[web] Amazon's AWS: EC2, EBS, S3, etc.
Author
Has anyone else worked with Amazon's Web Services lately? About a month or two ago I moved LucidChart onto an EC2 instance with them. Here are some observations:
You definitely need to be a system administrator to effectively use EC2 to host a traditional web site/web application. It took me about one full work day to get a basic LAMP stack up and running with all the cool features turned on, and I'm reasonably experienced in system administration in Linux.
The feature set, though, is so compelling that I will never be back to a traditional shared/VPS/dedicated hosting provider.
My persistent storage is logically separate from my server "instance", so in the case of my server crashing I can simply boot up a new one and attach my storage to it (e.g. database, code, etc). I recently moved to a higher-CPU instance at no downtime due to this feature.
I get instant incremental point-in-time snapshots of your storage (EBS, Elastic Block Store) volumes that automatically get backed up to multiple data centers. I have full hourly MySQL backups that only lock the database against new writes (not at all against reads) for about 1 second. This took a little engineering to get just right, but nothing too grueling.
Creating new MySQL slaves is therefore a zero-pain procedure. If you've ever set up MySQL replication before, you'll understand how important that is. I lock the database, dump a SHOW MASTER STATUS to the persistent storage, lock the file system, take a snapshot, and unlock the file system and database. This takes about 1-2 seconds. I then create a new MySQL server instance from a server image I have created, attach a new EBS volume made from that snapshot, set the slave parameters based on what's in the master status file, and start up MySQL. It's pretty much up to date with the master already, and there was basically no downtime.
Elastic IP addresses are what glue all this together. You can allocate up to 5 static IP addresses that you can remap instantly from one instance to another. So on major system upgrades, I just create a new instance from an image of my current instance, do whatever upgrade is necessary, and remap the production IP address to that instance. Done. Zero downtime.
And I only get charged for exactly what I use! No more "500GB bandwidth included, $2/GB overages". I pay $0.17/GB with no minimums. I pay by the server-hour for processing power and by the GB/month for persistent and backup storage (again, no minimums). They recently introduced "reserved" instances, which mean you can reserve a certain instance type for 1 or 3 years (at up-front cost) and then pay dramatically-reduced rates. I'm paying $0.03/hr for my MySQL master, having paid $500 upfront to reserve that instance for three years. That works out to something like $35/mo for a 1.4Ghz Xeon with 1.7GB of RAM.
One major frustration: You can't assign multiple elastic IP addresses to a single instance. So putting multiple SSL certificates on one Apache server is out (you can still get by with a UCC certificate for multiple domains).
Anyway, anyone else used AWS lately?
Nope. I'm pretty much the exact opposite of you. I don't like to be dependent on, or tied to anything. I run my own webserver, nameservers, jabber server, mailserver, etcetera. I even wrote my own SpamAssassin-like baysian spamfilter for my blog because I don't want to have to rely on external services like Akismet (I couldn't find any existing stand-alone comment spam filters). The only thing I need from the outside is an internet connection :-)
Sounds like it would be very cool, if I could ever get it set up. I'm an experienced developer but I hate system configuration and *nix especially.
Quote:Original post by Sander
Nope. I'm pretty much the exact opposite of you. I don't like to be dependent on, or tied to anything. I run my own webserver, nameservers, jabber server, mailserver, etcetera. I even wrote my own SpamAssassin-like baysian spamfilter for my blog because I don't want to have to rely on external services like Akismet (I couldn't find any existing stand-alone comment spam filters). The only thing I need from the outside is an internet connection :-)
Thing is, if you are trying to run a business, especially a start up whose business is built around the whole software as a service meme you need to be able to guarantee bandwith and uptime. More importantly you want to be outsource as much work which is not your strength of focus as is possible. While managing finances, attempting to get new clients, going to meetings, doing marketing, writing the software, the option to not have to or pay people to run and manage a robust scalable data centre is a very attractive option.
@BeanDog
Thank you for the review. It would be good to see other contrasting opinions who have used this service. I have heard of and been skeptical of this cloud computing hype. However the extremely cheap overall price for the hosting you are getting with such a glowing report speaks extremely well of the service. The ability to scale both costs and resource usage with almost no effort on your part makes this option seem a clear winner. Not to mention that under this, a well engineered system will grow costs much slower than users and wasting of resources and over charging will likely be less of an issue.
I suppose cloud computing makes sense in the same vein that people no longer grow their own food or make their own clothes, people will no longer host their own data.
I'm very interested in this. Thus far I have moved all of my storage to S3, but I I'm also seriously considering moving away from VPS and completely into a cloud based environment. You mention you are using EC2 as your web host. I've heard stories of EC2 instances going down, and of course there is no automatic failover. Are you using a framework such as weoceo, or did you roll your own to handle failover situations? Do you have separate EC2 instance(s) for your MySQL servers? Did you build your own AMI or use some of the off-the-shelf ones available, and if so how was the process? Sorry for the barrage of questions, but it is good to hear war stories from other guys!
Can you provide any additional info or useful links on how you achieved this? This seems to be a critical component in making a cloud based server both scalable and fault tolerant, but I'm having trouble seeing how to implement.
Quote:My persistent storage is logically separate from my server "instance", so in the case of my server crashing I can simply boot up a new one and attach my storage to it (e.g. database, code, etc). I recently moved to a higher-CPU instance at no downtime due to this feature.
I get instant incremental point-in-time snapshots of your storage (EBS, Elastic Block Store) volumes that automatically get backed up to multiple data centers. I have full hourly MySQL backups that only lock the database against new writes (not at all against reads) for about 1 second. This took a little engineering to get just right, but nothing too grueling.
Can you provide any additional info or useful links on how you achieved this? This seems to be a critical component in making a cloud based server both scalable and fault tolerant, but I'm having trouble seeing how to implement.
I do remote back-ups to S3 for about ~$3 a month of all my important code and business documents. I've also made use of EC2 for massively parallel computations, and am currently trying to get the background processes of my website to dynamically scale using EC2...
I like the products. There is no way I could host a cloud for my business yet, so it is nice having EC2 at my service.
I like the products. There is no way I could host a cloud for my business yet, so it is nice having EC2 at my service.
Author
Quote:Original post by CodeMunkie
You mention you are using EC2 as your web host. I've heard stories of EC2 instances going down, and of course there is no automatic failover. Are you using a framework such as weoceo, or did you roll your own to handle failover situations?
Especially during my early experimentation and implementation, I frequently deeply, horribly crashed my EC2 instance (my own stupidity... locking an XFS file system, then trying to write to it). This was frustrating at the time, but it gave me a great opportunity to see how failover would work in practice. Once I understood the basic process, it was really easy to make it as automated as can be reasonable. I haven't yet set up completely automated failover, although apparently it's pretty trivial with the use of AWS's new elastic load balancers. At this point, it's me logging into AWS's web-based management console, clicking to start a new instance from my server image, SSH'ing in and running a script. It takes about a minute or two, even if I'm setting up the replacement in a different data center.
Quote:Original post by CodeMunkie
Do you have separate EC2 instance(s) for your MySQL servers? Did you build your own AMI or use some of the off-the-shelf ones available, and if so how was the process?
I started with an off-the-shelf Fedora server image. I SSH'd in as root and installed and modified everything to be as I wanted it (e.g. moving webroot/MySQL data onto an EBS volume), then created an AMI from the server's current state. Creating an AMI, once I figured out how to do it, is actually pretty quick and painless. And the good news is that since everything in the file system that changes frequently (read: database) is on the EBS volume, I can make an updated AMI from the production server without any downtime.
Yes, I have since split my MySQL server from my web server for performance reasons.
Quote:Original post by CodeMunkieQuote:My persistent storage is logically separate from my server "instance", so in the case of my server crashing I can simply boot up a new one and attach my storage to it (e.g. database, code, etc). I recently moved to a higher-CPU instance at no downtime due to this feature.
I get instant incremental point-in-time snapshots of your storage (EBS, Elastic Block Store) volumes that automatically get backed up to multiple data centers. I have full hourly MySQL backups that only lock the database against new writes (not at all against reads) for about 1 second. This took a little engineering to get just right, but nothing too grueling.
Can you provide any additional info or useful links on how you achieved this? This seems to be a critical component in making a cloud based server both scalable and fault tolerant, but I'm having trouble seeing how to implement.
Creating EBS volumes and attaching them to instances can be done very easily using AWS's new online Management Console. Doing it automatically is also pretty easy using the command-line tools that Amazon provides for Linux (also Windows?).
It's important to use an XFS or other journaling, lockable file system to get real reliable EBS snapshots. Here's my actual EBS backup script that performs as I described:
#!/bin/bashexport EC2_HOME=/root/ec2-api-tools-1.3-36506/export JAVA_HOME=/usr/java/jre1.6.0_13/echo "SHOW MASTER STATUS;" > /root/showmasterstatus.sqlmysql -pXXXXXXXXXXX <<EOFPURGE BINARY LOGS BEFORE DATE_ADD(NOW(), INTERVAL -15 MINUTE);FLUSH TABLES WITH READ LOCK;SYSTEM mysql -pXXXXXXXXXXXX >/vol/aws/masterstatus </root/showmasterstatus.sqlSYSTEM xfs_freeze -f /volSYSTEM /root/ec2-api-tools-1.3-36506/bin/ec2-create-snapshot -K /vol/aws/private.pem -C /vol/aws/public.pem vol-45fa172cSYSTEM xfs_freeze -u /volUNLOCK TABLES;EOFBasically, the whole script is run through the MySQL CLI: First, I purge old binary logs (just as a space-saving measure). I then FLUSH TABLES WITH READ LOCK, which makes sure everything's written to disk and the database is locked against new writes. I then dump a SHOW MASTER STATUS to the EBS volume, so that if I set up a replication slave from this snapshot, I'll have all the information I need to have the slave catch up to the master from this snapshot's state. I freeze the file system and create the snapshot (vol-45fa172c is the EBS volume's ID number), and unfreeze the file system and database.
Not rocket science, but it took me a bit to figure that one out. As for rotating those backups, here's a script I made for that purpose:
#!/usr/bin/php -q<?phpif(sizeof($argv) != 3) { die("Usage: $argv[0] snapshot-id total-backups-to-keep\nExample: $argv[0] vol-45fa172c 100\n");}$Snapshots = array(); //Snapshot ID => Created Timestampputenv('EC2_HOME=/root/ec2-api-tools-1.3-36506/');putenv('JAVA_HOME=/usr/java/jre1.6.0_13/');$data = `/root/ec2-api-tools-1.3-36506/bin/ec2-describe-snapshots -K /vol/aws/private.pem -C /vol/aws/public.pem`;$data = split("\n", $data);foreach($data as $row) { if(trim($row) == '') continue; list($junk,$SnapshotId,$VolumeId,$Status,$Created,$Progress) = split("\t", $row); if($VolumeId == $argv[1] && $Status == 'completed') { echo "Found snapshot started at ".date('m/d/Y g:iA', strtotime($Created))."\n"; $Snapshots[$SnapshotId] = strtotime($Created); }}$Keep = PruneBackups($Snapshots, $argv[2]);//FYI: Date/time of backups to be kept.arsort($Keep);foreach($Keep as $id => $date) echo date('m/d/Y hA', $date)."\n";//Delete the ones not in $Keepforeach($Snapshots as $id => $date) { if(!isset($Keep[$id])) { echo "Kill $id - ".date('m/d/Y hA', $date)."\n"; echo `/root/ec2-api-tools-1.3-36506/bin/ec2-delete-snapshot -K /vol/aws/private.pem -C /vol/aws/public.pem $id`; }}/*We will keep 1/4 of the total requested backups from the last 24 hours, keeping all of the most recent.We will keep 1/2 of the total requested backups from the last 30 days, one per day.We will keep 1/4 of the total requested backups from older than 30 days, one per week.Any left over will go to the newest backups available.*/function PruneBackups($Snapshots, $Total, $Now=NULL) { if($Now === NULL) $Now = time(); $ret = $Snapshots; $Remaining = $Total; arsort($Snapshots); $UsedDays = array(); $UsedWeeks = array(); foreach($Snapshots as $id => $Created) { if($Created >= strtotime('-1 day', $Now)) { if($Remaining > $Total * 0.75) { $UsedDays[date('Y-m-d', $Created)] = true; $UsedWeeks[date('Y-W', $Created)] = true; $Remaining--; continue; } } if($Created >= strtotime('-30 days', $Now)) { if($Remaining > $Total * 0.25 && !isset($UsedDays[date('Y-m-d', $Created)])) { $UsedDays[date('Y-m-d', $Created)] = true; $UsedWeeks[date('Y-W', $Created)] = true; $Remaining--; continue; } } if($Remaining > 0 && !isset($UsedWeeks[date('Y-W', $Created)])) { $UsedWeeks[date('Y-W', $Created)] = true; $Remaining--; continue; } unset($ret[$id]); } arsort($Snapshots); foreach($Snapshots as $id => $Created) { if(!isset($ret[$id])) { $ret[$id] = $Created; $Remaining--; if($Remaining <= 0) break; } } return $ret;}?>I take backups hourly, and I keep 96 backups of each of my two EBS volumes (one is MySQL data, other is source/other webserver stuff). This script keeps the rotation sane, as described in a comment in the script.
As for setting up MySQL on XFS on EBS on EC2 in general, here's a tutorial I roughly followed: clicky
That tutorial was crucial in getting me in the right direction, but I had to do some things slightly differently--though I couldn't really tell you now exactly what.
Disclaimer: My scripts work for me. They may or may not work for you.
Quote:Original post by Daerax
Thing is, if you are trying to run a business, especially a start up whose business is built around the whole software as a service meme you need to be able to guarantee bandwith and uptime. More importantly you want to be outsource as much work which is not your strength of focus as is possible. While managing finances, attempting to get new clients, going to meetings, doing marketing, writing the software, the option to not have to or pay people to run and manage a robust scalable data centre is a very attractive option.
Attractive, but not smart. If you're running SaaS applications then hosting is part of your core business. If something is part of your core business then you should do it yourself. If you don't have the skill to do it yourself then you shouldn't be in the SaaS business but in the application business. Here's a good article about that.
Quote:Find the dependencies -- and eliminate them ... If it's a core business function -- do it yourself, no matter what.
Great info all around! Sander, do you mean that you have your own data center? What were the challenges setting that up and what were your capital costs? Also, how many users are you serving? Any issues scaling up? Love to hear alternative approaches.
Author
Quote:Original post by Sander
Attractive, but not smart. If you're running SaaS applications then hosting is part of your core business. If something is part of your core business then you should do it yourself. If you don't have the skill to do it yourself then you shouldn't be in the SaaS business but in the application business. Here's a good article about that.Quote:Find the dependencies -- and eliminate them ... If it's a core business function -- do it yourself, no matter what.
I completely disagree.
The core business of modern SaaS is the production of performant, scalable applications served through a web browser. The core business is not building and maintaining multiple data centers around the world, any more than the core business is developing a highly-performant and ACID-compliant database engine.
I'm not interested in recreating Oracle or MySQL. And I'm definitely not interested in maintaining physical network infrastructure. Some dependencies really are pragmatic.
In contrast, Google App Engine is an example of a horribly vendor-specific and locked-in dependency. To get the advantages of the app engine, you have to give up your choice of hosting options, and even take their database engine into account in your application's architecture.
If Amazon shut down AWS next month, I could be back up and running on Rackspace in a few days. If Google cut off their App Engine, their users are utterly screwed.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
