Recounting EC2 One Year Later
December 29th, 2010 | Published in Analysis | 20 Comments
It’s been over a year since my original Anatomy of an EC2 Resource ID post. In what became my little claim to fame in the industry, I uncovered the pattern behind those cryptic IDs AWS assigns to every object allocated (such as an instance, EBS volume, etc.). The discovery revealed that underlying the IDs is a regular serial number that increases with each resource allocated. While this may sound technical and insignificant, it turned out to be very valuable: it enabled, for the first time, a glimpse into the magnitude of Amazon’s cloud.
The numbers gathered in that post back in September 2009 showed that approximately 50,000 instances were being spun up every day on EC2 (in the us-east-1 region). So what’s happened since? I joined forces with CloudKick, providers of a cloud management and monitoring platform, to dig up more data. Here’s what we found: (click to expand to an interactive chart)
The chart above plots the number of instances launched per day, from mid-2007 till present day. Growth is, well, unmistakable. A couple of peaks dominate the landscape around February and October 2010, peaks which somewhat correlate to availability of new AWS services (see interactive map). The evidence is highly circumstantial though as I find it hard to draw a direct conclusion on how these specific events pushed the daily launch count as high up as 150,000.
Let’s zoom out now and look at EC2′s growth over the years:
For this chart, we averaged out the instance launch counts over each year (data for 2007 and 2010 may be partial). Based on the results, activity on EC2 has been multiplying several times over every year. The biggest step was 2008-2009, exhibiting 375% growth (that’s almost 5X). 2009 was the year AWS really exploded, but what strikes me as odd is that growth actually slowed down the following year (to 121%). One theory would be that the cloud has begun to saturate the early adopters and it is truly time to cross the chasm. Crossing, however, is turning out to be a difficult feat.
Responding to my previous research, a top Amazon official commented that a count of instance launches doesn’t really reflect anything meaningful (like the actual customer base, server count or revenues – all of which we’d all love to figure out). I respond that it’s examining the numbers one year later that provides the real value: it’s like looking at a mysterious dial on your car’s dashboard: even without understanding the exact parameter measured, if it shoots up then there’s a decent chance you’re driving faster.
December 29th, 2010 at 4:04 pm (#)
Congrats on the latest installment of reading the tea leaves :-)
I find it reasonable that growth would be less mad in 2010 compared with 2009, since the growth is from a higher base.
I’d love to continue the conversation on whether this is just “we’re bigger so we cannot grow so fast” or “cloud has begun to saturate the early adopters and it is truly time to cross the chasm”. I wonder how we could get data for this debate.
December 29th, 2010 at 4:08 pm (#)
[...] This post was mentioned on Twitter by Joe Devon and nubes. nubes said: Recounting EC2 One Year Later – http://tinyurl.com/2agn6uo #cloud [...]
December 29th, 2010 at 6:03 pm (#)
I think allot of the tools are still not as user friendly, especially when you look at how much people manage a dedicated server using a control panel to setup domains/email,etc.
December 29th, 2010 at 6:50 pm (#)
I should note for the record: Rackspace acquired CloudKick earlier this month. Our collaboration on this research was performed before the acquisition was announced.
December 29th, 2010 at 7:43 pm (#)
Great chart, Guy. Thanks for putting it up. Do you have any idea how long the average instance runs before it is shut down? If someone spins up an instance and keeps it up for a week, it will show up in your chart as a new instance only on the first day when it is created, right?
December 29th, 2010 at 7:56 pm (#)
@Turkey Fan – indeed, the instance will show up only on the day it was created. I don’t have any data on average instance lifetime I’m afraid.
December 29th, 2010 at 9:51 pm (#)
Guy, do you have any data on server size (small, med, large, hi CPU etc)? It would be interesting to correlate this data to the # of servers launched.
December 30th, 2010 at 12:31 am (#)
@Ray, unfortunately I don’t have this kind of data to share. I’ve seen some evidence in the past to suggest that small instances are the lion’s share (maybe even half) of launches.
December 30th, 2010 at 12:04 pm (#)
Great post Guy!
I think we will keep seeing a huge growth in 2011 (maybe not 100%+ but way above 50% for sure).
There are many companies that are just starting to explore this option and most of them are starting with AWS.
I assume you won’t mind me using some of these data as a reference in my blog (with credit).
December 30th, 2010 at 6:58 pm (#)
[...] שניתן לעקוב אחריו. שנה מאוחר יותר (למען הדיוק אתמול), מפרסם גיא את תוצאות המעקב שלו (יחד עם CloudKick) אחרי מספר השרתים [...]
December 30th, 2010 at 8:37 pm (#)
It’s possible that spikes in instance count correspond with new features (e.g. m2.xlarge instance type) simply because users want to try them out. For example, how many folks do you suppose launched m2.xlarge instances just to try them out, and then quickly shut them down? I know I did :-).
December 30th, 2010 at 8:48 pm (#)
P.S. I would love to see an updated chart that includes Black Friday and the ensuing holiday shopping season.
January 4th, 2011 at 3:11 am (#)
[...] Recounting EC2 One Year Later (From Jack of All Clouds) This estimate/analysis of instances launched sequentially over the past four years is very telling, even in its vagueness. What’s clear is that Amazon EC2 images are more prevalent every year. [...]
January 4th, 2011 at 7:06 pm (#)
[...] Jack of All Clouds [...]
January 11th, 2011 at 7:19 am (#)
[...] "76", "http://www.cloudcensus.com"); Guy Rosen has some interesting analysis as to the usage of EC2 in the Eastern U.S. data center.Notice the explosive growth in [...]
February 28th, 2011 at 1:51 pm (#)
Hi Guy
Great post indeed!
Is there a way to estimate the size of Amazon’s AMI market place out of all of EC2?
I’m trying to find out the market size of Machine Images or Virtual Appliances out of IaaS.
Thanks
Eitan
February 28th, 2011 at 2:00 pm (#)
The same logic I use for counting instances should apply to AMIs as well. You can use the decoding code here and feed it some AMI IDs. Should give you a feel for both the total number of images in a given region and the speed at which new images are added. I invite you to poke around and write a guest post here on the blog with your findings. Shoot me an email.
September 19th, 2011 at 4:14 am (#)
[...] services are growing fast despite the fact that a lot of people do not fully trust them. Just look at the number of launched Amazon EC2 instances in the datacenter US East 1 region (Courtesy of Jack of All Clouds). In 2008, the highest [...]
September 19th, 2011 at 12:08 pm (#)
[...] services are growing fast despite the fact that a lot of people do not fully trust them. Just look at the number of launched Amazon EC2 instances in the datacenter US East 1 region (Courtesy of Jack of All Clouds). In 2008, the highest peak [...]
September 19th, 2011 at 9:46 pm (#)
[...] services are growing fast despite the fact that a lot of people do not fully trust them. Just look at the number of launched Amazon EC2 instances in the datacenter US East 1 region (Courtesy of Jack of All Clouds). In 2008, the highest peak [...]