After a brief intermission, we’re back this month with an action-packed State of the Cloud report. In this month’s analysis of the top cloud providers we’ll be debuting a newcomer into the charts which makes quite an entrance. We’ll also run the analysis with an alternative data set and see if it confirms or refutes our findings.

Snapshot for July 2010

Here are the results for this month. Welcome Linode.

The top pair continue their steady march forward with 13% and 19% growth for Amazon EC2 and Rackspace Cloud Servers respectively, as compared to the last report two months ago. Amazon EC2 is the first of our contenders to smash the 3000-site barrier.

Linode is this month’s surprise, jumping straight into third place. I was deliberating whether Linode should be included in the report. Following some lively discussions on Twitter, the consensus was that Linode looks like a duck and walks like a duck, even if it doesn’t bother quacking. (This, as opposed to some providers that work hard to market themselves cloud while they don’t really seem to be.)

Linode offer a rich feature set and have an outstanding reputation among their customer community. The last time I saw this picture it was Slicehost, who ended up being acquired by Rackspace to jumpstart their Cloud Servers offering. What does the future hold for Linode?

Trends

Uncovering Linode’s footprint in the historical data collected, we witness remarkable growth. Linode has grown by 270% over the past 11 months, more than any other of the providers tracked.

Alternative Data Sets

Last month we skipped a report due to a bug in the top site list published by Quantcast, which serves as the input for this research. This prompted me to consider other data sets that could also be used. So what would our results be like if we used another well-known source of top site rankings? I chose to take Alexa, a well-known source, for a spin and see what happens. Here’s what I found -

I was pleased to find the results do not differ greatly between the different sets. The overall rankings are preserved although some providers do exhibit some variation (particularly Rackspace, which loses over 20% when we use Alexa). Even if it is tempting to switch data sets, continuity is of grave importance for research like this one. The findings above do confirm that the standings are in the same ballpark even when viewed through a different lense. Hence I am happy to continue using Quantcast (well, unless last month’s issues pop up again!).

Next month marks a year since the first State of the Cloud. This milestone will serve as an opportunity to look back and see what progress and changes we’ve seen over the past year – as well as to take a guess at what the future may hold for this industry.

We interrupt our regular programming for the following update.

Unfortunately, this month we have no regular post in the State of the Cloud series. Quantcast’s Top 1M Site list, which is the input data set used by the research, is broken. Instead of 1 million sites, the list contains only 78,000. Clearly, this makes it difficult to continue tracking using the same, consistent methodology we’ve had in place for almost a year.

Quantcast are aware of the problem, but “do not have an ETA on the fix”.

What you can do

Help us prioritize Quantcast’s investigation of this issue: go to http://www.quantcast.com/contact and tell Quantcast what you think. Here’s an example of what you could write:

A new month, a new State of the Cloud post! In this month’s post we’ll revisit the relative sizes of the top providers and see just how much of the cloud market the biggest players own.

But first, this month’s figures -

Looking at this month’s trends, we find a relatively slow month for the cloud. Overall growth was just 1.7%. The largest month-to-month percentage growth goes to OpSource which grew by 11%. In terms of absolute size, Rackspace grew the most with 70 new sites. Amazon gained just 10 new sites this month.

Let’s take a look at cloud providers from another angle. We last did this back in February.

Our first conclusion: Amazon now controls more than 50% of cloud-hosted sites. The second conclusion: by this metric at least, the cloud race continues to be a two-horse race: Amazon and Rackspace together control 94%, and all the rest of the providers retain but a sliver of control.

Welcome to update #10 in the regular State of the Cloud series. This month we’ll continue to examine how many of the world’s top websites are using cloud providers.

Many of the smaller providers have had a weak month, some even showing up less this month in the sample than they did previously. Only Amazon and Rackspace continue to plough ahead with Amazon gaining 6% and Rackspace 3%.

Overall cloud growth this month is at 3.9% (=58% CAGR).

A couple of weeks ago I gave a keynote talk at CloudConnect to share some of the findings that I’ve been publishing here on the blog. CloudConnect turned out to be a great event which really brought together many people from the cloud community – this is a good opportunity to congratulate and thank the organizers for a job well done.

Here’s the video of my talk and the presentation.

Welcome to this month’s update on adoption of cloud providers by public-facing websites. For details on how this is calculated, please see the first post in the series. This month we’ll take a step back and try to appreciate the overall size of the cloud as a whole. But first, to the regular numbers -

Snapshot for March 2010

Total Cloud Adoption

It’s easy to get caught up in the differences between each provider, so this time let’s try to zoom out for a moment. Last month I noted that all these providers combined still hold less than 1% of the sites surveyed. Well, this month that threshold was crossed. The cloud now constitutes 1.01% of our sample. There is plenty of room to grow…

Last Tuesday I joined John Rowell for a chat about cloud trends. Below is the recording of the show.

Like many in the cloud community, these days I am gearing up for CloudConnect in Santa Clara next month. If you’re going to be there, be sure to catch me and say hi. Don’t miss my appearance on the main stage on the morning of Tuesday 16th.

See you there!

Next Tuesday I’ll be joining John Rowell, OpSource co-founder and CTO, for the second episode of The Real Cloud. In the spirit of this blog, we’ll be chatting about cloud adoption trends and calling out the real numbers behind the hype.

Be sure to come and watch, as well as to post your questions during the show.

Tuesday, Feb 16th at 11am PST. Register here.

Back in September, I published the Anatomy of an EC2 Resource ID where I pointed out some curious patterns in EC2’s ID scheme and proposed a method of “decoding” these patterns to reveal an underlying serial number. In that post I was careful to write that “while the patterns are indisputable, there remain unknowns and quirks that remind us that such “black box” observation has its limits”.

This week, the black box became a little bit whiter.

Sören Bleikertz, a computer science student writing his Masters thesis on EC2 security, poked into the Xen hypervisor used by EC2 and made some observations regarding EC2’s underlying architecture. Among his findings on the storage and networking configurations, Sören pointed out that each instance was given a unique name (the “Xen domain”) such as dom_32504936 and that this seemed to behave like a serial number, growing from day to day. Sound familiar yet?

Well, it turns out that this Xen domain is none other than the underlying instance ID uncovered in my previous research! This revelation gives us an important conclusion: the decoding method was accurate. The serial number exists and based on everyone’s input we even got the formula right.

With Sören’s technique at hand we can now uncover the constants needed for all EC2 regions. Except for us-east-1 which thanks to RightScale enjoyed a 3-year history, we did not have enough data to extract the constants for other regions. Surprisingly, it turns out that the constants are in fact identical for all regions. What threw us off the scent is that as opposed to us-east-1 which very likely started the serial number from zero, the other regions do not. For example, the serial numbers for the 3-month-old us-west-1 region are already in the range of 752 million. Those for eu-west-1 are in the 500 million range. We can safely assume that hundreds of millions of instances have not in fact been spun up. What makes more sense is that each region was assigned a different starting point in order to ensure globally unique instance IDs.

An additional finding of Sören’s is that the image file for the root disk points to a filename on the VM host such as /mnt/instance_image_store_3/262768. It turns out that the number at the end of this file is, again, simply the AMI ID – decoded. For example, we can re-encode 262768 to yield ami-19a34270, which is Alestic’s Ubuntu Karmic Base image. Similar to instance IDs, the underlying image ID also seems to have different ranges in each AWS region.

As a bonus of Sören’s discoveries and the connection to the IDs, it’s now possible to infer your instance ID (and image ID) locally, without even consulting the EC2 user-data. Why someone would prefer this to the user-data is a good question, but it’s a fun exercise nonetheless. Here’s a Ruby script that does just that:

#!/usr/bin/ruby
$stderr.puts("Detecting VM domain ID (may take a few moments)")
dom_id = nil
(1..65535).each do |i|
        if system("xenstore-ls /local/domain/#{i} > /dev/null 2>&1")
                dom_id = i
                break
        end
end

$stderr.puts("VM domain ID is #{dom_id}")

dom_name = `xenstore-read /local/domain/#{dom_id}/name`
$stderr.puts("VM domain name is #{dom_name}")

numeric_id = dom_name.split("_").last.to_i
c1 = numeric_id >> 24
c2 = (numeric_id >> 16) & 0xFF
c3 = numeric_id & 0xFFFF
c3_1 = (numeric_id >> 8) & 0xFF
c3_2 = numeric_id & 0xFF

d1 = c1 ^ c3_2 ^ 0x69
d2 = c2 ^ c3_1 ^ 0x40 ^ 0xe5
d3 = c3 ^ 0x4000

instance_id = sprintf("i-%02x%02x%04x", d1, d2, d3)
puts(instance_id)

This requires xen-utils to be installed on the machine (on Ubuntu, run apt-get install xen-utils-3.3). Here’s an example run:

# ./get_instance_id.rb
Detecting VM domain ID (may take a few moments)
VM domain ID is 1423
VM domain name is dom_32900610
i-6a554602

Thanks once more to Sören for the great detective work.

Welcome to the eighth update for the State of the Cloud series. In case you are just joining – this monthly report measures the adoption of leading cloud providers amongst public-facing websites. The data set is based on the top 500,000 sites as measured by QuantCast. As always I’d like to point out the caveats of this method which were laid out in the first post in the series.

This month UK-based FlexiScale joins the report. FlexiScale has been on the radar for a while, but due to the US-oriented nature of QuantCast’s data it was so under-represented that it was as invisible. Since then, FlexiScale’s footprint has expanded enough so as to make it reasonable to include it the report. It’s worth mentioning that FlexiScale is probably still under-represented, so its standings should be taken with a pinch of salt.

Snapshot for February 2010

This month I’m plotting the numbers both in the usual bar chart and also in pie chart format. The pie-chart really visualizes that there are two leagues here. it’s increasingly difficult to see anyone from the minor league crossing over to the major league, which holds a whopping 93% of the total cloud-hosted sites found. Amazon EC2 hosts a shade under 50% of these sites while Rackspace Cloud Servers hosts 43%. Joyent, GoGrid, OpSource and FlexiScale together comprise just 7% of all cloud-hosted sites found.

I should remind readers that these percentages are not of the full 500,000 sites surveyed. In fact, all the cloud providers together still host a meager 1% of the sample.

Trends

Here are the results as observed over the past months:

Tune in next month for further updates!