This week, the black box became a little bit whiter.

Sören Bleikertz, a computer science student writing his Masters thesis on EC2 security, poked into the Xen hypervisor used by EC2 and made some observations regarding EC2’s underlying architecture. Among his findings on the storage and networking configurations, Sören pointed out that each instance was given a unique name (the “Xen domain”) such as dom_32504936 and that this seemed to behave like a serial number, growing from day to day. Sound familiar yet?

Well, it turns out that this Xen domain is none other than the underlying instance ID uncovered in my previous research! This revelation gives us an important conclusion: the decoding method was accurate. The serial number exists and based on everyone’s input we even got the formula right.

With Sören’s technique at hand we can now uncover the constants needed for all EC2 regions. Except for us-east-1 which thanks to RightScale enjoyed a 3-year history, we did not have enough data to extract the constants for other regions. Surprisingly, it turns out that the constants are in fact identical for all regions. What threw us off the scent is that as opposed to us-east-1 which very likely started the serial number from zero, the other regions do not. For example, the serial numbers for the 3-month-old us-west-1 region are already in the range of 752 million. Those for eu-west-1 are in the 500 million range. We can safely assume that hundreds of millions of instances have not in fact been spun up. What makes more sense is that each region was assigned a different starting point in order to ensure globally unique instance IDs.

An additional finding of Sören’s is that the image file for the root disk points to a filename on the VM host such as /mnt/instance_image_store_3/262768. It turns out that the number at the end of this file is, again, simply the AMI ID – decoded. For example, we can re-encode 262768 to yield ami-19a34270, which is Alestic’s Ubuntu Karmic Base image. Similar to instance IDs, the underlying image ID also seems to have different ranges in each AWS region.

As a bonus of Sören’s discoveries and the connection to the IDs, it’s now possible to infer your instance ID (and image ID) locally, without even consulting the EC2 user-data. Why someone would prefer this to the user-data is a good question, but it’s a fun exercise nonetheless. Here’s a Ruby script that does just that:

#!/usr/bin/ruby
$stderr.puts("Detecting VM domain ID (may take a few moments)")
dom_id = nil
(1..65535).each do |i|
        if system("xenstore-ls /local/domain/#{i} > /dev/null 2>&1")
                dom_id = i
                break
        end
end

$stderr.puts("VM domain ID is #{dom_id}")

dom_name = `xenstore-read /local/domain/#{dom_id}/name`
$stderr.puts("VM domain name is #{dom_name}")

numeric_id = dom_name.split("_").last.to_i
c1 = numeric_id >> 24
c2 = (numeric_id >> 16) & 0xFF
c3 = numeric_id & 0xFFFF
c3_1 = (numeric_id >> 8) & 0xFF
c3_2 = numeric_id & 0xFF

d1 = c1 ^ c3_2 ^ 0x69
d2 = c2 ^ c3_1 ^ 0x40 ^ 0xe5
d3 = c3 ^ 0x4000

instance_id = sprintf("i-%02x%02x%04x", d1, d2, d3)
puts(instance_id)

This requires xen-utils to be installed on the machine (on Ubuntu, run apt-get install xen-utils-3.3). Here’s an example run:

# ./get_instance_id.rb
Detecting VM domain ID (may take a few moments)
VM domain ID is 1423
VM domain name is dom_32900610
i-6a554602

Thanks once more to Sören for the great detective work.

To the results!

First, a plot of the instance counts for each region rising over the course of the day and a half sampled.

Next, a more direct comparison of the count of instances launched per day.

Conclusions?

• us-west-1 is a great success. Just two weeks after its official launch its level of activity is 73% of eu-west-1, which has been around for a whole year!
• us-east-1 continues to maintain a giant (and unsurprising) lead. We can also observe that its numbers are in the same range as the measurements back in September.

It will be interesting to see when (in my opinion it’s “when” not “if”) us-west-1 surpasses eu-west-1. It will probably not take too long. What is more intriguing is to try and guess the relative sizes, say, 1 year from now. Will us-east-1 retain its lead? While it enjoys its status as the default for AWS operations, the new region will definitely put up a fight due to its proximity to the California-centered tech industry.

One segment, applications with Salesforce integration, were up to now severely limited due to Salesforce’s datacenter being located in California and Amazon’s in Virginia. Many chose simply not to go for AWS. The new region now provides this segment with low-latency communications between EC2 and Salesforce. This segment alone may turn out to be a significant factor in the growth of us-west-1.

I’ll continue to follow from time to time. AWS are planning some new Asia regions in 2010, first Singapore and then Japan, so there will definitely be lots of data to keep us busy!

Happy Holidays!

Quite like GoGrid, Rackspace Cloud Servers’ systems make the task of measuring usage quite straightforward. Server IDs are serial numbers and there are no hoops to jump through in order to make the calculations. I set about collecting samples over a period of approximately two weeks. Here’s what I found:

Over the timespan of two weeks 7241 servers were provisioned by Rackspace Cloud users. On average the result is 488 servers/day.

Of the three providers surveyed, the Rackspace Cloud retains its position in second place. While among the top-ranked public websites Rackspace comes in at a close second, in terms of servers provisioned the gap is tremendous. The Rackspace Cloud is ahead of GoGrid’s 181 servers/day but still a hundred times smaller than Amazon’s whopping 50,000 servers/day.

As I speculated regarding GoGrid, some of the difference might be explained by the more elastic nature of a typical Amazon deployment and the ecosystem of tools available for Amazon. I do suspect that Rackspace may be gaining traction at least among the sporadic development and testing use cases – simply due to their low entry-level pricing. I myself have found that if in need of a quick server with minimal demands then Rackspace’s $0.015/hour definitely beats Amazon’s $0.085/hour.

In today’s post, we’ll be discussing such an analysis of GoGrid.

GoGrid’s technology seems to make the process almost trivial – the server ID that is assigned to each and every server provisioned appears to be a straightforward serial number. Hence, this time we need not find patterns or XOR bytes, and can instead get down to business. As always, I will warn that the numbers are circumstantial and only GoGrid knows if they are the real thing.

In total, during a time span of just over 13 days this research witnessed 2413 servers provisioned on GoGrid. On average, that comes to approximately 181 servers launched per day.

How does this compare to Amazon EC2? We calculated 50,000 servers/day for EC2, so the difference is significant. To put it another way – by this count, if GoGrid was the size of the Texas then EC2 would be the Pacific Ocean. Let’s try to dig a little deeper though: this measurement is more a reflection of a provider’s level of elasticity than of its absolute size. With a thriving ecosystem of tools and services such as Elastic Load Balancing and Elastic MapReduce it’s not surprising to find the use of EC2 servers is more dynamic.

On a personal note, I must say I was put off by the time it takes to launch a server on GoGrid (sometimes up to 8 minutes) as compared to what I was used to from EC2. For me, that means that if I need a quick server just to run something for a minute or two there would be no question.

Coming up next time – a similar analysis of Rackspace Cloud Servers.

• RightScale decided to apply the findings to the mountain of EC2 data they have – a few years worth. Firstly, this solved a few of the remaining puzzles in the ID formula, on the Series ID and Superseries ID (I’ve updated the original post to reflect this). Moreover, the wider perspective led them to estimate that the total number of instances launched is actually a whopping 15.5 million. RightScale’s full findings can be found here.

• Randy Bias published an excellent post stating that Amazon’s EC2 Generating 220M+ Anually – based on “actual verified EC2 numbers plus some guesses and a rough model of its current annual usage”. Bias’s sources tell him Amazon has approximately 40,000 servers running (note that’s servers not instances).

So what is the bottom line? CIO Magazine’s Bernard Golden described it well in his coverage of my research, concluding that “it’s hard to look at these numbers and not conclude that something big is going on, and not just in “toy” applications.”

Something big indeed.

Each time you allocate a resource using EC2 – an instance, a volume or a snapshot – you receive a unique identifier. This is the EC2 resource ID. Have you ever wondered what this ID represents? Well, I did. After noticing similarities between the IDs of resources requested in close succession, I started digging.

The outcome of this digging is a definition of the components that formulate an EC2 resource ID. The marvel is that this definition allows us to externally count the number of resources provisioned within a certain time frame – enabling us for the first time to observe EC2’s usage patterns. For example, we can count how many instances are launched on a certain day, in a given EC2 region.

Before continuing, I’d like to emphasize that these findings are circumstantial. While the patterns are indisputable, there remain unknowns and quirks that remind us that such “black box” observation has its limits. Note also that we can estimate how many new resources are created but not how many are already active, how many were later deleted, etc. The total number of servers running on EC2 remains a mystery.

Results

In one 24-hour period measured in September 2009, the estimation indicated the following volume of usage on Amazon EC2’s us-east-1 region:

• 50,242 instances requested
• 12,840 EBS volumes requested
• 30,925 EBS snapshots requested
• 41,121 reservations requested
  • Disambiguation: a reservation in this context is an atomic launch of one or more instances. This does not imply a reserved instance. For example, if you launch 1 instance, you get 1 instance ID and 1 reservation ID; if you launch 2 instances in one command, you get 2 instance IDs and still 1 reservation ID.

These numbers are impressive, to say the least. Even more impressive is a small hint, lurking between the numbers, that implies that just over the past month Amazon crossed a significant threshold (see below for more details):

8.4 million EC2 instances launched (since EC2’s debut).

UPDATE (Oct 7th 2009): RightScale applied the findings for the two years worth of data they have in their systems. Based on that data, they estimate the number of instances launched is actually 15.5 million! They also plotted the numbers over two years – worth checking out.

Anatomy of a Resource ID

So how were the numbers above calculated? To find out, let’s decompose an EC2 resource ID. After comparing hundreds of IDs, this opaque identifier turned out to be a little more transparent than you’d expect.

Type

The most trivial of the fields, the type is one of the following values, depending on the resource type:

• i – instance
• r – reservation
• vol – EBS volume
• snap – EBS snapshot
• ami – Amazon machine image
• aki – Amazon kernel image
• ari – Amazon ramdisk image

Inner ID

The Inner ID is a 16-bit counter of resources allocated. Each time a resource is requested, the Inner ID increments by one. For instance and reservation IDs, it increments by two (i.e., these Inner IDs are always even). Instead of counting from 0-FFFF as you’d expect, the Inner ID uses the following cycle:

• 4000-7FFF
• 0000-3FFF
• C000-FFFF
• 8000-BFFF

(This cycle can be easily normalized by XORing with 4000.) When the Inner ID has exhausted its space, a new series begins (see below) and the cycle restarts.

Series Marker

For a given resource type, there is one active 8-bit Series ID. This Series ID, however, is not embedded directly into the resource ID. Instead, it is XORed to the leftmost 8 bits of the Inner ID. The result, which I call the Series Marker, is embedded in the ID to the left of the Inner ID.

For example, on the resource ID above the Series ID would be e5 = a7 XOR 42.

Series IDs usually decrement by one each time the Inner ID completes a cycle. I say “usually” because while this is the most common behavior, from time to time Series IDs seem to jump around in a pattern which is yet to be explained.

UPDATE (Oct 7th 2009): RightScale contributed the missing piece: to normalize a series ID, XOR with E5 – this irons out the “jumps” I noticed perfectly.

Superseries Marker

For a given resource type, there is one active 8-bit Superseries ID. Like the Series ID, the Superseries ID is not embedded directly into the resource ID. Instead, it is XORed to the rightmost 8 bits of the Inner ID. The result – the Superseries Marker – is the leftmost byte of the resource ID.

For example, on the resource ID above the Superseries ID would be 69 = 31 XOR 58.

The Superseries ID changes so rarely that originally I had assumed it was some kind of checksum. This would have been odd as it limits the total available IDs to 2²⁴ = 16.8 million. Up to very recently, the Superseries ID for all resource types – instances, images, volumes, snapshots, etc. – was 69 (in the us-east-1 region (for eu-west-1 the Superseries ID is 74). These days, new instances use the Superseries ID 68. This subtle change, unnoticed by the industry, may hint at an astonishing achievement: 8.4 million instances launched since EC2’s debut! (Instance IDs are even so 8.4M = 16.8M / 2.)

UPDATE (Oct 7th 2009): RightScale suggested to normalize the Superseries ID by XORing with 69. In this technique, the superseries ID for us-east-1 was 0, and the recent change incremented it to 1.

Regions

Note that since each EC2 region is a completely separate system, the IDs in each region are independent of each other.

Counting Resources

Now that we have an idea of what an ID represents, how do we use that knowledge to estimate the number of resources provisioned by EC2 in a given time frame? The process is quite straightforward, and can be applied to time frames ranging from minutes up to weeks, months and years.

During the 24-hour period measured, one resource of each type was requested from EC2 every hour. In practice this means an instance was launched, an EBS volume was created and an EBS snapshot taken. The IDs that EC2 assigned to these resources were recorded, along with the time of their creation (as indicated in the timestamp returned from EC2 itself). Finally, the resource were released (instance terminated, volume and snapshot deleted) in order to minimize expenses. This process repeated every hour, which seems to be frequent enough so as not to miss any series rollovers.

The results – IDs and timestamps – were then analyzed using a combination of scripts and Excel spreadsheets. The Superseries, Series and Inner IDs were extracted from the resource IDs. Finally, the IDs were normalized and combined to yield a single number – a number that represents the continuum of resource IDs.

With this number, it’s plain sailing to measure or plot how many resources EC2 provisioned between any two samples.

Summary

The analysis, measurements and description above are based purely on observation. I cannot make any guarantees as to the accuracy of the technique.  Even with confidence regarding the analysis of an ID, whether or not we can use that to infer overall usage is open to debate. In theory, Amazon could be allocating resources internally for various purposes. Is this performed on a scale large enough to throw the figures off course? Only time (and Amazon) will tell.

Final word: if you have any insights, corrections or additions to this research – please feel free to jump in the conversation or email me. I’ll be sure to give credit in updates or future posts.

Thanks to Eric Hammond, Nati Shalom and Avner Algom and Peter Weinstein of the IGT for reading drafts of this.