We here at BioTeam have been kicking tires and generally exploring around the edges of the new Amazon cc1.4xlarge “compute cluster” EC2 instance types. Much of our experimentation has been centered around simplistic benchmarking techniques as a way of slowly zeroing in on the methods, techniques and orchestration approaches most likely to have a significant usability, performance or wallclock-time-to-scientific-results outcome for the work we do professionally for ourselves and our clients.

We are asking very broad questions and testing assumptions along the lines of:

• Does the hot new 10 Gigabit non-blocking networking fabric backing up the new instance types really mean that “legacy” compute farm and HPC cluster architectures which make heavy use of network filesharing possible?
• How does filesharing between nodes look and feel on the new network and instance types?
• Are the speedy ephemeral disks on the new instance types suitable for bundling into NFS shares or aggregating into parallel or clustered distribtued filesystems?
• Can we use the replication features in GlusterFS to mitigate some of the risks of using ephemeral disk for storage? 
• Should the shared storage built from ephermeral disk be assigned to “/scratch” or other non-critical duties due to the risks involved? What can we do to mitigate the risks?
• At what scale is NFS the easiest and most suitable sharing option? What are the best NFS server and client tuning parameters to use? 
• When using parallel or cluster filesystems like GlusterFS, what rough metrics can we use to figure out how many data servers to dedicate to a particular cluster size or workflow profile?

GlusterFS & NFS Initial Testing

Over the past week we have been running tests on two types of network filesharing. We’ve only tested against a single client so obviously these results say nothing about at-scale performance or operation.

Types of tests:

1. Take the pair of ~900GB ephemeral disks on the instance type, stripe them together as a RAID0 set, slap an XFS filesystem on top and export the entire volume out via NFS
2. Take the pair of ~900GB ephemeral disks on the instance type, slap a single large partition on each drive, format each drive with an EXT3 filesystem and then use GlusterFS to create, mount and export the volume via the GlusterFS protocol

For each of the above two test types we repeatedly ran (at least 4x times) our standard bonnie++ benchmark tests (methodology described in the earlier blog posts). The tests were run on a single remote client that was either NFS mounting or GlusterFS mounting the file share.

GlusterFS parameters

• None really. We used the standard volume creation command and mounted the file share via the GlusterFS protocol over TCP. Eventually we want to ask some of our GlusterFS expert friends for additional tuning guidance

NFS parameters:

• Server export file:  ”/nfs    <host>(rw,async)?”
• NFS Server config: boosted the number of nfsd daemons to 16 via edits to /etc/sysconfig/nfs file
• Client mount options:  ”mount -t nfs -o rw,async,hard,intr,retrans=2,rsize=32768,wsize=32768,nfsvers=3,tcp <host>:/nfs /nfs-scratch?”

Lessons Learned So Far  - NFS vs GlusterFS

• GlusterFS was incredibly easy to install and creating and exporting parallel filesystem shares was straightforward. The methods involved are easily scripted/automated or built into a server orchestration strategy. The process was so simple that initially we were thinking that GlusterFS would be our default sharing option for all our work on the new compute cluster instances
• GlusterFS has ONE HUGE DOWNSIDE. It turns out that GlusterFS recommends that the participating disk volumes be formatted with an ext3 filesystem for best results. This is … problematic … with the 900GB ephemeral disks because formatting a 900 gb disk with ext3 takes damn near forever. We estimate about 15-20 minutes of wallclock time wasted while waiting for the “mkfs.ext3″ command to complete.
• The wallclock time lost to formatting ext3 volumes for GlusterFS usage is significant enough to affect how we may or may not use GlusterFS in the future. Maybe there is a different filesystem we can use with a faster formatting profile. Using XFS and software RAID we can normally stand up and export filesystems in a matter of a few seconds or a minute or two. Sadly, XFS is not recommended at all with current versions of GlusterFS.
• Using GlusterFS with the recommended ext3 configuration seems to mean that we have to accept a minimum delay of 15 minutes or even more when standing up and exporting new storage. This is unacceptable for small deployments or workflows where you might only be running the EC2 instances for a short time.
• The possibility of using the GlusterFS replication features to mitigate against the risks of using ephermeral storage might be significant. We need to do more testing in this configuration. 
• Given the extensive wallclock time delays inherent in waiting for ext3 filesystem formatting to complete in a GlusterFS scenario it seems likely that we might default to using a tuned NFS server setup for (a) small clusters & compute farms or (b) systems that we plan to stand up only for a few hours.
• The overhead of provisioning GlusterFS becomes less significant when we have very large clusters that can benefit from the inherent scaling ability of GlusterFS or when we plan to stand up the clusters for longer periods of time

Benchmark Results

In all the results shown below I’ve included data from a 2-disk RAID0 ephemeral storage  setup. This is so that the network filesharing data can be contrasted against the results seen from running bonnie++ locally.

Click on the images for a larger version.

For background and summary writeups of all the various blog posts we have dealing with the new Amazon EC2 “compute cluster” cc1.4xlarge instance types please refer to this summary page: http://blog.bioteam.net/2010/07/19/exploring-the-new-aws-compute-cluster-ec2-instances/?

Related post

We talked about the performance of the boot and ephemeral storage in this post: http://blog.bioteam.net/2010/07/19/local-storage-performance-of-aws-cluster-compute-instances/?

This post

In this post I’ve finally collected enough data to cover repeated bonnie++ benchmark tests against all the main types of block storage available to the new Amazon cc1.4xlarge “compute cluster” instance types:

• Local boot disk performance
• Performance of a single ephemeral disk
• Performance of the two available ephemeral? disks when striped together with software RAID0
• Performance of a single EBS volumes attached to the instance
• Performance of 4 EBS volumes striped with RAID0 and attached to the instance
• Performance of 8 EBS volumes striped with RAID0 and attached to the instance?

Lessons Learned

1. Don’t use the boot disk for anything other than booting the operating system. As the results show, the performance of the (no PV driver support) cc1.4xlarge boot disk is the slowest of all possible block storage options available. Really slow. Not worth using for anything other than OS stuff. This also includes not bothering to mess with the size of the available volume.

2. The instance ephemeral storage volumes are fast and should not be ignored. Every cc1.4xlarge EC2 instance comes with a pair of ~840GB ephemeral disk volumes. In living by our own “don’t trust anything to non-persistant storage!” best practice we are guilty of ignoring these drives in situations where they could have been of significant benefit. This will change. The performance of a single ephemeral volume beats the performance we see out of a single EBS volume. A striped pair of ephemeral volumes performs even better and stacks up well even to multiple EBS volumes striped together. The RAID0 pairing of the two ephemeral drives seems to consistently outperform even 8-drive EBS RAID0 volumes when you look at the bonniee++ results for random and sequential file creation and deletion tests. This has major implications for HPC and scientific pipeline processing on the cloud. In particular I can easily envision using the ephemeral drives to build a shared parallel scratch filesystem (think PVFS or GlusterFS) in cluster configurations. This would give you a nice shared scratch storage pool. Even in simpler cluster setups it looks like it would be a win to stage data into the ephemeral storage so it can be used as the target drive for  scientific processing (where the input data is not unique and has backup copies elsewhere). We can run the IO heavy analysis against the fast ephemeral storage and send our result data into S3 buckets or a proper EBS volume for downstream handling.

3. Striping EBS volumes into software RAID0 sets is a valid practice. We clearly see performance gains when using more than one EBS volume, the performance gain is significant enough to justify the hassles involved in backing up and protecting EBS-resident sofware RAID sets. We need to do more work (and really need to test 2-volume EBS stripes) but it is clear that there is a measurable performance gain to be had. Not sure if we’d use 8-disk RAID0 sets for production work but looking at 2-disk and 4-disk methods is something that we will be looking seriously at.

Obviously there is much more to be drawn from the data but benchmarking is hard (and controversial) in regular settings let alone trying to get repeatable and consistent results out of a virtualized multi-tennant cloud framework. For now I’d prefer to stick to broad general conclusions and “lessons learned” rather than trying to divine highly specific things out of the raw data.

Test Results

As usual, you can find all of the raw data in this google spreadsheet. We did not finesse the data at all, the only data munging we did was to run tests repeatedly and then average out the results in order to arrive at the numbers used in the graphs.

Here are the numerical numbers behind the graphs, click on the image for the full-size version:

And here are the graphs. We’ve broken out the graphs to represent the results measured in “K/sec” versus just ” /sec”.

Read, Write & Rewrite results for all cc1.x4large storage types (click on image for full size):

Sequential create, delete & seek results for all cc1.x4large storage types (click on image for full size):

In our work with Amazon Web Services we try to spend as much time as we can “kicking the tires” so we become better at building stuff that we and our clients can actually use. We also try to share our information as much as possible in the spirit of scientific collaboration & honest exchange.

This is a ’roundup’ or summary blog post where we’ll list out blog post or articles that discuss the new Amazon cc1.4xlarge Compute Cluster instances.

• Grid Engine on the new Amazon Compute Cluster Instances
• How to resize an EC2 AMI when the boot disk is on EBS
• Preliminary EBS performance on Amazon Compute Cluster cc1.4xlarge instance types
• Local storage performance of AWS cluster compute instances
• Combined summary of local, ephemeral & EBS storage on cc1.4xlarge instance types
• NFS & GlusterFS network filesharing on the cc1.4xlarge instance types

Key Results:

• Performance of the root/boot disk is way slower than any other type of block based storage. This is to be expected as the boot disk (even though it comes via an EBS-resident AMI) does not get the benefit of paravirtualization acceleration. The take home message is that the boot/root disk volume should not really be used for anything. This also means that this blog post showing how to increase the size of the local OS disk is useful only for playing around and not for anything serious
• The performance of the ephemeral disks is better and striping the two available drives together as a RAID0 volume has measurable benefits across the board

What this means in the real world:

1. Don’t use the boot/root disk for anything but the OS and don’t bother trying to expand it’s size
2. It is reasonable to stripe the ephemeral storage together and use it for “real” work, especially as indications are that the speed may be faster than an EBS mounted volume.

Other people have mentioned that this is worth doing even if one includes the time it takes to rsync or stage data into the ephemeral storage. Future BioTeam cluster building practices may use the ~800GB of ephemeral storage to service a NFS or parallel filesystem that offers input data to pipelines running on EC2 compute farms. Since we can’t trust ephemeral storage for anything unique we’d have a second shared filesystem (backed by EBS) to handle capturing pipeline results.

Obviously there is one other comparison to make — how do these performance numbers measure against the 1-disk. 4-disk and 8-disk EBS RAID0 stripesets that we’ve been testing all week?

That is a topic for the next blog posting …

Here are the results of our tests against local storage on a cc1.4xlarge instance. As usual the raw data is available in our public spreadsheet.

We ran tests multiple times and averaged the results. All file systems were XFS.

Summarized/averaged values used to generate the graphs:

Here is the graph showing block and character based read & write tests. We did not capture character-based test data for the local root disk because it was so slow already.

And here is a graph of the bonnie++ tests that deal with Seeks and Sequential/Random file creation & deletion:

Screwing around with the boot volume is part of our regular “explore around the edges” work before we get serious with how we are going to configure and orchestrate the new systems. The boot volume in this scenario does not have PV driver support and thus will perform slower than the actual ephemeral storage. Our need was for the boot volume to be big enough to hammer with bonnie++ – this is not something we’d do in a production scenario.

Background

All of the cloud nerds at BioTeam are thrilled now that the Amazon Compute Cluster nodes have been publicly launched. If you missed the exciting news please visit the announcement post over at the AWS blog - http://aws.typepad.com/aws/2010/07/the-new-amazon-ec2-instance-type-the-cluster-compute-instance.html

We’ve been madly banging on the new instance types and trying to (initially) perform some basic low level benchmarks before we go on to the cooler benchmarks where we run actual life science & informatics pipelines against the hot new gear. Using our Chef Server it’s trivial for us to orchestrate these new systems into working HPC clusters in just a matter of minutes. We plan to start blogging and demoing live deployment of elastic genome assembly pipelines and NextGen DNA sequencing instrument pipelines (like the Illumina software) on AWS soon.

Like James Hamilton says, the real value in these new EC2 server types lies in the non-blocking 10Gigabit ethernet network backing them up. All of a sudden our “legacy” cluster and compute farm practices that involved network filesharing among nodes via NFS, GlusterFS, Lustre and GPFS seem actually feasible rather than a sick masochistic exercise in cloud futility.

We expect to see quite a bit of news in the near future about people using NFS, pNFS and other parallel/cluster filesystems for HPC data sharing on AWS – seems like a no-brainer now that we have full bisectional 10GbE bandwidth between our Compute Cluster cc1.4xlarge instance types.

However, despite the fact that the coolest stuff is going to involve what we can do now node-to-node over the fast new networking fabric there is still value in doing the low-level “what does this new environment look and feel like?” tests involving EBS disk volumes, S3 access and the like.

The Amazon AWS team did a great job preparing the way for people who want to quickly experiment with the new HPC instance types. The EC2 AMI images have to boot off of EBS and run under HVM virtualization rather than the standard paravirtualization used on the other instance types.

Recognizing that bootstrapping a HVM-aware EBS-booting EC2 server instance is a non-trivial exercise, AWS created a QuickStart public AMI with CentOS Linux that anyone can use right away:

The Problem – how to grow the default 20GB system disk in the QuickStart AMI?

As beautiful as the CentOS HVM AMI is, it only gives us 20GB of disk space in it’s native form. This is perfectly fine for most of our use cases but presented a problem when we decided we wanted to use bonnie++ to perform some disk IO benchmarks on the local boot volume to complement our tests against more traditional mounted EBS volumes and RAID0 stripe sets.

Bonnie++ really wants to work against a filesystem that is at least twice the size of available RAM so as to mitigate any memory-related caching issues when testing actual IO performance.

The cc1.4xlarge “Cluster Compute” instance comes with ~23GB of physical RAM. Thus our problem — we wanted to run bonnie++ against the local system disk but the disk is actually smaller than the amount of RAM available to the instance!

For this one particular IO test we really wanted a HVM-compatible AMI that had at least 50GB of storage on the boot volume.

The Solution

I was shocked and amazed to find that in about ~20 minutes of screwing around with EBS snapshot sizes, instance disk partitions and LVM settings I was able to achieve the goal of converting the Amazon Quickstart 20GB AMI into a custom version where the system disk was 80Gb in size.

The fact that this was possible and achievable out of the box without having to debug mysterious boot failures, kernel panics and all the other sorts of things I’m used to dealing with when messing with low level disk and partition issues is the ultimate testament to both Amazon’s engineering prowess (how cool is it that we can launch EBS snapshots of arbitrary size?) as well as the current excellent state of Linux, Grub and LVM2.

I took a bunch of rough notes so I’d remember how the heck I managed to do this. Then I decided to clean up the notes and really document the process in case it might help someone else.

The Process

I will try to walk through step-by-step the commands and methods used to increase the system boot disk from 20GB in size to 80GB in size.

It boils down to two main steps

• Launch the Amazon QuickStart AMI but override & increase the default 20GB boot disk size
• Get the CentOS Linux OS to recognize that it’s now running on a bigger disk

You can’t do the following step using the AWS Web Management Console as the webUI does not let you alter the parameters of the block device settings. You will need the command-line EC2 utilities installed and working in order to proceed.

On the command line we can easily tell Amazon that we want to start the QuickStart AMI but instead of launching it within a 20GB snapshot of the EBS boot volume we will launch it against a much larger snapshot.

If you look at the info for AMI ami-7ea24a17 within the web page you will see this under the details for the block devices that will be available to the system at boot:

Block Devices: /dev/sda1=snap-1099e578:20:true

That is basically saying that Linux Device “/dev/sda1″ will be built from EBS snapshot volume “snap-1099e578″. The next “:” delimited parameter sets the size to 20GB.

We are going to change that from 20GB to 80GB.

Here is the command to launch that AMI using a disk of 80GB in size instead of the default 20GB.

In the following screenshot, note how we are starting the Amazon AMI ami-7ea24a17 with a block device (the “-b” switch) that is bootstrapping itself from the snapshot “snap-1099e578″.

All we needed to do in order to make the EC2 server have a larger boot disk is pass in “80″ to override the default value of 20GB. Confused? Look at the “-b” block device argument below, the 80GB is set right after the snapshot name:

Your EBS volume might take a bit longer than normal to boot up but once it is online and available you can login normally.

Of course, the system will appear to have the default 20GB system disk:

Even the LVM2 physical disk reports show the ~20GB settings:

However, if we actually use the ‘fdisk’ command to examine the disk we see that the block device is, indeed, much larger than the 20GB the Operating System thinks it has to utilize:

Fdisk tells us that disk device /dev/hda has 85.8 GB of physical capacity.

Now we need to teach the OS to make use of that space!

There are only two partitions on this disk, /dev/hda1 is the 100MB /boot partition common to RedHat varients. Lets leave that alone.

The second partition, /dev/hda2 is already set up for logical volumes under LVM. We are going to be lazy. We are just going to use ‘fdisk’ to delete the /dev/hda2 partition so that we can immediately recreate it so that it spans the full remaining space on the physical drive.

After typing “fdisk /dev/hda” we type “d” and delete partition “2″. Then we type “n” for a new partition of type “p” for primary and “2″ to name it as the second partition. After that we just hit return to accept the default suggestions for the begin and end of the recreated second partion.

If it all worked, we can type the “p” command to print the new partition table out.

Note how /dev/hda2 now has many more blocks? Cool!

We are not done yet. None of our partition changes have actually been written to disk yet. We still need to type “w”  to write the new partition table down to disk and “q” to exit.

Obviously we can’t make live changes on a running boot disk. The new partition settings will come into effect after a system reboot.

Now we reboot the system and wait for it to come back up.

When it comes back up, don’t be alarmed that both ‘df’ and ‘pvscan’ still show the incorrect size:

We can fix that! Now we are in the realm of LVM so we need to use the “pvresize <device>” command to rescan the physical disk. Since our LVM2 partition is still /dev/hda2 that is the physical device path we give it:

Success! LVM recognizes that the drive is larger than 20GB.

With LVM aware that the disk is larger we are pretty much done. We can resize an existing logical volume or add a new one to the default Volume Group (“VolGroup00″).

Since I’m lazy AND I want to mount the extra space away from the root (“/”) volume I chose to create a new logical volume that shares the same /dev/hda2 physical volume (“PV”) and Volume Group (“VG”).

We are going to use the command “lvcreate VolGroup00 –size 60G /dev/hda2?” to make a new 60GB logical volume that is part of the existing Volume Group named “VolGroup00″:

Success. Note that our new logical volume got assigned a default name of “lvol0″ and it now exists in the LVM device path of “/dev/mapper/VolGroup00-lvol0″.

Now we need to place a Linux filesystem on our new 60GB of additional space and mount it up. Since I am a fan of XFS on EC2 I need to first install the “xfsprogs” RPM and then format the volume. A simple “yum -y install xfsprogs” does the trick and now I can make XFS filesystems on my server:

Success. We now have 60GB more space, visible to the OS and formatted with a filesystem. The final step is to mount it.

And we are done. We’ve successfully converted the 20GB Amazon QuickStart AMI into a version with a much larger boot volume.

Conclusion

None of this is rocket science. It’s actually just Linux Systems Administration 101.

The real magic here is how easily this is all accomplished on our virtual cloud system using nothing but a web browser and some command-line utilities.

What makes this process special for me is how quick and easy it was – anyone who has spent any significant amount of time managing many physical Linux server systems knows the pain and hours lost when trying to do this stuff in the real world on real (and flaky) hardware.

I can’t even count how many hours of my life I’ve lost trying to debug Grub bootloader failures, mysterious kernel panics and other hard-to-troubleshoot booting and disk resizing efforts on production and development server systems when I’ve altered settings that we’ve covered in this post. In cluster environments we often have to do this debugging via a 9600 baud serial console or via flaky IPMI consoles. It’s just nasty.

The fact that this method worked so quickly and so smoothly is probably only amazing to people who know the real pain of having done this in the field, crouched on the floor of a freezing cold datacenter and trying not to pull your hair out as text scrolls slowly by at 9600 baud.

Congrats to the Amazon AWS team. Fantastic work. It’s a real win when virtual infrastructure is this easy to manipulate.

• July 13th – Original post
• July 14th – More results from cc1.4xlarge single-disk & initial results from c1.xlarge instance type, uploaded new version of the raw data spreadsheet to Google Docs. Updated all graphs.
• July 19th – Lots more data (including ephemeral storage) added to the raw data spreadsheet on Google Docs
• July 20th – Added a new blog post specifically talking about local, ephemeral and EBS performance on cc1.4xlarge instances

Background

Now that Amazon Web Services has opened their new “Compute Cluster” cc1.4xlarge instance types to the public we’ve spent the day running bonnie++ disk performance benchmarks against single and RAID0 striped EBS volumes.

This is because we are life science types who do lots of high performance computing and cluster building. The single biggest performance bottleneck for people who want to do biology “in the cloud” is the generally poor performance of disk IO and storage in general. We tend to be more bottlenecked by the speed of disk than the speed of CPU in many common informatics and genomics applications.

Following in the footsteps of many others before us (example 1, example 2) we have learned that we can tease additional performance out of Amazon EBS disks by striping together multiple drives into a software RAID0 set.

There is a whole body of experimentation going on right now trying to find the “optimal” combination of:

• EC2 instance type
• EBS volume size
• # of EBS volumes
• Which filesystem to put on the software RAID set
• What software RAID settings to use when creating the RAID set
• What linux IO scheduler to use
• What volume mount options to use
• What other tweaks/parameters for increasing performance

Nobody has really discovered the “ultimate’” solution and things are further complicated by the fact that performance “on the cloud” can vary minute by minute, hour by hour and day by day. It’s extremely difficult to get any sort of reliably repeatable data from cloud systems.

I also hate benchmarking because

• Performance “in the cloud” is insanely variable for reasons that are invisible to mortals
• Nobody is ever satisfied with the results
• It’s a lot of work, and even harder to do it reasonably correctly
• Everybody has different needs, demands and requirements

Goals

At this point all we really want to see is what the effect of having the new non-blocking 10 Gigabit Ethernet network operating behind the new EC2 “Cluster Compute” instance types does for performance on EBS volumes.

Obviously there are a lot of possibilities for non-oversubscribed 10GbE networking for people used to clusters and compute farms. We are also going to test node-to-node file transfers and even vanilla NFS between systems to see if it is now sensible to actually orchestrate actual Platform LSF, PBS and Grid Engine managed clusters on AWS.

Methodology

We chose a 160GB disk as our target size and made the following EBS volumes:

• Single 160GB EBS volume
• Four 40GB EBS volumes (to be striped at RAID0 into 160GB disk)
• Eight 20GB EBS volumes (to be striped at RAID0 into 160GB disk)

Basically we wanted to run bonnie++ multiple times against 160GB single-disk, four-disk and eight-disk XFS volumes using different Linux IO schedulers to see what would happen.

Filesystem. We chose XFS as the Linux filesystem to use, based largely on the work of others in this area, the filesystem was created with the standard “mkfs.xfs <device>” command. Nothing special

Software RAID. Except for choosing a “–chunksize=256″ option, we did nothing special with the creation of the /dev/md0 RAID0 device. An example command for our 8-disk stripeset would look like “mdadm –create –verbose –level=0 –chunk=256 –raid-devices=8 /dev/md0 /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm“

Volume mounting: XFS volumes were mounted with the following options (noatime,nodiratime,logbufs=8?) example: “mount -t xfs -o noatime,nodiratime,logbufs=8 /dev/md0 /eightdisk?“

Linux blockdev ra attribute. Following in the footsteps of others who seemed to find that the value of the Linux “readahead” value was perhaps set too conservatively (especially on RedHat varients) we increased the readahead value for every disk via the command “blockdev –setra 65536? <device>“

Linux IO Scheduler. Various people have reported that the Linux IO scheduler matters. In our tests we wanted to see performance under different schedulers. We tested against the “noop”, “deadline” and “cfq” schedulers by altering the contents of “/sys/block/<device>/queue/scheduler“.

Bonnie++ Command: We ran the same bonnie++ command for each test, differing only in the name of the output log file. An example command: “bonnie++ -u nobody -n cc1-eightdisk-noop -s 50000 -x 2 -d /eightdisk/ -q -f 2>&1 | tee /opt/results/eightdisk-noop-results.log?“

Access Our Raw Data

Data collected so far has been posted to a Google Docs spreadsheet.

“Cluster compute” cc1.4large Results

We’ve created new blog posts to specifically talk about what we see just on the new EC2 instance types

• Performance of local storage only on cc1.4xlarge instance types
• Combined performance of local & EBS attached storage on cc1.4xlarge instance types

Misc. Results

Incomplete so far -

• Something seems “off” with our single-node 160GB EBS volume test. We might blow the volume and instance away and re-run just to see if we get any major shift in the numbers
• Testing the single-node 160GB EBS volume is so slow that we were only able to complete the single drive tests with the noop IO scheduler. For the 4-drive and 8-drive stripe sets we were able to test with noop, cfq and deadline
• We also have no data yet from non compute-cluster node instance types. We plan on collecting that data over the coming days so we can compare it.

Along with the raw data, here are some graphs. We averaged the values of the repeated tests.

Interpretation and more results will be forthcoming, we’ll update this blog post as we learn and do more.

Sequential Output

Sequential Input

Seek

Sequential Create & Delete

Random Create & Delete

Amazon made a very important announcement today, releasing new EC2 server types and network configurations that significantly enhance the Amazon AWS environment for people who are interested in cluster computing, compute farming and high performance computing (HPC) on the cloud.

The announcement is here for those who are interested:

http://aws.typepad.com/aws/2010/07/the-new-amazon-ec2-instance-type-the-cluster-compute-instance.html?

I’m thrilled that this news is now public, the service is up for use and I can finally start testing, blogging and benchmarking in the “real” production environment.

In the next few days I’ll be blogging over on http://blog.bioteam.net, concentrating initially on seeing how storage and storage IO speeds differ on the new instance types. For life science types like myself, one of the biggest hassles in the cloud is due to the fact that we tend to be more performance bound by the speed of storage and file IO than anything else. The 10GbE networking changes and non-oversubscription of the network links along with the ability to group nodes together may mean very very interesting things are now much more feasible on the AWS platform.

Because I’m going to first concentrate on storage and IO stuff on the new offering I wanted to quickly show Grid Engine running on the new server types.

Even a single node SGE cluster can do reasonable work now as the cc1 instance type includes a pair of quad-core Nehalem CPUs along with ~23GB memory and a 10GbE ethernet backend.

We will be blogging and talking much more about how to use Chef Server to orchestrate self-assembling Grid Engine clusters and compute farms on this new service but since that may not happen until later — I just wanted to throw up a teaser post showing SGE 6.2u5 running in single-node mode on the new HPC offerings from Amazon.

qstat output showing 16 CPUs (click for full-size):

qhost output showing system resources (click for full-size):

Link:

http://blog.bioteam.net/wp-content/uploads/2010/07/2010-ISMB-Cloud-Workshop_v1.pdf

We deployed the first incarnation of this system in 2004. At that time, it was a homogenous system built from single CPU G4 Xserve machines from Apple. I believe that it ran OS X 10.2 Server, installed via NetBoot from a portal that was configured by copying a bootable OS image from a USB drive.

In the ensuing six years, it has been in near constant use as a BLAST farm for the department of environmental engineering at MIT. We’ve upgraded it with each incarnation of the XServe as they came out – and when we ran out of physical space in the co-location facility (hard limit of three racks), we started rolling in new machines by ousting the oldest ones. These old servers moved to a corner of a wet lab to serve as a development cluster.

The system is – to some extent – a crazy quilt. It’s cobbled together, running three different major versions of OS X. We’ve been wanting to upgrade the ethernet backplane for about four years now – but somehow it’s never been important enough to actually do. There are separate NFS servers for BLAST databases and home directories. A small pile of scripts integrate with Sun Grid Engine to ensure that data-staging and software updates do not collide with running jobs. On the other hand, this system has cranked out a ridiculous amount of scientific analysis.

I had a blast. At the end of two days of work, I re-enabled the queues and had the satisfaction of watching all the little blue lights spin up almost immediately, indicating that user jobs were flowing out onto the hardware.

It reminds me, to some extent, of Admiral Hyman G. Rickover’s famous 1953 quote, to a congressional hearing, about the difference between “paper” and “real” nuclear reactors:

An academic reactor or reactor plant almost always has the following basic characteristics: (1) It is simple. (2) It is small. (3) It is cheap. (4) It is light. (5) It can be built very quickly. (6) It is very flexible in purpose. (7) Very little development will be required. It will use off-the-shelf components. (8) The reactor is in the study phase. It is not being built now.

On the other hand a practical reactor can be distinguished by the following characteristics: (1) It is being built now. (2) It is behind schedule. (3) It requires an immense amount of development on apparently trivial items. (4) It is very expensive. (5) It takes a long time to build because of its engineering development problems. (6) It is large. (7) It is heavy. (8) It is complicated.

The systems engineering required to keep pace with biology these days still falls in the “heavy and complicated” category. Under the hood of the virtual, there must always be the real. It’s therefore important that someone on the team break out the athletic tape and the power driver from time to time. It keeps us honest.

I had the unenviable task of giving the first talk after James Hamilton presented the opening keynote at the Amazon Web Services Genomics meeting

James is one of the very very few people writing and talking in public about the sorts of things the mega internet scale companies are doing to drive costs down and recapture efficiencies. There are amazing lessons to be learned from what Microsoft, Amazon and Google (etc.) are doing deep within their datacenters but that info is very hard to find because it represents a competitive advantage. This is why whenever you see somethign cool coming out of a Google datacenter you can be sure that the info is at least a few years old …

This is the first time I’ve gotten a chance to see James present in person and it was fantastic. Hopefully AWS will release the video and slides soon.

Many of my “cloud talks” are focused on how to map HPC workflows onto cloud platforms and the various technical challenges & best practices involved. Today’s talk was far lighter and more “big picture” since I knew that the speakers coming after me were presenting deeply specific and technical works.

Talk slides in PDF form can be downloaded by clicking on the image below. Feedback welcome.

Video from the event is likely to be posted soon by Amazon.

UPDATE:  Crap. I misspoke during my talk when I was talking about how surprising it was to hear about Chef Usage at the 2010 BioITWorld Cloud Computing Workshop. I spaced out and said that “we” organized the cloud meeting. Totally wrong – the meeting was organized and run by Jason and the gang over at Cycle Computing. Really sorry about that – not intentional!