In our work with Amazon Web Services we try to spend as much time as we can “kicking the tires” so we become better at building stuff that we and our clients can actually use. We also try to share our information as much as possible in the spirit of scientific collaboration & honest exchange.

This is a ’roundup’ or summary blog post where we’ll list out blog post or articles that discuss the new Amazon cc1.4xlarge Compute Cluster instances.

• Grid Engine on the new Amazon Compute Cluster Instances
• How to resize an EC2 AMI when the boot disk is on EBS
• Preliminary EBS performance on Amazon Compute Cluster cc1.4xlarge instance types
• Local storage performance of AWS cluster compute instances
• Combined summary of local, ephemeral & EBS storage on cc1.4xlarge instance types
• NFS & GlusterFS network filesharing on the cc1.4xlarge instance types

• July 13th – Original post
• July 14th – More results from cc1.4xlarge single-disk & initial results from c1.xlarge instance type, uploaded new version of the raw data spreadsheet to Google Docs. Updated all graphs.
• July 19th – Lots more data (including ephemeral storage) added to the raw data spreadsheet on Google Docs
• July 20th – Added a new blog post specifically talking about local, ephemeral and EBS performance on cc1.4xlarge instances

Background

Now that Amazon Web Services has opened their new “Compute Cluster” cc1.4xlarge instance types to the public we’ve spent the day running bonnie++ disk performance benchmarks against single and RAID0 striped EBS volumes.

This is because we are life science types who do lots of high performance computing and cluster building. The single biggest performance bottleneck for people who want to do biology “in the cloud” is the generally poor performance of disk IO and storage in general. We tend to be more bottlenecked by the speed of disk than the speed of CPU in many common informatics and genomics applications.

Following in the footsteps of many others before us (example 1, example 2) we have learned that we can tease additional performance out of Amazon EBS disks by striping together multiple drives into a software RAID0 set.

There is a whole body of experimentation going on right now trying to find the “optimal” combination of:

• EC2 instance type
• EBS volume size
• # of EBS volumes
• Which filesystem to put on the software RAID set
• What software RAID settings to use when creating the RAID set
• What linux IO scheduler to use
• What volume mount options to use
• What other tweaks/parameters for increasing performance

Nobody has really discovered the “ultimate’” solution and things are further complicated by the fact that performance “on the cloud” can vary minute by minute, hour by hour and day by day. It’s extremely difficult to get any sort of reliably repeatable data from cloud systems.

I also hate benchmarking because

• Performance “in the cloud” is insanely variable for reasons that are invisible to mortals
• Nobody is ever satisfied with the results
• It’s a lot of work, and even harder to do it reasonably correctly
• Everybody has different needs, demands and requirements

Goals

At this point all we really want to see is what the effect of having the new non-blocking 10 Gigabit Ethernet network operating behind the new EC2 “Cluster Compute” instance types does for performance on EBS volumes.

Obviously there are a lot of possibilities for non-oversubscribed 10GbE networking for people used to clusters and compute farms. We are also going to test node-to-node file transfers and even vanilla NFS between systems to see if it is now sensible to actually orchestrate actual Platform LSF, PBS and Grid Engine managed clusters on AWS.

Methodology

We chose a 160GB disk as our target size and made the following EBS volumes:

• Single 160GB EBS volume
• Four 40GB EBS volumes (to be striped at RAID0 into 160GB disk)
• Eight 20GB EBS volumes (to be striped at RAID0 into 160GB disk)

Basically we wanted to run bonnie++ multiple times against 160GB single-disk, four-disk and eight-disk XFS volumes using different Linux IO schedulers to see what would happen.

Filesystem. We chose XFS as the Linux filesystem to use, based largely on the work of others in this area, the filesystem was created with the standard “mkfs.xfs <device>” command. Nothing special

Software RAID. Except for choosing a “–chunksize=256″ option, we did nothing special with the creation of the /dev/md0 RAID0 device. An example command for our 8-disk stripeset would look like “mdadm –create –verbose –level=0 –chunk=256 –raid-devices=8 /dev/md0 /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm“

Volume mounting: XFS volumes were mounted with the following options (noatime,nodiratime,logbufs=8?) example: “mount -t xfs -o noatime,nodiratime,logbufs=8 /dev/md0 /eightdisk?“

Linux blockdev ra attribute. Following in the footsteps of others who seemed to find that the value of the Linux “readahead” value was perhaps set too conservatively (especially on RedHat varients) we increased the readahead value for every disk via the command “blockdev –setra 65536? <device>“

Linux IO Scheduler. Various people have reported that the Linux IO scheduler matters. In our tests we wanted to see performance under different schedulers. We tested against the “noop”, “deadline” and “cfq” schedulers by altering the contents of “/sys/block/<device>/queue/scheduler“.

Bonnie++ Command: We ran the same bonnie++ command for each test, differing only in the name of the output log file. An example command: “bonnie++ -u nobody -n cc1-eightdisk-noop -s 50000 -x 2 -d /eightdisk/ -q -f 2>&1 | tee /opt/results/eightdisk-noop-results.log?“

Access Our Raw Data

Data collected so far has been posted to a Google Docs spreadsheet.

“Cluster compute” cc1.4large Results

We’ve created new blog posts to specifically talk about what we see just on the new EC2 instance types

• Performance of local storage only on cc1.4xlarge instance types
• Combined performance of local & EBS attached storage on cc1.4xlarge instance types

Misc. Results

Incomplete so far -

• Something seems “off” with our single-node 160GB EBS volume test. We might blow the volume and instance away and re-run just to see if we get any major shift in the numbers
• Testing the single-node 160GB EBS volume is so slow that we were only able to complete the single drive tests with the noop IO scheduler. For the 4-drive and 8-drive stripe sets we were able to test with noop, cfq and deadline
• We also have no data yet from non compute-cluster node instance types. We plan on collecting that data over the coming days so we can compare it.

Along with the raw data, here are some graphs. We averaged the values of the repeated tests.

Interpretation and more results will be forthcoming, we’ll update this blog post as we learn and do more.

Sequential Output

Sequential Input

Seek

Sequential Create & Delete

Random Create & Delete

• Link: “Maximizing Utility on the Cloud” presentation

Slide download link: 2009 Grid Engine & Amazon EC2

2009-NYC-AWS-BioTeam

A few pictures from the event are posted below. It was a great event, standing room only for all of the presentations.

The tour details and dates are here:

http://aws.amazon.com/startupproject/

The agenda for the May 28th event is:

1:00-2:00 Doors Open
2:00-2:10 Opening Statements
2:10-2:50 AWS Overview – Adam Selipsky, Vice President, Amazon Web Services
2:50-3:10 Break
3:10-3:50 Customer Presentations:
Sam Lessin, CEO, drop.io
Dan Gill, VP Business Development, Gotuit
Chris Dagdigian, Founding Partner, BioTeam
Brian Adams, Co-Founder and CTO, Admeld
3:50-4:10 Customer Q&A
4:10-4:50 Architecting Cloud Apps – Matt Tavis, Solutions Architect, Amazon Web Services
4:50-5:00 Closing Statements
5:00-7:00 Networking Reception + Solutions Corner Expo

FreedomOSS, RightScale, SOASTA, Pentaho and Kaavo will be the vendors in the Solutions corner.

I’ve agreed to participate in a webinar and answer questions about the project, click on through for the full event details.

Date: Wednesday, September 17th

Time: 2:00 pm – 3:00 pm Central Standard Time

Register at: http://www.univaud.com/hpc/webinar20080917.php

Presented by leaders in HPC and Web Services, this webinar will provide IT decision-makers and System Administrators a unique opportunity to access real world information and examples of deploying clusters in a cloud environment.

Attendees will:

-Learn about Amazon’s EC2 web services, the UniCluster open sourcesoftware stack, and deployment of this software within EC2

-Gain insight into key data points from the people out in the field making it happen

-Be able to engage in a technical Q and A with the author of the technical white paper: “Deploying UniCluster in Amazon EC2″

Learn more about the webinar and register to attend at:

http://www.univaud.com/hpc/webinar20080917.php

Chris Dagdigian recently test-drove Univa UD’s UniCluster Express, integrated it with the Amazon Elastic Compute Cloud (EC2).

Links to the Howto Whitepaper and various screencast video recordings are available via the gridengine.info post:

http://gridengine.info/articles/2008/07/14/univa-ud-unicluster-express-and-amazon-ec2