We here at BioTeam have been kicking tires and generally exploring around the edges of the new Amazon cc1.4xlarge “compute cluster” EC2 instance types. Much of our experimentation has been centered around simplistic benchmarking techniques as a way of slowly zeroing in on the methods, techniques and orchestration approaches most likely to have a significant usability, performance or wallclock-time-to-scientific-results outcome for the work we do professionally for ourselves and our clients.

We are asking very broad questions and testing assumptions along the lines of:

• Does the hot new 10 Gigabit non-blocking networking fabric backing up the new instance types really mean that “legacy” compute farm and HPC cluster architectures which make heavy use of network filesharing possible?
• How does filesharing between nodes look and feel on the new network and instance types?
• Are the speedy ephemeral disks on the new instance types suitable for bundling into NFS shares or aggregating into parallel or clustered distribtued filesystems?
• Can we use the replication features in GlusterFS to mitigate some of the risks of using ephemeral disk for storage? 
• Should the shared storage built from ephermeral disk be assigned to “/scratch” or other non-critical duties due to the risks involved? What can we do to mitigate the risks?
• At what scale is NFS the easiest and most suitable sharing option? What are the best NFS server and client tuning parameters to use? 
• When using parallel or cluster filesystems like GlusterFS, what rough metrics can we use to figure out how many data servers to dedicate to a particular cluster size or workflow profile?

GlusterFS & NFS Initial Testing

Over the past week we have been running tests on two types of network filesharing. We’ve only tested against a single client so obviously these results say nothing about at-scale performance or operation.

Types of tests:

1. Take the pair of ~900GB ephemeral disks on the instance type, stripe them together as a RAID0 set, slap an XFS filesystem on top and export the entire volume out via NFS
2. Take the pair of ~900GB ephemeral disks on the instance type, slap a single large partition on each drive, format each drive with an EXT3 filesystem and then use GlusterFS to create, mount and export the volume via the GlusterFS protocol

For each of the above two test types we repeatedly ran (at least 4x times) our standard bonnie++ benchmark tests (methodology described in the earlier blog posts). The tests were run on a single remote client that was either NFS mounting or GlusterFS mounting the file share.

GlusterFS parameters

• None really. We used the standard volume creation command and mounted the file share via the GlusterFS protocol over TCP. Eventually we want to ask some of our GlusterFS expert friends for additional tuning guidance

NFS parameters:

• Server export file:  ”/nfs    <host>(rw,async)?”
• NFS Server config: boosted the number of nfsd daemons to 16 via edits to /etc/sysconfig/nfs file
• Client mount options:  ”mount -t nfs -o rw,async,hard,intr,retrans=2,rsize=32768,wsize=32768,nfsvers=3,tcp <host>:/nfs /nfs-scratch?”

Lessons Learned So Far  - NFS vs GlusterFS

• GlusterFS was incredibly easy to install and creating and exporting parallel filesystem shares was straightforward. The methods involved are easily scripted/automated or built into a server orchestration strategy. The process was so simple that initially we were thinking that GlusterFS would be our default sharing option for all our work on the new compute cluster instances
• GlusterFS has ONE HUGE DOWNSIDE. It turns out that GlusterFS recommends that the participating disk volumes be formatted with an ext3 filesystem for best results. This is … problematic … with the 900GB ephemeral disks because formatting a 900 gb disk with ext3 takes damn near forever. We estimate about 15-20 minutes of wallclock time wasted while waiting for the “mkfs.ext3″ command to complete.
• The wallclock time lost to formatting ext3 volumes for GlusterFS usage is significant enough to affect how we may or may not use GlusterFS in the future. Maybe there is a different filesystem we can use with a faster formatting profile. Using XFS and software RAID we can normally stand up and export filesystems in a matter of a few seconds or a minute or two. Sadly, XFS is not recommended at all with current versions of GlusterFS.
• Using GlusterFS with the recommended ext3 configuration seems to mean that we have to accept a minimum delay of 15 minutes or even more when standing up and exporting new storage. This is unacceptable for small deployments or workflows where you might only be running the EC2 instances for a short time.
• The possibility of using the GlusterFS replication features to mitigate against the risks of using ephermeral storage might be significant. We need to do more testing in this configuration. 
• Given the extensive wallclock time delays inherent in waiting for ext3 filesystem formatting to complete in a GlusterFS scenario it seems likely that we might default to using a tuned NFS server setup for (a) small clusters & compute farms or (b) systems that we plan to stand up only for a few hours.
• The overhead of provisioning GlusterFS becomes less significant when we have very large clusters that can benefit from the inherent scaling ability of GlusterFS or when we plan to stand up the clusters for longer periods of time

Benchmark Results

In all the results shown below I’ve included data from a 2-disk RAID0 ephemeral storage  setup. This is so that the network filesharing data can be contrasted against the results seen from running bonnie++ locally.

Click on the images for a larger version.

For background and summary writeups of all the various blog posts we have dealing with the new Amazon EC2 “compute cluster” cc1.4xlarge instance types please refer to this summary page: http://blog.bioteam.net/2010/07/19/exploring-the-new-aws-compute-cluster-ec2-instances/?

Related post

We talked about the performance of the boot and ephemeral storage in this post: http://blog.bioteam.net/2010/07/19/local-storage-performance-of-aws-cluster-compute-instances/?

This post

In this post I’ve finally collected enough data to cover repeated bonnie++ benchmark tests against all the main types of block storage available to the new Amazon cc1.4xlarge “compute cluster” instance types:

• Local boot disk performance
• Performance of a single ephemeral disk
• Performance of the two available ephemeral? disks when striped together with software RAID0
• Performance of a single EBS volumes attached to the instance
• Performance of 4 EBS volumes striped with RAID0 and attached to the instance
• Performance of 8 EBS volumes striped with RAID0 and attached to the instance?

Lessons Learned

1. Don’t use the boot disk for anything other than booting the operating system. As the results show, the performance of the (no PV driver support) cc1.4xlarge boot disk is the slowest of all possible block storage options available. Really slow. Not worth using for anything other than OS stuff. This also includes not bothering to mess with the size of the available volume.

2. The instance ephemeral storage volumes are fast and should not be ignored. Every cc1.4xlarge EC2 instance comes with a pair of ~840GB ephemeral disk volumes. In living by our own “don’t trust anything to non-persistant storage!” best practice we are guilty of ignoring these drives in situations where they could have been of significant benefit. This will change. The performance of a single ephemeral volume beats the performance we see out of a single EBS volume. A striped pair of ephemeral volumes performs even better and stacks up well even to multiple EBS volumes striped together. The RAID0 pairing of the two ephemeral drives seems to consistently outperform even 8-drive EBS RAID0 volumes when you look at the bonniee++ results for random and sequential file creation and deletion tests. This has major implications for HPC and scientific pipeline processing on the cloud. In particular I can easily envision using the ephemeral drives to build a shared parallel scratch filesystem (think PVFS or GlusterFS) in cluster configurations. This would give you a nice shared scratch storage pool. Even in simpler cluster setups it looks like it would be a win to stage data into the ephemeral storage so it can be used as the target drive for  scientific processing (where the input data is not unique and has backup copies elsewhere). We can run the IO heavy analysis against the fast ephemeral storage and send our result data into S3 buckets or a proper EBS volume for downstream handling.

3. Striping EBS volumes into software RAID0 sets is a valid practice. We clearly see performance gains when using more than one EBS volume, the performance gain is significant enough to justify the hassles involved in backing up and protecting EBS-resident sofware RAID sets. We need to do more work (and really need to test 2-volume EBS stripes) but it is clear that there is a measurable performance gain to be had. Not sure if we’d use 8-disk RAID0 sets for production work but looking at 2-disk and 4-disk methods is something that we will be looking seriously at.

Obviously there is much more to be drawn from the data but benchmarking is hard (and controversial) in regular settings let alone trying to get repeatable and consistent results out of a virtualized multi-tennant cloud framework. For now I’d prefer to stick to broad general conclusions and “lessons learned” rather than trying to divine highly specific things out of the raw data.

Test Results

As usual, you can find all of the raw data in this google spreadsheet. We did not finesse the data at all, the only data munging we did was to run tests repeatedly and then average out the results in order to arrive at the numbers used in the graphs.

Here are the numerical numbers behind the graphs, click on the image for the full-size version:

And here are the graphs. We’ve broken out the graphs to represent the results measured in “K/sec” versus just ” /sec”.

Read, Write & Rewrite results for all cc1.x4large storage types (click on image for full size):

Sequential create, delete & seek results for all cc1.x4large storage types (click on image for full size):

In our work with Amazon Web Services we try to spend as much time as we can “kicking the tires” so we become better at building stuff that we and our clients can actually use. We also try to share our information as much as possible in the spirit of scientific collaboration & honest exchange.

This is a ’roundup’ or summary blog post where we’ll list out blog post or articles that discuss the new Amazon cc1.4xlarge Compute Cluster instances.

• Grid Engine on the new Amazon Compute Cluster Instances
• How to resize an EC2 AMI when the boot disk is on EBS
• Preliminary EBS performance on Amazon Compute Cluster cc1.4xlarge instance types
• Local storage performance of AWS cluster compute instances
• Combined summary of local, ephemeral & EBS storage on cc1.4xlarge instance types
• NFS & GlusterFS network filesharing on the cc1.4xlarge instance types

Content-wise it’s very similar to my NHGRI meeting slides except I added more introductory material about cloud segments and definitions.

Hadoop for Bioinfomatics – Deepak Singh from Cloudera on Vimeo.

2009-NYC-AWS-BioTeam

A few pictures from the event are posted below. It was a great event, standing room only for all of the presentations.

The tour details and dates are here:

http://aws.amazon.com/startupproject/

The agenda for the May 28th event is:

1:00-2:00 Doors Open
2:00-2:10 Opening Statements
2:10-2:50 AWS Overview – Adam Selipsky, Vice President, Amazon Web Services
2:50-3:10 Break
3:10-3:50 Customer Presentations:
Sam Lessin, CEO, drop.io
Dan Gill, VP Business Development, Gotuit
Chris Dagdigian, Founding Partner, BioTeam
Brian Adams, Co-Founder and CTO, Admeld
3:50-4:10 Customer Q&A
4:10-4:50 Architecting Cloud Apps – Matt Tavis, Solutions Architect, Amazon Web Services
4:50-5:00 Closing Statements
5:00-7:00 Networking Reception + Solutions Corner Expo

FreedomOSS, RightScale, SOASTA, Pentaho and Kaavo will be the vendors in the Solutions corner.

Summary:

Buying CPUs by the hour is back. Marketing apparently decided that ‘On Demand’ sounds pushy and ‘Utility’ and ‘Grid’ are too rigid. This time it will be called a ‘Cloud’, and it will be running inside Amazon.com.

Article link:

http://www.bio-itworld.com/issues/2007/nov/inside-the-box-buying-cpus/