Amazon made a very important announcement today, releasing new EC2 server types and network configurations that significantly enhance the Amazon AWS environment for people who are interested in cluster computing, compute farming and high performance computing (HPC) on the cloud.

The announcement is here for those who are interested:

http://aws.typepad.com/aws/2010/07/the-new-amazon-ec2-instance-type-the-cluster-compute-instance.html?

I’m thrilled that this news is now public, the service is up for use and I can finally start testing, blogging and benchmarking in the “real” production environment.

In the next few days I’ll be blogging over on http://blog.bioteam.net, concentrating initially on seeing how storage and storage IO speeds differ on the new instance types. For life science types like myself, one of the biggest hassles in the cloud is due to the fact that we tend to be more performance bound by the speed of storage and file IO than anything else. The 10GbE networking changes and non-oversubscription of the network links along with the ability to group nodes together may mean very very interesting things are now much more feasible on the AWS platform.

Because I’m going to first concentrate on storage and IO stuff on the new offering I wanted to quickly show Grid Engine running on the new server types.

Even a single node SGE cluster can do reasonable work now as the cc1 instance type includes a pair of quad-core Nehalem CPUs along with ~23GB memory and a 10GbE ethernet backend.

We will be blogging and talking much more about how to use Chef Server to orchestrate self-assembling Grid Engine clusters and compute farms on this new service but since that may not happen until later — I just wanted to throw up a teaser post showing SGE 6.2u5 running in single-node mode on the new HPC offerings from Amazon.

qstat output showing 16 CPUs (click for full-size):

qhost output showing system resources (click for full-size):

Excerpt:

“IT MAY NO LONGER BE FAIR to characterize large pharmaceutical firms as late adopters of information technology (IT).

Having spent the past five years catching up to other industries in the deployment of enterprise software systems that link researchers and laboratories companywide, big drug firms are now starting to push data storage and processing onto the Internet to be managed for them by companies such as Amazon, Google, and Microsoft on computers in undisclosed locations…“

Article link: http://pubs.acs.org/cen/email/html/cen_coverstory_87_8721cover.html

Simply put, Amazon is now willing to accept delivery of eSATA and USB disks for large-scale data ingestion into the S3 storage cloud service. This is far faster than internet based methods, particularly if you are dealing with daily terabyte-scale data that you’d like to park in some sort of external utility service.

I think this may be a big deal for life science and can forsee quite a bit of scientific data making “1-way” trips into the cloud for long term storage and even secondary processing via cloud server instances. We’ll see.

These links cover the just-launched service in detail:

• http://aws.typepad.com/aws/2009/05/send-us-that-data.html
• http://aws.amazon.com/importexport/

Chris Dagdigian recently test-drove Univa UD’s UniCluster Express, integrated it with the Amazon Elastic Compute Cloud (EC2).

Links to the Howto Whitepaper and various screencast video recordings are available via the gridengine.info post:

http://gridengine.info/articles/2008/07/14/univa-ud-unicluster-express-and-amazon-ec2

Congratulations are in order for Mike who was announced as the winner of the “Win your Genome” contest being run by 23andMe. The announcement can be read online here:

http://spittoon.23andme.com/2008/05/14/and-the-winner-is/

Click on the image to download the full article in PDF form.

“… The wikiLIMS system is
“something of a shared brain”
for a lab, “where we can all
agree on a common dialog, a
common vocabulary, common
semantics, and we have the
infinite memory of a hard
drive.”

The full article in PDF form is available here: