Employee Posts
Playing with NFS & GlusterFS on Amazon cc1.4xlarge EC2 instance types
Early single-client tests of shared ephemeral storage via NFS and parallel GlusterFS We here at BioTeam have been kicking tires and generally exploring around the edges of the new Amazon cc1.4xlarge “compute cluster” EC2 instance types. Much of our experimentation has been centered around simplistic benchmarking techniques as a way of slowly zeroing in on [...]
Boot, ephemeral & EBS storage performance on amazon cc1.4xlarge instance types
Backstory For background and summary writeups of all the various blog posts we have dealing with the new Amazon EC2 “compute cluster” cc1.4xlarge instance types please refer to this summary page: http://blog.bioteam.net/2010/07/19/exploring-the-new-aws-compute-cluster-ec2-instances/? Related post We talked about the performance of the boot and ephemeral storage in this post: http://blog.bioteam.net/2010/07/19/local-storage-performance-of-aws-cluster-compute-instances/? This post In this post I’ve finally collected [...]
Local storage performance of AWS cluster compute instances
Lots more data collected over the weekend as we were finally able to run bonnie++ against the local boot disk as well as single and striped versions of the ephermeral storage volumes that come along with every cc1.4xlarge instance type. Key Results: Performance of the root/boot disk is way slower than any other type of [...]
How to resize an Amazon EC2 AMI when boot disk is on EBS
Note Screwing around with the boot volume is part of our regular “explore around the edges” work before we get serious with how we are going to configure and orchestrate the new systems. The boot volume in this scenario does not have PV driver support and thus will perform slower than the actual ephemeral storage. [...]
Preliminary EBS performance on Amazon Compute Cluster cc1.4xlarge instance types
Post Update History: July 13th – Original post July 14th – More results from cc1.4xlarge single-disk & initial results from c1.xlarge instance type, uploaded new version of the raw data spreadsheet to Google Docs. Updated all graphs. July 19th – Lots more data (including ephemeral storage) added to the raw data spreadsheet on Google Docs [...]
Grid Engine on the new Amazon Compute Cluster Instances
{ crossposted to blog.bioteam.net and gridengine.info } Amazon made a very important announcement today, releasing new EC2 server types and network configurations that significantly enhance the Amazon AWS environment for people who are interested in cluster computing, compute farming and high performance computing (HPC) on the cloud. The announcement is here for those who are [...]
2010 ISMB Amazon Cloud Workshop Slides
Later than I’d like but as promised to attendees of the ISBM 2010 meeting in Boston, here are my presentation slides from the Amazon Cloud Computing Workshop. Link: http://blog.bioteam.net/wp-content/uploads/2010/07/2010-ISMB-Cloud-Workshop_v1.pdf
Slides from the Amazon Web Services Genomics meeting
I had the unenviable task of giving the first talk after James Hamilton presented the opening keynote at the Amazon Web Services Genomics meeting James is one of the very very few people writing and talking in public about the sorts of things the mega internet scale companies are doing to drive costs down and [...]
BioITWorld – Slides from HPC Trends Talk
I was originally supposed to moderate the morning session of the IT/Infrastructure conference track but was asked at the list minute to fill in for Phil Butcher who could not make it to Boston because of the volcano-ash disruption of EU-US air travel. The slides from this talk are not really new as they are [...]
BioITWorld Cloud Workshop – Talk Slides
Click on the graphic down below to download the PDF talk slides. This was a short 30 minute talk starting off the 2010 BioITWorld Cloud Computing Workshop. Content-wise it’s very similar to my NHGRI meeting slides except I added more introductory material about cloud segments and definitions.
Instead of spending 20 minutes to format an ext3 volume every time you need one, you could just snapshot a ...
Let me add a few remarks about Ext3 as well.
The reason we usually recommend Ext3 instead of XFS is because ...
I'm using these slides to tune SGE 6.2 @ European Space Agency, thanks a lot! 
Great post.
Can you also try our native scale out NFS protocol? Next 3.1 release of Gluster will include it ...
So far my mental architecture is shaping up to be something like this:
Use the ephemeral disks for scratch space and ...
This shows that on-demand HPC still has a ways to go, it is very unfortunate that only ext3 is supported ...
We have had sge62u3 working fine in a 10.5 cluster using cron to startup the sge_execd process on clients. Trying ...
Instead of increasing your disk image size, you could have decreased the memory by writing a very short C program ...