This post is a follow-up to the article discussing Bill Van Etten’s OS X Launchd scripts for Grid Engine.

We now have a perl script that will automatically create site-appropriate launchd plist files for Grid Engine - the files are constructed by querying the SGE environment variables and running one of the SGE utility scripts. This may be a more convenient method than downloading our plist files and hand-editing them via the Apple plist editor program.

The new-in-Leopard “launchd” framewortk is Apple’s preferred system for handling system services and daemons. For more background on why launchd may be critical to Grid Engine on 10.5 systems (especially OS X Server 10.5.4 and later …) please visit the Grid Engine wiki page covering Apple Launchd:

http://wiki.gridengine.info/wiki/index.php/GridEngine_launchd

Download the script here:

sge-launchd-scriptmaker

Chris Dagdigian recently test-drove Univa UD’s UniCluster Express, integrated it with the Amazon Elastic Compute Cloud (EC2).

Links to the Howto Whitepaper and various screencast video recordings are available via the gridengine.info post:

http://gridengine.info/articles/2008/07/14/univa-ud-unicluster-express-and-amazon-ec2

Excerpt:

“…One terabyte of data is not large by today’s standards. An external terabyte disk can be purchased for less than a few hundred dollars at any office supply store. Accumulating one terabyte per day, maintaining it for rapid online access, and archiving it for permanent storage is a familiar problem to the sequencing center and maybe even the core facility, but this is a novel problem for the small research group and individual principal investigator.“

Link to the full article:

GEN 2008, “Managing Data from Next-Gen Sequencing” (p.42,44)

Link to a PDF reprint:

“…We’re now capturing data electronically with no human intervention required. A single device doing this isn’t particularly compelling, but two or more can be. The wiki becomes an electronic lab notebook written entirely by your machines. Because it can be viewed and edited with a Web browser, scientists can begin by using it as a web portal to recently completed analyses. Each discovery can be recorded into the wiki, one click away from the supporting raw data. Simple programs can build reports from or read and write into the wiki. An automatic full history of every page provides a safety net, and a way to view how our understanding of the data has evolved. This enables cycles in which humans and software each complement the other’s strengths.”

Read the full article here:

http://www.bio-itworld.com/issues/2008/april/inside-box-wikilims.html

Download a PDF reprint here:

William Van Etten has produced new scripts for the Apple OS X “Launchd” framework used by Mac OS X 10.5 “Leopard” systems. Moving forward it seems that Apple eventually intends to phase out the SystemStarter methods for starting/stopping daemons and services in favor of the newer launchd mechanisms.

A Wiki entry over at http://gridengine.info explains some of the problems with Launchd and shows an (older) potential workaround.

The new scripts referenced in this blog post take advantage of a newly discovered (by us) environment variable called “SGE_ND” — if this variable is set, the SGE daemons will not fork off and daemonize (something that Apple Launchd does not allow). By including the “SGE_ND=1” environment variable into the Grid Engine startup environment we are able to produce launchd-compatible scripts that function well at automatically enabling Grid Engine startup and shutdown (and theoretically automatic restart of crashed daemons).

Bill has created three launchd.plist(s)…

1) sgeqmaster.plist
2) sgeschedd.plist
3) sgeexecd.plist

Using these plists, one can load with …

launchctl load /Library/LaunchDaemons/net.sunsource.gridengine.sgeqmaster.plist
launchctl load /Library/LaunchDaemons/net.sunsource.gridengine.sgeschedd.plist
launchctl load /Library/LaunchDaemons/net.sunsource.gridengine.sgeexecd.plist

and start/stop with…

launchctl start net.sunsource.gridengine.sgeqmaster
launchctl start net.sunsource.gridengine.sgeschedd
launchctl start net.sunsource.gridengine.sgeexecd

launchctl stop net.sunsource.gridengine.sgeqmaster
launchctl stop net.sunsource.gridengine.sgeschedd
launchctl stop net.sunsource.gridengine.sgeexecd

If these scripts are installed in “/Library/LaunchDaemons/” they will also activate automatically on system boot or restarts.

Download link & Installation Instructions

The .plist scripts are available here: sge_launchd.tgz

To install them:

1. Download the gzipped tarball
2. Examine each .plist file in a text editor or the Apple “Plist Property Editor” program - there will be paths and variables that may have to be altered to match your local site configuration in each file
3. Install as root in “/Library/LaunchDaemons/”

To make an Apple 10.5.x system aware of the new scripts without rebooting or restarting, use the “launchctl load” command shown above.

We’ve put together a tri-fold formatted brochure style quick reference guide for Grid Engine 6.1 and later. Most of the content is from an excellent cheatsheet that Dan Templeton posted to the sge-users mailing list.

Check it out and feel free to leave comments and feedback. We’ll update the PDF and the link below will always work for obtaining the latest version:

Download links (updated 18 February 2008):

• Grid Engine Quick Reference v3 (US 8.5 x 11 paper)
• Grid Engine Quick Reference v3 (A4 layout)

Feb 18 Update: A better link to the article was substituted into this post. The new URL:
http://www.bioinform.com/issues/12_5/webreprints/145121-1.html

A short excerpt the article:

“…These issues are particularly acute for many customers of next-gen sequencing instruments, Cariaso noted, because they are still familiarizing themselves with the type — and volume — of data being generated.

Dick McCombie, a Cold Spring Harbor researcher participating in the lab’s partnership with the BioTeam, agreed. “The data management with these new systems is the single biggest challenge, so we wanted to act pretty aggressively and go with people who we thought could speed up our getting a handle on this,” he said.

McCombie said that his group developed an in-house LIMS system for handling gel and capillary electrophoresis sequencing data, but when it came time to select a similar system for the lab’s Illumina Genome Analyzers, he decided to work with the BioTeam for several reasons.

First of all, he said, his team is still manually analyzing weekly production statistics and tracking the performance of its machines — a task that he’d like to automate as soon as possible, and thinks the BioTeam will likely “get that in place quicker than we would have by hiring people in-house.”

Summary:

A few years ago, having gained familiarity with various microarray platforms, I figured out how to run my own DNA and extract the details. By cataloguing my single nucleotide polymorphisms (SNPs), I knew 500,000 facts about myself, but had no idea about their implications. As my resources were more technical than financial, starting a wiki made more sense than starting a genetic testing company.

Article link:

http://www.bio-itworld.com/issues/2007/dec-jan/inside-the-box-snpedia/

Summary:

Buying CPUs by the hour is back. Marketing apparently decided that ‘On Demand’ sounds pushy and ‘Utility’ and ‘Grid’ are too rigid. This time it will be called a ‘Cloud’, and it will be running inside Amazon.com.

Article link:

http://www.bio-itworld.com/issues/2007/nov/inside-the-box-buying-cpus/