As of early 2010, Grid Engine is effectively not installable out of the box on modern version of Mac OS X. We have seen this recently with server and client versions of OS X 10.5.* as well as the new Snow Leopard (10.6.*) releases.

This used to not be a big deal and the workarounds were trivial. Something seems to have changed, however, in recent updates to OS X that render the Grid Engine binaries unable to function when :

• When started during the installation process
• SystemStarter() scripts that SGE tries to install on Apple systems
• Manually started by user root via the command line

This is obviously a show-stopper now for new SGE users. We have no idea why this is the case but have witnessed the behavior on many 10.5 and 10.6 systems (many of them running OS X Server).

The problem with Grid Engine on modern versions of OS X can be simply stated:

SGE binaries will not launch or will be unreliable when started by ANY METHOD other than the system level OS X “launchd” service framework.

… and since neither the SGE installation scripts nor “traditional” SystemStarter() scripts that SGE install on Mac OS X systems use launchd framework this basically means that SGE 6.2 is unusable out of the box without manual intervention and custom starter scripts.

How to manually get SGE 6.2x working on Mac OS X systems

This process will be very simple to someone who already understands Grid Engine administration and is comfortable with SGE admin commands – unfortunately it may be confusing to novice or beginners because we have to interrupt the automatic SGE installation process and complete things by hand using SGE admin commands.

The method:

1. Run the “install_qmaster” script normally. The fact that it will fail is not a big deal – before it fails it will construct the SGE_CELL directory, configure spooling and otherwise do all of the behind the scenes steps necessary to support a functional SGE qmaster process.
2. When the install_qmaster script fails, exit out and manually kill any zombie sge_qmaster daemons that may be hanging around on the system
3. Download and run the sge-launchd-scriptmaker tarball from this BioTeam Blog post.
4. Run the sge-launchd-scriptmaker utility. This simple perl script will query your SGE environment and construct several .plist files suitable for copying into the /Library/LaunchdDaemons/ OS X folder
5. Using OS X “launchctl” commands, restart sge_qmaster
6. At this point the SGE qmaster is running and functional and we can manually complete a few additional configuration steps…
7. Create and populate a hostgroup named “@allhosts”
8. Create and configure the default cluster queue object (“all.q”)
9. At this point the SGE install_execd script should work but you can also skip that step and just launchd sge_execd via the launchctl framework (a step that will be necessary even if the exexhost installer script functions fine)

The screencast below shows a recording of me stepping through this process on a small Apple Mac Mini running Mac OS X 10.5.8 Server (sorry, I thought it was a Snow Leopard box when I began the work …).

If you don’t want to watch the embedded video below, you can navigate directly to the screencast site and download the full movie file. The video is hosted here: http://www.screencast.com/t/NjMyNGJiNWM

Feedback welcome.

Please do not redistribute without attribution.

Another reason for releasing the materials is that they themselves are quite dry and boring – real Grid Engine training needs to be done by someone who has used the product in demanding settings and is capable of talking about real world experiences, failures and best practices!

All we ask is that our effort be respected – please don’t strip out our company name, logo or otherwise try to pass these materials off as your own.

BioTeam continues to offer custom Grid Engine Administration (and End User) training suitable for training sessions lasting from two hours to two full days. Please contact us for more information.

01 - SGE Admin - Overview

02 - SGE Admin - Installation

03 - SGE Admin - Configuration

04 - SGE Admin - Application Integration

05 - SGE Admin - Monitoring

06 - SGE Admin - Policies

07 - SGE Admin - Troubleshooting

Slide download link: 2009 Grid Engine & Amazon EC2

2009 Sun HPC Workshop

Regensburg, Germany

The SGE Workshop returns to Regensburg, now operating as a conference track within the 2009 Sun HPC Workshop event.

Go here for details & registration:

http://hpcworkshop.com/

Pictures I took at the 2007 event:

• Regensburg 2007 Pictures
• 2007 Workshop Pictures

I’ve agreed to participate in a webinar and answer questions about the project, click on through for the full event details.

Date: Wednesday, September 17th

Time: 2:00 pm – 3:00 pm Central Standard Time

Register at: http://www.univaud.com/hpc/webinar20080917.php

Presented by leaders in HPC and Web Services, this webinar will provide IT decision-makers and System Administrators a unique opportunity to access real world information and examples of deploying clusters in a cloud environment.

Attendees will:

-Learn about Amazon’s EC2 web services, the UniCluster open sourcesoftware stack, and deployment of this software within EC2

-Gain insight into key data points from the people out in the field making it happen

-Be able to engage in a technical Q and A with the author of the technical white paper: “Deploying UniCluster in Amazon EC2″

Learn more about the webinar and register to attend at:

http://www.univaud.com/hpc/webinar20080917.php

Starting first in the Boston area and expanding elsewhere if demand allows we are offering the following Grid Engine training workshops:

1-Day Grid Engine User/Usage Workshop Dates

• April 15, 2008
• June 17, 2008

2-Day Grid Engine Administration Workshop Dates

• April 16-17, 2008
• June 18-19, 2008

Registration & Enrollment Deadlines

To register, contact info@bioteam.net

• Registration ends for the April workshops on April 7th, 2008
• Registration ends for the June workshops on June 9th, 2008

Workshop Location:

Residence Inn by Marriott (link)
6 Cambridge Center
Cambridge, MA, 02142 USA

SGE User Workshop Description

BioTeam is the first organization to offer open-enrollment training focused on making people more comfortable and more productive. Our goal is to help users, application integrators and developers understand Grid Engine features and capabilities in a way that allows them to become more productive at home. New SGE cluster operators or systems administrators may also benefit from the user and usage-centric perspective.

SGE Admin Workshop Description

Intended to provide comprehensive coverage of Grid Engine administration in a small-group, highly interactive setting. This workshop covers more in-depth technical material than other “Advanced” SGE training offerings. However, we do not use the word “Advanced” to describe this workshop, as Grid Engine sourcecode details, internal data structures, JGDI, ARCo queries and the DRMAA API are not typically addressed.

Workshop Brochure (click to download)

SGE User Workshop Details

Duration

1 Day

Cost

$375 USD (special pricing through June 2008). List price: $475 USD.

Intended Audience

Anyone interested in a user-centric view of distributed computing with Grid Engine. Note: this is not an advanced operator/admin course.

Our goal is to help users, application integrators and developers understand Grid Engine features and capabilities in a way that allows them to become more productive at home. New SGE cluster operators or administrators also may benefit from the user and usage-centric perspective.

Scope

Our materials use the open source Sun Grid Engine 6 product from http://gridengine.sunsource.net. Lab systems may run version 6.0 although 6.1 specific features can be addressed. Upon request, SGE 5.x behavior and features can also be covered.

Difficulty

No Grid Engine experience necessary but attendees should be comfortable with Linux and a bit of shell/perl scripting in order to work with demo code and examples presented in the longer seminar format. The seminar is targeted towards new or moderately experienced Grid Engine users. Individuals with advanced Grid Engine skills will not gain much from this 1-day seminar – we are considering offering future sessions targeted specifically at advanced end-user audiences.

Prerequisites

Attendees must bring wireless-capable laptops. A SSH client is required in order to connect to the classroom and lab systems.

Workshop Content Outline

I. Clustering Overview
II. Grid Problem Types
III. Distributed Resource Management
- Overview
- What it provides
IV. How Grid Engine Works
- Job Types
- Resource Allocation
- Policies
- Tickets
- Architecture
- Scheduling
V. Basic Usage
- Running Jobs
- Checking Status
- Troubleshooting
VI. Advanced Usage
- Resource requests
- Job Arrays
- Job Dependencies
- Simple Workflows
- Parallel Environments
- Loose vs Tight parallel environment integration
- Understanding resources & the SGE complex
VII. Policy Based Resource Allocation
- Overview
- How it works
- Effect on end-users
- Commonly seen implementations
VIII. Grid Engine Application Integration
- Standalone applications & web front-ends
- Parallel applications
- Licensed applications

SGE Administration Workshop Details

Duration

2 Days

Cost

$1,100 USD (special pricing through June 2008). List price: $1,400 USD.

Intended Audience

New or existing Grid Engine administrators or cluster operators.

Intended to provide comprehensive coverage of Grid Engine administration in a small-group, highly interactive setting.

Advanced Course?

This workshop covers more in-depth technical material than other “Advanced” SGE training offerings. However, we do not use the word “Advanced” to describe this workshop, as Grid Engine sourcecode issues, internal data structures, JGDI, ARCo queries and the DRMAA API are not typically addressed.

Scope

Our materials use the open source Sun Grid Engine 6 product from http://gridengine.sunsource.net. Lab systems may run version 6.0 although 6.1 specific features can be addressed. Differences between Grid Engine versions 5.x through 6.2 will be addressed.

Difficulty

Prior Grid Engine experience is highly desirable but not strictly required. We’ve run this workshop successfully with audiences of Linux and cluster system administrators having no previous SGE exposure.

Attendees should be comfortable and capable with Linux, command line operating envirinments and shell/perl scripting in order to work with demo code and classroom exercises.

Prerequisites

Attendees must bring wireless-capable laptops. A SSH client is required in order to connect to the classroom and lab systems.

Workshop Content Outline

I. Overview
- Grid Problem Types / High-throughput vs. High-performance
- Distributed Resource Management
-- DRM Benefits for Users & Workgroups
-- DRM Benefits for operators & organizations
- How Grid Engine Works
- Grid Engine Architecture
- Grid Engine Job Lifecycle
II. Installation & Deployment
- Pre-installation decisions
- Spooling & High Availability issues / decision points
- Template driven autoinstallation
- CSP mode installation
III. CLI Configuration
- Command line operation
- Host and hostgroup configuration
- Cluster configuration
- Queue configuration
- Parallel Environments
- Resources & the SGE Complex
- Load Sensors
- Access Control
- Aliases & Default requests
IV. SGE 5.x vs. 6.x vs. 6.1x
- Feature & behavior differences between major SGE release milestones
V. SGE Usage for Admins
- Brief review of SGE usage with focus on User-centric needs
VI. Monitoring & Reporting
- System Monitoring
- SGE Accounting
- SGE Reporting
- Accounting & Reporting tools
- 3rd party tools & utilities
VII. Policies & Resource Allocation
- Ticket-based Policies
-- Share Tree
-- Functional
- Urgency & Override Policies
- Resource Reservations & Backfilling
- Resource Quotas
IX. Application & Workflow Integration
X. Troubleshooting
- Cluster, job and general-purpose troubleshooting & diagnostics


William Van Etten has produced new scripts for the Apple OS X “Launchd” framework used by Mac OS X 10.5 “Leopard” systems. Moving forward it seems that Apple eventually intends to phase out the SystemStarter methods for starting/stopping daemons and services in favor of the newer launchd mechanisms.

A Wiki entry over at http://gridengine.info explains some of the problems with Launchd and shows an (older) potential workaround.

The new scripts referenced in this blog post take advantage of a newly discovered (by us) environment variable called “SGE_ND” — if this variable is set, the SGE daemons will not fork off and daemonize (something that Apple Launchd does not allow). By including the “SGE_ND=1” environment variable into the Grid Engine startup environment we are able to produce launchd-compatible scripts that function well at automatically enabling Grid Engine startup and shutdown (and theoretically automatic restart of crashed daemons).

Bill has created three launchd.plist(s)…

1) sgeqmaster.plist
2) sgeschedd.plist
3) sgeexecd.plist

Using these plists, one can load with …

launchctl load /Library/LaunchDaemons/net.sunsource.gridengine.sgeqmaster.plist
launchctl load /Library/LaunchDaemons/net.sunsource.gridengine.sgeschedd.plist
launchctl load /Library/LaunchDaemons/net.sunsource.gridengine.sgeexecd.plist

and start/stop with…

launchctl start net.sunsource.gridengine.sgeqmaster
launchctl start net.sunsource.gridengine.sgeschedd
launchctl start net.sunsource.gridengine.sgeexecd

launchctl stop net.sunsource.gridengine.sgeqmaster
launchctl stop net.sunsource.gridengine.sgeschedd
launchctl stop net.sunsource.gridengine.sgeexecd

If these scripts are installed in “/Library/LaunchDaemons/” they will also activate automatically on system boot or restarts.

Download link & Installation Instructions

The .plist scripts are available here: sge_launchd.tgz

To install them:

1. Download the gzipped tarball
2. Examine each .plist file in a text editor or the Apple “Plist Property Editor” program – there will be paths and variables that may have to be altered to match your local site configuration in each file
3. Install as root in “/Library/LaunchDaemons/”

To make an Apple 10.5.x system aware of the new scripts without rebooting or restarting, use the “launchctl load” command shown above.

The above image shows a pair of Illumina Genome Analyzer systems installed in a laboratory setting. The Illumina systems are one example of what people are calling “Next Generation” systems for doing DNA and whole-Genome analysis. Everyone is working hard to produce the fabled “$1000 Genome” and there is a fantastic scientific and technology race currently ongoing to see which system, which platform and which vendor will succeed. There is even a $10 Million Dollar X-Prize for the first team to do 100 human genomes in the space of ten days. Exciting times!

What does this have to do with Grid Engine? Well, each one of these instruments is individually capable of producing terabyte-level volumes of data in just a few instrument runs. The compute power required to analyze the data coming off of these lab instruments greatly exceeds the capacity of the workstation class machines that drive the instrument. In fact, the 2 instruments you see in the image above are supported by a cluster in the basement with more than 40 terabytes of raw online storage attached.

In the case of the above Illumina/Solexa machines, Grid Engine is actually a key part of the analysis pipeline. In order to operate the instruments shown above, a working SGE installation is required. My employer has been doing quite a lot of consulting and systems integration work these days helping users and labs integrate these devices into local infrastructures and workflows. Along the way we’ve met a number of people who have expressed an interest in a small mailing list or community for swapping tips and best practices.

I also believe (but can’t speak in public on the topic …) that the Illumina/Solexa devices are just the first wet-lab oriented SGE-using instruments to enter the market. There will likely be more.

Thus was born a small google group forum:

http://groups.google.com/group/sge-lifescience-sig

The goal of the group is to build a tightly focused community that can share knowledge and tips on how to integrate these SGE-using devices into new or existing settings. Many vendors also assume that their customers will exclusively buy their technology platform and will use their analysis systems on dedicated hardware — this is not what we see out in the real world where we are already trying to support multi-vendor sequencing environments and shared cluster/storage resources for the back end analysis pipelines.

We’ve made the group name generic rather than making it Illumina/Solexa specific in order to accommodate future products and technologies. If interested, visit http://groups.google.com/group/sge-lifescience-sig and sign up.

We’ve put together a tri-fold formatted brochure style quick reference guide for Grid Engine 6.1 and later. Most of the content is from an excellent cheatsheet that Dan Templeton posted to the sge-users mailing list.

Check it out and feel free to leave comments and feedback. We’ll update the PDF and the link below will always work for obtaining the latest version:

Download links (updated 18 February 2008):

• Grid Engine Quick Reference v3 (US 8.5 x 11 paper)
• Grid Engine Quick Reference v3 (A4 layout)