Screwing around with the boot volume is part of our regular “explore around the edges” work before we get serious with how we are going to configure and orchestrate the new systems. The boot volume in this scenario does not have PV driver support and thus will perform slower than the actual ephemeral storage. Our need was for the boot volume to be big enough to hammer with bonnie++ – this is not something we’d do in a production scenario.

Background

All of the cloud nerds at BioTeam are thrilled now that the Amazon Compute Cluster nodes have been publicly launched. If you missed the exciting news please visit the announcement post over at the AWS blog - http://aws.typepad.com/aws/2010/07/the-new-amazon-ec2-instance-type-the-cluster-compute-instance.html

We’ve been madly banging on the new instance types and trying to (initially) perform some basic low level benchmarks before we go on to the cooler benchmarks where we run actual life science & informatics pipelines against the hot new gear. Using our Chef Server it’s trivial for us to orchestrate these new systems into working HPC clusters in just a matter of minutes. We plan to start blogging and demoing live deployment of elastic genome assembly pipelines and NextGen DNA sequencing instrument pipelines (like the Illumina software) on AWS soon.

Like James Hamilton says, the real value in these new EC2 server types lies in the non-blocking 10Gigabit ethernet network backing them up. All of a sudden our “legacy” cluster and compute farm practices that involved network filesharing among nodes via NFS, GlusterFS, Lustre and GPFS seem actually feasible rather than a sick masochistic exercise in cloud futility.

We expect to see quite a bit of news in the near future about people using NFS, pNFS and other parallel/cluster filesystems for HPC data sharing on AWS – seems like a no-brainer now that we have full bisectional 10GbE bandwidth between our Compute Cluster cc1.4xlarge instance types.

However, despite the fact that the coolest stuff is going to involve what we can do now node-to-node over the fast new networking fabric there is still value in doing the low-level “what does this new environment look and feel like?” tests involving EBS disk volumes, S3 access and the like.

The Amazon AWS team did a great job preparing the way for people who want to quickly experiment with the new HPC instance types. The EC2 AMI images have to boot off of EBS and run under HVM virtualization rather than the standard paravirtualization used on the other instance types.

Recognizing that bootstrapping a HVM-aware EBS-booting EC2 server instance is a non-trivial exercise, AWS created a QuickStart public AMI with CentOS Linux that anyone can use right away:

The Problem – how to grow the default 20GB system disk in the QuickStart AMI?

As beautiful as the CentOS HVM AMI is, it only gives us 20GB of disk space in it’s native form. This is perfectly fine for most of our use cases but presented a problem when we decided we wanted to use bonnie++ to perform some disk IO benchmarks on the local boot volume to complement our tests against more traditional mounted EBS volumes and RAID0 stripe sets.

Bonnie++ really wants to work against a filesystem that is at least twice the size of available RAM so as to mitigate any memory-related caching issues when testing actual IO performance.

The cc1.4xlarge “Cluster Compute” instance comes with ~23GB of physical RAM. Thus our problem — we wanted to run bonnie++ against the local system disk but the disk is actually smaller than the amount of RAM available to the instance!

For this one particular IO test we really wanted a HVM-compatible AMI that had at least 50GB of storage on the boot volume.

The Solution

I was shocked and amazed to find that in about ~20 minutes of screwing around with EBS snapshot sizes, instance disk partitions and LVM settings I was able to achieve the goal of converting the Amazon Quickstart 20GB AMI into a custom version where the system disk was 80Gb in size.

The fact that this was possible and achievable out of the box without having to debug mysterious boot failures, kernel panics and all the other sorts of things I’m used to dealing with when messing with low level disk and partition issues is the ultimate testament to both Amazon’s engineering prowess (how cool is it that we can launch EBS snapshots of arbitrary size?) as well as the current excellent state of Linux, Grub and LVM2.

I took a bunch of rough notes so I’d remember how the heck I managed to do this. Then I decided to clean up the notes and really document the process in case it might help someone else.

The Process

I will try to walk through step-by-step the commands and methods used to increase the system boot disk from 20GB in size to 80GB in size.

It boils down to two main steps

• Launch the Amazon QuickStart AMI but override & increase the default 20GB boot disk size
• Get the CentOS Linux OS to recognize that it’s now running on a bigger disk

You can’t do the following step using the AWS Web Management Console as the webUI does not let you alter the parameters of the block device settings. You will need the command-line EC2 utilities installed and working in order to proceed.

On the command line we can easily tell Amazon that we want to start the QuickStart AMI but instead of launching it within a 20GB snapshot of the EBS boot volume we will launch it against a much larger snapshot.

If you look at the info for AMI ami-7ea24a17 within the web page you will see this under the details for the block devices that will be available to the system at boot:

Block Devices: /dev/sda1=snap-1099e578:20:true

That is basically saying that Linux Device “/dev/sda1″ will be built from EBS snapshot volume “snap-1099e578″. The next “:” delimited parameter sets the size to 20GB.

We are going to change that from 20GB to 80GB.

Here is the command to launch that AMI using a disk of 80GB in size instead of the default 20GB.

In the following screenshot, note how we are starting the Amazon AMI ami-7ea24a17 with a block device (the “-b” switch) that is bootstrapping itself from the snapshot “snap-1099e578″.

All we needed to do in order to make the EC2 server have a larger boot disk is pass in “80″ to override the default value of 20GB. Confused? Look at the “-b” block device argument below, the 80GB is set right after the snapshot name:

Your EBS volume might take a bit longer than normal to boot up but once it is online and available you can login normally.

Of course, the system will appear to have the default 20GB system disk:

Even the LVM2 physical disk reports show the ~20GB settings:

However, if we actually use the ‘fdisk’ command to examine the disk we see that the block device is, indeed, much larger than the 20GB the Operating System thinks it has to utilize:

Fdisk tells us that disk device /dev/hda has 85.8 GB of physical capacity.

Now we need to teach the OS to make use of that space!

There are only two partitions on this disk, /dev/hda1 is the 100MB /boot partition common to RedHat varients. Lets leave that alone.

The second partition, /dev/hda2 is already set up for logical volumes under LVM. We are going to be lazy. We are just going to use ‘fdisk’ to delete the /dev/hda2 partition so that we can immediately recreate it so that it spans the full remaining space on the physical drive.

After typing “fdisk /dev/hda” we type “d” and delete partition “2″. Then we type “n” for a new partition of type “p” for primary and “2″ to name it as the second partition. After that we just hit return to accept the default suggestions for the begin and end of the recreated second partion.

If it all worked, we can type the “p” command to print the new partition table out.

Note how /dev/hda2 now has many more blocks? Cool!

We are not done yet. None of our partition changes have actually been written to disk yet. We still need to type “w”  to write the new partition table down to disk and “q” to exit.

Obviously we can’t make live changes on a running boot disk. The new partition settings will come into effect after a system reboot.

Now we reboot the system and wait for it to come back up.

When it comes back up, don’t be alarmed that both ‘df’ and ‘pvscan’ still show the incorrect size:

We can fix that! Now we are in the realm of LVM so we need to use the “pvresize <device>” command to rescan the physical disk. Since our LVM2 partition is still /dev/hda2 that is the physical device path we give it:

Success! LVM recognizes that the drive is larger than 20GB.

With LVM aware that the disk is larger we are pretty much done. We can resize an existing logical volume or add a new one to the default Volume Group (“VolGroup00″).

Since I’m lazy AND I want to mount the extra space away from the root (“/”) volume I chose to create a new logical volume that shares the same /dev/hda2 physical volume (“PV”) and Volume Group (“VG”).

We are going to use the command “lvcreate VolGroup00 –size 60G /dev/hda2?” to make a new 60GB logical volume that is part of the existing Volume Group named “VolGroup00″:

Success. Note that our new logical volume got assigned a default name of “lvol0″ and it now exists in the LVM device path of “/dev/mapper/VolGroup00-lvol0″.

Now we need to place a Linux filesystem on our new 60GB of additional space and mount it up. Since I am a fan of XFS on EC2 I need to first install the “xfsprogs” RPM and then format the volume. A simple “yum -y install xfsprogs” does the trick and now I can make XFS filesystems on my server:

Success. We now have 60GB more space, visible to the OS and formatted with a filesystem. The final step is to mount it.

And we are done. We’ve successfully converted the 20GB Amazon QuickStart AMI into a version with a much larger boot volume.

Conclusion

None of this is rocket science. It’s actually just Linux Systems Administration 101.

The real magic here is how easily this is all accomplished on our virtual cloud system using nothing but a web browser and some command-line utilities.

What makes this process special for me is how quick and easy it was – anyone who has spent any significant amount of time managing many physical Linux server systems knows the pain and hours lost when trying to do this stuff in the real world on real (and flaky) hardware.

I can’t even count how many hours of my life I’ve lost trying to debug Grub bootloader failures, mysterious kernel panics and other hard-to-troubleshoot booting and disk resizing efforts on production and development server systems when I’ve altered settings that we’ve covered in this post. In cluster environments we often have to do this debugging via a 9600 baud serial console or via flaky IPMI consoles. It’s just nasty.

The fact that this method worked so quickly and so smoothly is probably only amazing to people who know the real pain of having done this in the field, crouched on the floor of a freezing cold datacenter and trying not to pull your hair out as text scrolls slowly by at 9600 baud.

Congrats to the Amazon AWS team. Fantastic work. It’s a real win when virtual infrastructure is this easy to manipulate.

BioTeam is a Massachusetts-based “virtual” company in that all of our employees work from home when not traveling on business. Being from Massachusetts is actually important because as of March 31st, 2010 the state has passed a fairly strict set of laws concerning how companies handle digital data.

Without going into the details of the Massachusetts data protection laws lets just say that there is one key attribute that we as a company fixated on:

• Any company that can prove that a lost/stolen/missing data storage device or computer was encrypted at the time it went missing is exempt from the rules covering mandatory event reporting and public breach disclosure.

Interesting, huh? Just to be straight here the rule does not exempt us from doing anything stupid or illegal. The law is just written so that mandatory press releases and other data loss notification events do not have to happen if the missing device was provably encrypted (and thus unreadable or unusable) at the time it got lost.

I’m totally fine with this law and I’m happy that MA is being aggressive about making sure companies are responsible with sensitive data types and records.

PGP Whole Disk Encryption & PGP Universal Server

I don’t want to shill for PGP so lets cut to the chase and namedrop the software products we chose to allow BioTeam to encrypt all employee digital data in a verifiable way:

• We use PGP Desktop with Whole Disk Encryption on all employee Apple Macbook Pro’s (and a few random Linux & Windows netbooks). The PGP Desktop product does a lot more than just disk encryption but we use it mainly for rendering our laptops unusable without the decryption key as well as to encrypt our USB storage and other portable media devices.
• The real magic, however, comes with PGP Universal Server. This product is an external server that manages our desktop PGP clients including enforcing policy rules, auditing the state of encryption and (handily) making sure that our employee laptops are always loaded with a secondary decryption key that an IT Administrator can use to decrypt the laptop if the staffer ever forgets his or her passphrase.

Make sense? PGP Desktop/w WDE is the product we use on the laptops to do the actual encryption and PGP Universal Server is the product that we use for policy management and to generate an audit trail. This is the software combo that fits nicely our needs under the Massachusetts data protection laws but it’s also just common sense IT procedures for any tech-centric business.

PGP and Small Business

I could write a whole blog post on the nightmare involved in getting someone at PGP to actually sell me this software. It was clear (at the time) that the company itself and it’s legion of resellers simply didn’t give a damn about small customers looking for less than 100 licensed seats. It was only after I left a whiny blog comment on the internet detailing my hassles that someone at PGP reached out and put me in touch with a reseller who understood small business and was willing to sell us a 10-seat enterprise license for Universal Server.

Not sure if things have changed but if you are an Apple shop looking to go down the same road drop me a line and I’ll put you in touch with the reseller who hooked us up. That said though, it looks like PGP now has “small business” and “workgroup” edition software now so maybe our use of Universal Server is overkill for the size of our shop.

Citrix XenEnterprise 4, Universal Server 2 and Apple OS X 10.5

Prior to Apple releasing version 10.6 (“Snow Leopard” of Mac OS X we had things running quite well:

• Virtualization system running Citrix XenEnterprise 4
• PGP Universal Server 2.x running virtual under XenEnterprise 4
• PGP Desktop 9.x running on our laptops

Running Universal Server under XenEnterprise is not supported by PGP but the install worked perfectly off of the supplied .iso image. The main problem we had was that PGP Universal Server would not reliably report encryption status of Apple disk devices. The devices were actually encrypted but the status change rarely showed up in the reporting tools. This was a “known bug” and we were told to sit tight for a future release.

The reason for this post:
PGP Universal Server 3, XenServer 5.5 and Apple OS X 10.6

Now we come for the actual reason for this blog post — documenting the process (painful) of getting PGP Universal Server 3 to actually function while virtualized under the open-source version of Citrix XenServer 5.5.

Warning: I’m leaving out tons of information about the mechanics of actually doing the upgrade/install including the details of preserving your Organizational keypair and an encrypted backup configuration file — both of which are necessary if you want to reload the configuration of your old Universal Server into the new 3.0 system that has to be install from scratch via a CentOS-based .ISO image. Read the PGP docs. Understand also that according to PGP running Universal Server under any virutalization system other than VMWare is also unsupported. PGP will also probably not like you mucking around as root within the Universal Server console as well.

Problems with Universal Server 3 and XenCenter 5.5

The problems I had trying to install from the .iso simplified down into the following issues:

1. Booting off of the .iso image and letting the “standard” install happen automatically seems to work all the way through the final reboot. However, when the system comes up after it’s inital reboot it freaks out over what appears to be ext3 related filesystem corruption and hangs forever during the boot process.
2. Choosing the “expert” install method results in a python crash that takes out the CentOS Anaconda installer software, leaving you stuck and only able to reboot and start over.
3. Choosing the “customnet” install method fails in the same way that “expert” does — a python stacktrace pops up and you are stuck only seconds into the install process

I did find a manual workaround that got us functional (see screen images below). The basic details are below. Hopefully this will help some other poor soul in the same position!

1. Boot the PGP .ISO image and choose the “noautopart” install option. For some reason I did not get the anaconda crash when using this method
2. Manually partition your disk. Easy for me since I’ve been using CentOS forever but might be hard for some. I chose a very simple partition layout: 100MB ext3 primary partition for “/boot”, a 2GB swap partition, a 2GB ext3 “/tmp” partition, a 3GB ext3 “/var” partition and finally an ext3 partition that spanned the rest of the disk as the “/” root partition.
3. The install will continue automatically. One side effect of the “noautopart” option is that the system will not prompt for network setup information and will auto-configure your first NIC to use the IP address 192.168.1.100. Fixing that is the next step
4. My system actually booted up after following the above steps except for the fact that the PostgreSQL database failed to start and initialize. I am not 100% sure if the problem was due to not having enough memory in the VM when I first booted it (I mistakenly left it at 256MB) or if the database really wanted a fully functional network before it would properly start up.
5. If your database actually initialized then you still have the problem of the hard-coded network settings. To fix these you’ll have to first reboot the server into single user mode by appending “s” or “single” to the kernel parameter in the grub boot menu
6. Once you are in single-user mode you can use ‘vi’ to edit the file “/opt/ovid/prefs.xml” — this is the central configuration file that PGP uses to generate all of the specific Linux configuration settings and files. You must edit this file, manually editing network settings or things like /etc/resolv.conf will get overwritten. The file is pretty straightforward XML, just enter the information for your interfaces. A key thing to remember is that if your primary interface is not “Interface 1″ then there are a number of lines in the prefs.xml file that refer to what interface is used for critical services — make sure you find and replace all instances of “Interface 1″ with the interface you actually plan to use.

There is a command line tool called “pgpsysconf” that can be used for things like “pgpsysconf –network –check” and “pgpsysconf –network” that will (a) check your network settings and (b) reconfigure the server to use them. They are worth using if only to make sure you did not leave any typos or mistakes in the prefs.xml file.

Didn’t mean to be so long winded, the simple process looks like this:

1. Boot the ISO, choose “noautopart” and manually partition your disk
2. Boot into single user mode so that you can hand edit and fix /etc/ovid/prefs.xml to enter the real networking and IP information

If you have postgres database startup or initialization problems, boot into single user mode and look at the log file that lives in /var/lib/pgsql. All I had to do was delete the “data/” directory there and reboot and PGP automatically reinitialized the database.

If you made it to this point you have a functional server and you can now use the web interface to upload your Organizational key and backup configuration file. No more need for hacking around within the VM console.

Some screenshots below.

Please do not re-distribute without attribution.

Please do not redistribute without attribution.

Apple provides a GUI for setting up the IP address and authentication information on the LOM. The problem, for me, with using a GUI for something like this is that it’s nearly impossible to script and automate. When I’m setting up a cluster of even a dozen nodes, going through a remote desktop to a GUI becomes intolerable in a hurry.

The command line tool for changing settings on the LOM is a standard one called ‘ipmitool’. While the internal documentation is good, the usage can be a little obscure. Here, therefore, are the commands that I use to set up and enable channel 1 (ethernet port 1, or eth0):

I feel compelled to note that you will need to use different IP addresses for the various hosts you’re setting up. Also, you *really* don’t want to set the LOM Ip address to conflict with the IP address of the operating system. Really, you’re setting up a small, separate, out of band machine to manage the big expensive server. Think of it as a different machine, and you’ll be all set.

For most of the clusters I build, I address the compute nodes as 192.168.2.1 = node001, and so on. For clusters of less than 100 nodes, I set the LOM address to the node address plus 100. Thus, these settings would be appropriate for node015.

ipmitool lan set 1 ipaddr 192.168.2.115
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.2.254
ipmitool lan set 1 access on
ipmitool lan set 1 arp respond on

At this point, the IP address of the LOM ought to respond to pings. All that’s left is to set up and enable a user account. Note that we’re now talking about user number two, rather than port number 1:

ipmitool user set name 2 admin
ipmitool user set password 2
** You get to type the password twice **
ipmitool user enable 2

FInally, I associate user number 2 with port number 1:

ipmitool user priv 2 4 1

I have yet to find a way around typing all those IP addresses into the Server Monitor tool.

* Intro to ssh tunnels
* Chaining ssh tunnels together

The problem addressed in this post is the same as the previous two, except that the remote administrator won’t open up *any* ports in their firewall. Machines on the customer’s network can connect out onto the internet, but there is no direct way to connect in from out here. This is remarkably common, either because the admin only has a limited number of public IP addresses, or because most organizations are very private with their machines. This post describes how to set up a pair of tunnels that meet in the middle on my co-located server, allowing me access to machines that don’t even have a public IP address.

Ssh tunnels are directional. When I open a tunnel with “-L”, I am opening a path from the originating machine forward to the target. The other tunnel argument, “-R”, opens a reverse tunnel from a port on the destination machine back to the originator. In the second post I set up a connection letting me see into remote.customer.com from demo.bioteam.net. However, someone sitting at remote.customer.com wouldn’t be able to see back down that tunnel.

It’s worth noting that at this point we run a risk of violating security policies at the customer site. By enlisting a co-conspirator inside the firewall, we’re getting access to resources that are very clearly supposed to be inaccessible. Bioteam’s practice is to be totally above board about what we are doing and why we’re doing it. Many security / networking folks agree that the goal of a network setup like this is to prevent unauthorized access. In that circumstance, we can frequently agree that my access is authorized, and that it’s simpler for short term or one time connections to use ssh tunnels to get the job done. However, if the policy is actually to prevent any remote access, there’s no getting around it. In those cases, I get in a car or on a plane. Many of my long term clients have started off giving me reverse ssh tunnels while we waited for the organizational paperwork on a “real” solution like VPN access.

Short version: There are times when it’s better to ask forgiveness than permission. Dealing with network security is not one of them. Building trust with the people responsible for a computing network requires both honesty and demonstrated competence over a period of time. If you sneak around behind people’s backs, you will eventually get burned.

The target machine is called: customer.private
I have an account on that machine: bioteam
Bioteam’s co-located server is still: demo.bioteam.net
My account on bioteam’s server: cdwan
The customer also gets an account on bioteam’s server: customer

The customer needs to be logged into customer.private and connect from there to demo.bioteam.net like this:

ssh demo.bioteam.net -l customer -R 9001:customer.private:80

That specification has three parts:

* 9001: The port on demo.bioteam.net on which the tunnel listens
* customer.private: The machine on which the tunnel ends. This is the part where terms like “localhost” get really confusing. In this case “localhost” would work just as well, but I’m making it explicit.
* 80: The port where the tunnel ends.

From my laptop, I open up a forward tunnel exactly as before:

ssh demo.bioteam.net -l cdwan -L 9000:localhost:9001

That’s it. We’ve got a tunnel from my side to port 9001 on demo.bioteam.net, and another tunnel from 9001 to port 80 on customer.private. I connect to http://localhost:9000 on my laptop, and I should see the webserver on customer.private. We’ve created the same chained pair of tunnels from the previous post, but with one originating on each side rather than me pushing both of them forward.

The attentive reader may ask “if there’s a tunnel listening at demo.bioteam.net:9001, can’t you just connect your web server directly to http://demo.bioteam.net:9001? This is a good question. The answer is “yes, but you need to explicitly ask for this functionality.” By default, ssh tunnels only listen to requests originating from within the machine itself. The tunnel listens on what is called the “loopback” interface and is not visible from outside the machine. While it’s possible to get around this, I’ve found it to be a universally bad idea in my work with Bioteam. It represents enough of a potential security hole that I just don’t do it.

Notice also that the customer didn’t have to give me an ssh account on their machine. Instead, we both connected to some machine in the middle (demo.bioteam.net) and I was able to see their web server. As before, as soon as either ssh connection closes, the access is gone. This has a pleasing symmetry insofar as I’m asking for access by giving the customer an account on my machine.

Now for the meat of it: Assume that the customer wants to let me log in via ssh. This means that the reverse tunnel needs to terminate at the ssh port (22) rather than the http port (80). Otherwise it’s exactly the same:

ssh demo.bioteam.net -l customer -R 9002:customer.private:22

With that in place, I can log in to demo.bioteam.net and then run this:

ssh localhost -l bioteam -p 9002

If everything has gone correctly, I will get a password prompt, type my password for cutomer.private, and see a shell prompt for a machine without a public internet IP.

Of course, I can create my own tunnels using that connection. For example, with the reverse tunnel from above in place, I can set up my very own tunnel from port 9010 on my laptop, through port 9011 on demo.bioteam.net, to the webserver on remote.customer.com. This is the part where the “localhosts” get almost impenetrably confusing:

ssh demo.bioteam.net -l cdwan -L 9010:localhost:9011
ssh localhost -l bioteam -p 9002 -L 9011:localhost:80

With this added complexity, we have hit the point where there are many, many ways to skin this particular cat. My aim in these posts is to show how I do things, demonstrating the underlying tools along the way. Hopefully, this will let you build up your own set of techniques in a way that makes sense to you.

A very common practice among the network security folks I meet is to ask “where will you be logging in from?” They then make an exception in their firewall rules for traffic from my particular machine. This makes a lot of sense, but it limits my flexiblity. If I give them the address of my home internet connection, then I won’t be able to connect while I’m on the road, and so on. In addition, if I need to let my colleagues see what I’m dealing with in order to get their help debugging something, we need a way to share that hole in the firewall.

My solution is to give an IP address of a server in Bioteam’s co-location facility, and then use ssh tunnels to bounce my connections through that server and into the customer’s network. This has the added social benefit of giving the customer a “bioteam.net” IP address rather than my home. I think that this looks a bit more professional.

My server is called “demo.bioteam.net”
I log into my server with the username “cdwan”

The customer’s server is “remote.customer.com”
I log into the customer’s server with the username “bioteam”

As before, the first thing to check is that straightforward, vanilla ssh connections work:

ssh demo.bioteam.net -l cdwan

I will be prompted for a password. This first time, I enter the password for “cdwan” on the bioteam machine. Next, within that ssh session, I connect to the customer’s machine:

ssh remote.customer.com -l bioteam

Again, the password prompt – but this time I have to enter the password for “bioteam” on the customer’s machine.

It feels a little pedantic to specify which password goes where, but I find that the second most common source of confusion about ssh tunnels is figuring out which machine is demanding a password. As you start to build up more and more complex systems, it’s key to remember where you are and where you are trying to go.

Assume that we want to take a look at the web server on the remote machine. If the security folks made a complete (all ports) exception for my host (demo.bioteam.net), then this is pretty simple. From my laptop, I do this:

ssh demo.bioteam.net -l cdwan -L 8999:remote.customer.com:80

This is exactly the same as the first example in the first post in this series, except that instead of connecting to “localhost,” I’m telling ssh to point at some external machine.

I will make a practice of changing the port number in these examples (8999 vs. 9000 in this case) to demonstrate that it doesn’t matter what number you use as long as it is not in use for anything else. If you try to open a tunnel on a port that is already in use by another process (someone else’s ssh tunnel to a different machine, for example) the command will fail.

There is a pretty safe range of ports from about 8900 to about 9100 that I tend to use for these purposes. I make a practice of picking numbers pretty much at random. You should do the same, rather than just hitting “up arrow, return” on some command cut and pasted from a blog post.

In actual practice, Bioteam sets up virtual machines – one per customer – to prevent us from stepping on each other’s connections and ssh keys. This has other benefits as well, the best being that we can give customers an account on our systems to share files – which is often simpler to do than convincing their networking team to let us into their machines.

Anyway, with the above tunnel in place, I can look at the URL below and see the remote machine:

http://localhost:8999

It’s worth noting that I could do almost the same thing by opening a remote desktop (or VNC) connection to demo.bioteam.net and running a web browser in that remote session. I find this to be inferior because remote desktops invariably have less pixels and worse mouse control than my local browser.

I would use exactly the same sort of command if the customer wanted me to use their “DMZ” gateway machine to access their network, rather than my own. This happens in lots of cases. The key here is that once I have ssh access to a machine – I can point whatever application I want at anything that machine can see.

The setup above works as long as there is a clear path from demo.bioteam.net to remote.customer.com. As an additional complexity, assume that we now encounter the firewall problem from my first post. The security folks only opened port 22 (ssh) and only from demo.bioteam.net.

To overcome this, we need to chain ssh tunnels together. Here’s the example:

ssh demo.bioteam.net -l cdwan -L 8999:localhost:9000

From within that session, connect to the remote machine like this:

ssh remote.customer.com -l bioteam -L 9000:localhost:80

And then look at the same URL:

http://localhost:8999

Here’s what happens: I have one tunnel leading from my laptop’s port 8999 to port 9000 on demo.bioteam.net. I have a second tunnel from port 9000 into remote.customer.com. Under the hood, ssh near-magically forwards packets along and does the right thing to tie them together.

There is absolutely no need for those tunnels to be opened in the same ssh session or from the same terminal. In fact, in the next post I will show how to open one tunnel from each side to meet in the middle.

It is possible to chain tunnels together arbitrarily. The longest series that I’ve ever built was three deep, but I’m not aware of any fundamental limitation on that – other than my own ability to keep track of accounts and passwords.

This post provides a very basic intro to forward tunnels. I hope to shed some light on reverse tunnels in a future post.

Firewalls and other network devices frequently limit the network ports that I am allowed to access. A very reasonable policy might only allow access over ssh (port 22) from machines outside of a local network. That’s well and good, but in my work I often need to see web servers (port 80) and/or Apple’s Remote Desktop or VNC (port 5900).

As a first puzzle, assume that a friendly administrator gave me an ssh account on her web server. However, because I’m not working onsite, her organization’s security scheme only allows me to get to port 22. I can ssh in, but I can’t really debug some problem with the web browser because port 80 is still blocked. Assume as well that we’ve already called up the network administrator, who said “no,” there are no exceptions to this security policy.

The remote machine is called: “remote.customer.com”
I will log into that remote machine with an account called “cdwan”

I run this perfectly ordinary ssh command to log in, and it works:

ssh remote.customer.com -l cdwan

Keep in mind that if that first command doesn’t work – there’s nothing I can do to make the rest work. With that in hand, I am going to add the forward tunnel specification:

ssh remote.customer.com -l cdwan -L 9000:localhost:80

The tunnel specification is in three parts:

* First is the port on my local machine into which I peer
* The middle is the target hostname I want to connect to as it is seen on the remote machine. In this simple example, I’m going to “localhost”, which really means “remote.customer.com”. I think that this is the most frequent source of confusion with tunnels, figuring out the context in which hostnames are evaluated.
* Last is the port to connect to on that target machine. That’s where the tunnel ends.

So, I’m setting up a connection from my local machine’s port 9000 to port 80 on remote.customer.com.

Once that tunnel is in place, I should be able to open a web browser and look at the URL below to see the web server on the remove machine:

http://localhost:9000

From the point of view of the web browser, I’m looking at port 9000 on my local machine. My ssh connection is catching those requests on port 9000, forwarding them along to the remote machine, and passing them to port 80. Responses come back by the same route.

The connection will only stay open as long as the ssh connection is active. As soon as I log out, my tunnel goes away. While I’m connected, all of my traffic goes over port 22, and is encrypted in exactly the same way as the rest of my ssh traffic. It’s actually a remarkably stable and secure solution to this problem.

I’ll close this post with the fact that ssh accepts multiple instances of the “-L” flag. So, if I want to connect to both a web server (port 80) and a remote desktop server (port 5900), I can do this:

ssh remote.customer.com -l cdwan -L 9000:localhost:80 -L 9001:localhost:5900

With that in place, I can point my web browser at localhost:9000, and my remote desktop client at localhost:9001.

Bioteam uses a pretty standard architecture, where the portal has the IP address and domain settings below. Modifying those for your particular setup is left as an exercise for the reader.

ldapsearch -h 192.168.2.254 -b "dc=cluster,dc=private" -x

slapconfig -setldapstatic 192.168.2.254

Apple’s “Directory Utility” GUI app is great, but for cluster maintenance you really can’t beat the command line.