Please do not re-distribute without attribution.

Please do not redistribute without attribution.

Apple provides a GUI for setting up the IP address and authentication information on the LOM. The problem, for me, with using a GUI for something like this is that it’s nearly impossible to script and automate. When I’m setting up a cluster of even a dozen nodes, going through a remote desktop to a GUI becomes intolerable in a hurry.

The command line tool for changing settings on the LOM is a standard one called ‘ipmitool’. While the internal documentation is good, the usage can be a little obscure. Here, therefore, are the commands that I use to set up and enable channel 1 (ethernet port 1, or eth0):

I feel compelled to note that you will need to use different IP addresses for the various hosts you’re setting up. Also, you *really* don’t want to set the LOM Ip address to conflict with the IP address of the operating system. Really, you’re setting up a small, separate, out of band machine to manage the big expensive server. Think of it as a different machine, and you’ll be all set.

For most of the clusters I build, I address the compute nodes as 192.168.2.1 = node001, and so on. For clusters of less than 100 nodes, I set the LOM address to the node address plus 100. Thus, these settings would be appropriate for node015.

ipmitool lan set 1 ipaddr 192.168.2.115
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.2.254
ipmitool lan set 1 access on
ipmitool lan set 1 arp respond on

At this point, the IP address of the LOM ought to respond to pings. All that’s left is to set up and enable a user account. Note that we’re now talking about user number two, rather than port number 1:

ipmitool user set name 2 admin
ipmitool user set password 2
** You get to type the password twice **
ipmitool user enable 2

FInally, I associate user number 2 with port number 1:

ipmitool user priv 2 4 1

I have yet to find a way around typing all those IP addresses into the Server Monitor tool.

* Intro to ssh tunnels
* Chaining ssh tunnels together

The problem addressed in this post is the same as the previous two, except that the remote administrator won’t open up *any* ports in their firewall. Machines on the customer’s network can connect out onto the internet, but there is no direct way to connect in from out here. This is remarkably common, either because the admin only has a limited number of public IP addresses, or because most organizations are very private with their machines. This post describes how to set up a pair of tunnels that meet in the middle on my co-located server, allowing me access to machines that don’t even have a public IP address.

Ssh tunnels are directional. When I open a tunnel with “-L”, I am opening a path from the originating machine forward to the target. The other tunnel argument, “-R”, opens a reverse tunnel from a port on the destination machine back to the originator. In the second post I set up a connection letting me see into remote.customer.com from demo.bioteam.net. However, someone sitting at remote.customer.com wouldn’t be able to see back down that tunnel.

It’s worth noting that at this point we run a risk of violating security policies at the customer site. By enlisting a co-conspirator inside the firewall, we’re getting access to resources that are very clearly supposed to be inaccessible. Bioteam’s practice is to be totally above board about what we are doing and why we’re doing it. Many security / networking folks agree that the goal of a network setup like this is to prevent unauthorized access. In that circumstance, we can frequently agree that my access is authorized, and that it’s simpler for short term or one time connections to use ssh tunnels to get the job done. However, if the policy is actually to prevent any remote access, there’s no getting around it. In those cases, I get in a car or on a plane. Many of my long term clients have started off giving me reverse ssh tunnels while we waited for the organizational paperwork on a “real” solution like VPN access.

Short version: There are times when it’s better to ask forgiveness than permission. Dealing with network security is not one of them. Building trust with the people responsible for a computing network requires both honesty and demonstrated competence over a period of time. If you sneak around behind people’s backs, you will eventually get burned.

The target machine is called: customer.private
I have an account on that machine: bioteam
Bioteam’s co-located server is still: demo.bioteam.net
My account on bioteam’s server: cdwan
The customer also gets an account on bioteam’s server: customer

The customer needs to be logged into customer.private and connect from there to demo.bioteam.net like this:

ssh demo.bioteam.net -l customer -R 9001:customer.private:80

That specification has three parts:

* 9001: The port on demo.bioteam.net on which the tunnel listens
* customer.private: The machine on which the tunnel ends. This is the part where terms like “localhost” get really confusing. In this case “localhost” would work just as well, but I’m making it explicit.
* 80: The port where the tunnel ends.

From my laptop, I open up a forward tunnel exactly as before:

ssh demo.bioteam.net -l cdwan -L 9000:localhost:9001

That’s it. We’ve got a tunnel from my side to port 9001 on demo.bioteam.net, and another tunnel from 9001 to port 80 on customer.private. I connect to http://localhost:9000 on my laptop, and I should see the webserver on customer.private. We’ve created the same chained pair of tunnels from the previous post, but with one originating on each side rather than me pushing both of them forward.

The attentive reader may ask “if there’s a tunnel listening at demo.bioteam.net:9001, can’t you just connect your web server directly to http://demo.bioteam.net:9001? This is a good question. The answer is “yes, but you need to explicitly ask for this functionality.” By default, ssh tunnels only listen to requests originating from within the machine itself. The tunnel listens on what is called the “loopback” interface and is not visible from outside the machine. While it’s possible to get around this, I’ve found it to be a universally bad idea in my work with Bioteam. It represents enough of a potential security hole that I just don’t do it.

Notice also that the customer didn’t have to give me an ssh account on their machine. Instead, we both connected to some machine in the middle (demo.bioteam.net) and I was able to see their web server. As before, as soon as either ssh connection closes, the access is gone. This has a pleasing symmetry insofar as I’m asking for access by giving the customer an account on my machine.

Now for the meat of it: Assume that the customer wants to let me log in via ssh. This means that the reverse tunnel needs to terminate at the ssh port (22) rather than the http port (80). Otherwise it’s exactly the same:

ssh demo.bioteam.net -l customer -R 9002:customer.private:22

With that in place, I can log in to demo.bioteam.net and then run this:

ssh localhost -l bioteam -p 9002

If everything has gone correctly, I will get a password prompt, type my password for cutomer.private, and see a shell prompt for a machine without a public internet IP.

Of course, I can create my own tunnels using that connection. For example, with the reverse tunnel from above in place, I can set up my very own tunnel from port 9010 on my laptop, through port 9011 on demo.bioteam.net, to the webserver on remote.customer.com. This is the part where the “localhosts” get almost impenetrably confusing:

ssh demo.bioteam.net -l cdwan -L 9010:localhost:9011
ssh localhost -l bioteam -p 9002 -L 9011:localhost:80

With this added complexity, we have hit the point where there are many, many ways to skin this particular cat. My aim in these posts is to show how I do things, demonstrating the underlying tools along the way. Hopefully, this will let you build up your own set of techniques in a way that makes sense to you.

A very common practice among the network security folks I meet is to ask “where will you be logging in from?” They then make an exception in their firewall rules for traffic from my particular machine. This makes a lot of sense, but it limits my flexiblity. If I give them the address of my home internet connection, then I won’t be able to connect while I’m on the road, and so on. In addition, if I need to let my colleagues see what I’m dealing with in order to get their help debugging something, we need a way to share that hole in the firewall.

My solution is to give an IP address of a server in Bioteam’s co-location facility, and then use ssh tunnels to bounce my connections through that server and into the customer’s network. This has the added social benefit of giving the customer a “bioteam.net” IP address rather than my home. I think that this looks a bit more professional.

My server is called “demo.bioteam.net”
I log into my server with the username “cdwan”

The customer’s server is “remote.customer.com”
I log into the customer’s server with the username “bioteam”

As before, the first thing to check is that straightforward, vanilla ssh connections work:

ssh demo.bioteam.net -l cdwan

I will be prompted for a password. This first time, I enter the password for “cdwan” on the bioteam machine. Next, within that ssh session, I connect to the customer’s machine:

ssh remote.customer.com -l bioteam

Again, the password prompt – but this time I have to enter the password for “bioteam” on the customer’s machine.

It feels a little pedantic to specify which password goes where, but I find that the second most common source of confusion about ssh tunnels is figuring out which machine is demanding a password. As you start to build up more and more complex systems, it’s key to remember where you are and where you are trying to go.

Assume that we want to take a look at the web server on the remote machine. If the security folks made a complete (all ports) exception for my host (demo.bioteam.net), then this is pretty simple. From my laptop, I do this:

ssh demo.bioteam.net -l cdwan -L 8999:remote.customer.com:80

This is exactly the same as the first example in the first post in this series, except that instead of connecting to “localhost,” I’m telling ssh to point at some external machine.

I will make a practice of changing the port number in these examples (8999 vs. 9000 in this case) to demonstrate that it doesn’t matter what number you use as long as it is not in use for anything else. If you try to open a tunnel on a port that is already in use by another process (someone else’s ssh tunnel to a different machine, for example) the command will fail.

There is a pretty safe range of ports from about 8900 to about 9100 that I tend to use for these purposes. I make a practice of picking numbers pretty much at random. You should do the same, rather than just hitting “up arrow, return” on some command cut and pasted from a blog post.

In actual practice, Bioteam sets up virtual machines – one per customer – to prevent us from stepping on each other’s connections and ssh keys. This has other benefits as well, the best being that we can give customers an account on our systems to share files – which is often simpler to do than convincing their networking team to let us into their machines.

Anyway, with the above tunnel in place, I can look at the URL below and see the remote machine:

http://localhost:8999

It’s worth noting that I could do almost the same thing by opening a remote desktop (or VNC) connection to demo.bioteam.net and running a web browser in that remote session. I find this to be inferior because remote desktops invariably have less pixels and worse mouse control than my local browser.

I would use exactly the same sort of command if the customer wanted me to use their “DMZ” gateway machine to access their network, rather than my own. This happens in lots of cases. The key here is that once I have ssh access to a machine – I can point whatever application I want at anything that machine can see.

The setup above works as long as there is a clear path from demo.bioteam.net to remote.customer.com. As an additional complexity, assume that we now encounter the firewall problem from my first post. The security folks only opened port 22 (ssh) and only from demo.bioteam.net.

To overcome this, we need to chain ssh tunnels together. Here’s the example:

ssh demo.bioteam.net -l cdwan -L 8999:localhost:9000

From within that session, connect to the remote machine like this:

ssh remote.customer.com -l bioteam -L 9000:localhost:80

And then look at the same URL:

http://localhost:8999

Here’s what happens: I have one tunnel leading from my laptop’s port 8999 to port 9000 on demo.bioteam.net. I have a second tunnel from port 9000 into remote.customer.com. Under the hood, ssh near-magically forwards packets along and does the right thing to tie them together.

There is absolutely no need for those tunnels to be opened in the same ssh session or from the same terminal. In fact, in the next post I will show how to open one tunnel from each side to meet in the middle.

It is possible to chain tunnels together arbitrarily. The longest series that I’ve ever built was three deep, but I’m not aware of any fundamental limitation on that – other than my own ability to keep track of accounts and passwords.

This post provides a very basic intro to forward tunnels. I hope to shed some light on reverse tunnels in a future post.

Firewalls and other network devices frequently limit the network ports that I am allowed to access. A very reasonable policy might only allow access over ssh (port 22) from machines outside of a local network. That’s well and good, but in my work I often need to see web servers (port 80) and/or Apple’s Remote Desktop or VNC (port 5900).

As a first puzzle, assume that a friendly administrator gave me an ssh account on her web server. However, because I’m not working onsite, her organization’s security scheme only allows me to get to port 22. I can ssh in, but I can’t really debug some problem with the web browser because port 80 is still blocked. Assume as well that we’ve already called up the network administrator, who said “no,” there are no exceptions to this security policy.

The remote machine is called: “remote.customer.com”
I will log into that remote machine with an account called “cdwan”

I run this perfectly ordinary ssh command to log in, and it works:

ssh remote.customer.com -l cdwan

Keep in mind that if that first command doesn’t work – there’s nothing I can do to make the rest work. With that in hand, I am going to add the forward tunnel specification:

ssh remote.customer.com -l cdwan -L 9000:localhost:80

The tunnel specification is in three parts:

* First is the port on my local machine into which I peer
* The middle is the target hostname I want to connect to as it is seen on the remote machine. In this simple example, I’m going to “localhost”, which really means “remote.customer.com”. I think that this is the most frequent source of confusion with tunnels, figuring out the context in which hostnames are evaluated.
* Last is the port to connect to on that target machine. That’s where the tunnel ends.

So, I’m setting up a connection from my local machine’s port 9000 to port 80 on remote.customer.com.

Once that tunnel is in place, I should be able to open a web browser and look at the URL below to see the web server on the remove machine:

http://localhost:9000

From the point of view of the web browser, I’m looking at port 9000 on my local machine. My ssh connection is catching those requests on port 9000, forwarding them along to the remote machine, and passing them to port 80. Responses come back by the same route.

The connection will only stay open as long as the ssh connection is active. As soon as I log out, my tunnel goes away. While I’m connected, all of my traffic goes over port 22, and is encrypted in exactly the same way as the rest of my ssh traffic. It’s actually a remarkably stable and secure solution to this problem.

I’ll close this post with the fact that ssh accepts multiple instances of the “-L” flag. So, if I want to connect to both a web server (port 80) and a remote desktop server (port 5900), I can do this:

ssh remote.customer.com -l cdwan -L 9000:localhost:80 -L 9001:localhost:5900

With that in place, I can point my web browser at localhost:9000, and my remote desktop client at localhost:9001.

Bioteam uses a pretty standard architecture, where the portal has the IP address and domain settings below. Modifying those for your particular setup is left as an exercise for the reader.

ldapsearch -h 192.168.2.254 -b "dc=cluster,dc=private" -x

slapconfig -setldapstatic 192.168.2.254

Apple’s “Directory Utility” GUI app is great, but for cluster maintenance you really can’t beat the command line.