alleft.com

I used to think that every server needed to have something like a DRAC (Dell Remote Access Card) or IBM RSA (IBM Remote System Administration) to be really manageable remotely. But the problem is that they are expensive and they are barely functional. In the case of DRAC, you have to use a browser with ActiveX controls. And although the IBM RSA utilizes java, the consoles have an annoying habit of typinggggggggggggg a lot of eeeeeeeeeeextra characters, especially towards the end of a really long command.

Technological annoyances aside, I really prefer serial consoles. They work great and there are even inexpensive ways to setup networks of remote serial consoles (Project Hydra). But what happens when you need remote access to the BIOS, or to use pxeboot to rescue boot because a kernel upgrade went wrong, or just to reinstall your system? Do you really need that $200 remote access card? The answer is no, provided you have IPMI 2.0 in your baseboard controller which is *very* common in server class hardware.

What is IPMI?

Just close that google tab, I’ll save you the effort: IPMI stands for “Intelligent Platform Management Interface” and is a standard for monitoring and controlling a machine remotely and independently from the operating system. This system management is typically handled by a BMC (Baseboard Management Controller) which is like a second computer inside your server with access to things like fan speed information, power control, system event logs, and SOL also known as Serial Over LAN which when combined with a serial console gives you a fully BIOS capable remote console. IPMI is thus a protocol for interfacing with the BMC, and SOL is the remote access gravy we are after.

How to setup IPMI

The first step is to configure your BMC so that it is remotely accessible. If you have a BMC, you will probably see an option during the POST to enter the BIOS configuration utility. On a Dell SC1435 it looks like this:

In this case, hitting Ctrl-E in the allotted time will take you into the BMC menu, which on a Dell SC1435 looks like this:

The options in this menu are:

IPMI over LAN: On 

This enables the IPMI protocol over the network.

NIC Selection: Shared

In “Shared” mode, the BMC network interface and the eth0 interface will share the same network port. In this configuration it is correct to think of your eth0 network port as a little hub with two computers connected to it since you will see two separate MAC addresses on this port.

LAN Parameters 

This popup menu is where you configure the network address (static or DHCP) for the BMC interface. One caveat – I have never been able to get the VLAN ID settings to work correctly.

LAN User Configuration 

You need to setup a user and password in here in order to enable remote access.

How to setup IPMI without console access

If you already have an OS installed and have remote shell access, you can configure the BMC for remote access from the command line. In Debian, this is done using the ipmitool package and loading a few kernel modules:

apt-get update; apt-get install ipmitool

for i in ipmi_msghandler ipmi_si ipmi_devintf ; do modprobe $i; done

ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr {YOUR DESIRED IP}
ipmitool lan set 1 netmask {YOUR NETMASK}
ipmitool lan set 1 auth ADMIN MD5,PASSWORD
ipmitool lan set 1 arp respond on
ipmitool lan set 1 access on
ipmitool user list 1
ipmitool user set password 2 {YOUR PASSWORD}
ipmitool lan print 1

NOTE: When you are setting remote access in shared mode on a running system, it is possible to wedge your eth0 connection, especially if you are trying to make the BMC work with VLANs. To avoid the hassle, I usually put the eth0 interface and the BMC interface in the same untagged VLAN.

Accessing the IPMI interface remotely

Now that you have enabled the network interface for your BMC, you can use the ipmitool to do all sort of things. Here are some particularly useful commands:

ipmitool -H {YOUR BMC IP} -U {YOUR USER} power [status|off|on|cycle|reset]  

This connects to the remote BMC and issues the specified power control. Very useful for hard resetting crashed machines.

ipmitool -H {YOUR BMC IP} -U {YOUR USER} shell 

Start a IPMI shell with interactive help – this allows you to issue multiple commands withoug having to type the password each time.

ipmitool -H {YOUR BMC IP} -U {YOUR USER} sel list 

List the contents of the “System Event Log” – this is where you catch things like ECC errors that are causing your server to reboot, etc.

How to setup the serial console redirection

So now you are probably wondering how you access the console thru IPMI, but we are missing a step. The IPMI interface will give us the ability to redirect a serial port over the network, but we need to attach a console to that serial port. This needs to be done in three places:

The BIOS

Reboot your system and go into the BIOS setup. Look for a section like “Serial Communication” or “Console Redirection”. Here is what the option looks like on a Dell SC1435:

Serial Communication: On with Console Redirection via COM2

This sets up a serial console using a virtual COM2 port.

External Serial Connector: COM1 

This machine has a physical serial port, which we set to COM1 so it won’t conflict with the console port.

Failsafe Baud Rate: 57600 

Setup the baud rate you desire

Remote Terminal Type: VT100/VT220 

Funny how the VAX terminal type keeps hanging around…

Redirection after Boot: Enabled

The bootloader and kernel init

The BIOS setup will give us remote access to the BIOS and POST before the bootloader. To get the bootloader and kernel init to use the serial console so we can choose a kernel and see initialization messages, we need to edit the bootloader configuration. For grub on Debian, this can be done with an entry in /boot/grub/menu.lst similar to

# kopt=root=/dev/mapper/sys-root ro console=ttyS1,57600n8

The OS

And finally, we want the booted up system to start a getty on the serial port as well. This can be done with an entry in /etc/inittab similar to T1:23:respawn:/sbin/getty -L ttyS1 57600 vt102.

Connecting to the IPMI SOL console

At this point, we have a serial console setup on com2 (ttyS1) and we have the BMC setup to make the serial port remote accessible via IPMI SOL. Now we can connect using ipmitool to the console remotely. The catch is that you must load the lanplus module in order to have access to SOL:

ipmitool -H {YOUR BMC IP} -U {YOUR USER} -I lanplus sol activate

If all goes well, you should now have full remote console access – you can reboot the machine and watch the entire POST, kernel init, and system login remotely. To get out of the console, just type “~.” and it will either close you out of ipmitool or drop you back into the ipmitool shell depending on where you started.

Summary

Once you get IPMI+SOL working, you will have everything you need to remotely manage the machine. I currently use this setup with PXEBoot to allow remote re-installation and rescue booting, which is really nice. Some of the older servers we have don’t do well with higher baud rates, and some require an occassional Ctrl-L to redraw the screen but for the most part they work great.