I used to think that every server needed to have something like a DRAC (Dell Remote Access Card) or IBM RSA (IBM Remote System Administration) to be really manageable remotely. But the problem is that they are expensive and they are barely functional. In the case of DRAC, you have to use a browser with ActiveX controls. And although the IBM RSA utilizes java, the consoles have an annoying habit of typinggggggggggggg a lot of eeeeeeeeeeextra characters, especially towards the end of a really long command.
Technological annoyances aside, I really prefer serial consoles. They work great and there are even inexpensive ways to setup networks of remote serial consoles (Project Hydra). But what happens when you need remote access to the BIOS, or to use pxeboot to rescue boot because a kernel upgrade went wrong, or just to reinstall your system? Do you really need that $200 remote access card? The answer is no, provided you have IPMI 2.0 in your baseboard controller which is *very* common in server class hardware.
What is IPMI?
Just close that google tab, I’ll save you the effort: IPMI stands for “Intelligent Platform Management Interface” and is a standard for monitoring and controlling a machine remotely and independently from the operating system. This system management is typically handled by a BMC (Baseboard Management Controller) which is like a second computer inside your server with access to things like fan speed information, power control, system event logs, and SOL also known as Serial Over LAN which when combined with a serial console gives you a fully BIOS capable remote console. IPMI is thus a protocol for interfacing with the BMC, and SOL is the remote access gravy we are after.
How to setup IPMI
The first step is to configure your BMC so that it is remotely accessible. If you have a BMC, you will probably see an option during the POST to enter the BIOS configuration utility. On a Dell SC1435 it looks like this:
In this case, hitting Ctrl-E in the allotted time will take you into the BMC menu, which on a Dell SC1435 looks like this:
The options in this menu are:
- IPMI over LAN: On
- This enables the IPMI protocol over the network.
- NIC Selection: Shared
- In “Shared” mode, the BMC network interface and the
eth0
interface will share the same network port. In this configuration it is correct to think of youreth0
network port as a little hub with two computers connected to it since you will see two separate MAC addresses on this port. - LAN Parameters
- This popup menu is where you configure the network address (static or DHCP) for the BMC interface. One caveat – I have never been able to get the
VLAN ID
settings to work correctly. - LAN User Configuration
- You need to setup a user and password in here in order to enable remote access.
How to setup IPMI without console access
If you already have an OS installed and have remote shell access, you can configure the BMC for remote access from the command line. In Debian, this is done using the ipmitool package and loading a few kernel modules:
apt-get update; apt-get install ipmitool for i in ipmi_msghandler ipmi_si ipmi_devintf ; do modprobe $i; done ipmitool lan set 1 ipsrc static ipmitool lan set 1 ipaddr {YOUR DESIRED IP} ipmitool lan set 1 netmask {YOUR NETMASK} ipmitool lan set 1 auth ADMIN MD5,PASSWORD ipmitool lan set 1 arp respond on ipmitool lan set 1 access on ipmitool user list 1 ipmitool user set password 2 {YOUR PASSWORD} ipmitool lan print 1
NOTE: When you are setting remote access in shared mode on a running system, it is possible to wedge your eth0
connection, especially if you are trying to make the BMC work with VLANs. To avoid the hassle, I usually put the eth0
interface and the BMC interface in the same untagged VLAN.
Accessing the IPMI interface remotely
Now that you have enabled the network interface for your BMC, you can use the ipmitool to do all sort of things. Here are some particularly useful commands:
ipmitool -H {YOUR BMC IP} -U {YOUR USER} power [status|off|on|cycle|reset]
- This connects to the remote BMC and issues the specified power control. Very useful for hard resetting crashed machines.
ipmitool -H {YOUR BMC IP} -U {YOUR USER} shell
- Start a IPMI shell with interactive help – this allows you to issue multiple commands withoug having to type the password each time.
ipmitool -H {YOUR BMC IP} -U {YOUR USER} sel list
- List the contents of the “System Event Log” – this is where you catch things like ECC errors that are causing your server to reboot, etc.
How to setup the serial console redirection
So now you are probably wondering how you access the console thru IPMI, but we are missing a step. The IPMI interface will give us the ability to redirect a serial port over the network, but we need to attach a console to that serial port. This needs to be done in three places:
The BIOS
Reboot your system and go into the BIOS setup. Look for a section like “Serial Communication” or “Console Redirection”. Here is what the option looks like on a Dell SC1435:
- Serial Communication: On with Console Redirection via COM2
- This sets up a serial console using a virtual COM2 port.
- External Serial Connector: COM1
- This machine has a physical serial port, which we set to COM1 so it won’t conflict with the console port.
- Failsafe Baud Rate: 57600
- Setup the baud rate you desire
- Remote Terminal Type: VT100/VT220
- Funny how the VAX terminal type keeps hanging around…
- Redirection after Boot: Enabled
The bootloader and kernel init
The BIOS setup will give us remote access to the BIOS and POST before the bootloader. To get the bootloader and kernel init to use the serial console so we can choose a kernel and see initialization messages, we need to edit the bootloader configuration. For grub on Debian, this can be done with an entry in /boot/grub/menu.lst similar to
# kopt=root=/dev/mapper/sys-root ro console=ttyS1,57600n8
The OS
And finally, we want the booted up system to start a getty on the serial port as well. This can be done with an entry in /etc/inittab similar to T1:23:respawn:/sbin/getty -L ttyS1 57600 vt102
.
Connecting to the IPMI SOL console
At this point, we have a serial console setup on com2 (ttyS1) and we have the BMC setup to make the serial port remote accessible via IPMI SOL. Now we can connect using ipmitool to the console remotely. The catch is that you must load the lanplus
module in order to have access to SOL:
ipmitool -H {YOUR BMC IP} -U {YOUR USER} -I lanplus sol activate
If all goes well, you should now have full remote console access – you can reboot the machine and watch the entire POST, kernel init, and system login remotely. To get out of the console, just type “~.” and it will either close you out of ipmitool or drop you back into the ipmitool shell depending on where you started.
Summary
Once you get IPMI+SOL working, you will have everything you need to remotely manage the machine. I currently use this setup with PXEBoot to allow remote re-installation and rescue booting, which is really nice. Some of the older servers we have don’t do well with higher baud rates, and some require an occassional Ctrl-L
to redraw the screen but for the most part they work great.
Dear all
I want to access a raw machine from remote means how to set pxe boot to a new fresh(raw newly assembled) machine.
Can any one let me know how this could be possible.
Deepak
If you have multiple servers and want to log your ipmi console sessions then check out conlog:
https://github.com/raowen/conlog
Thanks for the writeup, it was really helpful when setting up a Dell SC1435 server.
My biggest mistake was getting the speed settings for the TTYS1 out of sync with what the BIOS was setup to support – until they’re both the same (seemingly this has to be 56700), you’ll see loads of ? [question marks] when connecting to the sol console, and nothing else.
David.
Pingback: ipmi / linux / dell poweredge SC1435 | David Goodwin
Hi,
Really Nice Post !!
I am installing OS on VM using kickstart server (VM) it is working fine. however i do not want to use F12 for Network boot , can we use ipmitool from kickstart server to remotely network boot the machine for installtion of OS.
Any help would be really appreciable..
Thanks alot 🙂
Hi Aarti,
Did you manage to get the network boot setup using IPMI?
Worked like a champ; dunno if it was eaten by WordPress but the %fn in the shell script shulod be %f\n (with a forward slash before the n to force a newline.)You can even hook this up to Nagios to fire off warnings then the temp is out of bounds or if a fan dies with very little extra work.
I have a switch with ipmitool configured on it to talk to 4 blades. I didn’t do any of the initial setup and actually have never used ipmi before until asked to troubleshoot why things aren’t working as expected.
3 of the 4 blades respond correctly to
ipmitool -C 1 -I lanplus -H -U -P -k gkey sol activate
the last one gives an error message : Error: Unable to establish IPMI v2 / RMCP+ session .. using -v with the command gives further ‘enlightenment’ RAKP 2 message indicates an error : unauthorized name using –v gives more information but nothing that seems to provide any additional insight into the issue (could just be me not knowing what I am looking at) I _thought_ that the user name/password was put on the impi server, in this case the switch.. but as I tried to use commands like user summary and user list, it seems that the query is actually being directed to the target i.e. the blades. but I can’t find any instruction on how to create the user and when I log into each of the blades directly it doesn’t appear that ipmi is installed. can any one give me any ideas of what I need to look at to figure out why the one blade is acting differently than the others?
My machine has a dedicated BMC port, and when I connect using this, I get near-perfect responsiveness from the BMC and console SOL logging works perfectly. When I attempt to share BMC with eth0, I get console logging only until eth0 shuts down, and then the logging stops. Ipmi status commands hang for a few seconds too, and then return status. How do I get consistent BMC communication during reboots when the BMC shares with eth0?