Using conserver to monitor iLO virtual serial ports (updated)

Sep.07, 2008 in System Administration

There are a lot of companies that use serial console servers (e.g. Cyclades) to monitor the serial ports of their servers. These servers are usually running an operating system like Linux or FreeBSD. By accessing the serial console, the administrator can control almost everything on the server, and even watch it boot. This type of server management originated with Unix servers that did not have a video console – just a legacy RS-232 DB9 serial port.

With the advent of blade servers like the HP BladeSystem, the use of serial console servers is no longer necessary. The blade server and blade chassis typically has this functionality built in. In the case of the HP BladeSystem, the serial console access is provided by the iLO (Integrated Lights-Out) embedded management CPU within each server blade. iLO, in combination with the ProLiant BIOS, provides a “virtual serial port”. With a terminal emulation program (e.g. PuTTY) you can Telnet or SSH to a blade’s iLO, and connect to its virtual serial port.

This works very well, but what if you want to automatically collect all your serial console output, yet still have serial console access when you need it, do run some console command? On top of that requirement, what if you have 100′s of blade servers to manage? That’s where a great little tool called conserver comes in. Conserver will log serial console output and give you login access to the consoles at the same time.

Prerequisites

To start using Conserver, it’s best to setup a separate Linux server that can be dedicated to console logging and other management tasks. It should have network access to the iLO’s and Onboard Administrator network connections. Install the latest version of Conserver from http://www.conserver.com and also make sure that expect is installed.

Next, gather up all your iLO IP addresses. Update 11/7/08: The example here assumes that the iLO IP addresses are in your /etc/hosts file and you’ve given them hostnames.You’ll also need to know the Administrator password on the iLO’s. If you are buying in bulk from HP, these two items are something can can easily be pre-configured for you byHP Factory Express.

Instead of using passwords to access iLO, you could also use SSH key-based authentication. This is an advanced topic that I will cover in another posting. Update 11/7/08: The sample scripts are written assuming that SSH key-based authentication is working (for the user Administrator).

One final thing you need to enable is the operating system’s serial console (e.g. ttyS0 for Linux) and make sure that kernel logging is enabled on that port. I have covered this in another blog entry located here: Enabling Linux Serial Console with a Script

Configuring conserver for iLO Virtual Serial Port Logging

Finally we get to the code! Here’s an example conserver.cf configuration file containing the template to use for iLO virtual serial port connections:

Example 1: Conserver config file /usr/local/etc/conserver.cf

## Sample conserver.cf file for accessing iLO Virtual consoles## Filename: /usr/local/etc/conserver.cf## Requires conserver v8.1.9 or later, ilo-vsplogin script# in /usr/local/etc and 'expect'.# access * { trusted localhost; }config * { daemonmode on; logfile /var/log/conserver.log; }default * { master localhost; rw *; } ## Default template for all iLO Virtual Serial Port consoles# default ilo-ssh-vsp { type exec; execsubst H=cs; exec ssh -o ConnectionAttempts=15 \ -o ServerAliveInterval=15 \ -i /root/.ssh/id_dsa Administrator@H; initcmd "/usr/local/etc/ilo-startvsp"; logfile "/var/log/conserver/&"; logfilemax 4m; timestamp 10l;} ## One line here for each iLO# console ilo1 { include ilo-ssh-vsp; }console ilo2 { include ilo-ssh-vsp; }console ilo3 { include ilo-ssh-vsp; }console ilo4 { include ilo-ssh-vsp; }console ilo5 { include ilo-ssh-vsp; }

The second part we have to install is the expect script that will allow us to automatically login to iLO and connect us to the Virtual Serial Port:

Example 2: iLO auto-login to VSP expect script (ilo-startvsp)

#!/usr/bin/expect -f####################################################################### Sample expect script for automating iLO VSP Logins with conserver ######################################################################## Filename: /usr/local/etc/ilo-startvsp # Hardware Supported: any server with iLO-2# Virtual Serial Port must be configured in RBSU.# To be useful, the operating System must be running a console on# the VSP (Virtual Serial Port). set send_slow {1 0.02} expect "</>hpiLO->"sleep 5 send -s "vsp"sleep 1 send "\r" exit 0

Running conserver

With the config file and expect script in place, you can now start ‘conserver’. To verify that conserver started successfully, check its log file at /var/log/conserver.log. To see the serial console data that is being logged, check the directory /var/log/conserver/. You should see a log file for each iLO that you are monitoring.

Accessing a serial console

Use the “console” command to access the serial console of any of your servers. If your iLO’s have the same hostnames as in the script above, here’s an example:

Example 3: Accessing a serial console from conserver

[root@linux ~]# console ilo1[Enter `^Ec?' for help] Red Hat Enterprise Linux AS release 4 (Nahant Update 6)Kernel 2.6.9-67.0.7.ELsmp on an x86_64 linux-blade-1 login: [disconnect][root@linux ~]#

After the console command connects you (and you see the “Enter ^Ec? for help” message) you will need to press Enter to get a new login prompt. At this point, it behaves as if you were connected directly to the serial port of the server.