Revision as of 18:52, 9 December 2011

AN!Wiki :: IPMI

IPMI is an acronym for Intelligent Platform Management Interface. This is a technology built into many server-grade mainboards, sometimes via an optional add-in card, that allows "out of band" access to a server. This means that, via an IPMI interface, a user can remotely connect to a server regardless of it's power state. Through this interface the user can check the power state of the server plus one or more of it's sensors. The user can initiate a reboot, power off, power on and so forth.

IPMI is used as the basis of numerous OEM remote access technologies. To learn more, please see the wikipedia entry.

IPMI Setup for Remote Access and Fencing

If you have already configured your IPMI BMC using software provided by your vendor or via the server's BIOS, you will be able to skip the installation and go directly to the section on testing and using IPMI.

EL6

This covers the installation and usage of IPMI on EL6 (version 6.x of RHEL, CentOS and the like). This was written using two HP Proliant DL165 G7 servers, which use basic IPMI instead of iLO.

Install IPMI

Installing IPMI is quite simple;

yum install freeipmi freeipmi-bmc-watchdog freeipmi-ipmidetectd OpenIPMI OpenIPMI-libs OpenIPMI-perl OpenIPMI-tools
chkconfig ipmi on
/etc/init.d/ipmi start

Talking to the BMC

We can check to make sure that we've got access to the local BMC using ipmitool;

ipmitool chassis status

System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : previous
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
Sleep Button Disable : allowed
Diag Button Disable  : allowed
Reset Button Disable : allowed
Power Button Disable : allowed
Sleep Button Disabled: false
Diag Button Disabled : false
Reset Button Disabled: false
Power Button Disabled: false

ipmitool mc info

Device ID                 : 18
Device Revision           : 1
Firmware Revision         : 4.22
IPMI Version              : 2.0
Manufacturer ID           : 11
Manufacturer Name         : Hewlett-Packard
Product ID                : 0 (0x0000)
Product Name              : Unknown (0x0)
Device Available          : yes
Provides Device SDRs      : no
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    Bridge
    Chassis Device
Aux Firmware Rev Info     : 
    0x03
    0x01
    0x00
    0x00

ipmitool fru print

FRU Device Description : Builtin FRU Device (ID 0)
 Unknown FRU header version 0x00

FRU Device Description : System FRU (ID 1)
 Product Manufacturer  : HP      
 Product Name          : ProLiant DL165 G7
 Product Serial        : xxxxxxxxxx
 Product Asset Tag     :

You can read sensor data using ipmitool as well.

ipmitool sdr list

CPU1 DIMM1       | 98h | ok  | 32.1 | 32.50 degrees C
CPU1 DIMM2       | 97h | ns  | 32.2 | No Reading
CPU1 DIMM3       | AEh | ns  | 32.3 | No Reading
CPU1 DIMM4       | 9Fh | ok  | 32.4 | 31.50 degrees C
CPU1 DIMM5       | E9h | ns  | 32.5 | No Reading
CPU1 DIMM6       | 9Dh | ns  | 32.6 | No Reading
CPU1 DIMM7       | B5h | ok  | 32.7 | 30.50 degrees C
CPU1 DIMM8       | A1h | ns  | 32.8 | No Reading
CPU1 DIMM9       | E3h | ns  | 32.9 | No Reading
CPU1 DIMM10      | EDh | ok  | 32.10 | 31 degrees C
CPU1 DIMM11      | EBh | ns  | 32.11 | No Reading
CPU1 DIMM12      | ECh | ns  | 32.12 | No Reading
CPU2 DIMM1       | B1h | ok  | 32.13 | 29.50 degrees C
CPU2 DIMM2       | B0h | ns  | 32.14 | No Reading
CPU2 DIMM3       | E7h | ns  | 32.15 | No Reading
CPU2 DIMM4       | B4h | ok  | 32.16 | 29 degrees C
CPU2 DIMM5       | B3h | ns  | 32.17 | No Reading
CPU2 DIMM6       | B2h | ns  | 32.18 | No Reading
CPU2 DIMM7       | E8h | ok  | 32.19 | 28.50 degrees C
CPU2 DIMM8       | 82h | ns  | 32.20 | No Reading
CPU2 DIMM9       | 1Dh | ns  | 32.21 | No Reading
CPU2 DIMM10      | 1Ch | ok  | 32.22 | 29 degrees C
CPU2 DIMM11      | 1Bh | ns  | 32.23 | No Reading
CPU2 DIMM12      | EAh | ns  | 32.24 | No Reading
Inlet Ambient    | C7h | ok  | 64.1 | 25 degrees C
CPU1             | 7Fh | ok  | 65.1 | 33 degrees C
CPU2             | 93h | ok  | 65.2 | 31 degrees C
PCI Outlet       | 0Dh | ok  | 66.1 | 50 degrees C
SR5670 Outlet    | 0Eh | ok  | 66.2 | 45 degrees C
Rear Board       | 0Fh | ok  | 66.3 | 50.50 degrees C
PCI Mid          | A8h | ok  | 66.10 | 36 degrees C
PCI Inlet        | E4h | ok  | 66.11 | 30 degrees C
Rear Board2      | AAh | ok  | 66.12 | 39 degrees C
CPU Inlet        | 83h | ok  | 66.13 | 28 degrees C
PS Inlet Ambient | ACh | ok  | 66.14 | 28 degrees C
IO Controller    | D3h | ok  | 66.15 | 55 degrees C
PCI16_RISER      | D8h | ok  | 66.16 | 36.50 degrees C
PCI4_RISER       | DAh | ok  | 66.17 | 32 degrees C
[root@il0051 ~]# ipmitool sdr list
CPU2 Therm Trip  | 0x01              | ok
CPU1 PROC Hot    | 0x01              | ok
CPU2 PROC Hot    | 0x01              | ok
CPU1 Present     | 0x00              | ok
CPU2 Present     | 0x00              | ok
CPU1 Therm Trip  | 0x01              | ok
NMI Detect       | 0x01              | ok
PVCORE1          | 1.02 Volts        | ok
PVNB CPU1        | 1.10 Volts        | ok
P5V              | 5.12 Volts        | ok
P5V STBY         | 5.05 Volts        | ok
P3V3             | 3.31 Volts        | ok
P3V3 STBY        | 3.33 Volts        | ok
P1V1 SR5670      | 1.11 Volts        | ok
Fan1 Inlet       | 5709.72 RPM       | ok
Fan1 Outlet      | 4702.12 RPM       | ok
Fan2 Inlet       | 5848.98 RPM       | ok
Fan2 Outlet      | 4702.12 RPM       | ok
Fan3 Inlet       | 5709.72 RPM       | ok
Fan3 Outlet      | 4702.12 RPM       | ok
Fan4 Inlet       | 5709.72 RPM       | ok
Fan4 Outlet      | 4702.12 RPM       | ok
Fan5 Inlet       | 5709.72 RPM       | ok
Fan5 Outlet      | 4702.12 RPM       | ok
Fan6 Inlet       | 5450.19 RPM       | ok
Fan6 Outlet      | 4440.89 RPM       | ok
Fan7 Inlet       | 5329.07 RPM       | ok
Fan7 Outlet      | 4440.89 RPM       | ok
Fan Redundant    | 0x01              | ok
CPU1 DIMM1       | 32.50 degrees C   | ok
CPU1 DIMM2       | no reading        | ns
CPU1 DIMM3       | no reading        | ns
CPU1 DIMM4       | 31.50 degrees C   | ok
CPU1 DIMM5       | no reading        | ns
CPU1 DIMM6       | no reading        | ns
CPU1 DIMM7       | 30.50 degrees C   | ok
CPU1 DIMM8       | no reading        | ns
CPU1 DIMM9       | no reading        | ns
CPU1 DIMM10      | 31 degrees C      | ok
CPU1 DIMM11      | no reading        | ns
CPU1 DIMM12      | no reading        | ns
CPU2 DIMM1       | 29.50 degrees C   | ok
CPU2 DIMM2       | no reading        | ns
CPU2 DIMM3       | no reading        | ns
CPU2 DIMM4       | 29 degrees C      | ok
CPU2 DIMM5       | no reading        | ns
CPU2 DIMM6       | no reading        | ns
CPU2 DIMM7       | 28.50 degrees C   | ok
CPU2 DIMM8       | no reading        | ns
CPU2 DIMM9       | no reading        | ns
CPU2 DIMM10      | 29 degrees C      | ok
CPU2 DIMM11      | no reading        | ns
CPU2 DIMM12      | no reading        | ns
Inlet Ambient    | 25 degrees C      | ok
CPU1             | 33 degrees C      | ok
CPU2             | 31 degrees C      | ok
PCI Outlet       | 49.50 degrees C   | ok
SR5670 Outlet    | 45 degrees C      | ok
Rear Board       | 50 degrees C      | ok
PCI Mid          | 36 degrees C      | ok
PCI Inlet        | 30 degrees C      | ok
Rear Board2      | 39 degrees C      | ok
CPU Inlet        | 28 degrees C      | ok
PS Inlet Ambient | 28 degrees C      | ok
IO Controller    | 55 degrees C      | ok
PCI16_RISER      | 36.50 degrees C   | ok
PCI4_RISER       | 32 degrees C      | ok
Watchdog         | 0x00              | ok
Chassis          | 0x01              | ok
ACPI State       | Not Readable      | ns

If you want just a subset of the sensor data, you can narrow it down by first seeing which categories are available, then requesting just that group's sensor readings. Let's look at just the Temperature readings;

ipmitool sdr type list

Sensor Types:
	Temperature                 Voltage                  
	Current                     Fan                      
	Physical Security           Platform Security        
	Processor                   Power Supply             
	Power Unit                  Cooling Device           
	Other                       Memory                   
	Drive Slot / Bay            POST Memory Resize       
	System Firmwares            Event Logging Disabled   
	Watchdog                    System Event             
	Critical Interrupt          Button                   
	Module / Board              Microcontroller          
	Add-in Card                 Chassis                  
	Chip Set                    Other FRU                
	Cable / Interconnect        Terminator               
	System Boot Initiated       Boot Error               
	OS Boot                     OS Critical Stop         
	Slot / Connector            System ACPI Power State  
	Watchdog                    Platform Alert           
	Entity Presence             Monitor ASIC             
	LAN                         Management Subsystem Health
	Battery                     Session Audit            
	Version Change              FRU State

ipmitool sdr type Temperature

CPU1 DIMM1       | 98h | ok  | 32.1 | 32.50 degrees C
CPU1 DIMM2       | 97h | ns  | 32.2 | No Reading
CPU1 DIMM3       | AEh | ns  | 32.3 | No Reading
CPU1 DIMM4       | 9Fh | ok  | 32.4 | 31.50 degrees C
CPU1 DIMM5       | E9h | ns  | 32.5 | No Reading
CPU1 DIMM6       | 9Dh | ns  | 32.6 | No Reading
CPU1 DIMM7       | B5h | ok  | 32.7 | 30.50 degrees C
CPU1 DIMM8       | A1h | ns  | 32.8 | No Reading
CPU1 DIMM9       | E3h | ns  | 32.9 | No Reading
CPU1 DIMM10      | EDh | ok  | 32.10 | 31 degrees C
CPU1 DIMM11      | EBh | ns  | 32.11 | No Reading
CPU1 DIMM12      | ECh | ns  | 32.12 | No Reading
CPU2 DIMM1       | B1h | ok  | 32.13 | 29.50 degrees C
CPU2 DIMM2       | B0h | ns  | 32.14 | No Reading
CPU2 DIMM3       | E7h | ns  | 32.15 | No Reading
CPU2 DIMM4       | B4h | ok  | 32.16 | 29 degrees C
CPU2 DIMM5       | B3h | ns  | 32.17 | No Reading
CPU2 DIMM6       | B2h | ns  | 32.18 | No Reading
CPU2 DIMM7       | E8h | ok  | 32.19 | 28.50 degrees C
CPU2 DIMM8       | 82h | ns  | 32.20 | No Reading
CPU2 DIMM9       | 1Dh | ns  | 32.21 | No Reading
CPU2 DIMM10      | 1Ch | ok  | 32.22 | 29 degrees C
CPU2 DIMM11      | 1Bh | ns  | 32.23 | No Reading
CPU2 DIMM12      | EAh | ns  | 32.24 | No Reading
Inlet Ambient    | C7h | ok  | 64.1 | 25 degrees C
CPU1             | 7Fh | ok  | 65.1 | 33 degrees C
CPU2             | 93h | ok  | 65.2 | 31 degrees C
PCI Outlet       | 0Dh | ok  | 66.1 | 49.50 degrees C
SR5670 Outlet    | 0Eh | ok  | 66.2 | 45 degrees C
Rear Board       | 0Fh | ok  | 66.3 | 50 degrees C
PCI Mid          | A8h | ok  | 66.10 | 36 degrees C
PCI Inlet        | E4h | ok  | 66.11 | 30 degrees C
Rear Board2      | AAh | ok  | 66.12 | 39 degrees C
CPU Inlet        | 83h | ok  | 66.13 | 28 degrees C
PS Inlet Ambient | ACh | ok  | 66.14 | 28 degrees C
IO Controller    | D3h | ok  | 66.15 | 55 degrees C
PCI16_RISER      | D8h | ok  | 66.16 | 36.50 degrees C
PCI4_RISER       | DAh | ok  | 66.17 | 32 degrees C

Perfect, we're now confident that our BMC is available and working properly.

Configure the BMC

BMC is an acronym for "Baseboard Management Controller", which is the name of the actual IPMI controller found inside your server. It can be thought of as a fully separate device, despite residing within the server or even integrated on the mainboard itself. It has it's own MAC and IP addresses, access credentials and so forth.

Finding the BMC's Channel

We will need to configure the BMC's access credentials so that we can talk to it from other nodes. To do this, we need to know which channel our BMC uses. This depends on your vendor, so we'll need to do a little poking around.

IPMI channels start at 1. So we'll use the following call, incrementing by one, until we get some output;

Try channel 1;

ipmitool lan print 1

Channel 1 is not a LAN channel

Nothing, let's try channel 2;

ipmitool lan print 2

Set in Progress         : Set Complete
Auth Type Support       : NONE MD5 PASSWORD 
Auth Type Enable        : Callback : NONE MD5 PASSWORD 
                        : User     : NONE MD5 PASSWORD 
                        : Operator : NONE MD5 PASSWORD 
                        : Admin    : NONE MD5 PASSWORD 
                        : OEM      : NONE MD5 PASSWORD 
IP Address Source       : Static Address
IP Address              : 10.20.1.51
Subnet Mask             : 255.255.0.0
MAC Address             : 78:e3:b5:0e:f8:14
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 2.0 seconds
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3
Cipher Suite Priv Max   : OOOOXXXXXXXXXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM

Hit! So our BMC uses channel 2. Knowing this, we can now configure the BMC!

Configuring Network Details

Knowing that we're using channel 2, we can now configure the BMC's network settings.

You can see the current configuration using the same command we used above when confirming we could talk to the BMC.

ipmitool lan print 2

Set in Progress         : Set Complete
Auth Type Support       : NONE MD5 PASSWORD 
Auth Type Enable        : Callback : NONE MD5 PASSWORD 
                        : User     : NONE MD5 PASSWORD 
                        : Operator : NONE MD5 PASSWORD 
                        : Admin    : NONE MD5 PASSWORD 
                        : OEM      : NONE MD5 PASSWORD 
IP Address Source       : Static Address
IP Address              : 10.20.1.51
Subnet Mask             : 255.255.0.0
MAC Address             : 78:e3:b5:0e:f8:14
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 2.0 seconds
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3
Cipher Suite Priv Max   : OOOOXXXXXXXXXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM

I want to switch the above IP address, but leave the rest as-is. For the sake of a tutorial though, let's look at resetting all the network options.

The following four commands will, in order, tell the BMC to use a static IP address (vs. dhcp), to use the IP address 10.20.1.51 with the subnet mask of 255.255.0.0 and use the default gateway 10.20.255.254. Remember that we're using channel 2.

ipmitool lan set 2 ipsrc static
ipmitool lan set 2 ipaddr 10.20.1.1

Setting LAN IP Address to 10.20.1.1

ipmitool lan set 2 netmask 255.255.0.0

Setting LAN Subnet Mask to 255.255.0.0

ipmitool lan set 2 defgw ipaddr 10.20.255.254

Setting LAN Default Gateway IP to 10.20.255.254

The changes will take effect immediately. You can confirm the new configuring by re-printing the configuration.

ipmitool lan print 2

Set in Progress         : Set Complete
Auth Type Support       : NONE MD5 PASSWORD 
Auth Type Enable        : Callback : NONE MD5 PASSWORD 
                        : User     : NONE MD5 PASSWORD 
                        : Operator : NONE MD5 PASSWORD 
                        : Admin    : NONE MD5 PASSWORD 
                        : OEM      : NONE MD5 PASSWORD 
IP Address Source       : Static Address
IP Address              : 10.20.1.1
Subnet Mask             : 255.255.0.0
MAC Address             : 78:e3:b5:0e:f8:14
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl   : 2.0 seconds
Default Gateway IP      : 10.20.255.254
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3
Cipher Suite Priv Max   : OOOOXXXXXXXXXXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM

You should now be able to ping the interface.

ping 10.20.1.1 -c 3

PING 10.20.1.1 (10.20.1.1) 56(84) bytes of data.
64 bytes from 10.20.1.1: icmp_seq=1 ttl=255 time=1.36 ms
64 bytes from 10.20.1.1: icmp_seq=2 ttl=255 time=0.282 ms
64 bytes from 10.20.1.1: icmp_seq=3 ttl=255 time=0.259 ms

At this point, I add the IP address to my cluster nodes' /etc/hosts file (or to DNS where available).

Configuring Authentication =

Last part of the configuration is to setup user access. To do is get a list of what user accounts exist on the BMC (remember that we're using channel 2), we can run the following command;

ipmitool user list 2

ID  Name	     Callin  Link Auth	IPMI Msg   Channel Priv Limit
1                    true    true       true       USER
2   Operator         true    true       true       OPERATOR
3   admin            true    true       true       ADMINISTRATOR
4   OEM              true    true       true       OEM
5   Operator         true    true       true       OPERATOR
6   admin            true    true       true       ADMINISTRATOR
7   OEM              true    true       true       OEM
8   Operator         true    true       true       OPERATOR
9   admin            true    true       true       ADMINISTRATOR
10  OEM              true    true       true       OEM
11  Operator         true    true       true       OPERATOR
12  admin            true    true       true       ADMINISTRATOR
13  OEM              true    true       true       OEM
14  Operator         true    true       true       OPERATOR
15  admin            true    true       true       ADMINISTRATOR
16  OEM              true    true       true       OEM

Let's set the user ID 3 (admin) password to secret now.

ipmitool user set password 3 secret

Now we're ready to use IPMI!

Testing Remote IPMI Access and Control

Now from another machine, let's check the power status of the node we just configured. In my case, my two nodes are called il0051 and il0052 and I've created entries is /etc/hosts for il0051.ipmi and il0052.ipmi, which I use below.

ipmitool -I lan -U admin -P secret -H il0051.ipmi chassis status

System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : previous
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false
Sleep Button Disable : allowed
Diag Button Disable  : allowed
Reset Button Disable : allowed
Power Button Disable : allowed
Sleep Button Disabled: false
Diag Button Disabled : false
Reset Button Disabled: false
Power Button Disabled: false

Hoozah!

Now the final test; Powering off and then restarting the machine.

ipmitool -I lan -U admin -P secret -H il0051.ipmi chassis power off

Chassis Power Control: Down/Off

The target server should now be offline! Let's start it back up.

ipmitool -I lan -U admin -P secret -H il0051.ipmi chassis power on

Chassis Power Control: Up/On

If it powered up, you're done!

Any questions, feedback, advice, complaints or meanderings are welcome.
`Alteeve's Niche!`	`Enterprise Support: Alteeve Support`	`Community Support`
© Alteeve's Niche! Inc. 1997-2024		Anvil! "Intelligent Availability®" Platform
`legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.`

IPMI: Difference between revisions