Here are two ways to update the firmware on Dell iDRAC6 remote access cards.

Both methods require downloading the BIOS from Dell and extracting it from the bundle. For example, this is the 1.70.21 firmware:

mkdir /tmp/dell
cd /tmp/dell
wget http://ftp.dell.com/esm/IDRAC6_FRMW_LX_R299265.BIN

Grab this and extract like this:

cd /tmp/dell
sh IDRAC6_FRMW_LX_R299265.BIN --extract ./idrac6-1.70.21

The firmware image is now in /tmp/dell/idrac6-1.70.21/payload/firmimg.d6

If you are just updating one machine, then the simplest way to perform the update is to use the Dell bmcfwul tool locally. This is supplied in the dell_ie_nitrogen package, and is installed to /usr/libexec/dell_dup/dell_ie_nitrogen/bmcfwul

Install the new firmware like this:

/usr/libexec/dell_dup/dell_ie_nitrogen/bmcfwul -input=/tmp/dell/idrac6-1.70.21/payload/firmimg.d6

If you have several machines to update, the most convenient way to perform the update is with tftp.

First, copy the firmware image to the tftp server, and put it in /tftproot, or wherever the root of your tftp server is located:

scp /tmp/dell/idrac6-1.70.21/payload/firmimg.d6 $ip_of_tftp_server:/tftproot

Then, trigger a firmware upgrade on the machines remotely using either racadm or ssh:

racadm -r host.to.update -u root -p calvin fwupdate -g -u -a $ip_of_tftp_server

or

ssh host.to.update racadm fwupdate -g -u -a $ip_of_tftp_server

We sometimes see problems updating our Dell machines to the latest firmware, ie. update_firmware -y fails:

Running updates...
-	Installing dell_dup_componentid_00159 - 1.4.7Installation failed for
package: dell_dup_componentid_00159 - 1.4.7
aborting update...

The error message from the low-level command was:

Could not parse output, bad xml for package: dell_dup_componentid_00159

Dell have been unable to tell me why this is, or provide a fix or workaround.

Here's what I did to get the firmware installed:

Continue reading

Dell distributes its OMSA software in RPM packages and even has a yum repo available so you'd think that updating to the next version would be as simple as yum update, right? Wrong!

You have to remove the old version first, and then install the new version. Oh, and you also need to stop the Dell services, restart ipmi, then restart the Dell services.

Something like this:

yum -y remove srvadmin-* \
  && rm -Rf /opt/dell \
  && yum -y install srvadmin-all dell_ft_install \
  && srvadmin-services.sh stop \
  && service ipmi restart \
  && srvadmin-services.sh start

We’ve had a bunch of new servers in place for around 3 months now. They seem to be working well and are performing just fine.

Then, out of the blue, our monitoring started throwing alerts on seemingly random servers. Our queues were building up – basically, database performance had dropped dramatically and our processing scripts couldn’t stuff data into the DBs fast enough.

What could be causing it?

Continue reading

I use cobbler to provision our new Dell servers, which is great but it needs the MAC addresses of the servers to identify each machine.

Previously, I have been doing this manually:

  1. log in to the DRAC web interface
  2. launch the java console
  3. rebooting the server
  4. go into the BIOS
  5. navigate to Embedded Devices
  6. manually record the MAC addresses

This takes quite a while, and is prone to error.

I recently had another 42 servers to deploy to I looked for a way to automate this process. I found one! Continue reading

It seems that several people have been having problems getting Dell OMSA 6.2 to work correctly on CentOS 5.4 x86_64. Specifically, the software does not detect any storage controllers, and therefore also doesn't find any disks. eg.

[root@b034 ~]# omreport storage pdisk controller=0
Invalid controller value. Read, controller=0
No controllers found.

After a little investigation, I found the source of the problem.

Continue reading

Having got racadm working on my workstation (see my previous post), the next step is to perform initial DRAC configuration, ie. change the root password, set the SSL cert values, etc.

First I checked that all DRACs were pingable:

for h in $(seq -w 1 34); do
    hn=b0$h.drac.example.com
    if ping -q -c 1 $hn >& /dev/null ; then
        echo OK
    else
        echo failed
    fi
done

Next, I created a drac config file (named drac.cfg) containing the settings that are common to all devices:

[cfgLanNetworking]
cfgDNSDomainName=drac.example.com
 
[cfgUserAdmin]
# cfgUserAdminIndex=2
cfgUserAdminUserName=root
cfgUserAdminPassword=secret
 
[cfgOobSnmp]
cfgOobSnmpAgentEnable=1
cfgOobSnmpAgentCommunity=my_community_name
 
[cfgRacSecurity]
cfgRacSecCsrKeySize=1024
# cfgRacSecCsrCommonName=
cfgRacSecCsrOrganizationName=example.com
cfgRacSecCsrOrganizationUnit=Web Services
cfgRacSecCsrLocalityName=My City
cfgRacSecCsrStateName=My State
cfgRacSecCsrCountryCode=IE
cfgRacSecCsrEmailAddr=contact@example.com

I then ran a script to apply the common configuration to all devices. I also set the device-specific settings in the same script:

for n in $(seq -w 1 34); do
    host=b0$hn
    domain=drac.example.com
    fullname=$host.$domain
    racadm -r $fullname -u root -p calvin config -g cfgLanNetworking -o cfgDNSRacName $host
    racadm -r $fullname -u root -p calvin config -g cfgRacSecurity -o cfgRacSecCsrCommonName $fullname
    racadm -r $fullname -u root -p calvin config -f drac.cfg
done

Notice that I don't change the default password until last.

Now, I just need to work out how to generate the CSR, sign it, and upload the new cert…

I wanted to install racadm on my local (non-Dell) workstation running Fedora 11. I tried installing via the repos as per the instructions, but without success. I eventually managed by downloading manually:

wget http://linux.dell.com/repo/hardware/OMSA_6.1/platform_independent/rh50_64/racadm/mgmtst-racadm-6.1.0-648.i386.rpm
yum localinstall mgmtst-racadm-6.1.0-648.i386.rpm

The next problem was that racadm wouldn't run – it failed with the error:

ERROR: Failed to initialize transport

Running under strace showed that it seemed to be having problems finding a suitable ssl library – probably because racadm is i686 and I'm running it on x86_64. Using yum provides I determined that a suitable ssl lib was shipped with Adobe Reader, installed in /opt/Adobe/ on my workstation.

The following couple of symlinks fixed it:

ln -s ../../opt/Adobe/Reader9/Reader/intellinux/lib/libssl.so.0.9.8 /lib/i686/
ln -s ../../opt/Adobe/Reader9/Reader/intellinux/lib/libcrypto.so.0.9.8 /lib/i686/