OpenNMS ships with the thresholds for some events already defined. For example, there is a memory threshold defined as:

<group name="netsnmp-memory-nonlinux" rrdRepository="/opt/opennms/share/rrd/snmp/">
        <expression type="low" expression="memAvailReal / memTotalReal * 100.0" ds-type="node" ds-label="" value="5.0" rearm="10.0" trigger="2"/>
</group>

ie. if free memory drops below 5% then an event will be created. The alert will be cancelled automatically if free memory subsequently rises above 10%

I wanted to configure some specific nodes with a different threshold, eg. generate an event when free memory drops below 2.5%.

Here's what I did.

Continue reading

I had a utility server running RHEL 6.2 (I installed it as part of a RHEV evaluation process). However, I have no RHEL entitlements so am not able to get updates.

So, I converted it to CentOS 6.2, with a little help from this post:

yum clean all
mkdir ~/centos
cd ~/centos
wget http://mirror.centos.org/centos/6.2/os/x86_64/RPM-GPG-KEY-CentOS-6
wget http://mirror.centos.org/centos/6.2/os/x86_64/Packages/centos-release-6-2.el6.centos.7.x86_64.rpm
wget http://mirror.centos.org/centos/6.2/os/x86_64/Packages/yum-3.2.29-22.el6.centos.noarch.rpm
wget http://mirror.centos.org/centos/6.2/os/x86_64/Packages/yum-utils-1.1.30-10.el6.noarch.rpm
wget http://mirror.centos.org/centos/6.2/os/x86_64/Packages/yum-plugin-fastestmirror-1.1.30-10.el6.noarch.rpm
rpm --import RPM-GPG-KEY-CentOS-6
rpm -e --nodeps redhat-release-server
rpm -e yum-rhn-plugin rhn-check rhnsd rhn-setup
rpm -Uhv --force *.rpm
yum upgrade
reboot

Nice!

When you install an update to a package on an RPM-based system, any configuration files which were originally distributed by the package and that have been changed are not replaced when the package is updated. Instead, rpm creates a new file with the extention ".rpmnew". It is then necessary to manually compare the existing file and the new one and decide how to deal with the new file.

So, the pattern I often need to use is "list all .rpmnew files and the corresponding original file". This is not a particularly complex issue, but is one that I don't use often enough to have at my fingertips. The trick is bash variable substitution; specifically pattern matching: ${variable%pattern}. This deletes the shortest possible match for pattern from the right of the contents of $variable.

An example using .rpmnew files created when upgading OpenNMS:

# ls -1 *.rpmnew| while read f ; do ls -l ${f%\.rpmnew}* ; done
-rw-rw-r-- 1 root root 7285 Jun  3 12:43 database-reports.xml
-rw-rw-r-- 1 root root 8232 Aug 10 19:01 database-reports.xml.rpmnew
-rw-rw-r-- 1 root root 333092 Aug 25 09:25 datacollection-config.xml
-rw-rw-r-- 1 root root 314141 May 11 01:31 datacollection-config.xml.rpmnew
-rw-rw-r-- 1 root root 3395 Jun  3 12:43 jasper-reports.xml
-rw-rw-r-- 1 root root 3664 Aug 10 19:01 jasper-reports.xml.rpmnew
-rw-rw-r-- 1 root root 25887 Aug 10 19:01 log4j.properties
-rw-rw-r-- 1 root root 25887 May 11 01:31 log4j.properties.rpmnew
-rw-rw-r-- 1 root root 19319 Apr 22 16:56 opennms.properties
-rw-rw-r-- 1 root root 19635 Aug 10 19:01 opennms.properties.rpmnew
-rw-rw-r-- 1 root root 9252 Apr 22 16:59 rrd-configuration.properties
-rw-rw-r-- 1 root root 9604 May 11 01:31 rrd-configuration.properties.rpmnew
-rw-r--r-- 1 root root 669444 Jun  3 13:51 snmp-graph.properties
-rw-rw-r-- 1 root root 604457 Aug 10 19:01 snmp-graph.properties.rpmnew

One of my clients uses Tripp Lite  PDUMH20AT power distribution units. They are rather primitive, and upgrading firmware on the management cards is done via ftp; you start an ftp session, upload the firmware image and the device reboots when you quit.

I recently ran into problems with the ftp sessions – I could connect OK, but any attempt to transfer files would fail.

It turns out the ftp server on the devices can't do passive ftp, only active. The fix is simple – issue the "passive" command before uploading the firmware image. Of course, this means it won't work very well through a firewall/NAT router, but that's not a problem as I do the upgrades from a local machine.

The full session looks something like this:

[root@a001 Files4step2]# ftp 8.pdu.a
Connected to 8.pdu.a (192.168.254.138).
220 NET+ARM FTP Server 1.0 ready.
Name (8.pdu.a:root): admin
331 User admin OK, send password.
Password:
230 Password OK.
Remote system type is NET+ARM.
ftp> bin
200 Type set to I.
ftp> passive
Passive mode off.
ftp> put rom.bin
local: rom.bin remote: rom.bin
200 PORT command Ok.
150 About to open data connection.
226 Transfer complete
2109474 bytes sent in 2.35 secs (8.8e+02 Kbytes/sec)
ftp> put pwralert.dat
local: pwralert.dat remote: pwralert.dat
200 PORT command Ok.
150 About to open data connection.
226 Transfer complete
679080 bytes sent in 0.793 secs (8.4e+02 Kbytes/sec)
ftp> bye
221 Goodbye.

Most of the servers I manage are 64-bit. I have one linode box that is 32-bit. I chose 32-bit because it has better memory usage than 64-bit, which is possibly important with a 512MB instance. This was probably a mistake as the management overhead involved with maintaining a 32-bit infrastructure for just one 32-bit machine is silly. No matter – we are where we are…!

I use the fnv_64 user-defined function from maatkit with MySQL. So, I need to build a 32-bit version for use on the 32-bit server.

Here's how to use mock to create a 32-bit build environment (in this case, for CentOS 5) on a 64-bit machine (the host is actually a Fedora 15 server).

Continue reading

Since my recent post about converting between unix time and date, it has come to my attention that Unix epochs are in seconds while Java epochs are in milliseconds. That means that you need to scale the epoch values by a factor of 1000 to convert between Unix and Java format.

So, date -u -d '2011-04-26 15:00:00' +%s gives 1303830000 (Unix epoch) which becomes 1303830000000 (Java epoch)

Converting the other way, the Java epoch 1306934061475 is 1306934061.475 in Unix format. date -u -d @1306934061.475 gives Wed Jun 1 13:14:21 UTC 2011 (assuming a UTC epoch).

Here are two ways to update the firmware on Dell iDRAC6 remote access cards.

Both methods require downloading the BIOS from Dell and extracting it from the bundle. For example, this is the 1.70.21 firmware:

mkdir /tmp/dell
cd /tmp/dell
wget http://ftp.dell.com/esm/IDRAC6_FRMW_LX_R299265.BIN

Grab this and extract like this:

cd /tmp/dell
sh IDRAC6_FRMW_LX_R299265.BIN --extract ./idrac6-1.70.21

The firmware image is now in /tmp/dell/idrac6-1.70.21/payload/firmimg.d6

If you are just updating one machine, then the simplest way to perform the update is to use the Dell bmcfwul tool locally. This is supplied in the dell_ie_nitrogen package, and is installed to /usr/libexec/dell_dup/dell_ie_nitrogen/bmcfwul

Install the new firmware like this:

/usr/libexec/dell_dup/dell_ie_nitrogen/bmcfwul -input=/tmp/dell/idrac6-1.70.21/payload/firmimg.d6

If you have several machines to update, the most convenient way to perform the update is with tftp.

First, copy the firmware image to the tftp server, and put it in /tftproot, or wherever the root of your tftp server is located:

scp /tmp/dell/idrac6-1.70.21/payload/firmimg.d6 $ip_of_tftp_server:/tftproot

Then, trigger a firmware upgrade on the machines remotely using either racadm or ssh:

racadm -r host.to.update -u root -p calvin fwupdate -g -u -a $ip_of_tftp_server

or

ssh host.to.update racadm fwupdate -g -u -a $ip_of_tftp_server