We currently deploy our app code to around 50 nodes using capistrano. We use the "copy" deployment method, ie. the code is checked out of svn onto the local deployment node, rolled into a tarball, then copied out to each target node where it is unrolled into a release dir before the final symlink is put in place.

As you might imagine, copying to 50 nodes generates quite a bit of traffic, and it takes ~5 mins to do a full deploy.

I was reading this interesting link today; one bullet in particular jumped out at me:

  • "… the few hundred MB binary gets rapidly pushed bia [sic] bit torrent."

Now that's an interesting idea – I wonder if I can knock up something in capistrano that deploys using bittorrent?

Dell distributes its OMSA software in RPM packages and even has a yum repo available so you'd think that updating to the next version would be as simple as yum update, right? Wrong!

You have to remove the old version first, and then install the new version. Oh, and you also need to stop the Dell services, restart ipmi, then restart the Dell services.

Something like this:

yum -y remove srvadmin-* \
  && rm -Rf /opt/dell \
  && yum -y install srvadmin-all dell_ft_install \
  && srvadmin-services.sh stop \
  && service ipmi restart \
  && srvadmin-services.sh start

We’ve had a bunch of new servers in place for around 3 months now. They seem to be working well and are performing just fine.

Then, out of the blue, our monitoring started throwing alerts on seemingly random servers. Our queues were building up – basically, database performance had dropped dramatically and our processing scripts couldn’t stuff data into the DBs fast enough.

What could be causing it?

Continue reading

I use cobbler to provision our new Dell servers, which is great but it needs the MAC addresses of the servers to identify each machine.

Previously, I have been doing this manually:

  1. log in to the DRAC web interface
  2. launch the java console
  3. rebooting the server
  4. go into the BIOS
  5. navigate to Embedded Devices
  6. manually record the MAC addresses

This takes quite a while, and is prone to error.

I recently had another 42 servers to deploy to I looked for a way to automate this process. I found one! Continue reading

I use the toggl time-tracking service to keep track of the hours I work for my various clients.

toggl make available desktop clients for Windows, Mac, & Linux, but the Linux packages are in .deb format for Ubuntu and, until recently, they did not provide x86_64 packages.

toggl recently released the desktop client as open source so I grabbed it and have built an RPM.

SRPM: TogglDesktop-2.5.1-1.fc12.src.rpm

RPM (Fedora 12, x86_64): TogglDesktop-2.5.1-1.fc12.x86_64.rpm

It seems that several people have been having problems getting Dell OMSA 6.2 to work correctly on CentOS 5.4 x86_64. Specifically, the software does not detect any storage controllers, and therefore also doesn't find any disks. eg.

[root@b034 ~]# omreport storage pdisk controller=0
Invalid controller value. Read, controller=0
No controllers found.

After a little investigation, I found the source of the problem.

Continue reading