On the third side

Among other hardware in our SAN, we have one of these nice and small, entry level SAN boxes from IBM. It’s called DS3400. It blends well into the rest of the SAN fabric, and seems to work quite well. So far, so good.

Today, I had to extend one of the volumes hosted on that box. So I started up the expected tool, the “IBM System Storage DS Storage Manager (Enterprise Management)” *pant*, found the box, the correct array, and volume (“Logical drive”), and expected to just right click and add capacity.

Not so.

Okay, so I looked at the “Modify” menu, which would let me “Add free capacity”, but in terms of disks to an array, not free space to a volume. After a bit of searching, I concluded that this fancy GUI does actually not let me grow a volume. How “Enterprise” is that?

After some extensive googling, I found that this was certainly right, it’s not possible to do this by the GUI. There is a CLI tool, available from the shell, or as a script editor in the GUI, and I should be able to grow a volume from that. Now, all I had to do was to find the correct syntax. I searched through the local help pages. There were no exciting hits for “expanding”, “extending” or “grow”. There were no matching entries in the FAQ. The script editor had a command reference, but without already knowing the correct command, it was useless. After even a bit more googling, I got the correct command line. It’s something like this, note the space and the semicolon at the end.


set logicaldrive ["MyVolumeName"] addCapacity=100 GB;


Or even from the shell:

# SMcli 10.11.12.13 -n our_nice_ibm_box -p secret_password \
  -c 'set logicaldrive ["MyVolumeName"] addCapacity=100 GB;'

Trivial when you know it, right? Why on Earth did they not implement a function in the GUI to support this?

For fun, I searched through the online manuals, and finally found a small note at page 96 in “IBM System Storage DS Storage Manager 10, Installation and Support Guide” mentioning the command among best practices using the cli.

No bonus points to IBM. It’s almost as one would think they did it on purpose.

Later today, I found another blog post, telling more or less the same as this one, and with a few noteworthy comments, like how to get a growing progress status – which of course is not visible in the GUI.

Edit: Memo to self:

sudo SMcli 10.11.12.13 -n ibm-box -p secretPassword \
-c 'show logicalDrive ["DataVolumeName"] actionProgress ;'

RHEL4, while being old, is still widely in use. From time to time, on such old systems, the rpm database tends to break. It is just like that. The quick fix is

pkill -9 up2date
pkill -9 rpmq
pkill -9 rpm
rm /var/lib/rpm/__db.*
rpm --rebuilddb

Now, there are times when this is not enough. The package database is completely broken, and rebuilddb fails or leaves you with a broken database. Unless you can restore /var/lib/rpm from backup, the database is gone, and maintaining that system becomes … let’s just say … difficult. Now, if you are lucky, you discover this before /var/log/rpmpkgs.* are all rotated out. In the following, I assume that /var/log/rpmpkgs.1 is intact. If you have such a file, this approach may help:

First move the old database away and initialize a new one

mv /var/lib/rpm /var/lib/rpm.broken
rpm --initdb

Then manually download the correct redhat-release package from RHN, install it, and import the GPG key

rpm -ivh --nodeps redhat-release-4ES-9.i386.rpm
rpm --import /usr/share/doc/redhat-release-4ES/RPM-GPG-KEY

Now check if you are able to install a package

up2date --justdb --install bash

If your up2date cache is broken (may happen from time to time), scratch it

rm -rf /var/spool/up2date
mkdir /var/spool/up2date

If you had to clear out the cache, the next run of up2date must repopulate that cache, and may take a long time. I’ve seen systems using 15-20 minutes on this.

up2date --justdb --install bash

Now force install metadata for the kernel package and any other kernel related packages, as those are usually prohibited in the up2date config from being automatically dragged in by dependencies.

sed 's/\(.*\)-.*-.*/\1/' < /var/log/rpmpkgs.1 | grep ^kernel | sort -u | xargs up2date --justdb --install --force

If that went well, update the system with metadata from the repo, using the latest working rpmpkgs log file. Depending on your repos, network connection and system hardware, this may take a long time. Be patient.

sed 's/\(.*\)-.*-.*/\1/' < /var/log/rpmpkgs.1 | sort -u | xargs up2date --install --justdb

The scenario above fetched all packages from the active repos in /etc/sysconfig/rhn/sources. If the system has been broken over time, the packages in the repo may be newer than the content you have got on disk. That may lead to confusion, since you only updated the database with the latest versions, and not the actual package content. So finally, you must find and fix everything that went wrong

diff -u <(sort /var/log/rpmpkgs.1) <(rpm -qa --qf "%{name}-%{version}-%{release}.%{arch}.rpm\n" | sort)

If some newly installed packages are newer, you have to reinstall (--replacepkgs) them with content (without --justdb). Packages can be downloaded and reinstalled with

cd /var/spool/up2date
up2date --get packagename
rpm -Uvh --replacepkgs packagename-ver-rel.arch.rpm