On the third side

RHEL4, while being old, is still widely in use. From time to time, on such old systems, the rpm database tends to break. It is just like that. The quick fix is

pkill -9 up2date
pkill -9 rpmq
pkill -9 rpm
rm /var/lib/rpm/__db.*
rpm --rebuilddb

Now, there are times when this is not enough. The package database is completely broken, and rebuilddb fails or leaves you with a broken database. Unless you can restore /var/lib/rpm from backup, the database is gone, and maintaining that system becomes … let’s just say … difficult. Now, if you are lucky, you discover this before /var/log/rpmpkgs.* are all rotated out. In the following, I assume that /var/log/rpmpkgs.1 is intact. If you have such a file, this approach may help:

First move the old database away and initialize a new one

mv /var/lib/rpm /var/lib/rpm.broken
rpm --initdb

Then manually download the correct redhat-release package from RHN, install it, and import the GPG key

rpm -ivh --nodeps redhat-release-4ES-9.i386.rpm
rpm --import /usr/share/doc/redhat-release-4ES/RPM-GPG-KEY

Now check if you are able to install a package

up2date --justdb --install bash

If your up2date cache is broken (may happen from time to time), scratch it

rm -rf /var/spool/up2date
mkdir /var/spool/up2date

If you had to clear out the cache, the next run of up2date must repopulate that cache, and may take a long time. I’ve seen systems using 15-20 minutes on this.

up2date --justdb --install bash

Now force install metadata for the kernel package and any other kernel related packages, as those are usually prohibited in the up2date config from being automatically dragged in by dependencies.

sed 's/\(.*\)-.*-.*/\1/' < /var/log/rpmpkgs.1 | grep ^kernel | sort -u | xargs up2date --justdb --install --force

If that went well, update the system with metadata from the repo, using the latest working rpmpkgs log file. Depending on your repos, network connection and system hardware, this may take a long time. Be patient.

sed 's/\(.*\)-.*-.*/\1/' < /var/log/rpmpkgs.1 | sort -u | xargs up2date --install --justdb

The scenario above fetched all packages from the active repos in /etc/sysconfig/rhn/sources. If the system has been broken over time, the packages in the repo may be newer than the content you have got on disk. That may lead to confusion, since you only updated the database with the latest versions, and not the actual package content. So finally, you must find and fix everything that went wrong

diff -u <(sort /var/log/rpmpkgs.1) <(rpm -qa --qf "%{name}-%{version}-%{release}.%{arch}.rpm\n" | sort)

If some newly installed packages are newer, you have to reinstall (--replacepkgs) them with content (without --justdb). Packages can be downloaded and reinstalled with

cd /var/spool/up2date
up2date --get packagename
rpm -Uvh --replacepkgs packagename-ver-rel.arch.rpm


This entry was posted on Tuesday, October 5th, 2010 at 14:23 and is filed under comp, sysadmin. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.