Archive for July, 2010

Changing a process’ file descriptor on the fly

Saturday, July 10th, 2010

I come back to a specific problem every once in a while: Changing a program’s file descriptor while the same program is running.

From time to time, we do stupid things, like running a very important shell command, and adding debug output to see that it works, then realizing that it will take hours to finish, and spitting gigabytes of debug to an xterm through ssh does not help.

An other typical example could be a finding an ill managed system with some daemon without proper logfile handling. Restarting that process right now is just out of the question, copy-truncating that 16GB logfile will take too much time, and by the way, the disk is almost full.

After a hint from the ever helping kjetilho, and a Google search, I ended up with what seems to be a well-known trick, but I did not know about it till now. Welcome to the dark side of gdb.

With the power of gdb at your hand, you can hook into the inner parts of any running program, and change, well, virtually anything. Sounds insanely dangerous for systems in production, right? Yes I agree. Still, this hack is not that ugly. It does just what the doctor ordered: It changes a process’ fds for you while it’s running.

Google found a variant of this hack by Robert McKay, and I added a couple of lines for robustness. Here it is:

For a simple test, (yes, do test this before smashing your production environment to pieces) make small shell script like this

while true; do ((n++)); echo $n; sleep 1; done

Run that script in a separate window. Pick up the pid, and find out what vty is connected to that process’ standard output

ls -l /proc/$pid/fd/0

Then run the script, be really afraid of all that scary gdb output, but watch that number pumping magically stop.

sudo /dev/pts/$vty /dev/null $pid

Have a look at the fds again

ls -l /proc/$pid/fd

You can use strace to check that the process is still happily pumping numbers, but now to /dev/null

sudo strace -p $pid -e write