I come back to a specific problem every once in a while: Changing a program’s file descriptor while the same program is running.
From time to time, we do stupid things, like running a very important shell command, and adding debug output to see that it works, then realizing that it will take hours to finish, and spitting gigabytes of debug to an xterm through ssh does not help.
An other typical example could be a finding an ill managed system with some daemon without proper logfile handling. Restarting that process right now is just out of the question, copy-truncating that 16GB logfile will take too much time, and by the way, the disk is almost full.
After a hint from the ever helping kjetilho, and a Google search, I ended up with what seems to be a well-known trick, but I did not know about it till now. Welcome to the dark side of gdb.
With the power of gdb at your hand, you can hook into the inner parts of any running program, and change, well, virtually anything. Sounds insanely dangerous for systems in production, right? Yes I agree. Still, this hack is not that ugly. It does just what the doctor ordered: It changes a process’ fds for you while it’s running.
For a simple test, (yes, do test this before smashing your production environment to pieces) make small shell script like this
#!/bin/bash n=0 while true; do ((n++)); echo $n; sleep 1; done
Run that script in a separate window. Pick up the pid, and find out what vty is connected to that process’ standard output
ls -l /proc/$pid/fd/0
Then run the script, be really afraid of all that scary gdb output, but watch that number pumping magically stop.
sudo fdswap.sh /dev/pts/$vty /dev/null $pid
Have a look at the fds again
ls -l /proc/$pid/fd
You can use strace to check that the process is still happily pumping numbers, but now to /dev/null
sudo strace -p $pid -e write