Today’s munin tip: Spooling a year of backlog to munin

A customer asked for a graph that we didn’t have. But we had the logs, so we said “sure, no worries”, and wrapped up some perl.

The problem with spooling a backlog to munin is missing values, and what to do with them. What was the value at the actual time? This case was about members on a mailing list, and the log looked like this:

1046900665 +manual handadded@somewhere.tld
1134930282 + wantstobemember@someotherplace.tld
1135008014 -probe deadalias@somewhere.tld
1135194225 - leavesthislist@someotherplace.tld
1105605482 +manual invalid@nope.tld?

In this case, the state is static when nothing happens in the log, so we just had to do something like this: 1) Get the state right now (just count the members in the list), 2) spool the log backwards, (using /usr/bin/tac) and fill the points in time with a counter running up or down, when a user has been added or removed from the list. If there is more than 300s between each entry, fill in the last value again.

Add a progress meter, and the code looks something like this. Optimization can of course be added, like in, not running rrdtool update several hundrer thousand times. This script finishes in some 5 minutes on my workstation.

#!/usr/bin/perl -w
use strict;
my $rrd="foo.rrd";
my $log="Log";
# Remove old rrd file and create a new one
unlink ($rrd); 
system (
	"rrdtool", "create",
	"-b", "1046900655", $rrd,
	"DS:42:GAUGE:600:0:U",
	"RRA:AVERAGE:0.5:1:576",
	"RRA:MIN:0.5:1:576",
	"RRA:MAX:0.5:1:576",
	"RRA:AVERAGE:0.5:6:432",
	"RRA:MIN:0.5:6:432",
	"RRA:MAX:0.5:6:432",
	"RRA:AVERAGE:0.5:24:540",
	"RRA:MIN:0.5:24:540",
	"RRA:MAX:0.5:24:540",
	"RRA:AVERAGE:0.5:288:450",
	"RRA:MIN:0.5:288:450",
	"RRA:MAX:0.5:288:450"
	);
# Take a snapshot of the state right now
my $t=time;
my $prevtime=$t;
my $r=`munin-run something | cut -d ' ' -f 2`; chomp $r;
my $then=$r;
my $prevval=$r;
# Some helper variables
my $this;
my $what;
my %dates;
my $i=0;
# Parse the log. Using /usr/bin/tac to read it from the bottom up, to
# get latest entries first
open (TAC,"/usr/bin/tac $log|") or
	die "Unable to open a pipe from tac, $!";
while () {
	next if /\?$/;
	if ( /^(\d+) ([\\+\\-])/ ) {
	    $this=$1;
	    $what=$2;
	    # Skip values older than some 13 months
	    if ( ($t-$this) > 34214400 ) { 
		last; 
	    }
	    else {
		$what eq "+" and $r--;
		$what eq "-" and $r++;
	    }
	}
	else { next; }
	# Fill out missing 5 minutes intervals
	if ( ($prevtime-$this) > 300 ) { 
	    for ( $i=$prevtime; $i>$this; $i=$i-300 ) {
			$dates{$i}=$prevval;
		}
	}
	$dates{$this}=$r;
	$prevval=$r;
	$prevtime=$this;
}
close TAC;
# Get a nice progress meter
my $all=$t-$this; my $tmp;
print "    Filling $rrd...";
# Fill the munin rrd graph
$t++; # one-off
for my $d (sort keys %dates) {
	system ("rrdtool", "update", $rrd, "-t", "42", "$d:$dates{$d}");
	$tmp=100*($d-$this)/$all; 
	print "\\r"; printf "%*d%%",2,$tmp;
}
system ("rrdtool", "update", $rrd, "-t", "42", "$t:$then");
print "Done\\n";
#EOF

One Response to “Today’s munin tip: Spooling a year of backlog to munin”

  1. ingvar says:

    Update: I now know that rrdtool update can take more than one argument at the time, so patching the script to support this may add some considerable speed.

Leave a Reply