Archive for the ‘munin’ Category

munin-node running on the iPhone

Friday, September 24th, 2010

Since it’s Friday, here’s another completely waste of time experiment: Would it be possible to get munin-node running on the iPhone. As Perl exists for the jailbroken iPhone, the answer is obvious: Of Course!

I used my good old jailbroken first generation iPhone “2G”, and started by adding the Perl for iPhone apt repo, and then installed the necessary packages for a basic munin node, that is make, p5-Net-Server, p5-Pod-Compiler and p5-Pod-Simple. In the process, I moved /usr/local to /var/usr_local, and /opt to /var/opt, and symlinked them back, to save precious place on the root filesystem.

To get make things working without warnings, I added a line

                   iphoneos  Unix

to /usr/local/lib/perl5/5.10.0/Module/Build.pm, around line 36, together with darwin (and reported this as an upstream bug).

Then I downloaded the latest munin stable release, unpacked it, and looked over the INSTALL file for a node-only installation from source. The setup was very easy. I just configured munin to run as the “mobile” user and group in Makefile.config, and ran

make
make install-common-prime install-node-prime install-plugins-prime
/opt/munin/sbin/munin-node-configure --shell --families=contrib,auto | sh

Finally I added my local munin master to /etc/opt/munin/munin-node.conf, made sure it ran as user and group “mobile”, and started munin with

/opt/munin/sbin/munin-node

That’s it. No magic at all. Thank you, Perl! If you try this, remember to kill munin-node afterwards, or your battery won’t last.

Here’s a test run

RPM packages of munin-1.4.4 for RHEL and derivates available

Thursday, March 4th, 2010

Kevin Fenzi has uploaded packages of munin-1.4.4 to Fedora Rawhide, and I have as usual recompiled the packages against EPEL 4 and 5.

The packages are available at http://users.linpro.no/ingvar/munin/1.4/.

rpm packages of munin-1.4

Friday, December 4th, 2009

Munin is a auto-configuring, multi-platform, and generally awsome monitoring and graphing framework.

Thanks to the munin team‘s awsome work, munin-1.4 was finally released ultimo november. A bugfix update 1.4.1 was released today. Great thanks also to Kevin Fenzi who maintains munin for the Fedora project. He has built a matching specfile for Fedora and EPEL. I used it on the 1.4.1 tarball, and built rpm packages for RHEL5, available at http://users.linpro.no/ingvar/munin/1.4/

Please report bugs in munin to the munin upstream project. rpm package bugs can be reported to the RedHat Bugzilla

Update: Uploaded packages for el4 and el5 to http://sourceforge.net/projects/munin/files/
Update: Packages of 1.4.2 and 1.4.3 available (not uploaded to sourceforge yet).

Today’s munin tip: Collecting similar graphs

Wednesday, March 25th, 2009

At a weekend, usually at saturday night, just after the kids are in bed, or you have arrived at the local pub and got a pint served, the monitoring system pages you, and tells you that the disk of one of your database partitions is filled to the rim, and that database will crash within minutes unless you fix it.

Now, since you have Munin installed, this could of course be avoided, by watching the graphs, and even looking into the future. But since you are responsible for a lot of other things, browsing those graphs may be tedious.

I threw together a small perl script that may help, at least a bit. It won’t read the graphs for you, but it collects them in a central page for quick browsing.

Just run it like this (or put it in cron)

gen-graph-collection.pl df

df is just an example. You might run it on any munin plugin you have graphs for. You will get a page of colletions at http://your.host.tld/path/to/munin/collection.html Read that page every thursday, and you might drink that pint in peace.

If the page doesn’t fit to your screen, buy a bigger one, or switch to Firefox, and use CTRL+mouse-scroll-up/down to zoom.

Yeah, and that script has probably lots of ugly bugs. Use with care.

Today’s munin tip: Reading the future

Thursday, November 13th, 2008

Last time: Adding info from the past. This time: Looking into the future. As Munin is one of the ravens of Odin, one of the Norse gods, this script is called Skuld, the future Norne, one of the three female destiny deities from Norse mythology

Using rrdtool, perl and some simple statistical analysis, the linear least squares method, we try to predict when a filesystem reaches 100%. Very useful for system administrators.

Script can be downloaded from http://users.linpro.no/ingvar/munin/skuld.pl.txt

UPDATE: An even more flashy version, with optional --time and --value available at above URL.

#!/usr/bin/perl
#
# skuld - a perl script looking into the future
#
# This software is covered by the GNU General Public License version 2 or later.
# See http://www.gnu.org/copyleft/gpl.html for details.
#

use strict;
use Statistics::Descriptive;

unless (defined $ARGV[0]) {
    print "usage: skuld some_munin_df_file.rrd\n";
    exit 1;
}
my $debug=0;

my $rrdtool="/usr/bin/rrdtool";
my $rrd=$ARGV[0];
my $type="";
my $name="";

my @dayx  = (); my $dayy  = Statistics::Descriptive::Full->new();
my @weekx = (); my $weeky = Statistics::Descriptive::Full->new();
my @monthx= (); my $monthy= Statistics::Descriptive::Full->new();
my @yearx = (); my $yeary = Statistics::Descriptive::Full->new();

open (DUMP,"LANG=C $rrdtool dump $rrd |") 
    or die "Unable to open a pipe from $rrdtool dump, $!";

while (<DUMP>) 
{
    # Search for correct name
    if ( /\<ds\>/ ) {
	while ( <DUMP> ) {
	    /\<name\> (\w+) \<\/name\>/ and $name=$1;
	    last if /\<\/ds\>/
	}
	unless ( $name eq "42" ) {
	    print "No munin rrd file\n";
	    exit 2;
	}
    }

    if ( /\<rra\>/ ) {
	while ( <DUMP> ) {
	    # Only parse MAX values
	    if ( /\<cf\> MAX \<\/cf\>/ ) {
		my $in=<DUMP>;
		if ( $in =~ /\<pdp_per_row\> (\d+) \<\/pdp_per_row\>/ ) {
		    my $pdp=$1;
		    if    ( $pdp ==   1 ) { $type = "d"; print "day\n"   if $debug; }
		    elsif ( $pdp ==   6 ) { $type = "w"; print "week\n"  if $debug; }
		    elsif ( $pdp ==  24 ) { $type = "m"; print "month\n" if $debug; }
		    elsif ( $pdp == 288 ) { $type = "y"; print "year\n"  if $debug; }
		} 
		else { $type=""; }	    
	    }
	    if ( /\<database\>/ and ! $type eq "" ) {
		while (<DUMP>) {
		    if ( /\<\!\-\-.+ \/ (\d+) \-\-\> \<row\>\<v\> (\d+\.\d+e.\d+) \<\/v\>\<\/row>$/ ) {
			print "type: $type, time=$1, val=$2\n" if $debug;

			if ( $type eq "d" ) { push @dayx,  ($1); $dayy->add_data($2)   };
			if ( $type eq "w" ) { push @weekx, ($1); $weeky->add_data($2)  };
			if ( $type eq "m" ) { push @monthx,($1); $monthy->add_data($2) };
			if ( $type eq "y" ) { push @yearx, ($1); $yeary->add_data($2)  };
		    }
		    last if /\<\/database\>/;				   
		}
	    }
	    last if /\<\/rra\>/;
	}
    }
}

my ($dq, $dm, $dr, $drms)=$dayy->least_squares_fit(@dayx);
my ($wq, $wm, $wr, $wrms)=$weeky->least_squares_fit(@weekx);
my ($mq, $mm, $mr, $mrms)=$monthy->least_squares_fit(@monthx);
my ($yq, $ym, $yr, $yrms)=$yeary->least_squares_fit(@yearx);

print "Daily forecast: Passes 100% at ";
print localtime((100-$dq)/$dm) . "\n";

print "Weekly forecast: Passes 100% at ";
print localtime((100-$wq)/$wm) . "\n";

print "Monthly forecast: Passes 100% at ";
print localtime((100-$mq)/$mm) . "\n";

print "Yearly forecast: Passes 100% at ";
print localtime((100-$yq)/$ym) . "\n";

Today’s munin tip: Spooling a year of backlog to munin

Tuesday, October 21st, 2008

A customer asked for a graph that we didn’t have. But we had the logs, so we said “sure, no worries”, and wrapped up some perl.

The problem with spooling a backlog to munin is missing values, and what to do with them. What was the value at the actual time? This case was about members on a mailing list, and the log looked like this:

1046900665 +manual handadded@somewhere.tld
1134930282 + wantstobemember@someotherplace.tld
1135008014 -probe deadalias@somewhere.tld
1135194225 - leavesthislist@someotherplace.tld
1105605482 +manual invalid@nope.tld?

In this case, the state is static when nothing happens in the log, so we just had to do something like this: 1) Get the state right now (just count the members in the list), 2) spool the log backwards, (using /usr/bin/tac) and fill the points in time with a counter running up or down, when a user has been added or removed from the list. If there is more than 300s between each entry, fill in the last value again.

Add a progress meter, and the code looks something like this. Optimization can of course be added, like in, not running rrdtool update several hundrer thousand times. This script finishes in some 5 minutes on my workstation.

#!/usr/bin/perl -w
use strict;
my $rrd="foo.rrd";
my $log="Log";
# Remove old rrd file and create a new one
unlink ($rrd); 
system (
	"rrdtool", "create",
	"-b", "1046900655", $rrd,
	"DS:42:GAUGE:600:0:U",
	"RRA:AVERAGE:0.5:1:576",
	"RRA:MIN:0.5:1:576",
	"RRA:MAX:0.5:1:576",
	"RRA:AVERAGE:0.5:6:432",
	"RRA:MIN:0.5:6:432",
	"RRA:MAX:0.5:6:432",
	"RRA:AVERAGE:0.5:24:540",
	"RRA:MIN:0.5:24:540",
	"RRA:MAX:0.5:24:540",
	"RRA:AVERAGE:0.5:288:450",
	"RRA:MIN:0.5:288:450",
	"RRA:MAX:0.5:288:450"
	);
# Take a snapshot of the state right now
my $t=time;
my $prevtime=$t;
my $r=`munin-run something | cut -d ' ' -f 2`; chomp $r;
my $then=$r;
my $prevval=$r;
# Some helper variables
my $this;
my $what;
my %dates;
my $i=0;
# Parse the log. Using /usr/bin/tac to read it from the bottom up, to
# get latest entries first
open (TAC,"/usr/bin/tac $log|") or
	die "Unable to open a pipe from tac, $!";
while () {
	next if /\?$/;
	if ( /^(\d+) ([\\+\\-])/ ) {
	    $this=$1;
	    $what=$2;
	    # Skip values older than some 13 months
	    if ( ($t-$this) > 34214400 ) { 
		last; 
	    }
	    else {
		$what eq "+" and $r--;
		$what eq "-" and $r++;
	    }
	}
	else { next; }
	# Fill out missing 5 minutes intervals
	if ( ($prevtime-$this) > 300 ) { 
	    for ( $i=$prevtime; $i>$this; $i=$i-300 ) {
			$dates{$i}=$prevval;
		}
	}
	$dates{$this}=$r;
	$prevval=$r;
	$prevtime=$this;
}
close TAC;
# Get a nice progress meter
my $all=$t-$this; my $tmp;
print "    Filling $rrd...";
# Fill the munin rrd graph
$t++; # one-off
for my $d (sort keys %dates) {
	system ("rrdtool", "update", $rrd, "-t", "42", "$d:$dates{$d}");
	$tmp=100*($d-$this)/$all; 
	print "\\r"; printf "%*d%%",2,$tmp;
}
system ("rrdtool", "update", $rrd, "-t", "42", "$t:$then");
print "Done\\n";
#EOF

Today’s Munin tip: Splitting out a noisy graph from munin

Monday, April 7th, 2008

I have more than once found a Munin multi graph set being rendered unreadable by single graph peaking 100 times over the rest of the graphs in the set. The classical example is the Linux (over)committed memory graph on a system leaking memory, peaking in the 100GB area, making the other system memory graphs that are playing along about 1-4GB almost invisible:

Graph 1

One could of course put a hard limit in the RRD file, in my example, at the pysical memory in the machine in question, but this means that I loose oversight by hiding what’s probably a problem. A better solution is to split the graph out, that is, prevent it from graphing in the common memory set, and put it in custom graph. Consider the following munin.conf snipplet:


[my.server.tld]
address my.server.tld
memory.committed.graph no

memory_committed.graph_title Committed Memory
memory_committed.update no
memory_committed.graph_category system
memory_committed.graph_order \
memory=my.server.tld:memory.committed

This produces the following graphs:

Graph 2 Graph 3

Problem solved.