Tag-Archive for » munin «

Thursday, May 20th, 2010 | Author:

eth0Readers of this little blog may know I’ve spent some time to have a munin setup that was tweaked to be optimized. But I’ve reached a point where my knowledge did not suffice to balance the design problems of munin. This is not a rant, I just say that when your infrastructure reaches a certain size munin reaches its limits. The pull based model, the graph generation (don’t tell me about the CGI graph, this thing never worked as expected) overloaded my management box. Talking with other peoples brought collectd to my attention so I gave it a try.

Collectd has many nice features : 10 seconds precision (versus 5 minutes), written in C for performance, multicast support (even if I don’t use it), and you can even create collectd relays. There are some packages for many different targets, even for my OpenWRT based access points ! Configuration can easily be automated (flat files, superior), a mandatory point to me.

Of course collectd comes without an UI but that’s no big deal : there are many around. As I said in the previous post, I use visage.

For the load generated a picture says a thousand words (click to enlarge) :

goodbyemunin

Next step is working on the IO congestion caused by so many RRD updates, collectd wiki has many tips about this, this will probably be fixed in a couple of hours.

Monday, February 22nd, 2010 | Author:

Absolut_nginxAt $WORK I started using Nginx a while ago, first as a front end to my mongrel instances for puppet. Recently I began to use it for one of its most know features : reverse proxy (and caching too). Of course this work had to be puppetized !

This is a summary of what I’ve done :

  • Basic setup
  • Automatic setup of the status page, exploited by a munin plugin
  • An “include” directory, can be specific to a host through the usual $fqdn source selection system (as well as the nginx.conf file).
  • A “reverse proxy” specific class that uses a template embedding some ruby (see the previous post). My cache dir is under tmpfs, to speed up the whole thing.

This setup is mostly inspired by this post. I use a local dnsmasq setup to resolve both internal & external requests. This way I can manage vhosts being accessible from inside ou outside our network. It’s incredibly flexible and allows you to get the most from your infrastructure.

The puppet class :

# @name : nginx
# @desc : classe de base pour nginx
# @info : nil
class nginx
{
 package { "nginx":
 ensure => installed
 }
 
 service { "nginx":
 ensure => running
 }
 
 file { "nginx.conf":
 name => "/etc/nginx/nginx.conf",
 owner => root,
 group => root,
 source => [ "puppet://$fileserver/files/apps/nginx/$fqdn/nginx-rp-secure.conf", "puppet://$fileserver/files/apps/nginx/nginx-rp-secure.conf"],
 ensure => present,
 notify => Service["nginx"]
 }
 
 # status is installed on all nginx boxens
 file { "nginx-status":
 name => "/etc/nginx/sites-enabled/nginx-status",
 owner => root,
 group => root,
 source => [ "puppet://$fileserver/files/apps/nginx/nginx-status", "puppet://$fileserver/files/apps/nginx/$fqdn/nginx-status"],
 ensure => present,
 notify => Service["nginx"]
 }
 
 # include dir, get the freshness here
 file { "include_dir":
 name => "/etc/nginx/includes",
 owner => root,
 group => root,
 source => [ "puppet://$fileserver/files/apps/nginx/includes.$fqdn", "puppet://$fileserver/files/apps/nginx/includes"],
 ensure => directory,
 recurse => true,
 notify => Service["nginx"],
 ignore => ".svn*"
 }
 
 # files managed by hand, no matter if it breaks
 file { "sites-managed":
 name => "/etc/nginx/sites-managed",
 owner => root,
 group => root,
 ensure => directory
 }
}
 
# @name : nginx::reverseproxy
# @desc : config nginx pour reverse proxy
# @info : utilisée en conjonction avec dnsmasq local
class nginx::reverseproxy
{
 include nginx
 include dnsmasq::reverseproxy
 
 # Vars used by the template below
 $mysqldatabase=extlookup("mysqldatabase")
 $mysqllogin=extlookup("mysqllogin")
 $mysqlpassword=extlookup("mysqlpassword")
 $mysqlserver=extlookup("mysqlserver")
 
 file { "nginx-cachedir":
 name => "/dev/shm/nginx-cache",
 owner => www-data,
 group => www-data,
 ensure => directory
 }
 
 file { "site_reverse-proxy":
 name => "/etc/nginx/sites-enabled/reverse-proxy",
 owner => root,
 group => root,
 content => template("nginx/$fqdn/reverse-proxy.erb"),
 ensure => present,
 notify => Service["nginx"],
 require => File["nginx-cachedir"]
 }
 
}

This is the munin plugins that are automatically distributed with the box.

One of the generated graphs :

nginx_requests-day

Category: BOFH Life, Puppet, SysAdmin  | Tags: , , ,  | One Comment
Wednesday, August 05th, 2009 | Author:

Disclaimer : this work is mostly based upon DavidS work, available on his git repo. In the scope of my work I needed to have munin support for freeBSD & Solaris. I also wrote a class for snmp_plugins & custom plugins. Some things are quite dependant from my infrastructure, like munin.conf generation script but it can easily be adapted to yours, by extracting data from your CMDB.

It requires the munin_interfaces fact published here (and merged into DavidS repo, thanks to him), and Volcane’s extlookup function to store some parameters. Enough talking, this is the code :

# Munin config class
# Many parts taken from David Schmitt's http://git.black.co.at/
# FreeBSD & Solaris + SNMP & custom plugins support by Nicolas Szalay <nico@gcu.info>
 
class munin::node {
	case $operatingsystem {
		openbsd: {}
		debian: { include munin::node::debian}
		freebsd: { include munin::node::freebsd}
		solaris: { include munin::node::solaris}
		default: {}
	}
}
 
class munin::node::debian {
 
	package { "munin-node": ensure => installed }
 
	file { 
	"/etc/munin":
		ensure => directory,
		mode => 0755,
		owner => root,
		group => root;
 
	"/etc/munin/munin-node.conf":
		source => "puppet://$fileserver/files/apps/munin/munin-node-debian.conf",
		owner => root,
		group => root,
		mode => 0644,
		before => Package["munin-node"],
		notify => Service["munin-node"],
	}
 
	service { "munin-node": ensure => running }
 
	include munin::plugins::linux 
}
 
class munin::node::freebsd {
	package { "munin-node": ensure => installed, provider => freebsd }
 
        file { "/usr/local/etc/munin/munin-node.conf":
                source => "puppet://$fileserver/files/apps/munin/munin-node-freebsd.conf",
                owner => root,
                group => wheel,
                mode => 0644,
                before => Package["munin-node"],
                notify => Service["munin-node"],
        }
 
	service { "munin-node": ensure => running }
 
	include munin::plugins::freebsd
}
 
class munin::node::solaris {
	# "hand made" install, no package.
	file { "/etc/munin/munin-node.conf":
		source => "puppet://$fileserver/files/apps/munin/munin-node-solaris.conf",
                owner => root,
                group => root,
                mode => 0644
	}
 
	include munin::plugins::solaris
}
 
class munin::gatherer {
	package { "munin":
		ensure => installed
	}
 
	# custom version of munin-graph : forks & generates many graphs in parallel
	file { "/usr/share/munin/munin-graph":
		owner => root,
		group => root,
		mode => 0755,
		source => "puppet://$fileserver/files/apps/munin/gatherer/munin-graph",
		require => Package["munin"]
	}
 
	# custon version of debian cron file. Month & Year cron are generated once daily
	file { "/etc/cron.d/munin":
		owner => root,
		group => root,
		mode => 0644,
		source => "puppet://$fileserver/files/apps/munin/gatherer/munin.cron",
		require => Package["munin"]
	}
 
	# Ensure cron is running, to fetch every 5 minutes
	service { "cron":
		ensure => running
	}
 
	# Ruby DBI for mysql
	package { "libdbd-mysql-ruby":
		ensure => installed
	}
 
	# config generator
	file { "/opt/scripts/muningen.rb":
		owner => root,
		group => root,
		mode => 0755,
		source => "puppet://$fileserver/files/apps/munin/gatherer/muningen.rb",
		require => Package["munin", "libdbd-mysql-ruby"]
	}	
 
	# regenerate munin's gatherer config every hour
	cron { "munin_config":
		command => "/opt/scripts/muningen.rb > /etc/munin/munin.conf",
		user => "root",
		minute => "0",
		require => File["/opt/scripts/muningen.rb"]
	}
 
	include munin::plugins::snmp
	include munin::plugins::linux
	include munin::plugins::custom::gatherer
}
 
 
# define to create a munin plugin inside the right directory
define munin::plugin ($ensure = "present") {
 
	case $operatingsystem {
		freebsd: { 
			$script_path = "/usr/local/share/munin/plugins"
			$plugins_dir = "/usr/local/etc/munin/plugins"
		}
		debian: { 
			$script_path = "/usr/share/munin/plugins"
			$plugins_dir = "/etc/munin/plugins"
		}
		solaris: { 
			$script_path = "/usr/local/munin/lib/plugins"
			$plugins_dir = "/etc/munin/plugins"
		}
		default: { }
	}
 
	$plugin = "$plugins_dir/$name"
 
	case $ensure {
		"absent": {
			debug ( "munin_plugin: suppressing $plugin" )
			file { $plugin: ensure => absent, } 
		}
 
		default: {
			$plugin_src = $ensure ? { "present" => $name, default => $ensure }
 
			file { $plugin:
				ensure => "$script_path/${plugin_src}",
				require => Package["munin-node"],
				notify => Service["munin-node"],
			}
		}
	}
}
 
# snmp plugin define, almost same as above
define munin::snmp_plugin ($ensure = "present") {
	$pluginname = get_plugin_name($name)
 
	case $operatingsystem {
		freebsd: { 
			$script_path = "/usr/local/share/munin/plugins"
			$plugins_dir = "/usr/local/etc/munin/plugins"
		}
		debian: { 
			$script_path = "/usr/share/munin/plugins"
			$plugins_dir = "/etc/munin/plugins"
		}
		solaris: { 
			$script_path = "/usr/local/munin/lib/plugins"
			$plugins_dir = "/etc/munin/plugins"
		}
		default: { }
	}
 
	$plugin = "$plugins_dir/$name"
 
	case $ensure {
		"absent": {
			debug ( "munin_plugin: suppressing $plugin" )
			file { $plugin: ensure => absent, } 
		}
 
		"present": {
			file { $plugin:
				ensure => "$script_path/${pluginname}",
				require => Package["munin-node"],
				notify => Service["munin-node"],
			}
		}
	}
}
 
class munin::plugins::base
{
	case $operatingsystem {
		debian: { $plugins_dir = "/etc/munin/plugins" }
		freebsd: { $plugins_dir = "/usr/local/etc/munin/plugins" }
		solaris: { $plugins_dir = "/etc/munin/plugins" }
		default: {}
	}
 
	file { $plugins_dir:
		source => "puppet://$fileserver/files/empty",
		ensure => directory,
		checksum => mtime,
		ignore => ".svn*",
		mode => 0755,
		recurse => true,
		purge => true,
		force => true,
		owner => root
	}
}
 
class munin::plugins::interfaces
{
	$ifs = gsub(split($munin_interfaces, " "), "(.+)", "if_\\1")
	$if_errs = gsub(split($munin_interfaces, " "), "(.+)", "if_err_\\1")
	plugin {
		$ifs: ensure => "if_";
		$if_errs: ensure => "if_err_";
	}
 
	include munin::plugins::base
}
 
class munin::plugins::linux 
{
	plugin { [ cpu, load, memory, swap, irq_stats, df, processes, open_files, ntp_offset, vmstat ]: 
		ensure => "present"
	}
 
	include munin::plugins::base
	include munin::plugins::interfaces
}
 
class munin::plugins::nfsclient
{
	plugin { "nfs_client":
		ensure => present
	}
}
 
class munin::plugins::snmp
{
	# initialize plugins
	$snmp_plugins=extlookup("munin_snmp_plugins")
	snmp_plugin { $snmp_plugins:
		ensure => present
	}
 
	# SNMP communities used by plugins
	file { "/etc/munin/plugin-conf.d/snmp_communities":
		owner => root,
		group => root,
		mode => 0644,
		source => "puppet://$fileserver/files/apps/munin/gatherer/snmp_communities"
	}
 
}
 
define munin::custom_plugin($ensure = "present", $location = "/etc/munin/plugins") {
	$plugin = "$location/$name"
 
	case $ensure {
		"absent": {
			file { $plugin: ensure => absent, } 
		}
 
		"present": {
			file { $plugin:
				owner => root,
				mode => 0755,
				source => "puppet://$fileserver/files/apps/munin/custom_plugins/$name",
				require => Package["munin-node"],
				notify => Service["munin-node"],
			}
		}
	}
}
 
class munin::plugins::custom::gatherer
{
	$plugins=extlookup("munin_custom_plugins")
	custom_plugin { $plugins:
		ensure => present
	}
}
 
class munin::plugins::freebsd 
{
	plugin { [ cpu, load, memory, swap, irq_stats, df, processes, open_files, ntp_offset, vmstat ]: 
		ensure => "present",
	}
 
	include munin::plugins::base
	include munin::plugins::interfaces
}
 
class munin::plugins::solaris 
{
	# Munin plugins on solaris are quite ... buggy. Will need rewrite / custom plugins.
	plugin { [ cpu, load, netstat ]: 
		ensure => "present",
	}
 
	include munin::plugins::base
	include munin::plugins::interfaces
}
Category: BOFH Life, Code, Puppet, SysAdmin, Tech  | Tags: ,  | 7 Comments
Wednesday, March 04th, 2009 | Author:

eth0Et voila, mon premier plugin pour munin ! C’est un plugin solaris only (utilisation du module Kstat oblige) qui permet de monitorer les I/O sur les pools ZFS. Je n’ai pas trouvé comment faire des stats par pool (comprendre à la “zpool iostat”) et si quelqu’un a une astuce à ce niveau là, je suis preneur. En attendant voici le fichier.

Thursday, February 26th, 2009 | Author:

eth0Tips rapide : pour accélerer la collecte des données avec munin-update, ajoutez dans /etc/munin/munin.conf du master :

fork yes

Ainsi munin fera un fork par machine à aller interroger, plutôt que bêtement les unes après les autres.

Category: Général  | Tags: ,  | Leave a Comment
Wednesday, February 25th, 2009 | Author:

eth0Au boulot j’ai remplacé cacti par munin. Munin c’est choupi, la conf est en flat file mais ça a deux défauts, ou un selon le point de vue : le munin-cgi-graph est tout simplement moisi car d’une lenteur sans nom, et générer les images lors du poll c’est long. Sauf quand on tombe sur ce petit hack convivial qui permet de lancer en parallèle la génération pour X nodes. Malheureusement, l’exemple donné en lien n’est valable que pour la version du trunk (au moment ou j’écris ceci) et pas pour la version 1.2.6 livrée avec lenny. J’ai donc modifié mon munin-graph en “portant” le hack disponible. Résultat : 2min30s au lieu de 9min pour la génération de 80 machines (dont certaines avec beaucoup d’infos)

Le code en question :

# Add this to munin-graph
sub myforked_processnode {
        my ($domain, $code) = @_;
 
        my $max=24;
        my $counter = 0;
 
        for my $key ( keys %{$config->{domain}->{$domain}->{node}}) {
                wait unless ++ $counter >= $max;
                #print "Processing ".$key."\n";
                die "Fork failed: $!\n" unless defined (my $pid = fork);
                exit $code -> ($domain,$key,$config->{domain}->{$domain}->{node}->{$key}) unless $pid;
        }
        1 until -1 == wait;
}
 
# Modify process_domain to get :
sub process_domain {
    my ($domain) = @_;
    myforked_processnode($domain, \&process_node);
}

Et voila !

Category: BOFH Life, SysAdmin, Tech  | Tags: , , , , ,  | 3 Comments