Archive for the Category » Code «

Wednesday, September 08th, 2010 | Author: Nico

eth0 Most people that work with puppet use a VCS : subversion, git, CVS, mercurial… Pick yours. My company uses subversion and each cmmit to the repository needs to be pulled by the master. Since I have two masters, I also want them to be synchronized. Once again it’s mcollective that comes to the rescue. I wrote a very simple agent (a 5 minutes work, to be improved) that can update a specified path. Grab it here. Once it is deployed you can use a post commit hook that calls it.

Example of mine :

#!/usr/local/bin/ruby
 
require 'mcollective'
include MCollective::RPC
 
mc = rpcclient("svnagent")
mc.progress = false
mc.class_filter "puppet::master"
mc.update(:path => "/etc/puppet")

The agent will only be called on machines being puppet masters by using the class filter.

Category: Code, Puppet, SysAdmin  | Tags: , ,  | 2 Comments
Wednesday, August 18th, 2010 | Author: Nico

RubyQuick post : installing a mongo server is good, monitoring it is better. I wrote a very basic check that ensure your mongo DB is healthy. Grab it on my github.

Category: Code, SysAdmin, Tech  | Tags: , ,  | Leave a Comment
Tuesday, May 18th, 2010 | Author: Nico

eth0Some friends told me for a while about collectd, why I should look at it, why munin is so painful and so on. If you’ve been reading my posts you know I have tweaked a little my $WORK munin install to make it faster and lighter. But I finally took time to explore collectd, and I regret to not have done this before. It has so many pros that I decided to implement it in parallel with munin (because I can’t afford being blind on metrics). But collectd comes without an UI : it “only” collectds data, but that’s not a problem. There are various web interfaces and after giving a look to a bunch of them I fell in love with Lindsay Holmwood’s Visage.

This piece of software is definitely cool : all graphs are rendered live in your browser in SVG. Yes ! Realtime graphs, no need for crappy flash s***, zoom. It is based on sinatra, haml and some JS libraries (I won’t talk about this, my JS foo is deeper than the Mariana Trench). But it lacked some features : it’s OK when you have a few hosts but when the hosts list starts being loooong then the interface needs some improvements. So I forked it on github and implemented (some parts of) what I needed. My github fork has host grouping & per host profiles. Check this out and enjoy Visage !

Now working on sets of graphs :)

PS : <3 Guigui2

Wednesday, April 14th, 2010 | Author: Nico

eth0I already blogged about my experiments with mcollective & xen but I had something a little bigger in my mind. A friend had sent me a video showing some vmware neat features (DRS mainly) with VMs migrating through hypervisors automatically.

So I wrote a “proof of concept” of what you can do with an awesome tool like mcollective. The setup of this funny game is the following :

  • 1 box used a iSCSI target that serves volumes to the world
  • 2 xen hypervisors (lenny packages) using open-iscsi iSCSI initiator to connect to the target. VMs are stored in LVM, nothing fancy

The 3 boxens are connected on a 100Mb network and the hypervisors have an additionnal gigabit network card with a crossover cable to link them (yes, this is a lab setup). You can find a live migration howto here.

For the mcollective part I used my Xen agent (slightly modified from the previous post to support migration), which is based on my xen gem. The client is the largest part of the work but it’s still less than 200 lines of code. It can (and will) be improved because all the config is hardcoded. It would also deserve a little DSL to be able to handle more “logic” than “if load is superior to foo” but as I said before, it’s a proof of concept.

Let’s see it in action :

hypervisor2:~# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   233     2     r-----    873.5
hypervisor3:~# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   232     2     r-----  78838.0
test1                                        6   256     1     -b----     18.4
test2                                        4   256     1     -b----     19.3
test3                                       20   256     1     r-----     11.9

test3 is a VM that is “artificially” loaded, as is the machine “hypervisor3″ (to trigger migration)

[mordor:~] ./mc-xen-balancer
[+] hypervisor2 : 0.0 load and 0 slice(s) running
[+] init/reset load counter for hypervisor2
[+] hypervisor2 has no slices consuming CPU time
[+] hypervisor3 : 1.11 load and 3 slice(s) running
[+] added test1 on hypervisor3 with 0 CPU time (registered 18.4 as a reference)
[+] added test2 on hypervisor3 with 0 CPU time (registered 19.4 as a reference)
[+] added test3 on hypervisor3 with 0 CPU time (registered 18.3 as a reference)
[+] sleeping for 30 seconds

[+] hypervisor2 : 0.0 load and 0 slice(s) running
[+] init/reset load counter for hypervisor2
[+] hypervisor2 has no slices consuming CPU time
[+] hypervisor3 : 1.33 load and 3 slice(s) running
[+] updated test1 on hypervisor3 with 0.0 CPU time eaten (registered 18.4 as a reference)
[+] updated test2 on hypervisor3 with 0.0 CPU time eaten (registered 19.4 as a reference)
[+] updated test3 on hypervisor3 with 1.5 CPU time eaten (registered 19.8 as a reference)
[+] sleeping for 30 seconds

[+] hypervisor2 : 0.16 load and 0 slice(s) running
[+] init/reset load counter for hypervisor2
[+] hypervisor2 has no slices consuming CPU time
[+] hypervisor3 : 1.33 load and 3 slice(s) running
[+] updated test1 on hypervisor3 with 0.0 CPU time eaten (registered 18.4 as a reference)
[+] updated test2 on hypervisor3 with 0.0 CPU time eaten (registered 19.4 as a reference)
[+] updated test3 on hypervisor3 with 1.7 CPU time eaten (registered 21.5 as a reference)
[+] hypervisor3 has 3 threshold overload
[+] Time to see if we can migrate a VM from hypervisor3
[+] VM key : hypervisor3-test3
[+] Time consumed in a run (interval is 30s) : 1.7
[+] hypervisor2 is a candidate for being a host (step 1 : max VMs)
[+] hypervisor2 is a candidate for being a host (step 2 : max load)
trying to migrate test3 from hypervisor3 to hypervisor2 (10.0.0.2)
Successfully migrated test3 !

Let’s see our hypervisors :

hypervisor2:~# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   233     2     r-----    878.9
test3                                       25   256     1     -b----      1.1
hypervisor3:~# xm list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0   232     2     r-----  79079.3
test1                                        6   256     1     -b----     18.4
test2                                        4   256     1     -b----     19.4

A little word about configuration options :

  • interval : the poll time in seconds.  this should not be too low, let the machine some time and avoid load peeks to distort the logic.
  • load_threshold : where you consider the machine load is too high and that it is time to move some stuff away (tampered with max_over, see below)
  • daemonize : not used yet
  • max_over : maximum time (in minutes) where load should be superior to the limit. When reached, it’s time, really. Don’t set it too low and at least 2*interval or sampling will not be efficient
  • debug : well….
  • max_vm_per_host : the maximum VMs a host can handle. If a host already hit this limit it will not be candidate for receiving a VM
  • max_load_candidate : same thing as above, but for the load
  • host_mapping : a simple CSV file to handle non-DNS destinations (typically my crossover cable address have no DNS entries)

What is left to do :

  • Add some barriers to avoid migration madness to let load go down after a migration or to avoid migrating a VM permanently
  • Add a DSL to insert some more logic
  • Write a real client, not a big fat loop

Enjoy the tool !

Files :

Tuesday, April 06th, 2010 | Author: Nico

eth0Another cool project I keep an eye on for some weeks is “the marionette collective“, aka mcollective. This project is leaded & develloped by R.I. Pienaar, one of the most active people in the puppet world too.

Mcollective is an framework for distributed sysadmin. It relies on a messaging framework and has many features included : flexibility, speed, easy to understand.

Some time ago, I had wrote a tool called “whosyourdaddy” to help me (and my memory as big as a goldfish one) to find on which Xen dom0 a Xen domU was living. It worked fine, expect the fact that is was not dynamic : if a VM was migrated  from a dom0 to another, I had to update the CMDB. Not really reliable (if an update fails the CMDB is no more accurate) and I didn’t want to have to embed this constraint in the Xen logic. So I decided to try out to write my own mcollective agent and here it is ! It is built on top of a (very) small ruby module for xen and has it own client.

You can find on which dom0 a domU resides :

master1:~# ./mc-xen -a find --domu test
hypervisor2              : Absent
hypervisor1              : Absent
master1:~# ./mc-xen -a find --domu domu2
hypervisor2              : Present
hypervisor1              : Absent

Or list your domUs :

master1:~# ./mc-xen -a list
hypervisor2              
 domu2

hypervisor1              
 no domU running

Download the agent & the client

Tuesday, March 02nd, 2010 | Author: Nico

kermitI’ve been lazy at maintaining my servers recently and decided to start playing with puppet reports. First I started with something simple that helps me to find on which machines my manifests have some failure.

So here’s a quick and dirty code that goes through Puppet’s reportdir and points out neglected machines.

#!/usr/bin/env ruby
 
require 'puppet'
require 'find'
require 'yaml'
require 'optparse'
 
Puppet[:config] = "/etc/puppet/puppet.conf"
Puppet.parse_config
 
def most_recent_file(path)
	reports = []
	Find.find(path) { |file|
		if File.file? file
			reports << File.basename(file,".yaml")
		end
	}
	reports.sort!.reverse!
	return path+"/"+reports[0].to_s+".yaml"
end
 
 
def scan_dir(path, debug=false)
	Find.find(path) { |entry|
		if entry != path # don't scan the basedir
			if File.directory? entry
				report = most_recent_file(entry)
				scan_file(report, debug)
			end
		end
	}
end
 
 
def scan_file(filename, debug=false)
	notify_on_field = [:failed]
 
	# debug
	if debug then  puts "scanning " + filename end
 
	fp=open(filename,"r")
	YAML::load_documents(fp) { |report|
		report.metrics["resources"].values.each { |value|
			if (notify_on_field.include? value[0]) and (value[2] > 0) then
				puts "#{report.host} has #{value[2]} #{value[0]} resource(s)"
				if debug then
					puts "log message(s) :"
					report.logs.each { |log|
						puts log.message
					}
				end
			end
		}
	}	
end
 
options = {}
myargs = Array
 
optparse = OptionParser.new { |opts|
	opts.banner = "Usage : report_check.rb"
 
	options[:show]=false
	opts.on("-d", "--debug", "runs in debug mode") do |debug|
		options[:debug]=true
	end
 
	opts.on("-h", "--help", "Displays this help") do
		puts opts
		exit
	end
 
}
 
optparse.parse!
 
scan_dir(Puppet[:reportdir], options[:debug])
Friday, February 26th, 2010 | Author: Nico

kermitOn my Solaris machines at $WORK I use iMil’s pkgin to install additional software. But until today, I add to do it by hand, on every machine… Not really what I like to do after a little more than a year using puppet. So I wrote a provider to manage packages with pkgin. It was very informative on puppet internals and I learned more about my favorite config management system.

Enough talking, here is the file : pkgin.rb

Example of use in a manifest :

class foo {
    package { "bla":
        ensure => installed,
        provider => pkgin
    }
}
Wednesday, January 27th, 2010 | Author: Nico

Today I started installing a reverse proxy at $WORK. I choose to follow this way, and all my DNS data is stored in my CMDB. Once again, the solution came from #puppet ! You can embed some “pure” ruby code in ERB templates. And, yes, you can query your database !

<%
dbh = DBI.connect("DBI:Mysql:yourbase:mysql.mycorp.com", "you", "XXXX")
query = dbh.prepare("your fancy query")
query.execute
while row = query.fetch do
todisplay=some_funny_things()
%>
<%= todisplay %>
<% end %>

I use this technique to generate the dnsmasq data file. Just use the subscribe function and all is done !

Wednesday, August 05th, 2009 | Author: Nico

Disclaimer : this work is mostly based upon DavidS work, available on his git repo. In the scope of my work I needed to have munin support for freeBSD & Solaris. I also wrote a class for snmp_plugins & custom plugins. Some things are quite dependant from my infrastructure, like munin.conf generation script but it can easily be adapted to yours, by extracting data from your CMDB.

It requires the munin_interfaces fact published here (and merged into DavidS repo, thanks to him), and Volcane’s extlookup function to store some parameters. Enough talking, this is the code :

# Munin config class
# Many parts taken from David Schmitt's http://git.black.co.at/
# FreeBSD & Solaris + SNMP & custom plugins support by Nicolas Szalay <nico@gcu.info>
 
class munin::node {
	case $operatingsystem {
		openbsd: {}
		debian: { include munin::node::debian}
		freebsd: { include munin::node::freebsd}
		solaris: { include munin::node::solaris}
		default: {}
	}
}
 
class munin::node::debian {
 
	package { "munin-node": ensure => installed }
 
	file { 
	"/etc/munin":
		ensure => directory,
		mode => 0755,
		owner => root,
		group => root;
 
	"/etc/munin/munin-node.conf":
		source => "puppet://$fileserver/files/apps/munin/munin-node-debian.conf",
		owner => root,
		group => root,
		mode => 0644,
		before => Package["munin-node"],
		notify => Service["munin-node"],
	}
 
	service { "munin-node": ensure => running }
 
	include munin::plugins::linux 
}
 
class munin::node::freebsd {
	package { "munin-node": ensure => installed, provider => freebsd }
 
        file { "/usr/local/etc/munin/munin-node.conf":
                source => "puppet://$fileserver/files/apps/munin/munin-node-freebsd.conf",
                owner => root,
                group => wheel,
                mode => 0644,
                before => Package["munin-node"],
                notify => Service["munin-node"],
        }
 
	service { "munin-node": ensure => running }
 
	include munin::plugins::freebsd
}
 
class munin::node::solaris {
	# "hand made" install, no package.
	file { "/etc/munin/munin-node.conf":
		source => "puppet://$fileserver/files/apps/munin/munin-node-solaris.conf",
                owner => root,
                group => root,
                mode => 0644
	}
 
	include munin::plugins::solaris
}
 
class munin::gatherer {
	package { "munin":
		ensure => installed
	}
 
	# custom version of munin-graph : forks & generates many graphs in parallel
	file { "/usr/share/munin/munin-graph":
		owner => root,
		group => root,
		mode => 0755,
		source => "puppet://$fileserver/files/apps/munin/gatherer/munin-graph",
		require => Package["munin"]
	}
 
	# custon version of debian cron file. Month & Year cron are generated once daily
	file { "/etc/cron.d/munin":
		owner => root,
		group => root,
		mode => 0644,
		source => "puppet://$fileserver/files/apps/munin/gatherer/munin.cron",
		require => Package["munin"]
	}
 
	# Ensure cron is running, to fetch every 5 minutes
	service { "cron":
		ensure => running
	}
 
	# Ruby DBI for mysql
	package { "libdbd-mysql-ruby":
		ensure => installed
	}
 
	# config generator
	file { "/opt/scripts/muningen.rb":
		owner => root,
		group => root,
		mode => 0755,
		source => "puppet://$fileserver/files/apps/munin/gatherer/muningen.rb",
		require => Package["munin", "libdbd-mysql-ruby"]
	}	
 
	# regenerate munin's gatherer config every hour
	cron { "munin_config":
		command => "/opt/scripts/muningen.rb > /etc/munin/munin.conf",
		user => "root",
		minute => "0",
		require => File["/opt/scripts/muningen.rb"]
	}
 
	include munin::plugins::snmp
	include munin::plugins::linux
	include munin::plugins::custom::gatherer
}
 
 
# define to create a munin plugin inside the right directory
define munin::plugin ($ensure = "present") {
 
	case $operatingsystem {
		freebsd: { 
			$script_path = "/usr/local/share/munin/plugins"
			$plugins_dir = "/usr/local/etc/munin/plugins"
		}
		debian: { 
			$script_path = "/usr/share/munin/plugins"
			$plugins_dir = "/etc/munin/plugins"
		}
		solaris: { 
			$script_path = "/usr/local/munin/lib/plugins"
			$plugins_dir = "/etc/munin/plugins"
		}
		default: { }
	}
 
	$plugin = "$plugins_dir/$name"
 
	case $ensure {
		"absent": {
			debug ( "munin_plugin: suppressing $plugin" )
			file { $plugin: ensure => absent, } 
		}
 
		default: {
			$plugin_src = $ensure ? { "present" => $name, default => $ensure }
 
			file { $plugin:
				ensure => "$script_path/${plugin_src}",
				require => Package["munin-node"],
				notify => Service["munin-node"],
			}
		}
	}
}
 
# snmp plugin define, almost same as above
define munin::snmp_plugin ($ensure = "present") {
	$pluginname = get_plugin_name($name)
 
	case $operatingsystem {
		freebsd: { 
			$script_path = "/usr/local/share/munin/plugins"
			$plugins_dir = "/usr/local/etc/munin/plugins"
		}
		debian: { 
			$script_path = "/usr/share/munin/plugins"
			$plugins_dir = "/etc/munin/plugins"
		}
		solaris: { 
			$script_path = "/usr/local/munin/lib/plugins"
			$plugins_dir = "/etc/munin/plugins"
		}
		default: { }
	}
 
	$plugin = "$plugins_dir/$name"
 
	case $ensure {
		"absent": {
			debug ( "munin_plugin: suppressing $plugin" )
			file { $plugin: ensure => absent, } 
		}
 
		"present": {
			file { $plugin:
				ensure => "$script_path/${pluginname}",
				require => Package["munin-node"],
				notify => Service["munin-node"],
			}
		}
	}
}
 
class munin::plugins::base
{
	case $operatingsystem {
		debian: { $plugins_dir = "/etc/munin/plugins" }
		freebsd: { $plugins_dir = "/usr/local/etc/munin/plugins" }
		solaris: { $plugins_dir = "/etc/munin/plugins" }
		default: {}
	}
 
	file { $plugins_dir:
		source => "puppet://$fileserver/files/empty",
		ensure => directory,
		checksum => mtime,
		ignore => ".svn*",
		mode => 0755,
		recurse => true,
		purge => true,
		force => true,
		owner => root
	}
}
 
class munin::plugins::interfaces
{
	$ifs = gsub(split($munin_interfaces, " "), "(.+)", "if_\\1")
	$if_errs = gsub(split($munin_interfaces, " "), "(.+)", "if_err_\\1")
	plugin {
		$ifs: ensure => "if_";
		$if_errs: ensure => "if_err_";
	}
 
	include munin::plugins::base
}
 
class munin::plugins::linux 
{
	plugin { [ cpu, load, memory, swap, irq_stats, df, processes, open_files, ntp_offset, vmstat ]: 
		ensure => "present"
	}
 
	include munin::plugins::base
	include munin::plugins::interfaces
}
 
class munin::plugins::nfsclient
{
	plugin { "nfs_client":
		ensure => present
	}
}
 
class munin::plugins::snmp
{
	# initialize plugins
	$snmp_plugins=extlookup("munin_snmp_plugins")
	snmp_plugin { $snmp_plugins:
		ensure => present
	}
 
	# SNMP communities used by plugins
	file { "/etc/munin/plugin-conf.d/snmp_communities":
		owner => root,
		group => root,
		mode => 0644,
		source => "puppet://$fileserver/files/apps/munin/gatherer/snmp_communities"
	}
 
}
 
define munin::custom_plugin($ensure = "present", $location = "/etc/munin/plugins") {
	$plugin = "$location/$name"
 
	case $ensure {
		"absent": {
			file { $plugin: ensure => absent, } 
		}
 
		"present": {
			file { $plugin:
				owner => root,
				mode => 0755,
				source => "puppet://$fileserver/files/apps/munin/custom_plugins/$name",
				require => Package["munin-node"],
				notify => Service["munin-node"],
			}
		}
	}
}
 
class munin::plugins::custom::gatherer
{
	$plugins=extlookup("munin_custom_plugins")
	custom_plugin { $plugins:
		ensure => present
	}
}
 
class munin::plugins::freebsd 
{
	plugin { [ cpu, load, memory, swap, irq_stats, df, processes, open_files, ntp_offset, vmstat ]: 
		ensure => "present",
	}
 
	include munin::plugins::base
	include munin::plugins::interfaces
}
 
class munin::plugins::solaris 
{
	# Munin plugins on solaris are quite ... buggy. Will need rewrite / custom plugins.
	plugin { [ cpu, load, netstat ]: 
		ensure => "present",
	}
 
	include munin::plugins::base
	include munin::plugins::interfaces
}
Category: BOFH Life, Code, Puppet, SysAdmin, Tech  | Tags: ,  | Leave a Comment
Thursday, July 30th, 2009 | Author: Nico

-Post en anglais, pour une fois-

Everyone using puppet knows DavidS awesome git repository : git.black.co.at. Unfornately for me, his puppet infrastructure seems to be almost only linux based. I have different OS in mine, including FreeBSD & OpenSolaris. Looking at his module-munin I decided to reuse it (and not recreate the wheel) but he used a custom fact that needed some little work. So this is a FreeBSD & (Open)Solaris capable version, to know what network interfaces have link up

# return the set of active interfaces as an array
# taken from http://git.black.co.at
# modified by nico <nico@gcu.info> to add FreeBSD & Solaris support
 
Facter.add("munin_interfaces") do
 
	setcode do
		# linux
		if Facter.value('kernel') == "Linux" then
			`ip -o link show`.split(/\n/).collect do |line|
					value = nil
					matches = line.match(/^\d*: ([^:]*): <(.*,)?UP(,.*)?>/)
					if !matches.nil?
						value = matches[1]
						value.gsub!(/@.*/, '')
					end
					value
			end.compact.sort.join(" ")
		#end
 
		# freebsd
		elsif Facter.value('kernel') == "FreeBSD" then
			Facter.value('interfaces').split(/,/).collect do |interface|
				status = `ifconfig #{interface} | grep status`
				if status != "" then
					status=status.strip!.split(":")[1].strip!
					if status == "active" then # I CAN HAZ LINK ?
						interface.to_a
					end
				end
			end.compact.sort.join(" ")
		#end
 
		# solaris
		elsif Facter.value('kernel') == "SunOS" then
			Facter.value('interfaces').split(/,/).collect do |interface|
				if interface != "lo0" then # /dev/lo0 does not exists
					status = `ndd -get /dev/#{interface} link_status`.strip!
					if status == "1" # ndd returns 1 for link up, 0 for down
						interface.to_a
					end
				end
			end.compact.sort.join(" ")
		end
	end
end

Thanks to Volcane from IRC for helping me.

Category: BOFH Life, Code, Puppet, SysAdmin, Tech  | Tags: ,  | Leave a Comment