When I wrote the asynchronous chef handler that I presented in the previous post, I had a little idea in mind. being able to track changes made by chef. The idea grew up a little and I now release a little dashboard I wrote. It’s still in a very early stage of development but I’ll try to present the idea behind it.

The changes (including diffs) are pushed in a queue. This queue is consumed by a script and datas are indexed in elasticsearch, an open source search engine. On the top of this I wrote a web interface, based on sinatra and twitter’s bootstrap) that allow you to see the latest runs, filter out runs with no changes and search on criterias (hostname, updated resources and inside the diffs)

To use this you will need a couple of things :

• a STOMP broker (I use rabbitmq, but activemq or stompserver will fit too)
• an elasticsearch instance (or cluster)
• sinatra and its dependencies + the stomp ruby gem

The code is available on github, feel free to get in touch on freenode IRC, you can find me on the #chef-hacking channel (nickname : nico)

Configuration management tools are awesome. Using them, you are now managing loads of servers, reaching the pub on time and you can focus on really fun stuff. A counterpart is that they almost work in your back : changes are propagated quickly, and even if you store your cookbooks/modules in a VCS, even if you review them, you still want to know what really happens on your servers. Puppet has a really nice automatic summary part to do that job, and some people are doing cool things with it. Chef also has summary of updated resources but they don’t get back to a master like puppet do. So I wrote a little report handler to push back data to wherever you want to. It is based on the stomp protocol, to be non blocking and easily scale if you have a large number of machine. With this you just need to have a consumer that will process data to your favorite datastore.

I’ve put that on my github chef repo, check the readme for the extensive file list & function.

Mcollective has been able to fire up puppetd runs for a while now, via a standalone RPC call or through the puppet commander binary (check it out, spread your load). I wanted to be able to fire up my chef clients with mcollective, to use metadata to filter what should be impacted. So I wrote a little piece of ruby, mostly based on the puppet one, to achieve this. You can now do the following :

mco rpc chef runonce or  mco rpc chef status
The plugin is available on my github mcollective repository.

It started like a toy, to learn a little more about mcollective agents but I finally turned into something useful (at least for me). I pushed my agent “smith” on my github account. It allows you to install or remove agents within mcollective. I usually use my configuration management tool to deploy such pieces of software but it can be useful in some case to go without it.

The mandatory internet meme reference : yes xzibit approves.

This post follows my previous one, dealing with the reuse of chef providers of chef in mcollective. In the comments Adam Jacob had an interesting word and when I wrote my second agent, to manage package I saw it would be a piece of cake to write a really generic agent, due to the nature of chef resource (and the way to invoke them)

So, this is a generic chef resource mcollective agent, with the associated example client code. It anyway deserves an little explanation; it is not mean to work with a command line invocation. Why ? Because I push quite “complex” data as the resourceactions parameter. The only way I found to make this work from command line is to use eval on the argument, which is no way acceptable. Anyway I hope some people will find this useful.

It has been quite calm for a couple of months here. I have switched job, it explains why I had less time to post some things.I now work at fotolia, and I switched from puppet to chef (no troll intended, I still think puppet is a great tool, please read this).

However, a tool I still have is the awesome mcollective. Unfortunately, the most used agents (package, service) relay on puppet providers to do their actions. Fortunately, open source is here, so I wrote a (basic) service agent that uses chef providers to start/stop or restart an agent. It still needs some polish for the status part (ho the ugly hardcoded path) but I was quite excited to share this. Freshly pushed on github !

Thanks to Jordan Sissel for minstrel, an awesome debug tool, the opscode team for the help on the provider and R.I. Pienaar for mcollective (and the support).

Just a little script that enables you to launch a cluster ssh based on mcollective discovery capacities. Now you can say “I want cssh to all my openbsd boxes” in only one command.

mc-cluster-ssh --wf kernel=OpenBSD

Grab it in my github

Basing more and more stuff on mcollective means relying more and more on one of its underlying components : the activeMQ middleware, and more precisely the stomp connector. I hit a weird bug a few days ago and realized that I was not functionnaly monitoring this part of the system. The port was bound and responded to connections, subscriptions were possible but messages didn’t pass through.So I wrote this little plugin that makes this possible : it creates a random string, sends it to a queue and then reads the queue to check if the result is the same.

This has been possible with the help of @ripienaar. Thanks for the explanation for the difference between topics & queues !

I recently switched from puppet daemon to mcollective commander : it kicks the “stuck in outer space puppet daemon” feature out of the way and brings me nice features (as load control). To do so I deployed the puppetd agent over my boxes.

As most of the sysadmins say : “if it’s not monitored, it doesn’t exist” I had placed a script in cron based on the puppetlast script to report by mail once a day which hosts had not checked in during the last 30 minutes. This method had 2 serious flaws : it runs only once a day (I hate being flooded by mails) and test machines keep nagging you until you remove the yaml files on the puppetmaster. Talking with Volcane on #mcollective I discovered that the agent was able to report when the client last run, so I decided to use this to check my puppet runs freshness with a nagios plugin.

Good bye cron job, say hello nagios.

Most people that work with puppet use a VCS : subversion, git, CVS, mercurial… Pick yours. My company uses subversion and each cmmit to the repository needs to be pulled by the master. Since I have two masters, I also want them to be synchronized. Once again it’s mcollective that comes to the rescue. I wrote a very simple agent (a 5 minutes work, to be improved) that can update a specified path. Grab it here. Once it is deployed you can use a post commit hook that calls it.

Example of mine :

#!/usr/local/bin/ruby
 
require 'mcollective'
include MCollective::RPC
 
mc = rpcclient("svnagent")
mc.progress = false
mc.class_filter "puppet::master"
mc.update(:path => "/etc/puppet")

The agent will only be called on machines being puppet masters by using the class filter.