When I wrote the asynchronous chef handler that I presented in the previous post, I had a little idea in mind. being able to track changes made by chef. The idea grew up a little and I now release a little dashboard I wrote. It’s still in a very early stage of development but I’ll try to present the idea behind it.

The changes (including diffs) are pushed in a queue. This queue is consumed by a script and datas are indexed in elasticsearch, an open source search engine. On the top of this I wrote a web interface, based on sinatra and twitter’s bootstrap) that allow you to see the latest runs, filter out runs with no changes and search on criterias (hostname, updated resources and inside the diffs)

To use this you will need a couple of things :

• a STOMP broker (I use rabbitmq, but activemq or stompserver will fit too)
• an elasticsearch instance (or cluster)
• sinatra and its dependencies + the stomp ruby gem

The code is available on github, feel free to get in touch on freenode IRC, you can find me on the #chef-hacking channel (nickname : nico)

Configuration management tools are awesome. Using them, you are now managing loads of servers, reaching the pub on time and you can focus on really fun stuff. A counterpart is that they almost work in your back : changes are propagated quickly, and even if you store your cookbooks/modules in a VCS, even if you review them, you still want to know what really happens on your servers. Puppet has a really nice automatic summary part to do that job, and some people are doing cool things with it. Chef also has summary of updated resources but they don’t get back to a master like puppet do. So I wrote a little report handler to push back data to wherever you want to. It is based on the stomp protocol, to be non blocking and easily scale if you have a large number of machine. With this you just need to have a consumer that will process data to your favorite datastore.

I’ve put that on my github chef repo, check the readme for the extensive file list & function.

Mcollective has been able to fire up puppetd runs for a while now, via a standalone RPC call or through the puppet commander binary (check it out, spread your load). I wanted to be able to fire up my chef clients with mcollective, to use metadata to filter what should be impacted. So I wrote a little piece of ruby, mostly based on the puppet one, to achieve this. You can now do the following :

mco rpc chef runonce or  mco rpc chef status
The plugin is available on my github mcollective repository.

It started like a toy, to learn a little more about mcollective agents but I finally turned into something useful (at least for me). I pushed my agent “smith” on my github account. It allows you to install or remove agents within mcollective. I usually use my configuration management tool to deploy such pieces of software but it can be useful in some case to go without it.

The mandatory internet meme reference : yes xzibit approves.

Just a little script that enables you to launch a cluster ssh based on mcollective discovery capacities. Now you can say “I want cssh to all my openbsd boxes” in only one command.

mc-cluster-ssh --wf kernel=OpenBSD

Grab it in my github

Basing more and more stuff on mcollective means relying more and more on one of its underlying components : the activeMQ middleware, and more precisely the stomp connector. I hit a weird bug a few days ago and realized that I was not functionnaly monitoring this part of the system. The port was bound and responded to connections, subscriptions were possible but messages didn’t pass through.So I wrote this little plugin that makes this possible : it creates a random string, sends it to a queue and then reads the queue to check if the result is the same.

This has been possible with the help of @ripienaar. Thanks for the explanation for the difference between topics & queues !

Another cool project I keep an eye on for some weeks is “the marionette collective“, aka mcollective. This project is leaded & develloped by R.I. Pienaar, one of the most active people in the puppet world too.

Mcollective is an framework for distributed sysadmin. It relies on a messaging framework and has many features included : flexibility, speed, easy to understand.

Some time ago, I had wrote a tool called “whosyourdaddy” to help me (and my memory as big as a goldfish one) to find on which Xen dom0 a Xen domU was living. It worked fine, expect the fact that is was not dynamic : if a VM was migrated  from a dom0 to another, I had to update the CMDB. Not really reliable (if an update fails the CMDB is no more accurate) and I didn’t want to have to embed this constraint in the Xen logic. So I decided to try out to write my own mcollective agent and here it is ! It is built on top of a (very) small ruby module for xen and has it own client.

You can find on which dom0 a domU resides :

master1:~# ./mc-xen -a find --domu test
hypervisor2              : Absent
hypervisor1              : Absent
master1:~# ./mc-xen -a find --domu domu2
hypervisor2              : Present
hypervisor1              : Absent

Or list your domUs :

master1:~# ./mc-xen -a list
hypervisor2              
 domu2

hypervisor1              
 no domU running

Download the agent & the client

Today I started installing a reverse proxy at $WORK. I choose to follow this way, and all my DNS data is stored in my CMDB. Once again, the solution came from #puppet ! You can embed some “pure” ruby code in ERB templates. And, yes, you can query your database !

<%
dbh = DBI.connect("DBI:Mysql:yourbase:mysql.mycorp.com", "you", "XXXX")
query = dbh.prepare("your fancy query")
query.execute
while row = query.fetch do
todisplay=some_funny_things()
%>
<%= todisplay %>
<% end %>

I use this technique to generate the dnsmasq data file. Just use the subscribe function and all is done !

Pour cross-compiler des applis pour son drobo :

• Décompresser le SDK (arm-2006q1-6-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2), par exemple /home/nico/drobo
• Ajouter /home/nico/drobo/bin au PATH
• créer un répertoire de build, par exemple /home/nico/drobo/build
• créer un répertoire de destination, par exemple /home/nico/drobo/target
• décompresser les sources à cross-compiler dans le rep de build
• lancer le configure de l’appli comme suit :

./configure --host=arm-none-linux-gnueabi --prefix=/home/nico/drobo/target

• puis make & make install classique

hop !

Tips rapide : pour accélerer la collecte des données avec munin-update, ajoutez dans /etc/munin/munin.conf du master :

fork yes

Ainsi munin fera un fork par machine à aller interroger, plutôt que bêtement les unes après les autres.