Puppet Beaker
Posted8 April 2015
I've been working with Puppet Beaker lately. It's a 'acceptance testing framework', primarily for Puppet, although it can do pretty much anything you want (if you want to try quite hard). As work is a Puppet shop, it's a natural extension to our existing Puppet Rspec unit testing (and whilst you can run unit tests in seconds, you can't test everything that way).
In short, Beaker is a means to spin up a number of virtual servers or containers, run Puppet in them to get something installed and then run a series of Rspec (well, ServerSpec) tests against them (and get results reported). Our primary use-case is to actually try running Puppet on 'bare metal', and maybe try an upgrade/downgrade of RPMs or whatever to make sure that they actually do what they're supposed to. All this really only needs a single VM or container, and so the tests you can do are pretty straight forward. We used to use some Vagrant VMs to do this sort of thing, using a handful of scripts and other hand-built stuff. We're gradually getting this moved over to use Beaker, with Docker containers instead of VMs.
My particular goal was to spin up multiple containers and have them communicate with each other. Since I've been doing quite a bit with Postgres recently, I thought I'd start with that. Our Puppet module is home-grown and can install multiple instances of Postgres, can run 'initdb' and can do all the required config for it too, so I figured I had all the raw ingredients to get cooking.
Getting Beaker to spin up multiple containers is super-easy - just define a few of them in nodesets/default.yml. It literally took me minutes to rip-off someone else's single host tests for their Puppet module and get multiple containers starting up and running a 'hello world' sort of Rspec test on the default one. Getting both containers to run Puppet with my manifest wasn't too hard either. I later went back and configured both of my containers to have a fixed IP address so that I could configure Postgres replication with known IPs rather than having to somehow figure out what they were in my Rspec tests. It seems that guests don't easily know the names or addresses of each other (although the hypervisor obviously does).
Setting up Postgres replication isn't a hard job, but it took me a while to get it to work in Beaker. I realised that I couldn't just let Puppet run 'initdb' on both containers and then hope to join them together. Instead, I had to do the basic puppet runs on both boxes, then stop Postgres on both boxes, and run a 'tar' command on the master to zip up the Postgres data files. I tried to copy that between containers, but it turns out that just because Beaker runs an 'sshd' on the containers doesn't mean you can use it yourself very easily. I've ended up copying back to the hypervisor and back out the the slave server. Some more arbitrary shelling out to the container and it's configured as a slave and happily replicating from the master.
To test that replication is actually working, I've got another loop around Beaker's list of containers to say "create table" on the master and then "select * from table" on the slaves. Assuming replication works quickly, this should work fine (although if the test runs before the slave has actually replicated the new table from the master then it'll fail, presumably intermittently). I'm not sure how to reliably avoid this, but so far it's not been a problem.
All in all, Beaker's actually been a pretty easy experience. I was worried it was a bit 'beta', but actually it's pretty solid. Some of the documentation could do with a bit of work (as seems to be the case with a lot of Puppetlabs stuff), and the community is yet to weed out and solve all the issues so Internet searching for solutions isn't always as fruitful as one might like. All that said though, it's pretty good.
More blog posts:
Previous Post: Migrate MySQL to Postgres | Next Post: Redis