Posted 8 January 2015
Towards the end of last year, I had an opportunity to play with Docker. Docker wasn't the purpose of the job at hand though, so it turned out to be one of those incidental things that takes up the majority of the time to complete a task. At work, we're starting to look at Docker as a way to provision apps across the estate, with the aim of making things a bit more 'cloudy' and dynamic than they are today.
At work, we have an RPM package builder that's based on SVN. That is, you lay out a bunch of files in filesystem format into an SVN repository. When you're ready, you commit them and minutes later, a Jenkins server checks out your work from SVN, makes an RPM package from it and puts it into a local repo that all the machines can access. This solution has been the backbone of countless things we've done - being able to make and deploy packages is the crux of any sort of repeatable build capability.
The RPM builder was knocked up over a few days by someone just trying a few things out to see if he could make it work. The heart of it (a Jenkins build job) was a massive Bash script, which (as these things tend to) had been modified here and there to make it do more than originally intended. Despite its humble demeanour though, this script actually proved to be pretty solid, building countless packages of various types over maybe two years. We even ran it on two machines, one was Redhat Enterprise 5, the other RHEL6. The reason was that this little script could (with some help) perform some more traditional build steps before zipping up the RPM. We used that capability in some cases to compile native code before making the package.
All this had one fatal flaw though... the very first commit you did for your new package would always fail. You'd always have to go back in an do a dummy commit to actually get an RPM out. Not terrible when you only make one package every couple of months, but a complete pain when I had to build a couple of dozen packages one afternoon.
I decided enough was enough, and now was the time to revamp the package builder. I planned to just rewrite the Bash script in Ruby (which is our preferred language these days). It quickly became apparent that I could break the task of "make RPM" into various steps and make a little pipeline in Jenkins. I now have a "prepare" step that does the SVN checkout and gets things set up. The 'create RPM' step optionally does the 'build native code' work, but ultimately uses FPM to zip up a load of files into an RPM. Lastly, there are 'copy to repo' and 'reindex repo' steps.
Of course, if you're going to potentially build native code when you're making an RPM, you need to do it on the same OS environment as you're planning to use it on. Here's where I figured Docker could help. I really didn't know much about Docker at this point, other than I'd heard it was sort of like a "skinny" VM. I think nowadays I'd describe it more like a "fat" chroot, but that's another matter.
My first problem was I needed a Redhat Docker image. I couldn't find a good starting point on the Docker Hub, so elected to make my own. In hindsight, I suspect a bit more looking around might have saved me time in the long run, but actually it means we've got exactly what we want (ie. the same as our real systems), rather than maybe something that works but isn't quite as we'd like it.
Making the initial image isn't hard, but it's not easy either. In short, you need to use 'yum' to install a bunch of packages into a directory on your system. When it's done that, you tar up that directory and feed it into Docker to make an image from it. That list of base packages is semi-documented on the Internet so it took a bit of trial and error to get it right. However, once I had a method, making the RHEL5, RHEL6 and RHEL7 variants of it was pretty easy.
The next step was to make that very simple base image into something a bit more like the company Standard Build (although without lots of the extras on it). The nice thing here is that you can do this inside a Docker container. You just run
/bin/bash in the base image, Docker makes a container and away you go. You can go ahead and install packages, edit configs or whatever you like. Docker keeps it all in the container that it made from the image. When you're done, you exit the shell and Docker stops the container. Redhat Satellite users (like us) need to be careful though because you probably don't want to register Docker images with the Satellite just to use 'yum' (you might want a non-Satellite RPM repo instead...?)
The learning curve for me was that containers and images are different. In this example, the image is the 'minimalist base', but the container (now) has all sorts of extra stuff in it. You can re-connect to a container after you've stopped it, but it's not especially convenient. You really need to 'commit' your container into an image, after which you can always create a new container from your image. In that sense, it provides a sort of 'snapshot' of your solution.
To finish off, I took my company-specific image, and made a new image which includes all the package building tools in it. This image is the one the RPM builder uses. It actually uses three of them (one for each of RHEL5, 6 and 7). Jenkins is set up to run three jobs in parallel, each of which starts up a Docker container from the appropriate image and runs the package builder inside it. The package builder may then go ahead and run gcc or whatever, but ultimately saves out an RPM file.
It turns out that re-indexing repos is an OS-specific activity as well these days. The latest RHEL6 versions of 'createrepo' won't create RHEL5-compatible repo meta data. RHEL7 does things very differently, and so can't be used for anything other than RHEL7 repos. Once again, Docker saves us here too - Jenkins runs an 'reindex repo' step by starting up an environment specific image and mapping the repo directory into it. It works very nicely, and actually means we can now run all three OS variants on a single VM.
In conclusion, I have to say Docker looks very promising. It absolutely doesn't work properly on RHEL6 though, and even on RHEL7 it has quite a few rough edges. I'd like to try the very latest versions of it to see if they fix these sorts of problems. Right now though, I'm not quite sure its ready to 'bet the house' on it, but we've done some pretty cool stuff with it, and I'm sure it'll eventually start replacing VMs in a lot of peoples datacentres.
More blog posts:Previous Post: First Post | Next Post: Git-Backed Website Content