Backing up your Jenkins configuration

After your team has been using Jenkins for a while you’ll want to start thinking about backing up your configuration regularly. It’s the first step toward making your Jenkins installation “highly available” (quotes intentional). One day I stopped and asked myself, “How painful would it be if I had to re-create all of our configuration data from scratch?” The answer was that it would probably cost us several days of reduced productivity, not a risk worth taking given the very rapid pace of development and deployment we’re trying to sustain. I looked at a few different backup solutions, and eventually ended up just extending a generic one that we already use in-house.

plattersBefore I get into the different ways you can do this, I highly recommend that you take a look at this white paper that Jenkins creator Kohsuke Kawaguchi wrote, 7 Ways to Optimize Jenkins [pdf]. One section is about backups, and gives a lot of good advice. This is what I used to determine which things were most important to back up.

There are a few Jenkins plugins which will implement backups for you. The Backup plugin is no longer maintained and hasn’t had a release since 2011. It also backs up everything in your JENKINS_HOME directory, which is unnecessary and will quickly become a massive amount of data. Not recommended.

The SCM Sync Configuration plugin will write your Jenkins and individual job configurations to an SCM system. It currently only supports Git and Subversion, and every time you change a config and save it the plugin will interrupt you to ask for a comment to go with the SCM commit. I would find that annoying because I make configuration changes regularly, and I already use the Job Config History plugin to track, diff, and revert changes when necessary with no SCM required.

The thinBackup plugin only backs up essential configuration and has a bunch of good configuration options so you can tune it to your needs (back up plugins too, for example). It can also restore your backup for you. This is probably the best of the plugin options, but it can only store the backup locally on your Jenkins master. You would still have to script a separate process to move the backups off to another system for safe keeping. That’s what keeps me from using it.

At BrightRoll we are heavy users of Amazon AWS, and we have a skeleton BASH script (!) that can be run from cron to back up data to S3, typically as a tar file. My Jenkins version of that runs these commands to generate the tar file:

DUMP=jenkins_data.$YEAR-$MONTH-$DAY.tar.gz
JENKINS_HOME=/home/us/jenkins/jenkins-data
nice -n 19 tar --exclude-from jenkins_backup_exclusions -zcf $DUMP $JENKINS_HOME

Not rocket science, that. What I really need to show you is the contents of jenkins_backup_exclusions, because that’s what tells tar what to leave out of the backup. We used to back up our entire JENKINS_HOME, and then I needed to restore it and I found out the backup was 600MB and was going to take awhile to pull out of S3. Now they’re 138MB, which is not bad for 114 jobs. Here’s what we exclude:

config-history/*
jobs/*/workspace*
jobs/*/builds/*/archive
plugins/*/*
plugins/*.bak
war
cache

All of those paths are relative to JENKINS_HOME. config-history contains all the configuration history managed by the Job Config History plugin that I mentioned earlier. No need to back that up. You don’t need to back up any job’s workspace, most likely, nor any archived build artifacts. You can always download your plugins again, but keeping these would probably make complete restoration faster. They bloat the backup by 224MB in my case though. [See update below.] The war and cache directories are 69MB and 79MB, respectively, in my case. Also not necessary for a restoration. You’ll want to look at the contents of your own JENKINS_HOME and possibly add other things to the exclusions list, depending on what plugins you use and so on.

So our BASH script just runs in crontab every night, creates this tar.gz file and uploads it to S3 using Amazon’s standard Linux command line tools. The storage costs there are low enough that we can justify keeping backups indefinitely, so I can restore from any point in time. This has come in handy on a few occasions where a job configuration was broken and no one realized it for a long time.

So that’s it! Now you have no excuse to not perform backups of your Jenkins configs. Get to work! Let me know in the comments if you have any questions.

[Updated 11/18/2013]
I didn’t look closely enough at what I’m doing with the backup exceptions list, and it has been awhile since I set this up. In the case of the plugins directory, I’m only excluding the uncompressed contents of each plugin and any previous plugin versions which Jenkins keeps (so you can roll back if you find a bug). The actual plugin .jpi files are backed up, and that’s all you need to do a restoration.

[follow_me]

Bookmark the permalink.

8 Responses to Backing up your Jenkins configuration

  1. Madhu says:

    Hi
    I tired using thinBackup plugin to take backup of all jenkins jobs and required things , I am able to take backup successfully. Also tried to restore the same using restore option, able to restore all the jobs successfully , but after restore manage Jenkins option not displaying.

    Please let me know is there any solution to this issue

  2. Matan G says:

    Very interesting, will start working on it now.
    Thanks

  3. Niristotle Okram says:

    Nice article, but need to update the “jenkins_backup_exclusions” in conjunction to the later versions… Cheers!

  4. John says:

    Nice article. I tried this on Ubuntu 14, putting #!/bin/bash at the to of the backup script. The result was a 500GB file!

    The bulk of this seems to be in the dir .sdkman and .m2 directories. How do we just backup the configuration?

    Tar gave the following warning: tar: Removing leading `/’ from member names

    Note that for jenkins home we set /var/lib/jenkins. not sure if this is correct. It has .java, .grails, .m2, .subversion, .sdkman, updates, notes, users, workspace, logs, log jobs and a bunch of other stuff

    It also doesn’t get the dates in the DUMP file name: $YEAR is not defined for example.

    FYI, I tried the SCM Sync Configuration plugin 0.0.9, and it was impossible to get it to authenticate with SVN at assembla.com, even though our projects have no issues. You can switch off the message prompts.

    • Owen Mehegan says:

      It sounds like you did not create the backup_exclusions file, and thus backed up all your project workspaces. Hence the large size of your backup.

      What I listed was not a complete script, as noted. You need to add some things to set the date variables etc.

      I recommend you get assistance from someone with experience doing BASH scripting.

  5. Lance Essert says:

    Wonderful article. With all the backup options for Jenkins, this provided some much needed clarity and context for me.

Leave a Reply

Your email address will not be published. Required fields are marked *