GitLab Backup


By the end of this video you will have learned how to take a GitLab backup of your important files - not just your code repositories, but your GitLab config files also.

As part of a regular GitLab backup, the gitlab-rake task won't actually back up your config files. If you have changed your GitLab config files (e.g. your /etc/gitlab/gitlab.rb) in any way, then be aware that this file, and any other files in the /etc/gitlab dir are not backed up by default.

GitLab Backup for Your Config Files

GitLab documentation - and best security practice - recommends that you do not store your configuration backup in the same place as your application data backup.

Your GitLab config files likely aren't changing that frequently. This makes it all the more likely that you will forget to back up these files regularly, and potentially end up with an out of date config should you ever be in the unfortunate position of needing to do a restore.

GitLab has the habit of becoming quite indespensible to your / your team / your organisation once it has been in use for just a short period of time.

I have been in the unfortunate position whereby my GitLab server has been lost (an unfortunate incident with a Digital Ocean droplet not coming back from scheduled hardware maintainence), and let me tell you, you will regret not taking a system config back up. I know I did.

Rooted

The tricky part here is that the /etc/gitlab directory is owned by the root user, and the root group (root:root).

Therefore, to backup this directory, you need to run the command as sudo:

# Example backup command for /etc/gitlab:
# Create a time-stamped .tar file in the current directory.
# The .tar file will be readable only to root.
sudo sh -c 'umask 0077; tar -cf $(date "+etc-gitlab-%s.tar") -C / etc/gitlab'

(Example from the official GitLab Backups documentation)

But we ideally don't want to be doing this manually, because that more than likely will mean it won't get done. Or it won't get done frequently.

The trick is to add this job to your root users crontab.

Note: this is not the same as the system crontab.

sudo crontab -e -u root

This allows you to edit your root user's crontab.

What we want to do is to create a shell script which can run a modified version of the GitLab config backup script:

sudo sh -c 'umask 0077; tar -cf /var/opt/gitlab/config-backups/$(date "+etc-gitlab-%s.tar") -C / etc/gitlab'

This will do exactly what the GitLab doc example does, with the difference being that we are specifying the output directory:

/var/opt/gitlab/config-backups/ - feel free to change it to whatever you like.

However, in our case, that directory doesn't exist yet. Let's create it:

sudo mkdir -p /var/opt/gitlab/config-backups/ && cd $_

This makes a directory (mkdir) at the given path, creating the full path if it doesn't exist (-p), and then changes to that directory: && cd $_.

Inside this directory, we need to create ourselves a new bash script:

sudo touch backup.sh && sudo vim backup.sh

This will open up the backup.sh file for us, into which copy and paste the following:

#!/bin/sh
umask 0077;
tar -cf /var/opt/gitlab/config-backups/$(date "+etc-gitlab-%s.tar") -C / etc/gitlab

Vim is not the easiest tool to use, so feel free to use nano or whatever.

Copy from the above, then inside vim press i to insert, and then use the mouse - right click, paste. That's easiest way I know.

Save by pressing esc and then typing in :wq! to write and quit.

Using vim is a little hardcore, so if anything goes wrong at any point, hit esc then type :q! to quit without saving.

Permissions

We aren't done yet.

We need to set up the correct permissions to allow our new script to be executable.

Back on the command line, type:

sudo chmod +x backup.sh

This makes our backup.sh file executable (+x).

If all goes well, you should see something like this:

you@your-server:/var/opt/gitlab/config-backups$ ls -l backup.sh 
-rw-r--r-- 1 root root 107 Sep 13 12:34 backup.sh
you@your-server:/var/opt/gitlab/config-backups$ sudo chmod +x backup.sh 
you@your-server:/var/opt/gitlab/config-backups$ ls -l backup.sh 
-rwxr-xr-x 1 root root 107 Sep 13 12:34 backup.sh

Cron Job

Finally, with our script created, we can add a new entry to our root user's crontab:

sudo crontab -e -u root

This job will execute our shell script at our given interval.

Inside the crontab, you need to paste the following:

0 1 * * 1 /var/opt/gitlab/config-backups/backup.sh

Then exit as above.

This will trigger our config files to be backed up at 1am every Monday.

If you want a different schedule, feel free to edit accordingly.

Testing Your Cron Job

If you aren't sure your cron job is configured properly, a very easy way to test this is to change the above job to:

* * * * * /var/opt/gitlab/config-backups/backup.sh

Save and exit.

Then, every minute you should get a new GitLab config backup file created. Once you are seeing that outcome, adjust your schedule accordingly.

Caution

One thing to note, as we covered in the previous video, if you are using the latest GitLab Omnibus edition then each and every time you run sudo apt-get upgrade, your GitLab instance could be potentially upgraded.

You may not be expecting this. And more importantly, your systems admin most likely won't be expecting this.

This introduces a potential risk, and if the process fails, that GitLab backup may just come in very handy indeed.

The Common or Garden Variety GitLab Backup

GitLab have some great documentation on configuring your cron to make daily backups.

I'm not going to repeat those steps here as they may well change in the future.

However, this will configure your backups to run every day, keeping a rolling 7 days worth of backups locally on your server.

From bitter experience I can tell you that if you have the misfortune for your GitLab server to die, getting access to the disk may be impossible. Storing your GitLab backup files on your disk is therefore not advised.

Ghetto Offsite Backup

What we will configure is a way of using our local machine to pull down a full copy of the server's GitLab backup directory using rsync.

I am going to assume you are wanting to pull down the files from the server to your home directory. You can change this directory to anything you like, just update the commands accordingly.

Inside our home directory, we are going to create a new folder: gitlab-backups.

cd ~
mkdir gitlab-backups && cd $_

Then we are going to test out the rsync command:

// linux
rsync -avz --progress root@your-server-ip-address:/var/opt/gitlab/backups /home/your-username/gitlab-backups

// mac
rsync -avz --progress root@your-server-ip-address:/var/opt/gitlab/backups /Users/your-username/gitlab-backups

The path to your home directory may be different depending on your OS.

Root Problems

However, because we are trying to log in to the server as root, depending on how you have set up your server - this should not be allowed.

To get round this, we need to tell the server that we allow access as root from our remote user (the desktop / laptop you are on currently) by way of adding our public SSH key to the root user's authorized_keys file.

If at this point you are unsure / unable to get your public SSH key, then follow this guide.

You will then need to get a copy of your public SSH key. From your local machine:

cat ~/.ssh/id_rsa.pub

Go back to your GitLab server:

sudo su -
cd ~
ls -la

If you see a directory called .ssh here you are good to go, otherwise follow this guide for setting up an SSH profile for your root user.

Then, if the authorized_keys file does not exist, we need to create it:

cd ~/.ssh
touch authorized_keys
chmod 600 authorized_keys
vi authorized_keys

Then insert / paste the key you got from your local id_rsa.pub file here. This will authorise you to log in as root.

You should now be able to go back to your local machine and successfully run the rsync command:

// linux
rsync -avz --progress root@your-server-ip-address:/var/opt/gitlab/backups /home/your-username/gitlab-backups

// mac
rsync -avz --progress root@your-server-ip-address:/var/opt/gitlab/backups /Users/your-username/gitlab-backups

After running this, you should have the contents of /var/opt/gitlab/backups on your local machine.

Episodes