My GitLab Runner Config.toml [Example]

I hit on an annoying issue this week, which I’m not sure of the root cause.

Last week I bumped GitLab from 10.6, to 10.8, and somehow broke my GitLab CI Runner.

Somewhere, I have a backup of the config.toml file I was using. I run my GitLab CI Runner in a Docker container. I only run one, as it’s only for my projects. And one is enough.

Somehow, the Runner borked. And annoyingly I neither had a reference of the running version (never use :latest unless you like uncertainty), and recreating without the config.toml file has been a pain.

So for my own future reference, here is my current GitLab Runner config.toml file:

user@8818901c05c8:/# cat /etc/gitlab-runner/config.toml

concurrent = 1
check_interval = 0

  name = "runner-1"
  url = "https://my.gitlab.url"
  token = "{redacted}"
  executor = "docker"
    tls_verify = false
    image = "docker:dind"
    privileged = true
    pull_policy = "if-not-present"
    disable_cache = false
    volumes = ["/var/run/docker/sock:/var/run/docker.sock","/cache"]
    shm_size = 0
    insecure = false

FWIW this isn’t perfect. I’m hitting on a major issue currently whereby GitLab CI Pipeline stages with multiple jobs in the stage are routinely failing. It’s very frustrating. It’s also not scheduled for fix until v11, afaik.

Almost a year on

Wow, it’s almost a year since I last destroyed my personal GitLab.

Back then I was running the omnibus edition. Since then I’ve been rocking sameersbn/docker-gitlab.

Highly recommended. Love me some GitLab.

And yes, last, because I have unfortunately destroyed my GitLab three times so far. Each time is an extreme sad panda situation. I have backups, thankfully, but it still sucks.

How I Fixed: “error authorizing context: authorization token required”

I love me some Dockerised GitLab. I have the full CI thing going on, with a private registry for all my Docker images that are created during the CI process.

It all works real nice.

Until that Saturday night, when suddenly, it doesn’t.

Though it sounds like I’m going off on a tangent, it’s important to this story that you know I recently I changed my home broadband ISP.

I host one of my GitLab instances at my house. All my GitLab instances are now Dockerised, managed by Rancher.

I knew that as part of switching ISPs, there might (read: 100% would) be “fun” with firewalls, and ports, and all that jazz.

I thought I’d got everything sorted, and largely, I had.

Except I decided that whilst all this commotion was taking place, I would slightly rejig my infrastructure.

I use LetsEncrypt for SSL. I use the LetsEncrypt certs for this particular GitLab’s private registry.

I had the LetsEncrypt container on one node, and I was accessing the certs via a file share. It seemed pointless, and added complexity (the afore mentioned extra firewall rules), which I could remove if I moved the container on to the same box as the GitLab instance.

I made this move.

Things worked, and I felt good.

Then, a week or so later, I made some code changes and pushed.

The build failed almost immediately. Not what I needed on a Saturday night.

In the build logs I could see this:

Error response from daemon: Get https://my.gitlab:5000/v2/: received unexpected HTTP status: 500 Internal Server Error

This happened when the CI process was trying to log in to the private registry.

After a bit of head scratching, I tried from my local machine and sure enough I got the same message.

My Solution

As so many of my problems seem to, it boiled down to permissions.

Rather than copy the certs over from the original node, I let LetsEncrypt generate some new ones. Why not, right?

This process worked.

The GitLab and the Registry containers used a bind mounted volume to access the LetsEncrypt cert inside the container on the path /certs/.

When opening each container, I would be logged in as root.

Root being root, I had full permissions. I checked each file with a cheeky cat and visually confirmed that all looked good.

GitLab doesn’t run as root, however, and as the files were owned by root, and had 600 permissions:

Completed 500 Internal Server Error in 125ms (ActiveRecord: 7.2ms)
Errno::EACCES (Permission denied @ rb_sysopen - /certs/privkey.pem):
lib/json_web_token/rsa_token.rb:20:in `read'
lib/json_web_token/rsa_token.rb:20:in `key_data'
lib/json_web_token/rsa_token.rb:24:in `key'
lib/json_web_token/rsa_token.rb:28:in `public_key'
lib/json_web_token/rsa_token.rb:33:in `kid'
lib/json_web_token/rsa_token.rb:12:in `encoded'

The user GitLab is running as doesn’t have permission to read the private key.

Some more error output that may help future Googlers:

21/01/2018 21:31:51 time="2018-01-21T21:31:51.048129504Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.7.6"my.gitlab:5000" http.request.method=GET http.request.remoteaddr="" http.request.uri="/v2/" http.request.useragent="docker/17.12.0-ce go/go1.9.2 git-commit/d97c6d6 kernel/4.4.0-109-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.12.0-ce (linux))" service=registry version=v2.6.2
21/01/2018 21:31:5110.42.16.142 - - [21/Jan/2018:21:31:51 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/17.12.0-ce go/go1.9.2 git-commit/d97c6d6 kernel/4.4.0-109-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.12.0-ce (linux))"


Thankfully I hadn’t deleted the old cert, so I went back and saw that I had previously set 0640  on the private key in the old setup.

Directory permissions for the certs was set to 0750 with execute being required as well as read.

In my case this was sufficient to satisfy GitLab.

When making the change on the new node, I could then immediately log back in.

A Tip To Spot This Sooner

I would strongly recommend that you schedule your project to run a build every ~24 hours, even if nothing has changed.

This will catch weird quirks that aren’t related to your project, but have inadvertently broken your project’s build.

It’s much easier to diagnose problems whilst they are fresh in your mind.

Also, ya’ know, better documentation! This is exactly why I’m now writing this post. So in the future when I inevitable make a similar mistake, I now know where to look first 🙂

Docker Tutorial For Beginners (Update)

I’ve been super busy this week behind the scenes working on the new site version. Typically, every time I think I’m done (we all know we are never done :)), I hit upon something I’ve either missed, or overlooked.

There are a bunch of CSS related jobs:

Yeah… if you could just fix that margin-top, that’d be great

And React seems to have completely broken my mailing list sign up forms:

You can’t tell from a static image, but these inputs don’t allow input 🙂

And some things I only noticed when testing my user journeys:

What, no remember me?

One thing I am really happy with is the new Profile panel:

4 tabs, a lot of work 🙂

There are 29 open tickets left before I consider this next version done (again, never really done :)).

I reckon I will be happy enough to open the beta when I’m down to 20 tickets. I genuinely wish I was a 10x developer, able to blitz stuff like this out in a weekend.

I’m really looking forward to switching over though. This new site is so much nicer to work with from a development point of view. And I have a bunch of improvements from a member / visitor point of view, too 🙂

Both the front end and back end are completely separated. Both are pushed through a GitLab CI pipeline, which automates the test runs, and if all goes to plan, builds a Docker image for both. I can then deploy these using Rancher.

It’s a great setup.

As you may have noticed, Docker is currently the main course topic on the site. This is a Docker tutorial for beginners. If you’ve ever wanted to learn Docker, and prefer to see real world setups rather than simplified examples then this is the right starting point for you.

Which brings me neatly on to this weeks…

Video Update

This week saw three new videos added to the site.

All of this week’s videos fall under the same theme:

Long, and short term storage of data in Docker.

One of the most confusing aspects of switching my stack from Virtual Machines to Docker was what was happening to my data?

In Virtual Machine-land (if such a place ever existed) the location of my data was fairly obvious. It would either be on the VM itself (on the virtual hard disk), or on some attached network drive.

Things are roughly the same in Docker, but it’s not a 100% like-for-like experience. Especially not on a Mac.

#1. Docker, Without Volumes

Are volumes essential for persistent storage in Docker?

It can sometimes seem that way. A lot of Docker tutorials I followed seemed to equate persisting data with Docker Volumes.

In this video we explore persisting data within Docker containers without Volumes.

Yes, it can be done.

Yes, it works just fine.

And yes, you will lose your data if you delete the container your data belongs too 🙂

So don’t fall in to that trap. Watch this video and learn how to avoid this mistake.

#2. [Part 1] – Docker Volumes – Bind Mounts

I’m going out on a limb here, and I’m going to guess you love a good Star Trek episode as much as me.

Nothing troubles my tribbles quite like a blurt of technobabble:

The temporal surge we detected was caused by an explosion of a microscopic singularity passing through this solar system. Somehow, the energy emitted by the singularity shifted the chroniton particles in our hull into a high state of temporal polarisation.

Now the good news, when watching Star Trek, is that you really don’t need to understand any of that to gain enjoyment from the episode. The writers make sure the characters appear to understand it, and the Federation once again, save the day.

But what about technobabble we do have to understand?

Docker Bind Mounts.

Well, in this video we run our tricorder over bind mounts and figure out just what the heck these things are.

The good news?

Beyond the name they are surprisingly straightforward.

Set tachyon cannons to maximum dispersion!

#3. [Part 2] – Docker Volumes – Volumes

Here’s the unusual thing:

Docker bind mounts sounds complicated, but as we saw in the previous video, they really aren’t.

Docker volumes, on the other hand, sound quite straightforward.

But they are more complicated.

Docker’s Volumes are the future of the way we work with data in Docker.

Therefore, understanding Volumes is a very useful skill to have.

Fortunately, even though Volumes are more involved than bind mounts, by watching the previous two videos you have all the pre-requisite information required to grasp this concept.

In this video we work through a bunch of examples to ensure the fundamentals are in place. Volumes will feature heavily throughout the rest of our dealings with Docker, so be sure to understand this section before moving on.

That’s all from me for this week. Until next week, have a great weekend, and happy coding.


How I solved “New runner. Has not connected yet” in GitLab CI

gitlab-brown-triangle-of-doomI hit on an issue with GitLab CI whereby newly added GitLab CI Runners would register successfully, but never start a build.

The error message was shown on the Projects > Settings as a brown triangle with the runner’s ID hash next to it.

The runner itself seemed to be active, but it had the message: “New runner. Has not connected yet” next to it.

Frustrated, I re-installed, re-installed again, tried doing a gitlab-ci-multi-runner register AND a sudo gitlab-ci-multi-runner register which only exacerbated the problem.

The Problem

Firstly, note that running a :

gitlab-ci-multi-runner register

does something different to:

sudo gitlab-ci-multi-runner register

Both will create a config.toml file which may not be immediately obvious.

Normally when you run a command without sudo and it doesn’t work, you would expect it to fail and give a helpful error message.

This is a case of GitLab being flexible, but confusing.

gitlab-ci-multi-runner register will create a file in your home directory called config.toml.

sudo gitlab-ci-multi-runner register will create the file: /etc/gitlab-runner/config.toml.

My Solution to “New runner. Has not connected yet”

In my case, the solution was to delete the config.toml in my home directory. That’s the home directory of the user who I was ssh‘d into the GitLab CI Runner instance.

As soon as I did this, the GitLab runner immediately started – no further intervention required.

If your’s doesn’t, my next course of action would be to delete both config.toml files, remove any references to that runner from GitLab CI Runner dashboard, and then re-run the command:

sudo gitlab-ci-multi-runner register