CodeReviewVideos.com Forum

How To Use VirtualBox as a GitLab CI Test Runner

Continuous Integration is one of the most beneficial things to add to any serious project. However, setting it up can be costly in terms of time, or money, or both.


This is a companion discussion topic for the original entry at https://codereviewvideos.com/course/your-own-private-github/video/how-to-use-virtualbox-as-a-gitlab-ci-test-runner

hi i got this following error so let me know how to fix this ? ERROR: Build failed (system failure): exec: "vboxmanage": executable file not found in $PATH.

My config.toml :
[[runners]]
name = "virtualbox-runner-demo"
url = "http://gitlab-virtualbox/ci"
token = "b2ca3ae8731d7974d90c86b4906f46"
executor = "virtualbox"
[runners.ssh]
user = "krish"
password = "msys@123"
[runners.virtualbox]
base_name = "gitlab"
disable_snapshots = false
[runners.cache]

From the error message, vboxmanage needs to be included in the $PATH of the user you are trying to run the runner as. The specifics of doing this vary depending on your OS - but a google for "add vboxmanage to path {my OS name here}" should get you the help you need.

Hello @chris,

I am trying to make a gitlab server run on debian 9.5 only command line and with virtualbox installed. The vm is executed in headless mode. Even though I put the nic in NAT, the error “ERROR: Build failed: dial tcp 127.0.0.1:43342: getsockopt: connection refused” occurs. Do you know if headless mode works?

Can I point the runner to another IP that has the virtualbox installed on a machine with desktop manager?

I never tested with headless mode, but don’t see why it wouldn’t work. Can you confirm there is either no firewall, or better, a firewall is running but the port(s) are open?

Can I point the runner to another IP that has the virtualbox installed on a machine with desktop manager?

I’m not sure I follow this - what do you mean?

FWIW I have found using Docker to run my GitLab runners to be much easier than using VirtualBox.

Here is my current GitLab CI Runner docker-compose.yml config. I’m not saying this is perfect, but it’s working for me:

version: '3'

services:

  dind:
    restart: always
    privileged: true
    volumes:
    - /var/lib/docker
    image: docker:dind
    command: 
    - --storage-driver=overlay2

  runner:
    restart: always
    image: gitlab/gitlab-runner:alpine
    volumes:
    - ./gitlab/runner:/etc/gitlab-runner:Z
    environment:
    - DOCKER_HOST=tcp://dind:2375

  register-runner:
    restart: 'no'
    image: gitlab/gitlab-runner:alpine
    volumes:
    - ./gitlab/runner:/etc/gitlab-runner:Z
    command:
    - register
    - --non-interactive
    - --locked=false
    - --name={your runner name here}
    - --executor=docker
    - --docker-image=docker:dind
    - --docker-volumes=/var/run/docker.sock:/var/run/docker.sock
    environment:
    - CI_SERVER_URL=https://{your_gitlab_ci_server_here}
    - REGISTRATION_TOKEN={your_token_here}

Thanks for the answer @chris

I allow all on the firewall and now the error has changed.

root@gitlab-t:~/.ssh# gitlab-ci-multi-runner --debug run
Runtime platform                                    arch=amd64 os=linux pid=5130 revision=7f00c780 version=11.5.1
Starting multi-runner from /etc/gitlab-runner/config.toml ...  builds=0
Checking runtime mode                               GOOS=linux uid=0
Running in system-mode.                            
                                                   
Configuration loaded                                builds=0
listenaddress: ""
sessionserver:
  listenaddress: ""
  advertiseaddress: ""
  sessiontimeout: 1800
metricsserveraddress: ""
concurrent: 1
checkinterval: 0
loglevel: null
logformat: null
user: ""
runners:
- name: puppet-t
  limit: 0
  outputlimit: 0
  requestconcurrency: 0
  runnercredentials:
    url: http://10.60.0.11/
    token: e596effa5faa074ffa3f98a2247139
    tlscafile: ""
    tlscertfile: ""
    tlskeyfile: ""
  runnersettings:
    executor: virtualbox
    buildsdir: ""
    cachedir: ""
    cloneurl: ""
    environment: []
    preclonescript: ""
    prebuildscript: ""
    postbuildscript: ""
    shell: ""
    ssh:
      user: ubuntu
      password: reverse
      host: ""
      port: ""
      identityfile: ""
    docker: null
    parallels: null
    virtualbox:
      basename: puppet-t
      basesnapshot: ""
      disablesnapshots: false
    cache:
      type: ""
      path: ""
      shared: false
      s3:
        serveraddress: ""
        accesskey: ""
        secretkey: ""
        bucketname: ""
        bucketlocation: ""
        insecure: false
      gcs:
        cachegcscredentials:
          accessid: ""
          privatekey: ""
        credentialsfile: ""
        bucketname: ""
      s3cachepath: ""
      cacheshared: false
      serveraddress: ""
      accesskey: ""
      secretkey: ""
      bucketname: ""
      bucketlocation: ""
      insecure: false
    machine: null
    kubernetes: null
sentrydsn: null
modtime: 2018-12-07T18:05:15.632452227Z
loaded: true
  builds=0
Waiting for stop signal                             builds=0
Listen address not defined, metrics server disabled  builds=0
Listen address not defined, session server disabled  builds=0
Starting worker                                     builds=0 worker=0
Feeding runners to channel                          builds=0
Dialing: tcp 10.60.0.11:80 ...                     
Checking for jobs... nothing                        runner=e596effa




Feeding runners to channel                          builds=0
Checking for jobs... nothing                        runner=e596effa
Feeding runners to channel                          builds=0
Checking for jobs... received                       job=47 repo_url=http://10.60.1.171/root/puppet-t.git runner=e596effa
Failed to requeue the runner:                       builds=1 runner=e596effa
Running with gitlab-runner 11.5.1 (7f00c780)        job=47 project=1 runner=e596effa
  on puppet-t e596effa                              job=47 project=1 runner=e596effa
Shell configuration: environment: []
dockercommand:
- sh
- -c
- "if [ -x /usr/local/bin/bash ]; then\n\texec /usr/local/bin/bash --login\nelif [
  -x /usr/bin/bash ]; then\n\texec /usr/bin/bash --login\nelif [ -x /bin/bash ]; then\n\texec
  /bin/bash --login\nelif [ -x /usr/local/bin/sh ]; then\n\texec /usr/local/bin/sh
  --login\nelif [ -x /usr/bin/sh ]; then\n\texec /usr/bin/sh --login\nelif [ -x /bin/sh
  ]; then\n\texec /bin/sh --login\nelif [ -x /busybox/sh ]; then\n\texec /busybox/sh
  --login\nelse\n\techo shell not found\n\texit 1\nfi\n\n"
command: bash
arguments:
- --login
passfile: false
extension: ""
  job=47 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"--version"}  
Using VirtualBox version 5.2.22r126460 executor...  job=47 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"showvminfo", "puppet-t-runner-e596effa-concurrent-0"} 
Executing VBoxManageOutput: []string{"showvminfo", "puppet-t-runner-e596effa-concurrent-0"} 
Creating new VM...                                  job=47 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"showvminfo", "puppet-t-runner-e596effa-concurrent-0", "--machinereadable"} 
Executing VBoxManageOutput: []string{"unregistervm", "puppet-t-runner-e596effa-concurrent-0"} 
Executing VBoxManageOutput: []string{"showvminfo", "puppet-t-runner-e596effa-concurrent-0"} 
Executing VBoxManageOutput: []string{"snapshot", "puppet-t", "list", "--machinereadable"} 
Executing VBoxManageOutput: []string{"snapshot", "puppet-t", "list", "--machinereadable"} 
Creating testing VM from VM puppet-t snapshot Base State ...  job=47 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"clonevm", "puppet-t", "--mode", "machine", "--name", "puppet-t-runner-e596effa-concurrent-0", "--register", "--snapshot", "Base State", "--options", "link"} 
Identify SSH Port...                                job=47 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"showvminfo", "puppet-t-runner-e596effa-concurrent-0"} 
Creating localhost ssh forwarding...                job=47 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"list", "vms", "-l"} 
Executing VBoxManageOutput: []string{"modifyvm", "puppet-t-runner-e596effa-concurrent-0", "--natpf1", "guestssh,tcp,127.0.0.1,33673,,22"} 
Using local 33673 SSH port to connect to VM...      job=47 project=1 runner=e596effa
Bootstraping VM...                                  job=47 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"startvm", "puppet-t-runner-e596effa-concurrent-0", "--type", "headless"} 
Waiting for VM to become responsive...              job=47 project=1 runner=e596effa
Connecting to SSH...                                job=47 project=1 runner=e596effa
Feeding runners to channel                          builds=1
Appending trace to coordinator... ok                code=202 job=47 job-log=0-174 job-status=running runner=e596effa sent-log=0-173 status=202 Accepted
Submitting job to coordinator... ok                 code=200 job=47 job-status= runner=e596effa
Submitting job to coordinator... ok                 code=200 job=47 job-status= runner=e596effa
Submitting job to coordinator... ok                 code=200 job=47 job-status= runner=e596effa
Executing VBoxManageOutput: []string{"controlvm", "puppet-t-runner-e596effa-concurrent-0", "poweroff"} 
Executing VBoxManageOutput: []string{"unregistervm", "puppet-t-runner-e596effa-concurrent-0", "--delete"} 
WARNING: Preparation failed: ssh: handshake failed: read tcp 127.0.0.1:40502->127.0.0.1:33673: read: connection reset by peer  job=47 project=1 runner=e596effa
Will be retried in 3s ...                           job=47 project=1 runner=e596effa

I would like to try to resolve this issue of virtualbox and then move to docker. What can be this error?

The thing that’s confusing me here is this line:

WARNING: Preparation failed: ssh: handshake failed: read tcp 127.0.0.1:40502->127.0.0.1:33673: read: connection reset by peer job=47 project=1 runner=e596effa

Specifically:

read tcp 127.0.0.1:40502->127.0.0.1:33673

With 127.0.0.1 being your local host, I don’t follow why it’s trying to establish an SSH connection to itself. And even if this makes sense (is your GitLab runner running on the same host / VM as your GitLab server?) then why would a local SSH connection fail?

Here’s what I would try:

From your GitLab CI runner, go to the CLI of that box.

From there, establish an SSH connection to your GitLab CI Server box using the credentials you have supplied in your GitLab toml config.

At a guess, you will see a prompt for the authenticity of host ... cannot be established or similar.

Select yes to continue connecting.

After doing this once on the CLI, you should be able to do this programmatically / from your GitLab CI runner process.

If this works, you would need to either document this process (the ssh connectivity step) or more preferably, automate the secure copy of an authorised key file on initial provision - that’s more complex, but as I see your config mentions puppet, I imagine you have some facility to do this?

This post has some helpful config tips also: https://forum.gitlab.com/t/problem-configuring-gitlab-ci-ssh-handshake-failed/5815/2

Yes, it runs on the same VM.

Idk

Now, logs changed!

Waiting for VM to become responsive...              job=63 project=1 runner=e596effa
Connecting to SSH...                                job=63 project=1 runner=e596effa
Feeding runners to channel                          builds=1
Appending trace to coordinator... ok                code=202 job=63 job-log=0-174 job-status=running runner=e596effa sent-log=0-173 status=202 Accepted
Submitting job to coordinator... ok                 code=200 job=63 job-status= runner=e596effa
Submitting job to coordinator... ok                 code=200 job=63 job-status= runner=e596effa
Creating default snapshot...                        job=63 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"snapshot", "puppet-t-runner-e596effa-concurrent-0", "take", "Started"} 
Checking VM status...                               job=63 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"showvminfo", "puppet-t-runner-e596effa-concurrent-0", "--machinereadable"} 
Identify SSH Port...                                job=63 project=1 runner=e596effa
Executing VBoxManageOutput: []string{"showvminfo", "puppet-t-runner-e596effa-concurrent-0"} 
Waiting VM to become responsive...                  job=63 project=1 runner=e596effa
Starting SSH command...                             job=63 project=1 runner=e596effa
Connecting to SSH server...                         job=63 project=1 runner=e596effa
Waiting for signals...                              job=63 project=1 runner=e596effa
Executing build stage                               build_stage=prepare_script job=63 project=1 runner=e596effa
Appending trace to coordinator... ok                code=202 job=63 job-log=0-353 job-status=running runner=e596effa sent-log=174-352 status=202 Accepted
Executing build stage                               build_stage=get_sources job=63 project=1 runner=e596effa
Executing build stage                               build_stage=upload_artifacts_on_failure job=63 project=1 runner=e596effa
WARNING: Job failed: Process exited with: 1. Reason was:  ()  duration=1m22.135868757s job=63 project=1 runner=e596effa
Appending trace to coordinator... ok                code=202 job=63 job-log=0-837 job-status=running runner=e596effa sent-log=353-836 status=202 Accepted
Submitting job to coordinator... ok                 code=200 job=63 job-status= runner=e596effa
Executing VBoxManageOutput: []string{"controlvm", "puppet-t-runner-e596effa-concurrent-0", "poweroff"} 
Checking for jobs... nothing                        runner=e596effa
Feeding runners to channel                          builds=0
Checking for jobs... nothing                        runner=e596effa
Feeding runners to channel                          builds=0
Checking for jobs... nothing                        runner=e596effa
Feeding runners to channel                          builds=0
Checking for jobs... nothing                        runner=e596effa

and

Running with gitlab-runner 11.5.1 (7f00c780)
  on puppet-t e596effa
Using VirtualBox version 5.2.22r126460 executor...
Creating new VM...
Creating default snapshot...
Waiting VM to become responsive...
Starting SSH command...
stdin: is not a tty
Running on debian via gitlab-t...
stdin: is not a tty
bash: line 79: git: command not found
stdin: is not a tty
bash: line 62: cd: builds/root/puppet-t: No such file or directory
ERROR: Job failed: Process exited with: 1. Reason was:  ()

My gitlab-ci.yml

stages:
  - test

job1:
  stage: test
  tags:
  - puppet-t
  script:
  - apt-get update
  - apt-get install curl
  - curl --insecure -sL https://xxx.xxx.xxx.xxx/install_puppet_agent.sh -o /tmp/install_puppet_agent.sh

From your gitlab runner error:

bash: line 79: git: command not found

Is git installed - can you run git from the CLI?

On the gitlab server? If so, it’s installed!

root@gitlab-t:~# git --version
git version 2.11.0

I’d try making the gitlab-ci.yml script section much simpler.

Maybe replace it with something like:

ping 127.0.0.1 -c 5

If that works, you know the process works.

After that, it’s a case of debugging the steps incrementally. From what I’ve just read on Google, it could be that the git command is running somehow not using the git on your PATH, but in some script environment (not sure on that one) but that’s what I’d try next.

The same error occurs! :frowning:

I also installed git in the test VM and ran the commands to check if git was in the PATH and everything is ok.

root@gitlab-t:/home/debian# ssh root @10.60.1.236 which git-receive-pack
root@10.60.1.236’s password:
/usr/bin/git-receive-pack
root@gitlab-t:/home/debian# ssh root @10.60.1.236 ‘echo $PATH’
root@10.60.1.236’s password:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11

To confirm then:

you have a base / master VirtualBox .vbox file that when you boot up that image, it gets the IP address of 10.60.1.236, and from the CLI of that machine you can run the command git --version and see some output like:

root@10.60.1.236# git --version
git version 2.19.1

You have tried replacing the three lines in your script setup with just the ping... entry, and yet you get the same error?

This implies the script section is never reached(?) and the error is coming from the way the GitLab runner is currently configured somehow (at a guess).

These lines:

bash: line 79: git: command not found
bash: line 62: cd: builds/root/puppet-t: No such file or directory

Do they mean anything to you? Are they from a script of yours? builds/root/puppet-t - is that something you’ve set up?

I have almost no experience with puppet, so am not sure if these are common paths, or specific to your setup.

I assume you are, but to confirm, are you running the gitlab ci runner with the debug flag on to see the verbose output?

If still struggling, why not try the docker version I listed above. You need to change two environment variables, and then run docker-compose up, and that’s about it. It’s vastly simpler, and could also potentially help isolate the cause of the problem(s) you’re experiencing.

It’s a long while since I used VirtualBox for this process. I remember it being extremely difficult. It was a major part in why I switched to using Docker generally, actually. For that, I am grateful.

hello @chris!!! Thank you for your patience in helping me!!! :smiley:

This VM took this IP because I put it in bridge to test the ssh access from the gitlab server to the VM.

root@debian:~# git --version
git version 1.7.10.4

I made that change and the same error occurred.

I decided to follow your tip of using docker. I installed docker on the same gitlab server and set up a new runner.

At first it gave error again, but with docker the error had more output and I could see that the address of my project was wrong. I had configured the gitlab server with dhcp and then switched to static and set an ip. The project continued with the old IP and possibly it was this problem with the virtualbox.

I solved by following this link below.

With docker it’s okay. I’m sorry for taking your time and thank you very much for your time and willingness to help!!!

I’m going to test with virtualbox and come back here to let you know if the problem has been resolved.

Ahh cool, well I’m glad you’re on your way to resolving the issue.

As I mentioned above, I did find the VirtualBox approach to be much harder / more tricky to get working than the Docker approach, and the maintenance with having a base image file is also more cumbersome than letting Docker handle everything, imo.

CI is such a worthwhile thing to get working, but it can be sooo frustrating when it’s not working.