Docker Isn't Always Plain Sailing

Mixing concerns rarely leads to happy results.

Right now, when the rancher-cluster.yml file is created, it ends up in the Ansible directory.

This creates confusion.

By the end of this video we will have:

• Created a new directory to hold Rancher specific config
• Used our Ansible setup to create files in this new directory

It doesn't sound like much, but in practice it's not super straightforward, either.

New Directory For Rancher Config

As it stands in the current Ansible directory, we have two very similar sounding filenames:

• rke-cluster.yml
• rancher-cluster.yml

These both relate to Rancher, and are both related to each other. In fact, one is directly responsible for the creation of the other.

This is confusing. Even with a small number of files in our Ansible project, it's messy.

I'm going to create a new directory to hold just the Rancher specific config.

# from the Ansible directory

mkdir ../rancher-2

Easy enough. You can use any directory structure you like. In my case it's easiest to go one level up, so both Ansible and Rancher configs live at the same level in my disk's directory structure.

In here, I'm going to start with a Makefile:

cd ../rancher-2
touch Makefile

Spellbinding stuff, I'm sure.

To this file I'm adding the following:

.PHONY: create_rancher_cluster_yaml

create_rancher_cluster_yaml: 
    @docker run --rm -it \
        -v $(CURDIR)/rancher-cluster.yml:/crv-ansible/rancher-cluster.yml \
        -v $(CURDIR)/../ansible:/crv-ansible \
        -w /crv-ansible \
        williamyeh/ansible:alpine3 \
        ansible-playbook -i production.yml rke-cluster.yml -vvv

Essentially:

• Map a file called rancher-cluster.yml from the current directory to the path of /crv-ansible/rancher-cluster.yml inside the resulting container.
• Map a relative directory, one level up from the current directory then down into ansible directory, to the path of /crv-ansible inside the resulting container
• Only run the rke-cluster.yml playbook

And this should run as a command, but it won't work as intended.

After running we end up with the following:

tree .
├── Makefile
└── rancher-cluster.yml
    └── rancher-cluster.yml.j2

1 directories, 2 files

There are two immediate issues. The directory name is wrong. And the output filename has the j2 extension. It's all a bit odd. Yet the file contents are correct. So that's something.

There is a hidden third issue, which is the real gotcha. We'll get back to that in a moment.

What's going on here is that when we run the command, the file rancher-cluster.yml does not exist locally. When we specify a volume to bind with a path that doesn't exist, Docker - counter intuitively - will assume we mean we want a directory there, and then it mounts that directory into the container.

Though why the file ends up with a .j2 extension is beyond me at this point.

Even though we ideally don't want to, we might think - "ok, let's create the rancher-cluster.yml file before running the make task".

And whilst that's a good shout, it doesn't quite work:

sudo rm -rf ./rancher-cluster.yml \
  && touch rancher-cluster.yml \
  && make create_rancher_cluster_yaml

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/home/chris/Development/docker/rancher-2/rancher-cluster.yml\\\" to rootfs \\\"/var/lib/docker/overlay2/e6fe304d666971a9a2332a9f070f64b0dc81e66ff1ca3e3b98bb045cd10b517d/merged\\\" at \\\"/var/lib/docker/overlay2/e6fe304d666971a9a2332a9f070f64b0dc81e66ff1ca3e3b98bb045cd10b517d/merged/crv-ansible/rancher-cluster.yml\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
Makefile:5: recipe for target 'create_rancher_cluster_yaml' failed
make: *** [create_rancher_cluster_yaml] Error 125

Meh.

Even if that had worked, it's not ideal. We don't want to have to remember to first, manually create a file with a specific name, then run a command. One command to make everything work, that's the easy way.

But isn't this error a little odd?

Are you trying to mount a directory onto a file (or vice-versa)?

Errrm, no?

We explicitly created a new, empty file. Why does Docker believe this is a directory?

Let's tidy this up:

sudo rm -rf ./rancher-cluster.yml && rm rancher-cluster.yml

Remember that third hidden issue?

Hidden (in plain sight) in the ansible directory is the cause of our problems:

➜  ansible git:(master) ✗ ls -la

total 100
drwxrwxr-x   7 chris chris 4096 Feb  9 12:38 .
drwxr-xr-x 106 chris chris 4096 Feb  9 11:39 ..
...
drwxr-xr-x   2 root  root  4096 Feb  9 12:38 rancher-cluster.yml
...

I've removed the rest for brevity.

Docker has also created the rancher-cluster.yml directory here. It's empty, but it's there.

And so every time we try to run the container, that path is conflicting with the one we really want.

Lets remove it:

sudo rm -rf ./rancher-cluster.yml

That left over file issue will remain until addressed. We shall have to do that inside our codereviewvideos.rke_cluster role.

This gives us a couple of issues to address:

• We need to create a new, empty rancher-cluster.yml file before running the Docker command
• We need to tidy up that third issue after a Docker / Ansible playbook run

Doing all this in a one line Makefile command would be tricky. Thankfully, we can write a small shell script and call that from our Makefile. This gives us the best of both worlds.

# from the Rancher dir

mkdir bin
touch bin/create_rancher_cluster_yaml.sh
# make the script executable
chmod +x bin/create_rancher_cluster_yaml.sh

And into that file:

#!/bin/sh

ANSIBLE_DIR=$(pwd)/../ansible

# Create the file if it doesn't exist
touch rancher-cluster.yml

docker run --rm -it \
        -v $(pwd)/rancher-cluster.yml:/crv-ansible/rancher-cluster.yml \
        -v ${ANSIBLE_DIR}:/crv-ansible \
        -w /crv-ansible \
        williamyeh/ansible:alpine3 \
        ansible-playbook -i production.yml rke-cluster.yml -vvv

# Remove the unwanted, empty left over directory created by Docker 
rm -f ${ANSIBLE_DIR}/rancher-cluster.yml

Watch the video for a better understanding on how we get to that.

Now update the Makefile:

create_rancher_cluster_yaml: 
    bin/create_rancher_cluster_yaml.sh

We're almost there.

If we run make create_rancher_cluster_yaml then we still hit a fairly unhelpful error:

TASK [codereviewvideos.rke_cluster : Create the rancher-cluster.yml file] ************************************************************************************

fatal: [127.0.0.1 -> localhost]: FAILED! => {
    "changed": false, 
    "checksum": "21dd8b64085260df7294a101cbb38733182c5dae", 
    "diff": [], 
    "invocation": {
        "module_args": {
            "_original_basename": "rancher-cluster.yml.j2", 
            "attributes": null, 
            "backup": false, 
            "checksum": "21dd8b64085260df7294a101cbb38733182c5dae", 
            "content": null, 
            "delimiter": null, 
            "dest": "/crv-ansible/rancher-cluster.yml", 
            "directory_mode": null, 
            "follow": false, 
            "force": true, 
            "group": null, 
            "local_follow": null, 
            "mode": null, 
            "owner": null, 
            "regexp": null, 
            "remote_src": null, 
            "selevel": null, 
            "serole": null, 
            "setype": null, 
            "seuser": null, 
            "src": "/root/.ansible/tmp/ansible-tmp-1549717720.69-12008923366637/source", 
            "unsafe_writes": null, 
            "validate": null
        }
    }, 
    "msg": "Unable to make /root/.ansible/tmp/ansible-tmp-1549717720.69-12008923366637/source into to /crv-ansible/rancher-cluster.yml, failed final rename from /crv-ansible/.ansible_tmpl4qAStrancher-cluster.yml: [Errno 16] Resource busy"
}
    to retry, use: --limit @/crv-ansible/rke-cluster.retry

PLAY RECAP ***************************************************************************************************************************************************
127.0.0.1                  : ok=1    changed=0    unreachable=0    failed=1   

On the plus side, our shell script removed the left over / unwanted ansible/rancher-cluster.yml directory.

In order to fix this we need to use the fun, and not at all worryingly titled unsafe_writes parameter:

# roles/codereviewvideos.rke_cluster/tasks/main.yml

---
- name: Create the rancher-cluster.yml file
  local_action:
    module: template
    src: rancher-cluster.yml.j2
    dest: /crv-ansible/rancher-cluster.yml
    unsafe_writes: yes

I think that's essential because we're using Docker. I haven't been able to find a better way around it.

Anyway, that's good enough to get everything working as intended. We should now have a populated rancher-cluster.yml file in the current directory. Safely extracted from Ansible, yet using it like the workhorse it is. Yeehaw.

We will use this config file in the very next video to deploy Kubernetes with RKE.