Category: How I Fixed

I have Ansible installed on a variety of machines – primarily Ubuntu, but also for setting up the virtual machines / servers I use when recording, I also have Ansible installed on OSX.

Up until today I was running Ansible v1.9.1, but then I hit on an issue with my nginx sites not creating properly (for some reason which I did not track down), and decided the best course of action would be to upgrade all the submodules I use as the basis of my playbook infrastructure.

Just to explain this a little further – when I first started using Ansible I would git clone any interesting repository to my local machine, and then take ownership of that repo into my own project. There are pros and cons to this approach:

Pros

• I could make changes to the repos to suit my needs
• I knew the repo contents wouldn’t change unless I changed them
• Everything was very explicit and obvious to new people

Cons

• I created a lot of work for myself

Yeah, ‘busy’ work such as managing an ever expanding list of other people’s projects was not a fun thing. Instead, I switched to ‘importing’ other peoples projects by way of git’s submodules, which works a treat, but there’s a drawback:

Other people have a habit of changing things.

My mistake was to blindly update all my submodules to their latest versions:

git submodule foreach 'git fetch origin; git checkout $(git rev-parse --abbrev-ref HEAD); git reset --hard origin/$(git rev-parse --abbrev-ref HEAD); git submodule update --recursive; git clean -dfx'

This worked fine, but it turned out that some of my tracked submodules now required Ansible 2 to run.

No problem, I thought, let’s just upgrade to Ansible 2.

How naive.

The first mistake I made in this process was to try and update an OSX installation of Ansible by way of using pip install ansible –upgrade.

Yeah… not so much. Plenty of errors here:

➜  ~ sudo pip install ansible --upgrade
Password:
The directory '/Users/codereview/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/codereview/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting ansible
  Downloading ansible-2.1.0.0.tar.gz (1.9MB)
    100% |████████████████████████████████| 1.9MB 576kB/s
Requirement already up-to-date: paramiko in /Library/Python/2.7/site-packages (from ansible)
Requirement already up-to-date: jinja2 in /Library/Python/2.7/site-packages (from ansible)
Requirement already up-to-date: PyYAML in /Library/Python/2.7/site-packages (from ansible)
Collecting setuptools (from ansible)
  Downloading setuptools-23.0.0-py2.py3-none-any.whl (435kB)
    100% |████████████████████████████████| 440kB 2.3MB/s
Requirement already up-to-date: pycrypto>=2.6 in /Library/Python/2.7/site-packages (from ansible)
Requirement already up-to-date: pyasn1>=0.1.7 in /Library/Python/2.7/site-packages (from paramiko->ansible)
Requirement already up-to-date: cryptography>=1.1 in /Library/Python/2.7/site-packages (from paramiko->ansible)
Requirement already up-to-date: MarkupSafe in /Library/Python/2.7/site-packages (from jinja2->ansible)
Requirement already up-to-date: cffi>=1.4.1 in /Library/Python/2.7/site-packages (from cryptography>=1.1->paramiko->ansible)
Collecting six>=1.4.1 (from cryptography>=1.1->paramiko->ansible)
  Downloading six-1.10.0-py2.py3-none-any.whl
Requirement already up-to-date: idna>=2.0 in /Library/Python/2.7/site-packages (from cryptography>=1.1->paramiko->ansible)
Requirement already up-to-date: ipaddress in /Library/Python/2.7/site-packages (from cryptography>=1.1->paramiko->ansible)
Requirement already up-to-date: enum34 in /Library/Python/2.7/site-packages (from cryptography>=1.1->paramiko->ansible)
Requirement already up-to-date: pycparser in /Library/Python/2.7/site-packages (from cffi>=1.4.1->cryptography>=1.1->paramiko->ansible)
Installing collected packages: setuptools, ansible, six
  Found existing installation: setuptools 1.1.6
    Uninstalling setuptools-1.1.6:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 736, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 742, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]

Which is honestly fair enough – this isn’t how I’d installed Ansible originally (my poor grep skills fooled me), and this led me down some dark alleys of trying to manually fudge the upgrade of dependencies I thought were at fault – pycrypto, paramiko, and others.

The command that worked was simple:

brew install ansible
➜  ~ ansible --version
ansible 2.1.0.0

Ok, so cool, up to date with Ansible, but the playbook still wouldn’t run properly.

➜  ansible-submodule git:(master) ✗ ansible-playbook -i hosts -l symfony-3-api.dev playbook/symfony-dev.yml -k -K -s
SSH password:
SUDO password[defaults to SSH password]:
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default).
This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
No handlers could be found for logger "paramiko.transport"
fatal: [symfony-3-api.dev]: UNREACHABLE! => {"changed": false, "msg": "'EntryPoint' object has no attribute 'resolve'", "unreachable": true}
	to retry, use: --limit @playbook/symfony-dev.retry

PLAY RECAP *********************************************************************
symfony-3-api.dev          : ok=0    changed=0    unreachable=1    failed=0

Quite frustrating. I knew I could SSH onto the box, so it was likely not my server at fault.

A simpler Ansible ping command helped a little:

➜  ansible-submodule git:(master) ✗ ansible symfony-3-api.dev -i hosts -m ping -vvvv
Using /Users/Shared/Development/ansible-submodule/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<192.168.1.64> ESTABLISH SSH CONNECTION FOR USER: deploy
<192.168.1.64> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/Users/codereview/.ansible/cp/ansible-ssh-%h-%p-%r 192.168.1.64 '/bin/sh -c '"'"'LANG=en_GB.UTF-8 LC_ALL=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 /usr/bin/python && sleep 0'"'"''
symfony-3-api.dev | UNREACHABLE! => {
    "changed": false,
    "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh",
    "unreachable": true
}

From here I found a helpful github issue which led me to this:

symfony-3-api.dev ansible_ssh_host=192.168.1.64 ansible_ssh_user=deploy ansible_connection=ssh

Note the inclusion of:

ansible_connection=ssh

But this needed a little further help to start working:

curl -O -L http://downloads.sourceforge.net/project/sshpass/sshpass/1.05/sshpass-1.05.tar.gz && tar xvzf sshpass-1.05.tar.gz
cd sshpass-1.05
./configure
make
sudo make install

After that, the ping and playbook command would run again.

➜  ansible-submodule git:(master) ✗ ansible symfony-3-api.dev -i hosts -m ping -vvvv
Using /Users/Shared/Development/ansible-submodule/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<192.168.1.64> ESTABLISH SSH CONNECTION FOR USER: deploy
<192.168.1.64> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/Users/codereview/.ansible/cp/ansible-ssh-%h-%p-%r 192.168.1.64 '/bin/sh -c '"'"'LANG=en_GB.UTF-8 LC_ALL=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 /usr/bin/python && sleep 0'"'"''
symfony-3-api.dev | SUCCESS => {
    "changed": false,
    "invocation": {
        "module_args": {
            "data": null
        },
        "module_name": "ping"
    },
    "ping": "pong"
}

Jackpot.