Deploying With Ansistrano

I have a major problem. I keep waaay too many open tabs on my phone. Every time I see something interesting – via Twitter, Hacker News, or Reddit – I open the link in a tab and promise myself I will check it out more thoroughly soon.

I currently have 47 open tabs 🙁

This is only amplified by the fact I do the same on desktop. Right now, across three desktops I have over a hundred open tabs.

It’s becoming an epidemic.

Anyway, once in a while, one of these tabs becomes useful.

Recently I hit on a problem whereby I needed to deploy a bunch of individual JavaScript files – node.js scripts – to a server to be used as Rabbit MQ workers.

I had a bunch of requirements for these workers:

  • start with specific flags (–harmony-async-await )
  • restart automatically if the server reboots
  • restart if the script crashes
  • can run multiple instances

And so on.

These turned out to be the superficial problems – and I say this because there’s a tool out there that already nails this problem – PM2.

Initially I thought these would be the hard problems.

What I hadn’t banked on was how much of a royal pain in the backside it might be to deploy my node.js scripts to dev / prod / wherever.

 

My requirements are fairly straightforward – they could be solved by using rsync. However, rsync becomes unmanageable as a project grows.

There’s the issue of remembering the right command, and then duplicating the command – altering slightly – for the prod deploy.

And what if it goes wrong? Well, you have to handle that yourself.

Call me spoiled, but having become accustomed to Deployer (Matt did a fantastic job on this course btw, you should check it out), I now use that as my baseline for deployments.

I have a similar tool I use on JavaScript projects called Flightplan. It offers a decent level of functionality, but with one major issue (from my p.o.v):

It is a pain to deploy more than one directory.

Flightplan works on the assumption – as best I can tell – that you will be running your project through some webpack-style setup first, producing a dist directory which contains everything you need to boot your single page app, or whatever.

This is cool, but I needed to run many different worker scripts – all ideally from one directory.

As best I understand it, webpack allows this via it’s multiple entry options, but I’m not using webpack. Actually, I tried to use webpack but it threw out a bunch of errors right away and I gave up.

I also tried Deployer. But that didn’t work much good either. JS mixed with PHP leads to mess.

Enter Ansistrano

Ok, so all that was a very long-winded precursor to the eventual solution.

However, I felt I needed to do justice to how much I have struggled to get this thing working. It’s taken 5 hours… ouch.

Needless to say I tried to give up on getting Ansistrano at numerous times (see how I ended up at Deployer, Flightplan, webpack etc).

In the end though, I cracked it. So here goes:

Firstly, my playbook:

---
- hosts: all

  vars:
    ansistrano_deploy_to: "/var/www/your/remote/path" # server side path you want to deploy too
    ansistrano_keep_releases: 3

    ansistrano_deploy_via: "git"

    ansistrano_git_repo: "ssh://git@your.gitlab.server/your-gitlab-user/your-project.git"
    ansistrano_git_branch: "master"


  roles:
    - { role: ansistrano.deploy }

Pay special attention to the ansistrano_git_repo entry, whereby I needed to add the prefix of ssh:// to make this work. If you don’t, you will find Ansible doesn’t understand the path you are providing, and blanks it out instead :/

I guess I wasn’t the only person to notice this.

Also, note that the typical git path given by gitlab will contains colons, which need to be replaced with spaces:

git@your.gitlab.com:your-user/your-project.git

// becomes

git@your.gitlab.com/your-user/your-project.git

 

Now note, this is an Ansible issue, not an Ansistrano issue.

This should be enough to get most of the way there.

However, I hit upon another issue.

No matter what I did, all the Ansistrano managed folders were being created as root .

Since the days of yore, I have been using the same set of flags on my runs of ansible-playbook, and today I was well and truly bitten on the backside:

ansible-playbook playbook/deploy.yml -i hosts -l my-server.dev -k -K  -vvvv

Ultimately this command sees me through. I’ve started using -vvvv on every playbook run as it saves me having to re-run when things inevitably go wrong. Also, for the love of God, use snapshots before running.

But yeah, my issue was I was running with the additional flag of -s which forced the playbook to run as root. Silly me.

Anyway, early signs are promising. It all works. I just wish it hadn’t taken me so much time to figure out these problems. Hopefully though, by sharing I can save someone some hassle in future.

Upgrading to Ansible 2.1 on OSX

I have Ansible installed on a variety of machines – primarily Ubuntu, but also for setting up the virtual machines / servers I use when recording, I also have Ansible installed on OSX.

Up until today I was running Ansible v1.9.1, but then I hit on an issue with my nginx sites not creating properly (for some reason which I did not track down), and decided the best course of action would be to upgrade all the submodules I use as the basis of my playbook infrastructure.

Just to explain this a little further – when I first started using Ansible I would git clone any interesting repository to my local machine, and then take ownership of that repo into my own project. There are pros and cons to this approach:

Pros

  • I could make changes to the repos to suit my needs
  • I knew the repo contents wouldn’t change unless I changed them
  • Everything was very explicit and obvious to new people

Cons

  • I created a lot of work for myself

Yeah, ‘busy’ work such as managing an ever expanding list of other people’s projects was not a fun thing. Instead, I switched to ‘importing’ other peoples projects by way of git’s submodules, which works a treat, but there’s a drawback:

Other people have a habit of changing things.

My mistake was to blindly update all my submodules to their latest versions:

git submodule foreach 'git fetch origin; git checkout $(git rev-parse --abbrev-ref HEAD); git reset --hard origin/$(git rev-parse --abbrev-ref HEAD); git submodule update --recursive; git clean -dfx'

This worked fine, but it turned out that some of my tracked submodules now required Ansible 2 to run.

No problem, I thought, let’s just upgrade to Ansible 2.

How naive.

The first mistake I made in this process was to try and update an OSX installation of Ansible by way of using pip install ansible –upgrade.

Yeah… not so much. Plenty of errors here:

➜  ~ sudo pip install ansible --upgrade
Password:
The directory '/Users/codereview/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/codereview/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting ansible
  Downloading ansible-2.1.0.0.tar.gz (1.9MB)
    100% |████████████████████████████████| 1.9MB 576kB/s
Requirement already up-to-date: paramiko in /Library/Python/2.7/site-packages (from ansible)
Requirement already up-to-date: jinja2 in /Library/Python/2.7/site-packages (from ansible)
Requirement already up-to-date: PyYAML in /Library/Python/2.7/site-packages (from ansible)
Collecting setuptools (from ansible)
  Downloading setuptools-23.0.0-py2.py3-none-any.whl (435kB)
    100% |████████████████████████████████| 440kB 2.3MB/s
Requirement already up-to-date: pycrypto>=2.6 in /Library/Python/2.7/site-packages (from ansible)
Requirement already up-to-date: pyasn1>=0.1.7 in /Library/Python/2.7/site-packages (from paramiko->ansible)
Requirement already up-to-date: cryptography>=1.1 in /Library/Python/2.7/site-packages (from paramiko->ansible)
Requirement already up-to-date: MarkupSafe in /Library/Python/2.7/site-packages (from jinja2->ansible)
Requirement already up-to-date: cffi>=1.4.1 in /Library/Python/2.7/site-packages (from cryptography>=1.1->paramiko->ansible)
Collecting six>=1.4.1 (from cryptography>=1.1->paramiko->ansible)
  Downloading six-1.10.0-py2.py3-none-any.whl
Requirement already up-to-date: idna>=2.0 in /Library/Python/2.7/site-packages (from cryptography>=1.1->paramiko->ansible)
Requirement already up-to-date: ipaddress in /Library/Python/2.7/site-packages (from cryptography>=1.1->paramiko->ansible)
Requirement already up-to-date: enum34 in /Library/Python/2.7/site-packages (from cryptography>=1.1->paramiko->ansible)
Requirement already up-to-date: pycparser in /Library/Python/2.7/site-packages (from cffi>=1.4.1->cryptography>=1.1->paramiko->ansible)
Installing collected packages: setuptools, ansible, six
  Found existing installation: setuptools 1.1.6
    Uninstalling setuptools-1.1.6:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 736, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 742, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
    copytree(src, real_dst, symlinks=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
    raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-ymaXaK-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]

Which is honestly fair enough – this isn’t how I’d installed Ansible originally (my poor grep skills fooled me), and this led me down some dark alleys of trying to manually fudge the upgrade of dependencies I thought were at fault – pycrypto, paramiko, and others.

The command that worked was simple:

brew install ansible
➜  ~ ansible --version
ansible 2.1.0.0

Ok, so cool, up to date with Ansible, but the playbook still wouldn’t run properly.

➜  ansible-submodule git:(master) ✗ ansible-playbook -i hosts -l symfony-3-api.dev playbook/symfony-dev.yml -k -K -s
SSH password:
SUDO password[defaults to SSH password]:
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and make sure become_method is 'sudo' (default).
This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [all] *********************************************************************

TASK [setup] *******************************************************************
No handlers could be found for logger "paramiko.transport"
fatal: [symfony-3-api.dev]: UNREACHABLE! => {"changed": false, "msg": "'EntryPoint' object has no attribute 'resolve'", "unreachable": true}
	to retry, use: --limit @playbook/symfony-dev.retry

PLAY RECAP *********************************************************************
symfony-3-api.dev          : ok=0    changed=0    unreachable=1    failed=0

Quite frustrating. I knew I could SSH onto the box, so it was likely not my server at fault.

A simpler Ansible ping command helped a little:

➜  ansible-submodule git:(master) ✗ ansible symfony-3-api.dev -i hosts -m ping -vvvv
Using /Users/Shared/Development/ansible-submodule/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<192.168.1.64> ESTABLISH SSH CONNECTION FOR USER: deploy
<192.168.1.64> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/Users/codereview/.ansible/cp/ansible-ssh-%h-%p-%r 192.168.1.64 '/bin/sh -c '"'"'LANG=en_GB.UTF-8 LC_ALL=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 /usr/bin/python && sleep 0'"'"''
symfony-3-api.dev | UNREACHABLE! => {
    "changed": false,
    "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh",
    "unreachable": true
}

From here I found a helpful github issue which led me to this:

symfony-3-api.dev ansible_ssh_host=192.168.1.64 ansible_ssh_user=deploy ansible_connection=ssh

Note the inclusion of:

ansible_connection=ssh

But this needed a little further help to start working:

curl -O -L http://downloads.sourceforge.net/project/sshpass/sshpass/1.05/sshpass-1.05.tar.gz && tar xvzf sshpass-1.05.tar.gz
cd sshpass-1.05
./configure
make
sudo make install

After that, the ping and playbook command would run again.

➜  ansible-submodule git:(master) ✗ ansible symfony-3-api.dev -i hosts -m ping -vvvv
Using /Users/Shared/Development/ansible-submodule/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<192.168.1.64> ESTABLISH SSH CONNECTION FOR USER: deploy
<192.168.1.64> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=deploy -o ConnectTimeout=10 -o ControlPath=/Users/codereview/.ansible/cp/ansible-ssh-%h-%p-%r 192.168.1.64 '/bin/sh -c '"'"'LANG=en_GB.UTF-8 LC_ALL=en_GB.UTF-8 LC_MESSAGES=en_GB.UTF-8 /usr/bin/python && sleep 0'"'"''
symfony-3-api.dev | SUCCESS => {
    "changed": false,
    "invocation": {
        "module_args": {
            "data": null
        },
        "module_name": "ping"
    },
    "ping": "pong"
}

Jackpot.

How I Fixed: File is absent cannot continue

Lately I’ve been tidying up my Ansible playbook scripts ahead of a forthcoming project deployment. Lots of change, including upgrading to PHP7 across the board.

Pretty cool, and exciting, but quite a lot of work all the same.

I decided to take the opportunity to restructure my log files. I mean, that’s a good way to spend a Friday night, right? Of course it is.

The relevant part of the nginx playbook is:

    - name: "create access log file"
      file: dest="/var/log/nginx/{{ item.directory }}-access.log"
            mode=644
            state=file
            owner="{{ nginx_user }}"
            group="{{ nginx_group }}"
      with_items: "{{ website_directories }}"

It’s admittedly been a while since I’ve been regularly hands-on with my Ansible setup, so I was a little rusty. See if you can spot the error in the above.

The with_items section lives in my host_vars/my-symfony-3.dev file, and looks like:

website_directories:
  - { name: "Root", directory: "{{ website_domain_name }}" }
  - { name: "API", directory: "api.{{ website_domain_name }}" }

So, anyway the error that was being spat out was as follows:

TASK [create access log file] **************************************************
failed: [my-symfony-3.dev] (item={u'stat': {u'exists': False}, '_ansible_item_result': True, '_ansible_no_log': False, u'changed': False, 'item': {u'directory': u'my-symfony-3.dev', u'name': u'Root'}, 'invocation': {'module_name': u'stat', u'module_args': {u'checksum_algorithm': u'sha1', u'mime': False, u'get_checksum': True, u'path': u'/var/www/my-symfony-3.dev', u'follow': False, u'get_md5': True}}}) => {"failed": true, "item": {"changed": false, "invocation": {"module_args": {"checksum_algorithm": "sha1", "follow": false, "get_checksum": true, "get_md5": true, "mime": false, "path": "/var/www/my-symfony-3.dev"}, "module_name": "stat"}, "item": {"directory": "my-symfony-3.dev", "name": "Root"}, "stat": {"exists": false}}, "msg": "file (/var/log/nginx/my-symfony-3.dev-access.log) is absent, cannot continue", "path": "/var/log/nginx/my-symfony-3.dev-access.log", "state": "absent"}
failed: [my-symfony-3.dev] (item={u'stat': {u'exists': False}, '_ansible_item_result': True, '_ansible_no_log': False, u'changed': False, 'item': {u'directory': u'api.my-symfony-3.dev', u'name': u'API'}, 'invocation': {'module_name': u'stat', u'module_args': {u'checksum_algorithm': u'sha1', u'mime': False, u'get_checksum': True, u'path': u'/var/www/api.my-symfony-3.dev', u'follow': False, u'get_md5': True}}}) => {"failed": true, "item": {"changed": false, "invocation": {"module_args": {"checksum_algorithm": "sha1", "follow": false, "get_checksum": true, "get_md5": true, "mime": false, "path": "/var/www/api.my-symfony-3.dev"}, "module_name": "stat"}, "item": {"directory": "api.my-symfony-3.dev", "name": "API"}, "stat": {"exists": false}}, "msg": "file (/var/log/nginx/api.my-symfony-3.dev-access.log) is absent, cannot continue", "path": "/var/log/nginx/api.my-symfony-3.dev-access.log", "state": "absent"}

Hopefully it make it easier to Google for this for someone in the future.

Anyway, the solution will make you kick yourself. I can’t imagine that many people will make this typo, but you never know:

    - name: "create access log file"
      file: dest="/var/log/nginx/{{ item.directory }}-access.log"
            mode=644
            state=touch
            owner="{{ nginx_user }}"
            group="{{ nginx_group }}"
      with_items: "{{ website_directories }}"

    - name: "create error log file"
      file: dest="/var/log/nginx/{{ item.directory }}-error.log"
            mode=644
            state=touch
            owner="{{ nginx_user }}"
            group="{{ nginx_group }}"
      with_items: "{{ website_directories }}"

Yeah… d’oh. RTFM.

Though to be fair, I had RTFM, it was just a long time ago, and I guess I rely too heavily on IDE code completion. When it comes to Sublime + YAML, I am going to make mistakes 🙂

If this looks interesting to you, and you want to know more about Ansible, be sure to check out the tutorial series I did here at Code Review Videos.

It’s shameless self promotion, I know, but hey, it is free. And hopefully you find it useful.

Installing PHP7 on Ubuntu with Ansible

In a bid to keep updated with the latest and greatest in PHP-land, I have been slowly updating my Ansible set-up to accommodate PHP7.

Fortunately, pretty much all of the hard work has been taken care of by Jeff Geerling, with his fantastic geerlingguy/php role. A mighty big thank you is in order.

But I’m writing this as I struggled with a particular issue getting my newly updated playbook to work properly.

The issue I had was that I had the following:

---
- hosts: php_servers
  sudo: True

  tasks:
    - name: "Add repository for PHP 7.0."
      apt_repository: 
        repo="ppa:ondrej/php" 
        update_cache=yes

  roles:
    - role: geerlingguy.php
      php_version: "7.0"
      php_packages:
        - php7.0-common
        - php7.0-cli
        - php7.0-dev
        - # snip
      php_date_timezone: "Europe/London"

And whilst everything looked good, whenever I ran the playbook:

ansible-playbook playbook/symfony-dev.yml -i hosts -l test-box.dev -k -K -s

The cheeky little blighter would not run the task to add the repository before it would try and run the roles content. Needless to say, red error text appeared in quantity.

Annoyingly, I could comment out the roles section, and the task would run – but as soon as I put the role definition back in, that would always run first.

I must admit, this was a new one on me.

However, the solution is incredibly simple:

---
- hosts: php_servers
  sudo: True

  pre_tasks:
    - name: "Add repository for PHP 7.0."
      apt_repository: 
        repo="ppa:ondrej/php" 
        update_cache=yes

  roles:
    - role: geerlingguy.php
      php_version: "7.0"
      php_packages:
        - php7.0-common
        - php7.0-cli
        - php7.0-dev
        - php7.0-fpm
        - libpcre3-dev
        - php7.0-gd
        - php7.0-curl
        - php7.0-imap
        - php7.0-json
        - php7.0-opcache
        - php7.0-xml
        - php7.0-mbstring
      php_date_timezone: "Europe/London"
      php_mysql_package: php7.0-mysql
      php_fpm_daemon: php7.0-fpm
      php_webserver_daemon: nginx
      php_conf_paths:
        - /etc/php/7.0/fpm
        - /etc/php/7.0/apache2
        - /etc/php/7.0/cli
      php_extension_conf_paths:
        - /etc/php/7.0/fpm/conf.d
        - /etc/php/7.0/apache2/conf.d
        - /etc/php/7.0/cli/conf.d
      php_fpm_pool_conf_path: "/etc/php/7.0/fpm/pool.d/www.conf"

Please note: this config is still very much a work in progress for me. I found this particular set of config from this ticket on Jeff Geerling’s Ansible Role PHP repo.

Yes, there are pre_tasks and post_tasks available – new ones on me, but right there in the docs. You may need to ctrl+f to find the exact examples, as I can’t directly link to the specific section unfortunately.

Very useful anyway, and with that, my task started behaving exactly as I wanted.

Ansible is absolutely awesome. If you haven’t already done so, be sure to check out my Ansible tutorial course here at Code Review Videos. It’s completely free.