Part 1/3 - Deploying with git

Our second deploy process is one that is often requested, but for me personally, not one I would advise:

Deploying with Git.

Git is fantastic.

I use it daily, and in my opinion, it's essential that you know the basics if you are getting paid to write code.

Here are two courses to get familiar, if you are not already:

• Beginners Guide To Git
• Git Goodness

Git is all about Version Control.

It is not all about deployment.

You can deploy your code with Git, but I don't recommend it. I would strongly urge you to consider the rsync approach, or more preferably, a tool-based rsync approach, such as Deployer.

Personally I find git-based deploy processes to be harder to setup and maintain, and generally more complicated than simply copying files from A-to-B. Your opinion may differ, and that's ok.

But I get asked about this so frequently, and it was heavily requested in the build up to recording this series, so here we go:

Deploying with Git

To begin with you should have a pre-configured LAMP or LEMP stack. If you have been following along with previous videos, please ensure you are working from a fresh build as changes to the initial Apache or nginx configurations may have occurred, which may lead you to incorrect configurations.

The high level overview of proceedings is as follows:

• We work with Git on our local machine, committing code as normal
• We setup a remote repository on our Production server
• We add the remote repository to our local Git setup
• We git push to the remote whenever we want to deploy some code to prod

We will need to setup the remote repository. We will also need to ensure we get our permissions sorted out. And we will need to write at least one Git Hook script to automate part of the deployment. A Hook is simply a point along the typical git work flow into which we can 'hook' into, to make our own customisations to the process.

Much like our initial excursions into deployment with rsync we are going to have some Symfony-specific permissions issues to address.

Local Setup

As we cloned the demo project from the GitHub repository we should already have git setup in our project's local working directory.

If you are using your own project, to deploy with git kinda implies that you are also already using git. If not:

cd {your project}
git init
git add .
git commit "initial commit"

That's about it for our initial local setup.

Remote / Production Server Setup

The following steps take place on our remote / production server:

ssh you@your-server

# e.g

ssh root@104.236.215.199

(Remember the precaution of using root - we are only doing so because this is a demo exercise, do not do this in production / real life)

There are two dependencies we will need to make this process work:

• Composer
• Git

You need to have both composer and git installed on your remote server for this process to work.

Make sure to globally install composer, if not done so already.

And to install git:

sudo apt-get install git

As it stands we have a directory created to hold our website's code:

cd /var/www

ls -la

total 16
drwxr-xr-x  4 root     root     4096 Nov 18 10:59 .
drwxr-xr-x 14 root     root     4096 Nov 11 09:41 ..
drwxr-xr-x  2 www-data www-data 4096 Nov 18 10:59 crvfakeexample.com
drwxr-xr-x  2 root     root     4096 Nov 11 09:41 html

Immediately we need to sort out a permissions issue here.

We're going to:

sudo chown -R $(whoami):www-data crvfakeexample.com

Which in my case:

sudo chown -R $(whoami):www-data crvfakeexample.com

ls -la

total 16
drwxr-xr-x  4 root  root     4096 Nov 18 10:59 .
drwxr-xr-x 14 root  root     4096 Nov 11 09:41 ..
drwxr-xr-x  2 root  www-data 4096 Nov 18 10:59 crvfakeexample.com
drwxr-xr-x  2 root  root     4096 Nov 11 09:41 html

You can, of course, switch out $(whoami) for a specific username. All this command does is replace the placeholder with your currently logged in username.

The permissions change here is important as we will be copying data in as our own user, but the webserver user still needs to be able to access our files in order to serve them to our visitors. The webserver user in our case is www-data, which is a member of the www-data group. This may be different if not using Ubuntu.

Bare Repository

Right alongside the crvfakeexample.com directory, I am going to create a new directory called crvfakeexample.git. You may choose to place this new directory anywhere that you like, but for simplicity I am storing mine alongside the existing directory:

sudo mkdir /var/www/crvfakeexample.git

ls -la

total 20
drwxr-xr-x  5 root  root     4096 Nov 18 11:07 .
drwxr-xr-x 14 root  root     4096 Nov 11 09:41 ..
drwxr-xr-x  2 root  www-data 4096 Nov 18 10:59 crvfakeexample.com
drwxr-xr-x  2 root  root     4096 Nov 18 11:07 crvfakeexample.git
drwxr-xr-x  2 root  root     4096 Nov 11 09:41 html

We needed to use sudo make a directory in /var/www due to root owning this directory. However, in using sudo our new directory is owned by root also, so let's change this so we can work with our new directory without issue:

sudo chown -R $(whoami):$(whoami) crvfakeexample.git

# or, in my case

sudo chown -R root:root crvfakeexample.git

Which leads to:

ls -la

total 20
drwxr-xr-x  5 root  root     4096 Nov 18 11:07 .
drwxr-xr-x 14 root  root     4096 Nov 11 09:41 ..
drwxr-xr-x  2 root  www-data 4096 Nov 18 10:59 crvfakeexample.com
drwxr-xr-x  2 root  root     4096 Nov 18 11:07 crvfakeexample.git
drwxr-xr-x  2 root  root     4096 Nov 11 09:41 html

Ok, now inside this new crvfakeexample.git directory we need to initialise a bare git repository:

cd crvfakeexample.git

git init --bare

ls -la

total 40
drwxr-xr-x 7 root  root 4096 Nov 18 11:10 .
drwxr-xr-x 5 root  root  4096 Nov 18 11:07 ..
drwxrwxr-x 2 root  root 4096 Nov 18 11:10 branches
-rw-rw-r-- 1 root  root   66 Nov 18 11:10 config
-rw-rw-r-- 1 root  root   73 Nov 18 11:10 description
-rw-rw-r-- 1 root  root   23 Nov 18 11:10 HEAD
drwxrwxr-x 2 root  root 4096 Nov 18 11:10 hooks
drwxrwxr-x 2 root  root 4096 Nov 18 11:10 info
drwxrwxr-x 4 root  root 4096 Nov 18 11:10 objects
drwxrwxr-x 4 root  root 4096 Nov 18 11:10 refs

A bare git repository is simply the contents of the normal, hidden .git folder, but in your current directory.

To put it another way, when we do a git init in a typical directory, git will create us a hidden .git directory and put the newly created contents into that .git directory.

When using --bare, the directory structure is created in the current directory. No hidden .git folder.

We use a bare repository as no one will ever work directly from this git repository. A bare repository doesn't contain any checked out copies of our code (the working tree), which means we shouldn't ever get conflicts when pushing code changes up to this repository.

Can We Push Up Some Code Already?

At this point we could add our new repository as a remote to our local repository.

However, if we do, and we then git push up some code, not a lot will happen.

We need to add in some automation.

The plan is that when we push up some code from local to remote, git will automatically run a script to checkout our latest code to /var/www/crvfakeexample.com, run a composer install, and ensure our permissions are sorted.

Why do we need to run a composer install?

Well, take a look at the standard .gitignore file for our Symfony project:

# on your local machine

cat .gitignore

/app/config/parameters.yml
/build/
/node_modules/
/phpunit.xml
/.php_cs
/var/*
!/var/cache
/var/cache/*
!var/cache/.gitkeep
!/var/data
!/var/logs
/var/logs/*
!var/logs/.gitkeep
!/var/sessions
/var/sessions/*
!var/sessions/.gitkeep
!var/SymfonyRequirements.php
/vendor/
/web/bundles/
/web/build/fonts/glyphicons-*
/web/build/images/glyphicons-*
/.idea/*

One of the entries is /vendor.

We don't commit our third party dependency code into git. Therefore when we use git to deploy, the /vendor directory isn't going to be copied across. Likewise, anything else in the .gitignore file is going to be ignored, too.

Adding Automation Via Git Hooks

We will need a small bash script to automate some important parts of our deployment process.

# on your server

touch /var/www/crvfakeexample.git/hooks/post-receive
chmod +x /var/www/crvfakeexample.git/hooks/post-receive

There's two important points here:

1. Spelling is everything. It must be post-receive, not post-recieve!
2. post-receive must be eXecutable (chmod +x)

Once the post-receive file is created and has the correct permissions, we can edit the file:

vim /var/www/crvfakeexample.git/hooks/post-receive

And add the following contents:

#!/bin/bash

export GIT_WORK_TREE=/var/www/crvfakeexample.com
export GIT_DIR=/var/www/crvfakeexample.git
export SYMFONY_ENV=prod

git --work-tree=$GIT_WORK_TREE --git-dir=$GIT_DIR checkout -f master

cd $GIT_WORK_TREE

composer install --no-dev --optimize-autoloader

php bin/console cache:clear --env=prod --no-debug --no-warmup
php bin/console cache:warmup --env=prod

At the very top of the file we have #!/bin/bash, typically pronouncing the #! as "hash bang". This simply tells Linux that this is a bash script.

We start proper by setting / exporting three environment variables: GIT_WORK_TREE, GIT_DIR, and SYMFONY_ENV.

These environment variable names are not arbitrary. They are those expected by git / git naming convention, and Symfony respectively.

The values used are hopefully self explanatory based on what we have already covered above.

The most technically involved / potentially confusing part of the process happens next:

git --work-tree=$GIT_WORK_TREE --git-dir=$GIT_DIR checkout -f master

We could extend this out:

git --work-tree=/var/www/crvfakeexample.com --git-dir=/var/www/crvfakeexample.git checkout -f master

Normally when checking out a new branch we wouldn't need to provide --work-tree, nor --git-dir. Git would figure these out using defaults, which would likely be our project's root directory for --work-tree, and the .git subdirectory in our project's root directory for --git-dir.

Feel free to change the branch name accordingly - maybe prod or something, and potentially restrict access to this branch.

The -f flag is to force the changes through. There should be no local changes on your server, but if there are, they are going to get ruthlessly binned.

After this we get our bash script to change directory to whatever is set as GIT_WORK_TREE.

We run a composer install without the development dependencies, and then clear and warmup the cache.

At this point our new code should be live on our site.

Don't Forget var

We must have remember to setup the var folder as per Symfony's requirements:

# from the server

mkdir /var/www/crvfakeexample.com/var
chown $(whoami):www-data /var/www/crvfakeexample.com/var

HTTPDUSER=$(ps axo user,comm | grep -E '[a]pache|[h]ttpd|[_]www|[w]ww-data|[n]ginx' | grep -v root | head -1 | cut -d\  -f1)
sudo setfacl -dR -m u:"$HTTPDUSER":rwX -m u:$(whoami):rwX var
sudo setfacl -R -m u:"$HTTPDUSER":rwX -m u:$(whoami):rwX var

You'll know if you've forgotten this as 500 errors ahoy!

Deployment In Action

Let's test this out.

To start proceedings we need to tell our local git repository about our new remote:

# from your local

git remote add prod {your_remote_user}@{your_remote_ip}:/var/www/crvfakeexample.git

# e.g.

git remote add prod root@104.236.215.199:/var/www/crvfakeexample.git

And now we can push up our changes:

git push prod master

# asked for password here

Counting objects: 2, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 245 bytes | 0 bytes/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Already on 'master'
remote: Loading composer repositories with package information
remote: Installing dependencies from lock file
remote: Package operations: 43 installs, 0 updates, 0 removals
remote:   - Installing doctrine/lexer (v1.0.1): Loading from cache
remote:   - Installing doctrine/annotations (v1.2.7): Loading from cache
remote:   - Installing twig/twig (v1.35.0): Loading from cache
remote:   - Installing symfony/polyfill-util (v1.6.0): Loading from cache
remote:   - Installing paragonie/random_compat (v2.0.11): Loading from cache
remote:   - Installing symfony/polyfill-php70 (v1.6.0): Loading from cache
remote:   - Installing symfony/polyfill-php56 (v1.6.0): Loading from cache
remote:   - Installing symfony/polyfill-mbstring (v1.6.0): Loading from cache
remote:   - Installing symfony/symfony (v3.3.10): Loading from cache
remote:   - Installing symfony/polyfill-intl-icu (v1.6.0): Loading from cache
remote:   - Installing symfony/polyfill-apcu (v1.6.0): Loading from cache
remote:   - Installing psr/simple-cache (1.0.0): Loading from cache
remote:   - Installing psr/log (1.0.2): Loading from cache
remote:   - Installing psr/link (1.0.0): Loading from cache
remote:   - Installing psr/container (1.0.0): Loading from cache
remote:   - Installing psr/cache (1.0.1): Loading from cache
remote:   - Installing fig/link-util (1.0.0): Loading from cache
remote:   - Installing doctrine/inflector (v1.1.0): Loading from cache
remote:   - Installing doctrine/collections (v1.3.0): Loading from cache
remote:   - Installing doctrine/cache (v1.6.2): Loading from cache
remote:   - Installing doctrine/common (v2.6.2): Loading from cache
remote:   - Installing doctrine/doctrine-cache-bundle (1.3.2): Loading from cache
remote:   - Installing jdorn/sql-formatter (v1.2.17): Loading from cache
remote:   - Installing doctrine/dbal (v2.5.13): Loading from cache
remote:   - Installing doctrine/doctrine-bundle (1.6.13): Loading from cache
remote:   - Installing doctrine/data-fixtures (v1.1.1): Loading from cache
remote:   - Installing doctrine/doctrine-fixtures-bundle (v2.4.1): Loading from cache
remote:   - Installing doctrine/instantiator (1.0.5): Loading from cache
remote:   - Installing doctrine/orm (v2.5.12): Loading from cache
remote:   - Installing erusev/parsedown (1.6.3): Loading from cache
remote:   - Installing ezyang/htmlpurifier (v4.9.3): Loading from cache
remote:   - Installing incenteev/composer-parameter-handler (v2.1.2): Loading from cache
remote:   - Installing composer/ca-bundle (1.0.8): Loading from cache
remote:   - Installing sensiolabs/security-checker (v4.1.6): Loading from cache
remote:   - Installing sensio/distribution-bundle (v5.0.21): Loading from cache
remote:   - Installing sensio/framework-extra-bundle (v3.0.28): Loading from cache
remote:   - Installing monolog/monolog (1.23.0): Loading from cache
remote:   - Installing symfony/monolog-bundle (v3.1.1): Loading from cache
remote:   - Installing swiftmailer/swiftmailer (v5.4.8): Loading from cache
remote:   - Installing symfony/swiftmailer-bundle (v2.6.7): Loading from cache
remote:   - Installing twig/extensions (v1.5.1): Loading from cache
remote:   - Installing pagerfanta/pagerfanta (v1.0.5): Loading from cache
remote:   - Installing white-october/pagerfanta-bundle (v1.0.8): Loading from cache
remote: Generating optimized autoload files
remote: > Incenteev\ParameterHandler\ScriptHandler::buildParameters
remote: Creating the "app/config/parameters.yml" file
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::buildBootstrap
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::clearCache
remote: 
remote:  // Clearing the cache for the prod environment with debug                      
remote:  // false                                                                       
remote: 
remote:  [OK] Cache for the "prod" environment (debug=false) was successfully cleared.  
remote: 
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::installAssets
remote: 
remote:  Trying to install assets as relative symbolic links.
remote: 
remote:  --- ------------------------------ ------------------ 
remote:       Bundle                         Method / Error    
remote:  --- ------------------------------ ------------------ 
remote:   ✔   WhiteOctoberPagerfantaBundle   relative symlink  
remote:  --- ------------------------------ ------------------ 
remote: 
remote:  [OK] All assets were successfully installed.                                   
remote: 
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::installRequirementsFile
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::prepareDeploymentTarget
remote: 
remote:  // Clearing the cache for the prod environment with debug                      
remote:  // false                                                                       
remote: 
remote:  [OK] Cache for the "prod" environment (debug=false) was successfully cleared.  
remote: 
remote: 
remote:  // Warming up the cache for the prod environment with debug                    
remote:  // false                                                                       
remote: 
remote:  [OK] Cache for the "prod" environment (debug=false) was successfully warmed.   
remote: 
To root@104.236.215.199:/var/www/crvfakeexample.git
   64fc662..d74a880  master -> master

We see all of this output in our local terminal.

Unfortunately we lose all the nice terminal colouring.

One key line here that's easy to miss is:

remote: Creating the "app/config/parameters.yml" file

Fortunately we have some usable defaults. If we didn't, this process would highly likely not work as expected.

At this point, however, we should now be able to browse our site.

If we wanted to make a change then we need to go through the change as normal, commit the change to master, and push the master branch up to prod.

touch some-file

git add some-file
git commit -m "added some-file"
[master 42ecfba] added some-file
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 some-file

git push prod master

# asked for password here

Counting objects: 2, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 267 bytes | 0 bytes/s, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Already on 'master'
remote: Loading composer repositories with package information
remote: Installing dependencies from lock file
remote: Nothing to install or update
remote: Generating optimized autoload files
remote: > Incenteev\ParameterHandler\ScriptHandler::buildParameters
remote: Updating the "app/config/parameters.yml" file
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::buildBootstrap
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::clearCache
remote: 
remote:  // Clearing the cache for the prod environment with debug                      
remote:  // false                                                                       
remote: 
remote:  [OK] Cache for the "prod" environment (debug=false) was successfully cleared.  
remote: 
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::installAssets
remote: 
remote:  Trying to install assets as relative symbolic links.
remote: 
remote:  --- ------------------------------ ------------------ 
remote:       Bundle                         Method / Error    
remote:  --- ------------------------------ ------------------ 
remote:   ✔   WhiteOctoberPagerfantaBundle   relative symlink  
remote:  --- ------------------------------ ------------------ 
remote: 
remote:  [OK] All assets were successfully installed.                                   
remote: 
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::installRequirementsFile
remote: > Sensio\Bundle\DistributionBundle\Composer\ScriptHandler::prepareDeploymentTarget
remote: 
remote:  // Clearing the cache for the prod environment with debug                      
remote:  // false                                                                       
remote: 
remote:  [OK] Cache for the "prod" environment (debug=false) was successfully cleared.  
remote: 
remote: 
remote:  // Warming up the cache for the prod environment with debug                    
remote:  // false                                                                       
remote: 
remote:  [OK] Cache for the "prod" environment (debug=false) was successfully warmed.   
remote: 
To root@104.236.215.199:/var/www/crvfakeexample.git
   d74a880..42ecfba  master -> master