Docker Compose Tutorial For Elixir and Phoenix Framework


In the previous video we covered the web service of our docker-compose.yml file. Now, let's look at our Postgres database:

version: '3'

services:

  postgres:
    image: postgres:9.6.5
    ports:
      - "5432:5432"

We won't be using a custom image of postgres. The official public image meets our needs just fine.

We will use Postgres version 9.6.5, which at the time of writing is the most recent. The nice part about Docker is just how easy it is to bump up to a newer version of any service, whenever it becomes available.

Port 5432 is the official default port for connecting to Postgres, so that's the one we will use, and want exposed.

I have deliberately missed a step here, which is to set up a volume for Postgres. We will cover why that's important as we continue through this video.

We're nearly there now. Almost ready to boot. Let's keep going.

Going Up

Thinking back to what Phoenix's installation instructions told us, to get started we need to do two things:

Then configure your database in config/dev.exs and run:

    $ mix ecto.create

Start your Phoenix app with:

    $ mix phx.server

We know from our docker-compose.yml file that we will run the mix phx.server command when our web service starts.

Let's try this now and see what happens:

docker-compose up

Creating network "hellophoenix_default" with the default driver
Creating hellophoenix_postgres_1
Creating hellophoenix_web_1
Attaching to hellophoenix_postgres_1, hellophoenix_web_1
postgres_1  | The files belonging to this database system will be owned by user "postgres".
postgres_1  | This user must also own the server process.
postgres_1  | 
postgres_1  | The database cluster will be initialized with locale "en_US.utf8".
postgres_1  | The default database encoding has accordingly been set to "UTF8".
postgres_1  | The default text search configuration will be set to "english".
postgres_1  | 
postgres_1  | Data page checksums are disabled.
postgres_1  | 
postgres_1  | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1  | creating subdirectories ... ok
postgres_1  | selecting default max_connections ... 100
postgres_1  | selecting default shared_buffers ... 128MB
postgres_1  | selecting dynamic shared memory implementation ... posix
postgres_1  | creating configuration files ... ok
postgres_1  | running bootstrap script ... ok
web_1       | [error] backend port not found: :inotifywait
web_1       | 
postgres_1  | performing post-bootstrap initialization ... ok
web_1       | [error] Postgrex.Protocol (#PID<0.244.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.245.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.242.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.243.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.240.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.241.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.239.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.238.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.237.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.246.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [info] Running HelloPhoenixWeb.Endpoint with Cowboy using http://0.0.0.0:4000
postgres_1  | syncing data to disk ... 
postgres_1  | WARNING: enabling "trust" authentication for local connections
postgres_1  | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1  | --auth-local and --auth-host, the next time you run initdb.
postgres_1  | ok
postgres_1  | 
postgres_1  | Success. You can now start the database server using:
postgres_1  | 
postgres_1  |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1  | 
postgres_1  | ****************************************************
postgres_1  | WARNING: No password has been set for the database.
postgres_1  |          This will allow anyone with access to the
postgres_1  |          Postgres port to access your database. In
postgres_1  |          Docker's default configuration, this is
postgres_1  |          effectively any other container on the same
postgres_1  |          system.
postgres_1  | 
postgres_1  |          Use "-e POSTGRES_PASSWORD=password" to set
postgres_1  |          it in "docker run".
postgres_1  | ****************************************************
postgres_1  | waiting for server to start....LOG:  could not bind IPv6 socket: Cannot assign requested address
postgres_1  | HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
postgres_1  | LOG:  database system was shut down at 2017-08-04 17:10:17 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started
web_1       | 17:10:18 - info: compiled 6 files into 2 files, copied 3 in 599 ms
postgres_1  |  done
postgres_1  | server started
postgres_1  | ALTER ROLE
postgres_1  | 
postgres_1  | 
postgres_1  | /docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1  | 
postgres_1  | LOG:  received fast shutdown request
postgres_1  | LOG:  aborting any active transactions
postgres_1  | waiting for server to shut down...LOG:  autovacuum launcher shutting down
postgres_1  | .LOG:  shutting down
postgres_1  | LOG:  database system is shut down
web_1       | [error] Postgrex.Protocol (#PID<0.244.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
web_1       | [error] Postgrex.Protocol (#PID<0.237.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (postgres:5432): connection refused - :econnrefused
postgres_1  |  done
postgres_1  | server stopped
postgres_1  | 
postgres_1  | PostgreSQL init process complete; ready for start up.
postgres_1  | 
postgres_1  | LOG:  database system was shut down at 2017-08-04 17:10:18 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started
postgres_1  | FATAL:  database "dev_db" does not exist
web_1       | [error] Postgrex.Protocol (#PID<0.238.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
postgres_1  | FATAL:  database "dev_db" does not exist
web_1       | [error] Postgrex.Protocol (#PID<0.243.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
postgres_1  | FATAL:  database "dev_db" does not exist
web_1       | [error] Postgrex.Protocol (#PID<0.239.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
postgres_1  | FATAL:  database "dev_db" does not exist
web_1       | [error] Postgrex.Protocol (#PID<0.245.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
postgres_1  | FATAL:  database "dev_db" does not exist
web_1       | [error] Postgrex.Protocol (#PID<0.241.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
postgres_1  | FATAL:  database "dev_db" does not exist
web_1       | [error] Postgrex.Protocol (#PID<0.240.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
postgres_1  | FATAL:  database "dev_db" does not exist
web_1       | [error] Postgrex.Protocol (#PID<0.246.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
postgres_1  | FATAL:  database "dev_db" does not exist
web_1       | [error] Postgrex.Protocol (#PID<0.242.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
^CGracefully stopping... (press Ctrl+C again to force)
Stopping hellophoenix_web_1 ... done
Stopping hellophoenix_postgres_1 ... done

Oh mercy.

It all looked so good, until it died spectacularly.

Still, the error - although constantly repeating - is pretty useful:

FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist

It is right of course, it doesn't exist. We set it as part of the DATABASE_URL environment variable in our docker-compose.yml file, but it's not just magically created for us.

Instead, we need to do what Phoenix's installation instructions told us to do:

Then configure your database in config/dev.exs and run:

    $ mix ecto.create

But how do we do this when the docker-compose.yml is setup to run a single command on container boot?

version: '3'

services:
  web:
    image: docker.io/codereviewvideos/docker-phoenix
    command: mix phx.server # <-- this bit

Do we change this to:

version: '3'

services:
  web:
    image: docker.io/codereviewvideos/docker-phoenix
    command: mix ecto.create && mix phx.server

This wouldn't work as expected as Docker would only run the first command (as best I am aware), and then exit. In other words, it would create the database and then halt without trying to run mix phx.server.

Anyway, this approach has another flaw - it would only work once.

If we try to run mix ecto.create every time we bring up our Docker infrastructure, our web service will die with a complaint that we are trying to create a database that already exists.

Now, actually this is less of a problem for us, as we aren't yet persisting the database data - we will fix that shortly. However, beyond anything but a basic demo project you would likely find the fact your data is deleted every time you restart your Docker setup to be mildly infuriating at best.

Here's what we will do:

First, we will tell Postgres to use a volume. This will stop our database data from being deleted whenever the postgres service goes down.

Secondly, we will run mix ecto.create once, by using the web service.

version: '3'
services:
  web:
    # ...
  postgres:
    image: postgres:9.6.1
    volumes:
      - "./volumes/postgres:/var/lib/postgresql/data"
    ports:
      - "5432:5432"

Great, so when we bring up the postgres service now, a new directory will be created in our current working directory - /volumes. Inside this directory will be a subdirectory, postgres.

Our container will internally map this directory to /var/lib/postgresql/data.

As soon as the container is booted, postgres will start storing persistent data into that directory. In other words, your database data will no longer be deleted whenever the container gets removed. Nice.

Using a container for your database is potentially unwise. This is a different discussion. My advice is to use a real database server still in production. In dev, however, DB containers are great.

One thing you may have also noticed is that the permissions of the newly created volumes/ directory (and contents) are not - most likely - those of your own user:

➜  hello_phoenix ls -la volumes

total 12
drwxr-xr-x  3 root  root  4096 Aug  4 14:54 .
drwxrwxr-x 10 chris chris 4096 Aug  4 14:54 ..
drwx------ 19   999 root  4096 Aug  4 14:55 postgres

Docker is a real stickler for permissions. My advice is not to mess around with them, but be aware that most every serious problem I've had with Docker has been permissions related in some way. They really are a fiddly beast.

This is our system, however, so let's go sudo su - and check that out:

sudo su -

ls -la /path/to/your/project/volumes/postgres

total 128
drwx------ 19  999 root    4096 Aug  4 14:55 .
drwxr-xr-x  3 root root    4096 Aug  4 14:54 ..
drwx------  5  999 docker  4096 Aug  4 14:54 base
drwx------  2  999 docker  4096 Aug  4 14:55 global
drwx------  2  999 docker  4096 Aug  4 14:54 pg_clog
drwx------  2  999 docker  4096 Aug  4 14:54 pg_commit_ts
drwx------  2  999 docker  4096 Aug  4 14:54 pg_dynshmem
-rw-------  1  999 docker  4492 Aug  4 14:55 pg_hba.conf
-rw-------  1  999 docker  1636 Aug  4 14:54 pg_ident.conf
drwx------  4  999 docker  4096 Aug  4 14:54 pg_logical
drwx------  4  999 docker  4096 Aug  4 14:54 pg_multixact
drwx------  2  999 docker  4096 Aug  4 14:55 pg_notify
drwx------  2  999 docker  4096 Aug  4 14:54 pg_replslot
drwx------  2  999 docker  4096 Aug  4 14:54 pg_serial
drwx------  2  999 docker  4096 Aug  4 14:54 pg_snapshots
drwx------  2  999 docker  4096 Aug  4 14:55 pg_stat
drwx------  2  999 docker  4096 Aug  4 14:55 pg_stat_tmp
drwx------  2  999 docker  4096 Aug  4 14:54 pg_subtrans
drwx------  2  999 docker  4096 Aug  4 14:54 pg_tblspc
drwx------  2  999 docker  4096 Aug  4 14:54 pg_twophase
-rw-------  1  999 docker     4 Aug  4 14:54 PG_VERSION
drwx------  3  999 docker  4096 Aug  4 14:54 pg_xlog
-rw-------  1  999 docker    88 Aug  4 14:54 postgresql.auto.conf
-rw-------  1  999 docker 22233 Aug  4 14:54 postgresql.conf
-rw-------  1  999 docker    37 Aug  4 14:55 postmaster.opts
-rw-------  1  999 docker    85 Aug  4 14:55 postmaster.pid

exit

Ok, so fine, we can see postgres is storing data.

This doesn't solve our immediate problem. We still need to create the database.

docker-compose up -d
# then
docker ps -a

Notice now both containers stay up?

However, examining the docker logs {insert your web container id here} still shows an issue:

[error] Postgrex.Protocol (#PID<0.238.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
[error] Postgrex.Protocol (#PID<0.242.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
[error] Postgrex.Protocol (#PID<0.241.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
[error] Postgrex.Protocol (#PID<0.240.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist
[error] Postgrex.Protocol (#PID<0.244.0>) failed to connect: ** (Postgrex.Error) FATAL 3D000 (invalid_catalog_name): database "dev_db" does not exist

To fix this we can do one of two things. We can run the command itself via docker-compose:

➜  hello_phoenix docker-compose run web mix ecto.create

The database for HelloPhoenix.Repo has been created

Or we can jump on to a terminal inside the container:

➜  hello_phoenix docker-compose exec web /bin/bash

root@40e2d6172541:/app# mix ecto.create

The database for HelloPhoenix.Repo has been created

root@40e2d6172541:/app# exit
exit

Note the subtle difference between run and exec.

Hey, guess what?

Our Phoenix server is up:

http://127.0.0.1:4000/

Phoenix Framework 1.3 inside Docker

Pretty cool, huh?

There's not much to see, unfortunately, beyond a few links. However, all is not lost as we are going to be focusing on JSON fairly exclusively anyway.

Cleaning Up

Ok, so with our docker containers running in the background, how do we stop them?

docker-compose down

Stopping hellophoenix_web_1 ... done
Stopping hellophoenix_postgres_1 ... done
Removing hellophoenix_web_run_1 ... done
Removing hellophoenix_web_1 ... done
Removing hellophoenix_postgres_1 ... done
Removing network hellophoenix_default

All good.

You again may want to docker system prune at this point.

With everything up and running, now would be a good time to see why running docker-compose in the foreground can be useful:

➜  hello_phoenix docker-compose up

Creating network "hellophoenix_default" with the default driver
Creating hellophoenix_postgres_1
Creating hellophoenix_web_1
Attaching to hellophoenix_postgres_1, hellophoenix_web_1
postgres_1  | LOG:  database system was shut down at 2017-08-04 14:09:13 UTC
postgres_1  | LOG:  MultiXact member wraparound protections are now enabled
postgres_1  | LOG:  database system is ready to accept connections
postgres_1  | LOG:  autovacuum launcher started
web_1       | 
web_1       | [info] Running HelloPhoenixWeb.Endpoint with Cowboy using http://0.0.0.0:4000
web_1       | 14:10:35 - info: compiled 6 files into 2 files, copied 3 in 607 ms

Now when we hit the "Welcome to Phoenix" page we can see something I find rather exciting:

web_1       | 14:10:35 - info: compiled 6 files into 2 files, copied 3 in 607 ms
web_1       | [info] GET /
web_1       | [debug] Processing with HelloPhoenixWeb.PageController.index/2
web_1       |   Parameters: %{}
web_1       |   Pipelines: [:browser]
web_1       | [info] Sent 200 in 21ms
web_1       | [info] GET /
web_1       | [debug] Processing with HelloPhoenixWeb.PageController.index/2
web_1       |   Parameters: %{}
web_1       |   Pipelines: [:browser]
web_1       | [info] Sent 200 in 205µs
web_1       | [info] GET /
web_1       | [debug] Processing with HelloPhoenixWeb.PageController.index/2
web_1       |   Parameters: %{}
web_1       |   Pipelines: [:browser]
web_1       | [info] Sent 200 in 204µs
web_1       | [info] GET /
web_1       | [debug] Processing with HelloPhoenixWeb.PageController.index/2
web_1       |   Parameters: %{}
web_1       |   Pipelines: [:browser]
web_1       | [info] Sent 200 in 200µs
web_1       | [info] GET /

The first request in takes 21ms, which is really quick as far as I am concerned.

What blows me away is the subsequent requests.

205µs - that funny µs symbol means MICRO seconds.

I'll take that all day long. Good grief.

Lastly we should update our project's Makefile to save ourselves the burden of remembering how to use Docker:

dev:
  @docker-compose down && \
    docker-compose build --pull --no-cache && \
    docker-compose \
      -f docker-compose.yml \
    up -d --remove-orphans

Then I can call make dev to spin up my environment really easily.

There you have it. This was a long form approach to explaining each of the individual steps to - hopefully - remove (or reduce) any questions you might have about this process.

In actuality, you can largely copy / paste this and be up and running in just a few short minutes.

With such a reliable and reproducible environment to work from, I find myself much more likely to try out new things as there's now so small a barrier to entry.

As ever, I endeavor to make this information as accurate and thorough as possible, but any errors, omissions, or corrections are more than welcome in the comments below.

Episodes