Beta Blockers

It’s been a busy week here at CodeReviewVideos.

I want to thank everyone who tried out the site in the “beta” test. This has now ended, and I’m ready to launch the new site version. Hooray.

The only reason to delay now is that I’m away all weekend (more on this below), and as such, don’t want to launch on a Friday. Instead, the launch will likely be Tuesday, all things being equal.

There’s a bunch of tasks for the launch – it’s not quite as straightforward as I’d like. The hardest part is around migrating the database. I’ve changed the structure quite a lot between versions, and keeping both in sync across two different servers is proving… interesting.

There have been so many lessons learned during this process. Just this week I realised none of the pages had HTML head tags. Whoops. But then it turns out populating these tags from an API response isn’t super straightforward either. There’s no server-side rendering to help me 🙁

Another problem that caught me totally off guard was the sitemap.

Being that CodeReviewVideos is an existing, live site I know that whatever SEO ‘stuff’ I have in place already must be mirrored to the new site.

What I hadn’t thought through was the location of the sitemap.xml  file.

In the current iteration, I’m using Symfony with Twig. Everything lives on one server, and as such, making the sitemap is really straightforward. I use a bundle (PrestaSitemapBundle) and it just works.

For the new site there are two areas where pages may come from:

  • Static pages – i.e. the FAQ, contact / support, privacy policy, etc
  • Dynamic pages – i.e. the Course / Episode pages

There are some libraries available that will help generate sitemaps for React-based projects. I looked at each in turn but none seemed to fit my needs. I found this odd, honestly, as surely I’m not the first person to meet this problem.

Instead, I decided to roll my own. Well, I say “roll my own”, but really it’s a combination of some Doctrine query output dumped into Samdark/sitemap.

This worked really well, but.

It’s a big but.

The sitemap is generated on a sub-domain – under api dot codereviewvideos dot com.

Apparently, having done some research, Google ain’t so keen on this. They want the sitemap to live on www dot codereviewvideos dot com.

The problem being that behind the scenes these are two entirely separate concepts. Both are Docker images with all the necessary data “baked in”. How could I dynamically generate a sitemap from one, and it appear on the other?

Well, it turned out Docker Volumes came to my rescue. I could bind mount a file on the underlying host, and make the file appear as though it were on the www site automatically.

There have been an absolute ton of unexpected surprises like this along the way.

Video Update

This week saw three new videos added to the site.

#1 – Docker MySQL Tutorial

There’s a school of thought that says Docker and Databases don’t make good bedfellows.

For my money, this debate is really only worth having if you’re heading into production. If you’re simply wanting to play around with a database, or you have some local development work to get done, then Docker is perfect for the task.

In this video you will learn how to quickly spin up a working MySQL database server, in almost no time at all. Once you’ve got the latest MySQL Docker image stored locally on your computer, getting a new container up and running takes seconds.

As with almost anything Docker related, there are a few potential gotchas. It’s really all about reading the instructions (or the docker logs  output) if things don’t go according to plan. In particular here, you will need to use one of the mandatory environment variables.

We finish up this video by taking the example docker run command provided in the docs, and turning it into a more practical, usable variant along with storing our database on a named volume.

#2 – Docker nginx PHP Tutorial

Getting MySQL up and running was fairly easy going. And earlier in this series we saw how we could quickly spin up a Dockerised nginx server without much effort, either.

However, beyond some simple examples, the truth is you will need to do quite a bit more to get a working development environment that replicates Vagrant or some Ansible-managed VM infrastructure.

The way we will work through this is to separate nginx from the actual application code. In other words, nginx, php, and our Symfony site will all be separate Docker images.

On the surface of it, this seems like we’re going a little overkill. However, stick with me through this, and the next two videos, as this approach has served me well in the real world.

#3 – Docker PHP 7 Tutorial (7, 7.1, and higher)

As covered in the previous Docker nginx tutorial, our stack will be split across multiple Docker images and containers.

We’re going to let nginx accept incoming requests for our PHP-powered site(s), but the nginx container won’t know how to process PHP code. Instead, it will hand off to a dedicated PHP-processing container.

For our purposes we are going to create a PHP 7.1 image. This will be relatively generic, containing all the stuff most every PHP site will need. This will include a bunch of system level dependencies such as curl , and git , through to PHP deps such as intl , pdo_mysql , and optionally xdebug .

However, we won’t directly use the PHP image we create. Instead we will use this as the starting point for all future PHP Docker images. For example, in the next video we will use it as the basis of a Symfony image. But I also use the image as the base for WordPress, and custom PHP projects too.

PHP North West

This weekend I will be at PHP North West. If you are heading to PHP North West then please say hello 🙂 I already know a few of you will be attending, and I’m looking forward to meeting up in person.

The other reason I mention this is that support response times may be a little slower than usual. Apologies in advance if this causes you any inconvenience.

This will be my fourth year attending PHP North West, and whilst I didn’t blog about my first visit (not sure why), I did capture the previous two. Expect similar for PHPNW17 some time next week.

 

Docker Tutorial For Beginners (Update)

I’ve been super busy this week behind the scenes working on the new site version. Typically, every time I think I’m done (we all know we are never done :)), I hit upon something I’ve either missed, or overlooked.

There are a bunch of CSS related jobs:

Yeah… if you could just fix that margin-top, that’d be great

And React seems to have completely broken my mailing list sign up forms:

You can’t tell from a static image, but these inputs don’t allow input 🙂

And some things I only noticed when testing my user journeys:

What, no remember me?

One thing I am really happy with is the new Profile panel:

4 tabs, a lot of work 🙂

There are 29 open tickets left before I consider this next version done (again, never really done :)).

I reckon I will be happy enough to open the beta when I’m down to 20 tickets. I genuinely wish I was a 10x developer, able to blitz stuff like this out in a weekend.

I’m really looking forward to switching over though. This new site is so much nicer to work with from a development point of view. And I have a bunch of improvements from a member / visitor point of view, too 🙂

Both the front end and back end are completely separated. Both are pushed through a GitLab CI pipeline, which automates the test runs, and if all goes to plan, builds a Docker image for both. I can then deploy these using Rancher.

It’s a great setup.

As you may have noticed, Docker is currently the main course topic on the site. This is a Docker tutorial for beginners. If you’ve ever wanted to learn Docker, and prefer to see real world setups rather than simplified examples then this is the right starting point for you.

Which brings me neatly on to this weeks…

Video Update

This week saw three new videos added to the site.

All of this week’s videos fall under the same theme:

Long, and short term storage of data in Docker.

One of the most confusing aspects of switching my stack from Virtual Machines to Docker was what was happening to my data?

In Virtual Machine-land (if such a place ever existed) the location of my data was fairly obvious. It would either be on the VM itself (on the virtual hard disk), or on some attached network drive.

Things are roughly the same in Docker, but it’s not a 100% like-for-like experience. Especially not on a Mac.

#1. Docker, Without Volumes

Are volumes essential for persistent storage in Docker?

It can sometimes seem that way. A lot of Docker tutorials I followed seemed to equate persisting data with Docker Volumes.

In this video we explore persisting data within Docker containers without Volumes.

Yes, it can be done.

Yes, it works just fine.

And yes, you will lose your data if you delete the container your data belongs too 🙂

So don’t fall in to that trap. Watch this video and learn how to avoid this mistake.

#2. [Part 1] – Docker Volumes – Bind Mounts

I’m going out on a limb here, and I’m going to guess you love a good Star Trek episode as much as me.

Nothing troubles my tribbles quite like a blurt of technobabble:

The temporal surge we detected was caused by an explosion of a microscopic singularity passing through this solar system. Somehow, the energy emitted by the singularity shifted the chroniton particles in our hull into a high state of temporal polarisation.

Now the good news, when watching Star Trek, is that you really don’t need to understand any of that to gain enjoyment from the episode. The writers make sure the characters appear to understand it, and the Federation once again, save the day.

But what about technobabble we do have to understand?

Docker Bind Mounts.

Well, in this video we run our tricorder over bind mounts and figure out just what the heck these things are.

The good news?

Beyond the name they are surprisingly straightforward.

Set tachyon cannons to maximum dispersion!

#3. [Part 2] – Docker Volumes – Volumes

Here’s the unusual thing:

Docker bind mounts sounds complicated, but as we saw in the previous video, they really aren’t.

Docker volumes, on the other hand, sound quite straightforward.

But they are more complicated.

Docker’s Volumes are the future of the way we work with data in Docker.

Therefore, understanding Volumes is a very useful skill to have.

Fortunately, even though Volumes are more involved than bind mounts, by watching the previous two videos you have all the pre-requisite information required to grasp this concept.

In this video we work through a bunch of examples to ensure the fundamentals are in place. Volumes will feature heavily throughout the rest of our dealings with Docker, so be sure to understand this section before moving on.

That’s all from me for this week. Until next week, have a great weekend, and happy coding.

Chris

How I am using Docker with Symfony, nginx, and MySQL

Behind the scenes this week I have been continuing my work on the next major iteration of the CodeReviewVideos website.

I’ve split the site into two parts:

  • A Symfony 3 powered JSON API for the back-end.
  • React / Redux / Redux Saga for the front-end

It’s a stack that I really enjoy working with, and is all stuff I’ve covered here on the site before.

Progress has gone fairly well this week. As a developer I always expect to be further ahead than I actually am. Partly this is impatience, and partly because I am still not so good at estimating 🙂

By next week’s newsletter I am hoping to be able to share access to the “beta” (for want of a better word) with existing site subscribers.

Become An Instructor

I really want to make CodeReviewVideos as useful to you as possible.

There are likely dozens of topics you’d like to see covered that either I have not yet had chance to record videos for, or do not have any experience in.

Topics like Angular 2/4, Vue, WordPress, Magento… all the things that we as developers have to work with in our real-world, day-to-day jobs.

If you have ever wanted to start recording screencasts / tutorials then this may well be the perfect opportunity to do so, and earn some extra recurring income in the process.

If you’d like to know more then please send an email to chris@codereviewvideos.com.

Video Update

This week saw three new videos added to the site.

#1. How to add a Flash Message on Successful, or Failed Login

I got asked this question:

how can I “$this->addFlash” for login? Because now its working only with registration. How can I make it work with login? Check if signing in was successful/not successful?

There is a solution built into Symfony’s security bundle (think: Login form configuration) to make this fairly easy to accomplish.

There is just one small problem: lack of documentation / example.

I’m hoping to correct that with this video. Before going too much further, you may be wondering:

What are Flash Messages?

Flash messages in Symfony are simply one time messages – notifications, by another name – that are stored in the session and are removed as soon as they have been shown once.

If still unsure, I have a video called the Beginners Guide to Symfony Flash Messages. It’s free to watch.

Symfony 3.3 comes with some improvements to the way Flash messages work.

We cover how to add a Flash Message using the success_handler and failure_handler services. The tricky part is that these two services must behave a certain way, and this video covers one approach to this problem.

I’m always open to answering questions like this, so if you have any that you feel would be best answered by video then do please send them in.

#2.  [Demo] – Docker with Symfony, nginx, and MySQL

Docker.

It’s popular.

It’s good for the CV.

It’s also complicated.

In this video we take a look at a Dockerised environment in which the latest version of Symfony 3, an nginx webserver, and a MySQL database can be “spun-up” in next to no time.

The idea here is to take a look at what’s possible, rather than me saying: Hey, this is all you need to know.

I’ve planned out the Docker series and it’s going to be a few shorter series that can be watched independently, or all together.

There are a bunch of topics in Docker that you can initially skip over. Only as and when you have a need for either a more complex setup, or want to satisfy your curiosity do you need to delve any deeper.

This is good, in my opinion, as Docker itself is both broad and deep, and if we covered all the fundamentals upfront then you may be sound asleep by the end of the first few videos.

Because seeing code is fun, here’s the docker-compose.yml file that this video covers:

# docker-compose.yml

version: '2'

services:

    db:
        image: mysql:5.7.17
        env_file:
          - ./.env
        volumes:
          - "./volumes/mysql/dev:/var/lib/mysql"
        networks:
          crv_dev_network:
            aliases:
              - mysql

    nginx:
        image: codereviewvideos/nginx.dev
        ports:
          - "81:80"
        restart: always
        volumes:
          - "./volumes/nginx/logs:/var/log/nginx/"
        volumes_from:
            - php
        networks:
          crv_dev_network:
            aliases:
              - nginx

    php:
        image: codereviewvideos/www.dev
        restart: always
        env_file:
          - ./.env
        volumes:
          - "./volumes/php/app/cache:/var/www/dev/app/cache/:rw"
          - "./volumes/php/app/logs:/var/www/dev/app/logs/:rw"
          - "./volumes/.composer:/var/www/.composer"
          - "./app:/var/www/dev"
        networks:
          crv_dev_network:
            aliases:
              - php

networks:
  crv_dev_network:
    driver: bridge

You can watch the video here, and then hit the Repo over on GitHub.

We will cover everything in this repo piece by piece, building more than just a Symfony stack in the process.

#3. Linux Permissions 101

Oh my, this sounds dull.

Yes, I hear you.

There are parts of Development that sound sexy and fun (e.g. Docker, the latest JS framework, real-time mobile apps, etc), and then there are the dry and boring sounding topics that unfortunately, you really need to know.

I realise permissions aren’t much fun.

That’s why I have compressed the essential knowledge down into a 4 minute long video.

If you have ever seen something like this:

➜  ~ ls -la Development/docker/nginx

total 24
drwxrwxr-x  4 www-data www-data 4096 Apr  3 19:43 .
drwxrwxr-x 74 chris    chris    4096 Aug 17 20:15 ..
drwxrwxr-x  2 www-data www-data 4096 Apr  3 19:37 conf.d
-rwxrwxr-x  1 www-data www-data  137 Apr 18 20:51 Dockerfile
-rwxrwxr-x  1 www-data www-data   13 Apr  3 19:42 .dockerignore
drwxrwxr-x  8 www-data www-data 4096 Aug 19 12:05 .git

And you have wondered just what the heck most of this means then this video will see you right.

To work with Docker it is essential you understand just a small amount of this output. Most every problem I have had with Docker has been related to permissions. Trust me on this, if you want to use Docker then you need to know this stuff.

And the good news is that it’s really not that bad. And it’s useful beyond working with Docker too. Win win!

Until next week, have a great weekend, and happy coding.

Chris

 

A Tweak For ELK On Docker

One annoyance I’ve hit whilst running ELK on Docker is, after rebooting my system, the same error keeps returning:

 * Starting periodic command scheduler cron
   ...done.
 * Starting Elasticsearch Server
   ...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
waiting for Elasticsearch to be up (26/30)
waiting for Elasticsearch to be up (27/30)
waiting for Elasticsearch to be up (28/30)
waiting for Elasticsearch to be up (29/30)
waiting for Elasticsearch to be up (30/30)
Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.
[2017-07-14T08:36:42,337][INFO ][o.e.n.Node               ] [] initializing ...
[2017-07-14T08:36:42,437][INFO ][o.e.e.NodeEnvironment    ] [71cahpZ] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/sde2)]], net usable_space [51.1gb], net total_space [146.6gb], spins? [possibly], types [ext4]
[2017-07-14T08:36:42,438][INFO ][o.e.e.NodeEnvironment    ] [71cahpZ] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-14T08:36:42,463][INFO ][o.e.n.Node               ] node name [71cahpZ] derived from node ID [71cahpZ4SjeKKAoH8X5dYg]; set [node.name] to override
[2017-07-14T08:36:42,463][INFO ][o.e.n.Node               ] version[5.3.0], pid[64], build[3adb13b/2017-03-23T03:31:50.652Z], OS[Linux/4.4.0-45-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-07-14T08:36:43,910][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [aggs-matrix-stats]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [ingest-common]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [lang-expression]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [lang-groovy]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [lang-mustache]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [lang-painless]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [percolator]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [reindex]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [transport-netty3]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] loaded module [transport-netty4]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService     ] [71cahpZ] no plugins loaded
[2017-07-14T08:36:45,703][INFO ][o.e.n.Node               ] initialized
[2017-07-14T08:36:45,703][INFO ][o.e.n.Node               ] [71cahpZ] starting ...
[2017-07-14T08:36:45,783][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 58:01:4e:51:11:f3:c9:da
[2017-07-14T08:36:45,835][INFO ][o.e.t.TransportService   ] [71cahpZ] publish_address {172.20.0.3:9300}, bound_addresses {0.0.0.0:9300}
[2017-07-14T08:36:45,839][INFO ][o.e.b.BootstrapChecks    ] [71cahpZ] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-07-14T08:36:45,840][ERROR][o.e.b.Bootstrap          ] [71cahpZ] node validation exception
bootstrap checks failed
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-07-14T08:36:45,842][INFO ][o.e.n.Node               ] [71cahpZ] stopping ...
[2017-07-14T08:36:45,909][INFO ][o.e.n.Node               ] [71cahpZ] stopped
[2017-07-14T08:36:45,909][INFO ][o.e.n.Node               ] [71cahpZ] closing ...
[2017-07-14T08:36:45,914][INFO ][o.e.n.Node               ] [71cahpZ] closed

The core of the fix to this problem is helpfully included in the output:

max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

And on Ubuntu fixing this is a one-liner:

sudo sysctl -w vm.max_map_count=262144

Only, this is not persisted across a reboot.

To fix this permenantly, I needed to do:

sudo vim /etc/sysctl.d/60-elasticsearch.conf

And add in the following line:

vm.max_map_count=262144

I am not claiming credit for this fix. I found it here. I’m just sharing here because I know in future I will need to do this again, and it’s easiest if I know where to start looking 🙂

Another important step if persisting data is to ensure the folder is owned by 991:991 .

Centrifugo Docker Compose Config

    centrifugo:
        image: centrifugo/centrifugo:1.7.3
        environment:
          - CENTRIFUGO_SECRET=potato
          - CENTRIFUGO_ADMIN_PASSWORD=potato
          - CENTRIFUGO_ADMIN_SECRET=potato
        command: centrifugo --web
        ports:
          - "8569:8000"
        networks:
          crv_network:
            aliases:
              - crv_centrifugo

Admittedly not likely useful by many, but hopefully will save someone some time in the future.

This is just stuff I found throughout various GitHub tickets that’s good enough to get Centrifugo up and running on http://127.0.0.1:8569/