Symfony + Docker: My Journey So Far

Please note: This is a copy of my weekly email message. Please feel free to join the mailing list (the form is on the sidebar <<<) if you like this kind of thing. No pressure, it’s cool 🙂

I want to start by saying a huge thank you to everyone who replied to last week’s email. I was very grateful of each and every one of your replies, and want to reassure everyone: the Symfony tutorials will continue 😀

Over the next few weeks I will touch on this feedback a little more, as it has led to some interesting changes in the way I am thinking about the site.

Again, thank you, I really do appreciate and value your feedback.

I did something I was reluctant to do this week: I started playing with Docker.

This isn’t the first time I’ve had a bash with it. I tried it in late 2014 when I first became aware of it. This was around version 1.3, as best I recall. The reliance on the CLI – many of the commands being long and unwieldy – completely put me off.

I put it down almost as quickly as I’d picked it up. New and shiny, maybe, but confusing as heck.

Fast forward a year and I was on site with a FTSE 250 company who were spinning up a big docker swarm. Based on my in-depth research (read: 2 hours cursing followed by giving up) only a year earlier, I thought they must be mad. But hey, if big businesses were giving this a shot, then maybe there must be something too it, right?

So I picked it back up, and had another go. Everything was roughly as I remembered it. I still didn’t gel with the CLI.

However, there was now a new way of working with Docker: docker-compose.

The benefit of docker-compose being that you can have a single config file which manages the creation of many docker containers. Awesome! Given that I had no understanding of building one container, being able to spin up many containers couldn’t possibly go wrong.

Anyway, I persisted somewhat but I kept hitting on hurdles. Figuring out how to get Symfony’s cache and log permissions to work was frustrating me. There weren’t that many blog posts or tutorials to be found, and given that Ansible was working just fine for me, I once again consigned docker to the “not ready for me yet” pile.

If docker had gone away at this point, I can’t say I would have missed it. I really didn’t get the hype.

Now, whilst all this was happening, GitLab released version 8.0.

One of the very coolest features in GitLab 8.0 was their integration with GitLab CI (Continuous Integration). I watched the release video and I thought – oh my lord, I need me some of that! (https://about.gitlab.com/2015/09/22/gitlab-8-0-released/)

I absolutely love GitLab. It’s probably my single most used tool on any given day. The idea of being able to commit my code and have it spin up an environment that matches my dev / live setup, and then run my test suite for each commit?! What is this magic?

Alas, in reality this was not quite as easy as they made it look / sound.

A lot of this was due to the way I was working. I heavily used Ansible. And GitLab heavily expected me to be using Docker.

Undeterred, I spent night after night hacking away until I figured out how to make GitLab CI runner work with VirtualBox. Then, because this entire process was such a massive pain in the backside, I documented it all and video’d it (https://codereviewvideos.com/course/your-own-private-github/video/how-to-use-virtualbox-as-a-gitlab-ci-test-runner). I had suffered so much I hoped others would never have too.

But it griped me. This process was painful. If things changed in my environment – which they inevitably always do – I would need to build a new image, copy that over to my runner, test it all out… testing the test runners… how deep does this rabbit hole go?

What frustrated me most was that when this process works, it is incredible.

The alternative was to use Docker. In doing so, GitLab could spin up containers as needed and remove all of this pain. The caveat to this would be changing my entire development workflow. And I was so invested in Ansible this was incredibly unappealing.

Once more, I consigned docker to the “cool, but not for me just yet” pile.

Then GitLab dun’ did it again.

https://about.gitlab.com/2017/03/22/gitlab-9-0-released/

I urge you to watch the release video and not think: GIMME GIMME GIMME!

Now, I watched this video on Wednesday dinner time. Shortly thereafter I had to do a major production deploy involving Symfony, RabbitMQ, LetsEncrypt, Redis, MySQL, Node JS with PM2, a raft of cron jobs… and that’s just what I remember without looking into the Ansible configs more deeply.

The idea is simple – run a single command and let Ansible deploy all this stuff, reliably and in a managed fashion.

In (my) reality, most any new build involves some nightmare. For example, in dev you don’t need to worry about SSL. In prod, you do. This is not part of your dev Ansible scripts so manages to cause a few hours of headscratching whilst the dev / prod environments are integrated. Often this involves commenting out part of a script, running the first script, uncommenting out that part, and running another script. Reliable? Reproducible? Hmmm.

Then what about RabbitMQ in production? It needs to be protected with SSL. How does this fit into the config for LetsEncrypt… more hours lost.

And so on. I know many of these problems are specific to the way I work. But I am 100% convinced I’m not the only one who ends up in situations like this.

Anyway, I rage quit.

Deploying this stack can wait. Fortunately it’s important, but not urgent.

During all this time I couldn’t help but be seduced – once more – by the allure of docker, and GitLab’s one click easy deploys.

So this time I set myself a goal. I would take my entire Ansible orchestrated stack for this complex environment, and I would docker-ise it.

At this point I wish I could say the CLI issues had gone away. They haven’t. But this time I spent most of Thursday morning with my head in the Docker documentation. Then I started implementing a super basic config from scratch.

From there, I composed a few containers, and suddenly, it started to make sense.

I hit on a whole range of issues getting Symfony to work with Docker though. Permissions, but of course. Cache and Logs, the two Symfony stalwarts. But I’m past that now. And amazingly, it’s insanely quick. Like 1ms page renders quick.

I still have some way to go with this. And I’ve not started integrating with GitLab CI at all just yet. But there is major promise there.

Hopefully within the next week or so I will be able to share my docker scripts. There’s likely ways to improve on these yet. Still, it’s highly exciting.

With all this in mind, I would be super interested in hearing from you if you have put Docker + Symfony into production. That seems like the next major hurdle.

If you have any Docker experience – good, bad, or otherwise – please do hit reply and let me know. I’m particularly interested in those unexpected pitfalls I just know are out there, and would definitely appreciate a heads up if you are aware of any.

Video Update

This week there have been three new videos added to the site.

https://codereviewvideos.com/course/react-redux-and-redux-saga-with-symfony-3/video/change-password-part-6-avoid-blocking-and-wrap-up

Firstly we finished up the “Change Password” implementation.

On the surface, it may seem a little crazy to have 6 videos covering how to accept a user’s password change form submission. After all, this is only the front end, right? We have another two videos (https://codereviewvideos.com/course/fosuser-fosrest-lexikjwt-user-login-registration-api/video/password-management-change-password-part-1) covering how to do this on the back end!

Well, the first time you do anything like this you inevitably have to lay a whole bunch of foundations. All of these foundations make subsequent work much faster.

An example of this would be in how we can now process Symfony form errors on our front end. This will come in handy for any time we need to make a POST request – not just for changing passwords, but for registration, login, and any other Create / Update you do in your app.

Also, we extracted the repeated password fields out into their own separate component. This will allow us to re-use this component wherever we need to accept passwords – such as in the Registration screen, which is up next.

Finally, we looked at a way that Redux Saga provides for us to speed things up by allowing us to run many processes in parallel. When combined with our form error helper function, this becomes super useful should you have a form with many fields, just as one example.

https://codereviewvideos.com/course/symfony-workflow-component-tutorial/video/workflow-events-part-1

Next up we got an intro into the many events that Symfony’s Workflow Component dispatches for us during any single transition.

In case you haven’t been following along, a transition is when an object that’s flowing through our workflow goes from one place to another. These sorts of things happen all the time in our applications, even if we don’t have a formal process (or shiny Symfony component) to manage them.

In the example we have been working through, a potential customer (a prospect) can sign up with our system. When signing up, this potential customer goes through the transition of ‘sign up’. This takes them from the place of ‘prospect’ to the place of ‘free customer’.

As part of this transition, the workflow component dispatches a bunch of useful events. Each of these events can be hooked into, and if needed, some extra action can be taken. This might be as simple as logging some data out somewhere, but could be much more involved – potentially even triggering a whole different workflow.

This video is free for all, and explains when and where each of these events comes from.

https://codereviewvideos.com/course/symfony-workflow-component-tutorial/video/workflow-events-part-2-custom-audit-trail-listener

As part of the Workflow Component, a helpful utility class is included called the AuditTrailListener.

It’s got a bit of a computer science-y name, but the concept is simple enough: It listens for events, and then logs information about those events out to your Symfony log file.

In the previous video we saw just how many of these events are dispatched for every single transition.

What the provided AuditTrailListener cannot do is hook into some of the more unique events. However, we can implement our own version of this listener to capture as much information as we need. This also demonstrates exactly how you would listen to, and act upon any workflow event in your system.

How very handy! 🙂

Next week will see the final parts of the React, Redux, and Redux Saga With Symfony 3 course being uploaded. I’ve really enjoyed creating and sharing this course, and I hope you’ve found it useful too.

Until then, have a great weekend. Cross your fingers, we may even get some sunshine. Hey, that’s a big deal here in the UK 🙂

Crisis? What Crisis?

Depending on how closely you follow the Symfony community, you may or may not have seen the Grafikart story last weekend. If you didn’t see it, the gist of it is that lawyers demanded Grafikart remove their Symfony tutorial videos from YouTube.

As best I understand it, the laywer(s) believed they had found potential issues with trademark violations on their tutorials.

The aftermath of this was particularly interesting to me, what with a large part of CodeReviewVideos content being in a very similar category – i.e. Symfony tutorial videos, a number of which are on YouTube.

Now, thankfully, without much delay, Fabien Potencier got involved and resolved the issue:

Phew. Ok, so crisis averted.

However, whilst all this was happening I do have to say that there was an anxious knot in my stomach. It concerns me that the vast majority of the content on the site is about Symfony, and if this had gone the other way, the site may be offline right now.

That got me to thinking: Are all my eggs in one Symfony-shaped basket?

Right now, I would say 8 out of 10 of my eggs are Symfony-shaped eggs.

Historically the vast majority of the content has been focused on Symfony. More recently the content is Symfony alongside some front-end framework / library. I favour React, but have covered Angular, and Ionic.

Whatever front end you choose, Symfony – in my experience – can make a wonderful back end.

I still believe that the Symfony framework is the very best way I can be productive with PHP in 2017. There are always other frameworks and languages popping up that try to seduce me with their new and shiny, and there likely always will be. But when stuff needs to get done, it’s PHP and Symfony that I go to.

As many know already, I started CodeReviewVideos because I found learning Symfony to be difficult and stressful. This stress is amplified many times over if you have to learn Symfony “on the job”. That stress can be greatly reduced if you have access to a library of videos that explain the various ways and means of working with the framework to help you through.

Everything I learn, and have learned on this journey I share with you via the site. If its working for me, then chances are, it will work for you too.

From the feedback I’ve had recently, I’m excited that many of you are open to different types of video / tutorial content. Node JS gets requested a lot. I would also like to cover Elixir. And of course, lots more React.

All of this said, I’m not stopping recording Symfony content. I just spent a week recording loads of new content, and have another week booked in to do even more at the end of this month.

I’m interested in your opinion on this. Please can you hit reply in the comments below and let me know your thoughts?

Video Update

This week there have been four new videos added to the site.

https://codereviewvideos.com/course/symfony-workflow-component-tutorial/video/workflow-guards-part-1

In this video we are getting started with Workflow Guards.

The concept behind a Workflow Guard is simple enough: We want to block a transition if some criteria that we set is not met.

This criteria can be as simple, or as complex as you require.

Guards are going to give us a surface level skim of Events inside the Symfony Workflow Component, which will stand us in good stead for our deeper dive in a couple of videos time.

https://codereviewvideos.com/course/symfony-workflow-component-tutorial/video/workflow-guards-part-2

There are three different ‘levels’ of guards. In the previous video we look at the first two, the more generic guards.

In this video we cover very specific guards, and their potential use cases.

We also look at a way of improving the error message output – this might be a little controversial, but hey, it’s working ok for me so far.

https://codereviewvideos.com/course/react-redux-and-redux-saga-with-symfony-3/video/change-password-part-4-converting-errors-from-symfony-to-redux-form

One of the pain points I encountered when working with a Symfony-backed JSON API and React on the front end was dealing with error messages. In particular, those returned from a Symfony form error.

There’s likely a bunch of ways of addressing this problem. My advice is to use TDD to ensure your solution behaves as expected. Handling errors is painful for sure, but it’s pain worth enduring so your end users aren’t left scratching their heads.

In this video we start building an error helper function which takes the returned JSON form errors, and extracts useful information from those errors, allowing display on our Redux Form front end.

https://codereviewvideos.com/course/react-redux-and-redux-saga-with-symfony-3/video/change-password-part-5-adding-more-tests

In the previous video we made a start on the form error helper.

Even if you’re not interested in JavaScript, I would recommend this video to you as it covers one of my personal bugbears: Nested conditionals.

Whenever you nest anything – in this case an ‘if’ inside and ‘if’ – you increase the complexity of your code. Nothing gripes me more than opening a file to find some “christmas tree” shape where ifs are buried in ifs, buried in ifs… oh and a lovely else at the end with another if inside it. Who are these people who can keep this in their heads?!

Anyway, soap box aside, as I say, if you’re not interested in JavaScript, at least consider watching this for the code structuring aspect.

Ok, until next week, have a jolly fine (and hopefully sunny) weekend, and happy coding!

Chris

How I Fixed: Missing Headers on Response in Symfony 3 API

The first time this caught me out, I didn’t feel so bad. The second time – i.e. just now – I knew I had already solved this problem (on a different project), and found my urge to kill rising.

I wanted to POST in some data, and if the resource is successfully created, then the response should contain a link – via a HTTP header – to the newly created resource.

Example PHP / Symfony 3 API controller action code snippet:

    public function postAction(Request $request)
    {
        $form = $this->createForm(MyResourceType::class, null, [
            'csrf_protection' => false,
        ]);

        $form->submit($request->request->all());

        if (!$form->isValid()) {
            return $form;
        }

        $myResource = $form->getData();

        $em = $this->getDoctrine()->getManager();
        $em->persist($myResource);
        $em->flush();

        $routeOptions = [
            'id'      => $myResource->getId(),
            '_format' => $request->get('_format'),
        ];

        return $this->routeRedirectView(
            'get_myresource', 
            $routeOptions, 
            Response::HTTP_CREATED
        );
    }

And from the front end, something like this:

export async function createMyResource(important, info, here) {

  const baseRequestConfig = getBaseRequestConfig();

  const requestConfig = Object.assign({}, baseRequestConfig, {
    method: 'POST',
    body: JSON.stringify({
      important,
      info,
      here
    })
  });

  /* global API_BASE_URL */
  const url = API_BASE_URL + '/my-resource';

  const response = await asyncFetch(url, requestConfig);

  return {
    myResource: {
      id: response.headers.get('Location').replace(`${url}/`, '')
    }
  };
}

Now, the interesting line here – from my point of view, at least – is the final line.

Because this is a newly created resource, I won’t know the ID unless the API tells me. In the Symfony controller action code, the routeRedirectView  will take care of this for me, adding on a Location header pointing to the new resource / record.

I want to grab the Location  from the Headers returned on the Response and by removing the part of the string that contains the URL, I can end up with the new resource ID. It’s brittle, but it works.

Only, sometimes it doesn’t work.

Response
body:(...)
bodyUsed: false
headers: Headers
  __proto__: Headers
ok:true
status:201
statusText:"Created"
type:"cors"
url:"http://api.my-api.dev/app_dev.php/my-resource"
__proto__:Response

Excuse the formatting.

From JavaScript’s point of view, the Headers array is empty.

This leads to an enjoyable error: “Cannot read property ‘replace’ of null”.

Confusingly, however, from the Symfony profiler output from the very same request / response, I can see the header info is there:

Good times.

Ok, so the solution to this is really simple – when you know the answer.

Just expose the Location  header 🙂

# /app/config/config.yml

# Nelmio CORS
nelmio_cors:
    defaults:
        allow_origin:  ["%cors_allow_origin%"]
        allow_methods: ["POST", "PUT", "GET", "DELETE", "OPTIONS"]
        allow_headers: ["content-type", "authorization"]
        expose_headers: ["Location"] # this being the important line
        max_age:       3600
    paths:
        '^/': ~

After that, it all works as expected.

Programming Phoenix – Video to Category Relationship

I’m currently working my way through the Programming Phoenix book, which I have to say, I am thoroughly enjoying.

That’s not to say it’s all plain sailing. If only I were that good.

I got to chapter 7, and got myself quite stuck.

I’m aware the book was written for an earlier version of Phoenix, and in an attempt to force myself to “learn the hard way”, I decided that whenever I hit on a deprecated function, I would use the given replacement.

Ok, so in the book on page 99 (Chapter 6: Building Relationships), there is the following code:

  @required_fields ~w(url title description)
  @optional_fields ~w(category_id)

  def changeset(model, params \\ :empty) do
    model
    |> cast(params, @required_fields, @optional_fields)
  end

When using this code I got a deprecation warning:

warning: `Ecto.Changeset.cast/4` is deprecated, please use `cast/3` + `validate_required/3` instead
    (rumbl) web/models/video.ex:34: Rumbl.Video.changeset/2
    (rumbl) web/controllers/video_controller.ex:33: Rumbl.VideoController.create/3
    (rumbl) web/controllers/video_controller.ex:1: Rumbl.VideoController.action/2
    (rumbl) web/controllers/video_controller.ex:1: Rumbl.VideoController.phoenix_controller_pipeline/2

Seeing as I made the commitment to using the newer functions, this seemed like as good a time as any to do so.

The suggested replacement: cast/3 (docs) is where my confusion started. Here’s the signature:

cast(data, params, allowed)

And here’s the signature for cast/4 :

cast(data, params, required, optional)

You may then be wondering why in the original code from the book then that we only have three arguments being passed in?

Good question.

Really there are four, though one is being passed in as the first argument by way of Elixir’s pipe operator – which passes the outcome of the previous statement into the first argument of the next:

  model
 |> cast(params, @required_fields, @optional_fields)

As seen in the code from the Programming Phoenix book, cast/3 expects some params, then a list of required_fields, and another list of optional_fields.

One nice thing happening in the original code is the use of Attributes (@ symbol) as constants.

Using the ~w sigil is a way to create a list from the given string. Ok, so lots of things happening here in a very little amount of code. This is one of the benefits, and drawbacks (when learning) of Elixir, in my experience.

With PHPStorm I’m so used to having the code-completion and method signature look ups (cmd+p on mac) that learning and remembering all this stuff is really the hardest part. Maybe ‘intellisense’ has made me overly lazy.

Anyway, all of this is nice to know but it’s not directly addressing ‘problem’ I faced (and I use the inverted commas there as this isn’t a problem, it’s my lack of understanding).

We’ve gone from having required and optional fields, to simply just allowed fields.

My replacement code started off like this:

  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:url, :title, :description])
    |> assoc_constraint(:category)
    |> validate_required([:url, :title, :description])

Everything looked right to me.

I had the required fields, but now as a list, rather than the ~w sigil approach.

I had specified my three required fields by way of validate_required.

And as best I could tell, I was associating with the :category.

But no matter what I did, my Video was never associated with a Category on form submission.

I could see the Category was being submitted from my form:

%{"category_id" => "3", "description" => "Quality techno", "title" => "Richie Hawtin @ ENTER Ibiza Closing Party 2014, Space Ibiza",
  "url" => "https://www.youtube.com/watch?v=-5t2gH0l99w"}

But the generated insert statement was missing my category data:

INSERT INTO "videos" ("description","title","url","user_id","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5,$6) ...

Anyway, it turns out I was over thinking this.

All I needed to do was add in :category to the list passed in as the allowed argument and everything worked swimmingly:

  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:url, :title, :description, :category_id])
    |> assoc_constraint(:category)
    |> validate_required([:url, :title, :description])
  end

With the outcome of the next form submission:

INSERT INTO "videos" ("category_id","description","title","url","user_id","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5,$6,$7) ...
[debug] QUERY OK db=1.1ms

Easy, when you know how.

This took me an embarrassingly long time to figure out. I share with you for two reasons:

  1. Learning the hard way makes you learn a whole bunch more than simply copy / pasting, or typing out what you see on the book pages.
  2. I couldn’t find anything about this after a lot of Googling, so if some other poor unfortunate soul is saved a lot of headscratching from this in the future then it was all worthwhile writing.

That said, the new edition of Programming Phoenix is due for release very soon (so I believe), so likely this will have been addressed already.

Symfony === A Better Workflow

Probably the most requested feature for CodeReviewVideos right now is a search box.

Every time I get an email about this I feel like heading over to ShameNun.com and ringing the bell a few times. I know, I know, 2017, a site about programming, and I don’t even have a search bar 🙁

Well, the good news is: it is coming.

The bad news is: it’s not imminent.

I have a complete re-work of CodeReviewVideos living currently in my development environment. It follows all the practices I’ve been covering lately on the site:

There’s a bunch of other improvements in there too. Some are code related, others are things you will see when using the site.

Along the way, however, I’ve noticed I’m either going to have to take a risk with a server upgrade, or – more likely – do a side-by-side / DNS load balancer switcheroo. All of which adds to the complexity of the upgrade. This is a big part as to why this isn’t imminent. I don’t want to muck this up, so it requires a significant amount of planning.

Unfortunately back when starting the site I had only just started using Ansible, and whilst was happy to use it in my local / development setup, I didn’t have the confidence with it to manage my production servers. As such, CodeReviewVideos – in production – is not Ansible managed. Again, this will change with the upgrade, but it’s not quite as simple as running a single playbook and being bang up to date.

Anyway, it’s coming, so stay tuned 🙂

Video Update

This week there have been three new videos added to the site.

https://codereviewvideos.com/course/symfony-workflow-component-tutorial/video/creating-more-complex-workflows

First up we take a look at the example we will use throughout the rest of this series.

There isn’t as much code to learn with the Workflow Component as you might expect. There’s a few commonly used methods, some of which are also available in your Twig templates under slightly different names, but overall it shouldn’t take you that long to get familiar with the API.

What is harder – in my opinion – is the concepts behind the component. I would strongly recommend you take a look at this page:

http://workflowpatterns.com/patterns/control/

As this contains a bunch of useful ‘templates’ as to how you can put workflows together.

What I have found tricky is the naming conventions for transitions, and places. I would strongly recommend you follow along with this example, as we will cover on all the most commonly used transitions I have used in the real world. Getting hands-on is absolutely essential for building confidence in not only how to use this component, but also when it might be a good idea to do so.

https://codereviewvideos.com/course/symfony-workflow-component-tutorial/video/two-ways-to-make-life-easier-for-you-the-developer

As you will see a little later in this course, the Workflow Component dispatches a ton of events every time you successfully apply a transition.

Out-of-the-box, the Workflow Component comes with an Audit Trail Listener – a utility class you can hook up to start seeing stuff in your logs. This is fairly useful as a starting point. In a future video we will implement our own version of this Listener, as the generic one can only take you so far.

We’re also going to touch on the use of constants. As of Symfony 3.2 we can use PHP constants in YAML files (http://symfony.com/blog/new-in-symfony-3-2-php-constants-in-yaml-files).

The syntax is a little… well, it’s not my favourite. However, whilst using constants in your YAML files may look quite verbose, if you can get past this, the benefits do outweigh the negatives.

If you are using Twig then you can also use your constants in your templates too, so that’s another strong reason to do so.

https://codereviewvideos.com/course/symfony-workflow-component-tutorial/video/how-to-transitions-that-split-into-two-or-more-places

A really common workflow pattern I’ve found myself using in real world projects is the “AND-split” / “Parallel Split”.

My brain prefers the Friends style title: “the one where one place splits in to two, or more places”.

This has a few consequences further down the line – such as how to to merge back together, or not, and also how your state will be represented in your database.

I found this quite confusing when viewed as part of larger workflow diagrams, but fortunately when you break each transition down to their simplest forms, there’s really nothing scary going on here.

In this video we will also touch on how some of the workflow methods can trigger guards, which may result in some unexpected outcomes – but only if you aren’t aware of this, which of course you will be after watching this video 🙂

There’s still a good chunk to cover in this Workflow Component series, more of which is to come next week.

Until then, have a great weekend, and happy coding.

Chris