It Shouldn’t Happen In Production

I wrote last week largely about some “comedy errors” that I had hit upon along my way when Deploying with Docker. I want to share another one this week.

But before I do, I want to quickly touch on a conversation I had with a friend and fellow developer earlier this week.

I was talking about last week’s email, and how I had shared a few of my blunders. This shocked my friend. He found it strange that I would tell others about my mistakes.

Now, let me tell you – everyone makes mistakes. It doesn’t matter whether you’ve been developing software for 1 week, 1 year, 10 years, or whatever. It also doesn’t matter what skill level you are at. Mistakes happen.

To put it another way, if we made fewer mistakes as we gained more experience, StackOverflow wouldn’t even be a thing.

Of course, the issues we encounter as we gain more experience tend to be a different sort of problem to those we make earlier in our careers. Hey ho.

I want to state publicly that I share my mistakes because a) they are often funny in hindsight, and b) they – hopefully – dispel the myth that life as a developer is easy. If there’s one thing that I find really disheartening when working with beginners it’s that they frequently believe that I never hit upon problems. and honestly, that could not be further from the truth.

Ok, so onto this week’s issue.

If you are a regular reader of this newsletter, you will know that I am on a journey to change my infrastructure from Ansible to Docker.

It’s been a fairly long, stressful, but ultimately rewarding experience. As a heads up, there is a full course coming on this, which I think is going to be really useful to a huge number of people, and not just Symfony / PHP devs.

When getting started with Symfony, you will likely (and rather quickly) come across the dev.log file. And in production there is the corresponding prod.log file. These log files keep a track of all the issues we hit upon that help debugging our code.

As your infrastructure grows in complexity, a single log file might not be as useful to you. If you have two Symfony servers behind a load balancer, it can be an extra, annoying step to have to probe about on two boxes to find the right log file, and then track down the issue in that file. Or worse, piece together a bigger picture from both files.

There’s a better solution to this problem: a centralised logging server.

An example of this would be Graylog.

This is fairly easy to setup. You need to have a running Graylog server (or in Docker-land, a Graylog container, a Mongo container, and an Elasticsearch container), and then with a few extra lines in your Monolog config, you are good to go.

Or are you?

Well, sort of.

See, yes you are good to go, and logs start arriving in your Graylog dashboard, and the World seems good.

Only, what if you put your Graylog instance on one server, and then your PHP stuff lives on another server? Should all work still, right?


Where the heck am I going with this then?

Well, it turns out when connectivity between these two boxes goes down – as it did to me earlier this week – then not only does the Graylog server stop receiving logs (as I kinda expected) – but the Symfony sites started throwing 500 errors as they couldn’t write to the Graylog server… ARGHHH?!?!

Now, I’m not saying this is because I am now using Docker. This problem occurs if you are using Ansible too (I know this because I checked it). And in hindsight, it’s somewhat obvious. With a little more time, I’m hoping to put in a PR to fix this, as I’m fairly sure it’s a bug. I’m hoping a simple try / catch might do the trick, but I haven’t taken a look at the underlying code yet, so that’s a total guess.

Anyway, it turns out there may be a better solution to this problem. The ELK stack. That’s Elasticsearch, Logstash, and Kibana.

As I understand it, rather than pushing logs out from the server as they occur, you instead write to your log files as normal, and then Logstash parses the log files and ships them off to Elasticsearch. I’m not 100% on this, it’s just something I have started investigating by way of this:

All this comes back to my original point. If mistakes didn’t happen to developers with a little more experience then ELK and Graylog and the myriad other alternatives wouldn’t exist. And I guess that would be good for me, as then I wouldn’t have encountered this error…

Anyway, I was going to cover setting up Graylog as part of the forthcoming Deploying with Docker course, but I might switch this now to ELK. It really depends on which performs more reliably given the conditions. Time will tell.

Video Update

This week there have been three new videos added to the site.

As a quick side note – and something I hadn’t realised at the time – this takes the number of videos on the site to 409, spread over 55 courses.

This week we continue on with the “Symfony 3 For Beginners” series, this time moving into securing our site and adding our first user. I’ve tried my very best to make this as easy to understand as possible. I appreciate security is perhaps the most complicated (and scary) part of the whole framework.

To quickly recap, we aren’t making use of FOSUserBundle in this series. Even though FOSUserBundle is widely used in so many live Symfony projects, it is super important to be able to confidently put together a similar setup, or you won’t understand what FOSUserBundle is doing for you “behind the scenes”, so to speak.

My theory is that if you can come away from this course with confidence in creating a secured Symfony site, then almost everything else you will need to do with the Symfony framework should be easier (by comparison).

Until next week, have a great weekend, and happy coding 🙂


Published by

Code Review

Code Review

CodeReviewVideos is a video training site helping software developers learn Symfony faster and easier.

Leave a Reply

Your email address will not be published. Required fields are marked *