Comedy Errors

The last 4 weeks have been an epic journey. I started off with next to no Docker experience and have finished up with a working Docker-in-production setup.

Along the way I have learned an absolute ton of new stuff. But oh my Lord have I missed writing some code. Any code. Like, I have written absolutely no code in the last 4 weeks, and it’s killing me.

But anyway, I want to burst the bubble. Deploying Docker is not simple. At least, my experience has been anything but plain sailing.

Take for instance the obvious stuff, like having to learn the fundamentals of Docker – from the Dockerfile through to Volumes, and Networking. You need to know this stuff for development, let alone production.

And then there’s the less obvious. The “gotchas”. The ‘enjoyable’ headscratchers that left me stumped for days on end. Allow me to share a couple of comedy errors:

I wanted to run my API and my WordPress site on one of my Rancher hosts. This simply means I have a couple of Digital Ocean droplets that could potentially host / run my Docker containers, and I specifically wanted one of them to run both the API and the WordPress site.

Now, both sites are secured by SSL – a freebie cert from LetsEncrypt. So far, so good.

Both sites using SSL means both need access to port 443.

My initial impression was that I should be exposing port 443 from both containers, and we should be good, right?

No.

Only one of them can use port 443. The other will simply not come online, and the error isn’t very obvious.

No problem, Rancher has a Load Balancer. Let’s use that.

So I get the Load Balancer up, and with a bit of effort, both of my sites are online and I feel good about things.

“Where’s the comedy error?” you might ask. Good question.

This load balancer, it’s pretty useful. Shortly after getting the websites online I’m feeling fairly adventurous and decide to migrate from GitLab on a regular, plain old standalone, Ansible-managed Digital Ocean Droplet to a fully Dockerised, Rancher-managed ‘stack’.

What could possibly go wrong?

Well, it turns out… everything.

Sure, my backups worked, but they kinda didn’t. My Droplet used GitLab CE, but my Docker image is built from source. I don’t know specifically why, but I couldn’t get the two to play ball. No major loss, just 2 years worth of my GitLab down the drain.

I soldiered on. I got GitLab up and running, but this was when the real fun started.

To get my CI pipeline going I needed to host my Docker images inside GitLab. That’s cool, GitLab has a Docker Registry feature baked in as of a late minor release of the 8.x.x branch, and I was rocking GitLab 9.0.5. Also, this was something that was working on my trusty old Droplet.

Of course, this all went totally pear shaped.

It took me a couple of days just to get the SSL certificates to play ball. All the while, my GitLab server wouldn’t show me any of the Docker images I’d uploaded. Fun times. I mean, they were there, but GitLab was just having none of it. Where there any helpful error messages? Of course not.

Ok, I swear the comedy errors are coming really soon.

Anyway, GitLab and the Docker Registry need a bunch of open ports: 80, 443, 5000… and 22.

Sweet, I know how to work with ports, so I’ll just add a bunch more into my Load Balancer and everything “just works”, right? Well, after about 3 more days, yes, everything “just worked”.

Cool.

However, for some reason that totally escapes me now, I needed to SSH into the specific Rancher Host that had my GitLab instance on it. Not the container, the virtual machine / Droplet.

Lo-and-behold, no matter what I tried this just wouldn’t work. I was absolutely stumped.

In Linux-land, particularly on the server, it’s fairly uncommon to reboot. It rarely works quite as well as the usual “turn it off and on again” routine / meme would suggest.

But I tried it. And here’s the really weird part:

I could get in. Boom, I was in. But was it a fluke? I immediately logged out, and sure enough, I couldn’t log in. Same thing. Permission denied (publickey).

What the…

5 hours later I realised the mistake. I had redirected port 22 on the Load Balancer to my GitLab server so I wouldn’t end up with funky looking URLs for my git repos :/

Yep, port 22 wasn’t going to my Rancher Host. Instead, Rancher’s Load Balancer was forwarding port 22 on to my GitLab container.

ARGHHH!

5 hours. Seriously. I wish I was joking.

It’s this sort of thing that’s surprisingly difficult to Google for.

I promised a couple of comedy errors, so here’s the second:

I had gone full-scale cheapskate on my Rancher setup and opted for the 1gb Droplets. 1gb to run multiple instances of MySQL, nginx, PHP, RabbitMQ, Graylog, GitLab.

Yeah? No.

I wanted to push the limits of what Docker could do, and it turns out, Java inside a container is still Java. It will chew through a gig of ram in no time.

I wasn’t overly concerned by that. For you see, I had a cunning plan. And if Baldrick has taught us anything about cunning plans, it’s that they cannot fail.

Being the resourceful young man that I am, I provisioned a third node. A more beefy node. A node that was in fact, my very own computer. Chris, you genius!

Using Rancher’s “scheduling” feature I forced GitLab from the 1gb droplets to instead run exclusively on my machine. All was good.

Until yesterday when my ISP decided to soil itself:

Anyway, during that downtime some gremlins crawled into the system and started tearing apart important and sadly not very backed up pieces of my shiny infrastructure.

Late in the day, my ISP’s hard working network gophers fixed whatever network-related mishap that had broken the Internet and I got back online. Around that time I noticed my own computer was showing as “disconnected” in Rancher. A bit odd, seeing as I was online on this very box.

At this point I should mentioned that I had migrated to GitLab midway through my journey into Rancher. Most of the containers had been provisioned from the original standalone GitLab that I had been using for the last two years. These containers were available via port 4567.

My GitLab Docker container version used port 5000 instead.

Wouldn’t it have been a silly move to reprovision a bunch of my containers at this point, being that my Docker GitLab was playing up, and my old server was no longer available? Yep. Still, I did that anyway. Whoops. I managed to down 2/3rds of my infrastructure. Bad times.

You see, the new Registry container on port 5000 wasn’t up, and couldn’t come up. It needed access to a volume, but the volume was available via NFS on the other node, and connectivity to that node had been lost during the ISP-related downtime.

Not quite understanding the problem with my SSH / port 22 issue, I’d rebooted the other server anyway, and NFS decided not to come back online automatically. Compounded problems, anyone?

Of course, I’d deleted my old GitLab droplet by this point, so the old 4567 registry wasn’t available either… ughh.

Anyway, I’m sure reading this it all sounds… if not obvious, then not that bad. Yeah, I guess not, but like a decent financial savings plan, the real wins are found in compounding. And in my case, it wasn’t so much wins as losses. Many, many small details stacked up to turn a “this will only take me, at most, 2 weeks” to “over 4 weeks and just about getting there”.

It’s been a total mission.

I tell you this as I get so disheartened when I see others say things like this are easy.

It’s not easy. Very little of “development” is easy.

That said, I firmly believe if you have a stubborn persistence, there is actual enjoyment to be had in here somewhere πŸ™‚

Ok, that’s enough ranting from me.

Video Update

This week saw three new videos added to the site:

https://codereviewvideos.com/course/symfony-3-for-beginners/video/learning-a-little-more-about-forms

Again, like last week I will start you off here and let you follow through to the next two videos in the series.

We’re almost ready to get started with the Security portion of this course, where – in my opinion – things get a lot more interesting.

I will leave you this week by saying thank you to everyone who has been in touch – whether via email, comments, or similar – as ever, I am extremely grateful for all your feedback. Thank you.

Have a great weekend, and happy coding.

Chris

Docker In Production – Course Outline

This week I’ve been planning out the new Docker series. There’s a ton of stuff to cover, and I’m not sure whether to start out with the very basics, or dive right into the difficult stuff. One possibility is to split it into two (or more) course, covering different levels of detail.

What I am planning on covering is how to deploy the sort of stack that a small site might need. This includes:

  • Symfony 3
  • MySQL
  • nginx
  • Redis
  • RabbitMQ
  • Graylog

    Being that I love Symfony, it’s probably unsurprising that we are going to use Symfony 3 as the example back end. But in truth, you could swap this part out for anything you like. Hopefully with some of the problems we will encounter in getting a Symfony site online using Docker, then other similar setups will be covered with only minimal modification to the process.

    Then if you go with this, you’d probably want a separate front end site, so:

  • WordPress
  • MySQL
  • nginx

    The MySQL and nginx containers would be completely different from the Symfony stack ones.

    That brings up another issue then – sharing ports. You’re going to hit on issues if you try and re-use port 80 for serving two different nginx containers. At least, I did.

    The solution to this is using a Load Balancer. So we will cover this too.

    With a load balancer in place, we will add in LetsEncrypt to give us free SSL. This will cover both the Symfony stack and the WordPress stack.

    But wait, there’s more.

    We probably want a separate front end stack also. For the app that talks to our Symfony 3 API stack. So we will need to setup this too.

    There’s a bunch of fun challenges in each one of these steps. And by fun I mean “will cost you hours of your life” kinds of fun. Even simple stuff that you likely take for granted can bite you – e.g. how might you run a cronjob in Docker-land?

    Lastly, all of these services would be fairly painful without some kind of deployment pipeline. Thankfully, GitLab has us covered here, so we will also need to add a GitLab instance into our stack. Again, this will be dockerised.

    Oh, and managing this would be a total nightmare without some kind of container orchestration software. We will be using Rancher.

    So, yeah. Plenty to cover πŸ™‚

    But as I say, plenty to cover and that’s without mentioning the basics.

    In other news this week (and some of last), there’s a really interesting read around the forthcoming release of Symfony 4 –Β (start hereΒ http://fabien.potencier.org/symfony4-compose-applications.html)

    I really like the direction that Symfony 4 sounds like it’s taking. I cannot wait to try out Symfony Flex. I’m even going to get back to using Makefiles by the sound of things – though I can’t say I was in love with them the last time I had to use them. That was years ago, mind.

Video Update

This week saw four new, free videos added to the site.

These videos start here:

https://codereviewvideos.com/course/symfony-3-for-beginners/video/walking-through-the-initial-app

And are all part of the same course. This series is a slower paced introduction to Symfony 3, aiming to cover in a little more depth the big questions I had when first starting with Symfony in general.

Whilst we have covered how to create a simple Contact Form before on CodeReviewVideos, this time we are going to use it as the basis for a slightly more interesting series. Once we’ve built the contact form and had a play around with some extra things you might wish to do with it, we are going to move on and secure the contact form.

This will involve implementing some basic user management features – but crucially we are going to do so without resorting to FOSUserBundle.

Now, don’t get me wrong here. I think FOSUserBundle is fantastic and am extremely grateful for its existence. But you may have jumped in and added FOSUserBundle without ever taking the time to figure out how to implement login and registration for yourself. And so this series will aim to cover this.

Ok, that just about covers it from me this week.

Thanks to everyone who has been in touch since last week. I truly appreciate your feedback.

Until next week, have a great Easter Weekend, and happy coding.

Chris

Docker In Production: It’s a Go!

I want to start by saying thank you to everyone who got in touch regarding last week’s post on learning how to deploy with Docker. Judging from the feedback I received, I’m not the only one who has found deploying docker to be difficult.

Last week I mentioned how I was 11 days into my docker deployment exercise, and at that point the end did not seem to be in sight. Thankfully this week, although now at 17 days (at the time of writing), I can say “GREAT SUCCESS!”

The truth is, it has been an absolute pain to get this far. The short version of events is that I wanted to deploy a fairly standard Symfony 3 stack:

  • Symfony 3 JSON API
  • MySQL database
  • nginx web server
  • Lets Encrypt for lovely free SSL
  • RabbitMQ for queuing
  • Node w/ PM2 for queue workers

And I hit upon a fair few problems doing this with Ansible.

Now, don’t get me wrong: Ansible is fantastic.

There is – however – a major stumbling block with Ansible. If you make a mistake, reverting that mistake can be quite painful. My solution to this has often been destroy the server, and simply re-run the Ansible playbook(s) again until a working stack is back up and running.

But here’s the thing: This is terrible in production.

“No kidding, Chris” – everyone, everywhere.

I get this. It’s not a solution. The alternative is to jump onto the box, and start hacking at the config until you get a working system again. Only now you have a problem – you need to ensure your Ansible playbook / role setup will reliably reproduce this exact same setup. And that can be really, really tricky.

Pretty much the only way to guarantee you have nailed this is to, ahem, wipe the box and re-run the playbook.

See, in development you have a nice easy life, for the most part. If things go wrong, you can flatten everything, and start again. It costs you time, and time (as they say) is money. But overall, the cost is cheap.

As part of your development process you can add in extra pieces to your stack without much concern. Need RabbitMQ? There’s a role for that. Need Redis? There’s a role for that. Need a firewall? Sure, there’s a role for that – but do you really want to bother with a firewall in development?

And as a firewall in development seems a little overkill, I am often sorely tempted to skip this step. Hey, I have a lot on my plate and if I don’t need it right now, then I put it off.

In a similar vein, SSL in development is a total luxury. Without a public DNS record I can’t do any LetsEncrypt magic. I could do a self signing effort (there’s very likely an Ansible role for that, too), but what’s the point? If I use a self signed cert in dev, but LetsEncrypt in prod, now I have two different environments. Actually, using LetsEncrypt in prod, and not using it in dev is still two different environments, but somehow that feels … better?

Ultimately I have always ended up with two varying environments. Dev is a stripped down, and subtly different version of prod.

Anyway, back to my deployment woes. The straw that broke the camel’s back for me was in deploying the Node queue workers. The queue workers are fairly simple Node JS scripts. The gist of it is that they are long running processes that get given messages (think: JSON data) from RabbitMQ. They parse the JSON, and take some appropriate action.

As ever with software, things inevitably go wrong. When things go wrong with Node, it has the tendency to throw it’s hands up in the air and give up. My scripts behave like a pop star with a sore throat. PM2 is a process manager designed to sit and watch these processes on a 24/7 basis, and pro-actively restart any that die, for any reason. To continue the shaky analogy, PM2 is the venue manager, with two burly thugs waiting to duff up said pop star if the awaiting concert fans aren’t promptly entertained.

Before PM2 could do its thing, I needed a way to reliably get it installed on my servers. You guessed it – there’s an Ansible role for that.

Each role adds (yet another) layer of abstraction around whatever particular task you are trying to achieve. This layer means an extra bunch of potential steps that may or may not be causing any problems you might encounter. In my case, my problem was that whilst my code would deploy, and PM2 would manage it, it was always an out-of-date version.

To try and fix this, I started using Ansistrano (hey, another Ansible role!) to manage the deployment of my code, which triggered PM2 to hopefully reload my latest code changes… except it didn’t.

Somewhere along the way, I’d become many layers deep into a problem that I’d used all these tools to try and avoid.

In hindsight, I reckon if I’d stuck with the problem for the past two and a bit weeks, I would likely have found a solution. But the truth is, my confidence in my deployment process was at a low. With no traffic to the site, it wasn’t hurting any end users (thank the Lord!), but I could quite easily picture a future point where I had either deployed and made a mess, or worse, become so scared of deployment that I had essentially stopped new development to avoid the issue.

I know I’m not alone in this. In fact, I’ve worked with numerous organisations of many sizes that suffer from this very problem.

Staring me in the face, right there on the PM2 web site, was the section on “Docker Integration”.

Again, in hindsight, it probably would have made more sense to start small and Dockerise just the Node Script / PM2 deployment.

But that’s not what I did. No siree. Instead, I decided – to heck with this – let’s go the whole hog and move this entire stack to Docker.

It’s OK, we have hot failover

So yeah, it’s been a long, hard two (and a bit) weeks.

But I’m there. Well, 95% of the way there. See, very late yesterday evening it suddenly dawned on me that I’d neglected to include the WordPress blog which I’m using on the site root. Symfony is brilliant, don’t get me wrong, but use the right tool for the job. WordPress might get a lot of stick for its code, but the product that the end user sees is brilliant.

Putting WordPress into Docker should be easy right? There’s tons of tutorials on this. Heck, even the official Docker Compose tutorialΒ has a guide on how to do just this.

Actually though, it’s not easy. Sure, it’s different to using Ansible. Docker gets you up and running faster, there is no denying that. It also uses a whole bunch less memory and disk space than having to provision a bunch of virtual machines. But really, it’s just a different set of difficult problems.

Taking WordPress as the example: where do you store user uploads? What about new plugins?

These are things you don’t give much of a second thought to in a ‘traditional’ deployment. In Docker, these are headscratchers. You’re discouraged from using Volumes in production, but without them you can’t persist any data…

Most every tutorial I have found completely gloss over these trivial details :/

Anyway, after all this I do have some working solutions to these problems which I am going to share with you. From the plan I’ve made for this series it could become sprawling. I don’t want this to be the case. I’d rather keep it concise, but the truth is there is an absolute ton of stuff that you need to know to actually work with Docker.

This series is going to have a particular focus on deploying Symfony with Docker. We will also cover Rancher, a tool for managing your containers in production. In my opinion, after deploying Symfony with Docker you will have encountered a whole bunch of real world problems that make working with Docker on pretty much any other code base a lot easier.

I’d still love to hear from you if you are working with Docker in any way – whether just tinkering with it; working with it only in development; or have fully embraced Docker and put it into production. I have plenty more war stories to share, and whilst I can’t promise to have answers to your questions, if I can help in any way, I will do my best to try.

Video Update

This week there have been three new videos added to the site:

https://codereviewvideos.com/course/symfony-workflow-component-tutorial/video/new-in-symfony-3-3-workflow-guard-expressions

Towards the end of recording the Symfony Workflow Component Tutorial series I spotted an upcoming feature in Symfony 3.3, which is Workflow Guard Expressions.

As a quick recap, a Guard is a way of blocking a Transition. A transition is the process that objects going through your workflow will need to pass through to get from place to place.

Earlier in the series we covered creating separate Symfony Services to ‘contain’ your Guard logic.

With Guard Expressions you may be able to replace these standalone services with simple, human readable one-liners. It won’t always be the solution, but I’ve found them incredibly useful for two reasons:

* They cut down on code (less code, less bugs)
* They put your guards right inside your workflow definitions (easier to understand at a glance)

They are cool, and useful, and if you’re using the Symfony Workflow Component then I encourage you to check them out.

https://codereviewvideos.com/course/how-to-import-a-csv-in-symfony/video/setup-and-manual-implementation

and

https://codereviewvideos.com/course/how-to-import-a-csv-in-symfony/video/importing-csv-data-the-easy-way

In this two part series we cover a frequently requested concept – working with CSV data inside a Symfony application.

This series was created in response to a question from a site member regarding how to import CSV data, and turn it into related Doctrine entities.

When thought about as one process – importing and converting – it can be quite overwhelming. But if you break this process down into two tasks:

  • reading a CSV file;
  • converting each row into related Doctrine entities

Then it becomes a lot easier.

The approach given in these videos is a not intended for production use. You will need to expand on these concepts to fit your own needs. The aim here is to cover the high level, rather than “this is the implementation you should be using!”, which I disagree with. Every project is unique and has its own requirements. Use this as a possible source of guidance.

All of this weeks videos are free, and have been put up on YouTube also.

Thanks for reading, and have a great weekend.

Until next week, happy coding!

Chris

Would You Like To Learn How To Deploy With Docker?

At the time of writing, I’m now 11 days into my journey into Docker-ising my workflow.

The idea here is to build a continious integration / continuous delivery pipeline. It sounds horrendously enterprise, but in reality it’s all about removing the friction involved in getting working code into production as easily as possible.

As a quick re-cap, what I’m trying to achieve is roughly:

  • Develop locally using Docker;
  • Commit code, and push up to GitLab;
  • For each commit, GitLab will perform a checkout of my code, build up the environment (using Docker), and run my automated test suite;
  • If the tests pass, then the Docker image should be stored into my private GitLab Container Registry (or to you and me, store the Docker image on my private GitLab server, rather than publicly e.g. on dockerhub);
  • Assuming I am happy with everything, I can then pull this image to my production environment and replace the running container with the new one.

For this last step I’m looking at using Rancher.

Honestly, even just writing that out makes it sound tricky.

In practice, it’s been… a bloomin’ nightmare.

As I say, I’m 11 days into this. 11 days that is, without writing any code on the project that actually matters.

I genuinely believe this entire process will be worth it. Once it all works – and believe me, I will make this work even if it dangerously shortens my life expectancy – then I can roll this out not just to my current project, but to every project. And not just projects that use PHP either.

And that’s a huge win for me. That’s why I’m persisting with it.

I asked last week if anyone had any experience with this. Somewhat unsurprisingly, I got zero replies πŸ™‚

Now, this may be because no one reads what I write. It’s highly probable. However, MailChimp gives me stats and the stats say that at least some of you do read these emails πŸ™‚ Thanks!

Anyway, my take away from this is that if no one replied, either it’s because no one has experience doing this (and judging by how much pain I’m going through, I don’t blame you), or maybe it’s just not interesting to you.

I’d be up for doing a series on this for CodeReviewVideos once I have a working setup. It should shave a ton of time of the process if you are interested, but don’t have the time / desire to invest 11+ days of your life into a similar setup.

I would be extremely grateful if you could reply and let me know if this is a video tutorial series that would interest you.

I hope that in next week’s email I can say that I have a working solution.

Video Update

This week there have been three new videos added to the site.

I’m not going to link to each of the individual videos in this instance, but rather only to the first as they follow on logically from each other:

https://codereviewvideos.com/course/react-redux-and-redux-saga-with-symfony-3/video/registration-part-1

In this final part of the “React, Redux, and Redux Saga With Symfony 3” we wrap up by adding in Registration.

Previously when dealing with registration I have felt a little overwhelmed. From a very high level, it’s a form like any other.

But in reality it’s one of the most important forms on your site. It’s a core piece of the first impression a (potential) customer will have. You need to get it right.

Fortunately though, we’ve already done the vast majority of the hard work.

We have a set of re-usable components we can call upon, such as the repeated password entry field, and our styled form field.

We have a set process to follow, where we create a container, then a component to hold our form. We create our functions and pass them in, which will dispatch the right messages when interesting things happen. It’s all stuff we’ve covered throughout this series.

Finally we cover how to work with both the happy path, and the unhappy paths. We cover how to display helpful form errors to guide the user through as and when things go wrong.

And at the end of all this we have a working registration system that’s useful regardless of the type of app you’re creating. Without too much extra effort this system can be expanded upon to add in a payment processor – such as Stripe. I know, as this is exactly the same setup that’s being used in the forthcoming update of CodeReviewVideos πŸ™‚

Blog Update

This week I blogged about getting RabbitMQ working with Docker:

The Baffling World Of RabbitMQ and Docker

That’s it from me this week. Until next week, happy coding πŸ™‚

Chris