Not Once, Not Twice, But Thrice

We’re apparently in for a lovely weekend of sunshine here in the UK. Perfect weather for sitting indoors coding ๐Ÿ™‚

It’s been a busy week for me. I’ve had a long stint of travel, followed by busy days on a contract. In the evenings I’ve been fighting GitLab CI, which seems to have gone haywire. Builds are now taking hours to finish. I’ve unfortunately still not resolved this. There’s always some fun challenge to tackle.

On the plus side, thanks to the travel I did managed to see the Flying Scotsman here in Preston earlier this week:

Ok, enough ramble, on with the show…

Thrice Weekly

Over the last 4+ years of creating content for CodeReviewVideos.com I’ve tried a whole bunch of different approaches. I’m always trying to optimise my process and make things that touch more efficient, with the ultimate aim of delivering the most interesting and useful software development tutorials for you.

A number of people have asked why I “drip” out content. Typically, at least until the start of this year, I would release three videos a week: Monday, Wednesday, and Friday.

This approach worked well for me.

But, I got those emails frequently enough, asking why I didn’t either:

  • Release more videos, or;
  • Upload loads at once, rather than a few at a time

I had a definitive answer to the first question:

I can typically only get three videos done per week. Each video takes a lot of effort, from planning, to creating the initial code, to writing up that process, then recording, editing, uploading, and finally publishing. On average, for any minute you see on screen, it’s taken me about ~30 minutes to create.

The second question, however, I wasn’t sure on. The only way to find out would be to test the system.

As such, at the start of this year I decided I would switch to releasing blocks of content – an entire course at once, when it was done.

There’s a bunch of reasons why I don’t think this approach has worked. Probably the most disappointing for me, personally, is having had a number of subscribers cancel with feedback that I appeared to have abandoned the site. Heartbreaking.

Rather than dwell on this as a negative, I choose to take it as an experiment that didn’t work out. I’ve learned some lessons, and I know – more definitively, rather than just “gut feel” – that this is not the right process, for me.

As such, I have, in recent weeks, switched back to regular video updates. From now on, the previous approach of three videos per week will return.

Livestreams?

Something I’m considering at the moment is livestreaming things that I am working on. If you’ve ever watched Twitch, you’ll know the sort of thing I’m on about.

The main reason for this is to capture the thought process that I go through when writing code. I think this would be incredibly interesting to share. I do try to capture this when creating the more traditional content.

But even so, sometimes that exploratory stage holds big reasons as to why the code ended up as it did. This is much harder to cover in the traditional video approach.

For each of these livestreaming sessions I would record the screen as normal, along with the audio. I’m not sure how to capture chat as of yet, as the screen real estate is already extremely limited. I record at 1280×720, which is great for clarity of font etc, but severely limiting for real world dev. These things will need to be addressed.

This idea stemmed from this tweet:

There would also be no formal, written notes created for these videos. And each video would likely be ~1 hour long. I know this isn’t for everyone, but I’d be really grateful to hear your feedback on this idea.

Video Updates

This week saw three new videos added to the site:

Defining A Custom POST Route [API Platform]

One thing that I found initially confusing when working with the API Platform was in the creation of custom routes. In particular, in this video we address the issue of using a URI that differs from auto-generated route name / path that’s based off our underlying entity.

This is really useful, and I use this concept in every API Platform project I’ve created.

Finishing POST [API Platform]

In this video we finish up the POST implementation for our API Platform setup.

The number of videos required to get a POST endpoint working is a little misleading. We could have done this much quicker, but the Behat tests “dogfood” our API, and as such are making use of the POST endpoint also.

This is all about killing multiple birds with a single stone.

GET’ting One Resource [API Platform]

A major selling point, for me, of the API Platform is the rapid application development potential.

As mentioned above, the POST videos make this look a lot less rapid than it really can be. We had to take a care of a lot of setup / boilerplate for our testing in the previous few videos. Now we can spread our wings a little, and leverage a lot of the benefits that the API Platform provides in getting a brilliant API up and running, fast.

In the next few videos we will continue on with GET ‘ting multiple resources in one API call, PUT for updating existing resources, and DELETE for, well, I’m sure you can figure that one out yourself.

Have a Great Weekend!

Ok, that about wraps it up from me this week.

If you haven’t yet done so, please do come and say hi on the forum. It’s early days on there, but the discussions I’ve been involved with so far have been good fun. Here’s to more of them ๐Ÿ™‚

Until next time, have a great weekend, and happy coding.

Chris

How I Fixed: argument “$entityManager” of method “__construct()” references class “Doctrine\ORM\EntityManager” but no such service exists

Ok, mega crazy title. And honestly, this is just the tip of the iceberg. Allow me to set the scene:

Lately I have had email conversations, read threads on hackernews, and even had a forum post challenging how and why I do things the way I do.

The summary of the email conversations being why I persist with Symfony / PHP generally, when other, “better” solutions exist. And the same can be said for the linked forum post.

And then yesterday I saw that linked Hacker News thread:

It was at about ~320 comments when I read it. The top reply was the most interesting for me:

There’s a bit more to it than that, and the thread itself is worth a read. There’s basically 400+ different suggested ways to “get a web app up quickly in 2018”. I’d disagree with a bunch of them, but then, they are the way I do things.

Wait, what?

Yeah, I’d disagree with Docker + Ansible + Terraform + nginx + (Symfony/Rails/Go/etc) + Postgres, etc, being quick to get up and running.

Sure, once you know the drill / have projects to copy / paste from, it can be quick, relative to the first time you had to learn and implement all this stuff. But it’s not quick quick. It still takes me ages.

And so I challenged myself: Just how quickly could I get a typical project up and running for myself? The perfect question for a Saturday night.

My Setup

The setup I most typically use is:

  • Terraform for spinning up a server
  • Ansible for prep’ing the box
  • Docker for running stuff
  • GitLab for code hosting + CI
  • nginx for my web server
  • Symfony / PHP 7 for the code
  • Postgres for the DB

This is a lot of stuff, and it’s not super quick to set up.

This is why I started by mentioning the email / forum conversations whereby people ask: is Symfony / PHP the best tool of choice?

Well, maybe not. I don’t know. I just know I’m more productive with Symfony and PHP generally than everything else – though JavaScript is a close second.

Over the past few years I’ve tried other setups. It’s hard to invest time in learning another stack when the end result may be basically identical – what did I gain from the time invested? Could that time have been better invested elsewhere? Hard questions to answer.

But yeah, Node and more recently, Golang have been stronger contenders than usual for my attention. Anyway, that’s a bit of a digression.

The Problem

As mentioned above, that’s my stack. Learning it all took ages (years?), but as each project is, from an infrastructure point of view, very similar, I can now spin up a new environment very quickly.

My challenge was to find out how quickly. I got most of the core stuff up and running in ~1.5 hours.

I didn’t get the Behat testing environment set up in that time. Because I hit on an issue.

I wanted a simple JSON API as the outcome of this process. By simple I mean basically CRUD.

With the basic stack up and running, I created a basic entity (one property), and updated the DB accordingly. Doctrine was used for DB interactivity. Again, very typical for my projects.

In order to get data out of my repo, I needed to create a repository. There’s an awesome post on this by Tomas Votruba called How to use Repository with Doctrine as Service in Symfony.

As a side note here: if you haven’t already, I would highly recommend reading Tomas’ blog, as it’s jam packed with things you’d likely find very useful and interesting. Also, check out his GitHub projects, with Rector in particular being incredible.

I followed the linked article, and hit upon the following:

What was weird to me at this point is that I’ve followed this article before, but never hit upon any problems.

Anyway, I did as I was told – I switched up the code to reference the EntityManagerInterface instead:

This is a really simple class.

For complete clarity, here’s basically the rest of the app at this point:

And the entity:

This is basically a generated entity with a couple of tweaks. It’s not the final form, so don’t take this as good practice, or whatever.

The purpose of what this class is supposed to do is also not relevant here, but will be discussed in a future video.

Anyway, the problem is evident in the code above. If you can spot it, then good stuff ๐Ÿ™‚

If not, keep reading.

So with three records in the DB, all the connectivity setup, things looking decent, I sent in a request to my only endpoint – GET /.

And it didn’t work. I hit a 504 Gateway Timeoutย  error from nginx.

Very confusing, overall. I mean, this is basically copy / paste from a different project that works just fine. Only, I’ve renamed the project name. What the heck?

I hit refresh a few times, you know, to make sure the computer wasn’t lying to me. And then everything started going unresponsive. Very odd. I’ve just bumped the system from 16gb to 32gb, and all I have is a few Docker containers running, a browser with admittedly too many open tabs, and one instance of PHPStorm. Surely this couldn’t be taxing the system. htopย  told me a different story:

Yeah, I know, that swap size is ridiculous. Forgive me.

The nginx logs weren’t really that helpful. I needed to look at the PHP log output, which in this case is achieved via docker logsย :

Line 23 ofย  TemporaryEmailRepositoryย  is:

I mucked around a bit, trying out injecting the ObjectManager instead, but hit the same issue.

Then I wondered if it was the act of injecting itself, or actually using the injected code (durr). So I commented out the call:

Reloading now, I was no longer seeing the massive RAM spike, and looking at what that call was doing pushed me down the right lines.

I’ll admit, it took me a much longer amount of time than I’d of liked to realise my mistake:

Now, I’m not 100% certain on the conclusion here, but this is my best guess.

I believe I had created a circular reference. I’d injected the Entity Manager into the repo. Immediately I’d asked for the entity. The entity has an annotation pointing at the repo, which triggered the endless loop.

Anyway, removing the repositoryClass attribute fixed it up. Kinda obvious in hindsight.

The Conclusion

I’m convinced I could get an environment up faster than this. Without hitting this issue I believe I would be at the ~2 hour mark to go from idea to having a solid setup that’s good to write code in a sane, reproducible, reliable / testable way.

I think back to 10+ years ago, where I’d be up and running so much faster. PHP is essentially a scripting language. With shared hosting, you’d have the DB ready, the web server ready, you just needed to write a bit of code, connect to the DB, push the code up somehow (FTP :)) and bonza, you’re up and running.

Looking at that way now, I’m amazed how far I’ve come. There’s a massive overhead with using frameworks – time spent learning (which never stops, unless your framework of choice goes EOL), patching, managing all this stuff, learning new ways to make things better… is it all worth it? I think so.

I think the biggest takeaway for me lately is that whilst within the last ~5 years I’ve shipped a lot less code to prod than in the 5 years preceding this, the code I do ship is more stable, and maintainable.

Nagging in my mind, however, is that what’s the point in this slow, methodical approach if the end result is it takes so long, I either don’t bother with entire ideas, or by the time I’ve shipped them, I’m so burned out by the seeming complexity of the whole thing that I lose interest in taking them further.

Anyway, I appreciate this is half helpful, half rant. I just needed to blog it and get these thoughts out of my head.

My GitLab Runner Config.toml [Example]

I hit on an annoying issue this week, which I’m not sure of the root cause.

Last week I bumped GitLab from 10.6, to 10.8, and somehow broke my GitLab CI Runner.

Somewhere, I have a backup of the config.toml file I was using. I run my GitLab CI Runner in a Docker container. I only run one, as it’s only for my projects. And one is enough.

Somehow, the Runner borked. And annoyingly I neither had a reference of the running version (never use :latest unless you like uncertainty), and recreating without the config.toml file has been a pain.

So for my own future reference, here is my current GitLab Runner config.toml file:

FWIW this isn’t perfect. I’m hitting on a major issue currently whereby GitLab CI Pipeline stages with multiple jobs in the stage are routinely failing. It’s very frustrating. It’s also not scheduled for fix until v11, afaik.

PayPal: Done; Discourse: Done; What’s Next?

I hope this post finds you well. It’s so lovely and sunny here in the UK, it’s (almost) a shame to be inside coding. I guess that’s what laptops are for, right? So you can still code whilst sat in the garden.

The last five weeks have been incredibly busy for me. Aside from starting a new role, I’ve managed to finally finish off a couple of big tasks that have been on my TODO list for waaaay too long. These are:

  • adding PayPal to the site
  • migrating away from Disqus commenting

I want to quickly cover both.

Adding PayPal

If you’ve been getting my mailings / reading the blog for any length of time, you’ll likely be sick of hearing about PayPal. When switching from the old Symfony-only implementation of CodeReviewVideos (version 1), I knew I’d want to offer more than just Stripe as a payment option. Therefore, I dutifully planned ahead and made the process of accepting payments as modular as possible, and made Stripe payment processing just one possible implementation.

This all worked absolutely fine. I was plenty comfortable with Stripe already, and had my original implementation to use as a reference.

What I did wrong, in hindsight, was base my implementation too heavily on Stripe.

To be clear, Stripe get a lot of things right. If you have to accept payments, working with Stripe’s API is a joy. It’s hard not of be influenced by how their system works.

As such, some of the ways I implemented things like the card details endpoint, the invoicing, and even little things like what data was being captured if using a debug level of logging were too heavily tied to Stripe.

The new subscribe form with PayPal option chosen

These things combined to make adding Braintree integration (aka PayPal’s API) take a lot longer than planned (~6 months, to my estimate of about 2-4 weeks). There were some other complications, such as getting my account approved was perhaps something I should have done upfront, but instead, I left this until I was about 8 weeks into development. In hindsight, if they had declined my application, I’d have wasted a lot of time. Not to mention, when I finally thought I might get rejected (it took a while, I got fearful) I stopped development entirely – for about 2 weeks.

The biggest mistake I made though was in the DB schema. Even though I knew upfront that I’d ultimately want to allow people to subscribe with PayPal, or Stripe, I made the relationship between a User account, and a Payment Information a one-to-one.

This was deployed to prod.

All worked fine when all I had was Stripe.

The problem dawned on me that if a User was paying with Stripe, then canceled their subscription, then rejoined with PayPal, then canceled again, and rejoined with Stripe, there was no way to get their previous payment info back. It sounds like an edge case, but if I’ve learned anything from CodeReviewVideos, it’s that all manner of unexpected circumstances can, and do arise. And more frequently than I’d ever have thought.

There was another issue. If a User was paying with Stripe, and then switched to PayPal, with a one-to-one setup they would lose their Stripe invoices. Again, major headaches.

So even when I’d finished the development, I still had a major migration ahead of me. And that consumed about 4 weeks in terms of planning, writing migration scripts, testing, setting up a whole new environment to test the thing end to end… phew, the work just kept on, and on.

Anyway, to cut a long story short, it’s done. The migration went well, and PayPal is now in prod. I think I celebrated by immediately cracking on with the Disqus migration. Ha.

Migrating To Discourse

One aspect of the CodeReviewVideos web site that I’ve never been happy with has been the use of Disqus.

There was a nasty user experience whereby you’d have to sign up once to use the site, and again – and entirely separately – to leave a comment. It sucked. But as far as pragmatic solutions go, it was good enough to get going.

I also read that Disqus would be enabling adverts on my comments section – though to the very best of my knowledge, that never happened. There was talk of a monthly fee. I don’t know. I don’t begrudge them charging for their service, but that wasn’t for me.

Adding Disqus wasn’t super easy, but at the same time, it wasn’t quite as hardcore as the PayPal change.

Surgical

The complications came by way of Single Sign On, hosting Discourse (via Docker), and replacing the existing comments.

Single Sign On wasn’t as bad as I’d expected. I thought that would be the hardest part. I found a Laravel package which I butchered crafted with surgical precision into something that worked nicely with Symfony.

Hosting Discourse wasn’t too bad. I use Rancher for my Docker container management, but Discourse’s Docker implementation just wasn’t playing ball. In the end I got a new VPS from Hetzner and hosted it there instead. There were some tweaks needed, but overall it wasn’t so bad.

Replacing the existing comments was the real tricky part. Disqus provide a one-way export – something I think is a bit weird. By which I mean you can export your data from Disqus, but they won’t let you revert to a previous ‘backup’. Anyway, I didn’t need that, I just needed the export, so that was fine and dandy.

Once I had the export I needed to get that data in to Docker, and then tweak the provided Disqus import script to run against my export. That all generally worked, but it only seems to have missed some of the comments off. I’m not sure why, but also I felt the end result was “good enough”.

The import worked by looking for any existing user, and then mapping a Disqus email address to the user’s Discourse email address. If the Disqus commentor never had a site membership, then now their comment will be assigned to some anonymous username like Disqus_312jklsdf2klย  or whatever. Not perfect, but again, good enough.

Now what happens is when I create a new video, the comments section automagically creates a new forum post under the username chris. As such, if you look at the forum today (and you should, because it’s ace), you’ll see I’ve been posting new topics like a mad man. This will slow down over the next few days.

As I write this I still have email functionality globally disabled on the forum. This will change, possibly over the weekend, once I’m suitably confident everything has settled down. You may recall receiving an email from the staging forum a few weeks back – yep, I made a boob there. Sorry about that. Once bitten, twice shy.

What’s Upcoming

I mentioned at the start that aside from the PayPal and Discourse changes, I have also recently started a new role.

Sometimes I get emails asking why I don’t have any new videos in a while, or why I haven’t updated X, Y, or Z to the latest and greatest. Believe me, I’d love to spend all day making new videos. Unfortunately CodeReviewVideos is not my full time job.

As some of you may know – I’m fairly open about it – I am a contractor by day.

One of the really nice things about being a contractor is getting to experience lots of different projects, in various scales of complexity, and reliability ๐Ÿ™‚

I get a lot of interesting ideas for videos from my day-to-day work experiences. And this means that the content on CodeReviewVideos is very much about actionable, real world stuff you can use right now, today in your projects. It’s also battle tested / used in real world production websites.

There’s a bunch of videos I wrote up but haven’t had time to record as of yet. The reason I mention the whole day job thing is that it means I have only a fixed amount of time per week to devote to the site, and I have to prioritise tasks accordingly. Above all else, I prefer making videos. This is why I stared, and continue to run CodeReviewVideos. I love sharing, and from the feedback I get from so many of you (thank you!) you find it useful, too.

But of course, over the last few weeks I’ve had these other big site changes (dare I say, improvements) to make. And that has meant I haven’t been recording. Thankfully all that can change, and I can get back to making recording new stuff.

Ok cool – so what should be being recorded, all being well, is the continuation of the Beginners Guide to Back End (JSON API) + Front End Development [2018] series. Next up is the API Platform section, which I both really enjoyed building, and writing up, and am equally looking forwards to recording and sharing. It’s a good one.

Let’s hope this sunshine continues through the weekend, and for those of you in the UK, enjoy your bank holiday / extended weekend.

One final thing before I go: please do come and say hi on the forum.

Until next time, happy coding!

Chris

Site Update, and GDPR

If you’ve been to the site today you may be wondering just what the heck has changed. And visually, not a lot ๐Ÿ™‚ But behind the scenes…

I’ve been working on the PayPal update for a good long while now. I estimated two weeks, and it turned into 6 months, give or take. Fun times.

A big part of why the PayPal update was so long in the making was that I needed to make a major structural change the DB. Originally I had planned for multiple payment providers when switching from the Twig approach, to the new API / hybrid approach.

All that’s cool, and works, but I made a boob in setting the mapping between a User account and a “Payment Information”. I made it one to one, whereas it should have been one to many. Oopsie daisy.

There’s probably a bunch of ways I could have addressed this problem. The way I chose was to have two copies of the database active, change the schema in the prod db, then query the old DB for the relevant info, and write that back to the new schema.

With a change like that I knew there would be downtime. I had a big checklist created on what to do, and how to do it, and largely it all went well.

Somewhere along the way, I decided I also wanted to add a forum. You may have received an email from the forum a few weeks ago. If you did, the link wouldn’t have worked – my mistake, so apologies about that. As it stands, this work is a precursor to the forum work, more on that in another email.

Anyway, I decided to split the PayPal work into two chunks. The first being the DB schema change (now complete), and the second, smaller piece being to enable PayPal. That will happen sometime this weekend. Right now I’m done changing stuff ๐Ÿ™‚

The biggest change today has been that now anyone can sign up without needing a credit card. Over the next few weeks, some videos will require you to log in / have a free account to watch.

The existing paid content will remain only available to paying subscribers.

The primary reason for the new free account structure is for the forum.

To be completely clear – for paying subscribers, nothing is changing.


Heads Up: If you experience any strangeness / unexpected errors, please first try logging out and back in again. This update moved the API from Symfony 3 to Symfony 4, and involved a complete new Docker stack. If problems persist, get in touch via the support form, or email.

GDPR

There will be some forthcoming revisions to the terms and conditions, and privacy policy regarding GDPR.

For what it’s worth, I don’t have much tracking. On the site I have Google Analytics (which I use), and Facebook Pixel (which I don’t, but everyone tells me I should).

I also track anonymous video interaction. In other words I know what videos are being watched, and for how long, but I don’t know who by. Personally I don’t like being tracked, and whilst it’s probably detrimental to the success of the site, I have the luxury of building CRV how I want, not how some slimey marketing guy dictates.

The gist of the change is simply this:

You always have the right to request your data. And you always have the right to request I delete any data I have about you.

For existing subscribers who add their name / company details for printing invoices, I wipe these details as and when your paid subscription ends.

Moving forwards I will be deleting inactive user accounts older than 90 days. You will receive an email asking you to log in to confirm you still want your account. If you don’t log in, the account is deleted. With forum integration, this will lead to anonymising your post, but the post will remain.

If you have any questions on this then please reply, email me directly, or use the support form.

Site News

I have been adding videos as regularly as time permits. There have been over 30 new videos uploaded since I last emailed (~6 weeks ago).

Video updates will continue as ever, primarily focusing on the Beginners Guide to Back End (JSON API) + Front End Development [2018] course. I have a bunch of other ad-hoc videos to share, and ~30 new videos written up waited to be recorded.

If you would like to make videos and share them on CodeReviewVideos.com then please get in touch. Beginners are very much welcome, and there are a bunch of benefits to doing this. If at all interested, hit reply and get in touch.

Have A Great Weekend

Ok, that’s about it from me. I’m off out to enjoy the sun.

As ever, have a great weekend, and happy coding.

Chris