I hit on an annoying issue this week, which I’m not sure of the root cause.
Last week I bumped GitLab from 10.6, to 10.8, and somehow broke my GitLab CI Runner.
Somewhere, I have a backup of the config.toml file I was using. I run my GitLab CI Runner in a Docker container. I only run one, as it’s only for my projects. And one is enough.
Somehow, the Runner borked. And annoyingly I neither had a reference of the running version (never use :latest unless you like uncertainty), and recreating without the config.toml file has been a pain.
So for my own future reference, here is my current GitLab Runner config.toml file:
FWIW this isn’t perfect. I’m hitting on a major issue currently whereby GitLab CI Pipeline stages with multiple jobs in the stage are routinely failing. It’s very frustrating. It’s also not scheduled for fix until v11, afaik.
This course will cover building a JSON-based API with the following back-end stacks:
‘raw’ Symfony 4 (PHP)
Symfony 4 with FOSRESTBundle (PHP)
API Platform (PHP)
Koa JS (JavaScript / node)
Behat will be used to test all of these APIs. One Behat project, four different API implementations – in two different languages (PHP and JS).
We’re going to be covering the happy paths of GET , POST , PUT , (optionally) PATCH , and DELETE.
We’ll also be covering the unhappy paths. Error handling and display is just as important.
Where possible we’re going to try and use just one Behat feature file. It’s not always possible – the various implementations don’t always want to behave identically.
There’s a ton of good stuff covered in these videos. But the back end is only half the battle.
Whether you want to “catch them all”, or you’re working with a dedicated front-end dev, it’s definitely useful to know the basics of both.
With that in mind, you can pick and choose whether to implement the back-end, or front-end, or both.
If you don’t want to implement a back-end yourself, cloning any of the projects and getting an environment up and running is made as easy as possible by way of Docker. But you don’t need to use Docker. You can bring-your-own database, and do it that way, too.
The Front End
Whatever back end you decide to spin up, the front end should play nicely.
We’re going to implement a few different front-ends. The two I’m revealing today are:
‘raw’ JavaScript
React
I have plans for a few others, but each implementation is a fair amount of work and I don’t want to over promise at this stage. There’s definitely at least two more coming, but let me first get these two on the site 🙂
The raw JavaScript approach aims to show how things were in the ‘bad old days‘. The days before your package manager would take up ~7gb of your hard disk with its cache directory.
The benefit of working this way is that there’s really no extra ‘stuff’ to get in the way. We can focus on making requests, and working with responses.
But that said, this is 2018 and the many modern JavaScript libraries and frameworks are fairly awesome. You’ll definitely get a renewed sense of appreciation for how much easier your life is once you’re comfortable using a library like React, after having done things the hard way.
Again, as mentioned we will cover more than just raw JS and React. Currently each implementation is between ten and fifteen videos. Each video takes a couple of hours to write up, and another couple of hours to record on average. I’m going as fast as I can, and will upload and publish as quickly as possible.
Behind the scenes over the past 10 weeks I have been working on integrating CodeReviewVideos with Braintree.
This is to enable support for PayPal.
I tried to create a ticket for everything I could think of ahead of starting development.
And I added a new ticket for any issue I hit during development. I’m not convinced I tracked absolutely everything, but even so I completely underestimated just how much work would be involved in this feature.
Being completely honest, I have never been more envious of Laravel’s Spark offering. For $99 they get Stripe and Braintree integration, and a whole bunch more. Staggering.
There’s a bunch of other new and interesting features in this release.
I’ve taken the opportunity to migrate from Symfony 3 to Symfony 4 for the API. There’s a bunch of new issues that arose during this transition – I hadn’t given it much prior thought, but with the new front controller (public/index.php) totally broke my Behat (app_acceptance.php) setup.
This work is also enabling the next major feature which I will start work on, once PayPal is live. More on that in my next update.
I appreciate that from the outside looking in, there doesn’t seem to have been a great deal of activity on the site over the last few weeks. I can assure you that behind the scenes, there has never been more activity.
Have A Great Weekend
Ok, that’s about it from me for the moment.
As ever, have a great weekend, and happy coding.
p. s. – I would be extremely grateful if you could help me spread the word by clicking here to tweet about the new course.
I love me some Dockerised GitLab. I have the full CI thing going on, with a private registry for all my Docker images that are created during the CI process.
It all works real nice.
Until that Saturday night, when suddenly, it doesn’t.
Though it sounds like I’m going off on a tangent, it’s important to this story that you know I recently I changed my home broadband ISP.
I host one of my GitLab instances at my house. All my GitLab instances are now Dockerised, managed by Rancher.
I knew that as part of switching ISPs, there might (read: 100% would) be “fun” with firewalls, and ports, and all that jazz.
I thought I’d got everything sorted, and largely, I had.
Except I decided that whilst all this commotion was taking place, I would slightly rejig my infrastructure.
I use LetsEncrypt for SSL. I use the LetsEncrypt certs for this particular GitLab’s private registry.
I had the LetsEncrypt container on one node, and I was accessing the certs via a file share. It seemed pointless, and added complexity (the afore mentioned extra firewall rules), which I could remove if I moved the container on to the same box as the GitLab instance.
I made this move.
Things worked, and I felt good.
Then, a week or so later, I made some code changes and pushed.
The build failed almost immediately. Not what I needed on a Saturday night.
In the build logs I could see this:
Error response from daemon: Get https://my.gitlab:5000/v2/: received unexpected HTTP status: 500 Internal Server Error
This happened when the CI process was trying to log in to the private registry.
After a bit of head scratching, I tried from my local machine and sure enough I got the same message.
My Solution
As so many of my problems seem to, it boiled down to permissions.
Rather than copy the certs over from the original node, I let LetsEncrypt generate some new ones. Why not, right?
This process worked.
The GitLab and the Registry containers used a bind mounted volume to access the LetsEncrypt cert inside the container on the path /certs/.
When opening each container, I would be logged in as root.
Root being root, I had full permissions. I checked each file with a cheeky cat and visually confirmed that all looked good.
GitLab doesn’t run as root, however, and as the files were owned by root, and had 600 permissions:
Thankfully I hadn’t deleted the old cert, so I went back and saw that I had previously set 0640 on the private key in the old setup.
Directory permissions for the certs was set to 0750 with execute being required as well as read.
In my case this was sufficient to satisfy GitLab.
When making the change on the new node, I could then immediately log back in.
A Tip To Spot This Sooner
I would strongly recommend that you schedule your project to run a build every ~24 hours, even if nothing has changed.
This will catch weird quirks that aren’t related to your project, but have inadvertently broken your project’s build.
It’s much easier to diagnose problems whilst they are fresh in your mind.
Also, ya’ know, better documentation! This is exactly why I’m now writing this post. So in the future when I inevitable make a similar mistake, I now know where to look first 🙂
In a bid to make getting up and running with CodeReviewVideos tutorials moving forwards, I have created a new repo called the Docker and Symfony3 starting point.
I will be using this as the basis for all projects going forwards as it dramatically simplifies the process of setting up each tutorial series, and should – in theory – make reliably reproducing my setups much easier for you.
I’ve had a lot of feedback lately to say it’s increasingly hard work to follow some of the older tutorials as the dependencies are out of date.
Taking this feedback on board, I will do my best to update these projects in the next few weeks-to-months. I am, however, going to wait for Symfony 4 to land before I spend any time updating stuff. No point duplicating a bunch of effort. Symfony 4 drops in November, in case you are wondering.
Video Update
This week saw two new videos added to the site. Unfortunately I was ill early in the week so didn’t get to record the usual third video.
Imagine we have some expensive operation: maybe a heavy computation, or some API request that needs to go across the slow, and unpredictable internet.
Wouldn’t it be great if we did this expensive operation only once, saved (or cached) the result, and then for every subsequent request, we sent back the locally saved result.
Yes, that sounds awesome.
Symfony has us covered here. The framework comes with a bunch of ways we can cache and store data.
Redis seems to be the one I find most larger organisations like to use, so that’s one of the reasons behind picking Redis out of the bunch.
At this point you may be thinking:
But Chris, I don’t have a Redis instance just laying around waiting for use, and I’m not rightly sure how I might go about setting one up!
Well, not to worry.
To make life as simple as possible, we are going to use Docker.
Docker… simple… the same sentence?!
Well, the jolly good news is that you don’t need to know anything about Docker to use this setup. At least, I hope you won’t. That’s the theory.
As mentioned above, I am making use of the new Docker / Symfony 3 starting point repo. I’ve tweaked this a touch for our Redis requirements.
By the end of this video you will have got a Symfony website up and running, and cached data into Redis. We will cover the setup required, both from a Docker perspective, and the config needed inside Symfony.
In the previous video we put all our caching logic into a controller action.
That’s really not going to cut it beyond your first steps. In reality you’re going to want to move this stuff out into a separate service.
To give you an example of how I might do it:
Controller calls CacheApiWrapper which calls Api if needed.
That might not be making much sense, so let’s break it down further.
Let’s imagine an app without caching.
We need to call a remote API to get the days prices of Widgets.
We hit a controller action which delegates the API call off to our Api service, which really manages the API call for us. When the API call completes (or fails), it returns some response to the calling controller action, which in turn returns the response to the end user.
If that API call always returns the same response every time it is called – maybe the prices of Widgets only change every 24 hours – then calling the remote API for every request is both pointless, and wasteful.
We could, if we wanted to, add the Caching functionality directly into the Api service.
I don’t like this approach.
Instead, I prefer to split the responsibility of Caching out into a separate ‘layer’. This layer / caching service has an identical API to the Api service. This caching service takes the Api service as a constructor argument, and wraps the calls, adding caching along the way.
If all this sounds complex, after seeing some code, hopefully it will make a bit more sense:
class WidgetController extends Controller
{
/**
* @Route("/widgets", name="widget_prices")
*/
public function widgetPriceAction(
WidgetPriceProvider $widgetPriceProvider
)
{
return $this->render(
':widget:widget_price.html.twig',
[
'widget_prices' => $widgetPriceProvider->fetchWidgetPrices(),
]
);
}
This should hopefully look familiar.
The unusual part is the WidgetPriceProvider. Well, they do say the two hardest parts of computer science are cache invalidation, and naming things…
Really WidgetPriceProvider is the best name I could think of to explain what’s happening. This service provides Widget Prices.
How it provides them is where the interesting part happens:
<?php
namespace AppBundle\Provider;
use AppBundle\Connectivity\Api\WidgetPrice;
use AppBundle\Widget\WidgetPriceInterface;
use Psr\Cache\CacheItemPoolInterface;
use Symfony\Component\Cache\Adapter\AdapterInterface;
class WidgetPriceProvider implements WidgetPriceInterface
{
/**
* @var CacheItemPoolInterface
*/
private $cache;
/**
* @var WidgetPrice
*/
private $widgetPriceApi;
public function __construct(
AdapterInterface $cache,
WidgetPrice $widgetPriceApi
)
{
$this->cache = $cache;
$this->widgetPriceApi = $widgetPriceApi;
}
public function fetchWidgetPrices()
{
$cacheKey = md5('fetch_widget_prices');
$cachedWidgetPrices = $this->cache->getItem($cacheKey);
if (false === $cachedWidgetPrices->isHit()) {
$widgetPrices = $this->widgetPriceApi->fetchWidgetPrices();
$cachedWidgetPrices->set($widgetPrices);
$this->cache->save($cachedWidgetPrices);
} else {
$widgetPrices = $cachedWidgetPrices->get();
}
return $widgetPrices;
}
}
As mentioned, WidgetPriceProvider is a wrapper / layer over the Api Service.
It also has the cache injected.
This way the API call is separated from the cache, and can be used directly if needed. Sometimes I don’t want to go via the cache. Maybe in the admin backend, for example.
Note that this service implements WidgetPriceInterface. This isn’t strictly necessary. This interface defines a public function of fetchWidgetPrices.
The reason for this is, as mentioned earlier, I want the caching layer (WidgetPriceProvider) to use the same method name(s) as the underlying API service.
Speaking of which:
<?php
namespace AppBundle\Connectivity\Api;
use AppBundle\Widget\WidgetPriceInterface;
use GuzzleHttp\Client;
use Psr\Log\LoggerInterface;
class WidgetPrice implements WidgetPriceInterface
{
/**
* @var LoggerInterface
*/
private $logger;
/**
* @var Client
*/
private $client;
/**
* WidgetPrice constructor.
*
* @param LoggerInterface $logger
* @param Client $client
*/
public function __construct(
LoggerInterface $logger,
Client $client
)
{
$this->client = $client;
$this->logger = $logger;
}
public function fetchWidgetPrices()
{
// better to inject via constructor, but easier to show like this
$url = "https://api.widgetprovider.com/prices.json";
try {
return json_decode(
$this->client->get($url)->getBody()->getContents(),
true
);
} catch (\Exception $e) {
$this->logger->alert('It all went Pete Tong', [
'url' => $url,
'exception' => [
'message' => $e->getMessage(),
'code' => $e->getCode()
],
]);
return [];
}
}
}
So really it’s about delegating responsibility down to a more appropriate layer.
Be Careful – don’t blindly copy this approach.
There is a gotcha in this approach which maybe not be immediately obvious.
If your API call fails, the bad result (and empty array in this case) will still be cached.
Adapt accordingly to your own needs. It may be better to move the try / catch logic out from the WidgetPrice class and into the WidgetPriceProvider instead.
Or something else. It’s your call.
Anyway, that’s an excursion into a potentially more real world setup.
Back when I was a server room techy there was a line my old boss used to say whenever I got wind of something new and shiny:
There are no new ideas in IT, just the same ones on repeat.
Of course he was being tongue-in-cheek, but there was a truth in what he said.
For example, around the time I was in this role, the IT industry was moving from distributed to centralised. That is to say we had a bunch of servers dotted all over the county, and the plan was to bring them into just two central server rooms.
This, I was assured, was very much like “the olden days” when individual desktops were replaced with “dumb terminals”, which relied heavily on a centralised Mainframe.
At some point, someone (likely an army of well paid consultants) espoused the failings of such an architecture (oh my, single point of failure!) and likely sold them a bunch of high powered standalone desktops.
This worked well for a while, then companies like Citrix came along and sold a different spin on “dumb terminals” using your existing high powered desktop, and so on, and so on.
What the heck does any of this have to do with Web Development, I hear you ask.
Good question.
As you may recall, I have been getting quite excited about launching the new, shiny revision of CodeReviewVideos.com. I had my zip lock baggy of party poppers at the ready. Things were looking super.
Then, over the previous weekend, it dawned on me:
How’s the SEO on this new site then?
Seeing as about 70% of the incoming visitors to CodeReviewVideos arrive via Google, I figured I should probably – ya’ know – give this some consideration.
I checked, and it turned out that I had, ahem, neglected to set any of the head information.
Yes, I felt quite the chimp.
But not to worry, I have all the SEO data just sat ready and waiting in the DB. After all, it’s exactly the same as for the existing site.
What happens with the existing site is what happens with pretty much any PHP site I have ever worked with:
A request comes in
The relevant data is fetched from the DB (thanks, Doctrine)
This data populates a template (thanks, Twig)
The response is returned to the end user (thanks, Symfony!)
It doesn’t matter if the request is from Google Bot, or from a real person. The process is always the same.
What this means is that the response contains everything needed to make a full page representation.
Google Bot can look at the page and see all the expected “stuff”: the header tags, and body content, it’s all there.
Sure, we can then augment this with a snazzy bit of JS here and there, but largely, it’s good to go.
Web 2 Point… Oh?!
Then sometime a few years back, Single Page Applications (SPA, and not the relaxing kind) became popular.
SPA – not as relaxing as you may have been led to believe
Fast forward a few years and boom, I’ve done a few of these here SPA’s with React and Angular, and I’m thinking: yeah, this is awesome, let’s make CodeReviewVideos all snazzy using all this wicked tech.
So I did.
The new version of the site uses React, and Redux, and Redux Saga, and it talks to a Symfony JSON API, and it’s all lovingly tested and fills me with warm fuzzy feels whenever I work with either code base.
Unfortunately, all this awesomeness does make the architecture more complex. Let’s revisit how a request / response works now:
If you’re a bot Yahoo, Bing, or Yandex, you’re now done.
Forgetaboutit
By all accounts these bots do not process JavaScript, so forgettaboutit.
This may very well be of no major consequence.
By way of some hard figures, the combined total visitors that CodeReviewVideos received over the last 7 days that came via a search engine that isn’t Google was:
11.
Fixing the world for the sake of these other search engines probably isn’t worth my time right now.
My guess is they have. I mean, this is Google. They are super smart, and it’s their job to get this right.
But can I risk killing the site on a maybe.
No, definitely not.
What Happens Next?
Back to the HTML from above. Thinking about the request / response sequence, what happens next?
Well, on the client side – which is to say in your visitor’s browser – assuming they have JavaScript enabled, the JS is read and run (or parsed and executed, if you’re a CS text book).
Cus i’m Shiny!
This is where all the cool stuff happens!
React builds the page from my components
The components instruct the browser to make requests to the back-end API for the real content
The real content is received
React renders the content into the right place(s)
The visitor is completely indifferent
By the way, if the visitor doesn’t have JavaScript enabled then they are going to have a bad time. But, ya know, 2017, etc, etc.
When I realised my mistake about SEO last weekend, my first thought was to simply grab my Helmet and populate the head tags using the API response.
Alas, as we have now seen, this simply will not do.
So I figured I would be a Smart Guy ™ and put a caching layer in front of my web server. Maybe I could trick the bots by returning a fully cached version of each page.
Ultimately though, it meant I had to postpone the scheduled launch. And that sucks.
Server Side Rendering anybody? No? SSR? No?
Of course I’m not the first person to have experienced this problem.
There is already a solution:
Server Side Rendering.
The idea here is rather convoluted, but stick with me:
We already have our shiny new React site, right? Yes, yes we do.
Server-side-rendiamous!
So, let’s get a Node JS instance to sit in front of our React site, and uses some of the stuff we learned in the Dark Arts class at Hogwarts to run this code, then convert the response to a string, and write the string to a template, which is turned into another string which is sent back to the browser so that we now have the necessary HTML to please Google like in the olden days.
Oh my.
It seems like we’re repeating ourselves here.
But now with more layers. More bug filled, confusing, maybe unnecessary layers.
Anyway, that’s where I’m at with CodeReviewVideos right now. Trying to migrate my existing front end code from client-side only, to this SSR setup to essentially reproduce the outcome I already have with plain old PHP.
What this means is right now I have no ETA on the next version, and once it is again “ready”, I will need to go back through a phase of testing before go-live.
Between this, and the next video we look at a question I got asked late last week:
How can I add a custom CSS class to the `input` / other form field when validation fails for that field?
This video explores one possible solution to this problem.
What happens however, is that we end up with a bunch of repeated logic living in a Twig template.
Generally, if you find yourself repeating yourself… or you end up with complex logic living inside a Twig template, you can probably extract it into a better location – a custom Twig function.
As such we look at extracting this repeated logic from our Twig template into a Twig Extension.
It’s one of those topics that sounds like it might be hard, or too nerdy, or whatever.
It’s just a strange name for a very useful concept.
If you use Twig in any seriousness, and you don’t yet know about Twig Extensions then this video should help you understand how to use one, and why it’s not as hard as you might think to do so.
In my personal experience there has been no better way to improve my overall knowledge of software development than by learning another language.
Elixir, in this case, is the language.
Phoenix is somewhat akin to Symfony, or Rails, or what have you. It’s a framework for web applications.
In this series we are building up a small but functional (ho, ho) JSON API using Elixir and Phoenix.
In this video we cover how to set up the Web server by way of docker-compose.yml. This series is a little more advanced than most of the content on CodeReviewVideos, but I hope you’re finding it enjoyable all the same.
That’s It, That’s All
That’s just about it from me this week.
My favourite snowboarding movie 🙂 Great soundtrack.
As mentioned last week I spent the previous weekend at the PHP North West conference.
The raw JavaScript approach aims to show how things were in the ‘bad old days‘. The days before your package manager would take up ~7gb of your hard disk with its 
Behind the scenes over the past 10 weeks I have been working on integrating CodeReviewVideos with Braintree.
Yes, that sounds awesome.
I checked, and it turned out that I had, ahem, neglected to set any of the 
