Tag: symfony

How I Fixed: Server channel error: 406, message: PRECONDITION_FAILED – inequivalent arg ‘type’ for exchange ‘my_exchange’ in vhost ‘/’: received ‘fanout’ but current is ‘direct’

Not a fun way to start a Saturday morning. With a bit of spare time this morning I wanted to continue some refactoring work on a tool I’ve been working on for checking broken links on any given website.

The project is quite cool (in my opinion), using a bunch of interesting software / tech such as RabbitMQ with Symfony’s Messenger component, STOMP for real time stuff, React with Hooks, Tailwinds for CSS… and a bunch more buzz-wordy, CV helping stuff that keeps me gainfully employed.

Anyway, the first thing I did was spin up the Symfony docker containers that run the various services to handle incoming broken link checking requests. And as ever, I ran a composer update to bring Symfony up to 4.3.x.

I’m not sure if bumping up to Symfony 4.3 was the cause of this problem. I suspect not. It’s been a while since I’ve worked on this part of the code, but it was all working the last time I brought the project up. And it’s working live and online, too, so something has gone awry.

Anyway, after the composer update completed successfully:

composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)

Prefetching 49 packages 🎶 💨
  - Downloading (100%)

Package operations: 7 installs, 42 updates, 1 removal
  - Removing symfony/contracts (v1.0.2)
  - Updating symfony/flex (v1.2.3 => v1.2.5): Loading from cache
  - Installing symfony/service-contracts (v1.1.2): Loading from cache
  - Installing symfony/polyfill-php73 (v1.11.0): Loading from cache
  - Updating symfony/console (v4.2.8 => v4.3.0): Loading from cache
  - Installing symfony/event-dispatcher-contracts (v1.1.1): Loading from cache
  - Updating symfony/event-dispatcher (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/css-selector (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/dom-crawler (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/messenger (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/process (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/serializer (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/routing (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/finder (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/filesystem (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/debug (v4.2.8 => v4.3.0): Loading from cache
  - Installing symfony/polyfill-intl-idn (v1.11.0): Loading from cache
  - Installing symfony/mime (v4.3.0): Loading from cache
  - Updating symfony/http-foundation (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/http-kernel (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/dependency-injection (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/config (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/var-exporter (v4.2.8 => v4.3.0): Loading from cache
  - Installing symfony/cache-contracts (v1.1.1): Loading from cache
  - Updating symfony/cache (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/framework-bundle (v4.2.8 => v4.3.0): Loading from cache
  - Installing symfony/translation-contracts (v1.1.2): Loading from cache
  - Updating symfony/validator (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/yaml (v4.2.8 => v4.3.0): Loading from cache
  - Updating nikic/php-parser (v4.2.1 => v4.2.2): Loading from cache
  - Updating symfony/translation (v4.2.8 => v4.3.0): Loading from cache
  - Updating nesbot/carbon (2.17.1 => 2.19.0): Loading from cache
  - Updating illuminate/contracts (v5.8.15 => v5.8.19): Loading from cache
  - Updating illuminate/support (v5.8.15 => v5.8.19): Loading from cache
  - Updating symfony/inflector (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/property-access (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/property-info (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/monolog-bridge (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/dotenv (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/phpunit-bridge (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/expression-language (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/stopwatch (v4.2.8 => v4.3.0): Loading from cache
  - Updating composer/xdebug-handler (1.3.2 => 1.3.3): Loading from cache
  - Updating symfony/var-dumper (v4.2.8 => v4.3.0): Loading from cache
  - Updating twig/twig (v2.9.0 => v2.11.0): Loading from cache
  - Updating symfony/twig-bridge (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/debug-bundle (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/twig-bundle (v4.2.8 => v4.3.0): Loading from cache
  - Updating symfony/web-profiler-bundle (v4.2.8 => v4.3.0): Loading from cache
  - Updating roave/security-advisories (dev-master 1dfa887 => dev-master 4c0ba8a)
Writing lock file
Generating autoload files
ocramius/package-versions:  Generating version class...
ocramius/package-versions: ...done generating version class

What about running composer global require symfony/thanks && composer thanks now?
This will spread some 💖  by sending a ★  to the GitHub repositories of your fellow package maintainers.

Executing script cache:clear [OK]
Executing script assets:install public [OK]

I tried to run my messenger consumer:

www-data@1fbf5db0f719:~/app.checkforbrokenlinks.com$ bin/console messenger:consume --bus messenger.bus.fetch fetch -vvv

                                                                                                                        
 [OK] Consuming messages from transports "fetch".                                                                       
                                                                                                                        

 // The worker will automatically exit once it has received a stop signal via the messenger:stop-workers command.       

 // Quit the worker with CONTROL-C.                                                                                     


In AmqpReceiver.php line 56:
                                                                                                                                                             
  [Symfony\Component\Messenger\Exception\TransportException]                                                                                                 
  Server channel error: 406, message: PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'fetch' in vhost '/': received 'fanout' but current is 'di  
  rect'                                                                                                                                                      
                                                                                                                                                             

Exception trace:
 () at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Transport/AmqpExt/AmqpReceiver.php:56
 Symfony\Component\Messenger\Transport\AmqpExt\AmqpReceiver->getEnvelope() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Transport/AmqpExt/AmqpReceiver.php:47
 Symfony\Component\Messenger\Transport\AmqpExt\AmqpReceiver->get() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Worker.php:92
 Symfony\Component\Messenger\Worker->run() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Worker/StopWhenRestartSignalIsReceived.php:54
 Symfony\Component\Messenger\Worker\StopWhenRestartSignalIsReceived->run() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Command/ConsumeMessagesCommand.php:224
 Symfony\Component\Messenger\Command\ConsumeMessagesCommand->execute() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/console/Application.php:939
 Symfony\Component\Console\Application->doRunCommand() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/framework-bundle/Console/Application.php:87
 Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/console/Application.php:273
 Symfony\Component\Console\Application->doRun() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/framework-bundle/Console/Application.php:73
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/console/Application.php:149
 Symfony\Component\Console\Application->run() at /var/www/app.checkforbrokenlinks.com/bin/console:39

In Connection.php line 348:
                                                                                                                                                             
  [AMQPExchangeException (406)]                                                                                                                              
  Server channel error: 406, message: PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'fetch' in vhost '/': received 'fanout' but current is 'di  
  rect'                                                                                                                                                      
                                                                                                                                                             

Exception trace:
 () at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Transport/AmqpExt/Connection.php:348
 AMQPExchange->declareExchange() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Transport/AmqpExt/Connection.php:348
 Symfony\Component\Messenger\Transport\AmqpExt\Connection->setup() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Transport/AmqpExt/Connection.php:311
 Symfony\Component\Messenger\Transport\AmqpExt\Connection->get() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Transport/AmqpExt/AmqpReceiver.php:54
 Symfony\Component\Messenger\Transport\AmqpExt\AmqpReceiver->getEnvelope() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Transport/AmqpExt/AmqpReceiver.php:47
 Symfony\Component\Messenger\Transport\AmqpExt\AmqpReceiver->get() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Worker.php:92
 Symfony\Component\Messenger\Worker->run() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Worker/StopWhenRestartSignalIsReceived.php:54
 Symfony\Component\Messenger\Worker\StopWhenRestartSignalIsReceived->run() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/messenger/Command/ConsumeMessagesCommand.php:224
 Symfony\Component\Messenger\Command\ConsumeMessagesCommand->execute() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/console/Application.php:939
 Symfony\Component\Console\Application->doRunCommand() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/framework-bundle/Console/Application.php:87
 Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/console/Application.php:273
 Symfony\Component\Console\Application->doRun() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/framework-bundle/Console/Application.php:73
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /var/www/app.checkforbrokenlinks.com/vendor/symfony/console/Application.php:149
 Symfony\Component\Console\Application->run() at /var/www/app.checkforbrokenlinks.com/bin/console:39

messenger:consume [-l|--limit LIMIT] [-m|--memory-limit MEMORY-LIMIT] [-t|--time-limit TIME-LIMIT] [--sleep SLEEP] [-b|--bus BUS] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-e|--env ENV] [--no-debug] [--] <command> [<receivers>...]

Knickers. It all blew up quite badly.

There’s a lot of info to process, and without some nice terminal colouring it’s all a bit of a blur.

The interesting line is:

Server channel error: 406, message: PRECONDITION_FAILED - inequivalent arg 'type' for exchange 'my_exchange' in vhost '/': received 'fanout' but current is 'direct'

What I think has gone wrong is that at some point in the past, I’ve switched over my RabbitMQ exchange to use direct, and by default, Symfony’s Messenger component will try to create an exchange with the type of fanout.

To clarify, my exchange and queue combo already exists at: amqp://{username}:{password}@rabbitmq:5672/%2f/fetch

It exists because I have previously configured my RabbitMQ instance to boot up with this exchange / queue combo ready and good to go.

Because Symfony’s Messenger component is not immediately aware that this queue will already exist, it tries to create it.

It cannot create it because the default type of exchange that Symfony’s Messenger component will try to use is fanout.

In order to make this work, I needed to manually specify the config that explicitly sets this exchange / queue combo to the desired setting of direct.

Finding this out via the documentation wasn’t super straightforward. Here’s a few of the steps I took:

bin/console config:dump-reference framework

This shows that for each framework.messenger.transports entry in your config/packages/messenger.yaml file, you can have a variety of additional settings.

As it was, my original config looked like this:

By providing just a DSN (by way of environment variables), all the default config would be used.

What I needed to do was swap over to this:

framework:
    messenger:
        transports:
             fetch:
                dsn: '%env(MESSENGER_TRANSPORT_DSN_FETCH)%'
                options:
                    exchange:
                        type: 'direct'
             scrape:
                dsn: '%env(MESSENGER_TRANSPORT_DSN_SCRAPE)%'
                options:
                    exchange:
                        type: 'direct'

And after doing so, it all started working again:

In short, this isn’t directly a Symfony / Symfony Messenger problem. It’s a config problem. The messaging could be a little more clear, as could the documentation for what things are viable as options.

How I Fixed: argument “$entityManager” of method “__construct()” references class “Doctrine\ORM\EntityManager” but no such service exists

Ok, mega crazy title. And honestly, this is just the tip of the iceberg. Allow me to set the scene:

Lately I have had email conversations, read threads on hackernews, and even had a forum post challenging how and why I do things the way I do.

The summary of the email conversations being why I persist with Symfony / PHP generally, when other, “better” solutions exist. And the same can be said for the linked forum post.

And then yesterday I saw that linked Hacker News thread:

It was at about ~320 comments when I read it. The top reply was the most interesting for me:

There’s a bit more to it than that, and the thread itself is worth a read. There’s basically 400+ different suggested ways to “get a web app up quickly in 2018”. I’d disagree with a bunch of them, but then, they are the way I do things.

Wait, what?

Yeah, I’d disagree with Docker + Ansible + Terraform + nginx + (Symfony/Rails/Go/etc) + Postgres, etc, being quick to get up and running.

Sure, once you know the drill / have projects to copy / paste from, it can be quick, relative to the first time you had to learn and implement all this stuff. But it’s not quick quick. It still takes me ages.

And so I challenged myself: Just how quickly could I get a typical project up and running for myself? The perfect question for a Saturday night.

My Setup

The setup I most typically use is:

  • Terraform for spinning up a server
  • Ansible for prep’ing the box
  • Docker for running stuff
  • GitLab for code hosting + CI
  • nginx for my web server
  • Symfony / PHP 7 for the code
  • Postgres for the DB

This is a lot of stuff, and it’s not super quick to set up.

This is why I started by mentioning the email / forum conversations whereby people ask: is Symfony / PHP the best tool of choice?

Well, maybe not. I don’t know. I just know I’m more productive with Symfony and PHP generally than everything else – though JavaScript is a close second.

Over the past few years I’ve tried other setups. It’s hard to invest time in learning another stack when the end result may be basically identical – what did I gain from the time invested? Could that time have been better invested elsewhere? Hard questions to answer.

But yeah, Node and more recently, Golang have been stronger contenders than usual for my attention. Anyway, that’s a bit of a digression.

The Problem

As mentioned above, that’s my stack. Learning it all took ages (years?), but as each project is, from an infrastructure point of view, very similar, I can now spin up a new environment very quickly.

My challenge was to find out how quickly. I got most of the core stuff up and running in ~1.5 hours.

I didn’t get the Behat testing environment set up in that time. Because I hit on an issue.

I wanted a simple JSON API as the outcome of this process. By simple I mean basically CRUD.

With the basic stack up and running, I created a basic entity (one property), and updated the DB accordingly. Doctrine was used for DB interactivity. Again, very typical for my projects.

In order to get data out of my repo, I needed to create a repository. There’s an awesome post on this by Tomas Votruba called How to use Repository with Doctrine as Service in Symfony.

As a side note here: if you haven’t already, I would highly recommend reading Tomas’ blog, as it’s jam packed with things you’d likely find very useful and interesting. Also, check out his GitHub projects, with Rector in particular being incredible.

I followed the linked article, and hit upon the following:

Cannot resolve argument $temporaryEmailRepository of "App\Controller\TemporaryEmailController::cget()": Cannot autowire service "App\Repository\TemporaryEmailRepository": argument "$entityManager" of method "__construct()" references class "Doctrine\ORM\EntityManager" but no such service exists. Try changing the type-hint to one of its parents: interface "Doctrine\ORM\EntityManagerInterface", or interface "Doctrine\Common\Persistence\ObjectManager".

What was weird to me at this point is that I’ve followed this article before, but never hit upon any problems.

Anyway, I did as I was told – I switched up the code to reference the EntityManagerInterface instead:

<?php

declare(strict_types=1);

namespace App\Repository;

use App\Entity\TemporaryEmail;
use Doctrine\ORM\EntityManagerInterface;
use Doctrine\ORM\EntityRepository;

final class TemporaryEmailRepository
{
    /**
     * @var EntityRepository
     */
    private $repository;

    /**
     * TemporaryEmailRepository constructor.
     *
     * @param EntityManagerInterface $entityManager
     */
    public function __construct(EntityManagerInterface $entityManager)
    {
        $this->repository = $entityManager->getRepository(TemporaryEmail::class);
    }

    /**
     * @return array
     */
    public function findAll(): array
    {
        return $this->repository->findAll();
    }
}

This is a really simple class.

For complete clarity, here’s basically the rest of the app at this point:

<?php

namespace App\Controller;

use App\Repository\TemporaryEmailRepository;
use FOS\RestBundle\Controller\Annotations;
use FOS\RestBundle\Controller\FOSRestController;

class TemporaryEmailController extends FOSRestController
{
    /**
     * @Annotations\Get("/")
     *
     * @param TemporaryEmailRepository $temporaryEmailRepository
     *
     * @return \FOS\RestBundle\View\View
     */
    public function cget(TemporaryEmailRepository $temporaryEmailRepository)
    {
        return $this->view([
            'data' => $temporaryEmailRepository->findAll(),
        ]);
    }
}

And the entity:

<?php

namespace App\Entity;

use Doctrine\ORM\Mapping as ORM;
use Symfony\Component\Validator\Constraints as Assert;

/**
 * @ORM\Entity(repositoryClass="App\Repository\TemporaryEmailRepository")
 * @ORM\Table(name="temporary_email")
 */
class TemporaryEmail implements \JsonSerializable
{
    /**
     * @ORM\Column(type="guid")
     * @ORM\Id
     * @ORM\GeneratedValue(strategy="UUID")
     */
    private $id;

    /**
     * @ORM\Column(type="string", name="domain", unique=true, nullable=false)
     * @Assert\Url()
     * @var string
     */
    private $domain;

    /**
     * @return mixed
     */
    public function getId()
    {
        return $this->id;
    }

    /**
     * @return string
     */
    public function getDomain(): string
    {
        return $this->domain;
    }

    /**
     * @param string $domain
     *
     * @return TemporaryEmail
     */
    public function setDomain($domain): self
    {
        $this->domain = $domain;

        return $this;
    }

    /**
     * @return array
     */
    public function jsonSerialize(): array
    {
        return [
            'id'     => $this->id,
            'domain' => $this->domain,
        ];
    }
}

This is basically a generated entity with a couple of tweaks. It’s not the final form, so don’t take this as good practice, or whatever.

The purpose of what this class is supposed to do is also not relevant here, but will be discussed in a future video.

Anyway, the problem is evident in the code above. If you can spot it, then good stuff 🙂

If not, keep reading.

So with three records in the DB, all the connectivity setup, things looking decent, I sent in a request to my only endpoint – GET /.

And it didn’t work. I hit a 504 Gateway Timeout  error from nginx.

2018/06/03 20:02:19 [error] 7#7: *25 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.18.0.1, server: temporary-email.dev, request: "GET / HTTP/1.1", upstream: "fastcgi://172.18.0.3:9000", host: "0.0.0.0:807

Very confusing, overall. I mean, this is basically copy / paste from a different project that works just fine. Only, I’ve renamed the project name. What the heck?

I hit refresh a few times, you know, to make sure the computer wasn’t lying to me. And then everything started going unresponsive. Very odd. I’ve just bumped the system from 16gb to 32gb, and all I have is a few Docker containers running, a browser with admittedly too many open tabs, and one instance of PHPStorm. Surely this couldn’t be taxing the system. htop  told me a different story:

Yeah, I know, that swap size is ridiculous. Forgive me.

The nginx logs weren’t really that helpful. I needed to look at the PHP log output, which in this case is achieved via docker logs :

docker logs php_api_te

[03-Jun-2018 20:06:12] WARNING: [pool www] child 6 said into stderr: "NOTICE: PHP message: PHP Fatal error:  Maximum execution time of 60 seconds exceeded in /var/www/api.temporary-email.dev/src/Repository/TemporaryEmailRepository.php on line 23"

Line 23 of TemporaryEmailRepository  is:

public function __construct(EntityManagerInterface $entityManager)

I mucked around a bit, trying out injecting the ObjectManager instead, but hit the same issue.

Then I wondered if it was the act of injecting itself, or actually using the injected code (durr). So I commented out the call:

    /**
     * TemporaryEmailRepository constructor.
     *
     * @param EntityManagerInterface $entityManager
     */
    public function __construct(EntityManagerInterface $entityManager)
    {
        // $this->repository = $entityManager->getRepository(TemporaryEmail::class);
    }

Reloading now, I was no longer seeing the massive RAM spike, and looking at what that call was doing pushed me down the right lines.

I’ll admit, it took me a much longer amount of time than I’d of liked to realise my mistake:

/**
- * @ORM\Entity(repositoryClass="App\Repository\TemporaryEmailRepository")
+ * @ORM\Entity()
 * @ORM\Table(name="temporary_email")
 */
class TemporaryEmail implements \JsonSerializable

Now, I’m not 100% certain on the conclusion here, but this is my best guess.

I believe I had created a circular reference. I’d injected the Entity Manager into the repo. Immediately I’d asked for the entity. The entity has an annotation pointing at the repo, which triggered the endless loop.

Anyway, removing the repositoryClass attribute fixed it up. Kinda obvious in hindsight.

The Conclusion

I’m convinced I could get an environment up faster than this. Without hitting this issue I believe I would be at the ~2 hour mark to go from idea to having a solid setup that’s good to write code in a sane, reproducible, reliable / testable way.

I think back to 10+ years ago, where I’d be up and running so much faster. PHP is essentially a scripting language. With shared hosting, you’d have the DB ready, the web server ready, you just needed to write a bit of code, connect to the DB, push the code up somehow (FTP :)) and bonza, you’re up and running.

Looking at that way now, I’m amazed how far I’ve come. There’s a massive overhead with using frameworks – time spent learning (which never stops, unless your framework of choice goes EOL), patching, managing all this stuff, learning new ways to make things better… is it all worth it? I think so.

I think the biggest takeaway for me lately is that whilst within the last ~5 years I’ve shipped a lot less code to prod than in the 5 years preceding this, the code I do ship is more stable, and maintainable.

Nagging in my mind, however, is that what’s the point in this slow, methodical approach if the end result is it takes so long, I either don’t bother with entire ideas, or by the time I’ve shipped them, I’m so burned out by the seeming complexity of the whole thing that I lose interest in taking them further.

Anyway, I appreciate this is half helpful, half rant. I just needed to blog it and get these thoughts out of my head.

PayPal: Done; Discourse: Done; What’s Next?

I hope this post finds you well. It’s so lovely and sunny here in the UK, it’s (almost) a shame to be inside coding. I guess that’s what laptops are for, right? So you can still code whilst sat in the garden.

The last five weeks have been incredibly busy for me. Aside from starting a new role, I’ve managed to finally finish off a couple of big tasks that have been on my TODO list for waaaay too long. These are:

  • adding PayPal to the site
  • migrating away from Disqus commenting

I want to quickly cover both.

Adding PayPal

If you’ve been getting my mailings / reading the blog for any length of time, you’ll likely be sick of hearing about PayPal. When switching from the old Symfony-only implementation of CodeReviewVideos (version 1), I knew I’d want to offer more than just Stripe as a payment option. Therefore, I dutifully planned ahead and made the process of accepting payments as modular as possible, and made Stripe payment processing just one possible implementation.

This all worked absolutely fine. I was plenty comfortable with Stripe already, and had my original implementation to use as a reference.

What I did wrong, in hindsight, was base my implementation too heavily on Stripe.

To be clear, Stripe get a lot of things right. If you have to accept payments, working with Stripe’s API is a joy. It’s hard not of be influenced by how their system works.

As such, some of the ways I implemented things like the card details endpoint, the invoicing, and even little things like what data was being captured if using a `debug` level of logging were too heavily tied to Stripe.

The new subscribe form with PayPal option chosen

These things combined to make adding Braintree integration (aka PayPal’s API) take a lot longer than planned (~6 months, to my estimate of about 2-4 weeks). There were some other complications, such as getting my account approved was perhaps something I should have done upfront, but instead, I left this until I was about 8 weeks into development. In hindsight, if they had declined my application, I’d have wasted a lot of time. Not to mention, when I finally thought I might get rejected (it took a while, I got fearful) I stopped development entirely – for about 2 weeks.

The biggest mistake I made though was in the DB schema. Even though I knew upfront that I’d ultimately want to allow people to subscribe with PayPal, or Stripe, I made the relationship between a User account, and a Payment Information a one-to-one.

This was deployed to prod.

All worked fine when all I had was Stripe.

The problem dawned on me that if a User was paying with Stripe, then canceled their subscription, then rejoined with PayPal, then canceled again, and rejoined with Stripe, there was no way to get their previous payment info back. It sounds like an edge case, but if I’ve learned anything from CodeReviewVideos, it’s that all manner of unexpected circumstances can, and do arise. And more frequently than I’d ever have thought.

There was another issue. If a User was paying with Stripe, and then switched to PayPal, with a one-to-one setup they would lose their Stripe invoices. Again, major headaches.

So even when I’d finished the development, I still had a major migration ahead of me. And that consumed about 4 weeks in terms of planning, writing migration scripts, testing, setting up a whole new environment to test the thing end to end… phew, the work just kept on, and on.

Anyway, to cut a long story short, it’s done. The migration went well, and PayPal is now in prod. I think I celebrated by immediately cracking on with the Disqus migration. Ha.

Migrating To Discourse

One aspect of the CodeReviewVideos web site that I’ve never been happy with has been the use of Disqus.

There was a nasty user experience whereby you’d have to sign up once to use the site, and again – and entirely separately – to leave a comment. It sucked. But as far as pragmatic solutions go, it was good enough to get going.

I also read that Disqus would be enabling adverts on my comments section – though to the very best of my knowledge, that never happened. There was talk of a monthly fee. I don’t know. I don’t begrudge them charging for their service, but that wasn’t for me.

Adding Disqus wasn’t super easy, but at the same time, it wasn’t quite as hardcore as the PayPal change.

Surgical

The complications came by way of Single Sign On, hosting Discourse (via Docker), and replacing the existing comments.

Single Sign On wasn’t as bad as I’d expected. I thought that would be the hardest part. I found a Laravel package which I butchered crafted with surgical precision into something that worked nicely with Symfony.

Hosting Discourse wasn’t too bad. I use Rancher for my Docker container management, but Discourse’s Docker implementation just wasn’t playing ball. In the end I got a new VPS from Hetzner and hosted it there instead. There were some tweaks needed, but overall it wasn’t so bad.

Replacing the existing comments was the real tricky part. Disqus provide a one-way export – something I think is a bit weird. By which I mean you can export your data from Disqus, but they won’t let you revert to a previous ‘backup’. Anyway, I didn’t need that, I just needed the export, so that was fine and dandy.

Once I had the export I needed to get that data in to Docker, and then tweak the provided Disqus import script to run against my export. That all generally worked, but it only seems to have missed some of the comments off. I’m not sure why, but also I felt the end result was “good enough”.

The import worked by looking for any existing user, and then mapping a Disqus email address to the user’s Discourse email address. If the Disqus commentor never had a site membership, then now their comment will be assigned to some anonymous username like Disqus_312jklsdf2kl  or whatever. Not perfect, but again, good enough.

Now what happens is when I create a new video, the comments section automagically creates a new forum post under the username `chris`. As such, if you look at the forum today (and you should, because it’s ace), you’ll see I’ve been posting new topics like a mad man. This will slow down over the next few days.

As I write this I still have email functionality globally disabled on the forum. This will change, possibly over the weekend, once I’m suitably confident everything has settled down. You may recall receiving an email from the staging forum a few weeks back – yep, I made a boob there. Sorry about that. Once bitten, twice shy.

What’s Upcoming

I mentioned at the start that aside from the PayPal and Discourse changes, I have also recently started a new role.

Sometimes I get emails asking why I don’t have any new videos in a while, or why I haven’t updated X, Y, or Z to the latest and greatest. Believe me, I’d love to spend all day making new videos. Unfortunately CodeReviewVideos is not my full time job.

As some of you may know – I’m fairly open about it – I am a contractor by day.

One of the really nice things about being a contractor is getting to experience lots of different projects, in various scales of complexity, and reliability 🙂

I get a lot of interesting ideas for videos from my day-to-day work experiences. And this means that the content on CodeReviewVideos is very much about actionable, real world stuff you can use right now, today in your projects. It’s also battle tested / used in real world production websites.

There’s a bunch of videos I wrote up but haven’t had time to record as of yet. The reason I mention the whole day job thing is that it means I have only a fixed amount of time per week to devote to the site, and I have to prioritise tasks accordingly. Above all else, I prefer making videos. This is why I stared, and continue to run CodeReviewVideos. I love sharing, and from the feedback I get from so many of you (thank you!) you find it useful, too.

But of course, over the last few weeks I’ve had these other big site changes (dare I say, improvements) to make. And that has meant I haven’t been recording. Thankfully all that can change, and I can get back to making recording new stuff.

Ok cool – so what should be being recorded, all being well, is the continuation of the Beginners Guide to Back End (JSON API) + Front End Development [2018] series. Next up is the API Platform section, which I both really enjoyed building, and writing up, and am equally looking forwards to recording and sharing. It’s a good one.

Let’s hope this sunshine continues through the weekend, and for those of you in the UK, enjoy your bank holiday / extended weekend.

One final thing before I go: please do come and say hi on the forum.

Until next time, happy coding!

Chris

The 2018 Beginners Guide to Back End (JSON API) + Front End Development

It’s been a few weeks in the making, but I am happy now to reveal my latest course here on CodeReviewVideos:

The 2018 Beginners Guide to Back End (JSON API) + Front End Development.

This course will cover building a JSON-based API with the following back-end stacks:

  1. ‘raw’ Symfony 4 (PHP)
  2. Symfony 4 with FOSRESTBundle (PHP)
  3. API Platform (PHP)
  4. Koa JS (JavaScript / node)

Behat will be used to test all of these APIs. One Behat project, four different API implementations – in two different languages (PHP and JS).

We’re going to be covering the happy paths of GET , POST , PUT , (optionally) PATCH , and DELETE.

We’ll also be covering the unhappy paths. Error handling and display is just as important.

Where possible we’re going to try and use just one Behat feature file. It’s not always possible – the various implementations don’t always want to behave identically.

There’s a ton of good stuff covered in these videos. But the back end is only half the battle.

Whether you want to “catch them all”, or you’re working with a dedicated front-end dev, it’s definitely useful to know the basics of both.

With that in mind, you can pick and choose whether to implement the back-end, or front-end, or both.

If you don’t want to implement a back-end yourself, cloning any of the projects and getting an environment up and running is made as easy as possible by way of Docker. But you don’t need to use Docker. You can bring-your-own database, and do it that way, too.

The Front End

Whatever back end you decide to spin up, the front end should play nicely.

We’re going to implement a few different front-ends. The two I’m revealing today are:

  1. ‘raw’ JavaScript
  2. React

I have plans for a few others, but each implementation is a fair amount of work and I don’t want to over promise at this stage. There’s definitely at least two more coming, but let me first get these two on the site 🙂

The raw JavaScript approach aims to show how things were in the ‘bad old days‘. The days before your package manager  would take up ~7gb of your hard disk with its cache  directory.

The benefit of working this way is that there’s really no extra ‘stuff’ to get in the way. We can focus on making requests, and working with responses.

But that said, this is 2018 and the many modern JavaScript libraries and frameworks are fairly awesome. You’ll definitely get a renewed sense of appreciation for how much easier your life is once you’re comfortable using a library like React, after having done things the hard way.

Again, as mentioned we will cover more than just raw JS and React. Currently each implementation is between ten and fifteen videos. Each video takes a couple of hours to write up, and another couple of hours to record on average. I’m going as fast as I can, and will upload and publish as quickly as possible.

You can watch them as they drop right here.

Site Update

Behind the scenes over the past 10 weeks I have been working on integrating CodeReviewVideos with Braintree.

This is to enable support for PayPal.

I tried to create a ticket for everything I could think of ahead of starting development.

And I added a new ticket for any issue I hit during development. I’m not convinced I tracked absolutely everything, but even so I completely underestimated just how much work would be involved in this feature.

Being completely honest, I have never been more envious of Laravel’s Spark offering. For $99 they get Stripe and Braintree integration, and a whole bunch more. Staggering.

There’s a bunch of other new and interesting features in this release.

I’ve taken the opportunity to migrate from Symfony 3 to Symfony 4 for the API. There’s a bunch of new issues that arose during this transition – I hadn’t given it much prior thought, but with the new front controller (public/index.php) totally broke my Behat (app_acceptance.php) setup.

This work is also enabling the next major feature which I will start work on, once PayPal is live. More on that in my next update.

I appreciate that from the outside looking in, there doesn’t seem to have been a great deal of activity on the site over the last few weeks. I can assure you that behind the scenes, there has never been more activity.

Have A Great Weekend

Ok, that’s about it from me for the moment.

As ever, have a great weekend, and happy coding.

p. s. – I would be extremely grateful if you could help me spread the word by clicking here to tweet about the new course.

Symfony 4: Removing the Mystery

The Beginners Symfony 4 tutorial is in progress. I have all but the final video recorded now. I ended up re-recording a number of sections in both of the recent video uploads:

Why?

One of the areas I found most confusing when first starting with Symfony was in the widespread use of Interfaces.

You may have encountered the following problem:

Let’s say you’re working under a tight deadline. You’re writing some Symfony Controller code and working with a form. It’s not going quite as smoothly as you’d like. You reckon something is going awry with the form submission.

Being the inquisitive developer, you remember the oft touted advice:

Read the source, Luke

The thing is, when you ctrl+click on $form->submit($request); you’re taken to… an interface.

This is good stuff.

Your life will be much easier if you code to an interface, rather than tie your methods to specific implementations.

However, with that deadline looming over your shoulder, such things are nice to know, but right now, just show me the code!

Finding An Implementation

When I first recorded this video I initially just said what the outcome of a call to the submit  method would be.

Watching back, I couldn’t help but think about that stuck, and stressed developer. Sat in a noisy office, headphones in, listening to music when you’d rather have peace and quiet.

Everyone around you seems to be goofing off whilst you’re struggling to think through this really important problem.

The last thing you need is to be met with this weird interface  thing. If only you could find the implementation then life would be a lot less stressful.

How can you find out what is really happening when you call $form->submit($request);?

And what happens when you find the implementation and even then the code is tricky to follow?

I know these feels.

That’s why when recording this video I worked hard to make sure you come out at the end with a good understanding of the code that makes this happen.

This is a beginners series for Symfony 4. This is the stuff that will make working with, and understanding Symfony that much easier.

I hope you enjoy it.

Site Stuff

There’s a ton of work going on behind the scenes at the moment.

Can I pay by PayPal?

This is one of the most frequently asked questions that I get.

Currently: no.

That sucks. I appreciate that.

The reasoning for this is that Stripe is super shiny and as a developer, they were high on my wish list of cool things to implement.

Also, from a code perspective Stripe is actually a joy to work with. They really are awesome.

But still, I get asked a lot for PayPal.

I’m adding PayPal.

It’s quite a big job, but I’m about 65% of the way through the implementation.

Here’s a sneak peak:

That’s the Stripe form using Stripe’s React elements.

PayPal functionality is provided through Braintree.

There’s a nice transition between the two options too, which came for free via Bootstrap 4 and I really like it.

I’ve been working on completely extracting the membership code.

Would you have any interest in seeing video tutorials on how to make your own Symfony bundles?

Leave a comment and let me know.

There’s some other cool features that this work enables, which I’ll share with you in a future update.

Until next time, have a great weekend and happy coding.

Chris