No One Likes Errors [API Platform]

At this point we have a nice, working API Platform implementation for our Album resources. We can GET, PUT, POST, and DELETE. But we have largely only tested the "happy path".

Back in the real world, most of the hard work in software development goes into ensuring we either entirely avoid, or defensively mitigate against the "unhappy path". Those pesky situations whereby an API consumer does something untoward, or unwanted. I won't say unexpected, as it's our job to try and plan for the moments of madness that end users seemingly enjoy exploring.

We have some existing "edge case" tests that we can start from.

Here's one:

  Scenario: Must have a track count of one or greater
    Given the request body is:
      """
      {
        "title": "My album title",
        "track_count": 0,
        "release_date": "2030-12-05T01:02:03+00:00"
      }
      """
    When I request "/album" using HTTP POST
    Then the response code is 400
    And the response body contains JSON:
    """
    {
        "status": "error",
        "errors": {
            "children": {
                "title": [],
                "release_date": [],
                "track_count": {
                    "errors": [
                        "This value should be greater than 0."
                    ]
                }
            }
        }
    }
    """

It's almost inevitable at this point that our API Platform implementation is not going to behave identically to this. The entire process of serializing errors is different to our previous implementations.

Let's send this in manually and see what we do get:

{
    "@context": "/contexts/ConstraintViolationList",
    "@type": "ConstraintViolationList",
    "hydra:title": "An error occurred",
    "hydra:description": "trackCount: This value should be greater than 0.",
    "violations": [
        {
            "propertyPath": "trackCount",
            "message": "This value should be greater than 0."
        }
    ]
}

This is really nice.

It's not at all what our Behat test expects. But it is the same behaviour. Only the presentation is different.

There's no simple way to munge our test suite to make this generic. We're going to need new tests.

Here's the revised test suite for error checking with JSON-LD:

Feature: Provide insight into how the API Platform behaves on the unhappy path

  In order to eliminate bad Album data
  As a JSON API developer
  I need to ensure Album data meets expected criteria

  Background:
    Given there are Albums with the following details:
      | title                              | track_count | release_date              |
      | some fake album name               | 12          | 2020-01-08T00:00:00+00:00 |
      | another great album                | 9           | 2019-01-07T23:22:21+00:00 |
      | now that's what I call Album vol 2 | 23          | 2018-02-06T11:10:09+00:00 |

  Scenario: Must have a non-blank title
    Given the request body is:
      """
      {
        "title": "",
        "track_count": 7,
        "release_date": "2030-12-05T01:02:03+00:00"
      }
      """
    When I request "/album" using HTTP POST
    Then the response code is 400
    And the response body contains JSON:
    """
    {
        "@context": "/contexts/ConstraintViolationList",
        "@type": "ConstraintViolationList",
        "hydra:title": "An error occurred",
        "hydra:description": "title: This value should not be blank.",
        "violations": [
            {
                "propertyPath": "title",
                "message": "This value should not be blank."
            }
        ]
    }
    """

  Scenario: Must have a track count of one or greater
    Given the request body is:
      """
      {
        "title": "My album title",
        "track_count": 0,
        "release_date": "2030-12-05T01:02:03+00:00"
      }
      """
    When I request "/album" using HTTP POST
    Then the response code is 400
    And the response body contains JSON:
    """
    {
        "@context": "/contexts/ConstraintViolationList",
        "@type": "ConstraintViolationList",
        "hydra:title": "An error occurred",
        "hydra:description": "trackCount: This value should be greater than 0.",
        "violations": [
            {
                "propertyPath": "trackCount",
                "message": "This value should be greater than 0."
            }
        ]
    }
    """

  Scenario: Must have a track count of one or greater
    Given the request body is:
      """
      {
        "title": "My album title",
        "track_count": -5,
        "release_date": "2030-12-05T01:02:03+00:00"
      }
      """
    When I request "/album" using HTTP POST
    Then the response code is 400
    And the response body contains JSON:
    """
    {
        "@context": "/contexts/ConstraintViolationList",
        "@type": "ConstraintViolationList",
        "hydra:title": "An error occurred",
        "hydra:description": "trackCount: This value should be greater than 0.",
        "violations": [
            {
                "propertyPath": "trackCount",
                "message": "This value should be greater than 0."
            }
        ]
    }
    """

  Scenario: Album ID must be numeric
    Given I request "/album/a" using HTTP GET
    Then the response code is 404

  Scenario: Album ID must be positive
    Given I request "/album/-99" using HTTP GET
    Then the response code is 404

Honestly I'd love to say there's some ultra slick way of coming up with these tests.

There isn't.

You simply have to try to imagine the numerous ways your API consumers might (read: will) screw up, and then document this in your test suite.

My approach is very unscientific. I simply use Postman to send in manual requests, and take a copy of the response. This becomes my test.

Yes, this means you need a working API in order to write your tests. No, this is not true TDD. But then, this isn't TDD - this is a loose form of Behaviour Driven Development.

Anyway, the outcome is what's important (imo). We document and test our system. The system behaves as we expect.

However you look at it, the nice thing is: we're done. That's our API Platform implementation complete.