API - Pagination, (Basic) Filtering, and Sorting


In this video we are going to add in Pagination, Filtering, and Sorting to our API implementation - again, making use of the KNP Paginator Bundle.

This is almost identical to what we have implemented for the Twig setup, but with some restrictions on what JMS Serializer should output.

With the Twig implementation we didn't need to worry about having extra properties exposed, but unless we directly address this problem in our API implementation, we will expose a whole bunch of things that will serve to confuse our API consumers.

Aside from that, the other difference is that we won't have a Twig helper function to render out a nice pagination widget for us. Instead, we must directly manipulate the URL to ensure we can paginate, filter, sort, and so on. This means... writing documentation! wooo, everyone loves documenting.

Let's get started as there's quite a lot to cover.

Pagination Configuration

The first thing to do is add in the configuration for KNP Paginator.

If you haven't yet done so, go ahead and follow the installation instructions.

However, as we won't be using Twig, there's no need to add in the template section, so our final config will look something like:

# KNP Paginator
knp_paginator:
    page_range: 5                      # default page range used in pagination control
    default_options:
        page_name: page                # page query parameter name
        sort_field_name: sort          # sort field query parameter name
        sort_direction_name: direction # sort direction query parameter name
        distinct: true                 # ensure distinct results, useful when ORM queries are using GROUP BY statements

Basically it's the same as we used for our Twig implementation, only without the template section. Easy.

With the config added we can move on to adding this to our API endpoint.

In the Twig implementation we added the pagination, filtering, and sorting setup to our listAction.

The equivalent action in our API implementation would be the cgetAction - the collection GET action.

Currently we have a one-liner in here:

// /src/AppBundle/Controller/BlogPostsController.php

    public function cgetAction()
    {
        return $this->getBlogPostRepository()->createFindAllQuery()->getResult();
    }

Which works fine, but won't allow us to paginate, filter, and sort.

We have covered the reasoning behind this in the previous three videos, but in summary, the paginator can work with a result but this is inefficient.

Instead, if we pass in an as-yet-unrun query, KNP Paginator can figure out what limit and offset to apply before running the query, reducing the stress on our database.

Another advantage, as we saw in the previous video is that we could apply filtering logic to our query before the KNP Paginator runs the query.

Combined, this looks like:

// /src/AppBundle/Controller/BlogPostsController.php

    public function cgetAction(Request $request)
    {
        $queryBuilder = $this->getBlogPostRepository()->createFindAllQuery();

        if ($request->query->getAlnum('filter')) {
            $queryBuilder->where('bp.title LIKE :title')
                ->setParameter('title', '%' . $request->query->getAlnum('filter') . '%');
        }

        return $this->get('knp_paginator')->paginate(
            $queryBuilder->getQuery(), /* query NOT result */
            $request->query->getInt('page', 1), /*page number*/
            $request->query->getInt('limit', 10)/*limit per page*/
        );
    }

Essentially this is very similar to how our Twig implementation operates. In the Twig implementation I created the query builder directly in the Controller action, but here that responsibility has been delegated to the BlogPostRepository:

<?php

// /src/AppBundle/Entity/Repository/BlogPostRepository.php

namespace AppBundle\Entity\Repository;

use Doctrine\ORM\EntityRepository;

class BlogPostRepository extends EntityRepository
{
    public function createFindAllQuery()
    {
        return $this->_em->getRepository('AppBundle:BlogPost')->createQueryBuilder('bp');
    }

    // * snip *

Whilst not immediately obvious, creating the query builder in this way does implicitly define a SELECT * FROM BlogPost (pseudocode btw). And also note, this returns a query builder - not a query. We must call getQuery() to... ahem, get the query.

The reason for getting the query builder rather than DQL is to allow us to modify the query, should we need to, which we might if the user has passed in the filter parameter:

if ($request->query->getAlnum('filter')) {
    $queryBuilder->where('bp.title LIKE :title')
        ->setParameter('title', '%' . $request->query->getAlnum('filter') . '%');
}

As we covered in the Simple Filtering in Twig video, this is a very basic filtering setup, and is likely not suitable for your real world needs. This is demo code, change the implementation accordingly.

Finally, the paginator expects a query, a page number, and a limit. This has all been inlined to save space, but you can expand this out if you wish:

    return $this->get('knp_paginator')->paginate(
        $queryBuilder->getQuery(), /* query NOT result */
        $request->query->getInt('page', 1), /*page number*/
        $request->query->getInt('limit', 10)/*limit per page*/
    );

If any of this is confusing to you, please do watch the previous videos.

Initial Output

This should now work. We can hit our API /posts endpoint and get the limited, paginated JSON output.

But there's a problem.

We get too much output. We get a whole bunch of extra info we don't want to expose:

{
    "current_page_number": 1,
    "num_items_per_page": 10,
    "items": [
        { 
            "id": 1,
            "title": "some interesting title",
            "body": "our body content"
        },
        {
            // etc
        }
    ],
    "total_count": 50,
    "paginator_options": {
        "pageParameterName": "page",
        "sortFieldParameterName": "sort",
        "sortDirectionParameterName": "direction",
        "filterFieldParameterName": "filterField",
        "filterValueParameterName": "filterValue",
        "distinct": true
    },
    "custom_parameters": [],
    "route": "get_posts",
    "params": {
        "page": 1
    },
    "page_range": 5,
    "template": "KnpPaginatorBundle:Pagination:sliding.html.twig",
    "sortable_template": "KnpPaginatorBundle:Pagination:sortable_link.html.twig",
    "filtration_template": "KnpPaginatorBundle:Pagination:filtration.html.twig"
}

Some of this is helpful. Some is really just noise. Some may even be exposing security / implementation details we don't want to expose.

Fixing this is easy enough: it's a copy / paste exercise.

/app/config/config.yml

#JMS Serializer
jms_serializer:
    metadata:
        directories:
            KnpPaginatorBundle:
                namespace_prefix: Knp\Bundle\PaginatorBundle
                path: %kernel.root_dir%/config/serializer/KnpPaginatorBundle
            KnpPager:
                namespace_prefix: Knp\Component\Pager
                path: %kernel.root_dir%/config/serializer/KnpPager

Based on this config, we need to add in two new files:

%kernel.root_dir%/config/serializer/KnpPaginatorBundle/Pagination.SlidingPagination.yml

and

%kernel.root_dir%/config/serializer/KnpPager/Pagination.AbstractPagination.yml

In Symfony-land, if you aren't already aware, %kernel.root_dir% will point to the app/ directory - unless you have been meddling, in which case it will be wherever you moved it too. My advice - don't move it.

So, really, these paths are:

/app/config/serializer/KnpPaginatorBundle

and

/app/config/serializer/KnpPager

Awesome.

The contents of these two files are:

# /app/config/serializer/KnpPager/Pagination.AbstractPagination.yml

Knp\Component\Pager\Pagination\AbstractPagination:
    exclusion_policy: ALL
    accessor_order: custom
    custom_accessor_order: [currentPageNumber, numItemsPerPage, totalCount, items]
    properties:
        items:
            expose: true
            access_type: public_method
            accessor:
                getter: getItems
            type: array
            serialized_name:
                data
        currentPageNumber:
            expose: true
            serialized_name:
                currentPage
        numItemsPerPage:
            expose: true
            serialized_name:
                itemsPerPage
        totalCount:
            expose: true
            serialized_name:
                totalItems

and

# /app/config/serializer/KnpPaginatorBundle/Pagination.SlidingPagination.yml

Knp\Bundle\PaginatorBundle\Pagination\SlidingPagination:
    exclusion_policy: ALL

After this, it would be advised to clear the cache:

php bin/console cache:clear -e=dev

And really all this is doing is excluding all the 'stuff' from being exposed, and only exposing the fields we explicitly tell it to.

I don't claim to have figured this out, I found all the answers I share here via the GitHub repo for KNP Paginator, and StackOverflow.

The last thing I have done here is changed the order in which the JSON is output. This makes it nicer for humans / developers.

This should now all work. Manually accessing this data is now good, but we'd really rather have this done for us by JavaScript, so we will use our development chops to make React and Angular do just that.

Code For This Video

Get the code for this video.

Episodes