The Happiest Path

After we have done some checks to ensure we have a valid URL, it is time to actually visit / fetch that URL.

As we did the earlier hard work of extracting our fetching process to a separate function, we can now easily mock the responses we expect, and get the most basic functionality of our URL checker to work.

import { visit } from "./visit";
import * as Fetcher from "./fetcher";

const validUrl = "https://some.valid.url";

describe("visit", () => {

  // ... other tests

  test("should return a good request representation if given a 2xx response", async () => {
    const fetcherSpy = jest.spyOn(Fetcher, "fetcher").mockResolvedValueOnce({
      ok: true,
      url: validUrl,
      status: 200,
      statusText: "OK",
      headers: { a: "b" },
    });

    expect(await visit(validUrl)).toEqual([
      {
        ok: true,
        url: validUrl,
        status: 200,
        statusText: "OK",
        headers: { a: "b" },
        redirected: false,
      },
    ]);

    expect(fetcherSpy).toHaveBeenCalledTimes(1);
    expect(fetcherSpy).toHaveBeenCalledWith(validUrl);
  });
Code language: TypeScript (typescript)

The “spyOn” function is used to track how the “fetcher” method is called and what it returns. You can read up on spyOn on the Jest docs.

Notice on line 2 that we import * as Fetcher. I’ve not found a way to use spyOn when importing individual functions from an external file. By which I mean:

import {fetcher} from './fetcher';

This seems a little verbose, but it only impacts our test code – and also only indirectly, as we shall soon see.

The .mockResolvedValueOnce function is used to specify that when the “fetcher” method is called, it should return a specific object (in this case, an object with properties “ok“, “url“, “status“, “statusText“, and “headers“) and only once.

With this setup, you can use the spy to check if the “fetcher” method was called and with what arguments, and also to check if the expected return value is being returned.

Seeing It Work

We have added a failing test:

We expect an array of one object, but our implementation currently returns an empty array.

Let’s do the simplest thing we can to actually see what the outcome of running the fetcher against a real, known working URL would be.

I’m going to use the URL of https://codereviewvideos.com, but change it up to anything you like.

Here’s there code change:

import { VisitedURL } from "./types";
import { badRequest } from "./bad-request";
import { isValidUrl } from "./is-valid-url";
import { fetcher } from "./fetcher";

export const visit = async (
  href: string,
  requests: VisitedURL[] = []
): Promise<VisitedURL[]> => {
  if (!isValidUrl(href)) {
    return [
      ...requests,
      {
        ...badRequest,
        url: href,
        statusText: `Invalid URL`,
      },
    ];
  }

  const hrefToUrl = new URL(href);

  if (!["http:", "https:"].includes(hrefToUrl.protocol)) {
    return [
      ...requests,
      {
        ...badRequest,
        url: href,
        statusText: `Unsupported protocol: "${hrefToUrl.protocol}"`,
      },
    ];
  }

  const result = await fetcher("https://codereviewvideos.com");
  console.log(result);

  return [];
};
Code language: TypeScript (typescript)

First we will need to compile our TypeScript code to JavaScript, so Node can run it from the command line. You can do this in several ways, but most simply:

// from your project root dir
node ./node_modules/.bin/tscCode language: JavaScript (javascript)

This should, if you are following along with the way I’ve been working, spit out lots of JavaScript files in your ./dist directory.

You can then call the index.js file using Node:

node dist/v2/index.js

(node:1706271) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)

{
  url: 'https://codereviewvideos.com/',
  status: 200,
  statusText: 'OK',
  ok: true,
  headers: {
    'cache-control': 'max-age=3, must-revalidate',
    'content-encoding': 'gzip',
    'content-length': '10162',
    'content-type': 'text/html; charset=UTF-8',
    date: 'Fri, 20 Jan 2023 15:27:56 GMT',
    'last-modified': 'Fri, 20 Jan 2023 15:19:07 GMT',
    server: 'nginx',
    vary: 'Accept-Encoding, Cookie',
    'x-powered-by': 'PHP/8.1.12'
  }
}

[]Code language: Shell Session (shell)

I’ve added some line breaks in.

Maybe this view is easier to read:

The gist is that we can see a bunch of interesting information, restricted down to the keys / values we asked that the fetcher should return.

At the bottom we also see the empty array which what our visit function returns.

Making It Pass

OK, so we have seen that our code works.

That’s quite nice.

Now, let’s make the test pass.

We already have everything we need. However, if we make the simplest change to our code to make this work, we will see a TypeScript error:

This is because we declared that our visit function returns:

Promise<VisitedURL[]>

And VisitedURL is this:

export type VisitedURL = {
  url: string;
  status: number;
  statusText: string;
  ok: boolean;
  redirected: boolean;
  headers: Record<string, string>;
};
Code language: TypeScript (typescript)

redirected is not something that our fetcher tells us.

We could hack this in, for now:

  const result = await fetcher("https://codereviewvideos.com");

  return [
    {
      ...result,
      redirected: false,
    },
  ];Code language: TypeScript (typescript)

Here we spread (...result) everything that is contained in result into a new object.

Then, to that new object we add in the key of redirected, with the hardcoded value of false.

That satisfies TypeScript, but our test still fails:

We need to change out that hardcoded URL to be the one given to the visit function:

  const result = await fetcher(href);

  return [
    {
      ...result,
      redirected: false,
    },
  ];
Code language: TypeScript (typescript)

And we are definitely OK to use href here, as we did all the validation previously to ensure this is a valid URL.

At this point we should have a full set of passing tests.

However, this is just the happiest of happy paths. We need to do more than this, so let’s continue on and look at some of the other possible outcomes.