End-to-End Testing with TestCafe Book Excerpt: Intercepting HTTP Requests

This article is an excerpt from one of the chapters of my upcoming book, End-to-End Testing with TestCafe. If you're interested in learning more about the book, visit https://testingwithtestcafe.com.

As more web applications shift to using more interactive user experiences, testing will get more complicated with all the moving parts in place. Since end-to-end testing verifies how a system works as a whole, testers also need to ensure that the communication between external services works as expected.

Usually, these requests happen asynchronously in the background through JavaScript. The client-side portion of the application won't see what's happening until the external service returns a response unless there's some progress indicator programmed in the interface. Even then, there's no guarantee on what's getting returned to the client, making it difficult to perform any assertions on the page.

Intercepting an HTTP request and its response gives you the ability to take control over what happens during asynchronous calls to other servers. Some of the primary uses for intercepting HTTP requests are:

Recording all requests and responses from a remote service

You can open up your preferred browser's developer tools during manual testing and see the different network requests made in the application. This information allows you to see everything related to the request and the response, like the headers sent to the server, and the data returned to the client.

However, during automated testing, you won't have a chance to check the developer tools to see what's going on during the testing process. TestCafe allows you to record any HTTP requests made during the testing process and verify that the data returned from the server is what you expect.

Reliably manage responses from third-party services to ensure repeatable tests

As mentioned, responses coming from an HTTP request are usually out of our control as testers. Some systems provide testing environments, such as a sandbox environment that gives you control over its responses. In most cases, your tests won't have the luxury of specialized testing setups, and you'll have to use a real interface to the service.

Depending on the service where you make a request, you may not always receive the same data every time, making it tricky to create robust validations in your tests. With TestCafe, you can manipulate a response from an HTTP request to create repeatable tests without worrying about the data returned from a service you don't manage.

Creating tests for services that are under development or difficult to replicate

Testing should take place early in the development process to bake in quality from the start. The problem is that when an application is under active development, you'll have rapid changes occurring every day. It's impossible to build a test suite if the service continually changes the way it responds.

Another issue teams may face, particularly small bootstrapped teams, is not having infrastructure available to set up complex systems for testing purposes. Sometimes an organization doesn't have the resources to set up an environment that's not easy to duplicate on a smaller scale.

Fortunately, you don't have to wait for the team to finish developing or setting up a system before you can write tests with TestCafe. By creating a mock that simulates the interfaces you need to verify, you can build tests for environments that aren't readily available for use.

Recording and manipulating external requests have many real-world uses that can save time and help build a reliable and stable test suite. Some examples are:

• It's not uncommon for developers to accidentally break an application because of a small change in the response for an HTTP request. You can include assertions to check that those requests and responses work as expected, and raise alerts early during testing if there's a change that can potentially break the application.
• If your application performs intensive calculations that take time to execute and complete, it can slow down your test suite or cause your tests to fail due to timeouts. You can bypass these demanding functions with a mock to keep your tests speedy.
• Testing doesn't always cover the happy path. You'll also need to test scenarios that don't occur under regular use, such as a network or database error. Typically, you shouldn't be able to trigger one of these errors on your own. However, you can mock an error response from an HTTP request to cover those scenarios during your test run.
• If you practice test-driven development (TDD), you don't need to wait for systems to get entirely built before creating new tests. Mocking these systems accelerates the process by allowing you to test complex interfaces before they exist.
• Sometimes you won't have access to some services because they contain sensitive information that you don't want to expose during testing. You can bypass any HTTP request made to these servers with a mock to keep your info safe inside of your tests.

Now that you understand the reasons you'd want to intercept HTTP requests in your tests, let's take a look at how you can do this in TestCafe. TestCafe has different hooks to grab HTTP requests, depending on whether you need to record or manipulate the responses.

Logging HTTP requests

First, let's see how you can log any HTTP request that occurs in the application during testing. TestCafe has a class called RequestLogger that checks any HTTP request that occurs while the test runs and records both the request the browser makes and the response it receives from the server.

To log your HTTP requests, you need to import the RequestLogger constructor to your tests and use it to create an instance of the class.

import { RequestLogger } from "testcafe";

const logger = RequestLogger();

The RequestLogger constructor accepts two optional parameters. The first optional parameter is a filter to tell the request logger which HTTP requests it should track. If a filter is not specified, the request logger tracks every HTTP request made during the test. Usually, this behavior isn't beneficial since it logs everything that loads on the page, like images and scripts. You'll want to be more specific about which requests to capture during your test run.

You can filter which HTTP requests you want to record in different ways:

By exact URL

You can pass a string to log any requests sent to a specific URL.

// Only logs requests made to this specific URL.
const logger = RequestLogger("https://teamyap.app/api/posts");

Multiple URLs

If you need to track requests to multiple URLs, you can use an array to specify more than one URL.

// Only logs requests made to the specific URLs
// defined in the array.
const logger = RequestLogger([
    "https://teamyap.app/api/posts",
    "https://teamyap.app/api/conversations"
]);

Using regular expressions

You can use regular expressions to match a URL by a pattern. Using regular expressions is useful when the request URL changes depending on the situation, like an ID associated with a resource in the app.

// Logs requests made to URLs matching this regular expression
// like "https://teamyap.app/api/posts/12345/comments"
const logger = RequestLogger(/\/api\/posts\/(\d+)\/comments/);

Filtering AJAX requests or by request method

If you use a string, an array of strings, or a regular expression, the request logger will record all responses made to any matching URL, regardless of whether it's an asynchronous (AJAX) request or the request method (like GET or POST). If you want to get more specific, you can use an object with the following properties:

• url - The URL you want to log requests from.
• method - The HTTP method you want to record (GET, POST, PUT, PATCH, or DELETE).
• isAjax - A boolean flag to only record asynchronous HTTP requests.

// Only logs asynchronous POST requests made to
// the specific URL.
const logger = RequestLogger({
    url: "https://teamyap.app/api/posts",
    method: "post",
    isAjax: true
});

More control using a predicate function

If you need to fine-tune the request logger even further, you can use a predicate function. The function takes a request parameter that contains different properties that you can use to match the exact HTTP requests you want to filter:

• url - The URL you want to log requests from.
• method - The HTTP method you want to record.
• isAjax - A boolean flag to only record asynchronous HTTP requests.
• body - A string containing the body of the request.
• headers - An object containing the request headers in key-value form.
• userAgent - A string identifying the user agent making the request.

You can run conditional statements on one or more of these properties inside of the predicate function. The request logger will only log requests where all the conditions are true.

// Only logs asynchronous PATCH requests made to
// the specific URL that have a header to indicate
// this is a JSON request.
const logger = RequestLogger(request => {
    return request.url === "https://teamyap.app/api/posts" &&
        request.method === "patch" &&
        request.isAjax &&
        request.headers['Content-Type'] === 'application/json';
});

The second optional parameter for a request logger is an object that lets you configure which information you want the logger to capture. By default, the request logger only returns basic details like the request and response timestamps, the request URL and method, and the response status code. If you need more information like the headers and body, you can set it in this optional object with the following boolean properties:

• logRequestHeaders - Lets you specify if you want the request logger to log the headers made by the HTTP request.
• logRequestBody - Lets you specify if you want the request logger to log the body of the HTTP request.
• stringifyRequestBody - If you set logRequestBody to true, the request body of the request gets recorded as a Node.js Buffer object by default. This option converts the request body to a string. If you set this option to true without setting logRequestBody to true, TestCafe will throw an error.
• logResponseHeaders - Lets you specify if you want the request logger to log the headers of the server response.
• logResponseBody - Lets you specify if you want the request logger to log the body of the server response.
• stringifyResponseBody - If you set logResponseBody to true, the response body of the request gets recorded as a Node.js Buffer object by default. This option converts the response body to a string. If you set this option to true without setting logResponseBody to true, TestCafe will throw an error.

// Logs requests made to the specific URL, including
// the response header and body.
const logger = RequestLogger("https://teamyap.app/api/posts", {
    logResponseHeaders: true,
    logResponseBody: true,
    stringifyResponseBody: true
});

After creating a request logger instance, you can access additional properties and methods that let you manage any captured requests in your tests. These allow you to look at all requests and responses, run assertions, and clear the logger instance.

If you want to view all the requests and responses the logger captures in a test, the requests property returns an array of objects containing details about every intercepted request.

// Somewhere inside of your tests after running a few actions.
logger.requests;

The objects inside of the requests array contain information about the request and the response from the server. Each object will also include the headers and body for the request and response if you set additional options like logRequestHeaders or logResponseBody. Here's an example of a few logged requests:

[
  {
    id: 'yqw_dSpIl',
    testRunId: 'p0P1LuJtg',
    userAgent: 'Chrome 83.0.4103.116 / macOS 10.15.5',
    request: {
      timestamp: 1592980791729,
      url: 'https://teamyap.app/api/posts',
      method: 'get'
    },
    response: {
      statusCode: 200,
      timestamp: 1592980791879,
      headers: [Object],
      body: ''
    }
  },
  {
    id: 'K79qg9fRi',
    testRunId: 'p0P1LuJtg',
    userAgent: 'Chrome 83.0.4103.116 / macOS 10.15.5',
    request: {
      timestamp: 1592980793340,
      url: 'https://teamyap.app/api/posts',
      method: 'post'
    },
    response: {
      statusCode: 201,
      timestamp: 1592980793505,
      headers: [Object],
      body: '{"id":1,"body":"Can someone leave a comment?"}}'
    }
  }
]

You can use the requests property to run assertions, like verifying the number of requests by checking the length of the array. However, the request logger has additional methods - contains and count - to help you validate specific requests, like checking if a request happened or if an exact number of requests occurred during testing.

The contains method lets you run an assertion by specifying a predicate function, similar to the second optional parameter when creating an instance of a request logger. If the conditions of the function match a logged request, it will return true. Otherwise, it returns false.

As an example, let's say you have a logger that intercepted the requests shown when we talked about the requests property. If you want to assert that the logger captured one POST request and received a status code of 201, you can run the following assertion:

await t.expect(logger.contains(request => {
  return request.request.method === "post" &&
      request.response.statusCode === 201;
})).ok()

The count method works similarly to contains. You can set a predicate function to set conditions for matching captured requests in the logger. The function then returns the number of requests that match those conditions.

You can use the count method to perform assertions to verify that the logger captured a specific amount of requests. For instance, using the same example logger requests as above, if you want to confirm that the test intercepted one GET request from the specified URL, you can use the following assertion:

await t.expect(logger.count(request => {
  return request.request.url === "https://teamyap.app/api/posts" &&
        request.request.method === "get"
})).eql(1)

Note

The contains and count methods both return a Promise object. If you use either of these methods in an assertion, TestCafe uses the Smart Assertion Query Mechanism, as discussed in Chapter 10.

Finally, if you need to clear the intercepted requests at any point in your tests, you can use the clear method.

// Somewhere inside of your tests after running a few actions.
logger.clear();

You usually won't need to clear requests captured by the request logger. You can fine-tune which requests to catch when instantiating the request logger object, and methods like contains and count help you refine your assertions. However, some applications make unnecessary and redundant requests that make it challenging to run assertions, so the clear method is useful in those scenarios.

Mocking HTTP requests

Besides logging HTTP requests and verifying what it captures, TestCafe also lets you alter the responses for these requests. The RequestMock class sets up a request mocker object, which you can then use to manipulate requests as needed for your tests. To set up a request mocker, import the RequestMock constructor method to your tests. Once imported, you can use it to create an instance of the class.

import { RequestMock } from "testcafe";

const mock = RequestMock();

However, the object won't do much by itself. Instances of RequestMock have two required methods that you must chain together to create a mock: onRequestTo and respond. Together, these methods form a request mocker that lets you manage how you want any of the application's HTTP requests to respond during testing.

The onRequestTo method allows you to specify which HTTP request you want to intercept to manage its response during the test run. The argument required by the method is the same as the optional filtering argument used when creating a RequestLogger instance:

• A string containing the exact URL.
• An array of URLs.
• A regular expression.
• An object that lets you specify the URL, method, and if it's an AJAX request.
• A predicate function.

The respond method allows you to specify what you want to use as the response during testing. With this method, you can manage what the browser receives for your testing needs. You can set the body of the response, its HTTP status code, and set the response headers. This method allows you to set three optional arguments.

The first optional argument is the mocked body of the response. You can return a string to simulate an HTML response, an object or array for a JSON response, or a function to customize the response body even further. Most applications use the response body to update the page, so setting the body lets you control how it reacts to a request. If not specified, the mock will return an empty HTML response.

The second optional argument is the numeric HTTP status code of the response. For instance, you can have the request return "200" for a successful response, "404" for a "Page Not Found" response, or "500" to simulate internal server errors. Many asynchronous requests check the status code to indicate if a request was successful or not, so this option may be necessary for your tests. By default, the request mocker will return a status code of "200".

The third optional argument is an object to set custom headers to the response. Some applications use specific headers for it to work correctly. You can use this argument to set those headers as needed. When this option is empty, TestCafe sets a content-type header according to the first argument:

• If the mocked body of the response is an array or object, the value of the content-type header will be application/json.
• If the mocked body of the response is a string, the value of the content-type header will be text/html; charset=utf-8.

Here are a few different examples of setting up a request mocker:

// Mock any call to the specified URL and return an
// object to simulate a JSON response.
RequestMock()
    .onRequestTo("https://teamyap.app/api/posts")
    .respond([{ id: 1, body: "Can someone leave a comment?" }]);

// Mock any call to URLs that match the regular expression
// pattern, and return an HTML response with a 404 status code.
RequestMock()
    .onRequestTo(/\/api\/posts\/(\d+)\/comments/)
    .respond("<div>Not found</div>", 404);

// Mock any call to URLs that satisfy the conditions of the
// predicate function, and return an empty HTML response with
// a 201 status code and a custom header.
RequestMock()
    .onRequestTo(request => {
        return request.url === "https://teamyap.app/api/posts" &&
            request.method === "post" &&
            request.isAjax;
    })
    .respond(, {
        "Content-Length": "100"
    });

You can also chain multiple onRequestTo and respond methods to a single RequestMock instance to mock more than one request, each with a different response:

// Mocks different URLs and returns a different response
// for each chained method.
RequestMock()
    .onRequestTo({
        url: "https://teamyap.app/api/posts",
        method: "post",
        isAjax: true
    })
    .respond(null, 201)
    .onRequestTo(/\/api\/posts\/(\d+)\/comments/)
    .respond("<div>Not found</div>", 404);

Note

One of the most common uses of setting custom headers when mocking an asynchronous request to an external service is to deal with the Cross-Origin Resource Sharing mechanism, also known as CORS. CORS is a security mechanism handled by your web browser that allows one application in a domain to specify who can access its resources.

On a basic level, the way CORS works is by performing an initial request to the API before our desired API request, known as a preflight request. The preflight request verifies if the originating server where the request is coming from is allowed to make a cross-domain request. If it is, then it returns a successful response with a few headers that tell the browser it's okay to make further API requests.

When you want to mock an API request that goes to a different domain, the browser still goes through its standard CORS check, even with a request mocker object. That means you need to set the appropriate headers in your mocks to make the browser think that the CORS check passed.

The headers that the browser needs to perform a successful CORS check vary, depending on the configuration of the application under test. If you need to set these headers, ask the application developers which headers are required to make cross-origin HTTP requests during testing.

Attaching to tests and fixtures

Creating a request logger or mocker instance won't work on its own. You need to tell TestCafe to use these hooks in a specific test or fixture to log and mock HTTP requests using the objects you create in your test suite.

The fixture and test functions have a method called requestHooks where you can attach either an instance of RequestLogger or RequestMock. Setting up a request logger or mocker instance in this method will automatically set them up for intercepting HTTP requests.

const logger = RequestLogger("https://teamyap.app/api/posts");
const mock = RequestMock()
    .onRequestTo("https://teamyap.app/api/posts")
    .respond(null, 201);

// Attaches the request logger to the fixture and logs
// HTTP requests for all tests under the fixture.
fixture("My test fixture")
    .requestHooks(logger);

// Attaches the request mocker and intercepts and
// mocks HTTP requests for this test only.
test
    .requestHooks(mock)
    ("Test to mock HTTP requests", async t => {
        // Your test code goes here
    });

If you need to attach more than one request hook in a fixture or test, you can pass them in an array or define them as multiple parameters.

// Both of these examples attach the request logger and
// mocker to the fixture.
fixture("My test fixture")
    .requestHooks([logger, mock]);

fixture("My test fixture")
    .requestHooks(logger, mock);

Once a hook is attached to a fixture or test, TestCafe will intercept HTTP requests as defined in the request logger or mocker objects as soon as you run your tests. You can use a request logger object inside the tests to validate any captured requests, and request mockers will automatically simulate any responses matching the hook.

In some cases, you may not want to have a request hook intercepting HTTP requests from the start of your test run. For instance, there might be scenarios where it's okay to allow a request to occur without any interference from a request mocker, and after a few actions, you want to begin catching those requests and mock the response.

You aren't limited to setting a request logger or mocker from the start of your tests. The test controller object has two methods to give you more control over request hooks so you can attach and detach hooks in the middle of a test as needed. The t.addRequestHooks method attaches a request hook at any point in your test, and the t.removeRequestHooks method detaches the hook when you no longer want to intercept the requests.

test("Test to mock HTTP requests", async t => {
    // Run a few actions without intercepting HTTP requests.

    // Attach a request mocker to begin intercepting requests.
    await t.addRequestHooks(mock);

    // Run more actions

    // Detach the mocker to stop intercepting requests.
    await t.removeRequestHooks(mock);
});

If you found this article useful, you can pre-order the End-to-End Testing with TestCafe book at https://testingwithtestcafe.com and receive $10 off the book's original price when pre-ordering before the expected release date (on or before July 15, 2020).

If you sign up to the mailing list on the book's website, you'll receive the first three chapters of the book for free. In addition to the book sample, you'll get an exclusive discount to pre-order the book.