Avoiding False Positives in Node.js Tests

When running tests, it's a great feeling to see dozens of green check marks indicating that a test suite is passing. It's especially gratifying after tackling a tricky bug or slogging through a tough feature. But those passing tests may be giving you a false sense of security.

Often, bugs lurk in passing tests, undermining trust in the test suite and your application. Such tests can cause more harm than good, giving you a hearty pat on the back while hiding broken functionality. It can take months to weed out a false positive test, and that might be only after a customer complaint.

This post examines several common false positive patterns that can crop up in test suites. Taken out of context, the examples may appear obvious, but they find their way into complex, real-world tests all the same.

Prerequisites

I will assume readers are familiar with ES6 JavaScript syntax and have written at least a handful of tests in NodeJS.

Although I don't intend to be technology-specific, inevitably some of the pitfalls are specific to quirks in the JavaScript ecosystem (particularly its UI testing libraries). I expect readers will be unfamiliar with some (or most) of the libraries featured in the post, but I hope the underlying principles will be relevant.

This post is current as of Node 22.11.1, jest 29.7.0, mocha 10.3.0, chai 5.1.0, chai-http 4.3.0, supertest 6.3.3, eslint 8.57.0, @playwright/test 1.42.1, and @testing-library/react 13.4.0.

Let's begin.

Common False Positive Test Patterns in Node.js

First, we'll look at some common patterns when it comes to false positive test results.

Using equal() Rather Than strictEqual()

As a warmup, check out this Mocha test with Chai assertions:

import { assert } from "chai";

describe("strict equality", () => {
  test("1 equals '1'", () => {
    assert.equal(1, "1");
  });

  test("0 equals false", () => {
    assert.equal(0, false);
  });

  test("null equals undefined", () => {
    assert.equal(null, undefined);
  });
});

Here's the output:

$ npx mocha strict-equality-bad.test.js


  strict equality
    ✔ 1 equals '1'
    ✔ 0 equals false
    ✔ null equals undefined


  3 passing (2ms)

This classically illustrates a surprising JS type coercion, courtesy of the loose equality operator ==. Replacing assert.equal() with assert.strictEqual() (akin to ===) gives the more desirable result — these values being unequal (some output omitted for brevity):

$ npx mocha strict-equality-good.test.js


  strict equality
    1) 1 equals '1'
    2) 0 equals false
    3) null equals undefined


  0 passing (3ms)
  3 failing

  1) strict equality
       1 equals '1':
     AssertionError: expected 1 to equal '1'
  2) strict equality
       0 equals false:
     AssertionError: expected +0 to equal false
  3) strict equality
       null equals undefined:
     AssertionError: expected null to equal undefined

The following assertions also fail, as expected:

// Jest:
expect(1).toBe("1");
expect(1).toEqual("1");

// Chai:
expect(1).to.eq("1");
expect(1).to.equal("1");

The theme here (which will be a recurring theme in this post) is that matchers don't always behave as expected.

These mistakes can be detected by experimenting with assertions to trigger failures. For example, consider the assertion:

assert.equal(add(-3, 3)).to.be(0);

Try plugging in temporary values like .to.be(-1), .to.be(undefined), and .to.be(false) to ensure the assertion fails. If they don't, the assertion is too weak and might be hiding a bug. Failures are good!

Test-driven development can help to avoid overly-permissive assertions. Tests that start out by failing before passing are more likely to function as intended than ones that pass from the outset, written against already implemented code.

Using Overly General Assertions

The next example looks at the dangers of broad matchers such as .toBeTruthy() and .toBeFalsy(), which are present in most assertion libraries. The following test has a serious bug, but passes:

describe("parser", () => {
  it("should parse `text` without throwing", () => {
    const parser = new Parser(text);
    expect(parser.parse).toBeTruthy();
  });
});

The author wants to assert that the return value of the parse() function is truthy; for example, that it returns an object. But the test author forgot to call the function, so the test is only asserting that the parse property exists!

Using .toBeDefined() and .toBeInstanceOf(Object) would fail in the same way. Variables and properties are typically defined and are often objects, so these assertions are too weak. Frequent use of loose assertions can indicate a code smell in the test or in the application under test. Functions should return values with predictable types, and assertions should be strict and specific to these types.

General assertions tend to read poorly and emit vague failure messages. Rewrite assertions like expect(meaningOfLife() === 42).toBe(true) to expect(meaningOfLife()).toBe(42).

See this Playwright question on Stack Overflow for an example of a false positive .toBeTruthy() assertion in the wild.

Using Shallow Equality for Deep Comparisons

Similarly, false positives can occur when comparing objects:

import { assert, expect } from "chai";

describe("deep equality", () => {
  test("[assert] two objects with same contents not equal", () => {
    assert.notStrictEqual({ foo: 42 }, { foo: 42 }); // incorrect, passes
    assert.notDeepEqual({ foo: 42 }, { foo: 42 }); // correct, fails
  });

  test("[expect] two objects with same contents not equal", () => {
    expect({ foo: 42 }).not.to.eq({ foo: 42 }); // incorrect, passes
    expect({ foo: 42 }).not.to.deep.eq({ foo: 42 }); // correct, fails
  });
});

The assertions labeled "incorrect" test identity rather than deep value equality.

Note that .not is a contributing factor to the problem. Avoiding .not whenever possible can improve readability, as it often does in boolean logic in application code. Positive assertions tend to match a narrower set of values and are therefore more meaningful.

Misunderstanding Assertion Behavior

I've discussed some of the gotchas when using overly broad assertions, but fine-grained assertions can hide subtle, surprising behavior as well.

Example: Playwright

Setting aside unit testing for a moment, the browser automation library Playwright offers a Jest-style matcher called .toBeHidden(). Playwright's documentation describes this matcher as follows (emphasis mine):

Ensures that Locator either does not resolve to any DOM node, or resolves to a non-visible one.

The first part of this sentence is surprising. .toBeHidden() is a broader assertion than its name suggests. If an application never renders an element, .toBeHidden() still passes, even if the author's intention was to ensure the element exists, but in a hidden state.

Example: React Testing Library

React Testing Library's getBy queries implicitly assert by throwing an error when an element isn't located, and are often used without expect.

However, it can be easy to forget that queryBy variants return null rather than throwing, and can't be used as implicit assertions like their getBy complements.

The lesson is to read the documentation for your assertions carefully. Names can be misleading. Always force assertions to fail to make sure they're working as advertised and intended.

Forgetting to Call a Matcher

Here's an embarrassing mistake I made migrating a Mocha/chai-http test to Jest/supertest.

expect(response).to.be.json functions as expected in Mocha, but becomes a no-op in Jest when naively updated to expect(response).toBe.json. The property .json is undefined, and even if it were defined, it'd be a no-op without parentheses to call the function. The same mistake can appear in assertions like expect(someValue).toBeTrue, which looks sensible from a linguistic perspective, but is do-nothing code.

Jest offers expect.assertions(number), which holds you accountable to calling a certain number of assertions in a test. This adds a bit more protection against forgetting to call matchers and asynchronous assertions that run after the test ends. Linting can also identify do-nothing expressions and unused variables.

Misusing Mocks

Writing mocks can be time-consuming so you can be tempted to cut corners. Poorly written test mocks can hide bugs in application code. Consider the following Sequelize database query:

const user = await User.findOne({ where: { id } });

If User.findOne was mocked to ignore its argument and unconditionally return {id: 1, name: "Alan"}, the test would still pass, even if the function call argument was incorrect:

import User from "./user";

jest.mock("./user", () => ({ findOne: jest.fn() }));

describe("User", () => {
  it("should create and retrieve a user", async () => {
    User.findOne.mockResolvedValueOnce({
      id: 1,
      name: "Alan",
      createdAt: new Date(),
      updatedAt: new Date(),
    });

    // a bad call made by the code under test
    const user = await User.findOne({ nonsenseArg: 42 });

    // passes, even though the call to findOne is broken!
    expect(user).toEqual(
      expect.objectContaining({
        id: 1,
        name: "Alan",
        createdAt: expect.any(Date),
        updatedAt: expect.any(Date),
      })
    );
  });
});

A solution is to assert that a particular argument was passed to the User.findOne() mock using expect(User.findOne).toHaveBeenCalledWith({where: 1}). Although this helps to avoid a false positive, it can make refactoring and writing tests frustrating.

Similarly, in the early days of React, the Enzyme testing library supported invasive mocks of implementation details, like the setState() hook. Such mocks can easily be misused to strongarm the suite into 100% coverage, failing to test the component's actual behavior by injecting values that cause the component to behave differently than it would at run time.

Using Regex Matching Incorrectly

Testing libraries like Playwright, React Testing Library, and Jest offer many matchers which accept regexes. It's easy to forget that /Hello./g matches "Hello world" and "Hello!" because . is a regex special character, not a period, and the regex doesn't have anchors. Perhaps /^Hello\.$/ was intended. Using a regex when a plain string match would suffice is a testing code smell.

Even when using plain strings, inadvertently using substring matches when exact matches were intended can cause tests to pass when they shouldn't. Luckily, modern UI testing frameworks like Playwright and React Testing Library offer strict assertions and queries by default, failing with a clear error if multiple elements match an overly broad query.

Copy-Paste Errors

Testing mistakes often come down to silly copy-paste errors and typos. These are particularly common in test suites with long chains of similar assertions.

For example, when testing a form with a few buttons, a tester might create the following suite:

import { expect, test } from "@playwright/test";

// simple page for demonstration
const html = `<!DOCTYPE html><html><body>
<form>
  <input type="submit" value="Submit">
  <input type="reset" value="Reset">
  <button type="button">Cancel</button>
</form>
</body></html>`;

test.describe("form", () => {
  test.beforeEach(({ page }) => page.setContent(html));

  test("submit button exists", async ({ page }) => {
    await expect(page.getByRole("button", { name: "Submit" })).toBeVisible();
  });

  test("cancel button exists", async ({ page }) => {
    await expect(page.getByRole("button", { name: "Cancel" })).toBeVisible();
  });

  test("reset button exists", async ({ page }) => {
    await expect(page.getByRole("button", { name: "Submit" })).toBeVisible();
  });
});

Here, the author forgot to adjust the content of the third test block after copy-pasting the first block twice. The test name was updated, so the output misleadingly makes it seem like the 'reset' button is being tested. Removing the <input type="reset" value="Reset"> element from the page doesn't cause the test case to fail as it should.

In the era of GPT-generated testing code, a hallucination like this is easy to overlook.

On more complex tests that trigger JS code, a coverage tool can help catch these mistakes. Static analysis tools available as IDE plugins can also detect duplicate code.

Using Incorrect Properties in Configuration Objects

Configuration objects are a readable way to pass arguments to a function, making up for the fact that JS doesn't support named arguments. Modern UI testing libraries use them liberally.

However, if you're using plain JS rather than TypeScript, it's easy to make a call to a function with a typo or incorrect property name in the configuration object:

screen.getByRole("heading", { value: "Hello" }); // incorrect!

This React Testing Library query should use the name: key rather than value:. JS will silently ignore the argument, possibly retrieving an unexpected element or failing to implicitly assert the header's user-visible text.

This scenario can also creep up in global configuration files and strings in general. In addition to type checking and linting, the usual strategies for false positive avoidance described throughout this post apply.

Misusing Snapshot Tests

Snapshot tests let you diff a live UI component (for example, a React or Vue component) against a version-controlled, stringified snapshot of the component. If the strings match, the test passes.

Be careful, though: snapshot testing can lead to false positives if the reference snapshot is updated to match a broken component. Jest makes it effortless to update all failing snapshots in one command, even when some shouldn't be updated. Hastily updating snapshots without careful examination can bake false positives into tests.

Unlike traditional assertions, it's not as apparent from glancing at a large snapshot whether it's accurate or not, allowing bad snapshots to be handwaved through a code review. In some cases, the content of external snapshot files may not be examined at all.

Having a strong code review culture that examines tests as critically and thoroughly as application code is another tool to mitigate false positives.

Wrapping Up

After reading this article, I hope you'll walk away with a more critical eye on your test suites. This is far from an exhaustive list, so be on the lookout for other pitfalls, especially when you're onboarding a codebase or using an unfamiliar testing library.

The testing errors we've examined are a subclass of logic errors in general applications. Working cautiously to guard against and surface silent errors helps ensure the code you're working with stays correct.

Happy debugging!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.