Writing Unit Tests For A Rewrite: A Case Study

This blog post is the first post in a series that discusses my efforts to convert Raspi IO to TypeScript and modernize its architecture. This blog post series will explore how to write unit tests specifically for rearchitecting or rewriting a project, how to create TypeScript base classes and functionality that is shared across multiple TypeScript and non-TypeScript projects, and how to convert an existing code base to TypeScript all in one go.

This post was first published to the Azure Medium publication.

All codebases age and mature over time. With age brings stability, and older projects are typically more reliable as a result.

However, age also brings with it creaks and groans as the original architecture struggles to keep up with modern user needs. Time also brings newer, better ways of developing these projects, and what was once cutting edge often becomes clunky and slow.

So the question for these projects becomes: to rewrite, or not to rewrite? I faced such a question with my oldest project still under development: Raspi IO.

Raspi IO is a plugin for the Johnny-Five Node.js robotics and IoT framework that enables Johnny-Five to run on the Raspberry Pi. I first created it as a monolith in 2014, but the original architecture quickly ran into limitations as I added more features. I rewrote the library the following year and split it into multiple modules. This rewrite allowed the project to scale as more features were added.

Raspi IO currently consists of 11 modules. Nine of these modules comprise what I call Raspi.js, which can be used independently of Raspi IO and Johnny-Five. These modules together provide a complete API for interacting with hardware on the Raspberry Pi in a uniform way. Raspi IO and Raspi IO Core together provide a translation layer from Raspi.js to the IO Plugin spec for Johnny-Five. Raspi IO Core is platform independent, and Raspi IO injects Raspi.js into Raspi IO Core to create a Raspberry Pi specific IO plugin.

Over time, all of Raspi.js has been converted to TypeScript and updated to modern coding practices. Raspi IO and Raspi IO Core, however, have remained more or less unchanged for three years. This is fine for Raspi IO, which only contains 32 lines of code, but not for Raspi IO Core. Inside, there are 1000 lines of dense JavaScript, replete with hacks for strange edge cases and bugs. This codebase definitely falls under the classic case of “afraid to make changes because it might break everything.” It’s also in dire need of updating to TypeScript and modern coding conventions.

With the need clear in my head, I sat down and devised a plan to rewrite Raspi IO Core without breaking it for my users. The first step in this rewrite was to implement unit tests with a high degree of code coverage, as Raspi IO Core did not have unit tests for historical reasons (unit tests involving hardware are tough).

While major refactors and rewrites bring a lot of advantages to them, such as state-of-the-art best practices and modern tooling, they are inherently risky from the standpoint of breaking your users. Unit tests act as insurance to make sure that the rewrite is as transparent to users as possible.

Methodology

So how does one implement unit tests for a project that has no unit tests and needs to be rewritten? Very methodically, and following a specification.

As previously mentioned, Raspi IO Core implements a published specification called the IO Plugin Spec. This spec provides a blueprint for how the module is supposed to behave, and in effect provides a blueprint for the unit tests themselves.

Not all projects implement an API spec, but hopefully there are design documents or other documentation describing what the project is supposed to do. If not, then the first step in implementing unit tests is to write such a spec. It’s a lot of work, but I promise it will help tremendously down the road. In addition to making it easier to implement unit tests, it provides a place for all stakeholders, not just coders, to provide input on the project and make it better. If you’re unsure where to start, Read the Docs has good content on writing quality specifications.

Next up was to decide on a unit testing tech stack. I decided to go with a common stack for open source Node.js modules because I’m already familiar with them, and didn’t want to learn new tools or platforms at this time:

• Jasmine: a Behavior Driven Development (BDD) test framework.
• Istanbul: a JavaScript code coverage tool. Code coverage tools measure how much of your codebase is executed by your unit tests, and provides a useful proxy measure of how much of your code is tested by unit tests.
• Travis CI: a hosted unit testing platform that makes it easy to run unit tests on GitHub activity (e.g. when a PR is submitted, when pushing/merging to master, etc). Although not strictly required for the rewrite, it’s generally a good idea to wire up unit tests to a hosted platform such as Travis CI. This allows developers who are considering using your library to see unit test results without having to download your code and run tests themselves.
• Coveralls: a hosted code coverage platform that integrates with Travis CI, and provides all the value that Travis CI does, except for code coverage instead of unit tests themselves.

With the specification and unit testing infrastructure in place, it was time to write my unit tests!

Walkthrough of a Unit Test

To illustrate how to write an effective unit test, I’m going to do a deep-dive walkthrough for one part of the IO spec: the digitalRead method. The IO Plugin spec has this to say about the digitalRead method:

digitalRead(pin, handler)

• Initiate a new data reading process for pin
• The recommended new data reading frequency is greater than or equal to 200Hz. Read cycles may reduce to 50Hz per platform capability, but no less.
• Invoke handler for all new data reads in which the data has changed from the previous data, with a single argument which is the present value read from the pin.
• A corresponding digital-read-${pin} event is created and emitted for all new data reads in which the data has changed from the previous data, with a single argument which is the present value read from the pin (This can be used to invoke handler).

We can break the things this spec says we must do down into a few different things we need to test, which will become our set of unit tests. Reading through the spec, I identified the following five tests:

• The third bullet point indicates we need to test reading a value via the handler argument as the pin value changes over time.
• The fourth bullet point indicates we need to test reading a value via the digital-read-${pin} event as the pin value changes over time.
• The second bullet point indicates we need to test that handler is called at 50hz or faster.
• The third and fourth bullet points indicate we need to test that the method doesn’t report the same value twice in a row.
• Implicit in this and other parts of the spec is that we need to test that digitalRead continues to read even when the mode is changed to output mode and reports the output value that was set via digitalWrite.

Now that we’ve identified five unit tests we want to write, the next step is to figure out how to write them. At the end of the day, unit tests exist to confirm that the correct outputs are generated given a reasonably complete sampling of inputs. So the first step in any unit test is to identify the inputs and outputs.

We tend to think of inputs and outputs as the arguments we pass to functions, and the values they return. These are not the only inputs that exist though. For example, if we’re testing a function that saves a value to the database, then the call to the database is also an output, in addition to what the function returns or the callback it calls. In the case of digitalRead, we’re calling other modules that talk to hardware (more outputs and inputs!). In general, it’s quite common for there to be two or more sets of inputs and outputs.

The trick in unit testing is to figure out how to measure the inputs and outputs on the “back end” of the diagram below. Most often, this is done using mocking, and is the solution I chose to use here. The architecture of Raspi IO Core makes this pretty straightforward to do because we can pass in mocked versions of all the modules in Raspi.js. The full set of inputs and outputs we’re testing are shown below:

These mocked versions include a virtual implementation of hardware, and expose the inputs/outputs to this module such that we can verify them in our unit tests. For this unit test, we use the DigitalInput mock, which has the following code:

class DigitalInput extends Peripheral {
  constructor(...args) {
    super([ 0 ]);
    this.value = OFF;
    this.args = args;
  }
  read() {
    return this.value;
  }
  setMockedValue(value) {
    this.value = value;
  }
}

We’ve added an extra method called setMockedValue that doesn’t exist in the real Raspi GPIO DigitalInput class. This allows us to precisely control what Raspi IO Core will be reading. We also add a new property called args that we can use to see what parameters were passed to the class constructor. With this in place, we can measure all of the inputs and outputs to the “back end” of the black box we’re testing.

Now it’s time for the unit tests themselves. We’re going to take a look at a single unit test that tests using the callback to read the value:

it('can read from a pin using the `digitalRead` method',
    (done) => createInstance((raspi) =>
{
  const pin = raspi.normalize(pinAlias);
  raspi.pinMode(pinAlias, raspi.MODES.INPUT);
  const { peripheral } = raspi.getInternalPinInstances()[pin];

  let numReadsRemaining = NUM_DIGITAL_READS;
  let value = 0;
  peripheral.setMockedValue(value);
  raspi.digitalRead(pinAlias, (newValue) => {
    expect(value).toEqual(newValue);
    if (!(--numReadsRemaining)) {
      done();
      return;
    }
    value = value === 1 ? 0 : 1;
    peripheral.setMockedValue(value);
  });
}));

We start with some initialization code to get a test pin ready to read. We then call getInternalPinInstances, which is a special hook method that’s only exposed when we’re running unit tests. This returns the mocked instance of DigitalInput so we can access the hooks in DigitalInput we discussed above.

Then, we set up some state monitoring variables. Since this method is supposed to read data continuously, we must test that it can read more than once. numReadsRemaining tracks how many reads we’ve performed and how many we have left to go. We toggle the value each callback since it won’t call the callback if the value doesn’t change. In each callback, we test that the value that Raspi IO Core reports is the same value that we set in the mocked DigitalInput class.

And with that, the unit test is complete! If you’d like to see all of the unit tests that comprise the DigitalInput tests, you can find them on GitHub.

Lessons Learned

Throughout this process, I’ve learned several important lessons about unit tests and rewrites.

Edge cases are more important than common cases.

We test our common cases a lot, and our code is written with these common cases in mind. Edge cases, more often than not, are found through trial and error, or user reports. As such, when we’re rewriting an existing codebase, we want to make sure that we port the edge cases over as they’re much less likely to be fixed “out of the gate.” Getting unit tests to test these edge cases is the most effective way to ensure we get these edge cases included in the rewrite.

Always be specific, not general

When writing unit tests, it’s easy to write something quick that more or less tests what we want. For example, if we’re testing whether or not a function throws an exception when it’s given an incorrect parameter, we could write something like this:

expect(() => {
  add(NaN, `I'm not a number`);
}.toThrow();

This will indeed pass, but how do we know it passed because the add method correctly detected that we tried to add two non-numbers? What if there was a legitimate bug in the code that coincidentally threw on the same inputs? We should instead write this test as:

expect(() => {
  add(NaN, `I'm not a number`);
}.toThrow(new Error(`non-numbers passed as arguments to "add"`);

This way, we can ensure that it’s throwing the way we expect. This also helps us to prevent typos if we aren’t copy-pasting the error message over. This may not seem like a big deal, but sometimes user’s code depends on the content of the error message because they need to make a decision based on which error is thrown. If we change our error message, we break this code. For an in-depth discussion of why error messages are important (and tricky), I recommend reading how the Node.js project itself is changing how it does error handling.

Good code coverage is more important for rewrites than it is for day-to-day development.

In an ideal world we’d all have 100% code coverage. In practice, however, 100% code coverage is rarely ideal, and sometimes impossible. Indeed, Raspi IO Core sits at 93% coverage because most of the code not being testing is dead code. Most of this dead code is runtime code introduced by Babel itself, which is admittedly an outdated version. The rest is code that I thought was necessary but is most likely dead code in practice. There are also cases where some code is so tightly bound to something not present during testing (like, say, an external sensor), that mocking everything necessary would lead to a unit test that really is only testing the mocks, not the code itself.

It’s expected not to have 100% code coverage, but it’s more important to have high code coverage for a rewrite than for day-to-day coding. This is because of statistics. During a rewrite, we are changing vast swaths of our code which end up being covered by large numbers of unit tests, and thus large numbers of edge cases. Day to day coding rarely has such far reaching changes though. As such, the chance of regressions is higher during a rewrite. Having high code coverage is the most effective way at preventing regressions in general, and so high code coverage is especially important when we’re dealing with changes that have high risk of regressions, such as a rewrite.

Writing unit tests against a spec also improves the spec

As much as we want to view specifications as infallible, they’re created by humans. And just like humans who create code, humans who create specifications sometimes make mistakes and introduce bugs in the spec. Writing unit tests against a spec will often highlight areas of the spec that are ambiguous or contain errors. In creating the unit tests for Raspi IO Core, I uncovered multiple issues with the spec. In three of the cases, we simply forgot to update the spec with some new features that were added. In two other cases, the spec was ambiguous. Going through the process of writing unit tests can be a surprisingly effective way at sussing out issues in the specification.

Conclusion

I’ve attempted to convert Raspi IO Core to TypeScript probably 4 or 5 times in the past. Each previous attempt failed because I quickly became uncertain that I could provide a painless upgrade path for my users. Without unit tests, I was not confident in my changes. Writing these unit tests was the key missing ingredient in these previous attempts, and now I’m set to move forward with converting Raspi IO Core to TypeScript, and re-architecting major parts of it in the process.

This effort has really reiterated the importance of unit tests, as well as the importance of understanding what we test, how we test it, and why.