The 6 Principles of Test Automation

The word ‘test’ originally referred to “a small vessel used in assaying precious metals”. This meant that testing was a method of ascertaining the quality of gold or silver. It was also used in the process of refining valuable alloys, such as tin.

Later, the term was adopted in other fields, and these days it’s common to find it in contexts such as education, medicine, or software development. Its essence, however, has not changed: testing is used to refine end value.

We use tests in software development to ensure that code works as expected. Tests can be manual or automated. Manual testing is similar to automobile manufacturers crashing cars to verify that they’re safe for the road. It works, but it’s far too expensive to do frequently, so it is typically done at the end of the production cycle. The trouble with this method is that problems found at this stage can delay a product’s launch by months.

Automated software testing has an entirely different cost structure. There’s an initial inversion plus periodical maintenance, but once automation is in place, we can run our tests as often as we need—for pennies.

With automation, developers get continuous feedback, allowing them to spot problems very early in the production cycle. Quick iteration results in improved design, better quality, and safer launches.

Principles of test automation

Entire books have been written exclusively on the topic of test automation. It’s a skill every developer needs to master at some point, and it’s better to do it sooner rather than later.

Here are six principles to ease the learning curve:

1. Tests should improve quality.
2. Tests should reduce the risk of introducing failures.
3. Testing helps to understand the code.
4. Tests must be easy to write.
5. A test suite must be easy to run.
6. A test suite should need minimal maintenance.

Principle 1: test automation improves quality

Quality is an elusive concept. Try as we might, it’s impossible to define it numerically. Yet, we know it when we see it. The software industry has come up with many metrics to measure quality: number of defects, code coverage, CI error rate, test failure rate, and so on. Each one captures some aspect of the idea of quality.

Automated tests improve quality metrics by continually running hundreds or thousands of tests; finding defects before they reach production, informing developers of potential problems, and checking if the system deviates from user expectations.

Test reports in Semaphore shows a high-level view of a project’s state

Metrics aside, we know that a solid design is a prerequisite for quality. When tests drive development, developers can easily try out different ideas and determine which one works best. This characteristic has been exploited to great success by practices such as Test-Driven Development (TDD) and Behavior-Driven Development (BDD).

Principle 2: test automation reduces risk

Code review and peer programming, albeit necessary and productive, cannot be relied upon to find bugs. Experience shows that more eyeballs do not translate to fewer errors.

The only way to reliably find errors is to build a comprehensive automated test suite. Tests can check the whole application from top to bottom. They catch errors before they can do any harm, find regressions, and run the application on various devices and environments at a scale that is otherwise prohibitively expensive to attempt manually.

Even if everyone on the team was an exceptionally clever developer that somehow never made a mistake, third-party dependencies can still introduce errors and pose risks. Automated tests can scan every line of code in the project for errors and security issues.

Trivy scanning a project for security issues.

Principle 3: tests help you understand the system

Too frequently, developers return to code written only a few days ago only to realize they have completely forgotten how it works. This is even worse when developers have to deal with code written by other people.

Often, reading tests is the best place to understand a system, as they show how things work by example. So, when in doubt, developers can refer to the test suite.

Tests, for instance, can show another developer how an API should respond, allowing them to skip looking at the documentation.

ctx := context.Background()
result, _, err := env.Client.Server.Create(ctx, ServerCreateOpts{
    Name:       "test",
    ServerType: &ServerType{ID: 1},
    Image:      &Image{ID: 2},
    SSHKeys: []*SSHKey{
        {ID: 1},
        {ID: 2},
    },
})

if err != nil {
    t.Fatalf("Server.Create failed: %s", err)
}
if result.Server == nil {
    t.Fatal("no server")
}
if result.Server.ID != 1 {
    t.Errorf("unexpected server ID: %v", result.Server.ID)
}
if result.RootPassword != "" {
    t.Errorf("expected no root password, got: %v", result.RootPassword)
}
if len(result.NextActions) != 1 || result.NextActions[0].ID != 2 {
    t.Errorf("unexpected next actions: %v", result.NextActions)
}

Not sure if a line of code is necessary? Comment it out to see which test fails. Have an idea to improve a function? Need to refactor a piece of code? Try it out and run the automated tests. You’ll be surprised how much you can learn about a system from its tests.

Principle 4: automated tests should be easy to write

Some tests start their lives as manual tests and get automated down the road. But, more often than not, this results in overcomplicated, slow, and awkward tests. The best results come when tests and code have a certain synergy. The act of writing a test nudges developers to produce more modular code, which in turn makes tests simpler and more granular.

Test simplicity is important because it’s not practical to write tests for tests. Code should also be straightforward to read and write. Otherwise, we risk introducing failures with the test themselves, leading to false positives and flakiness.

Many testing frameworks use Domain Specific Languages (DSLs) to define tests in plain English. Perhaps the most notable example is Gherkin, the language used by the Cucumber testing framework:

Feature: Is it Friday yet?
  Everybody wants to know when it's Friday

  Scenario: Sunday isn't Friday
    Given today is Sunday
    When I ask whether it's Friday yet
    Then I should be told "Nope"

To sum up, it’s a good idea to stick to a few fundamentals when writing tests:

• Write only one assertion per test.
• Keep code separate from tests, i.e. production code should not include tests.
• Keep tests independent from each other, as dependencies can quickly snowball into a headache-inducing mess.
• Keep test overlap to the minimum, i.e. don’t test the same code twice.
• Do not break the encapsulation of the tested code. Intead, only test external interfaces.

Principle 5: tests should be easy to run

If a developer needs to open a checklist in order to start a test run, your tests won't be run as often as they should be.

Ideally, tests would run every time code changes without any intervention. We’re in luck here, as developer tools are quite sophisticated. Most modern IDEs can detect changes in files and start the test suite automatically, and the same can be achieved with command-line programs like nodemon, live reload, fswatch, or testmon.

VS Code running tests in the background

For tests to be easy to run, some conditions must be fulfilled:

• Idempotency: tests should not have side effects. Side effects include writing to files, saving to a database, or generally changing data. Developers should be able to safely run the same tests any number of times.
• Deterministic: tests should always give the same result given the same inputs. When tests need external data that is out of the developer’s control, such as the date/time or a response from an API, these should be faked with mocks or stubs.
• Independent: tests should be independent of each other, and developers must be able to run them in any order.
• Lightweight: tests must be lightweight enough to run on the developer’s machine in a reasonable time.
• Granular: developers must be able to run the test suite piecemeal.

Running tests on the developer’s machine is just part of the equation. Testing must also take place within your continuous integration pipeline. Your CI/CD pipeline acts as a quality gate; it runs the test suite on each commit, giving instant feedback and allowing developers to detect when a failure has been introduced.

Principle 6: an automated test suite should require low maintenance

The last principle is a corollary of the previous five. That is, you get it for free if you fulfill the others well. Still, it is important, so it’s good to make a point of it.

Developers want to do creative and rewarding work. Automatization lets machines take care of the drudgery of testing. A positive feedback loop is created when tests are easy to write and are executed frequently. Developers tend to appreciate how automation makes their lives easier and, thus, are incentivized to write and maintain tests.

Some periodic maintenance will, of course, be needed to keep your tests in good shape. Here are four recommendations for writing and maintaining your test suite:

• Write just enough tests to be effective (but not more). If errors are slipping by, you need more tests. Conversely, if you find that tests break with small changes, you need to remove some tests.
• Choose the best type of test for the situation. Unit tests are fast and laser-focused, while end-to-end tests cover the UI and are heavy and more comprehensive. A test suite that follows the test pyramid has a healthy variety of tests.
• Keep tests reliable. A test that fails when the code is correct is called a false positive. Tests that sometimes fail for no apparent reason are called flaky tests. Both cause problems in a test suite because they’re huge time-wasters and sources of frustration.
• Keep tests fast. A slow test suite will put the brakes on development.

Conclusion

Those who think testing is expensive are not fully aware of the cost of poor quality. Individually, the impact of bugs and defects on product value may be hard to measure, but can quickly spiral out of control if they are not addressed. Luckily, you can prevent this by building and refining your automated test suite, to serve as the foundation for a great developer experience and outstanding, quality software.