An exploration of the four major categories of software testing
Spoiler
There are hundreds of different categories of software tests, such as performance tests, functional tests, visual tests, usability tests, and a bunch of others. But out of all these categories, there are four that keep showing up in most projects. They are:
- Static tests
- Unit tests
- Integration tests
- End-to-end tests
In this article, we'll go over each of these four most popular categories, so you'll be able to understand their differences and decide which ones you should use in your project.
Spoiler alert: you should be using all of them.
Introduction - What is Software Testing
You can follow the code examples in this article by cloning this public Github repository.
First things first: What is software testing?
Software testing is the process of evaluating software to make sure it avoids regressions and doesn't introduce new bugs. In other words, it's just making sure the software does what it's supposed to do.
There are many ways of doing that. If you want to know all of them, you can check out the references at the end. In this article, we'll explore the four most popular options:
- Static tests
- Unit tests
- Integration tests
- End-to-end tests
Static Tests
A static test means that we are testing code without executing it. That's why it's called static.
We can use this to catch typos, type errors, and a lot of other nits. Here are some examples of static tests:
Linting
Linting is the process of checking your source code to enforce stylistic conventions and safety measures.
This is done using a lint tool, also known as linter. Linters are available for most languages. Some renowned linters are ESLint, CSSLint, and Pylint.
The following is an example of how a rule from ESlint would work.
var someFunction = () => {
//=> 🚨 ESLINT (no-var): Unexpected var, use let or const instead
console.log('someFunction');
};
In this example, we're using the ESLint no-var rule to enforce the usage of let and const statements instead of var statements. So ESLint is statically analyzing our code without executing it to ensure we're not using var. But since we are using var, ESLint is yelling at us.
This is just one simple example of a linting check, but linters are way more powerful than that. They can check a thousand different rules, and you can also write your own custom rules, if necessary.
You can even have auto-fix for some linting rules, such as automatically replacing var statements with let or const statements.
Another exciting thing that you can do is create a base linter configuration, and then you can reuse that configuration in all your projects. Many companies do that. Facebook, Airbnb, and others have created their own style conventions and linting configurations that enforce those style conventions. Then they reuse those linting rules for their projects.
Type checking
In programming languages, we have strong and weak type systems. When the type system is strong, the compiler warns you in the event of typos and errors. But when the type system is weak, like in JavaScript, some mistakes are hard to detect.
An example of a strong type language is TypeScript. In TypeScript, if you have a function that expects numbers as its arguments, TypeScript won't let you give it a string. It will make sure that you give it a number.
const multiply = (a: number, b: number): number => {
return a * b;
};
multiply('1', 2);
//=> ⚠️ COMPILER ERROR: Argument of type 'string' is not assignable to parameter of type 'number'.
But the same code in JavaScript won't give you any warnings because you can't set parameter types in JavaScript. Since you can't tell JavaScript that a function expects numbers, it will have no problem accepting strings.
Sometimes that will be fine, and your code will indeed work properly. In other times… things might behave unexpectedly.
const multiply = (a, b) => {
return a * b;
};
multiply('1', 2);
//=> 2
multiply('hello', 2);
//=> NaN
Static code analysis
Finally, other tools help ensure code quality by analyzing your code and providing valuable insights. Suites like SonarQube, PhpMetrics, or SpotBugs, can provide metrics such as vulnerability reports, testing coverage reports, and feedback about technical debt. These checks are called static code analysis.
Besides type-checking, TypeScript is also able to do some static code analysis. For example, it can detect misuse of the delete operator.
const x = 1;
delete x;
//=> ⚠️ COMPILER ERROR: The operand of a 'delete' operator must be a property reference
Here, TypeScript warns us not to use the delete operator on a variable.
This is different than type-checking. In this case, TypeScript sees that you're making a mistake, but this mistake is not based on a type-check. It's based on how the delete operator was meant to be used. That's why this example fits into the static analysis category, not the type-checking category.
Other static tests
Besides the examples above, there are many other types of static tests, such as:
- Informal Reviews
- Walkthroughs
- Technical Reviews
- Inspections
The thing is, all of these practices involve analyzing your code without executing it.
Unit Tests
Unit tests involve testing the smallest units of your code. A "unit" is usually just a function in functional programming or a class in object-oriented programming, but it can be more than that. At the end of the day, you decide what a "unit" means in the context of your project.
Let’s see a simple example of a unit test.
export const sum = (a, b) => {
return a + b;
};
Given a function called sum, that takes two numbers and returns their sum. We could write some unit tests, for example:
- Calling
sum()with1and2should return3 - Calling
sum()with-3and10should return7
That's how the code would look like in a real unit test:
import { sum } from './sum.js';
describe('sum()', () => {
it('should sum two positive numbers', () => {
const actual = sum(1, 2);
const expected = 3;
expect(actual).toEqual(expected);
});
it('should sum positive and negative numbers', () => {
const actual = sum(-3, 10);
const expected = 7;
expect(actual).toEqual(expected);
});
});
Now that we have written our unit tests, we can run them using a test runner, such as Jasmine or Jest.
> npx jasmine sum.spec.js
Started
Jasmine started
sum()
✓ should sum two positive numbers
✓ should sum positive and negative numbers
2 specs, 0 failures
Finished in 0.011 seconds
Executed 2 of 2 specs SUCCESS in 0.011 sec.
Integration Tests
While unit testing is about testing an individual unit in isolation. Integration testing is about testing that a combination of units can work together.
To illustrate, let’s say that we have a function called showUserName that returns the name of a user. But this function uses another function called findUserById to actually find this user.
const USERS = [
{ id: 1, name: 'John' },
{ id: 2, name: 'Jane' },
{ id: 3, name: 'Joe' }
];
export const findUserById = (id) => {
return USERS.find((user) => user.id === id);
};
export const showUserName = (userId) => {
const userName = findUserById(userId).name;
return userName;
};
An integration test for these two modules would look like this:
import { showUserName } from './show-user-name.js';
describe('Integration between showUserName() and findUserById()', () => {
it('should return the correct user name', () => {
const name1 = showUserName(1);
expect(name1).toEqual('John');
const name2 = showUserName(2);
expect(name2).toEqual('Jane');
const name3 = showUserName(3);
expect(name3).toEqual('Joe');
});
});
And just like our unit test, we can run our integration test with Jasmine.
> npx jasmine user.spec.js
Started
Jasmine started
Integration between showUserName() and findUserById()
✓ should return the correct user name
1 spec, 0 failures
Finished in 0.006 seconds
Executed 1 of 1 spec SUCCESS in 0.006 sec.
This is a simple example. As I said before, you define what a "unit" means. We can get way more complicated than that. We could write integration tests that simulate HTTP requests to a server. But you get the idea, we just want to make sure that the units can work together.
End-to-End Tests
Now let's get out of our code editor and picture ourselves as real users.
A real user won't mind if we pass the wrong parameter to a function, or if we're receiving the correct data from an API call. The user is only interested in what he can actually see and interact with. And for that, we have end-to-end tests, also known as e2e.
End-to-end tests are all about testing the end-user interaction, but instead of hiring humans, we can use a tool that simulates our users.
An end-to-end test runner will run tests against your entire application using the same interface as your end-users. For example, a web application runs in the browser, so your end-to-end test runner should interact with your application using a browser, just like a real user.
Cypress is a very popular and modern e2e test runner for browsers. Let's see an example e2e test written for Cypress:
describe('End-to-end testing example', () => {
it('should have the correct title', () => {
cy.visit('/');
const fruits = ['Apple', 'Watermelon', 'Banana', 'Peach', 'Orange'];
cy.get('.title').should('contain', 'Fruits');
fruits.forEach((fruit) => {
cy.get('.fruits li').should('contain', fruit);
});
});
});
In this case, we want to test that if a user visits our fruit application, he can see the correct title of the website and the correct list of fruits.
Running this test is not as simple as running unit or integration tests because, as I said before, end-to-end tests require your application to be fully running. In our case, our application is a simple file server, and we can start it by running npm start.
> npm start
┌───────────────────────────────────────────────────┐
│ │
│ Serving! │
│ │
│ - Local: http://localhost:3000 │
│ - On Your Network: http://192.168.0.108:3000 │
│ │
│ Copied local address to clipboard! │
│ │
└───────────────────────────────────────────────────┘
Now that our application is alive, we can run our end-to-end tests against it. Talking specifically about Cypress, there are two ways of running our tests:
- Using the Cypress CLI
- Using the Cypress GUI
Running the tests with the Cypress CLI is very similar to how we've been doing things so far. You simply execute cypress run, and it will run the tests and show the results on your terminal.
> npx cypress run
====================================================================================================
(Run Starting)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 9.6.0 │
│ Browser: Electron 94 (headless) │
│ Node Version: v16.14.0 (/Users/lucas/.nvm/versions/node/v16.14.0/bin/node) │
│ Specs: 1 found (fruits.spec.js) │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: fruits.spec.js
End-to-end testing example
✓ should have the correct title (200ms)
1 passing (243ms)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 1 │
│ Passing: 1 │
│ Failing: 0 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 0 │
│ Video: true │
│ Duration: 0 seconds │
│ Spec Ran: fruits.spec.js │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
(Video)
- Started processing: Compressing to 32 CRF
- Finished processing: /Users/lucas/Downloads/type-of-tests-master/end-to-end-test (0 seconds)
s/cypress/videos/fruits.spec.js.mp4
====================================================================================================
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✔ fruits.spec.js 236ms 1 1 - - - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✔ All specs passed! 236ms 1 1 - - -
But hey, if it were a real human running this test, you'd be able to see the person opening Google Chrome and clicking around. And guess what? Cypress allows you to have a similar experience!
Running cypress open will open a Cypress window. Then you can click on the fruits.spec.ts file and see your browser running your test, controlled by Cypress.
Pretty cool, right?
It's so cool that you might think that it's a good idea to forget about the other types of tests and just write end-to-end tests. And hey, I'm not gonna lie, that can work. But…
As you may have noticed, end-to-end tests take waaaay longer than unit and integration tests. This simple test took 236ms using the Cypress CLI. Imagine hundreds of these. You will quickly get into a state where it takes an hour to run all your e2e tests.
To work around that, most end-to-end test runners allow you to throw money at the problem and run your tests in parallel. But it will never be as fast as running unit and integration tests. This means that your developers will be annoyed at how long it takes, and they will eventually stop running the tests locally.
Also, end-to-end tests require all the setup of actually running your whole application. Which is yet another barrier for your developers.
Verdict
So here's my personal verdict, and you can ignore it if you disagree:
- I think we should have strict style conventions and use static tests to enforce them. This will save hours debating whether a new project should use tabs or spaces.
- I think we should have fast unit and integration tests that run every time we modify a file. That way, we can have quick feedback if our code breaks something.
- And lastly, I think we should have end-to-end tests, particularly for the primary features of our application. No matter how many unit and integration tests I have, end-to-end tests are necessary. Personally, I would not feel confident deploying an application without running some end-to-end tests first.
Don't Stop Here
If you want to dive deeper into software testing or the technologies used in this article, such as JavaScript, TypeScript, and Cypress, consider subscribing to our newsletter. It's spam-free. We keep the emails few and valuable. If you prefer to watch our content, we also have a YouTube channel that you can subscribe to.
And if your company is looking for remote web developers, consider contacting my team and me. You can do so through email, Twitter, or Instagram.
Have a great day!
– Lucas
References
- Github repository with the code examples
- Static vs Unit vs Integration vs E2E Testing for Frontend Apps
- 20 Types of Tests Every Developer Should Know - Semaphore
- Lint Code: What Is Linting + When To Use Lint Tools | Perforce.
- Types of Software Testing: Different Testing Types with Details
- What is Unit Testing? | Autify Blog
- Integration Tests (with examples) | by Team Merlin | Government Digital Services, Singapore
- Types of Software Testing | The Complete List | Edureka
- JavaScript Static Code Analysis & Security Review Tool | SonarQube
- How to Perform End-to-End Testing
- What Is End-To-End Testing: E2E Testing Framework with Examples
- ESLint
no-varrule - Airbnb style conventions
- Facebook style conventions for JavaScript projects
Top comments (0)