Testing is an activity with a very wide scope. We should test all critical components of our project, and it goes much further than the software features.
Let's remove two misconceptions about test:
- All tests are not code (unit or functional test) and automated. Test may be a simple script started on demand or a visual check.
- Tests have different frequencies: some may be run once a day only or for each release according to the risk. The test flow follows the development flow, it starts from developer workspace (pre-commit, git hook) and ends in production so some test may be outside the continuous integration tool and not be run for each commit.
Now let's focus on the target of our tests: WHAT to test ?
Testing Input and Output
In our project, you should test all input and output of your system, user input is generally well tested, but some internal features are not. It may be hidden code provided by an external library or written in a different language. Let's analyze the code to find code-related critical features.
Here are some examples:
- Telemetry (logs, metrics and traces): do we really want a new release that doesn't write any logs ? If no, then we should probably find a way to get alerted if the telemetry brutally stop (Prometheus endpoint unavailable or empty record)
- Alert: we can raise alert to a warning level to ensure production outage will be reported. This kind of test may be done in a QA or staging environment.
- Automated process (scheduled job or database stored procedure): all critical job's execution must be tracked, any lack of the launch should raise an alert.
- Package: ensure the package contain the right content (config files for all environment) and doesn't contain sensitive data. For this kind of test the best place is in a pre-commit rule to avoid these secret were versioned.
- Release: smoke test ensure the release is valid, but we can add test to ensure retro-compatibility; for example checking the new application works with the old version of the database (if this case may happen). This test is more complex than a simple unit-test, but it was greatly simplified by containers.
- Resource's policy: ensure your service can access to all required resource (storage, database, message queue...) and read/write data. These cases are probably already present in the functional test suite.
- Security procedure: this document list all the steps to be taken in response to an incident. To ensure all the step are correct and everyone in the team understand it, you should simulate commons incident and test your process.
Code tests
Back-to-back testing
This vocabulary term is used when we run the same test launched on two different versions of the code, for example when we use git bisect - To learn more about this Git feature, read this short but efficient introduction.
Canary test
A Health check is a constant call to a production service to ensure the service is up. This kind of test must be quick, it uses specific endpoint with basic response and it cannot guarantee that all is OK (such as the database).
A canary test is a sequence of actions that simulates a real user behavior of the service. In return for its longer duration, this test guarantees that all major components are alright. We can furthermore use the metrics of this test to follow reliability and detect problems (duration, latency).
Hypothesis testing
A new kind of testing framework (like Hypothesys for Python) allow defining the type of the input instead of defined a fixed value. At run, values used in the test will be randomly generated. These tests are not repeatable, it means the same test run twice will not use the same input data.
It's a good way to find edge case, but you should use hypothesis tests in addition to classic unit tests to ensure nominal and specific data are well tested. Furthermore, for some complex and specific cases, this kind of test may rise some false-negative.
Fault seeding
Deliberately adding a new bug in the code and monitor the detection by the test suite. This can be done with a Mutation testing tool which modify your code (for example the value in a conditional statement) and run all tests to ensure this modification (called mutant) is found.
This kind of test consume a lot of resource, use it only for small and critical algorithm.
Security testing
Security is a big topic, let's review the main points:
- Consider OWASP Top 10 security issues and tools like BURG or OWASP ZAP to test your code.
- Take a look at Static Application Security Testing or Dynamic Application Security Testing tool like Arachni to find vulnerabilities.
- See OWASP SAMM to find unsecure practices beyond source code.
Privacy testing
Privacy is related to security but because this topic is often forgotten, I put it in a dedicated section.
In the case who some features are implemented to mitigate data leaks (hashing, database encryption or column level security) these features must be periodically tested. See OWASP's recommendations to protect data everywhere and add a test for each countermeasure present in the project.
Domain-specific test
Depending on your application you may add some tests for usability, localization, accessibility or performance for web application.
Architecture tests
Endurance test
Endurance test is a load test during a long time because if the load test duration is shorter than some cache or TTL specified in your system, maybe you'll miss something. Using real-life context and usage, you will increase tests quality.
Stress test
Unavailability (full or partial) of one or more resource (application server, database, LDAP...) to ensure queues or high-availability is working.
These target may be randomly disconnected like for chaos engineering or we can only add latency (partial unavailability) in some server to see application behavior and check degraded mode.
To learn more you can look at the principles of chaos engineering.
Incident-proof testing
Because ransomware or disk crash sometimes happens, you can do recoverability test to check your service can restart after incident or data corruption.
We can also mention spike test which ensure ability of the service to work after peak loads and scalability test which check the system can get more worker to deal with increasing load. This work is usually done by an orchestration tool like Kubernetes.
Resources
API
A lot of tools have an API. For example, to check monitoring, we can use CURL and call Grafana and Prometheus APIs. Remember, learning about these APIs is a great way to automate things later.
Cause-effect diagram
Writing a good test require to find the right problem you want to avoid.
When you're stuck with a production problem, an interesting methodology to find the root causes is fishbone diagram or Ishikawa diagram.
You can browse and look at the other "problem-solving" methodologies explained by Untools.
ISTQB Glossary
On this testing certification website you can find terms used in software testing. The full list of terms is no longer available.
Exploratory testing
This methodology provide good resources to find edge case for some tests. You can find some Test Heuristics Cheat Sheet to extend your test cases.
Conferences
To learn more about this fast-growing topic, I warmly recommend watching the conferences on the subject. FOSDEM has a Testing track and this subject is present in Devoxx.
Top comments (0)