The Uncreative Software Engineer's Compendium to Testing

Testing is an integral part of software engineering, specifically for all the stakeholders involved. The users get to enjoy software that is user-friendly and satisfies their needs, while the engineers enjoy building software that is error- and bug-free.

Benefits of Testing

Bug detection and prevention: developers can identify bugs, errors, and defects in the software during development and get them resolved to avoid causing user inconveniences. This also saves time, cost, and effort to fix them in production.

Quality assurance: testing ensures that the software meets specific requirements in regard to functionality, performance, security, and user satisfaction.

User experience: usability testing allows the developer to identify areas that need improvement to create an application that is user-friendly and easy to use.

Risk mitigation: security and performance testing allow the discovery of potential vulnerability points and bottlenecks that can be resolved before deployment.

Continuous integration and Continuous delivery (CI/CD): automated tests are done at each stage of development to ensure changes made do not cause a break in the application or introduce new issues, and for a faster delivery process.

Customer confidence: testing increases the trust of clients that the software will function as expected without critical errors, which leads to customer adoption and loyalty.

Regulatory compliance: the software may be subject to legal compliance; testing helps verify that the software is legally compliant, which reduces the risk of penalties and legal actions.

Maintainability and scalability: well-tested software provides room for change without introducing unexpected issues, which makes it easier to scale and maintain.

Team collaboration: different teams while building get more involved; like testers, product owners, and developers, they communicate and issue feedback, which leads to a better understanding of requirements and potential requirements.

Types of Tests

Unit Testing

As atoms are the smallest unit of matter, unit testing tests the smallest unit of software at the lowest level. This includes functions, methods, classes, components, and modules.

There are different unit-testing tools depending on the language being used. In Python, for example, there are a range of tools that you can use for this:

Unittest: a built-in unit testing framework for Python that provides test discovery, test fixtures, and various assertion methods.

Pytest: is a third-party testing framework that supports fixtures, parameterized testing, and easy test discovery while having room to add plugins to extend its functionality.

Nose2: It offers test discovery, fixture support, and plugin support. It is almost identical to Pytest but is preferred due to its features and simplicity.

Doctest: this is embedding test cases within the docstrings of your code. This module allows you to write tests directly in the docstrings and run them to ensure the code examples provided in the documentation are accurate.

Tox: is a testing tool that automates testing across multiple Python environments. Tox ensures the code works in different Python versions and environments.

Hypothesis: a property-based testing tool that generates random test data and verifies that the fields and properties hold true for various scenarios.

Mock/unittest.mock: provides tools for mocking and patching objects and functions, specifically useful for isolating components during testing.

Integration Testing

Integration testing involves testing the correlation between different units to ensure they work together as expected.

Some tools can be used for unit and integration testing, for example: Unittest, Pytest, Nose2, Mock, and Doctest.

Specific tools for integration include:

Requests: is an HTTP library in Python that permits HTTP requests and verifies the responses made. It tests for interactions with RESTful APIs or other web services.

Selenium: is used for automating web browsers and is used to test interactions across different web pages and components.

HTTPretty: allows you to mock HTTP requests and responses. It's mainly used to simulate interactions with external APIs.

Faker: a library that generates fake data that can be useful when you need data to test for various components.

Regression Testing

Regression testing is re-running tests to check that the changes made, which include additions and reductions to the software, do not introduce any new defects.

Regression testing can be performed in three different ways. The first one is re-testing the whole application or specific functionality affected by the incoming changes. The second is re-executing the test suite to confirm that the changes did not introduce a breaking change to the existing functionality. The third is comparing the current version of the software with the previous version to ensure there is no breakage in functionality.

Performance Testing

Performance testing is a test that evaluates how the software acts under a variety of conditions, for example, speed, responsiveness, and stability during peak times. Performance testing identifies bottlenecks, potential performance issues, reliability, and the ability of the system to handle certain loads.

Tools used for performance testing include:

Locust: is an open-source load testing framework that allows you to define scenarios using Python code. It simulates a large number of concurrent users and measures the performance of your web applications and APIs.

JMeter: is a Java-based performance testing tool that supports scripting in Python using plugins. JMeter is used for load testing different applications, including web services.

Pytest-benchmark: Pytest permits you to track and compare the performance of functions and code snippets over time. It identifies performance declines and improvements in your Python code.

Hey: is a fast HTTP load testing tool used to test web applications and APIs. It provides a CLI (command-line interface) and supports concurrent requests.

Apache Benchmark(ab): is a command-line tool that comes with the Apache HTTP server and is used to perform load testing.

Vegeta: is a Go-based load testing tool that tests the performance of your web services and APIs. Additionally, it supports rate limiting and provides detailed reports.

Performance testing involves testing for different conditions as shown below:

a. Load Testing

Load testing is testing the stability of an application by applying a load that is equal to or less than the expected load. For instance, if an application can handle 4300 users at a time with a response time of 2.5 seconds, load testing can be done by applying the maximum load for 4300 users or less, like 3000 users, to verify that the application responds within 2.5 seconds for all users.

b. Stress Testing

Stress testing is testing the stability of an application by applying a load that is greater than the expected load. For instance, if an application can handle 4300 users at a time with a response time of 2.5 seconds, stress testing can be done by adding a load of 500 more users and checking the response time for all the users. The goal is to check how the application behaves under stress.

c. Scalability Testing

Scalability testing tests the stability as well as the response time of an application by applying more load than expected. For instance, if an application can withstand 4300 users at a time with a response time of 2.5 seconds, scalability testing can be done by applying a load of more than 1100 users and increasing it periodically to find the point of crash.

4300 users, 2.5 seconds
5800 users, 3 seconds
6300 users, 3.5 seconds
7000 users in 4 seconds
8500 users crash: point identified during scalability testing

d. Volume Testing

Volume testing tests the stability of an application and the response time in relation to the capacity of the data being transferred to a database. Simply put, how the database handles the transfer of huge amounts of data.

e. Endurance/Soak Testing

Endurance testing is testing an application's stability and response time by applying the load continuously for longer time periods to ensure the application still works fine throughout.

Acceptance Testing

Acceptance testing is done during the final phase of testing before an application is shared with end users. It involves testing the software in relation to the requirements and, if the users are satisfied with the output, validating the functionality of the application.

Tools for acceptance testing:

Behave: is a Behavior-Driven Development (BDD) Python framework where you write acceptance tests in Gherkin, which is a human-readable language. The tests are written in regular text, which makes them easier for all stakeholders to understand. The tests are executed by the framework.

Pytest-bdd: is a Pytest plugin with BDD enabled to test using Gherkin while integrating seamlessly with Pytest.

Splinter: is a web testing library used for testing web applications through the abstraction of web drivers to interact with web pages and perform acceptance tests.

PyAutoGUI: is a library used for acceptance testing of desktop applications by programmatically controlling the mouse and keyboard to automate the interactions with the application.

Security Testing

Security testing identifies vulnerabilities and weaknesses, both internal and external, in the software application that can be exploited. The aim of this test is to protect against unauthorized access and data breaches, among other ramifications.

a. Penetration Testing
Pen testing is testing carried out by external contractors to find out the weak points of the system in terms of security. The contractors carry out operations like SQL injections, URL manipulation, Privilege Elevation, session expiry, and then provide a report to the organisation.

Tools for security testing:

Bandit: is a tool designed for Python applications to analyse your code for potential security issues like insecure use of functions, hardcoded password and much more.

Safety: is a command line utility that scans Python dependencies against PyPI(Python Package Index) security advisories database for known security vulnerabilities.

PySAP(Python Static Analysis Plugin): is an analysis tool that scans Python code to check for security issues, potential bugs and other quality problems.

PyLint: helps identify potential security vulnerabilities and coding issues that may cause security issues.

Automation

Automation in testing is the use of tools and scripts to execute test cases and compare the output with the expected results. It saves time, increases coverage and prevents monotonous tasks. Tools for this include Selenium, Pytest and Unittest.

Smoke Testing

Smoke testing is a quick test conducted to check whether the essential functionalities are working as anticipated after a new build or deployment.The test validates the build to ensure no issues exist.

Sanity Testing

Sanity testing entails testing a narrow and focused area to verify that the added functionality or bug fix has not affected the core features of the application. It is a subset of the regression test.

Exploratory Testing

Exploratory testing is an informal testing approach where testers perform tests without a pre-defined test plan. They execute tests based on their knowledge and understanding of the software.

Resources and tools that aid exploratory testing include:

Test Management Tools like TestRail help testers organise their exploratory sessions, track and document their findings.

Session-Based Test Management (SBTM) Tools like Jira to help testers structure their sessions into time-boxed sessions for easier reporting and communication of reports.

Browser Developer Tools like Chrome DevTools allow testers to inspect elements, monitor network activity and debug JavaScript during exploratory testing of web applications.

Fiddler: is a web debugging tool that inspects web traffic to analyse API calls.

Non-destructive Testing

Non-destructive Testing concerns methods that permit you to inspect and verify behaviour of the code without altering its state or affecting the applications functionality.

Resources and tools for non-destructive testing include:

PyTest Snapshot Pluginpermits visual inspection of outputs without modifying actual implementation.

Code Coverage Analysis assess the code portions tested by the current test suites without altering the code.

Monkeypatching with pytestprovides a temporal replacement of functions without altering the original implementation.

Code review and inspection: where team members review code to identify issues and improve the code quality without altering the codebase.

System Integration Testing

System integration is joining individual components into a working system. System integration testing is the verification of the interactions between different components of an application to ensure they work together as expected and that data flows smoothly between them.

Continuous Testing

Continuous testing is executing tests automatically and as frequent as changes are being made to the software. This is also a part of continuous integration and continuous delivery.
The goal is to identify issues early, ensure quality and provide fast feedback.

Some common tools for continuous testing include:

Testinfra: is a testing framework for infrastructure used to test system configurations and infrastructure as code.

GitHub Actions / GitLab CI / Jenkins: used to automate continuous testing workflow and can be configured to run scripts for every code push or pull request.

CirlceCI: supports continuous testing for Python projects

Travis CI: integrated with Github to be used for continuous testing for Python projects.

API Testing

API(Application Programming Interfaces) entails testing the endpoints to verify that they are functional, reliable and secure.

Tools used for API testing include:

Dredd: used to test APIs based on the API blueprint or OpenAPI specification, to ensure implementation matches the specification.

Postman: provides a user-friendly interface for managing, creating and testing APIs.

TestFLO for Jira: is a Jira plugin that assists in executing API tests as part of the testing process

Schemathesis: executes API tests based on OpenAPI/Swagger specifications.

Agile Testing

Agile testing is a testing approach that aligns with the agile methodology which is iterative, incremental, and adaptive to changing requirements of the application due to feedback from the users. The tests are mainly carried out by the engineers involved. Popular tools for this include Jira, Trello, and Confluence.

Gray Box Testing

Gray box testing is a blending of black box and white box testing. In this case, the tester has some knowledge of the internal under-workings of the application. The incomplete knowledge allows testers to create tests that target specific areas while considering the overall system behaviour.

Data Driven Testing

Data driven testing is where the test data is stored in a separate database. This will allow the tester to validate multiple test cases using different data sets.

Tools for data driven tests include:

DDT (Data-Driven Tests): DDT is a library that allows you to use the @ddt.data decorator to define test cases and use the data sets they provide.

JSON / YAML / CSV files: Python has built-in support for the file formats to easily store and retrieve data for the data-driven test scenarios.

Paramiko: entails testing SSH connections in Python.

Model-based Testing

Model-based testing is where a formal model of the systems behaviour is used to generate test cases systematically.

Tools and resources to help in model testing include:

PyModel: is a Python library that creates state models and generates test cases based on the models.

Specmate: is an open-source model-based testing tool that supports web based applications to create models and provide test cases

GraphWalker: is an open-source tool that allows creation of models in graph format and generates tests based on the graphs.

Basis Path Testing

Basis Path testing is a technique that tests the control flow paths through a program. The tests need to be carried out at least once. Test cases are created based on the cyclomatic complexity, that is calculated from the number of linearly independent paths in the control flow path.

Conclusion

Throughout this article, we have explored various testing methodologies in software engineering and the tools available to support each type of testing. We learned that testing is an essential and ongoing process in software development, as changes and updates to software can introduce new defects. Continuous testing is crucial to ensure that software remains reliable and meets the needs of users.

To enhance your understanding of testing and stay up-to-date with emerging trends in the software testing domain, I recommend exploring the following resources:

Atlassian

Testing Help

Testing types

Testing Maturity

Software Testing List

Hey Athena

Maximizing performance