The Importance of Testing and the Impact of a Bug in Production

In literature there are several articles that talk about how the resolution of a bug (or defect) is more expensive the later the problem is discovered.

Wanting to cite some, we find the article The Cost of Finding Bugs Later in the SDLC where reference is made to the study «IBM System Science Institute Relative Cost of Fixing Defects» which is taken up in a very critical sense by the article Everyone cites that 'bugs are 100x more expensive to fix in production' research, but the study might not even exist.

Regardless of where the scales lean, I think it is clear that being able to identify an issue as early as possible allows you to save time and money thanks to the reduction in effort needed to perform a fix.

In fact, if a problem is identified in the development phase, the development team will directly perform the fix in the work environment that they master 100% (e.g. the development server) before the software is released in subsequent environments where all the application tests are also performed (non-regression testing). Otherwise, if a problem is identified in a Testing environment or, worse, in a Production environment, it will be necessary to run the fix in all the underlying environments and then repeat all the non-regression tests to validate the quality of the intervention (i.e., make sure you have not introduced further bugs - regressions - following the fix).

In my opinion, the Typemock infographic illustrates these concepts very well. Below is an "excerpt" with a focus on what was described above:

The data reported in the infographic refer to the study of «B. Boehm and v. Basili, "software defect reduction Top 10 List", IEEE Computer» conducted in 2001 but still valid today in terms of principles and best practices.

🔗 Do you like Techelopment? Check out the site for all the details!

What to do to reduce the risk of having issues?

Testing, testing and more testing!

Testing is the secret to reducing the probability of having bugs or unexpected results in the developed software.
Testing can be integrated into different phases of the software life cycle and in different ways.

Testing phases

Testing can be integrated at different levels:

• Design: prepare the development with a testing-aware approach that includes the use of Test-Driven Development (TDD) techniques.
• Unit testing: is the test of the single functionality that a developer is implementing. The type of technology is not relevant, whether it is Front-End development (e.g. JavaScript), API (e.g. serverless functions), Back-End (e.g. Java, Node.js, PHP, ...), Database, or Desktop and Mobile applications, there must always be at least one unit test for each developed functionality. It does not necessarily have to be run manually, it can be automated with TDD techniques.
• Integration testing: each functionality is not an end in itself, it is always part of a more extensive development that includes the other features of the product (or application) that we are creating. For this reason, immediately after unit testing, it is necessary to run integration tests that verify the functioning of the individual functions once integrated with all the others.
• System testing: this is the testing phase that we could define as "the dress rehearsal" of a show. It is the moment in which our application is tested in its entirety in terms of integrations with external systems such as third-party APIs, legacy systems, distributed databases, trusted network connections and so on.
• Non-regression testing (abbreviated to NRT): perhaps the most important phase. Unlike the other phases that are performed sequentially, non-regression tests belong to a specific phase but must be performed iteratively every time changes are made to the code in any phase of the software life cycle. In fact, this type of testing aims to verify that the changes to the code introduced by new features and any fixes performed have not created malfunctions (regressions) in the parts of the software (or application) that worked correctly before the latest changes to the code. In short: what worked before must continue to work.

Testing methods (or types)

Of course, the basic approach involves manually executing tests for the following types:

• Functional testing: testing the functionality and UI of an application in terms of satisfying functional requirements (e.g. submitting a form in the Desktop version and Mobile version of the site)
• Non-functional testing or Technical testing: this type of testing focuses on the following aspects:

1. Security: evaluating the robustness of the application from the point of view of potential cyber attacks
2. Performance: measuring the performance of the application to identify potential bottlenecks or degradation of response times
3. Reliability: reliability of the application in terms of its ability to handle errors and ensure usability of the service offered.
4. Load & Stress Testing: identifying the load limits of the application to evaluate its adequacy with non-technical requirements such as number of requests per second and number of users using the application.
5. Usability: specific tests to evaluate whether the UI of an application respects the principles of usability such as easy identification of menus and focus areas of the screens.

A more advanced approach than that of performing manual testing involves the adoption of Testing Automation.

Testing Automation is a real branch of computer science that consists in the development or integration of systems capable of automating both functional and non-functional tests. Thanks to automation, human errors or errors in the evaluation of the expected result will be avoided.

Main Testing Automation Systems

Functional Testing:

1. Selenium - https://www.selenium.dev/: One of the most important and long-lived tools. One of its strengths is represented by the browser extensions that allow you to record the execution steps of a test and then re-run them automatically. ⭐ Open-source
2. Appium - https://appium.io/: This tool allows you to automate testing also for mobile (iOS, Android, Tizen). ⭐ Open-source
3. Playwright - https://playwright.dev/: open-source library created by Microsoft for web testing automation. ⭐ Open-source
4. Katalon Studio - https://katalon.com/: an integrated environment to record and run your tests. 💲Free / Payment
5. LambdaTest - https://www.lambdatest.com/: cloud-based system for various types of testing automation: Web, Mobile, UI. 💲Free (very limited) / Payment
6. Testsigma - https://testsigma.com/: integrated suite for enterprise testing. 💲Payment

Non-functional testing:

1. Apache JMeter - https://jmeter.apache.org/: ideal for stress testing and load testing ⭐ Open-source
2. OpenText LoadRunner - https://www.opentext.com/ 💲Payment
3. Tricentis Neoload - https://www.tricentis.com/ 💲Payment

Follow me #techelopment

Official site: www.techelopment.it
Medium: @techelopment
Dev.to: Techelopment
facebook: Techelopment
instagram: @techelopment
X: techelopment
telegram: @techelopment_channel
youtube: @techelopment
whatsapp: Techelopment

References

• The Cost of Finding Bugs Later in the SDLC
• Everyone cites that 'bugs are 100x more expensive to fix in production' research, but the study…
• Infographic: The Severity of Bugs, Are We Doomed?
• https://www.researchgate.net/figure/Source-B-Boehm-and-v-Basili-software-defect-reduction-Top-10-List-IEEE-Computer_fig1_228983772