Unit Tests Do Find Bugs

(Sorry, lazily reused the same graphic above.)

This article is my "attempt" to define the benefits of proper levels of testing. I had a larger project in the past that showed clear benefits from developer-level Unit and Integration Testing.

I will update the contents here from time-to-time as I get more concrete numbers.

Script Parser and Engine

~120,000+ LINES-OF-CODE
58 BUGS found before reaching production BY TESTING
1 BUG found once the code was in production BY USERS

This means ...

1 BUG per ~2,034 LINES-OF-CODE

From December 2016 until April 2017, about five months, I wrote the 120,000+ lines of code that became a parser and engine for a "user" scripting language. This language replicated (in JavaScript) one created on a mainframe with almost three decades of fine-tuning and optimization.

While writing this code, I began writing Unit and Integration Tests. The integration tests being written were designed to tests the parts of this system against each other (for example, could I pass in a script to the parser and get an appropriate Active Syntax Tree back). I was not using Test-Driven Development (TDD).

I also kept track of the number of bugs in the original code that did not make it to production ... 58 bugs found

Once the code was released into production, there were two (2) issues and one (1) bug located by users running scripts against this code. The issues were due to incorrect specifications between what I was actually designing and what was needed.

Testing Checklist

~700 LINES-OF-CODE
5 BUGS found before reaching production BY TESTING
0 BUGS found once the code was in production BY USERS

This means ...

1 BUG per ~140 LINES-OF-CODE

This application was actually the catalyst for this article. I designed an application for developers, designed to reinforce the testing ideals we were trying to propagate to the client's teams.

Proper testing, in this case, allowed me to see a refactor of the logging methodologies that made the code easier to read and understand, as well as simpler to test.

Conclusions

This article is simply my "attempt" to define the benefits of proper testing levels. Having had a larger project in the past that showed clear benefits from developer-level Unit and Integration Testing, this seemed like a clear direction to go.

I will update the contents here from time-to-time as I get more concrete numbers.