DEV Community

100% test coverage is not enough...

Shahjada Talukdar on December 05, 2019

Even if your unit tests cover everything and pass at build time, your app can still be completely broken in production. 🧐🤔 In web application deve...

Read full post

Michiel Hendriks • Dec 6 '19

100% coverage is useless. 80%-90% coverage by unit tests is generally good enough, above that you are starting to waste time. Time better spend on other ways of testing, like this article mentions.

Besides these forms of tests

Unit
Integration
End-to-End

There are also other important tests which are often forgotten:

Load/stress tests: To find out how well the system behaves under high load. It builds upon integration or e2e tests, but with 2-3 times the peak volume you would normally expect. It is a test of scalability.
Endurance tests: To find out if your system keeps running for days, weeks, months. Similar to a load test, but instead of stressing the system you want to find out if the system remains stable when it processes a peak load for a day, or week. This helps you to find resource leaks, or performance degradation. You may also want to build is some load trends, like the usual sine of peak load to none, back to peak again. And maybe add a large pause in the middle of every 3rd peak load. To test 100% to 0% and back to 100%, this quite often has side effects.

Shahjada Talukdar • Dec 6 '19

Thanks for your views! 👋
Yeah, fully agree with Load tests and etc.

Definitely those are important and forgotten stuff.
This Article was mostly written from the initial ground as a first step.
I will try to continue writing more on other detailed ways.

What kind of tools you normally use for Load or Endurance testing?

Michiel Hendriks • Dec 6 '19

Which tools are available really depends on the software under test. Tools like JMeter or Gatling are good if you can produce load with rather simple sequence of HTTP calls.

Taurus is an interesting "wrapper" around a bunch of test frameworks to turn them into load tests.

The software I work on has rather complex asynchronous "conversations" which mostly happen via message queues. I was able to create an integration test in JMeter, but it was unsuitable for performance load testing as the test framework was the bottleneck. So I need to develop our own tooling. For this I am planning to use the Apache Camel, and maybe re-use parts of the Citrus Framework (we are a Java shop).

So maybe there is no tool available for your software which can produce significant load, you can often use parts of other integration testing frameworks. Most integration frameworks are not geared towards running the same tests over and over again in multiple threads. But there is a chance you can create a wrapper around this framework which will do exactly that (like Taurus has done).

Note, with load and endurance testing you do not have to fully validate everything. Shallow verification are often sufficient enough. You do not want the verification tests to slow how the test tool. Detailed validation should have been covered in the integration tests. If the tooling support is, you could do samples of detail validation (e.g. every 1000th run is verified to greater detail.)

Shahjada Talukdar • Dec 6 '19 • Edited

As you said, you are a Java shop, you mostly do API implementation and do load testing for the REST APIs only? or you also do the testing for the whole web app end-to-end?

Michiel Hendriks • Dec 6 '19

It's message oriented enterprise software for business process automation. So the load tests we do are via message queues for specific business processes. We don't really have REST APIs. It's complex data structures with somewhat long complex transactions, often various sequential asynchronous transactions which eventually produce an output in the form of a message on a different queue.

So the load testing is in the form of pumping messages on a queue, with a different process reading messages from a different queue, which is needed to send a follow message. This a few times, to complete a whole "operation". Not the easiest to load test.

Comment deleted

Nick Holmes • Dec 9 '19 • Edited

Unit Tests (100% or otherwise) are not an appropriate tool for assessing the design of your code.

Could be 100% Code Coverage, properly modularized & SOLID, and still have a blatant bug in it;

Here's some code:

public string foo(bool a, bool b)
{
    string x = "Hello World";

    if(a)
        x = null;

    if(b)
        x = x.ToUpper();

    return x;
}

and here are the tests for 100% code coverage:

Assert.AreSame("Hello World", target.foo(false, false));
Assert.AreSame(null, target.foo(true, false));
Assert.AreSame("HELLO WORLD", target.foo(true, true));

Maximilian Burszley • Dec 9 '19

"Blatant bug" or intentionally sabotaged?

Shahjada Talukdar • Dec 6 '19

That's a good point you mentioned.
If you can really do "properly modularized & follows SOLID", then Unit testing can help of course, for good. I am not against Unit Testing. I just wanted to make sure - this is not enough, not the only thing!
But do you think it's really easy to do "properly modularized & SOLID" and find out all the cases for Unit testing?
Maybe you and your team do it properly.
Actually, in reality, it does not happen always :(

Shahjada Talukdar • Dec 10 '19

I agree with your comment, Andy!
From my point of view, Having 20%-50% Unit test coverage can be enough just to make sure the app is not broken in compile/build time. I am saying 20-50 because it can vary person-to-person, company-to-company or product-to-product.

Other than that the app should have lots of Integration and E2E tests!

Shahjada Talukdar • Dec 5 '19

You are welcome!
If you like, feel free to share 👋

Shahjada Talukdar • Dec 5 '19

😀🤝

Shahjada Talukdar • Dec 10 '19

Yeah, the concept is the point here!