This article originally appeared on IG's blog
It is funny how things turn around. For fifteen years I have been preaching TDD, or at least for developers to write some unit tests. However, in recent times I have found myself saying more often, "Why did you write that test?" instead of, "You should write a test."
What is going on?
While walking around the office, I was asked by a developer to help him with some unit tests. It seems that he had trouble using Mockito to test the following piece of code:
I think he was very surprised with my response: "You don't need to test that."
"But I have to!" he said. "How do I know then if the code works?!"
"The code is obvious. There are no conditionals, no loops, no transformations, nothing. The code is just a little bit of plain old glue code."
"But without a test, anybody can come, make a change and break the code!"
"Look, if that imaginary evil/clueless developer comes and breaks that simple code, what do you think he will do if a related unit test breaks? He will just delete it."
"But what if you had to write the test?"
"In that case, this is how I would test it:"
"But you are not using Mockito!"
"So what? Mockito is not helping you. Quite the opposite: it is getting in your way and it is not going to make the test more readable or simpler."
"But we decided to use Mockito for all the tests!"
Me: "…"
Next time that I bumped into him, he proudly stated that he had managed to write the test with Mockito. I understand the mental satisfaction of getting it working, but nonetheless it made me sad.
Another example
I got pulled in by a developer all excited about the high code coverage of one of their new applications and their new found love for BDD. Looking around the code we found the following Cucumber test:
If you have used Cucumber before, you will not be surprised about the amount of supporting code that it needs:
And all of that to test:
Yes, a simple map lookup.
I had enough trust with the developer to bluntly say, "That is a big waste of time."
"But my boss expects me to write test for all classes," he replied.
"At the expense of?"
"Expense?"
"Anyway, those tests have nothing to do with BDD."
"I know, but we decided to use Cucumber for all tests"
Me: "…"
I understand the mental satisfaction of bending the tools to your will, but nonetheless it made me sad.
Where is the tragedy?
The tragedy is that two bright developers (both of whom I would take to a team interview) are wasting time writing those kinds of tests, tests that are pointless, and that will need to be maintained by future generations of IG developers.
The tragedy is that instead of using the correct tool for the job, we decide to keep plugging away with the wrong ones, for no particular good reason.
The tragedy is that once a "good practice" becomes mainstream we seem to forget how it came to be, what its benefits are, and most importantly, what the cost of using it is.
Instead, we just mechanically apply it without too much thought, which usually means that we end up with at best mediocre results, losing most of the benefits but paying all (or even more) of the cost. In my experience writing good unit tests is hard work.
So is 100% code coverage worth pursuing?
Yes, everybody should achieve it … in one project. I am of the opinion that you have to go to the extreme to know what the limit is.
We already have plenty of experience of one extreme: projects that have 0 unit tests, so we know the pain of working on those. What we are usually lacking is the experience in the other extreme: projects where a 100% code coverage is enforced and everything is TDD.
Unit testing (especially the test first approach) is a very good practice but we should learn which tests are useful and which ones are counterproductive.
But remember nothing is free, nothing is a silver bullet. Stop and think.
Top comments (42)
We decided to use screwdrivers for everything.
Finally got that nail in. I think screwdrivers are hard to use. For our next project let's use hammers for everything. They look simpler.
Rolf. Very accurate summary :)
😄👍🏻
This reminds me of what I read from Uncle Bob and DHH (linked from Uncle Bob's article): blog.cleancoder.com/uncle-bob/2017...
Recently I've come to the conclusion that we made an Idol out of unit tests and greatly undervalue integration tests. The examples above are clearly show code that could be tested as part of the bigger picture, because in itself it doesn't mean anything: it's glue code and integration tests are supposed to check exactly that... The gluing of components together!
So the bottom line is: write a good mixture of unit and integration tests and, most of all, THINK! That's our job.
The most important thing you have pointed out is : "THINK".
There is a perfect gif to illustrate your point: 9gag.com/gag/a0pbDeX/programmers-w...
I believe that 100% test coverage is not a good idea in applications. However, I find it very very valuable in libraries. I have a few personal recommendations:
Avoid white-box testing. It makes refactoring harder. We should be able to achieve 100% test coverage using only the public API of the library.
Every reported bug should be reproduced with a unit test before it is fixed. These unit tests are more important than achieving 100% coverage.
Yaaas! Blindly following metrics and using tools without applying critical thinking of cost vs benefit is the worst! I especially hate when it causes other devs or management to get a bad taste in their mouth around quality and start pushing devs to skip things that are actually important.
I've found tools like cucumber are awesome at the acceptance test level, but the cost vs benefit breaks down very quickly as you try to apply it at lower levels. You're basically maintaining this alternate human readable text and its mapping to code in addition to your regular unit test code. But is any non-developer going to read that feature file? No. So why do it when developers can easily (and probably more easily) just read well-written unit test code??
Definitely going to refer my teams to this article. Thanks for writing it!
Thanks a lot for the feedback!
When doing TDD I think it's important to treat your tests just as importantly as you treat your code. That means also applying the KISS and DRY principles when writing your tests. When I'm writing tests for a class or something, I'll usually write a test that I can instantiate the thing first, but after I have written the tests testing what the class actually does, I'll prune off all the pointless tests that served more as progress markers during the TDD process rather than actual, meaningful tests.
That is a very mature and, in my experience, unusual practice. Congrats!
I haven't done much formal testing myself, and definitely not any unit testing. While trying to understand it a bit more, I came across a really good paper: rbcs-us.com/documents/Why-Most-Uni...
What are your thoughts?
Mr. Coplein is above my pay-grade, so I will let Uncle Bob argue with him :)
m.youtube.com/watch?v=KtHQGs3zFAM
Personally, I always say that system level test are the best and only ones that we should write, if they could be run in a few seconds, in isolation by several developers at the same time.
In my personal experience, the systems that I work with are composed of several docens of moving parts, so system level test end up being too slow, brittle and hard to debug.
I wasn't there, but I think that is a big reason why unit test were born, out of pragmatism.
As Stéphane says in the comment, you need a good mix of test, but tests are not a substitute for thinking!
I would recommend Rich Hickey talk "Hammock Driven Development" m.youtube.com/watch?v=f84n5oFoZBc (all talks by Rich Hickey are excellent!) and Uncle Bob blog blog.cleancoder.com
Thanks for the question and the pdf!
100x this!
Oh and don't forget: Collaborate with the business representative (product manager, product owner, business analyst, whatever) on your high-level test cases! You will be surprised how good both of your understanding of that tested feature will become.
A better response to "How do I know then if the code works?!" might be to say, "No amount of unit tests can guarantee that your code works."
All tests are a cost benefit trade-off. You absolutely need them, but beyond a certain point you get diminishing returns. Writing useful tests is all about value, getting the most bang for your buck.
Testing simple accessors and "glue code" typically has little to no value. However, if it's the case that the "glue code" has a lot of churn, or something about it feels fragile, that it could break, a high level integration or acceptance test can prove very useful. Even if it doesn't always pinpoint the exact "glue code" that broke, you at least have broad coverage over multiple possible points of failure.
Very true.
Most (maybe all?) things in live, not just programming, have trade-offs.
If you don't stop to think about those trade-offs, you cannot make any good decisions. In fact, you are not even making decisions.
We are paid to think, or that is what I want to believe.
Thanks,
Dan
Thank you for this article. I can now go back and remove some tests which I felt at the back of my mind were a waste of time! :-)
Having said quite a bit of learning does come about in terms of architecture (is that too big word?) when writing unit tests but only if we know what we are testing.
the problem with this is that it's subjective. as organization grows, as people of different levels and personal preferences come and go, subjectiveness in these best practices will inevitably lead to lower software quality.
The only reason not to test something is if it's hard or tooling is missing. Those cases require case-by-case analysis and thorough code review.
Hi Alexander,
Thanks a lot for the comment.
I am going to argue that “there are no conditionals, no loops, no transformation” is quite objective. This rule can be follow by any developer of any level. There is no room for interpretation.
On the other hand I find that “not to test something if it’s hard” is very subjective, as it will depend a lot on each developer’s experience what they find hard to do.
That said, I think you are right that a big chunk of what we do as developers is subject to a high degree of subjectiviness: good architecture and design, code quality, simple code, easy to understand, obvious code, engineering vs craftsmanship, dynamic vs static...
I think this is why a lot of people say that what we do is “art” and not engineering. What I think is a masterpiece, you can think is a big pile of crap, and neither of us can provide objective enough reasons to convince the other.
I think the only reason to not automate a test is if it brings no or negative value. The value of a test is he cost of writing and maintaining it vs its benefit: time saved by catching bugs. Unfortunately, as you point out, the value is very subjective.
I am curious about the “tooling is missing” statement. Do you mean UI tests?
Cheers,
Dan
Very resonating post. If one wants to be called software ENGINEER, he/she must start thinking about business tradeoffs and invest time where it's most valuable.
An article that has been really enlightening for me in the past:
blog.stevensanderson.com/2009/11/0...
Hi Raptis,
Thanks a lot for the link, it is indeed very good.
Cheers,
Dan
This is where I draw the line between experience and knowledge of developer tools. Most developer feel that they must use every tool, apply every methodology without critically thinking through the use cases.
I will share with my team.
Well, at least you found something :). Also, in a couple of months you can reflect on how useful your task was. I would love to hear about it.
There is always a tension on how big is the unit under test, how many paths the test can cover and how confident you are about your test suite.
Hard to get right, I wrote some very fuzzy and not very specific advise.
From my experience code coverage metrics are rarely defined by the development team but a management team. For every marginal increase in test coverage there is a less proportional increase in value. Sounds like this organisation has more money then sense to be pursuing 100% coverage. Apply tests (both unit & integration) where it makes most sense.
Docker and testcontainers are changing the game. Pair that with Jenkins on Cloud wth auto scale group - you get what you wanted below - it's not seconds but it's tens of seconds for single test and minutes for full test suite. Oh and it's easy to debug.
I think the real problem is this DTOMapper. Sounds like a really sad architecture/ framework. It should have never been created at first place, then no one would have idea of testing it.
I know - if you follow "industry standard" of having anemic domain model with javabeans - you quickly get that abominations - your choice.
Don't get me started with the new industry standard: annotate-all-the-things.
See youtube.com/watch?v=-6zT60l5hDc for a good presentation about it. Maybe you are familiar ;).
Cheers,
Dan