Tests are our safety nets. If we don't trust on their integrity, we will be in great danger.
TL;DR: Don't write non-deterministic tests.
Problems
Determinism
Confidence loss
Wasted time
Solutions
Test should be in full control. There should be no space for erratic behavior and degrees of freedom.
Remove all tests coupling.
Coupling: The one and only software design problem
Maxi Contieri ・ Feb 6 '21
Examples
- Fragile, Intermittent, Sporadic or Erratic tests are common in many organizations.
Nevertheless, they mine the developers trust.
We must avoid them.
Sample Code
Wrong
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import components.set.Set;
import components.set.Set1L;
public abstract class SetTest {
protected abstract Set<String> constructor();
@Test
public final void testAddEmpty() {
Set<String> s = this.constructor();
s.add("green");
s.add("blue");
assertEquals("{green. blue}", s.toString());
//This is fragile since it dependes on set sort (which is not defined)
}
}
Right
import static org.junit.Assert.assertEquals;
import org.junit.Test;
import components.set.Set;
import components.set.Set1L;
public abstract class SetTest {
protected abstract Set<String> constructor();
@Test
public final void testAddEmpty() {
Set<String> s = this.constructor();
s.add("green");
assertEquals("{green}", s.toString());
}
@Test
public final void testEntryAtSingleEntry() {
Set<String> s = this.createFromArgs("red");
Boolean x = s.contains("red");
assertEquals(true, x);
}
}
Detection
Detection can be done with test run statistics.
It is very hard to put some test in maintenance since we are removing a safety net.
More Info
%[https://softwareengineering.stackexchange.com/questions/109703/how-to-avoid-fragile-unit-tests]
Relations
Tags
Coupling
Determinism
Conclusion
Fragile tests show system coupling and not deterministic or erratic behavior.
Developers spend lots of time and effort fighting against this false positives.
Credits
Photo by Jilbert Ebrahimi on Unsplash
The amateur software engineer is always in search of magic.
Grady Booch
Software Engineering Great Quotes
Maxi Contieri ・ Dec 28 '20
This article is part of the CodeSmell Series.
How to Find the Stinky parts of your Code
Maxi Contieri ・ May 21 '21
Last update: 2021/06/12
Top comments (1)
Should tests verify the internals (private fields) of a class?
I've seen many test cases that assert every internal variable. I don't like that.
My reason is: Every time you refactor the class you have to rewrite the tests. However, tests are supposed to ensure the external behaviour of a class doesn't change unexpectedly. If the tests are rewritten, it's hard to confirm whether the external behaviour changed.