The problem with tests
Recently I had problem with personal job in the pipeline which was designed to build and run all the unit tests and integration/heavier tests after. The idea seemed to be perfect - run all the tests on every change developer does when working in feature branch.
The reality quickly verified the idea: job duration was initially around 30 minutes, slowly but steadily growing to 60 minutes. Let's wait for an hour for simple change to be tested...
The content of each feature branch are extra tests as well so their number is growing constantly. What will happen in few months?
2 hours, 3 hours of waiting?
Important thing: application under test is 100+ modules monster with more than 10 top level ones. The bigger monster, the faster it grows.
The quickfix
I came up with idea to test only area each commit is changing. So if there is a change in moduleA, unit tests related only to moduleA and its direct dependencies are run. Also, only integrations tests related to this module directly come to play. (It turned out I have to skip indirectly dependent integration tests as all of them were run even for single module changes most of the time).
The result was really good: for changes in some modules which do not have many dependencies the duration was cut to 20 minutes. Building, testing and integration - great !
The final solution which I didn't do yet but I think I should
It is important to keep valuable tests only and delete the trash tests (what is the valuable test is the big topic for separate post for sure). One should also be suspicious if number of tests is growing quickly. It is a sign the process of creating them is wrong, like some kind of aggressive and fanatic TDD. If no action is taken, they will very soon spoil development by slowing it down first and then bring it to the complete standstill. The monster will eat all your resources. But that's easy stuff: persuade fanatic TDDevelopers they should stop...
I think main pipeline should also test only commit related areas and not "everything". The fear of missing the right coverage is huge however and in reality I know, managers prefer to slow down feature delivery than to rely on smaller subset of tests. Just persuade fearful managers now...
The goal of this text is more to share experience and ideas with you than to announce ex cathedra golden rules and shooting silver bullets.
That is why I am very much interested what you think about it.
Top comments (0)