I'm sure many people have hit this particular wall, before the Nightly Tests run the Smokes that check the setup fail. Often we go by the mantra - if the path to the Nightly is not Green don't run it. We'll just have too many failures. Heads will explode as people try to figure out why their neighbors tests pass while theirs failed.
The world is going to burn. Ships will sink. Chaos will be everywhere.
Lately, we've begun to have second thoughts on this.
Sure some tests will fail, but there will be a lot that will pass. Those teams at least will get some insight into their tests, and know how well they are doing, but others will see red and wonder what happened. Perhaps looking back at some PR that was merged, and didn't quite do as they wanted, so their code broke that particular part of the build (yes, we still run a monolith) and anyone relying on it but everyone else was A-OK.
The Quality part of me says this is not a good thing, we want ALL of the environment to be working so we get good tests. Good results. Confidence that the code is doing what we want. Why do we want to test on something that is partly broken? That's just bad. Quality part of me is a real sticker for certain things and likes gateways to pass and work and be all happy.
The Release part of me says we're not going to release this. It's just a daily. A blip on the trend report that will show bad for some, and good for others. We want to know how badly this particular piece is going to affect people, and maybe then we see what needs to be fixed. Also, we have a release coming up and we want to get as much testing done as we can before code freeze, even on a partly failing build. Data is good and if some people can see progress great! At least one team is not holding up everyone.
I'm trying to soften up my Quality side, I've been more focused on that side of me for quite a long time so it can take awhile to accept this sort of change. Though I have decided its time to let go of that all or nothing aspect and lets get some movement going. While its tenuous at least we can look forward to the next daily run and hopefully THAT one will be good and will run true.
Top comments (0)