A few months ago, we noticed that our sprint spillovers were constantly increasing in size. Upon taking a closer look, we realised that our existing workflow had an unhealthy emphasis on the number of merges into master. This emphasis was causing a false sense of development efficiency while introducing inefficiency in our delivery process.

The Problem: We left Quality Ownership to our Quality Engineer (QE)

A closer look at our agile workflow signalled that our Quality ownership was not evenly distributed.

We use GitLab's CI service for our automated pipeline, which runs on every new commit to the master branch. It runs our unit and integration tests before deploying the new build to our QA and UAT environments.

In an ideal world, this would be the perfect setup — after all, everything was ~automated~. But here comes the problem: our integration tests were not stable. With this workflow, the burden fell on our QE to investigate and resolve each integration test failure. This task became exponentially harder with multiple MRs being merged at once by different devs, as well as the existence of undocumented flaky tests. It also didn't help that at that time, our devs to QE ratio was 5:1.

This caused a bottleneck in our delivery process, which became especially visible at the start of each sprint, when we looked at the spillover from the previous sprint. Our spillover kept growing with majority of the stories stuck in the testing phase, but since the devs had "finished" all our work in the previous sprint, we continued to take up new stories.

We were now taking up new stories every sprint, but they were not being delivered to production at the same rate to create value for our users.

The "Chicken" process

Our old process went on for a while, frequently causing us to go weeks at once without a single green build. After many sprints of digging through numerous merges to debug pipeline failures, Jin Jie (the pained QE mentioned above) and Dickson (a developer who was helping our poor QE) from our team came up with the "Chicken" process. These were the rules for the new process:

1. One Merge at a time — Only one MR should be merged into master at any time, with the author of this MR being the "Chicken".
2. The "Chicken" owns the pipeline — The "Chicken" has to ensure that our CI pipeline runs to completion with a green build for their new merge. If there are any genuine integration test failures due to the new merge, open new MRs to fix them. The "Chicken" is free to open and merge as many MRs needed for a green build.
3. Ownership lies with the "Chicken", but Responsibility is shared by the team — The "Chicken" is not expected to fix all pipeline failures by themselves. If there are too many integration test failures, the "Chicken" is free to ask for help from the rest of the team.

We have been trying out this new process for a few months now, have observed some significant benefits.

Quality is now a Shared Responsibility

Essentially, the goal of this new process is to eliminate the Somebody Else's Problem effect.

"The Somebody Else's Problem (SEP) field... relies on people's natural predisposition not to see anything they don't want to, weren't expecting, or can't explain"

— Douglas Adams in Life, the Universe and Everything

The "Chicken" process helped to mitigate the SEP field in our team with the following:

1. Greater visibility on our processes — Now that the definition of "Finish" includes ensuring that the corresponding merge passes the full pipeline, we are more inclined to learn what happens after merging our MRs into the master branch.
2. More context of the codebase — Investigating the codebase to fix tests gave us the opportunity to learn more about some of the long-lost contextual knowledge behind the implementation of older components.

Rather than having a single QE shoulder the quality burden, quality ownership was now a shared responsibility.

Greater Code Standards

We also saw an improvement in our code standards:

1. Greater focus on code quality — Having to experience the pain of investigating integration test failures, we have become more mindful of whether each MR we author will break existing integration tests. The old "merge and forget" mentality was now thrown out of the window; there is greater focus on writing reliable code and tests.
2. Discovering old bugs hidden in the codebase — With more effort in investigating test failures, we have also uncovered several bugs that were not caught during the development and review processes.
3. Fewer flaky tests — We spent a huge amount of time in investigating some test failures, only to realise that they were flaky. In other words, these tests would only sometimes fail, and the failures were largely due to the choice of testing strategy rather than incorrect implementation. This has motivated the team to work together to improve the overall reliability of our tests, rather than just clicking "retry" N times.

Improved Delivery Efficiency

Before the "chicken" process, we used to see 1-2 green builds per two-week sprint. Now, we are seeing 1-2 green builds every two days. On good days, we might even get up to 3 green builds in a day! 🚀 This was all possible due to

1. Resource reallocation to remove bottleneck — Rather than one QE investigating test failures for every single merge, the whole team was now invested to do so together. This allowed us to deliver stories at a rate closer to the rate at which we pick them up.
2. Debugging is easier when the diff is smaller — With only one merge going into the master branch at once, it was easier to inspect the diff due to the new merge. This made it a lot quicker to pinpoint where the failure point was when the pipeline failed on a new merge.

Since adopting this new process, we have also made several modifications to it. One example is allowing multiple merges at once as long as there is still at least one person appointed as the "chicken". This modification helped us reap the benefits of the "chicken" process while also catering to our larger team size. We hope to continue improving on this process as our team's needs evolve.

Thank you Jin Jie and Dickson for your great efforts in improving our processes and also Joseph for providing feedback on this piece! Also thank you Jin Jie for providing the diagrams.

If you are now intrigued to work in a team like ours: we are hiring! Drop me an email @ ting_sze_ying@tech.gov.sg if you are interested to find out more 🌈