(Image credit: midjourney)
Trunk What?
Trunk Based Development (TBD) is committing small, frequent code-changes either directly to main branch or using few branches that are merged to main in less than a day. This is a way of working that is scientifically shown to be a predictor of high performing teams, with high performing teams having twice the impact on their organizations along metrics such as profitability, productivity, number of customers, customer satisfaction, and more. This is established research that's well worth paying attention to.
If you’re curious about the details of the actual science you can find it all documented in the amazing book Accelerate: The Science of Lean Software and DevOps, and in the yearly Accelerate State of DevOps Report publications that are going stronger than ever. This research has consistently shown for almost a decade that, amongst many other fascinating findings, trunk based development is a real, actual practice that is likely to cause an increase in team performance.
A few people I talk to have very bad reactions to being made aware of this research, with scoffing or actual anger at being told what ways of working are better than others. “How dare they say that with such finality”, y’know? But this science has been true for nearly a decade at this point, the signs all point the same way, and we just can’t keep having the same regressive discussions. But thankfully most of you are fine with the idea of optimizing how we to work to find ways of shortening feedback loops, and Trunk Based Development is merely one such technique to consider.
Teams that did well had fewer than three active branches at any time, their branches had very short lifetimes (less than a day) before being merged into trunk and never had “code freeze” or stabilization periods. It’s worth re-emphasizing that these results are independent of team size, organization size, or industry.
- Forsgren PhD, Nicole; Humble, Jez; Kim, Gene. Accelerate: The Science of Lean Software and DevOps
At this point I will simply conclude that Trunk Based Development is indeed worth pursuing, and that’ll be the end of my problem statement. Let's move on!
What’s the trick then?
There's no silver bullet, so here I'll step away from the sharp science-talk of the introduction because all I can offer is my angle on it. I'm just sharing to get to hear what tradeoffs other people have tried, so we can all celebrate the wonderful permutations that exist when we pursue quicker feedback cycles.
So for this article I want to capture the thinking behind my most favorite way of working: Easily and simply committing straight to main, with no branches or pull-requests no external gating getting in the way. Just code changes, into prod, faster.
If that sounds scary then of course remember that whatever we do locally, we'll always run Continuous Integration (CI) pipelines on main commits. So bad commits that would have resulted in branch-tests failing will now fail main. A broken main is definitely something to fix ASAP, but it should be a very rare event as we’ll push towards local workflows that prevent broken code from ever reaching CI.
What's in a branch?
First of all, it's important to deeply understand what safety comes from using branching and other gating mechanisms, and then reimagining them as local workflows. We’ll definitely run tests locally prior to shipping code, that's just a given. But quite possibly your context benefits from more checks to further demonstrate a change is good, and those should all be candidates for being run locally.
Are there any other advantages to branching we should consider? I think for a lot of you, you’ll suggest someone reviewing your code is actually the entire point of branching. And, well, actually Trunk Based Development can work fine with branching as long you keep fewer than 3 branches at a time and none live beyond a day.
That compromise isn't what I want to pursue in these articles though, because there is an even better nirvana just a few steps further in: Reviewing should also be a local activity. How can reviewing be done locally? By doing pair- and team-programming. But continuous reviewing will have to be a topic for another article or this one will drag on forever, so for now I’ll just say you need to sort out reviewing one way or another in order to move the needle on TBD, and today we'll explore how to push towards the ultimate TBD 😊
Oh and in case you suffer from Change Approval Boards (CAB) then note this part of the research:
Approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.
- Forsgren PhD, Nicole; Humble, Jez; Kim, Gene. Accelerate: The Science of Lean Software and DevOps
Preventing drift
Secondly, our development environments must not drift, because then code may behave differently and a change could pass on our machine but fail in production. There are many tools for locking down environments, e.g nix, pkgx, asdf, containers, etc., and they all share the common goal of being able to lock down dependencies for an environment accurately and deterministically. And that needs to be enforced in our local workflow so we don't have to rely on CI environments for correctness. All developers must have environments that are effectively identical to what runs in CI (which itself should be representative of the production environment).
CD
And thirdly, ideally all of this Trunk Based Development energy should achieve Continuous Deployment (CD), where pipelines actually deploy changes fully and entirely to production. Every change should get deployed quickly.
CD can be a difficult topic because some developers just really struggle to conceptualize how that can even work. But to truly hook into the scientifically-validated research that have identified patterns of high performing teams, we need the time it takes from code committed to code running in production to be less than an hour (and ideally deploys happen per-commit). Quite possibly feature-flags and progressive deployments and other derisking strategies are appropriate here, although this isn't the article for such solutions. But my best suggestion if CD sounds scary is to dare to be bold and deploy every change.
Green field
Finally, I must say I’ve unfortunately found it very difficult to retrofit all the above onto existing, slow, complex monsters. It's a right nightmare, actually! In my experience the best approach has been to start fresh, e.g. with a new service, and lock down all the Trunk Based Development practices from day one. That will generate learnings that may in time be applied back to legacy services.
But that was all a lot of theory, now shall we get practical? Let's next tackle the actual nuts and bolts of how to set up a Trunk Based Development-based workflow, ok?
Top comments (3)
Love how dynamic this makes the deployment process. Also that it embraces that error are OK and can be fixed. This gate keeping we have with PRs often feels like perfectionism.
I am suffering from Change Approval Boards, as you so accurately put it, and agree, most of the time the peeps are too far away from what I am doing to truly understand the list of changes.
Anyone not closely involved with the concepts behind my PR will only be able to comment on form, which is a whole different monster which fuels my imposter syndrome.
Besides the automatic test, do you also combine this with user tests? Not sure what your needs are, but for games I could imagine that this would mean we needed more in-person game testing rounds.
Could not agree more on external reviewing, it's a net-negative every time.
I started my career in gaming, and I found it a difficult space to remain agile in because every feature can impact the whole. It's so much art intersecting with tech it was difficult to preserve simplicity.
But I don't think the point of test-automation is to test everything. A lot of release-processes guard themselves behind heavy end-to-end testing, with humans meticulously going through every screen and clicking every detail before a change is brought to production. And it's the wrong response. It's like an autoimmune disease where the life-giving response is out of proportion and mistargeted. Hiding behind thicker and thicker armor weighs down the processes until the product suffocates.
I think it's very important to ambitiously imagine how a process would look if anchored around releasing code both quickly and safely. Because the research also show that speed and quality go hand in hand: High performing teams are both faster and safer than other performance-groups. What would that look like in games? If the game is long-lived and will be iterated on, then perhaps feature toggles could allow changes to be deployed instantly and then progressively rolled out? It would eliminate things like separate builds for user tests, and probably most changes are improvements that can be rolled out automatically.
I of course don't know your context and I've been out of games for a decade now so regardless my knowledge is likely out-of-date, but I hope there are many exciting ways to reimagine developments because otherwise games will miss out on some really amazing gains in job satisfaction, business impacts, and more.
And thanks for reading!
True, safeguarding is not the point here.
Feature flags and roll-outs are some safety measures we employ as well. If a bug is discovered in the rollout it can indeed be fixed quickly.
Something we found is helpful though is when the team play tests the game regularly or are playing the rollout. Not as testers running endless test scenarios but as players. The advantage is that the team understands the user perspective which improves decision making during development which in turn has an impact on development time.
Thank you for taking the time to answer in such detail!