Acknowledgments

This article would not be possible without the valuable input and help from Bryan Kimball. Thank you Bryan.

Synopsys

Starting your software development journey is like entering a maze, where every project brings its own set of challenges and intricacies.
Navigating the maze requires a successful and efficient development process, relying in part on effective software release planning.
And, as development teams strive for timely and high-quality product releases, they deploy various branching methodologies to do so.
In this blog post, we delve into the diverse landscape of existing Git branching strategies, highlight where they might fall short, and offer you a new alternative.

Existing Git Branching Strategies

Not all software projects are created equal and often require using a variety of Git branching options.
Here are a few widely employed approaches by product development teams:

1. GitFlow

GitFlow, a popular branching model proposed by Vincent Driessen, has been a stalwart in the version control arena.
Its structured approach aims to streamline development and release processes.

2. Feature Branching

Feature Branching allows developers to isolate new features in dedicated branches, fostering a modular and organized development environment. Important note: it is not well outlined in the provided link, and we assume that the described approach adheres to our Definition of Done and has the features fully tested in isolation before assuming those to be complete.

3. GitHub Flow

GitHub Flow simplifies the development process by promoting a continuous delivery approach.
It emphasizes small, incremental changes and rapid iterations.

4. Git Feature Flow

This may be considered to be the closest relative to our new approach since this article drives toward the same ultimate goal: bug-free and on-time releases. It may also be a good intro to the different aspects of code management techniques and their problems.

Critiques And Alternatives

While these strategies offer valuable frameworks, none are without their critiques and alternatives.
Various voices on the internet have raised concerns and proposed alternative models, especially with regard to GitFlow.
Here are a few noteworthy perspectives:

• A scalable Git branching model
• The problem with Git flow
• Why Git Flow is Outdated: A Better Approach for Managing Changes

But alternatives are not flawless either, especially considering the project with dozens (if not more) of contributors and sometimes very strict release schedules.
Here are key issues depending on the alternative used:

• Difficulty in managing pollution and the exclusion of undesirable and un-tested changes in a planned release.
• Challenges with the feasibility of adjusting the release scope
• Inability to properly surface and identify changes in release scope
• Burdening of technical release coordinator and team to untangle complicated merge conflict scenarios
• Overreliance on technical competence by team members responsible for release management

Embracing Change: The FeiFlow Solution

Please note that FeiFlow is not a "one size fits all" solution. It is intended for very active projects with many contributors (a dozen or more) with a strict release schedule. Simpler projects can do just fine with less sophisticated strategies, although many projects can indirectly benefit from FeiFlow as it implies better time and work tracking.

During our development journey, we relied heavily on GitFlow, working through both its strengths and weaknesses.
However, a pivotal moment arrived when we recognized the need for a more effective alternative.
Enter FeiFlow – an approach that we consider to be a game-changer, disrupting (or shall we say improving) our established practices and taking the pain out of release management.

So with that, sit back and grab a cup of coffee as we begin to explore FeiFlow!

Terms, Abbreviations, And Jargon

To keep things clear and concise in this post, we'll be using a few handy abbreviations and terms:

• BA - Business Analyst(s), Product Owner(s), and other business stakeholders.
• QC - Quality Control team or person(s).
• PBI - Product Backlog Item - a trackable unit of work. Also includes "on-board" defects and features.
• PBI Branch - Git branch where development work is being done for respective PBI; From FeiFlow standpoint we do not differentiate between PBI and Feature branches; Both types can be short and relatively long living branches; there are a few subtle differences which will not be discussed in this article.
• PR - Pull request; when PR is raised a few automated and manual validation procedures take place: human code review, validation build, and unit tests run to ensure minimally required code quality before it gets into testing.
• PR Commit - Commit on the PBI Branch which gets merged into the Integration Branch through the pull request.
• Release Branch - From FeiFlow's standpoint is the branch where "only" the released code lives; In practice, it's usually "master" or "main".
• Integration Branch - Target branch of the majority of PRs. This branch gets deployed to a lower environment for feature and integration testing; In practice, it's usually "develop"
• RC - Release Candidate
• RC branch - Release Candidate Branch; the branch where all the planned work for upcoming releases is merged for the final round of integration testing and bug fixes. The builds from this branch are usually deployed to higher environments with production-like datasets and integrations.
• ADO - Azure DevOps; Collaboration, Planning, and CI/CD platform.

Definition Of Done

It's important that we are on the same page about the meaning of done regarding feature completion and acceptance.

Our approach has 3 distinct stages of PBI/Feature acceptance:

1. Developer(s) conduct and complete coding and testing locally.
2. QCs perform functional testing.
3. Integration testing is led by QC to ensure proper functionality through the merging of multiple features. It's worth noting that steps #2 and partially #3 can be executed simultaneously.

Both #1 and #2 must be completed for a PBI to progress into the Done state.

Submerging Into The "Pain"

Unless one has experienced it, the cure will never be appreciated.

Have a bottle of painkillers of your choice ready. Tylenol, Gin, or Scotch may work for some. And save some Champagne for later. We are submerging now...

We start by presenting a very nice write-up (see The problem section) made by Rick Lannan. We should always remember to apply the DRY principle (don't repeat yourself) in combination with the all-favorite "copy/paste", especially when you're lazy. 😉

Coordination And Tracking Challenges:

Feature Branching:
One notable challenge with Feature Branching lies in the complexity of merging isolated features.
Resolving merge conflicts might require a deep understanding of each feature's context and possible interaction with other features, and this can be a nightmare if you're the
release coordinator. Not to mention difficulties in tracking all of those outstanding features.

Feature Branching/GitFlow/Trunk-Based Development:
Since the same code presumably exists between both the RC and develop/feature branches, determining where bug fixes should be applied becomes intricate. Should testing resources
identify a bug in multiple environments, you may need to coordinate cherry-picking and/or back-merging to have the issue resolved on multiple branches. This becomes especially complicated when having to support multiple releases simultaneously.

Limited Human/Environment Resources:

Feature Branching:
Because functional testing is part of our definition of done, feature branching would require testing each feature separately. This might involve spinning up isolated environments for each feature, leading to potential costs and delays in having to queue up individual features to be tested one at a time in a single and/or limited set of environments.

Finalizing The Release Candidate:

GitFlow/Trunk-Based Development:
It's all or nothing. All of the features complete or incomplete are part of the RC. This implies that everything must be tested and made production-ready by the release deadline. If only a percentage of the features are functionally tested and verified, what then might you say to stakeholders about the remaining percentage
that has gone untested or has unresolved bugs? This leads us to the next point.

Adjusting Release Scope/Repetitive Merges And Conflict Resolutions:

Feature Branching:
If a decision is made to extend or reduce the release scope you may be required to re-merge all the feature branches again, which is hardly an easy process. And this doesn't account for the time spent doing the re-merge and the probability of costly mistakes.

GitFlow/Trunk-Based Development:
When things don't go as planned, you may be required to make adjustments to the RC, taking out features that didn't quite make it through testing or
whose functionality requires further deliberation. A simple revert isn't enough, as features might be deeply linked and require careful unwinding and/or disabling. So, you'd ultimately be left with using extremely complex techniques that will almost always make your life miserable down the road, by which point you'd be pretty much toast.

Painful, yes? I agree. Okay, moving on...

The Solution At A High Level

It would be unfair to overwhelm you with challenges without offering a solution. We'll take a step-by-step approach, starting with the big picture.

Here's the solution in a visual representation:

There are a few things to note here:

1. All PBI branches are created from the Release branch
2. PBI branches go through PR and are merged into the Integration branch.
3. The Integration branch is created out of the Release branch and never merged anywhere. Periodically, it can be recreated to eliminate any accumulated clutter from abandoned code changes.

Notice that the diagram appears to blend elements of Feature branching and GitFlow. It shares the characteristics of Feature branching, allowing the testing of PBI branches independently. However, these branches are pre-merged into the Integration branch early on, serving the dual purposes of integration testing and conflict resolution. This, of course, minimizes the necessity for extra test environments and resources.

However, there are potential concerns:

1. As previously mentioned, in a highly dynamic project with numerous developers making contributions, how can we effectively track all the ongoing PBI branches to ensure that when it's time to cut the RC, we don't overlook or lose any essential work?
2. If conflicts arise, considering developers cannot merge the Integration branch into a PBI branch, the question arises: what should be merged, and how should conflicts be resolved?
3. Given that people may make mistakes, how can we guarantee that developers adhere to the rules, specifically refraining from branching out of the Integration branch and avoiding merging it into PBI branches to resolve conflicts?
4. Concerning the cutting of the RC branch, are you proposing manual merges of a certain number of PBI branches? Granted, if done correctly there is a guarantee that the dev doing the merges will not encounter any conflicts, as those would have already been resolved when the PBI branches were PRed/merged into the Integration branch.

Addressing these concerns requires safeguards and automation. Our "in-house" solution is based on powershell for git scripting and logic and ADO tools set for git repo hosting, tracking, planning, linking, building, execution, and querying purposes.

Git Building Blocks

The majority of the algorithms described below will need to walk and analyze the git commit graphs. Here are the Git building blocks that will be used over and over to get the job done.

Oldest Common Ancestor (OCA) (Fork Point)

Please consider the following commit graph:

The goal is to find OCA commit between any arbitrary pair of blue and yellow commits. This article should help.

Some Useful And Frequently Used Commands

• git rev-list, mostly using the options --ancestry-path, --first-parent, and --format
• git log, (how could we live without it!) with the --merge option to discover conflicting commits.

ADO Look And Feel

If you have never seen the ADO interface, please consider looking into the official documentation. I'll be referring to some of the screens as we go.

• Pull Request
• Product Backlog Item
• Build Pipelines
• Queries

Our "in-house" ADO PBI template was extended to include two extra fields:

PR Validation

The best way to teach and avoid costly mistakes is to do early detection in order to "slap the wrists" of the "wrongdoers". We already had PR validation build pipeline in place, which fires for every PR raised. It was performing validation build and unit tests execution. Now, we extended it a little bit to include the PR validation step, which gets executed at the very beginning of the build to provide the quickest feedback possible.

Source Branch Validation

The purpose of this script/logic is to detect merges from the Integration branch and fail the validation build to prevent PR merge. The algorithm used can be described in the following few sentences:

1. Find the oldest common ancestor commit (OCA - fork point) between the PBI branch and the Release branch (main/master);
2. Find a commit (H) on the Integration branch (develop) that was created right before the first commit (C) on the PBI branch since the oldest common ancestor (OCA).
3. If there is an ancestry path (in red) from the tip of the PBI branch (HEAD) and the commit that was detected at step #2 (H), then fail validation.

Enforce Proper Linkage

For automation to work, we have to trust the state/information of our ADO Scrum board. Each PR should be linked with a related PBI. One way to prevent an incorrect PBI linkage would be to check all the PBIs linked to the PR and find at least one where the Target Branch PBI field matches the PBI branch name. Also, we detect and prevent linkage to PBIs in the Done state, which would indicate invalid planning or execution of the plan. A nice side effect of these checks is that they enforce work and time tracking, as no pull request can be merged without a corresponding PBI on the board. The Target Branch field will be discussed in more detail in just a bit.

Merge Conflicts

I dream of a world where merge conflicts are completely outlawed. But, until that glorious day arrives, we're stuck giving them a reluctant hug and inviting them in for tea.

With the FeiFlow branching strategy, DEVs may encounter more frequent merge conflicts compared to GitFlow or similar methods. However, these conflicts are now distributed amongst DEVs rather than causing constant trouble for a few. Subject matter experts, who encounter conflicts more often, are best equipped to resolve them due to their familiarity with the affected code. Later, we'll explore approaches to minimize conflicts.

As previously mentioned, we cannot merge the Integration branch to resolve conflicts. When a DEV working on a PBI branch submits a pull request and conflicts are detected, we require a tool/script (let's call it How-To-Merge). This tool, available as a stand-alone script, can be executed locally on DEV's machine and/or as an ADO Build pipeline. The script's output might look like the following:

git fetch

# Resolve conflicts and commit after each merge command
git merge 45986b5

git merge 00a94ff

How-To-Merge Inner Workings

Please keep the following git commits graph in mind:

Let's assume that our PBI branch (which is not in the picture) is conflicting with commit C10. The idea behind this script is to find and suggest merging either C10, P11, P12, or P22. Our implementation will choose P12, this is what we call optimistic pollution. Here are the reasons behind this choice:

• We exclude C10 from consideration as it could be a work-in-progress with broken functionality or unit tests.
• P11 may not be the best either as there is a follow-up P12, which we hope addressed certain issues found since P11; it should be evident now why we call it optimistic.
• And finally, P22. You might have noticed that any conflict introduces a bond or dependency between unrelated PBIs. We try to keep the pollution to a minimum to retain the flexibility of being selective when it comes to RC cutting.

Developers Code Of Conduct

Now we are ready to formulate a very precise and easy set of rules for DEVs to follow:

1. Before starting work on a PBI branch, first, check the Target Branch field of the PBI. If it's set, switch to that branch (i.e. via git checkout ) for your upcoming tasks. If it's empty, you're free to create a new branch, publish it (i.e. push to origin), and update the Target Branch field on the related PBI accordingly. The Target Branch field can also serve as a recommendation from senior DEVs when there are dependencies across PBIs that need to be communicated. In such cases, the same branch name is typically set on the dependent PBIs.
2. Create the PBI branch out of the Release branch (main/master).
3. Raise a PR targeting the Integration branch (develop).
4. In case of conflicts, use the How-To-Merge script/pipeline and follow the produced conflict resolution instructions.

Release Candidate

Finally, we've reached the best part. If you happen to be the tech lead reading this, I suggest leaning back and popping open a bottle of your finest Champagne. This stage is reserved exclusively for the BAs and QC.

Planning

"Man plans, God laughs" - Unknown

Last time I checked, the prices for "Crystal Balls" were sky-high, yet even the latest models offer only so-so results beyond a 2 to 3-day forecast. So, we've ditched wishful thinking and opted for factual planning.

BAs lead the planning ceremony, where they assess the RC's scope. They go through all the PBIs, preferably those marked as Done, and fill in the Release ID field. Using specific ADO queries helps them filter tasks efficiently. This is also a chance for BAs to spot work that needs prioritizing for the release or to be a subject for negotiation with the customer, to get it excluded or to have the Release date shifted.

Planning can happen frequently, and the scope can adjust as necessary, making it equally easy to expand or shrink the scope as needed.

At the start of Release preparation, we aim to include as many completed PBIs as possible. Since they've already been tested on lower environments, we have time for retesting. As the Release date nears, we become more selective, only including changes or fixes crucial for a timely, bug-free release.

It's also important to note that planning occurs close to the actual RC cutting period.

Release Candidate Cutting ADO Pipeline

Once the planning phase is completed. We execute an ADO Build pipeline providing the Release ID as an input. This pipeline/script queries ADO to get all the PBIs marked with the provided Release ID. Remember, we enforced very strict linking rules. And for each PBI we can get the list of linked PR commits. Please note that the Target Branch field is irrelevant now, and we rely on linked commits instead. What happens during script execution can be summarized as follows:

• Each PBI is validated. Those PBIs not in the Done state will result in a warning. PBIs that have no associated commits or active PRs will result in an error. Those without associated commits, however, aren't necessarily a problem since they might have been created to track work that didn't require a code change or other configuration work. That decision is to be made manually. If confirmed not to be a planning/tracking issue, we can apply a special branch name (eg. N/A) onto the Target Branch field to disable this error.

• Each PR commit is validated. We repeat the Branch validation logic, which was discussed earlier (see PR Validation). We also detect and ignore PR commits that are fully merged into the Release branch.

• Optimization and sorting logic detects commits that can be excluded as they are implicitly included by the descendants. (see commit P11 in the How-To-Merge Inner Workings commits graph as an example).

• Finally, the RC branch is created out of the Release branch and the filtered commits get merged into it in chronological order.

In the event of consecutive RC cutting, the previous Release branch is renamed, so that we can always easily compare and analyze the difference between the previous and current RC branch composition code-wise.

Resulting Artifact

With FeiFlow automation in place, non-technical staff such as BAs and QC are empowered to handle RC branch creation, since the pipeline's output is tailored to a non-technical audience. It generates a Markdown file as an artifact, conveniently linked to the pipeline instance for quick access and review. This interactive artifact allows easy exploration of any aspect of the RC with just a click. Here are a couple of examples:

The Release

Please note that in FeiFlow world not all the releases are meant to be delivered to the customer/production. Please see Periodic Releases and Technical Releases (see below)

The release day finally arrives. Our RC branch and corresponding deployment package are fully tested. It's time to "officially release" the RC. And you'd be right if you guessed that we use another ADO pipeline to make this happen. The release pipeline/script's input is the RC branch name and an optional indicator if the "versioned release" is to be made. The pipeline merges the RC branch into the Release branch, applies git tags, and merges the Release branch into the Integration branch. Still, we are to make a few validations before committing as follows:

• Ensure that the Release branch (main/master) is in a "fast-forward" state compared to the RC branch. If not, it indicates that another release has been created while the current RC is still in testing. Not a big deal though, since we can "re-cut" the RC (see Release Candidate) and test it a little bit more. Also, this is the reason why we don't use a "numbered" version (eg. v1.1.0) for Release ID (more on this in just a bit).

Delta Branch

At this point, we've established some pretty straightforward rules for DEVs to adhere to. However, rules are made to be broken and exceptions just make them stronger. Consider a scenario where you're putting the final touches on your RC branch and a bug is identified. You fix it on a PBI branch and create a PR against the Integration branch, intending to get it tested and integrated into the RC branch, only to encounter a merge conflict. At this point, resolving the conflict as outlined in Merge Conflicts implies introducing additional pollution to the RC branch. In such a case, creating the PR against the RC branch directly might be preferred. This means, however, that you'd likely run into the same merge conflict when the release pipeline attempts to merge the RC branch into the Release branch and then the Release branch into the Integration branch. Not to worry though as the Release pipeline/script that we established earlier will/should catch it and fail gracefully. Running the pipeline should generate and publish a "Delta" branch (derived from the RC branch) along with a fresh set of How-To-Merge Script Instructions. Once the Delta branch is updated and the Release pipeline is re-run, we merge and tag the Release branch. Then, a few commits from the Delta branch are applied after the tags on the Release branch, and then the Release branch gets merged into the Integration branch.

Final Thoughts And Considerations

Our journey is almost complete. But just in case, let's fill in a few potential gaps.

Periodic Releases

For FeiFlow to work efficiently, we should try not to allow the Release branch to get too far behind the Integration branch. Otherwise, the probability of conflicts may increase and development and testing of the PBI branches on local machines may get slightly more complicated. Therefore, unless there is a customer release on the horizon, we can consider doing periodic releases when fully tested PBIs get merged into the Release branch. Such releases can be tied, for example, to the Scrum sprint schedule or to when a desired number of PBIs have been completed.

Technical Releases

Another reason for a release can be code refactoring which touches many files and produces almost guaranteed conflicts for other PBIs in work. This type of release is almost always a release of a single or very few PBIs.

Sacredness Of The Release Branch

Our Release branch is not "sacred" as it may contain code that is yet to be integration tested, which will certainly happen when we start preparing for the next customer release. Instead, we rely on git tags to keep track of important milestones and versions.

Support Namespace

With FeiFlow, you might have noticed that the Release branch frequently gets updated with new commits beyond what was last deployed to production. What if the next customer release is in the distant future and a bug has been found in the production environment, requiring an immediate hotfix?

Well, in this case, we use an extended GitFlow-like branching approach with the support branches and continue to exercise FeiFlow in a newly created support branch namespace. Here's an example of a timeline and mitigation actions:

• The last customer/production release was v1.1.0
• Since then, a few periodic and/or technical releases have been produced
• A bug is found on production and requires urgent attention and a hotfix.

The mitigation steps are as follows:

• Create support Release and Integration branches such as support/1.1/master and support/1.1/develop out of release tag v1.1.0.
• Continue to apply FeiFlow in the allocated namespace and produce patch releases such as v1.1.1, v1.1.2, and so on, as needed.
• Use PBI branches in the main namespace to merge patch releases.
• Once the customer(s) are updated with a new production release (eg. v1.2.0), the support/1.1 namespace can be abandoned.

Keeping The Plans Apart

Unlike GitFlow and Trunk-Based strategies, whose release timing and procedures are often tightly coupled with the timing and planning scope of Scrum/sprint, FeiFlow completely decouples Scrum timing from release management. This means much fewer headaches for planners dealing with uncertainties.