Originally published on peateasea.de.
Not only should the final commit in a pull request patch series pass all its tests but also each commit back to where the pull request branched from main
. A Git alias helps to automate this process. Introducing: git test-fork
.
I take pride in delivering quality code in my pull requests, be it either for clients,1 or for open source software submissions. One aspect of that quality is to deliver small chunks of working functionality in each commit. To ensure that a pull request has this property, I like to ensure that each commit builds correctly, that the test suite passes and, depending on the project, that the linter checks also all pass. Of course, for a multi-commit pull request checking each commit manually is a lot of work. To avoid unnecessary work, Iāve automated this process with a Git alias: git test-fork
. Hereās why I use it and how its inner workings, erm, work.
My main focus: quality
Have you ever been doing software archaeology on a codebase and wondered why the test suite on an old commit suddenly no longer works? And have you then spent the next several hours trying to work out what the test failures have to do with the bug only to find out that theyāre not related? If so, you know how annoying if not downright painful such a situation is. Also, youāll know how stressful this can be because youāre usually under pressure when bug-hunting and wild goose chases are the last thing you need. Wouldnāt it have been nice if those who came before you2 had ensured that the test suite passed at every commit? This would have saved you time, stress and grey hairs. Also, this would have reduced friction in your bug-hunting, letting you find (and hopefully solve!) the bug earlier. Instead, you went on a minor odyssey, delving into problems unrelated to those you were actually trying to solve.
Iāve had that kind of experience one too many times in the past. My solution: make sure that each and every commit in a pull request passes its test suite and, if relevant, passes the projectās linter tests. Before I show the command and explain how its internals work, why do I think going to this effort is worth it?
To me, a passing test suite is a sign of quality and that whoever wrote the code cared enough to submit robust, well-tested code. Itās sort of like a craftsperson taking the time to build in quality and taking pride in their work and a job well done. Software isnāt a work of art in the way that Biedermeier furniture or a Stradivarius violin is, however a strong focus on quality and craftsmanship (for want of a better word) makes for better software.
A codebase in which each commit passes its tests has technical benefits: itās possible to use git bisect
automatically and any commit could be pushed to production at any time, affording a high level of development flexibility and responsiveness.
Personally, I like the feeling of a solid foundation; itās something I know I can depend upon and build upon.
Swings and roundabouts
Of course, running the full test suite on each commit has its downsides. It slows you down and some might feel that it adds unnecessary friction to the development workflow, especially when the test suite is slow. Others might feel that itās just another exercise in gold plating or some kind of over-the-top obsessive programmer behaviour.
These are valid points and there needs to be a balance between a desire for high quality and getting code āout the doorā. After all, one can go too far and being extremist about things is usually a bad sign.
That being said, sometimes itās a good idea to slow down when developing software so that our brains can mull over what weāre doing and contemplate the bigger picture. There have been times when Iāve been running the tests on each commit for a given branch and have realised that a commit wasnāt necessary, or the idea behind a direction of development was plain wrong. This extra cogitation time allowed me to rethink what I was doing and ultimately led to better software. Also, by not submitting some code, I saved my colleaguesā time, because it was code they didnāt have to review!
Sometimes, ensuring that each commit passes the test suite along a feature branch picks up on things Iād missed during development and should have fixed. Recently, I was refactoring some code and had finished a long-ish feature branch. I ran the tests together with the linter checks and the linter spotted a bug: while renaming a module Iād not updated the imports in a file. This was the code ātalkingā to me. It showed that Iād missed a particular code change and that there was a hole in my test coverage. This was a big win because it gave me the opportunity to improve the tests which will reduce risk and friction when refactoring in the future.
Also, if you notice that it takes ages to test each commit on a feature branch, then this is not a hint that you shouldnāt be testing each commit, but a sign that the test suite is too slow. Thatās something that you could invest time in in the future. Itās like an application of āif it hurts, do it more oftenā.
Another criticism of this technique is that it takes a long time on branches with many commits. This is a good thing: it provides feedback to you to keep your pull requests and feature branches small and focused. Again, āif it hurts, do it more oftenā!
Automatically testing forks via Git
Obviously, testing all commits on a fork of the main
branch isnāt something one wants to do manually. A single commit? Fine. Ten commits on a feature branch? Nah, Iāll pass, thanks. š
So how do you know when the current branch forked from the main
branch? And how do you make Git run tests on each commit? Letās get to that now.
Hereās the alias I have set up in my .gitconfig
:
test-fork = !"f() { \
[x$# = x1] && testcmd=\"$1\" || testcmd='make test'; \
upstream_base_branch=$(git branch --remotes | grep 'origin/\\(HEAD\\|master\\|main\\)' | cut -d'>' -f2 | head -n 1); \
current_branch=$(git rev-parse --abbrev-ref HEAD); \
fork_point=$(git merge-base --fork-point $upstream_base_branch $current_branch); \
git rebase $fork_point -x \"git log -1 --oneline; $testcmd\"; \
}; f"
Thereās a lot going on in here, so I tried representing it as a diagram:
whichāalong with the detailed explanations belowāI hope aids understanding of the alias code above.
A big shell function
Letās focus on the full test-fork
alias. In essence, this command uses git rebase
to execute a command (via the -x
option) on each commit from a given base commit.
This is a very long command, and as far as Git is concerned, this command is all one line. However, to make it easier to edit within the .gitconfig
file, Iāve split it across several lines by using trailing backslashes (\
).
On the left-hand side of the equals sign is the name of the alias: test-fork
. On the right-hand side is where all the action is happening: a shell function that Git runs when the user enters the command
$ git test-fork
within a Git repository.
We define the shell function within a large double-quoted string, and hence we have to be careful when embedding double quotes. The exclamation mark at the beginning means that Git treats the alias as a shell command and hence it wonāt prefix the alias by the git
command as would be the case without the exclamation mark.
The shell function has this form:
f() { ... code ... }; f
meaning that we define the function and then immediately run it. The semicolon separates the function definition from its call; by entering f
at the end we call the function that we just defined.
Defining the test command to run
The first line of the function is
[x$# = x1] && testcmd=\"$1\" || testcmd='make test'
This code tests to see if we have an argument and if so, sets the variable testcmd
to its value (i.e. the variable $1
). Otherwise, we set testcmd
to the default value of make test
. In other words, if you run
$ git test-fork 'some-test-command'
then some-test-command
will test each commit in your branch. If you donāt specify a command explicitly, then the alias falls back to using make test
.
The variable $#
is a special parameter in Bash and expands to the number of positional parameters. In other words, if there is a single argument, $#
will be the value 1
and the test
[x$# = x1]
will evaluate as true. The code will then take the first branch of this implicit if
block (i.e. the bit after &&
) setting testcmd
to the value passed in on the command line and using this as the test command for the rest of the code in the shell function.
The origin of branches
The next line in the function is
upstream_base_branch=$(git branch --remotes | grep 'origin/\\(HEAD\\|master\\|main\\)' | cut -d'>' -f2 | head -n 1)
which determines the name of the upstream (a.k.a. origin
) branch from which the local feature branch is based. This information will later help us work out where the feature branch forked off from the main line of development. This is the first Git-related command, which Iāve indicated by the number ā and the colour blue in the diagram above.
This line runs the command within the $()
and returns its result as the value of the variable. The $()
is a Bash feature called command substitution and
allows the output of a command to replace the command itself.
The command
$ git branch --remotes
returns a list of all locally-known remote branches. The one weāre interested in is either origin/master
or origin/main
, or what origin/HEAD
is currently pointing to.
The output from git branch --remotes
can be different in certain situations. Usually, you will see output like the following:
$ git branch --remotes
origin/HEAD -> origin/master
origin/master
origin/rename-blah-to-fasl
origin/refactor-foo-baa
where we have a reference origin/HEAD
which points to the actual upstream main branch, which in this case is origin/master
.
In other situations (and I havenāt been able to work out exactly why; I think this has to do with sharing an upstream repository with others, but Iām not sure), the output omits a pointer from origin/HEAD
to the main remote branch, giving e.g.
$ git branch --remotes
origin/main
where Iāve used the now more common main
name for the main branch in the upstream repository.
The filtering commands after git branch --remotes
handle both situations. By filtering on HEAD
, master
or main
with the grep
command, we ensure that all variations are in the filtered output. The cut
extracts the reference that origin/HEAD
is pointing to (if origin/HEAD
exists in the output) and the head
ensures we only select the first entry should there be multiple matches. If origin/HEAD
doesnāt exist in the output, the cut
passes its output to head
and we again select the first entry in the list of appropriate upstream branch names.
When constructing grep
regular expressions in the shell, we have to escape Boolean-or expressions (the pipe character |
) and groups (parentheses, ()
) with a backslash (\
). In the case we have here, we have to āescape the escape characterā within the Git alias by using two backslashes (\\
). Thus, when Git passes the command to the shell, there is only a single backslash character present and the regular expression is formed correctly.
After all this hard work, the variable upstream_base_branch
contains the name of the upstream base branch.
Where are we now?
The third line in our shell function
current_branch=$(git rev-parse --abbrev-ref HEAD)
finds out the name of the current branch. I.e. this is the name of the feature branch that we want to test. Iāve highlighted this information by the number ā” and the colour green in the diagram above.
We use this information, along with the upstream branchās name, to work out where the current branch forked from the main line of development. This is the purpose of the next line.
Where did we forkān come from?
Now weāre in a position to work out where the feature branch forked3 from the main line of development. In particular, we want to find the commit id of this fork point. Hence the next line of code assigns a variable called fork_point
:
fork_point=$(git merge-base --fork-point $upstream_base_branch $current_branch)
finds the best common ancestor(s) between two commits to use in a three-way merge.
When using the --fork-point
option, the command takes the form
git merge-base --fork-point <ref> [<commit>]
The --fork-point
option is key for us here because it finds
the point at which a branch (or any history that leads to
<commit>
) forked from another branch (or any reference)<ref>
.
In our case, we use git merge-base
to find the commit at which the current_branch
diverged from upstream_base_branch
. Iāve denoted this with the number ā¢ and the colour red in the diagram above.
It was a fair bit of work to get to this point, but now weāre in a position to use git rebase
to run our tests.
Skip to the commit, my darling
Now we get to the very heart of the matter: iterating over each commit in the branch and running the test command on each commit as we go. Iāve referenced this process by the orange number ā£ and arrows in the diagram above.
We run the test command via the -x/--exec
argument to git
:
rebase
git rebase $fork_point -x \"git log -1 --oneline; $testcmd\"
Here, we rebase the branch we are currently on (i.e. the feature branch) onto where it forked from the upstream base branch. Note that git rebase
operates on a reference which exists upstream and aborts the rebase if there is no upstream configured. This is why we use the upstream base branch when working out the fork point. Also, the upstream branch is usually the branch used for comparisons to the feature branch when submitting a pull request on systems such as GitHub, GitLab, Gitea, etc. Thus it makes sense to consider the state of the upstreamās main
branch rather than the local main
branchās current state.
The -x
option takes a string argument of the shell command to run. In our case here, we need to escape the double quotes so that they donāt conflict with the quotes enclosing the entire alias code. Doing so ensures that quotes still enclose the command to run for each commit in the rebase process. Note that we need double quotes here as well so that the value of $testcmd
is interpolated into the command ultimately executed by git rebase
.
To provide context for the test command, and to indicate where the git rebase
process currently is, we precede it with
git log -1 --oneline
This prints the abbreviated commit id and subject line on a single line for the current commit, which can be useful to know if the test command fails.
Finally, we run the test command defined in the variable $testcmd
. This is either the command specified as an argument to git test-fork
or is make test
if no arguments were given.
Thatās it!
Thatās the guts of the test-fork
alias in detail.
Clear as mud? Ok, letās see the command in action and hopefully its use and utility will make more sense.
git test-fork
in action
Sometimes itās easier to understand whatās going on if one sees something run. I canāt do that dynamically here, but I can show a representative example.
A quick but detailed example
Hereās an example from a Python project where I was wanting to reduce technical debt with the aid of the pylint
code checker. Iād used pylint
to sniff out any code smells which might need addressing and had a few commits on a feature branch which had fixed these issues. I now wanted to make sure that Iād not broken anything in the process, hence I wanted to run the test suite on each commit in the feature branch. Since I have a Makefile
which wraps the actual test command behind a simple test
target, I only needed to run git test-fork
. This is the output:
$ git test-fork
Executing: git log -1 --oneline; make test
8e6ff5b (HEAD) Fix import ordering
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================
<snip-lots-of-test-output>
======================= 257 passed in 239.65s (0:03:59) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Executing: git log -1 --oneline; make test
2c5c1d9 (HEAD) Remove unnecessary "dunder" calls
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================
<snip-lots-of-test-output>
======================= 257 passed in 248.33s (0:04:08) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Executing: git log -1 --oneline; make test
bccc026 (HEAD) Remove reimported module
make --directory=src test
make[1]: Entering directory '/home/cochrane/a-python-project/src'
. ../venv/bin/activate; pytest
============================= test session starts =============================
<snip-lots-of-test-output>
======================= 257 passed in 254.63s (0:04:14) =======================
make[1]: Leaving directory '/home/cochrane/a-python-project/src'
Successfully rebased and updated refs/heads/address-technical-debt.
There are a few things to note about the output:
-
git rebase
echoes the command itās running:Executing: git log -1 --oneline; make test
. This lets us check that Git is running the command we want it to run. - We see the
git log -1 --oneline
output fairly clearly. Bash displays this with nice, bright colours and is easy to see in the terminal. Unfortunately, I couldnāt reproduce that here though. Sorry. š -
make
changes into thesrc/
directory to then run thetest
target within that directory:make --directory=src test
andmake[1]: Entering directory '/home/cochrane/a-python-project/src'
. - Now we see the
pytest
invocation thatmake test
runs. - There is lots of output from
pytest
. Iāve removed a lot so we can focus on the main points in this discussion. - We see that the tests passed, yay! š
-
make
returns to the original directory after the commands in thetest
target have completed successfully:make[1]: Leaving directory '/home/cochrane/a-python-project/src'
. - Git checks out the next commit on the feature branch and the process repeats.
Rebase starts from the base
Note that the rebase command starts running from the projectās base directory. This is independent of where you run the git test-fork
command. So, if you want to run pytest
on an individual file, youāll need to explicitly change into the appropriate directory as part of the test command. In other words, if you have to be in the src/
directory to run a single test like this:
$ pytest tests/test_views.py
then using
$ git test-fork 'pytest tests/test_views/py'
wonāt work, because git rebase
operates from the base directory and pytest
wonāt be able to find the files. Also, using the full path to the test file by running
$ git test-fork 'pytest src/tests/test_views/py'
probably wonāt work. At least, it doesnāt work in my case because I have a pytest.ini
file which pytest
reads and itās in the src/
dir. Thus, the only command that will allow you to run individual test files in such a situation is:
$ git test-fork 'cd src && pytest tests/test_file.py'
where each command execution by git rebase
also changes into the required directory to run the individual test file. Running this command has output much like the detailed output included above.
Common usage variations
Other common invocations I use are:
$ git test-fork 'make lint'
which runs the linter checks on the entire codebase for each commit in the branch.
Also, I tend to use this one a lot:
$ git test-fork 'make test && make lint'
which runs the full test suite and the linter checks for each commit in the feature branch. Note that we can chain commands in the argument passed to git test-fork
by using the Bash Boolean-and operator: &&
.
What to do if things go wrong
Nobodyās perfect and something could go wrong in the middle of the rebase process. Actually, this is exactly what weāre trying to do: we want to sniff out any problems before they make their way upstream into a pull request. This way we avoid our colleagues from having to stumble across problems when reviewing the code.
So what happens if the test suite fails in the middle of a rebase? Git interrupts the rebase. Thatās all. Actually, itās great: youāre dropped right into the middle of where the problem is, which is the best place to be able to fix it.
After fixing the issue, run make test
(or the equivalent command) to check that everything now works. Add the changes with
$ git add -p
or use git add
on each of the files. Then itās simply a matter of amending the commit
$ git commit --amend
before then continuing the rebase with:
$ git rebase --continue
Or, if things look to be too complicated and you might need some thinking time, just abort:
$ git rebase --abort
Then, take a step back, take a deep breath, and dig in again.
Wrapping up
The test-fork
alias can be really helpful in finding test or linting issues locally before pushing code to colleagues or collaborators.
Iām fairly sure this code could be improved upon. Still, it works well for my purposes and is a standard part of my process to provide high-quality work to internal teams and external customers.
So what are you waiting for? Go test that forking code!
Iām available for freelance Python/Perl backend development and maintenance work. Contact me at paul@peateasea.de and letās discuss how I can help solve your businessā hairiest problems. ā©
Of course, ātheyā could have been an earlier you! ā©
I could have said ābranchedā here, but the phrase āa branch branchedā sounded a bit odd. ā©
Top comments (0)