Over the last week I've been working hard on migrating our Frontend Monorepo from a on-premise Jenkins pipeline to a cloud based solution.
One of the big obstacles I had to overcome was dealing with the sheer size of our monorepo. If we include the whole history (all branches on remote, all tags, full history), the size of our repo amounts to more than 1GB.
Downloading that much data on each run is a waste of bandwith and time, so I've dived into the git documentation and found several possibilities that I will share with you.
This simple flag (
--single-branch) makes sure you only fetch the history of your main branch. The more active branches live on your remote, the bigger the benefit and with more than 400 active contributors in our repo, that helps!
Another simple flag! This one omits the tags from the git history. Tags are often version numbers such as
<package-name>@1.2.67. Although not a huge data saver, it helps. Be careful though, some CI pipelines need tags, so double check if you need it.
Oh yes, now we are talking. This flag restricts the history to the latest commits. Really useful if you only care about the newest state of a certain branch. For example if you know you need to build a feature branch (or your main branch) and you don't need all it's history! Potential huge data savings 🤤.
If you just need the latest state of a single branch this is your command! You can also modify this command to checkout a specific branch by adding
--branch=<branch-name> if you are building a branch that is not your main.
This is where things become a bit more tricky. In this scenario
git clone might not be our best bet because it only allows for cloning all branches or one branch.
This was also the scenario which was most tricky for me, because I needed to determine a single "ancestor" commit where a feature branch started to branch of the main branch.
As we saw this command fetches all active branches with limited history. This might work for you if you are interested in the most recent state of two (or more branches).
Downsides of this command is that it still fetches all branches and that determining "ancestor" commits is difficult/impossible (depends on depth, but can still be inaccurate).
This one might seem scary, but can actually be really effective. You have full control over the number of branches you want to track, if you would like tags and still control depth of the history
- Initialize an empty git repository by running
Add the following snippet to your
# .git/config [core] repositoryformatversion = 0 filemode = true bare = false logallrefupdates = true [remote \"origin\"] # replace with your own repo! url = firstname.lastname@example.org:patricksevat/foo.git # list the branches that you want to track fetch = +refs/heads/main:refs/remotes/origin/main fetch = +refs/heads/development:refs/remotes/origin/development fetch = +refs/heads/feature:refs/remotes/origin/feature # Replace main with master or whatever your main branch name is [branch \"main\"] remote = origin merge = refs/heads/main
Now you can
git fetchwith whatever flags you like.
Do not add any flags for full history of the listed branches, or use
--depth=1to limit history.
Don't forget to run
git checkoutwith one of your fetched branches or you'll have an empty working directory!
I hope these commands help you in optimizing your own git CI strategy. If you have any useful tips or improvements please leave them in the comments!