Disclaimer

This post is about open source contributions, that happen during business hours and the company is paying for it. Why? Because the company uses open source projects and it is in the company's best interest to get certain/blocking stuff fixed.
This post is not about contributions that happen in people's free time.

Background

I have been a JAVA dev since the start of my career. Started as a desktop application developer in a Danish company, that produces high-end hearing aids for the hearing impaired people. After 7 great years, I switched to another high-tech Danish company, but this time as DevOps engineer. My primary task was to support 400 engineers by providing stable and reliable CI/CD system. This was the time, when my new colleagues introduced me to the Open Source and why it mattered. Great times - I can promise you :) One year into the new job, the company has decided to move from internally hosted Gitlab solution to Github. Yay!!! For our team it meant to support the migration and keep the level of build system at the same or even better performance level. Even though the time has been spent on verifying Github for our organization needs, the true issue came out only after the migration :)

The Problem

We have had many repositories in our organization. More than 1200 actually. Some of them have been built by TeamCity and some of them by Jenkins. There was no overlap between the 2 groups of projects. While all projects in TeamCity have been created by hand (using templates), the projects in Jenkins have been managed by configuration-as-code CasC. Great - it worked well in Gitlab - what could go wrong in Github? The answer is Repository discovery. It is a process performed by build systems to detect projects to build. Suddenly for every Jenkins restart, the repository discovery took 10+ min. These minutes got introduced into startup of Jenkins and caused a series of failures. Just to name a few: timeouts, health check problems, tests failing to start (yes, we validated every Jenkins installation), memory issues. Unfortunately, we could not silently ignore the problem and wait for the fix to appear. Engineers in our organization wanted new Jenkins features, bugfixes. We had to regularly update plugins, resize disks etc and therefore restart Jenkins. The life was not easy.

The Solution

Well, we thought, let's find the plugin responsible for Github repository discovery in Jenkins and fix the problem. Ok. That is not hard. Few days later we came up with the fix. The improvement is huge: the repository discovery went from 10+ min to 2+ min. You might think that the whole community has been waiting for the update, right? Or ...

Here is what happened next

Unfortunately, and to our biggest surprise, no one from the project maintainers was really interested in reviewing the change and get it released. Here is what happened next (look at the dates!):
Oct 1, 2020 Pull request (PR) published
Oct 2, 2020 First ask for a review from the maintainers
Oct 22, 2020 github-source-branch plugin comments on the PR
Oct 22, 2020 First review received
Oct 27, 2020 Review feedback addressed
Oct 27, 2020 First reviewer asks for a second pair of eyes
Jan 11, 2021 Approval for PR received
Sep 7, 2021 A ping to project maintainers
Sep 13, 2021 Third reviewer joins the conversation
Sep 13, 2021 Merge issues detected in the PR
Sep 20, 2021 Merge issues solved
Oct 4, 2021 Changes approved and merged
Dec 27, 2021 A new version of plugin released

What happened behind the scenes

16 releases happened between Oct 1, 2020 (the day that we published the fix) and Dec 27, 2021 (the day, when the fix was made available to everyone). Meaning we had to keep the fork of the project updated: integrate new changes, build and deploy the new version of the plugin. Everyone who has ever tried to keep Jenkins plugins updated in your Jenkins installation, knows what a mess it can be: plugins, that depend on other plugins. Oh, and do not forget transitive dependencies, that has to match. Well, should I have invested into automation? Maybe yes. Except, that my expectation was "I am very close. It will get the fix released soon."

How will I go about contributing to open source next time

Well, myself being an open source project maintainer, I understand, that the issue is in man power. And that is pretty much it. It takes time to correspond, review and release. The only thing, that I would like to improve next time is to align expectations with the project maintainers and as early as possible get in touch with them. Normally open source projects have CONTRIBUTING.md or a public slack channel to get started. As the last resort I would create an issue in the project repository and present my ideas there. Basically I would like to get a feel of:

• How active the project is
• How busy the people behind the project are
• How much my feature is relevant/important for the project etc

This information would help to keep things more transparent with my colleagues and my boss and potentially invest into automation while waiting for the fix to get released.

Acknowledgments

I would like to thank Rasmus Jelsgaard for encouraging to write this post and Sameena Syed for proof-reading.