Estimating defect rate using GitHub API

Why?

How do you know if your team of software developers is improving?

What you really want to know about are the outcomes: revenue, customer satisfaction, retention - all that good stuff means clients are choosing your product over competitors. These are hard to pin down to specific departments, let alone individual contributors (ICs).

So instead, many resort to measuring the output: completed tasks, commits, or lines of code (LOC). These can be easily compared between ICs, but unfortunately, they can also be just as easily misinterpreted or manipulated.

If we closed 30 tasks in the previous sprint and 40 tasks in the current one, clearly we're getting more productive, right? Maybe. Or maybe we just started shipping half-baked code before it's ready, which resulted in more bug reports. Productive? Technically yes, you are producing more. Desirable? Not exactly.

"When a measure becomes a target, it ceases to be a good measure".
Goodhart's law

This isn't even malicious in most cases; it's just people responding to subtle incentives.

A good tech lead must know the difference between the output and the outcome. The two share a relationship, but rarely a straightforward one. At best, output is the means to an end. At worst, it can be a distraction from actual business goals.

So what do you do instead?

For mature and mostly feature - complete products, I suggest measuring defects instead.

PMs and POs will usually ensure the team keeps shipping features and improvements. As a tech lead, it's your responsibility to ensure the consistent quality, and absence of defects strikes me as a pretty indicator of that.

You can define defects in a way that fits your team best. It could be bug reports, exceptions, or in my case - hotfixes.

I went with hotfixes because these are the least ambiguous - by definition, it's something to be fixed urgently. It's not a perfect definition, as not every defect calls for a hotfix. Still, this should give you a better idea of how much time your team spends firefighting.

How?

What you need to know about our git workflow for the following to make sense:

• Feature branches are based on and merged into the develop branch.
• Hotfix branches are based on and merged into the main branch.

Pretty vanilla stuff, really.

With that in mind, if we divide the number of hotfix PRs by the number of feature PRs, we'd get a rate of how many defects we're introducing with each released change (on average):

DEFECT_RATE = HOTFIX_PRS / FEATURE_PRS

Implementation

Set up and authenticate GitHub CLI if you haven't already:

brew install gh
gh auth login 

Navigate to your repository locally and run the following snippet. I'll explain what it does below.

gh pr list --state merged --search "sort:updated-desc" --limit 1000 --json "mergedAt,baseRefName" -q 'group_by(.mergedAt[:7]) | .[1:] | map({ month : .[0].mergedAt[:7], total_prs: . | length, feature_prs: map(select(.baseRefName == "develop")) | length, hotfix_prs: map(select(.baseRefName == "main")) | length }) | map (. + { defect_rate: (.hotfix_prs / .feature_prs * 100 | round / 100) })'

What this does:

• Lists the 1000 most recent merged PRs (1000 is a GitHub API limit).
• Selects the timestamp of merge and base branch name, in JSON.

At this point, we pass the JSON to JQ using the --jq flag, which then:

• Groups pull requests by month.
• Drops the oldest month since it's almost guaranteed to be incomplete (limit of 1000, remember).
• Counts total PRs within the month, feature PRs, and hotfix PRs.
• Calculates the rate of hotfixes to features.
• Rounds it to two decimal points.

Finally, you will get the output like this:

[
  {
    "defect_rate": 0.46,
    "feature_prs": 128,
    "hotfix_prs": 59,
    "month": "2023-10",
    "total_prs": 188
  },
  {
    "defect_rate": 0.95,
    "feature_prs": 108,
    "hotfix_prs": 103,
    "month": "2023-11",
    "total_prs": 214
  },
  {
    "defect_rate": 0.38,
    "feature_prs": 138,
    "hotfix_prs": 52,
    "month": "2023-12",
    "total_prs": 191
  },
  {
    "defect_rate": 0.32,
    "feature_prs": 164,
    "hotfix_prs": 53,
    "month": "2024-01",
    "total_prs": 219
  },
  {
    "defect_rate": 0.36,
    "feature_prs": 113,
    "hotfix_prs": 41,
    "month": "2024-02",
    "total_prs": 155
  }
]

If you want to go a step further and produce a more graphical representation, here's how you draw a very basic chart using some more JQ:

gh pr list --state merged --search "sort:updated-desc" --limit 1000 --json "mergedAt,baseRefName" -q 'group_by(.mergedAt[:7]) | map({ month : .[0].mergedAt[:7], total_merges: . | length, feature_prs: map(select(.baseRefName == "develop")) | length, hotfix_prs: map(select(.baseRefName == "main")) | length }) | .[1:] | map (. + { defect_rate: (.hotfix_prs / .feature_prs * 100 | round / 100) }) | .[] | "\(.month):\t" + "\(.defect_rate)\t" + "*" * (.defect_rate * 100)'

Will produce:

2023-10:    0.46    **********************************************
2023-11:    0.95    ***********************************************************************************************
2023-12:    0.38    **************************************
2024-01:    0.32    ********************************
2024-02:    0.36    ************************************

So now you know! What you do with all that data is a whole other blog post.