Build metrics and budgets with git-metrics

Monitoring metrics for a project can be challenging. Deciding what to track, determining the cardinality of each tag, naming the metrics, and choosing what should be split into a tag or kept as a metric are all complex tasks that require experience.

But monitoring the run of your project is not the only thing we need to monitor; the development phase is also important. Code coverage is a good example of how we monitor the build of a project. Many projects implement rules to prevent code coverage from decreasing. In the frontend world, there is a growing trend to track metrics such as page size and load time. This kind of monitoring might seem less complicated, but determining where and how to store the metrics is more challenging.

In recent years, I've been working on several projects where I wanted to put in place such boundaries. When looking at how to implement that, I always ended up using multiple external services or deploying applications to store those metrics that I would have to keep running all the time. This didn't match my requirements. I wanted to keep that cheap in terms of resource usage and low in terms of maintenance. I don't want to have a VPS running 24/7 just to push some metrics.

With global warming already well established, and human activity being the principal reason for it, we should rethink the way we build software. We cannot keep increasing exponentially the resource usage of our systems. Therefore, we need to put in place boundaries on the projects we build.

State of the Art

For open-source projects, many SaaS platforms offer free tiers for monitoring. For tracking code coverage, you can use Codecov or Coveralls. For tracking complexity, CodeClimate is a good option. These platforms integrate well with GitHub repositories.

For closed-source projects, these SaaS platforms offer paid subscriptions, which may require budget considerations. Alternatively, if your DevOps team has the bandwidth, you can set up tools like SonarQube for in-house monitoring.

However, be prepared for potential vendor lock-in, which can make switching services or losing historical data problematic.

Alternative Approach

Shouldn’t the metrics related to the code be attached to the code itself?

Some projects, such as typescript-action, version code coverage alongside the code. This approach isn't perfect, as it requires developers to update the metrics for each commit, or you need a CI job to keep them in sync. However, keeping metrics directly in Git is appealing as it provides easy access and avoids reliance on external services. Switching from GitHub to GitLab, for instance, would be as simple as pushing the repository.

To address synchronization issues, instead of writing these metrics with the code, they can be kept next to it. Git offers a feature called notes, which allows attaching textual data to a commit reference.

Solution Implementation

This is where git-metrics comes into play.

Basic Usage

With git-metrics, you can attach, replace, or remove metrics for a given commit with a simple command:

$ git-metrics add my-metric-name --tag "foo: bar" 1024.0

To show the metrics attached to a given commit:

$ git metrics show <commit-sha>
my-metric-name{foo="bar"} 1024.0

To display a list of all commits, or within a commit range:

$ git metrics diff HEAD~2..HEAD
- my-metric-name{foo="bar"} 512.0
+ my-metric-name{foo="bar"} 1024.0 (+200.00 %)

These commands provide a solid foundation, but there are additional features worth exploring.

Extra features

Budget management

Tracking metrics is useful, but the goal is to improve the codebase or project. One way to ensure this is by blocking commits that don’t meet certain metric criteria, a feature already used by many external services.

With git-metrics, you can configure rules by adding a .git-metrics.toml file to your repository:

# fails when increase size is above 10%
[[metrics.binary-size.rules]]
type = "max-increase"
ratio = 0.1

# fails when the size increases by more that 1MB
[[metrics.binary-size.rules]]
type = "max-increase"
value = 1048576.0

# fails when binary size is more than 10MB
[[metrics.binary-size.rules]]
type = "max"
value = 10485760.0

# fails when the code coverage goes below 80%
[[metrics."coverage.lines.percentage".rules]]
type = "min"
value = 0.8

# fails when the code coverage decreases of more than 5%
[[metrics."coverage.lines.percentage".rules]]
type = "max-decrease"
ratio = 0.05

Running git-metrics check from your CI after adding your metrics will inform you if your changes meet the specified budget.

Importing from other file formats

Manually adding metrics with the add command can be cumbersome, especially for derived metrics. For instance, after computing code coverage, you might end up with an lcov file, which needs to be processed to extract useful metrics. git-metrics can handle this:

$ git-metrics import lcov ./lcov.info

This command adds summary metrics such as coverage.lines.count, coverage.functions.hit, or coverage.branches.percentage.

Currently, git-metrics supports only lcov files, but more formats will be supported as the tool evolves.

Conclusion

git-metrics is in its early stages but already provides essential commands for tracking metrics, creating budgets, and blocking contributions that don’t meet the budget criteria, all within your git repository.

Similar to the mentioned SaaS solutions, git-metrics offers GitHub actions for installation, tracking, and checking metrics. GitLab support is expected soon.

Article also available here