Thanos: Musings of a Mentee

This is my account of my experience as a Fall ‘21 LFX mentee working with the Thanos project. The mentorship, as an experience, deserved an exhaustive post in its own right and this is my attempt at penning down the events this fall, in as much detail as I could.

This post is divided into 4 sections:

1. Project
2. The Thanos Process
3. Getting started as a contributor
4. Learnings

Project
My project involved adding a feature flag to enable Prometheus metrics which tracked the progress of compaction, downsampling and retention processes. These metrics are useful when performing a time-consuming compaction on large blocks of TSDB data. Prior to the project, compaction progress would have to be manually monitored using the bucket UI.The added metrics export data that can be visualised graphically and would give an estimate about some progress parameters related to these processes.I also added unit tests and the required documentation for the functions added as part of the project.

Some other additions I worked on include adding tracing support for compaction and adding the enable-feature flag for negative offsets and @ modifiers in Thanos compact.

For understanding my work better, these are some links:

1. The PR for compaction and downsampling progress metrics
   a. Diagram to understand compaction metrics.

2. The PR for retention metrics

3. The PR for compaction tracing support

The Thanos Process
Since my mentor and I were in timezones 11 and a half hours apart, our weekly sync up was on Sunday morning(for me)/Saturday night(for him). Time zones notwithstanding, we frequently communicated synchronously.

The first few meetings involved clarifying the project goals and understanding the Thanos compactor. Each meeting had a defined agenda and actionables for the next week.

Most meetings had me relentlessly asking my mentor all kinds of queries (Some in hindsight, very trivial). Each meeting would end with my mentor detailing the week’s actionables to me and each week would be peppered with queries on the nuts and bolts of the implementation. When I started on my first major PR(which was ultimately not merged), I was completely off the mark when coding my first draft. My mentor had to then break down the implementation into steps to get me onto the right track.

My project PRs were frequently reviewed by my mentor and I tried to resolve the comments as fast as possible. This led to PRs with a short review-response cycle with some intense discussions and a very steep learning curve from the reviews.

Getting started as a contributor
The first step to familiarising myself was setting up Thanos locally. In the process, I discovered that an interactive example lacked instructions and opened a PR for the same. This was my first PR and undoubtedly an easy one, which boosted my confidence a teeny bit.

The mentorship was the first time I was contributing to Thanos. To get acquainted with the code, my mentor suggested I take on some good first issues. My first such issue involved refactoring a file and had earlier PRs refactoring other files similarly, for reference. Even though this PR was relatively straightforward in terms of coding(since I wasn’t completely new to Go), it taught me about the Thanos PR process, which, incidentally, is very thorough. The reviewers found nits as small as an extra pair of parentheses in my commits!

Taking the initiative to work on small issues definitely paved the way for a smoother mentorship when I was working on my project. It helped me in many ways, the most obvious one being getting familiarised with navigating the codebase and the review process. In hindsight, I realise that one of the other ways it benefited me was the confidence I developed when I took on an issue, however trivial, discussed the approach with a maintainer and then iterated based on the reviews. The supportive community definitely merits a mention here and had it not been for the encouragement extended to the mentees, this would have been an intimidating task.

Learnings
Prior to the mentorship, I thought I was experienced enough in Go, having coded in it for roughly a year. What did I know, that myth would be soon busted! One of the things that caught my eye as a new contributor was the style guide. Opinionated and quite exhaustive, the style guide has some really interesting Go conventions. The few that struck with me are:

• Pre-allocating slices and maps if we are aware of their size
• Using values of type struct{} with maps since they are allocated with 0 bytes
• Handling errors from deferred functions.

Since my project involved adding metrics to a Prometheus exporter, I learnt about the types of metrics and developed a small side project to export some data. This was interesting and came in useful when I had to add metrics related to compaction, downsampling and retention to the upstream codebase.

Initially, I struggled to understand how I would pare down the compaction process to its essentials to simulate it and predict the blocks and compaction runs. My mentor then had to explain it to me using skeletal pseudo code, which cleared the fog a little but the implementations of some concepts like filter and syncer were still hazy. I understood those better in the last phase when after asking my nth trivial doubt, my mentor explained a crucial fact about syncer’s working which, in hindsight, explained the implementation details of code I had already worked on for a prior feature. The next phase, working on downsampling simulation, saw me a tiny bit more independent since the codebase is relatively more straightforward compared to the compaction codebase. I still considered adding some non-essentials to the process, which my mentor had to tell me were unnecessary. The last leg of the project involved working on a stretch goal, adding retention metrics. This was fairly straightforward since the retention codebase seemed more intuitive and it was the third time I was adding metrics for a process. Third time’s the charm :)

The exhaustive review process(easily a 100+ review comments, led by my indefatigable mentor) taught me a lot about considering some of the finer aspects of a feature, usually related to Prometheus and the Go SDK. These ranged from always specifying a registry when creating a new metric with promauto to avoid registering with the default global registry, using Reset() to clear earlier exported values and the basics of promtestutil for related testing.

Another area I learnt a lot about was structuring my code including,surprisingly, naming functions/variables and writing comments!
The first draft of my PR lacked the ‘code consistency’ of the final PR in terms of proper use of interfaces, consistent function signatures and abstraction of related data structures. Aside: the Thanos codebase is a pretty good example of a well structured Go codebase. Definitely worth your while exploring it!
I could never get variable/struct/function names right and fixing them seemed like a never-ending process during the reviews. Turns out, I either did not comment enough or when I did, my comments were too detailed and not abstract enough. Always remember: All comments at Thanos end with a period. Period.

The first major PR, which I worked hard on for a couple of weeks,was made redundant. It not being merged did dampen my morale for a tiny bit. However, apart from the technical learnings about the internals of compactor, the overarching lesson was not letting my enthusiasm dampen when faced with such a situation and not considering the time spent as a sunk cost. This, I would say, was one of the most important lessons from the mentorship.

To sum it up, the mentorship gave my progression as a developer a fillip in terms of not just technical and non-technical knowledge, but fostered a connection with an inspiring, supportive community and made me aware of interesting opportunities and perspectives which I wouldn’t have been if not for the mentorship. Truly thankful to everyone who helped me in this journey, my mentor, my co-mentees, the Thanos maintainers and the wider Thanos community!