Adopting GitOps for Prometheus Rule Management — Git Server Setup Guide Included

When managing Prometheus configuration or rule files, it sometimes can be challenging due to the following issues:

1. Syntax Errors, Rule Validation: Incorrect syntax can cause unexpected behavior. It requires careful testing and validation to avoid false positives or false negatives in the alerting system.
2. File Versions, Undoing: It can be frustrating if changes to files cannot be easily reverted.
3. Weak Collaboration: Operators do not know if they have exactly the same copy of the configuration files.
4. Rule Visualization: Users do not know how the rule behaves on metrics unless they query the data from Prometheus explicitely.

Recognizing these challenges, the answer lies in the adoption of a system that addresses these issues. This is where distributed version control system Git, along with a suitable visualization tool, can provide comprehensive solutions to manage Prometheus configurations and rules.

Git

Widely recognized among software developers and programmers, Git is a free and open-source distributed version control system with several standout features:

• Distributed Version Control: Every developer has a full copy of the project repository, facilitating offline and independent work.
• High Performance: Git operates swiftly, handling large projects and comprehensive histories with ease.
• Seamless Collaboration: Git enables easy collaboration via features like remote repositories, pull requests, and code reviews.
• Undo and Versioning Capabilities: Changes can be readily undone, and previous code versions are easily retrievable.
• Cross-platform Accessibility: With numerous cross-platform clients, Git is accessible and user-friendly across various operating systems.

GitOps leverages Git-based workflows, extending from application development to infrastructure configuration. This ensures every change throughout the application lifecycle is auditable and traceable within the Git repository.

GreptimeCloud Provides a Git Workflow for Prometheus rule management

GreptimeCloud, a Cloud Native Time-Series Data Platform, has uniquely adopted the GitOps philosophy, using Git as the CRUD API to manage user rules. The choice of Git, thanks to all the advantages mentioned above, and broad acceptance among developers, makes it an unparalleled solution for rule management.

But how exactly does GreptimeCloud handle Prometheus rule files?

Prometheus rule management in a GitOps way

Creating a service in GreptimeCloud means initiating a database instance in a specific region. Upon creating this service, you'll receive a corresponding Git repository which you can utilize to manage your Prometheus configuration files such as prometheus.yml , and alerting/recording rule files.

You can interact with this repository just like you would with any Git repository. Clone the repository to your local system, make modifications to your rule files, commit these changes, and push the updated files back to the remote repository. Server-side hooks are in place to validate your rule files and provide valuable feedback.

Git Operations in GreptimeCloud

Once you successfully push the rule files, GreptimeCloud Prometheus Workbench combines all the rule files with the data remote written from Prometheus to generate dashboards. Moreover, you can directly edit rule files in GreptimeCloud Web, and the dashboard will be updated in real-time. Also, you can sync these changes to your local copy via git pull or git fetch .

Chart View in GreptimeCloud

File View in GreptimeCloud

If the condition matches, alertings will be sent to your AlertManager or Slack webhook. Additionally, you can export your rules in Grafana's JSON format, and then import these JSON files directly into your Grafana instance.

Looking ahead, we plan to offer a Dashboarding-as-Code solution. This will facilitate the management of Grafana dashboards in a declarative style, making data visualization more aligned with GitOps principles. Keep an eye out for our upcoming updates.

Leverage Git Workflow on Your Own Server

As we unveiled our use of GitOps for rule management in GreptimeCloud, you might be curious about our implementation of the GitOps workflow. Furthermore, you may want to utilize Git operations to manage your own service configuration files for a variety of advantages.

To help you establish your own Git Server and effectively harness the full power of Git operations, we'll now outline our server setup methodology for your reference.

Git Smart HTTP Server Workflow Setup

In a typical client-server setup, there is a central server that stores the repository, and clients interact with the server to perform various operations. Git can be used in this way by designating one repository as the central server repository, often referred to as the "origin" repository.

Server

Git can communicate over HTTP using two different modes. After Git 1.6.6, Smart HTTP protocol is able to intelligently negotiate data transfer in a manner similar to how it does over SSH. It runs over standard HTTPS ports and can use various HTTP authentication mechanisms.

Setting up Smart HTTP is basically just enabling a CGI script that is provided with Git called git-http-backend on the server. When invoked it will do all the negotiation to send and receive data over HTTP. It does not implement any authentication itself, but that can easily be controlled at the layer of the web server. You can do this with nearly any CGI-capable web server, like Apache or Nginx.

If you want to set up the Smart Git HTTP Server using a popular programming language, such as Go, it isn't too complex. You can refer asim/git-http-backend for some ideas. The most frequently used paths are /git-upload-pack, /git-receive-pack and /info/refs, these will handle the git operations like clone, push, pull, fetch. And you can also implement your own access control mechanism to protect your repositories.

Hook

Now that your Smart HTTP Server setup is complete, how can you customize it to align with your own workflow? Like many other Version Control Systems, Git has a way to fire off custom scripts when certain important actions occur: Git Hooks. There are two groups of these hooks: client-side and server-side. Client-side hooks are triggered by operations such as committing and merging, while server-side hooks run on network operations such as receiving pushed commits. You can use these hooks for all sorts of reasons. Server-Side hooks include pre-receive, update and post-receive, you can make use of any of the server-side hooks to enforce nearly any kind of policy for your project, fire http request, store data into database, publish event into to message queue, etc. The pre-receive hooks can exit non-zero at any time to reject the push as well as print an error message back to the client.

Since your Smart HTTP Server is ready to serve your git operations, you can clone repository, add your configuration files, commit and push to the remote server, thus triggering the server-side hooks to continue your CI/CD workflows.

Conclusion

In conclusion, the versatility of Git extends beyond its commonly recognized role as a distributed version control system. When leveraged on one's own server, it offers robust features, heightened control, privacy and security. The integration of GreptimeCloud with Git workflows demonstrates the power of GitOps philosophy in managing Prometheus configuration files.

As we continue to explore with Git and GitOps, we encourage you to stay tuned to our webiste for the latest updates and features. Should you encounter any questions, feel free to contact us via Slack.

Recently GreptimeCloud announced its Public Tech Preview. A highlight of this release is Prometheus Workbench, a managed Prometheus solution offering:

• Support for Prometheus remote write and remote read
• Ability to function as a Prometheus datasource in Grafana
• Capability to check alerting rules following Prometheus way
• Management of rule files in line with GitOps practices

Sign up for our free trial today and experience how our cloud service can revolutionize your handling of time-series data before the trial ends. We're eagerly anticipating your valuable feedback!