DevOps is a set of practices that combines software development (Dev) and IT operations (Ops). It aims to shorten the systems development life cycle and provide continuous delivery with high software quality. DevOps is complementary with Agile software development; several DevOps aspects came from the Agile methodology.

Helping to support such a vast operation we as DevOps teams use multiple 3rd party application that need constant monitoring.
One of those tools is Grafana. A great application for better observability across multi-data centers.
Using Grafana we are able to keep track on data coming from MSSQL ElasticSearch , Prometheus , Coralogix , API and more.
With such an extensive operation we need a way to back up all the dashboards we have per environment.
This prove to be a bit tricky since :

[X] How do we keep the same folder structures ?
[X] How easy will it be to restore one or many dashboards ?
[X] Will i have multiple backups per environment ?

---

What Do We Need To Backup

A dashboard in Grafana is represented by a JSON object, which stores metadata of its dashboard. Dashboard metadata includes dashboard properties, metadata from panels, template variables, panel queries, etc.
See an example dashboard for RabbitMQ:
RabbitMQ-Overview

As you can see the dashboard only points to the data source and to the query type and content.
So we now know that we need to backup just the JSON file of each dashboard stored in Grafana DB:

---

Backup Options
We explored a few option for backing up Grafana:

1. Server backup
   In case it's needed- we will need to create an new server and dump the image on it. This method is in effect by the SRE-IT department but the disadvantages oversees the good ones

2. DB backup
   In case it's needed- We will need to spin up a new DB ( MySQL or PostgresSQL ) and do a restore. Usually will require some assistance from DBA/DataOps

3. Dashboard backup
   This method , with it's challenges , proved to be a task that will require the DevOps team with no other dependency.

---

DevOps Building Blocks

Azure Repos

cloud-hosted private Git repos for your project
Microsoft Azure provide free private Git repositories that we use to store our code as well as other projects.

Azure Pipelines

Continuously build, test, and deploy to any platform and cloud
Cloud-hosted pipelines for Linux, macOS, and Windows. Build web, desktop and mobile applications. Deploy to any cloud or on‑premises using installed agents per environment.

---

Let's start backing up

First we will lay the architecture more clearly :

Azure agents will have the option to access each environment and query the dedicated Grafana API and return the result back to Azure Pipelines.
Those step will repeat each environment we would like to backup i.e QA/Sandbox/Staging/Production

let's see a more detailed view of the flow in one of the above env:

The flow will backup each environment to it's own GIT branch resulting in the below:

Code

The full code can be found here:
https://github.com/kfirlevin/grafana-dashboards-backup

Conclusion

With very easy setup using Azure Devops Repos and Pipelines we were able to backup all of our environments on a daily basis using a simple Powershell code that iterate over our environment .