In the previous post, we saw how to implement metrics in a simple Python Flask application. In this post we will see how to:

• start a local monitoring stack with a Docker composition
• configure Prometheus to scrape metrics from our application
• build a Grafana dashboard
• commit the Dashboard definition so other developers can also use it

Before launching an application in a Kubernetes cluster, we need to make sure that it generates the right metrics. This will be done using a Prometheus stack running in a Docker composition.

This test Prometheus stack will be made of:

• Prometheus: Core component that will:
  • scrape application metrics
  • store metrics in a time series database
  • expose metrics for use with Grafana
• Grafana: Visualization of Prometheus metrics

To be able to run this Docker composition, you will need to install Docker and install docker-compose.

Start Prometheus

Let's write a docker-compose.yml file that will start a Docker container for Prometheus:

version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    ports:
      - "9090:9090"

Be sure to use only spaces for indentation in the YAML file, not tabs.

Start the Docker composition with:

docker-compose up -d

This composition starts Prometheus with the default configuration and allows access to the web interface on http://localhost:9090/.
You will be redirected to the "Graph" page where you can query metrics using the PromQL format.

In the Status → Targets menu, you can find all exporters from which Prometheus retrieves metrics.

In the default setup, there is only one defined target, which points to the metrics endpoint of Prometheus itself.

Now it's time to configure Prometheus to retrieve metrics from our application!

Prometheus is running in a container but needs to access the application that runs on the host. We will start the application with:

flask run --host 0.0.0.0

With the new --host option, our application is running on all network interfaces, including the interface used by containers. From a container, you can connect to the host with the special IP address 172.17.0.1 (the bridge address for the 172.17.0.0 network, linked to the host). So let's create a prometheus.yml file that contains the Prometheus configuration:

scrape_configs:
  - job_name: view_buy
    metrics_path: /metrics
    static_configs:
      - targets:
        - 172.17.0.1:5000

• line 2: define the name of the Prometheus exporter; here we use the name of the application
• line 5: IP address and port of the application

The prometheus.yml file must be in the same folder as docker-compose.yaml.

We edit the docker-compose.yaml as follow to inject the Prometheus configuration in the container:

version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

Let’s restart the Docker composition with:

docker-compose up -d

You should now be able to see a target that points to the application running on the host on the target list page.

This target has a static label with the name of the application set in the prometheus.yml file: job="view_buy".

We can now display all metrics from this target on the Graph page. In order to query the time series for our label, you just need to surround the label and value with curly braces: {job="view_buy"}.

It's time to generate some traffic!

Open 3 terminals and run the following commands:

• watch -n 1 "curl localhost:5000/view/product1"
• watch -n 2 "curl localhost:5000/view/product2"
• watch -n 3 "curl localhost:5000/buy/product1"

On the Graph page of the Prometheus Web UI, you can query the view_total metric using the following PromQL expression: view_total. A visual representation of the metrics is available in the Graph tab, just under the "Execute" button.

Start Grafana

Add a new service to the Docker composition by editing the docker-compose.yml file:

version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
    ports:
      - "3000:3000"

We need to create the datasource.yaml file with the following content:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    isDefault: true
    url: http://prometheus:9090/

This file configures the connection between Grafana and Prometheus. The hostname of the Prometheus container is prometheus because this is the name of the service in the Docker composition. Restart the Docker composition with:

docker-compose up -d

You can now access the Grafana Web UI using the default login/password admin/admin.

Skip the password change for now:

In the Explore view, you can use the following expression to visualize the number of views for each product: view_total.

The view_total metric is a Counter, a metric that always increases. As suggested by Grafana, it is better to display a temporal derivative of the value, which can be achieved using the rate() function.

Now we have more meaningful information about the number of views of products. The unit is view per second. For example, "product2" has 0.5 view per second, which makes sense as it corresponds to one view each 2 seconds.

Finally, in the legend, you can find the labels of each time series, one per product.

Let's create a dashboard now.

Creating a Grafana Dashboard

In the Grafana web UI, you can create a new dashboard. Choose "Add an empty panel". Then type the following expression: rate(view_total[5m]) and validate with 'Shift+Enter'.

To improve the legend, you can define the "Legend Format" and extract label values with the expression {{<label name>}}. For our application, we will use {{product}}.

The legend looks better as we now only have the name of the product. It's also possible to add a prefix and a suffix, using e.g.: Product {{product}} views.

Now you can click on the "Apply" button to save modifications of the dashboard. You can then add a new panel for purchases. Finally, click on the save 💾 button.

Persisting Metrics and Dashboards

If you stop or restart the current Docker composition, you will lose all the metrics retrieved by Prometheus and the Dashboard you just built. To persist data, with this local setup, we will use Docker volumes.

We will first create a Docker volume to persist Prometheus metrics. The time series database is stored in /prometheus/ inside the container so we will mount the volume on this path:

version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus:/prometheus
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
    ports:
      - "3000:3000"
volumes:
  prometheus:

• line 7: we add a volume to persist the content of /prometheus folder
• last 2 lines: a top level volumes key is used to define volumes used in the Docker composition

Finally, you can apply changes with docker-compose.yml. Metrics are now persisted across restarts of the Docker composition. Note that mounting the volume resets all metrics previously retrieved.

In order to persist the Grafana dashboard, we will create a file that contains the definition of the dashboard:

• Switch back to the Grafana web ui,
• Open your dashboard
• Go to settings ⚙ and then "JSON Model"
• Copy the JSON definition of the dashboard
• Paste the copied content into a new dashboard.json file

The last step is to mount this dashboard in the Grafana container.

Create a dashboards.yaml file with the following contents:

apiVersion: 1

providers:
  - name: 'provisionned dashboards'
    orgId: 1
    folder: ''
    folderUid: ''
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 10
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

This file tells Grafana to load every dashboard defined in the /var/lib/grafana/dashboards. Finally, modify the Docker composition to include dashboards.yaml and dashboard.json in the Grafana container:

version: '3'
services:
  prometheus:
    image: prom/prometheus:v2.26.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus:/prometheus
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana:7.5.2
    volumes:
      - grafana:/var/lib/grafana
      - ./datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml
      - ./dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml
      - ./dashboard.json:/var/lib/grafana/dashboards/dashboard.json
    ports:
      - "3000:3000"

volumes:
    prometheus:
    grafana:

Update the Docker composition with docker-compose up -d. Your dashboard should be automatically loaded into Grafana.

Dashboard Modification Workflow

If you modify the dashboard, remember to copy the new JSON Model to the dashboard.json file and restart Grafana with docker-compose restart grafana. Now, your modifications are persisted as code in the dashboard.json and you can commit the dashboard definition within your project repository.

If someone wants to try your project, they just need to run the application and the monitoring stack with docker-compose up -d. They can then access your dashboard, see the evolution of metrics and create or modify graphs.

In this post, we were able to start a monitoring stack to retrieve metrics from our test application and build a dashboard to visualize the evolution of product views and purchases. We created several files that can be committed in Git and used later by other developers.

Conclusion

You are now ready to launch a monitoring stack aside your application, whichever language you’re using. You can also integrate the application as a new service in the Docker composition to ease test and demo.
This Docker composition should help you get your application monitoring code ready for production, so you can now deploy it on a Kubernetes cluster, and avoid the usual pitfalls of a new Kubernetes deployment.

In the next post, we will deploy the application and its monitoring in a Kubernetes cluster and see how to:

• integrate the application monitoring with the Prometheus Operator
• create alerts based on application metrics
• package everything in a Helm Chart