In the previous post, we introduced the principles of Prometheus metrics series, labels, and saw which kind of metrics were useful to observe applications.

In this post, we will implement some metrics in a Flask application with 2 endpoints:

• /view/<product>: Display the product information. This page simply displays the product name.
• /buy/<product>: Purchase the product. Actually, this endpoint just displays a message.

We will start with the following implementation for our Flask application in a app.py file:

from flask import Flask

app = Flask(__name__)

@app.route('/view/<id>')
def view_product(id):
    return "View %s" % id

@app.route('/buy/<id>')
def buy_product(id):
    return "Buy %s" % id

First install Flask with pip install Flask or pip3 install Flask and then run application with flask run.

The purpose is to generate the following metrics to monitor product views and purchases:

# HELP view_total Product view
# TYPE view_total counter
view_total{product="product1"} 8
view_total{product="product2"} 2
# HELP buy_total Product buy
# TYPE buy_total counter
buy_total{product="product1"} 3

How to Generate Metrics

In order to generate metrics, we need HTTP endpoints that output text-based metrics. The default path is /metrics, but it can be modified. This can be implemented with a standard "route":

1 @app.route('/metrics')
2 def metrics():
3     metrics = ""
4     for id in view_metric:
5         metrics += 'view_total{product="%s"} %s\n'
6                        % (id, view_metric[id])
7     for id in buy_metric:
8         metrics += 'buy_total{product="%s"} %s\n'
9                        % (id, buy_metric[id])
10    return metrics

In this example, the view_metric and buy_metric variables contain a mapping between the product name and the count of views or purchases.

• line 1: We create a new HTTP endpoint with the path /metrics; this endpoint will be used by Prometheus.
• line 3: We initialize the result as an empty string
• lines 4 to 6: For each product, we generate a line with:
  • metric name: view
  • label: product=<product name>
  • value: the count of views; this value is fetched from view_metric
• lines 7 to 9: Same for product purchases

Maintain the Metrics Variable up-to-date

In this approach, view_metric and buy_metric need to be updated elsewhere in the code when the product is viewed or bought. So the first thing to implement is a variable or object that holds metric values within application:

1 view_metric = {}
2
3 @app.route('/view/<id>')
4 def view_product(id):
5     # Update metric
6     if product not in view_metric:
7         view_metric[id] = 1
8     else:
9         view_metric[id] += 1
10   return "View %s" % id

• line 1: a global variable that holds the total number of views for each product
• lines 7 and 9: the global variable is updated

The code for the metrics() function is really simple, it does not compute anything, but simply retrieves values from an existing variable that is kept up-to-date. So the complexity is in the view_product() function. Each time a product is "viewed", a piece of code is run to maintain the view_metric counter up-to-date.

On-demand Metrics

Sometimes, the cost of maintaining these kinds of variables is higher than computing the values on-the-fly when the metrics() function is called. With the default configuration, Prometheus queries the /metrics endpoint once every 30 seconds. So if the variable needs to be updated several times during this interval, it might be a good idea to compute metrics on demand:

1 @app.route('/metrics')
2 def metrics():
3     metrics = ""
4     res = query('SELECT product, count(*) FROM product_view'\
5                 'GROUP BY product')
6     for line in res:
7         metrics += 'view_total{product="%s"} %s\n'
8                     % (line[0], line[1])
9     return metrics

With this approach, we don't need to modify other parts of the code. Instead, we query the database to retrieve this information.
Note that, in this example, the metrics concerning the purchases are removed to improve readability.

Using the Prometheus Client Library

There is probably a library for your language that will take care of producing Prometheus text format.

This will provide Counter and Gauge objects to implements your metrics:

1  from prometheus_client import Counter
2
3  view_metric = Counter('view', 'Product view')
4
5  @app.route('/view/<id>')
6  def view_product(id):
7      # Update metric
8      view_metric.inc()
9      return "View %s" % id

• line 1: Loading the Prometheus client library
• line 3: Creation of Counter metrics with name and description
• line 8: Here the code is far more simple as we only need to call the inc() method of Counter object

Note that a _total suffix will be added to the metric name because it's a Counter and another metric with a _created suffix will contain the timestamp of the creation of the counter.

Adding Labels

So far, we haven’t handled labels. Let's see how we can add them now:

1 from prometheus_client import Counter
2 
3 view_metric = Counter('view', 'Product view', ['product'])
4
5 @app.route('/view/<id>')
6 def view_product(id):
7     # Update metric
8     view_metric.labels(product=id).inc()
9     return "View %s" % id

• line 3: an additional parameter defines the allowed labels for the view metric
• line 8: a call to labels() allows to set label values and thus select the time series that will be incremented

Finally, in the metrics() function, we just need to retrieve all the metrics in the Prometheus text format using the generate_latest() function:

1 from prometheus_client import generate_latest
2
3 @app.route('/metrics')
4 def metrics():
5     return generate_latest()

Here is a full example:

from flask import Flask
from prometheus_client import Counter, generate_latest

app = Flask(__name__)
view_metric = Counter('view', 'Product view', ['product'])
buy_metric = Counter('buy', 'Product buy', ['product'])

@app.route('/view/<id>')
def view_product(id):
    view_metric.labels(product=id).inc()
    return "View %s" % id

@app.route('/buy/<id>')
def buy_product(id):
    buy_metric.labels(product=id).inc()
    return "Buy %s" % id

@app.route('/metrics')
def metrics():
    return generate_latest()

Using Python Decorator

The python library also has some nice decorators.

For example, you can track the time spent in a function by using the @<metric>.time() decorator:

import time
import random
from prometheus_client import Summary

duration = Summary('duration_compute_seconds', 'Time spent in the compute() function')

@duration.time()
def compute():
    time.sleep(random.uniform(0, 10))

With a Counter, you can keep track of exceptions thrown in particular functions:

import time
import random
from prometheus_client import Counter

exception = Counter('compute_exception', 'Exception thrown in compute() function')

@exception.count_exceptions()
def compute():
    if random.uniform(0, 10) > 7:
        raise Exception("Random error")

On-demand Metrics with the Prometheus Library

The previously seen on-demand pattern can be implemented with the Prometheus Library by setting a callback function when defining the metric:

stock_metric = Counter('stock', 'Stock count')
stock_metric.set_function(compute_stock)
def compute_stock():
    res = query('SELECT count(*) FROM product_stock')
    for line in res:
        return line[0]

You can of course mix metrics managed with inc() and set() with other metrics using a callback function.

In the next blog post, we will see:

• how to visualize the application metrics using a Docker composition that includes Prometheus and Grafana
• how to leverage the new metrics in order to monitor the application in a Kubernetes cluster.