Monitor an operator

Having an Kubernetes operator is quite cool, but if everything goes well. To track this and potentially adding alerts on it, we must have metrics.

And that's good, by default with the Golang Operator SDK, we already have a couple of metrics which are available.

Available metrics

The following list is an overview of all the metrics available. To see the full list, please check the Kubebuilder reference.

• controller_runtime_reconcile_total : Counter - Total number of reconciliations per controller.
• controller_runtime_reconcile_errors_total : Counter - Total number of reconciliation errors per controller.
• controller_runtime_reconcile_time_seconds : Histogram - Length of time per reconciliation per controller.
• controller_runtime_webhook_requests_total : Counter - Total number of admission requests by HTTP status code.

Add metrics

Even if the generic metrics from Kubebuilder are useful, they may not be enough for your use cases.

For this, there is something to do to easily add more metrics. With the package github.com/prometheus/client_golang/prometheus, create all the metrics you may need

import ( 
    "github.com/prometheus/client_golang/prometheus" 
)

var ( 
    goobers = prometheus.NewCounter( 
        prometheus.CounterOpts{ 
            Name: "goobers_total", 
            Help: "Number of goobers proccessed", 
        }, 
    ) 
    gooberFailures = prometheus.NewCounterVec( 
        prometheus.CounterOpts{ 
            Name: "goober_failures_total",
            Help: "Number of failed goobers", 
        }, []string{"label_1"}
    )
)

Then register them in the metric registry of the operator

import ( 
    "sigs.k8s.io/controller-runtime/pkg/metrics" 
)

func init() { 
    metrics.Registry.MustRegister(goobers, gooberFailures) 
}

Once it's done, congrats! You can use these metrics wherever you want!

goobers.Inc()
goobers.Add(2)

gooberFailures.WithLabelValues("toto").Inc()
gooberFailures.WithLabelValues("titi").Add(2)

Metrics with labels

As you can see in the previous example, if you want to create a metric with labels, you can use New...Vec. It will ask you to give a list of labels name, and then will ask you to use WithLabelValues to fill the value for each label when you will want to update a metric value. Quite useful if you always have the same labels.

Otherwise, you can use .With() to add all the labels you want when updating a metric value.

ex: .With(map[string]string{"toto": "titi"}).Inc()

At the end, an operator is a service like all the others and should be monitored to be sure that everything is going well in your stack. Like a lot of other things, Operator SDK and Kubebuilder help us a lot by simplifying all this process.

I hope it will help you and if you have any questions (there are not dumb questions) or some points are not clear for you, don't hesitate to add your question in the comments or to contact me directly on LinkedIn.