Continuous Profiling in Kubernetes Using Pyroscope

Developers typically need to look at performance bottlenecks in production applications to determine the cause of the problem. To do this, you usually need information that can be collected through logs and code tools. Unfortunately, this approach is usually time-consuming and does not provide enough details about the underlying problem.

A modern and more advanced approach is to apply and use profiling techniques and tools that highlight the slowest application code, that is, the area consuming most of your resources.

In this blog post, we will be talking about continuous profiling, and then instrument a couple of microservices running on Kubernetes using an open source tool called Pyroscope.

What is profiling?

Code must be analyzed, debugged, and reviewed to determine the most effective way to make it run faster. Using a profiling tool to examine an application’s code helps us to locate and fix performance bottlenecks. This can quickly diagnose how an application performs and enable programmers to get down to core details of poor performance. The result is a streamlined codebase that decreases CPU/memory consumption and makes the user experience better!

Profiling is a program analysis that measures the memory, time complexity of a program or the frequency and duration of function calls. Profiling information serves to aid program optimization and performance. Profiler programs can track each individual line of code.

Continuous profiling

Continuous Profilers are used to make troubleshooting even faster and easier. Continuous Profilers are production code profilers that allow you to analyze code-level performance across your environment over time. As profiles are collected continuously, they can reveal the most resource-intensive features (or lines of code) quickly after new code is introduced. Optimization can reduce end-user delays and cloud provider accounts.

What continuous profilers are out there?

So, here’s a list of some of the profilers you may have come across:

Pyroscope

Pyroscope is an open source platform, consisting of server and agent. It allows the user to collect, store, and query the profiling data in a CPU and disk efficient way.

Parca

Parca collects, stores and makes profiles available to be queried over time. It is open source and can be deployed on production environments as Parca focuses on sampling profiling two main types of profiles: tracing and sampling.

Datadog

Datadog Continuous Profiler analyzes and compares code performance all the time and in any environment, including production. It pinpoints the hard to replicate production issues caused by inefficient code. Also has automated code profiling insights.

Google - Cloud Profiler

Cloud Profiler is a statistical, low-overhead profiler that continuously gathers CPU usage and memory-allocation information from your production applications. It has Actionable application profiling, Low-impact production Profilin and Broad Platform support.

Why use Pyroscope

Before we start exploring Pyroscope, let’s see how it is different from few of the other continuous profiling tools available in the market. DataDog and Google Cloud Profiler are widely used in the industry. As pointed out by one of the Reddit user, below are some of the reasons why Pyroscope is better compared to the other two.

Source: Comparison of Datadog, Google Cloud, and Pyroscope continuous profiling capabilities

Pyroscope, focuses on building a storage engine that is built specifically for profiling data to both store and query that data as efficiently as possible. It uses an agent-server model to send the profiles from applications to the Pyroscope server:

Source: How does Pyroscope Work?

Pyroscope allows for profilers from any language to send data to it and have that data efficiently stored by the storage engine. For example, Pyroscope has language specific agents for Go, Python, Ruby, eBPF, Java, .NET, PHP, and Rust.

Parca on the other hand takes a slightly different approach and relies on eBPF for compiled languages like C, C++, Go, etc. At the time of writing this article, support for other languages is work in progress. Similar to Pyroscope it can read from any pprof formatted profiles from HTTP endpoints as well.

Theoretically, since all these languages eventually compile down and run on the kernel, eBPF should work for any of these languages. However, in practice, if you actually run eBPF for interpreted languages like Python, function names are unreadable for humans in many cases. This is because symbols are not stored in those languages.

For this reason Pyroscope supports both language specific profilers as well as eBPF profiling. This comes at the expense of slightly more work to integrate language specific agents as compared to eBPF, which can just run at the kernel level. But it also comes with the benefit of having much more actionable and human-readable profiles.

How to install Pyroscope?

You can start the server followed by the agent no matter what you use, Docker, Linux, or are looking for Ruby or Go docs, Pyroscope covers you. Even if you aim for ten seconds or ten months of software profiling data, their custom-designed storage engine makes fast queries.

— Pyroscope website

We will use minikube for running a Kubernetes cluster. Create a cluster using minikube:

minikube start

Add the Helm chart repo:

helm repo add pyroscope-io https://pyroscope-io.github.io/helm-chart

Install Helm chart:

helm install pyroscope pyroscope-io/pyroscope --set service.type=NodePort

To check Pyroscope Helm chart installed successfully:

helm list

Check if Pyroscope is running:

kubectl get all

Now we have Pyroscope running in our Kubernetes cluster, we will proceed with the steps of using the application with it.
We will be using Google microservices, for this demo.

Integrating Google microservices demo with Pyroscope

We will modify our container images to use pyroscope binary. This binary will start our application and inject itself to monitor. You can refer more in this Pyroscope document.

We will be working on Python, Go, and .NET microservice from Google microservices for the demo. All the modifications are pushed to the fork of Google microservices on GitHub, let's take a look at those changes per serivce.

Note: To try out Pyroscope with Google microservices demo, you don't need to build the Docker images yourself. You can just apply the Kubernetes manifest as shown by the Obtaining the profiling data from microservice section.

Python

We will use the Email Service application which is written in Python.
Following changes in the Dockerfile are required, for using a Python application with Pyroscope.

COPY --from=pyroscope/pyroscope:latest /usr/bin/pyroscope /usr/bin/pyroscope
CMD [ "pyroscope", "exec", "python", "email_server.py" ]

After editing the Dockerfile, under the same folder, we proceeded with building and pushing the image.

docker build . -t beellzrocks/emailservice:latest
docker push beellzrocks/emailservice:latest

.NET

We will use the application Cart Service for .NET.
For using the .NET application with Pyroscope, the following changes in Dockerfile are required.

COPY --from=pyroscope/pyroscope:latest /usr/bin/pyroscope /usr/bin/pyroscope
ENTRYPOINT ["pyroscope", "exec", "-spy-name", "dotnetspy", "/app/cartservice"]

After editing the Dockerfile, we proceeded with building and pushing the image.

Go

We will take the Product Catalog Service application which is written in Go.
For using the Go application with Pyroscope following changes in server.go are required.

import (
  pyroscope "github.com/pyroscope-io/pyroscope/pkg/agent/profiler"
)

func main() {

  pyroscope.Start(pyroscope.Config{
    ApplicationName: os.Getenv("APPLICATION_NAME"),
    ServerAddress:   os.Getenv("SERVER_ADDRESS"),
  })
  // code here
)

After editing the server.go, we proceeded with building and pushing the image.

Obtaining the profiling data from microservice

We modified the Kubernetes manifests to use our images with Pyroscope.

The kubernetes-manifests.yaml file contains resources for all the applications. We edited it to use the images which we built in above steps i.e. emailservice, cartservice, productcatalogservice.

containers:
  - name: server
    image: beellzrocks/emailservice

When running Pyroscope in Kubernetes, we need to do following changes:

• Add SYS_PTRACE capability.
• Tell the agent about the location of Pyroscope server, and the application name using environment variables.

      containers:
      - name: server
        env:
        - name: PYROSCOPE_SERVER_ADDRESS # To change Pyroscope Server Port change the value
          value: "http://pyroscope:4040"
        - name: PYROSCOPE_APPLICATION_NAME # Application name shown in the UI
          value: "email.service" 
        securityContext:
          capabilities:
            add:
            - SYS_PTRACE

Now, to deploy all the services, you can apply the Kubernetes manifest to your cluster.

kubectl apply -f https://raw.githubusercontent.com/infracloudio/microservices-demo-dev/master/release/kubernetes-manifests.yaml

Get the service url for Pyroscope:

minikube service pyroscope

|-----------|-----------|-------------|---------------------------|
| NAMESPACE |   NAME    | TARGET PORT |            URL            |
|-----------|-----------|-------------|---------------------------|
| default   | pyroscope | http/4040   | http://192.168.49.2:30639 |
|-----------|-----------|-------------|---------------------------|
🎉  Opening service default/pyroscope in default browser

To access the Pyroscope UI, you can go to the URL: http://192.168.49.2:30639 (your's will be different).

Pyroscope UI with Pyroscope server CPU

As you can see in above screenshot, Pyroscope itself takes low CPU usage while storing the data locally. It uses Badger database to store data locally.

Pyroscope resource utilization

Monitoring Kubernetes pods is also important in the context of resource usage, utilization, and cost control. Pyroscope uses low resources with low overhead.

Pyroscope CPU utilization

Monitoring using Pyroscope

Pyroscope profiles the code using different agents depending upon the programming language. Here are some examples of the profiled application's flame graph using Pyroscope.

Pyroscope with Go Product Catalogue Service application

Pyroscope with .Net Cart application

Pyroscope with Python Email application

Conclusion

Continuous profiling performance is a crucial factor in fulfilling the expectation of end-users. And if performance issues occur, you must be ready to diagnose the issue before impacting the end-user experience.

Hence, keep optimizing your applications and fix the issues immediately to continue delivering super-fast application performance to the users using tools like Pyroscope. Pyroscope showcases a layer of visibility to help you understand how to improve the performance of your code in Production and reduce cloud infrastructure costs.

That’s a wrap folks :) Hope the article was informative and you enjoyed reading it. I’d love to hear your thoughts and experience - let’s connect and start a conversation on LinkedIn.