Tracing On Kubernetes

Observability comes in all shapes and sizes. It could be anything from an entire monitoring stack that gives you everything from alerts and APM to particular tools that give you bits and pieces of the stack. Outside of metrics and logging, there’s the component that tells you how an application is performing.

That component is tracing.

In this blog post, you’re going to learn both from a theoretical perspective and a hands-on perspective why tracing matters, how to think about it on Kubernetes, and how to implement it with Lumigo.

Prerequisites

If you’d like to follow along with this blog post from a hands-on perspective, you should have the following:

1. A Kubernetes cluster running in EKS.

If you don’t have your own cluster, there’s a section in this blog post that’ll show you the demo environment.

Tracing

You get alerts if an application fails and if the system that the application is running on fails. You get logs about the application and the system.

But when do you truly ever see and understand how an application is performing in real-time or over time in a forecasted fashion? Engineers hear a lot about overall monitoring, metrics, and alerting, but there’s arguably an even more important piece to the puzzle - tracing.

Tracing shows you, from an end-to-end perspective, how an application is performing in real-time or over a particular time (typically called history or a forecast). When you hear “end-to-end”, it means how the application itself is performing and the performance between the route that it’s taking. For example, an application itself (as in the code running inside of it) can be performing, but is the performance in terms of where it’s supposed to be going (like to a database, another app, a service, etc.) performing as expected? The whole idea around “end-to-end” is wherever that application is supposed to be running and where it’s going matters. Tracing does this by tracking application requests as they go from frontend to backend (databases, services, etc.) or even frontend to middleware depending on your stack.

Regardless of whether you’re using microservices or monolithic stacks, you can still use tracing.

From a pure application perspective, you’ll typically see a tracing tool embedded in the code. For example, if you’re writing Go code, you may implement libraries and functions into your application stack for the tracing tool to collect the data it needs.

In short: Tracing allows to to see how services interact with each other.

You may sometimes hear tracing called Distributed Tracing as well.

Tracing On Kubernetes

Within Kubernetes, you can implement tracing in a few ways:

1. Via the application code like with any other tracing implementation.
2. Directly on Kubernetes.

If you go with option one, the implementation won’t be different from the explanation given in the previous section. The tracing libraries/functions would be embedded in the application code. The only difference is that app code may be bundled into a container image and deployed to Kubernetes.

The second option is deploying the tracing tool directly on Kubernetes and have it running as an Operator, Pods, etc…

Truth be told, the section option is way better from a configuration perspective for two reasons. Number one, you don’t have to worry about putting the tracing code into your application code. This saves you headaches later if you want to take it out. Number two, if you deploy the tracing tool to Kubernetes and it can trace any Pods running on the cluster, that means you don’t have to put the tracing code into the application code which will save you a ton of time and configuration. Imagine if you have twenty app stacks that need the tracing tool code. You’d have to put that code into the app stacks one by one and manage it one by one. If you deploy the tracing tool to Kubernetes instead, you don’t have to do any of that management.

Lumigo - Quick Setup

Now that you know a bit about tracing and how to think about implementing it on Kubernetes, let’s see how we can use Lumigo to get the job done.

First things things, you’ll need to set up Lumigo. Luckily, it’s a SaaS, so you literally just have to click the Login button, sign up with your email, and start using it for free.

You can do so at this link: https://lumigo.io/

Implementing Lumigo In Kubernetes

Once you have an account, you can set up Lumigo with Helm.

First, you’ll have to add the Helm Chart and ensure you specify the name of your cluster.

helm repo add lumigo https://lumigo-io.github.io/lumigo-kubernetes-operator \
&& helm install lumigo lumigo/lumigo-operator \
--namespace lumigo-system --create-namespace \
--set cluster.name=NAME_OF_YOUR_CLUSTER

Next, create a Kubernetes Secret with the token you can receive from Lumigo’s portal. This will give you the ability to create the connection in a secure way from your Kubernetes cluster to Lumigo.

kubectl create secret generic \
--namespace NAMESPACE_YOU_WANT_TO_TRACE lumigo-credentials \
--from-literal token=TOKEN_FROM_LUMIGO

Lastly, use the Lumigo Operator to connect your Kubernetes cluster to Lumigo. The connection is done via the operator and the authentication/authorization is done via the secret you created in the previous section.

echo '{
      "apiVersion": "operator.lumigo.io/v1alpha1",
      "kind": "Lumigo",
      "metadata": {
        "name": "lumigo"
      },
      "spec": {
        "lumigoToken": {
          "secretRef": {
            "name": "lumigo-credentials",
            "key": "token"
          } 
        }
      }
    }' | kubectl apply -f - --namespace NAMESPACE_YOU_WANT_TO_TRACE

Using The Demo Environment

If you don’t have a Kubernetes cluster via EKS, you can use the demo environment in one of two ways:

1. Via the Sandbox.
2. Via the Demo button on your Lumigo portal.

To use the Sandbox environment, click on the following link, scroll down, and click the yellow T*ry in sandbox* button.

https://lumigo.io/

From your portal, you can toggle the Demo button.

Tracing On Lumigo

Once Kubernetes is connected or you decide to go with the demo environment, you can start seeing the tracing information.

Clicking on the Kubernetes button on the left pain, you can see any workloads that are running with the success and failure rates based on request.

If you click on one of the workloads, you can access the logs.

You can also see live interactions which will show you tracing information in real-time.

Another way (probably my favorite way) to look at tracing is with the System Map. The System Map tells you what services/apps are talking to other services/apps along with any errors, duration, cost, and memory information.

You can also see the communication directionally via the System Map.

The cool thing about Lumigo is it can also be used for standalone containerized applications running in ECS as well. Regardless of if you’re running containerized applications in ECS or EKS, you can the tracing information you need.