In this guide, you will learn hands-on how to create and visualize traces with OpenTelemetry Go without prior knowledge.
We will start with creating a simple to-do app that uses Mongo and the Gin framework. Then, we will send tracing data to Jaeger Tracing and to Aspecto for visualization. You can find all the relevant files in this Github repository.
What to Expect
Intro to OpenTelemetry
OpenTelemetry is a collection of APIs and SDKs that allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).
It is a CNCF community-driven open-source project (Cloud Native Computing Foundation, the folks in charge of Kubernetes).
In a cloud-native environment, we use OpenTelemetry (OTel for short) to gather data from our system operations and events. In other words, to instrument our distributed services. This data enables us to understand and investigate our software’s behavior and troubleshoot performance issues and errors.
OpenTelemetry serves as a standard observability framework that captures all data under a single specification. It provides several components, including:
- APIs and SDKs per programming language for generating telemetry
- The OpenTelemetry Collector; receives, processes, and exports telemetry data to different destinations.
- OTLP protocol for shipping telemetry data
To get a deeper understanding of this technology, including its structure and the primary motivation, visit this guide.
For this OpenTelemetry Golang guide, here are the terms you need to know:
Span: The most basic unit. A span represents an event in our system (e.g., an HTTP request or a database operation that spans over time). A span would usually be the parent of another span, its child, or both.
Trace: ‘Call-stacks’ for distributed services. Traces represent a tree of spans connected in a child/parent relationship. Traces specify the progression of requests across different services and components in our app (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.
Exporter: Once we create a span, the exporter handles sending the data to our backend (e.g., in memory, Jaeger Tracing, or console output)
Context propagation – The mechanism that allows us to correlate events across distributed services. Context is referred to as the metadata we collect and transfer*. Propagation* is how the context is packaged and transferred across services, often via HTTP headers. Context propagation is one of the areas where OpenTelemetry shines.
Instrumentation – instrumentation libraries gather the data and generate spans based on different libraries in our applications (Kafka, Mongo, Gin, etc.).
There are two ways to instrument our app – manually and automatically:
- Auto instrumentation: Automatically create spans from the application libraries we use with ready-to-use OpenTelemetry libraries.
- Manual instrumentation: Manually add code to your application to define the beginning and end of each span and the payload.
To understand more of the OpenTelemetry jargon, visit the official documentation.
Hello world: OpenTelemetry Go example
We will start by creating our to-do service and installing two libraries (Gin and Mongo) to understand how instrumentations work.
Step 1: Create main.go file for our to-do app
1) Install Gin and Mongo-driver
go get -u github.com/gin-gonic/gin
go get go.mongodb.org/mongo-driver/mongo
2) Set up gin and mongo to listen on “/todo”
3) Create some to-do’s to seed Mongo
package main
import (
"context"
"net/http"
"github.com/gin-gonic/gin"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
var client *mongo.Client
func main() {
connectMongo()
setupWebServer()
}
func connectMongo() {
opts := options.Client()
opts.ApplyURI("mongodb://localhost:27017")
client, _ = mongo.Connect(context.Background(), opts)
//Seed the database with todo's
docs := []interface{}{
bson.D{{"id", "1"}, {"title", "Buy groceries"}},
bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
}
client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
func setupWebServer() {
r := gin.Default()
r.GET("/todo", func(c *gin.Context) {
collection := client.Database("todo").Collection("todos")
//Important: Make sure to pass c.Request.Context() as the context and not c itself - TBD
cur, findErr := collection.Find(c.Request.Context(), bson.D{})
if findErr != nil {
c.AbortWithError(500, findErr)
return
}
results := make([]interface{}, 0)
curErr := cur.All(c, &results)
if curErr != nil {
c.AbortWithError(500, curErr)
return
}
c.JSON(http.StatusOK, results)
})
_ = r.Run(":8080")
}
Now that our small todo app is ready, let’s introduce OpenTelemetry.
Step 2: Install OpenTelemetry Go
We will be configuring OpenTelemetry to instrument our Go app.
1) To install the OTel SDK, run:
go get go.opentelemetry.io/otel /
go.opentelemetry.io/otel/sdk /
2) Instrument our Gin and Mongo libraries to generate traces.
3) Gin instrumentation: Install otelgin
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin
4) Mongo instrumentation: Install otelmongo
go get go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo
Gin instrumentation: gin.Context
We previously discussed the idea of context propagation – the way to transfer metadata between distributed services to correlate events in our system.
The Gin framework has its own type gin.Context
which gets passed as a parameter to an HTTP handler. However, the context that should be passed down to the mongo operations is the standard Go library Context object, available in gin.Context.Request.Context
.
//Make sure to pass c.Request.Context() as the context and not c itself
cur, findErr := collection.Find(c.Request.Context(), bson.D{})
So make sure that you pass the Context to the mongodb operation. Check out this issue for more info.
We now have our todo app ready and instrumented. It’s time to utilize OpenTelemetry to its full potential. Our ability to visualize traces is where the true troubleshooting power of this technology comes into play.
For visualization, we’ll be using the open-source Jaeger Tracing and Aspecto.
Visualization with Jaeger and Aspecto
The setup to export traces to Jaeger or Aspecto is relatively similar. Follow along with the Jaeger setup, then switch to Aspecto by changing a few lines of code.
OpenTelemetry Go and Jaeger Tracing: Export traces to Jaeger
Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.
Here’s what the setup looks like:
1) Install the Jaeger exporter
go get go.opentelemetry.io/otel/exporters/jaeger
2) Create a tracing folder and a jaeger.go file
3) Add the following code to the file
package tracing
import (
"go.opentelemetry.io/otel/exporters/jaeger"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)
func JaegerTraceProvider() (*sdktrace.TracerProvider, error) {
exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint("http://localhost:14268/api/traces")))
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp),
sdktrace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceNameKey.String("todo-service"),
semconv.DeploymentEnvironmentKey.String("production"),
)),
)
return tp, nil
}
4) Go back to the main.go file and modify our code to use the JaegerTraceProvider
function we just created
func main() {
tp, tpErr := tracing.JaegerTraceProvider()
if tpErr != nil {
log.Fatal(tpErr)
}
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
connectMongo()
setupWebServer()
}
Next, we are going to hook up the instrumentations we installed.
5) Add the Mongo instrumentation. In our connectMongo
function by adding this line
opts.Monitor = otelmongo.NewMonitor()
The function shold look like this
func connectMongo() {
opts := options.Client()
//Mongo OpenTelemetry instrumentation
opts.Monitor = otelmongo.NewMonitor()
opts.ApplyURI("mongodb://localhost:27017")
client, _ = mongo.Connect(context.Background(), opts)
//Seed the database with some todo's
docs := []interface{}{
bson.D{{"id", "1"}, {"title", "Buy groceries"}},
bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
}
client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
Now, add the Gin instrumentation.
6) Go to the startWebServer
function and add this line right after we create the gin instance
r.Use(otelgin.Middleware("todo-service"))
The function should look like this
func startWebServer() {
r := gin.Default()
//Gin OpenTelemetry instrumentation
r.Use(otelgin.Middleware("todo-service"))
r.GET("/todo", func(c *gin.Context) {
collection := client.Database("todo").Collection("todos")
//make sure to pass c.Request.Context() as the context and not c itself
cur, findErr := collection.Find(c.Request.Context(), bson.D{})
if findErr != nil {
c.AbortWithError(500, findErr)
return
}
results := make([]interface{}, 0)
curErr := cur.All(c, &results)
if curErr != nil {
c.AbortWithError(500, curErr)
return
}
c.JSON(http.StatusOK, results)
})
_ = r.Run(":8080")
}
Below is the complete main.go file. Now we’re finally ready to export to Jaeger.
package main
import (
"context"
"log"
"net/http"
"github.com/aspecto-io/opentelemerty-examples/tracing"
"github.com/gin-gonic/gin"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
"go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/propagation"
)
var client *mongo.Client
func main() { //Export traces to Jaeger
tp, tpErr := tracing.JaegerTraceProvider()
if tpErr != nil {
log.Fatal(tpErr)
}
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
connectMongo()
startWebServer()
}
func connectMongo() {
opts := options.Client()
//Mongo OpenTelemetry instrumentation
opts.Monitor = otelmongo.NewMonitor()
opts.ApplyURI("mongodb://localhost:27017")
client, _ = mongo.Connect(context.Background(), opts)
//Seed the database with some todo's
docs := []interface{}{
bson.D{{"id", "1"}, {"title", "Buy groceries"}},
bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
}
client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
func startWebServer() {
r := gin.Default()
//gin OpenTelemetry instrumentation
r.Use(otelgin.Middleware("todo-service"))
r.GET("/todo", func(c *gin.Context) {
collection := client.Database("todo").Collection("todos")
//Make sure to pass c.Request.Context() as the context and not c itself
cur, findErr := collection.Find(c.Request.Context(), bson.D{})
if findErr != nil {
c.AbortWithError(500, findErr)
return
}
results := make([]interface{}, 0)
curErr := cur.All(c, &results)
if curErr != nil {
c.AbortWithError(500, curErr)
return
}
c.JSON(http.StatusOK, results)
})
_ = r.Run(":8080")
}
Export traces to Jaeger
- Run the todo-service with go run main.go
- Make an HTTP GET request to localhost:8080/todo to generate some traces in Go
- Open Jaeger at http://localhost:16686/search to view those traces
You can now see the Jaeger UI. Select todo-service and click on Find traces. You should see your trace on the right:
By clicking the trace, you can drill down and see more details about it that allow you to further investigate on your own:
Visualization with OpenTelemetry Go and Aspecto
So now you know the basic concepts of OpenTelemetry. You used it on your own to instrument your Go libraries, create traces and export the data to Jaeger for visualization.
Let’s take our visualization capabilities to the next level with Aspecto.
Exporting and visualizing our data in Aspecto is easy (we will see that in a moment). You can try it yourself with the free-forever plan that has no limited features. Give this Live Playground a try to get a better idea.
This is how we do it:
- Create a new free account at www.aspecto.io
- Install the otlp exporter by running
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
- Create a file called aspecto.go in your tracing folder and add the code below
- Replace the {ASPECTO_AUTH} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens)
func AspectoTraceProvider() (*sdktrace.TracerProvider, error) {
exp, err := otlptracegrpc.New(context.Background(),
otlptracegrpc.WithEndpoint("collector.aspecto.io:4317"),
otlptracegrpc.WithHeaders(map[string]string{
"Authorization": "<ADD YOUR TOKEN HERE>",
}))
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp),
sdktrace.WithResource(resource.NewWithAttributes(
semconv.SchemaURL,
semconv.ServiceNameKey.String("todo-service"),
semconv.DeploymentEnvironmentKey.String("production"),
)),
)
return tp, nil
}
- Back in the main.go file, replace the
JaegerTraceProvider
withAspectoTraceProvider
.
The main function should now look like this
func main() { //Export traces to Aspecto
tp, tpErr := tracing.AspectoTraceProvider()
if tpErr != nil {
log.Fatal(tpErr)
}
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
connectMongo()
startWebServer()}
Exporting traces to Aspecto
- Run the todo-service with go run main.go
- Make an HTTP GET request to localhost:8080/todo to generate some traces
- Navigate to app.aspecto.io to view the traces
We also sent a few traces with errors to see what it looks like. Our traces should look something like this:
In the Trace Search view, using the left filters pane, simply filter out by the service name (you can also filter by error, HTTP method, and much more).
If we drill down into one of these traces, we can see in more detail how long each request took and clear visualization of the entire workflow.
In more complex workflows, traces would look something like the below:
That’s all folks! We hope this guide was informative and easy to follow. You can find all files ready to use in our Github repository.
If you have any questions or issues, feel free to reach out to us via our chat.
To learn more about it, check out our OpenTelemetry Bootcamp. It’s a 6 episode YouTube series that covers OpenTelemetry from zero to 100 including the implementation of opentelemetry in production, security, sampling, and more. Completely free and vendor-neutral.
Follow these guides below, if you want to install OpenTelemetry in more officially supported languages:
Top comments (0)