DEV Community

Kim CH
Kim CH

Posted on • Updated on

Simplify observability .NET with OpenTelemetry Collector

#observability #dotnet #opentelemetry #collector

Overview

The Observability concept become the standard of almost system in recently. It helps team to troubleshoot what's happening inside the system. There are 3 pillars of Observability - Traces, Metrics and Logs.

Because of the various exporting ways, we have to consider one of these options when implementing

  • Support all type of exporting then toggle via settings. For example, only export to Zipkin if it's enabled
  • Or, just only export to Zipkin or Jaeger

Only one answer for the concerns

❓ Concerns

  • Is there any way that just only one export for multiple consumers? Or,
  • Is there any way that just only one export but change consumer without changing the code?

🌟 Only one answer

Objectives

  • Usability: Reasonable default configuration, supports popular protocols, runs and collects out of the box.
  • Performance: Highly stable and performant under varying loads and configurations.
  • Observability: An exemplar of an observable service.
  • Extensibility: Customizable without touching the core code.
  • Unification: Single codebase, deployable as an agent or collector with support for traces, metrics, and logs (future).
  • An image more than thousand words

Image description

💻 Let our hand dirty

👉 The below steps are just the showcase of using OTEL Collector within .NET 7. The full of implementation can be found at - .NET with OpenTelemetry Collector

👉 In which, we'll export the telemetry signals from application to OTEL Collector then they'll be exported to - Zipkin or Jaeger for tracings; Prometheus; and Loki for logs

Nuget packages Directory.Packages.props 


<!-- OpenTelemetry: Traces & Metrics -->
<ItemGroup>
    <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.4.0" />
    <PackageVersion Include="OpenTelemetry.Extensions.Hosting" Version="1.4.0" />
    <PackageVersion Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc9.14" />
</ItemGroup>

<!-- Serilog -->
<ItemGroup>
    <PackageVersion Include="Serilog.AspNetCore" Version="6.1.0" />
    <PackageVersion Include="Serilog.Enrichers.Context" Version="4.6.0" />
    <PackageVersion Include="Serilog.Sinks.OpenTelemetry" Version="1.0.0-dev-00129" />
    <PackageVersion Include="OpenTelemetry.Exporter.OpenTelemetryProtocol.Logs" Version="1.4.0-rc.4" />
</ItemGroup>
Enter fullscreen mode Exit fullscreen mode

Register OpenTelemetry, typically from Program.cs 


var builder = WebApplication.CreateBuilder(args);

builder.Host.AddSerilog();

builder.Services
       .AddOpenTelemetry()
       .AddTracing(observabilityOptions)
       .AddMetrics(observabilityOptions);
Enter fullscreen mode Exit fullscreen mode

Configure Tracings 

private static OpenTelemetryBuilder AddTracing(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
    builder.WithTracing(tracing =>
    {
       tracing
                .AddSource(observabilityOptions.ServiceName)
                .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(observabilityOptions.ServiceName))
                .SetErrorStatusOnException()
                .SetSampler(new AlwaysOnSampler())
                .AddAspNetCoreInstrumentation(options =>
                {
                    options.EnableGrpcAspNetCoreSupport = true;
                    options.RecordException = true;
                });

        /* Add more instrument here: MassTransit, NgSql ... */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */
        tracing
                .AddOtlpExporter(options =>
                {
                    options.Endpoint = observabilityOptions.CollectorUri;
                    options.ExportProcessorType = ExportProcessorType.Batch;
                    options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
                });
        });

        return builder;
}
Enter fullscreen mode Exit fullscreen mode

Configure for Metrics 

private static OpenTelemetryBuilder AddMetrics(this OpenTelemetryBuilder builder, ObservabilityOptions observabilityOptions)
{
        builder.WithMetrics(metrics =>
        {
            var meter = new Meter(observabilityOptions.ServiceName);

       metrics
                .AddMeter(meter.Name)
                .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(meter.Name))
                .AddAspNetCoreInstrumentation();

        /* Add more instrument here */

        /* ============== */
        /* Only export to OpenTelemetry collector */
        /* ============== */

        metrics
                .AddOtlpExporter(options =>
                {
                    options.Endpoint = observabilityOptions.CollectorUri;
                    options.ExportProcessorType = ExportProcessorType.Batch;
                    options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.Grpc;
                });
        });

        return builder;
}
Enter fullscreen mode Exit fullscreen mode

Configure Logs 

public static IHostBuilder AddSerilog(this IHostBuilder hostBuilder)
{
     hostBuilder
            .UseSerilog((context, provider, options) =>
            {
                var environment = context.HostingEnvironment.EnvironmentName;
                var configuration = context.Configuration;

                ObservabilityOptions observabilityOptions = new();

                configuration
                    .GetSection(nameof(ObservabilityOptions))
                    .Bind(observabilityOptions);

                var serilogSection = $"{nameof(ObservabilityOptions)}:{nameof(ObservabilityOptions)}:Serilog";

                options
                    .ReadFrom.Configuration(context.Configuration, serilogSection)
                    .Enrich.FromLogContext()
                    .Enrich.WithEnvironment(environment)
                    .Enrich.WithProperty("ApplicationName", observabilityOptions.ServiceName);

                 /* ============== */
                 /* Only export to OpenTelemetry collector */
                 /* ============== */
                 options.WriteTo.OpenTelemetry(cfg =>
                 {
                     cfg.Endpoint = $"{observabilityOptions.CollectorUrl}/v1/logs";
                     cfg.IncludedData = IncludedData.TraceIdField | IncludedData.SpanIdField;
                     cfg.ResourceAttributes = new Dictionary<string, object>
                                                {
                                                    {"service.name", observabilityOptions.ServiceName},
                                                    {"index", 10},
                                                    {"flag", true},
                                                    {"value", 3.14}
                                                };
                  });

        });
        return hostBuilder;
}
Enter fullscreen mode Exit fullscreen mode

The interesting here

1️⃣ - Refer to docker-compose.observability.yaml

Image description

2️⃣ - Refer to otel-collector.yaml to configure OTEL Collector

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  logging:
    loglevel: info

  prometheus:
    endpoint: 0.0.0.0:8889

  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"
    format: proto

  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    format: json
    labels:
      resource:
        service.name: "service_name"
        service.instance.id: "service_instance_id"

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, jaeger, zipkin]

    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, prometheus]

    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, loki]
Enter fullscreen mode Exit fullscreen mode

Let's 👍 OTEL Collector and take fully implementation at - .NET with OpenTelemetry Collector

Cheers!!! 🍻

Top comments (1)

Collapse
 
pvsunil profile image
Sunil • • Edited

Thank you for the well written article. However I am using an issue with Logging.

I am using Serilog in .Net Framework 4.8 to send logs to Opentelemetry. I am using Elastic as OpenTelemetry backend . I am able to send traces and but not logs. Please find my below code where it writes the log to the file but not sending logs to opentelemetry. Can somebody help?

    // TRACES ----- WORKING 
    _tracerProvider = Sdk.CreateTracerProviderBuilder()
   .AddAspNetInstrumentation()
   .AddHttpClientInstrumentation()
   .AddOtlpExporter(config =>
   {
       config.Endpoint = new Uri("https://abcd.es.io:443");
       config.Headers = "Authorization=ApiKey xyz";
   })
   .AddSource("SK")
   .SetResourceBuilder(
       ResourceBuilder.CreateDefault()
           .AddService(serviceName: "NLogger", serviceVersion: "1.0.0")
           .AddAttributes(resourceAttributes))
   .Build();

   var endpoint = "https://abcd.es.io:443/v1/logs";
   var protocol = OtlpProtocol.HttpProtobuf;

   // LOGGING --- NOT WORKING
   Log.Logger = new LoggerConfiguration()
   .MinimumLevel.Error()
   .WriteTo.Console()
   .WriteTo.File(@"c:\tt\1.txt")
   .WriteTo.OpenTelemetry(
   options => {
        options.Endpoint = endpoint;
        options.Protocol = protocol;
        options.IncludedData =
                IncludedData.SpanIdField
                | IncludedData.TraceIdField
                | IncludedData.MessageTemplateTextAttribute
                | IncludedData.MessageTemplateMD5HashAttribute;
        options.ResourceAttributes = new Dictionary<string, object>
        {
            ["service.name"] = "NLogger",
            ["index"] = 10,
            ["flag"] = true,
            ["pi"] = 3.14
         };
         options.Headers = new Dictionary<string, string>
         {
            ["Authorization"] = "Basic xyz", // user:abc123
         };
         options.BatchingOptions.BatchSizeLimit = 2;
         options.BatchingOptions.Period = TimeSpan.FromSeconds(2);
         options.BatchingOptions.QueueLimit = 10;
     })
     .CreateLogger();
Enter fullscreen mode Exit fullscreen mode

I tried "ApiKey" instead of "Basic" in the authorization header, but still it doesn't work. It writes the log successfully to the text file though. Please help.

Followed this page github.com/serilog/serilog-sinks-o... as reference.

Read next

kdhamric profile image

Cloud Native Observability and Cypress - an Unlikely Marriage

Ken Hamric -

me_janki profile image

Why .NET is Poised for Massive Growth in 2024

Janki Mehta -

lovelacecoding profile image

Learning C# in 2024? How to do it in 3 steps

Lou Creemers -

mhamzap10 profile image

How to Create A Word Document in C#

Mehr Muhammad Hamza -