DEV Community

loading...

Structured logging with Serilog and Seq and ElasticSearch under Docker

Ali MSELMI
・6 min read

This blog post demonstrates Structured Logging with Serilog, Seq, ElasticSearch and kibana under Docker containers.
This post is a follow up on the beginner post I wrote on Serilog.

If you are a newbie on Serilog then I will recommend that you give a quick read to official Serilog page and this blog post.

Example application

The complete below example shows serilog structured logging in a containerized web application with microservices style using docker, with events sent to the Seq and elastic search as well as a date-stamped rolling log file with the use of available below sinks:

  • Serilog.Sinks.File
  • Serilog.Sinks.Http
  • Serilog.Sinks.Seq
  • Serilog.Sinks.ElasticSearch

Create a new empty solution

To start, we need create a new solution using Visual studio or your favorite IDE.
Create a new empty solution

Create a new web Application project

Create new Web application project like below:
Create a new web Application project

Install the core Serilog package and the File, Seq and ElasticSearch sinks

In Visual Studio, open the Package Manager Console and type:

Install-Package Autofac
Install-Package Autofac.Extensions.DependencyInjection
Install-Package Microsoft.AspNetCore.Mvc.NewtonsoftJson
Install-Package Microsoft.Extensions.DependencyInjection
Install-Package Microsoft.Extensions.Hosting
Install-Package Microsoft.Extensions.Logging
Install-Package Microsoft.Extensions.Options
Install-Package Newtonsoft.Json
Install-Package Serilog
Install-Package Serilog.AspNetCore
Install-Package Serilog.Extensions.Hosting
Install-Package Serilog.Extensions.Logging
Install-Package Serilog.Settings.AppSettings
Install-Package Serilog.Settings.Configuration
Install-Package Serilog.Sinks.ElasticSearch
Install-Package Serilog.Sinks.File
Install-Package Serilog.Sinks.Http
Install-Package Serilog.Sinks.Seq

Add the following code to Program.cs

Create GetConfigurationmethod

private static IConfiguration GetConfiguration()
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.Development.json", optional: true)
        .AddEnvironmentVariables();

    return builder.Build();
}

Create CreateSerilogLoggermethod

private static ILogger CreateSerilogLogger(IConfiguration configuration)
{
    var seqServerUrl = configuration["Serilog:SeqServerUrl"];
    return new LoggerConfiguration()
        .MinimumLevel.Verbose()
        .Enrich.WithProperty("ApplicationContext", AppName)
        .Enrich.FromLogContext()
        .WriteTo.File("catalog.api.log.txt", rollingInterval: RollingInterval.Day)
        .WriteTo.Elasticsearch().WriteTo.Elasticsearch(ConfigureElasticSink(configuration, "Development"))
        .WriteTo.Seq(string.IsNullOrWhiteSpace(seqServerUrl) ? "http://seq" : seqServerUrl)
        .CreateLogger();
}

Create ConfigureElasticSinkmethod

private static ElasticsearchSinkOptions ConfigureElasticSink(IConfiguration configuration, string environment)
{
    return new ElasticsearchSinkOptions(new Uri(configuration["Serilog:ElasticConfiguration"]))
    {
        BufferCleanPayload = (failingEvent, statuscode, exception) =>
        {
            dynamic e = JObject.Parse(failingEvent);
            return JsonConvert.SerializeObject(new Dictionary<string, object>()
                {
                    { "@timestamp",e["@timestamp"]},
                    { "level","Error"},
                    { "message","Error: "+e.message},
                    { "messageTemplate",e.messageTemplate},
                    { "failingStatusCode", statuscode},
                    { "failingException", exception}
                });
        },
        MinimumLogEventLevel = LogEventLevel.Verbose,
        AutoRegisterTemplate = true,
        AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
        CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage: true),
        IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}",
        EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | 
                               EmitEventFailureHandling.WriteToFailureSink |
                               EmitEventFailureHandling.RaiseCallback
    };
}

Create GetDefinedPorthelper method

private static int GetDefinedPort(IConfiguration config)
{
    var port = config.GetValue("PORT", 80);
    return port;
}

Create CreateHostBuildermethod

private static IWebHost CreateHostBuilder(IConfiguration configuration, string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .UseConfiguration(configuration)
        .CaptureStartupErrors(false)
        .ConfigureKestrel(options =>
        {
            var httpPort = GetDefinedPort(configuration);
            options.Listen(IPAddress.Any, httpPort, listenOptions =>
            {
                listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
            });
        })
        .UseStartup<Startup>()
        .UseContentRoot(Directory.GetCurrentDirectory())
        .UseSerilog()
        .Build();

Create Main method

public static readonly string Namespace = typeof(Program).Namespace;
public static readonly string AppName = Namespace.Substring(Namespace.LastIndexOf('.', Namespace.LastIndexOf('.') - 1) + 1);

public static int Main(string[] args)
{
    var configuration = GetConfiguration();
    Log.Logger = CreateSerilogLogger(configuration);
    try
    {
        Log.Information("Configuring web host ({ApplicationContext})", AppName);
        var host = CreateHostBuilder(configuration, args);
        Log.Information("Starting web host ({ApplicationContext})", AppName);
        host.Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Program terminated unexpectedly ({ApplicationContext})!", AppName);
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}

Add the following code to Startup.cs

Inject IConfigurationinto the constructure

public Startup(IConfiguration configuration)
{
    Configuration = configuration;
}
public IConfiguration Configuration { get; }

Create ConfigureServices method

public IServiceProvider ConfigureServices(IServiceCollection services)
{
    services.AddCustomMVC(Configuration);

    var container = new ContainerBuilder();
    container.Populate(services);

    return new AutofacServiceProvider(container.Build());
}

Create Configuremethod

public void Configure(IApplicationBuilder app, IWebHostEnvironment env, ILoggerFactory loggerFactory)
{
    var pathBase = Configuration["PATH_BASE"];

    if (!string.IsNullOrEmpty(pathBase))
    {
        loggerFactory.CreateLogger<Startup>().LogDebug("Using PATH BASE '{pathBase}'", pathBase);
        app.UsePathBase(pathBase);
    }

    app.UseCors("CorsPolicy");
    app.UseRouting();
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapDefaultControllerRoute();
        endpoints.MapControllers();
    });
}

Create the CustomExtensionMethodsextension method

public static class CustomExtensionMethods
{
    public static IServiceCollection AddCustomMVC(this IServiceCollection services, IConfiguration configuration)
    {
        services.AddControllers(options =>
        {

        }).AddNewtonsoftJson();

        services.AddCors(options =>
        {
            options.AddPolicy("CorsPolicy",
                builder => builder
                .SetIsOriginAllowed((host) => true)
                .AllowAnyMethod()
                .AllowAnyHeader()
                .AllowCredentials());
        });

        return services;
    }
}

Add appsettings.json / appsettings.Development.json

It's very important to use the container name as ElasticSearchurl and not http://localhost:9200

{
  "Serilog": {
    "SeqServerUrl": "http://seq",
    "LogstashgUrl": "http://locahost:8080",
    "ElasticConfiguration": "http://elasticsearch:9200",
    "MinimumLevel": {
          "Default": "Debug",
          "Override": {
            "Microsoft": "Debug",
            "CatalogAPI": "Debug",
            "MYSHOP": "Debug",
            "System": "Warning"
          }
        }
      }
  }

Add Docker support to our solution

Add Docker compose project to our solution

Add Docker compose project to our solution

Add Dockerfile to the project

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/core/sdk:aspnet:3.1-buster AS build
WORKDIR /src

COPY "Services/Catalog/Catalog.API/CatalogAPI.csproj" "Services/Catalog/Catalog.API/CatalogAPI.csproj"
COPY "docker-compose.dcproj" "docker-compose.dcproj"

COPY "NuGet.config" "NuGet.config"

RUN dotnet restore "MyShop.sln"

COPY . .
WORKDIR /src/Services/Catalog/Catalog.API
RUN dotnet publish --no-restore -c Release -o /app

FROM build AS publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "CatalogAPI.dll"]

Add docker-compose.yml to the docker project

version: '3.4'

services:

  seq:
    image: datalust/seq:latest

  catalog.api:
    image: ${REGISTRY:-myshop}/catalogapi
    build:
      context: .
      dockerfile: Services/Catalog/CatalogAPI/Dockerfile

networks:
  elastic:
    driver: bridge

volumes:
  elasticsearchdata:
    external: true

Add Docker docker-compose.overrides.yml to the docker project

version: '3.4'

services:

  seq:
    environment:
      - ACCEPT_EULA=Y
    ports:
      - "5340:80"

  elasticsearch:
    build:
      context: elk/elasticsearch/
    volumes:
      - ./elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"

  logstash:
    build:
      context: elk/logstash/
    volumes:
      - ./elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./elk/logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "8080:8080"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: elk/kibana/
    volumes:
      - ./elk/kibana/config/:/usr/share/kibana/config:ro
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  catalog.api:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://0.0.0.0:80
      - PORT=80
      - PATH_BASE=/catalog-api
    ports:
      - "4201:80"

Create Configuration for ElasticSearchand Kibana

Create the folder structure

Create "elk" folder in the root folder of the solution with following structure
Create the folder structure

Create elasticsearch configuration

Under elk/elasticsearch create new Dockerfileand new config folder like bellow:
Create elasticsearch configuration
Add the following code to the Dockerfile

# https://github.com/elastic/elasticsearch-docker
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2

# Add your elasticsearch plugins setup here
# Example: RUN elasticsearch-plugin install analysis-icu

Create new elasticsearch.yml under the config folder

---
## Default Elasticsearch configuration from elasticsearch-docker.
## from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node

Create kibana configuration

Under elk/kibana create new Dockerfileand new config folder like bellow:

Add the following code to the Dockerfile

# https://github.com/elastic/kibana-docker
FROM docker.elastic.co/kibana/kibana-oss:7.6.2

# Add your kibana plugins setup here
# Example: RUN kibana-plugin install <name|url>
Create new kibana.yml under the config folder

## Default Kibana configuration from kibana-docker.
## from https://github.com/elastic/kibana-docker/blob/master/build/kibana/config/kibana.yml
#
server.name: kibana
server.host: "0"
elasticsearch.hosts: "http://elasticsearch:9200"

Conclusion

We have seen how to configure .net core web application to log in Elasticsearch, Kibanaand Sequsing Serilog. This solution offers many advantages and I invite you to discover the range of possibilities on the Elastic and Seq websites.
The sources are here feel free to clone the solution and play with it.

Discussion (0)