DEV Community

Ali MSELMI
Ali MSELMI

Posted on

Structured logging with Serilog and Seq and ElasticSearch under Docker

#aspnetcore #docker #serilog #elasticsearch

This blog post demonstrates Structured Logging with Serilog, Seq, ElasticSearch and kibana under Docker containers.
This post is a follow up on the beginner post I wrote on Serilog.

If you are a newbie on Serilog then I will recommend that you give a quick read to official Serilog page and this blog post.

Example application

The complete below example shows serilog structured logging in a containerized web application with microservices style using docker, with events sent to the Seq and elastic search as well as a date-stamped rolling log file with the use of available below sinks:

  • Serilog.Sinks.File
  • Serilog.Sinks.Http
  • Serilog.Sinks.Seq
  • Serilog.Sinks.ElasticSearch

Create a new empty solution

To start, we need create a new solution using Visual studio or your favorite IDE.
Create a new empty solution

Create a new web Application project

Create new Web application project like below:
Create a new web Application project

Install the core Serilog package and the File, Seq and ElasticSearch sinks

In Visual Studio, open the Package Manager Console and type:

Install-Package Autofac
Install-Package Autofac.Extensions.DependencyInjection
Install-Package Microsoft.AspNetCore.Mvc.NewtonsoftJson
Install-Package Microsoft.Extensions.DependencyInjection
Install-Package Microsoft.Extensions.Hosting
Install-Package Microsoft.Extensions.Logging
Install-Package Microsoft.Extensions.Options
Install-Package Newtonsoft.Json
Install-Package Serilog
Install-Package Serilog.AspNetCore
Install-Package Serilog.Extensions.Hosting
Install-Package Serilog.Extensions.Logging
Install-Package Serilog.Settings.AppSettings
Install-Package Serilog.Settings.Configuration
Install-Package Serilog.Sinks.ElasticSearch
Install-Package Serilog.Sinks.File
Install-Package Serilog.Sinks.Http
Install-Package Serilog.Sinks.Seq

Add the following code to Program.cs

Create GetConfigurationmethod 

private static IConfiguration GetConfiguration()
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(Directory.GetCurrentDirectory())
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile($"appsettings.Development.json", optional: true)
        .AddEnvironmentVariables();

    return builder.Build();
}
Enter fullscreen mode Exit fullscreen mode

Create CreateSerilogLoggermethod 

private static ILogger CreateSerilogLogger(IConfiguration configuration)
{
    var seqServerUrl = configuration["Serilog:SeqServerUrl"];
    return new LoggerConfiguration()
        .MinimumLevel.Verbose()
        .Enrich.WithProperty("ApplicationContext", AppName)
        .Enrich.FromLogContext()
        .WriteTo.File("catalog.api.log.txt", rollingInterval: RollingInterval.Day)
        .WriteTo.Elasticsearch().WriteTo.Elasticsearch(ConfigureElasticSink(configuration, "Development"))
        .WriteTo.Seq(string.IsNullOrWhiteSpace(seqServerUrl) ? "http://seq" : seqServerUrl)
        .CreateLogger();
}
Enter fullscreen mode Exit fullscreen mode

Create ConfigureElasticSinkmethod 

private static ElasticsearchSinkOptions ConfigureElasticSink(IConfiguration configuration, string environment)
{
    return new ElasticsearchSinkOptions(new Uri(configuration["Serilog:ElasticConfiguration"]))
    {
        BufferCleanPayload = (failingEvent, statuscode, exception) =>
        {
            dynamic e = JObject.Parse(failingEvent);
            return JsonConvert.SerializeObject(new Dictionary<string, object>()
                {
                    { "@timestamp",e["@timestamp"]},
                    { "level","Error"},
                    { "message","Error: "+e.message},
                    { "messageTemplate",e.messageTemplate},
                    { "failingStatusCode", statuscode},
                    { "failingException", exception}
                });
        },
        MinimumLogEventLevel = LogEventLevel.Verbose,
        AutoRegisterTemplate = true,
        AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
        CustomFormatter = new ExceptionAsObjectJsonFormatter(renderMessage: true),
        IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}",
        EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog | 
                               EmitEventFailureHandling.WriteToFailureSink |
                               EmitEventFailureHandling.RaiseCallback
    };
}
Enter fullscreen mode Exit fullscreen mode

Create GetDefinedPorthelper method 

private static int GetDefinedPort(IConfiguration config)
{
    var port = config.GetValue("PORT", 80);
    return port;
}
Enter fullscreen mode Exit fullscreen mode

Create CreateHostBuildermethod 

private static IWebHost CreateHostBuilder(IConfiguration configuration, string[] args) =>
    WebHost.CreateDefaultBuilder(args)
        .UseConfiguration(configuration)
        .CaptureStartupErrors(false)
        .ConfigureKestrel(options =>
        {
            var httpPort = GetDefinedPort(configuration);
            options.Listen(IPAddress.Any, httpPort, listenOptions =>
            {
                listenOptions.Protocols = HttpProtocols.Http1AndHttp2;
            });
        })
        .UseStartup<Startup>()
        .UseContentRoot(Directory.GetCurrentDirectory())
        .UseSerilog()
        .Build();
Enter fullscreen mode Exit fullscreen mode

Create Main method 

public static readonly string Namespace = typeof(Program).Namespace;
public static readonly string AppName = Namespace.Substring(Namespace.LastIndexOf('.', Namespace.LastIndexOf('.') - 1) + 1);

public static int Main(string[] args)
{
    var configuration = GetConfiguration();
    Log.Logger = CreateSerilogLogger(configuration);
    try
    {
        Log.Information("Configuring web host ({ApplicationContext})", AppName);
        var host = CreateHostBuilder(configuration, args);
        Log.Information("Starting web host ({ApplicationContext})", AppName);
        host.Run();
        return 0;
    }
    catch (Exception ex)
    {
        Log.Fatal(ex, "Program terminated unexpectedly ({ApplicationContext})!", AppName);
        return 1;
    }
    finally
    {
        Log.CloseAndFlush();
    }
}
Enter fullscreen mode Exit fullscreen mode

Add the following code to Startup.cs

Inject IConfigurationinto the constructure 

public Startup(IConfiguration configuration)
{
    Configuration = configuration;
}
public IConfiguration Configuration { get; }
Enter fullscreen mode Exit fullscreen mode

Create ConfigureServices method 

public IServiceProvider ConfigureServices(IServiceCollection services)
{
    services.AddCustomMVC(Configuration);

    var container = new ContainerBuilder();
    container.Populate(services);

    return new AutofacServiceProvider(container.Build());
}
Enter fullscreen mode Exit fullscreen mode

Create Configuremethod 

public void Configure(IApplicationBuilder app, IWebHostEnvironment env, ILoggerFactory loggerFactory)
{
    var pathBase = Configuration["PATH_BASE"];

    if (!string.IsNullOrEmpty(pathBase))
    {
        loggerFactory.CreateLogger<Startup>().LogDebug("Using PATH BASE '{pathBase}'", pathBase);
        app.UsePathBase(pathBase);
    }

    app.UseCors("CorsPolicy");
    app.UseRouting();
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapDefaultControllerRoute();
        endpoints.MapControllers();
    });
}
Enter fullscreen mode Exit fullscreen mode

Create the CustomExtensionMethodsextension method 

public static class CustomExtensionMethods
{
    public static IServiceCollection AddCustomMVC(this IServiceCollection services, IConfiguration configuration)
    {
        services.AddControllers(options =>
        {

        }).AddNewtonsoftJson();

        services.AddCors(options =>
        {
            options.AddPolicy("CorsPolicy",
                builder => builder
                .SetIsOriginAllowed((host) => true)
                .AllowAnyMethod()
                .AllowAnyHeader()
                .AllowCredentials());
        });

        return services;
    }
}
Enter fullscreen mode Exit fullscreen mode

Add appsettings.json / appsettings.Development.json

It's very important to use the container name as ElasticSearchurl and not http://localhost:9200

{
  "Serilog": {
    "SeqServerUrl": "http://seq",
    "LogstashgUrl": "http://locahost:8080",
    "ElasticConfiguration": "http://elasticsearch:9200",
    "MinimumLevel": {
          "Default": "Debug",
          "Override": {
            "Microsoft": "Debug",
            "CatalogAPI": "Debug",
            "MYSHOP": "Debug",
            "System": "Warning"
          }
        }
      }
  }
Enter fullscreen mode Exit fullscreen mode

Add Docker support to our solution

Add Docker compose project to our solution

Add Docker compose project to our solution

Add Dockerfile to the project 

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1-buster-slim AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/core/sdk:aspnet:3.1-buster AS build
WORKDIR /src

COPY "Services/Catalog/Catalog.API/CatalogAPI.csproj" "Services/Catalog/Catalog.API/CatalogAPI.csproj"
COPY "docker-compose.dcproj" "docker-compose.dcproj"

COPY "NuGet.config" "NuGet.config"

RUN dotnet restore "MyShop.sln"

COPY . .
WORKDIR /src/Services/Catalog/Catalog.API
RUN dotnet publish --no-restore -c Release -o /app

FROM build AS publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "CatalogAPI.dll"]
Enter fullscreen mode Exit fullscreen mode

Add docker-compose.yml to the docker project 

version: '3.4'

services:

  seq:
    image: datalust/seq:latest

  catalog.api:
    image: ${REGISTRY:-myshop}/catalogapi
    build:
      context: .
      dockerfile: Services/Catalog/CatalogAPI/Dockerfile

networks:
  elastic:
    driver: bridge

volumes:
  elasticsearchdata:
    external: true
Enter fullscreen mode Exit fullscreen mode

Add Docker docker-compose.overrides.yml to the docker project 

version: '3.4'

services:

  seq:
    environment:
      - ACCEPT_EULA=Y
    ports:
      - "5340:80"

  elasticsearch:
    build:
      context: elk/elasticsearch/
    volumes:
      - ./elk/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"

  logstash:
    build:
      context: elk/logstash/
    volumes:
      - ./elk/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./elk/logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "8080:8080"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    depends_on:
      - elasticsearch

  kibana:
    build:
      context: elk/kibana/
    volumes:
      - ./elk/kibana/config/:/usr/share/kibana/config:ro
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

  catalog.api:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=http://0.0.0.0:80
      - PORT=80
      - PATH_BASE=/catalog-api
    ports:
      - "4201:80"
Enter fullscreen mode Exit fullscreen mode

Create Configuration for ElasticSearchand Kibana

Create the folder structure

Create "elk" folder in the root folder of the solution with following structure
Create the folder structure

Create elasticsearch configuration

Under elk/elasticsearch create new Dockerfileand new config folder like bellow:
Create elasticsearch configuration
Add the following code to the Dockerfile

# https://github.com/elastic/elasticsearch-docker
FROM docker.elastic.co/elasticsearch/elasticsearch-oss:7.6.2

# Add your elasticsearch plugins setup here
# Example: RUN elasticsearch-plugin install analysis-icu
Enter fullscreen mode Exit fullscreen mode

Create new elasticsearch.yml under the config folder

---
## Default Elasticsearch configuration from elasticsearch-docker.
## from https://github.com/elastic/elasticsearch-docker/blob/master/build/elasticsearch/elasticsearch.yml
#
cluster.name: "docker-cluster"
network.host: 0.0.0.0

# minimum_master_nodes need to be explicitly set when bound on a public IP
# set to 1 to allow single node clusters
# Details: https://github.com/elastic/elasticsearch/pull/17288
discovery.zen.minimum_master_nodes: 1

## Use single node discovery in order to disable production mode and avoid bootstrap checks
## see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
#
discovery.type: single-node
Enter fullscreen mode Exit fullscreen mode

Create kibana configuration

Under elk/kibana create new Dockerfileand new config folder like bellow:

Add the following code to the Dockerfile

# https://github.com/elastic/kibana-docker
FROM docker.elastic.co/kibana/kibana-oss:7.6.2

# Add your kibana plugins setup here
# Example: RUN kibana-plugin install <name|url>
Create new kibana.yml under the config folder

## Default Kibana configuration from kibana-docker.
## from https://github.com/elastic/kibana-docker/blob/master/build/kibana/config/kibana.yml
#
server.name: kibana
server.host: "0"
elasticsearch.hosts: "http://elasticsearch:9200"
Enter fullscreen mode Exit fullscreen mode

Conclusion

We have seen how to configure .net core web application to log in Elasticsearch, Kibanaand Sequsing Serilog. This solution offers many advantages and I invite you to discover the range of possibilities on the Elastic and Seq websites.
The sources are here feel free to clone the solution and play with it.

Latest comments (1)

Collapse
 
moslemhady profile image
Moslem •

In serilog configuration, there is a property called "LogstashgUrl". Is this a typo? shouldn't it be "LogstashUrl"? or does it mean something I dont know?!

Read next

pradumnasaraf profile image

How to collaborate better with your team on Localhost

Pradumna Saraf -

massivebrains profile image

Use same Dockerfile for Dev & Production

Segun Olaiya -

plv8docker profile image

PostgreSQL 16: A Deep Dive and Our New Docker Image

plv8-docker -

abhirockzz profile image

Run and test DynamoDB applications locally using Docker and Testcontainers

Abhishek Gupta -