.Net 6 WebAPI - Intro to ElasticSearch & Kibana - Step by Step

#dotnet #elasticsearch #docker #api

In this article we will explore .NET 6 with ELK stack including ElasticSearch, Kibana and Serilog

What are we covering in this article

What is a Elastic Search
What is Kibana
Why use ElasticSearch
How everything fits together
Companies which use ELK
Ingredients
Code

You can watch the full video on Youtube

And you can find the link to the source code on GitHub
https://github.com/mohamadlawand087/Net6-ELK

What is Elastic Search

Elasticsearch is a distributed, open-source search and analytics engine, Elasticsearch allows you to store, search, and analyze huge volumes of data quickly and in near real-time and give back answers in milliseconds.

ElasticSearch utilisation

Search your way
Analyze at scale

What is Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elasticsearch Clusters.

The core feature of Kibana is data querying & analysis. In addition, Kibana’s visualization features allow you to visualize data in alternate ways

With Kibana, it is easy to understand big data, and you can quickly build and share dynamic dashboards that frame-out changes to the Elasticsearch query in real-time.

Why use ElasticSearch

E*asy to get started*: Just fire up a docker compose file containing ElasticSearch and Kibana containers and you’re ready to start logging and searching.
Easy to Query: ElasticSearch has a built-in full-text search engine that is based on Apache Lucene. Compared to other databases, Lucene is easy to query. Even non-technical people would be able to write common queries.
Fast: Querying a large SQL database can easily take 10 or 20 seconds. It’s quite common for similar queries on a large ElasticSearch database to return results in under 10 milliseconds.
Free: Well almost. The basic features are free, well mostly. If you need security and alerting features in Kibana, you can buy the commercial X-pack subscription for Kibana, or you can install some open source alternatives.
Full Text Search Engine
Analytical Engine
RESTful API: ElasticSearch has a RESTful API. Query results are returned in JSON which means results are easy to work with. Querying and inserting data via the RESTful API means it’s easy to use any programming language to work with ElasticSearch.
S*calable:* It’s easy to scale. Combined with the fact that it's open source means it's easy on the wallet too.

Companies which use ELK

Netflix
Ebay
Walmart
Adobe
LinkedIn
Stack Overflow

Code Time

We create a docker file

version: '3.1'

services:
  elasticsearch:
    container_name: els
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.1
    ports:
      - 9200:9200
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    environment:
      - xpack.monitoring.enabled=true
      - xpack.watcher.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    networks:
      - elastic

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.16.1
    ports:
      - 5601:5601
    depends_on:
      - els
    environment:
      - ELASTICSEARCH_URL=http://localhost:9200
    networks:
      - elastic

networks:
  elastic:
    driver: bridge

volumes:
  elasticsearch-data:

Execute the command

docker-compose up -d

To test they are running we visit the following URLs

ELS: http://localhost:9200
Kibana: http://localhost:5601

Now we create our web application

dotnet new webapi --no-https -n DotnetELK

Install the packages

dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Enrichers.Environment
dotnet add package Serilog.Sinks.Debug
dotnet add package Serilog.Sinks.Elasticsearch
dotnet add package Serilog.Exceptions
dotnet restore

Update appsettings.json

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Information",
        "System": "Warning"
      }
    }
  },
  "ElasticConfiguration": {
    "Uri": "http://localhost:9200"
  },
  "AllowedHosts": "*"
}

We need to update our program.cs

using System.Reflection;
using Serilog;
using Serilog.Sinks.Elasticsearch;

var builder = WebApplication.CreateBuilder(args);

ConfigureLogging();
builder.Host.UseSerilog();

// Add services to the container.
builder.Services.AddControllersWithViews();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
}
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Home}/{action=Index}/{id?}");

app.Run();

void ConfigureLogging()
{
    var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
    var configuration = new ConfigurationBuilder()
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile(
            $"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")}.json",
            optional: true)
        .Build();

    Log.Logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .Enrich.WithExceptionDetails()
        .WriteTo.Debug()
        .WriteTo.Console()
        .WriteTo.Elasticsearch(ConfigureElasticSink(configuration, environment))
        .Enrich.WithProperty("Environment", environment)
        .ReadFrom.Configuration(configuration)
        .CreateLogger();
}

ElasticsearchSinkOptions ConfigureElasticSink(IConfigurationRoot configuration, string environment)
{
    return new ElasticsearchSinkOptions(new Uri(configuration["ElasticConfiguration:Uri"]))
    {
        AutoRegisterTemplate = true,
        IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}"
    };
}

Now we run our application

dotnet run

Add integration

click on menu
Under Analytics click discover
create index pattern
in the Name type: dotnetelk-*
in timestamp field choose @timestamp
click on create index pattern
Now go back to discover and we can see all of the logs so far there

Now let us create a custom message, inside our controller we update the following

[HttpGet(Name = "GetWeatherForecast")]
public IEnumerable<WeatherForecast> Get()
{
    _logger.LogInformation("WeatherForecastController Get - this is a nice message to test the logs", DateTime.UtcNow);
    return Enumerable.Range(1, 5).Select(index => new WeatherForecast
    {
        Date = DateTime.Now.AddDays(index),
        TemperatureC = Random.Shared.Next(-20, 55),
        Summary = Summaries[Random.Shared.Next(Summaries.Length)]
    })
    .ToArray();
}

Now we can search for it by typing this in the search box

message: "WeatherForecastController Get"

Now let us test errors, inside our controller

try
{
    throw new Exception("Testing exceptions - ML");
}
catch (Exception ex)
{
    _logger.LogError(ex, "An unknown error occurred on the Index action of the HomeController");
}