DEV Community

Cover image for .Net 6 WebAPI - Intro to ElasticSearch & Kibana  - Step by Step
Mohamad Lawand
Mohamad Lawand

Posted on

.Net 6 WebAPI - Intro to ElasticSearch & Kibana - Step by Step

In this article we will explore .NET 6 with ELK stack including ElasticSearch, Kibana and Serilog

What are we covering in this article

  • What is a Elastic Search
  • What is Kibana
  • Why use ElasticSearch
  • How everything fits together
  • Companies which use ELK
  • Ingredients
  • Code

You can watch the full video on Youtube

And you can find the link to the source code on GitHub
https://github.com/mohamadlawand087/Net6-ELK

What is Elastic Search

Elasticsearch is a distributed, open-source search and analytics engine, Elasticsearch allows you to store, search, and analyze huge volumes of data quickly and in near real-time and give back answers in milliseconds.

ElasticSearch utilisation

  • Search your way
  • Analyze at scale

What is Kibana

Kibana is a visual interface tool that allows you to explore, visualize, and build a dashboard over the log data massed in Elasticsearch Clusters.

The core feature of Kibana is data querying & analysis. In addition, Kibana’s visualization features allow you to visualize data in alternate ways

With Kibana, it is easy to understand big data, and you can quickly build and share dynamic dashboards that frame-out changes to the Elasticsearch query in real-time.

Why use ElasticSearch

  • E*asy to get started*: Just fire up a docker compose file containing ElasticSearch and Kibana containers and you’re ready to start logging and searching.
  • Easy to Query: ElasticSearch has a built-in full-text search engine that is based on Apache Lucene. Compared to other databases, Lucene is easy to query. Even non-technical people would be able to write common queries.
  • Fast: Querying a large SQL database can easily take 10 or 20 seconds. It’s quite common for similar queries on a large ElasticSearch database to return results in under 10 milliseconds.
  • Free: Well almost. The basic features are free, well mostly. If you need security and alerting features in Kibana, you can buy the commercial X-pack subscription for Kibana, or you can install some open source alternatives.
  • Full Text Search Engine
  • Analytical Engine
  • RESTful API: ElasticSearch has a RESTful API. Query results are returned in JSON which means results are easy to work with. Querying and inserting data via the RESTful API means it’s easy to use any programming language to work with ElasticSearch.
  • S*calable:* It’s easy to scale. Combined with the fact that it's open source means it's easy on the wallet too.

Companies which use ELK

  • Netflix
  • Ebay
  • Walmart
  • Adobe
  • LinkedIn
  • Stack Overflow

Code Time

We create a docker file

version: '3.1'

services:
  elasticsearch:
    container_name: els
    image: docker.elastic.co/elasticsearch/elasticsearch:7.16.1
    ports:
      - 9200:9200
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    environment:
      - xpack.monitoring.enabled=true
      - xpack.watcher.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - discovery.type=single-node
    networks:
      - elastic

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.16.1
    ports:
      - 5601:5601
    depends_on:
      - els
    environment:
      - ELASTICSEARCH_URL=http://localhost:9200
    networks:
      - elastic

networks:
  elastic:
    driver: bridge

volumes:
  elasticsearch-data:
Enter fullscreen mode Exit fullscreen mode

Execute the command

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

To test they are running we visit the following URLs

Now we create our web application

dotnet new webapi --no-https -n DotnetELK
Enter fullscreen mode Exit fullscreen mode

Install the packages

dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Enrichers.Environment
dotnet add package Serilog.Sinks.Debug
dotnet add package Serilog.Sinks.Elasticsearch
dotnet add package Serilog.Exceptions
dotnet restore
Enter fullscreen mode Exit fullscreen mode

Update appsettings.json

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Information",
        "System": "Warning"
      }
    }
  },
  "ElasticConfiguration": {
    "Uri": "http://localhost:9200"
  },
  "AllowedHosts": "*"
}
Enter fullscreen mode Exit fullscreen mode

We need to update our program.cs

using System.Reflection;
using Serilog;
using Serilog.Sinks.Elasticsearch;

var builder = WebApplication.CreateBuilder(args);

ConfigureLogging();
builder.Host.UseSerilog();

// Add services to the container.
builder.Services.AddControllersWithViews();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (!app.Environment.IsDevelopment())
{
    app.UseExceptionHandler("/Home/Error");
}
app.UseStaticFiles();

app.UseRouting();

app.UseAuthorization();

app.MapControllerRoute(
    name: "default",
    pattern: "{controller=Home}/{action=Index}/{id?}");

app.Run();

void ConfigureLogging()
{
    var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
    var configuration = new ConfigurationBuilder()
        .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
        .AddJsonFile(
            $"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")}.json",
            optional: true)
        .Build();

    Log.Logger = new LoggerConfiguration()
        .Enrich.FromLogContext()
        .Enrich.WithExceptionDetails()
        .WriteTo.Debug()
        .WriteTo.Console()
        .WriteTo.Elasticsearch(ConfigureElasticSink(configuration, environment))
        .Enrich.WithProperty("Environment", environment)
        .ReadFrom.Configuration(configuration)
        .CreateLogger();
}

ElasticsearchSinkOptions ConfigureElasticSink(IConfigurationRoot configuration, string environment)
{
    return new ElasticsearchSinkOptions(new Uri(configuration["ElasticConfiguration:Uri"]))
    {
        AutoRegisterTemplate = true,
        IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower().Replace(".", "-")}-{environment?.ToLower().Replace(".", "-")}-{DateTime.UtcNow:yyyy-MM}"
    };
}
Enter fullscreen mode Exit fullscreen mode

Now we run our application

dotnet run
Enter fullscreen mode Exit fullscreen mode

Add integration

  • click on menu
  • Under Analytics click discover
  • create index pattern
  • in the Name type: dotnetelk-*
  • in timestamp field choose @timestamp
  • click on create index pattern
  • Now go back to discover and we can see all of the logs so far there

Now let us create a custom message, inside our controller we update the following

[HttpGet(Name = "GetWeatherForecast")]
public IEnumerable<WeatherForecast> Get()
{
    _logger.LogInformation("WeatherForecastController Get - this is a nice message to test the logs", DateTime.UtcNow);
    return Enumerable.Range(1, 5).Select(index => new WeatherForecast
    {
        Date = DateTime.Now.AddDays(index),
        TemperatureC = Random.Shared.Next(-20, 55),
        Summary = Summaries[Random.Shared.Next(Summaries.Length)]
    })
    .ToArray();
}
Enter fullscreen mode Exit fullscreen mode

Now we can search for it by typing this in the search box

message: "WeatherForecastController Get"
Enter fullscreen mode Exit fullscreen mode

Now let us test errors, inside our controller

try
{
    throw new Exception("Testing exceptions - ML");
}
catch (Exception ex)
{
    _logger.LogError(ex, "An unknown error occurred on the Index action of the HomeController");
}
Enter fullscreen mode Exit fullscreen mode

Discussion (0)