DEV Community

loading...
Cover image for Create a dashboard to visualize application logs in Kibana

Create a dashboard to visualize application logs in Kibana

Mohsen Esmailpour
I'm a software developer with several years of experience in developing software mainly working on web platforms with Microsoft .NET stack. I'm looking for a job opportunity abroad.
・5 min read

In this article, I'm going to show you how to create a dashboard in Kibana to visualize application logs, and of course if you are using Elasticsearch to store your application logs.

Step 1- Setup Elasticsearch and Kibana

I use docker to run an instance of Elastic and Kibana.

  • Create docker-compose.yml file and add the following content:
version: "3.0"
services:
  elasticsearch:
    container_name: es-container
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    environment:
      - xpack.security.enabled=false
      - "discovery.type=single-node"
    networks:
      - es-net
    ports:
      - 9200:9200
  kibana:
    container_name: kb-container
    image: docker.elastic.co/kibana/kibana:7.12.0
    environment:
      - ELASTICSEARCH_HOSTS=http://es-container:9200
    networks:
      - es-net
    depends_on:
      - elasticsearch
    ports:
      - 5601:5601
networks:
  es-net:
    driver: bridge
Enter fullscreen mode Exit fullscreen mode
  • Run thedocker-compose up command Now you should have access to Elasticsearch via http://localhost:9200/ and Kibana through http://localhost:5601/

Step 2- Setup ASP.NET Core Web API project

This step is optional if you are not a .NET developer, just create an application store some logs into Elasticsearch.

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.UseStartup<Startup>();
        })
        .UseSerilog((hostingContext, loggerConfiguration) =>
            loggerConfiguration.ReadFrom.Configuration(hostingContext.Configuration));
Enter fullscreen mode Exit fullscreen mode
  • Open appsettings.json file and get rid of the logging section:
"Logging": {
  "LogLevel": {
    "Default": "Information",
    "Microsoft": "Warning",
    "Microsoft.Hosting.Lifetime": "Information"
  }
}
Enter fullscreen mode Exit fullscreen mode
  • Add Serilog configuration section to appsettings.json file
"Serilog": {
  "MinimumLevel": {
    "Default": "Debug",
    "Override": {
      "Microsoft": "Information",
      "System": "Information"
    }
  },
  "Enrich": [ "FromLogContext" ],
  "WriteTo": [
    {
      "Name": "Elasticsearch",
      "Args": {
        "nodeUris": "http://localhost:9200",
        "indexFormat": "demo-api-{0:yyyy.MM}",
        "autoRegisterTemplate": true,
        "autoRegisterTemplateVersion": "ESv7"
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode
  • Open Startup class and in the Configure method add the following code
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    ....
    app.UseSerilogRequestLogging();
Enter fullscreen mode Exit fullscreen mode
  • Run the API project and make several API call through Swagger

Step 3- Create index in Kibana

  • In Kibana dashboard from the menu go to Management -> Stack Management
    Alt Text

  • In Stack Management page and in Kibana section click on Index Patterns
    Alt Text

  • In Index Pattern click on Create index pattern button
    Alt Text

  • In Create index pattern page you should see your index pattern demo-api-2021.04 name (if your logs are saved successfully in Elasticsearch) and in Index pattern name input enter your index name demo-api-* and click Next button
    Alt Text

  • In the next step in the Time field drop down select @timestamp and click on Create index pattern button
    Alt Text

  • From the menu go to Analytics -> Discover page and select your index to view logs
    Alt Text

    Step 4- Create Visualizer

    Before creating a dashboard you need to create several visualizers to add your log dashboard.

  • From the menu click on Analytics -> Visualize library then click on Create new visualization

  • From New visualization popup window click on Aggregation based (easiest way to create visualizer is using Lens but after creating visualizer aggregation based method, you can create other visualizers very quickly)
    Alt Text

Total logs metric visualizer
  • Click on Metric visualizer Alt Text
  • After choosing Metric visualizer then click on your index demo-api-*
  • By now I have 100 logs Alt Text
  • You just need to save the visualizer, click on the save button name it Total logs Alt Text
Log level metric visualizer
  • Follow the above steps and create another metric visualizer
  • In Buckets section click on Add then Split group Alt Text
  • From Aggregation dropdown select Terms Alt Text
  • After selecting Terms in Field dropdown select level.raw and then click the Update button at the bottom Alt Text Now you have all log levels count and save the visualizer and name it Log levels count Alt Text
Log level pie visualizer
  • Create a new visualizer and choose Pie
  • In Buckets section click on Add then Split slices
  • In Aggregation dropdown select Terms
  • In Field dropdown select level.raw
  • Click the Update button at the bottom
  • Save the visualizer name it Log levels percentage Alt Text You can click on Options and change the appearance of the pie Alt Text You can also change the color of each level by clicking on the level label Alt Text
Vertical bar visualizer
  • Create a new visualizer and choose Vertical bar
  • In Buckets section click on Add then X-axis
  • In Aggregation dropdown select Date Histogram
  • Click the Update button at the bottom
  • Save the visualizer name it Total logs bar Alt Text
Table visualizer
  • Create a new visualizer and choose Table
  • In Buckets section click on Add then Split rows
  • In Aggregation dropdown select Terms
  • In Field dropdown select fields.SourceContext.raw
  • Click the Update button at the bottom
  • Add another Split rows
  • In Aggregation dropdown select Terms
  • In Field dropdown select level.raw
  • Save the visualizer name it Log level sources Alt Text
Error log level search
  • From the Analytic menu click on Discover
  • In KQL input enter level.raw:"Error
  • In Date filter choose Today
  • Save the search name it Today error level search Alt Text

It's time to create a dashboard.

  • From the Analytic menu click on Dashboard
  • Click on Create new dashboard button Alt Text
  • From the right sidebar click all visualizer we have created previously Alt Text
  • From Types dropdown click on Saved search then click on Today error level search Alt Text
  • Click on Save button and save the dashboard Alt Text

In the end, you can drag and drop or resize visualizers and arrange visualizers as you like.
Alt Text
Alt Text

Discussion (2)

Collapse
rmaurodev profile image
Ricardo

Just did and it's awesome! Thanks for sharing.

Collapse
rmaurodev profile image
Ricardo

Very nice! Thanks again for sharing.

I'll try that at my work. :)

Forem Open with the Forem app