Elasticsearch Guide on Docker: A Comprehensive Walkthrough

Introduction

Elasticsearch is a powerful, open-source search and analytics engine designed for handling large volumes of data in real-time. Whether you need full-text search, structured search, or analytics, Elasticsearch offers a scalable solution capable of managing and analyzing large datasets quickly and efficiently.

Why Use Elasticsearch?

1. Speed and Scalability: Elasticsearch is built on top of Apache Lucene and is known for its speed and scalability. It can handle large amounts of data and provide near real-time search results.
2. Full-Text Search Capabilities: With Elasticsearch, you can perform complex full-text searches, including natural language processing, filtering, and ranking, making it ideal for applications that require advanced search functionalities.
3. Real-Time Analytics: Elasticsearch allows for real-time data analysis, making it a popular choice for logging, monitoring, and observability use cases.
4. Distributed Nature: Elasticsearch is designed to scale horizontally across multiple nodes, making it resilient and highly available.

How to Use Elasticsearch

Elasticsearch is commonly used in:

• Web and Application Search: Enhancing search capabilities in websites, e-commerce platforms, and applications.
• Log and Event Data Management: Storing, searching, and analyzing log and event data in real-time.
• Data Analytics: Performing complex queries and aggregations on large datasets.

Installing Elasticsearch on Docker

Step 1: Set Up Elasticsearch with Docker

1. Pull the Elasticsearch Docker Image:

   docker pull elasticsearch:8.10.2

1. Run Elasticsearch in a Docker Container:

   docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:8.10.2

Step 2: Troubleshooting Common Issues

During the setup, several issues can arise. Here’s a summary of the troubleshooting steps:

1. Port Conflict:

   • If Docker fails to start due to a port conflict, ensure that no other process is using ports 9200 or 9300. Use the following commands to find and stop conflicting processes:

    netstat -ano | findstr :9200
    taskkill /PID <PID> /F
   

2. Empty Reply from Server:

   • If you encounter an "Empty reply from server" error, it may be due to Elasticsearch not responding properly. Ensure that the container is running:

    docker ps
   

• Check the logs for any issues:
  

   docker logs elasticsearch
  

1. Authentication Issues:

   • If you receive a security exception when accessing Elasticsearch, reset the password for the elastic user:

    docker exec -it elasticsearch bin/elasticsearch-reset-password -u elastic
   

2. Security Configuration:

   • Elasticsearch may require HTTPS and authentication. Use the curl command with the -k flag to bypass certificate warnings:

    curl -k -u elastic:your_password https://localhost:9200
   

3. Disk Space Warning:

   • If Elasticsearch logs a warning like low disk watermark [85%] exceeded, it means the disk space is running low. To resolve this, you can increase the available disk space using the following command in your Docker environment:

    wsl -d docker-desktop sysctl -w vm.max_map_count=262144
   

• Additionally, you can adjust the disk watermark settings in the elasticsearch.yml configuration file:
  

   cluster.routing.allocation.disk.watermark.low: "5%"
   cluster.routing.allocation.disk.watermark.high: "10%"
   cluster.routing.allocation.disk.watermark.flood_stage: "2%"
  

• After making changes, restart the Elasticsearch container to apply the settings:
  

   docker restart elasticsearch
  

Building a .NET Core POC with Elasticsearch

Step 1: Set Up the .NET Core Project

1. Create a New Console Application:

   dotnet new console -n ElasticsearchPOC
   cd ElasticsearchPOC

1. Install the NEST Client:

   dotnet add package NEST --version 7.17.0

Step 2: Connect to Elasticsearch and Index Data

1. Connect to Elasticsearch:

   var settings = new ConnectionSettings(new Uri("https://localhost:9200"))
                  .DefaultIndex("my_index")
                  .BasicAuthentication("elastic", "your_password")
                  .ServerCertificateValidationCallback(CertificateValidations.AllowAll);

   var client = new ElasticClient(settings);

1. Index Sample Data:

   var person = new { Id = 1, Name = "John Doe", Age = 30, Occupation = "Software Developer" };
   var indexResponse = client.IndexDocument(person);

1. Search the Data:

   var searchResponse = client.Search<object>(s => s
       .Query(q => q.Match(m => m.Field("name").Query("John Doe")))
   );

Step 3: Run the Application

dotnet run

Conclusion

Elasticsearch is a powerful tool for search and analytics, and integrating it with .NET Core provides a robust solution for managing and querying data. By running Elasticsearch on Docker and connecting it with .NET Core, you can quickly deploy a scalable search solution with minimal setup.