DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’» is a community of 963,673 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in
Cover image for How to Create a Full Autocomplete Search Application with Elasticsearch, Kibana and NestJS - Part 3
Yusuf Ganiyu
Yusuf Ganiyu

Posted on • Updated on

How to Create a Full Autocomplete Search Application with Elasticsearch, Kibana and NestJS - Part 3

Hi guys, welcome to back to the Part 3 of the Elastic, Kibana and NestJS series. In the part 1 of this series, we installed and configured elasticsearch (check it here if you missed it), in the part 2, we connected elasticsearch with Kibana and ran a few queries (check here for part 2).

In this article, we will be writing the NodeJS code that will connect and query elasticsearch.

Loading data into elasticsearch

To enable us write our code effectively, we need data loaded into our elasticsearch. We will be using a sample dataset from Kaggle (Download it here.

Follow the FOUR steps below to load it up into elasticsearch

  1. Open up Kibana (http://localhost:5601) kibana upload
  2. Under Get started by adding integrations click on Upload a file upload file
  3. Click on import and enter the name of the index you want to put the data in. index name
  4. Click on import (final page) import

If you made it to this point, you have successfully imported data into elasticsearch.

Querying for sample

Goto DevTools (Hamburger on the top left corner of the screen > Scroll down to Management > DevTools)

Run the query below (select it and click on the play button)

GET tmdb_movies/_search
Enter fullscreen mode Exit fullscreen mode

If you see this, we are good to go!

querying elasticsearch


Now, Let's dive into coding, shall we 😊?

NestJs

NestJS is a progressive Node. js framework that helps build server-side applications. Nest extends Node. js frameworks like Express or Fastify adding modular organization and a wide range of other libraries to take care of repetitive tasks. It's open-source, uses TypeScript, and is a very versatile Node.

Creating a NestJs Application

Run the command below to install nestcli and create a new NestJs Application (in the article, the name of the app will be nest-elastic).

$ npm i -g @nestjs/cli
$ nest new nest-elastic
Enter fullscreen mode Exit fullscreen mode

You will be asked to select a package manager, you can select npm, yarn or pnpm. I will be selecting yarn (you can choose any other one you want πŸ˜‰). Your project will be setup and we should be ready to code!

project setup

Adding elasticsearch to your app

Run the command below to add elasticsearch to your nest-elastic and other dependencies.

yarn add @elastic/elasticsearch @nestjs/elasticsearch @nestjs/config
Enter fullscreen mode Exit fullscreen mode

In your root folder add your .env file with the following contents

ELASTICSEARCH_NODE=https://localhost:9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=elasticPasswordGoesHere
ELASTICSEARCH_MAX_RETRIES=10
ELASTICSEARCH_REQ_TIMEOUT=50000
ELASTICSEARCH_INDEX=tmdb_movies
Enter fullscreen mode Exit fullscreen mode

Let's create a separate module that handles only search using elasticsearch. A simple shortcut is to use the command below (you are welcome to do it manually, if you want too)

nest g resource search
Enter fullscreen mode Exit fullscreen mode

added search module

Update the search.module.ts to have the content below

import { Module } from '@nestjs/common';
import { SearchService } from './search.service';
import { SearchController } from './search.controller';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { ElasticsearchModule } from '@nestjs/elasticsearch';

@Module({
  imports: [
    ConfigModule,
    ElasticsearchModule.registerAsync({
      imports: [ConfigModule],
      useFactory: async (configService: ConfigService) => ({
        node: configService.get('ELASTICSEARCH_NODE'),
        auth: {
          username: configService.get('ELASTICSEARCH_USERNAME'),
          password: configService.get('ELASTICSEARCH_PASSWORD'),
        },
        maxRetries: configService.get('ELASTICSEARCH_MAX_RETRIES'),
        requestTimeout: configService.get('ELASTICSEARCH_REQ_TIMEOUT'),
      }),
      inject: [ConfigService],
    }),
  ],
  controllers: [SearchController],
  providers: [SearchService],
  exports: [SearchService],
})
export class SearchModule {}
Enter fullscreen mode Exit fullscreen mode

Update search.service.ts with the content below:

import { ConfigService } from '@nestjs/config';
import { Injectable } from '@nestjs/common';
import { ElasticsearchService } from '@nestjs/elasticsearch';

type dataResponse = {
  UnitPrice: number;
  Description: string;
  Quantity: number;
  Country: string;
  InvoiceNo: string;
  InvoiceDate: Date;
  CustomerID: number;
  StockCode: string;
};

@Injectable()
export class SearchService {
  constructor(
    private readonly esService: ElasticsearchService,
    private readonly configService: ConfigService,
  ) {}

  async search(search: {key: string}) {
    let results = new Set();
    const response = await this.esService.search({
      index: this.configService.get('ELASTICSEARCH_INDEX'),
      body: {
        size: 50,
        query: {
          match_phrase: search
        },
      },
    });
    const hits = response.hits.hits;
    hits.map((item) => {
      results.add(item._source as dataResponse);
    });

    return { results: Array.from(results), total: response.hits.total };
  }
}
Enter fullscreen mode Exit fullscreen mode

Now, let's add movies modules

nest g resource movies
Enter fullscreen mode Exit fullscreen mode

Update movies.controller.ts with the content below:

import { SearchService } from './../search/search.service';
import { Body, Controller, Post } from '@nestjs/common';

@Controller('movies')
export class MoviesController {
  constructor(private readonly searchService: SearchService) {}

  @Post('search')
  async search(@Body() body) {
    return await this.searchService.search(body.data);
  }
}
Enter fullscreen mode Exit fullscreen mode

Then movies.module.ts

import { SearchModule } from './../search/search.module';
import { Module } from '@nestjs/common';
import { MoviesService } from './movies.service';
import { MoviesController } from './movies.controller';

@Module({
  imports: [SearchModule],
  controllers: [MoviesController],
  providers: [MoviesService],
})
export class MoviesModule {}

Enter fullscreen mode Exit fullscreen mode

Finally, update app.module.ts

import { MoviesModule } from './movies/movies.module';
import { ConfigModule } from '@nestjs/config';
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { SearchModule } from './search/search.module';

@Module({
  imports: [MoviesModule, ConfigModule.forRoot(), SearchModule],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}
Enter fullscreen mode Exit fullscreen mode

Your package.json should look like this

{
  "name": "nest-elastic",
  "version": "0.0.1",
  "description": "",
  "author": "Yusuf Ganiyu",
  "private": true,
  "license": "UNLICENSED",
  "scripts": {
    "prebuild": "rimraf dist",
    "build": "nest build",
    "format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
    "start": "nest start",
    "start:dev": "nest start --watch",
    "start:debug": "nest start --debug --watch",
    "start:prod": "node dist/main",
    "lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
    "test": "jest",
    "test:watch": "jest --watch",
    "test:cov": "jest --coverage",
    "test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
    "test:e2e": "jest --config ./test/jest-e2e.json"
  },
  "dependencies": {
    "@elastic/elasticsearch": "^8.4.0",
    "@nestjs/common": "^9.0.0",
    "@nestjs/config": "^2.2.0",
    "@nestjs/core": "^9.0.0",
    "@nestjs/elasticsearch": "^9.0.0",
    "@nestjs/mapped-types": "*",
    "@nestjs/platform-express": "^9.0.0",
    "reflect-metadata": "^0.1.13",
    "rimraf": "^3.0.2",
    "rxjs": "^7.2.0"
  },
  "devDependencies": {
    "@nestjs/cli": "^9.0.0",
    "@nestjs/schematics": "^9.0.0",
    "@nestjs/testing": "^9.0.0",
    "@types/express": "^4.17.13",
    "@types/jest": "28.1.4",
    "@types/node": "^16.0.0",
    "@types/supertest": "^2.0.11",
    "@typescript-eslint/eslint-plugin": "^5.0.0",
    "@typescript-eslint/parser": "^5.0.0",
    "eslint": "^8.0.1",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-prettier": "^4.0.0",
    "jest": "28.1.2",
    "prettier": "^2.3.2",
    "source-map-support": "^0.5.20",
    "supertest": "^6.1.3",
    "ts-jest": "28.0.5",
    "ts-loader": "^9.2.3",
    "ts-node": "^10.0.0",
    "tsconfig-paths": "4.0.0",
    "typescript": "^4.3.5"
  },
  "jest": {
    "moduleFileExtensions": [
      "js",
      "json",
      "ts"
    ],
    "rootDir": "src",
    "testRegex": ".*\\.spec\\.ts$",
    "transform": {
      "^.+\\.(t|j)s$": "ts-jest"
    },
    "collectCoverageFrom": [
      "**/*.(t|j)s"
    ],
    "coverageDirectory": "../coverage",
    "testEnvironment": "node"
  }
}
Enter fullscreen mode Exit fullscreen mode

Running the app

You can fire up the app in dev environment using yarn start:dev

running terminal

Testing

postman -1

postman -2

You can access the full source code here

Summary

In this article, we are able to import data into elastic using kibana and also connect our NestJs backend app to the leverage the power of elasticsearch.

In the next article, we will be building a simple frontend to query and visualize results from elasticsearch in realtime.

Thanks for reading!

Top comments (0)

Must read for JS devs:

"I made 10x faster JSON.stringify() functions, even type safe"