Hi guys, welcome to back to the Part 3 of the Elastic, Kibana and NestJS series. In the part 1 of this series, we installed and configured elasticsearch (check it here if you missed it), in the part 2, we connected elasticsearch with Kibana and ran a few queries (check here for part 2).
In this article, we will be writing the NodeJS code that will connect and query elasticsearch.
Loading data into elasticsearch
To enable us write our code effectively, we need data loaded into our elasticsearch. We will be using a sample dataset from Kaggle (Download it here.
Follow the FOUR steps below to load it up into elasticsearch
- Open up Kibana (http://localhost:5601)
- Under
Get started by adding integrationsclick on Upload a file
- Click on import and enter the name of the index you want to put the data in.
- Click on import (final page)
If you made it to this point, you have successfully imported data into elasticsearch.
Querying for sample
Goto DevTools (Hamburger on the top left corner of the screen > Scroll down to Management > DevTools)
Run the query below (select it and click on the play button)
GET tmdb_movies/_search
If you see this, we are good to go!
Now, Let's dive into coding, shall we 😊?
NestJs
NestJS is a progressive Node. js framework that helps build server-side applications. Nest extends Node. js frameworks like Express or Fastify adding modular organization and a wide range of other libraries to take care of repetitive tasks. It's open-source, uses TypeScript, and is a very versatile Node.
Creating a NestJs Application
Run the command below to install nestcli and create a new NestJs Application (in the article, the name of the app will be nest-elastic).
$ npm i -g @nestjs/cli
$ nest new nest-elastic
You will be asked to select a package manager, you can select npm, yarn or pnpm. I will be selecting yarn (you can choose any other one you want 😉). Your project will be setup and we should be ready to code!
Adding elasticsearch to your app
Run the command below to add elasticsearch to your nest-elastic and other dependencies.
yarn add @elastic/elasticsearch @nestjs/elasticsearch @nestjs/config
In your root folder add your .env file with the following contents
ELASTICSEARCH_NODE=https://localhost:9200
ELASTICSEARCH_USERNAME=elastic
ELASTICSEARCH_PASSWORD=elasticPasswordGoesHere
ELASTICSEARCH_MAX_RETRIES=10
ELASTICSEARCH_REQ_TIMEOUT=50000
ELASTICSEARCH_INDEX=tmdb_movies
Let's create a separate module that handles only search using elasticsearch. A simple shortcut is to use the command below (you are welcome to do it manually, if you want too)
nest g resource search
Update the search.module.ts to have the content below
import { Module } from '@nestjs/common';
import { SearchService } from './search.service';
import { SearchController } from './search.controller';
import { ConfigModule, ConfigService } from '@nestjs/config';
import { ElasticsearchModule } from '@nestjs/elasticsearch';
@Module({
imports: [
ConfigModule,
ElasticsearchModule.registerAsync({
imports: [ConfigModule],
useFactory: async (configService: ConfigService) => ({
node: configService.get('ELASTICSEARCH_NODE'),
auth: {
username: configService.get('ELASTICSEARCH_USERNAME'),
password: configService.get('ELASTICSEARCH_PASSWORD'),
},
maxRetries: configService.get('ELASTICSEARCH_MAX_RETRIES'),
requestTimeout: configService.get('ELASTICSEARCH_REQ_TIMEOUT'),
}),
inject: [ConfigService],
}),
],
controllers: [SearchController],
providers: [SearchService],
exports: [SearchService],
})
export class SearchModule {}
Update search.service.ts with the content below:
import { ConfigService } from '@nestjs/config';
import { Injectable } from '@nestjs/common';
import { ElasticsearchService } from '@nestjs/elasticsearch';
type dataResponse = {
UnitPrice: number;
Description: string;
Quantity: number;
Country: string;
InvoiceNo: string;
InvoiceDate: Date;
CustomerID: number;
StockCode: string;
};
@Injectable()
export class SearchService {
constructor(
private readonly esService: ElasticsearchService,
private readonly configService: ConfigService,
) {}
async search(search: {key: string}) {
let results = new Set();
const response = await this.esService.search({
index: this.configService.get('ELASTICSEARCH_INDEX'),
body: {
size: 50,
query: {
match_phrase: search
},
},
});
const hits = response.hits.hits;
hits.map((item) => {
results.add(item._source as dataResponse);
});
return { results: Array.from(results), total: response.hits.total };
}
}
Now, let's add movies modules
nest g resource movies
Update movies.controller.ts with the content below:
import { SearchService } from './../search/search.service';
import { Body, Controller, Post } from '@nestjs/common';
@Controller('movies')
export class MoviesController {
constructor(private readonly searchService: SearchService) {}
@Post('search')
async search(@Body() body) {
return await this.searchService.search(body.data);
}
}
Then movies.module.ts
import { SearchModule } from './../search/search.module';
import { Module } from '@nestjs/common';
import { MoviesService } from './movies.service';
import { MoviesController } from './movies.controller';
@Module({
imports: [SearchModule],
controllers: [MoviesController],
providers: [MoviesService],
})
export class MoviesModule {}
Finally, update app.module.ts
import { MoviesModule } from './movies/movies.module';
import { ConfigModule } from '@nestjs/config';
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { SearchModule } from './search/search.module';
@Module({
imports: [MoviesModule, ConfigModule.forRoot(), SearchModule],
controllers: [AppController],
providers: [AppService],
})
export class AppModule {}
Your package.json should look like this
{
"name": "nest-elastic",
"version": "0.0.1",
"description": "",
"author": "Yusuf Ganiyu",
"private": true,
"license": "UNLICENSED",
"scripts": {
"prebuild": "rimraf dist",
"build": "nest build",
"format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
"start": "nest start",
"start:dev": "nest start --watch",
"start:debug": "nest start --debug --watch",
"start:prod": "node dist/main",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"test": "jest",
"test:watch": "jest --watch",
"test:cov": "jest --coverage",
"test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
"test:e2e": "jest --config ./test/jest-e2e.json"
},
"dependencies": {
"@elastic/elasticsearch": "^8.4.0",
"@nestjs/common": "^9.0.0",
"@nestjs/config": "^2.2.0",
"@nestjs/core": "^9.0.0",
"@nestjs/elasticsearch": "^9.0.0",
"@nestjs/mapped-types": "*",
"@nestjs/platform-express": "^9.0.0",
"reflect-metadata": "^0.1.13",
"rimraf": "^3.0.2",
"rxjs": "^7.2.0"
},
"devDependencies": {
"@nestjs/cli": "^9.0.0",
"@nestjs/schematics": "^9.0.0",
"@nestjs/testing": "^9.0.0",
"@types/express": "^4.17.13",
"@types/jest": "28.1.4",
"@types/node": "^16.0.0",
"@types/supertest": "^2.0.11",
"@typescript-eslint/eslint-plugin": "^5.0.0",
"@typescript-eslint/parser": "^5.0.0",
"eslint": "^8.0.1",
"eslint-config-prettier": "^8.3.0",
"eslint-plugin-prettier": "^4.0.0",
"jest": "28.1.2",
"prettier": "^2.3.2",
"source-map-support": "^0.5.20",
"supertest": "^6.1.3",
"ts-jest": "28.0.5",
"ts-loader": "^9.2.3",
"ts-node": "^10.0.0",
"tsconfig-paths": "4.0.0",
"typescript": "^4.3.5"
},
"jest": {
"moduleFileExtensions": [
"js",
"json",
"ts"
],
"rootDir": "src",
"testRegex": ".*\\.spec\\.ts$",
"transform": {
"^.+\\.(t|j)s$": "ts-jest"
},
"collectCoverageFrom": [
"**/*.(t|j)s"
],
"coverageDirectory": "../coverage",
"testEnvironment": "node"
}
}
Running the app
You can fire up the app in dev environment using yarn start:dev
Testing
You can access the full source code here
Summary
In this article, we are able to import data into elastic using kibana and also connect our NestJs backend app to the leverage the power of elasticsearch.
In the next article, we will be building a simple frontend to query and visualize results from elasticsearch in realtime.
Thanks for reading!
Top comments (0)