Building a Serverless Application with NestJS and the Serverless Framework: A Monorepo Approach

Table of Contents:

1. Introduction
2. Serverless Configuration and DynamoDB Setup for Two Services
3. Deploying to AWS
4. Running It Offline

Introduction

This blog post explores the utilization of NestJS features in a monorepo mode to build a serverless application using a combination of AWS and the Serverless Framework. The goal is to address the challenge of combining a serverless framework with a monorepo, specifically in the context of creating multiple NestJS apps within the same repository, where each app handles the logic for different endpoints and plays a controller-like role.

To get started, make sure you have Node.js 16.x installed (you can use nvm for easy version management) along with the Serverless Framework and basic knowledge of how the Serverless Framework and AWS Lambda work, as well as familiarity with DynamoDB.

To install the Serverless Framework globally, use the following command:

npm install -g serverless

Next, install the NestJS CLI globally by running:

npm i -g @nestjs/cli

For more detailed information and a brief introduction on how to get started with these tools, refer to the following links:

• Serverless Framework: https://www.serverless.com/framework/docs/getting-started
• NestJS: https://docs.nestjs.com/first-steps

Now, let’s dive into the coding part. Start by creating a standard NestJS application structure using the following command:

nest new users

This will generate a folder structure similar to the one below:

node_modules
src
  app.controller.ts
  app.module.ts
  app.service.ts
main.ts
nest-cli.json
package.json
tsconfig.json
.eslintrc.js

To convert our project into a monorepo structure, use the command:

nest generate app items

This command will convert the project structure into a monorepo structure that looks like thi

apps
  users
    src
      app.controller.ts
      app.module.ts
      app.service.ts
      main.ts
    tsconfig.app.json
  items
    src
      app.controller.ts
      app.module.ts
      app.service.ts
      main.ts
    tsconfig.app.json
nest-cli.json
package.json
tsconfig.json
.eslintrc.js

At this point, we have set up the necessary NestJS structure for our project. Now, let’s introduce the powerful Serverless Compose file, which was recently introduced in version 3.15.0 of the Serverless Framework. The documentation here provides more insights into how to compose Serverless Framework services.

In the root of our project, create a new file named serverless-compose.yaml with the following content:

services:
  items:
    path: apps/items
  users:
    path: apps/users

To deploy our services to AWS, run the following command:

npx serverless deploy

To run any plugin associated with the users or items service, use the following format:

npx serverless items:<plugin-name>

With this setup, you can utilize the power of NestJS and the Serverless Framework in a monorepo structure to build your serverless application on AWS.

Serverless Configuration and DynamoDB Setup for Users and Items Services

Serverless Configuration

Let’s now dive into the “apps/users” directory and embark on the journey of crafting the serverless configuration for our users service. Below, you’ll find an upgraded version of the configuration file, accompanied by a more detailed technical explanation:

#apps/users/serverless.yaml
service: users

plugins:
  - serverless-offline

custom:
    serverless-offline:
        httpPort: 3003
        lambdaPort: 3005

functions:
  getUser:
    handler: dist/main.getUser
    events:
      - http:
          method: GET
          path: /users/{id}
          request: 
            parameters: 
              paths: 
                id: true
  getUsers:
    handler: dist/main.getUsers
    events:
      - http:
          method: GET
          path: /users

provider:
  name: aws
  region: eu-west-3
  runtime: nodejs16.x
  stage: dev

Below is a comprehensive breakdown of each section in the serverless configuration file:

• service: This denotes the unique identifier for the service that will be created. In this instance, the service is aptly named "users".
• plugins: This section enumerates the plugins employed within the service. For this case, the "serverless-offline" plugin is utilized. The "serverless-offline" plugin emulates the behavior of AWS λ and API Gateway on your local machine, significantly expediting the development process. It initializes an HTTP server that manages the lifecycle of requests and calls your handlers.
• custom: This section comprises customized configuration options for the service. Specifically, the custom configuration pertains to the "serverless-offline" plugin. The properties "httpPort", and "lambdaPort" allow for the configuration of ports utilized when running the service in offline mode.
• functions: This section catalogs the individual functions to be created within the service. In this scenario, there are two functions: "getUser" and "getUsers".
• handler: This property specifies the file housing the code for a given function. In this instance, the function "getUser" is defined in the file "dist/main.getUser", while "getUsers" resides in "dist/main.getUsers".
• events: This section enumerates the events that trigger the execution of functions. Both functions in this case are triggered by HTTP requests. The "getUser" function is triggered by HTTP requests to the path "/users/{id}", wherein "id" represents a path parameter. Similarly, the "getUsers" function is triggered by HTTP requests to the path "/users".
• provider: This section specifies the cloud provider to be utilized for deploying the service. In this case, the chosen provider is AWS (Amazon Web Services).
• name: This property designates the name of the AWS service that will be generated. In this case, the AWS service will be named "users".
• region: This property signifies the AWS region where the service will be deployed. In this instance, the chosen AWS region is "eu-west-3".
• runtime: This property defines the runtime environment for the service. In this case, the service will be executed within the Node.js 16.x runtime environment.
• stage: This property denotes the stage of the service. In this instance, the stage is designated as "dev".

Users Service Code Implementation

Let’s examine the implementation of the Users service code. First, we have the main.ts file located in the apps/users directory:

// apps/users/main.ts
import { HttpStatus } from '@nestjs/common';
import { NestFactory } from '@nestjs/core';
import { Callback, Context, Handler } from 'aws-lambda';
import { UsersModule } from './users.module';
import { UsersService } from './users.service';

export const getUser: Handler = async (
  event: any,
  _context: Context,
  _callback: Callback,
) => {
  const appContext = await NestFactory.createApplicationContext(UsersModule);
  const appService = appContext.get(UsersService);
  const { id } = event.pathParameters;
  try {
    const res = await appService.getUser(id);
    return {
      statusCode: HttpStatus.OK,
      body: JSON.stringify(res),
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: HttpStatus.BAD_REQUEST,
      body: JSON.stringify(error.response ?? error.message),
    };
  }
};

export const getUsers: Handler = async (
  _event: any,
  _context: Context,
  _callback: Callback,
) => {
  const appContext = await NestFactory.createApplicationContext(UsersModule);
  const appService = appContext.get(UsersService);
  try {
    const res = await appService.getUsers();
    return {
      statusCode: HttpStatus.OK,
      body: JSON.stringify(res),
    };
  } catch (error) {
    console.log(error);
    return {
      statusCode: HttpStatus.BAD_REQUEST,
      body: JSON.stringify(error.response ?? error.message),
    };
  }
};

In the main.ts file, we define two AWS Lambda handlers, getUser and getUsers, which are responsible for handling the corresponding API endpoints. Here's a breakdown of the code:

• getUser handler: This function receives an HTTP event containing the id parameter in the path. It creates an application context using NestFactory.createApplicationContext with the UsersModule. Then, it retrieves an instance of the UsersService from the application context. The getUser method of the service is called with the id, and the result is returned as an HTTP response with a status code of 200 (OK).
• getUsers handler: This function doesn't require any parameters. It follows a similar process to the getUser handler. It creates an application context, retrieves the UsersService, and calls the getUsers method. The resulting array of users is returned as an HTTP response with a status code of 200 (OK).

Next, let’s take a look at the UsersService implementation in the users.service.ts file:

// apps/users/users.service.ts
import { Injectable, InternalServerErrorException } from '@nestjs/common';
import { DynamoDB } from 'aws-sdk';

const db = new DynamoDB.DocumentClient({
  convertEmptyValues: true,
  paramValidation: true,
});

@Injectable()
export class UsersService {
  async getUser(id: string) {
    const res = await db
      .get({
        TableName: process.env.DYNAMODB_TABLE,
        Key: { id },
        AttributesToGet: ['id', 'email', 'firstName', 'lastName'],
      })
      .promise();
    if (res.$response.error || !res.Item) {
      throw new InternalServerErrorException(res.$response.error);
    }
    return res.Item;
  }
 async getUsers() {
    const res = await db
      .scan({
        TableName: process.env.DYNAMODB_TABLE,
        AttributesToGet: ['id', 'email', 'firstName', 'lastName'],
      })
      .promise();
    if (res.$response.error) {
      throw new InternalServerErrorException(res.$response.error.message);
    }
    return res.Items;
  }
}

The UsersService class provides the business logic for handling user-related operations. Here's a breakdown of the code:

getUser(id: string): This method retrieves a user from the DynamoDB table based on the provided id. It uses the get method of the DynamoDB.DocumentClient to fetch the user's data from the table. If there is an error or the item is not found, an InternalServerErrorException is thrown. Otherwise, the retrieved user is returned.

getUsers(): This method retrieves all users from the DynamoDB table. It uses the scan method of the DynamoDB.DocumentClient to perform a scan operation on the table. If there is an error during the scan operation, an InternalServerErrorException is thrown. Otherwise, the array of retrieved users is returned.

Finally, let’s look at the UsersModule defined in the users.module.ts file:

// apps/users/users.module.ts
import { Module } from '@nestjs/common';
import { UsersService } from './users.service';
@Module({
  imports: [],
  providers: [UsersService],
})
export class UsersModule {}

The UsersModule is a basic NestJS module that imports no other modules (imports: []) and provides the UsersService as a provider (providers: [UsersService]).

In summary, the code provided demonstrates the implementation of the Users service in the NestJS monorepo. It includes Lambda handlers for handling API requests, a UsersService class for performing user-related operations using DynamoDB, and a UsersModule that defines the service as a provider.

In addition to that, a typescript configuration file needs to be added that provides compiler options for the Users service. It extends the root tsconfig.json file located in the monorepo root directory and provides specific compiler options and file inclusion/exclusion rules for the Users service within the NestJS

//apps/users/tsconfig.app.json
{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "declaration": false,
    "outDir": "./dist",
  },
  "include": ["src/**/*", "src/*"],
  "exclude": ["node_modules", "dist", ".build", "test", "**/*spec.ts"]
}

DynamoDB Setup

As you can see or notice we are using DynamoDB as the main database for our application and it needs some setup with a serverless framework to work properly so below you can see how our file will look when we add Dynamodb.

#apps/users/serverless.yaml
service: users

plugins:
  - serverless-offline

custom:
    serverless-offline:
        httpPort: 3003
        lambdaPort: 3005

functions: 
  getUser:
    handler: dist/main.getUser
    events:
      - http:
          method: GET
          path: /users/{id}
          request: 
            parameters: 
              paths: 
                id: true
  getUsers:
    handler: dist/main.getUsers
    events:
      - http:
          method: GET
          path: /users

provider:
  name: aws
  region: eu-west-3
  runtime: nodejs16.x
  stage: dev
  environment:
    DYNAMODB_TABLE: ${self:service}-${opt:stage, self:provider.stage}
  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:Query
        - dynamodb:Scan  
        - dynamodb:GetItem
        - dynamodb:UpdateItem
      Resource: 
        Fn::Join:
          - ''
          - - "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/"
            - ${self:provider.environment.DYNAMODB_TABLE}

resources:
  Resources:
    UsersTable:
      Type: AWS::DynamoDB::Table
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        TableName: ${self:provider.environment.DYNAMODB_TABLE}

So as we can see our serverless file has been updated to add the following things:

IAM permissions: The IAM role for the users service has been updated to allow the following DynamoDB operations:

• Scan
• GetItem

DynamoDB table: The DynamoDB table is used to store data for the users service. The table name is a concatenation of the service name and the stage. This allows you to have multiple tables for the same service, each with a different stage.

Resources: A new DynamoDB table resource has been added to the users service. This resource defines the DynamoDB table that will be created for the service.

Items Service Implementation

Now let’s focus on the implementation of the Items service. We’ll take a closer look at the files that describe the Items service, its model, main file, and serverless.yaml. We’ll also cover the setup of DynamoDB and the usage of the “serverless offline” plugin

#apps/items/serverless.yaml
service: items

plugins:
  - serverless-offline

custom:
    serverless-offline:
        httpPort: 3006
        lambdaPort: 3008

functions:
  createItem:
    handler: dist/main.createItem
    events:
      - http:
          method: POST
          path: /items
  getItem:
    handler: dist/main.getItem
    events:
      - http:
          method: GET
          path: /items/{id}
          request: 
            parameters: 
              paths: 
                id: true
  getItems:
    handler: dist/main.getItems
    events:
      - http:
          method: GET
          path: /items

provider:
  name: aws
  region: eu-west-3
  runtime: nodejs16.x
  stage: dev
  environment:
    DYNAMODB_TABLE: ${self:service}-${opt:stage, self:provider.stage}

  iamRoleStatements:
    - Effect: Allow
      Action:
        - dynamodb:Scan  
        - dynamodb:GetItem
        - dynamodb:PutItem
      Resource: 
        Fn::Join:
          - ''
          - - "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/"
            - ${self:provider.environment.DYNAMODB_TABLE}

resources:
  Resources:
    ItemsTable: 
      Type: AWS::DynamoDB::Table
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        TableName: ${self:provider.environment.DYNAMODB_TABLE}  

//apps/items/items.module.ts
import { Module } from '@nestjs/common';
import { ItemsService } from './items.service';

@Module({
  imports: [],
  providers: [ItemsService],
})
export class ItemsModule {}

//app/items/items.service.ts
import { Injectable, InternalServerErrorException } from '@nestjs/common';
import { v1 } from 'uuid';
import { DynamoDB } from 'aws-sdk';

const db = new DynamoDB.DocumentClient();

@Injectable()
export class ItemsService {
  async createItem(item: any) {
    const { title, description } = item;
    const createdOn = new Date().getTime();

    const data = {
      TableName: process.env.DYNAMODB_TABLE,
      Item: {
        id: v1(),
        title,
        description,
        createdOn,
      },
    };

    try {
      await db.put(data).promise();
      return item;
    } catch (error) {
      throw new InternalServerErrorException(error.message);
    }
  }

  async getItem(id: string) {
    const params = {
      TableName: process.env.DYNAMODB_TABLE,
      Key: { id },
    };

    try {
      const result = await db.get(params).promise();
      return result.Item;
    } catch (error) {
      throw new InternalServerErrorException(error.message);
    }
  }

  async getItems() {
    const params = {
      TableName: process.env.DYNAMODB_TABLE,
    };

    try {
      const result = await db.scan(params).promise();
      return result.Items;
    } catch (error) {
      throw new InternalServerErrorException(error.message);
    }
  }
}

The tsconfig.app.json file for the Items service is the same as the one used for the Users service.

Deploying to AWS

To deploy our serverless application to AWS using the Serverless Framework, we can start by adding the following scripts to our package.json file:

"scripts": {
  "build:items": "nest build --tsc items",
  "build:users": "nest build --tsc users",
  "build:all": "npm run build:users && npm run build:items",
  "deploy": "npm install && npm run build:all && npx serverless deploy",
  ....
 },

These scripts provide a convenient way to build and deploy the application, allowing you to easily compile the TypeScript code and package it for deployment using the Serverless framework.

1. "build:api": "nest build --tsc api": This script builds the API module using the NestJS CLI. It compiles the TypeScript code in the api module and generates the corresponding JavaScript files.
2. "build:items": "nest build --tsc items": This script builds the items module using the NestJS CLI. It compiles the TypeScript code in the items module and generates the corresponding JavaScript files.
3. "build:users": "nest build --tsc user": This script builds the users module using the NestJS CLI. It compiles the TypeScript code in the users module and generates the corresponding JavaScript files.
4. "build:all": "npm run build:items && npm run build:users": This script is a convenience script that runs the build scripts for all the modules (items, and users) sequentially. It ensures that all the modules are built before proceeding to the deployment.
5. "deploy": "npm install && npm run build:all && npx serverless deploy": This script handles the deployment process. It first installs the required dependencies (npm install), builds all the modules (npm run build:all), and then deploys the application using the Serverless framework (npx serverless deploy).

Adding the necessary packages and deploying to AWS:

package:
  exclude:
    - ../../node_modules/**
    - ./src/**
    - ./test/**
  include:
    - '../../node_modules/@nestjs/common/**'
    - '../../node_modules/@nestjs/core/**'
    - '../../node_modules/@nestjs/schematics/**'
    - '../../node_modules/@nestjs/testing/**'
    - '../../node_modules/tslib/**'
    - '../../node_modules/reflect-metadata/**'
    - '../../node_modules/uid/**'
    - '../../node_modules/rxjs/**'
    - '../../node_modules/iterare/**'
    - '../../node_modules/@nuxtjs/**'
    - '../../node_modules/fast-safe-stringify/**'
    - '../../node_modules/path-to-regexp/**'
    - '../../node_modules/cache-manager/**'
    - '../../node_modules/class-transformer/**'
    - '../../node_modules/class-validator/**'
    - '../../node_modules/cache-manager/**'
    - '../../node_modules/@angular-devkit/**'
    - '../../node_modules/jsonc-parser/**'
    - '../../node_modules/pluralize/**'
    - '../../node_modules/body-parser/**'
    - '../../node_modules/cors/**'
    - '../../node_modules/express/**'
    - '../../node_modules/multer/**'
    - '../../node_modules/@vendia/**'

In the provided code snippet, the package key is added to the serverless files for both the items and users services. This key is used to configure the packaging process of the Lambda functions, determining which files and directories should be included or excluded in the deployment package.

The exclude property within the package the configuration specifies patterns to exclude files and directories from the deployment package. In this case, the patterns ../../node_modules/**, ./src/**, and ./test/** are used. The ../../node_modules/** pattern excludes all files and directories within the node_modules directory, ensuring that dependencies installed via npm or yarn are not included in the deployment package. The ./src/** and ./test/** patterns exclude the source code and test files, respectively, as they are not required for the deployed application to function properly.

On the other hand, the include property lists specific packages that are necessary for the Lambda functions to execute correctly. The packages specified in the include property are included in the deployment package, ensuring that the functions have access to their required dependencies. In the provided code snippet, various NestJS packages (@nestjs/common, @nestjs/core, @nestjs/schematics, @nestjs/testing), as well as other essential packages (tslib, reflect-metadata, uid, rxjs, iterare, @nuxtjs, fast-safe-stringify, path-to-regexp, cache-manager, class-transformer, class-validator, @angular-devkit, jsonc-parser, pluralize, body-parser, cors, express, multer, @vendia) are included.

By configuring the package property in this way, we optimize the deployment package size by excluding unnecessary files and only including the required packages. This results in faster deployment times and reduces the memory footprint of the Lambda functions.

Once the package configuration is in place, and running the command npm run deploy in the root directory triggers the deployment process. The Serverless framework packages the Lambda functions along with the specified packages and deploys them to AWS.

Overall, the package configuration helps address two important issues: including the required packages in the deployment and excluding unnecessary files to optimize the package size.

Running it offline

Before we dive into the technical details, let’s address a fundamental question: why should we run our serverless functions offline in the first place? By doing so, we unlock several advantages that significantly enhance our development process.

1. Ease of Development:

Running serverless functions offline provides developers with the freedom to work on their code locally, without the need for a live serverless infrastructure, resulting in a faster and more efficient development workflow.

1. Cost Savings: Optimizing Your Budget

Serverless functions are billed based on their usage, such as the number of invocations and execution duration. Deploying and testing functions in a live environment during development can lead to unexpected costs.

1. Rapid Iterations: Accelerating Your Progress

Offline execution empowers developers to iterate rapidly and test their serverless functions without any deployment delays. With the ability to make code changes, run functions locally, and observe immediate results.

1. Isolation and Debugging: Mastering the Art of Troubleshooting

Running serverless functions offline provides a controlled and isolated environment for debugging, this isolation simplifies the troubleshooting process, enabling us to identify and fix issues without the complexities introduced by cloud-based environment.

Now, let’s explore how we can implement offline execution for our serverless functions using the Serverless Framework. Going back to our package.json file, we can add the following scripts to enable offline execution for specific services:

"scripts": {
  "start:dev:api": "npm run build:api && serverless api:offline",
  "start:dev:items": "npm run build:items && serverless items:offline",
  "start:dev:users": "npm run build:users && serverless users:offline",
  "start:dev": "concurrently \"npm run start:dev:items\" \"npm run start:dev:users\"",
  ...
}

The added scripts in the package.json file allows us to run the items and users services offline for development and testing purposes. Here's an explanation of each script:

1. "start:dev:items": "npm run build:items && serverless items:offline" azeazthis script builds the items service by running npm run build:items, which compiles the TypeScript code into JavaScript, after the build process, it starts the items service in offline mode using the serverless items:offline command, running the items service offline allows us to test and work with it locally without the need for a live serverless infrastructure.
2. "start:dev:users": "npm run build:users && serverless users:offline" This script builds the users service by running npm run build:users, which compiles the TypeScript code into JavaScript, after the build process, it starts the users service in offline mode using the serverless users:offline command, running the users service offline enables us to test and interact with it locally without deploying it to a production environment.
3. "start:dev": "concurrently \"npm run start:dev:items\" \"npm run start:dev:users\"" This script uses the concurrently package to run multiple scripts concurrently, It starts the items and users services simultaneously in offline mode running both services together allows us to test the integration between the items and users services locally, mimicking the behavior of a real production environment.

By running these scripts, we can easily start the items and users services locally and test them in an offline mode. This setup enables faster development iterations and facilitates debugging without the need for a fully deployed infrastructure. It provides a convenient way to work on and test specific services independently or together as part of a larger system.

In this post, we have built our application using a serverless framework combined with Nestjs, there still be a lot of features in both of these tools that we will discover in the upcoming posts, and we will also fix some mistakes and improve our app by adding more features to it.

References:

• Documentation | NestJS - A progressive Node.js framework
• Documentation | NestJS - A progressive Node.js framework
• Documentation | NestJS - A progressive Node.js framework
• Documentation | NestJS - A progressive Node.js framework
• Serverless Framework - Packaging
• Tutorial: Your First Serverless Framework Project

• Serverless Framework - Composing services

• https://www.serverless.com/guides/dynamodb

• https://www.udemy.com/course/aws-lambda-serverless/