Amazon Simple Queue Service with NodeJs (Amazon SQS)

As we all know Amazon is providing countless services to make our (as developer) life easier. Simple Queue Service or aka Amazon SQS is boon for microservice or serverless application.

Need for SQS will be realized mostly when one part of the service works very quickly and another at a different pace. For example, You have a job that sends a daily email to your subscribers. It can get hundreds or thousands of email addresses to send emails to in a matter of milliseconds. However, sending emails is a bit slower process. SQS comes handy in this scenario. Using SQS, your first job could send email addresses to queue, a lambda function is triggered whenever there is a new message in the queue and get the email address and finally send it.

This article is divided into three sections. In section one we setup AWS and create two functions with minimum code and deploy them to AWS. In section two, we would actually work on creating an SQS queue and sending messages. In the last section, we work on functionTwo to make it receive and process queue messages.

If you just want to see the code, click Here

Section One - Setup

Let's now talk about how can we implement SQS using NodeJs. In this article, we will be using the Serverless Framework, which makes it easier to get started and deploy our application stack to AWS.

First, install the Serverless Framework globally using this command in your terminal. And configure your AWS account using your key and secret key.

# Install serverless
npm install -g serverless
# Configure AWS
serverless config credentials --provider aws --key <your_key> --secret <your_secret_key>

Next, let's create our sample app using serverless.

serverless create --template aws-nodejs-typescript --path sqs-demo

This should create a project inside the sqs-demo folder with the following files.

We need to install all the NPM packages in this project. So let's go to that folder and install them. Following command be used, if you are unsure.

# Change the current directory to project directory
cd sqs-demo
# Install required NPM packages
npm install

We will also need aws-sdk to send a message. Please install it as developer dependencies using the following command.

npm i -D aws-sdk @types/aws-sdk

To test sending and receiving a message with SQS, we'd need two lambda functions. One to send a message and another to receive and process it. Let's go to handler.ts file and delete everything to start fresh.

Let's create two functions functionOne and functionTwo, where functionOne will send messages to SQS and functionTwo would receive and process that message. At this point, your handler.ts should have these lines of code.

// handler.ts
import { Handler, SQSHandler } from 'aws-lambda';
import 'source-map-support/register';

export const funtionOne: Handler = async (_event, _context) => {
  console.log("I am funtionOne");
}

export const funtionTwo: SQSHandler = async (_event, _context) => {
  console.log("I am funtionTwo");
}

We need to put these functions into the serverless.yml file in order for Serverless to create them. Simply remove the hello function under the functions section and add these two. We also need to give them SQS permission. serverless.yml file after adding these two functions would look like this.

service:
  name: sqs-demo

custom:
  webpack:
    webpackConfig: ./webpack.config.js
    includeModules: true

plugins:
  - serverless-webpack

provider:
  name: aws
  runtime: nodejs12.x
  region: ap-southeast-2
  apiGateway:
    minimumCompressionSize: 1024 # Enable gzip compression for responses > 1 KB
  environment:
    AWS_NODEJS_CONNECTION_REUSE_ENABLED: 1
  # SQS Permission given
  iamManagedPolicies:
  - 'arn:aws:iam::aws:policy/AmazonSQSFullAccess'

functions:
  funtionOne:
    handler: handler.funtionOne
  funtionTwo:
    handler: handler.funtionTwo

You can deploy these functions using Serverless easily. Just put this command.

serverless deploy

This should create two functions in AWS Lambda. Now we move into section two, where we actually start working on the SQS side.

Section Two - SQS Setup + Send Message

Let's login to AWS Console and search for Simple Queue Service and go there. Chose the appropriate region on the top right corner of the webpage and hit Get Started Now button. This should start the process of creating a new queue.

Enter the queue name you want to give it, I'd put sqs-demo. Now there are two types of the queue to choose from between Standard Queue and FIFO Queue. You can read the description to understand them better, but for this demo purpose, we are going with Standard Queue. Now hit the Quick-Create Queue button to create it.

Now the queue should be created and ready for use. Your page should look like this. Please note the ARN and URL, we will need it later.

Now let's enable our functionOne to send a message. To achieve this we need an SQS package from aws-sdk. Simply import it at the top of this file import {SQS} from 'aws-sdk and add the following code in the functionOne. I will explain it later.

try {
    const sqs = new SQS({
      region: 'ap-southeast-2'
    });
    const response = await sqs.sendMessage({
      MessageBody: 'hello world',
      QueueUrl: '<your queue url>'
    }).promise();
    console.log(`Message put on queue`, response);
  } catch (e) {
    console.log('Exception on queue', e);
  }

In this snippet, we are creating a new SQS object providing our region (mine is ap-southeast-2). Then we are using its sendMessage method to send our message to the queue. We'd need to provide MessageBody and QueueUrl, which we saved from earlier steps. Next, deploy it using the serverless deploy command.

Now we test our functionOne function. Go to AWS Lambda service and look for functionOne in Functions tab and click on its name to go to its detail page. On top right corner there is Test button, click on it, enter any name if asked for event name and hit Test. It should display execution result and there should be the console of the successful message sent as below.

Now if we go to SQS console, we should be seeing Messages available with the higher count.

At this point, we have successfully send a message to the queue. Now we work on functionTwo to receive this message from the queue and process it.

Section Three- SQS Receive and Process Message

First of all, let's add SQS event trigger for functionTwo. To do this we need to edit serverless.yml file under functionTwo section, we need to add the following after its handler.

events:
      - sqs: <your sqs queue arn number copied earlier>

Now the function should look like this.

Next, change the functionTwo to log all Records from the event object. You can replace functionTwo with the following code.

export const funtionTwo: SQSHandler = async (_event, _context) => {
  console.log("I am function two");
  for (let message of _event.Records) {
    console.log(`Handling message: ${message}`);
  }
}

All this functionTwo is doing is logging the message to console. You can change it to send an email or whatever you require the queue for. After this let's deploy it one last time.

As soon as your new functionTwo is deployed, it will start processing those messages in the queue and wait for new incoming messages.

You can also check logs for functionTwo. Mine is as below.

Overall, it is quite easy to create a new Amazon Simple Queue Service Queue, put message init and trigger another lambda function to pick those messages and process them.

You can find the complete project in GitHub

I would really appreciate your feedback.

Thank you for reading.