Send emails asynchronously with Sendgrid and node.js, AWS SQS, AWS Lambda

Recently in TheGoodPsy we experienced an uptick in the number of signups & concurrent users.
This is generally a good thing, an increase in traffic means that a project is gaining traction (or a DDoS attack but we surely hope it wasn't that and that kind of attacks would generate way more traffic than the numbers we were experiencing).

Our backend is in node.js and we deploy in the AWS ecosystem. As part of our user flow, we use email both for notifications as a response to a user action (transactional) and some custom marketing automation. Sendgrid is our service of choice for those matters.

Initially in order to send emails the approach was quite simple, send an email to a user using a given template whenever we saw the need to:

const sendCustomMailTemplate = async (templateId, toMail, data) => {
    const msg = {
      to: toMail,
      from: fromMail,
      templateId: templateId,
      dynamicTemplateData: {
        ...data
      },
    };
    await sgMail.send(msg);
}

This approach worked fine at the beginning. The problem with it though is that sending an email is synchronous and blocks the main thread. If enough emails are needed to be sent at the same time we'd run into problems.

A possible solution for this would be to use native node.js Worker Threads and offload the email sending into them. That would be initially possible but those workers live in memory and we'd lose on features like persist, retry on failures, batch processing of multiple emails, logging, etc... Or at least we'd need to implement them manually.

Eventually, we settled for a simpler solution using AWS SQS queues.

By using an external queue to process email sending jobs, we'd offload all the email processing & sending outside of the backend leaving it free to do other work.

The architecture of the new system is simple:

The backend enqueues messages in SQS and a lambda function consumes them and is in charge of processing them and sending the emails. As simple as that. This situation can be leveraged with lambda because one of the triggers that lambdas integrate with are messages being added to a SQS queue, so all the heavy lifting is done for us, nice.

Now, we can manually create both the SQS queue and the lambda function through the AWS console and play around with their multiple parameters, but the Serverless framework wraps around all that complexity and provides developers with a bunch of tooling to automate and easily create serverless applications. It even has a Node SQS worker template that we can use to serve as a starting point.

The template utilizes the lift plugin to leverage AWS CDK and expand Serverless' functions to avoid all the yak-shaving at the beginning. One can always tweak the parameters afterward, be it from the serverless.yml configuration file or directly from the AWS console.

So to have a working SQS/Lambda pair with the trigger already configured, we create the function:

$ serverless create --template aws-nodejs --name email-consumer

We install the necessary packages:

$ npm install --save @sendgrid/client @sendgrid/mail serverless serverless-lift

And we tweak the serverless.yml configuration to use serverless-lift and set Sendgrid's credentials:

service: email-sender

frameworkVersion: '3'

provider:
  name: aws
  stage: <your stage>
  region: <your_region>
  runtime: nodejs14.x

constructs:
  email-queue:
    type: queue
    worker:
      handler: handler.consumer
      environment:
        SENDGRID_API_KEY: <SENDGRID_API_KEY>

plugins:
  - serverless-lift

When we hit deploy, Serverless will take care of creating the resources:

serverless deploy



Deploying worker-project to stage dev (us-east-1)

✔ Service deployed to stack worker-project-dev (175s)

functions:
  worker: worker-dev-jobsWorker (167 kB)
jobs: https://sqs.us-east-1.amazonaws.com/000000000000/email-sender

The URL in jobs is your SQS URL. What's left is to program the consumer logic in the lambda and substitute calls to sendCustomMailTemplate() in the backend by our new enqueuing logic: enqueueMail().

The consumer:

const setupMailClient = async () => {
    sgMail.setApiKey(API_KEY);
    sgClient.setApiKey(API_KEY);
    sgClient.setDefaultHeader("Content-Type", "application/json");
}

const sendCustomMailTemplate = async ({
    ... we have the same sync email sending logic here ...
}


const consumer = async (event) => {
    //Setup the mail client with the Sendgrid API key
    await setupMailClient();

    //Go through all records (1 by default, change in serverless.yml)
    //extract info about the email and send it calling Sendgrid.
    const promises = event.Records.map(async record => {
        const { body, messageAttributes } = record;

        const parsedBody = JSON.parse(body);

        const {
            templateId,
            toMail,
            data
        } = parsedBody;

        await sendCustomMailTemplate({
            templateId,
            toMail,
            data
        });
    });

    await Promise.all(promises);
}

And in the backend, the enqueueMail() logic:

const enqueueMail = async ({ templateId, toMail, data }) => {
    const AWS_SQS_EMAIL_QUEUE_URL = "YOUR_SQS_URL>";

    const messageBody = JSON.stringify({
        templateId,
        toMail,
        data
    });

    const messageParams = {
        // Remove DelaySeconds parameter and value for FIFO queues
        // DelaySeconds: 10,
        MessageAttributes: {},
        MessageBody: messageBody,
        QueueUrl: AWS_SQS_EMAIL_QUEUE_URL
    };

    const promise = new Promise(
        (resolve, reject) => 
            SQS.sendMessage(messageParams, (err, data) => err ? 
                reject(err) : 
                resolve(data)
            )
    );
    return promise;
}

And that's it!
Have fun!

Permalink: https://www.estebansastre.com/send-emails-node-sendgrid-aws-sqs-lambda/