Building a larger Serverless application - Part 3: Modular Monorepos

I've decided that I'm using AWS, Serverless and Node.js for my actual code. Now to decide how to actually structure it.

I want to keep distinct areas actually distinct from each other. Allow for a truly modular approach. I also want to use a single repository and not lots of them, because it's just easier for a single person or small team that way.

Surprisingly, this is not easy.

AWS CloudFormation - and thus AWS SAM, and thus Serverless - builds our infrastructure in Stacks. A single stack represents a single unit of infrastructure, and everything that goes with it. So this would be Lambdas, DynamoDB tables, Queues, IAM roles, everything. It also supports what are called Nested Stacks, where one stack is actually composed of others.

Why use Nested Stacks?

One obvious question is - why should we use a nested stack? One of the benefits is that you can create and destroy the entire stack in one go. However, this isn't the real selling point.

AWS has a hard limit of 200 resources in any single stack. A stack that contains a single Lambda attached to an API Gateway, and a single DynamoDB table will contain up to 13 of these resources. This means that we can have 15 such stacks until we run out. Nested stacks let us circumvent this because we can have a parent stack containing many nested stacks, each of which have this 200 resource limit. We can nest these as deep as we want too, so we can structure this however we want.

Shared API Gateways

One obvious problem with multiple stacks is the URLs. The default behaviour would be that each stack has it's own API Gateway, and thus it's own URL. That is not helpful.

However, we can solve this. AWS already lets us share resources between stacks as long as they belong to the same IAM user. Serverless lets us achieve this as well.

What we will do is:

• Create a stack that contains nothing but the API Gateway
• Refer to this same API Gateway in all of our other stacks

This means that they will all be on the same URL, and we've only had to define it once.

Our API Gateway stack will look like this:

service: serverless-gateway

provider:
  name: aws
  stage: dev

resources:
  Resources:
    ServerlessGW:
      Type: AWS::ApiGateway::RestApi
      Properties:
        Name: ServerlessGW-${opt:stage, self:provider.stage}
  Outputs:
    apiGatewayRestApiId:
      Value:
        Ref: ServerlessGW
      Export:
        Name: ServerlessGW-restApiId-${opt:stage, self:provider.stage}
    apiGatewayRestApiRootResourceId:
      Value:
        Fn::GetAtt:
          - ServerlessGW
          - RootResourceId
      Export:
        Name: ServerlessGW-rootResourceId-${opt:stage, self:provider.stage}

This creates an API Gateway with the name "ServerlessGW" and the name of the stage. The use of the stage means that we can deploy multiple stages at the same time - very useful when we come to do testing!

We then refer to this in our other files as follows:

service: serverless-users

provider:
  name: aws
  apiGateway:
    restApiId:
      "Fn::ImportValue": ServerlessGW-restApiId-${opt:stage, self:provider.stage}
    restApiRootResourceId:
      "Fn::ImportValue": ServerlessGW-rootResourceId-${opt:stage, self:provider.stage}

The provider.apiGateway key tells Serverless that this stack will use that gateway, instead of defining it's own. Instantly we've got what we wanted.

Implementing a modular structure

Unfortunately, the tooling doesn't easily support nested stacks. AWS CloudFormation and AWS SAM do, but to do so every single nested stack needs to be in S3 - both the CloudFormation scripts and the archive of contents. Only when all of these sub-stacks are in S3 can you then deploy the parent one. Serverless doesn't have any way to support this at present. It does have some plugins that get some of the way there, but none of them do the exact job that I want.

As such, I've settled on a more DIY approach. It's possible to have stacks that are not nested but still relate to each other, so we can simply have all the various parts of the application as different stacks with no direct connections between them except in terms of names. It's not perfect, but it works.

This puts us into the following:

.
├── stacks
│   ├── gateway
│   │   ├── serverless.yml
│   └── users
│       ├── serverless.yml

Every directory inside stacks is then a single stack to be deployed.

The next problem is that we need to do this in the correct order.

My first thought was to do it a Node.js way. What we've actually got here is many projects that we want to orchestrate together. That should be easy, right? Wrong.

The obvious thing to do is stick within the tooling stack. I'm using Node, so lets use Node tools.

Lerna

There's a tool called Lerna that is explicitly designed for monorepos, and running tasks in multiple sub-modules correctly. So let use this.

In order to do this, you need to have a package.json in each module, and one at the top level. The top-level one will depend on lerna and serverless, and have a few scripts entries to help run things. Our per-module ones will then have scripts entries for the actual work.

// ./package.json
{
  "name": "serverless",
  "scripts": {
    "sls:deploy": "lerna run sls:deploy",
    "sls:remove": "lerna run sls:remove"
  },
  "devDependencies": {
    "lerna": "^3.17.0",
    "serverless": "^1.54.0"
  }
}

// ./stacks/gateway/package.json
{
  "name": "serverless-gateway",
  "scripts": {
      "sls:package": "sls package",
      "sls:deploy": "sls deploy",
      "sls:remove": "sls remove"
  }
}

We then need a lerna.json file to orchestrate this:

{
  "npmClient": "yarn",
  "packages": [ "stacks/gateway", "stacks/users" ],
  "version": "independent"
}

We can now run yarn sls:deploy at the top level and it will execute it in the sub-stacks, in the correct order, and deploy everything.

Success? Not quite. This works fantastically for setting things up, but completely fails when tearing them down. If we execute yarn sls:remove then it will try to remove gateway before users, and that will fail. And there is no way at present to get Lerna to run in reverse direction. (There is an open issue for it though, so you never know!)

Gulp

Next attempt. Gulp. There is a gulp plugin explicitly for serverless, and it will let you run serverless commands in directories. That's perfect.

So we can set up our top-level package.json file as follows:

{
  "name": "serverless",
  "scripts": {
    "sls:deploy": "gulp deploy",
    "sls:remove": "gulp remove"
  },
  "devDependencies": {
    "gulp": "^4.0.2",
    "serverless": "^1.54.0",
    "serverless-gulp": "^1.0.10"
  }
}

Then we have a gulpfile.js file to orchestrate this:

const gulp = require("gulp");
const serverlessGulp = require("serverless-gulp");

const paths = {
  serverless: ["gateway", "users"].map(p => `stacks/${p}/serverless.yml`)
};

gulp.task("deploy", () => {
  return gulp
    .src(paths.serverless, { read: false })
    .pipe(serverlessGulp.exec("deploy", { stage: "dev" }));
});

gulp.task("remove", () => {
  return gulp
    .src(paths.serverless.reverse(), { read: false })
    .pipe(serverlessGulp.exec("remove", { stage: "dev" }));
});

With the above, when we run yarn sls:deploy then we set everything up in the correct direction, but yarn sls:remove now tears everything down in the opposite direction. Perfect.

Only there's no easy way for Gulp to handle other monorepo tasks, like running tests across all the modules

Combined

So we can achieve this by combining both tools. Gulp for the serverless tasks that need to have a strict order, and Lerna for the tasks that can happen in any order. Lerna can use wildcards for finding the modules, so we only need to maintain our ordered list in the Gulp configuration and all is good.

Summary

Building a Serverless application in a monorepo is not easy, but it is doable. And if you're a small team or an individual then it's worth the effort up front to make the rest of the process smoother.