CI/CD, AWS, and Serverless: 5 tips I learned the hard way

So you want to build a serverless app on AWS?

Great! There are lots of cool posts that show you how to get started.

An Intro to the Power of Serverless with AWS Amplify

Jared Short for Trek10 ・ May 29 '19

#aws #graphql #tutorial

But how do you get that app off your laptop and into the cloud? You could zip your code and upload it manually into the AWS console every time you change something, but that would get old pretty fast.

What you probably want is some kind of CI/CD (continuous integration / continuous delivery) system, where code is automatically tested and released to your environment when you push changes to a source control repository.

An example CI/CD pipeline

I've been building CI/CD pipelines for serverless on AWS for years, and I've picked up a few tricks along the way. Here are five tips I wish somebody had told me when I first got started.

Test code locally and services in the cloud

There are several ways to run a Lambda function on your local developer machine -- for example, the Serverless Framework's serverless-offline plugin or the AWS SAM CLI.

This works great for quickly testing changes to your code. But a serverless app is a lot more than just code: permissions, service configurations, and other things can go wrong as you glue services together to build a truly cloud-native application.

So local testing is not enough. The more external services you integrate with, the more you'll have to mock, and the more your tests will take on a tinge of unreality. You'll be testing a system that doesn't exist.

Instead, my rule of thumb is to test code locally and services in the cloud. This might involve the following steps:

1. Make an initial deployment in AWS that contains the definitions for services your code depends on (for example, deploying an underlying IAM role or a DynamoDB table).
2. Run your Lambda function locally as you develop and test the code, reaching out to cloud services for your dependencies.

This way, you can quickly iterate on code, but if something breaks on the cloud side, you'll know! When your Lambda function works locally as expected, you can push it to the cloud as well, where you should...

Prioritize end-to-end functionality tests over unit tests

As a general rule, we want to be writing less code in the serverless world. It's quite possible to write a useful service in AWS using mostly configuration. For example, you can build a CRUD API using API Gateway and DynamoDB alone - no Lambda functions in the middle required.

The more "serverless" you get, the less code you can usefully unit test, and the more you have to rely on tests of your deployed infrastructure.

Lately I've been using Cypress for end-to-end tests of serverless APIs. It's a Javascript test framework that works with assertion libraries like Jest and Chai. You can query your API directly, no inside knowledge of the code required. This is testing your service the way your users will experience it, so you can be sure your expectations match reality.

If you want to test AWS resources directly, a cool project to check out is Erez Rokah's AWS Testing Library, which lets you write tests directly against deployed resources like DynamoDB tables or SQS queues.

If you're writing tests, of course, you'll need some place to run them. AWS provides a managed build service called CodeBuild that you should be paying attention to, because...

AWS CodeBuild is hugely underrated

Ever had to babysit a build server that was constantly crashing because one rogue job ate all the disk space? That's not a very "serverless" thing to do.

Fortunately, AWS CodeBuild spins up a completely separate Docker-based build environment for every invocation of a build job. It's truly ephemeral compute, pay only for what you use, and it's pretty cheap -- 500 build minutes a month will run you a cool $2.00. The containers can take a few seconds to initialize, but that's way better than managing a pool of build runners that may flake out on you at any time.

Once you've got CodeBuild in your corner, you should take a look at another AWS tool for developers, because ...

AWS CodePipeline is the single best way to manage CloudFormation releases

CloudFormation is AWS's infrastructure-as-config service: you create a template that defines your Lambda functions, IAM roles, etc. (The Serverless Application Model, or SAM, which makes it easier to define serverless apps on AWS, is just CloudFormation with a little special sauce.)

Another annoying CI/CD thing I've had to do a lot is write scripts that poll AWS CloudFormation for updates. CodePipeline is really good at doing that for me. So I can trust it to manage my serverless infrastructure deployments as they roll from development, to staging, to production environments.

None of these services come without gotchas, of course. The most important one I can warn you about right now is this:

CodePipeline only supports one branch per pipeline (at least for now)

No doubt this will change someday, but at present you can trigger a CodePipeline from only one source, such as a repository branch. If you have branch-based workflows, this gets frustrating quickly.

One workaround is to dynamically create a new CodePipeline per branch using a Lambda function, which has worked pretty well for us at Trek10 -- well enough that we recently open-sourced the idea as an AWS Quickstart. That Quickstart contains the CloudFormation templates needed to spin up dynamic CodePipelines that include CodeBuild jobs, Lambda tests, and lots more -- so what are you waiting for? Give it a try, and hit me up in the comments if you have other questions.