Using AWS SAM Cookiecutter Project Templates

For the upcoming Lamby work, we really want to improve our "getting started" experience. Creating a new Rails app on AWS Lambda should feel just as natural as running the $ rails new command line tool. One option to explore could be AWS' Serverless Application Repository. Commonly called SAR (czar, not es•ay•are) for short, it offers re-usable applications similiar to Heroku's deploy button.

Learning SAR is definitely on my list. However, initial research showed that it was not well suited for generator style project templates. Nor did it appear to put an emphasis on leaving the user with some local version-controllable artifact to move the application forward using various CI/CD tools. However, I have known that the SAM CLI tool does support a popular Python project called Cookiecutter which might be perfect for Lamby. So I set out to learn more about it.

👟 Want to Skip Ahead?

Like learning from the outside in? I took all the work (🚧) below and made a simple demo Lambda cookiecutter publicly available at customink/cookiecutter-ruby on GitHub. Feedback on your experience or how we can do this better is very much appreciated!

If you already have AWS SAM CLI installed, run this command.

$ sam init --location "gh:customink/cookiecutter-ruby"

If you want to avoid installing SAM CLI alltogether, you can use this Docker command.

$ docker run \
  --interactive \
  --volume "${PWD}:/var/task:delegated" \
  lambci/lambda:build-ruby2.7 \
  sam init --location "gh:customink/cookiecutter-ruby"

Commit your newly generated Lambda project to GitHub and the project's README.md for usage and next steps.

🚧 Learning Cookiecutter

Abstract learning without a goal to apply it, rarely helps me explore a technology. So for this exercise, I set out to build a suite of Lambda starters to help our Custom Ink teams adopt a "serverless-first" mindset by starting on the small/medium sized workload needs. Here is a checklist of features I think we needed.

• Mimic Rails' CLI "app_name" argument to customize file, folder, and class namespaces.
• Expose a configuration option for simple event-driven workload or API responses.
• Integrate a GitHub Actions workflow template to run tests and deploy.
• Leverage Docker containers to make development, test, and deployment easy.
• Remove the need to copy our SAM bin script conventions to new projects.
• Explore the new HTTP API version of API Gateway.

While learning, I made heavy use of the great documentation by the Cookiecutter team, notably the "Advanced Usage" section. Searching GitHub issues is also a fine art. Especially when you lack the proper project-specific keywords to use. Sharing both below.

• Cookiecutter Docs
• Cookiecutter GitHub Repo

🚧 Inflecting a Single Input

Cookiecutter's input parameters are driven by a cookiecutter.json file at the root of the repository. The user can customize these via the CLI prompt after running sam init. Most Cookiecutter projects have a project_name variable and this is the one we want to mimic Rails' new command which drives all naming conventions for the newly created project. So if someone typed "my_new_service" it would be converted to "MyNewService" and ideally used via Cookiecutter's template code like {{cookiecutter.class_name}}. These are the derived project name variables we needed:

• MyNewService - Class name or module namespace.
• my_new_service - File or folder namespace.
• my-new-service - Other places like function/stack names.

The problem is that Cookiecutter was not built to directly support this. Sure we could use some inline, possibly verbose, Python to transform a single variable. Especially since Cookiecutter does not natively support succinct rails-like ActiveSupport inflector methods. Nor the ability to import ad-hoc code. The solution is somewhat hacky, but involves these steps:

1. Leverage Cookiecutter's "pre" and "post" gen hooks.
2. Add some inflector code to the pre_gen_project.py file.
3. Write individual text files, one for each derived property.
4. Include those values within in your templates.
5. Clean up the project's build directory using post_gen_project.py file.

So instead of being able to use {{cookiecutter.class_name}} we have to use something like {% include "_cctmp/class_name.txt" %}. It also means that file renaming now needs to be done in the post gen hook via Python's os.rename method. A small price to keep the outside user experience clean.

How can Cookiecutter make this better? A little would go a long way. These two issues on their site are good ones to watch if you want to advocate for improving things.

• Allow Jinja Extensions to be Bundled
• Help generating GUID/UUID values for Context Variables

🚧 GitHub Actions for CI/CD

Your newly created Lambda project includes a simple test suite with some events to get you started. It even includes a .github/workflows/cicd.yml file to run those tests with GitHub Actions and even deploys your Lambda when changes land in master. We leverage the Configure AWS Credentials action which requires you to provide both a AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via their managed secrets interface. To survive the Cookiecutter templating process we had to quote this literal ${{ secrets.AWS_ACCESS_KEY_ID }} format like so.

aws-access-key-id: {{ "${{ secrets.AWS_ACCESS_KEY_ID }}" }}
aws-secret-access-key: {{ "${{ secrets.AWS_SECRET_ACCESS_KEY }}" }}

One easy way to make a "deploy" user with limited IAM capabilities is to do the first deployment yourself in a role with elevated permissions, and then create a user with more restricted permissions to do the updates. For example, assuming your local machine has the AWS CLI configured, this would deploy your Lambda:

$ STAGE_ENV=production ./bin/deploy

To create a simple "deploy" user with keys, we recommend getting started with the AWSLambdaFullAccess managed policy. To do so, in the "AWS Console → IAM → Users → Add User":

1. Check "Programmatic access" option.
2. Select "Attach existing policies directly" option.
3. Select "AWSLambdaFullAccess" policy.
4. Copy the "Access key ID" and "Secret access key"

Now within your GitHub repo page. Click Settings → Secrets → Add a new secret

• Name: AWS_ACCESS_KEY_ID Value: Value from the step above.
• Name: AWS_SECRET_ACCESS_KEY Value: Value from the step above.

🚧 BYO Events or HTTP API?

Our demo Cookiecutter project has an option to bring your own events or to make a simple HTTP microservice using the newly released HTTP API for API Gateway. I recommend Yan Cui's "HTTP API GOES GA TODAY!" post if HTTP API is new to you.

To enable the event options, we simply added this http_api variable to our cookiecutter.json file. It defaults to yes because it is the first option in the array. In our template code we can check for this using simple conditions like {%- if cookiecutter.http_api == "yes" %}.

{
  "project_name": "my_awesome_lambda",
  "http_api": ["yes", "no"]
}

I was overjoyed to find that implementing HTTP API via SAM was super terse and easy. Every AWS::Serverless::HttpApi property like DefinitionBody was optional and the default behavior was a full proxy. So this would be the conditional resource added in the template:

MyNewServiceHttpApi:
  Type: AWS::Serverless::HttpApi
  Properties:
    StageName: !Ref StageEnv

And this would be the conditional event added to the Lambda resource.

Events:
  MyNewServiceHttpApiProxy:
    Type: HttpApi
    Properties:
      ApiId: MyNewServiceHttpApi

Amazing! This is the first time I felt like HTTP events in SAM was well into the convention-over-configuration camp. Well done, team! That said, they do have some work ahead of them to enable all the features of AWS::ApiGatewayV2::Api. For example, tags are not passed down via CloudFormation and you have no way to use them with AWS::Serverless::HttpApi. Also, our bin/server script does not work since SAM CLI has yet to add support via the start-api command. But I'm sure that is coming soon and not technically needed in this demo project.

🚧 Everything else

I could go on and on but for brevity's sake, we can stop here. If you want to learn how to build your own Cookiecutter Lambda starter, use our customink/cookiecutter-ruby project for inspiration. Our internal ones span multiple languages (Ruby, Node, & Python) and do a lot more. SAM Cookiecutters are a great way to remove process or boilerplate burdens for your team. Here are some things we put an emphasis on:

Docker Images

By using docker-compose, we are able to provide a shared development, testing, and deployment process. Not only does this allow a unified developer experience, but it effectively commoditizes the deployment process, making it more portable. We do this for Lambda by leveraging @hichaelmart's community lambci/docker-lambda project. These are the same images that AWS SAM CLI uses too.

Strap Scripts

Following ideas like this scripts to rule them all project, we make heavy use of bin script conventions at Custom Ink. Every project no matter the language or implementation should have a unified interface to bootstrap, setup, and test the project. Our adoption of Lambda and Infrastructure as Code extends these concepts to deploy as well.

Multi-Account Deploys

From development to staging and production. Using a combination of environment variables like STAGE_ENV or RAILS_ENV along with our script conventions above, we make it easy to deploy to any AWS account. Docker compose also makes this easy by leveraging its environment option. It even support defaults. For example, this line will use the host's STAGE_ENV or default to development; combine this with AWS_PROFILE and you can get some really neat results:

- STAGE_ENV=${STAGE_ENV-development}

Resources

As always, thanks for reading! Below are some quick links for reference. If you have any ideas or feedback, leave a comment. I'd love to hear from you.

• Demo AWS SAM Lambda Cookiecutter Ruby Project
• Yan Cui's HTTP API Rundown
• AWS Serverless Application Repository: Lambda and Beyond
• The Cookiecutter GitHub Project
• LambCI Docker images used by SAM