In Depth Guide to Serverless APIs with AWS Lambda and AWS API Gateway (Part 1)

TL;DR The repository with an example project can be found on GitHub

Building your software products around an API is THE thing for years now and doing it with serverless technology right from the start seems rather intriguing for many reasons — on-demand pricing, auto-scaling, and less operational overhead.

Why

When I read what people are talking about serverless technology, I have the feeling there are still many questions open.

How to

1. use a query string or route parameters?
2. use the params or body of a POST request?
3. set HTTP status codes?
4. set a response header?
5. call a Lambda from a Lambda?
6. store keys for third-party APIs?

So I decided to write an article about how to build an API with serverless technology, specifically AWS Lambda and API-Gateway.

This article is split into two parts.

This one, the first, is about the architecture, setup, and authentication.

The second one is about actual work, uploading images, tagging them with the third-party API, and retrieving the tags and the images later.

What

We will build a simple RESTful image tagging API. It lets us upload images, tags them automatically with a third-party API called Clarifai, and stores the tags and image names into Elasticsearch.

The work-flows I have in mind are:

• Signing up and in
• Uploading an image
• Getting all tags of all images
• Getting all images and their tags
• Getting an image by tag

How

For the API, we use API-Gateway, which is Amazons all-round serverless HTTP solution. It's optimized for RESTful APIs and works as the entry-point for our system.

Amazon Cognito handles the authentication. Cognito is a managed serverless authentication, authorization, and data synchronization solution. We use it to sign our users up, and in so we don't have to reinvent the wheel here.

The actual computing work of our API is done by AWS Lambda, a function as a service solution. Lambda is a serverless event-based system that allows triggering functions when something happens, for example, an HTTP request hit our API, or someone uploaded a file directly to S3.

The images are stored in an Amazon S3 bucket. S3 is a serverless object-based storage solution. It allows direct access and uploads of files via HTTP and can, as API-Gateway, be an event source for Lambda.

The tag data and the corresponding image-names are stored in Amazon Elasticsearch Service; an AWS managed version of Elasticsearch. Sadly this is not serverless but built on EC2 instances, but there is a free tier for the first 12 months. Elasticsearch is a very flexible document storage and comes with a powerful query language.

A non-AWS service called Clarifai provides image recognition magic. AWS has its service for this, called Rekognition, but by using Clarifai, we can learn how to store third-party API keys.

The whole infrastructure we build is managed by AWS SAM, the Serverless Application Model. SAM is an extension for AWS CloudFormation that reduces some boilerplate code needed to set up AWS Lambda and API-Gateway resources.

We use AWS Cloud9 as an IDE because it comes with all the tools and permissions pre-installed to use AWS resources.

Prerequisites

• A browser
• An AWS account with a Cloud9 environment. (Setup can be found here in the first 5 steps.)
• Basic JavaScript knowledge
• Basic Lambda, API-Gateway, and SAM knowledge

Setup

Let's start by getting a basic serverless API going that just implements a login with the help of AWS SAM, API-Gateway and Cognito.

mkdir serverless-api
cd serverless-api
mkdir functions
touch template.yaml

First, we create the folder structure and a template.yaml file that holds the definition of the infrastructure we create with SAM.

Implementing the SAM Template

The content of our SAM template is as follows:

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: "An example REST API build with serverless technology"

Globals:
  Function:
    Runtime: nodejs8.10
    Handler: index.handler
    Timeout: 30
    Tags:
      Application: Serverless API

Resources:
  ServerlessApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
      Cors: "'*'"
      Auth:
        DefaultAuthorizer: CognitoAuthorizer
        Authorizers:
          CognitoAuthorizer:
            UserPoolArn: !GetAtt UserPool.Arn
      GatewayResponses:
        UNAUTHORIZED:
          StatusCode: 401
          ResponseParameters:
            Headers:
              Access-Control-Expose-Headers: "'WWW-Authenticate'"
              Access-Control-Allow-Origin: "'*'"
              WWW-Authenticate: >-
                'Bearer realm="admin"'

  # ============================== Auth =============================
  UserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      UserPoolName: ApiUserPool
      LambdaConfig:
        PreSignUp: !GetAtt PreSignupFunction.Arn
      Policies:
        PasswordPolicy:
          MinimumLength: 6

  UserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref UserPool
      ClientName: ApiUserPoolClient
      GenerateSecret: no

  PreSignupFunction:
    Type: AWS::Serverless::Function
    Properties:
      InlineCode: |
        exports.handler = async event => {
          event.response = { autoConfirmUser: true };
          return event;
        };

  LambdaCognitoUserPoolExecutionPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt PreSignupFunction.Arn
      Principal: cognito-idp.amazonaws.com
      SourceArn: !Sub "arn:${AWS::Partition}:cognito-idp:${AWS::Region}:${AWS::AccountId}:userpool/${UserPool}"

  AuthFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: functions/auth/
      Environment:
        Variables:
          USER_POOL_ID: !Ref UserPool
          USER_POOL_CLIENT_ID: !Ref UserPoolClient
      Events:
        Signup:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /signup
            Method: POST
            Auth:
              Authorizer: NONE
        Signin:
          Type: Api
          Properties:
            RestApiId: !Ref ServerlessApi
            Path: /signin
            Method: POST
            Auth:
              Authorizer: NONE

Outputs:
  ApiUrl:
    Description: The target URL of the created API
    Value: !Sub "https://${ServerlessApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/"
    Export:
      Name: ApiUrl

First, we define a ServerlessApi resource. In a SAM template, this resource is usually created implicitly for us, but when we want to enable CORS or use an authorizer, we need to define it explicitly.

We also set the default authorizer for all routes to be the CognitoAuthorizer and then configure it right away with a Cognito user pool we define later in the template.

Then we define an UNAUTHORIZED gateway response because API-Gateway won't add CORS headers to our responses on its own. In our Lambda backed routes, we can do this via JavaScript code, but when we dont't have permissions to call them, this doesn't help. Gateway responses are a way to add headers directly with API-Gateway.

Next, we define the Cognito related resources:

• UserPool is the part of Cognito that holds our users' accounts.
• UserPoolClient is the part of Cognito that allows programmatic interaction with a user pool.
• PreSignupFunction is a Lambda function that is called before the actual signup with the user pool happens. It allows us to activate users without the need to send an email to them. It only has a few lines of code, so we inline it.
• LambdaCognitoUserPoolExecutionPermission grant the manged Cognito service the permission to execute our PreSignupFunction.

Finally, we define a Lambda function that is called by API-Gateway routes. Specifically POST /signup and POST /signin. This AuthFunction also gets the user pool ID and the user pool client ID as environment variables. These values are dynamically generated at deployment time, but environment variables are a way to pass such values from SAM/Cloudformation into a function.

With the Global/Function/Handler we set at the top and the CodeUri in our AuthFunction Properties, we can determine where our JavaScript file has to be and how it has to export a handler function for Lambda.

...
Handler: index.handler
...
CodeUri: functions/auth/

The file has to be called index.js, it has to export a function called handler and it has to be in the functions/auth/ directory.

The Auth property of our endpoint definitions are set to Authorizer: NONE so API-Gateway lets us request the endpoints without the need of a token.

At the end of the file, we have one output called ApiUrl; we use it after the deployment to fetch the actual API URL from CloudFormation.

Implementing the Auth Lambda Function

We created the functions directory at the beginning, so we only need to add an auth directory with an index.js

The index.js holds the following code:

const users = require("./user-management");

exports.handler = async event => {
  const body = JSON.parse(event.body);
  if (event.path === "/signup") return signUp(body);
  return signIn(body);
};

const signUp = async ({ username, password }) => {
  try {
    await users.signUp(username, password);
    return createResponse({ message: "Created" }, 201);
  } catch (e) {
    console.log(e);
    return createResponse({ message: e.message }, 400);
  }
};

const signIn = async ({ username, password }) => {
  try {
    const token = await users.signIn(username, password);
    return createResponse({ token }, 201);
  } catch (e) {
    console.log(e);
    return createResponse({ message: e.message }, 400);
  }
};

const createResponse = (
  data = { message: "OK" },
  statusCode = 200
) => ({
  statusCode,
  body: JSON.stringify(data),
  headers: { "Access-Control-Allow-Origin": "*" }
});

It requires a user-management.js to do its work, but we talk about this later. First, let us look at what this file does.

As we said in the template.yaml, It exports a handler function that receives the HTTP request event from API-Gateway when someone sends a POST request to the /signup or /signin endpoints.

Here we can see an answer to one of the most frequent questions.

How to access the request body?

exports.handler = async event => {
  const body = JSON.parse(event.body);
  ...
};

The first parameter of a JavaScript Lambda function holds an event object. When the function is called with an API-Gateway event, this object has a body attribute that holds a string of the request body if the client sent one.

In our case, we expect a JSON with a username and password so we can create new user accounts or sign users into their accounts, so we need to JSON.parse() the body first.

Next, we have two functions, signUp and signIn that use the required user-management module, here called users to do their work. They are both passed username and password as arguments.

createResponse is a utility function that builds a response object. This object is what a Lambda function has to return.

Here, we have an answer to other questions from the beginning.

How to set HTTP status codes?

exports.handler = async event => {
  ...
  return { statusCode: 404 };
};

Every Lambda function needs to return a response object. This object has to have at least a statusCode attribute. Otherwise, API-Gateway considers the request failed.

How to set an HTTP response header?

exports.handler = async event => {
  ...
  return {
    statusCode: 200,
    headers: {
      "Access-Control-Allow-Origin": "*"
    }
  };
};

The response object our Lambda function has to return can also have a headers attribute, it has to be an object with header names as the objects keys and the header values as the values of the object.

Here, we can see that we need to set CORS headers manually. Otherwise, browsers won't accept the response.

Now, let's look at the user-management.js file we required.

global.fetch = require("node-fetch");
const Cognito = require("amazon-cognito-identity-js");

const userPool = new Cognito.CognitoUserPool({
  UserPoolId: process.env.USER_POOL_ID,
  ClientId: process.env.USER_POOL_CLIENT_ID
});

exports.signUp = (username, password) =>
  new Promise((resolve, reject) =>
    userPool.signUp(username, password, null, null, (error, result) =>
      error ? reject(error) : resolve(result)
    )
  );

exports.signIn = (username, password) =>
  new Promise((resolve, reject) => {
    const authenticationDetails = new Cognito.AuthenticationDetails({
      Username: username,
      Password: password
    });

    const cognitoUser = new Cognito.CognitoUser({
      Username: username,
      Pool: userPool
    });

    cognitoUser.authenticateUser(authenticationDetails, {
      onSuccess: result => resolve(result.getIdToken().getJwtToken()),
      onFailure: reject
    });
  });

First, we use the node-fetch package to polyfill the fetch browser API. This polyfill is needed for the amazon-cognito-identity-js package we require next, which does the heavy lifting of talking to the Cognito service for us.

We create a CognitoUserPool object with the help of the environment variables we got from the SAM template and use this object inside the signIn and signUp functions we export.

The most exciting part is in the signIn function that fetches an ID token from the Cognito user pool and returns it.

This token needs to be passed inside an Authorization request header with a Bearer prefix on every request to our API, and it needs to be re-fetched when it expires. The CognitoAuthorizer in the API configuration of our SAM template told API-Gateway how to handle everything else with Cognito.

Now we need to install the packages we used. The node-fetch polyfill package and the amazon-cognito-identity-js package.

For this, we need to go into the functions/auth directory and init an NPM project, then install the packages.

npm init -y
npm i node-fetch amazon-cognito-identity-js

All files inside the functions/auth directory are uploaded to S3 when we deploy the function to Lambda with the rest of our API.

Deploying the API

To deploy our project to AWS, we use the sam CLI tool.

First, we need to package our Lambda function source and upload it to an S3 deployment bucket. We can create this bucket with the aws CLI.

aws s3 mb s3://<DEPLOYMENT_BUCKET_NAME>

The bucket name has to be globally unique, so we need to invent one.

When the creation worked, we need the DEPLOYMENT_BUCKET_NAME to package our Lambda source.

sam package --template-file template.yaml \
--s3-bucket <DEPLOYMENT_BUCKET_NAME> \
--output-template-file packaged.yaml

This command creates a packaged.yaml file that holds URLs to our packaged Lambda sources on S3.

Next, we need to do the actual deployment with CloudFormation.

sam deploy --template-file packaged.yaml \
--stack-name serverless-api \
--capabilities CAPABILITY_IAM

If everything went well, weuse the following command to get the APIs base URL.

aws cloudformation describe-stacks \
--stack-name serverless-api \
--query 'Stacks[0].Outputs[?OutputKey==`ApiUrl`].OutputValue' \
--output text

The URL should look something like this:

https://<API_ID>.execute-api.<REGION>.amazonaws.com/Prod/

This URL can now be used to issue POST requests to the /signup and /signin endpoints we created.

Conclusion

This article was the first of two articles about creating serverless APIs on AWS. We talked about the motivations to do so, the AWS services we need to get things done and implemented token-based authentication with the help of AWS Cognito.

We also answered a few of the most pressing questions that arise when building an API with API-Gateway and Lambda.

In the next part, we implement two Lambda functions that allow us to work with the API.

• ImagesFunction is responsible for creating upload links for an S3 bucket.
• TagsFunction handles the S3 uploads, third-party API integration, and the listing of created tags.

Originally published on the Moesif Blog