DEV Community

loading...
Cover image for Moving your cron jobs to AWS lambda with SAM cli

Moving your cron jobs to AWS lambda with SAM cli

tathagata profile image Tathagata ・5 min read

Every now and then, you need to a script to run at regular interval and do something interesting. Before the advent of serverless computing, cron jobs were the defacto way of doing this. But now we have several options that are more cost effective. ECS scheduled tasks and lambdas are two serverless alternatives for task scheduling in AWS. Depending on your workload you can decide which option would be the right choice for you. Lambdas, with 15-minute runtime limit are ideal for small workloads. Let's explore how we can get this working.

Install SAM cli

AWS Sam cli is a tool with an adorable chipmunk as mascot. It simplifies building and deploying lambdas.

pip install --user aws-sam-cli

Create a project

sam init --name lambda-cron --runtime python3.7
[+] Initializing project structure...

Project generated: ./lambda-cron

Steps you can take next within the project folder
===================================================
[*] Invoke Function: sam local invoke HelloWorldFunction --event event.json
[*] Start API Gateway locally: sam local start-api

Read lambda-cron/README.md for further instructions

[*] Project initialization is now complete

Lets take a look at the generated code

tree lambda-cron/
lambda-cron/
|-- README.md
|-- event.json
|-- hello_world
|   |-- __init__.py
|   |-- __pycache__
|   |   |-- __init__.cpython-37.pyc
|   |   `-- app.cpython-37.pyc
|   |-- app.py
|   `-- requirements.txt
|-- template.yaml
`-- tests
    `-- unit
        |-- __init__.py
        |-- __pycache__
        |   |-- __init__.cpython-37.pyc
        |   `-- test_handler.cpython-37.pyc
        `-- test_handler.py

5 directories, 12 files

This might seem an overkill of artifacts compared to a line in your cron tab. But if you think you are getting rid of the entire server, this doesn't look that unreasonable. Let's take a look at the non-obvious files.

event.json

sam's scaffolding is targets the most popular lambda use case - a function servicing an API gateway endpoint. We can ignore this.

app.py

This file is of primary importance, as this is where your code goes. By default, lambda_handler is the name of the function which you populate. For now, we will just limit it to a printing a string.

def lambda_handler(event, context):
    print("Do something interesting!")

requirements.txt

Standard stuff - your package dependencies go here.

template.yaml

This file is responsible for orchestrating the serverless-ness. It declaratively specifies what components make up your stack, where the stack is your entire software architecture deployed as one unit in AWS CloudFromation. If this is not magic, I don't know what is. I am a big fan of CloudFormation.

Note this file was generated to build a lambda function that gets triggered by an HTTP request to the API gateway endpoint. But our goal is to run it at scheduled intervals. Take a look at the section which describes the events.

      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

We need to replace it with the schedule information. Sam template supports schedule expression in both cron syntax and a simpler rate expression.

Every minute using rate

      Events:
        HelloWorld:
          Type: Schedule
          Properties:
            Schedule: rate(1 minute)

Every minute using cron

      Events:
        HelloWorld:
          Type: Schedule
          Properties:
            Schedule: cron(1 * * * ? *)

test_handler.py

This is pre-generated unittest stubs for pytest. You can install pytest (pip install pytest) and run pytest like usual.

Build

With your tests passing, it is now time to do the builds

cd lambda-cron/
t lambda-cron $ sam build --use-container
2019-03-30 19:42:24 Starting Build inside a container
2019-03-30 19:42:24 Building resource 'HelloWorldFunction'

Fetching lambci/lambda:build-python3.7 Docker container image......
2019-03-30 19:42:25 Mounting /Users/t/projects/python/playground/lambda-deep-dive/lambda-cron/hello_world as /tmp/samcli/source:ro inside runtime container

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Note the first time you run this, it will download the docker image it needs for running.

Local testing

One of the best functionality that sam offers is the ability to test your code locally in a docker container. This makes quick iteration possible without having to acutally deploy on AWS lambda environment.

sam local invoke --no-event -t template.yaml
2019-03-30 21:01:43 Found credentials in shared credentials file: ~/.aws/credentials
2019-03-30 21:01:43 Invoking app.lambda_handler (python3.7)

Fetching lambci/lambda:python3.7 Docker container image....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
2019-03-30 21:02:53 Mounting /Users/t/projects/python/playground/lambda-deep-dive/lambda-cron/hello_world as /var/task:ro inside runtime container
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
Do something intersting!
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72
REPORT RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72  Duration: 16.52 ms  Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 22 MB
null

Package your code

Let's create a bucket and pass it to the packaging command. Note we are passing a file called packaged.yaml which is the generated cloud formation template that we will be using in the next command to deploy it.

aws s3 mb s3://lambda-cron-bucket
sam package --s3-bucket lambda-cron-bucket --output-template-file packaged.yaml

Deploy

Finally deploy your cron job. You'll need to provide a name for your stack and tell cloudformation explicitly that you are ok with the creation of IAM related resources by passing the CAPABILITY_IAM flag.

aws cloudformation deploy --template-file /absolute/path/packaged.yaml --stack-name lambda-cron --capabilities CAPABILITY_IAM

Validate

Lets take a look if our cron is executing every minute.

sam logs -n HelloWorldFunction --stack-name lambda-cron --tail

You should see something like the following:

2019-03-30 20:35:19 Found credentials in shared credentials file: ~/.aws/credentials
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.613000 START RequestId: c2c8a898-dfab-450f-944b-d43d77741c35 Version: $LATEST
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.618000 Do something intersting!
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.638000 END RequestId: c2c8a898-dfab-450f-944b-d43d77741c35
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:29:12.638000 REPORT RequestId: c2c8a898-dfab-450f-944b-d43d77741c35  Duration: 24.83 ms  Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 63 MB
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.272000 START RequestId: 93645e34-850b-47b0-a4a3-00d10e502057 Version: $LATEST
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.278000 Do something intersting!
2019/03/31/[$LATEST]7cc1e71b6eb14c89881cb91d6c29b920 2019-03-31T01:30:12.298000 END 
...

Congratulations on your first serverless cron job!

What's with the cover picture?

Keeping with the theme of my other dev.to posts, it has nothing to do with the post. It's view of our beautiful city, Chicago, from my balcony.

Discussion (0)

pic
Editor guide