DEV Community

Cover image for Migrate python async worker to asynchrounous Lambda

Migrate python async worker to asynchrounous Lambda

kokospapa8 profile image Jinwook Baek ・8 min read

Original blog post


When you build a web service, there is a time that you need to implement simple async worker for background process. I am big fan of Django and whenever I need a async worker, I used celery or rq for convenience. celery or other async framework provides a lot of cool features but most of the time all I need is a simple background process that would not interrupt user experiences.

Overtime, as paradigm shifts to cloud, I have been thinking lambda as new async worker solution due to deployment complexity, scaling and cost constraint. Here are pros and cons for using lambda instead of celery on cloud environment(non-local environment).

Alt Text


  • FasS advantage
    • scaling benefit - cost, ops, etc
    • Resiliency
    • faster development
  • No need to maintain Broker (e.g elastic search redis)
  • Smaller side effect on code change
  • Fully-managed


  • FasS disadvantage
    • limited state
  • Hard to reuse app code
    • Django ORM or settings module
    • You can still use them with django package installed but inefficient
    • can't reuse code from django app
    • Django package is too heavy to use on lambda (still usable but too complex to setup
  • Latency on startup
  • Json formatted Invocation parameter
    • If you need to pass binary or big size paras, need to consider S3 or other medium


Most of early stage limitation are lifted as lambda is in mature state (e.g conccurent executions, layers, etc)

AWS Lambda quotas

  • timeout - 15min
    • If your async worker take more than 5min, you need to break up codes in to multiple lambda functions or use other distributed system such as EMR, etc.
  • Invocation payload
    • asynchronous - 256KB


  • VPC, Subnet, Security Group
    • If you are accessing Resources in private VPC, you need to consider placing lambda in same subnet.
    • If you are lambda needs to talk to outside world, private subnets needs NAT gateway
  • Maintenance


AWS recently came up with SAM(Serverless Application Model) which utilize cloudformation for easier deployment of lambda application. SAM provides functionality such as API gateway and statemachine, but we will only be using AWS::Serverless::Function resource for our purpose. You can consider using Zappa for other option.


refer to this repo for details



You need following AWS resource created in order to invoke asynchronous lambda.


In order use asynchronous lambda, you need to provide DLQ in case lambda fails and retry. Click link for details. I will create SNS topic in this post.

Alt Text

Asynchronous invocation


Let's assume that you have mysql db in public VPC or s3 and need to access them in order to process some data, you can add secrets to lambda env, but it is not a good idea to put secret in plaintext for env variable. You can use aws secret manger , ssm parameterstore or vault . I will use ssm parameterstore for this post. Refer to previous post for creating secrets.(parameter store section)

How to deploy django app to ECS Fargate part3


Alt Text


You need lambda execution role with following permission and polcies attached.

- AWSLambdaExecute
- SNS – sns:Publish
- kms - kms:Decrypt
- ssm - ssm:GetParameters, ssm:DescribeParameters, ssm:GetParameter
You need following permission if you want to place your lambda function in a designated VPC
- ec2 - ec2:CreateNetworkInterface, ec2:DeleteNetworkInterface, ec2:DescribeSecurityGroups


What is the AWS Serverless Application Model (AWS SAM)?

Install (on macOS)

Installing the AWS SAM CLI on macOS

$ aws configure
AWS Access Key ID [None]: your_access_key_id
AWS Secret Access Key [None]: your_secret_access_key
Default region name [None]: 
Default output format [None]:

$ /bin/bash -c "$(curl -fsSL"
$ brew tap aws/tap
$ brew install aws-sam-cli


$ sam init

Alt Text

This code will create following structure

   ├── events/
   │   └── event.json
   ├── hello_world/
   │   ├──
   │   ├──            #Contains your AWS Lambda handler logic.
   │   └── requirements.txt  #Contains any Python dependencies the application requires, used for sam build
   ├── template.yaml         #Contains the AWS SAM template defining your application's AWS resources.
   └── tests/
       └── unit/


Update template file if you need more info on syntax, refer to the link.


  • template.yaml
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  Sample SAM Template for sample-app

# More info about Globals:
    Timeout: 3

    Type: AWS::Serverless::Function # More info about Function Resource:
      CodeUri: sample_app/
        Type: SNS
        TargetArn: arn:aws:sns:<region>:<account_id>:sample-dlq
      Handler: app.lambda_handler
      Runtime: python3.8
      Description: sample lambda
        MaximumEventAgeInSeconds: 60
        MaximumRetryAttempts: 2
      FunctionName: SampleApp
      Role: arn:aws:iam::<account_id>:role/sample_lambda_execution_role
          S3_URL: ""

    Description: "Sample app Function ARN"
    Value: !GetAtt SampleAppdFunction.Arn
  • requirements.txt
  • This sample app receives image_url as parameter and puts the image on s3bucket name provided on s3 bucket. (AWSLambdaExecute policy has s3 access permission)
import boto3
import os
import requests

def lambda_handler(event, context):
    image_url = event['image_url']
    S3_BUCKET_PARAM = os.environ['S3_BUCKET_PARAM']

    # download

    image = requests.get(image_url).content

    ssm_client = boto3.client('ssm', region_name="ap-northeast-2")
    response = ssm_client.get_parameter(
    bucket_name = response['Parameter']['Value']

    s3_client = boto3.client('s3', region_name="ap-northeast-2")
    response = s3_client.put_object(

    return response


Once your logic code is ready, you can build it using following command

sam build -t template.yaml --region <region_name>

You can test your function with following command

$ sam local invoke
# if you need to provide environment variable or paramters, use following command
$ sam local invoke -e events/event.json --env-vars env.json
  "SampleAppdFunction": {
        "S3_BUCKET_PARAM_ARN": "parameterstore_arn",
  "image_url": ""


Once your function is tested locally, let's deploy the function to cloud. You have two options. Enter appropriate response to prompt

$ sam deploy --guided

Configuring SAM deploy

    Looking for samconfig.toml :  Found
    Reading default arguments  :  Success

    Setting default arguments for 'sam deploy'
    Stack Name [sample-app]: y
    AWS Region [ap-northeast-2]: y
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [Y/n]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: y
    Save arguments to samconfig.toml [Y/n]: y

output should look like this

    Looking for resources needed for deployment: Found!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
        A different default S3 bucket can be set in samconfig.toml

    Deploying with following values
    Stack name                 : sample-app
    Region                     : ap-northeast-2
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at

    Deploying with following values
    Stack name                 : sample-app
    Region                     : ap-northeast-2
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment

Waiting for changeset to be created..

CloudFormation stack changeset
Operation                                                    LogicalResourceId                                            ResourceType                                               
+ Add                                                        SampleAppdFunctionEventInvokeConfig                          AWS::Lambda::EventInvokeConfig                             
+ Add                                                        SampleAppdFunction                                           AWS::Lambda::Function                                      

Changeset created successfully. arn:aws:cloudformation:ap-northeast-2:982947632035:changeSet/samcli-deploy1602235720/1bb1b8a3-da32-4de9-9a14-49aaae3f066f

Previewing CloudFormation changeset before deployment
Deploy this changeset? [y/N]: y

2020-10-09 18:29:04 - Waiting for stack create/update to complete

CloudFormation events from changeset
ResourceStatus                                ResourceType                                  LogicalResourceId                             ResourceStatusReason                        
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::Function                         SampleAppdFunction                            Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::Function                         SampleAppdFunction                            -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_IN_PROGRESS                            AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           Resource creation Initiated                 
CREATE_COMPLETE                               AWS::Lambda::EventInvokeConfig                SampleAppdFunctionEventInvokeConfig           -                                           
CREATE_COMPLETE                               AWS::CloudFormation::Stack                    sample-app                                    -                                           

CloudFormation outputs from deployed stack
Key                 SampleAppdFunction                                                                                                                                               
Description         Hello World Lambda Function ARN                                                                                                                                  
Value               arn:aws:lambda:ap-northeast-2:982947632035:function:SampleApp                                                                                                    

Successfully created/updated stack - sample-app in ap-northeast-2

Check cloudformation and lambda console for successful deployment.
Alt Text
Alt Text

Invoke function in lambda console

Once Lambda is deployed, you can create test event in console.
Alt Text
You will get successful log data.
Alt Text

Invoke from python app

Now you need to invoke lambda function from your app code. Here are examples you can use.

import boto3
import json

    "AWS_DEFAULT_REGION": "ap-northeast-2",
    "ENV": "prod"
def main():
    lambda_client = boto3.client('lambda', region_name=SETTINGS['AWS_DEFAULT_REGION'])

    payload = {
        "image_url": ""

    ret = lambda_client.invoke(
        InvocationType="DryRun" if SETTINGS['ENV'] == "test" else "Event",

if __name__ == "__main__":


Celery or rq provides native or 3rd party too for monitoring such as sentry . There are some options for monitoring lambda functions but SAM application also provides minimal monitoring environment. Go to lambda service and application menu. Select Monitoring tab to dashboard and cloudwatch logs. You can also configure x-ray for tracing.
Alt Text
Alt Text


Let's talk about CD pipeline, I will use conditional trigger on github workflow and run the

Github workflow

following workflow file will build and deploy to lambda. You need to set secret key with AWS credential.
Alt Text

      - master

name: Lambda deployment
    name: Lambda deploy
    runs-on: ubuntu-latest

    - name: Checkout
      uses: actions/checkout@v2

    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ap-northeast-2

    - name: Build, and deploy
      run: |
        /bin/bash -c "$(curl -fsSL"
        test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
        test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
        test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
        brew --version
        brew tap aws/tap
        brew install aws-sam-cli
        sam --version

        sam build
        sam deploy --stack-name sample-app --region ap-northeast-2 -t .aws-sam/build/template.yaml --capabilities CAPABILITY_IAM --no-confirm-changeset --s3-bucket aws-sam-cli-managed-default-samclisourcebucket-fdwl3buruuk1

If you want to implement canary or linear deployment using codedeploy, refer to the following link for more detail.

Deploying serverless applications gradually


Once lambda is deployed, I removed 20% of my app code related to RQ. I also removed broker(Elasticsearch Redis) service anymore. Moreover since I was using ECS and EKS, I can remove additional task and pod for more api tasks and pods. I am pretty happy with the result that I don't need to manage another container on production. If you have simple workers running on RQ or Celery, I recommend that you try this setup. Thank you for reading.

Discussion (0)

Editor guide