Overview
Recently, I came across a need to finally use Lambda in AWS. I was looking to query an Oracle database and export the Oracle table to a CSV file. While using Python I realized the Oracle client is not available in Lambda so I needed a way to add libraries and dependancies in order to execute my code. Lambda layers allows you to do this. You can add several layers to your Lambda function. This article documents how to create the Oracle Layer for Python in Lambda with the required libraries.
The main limitation that I discovered using Lambda layers is the size limit of 250 megs for all your code and libraries. Fortunately, I was just under this limitation but there are several ways to get around this if you run into this problem. One is to break your code into various micro services. The other is to to use EFS and place all of your libraries in the EFS volume and mount the volume on your Lambda function.
Building the Oracle Layer
Optional, but preferred, commission an EC2 instance so you can start from a clean environment.
Verify the Python version
python3 --version
Create your project directory
mkdir project
Create a directory called python
mkdir python
Make library directory
mkdir lib
Install cx_Oracle Python library
pip3 install cx_Oracle -t python/
Download the latest Oracle Client from https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html
wget https://download.oracle.com/otn_software/linux/instantclient/218000/instantclient-basic-linux.x64-21.8.0.0.0dbru.zip -O oracli.zip
Unzip the Oracle Client into the library folder
unzip -j oracli.zip -d lib/
Ensure Libaio is insalled on the EC2 instance (it should be)
sudo yum install libaio -y
Copy Libaio to library
cp /lib64/libaio.so.1 lib/libaio.so.1
Zip up all your libraries to be used for the Lambda Layer
zip -y -r oracletable.zip python/ lib/
Copy your zipped library file to a S3 bucket so that the Lambda function can access it.
Adding Layer into Lambda
- Copy the url to be use when creating the layer
- Go to lambda and create a layer using AWS Lambda Layer documentation.
- Create the Lambda function using AWS Lambda Documentation.
- Add your code
- Add the layer your created
- Run the Lambda function
- Using CloudFormations
Below is an example of the CloudFormation code to create a Lambda Layer and Lambda function.
Note: Amazon Python Power Tools Layer documentation is at https://awslabs.github.io/aws-lambda-powertools-python/2.6.0/
Role:
Type: 'AWS::IAM::Role'
Properties:
RoleName: lambda-role-name
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
- s3.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: lambda-policy
PolicyDocument:
Version: "2012-10-17"
Statement:
# Add access to secrets manager to give access to Lambda function to use
- Effect: Allow
Action:
- secretsmanager:GetResourcePolicy
- secretsmanager:GetSecretValue
Resource: 'arn:aws:secretsmanager:<region>:<account>:secret:<secretname>'
- Effect: "Allow"
Action:
- s3:ListBucket
Resource:
- !Sub 'arn:aws:s3:::<S3 Bucket>'
- Effect: "Allow"
Action:
- s3:GetObject
- s3:PutObject
- s3:PutObjectAcl
Resource:
- !Sub 'arn:aws:s3:::<S3 Bucket>/*'
- !Sub 'arn:aws:s3:::<S3 Bucket>/*'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
- 'arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole'
# Create the Oracle Layer for Python
OrcaleLayer:
Type: AWS::Lambda::LayerVersion
Properties:
CompatibleRuntimes:
- python3.7
Content:
S3Bucket: <S3 Lambda Bucket with code>
S3Key: <layer ip file>
Description: "Oracle Layer"
LayerName: oracle-layer
# Create Lambda Function
OracleFunction:
Type: AWS::Lambda::Function
Properties:
FunctionName: oracle-lambda
Handler: oraclelambda.lambda_handler
Layers:
# Oracle Client Library Layer
- !Ref OracleLayer
# Add Python Power Tools
- 'arn:aws:lambda:us-east-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:13'
Role:
Fn::GetAtt:
- Role
- Arn
Environment:
#Pass Secret Name, Target Bucket, and S3 Prefix to the Python code
Variables:
SecretName: <secret name>
TargetBucket: <target bucket>
S3Prefix: <target prefix>
# Location of the Python Code
Code:
S3Bucket: <S3 Lambda Bucket with code>
S3Key: <pthon code zip file>
Runtime: python3.7
Timeout: 90
TracingConfig:
Mode: Active
VpcConfig:
SecurityGroupIds: <security group id>
- !Ref SecurityGroupId
SubnetIds:
- <subnet 1>
- <subnet 2>
What’s next?
Currently, the process for updating the layers and code is very manual. I’m looking to find ways to automate getting the CloudFormation templates and Lambda to get updated when new changes are made.
Oldest comments (0)