Update: Well turns out the new provisioned concurrency feature is mostly unusable. The hidden cost of Lambda provisioned concurrency pricing explains why but unfortunately the cost is not at all hidden. Uclusion was billed $3,347.63 for 803,424,893.570 Lambda-GB-Second on Lambdas that were not costing anything under the “warmup” way. We are disputing the charge but will have to revert to trying to solve cold starts with the synthetic traffic plugin.
Uclusion uses Serverless to deploy Lambdas in AWS. There are many aspects to our configuration so this post just walks you through a basic abbreviated file. You can use Serverless documentation to understand the key words.
service: uclusion-market-api
plugins:
- serverless-plugin-aws-alerts
custom:
alerts:
topics:
alarm:
topic: uclusion-market-api-${opt:stage, self:provider.stage}-alerts-alarm
notifications:
- protocol: email
endpoint: [an email address]
alarms:
- functionErrors
- functionThrottles*
*provider:
name: aws
runtime: python3.7
region: us-west-2
versionFunctions: false
usagePlan:
quota:
limit: ${ssm:/usageplan/limit}
period: DAY
throttle:
burstLimit: 400
rateLimit: 100
apiGateway:
apiKeySourceType: AUTHORIZER
environment:
usersServicePrefix: uclusion-users-dev-
marketsServicePrefix: uclusion-markets-dev-
USER_POOL_ID: us-west-2_${ssm:/userpool/id}
COGNITO_CLIENT_ID: ${ssm:/cognito/client/id~true}
GLOBAL_CAPABILITY_SECRET_KEY: ${ssm:/capability/secret/key~true}
CAPABILITY_SIGNING_ALGORITHM: HS256
myRegion: us-west-2
package:
exclude:
- node_modules/**
- .idea/**
- env/**
- tests/**
- README.md
- setup.cfg
- package.json
- package-lock.json
functions:
authorizerFunc:
provisionedConcurrency: 5
handler: authorizers/long_capability_auth.lambda_handler
role: arn:aws:iam::${ssm:/account/id}:role/requests/generalapi/policy/GeneralAPIRoleShared
layers: ${file(layers.yml):${opt:stage, self:provider.stage}}
update_investment:
provisionedConcurrency: ${ssm:/account/provisioned}
role: arn:aws:iam::${ssm:/account/id}:role/requests/generalapi/policy/GeneralAPIRoleShared
handler: handlers/investment_update.update
layers: ${file(layers.yml):${opt:stage, self:provider.stage}}
timeout: 20
events:
- http:
path: invest
method: post
private: true
cors: true
authorizer: ${file(authorizer.yml):${opt:stage, self:provider.stage}}
remove_investment:
provisionedConcurrency: ${ssm:/account/provisioned}
role: arn:aws:iam::${ssm:/account/id}:role/requests/generalapi/policy/GeneralAPIRoleShared
handler: handlers/investment_remove.delete
layers: ${file(layers.yml):${opt:stage, self:provider.stage}}
timeout: 20
events:
- http:
path: invest/{investible_id}
method: delete
private: true
cors: true
authorizer: ${file(authorizer.yml):${opt:stage, self:provider.stage}}
get_market:
provisionedConcurrency: ${ssm:/account/provisioned}
role: arn:aws:iam::${ssm:/account/id}:role/requests/generalapi/policy/GeneralAPIRoleShared
handler: handlers/get_market.get
layers: ${file(layers.yml):${opt:stage, self:provider.stage}}
events:
- http:
path: get
method: get
private: true
cors: true
authorizer: ${file(authorizer.yml):${opt:stage, self:provider.stage}}
update_market:
provisionedConcurrency: ${ssm:/account/provisioned}
role: arn:aws:iam::${ssm:/account/id}:role/requests/generalapi/policy/GeneralAPIRoleShared
handler: handlers/market_update.update
layers: ${file(layers.yml):${opt:stage, self:provider.stage}}
events:
- http:
path: update
method: patch
private: true
cors: true
authorizer: ${file(authorizer.yml):${opt:stage, self:provider.stage}}
resources:
Resources:
GatewayResponseDefault4XX: *# See https://serverless.com/blog/cors-api-gateway-survival-guide
*Type: 'AWS::ApiGateway::GatewayResponse'
Properties:
ResponseParameters:
gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
gatewayresponse.header.Access-Control-Allow-Headers: "'*'"
ResponseType: DEFAULT_4XX
RestApiId:
Ref: 'ApiGatewayRestApi'
Here the authorizer.yml is
dev:
name: authorizerFunc
authorizer_type: TOKEN
identitySource: method.request.header.Authorization
and the layers.yml is
dev:
- ${cf:udcommon-layer-dev.UdcommonLayerExport}
- ${cf:ucommon-layer-dev.UcommonLayerExport}
- ${cf:ubcommon-layer-dev.UbcommonLayerExport}
and you use the environment section in your Lambda Python code like so
os.environ['usersServicePrefix']
One of the more confusing points when you initially setup using AWS Lambdas is the API Gateway stage feature. So far Uclusion has not found a way to make use of that feature at all. We created an organization with three accounts — dev, stage and production and use the same Serverless yml to deploy to each of them (using CircleCI). So each environment has only one API Gateway stage.
In theory you could canary test a new version of the API in production by having a second stage but in practice the new stage shares SSM configuration (see ssm: usage above), DynamoDB, DynamoDB streams, etc. and so if the canary dies the whole site might die as well.
Another confusing aspect is the provisionedConcurrency. Any call you make to an outside static resource like os.environ[‘usersServicePrefix’] must be globally declared in your Python file — not inside a method. Otherwise instead of the call being made once when the Lambda is provisioned it will be made every time the method is called.
Also confusing with provisionedConcurrency you will see in your logs something like
Init Duration: 2285.65 ms
when you run a Lambda. Even more confusing is that if you run that Lambda twice in a row the init duration won’t be there on the second call. I can’t say for certain what is happening but it was the same with the old warm-up plugin running every 5 minutes.
Finally make sure to thoroughly read the API Gateway limits ahead of time; I promise that will be time well spent!
Top comments (0)