In this article, I will describe how I got rid of two Lambda functions in my Data Lake loading process.
All thanks to the change announced in October 2021, Step Functions received native API integrations with over 200 AWS services. Yep, I call them native API integrations because for me, it's cloud-native low-code solution. AWS refers to them as AWS SDK Service Integrations, which is also fine 😉
Anyhow, I must admit that I waited for this particular update since summer.
Some prelude to the update has been available for a long time. I'm talking about this new way of calling Lambda functions from Step Functions. The difference is only visible in a language defining the machine. In the old configuration, we defined a Lambda call like that:
StateName: Type: Task Resource: !GetAtt functionName.Arn
and in the new way like that:
StateName: Type: Task Resource: arn:aws:states:::lambda:invoke Parameters: FunctionName: !GetAtt functionName.Arn Payload.$: $
At first glance, you can notice differences. Instead of your own resource, we use a universal client in the
Resource field, and provide the function name as a parameter.
This change to us - users - had no significance, both excerpts are equivalent. However, I'm eager to speculate that it was a milestone step on the way to providing the native integration of Step Functions with other AWS services via API.
From now on, we have access to 200 different integrations that we call similarly:
StateName: Type: Task Resource: arn:aws:states:::aws-sdk:<service>:<action> Parameters: ParametrApi1: value ParametrApi2: value
AWS SDK - feels like home 😃
In this way, you can copy a file from S3 (
arn:aws:states:::aws-sdk:s3:copyObject), run the EC2 instance(
arn:aws:states:::aws-sdk:ec2:startInstances), and so on. There are myriad possibilities.
And here is the biggest deal: it costs nothing!
I repeat: it's free 😃
I'm talking about the SDK API call (if it is free, and most of them are). For the state machine and the operation of the invoked service, of course, we pay as we used to be.
If it does not cost anything, then it is worth replacing Lambda functions that invoke other services with native integration. Thanks to that, we're going to save on Lambda cost, but also in terms of implementation (developer's time).
Because, as always: No code is the best code.
The over mentioned developer's time-savings, I must admit, was not so obvious. When I refactored two functions to native integrations, it took me 3 hours. A lot of time, but we all know that:
That's how it looked in my case because AWS API does not response with exactly what I needed at the moment. Variable names returned by AWS API must be processed, which is difficult, as there is no Lambda function at your disposal. And I didn't read the documentation 🤣
In my case, I changed the left implementation to the right one. I remind you, the entire process loads data to a Data Lake, hence references to the Glue service. In your case, this may be any other AWS service.
The entire state machine is called with a Payload, where the Glue Crawler name is given. The former Lambda function expected a variable named
crawlerName, while API method named
That was not so difficult to solve:
StartCrawler: Type: Task Resource: arn:aws:states:::aws-sdk:glue:startCrawler Parameters: Name.$: $.crawlerName
However, a big problem for me was to embrace the output data from this step because
startCrawler does not return any data, and by default, Step Functions relays the current result to the next step. Two steps further,
GetCrawler will need a Crawler name to check a condition (Has Crawler finished running?).
I solved the problem by reaching for a global context of the state machine and using the
ResultSelector filter to build my own object at the state exit.
StartCrawler: Type: Task Resource: arn:aws:states:::aws-sdk:glue:startCrawler Parameters: Name.$: $.crawlerName ResultSelector: crawlerName.$: $$.Execution.Input.crawlerName
This is just an illustrative example and does not include error handling and mapping them to the exit state.
That mapping realized what I required, an exit state with a variable
crawlerName and proper value.
The next step was to get the current state of the Crawler using the
GetCrawler method in the loop, that lasts until it is done. I used a similar solution here as before. The value of
state is used by the condition (Has Crawler finished running?) in a loop, and
crawlerName is used as an input parameter for the next state
Crawler.State comes from the results returned by the API
ResultSelector: crawlerName.$: $$.Execution.Input.crawlerName state.$: $.Crawler.State
Of course, it was!
Firstly, no code is the best code. Here, we replace the custom Lambda with declarative definition of states. This means that any person in the world knowing Step Functions will understand how it works, without deep-diving into our beautiful Lambda code.
Secondly, practice makes perfect.
Juggling Mapping values on the output is only difficult at the beginning. When you figure out how this works, you get more efficient in writing them. I hope this article will help you avoid many problems I have encountered 🙂
Thirdly, native integrations are so versatile, that from now on, practically every state machine you will use them. This means that the investment in mastering them will definitely pay off in the future projects.
Fourthly, the declarative code of the states is easier to reuse (copy-paste) than the Lambda code.