Orchestrate AWS Lambdas using MongoDB - Part 1

This post assumes you are a developer with working knowledge on AWS, Lambda, EventBridge, MongoDB, NodeJs.

Usecase: Let's say you have several lambdas that you need to run some in parallel, in sequential and some in conditional, and their collective goal is to finish a task. To orchestrate these jobs we need a state-machine(https://statecharts.dev/what-is-a-state-machine.html), which ensures the jobs are run based on the given conditions and will terminated itself at the end.

Solution: For the similar problem, AWS Lambda provides a solution called step-functions where you could chain these lambdas, I felt its too difficult to establish and manage them, so I created a custom solution using MongoDB Triggers and AWS EventBridge. Now lets work on the process.

Components
_1. Statemachine Configuration: A configuration which consists of all the phases and jobs to run in order. This will be saved to a collection state-machines.

2. Status Event: Whenever a job status changes, i.e, success, failed, started we insert an event to database which will be consumed by other necessary jobs. This events will be saved statuses collections. This is the collection where MongoDB trigger is configured.

3. Statemachine Job: An additional job apart from the business jobs to update the state machine whenever a job completes or fails. This job only receives success events.

4. Orchestrator Job: An additional job apart from the business jobs to trigger the next jobs. This job only receives success events.

5. Notification Job: You could have another lambda to send messages to slack or any other messaging platform. This job only receives all events.

How it works:

1. A state machine will be created by either an API or any such equivalent method based on the requirements. And sends an status event as the start of the process. Now this event is received by statemachine_job, orchestrator_job and notifications_job

2. statemachine_job will update the corresponding statemachine with the finished job. Check if it has to either end the state machine or move onto next phase.

3. orchestrator_job will find the next job to run based on the given configuration, until nothing left to run.

4. notifications_job will send the current event message to configured messaging medium.

5. This cycle continues until all the phases finished & satisfied with finished jobs.

Set Up:
As mentioned we need to two collections state-machines, statuses. The model will be explained later. Create a trigger on statuses collection, for insert event such that whenever an item inserted into statuses it will be sent to the AWS Eventbridge. Set up the database trigger for a collection statuses. Follow this link to setup events for AWS Eventbridge https://www.mongodb.com/docs/atlas/triggers/.
The above created event bridge will invoke statemachine_job, orchestrator_job and notifications_job

Lets work on the technical implementation on the next part.