While working on a backend service, I came across this problem where I had to retrieve a set of data of a specific type, but that type wasn’t having any index and the data was also updating frequently.
Since the data was constantly changing (add/update), first thing that came to my mind was to have a trigger but then as I looked at the documentation of how to add triggers to Dynamo in AWS, I pursued a different solution :p
So, there were three straightforward solutions that were present before me without thinking much -
→ Complete scan of the table and filter the results
This solution was screaming “slow” right from the moment it entered my mind.
→ Add an Index for that type
There were a couple of reasons I didn’t go along with this idea
- That wasn't a very frequently queried data
- Already had so many indexes so didn't want to add one more just yet before trying other solutions
→ Run a cron-job
A cronjob that will scan the table and store the result in a cache or in another table.
A Great man once said, If you can't think of anything else, run a cron-job.
Back in my mind, I wasn’t quite convinced with any of the above solutions and so I gave Dynamo db triggers try. I was aware of Dynamo db streams but never tried them.
DynamoDB Streams are basically triggers like that we have in Relational DBs but the only difference is that it generates a stream of events when there is a data change and you can easily pipe that stream into Kinesis or a Lambda (In my case it was lambda).
AWS made the things look difficult but the process was not complex at all.
All I had to do was the following -
- Setup a stream on the dynamodb table which was just a one click action.
- Create a new lambda function and attach it to the dynamodb stream which was also fairly easy
- Rest was basic stuff of listening the data then processing it and finally saving it to the new table
DynamoDB streams are much more than what I have covered in my use case. You can read more about it here : -https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/