From reading AWS documentation understand you can trigger a Lambda from a SNS. If the Lambda fails AWS will retry it twice some time later. You can configure the SNS/Lambda to send the failed messages (after the 3 attempts) to a dead-letter queue. Why/When should I put a SQS between the SNS and the Lambda?
My professional interests are cloud and big data technologies, machine learning, and software development. I like to read source code and research papers to understand how stuff works.
Pronoun: He
When putting SQS between SNS and Lambda you don't need a dead letter queue (DLQ). In my opinion it depends on what you are doing with events in the DLQ. If those events are treated differently then you can use a DLQ. But if you are feeding them back into the input queue because you want to keep trying to process them until they succeed, just put SQS in front in the first place.
my thinking is that if a message failed to be processed 3 times (with delays between retries), must be something wrong with the message and needs to be verified kind of manually or will be going around the SQS forever. Another case can be that an external component (3rd party service, database...) is down but then do we want to immediately put the message back in the SQS and try to reprocess again? Better to move to the DLQ and wait little bit longer.
My professional interests are cloud and big data technologies, machine learning, and software development. I like to read source code and research papers to understand how stuff works.
Pronoun: He
The message processing might fail because there is something wrong with the message, yes. It might also fail however because a downstream system is not available and the Lambda fails because of this. Then it's a useful retry mechanism.
In the end it's an architecture decision and there are no right solutions, only trade-offs. But you're asking the right questions!
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
From reading AWS documentation understand you can trigger a Lambda from a SNS. If the Lambda fails AWS will retry it twice some time later. You can configure the SNS/Lambda to send the failed messages (after the 3 attempts) to a dead-letter queue. Why/When should I put a SQS between the SNS and the Lambda?
When putting SQS between SNS and Lambda you don't need a dead letter queue (DLQ). In my opinion it depends on what you are doing with events in the DLQ. If those events are treated differently then you can use a DLQ. But if you are feeding them back into the input queue because you want to keep trying to process them until they succeed, just put SQS in front in the first place.
Does that make sense?
Hi Frank,
my thinking is that if a message failed to be processed 3 times (with delays between retries), must be something wrong with the message and needs to be verified kind of manually or will be going around the SQS forever. Another case can be that an external component (3rd party service, database...) is down but then do we want to immediately put the message back in the SQS and try to reprocess again? Better to move to the DLQ and wait little bit longer.
Thanks for your input.
The message processing might fail because there is something wrong with the message, yes. It might also fail however because a downstream system is not available and the Lambda fails because of this. Then it's a useful retry mechanism.
In the end it's an architecture decision and there are no right solutions, only trade-offs. But you're asking the right questions!