My professional interests are cloud and big data technologies, machine learning, and software development. I like to read source code and research papers to understand how stuff works.
Pronoun: He
I assume you already setup a bucket notification that publishes messages to an SNS topic. If you want to use SNS to receive the notifications (because you want to fan-out) then you can either subscribe via HTTP or SQS.
In case you are using HTTP, the EC2 instance needs to expose an HTTP endpoint. If you want a more resilient and secure setup then you can subscribe with SQS and make the EC2 instance poll from SQS.
In case you don't have multiple subscribers on your SNS topic you may also directly notify SQS from S3 and skip SNS inbetween.
Another option is to replace your EC2 instance with a Lambda function and then either directly notify the Lambda or make the Lambda poll the SQS queue (see dev.to/frosnerd/understanding-the-...).
Without knowing more of your context it is hard to recommend something concrete but if possible I'd go for the following setup if I were you: S3 -> SQS -> Lambda. That's easy to manage and gives a reasonable amount of resilience. Another, simpler option would be S3 -> Lambda but then you might lose events in case your Lambda function is broken as S3 will only retry a couple of times as it invokes the Lambda synchronously.
Sure!
I assume you already setup a bucket notification that publishes messages to an SNS topic. If you want to use SNS to receive the notifications (because you want to fan-out) then you can either subscribe via HTTP or SQS.
In case you are using HTTP, the EC2 instance needs to expose an HTTP endpoint. If you want a more resilient and secure setup then you can subscribe with SQS and make the EC2 instance poll from SQS.
In case you don't have multiple subscribers on your SNS topic you may also directly notify SQS from S3 and skip SNS inbetween.
Another option is to replace your EC2 instance with a Lambda function and then either directly notify the Lambda or make the Lambda poll the SQS queue (see dev.to/frosnerd/understanding-the-...).
Without knowing more of your context it is hard to recommend something concrete but if possible I'd go for the following setup if I were you:
S3 -> SQS -> Lambda. That's easy to manage and gives a reasonable amount of resilience. Another, simpler option would beS3 -> Lambdabut then you might lose events in case your Lambda function is broken as S3 will only retry a couple of times as it invokes the Lambda synchronously.I hope that helps!
References:
Thanks a lot for the details Frank. to give you more context, below digram might help with the requirement: