Thanks for the article. Where do Amazon SNS and Google Pub/Sub fit in the picture? When would you recommend these services over Apache Kafka?
Those are both great for managed pub/sub messaging. However, they don't guarantee ordering. If order matters to you, such as for time-ordered logs or metrics, then you might want to use Apache Kafka for Heroku or Amazon Kinesis instead. Also consider the delivery guarantees. SNS tries a limited number of times. For more control over expiration, or to reprocess data, then something like Kafka will be better.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.