So glad I dropped by on dev.to today as I'm pretty much set on using a message based architecture in my next project. I've been looking for some good commentary on the merits to such an approach and this article helps greatly.
Do you have any opinions event routers/queues there's so many out there and I'd love to have some feedback on how big the hammer should be to crack the nut. I've used simple mqqt protocols and most recently Apache Kafka but there's so many more out there.
I think the right tool will change depending on the level / volume of messages.
For the benefit of other potential readers, the answer to this question has nothing to do specifically with message-based APIs. It is about how get different services listening to each other's events.
If we're talking about events which might be listened to by different services, but which are part of the same logical system (maintained by the same team or family of teams). Then I don't see anything wrong with the different services reading events directly from a shared event database. Polling or subscribing could be used, depending on load. For example, Event Store has support for subscribing to events built into the database. Postgres has NOTIFY, which is a primitive that enables you to build event notification too.
Where you start running into a problem with this approach is when you have different teams or business units which can't effectively share resources. Maybe they can't agree on how to manage those resources. Or maybe their needs are vastly different. This communication between logically disparate systems is where a more robust messaging solution like Kafka shines. Apps from different business units can all drop their messages into a reliable stream. Other units can subscribe to only what they are interested in, and keep their own model of the outside world that makes sense for their domain. See Conway's Law.
Another reason a distributed message system could make sense without the organizational factors above is if the volume of messages is simply too much for one database to handle.
The reason I specifically mention organizational factors as a reason to use a heavy-weight tool like a distributed message system is because operating it can have a non-trivial cost. You certainly could use it for a single system with low message volume, but the cost-to-benefit ratio suffers.
I can't give you first hand account of operating a distributed message stream, but I'll say that I was looking at giving AWS IoT a try when those needs arise. I've read that it is a bit more responsive than AWS Kinesis. Kafka is pretty awesome by all accounts, but it takes non-trivial effort to setup and maintain. Whereas the AWS stuff is bit more approachable for our needs. I'll probably be looking at Confluent's hosted Kafka more seriously when they get their tooling further along.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
So glad I dropped by on dev.to today as I'm pretty much set on using a message based architecture in my next project. I've been looking for some good commentary on the merits to such an approach and this article helps greatly.
Do you have any opinions event routers/queues there's so many out there and I'd love to have some feedback on how big the hammer should be to crack the nut. I've used simple mqqt protocols and most recently Apache Kafka but there's so many more out there.
I think the right tool will change depending on the level / volume of messages.
For the benefit of other potential readers, the answer to this question has nothing to do specifically with message-based APIs. It is about how get different services listening to each other's events.
If we're talking about events which might be listened to by different services, but which are part of the same logical system (maintained by the same team or family of teams). Then I don't see anything wrong with the different services reading events directly from a shared event database. Polling or subscribing could be used, depending on load. For example, Event Store has support for subscribing to events built into the database. Postgres has NOTIFY, which is a primitive that enables you to build event notification too.
Where you start running into a problem with this approach is when you have different teams or business units which can't effectively share resources. Maybe they can't agree on how to manage those resources. Or maybe their needs are vastly different. This communication between logically disparate systems is where a more robust messaging solution like Kafka shines. Apps from different business units can all drop their messages into a reliable stream. Other units can subscribe to only what they are interested in, and keep their own model of the outside world that makes sense for their domain. See Conway's Law.
Another reason a distributed message system could make sense without the organizational factors above is if the volume of messages is simply too much for one database to handle.
The reason I specifically mention organizational factors as a reason to use a heavy-weight tool like a distributed message system is because operating it can have a non-trivial cost. You certainly could use it for a single system with low message volume, but the cost-to-benefit ratio suffers.
I can't give you first hand account of operating a distributed message stream, but I'll say that I was looking at giving AWS IoT a try when those needs arise. I've read that it is a bit more responsive than AWS Kinesis. Kafka is pretty awesome by all accounts, but it takes non-trivial effort to setup and maintain. Whereas the AWS stuff is bit more approachable for our needs. I'll probably be looking at Confluent's hosted Kafka more seriously when they get their tooling further along.