DEV Community

Discussion on: Redefining Event Sourcing

 
javier_toledo profile image
Javier Toledo

Yes, it certainly can become complex, but the advantages really pay off in some scenarios.

Talking about a close example to me, in e-commerce applications, the go-to way to scale is caching the catalog, and that helps a lot to keep the site online on sales days, but there are situations when the number of concurrent orders is such that the database becomes a bottleneck. When an order is processed a lot of transactional things happen under the hoods, you need to check that you have enough stock (and prevent others to pick it while you're processing the order), that the payment processor confirms the deposit, and trigger the warehouse processes to prepare the order, etc. In most e-commerce implementations all these things happen synchronously while the user waits, and some user actions could be blocked waiting for others to finalize, to the point that users get bored and leave or some calls time out and fail, with the consequent loss of orders (and money). With event-sourcing, you have an append-only database that can ingest way more data, it's easier to shard the writes, and the checks and everything else can be done asynchronously (you can still manage transactions with sagas). This requires a mindset change in the way the whole system is designed, and for sure it's not "easier", but this way no orders are ever dropped.

The main advantage I see over inter-services REST APIs is the inversion of dependence. When you use a REST API, the sender needs to understand the API contract and implement a client to consume it. When that API changes, you need to go back and find all the services that consume that API and change their API client implementation. Otherwise, everything fails. This can become challenging when you have more than a couple of services. In event-sourcing, as I mentioned in my previous message, one service that changes the contract could live with other services that are using a previous version of the message, reducing this risk. Dependencies happen at the data level, not at the code level, and that's generally easier to deal with.

With RabbitMQ you could deal with this dependency in a similar way, but the disadvantage I see is that RabbitMQ works with a real-time push mechanism that can easily overwhelm the consumers in some situations. I've seen a couple of times that because of errors or sudden user peaks, the consumers can't consume the messages fast enough until the hard drive is full and stops accepting messages. Event-sourced databases tend to be easier to scale and the messages are ingested and persisted until you can process them, so you don't need it to happen in real-time. The consumers can go at their own pace, they don't even need to be online when events are stored, and it's easier to recover from failures because no events are dropped.

Don't get me wrong, RabbitMQ is a fantastic piece of technology, and when the system is properly designed taking into account these edge cases, it can definitely work. Indeed, there are some implementations of Event Sourcing that use RabbitMQ under the hoods for notification events, so they are not really "competing" technologies.

Event-sourcing is not a silver bullet, but a way to store data that has some benefits in some situations like the one I described, but as with every other technique, I think it shines especially when you don't overuse it and combine it with other tools and techniques in a balanced way.