Combining event sourcing and stateful systems

In this two-part series, my colleague Freek and I will discuss the architecture of a project we're working on. We will share our insights and answers to problems we encountered along the way. This part will be about the design of the system, while Freek's part will look at the concrete implementation.

Let's set the scene.

This project is one of the larger ones we've worked on. In the end it will serve hundreds of thousands of users, it'll handle large amounts of financial transactions, standalone tenant-specific installations need to be created on the fly.

One key requirement is that the product ordering flow — the core of the business — can be easily reported on, as well as tracked throughout history.

Besides this front-facing client process, there's also a complex admin panel to manage products. Within this context, there's little to no need for reporting or tracking history of the admin activities; the main goal here is to have an easy-to-use product management system.

I hope you understand that I deliberately am keeping these terms a little vague because obviously this isn't an open-source project, though I think the concepts of "product management" and "orders" is clear enough for you to understand the design decisions we've made.

Let’s first discuss an approach of how to design this system based on my Laravel beyond CRUD series.

In such a system there would probably be two domain groups: Product and Order, and two applications making use of both these domains: an AdminApplication and a CustomerApplication.

A simplified version would look something like this:

Having used this architecture successfully in previous projects, we could simply rely on it and call it a day. There are a few downsides with it though, specifically for this new project: we have to keep in mind that reporting and historical tracking are key aspects of the ordering process. We want to treat them as such in our code, and not as a mere side effect.

For example: we could use our activity log package to keep track of "history messages" about what happened with an order. We could also start writing custom queries on the order and history tables to generate reports.

However, these solutions only work properly when they are minor side effects of the core business. In this case, they are not. So Freek and I were tasked with figuring out a design for this project that made reporting and historical tracking an easy-to-maintain and easy-to-use, core part of the application.

Naturally we looked at event sourcing, a wonderful and flexible solution that fulfills the above requirements. Nothing comes for free though: event sourcing requires quite a lot of extra code to be written in order to do otherwise simple things. Where you'd normally have simple CRUD actions manipulating data in the database, you now have to worry about dispatching events, handling them with projectors and reactors, always keeping versioning in mind.

While it was clear that an event sourced system would solve many of the problems, it would also introduce lots of overhead, even in places where it wouldn't add any value.

Here's what I mean with that: if we decide to event source the Orders module, which relies on data from the Products module, we also need to event source that one, because otherwise we could end up with an invalid state. If Products weren't event sourced, and one was deleted, we'd couldn't rebuild the Orders state anymore, since it's missing information.

So either we event source everything, or find a solution for this problem. You can read about it here.