The most common question I encounter when training or consulting with developers, Engineers and Software development laboratories about a new Event Sourcing project is how and where do we start. This question makes so much sense. I remember trying to get my head around Object Oriented Programming in practice (not the crap I learnt at school), let alone understanding Domain-Driven Design!
Almost a decade ago, I was building a relatively complex online and offline sales and logistics system for what was, back in the day one of my largest clients: a manufacturer, wholesaler and distributor of industrial Refrigeration and Air-conditioning systems. Three months in the project, I had my team set up with three Software Engineers, two backend web developers and one frontend developer. We were making excellent progress. We were building this system using a DDD model and Laravel as the application framework. It was good, and we were productive. Then, the client hired an internal CTO. He asked to meet me, and I obliged. He asked me how it was working in Event Sourcing with PHP. My answer was more of a series of questions 😳.
I already knew about Event Sourcing, I read the books but I never, ever needed to implement such a thing. I did not know where to start. I mean, I had no idea. I knew the concept and the theory, but that was it.
In my previous article, I explained how I get started with a new Event Sourcing Project. With some theory and Event Storming covered, it is time to start thinking about other equally important factors, the Technology Stack for an Event Sourcing project.
I chose PHP as the primary backend programming language. However, I am confident that the planning factors apply to any programming language you'd like to use for your Event Sourcing project. In this article, I will outline what the considerations for planning the Infrastructure and the Application glue in between are. I will also describe and provide a working example of my local environment deployment, code structure, and what I deem as the starting point for such a system, the testing framework.
How to plan the infrastructure services for an event sourcing project
Arguably, I believe that in the majority of cases, when working with a team, preparing for these requirements will depend on several factors and other project stakeholder's needs. Although I am making these decisions by myself for this project, the process is similar for teams and projects commissioned by your clients. The main difference is that I can choose which backend services I am most comfortable using and that I don't have external dependencies beyond my control. Such as migrating or integrating data from an MSSQL database into a MySQL Database 😡.
During my career, I had an equal number of opportunities for developing new systems and migrating legacy systems into an event sourcing solution. In my opinion, the latter is a lot more complicated. You can get in touch with me if you need help with refactoring a legacy system to use an event sourcing pattern and also to migrate your existing data.
I intentionally kept this example Domain simple so to not distract you from the primary topic, which is to build an Event Sourcing project with practical examples. The example code in this project is a sample from a commercial project I manage. Please feel free to ask me questions in the comments section further below about this or other Business Domains. With this out of the way, lets us start planning the Technology Stack for our example project because we have a lot to cover.
Event store database
The most critical persistent storage for Event Sourcing is the Event Store Database. As I'm sure you are aware, there are several different database engines we can use for running our event store. I will not be comparing any of these engines in this post because all we need to know is that we need an append optimised database.
Given the immutable nature of Event Sourcing, we certainly don't need to update any of our persisted events. We also don't need to run any complex read queries. As I will mention in a future article, the Event Store implementation will query the persisted events as identified by the Aggregate's UUID (Universally Unique Identifier). That is quite a simple query.
Top comments (4)
Thanks for your articles, Keith!
I'm implementing my experimental event-sourcing project currently. It is in a different language, but I use PostgreSQL to build Event Store.
I feel the need to lock event stream for an aggregate that's being updated. Otherwise, it is hard to guarantee a correct event order. Based on your experience, what is the way to do it?
I know, some people suggest that you just write all events in a single thread. But, put scaling aside, it doesn't feel reliable to tell that this instance is the only one writing to the event store. Suppose, Kubernetes is said to maintain one instance of a service. But an instance is dead if it stops responding to liveness check. What if packets are lost, but the instance is still alive and writing to the event store?
Anyway, I'm looking for your advice about enforcing correct event ordering.
Hello :)
Thank you for your feedback. I really appreciate it :)
I have used PostgreSQL as an event store before without issues. However, I didn't understand why you will need to lock down the events for an entire Aggregate (assuming it is identified by a UUID).
Can you show me why you needed to do this, please? (with an example) :)
Kind regards,
from you dev friend..Keith!
An example domain is push notification settings in a mobile app. Push notifications are delivered by an onmichannel communications provider. A customer may have a number of mobile devices which are bound to his (single) phone number. A device is bound to a phone number through a process called personalization. One of a customer's devices is considered main - push notifications are delivered to this device. A customer may switch main device as he wishes. Next, there are a few kinds of messages, and a customer may choose a delivery channel for each message type.
I've chosen Customer to be an aggregate, so a single event stream stores state for a particular customer.
To start getting notifications, the following events should occur:
Because of the last synchronization step commands are stored and processed asynchronously. An omnichannel provider has API with a limited throughput, and it regularly fails and needs retry. But we don't want a customer to wait.
A command processor needs to know a current state of a Customer to decide, whether to reject or to store an event, and with what event data.
The point is, if I read from a database some state that I rely on, the next moment it may become outdated. And I may get conflicting events for a customer. So, I supposed that I need a way to lock a particular customer so that his event stream would reliably remain in the same state that I rely on to make up the next decision about changing this state.
EDIT: a device also may be depersonalized, which is important for security reasons. So, for example, a device that is not personalized may not be set as main.
Thanks for the long explanation :)
Please don't just assume that my comments will be right or that they fit your domain; I'm merely commenting more as a question than a statement :)
I think that these events are not business domain ones but more an application(s) matter. They seem to describe tech clients (browsers, devices etc..) and not what the Customer can do. For example, user login is not a Customer's aggregate problem but an application access one.
I think, Eventual Consistency is key here and that you should (if possible) try to plan the system with it in mind.
Please let me know your thoughts on this. I enjoy such discussions :)