I recenlty spoke at the Google DevFestVeneto 2019, in Venice last saturday.
Here are my slides: I'll follow with a more detailed writeup of my experiences. Enjoy!
For further actions, you may consider blocking this person and/or reporting abuse
Read next
How I Built a Cardiovascular Disease Detector using Machine Learning and FastAPI
Daniel Okello -
Migrating from nvm to mise
hverlin -
Test Cases: A Comprehensive Guide
keploy -
Idea: Make Apache Flight (gRPC) work over HTTP/3 (QUIC) and make use of RDMA (fast memory transfer). All made in Rust
Radu Marias -
Top comments (3)
Meta note: it might have been my ad blocker, but none of the SlideShare controls seem to work for me here. Can you also link the slideshare?
Thank you for posting this!
I am curious to know how many requests/s and events/s the system was handling. The techs chosen seem to be geared for high throughput.
The presentation has pretty valuable information for others looking to use Kafka as an event store. Other event storage strategies will face slightly different challenges.
How did you handle snapshot structural changes? For example, changes to business logic which require tracking different data in the write-side snapshot. My best guesses are: doing a long replay onto new read-side and cut-over; or writing code to migrate snapshots to new structure.
Here is the complete link:
slideshare.net/michele.mauro/5-thi...
For the time I was operating it, a few hundreds/minute at peak. The main focus was multitenancy, and there was a queue of clients to be migrated on the new system.
Kafka is one of the few "distributed logs" on the market. The critical feature is the durability of the log, since it may happen that it needs to be replayed. You can't do it with a pure messaging platform because they usually are not optimized for long-term durability. A database (relational or not) must have specific features to manage append-style tables quickly to be suitable for this load. Cassandra is a popular choice (and one of the default production event store reccommended for Akka Persistent Actors).
With care.
Your changes must be as back-compatible as they can, and the write side must be ready to be very liberal in what it reads. If your domain changes wildly, you have to manage this change and assign it to non-structural fields; that is, prepare to have lots of
Map<String, String>and such. If you use them with the appropriate care, you can go a long way without needing to change the event or snapshot schema. A bit of analysis up-front can save you lots of problems in the operation of the system.I think the only way to work this is to re-read history. Or call it a different system and restart from 0. If the new data needs the old events, you have little choice.
Snapshots are so short-lived that you usually consider only one, the last one. So, your code (or, to be more precise, your unmarshalling layer) must be able to read just the last one. Each change you make, you can remove the code to read version n-2, because there will not be any snapshots around in that format.
Thanks for sharing! This is a lot of valuable information.
We have an event sourced system in production and are developing another one. We use different strategies and techs, so our trade-offs are a bit different. It was very interesting to see how your choices played out in trade-offs. I very much appreciate you sharing.