There has been much talk about the ideal aggregate store in domain driven design (DDD) over the years. One particular post that drew my attention back then was this one from Vaughn Vernon https://kalele.io/the-ideal-domain-driven-design-aggregate-store/ that reached the consensus that some form of serialized json in a document store was optimal. In this approach fields themselves can still remain queryable.

This approach had the added benefit of limiting the complicated mapping that inevitably ensued from object-relational-mapping (ORM) tools. For anyone who has tried DDD without event sourcing and hit some of the encapsulation issues, you will know what I mean.

Change data capture (CDC) (transactional outbox)

One additional highlight of that article is the reference to domain events and persisting these within an ACID transaction with the aggregate.

A pattern that has certainly grown a lot more popularity recently is the CDC outbox pattern which prescribes an outbox table where data can be streamed out to analytics and big data products such as Apache Pinot.

CDC Outbox pattern with MySQL, Debezium & Apache Pinot

For further information on this pattern there are detailed explanations on the concept in the following posts:

• https://microservices.io/patterns/data/transactional-outbox.html
• https://debezium.io/blog/2019/02/19/reliable-microservices-data-exchange-with-the-outbox-pattern/

Event Sourcing

A number of previous organisations I have been part of have been led down the Event Sourcing (ES) path due to the promises of intrinsic audit and the ability to recreate views on records to previous states. The reality of these approaches can often be fraught with danger without significant experience. In my experience, it has often resulted in applications that were difficult to design, adapt and manage.

This is where a database technology such as Quantum Ledger Database from Amazon Web Services (AWS) can really help simplify the engineering process and help deliver similar benefits to the event sourcing approach but without the added complexity that it can bring.

Quantum Ledger Database (QLDB)

QLDB is a fully managed ledger database that provides a transparent, immutable, and cryptographically verifiable transaction log owned by a central trusted authority. Amazon QLDB tracks each and every application data change and maintains a complete and verifiable history of changes over time. QLDB Guide

Matthew Lewis has written a very useful introduction around the key concepts behind this new style of database and for that reason I will not repeat that detail here. Intro to QLDB

For the purposes of the reference to the perfect aggregate store, QLDB supports the following key attributes:

• create cryptographically verifiable versions of your document
• documents can be stored with an intention on querying the materialized view of the events within journal
• while it doesn't allow you to stream from an outbox table, you can stream from the ledger and filter/reconstruct an event message from something like AWS Lambda

QLDB and the CDC Outbox pattern

QLDB provides QLDB Streams to support attaching a kinesis data stream that allows you to stream data from the journal into kinesis from any point in time. This then enables functionality such as additional views in search databases such as Elastic Search or analytics tooling such as Apache Pinot. QLDB streams takes care of all the configuration around partition keys for kinesis.

The figure above illustrates the mechanism by which data can be transmitted to Online Analytical Processing (OLAP) tooling. I will be producing some demos to back up this post showing CDC Outbox in to Apache Pinot using the new Kinesis connector which was unveiled in a sneak preview by Neha Pawar in Jan 21.

For those of you that want to take a look at the CDC Outbox style pattern with QLDB and Kinesis, Matthew Lewis has created a demo available on github which highlights this process. It is also available as a running reference example that you can play around with at demo.qldbguide.com.

So is QLDB the perfect aggregate store?

Possibly, time will tell as the product evolves and it gains more traction. It does however provide some critical features not available in other database technology that provides greater verifiable integrity of your records.

Thanks for taking the time to read!