Data Modeling for Apache Kafka – Streams, Topics & More with Dani Traphagen

Helping users be successful when it comes to using Apache Kafka® is a large part of Dani Traphagen’s role as a senior systems engineer at Confluent. Whether she’s advising companies on implementing parts of Kafka or rebuilding their systems entirely from the ground up, Dani is passionate about event-driven architecture and the way streaming data provides real-time insights on business activity. 

She explains the concept of a stream, topic, key, and stream-table duality, and how each of these pieces relate to one another. When it comes to data modeling, Dani covers importance business requirements, including the need for a domain model, practicing domain-driven design principles, and bounded context. She also discusses the attributes of data modeling: time, source, key, header, metadata, and payload, in addition to exploring the significance of data governance and lineage and performing joins.

EPISODE LINKS

• Convert from table to stream and stream to table 
• Distributed, Real-Time Joins and Aggregations on User Activity Events Using Kafka Streams
• KSQL in Action: Real-Time Streaming ETL from Oracle Transactional Data
• KSQL in Action: Enriching CSV Events with Data from RDBMS into AWS
• Journey to Event Driven – Part 4: Four Pillars of Event Streaming Microservices
• Join the Confluent Community Slack
• Fully managed Apache Kafka as a service! Try free.

Episode source