Using ksqlDB you can enrich streams of data, and write the resulting Kafka topic to a database. You can also build stateful aggregations and write these to a database.
🎥 https://rmoff.dev/ksqldb-jdbc-sink-video
Check out this tutorial for a walkthrough of how to do this and learn tricks such as:
- Troubleshooting Kafka Connect JDBC sink
- Un-nesting data for loading into the database (solves the error "(STRUCT) type doesn't have a mapping to the SQL database column type")
- Handling timestamp fields
- Idempotent database updates
- Update rows in place based on a declared key
Whilst the tutorial is built around ksqlDB, the tips & tricks for the JDBC sink apply equally to Kafka Connect used on its own.
You can find the Docker Compose and notes on how to try this out for yourself here: https://rmoff.dev/ksqldb-jdbc-sink
Top comments (0)