DEV Community

Cover image for 🎯 Sending flat files to Kafka
Daniel Santos for opt-nc

Posted on

🎯 Sending flat files to Kafka

Hello πŸ–οΈ !

Following the mechanisms described here to send CSV files into Kafka in JSON using kcat through shell pipelines, we packed πŸ“¦ all of that in a new home-made tool named producer-file. Here are the features :

  • πŸ’¨ Sending line by line
  • πŸ’ͺ CSV to JSON conversion
  • πŸ”¦ Support message headers by file and/or by line
  • ⚑️ Delta-changes mode (only send changes occurred after last send)

☝️ More about the shell way :


Using jq + kcat with shell pipelines do the job well, but this doesn't fit the company's requirements today about software packaging and logging capabilities πŸ€·πŸ»β€β™‚οΈ.

πŸ“½οΈ Demo

Short video demonstrating how to use this tool to send a CSV file in JSON into Kafka within a Docker environnement :

πŸ“– Ressources

Top comments (2)

adriens profile image

Here is a related and worhtwhile content :

GitHub logo opt-nc / atelier-spring-kafka

Projet d'exemple d'utilisation de la lib `org.springframework.kafka:spring-kafka`

Project Website Confluent tutorial

❔ A propos

Ce projet explique comment dΓ©velopper un producer/consumer sans la lib interne historique au profit de celle dΓ©veloppΓ©e et maintenue par Spring.

βœ‹Pourquoi Kafka


πŸ“ Description

Projet d'exemple d'utilisation de la lib org.springframework.kafka:spring-k...

βœ… PrΓ©-requis

  • Instance kafka
  • docker & docker-compose (si utilisation de kafka.yml)

πŸ‘Ά Kafka ?! WTF ❔

Avant de passer Γ  la suite, prendre soin de consulter les ressources ci-dessous:

πŸ§‘β€πŸŽ“ Supports custome

πŸ“œ Scripts

1️⃣ Initialiser le topic demo.user

Entrer dans le container kafka :

docker exec -ti kafka bash

CrΓ©er le producer : --broker-list kafka:9092 --topic demo.user --property "parse.key=true" --property "key.separator=|"

Copier les lignes :

112233|{"phoneNumber":"112233", "firstName":"Hubert", "lastName":"Bonisseur de la Bath"}
998877|{"phoneNumber":"998877", "firstName":"Jean", "lastName":"Soudajman"}
446655|{"phoneNumber":"446655", "firstName":"Henri", "lastName":"Tathan"}

2️⃣ Ecriture automatique dans le topic demo.sms


adriens profile image