I was working on some demos recently on the Apache Kafka source connectors (hi #KafkaSummit!), and trying to display the stream of changes in the resulting Apache Kafka topic.
My standard approach is to vertically divide the terminal and have on the left the database connection where I can issue commands and on the right the output of kcat. This setup works really well for simple messages, but when using it with a Debezium source connector, the amount of fields pushed in a single message makes it hardly parsable by a live audience watching a screen.
In the above image you can notice that the kcat
output is a JSON format without syntax highlight making it almost unreadable.
jq
to the rescue
I knew and was using since long time jq a tool that allows you to beautify and parse JSON outputs. After installing you can just use it as
echo '{"name":"Francesco", "preferences":["pasta","pizza","espresso"]}' | jq '.'
and you get your amazing JSON displayed nicely
{
"name": "Francesco",
"preferences": [
"pasta",
"pizza",
"espresso"
]
}
jq
and kcat
Now, how can I apply jq
to the kcat
consumer output? I tried a direct pipe like:
kcat -F kcat.config -C \
-t my_pgsource.public.players | jq
But unfortunately I only got the output comments without the actual data being displayed
% Reading configuration from file kcat.config
% Reached end of topic my_pgsource.public.players [0] at offset 3
Seen on screen as
I struggled long time, but then reached out to a friend (kudos @rmoff) who pointed me to the right solution: the -u
flag ( Unbuffered output) in kcat
. Once added the -u
flag I'm now able to see my beautified JSON!
Top comments (0)