DEV Community

Cover image for Beautify kcat consumer output by piping to jq
Francesco Tisiot
Francesco Tisiot

Posted on • Originally published at


Beautify kcat consumer output by piping to jq

I was working on some demos recently on the Apache Kafka source connectors (hi #KafkaSummit!), and trying to display the stream of changes in the resulting Apache Kafka topic.

My standard approach is to vertically divide the terminal and have on the left the database connection where I can issue commands and on the right the output of kcat. This setup works really well for simple messages, but when using it with a Debezium source connector, the amount of fields pushed in a single message makes it hardly parsable by a live audience watching a screen.

Unparsable JSON

In the above image you can notice that the kcat output is a JSON format without syntax highlight making it almost unreadable.

jq to the rescue

I knew and was using since long time jq a tool that allows you to beautify and parse JSON outputs. After installing you can just use it as

echo '{"name":"Francesco", "preferences":["pasta","pizza","espresso"]}' | jq '.'
Enter fullscreen mode Exit fullscreen mode

and you get your amazing JSON displayed nicely

  "name": "Francesco",
  "preferences": [
Enter fullscreen mode Exit fullscreen mode

jq and kcat

Now, how can I apply jq to the kcat consumer output? I tried a direct pipe like:

kcat -F kcat.config -C \
    -t my_pgsource.public.players | jq 
Enter fullscreen mode Exit fullscreen mode

But unfortunately I only got the output comments without the actual data being displayed

% Reading configuration from file kcat.config
% Reached end of topic my_pgsource.public.players [0] at offset 3
Enter fullscreen mode Exit fullscreen mode

Seen on screen as

No output of JSON

I struggled long time, but then reached out to a friend (kudos @rmoff) who pointed me to the right solution: the -u flag ( Unbuffered output) in kcat. Once added the -u flag I'm now able to see my beautified JSON!

Beautified JSON

Top comments (0)

50 CLI Tools You Can't Live Without

The top 50 must-have CLI tools, including some scripts to help you automate the installation and updating of these tools on various systems/distros.