As a data engineer or, more specifically, data platform engineer, a service with high dependency may be handed over to you. Upgrading such a service is a terrifying process. Suppose that service is Kafka, and it's the main component of your data stack at the company. However, the solution isn't ignoring the complexity because every bug fix or new feature can save you from downtime and help you increase the performance of the services. So, what is the solution? How can we ensure all services that depend on Kafka work fine after the upgrade? In this post, I will share my experience through this process.
Main concerns
When we talk about services like Kafka, we know many producers and consumers are in between. So, what happens to them after an upgrade? Do they continue to produce/consume? What about the schema registry and other components that depend on Kafka? So, one of the main concerns is the healthiness of the dependent element.
Also, we want to upgrade Kafka for two significant versions; how should we check deprecated configs? Should I read all the changelogs one by one? There is a better approach that minimizes the time spent and the probability of downtime.
Proposed approach
Honestly, every time I think about Docker, I wonder what a beautiful tool this is :D You know? Amazingly, you can independently set up a whole stack in a separate network with tools like docker-compose.
A better approach is to use Docker to simulate production services in a safe environment. We can set up a whole stack with the same configs but fewer resources, simulate upgrades, and check each component's behavior.
Applied approach for Kafka
To simulate the upgrade process for Kafka, I am supposed to create a stack including these components:
- Zookeeper Instances -> Coordinator for Kafka Cluster
- Kafka Instances -> Main component
- Schema Registry -> Persist schema of produced messages
- Kafka UI -> Monitor Kafka cluster and see incoming messages in topics
- Producers -> Python code to produce data into Kafka topic in Avro format.
- Consumer -> Python code to consume data produced by
Producer
. - Clickhouse -> Analytical database to store data coming from Kafka
- Postgres -> OLTP database stores transactional data
- Postgres Producer -> Python code, which Inserts one record every 0.1 seconds into the
Postgres
database - Debezium -> Capture each change in
Postgres
and send it to the corresponding Kafka topic in Avro format.
Now, it's time to prepare the appropriate docker-compose.yaml
Implement detail
-
Zookeeper
- Image: Official Image
- Configs:
- Mount
zoo.cfg
into the container -
myid
and data directory created usingzookeeper_conf_creator.py
- Mount
-
Kafka
- Image: Bitnami Image
- Image customized by
Dockerfile-Kafka
(https://hub.docker.com/r/bitnami/kafka)
- Image customized by
- Configs:
- Set as an environment variable
-
server.properties
converted toserver.env
usingkafka_env_creator.py
- This image didn't support
SCRAM-SHA
for authentication. Solibkafka.sh
(which is bitnami's Kafka library), rewritten.
- Image: Bitnami Image
-
Schema Registry:
- Image: Official Image
- Configs:
- Set as an environment variable
-
schema-registry.properties
converted toschema-registry.env
usingschema_registry_config_creator.py
-
Kafka UI
- Image: Official Image
- Configs:
- Set as an environment variable
- Directly in
docker-compose.yaml
- *Producer and Consumer *
- Image: Official Python Image
- Image customized by
Dockerfile-Python
- Image customized by
- Code:
producer.py
andconsumer.py
- Configs: Set as environment variables directly in
docker-compose.yaml
- Image: Official Python Image
-
Clickhouse
- Image: Official Image
- Tables: Tables DDL defined here and then mounted into
/docker-entrypoint-initdb.d
- For each table in
Postgres
three tables are defined here:- Base table -> data persist here
- Kafka table -> read data from kafka
- Materialize view -> ship data from the Kafka table into the base table.
- For each table in
- Configs: Default configs used only
kafka.xml
mounted into/etc/clickhouse-server/config.d/kafka.xml
- Logs: For debug purposes, logs mounted into local directory
-
Postgres
- Image: Debezium Example Image
- This Postgres contains sample sale data.
- Image: Debezium Example Image
-
Postgres Producer
- Same as
Producer
andConsumer
but this code used
- Same as
-
Debezium
- Image: Official Image
- Configs:
- Set as an environment variable
-
kafka-connect.properties
converted tokafka-connect.env
usingkafka_connect_config_generator.py
Some Extra Containers
-
kafka-setup-user
- It uses the same image as
Kafka
; it runs afterkafka1
becomes healthy. Some users are created after this container runs (exit with status 0). See them here - It needs one Kafka broker and also a Zookeeper cluster because
SCRAM-SHA
needs to persist on Zookeeper.
- It uses the same image as
-
kafka-setup-topic
- It uses the same image as
Kafka
and creates some topics. See the list here
- It uses the same image as
-
submit-connector
- It use curl image to submit this connector into
Debezium
. The connector captures the changes inPostgres
, sends events toKafka
, and thenClickhouse
consumes the data into appropriate tables.
- It use curl image to submit this connector into
Some Extra Notes:
The version of all containers defined in the
.env
file. You can change them from this file.Container dependencies are defined accurately. So, if one container depends on another to come up, appropriate
healthcheck
anddepends_on
conditions are defined for it.If you take a look at the
healthcheck
of containers, for example, kafka, you see this command:
(echo > /dev/tcp/kafka1/9092) &>/dev/null && exit 0 || exit 1
This shell script helps check the TCP port in a container without telnet
.
Simulation Process
To run the simulation, you can follow these steps.
Result
All tests were successful. By successful, I mean the producer can still produce messages without errors, and consumers can consume messages without errors. No other criteria were investigated; you can define your metrics for this simulation. Only one problem was seen in this process.
Problems:
- In
Setup Kafka User
:java.lang.ClassNotFoundException: kafka.security.auth.SimpleAclAuthorizer
occurred- It deprecated after 2.4.0. See here
- Doc recommends to use
kafka.security.authorizer.AclAuthorizer
instead. It's fully compatible with deprecated class, so it was replaced in docker-compose and it worked
Conclusion:
- As there is the official document for upgrading from any version to 3.6.1 (and another previous version), there is no obstacle in this process. Also, our test shows this process works, and we can upgrade our Kafka to whatever version we want.
Conclusion
This article is a suggestion for the best approach for upgrading highly dependent services. We talked about the details of implementing this process, and then, as we saw in the Result section, one problem was found before upgrading so we can upgrade our Kafka cluster seamlessly, with zero-downtime :)
Top comments (0)