DEV Community

Maxime Guilbert
Maxime Guilbert

Posted on • Updated on

Kafka Architecture : Synchronous to Asynchronous [1]

Kafka is a powerful stream processing tool, but it's an asynchronous tool. So we know when we send the request but we don't know when the answer will come. And in some cases, there are some synchronous applications which fronts Kafka.

So today we will see the first of 3 cases to make this communication between the synchronous application more efficient depending your case.


1st solution - Kafka Based Solution

This solution is pretty simple. When you are send a message to kafka, you can add 2 headers to inform where the answer should be sent.

  • REPLY_TOPIC : Name of the topic where the answer must be sent
  • REPLY_PARTITION : Index of the partition in the REPLY_TOPIC where the answer must be sent

Example

Kafka Based solution schema

In this example we have 2 topics :

  • TOPIC REQUEST
  • TOPIC RESPONSE

The 2 instances of the API X write in the TOPIC REQUEST and are waiting an answer in the TOPIC RESPONSE. This topic has 4 partitions which are divided equally between the two instances.

Behind these topics, a MicroService Y is reading request from TOPIC REQUEST and is answering on the TOPIC RESPONSE (defined in the REPLY_TOPIC header of the request) on the partition asked by the header REPLY_PARTITION.

Like this when the first instance send a request, it will receive its answer.

Pros & Cons

Pros Cons
Messages are consumed only once Messages can be lost on a rebalancing
The MicroService Y must be setup to read the headers REPLY_TOPIC and REPLY_PARTITION and use this values to answer
The service account of the microservice Y must be able to write on all the topic it can received on REPLY_TOPIC

The principal point here is the rebalancing. When you scale up/down or restart a service, the partitions will be rebalanced between all the instances of the service.

If the first instance of the API was listening on the partition 1 and 3 before a restart, after a rebalacing it can listen on 2 other partitions. So, if a request was sent from the first instance of the API with REPLY_PARTITION = 1 and the rebalancing appends, then the message can be lost (if the partition 1 is given to another instance than the first one).

/!\ This solution is working only if you can lose messages.


In the following part, we will see a more data-secure solution!

I hope it will help you! 🍺


You want to support me?

Buy Me A Coffee

Oldest comments (0)