First time to read about Kafka

If you don’t know anything about Kafka this article for you , we are going to start a short journey to get to know Kafka. Now let’s go.

Suppose we have two parties on applications (A and B) and we need to exchange a message which may be is (Text, JSON), What should we do?

First thing to think we will build an APIs in both, It’s good if you have a short message and will be between two apps only.

Now let’s imagine an application A is an online customer service that receive requests from the customer.
And an application B is for employees to display the customer requests and working on them.
And using an APIs application A will receive the requests from all users and send them to application B which is store them into data storage then employees can display the request and working on them.

Maybe you think, we can integrate both with one data storage application A receive requests and store them directly then application B will read them directly it will be good if application A receive small number of requests.
So what if application A receive 100 request per minute ?
, what if application A receive 1000 request per minute ?
and what if application A receive 1000 request per second ?
and so on …etc.

Now what about the performance ? it will be no good.

So integrate both applications using one database is not better solution :( .
What if we have a solution to make application A just receive any number of messages per second then send them to another component made to transfer a huge amount of data quickly 😊 it will be good, right ?

Yes, you right This is Kafka.

Simply we can say Kafka is a platform which is used to handle real-time data storage and exchange message (data) between two parties.
When you work with Kafka will listen some of words like :
Message, Topic, Partition, Producer, Consumer…etc.

Anything will be exchange between two parties called Message.
Now we will transfer data from application A to application B but using Kafka.
Application A will send a message This is known as a Producer.
Application B will receive a message This is known as a Consumer.

Let’s image a scenario, application A (Producer) will receive a huge number of requests at the same time, it will to send them to Kafka directly. GOOD !.

Kafka will receive them and need to store them in place till application B (Consumer) receive them, This place known as Topic.
Topic is a place which use to store a messages, it identified by name each topic has unique name because this name used by producer to send a message to it.

Let’s say we have a topic called *company_customer_services_topic * on Kafka, when you implement the producer you will add the topic name in configuration setup to know the producer, where it can send the messages?

Now our messages store on the topic on Kafka, and application B need those messages (Also when you implement application B as Consumer will add topic name in configuration setup to know where it can get the message? )
Actually, the consumer app always talking with Kafka to tell it, I’m here if and message stored in topic company_customer_services_topic I need to get it.
So now the consumer will pull the messages from the topic and store it in the database, and can work anything with any request.

Summary
Now both applications talk together via Kafka to improve the performance of exchange data, application A(Producer) just receive a requests from customers and produce them to Kafka in specific Topic in the same time application B talk Kafka to consume/subscribe the same topic, once the message arrives and stores in the topic application B(Consumer) will pull it .