Recently, I got an opportunity to write a microservice using NodeJS that consume messages from Kafka, transforms it and produce to another topic.
However, I had to go through the interesting phase of convincing fellow developers and other stakeholders why we should be using NodeJS based microservice instead of Spring Boot.
There are a few existing microservices that are written in NodeJS / Python and are integrated with Kafka. These services are written in the span of last 2 to 3 years. Few libraries were tried and apparently the best at that time was chosen (kafka-node). These services are causing some trouble in the production, sometimes dropping the messages.
I have been following KafkaJS npm package and it looks modern and promising so I proposed it.
But I had to work hard and overtime on the proof of concept. My goal was to address all the concerns raised by other developers who had bad experience with NodeJS + Kafka stack.
Primary responsibility of the microservice is
- Consume Json message
- Transform the Json into multiple small Json objects
- Produce it on a few different Kafka topics based of business logic
I compared the microservices based on SpringBoot vs NodeJs.
Following are my observations
Of course, it is well known fact that NodeJs is much better than Java in terms of resource consumption, I had to add these details as well to clear the doubts from few.
The resource requirement for Java application is 6 times+ more than NodeJS application, so is the monthly AWS bill.
I used streaming feature, consuming one message at a time to keep it simple. Batch processing requires extra care and love.
Throughput can be increased by having more partitions.
- KafkaJS may not be reliable in long run
KafkaJS usage is steadily increasing and they have got a very supportive developers and community. It is less likely to go away in near future.
- There are few open issues in the Library
There are open issues in all well established technologies that includes Java and Springboot. This cannot be the ground to reject the proposal. It is proved in POC that the functionality we needed works fine.
- Does KafkaJS support particular version and implementation of Kafka that we are using?
This was proved in POC
- Is Consumer/Producer Rebalancing supported?
When one consumer/producer goes down, another processor should attach itself to the partition and consume/produce the messages. This was proved as part of POC
- Does it recover from broker going down and another instance coming up?
When one of the brokers goes down, consumer application should be able to reestablish the connection with new instance. This was proved in POC.
To prove that KafkaJS library is good enough, I prepared demo consumer/producer microservices and ran those over 3 to 4 days. These services processed thousands of messages in this time without dropping a single message in all failure situations.
Finally, the POC helped to make the way for KafkaJS in our tech stack. Nevertheless, I really appreciate my team and all for raising concerns and completing POC more convincingly.
At the end, I also believe that however much good are the platform and technology, it is up to a developer how he/she writes the code and take care of corner cases. Development cannot be always plug n play :).