In the the initial post of the Event-driven microservices with Kafka series (see here), I talked about the advantages of using event-driven communication and Kafka to implement stateful microservices instead of the standard stateless RESTful ones.
I also presented the architecture and the source code of a related proof of concept application.
In this post, I would like to show how to extend the asynchronous event-driven communication all the way from Kafka to the Web frontend passing through the Java backend.
Hence, in the first post of this series, we got rid of HTTP as the communication protocol among microservices in the backend, and now we are also replacing it (with WebSockets) as the communication protocol between the frontend and the backend.
Ok, but why would you do that?
Because it provides a better experience to the end user!.
Using WebSockets you can build legit real-time user interfaces, the updates are pushed immediately from the server to the client, no need for the user to click any button nor to refresh the browser.
Wait, but we can do the same using good old HTTP, can't we?
Well... HTTP was not designed to provide real-time communication, but since people really wanted it, a number of workaround techniques and technologies have been create on top of it, such as:
- Long polling
- Short polling
- Server-Sent Events
Those are little more than hacks, IMHO.
WebSockets, on the other hand, provide full-duplex bidirectional communication, which means that information can flow from the client to the server and also in the opposite direction simultaneously, it cannot get more real-time than that (that's what I meant when I said 'legit real-time applications' before).
The example
I added a new service to the same PoC application which I used in the previous post and also an Angular web frontend.
Ok, but what does this new service do?
It allows the frontend user to receive information about the current balance of any account of her/his choice as well as live updates when the balance changes.
Also, information about any transactions related to that currently selected account (i.e., any amount of money credited or debited) past or live, is displayed to the users.
Architecture
The new Transfers WebSockets service main components are:
- Two Kafka consumers (one for each topic) to retrieve messages from the Kafka cluster
- Two Kafka Streams local stores to retrieve the latest data associated with a given key (id)
- A custom local store implemented using a simple Map to store the list of transactions for a given account.
- Two dispatchers which keep track of what WebSockets sessions (i.e., frontend clients) are registered to receive live balance updates or live new transfers data related to which account, and then dispatch the updates accordingly.
And these are the main components of the Angular frontend:
- A WebSockets service which handles the connection lifecycle and dispatches the received messages to the corresponding
RxJS Subject. Please note that different types of JSON messages (requests, responses and updates) flow through the only WebSockets connection. - A form which passes the user input (i.e., the id of the chosen account) to the WebSockets service so it can be sent to the remote server.
- An Angular component to display the account balance which gets asynchronously notified of balance changes (updates) through an
RxJS Observerwhich is subscribed to the correspondingRxJS Subjectin the previously mentioned WebSockets service, and then re-renders the value accordingly. - Another Angular component which similarly as the previous one, gets asynchronously notified of any new transaction related to the selected account and adds it to the list of displayed transaction data.
See it in action
Check out this Youtube video for a demo, where a test process which continuously generates Kafka messages representing random money transfers is used to feed the application.
And on the Web UI the user can see how the transfers pop up as they happen and the balance of the related account changes accordingly, all in real time.
Final Thoughts
HTTP protocol has had a huge success, it made possible the WWW as we know it today.
It has been so successful that it has spread to the backend (RESTful services and open APIs) where it has been the king for more than a decade already.
However, technology evolves, paradigms come and go and for some use cases other tools and designs may be a better choice.
I do not know how software applications will look like in the future, all I can do is guess.
Event-driven microservices look very powerful and implementing them with Kafka and adding WebSockets to the mix seems promising.
In fact, some companies are able to see beyond REST APIs and are starting to offer WebSockets APIs to their clients for faster streaming access to data (example).
And there are other alternative ways to implement event-driven microservices, for example, recently I learnt about Uber Cadence but that's for a future post ☺️
Top comments (7)
Is there any way to send messages directly from KTable to websocket?
I have data that are constantly updating. I keep them in Kafka compacted topic. When new websocket customer arrive I would like to send him a current state of my data, and then I want to send him automatically every change of the data. I believe that the best way it would be to somehow bind KTable (or GlobalKTable, or some state store maybe) with websocket. Is something like that possible?
Hi, sorry for the late response.
Well, since a KTable is basically (please allow me the rough simplification) a view based on a Kafka topic, I guess what you are really asking is whether out-of-the-box integration with Kafka using Websockets exists.
And unfortunately the answer is no, it does not exist.
As per this post, it looks like they experimented with the idea at Confluent, but I could not find anything more about websockets in their blog nor documentation.
I also found this (random) repository on Github but it is 5 years old and looks abandoned.
I even considered to try building a Kafka connector for Websockets myself but I never started it.
Kafka's API provides poll(timer) to read messages from queue; let's call that a pull. Do websockets rely on this method (under the covers)? If it's truly event driven then don't we need push semantics exposed within Kafka's APIs to get the 'real-time' eventing that's being shown here? Make sense?
Thanks a bunch!
Hi! Sorry for such a late response.
Your comment raises a good point, it is true that Kafka does not provide something such as JMS' MessageListener or WebSockets'
MessageHandler.Whole and hence, one needs to explicitly perform the periodic pulling in order to retrieve messages from Kafka.
But this is implemented this way because of performance reasons (so that the Kafka broker does not need to keep track of what messages have been received by which client) and one important detail to note is that a single pull/poll operation may grab more than one message from more than one Kafka topic in one go.
Also, the polling interval is supposed to be very short when querying Kafka, much shorter than the traditional polling we see being used in other systems whose original use case was synchronous communication (e.g. HTTP request/response).
Hence, because of the two points above (the ability to receive messages in bulk and the short polling interval), Kafka is considered a real-time system, despite not providing explicit push (asynchronous receiving) semantics.
I hope this makes sense!
What is not clear here, is who picks the materialized records and publish it back to kafka topic? Is there a backend service? Does it run on a clock (polling) ? Because database itself is not an actor that will send newly added records to kafka by itself!
Is the github link valid? I am not able to check the source code?
Hi, sorry for the late response.
Yes, the Github link works --> github.com/VictorGil/transfers_web...
It is a publicly accessible repository and in the
README.mdfile there are links to other related repos.