DEV Community

Cover image for Url shortener using Apache Kafka
Zakaria Maaraki
Zakaria Maaraki

Posted on

Url shortener using Apache Kafka

Url shortener

URL shortening is used to create shorter aliases for long URLs. We call these shortened aliases short links. Users are redirected to the original URL when they hit these short links. Short links save a lot of space when displayed, printed messaged or tweeted. Additionally, users are less likely to misstype shorter URLs.

Source code

Before starting you can find the source code of the project at the following link


Functional requirements

  • Given a URL, the system should generate a shorter and unique alias of it. This is called a short link
  • When users access a short link, the system should redirect them to the original link.
  • The system should be able to keep tracking of the number of visists of each url

Non Functional requirements

  • The system should be highly available. This is because if our system is down, all the URL redirections will start failing.
  • URL redirection should happen in real-time with minimal latency.
  • Shortned links should not be guessable (not predictable)


  • How many times a redirection happened (visited links)
  • The system should be accessible through REST API call by other services.


url-shortener architecture

System APIs

Each service uses Rest APIs to expose the functionalities. Following could be the definition of APIs :

url-shortener apis

CQRS pattern

CQRS stands for Command and Query Responsibility Segregation, a pattern that separates read and update operations for a data store. Implementing CQRS can maximize performance of the application, scalability, and security. The flexibility created by migrating to CQRS allows a system to better evolve over time and prevents update commands from causing merge conflicts at the domain level.

Our datastore in this case is the in memory database RocksDB used by default by Kafka Streams.

Writes are done by Url shortener service through Kafka topic and reads are done by Url redirector using in memory database RocksDB

Interactive queries

Kafka Streams natively provides all of the required functionality for interactively querying the state of the application.
A Kafka Streams application typically runs on multiple instances. The state that is locally available on any given instance is only a subset of the application’s entire state. Querying the local stores on an instance will only return data locally available on that particular instance.


Kafka Streams materializes one state store per stream partition. This means the application will potentially manage many underlying state stores. The API enables to query all of the underlying stores without having to know which partition the data is in.

Encoding actual URL

We can compute a unique hash (MD5 or SHA256, etc.) of the given URL. The hash can then be encoded for displaying. For the implementation i have chosen SHA256.


Check my Github for more details about the implementation :

Discussion (0)