Act on storage events

I'm a big fan of writing event driven applications both in intra app communication as well as cross app communication, like message queues, job queues or sockets.

One area which, when you look at the examples out there, seems to have been fallen behind when it comes to this is to act on data storage and the events there might be from the database engine.

The classic case in the greater database engines out there, like MongoDB, is to listen for updates on a specific table or document storage. But I want a more dynamic solution to solve more use cases.

Document Database Scenario

Quite a few years ago now, there came a new star in this market when it comes to storage for realtime applications, RethinkDB. It's a document database, or rather "json database" as they call it, which is built around the concept of listening to queries and get updates pushed to you rather then polling for new data.

Yes, that's correct - you get new data pushed not only on table level, but on each and every query, no matter how detailed it may be.

You can start by using it as an ordinary database, doing a query by their simple query language - ReQL. When you've got your query right, you'll just add .changes() to start listening for changes on that specific query.

For example if you think of a chat application where messages as posted into a database table, separated by their room id, room_id.

const ROOM_ID = "#myChatRoom";

r.table("messages").filter(
  r.row("room_id").eq(ROOM_ID)
).changes().run(conn, callback);

This would give us a callback, calling the defined callback method, each time a new message appears in the message table with our room id defined.

Going one step further - this do not only apply to new data being added to the query of your choice, but also a lot of other events, i.e. you can build whatever you want on whatever event that you want. What a world of possibilities!

Key value Database Scenario, the simpler one

In a lot of scenarios you might not need a document database, but you still want to act on data changes. It might be when things are set or get, but maybe even more likely on expiration an deletions.

Redis is one of the most well known key value databases out there. It is also quite well known that they have a PubSub solution, i.e. you can use Redis as an message queue. Though what is less known is that they do have an event subscriber which isn't related, that is used to listen for changes on your ordinary key storage. They call this Keyspace Notifications. It is quite a threshold to understand and get started with, so I will walk you through it.

Configuration

Step one in to configure your Redis instance to send these push events, as this is off by default due to performance optimization.

There are quite a few keys that can be set but in this example we will set KEA, i.e. all keyspace and keyevent events.

This can be done either through the cli $ redis-cli config set notify-keyspace-events KEA or by your application client that integrates with redis. Here is an example in JavaScript.

const redis = require("redis");
const redisSubscriber = redis.createClient();
redisSubscriber.config("SET", "notify-keyspace-events", "KEA");

Subscribe

When Redis is configured, you have to define the database you want to subscribe (a Redis has 16 databases, where the default one is 0) specific kind of event that you want to key notifications for, or if you want to get it for all of them (then subscribe to *).

const database = 0;
const event = "set";
redisSubscriber.subscribe(`__keyevent@${database}__:${event}`);

Event listener

When those are set up, you're ready to start listen to events.

First you connect to the subscription and then you start listening for messages.

redisSubscriber.on("subscribe", channel => {
  console.log(`subscribed to event of type ${channel}`);
  redisSubscriber.on("message", (channel, key) => {
    console.log(`Event type: ${channel}`);
    console.log(`KEY: ${key}`);

    // Here you can go and fetch data on that key 
    // or take whatever action that you might want to.
  });
});

Use cases

As seen for the document database there are endless possibilities in creating realtime application with these kinds of event - but for the key value storage my main usage of subscriptions has been to work with TTL and expiration of keys, taking actions on expiring cache etc.

The concept though is the same - why should you long poll for data from your database, when you wouldn't want to implement long polling between your clients and your server?

What use cases do you see for these kind of subscriptions?