Cache Master data in Redis using Python

Introduction

According to its GitHub repository, Redis (stands for REmote DIctionary Server) is an in-memory data structure store. It is a disk-persistent key-value database with support for multiple data structures such as strings, hashes, lists, sets, bitmaps, etc.

Data structures implemented into Redis have a few special properties:

• Redis cares to store them on disk, even if they are always served and modified into the server memory. This means that Redis is fast, but that it is also non-volatile.
• The implementation of data structures emphasizes memory efficiency, so data structures inside Redis will likely use less memory compared to the same data structure modelled using a high-level programming language.
• Redis offers a number of features that are natural to find in a database, like replication, durability, clustering, and high availability.

Use Cases

When speaking of use cases of using Redis, the most generic use case that comes to many is for caching data and for session store (i.e. web session). Redis has a lot of use cases that can be applied and useful for any situations especially that involves speed and scalability wise, it's easy to manage either scaling up or scaling down.

Few use cases are:

1. Caching
2. Session store
3. Chat and messaging application
4. Gaming leader board applications

Caching Data in Redis

The cache is temporary storage where data is stored so that in the future data can be accessed faster. So, caching is the process of storing data in Cache. One of the caching use cases is caching the master data that can be database query results, pandas data frame, etc.

Prerequisite
Install python and Redis. Following versions are used in this blog:

python==3.9.0

redis==4.1.0

redis-py requires a running Redis server, and Python 3.6+. See the Redis quickstart for Redis installation instructions.

redis-py can be installed using pip via “pip install redis”.

To test that the Redis connection to the server is successful, we can open a new terminal window and run:

% redis-cli ping

The output should be

PONG

Create instance of Redis Class

We need to create instance of Redis class that expects mainly Redis broker URL, port, and database, and it can be further used to do all CRUD operations on Redis.

Key-Value storage
Cache is maintained in key-value pair, which means we can store any data with respect to a unique key like dictionary. Here key can be any string with max size of 512 MB. Values, on the other hand, can vary in size depending on their type. For aggregate data types (i.e. hash, list, set, and sorted set), the maximum value size is 512 MB for each element, although the data structure itself can have up to 2^32 - 1 elements.

Data structures supported in values by Redis:

• Binary-safe strings
• Lists
• Sets
• Sorted sets
• Hashes
• Bit arrays (or simply bitmaps)

Note: Redis can handle up to 232 keys and was tested in practice to handle at least 250 million keys per instance.

Every hash, list, set, and sorted set, can hold 232 elements.

In other words, your limit is likely the available memory in your system.

Check if key exist in Redis
Since data is saved in Redis as key-value pair, we can check if key is already there in cache:

app_redis_handler.exists(key)

For example:

if not app_redis_handler.exists("balance"):
    return None

Create cache using key
We can create a cache based on provided key and value:

app_redis_handler.set(key, data)

For example:

app_redis_handler.set("balance",1777)

Save json dictionary as value
We can also save json as data in Redis cache:

app_redis_handler.set(key, json.dumps(data))

For example:

app_redis_handler.set("balance", json.dumps({"due":577,\
 "credit":680}))

Set expiry on Cache key
We can provide expiry of cache key in seconds. Keys expiring information is stored as absolute Unix timestamps (in milliseconds in case of Redis version 2.6 or greater).

app_redis_handler.expire(key, expiry)

For example:

app_redis_handler.expire("balance", 10)

Create cache using key with expiry
We can create a cache based on provided key and value along with expiry:

app_redis_handler.setex(key, expiry, data)

For example:

app_redis_handler.setex("balance",60, 1777)

Get data based on key
Value can be fetched from cache based on key:

data = app_redis_handler.get(key)

For example:

data = app_redis_handler.get("balance")

Delete/invalidate data based on key:
Delete/invalidate cache key once not required:

app_redis_handler.delete(key)

For example:

app_redis_handler.delete("balance")

Scan all keys prefixed with "prefix_text":
Cache key is searchable based on prefix, and it fetches all keys prefixed with the provided text:

redis_handler.scan_iter(f'{prefix_text}*')

For example:

[key for key in app_redis_handler.scan_iter(f'{bal}*')]

Save master data in cache
We can also save the master data that can be database query results, pandas data frame data, etc. in cache:

app_redis_handler.set("employee", [(1, 'John', 'Finance'), \
(2, 'Richard', 'Energy'), (3, 'Carter', 'HR'),……])

Here we can save the master data in cache, and fetch it whenever required and load it in data frame like structure to apply filters like "department=HR". It can be useful if we want to show top employees of all departments from cache, so it is better to save it in one cache and apply filter on cache data to save the extra transactions to database and managing the extra keys in cache.

Summary

Redis offers highly performant and efficient read and write along with few operations mentioned in this article. There are lots more. For more information, you can visit the documentation.

References

https://redis.io

https://github.com/redis/redis

https://redis-py.readthedocs.io/en/stable/

https://www.kite.com/python/docs/redis.StrictRedis

Disclaimer: This is a personal blog/post. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.