DEV Community

Cover image for Django + memcached + namespace
Peter van der Does
Peter van der Does

Posted on

Django + memcached + namespace

Introduction

One of the drawbacks of using memcached is that you can not delete keys using a wildcard. With Django, you can delete many keys but you have to give it a list of keys to delete.

A way around this is by using namespaces. You will organize keys by namespace. For example, every key related to the products you sell falls in the namespace product. And your clients are in a namespace client. If you delete the namespace all keys within the namespace will be deleted as well. Unfortunately, we run into another snag, namespaces are not supported by memcached.

With the code presented in this article we fake namespaces and have a way to "delete" all keys in a namespace

TL;DR Check out the code.

How do we do this?

When we store data in memcached you need a key. What we will do is prepend this key with the namespace we want to use. For the key guitar, we will prepend the key like this product:guitar.

Wait! How will we invalidate all keys in the namespace product? Instead of using product as the prefix, we will store the namespace in memcached with a value. We use that value as the actual namespace.

The namespace key for memcached will be namespace:product and we store the value 1. I hear you say it: "Ha, but that will cause a conflict with other namespaces". You are right so let us prepend this value with the namespace itself like this product1. Now the guitar key will look like this product1:guitar.
When we change the value of the product value to product2, the guitar key will be product2:guitar. Now all keys in the namespace product1 will no longer be reachable and memcached will take care of removing them.

Code

Wrapping this all up in a class to be used within Django

# code/cache.py

import time

import xxhash
from django.core.cache import cache as django_cache


IS_DEVELOPMENT = False
CACHE_PREFIX = "MyApp"
HOUR = 3600
DAY = HOUR * 24


class MyCache:
    """
    This class is used to create a cache for MyApp.
    """

    def __init__(self, timeout=DAY):
        """
        Initialize the cache.
        """
        self.timeout = timeout if not IS_DEVELOPMENT else 20

    def __str__(self):
        """
        Return a string representation of the cache.
        """
        return "MyCache"

    def get(self, namespace, key):
        """
        Get a value from the cache.

        Parameters
        ----------
        namespace: str
        key: str
        """
        try:
            cache_key = self.safe_cache_key(namespace=namespace, key=key)
            return django_cache.get(cache_key)
        except Exception:
            return None

    def set(self, namespace, key, value, timeout=None):
        """
        Set a value in the cache.

        Parameters
        ----------
        namespace: str
        key: str
        value: str|int|dict|list
        timeout: int or None
        """
        try:
            cache_key = self.safe_cache_key(namespace=namespace, key=key)
            timeout = timeout or self.timeout
            django_cache.set(cache_key, value, timeout)
        except Exception:
            pass

    def delete(self, namespace, key):
        """
        Delete a value from the cache.

        Parameters
        ----------
        namespace: str
        key: str
        """
        try:
            cache_key = self.safe_cache_key(namespace=namespace, key=key)
            django_cache.delete(cache_key)
        except Exception:
            pass

    def delete_namespace(self, namespace):
        """
        Delete the namespace

        Parameters
        ----------
        namespace:str
        """
        self.update_cache_namespace(namespace=namespace)

    def safe_cache_key(self, namespace, key):
        """
        Create a key that is safe to use in memcached

        Parameters
        ----------
        namespace: str
        value: str

        Returns
        -------
        str
        """

        namespace = self.get_namespace(namespace=namespace)
        new_key = "{}:{}:{}".format(
            CACHE_PREFIX, namespace, xxhash.xxh3_64_hexdigest(key)
        )
        return new_key

    def get_namespace(self, namespace):
        """
        Get the namespace value for the given namespace

        Parameters
        ----------
        namespace: str

        Returns
        -------
        str
        """
        key = self.get_namespace_key(namespace=namespace)
        rv = django_cache.get(key)
        if rv is None:
            value = self.get_namespace_value(namespace=namespace)
            django_cache.add(key, value, DAY)
            # Fetch the value again to avoid a race condition if another
            # caller added a value between the first get() and the add()
            # above.
            return django_cache.get(key, value)

        return rv

    def update_cache_namespace(self, namespace):
        """
        Update the value for the namespace key
        Parameters
        ----------
        namespace: str
        """
        key = self.get_namespace_key(namespace=namespace)
        value = self.get_namespace_value(namespace=namespace)
        django_cache.set(key, value, DAY)

    def get_namespace_key(self, namespace):
        """

        Parameters
        ----------
        namespace: str

        Returns
        -------
        str
        """
        return xxhash.xxh3_64_hexdigest(f"namespace:{namespace}")

    def get_namespace_value(self, namespace):
        """
        Create value for the namespace value

        The namespace is used to make sure the hashed value is unique

        Parameters
        ----------
        namespace: str

        Returns
        -------
        str
        """
        namespace_value = namespace + str(int(round(time.time() * 1000)))
        return xxhash.xxh3_64_hexdigest(namespace_value)


cache = MyCache()

Enter fullscreen mode Exit fullscreen mode

Usage

Set a value

cache.set("product", "combined_pricing", 5000)
Enter fullscreen mode Exit fullscreen mode

Get a value

pricing = cache.get("product", "combined_pricing")
Enter fullscreen mode Exit fullscreen mode

Delete single key

pricing = cache.delete("product", "combined_pricing")
Enter fullscreen mode Exit fullscreen mode

Delete namespace

cache.delete_namespace("product")
Enter fullscreen mode Exit fullscreen mode

Conclusion

Using this code allows you to group memcached keys and invalidate the keys in that namespace all at once.

Notes

  • I hash all memcached keys as well as the namespace value because it is part of a key. Parts of the key are not predictable and could potentially have characters that are incompatible with a key.
  • Using time for the namespace value eliminates the need for an increase method.
  • I prefer xxhash over any other hasher, it's super fast.

References

Found a typo?

If you've found a typo, a sentence that could be improved or anything else that should be updated on this blog post, you can access it through a git repository and make a pull request. Instead of posting a comment, please go directly to GitHub and open a new pull request with your changes.


Photo by Colin Lloyd - https://unsplash.com/photos/62OEfKjU1Vs

Discussion (0)