Spring Cache with Caffeine

Introduction

In the last article it was explained in detailed how Spring Cache works. Our customer app benefited from this by reducing the number of accesses to the service layer methods (and by extension to the repository layer). The application used the default simple implementation which is based on a ConcurrentHashMap. The main disadvantage of this option is that it does not offer a cache eviction policy. Hence, entries must be removed explicitly. That was achieved with a fixed scheduler to clean up the HashMap.

This is a rudimentary solution. That is why in this article we are going to take a look at a cache implementation supported in Spring: The Caffeine library.

Caffeine

Caffeine is a java caching library known for its efficiency. Under the hood, Caffeine employs the Window TinyLfu (build upon Bloom filter theory) policy providing high hit rate (the ratio between the number of cache hits and the total number of data accesses) and low memory footprint. This algorithm has better performance than LRU (Least Recently Used) and it is a good choice for general purpose cache.

Visit the official Caffeine git project and documentation here for more information if you are interested in the subject.

Main Features

Caffeine provides the following optional features:

1. Automatic loading of entries into the cache, optionally asynchronously.

2. Three types of eviction: size-based eviction, time-based eviction, and reference-based eviction.

   Size-based eviction when a maximum is exceeded based on
   frequency and recency.

   Time-based expiration of entries, measured since last access or
   last write.

   Reference-based evicts when neither the key nor value are
   strongly reachable.

3. Asynchronously refresh when the first stale request for an entry occurs.

4. Keys automatically wrapped in weak references.

5. Values automatically wrapped in weak or soft references.

6. Notification of evicted entries. Removal listeners can be setup to perform an operation when an entry is removed.

7. Writes propagated to an external resource.

8. Accumulation of cache access statistics.

Dependencies

Caffeine dependency has to be added to the pom.xml. We will download latest version 3.1.8 at the time of writing this article. This version is compatible with java 11+. If you are still running your apps in java 8 you must pick any 2.x version.

<dependency>
    <groupId>com.github.ben-manes.caffeine</groupId>
    <artifactId>caffeine</artifactId>
    <version>3.1.8</version>
</dependency>

Configuration

There are two ways to configure Caffeine in Spring. The first one is by setting the cache properties in the application configuration file. In our case, it is the application.properties file but it could be the yml too.

The below sample defines the properties for two cache regions, customer and customersSearch. Each region will have an initial capacity of 10 entries, a maximum of 100 entries and entries will be automatically removed from the cache after 1 hour since the last read or write (also called TTL time to live).

spring.cache.cache-names=customers,customersSearch
spring.cache.caffeine.spec=initialCapacity=10,maximumSize=100,expireAfterAccess=1h

The caffeine spec can be parsed from a string. The syntax is a collection of comma separated key-value pairs. Each setting corresponds to a builder method of the Caffeine class. Full list of spec props can be found here.

However, this configuration is global and has its limitations. It applies the same settings to all the caches which may not be suitable for many applications handling multiple cache regions.

Now let's have a look at the second approach. Caffeine can also be configured programatically. We start declaring a Caffeine Bean holding the spec of the Cache as shown below

Caffeine caffeineSpec() {
    return Caffeine.newBuilder()
        .initialCapacity(10)
        .maximumSize(100)
        .expireAfterAccess(1, TimeUnit.HOURS)x;
}

The next step is to create a CacheManager passing the above configuration. We can place this in a new class as demostrated here

@Configuration
@EnableCaching
public class CaffeineCacheConfig {

    @Bean
    public CacheManager cacheManager(Caffeine caffeine) {
        CaffeineCacheManager cacheManager = new 
            CaffeineCacheManager("customers", "customersSearch");
        cacheManager.setCaffeine(caffeine);
        return cacheManager;
    }

    @Bean
    Caffeine caffeineSpec() {
        return Caffeine.newBuilder()
                .initialCapacity(10)
                .maximumSize(100)
                .expireAfterAccess(1, TimeUnit.HOURS);
    }
}

As you can see Spring already comes with a specific cache implementation for Caffeine, so it makes it easier to integrate with Caffeine. The CaffeineCacheManager constructor can accept more than one cache name/region in the overloaded varargs version.

Let's test the find customer endpoint from the CLI app (check out previous articles on Spring Shell series and Spring Rest app).

The first request to the endpoint to get the customer id 2 from our cli client app.

shell(administrator1):>find-customer 2

will trigger a db call in the RestApi app

org.hibernate.SQL                        : select c1_0.id,c1_0.date_of_birth,d1_0.customer_id,d1_0.created_on,d1_0.info,d1_0.vip,c1_0.email,c1_0.name,c1_0.status from customer c1_0 join customer_details d1_0 on c1_0.id=d1_0.customer_id where c1_0.id=?
org.hibernate.orm.jdbc.bind              : binding parameter [1] as [BIGINT] - [2]
d.n.r.aspect.LoggingExecutionTimeAspect  : Method [Optional dev.noelopez.restdemo1.service.CustomerService.findById(Long)] with params [2] executed in 42 ms
d.n.r.aspect.LoggingExecutionTimeAspect  : Method [ResponseEntity dev.noelopez.restdemo1.controller.CustomerController.findCustomer(Long)] with params [2] executed in 60 ms

Then the second and subsequence calls will be loaded from the cache.

d.n.r.aspect.LoggingExecutionTimeAspect  : Method [ResponseEntity dev.noelopez.restdemo1.controller.CustomerController.findCustomer(Long)] with params [2] executed in 0 ms

Configuring Multiple Cache Regions

In the previous section the Caffeine Bean sets the spec/configuration for two regions. As each region manages two different set of data, they will probably have different needs. For example, customers region stores a single customer while customersSearch stores list of customers. Hence we may require distinct max size for the regions. Another example, suppose data A is updated less frequently than data B. In this scenario, it could make more sense to have a longer time to live set for data A. Tuning the cache properties will depend on how the data is accessed/updated/deleted/stored in you application and the constrains in resources.

Back to our application, we are interested in having particular TTL and max size for the regions. To achive it we need to make a few changes in class CaffeineCacheConfig.

@Bean
public CacheManager cacheManager() {
    CaffeineCacheManager cacheManager = new CaffeineCacheManager();
    cacheManager.registerCustomCache("customers", 
        buildCache(100,200, 2));
    cacheManager.registerCustomCache("customersSearch",
        buildCache(50,100, 1));
    return cacheManager;
}

private Cache buildCache(
    int initialCapacity, int maximumSize, int durationInHours) {
    return Caffeine.newBuilder()
        .initialCapacity(initialCapacity)
        .maximumSize(maximumSize)
        .expireAfterAccess(durationInHours, TimeUnit.HOURS)
        .build();
}

CaffeineCacheManager provides a convinient method to register a native a Caffeine Cache instance. Any number of custom caches may be registered by invoking this method as many times as needed. This allows for custom settings per cache.
The private method buildCache creates a Cache instance with some features. This is typically used with the Caffeine builder API. The build() method returns a Cache with the features set while constructing the builder.

Notification on Eviction

Caffeine provides a mechanism to notify when an entry is removed from the cache. Listeners can be added to the Cache config. There are two type of listeners:

1. Eviction Listener: it is triggered when an eviction happens ( meaning removal due to the policy). This operation is performed synchronously.
2. Removal Listener: it is triggered as a consequence of eviction or invalidation (meaning manual removal by the caller). The operation is executed asynchronously using an Executor, where the default executor is ForkJoinPool.commonPool() and can be overridden via Caffeine.executor(Executor).

Both listeners receive a RemovalListener which is a functional interface. Its abstract method signature is

void onRemoval​(@Nullable K key, @Nullable V value, RemovalCause cause)

key and value are normally Object type and RemovalCause is an enum with the specific cause (EXPLICIT,REPLACED,COLLECTED,EXPIRED,SIZE).

Registering listeners in the Cache config is as simple as the next code

private Cache<Object, Object> buildCache(
    int initialCapacity, int maximumSize, int durationInHours) {
    return Caffeine.newBuilder()
        .initialCapacity(initialCapacity)
        .maximumSize(maximumSize)
        .expireAfterAccess(durationInHours, TimeUnit.MINUTES)
        .evictionListener((Object key, Object value, 
            RemovalCause cause) ->
            logger.info(String.format(
                "Key %s was evicted (%s)%n", key, cause)))
        .removalListener((Object key, Object value, 
            RemovalCause cause) ->
            logger.info(String.format(
                "Key %s was removed (%s)%n", key, cause)))
        .build();
}

Let's have a look at 3 user cases to verify the listeners.

• Deleting a customer:

This operation is done using the command

shell(administrator1):>delete-customer 2
Customer 2 deleted successfully

The customer is loaded first hence it is added to the cache via @Cacheable. Then, it is deleted from the db and it is evicted manually from cache via @CacheEvict. This last action causes the listener to be executed

d.n.r.config.CaffeineCacheConfig: Key 2 was removed (EXPLICIT)

• Excedding the max size in the cache:

If the cache reaches its max size, then new entries will cause old entries to be evicted.

Finding a new customer when the cachee is full

shell(administrator1):>find-customer 21

causes the removal of another entry. Note that both listeners were executed this time

d.n.r.config.CaffeineCacheConfig: Key 2 was evicted (SIZE)
d.n.r.config.CaffeineCacheConfig: Key 2 was removed (SIZE)

• Expired entry:

Customer 1 is searched via the find-customer command and added to the cache with key 1. Then, there are no more accesses to key 1 and time to live expires. However, the entry is not removed until a new customer is searched. In this case customer 2 is fetched and the removal of key 1 occurs.

d.n.r.config.CaffeineCacheConfig: Key 1 was evicted (EXPIRED)
d.n.r.config.CaffeineCacheConfig: Key 1 was removed (EXPIRED)

Cleanup

By default, Caffeine does not perform cleanup and evict values "automatically" or instantly after a value expires. Instead, it carries out small pieces maintenance of work after write operations or occasionally after read operations if writes are rare.

If your cache is mainly read and seldom writes, it is possible to delegate the cleanup task to a thread. A scheduler can be specified to prompt removal of expired entries regardless of whether any cache activity is occurring at that time.

return Caffeine.newBuilder()
    .initialCapacity(initialCapacity)
    .maximumSize(maximumSize)
    .expireAfterAccess(durationInHours, TimeUnit.MINUTES)
    .evictionListener((Object key, Object value, 
        RemovalCause cause) -> logger.info(
           String.format("Key %s was evicted (%s)%n", key, cause)))
    .removalListener((Object key, Object value, 
        RemovalCause cause) -> logger.info( 
           String.format("Key %s was removed (%s)%n", key, cause)))
    .scheduler(Scheduler.systemScheduler())
.build();

After adding the systemScheduler to the cache config, the entries are automatically removed once they expire.

d.n.r.config.CaffeineCacheConfig : Key 21 was evicted (EXPIRED)
d.n.r.config.CaffeineCacheConfig : Key 21 was removed (EXPIRED)
d.n.r.config.CaffeineCacheConfig : Key 22 was evicted (EXPIRED)
d.n.r.config.CaffeineCacheConfig : Key 22 was removed (EXPIRED)

The scheduling is best-effort and does not make any hard guarantees of when an expired entry will be removed. This means, the entries may not be removed immediately at the time of expiration.

Enable Statistics

Statistics can be turned on with the recordStats in the Caffeine builder. Then, the Cache.stats() method returns a CacheStats object which provides statistics such as

• hitRate(): returns the ratio of hits to requests
• evictionCount(): the number of cache evictions
• averageLoadPenalty(): the average time spent loading new values

For instance, we can code a Cache Controller to present the stats as well as the list of entries in the cache regions.

@RestController
@RequestMapping("admin/cache")
public class CacheController {
    private CacheManager cacheManager;

    public CacheController(CacheManager cacheManager) {
        this.cacheManager = cacheManager;
    }

    @GetMapping()
    public List<cacheInfo> getCacheInfo() {
        return cacheManager.getCacheNames()
            .stream()
            .map(this::getCacheInfo)
            .toList();
    }

    private cacheInfo getCacheInfo(String cacheName) {
        Cache<Object, Object> nativeCache = 
          (Cache)cacheManager.getCache(cacheName).getNativeCache();
        Set<Object> keys = nativeCache.asMap().keySet();
        CacheStats stats = nativeCache.stats();
        return new cacheInfo(
          cacheName, keys.size(), keys, stats.toString());
    }

    private record cacheInfo(
        String name, int size, Set<Object> keys, String stats) {}
}

Accessing the new endpoint returns valuable info that can be used to fine-tune the caches, specially in performance-critical applications.

[
    {
        "name": "customersSearch",
        "size": 5,
        "keys": [
            [null,null,null,null],
            [null,"ACTIVATED",null,null],
            [null,null,null,false],
            [null,null,null,true],
            ["name","ACTIVATED",null,null]
        ],
        "stats": "CacheStats{hitCount=0, missCount=11, loadSuccessCount=0, loadFailureCount=0, totalLoadTime=0, evictionCount=0, evictionWeight=0}"
    },
    {
        "name": "customers",
        "size": 10,
        "keys": [1,2,3,4,5,7,9,13,888,910],
        "stats": "CacheStats{hitCount=2, missCount=12, loadSuccessCount=0, loadFailureCount=0, totalLoadTime=0, evictionCount=0, evictionWeight=0}"
    }
]

Spring Actuator Cache Metrics

Spring Actuator also provides cache related information. The endpoint /caches displays a list of all configured caches. In our app, accessing the url http://localhost:8080/actuator/caches will send back the json

{
    "cacheManagers": {
        "cacheManager": {
            "caches": {
                "customersSearch": {
                    "target": "com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache"
                },
                "customers": {
                    "target": "com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalManualCache"
                }
            }
        }
    }
}

The metrics endpoint supports Caffeine library and provides a basic set of metrics. However, only caches that are configured on startup are bound to the registry.

The url http://localhost:8080/actuator/metrics/cache.size?tag=cache:customers displays the size of the cache region customers

{
    "name": "cache.size",
    "description": "The number of entries in this cache. This may be an approximation, depending on the type of cache.",
    "measurements": [
        {
            "statistic": "VALUE",
            "value": 10
        }
    ],
    "availableTags": [
        {
            "tag": "cache.manager",
            "values": [
                "cacheManager"
            ]
        },
        {
            "tag": "name",
            "values": [
                "customers"
            ]
        }
    ]
}

There are other metrics but they are coming empty. For this reason I would recommend to code your own controller or service to get the statistics directly from the cache provider just like it was done in section Enable Statistics.

Summary

We come to a conclusion now. We have learned how the library caffeine works internaly and its main features. Then we have seen how to configure it globlaly and with particular settings per cache region. We also explored other not so common functionality of Caffeine such as removal listeners, clean up, statistics with plenty of examples to demostrate it.

Git repo with the code is available here.

Feel free to share this post with your colleagues and subcribe for more Java and Spring related content!