I'm a .NET backend developer and I'm using Redis on a daily basis. From time to time it seems that my Azure Cache for Redis is not working properly and my services fail to connect to it.
In this state response times go crazy because of multiple Redis calls, the build in retry policies and their timeouts. This high response times may also result in timeouts on client side.
Sure you can optimize your redis settings, but you still don't solve the issue that an external dependency can fail and in my case I want to fail fast(er).

I would prefer to detect an unhealthy state of my redis and start to ignore this cache layer for a certain amount of time. For now I didn't found a ready to use solution. So I implemented one.

Idea

My idea was to build a decorator for the IDistributedCache interface that is able to use injected Polly policies and wrap all actual IDistributedCache calls in them.
With using the decorator pattern I can easily add this package to existing applications and benefit from the new level of resiliency. As a build in policy I choose the CircuitBreaker.

Packages that were used

Polly

https://github.com/App-vNext/Polly
"Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner."

Scrutor

https://github.com/khellang/Scrutor
Scutor adds Assembly scanning and decoration to the .NET dependency injection. Helpful when you want to use decorators or when you want to register all classes that implement an interface.

Microsoft.Extensions.Caching.Abstractions

Used in the package because I going to use the IDistributedCache interface from it.

Code

Here is an extract of the PolicyDistributedCacheDecorator:

    public class PolicyDistributedCacheDecorator : IDistributedCache
    {
        private readonly IDistributedCache _distributedCache;
        private readonly ISyncPolicy _syncPolicy;
        private readonly IAsyncPolicy _asyncPolicy;

        public PolicyDistributedCacheDecorator(IReadOnlyPolicyRegistry<string> registry, IDistributedCache distributedCache)
        {
            _distributedCache = distributedCache;
            _syncPolicy = registry.Get<ISyncPolicy>(PollyPolicies.SyncPolicy);
            _asyncPolicy = registry.Get<IAsyncPolicy>(PollyPolicies.AsyncPolicy);
        }

        public byte[] Get(string key)
        {
            return _syncPolicy.Execute(() => _distributedCache.Get(key));
        }

        public async Task<byte[]> GetAsync(string key, CancellationToken token = default)
        {
            return await _asyncPolicy.ExecuteAsync(() => _distributedCache.GetAsync(key, token)).ConfigureAwait(false);
        }

So we have a IDistributedCache implementation that also get's this interface injected. The injected on will be a "real" implementation or another decorator - depending on your code (when you have multiple decorators for IDistributedCache they will get called first).

You can compare this pattern to the ASP.NET Core middlewares.
Every function will wrap the call of the injected interface in a Polly policy. Now the IDistributedCache works as before while it also adds the policy behaviors we wanted.

Let's come to the dependency injection. You can't simple add PolicyDistributedCacheDecorator like 'services.AddScoped()' since this would result in an endless loop.
But with scrutor you can just do this 'services.Decorate()' and everything will work fine

Included CircuitBreaker

Because in my error cases I get SocketException my circuitbreakers only handle this exceptions

    var breaker = Policy.Handle<SocketException>()
        .AdvancedCircuitBreaker(
            circuitBreakerSettings.FailureThreshold,
            circuitBreakerSettings.SamplingDuration,
            circuitBreakerSettings.MinimumThroughput,
            circuitBreakerSettings.DurationOfBreak);

    return breaker;

You should now that Polly differentiate between sync and async policies - that's why you need to specify both.

When you need to handle more exception types you can do this like this:

Policy.Handle<SocketException>()
      .Or<TimeoutException>() // add as many Or<TException> as you need 

How to use it

1. install the package:

   dotnet add package DistributedCachePollyDecorator --version 1.0.0

2. add the decorator to your dependency injection
   

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddControllers();
        // your settings for Redis
        services.AddStackExchangeRedisCache(options =>
        {
            options.Configuration = "localhost:6379,allowAdmin=true,DefaultDatabase=1";
        });

        // Decorate the inferface with a simple circuit breaker
        services.AddDistributedCacheDecorator(new CircuitBreakerSettings()
        {
            DurationOfBreak = TimeSpan.FromMinutes(5),
            ExceptionsAllowedBeforeBreaking = 2
        });

        // OR decorate it with an advanced one: (don't add both)
        services.AddDistributedCacheDecorator(new CircuitBreakerAdvancedSettings()
        {
            DurationOfBreak = TimeSpan.FromMinutes(5),
            FailureThreshold = 0.5,
            MinimumThroughput = 100,
            SamplingDuration = TimeSpan.FromMinutes(2)
        });

        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo { Title = "WebApplication1", Version = "v1" });
        });
    }

Conclusion

With the use of decorators it was quite easy to extend the IDistributedCache to use Polly for it's calls.

I never published Nuget packages myself, but with the help of Dotnet-Boxed it was super easy.

You can find the complete source code on GitHub