Spring Cloud Loadbalancer with Eureka Server

Introduction

I built a very simple application that takes advantage of the Eureka Discovery Service.

It consists of a web application that needs to consume an external service in order to complete his work.

At some point, there will be several service instances running on different machines. The web application uses Eureka to discover the available service instances.

The idea is to simulate an application that reacts to a high load situation, adding more services in order to maintain the response time under certain limits.

I found some difficulties trying to build a version that works well under relatively high load so I will describe the solution that worked for me.

I use Spring Cloud 2021.0.4 and Spring Boot 2.6.11 version.

.

The problem

When several service instances were running and processing the requests sent by the web application, If one of them was shut down the loadbalancer still sent requests to him.

Therefore the application was not able to process the request and an error was sent to the client.

This graph belongs to JMeter, which was used to simulate concurrent application’s clients.

The green line represents the number of failed transactions that the clients received after one of the service instances was shut down.

After several seconds the loadbalancer started to send requests to available service instances again and the problem was solved.

Expected behavior

The loadbalancer executing in the web application should only send requests to available service instances.The client should not receive an error when a service instance is shut down.

First intent.

The time period where the failed requests were registered corresponds to the loadbalancer cache time to live (ttl) configuration. So decreasing the cache refresh time improves the situation.

#Configuration of web client application.
spring.cloud.loadbalancer.cache.ttl=10s

To decrease the loadbalancer cache expiration time seems to be a work around due to the fact that the loadbalancer should avoid sending requests to an unavailable instance even with an out of date cache.

Second intent.

Another possible idea was to surround the RestTemplate call with a retry mechanism using a loop or an already retry implementation like Resilience4j, which is already integrated with Spring Boot.

The problem with this approach is that the loadbalancer is a singleton component and shares the state with all the incoming requests.

When the balancer uses a Round Robin algorithm (even the problem applies to other implementations) there is no guarantee that when a request is sent to an unavailable instance the second try will not be processed for the same instance again and again.

Actually with few service instances it is very easy to reproduce the problem.

Third intent

According to the Spring Cloud documentation we need to add a dependency to our web client application.

        <dependency>
            <groupId>org.springframework.retry</groupId>
            <artifactId>spring-retry</artifactId>
        </dependency>

When present, this retry implementation is used internally by the Spring Cloud loadbalancer component.

Now, after running the first test again, the client does not get any error when a service instance is shut down. As the graph shows, the web application throughput was affected for a moment, but at least the client did not receive any failed request.

As the metrics show, the loadbalancer still is sending request to the unavailable instance, but at least in conjunction with Spring Retry is able to deal with the ConnectionException in a more accurate way.

Finally, the solution that worked better for me was to disable the LoadBalancer cache an to increase the retries max to three.

spring.cloud.loadbalancer.cache.enabled=false
spring.cloud.loadbalancer.retry.max-retries-on-next-service-instance=3

Conclusion

It is mandatory to include the Spring Retry dependency in order to improve the resilience of the Spring Cloud loadbalancer implementation.