Vert.x Circuit Breaker

Scenario

We can imagine a scenario where we have a restaurant. We have the kitchen service and the orders service that communicate with each other. When a new order arrives, this will be sent to the kitchen, which gets to work and prepares the dishes. When the dishes are ready, they are delivered to the customer. But if we suppose that a really large number of customers come, so consequently a large number of orders, the kitchen gets overloaded by all these orders and therefore it cannot prepare the dishes at the same rate as the orders arrive. At this point the customers are going to wait much time before receiving their dishes because the kitchen is using all the resources for the previous orders.
So we have a deadlock. To make the situation worse, we can immagine that the chef, due to the excessive work, has cut themself and for some time they cannot work.
The scenario just described is defined as cascading failure.

A definition that we can give of a cascading failure is: a process in a system of interconnected parts in which the failure of one or few parts can trigger the failure of other parts and so on. Such a failure may happen in many types of systems, including power transmission, computer networking, finance, transportation systems, organisms, the human body, and ecosystems.

Explore the pattern

The Circuit Breaker Pattern is one solution for the cascading failure. This pattern can prevent an application from repeatedly trying to execute an operation that is likely to fail. Allowing it to continue without waiting for the fault to be fixed. The Circuit Breaker pattern also enables an application to detect whether the fault has been resolved. If the problem appears to have been fixed, the application can try to invoke the operation.

In the previous scenario the circuit breaker is applied to the orders service: every time a new customer comes, the orders service asks the kitchen if it can prepare the dish in a short time. If the kitchen's response is negative, the orders service consider it as a failure and so it understands that the kitchen service is broken and advises the new customer to have a drink at the bar service in order to not overload the kitchen's one.

So a circuit breaker acts as a proxy for operations that might fail. The proxy should monitor the number of recent failures that have occurred and use this information to decide whether to allow the operation to proceed, or simply return an exception immediately.

In this article we see how to use an implentation of this pattern: Vertx Circuit Breaker.

Setup

To use the Vert.x Circuit Breaker, add the following dependency to the dependencies section of your build descriptor:

• Maven (in your pom.xml)

<dependency>
 <groupId>io.vertx</groupId>
 <artifactId>vertx-circuit-breaker</artifactId>
 <version>4.0.3</version>
</dependency>

• Gradle (in your build.gradle file):

compile 'io.vertx:vertx-circuit-breaker:4.0.3'

So now you can use a circuit breaker in your java/kotlin file, here is an example in kotlin:

val breaker = CircuitBreaker.create("test", vertx, CircuitBreakerOptions()
  .setMaxFailures(2)
  .setTimeout(1000)
  .setMaxRetries(10)
  .setResetTimeout(10000)
)

Don’t recreate a circuit breaker on every call. A circuit breaker is a stateful entity. It is recommended to store the circuit breaker instance in a class field.

Inside CircuitBreakerOptions we can set some settings:

• setMaxFailures: number of failures before opening the circuit
• setTimeout: consider a failure if the operation does not succeed in time
• setMaxRetries: number of retries if the operation fails
• setResetTimeout: time to wait for switch from OPEN to HALF OPEN state

For executing code with the circuit breaker we just use execute function.
The executed block receives a Future object as parameter, to denote the success or failure of the operation as well as the result.

// we set promise type to String
breaker.execute<String> { promise ->
  stuff()
  // some code executing with the breaker
  // the code reports failures or success on the given promise.
}.onComplete {
  // Get the operation result.
  println("stuff done") 
}

Let's see how it works this circuit breaker.

State Machine

The circuit breaker uses a state machine for registering all events and behave accordingly.
Below we can see an image of this state machine.

• Circuit Breaker goes into the OPEN state when the service is failed
• Circuit Breaker goes into the CLOSE state when the service it works correctly
• Circuit Breaker goes into the HALF OPEN state when the reset timeout is expired

Open state

When a circuit is in the open state, it means that no code inside an execute block will be ran for the same instance.
The circuit opens when the maxFailures property is satisfied.

Let's see an example:

circuitBreaker.execute<String> { promise ->
  promise.fail("error")
}.onComplete { println(it) }

So we can declare a failure when an execution block fails independently of how many calls fail inside that block. In this case the count of failure will be set to 1.

Close state

When a circuit is in the closed state, it means that the number of failures is less than the maximum. In this state all code will be executed inside the execute block.

In this state it is possible to define the retry number (retries are available just when the circuit is in the closed state). This number specifies how many times the circuit may execute the code before failing. So for example, if we have setMaxRetries to 2, the operation may be called 3 times: the initial attempt plus 2 retries. So at the end of the third call that executed block fails and will increment the number of failures by 1.

Retries can be useful when the answer is not deterministic. In fact, we can take advantage of our scenario by imagining that the order service asks if the dish is ready in the kitchen. This request has an nondeterministic answer as, depending on the moment, different answers can be received. In fact, if the kitchen is busy and the dish is not ready, you may not receive a response, thus sending the request to timeout and creating a failure. Otherwise, if the kitchen has finished, the answer will be successful so the dish will be delivered to the customer. We don't want the first situation to happen because the circuit breaker would immediately count the failure, instead we want to make several calls because we assume that the resource we want takes a longer time to be ready. So we set up retries for this very reason. If in the end the resource is still not available then we will be sure that there is a problem and we will activate the circuit breaker.

By default the timeout between retries is set to 0 which means that retries will be executed one after another without any delay. This, however, will result in increased load on the called service and may delay its recovery. In order to mitigate this problem it is recommended to execute retries with delays.
The retryPolicy method can be used to specify retry policy. Retry policy is a function which receives retry count as a single argument and returns the timeout in milliseconds before retry is executed and allows to implement a more complex policy. Below is an example of retry timeout which grows linearly with each retry count:

val circuitBreaker = CircuitBreaker.create("my-circuit-breaker", vertx, CircuitBreakerOptions()
  .setMaxFailures(5)
  .setMaxRetries(5)
  .setTimeout(2000)
).retryPolicy(retryCount -> retryCount * 100L)

Half open state

When a circuit is in the open state, every call that you want to execute will fail, without executing really the code. At this point, after some time (called ResetTimeout) which by default is set to 30000 ms, the circuit will be set to half open state. In this state the next execution of protected code will define the next state of circuit. If execution succeeds then the state will be set to closed otherwise it will be set to open.

Event Bus

Every time that the state of circuit changes, an event is published on the event bus. The address of the bus can be configured using setNotificationAddress. If null is passed to this method, the notifications will be disabled. By default, the used address is vertx.circuit-breaker.

Each message contains at least:

• state
• name
• failures

For doing this you just need few row of code:

vertx.eventBus().localConsumer<JsonObject>("vertx.circuit-breaker") {
  println(it.body())
}

Handlers

It is possible to configure some callbacks when circuit changes the state. Here's an example:

val breaker = CircuitBreaker.create("my-circuit-breaker", vertx, CircuitBreakerOptions()
  .setMaxFailures(5)
  .setTimeout(2000)
).openHandler {
  println("Circuit opened")
}.closeHandler {
  println("Circuit closed")
}.halfOpenHandler {
  println("Circuit half-opened")
}

Conclusion

In conclusion this pattern is great when you have a system with a large number of service calls which are used to manage a large number of resources.

So as a last tip we recommend to use this pattern to prevent an application from trying to invoke a remote service or access a shared resource if this operation is highly likely to fail.

Don't use this pattern as a replacement for exception handling in your application business logic.

Now you can implement circuit breaker in your project. :)