Circuit Breaker: What to do when a service fails constantly?

As a project grows, it’s common that each service communicates with a variety of others. For example, a checkout process to reserve a hotel room might include the following steps:

1. Verify availability of the resource we want to reserve
2. Validate the user’s information
3. Validate the credit card information
4. Confirm the reservation
5. Notify the user by email of the completion

It would be difficult for all of these steps to be in one service — frankly, they shouldn’t be. Some of these steps could be carried out by third party libraries.

Let’s suppose that the process that validates our credit card stops working. How should our application respond to this failure? Can it recover from this error? The answers to these questions help us to measure if our application is resilient. We can say that an application is resilient if it has the capacity to recover from a failure and continue operating.

One form to achieve resiliency is through a development pattern known as a Circuit Breaker.

What is, and how does a circuit breaker work?

This pattern is used often in microservices and gets its name from electric engineering. It references a piece of hardware that has the capability of cutting the electrical current inside of itself. In our case, we can use this name for software that cuts communication between processes when one of the processes stops responding.

Why would we use this design pattern?

1. To save money
   What happens if the step just before a failing service uses a paid service, or one that has an API rate limit? We’re wasting resources, given that the step immediately following the paid service isn’t working.
2. Save resources
   Does it make sense to respond to a request, or invoke a method if we know a service isn’t responding?
3. Avoid a bad user experience
   What kind of user experience are we providing if, after completing various steps, a user is faced with a broken checkout system, and can’t complete their reservation?

This brings us to the question, should we protect all of our resources with a circuit breaker? In my opinion, we only need to worry about those which are critical due to integrations with multiple services.

States of a circuit breaker

A circuit breaker works with three distinct states:

1. Open: Fails immediately and does not execute a request.
2. Half-open: A request is executed after a set time, here referred to as a timeout. If the request executes successfully, then the state is changed to closed. If it continues failing, the state returns to open.
3. Closed: A request is executed as normally would be expected.

Note: The name of the states can generate confusion, but it can be helpful to remember that the circuit breakers we use here are analogous to electrical circuits.

How do we determine the correct state?

We use thresholds to determine when to change from one state to another. This value is adjusted according to whether a resource executes correctly or not. When a resource fails, we add 1 to our count of errors. Once we reach a specified value in the threshold, we change the circuit breaker to open — or in this case, we indicate that we should not execute any more calls to this resource. Instead, we should immediately return with an error message, without advancing in our execution.

After a certain amount of time (let’s say 5 minutes since our last request), we can change the state to half-open, and allow a request to proceed to the resource. If it doesn’t fail, then we can reset our failure counter, and change the status once again to closed.

How do we implement a circuit breaker?

In Python, there are quite a few options; in this case, we will explore pybreaker as an example.

fail_max indicates the maximum number of allowed errors before changing our status to open.

reset_timeout indicates the time (in seconds) before changing to half-open since our last error.

Next, let’s take a look at how pybreaker implements the states of the circuit breaker.

Implementation for ClosedState and HalfOpenState

In the implementation for these states, we see that the request is executed in the following line:

ret = func(*args, **kwargs)

In this case, func is validate_credit_card in our example. This execution allows for two possible responses:

1. Success: In this case, we reset our error count (_handle_success)
2. Error: If our system generates an error, we check to see if the exception is not excluded; pybreaker allows us to exclude certain exceptions. If it’s not excluded, then we increment our error count; if it is, we consider func to have executed successfully.

Implementation of on_success for ClosedState

If the current state is closed, then we don’t need to do anything in on_success.

Hook for HalfOpenState

If the current state is half_open, then we change the state to closed in on_success.

Implementation for OpenState

In this implementation, we should check timeout; if we haven’t completed the wait period specified in timeout, then we shouldn’t call the resource, and we should interrupt execution. If the inverse is true, we change the state to half_open and we execute the process (the implementation is described in the earlier example.)

Conclusion

Circuit Breaker allows us to protect the critical processes of our application, making them more resilient, and allowing them to adapt to the current state of the services in use. We should consider using circuit breakers in our processes that invoke different services, or those that have costs in terms of processing, time or money. The usage of circuit breakers also has the potential to generate a better user experience, given that they won’t have to complete a series of steps, all to at the end, encounter a fatal error.

Sources:

1. “Release it!” by Michael Nygard
2. https://pypi.org/project/pybreaker/