Circuit breakers and graceful degradation under failure

A slow dependency can hold connections and request workers until unrelated requests start failing. A circuit breaker reduces repeated calls after failures accumulate. Graceful degradation defines what the application can still return while that dependency is unavailable.

Both need limits around them. Calls already in flight still need deadlines, and the fallback needs enough capacity to survive the outage. Start with one optional dependency and prove its failure behavior before extending the pattern across your application.

Decide what a degraded response means

Consider a hypothetical shop whose product page fetches recommendations from a separate service. The product details can still load if recommendations fail. Showing a small, locally stored list might be acceptable; blocking the whole page until recommendations recover would defeat that choice.

Write down the response contract first. For this example, it could be:

This is a proposed design, not a measured result or a policy every shop should adopt. Choose the allowed cache age with the person responsible for the data. A stock count, account balance or authorization decision has different consequences from a list of suggested products.

Keep the fallback cheaper than the normal request. Sending every cache miss to the same failing database adds load during the outage. Google's overload guidance describes reducing work and serving less expensive responses. Exercise your fallback during routine testing, including when its cache is empty.

What each breaker state permits

In the closed state, the breaker admits calls and records their outcomes. Once its configured failure rule is met, it opens and rejects new calls through that breaker. The application must handle that rejection, usually with a fallback or an explicit error.

After a waiting period, a half-open state admits limited trial traffic. Successful trials can restore normal admission; failed trials can reopen the breaker. Exact transition rules depend on the library. Microsoft's circuit-breaker pattern explains this cycle and the need to handle rejected calls.

Avoid implementing only a failure counter and an open flag. That leaves unanswered how calls resume, how concurrent trials are limited, and when old failures expire. Use the state machine supplied by your library, then test its behavior with your client and error types.

A breaker also needs a useful lifetime. Recreating it for each request discards the history that makes it work. Scope it to the dependency or operation that should share a failure decision. A single global breaker can make an optional recommendation failure block unrelated calls.

Sampling changes when a breaker opens

A failure percentage needs a denominator. Resilience4j supports count-based and time-based windows, plus a minimum number of recorded calls before evaluating failure or slow-call rates. Its slow-call threshold records duration; it does not terminate the call. The Resilience4j documentation also makes clear that the window size does not limit concurrent requests.

For a hypothetical count-based policy, suppose the window holds ten completed calls, the minimum sample is ten, and the breaker opens at a failure rate of at least 50%. Nine failed calls are still below the minimum sample. If the tenth succeeds, nine of ten have failed, so the rule opens the breaker. These numbers demonstrate the rule; they are not recommended production settings.

Time-based sampling has a different low-traffic problem. If too few requests arrive within the sampling period, the minimum may never be reached. Polly documents this behavior: below its minimum throughput, even an entirely failing sample does not open the circuit. Check request volume per breaker instance, including quiet periods, before choosing a threshold.

Classify outcomes explicitly. A rejected business operation, such as an invalid product identifier, need not indicate dependency failure. Decide which exceptions and HTTP results count, and how cancellations are treated. Polly exposes a predicate for this choice. Its breaker rethrows failures and does not perform retries itself.

Bound time, concurrency and retries separately

Set client timeouts that fit inside the caller's total deadline. Include the work before and after the dependency call, plus any retry delay. Check whether the client timeout covers connection setup, DNS and TLS. A timeout means the caller stopped waiting; it does not prove the remote operation had no effect. AWS explains these limits in Timeouts, retries, and backoff with jitter.

Use a bulkhead to cap concurrent work for this dependency. A bulkhead separates capacity, such as a connection pool or a fixed number of execution slots, so one failing service cannot consume every slot available to other work. Put a bound on queued work too. Microsoft's bulkhead guidance covers isolating resource pools and testing those boundaries under failure.

If retries are appropriate, give one layer responsibility for them, limit the attempts, and use backoff with jitter. Retry only operations whose effects can safely be repeated. Stop retrying when the breaker rejects admission. Otherwise, retries in several layers can multiply traffic or waste the remaining request budget on immediate rejections. AWS discusses retry amplification and retry limits; Microsoft's breaker guidance calls out handling non-transient breaker rejections.

Record where the retry logic sits relative to the breaker. A breaker around the entire retry operation sees its final outcome. A breaker around each attempt sees individual attempts. This follows from what each wrapper observes, and it changes how quickly its sample fills. Verify the arrangement your client stack actually uses.

Recovery across several application instances

An in-memory breaker normally tracks one process's experience. Resilience4j's registry is in memory, so separate processes can disagree about the same dependency. That can be useful when only one caller has a broken network path, but it also means one process opening its breaker does not stop traffic from the others.

Account for aggregate trial traffic. In a hypothetical deployment with twenty independent instances, allowing two simultaneous trials per instance can admit forty trials across the fleet. A local limit alone cannot establish a fleet-wide limit. Include the number of callers when load-testing recovery.

A timer expiring only makes recovery eligible to be tested. Check whether your library waits for the next call or performs an automatic transition. Do not manually close every breaker merely because a public health endpoint turns green. That endpoint may use a different route, credential or operation from the failed request.

Run a failure drill and keep the evidence

Use an isolated dependency stub or a staging service where you can inject delays and failures without affecting customers. Record the configured policy, request count at the dependency, breaker transitions and caller-visible result. The following are acceptance cases to run, not results from a lab performed for this article.

Also test a rejected business request and a cancelled caller. Check that neither silently enters the wrong failure category. Repeat recovery with several application instances, since a single-process test misses their combined load.

Track fallback use separately from HTTP success. A page can return 200 while an optional feature remains unavailable for hours. Record dependency errors, rejected calls and time spent open, alongside the age of data served from the fallback.

External checks add a view of the public response. Internal metrics explain the breaker and fallback decisions behind it. Google's monitoring guidance distinguishes these black-box and white-box signals. For the shop example, check that product details still render, then use application telemetry to see whether recommendations are live, cached or omitted. A successful public check alone cannot establish that the dependency recovered.