The circuit breaker pattern in microservices stops calls to a dependency that keeps failing. Your service answers from a fallback in microseconds while the dependency recovers. Without it, one sick service drags down every caller that waits on it, and a small feature takes the whole product with it.
What happens when a node dies
Picture a checkout that calls a pricing service. Pricing loses its database, or its pods crash during a bad deploy. Checkout keeps calling.
If pricing fails fast with connection refused, checkout gets errors and stays up. The slow failure hurts more. Pricing accepts connections and never answers, because its threads are stuck on the dead database. Each checkout request now waits for its full timeout. At peak, checkout’s threads or event loop fill with those waits. Checkout stops answering its own health checks and its own callers. Then the outage spreads up the chain to the storefront.
Timeouts and retries limit the damage per request. The timeouts and retries recipe covers both. They don’t stop the next request from paying the same timeout, and the one after that. A breaker counts the failures, and past a threshold it stops calling for a while.
A breaker has three states:
- Closed. Calls go through. The breaker records successes, failures and slow calls in a sliding window.
- Open. The failure rate crossed the threshold. Calls fail at once with an exception, and your code serves a fallback.
- Half-open. After a wait, the breaker lets a few trial calls through. If they pass, it closes. If they fail, it opens again.
These are the errors your logs show when a breaker is open:
io.github.resilience4j.circuitbreaker.CallNotPermittedException: CircuitBreaker 'pricing' is OPEN and does not permit further calls
circuit breaker is open
too many requests
pybreaker.CircuitBreakerError: Failures threshold reached, circuit breaker opened
The second and third come from Go’s gobreaker. “too many requests” means a half-open breaker refused a call beyond its trial count.
Recovery with the defaults. The library defaults are loose for a user-facing call. Resilience4j needs 100 calls before it judges the failure rate. It counts a call as slow past 60 seconds and stays open for 60 seconds. gobreaker trips after more than 5 consecutive failures and waits 60 seconds. pybreaker trips after 5 consecutive failures and waits 60 seconds. Without a client timeout, a breaker that treats 59-second calls as healthy won’t open on a hung dependency in time to help.
The failover setup
Each dependency gets its own breaker, its own timeout and a fallback. The fallback is a product decision. Pricing can serve the last known price with a “may have changed” flag. Recommendations can serve a static list. A payment call has no safe fallback, so it fails fast with a clear message.
Java with Resilience4j. Add io.github.resilience4j:resilience4j-spring-boot3, plus spring-boot-starter-aop, spring-boot-starter-actuator and micrometer-registry-prometheus. Then configure the breaker in application.yml:
resilience4j:
circuitbreaker:
instances:
pricing:
slidingWindowType: TIME_BASED
slidingWindowSize: 30
minimumNumberOfCalls: 20
failureRateThreshold: 50
slowCallDurationThreshold: 2s
slowCallRateThreshold: 80
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 5
automaticTransitionFromOpenToHalfOpenEnabled: true
recordExceptions:
- java.io.IOException
- org.springframework.web.client.ResourceAccessException
- org.springframework.web.client.HttpServerErrorException
ignoreExceptions:
- org.springframework.web.client.HttpClientErrorException
retry:
instances:
pricing:
maxAttempts: 2
waitDuration: 200ms
ignoreExceptions:
- io.github.resilience4j.circuitbreaker.CallNotPermittedException
management:
endpoints:
web:
exposure:
include: health,prometheus
Each setting explained:
slidingWindowType: TIME_BASEDwithslidingWindowSize: 30judges the last 30 seconds of calls. A count-based window of 100 calls can hold minutes of history on a quiet endpoint.minimumNumberOfCalls: 20stops the breaker from judging on two or three calls. Set it to what your service sees in a few seconds at normal traffic.failureRateThreshold: 50opens the breaker when half the calls in the window fail.slowCallDurationThreshold: 2sandslowCallRateThreshold: 80treat a call slower than 2 seconds as bad, and open when 80% are slow. This catches the hung dependency. Set the duration a little above the dependency’s p99 at peak, and below your client timeout.waitDurationInOpenState: 30sis how long the breaker stays open before trial calls. Shorter recovers faster, but probes a sick service more often.permittedNumberOfCallsInHalfOpenState: 5sets the trial calls. One lucky call shouldn’t close the breaker.recordExceptionsandignoreExceptionsdecide what counts. A 404 or 400 says nothing about pricing’s health, so it’s ignored.- The retry’s
ignoreExceptionsentry stops retries when the breaker is open. Retrying a refused call wastes the wait.
The service wraps the call and names the fallback. RestClient needs Spring Framework 6.1, and the timeouts sit on the request factory:
import io.github.resilience4j.circuitbreaker.CallNotPermittedException;
import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker;
import io.github.resilience4j.retry.annotation.Retry;
import java.time.Duration;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestClient;
@Service
public class PriceClient {
private final RestClient rest;
private final PriceCache cache; // your own last-known-price store
public PriceClient(RestClient.Builder builder, PriceCache cache) {
var factory = new SimpleClientHttpRequestFactory();
factory.setConnectTimeout(Duration.ofMillis(500));
factory.setReadTimeout(Duration.ofSeconds(3));
this.rest = builder.baseUrl("http://pricing.internal").requestFactory(factory).build();
this.cache = cache;
}
@Retry(name = "pricing")
@CircuitBreaker(name = "pricing", fallbackMethod = "lastKnownPrice")
public Price getPrice(String sku) {
Price price = rest.get().uri("/v1/prices/{sku}", sku).retrieve().body(Price.class);
cache.put(sku, price);
return price;
}
// Runs when the breaker refuses the call; other errors reach the caller.
private Price lastKnownPrice(String sku, CallNotPermittedException e) {
return cache.get(sku).markStale();
}
}
Resilience4j applies its aspects as Retry ( CircuitBreaker ( ... ) ) by default. Each retry counts as a call in the breaker’s window, and the open breaker stops the retries. Keep that order.
If you turn on registerHealthIndicator, an open breaker reports the app as DOWN on /actuator/health. I keep breaker health out of the Kubernetes readiness probe. Otherwise a pricing outage marks every checkout pod unready, and Kubernetes pulls all of them from the Service.
Go with gobreaker. ReadyToTrip here trips on a failure rate after a minimum number of calls, which matches the Java setup:
package main
import (
"context"
"errors"
"fmt"
"io"
"log/slog"
"net/http"
"time"
"github.com/sony/gobreaker"
)
var errUpstream = errors.New("upstream 5xx")
var pricing = gobreaker.NewCircuitBreaker(gobreaker.Settings{
Name: "pricing",
MaxRequests: 5, // trial calls allowed while half-open
Interval: 60 * time.Second, // closed state clears its counts this often
Timeout: 30 * time.Second, // open state lasts this long before half-open
ReadyToTrip: func(c gobreaker.Counts) bool {
// Trip on a failure rate, and only after enough calls to judge it.
return c.Requests >= 20 && float64(c.TotalFailures)/float64(c.Requests) >= 0.5
},
IsSuccessful: func(err error) bool {
// A cancelled caller says nothing about the dependency's health.
return err == nil || errors.Is(err, context.Canceled)
},
OnStateChange: func(name string, from, to gobreaker.State) {
slog.Warn("circuit breaker state change", "breaker", name, "from", from.String(), "to", to.String())
},
})
var client = &http.Client{Timeout: 2 * time.Second}
func fetchPrice(ctx context.Context, url string) ([]byte, error) {
body, err := pricing.Execute(func() (interface{}, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 500 {
return nil, fmt.Errorf("%w: %d", errUpstream, resp.StatusCode)
}
return io.ReadAll(resp.Body)
})
if errors.Is(err, gobreaker.ErrOpenState) || errors.Is(err, gobreaker.ErrTooManyRequests) {
return cachedPrice(), nil // fallback: last known price, flagged as stale upstream
}
if err != nil {
return nil, err
}
return body.([]byte), nil
}
func cachedPrice() []byte { return []byte(`{"stale":true}`) }
gobreaker counts in fixed Interval windows, not a sliding one. Its v2 module (github.com/sony/gobreaker/v2) adds generics, so Execute returns your type with no cast. Export the state as a Prometheus gauge from OnStateChange if you run Go services, since the library has no metrics of its own.
Python with pybreaker. pybreaker trips on consecutive failures. That suits a call with steady traffic, and it reacts slower than a rate on a mixed result stream:
import logging
import pybreaker
import requests
log = logging.getLogger(__name__)
class LogStateChange(pybreaker.CircuitBreakerListener):
def state_change(self, cb, old_state, new_state):
log.warning("circuit breaker %s: %s -> %s", cb.name,
old_state.name if old_state else None, new_state.name)
class NotFound(Exception):
"""A 404 says nothing about the dependency's health."""
pricing_breaker = pybreaker.CircuitBreaker(
name="pricing",
fail_max=10, # consecutive failures that open the breaker
reset_timeout=30, # seconds open before a trial call
exclude=[NotFound],
listeners=[LogStateChange()],
)
session = requests.Session()
@pricing_breaker
def _fetch_price(sku):
resp = session.get(f"http://pricing.internal/v1/prices/{sku}", timeout=(0.5, 2))
if resp.status_code == 404:
raise NotFound(sku)
resp.raise_for_status() # 5xx counts as a failure
return resp.json()
def get_price(sku):
try:
return _fetch_price(sku)
except pybreaker.CircuitBreakerError:
return last_known_price(sku) # fallback while the breaker is open
def last_known_price(sku):
return {"sku": sku, "stale": True}
Each Gunicorn worker process holds its own breaker state. Pass state_storage=pybreaker.CircuitRedisStorage(...) to share it across workers through Redis.
A breaker per process is the norm in every library above. Twenty pods each learn about an outage on their own, which takes minimumNumberOfCalls failures per pod. Size that number for one pod’s traffic, not the fleet’s.
Test the failover
I break the dependency on purpose, in staging first, under load. Keep a fixed arrival rate on the caller with k6, so a slower checkout doesn’t get less traffic:
import http from 'k6/http';
export const options = {
scenarios: {
peak: {
executor: 'constant-arrival-rate',
rate: 200, timeUnit: '1s', duration: '20m',
preAllocatedVUs: 200, maxVUs: 2000,
},
},
thresholds: { http_req_duration: ['p(99)<800'] },
};
export default function () {
http.get('https://staging.example.com/checkout/quote?sku=123');
}
Run three faults, five minutes apart, each for five minutes:
- Hard down.
kubectl -n staging scale deployment/pricing --replicas=0. Calls fail fast with connection errors. - Hung. Put Toxiproxy between checkout and pricing and add
toxiproxy-cli toxic add -t timeout -a timeout=0 pricing. Connections open and nothing comes back. This is the case the slow-call threshold exists for. - Partial. Make one pricing pod return 503, or add latency to half the traffic. The failure rate should sit near your threshold, which shows whether the breaker flaps.
Watch the breaker state in /actuator/prometheus or your logs, with checkout’s p99 and error rate beside it. Track the share of fallback responses and checkout’s thread pool or event loop lag. Then restore pricing and watch the breaker go half-open and close.
Write the pass criteria first, as your own targets. The breaker opens within the time you set, such as 30 seconds, for the hard and hung cases. While it’s open, checkout’s p99 stays under budget and every quote returns, with a stale flag where needed. Checkout’s pods stay ready. After pricing recovers, the breaker closes within waitDurationInOpenState plus a few seconds, without a spike of errors. In the partial case it shouldn’t flip state more than a few times.
Keep it healthy
Spring Boot exposes the Resilience4j Micrometer metrics on /actuator/prometheus. The state gauge reports 1 for the current state. The failure rate is a percentage, and it reads -1 until the window holds enough calls:
groups:
- name: circuit-breakers
rules:
- alert: CircuitBreakerOpen
expr: max by (job, name) (resilience4j_circuitbreaker_state{state="open"}) == 1
for: 1m
labels:
severity: critical
annotations:
summary: "Circuit breaker {{ $labels.name }} in {{ $labels.job }} is open"
description: "Calls to {{ $labels.name }} are failing over to the fallback. Check that dependency's error rate, pods and database; the breaker closes on its own once trial calls pass."
- alert: CircuitBreakerFlapping
expr: max by (job, name) (changes(resilience4j_circuitbreaker_state{state="open"}[30m])) > 6
for: 0m
labels:
severity: warning
annotations:
summary: "Circuit breaker {{ $labels.name }} in {{ $labels.job }} keeps opening and closing"
description: "The failure rate sits near the threshold. Fix the partial failure in {{ $labels.name }}, or raise permittedNumberOfCallsInHalfOpenState so one good trial can't close it."
- alert: CircuitBreakerFailureRateRising
expr: max by (job, name) (resilience4j_circuitbreaker_failure_rate) > 25
for: 5m
labels:
severity: warning
annotations:
summary: "Failure rate through breaker {{ $labels.name }} is above 25%"
description: "The breaker opens at 50%. Check {{ $labels.name }} now, before calls start going to the fallback."
- alert: CircuitBreakerRejectingCalls
expr: sum by (job, name) (rate(resilience4j_circuitbreaker_not_permitted_calls_total[5m])) > 0
for: 10m
labels:
severity: warning
annotations:
summary: "Breaker {{ $labels.name }} has refused calls for 10 minutes"
description: "The dependency hasn't recovered. Check whether the fallback is still fit to serve, such as how stale cached prices are now."
CircuitBreakerFlapping uses for: 0m because changes() already counts over 30 minutes. Set the 25% warning from your own baseline failure rate. For Go and Python, export a state gauge with the same labels from the state-change hook. One set of rules then covers all three.
Roll back
Keep the breaker settings in config, such as application.yml or environment variables, and note the current values before you change them. To roll back, restore the old values and redeploy. To take a breaker out of the path, remove the annotation or the wrapper and ship that. The call then goes straight to the dependency.
A breaker that opens on healthy traffic is causing errors, so roll it back now. The cause tends to be a minimumNumberOfCalls set too low, or a slow-call threshold below the dependency’s normal p99. Raise those before you remove the breaker. For any other reason, roll back off-peak, because the next dependency failure will then reach your users with no fallback.
More recipes
- Stop a Cache Stampede with Coalescing and Early Refresh
Let one request rebuild an expired key while the rest wait or get the stale copy, so a thundering herd stops hitting your database when a hot key expires.
- Exponential Backoff and Jitter for Timeouts and Retries
Give every call a deadline, a per-try timeout and jittered backoff under a retry budget, so a slow dependency can't set off a retry storm.
- Improve Cache Hit Ratio to Protect Your Origin at Peak
Trim the cache key, send cache headers from the origin and serve stale while revalidating, so the CDN absorbs peak traffic instead of passing it to your origin.
- Fix ALB 502 Bad Gateway and 504 Gateway Timeout
Set each target's keep-alive above the ALB idle timeout and keep slow requests under it, so the load balancer stops answering peak traffic with 502 and 504.