| name | Circuit Breaker Builder |
| description | Wraps flaky upstream dependencies in circuit breakers, aggressive timeouts, and per-dependency bulkheads so a slow or failing service degrades gracefully instead of cascading into a full outage. Use when a slow or unavailable upstream is stalling your threads, outbound calls hang with no timeout, one dependency's outage is taking down unrelated traffic, or you are integrating a network call that can realistically be slow or down. Do NOT use when the goal is staying under a provider's request quota or handling 429s — use rate-limit-handler instead; do NOT use to size a database connection pool — use connection-pool-tuner instead. |
Circuit Breaker Builder
Protect a service from slow or unavailable upstreams: fail fast, fall back, and isolate the blast radius so one sick dependency can't drain your whole capacity.
Workflow
-
Set an explicit timeout on every outbound call. Configure connect and read timeouts separately; never rely on library defaults (often minutes or infinite). The total budget must be shorter than your own caller's timeout so you fail before they give up. A call with no timeout is a guaranteed resource leak under load.
-
Wrap the call in a three-state breaker. Closed: calls pass through. Open: calls fail instantly without touching the dependency. Half-open: a few probe calls test recovery. Implement these states explicitly per dependency, not a single global breaker.
-
Trip on a rolling error rate, not a raw count. Open the breaker when the error rate crosses a threshold over a sliding window with a minimum request volume (e.g. >50% over the last N calls, ignore until at least M calls). A raw failure count flaps at low traffic and trips on noise.
-
Define a fallback per call site. While open, return immediately: stale cache for reads, queue-for-later or accept-and-reconcile for writes, a sane default or degraded response where one exists, and a clean fast error only when there is no safe degradation. The point is to stop spending threads, connections, and latency on a call you expect to fail.
-
Probe and recover from half-open. After a cooldown, allow a limited number of probe calls. If they succeed, close the breaker. If any fail, re-open and reset (and back off) the cooldown. Never flood the recovering dependency with full traffic the instant the cooldown expires.
-
Give each dependency its own bulkhead. Isolate every upstream behind its own concurrency limit — a dedicated connection pool, thread pool, or concurrency semaphore. A saturated dependency then exhausts only its own bulkhead and leaves capacity for unrelated traffic. Never share one pool across multiple upstreams.
-
Emit metrics and alert. Record state transitions, trip counts, fallback rates, and timeout rates per dependency. Alert when a breaker opens. Tune thresholds and cooldowns against this real data, not against guesses.
Quality bar
- Every outbound network call has explicit connect and read timeouts, each shorter than the inherited caller budget.
- Breaker trip logic uses a rolling error rate with a minimum-volume guard, not a bare count.
- Each protected dependency has its own bulkhead; no two upstreams share a pool.
- Every call site behind a breaker has a defined open-state fallback.
- State transitions are observable and alert on open.
Do NOT
- Do NOT trust default library timeouts, or ship any outbound call without one.
- Do NOT retry through an open breaker — that defeats the fail-fast behavior; let the breaker reject.
- Do NOT trip on a raw failure count at low volume; it flaps and masks real signal.
- Do NOT share a connection or thread pool across dependencies; one slow upstream then stalls all of them.
- Do NOT wrap fast, in-process, or highly reliable local calls — the overhead and false trips outweigh the benefit.
- Do NOT use a breaker to ration calls against a provider quota (that is rate-limit-handler's job) or to decide a DB pool's maximum size (that is connection-pool-tuner's job).
