request_info that says what is missing, in place of no answer or a guessed one.The roof-data provider is having a bad morning. In scenario UW-R03 it stops answering altogether. The submission is SUB-1006, the facts of shared case UW-006: a masonry office whose application says the roof is 12 years old. The workflow needs the provider's figure to check that claim (rule R-ROOF-03 compares the two ages).
One run handles this well, as Retries showed: three attempts of 2000 ms each, with backoff between them, inside the deadline. In this example step 2 gives up after 6960 ms of simulated time.
Now the morning goes on. Brokers keep sending quote requests, and every new run does the same thing: three attempts, three timeouts, about seven seconds of a worker's time spent waiting on a provider that is not there. Take an illustrative 50 runs started in the same minute. That is 150 calls sent to a provider that is already down or overloaded, and 50 workers each held for seven seconds doing nothing useful. If the provider is down because it is overloaded, those 150 calls are the very load that keeps it down. And while every worker waits on the roof provider, submissions that do not need it at all wait behind them.
The naive workflow makes it far worse. In UW-R03 it retries at once, with no cap, until the run's deadline: 60 attempts in 120000 ms, and no recommendation at the end.
Retries were the right answer to one failed call. Against a dependency that is failing for everyone, retries are the problem. This page is about the opposite move: stop calling, refuse some work on purpose, and give a smaller answer that tells the truth.
Plain English. When something you depend on is down, asking it harder does not bring it back. It uses up your own time and its remaining capacity. The reliable move is to notice quickly, stop asking for a while, and do the best honest thing without it.
How retries turn one failing dependency into an overload#
A cascading failure is a failure that feeds itself: one part fails, which makes the parts around it more likely to fail. Google's SRE book describes it as growth through positive feedback, and names retries as a common amplifier: a backend near overload gets more traffic from the clients it is failing, and retries at several layers multiply (Site Reliability Engineering, chapter 22, Addressing Cascading Failures).
Walk the cause and effect for the insurance agent. The provider slows down. Calls start to take the full 2000 ms timeout. Each run now holds a worker three times longer than usual, so fewer workers are free, so new messages wait longer on the queue. The provider, meanwhile, receives every attempt of every run. If its trouble was load, it now has more. Nothing in this loop pushes back, because each run decides alone, from its own three attempts, and no run knows that the last forty runs saw the same thing.
That last sentence is the gap. A retry budget limits how many retries one story spends (Retries); it does not tell a new run that the provider has been down for a minute. Something must remember, across runs, that this dependency is failing.
A circuit breaker stops calling a dependency that keeps failing#
The name comes from the electrical breaker in a house. When too much current flows, it trips and cuts the circuit, so the wiring does not burn. Someone fixes the fault and resets it.
The software version sits in front of one dependency and watches how its calls end. In this example: if 3 of the last 5 calls to the roof-data provider failed, the breaker trips, and for the next 10000 ms every call to the provider is refused at once, without being sent. That component is a circuit breaker: it stops calling a dependency that keeps failing, and lets calls through again after a pause.
Refusing at once has its own name. To fail fast is to return an error immediately when the outcome is already known to be bad, instead of waiting for a timeout to say so. A refused call costs microseconds; a timed-out call costs 2000 ms of a worker and one more request on a struggling provider.
A breaker has three states. Each answers one question: what happens to the next call?
| State | When it is entered | What it does with a call | When it leaves |
|---|---|---|---|
| closed | At the start, and after a successful probe | Sends it; remembers how it ended (the last 5 results here) | When 3 of the last 5 sent calls failed: to open |
| open | When the failure threshold is reached, or a probe fails | Refuses it at once: fail fast, nothing is sent | When the open time (10000 ms here) has passed: to half-open |
| half-open | When the open time is over | Sends it as a probe (1 probe here) | Probe succeeds: to closed. Probe fails: back to open, and the clock restarts |
The half-open state is what makes a breaker recover by itself. Without it, an open breaker either stays open for ever (a person must reset it) or closes on a timer and lets the full flood through at once. Half-open lets one call find out whether the dependency is back, and only that one call pays if it is not. Martin Fowler's description of the pattern, which credits Michael Nygard's book "Release It!" with popularising it, has the same three states and the same trial call after a reset timeout (CircuitBreaker, martinfowler.com).
Diagram: The three states of a circuit breaker · what happens to the next call
Read the arrows as transitions. In closed, calls go through and the breaker remembers the last 5 results. Three failures among them open it. In open, every call is refused at once and nothing reaches the dependency. When the open time is over the breaker becomes half-open and lets one probe through: success closes it, failure opens it again. The numbers are this example's settings and are illustrative.
- Closed · calls go through
- closed: The normal state. Every call is sent to the dependency and its result joins a window of the last 5 results. A 429 counts as a success: the dependency is alive.
- Open · fail fast
- open: Calls are refused immediately and never sent. The caller gets an error in microseconds instead of a 2000 ms timeout, and the dependency gets no load from this caller.
- Half-open · one probe
- half-open: After the open time, one call is sent as a probe to find out whether the dependency is back. Only that call pays if it is not.
Here is the breaker of the example, CircuitBreaker in breaker.py, the two methods that hold the state machine (dedented from the class, otherwise verbatim). allow answers "may a call be sent now?" and moves open to half-open when the time is up. record is told how a sent call ended.
def allow(self, now_ms: int) -> bool:
"""May a call be sent now? Moves from open to half_open when `open_ms` has passed."""
if self.state == "open" and now_ms - self.opened_at_ms >= self.open_ms:
self.state, self.probe_successes = "half_open", 0
return self.state != "open"
def record(self, ok: bool, now_ms: int) -> None:
"""Tell the breaker how a call that was sent ended."""
if self.state == "half_open":
self.probe_successes = self.probe_successes + 1 if ok else 0
if not ok:
self._open(now_ms)
elif self.probe_successes >= self.half_open_probes:
self.state, self.window = "closed", []
return
self.window = (self.window + [ok])[-self.window_calls:]
if self.window.count(False) >= self.failure_threshold:
self._open(now_ms)
One line in CircuitBreaker.call, next to these, matters as much: a 429 is recorded as a success for the breaker, with the comment "A 429 does not count as a failure: the dependency is alive." A provider that says "slow down, try in 1500 ms" is working; it is managing its load. Opening the breaker on 429s would stop calling a healthy dependency. The breaker counts only the errors that say the dependency is not answering: timeouts and unavailability.
UW-R03: the breaker opens, and the run degrades#
The sequence first, then the run that produced it.
Diagram: UW-R03 · three bounded attempts, the breaker opens, the run degrades to request_info
Time runs down the page. The worker calls the roof-data provider through the breaker. Two timeouts, two waits, a third timeout: the third failure in the window opens the breaker, and the attempts are used up at the same moment. The run does not stop. It goes on without the roof assessment and stores a recommendation that asks for it. A person decides. Times are from the simulated clock; the agent is a scripted stand-in.
- Worker (the run) to Roof-data provider: Attempt 1 via the breaker. The breaker is closed, so the call is sent. No answer within 2000 ms: a timeout. The breaker records one failure.
- Worker (the run) to Roof-data provider: Attempt 2 via the breaker. Sent again after the backoff. A second timeout; two failures in the window.
- Worker (the run) to Roof-data provider: Attempt 3 via the breaker. A third timeout. With 3 failures in the last 5 calls the breaker opens. The run has also used its 3 attempts.
- Breaker (provider): Open for 10000 ms: later calls refused at once. From now until the open time is over, any run that reaches step 2 is refused without a call. That is where the breaker saves time and load: in the runs that follow, not in this one.
- Worker (the run): Degrade: go on without the roof assessment. Step 2 ends with the roof marked unavailable and the run marked degraded. Steps 3 to 5 run normally; the stand-in model writes request_info.
- Worker (the run) to Store and underwriter: Store request_info, missing roof_assessment, degraded true. A smaller honest result: it names what is missing and does not approve or decline without it.
- Store and underwriter to Worker (the run): Run ends: degraded_request_info. A named terminal state, countable apart from normal runs. The model recommends; a person decides.
Run the scenario with the careful workflow, in a copy of examples/uw-reliability. A successful run proves that, in this simulation, the run stops trying after three bounded attempts, the breaker records that the provider is failing, and the run still ends in a named terminal state with an honest recommendation. It does not measure anything about a real provider.
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R03
The timeline and the end state, verbatim (the header and the seven invariant lines are left out; all seven hold):
t_ms worker step att event
0 queue - - message sent: msg-285e
0 worker-A - - message received: msg-285e, receive 1
0 worker-A - - run created: trace_id 344c86900ed5796442ac8654f744dc43
0 worker-A - - lease granted: fencing token 41
0 worker-A load_submission - step done: checkpoint saved
2000 worker-A get_roof_assessment 1 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
2000 worker-A get_roof_assessment 1 backoff: 52 ms
4052 worker-A get_roof_assessment 2 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
4052 worker-A get_roof_assessment 2 backoff: 908 ms
6960 worker-A get_roof_assessment - breaker open: dependency provider
6960 worker-A get_roof_assessment 3 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
6960 worker-A get_roof_assessment - degrade: go on without the roof assessment: request_info
6960 worker-A get_roof_assessment - step done: checkpoint saved
7200 worker-A search_guidance 1 attempt ok
7350 worker-A search_guidance - step done: checkpoint saved
7354 worker-A check_eligibility - step done: checkpoint saved
8754 worker-A draft_recommendation 1 attempt ok
8754 worker-A draft_recommendation - step done: checkpoint saved
8766 worker-A record_recommendation 1 attempt ok
8766 worker-A record_recommendation - step done: checkpoint saved
8778 worker-A open_review_task 1 attempt ok
8778 worker-A open_review_task - step done: checkpoint saved
8778 worker-A await_underwriter - run ended: degraded_request_info
8778 worker-A await_underwriter - step done: checkpoint saved
8778 worker-A - - acknowledged
end state:
run run-bef5 state degraded_request_info resume_count 0 ended at 8778 ms (deadline 120000 ms)
recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
REC-0001 pending_review request_info cites none evidence EV-RULE-0001
run run-bef5 route primary degraded true fencing_token 41 key run-bef5:record_recommendation
Be precise about what the breaker did here, because it is easy to overstate. In this one run the breaker opened at 6960 ms, on the third failure, which is also the moment the run ran out of attempts. It did not save this run a single call. Its value is for the runs that come next: for the following 10000 ms, any run that reaches step 2 is refused at once. From the code (this example runs one message per scenario, so no second run is shown): _through_breaker raises CircuitOpen, classify in errors.py sorts it as "give up", and get_roof_assessment catches it and degrades straight away. That run spends no 6960 ms and sends the provider nothing.
The breaker lab below shows the effect over many calls. With a dependency that fails on every call and the settings of this example, 40 calls one second apart (from breaker.py:simulate_breaker, run in a copy of the example): 6 calls are sent, 34 are refused before they reach the provider, and the breaker opens 4 times. The 6 sent calls are the 3 that tripped it and one failed probe after each open period.
Try it: forty calls through a breaker#
Set how often the dependency fails, how many failures in the last 5 calls open the breaker, and how long it stays open. Step through 40 calls, one per second.
Without JavaScript: the three states, as the lab uses them, with this example's defaults.
| State | Enters when | A call in this state | Leaves when |
|---|---|---|---|
| closed | start; or a probe succeeded | sent; the result joins the last 5 | 3 of the last 5 failed |
| open | threshold reached; or a probe failed | refused, never sent (counted as spared) | 10000 ms have passed |
| half-open | the open time is over | sent as the one probe | probe ok: closed; probe failed: open |
The number to watch in the lab is "refused that would have worked". Lower the failure share to a level where most calls succeed, and the breaker sometimes refuses calls that would have been fine. That is the price of failing fast: a breaker trades a few good calls for not drowning in bad ones. Thresholds decide the exchange rate.
Graceful degradation: a smaller honest result in place of none#
Once the provider is out of reach, the run has three choices. It can fail and leave the broker with nothing. It can guess: take the application's 12 years as the roof age and carry on. Or it can finish with what it knows and say plainly what it does not.
The second choice is the dangerous one. Rule R-ROOF-03 exists because the application and the provider sometimes disagree (in UW-004 by 13 years). Using the application's figure as if it were verified would let exactly the case the rule is for slip through. A wrong answer that looks complete is worse than a late one.
The careful workflow takes the third choice. It stores this recommendation (from out/UW-R03-careful.json):
{"recommendation": "request_info",
"reasons": ["The roof-data provider did not answer, so the roof age on the application (12 years) is not verified.",
"No recommendation to approve or decline is made without that check.",
"Next step: ask the provider again later or request an inspection report."],
"citations": [], "missing": ["roof_assessment"], "stated_confidence": 0.7,
"requires_human": true, "policy_version": "2026.2"}
The row around it carries degraded: true, and the run ends in its own terminal state, degraded_request_info, so a dashboard can count these runs apart from normal ones. This is graceful degradation: when a dependency is unavailable, give a smaller result that is honest about what is missing, instead of no result or a result that pretends. INV-7 checks the honesty part: a degraded result must say so.
The SRE book uses the same word for a different lever: reducing the work done per request, for example by searching a cached subset, when capacity is short. Both meanings share the core: less than the full answer, chosen on purpose, and still useful. The model recommends; a person decides, and here the person learns at a glance that the roof was not checked.
Common confusion. "Degraded" is not "partial success". A partial success hides which parts are missing. A degraded result names them (missing: ["roof_assessment"]) and changes what it recommends because of them: it asks for information instead of approving or declining.
Bulkheads: one sick dependency must not take every worker#
A ship's hull is divided by walls called bulkheads. If the hull is breached, only one compartment floods. Microsoft's description of the pattern takes the name from there and applies it to resources: give each dependency its own pool (of connections, threads or workers), so that when one dependency hangs, it exhausts only its own pool (Bulkhead pattern, Azure Architecture Center).
For the insurance agent, picture 20 worker slots, shared. The roof provider starts timing out. Every run reaches step 2 and holds its slot for up to 7 seconds. Soon all 20 slots wait on the provider, and a run that is resuming at step 6 (it already has its roof data from a checkpoint) cannot get a slot to do 12 ms of work. A bulkhead caps how many slots may be inside a call to one dependency at the same time, say 8 for the roof provider (an illustrative number). The ninth run that wants the provider is told "not now" at once; the other 12 slots keep serving everything else.
A bulkhead and a breaker work on different signals. The breaker reacts to failures: errors in a row. The bulkhead reacts to occupancy: too many calls in flight, which is often the first sign of a slow dependency, before any call has timed out. The example has the primitive, ConcurrencyLimit in limits.py (its docstring calls it "a bulkhead for one dependency"), but it is not wired into the workflow; Concurrency is its home.
Load shedding: decide what to refuse first#
Sometimes the trouble is not one dependency but the whole system: more work arrives than the workers, the model quota or the provider can handle. Something will not be done. The only question is whether the system chooses what, or the overload chooses for it.
If nobody chooses, every request gets slower together. Requests start to exceed their timeouts. The server keeps working on requests whose callers have already given up, which is pure waste, and those callers retry. Amazon's Builders' Library describes this collapse of useful throughput, and the answer: when a server nears overload, reject the excess early, so that the requests it accepts are served fast (Using load shedding to avoid overload). That is load shedding: refusing part of the incoming work on purpose, so the rest succeeds.
Refusing at random is better than collapsing, but refusing in order is better still. Both sources describe priority. The SRE book's handling-overload chapter gives requests a criticality and rejects lower ones first (chapter 21, Handling Overload); the Builders' Library article gives the example of serving calls that finish started work ahead of calls that start new work. For the insurance agent, a design (this site's proposal, not a measured policy) could rank work like this:
| Priority | Work | Why at this level |
|---|---|---|
| 1, never shed | Resumed runs at steps 6 to 8, and compensations | The run has already paid for the model; a half-finished run is worse than an unstarted one |
| 2 | Redeliveries of runs in progress | Each one finishes something that started |
| 3 | New runs for submissions near a broker's deadline | The insurer has a promise to keep |
| 4, shed first | New runs with no urgency; re-runs a person asked for "when there is time" | Waiting on the queue costs them least |
Shedding a new run here does not mean losing it. The worker does not take the message, it stays on the queue, and it is picked up when there is room. What must never happen is shedding in the middle of a write without a key: that is where a refusal becomes a duplicate.
Backpressure: let the slow part slow down the fast part#
The queue in front of the workers absorbs bursts. That is its job. But an unbounded buffer hides overload instead of handling it: the queue grows, every message waits longer, and by the time a run starts, the broker's request may be hours old.
Backpressure is the signal that flows backwards from a slow stage to the stage feeding it, saying "send less". In this workflow it can be as plain as this: a worker takes a new message only when it has a free slot and the breakers it needs are not open. The queue then grows visibly, which is the honest place for waiting work to sit, and the queue's age becomes the alarm. Both sources make the same two points about queues: keep them bounded, and drop or refuse work that has waited too long, because its caller has likely given up (SRE chapter 22, "Queue management"; the Builders' Library article on size and age limits).
Before AI. A Spring service calling a slow downstream API with a shared thread pool shows the same chain: the pool fills with threads blocked on one host, and unrelated endpoints start timing out. The classic answers are the same three: a breaker in front of the client, a separate pool per downstream (a bulkhead), and a bounded executor queue that rejects when full. Microsoft's bulkhead page names Resilience4j as one Java library for building consumer bulkheads; what it provides, from its own documentation, is on Frameworks. What is different for an agent: one run holds a slot for seconds or minutes, not milliseconds, so slots run out far sooner; the most expensive dependency, the model, is usually rate-limited by quota rather than by failure; and the degraded answer must be marked as such, because a person acts on it.
How these mechanisms go wrong#
| Mistake | What happens |
|---|---|
| A breaker with no half-open state | It never closes by itself, and the workflow degrades for ever after the dependency recovered |
| Closing on a timer with no probe | Recovery sends the full queued load at once to a dependency that has barely come back |
| Counting 429s or validation errors as failures | The breaker opens on a healthy, busy dependency, or on the model's formatting, which a breaker cannot fix |
| A threshold that is too sensitive | The breaker flaps open and closed; runs degrade that did not need to |
| One breaker per process, many processes | Each process must learn about the failure on its own; the dependency sees many first failures (the example shares one breaker across the workers of a story, and says so in its README) |
| Degradation that does not say so | A request_info without missing, or a fallback answer without degraded: true: the person cannot tell a weak answer from a strong one (INV-7) |
| Shedding writes that are in progress | A refused acknowledgement leads to a retry without a key: a duplicate |
| An unbounded queue as the only buffer | Overload turns into hours of delay, and every run starts late |
The same idea for a coding agent#
A coding agent pushes a change and waits for CI. The CI service is down. A naive agent re-triggers the pipeline every few seconds, each trigger queuing a job on a system that is already struggling, and after twenty minutes it reports a timeout, or worse, reports that it "could not confirm the tests failed". With a breaker on the CI dependency, the agent stops triggering after a few failures, probes once in a while, and degrades honestly: it opens the pull request marked "tests not run: CI unavailable" and does not claim that anything passed. The runtime that holds such a breaker is the subject of https://harness.sammuthu.com; the path from change to production, of https://diff-to-deploy.sammuthu.com.
What this is not. Seeing that a dependency is failing (the error-rate graph, the trace that shows three timeouts, the alert) is observability: https://observe.sammuthu.com. This page owns what the workflow does once the failure is seen. Switching the model to a second route when the first one is down is Fallback, which uses the breaker taught here.
The four blocks below hold the detailed layer.
Choosing the window, the threshold and the open timeCount-based or rate-based, how many probes, and what each setting trades
The example uses a count-based window: the last 5 calls, open at 3 failures. That reacts fast at low traffic and is easy to test, but at high traffic five calls are a tiny sample, and three unlucky timeouts in a row can open it. A rate-based window (for example, the share of failures over the last 30 seconds, with a minimum number of calls before it may open) is steadier under load. Fowler's article mentions both a failure count and a failure rate as triggers, and different thresholds for different kinds of error.
The open time trades how long a recovered dependency goes unused against how often a still-broken one is probed. A fixed 10000 ms is fine for a provider that recovers in minutes. For long outages, growing the open time after each failed probe (with jitter, so that many processes do not probe together) reduces probe traffic. More than one probe in half-open gives more confidence before closing; one probe, as here, closes faster.
Thresholds are settings, not facts. Choose them from the dependency's normal error rate and latency (from your observability data), and test them by injecting the failure (Fault injection).
Where the breaker sits relative to retriesBreaker inside the retry loop, or retry loop inside the breaker
In the example the breaker wraps each attempt: Workflow._call puts _through_breaker inside call_with_retry. Each attempt is recorded by the breaker, and when the breaker opens, the next attempt is refused at once and the retry loop gives up (an open breaker is not retryable). This is the arrangement that lets the breaker see every failed call.
The other arrangement (the whole retry sequence as one call through the breaker) records one failure per exhausted sequence. The breaker then opens later, after several runs have each spent their full attempts. It also hides from the breaker the case where the third attempt succeeded, which is information about a degrading dependency.
A second question is scope. One breaker per dependency (the roof provider) is simple. One per dependency and region, or per host behind a load balancer, avoids opening against all hosts because one is broken, at the cost of more state. For the model, a breaker per route is natural, and it is what the example uses (model.primary, model.fallback), which is how Fallback knows when to switch.
Adaptive client-side throttling as an alternativeRefuse a share of calls locally, in proportion to how many the backend is rejecting
A breaker is binary: send everything or send almost nothing. The SRE book's handling-overload chapter describes a smoother client-side scheme: each client tracks, over a recent window, how many requests it sent and how many the backend accepted, and starts rejecting a share of new requests locally once requests exceed a multiple of accepts. Requests above that cap fail without reaching the network. The rejection share rises and falls with the backend's health, so there is no flapping and no separate half-open step.
It fits a dependency that is overloaded and still answering most calls. A breaker fits a dependency that is down. Many systems use both: throttling for the gray zone, a breaker for the outage. The SRE chapter also notes that it is a mistake to expect an overloaded backend to stop accepting all traffic; it should keep serving what it can and reject the rest cleanly.
Designing the degraded answerWhat may be dropped, what must not, and who decides
Not every dependency can be degraded around. Step 3 (search_guidance) has no degraded mode in this workflow: without the policy fixture's passages there is nothing to ground a recommendation in, so the message goes back to the queue instead (and may end in the dead-letter queue). Step 2 can be degraded because there is an honest smaller answer: ask for the missing information. Decide this per dependency, in writing, with the people who own the outcome, here the underwriters.
Three rules keep a degraded answer honest. It must name what is missing, in a field a machine can read (missing). It must not reach a conclusion that the missing input could overturn (no approve or decline here). And it must be countable apart from normal results (its own terminal state, degraded: true), so a rise in degraded runs shows up as what it is: an outage, measured in outcomes.
Checklist#
| Question | Good answer |
|---|---|
| Does every outside dependency the workflow calls have a breaker? | Yes, one per dependency (or per route, for the model) |
| Does the breaker count only failures that mean "not answering"? | Timeouts and unavailability, not 429s or validation errors |
| Does it recover by itself through a half-open probe? | Yes, and state changes are logged |
| Is the number of calls in flight to each dependency capped? | Yes, a bulkhead per dependency |
| Is there a written order of what to refuse first under overload? | Yes; work that finishes a run comes before work that starts one |
| Is the queue bounded, and is its oldest message's age watched? | Yes |
| For each dependency, is there a degraded answer, or a written reason why not? | Yes, and the degraded answer names what is missing |
| Can a dashboard count degraded runs apart from normal ones? | Yes: a terminal state and a flag |
| Was each of these tested by injecting the failure? | Yes |
In one breath#
Retries help one failed call, but against a dependency that is down for everyone they become the load that keeps it down. A circuit breaker remembers across runs that the dependency is failing, refuses calls at once while it is open, and lets one probe through to find out when it is back. Bulkheads keep one slow dependency from taking every worker, load shedding refuses the least important work first, and a degraded request_info that names what is missing is better than no answer or a guessed one.
Next: many runs, one quota#
The insurance agent now protects itself from a dependency that fails. The next danger comes from success: many healthy runs at once, all sharing one model quota, one budget of cost units and sometimes one submission. A provider that answers 429 is not broken, and a breaker is the wrong tool for it; two workers that both believe they own SUB-1004 are not a dependency failure at all. Concurrency is about rate limits, budgets and the lease with a fencing token that settles who may write.