It is a Tuesday afternoon and nothing was deployed. The underwriting agent is working through quote requests as usual: it validates the submission, gets the roof assessment, retrieves the guidance and runs the rules. Then it asks the model to draft the recommendation, and the model provider's primary route answers 503, service unavailable. Every call, for every run. The model recommends; a person decides, but right now there is nothing to decide on.
This is scenario UW-R13, on submission SUB-1004: the application says the roof is 9 years old, the roof-data provider says 22, so the expected recommendation is escalate, citing R-ROOF-03. Here is what the naive workflow does with a route that stays down, from the example's run (folded by the runner):
1574 worker-A draft_recommendation 1 attempt failed: Unavailable: model route primary: unavailable (503)
2574 worker-A draft_recommendation 2 attempt failed: Unavailable: model route primary: unavailable (503)
... 117 more times: attempt failed ...
120574 worker-A draft_recommendation 120 attempt failed: Unavailable: model route primary: unavailable (503)
120574 worker-A draft_recommendation - DEADLINE EXCEEDED: draft_recommendation: the deadline passed after 120 attempt(s)
120574 worker-A draft_recommendation - run ended: deadline_exceeded
One hundred and twenty attempts, two minutes of a worker's time, and no recommendation. INV-4 (a named terminal state inside the deadline) and INV-5 (no step over its maximum attempts) are broken. Breakers already fixed half of this: the careful workflow stops after three attempts and the breaker stops further calls. But stopping only gets you an honest failure. The broker is still waiting for a quote, and the steps before the model all succeeded.
The model is the one dependency in this workflow that has a natural substitute. A second route to a model can usually be bought or configured. This page is about using it well: when to switch, what the answer loses on the way, and how to make sure the underwriter can tell.
Plain English. If the first model cannot answer, ask a second one. But the second one is not the first one. Its answer may be shorter, less sure or different, so write on the result where it came from, and never let the switch hide a worse answer.
Fallback and failover: two different second routes#
The words are close, and the difference matters for quality.
Provider failover sends the same request to another copy of the same thing: the same model, served from another region or another endpoint of the provider. Tiny example: the primary region of the model provider is down, so the call goes to a second region that serves the identical model id. If the copies really are identical, the answer is of the same kind.
Model fallback sends the request to a different model: another size, another version, another provider. Tiny example: the large model is unavailable, so a smaller, faster model drafts the recommendation instead. The run continues, but with a different mind behind the answer.
Jacob Gabrielson's Builders' Library article draws the same line in general terms: fallback uses a different mechanism to reach the same result and runs only when the primary fails, while failover sends the same work to another copy of an endpoint (Avoiding fallback in distributed systems). For a deterministic service the difference is about how well the second path is tested. For a model it is also about what the answer says.
A model route is one configured way to reach a model: a provider, an endpoint or region, a model id, and the settings that go with it (prompt version, timeout, breaker). The example has two routes, primary and fallback, each with its own breaker (model.primary, model.fallback).
What this is not. Choosing which agent or which step handles a request, routing between agents and the control plane that decides what runs next, belongs to https://orchestrate.sammuthu.com. A model route here is narrower: the same step, the same prompt, a second way to get the answer when the first way is down.
The route order and what triggers a switch#
A fallback needs three decisions written down before the outage: the order of the routes, what makes the workflow leave a route, and what does not.
The example makes all three in one function, router.py:route_call, quoted verbatim:
def route_call(routes: tuple[str, ...], call: Callable[[str], Any],
on_switch: Callable[[str, Exception], None] | None = None) -> tuple[Any, str]:
"""Try each route in order. Returns (result, route used).
A route is left when its attempts are exhausted or its breaker is open.
Malformed output is NOT a reason to switch: the workflow handles it (ask
again, then escalate), so that error is passed up with the route on it.
"""
last: Exception | None = None
for route in routes:
try:
return call(route), route
except (RetriesExhausted, CircuitOpen) as error:
if isinstance(getattr(error, "last", None), MalformedOutput):
error.route = route # type: ignore[attr-defined]
raise
last = error
if on_switch and route != routes[-1]:
on_switch(route, error)
assert last is not None
raise last
Read the triggers off the code. A route is left when its attempts are exhausted (each route call goes through the same retry with backoff that Retries teaches, so a single 503 does not cause a switch) or when its breaker is open (the route has been failing across runs). A 429 with retry_after_ms is not a switch either: the retry waits as told, and the breaker counts a 429 as a sign that the route is alive. And malformed output is explicitly not a reason to switch.
That last rule deserves its own sentence. A route that answers with text that fails validation is up. Switching would hide a problem with the prompt or the output contract behind a different model that might fail the same way, or pass by luck. The workflow handles malformed output on its own path: ask again, at most twice, then a code-written escalate with the reason "output could not be validated" (Failure modes and https://schema.sammuthu.com cover the validation itself).
| Event on the primary route | Switch to fallback? | Why |
|---|---|---|
| One timeout or 503 | No, retry first | Transient errors usually pass; the retry is cheaper than a different model |
| Attempts exhausted (3 here) | Yes | The route did not answer within its share of the deadline |
| Breaker open | Yes, at once | Other runs have already shown the route is down |
429 with retry_after_ms | No | The route is alive and told you when to come back |
| Malformed output | No | The route answered; the problem is the answer, handled by re-ask and escalate |
| Cost ceiling reached | No | A second route spends more; the run ends with a code-written escalate |
UW-R13 with a fallback route#
Now the same scenario with the careful workflow. The command runs in a scratch copy of examples/uw-reliability. A successful run proves that, in this simulation, the run leaves the primary route after its attempts are used, the fallback route drafts the recommendation, and the stored row says so. It says nothing about any real model or provider: both routes are scripted.
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R13
The timeline from step 5 and the end state, verbatim (all seven invariants hold):
1574 worker-A draft_recommendation 1 attempt failed: Unavailable: model route primary: unavailable (503)
1574 worker-A draft_recommendation 1 backoff: 52 ms
2626 worker-A draft_recommendation 2 attempt failed: Unavailable: model route primary: unavailable (503)
2626 worker-A draft_recommendation 2 backoff: 908 ms
4534 worker-A draft_recommendation - breaker open: dependency model.primary
4534 worker-A draft_recommendation 3 attempt failed: Unavailable: model route primary: unavailable (503)
4534 worker-A draft_recommendation - fallback route: route primary gave up; the result will be marked degraded
5434 worker-A draft_recommendation 1 attempt ok
5434 worker-A draft_recommendation - step done: checkpoint saved
5446 worker-A record_recommendation 1 attempt ok
5446 worker-A record_recommendation - step done: checkpoint saved
5458 worker-A open_review_task 1 attempt ok
5458 worker-A open_review_task - step done: checkpoint saved
5458 worker-A await_underwriter - run ended: waiting_for_human
5458 worker-A await_underwriter - step done: checkpoint saved
5458 worker-A - - acknowledged
end state:
run run-06f3 state waiting_for_human resume_count 0 ended at 5458 ms (deadline 120000 ms)
recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
REC-0001 pending_review escalate cites R-ROOF-03 evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
run run-06f3 route fallback degraded true fencing_token 41 key run-06f3:record_recommendation
The run ends in 5458 ms of simulated time with one recommendation waiting for the underwriter, where the naive run ended after 120574 ms with none. The attempt counter starts again at 1 on the fallback route, because each route has its own attempts. And the last line carries the two fields this page is about: route fallback and degraded true.
The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
Diagram: UW-R13 · step 5 leaves the primary route, the fallback route drafts, and the row says so
Read left to right. Step 5 asks the primary route three times, with backoff; the third failure also opens its breaker. The router then tries the fallback route, which answers in 900 ms. The recommendation is the same, escalate citing R-ROOF-03, but shorter and less sure, and the stored row carries model_route fallback and degraded true. Malformed output would not have caused the switch. Times are simulated; the agent and both routes are scripted stand-ins.
- Primary route
- Primary: 3 attempts: Each attempt fails after 1000 ms; backoff of 52 ms and 908 ms between them.
- Breaker opens: Three failures in the last five calls open the primary route's breaker, so later runs skip it at once.
- Router decides
- Switch route: route_call leaves a route only when its attempts are exhausted or its breaker is open; malformed output and 429 never cause a switch.
- Fallback route
- Fallback drafts: A more cautious stand-in: never writes approve, two reasons instead of three, stated confidence 0.2 lower.
- Stored and marked
- Row marked degraded: The recommendation row carries model_route fallback and degraded true, so the underwriter can tell. INV-7 checks the mark.
What the second route changes: quality impact#
The fallback answer in UW-R13 has the right recommendation: escalate, citing R-ROOF-03. It would be easy to stop there and call the fallback free. Compare the two stored recommendations field by field (the primary one from UW-R01, the fallback one from UW-R13, both from the example's output files):
| Field | Primary route | Fallback route |
|---|---|---|
recommendation | escalate | escalate |
citations | R-ROOF-03 | R-ROOF-03 |
| Reasons | 3: the two roof ages, the rule sentence quoted from the policy fixture, "escalate to a senior underwriter to settle the roof age" | 2: the two roof ages, "the underwriter reviews this recommendation and decides" |
stated_confidence | 0.8 | 0.6 |
model_route, degraded | primary, false | fallback, true |
| Latency, cost | 1400 ms, 1340 cost units | 900 ms, 620 cost units |
The underwriter reading the fallback version gets the same conclusion with less support: no quoted rule, a generic next step, lower stated confidence. That difference is the quality impact of the fallback: what changes in the result, not in the latency, when the second route answers.
The example scripts exactly four differences between the routes. Three sit in agent_sim.py:draft; the fourth, 900 ms and 620 cost units, sits in the route settings. One of the three is a deliberate safety rule. The branch for the fallback route, dedented, otherwise verbatim:
if route == "fallback":
closing = NEXT_STEP
if output["recommendation"] == "approve":
output["recommendation"] = "escalate"
closing = "The fallback route does not write approve. Next step: a senior underwriter reviews this submission."
output["reasons"] = [output["reasons"][0], closing]
output["stated_confidence"] = round(output["stated_confidence"] - 0.2, 2)
The fallback route never writes approve. For a clean submission such as UW-002, where the primary route would recommend approval, the fallback writes escalate with the sentence "The fallback route does not write approve." That is the example's version of a principle worth keeping in any design: when the second route is less trusted, let it say less. It may still recommend the cautious outcomes; it may not recommend the one that would most easily be waved through.
A real fallback model differs in ways nobody scripted. It may cite a different rule, miss the conflict between the two roof ages, or read the policy fixture differently. Those are quality regressions, and measuring them before the outage, on a golden set, is the work of evaluation.
What this is not. Measuring how much worse the fallback model's recommendations are, with graders and golden cases, is eval: https://eval.sammuthu.com. This page owns the mechanism: when the route switches, how the result is marked, and what the run does when no route answers.
Marking a degraded result#
A degraded result is a smaller or less trusted result, given because a dependency was unavailable, and labelled as such. Breakers introduced the idea with request_info when the roof provider stays down. The fallback route produces the second kind in this example: a full recommendation, from a less trusted route.
The mark is two fields on the stored row, model_route: "fallback" and degraded: true, set in Workflow.draft_recommendation and written with the recommendation. INV-7 checks that every degraded or fallback result says so. The underwriter cockpit can then show it, sort it, or route it to a more senior reviewer.
Why does a mark matter if the recommendation is the same? Because nobody can know that in general. The fallback lab makes the arithmetic visible. Out of 1000 runs, with the primary route failing 10% of the time and the fallback answer differing from what the primary would have written in 30% of those runs (both synthetic settings), 98 runs go to the fallback and 28 of them get a different answer. With the mark, all 28 carry degraded: true. Without it, the same 28 look exactly like the 902 primary answers. That is the worst outcome on this page: answered, quietly different, and trusted as if nothing happened.
Try it: how much a fallback buys, and what it hides#
Set how often the primary route fails and how often the fallback answer differs, and switch the fallback and the mark on and off.
Without JavaScript, the lab's default (primary fails 10%, fallback differs 30%, 1000 runs, seed 7), from router.py:simulate_fallback run in a copy of the example:
| Setting | Primary answered | Fallback answered | Different and marked | Different and silent | No draft |
|---|---|---|---|---|---|
| Fallback on, mark on | 902 | 98 | 28 | 0 | 0 |
| Fallback on, mark off | 902 | 98 | 0 | 28 | 0 |
| Fallback off | 902 | 0 | 0 | 0 | 98 |
The third row is not a disaster in a careful workflow. Those 98 runs still have to end in an honest terminal state, as the next section shows. The second row is the one to avoid.
When not to fall back#
A fallback trades a late answer for a different one. For a quote that a person will review, a late answer is often the better side of that trade. Four cases call for no switch, or a narrower one.
When the answer would be wrong in a way that is hard to see. A recommendation that is plausible and wrong costs more than one that arrives an hour later, because the underwriter may act on it. If the fallback model has not been evaluated on cases like this one, a degraded request_info or a wait is more honest than its draft. The example encodes a small piece of this: no approve from the fallback.
When the fallback is rarely used. Gabrielson's article explains why fallback paths are dangerous in general: they are hard to test, they may not run for months, they can carry latent bugs that appear during the outage, and they can make the outage worse, for example when every client falls back to the same smaller dependency at once. Its preferred cure is to run both paths all the time, so the second one is exercised constantly and the fallback becomes a failover. For a model, that means sending a steady small share of real runs to the fallback route even on a good day, and comparing.
When the trigger is not "down". Malformed output, a 429 that names its wait, a cost ceiling reached: each has its own answer on this site, and a switch would hide it.
When both routes are down. A second route adds an option, not a guarantee. To see what the example does, both routes were made unavailable in a variant of UW-R13 built for this page in a scratch copy (it is not one of the fifteen scenarios). The primary route gives up at 4534 ms and the fallback route at 8411 ms. The message goes back to the queue, and four more workers resume the run from its checkpoint, find both breakers still open, and give it back at once. After the fifth receive the message moves to the dead-letter queue at 13411 ms: 0 recommendation rows, run state dead_lettered, all seven invariants holding. Honest, but harsh: the redelivery interval of 1000 ms used up five receives inside the breakers' 10000 ms open time. A design for a long model outage should hold the run (or end it as a degraded request_info) instead of spending its receives against open breakers.
Common confusion. "With two routes, the model step is twice as available." Only if the two fail independently. Two routes that share a provider, a region, a network path or a quota can fail together, and so can two models behind the same gateway. Ask what the routes have in common before counting on the second one.
Hedged requests: asking twice on purpose#
A fallback waits for the primary to fail. A hedged request does not wait for a failure: it sends a second copy of a request when the first one is taking longer than usual, uses whichever answer arrives first, and cancels the other. Jeffrey Dean and Luiz André Barroso described the technique in "The Tail at Scale" (Communications of the ACM, 2013): defer the second request until the first has been outstanding longer than the 95th-percentile expected latency for that kind of request, which in their description limits the extra load to about 5% (PDF hosted by the second author). It is a weapon against tail latency, not against outages.
For an agent, hedging has two costs a search backend does not have. A model call spends cost units, so every hedge is paid for, and two copies of a model call can return two different answers; the run must take one and record which. And a hedge is only safe for a step with no side effect. Hedging the draft is possible. Hedging record_recommendation buys nothing and adds a race: two copies of the same write arrive together, and only the idempotency key stands between them and a duplicate row.
| Hedge this step? | Answer |
|---|---|
get_roof_assessment (a read) | Possible, if the provider's tail is long and its load allows it |
draft_recommendation (the model) | Possible, at a cost in cost units; record which answer was used |
record_recommendation, open_review_task (writes) | No |
The example does not implement hedging.
How fallback goes wrong#
| Mistake | What happens | On this site |
|---|---|---|
| No fallback and no bounded retry | The run retries until the deadline and ends with nothing | Naive UW-R13: 120 attempts, INV-4, INV-5 |
| Fallback without the mark | Different answers look like primary answers | Lab: 28 silent of 1000 |
| Switching on malformed output | A prompt or contract problem is hidden behind another model | route_call refuses this |
| Switching on one transient error | The weaker route answers runs the primary would have handled | Retry first |
| Fallback never exercised | It fails, or overloads, on the day it is needed | Gabrielson's article |
| Two routes with a shared failure | Both go down together | Common confusion above |
| Fallback allowed to write the riskiest outcome | A less trusted route writes approve | The example forbids it |
| Hedging a write | Two writes race | Hedge reads and drafts only |
Before AI. A Java service with a primary and a replica database, or two payment gateways, has the same pattern: try the primary, fail over after retries or an open breaker, and alarm when the secondary is in use. The new part for an agent is that the secondary may be correct in form and different in content. A database replica returns the same rows; a second model returns its own answer to the same prompt. That is why the marking and the quality comparison carry weight here that they do not carry for a replica. Model gateways and SDKs that offer route lists and retries are described, each from its own documentation, on Frameworks.
The same idea for a coding agent#
A coding agent's primary model is unavailable in the middle of a task. Falling back to a smaller model to finish the change is possible, and sometimes fine for a formatting fix. For a change to authentication code it is the case where a late pull request beats a wrong one. A good harness records the route on the pull request ("drafted by the fallback model"), so the reviewer knows to read more carefully, and it lets the fallback open a draft pull request but not merge. The runtime that holds the routes is https://harness.sammuthu.com; the review lanes a change passes through are https://diff-to-deploy.sammuthu.com.
The blocks below hold the detailed layer.
One prompt, two models: the prompt is part of the routeA prompt tuned for one model is not automatically right for another
The example sends the same request to both routes. In practice a prompt is often tuned for one model: its length, its examples, the output format it follows reliably. Sent unchanged to another model, it may produce malformed output more often, which the workflow then re-asks and escalates. Treat the prompt version as part of the route's configuration, evaluate each pair (model, prompt) on its own, and put both in the release record that Deployment describes, so that a change to the fallback prompt gets a canary like any other change.
The output contract must not differ. Both routes are validated by the same validate_output, and the stored row has the same shape. Only the content and the mark may differ.
Watching the fallback as its own SLIThe share of runs on the fallback route is an early signal, and a budget
Count the share of runs answered by the fallback route. On a normal day it should be small and steady; a rise means the primary route is struggling before any run fails. Gabrielson's article gives the matching advice for retries: monitor their rate and alarm on unusual increases, so a retry or a fallback does not quietly become the normal path.
The share can also carry an objective of its own on SLOs: for example, at most a small percentage of a week's recommendations may be degraded (an illustrative target). When it is exceeded, the team knows that underwriters have spent the week reading weaker recommendations, even if every run ended well.
Fallback and resume: which route finishes the runThe route is part of the checkpoint, so a resume does not switch again
The checkpoint after step 5 holds state.recommendation, state.model_route and state.degraded. A run that crashed after a fallback draft resumes with the fallback answer and its mark; it does not ask the primary route again because the primary is back. Asking again would replace an answer the run already had with a different one under the same idempotency key, the replay problem of Checkpoints.
The opposite case is a run that resumes before step 5 while the primary is back. It uses whichever route is up at that moment, and the mark follows the route that actually drafted. The rule is the same as for versions: what the run used is recorded once and never quietly changed.
Checklist#
| Question | Good answer |
|---|---|
| Is the route order written down, with each route's own retries and breaker? | Yes |
| Does a route switch only after its attempts are exhausted or its breaker is open? | Yes, not on one error |
| Do malformed output, 429 and the cost ceiling keep their own handling? | Yes, none of them switches routes |
Does every fallback result carry model_route and degraded: true, checked by a test? | Yes (INV-7) |
| Is the fallback forbidden from the riskiest outcome where that matters? | Yes, for example no approve |
| Has the fallback model, with its prompt, been evaluated on your cases? | Yes, before the outage |
| Is the fallback route exercised on a normal day? | Yes, a steady small share |
| Do you know what the routes have in common? | Yes: provider, region, gateway, quota |
| Does a run end honestly when every route is down? | Yes, in a named terminal state |
| Are writes never hedged? | Yes |
In one breath#
When the primary model route is down, a fallback route can keep the run alive, but it is a different model, so its answer may be shorter, less sure or different. Switch only when the primary's attempts are exhausted or its breaker is open, never on malformed output, and mark every fallback result with model_route: "fallback" and degraded: true so the underwriter can tell. A wrong answer is worse than a late one: keep the fallback away from the riskiest outcomes, exercise it on normal days, and let a run end honestly when no route answers.
Next: proving the recovery paths work#
Every page so far showed one mechanism on one scenario: a retry, a key, a checkpoint, a breaker, a canary, a fallback. Each is a recovery path, and a recovery path that has never run under the faults it was built for is a guess. Fault injection runs all of them together, on purpose, against invariants that must hold after every fault, so the careful workflow's 295 passing runs are a result and not a hope.