All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Operations · 11

Model fallback and provider failover: what a second route costs in quality

When the first model or provider cannot answer, a second route keeps the run alive. Routing rules, what changes in the answer, and how to say that a result came from the second choice.

What you will understand
A second route
How a run switches from the primary model route to a fallback when the first one is down, and what decides the switch.
What it costs in quality
Why the second route's answer is not the same answer, and how the stored recommendation says it came from the fallback.
When not to switch
Malformed output, hedged requests, and the cases where a late answer beats a wrong one.

It is a Tuesday afternoon and nothing was deployed. The underwriting agent is working through quote requests as usual: it validates the submission, gets the roof assessment, retrieves the guidance and runs the rules. Then it asks the model to draft the recommendation, and the model provider's primary route answers 503, service unavailable. Every call, for every run. The model recommends; a person decides, but right now there is nothing to decide on.

This is scenario UW-R13, on submission SUB-1004: the application says the roof is 9 years old, the roof-data provider says 22, so the expected recommendation is escalate, citing R-ROOF-03. Here is what the naive workflow does with a route that stays down, from the example's run (folded by the runner):

   1574  worker-A  draft_recommendation     1  attempt failed: Unavailable: model route primary: unavailable (503)
   2574  worker-A  draft_recommendation     2  attempt failed: Unavailable: model route primary: unavailable (503)
                                               ... 117 more times: attempt failed ...
 120574  worker-A  draft_recommendation   120  attempt failed: Unavailable: model route primary: unavailable (503)
 120574  worker-A  draft_recommendation     -  DEADLINE EXCEEDED: draft_recommendation: the deadline passed after 120 attempt(s)
 120574  worker-A  draft_recommendation     -  run ended: deadline_exceeded

One hundred and twenty attempts, two minutes of a worker's time, and no recommendation. INV-4 (a named terminal state inside the deadline) and INV-5 (no step over its maximum attempts) are broken. Breakers already fixed half of this: the careful workflow stops after three attempts and the breaker stops further calls. But stopping only gets you an honest failure. The broker is still waiting for a quote, and the steps before the model all succeeded.

The model is the one dependency in this workflow that has a natural substitute. A second route to a model can usually be bought or configured. This page is about using it well: when to switch, what the answer loses on the way, and how to make sure the underwriter can tell.

Plain English. If the first model cannot answer, ask a second one. But the second one is not the first one. Its answer may be shorter, less sure or different, so write on the result where it came from, and never let the switch hide a worse answer.

Fallback and failover: two different second routes#

The words are close, and the difference matters for quality.

Provider failover sends the same request to another copy of the same thing: the same model, served from another region or another endpoint of the provider. Tiny example: the primary region of the model provider is down, so the call goes to a second region that serves the identical model id. If the copies really are identical, the answer is of the same kind.

Model fallback sends the request to a different model: another size, another version, another provider. Tiny example: the large model is unavailable, so a smaller, faster model drafts the recommendation instead. The run continues, but with a different mind behind the answer.

Jacob Gabrielson's Builders' Library article draws the same line in general terms: fallback uses a different mechanism to reach the same result and runs only when the primary fails, while failover sends the same work to another copy of an endpoint (Avoiding fallback in distributed systems). For a deterministic service the difference is about how well the second path is tested. For a model it is also about what the answer says.

A model route is one configured way to reach a model: a provider, an endpoint or region, a model id, and the settings that go with it (prompt version, timeout, breaker). The example has two routes, primary and fallback, each with its own breaker (model.primary, model.fallback).

What this is not. Choosing which agent or which step handles a request, routing between agents and the control plane that decides what runs next, belongs to https://orchestrate.sammuthu.com. A model route here is narrower: the same step, the same prompt, a second way to get the answer when the first way is down.

The route order and what triggers a switch#

A fallback needs three decisions written down before the outage: the order of the routes, what makes the workflow leave a route, and what does not.

The example makes all three in one function, router.py:route_call, quoted verbatim:

def route_call(routes: tuple[str, ...], call: Callable[[str], Any],
               on_switch: Callable[[str, Exception], None] | None = None) -> tuple[Any, str]:
    """Try each route in order. Returns (result, route used).

    A route is left when its attempts are exhausted or its breaker is open.
    Malformed output is NOT a reason to switch: the workflow handles it (ask
    again, then escalate), so that error is passed up with the route on it.
    """
    last: Exception | None = None
    for route in routes:
        try:
            return call(route), route
        except (RetriesExhausted, CircuitOpen) as error:
            if isinstance(getattr(error, "last", None), MalformedOutput):
                error.route = route                               # type: ignore[attr-defined]
                raise
            last = error
            if on_switch and route != routes[-1]:
                on_switch(route, error)
    assert last is not None
    raise last

Read the triggers off the code. A route is left when its attempts are exhausted (each route call goes through the same retry with backoff that Retries teaches, so a single 503 does not cause a switch) or when its breaker is open (the route has been failing across runs). A 429 with retry_after_ms is not a switch either: the retry waits as told, and the breaker counts a 429 as a sign that the route is alive. And malformed output is explicitly not a reason to switch.

That last rule deserves its own sentence. A route that answers with text that fails validation is up. Switching would hide a problem with the prompt or the output contract behind a different model that might fail the same way, or pass by luck. The workflow handles malformed output on its own path: ask again, at most twice, then a code-written escalate with the reason "output could not be validated" (Failure modes and https://schema.sammuthu.com cover the validation itself).

Event on the primary routeSwitch to fallback?Why
One timeout or 503No, retry firstTransient errors usually pass; the retry is cheaper than a different model
Attempts exhausted (3 here)YesThe route did not answer within its share of the deadline
Breaker openYes, at onceOther runs have already shown the route is down
429 with retry_after_msNoThe route is alive and told you when to come back
Malformed outputNoThe route answered; the problem is the answer, handled by re-ask and escalate
Cost ceiling reachedNoA second route spends more; the run ends with a code-written escalate

UW-R13 with a fallback route#

Now the same scenario with the careful workflow. The command runs in a scratch copy of examples/uw-reliability. A successful run proves that, in this simulation, the run leaves the primary route after its attempts are used, the fallback route drafts the recommendation, and the stored row says so. It says nothing about any real model or provider: both routes are scripted.

/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R13

The timeline from step 5 and the end state, verbatim (all seven invariants hold):

   1574  worker-A  draft_recommendation     1  attempt failed: Unavailable: model route primary: unavailable (503)
   1574  worker-A  draft_recommendation     1  backoff: 52 ms
   2626  worker-A  draft_recommendation     2  attempt failed: Unavailable: model route primary: unavailable (503)
   2626  worker-A  draft_recommendation     2  backoff: 908 ms
   4534  worker-A  draft_recommendation     -  breaker open: dependency model.primary
   4534  worker-A  draft_recommendation     3  attempt failed: Unavailable: model route primary: unavailable (503)
   4534  worker-A  draft_recommendation     -  fallback route: route primary gave up; the result will be marked degraded
   5434  worker-A  draft_recommendation     1  attempt ok
   5434  worker-A  draft_recommendation     -  step done: checkpoint saved
   5446  worker-A  record_recommendation    1  attempt ok
   5446  worker-A  record_recommendation    -  step done: checkpoint saved
   5458  worker-A  open_review_task         1  attempt ok
   5458  worker-A  open_review_task         -  step done: checkpoint saved
   5458  worker-A  await_underwriter        -  run ended: waiting_for_human
   5458  worker-A  await_underwriter        -  step done: checkpoint saved
   5458  worker-A  -                        -  acknowledged
end state:
  run run-06f3  state waiting_for_human  resume_count 0  ended at 5458 ms (deadline 120000 ms)
  recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
  REC-0001  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-06f3  route fallback  degraded true  fencing_token 41  key run-06f3:record_recommendation

The run ends in 5458 ms of simulated time with one recommendation waiting for the underwriter, where the naive run ended after 120574 ms with none. The attempt counter starts again at 1 on the fallback route, because each route has its own attempts. And the last line carries the two fields this page is about: route fallback and degraded true.

The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

Diagram: UW-R13 · step 5 leaves the primary route, the fallback route drafts, and the row says so

Read left to right. Step 5 asks the primary route three times, with backoff; the third failure also opens its breaker. The router then tries the fallback route, which answers in 900 ms. The recommendation is the same, escalate citing R-ROOF-03, but shorter and less sure, and the stored row carries model_route fallback and degraded true. Malformed output would not have caused the switch. Times are simulated; the agent and both routes are scripted stand-ins.

  1. Primary route
    1. Primary: 3 attempts: Each attempt fails after 1000 ms; backoff of 52 ms and 908 ms between them.
    2. Breaker opens: Three failures in the last five calls open the primary route's breaker, so later runs skip it at once.
  2. Router decides
    1. Switch route: route_call leaves a route only when its attempts are exhausted or its breaker is open; malformed output and 429 never cause a switch.
  3. Fallback route
    1. Fallback drafts: A more cautious stand-in: never writes approve, two reasons instead of three, stated confidence 0.2 lower.
  4. Stored and marked
    1. Row marked degraded: The recommendation row carries model_route fallback and degraded true, so the underwriter can tell. INV-7 checks the mark.

What the second route changes: quality impact#

The fallback answer in UW-R13 has the right recommendation: escalate, citing R-ROOF-03. It would be easy to stop there and call the fallback free. Compare the two stored recommendations field by field (the primary one from UW-R01, the fallback one from UW-R13, both from the example's output files):

FieldPrimary routeFallback route
recommendationescalateescalate
citationsR-ROOF-03R-ROOF-03
Reasons3: the two roof ages, the rule sentence quoted from the policy fixture, "escalate to a senior underwriter to settle the roof age"2: the two roof ages, "the underwriter reviews this recommendation and decides"
stated_confidence0.80.6
model_route, degradedprimary, falsefallback, true
Latency, cost1400 ms, 1340 cost units900 ms, 620 cost units

The underwriter reading the fallback version gets the same conclusion with less support: no quoted rule, a generic next step, lower stated confidence. That difference is the quality impact of the fallback: what changes in the result, not in the latency, when the second route answers.

The example scripts exactly four differences between the routes. Three sit in agent_sim.py:draft; the fourth, 900 ms and 620 cost units, sits in the route settings. One of the three is a deliberate safety rule. The branch for the fallback route, dedented, otherwise verbatim:

if route == "fallback":
    closing = NEXT_STEP
    if output["recommendation"] == "approve":
        output["recommendation"] = "escalate"
        closing = "The fallback route does not write approve. Next step: a senior underwriter reviews this submission."
    output["reasons"] = [output["reasons"][0], closing]
    output["stated_confidence"] = round(output["stated_confidence"] - 0.2, 2)

The fallback route never writes approve. For a clean submission such as UW-002, where the primary route would recommend approval, the fallback writes escalate with the sentence "The fallback route does not write approve." That is the example's version of a principle worth keeping in any design: when the second route is less trusted, let it say less. It may still recommend the cautious outcomes; it may not recommend the one that would most easily be waved through.

A real fallback model differs in ways nobody scripted. It may cite a different rule, miss the conflict between the two roof ages, or read the policy fixture differently. Those are quality regressions, and measuring them before the outage, on a golden set, is the work of evaluation.

What this is not. Measuring how much worse the fallback model's recommendations are, with graders and golden cases, is eval: https://eval.sammuthu.com. This page owns the mechanism: when the route switches, how the result is marked, and what the run does when no route answers.

Marking a degraded result#

A degraded result is a smaller or less trusted result, given because a dependency was unavailable, and labelled as such. Breakers introduced the idea with request_info when the roof provider stays down. The fallback route produces the second kind in this example: a full recommendation, from a less trusted route.

The mark is two fields on the stored row, model_route: "fallback" and degraded: true, set in Workflow.draft_recommendation and written with the recommendation. INV-7 checks that every degraded or fallback result says so. The underwriter cockpit can then show it, sort it, or route it to a more senior reviewer.

Why does a mark matter if the recommendation is the same? Because nobody can know that in general. The fallback lab makes the arithmetic visible. Out of 1000 runs, with the primary route failing 10% of the time and the fallback answer differing from what the primary would have written in 30% of those runs (both synthetic settings), 98 runs go to the fallback and 28 of them get a different answer. With the mark, all 28 carry degraded: true. Without it, the same 28 look exactly like the 902 primary answers. That is the worst outcome on this page: answered, quietly different, and trusted as if nothing happened.

Try it: how much a fallback buys, and what it hides#

Set how often the primary route fails and how often the fallback answer differs, and switch the fallback and the mark on and off.

Without JavaScript, the lab's default (primary fails 10%, fallback differs 30%, 1000 runs, seed 7), from router.py:simulate_fallback run in a copy of the example:

SettingPrimary answeredFallback answeredDifferent and markedDifferent and silentNo draft
Fallback on, mark on902982800
Fallback on, mark off902980280
Fallback off90200098

The third row is not a disaster in a careful workflow. Those 98 runs still have to end in an honest terminal state, as the next section shows. The second row is the one to avoid.

When not to fall back#

A fallback trades a late answer for a different one. For a quote that a person will review, a late answer is often the better side of that trade. Four cases call for no switch, or a narrower one.

When the answer would be wrong in a way that is hard to see. A recommendation that is plausible and wrong costs more than one that arrives an hour later, because the underwriter may act on it. If the fallback model has not been evaluated on cases like this one, a degraded request_info or a wait is more honest than its draft. The example encodes a small piece of this: no approve from the fallback.

When the fallback is rarely used. Gabrielson's article explains why fallback paths are dangerous in general: they are hard to test, they may not run for months, they can carry latent bugs that appear during the outage, and they can make the outage worse, for example when every client falls back to the same smaller dependency at once. Its preferred cure is to run both paths all the time, so the second one is exercised constantly and the fallback becomes a failover. For a model, that means sending a steady small share of real runs to the fallback route even on a good day, and comparing.

When the trigger is not "down". Malformed output, a 429 that names its wait, a cost ceiling reached: each has its own answer on this site, and a switch would hide it.

When both routes are down. A second route adds an option, not a guarantee. To see what the example does, both routes were made unavailable in a variant of UW-R13 built for this page in a scratch copy (it is not one of the fifteen scenarios). The primary route gives up at 4534 ms and the fallback route at 8411 ms. The message goes back to the queue, and four more workers resume the run from its checkpoint, find both breakers still open, and give it back at once. After the fifth receive the message moves to the dead-letter queue at 13411 ms: 0 recommendation rows, run state dead_lettered, all seven invariants holding. Honest, but harsh: the redelivery interval of 1000 ms used up five receives inside the breakers' 10000 ms open time. A design for a long model outage should hold the run (or end it as a degraded request_info) instead of spending its receives against open breakers.

Common confusion. "With two routes, the model step is twice as available." Only if the two fail independently. Two routes that share a provider, a region, a network path or a quota can fail together, and so can two models behind the same gateway. Ask what the routes have in common before counting on the second one.

Hedged requests: asking twice on purpose#

A fallback waits for the primary to fail. A hedged request does not wait for a failure: it sends a second copy of a request when the first one is taking longer than usual, uses whichever answer arrives first, and cancels the other. Jeffrey Dean and Luiz André Barroso described the technique in "The Tail at Scale" (Communications of the ACM, 2013): defer the second request until the first has been outstanding longer than the 95th-percentile expected latency for that kind of request, which in their description limits the extra load to about 5% (PDF hosted by the second author). It is a weapon against tail latency, not against outages.

For an agent, hedging has two costs a search backend does not have. A model call spends cost units, so every hedge is paid for, and two copies of a model call can return two different answers; the run must take one and record which. And a hedge is only safe for a step with no side effect. Hedging the draft is possible. Hedging record_recommendation buys nothing and adds a race: two copies of the same write arrive together, and only the idempotency key stands between them and a duplicate row.

Hedge this step?Answer
get_roof_assessment (a read)Possible, if the provider's tail is long and its load allows it
draft_recommendation (the model)Possible, at a cost in cost units; record which answer was used
record_recommendation, open_review_task (writes)No

The example does not implement hedging.

How fallback goes wrong#

MistakeWhat happensOn this site
No fallback and no bounded retryThe run retries until the deadline and ends with nothingNaive UW-R13: 120 attempts, INV-4, INV-5
Fallback without the markDifferent answers look like primary answersLab: 28 silent of 1000
Switching on malformed outputA prompt or contract problem is hidden behind another modelroute_call refuses this
Switching on one transient errorThe weaker route answers runs the primary would have handledRetry first
Fallback never exercisedIt fails, or overloads, on the day it is neededGabrielson's article
Two routes with a shared failureBoth go down togetherCommon confusion above
Fallback allowed to write the riskiest outcomeA less trusted route writes approveThe example forbids it
Hedging a writeTwo writes raceHedge reads and drafts only
Before AI. A Java service with a primary and a replica database, or two payment gateways, has the same pattern: try the primary, fail over after retries or an open breaker, and alarm when the secondary is in use. The new part for an agent is that the secondary may be correct in form and different in content. A database replica returns the same rows; a second model returns its own answer to the same prompt. That is why the marking and the quality comparison carry weight here that they do not carry for a replica. Model gateways and SDKs that offer route lists and retries are described, each from its own documentation, on Frameworks.

The same idea for a coding agent#

A coding agent's primary model is unavailable in the middle of a task. Falling back to a smaller model to finish the change is possible, and sometimes fine for a formatting fix. For a change to authentication code it is the case where a late pull request beats a wrong one. A good harness records the route on the pull request ("drafted by the fallback model"), so the reviewer knows to read more carefully, and it lets the fallback open a draft pull request but not merge. The runtime that holds the routes is https://harness.sammuthu.com; the review lanes a change passes through are https://diff-to-deploy.sammuthu.com.

The blocks below hold the detailed layer.

One prompt, two models: the prompt is part of the routeA prompt tuned for one model is not automatically right for another

The example sends the same request to both routes. In practice a prompt is often tuned for one model: its length, its examples, the output format it follows reliably. Sent unchanged to another model, it may produce malformed output more often, which the workflow then re-asks and escalates. Treat the prompt version as part of the route's configuration, evaluate each pair (model, prompt) on its own, and put both in the release record that Deployment describes, so that a change to the fallback prompt gets a canary like any other change.

The output contract must not differ. Both routes are validated by the same validate_output, and the stored row has the same shape. Only the content and the mark may differ.

Watching the fallback as its own SLIThe share of runs on the fallback route is an early signal, and a budget

Count the share of runs answered by the fallback route. On a normal day it should be small and steady; a rise means the primary route is struggling before any run fails. Gabrielson's article gives the matching advice for retries: monitor their rate and alarm on unusual increases, so a retry or a fallback does not quietly become the normal path.

The share can also carry an objective of its own on SLOs: for example, at most a small percentage of a week's recommendations may be degraded (an illustrative target). When it is exceeded, the team knows that underwriters have spent the week reading weaker recommendations, even if every run ended well.

Fallback and resume: which route finishes the runThe route is part of the checkpoint, so a resume does not switch again

The checkpoint after step 5 holds state.recommendation, state.model_route and state.degraded. A run that crashed after a fallback draft resumes with the fallback answer and its mark; it does not ask the primary route again because the primary is back. Asking again would replace an answer the run already had with a different one under the same idempotency key, the replay problem of Checkpoints.

The opposite case is a run that resumes before step 5 while the primary is back. It uses whichever route is up at that moment, and the mark follows the route that actually drafted. The rule is the same as for versions: what the run used is recorded once and never quietly changed.

Checklist#

QuestionGood answer
Is the route order written down, with each route's own retries and breaker?Yes
Does a route switch only after its attempts are exhausted or its breaker is open?Yes, not on one error
Do malformed output, 429 and the cost ceiling keep their own handling?Yes, none of them switches routes
Does every fallback result carry model_route and degraded: true, checked by a test?Yes (INV-7)
Is the fallback forbidden from the riskiest outcome where that matters?Yes, for example no approve
Has the fallback model, with its prompt, been evaluated on your cases?Yes, before the outage
Is the fallback route exercised on a normal day?Yes, a steady small share
Do you know what the routes have in common?Yes: provider, region, gateway, quota
Does a run end honestly when every route is down?Yes, in a named terminal state
Are writes never hedged?Yes

In one breath#

When the primary model route is down, a fallback route can keep the run alive, but it is a different model, so its answer may be shorter, less sure or different. Switch only when the primary's attempts are exhausted or its breaker is open, never on malformed output, and mark every fallback result with model_route: "fallback" and degraded: true so the underwriter can tell. A wrong answer is worse than a late one: keep the fallback away from the riskiest outcomes, exercise it on normal days, and let a run end honestly when no route answers.

Next: proving the recovery paths work#

Every page so far showed one mechanism on one scenario: a retry, a key, a checkpoint, a breaker, a canary, a fallback. Each is a recovery path, and a recovery path that has never run under the faults it was built for is a guess. Fault injection runs all of them together, on purpose, against invariants that must hold after every fault, so the careful workflow's 295 passing runs are a result and not a hope.