The first version of the quote workflow had one rule for trouble: if a step raises an exception, try the step again. It is the rule most teams start with, and on a quiet day it works. Then three requests arrive in one morning, and the rule meets three different kinds of trouble.
For the first request, the model provider answers "too many requests, come back in 1.5 seconds". The rule tries again at once, is refused, tries again, is refused, and keeps going: 33 attempts before one gets through. For the second, the submission is missing its wind zone, so validation fails. Trying again cannot add the missing field, so the request comes back again and again, twelve deliveries, until someone stops it by hand. For the third, the underwriter cockpit refuses for good to open the review task. A retry cannot change that answer, and the rule has no other move: the recommendation stays stored with no task, and nobody will ever see it.
Those are scenarios UW-R04, UW-R11 and UW-R12 of the example, run with the naive workflow. One rule, three wrong outcomes. A rate limit needed a wait of a known length. A broken submission needed to be set aside for a person. A permanent refusal needed the earlier write to be undone in meaning. The rule treated three different failures as one, because it only asked "did it fail?" and never "how did it fail, and will that pass?"
The previous page separated a fault, its error and the failure the business sees. This page looks at the errors themselves: where they come from, which ones pass with time, how each is noticed, and how far each can spread.
The six things an agent depends on#
A dependency is anything outside a step that the step needs in order to do its work. A tiny example: step 2 of the workflow cannot get a roof assessment unless the outside roof-data provider answers, so the provider is a dependency of step 2. The quote workflow has six kinds. They are the same six in almost every agent, under different names.
The model. Step 5 sends the facts and the guidance to a model and asks for a recommendation. A model provider can refuse for a moment because of a rate limit, can be down or slow, and can answer with text that is not a valid recommendation. The last way is new: the call itself succeeds.
A tool. Step 2 calls the roof-data provider, an outside service. It can fail to answer inside the time the workflow is willing to wait, or answer that it is unavailable. A tool is anything the agent calls to read or change the world; in the example the four read tools and two write tools are listed in tools.py.
The network. Every arrow between two boxes crosses a network: to the provider, to the store, from the queue. The network's special failure is losing an answer on the way back after the request was applied. The caller sees a timeout and cannot tell whether the work happened.
The database. Steps 6 and 7 write to the store. A write can fail before it commits, which is harmless to repeat. It can also be refused for a reason that will never change: the same key with different content, or a write from a worker that has lost its claim on the submission.
A policy. Rules around the run can refuse it: a submission that never validates, a run that has spent its cost ceiling, a recommendation under an old version of the policy fixture. These refusals are explicit and deliberate, and the same input meets the same rule every time.
A person. Steps 7 and 8 involve the underwriter cockpit and the underwriter. A person can be slow or away for hours, and the cockpit can refuse a task for good. Neither is something a retry can fix.
Diagram: Six dependencies of the quote workflow · where each one sits and how it fails
Read left to right: outside the insurer, between every pair of boxes, inside the insurer. Each box is one dependency with its typical failure and the scenario that injects it. Failures on the left are mostly transient and out of the insurer's control; failures on the right are more often permanent and a matter of design.
- Outside the insurer
- Model: The model provider behind step 5. It can rate-limit, be unavailable, or answer with text that is not a valid recommendation.
- Tool: The roof-data provider behind step 2. It can fail to answer within 2000 ms or say it is unavailable.
- Between every pair of boxes
- Network: Every call crosses a network. A request can be applied and its answer lost, so the caller sees a timeout and cannot tell.
- Inside the insurer
- Database: The store behind steps 6 and 7. A write can fail before it commits, or be refused for good.
- Policy: Rules around the run: validation, the cost ceiling, the policy fixture version. The same input meets the same rule every time.
- Person: The underwriter cockpit and the underwriter. Slow or away for hours, and the cockpit can reject a task for good.
The diagram groups the six by where they sit. Two are outside the insurer (the model provider and the roof-data provider), one is between every pair of boxes (the network), and three are inside it (the store, the rules, and the people). The grouping matters for one reason: a failure outside the insurer is usually transient and not under its control, while a failure inside it is more often permanent and a matter of design.
The first question: will it pass with time?#
Every error the workflow sees deserves one question before any other: if we wait and try again, can this succeed?
A transient error passes with time. The provider was overloaded for a few seconds; the database lost a connection; the model provider asked us to slow down. Trying again later may well succeed. A permanent error will never pass by itself. The submission lacks a required field; the cockpit refuses the task; the same key arrived with a different payload. Trying again sends the same request into the same rule and gets the same answer.
HTTP already draws this line. RFC 9110 describes the 4xx status codes as client errors, for a request with bad syntax or one that cannot be fulfilled, and the 5xx codes as server errors, where the server failed to fulfil an apparently valid request (RFC 9110, section 15). Status 503 says the server cannot handle the request right now, usually because of a temporary overload or maintenance, and may carry a Retry-After header (section 15.6.4). Status 429, defined in RFC 6585, says the client sent too many requests in a given amount of time, and may also say how long to wait (RFC 6585, section 4). The Amazon Builders' Library article on timeouts and retries gives the same rule of thumb, with a caveat: client errors are generally not worth retrying with the same request, server errors may succeed later, and eventual consistency can blur the line (Timeouts, retries, and backoff with jitter).
An agent adds two kinds that HTTP does not have. Malformed output is a model answer that arrived fine and does not pass validation: neither transient in the usual sense nor permanent, because asking again may produce a valid answer. And a crash is the process itself dying; nothing inside the workflow can catch it, so the answer to a crash is never a retry but a resume from saved progress.
The example writes this down as code. The function below is classify from examples/uw-reliability/errors.py, quoted as it is. Every retry decision in the workflow starts by calling it.
def classify(error: BaseException) -> str:
"""What to do after this error: "retry", "retry_after", "reask" or "give_up"."""
if isinstance(error, RateLimited):
return "retry_after" # wait as long as the provider said, then try again
if isinstance(error, TransientError):
return "retry" # back off, then try again
if isinstance(error, MalformedOutput):
return "reask" # ask the model again, a bounded number of times
return "give_up" # permanent, breaker open, deadline gone: a retry cannot help
Four answers, and the order of the checks matters: a rate limit is a transient error with extra information, so it is checked first. The script below feeds one error of each class in errors.py to classify. It was written for this page and run in a copy of the example. A successful run shows the taxonomy in one screen; it does not show how often any of these errors happens in a real system.
from errors import (ToolTimeout, RateLimited, Unavailable, DbTransient, AckLost,
MalformedOutput, InvalidSubmission, Rejected, KeyConflict,
StaleFencingToken, BudgetExhausted, classify)
seen = [ToolTimeout("get_roof_assessment: no answer after 2000 ms"),
Unavailable("roof-data provider: 503"),
RateLimited(1500),
DbTransient("before the commit"),
AckLost("committed, but the acknowledgement was lost"),
MalformedOutput("not valid JSON"),
InvalidSubmission("field wind_zone is missing"),
Rejected("cockpit refused the review task"),
KeyConflict("same key, different payload"),
StaleFencingToken("token 41 is older than 42"),
BudgetExhausted("6000 cost units spent")]
for error in seen:
print(f"{type(error).__name__:18} -> {classify(error)}")
ToolTimeout -> retry
Unavailable -> retry
RateLimited -> retry_after
DbTransient -> retry
AckLost -> retry
MalformedOutput -> reask
InvalidSubmission -> give_up
Rejected -> give_up
KeyConflict -> give_up
StaleFencingToken -> give_up
BudgetExhausted -> give_up
Diagram: One error, four answers · what classify decides
Read left to right. Every error the workflow sees goes through one question: can trying again help? A rate limit says how long to wait. Other transient errors get a growing wait. Malformed output gets a few re-asks. Everything else stops retrying and takes another path to an acceptable outcome.
- The error
- An attempt fails: A step of the workflow raises an error instead of returning a result.
- The question
- classify(error): One function decides what to do next from the class of the error.
- The answer
- retry_after: A rate limit: wait as long as the provider said, then try again.
- retry: Another transient error: wait a growing time, within the attempt limit and the deadline.
- reask: The model answered, but the answer is not valid: ask again a bounded number of times.
- give_up: A permanent error, an open breaker or no time left: stop retrying.
- The other path
- Another way to an acceptable end: A permanent error leads to the dead-letter queue, a compensation, or an escalation written by code.
The diagram is the same decision drawn as a path. Three of the four answers try again in different ways; the fourth stops trying and must lead somewhere else, because a permanent error still has to end in an acceptable outcome.
One line deserves a second look: AckLost -> retry. A lost acknowledgement is transient, so the workflow tries again. But the first write may already be in the store. The retry is safe only because the careful workflow sends it with the same idempotency key, so the store can recognize it. "Retryable" is a property of the error; "safe to retry" is a property of the operation. The idempotency page closes that gap.
Before AI. Java drew the same line in JDBC long ago.SQLTransientExceptionis thrown when a failed operation might succeed if retried without any intervention by the application, and it has subclasses such asSQLTimeoutExceptionandSQLTransientConnectionException.SQLNonTransientExceptionis for failures where the same operation would fail again until the cause is corrected, with subclasses such asSQLIntegrityConstraintViolationExceptionandSQLSyntaxErrorException(Java SE 21 API). A Spring service that retries only on the transient branch is doing whatclassifydoes. The difference for an agent is the third answer,reask, which exists because a model can return a wrong answer with a successful status.
Four failure modes worth naming#
A failure mode is one specific way in which a dependency fails, with its own signature and its own right response. Four modes cause most of the trouble in agent workflows. Each comes with its scenario from the example.
The rate limit: "not now, come back later"#
A rate limit error says that the caller sent too many requests in a window and should come back later. Model providers use rate limits to share capacity, and in HTTP the signal is status 429, often with a hint of how long to wait. RFC 6585 leaves it to the server how it counts and whom it counts: per resource, per server, per user (RFC 6585, section 4). So the client cannot compute the wait; it has to read it.
In UW-R04 the model answers 429 twice, each time with retry_after_ms 1500. The careful workflow does what it was told, from a copy of the example:
674 worker-A draft_recommendation 1 attempt failed: RateLimited: rate limited (429); retry after 1500 ms
674 worker-A draft_recommendation 1 wait as told (retry_after_ms): 1500 ms
2274 worker-A draft_recommendation 2 attempt failed: RateLimited: rate limited (429); retry after 1500 ms
2274 worker-A draft_recommendation 2 wait as told (retry_after_ms): 1500 ms
5174 worker-A draft_recommendation 3 attempt ok
Three attempts. The naive workflow, which tries again at once, needs 33 attempts for the same result, and breaks INV-5 because it exceeds the maximum of three attempts and overdraws the retry budget. Every one of the 32 refused calls also counts against the same quota that the provider is protecting. Ignoring a rate limit makes the rate limit last longer.
The timeout: "I stopped listening"#
A timeout is the longest we are willing to wait for one call. In this example the roof-data provider gets 2,000 ms; after that, the workflow stops listening. A timeout is transient by default, and in UW-R02 attempt 3 succeeds after two of them. But the timeout is the most misread error of all, because it is a statement about the caller, not about the callee. It says "I gave up waiting". It does not say whether the other side received the request, is still working on it, or finished it a moment ago. For a read, that does not matter. For a write, it is the whole problem: a timeout tells you nothing about whether the write happened. How long to wait, and how a whole run's time limit is shared between its calls, is the subject of retries.
Malformed output: "the call worked, the answer did not"#
Malformed output is a model answer that came back successfully and does not pass validation: it is not valid JSON, it uses a value outside the four allowed recommendations, it cites a rule that does not exist, or it lacks a required field. Nothing at the transport level signals it. The status is fine, the latency is normal, the body is text. Only a check of the content finds it.
In UW-R05 the first answer is cut off in the middle of a string. The careful workflow parses it, sees the error, and asks again; the second answer is valid. It asks again at most twice; if both re-asks fail, it writes escalate itself, with the reason "output could not be validated", so that a person sees the case instead of a broken row. The naive workflow stores the broken text as the recommendation, and INV-3 and INV-6 fail. The bounded "ask again" is a retry of a special kind: it may help, but only a few times, because a model that keeps producing broken output for one input is telling you something about the input or the prompt.
What this is not. How to design the output schema, validate it and repair a near-miss is the subject of Structured Output Engineering. This site owns one part: treating a failed validation as an error with a bounded number of re-asks.
Gray failure: "everything looks healthy"#
The hardest mode has no error at all. A gray failure is a dependency that is not working properly while the checks that watch it say it is fine. Researchers at Microsoft named it in a 2017 paper and called its key trait differential observability: the system's failure detector sees a healthy component while the applications that use it see problems (Huang et al., HotOS 2017).
An agent offers many places for this. The following are illustrations; the example does not inject a gray failure. A roof-data provider answers every call quickly with status 200, but for one region it has started returning a default roof age instead of the real one. A model route passes its health check, which sends one short prompt, while it truncates long prompts. A retrieval index answers every query, but a failed rebuild left it with last year's guidance. In each case every call succeeds, every dashboard of call errors is green, and the recommendations are getting worse.
Gray failures are found by checking outcomes, not calls: an invariant over what was stored (INV-6 would catch a recommendation under an old policy_version), a comparison between two sources (the conflict rule R-ROOF-03 exists because the application and the provider can disagree), or a sample of results that a person reads. The closest the example comes is UW-R05: the call succeeded, and only validation saw the problem.
How each failure is detected#
Each mode leaves a different trace, and a workflow can only respond to what it detects. The table puts the four modes and the two remaining dependencies side by side.
| Failure | What the workflow sees | Detected by | Detected when |
|---|---|---|---|
| Rate limit | Status 429, a wait hint | The status code | At once |
| Timeout or unavailable | No answer in time, or 503 | The timeout of one call, or the status | After the timeout |
| Lost acknowledgement | A timeout on a write | Nothing certain: only the store knows | Never, by the caller alone |
| Malformed output | A normal answer | Validation of the content | Before the output is used, if checked |
| Policy refusal | An explicit rejection by code | The rule that refused | At once, every time |
| Gray failure | Nothing unusual | An invariant over outcomes, a cross-check, a person | Late, and only if someone looks |
Read the last column from top to bottom. The failures near the top announce themselves; the ones near the bottom have to be looked for. Reliability work tends to start at the top because those are easy to handle, and the expensive incidents tend to come from the bottom.
What this is not. Collecting the signals (logs, metrics, traces) that make a failure visible belongs to Observability Engineering. This page names what to look for; that site shows how to see it.
Blast radius: how far one failure reaches#
A failure that stays inside one attempt is a nuisance. The same failure spread across every run is an outage. The blast radius of a failure is how much it can affect: one attempt, one step, one run, every run that shares a dependency, or every submission.
| Radius | Example | What keeps it there |
|---|---|---|
| One attempt | The provider times out once (UW-R02) | A bounded retry |
| One step | The provider stays down for this run (UW-R03) | Maximum attempts, then degrade to request_info |
| One run | The process crashes after the write (UW-R08) | Resume from the checkpoint; the run's deadline |
| Every run on a dependency | The provider is down for an hour; every run waits on it | A circuit breaker that stops calling it for a while |
| Every submission | A new version of the policy fixture is deployed with a wrong roof limit | Deployment in small steps and a way back |
The radius is not a property of the fault alone. It depends on what the system does next. A provider outage has a one-step radius in the careful workflow, because each run tries three times and then goes on without the roof data. The same outage in the naive workflow reaches further: every run retries until its deadline, holding a worker the whole time, and the retries all land on the provider while it is trying to recover. Retries that ignore the radius can turn a one-dependency fault into a whole-system failure, which the Google SRE book describes as a cascading failure (Addressing Cascading Failures). The breakers page shows how to stop that spread, and deployment covers the widest row.
The error taxonomy of the example#
All of this is written down in one file, errors.py, and every scenario uses it. The table is the reference for the rest of the site.
| Class | Kind | classify says | Dependency | Scenario |
|---|---|---|---|---|
ToolTimeout | Transient | retry | Tool, model | UW-R02, UW-R03 |
Unavailable | Transient | retry | Tool, model route | UW-R13 |
RateLimited | Transient, with a wait | retry_after | Model | UW-R04 |
DbTransient | Transient, nothing written | retry | Database | UW-R07 |
AckLost | Transient, written but not confirmed | retry (same key only) | Network, database | UW-R06 |
MalformedOutput | Its own kind | reask, at most twice | Model | UW-R05 |
InvalidSubmission | Permanent | give_up, then the dead-letter queue | Policy | UW-R11 |
Rejected | Permanent | give_up, then compensation | Person (the cockpit) | UW-R12 |
KeyConflict | Permanent | give_up | Database | (unit tests) |
StaleFencingToken | Permanent | give_up | Database | UW-R14 |
BudgetExhausted | Permanent | give_up, then a code-written escalate | Policy | (unit tests) |
Crash | Not catchable | resume, never retry | The process | UW-R08, UW-R09 |
Two design choices sit in this table. Crash derives from BaseException in the example, so no except Exception can catch it, which forces the recovery to happen outside the failing process. And every permanent error has a named next step (the dead-letter queue, compensation, a code-written escalation) instead of "log and drop". A permanent error is not the end of the run's story; it is the signal to take a different path to an acceptable outcome.
Try it: six dependencies, one at a time#
The lab below shows the six dependencies. Click one to see how it fails, how the failure is detected, whether a retry may help, and what the example does about it.
If the lab does not load, this table holds the same content (from the lab data in scripts/build_data.py, built from errors.py, the workflow table and the scenarios).
| Dependency | How it fails | Detected by | Retry helps? | Response in the example |
|---|---|---|---|---|
| Model | 429; route down or too slow; malformed answer | Status or timeout; only validation finds a malformed answer | May help | UW-R04 waits 1500 ms twice; UW-R05 asks again; UW-R13 uses the fallback route, marked degraded |
| Tool | No answer in 2000 ms; 503 | The timeout or the error status | May help | UW-R02 succeeds on attempt 3; UW-R03 opens the breaker and goes on with request_info |
| Network | Request applied, answer lost; queue acknowledgement lost | Only as a timeout | May help, with the same key | UW-R06 replays with the same key; UW-R10 finds the same run by message_id |
| Database | Fails before commit; stale fencing token; key conflict | An error from the store | Depends | UW-R07 one retry; UW-R14 refuses the stale write |
| Policy | Submission never validates; cost ceiling reached; old policy version | An explicit refusal by code; INV-6 | Cannot help | UW-R11 dead-letters after 5 receives |
| Human | Person slow or away; cockpit rejects the task | A run in waiting_for_human; a permanent rejection | Cannot help | UW-R12 marks the row withdrawn and ends in failed_compensated |
Then test yourself: five errors from the example, and for each one the right response.
If the quiz does not load, here are the questions with the answers and the reasons (from scripts/build_data.py).
| Error | Right response | Why |
|---|---|---|
The model answers 429 with retry_after_ms 1500 (UW-R04) | Wait 1500 ms, then try again | A rate limit passes with time and the provider said how long; trying at once was refused 32 more times |
The cockpit rejects the review task for good (UW-R12) | No retry | A permanent error never passes; the careful workflow compensates by marking the recommendation withdrawn |
The write times out and nobody knows whether it committed (UW-R06) | Retry with the same idempotency key | The store can answer "already written"; a fresh key makes the retry a second write |
The model's output fails validation (UW-R05) | Ask again, at most twice, then escalate with the reason | Asking again may help a bounded number of times |
A submission never validates (UW-R11) | Move it to the dead-letter queue after 5 receives | The same input fails the same way; the dead-letter queue keeps it for a person |
The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
The same idea for a coding agent#
A coding agent depends on the same six things under other names. Its model can rate-limit it. Its tools (the test runner, the package registry, the hosting service's API) can time out. A git push can apply on the server and lose its answer on the way back, which is the coding agent's lost acknowledgement. CI is its database of record, and a flaky test is its gray failure: the job reports a result, and the result does not reflect the code. A protected branch that refuses a direct push is a policy failure, permanent for that request. And the reviewer is the person who may be away for the weekend. Retrying a flaky test until it passes is the coding agent's version of treating a gray failure as transient. The runtime that classifies these errors around the model loop is the subject of Harness Engineering, and CI as a gate is covered by Diff to Deploy.
How each framework and model SDK classifies errors and which it retries by default is described from their own documentation on the frameworks page.
When the classification is wrongPermanent as transient, transient as permanent, two classes as one
A taxonomy helps only if errors land in the right class. Misclassification goes wrong in two directions, and both have a scenario.
A permanent error treated as transient. The naive UW-R11 retries a submission that can never validate: twelve deliveries, twelve runs, and the simulation has to stop it. In production nothing stops it, and the message keeps a worker busy forever while the request behind it waits. The same happens to a 400-class response retried in a loop, or to a model asked again and again for an input it cannot handle.
A transient error treated as permanent. The opposite mistake gives up too early. A workflow that treats every timeout as final would fail UW-R02, where attempt 3 succeeds, and would send many runs to a person that a short wait would have finished.
One class for an error that has two. The database dependency shows the subtle case: "the store returned an error" can mean "nothing was written, try again" (DbTransient) or "this write must never happen" (StaleFencingToken). A wrapper that turns every store error into one generic exception has thrown away the one fact the caller needed.
"Transient" is a forecast, not a factWhy the class of an error is a bet on the future
Calling an error transient predicts that the condition behind it will end soon. Usually that bet is right: an overloaded provider recovers, a connection is re-established. But the error itself carries no proof. A provider that returns 503 for an hour is, from the caller's point of view, permanently unavailable for that run. That is why a transient classification is always paired with limits: a maximum number of attempts, a retry budget, and the run's deadline. The limits convert "transient, so retry" into "transient, so retry a little, then treat it as permanent for this run". UW-R03 is that conversion in action.
The bet can fail the other way too. The Builders' Library article notes that eventual consistency can make a client error turn into a success later: a request for a resource that was created a moment ago may first see "not found". A workflow that treats every 404 as permanent will sometimes be wrong. The answer is not to retry every 404; it is to know which reads in your system can lag behind writes, and to classify those specifically.
The person as a dependencySlow, unavailable, and never retried
It is uncomfortable to list a person next to a database, but for reliability the comparison is useful. The underwriter has an availability (working hours, holidays), a latency (minutes to days), a capacity (tasks per day) and failure modes (a task never picked up, a decision with a missing reason). None of them responds to a retry: sending the same task twice makes things worse. The example stops at waiting_for_human and models one human-side failure, the cockpit's permanent rejection in UW-R12, which is handled by compensation. Approval timeouts, escalation to another person and capacity planning for the review queue are on the advanced page.
Checklist#
| Question | A good answer looks like |
|---|---|
| Is every error your workflow can see in a named class? | A table like the one above, in code, with a test per class |
| Does a rate limit wait as long as the provider said? | The wait comes from the response, never a fixed guess |
| Is model output validated before it is used? | Parse and check, then a bounded number of re-asks, then a person |
| Does each permanent error have a next step? | Dead-letter, compensate or escalate; never only "log" |
| Can your store tell a transient failure from a refusal? | Different errors for "not written" and "must not be written" |
| How would you notice a gray failure? | At least one invariant or cross-check over stored outcomes |
| What is the blast radius of your busiest dependency failing for an hour? | One step per run, not every worker held until its deadline |
In one breath#
An agent depends on six things (the model, its tools, the network, the store, its policies and people), and each fails in its own way. Before trying again, ask whether the error will pass with time: rate limits, timeouts and unavailability may; a broken submission, a refused task or a key conflict never will, and malformed output gets a few re-asks and then a person. The dangerous failures are the quiet ones: a timeout that hides a completed write, and a gray failure where every call succeeds and the outcomes are wrong. And how far a failure spreads depends less on the fault than on what the system does next.
You can now sort the errors of the quote workflow into those worth trying again and those that need another path. What you do not yet have is a safe way to try again: how long to wait for one call, how long to wait between attempts, when to stop, and how to keep a thousand runs from retrying in step against a provider that is already struggling. Retries builds that, starting from the timeout.