Retry-After tells you when the other side wants you back.Submission SUB-1006 reaches step 2 of the workflow. The underwriting agent asks the roof-data provider for the roof assessment, and nothing comes back. The provider is down, and it will stay down for the whole story (scenario UW-R03). The agent will write a recommendation for an underwriter. The model recommends; a person decides. But first it needs to know what to do with a call that does not answer.
The first idea most engineers have is to try again. The naive workflow of the example does exactly that: it tries again at once, with no cap, until the run has no time left. This is what it did:
2000 worker-A get_roof_assessment 1 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
4000 worker-A get_roof_assessment 2 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
... 57 more times: attempt failed ...
120000 worker-A get_roof_assessment 60 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
120000 worker-A get_roof_assessment - DEADLINE EXCEEDED: get_roof_assessment: the deadline passed after 60 attempt(s)
120000 worker-A get_roof_assessment - run ended: deadline_exceeded
Sixty calls to a provider that was already failing. Two minutes of a worker's time. And at the end, nothing for the underwriter: no recommendation, not even an honest "we could not get the roof data". The retry made the outcome worse than giving up after the first call would have.
Now picture every worker of the insurer doing this on the morning the provider has trouble. That is how a small outage becomes a long one.
This page builds the careful version, one decision at a time: how long to wait for one call, how long the whole run may take, which errors deserve another try, how long to pause between tries, and how many retries the system as a whole can afford. One note before the first result: the agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
1. Two clocks: a timeout for one call, a deadline for the run#
Start with the smallest question. When the agent calls the provider, how long does it listen before it decides that no answer is coming?
A timeout is the longest we are willing to wait for one call. In this example the roof-data provider gets 2000 ms per attempt; after that the agent stops listening and treats the call as failed. Without a timeout, a call to a provider that hangs keeps the worker waiting forever, and a worker that waits forever is a worker that serves nobody else.
A timeout alone does not stop the naive loop above. Every one of its sixty calls respected the 2000 ms timeout. What it lacked was a limit on the whole job.
A deadline is the moment by which the whole run must be finished. In this example the automated part of a run gets 120000 ms from the moment the run is created. The deadline is a point in time, not a length: at 90000 ms into the run, 30000 ms are left, whatever has happened so far.
The two clocks work together through one rule, called deadline propagation: a call never gets a longer timeout than the time left before the deadline. If 1500 ms are left, the next call gets 1500 ms, not 2000. Google's SRE book recommends the same thing across services: pass the remaining time down the chain of calls instead of letting each server pick a fresh deadline, so that nobody keeps working on a request that its caller has already abandoned.
| Timeout | Deadline | |
|---|---|---|
| Bounds | One attempt of one call | The whole automated part of the run |
| In this example | 2000 ms (RetryPolicy.timeout_ms) | 120000 ms from run creation (RUN_DEADLINE_MS) |
| Kind of value | A length | A moment |
| When it runs out | This attempt failed; maybe try again | Stop. No further attempt, no further wait |
| Who passes it on | Set per call | Carried by the run; every timeout is cut to fit it |
Common confusion. "We have a timeout, so we cannot hang." A timeout bounds one call. A loop of calls, each with a timeout, can still run forever. Only a deadline bounds the loop.
Before AI. A Spring service calling another over HTTP sets a connect timeout and a read timeout on its client, and a gateway in front of it often sets a request deadline. That is the same pair of clocks. What changes for an agent is the length of the run. A web request lasts a few hundred milliseconds; one agent run calls a provider, a retrieval index, a model and a database, may wait for a rate limit, and can last minutes. The longer the run, the more a forgotten deadline costs.
2. Retry only what might pass#
The second question: after an error, is another attempt worth anything at all?
Some errors pass with time. The provider was restarting, a network link dropped a packet, the model provider was briefly over its limit. A transient error like that may be gone by the next attempt. Other errors will never pass by themselves. A submission without a wind zone will be invalid on the hundredth try exactly as on the first. That is a permanent error, and retrying it only burns time and load. The full taxonomy, with the six dependencies of the agent and how each one fails, is on Failure modes.
A retry is another attempt of the same step after an error. The word is narrow on this site on purpose: a retry happens inside one run, in the same process, a few hundred milliseconds after the error. A new process picking up a crashed run is a resume, which is a different mechanism on a different page (Checkpoints).
The example puts the decision in one function. It looks at the class of the error and answers with one of four actions. This is errors.py:classify, verbatim:
def classify(error: BaseException) -> str:
"""What to do after this error: "retry", "retry_after", "reask" or "give_up"."""
if isinstance(error, RateLimited):
return "retry_after" # wait as long as the provider said, then try again
if isinstance(error, TransientError):
return "retry" # back off, then try again
if isinstance(error, MalformedOutput):
return "reask" # ask the model again, a bounded number of times
return "give_up" # permanent, breaker open, deadline gone: a retry cannot help
| Action | Errors that get it | What the loop does next |
|---|---|---|
retry | ToolTimeout, Unavailable, DbTransient, AckLost | Waits with backoff (section 3), then tries again |
retry_after | RateLimited (a 429 in a real system) | Waits exactly as long as the provider said (section 6) |
reask | MalformedOutput from the model | Asks again at once, at most twice |
give_up | Anything permanent, an open breaker, a passed deadline | Raises the error now; no retry |
Notice that give_up is the default. An error nobody planned for is not retried. That choice matters: a loop that retries everything it does not recognize turns every new bug into extra load.
What this is not. Validating the model's output and repairing it is the subject of https://schema.sammuthu.com. This site owns only the bounded "ask again" as a kind of retry.
Production pitfall.ToolTimeoutandAckLostare both in theretryrow, and both hide a question that a retry cannot answer: did the first attempt already change something? For the roof lookup it does not matter, because a read changes nothing. For step 6, which writes the recommendation row, it matters a great deal. A timeout tells you nothing about whether the write happened. The next chapter, Idempotency, exists for that question.
3. How long to wait before the next attempt#
The provider in scenario UW-R02 times out twice and then answers. So a retry will help. The remaining question is when to send it.
At once is the worst choice when the provider is overloaded, because the retry arrives while the overload is still there. The common answer is to wait, and to wait longer after each failure. That is exponential backoff: the ceiling of the wait doubles with every attempt, up to a cap. In this example the base is 500 ms and the cap is 8000 ms, so the ceilings after attempts 1, 2, 3 and 4 are 500, 1000, 2000 and 4000 ms.
Backoff alone leaves one problem. If a hundred workers failed at the same moment, they all wait the same 500 ms and all come back at the same moment. Jitter is randomness added to the wait, so that clients who failed together do not retry together. The example offers three kinds. This is retry.py:backoff_delay, verbatim:
def backoff_delay(attempt: int, base_ms: int, cap_ms: int, jitter: str, rng: Mulberry32) -> int:
"""How long to wait after failed attempt number `attempt` (1 is the first).
The ceiling doubles with every attempt and stops at the cap.
none wait the ceiling itself (every client waits the same: a herd)
full wait a random time between 0 and the ceiling
equal wait half the ceiling plus a random time up to the other half
"""
ceiling = min(cap_ms, base_ms * 2 ** (attempt - 1))
if jitter == "none":
return ceiling
if jitter == "full":
return int(rng.next_float() * ceiling)
if jitter == "equal":
half = ceiling // 2
return half + int(rng.next_float() * (ceiling - half))
raise ValueError(f"unknown jitter {jitter!r}")
The next command prints the four waits one client would use between five attempts, once per kind of jitter. What it proves is narrow: each wait stays at or under its ceiling, and the random draws come from the seeded generator, so the same seed gives the same numbers on every machine. It does not prove that any of these settings suits a real provider.
cd examples/uw-reliability
/opt/homebrew/bin/python3 -c "
from retry import schedule
for jitter in ('none', 'full', 'equal'):
print(jitter, schedule(5, 500, 8000, jitter, 7))
"
none [500, 1000, 2000, 4000]
full [5, 61, 1953, 2796]
equal [252, 530, 1976, 3398]
Without jitter, the waits are the ceilings. With full jitter, any value from 0 to the ceiling is possible: 5 ms is as legal as 499 ms. With equal jitter, the wait is never below half the ceiling. The careful workflow uses full jitter, which is why UW-R02 waited 52 ms after the first timeout (ceiling 500) and 908 ms after the second (ceiling 1000).
| Jitter | Wait after attempt n (ceiling c = min(cap, base times 2 to the power n minus 1)) | Strength | Weakness |
|---|---|---|---|
| none | c | Predictable | Clients who failed together come back together |
| full | random between 0 and c | Spreads clients the most | A single client may retry almost at once |
| equal | c/2 plus random up to c/2 | Keeps a minimum pause | Spreads less than full jitter |
The Amazon Builders' Library article on timeouts and retries and the SRE book both recommend capped exponential backoff with randomness, and a small number of retries.
4. A hundred clients fail at once: the retry storm#
One client's schedule hides the problem jitter solves. It shows up only when many clients fail together, which is exactly what happens when a shared dependency has trouble.
A retry storm is the wave of retries that follows a shared failure: every client retries, the retries arrive together, and the dependency that was struggling now receives more traffic than before it failed. Without a limit it can keep the dependency down long after the original cause is gone.
The example's retry.py:herd lets 100 clients fail at time 0 and counts when their retries land, in 250 ms buckets. Every attempt fails, so the picture shows the retry timing alone. The command runs it without jitter and with full jitter; the claim it supports is about timing, not about any real service:
cd examples/uw-reliability
/opt/homebrew/bin/python3 -c "
from retry import herd
for jitter in ('none', 'full'):
h = herd(100, 4, 500, 8000, jitter, 7, 250)
print(jitter, h['retries'], 'retries, peak', h['peak'], 'at', h['peak_at_ms'], 'ms, buckets', h['buckets'])
"
none 300 retries, peak 100 at 500 ms, buckets [0, 0, 100, 0, 0, 0, 100, 0, 0, 0, 0, 0, 0, 0, 100]
full 300 retries, peak 85 at 250 ms, buckets [45, 85, 28, 24, 30, 19, 19, 9, 9, 11, 8, 8, 5]
The same 300 retries, two very different loads. Without jitter, all 100 clients hit the dependency in the same quarter second, three times: at 500 ms, at 1500 ms and at 3500 ms, with silence in between. With full jitter the first wave still peaks at 85 in one bucket, because the first ceiling is only 500 ms wide, but the later retries spread over three seconds and the largest later bucket holds 30.
Diagram: A hundred clients fail at once · the same 300 retries, without and with jitter
Each lane is the same hundred clients, failing together at time 0 and retrying three times. Read a lane left to right. Without jitter every retry wave hits the dependency in one quarter second. With full jitter the first wave is still narrow, because its ceiling is only 500 ms, and the later waves spread out. The numbers come from retry.py:herd with seed 7 and are synthetic.
- No jitter
- 100 at 500 ms: All hundred clients waited exactly 500 ms, so all hundred retries arrive in the same quarter second.
- 100 at 1500 ms: They all failed again together and all waited 1000 ms: the second wave is as sharp as the first.
- 100 at 3500 ms: The third wave, after a wait of 2000 ms. Between the waves the dependency sees nothing; during them it sees everything.
- Full jitter
- Spread over 0 to 500 ms: Each client waits a random time up to 500 ms. The window is narrow, so the first wave still peaks.
- Spread wider: The second ceiling is 1000 ms and each client starts from a different moment. From here the waves overlap: one bucket mixes second and third retries of different clients.
- A thin tail: By the third retry the hundred clients are spread over seconds. The total is still 300 retries: jitter spreads load, the retry budget reduces it.
Jitter spreads the retries. It does not reduce their number. Three hundred retries are still three hundred extra calls to a dependency that is failing, and that is the job of the next section.
Try it yourself. The lab below lets you change the number of attempts, the base delay, the cap, the deadline and the jitter. It draws one client's waits against their ceilings, says whether the worst case fits the deadline, and repeats the 100-client experiment.
The same information without the lab, at its default settings (base 500 ms, cap 8000 ms, from schedule and herd above):
| Attempt that failed | Ceiling of the wait | No jitter waits | Full jitter waits | Equal jitter waits |
|---|---|---|---|---|
| 1 | 500 ms | 500 ms | 0 to 500 ms | 250 to 500 ms |
| 2 | 1000 ms | 1000 ms | 0 to 1000 ms | 500 to 1000 ms |
| 3 | 2000 ms | 2000 ms | 0 to 2000 ms | 1000 to 2000 ms |
| 4 | 4000 ms | 4000 ms | 0 to 4000 ms | 2000 to 4000 ms |
The worst case against the deadline, in this example: three attempts of 2000 ms plus waits of at most 500 and 1000 ms is 7500 ms, far inside the 120000 ms deadline. So one step can never spend the run's whole deadline on retries, which is the failure we started with.
5. How many retries the system can afford: the retry budget#
Each step in the careful workflow makes at most 3 attempts. That caps one call. It does not cap the system: if every one of a thousand runs retries twice, the provider sees three times its normal load, from callers who each behaved correctly.
A retry budget is a limit on retries for a whole process or system, not for one call. The example implements it as a bucket of tokens shared by all steps. Every first attempt drops a small amount in, every retry takes one token out, and a retry with no token left is refused. The Builders' Library article describes this kind of local token bucket for retries, and the SRE book describes a per-process retry budget with the same purpose. This is retry.py:RetryBudget, verbatim:
class RetryBudget:
"""A token bucket for retries, shared by all steps.
Every first attempt deposits `ratio` tokens. Every retry withdraws one.
When the bucket is empty the retry is refused, so retries can never grow
into a multiple of the normal traffic.
"""
def __init__(self, ratio: float = 0.2, initial: float = 10.0, cap: float = 10.0) -> None:
self.ratio, self.cap, self.tokens = ratio, cap, initial
self.retries = self.refused = 0
def first_attempt(self) -> None:
self.tokens = min(self.cap, self.tokens + self.ratio)
def try_retry(self) -> bool:
if self.tokens < 1:
self.refused += 1
return False
self.tokens -= 1
self.retries += 1
return True
The settings of the example are a start of 10 tokens, a cap of 10, a deposit of 0.2 per first attempt and a withdrawal of 1 per retry. The next command empties a full bucket and then refills it with five first attempts. It shows the rule at work; it does not show that 0.2 is the right ratio for any real system.
cd examples/uw-reliability
/opt/homebrew/bin/python3 -c "
from retry import RetryBudget
b = RetryBudget(ratio=0.2, initial=10.0, cap=10.0)
granted = 0
while b.try_retry():
granted += 1
print('retries granted from a full bucket:', granted, '· refused:', b.refused)
for _ in range(5):
b.first_attempt()
print('after 5 first attempts:', round(b.tokens, 2), 'tokens · next retry allowed:', b.try_retry())
"
retries granted from a full bucket: 10 · refused: 1
after 5 first attempts: 1.0 tokens · next retry allowed: True
Read the rule as a sentence. A burst of up to 10 retries is allowed at any time. After that, the system earns one retry for every five first attempts. Over a long stretch, retries can never be more than about one fifth of normal traffic, plus the burst. When the provider fails for everyone, the bucket empties within seconds and the system stops adding load: errors come back fast instead of slowly, and the circuit breaker takes over from there.
The invariant INV-5 checks this after every scenario: no step exceeds its maximum attempts, and the retry budget is never overdrawn. The naive run of UW-R03 fails it in so many words:
INV-5 FAIL run-bef5: step get_roof_assessment made 60 attempts (maximum 3); 59 retries, the retry budget allows 10
6. When the other side tells you when to come back: Retry-After#
Scenario UW-R04 is a different kind of refusal. The model provider answers 429, "too many requests", and adds how long to wait: retry_after_ms 1500. It does this twice.
Backoff is a guess about when the other side will be ready. A rate-limit answer can replace the guess with a statement. RFC 6585 defines status 429 and says the response may carry a Retry-After header; RFC 9110, section 10.2.3, defines that header as either a date or a number of seconds to wait. Note the unit: the HTTP header counts seconds, while the example's field retry_after_ms counts milliseconds. A client that reads one as the other waits a thousand times too long or too short.
The example honors the value in one line. This is retry.py:retry_delay, verbatim:
def retry_delay(action: str, error: Exception, attempt: int, policy: RetryPolicy, rng: Mulberry32) -> int:
"""The wait before the next attempt: as told (429), none (ask again) or backoff with jitter."""
if action == "retry_after":
return error.retry_after_ms # type: ignore[attr-defined]
if action == "reask":
return 0
return backoff_delay(attempt, policy.base_ms, policy.cap_ms, policy.jitter, rng)
The careful run waits as told, twice, and the third attempt succeeds:
674 worker-A draft_recommendation 1 attempt failed: RateLimited: rate limited (429); retry after 1500 ms
674 worker-A draft_recommendation 1 wait as told (retry_after_ms): 1500 ms
2274 worker-A draft_recommendation 2 attempt failed: RateLimited: rate limited (429); retry after 1500 ms
2274 worker-A draft_recommendation 2 wait as told (retry_after_ms): 1500 ms
5174 worker-A draft_recommendation 3 attempt ok
The naive run ignores the header and asks again at once. The stand-in provider refuses every call that comes back early and says how much of the wait is left:
674 worker-A draft_recommendation 1 attempt failed: RateLimited: rate limited (429); retry after 1500 ms
774 worker-A draft_recommendation 2 attempt failed: RateLimited: rate limited (429); retry after 1400 ms
... 29 more times: attempt failed ...
3774 worker-A draft_recommendation 32 attempt failed: RateLimited: rate limited (429); retry after 0 ms
5174 worker-A draft_recommendation 33 attempt ok
Both runs finish at 5174 ms. The naive one made 33 attempts to get there, 32 of them refused, each one counted against the insurer's quota by a real provider. Ignoring Retry-After bought no time and spent a great deal of goodwill. In this example it also breaks INV-5.
7. The whole loop, and the attempt number that names each try#
Now the pieces fit into one function. retry.py:call_with_retry takes the call, the policy, the clock, the deadline and the budget. Before each attempt it cuts the timeout to the time left. After a failure it asks classify, refuses a wait that would pass the deadline, takes a token from the budget, and only then waits. Quoted verbatim:
def call_with_retry(fn: Callable[[int], Any], policy: RetryPolicy, clock: Clock, deadline: Deadline, budget: RetryBudget,
tracer: Tracer, name: str, rng: Mulberry32, kind: str = "tool",
stats: dict[str, Any] | None = None, attributes: dict[str, Any] | None = None) -> Any:
"""Call `fn(timeout_ms)` until it succeeds, within the policy, the retry budget and the deadline.
The timeout handed to `fn` is never longer than the time left before the
deadline. The loop never sleeps past the deadline. An error that a retry
cannot help is raised at once.
"""
budget.first_attempt()
error: Exception | None = None
for attempt in range(1, policy.max_attempts + 1):
timeout_ms = min(policy.timeout_ms, deadline.remaining(clock))
if timeout_ms <= 0:
raise DeadlineExceeded(f"{name}: the deadline passed before attempt {attempt}")
result, error = attempt_once(fn, timeout_ms, tracer, name, kind, attempt, stats, attributes)
if error is None:
return result
action = classify(error)
if action == "give_up":
raise error
if attempt == policy.max_attempts:
break
delay = retry_delay(action, error, attempt, policy, rng)
if delay >= deadline.remaining(clock):
raise DeadlineExceeded(f"{name}: a wait of {delay} ms would pass the deadline")
if not budget.try_retry():
raise RetriesExhausted(name, attempt, error, "the retry budget is empty")
tracer.event(name, WAIT_EVENT[action], f"{delay} ms", attempt)
clock.advance(delay)
assert error is not None
raise RetriesExhausted(name, policy.max_attempts, error)
The diagram follows one failed attempt through the four checks in the order the code makes them.
Diagram: One failed attempt, four checks · what call_with_retry does before it tries again
Read left to right. An attempt gets a timeout cut to the time left. If it fails, the error is classified first: an error a retry cannot help ends the loop at once. Then three limits are checked in order: the maximum attempts, the deadline, the retry budget. Only a retry that passes all of them waits and goes back to the first box.
- 1 · Try
- One attempt: The call is sent with a timeout that is never longer than the time left before the run's deadline. The attempt is recorded as a span with its attempt_number.
- 2 · Can a retry help?
- classify(error): The class of the error decides. Transient errors may pass with time; permanent and unknown errors give up at once.
- Give up now: No retry. The error goes to the step, which may degrade, compensate or hand the message back to the queue.
- 3 · Three limits
- Attempts left?: A step makes at most three attempts in this example. After the third failure the loop stops with RetriesExhausted.
- Wait fits the deadline?: If the wait would reach the deadline, the loop stops now instead of sleeping past it.
- Token in the budget?: Every retry takes one token; every first attempt adds 0.2. With no token left the retry is refused.
- 4 · Wait, then try again
- Wait: The wait is a random time up to a doubling ceiling, or exactly the retry_after_ms the provider sent. Then the next attempt starts in box 1.
The loop counts with attempt, and that number travels further than the loop. The attempt number (attempt_number on this site) names one try of one step inside one run: 1 for the first try, 2 for the first retry. attempt_once records every attempt as its own span in the run's trace, with attempt_number as an attribute. On this site there is one trace per run; a retry is a new span in the same trace, not a new trace. How spans are collected and read is the territory of https://observe.sammuthu.com.
attempt_number has one job, and it is easy to give it a second one by mistake. It changes on every retry, so it must never be part of anything that should stay the same across retries. The idempotency key of a write is the main example: a key that contains the attempt number turns every retry into a "new" write. The next chapter shows that failure in its lab.
Here is the careful run of UW-R02, the provider that times out twice and then answers, from run_scenario.py --scenario UW-R02. A successful run proves that the loop retried a transient error with jittered waits inside the deadline and the run reached its normal end. It does not prove anything about a real provider.
cd examples/uw-reliability
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R02
2000 worker-A get_roof_assessment 1 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
2000 worker-A get_roof_assessment 1 backoff: 52 ms
4052 worker-A get_roof_assessment 2 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
4052 worker-A get_roof_assessment 2 backoff: 908 ms
5140 worker-A get_roof_assessment 3 attempt ok
5140 worker-A get_roof_assessment - step done: checkpoint saved
end state:
run run-f57d state waiting_for_human resume_count 0 ended at 6958 ms (deadline 120000 ms)
recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
result: 7 of 7 invariants hold
(These are the lines about step 2 and the end state; the full output also lists the other steps, the recommendation row and each invariant.) The naive workflow passes UW-R02 as well, retrying at once, at 4000 ms and 4180 ms. It works here by luck: the provider happened to recover after two calls.
And the run we opened with, UW-R03, with the careful workflow. Three attempts, then the breaker opens, and the step goes on without the roof assessment:
2000 worker-A get_roof_assessment 1 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
2000 worker-A get_roof_assessment 1 backoff: 52 ms
4052 worker-A get_roof_assessment 2 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
4052 worker-A get_roof_assessment 2 backoff: 908 ms
6960 worker-A get_roof_assessment - breaker open: dependency provider
6960 worker-A get_roof_assessment 3 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
6960 worker-A get_roof_assessment - degrade: go on without the roof assessment: request_info
end state:
run run-bef5 state degraded_request_info resume_count 0 ended at 8778 ms (deadline 120000 ms)
recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
REC-0001 pending_review request_info cites none evidence EV-RULE-0001
The underwriter now has an honest recommendation, request_info, with missing: ["roof_assessment"], at 8778 ms instead of nothing at 120000 ms. The retries did not produce that outcome. Bounding them did: they stopped early enough for the workflow to do something useful with the time. The step after the retries, the breaker and the smaller honest result, is the subject of Breakers.
| Scenario | Fault | Careful workflow | Naive workflow |
|---|---|---|---|
UW-R02 | Provider times out twice, then answers | 3 attempts, waits 52 and 908 ms, ends at 6958 ms, 1 row | 3 attempts at once; works, by luck |
UW-R03 | Provider stays down (SUB-1006) | 3 attempts, breaker opens, degraded_request_info at 8778 ms, 1 row | 60 attempts, deadline_exceeded at 120000 ms, no row; breaks INV-4, INV-5 |
UW-R04 | Model answers 429 with retry_after_ms 1500, twice | Waits 1500 ms twice, third attempt ok, ends at 5198 ms | 33 attempts, 32 refused; breaks INV-5 |
8. How retries go wrong even when they are there#
Retries usually go wrong not because they are missing but because one of their limits is. Five ways, each prevented by one line of call_with_retry:
A retry without a budget. Each call is polite, the sum is not. Under a shared failure, three attempts per call triple the load on the dependency that is already failing. The budget is the line that stops it.
Retries at several layers. If the agent retries three times, the HTTP client below it retries three times, and the provider's own gateway retries three times, one failing call becomes 27 calls at the bottom. The Builders' Library article makes the same point with five layers (243 times the load) and the SRE book with three layers (64 attempts). Retry at one layer, and know which libraries under you already retry.
Retrying a permanent error. The poison submission of UW-R11 will fail validation forever. Retrying it inside the step only delays the moment it is set aside. classify returns give_up, and the queue decides what happens next (Queues).
A timeout longer than the time left. A call that starts 500 ms before the deadline with a 2000 ms timeout can finish 1500 ms after it, and its result arrives at a run that has already ended. Deadline propagation is the line timeout_ms = min(policy.timeout_ms, deadline.remaining(clock)).
Retrying a write that may have succeeded. A retry of a read is harmless. A retry of a write, after a timeout or a lost acknowledgement, can write twice. Retries make writes happen more than once; only idempotency makes that safe.
The same idea for a coding agent#
A coding agent that edits a repository, runs the tests and waits for CI has the same two clocks. Each poll of the CI status gets a short timeout; the whole task gets a deadline, and the time left is passed into every tool call, so a test command started near the end cannot outlive the task. A network error from the CI service is transient and worth a jittered retry. A failing assertion is permanent for this commit; re-running it in a loop only hides it. The harness around such an agent is the subject of https://harness.sammuthu.com; the pipeline belongs to https://diff-to-deploy.sammuthu.com.
How frameworks package this#
The tools a team meets here include the model providers' SDKs, libraries such as Resilience4j, and workflow engines such as Temporal and AWS Step Functions. Whether each one retries for you, by default or on request, with which backoff and which limits, is on Frameworks, checked against each tool's own documentation. Whatever you choose, ask one question first: if this library retries and my code also retries, how many attempts does the provider see?
Choosing the timeout from latency, not from a round numberA percentile and an accepted rate of false timeouts
The Builders' Library article gives a method worth stating as a rule: decide what share of healthy calls you are willing to cut off by mistake, then set the timeout at the dependency's latency for that percentile. If you accept 0.1% false timeouts, the timeout sits at the dependency's p99.9 latency (the value that 99.9% of calls stay under), with padding where the network path is long. Percentiles and how to measure them are the subject of SLOs.
Two consequences. A timeout is not a constant: when the dependency changes, its latency distribution changes, and the timeout must be checked again. And the timeout interacts with the deadline: three attempts at the p99.9 timeout plus the backoff waits must fit inside the share of the deadline you are willing to give this step. In this example the arithmetic is 7500 ms of a 120000 ms deadline; a run with ten such steps in sequence would need to budget more carefully.
For a model call the distribution is wide, because output length varies. A timeout set for short answers cuts off long ones. Streaming changes the measure: the time to the first token and the gap between tokens are often better clocks than the total.
Where the retry budget lives, and what it does not coverPer process, per dependency, or per run
The example has one bucket for the whole world, shared by all workers of one story and surviving a simulated crash, because the whole world is one Python process; the README names this as a limit. Real systems choose a scope, and each scope protects something different. A bucket per process, as the SRE book describes, limits what one worker can add during an outage, but a thousand processes each with their own bucket still add a thousand bursts. A bucket per dependency stops a failing provider from draining the budget that the healthy database needs. A cap per run (the example's maximum of three attempts per step) protects the run's deadline, not the dependency.
A budget also does not tell you why it emptied. An empty bucket during an outage is the budget working; an empty bucket on a normal day means something retries far more than it should, and that deserves an alert. The budget converts a slow, expensive failure into a fast, cheap one. It does not make the call succeed. That is why it sits next to the circuit breaker, which stops calling a dependency that keeps failing, and next to graceful degradation, which decides what to deliver instead (Breakers).
Hedged requests, which send a second copy of a slow call before the first has failed, are the opposite trade: they spend load to cut tail latency. They belong with model routing on Fallback, and they need a budget of their own.
Full jitter, equal jitter, and why the first wave still peaksReading the herd output carefully
With full jitter the first retry of each client lands anywhere from 0 to 500 ms, and the herd output shows 45 retries in the first bucket and 85 in the second. Jitter cannot spread a wave wider than its ceiling. The first wave is narrow because the base is small; a larger base spreads it more, at the price of a slower recovery for a single client who hit a one-off error.
Equal jitter keeps half of every ceiling as a fixed pause, so no client comes back almost at once. The herd with equal jitter (not printed above; same call with "equal") puts all 100 first retries between 250 and 500 ms, in one bucket. It is a reasonable choice when a minimum pause matters more than spreading, for example when the dependency needs a moment to restart.
None of this is a measurement of a real dependency. The 100 clients fail at the same instant and every attempt fails; that is a stress picture of the timing, not a model of traffic.
Checklist#
| Check | What to look for in your own system |
|---|---|
| Every call has a timeout | Chosen from the dependency's latency at a stated percentile, not a round number |
| Every run has a deadline | A moment, carried with the run; every timeout is cut to the time left |
| Retries are selective | Transient errors only; permanent errors and unknown errors give up at once |
| Attempts are capped | A maximum per step (3 in this example) |
| Waits back off and are jittered | Ceilings double up to a cap; full or equal jitter |
| A retry budget exists | Retries limited to a share of normal traffic, with a small burst |
Retry-After is honored | The unit is checked: the HTTP header counts seconds |
| One layer retries | You know which libraries below you already retry |
| Writes are retried only with an idempotency key | See Idempotency |
| Each attempt is visible | One span per attempt with attempt_number, inside the run's single trace |
In one breath#
Give every call a timeout and every run a deadline, and never hand a call more time than the run has left. Retry only errors that can pass, wait longer each time with jitter so failing clients do not return together, and cap retries with a budget so the system never multiplies its own load. When a provider says when to come back, come back then. And remember that a retry is not a recovery: it repeats an attempt, and it cannot tell whether the last one already changed something.
The careful workflow now survives a provider that times out, a provider that stays down and a model that rate-limits, inside its deadline and inside its budget. One question is still open, and it is the dangerous one: step 6 writes a row, and when that write times out the agent cannot know whether the row exists. A retry there could leave the underwriter two recommendations for one submission. Idempotency makes that retry safe.