All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources

Reliability for agentic software · what is still true after this fails?

Every step can fail. The outcome still has to be acceptable

Reliability means a system keeps delivering acceptable outcomes despite failures, retries, load and changing dependencies.

For engineers and architects who run agents that call models, call tools and write to systems that remember. One running example, fifteen fault scenarios, and code that runs without a model, a network or an API key.

You will understand
Why a quote run that dies in the middle and is tried again can leave two recommendations behind.
You will be able to say
What reliable means for an agent, in one sentence, and why a retry is not a recovery.
You will know
The insurance example that every page breaks and repairs, and which page answers which question.

An underwriter opens the review queue in the morning. Two tasks are waiting for the same building, a masonry office with the reference SUB-1004. Both say the same thing: the application gives the roof age as 9 years, an outside provider says 22, so a senior underwriter should settle it. The two recommendations were written 30 seconds apart, by two different processes, from one quote request.

Nobody pressed a button twice. Here is what happened. The software that prepares these recommendations saved the first one to its database. A fraction of a second later, before it could note that this step was done, its process crashed. The quote request was still on the queue, so after 30 seconds another process picked it up. That process had no record of the first one's progress. It started from the beginning, did every step again, and saved a second recommendation.

Each component did what it was built to do: the queue redelivered an unconfirmed request, the second process did work that looked undone, the database stored what it was given. The result is still wrong: one request, two recommendations, and an underwriter who now has to work out which one counts. In the example on this site this is scenario UW-R08, and its naive version ends exactly like this.

end state:
  run run-3708  state running  resume_count 0  not finished (deadline 120000 ms)
  run run-d159  state waiting_for_human  resume_count 0  ended at 31998 ms (deadline 150000 ms)
  recommendation rows 2 (pending_review 2, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0

Here the two recommendations happen to agree, because the agent in the example is a scripted stand-in. With a real model the second one could say something different from the first. The underwriter would then hold two answers to one question and no rule for choosing between them.

What "reliable" means on this site#

The word reliable often gets read as "never fails". That target is wrong in two ways. Every dependency of an agent fails sometimes: a provider times out, a model refuses for a moment, a process dies during a deployment. And chasing zero failures costs more with every step closer, a point the Google SRE book makes at length in its chapter on embracing risk. This site uses a different definition, and it is the first sentence of every page that follows.

Reliability means a system keeps delivering acceptable outcomes despite failures, retries, load and changing dependencies.

The key word is acceptable outcome: an end state the insurer can live with. For the quote above, three end states are acceptable. One stored recommendation waits for an underwriter. Or the system says honestly that it could not get the roof data and asks for it. Or the work stops cleanly and every partial write is marked as withdrawn. Two recommendations for one request are not acceptable, and neither is a request that vanished without a trace, nor a recommendation that cites evidence the run never fetched.

That definition turns reliability into a question you can ask of any design and any incident: what is still true after this fails? If the provider times out three times, is there still exactly one recommendation? If the process crashes after the write, does the underwriter still get one task? Every page of this site asks that question of one mechanism.

Why a retry is not a recovery#

The first fix most engineers reach for is a retry: if a call fails, call it again. For the opening story that is not enough, and the reason is worth stating carefully.

A retry repeats one step of the work after an error. It answers the question "can this call succeed if I try again?" A recovery has a larger job. It must bring the whole piece of work to an acceptable end state, and to do that it must know what already happened. In the opening story the second process did retry, in a sense: it did the whole job again. What it lacked was knowledge. It did not know that the recommendation had already been saved, so it saved another.

There is a second trap inside the same story. Suppose the save had not crashed the process but had timed out: the database took too long to answer, and the caller gave up waiting. Did the row get written? The caller cannot tell. The write may have failed before the database committed it, or it may have committed and the answer may have been lost on the way back. A timeout tells you nothing about whether the write happened. A retry that ignores this repeats a write that may already be done.

So the site keeps three sentences exact wherever they appear: A retry is not a recovery. A timeout tells you nothing about whether the write happened. The model recommends; a person decides.

Before AI: a bank transfer and a Kafka consumer#

None of this is new. Engineers who built payment systems or event pipelines met these problems long before agents existed, and the answers they found carry over.

Take a bank transfer from a mobile app. The customer taps "send", the network drops for a moment, and the app shows an error. Did the money move? The app cannot know from the error alone. If it sends the transfer again without care, the customer may pay twice. Payment systems handle this by giving the transfer a reference when the customer first taps "send" and sending the same reference on every repeat. The bank stores the reference with the result, so a repeat finds the stored result and moves no money. That pattern has a name, the idempotency key, and it has its own page here.

Now take a consumer that reads records from Apache Kafka and writes each one to a database. The Kafka documentation describes two orders of work. If the consumer saves its position first and then processes, a crash in between loses records: at-most-once delivery. If it processes first and then saves its position, a crash in between means the next process starts from the older position and handles some records a second time: at-least-once delivery. The documentation adds that a second processing is harmless when the write is keyed, because writing the same record again replaces it with an identical copy (Kafka 4.4 design: message delivery semantics). The opening story is the second case. The fix is the same kind of fix: make the second write recognize the first.

Before AI. An at-least-once consumer that writes by primary key, a payment request with a client reference, a database transaction that commits the work and the progress marker together: the patterns on this site are these patterns. What changes for an agent is how long the work runs, what a repeat produces, and how much each repeat costs.

What is different for an agent#

Three properties make an agent harder to keep reliable than the consumer above. Each one makes the opening story more likely, or more damaging.

Agent runs are long. A Kafka consumer handles one record in milliseconds. An agent run calls a provider, retrieves documents, calls a model that takes seconds to answer, and writes to two systems. In the example a run has eight steps, and the anchor scenario takes 30,024 ms on the simulated clock from the first delivery to the end. Every step is a place where a process can die, and the longer the run, the more of them it passes. Starting over is also expensive: every earlier step runs again.

The model is probabilistic. The same input can give a different output on a second call. For ordinary code, running a step again reproduces its result, so a repeat is at worst wasteful. For a model, running the step again produces a new answer, which may disagree with the one already stored. A re-run is not a replay. That is why the opening story is worse with a real model than with the stand-in: the two recommendations could point in different directions.

Agents have side effects. A side effect is a change outside the run that stays after the run ends: a row in a table, a task in an underwriter's queue, an email to a broker. Reads can be repeated freely. Side effects cannot, unless the system on the other end can recognize a repeat. The more tools an agent can call, the more of its steps are side effects.

PropertyPre-AI serviceAgent runWhat it means for reliability
LengthMilliseconds, one or two callsSeconds to minutes, many callsMore places to fail; starting over repeats a lot of work
Repeat of a stepSame resultPossibly a different resultA re-run can contradict what is already stored
Side effectsA few, well knownAny write tool the model may callEvery write must be safe to repeat, or never repeated
Cost of one attemptSmallA model call, counted in tokensRetries and re-runs are paid for; budgets matter

The last row is a fourth difference that sits underneath the other three. A model call is paid for, and in the example it is counted in cost units (synthetic; no money). In scenario UW-R05 the careful workflow asks the model a second time after a broken answer, and the run spends 2,680 cost units in place of 1,340: the broken answer was paid for too.

The insurance example in one minute#

Every page uses one example, so here it is in full, once. All of it is synthetic.

An insurer sells property insurance for commercial buildings through brokers. A broker's quote request arrives as an email. An intake step turns the email into a submission: one structured record that every later step reads. In the scenario the record has 63 fields; the example code carries the ten below.

{
  "submission_id": "SUB-1004",
  "occupancy": "office",
  "construction": "masonry",
  "year_built": 2001,
  "area_sq_ft": 20000,
  "building_value_usd": 2400000,
  "sprinklered": true,
  "roof_age_years": 9,
  "wind_zone": "C",
  "prior_losses_5y": 0
}

The intake step puts the submission on a queue. The underwriting agent (short: the agent) takes it from the queue and works through eight fixed steps. It validates the submission, asks an outside roof-data provider for a roof assessment, retrieves passages from the insurer's guidance (an illustrative rule set called the policy fixture), runs the eligibility rules as ordinary code, and has a model draft a recommendation. Then it writes two things: the recommendation row, and a review task in the underwriter cockpit, the web app where people work. The last step waits for a person.

Diagram: The eight steps of one run · reads, the model, two writes, a person

Read left to right. Steps 1 to 4 only read, so repeating them changes nothing. Step 5 is the model: a repeat can give a different answer. Steps 6 and 7 write, so they are the steps a careless retry can double. Step 8 belongs to a person: the model recommends; a person decides.

  1. Read: safe to repeat
    1. 1 · Load the submission: Validates the submission taken from the queue. A submission that never validates is sent aside for a person.
    2. 2 · Get the roof assessment: Asks the outside roof-data provider about the roof. A read: it changes nothing, so a retry is harmless.
    3. 3 · Search the guidance: Retrieves passages from the illustrative policy fixture that bear on this building.
    4. 4 · Check eligibility: Runs the eligibility rules as ordinary code. The same facts always give the same result.
  2. Model: a repeat may differ
    1. 5 · Draft the recommendation: A model writes the recommendation with reasons and citations, and the output is checked before use. Here a scripted stand-in plays the model.
  3. Write: side effects
    1. 6 · Record the recommendation: Stores the recommendation row. A change that stays after the run: repeating it carelessly writes a second row.
    2. 7 · Open the review task: Opens a task for the underwriter in the web app where people work. Also a side effect.
  4. Decide
    1. 8 · Wait for the underwriter: The automated part ends. The run waits in the state waiting_for_human until a person decides.

Read the diagram by lane. The first four steps only read; repeating them wastes time but changes nothing. Step 5 is the model, and the one place where a repeat can give a different answer. Steps 6 and 7 are the side effects, so they are the steps a careless retry can double. Step 8 belongs to a person. One execution of all eight steps for one submission is a run, and it has its own id.

For SUB-1004 the provider says the roof is 22 years old while the application says 9. The policy fixture has a rule for that: R-ROOF-03 says that when the two ages differ by more than 5 years, a senior underwriter must settle it. So the stored recommendation is this:

{
  "recommendation": "escalate",
  "reasons": [
    "The application says the roof is 9 years old. The roof-data provider says 22 years.",
    "R-ROOF-03: Roof age on the application and from the provider differ by more than 5 years: escalate to a senior underwriter.",
    "Next step: escalate to a senior underwriter to settle the roof age."
  ],
  "citations": ["R-ROOF-03"],
  "missing": [],
  "stated_confidence": 0.8,
  "requires_human": true,
  "policy_version": "2026.2"
}

The recommendation is advice for a human underwriter, the person at the insurer who decides whether to accept a risk. The model recommends; a person decides. The agent never approves, declines or quotes anything on its own.

Four identifiers carry most of the story, and they are never mixed up. Each has one job.

IdentifierExampleWhat it names
message_idmsg-51c2One quote request as it sits on the queue. A second delivery of the same request has the same id
run_idrun-7f3aOne run of the workflow for a submission. It is looked up by message_id, so a second delivery finds the same run
attempt_number1, 2, 3One try of one step inside a run. It grows with every retry
idempotency_keyrun-7f3a:record_recommendationOne intended write, built from run_id and the step name. It never contains the attempt number

The example also defines fifteen fault scenarios, UW-R01 to UW-R15. Each one is a case plus a fault plan (which call fails, how, how many times), and each runs in two versions of the workflow. The careful workflow has every protection the site teaches. The naive workflow has none: it tries again at once, writes without a key, keeps no progress record. After every run, seven invariants are checked: statements that must be true whatever failed on the way, such as "at most one pending recommendation per submission".

Honest boundary. This is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider, every timing and every number are invented. Nothing here reports the reliability or the quality of a real system or model.

Three faults, one outcome: the anchor scenario#

The opening story had one fault. The scenario the whole site builds towards, UW-R15, has three in the same run. The roof-data provider times out twice. Then the process crashes right after the recommendation is saved and before it records that step as done. Then the queue delivers the request once more after the run has finished. The question is the site's question: is there still exactly one recommendation and one review task at the end?

The command below runs UW-R15 with the careful workflow, in a copy of examples/uw-reliability/. What a successful run shows: the end state and all seven invariants after the three faults. What it does not show: how often a real provider times out or how a real model behaves. The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

cd examples/uw-reliability
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R15
Scenario UW-R15 · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
The anchor: provider times out twice, the process crashes after the write, the queue redelivers
case UW-004 · submission SUB-1004 · message msg-51c2 · run run-7f3a
fault plan:
  get_roof_assessment          timeout       on call 1, 2
  process                      crash         on call 1  (after_step record_recommendation, when before_checkpoint)
  queue                        duplicate     on call 1
timeline:
   t_ms  worker    step                   att  event
      0  queue     -                        -  message sent: msg-51c2
      0  worker-A  -                        -  message received: msg-51c2, receive 1
      0  worker-A  -                        -  run created: trace_id 3e00680d22530c83b2c9332e533e599c
      0  worker-A  -                        -  lease granted: fencing token 41
      0  worker-A  load_submission          -  step done: checkpoint saved
   2000  worker-A  get_roof_assessment      1  attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
   2000  worker-A  get_roof_assessment      1  backoff: 52 ms
   4052  worker-A  get_roof_assessment      2  attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
   4052  worker-A  get_roof_assessment      2  backoff: 908 ms
   5140  worker-A  get_roof_assessment      3  attempt ok
   5140  worker-A  get_roof_assessment      -  step done: checkpoint saved
   5380  worker-A  search_guidance          1  attempt ok
   5530  worker-A  search_guidance          -  step done: checkpoint saved
   5534  worker-A  check_eligibility        -  step done: checkpoint saved
   6934  worker-A  draft_recommendation     1  attempt ok
   6934  worker-A  draft_recommendation     -  step done: checkpoint saved
   6946  worker-A  record_recommendation    1  attempt ok
   6946  worker-A  record_recommendation    -  PROCESS CRASH: after the step, before checkpoint
   6946  worker-A  -                        -  process is gone: no acknowledgement; visible again after 30000 ms
  30000  worker-B  -                        -  message received: msg-51c2, receive 2
  30000  worker-B  -                        -  run found: by message_id; resume_count 1
  30000  worker-B  -                        -  lease granted: fencing token 42
  30000  worker-B  draft_recommendation     -  resume from checkpoint: steps 1 to 5 are not run again
  30012  worker-B  record_recommendation    1  attempt ok
  30012  worker-B  record_recommendation    -  replay: same key: the stored row was returned, nothing was written
  30012  worker-B  record_recommendation    -  step done: checkpoint saved
  30024  worker-B  open_review_task         1  attempt ok
  30024  worker-B  open_review_task         -  step done: checkpoint saved
  30024  worker-B  await_underwriter        -  run ended: waiting_for_human
  30024  worker-B  await_underwriter        -  step done: checkpoint saved
  30024  worker-B  -                        -  acknowledgement lost: the queue will deliver the message again
  30024  worker-C  -                        -  message received: msg-51c2, receive 3
  30024  worker-C  -                        -  duplicate delivery: run-7f3a already ended in waiting_for_human: nothing to do
  30024  worker-C  -                        -  acknowledged
end state:
  run run-7f3a  state waiting_for_human  resume_count 1  ended at 30024 ms (deadline 120000 ms)
  recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
  REC-0001  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-7f3a  route primary  degraded false  fencing_token 41  key run-7f3a:record_recommendation
expected (careful): 10 of 10 values match
invariants:
  INV-1  pass  holds
  INV-2  pass  holds
  INV-3  pass  holds
  INV-4  pass  holds
  INV-5  pass  holds
  INV-6  pass  holds
  INV-7  pass  holds
result: 7 of 7 invariants hold
wrote out/UW-R15-careful.json

Read the timeline for which mechanism handled which fault. The two timeouts at 2,000 ms and 4,052 ms were handled by a retry with a growing wait between attempts; attempt 3 got the roof assessment. That is the only fault a retry fixed. The crash at 6,946 ms was handled by three other things. The second process found the existing run by message_id in place of creating a new one. It found the saved progress record after step 5, so it did not call the provider or the model again. And when it repeated step 6, the write carried the same key as before, so the store returned the row it already had: "replay, nothing was written". The third delivery found a finished run and did nothing.

The naive workflow meets the same three faults. Its retries also get the roof assessment. Then the crash sends it back to the start, twice, and it ends like this.

UW-R15Careful workflowNaive workflow
Runs created1 (run-7f3a)3 (run-7f3a, run-85a7, run-9e62)
Recommendation rows1 (REC-0001)3
Review tasks12
End state of run-7f3awaiting_for_humanleft in running, never finished
Invariants that hold7 of 75 of 7 (INV-1 and INV-4 broken)

Diagram: UW-R15 twice · both workflows retry, only one recovers

Each lane is one workflow facing the same three faults on submission SUB-1004. Read a lane left to right. The first column is the same in both lanes: the retry gets the roof assessment. From the crash on, the lanes split, and the difference is what each workflow knows about its own past.

  1. Naive workflow
    1. Retries at once: The naive workflow tries again immediately, with no wait. Here the third attempt succeeds, by luck.
    2. Starts over: The next process finds no record of the first run, creates a new one and does all eight steps again.
    3. Starts over again: The extra delivery also looks like new work, so a third run writes a third row.
    4. 3 rows, 2 tasks: Three recommendations for one request, and the first run is left in the state running forever.
  2. Careful workflow
    1. Retries with backoff: The careful workflow waits a growing, random time between attempts, within limits.
    2. Resumes the same run: The next process finds run-7f3a by its message id, skips steps 1 to 5 from the saved progress, and repeats step 6 with the same key. The store returns the stored row.
    3. Finds a finished run: The third delivery finds run-7f3a already waiting for a person and only confirms the message.
    4. 1 row, 1 task: One recommendation, escalate citing R-ROOF-03, waiting for the underwriter.

Both workflows retried. Only one recovered. The difference is not how hard each one tried. It is what each one knew about its own past: which run this delivery belongs to, which steps are done, and which write has already happened.

Try it: the game day#

The lab below is a small version of a game day, the practice of breaking a system on purpose to see whether its recovery works. It runs submission SUB-1004 with five faults you can switch on and four protections you can switch off. The anchor preset switches on the two faults of UW-R15 that the lab offers: the provider times out twice, and the process crashes after the write (the queue then delivers the request again by itself, because nobody confirmed it). All four protections start on. Switch them off one at a time and watch which invariants break. Every combination is a real run of the example, computed in advance.

If the lab does not load, this table holds the same results. The first three rows come from the lab's own function (run_scenario.py:gameday, seed 7), run in a copy of the example; in the lab, inbox deduplication and the other protections not offered as switches stay on. The last row is the full naive workflow of UW-R15 from the command line, with all ten protections off and the extra duplicate delivery.

Faults onProtections onEnd stateRecommendation rowsReview tasksInvariants that fail
Provider timeout twice, crash after the writeAll four: retry with backoff, output validation, idempotency key, checkpointwaiting_for_human at 30,024 ms11none
The same twoAll except the idempotency keywaiting_for_human21INV-1
The same twoAll except the checkpointwaiting_for_human at 31,998 ms; provider called 4 times, model twice11none
UW-R15 on the command line, naiveNone of the tenrun-7f3a left in running; two more runs end in waiting_for_human32INV-1, INV-4

Read the second and third rows together. Without the key, the resumed process repeats step 6 and the store cannot tell it is a repeat, so a second row appears. Without the checkpoint, the resumed process starts at step 1 again and calls the provider and the model a second time; the key still stops the second row, but the run pays for every step twice (2,680 cost units in place of 1,340). Each protection covers a different hole, which is why the pages that follow teach them one at a time.

The same idea for a coding agent#

The second, smaller example of this site is a coding agent that edits a repository, runs the tests, pushes a branch, opens a pull request and waits for CI. Its version of the opening story: the agent asks the hosting service to open the pull request, the call times out, and the agent tries again. Now there are two pull requests for one change, and two reviewers may each review one of them. The fix has the same shape as in the insurance example. Before creating, look up whether a pull request for this branch already exists, and treat "already exists" as success. The software around the model loop, where such a check lives, is the subject of Harness Engineering; shipping the change through review and CI is the subject of Diff to Deploy.

What this site is not#

A map to adapt, not a standard, and not a scoreboard. The page order and the words are one way of dividing the subject. The agent in the example is a scripted stand-in and every fault is injected, so no result here says how reliable a model, a provider or a product is. Tools are described on the frameworks page, from their own documentation with the date it was read, and none is called best.
What this is not. This site does not teach how a failure is seen in logs, metrics and traces; that is Observability Engineering. It starts once the failure is seen.
What this is not. Deciding which step runs next is the control plane of Agent Orchestration Engineering. This site owns what happens when a step fails.
What this is not. Measuring whether recommendations got worse after a change is Eval Engineering. A recommendation can be reliable here (one row, stored, waiting for a person) and still be wrong there.
Why "acceptable outcome" and not "success"The definition is about end states, and it has to include honest failure

A run that "returned something" is not the unit that matters. The naive workflow in UW-R05 returns at 1,998 ms with a stored row, and the row holds text that is not a recommendation at all. Counting it as a success would hide the worst outcome in the table. The opposite error is also common: counting every non-happy end as a failure. When the provider stays down (UW-R03), the careful workflow stores request_info with missing: ["roof_assessment"] and the run ends in degraded_request_info. Nothing was approved on missing data, the underwriter knows what is missing, and the work can continue. That is an acceptable outcome, even though the happy path did not happen.

So the set of acceptable outcomes is a design decision, written down before an incident: here, the four terminal states waiting_for_human, degraded_request_info, failed_compensated and dead_lettered, each with its row counts. Everything else, including "still running after the deadline", is unacceptable. The page on service level objectives turns that set into a measurement.

What the example leaves outOne process, simulated time, a scripted model

The example runs in one Python process on a simulated clock. "Two workers at once" means that while one worker stalls, the simulation lets the clock run and hands the message to the next. There is no real network, so a timeout is a number added to the clock, with no partial responses and no slow connections. The store is lists in memory, with no isolation levels and no replication lag, so "the key and the row are written in the same commit" is true by construction. The stand-in model writes what the rules imply; it cannot be wrong in the many ways a model can. The breaker and the retry budget are shared by all workers of one story and survive a simulated crash, where a real system would start each process with its own.

These limits are deliberate. They make every result repeatable from a seed, so a reader can rerun a scenario and see the same timeline. They also mean a passing suite here is evidence about the mechanisms, and only about the faults the example knows how to inject. A real system needs the same scenarios run against its real queue, store and providers.

Where this fits#

This site is one of a series on building with agents. Each site owns one part of the subject and links to the others where they meet, in place of repeating them. Ten sites are being written together; five more are published.

The ten sites written together (this one is marked).

Published sister sites.

Three borders matter most for this site. Memory decides what state is worth keeping; this site uses saved state to recover. Orchestration decides which step runs next; this site decides what happens when a step fails. Observability makes a failure visible; this site decides what to do once it is visible, and owns the arithmetic of service level objectives.

How to read this site#

The pages climb in one order. Two pages give words for the problem (four qualities, then failure modes). Six pages are mechanisms, each answering a failure the previous ones left open: retries, idempotency, queues, checkpoints, circuit breakers, concurrency limits. Five pages are about running the system: service level objectives, deployment, model fallback, fault injection, incidents. Reference pages close it: frameworks, advanced topics, and the full insurance lab with every scenario and its code.

Each chapter page follows the same ladder: a moment from the insurance example where the problem bites, the idea in plain words, the smallest working piece of code from the example with its real output, an interactive lab, what goes wrong when the mechanism is missing or misused, and a checklist. Depth for experienced readers sits in folded blocks like the two above, so the first reading stays short. Terms with a dotted underline open a definition.

If your question isStart at
Is "it answered" the same as "it worked"?Four qualities
Which failures should I retry, and which never?Failure modes, then Retries
How do I stop a retry from writing twice?Idempotency
How does a long run survive a crash?Checkpoints
How reliable is reliable enough, in numbers?SLOs
How do I know my recovery path works before production tests it?Fault injection
Where is the whole example with its code?The insurance lab

Checklist#

Questions to ask of any agent workflow before it touches a real request.

QuestionA good answer looks like
Have you written down the acceptable end states?A short list of terminal states, each with the rows it leaves
Which steps are side effects?Every write named, with how a repeat of it is recognized
What happens if the process dies after a write?A new process finds the same run and the same saved progress
What does a timeout on a write mean in your code?"Unknown", never "failed"; the retry carries the same key
Can a second delivery start a second run?No: the run is looked up by the message id
Which invariants does every test run check?At least "one recommendation per request" and "every run ends in a named state"

In one breath#

Reliable does not mean nothing fails. It means the outcome stays acceptable when things do fail: one recommendation waiting for a person, an honest request for missing data, or a clean stop. A retry gets one call through; it does not tell you what already happened, and a timeout on a write leaves that question open. Recovery comes from knowing your own past: which run this is, which steps are done, and which write already landed.

You now have the site's definition, its question and its example, and you have seen one run survive three faults. What you do not have yet are precise words for what went wrong in each case: "the system answered", "the answer was right" and "the answer was kept" are three different claims, and the opening story broke only one of them. Four qualities separates them, one incident each.