All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Reference · 16

The insurance lab: one quote, fifteen ways to fail, end to end

The whole example in one place: the workflow, the fault plans, the retries, the resume, the single stored recommendation, and the fault-injection suite that proves it, with the code.

What you will understand
One quote, end to end
The eight steps of the workflow, the ten protections around them, and the rows a run leaves in the store.
The anchor, in full
UW-R15: three faults in one run, and the 30,024 ms of simulated time in which the careful workflow still ends with one recommendation.
All fifteen, proved
Every scenario with its careful and naive outcome, the seven invariants, the 295-run suite, and where each concept lives in the code.
Honest boundary. This is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider, every timing and every number are invented. Nothing here reports the reliability or the quality of a real system or model.

Every chapter took one fault out of the insurance example and repaired it; this page puts it back together. A broker's quote request for submission SUB-1004 arrives as a message on a queue: an office building, masonry, built 2001, sprinklered, in wind zone C, whose application says the roof is 9 years old. The roof-data provider says 22. Under the policy fixture (UW-GUIDE version 2026.2, illustrative) rule R-ROOF-03 applies: the two ages differ by more than 5 years, so the underwriting agent should recommend escalate. The model recommends; a person decides.

In scenario UW-R15 three things go wrong on the way. The provider times out twice. The worker process crashes right after it stored the recommendation, before it could record that it had. And the queue, never told that the work was done, delivers the same message again, twice in all. Each fault on its own was the subject of a chapter. Together they are the day the design was made for.

What is still true after this fails? One run, one recommendation row, one review task, and a person deciding.

The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

The workflow and its identifiers#

The workflow has eight steps, always in this order, with fixed step ids. A workflow step is one unit of that sequence: it has an id, an input from the steps before it, at most one kind of side effect, and a rule for what happens when it fails. Tiny example: step 6, record_recommendation, writes one row and is retried with the same idempotency key.

#Step idWhat it doesSide effectIf it fails
1load_submissionValidates the submission from the queue messagenonePermanent: the message is poison and ends in the dead-letter queue
2get_roof_assessmentCalls the roof-data providernoneRetry with backoff inside the deadline; breaker; then go on with request_info, missing: ["roof_assessment"]
3search_guidanceRetrieves passages of the policy fixture and similar past decisionsnoneRetry; without guidance nothing is drafted and the message goes back to the queue
4check_eligibilityRuns the rules (code)noneDeterministic: a failure here is a bug, not a fault
5draft_recommendationThe stand-in model writes the recommendation; the output is validatednone429: wait retry_after_ms. Malformed: ask again, at most twice, then a code-written escalate. Primary route down: fallback route, marked degraded
6record_recommendationStores the recommendation rowwriteRetry with the same idempotency_key; a replay returns the stored row
7open_review_taskOpens the task in the underwriter cockpitwriteRetry with its own key; a permanent rejection compensates step 6 (row marked withdrawn)
8await_underwriterThe automated part is overnoneThe run sits in waiting_for_human

Diagram: The eight steps of one run · reads first, the model in the middle, two writes, then a person

Read left to right. Steps 1 to 4 read and compute: repeating them costs time, nothing else. Step 5 is the model, a scripted stand-in here, whose answer must be saved, never asked for twice. Steps 6 and 7 are the only writes and carry idempotency keys; step 7 can compensate step 6. Step 8 hands the run to a person. A checkpoint is saved after every step.

  1. Read and compute
    1. 1 · load_submission: Validates the submission taken from the queue message. A submission that never validates is a poison message.
    2. 2 · get_roof_assessment: Calls the roof-data provider with a 2000 ms timeout, retries with backoff, and can degrade to request_info.
    3. 3 · search_guidance: Retrieves passages of the policy fixture and similar past decisions.
    4. 4 · check_eligibility: Runs the deterministic rules. A failure here is a bug, not a fault.
  2. The model
    1. 5 · draft_recommendation: The stand-in model writes the recommendation; the output is parsed and checked before it is used. The answer goes into the checkpoint.
  3. Two writes
    1. 6 · record_recommendation: Stores the recommendation row. A retry or a resume sends the same key and gets the stored row back.
    2. 7 · open_review_task: Opens the task in the underwriter cockpit. A permanent rejection marks the row of step 6 withdrawn.
  4. A person
    1. 8 · await_underwriter: The automated part ends. An underwriter reads the task and decides; a different decision is an override.

Three nouns from the table need a sentence each. The policy fixture is the invented rule book the agent reads: UW-GUIDE version 2026.2, effective 2026-07-01, with an earlier version 2025.4 in which the roof limit of R-ROOF-01 was 30 years instead of 25. The underwriter cockpit is the web application where underwriters see their work. A review task is the item in that cockpit that puts one stored recommendation in front of an underwriter; without it, the recommendation is in the table and nobody sees it.

The run carries identifiers that each have one job. In UW-R15 they take these values:

IdentifierValue in UW-R15What it names
message_idmsg-51c2The quote request as it sits on the queue; the same on every redelivery
submission_idSUB-1004The broker's request as a record
run_idrun-7f3aThis execution of the workflow, found by message_id
attempt_number1 to 3 on step 2One try of one step
resume_count0, then 1How many times a new process picked the run up
idempotency_keyrun-7f3a:record_recommendationThe one intended write of step 6; no attempt number, no timestamp
trace_id3e00680d22530c83b2c9332e533e599cOne trace for the whole run, kept across the resume
fencing_token41, then 42Grows with each lease; a write with an old token is refused
evidence_idsEV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007The evidence the recommendation rests on

Around the steps sit ten protections, each a flag in workflow.py:FLAGS. The careful workflow switches all ten on; the naive workflow switches all ten off.

FlagOnOffChapter
retry_backoffBounded attempts, backoff with jitter, retry budget, deadline-aware timeoutsTries again at once, with no cap, until the deadline is goneRetries
validate_outputThe model's text is parsed and checked before useWhatever came back is storedFailure modes
idempotency_keyWrites carry <run_id>:<step id>Writes carry no keyIdempotency
checkpointState saved after every step; a new process resumesA new process starts at step 1Checkpoints
dedup_inboxThe run is found by message_id; one run per messageEvery delivery creates a new runQueues
breakerA dependency that keeps failing is not called for a whileEvery call is sentBreakers
compensationA rejected review task withdraws the recommendationThe recommendation stays with no taskCheckpoints
fencingThe store refuses a write with an old fencing tokenThe store cannot tell a lease was lostConcurrency
fallback_routePrimary model route, then fallback routePrimary route onlyFallback
dead_letterAfter 5 receives the message moves to the dead-letter queueThe message comes back until the simulation stops itQueues

The anchor: UW-R15, step by step#

The fault plan of UW-R15 is three rules in its scenario file. The dry run prints it together with the steps and the protections, and runs nothing:

/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R15 --dry-run
Scenario UW-R15 · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
The anchor: provider times out twice, the process crashes after the write, the queue redelivers
case UW-004 · submission SUB-1004 · message msg-51c2 · run run-7f3a
dry run: nothing is executed
fault plan:
  get_roof_assessment          timeout       on call 1, 2
  process                      crash         on call 1  (after_step record_recommendation, when before_checkpoint)
  queue                        duplicate     on call 1
steps:
  1 load_submission
  2 get_roof_assessment
  3 search_guidance
  4 check_eligibility
  5 draft_recommendation
  6 record_recommendation
  7 open_review_task
  8 await_underwriter
protections: retry_backoff on, validate_output on, idempotency_key on, checkpoint on, dedup_inbox on, breaker on, compensation on, fencing on, fallback_route on, dead_letter on
teaches: Three faults, one outcome: one run, one recommendation row, one review task, waiting for a person. A retry is not a recovery; the key, the checkpoint and the run lookup are.

The command below runs the scenario on the careful workflow in a scratch copy of examples/uw-reliability. A successful run proves that in this simulation the three faults leave exactly one run, one recommendation row and one review task, and that all seven invariants hold. The second tab is the same plan on the naive workflow.

Careful

/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R15
Scenario UW-R15 · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
The anchor: provider times out twice, the process crashes after the write, the queue redelivers
case UW-004 · submission SUB-1004 · message msg-51c2 · run run-7f3a
fault plan:
  get_roof_assessment          timeout       on call 1, 2
  process                      crash         on call 1  (after_step record_recommendation, when before_checkpoint)
  queue                        duplicate     on call 1
timeline:
   t_ms  worker    step                   att  event
      0  queue     -                        -  message sent: msg-51c2
      0  worker-A  -                        -  message received: msg-51c2, receive 1
      0  worker-A  -                        -  run created: trace_id 3e00680d22530c83b2c9332e533e599c
      0  worker-A  -                        -  lease granted: fencing token 41
      0  worker-A  load_submission          -  step done: checkpoint saved
   2000  worker-A  get_roof_assessment      1  attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
   2000  worker-A  get_roof_assessment      1  backoff: 52 ms
   4052  worker-A  get_roof_assessment      2  attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
   4052  worker-A  get_roof_assessment      2  backoff: 908 ms
   5140  worker-A  get_roof_assessment      3  attempt ok
   5140  worker-A  get_roof_assessment      -  step done: checkpoint saved
   5380  worker-A  search_guidance          1  attempt ok
   5530  worker-A  search_guidance          -  step done: checkpoint saved
   5534  worker-A  check_eligibility        -  step done: checkpoint saved
   6934  worker-A  draft_recommendation     1  attempt ok
   6934  worker-A  draft_recommendation     -  step done: checkpoint saved
   6946  worker-A  record_recommendation    1  attempt ok
   6946  worker-A  record_recommendation    -  PROCESS CRASH: after the step, before checkpoint
   6946  worker-A  -                        -  process is gone: no acknowledgement; visible again after 30000 ms
  30000  worker-B  -                        -  message received: msg-51c2, receive 2
  30000  worker-B  -                        -  run found: by message_id; resume_count 1
  30000  worker-B  -                        -  lease granted: fencing token 42
  30000  worker-B  draft_recommendation     -  resume from checkpoint: steps 1 to 5 are not run again
  30012  worker-B  record_recommendation    1  attempt ok
  30012  worker-B  record_recommendation    -  replay: same key: the stored row was returned, nothing was written
  30012  worker-B  record_recommendation    -  step done: checkpoint saved
  30024  worker-B  open_review_task         1  attempt ok
  30024  worker-B  open_review_task         -  step done: checkpoint saved
  30024  worker-B  await_underwriter        -  run ended: waiting_for_human
  30024  worker-B  await_underwriter        -  step done: checkpoint saved
  30024  worker-B  -                        -  acknowledgement lost: the queue will deliver the message again
  30024  worker-C  -                        -  message received: msg-51c2, receive 3
  30024  worker-C  -                        -  duplicate delivery: run-7f3a already ended in waiting_for_human: nothing to do
  30024  worker-C  -                        -  acknowledged
end state:
  run run-7f3a  state waiting_for_human  resume_count 1  ended at 30024 ms (deadline 120000 ms)
  recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
  REC-0001  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-7f3a  route primary  degraded false  fencing_token 41  key run-7f3a:record_recommendation
expected (careful): 10 of 10 values match
invariants:
  INV-1  pass  holds
  INV-2  pass  holds
  INV-3  pass  holds
  INV-4  pass  holds
  INV-5  pass  holds
  INV-6  pass  holds
  INV-7  pass  holds
result: 7 of 7 invariants hold
wrote out/UW-R15-careful.json

Naive

/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R15 --variant naive
Scenario UW-R15 · variant naive · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
The anchor: provider times out twice, the process crashes after the write, the queue redelivers
case UW-004 · submission SUB-1004 · message msg-51c2 · run run-7f3a
fault plan:
  get_roof_assessment          timeout       on call 1, 2
  process                      crash         on call 1  (after_step record_recommendation, when before_checkpoint)
  queue                        duplicate     on call 1
timeline:
   t_ms  worker    step                   att  event
      0  queue     -                        -  message sent: msg-51c2
      0  worker-A  -                        -  message received: msg-51c2, receive 1
      0  worker-A  -                        -  run created: trace_id 3e00680d22530c83b2c9332e533e599c
      0  worker-A  -                        -  lease granted: fencing token 41
      0  worker-A  load_submission          -  step done: no checkpoint
   2000  worker-A  get_roof_assessment      1  attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
   4000  worker-A  get_roof_assessment      2  attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
   4180  worker-A  get_roof_assessment      3  attempt ok
   4180  worker-A  get_roof_assessment      -  step done: no checkpoint
   4420  worker-A  search_guidance          1  attempt ok
   4570  worker-A  search_guidance          -  step done: no checkpoint
   4574  worker-A  check_eligibility        -  step done: no checkpoint
   5974  worker-A  draft_recommendation     1  attempt ok
   5974  worker-A  draft_recommendation     -  step done: no checkpoint
   5986  worker-A  record_recommendation    1  attempt ok
   5986  worker-A  record_recommendation    -  PROCESS CRASH: after the step, before checkpoint
   5986  worker-A  -                        -  process is gone: no acknowledgement; visible again after 30000 ms
  30000  worker-B  -                        -  message received: msg-51c2, receive 2
  30000  worker-B  -                        -  run created: trace_id b57e8a831a9b02ce226a20653624ddad
  30000  worker-B  -                        -  lease granted: fencing token 42
  30000  worker-B  load_submission          -  step done: no checkpoint
  30180  worker-B  get_roof_assessment      1  attempt ok
  30180  worker-B  get_roof_assessment      -  step done: no checkpoint
  30420  worker-B  search_guidance          1  attempt ok
  30570  worker-B  search_guidance          -  step done: no checkpoint
  30574  worker-B  check_eligibility        -  step done: no checkpoint
  31974  worker-B  draft_recommendation     1  attempt ok
  31974  worker-B  draft_recommendation     -  step done: no checkpoint
  31986  worker-B  record_recommendation    1  attempt ok
  31986  worker-B  record_recommendation    -  step done: no checkpoint
  31998  worker-B  open_review_task         1  attempt ok
  31998  worker-B  open_review_task         -  step done: no checkpoint
  31998  worker-B  await_underwriter        -  run ended: waiting_for_human
  31998  worker-B  await_underwriter        -  step done: no checkpoint
  31998  worker-B  -                        -  acknowledgement lost: the queue will deliver the message again
  31998  worker-C  -                        -  message received: msg-51c2, receive 3
  31998  worker-C  -                        -  run created: trace_id 6318c9d4cd8e80819aa606bcd5728913
  31998  worker-C  -                        -  lease granted: fencing token 43
  31998  worker-C  load_submission          -  step done: no checkpoint
  32178  worker-C  get_roof_assessment      1  attempt ok
  32178  worker-C  get_roof_assessment      -  step done: no checkpoint
  32418  worker-C  search_guidance          1  attempt ok
  32568  worker-C  search_guidance          -  step done: no checkpoint
  32572  worker-C  check_eligibility        -  step done: no checkpoint
  33972  worker-C  draft_recommendation     1  attempt ok
  33972  worker-C  draft_recommendation     -  step done: no checkpoint
  33984  worker-C  record_recommendation    1  attempt ok
  33984  worker-C  record_recommendation    -  step done: no checkpoint
  33996  worker-C  open_review_task         1  attempt ok
  33996  worker-C  open_review_task         -  step done: no checkpoint
  33996  worker-C  await_underwriter        -  run ended: waiting_for_human
  33996  worker-C  await_underwriter        -  step done: no checkpoint
  33996  worker-C  -                        -  acknowledged
end state:
  run run-7f3a  state running  resume_count 0  not finished (deadline 120000 ms)
  run run-85a7  state waiting_for_human  resume_count 0  ended at 31998 ms (deadline 150000 ms)
  run run-9e62  state waiting_for_human  resume_count 0  ended at 33996 ms (deadline 151998 ms)
  recommendation rows 3 (pending_review 3, withdrawn 0) · review tasks 2 · decisions 0 · dead letters 0
  REC-0001  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-7f3a  route primary  degraded false  fencing_token 41  key None
  REC-0002  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-85a7  route primary  degraded false  fencing_token 42  key None
  REC-0003  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-9e62  route primary  degraded false  fencing_token 43  key None
expected (naive): 10 of 10 values match
invariants:
  INV-1  FAIL  SUB-1004 has 3 pending_review rows (a duplicate write)
  INV-2  pass  holds
  INV-3  pass  holds
  INV-4  FAIL  run-7f3a is left in state running
  INV-5  pass  holds
  INV-6  pass  holds
  INV-7  pass  holds
result: 5 of 7 invariants hold · broken: INV-1, INV-4
wrote out/UW-R15-naive.json

Read the careful timeline as three acts.

Act one, the provider (0 to 5,140 ms). Worker A receives msg-51c2, creates run-7f3a, takes the lease with fencing token 41 and checkpoints step 1. The roof-data provider gives no answer within the 2,000 ms timeout, twice. Between attempts the worker waits 52 ms, then 908 ms: backoff with full jitter in this example. Attempt 3 answers at 5,140 ms. This is the only act a retry can handle. The naive workflow also gets through it, by luck (UW-R02).

Act two, the crash (5,140 to 30,000 ms). Guidance, the rule check and the draft follow, each checkpointed; the draft is escalate, citing R-ROOF-03. At 6,946 ms worker A stores the recommendation row REC-0001 under the key run-7f3a:record_recommendation, and dies before it can save the checkpoint of step 6. The store has the row. The checkpoint still says step 5. The message was never acknowledged, so it becomes visible again after the queue's visibility timeout of 30,000 ms. No retry loop survives this: the loop died with the process. A retry is not a recovery.

Act three, the recovery (30,000 to 30,024 ms). Worker B receives the message (receive 2). Three mechanisms, each from a different chapter, act in order. The inbox finds run-7f3a by message_id, so no second run is created. The new lease carries fencing token 42. The checkpoint says step 5 is done, so steps 1 to 5 are skipped: no second provider call, no second model call. Step 6 runs again, because worker B cannot know the write committed (a timeout tells you nothing about whether the write happened, and neither does a crash), and sends the same key; the store returns the stored row and writes nothing. Step 7 opens TASK-0001; the run ends in waiting_for_human at 30,024 ms. Then the acknowledgement of that delivery is lost too, the queue delivers a third time, and worker C finds the run already ended: nothing to do.

The naive tab shows the same three faults without those mechanisms. Worker B finds no run for the message and starts run-85a7 at step 1; worker C starts run-9e62. Three recommendation rows wait for review for one submission (INV-1), two review tasks reach the cockpit, and run-7f3a is left in running forever (INV-4).

Diagram: UW-R15 · three faults, three workers, one run, one row

Time runs down the page, in simulated milliseconds. Worker A survives the provider timeouts by retrying, stores the recommendation and crashes before its checkpoint. Worker B gets the redelivered message, finds the same run, skips steps 1 to 5 and replays step 6 by its key. Worker C gets a third delivery and finds the run already ended. The agent is a scripted stand-in; every fault is injected.

  1. Queue to Worker A: Deliver msg-51c2 (receive 1) at 0 ms. Worker A creates run-7f3a and takes the lease with token 41.
  2. Worker A to Roof-data provider: Attempts 1 and 2: no answer. Two timeouts at 2000 and 4052 ms, with backoff of 52 ms and then 908 ms.
  3. Roof-data provider to Worker A: Attempt 3 answers: 22 years (5140 ms). The retry worked. This is the only fault a retry can handle.
  4. Worker A to Store: Steps 3 to 5, checkpoint after each; step 6 writes REC-0001 (6946 ms). The row is stored with key run-7f3a:record_recommendation and fencing token 41.
  5. Worker A: Crash before the checkpoint of step 6. The store has the row; the checkpoint still says step 5. The message was never acknowledged.
  6. Queue to Worker B: Visible again: receive 2 at 30000 ms. After the visibility timeout the same message goes to worker B.
  7. Worker B to Store: Find run by message_id; load checkpoint; skip steps 1 to 5. No second run, no second provider call, no second model call.
  8. Worker B to Store: Step 6 again, same key (30012 ms). Worker B cannot know the write committed, so it sends it again with the same key.
  9. Store to Worker B: Replay: the stored row, nothing written. Still one recommendation row for SUB-1004.
  10. Worker B to Store: Step 7: TASK-0001; run ends waiting_for_human (30024 ms). The acknowledgement of this delivery is lost as well, so the queue delivers once more.
  11. Queue to Worker C: Receive 3 at 30024 ms: run already ended. Worker C finds run-7f3a in waiting_for_human and only acknowledges the message.

The one row the careful run leaves, from out/UW-R15-careful.json:

{"recommendation_id": "REC-0001", "submission_id": "SUB-1004", "run_id": "run-7f3a",
 "idempotency_key": "run-7f3a:record_recommendation",
 "recommendation": {"recommendation": "escalate",
                    "reasons": ["The application says the roof is 9 years old. The roof-data provider says 22 years.",
                                "R-ROOF-03: Roof age on the application and from the provider differ by more than 5 years: escalate to a senior underwriter.",
                                "Next step: escalate to a senior underwriter to settle the roof age."],
                    "citations": ["R-ROOF-03"], "missing": [], "stated_confidence": 0.8,
                    "requires_human": true, "policy_version": "2026.2"},
 "evidence_ids": ["EV-ROOF-0412", "EV-GUIDE-0031", "EV-RULE-0007"],
 "model_route": "primary", "degraded": false, "status": "pending_review",
 "fencing_token": 41, "created_by": "underwriting-agent", "created_ms": 6946}

fencing_token 41 shows worker A wrote it under its lease, before the crash. requires_human: true and status: pending_review mark a recommendation waiting for a person, not a decision. policy_version and evidence_ids say what it rests on (INV-6).

What happens next is outside the example. An underwriter opens TASK-0001, reads the recommendation and decides. If the decision differs from the recommendation, it is an override: the person's decision, recorded with who decided, what and why, as human_decision with override: true. Shared case UW-009 is such a case: the agent's facts support approval, and the underwriter declines because the broker's loss run shows an open claim missing from the application. Only a person writes that record; no function in the example writes a decisions row, and INV-3 checks it.

Try the anchor yourself#

The anchor preset switches on the two faults of UW-R15 that the lab offers. The six protections that are not switches, inbox deduplication among them, stay on, so the lab never creates a second run. Every combination is a real run of the example, computed in advance.

Without JavaScript: the anchor faults (provider timeout twice, crash after the write) with one protection removed at a time, from run_scenario.py:gameday (seed 7) in a copy of the example, and the command-line naive run for comparison.

ProtectionsEnd stateRowsTasksProvider callsModel callsCost unitsInvariants that fail
All four (the preset)waiting_for_human at 30,024 ms11311,340none
Without retry with backoffwaiting_for_human at 30,024 ms11311,340none
Without output validationwaiting_for_human at 30,024 ms11311,340none
Without the idempotency keywaiting_for_human at 30,024 ms21311,340INV-1
Without the checkpointwaiting_for_human at 31,998 ms11422,680none
UW-R15 naive, command line (all ten off, third delivery)run-7f3a left running32INV-1, INV-4

Rows two and three do not make backoff and validation useless: here the immediate retry happens to work and no model fault is on. Row five is the quiet one: every invariant holds and the run pays twice.

All fifteen scenarios#

Each scenario is a shared case plus a fault plan. Unless noted, the submission is SUB-1004. The table comes from running every scenario for both variants in a scratch copy of the example; "ends" is simulated time.

ScenarioFaultCarefulNaiveNaive breaksChapter
UW-R01nonewaiting_for_human at 1,998 ms; 1 row, 1 taskthe samenoneStart
UW-R02Provider times out twice, then answers3 attempts, backoff 52 and 908 ms; ends 6,958 ms; 1 row3 attempts at once; ends 5,998 ms; 1 row (luck)noneRetries
UW-R03Provider stays down (SUB-1006)Breaker opens; degraded_request_info at 8,778 ms; 1 row, missing: ["roof_assessment"]60 attempts; deadline_exceeded at 120,000 ms; 0 rowsINV-4, INV-5Breakers
UW-R04Model answers 429 twiceWaits 1,500 ms twice; ends 5,198 ms33 attempts; ends 5,198 msINV-5Retries
UW-R05Model output malformed onceAsks again; ends 3,398 ms; 2,680 cost unitsStores the broken text; ends 1,998 msINV-3, INV-6Failure modes
UW-R06Write commits, acknowledgement lostRetry is a replay; 1 row; ends 2,062 ms2 rowsINV-1Idempotency
UW-R07Store fails before the commit, onceRetry; 1 row; ends 2,062 msRetry; 1 rownoneIdempotency
UW-R08Crash after the write, before the checkpointResume replays step 6; 1 row; ends 30,024 msNew run: 2 rows; first run left runningINV-1, INV-4Checkpoints
UW-R09Crash after step 2Resume at step 3; 1 provider call; ends 31,818 msNew run; 2 provider calls; first run left runningINV-4Checkpoints
UW-R10Queue delivers twiceSecond delivery finds the run; 1 run, 1 row2 runs, 2 rows, 2 tasksINV-1Queues
UW-R11Poison message5 receives, dead_lettered at 5,000 ms; 0 rows12 deliveries, 12 runs left runningINV-4Queues
UW-R12Cockpit rejects the review task for goodREC-0001 marked withdrawn; failed_compensated at 1,998 mspending_review row with no task; state failedINV-4Checkpoints
UW-R13Primary model route downBreaker, fallback route; degraded: true; ends 5,458 ms; 620 cost units120 attempts; deadline_exceeded; 0 rowsINV-4, INV-5Fallback
UW-R14A worker stalls 45,000 ms past its leaseStale write with token 41 refused; 1 rowBoth write: 2 runs, 2 rowsINV-1Concurrency
UW-R15Provider timeouts, crash after the write, redelivery1 run, 1 row, 1 task; ends 30,024 ms3 runs, 3 rows, 2 tasksINV-1, INV-4this page

The careful workflow always ends in a terminal state: waiting_for_human twelve times (UW-R13 among them, with its fallback result marked degraded), degraded_request_info once, failed_compensated once and dead_lettered once. The naive workflow passes three, UW-R01, UW-R02 and UW-R07, which are exactly the scenarios where an unprotected retry is enough.

The duplicate, the resume and the week#

Three labs replay parts of the anchor in isolation. The first chooses where the failure hits the write and how the key is built.

The same results without JavaScript, from idempotency.py:simulate_writes with two retries after the failure:

Failure pointNo keyrun_id:stepKey with attempt_numberRandom key per callRight key, other payload
Before the commit (UW-R07)1 row1 row1 row1 row1 row
After the commit, acknowledgement lost (UW-R06)3 rows1 row3 rows3 rows1 row, retries refused as conflicts
Crash before the checkpoint (UW-R08)3 rows1 row3 rows3 rows1 row, retries refused as conflicts

The second lab crashes the first process after a step you choose, with checkpoints on or off, and can add a permanent rejection at step 7.

Without JavaScript, the crash after step 6 (in the lab the crash comes after that step's checkpoint), from workflow.py:simulate_resume:

StepCheckpoints on: process 1Checkpoints on: process 2Checkpoints off: process 1Checkpoints off: process 2
1 to 5ranskippedranran again
6 record_recommendationranskippedranreplayed by its key
7 open_review_tasknot reachedrannot reachedran
8 await_underwriternot reachedrannot reachedran
Provider and model calls in total1 and 12 and 2
Endwaiting_for_human, 1 row, 1 taskwaiting_for_human, 1 row, 1 task

With a permanent rejection at step 7 and no crash (UW-R12): steps 1 to 5 run, step 6 ends as compensated, step 7 as failed, and the run ends in failed_compensated with 1 row marked withdrawn and 0 tasks.

The third lab leaves the single run and looks at a week of invented runs against a service level objective.

Without JavaScript, the three weeks at the lab's default settings (objective 95% good runs, a good run under 8,000 ms), from slo.py:summarize and first_alert:

WeekRunsp50p95p99Unacceptable outcomesError budget usedBurn-rate alert
Healthy2,3153,256 ms4,475 ms9,739 ms18 (0.78%)58%does not fire
Provider brown-out2,3153,415 ms17,471 ms29,695 ms35 (1.51%)220%hour 63
Bad deployment2,3153,256 ms4,475 ms9,739 ms87 (3.76%)117%hour 104

The bad deployment hides from latency: its percentiles equal the healthy week's. Only the unacceptable outcome rate, mostly duplicate writes in this profile, shows it. The brown-out is the incident walked through on Incidents.

The seven invariants and the suite#

After every run seven invariants are checked against the rows left in the store. Their reasoning is on Fault injection; here is the list the suite applies.

IdInvariant
INV-1At most one recommendation row with status pending_review per submission, and exactly one when the run ended in waiting_for_human or degraded_request_info
INV-2At most one review task per run
INV-3No decisions row written by the agent; every stored recommendation has requires_human: true
INV-4Every run ends in a named terminal state inside its deadline
INV-5No step exceeds its maximum attempts; the retry budget is never overdrawn
INV-6The stored recommendation carries the current policy_version and only evidence_ids that exist in the run
INV-7A degraded or fallback result says so (degraded: true), and a withdrawn recommendation is still in the table

The suite runs the 15 scenarios, a matrix of the ten shared cases against no fault and seven single faults (80 runs), and 200 random fault plans with seed 7: 295 runs per variant. A successful careful run proves that in this simulation every invariant held in every one of them. The command exits with 1 when any run breaks an invariant, as the naive run does.

Careful

/opt/homebrew/bin/python3 fault_suite.py
Fault-injection suite · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose

Part 1: the 15 named scenarios
scenario  terminal state         rows tasks  expected  broken invariants
UW-R01    waiting_for_human         1     1  match     -
UW-R02    waiting_for_human         1     1  match     -
UW-R03    degraded_request_info     1     1  match     -
UW-R04    waiting_for_human         1     1  match     -
UW-R05    waiting_for_human         1     1  match     -
UW-R06    waiting_for_human         1     1  match     -
UW-R07    waiting_for_human         1     1  match     -
UW-R08    waiting_for_human         1     1  match     -
UW-R09    waiting_for_human         1     1  match     -
UW-R10    waiting_for_human         1     1  match     -
UW-R11    dead_lettered             0     0  match     -
UW-R12    failed_compensated        1     0  match     -
UW-R13    waiting_for_human         1     1  match     -
UW-R14    waiting_for_human         1     1  match     -
UW-R15    waiting_for_human         1     1  match     -

Part 2: the matrix, 10 shared cases x (no fault + 7 single faults). A cell shows the broken invariants.
case    none              provider-timeout  model-429         malformed         db-transient      ack-lost          crash-after-write duplicate        
UW-001  ok                ok                ok                ok                ok                ok                ok                ok               
UW-002  ok                ok                ok                ok                ok                ok                ok                ok               
UW-003  ok                ok                ok                ok                ok                ok                ok                ok               
UW-004  ok                ok                ok                ok                ok                ok                ok                ok               
UW-005  ok                ok                ok                ok                ok                ok                ok                ok               
UW-006  ok                ok                ok                ok                ok                ok                ok                ok               
UW-007  ok                ok                ok                ok                ok                ok                ok                ok               
UW-008  ok                ok                ok                ok                ok                ok                ok                ok               
UW-009  ok                ok                ok                ok                ok                ok                ok                ok               
UW-010  ok                ok                ok                ok                ok                ok                ok                ok               

Part 3: 200 random fault plans (1 to 3 faults each, seeded)
  end states: waiting_for_human 131, degraded_request_info 31, failed_compensated 22, dead_lettered 16
  runs with every invariant holding: 200 of 200

Runs that break each invariant (scenarios / matrix / random):
  INV-1    0 /   0 /   0
  INV-2    0 /   0 /   0
  INV-3    0 /   0 /   0
  INV-4    0 /   0 /   0
  INV-5    0 /   0 /   0
  INV-6    0 /   0 /   0
  INV-7    0 /   0 /   0
  wrong recommendation without any fault: 0 of 10 cases
SUMMARY: 295 runs · 295 pass · 0 fail · every invariant held in every run
wrote out/suite-careful.json

Naive

/opt/homebrew/bin/python3 fault_suite.py --variant naive
Fault-injection suite · variant naive · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose

Part 1: the 15 named scenarios
scenario  terminal state         rows tasks  expected  broken invariants
UW-R01    waiting_for_human         1     1  match     -
UW-R02    waiting_for_human         1     1  match     -
UW-R03    deadline_exceeded         0     0  match     INV-4, INV-5
UW-R04    waiting_for_human         1     1  match     INV-5
UW-R05    waiting_for_human         1     1  match     INV-3, INV-6
UW-R06    waiting_for_human         2     1  match     INV-1
UW-R07    waiting_for_human         1     1  match     -
UW-R08    running                   2     1  match     INV-1, INV-4
UW-R09    running                   1     1  match     INV-4
UW-R10    waiting_for_human         2     2  match     INV-1
UW-R11    running                   0     0  match     INV-4
UW-R12    failed                    1     0  match     INV-4
UW-R13    deadline_exceeded         0     0  match     INV-4, INV-5
UW-R14    waiting_for_human         2     2  match     INV-1
UW-R15    running                   3     2  match     INV-1, INV-4

Part 2: the matrix, 10 shared cases x (no fault + 7 single faults). A cell shows the broken invariants.
case    none              provider-timeout  model-429         malformed         db-transient      ack-lost          crash-after-write duplicate        
UW-001  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-002  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-003  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-004  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-005  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-006  4,5,R             4,5               4,5               4,5               4,5               4,5               4,5               4,5              
UW-007  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-008  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-009  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-010  ok                ok                5                 3,6               ok                1                 1,4               1                

Part 3: 200 random fault plans (1 to 3 faults each, seeded)
  end states: waiting_for_human 81, running 57, deadline_exceeded 47, failed 15
  runs with every invariant holding: 29 of 200

Runs that break each invariant (scenarios / matrix / random):
  INV-1    5 /  27 /  43
  INV-2    0 /   0 /  18
  INV-3    1 /   9 /  26
  INV-4    7 /  17 / 121
  INV-5    3 /  17 /  72
  INV-6    1 /   9 /  26
  INV-7    0 /   0 /   0
  wrong recommendation without any fault: 1 of 10 cases
SUMMARY: 295 runs · 59 pass · 236 fail · see the broken invariants above
wrote out/suite-naive.json
CarefulNaive
Runs that pass295 of 29559 of 295
Scenarios that pass15 of 153 of 15
Random plans with every invariant holding200 of 20029 of 200
Invariant broken most oftennoneINV-4: 7 scenarios, 17 matrix runs, 121 random plans
Broken only in random plansnoneINV-2: 18 runs

Where each concept lives in the code#

The example is standard-library Python in examples/uw-reliability/. The map from concept to code, from its README:

ConceptWhere
Transient, permanent, malformed, crasherrors.py: the classes; errors.py:classify
Fault plan, fault injectionfaults.py:FaultPlan, faults.py:FaultInjector.fire; a scenario's fault_plan
Timeout (one call) and deadline (the run)retry.py:RetryPolicy.timeout_ms, retry.py:Deadline; workflow.py:RUN_DEADLINE_MS
Deadline propagationretry.py:call_with_retry
Exponential backoff, jitterretry.py:backoff_delay, retry.py:schedule
Retry stormretry.py:herd
Retry budgetretry.py:RetryBudget
Retry only what is retryable; Retry-Afterretry.py:call_with_retry, retry.py:retry_delay
attempt_numberretry.py:attempt_once
The retry without protectionretry.py:call_naive
Idempotency keyidempotency.py:make_key
Deduplication of a write, replay, key conflictidempotency.py:IdempotencyStore.execute
Lost acknowledgementstore.py:Store.write
At-least-once, visibility timeout, redeliveryqueue_sim.py:Queue.receive, Queue.ack, Queue.nack
Dead-letter queue, poison messagequeue_sim.py:Queue.receive; world.py:World._note_dead_letters; tools.py:validate_submission
Inbox (one run per message)workflow.py:Worker._find_or_create_run, store.py:Store.insert_run
Checkpointcheckpoint.py:CheckpointStore.save, workflow.py:Workflow._save_checkpoint
Resumeworkflow.py:Workflow.restore, workflow.py:Workflow.execute
Saga, compensationworkflow.py:Workflow.open_review_task, Workflow.compensate_record_recommendation, store.py:Store.withdraw
Terminal statesworkflow.py:TERMINAL_STATES, Workflow.finish, Workflow.await_underwriter
Circuit breaker, fail fastbreaker.py:CircuitBreaker.allow, record, call
Graceful degradationworkflow.py:Workflow.get_roof_assessment
Bounded re-ask of the modelagent_sim.py:validate_output; errors.py:classify; Workflow.draft_recommendation
Token bucket, concurrency limit, cost budgetlimits.py:TokenBucket, ConcurrencyLimit, Budget; Workflow._call_model
Lease, fencing tokenlease.py:LeaseManager.acquire, store.py:Store.check_fencing
Model fallback, degraded markingrouter.py:route_call, agent_sim.py:draft, Workflow.draft_recommendation
One trace per run; a resume reuses ittracing.py:Tracer, workflow.py:Workflow.__init__
Invariantsinvariants.py:check_invariants
Fault-injection suitefault_suite.py:part_scenarios, part_matrix, part_random
Careful and naive; single protectionsworkflow.py:flags_for, workflow.py:FLAGS
Percentile, SLI, error budget, burn rate, alertslo.py:percentile, sli, error_budget, budget_consumed, burn_rate, burn_alert
Canary and rollbackrollout.py:simulate_rollout

TokenBucket and ConcurrencyLimit are not wired into the workflow; they exist for the rate-limit lab and the tests.

To run everything yourself, copy the folder first, because the runner rewrites out/:

cp -R examples/uw-reliability "$TMPDIR/uwr"
cd "$TMPDIR/uwr"
PY=/opt/homebrew/bin/python3
$PY run_scenario.py --scenario UW-R15
$PY run_scenario.py --scenario UW-R15 --variant naive
$PY fault_suite.py
$PY -m unittest

The 77 unit tests check mechanisms; the suite checks outcomes.

What the example leaves out#

The README lists what the simulation leaves out; each item matters when the design moves to production:

Left outConsequence for the reader
Real parallelism"Two workers at once" means one stalls while the clock runs; real races are harder
A real networkA timeout is a number added to the clock; no partial responses, no slow connections
A real databaseThe store is lists in memory: no isolation levels, no replication lag; "key and row in one commit" is trivially true
Real timeThe deadline, the visibility timeout and the lease are compared with a counter
A real modelThe stand-in writes what the rules imply and cannot be wrong the way a model can; its two routes differ in four scripted ways
Per-process stateThe breaker and the retry budget are shared by all workers of one story and survive a crash
The human stepThe run ends in waiting_for_human; approval timeouts and escalation are discussed on Advanced
Faults outside the catalogA fault kind not in faults.py:KINDS was not tried
What this is not. This page proves that the workflow keeps its outcomes acceptable under injected faults. Whether the recommendations are good is measured on https://eval.sammuthu.com, and which step runs next in a larger system is the subject of https://orchestrate.sammuthu.com.

The same idea for a coding agent#

The coding agent's anchor has the same shape: CI times out twice, the process dies right after opening the pull request and before recording it, and a new process takes the task. The careful version finds the task by its id, resumes after the last recorded step, looks for the open pull request of its branch before creating one, and ends with one pull request waiting for review. The runtime that carries those mechanisms is the subject of https://harness.sammuthu.com; the review and CI path, of https://diff-to-deploy.sammuthu.com.

The three blocks below hold the detailed layer.

Is the naive workflow a straw man?All ten protections off is a reference point, not a claim about real systems

No real team ships the naive workflow. It exists to give every protection a baseline: a scenario that the naive workflow passes (UW-R01, UW-R02, UW-R07) does not test any protection, and a scenario it fails shows which protection matters there. Real systems sit in between, with some protections and not others, and that middle ground is where the surprises are. The game-day lab shows it: with all five faults and only the idempotency key switched on, the run ends in failed on a key conflict, worse than with no protection at all (Fault injection). For an existing system, the useful exercise is to set the flags to what the system actually has and run the suite.

Two protections against the same duplicateIn `UW-R14` the key and the fencing token would each stop the second row

In UW-R14 worker A stalls for 45,000 ms after drafting, past its 20,000 ms lease. Worker B takes the lease with token 42, resumes and finishes. When A wakes up and writes with token 41, the store refuses it. The README notes that the idempotency key would also have stopped the second row, because both workers make the same intended write. The fencing token stops the stale worker one step earlier and does not depend on the two writes being equal: if A had drafted different text, the key would have raised a conflict, while the token refuses A whatever it wrote. Defence in depth here is not redundancy for its own sake; each mechanism covers a case the other does not.

Why every number on this page repeats exactlySeed 7, a simulated clock and a scripted model

Each command takes --seed (default 7). The seed drives the jitter and the ids, so run-7f3a, the backoff of 52 ms and 908 ms and the trace id are the same on every machine. Time is a counter, so 30,024 ms is not a measurement and never varies. The model is scripted, so the draft is the same text every time. The only wall-clock number in the folder is the time the unit tests take.

This is what turns the suite into a regression test: any change to the code that moves a number on this page is visible in a diff of the output. It is also its limit. A real model and a real provider vary, and their variation is part of what a production system must survive; the example removes it so that the mechanisms can be seen.

Checklist#

QuestionGood answer
Can you name, for each step of your workflow, its side effect and what happens when it fails?Yes, in one table like the eight-step table above
Does every identifier have one job, and is none of them reused for another?Yes: message, submission, run, attempt, resume, key, trace, token
Does a redelivered message find the existing run?Yes, by message_id
Does a resumed run skip completed steps and replay writes by their keys?Yes
Does every scenario end in a named terminal state?Yes, four of them
Is every stored recommendation marked as needing a person, and does only a person write decisions and overrides?Yes
Does a suite run named scenarios, a matrix and seeded random plans against the invariants on every change?Yes
Do you know what your simulation leaves out?Yes, written down

In one breath#

One quote request, eight steps, ten protections: the provider times out twice, the process dies after the write, the queue delivers again, and the careful workflow still ends with one run, one recommendation and one review task waiting for a person. The retry handled the timeouts; the run lookup, the checkpoint and the idempotency key handled the crash and the redelivery, because a retry is not a recovery. Fifteen scenarios and a 295-run suite check seven invariants after every run, and the naive workflow, with nothing on, passes only the three scenarios where luck is enough.

Where to go from here#

You can now read the whole example and run it: change a flag, add a fault to a plan, and watch which invariant breaks. What it cannot tell you is how your own dependencies fail, how your model behaves under pressure, or whether your people know their roles; those come from fault experiments and incidents on your own system. To revisit a mechanism, start again from the beginning, look up a word in the Glossary, or see the sources and sister sites on About.