UW-R15: three faults in one run, and the 30,024 ms of simulated time in which the careful workflow still ends with one recommendation.Honest boundary. This is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider, every timing and every number are invented. Nothing here reports the reliability or the quality of a real system or model.
Every chapter took one fault out of the insurance example and repaired it; this page puts it back together. A broker's quote request for submission SUB-1004 arrives as a message on a queue: an office building, masonry, built 2001, sprinklered, in wind zone C, whose application says the roof is 9 years old. The roof-data provider says 22. Under the policy fixture (UW-GUIDE version 2026.2, illustrative) rule R-ROOF-03 applies: the two ages differ by more than 5 years, so the underwriting agent should recommend escalate. The model recommends; a person decides.
In scenario UW-R15 three things go wrong on the way. The provider times out twice. The worker process crashes right after it stored the recommendation, before it could record that it had. And the queue, never told that the work was done, delivers the same message again, twice in all. Each fault on its own was the subject of a chapter. Together they are the day the design was made for.
What is still true after this fails? One run, one recommendation row, one review task, and a person deciding.
The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
The workflow and its identifiers#
The workflow has eight steps, always in this order, with fixed step ids. A workflow step is one unit of that sequence: it has an id, an input from the steps before it, at most one kind of side effect, and a rule for what happens when it fails. Tiny example: step 6, record_recommendation, writes one row and is retried with the same idempotency key.
| # | Step id | What it does | Side effect | If it fails |
|---|---|---|---|---|
| 1 | load_submission | Validates the submission from the queue message | none | Permanent: the message is poison and ends in the dead-letter queue |
| 2 | get_roof_assessment | Calls the roof-data provider | none | Retry with backoff inside the deadline; breaker; then go on with request_info, missing: ["roof_assessment"] |
| 3 | search_guidance | Retrieves passages of the policy fixture and similar past decisions | none | Retry; without guidance nothing is drafted and the message goes back to the queue |
| 4 | check_eligibility | Runs the rules (code) | none | Deterministic: a failure here is a bug, not a fault |
| 5 | draft_recommendation | The stand-in model writes the recommendation; the output is validated | none | 429: wait retry_after_ms. Malformed: ask again, at most twice, then a code-written escalate. Primary route down: fallback route, marked degraded |
| 6 | record_recommendation | Stores the recommendation row | write | Retry with the same idempotency_key; a replay returns the stored row |
| 7 | open_review_task | Opens the task in the underwriter cockpit | write | Retry with its own key; a permanent rejection compensates step 6 (row marked withdrawn) |
| 8 | await_underwriter | The automated part is over | none | The run sits in waiting_for_human |
Diagram: The eight steps of one run · reads first, the model in the middle, two writes, then a person
Read left to right. Steps 1 to 4 read and compute: repeating them costs time, nothing else. Step 5 is the model, a scripted stand-in here, whose answer must be saved, never asked for twice. Steps 6 and 7 are the only writes and carry idempotency keys; step 7 can compensate step 6. Step 8 hands the run to a person. A checkpoint is saved after every step.
- Read and compute
- 1 · load_submission: Validates the submission taken from the queue message. A submission that never validates is a poison message.
- 2 · get_roof_assessment: Calls the roof-data provider with a 2000 ms timeout, retries with backoff, and can degrade to request_info.
- 3 · search_guidance: Retrieves passages of the policy fixture and similar past decisions.
- 4 · check_eligibility: Runs the deterministic rules. A failure here is a bug, not a fault.
- The model
- 5 · draft_recommendation: The stand-in model writes the recommendation; the output is parsed and checked before it is used. The answer goes into the checkpoint.
- Two writes
- 6 · record_recommendation: Stores the recommendation row. A retry or a resume sends the same key and gets the stored row back.
- 7 · open_review_task: Opens the task in the underwriter cockpit. A permanent rejection marks the row of step 6 withdrawn.
- A person
- 8 · await_underwriter: The automated part ends. An underwriter reads the task and decides; a different decision is an override.
Three nouns from the table need a sentence each. The policy fixture is the invented rule book the agent reads: UW-GUIDE version 2026.2, effective 2026-07-01, with an earlier version 2025.4 in which the roof limit of R-ROOF-01 was 30 years instead of 25. The underwriter cockpit is the web application where underwriters see their work. A review task is the item in that cockpit that puts one stored recommendation in front of an underwriter; without it, the recommendation is in the table and nobody sees it.
The run carries identifiers that each have one job. In UW-R15 they take these values:
| Identifier | Value in UW-R15 | What it names |
|---|---|---|
message_id | msg-51c2 | The quote request as it sits on the queue; the same on every redelivery |
submission_id | SUB-1004 | The broker's request as a record |
run_id | run-7f3a | This execution of the workflow, found by message_id |
attempt_number | 1 to 3 on step 2 | One try of one step |
resume_count | 0, then 1 | How many times a new process picked the run up |
idempotency_key | run-7f3a:record_recommendation | The one intended write of step 6; no attempt number, no timestamp |
trace_id | 3e00680d22530c83b2c9332e533e599c | One trace for the whole run, kept across the resume |
fencing_token | 41, then 42 | Grows with each lease; a write with an old token is refused |
evidence_ids | EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007 | The evidence the recommendation rests on |
Around the steps sit ten protections, each a flag in workflow.py:FLAGS. The careful workflow switches all ten on; the naive workflow switches all ten off.
| Flag | On | Off | Chapter |
|---|---|---|---|
retry_backoff | Bounded attempts, backoff with jitter, retry budget, deadline-aware timeouts | Tries again at once, with no cap, until the deadline is gone | Retries |
validate_output | The model's text is parsed and checked before use | Whatever came back is stored | Failure modes |
idempotency_key | Writes carry <run_id>:<step id> | Writes carry no key | Idempotency |
checkpoint | State saved after every step; a new process resumes | A new process starts at step 1 | Checkpoints |
dedup_inbox | The run is found by message_id; one run per message | Every delivery creates a new run | Queues |
breaker | A dependency that keeps failing is not called for a while | Every call is sent | Breakers |
compensation | A rejected review task withdraws the recommendation | The recommendation stays with no task | Checkpoints |
fencing | The store refuses a write with an old fencing token | The store cannot tell a lease was lost | Concurrency |
fallback_route | Primary model route, then fallback route | Primary route only | Fallback |
dead_letter | After 5 receives the message moves to the dead-letter queue | The message comes back until the simulation stops it | Queues |
The anchor: UW-R15, step by step#
The fault plan of UW-R15 is three rules in its scenario file. The dry run prints it together with the steps and the protections, and runs nothing:
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R15 --dry-run
Scenario UW-R15 · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
The anchor: provider times out twice, the process crashes after the write, the queue redelivers
case UW-004 · submission SUB-1004 · message msg-51c2 · run run-7f3a
dry run: nothing is executed
fault plan:
get_roof_assessment timeout on call 1, 2
process crash on call 1 (after_step record_recommendation, when before_checkpoint)
queue duplicate on call 1
steps:
1 load_submission
2 get_roof_assessment
3 search_guidance
4 check_eligibility
5 draft_recommendation
6 record_recommendation
7 open_review_task
8 await_underwriter
protections: retry_backoff on, validate_output on, idempotency_key on, checkpoint on, dedup_inbox on, breaker on, compensation on, fencing on, fallback_route on, dead_letter on
teaches: Three faults, one outcome: one run, one recommendation row, one review task, waiting for a person. A retry is not a recovery; the key, the checkpoint and the run lookup are.
The command below runs the scenario on the careful workflow in a scratch copy of examples/uw-reliability. A successful run proves that in this simulation the three faults leave exactly one run, one recommendation row and one review task, and that all seven invariants hold. The second tab is the same plan on the naive workflow.
Careful
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R15
Scenario UW-R15 · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
The anchor: provider times out twice, the process crashes after the write, the queue redelivers
case UW-004 · submission SUB-1004 · message msg-51c2 · run run-7f3a
fault plan:
get_roof_assessment timeout on call 1, 2
process crash on call 1 (after_step record_recommendation, when before_checkpoint)
queue duplicate on call 1
timeline:
t_ms worker step att event
0 queue - - message sent: msg-51c2
0 worker-A - - message received: msg-51c2, receive 1
0 worker-A - - run created: trace_id 3e00680d22530c83b2c9332e533e599c
0 worker-A - - lease granted: fencing token 41
0 worker-A load_submission - step done: checkpoint saved
2000 worker-A get_roof_assessment 1 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
2000 worker-A get_roof_assessment 1 backoff: 52 ms
4052 worker-A get_roof_assessment 2 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
4052 worker-A get_roof_assessment 2 backoff: 908 ms
5140 worker-A get_roof_assessment 3 attempt ok
5140 worker-A get_roof_assessment - step done: checkpoint saved
5380 worker-A search_guidance 1 attempt ok
5530 worker-A search_guidance - step done: checkpoint saved
5534 worker-A check_eligibility - step done: checkpoint saved
6934 worker-A draft_recommendation 1 attempt ok
6934 worker-A draft_recommendation - step done: checkpoint saved
6946 worker-A record_recommendation 1 attempt ok
6946 worker-A record_recommendation - PROCESS CRASH: after the step, before checkpoint
6946 worker-A - - process is gone: no acknowledgement; visible again after 30000 ms
30000 worker-B - - message received: msg-51c2, receive 2
30000 worker-B - - run found: by message_id; resume_count 1
30000 worker-B - - lease granted: fencing token 42
30000 worker-B draft_recommendation - resume from checkpoint: steps 1 to 5 are not run again
30012 worker-B record_recommendation 1 attempt ok
30012 worker-B record_recommendation - replay: same key: the stored row was returned, nothing was written
30012 worker-B record_recommendation - step done: checkpoint saved
30024 worker-B open_review_task 1 attempt ok
30024 worker-B open_review_task - step done: checkpoint saved
30024 worker-B await_underwriter - run ended: waiting_for_human
30024 worker-B await_underwriter - step done: checkpoint saved
30024 worker-B - - acknowledgement lost: the queue will deliver the message again
30024 worker-C - - message received: msg-51c2, receive 3
30024 worker-C - - duplicate delivery: run-7f3a already ended in waiting_for_human: nothing to do
30024 worker-C - - acknowledged
end state:
run run-7f3a state waiting_for_human resume_count 1 ended at 30024 ms (deadline 120000 ms)
recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
REC-0001 pending_review escalate cites R-ROOF-03 evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
run run-7f3a route primary degraded false fencing_token 41 key run-7f3a:record_recommendation
expected (careful): 10 of 10 values match
invariants:
INV-1 pass holds
INV-2 pass holds
INV-3 pass holds
INV-4 pass holds
INV-5 pass holds
INV-6 pass holds
INV-7 pass holds
result: 7 of 7 invariants hold
wrote out/UW-R15-careful.jsonNaive
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R15 --variant naive
Scenario UW-R15 · variant naive · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
The anchor: provider times out twice, the process crashes after the write, the queue redelivers
case UW-004 · submission SUB-1004 · message msg-51c2 · run run-7f3a
fault plan:
get_roof_assessment timeout on call 1, 2
process crash on call 1 (after_step record_recommendation, when before_checkpoint)
queue duplicate on call 1
timeline:
t_ms worker step att event
0 queue - - message sent: msg-51c2
0 worker-A - - message received: msg-51c2, receive 1
0 worker-A - - run created: trace_id 3e00680d22530c83b2c9332e533e599c
0 worker-A - - lease granted: fencing token 41
0 worker-A load_submission - step done: no checkpoint
2000 worker-A get_roof_assessment 1 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
4000 worker-A get_roof_assessment 2 attempt failed: ToolTimeout: get_roof_assessment: no answer after 2000 ms
4180 worker-A get_roof_assessment 3 attempt ok
4180 worker-A get_roof_assessment - step done: no checkpoint
4420 worker-A search_guidance 1 attempt ok
4570 worker-A search_guidance - step done: no checkpoint
4574 worker-A check_eligibility - step done: no checkpoint
5974 worker-A draft_recommendation 1 attempt ok
5974 worker-A draft_recommendation - step done: no checkpoint
5986 worker-A record_recommendation 1 attempt ok
5986 worker-A record_recommendation - PROCESS CRASH: after the step, before checkpoint
5986 worker-A - - process is gone: no acknowledgement; visible again after 30000 ms
30000 worker-B - - message received: msg-51c2, receive 2
30000 worker-B - - run created: trace_id b57e8a831a9b02ce226a20653624ddad
30000 worker-B - - lease granted: fencing token 42
30000 worker-B load_submission - step done: no checkpoint
30180 worker-B get_roof_assessment 1 attempt ok
30180 worker-B get_roof_assessment - step done: no checkpoint
30420 worker-B search_guidance 1 attempt ok
30570 worker-B search_guidance - step done: no checkpoint
30574 worker-B check_eligibility - step done: no checkpoint
31974 worker-B draft_recommendation 1 attempt ok
31974 worker-B draft_recommendation - step done: no checkpoint
31986 worker-B record_recommendation 1 attempt ok
31986 worker-B record_recommendation - step done: no checkpoint
31998 worker-B open_review_task 1 attempt ok
31998 worker-B open_review_task - step done: no checkpoint
31998 worker-B await_underwriter - run ended: waiting_for_human
31998 worker-B await_underwriter - step done: no checkpoint
31998 worker-B - - acknowledgement lost: the queue will deliver the message again
31998 worker-C - - message received: msg-51c2, receive 3
31998 worker-C - - run created: trace_id 6318c9d4cd8e80819aa606bcd5728913
31998 worker-C - - lease granted: fencing token 43
31998 worker-C load_submission - step done: no checkpoint
32178 worker-C get_roof_assessment 1 attempt ok
32178 worker-C get_roof_assessment - step done: no checkpoint
32418 worker-C search_guidance 1 attempt ok
32568 worker-C search_guidance - step done: no checkpoint
32572 worker-C check_eligibility - step done: no checkpoint
33972 worker-C draft_recommendation 1 attempt ok
33972 worker-C draft_recommendation - step done: no checkpoint
33984 worker-C record_recommendation 1 attempt ok
33984 worker-C record_recommendation - step done: no checkpoint
33996 worker-C open_review_task 1 attempt ok
33996 worker-C open_review_task - step done: no checkpoint
33996 worker-C await_underwriter - run ended: waiting_for_human
33996 worker-C await_underwriter - step done: no checkpoint
33996 worker-C - - acknowledged
end state:
run run-7f3a state running resume_count 0 not finished (deadline 120000 ms)
run run-85a7 state waiting_for_human resume_count 0 ended at 31998 ms (deadline 150000 ms)
run run-9e62 state waiting_for_human resume_count 0 ended at 33996 ms (deadline 151998 ms)
recommendation rows 3 (pending_review 3, withdrawn 0) · review tasks 2 · decisions 0 · dead letters 0
REC-0001 pending_review escalate cites R-ROOF-03 evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
run run-7f3a route primary degraded false fencing_token 41 key None
REC-0002 pending_review escalate cites R-ROOF-03 evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
run run-85a7 route primary degraded false fencing_token 42 key None
REC-0003 pending_review escalate cites R-ROOF-03 evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
run run-9e62 route primary degraded false fencing_token 43 key None
expected (naive): 10 of 10 values match
invariants:
INV-1 FAIL SUB-1004 has 3 pending_review rows (a duplicate write)
INV-2 pass holds
INV-3 pass holds
INV-4 FAIL run-7f3a is left in state running
INV-5 pass holds
INV-6 pass holds
INV-7 pass holds
result: 5 of 7 invariants hold · broken: INV-1, INV-4
wrote out/UW-R15-naive.jsonRead the careful timeline as three acts.
Act one, the provider (0 to 5,140 ms). Worker A receives msg-51c2, creates run-7f3a, takes the lease with fencing token 41 and checkpoints step 1. The roof-data provider gives no answer within the 2,000 ms timeout, twice. Between attempts the worker waits 52 ms, then 908 ms: backoff with full jitter in this example. Attempt 3 answers at 5,140 ms. This is the only act a retry can handle. The naive workflow also gets through it, by luck (UW-R02).
Act two, the crash (5,140 to 30,000 ms). Guidance, the rule check and the draft follow, each checkpointed; the draft is escalate, citing R-ROOF-03. At 6,946 ms worker A stores the recommendation row REC-0001 under the key run-7f3a:record_recommendation, and dies before it can save the checkpoint of step 6. The store has the row. The checkpoint still says step 5. The message was never acknowledged, so it becomes visible again after the queue's visibility timeout of 30,000 ms. No retry loop survives this: the loop died with the process. A retry is not a recovery.
Act three, the recovery (30,000 to 30,024 ms). Worker B receives the message (receive 2). Three mechanisms, each from a different chapter, act in order. The inbox finds run-7f3a by message_id, so no second run is created. The new lease carries fencing token 42. The checkpoint says step 5 is done, so steps 1 to 5 are skipped: no second provider call, no second model call. Step 6 runs again, because worker B cannot know the write committed (a timeout tells you nothing about whether the write happened, and neither does a crash), and sends the same key; the store returns the stored row and writes nothing. Step 7 opens TASK-0001; the run ends in waiting_for_human at 30,024 ms. Then the acknowledgement of that delivery is lost too, the queue delivers a third time, and worker C finds the run already ended: nothing to do.
The naive tab shows the same three faults without those mechanisms. Worker B finds no run for the message and starts run-85a7 at step 1; worker C starts run-9e62. Three recommendation rows wait for review for one submission (INV-1), two review tasks reach the cockpit, and run-7f3a is left in running forever (INV-4).
Diagram: UW-R15 · three faults, three workers, one run, one row
Time runs down the page, in simulated milliseconds. Worker A survives the provider timeouts by retrying, stores the recommendation and crashes before its checkpoint. Worker B gets the redelivered message, finds the same run, skips steps 1 to 5 and replays step 6 by its key. Worker C gets a third delivery and finds the run already ended. The agent is a scripted stand-in; every fault is injected.
- Queue to Worker A: Deliver msg-51c2 (receive 1) at 0 ms. Worker A creates run-7f3a and takes the lease with token 41.
- Worker A to Roof-data provider: Attempts 1 and 2: no answer. Two timeouts at 2000 and 4052 ms, with backoff of 52 ms and then 908 ms.
- Roof-data provider to Worker A: Attempt 3 answers: 22 years (5140 ms). The retry worked. This is the only fault a retry can handle.
- Worker A to Store: Steps 3 to 5, checkpoint after each; step 6 writes REC-0001 (6946 ms). The row is stored with key run-7f3a:record_recommendation and fencing token 41.
- Worker A: Crash before the checkpoint of step 6. The store has the row; the checkpoint still says step 5. The message was never acknowledged.
- Queue to Worker B: Visible again: receive 2 at 30000 ms. After the visibility timeout the same message goes to worker B.
- Worker B to Store: Find run by message_id; load checkpoint; skip steps 1 to 5. No second run, no second provider call, no second model call.
- Worker B to Store: Step 6 again, same key (30012 ms). Worker B cannot know the write committed, so it sends it again with the same key.
- Store to Worker B: Replay: the stored row, nothing written. Still one recommendation row for SUB-1004.
- Worker B to Store: Step 7: TASK-0001; run ends waiting_for_human (30024 ms). The acknowledgement of this delivery is lost as well, so the queue delivers once more.
- Queue to Worker C: Receive 3 at 30024 ms: run already ended. Worker C finds run-7f3a in waiting_for_human and only acknowledges the message.
The one row the careful run leaves, from out/UW-R15-careful.json:
{"recommendation_id": "REC-0001", "submission_id": "SUB-1004", "run_id": "run-7f3a",
"idempotency_key": "run-7f3a:record_recommendation",
"recommendation": {"recommendation": "escalate",
"reasons": ["The application says the roof is 9 years old. The roof-data provider says 22 years.",
"R-ROOF-03: Roof age on the application and from the provider differ by more than 5 years: escalate to a senior underwriter.",
"Next step: escalate to a senior underwriter to settle the roof age."],
"citations": ["R-ROOF-03"], "missing": [], "stated_confidence": 0.8,
"requires_human": true, "policy_version": "2026.2"},
"evidence_ids": ["EV-ROOF-0412", "EV-GUIDE-0031", "EV-RULE-0007"],
"model_route": "primary", "degraded": false, "status": "pending_review",
"fencing_token": 41, "created_by": "underwriting-agent", "created_ms": 6946}
fencing_token 41 shows worker A wrote it under its lease, before the crash. requires_human: true and status: pending_review mark a recommendation waiting for a person, not a decision. policy_version and evidence_ids say what it rests on (INV-6).
What happens next is outside the example. An underwriter opens TASK-0001, reads the recommendation and decides. If the decision differs from the recommendation, it is an override: the person's decision, recorded with who decided, what and why, as human_decision with override: true. Shared case UW-009 is such a case: the agent's facts support approval, and the underwriter declines because the broker's loss run shows an open claim missing from the application. Only a person writes that record; no function in the example writes a decisions row, and INV-3 checks it.
Try the anchor yourself#
The anchor preset switches on the two faults of UW-R15 that the lab offers. The six protections that are not switches, inbox deduplication among them, stay on, so the lab never creates a second run. Every combination is a real run of the example, computed in advance.
Without JavaScript: the anchor faults (provider timeout twice, crash after the write) with one protection removed at a time, from run_scenario.py:gameday (seed 7) in a copy of the example, and the command-line naive run for comparison.
| Protections | End state | Rows | Tasks | Provider calls | Model calls | Cost units | Invariants that fail |
|---|---|---|---|---|---|---|---|
| All four (the preset) | waiting_for_human at 30,024 ms | 1 | 1 | 3 | 1 | 1,340 | none |
| Without retry with backoff | waiting_for_human at 30,024 ms | 1 | 1 | 3 | 1 | 1,340 | none |
| Without output validation | waiting_for_human at 30,024 ms | 1 | 1 | 3 | 1 | 1,340 | none |
| Without the idempotency key | waiting_for_human at 30,024 ms | 2 | 1 | 3 | 1 | 1,340 | INV-1 |
| Without the checkpoint | waiting_for_human at 31,998 ms | 1 | 1 | 4 | 2 | 2,680 | none |
UW-R15 naive, command line (all ten off, third delivery) | run-7f3a left running | 3 | 2 | INV-1, INV-4 |
Rows two and three do not make backoff and validation useless: here the immediate retry happens to work and no model fault is on. Row five is the quiet one: every invariant holds and the run pays twice.
All fifteen scenarios#
Each scenario is a shared case plus a fault plan. Unless noted, the submission is SUB-1004. The table comes from running every scenario for both variants in a scratch copy of the example; "ends" is simulated time.
| Scenario | Fault | Careful | Naive | Naive breaks | Chapter |
|---|---|---|---|---|---|
UW-R01 | none | waiting_for_human at 1,998 ms; 1 row, 1 task | the same | none | Start |
UW-R02 | Provider times out twice, then answers | 3 attempts, backoff 52 and 908 ms; ends 6,958 ms; 1 row | 3 attempts at once; ends 5,998 ms; 1 row (luck) | none | Retries |
UW-R03 | Provider stays down (SUB-1006) | Breaker opens; degraded_request_info at 8,778 ms; 1 row, missing: ["roof_assessment"] | 60 attempts; deadline_exceeded at 120,000 ms; 0 rows | INV-4, INV-5 | Breakers |
UW-R04 | Model answers 429 twice | Waits 1,500 ms twice; ends 5,198 ms | 33 attempts; ends 5,198 ms | INV-5 | Retries |
UW-R05 | Model output malformed once | Asks again; ends 3,398 ms; 2,680 cost units | Stores the broken text; ends 1,998 ms | INV-3, INV-6 | Failure modes |
UW-R06 | Write commits, acknowledgement lost | Retry is a replay; 1 row; ends 2,062 ms | 2 rows | INV-1 | Idempotency |
UW-R07 | Store fails before the commit, once | Retry; 1 row; ends 2,062 ms | Retry; 1 row | none | Idempotency |
UW-R08 | Crash after the write, before the checkpoint | Resume replays step 6; 1 row; ends 30,024 ms | New run: 2 rows; first run left running | INV-1, INV-4 | Checkpoints |
UW-R09 | Crash after step 2 | Resume at step 3; 1 provider call; ends 31,818 ms | New run; 2 provider calls; first run left running | INV-4 | Checkpoints |
UW-R10 | Queue delivers twice | Second delivery finds the run; 1 run, 1 row | 2 runs, 2 rows, 2 tasks | INV-1 | Queues |
UW-R11 | Poison message | 5 receives, dead_lettered at 5,000 ms; 0 rows | 12 deliveries, 12 runs left running | INV-4 | Queues |
UW-R12 | Cockpit rejects the review task for good | REC-0001 marked withdrawn; failed_compensated at 1,998 ms | pending_review row with no task; state failed | INV-4 | Checkpoints |
UW-R13 | Primary model route down | Breaker, fallback route; degraded: true; ends 5,458 ms; 620 cost units | 120 attempts; deadline_exceeded; 0 rows | INV-4, INV-5 | Fallback |
UW-R14 | A worker stalls 45,000 ms past its lease | Stale write with token 41 refused; 1 row | Both write: 2 runs, 2 rows | INV-1 | Concurrency |
UW-R15 | Provider timeouts, crash after the write, redelivery | 1 run, 1 row, 1 task; ends 30,024 ms | 3 runs, 3 rows, 2 tasks | INV-1, INV-4 | this page |
The careful workflow always ends in a terminal state: waiting_for_human twelve times (UW-R13 among them, with its fallback result marked degraded), degraded_request_info once, failed_compensated once and dead_lettered once. The naive workflow passes three, UW-R01, UW-R02 and UW-R07, which are exactly the scenarios where an unprotected retry is enough.
The duplicate, the resume and the week#
Three labs replay parts of the anchor in isolation. The first chooses where the failure hits the write and how the key is built.
The same results without JavaScript, from idempotency.py:simulate_writes with two retries after the failure:
| Failure point | No key | run_id:step | Key with attempt_number | Random key per call | Right key, other payload |
|---|---|---|---|---|---|
Before the commit (UW-R07) | 1 row | 1 row | 1 row | 1 row | 1 row |
After the commit, acknowledgement lost (UW-R06) | 3 rows | 1 row | 3 rows | 3 rows | 1 row, retries refused as conflicts |
Crash before the checkpoint (UW-R08) | 3 rows | 1 row | 3 rows | 3 rows | 1 row, retries refused as conflicts |
The second lab crashes the first process after a step you choose, with checkpoints on or off, and can add a permanent rejection at step 7.
Without JavaScript, the crash after step 6 (in the lab the crash comes after that step's checkpoint), from workflow.py:simulate_resume:
| Step | Checkpoints on: process 1 | Checkpoints on: process 2 | Checkpoints off: process 1 | Checkpoints off: process 2 |
|---|---|---|---|---|
| 1 to 5 | ran | skipped | ran | ran again |
6 record_recommendation | ran | skipped | ran | replayed by its key |
7 open_review_task | not reached | ran | not reached | ran |
8 await_underwriter | not reached | ran | not reached | ran |
| Provider and model calls in total | 1 and 1 | 2 and 2 | ||
| End | waiting_for_human, 1 row, 1 task | waiting_for_human, 1 row, 1 task |
With a permanent rejection at step 7 and no crash (UW-R12): steps 1 to 5 run, step 6 ends as compensated, step 7 as failed, and the run ends in failed_compensated with 1 row marked withdrawn and 0 tasks.
The third lab leaves the single run and looks at a week of invented runs against a service level objective.
Without JavaScript, the three weeks at the lab's default settings (objective 95% good runs, a good run under 8,000 ms), from slo.py:summarize and first_alert:
| Week | Runs | p50 | p95 | p99 | Unacceptable outcomes | Error budget used | Burn-rate alert |
|---|---|---|---|---|---|---|---|
| Healthy | 2,315 | 3,256 ms | 4,475 ms | 9,739 ms | 18 (0.78%) | 58% | does not fire |
| Provider brown-out | 2,315 | 3,415 ms | 17,471 ms | 29,695 ms | 35 (1.51%) | 220% | hour 63 |
| Bad deployment | 2,315 | 3,256 ms | 4,475 ms | 9,739 ms | 87 (3.76%) | 117% | hour 104 |
The bad deployment hides from latency: its percentiles equal the healthy week's. Only the unacceptable outcome rate, mostly duplicate writes in this profile, shows it. The brown-out is the incident walked through on Incidents.
The seven invariants and the suite#
After every run seven invariants are checked against the rows left in the store. Their reasoning is on Fault injection; here is the list the suite applies.
| Id | Invariant |
|---|---|
INV-1 | At most one recommendation row with status pending_review per submission, and exactly one when the run ended in waiting_for_human or degraded_request_info |
INV-2 | At most one review task per run |
INV-3 | No decisions row written by the agent; every stored recommendation has requires_human: true |
INV-4 | Every run ends in a named terminal state inside its deadline |
INV-5 | No step exceeds its maximum attempts; the retry budget is never overdrawn |
INV-6 | The stored recommendation carries the current policy_version and only evidence_ids that exist in the run |
INV-7 | A degraded or fallback result says so (degraded: true), and a withdrawn recommendation is still in the table |
The suite runs the 15 scenarios, a matrix of the ten shared cases against no fault and seven single faults (80 runs), and 200 random fault plans with seed 7: 295 runs per variant. A successful careful run proves that in this simulation every invariant held in every one of them. The command exits with 1 when any run breaks an invariant, as the naive run does.
Careful
/opt/homebrew/bin/python3 fault_suite.py
Fault-injection suite · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
Part 1: the 15 named scenarios
scenario terminal state rows tasks expected broken invariants
UW-R01 waiting_for_human 1 1 match -
UW-R02 waiting_for_human 1 1 match -
UW-R03 degraded_request_info 1 1 match -
UW-R04 waiting_for_human 1 1 match -
UW-R05 waiting_for_human 1 1 match -
UW-R06 waiting_for_human 1 1 match -
UW-R07 waiting_for_human 1 1 match -
UW-R08 waiting_for_human 1 1 match -
UW-R09 waiting_for_human 1 1 match -
UW-R10 waiting_for_human 1 1 match -
UW-R11 dead_lettered 0 0 match -
UW-R12 failed_compensated 1 0 match -
UW-R13 waiting_for_human 1 1 match -
UW-R14 waiting_for_human 1 1 match -
UW-R15 waiting_for_human 1 1 match -
Part 2: the matrix, 10 shared cases x (no fault + 7 single faults). A cell shows the broken invariants.
case none provider-timeout model-429 malformed db-transient ack-lost crash-after-write duplicate
UW-001 ok ok ok ok ok ok ok ok
UW-002 ok ok ok ok ok ok ok ok
UW-003 ok ok ok ok ok ok ok ok
UW-004 ok ok ok ok ok ok ok ok
UW-005 ok ok ok ok ok ok ok ok
UW-006 ok ok ok ok ok ok ok ok
UW-007 ok ok ok ok ok ok ok ok
UW-008 ok ok ok ok ok ok ok ok
UW-009 ok ok ok ok ok ok ok ok
UW-010 ok ok ok ok ok ok ok ok
Part 3: 200 random fault plans (1 to 3 faults each, seeded)
end states: waiting_for_human 131, degraded_request_info 31, failed_compensated 22, dead_lettered 16
runs with every invariant holding: 200 of 200
Runs that break each invariant (scenarios / matrix / random):
INV-1 0 / 0 / 0
INV-2 0 / 0 / 0
INV-3 0 / 0 / 0
INV-4 0 / 0 / 0
INV-5 0 / 0 / 0
INV-6 0 / 0 / 0
INV-7 0 / 0 / 0
wrong recommendation without any fault: 0 of 10 cases
SUMMARY: 295 runs · 295 pass · 0 fail · every invariant held in every run
wrote out/suite-careful.jsonNaive
/opt/homebrew/bin/python3 fault_suite.py --variant naive
Fault-injection suite · variant naive · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
Part 1: the 15 named scenarios
scenario terminal state rows tasks expected broken invariants
UW-R01 waiting_for_human 1 1 match -
UW-R02 waiting_for_human 1 1 match -
UW-R03 deadline_exceeded 0 0 match INV-4, INV-5
UW-R04 waiting_for_human 1 1 match INV-5
UW-R05 waiting_for_human 1 1 match INV-3, INV-6
UW-R06 waiting_for_human 2 1 match INV-1
UW-R07 waiting_for_human 1 1 match -
UW-R08 running 2 1 match INV-1, INV-4
UW-R09 running 1 1 match INV-4
UW-R10 waiting_for_human 2 2 match INV-1
UW-R11 running 0 0 match INV-4
UW-R12 failed 1 0 match INV-4
UW-R13 deadline_exceeded 0 0 match INV-4, INV-5
UW-R14 waiting_for_human 2 2 match INV-1
UW-R15 running 3 2 match INV-1, INV-4
Part 2: the matrix, 10 shared cases x (no fault + 7 single faults). A cell shows the broken invariants.
case none provider-timeout model-429 malformed db-transient ack-lost crash-after-write duplicate
UW-001 ok ok 5 3,6 ok 1 1,4 1
UW-002 ok ok 5 3,6 ok 1 1,4 1
UW-003 ok ok 5 3,6 ok 1 1,4 1
UW-004 ok ok 5 3,6 ok 1 1,4 1
UW-005 ok ok 5 3,6 ok 1 1,4 1
UW-006 4,5,R 4,5 4,5 4,5 4,5 4,5 4,5 4,5
UW-007 ok ok 5 3,6 ok 1 1,4 1
UW-008 ok ok 5 3,6 ok 1 1,4 1
UW-009 ok ok 5 3,6 ok 1 1,4 1
UW-010 ok ok 5 3,6 ok 1 1,4 1
Part 3: 200 random fault plans (1 to 3 faults each, seeded)
end states: waiting_for_human 81, running 57, deadline_exceeded 47, failed 15
runs with every invariant holding: 29 of 200
Runs that break each invariant (scenarios / matrix / random):
INV-1 5 / 27 / 43
INV-2 0 / 0 / 18
INV-3 1 / 9 / 26
INV-4 7 / 17 / 121
INV-5 3 / 17 / 72
INV-6 1 / 9 / 26
INV-7 0 / 0 / 0
wrong recommendation without any fault: 1 of 10 cases
SUMMARY: 295 runs · 59 pass · 236 fail · see the broken invariants above
wrote out/suite-naive.json| Careful | Naive | |
|---|---|---|
| Runs that pass | 295 of 295 | 59 of 295 |
| Scenarios that pass | 15 of 15 | 3 of 15 |
| Random plans with every invariant holding | 200 of 200 | 29 of 200 |
| Invariant broken most often | none | INV-4: 7 scenarios, 17 matrix runs, 121 random plans |
| Broken only in random plans | none | INV-2: 18 runs |
Where each concept lives in the code#
The example is standard-library Python in examples/uw-reliability/. The map from concept to code, from its README:
| Concept | Where |
|---|---|
| Transient, permanent, malformed, crash | errors.py: the classes; errors.py:classify |
| Fault plan, fault injection | faults.py:FaultPlan, faults.py:FaultInjector.fire; a scenario's fault_plan |
| Timeout (one call) and deadline (the run) | retry.py:RetryPolicy.timeout_ms, retry.py:Deadline; workflow.py:RUN_DEADLINE_MS |
| Deadline propagation | retry.py:call_with_retry |
| Exponential backoff, jitter | retry.py:backoff_delay, retry.py:schedule |
| Retry storm | retry.py:herd |
| Retry budget | retry.py:RetryBudget |
| Retry only what is retryable; Retry-After | retry.py:call_with_retry, retry.py:retry_delay |
attempt_number | retry.py:attempt_once |
| The retry without protection | retry.py:call_naive |
| Idempotency key | idempotency.py:make_key |
| Deduplication of a write, replay, key conflict | idempotency.py:IdempotencyStore.execute |
| Lost acknowledgement | store.py:Store.write |
| At-least-once, visibility timeout, redelivery | queue_sim.py:Queue.receive, Queue.ack, Queue.nack |
| Dead-letter queue, poison message | queue_sim.py:Queue.receive; world.py:World._note_dead_letters; tools.py:validate_submission |
| Inbox (one run per message) | workflow.py:Worker._find_or_create_run, store.py:Store.insert_run |
| Checkpoint | checkpoint.py:CheckpointStore.save, workflow.py:Workflow._save_checkpoint |
| Resume | workflow.py:Workflow.restore, workflow.py:Workflow.execute |
| Saga, compensation | workflow.py:Workflow.open_review_task, Workflow.compensate_record_recommendation, store.py:Store.withdraw |
| Terminal states | workflow.py:TERMINAL_STATES, Workflow.finish, Workflow.await_underwriter |
| Circuit breaker, fail fast | breaker.py:CircuitBreaker.allow, record, call |
| Graceful degradation | workflow.py:Workflow.get_roof_assessment |
| Bounded re-ask of the model | agent_sim.py:validate_output; errors.py:classify; Workflow.draft_recommendation |
| Token bucket, concurrency limit, cost budget | limits.py:TokenBucket, ConcurrencyLimit, Budget; Workflow._call_model |
| Lease, fencing token | lease.py:LeaseManager.acquire, store.py:Store.check_fencing |
| Model fallback, degraded marking | router.py:route_call, agent_sim.py:draft, Workflow.draft_recommendation |
| One trace per run; a resume reuses it | tracing.py:Tracer, workflow.py:Workflow.__init__ |
| Invariants | invariants.py:check_invariants |
| Fault-injection suite | fault_suite.py:part_scenarios, part_matrix, part_random |
| Careful and naive; single protections | workflow.py:flags_for, workflow.py:FLAGS |
| Percentile, SLI, error budget, burn rate, alert | slo.py:percentile, sli, error_budget, budget_consumed, burn_rate, burn_alert |
| Canary and rollback | rollout.py:simulate_rollout |
TokenBucket and ConcurrencyLimit are not wired into the workflow; they exist for the rate-limit lab and the tests.
To run everything yourself, copy the folder first, because the runner rewrites out/:
cp -R examples/uw-reliability "$TMPDIR/uwr"
cd "$TMPDIR/uwr"
PY=/opt/homebrew/bin/python3
$PY run_scenario.py --scenario UW-R15
$PY run_scenario.py --scenario UW-R15 --variant naive
$PY fault_suite.py
$PY -m unittest
The 77 unit tests check mechanisms; the suite checks outcomes.
What the example leaves out#
The README lists what the simulation leaves out; each item matters when the design moves to production:
| Left out | Consequence for the reader |
|---|---|
| Real parallelism | "Two workers at once" means one stalls while the clock runs; real races are harder |
| A real network | A timeout is a number added to the clock; no partial responses, no slow connections |
| A real database | The store is lists in memory: no isolation levels, no replication lag; "key and row in one commit" is trivially true |
| Real time | The deadline, the visibility timeout and the lease are compared with a counter |
| A real model | The stand-in writes what the rules imply and cannot be wrong the way a model can; its two routes differ in four scripted ways |
| Per-process state | The breaker and the retry budget are shared by all workers of one story and survive a crash |
| The human step | The run ends in waiting_for_human; approval timeouts and escalation are discussed on Advanced |
| Faults outside the catalog | A fault kind not in faults.py:KINDS was not tried |
What this is not. This page proves that the workflow keeps its outcomes acceptable under injected faults. Whether the recommendations are good is measured on https://eval.sammuthu.com, and which step runs next in a larger system is the subject of https://orchestrate.sammuthu.com.
The same idea for a coding agent#
The coding agent's anchor has the same shape: CI times out twice, the process dies right after opening the pull request and before recording it, and a new process takes the task. The careful version finds the task by its id, resumes after the last recorded step, looks for the open pull request of its branch before creating one, and ends with one pull request waiting for review. The runtime that carries those mechanisms is the subject of https://harness.sammuthu.com; the review and CI path, of https://diff-to-deploy.sammuthu.com.
The three blocks below hold the detailed layer.
Is the naive workflow a straw man?All ten protections off is a reference point, not a claim about real systems
No real team ships the naive workflow. It exists to give every protection a baseline: a scenario that the naive workflow passes (UW-R01, UW-R02, UW-R07) does not test any protection, and a scenario it fails shows which protection matters there. Real systems sit in between, with some protections and not others, and that middle ground is where the surprises are. The game-day lab shows it: with all five faults and only the idempotency key switched on, the run ends in failed on a key conflict, worse than with no protection at all (Fault injection). For an existing system, the useful exercise is to set the flags to what the system actually has and run the suite.
Two protections against the same duplicateIn `UW-R14` the key and the fencing token would each stop the second row
In UW-R14 worker A stalls for 45,000 ms after drafting, past its 20,000 ms lease. Worker B takes the lease with token 42, resumes and finishes. When A wakes up and writes with token 41, the store refuses it. The README notes that the idempotency key would also have stopped the second row, because both workers make the same intended write. The fencing token stops the stale worker one step earlier and does not depend on the two writes being equal: if A had drafted different text, the key would have raised a conflict, while the token refuses A whatever it wrote. Defence in depth here is not redundancy for its own sake; each mechanism covers a case the other does not.
Why every number on this page repeats exactlySeed 7, a simulated clock and a scripted model
Each command takes --seed (default 7). The seed drives the jitter and the ids, so run-7f3a, the backoff of 52 ms and 908 ms and the trace id are the same on every machine. Time is a counter, so 30,024 ms is not a measurement and never varies. The model is scripted, so the draft is the same text every time. The only wall-clock number in the folder is the time the unit tests take.
This is what turns the suite into a regression test: any change to the code that moves a number on this page is visible in a diff of the output. It is also its limit. A real model and a real provider vary, and their variation is part of what a production system must survive; the example removes it so that the mechanisms can be seen.
Checklist#
| Question | Good answer |
|---|---|
| Can you name, for each step of your workflow, its side effect and what happens when it fails? | Yes, in one table like the eight-step table above |
| Does every identifier have one job, and is none of them reused for another? | Yes: message, submission, run, attempt, resume, key, trace, token |
| Does a redelivered message find the existing run? | Yes, by message_id |
| Does a resumed run skip completed steps and replay writes by their keys? | Yes |
| Does every scenario end in a named terminal state? | Yes, four of them |
| Is every stored recommendation marked as needing a person, and does only a person write decisions and overrides? | Yes |
| Does a suite run named scenarios, a matrix and seeded random plans against the invariants on every change? | Yes |
| Do you know what your simulation leaves out? | Yes, written down |
In one breath#
One quote request, eight steps, ten protections: the provider times out twice, the process dies after the write, the queue delivers again, and the careful workflow still ends with one run, one recommendation and one review task waiting for a person. The retry handled the timeouts; the run lookup, the checkpoint and the idempotency key handled the crash and the redelivery, because a retry is not a recovery. Fifteen scenarios and a 295-run suite check seven invariants after every run, and the naive workflow, with nothing on, passes only the three scenarios where luck is enough.
Where to go from here#
You can now read the whole example and run it: change a flag, add a fault to a plan, and watch which invariant breaks. What it cannot tell you is how your own dependencies fail, how your model behaves under pressure, or whether your people know their roles; those come from fault experiments and incidents on your own system. To revisit a mechanism, start again from the beginning, look up a word in the Glossary, or see the sources and sister sites on About.