The underwriting agent of this site handles submission SUB-1004: the application says the roof is 9 years old, the roof-data provider says 22, so the expected recommendation is escalate, citing R-ROOF-03. The model recommends; a person decides.
Picture a team that has built the protections of the earlier pages one at a time and tested each one on its own. The idempotency key passed its test: in UW-R06 the acknowledgement of the write is lost, the retry carries the same key, and the store returns the stored row. Crash recovery passed its test too. Output validation is still on the backlog, and so are checkpoints. Every test is green.
Then someone runs a small game day. Two faults at once: the model's first answer is malformed, and the process crashes after the write. Retry with backoff and the idempotency key are on; validation and checkpoints are off. Here are selected lines of that run, verbatim, from the example's game-day function (run_scenario.py:gameday, seed 7) in a scratch copy of examples/uw-reliability. Note that the malformed answer at 1974 ms is logged as "attempt ok": with validation off, nothing looked at it.
t_ms worker step att event
1974 worker-A draft_recommendation 1 attempt ok
1986 worker-A record_recommendation 1 attempt ok
1986 worker-A record_recommendation - PROCESS CRASH: after the step, before checkpoint
30000 worker-B - - run found: by message_id; resume_count 1
30000 worker-B load_submission - step done: no checkpoint
31974 worker-B draft_recommendation 1 attempt ok
31986 worker-B record_recommendation 1 attempt failed: KeyConflict: key run-7f3a:record_recommendation was first used with another payload
31986 worker-B record_recommendation - run ended: failed
Worker A stored the broken model output, because nothing checked it, and died. Worker B had no checkpoint, so it started at step 1 and asked the model again. This time the answer was well formed, and it differed from what was stored under the same key. The store did what an idempotency store must do with the same key and a different payload: it refused. The run ended in failed, which is not one of the four terminal states, with a broken recommendation in the table and no review task.
No single-fault test could have shown this. The key works. The crash recovery works. Their combination with a missing protection produced an outcome that nobody wrote a test for, because nobody imagined it. This page is about finding such outcomes before production does.
Plain English. A recovery path is code that runs only when something goes wrong. If nothing has gone wrong yet, that code has never run, and code that has never run usually has a bug. Fault injection makes things go wrong on purpose so the recovery code runs while you are watching.
Why waiting for real failures does not work#
Most of the protections on this site sit on paths that a healthy system never takes. The retry loop of Retries runs only after a timeout. The replay of Idempotency runs only after a lost acknowledgement. The resume of Checkpoints runs only after a crash. In this example a healthy run, UW-R01, touches none of them: one attempt per step, done at 1998 ms.
So the usual test suite, which feeds good inputs to a healthy system, says nothing about them. Waiting for production to exercise them has three problems. Real faults are rare, so a broken path can sit for months. They arrive at the worst time, during a provider outage when many runs fail together. And they arrive in combinations: a crash during a retry, a redelivery after a resume.
An agent makes this worse in two ways. Its runs are long, so many faults can land inside one run (UW-R15 has three). And one of its steps, the model, does not give the same answer twice, so a path that repeats the model step can behave differently on the second pass, as the opening run showed.
The way out is to stop waiting and cause the failures yourself.
Fault injection: making a dependency fail on purpose#
Fault injection is causing a chosen fault at a chosen place, on purpose, to see how the system responds. A tiny example: "the roof-data provider does not answer the first two calls for SUB-1004". In the example code nothing fails by chance; every fault is injected.
The written form of that intent is a fault plan: a list of rules, each naming a target, the kind of fault and which calls fail. Here is the plan of UW-R15, from its scenario file:
[{"target": "get_roof_assessment", "kind": "timeout", "on_calls": [1, 2]},
{"target": "process", "kind": "crash", "after_step": "record_recommendation",
"when": "before_checkpoint", "on_calls": [1]},
{"target": "queue", "kind": "duplicate", "on_calls": [1]}]
Three rules, three faults: the provider times out on its first two calls; the process crashes after the write and before its checkpoint, once; the queue loses one acknowledgement, so it delivers the message again. A plan is data, not code, so it can be read by a person, stored next to the scenario, diffed in a review and replayed exactly.
The component that reads the plan and decides, call by call, whether a fault fires is the injector. In the example it is a counter per rule. Every call to a target is registered; when the count is in on_calls, the rule fires and the caller raises the matching error. From faults.py, the method FaultInjector.fire, verbatim:
def fire(self, target: str, **where: Any) -> dict[str, Any] | None:
"""Register one call to `target`. Returns the rule that fires on this call, or None.
`where` narrows the match: `fire("process", after_step="record_recommendation",
when="before_checkpoint")` only counts for rules that carry the same values.
"""
for i, rule in enumerate(self.plan.rules):
if rule["target"] != target or any(rule.get(k) != v for k, v in where.items()):
continue
self.hits[i] += 1
if rule["on_calls"] == "always" or self.hits[i] in rule["on_calls"]:
self.fired.append({"target": target, "kind": rule["kind"]})
return rule # a call that one rule takes is not counted by the rules after it
return None
Two design choices in it matter. The fault is decided by call number, not by a random draw, so the same plan gives the same run every time. And the injector sits at the boundary of each dependency (the tool call, the model call, the store write, the queue), which is where real faults appear as errors. The workflow code does not know a fault is injected; it sees a ToolTimeout or an AckLost exactly as it would see a real one.
Before AI. A Java team tests the same way with a test double: a payment client that throws on the second call, or a Kafka consumer killed between processing a record and committing its offset. What changes for an agent is the step in the middle whose output is not repeatable. A fault that forces a step to run again can change the content of the result, not only its timing, so the plan must include faults that repeat the model step.
Steady state and invariants: what "still works" means#
Injecting a fault is easy. Deciding whether the system survived it is the hard part. "It did not crash" is not enough: the naive workflow did not crash in UW-R06 and left two recommendation rows behind.
The Principles of Chaos Engineering, a short statement published at principlesofchaos.org, start an experiment from steady state: a measurable output of the system that shows normal behaviour, such as throughput, error rate or latency. You state a hypothesis that this output will not change when the fault is added, and then try to show that it does. The page stresses the output a user sees over the internals of the system.
For this workflow the steady state of one run is the outcome of UW-R01: one run ends in waiting_for_human, one pending_review row, one review task, inside the deadline. Across many runs it is the service level indicator of SLOs: the share of runs with an acceptable outcome. A fault experiment asks whether either moves.
For a single run, the example turns steady state into seven statements that must be true afterwards, whatever happened on the way. Each one is an invariant: a property that must hold after every run, checked only on what is left. A tiny example: "a submission never has two recommendation rows waiting for review". The seven of this site:
| Id | What must be true after every run | What it catches |
|---|---|---|
INV-1 | At most one pending_review row per submission, and exactly one when the run ended in waiting_for_human or degraded_request_info | Duplicate writes; a run that claims success with no row |
INV-2 | At most one review task per run | A duplicate task in the underwriter cockpit |
INV-3 | No decisions row written by the agent; every stored recommendation has requires_human: true | The agent acting as the decider; broken output stored as a recommendation |
INV-4 | Every run ends in a named terminal state inside its deadline | Lost runs, runs stuck in running or failed |
INV-5 | No step exceeds its maximum attempts; the retry budget is never overdrawn | Retry storms, ignored retry_after_ms |
INV-6 | The stored recommendation carries the current policy_version and only evidence_ids that exist in the run | Citations to evidence the run never saw |
INV-7 | A degraded or fallback result says so, and a withdrawn recommendation is still in the table | A smaller answer passed off as a full one; evidence deleted |
The key property of an invariant is that it does not depend on the fault. The same check judges a crash, a 429 and a redelivery. That is what lets a suite run hundreds of different fault plans without writing an expected result for each one. The checker in invariants.py reads only the tables of the store; here are its entry point and the first check, verbatim:
def check_invariants(store: Store, run: dict[str, Any] | None, scenario: dict[str, Any] | None = None) -> list[dict[str, Any]]:
"""Check INV-1 to INV-7 on the store. `run` is the first run of the story (None if none was created).
The checks read only the store, so `scenario` is not needed to judge a
run; callers may pass it to keep the three arguments together.
Returns one result per invariant: `{"id": "INV-1", "passed": bool, "detail": str}`.
"""
checks = (_inv1, _inv2, _inv3, _inv4, _inv5, _inv6, _inv7)
results = []
for number, check in enumerate(checks, start=1):
problems = check(store) if run is not None else ["no run was created for the message"]
results.append({"id": f"INV-{number}", "passed": not problems, "detail": "; ".join(problems) or "holds"})
return results
def _inv1(store: Store) -> list[str]:
problems = []
runs, rows = store.tables["runs"], store.tables["recommendations"]
for submission_id in sorted({r["submission_id"] for r in runs}):
pending = sum(1 for r in rows if r["submission_id"] == submission_id and r["status"] == "pending_review")
needs_one = any(r["submission_id"] == submission_id and r["state"] in ENDS_WITH_A_ROW for r in runs)
if pending > 1:
problems.append(f"{submission_id} has {pending} pending_review rows (a duplicate write)")
if needs_one and pending == 0:
problems.append(f"{submission_id}: a run waits for a person but no pending_review row exists")
return problems
Common confusion. An invariant is not an expected output. "The recommendation isescalate" is an expected output: it belongs to one case and is wrong forUW-002. "At most one row waits for review" is an invariant: it holds for every case and every fault. A suite needs both, but only invariants scale to random fault plans.
The loop that ties these pieces together has five parts.
Diagram: One fault experiment · from steady state to a verdict, and back into the suite
Read left to right. First say what normal looks like. Then write the fault plan as data. The injector makes the chosen calls fail while the real workflow runs. Afterwards seven invariants judge only what is left in the store. A failure becomes a new named scenario, so the same story runs on every change. The agent here is a scripted stand-in and every fault is injected on purpose.
- Before
- Steady state: A measurable output that shows normal behaviour. For one run: UW-R01 ends in waiting_for_human with 1 row and 1 task. For many runs: the SLI.
- Fault plan: A list of rules written as data: a target, a kind of fault, and the call numbers that fail.
- During
- Inject: The injector counts calls per rule and raises the matching error when a rule fires. The workflow cannot tell an injected fault from a real one.
- Run the real workflow: The same eight steps that run without faults, with retries, keys, checkpoints and the rest switched on or off.
- After
- Check invariants: Seven statements that must hold after every run, read from the tables of the store, whatever the fault was.
- Keep
- Name it and keep it: A plan that broke an invariant is shrunk to its smallest form, given a story and an expected end state, and added to the named scenarios.
The suite: named scenarios, a matrix and random plans#
One scenario tests one story. The example's fault-injection suite (fault_suite.py) runs three kinds of story and checks all seven invariants after each.
| Part | What it runs | What it is for |
|---|---|---|
| 1 · Named scenarios | UW-R01 to UW-R15, each with its own fault plan and an expected end state | The faults someone thought of, each with a story a person can read |
| 2 · Matrix | The ten shared cases UW-001 to UW-010 × eight columns (no fault, and seven single faults) = 80 runs | Every case meets every common fault; also checks each case gets its expected recommendation when nothing fails |
| 3 · Random plans | 200 plans, each a case plus one to three faults drawn from a catalog of 23, with a seeded generator | The combinations nobody thought of |
Part 3 is where the opening surprise lives. A person writes scenarios for the faults they fear. A generator draws pairs and triples that a person would not pick: a poison message with a lease pause, a 429 with a crash before the checkpoint. Because the generator is seeded (seed 7 here), a failing random plan is not a flake: the same seed reproduces it exactly, and its plan can be copied into a new named scenario. From fault_suite.py, the function that draws one plan, verbatim:
def random_plan(rng: Mulberry32) -> tuple[str, list[dict[str, Any]]]:
"""One random story: a shared case and one to three different faults from the catalog."""
case_id = CASE_IDS[rng.below(len(CASE_IDS))]
picks: list[int] = []
for _ in range(1 + rng.below(3)):
pick = rng.below(len(CATALOG))
if pick not in picks:
picks.append(pick)
return case_id, [CATALOG[i] for i in picks]
Diagram: The fault-injection suite · three parts, one judge, 295 runs
Three sources of stories feed one checker. The named scenarios are the faults someone feared. The matrix crosses every shared case with every common single fault. The random plans draw one to three faults from a catalog with a fixed seed, which is where combinations nobody imagined appear. Every run, from any part, is judged by the same seven invariants. Counts are from this example, seed 7.
- Stories
- 15 named scenarios: UW-R01 to UW-R15, each with a story, a fault plan and an expected end state for both workflows.
- Matrix: 10 cases x 8 columns: The ten shared cases UW-001 to UW-010, each with no fault and with seven single faults: 80 runs.
- 200 random plans: A seeded generator picks a case and one to three different faults from a catalog of 23. Same seed, same plans.
- Judge
- Seven invariants: The same check for every run, so no plan needs a hand-written expected result.
- Verdict
- Careful: 295 of 295: With all ten protections on, no run broke an invariant in this example.
- Naive: 59 of 295: With every protection off, most runs break something. A duplicate review task appeared only in random plans.
Run it: the careful workflow#
The command runs the whole suite on the careful workflow, all ten protections on, in a scratch copy of the example. A successful run proves that in this simulation every invariant held after each of the 295 runs. It proves nothing about faults that are not in the catalog, and nothing about a real model or provider.
/opt/homebrew/bin/python3 fault_suite.py
Fault-injection suite · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
Part 1: the 15 named scenarios
scenario terminal state rows tasks expected broken invariants
UW-R01 waiting_for_human 1 1 match -
UW-R02 waiting_for_human 1 1 match -
UW-R03 degraded_request_info 1 1 match -
UW-R04 waiting_for_human 1 1 match -
UW-R05 waiting_for_human 1 1 match -
UW-R06 waiting_for_human 1 1 match -
UW-R07 waiting_for_human 1 1 match -
UW-R08 waiting_for_human 1 1 match -
UW-R09 waiting_for_human 1 1 match -
UW-R10 waiting_for_human 1 1 match -
UW-R11 dead_lettered 0 0 match -
UW-R12 failed_compensated 1 0 match -
UW-R13 waiting_for_human 1 1 match -
UW-R14 waiting_for_human 1 1 match -
UW-R15 waiting_for_human 1 1 match -
Part 2: the matrix, 10 shared cases x (no fault + 7 single faults). A cell shows the broken invariants.
case none provider-timeout model-429 malformed db-transient ack-lost crash-after-write duplicate
UW-001 ok ok ok ok ok ok ok ok
UW-002 ok ok ok ok ok ok ok ok
UW-003 ok ok ok ok ok ok ok ok
UW-004 ok ok ok ok ok ok ok ok
UW-005 ok ok ok ok ok ok ok ok
UW-006 ok ok ok ok ok ok ok ok
UW-007 ok ok ok ok ok ok ok ok
UW-008 ok ok ok ok ok ok ok ok
UW-009 ok ok ok ok ok ok ok ok
UW-010 ok ok ok ok ok ok ok ok
Part 3: 200 random fault plans (1 to 3 faults each, seeded)
end states: waiting_for_human 131, degraded_request_info 31, failed_compensated 22, dead_lettered 16
runs with every invariant holding: 200 of 200
Runs that break each invariant (scenarios / matrix / random):
INV-1 0 / 0 / 0
INV-2 0 / 0 / 0
INV-3 0 / 0 / 0
INV-4 0 / 0 / 0
INV-5 0 / 0 / 0
INV-6 0 / 0 / 0
INV-7 0 / 0 / 0
wrong recommendation without any fault: 0 of 10 cases
SUMMARY: 295 runs · 295 pass · 0 fail · every invariant held in every run
wrote out/suite-careful.json
Read the end states of part 3. Not every run ends in waiting_for_human, and that is correct: a provider that stays down ends honestly in degraded_request_info, a rejected review task in failed_compensated, a poison message in dead_lettered. The suite does not demand the happy ending. It demands an acceptable one.
Run it: the naive workflow#
The same suite with every protection off. The exit code is 1, because invariants break.
/opt/homebrew/bin/python3 fault_suite.py --variant naive
Fault-injection suite · variant naive · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose
Part 1: the 15 named scenarios
scenario terminal state rows tasks expected broken invariants
UW-R01 waiting_for_human 1 1 match -
UW-R02 waiting_for_human 1 1 match -
UW-R03 deadline_exceeded 0 0 match INV-4, INV-5
UW-R04 waiting_for_human 1 1 match INV-5
UW-R05 waiting_for_human 1 1 match INV-3, INV-6
UW-R06 waiting_for_human 2 1 match INV-1
UW-R07 waiting_for_human 1 1 match -
UW-R08 running 2 1 match INV-1, INV-4
UW-R09 running 1 1 match INV-4
UW-R10 waiting_for_human 2 2 match INV-1
UW-R11 running 0 0 match INV-4
UW-R12 failed 1 0 match INV-4
UW-R13 deadline_exceeded 0 0 match INV-4, INV-5
UW-R14 waiting_for_human 2 2 match INV-1
UW-R15 running 3 2 match INV-1, INV-4
Part 2: the matrix, 10 shared cases x (no fault + 7 single faults). A cell shows the broken invariants.
case none provider-timeout model-429 malformed db-transient ack-lost crash-after-write duplicate
UW-001 ok ok 5 3,6 ok 1 1,4 1
UW-002 ok ok 5 3,6 ok 1 1,4 1
UW-003 ok ok 5 3,6 ok 1 1,4 1
UW-004 ok ok 5 3,6 ok 1 1,4 1
UW-005 ok ok 5 3,6 ok 1 1,4 1
UW-006 4,5,R 4,5 4,5 4,5 4,5 4,5 4,5 4,5
UW-007 ok ok 5 3,6 ok 1 1,4 1
UW-008 ok ok 5 3,6 ok 1 1,4 1
UW-009 ok ok 5 3,6 ok 1 1,4 1
UW-010 ok ok 5 3,6 ok 1 1,4 1
Part 3: 200 random fault plans (1 to 3 faults each, seeded)
end states: waiting_for_human 81, running 57, deadline_exceeded 47, failed 15
runs with every invariant holding: 29 of 200
Runs that break each invariant (scenarios / matrix / random):
INV-1 5 / 27 / 43
INV-2 0 / 0 / 18
INV-3 1 / 9 / 26
INV-4 7 / 17 / 121
INV-5 3 / 17 / 72
INV-6 1 / 9 / 26
INV-7 0 / 0 / 0
wrong recommendation without any fault: 1 of 10 cases
SUMMARY: 295 runs · 59 pass · 236 fail · see the broken invariants above
wrote out/suite-naive.json
Three readings matter. First, the naive workflow passes three scenarios, UW-R01, UW-R02 and UW-R07. A team that tested only those would ship it: a transient fault before the commit is the case where an unprotected retry happens to work. Second, the matrix columns are uniform: every case breaks INV-1 under ack-lost, because the fault, not the case, decides the outcome. (UW-006 fails in every column because its provider never answers; R marks that it does not get its expected recommendation even with no added fault.) Third, INV-2 breaks only in the random plans, 18 times. No named scenario and no single fault produced a duplicate review task; only combinations did. That is the reason part 3 exists.
The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
In our insurance agent.INV-7never breaks under the naive workflow. That is not a sign of health: the naive workflow has no fallback route and no degradation, so it never produces the kind of resultINV-7checks. An invariant that always passes may be checking a path that never ran. Look for that before trusting a green column.
The game day: switching faults and protections by hand#
A suite runs in minutes and runs every time. A game day is a scheduled session in which a team causes failures on purpose, by hand or from a prepared plan, and watches what the system and the people do: which alerts fire, which dashboards help, which runbook steps work. The suite tests the code; a game day also tests the response around it, which is the subject of the next page.
The lab below is a game day on one run. It starts from the worst preset: all five faults on, all four protections off. Switch protections on one at a time and read which invariants come back. In the lab the remaining protections of the careful workflow, such as inbox deduplication and the breaker, stay on.
Without JavaScript, the same results, from the lab's own function run_scenario.py:gameday (seed 7) run in a copy of the example. All five faults are on in every row: a provider timeout twice, a model 429, a malformed model output, a transient store failure before the commit, and a crash after the write.
| Protections on | End state | Rows | Review tasks | Model calls | Invariants that fail |
|---|---|---|---|---|---|
| None (the preset) | waiting_for_human at 31,998 ms; broken output stored | 2 | 1 | 18 | INV-1, INV-3, INV-5, INV-6 |
| Retry with backoff only | waiting_for_human; broken output stored | 2 | 1 | 3 | INV-1, INV-3, INV-6 |
| Output validation only | waiting_for_human; 18 attempts at the model | 2 | 1 | 19 | INV-1, INV-5 |
| Idempotency key only | failed at 31,986 ms (a key conflict) | 1 | 0 | 18 | INV-3, INV-4, INV-5, INV-6 |
| Checkpoint only | waiting_for_human at 30,024 ms; broken output stored | 2 | 1 | 17 | INV-1, INV-3, INV-5, INV-6 |
| All four | waiting_for_human at 30,024 ms; escalate | 1 | 1 | 3 | none |
The fourth row is the opening surprise again, with more faults: adding the idempotency key alone turns a duplicate into a failed run with no review task, because the replay meets a different payload. A protection is not a module you add for a fixed gain. Protections depend on each other: the key assumes the payload of a replay is the same, which needs either validation before the first write or a checkpoint that keeps the first answer. Only the combination holds.
Production pitfall. The last row says every invariant holds, and it hides a cost. Switch off only the checkpoint in the lab and the invariants still hold, but the resumed process starts at step 1, calls the provider and the model again, and spends 4,020 cost units in place of 2,680 in this example. Invariants check correctness, not cost or latency. Track those as service level indicators next to them (SLOs).
Synthetic production scenarios#
Everything above ran inside the example, on a simulated clock. A real system adds what a simulation leaves out: real network behaviour, a real database, a real model provider and its quotas. The Principles of Chaos Engineering argue for experimenting on production traffic, because only production has the real request paths, and for keeping the blast radius small.
For an agent that writes recommendations for a person, "experiment in production" needs care. A real broker's submission must never be the subject of an experiment. The usual answer is a synthetic scenario: an invented input with a known expected outcome and a fault plan, run through the real production path. In this workflow it would be a test submission, such as one with the facts of UW-004, marked as synthetic from the moment it enters the queue.
| Rule for a synthetic production scenario | Why |
|---|---|
| Mark it at intake and carry the mark on every row it writes | So the cockpit never shows it to an underwriter as real work, and dashboards can separate it |
| Give it a known expected end state and run the seven invariants on its rows | So the run is judged, not only observed |
| Inject faults only at the boundary of this run (its own tool calls, its own write) | Blast radius: one run, never the shared provider for everyone |
| Set a stop condition before starting: an SLO burn alert, or any invariant broken on a real run | So the experiment ends before it harms real work |
| Count its cost units against a separate budget | Model calls in an experiment are real spending |
This is a design proposed here, not something the example runs. The example has no production. What it does have is the same fault plan format, so a scenario that ran in the simulation can run unchanged as a synthetic scenario elsewhere.
What this is not. How a failure becomes visible (logs, metrics, traces of an experiment) is observability: https://observe.sammuthu.com. Whether the recommendation itself got better or worse is measured by evaluation: https://eval.sammuthu.com. This page asks one question of every injected fault: does the outcome stay acceptable?
Dedicated fault-injection products and services exist for real infrastructure. This site does not describe their features here; any named tool belongs on Frameworks, from its own documentation.
The same idea for a coding agent#
A coding agent edits a repository, runs the tests, pushes a branch, opens a pull request and waits for CI. Its fault plan could read: the CI job times out once; the process crashes after git push and before the agent records the push; the call that opens the pull request loses its response. Its invariants: at most one open pull request per task; the branch on the remote contains exactly the commits the agent recorded; no merge without a person's approval. A seeded generator that draws pairs of those faults finds the same class of surprise as the insurance game day, such as a resumed agent that regenerates a slightly different commit and force-pushes over its own branch. The runtime where such faults are injected is the subject of https://harness.sammuthu.com; the CI path of https://diff-to-deploy.sammuthu.com.
How fault injection goes wrong#
| Mistake | What happens | Seen here |
|---|---|---|
| Faults only before the commit | Every design passes, even one with no key | Naive UW-R07 passes |
| Single faults only | Combinations go untested | INV-2 breaks only in random plans (18 naive runs) |
| Assertions on outputs instead of invariants | Each new plan needs a hand-written expectation, so few plans get written | The suite's 200 random plans have no per-plan expectation |
| Unseeded randomness | A failure cannot be reproduced, so it is dismissed as a flake | Seed 7 reproduces every run here |
| An invariant that never had a chance to fail | A green column that proves nothing | INV-7 under naive |
| Invariants only | Cost and latency regressions pass silently | Checkpoint off: 4,020 cost units, all invariants hold |
| Faults from a fixed list | A fault outside the list is never tried | faults.py:KINDS has no "slow but successful" provider |
The four blocks below hold the detailed layer.
Coverage: what the suite cannot claim295 green runs say nothing about a fault kind that is not in the catalog
The suite draws its faults from faults.py:KINDS: timeouts and 503s for the two read tools; 429, unavailable, timeout and malformed for the two model routes; a transient failure and a lost acknowledgement for the two writes, plus a permanent rejection for the review task; a duplicate or a poison message on the queue; a crash at a chosen step; a lease pause. The README says it plainly: a fault that is not in that list was not tried.
Missing kinds an expert would add next: a provider that answers slowly but successfully (a gray failure, see Failure modes); a model that answers well formed and wrong; a store that commits the key but not the row (impossible here because both are plain Python writes in one function); clock skew between workers; two workers truly in parallel rather than one stalled. Each needs a new fault kind and, for some, a richer simulation. Coverage of a fault-injection suite is a statement about the catalog, never about the system.
The combination space is also larger than 200 plans. With 23 catalog entries and one to three different faults per plan there are 2,047 distinct sets of faults (23 + 253 + 1,771) before the ten cases multiply them. Random sampling with a fixed seed gives breadth and reproducibility; it does not give completeness. A deliberate pairwise design (every pair of catalog entries at least once) is a reasonable next step when a suite is cheap, as this one is.
From a failing random plan to a named scenarioReproduce, shrink, name, keep
When part 3 finds a failing plan, the plan, the case and the seed are in out/suite-<variant>.json. The useful next move is to shrink it: remove one fault at a time and keep the smallest plan that still breaks the same invariant. The opening run of this page has the shape of a shrunk plan: two faults (malformed output once, crash after the write) and one protection set, the smallest story that still ends in failed. Here it is done by hand, because plans are short.
The shrunk plan then becomes a named scenario with a story, an expected end state for both variants and a "teaches" line, like UW-R01 to UW-R15. Named scenarios are the regression set: once a combination has broken the workflow, it runs on every change forever. The next page uses the same move for failures found in production.
Running experiments against a live agentBlast radius, abort conditions and the human queue
The Principles of Chaos Engineering ask for experiments on production traffic, automated to run continuously, with a minimized blast radius. For an agent three constraints shape that advice.
The human queue is part of the system. A synthetic recommendation that reaches the underwriter cockpit costs a person's time and can be mistaken for real work. Experiments must be invisible to the cockpit, or run in a cockpit view reserved for them.
Spending is real. Every injected model fault that forces a re-ask spends cost units on the real provider. Budget the experiment and stop it at a ceiling, as Concurrency does for runs.
Shared dependencies are shared. Making the real roof-data provider fail is not something the insurer controls, and making it look failed for all runs is an outage, not an experiment. Inject at the client of one run (the fault plan is per run here for that reason), never at the dependency.
Stop conditions come first: the SLO burn alert of SLOs firing, or any invariant failing on a non-synthetic run, ends every experiment at once.
Determinism is what makes a fault test a testSimulated time, seeded randomness and a scripted model
Every number on this page repeats exactly because three sources of variation are fixed: time is a simulated clock in milliseconds, jitter and ids come from a seeded mulberry32 generator, and the model is a scripted stand-in. A failing run is a bug, never a flake.
A real agent adds a source the example removes: the model. Asked twice, it may answer differently. In a fault test against a real model that variation is a fault in its own right, which the opening run imitated with a malformed first answer. Two practices keep such tests meaningful: record the model's answers once and replay them in the regression suite (so the test checks the workflow, not the model), and run a separate, smaller set of live trials whose result is a rate, not a pass or fail. The second belongs to evaluation; the first is a fault test.
Checklist#
| Question | Good answer |
|---|---|
| Does every recovery path (retry, replay, resume, compensation, dead letter, fallback) have a test that makes it run? | Yes, through a fault plan |
| Are fault plans data that can be stored, reviewed and replayed? | Yes, next to the scenario |
| Are faults injected after the commit as well as before it? | Yes; ack_lost and a crash before the checkpoint |
| Is every run judged by invariants that read only what is left? | Yes, all of them after every run |
| Do the tests include combinations, and are they seeded? | Yes: random plans with a fixed seed, failures reproducible |
| Has each invariant been seen to fail at least once? | Yes, on the unprotected variant |
| Are cost and latency tracked next to the invariants? | Yes, as service level indicators |
| Is there a scheduled game day that tests the people and the alerts too? | Yes, with a stop condition |
| Can a synthetic run in production never reach a real underwriter? | Yes: marked at intake, filtered in the cockpit |
In one breath#
A recovery path that never ran is a guess, so we make dependencies fail on purpose with a written, replayable fault plan and judge every run by invariants about what is left in the store, not by the output we expected. Named scenarios cover the faults we feared, a matrix crosses every case with every common fault, and seeded random plans find the combinations we did not imagine, like an idempotency key that turns a duplicate into a failed run when validation and checkpoints are missing. Invariants do not see cost or latency, and the catalog limits what was tried, so a game day also tests the alerts and the people.
Next: when a real failure gets through anyway#
The suite says the workflow keeps its invariants under every fault in its catalog. Production will still find a fault outside it, such as a provider that is slow without failing. When that happens, someone has to notice, take charge, limit the damage, tell the people affected, and afterwards turn the surprise into a new scenario. Incidents is about that response, and about how one bad day becomes a test that runs forever.