All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Operations · 12

Fault injection: breaking it on purpose, before production does

A recovery path that has never run is a guess. Injected faults, invariants that must hold after each one, synthetic production scenarios, and a game day you can replay.

What you will understand
Fault injection
Why a recovery path that has never run is a guess, and how a written fault plan makes a dependency fail on purpose, at a chosen call.
Invariants
How seven statements about what is left in the store judge every run, whatever failed on the way, and what they cannot see.
Suite and game day
How 15 named scenarios, a matrix and 200 seeded random plans test the combinations nobody thought of, and what a game day adds.

The underwriting agent of this site handles submission SUB-1004: the application says the roof is 9 years old, the roof-data provider says 22, so the expected recommendation is escalate, citing R-ROOF-03. The model recommends; a person decides.

Picture a team that has built the protections of the earlier pages one at a time and tested each one on its own. The idempotency key passed its test: in UW-R06 the acknowledgement of the write is lost, the retry carries the same key, and the store returns the stored row. Crash recovery passed its test too. Output validation is still on the backlog, and so are checkpoints. Every test is green.

Then someone runs a small game day. Two faults at once: the model's first answer is malformed, and the process crashes after the write. Retry with backoff and the idempotency key are on; validation and checkpoints are off. Here are selected lines of that run, verbatim, from the example's game-day function (run_scenario.py:gameday, seed 7) in a scratch copy of examples/uw-reliability. Note that the malformed answer at 1974 ms is logged as "attempt ok": with validation off, nothing looked at it.

   t_ms  worker    step                   att  event
   1974  worker-A  draft_recommendation     1  attempt ok
   1986  worker-A  record_recommendation    1  attempt ok
   1986  worker-A  record_recommendation    -  PROCESS CRASH: after the step, before checkpoint
  30000  worker-B  -                        -  run found: by message_id; resume_count 1
  30000  worker-B  load_submission          -  step done: no checkpoint
  31974  worker-B  draft_recommendation     1  attempt ok
  31986  worker-B  record_recommendation    1  attempt failed: KeyConflict: key run-7f3a:record_recommendation was first used with another payload
  31986  worker-B  record_recommendation    -  run ended: failed

Worker A stored the broken model output, because nothing checked it, and died. Worker B had no checkpoint, so it started at step 1 and asked the model again. This time the answer was well formed, and it differed from what was stored under the same key. The store did what an idempotency store must do with the same key and a different payload: it refused. The run ended in failed, which is not one of the four terminal states, with a broken recommendation in the table and no review task.

No single-fault test could have shown this. The key works. The crash recovery works. Their combination with a missing protection produced an outcome that nobody wrote a test for, because nobody imagined it. This page is about finding such outcomes before production does.

Plain English. A recovery path is code that runs only when something goes wrong. If nothing has gone wrong yet, that code has never run, and code that has never run usually has a bug. Fault injection makes things go wrong on purpose so the recovery code runs while you are watching.

Why waiting for real failures does not work#

Most of the protections on this site sit on paths that a healthy system never takes. The retry loop of Retries runs only after a timeout. The replay of Idempotency runs only after a lost acknowledgement. The resume of Checkpoints runs only after a crash. In this example a healthy run, UW-R01, touches none of them: one attempt per step, done at 1998 ms.

So the usual test suite, which feeds good inputs to a healthy system, says nothing about them. Waiting for production to exercise them has three problems. Real faults are rare, so a broken path can sit for months. They arrive at the worst time, during a provider outage when many runs fail together. And they arrive in combinations: a crash during a retry, a redelivery after a resume.

An agent makes this worse in two ways. Its runs are long, so many faults can land inside one run (UW-R15 has three). And one of its steps, the model, does not give the same answer twice, so a path that repeats the model step can behave differently on the second pass, as the opening run showed.

The way out is to stop waiting and cause the failures yourself.

Fault injection: making a dependency fail on purpose#

Fault injection is causing a chosen fault at a chosen place, on purpose, to see how the system responds. A tiny example: "the roof-data provider does not answer the first two calls for SUB-1004". In the example code nothing fails by chance; every fault is injected.

The written form of that intent is a fault plan: a list of rules, each naming a target, the kind of fault and which calls fail. Here is the plan of UW-R15, from its scenario file:

[{"target": "get_roof_assessment", "kind": "timeout", "on_calls": [1, 2]},
 {"target": "process", "kind": "crash", "after_step": "record_recommendation",
  "when": "before_checkpoint", "on_calls": [1]},
 {"target": "queue", "kind": "duplicate", "on_calls": [1]}]

Three rules, three faults: the provider times out on its first two calls; the process crashes after the write and before its checkpoint, once; the queue loses one acknowledgement, so it delivers the message again. A plan is data, not code, so it can be read by a person, stored next to the scenario, diffed in a review and replayed exactly.

The component that reads the plan and decides, call by call, whether a fault fires is the injector. In the example it is a counter per rule. Every call to a target is registered; when the count is in on_calls, the rule fires and the caller raises the matching error. From faults.py, the method FaultInjector.fire, verbatim:

def fire(self, target: str, **where: Any) -> dict[str, Any] | None:
    """Register one call to `target`. Returns the rule that fires on this call, or None.

    `where` narrows the match: `fire("process", after_step="record_recommendation",
    when="before_checkpoint")` only counts for rules that carry the same values.
    """
    for i, rule in enumerate(self.plan.rules):
        if rule["target"] != target or any(rule.get(k) != v for k, v in where.items()):
            continue
        self.hits[i] += 1
        if rule["on_calls"] == "always" or self.hits[i] in rule["on_calls"]:
            self.fired.append({"target": target, "kind": rule["kind"]})
            return rule                 # a call that one rule takes is not counted by the rules after it
    return None

Two design choices in it matter. The fault is decided by call number, not by a random draw, so the same plan gives the same run every time. And the injector sits at the boundary of each dependency (the tool call, the model call, the store write, the queue), which is where real faults appear as errors. The workflow code does not know a fault is injected; it sees a ToolTimeout or an AckLost exactly as it would see a real one.

Before AI. A Java team tests the same way with a test double: a payment client that throws on the second call, or a Kafka consumer killed between processing a record and committing its offset. What changes for an agent is the step in the middle whose output is not repeatable. A fault that forces a step to run again can change the content of the result, not only its timing, so the plan must include faults that repeat the model step.

Steady state and invariants: what "still works" means#

Injecting a fault is easy. Deciding whether the system survived it is the hard part. "It did not crash" is not enough: the naive workflow did not crash in UW-R06 and left two recommendation rows behind.

The Principles of Chaos Engineering, a short statement published at principlesofchaos.org, start an experiment from steady state: a measurable output of the system that shows normal behaviour, such as throughput, error rate or latency. You state a hypothesis that this output will not change when the fault is added, and then try to show that it does. The page stresses the output a user sees over the internals of the system.

For this workflow the steady state of one run is the outcome of UW-R01: one run ends in waiting_for_human, one pending_review row, one review task, inside the deadline. Across many runs it is the service level indicator of SLOs: the share of runs with an acceptable outcome. A fault experiment asks whether either moves.

For a single run, the example turns steady state into seven statements that must be true afterwards, whatever happened on the way. Each one is an invariant: a property that must hold after every run, checked only on what is left. A tiny example: "a submission never has two recommendation rows waiting for review". The seven of this site:

IdWhat must be true after every runWhat it catches
INV-1At most one pending_review row per submission, and exactly one when the run ended in waiting_for_human or degraded_request_infoDuplicate writes; a run that claims success with no row
INV-2At most one review task per runA duplicate task in the underwriter cockpit
INV-3No decisions row written by the agent; every stored recommendation has requires_human: trueThe agent acting as the decider; broken output stored as a recommendation
INV-4Every run ends in a named terminal state inside its deadlineLost runs, runs stuck in running or failed
INV-5No step exceeds its maximum attempts; the retry budget is never overdrawnRetry storms, ignored retry_after_ms
INV-6The stored recommendation carries the current policy_version and only evidence_ids that exist in the runCitations to evidence the run never saw
INV-7A degraded or fallback result says so, and a withdrawn recommendation is still in the tableA smaller answer passed off as a full one; evidence deleted

The key property of an invariant is that it does not depend on the fault. The same check judges a crash, a 429 and a redelivery. That is what lets a suite run hundreds of different fault plans without writing an expected result for each one. The checker in invariants.py reads only the tables of the store; here are its entry point and the first check, verbatim:

def check_invariants(store: Store, run: dict[str, Any] | None, scenario: dict[str, Any] | None = None) -> list[dict[str, Any]]:
    """Check INV-1 to INV-7 on the store. `run` is the first run of the story (None if none was created).

    The checks read only the store, so `scenario` is not needed to judge a
    run; callers may pass it to keep the three arguments together.
    Returns one result per invariant: `{"id": "INV-1", "passed": bool, "detail": str}`.
    """
    checks = (_inv1, _inv2, _inv3, _inv4, _inv5, _inv6, _inv7)
    results = []
    for number, check in enumerate(checks, start=1):
        problems = check(store) if run is not None else ["no run was created for the message"]
        results.append({"id": f"INV-{number}", "passed": not problems, "detail": "; ".join(problems) or "holds"})
    return results


def _inv1(store: Store) -> list[str]:
    problems = []
    runs, rows = store.tables["runs"], store.tables["recommendations"]
    for submission_id in sorted({r["submission_id"] for r in runs}):
        pending = sum(1 for r in rows if r["submission_id"] == submission_id and r["status"] == "pending_review")
        needs_one = any(r["submission_id"] == submission_id and r["state"] in ENDS_WITH_A_ROW for r in runs)
        if pending > 1:
            problems.append(f"{submission_id} has {pending} pending_review rows (a duplicate write)")
        if needs_one and pending == 0:
            problems.append(f"{submission_id}: a run waits for a person but no pending_review row exists")
    return problems
Common confusion. An invariant is not an expected output. "The recommendation is escalate" is an expected output: it belongs to one case and is wrong for UW-002. "At most one row waits for review" is an invariant: it holds for every case and every fault. A suite needs both, but only invariants scale to random fault plans.

The loop that ties these pieces together has five parts.

Diagram: One fault experiment · from steady state to a verdict, and back into the suite

Read left to right. First say what normal looks like. Then write the fault plan as data. The injector makes the chosen calls fail while the real workflow runs. Afterwards seven invariants judge only what is left in the store. A failure becomes a new named scenario, so the same story runs on every change. The agent here is a scripted stand-in and every fault is injected on purpose.

  1. Before
    1. Steady state: A measurable output that shows normal behaviour. For one run: UW-R01 ends in waiting_for_human with 1 row and 1 task. For many runs: the SLI.
    2. Fault plan: A list of rules written as data: a target, a kind of fault, and the call numbers that fail.
  2. During
    1. Inject: The injector counts calls per rule and raises the matching error when a rule fires. The workflow cannot tell an injected fault from a real one.
    2. Run the real workflow: The same eight steps that run without faults, with retries, keys, checkpoints and the rest switched on or off.
  3. After
    1. Check invariants: Seven statements that must hold after every run, read from the tables of the store, whatever the fault was.
  4. Keep
    1. Name it and keep it: A plan that broke an invariant is shrunk to its smallest form, given a story and an expected end state, and added to the named scenarios.

The suite: named scenarios, a matrix and random plans#

One scenario tests one story. The example's fault-injection suite (fault_suite.py) runs three kinds of story and checks all seven invariants after each.

PartWhat it runsWhat it is for
1 · Named scenariosUW-R01 to UW-R15, each with its own fault plan and an expected end stateThe faults someone thought of, each with a story a person can read
2 · MatrixThe ten shared cases UW-001 to UW-010 × eight columns (no fault, and seven single faults) = 80 runsEvery case meets every common fault; also checks each case gets its expected recommendation when nothing fails
3 · Random plans200 plans, each a case plus one to three faults drawn from a catalog of 23, with a seeded generatorThe combinations nobody thought of

Part 3 is where the opening surprise lives. A person writes scenarios for the faults they fear. A generator draws pairs and triples that a person would not pick: a poison message with a lease pause, a 429 with a crash before the checkpoint. Because the generator is seeded (seed 7 here), a failing random plan is not a flake: the same seed reproduces it exactly, and its plan can be copied into a new named scenario. From fault_suite.py, the function that draws one plan, verbatim:

def random_plan(rng: Mulberry32) -> tuple[str, list[dict[str, Any]]]:
    """One random story: a shared case and one to three different faults from the catalog."""
    case_id = CASE_IDS[rng.below(len(CASE_IDS))]
    picks: list[int] = []
    for _ in range(1 + rng.below(3)):
        pick = rng.below(len(CATALOG))
        if pick not in picks:
            picks.append(pick)
    return case_id, [CATALOG[i] for i in picks]

Diagram: The fault-injection suite · three parts, one judge, 295 runs

Three sources of stories feed one checker. The named scenarios are the faults someone feared. The matrix crosses every shared case with every common single fault. The random plans draw one to three faults from a catalog with a fixed seed, which is where combinations nobody imagined appear. Every run, from any part, is judged by the same seven invariants. Counts are from this example, seed 7.

  1. Stories
    1. 15 named scenarios: UW-R01 to UW-R15, each with a story, a fault plan and an expected end state for both workflows.
    2. Matrix: 10 cases x 8 columns: The ten shared cases UW-001 to UW-010, each with no fault and with seven single faults: 80 runs.
    3. 200 random plans: A seeded generator picks a case and one to three different faults from a catalog of 23. Same seed, same plans.
  2. Judge
    1. Seven invariants: The same check for every run, so no plan needs a hand-written expected result.
  3. Verdict
    1. Careful: 295 of 295: With all ten protections on, no run broke an invariant in this example.
    2. Naive: 59 of 295: With every protection off, most runs break something. A duplicate review task appeared only in random plans.

Run it: the careful workflow#

The command runs the whole suite on the careful workflow, all ten protections on, in a scratch copy of the example. A successful run proves that in this simulation every invariant held after each of the 295 runs. It proves nothing about faults that are not in the catalog, and nothing about a real model or provider.

/opt/homebrew/bin/python3 fault_suite.py
Fault-injection suite · variant careful · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose

Part 1: the 15 named scenarios
scenario  terminal state         rows tasks  expected  broken invariants
UW-R01    waiting_for_human         1     1  match     -
UW-R02    waiting_for_human         1     1  match     -
UW-R03    degraded_request_info     1     1  match     -
UW-R04    waiting_for_human         1     1  match     -
UW-R05    waiting_for_human         1     1  match     -
UW-R06    waiting_for_human         1     1  match     -
UW-R07    waiting_for_human         1     1  match     -
UW-R08    waiting_for_human         1     1  match     -
UW-R09    waiting_for_human         1     1  match     -
UW-R10    waiting_for_human         1     1  match     -
UW-R11    dead_lettered             0     0  match     -
UW-R12    failed_compensated        1     0  match     -
UW-R13    waiting_for_human         1     1  match     -
UW-R14    waiting_for_human         1     1  match     -
UW-R15    waiting_for_human         1     1  match     -

Part 2: the matrix, 10 shared cases x (no fault + 7 single faults). A cell shows the broken invariants.
case    none              provider-timeout  model-429         malformed         db-transient      ack-lost          crash-after-write duplicate        
UW-001  ok                ok                ok                ok                ok                ok                ok                ok               
UW-002  ok                ok                ok                ok                ok                ok                ok                ok               
UW-003  ok                ok                ok                ok                ok                ok                ok                ok               
UW-004  ok                ok                ok                ok                ok                ok                ok                ok               
UW-005  ok                ok                ok                ok                ok                ok                ok                ok               
UW-006  ok                ok                ok                ok                ok                ok                ok                ok               
UW-007  ok                ok                ok                ok                ok                ok                ok                ok               
UW-008  ok                ok                ok                ok                ok                ok                ok                ok               
UW-009  ok                ok                ok                ok                ok                ok                ok                ok               
UW-010  ok                ok                ok                ok                ok                ok                ok                ok               

Part 3: 200 random fault plans (1 to 3 faults each, seeded)
  end states: waiting_for_human 131, degraded_request_info 31, failed_compensated 22, dead_lettered 16
  runs with every invariant holding: 200 of 200

Runs that break each invariant (scenarios / matrix / random):
  INV-1    0 /   0 /   0
  INV-2    0 /   0 /   0
  INV-3    0 /   0 /   0
  INV-4    0 /   0 /   0
  INV-5    0 /   0 /   0
  INV-6    0 /   0 /   0
  INV-7    0 /   0 /   0
  wrong recommendation without any fault: 0 of 10 cases
SUMMARY: 295 runs · 295 pass · 0 fail · every invariant held in every run
wrote out/suite-careful.json

Read the end states of part 3. Not every run ends in waiting_for_human, and that is correct: a provider that stays down ends honestly in degraded_request_info, a rejected review task in failed_compensated, a poison message in dead_lettered. The suite does not demand the happy ending. It demands an acceptable one.

Run it: the naive workflow#

The same suite with every protection off. The exit code is 1, because invariants break.

/opt/homebrew/bin/python3 fault_suite.py --variant naive
Fault-injection suite · variant naive · seed 7 · agent: scripted stand-in (no model called) · every fault injected on purpose

Part 1: the 15 named scenarios
scenario  terminal state         rows tasks  expected  broken invariants
UW-R01    waiting_for_human         1     1  match     -
UW-R02    waiting_for_human         1     1  match     -
UW-R03    deadline_exceeded         0     0  match     INV-4, INV-5
UW-R04    waiting_for_human         1     1  match     INV-5
UW-R05    waiting_for_human         1     1  match     INV-3, INV-6
UW-R06    waiting_for_human         2     1  match     INV-1
UW-R07    waiting_for_human         1     1  match     -
UW-R08    running                   2     1  match     INV-1, INV-4
UW-R09    running                   1     1  match     INV-4
UW-R10    waiting_for_human         2     2  match     INV-1
UW-R11    running                   0     0  match     INV-4
UW-R12    failed                    1     0  match     INV-4
UW-R13    deadline_exceeded         0     0  match     INV-4, INV-5
UW-R14    waiting_for_human         2     2  match     INV-1
UW-R15    running                   3     2  match     INV-1, INV-4

Part 2: the matrix, 10 shared cases x (no fault + 7 single faults). A cell shows the broken invariants.
case    none              provider-timeout  model-429         malformed         db-transient      ack-lost          crash-after-write duplicate        
UW-001  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-002  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-003  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-004  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-005  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-006  4,5,R             4,5               4,5               4,5               4,5               4,5               4,5               4,5              
UW-007  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-008  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-009  ok                ok                5                 3,6               ok                1                 1,4               1                
UW-010  ok                ok                5                 3,6               ok                1                 1,4               1                

Part 3: 200 random fault plans (1 to 3 faults each, seeded)
  end states: waiting_for_human 81, running 57, deadline_exceeded 47, failed 15
  runs with every invariant holding: 29 of 200

Runs that break each invariant (scenarios / matrix / random):
  INV-1    5 /  27 /  43
  INV-2    0 /   0 /  18
  INV-3    1 /   9 /  26
  INV-4    7 /  17 / 121
  INV-5    3 /  17 /  72
  INV-6    1 /   9 /  26
  INV-7    0 /   0 /   0
  wrong recommendation without any fault: 1 of 10 cases
SUMMARY: 295 runs · 59 pass · 236 fail · see the broken invariants above
wrote out/suite-naive.json

Three readings matter. First, the naive workflow passes three scenarios, UW-R01, UW-R02 and UW-R07. A team that tested only those would ship it: a transient fault before the commit is the case where an unprotected retry happens to work. Second, the matrix columns are uniform: every case breaks INV-1 under ack-lost, because the fault, not the case, decides the outcome. (UW-006 fails in every column because its provider never answers; R marks that it does not get its expected recommendation even with no added fault.) Third, INV-2 breaks only in the random plans, 18 times. No named scenario and no single fault produced a duplicate review task; only combinations did. That is the reason part 3 exists.

The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

In our insurance agent. INV-7 never breaks under the naive workflow. That is not a sign of health: the naive workflow has no fallback route and no degradation, so it never produces the kind of result INV-7 checks. An invariant that always passes may be checking a path that never ran. Look for that before trusting a green column.

The game day: switching faults and protections by hand#

A suite runs in minutes and runs every time. A game day is a scheduled session in which a team causes failures on purpose, by hand or from a prepared plan, and watches what the system and the people do: which alerts fire, which dashboards help, which runbook steps work. The suite tests the code; a game day also tests the response around it, which is the subject of the next page.

The lab below is a game day on one run. It starts from the worst preset: all five faults on, all four protections off. Switch protections on one at a time and read which invariants come back. In the lab the remaining protections of the careful workflow, such as inbox deduplication and the breaker, stay on.

Without JavaScript, the same results, from the lab's own function run_scenario.py:gameday (seed 7) run in a copy of the example. All five faults are on in every row: a provider timeout twice, a model 429, a malformed model output, a transient store failure before the commit, and a crash after the write.

Protections onEnd stateRowsReview tasksModel callsInvariants that fail
None (the preset)waiting_for_human at 31,998 ms; broken output stored2118INV-1, INV-3, INV-5, INV-6
Retry with backoff onlywaiting_for_human; broken output stored213INV-1, INV-3, INV-6
Output validation onlywaiting_for_human; 18 attempts at the model2119INV-1, INV-5
Idempotency key onlyfailed at 31,986 ms (a key conflict)1018INV-3, INV-4, INV-5, INV-6
Checkpoint onlywaiting_for_human at 30,024 ms; broken output stored2117INV-1, INV-3, INV-5, INV-6
All fourwaiting_for_human at 30,024 ms; escalate113none

The fourth row is the opening surprise again, with more faults: adding the idempotency key alone turns a duplicate into a failed run with no review task, because the replay meets a different payload. A protection is not a module you add for a fixed gain. Protections depend on each other: the key assumes the payload of a replay is the same, which needs either validation before the first write or a checkpoint that keeps the first answer. Only the combination holds.

Production pitfall. The last row says every invariant holds, and it hides a cost. Switch off only the checkpoint in the lab and the invariants still hold, but the resumed process starts at step 1, calls the provider and the model again, and spends 4,020 cost units in place of 2,680 in this example. Invariants check correctness, not cost or latency. Track those as service level indicators next to them (SLOs).

Synthetic production scenarios#

Everything above ran inside the example, on a simulated clock. A real system adds what a simulation leaves out: real network behaviour, a real database, a real model provider and its quotas. The Principles of Chaos Engineering argue for experimenting on production traffic, because only production has the real request paths, and for keeping the blast radius small.

For an agent that writes recommendations for a person, "experiment in production" needs care. A real broker's submission must never be the subject of an experiment. The usual answer is a synthetic scenario: an invented input with a known expected outcome and a fault plan, run through the real production path. In this workflow it would be a test submission, such as one with the facts of UW-004, marked as synthetic from the moment it enters the queue.

Rule for a synthetic production scenarioWhy
Mark it at intake and carry the mark on every row it writesSo the cockpit never shows it to an underwriter as real work, and dashboards can separate it
Give it a known expected end state and run the seven invariants on its rowsSo the run is judged, not only observed
Inject faults only at the boundary of this run (its own tool calls, its own write)Blast radius: one run, never the shared provider for everyone
Set a stop condition before starting: an SLO burn alert, or any invariant broken on a real runSo the experiment ends before it harms real work
Count its cost units against a separate budgetModel calls in an experiment are real spending

This is a design proposed here, not something the example runs. The example has no production. What it does have is the same fault plan format, so a scenario that ran in the simulation can run unchanged as a synthetic scenario elsewhere.

What this is not. How a failure becomes visible (logs, metrics, traces of an experiment) is observability: https://observe.sammuthu.com. Whether the recommendation itself got better or worse is measured by evaluation: https://eval.sammuthu.com. This page asks one question of every injected fault: does the outcome stay acceptable?

Dedicated fault-injection products and services exist for real infrastructure. This site does not describe their features here; any named tool belongs on Frameworks, from its own documentation.

The same idea for a coding agent#

A coding agent edits a repository, runs the tests, pushes a branch, opens a pull request and waits for CI. Its fault plan could read: the CI job times out once; the process crashes after git push and before the agent records the push; the call that opens the pull request loses its response. Its invariants: at most one open pull request per task; the branch on the remote contains exactly the commits the agent recorded; no merge without a person's approval. A seeded generator that draws pairs of those faults finds the same class of surprise as the insurance game day, such as a resumed agent that regenerates a slightly different commit and force-pushes over its own branch. The runtime where such faults are injected is the subject of https://harness.sammuthu.com; the CI path of https://diff-to-deploy.sammuthu.com.

How fault injection goes wrong#

MistakeWhat happensSeen here
Faults only before the commitEvery design passes, even one with no keyNaive UW-R07 passes
Single faults onlyCombinations go untestedINV-2 breaks only in random plans (18 naive runs)
Assertions on outputs instead of invariantsEach new plan needs a hand-written expectation, so few plans get writtenThe suite's 200 random plans have no per-plan expectation
Unseeded randomnessA failure cannot be reproduced, so it is dismissed as a flakeSeed 7 reproduces every run here
An invariant that never had a chance to failA green column that proves nothingINV-7 under naive
Invariants onlyCost and latency regressions pass silentlyCheckpoint off: 4,020 cost units, all invariants hold
Faults from a fixed listA fault outside the list is never triedfaults.py:KINDS has no "slow but successful" provider

The four blocks below hold the detailed layer.

Coverage: what the suite cannot claim295 green runs say nothing about a fault kind that is not in the catalog

The suite draws its faults from faults.py:KINDS: timeouts and 503s for the two read tools; 429, unavailable, timeout and malformed for the two model routes; a transient failure and a lost acknowledgement for the two writes, plus a permanent rejection for the review task; a duplicate or a poison message on the queue; a crash at a chosen step; a lease pause. The README says it plainly: a fault that is not in that list was not tried.

Missing kinds an expert would add next: a provider that answers slowly but successfully (a gray failure, see Failure modes); a model that answers well formed and wrong; a store that commits the key but not the row (impossible here because both are plain Python writes in one function); clock skew between workers; two workers truly in parallel rather than one stalled. Each needs a new fault kind and, for some, a richer simulation. Coverage of a fault-injection suite is a statement about the catalog, never about the system.

The combination space is also larger than 200 plans. With 23 catalog entries and one to three different faults per plan there are 2,047 distinct sets of faults (23 + 253 + 1,771) before the ten cases multiply them. Random sampling with a fixed seed gives breadth and reproducibility; it does not give completeness. A deliberate pairwise design (every pair of catalog entries at least once) is a reasonable next step when a suite is cheap, as this one is.

From a failing random plan to a named scenarioReproduce, shrink, name, keep

When part 3 finds a failing plan, the plan, the case and the seed are in out/suite-<variant>.json. The useful next move is to shrink it: remove one fault at a time and keep the smallest plan that still breaks the same invariant. The opening run of this page has the shape of a shrunk plan: two faults (malformed output once, crash after the write) and one protection set, the smallest story that still ends in failed. Here it is done by hand, because plans are short.

The shrunk plan then becomes a named scenario with a story, an expected end state for both variants and a "teaches" line, like UW-R01 to UW-R15. Named scenarios are the regression set: once a combination has broken the workflow, it runs on every change forever. The next page uses the same move for failures found in production.

Running experiments against a live agentBlast radius, abort conditions and the human queue

The Principles of Chaos Engineering ask for experiments on production traffic, automated to run continuously, with a minimized blast radius. For an agent three constraints shape that advice.

The human queue is part of the system. A synthetic recommendation that reaches the underwriter cockpit costs a person's time and can be mistaken for real work. Experiments must be invisible to the cockpit, or run in a cockpit view reserved for them.

Spending is real. Every injected model fault that forces a re-ask spends cost units on the real provider. Budget the experiment and stop it at a ceiling, as Concurrency does for runs.

Shared dependencies are shared. Making the real roof-data provider fail is not something the insurer controls, and making it look failed for all runs is an outage, not an experiment. Inject at the client of one run (the fault plan is per run here for that reason), never at the dependency.

Stop conditions come first: the SLO burn alert of SLOs firing, or any invariant failing on a non-synthetic run, ends every experiment at once.

Determinism is what makes a fault test a testSimulated time, seeded randomness and a scripted model

Every number on this page repeats exactly because three sources of variation are fixed: time is a simulated clock in milliseconds, jitter and ids come from a seeded mulberry32 generator, and the model is a scripted stand-in. A failing run is a bug, never a flake.

A real agent adds a source the example removes: the model. Asked twice, it may answer differently. In a fault test against a real model that variation is a fault in its own right, which the opening run imitated with a malformed first answer. Two practices keep such tests meaningful: record the model's answers once and replay them in the regression suite (so the test checks the workflow, not the model), and run a separate, smaller set of live trials whose result is a rate, not a pass or fail. The second belongs to evaluation; the first is a fault test.

Checklist#

QuestionGood answer
Does every recovery path (retry, replay, resume, compensation, dead letter, fallback) have a test that makes it run?Yes, through a fault plan
Are fault plans data that can be stored, reviewed and replayed?Yes, next to the scenario
Are faults injected after the commit as well as before it?Yes; ack_lost and a crash before the checkpoint
Is every run judged by invariants that read only what is left?Yes, all of them after every run
Do the tests include combinations, and are they seeded?Yes: random plans with a fixed seed, failures reproducible
Has each invariant been seen to fail at least once?Yes, on the unprotected variant
Are cost and latency tracked next to the invariants?Yes, as service level indicators
Is there a scheduled game day that tests the people and the alerts too?Yes, with a stop condition
Can a synthetic run in production never reach a real underwriter?Yes: marked at intake, filtered in the cockpit

In one breath#

A recovery path that never ran is a guess, so we make dependencies fail on purpose with a written, replayable fault plan and judge every run by invariants about what is left in the store, not by the output we expected. Named scenarios cover the faults we feared, a matrix crosses every case with every common fault, and seeded random plans find the combinations we did not imagine, like an idempotency key that turns a duplicate into a failed run when validation and checkpoints are missing. Invariants do not see cost or latency, and the catalog limits what was tried, so a game day also tests the alerts and the people.

Next: when a real failure gets through anyway#

The suite says the workflow keeps its invariants under every fault in its catalog. Production will still find a fault outside it, such as a provider that is slow without failing. When that happens, someone has to notice, take charge, limit the damage, tell the people affected, and afterwards turn the surprise into a new scenario. Incidents is about that response, and about how one bad day becomes a test that runs forever.