All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
About

About this site and its sources

What this site is, what it is not, where its ideas come from, the sister sites, and how the facts were checked.

This is a teaching site about keeping the outcomes of agentic software acceptable when its parts fail. It starts from one sentence, "reliability means a system keeps delivering acceptable outcomes despite failures, retries, load and changing dependencies", and climbs one mechanism at a time: timeouts and retries, idempotency, queues, checkpoints and sagas, circuit breakers, concurrency limits, service level objectives, deployment, fallback routes, fault injection and incident response. At every step it asks the same question: what is still true after this fails?

It is written for three readers at once: someone new to building with models, an experienced backend engineer who knows Java, Spring, Kafka and relational databases, and a principal engineer who wants the trade-offs and the failure cases. The first reading of each page is for the first two; the folded blocks are for the third.

One example runs through every page. An insurer receives a broker's request for a property quote. An underwriting agent reads the submission, asks an outside provider about the roof, retrieves the guidance, runs the rules and writes a recommendation, which a human underwriter then decides on. The model recommends; a person decides. Every page breaks that workflow in one way and repairs it.

What it is and what it is not#

It isIt is not
A ladder: one mechanism per page, each word in plain English with a tiny example before it is usedA standard, or the only way to divide the subject
One running example, the eight-step insurance workflow and its fifteen fault scenarios UW-R01 to UW-R15A record of any real insurer, broker, applicant or provider
A demonstration of mechanisms: what a retry, a key, a checkpoint or a breaker keeps true, shown by injecting the faultA measurement of any model, provider or product. No page reports how reliable a real system is
Code that was run: a small example in standard-library Python, with its real output on the pagesA framework to install. It is short so that it can be read whole, and it leaves things out on purpose
Tools described from their own documentation, by what each keeps reliable for youA ranking or a buying guide. No tool is called best, and no price is quoted
Designs, labelled as designs, where the site proposes something (the key format, the escalation path)Advice that a design fits every system. Each page says what its design does not solve

The honest boundary#

The insurance example is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider, every timing and every number are invented. Nothing here reports the reliability or the quality of a real system or model. The rules of the policy fixture (for example "a roof older than 25 years in wind zone C is not eligible") are synthetic, not insurance practice.

The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

The table says what stands in for what.

In the exampleWhat it really is
The underwriting agentA script that writes what the rules imply. No model is called; there is no network and no key
The model providerA stand-in with two routes, a fixed latency and a fixed cost in cost units (simulated tokens, never money)
The roof-data provider and the other toolsStand-ins that return what the case file tells them, after a fixed latency
The store, the queue, the checkpointsPython lists and dictionaries in one process. No isolation levels, no replication, no real disk
TimeA simulated clock in milliseconds. Nothing waits
FaultsWritten in a fault plan per scenario, or drawn from a seeded generator, so every reader sees the same run
The human underwriterNot simulated. The automated run ends when the review task is open
The synthetic week of runs behind the SLO labThree invented profiles (healthy, a provider brown-out, a bad deployment), not measurements

The fault-injection suite of the example runs 295 runs (15 scenarios, 80 matrix runs and 200 random fault plans). Under the careful workflow every invariant holds in every run; under the naive workflow 59 runs pass. Those counts come from a script and a seed. They show that the mechanisms do what the pages say, against the faults the example knows how to inject, and nothing more. The full output is on The insurance lab.

How the facts were checked#

Kind of claimHow it was checked
A named standard or toolRead on its own documentation on the day of writing, 2026-11-17, and logged with the address and the date. Where the documentation is silent, the page says "the documentation read here does not say". Statements about tools live on Frameworks only; other pages name a tool and link there
An idea from a sourceThe source was opened and read on the day of writing, logged in the fact file of the page that uses it, and cited where the idea is taught
A surpriseA deprecation, a moved page or a new revision found during reading is marked "needs a second read" in the fact file, so it can be fetched again before it is relied on
The exampleRun. Every command a page shows was executed in a scratch copy of examples/uw-reliability, and the output on the page is what it printed. The folder has 77 unit tests
The labsThey read data generated from the example with a fixed seed. Their arithmetic is a second implementation of the Python functions, and a test fails if the two disagree
NumbersSynthetic and marked "in this example" or "illustrative" unless a source is linked. A number from the example is the result of a script, never a measurement
Other codeA block in a tool's API says how far it was checked: "documented shape; not run here" (compiled or parsed only)
Links and structureA build step checks every internal link, every diagram and every glossary reference before the site is published

The fact files sit in the repository under docs/facts/, one per page: each row is a claim, the address it came from, and the date it was read. Documentation changes. Treat every statement about a tool as true on the day it was read, and read the source again before you build on it.

What is original here and what is borrowed#

Most of the mechanisms on this site were published by other people first, many of them long before agents. The site uses those ideas with attribution and does not copy their prose, examples or figures.

PartStatus
Timeouts, retries, backoff with jitter, idempotent APIs, load shedding, fallbackPractice described in the Amazon Builders' Library and Google's SRE books; cited on the pages that teach them
Error budgets, burn-rate alerts, cascading failures, incident roles, blameless postmortemsFrom Google's SRE book and workbook
Fault, error and failure as separate wordsFrom the published taxonomy of dependable computing (see the reading list)
Saga and compensationFrom the 1987 paper on sagas; the pivot step from Microsoft's saga pattern page
Circuit breaker, bulkhead, feature toggle, transactional outbox, fencing tokenEstablished patterns; each page cites the description it read
Chaos engineering and steady stateFrom the Principles of Chaos Engineering
Hedged requests, tail latencyFrom "The Tail at Scale"
Little's lawFrom its author's 50th-anniversary paper
The eight-step workflow, the fifteen scenarios UW-R01 to UW-R15, the seven invariants, the fault plansWritten for this site. All synthetic
The key design <run_id>:<step id>, the one-trace-per-run rule, the approval path on AdvancedDesigns proposed on this site
The example code, its 77 tests, the fault-injection suiteWritten for this site, in standard-library Python
The interactive labs and the diagramsWritten for this site
The shared cases UW-001 to UW-010 and the policy fixtureShared with the sister sites of the series; synthetic

Where a page takes an idea from a source, it says so. Where this site proposes something, it calls it a design.

Sources and reading list#

Every source below was opened on the day the pages were written and is logged, with the claims taken from it, in the fact file of the page named in the last column. The titles are the sources' own.

Reliability practice#

SourceWhy read itUsed on
Site Reliability Engineering, GoogleThe book's chapters on embracing risk, service level objectives, handling overload, cascading failures, data integrity, managing incidents and postmortem culture; the introduction for capacity planningStart, Four qualities, Failure modes, Retries, Breakers, SLOs, Incidents, Advanced
The Site Reliability Workbook, GoogleImplementing SLOs, alerting on SLOs and canarying releasesSLOs, Deployment
Timeouts, retries and backoff with jitter, Amazon Builders' LibraryChoosing a timeout from latency, retrying only what may succeed, and jitterFailure modes, Retries, Idempotency
Making retries safe with idempotent APIs, Amazon Builders' LibraryA request identifier chosen by the callerIdempotency
Using load shedding to avoid overload, Amazon Builders' LibraryWhat to refuse first when a service is overloadedBreakers
Avoiding insurmountable queue backlogs, Amazon Builders' LibraryBacklogs, dead-letter queues and messages that can never succeedQueues
Avoiding fallback in distributed systems, Amazon Builders' LibraryWhy a fallback path that rarely runs is a risk of its ownFallback
Leader election in distributed systems and Fairness in multi-tenant systems, Amazon Builders' LibraryLeases and their hazards; token buckets and per-tenant quotasConcurrency
Automating safe, hands-off deployments and Ensuring rollback safety during deployments, Amazon Builders' LibraryStaged rollout, bake time, automatic rollback, and changes that can be rolled backDeployment
Principles of Chaos EngineeringSteady state, hypotheses and experiments with real-world eventsFault injection

The Builders' Library articles now open on AWS Builder Center: the old aws.amazon.com/builders-library/... addresses redirect there, and the links above point at the pages that were read.

Patterns#

SourceWhy read itUsed on
Sagas, 1987 (a PDF hosted for a university course)The origin of the saga: a long transaction as a sequence of smaller ones, each with a compensationCheckpoints
Saga design pattern and Bulkhead pattern, Microsoft Azure Architecture CenterThe pivot step after which nothing can be compensated; isolating resources into poolsCheckpoints, Breakers
Circuit Breaker and Feature Toggles, martinfowler.comThe breaker's states; the kinds of toggle and the kill switchBreakers, Deployment
Transactional outbox, microservices.ioWriting the message in the same transaction as the changeIdempotency, Queues
How to do distributed locking, 2016Why a lease alone does not stop a paused worker, and the fencing tokenConcurrency

Research#

SourceWhy read itUsed on
Basic Concepts and Taxonomy of Dependable and Secure Computing, 2004 (record of the University of Maryland repository)Fault, error and failure as three different thingsFour qualities
Gray Failure: The Achilles' Heel of Cloud-Scale Systems, HotOS 2017Failures that a health check does not seeFailure modes
The Tail at Scale, Communications of the ACM, 2013Tail latency and hedged requestsFallback
Little's Law as Viewed on Its 50th Anniversary, Operations Research, 2011 (a copy hosted for a university course)L = λW, and why it holds over a finite period with no assumption about arrivals or order of serviceAdvanced

Standards#

SourceWhy read itUsed on
RFC 9110, HTTP SemanticsIdempotent methods, Retry-After, 503Failure modes, Retries, Idempotency, Frameworks
RFC 6585Status 429 Too Many RequestsFailure modes, Retries, Frameworks
The Idempotency-Key HTTP Header Field, an IETF Internet-DraftA standard header for idempotency keys, still a draftIdempotency, Frameworks

Tool documentation#

What each tool keeps reliable for you, and what it leaves to you, is described on Frameworks and only there. These are the entry points of each tool's own documentation that the page read.

Sister sites#

This site is one of a series on building with agents. Each site owns one part of the subject and links to the others where they meet, rather than repeating them. Ten sites are being written together; some of them may not be published yet when you read this.

SiteThe question it answers
Prompt EngineeringWhat instructions does the model read, and do they say what you mean?
Context EngineeringWhat goes into the model's window for one call, and what stays out?
Memory EngineeringWhich state persists between calls and runs, and how is it designed?
Agent Communication EngineeringHow do agents send each other messages, and over which protocols?
Observability EngineeringWhat evidence shows what happened: logs, metrics, traces, audit events?
Guardrails EngineeringWhich enforceable checks surround a probabilistic agent?
Tools EngineeringHow are callable capabilities designed, described and executed?
Structured Output EngineeringHow is model output given a shape, validated and repaired?
Agent Orchestration EngineeringWhich step runs next, and under which condition?
Agent Reliability Engineering, this siteWhat is still true after something fails?

Five sister sites are already published.

SiteThe question it answers
Harness EngineeringWhat does the software around a model call have to do to turn a model into an agent?
Eval EngineeringHow do you measure what an agent does, and what may you claim from the number?
Diff to DeployHow does a change made by a coding agent travel through review and CI to production?
Ask to ReleaseHow is a change planned before the first meeting?
AI EnablementHow does an organization adopt AI tools, role by role?

Three borders matter most for this site. Memory decides what state is worth keeping; this site uses saved state to recover. Orchestration decides which step runs next; this site decides what happens when a step fails. Observability makes a failure visible; this site decides what to do once it is visible.

Who wrote it#

Written by Sam Muthu. More of his work is at sammuthu.com.

The site is a working version. If a claim is wrong or out of date, the fix is to read the source again and correct the sentence; the fact files make that possible line by line. Every term the pages use is collected in the Glossary.