This is a teaching site about keeping the outcomes of agentic software acceptable when its parts fail. It starts from one sentence, "reliability means a system keeps delivering acceptable outcomes despite failures, retries, load and changing dependencies", and climbs one mechanism at a time: timeouts and retries, idempotency, queues, checkpoints and sagas, circuit breakers, concurrency limits, service level objectives, deployment, fallback routes, fault injection and incident response. At every step it asks the same question: what is still true after this fails?
It is written for three readers at once: someone new to building with models, an experienced backend engineer who knows Java, Spring, Kafka and relational databases, and a principal engineer who wants the trade-offs and the failure cases. The first reading of each page is for the first two; the folded blocks are for the third.
One example runs through every page. An insurer receives a broker's request for a property quote. An underwriting agent reads the submission, asks an outside provider about the roof, retrieves the guidance, runs the rules and writes a recommendation, which a human underwriter then decides on. The model recommends; a person decides. Every page breaks that workflow in one way and repairs it.
What it is and what it is not#
| It is | It is not |
|---|---|
| A ladder: one mechanism per page, each word in plain English with a tiny example before it is used | A standard, or the only way to divide the subject |
One running example, the eight-step insurance workflow and its fifteen fault scenarios UW-R01 to UW-R15 | A record of any real insurer, broker, applicant or provider |
| A demonstration of mechanisms: what a retry, a key, a checkpoint or a breaker keeps true, shown by injecting the fault | A measurement of any model, provider or product. No page reports how reliable a real system is |
| Code that was run: a small example in standard-library Python, with its real output on the pages | A framework to install. It is short so that it can be read whole, and it leaves things out on purpose |
| Tools described from their own documentation, by what each keeps reliable for you | A ranking or a buying guide. No tool is called best, and no price is quoted |
| Designs, labelled as designs, where the site proposes something (the key format, the escalation path) | Advice that a design fits every system. Each page says what its design does not solve |
The honest boundary#
The insurance example is an illustrative reconstruction for teaching. It follows the shape of a real project by the author, but the cases, the rules, the fields, the provider, every timing and every number are invented. Nothing here reports the reliability or the quality of a real system or model. The rules of the policy fixture (for example "a roof older than 25 years in wind zone C is not eligible") are synthetic, not insurance practice.
The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
The table says what stands in for what.
| In the example | What it really is |
|---|---|
| The underwriting agent | A script that writes what the rules imply. No model is called; there is no network and no key |
| The model provider | A stand-in with two routes, a fixed latency and a fixed cost in cost units (simulated tokens, never money) |
| The roof-data provider and the other tools | Stand-ins that return what the case file tells them, after a fixed latency |
| The store, the queue, the checkpoints | Python lists and dictionaries in one process. No isolation levels, no replication, no real disk |
| Time | A simulated clock in milliseconds. Nothing waits |
| Faults | Written in a fault plan per scenario, or drawn from a seeded generator, so every reader sees the same run |
| The human underwriter | Not simulated. The automated run ends when the review task is open |
| The synthetic week of runs behind the SLO lab | Three invented profiles (healthy, a provider brown-out, a bad deployment), not measurements |
The fault-injection suite of the example runs 295 runs (15 scenarios, 80 matrix runs and 200 random fault plans). Under the careful workflow every invariant holds in every run; under the naive workflow 59 runs pass. Those counts come from a script and a seed. They show that the mechanisms do what the pages say, against the faults the example knows how to inject, and nothing more. The full output is on The insurance lab.
How the facts were checked#
| Kind of claim | How it was checked |
|---|---|
| A named standard or tool | Read on its own documentation on the day of writing, 2026-11-17, and logged with the address and the date. Where the documentation is silent, the page says "the documentation read here does not say". Statements about tools live on Frameworks only; other pages name a tool and link there |
| An idea from a source | The source was opened and read on the day of writing, logged in the fact file of the page that uses it, and cited where the idea is taught |
| A surprise | A deprecation, a moved page or a new revision found during reading is marked "needs a second read" in the fact file, so it can be fetched again before it is relied on |
| The example | Run. Every command a page shows was executed in a scratch copy of examples/uw-reliability, and the output on the page is what it printed. The folder has 77 unit tests |
| The labs | They read data generated from the example with a fixed seed. Their arithmetic is a second implementation of the Python functions, and a test fails if the two disagree |
| Numbers | Synthetic and marked "in this example" or "illustrative" unless a source is linked. A number from the example is the result of a script, never a measurement |
| Other code | A block in a tool's API says how far it was checked: "documented shape; not run here" (compiled or parsed only) |
| Links and structure | A build step checks every internal link, every diagram and every glossary reference before the site is published |
The fact files sit in the repository under docs/facts/, one per page: each row is a claim, the address it came from, and the date it was read. Documentation changes. Treat every statement about a tool as true on the day it was read, and read the source again before you build on it.
What is original here and what is borrowed#
Most of the mechanisms on this site were published by other people first, many of them long before agents. The site uses those ideas with attribution and does not copy their prose, examples or figures.
| Part | Status |
|---|---|
| Timeouts, retries, backoff with jitter, idempotent APIs, load shedding, fallback | Practice described in the Amazon Builders' Library and Google's SRE books; cited on the pages that teach them |
| Error budgets, burn-rate alerts, cascading failures, incident roles, blameless postmortems | From Google's SRE book and workbook |
| Fault, error and failure as separate words | From the published taxonomy of dependable computing (see the reading list) |
| Saga and compensation | From the 1987 paper on sagas; the pivot step from Microsoft's saga pattern page |
| Circuit breaker, bulkhead, feature toggle, transactional outbox, fencing token | Established patterns; each page cites the description it read |
| Chaos engineering and steady state | From the Principles of Chaos Engineering |
| Hedged requests, tail latency | From "The Tail at Scale" |
| Little's law | From its author's 50th-anniversary paper |
The eight-step workflow, the fifteen scenarios UW-R01 to UW-R15, the seven invariants, the fault plans | Written for this site. All synthetic |
The key design <run_id>:<step id>, the one-trace-per-run rule, the approval path on Advanced | Designs proposed on this site |
| The example code, its 77 tests, the fault-injection suite | Written for this site, in standard-library Python |
| The interactive labs and the diagrams | Written for this site |
The shared cases UW-001 to UW-010 and the policy fixture | Shared with the sister sites of the series; synthetic |
Where a page takes an idea from a source, it says so. Where this site proposes something, it calls it a design.
Sources and reading list#
Every source below was opened on the day the pages were written and is logged, with the claims taken from it, in the fact file of the page named in the last column. The titles are the sources' own.
Reliability practice#
| Source | Why read it | Used on |
|---|---|---|
| Site Reliability Engineering, Google | The book's chapters on embracing risk, service level objectives, handling overload, cascading failures, data integrity, managing incidents and postmortem culture; the introduction for capacity planning | Start, Four qualities, Failure modes, Retries, Breakers, SLOs, Incidents, Advanced |
| The Site Reliability Workbook, Google | Implementing SLOs, alerting on SLOs and canarying releases | SLOs, Deployment |
| Timeouts, retries and backoff with jitter, Amazon Builders' Library | Choosing a timeout from latency, retrying only what may succeed, and jitter | Failure modes, Retries, Idempotency |
| Making retries safe with idempotent APIs, Amazon Builders' Library | A request identifier chosen by the caller | Idempotency |
| Using load shedding to avoid overload, Amazon Builders' Library | What to refuse first when a service is overloaded | Breakers |
| Avoiding insurmountable queue backlogs, Amazon Builders' Library | Backlogs, dead-letter queues and messages that can never succeed | Queues |
| Avoiding fallback in distributed systems, Amazon Builders' Library | Why a fallback path that rarely runs is a risk of its own | Fallback |
| Leader election in distributed systems and Fairness in multi-tenant systems, Amazon Builders' Library | Leases and their hazards; token buckets and per-tenant quotas | Concurrency |
| Automating safe, hands-off deployments and Ensuring rollback safety during deployments, Amazon Builders' Library | Staged rollout, bake time, automatic rollback, and changes that can be rolled back | Deployment |
| Principles of Chaos Engineering | Steady state, hypotheses and experiments with real-world events | Fault injection |
The Builders' Library articles now open on AWS Builder Center: the old aws.amazon.com/builders-library/... addresses redirect there, and the links above point at the pages that were read.
Patterns#
| Source | Why read it | Used on |
|---|---|---|
| Sagas, 1987 (a PDF hosted for a university course) | The origin of the saga: a long transaction as a sequence of smaller ones, each with a compensation | Checkpoints |
| Saga design pattern and Bulkhead pattern, Microsoft Azure Architecture Center | The pivot step after which nothing can be compensated; isolating resources into pools | Checkpoints, Breakers |
| Circuit Breaker and Feature Toggles, martinfowler.com | The breaker's states; the kinds of toggle and the kill switch | Breakers, Deployment |
| Transactional outbox, microservices.io | Writing the message in the same transaction as the change | Idempotency, Queues |
| How to do distributed locking, 2016 | Why a lease alone does not stop a paused worker, and the fencing token | Concurrency |
Research#
| Source | Why read it | Used on |
|---|---|---|
| Basic Concepts and Taxonomy of Dependable and Secure Computing, 2004 (record of the University of Maryland repository) | Fault, error and failure as three different things | Four qualities |
| Gray Failure: The Achilles' Heel of Cloud-Scale Systems, HotOS 2017 | Failures that a health check does not see | Failure modes |
| The Tail at Scale, Communications of the ACM, 2013 | Tail latency and hedged requests | Fallback |
| Little's Law as Viewed on Its 50th Anniversary, Operations Research, 2011 (a copy hosted for a university course) | L = λW, and why it holds over a finite period with no assumption about arrivals or order of service | Advanced |
Standards#
| Source | Why read it | Used on |
|---|---|---|
| RFC 9110, HTTP Semantics | Idempotent methods, Retry-After, 503 | Failure modes, Retries, Idempotency, Frameworks |
| RFC 6585 | Status 429 Too Many Requests | Failure modes, Retries, Frameworks |
| The Idempotency-Key HTTP Header Field, an IETF Internet-Draft | A standard header for idempotency keys, still a draft | Idempotency, Frameworks |
Tool documentation#
What each tool keeps reliable for you, and what it leaves to you, is described on Frameworks and only there. These are the entry points of each tool's own documentation that the page read.
| Tool | Documentation |
|---|---|
| Temporal | docs.temporal.io/workflows |
| LangGraph | docs.langchain.com, checkpointers |
| AWS Step Functions | Error handling |
| Apache Kafka | Design |
| Amazon SQS | Visibility timeout |
| RabbitMQ | Consumer acknowledgements and publisher confirms |
| Resilience4j | Getting started |
| Anthropic Python SDK | platform.claude.com, Python SDK |
| OpenAI Python SDK | github.com/openai/openai-python |
Sister sites#
This site is one of a series on building with agents. Each site owns one part of the subject and links to the others where they meet, rather than repeating them. Ten sites are being written together; some of them may not be published yet when you read this.
| Site | The question it answers |
|---|---|
| Prompt Engineering | What instructions does the model read, and do they say what you mean? |
| Context Engineering | What goes into the model's window for one call, and what stays out? |
| Memory Engineering | Which state persists between calls and runs, and how is it designed? |
| Agent Communication Engineering | How do agents send each other messages, and over which protocols? |
| Observability Engineering | What evidence shows what happened: logs, metrics, traces, audit events? |
| Guardrails Engineering | Which enforceable checks surround a probabilistic agent? |
| Tools Engineering | How are callable capabilities designed, described and executed? |
| Structured Output Engineering | How is model output given a shape, validated and repaired? |
| Agent Orchestration Engineering | Which step runs next, and under which condition? |
| Agent Reliability Engineering, this site | What is still true after something fails? |
Five sister sites are already published.
| Site | The question it answers |
|---|---|
| Harness Engineering | What does the software around a model call have to do to turn a model into an agent? |
| Eval Engineering | How do you measure what an agent does, and what may you claim from the number? |
| Diff to Deploy | How does a change made by a coding agent travel through review and CI to production? |
| Ask to Release | How is a change planned before the first meeting? |
| AI Enablement | How does an organization adopt AI tools, role by role? |
Three borders matter most for this site. Memory decides what state is worth keeping; this site uses saved state to recover. Orchestration decides which step runs next; this site decides what happens when a step fails. Observability makes a failure visible; this site decides what to do once it is visible.
Who wrote it#
Written by Sam Muthu. More of his work is at sammuthu.com.
The site is a working version. If a claim is wrong or out of date, the fix is to read the source again and correct the sentence; the fact files make that possible line by line. Every term the pages use is collected in the Glossary.