The fault-injection suite of the previous page is green: 295 runs, every invariant held. The underwriting agent goes on writing recommendations for submissions like SUB-1004, the office building whose roof is 9 years old on the application and 22 according to the roof-data provider. The model recommends; a person decides.
Then, in the synthetic week that the example uses for its service level lab, the provider has a bad day. From hour 60 to hour 83 it is slow and flaky: it answers, often late, sometimes not at all. Nothing is down. No single call does anything the suite did not try. But across thousands of runs the shape is new: many runs take much longer than usual, a few lose the roof assessment, a few are lost.
In this week, at the default settings of the SLOs lab (95% of runs good, a good run under 8000 ms), the 95th percentile of run time for the week rises from 4,475 ms in a healthy week to 17,471 ms, and 255 runs are bad where the week's error budget allows about 116. The burn-rate alert fires in hour 63, three hours after the trouble began. All numbers here come from invented settings in slo.py, not from measurements. The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
At that moment a person's phone rings. What happens in the next hour decides how much harm the brown-out does: how many broker requests wait, how many recommendations go out without a roof assessment, whether anyone makes it worse by turning off a protection to "get things moving". The code cannot decide this. People can, if they know their roles and have written down beforehand what to do.
Plain English. An incident is the moment a failure stops being a log line and becomes a job for people. Good incident response is a short list of habits: notice early, say out loud that it is an incident, stop the harm before hunting the cause, keep everyone informed, and afterwards write down what happened without blaming anyone.
What makes a failure an incident#
Most failures are handled by the code: a retry gets the roof assessment on the third attempt (UW-R02), a breaker opens and the run ends honestly in degraded_request_info (UW-R03). Nobody needs to wake up for those.
An incident is an unplanned event that harms or threatens the outcome for users badly enough that people must coordinate a response. Tiny example: the provider brown-out above, once the burn-rate alert says the week's error budget is burning fast.
The difference from an alert matters. An alert is a signal; an incident is a decision, by a person, that the signal needs an organized response. Google's SRE book, in its chapter "Managing Incidents", suggests declaring early rather than late and gives three tests, any one of which is enough: a second team is needed to fix the problem, the outage is visible to customers, or the problem is still unsolved after an hour of focused analysis (Site Reliability Engineering, "Managing Incidents").
For the insurance agent the second test needs translating. The customers of the workflow are the underwriters who read the recommendations and, behind them, the brokers who wait for a quote. A brown-out that turns many recommendations into request_info is visible to underwriters at once, even when every run ends in a named terminal state.
Common confusion. "Every invariant holds, so there is no incident." Invariants say each run ended acceptably. They do not say enough runs ended well enough, fast enough. A day of honest degraded_request_info results is acceptable per run and still an incident for the people waiting on full recommendations. That gap is why the alert watches a service level indicator, not the invariants.
The five phases of a response#
An incident moves through five phases. They overlap in practice, but their order shows what to do first.
| Phase | Question it answers | In the brown-out |
|---|---|---|
| Detect | Is something wrong, and how badly? | The burn-rate alert fires in hour 63 |
| Declare | Is this an incident, and who leads? | The on-call engineer declares it and takes the lead |
| Mitigate | How do we stop the harm now, even without knowing the cause? | Keep the breaker on, accept degraded_request_info, slow intake to the provider |
| Communicate | Who needs to know what, and when? | Underwriters: expect more request_info with a missing roof assessment; nothing is lost |
| Resolve | Is the service back to normal, and what is left to clean up? | The provider recovers at hour 84; degraded submissions get a fresh run on request |
Diagram: The provider brown-out · five phases of a response, then the learning
Read left to right. The alert is a signal; declaring turns it into an organized response with a named lead. Mitigation comes before the cause: on our side we keep the breaker on, accept honest request_info results and slow intake. Communication runs alongside. Resolution includes the cleanup. The postmortem and the promoted tests are what make the next brown-out cheaper. Hours are from the synthetic week of the example.
- Notice
- Detect: A signal that the share of good runs is falling fast enough to threaten the SLO for the week.
- Declare: Someone says out loud that this is an incident, takes the role of commander and opens the incident document.
- Stop the harm
- Mitigate: Changes on our side that make the next runs end better, without repairing the provider.
- Communicate: Regular updates for responders and for the underwriters who read the recommendations.
- Close
- Resolve: The provider answers normally again and the burn rate falls; changed settings are reverted and the store is checked.
- Learn
- Blameless postmortem: Impact, timeline, causes, what went well and actions, without blaming a person.
- Promote: The incident becomes a fault scenario here and, if an answer misled, a golden case for evaluation.
Mitigation is any action that reduces the harm of an incident without necessarily fixing its cause. Tiny example: a breaker that stops calling the provider does nothing for the provider, but it stops workers waiting 2000 ms on every call. The SRE book lists "prioritize" first among its practices: stop the bleeding and restore service, and keep the evidence for finding the cause later.
The order matters because cause is slow and harm is fast. During the brown-out nobody at the insurer can repair the provider. Hunting the cause for two hours while runs pile up is the classic mistake. The useful question in the first minutes is "what can we change, on our side, that makes the next hundred runs end better?"
The measure of the whole response is time to recover: how long from the start of the harm until the service delivers acceptable outcomes at its normal rate again. It splits naturally into time to detect (here three hours, from hour 60 to the alert in hour 63), time to mitigate, and time to resolve. In this synthetic week the first part is the most visible: an alert tuned to avoid false alarms costs three hours of slow runs before anyone is asked to act. That trade-off is discussed on SLOs.
Roles, not people#
In a small team one person may hold every role. The roles still matter, because each one is a different job, and when one person does three jobs at once one of them stops being done. The SRE book separates four. They are roles that people take for one incident, not job titles.
| Role | What it does | What it does not do | In the brown-out |
|---|---|---|---|
| Incident commander | Holds the overall picture, assigns work, decides; keeps any role not handed out | Type commands into production | Decides to keep the breaker on and slow intake |
| Operations lead | Makes the changes to the system, with the commander | Talk to stakeholders | Changes the intake rate; watches the queue and the breaker |
| Communications lead | Sends regular updates to responders and affected people; keeps the incident document current | Change the system | Tells the underwriting team what to expect and when the next update comes |
| Planning lead | Handles the longer view: bugs to file, hand-offs, a record of what was changed from normal so it can be undone | Fight the fire | Lists every setting changed so it can be reverted at resolve |
Two rules from the same chapter carry most of the value. Only the operations lead changes the system during the incident, so two people never "fix" the same thing in opposite directions. And when the lead role passes to another person, the hand-off is explicit and confirmed, so everyone knows who leads. The chapter also asks for a single place to find the commander and one living document of the incident's current state, with the most important facts at the top.
In our insurance agent. An agent adds a role nobody had in a classic service: the people who receive its output. Underwriters see every degraded recommendation. They need to hear from the communications lead thatrequest_infowithmissing: ["roof_assessment"]is the workflow working as designed during a brown-out, so they do not escalate it as a model error, and that a fresh run will be possible when the provider recovers.
The runbook: written steps for a known failure#
At three in the morning nobody reasons well from first principles. A runbook is a written, tested procedure for a known kind of failure: how to confirm it, what to do, in what order, what not to do, and how to tell that it worked. Tiny example: "if the breaker on the roof-data provider has opened more than ten times in an hour, follow the brown-out runbook".
Here is a runbook for the example's most likely real incident. It is a design this site proposes, written against the settings of the example; the example itself has no operations, so nothing here was run.
| Step | Action | Why |
|---|---|---|
| Trigger | The burn-rate alert fires and the share of runs ending in degraded_request_info rises; the breaker on the roof-data provider keeps opening | These two together point at the provider, not at a change of ours |
| 1 · Confirm | Check the latest deployment and the policy fixture version. If either changed in the last hours, switch to the deployment runbook (Deployment) | A change of ours is the most common cause and the one we can undo |
| 2 · Declare | Declare an incident, name the commander, open the incident document | The alert alone organizes nobody |
| 3 · Keep the breaker on | Do not raise the provider timeout above 2000 ms, do not raise maximum attempts above 3, do not switch the breaker off | Each of these makes runs wait longer on a provider that cannot answer, and adds load to it (Breakers) |
| 4 · Accept the smaller answer | Let runs end in degraded_request_info with missing: ["roof_assessment"]; check that every such row has degraded: true (INV-7) | An honest request for information is an acceptable outcome; a guessed roof age is not |
| 5 · Slow intake if workers starve | Lower the number of runs started per minute, or use the "stop starting new runs" kill switch; messages wait safely on the queue | Messages on the queue are not lost; runs stuck waiting hold workers (Deployment for the kill switch) |
| 6 · Communicate | Every 30 minutes (an illustrative interval): what underwriters will see, what is safe, when the next update comes | Silence makes people invent their own workarounds |
| 7 · Watch for duplicates | Run the INV-1 query against the live store: any submission with two pending_review rows | A brown-out produces timeouts, and timeouts are where duplicate writes are born |
| Exit | The provider answers normally for an hour (an illustrative window) and the burn rate is back under 1 | Recovery is a measured state, not a feeling |
| Never | Never let the agent write a recommendation without the roof assessment as if it had one; never delete rows to "clean up" | The model recommends; a person decides. Compensation marks, it does not delete (Checkpoints) |
A runbook earns trust the same way a recovery path does: by running. The game day of the previous page is where this one gets tested, by making the provider brown out for a synthetic submission and following the steps.
Production pitfall. The most tempting step in a brown-out is the wrong one: "the provider is slow, so raise the timeout and the retries". It feels like helping the runs finish. It turns every run into a longer wait on a provider that is already overloaded, and the retries multiply its load. Write the "do not" lines into the runbook, because the pressure in the moment pushes the other way.
Communicate, then resolve#
Communication during an incident has one job: give each audience what it needs to act, at a known rhythm. Engineers need the current state and who is doing what. Underwriters need to know which results to trust, which to set aside, and when to expect normal service. People outside the insurer, such as brokers, hear through the insurer's own channels, not from the engineering team.
Resolution is more than "the provider is back". The brown-out leaves traces in the store that must be dealt with on purpose:
| Left behind | What to do |
|---|---|
Runs that ended in degraded_request_info | Nothing automatic. An underwriter may ask for a fresh run once the roof assessment is available; that is a new run_id for the same submission_id |
| Runs still waiting on the queue after the kill switch | Lift the kill switch; they start in order. Their message_id makes each one a single run |
| Settings changed during the incident | Revert from the planning lead's list, one at a time, watching the burn rate |
Any duplicate found by the INV-1 query | Mark the extra row withdrawn with a reason; never delete it |
Then the incident is closed, and the learning starts.
The blameless postmortem#
A postmortem is a written record of an incident: its impact, what was done to mitigate and resolve it, its causes, and the follow-up actions that stop it from happening again or reduce its harm. The SRE book's chapter "Postmortem Culture" suggests agreeing the triggers before any incident, for example a visible degradation beyond a threshold, any data loss, an on-call engineer having to intervene, a resolution that took too long, or an incident found by a person rather than by monitoring (Site Reliability Engineering, "Postmortem Culture").
The same chapter makes the postmortem blameless: it looks for the causes in the system, assumes everyone acted in good faith on the information they had, and does not single out a person. The reason is practical. People who expect blame hide what they saw, and hidden problems grow.
"Causes", plural. A root cause is a reason without which the incident would not have happened. In complex systems there is rarely one. For the brown-out, a postmortem might read like this (a sketch built from the synthetic week):
| Section | Content |
|---|---|
| Summary | The roof-data provider was slow and flaky for one day; the share of good runs fell below the objective for the week |
| Impact | In the synthetic week: 255 bad runs against an error budget of about 116; p95 of run time 17,471 ms for the week; 19 runs without required evidence and 11 lost runs, against 6 and 7 in a healthy week |
| Detection | Burn-rate alert in hour 63; the trouble began in hour 60 |
| Causes | The provider degraded (outside our control). Contributing: our fault-injection catalog had no "slow but successful" provider, so we never saw how runs behave when most calls answer late; the alert rule needs three hours of burn to fire on this shape |
| What went well | The breaker opened; degraded results were marked; duplicate writes stayed at the level of a healthy week (5 in each) |
| Actions | Add a slow-provider fault kind and a scenario; test the brown-out runbook on the next game day; review the alert windows; promote the misleading cases to the golden set (next section) |
The "contributing" line is where a blameless postmortem pays off. Blame would stop at "the provider failed". The useful finding is that our tests did not include this shape of failure, which is a gap in the system we can close.
Before AI. A Spring service team behind a slow payment provider writes the same postmortem: impact, timeline, causes, actions. Two things are new for an agent. Its harm can be a wrong answer that looks right, which no latency graph shows, so the impact section must ask about the quality of what was written, not only how fast. And the follow-up actions have two homes: a fault scenario for the workflow and a case for the evaluation of the model's answers.
Promotion: from one bad day to a test that runs forever#
A postmortem that ends in a document is read once. A postmortem that ends in a test is checked on every change. Turning an incident into permanent tests is golden-set promotion on this site: the incident's story becomes a fault scenario in the reliability suite and, when the model's answer was wrong or misleading, an input with its correct expected answer becomes a case in the golden set that evaluation owns.
Diagram: Promotion · one incident, two permanent tests, two owners
The postmortem asks two questions. Did the workflow behave badly under the fault? Then the story becomes a fault scenario in this site's suite, judged by the seven invariants. Was the model's answer wrong or misleading? Then the input and its correct answer become a golden case, owned by the evaluation site. The runbook is updated either way.
- Source
- Postmortem actions: The list of follow-up actions at the end of the postmortem.
- Workflow behaved badly
- Fault scenario: A new fault kind if needed, a fault plan and an expected end state, added to the named scenarios of the suite.
- Fault-injection suite: The scenario runs with the others on every change and is judged by INV-1 to INV-7.
- Answer was wrong
- Golden case: The input that produced a wrong or misleading recommendation, with the recommendation it should have had.
- People
- Runbook update: The steps that worked stay; the steps that could not be followed are fixed; the never-do lines grow.
Two different questions, two different homes:
| What the incident showed | Becomes | Owned by | Checks |
|---|---|---|---|
| The workflow behaved badly under a fault (waits too long, a duplicate, a lost run) | A fault scenario: a case, a fault plan, an expected end state, the seven invariants | This site: the fault-injection suite (Fault injection) | That the outcome stays acceptable under that fault |
| The model's recommendation was wrong or misleading under those conditions | A golden case: the input, the expected recommendation, the grading | Evaluation: https://eval.sammuthu.com | That the answer is right |
For the brown-out the first row is the larger one. The suite's catalog (faults.py:KINDS) has timeouts and 503s for the provider, but no fault in which the provider answers late and successfully. The promotion is a new fault kind and a new scenario built on the facts of UW-004: provider calls that take most of their timeout and then answer. Its expected end state is still waiting_for_human with one row, inside the deadline. This is a design; the example does not contain it yet.
The second row applies if the postmortem finds a recommendation that misled: say, a degraded recommendation that read as a full one to an underwriter. The input with its expected recommendation joins the golden set, so every later model or prompt change is graded on it. How that set is built, graded and used as a release gate is the subject of the evaluation site, not of this page.
What this is not. Measuring whether recommendations got better or worse, golden sets and release gates belong to https://eval.sammuthu.com. How a failure is seen in logs, metrics and traces belongs to https://observe.sammuthu.com. This page owns the response once a failure is seen, and the promotion of the incident into a fault scenario.
The same idea for a coding agent#
A coding agent opens pull requests for a team. One afternoon the CI service is slow, CI jobs time out, the agent's harness retries the whole task, and each retry opens a new pull request. The commander pauses the agent (stop starting new tasks; let tasks in flight finish), the communications lead tells the team which pull requests to close, and the operations lead marks the duplicates closed with a comment rather than deleting branches. The postmortem promotes two things: a fault scenario in which CI times out and the agent must find its existing pull request before opening one, and, if the agent's code was wrong under pressure, a case for its evaluation. The harness where that pause lives is the subject of https://harness.sammuthu.com; the CI path of https://diff-to-deploy.sammuthu.com.
How incident response goes wrong#
| Mistake | What happens |
|---|---|
| Declaring late | Hours of harm while one person works alone and nobody else knows |
| Hunting the cause before mitigating | Harm keeps growing while the team reads logs |
| Everyone changes the system | Two fixes in opposite directions; nobody knows what state production is in |
| Turning protections off "to get things moving" | Raising timeouts and retries during a brown-out multiplies the load and the waits |
| Silence towards the people using the output | Underwriters treat honest request_info results as model errors, or build workarounds |
| Cleanup by deleting rows | Evidence gone; an audit cannot see what the agent wrote |
| A postmortem that names a person | People hide what they saw next time |
| A postmortem with no promoted test | The same shape of failure surprises the team again |
The four blocks below hold the detailed layer.
Measuring the response without fooling yourselfTime to detect, to mitigate, to resolve, and why an average of few incidents says little
Time to recover is easy to define and hard to use. Split it into its parts, because each part has a different fix: time to detect belongs to the alerts and the indicators (SLOs), time to mitigate to the runbooks and the protections, time to resolve partly to dependencies outside your control. In the synthetic week of this example, detection takes three hours by design of the illustrative alert rule, and the provider's recovery is fixed by the profile at hour 84; nothing the team does shortens the latter.
Averages over incidents mislead when incidents are few and different. One brown-out and one bad deployment have little in common; their mean time to recover describes neither. Report each incident's phases separately, and track the share of incidents found by monitoring rather than by a person (one of the SRE book's postmortem triggers).
For an agent add one measure the classic list lacks: the number of unacceptable outcomes left behind (duplicates, recommendations without evidence, lost runs), counted after cleanup. An incident can recover quickly and still leave rows that someone must find.
Should an agent help run the incident?Drafting, summarizing and querying: yes; changing production: through the operations lead
An agent can be useful inside an incident: summarizing the incident document for a new commander, running the INV-1 duplicate query, drafting the next update for the communications lead, collecting the timeline for the postmortem. Each of these reads or drafts; none changes production.
The same rule that keeps two people from fixing the same thing applies to an agent. Any change to the system goes through the operations lead, and an agent's proposed change is a proposal that a person accepts. An incident is the worst time to let a probabilistic component act on its own: inputs are unusual, the pressure is high, and a wrong action multiplies the harm. The pattern is the one this whole site uses for recommendations: the agent drafts; a person decides.
Quality incidents: when every dashboard is greenA model or prompt change that makes answers worse without making anything fail
A classic incident shows itself as errors or latency. An agent can have an incident with neither: the model provider changes a model behind the same name, or a new prompt makes recommendations subtly worse, and every run still ends in waiting_for_human with one row inside the deadline. Every invariant on this site holds.
Such an incident is detected by evaluation (graded samples of live output, underwriter overrides rising, a golden set re-run after the change), not by the reliability indicators. The response is the same five phases, and the most common mitigation is a rollback of the version that changed (Deployment). The promotion goes mostly to the golden set. A reliability team should still record the version of every part of the agent with each run, because without it nobody can tell which runs were written under the bad version.
Keeping runbooks trueA runbook is code that runs rarely, so it rots like a recovery path
A runbook written against "timeout 2000 ms, 3 attempts, breaker at 3 of 5" is wrong the day those settings change. Keep the numbers in one place that both the code and the runbook read, or link the runbook to the configuration rather than copying values into it.
Test runbooks on game days, by following them literally against a synthetic incident. Every step that could not be followed as written is a finding. Rotate who leads the game day, as the SRE book suggests for incident roles, so the runbook is tested by someone who did not write it.
Checklist#
| Question | Good answer |
|---|---|
| Are the criteria for declaring an incident written down before one happens? | Yes, including "visible to underwriters" |
| Does the alert watch a service level indicator, not every error? | Yes, a burn-rate alert (SLOs) |
| Are the four roles named in each incident, even when one person holds several? | Yes |
| Does only one person change production during an incident? | Yes, the operations lead |
| Is there a runbook for each known failure, with "never do" lines? | Yes, starting with the provider brown-out |
| Is there a kill switch that stops new runs without interrupting runs in flight? | Yes |
| Are the people who read the agent's output told what to expect? | Yes, at a known rhythm |
| Are postmortem triggers agreed in advance, and are postmortems blameless? | Yes |
| Does every postmortem promote at least one test? | Yes: a fault scenario here, a golden case on evaluation when an answer was wrong |
In one breath#
An incident is when a failure needs people, so we declare early, stop the harm before hunting the cause, give one person the lead and one person the hands on production, and tell the underwriters what they will see. A written runbook for known failures, like a provider brown-out, says what to do and, equally important, what not to touch. Afterwards a blameless postmortem finds the gaps in the system rather than a person to blame, and every incident becomes a test: a fault scenario in our suite, and a golden case for evaluation when an answer was wrong.
Next: the tools that keep some of this for you#
Every mechanism on this site so far was built by hand in the example: retries, keys, a queue, checkpoints, a breaker, a fault suite. In production much of it comes from standards and tools: HTTP's rules for retries and idempotency, durable workflow engines, message queues, resilience libraries. Frameworks describes what each one keeps reliable for you, from its own documentation, and where plain code is still the better choice.