All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Operations · 10

Deployment: canary, blue-green, feature flags and rollback

Most outages start with a change. How to expose a new prompt, model or version to a few runs first, what to watch, and how to go back when runs are still in flight.

What you will understand
What a version is
For an agent a release is more than code: the prompt, model id, tool schema and policy version each change behaviour, and each needs a version.
Expose a few runs first
Canary, blue-green, feature flags and the kill switch: how a bad change harms a few runs, not all of them, and how to go back.
Runs in flight
Which version finishes a run that started on the old one, and why a run should pin its versions in its checkpoint.

On SLOs the insurance agent ended its review week with one uncomfortable line. In the bad-deployment week a new version went out at hour 100. It wrote duplicate recommendation rows in about 3 of every 10 runs (a synthetic setting). The burn-rate alert fired at hour 104. The version stayed in production until hour 112, and by the end of the week it had spent 117% of the error budget. Its latency was identical to a healthy week, so no latency graph moved.

Look at what "a new version" was. Nobody changed the workflow's Python. The team changed the instructions the underwriting agent gives the model when it drafts a recommendation, and switched the model id of the primary route to a newer one. Both looked harmless in review. Together, on every run at once, they spent a week of budget in twelve hours.

Most outages begin with a change like this. The SRE workbook says that a majority of incidents are triggered by pushes of new binaries or configuration (Canarying Releases). So the question of this page is not how to avoid change. It is how to change the agent so that a bad version harms a few runs, is noticed quickly, and can be taken back, while runs that started on the old version still end correctly.

Plain English. Try the new version on a small slice of real work first, watch the same numbers your objective uses, and be able to switch back in one move. And remember that some runs are halfway through when you switch.

What "a version" is for an agent#

For a Java service, a version is usually a build: one artifact, one number. For an agent, at least five things change behaviour on their own, and most of them never pass through a compiler.

PartExample in the insurance agentHow it changes behaviour
CodeThe workflow and its eight stepsNew steps, new retry settings, new bugs
PromptThe instructions for draft_recommendationDifferent reasons, different recommendation in borderline cases
Model idThe model behind the primary and fallback routesDifferent answers to the same prompt; different latency and cost units
Tool schemaThe fields record_recommendation acceptsA field the old code does not send, or the new code does not read
Policy versionThe policy fixture, 2026.2 (earlier 2025.4)Different rules: in 2025.4 the roof limit was 30 years

Any one of these can be the bad change. So a release of an agent is a set of versions, and it is worth writing the set down as one record. Here is a shape this site proposes; it is a design, not part of the example, and every value is illustrative:

{
  "agent_release": "uw-agent-2026.11.3",
  "code": "git:4c1e9a2",
  "prompts": {"draft_recommendation": "draft-rec-v15"},
  "model_routes": {"primary": "model-a-2026-09", "fallback": "model-b-2026-06"},
  "tool_schemas": {"record_recommendation": "v3", "open_review_task": "v2"},
  "policy_version": "2026.2"
}

With such a record, "what changed?" has an answer in one line, the canary can compare two records instead of two vague builds, and a rollback can name exactly which record it returns to. The policy version is the odd one out: it changes because the insurer's rules change, not because engineers improved something. A change of rules still needs the same care, as the section on version skew below shows.

What this is not. Reviewing a change, its CI lanes and the path from a diff to production are covered on https://diff-to-deploy.sammuthu.com. This page starts where that path ends: the new version meets real runs, and some of those runs are already in flight.

A canary: a few runs first, compared with the rest#

A canary is a release that sends a small share of the work to the new version, for a limited time, and compares its results with the rest before going further. The SRE workbook defines canarying as a partial and time-limited deployment of a change and its evaluation, and calls the part on the new version the canary and the rest the control.

For the insurance agent: route 5% of new runs to release v15 and 95% to v14. Measure both groups with the SLIs of SLOs: unacceptable outcomes, latency, cost units per run. If the canary is worse than a threshold you set in advance, roll back; if not, widen the share step by step.

Why it helps is plain arithmetic. The workbook's own example: a canary on 5% of traffic with a 20% error rate adds only about 1% to the overall error rate. A bad change on a canary spends a small slice of the error budget. The same change on everything spends it all.

Two rules from the workbook matter for an agent in particular. Pick a few metrics that reflect what users feel and that the change could cause, starting with the SLIs. And compare the canary with the control at the same time, not with last week, because time itself changes the traffic. For a single agent run there is a third rule, from the workbook's section on systems that are not interactive: keep one unit of work on the same side, canary or control, through all its stages, and let the canary last at least as long as one unit of work. That rule is the seed of version pinning, later on this page.

The rollout lab: canary against all at once#

rollout.py:simulate_rollout compares the same stream of runs released two ways. Each run draws one random number that decides whether it goes to the new version (canary only) and one that decides whether it ends badly. Every check_every_runs runs, the share of bad runs seen on the new version is compared with the threshold, and above it the new version is rolled back. "Harmed" counts bad runs on the new version. Verbatim from rollout.py:

def _release(share: float, bad_rate_new: float, bad_rate_old: float, rollback_threshold: float, check_every_runs: int,
             total_runs: int, seed: int) -> dict[str, Any]:
    rng = Mulberry32(seed)
    new_runs = harmed = bad_old = 0
    rollback_at: int | None = None
    for run in range(1, total_runs + 1):
        to_new, luck = rng.next_float() < share, rng.next_float()
        if rollback_at is None and to_new:
            new_runs += 1
            harmed += 1 if luck < bad_rate_new else 0
        else:
            bad_old += 1 if luck < bad_rate_old else 0
        if rollback_at is None and run % check_every_runs == 0 and new_runs and harmed / new_runs > rollback_threshold:
            rollback_at = run
    return {"runs_on_new": new_runs, "harmed": harmed, "bad_on_old": bad_old, "rollback_at_run": rollback_at,
            "rolled_back": rollback_at is not None}

The script below is not part of the example; it sits in a scratch copy of examples/uw-reliability and calls the function with the lab's defaults (canary 5%, the new version harms 20% of its runs, the old one 1%, 2000 runs, a check every 50), then once more with the threshold raised from 5% to 25%. A successful run proves only the arithmetic for seed 7; the harm rates are settings, not measurements.

from rollout import simulate_rollout

for threshold in (0.05, 0.25):
    r = simulate_rollout(canary_share=0.05, bad_rate_new=0.2, bad_rate_old=0.01, rollback_threshold=threshold,
                         check_every_runs=50, total_runs=2000, seed=7)
    for name in ("canary", "all_at_once"):
        x = r[name]
        print(f"threshold {threshold:.2f}  {name:11}  on new {x['runs_on_new']:4}  harmed {x['harmed']:3}  "
              f"rollback at run {x['rollback_at_run']}")
threshold 0.05  canary       on new    2  harmed   2  rollback at run 50
threshold 0.05  all_at_once  on new   50  harmed  10  rollback at run 50
threshold 0.25  canary       on new    2  harmed   2  rollback at run 50
threshold 0.25  all_at_once  on new 2000  harmed 403  rollback at run None

Two lessons are in those four lines. With a sensible threshold, both releases roll back at the first check, and the canary harmed 2 runs where the all-at-once release harmed 10. With a threshold above the real harm (25% against 20%), the all-at-once release never rolls back and harms 403 of 2000 runs. The canary still rolled back, but look at why: 2 harmed out of 2 runs is 100%. A canary of 2 runs reports almost noise. The README of the example says the same about small check intervals: a rollback can fire by chance. In this example, a new version that harms only 3% of runs, below the 5% threshold, is still rolled back at run 50 because 1 of its 2 canary runs went badly. The workbook's advice answers this: a canary has to be large enough and last long enough to be representative.

The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

Try it: canary or all at once#

Set the canary share, how often the new version harms a run, and the rollback threshold. The lab checks every 50 of 2000 runs.

Without JavaScript, the lab's default (canary 5%, harm 20%, old version 1%, threshold 5%, seed 7), from rollout.py:simulate_rollout run in a copy of the example:

ReleaseRuns on the new versionRuns harmedRolled back
Canary, 5%22at run 50
All at once5010at run 50
All at once, threshold 25%2000403never

Diagram: A canary release of the agent: a small share on the new release, judged against the rest, then widen or roll back

Read left to right. New runs arrive from the queue. A router sends 5 percent to release v15 and the rest to v14. Both groups are measured with the same indicators at the same time. If the canary burns budget faster than the threshold, the router sends everything back to v14 automatically; otherwise the share grows step by step. The numbers in the examples are the lab's synthetic defaults.

  1. Split
    1. Router: Decides, per new run, which release takes it. A run keeps its release until it ends.
  2. Run
    1. Canary: v15: The small group of runs on the new release record.
    2. Control: v14: Everything else, on the release already in production, measured at the same time.
  3. Judge
    1. Compare SLIs: The share of unacceptable outcomes, the latency percentiles and the cost units per run of each group, compared with a threshold set in advance.
  4. Decide
    1. Roll back: The router sends every new run to v14 again. Runs already on v15 finish on v15.
    2. Widen: No sign of harm: the share grows in steps, each with its own wait and check.

Blue-green: two environments and a switch#

A blue-green deployment keeps two complete environments. One (say green) serves all the work; the other (blue) gets the new release. When blue is ready, the router moves the work across. Rolling back means moving the router back. The SRE workbook describes it this way and names the cost: about twice the resources of a normal deployment. It can also act as a canary by moving the work across in steps.

For the agent, blue and green are two pools of workers, each with its own release record, reading from the same queue through a router that decides which pool takes a new message. The switch is fast. What it does not switch is the store: both pools write recommendation rows into the same tables, and a run that blue began may still be running when the router points back at green. Blue-green makes the code switch cheap; it does not make data or in-flight runs go away.

Feature flags and the kill switch#

A feature flag is a setting, read at run time, that turns one behaviour on or off without a new deployment. Tiny example: a flag draft_prompt_v15 that is on for 5% of submissions and off for the rest. The code for both prompts ships; the flag decides which one a run uses. Pete Hodgson's article on feature toggles sorts flags into four kinds (release, experiment, ops and permissioning) and warns that each flag is inventory with a carrying cost: one more code path to test, which should be removed once it is no longer needed (Feature Toggles).

Flags suit an agent well, because so much of its version is data: a prompt, a model id, a route order. Changing a flag is faster than a deployment, and it can be scoped by submission, by broker or by share.

A kill switch is a flag kept for emergencies, which turns a risky behaviour off at once. The same article describes a few long-lived ops toggles used this way, calls them a manually managed circuit breaker, and says that changing one should not need a release. For the insurance agent, a sensible kill switch (a design this site proposes, not part of the example) is "stop starting new runs": workers stop taking messages, the messages wait on the queue, and runs already in flight continue to a terminal state. Killing runs halfway, between a write and its checkpoint, would create exactly the partial states the earlier pages work so hard to prevent. A kill switch should stop new work, not interrupt old work.

MechanismWhat it switchesHow fast backWhat it costs
CanaryA share of new runs to the new releaseRoute the share backNeeds enough runs to judge
Blue-greenAll new work between two poolsMove the router backTwo environments; shared store
Feature flagOne behaviour, per run, without deployingTurn the flag offA second code path until removed
Kill switchA risky behaviour, or new runs entirelyImmediateWork waits; nothing new is done

Rollback: going back must be safe too#

A rollback returns the system to the previous release record. It is the safety net under every mechanism above, and it is also a change, with its own risks. (This site keeps the word for deployments and database transactions; undoing the meaning of a stored row is a compensation, taught on Checkpoints.)

The classic trap is data. If v15 writes something v14 cannot read, rolling back to v14 breaks it. Amazon's Builders' Library article on rollback safety gives a two-phase answer: first deploy a version that can read the new format but still writes the old one (prepare), and only after that is everywhere deploy the version that writes the new format (activate). Each phase can be rolled back alone (Ensuring rollback safety during deployments).

For the agent, the tool schema is the usual place for this. If v15 adds a field to the recommendation row that the cockpit or v14 cannot handle, prepare first (everyone tolerates the field), activate second (v15 starts writing it). The checkpoint is another such format: v14 must be able to read a checkpoint v15 wrote, or a rollback strands those runs.

Amazon's article on hands-off deployments adds the operational half: rollback is automatic when alarms fire, it needs nobody watching, and it returns to the previously deployed package. The alarms it lists include fault rates and latency percentiles, and a first stage serves at most about 10% of requests on one box before waves of growing size, with a wait (bake time) between stages (Automating safe, hands-off deployments). For an agent, the alarm to wire to automatic rollback is the burn rate of the canary's own runs.

Version skew: the run that started on the old version#

A web request lasts milliseconds, so during a deployment it runs on one version or the other. An insurance run lasts seconds to minutes in the automated part, and a crash or a deployment can cut it in the middle. Then a question appears that a web service rarely meets: which version finishes the run?

An in-flight run is a run that has started and has not yet reached a terminal state. Version skew is a run, or a pair of systems, working with parts from two versions at once.

Here is how it happens in this example's world. Worker A, on release v14, takes the message for SUB-1004 and runs steps 1 to 5. The model, on the v14 prompt, drafts escalate citing R-ROOF-03, and the checkpoint saves it. Then the deployment of v15 replaces worker A's process. This is the crash of UW-R08, caused by the deployment instead of a fault. The queue redelivers after the visibility timeout, and worker B, now running v15, resumes the run from its checkpoint. The checkpoint was written by v14. Step 6 will be executed by v15.

Three things can go wrong, and they go wrong silently:

SkewWhat happens
v15 renamed or reordered a stepThe checkpoint's step_id points at a step v15 does not have, or resumes at the wrong place
v15 changed the prompt or model and re-draftsThe run asks a different model for a new recommendation, under the same idempotency key; the stored row and the new text disagree
The policy fixture moved from 2025.4 to 2026.2 mid-runSteps 1 to 5 used one rule set and step 6 records another policy_version; INV-6 expects the current one

The design this site proposes is to pin the versions in the checkpoint. When a run starts, it records its release record (code, prompt, model ids, tool schemas, policy version) in the run row and in every checkpoint. A worker that resumes the run reads the pin and either continues with exactly those versions, or hands the run to a pool that still has them. The example does not do this: its checkpoint holds run_id, step_id, trace_id, resume_count and the state, and its recommendation carries policy_version, but nothing names the code or prompt. It also does not need to, because it never deploys mid-run.

Diagram: Version skew: a run starts on v14, a deployment kills the worker, a v15 worker resumes it (a proposed design)

Time runs down the page. Worker A on release v14 drafts the recommendation and saves a checkpoint that pins the release record. The deployment of v15 replaces worker A before step 6 is checkpointed, like the crash of UW-R08. Worker B runs v15. It reads the pin, sees v14, and continues with the v14 prompt and model, or hands the run to a v14 worker. The pin is a design this site proposes; the example's checkpoint does not hold a release record.

  1. Worker A (release v14) to Store (run row, checkpoint): Run starts: pin release v14. The run row records the code, prompt, model ids, tool schemas and policy version it started with.
  2. Worker A (release v14) to Store (run row, checkpoint): Step 5 done: checkpoint with the draft and the pin. The model's recommendation, drafted under the v14 prompt, is saved with the pin.
  3. Deployment to Worker A (release v14): Replace process with v15. The deployment stops worker A between the write of step 6 and its checkpoint.
  4. Worker B (release v15) to Store (run row, checkpoint): Resume: load checkpoint, read the pin. Worker B finds the run by message_id and sees that it was started on v14.
  5. Store (run row, checkpoint) to Worker B (release v15): Pin says v14. Worker B must not re-draft with the v15 prompt or model. It continues with the saved draft and the v14 tool schema, or hands the run to a v14 worker.
  6. Worker B (release v15) to Store (run row, checkpoint): Step 6 again, same key: replay. The write is replayed with the same idempotency key, so the run still has one recommendation row, recorded under v14.

Pinning does not answer every case. The policy version is a business decision: if the insurer's new rules take effect at midnight, a run that started at 23:59 under 2025.4 and records at 00:01 may need to be re-checked under 2026.2, not finished under the old rules. That is a choice for the people who own the policy fixture, and the run must make it visible, which is what INV-6 checks. For code, prompt and model, finishing on the pinned version is the safe default, because the model's answer is already in the checkpoint and was drafted under those instructions.

Common confusion. "Roll back the prompt and the runs it drafted are fixed." Rows written under v15 stay in the store after the rollback. A rollback stops new damage. Recommendations already waiting for the underwriter were produced by v15 and should be findable by their release record, so that a person can review or withdraw them.

How deployment goes wrong#

MistakeWhat happensConnection on this site
Treat a prompt or model change as "configuration", not a releaseIt skips the canary and reaches every run at onceThe bad-deployment week of SLOs
Canary judged on latency onlyA fast version that writes duplicates passesSame percentiles, 87 unacceptable outcomes
Canary too small or too shortRollbacks by chance, or a bad version promoted by luck1 harmed of 2 runs rolls back a 3% version here
Rollback threshold above the harm you cannot acceptThe bad version is never rolled back403 of 2000 runs harmed, all at once, threshold 25%
New version writes a format the old one cannot readRollback breaks what it was meant to savePrepare, then activate
Kill switch that interrupts runs mid-stepPartial writes and runs that never endINV-1, INV-4
Resumed runs switch to the new versionRe-drafts, key conflicts, wrong policy_versionUW-R08 caused by a deployment
Releasing into a spent error budgetEvery further change adds risk to a service already below its objectiveError budget policy on SLOs
Before AI. A Spring service on Kubernetes has most of this: rolling updates, a canary deployment by replica count, blue-green with two services and one ingress, feature flags from a configuration service, and expand-then-contract database migrations so the old code still works during the switch. Two things are new for an agent. Much of its version is not code (a prompt, a model id, a policy), so it escapes the pipeline unless it is versioned on purpose. And its unit of work is long and resumable, so a deployment does not end requests, it interrupts runs that a different version will finish. Products that run durable workflows have their own answers for changing code under running workflows; what each provides, from its own documentation, is on Frameworks.

The same idea for a coding agent#

A coding agent works through a task: it reads the repository, edits files, runs the tests and opens a pull request. Its release record has the same parts: the harness code, the system instructions, the model id, the tool definitions, and the repository's own instruction file. Ship a new model to 5% of tasks first and compare the share of tasks that end with one green pull request. If a task is resumed after a deployment, it should continue on the model and instructions it started with, or a half-made change gets finished by a different mind with different habits. The runtime that holds those versions is the subject of https://harness.sammuthu.com, and the path from change to production of https://diff-to-deploy.sammuthu.com.

The blocks below hold the detailed layer.

When the model changes and you did not deployA model id that points at a moving target is a release you never canaried

If a model identifier is an alias that its provider can point at a newer model, and the release record holds that alias, the provider can change the agent's behaviour without any deployment by the team, and no canary sees it. Prefer identifiers that name a fixed version, write the resolved id into the run row, and treat a move to a new snapshot as a release with its own canary. What each provider promises about its identifiers belongs to that provider's documentation and, for any provider named on this site, to Frameworks.

The same holds for anything the run fetches at run time: the retrieved guidance passages, the roof-data provider's model, the cockpit's API. Each can change under the agent. The canary's comparison with a control group at the same time is what separates "our release made it worse" from "the world changed today", which is the workbook's reason to avoid before-and-after comparisons.

How big a canary must be for an agentRare bad outcomes need many runs, and runs are slow

An unacceptable outcome rate of 1% means a canary of 50 runs expects half a bad run. It cannot tell a version that harms 1% from one that harms 3%. The lab shows the extreme: a check after 2 canary runs. Three levers help. Longer canaries (more runs at the same share). Higher shares once the first small step has shown no gross failure, which is how Amazon's waves grow. And replaying recorded or synthetic work through the canary before live traffic: the fault scenarios of Fault injection and a golden set from https://eval.sammuthu.com both give a release many runs in minutes, without harming a broker.

A canary also has to respect the error budget. If the budget is nearly spent, even a small canary is a risk the policy may forbid. If it is full, a larger canary that finishes in a day may cost less in total than a tiny one that runs for a week.

Pin, migrate or restart: three answers to in-flight runsEach costs something; the choice depends on how long runs live

Pin: in-flight runs finish on the versions they started with. Correct by construction, but the old version must stay deployable until the last pinned run ends. For runs that wait days for a person, the automated part after the human step may still be weeks away; keep the old pool small but alive, or give each later step its own pin decision.

Migrate: the new version converts the checkpoint and continues. Cheaper to operate, but every migration is code that must be written, tested and rolled back, and the model's answer drafted under the old prompt is now finished under new code. Use it when the change is to code only and the checkpoint format is compatible.

Restart: withdraw the run's effects (compensation) and start a fresh run on the new version. Clean, but it repeats the model call and the provider call, and it is only safe if every effect so far can be compensated. For a run already waiting for the underwriter, it means a person sees a withdrawn recommendation and a new one; that should be rare and visible.

Checklist#

QuestionGood answer
Is every prompt, model id, tool schema and policy version part of a release record?Yes, one record per release, stored with each run
Does every change, including prompt and model changes, go through a canary?Yes
Is the canary judged on the SLIs, including unacceptable outcomes, against a control at the same time?Yes
Is the rollback threshold below the harm you cannot accept, and is the canary large enough to see it?Yes
Is rollback automatic on the canary's burn rate?Yes
Can the old version read everything the new one writes, including checkpoints?Yes: prepare first, activate second
Do in-flight runs finish on their pinned versions, or by a stated migration?Yes, decided per change
Does the kill switch stop new runs without interrupting old ones?Yes
Can you find every row written by a given release?Yes, by its release record

In one breath#

For an agent a version is a set: code, prompt, model id, tool schema and policy version, and any of them can be the bad change. Release it to a small share of runs first, compare that share's SLIs with the rest at the same time, and roll back automatically when the canary burns budget, with a format the old version can still read. Runs in flight should pin their versions in their checkpoint, so a run that started on the old release is never finished by a different prompt or model by accident.

Next: when the model itself is down#

A canary protects the agent from its own changes. It does nothing when the model provider's primary route stops answering on a Tuesday afternoon, with no release in sight. Fallback is about a second route for the model call: when to switch, what the answer loses on the way, and how to say so in the recommendation the underwriter reads.