All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Operations · 9

SLOs: saying how reliable is reliable enough

An indicator measures, an objective sets the line, an error budget says how much failure is left to spend. Latency distributions, unacceptable outcomes, burn rate and alerts worth waking for.

What you will understand
Good and bad events
What counts as a good run for an agent: an acceptable outcome, fast enough, and the share of good runs as the number to watch.
Latency is a spread
Why the average hides the slow runs, and how p50, p95 and p99 show what the slowest brokers live through.
Budget and burn
How an objective turns into an error budget, how fast it is being spent, and an alert that fires on the burn, not on every error.

Every mechanism of this site now protects the insurance agent. The underwriting agent reads a broker's submission, fetches the roof assessment, retrieves the guidance, runs the rules and writes a recommendation that waits for an underwriter. The model recommends; a person decides. Retries, keys, checkpoints, breakers and fenced leases keep each run to one row and a named end, as the earlier pages showed one scenario at a time.

Now picture a Monday review of the past week. Over 168 hours the agent handled 2315 runs (a synthetic week, invented for this page). On Wednesday the roof-data provider was slow and flaky for a whole day. Someone asks three questions. Was the week good enough? When should someone have been woken? And may the team ship the new prompt it planned for Thursday?

Nobody can answer from the logs. One engineer says "the error rate was low": most runs did finish. Another says "Wednesday was terrible": 1 run in 20 took more than 17 seconds. Both are true. Neither is an answer, because nobody wrote down, before the week began, what "good enough" means.

This page writes it down. It turns "reliable" into a measurement, a target, and an amount of failure the team agreed to accept.

Plain English. You cannot manage what you have not defined. Decide in advance what a good run is, what share of runs must be good, and what happens when that share drops. Then the Monday review is arithmetic, not an argument.

A good event: the unit of reliability#

Start with the smallest thing you can count. For a web page that is a request: it answered, or it did not. For the insurance agent the natural unit is a run, one execution of the workflow for one submission. So the first question is: when is a run good?

"It finished" is not enough. Qualities showed that a run can answer and still leave an unacceptable outcome: a duplicate recommendation row, a run that never reaches a terminal state, a recommendation that cites evidence the run never fetched. And a run that ends well after 40 seconds may be correct and still too slow for the broker waiting on the quote.

So this example defines a good event as a run that ended in an acceptable outcome and took no longer than a latency threshold (8000 ms by default here). Everything else is a bad event. In code, from slo.py:_bad_in, the bad events of one hour are the unacceptable outcomes plus the runs slower than the threshold:

def _bad_in(bucket: dict[str, Any], latency_threshold_ms: int) -> int:
    """Bad events of one hour: unacceptable outcomes plus runs slower than the threshold.

    A run that is both is counted twice here. With the rates of the three
    profiles that overlap is small; the lab says so.
    """
    slow = sum(1 for ms in bucket["latencies_ms"] if ms > latency_threshold_ms)
    return min(bucket["runs"], slow + sum(bucket[k] for k in KINDS))

KINDS names the three unacceptable outcomes the synthetic week can contain: duplicate_write, lost_run and missing_evidence. The docstring is honest about a shortcut: a run that is both slow and unacceptable counts twice. A production definition would count each run once.

In our insurance agent. The latency of a run here is the automated part only: from the message to waiting_for_human. The days a recommendation waits for the underwriter are not agent latency. They belong to the human step, which Advanced treats as a dependency of its own.

SLI, SLO, SLA: a measurement, a target, a contract#

Three words do the work from here, and they are easy to mix up.

A service level indicator (SLI) is a measurement of one aspect of the service. Tiny example: "of the 2315 runs this week, 2248 were good" gives an SLI of 97.11%. The SRE workbook recommends writing an SLI as a ratio of good events to total events, so it runs from 0% (nothing works) to 100% (nothing is broken) (Implementing SLOs).

A service level objective (SLO) is the target for that indicator over a period. Tiny example: "at least 95% of runs in a week are good." The SLI is what happened; the SLO is the line it is compared with.

A service level agreement (SLA) is a contract that names consequences when an objective is missed. The SRE book gives a quick test: if nothing explicit happens when the target is missed, it is an SLO, not an SLA (Service Level Objectives). The insurer in this example has no SLA with brokers. Every target on this page is an internal SLO, and that is the normal case: most teams need objectives long before anyone signs a contract about them.

WordQuestion it answersIn this example
SLIWhat share of runs were good?97.11% in the healthy week
SLOWhat share must be good?95% of runs per week (illustrative)
SLAWhat happens to us if we miss it?None: no contract exists
Error budgetHow many bad runs may we have?5% of 2315 runs, 115.75 runs

The last row is the idea the rest of the page builds on. It needs the latency spread first.

Latency is a distribution: p50, p95, p99#

Ten runs took these times, in milliseconds (illustrative numbers): 2100, 2300, 2600, 3000, 3100, 3300, 3900, 4200, 9800, 24000. Their average is 5830 ms. That number describes nobody: eight of the ten runs were faster, and the two slow ones were far slower.

A percentile says how long a given share of runs took at most. The 50th percentile (p50, the median) is 3100 ms here: half of the runs took that long or less. The 90th percentile is 9800 ms: nine runs in ten were at or below it. The example computes it by nearest rank, in slo.py:percentile:

def percentile(values: list[int], p: float) -> int:
    """Nearest-rank percentile: the smallest value with at least p percent of the values at or below it."""
    if not values:
        return 0
    ordered = sorted(values)
    rank = max(1, math.ceil(p * len(ordered) / 100))
    return ordered[rank - 1]

Tail latency is the latency of the slowest runs: p95, p99 and above. The SRE book makes the same point with its own example: a typical request was fast, yet 1 in 20 was about twenty times slower, and an average would not show it. High percentiles show a plausible worst case; the median shows the typical case (Service Level Objectives, linked above).

For an agent the tail matters more than for a web request. One run calls a provider, a retrieval step, a model and two writes. Any one of them can be slow, and a retry with backoff adds its waits on top. In UW-R02 from Retries, two provider timeouts and two backoffs stretch a run that normally ends at 1998 ms to 6958 ms in this example. Nothing failed in the end. The run sits in the tail.

Common confusion. "Our p99 is 9.7 seconds, so 99% of brokers wait less than that." It is 99% of runs. A broker who sends ten requests a day meets the p99 run far more often than 1 time in 100 days. The more steps and the more requests per person, the more of the tail each person sees.

Unacceptable outcomes and cost are SLIs too#

A web service often has two SLIs: availability and latency. An agent needs at least two more.

The unacceptable outcome rate is the share of runs that ended in an outcome the insurer cannot live with: a duplicate write, a lost run, missing evidence. This is the agent's version of "correctness of the side effects", and it is the one a latency graph cannot see. The synthetic week makes that concrete. In the bad-deployment week, a new version writes duplicates for twelve hours. Its p50, p95 and p99 are exactly those of the healthy week: 3256, 4475 and 9739 ms. Only the unacceptable outcomes move, from 18 to 87 in this example.

Cost is the third. A run that loops through retries or re-asks the model may end well, fast, and still spend three times its usual cost. This site counts cost in simulated cost units, never in money (Concurrency sets the ceiling of 6000 per run). An SLI for cost can be "cost units per run" on a dashboard, or a ratio in the same shape as the others: the share of runs that stayed under a cost threshold. In the synthetic week the cost per run is 2996 cost units healthy, 3052 during the brown-out and 3115 with the bad deployment (the week has its own invented cost model; it is not the sum of the scenario's calls).

SLIGood eventWhat it catches that the others miss
LatencyRun done within 8000 msA slow provider, retry waits, a slow model route
Unacceptable outcome rateNo duplicate, no lost run, no missing evidenceA bad version that is fast and wrong in its effects
CostRun under its cost thresholdA loop that re-asks the model and still ends well
Combined (this example)Acceptable outcome AND within the latency thresholdOne number for the budget
What this is not. Whether the recommendation itself was the right one, measured with graders and a golden set, is quality measurement: https://eval.sammuthu.com. An unacceptable outcome here is a fact about the stored rows (two rows for one submission), checkable without judging the content.

The error budget: the failure you agreed to accept#

An SLO of 95% says something useful in reverse: 5% of runs may be bad. That allowance is the error budget. The SRE workbook defines it as 100% minus the SLO; with 2315 runs in the week and an objective of 95%, the budget is 115.75 bad runs.

The budget changes the conversation. A team that aims for 100% treats every bad run as a failure and every change as a risk, so it either stops changing or stops believing its own target. The workbook says plainly that 100% is the wrong target, and that change is the main source of outages (Implementing SLOs, linked above). With a budget, a bad run is a withdrawal from an account. While budget is left, the team ships. When it runs out, the agreed error budget policy says what happens; the workbook's common example is to stop feature launches until the service is back inside its objective. Deployment applies that policy to the agent's releases.

Here is the arithmetic in the example, verbatim from slo.py:

def sli(good: int, total: int) -> float:
    """The service level indicator: the share of good events."""
    return good / total if total else 1.0


def error_budget(slo_target: float, total: int) -> float:
    """How many bad events the objective allows among `total` events."""
    return (1 - slo_target) * total


def budget_consumed(bad: int, slo_target: float, total: int) -> float:
    """The share of the error budget that is used up. Above 1 the objective is missed."""
    allowed = error_budget(slo_target, total)
    return bad / allowed if allowed else (0.0 if bad == 0 else float(bad))


def burn_rate(bad_share: float, slo_target: float) -> float:
    """How fast the budget burns: 1 uses it up exactly over the SLO period, 2 in half the time."""
    return bad_share / (1 - slo_target)

Burn rate: how fast the budget is going#

A budget tells you how much failure is left. It does not tell you how quickly it is going. 115.75 bad runs spread evenly over a week is a normal week. The same 115.75 in one afternoon is an incident.

The burn rate is the speed of spending, relative to the objective. A burn rate of 1 uses the whole budget exactly by the end of the period. A burn rate of 2 uses it in half the period. Tiny example with this page's 95% objective: if 10% of the last hour's runs were bad, the burn rate is 0.10 / 0.05 = 2. If 41% were bad, it is about 8, and a week's budget would be gone in about 20 hours. The SRE workbook defines burn rate the same way and works it through for a 99.9% objective over 30 days, where a burn rate of 10 uses the budget in 3 days (Alerting on SLOs).

Diagram: From one run to one alert: good events, the SLI, the error budget, the burn rate

Read left to right. Every run that ends is judged good or bad. The share of good runs over the week is the SLI, and the objective of 95 percent turns into a budget of bad runs. The burn rate compares how fast that budget is spent with the pace the objective allows, over a long and a short window. The alert fires only when both windows burn too fast. Numbers are from the synthetic brown-out week of the example, seed 7; they are invented.

  1. Each run
    1. A run ends: The automated part of one run is over: it has a terminal state, a set of rows and a duration up to waiting_for_human.
    2. Good or bad?: Good means an acceptable outcome (no duplicate write, no lost run, no missing evidence) and a latency no longer than the threshold.
  2. The week
    1. SLI: The share of good runs over the period, from 0 to 100 percent.
    2. Error budget: How many bad runs the objective allows in the period.
  3. Right now
    1. Burn rate: How fast the budget is being spent: 1 spends it exactly over the period, 8 in one eighth of it.
  4. Act
    1. Alert: A person is asked to act when both the long and the short window burn faster than the rule's threshold.

Alerts on burn, not on every error#

An alert is a notification that asks a person to act now. It costs a person's attention, often at night, so it should fire when the objective is in danger and stay quiet otherwise.

The obvious rule, "page when a run fails", fires on every one of the 18 unacceptable outcomes of a healthy week. Those were inside the budget; nobody needed to wake up. The opposite rule, "page when the week's SLI drops below 95%", fires on Friday for a brown-out that began on Wednesday.

Burn rate gives the middle path. The SRE workbook walks through six ways to alert, from the plain error rate to its recommendation: multiwindow, multi-burn-rate alerts (Alerting on SLOs). Each rule has a long window and a short window. The long window shows that the problem is real and large: a few bad minutes cannot push a 6-hour burn rate over 6. The short window shows that it is still happening, so the alert stops soon after the problem does. The rule fires only when both windows burn faster than its threshold.

The example's version is slo.py:burn_alert, with two illustrative rules in ALERT_RULES:

def burn_alert(windows: list[dict[str, float]]) -> dict[str, Any]:
    """A multi-window rule. Each window is `{"long_burn", "short_burn", "threshold"}`.

    A window fires when the long AND the short burn rate are above its
    threshold: the long one shows the problem is real, the short one shows it
    is still going on. The alert fires when any window fires.
    """
    fired = [w["long_burn"] > w["threshold"] and w["short_burn"] > w["threshold"] for w in windows]
    return {"fires": any(fired), "by_window": fired}
Rule (this example, illustrative)Long windowShort windowFires when both burn faster than
16 hours1 hour6
224 hours6 hours3

These are not Google's numbers. The workbook's table, for a 99.9% objective over 30 days, pages at a burn rate of 14.4 over 1 hour (with a 5-minute short window), pages at 6 over 6 hours (30 minutes), and opens a ticket at 1 over 3 days (6 hours); its short window is one twelfth of the long one. This example has only hourly buckets and a one-week period, so its windows are coarser. The structure is the same.

The synthetic week, run#

The script below sits in a scratch copy of examples/uw-reliability (it is not part of the example; it only calls its functions). It prints the ten-run percentile example from above, then summarizes the three synthetic weeks at the default objective of 95% and a threshold of 8000 ms. A successful run proves the arithmetic: the same functions the lab uses give these numbers for seed 7. It proves nothing about a real agent; the weeks are invented settings, not measurements.

from slo import synthetic_week, summarize, percentile

latencies = [2100, 2300, 2600, 3000, 3100, 3300, 3900, 4200, 9800, 24000]   # ten illustrative runs, in ms
print("mean", sum(latencies) // len(latencies), "p50", percentile(latencies, 50), "p90", percentile(latencies, 90))

for profile in ("healthy", "provider_brownout", "bad_deploy"):
    s = summarize(synthetic_week(profile), slo_target=0.95, latency_threshold_ms=8000)
    print(f"{profile:18} runs {s['runs']}  p50 {s['p50_ms']}  p95 {s['p95_ms']}  p99 {s['p99_ms']}  "
          f"unacceptable {s['unacceptable_total']}  cost/run {s['cost_units_per_run']:.0f}")
    print(f"{'':18} bad {s['bad']} of {s['error_budget_events']:.2f} allowed  SLI {s['sli']:.4f}  "
          f"budget used {s['budget_consumed']:.0%}  alert hour {s['alert']['first_hour']}")
mean 5830 p50 3100 p90 9800
healthy            runs 2315  p50 3256  p95 4475  p99 9739  unacceptable 18  cost/run 2996
                   bad 67 of 115.75 allowed  SLI 0.9711  budget used 58%  alert hour None
provider_brownout  runs 2315  p50 3415  p95 17471  p99 29695  unacceptable 35  cost/run 3052
                   bad 255 of 115.75 allowed  SLI 0.8898  budget used 220%  alert hour 63
bad_deploy         runs 2315  p50 3256  p95 4475  p99 9739  unacceptable 87  cost/run 3115
                   bad 136 of 115.75 allowed  SLI 0.9413  budget used 117%  alert hour 104

Read the three weeks against each other. The healthy week used 58% of its budget and no alert fired: 67 bad runs, most of them slow ones in the normal tail (49 runs over 8000 ms). The brown-out begins at hour 60; the alert fires at hour 63, three hours in, on rule 1 (the last 6 hours burned at 8.21 and the last hour at 15.45). The bad deployment begins at hour 100; the alert fires at hour 104 on rule 1 (6.44 and 8.18). Its latency percentiles are identical to the healthy week. Without the unacceptable outcome SLI, the latency dashboards would have said nothing at all.

The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

Try it: a week against its objective#

Pick a week, move the objective and the latency threshold, and watch the budget, the burn rate and the hour the alert fires. The lab opens on the brown-out week.

Without JavaScript, the three weeks at the lab's default settings (objective 95%, threshold 8000 ms), from slo.py:summarize run in a copy of the example:

WeekRunsp50p95p99Unacceptable (rate)Budget consumedAlert fires
Healthy23153256 ms4475 ms9739 ms18 (0.78%)67 of 115.75 (58%)no
Provider brown-out (hours 60 to 83)23153415 ms17471 ms29695 ms35 (1.51%)255 of 115.75 (220%)hour 63
Bad deployment (hours 100 to 111)23153256 ms4475 ms9739 ms87 (3.76%)136 of 115.75 (117%)hour 104

Then raise the objective to 99%. The budget shrinks to 23.15 bad runs, and in this example the alert fires in hour 3 of every week, the healthy one included. The normal tail alone burns faster than a 99% objective allows. An objective the system never meets turns its alert into noise that everyone learns to ignore. The SRE book warns against both extremes: against absolutes such as "always available", and against copying today's performance as the target without asking what users need.

How SLOs go wrong#

MistakeWhat happensIn this example
Latency is the only SLIA version that writes duplicates fast looks healthyBad deployment: percentiles unchanged, 87 unacceptable outcomes
Alert on every errorPages for failures that were inside the budget; people mute the pager18 unacceptable outcomes in the healthy week, no action needed
Alert on the period's SLIThe alert comes days lateThe brown-out would page on Friday instead of at hour 63
Objective tighter than realityThe alert fires all week99% objective: alert in hour 3 of the healthy week
Average latency in place of percentilesThe slow runs disappearTen runs: average 5830 ms, worse than 8 of them and far better than the slowest
Human wait counted as agent latencyEvery run looks days slowLatency stops at waiting_for_human
No error budget policyThe budget runs out and nothing changesThe Thursday prompt ships into a spent budget
What this is not. Collecting the metrics, logs and traces that feed an SLI, and seeing a failure as it happens, is observability: https://observe.sammuthu.com. This page owns the arithmetic on top: what counts as good, the objective, the budget, the burn and when to act.
Before AI. A Spring Boot service behind a load balancer has the same machinery: a request counter by status code, a latency histogram, p99 on a dashboard, and an alert on the error ratio. A Kafka consumer adds consumer lag. All of that carries over. Three things differ for an agent. The unit is a run of seconds or minutes, not a request of milliseconds, so there are far fewer events per hour and each one weighs more. "Good" includes the effects the run left behind, which a status code does not show: a run can return 200 and leave two rows. And a third SLI, cost, matters because one run can spend many model calls.

The same idea for a coding agent#

A coding agent takes a task, edits the repository, runs the tests, opens a pull request and waits for CI. A good event can be: the task ended with exactly one pull request, CI passed on it, and it took under 30 minutes of agent time (waiting for a human reviewer excluded). Bad events include a second pull request for the same task, a run that never ended, and a pull request whose CI failed on a test the agent never ran. The burn-rate alert then tells the team when a new model route or a changed instruction file starts producing bad runs faster than the budget allows. The runtime around such an agent is https://harness.sammuthu.com; the path from a change to production is https://diff-to-deploy.sammuthu.com.

The blocks below hold the detailed layer.

Few events: when one bad run is a large shareAn agent has far fewer events than a web service, and that changes the windows

The synthetic week has 6 to 24 runs per hour. With 10 runs in an hour, one bad run is a 10% bad share, a burn rate of 2 against a 95% objective; three bad runs make a burn rate of 6. A short window of one hour therefore swings hard, which is why the example's rules pair it with a 6-hour long window. Google's workbook has a section on exactly this problem, low-traffic services, and lists the options it considered: generate artificial traffic, combine several small services into one objective, change the service so that fewer events are needed, or lower the objective or lengthen the window. For an agent, synthetic runs of the fault scenarios of Fault injection are a natural source of artificial traffic, as long as they are kept out of the business numbers.

The second consequence is statistical. A week of 2315 runs at a true bad share of 3% will show anything from about 2.3% to 3.7% by chance alone (a rough binomial spread, given as an illustration). An objective of 97% on such a service is a coin toss each week. Choose the period and the objective so that the noise is small next to the margin.

Bad events that are only known laterA duplicate row or missing evidence may be found hours after the run ended

A slow run is known the moment it ends. An unacceptable outcome is often not: the duplicate row is found by a nightly reconciliation, the missing evidence by an auditor's query, the lost run by a sweep for runs stuck in running. The SLI of yesterday then changes today.

Two designs handle this. The first computes the SLI from checks that run at the end of every run, the way invariants.py:check_invariants reads the rows left in the store; the invariants of Fault injection become production checks. The second accepts late data and recomputes past windows, alerting on the fast checks and reporting the full SLI with a delay. Either way, write down when a run's verdict becomes final. An alert that waits for the nightly job cannot fire at hour 63.

Percentiles do not add upThe p99 of a week is not the average of the hourly p99s

A percentile is a property of a whole set. The p99 of a week cannot be computed from 168 hourly p99s, and averaging them gives a number with no meaning. Either keep the raw latencies (the synthetic week does: every run's latency is in its hour), or keep a histogram with fixed buckets that can be summed, and read percentiles from the summed histogram, at the cost of precision inside a bucket.

The ratio form of a latency SLI avoids the problem: "share of runs under 8000 ms" is a count of good runs over a count of all runs, and counts add up across hours, services and regions. That is one reason the workbook prefers SLIs as ratios. The workbook's worked example also uses two latency grades, 90% of requests under 450 ms and 99% under 900 ms, so that both the typical case and the tail have a line.

One combined SLI or severalThe example folds latency and outcomes into one budget; that is a choice with a cost

slo.py:summarize counts a run as bad if it was slow or unacceptable, and spends one budget. That is simple and gives one answer to "may we ship?". The cost is that a slow week and a duplicate-writing week look alike in the budget, while they need different responses: the first points at a provider, the second at a release. They also deserve different weights: one duplicate recommendation in front of an underwriter is worse than one run that took 9 seconds.

The alternative is one objective per SLI, each with its own budget and burn alert: for instance 95% of runs within 8000 ms, and 99.5% of runs free of unacceptable outcomes (illustrative). It costs more alert rules and more care, and it answers the Monday questions separately. A middle path keeps one combined budget for release decisions and separate burn alerts for diagnosis.

Checklist#

QuestionGood answer
Is a good run defined in writing, before the period starts?Yes: acceptable outcome, within a latency threshold, maybe under a cost threshold
Does the SLI include the effects a run left behind?Yes: duplicates, lost runs and missing evidence count as bad
Is latency reported as percentiles, not an average?Yes: p50 and p99 at least
Does agent latency stop at the human step?Yes
Is cost tracked per run, in units, not only per month?Yes
Is the objective below 100% and near what users need?Yes, and the system can meet it in a normal week
Do alerts fire on burn rate over a long and a short window?Yes, not on single errors and not on the period's SLI
Is there an agreed error budget policy?Yes: what stops when the budget is spent

In one breath#

A run is good when it ends in an acceptable outcome and fast enough, and the share of good runs is the SLI; the SLO is the target for that share, and one minus the target is the error budget. Latency is a distribution, so watch p95 and p99, and add an unacceptable outcome rate and cost per run, because a bad agent version can be fast and still write duplicates. Alert on how fast the budget burns, over a long and a short window together, not on every error.

Next: the change that spends the budget#

The Monday questions now have answers: the brown-out week missed its objective, the alert should have fired at hour 63, and the Thursday prompt has to wait for the budget to recover. That last answer points at the most common cause of a spent budget: a change. In the bad-deployment week the alert fired at hour 104, yet the bad version kept writing duplicates until its rollback at hour 112. Deployment is about releasing a new prompt, model or version to a few runs first, so that a bad change spends a little of the budget instead of all of it.