All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Mechanisms · 8

Concurrency: rate limits, budgets and distributed locks

Many runs share one provider quota, one budget and one record. Concurrency limits, token buckets, spending budgets, and why a lock needs a lease and a fencing token.

What you will understand
Limits
A concurrency limit caps calls in flight; a token bucket caps calls per second. Why an agent should pace itself before the provider has to refuse it.
Budgets
Why every run, and every day, needs a ceiling on cost units that is checked before a model call, not after.
Locks that expire
Why a lock on a submission needs a lease, why a lease alone is unsafe, and how a fencing token lets the store refuse a stale write.

Worker A holds submission SUB-1004. It has validated it, fetched the roof assessment, retrieved the guidance and had the model draft the recommendation: escalate, citing R-ROOF-03. The model recommends; a person decides. At 1974 ms of simulated time, right before it stores the recommendation, worker A stops. Not a crash: a pause. In a real system this is a long garbage-collection stop, a virtual machine moved to another host, a laptop lid closed on a developer's test run. The process is alive and holds all its state, but it is not running.

This is scenario UW-R14. Worker A holds a lease on the submission that lasts 20000 ms in this example, so that a dead worker cannot block it for ever. The pause lasts 45000 ms. While worker A is frozen, its lease runs out, the queue's visibility timeout passes, and at 30000 ms worker B receives the same message and takes the lease. Worker B finishes the run. At 46974 ms worker A wakes up, with no idea that time has passed, and carries on from where it was: it writes its recommendation.

The naive workflow lets it. The end state:

  recommendation rows 2 (pending_review 2, withdrawn 0) · review tasks 2 · decisions 0 · dead letters 0
  REC-0001  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-cc48  route primary  degraded false  fencing_token 42  key None
  REC-0002  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-0d05  route primary  degraded false  fencing_token 41  key None

Two recommendations and two review tasks for one submission (INV-1 broken). Two underwriters may now pick up the same file. Nothing failed in the usual sense: no timeout, no error, no crash. Two healthy workers each did their job, at the same time, on the same record.

This page is about what many runs share. Once the insurance agent runs more than one submission at a time, its runs compete for three things: the provider's and model's quota, the money (here, cost units) the insurer is willing to spend, and individual records such as SUB-1004. Each needs its own mechanism.

Many runs at once share quotas, budgets and records#

A race condition is a bug whose outcome depends on the timing of two activities that touch the same thing. The tiny example: two workers each check "is there a recommendation for SUB-1004?", both see "no", and both insert one. Run them one after the other and the second sees the first row; run them at the same moment and both write. Most of the time the timing is harmless, which is what makes a race hard to find.

The three shared things fail in different ways:

Shared thingWhat goes wrong without a limitThe mechanism on this page
Calls in flight to one dependencyOne slow dependency takes every workerConcurrency limit
Calls per second to the model or the providerThe provider answers 429 to everyone, or a quota runs out for the dayRate limit (token bucket)
Cost units per run and per dayA run that loops on the model spends without endBudget, with a ceiling
One record, such as a submissionTwo workers both act on itLease, with a fencing token

The first two protect a dependency from us. The third protects the insurer from a runaway run. The fourth protects the record from two workers. The rest of the page takes them in that order.

Diagram: Four shared things, four limits · what each one protects

Read each column top to bottom: the thing many runs share, then the limit that guards it, then what goes wrong without it. The first two protect a dependency from the agent; the budget protects the insurer from a runaway run; the lease and fencing token protect one record from two workers.

  1. Calls in flight
    1. Concurrency limit: Caps how many calls to one dependency are in progress at the same time. Not wired into the example's workflow.
  2. Calls per second
    1. Token bucket: Admits a call only if a token is available; tokens refill at a fixed rate up to a burst size. Not wired into the example's workflow.
  3. Cost units
    1. Budget: Refuses a model call the run cannot afford, before it is sent. Cost is counted in cost units, never money.
  4. One record
    1. Lease: Gives one worker the submission for a limited time, so a dead worker cannot hold it for ever.
    2. Fencing token: A number that grows with every lease; the store refuses a write with a lower one, so a worker whose lease expired cannot write.

A concurrency limit caps how many calls are in flight#

A concurrency limit is the most calls that may be in progress at the same time. If the limit for the roof-data provider is 8, the ninth run that wants the provider is told "not now" until one of the eight finishes. Breakers introduced it as a bulkhead: a cap per dependency, so one slow dependency cannot take every worker.

The example's version is ConcurrencyLimit in limits.py:

class ConcurrencyLimit:
    """At most `n` calls in flight at the same time (a bulkhead for one dependency)."""

    def __init__(self, n: int) -> None:
        self.n, self.in_flight = n, 0

    def try_acquire(self) -> bool:
        if self.in_flight >= self.n:
            return False
        self.in_flight += 1
        return True

    def release(self) -> None:
        self.in_flight = max(0, self.in_flight - 1)

ConcurrencyLimit and the TokenBucket below are not wired into the workflow of the example. They exist for the labs and the unit tests; the README says so. The workflow's protection against a failing dependency is the breaker, and against a busy one, the 429 handling of Retries. A production system would add both limits.

A rate limit caps calls per second: the token bucket#

A concurrency limit counts calls at the same moment. A rate limit counts calls over time: at most 5 calls per second to the model, say. Model and data providers set rate limits on their side and answer 429 when a client goes over (the error is described on Failure modes). A client that waits for the 429 is letting the provider do its pacing for it, and every refused call is a wasted round trip that may also count against its quota.

The usual way to enforce a rate on your own side is a token bucket. Picture a bucket that holds at most 10 tokens and gains 5 tokens every second. Each call takes one token. If the bucket has a token, the call goes now; if it is empty, the call waits or is refused. The size of the bucket is the burst (how many calls may go at once after a quiet period); the refill rate is the long-run rate. Amazon's Builders' Library describes the token bucket as a common way to admit requests, with tokens refilling at a set rate up to a maximum burst, and per-tenant quotas so that one tenant's surge does not fail the others (Fairness in multi-tenant systems).

The example's TokenBucket, verbatim from limits.py:

class TokenBucket:
    """A rate limit: `rate_per_s` tokens flow in every second, up to `burst`. A call takes one."""

    def __init__(self, rate_per_s: float, burst: float) -> None:
        self.rate_per_s, self.burst, self.tokens = rate_per_s, burst, burst
        self.last_ms = 0

    def try_take(self, now_ms: int, n: float = 1) -> bool:
        """Refill for the time that passed, then take `n` tokens if they are there."""
        self.tokens = min(self.burst, self.tokens + (now_ms - self.last_ms) * self.rate_per_s / 1000)
        self.last_ms = now_ms
        if self.tokens < n:
            return False
        self.tokens -= n
        return True

It stores no timer and no queue: on each call it adds the tokens earned since the last call, capped at the burst, then tries to take one. That is why a bucket is cheap enough to sit in front of every call.

Try it: a token bucket over one minute#

Set the refill rate, the burst, how many calls arrive per second, and what happens when no token is left: wait (up to a longest wait) or refuse.

Without JavaScript, the lab's default run (from limits.py:simulate_bucket, run in a copy of the example): refill 5 tokens per second, burst 10, 8 calls arriving every second for 60 seconds, policy "wait up to 3000 ms".

ArrivedAdmitted at onceAdmitted after a waitRefused (waited too long)Still waiting at the end
4802028514827

The arithmetic explains the table. Calls arrive at 8 per second and tokens at 5, so 3 calls per second more than the bucket can admit. The burst of 10 hides the gap for the first two seconds; after that a queue builds, and calls that would wait longer than 3000 ms are refused. Switch the policy to "refuse" and the same minute admits 305 calls and refuses 175 at once. A rate limit does not create capacity. It decides who waits and who is told no, early and cheaply, instead of letting the provider decide with 429s.

Production pitfall. A token bucket inside each worker process limits each process, not the system. Ten workers with a bucket of 5 per second each send up to 50 per second to a provider whose limit is 5. A rate limit that must hold for the whole system lives in one shared place (a shared store or a gateway in front of the provider), or each process gets a share of the limit that is updated as processes come and go.

Budgets: a ceiling on cost units per run and per day#

A rate limit protects the provider. A budget protects the insurer: a ceiling on what one run, or all runs in a day, may spend. On this site cost is counted in simulated tokens called cost units, never in money (decision Z11 of the series). In this example a draft on the primary model route costs 1340 cost units and the ceiling for one run is 6000.

Why would a run spend more than one draft? Because the reliability mechanisms of the earlier pages each add model calls. UW-R05 asks again after one malformed answer and spends 2680 cost units: the broken answer was paid for. A resume without a checkpoint re-drafts. A bug that loops between drafting and validation would draft until something stops it. Each mechanism is bounded on its own page; the budget is the bound on their sum.

The example's Budget in limits.py asks before a call whether the run can afford it, and charges after. The script below was written for this page and imports the example's classes; it is not part of the example's files. It shows the three limits refusing, nothing more. Run in a copy of examples/uw-reliability:

from errors import BudgetExhausted
from limits import Budget, ConcurrencyLimit, TokenBucket

bucket = TokenBucket(rate_per_s=2, burst=3)         # 2 tokens per second, at most 3 saved up
print("at 0 ms:  ", [bucket.try_take(now_ms=0) for _ in range(4)])
print("at 500 ms:", bucket.try_take(now_ms=500))     # half a second refills 1 token

slots = ConcurrencyLimit(2)                           # at most 2 calls in flight
print("in flight:", [slots.try_acquire() for _ in range(3)])

budget = Budget(limit_units=6000)                     # the cost ceiling of one run, in cost units
for draft in range(1, 6):
    try:
        print("draft", draft, "spent", budget.charge(1340))
    except BudgetExhausted as error:
        print("draft", draft, "refused:", error)
at 0 ms:   [True, True, True, False]
at 500 ms: True
in flight: [True, True, False]
draft 1 spent 1340
draft 2 spent 2680
draft 3 spent 4020
draft 4 spent 5360
draft 5 refused: 1340 cost units asked, 640 left of 6000

The fifth draft is refused before it is sent, because charge calls require first, and the workflow calls require before every model call (Workflow._call_model). A ceiling checked after the call is a report, not a limit: the money is already spent. When the ceiling is reached the workflow does not crash and does not quietly stop. BudgetExhausted is a permanent error, so it is not retried, and draft_recommendation writes an escalate in code with the reason "the run reached its cost ceiling", marked degraded. A person sees why the run stopped.

A per-day budget is the same idea one level up: a shared counter of cost units for all runs, checked before each run starts or each model call. The example has only the per-run ceiling; a per-day budget is a design this site recommends, and the cost ceilings that stop a runaway are developed further on Advanced.

A distributed lock needs a lease, and a lease is not enough#

Back to UW-R14. The insurer wants one worker at a time on a submission. Inside one process a mutex would do it. Across many processes, on many machines, the lock must live somewhere they all can see: a distributed lock, a record in a shared store that says which worker currently owns SUB-1004.

A plain lock has a fatal flaw for workers that can crash: a worker that dies holding it holds it for ever. So the lock is granted for a limited time and must be renewed. That is a lease: a lock that expires. In the example, LeaseManager.acquire in lease.py grants a lease of 20000 ms, and a worker that does not come back in time loses it, so another worker can take over.

def acquire(self, resource: str, owner: str, ttl_ms: int) -> int | None:
    """The fencing token if the lease is free or expired, else None."""
    held = self.leases.get(resource)
    if held and held["owner"] != owner and held["expires_ms"] > self.clock.now():
        return None
    token, self.next_token = self.next_token, self.next_token + 1
    self.leases[resource] = {"owner": owner, "token": token, "expires_ms": self.clock.now() + ttl_ms}
    return token

Expiry solves the dead worker and creates the slow one. A worker that pauses does not know that its lease ran out while it was paused. Checking "do I still hold the lease?" immediately before writing does not help, because the pause can fall between the check and the write. Martin Kleppmann's essay on distributed locking makes exactly this argument, with garbage-collection pauses, page faults and delayed network packets as causes (How to do distributed locking), and Amazon's Builders' Library article on leader election names slow networks and garbage-collection pauses between checking a lease and doing the work as the hard part of using leases (Leader election in distributed systems).

The worker cannot be trusted to know. The store can.

The fencing token: the store refuses a write from an old lease#

Look again at the return value of acquire: a number. Every time a lease on any submission is granted, the number goes up by one. In this example the first lease of the story gets 41 (the code's comment: the store has granted forty leases before this story starts). Worker A got 41. When worker B takes the expired lease, it gets 42.

That number is a fencing token: a number that grows with every lease granted, sent with every write, and checked by the store, which refuses any write carrying a number lower than one it has already seen. Kleppmann's essay describes this design, with the storage server rejecting a write whose token is older than one it has processed. The check lives in the store, not in the worker, because the worker is the party that may be wrong about time.

The example's check, Store.check_fencing in store.py (dedented from the class). The worker shows its token to the store as soon as it gets the lease (note_token), and every write to the recommendations table passes through this check before the insert:

def check_fencing(self, submission_id: str, token: int | None) -> None:
    """Refuse a token lower than the highest seen for this submission."""
    if not self.fencing or token is None:
        return
    highest = self.highest_token.get(submission_id, 0)
    if token < highest:
        raise StaleFencingToken(f"fencing token {token} is older than {highest}: the lease was lost")
    self.highest_token[submission_id] = token

The sequence of UW-R14, careful:

Diagram: UW-R14 · a stalled worker, an expired lease, and the store that refuses token 41

Time runs down the page. Worker A takes the lease with fencing token 41 and stalls before its write. The lease expires while it is frozen. When the queue redelivers the message, worker B takes the lease with token 42, resumes the run from its checkpoint and writes. When A wakes up it still believes it may write; the store compares tokens and refuses. Times are from the simulated clock of the example, which runs in one process.

  1. Lease manager and queue to Worker A (token 41): Lease on SUB-1004, token 41. Worker A gets the lease for 20000 ms and shows token 41 to the store at once.
  2. Worker A (token 41): Stall for 45000 ms before the write. Steps 1 to 5 are done and checkpointed. At 1974 ms the worker freezes. It cannot see its own pause.
  3. Lease manager and queue: Lease expires; message visible again. The lease ran out at 20000 ms. The message was never acknowledged, so after the visibility timeout the queue can deliver it again.
  4. Lease manager and queue to Worker B (token 42): Redeliver; lease granted, token 42. Worker B finds the same run by message_id and gets the next token. The store now knows 42 is the highest token for this submission.
  5. Worker B (token 42) to Store: Write the recommendation with token 42. B resumes from the checkpoint (no second model call) and writes. 42 is not lower than 42: accepted.
  6. Worker A (token 41) to Store: Wake up; write with token 41. Worker A carries on as if no time had passed. It never checked the clock, and checking would not help: the pause can fall between a check and a write.
  7. Store to Worker A (token 41): Refused: 41 is older than 42. A permanent error, StaleFencingToken. Worker A stops and does not retry. One recommendation row; all seven invariants hold.

Run it in a copy of examples/uw-reliability. A successful run proves that, in this simulation, the stale worker's write is refused and the store keeps one recommendation. It does not prove anything about real clocks or real pauses: the example runs in one process, and "two workers at once" is simulated by letting the clock run while one worker stalls (the README says so in "What this example leaves out").

/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R14

The timeline from the stall on, and the end state, verbatim:

   1974  worker-A  record_recommendation    -  WORKER STALLS: for 45000 ms, before its write
  30000  worker-B  -                        -  message received: msg-83a3, receive 2
  30000  worker-B  -                        -  run found: by message_id; resume_count 1
  30000  worker-B  -                        -  lease granted: fencing token 42
  30000  worker-B  draft_recommendation     -  resume from checkpoint: steps 1 to 5 are not run again
  30012  worker-B  record_recommendation    1  attempt ok
  30012  worker-B  record_recommendation    -  step done: checkpoint saved
  30024  worker-B  open_review_task         1  attempt ok
  30024  worker-B  open_review_task         -  step done: checkpoint saved
  30024  worker-B  await_underwriter        -  run ended: waiting_for_human
  30024  worker-B  await_underwriter        -  step done: checkpoint saved
  30024  worker-B  -                        -  acknowledged
  46974  worker-A  record_recommendation    -  worker wakes up: still carrying fencing token 41
  46986  worker-A  record_recommendation    1  attempt failed: StaleFencingToken: fencing token 41 is older than 42: the lease was lost
  46986  worker-A  record_recommendation    -  write refused: fencing token 41 is older than 42: the lease was lost; this worker stops
end state:
  run run-0d05  state waiting_for_human  resume_count 1  ended at 30024 ms (deadline 120000 ms)
  recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
  REC-0001  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-0d05  route primary  degraded false  fencing_token 42  key run-0d05:record_recommendation

Worker A's write is refused at 46986 ms with token 41, and worker A stops; it does not retry, because StaleFencingToken is a permanent error. The stored row carries token 42. All seven invariants hold. Worker B resumed the same run (resume_count 1) from its checkpoint, so the model was not asked a second time.

The example's README adds a fair point: in UW-R14 the idempotency key alone would also have stopped the second row, because both workers make the same intended write, run-0d05:record_recommendation. The fencing token stops the stale worker one step earlier and does not depend on the two writes being equal. If worker A had held a different draft (a model that answered differently, a policy fixture that changed while it slept), the key would have refused it as a conflict at best; the token refuses it because its authority expired, which is the actual reason it must not write.

Try it: two workers, one submission#

Set the lease length, how long worker A stalls before its write, and switch the fencing token on or off.

Without JavaScript, three runs of lease.py:simulate_lease (lease 20000 ms, 2000 ms of work before the stall), from a copy of the example:

Worker A stallsFencingWhat happensRows
10000 msonA writes at 12000 ms, inside its lease, and releases it; B then finds the row and writes nothing1
45000 msonA's lease expires at 20000 ms; B gets token 42 and writes at 22000 ms; A wakes and writes with token 41 at 47000 ms: refused (as in UW-R14)1
45000 msoffThe same, but the store cannot tell that A's lease is gone: A's write is accepted2

The timings in the lab differ from the scenario (there B arrives at 20000 ms, the moment the lease expires; in UW-R14 B arrives at 30000 ms, when the queue redelivers). The outcome is the same.

The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

Before AI: optimistic concurrency and the version column#

A backend engineer has met the fencing token before under another name. A relational table with a version column, and an update that only applies if the version is still the one the reader saw, is optimistic concurrency. The shape below is generic SQL, illustrative and not run here:

-- illustrative: the update applies only if nobody changed the row since it was read at version 7
UPDATE submissions
SET    status = 'in_review', version = version + 1
WHERE  submission_id = 'SUB-1004' AND version = 7;
-- 0 rows updated means someone else got there first: re-read, decide again

The common idea is a number that only grows, checked by the storage at the moment of the write. The version column puts it on the row and needs no lock at all: readers proceed freely and the losers find out at write time. The fencing token puts it on the lease, which suits work that should not even be attempted twice, because the attempt is expensive. For an agent the attempt is the expensive part: a model call that costs cost units and takes seconds. A version column would refuse worker A's write too, but only after both workers had done all the work. What is new for an agent is also the length of the critical section: a database transaction holds its lock for milliseconds, while a run holds its lease across a model call and tool calls that take seconds, which is exactly when pauses and expiries happen.

How these mechanisms go wrong#

MistakeWhat happens
A lock with no expiryA worker that dies holds the submission for ever
A lease with no fencing tokenA paused worker writes after its lease expired (naive UW-R14: 2 rows)
The worker checks its lease before writing, instead of the store checking the tokenThe pause falls between the check and the write; the check proves nothing
The lease length is set by the worker's wall clockA clock that jumps changes when leases expire; the Builders' Library article notes that leases rely on elapsed time, not synchronized wall clocks
A rate limiter per process for a system-wide limitN processes send N times the limit
A budget checked after the model callThe overspend has already happened
A budget counted in currency in code and testsPrices change and leak into tests; count units, convert at the reporting edge
A concurrency limit with no release on errorEach failed call leaks a slot until the limit is reached and everything waits

The same idea for a coding agent#

Two instances of a coding agent are assigned the same issue, one after a timeout on the first. Both create the same branch and push. The second push overwrites the first agent's commits, or both open pull requests. The fix has the same parts: a lease on the issue (with expiry, so a dead agent releases it), and a write that the remote accepts only if the branch is still at the commit the agent last saw, which is optimistic concurrency on the branch. A per-task budget of model tokens stops an agent that loops on a failing test from spending all day on it. The runtime that enforces such budgets is the subject of https://harness.sammuthu.com; the review and CI path, of https://diff-to-deploy.sammuthu.com.

What this is not. Deciding which worker or which agent should take which piece of work (scheduling, routing between agents, the control plane) is orchestration: https://orchestrate.sammuthu.com. This page owns what keeps two of them from damaging the same record or the same budget once they are running.

The four blocks below hold the detailed layer.

Efficiency locks and correctness locksDecide what a lock failure costs before choosing how strong the lock must be

Kleppmann's essay separates two reasons to lock. An efficiency lock saves duplicate work: if it fails, two workers do the same job and the cost is some extra spending. A correctness lock protects state: if it fails, data is corrupted or duplicated. The first can live with a simple lease in a fast store. The second needs the store that holds the data to enforce a fencing token (or a version check) on every write, because no lease service alone can stop a paused client.

In this workflow the lease on a submission is both. It saves cost units (worker B does not redraft while A works), and it protects INV-1 (one pending_review row per submission). The design therefore uses the lease for efficiency and puts the correctness guarantee where the row is written: the fencing check and the idempotency key, both inside the store's write path, both in the same commit as the insert.

Choosing the lease length, and renewing itToo short and workers lose leases while working; too long and a dead worker blocks the submission

A lease must outlast the longest normal gap between renewals, and expire soon enough that a dead worker's submission is picked up in acceptable time. In this example the lease (20000 ms) is shorter than the queue's visibility timeout (30000 ms), so by the time the message is redelivered the old lease is gone, and the new worker gets it at once. If the lease were longer than the visibility timeout, a redelivered message would find the lease busy and go back to the queue (Worker.handle returns "nack" on "lease busy").

A run that holds its lease across a slow model call should renew it, for example after each step, and should treat a failed renewal as a lost lease: stop before the next write. Renewal narrows the window; it does not close it, because the pause can still fall between a renewal and a write. The fencing token is what closes it.

Fair shares between brokers, and per-day budgetsOne broker's batch must not use the whole model quota

A single rate limit for the model protects the provider, but not the brokers from each other. If one broker sends two hundred submissions at nine in the morning, a shared bucket admits them first and everyone else waits. The Builders' Library fairness article describes quotas at the granularity of a tenant or a workload for this reason. For the insurer, a design could give each broker (or each line of business) its own bucket, sized from its normal volume, plus a shared bucket for bursts.

The same holds for budgets. A per-day ceiling of cost units for all runs is easy to state and easy to hit unfairly. Splitting it per broker, with a reserve for resumed runs (which have already spent and are cheap to finish), keeps a runaway in one place from stopping the rest. These are designs; the example has only the per-run ceiling.

Accounting for a call whose cost is not known in advanceReserve the maximum, charge the actual, release the difference

Budget.require asks for the cost of a draft before the call, and charge books the actual cost after. In the example the two are equal, because the scripted model has a fixed cost per route. A real model call's cost depends on the length of its output, which is known only afterwards. The usual pattern is to reserve the maximum the call may cost (the route's output limit), refuse the call if the reservation does not fit, then charge the actual amount and release the rest.

Two consequences. Concurrent runs that share a per-day budget must reserve atomically in the shared store, or two runs that each see "enough left" both spend it: a race condition on the budget itself. And a retried call must be charged again, because the provider may have done the work for the failed attempt; a budget that charges only successful calls undercounts exactly when things go wrong.

Checklist#

QuestionGood answer
Is the number of calls in flight to each dependency capped?Yes, and slots are released on every path, including errors
Does the agent pace its own calls to each rate-limited provider?Yes, with a token bucket shared across processes
Is each broker's share of the quota bounded?Yes, per broker or per workload
Does every run have a cost ceiling in cost units, checked before each model call?Yes, and reaching it ends the run in a named, explained state
Is there a per-day ceiling, reserved atomically?Yes
Does every lock on a record expire?Yes, a lease, renewed while the run works
Does the store refuse writes from an expired lease?Yes, by fencing token (or version), checked inside the write
Is a stale-token refusal treated as permanent, not retried?Yes; the worker stops
Was each mechanism tested with an injected pause or burst?Yes

In one breath#

Once many runs share a provider, a budget and a record, each needs its own limit: a concurrency limit for calls in flight, a token bucket for calls per second, and a ceiling of cost units checked before every model call. A lock on a record must expire, so it is a lease, and a lease alone lets a paused worker write after it lost the lease. A fencing token, a number that grows with every lease and is checked by the store, makes the store refuse that stale write, the same idea as a version column in a relational table.

Next: how reliable is reliable enough#

The insurance agent now has every mechanism of this series: retries, idempotent writes, a queue with dead letters, checkpoints and compensation, breakers and degradation, limits, budgets and fenced leases. Each one was shown working on one scenario. What none of them says is how often the agent may fail, how slow it may be, or how much it may spend before someone should act. SLOs turns those questions into numbers: indicators, objectives, and an error budget that tells the team when reliability needs attention more than new features do.