All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Mechanisms · 4

Idempotency: when the same request arrives twice

A retry cannot know whether the first try was applied. Idempotency keys, deduplication and the designs that make a repeated write leave one result.

What you will understand
The lost acknowledgement
Why a retry of a write cannot know whether the first try already happened, and what that costs.
The key
An idempotency key names one intended write: who makes it, what it is made of, where it lives, and why the attempt number must stay out of it.
Beyond one table
Deduplication, the honest meaning of "exactly once", and the outbox for a change that must also reach another system.

The underwriting agent has finished its work on submission SUB-1004. The roof is 9 years old according to the application and 22 according to the roof-data provider, so the draft is a recommendation to escalate, citing rule R-ROOF-03. The model recommends; a person decides. Step 6, record_recommendation, stores that recommendation as a row so the underwriter can review it.

In scenario UW-R06 the store receives the write, commits the row, and sends back its confirmation. The confirmation never arrives. From the agent's side, the call failed exactly like a timeout:

   1986  worker-A  record_recommendation    1  attempt failed: AckLost: store.record_recommendation: committed, but the acknowledgement was lost

The previous chapter built a careful retry loop, and both workflows of the example use some retry here. Retrying is the right instinct: the error looks transient, and the deadline has plenty of room. The naive workflow retries, the second attempt succeeds, and the run ends normally. Then you look at the table:

  recommendation rows 2 (pending_review 2, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
  REC-0001  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-4708  route primary  degraded false  fencing_token 41  key None
  REC-0002  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-4708  route primary  degraded false  fencing_token 41  key None

Two recommendations for one submission, both waiting for review. Nothing crashed, and every component did what it was built to do. The underwriter now has two items for one broker request, and if they ever differed, no way to know which is the agent's answer. The invariant INV-1 (at most one pending_review row per submission) catches it.

This chapter is about one fact and what follows from it: a timeout tells you nothing about whether the write happened. One note before more results: the agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

1. Three places a write can fail, and why the caller cannot tell them apart#

A write to the store passes through two moments that matter. First the store commits: the row is now durable. Then the store's answer travels back to the caller. A failure can land before the commit, after it, or after the caller has received the answer but before it has saved its own progress. The example has a scenario for each.

In UW-R07 the store fails before the commit. Nothing was written. A retry is the first real write, and one row results whether or not anything else protects the write.

In UW-R06 the store fails after the commit: the row exists, but the answer is lost. This is the lost acknowledgement. The caller receives an error that is indistinguishable from the first case. A retry now writes a second row, unless the store can recognize it.

In UW-R08 the write and its answer both succeed, and then the process crashes before it saves its checkpoint (the record of which steps are done). A new process resumes the run from the last checkpoint, which says step 5 is the latest finished step, so it executes step 6 again. Same problem, larger gap: the second attempt comes 28 seconds later from a different machine.

ScenarioWhere the failure landsRow exists after the failure?What the caller seesNaive outcome
UW-R07Before the commitNoAn error1 row: a retry is enough here
UW-R06After the commit; the acknowledgement is lostYesThe same kind of error2 rows
UW-R08After the answer, before the checkpointYesNothing: the process is gone; a new one resumes2 rows, and a run left in running

The middle column is the whole problem. The caller sees the third column; only the store knows the second. So the decision "is this a new write or a repeat?" cannot be made by the caller. It has to be made by the store, and the caller has to give the store enough information to make it.

Diagram: UW-R06, careful · the row commits, the answer is lost, the same key turns the retry into a replay

Time runs down the page. The worker sends the write with its key. The store commits the row and the key together, then the answer is lost, so the worker sees an error it cannot tell apart from a failure before the commit. The retry carries the same key, and the store answers with the stored row. Times come from the simulated clock of the example.

  1. Step 6 (worker-A) to Store (rows and keys): Write the recommendation, key run-4708:record_recommendation. Attempt 1 at 1974 ms. The payload is the escalate recommendation; a fingerprint of it travels with the key.
  2. Store (rows and keys): Commit the row and the key together. REC-0001 and its key are now durable. From here on the store knows this write happened.
  3. Store (rows and keys) to Step 6 (worker-A): Answer lost: the worker sees an error. The injected fault ack_lost fires after the commit. The worker receives AckLost at 1986 ms, which looks the same as a failure before the commit.
  4. Step 6 (worker-A): Back off 52 ms. The error is transient, so the retry loop waits with full jitter, inside the deadline and the retry budget.
  5. Step 6 (worker-A) to Store (rows and keys): Write again, the same key. Attempt 2. The key is the same because it is built from the run and the step, never from the attempt number.
  6. Store (rows and keys) to Step 6 (worker-A): Stored row REC-0001, replayed. The key is known and the fingerprint matches, so nothing is written. The worker receives the same row the first attempt would have received, at 2050 ms.

2. Idempotent: doing it twice leaves what doing it once left#

Some operations are safe to repeat by their nature. Reading the roof assessment twice changes nothing. Setting the status of a review task to open twice leaves it open. An operation is idempotent when doing it twice leaves the same result as doing it once. RFC 9110, section 9.2.2, gives the HTTP version of this definition: the intended effect of several identical requests equals the effect of one. It lists PUT, DELETE and the read-only methods as idempotent, and it says a client should not automatically retry a request that is not idempotent unless it knows the request is safe to repeat or can detect that the first one was never applied.

"Insert a recommendation row" is not idempotent. Each insert adds a row. Most writes an agent makes are of this kind: create a task, send an email, open a pull request, charge a card. Each is "do it", not "make it so".

There are two ways to make such a write safe to repeat. One is to change the operation into a "make it so": "ensure that the recommendation row for this run exists, with this content." The other is to keep the insert and give the store a way to recognize the repeat. In practice they are the same mechanism, and its central piece is a key.

Before AI. A payment service has met this problem for decades: a client times out on "charge 40 dollars" and must not charge twice. @Transactional in Spring does not solve it: the transaction committed, and the commit is the problem. The classic answers are a unique business key with a database constraint, or a request id stored with the result. An agent faces the same problem more often, for three reasons. Its runs are long, so a crash in the middle is more likely. Its steps are retried by more layers (the loop, the queue, the resume). And its writes reach other people: a duplicate recommendation is work for an underwriter, not a line in a log.

3. The idempotency key: one name for one intended write#

The caller attaches a name to the write, the same name on every attempt. The store remembers the names it has seen together with the result. When a name arrives that it has seen, it does not write again; it returns the stored result. That name is the idempotency key.

Four questions decide whether a key works: who makes it, what it is made of, where it is stored, and for how long.

Who makes it: the caller, before the first attempt. Only the caller knows that attempt 2 is a repeat of attempt 1. A key the store generated would be different on each attempt, because the store cannot tell them apart. The Amazon Builders' Library article on idempotent APIs makes the same point: the caller supplies a unique request identifier, because the service cannot tell from the parameters alone whether two identical requests were meant as one or as two.

What it is made of: what stays the same across every repeat of the same intention. In this example that is the run and the step: <run_id>:<step id>, for instance run-7f3a:record_recommendation. It contains no attempt number, no timestamp and no random value. This is idempotency.py:make_key, verbatim:

def make_key(run_id: str, step_id: str) -> str:
    """The idempotency key of one intended write: `<run_id>:<step id>`.

    No attempt number, no timestamp, no random value. Any of those would make
    a retry look like a new write.
    """
    return f"{run_id}:{step_id}"

Why the run and not the submission? Because a person may legitimately ask for a fresh run of the same submission, for example after the broker corrects the roof age. That is a new intention, it gets a new run_id, and it should produce a new recommendation. A key built from submission_id alone would wrongly swallow it. The run_id is the right width: one run, one intended write per step.

Where it is stored: with the row, in the same commit. If the key is saved in one place and the row in another, a crash between the two leaves a row without a key (the next retry writes a second row) or a key without a row (the next retry is told "done" and nothing exists). The Builders' Library article states the same requirement: recording the token and making the change succeed or fail together.

How long it is kept: longer than any repeat can arrive. A key that expires while a repeat is still possible protects nothing. In this example the slowest repeat is a resume after the queue's visibility timeout of 30000 ms, so keys must outlive that by a wide margin. A real store keeps keys for days and states the period.

This is the store side, idempotency.py:IdempotencyStore.execute, verbatim (shown without the indentation of its class). It runs the write once per key, returns the stored result to a repeat, and refuses a key that comes back with a different payload (section 5):

def execute(self, key: str, payload_fingerprint: str, fn: Callable[[], dict[str, Any]]) -> dict[str, Any]:
    """Run `fn` once per key.

    First time: run `fn`, store its result under the key, return it.
    Same key, same fingerprint: return the stored result with `replayed: true`.
    Same key, different fingerprint: raise KeyConflict.
    """
    seen = self.table.get(key)
    if seen is not None:
        if seen["fingerprint"] != payload_fingerprint:
            raise KeyConflict(f"key {key} was first used with another payload")
        return {**seen["result"], "replayed": True}
    result = fn()
    self.table[key] = {"fingerprint": payload_fingerprint, "result": result}
    return {**result, "replayed": False}

A repeat receives the stored row, the answer the first attempt would have received, with replayed: true: not an error, not "already exists". The Builders' Library article argues for exactly this: a response with the same meaning as the original success, so the caller knows its own request succeeded.

In store.py:Store.write the call sits between the two injected faults, so that in UW-R06 the row and its key are committed before the acknowledgement is lost. The docstring says it: "fault before the commit, fencing check, then the insert together with its idempotency key, then the fault after the commit."

Under the hood. In the example "the same commit" holds because both are plain Python writes in one function (the README says so). In a relational database it means one transaction and a unique index on the key column; the folded note at the end covers the race that index prevents.

4. Why the attempt number or a timestamp in the key breaks it#

The most common mistake with idempotency keys is to make them too unique. A developer wants keys that never collide and adds the attempt number, or the current time, or a fresh UUID per call. Each of these changes between attempts, and a key that changes between attempts names attempts, not the intended write. The store then sees three different names and writes three rows.

The example's lab function idempotency.py:simulate_writes makes one intended write fail twice with a lost acknowledgement, then succeed, under each key design. The command below prints the rows left and the keys each attempt sent. What a successful run proves: with the run_id:step key the table ends with one row; with a key that changes per attempt it ends with three. It does not prove anything about a real database.

cd examples/uw-reliability
/opt/homebrew/bin/python3 -c "
from idempotency import simulate_writes
for use_key, design in ((False, 'run_step'), (True, 'run_step'), (True, 'includes_attempt'), (True, 'random_per_call')):
    r = simulate_writes('ack_lost', use_key, design, 2)
    print(design if use_key else 'no key', '->', r['rows'], 'rows; keys sent:', [a['key'] for a in r['attempts']])
"
no key -> 3 rows; keys sent: [None, None, None]
run_step -> 1 rows; keys sent: ['run-7f3a:record_recommendation', 'run-7f3a:record_recommendation', 'run-7f3a:record_recommendation']
includes_attempt -> 3 rows; keys sent: ['run-7f3a:record_recommendation:1', 'run-7f3a:record_recommendation:2', 'run-7f3a:record_recommendation:3']
random_per_call -> 3 rows; keys sent: ['k-02ff152c', 'k-0fdc7f12', 'k-fa169e5e']

A key with the attempt number protects exactly as much as no key at all. A timestamp taken at each attempt behaves like the random value: two attempts get two keys. A timestamp taken once and reused would work, but then it is a stored name, and the run_id is a better one: it already exists, and a resumed process can find it.

There is a second, quieter reason to keep attempt_number out. It starts at 1 again in a resumed process, because a new process starts a new retry loop. In UW-R08 the resumed worker's write is attempt 1, exactly like the crashed worker's write. A key built from the attempt number would collide across processes for the wrong reason, and only by luck. The run_id survives the crash because it lives in the store, not in the process.

Key designChanges between attempts?Rows after a lost acknowledgement (2 retries)Verdict
nonenot applicable3The store cannot tell a retry from a new write
run_id:step (this site's design)No1Correct: one name per intended write
run_id:step:attempt_numberYes3Names attempts, not the write
A random value or a timestamp per callYes3Same failure
submission_id:stepNo1, but also blocks a fresh run a person asked forToo broad

5. The same key with a different payload#

A key says "this is the same write as before". What if the payload is not the same? Suppose a bug makes the retry send request_info where the first attempt sent escalate, with the same key. Returning the stored escalate row would hide the bug. Writing the request_info row would break the promise of the key. The store does neither: it refuses.

IdempotencyStore.execute keeps a short fingerprint (a hash) of the payload with each key. A repeat with the same key and a different fingerprint raises KeyConflict, which errors.classify treats as permanent: no retry, look at it. The lab shows the result: one row, and a conflict to investigate. The Builders' Library article describes the same rule for its client tokens (a mismatch is a validation error), and the IETF Internet-Draft for an Idempotency-Key HTTP header assigns status 422 to a key reused with a different payload.

Common confusion. "Same key, so return the stored result" is right only for the same payload. A key without a payload check turns a bug in the caller into silently wrong data.

6. The example, careful and naive#

Here is the careful run of UW-R06, from run_scenario.py --scenario UW-R06. A successful run proves that the retry of step 6 carried the same key and the store answered it with the stored row. It does not prove that a real store commits the key and the row together; the example's store does, by construction.

cd examples/uw-reliability
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R06
   1986  worker-A  record_recommendation    1  attempt failed: AckLost: store.record_recommendation: committed, but the acknowledgement was lost
   1986  worker-A  record_recommendation    1  backoff: 52 ms
   2050  worker-A  record_recommendation    2  attempt ok
   2050  worker-A  record_recommendation    -  replay: same key: the stored row was returned, nothing was written
   2050  worker-A  record_recommendation    -  step done: checkpoint saved
end state:
  run run-4708  state waiting_for_human  resume_count 0  ended at 2062 ms (deadline 120000 ms)
  recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
  REC-0001  pending_review  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
            run run-4708  route primary  degraded false  fencing_token 41  key run-4708:record_recommendation
result: 7 of 7 invariants hold

(These are the lines about step 6 and the end state.) Both workflows saw the same error and both retried. The difference is the line replay: same key. The key is what lets the store answer the question the caller cannot.

UW-R08 shows the same key working across a crash. The first worker writes the row at 1986 ms and dies before its checkpoint. The queue delivers the message again at 30000 ms, the second worker finds the run by its message_id, resumes from the checkpoint after step 5, and executes step 6 again with the key run-3708:record_recommendation:

   1986  worker-A  record_recommendation    1  attempt ok
   1986  worker-A  record_recommendation    -  PROCESS CRASH: after the step, before checkpoint
   1986  worker-A  -                        -  process is gone: no acknowledgement; visible again after 30000 ms
  30000  worker-B  -                        -  message received: msg-02a7, receive 2
  30000  worker-B  -                        -  run found: by message_id; resume_count 1
  30000  worker-B  -                        -  lease granted: fencing token 42
  30000  worker-B  draft_recommendation     -  resume from checkpoint: steps 1 to 5 are not run again
  30012  worker-B  record_recommendation    1  attempt ok
  30012  worker-B  record_recommendation    -  replay: same key: the stored row was returned, nothing was written

The naive workflow creates a new run here, writes a second row and leaves the first run in running: INV-1 and INV-4 fail. In the careful workflow three things cooperate. The lookup by message_id finds the same run, so the run_id is the same. The checkpoint skips steps 1 to 5. And the key, built from that run_id, turns the repeated step 6 into a replay. Remove any one and the outcome changes. That is the meaning of "a retry is not a recovery": the retry loop played no part here at all. How the checkpoint and the resume work is the subject of Checkpoints.

The lab lets you place the failure yourself: before the commit, after the commit with the acknowledgement lost, or as a crash before the checkpoint; with or without a key; and with the key built in each of the ways from section 4. It shows each attempt and the rows left, against the expected one.

The same results without the lab (from simulate_writes, two retries after the failure):

Failure pointNo keyrun_id:stepWith attempt_numberRandom per callRight key, other payload
Before the commit (UW-R07)1 row1 row1 row1 row1 row
After the commit, acknowledgement lost (UW-R06)3 rows1 row, 2 replays3 rows3 rows1 row, conflict
Crash before the checkpoint (UW-R08)3 rows1 row, 2 replays3 rows3 rows1 row, conflict

The first row is the trap: if tests inject failures only before the commit, every design passes, even no key. A test for idempotency must inject the failure after the commit.

7. Deduplication, and the honest meaning of "exactly once"#

What the store does with a repeated key has a general name. Deduplication is recognizing that a request, a message or a write has been seen before and not acting on it a second time. The store deduplicates writes by key. In the next chapter a consumer deduplicates messages by message_id, which is the same idea one level up.

A duplicate write is what deduplication prevents: the same intended change stored more than once. In this example it is two pending_review rows for one submission, which INV-1 forbids.

People often ask for a write to happen "exactly once". Across a network that cannot be promised for the attempts. A caller that has lost an acknowledgement must try again or give up, and if it tries again the store receives the write twice. What can be promised is the result: however many attempts arrive, the effect is applied once. This site calls that an exactly-once effect: at-least-once attempts plus deduplication. The phrase is narrower than "exactly once" on purpose, and the next chapter explains why exactly-once delivery is not on offer either.

What this is not. How to design the write tool itself, its schema and its arguments, is the subject of https://tools.sammuthu.com. This site owns one property of that tool: it must be safe to call twice.

8. When the change must also reach another system: the outbox#

The key works because the row and the key live in one store and commit together. Some steps change two systems. Imagine a variant of step 7 in which the agent stores the recommendation and must also publish an event, "a recommendation is ready", to a separate message broker that the underwriter cockpit listens to. The database and the broker cannot commit together.

Both obvious orders fail. Publish first: if the commit then fails, the cockpit announces a recommendation that does not exist. Commit first: if the process crashes before publishing, nobody is ever told, and there is no process left to retry.

The outbox is a design that removes the gap. In the same transaction as the business row, the service also inserts the event into an outbox table in the same database. Either both rows commit or neither does. A separate relay process reads the outbox, publishes each event to the broker, and marks it sent. The microservices.io description of the transactional outbox pattern gives this design and names its cost: the relay can publish an event more than once (for example if it crashes after publishing and before marking the event sent), so whoever consumes the event must deduplicate.

Diagram: The outbox · one transaction for the row and the event, a relay for the rest

Read left to right. The service writes the business row and the outgoing event in one transaction of its own database. A separate relay later publishes the event to the broker. The relay can publish an event twice, so the consumer deduplicates on the event id. This is a design; the example does not implement it.

  1. 1 · One transaction
    1. Recommendation row: The business change, with its idempotency key and a unique index on that key.
    2. Outbox row: The outgoing event, inserted in the same transaction. Both rows commit, or neither does.
  2. 2 · Later, separately
    1. Relay: A process that reads unsent outbox rows, publishes each to the broker and marks it sent. A crash between the two steps publishes it again.
  3. 3 · The other system
    1. Message broker: The broker delivers the event to its consumers, at least once.
    2. Consumer deduplicates: The consumer records the event ids it has handled, in the same commit as its own effect, and skips a repeat.

The example does not implement an outbox: its step 7 is a second write with its own idempotency key, run-7f3a:open_review_task, into the same simulated store. The SQL below shows the shape of the design. Documented shape; not run here.

-- documented shape; not run here
BEGIN;
INSERT INTO recommendations (recommendation_id, run_id, submission_id, idempotency_key, body, status)
VALUES ('REC-0001', 'run-7f3a', 'SUB-1004', 'run-7f3a:record_recommendation', '{"recommendation": "escalate"}', 'pending_review')
ON CONFLICT (idempotency_key) DO NOTHING;
INSERT INTO outbox (event_id, topic, payload, sent)
VALUES ('run-7f3a:recommendation_ready', 'recommendation_ready', '{"recommendation_id": "REC-0001"}', false)
ON CONFLICT (event_id) DO NOTHING;
COMMIT;
-- the relay, separately: read unsent rows, publish each, then mark it sent

Notice that the event id is built like the idempotency key, from the run_id and a step name. A replayed step inserts nothing new, and the consumer can deduplicate on the event id.

ApproachWhat happens after a crash in the middleDuplicates possible?
Publish, then commitAn event for a row that may not existYes, and phantom events
Commit, then publishA row nobody is told aboutNo, but lost events
Outbox in the same transaction, relay publishesThe relay publishes on its next passYes, from the relay; consumers deduplicate

9. How idempotency goes wrong even when it is there#

Section 4 showed the key that changes between attempts and the key that is too broad. Four more ways a key exists and still fails:

Stored apart from the change. A key in a cache and a row in the database can disagree after a crash: a row without a key, or a key without a row.

Expired too early. A repeat that arrives after the key was deleted is a new write. Keys must outlive the slowest repeat path: retries, redelivery, resume.

No payload check. The same key with a different payload returns stale data or overwrites good data (section 5).

Read, then insert, without a constraint. Two concurrent attempts can both find no key and both insert. A unique index closes the gap; the folded note below explains.

The same idea for a coding agent#

A coding agent finishes a change, runs git push, and the command times out. Did the push reach the remote? Pushing the same commit to the same branch again is safe, because it is a "make the branch point here" operation, idempotent in effect. The next step is the dangerous one. "Open a pull request" is a create, and an agent that retries it, or that crashes and is resumed, can open two pull requests for one task, each starting its own CI run and asking for review twice. The fix has the same shape as run_id:step: derive the branch name from the task id, and before creating a pull request, look for an open one from that branch and return it if it exists. The harness that runs such an agent is https://harness.sammuthu.com; the pull request and CI flow is https://diff-to-deploy.sammuthu.com.

How frameworks and standards package this#

There is a standards effort for exactly this mechanism: the IETF Internet-Draft "The Idempotency-Key HTTP Header Field" (an Internet-Draft, not yet an RFC). The Builders' Library article describes a client token on Amazon's own APIs for the same purpose. Workflow engines such as Temporal and AWS Step Functions are compared on Frameworks, including what each expects from the steps it may run again, checked against its own documentation. None of them can choose your key for you: what counts as "the same intended write" is a decision about your workflow.

The race between two attempts, and the in-progress stateWhy a unique index, and what a repeat should see while the first attempt is still running

IdempotencyStore.execute checks for the key, then runs the write, then stores the key. In the example that is safe because nothing runs concurrently. In a real service two attempts can overlap: the first is still writing when the retry arrives, typically because the caller's timeout was shorter than the write. Both look for the key, both find nothing, both insert.

Two defences, usually together. A unique index on the key column makes the database refuse the second insert, so at most one row can exist whatever the timing; the losing attempt then reads and returns the winner's row. And an explicit in-progress state: the first attempt inserts the key with status "in progress" before doing the work, and a repeat that finds that status is told to come back later instead of being given a result that does not exist yet. The IETF Internet-Draft assigns status 409 to that case.

The in-progress record needs its own expiry. If the first attempt crashes after claiming the key, the key must not stay "in progress" forever, or the write can never happen. That expiry is a lease, with the same hazards: Concurrency covers leases and fencing tokens.

How long to keep keys, and what a late repeat should getRetention is derived from the slowest path a repeat can take

List every path by which the same intended write can arrive again, and its worst-case delay. In this example: the retry loop (seconds), queue redelivery after the visibility timeout (30000 ms), a resume after a crash (the same), and a message that sits in the dead-letter queue until a person redrives it after a fix (hours or days). Retention must cover the slowest path that is still allowed to re-execute the write. If a redrive from the dead-letter queue can replay step 6, keys must outlive the dead-letter queue's retention, or the redrive must start a fresh run on purpose.

What a late repeat receives also matters. The Builders' Library article describes returning a response consistent with the original request even when the resource has since changed state. For this example that would mean a repeat after the underwriter has already decided still receives the stored recommendation row, not an error, and certainly not a fresh write.

Storage is the cost: one key and one fingerprint per write, kept for the retention period. For an agent that writes a handful of rows per run that is small; for a high-volume event stream it is a design input.

When the other side offers no keyRead before write, reconcile, or route through something that deduplicates

Many external APIs an agent calls accept no idempotency key: a legacy policy system, an email gateway, a ticketing tool. Three options, from strongest to weakest.

Make the operation a "make it so" by reading first: search for the object by a natural identifier you control (a reference field set to run_id:step), and create it only if it is absent. This has the race described above, so it needs a lease or a lock if two workers can act at once.

Route the call through something you own that deduplicates: an outbox row with an id built like the key, and a relay that records which ids it has delivered. The external call can still happen twice if the relay crashes between the call and the record, so the remaining window is small but not zero.

Accept the duplicate and reconcile: detect duplicates afterwards and compensate (withdraw one, apologize for one email). Checkpoints covers compensation; some effects, like an email already read, cannot be undone, which is why the stronger options come first.

Checklist#

CheckWhat to look for in your own system
Every write that can be retried carries a keyIncluding writes repeated by a queue redelivery or a resume, not only by the retry loop
The caller makes the key, before the first attemptNever generated by the server; never regenerated per attempt
The key names the intended writeBuilt from stable ids such as run_id and the step id; no attempt number, timestamp or random value
The key is stored with the changeOne transaction; a unique index on the key column
A repeat gets the original resultThe stored row, marked as a replay; not an error and not "already exists"
A payload mismatch is refusedA fingerprint stored with the key; a conflict is permanent and investigated
Keys outlive every repeat pathRetries, redelivery, resume, dead-letter redrive
Changes to two systems use an outboxAnd the consumer deduplicates on the event id
Tests inject failures after the commitA test that fails only before the commit passes with no key at all

In one breath#

When a write times out, the caller cannot know whether it happened, so the store has to decide, and the caller has to give it a name for the write. That name is the idempotency key: made by the caller, the same on every attempt, built from the run and the step and never from the attempt number, stored in the same commit as the row, kept longer than any repeat can arrive. A repeat then receives the stored result instead of writing again. Attempts may arrive many times; the effect happens once.

The careful workflow now survives a lost acknowledgement and a crash after the write with exactly one recommendation row. But both of those repeats were started by something outside the retry loop: in UW-R08, the queue delivered the message again after the worker died. Where messages come from, when they are delivered twice, and what happens to one that can never succeed is the subject of Queues.