All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Mechanisms · 5

Queues: delivery semantics, dead letters and poison messages

A queue holds work until a worker finishes it. At-most-once and at-least-once delivery, why a message can arrive twice, and where a message goes when it can never succeed.

What you will understand
Hand it over safely
Why work goes through a queue, and how an acknowledgement and a visibility timeout make sure a crashed worker's message comes back.
Twice, or maybe never
At-least-once and at-most-once delivery, and why exactly-once delivery is not on offer.
When a message cannot succeed
The inbox that makes a second delivery harmless, and the dead-letter queue that stops a poison message from looping.

A broker's email asks for a quote on SUB-1004. The intake step turns it into a submission and puts a message on the queue: msg-32f6. A worker takes it, and the underwriting agent runs all eight steps. At 1998 ms the run ends where it should: one recommendation to escalate, waiting in the underwriter cockpit for a person. The model recommends; a person decides. The worker tells the queue "done".

In scenario UW-R10 that last message, the "done", is lost. The queue never hears it. So, as it was built to do, it delivers msg-32f6 again, and a second worker picks it up:

   1998  worker-A  -                        -  acknowledgement lost: the queue will deliver the message again
   1998  worker-B  -                        -  message received: msg-32f6, receive 2
   1998  worker-B  -                        -  run created: trace_id 1fb3e47c4d94b230e1c4af3417e2aa5a

That is the naive workflow. Worker B does not know that worker A finished, so it starts a new run and does all the work again. Two seconds later the underwriter has two recommendations and two review tasks for one broker request:

  run run-4114  state waiting_for_human  resume_count 0  ended at 1998 ms (deadline 120000 ms)
  run run-0ad4  state waiting_for_human  resume_count 0  ended at 3996 ms (deadline 121998 ms)
  recommendation rows 2 (pending_review 2, withdrawn 0) · review tasks 2 · decisions 0 · dead letters 0

The queue did nothing wrong. Redelivering a message whose acknowledgement never arrived is its most important feature: it is what brings back the work of a worker that crashed. The problem is on the receiving side, and the previous chapter already met its shape. The worker cannot tell the queue "I did it" and do the work in one atomic step, so one of the two can happen without the other. This chapter is about living with that: how a queue delivers, what it promises and what it cannot, and how a consumer turns "possibly twice" into "once". One note before more results: the agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

1. Why the work goes through a queue at all#

The intake step could call the agent directly. It does not, for three reasons that all come down to failure.

A queue is a durable list of work items between a producer and the workers that consume them. The producer adds a message and moves on. A worker takes a message when it has capacity. In this example the message carries the submission, and the queue holds it until a worker has finished with it.

The first reason is that the producer must not wait for the agent. A run takes seconds when all goes well and minutes when a provider is slow. The intake step should accept the next email, not hold a connection open. The second is load: on a busy morning more submissions arrive than the workers can process at once, and the queue absorbs the difference instead of refusing work. The third, and the one this chapter is about, is crashes. If a worker dies in the middle of a run, the request must not die with it. The queue still holds the message, and another worker will receive it.

Without a queueWith a queue
The intake step waits for the whole runThe intake step adds a message and is done
A burst of submissions is refused or times outThe burst waits in the queue; workers drain it
A worker crash loses the requestThe message comes back to another worker
Duplicates are rare, losses are commonLosses are rare, duplicates are possible

The last row is the trade. A queue that never loses work must sometimes deliver it twice. The rest of the chapter follows from that one line.

What this is not. Deciding which step of a workflow runs next, and routing work between agents, is orchestration: https://orchestrate.sammuthu.com. Messages exchanged between agents are https://a2a.sammuthu.com. This chapter is about one queue in front of one kind of worker, and what happens when a delivery fails.

2. Receive, acknowledge, or come back: the visibility timeout#

A queue that deletes a message the moment a worker takes it would lose the message if the worker crashed. A queue that keeps it visible would hand it to every worker at once. The common design sits in between.

When a worker receives a message, the queue hides it from other workers for a fixed time. That time is the visibility timeout: 30000 ms in this example. If the worker finishes, it sends an acknowledgement, a short message to the queue that says "this one is done, delete it". If no acknowledgement arrives before the visibility timeout ends, the queue assumes the worker is gone and makes the message visible again. The next worker receives it, and the queue counts how many times it has been received.

This is queue_sim.py:Queue.receive and Queue.ack, verbatim. receive hides the message and counts the receive; ack deletes it, unless the acknowledgement is lost, in which case the message shows up again:

def receive(self) -> dict[str, Any] | None:
    """The next visible message, hidden from now on for the visibility timeout.

    A message that was already received `max_receives` times is moved to the
    dead-letter queue instead of being delivered again.
    """
    for message in list(self.messages):
        if message["visible_at_ms"] > self.clock.now():
            continue
        if self.max_receives is not None and message["receive_count"] >= self.max_receives:
            self.messages.remove(message)
            self.dead_letters.append(message)
            continue
        message["receive_count"] += 1
        message["visible_at_ms"] = self.clock.now() + self.visibility_timeout_ms
        if self.mode == "at_most_once":
            self.messages.remove(message)
        return {"message_id": message["message_id"], "body": message["body"], "receive_count": message["receive_count"]}
    return None

def ack(self, message_id: str, lost: bool = False) -> None:
    """Delete the message. With `lost`, the acknowledgement never arrives and the message shows up again."""
    for message in self.messages:
        if message["message_id"] == message_id:
            if lost:
                message["visible_at_ms"] = self.clock.now()
            else:
                self.messages.remove(message)
            return

(Both are methods of the class Queue; they are shown without the class indentation.) The anchor scenario of the site, UW-R15, shows the visibility timeout at work: worker A crashes at 6946 ms without acknowledging, and the message becomes visible again at exactly 30000 ms, when worker B receives it.

Diagram: The life of msg-51c2 in UW-R15 · three deliveries, one run

Time runs down the page. Worker A receives the message and crashes after the write without acknowledging, so the message comes back when its visibility timeout ends. Worker B finds the run by message_id and finishes it, but its acknowledgement is lost. Worker C receives the message a third time, sees that the run has ended and acknowledges. Times come from the simulated clock.

  1. Queue to Worker A: Receive 1, hidden for 30000 ms. The message is delivered and hidden from other workers until 30000 ms, unless an acknowledgement arrives first.
  2. Worker A: Steps 1 to 6, then crash before the checkpoint. The recommendation row is written at 6946 ms; the process dies before it can save the checkpoint or acknowledge.
  3. Queue: No acknowledgement: visible again at 30000 ms. The queue cannot tell a crash from a slow worker. When the visibility timeout ends it offers the message again.
  4. Queue to Worker B: Receive 2. The same message, the same message_id. At-least-once delivery means this second delivery is normal, not an error.
  5. Worker B: Run found by message_id; resume; write replayed. The inbox lookup finds run-7f3a still running. The checkpoint skips steps 1 to 5; the idempotency key turns step 6 into a replay; step 7 opens the task.
  6. Worker B to Queue: Acknowledgement, lost. The injected duplicate fault: the acknowledgement never arrives, so the queue keeps the message.
  7. Queue to Worker C: Receive 3. A third delivery of the same message.
  8. Worker C to Queue: Run already ended: acknowledge. The run is in waiting_for_human, so there is nothing to do. The acknowledgement arrives and the queue deletes the message.

The diagram follows msg-51c2 through UW-R15. Notice that the queue never learns why a worker went quiet. A crash, a network partition and a worker that is still busy after 30 seconds look the same from the queue's side: no acknowledgement. That last case matters, and section 7 returns to it.

3. At-least-once, at-most-once, and why not exactly once#

When does the worker acknowledge? The choice decides what the queue can promise.

Acknowledge after the work. If the worker crashes before the acknowledgement, the message comes back and the work is done again. Nothing is lost, but the work may happen twice. This is at-least-once delivery: every message is processed one or more times. It is the default of the example, and of most systems that must not lose work.

Acknowledge before the work (or, the same thing, let the queue delete the message on receive). If the worker crashes after acknowledging, the message is gone and the work never happens. Nothing is duplicated, but work can be lost. This is at-most-once delivery: every message is processed zero or one times. It suits work whose loss is cheaper than its repetition, such as a metric sample. A quote request is not that kind of work.

What everyone wants is the third option, exactly-once delivery: every message processed one time, never zero, never two. For a worker whose work changes something outside the queue, that cannot be built from the queue alone. The reason is the lost acknowledgement again. The worker does its work, then acknowledges. Between the two there is a moment where the work is done and the queue does not know it. A crash in that moment, or a lost acknowledgement, forces a choice: redeliver (and risk doing it twice) or not (and risk not doing it at all). No ordering of "work" and "acknowledge" removes that moment, because they happen in two different systems.

Kafka's design documentation describes the same two choices in its own terms: a consumer that saves its position before processing gets at-most-once, one that saves it after processing gets at-least-once. For results written to an outside system, it points to the same way out this site uses: store the position together with the output, or make the output idempotent.

What can be achieved is an exactly-once effect: at-least-once delivery, plus a consumer that recognizes a message it has already handled. The Idempotency chapter defined that phrase. The next section builds the consumer.

DeliveryAcknowledgeAfter a crash at the wrong momentGood for
At-most-onceBefore the work, or delete on receiveThe work is lostWork that is cheap to lose
At-least-onceAfter the workThe work is done againWork that must not be lost; the consumer must deduplicate
Exactly-once deliveryNot available for outside side effects
Exactly-once effectAfter the work, plus an inboxThe repeat is recognized and skippedQuote requests, payments, anything with a side effect
Common confusion. "Our queue offers exactly-once delivery." Read what the documentation promises carefully. A promise about delivery inside the messaging system (no duplicate copies in the queue, or an atomic read-process-write between topics of the same system) is not a promise that your database write or your email happens once. Feature claims about specific products are on Frameworks.

4. The inbox: one run per message#

The consumer needs to remember which messages it has already handled, and to remember it in a way that survives its own crash. That memory is the inbox: a record, kept in the consumer's own store, of the message ids it has accepted, written together with the work they caused.

In this example the inbox is the runs table itself. Every run row carries the message_id that created it, and a unique index allows only one run per message. When a message arrives, the worker first looks up the run by message_id. This is workflow.py:Worker._find_or_create_run, verbatim:

def _find_or_create_run(self, message: dict[str, Any]) -> dict[str, Any]:
    """Look the run up by `message_id`, so a redelivered message finds the same run."""
    env = self.env
    run = env.store.find_run(message["message_id"]) if env.flags["dedup_inbox"] else None
    if run is not None:
        if run["state"] == "running":
            run["resume_count"] += 1
            env.log(self.name, "run found", f"by message_id; resume_count {run['resume_count']}", run["run_id"])
        return run
    submission = message["body"].get("submission") or {}
    run = {"run_id": env.next_run_id(), "message_id": message["message_id"],
           "submission_id": submission.get("submission_id", "unknown"), "state": "running",
           "trace_id": new_trace_id(env.ids), "created_ms": env.clock.now(),
           "deadline_ms": env.clock.now() + RUN_DEADLINE_MS, "finished_ms": None, "resume_count": 0, "last_step": None,
           "attempts": {}, "first_attempts": 0, "retries": 0, "cost_units": 0, "evidence_ids": [], "recommendation_id": None}
    env.log(self.name, "run created", f"trace_id {run['trace_id']}", run["run_id"])
    return env.store.insert_run(run)

(A method of Worker, shown without the class indentation.)

Diagram: The inbox · one lookup by message_id decides what a delivery does

Read left to right. Every delivery starts with the same lookup. A new message creates a run, and a unique index on message_id refuses a second one. A message whose run has ended is acknowledged with nothing done. A message whose run is still running means a worker crashed, so this worker resumes the run from its checkpoint.

  1. 1 · A delivery
    1. Message arrives: A worker receives a message. It has not yet acknowledged it, so if it dies now the message comes back.
  2. 2 · The inbox lookup
    1. Find the run: Worker._find_or_create_run looks in the runs table for a run created by this message_id.
  3. 3 · Three outcomes
    1. No run yet: Insert a new run. The unique index on message_id refuses a second insert, so two racing workers cannot both create a run.
    2. Run has ended: The state is a terminal state, so the work is done. The worker acknowledges and stops.
    3. Run still running: A previous worker crashed or stalled. This worker resumes the same run, with resume_count one higher.
  4. 4 · The effect
    1. One run, one row: Because every delivery of one message reaches the same run, and the run's writes carry idempotency keys, the effect happens once.

Three outcomes follow from one lookup. No run yet: create one, and the unique index on message_id refuses a second insert if two workers race. A run that already ended: the worker in Worker.handle sees a state other than running, does nothing and acknowledges. A run still running: a previous worker crashed, so this one resumes it from its checkpoint, with resume_count one higher.

This lookup is what keeps the identifiers of the site apart. The message_id names the request on the queue and never changes across redeliveries. The run_id is found by message_id, so a redelivered message finds the same run. And the idempotency keys are built from the run_id, so the writes of a resumed run are replays. Each identifier has one job, and the chain only holds because each one is derived from the one before.

Here is the careful run of UW-R10. A successful run proves that the second delivery found the finished run and did nothing; it does not prove anything about a real queue's redelivery behaviour.

cd examples/uw-reliability
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R10
   1998  worker-A  await_underwriter        -  run ended: waiting_for_human
   1998  worker-A  await_underwriter        -  step done: checkpoint saved
   1998  worker-A  -                        -  acknowledgement lost: the queue will deliver the message again
   1998  worker-B  -                        -  message received: msg-32f6, receive 2
   1998  worker-B  -                        -  duplicate delivery: run-4114 already ended in waiting_for_human: nothing to do
   1998  worker-B  -                        -  acknowledged
end state:
  run run-4114  state waiting_for_human  resume_count 0  ended at 1998 ms (deadline 120000 ms)
  recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
result: 7 of 7 invariants hold

(The last lines of the timeline and the end state.) Two deliveries, one run, one row, one task.

Before AI. A Spring service with a Kafka listener or a JMS consumer meets the same choices: commit the offset or acknowledge after processing, accept redelivery, and keep a processed-message table so a redelivery does no harm. Nothing on this page is new to such a team. Two things are different for an agent. Processing takes minutes, not milliseconds, so the window in which a crash or a lost acknowledgement causes a redelivery is far wider, and the visibility timeout must be chosen with long runs in mind. And a repeated run is expensive and visible: it calls a paid model again and puts a second task in front of a person.

The lab lets you choose the delivery mode, when the worker fails, whether it has an inbox, and how many receives are allowed before the dead-letter queue. It shows each delivery, the effects (expected: 1) and whether the message ended in the dead-letter queue.

The same results without the lab (from queue_sim.py:simulate_queue, maximum 5 receives):

Delivery modeWhen the worker failsInboxDeliveriesEffectsDead-lettered
At-least-onceNeveroff or on11no
At-least-onceBefore the work, first deliveryoff or on21no
At-least-onceAfter the work, before the acknowledgementoff22no
At-least-onceAfter the work, before the acknowledgementon21no
At-least-onceEvery delivery (a poison message)off or on50yes
At-most-onceBefore the work, first deliveryoff or on10: the work is lostno
At-most-onceAfter the work, before the acknowledgementoff or on11no

Read the third and fourth rows together: the inbox is the only difference, and it is the difference between one effect and two. And note the at-most-once row with 0 effects: no setting of the inbox can bring back a message the queue has already deleted.

5. The message that can never succeed: poison and the dead-letter queue#

At-least-once delivery has a dark side. A message that fails every time comes back every time.

A poison message is a message that will fail on every delivery, because the fault is in the message, not in the worker or a dependency. In scenario UW-R11 the submission has no wind_zone. Step 1 validates it and raises InvalidSubmission, a permanent error: nothing about it will change by trying again. The Retries chapter made sure such an error is not retried inside the step. But the worker cannot acknowledge the message as done, because nothing was done, so it hands the message back. In this example a handed-back message becomes visible again after 1000 ms.

Without a limit, that loop never ends. The naive workflow shows it: twelve deliveries, twelve workers, twelve runs left in running, until the simulation stops it.

  11000  worker-L  load_submission          -  poison message: field wind_zone is missing or not a str; not acknowledged
  11000  worker-L  -                        -  not acknowledged: visible again after 1000 ms
  11000  queue     -                        -  SIMULATION STOPPED: after 12 deliveries the message is still not done

In a real system there is no simulation to stop it. The message takes a worker's time on every delivery, forever, and if several such messages arrive they can occupy every worker while valid submissions wait behind them.

The answer is a limit on receives and a place to put the messages that reach it. A dead-letter queue is a separate queue that holds messages a consumer gave up on, so that they stop circulating and a person can look at them. In this example the limit is 5 receives: Queue.receive moves a message that has already been received 5 times to the dead-letter queue instead of delivering it. The careful run of UW-R11:

cd examples/uw-reliability
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R11
   4000  worker-E  -                        -  message received: msg-7dfd, receive 5
   4000  worker-E  -                        -  run found: by message_id; resume_count 4
   4000  worker-E  -                        -  lease granted: fencing token 45
   4000  worker-E  load_submission          -  poison message: field wind_zone is missing or not a str; not acknowledged
   4000  worker-E  -                        -  not acknowledged: visible again after 1000 ms
   5000  queue     -                        -  dead-lettered: msg-7dfd after 5 receives
end state:
  run run-d8ff  state dead_lettered  resume_count 4  ended at 5000 ms (deadline 120000 ms)
  recommendation rows 0 (pending_review 0, withdrawn 0) · review tasks 0 · decisions 0 · dead letters 1
result: 7 of 7 invariants hold

(The fifth delivery, the dead-lettering and the end state.) Five receives, one run (the inbox found the same run each time), zero rows, and a named terminal state, dead_lettered, inside the deadline. Zero rows is the acceptable outcome here: a submission that cannot be read must not produce a recommendation.

A dead-letter queue is not a bin. Every message in it is a request from a broker that nobody has answered, and usually a bug in the intake step. The Builders' Library article on queue backlogs describes alarming on any message in a dead-letter queue for exactly that reason, and reprocessing the messages after the fix.

ScenarioCareful workflowNaive workflow
UW-R102 deliveries, the second finds the finished run; 1 run, 1 row, 1 task2 runs, 2 rows, 2 tasks; breaks INV-1
UW-R115 receives, then the dead-letter queue at 5000 ms; 0 rows; dead_lettered12 deliveries, 12 runs left in running; breaks INV-4
Production pitfall. A maximum-receives limit cannot tell a poison message from a message that met five transient failures in a row, for example during a long provider outage when step 3 keeps giving up. Both end in the dead-letter queue. So the dead-letter queue needs a person and a redrive procedure, and the limit should be high enough that a normal outage does not fill it. A permanent error recognized in the message itself can go to the dead-letter queue at once; the folded note discusses that design.

6. Order, and why this workflow does not depend on it#

Unless a queue promises order, assume none, especially once redelivery and several workers are involved. Message 2 can be processed while message 1 waits for its visibility timeout.

This workflow is built so that order does not matter: one message per submission, and every message carries the whole submission. Order starts to matter when two messages describe changes to the same thing, for example when the broker sends a correction to SUB-1004 while the first request is still running. Then the consumer needs a rule: carry a version number in each message and ignore an older version, or hold a lease on the submission so that two runs cannot act on it at once. Leases, and the fencing token that protects against a worker that lost its lease, are on Concurrency.

7. When the visibility timeout is shorter than the work#

The queue cannot tell a dead worker from a slow one. If a run takes longer than the visibility timeout, the message becomes visible while the first worker is still working, and a second worker receives it. The Builders' Library article on queue backlogs warns about exactly this: the same message processed in parallel by two consumers.

In this example a careful run takes about 2 seconds without faults and the visibility timeout is 30000 ms, so the case does not arise from normal work. It arises from a stall. Scenario UW-R14 has a worker pause for 45000 ms, past both its visibility timeout and its lease, and a second worker take over. The inbox lets the second worker find the same run, and the fencing token stops the stalled worker's late write.

Three things keep this window small. Choose the visibility timeout longer than the slowest normal run, with room for its retries. Extend the timeout while working, if the queue allows it (a heartbeat). And assume it will happen anyway: the inbox, the idempotency keys and the fencing token are what make a parallel delivery harmless, not the timeout.

The same idea for a coding agent#

A CI system calls a webhook when a pipeline fails, and a coding agent listens and starts a run to fix the build. Suppose the sender retries when it does not get a quick success answer. The agent's endpoint is slow because it starts a run before answering, so the sender gives up waiting and delivers the event again, and two agent runs start fixing the same build, each pushing a branch and opening a pull request. The cure has the shape of the inbox: answer the webhook at once, put the event on a queue, and create at most one run per delivery id, looked up before anything else. The harness around such an agent is https://harness.sammuthu.com; the CI and pull request flow is https://diff-to-deploy.sammuthu.com.

How frameworks package this#

Apache Kafka, Amazon SQS and RabbitMQ make different choices about acknowledgement, redelivery, ordering and dead letters. One contrast is conceptual and worth holding before reading their documentation. A queue in the sense of this page deletes a message once it is acknowledged. Kafka keeps messages in a log for a retention period, and each consumer stores its own position in that log; "acknowledging" means moving the position forward, and a consumer can move it back to read again. The delivery choices are the same in both models. What each product promises, and how, is on Frameworks, checked against its own documentation.

A queue (this page's model)A log with consumer positions
After the consumer is doneThe message is deletedThe message stays; the consumer's position moves forward
A crashed consumerIts message becomes visible again after the visibility timeoutIt restarts from its last saved position
Reading againNot possible after deletionPossible: move the position back
DuplicatesAfter a lost acknowledgement or a timeoutAfter a crash between processing and saving the position
What the consumer still needsAn inboxAn inbox, or the position stored with the output
Dead-letter at once on a permanent error, or let the receive limit do itThe example's choice and the alternative

In UW-R11 the worker recognizes the problem on the first delivery: InvalidSubmission is a permanent error. Yet it hands the message back, and the queue redelivers it four more times before the receive limit moves it to the dead-letter queue. Each redelivery costs a worker a moment and, in this example, raises the run's resume_count, which reaches 4.

The alternative is to dead-letter explicitly: on a permanent error in the message itself, the worker moves the message to the dead-letter queue (or records it as dead and acknowledges it) on the first delivery. That saves four deliveries and makes the reason exact. The cost is that the worker must be right about "permanent". A validation rule deployed with a bug would then send every message straight to the dead-letter queue without a second chance, whereas the receive limit gives a fixed deployment a few deliveries to catch up.

A reasonable design uses both: explicit dead-lettering for errors that are certainly in the message (it does not parse, a required field is missing), the receive limit as the safety net for everything else. Either way, the run must reach a named terminal state, so that INV-4 holds and someone can see it.

Redriving the dead-letter queue without creating duplicatesA redrive is a redelivery that arrives days later

After the bug is fixed, the messages in the dead-letter queue are sent back to the main queue: a redrive. To the consumer, a redriven message is a redelivery with a very long delay. Everything that protects a redelivery must still be in place when it arrives.

The inbox must still hold the message_id, so that a message whose run did finish before it was dead-lettered (possible when the last acknowledgement was lost) does not start a second run. The idempotency keys must still be stored, so that a run that wrote its row before failing does not write it again. Both retention periods are therefore set by the dead-letter queue's retention, not by the visibility timeout.

There is also a choice of meaning. A submission that sat in the dead-letter queue for three days may now fall under a newer policy fixture. Resuming the old run would write a recommendation under the old policy_version; INV-6 would catch it. Starting a fresh run on purpose is often the right call, and it is a decision for a person, recorded as such.

At-least-once at every hop: inbox and outbox togetherEach hand-off between systems repeats the same problem

A request crosses several hand-offs: email to intake, intake to queue, queue to worker, worker to store, store to cockpit. Each hand-off between two systems that cannot commit together has the lost-acknowledgement moment, so each one is at-least-once at best. A design that is safe end to end needs a deduplication point after every such hop.

Two patterns pair up. On the sending side, the outbox (Idempotency) makes sure an event leaves only if the change committed, at the price of possible duplicates. On the receiving side, the inbox makes sure a duplicate does no harm. With both, the producer cannot lose an event and the consumer cannot double its effect.

The intake step deserves the same care as the worker. If it can put the same email on the queue twice, two messages with two different message_ids arrive, and the inbox, which deduplicates by message_id, cannot see that they are one request. The intake step needs its own key, for instance derived from the email's message header, so that one email yields one queue message.

Checklist#

CheckWhat to look for in your own system
Work with side effects goes through a durable queueA crashed worker's request comes back instead of disappearing
The worker acknowledges after the workAt-least-once by design; at-most-once only for work that is cheap to lose
Every consumer has an inboxOne unit of work per message_id, enforced by a unique index, written with the effect
The visibility timeout exceeds the slowest normal runPlus its retries; extended while working where the queue allows it
A receive limit and a dead-letter queue existEvery dead-lettered message raises an alarm and reaches a person
Permanent errors are not retried inside the stepThe message is handed back or dead-lettered, never looped
Every run reaches a named terminal stateIncluding dead_lettered
Redrives are safeInbox and idempotency keys outlive the dead-letter queue's retention
The producer deduplicates tooOne email gives one message

In one breath#

A queue keeps work from being lost by delivering it again whenever an acknowledgement does not arrive, so a consumer must expect every message more than once. Acknowledge after the work, keep an inbox that allows one run per message id, and let the idempotency keys of that run turn any repeated write into a replay: that is an exactly-once effect, the only kind of exactly once on offer. A message that can never succeed gets a receive limit and a dead-letter queue, and a person.

The careful workflow now survives a duplicate delivery and a poison message, and a redelivered message after a crash finds its run. Finding the run is half of a recovery. The other half is knowing how far the run got, so the new worker continues at step 6 instead of starting over at step 1 and calling every provider again. That is the work of Checkpoints.