The insurance workflow of this site runs on a few hundred lines of plain Python. The underwriting agent reads a submission, calls the roof-data provider, retrieves the guidance, runs the rules, drafts a recommendation and stores it. The model recommends; a person decides. Every protection in that code was written by hand: the retry loop, the idempotency key, the checkpoint, the breaker, the dead-letter queue.
Then the team decides to stop hand-building. In one week three proposals arrive. Move the workflow onto a workflow engine, because "it retries for us". Put the model call behind the provider's SDK, because "the SDK already retries". Replace the simulated queue with a real broker, because "the broker guarantees delivery".
Take scenario UW-R04 from the example: the model answers 429 (too many requests) twice, then succeeds. The hand-built step allows 3 attempts and waits as the provider asks. Now wrap each of those attempts around an SDK that retries a 429 twice on its own, inside an engine whose default retry policy for a step has no attempt limit. Nobody wrote a loop on purpose, and yet the provider can see three times three requests for one step, and the engine will keep starting new ones. Three layers each did their job. Together they built the retry storm that the hand-built code was designed to prevent.
The question that sorts the tools is not which one is better. It is the one this whole site asks, applied to each tool in turn: what is still true after this fails, because of the tool, and what is still my job?
Plain English. A standard is an agreement about meaning: what a status code says, what a header promises. A tool is code someone else wrote and runs: a library in your process, a broker, a workflow engine. A standard cannot retry anything. A tool can, and that is exactly why you must know when it does.
This is a reference page. It is the only page of this site that makes statements about named tools and standards; the mechanism pages link here. One note before any result: the agent in the example is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
How each item was read#
Every item below was read on its own documentation on 2026-11-17, and each claim is logged with its address in the repository under docs/facts/frameworks.md. Four rules hold for the whole page.
- Only what the documentation says. Where the pages read do not state something, the text says "the documentation read here does not say". That is a statement about the reading, not about the tool.
- No prices, no ranking. No item is called the best. Documentation changes faster than this page; check the linked source before you rely on a sentence.
- Code in a tool's API is "documented shape; not run here". It follows the calls the documentation shows and was checked for syntax only. Nothing was installed. The only code on this site that was run is
examples/uw-reliability/. - Standards are kept apart from tools. An RFC (a published Internet standard) or an Internet-Draft (a working document that may still change) defines meaning. Products and libraries implement behaviour. The page treats them as two groups.
Each item follows the same template, so you can read across: what it is, the problem it solves, its model (protocol or data model), a minimum example, then strengths, limitations, integration and security, when plain code is better, and the version and sources. The last four parts are folded under each item, so the first reading stays short.
The diagram places the twelve items by the layer they act on. Read it from the wire upwards.
Diagram: Twelve items by the layer they act on, and where each one can repeat work
Read from the left. Standards define what a request, a status code and a header mean; they run nothing. In-process libraries wrap one call and retry it inside your process. Brokers hold messages and deliver them again after a crash. Workflow engines save a run's progress and resume it. Every layer to the right of the standards can repeat work, and each counts its own attempts, so a design names one owner for the retry of each step. The idempotency key and the invariants stay with the application in every layer.
- Standards · meaning on the wire
- RFC 9110: HTTP Semantics: which methods may be repeated automatically, how long to wait, what a temporary overload looks like.
- RFC 6585: A status code for a rate limit, which may carry Retry-After.
- Idempotency-Key draft: A request header carrying a client-generated key, so a server can recognize a repeated request.
- In-process libraries · one call
- Resilience4j: Java decorators around a call. Retries in the same process; gone after a crash.
- Model SDKs: Client libraries that retry connection errors, 408, 409, 429 and 5xx twice by default, with a 10-minute default timeout.
- Brokers · delivery
- Apache Kafka: At-least-once by default; exactly once between Kafka topics with transactions; an external store must cooperate.
- Amazon SQS: A received message is hidden for the visibility timeout and comes back if not deleted; a redrive policy moves it to a dead-letter queue.
- RabbitMQ: Unacknowledged deliveries are requeued with the redeliver flag; quorum queues dead-letter after a delivery limit.
- Workflow engines · run progress
- Temporal: Replays a workflow's history after a crash; finished Activities are not run again; Activities retry by policy.
- LangGraph: Saves graph state per super-step; resumes unfinished work by thread id; replay re-runs a node from its start.
- AWS Step Functions: Declared retries and fallbacks per state; Standard workflows up to one year; a task token pauses for an outside answer.
- Still yours · every layer
- Key and invariants: No tool knows which two requests are the same intended write, or that one submission must leave one recommendation row.
The observation to carry through the page: each layer can retry, and each layer counts its own attempts. A design must name one layer that owns the retry of each step, and turn the retries of the others off or down.
What this is not. Choosing which step runs next, branching and routing between agents is orchestration: https://orchestrate.sammuthu.com. This page describes engines only by what they keep reliable: retries, saved progress, redelivery and the waiting step.
Group 1: standards#
Three documents define words that every retry depends on. None of them runs code. They matter because the model provider, the roof-data provider and your own write tool speak HTTP, and a retry is only safe when both sides agree on what a repeated request means.
RFC 9110: idempotent methods, Retry-After and 503#
What it is. RFC 9110, "HTTP Semantics" (June 2022, an Internet Standard, STD 97), defines what HTTP methods, header fields and status codes mean. Three parts of it carry the retry rules of this site.
The problem it solves. A client whose connection dropped before the answer arrived does not know whether the server acted. RFC 9110 says when the client may repeat the request without asking anyone: when the method is idempotent. A method is idempotent when the intended effect of several identical requests equals the effect of one. PUT, DELETE and the safe methods (such as GET) are idempotent; POST is not.
Its model. Section 9.2.2 adds the rule this site leans on: a client should not automatically retry a non-idempotent request unless it knows the request is idempotent anyway, or can detect that the first one was never applied. A proxy must not retry such requests, and a client should not retry a failed automatic retry. Section 10.2.3 defines Retry-After: how long the client ought to wait before a follow-up request, as an HTTP date or as a number of seconds. Section 15.6.4 defines 503 Service Unavailable: a temporary overload or maintenance that will likely pass after some delay; the server may send Retry-After with it, and is not required to use 503 at all when overloaded (it may refuse the connection).
Minimum example. A 503 that tells the client to wait two minutes, as RFC 9110 writes it:
HTTP/1.1 503 Service Unavailable
Retry-After: 120
In the example, record_recommendation is the non-idempotent request in this sense: a plain repeat would store a second row (UW-R06). The idempotency key is the "means to detect that the original request was applied" that 9.2.2 asks for.
RFC 9110: strengths, limits, integration, sourcesWhat the standard settles, and the half of the problem it leaves open
Strengths. It is the shared vocabulary of every HTTP library, proxy and provider. A client that follows 9.2.2 cannot create a duplicate write by retrying on its own.
Limitations. Idempotency in 9.2.2 is about what the user requested; the RFC says a server may still log each request separately. It says nothing about how a server makes a POST safe to repeat. And Retry-After is advice from the server: nothing forces a client to honour it.
Integration and security. Check what every layer between you and the provider does with a failed request: an HTTP client library, a service mesh or a gateway may retry on its own. 9.2.2 forbids a proxy from retrying a non-idempotent request; this site's reading is that a mesh configured to retry POST would break that rule, so check its configuration. Cap a Retry-After value before obeying it, so a wrong header cannot park a run beyond its deadline.
When plain code is better. Always, in the sense that a standard is not code: you implement it. The example reads retry_after_ms from the stand-in model and waits that long inside the deadline (retry.py:retry_delay).
Version and sources, read 2026-11-17. RFC 9110, sections 9.2.2, 10.2.3 and 15.6.4.
RFC 6585: 429 Too Many Requests#
What it is. RFC 6585 (April 2012, a Proposed Standard) adds four HTTP status codes. Section 4 defines 429 Too Many Requests.
The problem it solves. Before 429, a server that wanted a client to slow down had no specific code to say so. 429 means the client sent too many requests in a given amount of time: a rate limit, which passes with time, so a later retry may succeed.
Its model. The response should explain the condition and may include Retry-After. The RFC deliberately does not say how the server identifies the client or counts requests: per resource, per server or across a set of servers are all allowed. A cache must not store a 429. Section 7.2 adds that a server is not required to use 429: under attack or heavy load, dropping connections may be the better choice.
Minimum example. The shape of the response, with the wait in seconds:
HTTP/1.1 429 Too Many Requests
Retry-After: 2
In UW-R04 the stand-in model answers 429 twice with retry_after_ms 1500. The careful workflow waits 1500 ms each time and succeeds on attempt 3, at 5198 ms in this example; the naive one retries at once and makes 33 attempts.
RFC 6585: strengths, limits, integration, sourcesA 429 says "slow down", not "how much" or "for whom"
Strengths. One code that every client can classify as transient and retryable with a wait.
Limitations. A 429 does not say which limit was hit or for how long, unless Retry-After or provider-specific headers do. And a 429 is not always transient in practice: a model provider documents a 429 for a reached monthly spend cap that will not pass by retrying (see the model SDK section below). Classify by the body or the headers, not by the number alone.
Integration and security. Rate limits are usually per account or per key, shared by every worker. One worker's retries spend the quota of all the others; the shared limit belongs to Concurrency.
When plain code is better. Reading the code, the wait and the limit is a few lines; the example's errors.py:classify and retry.py:retry_delay do it.
Version and sources, read 2026-11-17. RFC 6585, sections 4 and 7.2.
The Idempotency-Key HTTP header (an Internet-Draft)#
What it is. "The Idempotency-Key HTTP Header Field", draft-ietf-httpapi-idempotency-key-header, is an Internet-Draft of the IETF httpapi working group. The revision read on 2026-11-17 is -08, last updated 22 October 2026. It is a working document, not an RFC, and may change.
The problem it solves. RFC 9110 says a POST must not be retried blindly unless the client can detect that it was applied. This draft standardizes one way to detect it: the client sends a key, and the server uses the key to recognize a repeat. It is the idempotency key of this site, written as an HTTP header.
Its model. The header carries one string, generated by the client; the draft recommends a UUID or a similar random identifier, and the key must not be reused with a different payload. Revision -08 gives the server's answers: a missing key where one is required gets 400; a repeat that arrives while the first request is still being processed gets 409 Conflict; the same key with a different payload gets 422 Unprocessable Content. A server may expire keys, and should publish how long it keeps them.
Minimum example. A write of the insurance example as an HTTP request. The key is this site's design, built from run_id and the step id:
POST /recommendations HTTP/1.1
Content-Type: application/json
Idempotency-Key: "run-7f3a:record_recommendation"
Idempotency-Key: strengths, limits, integration, sourcesA random key per request versus a key per intended write
Strengths. One header name and one set of status codes in place of every provider's own. The 409 for a request still in progress closes the race between two overlapping attempts that Idempotency describes.
Limitations. The draft recommends a random identifier. A random value is safe only if the client creates it once and stores it before the first attempt. If the client generates it inside the retry loop, every attempt carries a new key and the server sees new requests: the duplicate that the duplicate lab shows for "a random value on every call". This site's key, <run_id>:<step id>, is derived from the run, so a resumed process rebuilds the same key without having stored it. Both meet the draft's rule that one key means one payload; they differ in what survives a crash.
Integration and security. A server that keeps keys must decide how long (the draft asks it to publish this), and must scope keys to the caller so one client cannot replay another's response. A 409 is retryable after a wait; a 422 is a bug in the caller and must not be retried. Note that the model SDKs read on this page retry 409 by default; that is right for "still in progress" and wrong for a conflict that will not pass.
When plain code is better. For a write tool inside your own system, a unique index on the key column does the same work without HTTP.
Version and sources, read 2026-11-17. Datatracker entry for the draft, revision -08. Check the revision again before relying on a detail.
Group 2: tools#
The nine tools fall into three kinds. Workflow engines (Temporal, LangGraph, AWS Step Functions) save a run's progress and resume it. Brokers (Apache Kafka, Amazon SQS, RabbitMQ) hold messages and redeliver them. In-process libraries (Resilience4j, the model SDKs) wrap one call with retries, timeouts or a breaker. Each section asks the same question: after a failure, what does this tool keep true, and what is still yours?
Temporal#
What it is. A platform for running workflows as code. A Workflow is code that coordinates the run; an Activity is a function that does one action with the outside world, such as calling a provider. Workers (your processes) execute both; the Temporal service records what happened.
The problem it solves. The crash of UW-R09. The documentation describes an Event History, "a complete, ordered log" of what already happened in a Workflow Execution. After a failure, Temporal re-runs the Workflow code and replays the history to rebuild its state; the results of Activities that already finished are taken from the history, so those Activities do not run again. This is what Checkpoints calls durable execution, though the Workflow page read here does not use that phrase. Workflows can keep running for years, so a durable wait for the underwriter fits the model.
Its model. Two rules carry the reliability. Workflow code must be deterministic on replay: no reading the clock directly, no random numbers, no network calls outside an Activity. And Activities are retried by a Retry Policy: by default an initial interval of 1 second, a backoff coefficient of 2.0, a maximum interval of 100 times the initial one, and unlimited attempts. Workflows themselves do not retry by default. Each Activity must have a Start-To-Close timeout (one attempt) or a Schedule-To-Close timeout (all attempts together); the documentation strongly recommends Start-To-Close, because without it the service cannot detect a worker that crashed.
Minimum example. Step 2 of the insurance workflow as a Temporal Activity call in Python, with this site's limits written into the policy. Documented shape; not run here. A successful run would show the provider called at most three times, each attempt cut off after 2 seconds; it would say nothing about whether a write Activity is safe to repeat.
# documented shape; not run here (compiled only)
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
from uw_activities import get_roof_assessment # your Activity: a call to the roof-data provider
@workflow.defn(name="UnderwriteSubmission")
class UnderwriteSubmission:
@workflow.run
async def run(self, submission_id: str) -> dict:
roof = await workflow.execute_activity(
get_roof_assessment,
submission_id,
start_to_close_timeout=timedelta(seconds=2), # one attempt
retry_policy=RetryPolicy(
initial_interval=timedelta(milliseconds=500),
backoff_coefficient=2.0,
maximum_interval=timedelta(seconds=8),
maximum_attempts=3,
non_retryable_error_types=["ProviderRejected"],
),
)
return roof
Temporal: strengths, limits, integration, sourcesReplay keeps the run; the Activity's write is still yours
Strengths. Resume after a crash without writing checkpoint code; recorded Activity results, so the model is not asked twice for an answer it already gave; long durable waits; timeouts that separate one attempt from the whole budget, as Retries separates a timeout from a deadline.
Limitations. The documentation recommends that Activities be idempotent "so retries can be processed without duplicate side effects", and the Activities page read here does not mention idempotency keys. An Activity that stored the recommendation and crashed before reporting success will be retried: that is UW-R08, and the key is still yours. The default of unlimited attempts means an unchanged policy retries a permanent error until a timeout stops it; mark permanent errors non-retryable. Determinism is a rule your code must follow, and a model call in Workflow code would break it; it belongs in an Activity.
Integration and security. Activity inputs and results are recorded in the history, so the recommendation, the evidence and anything else passed between steps is stored by the service. Decide what may be stored there; the pages read here do not describe encryption of payloads. The Python SDK runs Workflow code in a sandbox; the pages read here only link to it.
When plain code is better. When runs are short, a crash may restart the run from the start without harm, and your store already holds a checkpoint per step, as the example's does.
Version and sources, read 2026-11-17. Python SDK 1.36.1 (dated 14 November on its releases page). Retry policies · Activities · Workflows · Detecting Activity failures · Python Activity timeouts · Python SDK releases.
LangGraph: checkpointers and durable execution#
What it is. A library for building agents as graphs of nodes. Its checkpointer saves the state of a graph run, called a thread, so the run can continue later.
The problem it solves. The same crash, inside an agent loop. A checkpoint is saved at each super-step (one round in which the scheduled nodes run). If one node fails in a super-step, the outputs of the nodes that succeeded are kept as pending writes and are not re-run on resume. To continue after an error, the documentation shows graph.invoke(None, config) with the same thread_id: it "loads the latest checkpoint and retries unfinished work".
Its model. Three durability modes decide when state is written: "exit" only when the graph exits (fastest; a crash mid-run leaves nothing to recover), "async" while the next step runs (a small risk of missing a checkpoint), and "sync" before the next step starts (the strongest guarantee, at some cost in speed). The default mode is not stated on the pages read here. Replay re-runs the nodes after the chosen checkpoint, including model calls and API requests, and a node interrupted for a human answer restarts from its beginning when resumed with Command(resume=...).
Minimum example. A graph compiled with a production checkpointer and resumed after a crash. Documented shape; not run here.
# documented shape; not run here (compiled only)
from langgraph.checkpoint.postgres import PostgresSaver
from uw_graph import builder # your StateGraph: the eight steps as nodes
DB_URI = "postgresql://uw@localhost:5432/uw"
config = {"configurable": {"thread_id": "run-7f3a"}} # one thread per run
checkpointer = PostgresSaver.from_conn_string(DB_URI)
checkpointer.setup()
graph = builder.compile(checkpointer=checkpointer)
graph.invoke(None, config, durability="sync") # after a crash: continue from the latest checkpoint
LangGraph: strengths, limits, integration, sourcesThe mode you pick decides what a crash costs
Strengths. Checkpointing inside the agent framework itself; pending writes so finished siblings are not repeated; resume and human interrupts on the same mechanism.
Limitations. Replay re-runs a node from its start, so a node that writes and then crashes writes again on resume unless the write is idempotent: the key is still yours. A model call inside a re-run node is asked again and may answer differently; this site's rule (record the answer once, use the record on replay) has to be built into the node. With "exit" or "async" a crash can lose progress. The in-memory saver loses everything on restart; the documentation calls the SQLite checkpointer "ideal for experimentation and local workflows" and the Postgres one "ideal for using in production".
Integration and security. Checkpoints contain the whole graph state: submission, tool results, model output. The default serializer can fall back to pickle when enabled, and an EncryptedSerializer encrypts persisted state with a key from LANGGRAPH_AES_KEY. Treat the checkpoint store as sensitive data. The thread_id is this site's run_id: look it up by message_id, or a redelivered message starts a second thread.
When plain code is better. When the workflow is a fixed sequence, as the eight steps here are, a checkpoint row per step in your own store is a small amount of code you fully understand.
Version and sources, read 2026-11-17. langgraph 1.2.22 (14 November 2026, MIT licence). Checkpointers · Durability type · Persistence · PyPI. The address of the durable-execution page served the persistence overview on the day of reading; the details above come from the checkpointers page.
AWS Step Functions#
What it is. A managed AWS service that runs state machines written in the Amazon States Language (JSON). Each Task state calls a service; the service records the execution.
The problem it solves. Retries, fallbacks and long waits declared next to each step instead of coded around it. A state that reports an error fails the whole execution, unless the state has a Retry or a Catch field.
Its model. A retrier matches error names (ErrorEquals) and sets IntervalSeconds (default 1), MaxAttempts (default 3; 0 means never retry), BackoffRate (default 2.0), an optional cap MaxDelaySeconds, and JitterStrategy, FULL or NONE, where the default is NONE. Retriers run first; when they are exhausted, a catcher sends the run to a fallback state. The choice of workflow type changes the guarantees: Standard workflows run up to one year with exactly-once workflow execution ("never run more than once, unless you have specified Retry behavior"); Express workflows run up to five minutes, at-least-once when started asynchronously and at-most-once when started synchronously, with idempotency "not automatically managed". The callback pattern .waitForTaskToken pauses a Standard workflow until someone returns a task token, and the documentation names human approval as a use.
Minimum example. Step 2 of the insurance workflow, with jitter switched on and the provider's permanent rejection sent to the degraded path. Documented shape; not run here (the JSON parses; the resource name is a placeholder).
{
"Type": "Task",
"Resource": "arn:aws:lambda:us-east-1:123456789012:function:GetRoofAssessment",
"TimeoutSeconds": 2,
"Retry": [
{"ErrorEquals": ["States.Timeout", "ProviderUnavailable"], "IntervalSeconds": 1,
"MaxAttempts": 2, "BackoffRate": 2.0, "MaxDelaySeconds": 8, "JitterStrategy": "FULL"}
],
"Catch": [
{"ErrorEquals": ["States.ALL"], "ResultPath": "$.roof_error", "Next": "DraftWithoutRoof"}
],
"Next": "SearchGuidance"
}
MaxAttempts here counts retries, so 2 means three tries in all, the same as this site's 3 attempts.
Step Functions: strengths, limits, integration, sourcesExactly-once is a property of the state machine, not of your write
Strengths. Retry, backoff, cap, jitter and fallback declared per state; the execution history; one-year Standard executions and the callback token for a human step.
Limitations. "Exactly-once" describes how the service runs states, and the page itself adds "unless you have specified Retry behavior". A Task that stored the recommendation and then timed out is retried by its retrier, and the store sees a second request: the key is still yours. Jitter is off unless you set FULL. Express workflows can run twice and keep no history in the service.
Integration and security. Callback tokens only work from principals in the same AWS account, and a waiting task without HeartbeatSeconds can wait until the one-year limit; set one, as Advanced argues for the underwriter's step. Retries are state transitions, which the documentation notes affect billing; no prices are given here.
When plain code is better. When the workflow does not run on AWS, or when each step is a function call inside one process.
Version and sources, read 2026-11-17. The developer guide is not versioned on the pages read. Error handling · Choosing the workflow type · Service integration patterns.
Apache Kafka#
What it is. A distributed log. Producers append messages to partitions of a topic; each consumer keeps its own position (offset) in the log and moves it forward as it works.
The problem it solves. Durable, ordered delivery of many events to many consumers, with the ability to read again. The design documentation lists the three delivery guarantees and says Kafka "guarantees at-least-once delivery by default".
Its model. The order of two actions in the consumer decides the guarantee: save the position and then process, and a crash in between loses the message (at-most-once); process and then save the position, and a crash in between processes it again (at-least-once). Since version 0.11.0.0 the producer offers an idempotent option: the broker gives each producer an ID and drops resent messages by their sequence number. Transactions let one producer write to several partitions atomically, and with Kafka Streams and read-committed consumers a read-process-write between Kafka topics can be exactly once. For an external system, the documentation's advice is to store the offset in the same place as the output.
Minimum example. The consumer of the insurance queue, in the order that gives at-least-once delivery. A sketch with names from the design documentation; not a client API.
# sketch: at-least-once consumer of the submissions topic
poll() -> message msg-51c2 at offset 1041
run_workflow(msg-51c2) -> run-7f3a found by message_id (the inbox), one recommendation row
commit_offset(1042) -> a crash before this line means msg-51c2 is read again
Kafka: strengths, limits, integration, sourcesExactly-once inside Kafka; your store is outside it
Strengths. Replay from any retained offset, ordering within a partition, idempotent producers, and transactions between topics.
Limitations. The exactly-once path covers writes to Kafka topics. The recommendation row is written to your store, outside Kafka, so the documentation's own condition applies: the store must cooperate, either by keeping the offset with the output or by an idempotency key. The design page read here does not describe a dead-letter queue for consumers; a poison message blocks progress on its partition until your code moves it aside (UW-R11).
Integration and security. A redelivered message is the normal case under at-least-once, so the inbox lookup by message_id is required, not optional. Retention decides how far back a replay can reach.
When plain code is better. For a single workflow with modest volume, a queue with a visibility timeout and a dead-letter queue (as in the example) is simpler.
Version and sources, read 2026-11-17. Documentation for Kafka 4.4 (by its address). Design: message delivery semantics.
Amazon SQS#
What it is. A managed message queue on AWS, in two types: standard queues and FIFO (first in, first out) queues.
The problem it solves. Hand work to workers and redeliver it when a worker dies. A received message stays in the queue but is hidden from other consumers for the visibility timeout; if the worker does not delete it in time, it becomes visible again. The default visibility timeout is 30 seconds, the same as the example's setting, and it can be extended per message up to 12 hours from the first receipt.
Its model. SQS uses an at-least-once model: the documentation says there is no guarantee that a message will not be delivered more than once, even within the visibility timeout. A redrive policy sets maxReceiveCount, after which a message moves to a dead-letter queue. FIFO queues offer what the documentation calls "exactly-once processing": a SendMessage retried within a 5-minute deduplication interval does not add a duplicate to the queue, keyed by a message deduplication ID or a hash of the body.
Minimum example. The worker loop of the example in SQS terms. A sketch with the action names from the documentation; not an SDK call.
# sketch: one worker, one message
ReceiveMessage -> msg-51c2 hidden for 30 s
run the workflow (inbox by message_id, key per write)
DeleteMessage -> done; a crash before this line: visible again after 30 s
ChangeMessageVisibility(msg, 60) -> optional: extend while a long step runs
SQS: strengths, limits, integration, sources"Exactly-once processing" is deduplication at send time
Strengths. Redelivery after a crash with no code; a dead-letter queue configured, not written; deduplication of sends on FIFO queues.
Limitations. Read the FIFO promise carefully: it stops a retried send from adding a second copy to the queue within five minutes. It does not stop a worker from processing one copy twice when the worker crashes before deleting it (UW-R08 with a redelivery). The inbox and the key are still yours. A visibility timeout shorter than a run lets a second worker take the message while the first is still working; the example's lease and fencing token exist for that (Concurrency).
Integration and security. The dead-letter queue must be in the same account and Region. For standard queues a message keeps its original enqueue time in the dead-letter queue, so the documentation advises a longer retention there. A maxReceiveCount of 1 sends a message to the dead-letter queue after a single failure; the documentation advises setting it high enough for sufficient retries.
When plain code is better. When the work never leaves one process, or the queue already exists in your database.
Version and sources, read 2026-11-17. The developer guide is not versioned on the pages read. Visibility timeout · Dead-letter queues · Exactly-once processing.
RabbitMQ#
What it is. A message broker. Publishers send messages to exchanges, which route them to queues; consumers receive deliveries and acknowledge them.
The problem it solves. The same handover as SQS, with the acknowledgement made explicit. In manual mode the consumer sends basic.ack after its work, or basic.nack or basic.reject. If the channel or connection closes first, unacknowledged deliveries are requeued and redelivered with the redeliver flag set. The documentation calls automatic acknowledgement "fire-and-forget" and says it "should be considered unsafe" for some workloads.
Its model. On the publishing side, publisher confirms tell the publisher the broker has taken responsibility for a message. On the consuming side, the documentation says consumers "must be prepared to handle redeliveries", with idempotence in mind. A message is dead-lettered when it is rejected without requeue, its time-to-live expires, a queue length limit drops it, or it exceeds a quorum queue's delivery limit, which defaults to 20 since version 4.0. Dead lettering is at-most-once by default; quorum queues offer at-least-once dead lettering.
Minimum example. A sketch of the consumer in AMQP 0.9.1 terms; method names from the documentation, not a client API.
# sketch: manual acknowledgement
deliver(msg-51c2, redeliver=false)
run the workflow
basic.ack -> removed from the queue
# a crash before basic.ack: requeued, delivered again with redeliver=true
# rejected for good: basic.reject requeue=false -> dead-letter exchange (x-death reason "rejected")
RabbitMQ: strengths, limits, integration, sourcesA poison-message limit by default, and dead letters that can be lost
Strengths. Explicit acknowledgements, a redelivery flag, a default delivery limit for poison messages on quorum queues, and an x-death header that records why and how often a message was dead-lettered.
Limitations. The redeliver flag says a message may have been seen before; it does not say whether the work was done. The inbox is still yours. Default dead lettering republishes without confirms, so a dead-lettered message can be lost if its target is unavailable; quorum queues with at-least-once dead lettering close that gap.
Integration and security. Configure dead lettering by policy rather than by queue arguments; the documentation recommends policies because arguments cannot be changed without redeploying. The user needs read access on the source queue and write access on the dead-letter exchange. Setting the delivery limit to -1 disables it; the documentation says this is not recommended.
When plain code is better. As for SQS: when the work stays in one process, or a table can serve as the queue.
Version and sources, read 2026-11-17. Documentation labelled version 4.4. Consumer acknowledgements and publisher confirms · Dead letter exchanges · Quorum queues.
Resilience4j#
What it is. A Java library that its documentation calls a "lightweight fault tolerance library designed for functional programming". It wraps a call in decorators: retry, circuit breaker, rate limiter, bulkhead, time limiter and cache. Version 2 requires Java 17.
The problem it solves. The per-call protections of Retries and Breakers for a Java or Spring service, without writing them. A familiar tool for the backend engineer who wants to see where an agent's retry differs.
Its model. Retry: maxAttempts defaults to 3, counting the first call; the wait is a constant 500 ms unless an interval function is set (IntervalFunction.ofExponentialBackoff(), or ofRandomized() for a randomized wait); and every exception is retried unless you restrict retryExceptions or add ignoreExceptions. CircuitBreaker: states CLOSED, OPEN and HALF_OPEN, a count-based window of 100 calls by default, opening at a 50% failure rate after at least 100 calls, 60 seconds open, then 10 trial calls in half-open. Calls refused while open fail with CallNotPermittedException.
Minimum example. The provider call of step 2 with this site's numbers. Documented shape; not run here.
// documented shape; not run here
RetryConfig retryConfig = RetryConfig.custom()
.maxAttempts(3) // the first call counts
.intervalFunction(IntervalFunction.ofExponentialBackoff(500, 2.0))
.retryExceptions(ProviderTimeoutException.class)
.ignoreExceptions(ProviderRejectedException.class) // permanent: never retried
.build();
CircuitBreakerConfig breakerConfig = CircuitBreakerConfig.custom()
.slidingWindowSize(5)
.minimumNumberOfCalls(5)
.failureRateThreshold(60) // 3 of the last 5 calls
.waitDurationInOpenState(Duration.ofSeconds(10))
.permittedNumberOfCallsInHalfOpenState(1)
.build();
Resilience4j: strengths, limits, integration, sourcesDefaults made for busy services, not for five calls a minute
Strengths. Small decorators that compose; the same vocabulary as this site; special breaker states (FORCED_OPEN, DISABLED, METRICS_ONLY) that act as a manual switch.
Limitations. The defaults retry every exception, including permanent ones, and wait a constant 500 ms with no jitter; both must be changed for this site's rules. The breaker's default of at least 100 calls before it can open suits a busy service; a workflow that calls the provider a few times a minute would never open it, which is why the example above sets a window of 5. The retry protects one call in one process: it does not survive a crash and knows nothing about the run's deadline.
Integration and security. In a Spring service the decorators often sit next to an HTTP client that retries too; count attempts across both. The randomized interval function named in the documentation is a jittered wait; whether it can be combined with exponential growth is not stated on the page read here.
When plain code is better. In Python, where the library does not apply, or when the retry must read a retry_after_ms and a deadline, as retry.py:call_with_retry in the example does in a few dozen lines.
Version and sources, read 2026-11-17. v2.4.0 is the latest release on the releases page (dated 14 March; the year is not shown). Getting started · Retry · CircuitBreaker · Releases.
The model SDKs: Anthropic and OpenAI Python SDKs#
What they are. The official Python client libraries of two model providers. Both wrap HTTP calls to the provider's API, and both retry and time out by themselves.
The problem they solve. Transient failures of the model call, without code. Both SDKs document the same defaults: an error is retried twice, with a short exponential backoff, for connection errors, 408, 409, 429 and 5xx responses. The default timeout is 10 minutes, a timed-out request raises APITimeoutError, and timed-out requests are retried twice too. Both offer max_retries per client or per request, where 0 turns retries off. The OpenAI README adds that requests are retried only when their body can be safely resent. Anthropic's error page says the SDK honours the retry-after header when present.
Their model. A 429 from Anthropic's API may carry retry-after in seconds, and its rate limits use a token bucket. One 429 is different: when an organization reaches its monthly spend cap, the 429 has no retry-after header and the documentation says retrying, "including the SDK's automatic retries", fails until access resumes. The Anthropic SDK also refuses a non-streaming request expected to take longer than about 10 minutes, unless you stream or set the timeout yourself.
Minimum example. The example's step 5, with the SDK's own retries turned off so that one layer (the workflow) owns the retry, and a timeout that fits the run's deadline. Documented shape; not run here; no key was used.
# documented shape; not run here (compiled only)
from anthropic import Anthropic
client = Anthropic(max_retries=0, timeout=20.0) # the workflow owns retries; one attempt may take 20 s
def draft_once(prompt: str) -> str:
message = client.messages.create(
model="claude-opus-5-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
return "".join(block.text for block in message.content if block.type == "text")
Model SDKs: strengths, limits, integration, sourcesTwo retries you did not write, and a 429 that will not pass
Strengths. Sensible defaults for a script: transient errors retried with backoff, a long timeout for long generations, typed errors that make classification easy.
Limitations. The retries are invisible to your workflow's counters. Inside the example's loop of 3 attempts they make up to 9 requests per step, and the retry budget of Retries never sees 6 of them. A timeout of 10 minutes is longer than the example's whole run deadline of 2 minutes; set it from the deadline. And 409 is retried by default: right for a conflict that passes, wrong for one that will not.
Integration and security. Keep the provider's key out of the agent's reach and out of logs. The request-id of each response is the handle to quote to the provider; store it on the attempt's span. A spend-cap 429 must be classified as permanent, or every run retries into it. The 2.x Anthropic SDK pins an API version per major version, so an SDK upgrade is a change to release with the care of Deployment.
When plain code is better. The SDK is the plain code for the HTTP call. What it should not own is the retry policy of a step that has a deadline, a budget and an idempotency key.
Version and sources, read 2026-11-17. anthropic 2.10.0 (17 November 2026); openai 3.55.0 (uploaded to PyPI at 05:22 UTC on 18 November 2026, the evening of 17 November in Pacific time). Both SDKs now name httpx2 as their HTTP layer. Anthropic Python SDK · Claude API errors · Claude API rate limits · OpenAI Python SDK · anthropic on PyPI · openai on PyPI.
The twelve side by side#
You have read each item against the same question. The table puts the answers in one place and adds nothing the sections did not prepare. Use the buttons to filter the rows.
How to read the cells: Retries on its own is yes when the item retries without configuration, opt-in when it retries only when you configure it, n/a for a standard or a broker that redelivers instead of retrying, and unstated when the pages read here do not say. Saves run progress is partly when it holds only a position or only in one mode. Can run a step twice is yes whenever a retry, a replay or a redelivery can repeat work.
| Item | Kind | Retries on its own | Saves run progress | Can run a step twice | Your key still needed | Read on 2026-11-17 |
|---|---|---|---|---|---|---|
| RFC 9110 | standard | n/a | no | n/a | yes | June 2022, STD 97 |
| RFC 6585 | standard | n/a | no | n/a | n/a | April 2012 |
| Idempotency-Key draft | standard | n/a | no | n/a | yes | revision -08, 22 October 2026 |
| Temporal | engine | yes | yes | yes | yes | Python SDK 1.36.1 |
| LangGraph | engine | unstated | yes | yes | yes | 1.2.22 |
| AWS Step Functions | engine | opt-in | partly | yes | yes | guide not versioned |
| Apache Kafka | broker | n/a | partly | yes | yes | docs 4.4 |
| Amazon SQS | broker | n/a | no | yes | yes | guide not versioned |
| RabbitMQ | broker | n/a | no | yes | yes | docs 4.4 |
| Resilience4j | library | opt-in | no | yes | yes | v2.4.0 |
| Anthropic Python SDK | library | yes | no | yes | yes | 2.10.0 |
| OpenAI Python SDK | library | yes | no | yes | yes | 3.55.0 |
Three notes keep the table honest. LangGraph's retry behaviour is unstated because the pages read describe resuming unfinished work on request, not an automatic retry policy. Step Functions' progress is partly because Standard workflows persist state between transitions and Express workflows do not. Kafka's progress is partly because what it saves is the consumer's position, not the state of your run.
Read the fifth column top to bottom. Every tool that can run a step twice leaves the idempotency key to you. That is the finding of this page in one column.
What stays yours, whichever tool you choose#
Each tool takes over part of a mechanism. None of them knows the insurance workflow, so four things stay with the team in every combination.
| Yours | Why no tool can own it | Where it is taught |
|---|---|---|
The idempotency key <run_id>:<step id> | Only the workflow knows which two requests are the same intended write | Idempotency |
| One layer that owns each retry | Each tool counts its own attempts; only the design sees all layers | Retries |
| The terminal states and the invariants | "One submission, one pending_review row" is a fact about insurance, not about queues | Checkpoints, Fault injection |
| The fault-injection suite | A tool's promise is tested by the tool's authors on their faults, not on yours | Fault injection |
Before AI. A Spring service that moves from hand-written retries to Resilience4j and from a database table to Kafka faces the same audit: which layer retries, which delivers twice, and which write is protected by a key. What is new for an agent is the model call in the middle, whose answer must be recorded once (no engine can make a model deterministic), and the human step at the end, which can wait for days.
The same idea for a coding agent#
A coding agent runs inside a CI job. The job runner retries a failed job, the agent's harness retries a failed tool call, and the model SDK retries a failed request. When the step that failed was "open a pull request", the three layers can open three. The same audit applies: pick the layer that owns the retry, turn the others off for that step, and make the step look for its earlier effect (an open pull request for the branch) before acting. The runtime around such an agent is https://harness.sammuthu.com; the path from change to production is https://diff-to-deploy.sammuthu.com.
When the plain code of the example is enoughFour signs that a product would add more than it removes
The example does without every tool on this page, in a few hundred lines. That is enough while four things hold. Runs are short enough that a crash can resume from a checkpoint row in your own store. The queue has one producer and a handful of workers. The team can read the whole recovery path in one sitting. And the fault-injection suite runs against that path on every change.
A product starts to pay when one of these breaks: runs that wait for weeks, many teams writing workflows, volumes where a broker's partitions and retention matter, or an audit that wants an execution history kept by a system the application cannot edit. Even then, the suite should keep running against the new stack: replace the simulated queue with the real broker, replace the hand-built checkpoint with the engine, and check that the same seven invariants hold. A tool that passes the same suite has earned its place; a tool adopted because it "guarantees" something has only been read about.
An adoption checklist for any engine or brokerQuestions to answer from the documentation before the first run
- Which errors does it retry by default, how many times, with what backoff and jitter, and under which total timeout?
- What does it record, and when: before the next step starts, while it runs, or only at the end?
- On resume or redelivery, which code runs again: the whole step, the node from its start, the message handler?
- What is its delivery or execution guarantee, in its own words, and under which conditions does the documentation say it does not hold?
- Where is the payload stored, for how long, and who can read it?
- How does a waiting step end: by a timeout you set, by a heartbeat, or never?
- Which of these answers did the pages you read not give? Write those down as open questions, not as assumptions.
Checklist#
| Question | Good answer |
|---|---|
| For each step, which single layer owns the retry? | Named in the design; the other layers are set to max_retries=0 or their equivalent for that step |
| Does any layer retry by default with unlimited attempts or a long timeout? | Found and bounded from the run's deadline |
Does every write that a tool may repeat carry <run_id>:<step id>? | Yes, whatever the tool promises about delivery |
Is the run found by message_id on every redelivery? | Yes; the broker's redelivery flag is not used as proof |
| Is a model's answer recorded once and used on replay? | Yes, in an Activity result, a checkpoint or your own table |
| Are permanent errors (a spend-cap 429, a 422, a rejection) marked non-retryable? | Yes, by type, in every layer |
| Does the fault-injection suite run against the real tool, not only the simulation? | Yes, with the same invariants |
| Is every statement about a tool dated, with its source? | Yes, in a fact log |
In one breath#
Standards say what a retry may assume; tools retry, save progress and redeliver for you. Every tool on this page can run a step twice, and none of them can choose the idempotency key, so the key, the terminal states and the invariants stay yours. Before adopting one, find every layer that retries, give each step one owner, and run the same fault-injection suite against the new stack.
Next: the dependencies no library wraps#
Every tool here protects calls between machines. The slowest and least available dependency of the insurance workflow is not a machine: it is the underwriter, who may be busy, on holiday or not authorized for this case. Two more limits are still open: a run that spends without stopping, and a queue in front of a person that grows faster than people can empty it. Advanced takes the human step, cost ceilings and capacity.