The underwriting agent has started a run for submission SUB-1004, the office building whose application says the roof is 9 years old while the roof-data provider says 22. The expected result is a recommendation to escalate, citing R-ROOF-03. The model recommends; a person decides.
In scenario UW-R09 the worker process dies 180 ms into the run. It has validated the submission and it has the roof assessment in memory. Then the machine is gone. Nothing threw an exception that a retry loop could catch, because the code that would have caught it is gone too.
Thirty seconds later the queue hands the same message to another worker, as Queues explains. Here is what the naive workflow does with it:
180 worker-A get_roof_assessment - PROCESS CRASH: after the step, after checkpoint
180 worker-A - - process is gone: no acknowledgement; visible again after 30000 ms
30000 worker-B - - message received: msg-e3c0, receive 2
30000 worker-B - - run created: trace_id e267a8b82db7b811b6b2f70312632f9e
30000 worker-B - - lease granted: fencing token 42
30000 worker-B load_submission - step done: no checkpoint
30180 worker-B get_roof_assessment 1 attempt ok
(The crash line says "after checkpoint" because that is where the fault plan places the crash; the naive workflow saves no checkpoint, so there is nothing to resume from.)
Worker B knows nothing about the first run. It creates a second run, calls the provider a second time and does all the work again. In this example the end result still has one recommendation row, so the damage looks small: one extra provider call. But the first run is still marked running and will never finish, and the work it did is thrown away. Move the crash a few steps later, after the recommendation was written, and the same "start over" leaves two recommendation rows for one submission (UW-R08, below).
This page is about the two things a run needs so that a crash costs one step and not the whole run: a saved position, and a way to undo the meaning of a step that cannot be finished.
Plain English. A retry tries one step again inside a process that is still alive. A crash takes the process away. What survives a crash is only what was written somewhere outside the process before it died. A retry is not a recovery.
Why a dead process cannot retry its way out#
Retries handle an error that the running code can see: a timeout, a 429, a failed write. The loop that retries lives in the memory of the worker. When the worker crashes (the machine restarts, the container is killed during a deployment, the process runs out of memory), the loop, the attempt counter and every intermediate result disappear with it.
For a short request the caller retries the whole request and loses a second of work. An agent run is long, calls a model whose answer is not the same every time, and writes to systems that remember (the index explains all three). Starting over repeats the slow part, asks the model again for an answer that may differ, and repeats writes that already happened.
So the run itself must leave a trail outside the process. That trail has a name.
A checkpoint is the saved state of a run after a completed step#
When the worker finishes step 2 of SUB-1004, it holds the validated submission and the provider's answer (roof 22 years old). If it writes those two things to the store before it starts step 3, a later process can read them back and start at step 3. Nothing about step 1 or step 2 has to happen again.
That saved record is a checkpoint: the saved state of a run after a completed step. In this example one checkpoint is kept per run, the latest one, and it is saved after every completed step of the eight in the workflow.
A checkpoint belongs to a run, and a run needs a name that a new process can find. The run id (run_id, for example run-9195) names one execution of the workflow for one submission. It is not the submission (SUB-1004 stays the same if a person asks for a fresh run) and it is not an attempt (attempt_number counts tries of one step inside a run). A redelivered message must lead to the same run, so the run is looked up by the message's message_id; Queues covers that lookup, the inbox.
What goes into the checkpoint follows from one question: what would the next step need, if this process vanished now? In this example the answer is the shape below, from the example's README.
| Field | Example (UW-R15) | Why it is saved |
|---|---|---|
run_id | run-7f3a | The key a new process uses to find the checkpoint |
step_id | draft_recommendation | The last completed step; the resume starts at the next one |
state.submission, state.roof, state.passages, state.facts, state.check | the validated input and every tool result | So steps 1 to 4 never run again, and the provider is not called again |
state.recommendation, state.model_route, state.degraded | the model's output and how it was produced | So the model is never asked again for an answer it already gave (see replay, below) |
state.evidence_ids | EV-ROOF-0412, EV-GUIDE-0031, ... | So the stored recommendation cites only evidence that exists in this run (INV-6) |
trace_id | 32 hex characters | So the resumed run writes into the same trace: one trace per run |
resume_count | 0, then 1 | How many times a new process picked the run up |
What stays out matters as much. A checkpoint holds data, not live objects: no open connection, no client, no lock. It holds no secret that the next step does not need. And it holds no attempt counter: attempt_number starts again at 1 in a new process, which is why the pair (resume_count, attempt_number) names an attempt in the trace.
What this is not. Which state is worth keeping, and the checkpoint as a data structure in its own right, belong to memory design: https://memory.sammuthu.com. This page uses a checkpoint for one purpose only, to recover a run.
Resume: find the run, load the checkpoint, continue#
To resume is to let a new process continue a run from its checkpoint. It is not a retry (a retry is another attempt of one step in the same process) and it is not a restart (a restart begins at step 1).
The example does it in two small methods of Workflow in workflow.py. restore loads the checkpoint and returns the index of the first step that still has to run; execute runs the steps from there and saves a checkpoint after each one. They are shown here dedented from the class, otherwise verbatim. Read them for the order of operations: run the step, then the crash point that the fault plan can trigger, then the checkpoint.
def restore(self) -> int:
"""Load the checkpoint, if any. Returns the index of the first step that still has to run."""
checkpoint = self.env.checkpoints.load(self.run["run_id"]) if self.env.flags["checkpoint"] else None
if checkpoint is None:
return 0
self.state = checkpoint["state"]
done = STEPS.index(checkpoint["step_id"]) + 1
self.report.update({step_id: "skipped" for step_id in STEPS[:done]})
self.tracer.event(checkpoint["step_id"], "resume from checkpoint", f"steps 1 to {done} are not run again")
return done
def execute(self, start: int) -> None:
"""Run the steps from `start`. After each completed step: save a checkpoint."""
with self.tracer.span("workflow", "underwrite_submission", {"message_id": self.run["message_id"]}):
for step_id in STEPS[start:]:
self.current_step = step_id
with self.tracer.span("step", step_id):
getattr(self, step_id)()
self.report.setdefault(step_id, "ran")
if self.run["state"] != "running" and step_id != "await_underwriter":
return # stopped early (step 7 was rejected)
self._crash_point(step_id, "before_checkpoint")
self._save_checkpoint(step_id)
self.tracer.event(step_id, "step done", "checkpoint saved" if self.env.flags["checkpoint"] else "no checkpoint")
self._crash_point(step_id, "after_checkpoint")
Now the same crash as in the opening, UW-R09, with the careful workflow. The command runs the scenario in a copy of examples/uw-reliability. A successful run proves that, in this simulation, the second process skips the two completed steps and calls the provider once in total. It does not prove anything about a real store's durability; the store here is memory inside one Python process.
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R09
The timeline and the end state, verbatim (the header and the seven invariant lines are left out; all seven hold):
t_ms worker step att event
0 queue - - message sent: msg-e3c0
0 worker-A - - message received: msg-e3c0, receive 1
0 worker-A - - run created: trace_id 33b54151e808a2b70642c3cd73a6e1f2
0 worker-A - - lease granted: fencing token 41
0 worker-A load_submission - step done: checkpoint saved
180 worker-A get_roof_assessment 1 attempt ok
180 worker-A get_roof_assessment - step done: checkpoint saved
180 worker-A get_roof_assessment - PROCESS CRASH: after the step, after checkpoint
180 worker-A - - process is gone: no acknowledgement; visible again after 30000 ms
30000 worker-B - - message received: msg-e3c0, receive 2
30000 worker-B - - run found: by message_id; resume_count 1
30000 worker-B - - lease granted: fencing token 42
30000 worker-B get_roof_assessment - resume from checkpoint: steps 1 to 2 are not run again
30240 worker-B search_guidance 1 attempt ok
30390 worker-B search_guidance - step done: checkpoint saved
30394 worker-B check_eligibility - step done: checkpoint saved
31794 worker-B draft_recommendation 1 attempt ok
31794 worker-B draft_recommendation - step done: checkpoint saved
31806 worker-B record_recommendation 1 attempt ok
31806 worker-B record_recommendation - step done: checkpoint saved
31818 worker-B open_review_task 1 attempt ok
31818 worker-B open_review_task - step done: checkpoint saved
31818 worker-B await_underwriter - run ended: waiting_for_human
31818 worker-B await_underwriter - step done: checkpoint saved
31818 worker-B - - acknowledged
end state:
run run-9195 state waiting_for_human resume_count 1 ended at 31818 ms (deadline 120000 ms)
recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0
Three lines carry the lesson. "run found: by message_id" is the inbox: the redelivered message led worker B to run-9195, not to a new run. "resume from checkpoint" is restore: steps 1 and 2 are skipped, so the provider is called once, where the naive workflow called it twice. And resume_count 1 records that a second process was needed, worth an alert if it keeps happening. The 30 seconds before the resume are the queue's visibility timeout in this example.
The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.
The gap between a write and its checkpoint#
A checkpoint is saved after a step completes. So there is always a moment when a step has done its work and the checkpoint does not yet say so. For a read, that gap costs one repeated read. For a write, it is the place where duplicates are born.
Scenario UW-R08 puts the crash exactly there. Worker A stores the recommendation row at 1986 ms and dies before the checkpoint after step 6 is saved. The latest checkpoint says "step 5 done". Worker B resumes, and from its point of view step 6 has not happened: it must run step 6 again. It cannot know whether the first write committed. A timeout tells you nothing about whether the write happened, and neither does a crash.
The sequence shows who holds what at each moment.
Diagram: UW-R08 · a crash between the write and its checkpoint, and the replay that saves the run
Time runs down the page. Worker A completes steps 1 to 5 and checkpoints each one. It stores the recommendation, then dies before the checkpoint for step 6. Thirty seconds later the queue hands the message to worker B. B finds the same run, loads the checkpoint that says step 5 is done, and runs step 6 again with the same key. The store returns the row it already has. Times are from the simulated clock of the example; the agent is a scripted stand-in.
- Queue to Worker A (first process): Deliver msg-02a7 (receive 1). The first delivery. Worker A creates run-3708 and takes the lease with fencing token 41.
- Worker A (first process) to Store (rows, keys, checkpoints): Steps 1 to 5 done: checkpoint after each. After every completed step the state is saved. The checkpoint after step 5 holds the model's recommendation.
- Worker A (first process) to Store (rows, keys, checkpoints): Step 6: store the recommendation, key run-3708:record_recommendation. The write commits: the row and its key are stored together.
- Worker A (first process): Crash before the checkpoint of step 6. The process dies. The store has the row, but the checkpoint still says step 5 is the last completed step. The message was never acknowledged.
- Queue to Worker B (second process): Visible again: deliver msg-02a7 (receive 2). After the visibility timeout the queue delivers the same message to another worker.
- Worker B (second process) to Store (rows, keys, checkpoints): Find the run by message_id; load its checkpoint. The inbox lookup finds run-3708, not a new run. The checkpoint says step 5 is done, so steps 1 to 5 are skipped.
- Worker B (second process) to Store (rows, keys, checkpoints): Step 6 again, same key. Worker B cannot know the first write committed, so it runs step 6 again. The key is the same because it is built from the run id and the step id only.
- Store (rows, keys, checkpoints) to Worker B (second process): Replay: the stored row, nothing written. The store has seen the key and returns REC-0001. Still one recommendation row for SUB-1004.
- Worker B (second process) to Store (rows, keys, checkpoints): Checkpoint; step 7; step 8: waiting_for_human. The run finishes normally and the message is acknowledged.
The careful workflow survives this because step 6 is idempotent: it sends the same idempotency_key, run-3708:record_recommendation, built from the run_id and the step id and from nothing else. The store has seen that key and returns the stored row. The output of UW-R08, from the resume on:
30000 worker-B - - message received: msg-02a7, receive 2
30000 worker-B - - run found: by message_id; resume_count 1
30000 worker-B - - lease granted: fencing token 42
30000 worker-B draft_recommendation - resume from checkpoint: steps 1 to 5 are not run again
30012 worker-B record_recommendation 1 attempt ok
30012 worker-B record_recommendation - replay: same key: the stored row was returned, nothing was written
30012 worker-B record_recommendation - step done: checkpoint saved
30024 worker-B open_review_task 1 attempt ok
30024 worker-B open_review_task - step done: checkpoint saved
30024 worker-B await_underwriter - run ended: waiting_for_human
Re-executing a step whose effect is already stored, and getting the stored result back, is called a replay. The naive workflow, with no checkpoint, no inbox and no key, starts a new run and writes again: two pending_review rows for SUB-1004, and the first run left in running forever (INV-1 and INV-4 broken).
This gives the rule that every checkpointed workflow depends on: any step that can run again after a resume must be idempotent. Checkpoints shrink how much repeats; they never make it zero. The key itself is taught on Idempotency; this page only applies it.
Common confusion. "Save the checkpoint before the write, then nothing repeats." Then the crash moves to the other side: the checkpoint says "step 6 done" for a write that never committed, and the resume skips it. The run ends with no recommendation row at all. There is no position for the checkpoint that removes the gap; there is only a choice of which side of it to be on. Saving after the step and making the step safe to repeat is the side that loses nothing.
Replay with a model that answers differently#
The step before the write is the model. In the example a scripted stand-in writes the same recommendation every time. A real model does not promise that: asked twice, it may word the reasons differently, or in a borderline case reach a different recommendation.
Look at what the checkpoint after step 5 holds: state.recommendation, the model's output. When worker B resumes in UW-R08, it does not ask the model again. It sends the recommendation from the checkpoint, so the replayed write carries the same content under the same key.
Switch checkpoints off and keep everything else, and the resume lab shows what changes. With a crash after step 7, the second process runs steps 1 to 5 again, calls the provider and the model a second time (two model calls, so twice the cost units of a draft in this example), and replays steps 6 and 7 by their keys. The table still ends with one row, but only because the stand-in wrote identical text both times. The example's IdempotencyStore.execute stores a fingerprint of what was written; from reading the code (not a scenario of the suite), a second draft with different text under the same key would be refused with KeyConflict, and the run would stop on a permanent error.
So the rule for a probabilistic step is: record its result once, and on every replay use the record. Determinism is not something to hope for from the model; it is something the workflow creates by never asking twice for an answer it has already been given. The same holds for anything else that changes between calls: the current time, a random value, the version of the policy fixture in force.
Under the hood. A runtime that records the outcome of each completed step durably, so that code can continue after a crash as if the crash had not happened, is said to provide durable execution. The eight steps of this example, with a checkpoint after each, are a small hand-built version. Products such as Temporal, LangGraph and AWS Step Functions offer it as a service or library; how each records progress, from its own documentation, is on Frameworks.
Try it: crash after any step, with checkpoints on or off#
The lab runs the real workflow on UW-004 for every crash point. Pick the step after which the first process dies, switch checkpoints on or off, and resume. A permanent rejection at step 7 shows the compensation of the next section.
Without JavaScript, here is one crash point, after step 2 (as in UW-R09, the lab crashes after the checkpoint), from workflow.py:simulate_resume run in a copy of the example:
| Step | Checkpoints on: process 1 | Checkpoints on: process 2 | Checkpoints off: process 1 | Checkpoints off: process 2 |
|---|---|---|---|---|
1 load_submission | ran | skipped | ran | ran |
2 get_roof_assessment | ran | skipped | ran | ran |
3 search_guidance | not reached | ran | not reached | ran |
4 check_eligibility | not reached | ran | not reached | ran |
5 draft_recommendation | not reached | ran | not reached | ran |
6 record_recommendation | not reached | ran | not reached | ran |
7 open_review_task | not reached | ran | not reached | ran |
8 await_underwriter | not reached | ran | not reached | ran |
| Provider calls in total | 1 | 2 | ||
| End | waiting_for_human, 1 row, 1 task | waiting_for_human, 1 row, 1 task |
And the compensation case, no crash, permanent rejection at step 7 (UW-R12): process 1 runs steps 1 to 5, step 6 ends as "compensated", step 7 as "failed", step 8 is not reached; the run ends in failed_compensated with 1 row, 0 pending_review, 1 withdrawn, 0 tasks.
The careful run without checkpoints still ends well here, because the inbox and the keys protect it. What it loses is work: a second provider call and, at later crash points, a second model call. Checkpoints are what make a crash cheap; keys are what make it safe.
When a later step fails for good: sagas and compensation#
Resume handles a crash, a failure that passes. Some failures do not pass. In UW-R12 the underwriter cockpit rejects the review task permanently, on every call. Step 6 has already stored a recommendation with status pending_review. Step 7 will never succeed.
Leave it there and the store holds a recommendation that no underwriter will ever see, because only a review task puts it in front of a person. That is the naive outcome: a pending_review row with no task, and a run in state failed, which is not one of the four terminal states (INV-4 broken).
A database transaction would roll back. The run is not one transaction: step 6 committed long ago in its own write, and a run that waits for a person cannot hold a database transaction open for days. What it can do is take a new action that reverses the meaning of the earlier one. That is a compensation. Here it marks the recommendation withdrawn, with a reason and a time. The row is not deleted.
A sequence of steps, each committed on its own, where every step that can be undone has a compensation that runs if a later step fails for good, is a saga. The word comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem of Princeton University, written about database transactions that run for a long time; they proposed breaking such a transaction into a sequence of smaller ones, each with a compensating transaction (the paper, as a PDF hosted by Cornell).
Diagram: UW-R12 · the saga of one run: forward steps, a permanent failure, and the compensation that runs back
Read left to right for the forward steps. Steps 1 to 5 change nothing outside the run, so they need no compensation. Step 6 writes a row, and that row has a compensation: mark it withdrawn. Step 7 is rejected for good by the underwriter cockpit, so the run turns back and compensates step 6, then ends in the named terminal state failed_compensated. Nothing is deleted.
- Forward · no side effects
- Steps 1 to 4: Reads and code. They change nothing outside the run, so there is nothing to compensate.
- Step 5 · draft: The model, a scripted stand-in here, writes the recommendation. Its output is saved in the checkpoint, not in another system.
- Forward · a write
- Step 6 · record: Stores the recommendation row with status pending_review. It can be compensated by marking the row withdrawn.
- Step 7 · open task: The underwriter cockpit refuses the review task permanently. A retry cannot help, so no retry is made.
- Back · compensation
- Mark withdrawn: The compensation of step 6. The row stays in the table with status withdrawn, a reason and a time, plus an audit event.
- End
- failed_compensated: The run ends in a state a person can read. One row, zero pending, one withdrawn, zero review tasks.
The compensation in the example, from workflow.py (method of Workflow, dedented) and store.py (method of Store, dedented). It writes; it never removes.
def compensate_record_recommendation(self) -> None:
"""The compensation of step 6: mark the recommendation `withdrawn`. It is never deleted."""
with self.tracer.span("store", "compensate_record_recommendation") as span:
span["output"] = self.env.store.withdraw(self.state["recommendation_id"], "the review task could not be opened")
self.report["record_recommendation"] = "compensated"
self.tracer.event("record_recommendation", "compensate", f"{self.state['recommendation_id']} marked withdrawn")
def withdraw(self, recommendation_id: str, reason: str) -> dict[str, Any]:
"""Mark a recommendation `withdrawn`. The row stays in the table."""
row = next(r for r in self.tables["recommendations"] if r["recommendation_id"] == recommendation_id)
row["status"], row["withdrawn_reason"], row["withdrawn_ms"] = "withdrawn", reason, self.clock.now()
self.tables["audit_events"].append({"event": "recommendations.withdraw", "id": recommendation_id,
"run_id": row.get("run_id"), "actor": AGENT, "actor_type": "agent",
"at_ms": self.clock.now()})
return row
Run UW-R12 with the careful workflow. Success means the run ends in a named terminal state with the row still present and marked; it does not mean the cockpit problem is solved, which is a matter for a person.
/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R12
1986 worker-A record_recommendation 1 attempt ok
1986 worker-A record_recommendation - step done: checkpoint saved
1998 worker-A open_review_task 1 attempt failed: Rejected: store.open_review_task: rejected for good
1998 worker-A record_recommendation - compensate: REC-0001 marked withdrawn
1998 worker-A open_review_task - run ended: failed_compensated
1998 worker-A - - acknowledged
end state:
run run-e513 state failed_compensated resume_count 0 ended at 1998 ms (deadline 120000 ms)
recommendation rows 1 (pending_review 0, withdrawn 1) · review tasks 0 · decisions 0 · dead letters 0
REC-0001 withdrawn escalate cites R-ROOF-03 evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007
Notice what was not retried. The rejection is a permanent error, so classify in errors.py returns "give up" at once: one attempt, no backoff. Retrying a rejection that will never pass only delays the compensation.
Why never delete? The row is evidence that the agent produced a recommendation, which an audit may ask about (INV-7 checks that a withdrawn recommendation is still in the table). "Mark as withdrawn with this reason" is idempotent, where a delete is one more write that can fail halfway. And a withdrawn row with a reason tells the people who must fix the cockpit what happened.
Before AI. A Java service that books a payment and then fails to ship the order has the same problem and the same answer: issue a refund (a new transaction), do not erase the payment. Inside one relational database a transaction gives you rollback; across services or across hours it does not. What is new for an agent is the length (a run can wait days for a person) and the model step in the middle, whose output must be saved so that a compensation and a replay talk about the same recommendation.
What cannot be compensated#
Compensation reverses meaning, not history. Some effects have no meaningful reverse:
| Effect | Why it cannot be undone | What to do instead |
|---|---|---|
| An email sent to the broker | Once read, it was read. A "please ignore" mail is a new message, not an undo | Put irreversible steps last, after every step that can still fail for good |
| A decision by the underwriter | Only a person writes human_decision; the agent must never reverse it | The agent's automated part ends before the decision; nothing in the run compensates a person |
| Cost units already spent on a model call | Spending is spent | Budgets and ceilings stop the spending before it happens (Concurrency) |
| Information that a person or another system has already acted on | The action happened outside the run | Compensate what the run owns; tell the owner of the rest |
The general advice that follows from the table: order the workflow so that the steps that can be compensated come first and the steps that cannot be undone come last. Microsoft's description of the saga pattern names the step after which no compensation is possible a pivot transaction, and says the steps after it must be retryable and idempotent so the saga can still reach its end (Saga design pattern). In this workflow the agent writes two things, a row that can be withdrawn and a task that, once open, belongs to the cockpit, and the decision that matters is outside the run entirely.
Terminal states: every run must end somewhere named#
A run that crashed, resumed, compensated or gave up must still end in a state that a person or a dashboard can read. A terminal state is a named end of a run after which the workflow does nothing more. This workflow has four:
| Terminal state | Meaning | Scenario |
|---|---|---|
waiting_for_human | The normal end of the automated part: one recommendation, one review task, a person decides | UW-R01, UW-R08, UW-R09 |
degraded_request_info | A dependency stayed down; the recommendation honestly asks for what is missing | UW-R03 (Breakers) |
failed_compensated | A later step failed for good; earlier steps were compensated | UW-R12 |
dead_lettered | The message could never be processed and was moved aside | UW-R11 (Queues) |
Everything else is a failure of the workflow itself: a run left in running (the first run of the naive UW-R08 and UW-R09), in failed (naive UW-R12), or in deadline_exceeded. INV-4 says every run ends in a named terminal state inside its deadline. The fault-injection suite checks it after each of its 295 runs: under the careful workflow it always holds; under the naive one it is the invariant broken most often (7 scenarios, 17 matrix runs and 121 of the 200 random plans in this example). A terminal state is a promise to the reader of the store: waiting_for_human means exactly one recommendation is waiting (INV-1).
The same idea for a coding agent#
A coding agent pushes a branch and opens a pull request. Its process dies after git push and before it records the push. A resumed agent pushes again (harmless: the same commit on the same branch) and then opens a second pull request. The fix is the same as here: checkpoint after each step, and make the writing step look for its earlier effect first, for example an open pull request for that branch. If CI rejects the change for good, the compensation is to close the pull request with a comment, not to force-delete the branch. The runtime around such an agent is the subject of https://harness.sammuthu.com, and the path from change to production of https://diff-to-deploy.sammuthu.com.
How this goes wrong#
| Mistake | What happens | Scenario or invariant |
|---|---|---|
| No checkpoint | Every crash restarts the run: repeated provider and model calls, and with no inbox a second run | Naive UW-R09 |
| Checkpoint, but write steps not idempotent | The step in the gap writes twice | Naive UW-R08, INV-1 |
| Checkpoint saved before the step it describes | A crash skips a step that never happened | (design error; the example does not offer it) |
| Model output not in the checkpoint | A replay asks the model again; the answer may differ from the one already stored | KeyConflict by code reading |
| Compensation deletes | Evidence gone; a failed delete is a new partial failure | INV-7 |
| No terminal state on failure | Runs stay running or failed; dashboards count them as in progress | INV-4 |
| Unbounded resumes | A run that crashes the worker every time is resumed for ever | Bound it as a poison message (Queues) |
What this is not. Choosing which step runs next, branching between steps and routing between agents is orchestration: https://orchestrate.sammuthu.com. This page owns what happens when a step fails: resume, replay, compensate, and end in a named state.
The four blocks below hold the detailed layer.
How often to checkpoint, and what it costsOne write per step is a choice; the right grain follows the expensive and the irreversible steps
A checkpoint is a write to durable storage. After every step it costs one write per step per run; in this example eight per run, small next to the model call. The benefit of a checkpoint is the work it saves on a crash, which is the cost of the steps since the previous one. So checkpoint immediately after anything expensive (the model call, a slow provider) and immediately after anything that must not be repeated without a key. Cheap, pure steps (the rule check, step 4) could share a checkpoint with their neighbour without much loss.
Size matters more than count. The checkpoint here copies every tool result, including the passages from the guidance search. In a real system with long retrieved documents, keep references (the evidence ids and a content hash) in the checkpoint and the documents in their own store, and check on resume that the referenced version still exists. A checkpoint that silently points at a newer version of a document breaks the determinism argument above.
Closing the gap: writing the effect and the checkpoint togetherWhen the step's write and the checkpoint live in one database, one transaction can hold both
The gap between a write and its checkpoint exists because they are two writes. If the recommendation row and the checkpoint live in the same database, one transaction can commit both, and a crash leaves either both or neither. This is the same move as the outbox on Idempotency: put the record of "what I did" in the same commit as the effect.
It removes the duplicate risk for that step, not the need for idempotency in general. The review task is opened in another system (the cockpit), so no local transaction can include it. A design should still assume that any step touching another system can be repeated, and keep the key. Note also that the example's store is plain Python, so "same commit" is trivially true there; the README says so in its list of what the example leaves out.
When the compensation itself failsA compensation is a step: it needs retries, a key and a state of its own
Compensation is ordinary code that writes to ordinary systems, so it can time out, crash halfway or be rejected. Treat it as a step: checkpoint before it, make it idempotent (marking withdrawn twice is harmless; "decrement the count" twice is not), retry its transient errors with backoff, and give the run a state for the case where compensation itself cannot finish, so a person is asked. Microsoft's saga page lists the same limitation: compensating transactions might not always succeed.
A second subtlety is isolation. Between step 6 and the compensation, the recommendation was visible as pending_review. Anything that read it in that window (a dashboard, a count of open work) saw a state that was later reversed. Here the window is 12 ms of simulated time; in a run that waits on a slow cockpit it could be minutes. The saga paper is explicit that other transactions might see the effects of a partially executed saga. If readers must never act on a recommendation whose task is not open, make the review task the thing they read, not the row.
Resuming on a different version of the codeA checkpoint written by version N may be read by version N+1
A crash during a deployment is common, because a deployment is often what kills the process. The resumed run may then execute on new code: a new prompt, a new step list, a new policy version. A checkpoint that says "step 5 done" means nothing if version N+1 renamed step 5. Store the workflow version in the run, and decide per change whether in-flight runs finish on the old version or are migrated. That decision, version skew, is taught on Deployment. For this page, the minimum is that restore must refuse a checkpoint it does not understand, rather than guess.
Checklist#
| Question | Good answer |
|---|---|
| Can a new process find the run of a redelivered message? | Yes: the run is looked up by message_id, with a unique index |
| Is a checkpoint saved after every completed step that is expensive or writes? | Yes, after the step, never before it |
| Does the checkpoint hold the model's output and every tool result the next step needs? | Yes, as data, with the trace_id and resume_count |
| Is every step that can run again after a resume idempotent? | Yes: writes carry <run_id>:<step id> keys |
| Does a replay ever ask the model again for an answer already given? | No |
| Does every step that can fail for good have a compensation, or come after the last one? | Yes; irreversible steps come last |
| Does compensation mark, never delete? | Yes, with a reason and a time |
| Does every path end in a named terminal state inside the deadline? | Yes, and a test checks it for each fault |
| Is the number of resumes per run bounded and visible? | Yes |
In one breath#
A retry cannot help a process that died, so a long run saves a checkpoint after every completed step and a new process resumes it by run_id. The step between a write and its checkpoint will run twice, so it must be idempotent, and the model's answer must come from the checkpoint, never from a second call. When a later step fails for good, a compensation marks the earlier effect withdrawn instead of deleting it, and every run ends in a named terminal state.
Next: when the dependency itself is down#
The insurance run now survives a crash, a redelivery and a permanent rejection, with one recommendation row and a named end every time. What it cannot yet survive well is a dependency that stays down: each run retries the roof-data provider three times before giving up, and a hundred runs at once turn one failing provider into a flood of doomed calls. Breakers is about stopping those calls early and giving a smaller, honest answer in place of none.