All chapters
  1. Start hereEvery step can fail. The outcome still has to be acceptable
  2. Foundations · 1Available, correct, durable, resilient: four qualities that are not the same
  3. Foundations · 2Failure modes: what breaks around an agent, and how it shows
  4. Mechanisms · 3Timeouts, deadlines and retries: trying again without making it worse
  5. Mechanisms · 4Idempotency: when the same request arrives twice
  6. Mechanisms · 5Queues: delivery semantics, dead letters and poison messages
  7. Mechanisms · 6Checkpoint, resume and sagas: finishing what a crash interrupted
  8. Mechanisms · 7Circuit breakers, load shedding and graceful degradation
  9. Mechanisms · 8Concurrency: rate limits, budgets and distributed locks
  10. Operations · 9SLOs: saying how reliable is reliable enough
  11. Operations · 10Deployment: canary, blue-green, feature flags and rollback
  12. Operations · 11Model fallback and provider failover: what a second route costs in quality
  13. Operations · 12Fault injection: breaking it on purpose, before production does
  14. Operations · 13Incidents: response, runbooks, postmortems and the case that joins the golden set
  15. Reference · 14Frameworks and standards: what each one keeps reliable for you
  16. Reference · 15Advanced: the human step, cost ceilings and capacity
  17. Reference · 16The insurance lab: one quote, fifteen ways to fail, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Mechanisms · 6

Checkpoint, resume and sagas: finishing what a crash interrupted

A run that dies in the middle should continue from its last saved step, not from the start. Checkpoints, resume, and compensation for the steps that cannot be undone.

What you will understand
Checkpoint and resume
Why a run must save its state after every completed step, and how a new process finds the run and continues where the dead one stopped.
Replay
Why the step between a write and its checkpoint runs twice, and why the model's answer must be saved, never asked for again.
Compensation
What to do when a later step fails for good: a new action that reverses the meaning of an earlier one, never a delete.

The underwriting agent has started a run for submission SUB-1004, the office building whose application says the roof is 9 years old while the roof-data provider says 22. The expected result is a recommendation to escalate, citing R-ROOF-03. The model recommends; a person decides.

In scenario UW-R09 the worker process dies 180 ms into the run. It has validated the submission and it has the roof assessment in memory. Then the machine is gone. Nothing threw an exception that a retry loop could catch, because the code that would have caught it is gone too.

Thirty seconds later the queue hands the same message to another worker, as Queues explains. Here is what the naive workflow does with it:

    180  worker-A  get_roof_assessment      -  PROCESS CRASH: after the step, after checkpoint
    180  worker-A  -                        -  process is gone: no acknowledgement; visible again after 30000 ms
  30000  worker-B  -                        -  message received: msg-e3c0, receive 2
  30000  worker-B  -                        -  run created: trace_id e267a8b82db7b811b6b2f70312632f9e
  30000  worker-B  -                        -  lease granted: fencing token 42
  30000  worker-B  load_submission          -  step done: no checkpoint
  30180  worker-B  get_roof_assessment      1  attempt ok

(The crash line says "after checkpoint" because that is where the fault plan places the crash; the naive workflow saves no checkpoint, so there is nothing to resume from.)

Worker B knows nothing about the first run. It creates a second run, calls the provider a second time and does all the work again. In this example the end result still has one recommendation row, so the damage looks small: one extra provider call. But the first run is still marked running and will never finish, and the work it did is thrown away. Move the crash a few steps later, after the recommendation was written, and the same "start over" leaves two recommendation rows for one submission (UW-R08, below).

This page is about the two things a run needs so that a crash costs one step and not the whole run: a saved position, and a way to undo the meaning of a step that cannot be finished.

Plain English. A retry tries one step again inside a process that is still alive. A crash takes the process away. What survives a crash is only what was written somewhere outside the process before it died. A retry is not a recovery.

Why a dead process cannot retry its way out#

Retries handle an error that the running code can see: a timeout, a 429, a failed write. The loop that retries lives in the memory of the worker. When the worker crashes (the machine restarts, the container is killed during a deployment, the process runs out of memory), the loop, the attempt counter and every intermediate result disappear with it.

For a short request the caller retries the whole request and loses a second of work. An agent run is long, calls a model whose answer is not the same every time, and writes to systems that remember (the index explains all three). Starting over repeats the slow part, asks the model again for an answer that may differ, and repeats writes that already happened.

So the run itself must leave a trail outside the process. That trail has a name.

A checkpoint is the saved state of a run after a completed step#

When the worker finishes step 2 of SUB-1004, it holds the validated submission and the provider's answer (roof 22 years old). If it writes those two things to the store before it starts step 3, a later process can read them back and start at step 3. Nothing about step 1 or step 2 has to happen again.

That saved record is a checkpoint: the saved state of a run after a completed step. In this example one checkpoint is kept per run, the latest one, and it is saved after every completed step of the eight in the workflow.

A checkpoint belongs to a run, and a run needs a name that a new process can find. The run id (run_id, for example run-9195) names one execution of the workflow for one submission. It is not the submission (SUB-1004 stays the same if a person asks for a fresh run) and it is not an attempt (attempt_number counts tries of one step inside a run). A redelivered message must lead to the same run, so the run is looked up by the message's message_id; Queues covers that lookup, the inbox.

What goes into the checkpoint follows from one question: what would the next step need, if this process vanished now? In this example the answer is the shape below, from the example's README.

FieldExample (UW-R15)Why it is saved
run_idrun-7f3aThe key a new process uses to find the checkpoint
step_iddraft_recommendationThe last completed step; the resume starts at the next one
state.submission, state.roof, state.passages, state.facts, state.checkthe validated input and every tool resultSo steps 1 to 4 never run again, and the provider is not called again
state.recommendation, state.model_route, state.degradedthe model's output and how it was producedSo the model is never asked again for an answer it already gave (see replay, below)
state.evidence_idsEV-ROOF-0412, EV-GUIDE-0031, ...So the stored recommendation cites only evidence that exists in this run (INV-6)
trace_id32 hex charactersSo the resumed run writes into the same trace: one trace per run
resume_count0, then 1How many times a new process picked the run up

What stays out matters as much. A checkpoint holds data, not live objects: no open connection, no client, no lock. It holds no secret that the next step does not need. And it holds no attempt counter: attempt_number starts again at 1 in a new process, which is why the pair (resume_count, attempt_number) names an attempt in the trace.

What this is not. Which state is worth keeping, and the checkpoint as a data structure in its own right, belong to memory design: https://memory.sammuthu.com. This page uses a checkpoint for one purpose only, to recover a run.

Resume: find the run, load the checkpoint, continue#

To resume is to let a new process continue a run from its checkpoint. It is not a retry (a retry is another attempt of one step in the same process) and it is not a restart (a restart begins at step 1).

The example does it in two small methods of Workflow in workflow.py. restore loads the checkpoint and returns the index of the first step that still has to run; execute runs the steps from there and saves a checkpoint after each one. They are shown here dedented from the class, otherwise verbatim. Read them for the order of operations: run the step, then the crash point that the fault plan can trigger, then the checkpoint.

def restore(self) -> int:
    """Load the checkpoint, if any. Returns the index of the first step that still has to run."""
    checkpoint = self.env.checkpoints.load(self.run["run_id"]) if self.env.flags["checkpoint"] else None
    if checkpoint is None:
        return 0
    self.state = checkpoint["state"]
    done = STEPS.index(checkpoint["step_id"]) + 1
    self.report.update({step_id: "skipped" for step_id in STEPS[:done]})
    self.tracer.event(checkpoint["step_id"], "resume from checkpoint", f"steps 1 to {done} are not run again")
    return done

def execute(self, start: int) -> None:
    """Run the steps from `start`. After each completed step: save a checkpoint."""
    with self.tracer.span("workflow", "underwrite_submission", {"message_id": self.run["message_id"]}):
        for step_id in STEPS[start:]:
            self.current_step = step_id
            with self.tracer.span("step", step_id):
                getattr(self, step_id)()
            self.report.setdefault(step_id, "ran")
            if self.run["state"] != "running" and step_id != "await_underwriter":
                return                                            # stopped early (step 7 was rejected)
            self._crash_point(step_id, "before_checkpoint")
            self._save_checkpoint(step_id)
            self.tracer.event(step_id, "step done", "checkpoint saved" if self.env.flags["checkpoint"] else "no checkpoint")
            self._crash_point(step_id, "after_checkpoint")

Now the same crash as in the opening, UW-R09, with the careful workflow. The command runs the scenario in a copy of examples/uw-reliability. A successful run proves that, in this simulation, the second process skips the two completed steps and calls the provider once in total. It does not prove anything about a real store's durability; the store here is memory inside one Python process.

/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R09

The timeline and the end state, verbatim (the header and the seven invariant lines are left out; all seven hold):

   t_ms  worker    step                   att  event
      0  queue     -                        -  message sent: msg-e3c0
      0  worker-A  -                        -  message received: msg-e3c0, receive 1
      0  worker-A  -                        -  run created: trace_id 33b54151e808a2b70642c3cd73a6e1f2
      0  worker-A  -                        -  lease granted: fencing token 41
      0  worker-A  load_submission          -  step done: checkpoint saved
    180  worker-A  get_roof_assessment      1  attempt ok
    180  worker-A  get_roof_assessment      -  step done: checkpoint saved
    180  worker-A  get_roof_assessment      -  PROCESS CRASH: after the step, after checkpoint
    180  worker-A  -                        -  process is gone: no acknowledgement; visible again after 30000 ms
  30000  worker-B  -                        -  message received: msg-e3c0, receive 2
  30000  worker-B  -                        -  run found: by message_id; resume_count 1
  30000  worker-B  -                        -  lease granted: fencing token 42
  30000  worker-B  get_roof_assessment      -  resume from checkpoint: steps 1 to 2 are not run again
  30240  worker-B  search_guidance          1  attempt ok
  30390  worker-B  search_guidance          -  step done: checkpoint saved
  30394  worker-B  check_eligibility        -  step done: checkpoint saved
  31794  worker-B  draft_recommendation     1  attempt ok
  31794  worker-B  draft_recommendation     -  step done: checkpoint saved
  31806  worker-B  record_recommendation    1  attempt ok
  31806  worker-B  record_recommendation    -  step done: checkpoint saved
  31818  worker-B  open_review_task         1  attempt ok
  31818  worker-B  open_review_task         -  step done: checkpoint saved
  31818  worker-B  await_underwriter        -  run ended: waiting_for_human
  31818  worker-B  await_underwriter        -  step done: checkpoint saved
  31818  worker-B  -                        -  acknowledged
end state:
  run run-9195  state waiting_for_human  resume_count 1  ended at 31818 ms (deadline 120000 ms)
  recommendation rows 1 (pending_review 1, withdrawn 0) · review tasks 1 · decisions 0 · dead letters 0

Three lines carry the lesson. "run found: by message_id" is the inbox: the redelivered message led worker B to run-9195, not to a new run. "resume from checkpoint" is restore: steps 1 and 2 are skipped, so the provider is called once, where the naive workflow called it twice. And resume_count 1 records that a second process was needed, worth an alert if it keeps happening. The 30 seconds before the resume are the queue's visibility timeout in this example.

The agent here is a scripted stand-in, and every fault is injected on purpose, so these results show how the mechanism works, not how reliable any model or provider is.

The gap between a write and its checkpoint#

A checkpoint is saved after a step completes. So there is always a moment when a step has done its work and the checkpoint does not yet say so. For a read, that gap costs one repeated read. For a write, it is the place where duplicates are born.

Scenario UW-R08 puts the crash exactly there. Worker A stores the recommendation row at 1986 ms and dies before the checkpoint after step 6 is saved. The latest checkpoint says "step 5 done". Worker B resumes, and from its point of view step 6 has not happened: it must run step 6 again. It cannot know whether the first write committed. A timeout tells you nothing about whether the write happened, and neither does a crash.

The sequence shows who holds what at each moment.

Diagram: UW-R08 · a crash between the write and its checkpoint, and the replay that saves the run

Time runs down the page. Worker A completes steps 1 to 5 and checkpoints each one. It stores the recommendation, then dies before the checkpoint for step 6. Thirty seconds later the queue hands the message to worker B. B finds the same run, loads the checkpoint that says step 5 is done, and runs step 6 again with the same key. The store returns the row it already has. Times are from the simulated clock of the example; the agent is a scripted stand-in.

  1. Queue to Worker A (first process): Deliver msg-02a7 (receive 1). The first delivery. Worker A creates run-3708 and takes the lease with fencing token 41.
  2. Worker A (first process) to Store (rows, keys, checkpoints): Steps 1 to 5 done: checkpoint after each. After every completed step the state is saved. The checkpoint after step 5 holds the model's recommendation.
  3. Worker A (first process) to Store (rows, keys, checkpoints): Step 6: store the recommendation, key run-3708:record_recommendation. The write commits: the row and its key are stored together.
  4. Worker A (first process): Crash before the checkpoint of step 6. The process dies. The store has the row, but the checkpoint still says step 5 is the last completed step. The message was never acknowledged.
  5. Queue to Worker B (second process): Visible again: deliver msg-02a7 (receive 2). After the visibility timeout the queue delivers the same message to another worker.
  6. Worker B (second process) to Store (rows, keys, checkpoints): Find the run by message_id; load its checkpoint. The inbox lookup finds run-3708, not a new run. The checkpoint says step 5 is done, so steps 1 to 5 are skipped.
  7. Worker B (second process) to Store (rows, keys, checkpoints): Step 6 again, same key. Worker B cannot know the first write committed, so it runs step 6 again. The key is the same because it is built from the run id and the step id only.
  8. Store (rows, keys, checkpoints) to Worker B (second process): Replay: the stored row, nothing written. The store has seen the key and returns REC-0001. Still one recommendation row for SUB-1004.
  9. Worker B (second process) to Store (rows, keys, checkpoints): Checkpoint; step 7; step 8: waiting_for_human. The run finishes normally and the message is acknowledged.

The careful workflow survives this because step 6 is idempotent: it sends the same idempotency_key, run-3708:record_recommendation, built from the run_id and the step id and from nothing else. The store has seen that key and returns the stored row. The output of UW-R08, from the resume on:

  30000  worker-B  -                        -  message received: msg-02a7, receive 2
  30000  worker-B  -                        -  run found: by message_id; resume_count 1
  30000  worker-B  -                        -  lease granted: fencing token 42
  30000  worker-B  draft_recommendation     -  resume from checkpoint: steps 1 to 5 are not run again
  30012  worker-B  record_recommendation    1  attempt ok
  30012  worker-B  record_recommendation    -  replay: same key: the stored row was returned, nothing was written
  30012  worker-B  record_recommendation    -  step done: checkpoint saved
  30024  worker-B  open_review_task         1  attempt ok
  30024  worker-B  open_review_task         -  step done: checkpoint saved
  30024  worker-B  await_underwriter        -  run ended: waiting_for_human

Re-executing a step whose effect is already stored, and getting the stored result back, is called a replay. The naive workflow, with no checkpoint, no inbox and no key, starts a new run and writes again: two pending_review rows for SUB-1004, and the first run left in running forever (INV-1 and INV-4 broken).

This gives the rule that every checkpointed workflow depends on: any step that can run again after a resume must be idempotent. Checkpoints shrink how much repeats; they never make it zero. The key itself is taught on Idempotency; this page only applies it.

Common confusion. "Save the checkpoint before the write, then nothing repeats." Then the crash moves to the other side: the checkpoint says "step 6 done" for a write that never committed, and the resume skips it. The run ends with no recommendation row at all. There is no position for the checkpoint that removes the gap; there is only a choice of which side of it to be on. Saving after the step and making the step safe to repeat is the side that loses nothing.

Replay with a model that answers differently#

The step before the write is the model. In the example a scripted stand-in writes the same recommendation every time. A real model does not promise that: asked twice, it may word the reasons differently, or in a borderline case reach a different recommendation.

Look at what the checkpoint after step 5 holds: state.recommendation, the model's output. When worker B resumes in UW-R08, it does not ask the model again. It sends the recommendation from the checkpoint, so the replayed write carries the same content under the same key.

Switch checkpoints off and keep everything else, and the resume lab shows what changes. With a crash after step 7, the second process runs steps 1 to 5 again, calls the provider and the model a second time (two model calls, so twice the cost units of a draft in this example), and replays steps 6 and 7 by their keys. The table still ends with one row, but only because the stand-in wrote identical text both times. The example's IdempotencyStore.execute stores a fingerprint of what was written; from reading the code (not a scenario of the suite), a second draft with different text under the same key would be refused with KeyConflict, and the run would stop on a permanent error.

So the rule for a probabilistic step is: record its result once, and on every replay use the record. Determinism is not something to hope for from the model; it is something the workflow creates by never asking twice for an answer it has already been given. The same holds for anything else that changes between calls: the current time, a random value, the version of the policy fixture in force.

Under the hood. A runtime that records the outcome of each completed step durably, so that code can continue after a crash as if the crash had not happened, is said to provide durable execution. The eight steps of this example, with a checkpoint after each, are a small hand-built version. Products such as Temporal, LangGraph and AWS Step Functions offer it as a service or library; how each records progress, from its own documentation, is on Frameworks.

Try it: crash after any step, with checkpoints on or off#

The lab runs the real workflow on UW-004 for every crash point. Pick the step after which the first process dies, switch checkpoints on or off, and resume. A permanent rejection at step 7 shows the compensation of the next section.

Without JavaScript, here is one crash point, after step 2 (as in UW-R09, the lab crashes after the checkpoint), from workflow.py:simulate_resume run in a copy of the example:

StepCheckpoints on: process 1Checkpoints on: process 2Checkpoints off: process 1Checkpoints off: process 2
1 load_submissionranskippedranran
2 get_roof_assessmentranskippedranran
3 search_guidancenot reachedrannot reachedran
4 check_eligibilitynot reachedrannot reachedran
5 draft_recommendationnot reachedrannot reachedran
6 record_recommendationnot reachedrannot reachedran
7 open_review_tasknot reachedrannot reachedran
8 await_underwriternot reachedrannot reachedran
Provider calls in total12
Endwaiting_for_human, 1 row, 1 taskwaiting_for_human, 1 row, 1 task

And the compensation case, no crash, permanent rejection at step 7 (UW-R12): process 1 runs steps 1 to 5, step 6 ends as "compensated", step 7 as "failed", step 8 is not reached; the run ends in failed_compensated with 1 row, 0 pending_review, 1 withdrawn, 0 tasks.

The careful run without checkpoints still ends well here, because the inbox and the keys protect it. What it loses is work: a second provider call and, at later crash points, a second model call. Checkpoints are what make a crash cheap; keys are what make it safe.

When a later step fails for good: sagas and compensation#

Resume handles a crash, a failure that passes. Some failures do not pass. In UW-R12 the underwriter cockpit rejects the review task permanently, on every call. Step 6 has already stored a recommendation with status pending_review. Step 7 will never succeed.

Leave it there and the store holds a recommendation that no underwriter will ever see, because only a review task puts it in front of a person. That is the naive outcome: a pending_review row with no task, and a run in state failed, which is not one of the four terminal states (INV-4 broken).

A database transaction would roll back. The run is not one transaction: step 6 committed long ago in its own write, and a run that waits for a person cannot hold a database transaction open for days. What it can do is take a new action that reverses the meaning of the earlier one. That is a compensation. Here it marks the recommendation withdrawn, with a reason and a time. The row is not deleted.

A sequence of steps, each committed on its own, where every step that can be undone has a compensation that runs if a later step fails for good, is a saga. The word comes from a 1987 paper by Hector Garcia-Molina and Kenneth Salem of Princeton University, written about database transactions that run for a long time; they proposed breaking such a transaction into a sequence of smaller ones, each with a compensating transaction (the paper, as a PDF hosted by Cornell).

Diagram: UW-R12 · the saga of one run: forward steps, a permanent failure, and the compensation that runs back

Read left to right for the forward steps. Steps 1 to 5 change nothing outside the run, so they need no compensation. Step 6 writes a row, and that row has a compensation: mark it withdrawn. Step 7 is rejected for good by the underwriter cockpit, so the run turns back and compensates step 6, then ends in the named terminal state failed_compensated. Nothing is deleted.

  1. Forward · no side effects
    1. Steps 1 to 4: Reads and code. They change nothing outside the run, so there is nothing to compensate.
    2. Step 5 · draft: The model, a scripted stand-in here, writes the recommendation. Its output is saved in the checkpoint, not in another system.
  2. Forward · a write
    1. Step 6 · record: Stores the recommendation row with status pending_review. It can be compensated by marking the row withdrawn.
    2. Step 7 · open task: The underwriter cockpit refuses the review task permanently. A retry cannot help, so no retry is made.
  3. Back · compensation
    1. Mark withdrawn: The compensation of step 6. The row stays in the table with status withdrawn, a reason and a time, plus an audit event.
  4. End
    1. failed_compensated: The run ends in a state a person can read. One row, zero pending, one withdrawn, zero review tasks.

The compensation in the example, from workflow.py (method of Workflow, dedented) and store.py (method of Store, dedented). It writes; it never removes.

def compensate_record_recommendation(self) -> None:
    """The compensation of step 6: mark the recommendation `withdrawn`. It is never deleted."""
    with self.tracer.span("store", "compensate_record_recommendation") as span:
        span["output"] = self.env.store.withdraw(self.state["recommendation_id"], "the review task could not be opened")
    self.report["record_recommendation"] = "compensated"
    self.tracer.event("record_recommendation", "compensate", f"{self.state['recommendation_id']} marked withdrawn")

def withdraw(self, recommendation_id: str, reason: str) -> dict[str, Any]:
    """Mark a recommendation `withdrawn`. The row stays in the table."""
    row = next(r for r in self.tables["recommendations"] if r["recommendation_id"] == recommendation_id)
    row["status"], row["withdrawn_reason"], row["withdrawn_ms"] = "withdrawn", reason, self.clock.now()
    self.tables["audit_events"].append({"event": "recommendations.withdraw", "id": recommendation_id,
                                        "run_id": row.get("run_id"), "actor": AGENT, "actor_type": "agent",
                                        "at_ms": self.clock.now()})
    return row

Run UW-R12 with the careful workflow. Success means the run ends in a named terminal state with the row still present and marked; it does not mean the cockpit problem is solved, which is a matter for a person.

/opt/homebrew/bin/python3 run_scenario.py --scenario UW-R12
   1986  worker-A  record_recommendation    1  attempt ok
   1986  worker-A  record_recommendation    -  step done: checkpoint saved
   1998  worker-A  open_review_task         1  attempt failed: Rejected: store.open_review_task: rejected for good
   1998  worker-A  record_recommendation    -  compensate: REC-0001 marked withdrawn
   1998  worker-A  open_review_task         -  run ended: failed_compensated
   1998  worker-A  -                        -  acknowledged
end state:
  run run-e513  state failed_compensated  resume_count 0  ended at 1998 ms (deadline 120000 ms)
  recommendation rows 1 (pending_review 0, withdrawn 1) · review tasks 0 · decisions 0 · dead letters 0
  REC-0001  withdrawn  escalate  cites R-ROOF-03  evidence EV-ROOF-0412, EV-GUIDE-0031, EV-RULE-0007

Notice what was not retried. The rejection is a permanent error, so classify in errors.py returns "give up" at once: one attempt, no backoff. Retrying a rejection that will never pass only delays the compensation.

Why never delete? The row is evidence that the agent produced a recommendation, which an audit may ask about (INV-7 checks that a withdrawn recommendation is still in the table). "Mark as withdrawn with this reason" is idempotent, where a delete is one more write that can fail halfway. And a withdrawn row with a reason tells the people who must fix the cockpit what happened.

Before AI. A Java service that books a payment and then fails to ship the order has the same problem and the same answer: issue a refund (a new transaction), do not erase the payment. Inside one relational database a transaction gives you rollback; across services or across hours it does not. What is new for an agent is the length (a run can wait days for a person) and the model step in the middle, whose output must be saved so that a compensation and a replay talk about the same recommendation.

What cannot be compensated#

Compensation reverses meaning, not history. Some effects have no meaningful reverse:

EffectWhy it cannot be undoneWhat to do instead
An email sent to the brokerOnce read, it was read. A "please ignore" mail is a new message, not an undoPut irreversible steps last, after every step that can still fail for good
A decision by the underwriterOnly a person writes human_decision; the agent must never reverse itThe agent's automated part ends before the decision; nothing in the run compensates a person
Cost units already spent on a model callSpending is spentBudgets and ceilings stop the spending before it happens (Concurrency)
Information that a person or another system has already acted onThe action happened outside the runCompensate what the run owns; tell the owner of the rest

The general advice that follows from the table: order the workflow so that the steps that can be compensated come first and the steps that cannot be undone come last. Microsoft's description of the saga pattern names the step after which no compensation is possible a pivot transaction, and says the steps after it must be retryable and idempotent so the saga can still reach its end (Saga design pattern). In this workflow the agent writes two things, a row that can be withdrawn and a task that, once open, belongs to the cockpit, and the decision that matters is outside the run entirely.

Terminal states: every run must end somewhere named#

A run that crashed, resumed, compensated or gave up must still end in a state that a person or a dashboard can read. A terminal state is a named end of a run after which the workflow does nothing more. This workflow has four:

Terminal stateMeaningScenario
waiting_for_humanThe normal end of the automated part: one recommendation, one review task, a person decidesUW-R01, UW-R08, UW-R09
degraded_request_infoA dependency stayed down; the recommendation honestly asks for what is missingUW-R03 (Breakers)
failed_compensatedA later step failed for good; earlier steps were compensatedUW-R12
dead_letteredThe message could never be processed and was moved asideUW-R11 (Queues)

Everything else is a failure of the workflow itself: a run left in running (the first run of the naive UW-R08 and UW-R09), in failed (naive UW-R12), or in deadline_exceeded. INV-4 says every run ends in a named terminal state inside its deadline. The fault-injection suite checks it after each of its 295 runs: under the careful workflow it always holds; under the naive one it is the invariant broken most often (7 scenarios, 17 matrix runs and 121 of the 200 random plans in this example). A terminal state is a promise to the reader of the store: waiting_for_human means exactly one recommendation is waiting (INV-1).

The same idea for a coding agent#

A coding agent pushes a branch and opens a pull request. Its process dies after git push and before it records the push. A resumed agent pushes again (harmless: the same commit on the same branch) and then opens a second pull request. The fix is the same as here: checkpoint after each step, and make the writing step look for its earlier effect first, for example an open pull request for that branch. If CI rejects the change for good, the compensation is to close the pull request with a comment, not to force-delete the branch. The runtime around such an agent is the subject of https://harness.sammuthu.com, and the path from change to production of https://diff-to-deploy.sammuthu.com.

How this goes wrong#

MistakeWhat happensScenario or invariant
No checkpointEvery crash restarts the run: repeated provider and model calls, and with no inbox a second runNaive UW-R09
Checkpoint, but write steps not idempotentThe step in the gap writes twiceNaive UW-R08, INV-1
Checkpoint saved before the step it describesA crash skips a step that never happened(design error; the example does not offer it)
Model output not in the checkpointA replay asks the model again; the answer may differ from the one already storedKeyConflict by code reading
Compensation deletesEvidence gone; a failed delete is a new partial failureINV-7
No terminal state on failureRuns stay running or failed; dashboards count them as in progressINV-4
Unbounded resumesA run that crashes the worker every time is resumed for everBound it as a poison message (Queues)
What this is not. Choosing which step runs next, branching between steps and routing between agents is orchestration: https://orchestrate.sammuthu.com. This page owns what happens when a step fails: resume, replay, compensate, and end in a named state.

The four blocks below hold the detailed layer.

How often to checkpoint, and what it costsOne write per step is a choice; the right grain follows the expensive and the irreversible steps

A checkpoint is a write to durable storage. After every step it costs one write per step per run; in this example eight per run, small next to the model call. The benefit of a checkpoint is the work it saves on a crash, which is the cost of the steps since the previous one. So checkpoint immediately after anything expensive (the model call, a slow provider) and immediately after anything that must not be repeated without a key. Cheap, pure steps (the rule check, step 4) could share a checkpoint with their neighbour without much loss.

Size matters more than count. The checkpoint here copies every tool result, including the passages from the guidance search. In a real system with long retrieved documents, keep references (the evidence ids and a content hash) in the checkpoint and the documents in their own store, and check on resume that the referenced version still exists. A checkpoint that silently points at a newer version of a document breaks the determinism argument above.

Closing the gap: writing the effect and the checkpoint togetherWhen the step's write and the checkpoint live in one database, one transaction can hold both

The gap between a write and its checkpoint exists because they are two writes. If the recommendation row and the checkpoint live in the same database, one transaction can commit both, and a crash leaves either both or neither. This is the same move as the outbox on Idempotency: put the record of "what I did" in the same commit as the effect.

It removes the duplicate risk for that step, not the need for idempotency in general. The review task is opened in another system (the cockpit), so no local transaction can include it. A design should still assume that any step touching another system can be repeated, and keep the key. Note also that the example's store is plain Python, so "same commit" is trivially true there; the README says so in its list of what the example leaves out.

When the compensation itself failsA compensation is a step: it needs retries, a key and a state of its own

Compensation is ordinary code that writes to ordinary systems, so it can time out, crash halfway or be rejected. Treat it as a step: checkpoint before it, make it idempotent (marking withdrawn twice is harmless; "decrement the count" twice is not), retry its transient errors with backoff, and give the run a state for the case where compensation itself cannot finish, so a person is asked. Microsoft's saga page lists the same limitation: compensating transactions might not always succeed.

A second subtlety is isolation. Between step 6 and the compensation, the recommendation was visible as pending_review. Anything that read it in that window (a dashboard, a count of open work) saw a state that was later reversed. Here the window is 12 ms of simulated time; in a run that waits on a slow cockpit it could be minutes. The saga paper is explicit that other transactions might see the effects of a partially executed saga. If readers must never act on a recommendation whose task is not open, make the review task the thing they read, not the row.

Resuming on a different version of the codeA checkpoint written by version N may be read by version N+1

A crash during a deployment is common, because a deployment is often what kills the process. The resumed run may then execute on new code: a new prompt, a new step list, a new policy version. A checkpoint that says "step 5 done" means nothing if version N+1 renamed step 5. Store the workflow version in the run, and decide per change whether in-flight runs finish on the old version or are migrated. That decision, version skew, is taught on Deployment. For this page, the minimum is that restore must refuse a checkpoint it does not understand, rather than guess.

Checklist#

QuestionGood answer
Can a new process find the run of a redelivered message?Yes: the run is looked up by message_id, with a unique index
Is a checkpoint saved after every completed step that is expensive or writes?Yes, after the step, never before it
Does the checkpoint hold the model's output and every tool result the next step needs?Yes, as data, with the trace_id and resume_count
Is every step that can run again after a resume idempotent?Yes: writes carry <run_id>:<step id> keys
Does a replay ever ask the model again for an answer already given?No
Does every step that can fail for good have a compensation, or come after the last one?Yes; irreversible steps come last
Does compensation mark, never delete?Yes, with a reason and a time
Does every path end in a named terminal state inside the deadline?Yes, and a test checks it for each fault
Is the number of resumes per run bounded and visible?Yes

In one breath#

A retry cannot help a process that died, so a long run saves a checkpoint after every completed step and a new process resumes it by run_id. The step between a write and its checkpoint will run twice, so it must be idempotent, and the model's answer must come from the checkpoint, never from a second call. When a later step fails for good, a compensation marks the earlier effect withdrawn instead of deleting it, and every run ends in a named terminal state.

Next: when the dependency itself is down#

The insurance run now survives a crash, a redelivery and a permanent rejection, with one recommendation row and a named end every time. What it cannot yet survive well is a dependency that stays down: each run retries the roof-data provider three times before giving up, and a hundred runs at once turn one failing provider into a flood of doomed calls. Breakers is about stopping those calls early and giving a smaller, honest answer in place of none.