The AI Incident Postmortem Template, What to Capture
Pull-quote: “A service incident ends with a stack trace. An agent incident ends with a transcript that reads perfectly well until you notice the world it describes does not exist.”
Why this matters
A failed service leaves an exception, a stack trace, and a timestamp. A misbehaving agent usually leaves a completed run: status 200, fluent prose, and a wrong action already taken. The update went to the wrong record, the summary cited a document that says the opposite, the tool was called with arguments the model invented. The SRE postmortem discipline transfers: blameless, written, reviewed, actioned. The artifacts do not. Run an agent postmortem with a service template and the finding comes back “the model behaved unexpectedly,” which is a weather report, not a cause. The template below is the one to write before the incident, because most of what it captures cannot be reconstructed afterward.
The five artifacts
Capture five things for every AI incident. Each answers a question the review will ask.
| Artifact | What it contains | Question it answers |
|---|---|---|
| 1. Trigger | The full input: user message plus everything assembled around it, retrieved documents, memory entries, system prompt version | What did the model actually see? |
| 2. Trajectory | The run tree: every model call, tool call, retrieval, and retry, in order, with timestamps and statuses | What did it do, and in what order? |
| 3. World state | Versions of everything: model ID, prompt template, tool schemas, index snapshot, flags and thresholds | Could we rebuild this exact run? |
| 4. Decision points | Where the agent chose between alternatives, and which guardrails evaluated, fired, or were bypassed | Where could this have been stopped? |
| 5. Blast radius | Every read, write, and side effect: records touched, messages sent, jobs triggered, who saw the output | What needs cleanup, rollback, or disclosure? |
The trigger is the one teams get wrong most often. The user’s message is rarely the cause; the assembled context is. A stale memory entry, a superseded document ranked first, a system prompt that changed the day before: most “the model went rogue” incidents are context incidents, and if you logged only the message, the cause is gone.
Reconstruction is a property of the system
14:02:11 user request arrives [trigger captured]
14:02:12 retrieval returns 8 docs [doc IDs logged]
14:02:14 plan: update carrier record [decision point]
14:02:15 guardrail: schema check PASS [checkpoint log]
14:02:15 tool: update_record(id=4471) [write: blast radius]
14:02:18 answer drafted and sent [output + recipient]
│
▼
postmortem replay: same context, same versions,
one question per artifact: which one was wrong?If reconstructing this timeline requires an engineer grepping application logs, the artifacts do not exist. Traces provide the trajectory; version pins provide the world state; the audit record provides attribution. Retention matters as much as capture: observability data expires on a storage budget, and incidents in regulated settings surface late. That is why regulated deployments record provenance per decision in an append-only audit trail kept separate from the tracing backend: reconstruction cannot depend on whether the logs happened to survive.
From mechanism to cause
“The model hallucinated” is a mechanism, not a root cause, the same way “the process crashed” never closed a postmortem. Causes live in the system around the model: retrieval surfaced a superseded document because nothing versioned the corpus; the tool accepted a free-text amount field; the loop had no budget; the approval gate was scoped to the wrong action class. Ask why five times against the system, not against the model. The strongest class of fix removes the possibility rather than patching the instance. A tool layer that validates every argument against a schema, rejecting the call before it reaches the data layer, is what a structural fix looks like: an incident class that cannot recur.
Action items that compound
Type every action item, or the review produces prose instead of controls: a new eval case built from the failing run, a checkpoint added or tightened, a tool contract fix, a budget or stop condition, an observability gap closed. The first one is non-negotiable: every postmortem ends with the incident promoted into the golden set, so the failure becomes a regression test the next change must pass. Quarterly, review the distribution of incidents rather than the latest one. The artifact that keeps being missing is your observability backlog, and the checkpoint that keeps being bypassed is your next design review.
Closing
Agents fail like distributed systems and testify like witnesses: fluently, and not always reliably. The postmortem template is how you stop litigating the testimony and start examining the record. Capture the trigger, the trajectory, the world state, the decision points, and the blast radius, at write time, with versions pinned. Then convert every incident into an eval case and, where possible, a structural control. Teams that do this do not have fewer first incidents. They have far fewer second ones.