Agents that check their own work
Pull-quote: “The output you should least trust from a critic is a clean pass written with confidence. That is exactly what a model produces when it recognises its own work.”
A model checking its own reasoning is the weakest form of verification available, and the published evidence on that is not ambiguous. Self-checking earns its cost in a specific, nameable set of cases. Outside them it does something worse than nothing: it attaches a confident review to a wrong answer, which is how a defect acquires the appearance of having been examined.
This post is for engineers who have a critic step in a production loop and are not certain it is doing anything. It is the long version of a rule stated in passing in prompt looping and loops that close: a critique helps when it receives information the generation step did not have. By the end you will be able to classify the four things people call self-checking, cite what the research found, recognise a laundered answer, design a check that is genuinely independent, and measure whether your critic is worth its tokens.
What you will be able to do
- Distinguish intrinsic self-correction, critic loops, self-consistency, and external verification.
- Cite what the published research found about each, with the numbers.
- Explain why a confident self-review is the least trustworthy output a critic produces.
- Name the four axes of independence, and pick the cheapest one that applies.
- Score your critic as a classifier and decide whether to keep it.
What do people mean when they say an agent checks its own work?
Four different mechanisms travel under that phrase, and they succeed and fail for unrelated reasons. Naming which one you built is the first diagnostic step, because two of them work and two mostly do not.
Intrinsic self-correction is the same model, in the same context, asked to critique and revise with no new input. A critic loop, which Anthropic calls the evaluator-optimizer workflow, uses a separate call to evaluate an output and feed structured criticism back. Self-consistency samples several independent reasoning paths and takes the most common final answer, with no critique at all. External verification runs a deterministic check or reads the world again, and compares the result to what was intended.
The four get implemented as if they were interchangeable. They are not, and the difference is entirely about what the checking step knows.
What does the research actually say about self-correction?
It says that asking a model to check its own reasoning with no external signal makes the answer worse, on clean experimental design.
Huang and colleagues, in “Large Language Models Cannot Self-Correct Reasoning Yet”, removed oracle labels from the self-correction loop and re-ran the procedure earlier work had reported gains from. Across GSM8K, CommonSenseQA, and HotpotQA, with GPT-3.5, GPT-4, and Llama-2, accuracy fell rather than rose. On GSM8K the model retained its initial answer 74.7% of the time, so the procedure was mostly a no-op, and the revisions it made were not reliably improvements. The earlier gains, roughly 7 points on GSM8K and 15 on CommonSenseQA, depended on ground-truth labels telling the loop when to stop revising.
The same paper measured what does work on that benchmark. Standard prompting produced 76.7%, and self-consistency reached 82.5% with three sampled responses, 85.3% with six, and 88.2% with nine. That is the self-consistency method of Wang and colleagues, which reported 17.9 points on GSM8K, 11.0 on SVAMP, and 12.2 on AQuA over chain-of-thought prompting. Nothing in it involves a model judging itself. It samples, and it counts.
The apparent counter-example is Self-Refine, where Madaan and colleagues reported roughly 20 points of absolute improvement on average across seven tasks using iterative self-feedback. Both results are correct, and reconciling them is the load-bearing idea here. Self-Refine’s tasks largely judge stated quality along named dimensions rather than binary correctness of a reasoning chain, and its feedback stage supplies a rubric the generation prompt did not contain. That rubric is new information. Ask what the checker knew that the generator did not, and the two findings stop disagreeing.
| Study | What it tested | What it found |
|---|---|---|
| Huang et al., arXiv:2310.01798 | Critique and revise with no oracle label | Accuracy fell on all three benchmarks; answer retained 74.7% of the time on GSM8K |
| Wang et al., arXiv:2203.11171 | Sample many paths, take the majority answer | 17.9 points on GSM8K over chain-of-thought prompting |
| Madaan et al., arXiv:2303.17651 | Iterative self-feedback against stated criteria | About 20 points absolute on average across seven tasks |
| Panickssery et al., arXiv:2404.13076 | Whether evaluators recognise their own output | GPT-4 at 73.5% self-recognition; above 90% after fine-tuning on 500 examples |
| Zheng et al., arXiv:2306.05685 | LLM judges against human preference | Over 80% agreement, 85% excluding ties, alongside a named self-enhancement bias |
Why can a self-check launder a wrong answer?
Because the same distribution that produced the error is asked to price it, and because a model can tell when it is looking at its own work. Together those produce a review that is fluent, confident, and worthless.
The recognition part is measured. Panickssery and colleagues found that GPT-4 distinguished its own summaries from those of other models and of humans with 73.5% accuracy unprompted, and that fine-tuning on 500 examples pushed that above 90%. Self-recognition strength correlates linearly with self-preference: the better a model spots its own output, the more it favours it. Zheng and colleagues named the same effect self-enhancement bias, in the work establishing that such judges agree with human preference over 80% of the time, and 85% with ties excluded.
The failure then has a clean signature. When critic and generator share a distribution, an output that is wrong in an on-distribution way reads as correct, because it looks like what the checker would have written. The review comes back clean and well argued. So the output you should least trust from a critic is a confident pass, which inverts how people read critic output in practice.
Two consequences follow. Verbal confidence is not a probability, so thresholding on “the critic was 95% sure” gives an operating point with no relationship to error rate. And agreement between a generator and a critic from the same family is weaker evidence than it appears, because the correlation is partly stylistic rather than epistemic.
When does self-checking genuinely help?
When the check receives information the generation step did not have. That rule covers every case and is testable per check: name the new input, or delete the step.
It helps in five recognisable situations. When an executable oracle exists, meaning a compiler, test suite, linter, schema validator, or dry-run endpoint, the model reads a real failure and repairs against it. When a rubric or requirements checklist arrives from outside the generation prompt, the critique has criteria the draft was not written against, which is the Self-Refine mechanism. When the check performs a fresh retrieval pass over different evidence, it can find the contradiction. When the model generates several candidates for a deterministic filter to choose between, something other than the model is doing selection. And when the failure mode is omission, a pass against an explicit list of required elements catches what the draft skipped.
It does not help in five equally recognisable situations. It does not establish the correctness of a reasoning chain, which is the Huang result stated plainly. It does not validate factual claims unless the check retrieves. It does not tell you whether a write landed, because only a read of the source system does that. It should never carry a safety or permission decision. And it is worst where the failure mode is a confident false belief, which is the case people most want it for.
What does an independent check look like?
Independence means the check differs from the generation on at least one of four axes: mechanism, evidence, context, or model. Ranked by what they buy per unit of cost, they give you a default order to try.
| Axis | What it means | What it costs | What it catches |
|---|---|---|---|
| Mechanism | A compiler, test, validator, or dry run | Almost nothing per call | Form, behaviour, and contract violations |
| Evidence | A second retrieval pass or a read of the source | One extra query or call | Claims contradicted by the record |
| Context | The critic sees the artifact only, never the reasoning | Free, it is a prompt change | Errors the chain of thought would have justified |
| Model | A different model family judges the output | A second inference | Some distribution-specific blind spots |
The context axis is the one most teams have never tried, and it is free. Passing the generator’s chain of thought to the critic imports the error along with the answer, because a plausible derivation is what persuades a model that a wrong conclusion is right. Give the critic the finished artifact and the requirements, nothing else. If the artifact cannot be judged without its reasoning, that is information too: usually a missing citation or an unstated assumption.
The model axis helps less than people expect. A different family removes the self-recognition effect Panickssery and colleagues measured, and does not remove correlated errors from overlapping training data. Treat cross-model agreement as a signal, not as statistical independence.
For irreversible actions, none of the four is sufficient alone. Use a two-key rule: an independent check of the mechanism or evidence kind, plus a person on the approval, with the exact action written out for them. That is the escalation discipline described in orchestrator and worker model loops, and it applies whether one model or ten produced the proposal.
How do you measure whether your critic is worth its cost?
Score it as a classifier against outcomes, because “the critic seems to help” is not a finding. Three measurements settle it.
First, precision and recall on the judgement “this output is wrong”. Log the critic’s verdict and the eventual ground truth for every run, from tests, user corrections, downstream failures, or sampled human review. A critic that passes 95% of outputs on a task whose true success rate is 70% is detecting nothing, and that comparison alone retires most critic steps.
Second, the revision keep rate. Keep the original, apply the revision, re-run the deterministic checks, and record how often the revision survived. A low rate means the critique produces text rather than signal. A high rate with no movement in the checks means the loop is rewriting for style and calling it verification.
Third, the cost against the cheapest check that would have caught the same defect class. If a schema validator catches four of the five defect types your critic finds, the critic is paid to find the fifth, and you should know its cost per catch. This instrumentation belongs with the rest of your run data, the subject of building an agent evaluation harness.
Where this goes wrong
The confident endorsement. The symptom is a critic pass rate near 98% while defects keep reaching users. The cause is the same model reviewing the same context with nothing new. Give the critic a rubric it did not write, or replace it with a deterministic check.
The critique that reads the reasoning. The symptom is a critic restating the generator’s error in different words, sometimes more persuasively. The cause is passing the chain of thought. Pass the artifact and the requirements only.
Revision accepted unconditionally. The symptom is a second draft that is longer, more hedged, and no better. The cause is a rewrite with no guard. Keep the original and discard the revision unless the checks improve.
Self-consistency on open-ended output. The symptom is majority voting over free text producing something bland and slightly wrong. The cause is applying a method that needs comparable final answers to a task that has none. Use a rubric instead.
Stated confidence used as a probability. The symptom is a confidence threshold that separates nothing at any value you try. The cause is treating verbal confidence as calibrated. Calibrate against outcomes, or use agreement across independent samples.
Critique applied uniformly. The symptom is roughly doubled cost with quality inside the noise. The cause is running the critic on every output rather than where a wrong pass is expensive. Route by risk, and let deterministic checks carry the rest.
Common questions
Does a second model checking the first solve the problem?
Partly. A different model family removes the self-recognition effect Panickssery and colleagues measured, so the reviewer is no longer favouring its own style. It does not remove errors both models share from overlapping training data, so treat cross-model agreement as a signal rather than an independent measurement.
Is self-consistency the same as self-checking?
No, and the distinction matters because one of them works. Self-consistency samples several reasoning paths independently and takes the most common answer, never asking a model to judge anything. Huang and colleagues found it improved GSM8K accuracy from 76.7% to 88.2% with nine samples, in the same experiments where intrinsic self-correction reduced accuracy.
Should I let the agent grade its own output in production?
Only where a wrong pass is cheap, or where the grading step has real new information such as a test result or a fresh read. For anything irreversible or externally visible, use a mechanism check plus a human approval, because a self-grade on a destructive action is a rubber stamp on the path you cannot undo.
Does telling the model to think carefully before checking help?
It helps a little and does not change the mechanism. A longer critique from the same model on the same context is still one distribution pricing its own output, so the improvement is bounded by the fact that no new information entered. Spend the same tokens on a retrieval pass or a validator instead.
What is the cheapest check that actually works?
A schema validator, followed by whatever executable oracle your domain already has: a test suite, a compiler, a linter, a SQL dry run, a database constraint. These cost almost nothing per call, have no self-preference, and catch the largest and most boring class of defects, freeing the expensive checks for judgements no program can make.
Next steps
Take every critic step in your system and write down, next to each, the information it has that the generator did not. Steps with an empty answer are candidates for deletion, and deleting them usually improves both cost and quality.
Then read building an agent evaluation harness, because scoring the critic as a classifier needs the same run data and ground truth pipeline as the rest of your evaluation. For the loop mechanics around a check, prompt looping and loops that close is the prerequisite, and when a check misses something, a failure taxonomy for production agents names what happened.
Resolving every extracted claim back to a source document rather than to the model’s confidence in it is the review discipline behind EvidAI in pharmaceutical evidence review, where a claim that cannot be traced to a document is treated as absent rather than probable.