Dual-Reviewer Screening with AI Agents in the Loop
Pull-quote: “An AI screener does not earn trust by being impressive. It earns trust the way any second reviewer does: measured agreement, adjudicated conflicts, and reasons on the record.”
Dual review is the control. Keep it.
Independent screening by two reviewers, with disagreements resolved by discussion or a third party, is the standard control against single-reviewer error in systematic reviews. The right way to introduce AI is not to retire that control but to hold the AI to it: the agent screens as a reviewer whose agreement is measured, whose conflicts are routed, and whose reasoning is inspectable. In a well-designed workflow, an ensemble of agents screens with consensus voting alongside human reviewers, and dual-reviewer enforcement is exactly that, enforcement: no record leaves screening on one opinion.
Measure agreement, not impressions
Cohen’s kappa is the working instrument: agreement between two reviewers corrected for chance, computed on the same records, whether the pair is human and human, human and agent, or agent and agent across the ensemble. The classical interpretation bands trace to Landis and Koch, and they translate directly into screening operations.
| Kappa | Reading | Operational response |
|---|---|---|
| Below 0.40 | Poor to fair | Stop. Rework criteria wording, recalibrate on a pilot set |
| 0.41 to 0.60 | Moderate | Tighten ambiguous criteria before scaling up |
| 0.61 to 0.80 | Substantial | Production screening territory, keep monitoring |
| 0.81 to 1.00 | Almost perfect | Healthy, but check the criteria are not too narrow |
Two disciplines make the number honest. Calibrate first: run a pilot set through all reviewers before full screening, and fix the criteria, not the reviewer, when kappa is low, because low agreement usually means the protocol is ambiguous. Track continuously: agreement drifts as the queue moves from obviously relevant records into the long ambiguous tail, and a kappa that was substantial in week one deserves re-checking in week three.
Two stages, two agreement profiles
Screening is two different jobs wearing one name. Title and abstract screening is high volume and deliberately permissive: a wrong exclusion loses a study, while a wrong inclusion only costs a full-text read. Full-text screening is lower volume and stricter, with an exclusion reason recorded per record. Agreement should be tracked separately for each stage, because they measure different skills, and an agent ensemble that is almost perfect at the first stage can still be merely moderate at the second, where the criteria do the fine work.
Conflicts are routed, never averaged
reviewer A reviewer B
(human or agent) (human or agent)
│ │
decision + reasons decision + reasons
└──────────┬──────────────────┘
▼
agreement?
│
yes ───────┼─────── no
│ │
▼ ▼
decision recorded conflict queue
│
▼
adjudication: third reviewer
or consensus discussion
│
▼
audit trail entry: model, prompt
version, input hash, reviewer identityA conflict between reviewers is information, not friction. Systematic conflicts cluster around specific criteria, and the conflict queue is the fastest diagnostic of a protocol that needs one more sentence of precision.
Reasoning chains are the difference
A bare include-or-exclude label from a model is unauditable. An explainable reasoning chain, this record fails the population criterion, here is the passage, is a decision a human adjudicator can confirm or overturn in seconds, and an auditor can reconstruct months later. Consensus voting adds a second signal: when the agent ensemble divides on a record, the division itself flags genuine ambiguity that deserves human eyes. Every decision, human and agent alike, belongs on an append-only audit trail with the model, prompt version, input hash, and reviewer identity attached.
Closing
AI in the screening loop is not a shortcut around dual review. It is a second reviewer held to the discipline the method always required: calibrated before trusted, measured with kappa, contradicted through a conflict queue, and explainable on every record. Humans stay in command because the workflow makes any other arrangement impossible.