Shadow Deployments for AI, the New Model Beside the Old One
Pull-quote: “The demo proves the new model can be better. Paired traffic proves it is better, on your distribution, at your latency, at your price.”
Why this matters
Model changes now arrive faster than confidence in them: provider releases, prompt rewrites, retrieval changes, fine-tune refreshes. Offline evals are necessary and not sufficient, because a golden set cannot fully represent live traffic, and latency and cost behave differently under production load. The shadow deployment closes the gap: the candidate runs beside the champion on mirrored traffic, users see only the champion, and the switch happens when paired numbers say so, not when the demo feels better.
Agents complicate the mirror
Shadowing a classifier is easy: score twice, compare. Shadowing an agent is not, because agents act, and a shadow that acts twice is an incident generator. The mirror must neutralize side effects at the tool boundary. Reads go to production or replicas with read-only scopes. Writes are stubbed, recording the intended operation without executing it. External actions, emails, payments, tickets, are mocked. Credentials are shadow-scoped, so the blast radius of a mistake is zero by construction. This is a hidden dividend of a typed tool layer: when every action passes through a tool contract, shadow mode is a property of the tool runtime, not something each team re-invents per model.
What to compare
| Dimension | What to look at | Why it gates promotion |
|---|---|---|
| Task success | Judged outcomes on paired requests | The reason the candidate exists |
| Failure taxonomy | Distribution across failure classes | A better average with a new failure mode is not better |
| Guardrail trips | Trip rate per checkpoint plane | Quality that arrives by skirting policy does not count |
| Calibration | Stated confidence vs observed outcomes | Downstream decisions consume the confidence, not the prose |
| Latency | p50 and p95 under real load | Demos never measure the queue |
| Cost | Cost per completed task, retries included | A cheaper model that fails more is more expensive |
Segment before you promote. Aggregate wins hide segment losses: a candidate that is better overall and worse on your largest customer’s traffic is not better. Calibration deserves its seat at the gate. Hold the candidate to the champion’s calibration bar, not just its accuracy, because operational consumers act on the stated probabilities, and a sharper model with drifting confidence is a downgrade in production terms.
The pipeline
production traffic
│
┌── mirror ──┐
▼ ▼
CHAMPION CANDIDATE
serves the shadow mode:
user reads real · writes stubbed ·
│ externals mocked
▼ ▼
response recorded intent
│ │
└── paired ──┘
▼
eval store: success · calibration ·
trips · latency · cost per task
▼
pre-registered gate ──► promote / extend / rejectPre-register the gate
Decide the promotion thresholds before the run starts, in writing: no regression on task success beyond the noise band, calibration within tolerance, cost per completed task inside the envelope, minimum volume per key segment, and a window long enough to cover weekly seasonality, because Tuesday’s traffic does not predict Saturday’s. Pre-registration is what keeps the decision honest; thresholds chosen after the data arrive drift toward whatever the loudest stakeholder wanted. The same principle should govern routing in steady state: models earn traffic through measured performance per task class, and the routing table changes when the measurements change, not when a release note is exciting. When the shadow passes, promote through a canary, a small slice of real traffic with real side effects and a rehearsed rollback. The shadow answered “is it better.” The canary answers “does it behave.”
Closing
Shadow deployments buy the one thing offline evals cannot: evidence from your own distribution at production load, with side effects neutralized. Mirror the traffic, stub the writes, compare paired outcomes on success, calibration, trips, latency, and cost, and let a pre-registered gate make the call. Teams that work this way change models more often than everyone else, precisely because each change carries less fear.