Small Models, Big Systems, Routing to the Cheapest Capable Model
Pull-quote: “The question is never which model is best. It is which model is the cheapest one that is good enough for this specific piece of work.”
Why this matters
A production LLM system is not one model. It is a workload: classification here, extraction there, synthesis at the end, a judgment call somewhere in the middle. Sending all of it to the largest frontier model is the LLM equivalent of running every query on the biggest warehouse cluster, and it fails the same way, on the bill. The mature pattern is routing: decompose the workload by task, assign each task the cheapest model that clears its quality bar, and reserve the expensive calls for the work that actually needs them.
The triangle you are actually optimizing
Every routing decision trades among three axes, and you can rarely win all three at once:
| Axis | Small model | Frontier model |
|---|---|---|
| Cost per call | Low | High |
| Latency | Low, often local | Higher, always remote |
| Quality ceiling | Sufficient for narrow tasks | Required for open-ended reasoning |
The insight that makes routing profitable is that quality requirements are task-shaped, not system-shaped. A classifier that labels a document’s language does not need the model that writes the executive summary. Measure each task’s quality bar independently, on your own data, and the routing table writes itself.
The cascade pattern
Static routing assigns tasks to models ahead of time. Cascades route dynamically, per request:
Request ──► Small model ──► Confidence check ──► Accept ──► Done (most traffic)
│
│ low confidence / verifier reject
▼
Larger model ──► Confidence check ──► Accept
│
│ still uncertain, high stakes
▼
Cross-family consensus ──► Agree: accept
└──► Disagree: human reviewTwo implementation details decide whether a cascade works:
- The confidence signal must be real. Self-reported confidence (“rate your certainty”) is weakly calibrated. Prefer signals with teeth: token log-probabilities where your server exposes them, agreement across sampled generations, or a cheap verifier model checking the output against the input.
- The escalation threshold is an evaluated parameter, not a guess. Set it on a labeled set where you know what the small model gets wrong; the threshold is where escalation cost crosses error cost.
Consensus for the decisions that matter
At the top of the cascade sits a different tool: cross-family consensus. For high-stakes judgments, run the same structured decision through models from different families and compare. Models within a family share training lineage and share blind spots; disagreement across families is a genuinely informative uncertainty signal. Screening and review workflows show the pattern at its best: when consensus holds, the decision proceeds; when it breaks, the record is flagged for human adjudication rather than guessed at. The same philosophy should govern the routing table itself. A route earns its place through measured performance on your tasks, not model reputation, and it gets re-justified whenever the model lineup changes.
Closing
Model routing is the FinOps of LLM engineering, and like FinOps it is mostly measurement. Decompose the workload, establish a per-task quality bar on your own data, cascade on confidence signals that are actually calibrated, and spend the frontier-model budget where escalation is cheaper than error. Small models do the volume. Big models do the judgment. Consensus guards the decisions you cannot afford to get wrong.