Quantization in Practice, What 4-Bit Costs Enterprise Accuracy

Pull-quote: “The headline benchmark says the 4-bit model lost one point. The tail of your workload is where that point was hiding, and the tail is where enterprises live.”

Why this matters

Quantization stores model weights at lower precision, 8-bit or 4-bit instead of 16-bit, cutting memory by half to three quarters and raising throughput on the same hardware. For self-hosted and air-gapped deployments it is usually the difference between the model that fits the GPU you own and the model you wanted. The formats are mature: GPTQ and AWQ for weight-only quantization on GPU serving stacks like vLLM, GGUF for llama.cpp-based runtimes such as Ollama, plus 8-bit variants and quantized KV caches as separate levers. What is less mature is how teams evaluate the cost. Aggregate benchmarks report a point or two of loss and everyone moves on. The loss is real, it is just not evenly distributed, and the places it concentrates are the places enterprise workloads care about most.

Where the cost lands

The pattern to internalize: quantization error behaves like a small amount of noise added everywhere. Tasks with many acceptable answers absorb it. Tasks with one correct answer, or long chains of dependent steps, expose it. An aggregate score averages over both and tells you neither.

The memory budget

  Fixed GPU, 48 GB total
  ──────────────────────
  16-bit weights (mid-size model)      ██████████████████░░░░  most of the card
  KV cache + batch headroom            ░░░░  small batches, short contexts

  4-bit weights (same model)           █████░░░░░░░░░░░░░░░░░  a quarter of the card
  KV cache + batch headroom            ████████████████░  longer contexts, real batches

This is the trade in one picture. Quantized weights are not just a smaller model; they are reclaimed room for the KV cache, which converts directly into batch size, context length, and throughput on hardware you cannot upgrade. The alternative use of the same budget is a bigger or less-quantized model at lower throughput. That choice should be made per workload, not by default.

The acceptance gate

Ship a quantized model the way you ship any model change: through an eval gate on your own tasks, with the tails represented on purpose.

1. Build the eval from production traces, including the rare classes, the exact-recall cases, and the longest reasoning chains you actually serve.
2. Compare quantizations head to head, 16-bit against 8-bit against 4-bit variants of the same checkpoint, on that set. The right precision is an empirical answer per task, not a policy.
3. Gate on the tail metrics, worst-class accuracy and exact-match on critical fields, not the average.
4. Re-run the gate on every runtime or format change. Two 4-bit files with the same label are not the same artifact across quantization methods and runtime versions.

A useful division of labor also falls out: quantize aggressively where volume lives, drafts, triage, first passes, and keep the checking role, the validator or judge, at the highest precision the budget allows. Cheap generation, careful verification.

Where this is daily practice

Air-gapped manufacturing deployments live this trade every day. Local LLMs run inside plants where the GPU in the rack is the entire capacity plan and nothing ships without an on-site acceptance run. Quantization is what makes capable models fit that constraint, and the acceptance evals are what decide which quantization level each task is allowed to run at. The plant does not care about a leaderboard delta. It cares whether the quality-event summaries still cite the right lot numbers, which is an exact-recall task, which is exactly where 4-bit needs watching.

Closing

Quantization is one of the best deals in enterprise inference: most of the capability, a fraction of the memory, on hardware you already own. Pay for it in evaluation rather than in surprises. Test the tails, gate on the worst class, choose precision per task, and let the aggressive quantizations carry the volume while full precision keeps the final word.