KV Cache to Prompt Cache, the Architecture of Cheap Inference

Pull-quote: “The cheapest token is the one you never recompute. Most inference bills are a record of prompts that were never designed to be cached.”

Why this matters

Inference cost is usually treated as procurement: pick a cheaper model, negotiate a discount. The larger lever is architectural. Modern serving is built around two caches: the KV cache that makes generation tractable inside a request, and the prompt cache that makes repetition cheap across requests. The first is the serving engine’s job. The second is yours, because whether it works at all is decided by how you lay out your prompts.

The two caches

During generation a transformer attends over everything that came before. The KV cache stores the attention keys and values of already-processed tokens, so each new token costs one incremental step instead of a reprocessing of the whole sequence. The price is memory that grows with context length. That is why serving engines page the cache in blocks, with vLLM’s PagedAttention as the canonical design, and why long contexts squeeze batch sizes on fixed hardware.

Prompt caching extends the same idea across requests. If a new request begins with exactly the same prefix as a recent one, the computed state of that prefix is reused instead of re-prefilled from scratch. Hosted APIs bill cached prefix tokens at a steep discount. Self-hosted engines return the saving as GPU time and faster time to first token. The discipline is identical in both cases, because the reuse depends on a byte-exact prefix match.

Layout, stable first, volatile last

  ┌─────────────────────────────────────┐
  │ system prompt, policies             │  stable for months    ─┐
  │ tool and schema definitions         │  stable for weeks      │ cacheable
  │ few-shot examples, reference docs   │  stable for days       │ prefix
  ├─────────────────────────────────────┤ ── cache boundary ─────┘
  │ session state                       │  changes per session
  │ user query, retrieved evidence      │  changes per request
  └─────────────────────────────────────┘
  one timestamp, request ID, or user name above the
  boundary invalidates everything below it, every request

Order the prompt by volatility. A timestamp in the first line, a request ID in a header, a user name interpolated into the system prompt: each one drags the cache boundary to the top of the document and turns every request into a full-price prefill.

The first row you get for free. The other three are design decisions someone has to make on purpose.

Measure hit rate like an SLO

Cache hit rate belongs on the same dashboard as latency and error rate. When it drops, the causes are enumerable: a prompt edit reordered sections, traffic spread across replicas that do not share cache state, request cadence fell behind the cache TTL. Each has a fix, but only if someone is watching the number and owns it.

Where the design pays fastest

High-volume narration is the textbook case: a pipeline that turns anomaly detections into analyst-readable narratives, thousands of times a day, over one shared instruction set. That workload should be built cache-first, with one stable instruction prefix shared by every narration and the per-event evidence entering last, so the marginal narration rides the cache instead of paying full prefill. Self-hosted deployments sharpen the calculus further. An air-gapped plant has no vendor discount because there is no vendor. Capacity is the GPU already in the room, and prefix reuse in the local serving engine returns directly as throughput, which is the entire budget.

Closing

Cheap inference is not a price sheet. It is a property of systems designed for their caches: prompts ordered by volatility, routing that respects prefix locality, and a hit-rate metric someone owns. Design for the cache and the discount follows. Ignore it and you pay list price for the same tokens, every request, indefinitely.