Cost Observability for LLM Systems, Envelopes and Attribution

Pull-quote: “A monthly invoice is a postmortem. LLM spend moves at request speed, and any budget that cannot act at request speed is decoration.”

Why this matters

The invoice is the last place to learn what your AI features cost. LLM spend moves at request speed: a prompt change doubles context, a new feature quietly loops, a single tenant starts hammering an endpoint, and the monthly number arrives too late to govern any of it. Most organizations still run one shared API key and one aggregate bill, so when the number spikes, nobody can say which feature, which customer, or which deploy did it. Cost observability is the discipline that fixes this: meter every call, attribute it to something the business recognizes, wrap budgets around the units that matter, and enforce them while the request is still in flight.

Meter at the span

Every model call already knows its own cost: tokens in, tokens out, price per token. Emit it as telemetry on the span, the same place latency lives, and tag it with the dimensions attribution will need: feature, tenant, environment, run ID, model. The tags are the entire game. Untagged spend is unattributable forever, so enforce tagging at the gateway: a call without a feature tag does not go through. That is a fail-closed rule, and it is the cheapest one you will ever deploy, because it costs nothing at request time and saves the archaeology later. From spans, roll up into a ledger: cost per feature per day, per tenant per month, per completed task. The reduction levers, caching, compaction, routing, are a sibling discipline; this post is about seeing and governing, because you cannot reduce what you cannot attribute.

Envelopes, not alerts

An alert tells you money left. An envelope decides what happens next. A budget envelope is an amount, a window, a scope, and pre-agreed actions at thresholds:

The degradation step is what separates governance from theater, and it is a product decision made in advance: which features may quietly step down to a cheaper model, which may serve cached answers, and which must never degrade and instead page a human. Scope envelopes to features and tenants, not just the organization. An org-level budget lets one runaway feature spend everyone’s quarter before anyone is warned.

The flow

  request ──► gateway ──► model call ──► response
                │              │
             [tags:         [meter:
              feature ·      tokens × price]
              tenant · run]      │
                │                ▼
                ▼           ledger (feature · tenant · task)
             policy  ◄───── envelope evaluator (80 / 100 / 120)
                │
        warn · degrade · stop
                │
                └──► next request feels the decision

The evaluator runs continuously against the ledger, not monthly against the invoice. Enforcement signals flow back to the gateway, where the next request feels them.

Attribution that survives finance

Attribution has three audiences. Engineering wants cost per feature, which is how a regression shows up hours after a deploy instead of weeks. Product wants cost per tenant, which sets pricing floors and catches abuse. Finance wants unit economics: cost per completed task, retries and failures included, which is the number that decides whether a feature survives. Start with showback, publishing the numbers to their owners, and move to chargeback only once the meters are trusted. And when the unit of value is a deliverable rather than an API call, meter the deliverable, so “what did this cost” has an auditable answer per artifact instead of a monthly guess. Model choice belongs in the same loop: route each task class on measured quality per dollar, because a routing table that ignores that measurement is a preference, not a policy. The meters are what turn model choice from a periodic argument into a standing decision.

Closing

Meter every call at the span with enforced tags, attribute spend to features, tenants, and completed tasks, wrap envelopes around the units the business recognizes, and enforce at the gateway while the request is in flight. Do this and the monthly invoice stops being news. It becomes a reconciliation of decisions your system already made, deliberately, at request speed.