Tracing Agents Like Distributed Systems

Pull-quote: “An agent run is a distributed trace wearing a trench coat: dozens of network calls, partial failures, retries, and fan-out. The industry solved this observability problem fifteen years ago. Use that solution.”

Why this matters

A single agent request can produce ten model calls, six tool invocations, three retrievals, and a retry loop, any of which can fail, degrade, or silently return the wrong thing. When the final answer is bad, “look at the logs” means scrolling a wall of interleaved JSON. Microservices had the same problem and answered it with distributed tracing: spans, parent-child trees, and correlation IDs. Agents deserve nothing less, and the mapping is nearly one-to-one.

Span everything

Every unit of work becomes a span with a start, an end, a status, and typed attributes:

Do not invent a private schema. The OpenTelemetry semantic conventions for generative AI define standard attribute names for model calls, token usage, and tool execution; adopting them means any OTel-compatible backend, and the observability platforms built specifically for LLM systems, can ingest your traces without adapters. Conventions outlive vendors.

The run tree is the debugging surface

  trace: support-request-8f31
  └── agent.run  (12.4s · $0.084)
      ├── llm.call  plan          (1.1s · 2.1k tok)
      ├── tool.call retrieve_docs (0.6s)
      │   └── retrieval  k=8, index=support-kb
      ├── llm.call  draft         (3.8s · 6.4k tok)
      ├── tool.call update_ticket (0.4s · FAILED 403)
      ├── llm.call  retry-plan    (1.2s · 2.3k tok)   ◄ loop begins
      ├── tool.call update_ticket (0.4s · FAILED 403) ◄ same failure
      └── llm.call  apologize     (1.9s · 1.8k tok)

Ten seconds with this tree tells you what an hour of log-grepping cannot: the agent burned two model calls retrying a tool that fails deterministically on permissions. The tree turns “the agent is being weird” into a specific span with a specific status code.

Replayable, or it didn’t happen

A trace you can read is good; a trace you can re-run is the real asset. Capture full inputs and outputs (with redaction policies applied at write time) so any production run can be replayed against a new prompt, a new model, or a fixed tool. Replayed traces are also the cheapest source of eval cases: yesterday’s failure becomes tomorrow’s regression test without anyone writing a synthetic scenario.

Classify failures, then count them

Individual traces answer “what happened here.” A failure taxonomy answers “what keeps happening.” Tag every failed run with a category (wrong tool selected, hallucinated arguments, unhandled tool error, context overflow, loop without progress, guardrail block) and review the distribution weekly. The taxonomy tells you whether to spend the sprint on tool schemas or on loop detection; without it, you fix whichever failure was seen most recently.

Cost lives on the spans

Because every model span carries token counts and price, cost attribution falls out of the trace for free: cost per run, per step, per tool, per customer. The retry loop in the tree above is not just a latency bug; it is a line item. Per-span cost is the raw material for the token-economics work every agent team eventually faces.

Operational traces answer the engineer’s question. When the output feeds a regulated decision, a different question arrives later: what did the AI contribute, and who approved it? That answer must outlive log retention, which is why AI provenance belongs in an append-only, hash-chained audit record kept per decision, separate from the tracing backend. Observability data expires; evidence does not.

Closing

Treat every agent as a distributed system, because it is one. Span every model and tool call on OpenTelemetry GenAI conventions, keep run trees replayable, maintain a failure taxonomy that directs engineering effort, and attribute cost per span. Teams that do this stop debating what the agent did and start deciding what to fix.