AI Agent Latency: Finding the Slow Span with LLM Tracing

Two runs of the same agent, same prompt template, same model. One comes back fast enough that nobody notices. The other takes long enough that someone files a ticket. Neither errored. The average on the dashboard still looks healthy. And nobody on the team can say which part of the slow run was slow, because the run was dozens of spans deep and only the outermost one was ever measured.

That is the ordinary condition of agent systems in production. Latency stops being a property of a request and becomes a property of a chain: planning, retrieval, tool calls, model calls, sub-agent handoffs, each waiting on the one before it. Anthropic’s engineering team reported in June 2025 that moving from sequential to parallel sub-agent execution cut research time by up to 90% on complex queries – the same work, reorganized in time. LLM tracing is what makes that kind of reorganization possible, because it is the only instrument that resolves a run into parts you can act on individually.

Below: the four numbers that "latency" hides, what a trace has to record to be diagnostic, a five-zone model for locating slow time, and how to turn the diagnosis into a latency budget that holds.

‍

LLM Latency Is Four Numbers, Not One

LLM latency splits into four measurements that fail independently, and collapsing them into a single average is the most common instrumentation mistake in production AI. Time to first token (TTFT) is the gap between issuing a request and receiving the first piece of the response. Time per output token is how fast the rest then streams. End-to-end duration covers the full operation. Tail latency – p95 and p99 – describes the slow minority of runs that generate every complaint. A median that looks fine says almost nothing about the run a user actually sat through.

The split matters because the two halves of an inference call have opposite drivers. A model processes every input token before it emits the first output token, so TTFT scales with context length: a 50,000-token prompt starts slower than a 500-token prompt on identical hardware. Generation time scales with output length instead. Two calls with the same end-to-end duration can therefore need opposite fixes – one wants a shorter prompt or a warm cache, the other wants a lower max-tokens ceiling or a faster model. Without step-level traces, you cannot tell which one you are holding.

‍

What a Trace Records That a Dashboard Doesn't

A span is a single timed unit of work inside a run – one model call, one tool execution, one retrieval – carrying a start time, a duration, a parent, and a set of attributes. A trace is the tree those spans form. LLM tracing is the practice of emitting spans for every step an AI application takes, so a run’s duration can be decomposed into the durations of its parts instead of reported as one opaque number. It is distributed tracing applied to a non-deterministic call graph: the mechanics are familiar, but the tree changes shape on every run.

The naming is no longer a matter of taste. The OpenTelemetry GenAI semantic conventions – moved to a dedicated repository in June 2026 and still marked Development – define the operations an agent system produces: chat for a model call, execute_tool for a tool invocation, retrieval for a vector lookup, plan for a decomposition step, and invoke_agent for the run as a whole. They also define the latency instruments directly, including gen_ai.client.operation.duration, gen_ai.client.operation.time_to_first_chunk, gen_ai.execute_tool.duration and gen_ai.invoke_agent.duration, plus counters for inference and tool calls per invocation. GenAI usage tokens ride alongside as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.

Three properties separate an AI agent tracing setup from ordinary APM. Session grouping ties every span produced by one goal into a single record, so you can reconstruct a user-facing wait end to end. Token attributes sit on the spans themselves, which lets you correlate duration against input size rather than guess at it. And the trace backend has to retain the real prompt and response bodies for the runs you care about – "this call took nine seconds" isn't diagnosable without knowing what went into it.

‍

The Agent Latency Ledger: Five Places Time Hides

Every millisecond in an agent run belongs to one of five zones. Naming them turns "it’s slow" into a routing decision.

The fifth zone is the one dashboards never surface. The most expensive latency in an agent trace is often time that belongs to no span at all – the gap between a parent span starting and its first child starting, or between one child ending and the next beginning. That dead air is orchestration: a planner deciding, a framework serializing, a lead agent waiting on the slowest of five sub-agents before it can proceed. Anthropic documents this bottleneck in its own production system, where sub-agents execute synchronously, and the whole system can block while a single one finishes. Sum the child durations and subtract from the parent total; whatever is missing is the gap, and it is frequently the largest line in the ledger.

This decomposition is what an AI agent tracing view is for. OptScale AI’s Team & Agent AI Performance pillar renders any run as a waterfall – every step in the chain, its duration, its model, its token count, and its cost – with sub-second trace lookup across millions of logged interactions, so the ledger can be read off a single run rather than assembled by hand.

Depth Costs Time. Width Costs Money.

Two numbers describe the shape of an agent run, and they aren't the same. Width is the total count of spans a run produced. Call stack depth is the longest chain of spans that must happen in order, each blocked on the previous result. Width sets what the run costs. Depth sets how long it takes.

That distinction explains most of the confusion in agent performance work. An agent firing forty tool calls in parallel and an agent firing forty in sequence produce a similar token bill and wildly different wall-clock times. Reducing width is a cost exercise; reducing depth is a latency exercise; and an optimization that helps one can hurt the other, since parallel fan-out shortens the critical path while multiplying concurrent spend.

So read the trace for the critical path, not for the biggest bar. Find the longest dependent chain from root to leaf, then ask of each link whether it genuinely needs its predecessor’s output. In most agent graphs, several links do not, and they run sequentially only because the framework’s default is to await.

If what you are chasing is spend rather than time – the same call repeated forty times, a task quietly drifting, recursion without a ceiling – that is the other axis, covered separately in Runaway Agents: Detecting Loops, Drift, and Recursion. Same traces, different question: that article asks what the agent did; this one asks how long each part of it took.

‍

A Five-Step Diagnostic Sequence

With LLM tracing in place, the following routine becomes repeatable rather than heroic. Step-level traces are a prerequisite for every step.

1. Group by session. Reconstruct the whole goal, not one call. Session grouping is what turns forty scattered spans into one user-facing wait.

2. Rank by p95, not by mean. Sort sessions by duration and work the tail. The median session is not the one anyone is complaining about.

3. Read the waterfall for gaps first. Before you look at the longest bar, check whether the children’s durations add up to the parent’s. What is missing is orchestration.

4. Split the longest span. Compare the time to the first chunk against total duration for that call. A long TTFT is an input-size problem; a long tail after the first chunk is an output-size or throughput problem.

5. Classify against the ledger and pick the matching lever. Prefill wants caching or compression; decode wants a cap or a different model; tools want parallelism or a timeout; orchestration wants a restructured graph.

Set the latency budget before you start, not after. Assign a ceiling per zone – say 150 ms for edge and gateway, 1.5 s for prefill, 3 s for decode, 2 s total for tools – and treat any zone over its ceiling as the defect. Without a stated budget, every trace looks acceptable in isolation, and nobody can say which of five plausible fixes to fund first.

‍

From Diagnosis to Enforcement

Tracing tells you where the time went. It cannot give any of it back. That takes a component sitting in the request path, which is where a gateway earns its position: every model call, tool call, and retrieval an agent makes has to leave the process, so routing that traffic through one control point creates the single place where latency-aware decisions can actually be made. Route a summarization step to a faster model while a reasoning step keeps the frontier one. Fail over automatically when a provider’s tail latency degrades. Compress context so prefill has less to chew on. Enforce a maximum execution time so a stalled run terminates instead of hanging. OptScale AI’s gateway adds under 2 ms of its own overhead – a rounding error against multi-second inference, and the price of having anywhere to enforce a budget at all.

The pairing is the point. LLM tracing supplies the evidence; the gateway supplies the lever. Neither does much alone.

‍

Conclusion

Agent latency is not one number and cannot be fixed as one. It is four measurements – TTFT, time per output token, end-to-end duration, and tail percentiles – distributed across five zones: edge and gateway, prefill, decode, tools and retrieval, and orchestration gaps. LLM tracing is what makes those zones visible, and the OpenTelemetry GenAI conventions now name the spans and metrics needed to record them consistently. Two rules do most of the diagnostic work: the most expensive latency in an agent trace is often the time that belongs to no span at all, and depth costs time while width costs money. Everything else follows from reading the critical path against a stated latency budget instead of eyeballing a waterfall.

Detection and correction are separate jobs. OptScale AI covers both: the Team & Agent AI Performance pillar captures every request flowing through the gateway with its model, cost, latency, and token count, and renders any run as a full waterfall with sub-second trace lookup – while the AI Gateway sits in the request path, where latency-aware routing, failover, context compression, and hard execution limits can actually be enforced.