Time to first token and time per output token
An LLM response has two latencies: the wait before the first word, and the pace of the words after it. They have different causes and different fixes, and a single latency number hides both.
For a conventional API call, latency is a single measurement from request to response. A streamed LLM response has two, and they respond to different conditions. A request-level latency metric aggregates them into a number that does not identify which has changed.
Time to first token (TTFT) is the interval before the client receives any output. Time per output token (TPOT) is the interval between successive tokens once output has begun. A 2,000-token response with a 400 ms TTFT and a 20 ms TPOT completes in approximately 40 seconds. A total-latency chart reports 40 seconds; a 500 ms increase in TTFT is invisible in it and fully visible to the user.
Composition
TTFT = queue wait + prefill. The request waits for a scheduler slot on a replica, then the full prompt is processed in a single pass to construct the attention state. Prefill is compute-bound: cost scales with prompt length, and a 6,000-token prompt costs roughly thirty times a 200-token prompt. Queue wait scales with load and with the replica's memory occupancy.
TPOT = one decode step. Each output token requires one forward pass for every sequence in the batch. Decode is memory-bandwidth-bound: step time is dominated by reading the weights and each sequence's attention cache, so it scales with batch size and context length and only indirectly with queue depth.
The two metrics therefore respond to different variables. TTFT moves with load and prompt length. TPOT moves with batch size and context length.
Factors affecting TTFT
- Queueing. The dominant contributor under load. Alert on queue age per replica, and route on tokens in flight rather than request count so that long-prompt requests do not concentrate on one replica.
- Prefill of long prompts. Chunked prefill distributes a long prompt across several scheduler iterations. Prefix caching eliminates prefill for the portion of the prompt matching a recent request, which for a shared system prompt is the majority.
- Cold replicas. A newly started replica has an empty cache and uncompiled kernels. Warm it before admitting traffic.
Factors affecting TPOT
- Batch size. More sequences per step increases work per step. This is the throughput-latency trade-off expressed as a single parameter.
- Context length. Each step reads the attention cache of every sequence; longer conversations increase step time for all sequences in the batch.
- Prefill arriving mid-batch. Without chunking, one new request's prefill extends the step for sequences already decoding. This appears as a TPOT spike correlated with another request's TTFT.
- Speculative decoding and quantisation reduce it. Both are engine-level choices.
Instrumentation
Two panels rather than one. TTFT at p50 and p99, alongside queue age. TPOT at p50 and p99, alongside batch size and mean context length. Total request latency is useful for capacity planning and should not be the alerting metric.
Set budgets independently. An interactive product might require TTFT under 600 ms and TPOT under 30 ms for readable streaming. A batch summarisation workload is indifferent to both and constrained only by tokens per second per GPU. Same model, same engine, different tuning.
Summary
TTFT measures the delay before output begins; TPOT measures the rate once it has. They have distinct causes, distinct controls, and require distinct alerts.