GPU utilization is not a load signal
GPU_UTIL reports whether a kernel was running, not how much of the device it used. A decode step reads 100 percent at batch size 1 and at 64. Scale on queue age and cache occupancy instead.
The utilisation figure from nvidia-smi or the DCGM GPU_UTIL metric is the fraction of the sampling period during which at least one kernel was executing on the device. It does not measure how many of the device's compute units were busy, how much of its memory bandwidth was in use, or how much useful work was done. A single kernel that keeps one streaming multiprocessor busy for the whole period reports 100 percent.
For LLM serving this makes it nearly useless as a load signal, because a decode step for one sequence and a decode step for sixty-four sequences both keep a kernel running for the whole period.
Why decode looks the same at any batch size
A decode step reads the full weight set from memory and applies it to every sequence in the batch. The weight read dominates. On a device with 3 TB/s of memory bandwidth and 14 GB of weights, the read alone takes about 5 ms, and that time is the same whether one sequence or sixty-four is being processed. The arithmetic for sixty-four sequences is more than for one, but it overlaps with the memory transfer and does not extend the step much until the batch is large.
So a replica serving one request runs a kernel continuously, reports 100 percent utilisation, and is delivering perhaps 2 percent of its potential throughput. A scaler watching that number concludes the replica is saturated and adds another, which then also runs one request at 100 percent.
What actually saturates
Two resources bound a serving replica, and neither is reported by utilisation.
KV cache memory. Each active sequence holds cache proportional to its context length. When free blocks run out, the replica cannot admit another request regardless of how idle its compute is. This is the usual bound for long-context workloads.
Memory bandwidth per step. As the batch grows, per-sequence cache reads accumulate until they rival the weight read, and step time starts to grow with batch size. This is the bound for short-context, high-concurrency workloads.
Signals that correspond to load
| Signal | Source | What it tells you |
|---|---|---|
| Queue depth and queue age | engine metrics | requests waiting for admission; the most direct overload signal |
| KV cache block utilisation | engine metrics | how close to the memory bound |
| Running vs waiting sequences | engine metrics | admitted work vs backlog |
| Tokens per second (prompt and generation) | engine metrics | delivered throughput |
| Time per output token, p50 and p99 | engine or client | whether the batch is large enough to slow decode |
| Pre-emption count | engine metrics | memory pressure severe enough to evict running sequences |
DCGM SM_ACTIVE and DRAM_ACTIVE | DCGM profiling metrics | actual compute and bandwidth occupancy, unlike GPU_UTIL |
vLLM, SGLang, and TensorRT-LLM all export the engine metrics above over Prometheus. DCGM's profiling counters (DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE) require the profiling module but report what utilisation is commonly assumed to.
Scaling rules that work
Scale out on queue age. A request waiting longer than budget means the fleet is under capacity. This is what the user experiences and it is not fooled by batch size.
Alert on KV cache utilisation above roughly 90 percent sustained. The replica is about to start pre-empting or refusing admission.
Use tokens per second per replica as the capacity model. Measure it at the target latency under a representative traffic replay, and compute replica count from forecast token demand divided by that figure.
Treat GPU_UTIL as a health check only. Zero utilisation on a replica that has admitted work means something is wrong. Any other value means the process is running.
Summary
GPU_UTIL reports whether a kernel was running, not how much of the device it used. For decode workloads it reads near 100 percent at any load. Scale on queue age and cache occupancy, model capacity in tokens per second, and if a device-level metric is wanted, use DCGM's SM and DRAM activity counters.