Whole-GPU Idle Under Bursty Request Arrival: Measuring the Gap Between Billed Instance-Hours and Consumed GPU-Seconds on an LLM Serving Endpoint

1. Introduction

The claim that self-hosted inference wastes a large fraction of GPU spend circulates widely, usually as an unmeasured percentage. TR-2026-01 replaced assertion with measurement on one workload class and found the claim false there: a saturated diffusion-transformer generation left nothing to recover. That result partitioned the hypothesis space — if a billed-versus-used gap exists, it must live in the arrival process, not the model.

This report tests that partition directly. The design question: on identical hardware, with an identical model, identical prompts, and identical generation lengths, how does GPU-busy fraction over the billing window respond to request arrival rate alone?

Outcome: confirmed — 2.0% (range 1.6–2.4% across 3 runs). One input assumption was wrong in a direction that strengthens the result: generations measured 0.284 s median, not ~1 s, so the offered duty cycle at λ = 0.05 was 1.4%, and the measurement sits just above it (§4.2).

2. Method

2.1 Environment

2.2 Design

Every request-level variable is held constant; only the arrival process varies. Fixed ~50-token prompt, max_tokens=128, temperature=0, default vLLM scheduling. Every completed request returned exactly 128 tokens.

Poisson and burst patterns are open-loop: arrivals are scheduled on the wall clock and fired as detached asynchronous tasks, so a slow response cannot delay the next arrival. Only the control is closed-loop, by design. Arrival schedules are seeded (seed = run index) and recorded.

2.3 Instrumentation

A dedicated thread samples NVML every 100 ms (identical to the TR-2026-01 rig): utilization.gpu, power, memory used.

GPU-busy is defined as the fraction of 100 ms samples with utilization.gpu > 0, taken over the entire wall-clock measurement window. The billing window is the denominator throughout — never request time — because the billing window is what the invoice denominates.

Each pattern: 3 discarded warm-up requests (cold start is a distinct phenomenon, excluded by design per TR-2026-01's stated caveat), 10 s settle to true idle, then the measured window. Each pattern run 3 times. No retries, no error suppression: 0 failed requests across all 15 runs.

2.4 Control (rig validation)

The saturating control returned 99.86% GPU-busy (range 99.75–99.92%), with Σ request durations equal to wall time (120.2 s = 120.2 s), as a closed loop must produce. The pre-declared abort threshold was ≥95%. Power under saturation averaged 303–313 W against a 117 W idle floor.

Saturation power sits well below the 700 W cap: single-stream decode of a 1.5B model is memory-latency-bound and does not electrically saturate an H100 the way TR-2026-01's diffusion workload did (~630 W). This does not affect the idle claim; the busy/idle power contrast remains unambiguous.

3. Results

Means over 3 runs; dollar figures at $2.00/hr (configurable).

Request-level statistics (pooled): duration median 0.284 s (p95 0.296 s), time-to-first-token median 6–10 ms, exactly 128 tokens per completion, ≈450 tok/s single-stream.

The row that defies intuition is not the sparsest one. At λ = 1.0 — one request every second, a genuinely active endpoint by early-stage standards — the GPU is idle for 69% of the billed window. Sparse-traffic waste is easy to dismiss as an edge case; a busy-sounding endpoint wasting two-thirds of its bill is not.

3.1 Dose-response: measured busy tracks offered load

If the instrument is honest, measured busy should equal the offered duty cycle (λ × 0.284 s) plus a small positive bias from NVML's internal averaging window:

The excess is small, positive, and shrinks with sparsity — the signature of counter smearing — and it biases against the idle hypothesis. Reported idle fractions are therefore a lower bound on true whole-GPU idle.

This table also yields the report's most practically useful output, a self-diagnosis rule requiring only two numbers an operator already knows:

A service handling 0.2 req/s of 1-second generations should expect ~20% GPU-busy — and ~80% of its instance-hours billed against an idle card.

3.2 Power corroborates utilization at every tier

Whenever the sampler reports busy, the GPU draws real additional power; whenever it reports idle, draw sits at the floor. The utilization signal is not an NVML artifact. The floor itself is a secondary finding: an idle H100 with weights resident draws ~117 W continuously — the invoice hides the idle time, and the idle time hides the energy.

3.3 The burst pattern: consolidation capacity, measured

All 20 concurrent requests completed in ~0.33 s each: vLLM's continuous batching absorbed the entire burst essentially simultaneously (~4,300 tok/s aggregate vs. 450 tok/s single-stream, ≈9.5× throughput at near-identical per-request latency). The GPU then sat at the idle floor for the remaining 359.5 s of the window.

This is the consolidation counterfactual measured directly: the same GPU that idles 98–99.9% of the time under sparse arrival can absorb an order of magnitude more concurrent work without measurable latency cost. The capacity to recover the idle exists; whether a given deployment can exploit it is addressed in §5.

4. Cost interpretation

Billed GPU-seconds per request = wall time ÷ requests completed:

The 68× figure is arithmetic, not per-request inefficiency — the identical request, executing identically, costs 68× more solely because of what its neighbors are doing. That the number is a tautology of wall-clock billing is precisely the finding: the invoice is denominated in a unit the workload does not consume, so per-request cost becomes a property of the arrival process rather than of the request.

Extrapolated to an always-on single-H100 endpoint at $2.00/hr ($1,460/mo):

These are extrapolations from 5–10 minute windows, not measurements of a production month. Real traffic is diurnal and burstier than homogeneous Poisson — which makes Poisson the conservative case: diurnal traffic concentrates load and lengthens idle gaps, pushing wall-clock idle higher, as the burst pattern (99.9% idle) demonstrates at the limit.

5. What this measurement cannot see

1. utilization.gpu > 0 means at least one kernel was resident — not that the GPU did useful work. A single kernel on one SM reads identically to full saturation. This method measures whole-GPU idle only; a busy-but-low-occupancy waste class exists and is invisible here (DCGM-class counters are required, and are future work in this series).
2. NVML's utilization counter averages over an internal window (1/6–1 s depending on product), smearing short kernels across adjacent samples. Quantified in §3.1 (+0.6 to +2.6 pp); the direction is against the hypothesis.
3. Idle memory is not free memory. vLLM pre-allocates KV cache: memory-in-use held at ~74 GiB throughout, including windows that were 98% idle. A second tenant cannot naively co-locate; recovering the idle requires scheduling — batching, multiplexing, or scale-to-zero — not mere co-residence.
4. Cold-start cost is excluded by design (warm-up + settle precede every window). A scale-to-zero strategy pays model-load and warm-up costs this experiment deliberately does not measure; TR-2026-01 measured that class separately (27% GPU-busy over a 29.6 s cold load).
5. Single GPU, single model size, single stack. By the duty-cycle relation, a 70B model with multi-second generations at the same λ shows proportionally higher busy fraction; the idle phenomenon concentrates in small-to-mid models and sparse endpoints.

6. Threats to validity

Synthetic arrivals (homogeneous Poisson, argued in §4 to be conservative relative to diurnal traffic); one hardware/stack configuration; three runs per pattern (ranges reported are tight — e.g., [0.016, 0.024] at λ = 0.05 — but n = 3 is n = 3); dollar figures assume on-demand pricing and do not model reserved or spot economics.

7. Conclusion

Together, TR-2026-01 and this report bracket the phenomenon on identical methodology: a saturated single-stream workload shows no whole-GPU idle to recover (99% busy), while a sparse serving endpoint spends 93–99.9% of its billed window (at λ ≤ 0.2) with zero resident kernels — and even a one-request-per-second endpoint idles 69% of it. The waste is not in the model or the hardware; it lives in the gaps between arrivals, is invisible to per-request latency metrics, and compounds to roughly $1,400/month per idle-heavy H100 at on-demand rates. It is real, large, and cheaply measurable — this entire experiment cost $2.96 of pod time — and, as the burst pattern demonstrates, the capacity to recover it through consolidation exists, where deployment constraints allow.

Data availability

Published alongside this report: results.json (per-run summaries, seeded arrival schedules, per-request send/first-token/completion timestamps), 15 raw 100 ms NVML sample streams (samples_<pattern>_<run>.csv), experiment.log, vllm.log. The harness (sampler.py, loadgen.py, run.py) aborts automatically if the saturating control reads below 95% busy. Reproduction:

pip install vllm pynvml numpy aiohttp
python run.py --hourly-rate 2.00

References

1. TR-2026-01: Phase-Level GPU Utilization Profiling of a Production Diffusion Transformer. This series.
2. Qwen Team. Qwen2.5: A Party of Foundation Models. huggingface.co/Qwen/Qwen2.5-1.5B-Instruct

Correspondence: github.com/AgentMulder404. Re-runs that disagree with these numbers are actively solicited — the arrival schedules are seeded and published for exactly that purpose.