vLLM Overload Study Shows Cancellation Alone Cannot Stop Retry Storms

Summary:A study of overloaded vLLM deployments finds that client- and proxy-side retries can keep an inference service unstable even after incoming load falls below capacity. Propagating cancellation did not restore service in any experiment, and only 12–38% of calls arriving after the overload were completed. Scheduler traces show that retried executions typically began prefill less than half a second before timing out and consumed 79–85% of the prompt tokens later wasted by cancellation; vLLM's prompt

---

Core Finding

In a vLLM deployment with automatic retries, recovery from overload fails because clients and proxies continually recreate work after requests time out. The study reproduces overload conditions with three open models and uses scheduler traces from one run to map every client attempt to its corresponding execution inside vLLM.

The experiments show that:

• With retries enabled, cancellation alone did not restore service in any run;
• Only 12%–38% of calls arriving after the overload were delivered;
• Retried executions typically began prefill less than half a second before their deadlines expired;
• Executions later aborted had already consumed 79%–85% of the prompt tokens computed by the engine;
• vLLM's prompt-token counter missed roughly one-third to one-half of all computed prompt tokens;
• A retry budget restored service by shedding retries, but left many calls undelivered;
• Request coalescing delivered every call, although some completed late, provided cancellation had not already aborted the running execution.

Overload control should therefore be evaluated through the executions introduced by retries and the responses clients actually receive—not merely arrival rates, active-request gauges, or whether cancellation signals were emitted.

Why Cancellation Alone Fails

A typical LLM API path contains a client, a gateway or proxy, and an inference engine. When a client times out, it retries; a proxy may forward that retry again; and vLLM maps these attempts to internal request executions.

Cancellation appears to reclaim work from callers that have left, but the relevant timing undermines that intuition:

1. After a client timeout, the original request may still be queued, prefilling, or decoding;
2. A retry enters the scheduler before the timeout window closes and may compete for queue position, KV Cache, and compute slots;
3. When cancellation arrives, the earlier execution has usually completed a substantial amount of prompt prefill;
4. Cancellation can stop one execution, but it cannot retract the queue pressure created by its replacement.

In other words, cancellation stops an execution, while retries create another one. If the client cannot distinguish the original attempt from retries and continues applying pressure after its deadline, cancellation only recovers part of the running computation.

The traces reveal a cost amplification effect. A retried execution often began prefill less than half a second before timing out, showing that these were not long-lived zombie requests but fresh executions admitted near the deadline boundary. Because prefill work generally grows with input length, a long prompt can consume substantial GPU time and KV Cache even when the request can no longer finish in time.

Token Accounting Can Hide the Waste

The paper finds that vLLM's recorded prompt-token count differs substantially from the scheduler traces, with an omission of roughly one-third to one-half. This affects operational decisions:

• Overload-loss estimates can undercount prefill work wasted by cancellation;
• Capacity models based on token throughput or cache-hit statistics may be systematically wrong;
• Throughput, TTFT, and effective-completion calculations may use inconsistent numerator and denominator definitions;
• Autoscaling controllers may infer spare capacity from an understated token rate.

Teams should distinguish client-submitted tokens, engine-received tokens, tokens entering prefill, and tokens contributing to delivered output. Existing engine counters alone do not fully describe the resources consumed by retry storms.

Trade-offs Among Three Controls

1. Propagation of Cancellation

Cancellation is useful for reclaiming work that clients have explicitly abandoned, particularly during long generations. Its value depends heavily on arrival timing. If prefill has already started, most of the computation has been spent. If retries continue, the engine can retain high admission pressure and low completion rates.

Production systems should propagate a request identifier and retry relationship so that clients, proxies, and the engine can recognize all attempts belonging to one logical call. Sending cancellation to the engine while allowing a proxy to keep retrying is not a complete overload-control mechanism.

2. Retry Budgets

A total attempt limit, cumulative attempt deadline, or per-client concurrent retry budget can prevent retry amplification. In the paper, this strategy restored service by dropping retries, but many calls from the overloaded period remained undelivered.

This is an explicit availability trade-off: sacrifice part of the request success rate to keep queues and GPU resources bounded. A budget that is too strict suppresses recovery-period traffic, while one that is too loose provides little protection. A dynamic policy based on request priority, remaining service time, and queue depth is preferable to a fixed attempt count.

3. Request Coalescing

Request coalescing maps multiple logically identical calls to a shared execution and distributes the result to all waiters. In the study, it preserved every call after the overload, although some responses arrived late.

The approach is well suited to idempotent or mergeable calls with exactly identical prompts and generation parameters, particularly when shared-prefix caching can reuse the relevant computation. Two boundaries matter:

• Semantic requirement: merged calls must genuinely be allowed to share one output; near-identical text alone is insufficient.
• Timing requirement: if cancellation has already aborted the original execution, new waiters can lose computation that could otherwise have been reused.

Coalescing should therefore be designed jointly with cancellation. Whether a cancelled shared execution should stop, be rescheduled, or continue for other waiters depends on the service-level objective.

Implications for vLLM Operations and Evaluation

An overload test should not treat an arrival rate below nominal capacity as proof of recovery. Operators should track:

• Effective completion rate: logical calls that return a response, not raw HTTP attempts;
• Retry amplification factor: total client attempts divided by original logical calls;
• Wasted execution cost: prompt tokens, GPU time, and KV Cache consumed by cancelled executions;
• Cancellation benefit: the proportion of work completed before cancellation arrived;
• Tail latency: coalesced, queued, and dropped requests should be analyzed separately;
• Recovery time: the interval between load returning below a threshold and completion quality stabilizing.

Because of the token-accounting gap, benchmark comparisons should prefer scheduler traces or execution-level instrumentation over API aggregates alone. Any comparison of cancellation, retry budgets, and coalescing must hold request semantics, client deadlines, proxy retry policy, model length, batch composition, and hardware configuration constant. Otherwise, differences in observed latency may reflect workload construction rather than the control mechanism.

Production Recommendations

Teams operating vLLM can implement a four-layer defense:

1. Limit retries at the proxy: enforce maximum attempts, exponential backoff with jitter, and per-client concurrent retry caps. Prefer retrying connection failures that never entered computation over requests already prefilling when they time out.
2. Separate logical calls from execution attempts: ensure cancellation covers all pending attempts for the same logical request, preventing an old execution to be canceled only for a new one to appear.
3. Use request coalescing selectively: merge only requests with strictly compatible parameters, and ensure the cancellation protocol does not prematurely destroy reusable execution state.
4. Instrument at execution level: track arrival QPS, prefilled tokens, execution attempts, successful completions, and actual GPU consumption together instead of relying on a single aggregate token counter.

The paper's contribution is not a new scheduling algorithm. It turns the familiar retry-timeout-cancellation chain into measurable system behavior and shows that overload recovery is a closed-loop problem spanning clients, proxies, and the inference scheduler.

---

Source: ArXiv AI (cs.AI) (2026-11-10)

— END —

Tags: #vLLM Follow #Inference Scheduling Follow #Overload Control Follow #Request Retries Follow #Request Coalescing Follow