vLLM Introduces Push-based KV Transfer for Enhanced LLM Inference Performance

Summary:vLLM, an open-source inference engine for large language models (LLMs), introduces a push-based KV transfer mechanism that significantly enhances inference performance. Benchmarks on Amazon EC2 P5en instances demonstrate up to a 3x improvement in time-to-first-token (TTFT) and a 30% reduction in time-per-output-token (TPOT) compared to the previous pull-based approach. This mechanism optimizes KV cache transfer by moving coordination off the critical path, reducing latency and increasing through

---

Core Breakthrough

vLLM, an open-source inference engine for large language models (LLMs), has introduced a push-based KV transfer mechanism to address the inefficiency of KV cache transfer during LLM inference. LLM inference consists of two phases: prefill and decode. The prefill phase processes the input prompt and builds the KV cache, while the decode phase uses this cache to generate output tokens one at a time. Traditionally, these phases run on separate engine instances, requiring KV cache transfer between the prefill and decode instances, which becomes a critical bottleneck affecting time-to-first-token (TTFT).

Technical Mechanism Analysis

The newly introduced push mode optimizes the KV transfer process through the following methods:

1. Eliminating Serialization Bottlenecks: The push mode removes the serialization bottleneck of the traditional pull mode by pre-registering target blocks and working in parallel with the prefill process. The decode instance registers its target blocks before the prefill completes, and the KV transfer begins as soon as the producer's blocks are ready, avoiding the ID discovery round trip on the consumer side.

2. Transfer Setup Time: The push mode shifts the transfer setup time from the decode instance to the prefill instance and runs it on a background thread. This means the decode instance does not need to spend additional cycles on setup during the latency-critical token generation phase.

3. Threading Model Optimization: The push connector uses a dedicated background transfer thread to separate model execution from KV transfer preparation. At the start of a request, the decode main loop allocates its destination KV blocks and registers them with the prefill instance, while the prefill main loop computes the prompt KV cache. The prefill main loop hands the KV blocks to the background thread once both are ready and quickly returns to model execution.

Engineering Trade-offs and Benchmark Results

In benchmarks conducted on Amazon EC2 P5en instances, the push mode demonstrated significant improvements in a 1P1D deployment. Compared to the pull mode, the push mode achieved up to a 3x improvement in TTFT and a 30% reduction in TPOT. The results show that the push mode excels in high request rates and long input prompts, as the pull mode suffers from latency due to synchronous NIXL descriptor processing in the decode worker thread, while the push mode moves this processing to a background thread, avoiding delays on the critical path.

Developer Deployment Recommendations

For applications requiring low TTFT, the push mode is the preferred choice. Developers can enable the push mode by configuring the prefill and decode servers to use NixlPushConnector. If the application requires load-aware routing or bidirectional KV transfer, the pull mode should be used. vLLM supports both modes, allowing developers to choose the approach that best fits their deployment needs.

Future Directions

As a future improvement, the vLLM team is exploring bringing the background NIXL threading model to the pull mode connector, so that pull mode workloads can benefit from the same optimizations. This work is being discussed in a GitHub pull request.

---

Source: AWS Machine Learning Blog (2026-10-19)

— END —

Tags: #vLLM Follow #KV transfer Follow #LLMs & Foundation Models Follow #Inference Optimization Follow #Open Source AI Follow