vLLM Releases Bi-directional KV Cache Transfer to Enhance Multi-turn Agentic Workload Performance
Summary:vLLM's open-source inference engine introduces bi-directional KV cache transfer, allowing prefill and decode nodes to reuse previously computed context across conversation turns, significantly reducing Time-To-First-Token (TTFT). Benchmarks on Amazon EC2 P5en instances show a 1.6x reduction in TTFT for Qwen3-32B and a 3.3x reduction for GLM-5-FP8 in multi-turn agentic workloads. This mechanism enhances accelerator utilization and reduces redundant computations, improving inference efficiency for
Technical Mechanism Analysis
The bi-directional KV cache transfer mechanism introduced by vLLM addresses the latency issues caused by redundant computations in multi-turn conversational scenarios. In the traditional Prefill-Decode (P-D) disaggregated architecture, KV cache is transferred only from the prefill node to the decode node in a single direction. The bi-directional transfer mechanism allows the decode node to return KV cache to the prefill node after generating a response, making it available for subsequent turns. This is achieved through the following steps:
-
KV Cache Retention and Reuse: The decode node retains eligible KV cache entries after generating a response and returns their metadata to the prefill node in the next request. The prefill node decides whether to pull KV cache from the decode node or recompute based on the configured reuse threshold.
-
KV Cache-Aware Router: vLLM introduces a lightweight KV cache-aware router that manages the mapping between the client, session ID, and KV cache metadata. The router records and updates the location and expiry of KV cache in each conversation turn, ensuring the prefill node can accurately retrieve reusable KV cache.
-
NIXL KV Connector Enhancement: The NVIDIA Inference Xfer Library (NIXL) KV connector is extended to allow the prefill node to pull KV blocks from remote sources. This enables more efficient collaboration between the prefill and decode nodes, reducing data transfer latency.
Engineering Trade-offs and Performance
While the bi-directional transfer mechanism improves performance, it also involves some trade-offs:
-
Increased KV Transfer Time: Although bi-directional transfer reduces redundant computations, the KV cache transfer itself introduces some latency. In the benchmark, KV transfer time ranges from 30ms to 130ms.
-
Cache Space Usage: The decode node needs to retain more KV cache, which increases显存 usage. However, by configuring appropriate TTL (Time-To-Live) and reuse thresholds, cache space usage can be effectively controlled.
In the benchmark on Amazon EC2 P5en instances, the bi-directional KV transfer mechanism showed excellent performance in multi-turn conversational scenarios:
-
Qwen3-32B: TTFT decreased from 158ms (2k prompt length) to 122ms (with bi-directional transfer enabled), and from 778ms to 460ms at 24k prompt length, achieving a 1.6x improvement.
-
GLM-5-FP8: TTFT decreased from 500ms (2k prompt length) to 358ms (with bi-directional transfer enabled), and from 2334ms to 702ms at 20k prompt length, achieving a 3.3x improvement.
Developer Implementation and Deployment Recommendations
To deploy the bi-directional KV transfer mechanism, developers should follow these steps:
-
Upgrade vLLM Version: Ensure you are using vLLM v0.21.0 or later.
-
Configure NixlConnector: Enable bi-directional KV transfer on both prefill and decode servers.
-
Deploy KV Cache-Aware Router: Use the example proxy code provided by vLLM and adapt it to your environment.
-
Adjust Configuration Parameters: Tune the KV reuse threshold and TTL based on model computation time, KV size, and network performance to achieve optimal performance.
Conclusion
The bi-directional KV transfer mechanism significantly improves the inference efficiency of multi-turn conversational tasks by reducing redundant computations and optimizing cache utilization. The mechanism has shown excellent performance in benchmarks on Amazon EC2 P5en instances and is particularly suitable for intelligent agent applications that require processing long contexts and multi-turn conversations.
— END —Source: AWS Machine Learning Blog (2026-10-19)
Tags: #vLLM #Bi-directional KV Transfer #Multi-turn Conversations #Inference Optimization #P-D Disaggregation
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments