NIXL Proposes Calibrated Routing Strategy to Revolutionize LLM Serving Performance
Summary:The NIXL team has introduced a calibrated routing strategy for disaggregated LLM serving architectures. This strategy estimates the additional completion time for each instance using exact prompt length, predicted output length, post-admission KV cache pressure, and SLO class. Developed and validated in a discrete event simulator, the method achieved the highest mean goodput (0.864) across various mixed and bursty traffic scenarios, outperforming round-robin, least-loaded, and length heuristic m
Background and Challenges
The disaggregated LLM serving architecture separates compute-heavy prefill and memory-heavy decode operations into different GPU pools. While systems like DistServe, Splitwise, and Mooncake have enabled this separation, the routing strategy still determines which instances handle each request.
Key Innovation
The NIXL team proposes a calibrated routing strategy that estimates the additional completion time for each instance using:
- Exact Prompt Length: Precise calculation based on the input prompt length.
- Predicted Output Length: Prediction of output length to optimize resource allocation.
- Post-Admission KV Cache Pressure: Assessment of KV cache occupancy.
- SLO Class: Classification based on Service Level Objective (SLO).
The strategy is developed in a discrete event simulator and validated on eight NVIDIA A40 GPUs, each running a vLLM engine, with NIXL transferring KV caches between pools. All workloads run at measured saturation.
Experimental Results
Across three mixed and bursty traffic scenarios, the calibrated routing strategy achieved the highest mean goodput (0.864), compared to 0.835 to 0.847 for round-robin, least-loaded, and length heuristic methods. The strategy also showed the lowest variance across scenarios, outperforming round-robin and length heuristic methods in all three scenarios and least-loaded in two. In the third scenario, it trailed by 0.003, within run-to-run noise.
Importance of Hardware Calibration
Hardware calibration is crucial: using simulator-derived constants costs 4.5 goodput points and roughly 40 percent of the tail latency advantage, reducing the scorer to little more than queue counting. As the decode pool size and traffic heterogeneity increase, the benefits also increase, but in pools with only three instances, queue counts are often enough. Under extreme scarcity, greedy cost minimization concentrates requests on the cheapest scored instance, and blind spreading performs better.
Conclusion
With calibrated costs, the learned router matches the goodput of round-robin using six GPUs instead of seven, demonstrating its significant advantage in resource utilization efficiency.
— END —Source: ArXiv AI (cs.AI) (2026-09-16)
Tags: #NIXL #LLMs & Foundation Models #Routing Strategy #GPU Optimization #Disaggregated Architecture
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments