ArXiv Proposes TOPAS: Optimizing Prefix Cache Scheduling for Multi-Agent LLM Serving
Summary:ArXiv introduces TOPAS, a Task-Oriented Prefix-Aware Scheduler, designed to optimize prefix cache management in multi-agent Large Language Model (LLM) serving. By balancing the expected reduction in each task's longest remaining service path against the near-term benefits of downstream prefix reuse, TOPAS enhances decision-making for cache retention and request scheduling. Experimental results demonstrate that TOPAS significantly reduces mean and p99 Job Completion Time (JCT) on synthetic worklo
Background and Challenges
In multi-agent Large Language Model (LLM) serving, prefix caching presents a fundamental tradeoff: retaining a longer system-prompt key-value (KV) cache accelerates future calls but reduces the GPU memory available for batching concurrent requests. Existing schedulers in multi-stage workflows often prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can lead to downstream delays or frequent prefix replacements, thereby prolonging the task-level Job Completion Time (JCT).
Innovations of TOPAS
To address these challenges, ArXiv introduces TOPAS (Task-Oriented Prefix-Aware Scheduler), featuring the following innovations:
- Joint Decision-Making: TOPAS jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution.
- Balanced Optimization: By trading off the expected reduction in each task's longest remaining service path against the near-term benefits of downstream prefix reuse, TOPAS optimizes the balance between cache management and request scheduling.
- Task-Level Aging Mechanism: A task-level aging mechanism is incorporated to prevent starvation, ensuring fair resource allocation across all tasks.
Experimental Results
TOPAS is implemented within the SGLang framework and evaluated on three synthetic DAGs and two MetaGPT software-development workflows. Compared with the best-performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%/49.4% on synthetic workloads and lowers the mean JCT by 9.8% on MetaGPT-SOP and the mean/p99 JCT by 22.0%/26.6% on MetaGPT-TL.
Industry Impact and Developer Recommendations
TOPAS offers a novel approach to prefix cache management in multi-agent LLM serving, particularly beneficial for applications requiring complex multi-stage workflows. Here are some recommendations for developers:
- Optimize Cache Strategies: Developers can leverage TOPAS's joint decision-making mechanism to optimize cache strategies and enhance overall system performance.
- Resource Allocation: In resource-constrained environments, TOPAS's task-level aging mechanism can help allocate GPU resources more effectively.
- Workflow Management: For multi-stage workflows, TOPAS can significantly reduce downstream delays, thereby improving overall efficiency.
— END —Source: ArXiv cs.CL (2026-08-26)
Tags: #ArXiv #Large Language Model #Cache Scheduling #Multi-Agent Systems #Performance Optimization
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments