ZICQ
中 Log in / Sign up
Newsroom LLMs & Foundation Models #ArXiv #Large Language Model #Cache Scheduling #Multi-Agent Systems #Performance Optimization

ArXiv Proposes TOPAS: Optimizing Prefix Cache Scheduling for Multi-Agent LLM Serving

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 15 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:ArXiv introduces TOPAS, a Task-Oriented Prefix-Aware Scheduler, designed to optimize prefix cache management in multi-agent Large Language Model (LLM) serving. By balancing the expected reduction in each task's longest remaining service path against the near-term benefits of downstream prefix reuse, TOPAS enhances decision-making for cache retention and request scheduling. Experimental results demonstrate that TOPAS significantly reduces mean and p99 Job Completion Time (JCT) on synthetic worklo


Background and Challenges

In multi-agent Large Language Model (LLM) serving, prefix caching presents a fundamental tradeoff: retaining a longer system-prompt key-value (KV) cache accelerates future calls but reduces the GPU memory available for batching concurrent requests. Existing schedulers in multi-stage workflows often prioritize either immediate prefix locality or overall workflow progress. However, under a shared KV cache budget, optimizing either objective in isolation can lead to downstream delays or frequent prefix replacements, thereby prolonging the task-level Job Completion Time (JCT).

Innovations of TOPAS

To address these challenges, ArXiv introduces TOPAS (Task-Oriented Prefix-Aware Scheduler), featuring the following innovations:

  • Joint Decision-Making: TOPAS jointly decides which agent prefixes to keep in the cache and which requests to schedule for execution.
  • Balanced Optimization: By trading off the expected reduction in each task's longest remaining service path against the near-term benefits of downstream prefix reuse, TOPAS optimizes the balance between cache management and request scheduling.
  • Task-Level Aging Mechanism: A task-level aging mechanism is incorporated to prevent starvation, ensuring fair resource allocation across all tasks.

Experimental Results

TOPAS is implemented within the SGLang framework and evaluated on three synthetic DAGs and two MetaGPT software-development workflows. Compared with the best-performing baseline for each workload and metric, TOPAS reduces the mean/p99 JCT by up to 39.8%/49.4% on synthetic workloads and lowers the mean JCT by 9.8% on MetaGPT-SOP and the mean/p99 JCT by 22.0%/26.6% on MetaGPT-TL.

Industry Impact and Developer Recommendations

TOPAS offers a novel approach to prefix cache management in multi-agent LLM serving, particularly beneficial for applications requiring complex multi-stage workflows. Here are some recommendations for developers:

  • Optimize Cache Strategies: Developers can leverage TOPAS's joint decision-making mechanism to optimize cache strategies and enhance overall system performance.
  • Resource Allocation: In resource-constrained environments, TOPAS's task-level aging mechanism can help allocate GPU resources more effectively.
  • Workflow Management: For multi-stage workflows, TOPAS can significantly reduce downstream delays, thereby improving overall efficiency.

Source: ArXiv cs.CL (2026-08-26)

— END —

Tags: #ArXiv #Large Language Model #Cache Scheduling #Multi-Agent Systems #Performance Optimization

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…