Octopus Rebuilt Multi-Agent Reinforcement Learning Around a Shared KV DAG
Summary:Octopus is a research framework for long-context multi-agent reinforcement learning that replaces duplicated context branches and text-summary communication with a GPU-resident directed acyclic graph of KV states sharing one model backbone. Its Zero-Copy Radix Forking creates branches as O(1) cursor views over shared physical KV pages, while Attention-Only KV Mounting directly exposes sibling terminal KV blocks to full-attention layers without propagating incompatible recurrent states. The fram
Core Idea: Share State Instead of Duplicating Context
Traditional multi-agent systems give every branch an isolated context, while single-agent systems append every search, verification, and reasoning step to one growing trajectory. Octopus combines these models with a GPU-resident directed acyclic graph of KV states. All threads share one model backbone and interact through five primitives: fork, read_thread, wait_threads, rewind, and commit.
The important contribution is not merely running multiple agents. It is allowing branches to reference shared KV pages directly and preserving structured dependencies between branches throughout reinforcement learning.
Zero-Copy Radix Forking: Creating Shared-Prefix Branches in O(1)
Octopus organizes cached states as physical KV pages in a Radix-tree-like structure. Creating a branch allocates a cursor and lightweight metadata rather than copying the Tensor blocks associated with the shared prefix. Forking is therefore O(1) at the system level, while physical KV capacity grows mainly with newly generated tokens.
This removes one of the largest costs in multi-agent research systems: duplicated prefix KV caches. Octopus allows multiple branches to read the same physical pages and allocates new pages only for independently appended tokens. The tradeoff is implementation complexity. Correct systems must handle page lifetimes, concurrent appends, reference counting, and reclamation. If a branch can mutate a shared page in place, snapshot reproducibility breaks immediately.
Attention-Only KV Mounting: Exposing State Without Refilling It
The second mechanism lets a thread mount a sibling thread's terminal KV blocks without allocating another copy. Text-summary communication requires re-prefilling another thread's results, adding compute and discarding fine-grained context. Direct mounting allows full-attention layers to read the sibling's existing KV representation.
The paper specifically avoids propagating hidden states from Gated DeltaNet, Mamba-2, and other linear-recurrent models. Recurrent states are order-dependent and cannot be mounted as independent KV blocks without introducing cross-thread contamination, described in the paper as off-task hijacking. The reported rate falls from 57.9% to 2.7%, a 21× reduction, while retaining 92.2% of the mutual information available from full text.
This mechanism is not free. Query-to-KV-head mappings under GQA or MQA, RoPE positions, causal masks, thread boundaries, and position offsets must all align. Incorrectly mounted KV can violate causality, while incorrect gradient routing can attribute sibling-thread states to the wrong branch.
Packed DAG-GRPO: Learning Credit Across Branches
Octopus trains with Packed DAG-GRPO, packing all branches from an episode into a block-sparse attention pass. The sparse block matrix encodes within-thread causality, permitted cross-thread reads, and synchronization dependencies, avoiding a separate full Transformer forward for every branch.
Credit assignment is also different from assigning one final trajectory reward to each branch. Leave-one-branch-out marginal rewards compare the contribution observed when a branch is retained or removed, then assign credit to the paths that genuinely helped. This is closer to counterfactual credit assignment than final-answer-only reinforcement learning, but it remains vulnerable to sparse rewards, unstable search quality, and strong correlations between branches.
For trajectory length $L$ and $B$ branches, naive independent attention costs roughly $O(BCL^2)$. A block-sparse implementation approaches $O(BL^2 + E_{\text{DAG}}L_b^2)$, where $E_{\text{DAG}}$ is the number of cross-branch dependency edges and $L_b$ is the sparse block size. Actual speed still depends on KV page layout, block-table management, GPU utilization, and branch imbalance.
Results and Evaluation Boundaries
The paper trains Qwen3.5-4B with GRPO over a sparse BM25 corpus and evaluates BrowseComp-Plus at a 143K context length. Octopus reaches 49.6% accuracy, 17.5 percentage points above single-agent GRPO and 12.5 points above text-summary multi-agent RL. It reports 51.2% on OpenResearcher, 3.3× higher training throughput, and 2.8× lower peak KV memory. The authors also report outperforming dense-retrieval agents with up to 250× more parameters.
The results support the value of shared KV state and joint credit assignment, but important limits remain. The reported setup uses one 4B model and a BM25 retrieval environment. The abstract does not provide independent runs, full hardware details, dense-retrieval configurations, or results in live-web environments. The 49.6% absolute accuracy also shows that long-horizon research remains difficult. Beating a model up to 250× larger should not be generalized to open-ended retrieval tasks where model quality and retrieval data differ.
Deployment Guidance
Teams implementing multi-branch RL should first reproduce Zero-Copy Radix Forking under a fixed policy, then add Attention-Only KV Mounting while validating KV-head mappings, RoPE, causal masks, and thread positions. Production implementations need reference-counted pages, snapshot consistency, GPU-OOM fallback, cross-thread contamination detection, and thread-level cancellation.
For applications, Octopus is most relevant to long-context retrieval, literature research, and hypothesis validation: workloads with parallel branches that share large prefixes. Its primary benefit is training efficiency and state reuse, not necessarily single-request generation latency. Teams optimizing for time to first token, synchronous serving throughput, or tool-call stability should use separate inference benchmarks rather than directly transferring the paper's training-throughput numbers.
Engineering Trade-offs
Octopus exchanges stricter KV sharing and DAG constraints for lower memory use and greater parallel efficiency. It replaces text summaries with direct state mounting, preserving more context while introducing contamination risks and architecture-compatibility constraints. The paper offers a compelling systems design, but the current evidence remains tied to a single model and a static retrieval corpus. Production adoption should prioritize reproducing stability under live web access, dynamic tool use, and multi-GPU execution.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-06)
Tags: #Octopus #Multi-Agent Reinforcement Learning #KV Cache #GRPO #Block-Sparse Attention
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments