Octopus: One Brain, Many Branches for Latent-KV Multi-Agent Reinforcement Learning
Paper · https://arxiv.org/abs/2611.06956
arXiv:2611.06956v1 Announce Type: new Abstract: Multi-agent LLM systems explore in parallel by spawning sub-agents with isolated context windows, whereas single-agent systems append every search and verification step into one growing trajectory. However, existing multi-agent frameworks duplicate prefix KV caches across branches, exchange findings by re-prefilling text summaries, and flatten or drop cross-branch dependencies during reinforcement learning (RL). We introduce Octopus, a single-model multi-agent RL framework in which all threads share one backbone and coordinate over a GPU-resident directed acyclic graph (DAG) of KV states through five primitives (fork, read_thread, wait_threads, rewind, commit). Octopus combines three co-designed mechanisms: (1) Zero-Copy Radix Forking, which spawns branches as $\mathcal{O}(1)$ cursor views over shared physical KV pages; (2) Attention-Only KV Mounting, which mounts a sibling thread's terminal KV blocks directly into full-attention layers while bypassing linear-recurrent states (Gated DeltaNet / Mamba-2), cutting off-task hijacking by $21\times$ ($57.9\% \to 2.7\%$) while retaining $92.2\%$ of full-text mutual information at zero extra KV allocation; and (3) a Packed DAG-GRPO Trainer, which packs all episode branches into one block-sparse attention pass and assigns counterfactual branch credit via leave-one-branch-out marginal rewards. Trained end-to-end with GRPO on Qwen3.5-4B over a sparse BM25 corpus, Octopus reaches $49.6\%$ accuracy on BrowseComp-Plus ($+17.5$ points over single-agent GRPO at 143K context and $+12.5$ points over text-summary multi-agent RL) and $51.2\%$ on OpenResearcher, outperforming dense-retrieval agents up to $250\times$ larger while increasing training throughput $3.3\times$ and cutting peak KV memory $2.8\times$.
First seen by Oriel on .
AI summary
The Octopus paper presents a single-model multi-agent RL framework that shares latent KV state across branches, improving research accuracy while reducing memory use and increasing training throughput.
- Octopus represents agent threads as a GPU-resident directed acyclic graph of KV states coordinated through fork, read_thread, wait_threads, rewind, and commit.
- Zero-Copy Radix Forking creates O(1) cursor views over shared physical KV pages, avoiding duplicated prefix caches.
- Attention-Only KV Mounting reduces off-task hijacking from 57.9% to 2.7% while retaining 92.2% of full-text mutual information without extra KV allocation.
- Packed DAG-GRPO packs episode branches into one block-sparse attention pass and uses leave-one-branch-out marginal rewards for counterfactual credit.
- On Qwen3.5-4B, Octopus reaches 49.6% on BrowseComp-Plus and 51.2% on OpenResearcher, outperforms agents up to 250× larger, and improves throughput 3.3× while reducing peak KV memory 2.8×.
#multi-agent reinforcement learning #kv cache #llm agents #sparse attention
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.06956v1 | https://arxiv.org/abs/2611.06956 Read discussion |