To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems
Paper · https://arxiv.org/abs/2610.28637
arXiv:2610.28637v1 Announce Type: new Abstract: Disaggregated \textsc{LLM} serving separates prefill and decode across node pools, placing a network fabric between \textsc{KV}-cache computation and consumption. This breaks a key assumption of classical cache policies: a miss no longer incurs only local recomputation, but also network transfer to the decode node. Likewise, admitting a block to the global cache requires an additional transfer, so poor retention decisions waste both memory and bandwidth. We present \textbf{KVLearn}, a learning-based framework for cost-aware \textsc{KV}-cache retention in disaggregated serving. KVLearn includes: (i) a lightweight Prefix Reuse Predictor (\textsc{PRP}) that estimates reuse probability from structural and temporal prefix features without modifying model weights; (ii) a Cost-Aware Retention Score (\textsc{CARS}) that converts reuse probability into keep/admit decisions using recomputation, transfer, and storage costs; and (iii) an Adaptive Threshold Controller (\textsc{ATC}) that adjusts admission thresholds online from observed hit rates and memory pressure. We integrate KVLearn into a globally disaggregated serving topology and evaluate it on text and multimodal workloads, including large image/video-derived \textsc{KV} blocks with heterogeneous reuse patterns. KVLearn reduces end-to-end time-to-first-token by up to 56% versus No-Cache, 38% versus LRU-Pool, and 33% versus Mooncake-style baselines, while cutting inter-node \textsc{KV} transfer volume by up to 53% versus LRU-Pool. On MM-Session, throughput remains within ${\sim}5%$ of oracle. Code is available at https://github.com/FastLM/KVLearn.
First seen by Oriel on .
AI summary
KVLearn is a learning-based KV-cache retention framework for disaggregated LLM serving that balances reuse probability against recomputation, transfer, and storage costs.
- Its Prefix Reuse Predictor estimates reuse from structural and temporal prefix features without modifying model weights.
- Cost-Aware Retention Score makes keep/admit decisions, while an Adaptive Threshold Controller adjusts admission thresholds using observed hit rates and memory pressure.
- Across text and multimodal workloads, KVLearn reduces time-to-first-token by up to 56% versus No-Cache, 38% versus LRU-Pool, and 33% versus Mooncake-style baselines.
- It cuts inter-node KV transfer volume by up to 53% versus LRU-Pool; on MM-Session, throughput stays within approximately 5% of oracle.
#llm serving #kv cache #cache management #multimodal
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2610.28637v1 | https://arxiv.org/abs/2610.28637 Read discussion |