Oriel — AI & technology worth noticing, every day.

Cache Engineering for Agents

Paper · https://arxiv.org/abs/2610.28577

Large Language Model (LLM) agents can handle increasingly complex tasks, navigating long trajectories of observations and actions. As these trajectories grow, key-value (KV) cache I/O and storage become the dominant cost. Trajectories can also exceed the LLM's maximum context length, requiring control over active cache growth. Prior work uses context engineering to reduce cache size by trimming or summarizing history. However, modifying the trajectory prefix invalidates subsequent cached states and triggers costly re-prefilling of retained context. Task accuracy can also suffer when too little context is retained to limit re-prefill costs. We propose cache engineering as a new scheme. It allows agents to directly control their active KV cache without modifying the trajectory or re-prefilling retained context. Specifically, agents can proactively use drop to exclude selected cache from future attention and repos to adjust retained keys' position encodings. We formalize these cache operations and identify history dependence as a serving challenge: identical token sequences can yield different cache states under different cache operation sequences, making prefix matching insufficient for correct cache reuse. Thus, we co-design the AKV serving engine. It provides a stateless interface compatible with existing serving APIs, while internally managing stateful cache reuse, eviction, and restoration. Experiments show that a tool-response rolling-drop cache policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7%, compared with conventional threshold-triggered summarization. With AKV, it achieves up to 1.2x the output throughput and a 1.5x speedup in P95 request latency. Recursive self-improvement of cache policies further reduces cost by 7.1% over expert design. Our code is at https://github.com/thu-nics/AKV.

First seen by Oriel on .

AI summary

This paper introduces cache engineering and the AKV serving engine, enabling LLM agents to control active KV caches without rewriting trajectories or re-prefilling retained context.

  • Agents use drop to exclude selected cache entries from future attention and repos to adjust retained keys' position encodings.
  • Cache operation histories make identical token sequences produce different cache states, so prefix matching alone cannot ensure correct cache reuse.
  • The AKV engine exposes a stateless interface while internally managing stateful cache reuse, eviction, and restoration.
  • A tool-response rolling-drop policy improves average benchmark scores by 1.7 points and reduces average cost by 27.7% versus threshold-triggered summarization.
  • AKV delivers up to 1.2x higher output throughput and 1.5x faster P95 request latency, while recursive policy improvement cuts cost by another 7.1%.

#llm agents #kv cache #model serving #context engineering

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 161, latest observation ; +77 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 1, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.28577https://arxiv.org/abs/2610.28577 Read discussion