Oriel — AI & technology worth noticing, every day.

Linear-Time Context Compression with State-Space Encoding

Paper · https://arxiv.org/abs/2610.33006

arXiv:2610.33006v1 Announce Type: new Abstract: Transformer-based large language models (LLMs) impose substantial computation and memory costs as context sizes increase, motivating methods that compress a context into a shorter representation before inference. Existing soft-prompt compression methods typically use the same Transformer backbone as used in the target model, where producing the compressed representation leads to significant latency overheads. This restricts them to the offline setting where context is compressed once and then reused across many queries. We propose the Boa Compressor, a plug-and-play online context compressor built on a Mamba state-space model (SSM) that encodes arbitrary-length context in a single linear-time pass, producing a representation on which we condition a frozen target LLM. The SSM is trained by distillation, matching the next-token logit distribution from the LLM ingesting the full context (as teacher) to the same LLM ingesting the compressed representation (as the student). At equal compression ratios, Boa matches or surpasses the performance of existing state-of-the-art offline compression baselines, while being an order of magnitude faster than Transformer-based implementations, thereby reducing end-to-end latency rather than adding compression time overhead. We demonstrate that Boa also scales well to longer contexts and to compression ratios as aggressive as 16x. By enabling effective compression that is cheap enough to run on every query at inference time, Boa becomes a practical and general tool for efficient long-context inference.

First seen by Oriel on .

AI summary

Boa Compressor uses a Mamba state-space model to compress arbitrary-length context in one linear-time pass, enabling online compression that reduces end-to-end latency for frozen target LLMs.

  • It is trained by distillation, matching next-token logit distributions between the target LLM given full context and the same LLM given compressed representations.
  • At equal compression ratios, Boa matches or surpasses state-of-the-art offline compression baselines and runs an order of magnitude faster than Transformer-based implementations.
  • The method scales to longer contexts and compression ratios up to 16x.
  • Its intended use is per-query context compression during inference, rather than offline compression reused across queries.

#mamba #state-space models #context compression #large language models

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.33006v1https://arxiv.org/abs/2610.33006 Read discussion