Oriel — AI & technology worth noticing, every day.

ResCom: Activation Residual Compensation for Large Language Model Quantization

Paper · https://arxiv.org/abs/2610.35955

arXiv:2610.35955v1 Announce Type: new Abstract: Low-bit weight--activation quantization reduces the memory and computation needed to deploy large language models, but activation rounding can still degrade accuracy after calibration. Reconstruction from rounded inputs cannot distinguish activations within the same quantization bin. We propose ResCom, a low-rank correction that uses the activation rounding residual to compensate for the resulting layer-output error. An output-aware calibration procedure selects residual directions by their contribution to output reconstruction and computes the correction in closed form, without gradient training or updates to the calibrated weights. On Llama-3-8B with 4-bit weights and activations, rank-32 attention corrections reduce WikiText-2 perplexity from 7.708 to 7.147, improve MMLU by 2.89 percentage points, and raise five-shot GSM8K accuracy from 23.96 to 36.16 percent. Evaluations on models from 1B to 70B parameters and on held-out C4 test the method beyond this setting. The evaluated FP32 implementation adds 2.25 percent to 4K prefill time; a separate compact-storage implementation retains a 27.7 percent peak-memory reduction relative to FP16.

First seen by Oriel on .

AI summary

ResCom is a low-rank, training-free correction that uses activation rounding residuals to recover accuracy lost during low-bit weight–activation quantization of large language models.

  • Output-aware calibration selects residual directions by their contribution to output reconstruction and computes corrections in closed form without changing calibrated weights.
  • On 4-bit Llama-3-8B, rank-32 attention corrections reduced WikiText-2 perplexity from 7.708 to 7.147 and improved MMLU by 2.89 percentage points.
  • Five-shot GSM8K accuracy increased from 23.96 percent to 36.16 percent, with evaluations also covering 1B-to-70B models and held-out C4.
  • The FP32 implementation added 2.25 percent to 4K prefill time, while compact storage retained a 27.7 percent peak-memory reduction versus FP16.

#llm quantization #low-rank correction #model compression #language models

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.35955v1https://arxiv.org/abs/2610.35955 Read discussion