Oriel — AI & technology worth noticing, every day.

Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

Paper · https://arxiv.org/abs/2610.37662

Discrete visual tokens represent vision in a discrete form analogous to language, offering a native interface for multimodal modeling under a shared next-token prediction objective. However, quantization error is widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity of discrete visual tokens and positioning continuous visual representations as an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and find quantization error unreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additional codebooks contribute progressively less semantic information. As one approach to alleviating this saturation, we introduce Factorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallel codebooks. It continues to improve at larger bit budgets, ultimately matching continuous visual representations in multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ

First seen by Oriel on .

AI summary

This paper finds discrete visual tokens can match continuous representations for multimodal understanding when their effective bit budget is increased using Factorized Vector Quantization.

  • Quantization error does not reliably predict semantic fidelity; discrete representations with larger errors can produce better multimodal understanding.
  • Downstream performance depends strongly on the effective bit budget, while existing quantizers saturate because additional codebooks provide progressively less semantic information.
  • Factorized Vector Quantization quantizes complementary fixed-dimensional feature views with symmetric parallel codebooks, continuing to improve at larger bit budgets.
  • The model achieves state-of-the-art results among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2% on a nine-metric average.
  • It matches continuous visual representations, including on fine-grained OCR tasks, and provides principles and a practical recipe for competitive discrete representations.

#visual tokens #vector quantization #multimodal learning #ocr

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 90, latest observation ; +2 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 2, latest observation ; +1 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.37662https://arxiv.org/abs/2610.37662 Read discussion