Token Pooling-Aware Training for Multi-Vector Embedding Models
Web page · https://huggingface.co/blog/token-pooling
First seen by Oriel on .
AI summary
This Hugging Face post presents pooling-aware training for multi-vector embedding models, preserving retrieval quality when Ward-clustered document token vectors are compressed to reduce storage.
- HierarchicalTokenPooling merges similar token embeddings with Ward hierarchical clustering, represents each group by its normalized mean, and leaves query embeddings unchanged by default.
- Pooling-aware training averages unpooled and compressed retrieval losses, using a straight-through estimator so encoder parameters learn which information must survive clustering.
- With pool_factor=4, pooling-aware training reached 51.23 mean nDCG@10 versus 49.03 for ordinary training, a 2.20-point gain across NanoBEIR@13.
- The trained example reduced document vectors from 58 to 16 while preserving the ranking of Mars, Jupiter, and Venus in a Red Planet query.
- The implementation supports fixed maximums, proportional pool factors, calibrated adaptive thresholds, GPU kernels, and Sentence Transformers loss wrappers for custom deployment budgets.
#multi-vector retrieval #token pooling #sentence transformers #embedding models
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Blog | https://huggingface.co/blog/token-pooling | https://huggingface.co/blog/token-pooling Read discussion |