Covariance-aware Scoring: A Geometric Correction for Vision-Language Test-Time Adaptation
Paper · https://arxiv.org/abs/2611.00024
arXiv:2611.00024v1 Announce Type: new Abstract: Test-time adaptation (TTA) enables vision--language models such as CLIP to adapt to distribution shifts without labels or access to source data. Existing methods adapt through feature caching, prototype estimation, or the online accumulation of feature statistics, but most ultimately score a test feature against a class representative in the ambient embedding space, typically through cosine similarity. We identify this shared decision rule as a limitation under anisotropic feature distributions, which means when a small number of directions account for a large fraction of feature variance, the score can be dominated by high-variance directions that are not necessarily discriminative. In light of this, we propose CARe, a covariance-aware scoring for TTA that replaces the ambient geometry with a Mahalanobis geometry induced by the target feature second moment. Theoretically, we show that under a shared-covariance Gaussian model the covariance-aware decision direction coincides with that of the Bayes-optimal linear discriminant, with ambient-space scoring as the isotropic special case, and establish that the potential benefit of covariance-aware scoring grows with feature anisotropy. Empirically, we show that CARe improves adaptation accuracy over its base cosine scoring on both standard and fine-grained distribution-shift benchmarks across two backbones. Consistent with our analysis, the gains are largest on datasets whose features are most anisotropic and vanish as the features approach isotropy.
First seen by Oriel on .
AI summary
CARe improves vision-language test-time adaptation by scoring features in a Mahalanobis geometry derived from target feature statistics, addressing limitations of cosine similarity under anisotropic distributions.
- CARe uses the target feature second moment to replace ambient-space scoring with covariance-aware scoring.
- Under a shared-covariance Gaussian model, its decision direction matches the Bayes-optimal linear discriminant; cosine scoring is the isotropic special case.
- The authors report accuracy gains over base cosine scoring on standard and fine-grained distribution-shift benchmarks across two backbones.
- Gains are largest on datasets with the most anisotropic features and disappear as features approach isotropy.
#vision-language models #test-time adaptation #covariance estimation #distribution shift
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.00024v1 | https://arxiv.org/abs/2611.00024 Read discussion |