Oriel — AI & technology worth noticing, every day.

Whispering Wisely: Smart Subset Selection for ASR Fine-Tuning

Paper · https://arxiv.org/abs/2610.32997

arXiv:2610.32997v1 Announce Type: new Abstract: Automatic Speech Recognition (ASR) models such as OpenAIs Whisper have substantially improved speech recognition quality but remain expensive to fine-tune, especially when large amounts of transcribed audio are available. This makes it challenging for many practitioners to adapt these models to new languages or domains. In this work, we introduce an embedding-based subset selection approach for selecting smaller training sets for ASR fine-tuning. Using Whispers pretrained encoder, we map each audio sample into a fixed-dimensional embedding space and apply a greedy k-center algorithm to select diverse and representative subsets of the training data. We evaluate this approach on both Hebrew and Danish, showing that k-center selection often improves upon random sampling, particularly at smaller subset sizes. An analysis of lexical diversity reveals that k-center subsets capture a broader range of linguistic content than random subsets of the same size, which may help explain their observed effectiveness. Overall, our results demonstrate that embedding-based subset selection provides a practical strategy for ASR subset selection in the two evaluated languages, with the clearest recognition benefits observed at smaller sample budgets.

First seen by Oriel on .

AI summary

The paper presents embedding-based subset selection for ASR fine-tuning and finds that greedy k-center sampling can improve recognition over random sampling, especially with small training budgets.

  • The method embeds audio with Whisper’s pretrained encoder and uses a greedy k-center algorithm to choose diverse, representative training samples.
  • Evaluations on Hebrew and Danish show the clearest recognition benefits at smaller subset sizes.
  • Selected subsets contain broader lexical diversity than random subsets of equal size, which may help explain their effectiveness.
  • The approach targets practitioners adapting ASR models to new languages or domains while reducing fine-tuning data requirements.

#automatic speech recognition #whisper #data selection

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.32997v1https://arxiv.org/abs/2610.32997 Read discussion