Oriel — AI & technology worth noticing, every day.

The Best Second Opinion Is Right Where the Primary Is Wrong: Partner Selection for Selective Span Tagging

Paper · https://arxiv.org/abs/2610.37678

arXiv:2610.37678v1 Announce Type: new Abstract: A span tagger that ranks its outputs by risk can defer the riskiest ones to a person, and a second model can sharpen this ranking. Which second model should it be? Partner selection is conditional, not global: the best second opinion is the model that is right where the primary is wrong. Across eight tagger types, three seeds and three English benchmarks (576 evaluated primary-partner pairings), choosing the partner by held-out accuracy on the primary's errors recovers 82% of the best partner's gain, against 69-70% for span F1 or a random pick. Frozen before a fourth dataset was chosen, the rule replicated there (Spearman 0.40 vs. 0.04 for span F1). Partner value is also directional. The predictive distribution is the second opinion: hard labels discard half of a partner's value, and scoring the served label improves 94% of pairings. Fixed before the fourth dataset was chosen, this score also replicated there. For LoRA-tuned decoders, independently trained partners beat every readout of the primary network. Applied zero-shot to 51 MASSIVE locales, every primary has a helpful trained partner in every locale, and the preferred partner follows competence.

First seen by Oriel on .

AI summary

The paper finds that selecting a second span tagger based on its accuracy specifically on the primary model’s errors improves selective tagging decisions across datasets and locales.

  • Across 576 primary-partner pairings, error-conditioned selection recovers 82% of the best partner’s gain, versus 69–70% for selection by span F1 or random choice.
  • The selection rule replicated on a fourth dataset held out from the initial analysis, with Spearman correlation 0.40 versus 0.04 for span F1.
  • Using a partner’s predictive distribution rather than hard labels preserves more value, and scoring the served label improves 94% of pairings.
  • For LoRA-tuned decoders, independently trained partners outperform readouts of the primary network; in 51 MASSIVE locales, every primary has a helpful trained partner.

#span tagging #selective prediction #model ensembles #lora

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.37678v1https://arxiv.org/abs/2610.37678 Read discussion