Oriel — AI & technology worth noticing, every day.

StateBench: Testing how language models track state changes over long documents

Web page · https://huggingface.co/blog/allenai/statebench

First seen by Oriel on .

AI summary

StateBench is a benchmark for measuring how language models track changing information across long documents; recurrent and hybrid models trained on its tasks generalize farther than transformers on harder state-tracking problems.

  • The benchmark defines four levels: immutable values, latest-value reassignment, self-referential updates, and cross-variable assignments such as swaps.
  • Most pretrained models handled simple tasks but degraded on longer aperiodic inputs and stayed near chance on general finite tasks; task format also affected results.
  • Four frontier models scored 13% or lower on the longest general finite problems, with accuracy falling sharply on soluble and general finite tasks as sequences grew.
  • Models trained from scratch on up to 256 assignments were tested on up to 64,000; Gated DeltaNet and hybrid models generalized strongly on three levels, while transformers failed to learn general finite tasks.
  • The benchmark assumes a fixed number of variables, and the capabilities of constrained Gated DeltaNet variants remain partly unresolved.

#language models #state tracking #transformers #gated deltanet

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
Hugging Face Bloghttps://huggingface.co/blog/allenai/statebenchhttps://huggingface.co/blog/allenai/statebench Read discussion