StateBench: Testing how language models track state changes over long documents
Web page · https://huggingface.co/blog/allenai/statebench
First seen by Oriel on .
AI summary
StateBench is a benchmark for measuring how language models track changing information across long documents; recurrent and hybrid models trained on its tasks generalize farther than transformers on harder state-tracking problems.
- The benchmark defines four levels: immutable values, latest-value reassignment, self-referential updates, and cross-variable assignments such as swaps.
- Most pretrained models handled simple tasks but degraded on longer aperiodic inputs and stayed near chance on general finite tasks; task format also affected results.
- Four frontier models scored 13% or lower on the longest general finite problems, with accuracy falling sharply on soluble and general finite tasks as sequences grew.
- Models trained from scratch on up to 256 assignments were tested on up to 64,000; Gated DeltaNet and hybrid models generalized strongly on three levels, while transformers failed to learn general finite tasks.
- The benchmark assumes a fixed number of variables, and the capabilities of constrained Gated DeltaNet variants remain partly unresolved.
#language models #state tracking #transformers #gated deltanet
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Blog | https://huggingface.co/blog/allenai/statebench | https://huggingface.co/blog/allenai/statebench Read discussion |