Oriel — AI & technology worth noticing, every day.

Recurrent Looped Transformer

Paper · https://arxiv.org/abs/2610.07591

State tracking requires an update at every input, but the depth a Transformer applies to each token is fixed regardless of sequence length. We introduce the Recurrent Looped Transformer (RLT), which splits its layers between a parallel causal encoder and a recurrent decoder. At each token, the decoder merges the encoder output with the previous token's final decoder state, so the computation path grows with sequence length at a fixed per-token cost. On six algorithmic tasks, we compare five splits of eight layers with an eight-layer Transformer over three seeds. Trained on at most 40 bits, two RLT splits generalize parity to 256 bits with 100% accuracy in every seed, while the Transformer stays at chance. On swap-based S_5 permutation tracking at eight times the training length, RLT reaches 97% final-state accuracy versus under 1% for the Transformer, and accuracy increases with decoder depth. On modular arithmetic beyond the training lengths, RLT reaches up to 93% versus 33% for the Transformer. Ablations show that these gains depend on the feedback: removing it drops parity and swap-based S_5 to chance at every split. Updating the feedback once per four-token chunk lets known tokens in a chunk run in parallel and keeps 64-bit parity at 99%, while permutation tracking depends on per-token feedback: chunking lowers length-64 swap-based S_5 from 100% to 20%.

First seen by Oriel on .

AI summary

The paper introduces Recurrent Looped Transformer, whose recurrent decoder extends computation paths with sequence length and substantially improves algorithmic generalization over a fixed-depth Transformer.

  • RLT divides layers between a parallel causal encoder and recurrent decoder, which combines encoder outputs with the previous token’s final decoder state.
  • Across six algorithmic tasks, researchers compare five eight-layer splits with an eight-layer Transformer using three random seeds and training sequences of at most 40 bits.
  • Two RLT configurations achieve 100% parity accuracy at 256 bits, while the Transformer remains at chance; RLT also reaches 97% swap-based $S_5$ accuracy at eight times training length.
  • For modular arithmetic beyond training lengths, RLT reaches up to 93% accuracy versus 33% for the Transformer, with accuracy increasing alongside decoder depth.
  • Removing feedback eliminates the gains, while four-token chunking preserves 99% 64-bit parity but reduces length-64 swap-based $S_5$ accuracy from 100% to 20%.

#transformers #sequence modeling #algorithmic generalization #state tracking

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 24, latest observation ; -11 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 2, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.07591https://arxiv.org/abs/2610.07591 Read discussion