Oriel — AI & technology worth noticing, every day.

baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity

Paper · https://arxiv.org/abs/2610.30154

arXiv:2610.30154v1 Announce Type: new Abstract: Nonlinear recurrent networks remain useful for streaming sequence modeling, yet gated cell design is still dominated by LSTM and GRU. We study whether recurrent computation can be reallocated away from gate transforms and toward candidate generation. We propose baseVultra, a compact recurrent cell that concentrates the dense recurrent budget on a single H x H candidate-generation matrix, while replacing recurrent gate matrices with O(H) diagonal interactions. The candidate update combines this dense branch with two low-rank auxiliary branches, an EMA cache, and learnable output blending; the gates additionally receive lightweight second-order access to h_{t-2} through diagonal terms. On TinyShakespeare character-level language modeling, baseVultra achieves a validation loss of 1.594 in the compact one-layer 200K setting, compared with 1.604 for a parameter-matched LSTM and 1.616 for a parameter-matched GRU, while its dense recurrent matrix-vector computation is about 54% of GRU under matched hidden size for this configuration. On text8, a roughly 45x larger benchmark, baseVultra is competitive with full-gate cells (1.854 bpc vs. 1.849 for GRU and 1.837 for LSTM). On formal-language probes, the largest gains appear under constrained hidden-state capacity (0.991 vs. 0.761 GRU and 0.691 LSTM OOD2 accuracy on Counter at H=16) and vanish as hidden size grows. These results characterize where and when candidate-focused recurrent capacity allocation helps: the benefit is concentrated in compact regimes, while full-gate cells remain competitive at scale.

First seen by Oriel on .

AI summary

The paper introduces baseVultra, a compact recurrent cell that reallocates computation from gate transforms to candidate generation, with reported advantages concentrated in small-capacity settings.

  • baseVultra uses one dense H x H candidate-generation matrix, O(H) diagonal recurrent gate interactions, two low-rank auxiliary branches, an EMA cache, and learnable output blending.
  • On TinyShakespeare, it reaches validation loss 1.594 in a one-layer 200K-parameter setting, versus 1.604 for a parameter-matched LSTM and 1.616 for a GRU.
  • With matched hidden size in that configuration, its dense recurrent matrix-vector computation is about 54% of GRU's.
  • On text8, it is competitive but trails LSTM: 1.854 bpc versus 1.849 for GRU and 1.837 for LSTM.
  • Formal-language gains are strongest at constrained hidden-state capacity and disappear as hidden size grows; on Counter OOD2 at H=16, accuracy is 0.991 versus 0.761 for GRU and 0.691 for LSTM.

#recurrent neural networks #sequence modeling #language modeling

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2610.30154v1https://arxiv.org/abs/2610.30154 Read discussion