OpenESM: Energy-Steered Models Are Scalable Foundation Learners
Paper · https://arxiv.org/abs/2610.33005
arXiv:2610.33005v1 Announce Type: new Abstract: Energy-based models have shown promising properties for domain-specific tasks but remain challenging in modeling general-purpose scenarios. To bridge this gap, we design energy-steered models (ESMs) for foundational language modeling. ESMs integrate energy steering and token generation within a single architecture to leverage the energy-guided search to refine token generation. Specifically, ESMs first optimize a latent state over the energy landscape, then leverages the refined latent state for token generation. By performing such energy-guided pre-training on large-scale corpora, ESMs learn generally effective energy landscape where favorable tokens correspond to lower energy values. To explore the scaling behavior of ESMs, we pre-train ESMs on three large-scale corpora, then evaluate across 26 downstream tasks spanning a broad range of domains. Our results demonstrate that ESMs have promising scaling behaviors with increasing model size and training data, and outperform GPTs on most downstream tasks. These findings highlight ESMs as a promising architecture for large-scale pre-training and general purpose problem solving. We fully release the model weights and training infrastructure to enable the community to completely reproduce and extend our results. GitHub: https://github.com/datamllab/openesm HuggingFace: https://huggingface.co/collections/guan-wang/openesm
First seen by Oriel on .
AI summary
OpenESM introduces energy-steered models that refine latent states through energy-guided optimization before token generation, showing promising scaling and broad downstream performance.
- ESMs combine energy steering and token generation in one architecture, optimizing a latent state over an energy landscape before producing tokens.
- Energy-guided pre-training teaches generally effective landscapes where favorable tokens have lower energy values.
- The authors pre-trained ESMs on three large-scale corpora and evaluated them across 26 downstream tasks spanning multiple domains.
- ESMs scale with model size and training data and outperform GPTs on most evaluated downstream tasks.
- The model weights and training infrastructure are fully released for reproduction and extension.
#energy-based models #language models #pre-training #scaling laws
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2610.33005v1 | https://arxiv.org/abs/2610.33005 Read discussion |