Oriel — AI & technology worth noticing, every day.

SWAM: Scaling World Action Models with Cross-Modal Alignment

Paper · https://arxiv.org/abs/2611.12870

World action models jointly predict video and actions conditioned on language. Existing studies focus primarily on architectures and training mechanisms, typically treating language-video alignment acquired through video pretraining as an established foundation. However, introducing action learning may impair this alignment. Principled guidance on data and supervision design for maintaining language-video alignment during action learning remains limited. To investigate this, we curate a pool of approximately 47,000 hours of general videos, egocentric videos, UMI demonstrations, and robot demonstrations. Building on this, we systematically study how language-video supervision, language-video-action supervision, and the joint use of these two forms affect WAM performance. In particular, we examine how language annotation quality affects pretrained language-video capabilities, and how action supervision from UMI and robot demonstrations extends these capabilities to language-video-action modeling. To better scale the model, we further conduct detailed studies on joint training recipe and model architecture. Under a fixed training budget, mixing egocentric videos and UMI demonstrations improves control generalization. Model architecture affects how effectively these experiences are used, while scaling pretraining under the selected configuration brings further gains. The resulting recipe, post-trained on RoboTwin Clean demonstrations only, achieves success rates of 88.10% and 61.56% under Easy (Clean-to-Clean) and Hard (Clean-to-Random) conditions, respectively. Our model achieves 40.00% SR across three real-world tasks, compared with 23.33% for the strongest baseline, indicating better generalization under unseen conditions. We release the curated data, training code, and model weights.

First seen by Oriel on .

AI summary

SWAM studies how data, supervision, training recipes, and architecture preserve cross-modal alignment while scaling world action models for stronger control generalization.

  • The authors curated approximately 47,000 hours of general, egocentric, UMI, and robot demonstration videos.
  • Under a fixed training budget, combining egocentric videos with UMI demonstrations improved control generalization, while architecture determined how effectively the model used these experiences.
  • After post-training only on RoboTwin Clean demonstrations, SWAM achieved 88.10% success in Easy conditions and 61.56% in Hard conditions.
  • Across three real-world tasks, the model reached 40.00% success, versus 23.33% for the strongest baseline under unseen conditions.
  • The curated data, training code, and model weights are being released.

#world action models #cross-modal alignment #robot learning #video models

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 24, latest observation ; +3 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 7, latest observation ; +5 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2611.12870https://arxiv.org/abs/2611.12870 Read discussion