Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent
Paper · https://arxiv.org/abs/2611.18397
We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.
First seen by Oriel on .
AI summary
Gamer is a generalist vision-language-action framework for long-horizon game tasks, combining multimodal perception, memory, reasoning, reinforcement learning, and real-time inference across 2D and 3D environments.
- Gamer is pre-trained on tens of thousands of hours of keyboard and mouse trajectories from simulated games and embodied environments.
- Sparse-thinking instruction tuning reduces redundant computation while preserving reasoning for complex decisions.
- Stable multi-turn reinforcement learning addresses finite context windows, accumulated rollout errors, and sparse rewards during long-horizon interaction.
- A cloud-edge inference engine concurrently serves numerous simulated game environments, supports unbounded-context execution, and maintains low online latency.
- Across challenging 3D open-world games and conventional 2D benchmarks, Gamer achieves generalization and operational efficiency comparable to skilled players.
#vision-language-action #game agents #reinforcement learning #embodied ai
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 100, latest observation ; +14 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 4, latest observation ; +2 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2611.18397 | https://arxiv.org/abs/2611.18397 Read discussion |