StreamEval: Evaluating Interactive Streaming Video Generation Models
Paper · https://arxiv.org/abs/2610.28402
As streaming video generation advances, enabling real-time interaction with generated video is becoming increasingly important. We introduce StreamEval, a unified benchmark for interactive streaming video generation spanning five tasks: avatar dialogue, avatar performance, world generation, instruction editing, and reference editing. StreamEval comprises 300 scripted test cases lasting 15-300s, each specifying initial assets, incrementally arriving multimodal requests, and explicit requirements for state changes and content preservation. Performance is assessed along four evaluation dimensions: task execution, state consistency, visual fidelity, and audiovisual coordination, instantiated through ten task-specific metrics on a 0-100 scale. Evaluating 37 distinct models across 300 test cases, we find that high visual quality and identity retention do not necessarily coincide with reliable request execution or sustained state consistency. In particular, request execution remains challenging: the highest task-specific scores are 36.90 for Following among seven avatar-performance models, 25.93 for Following in text-conditioned world generation, and 15.00 for Edit Success among three reported reference-editing models. These findings identify reliable request execution under timing and preservation constraints as a central challenge for advancing omni-modal real-time interactive systems. The leaderboard is available at https://streameval.ai
First seen by Oriel on .
AI summary
StreamEval is an open-source benchmark for interactive streaming video generation, evaluating 37 models and finding that visual quality and identity retention do not guarantee reliable request execution or sustained consistency.
- Five tasks cover avatar dialogue, avatar performance, world generation, instruction editing, and reference editing.
- The benchmark includes 300 scripted cases lasting 15 to 300 seconds and uses ten metrics for task execution, state consistency, visual fidelity, and audiovisual coordination.
- Benchmark data, evaluation code, protocols, and scoring rules are publicly available.
- The authors plan monthly evaluations of new and updated models with published leaderboard results.
#video generation #benchmarking #streaming video #multimodal evaluation
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 55, latest observation ; -599 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 2, latest observation ; 0 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2610.28402 | https://arxiv.org/abs/2610.28402 Read discussion |