Oriel — AI & technology worth noticing, every day.

StreamEval: Evaluating Interactive Streaming Video Generation Models

Paper · https://arxiv.org/abs/2610.28402

As streaming video generation advances, enabling real-time interaction with generated video is becoming increasingly important. We introduce StreamEval, a unified benchmark for interactive streaming video generation spanning five tasks: avatar dialogue, avatar performance, world generation, instruction editing, and reference editing. StreamEval comprises 300 scripted test cases lasting 15-300s, each specifying initial assets, incrementally arriving multimodal requests, and explicit requirements for state changes and content preservation. Performance is assessed along four evaluation dimensions: task execution, state consistency, visual fidelity, and audiovisual coordination, instantiated through ten task-specific metrics on a 0-100 scale. Evaluating 37 distinct models across 300 test cases, we find that high visual quality and identity retention do not necessarily coincide with reliable request execution or sustained state consistency. In particular, request execution remains challenging: the highest task-specific scores are 36.90 for Following among seven avatar-performance models, 25.93 for Following in text-conditioned world generation, and 15.00 for Edit Success among three reported reference-editing models. These findings identify reliable request execution under timing and preservation constraints as a central challenge for advancing omni-modal real-time interactive systems. The leaderboard is available at https://streameval.ai

First seen by Oriel on .

AI summary

StreamEval is an open-source benchmark for interactive streaming video generation, evaluating 37 models and finding that visual quality and identity retention do not guarantee reliable request execution or sustained consistency.

  • Five tasks cover avatar dialogue, avatar performance, world generation, instruction editing, and reference editing.
  • The benchmark includes 300 scripted cases lasting 15 to 300 seconds and uses ten metrics for task execution, state consistency, visual fidelity, and audiovisual coordination.
  • Benchmark data, evaluation code, protocols, and scoring rules are publicly available.
  • The authors plan monthly evaluations of new and updated models with published leaderboard results.

#video generation #benchmarking #streaming video #multimodal evaluation

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 55, latest observation ; -599 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 2, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.28402https://arxiv.org/abs/2610.28402 Read discussion