FORCE: Factorized Object Relations for Counterfactual Editing in Video World Models
Paper · https://arxiv.org/abs/2611.00023
arXiv:2611.00023v1 Announce Type: new Abstract: World models predict how scenes evolve, but they are difficult to interrogate: given a model that forecasts a video, one cannot easily ask what would have happened had a particular interaction between two objects been different. We present Factorized Object Relations for Counterfactual Editing (FORCE), a world model that infers pairwise interactions from raw video without supervision and represents each as an explicit latent per object pair, so that a relation can be removed, changed, transferred or introduced directly. Objects are extracted by slot attention over frozen self-supervised features; a relation encoder infers each pair latent from a short observation window, and an interaction head turns it into a force. The latent is held fixed across the rollout, so an edit persists rather than being recomputed, and every edit acts on the model's own quantities with no mapping to simulator parameters. We evaluate in a simulated environment where the true outcome of an edit is available, scoring the change each edit induces against the change the simulator produces under the same operation. On held-out clips, FORCE's edits made from video agree with the simulator's response at 0.19 to 0.37 on our effect-based score; trained and scored on exact object positions instead, the same architecture reaches 0.57 to 0.85. The gap therefore lies not in the learned relational dynamics but in perception: what bounds an edit is how precisely objects can be located from video, and the precision editing needs lies just beyond what current object-centric perception delivers. Closing that gap is a concrete target for making counterfactual world models work from video.
First seen by Oriel on .
AI summary
FORCE is a video world model that supports persistent counterfactual edits to pairwise object interactions, but its video-based edit accuracy is limited by object localization.
- Slot attention over frozen self-supervised features extracts objects, while a relation encoder infers a latent for each object pair from a short observation window.
- An interaction head converts each relation latent into a force; holding the latent fixed during rollout lets edits persist without mapping them to simulator parameters.
- On held-out clips, video-based edits scored 0.19 to 0.37 against simulator outcomes, compared with 0.57 to 0.85 when trained and scored using exact object positions.
- The authors attribute the performance gap to perception precision and identify accurate object localization as a requirement for counterfactual editing from video.
#world models #counterfactual editing #object-centric learning #video perception
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.00023v1 | https://arxiv.org/abs/2611.00023 Read discussion |