Oriel — AI & technology worth noticing, every day.

JVT: Just Video Transformer makes pixel-space video diffusion simple

Paper · https://arxiv.org/abs/2611.18549

Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.

First seen by Oriel on .

AI summary

JVT is a simple large-patch DiT-style Transformer for pixel-space video diffusion, achieving quality competitive with latent diffusion while enabling aligned RGB-disparity generation.

  • JVT operates directly on raw video pixels without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections.
  • The study evaluates video patch embeddings, register tokens, and adaptive layer normalization to balance generation quality and computational efficiency.
  • JVT performs competitively with latent diffusion baselines in both class-conditioned and unconditional video generation settings.
  • Camera-conditioned RGB-disparity generation improves camera following over RGB-only baselines and provides explicit scene geometry without an inference-time reconstruction model.

#video diffusion #transformers #pixel-space generation #disparity estimation

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 15, latest observation ; +1 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 2, latest observation ; +1 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2611.18549https://arxiv.org/abs/2611.18549 Read discussion