Oriel — AI & technology worth noticing, every day.

UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

Paper · https://arxiv.org/abs/2610.24791

Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities. Existing methods rely heavily on paired corpora, which are scarce for joint generation and even rarer for editing. Consequently, they are spent learning single-branch generative and editing capabilities that abundant unimodal data could readily teach, rather than on the cross-modal coupling. Furthermore, prevailing systems apply a single shared noise schedule to both modalities, neglecting two critical limitations. First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise. Second, lockstep denoising leaves asymmetric workflows, such as conditioning on one modality to synthesize or edit another, untrained. We present UniAVC, which addresses both obstacles. It first trains each branch on unimodal dataset to full generative and editing competence, and subsequently dedicates paired data to exploit cross-modal correspondence. Modality-decoupled schedulers meanwhile warp the noise schedule independently from a shared timestep, while subtasks with one clean modality directly supervise asymmetric configurations. These settings respectively enable intent-aware editing, which preserves one modality intact while enlisting it to constrain the other, and post-inference self-refinement, which regenerates only an unsatisfactory branch against an accepted one. UniAVC thus spans multiple creation tasks from a single set of weights without task-specific adapters.

First seen by Oriel on .

AI summary

UniAVC is a unified audio-video model trained progressively with unimodal and paired data, using asymmetric noise scheduling to support joint creation, cross-modal generation, and editing from one set of weights.

  • Each modality first learns full generative and editing capabilities from abundant unimodal datasets before paired data trains cross-modal correspondence.
  • Modality-decoupled schedulers independently warp audio and video noise schedules from a shared timestep, reflecting their different information densities.
  • Clean-modality subtasks supervise asymmetric workflows, enabling one modality to condition or constrain synthesis and editing of the other.
  • The model supports intent-aware editing and post-inference self-refinement, regenerating an unsatisfactory branch while preserving an accepted modality.
  • UniAVC covers multiple creation tasks without task-specific adapters.

#audio-video generation #multimodal models #diffusion models #cross-modal editing

Generated by gpt-5.6-sol. Check the original for details.

Metric trends

Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained

Points

Score reported by the source.

  • Hugging Face Papers: 14, latest observation ; +6 over observed period since

Comments

Comment count reported by the source.

  • Hugging Face Papers: 1, latest observation ; 0 over observed period since

Aliases

SourceExternal IDURL
Hugging Face Papers2610.24791https://arxiv.org/abs/2610.24791 Read discussion