WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning
Paper · https://arxiv.org/abs/2611.12527
This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral v-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen
First seen by Oriel on .
AI summary
WaveGen is a diffusion-based text-to-waveform framework that jointly learns internal spectral and waveform trajectories, achieving state-of-the-art performance on the Emilia training dataset without separately trained intermediate representations.
- Spectral Forcing uses disentangled diffusion heads and a multi-scale spectral v-loss to support direct waveform generation.
- The single model performs internal text-speech alignment without duration predictors, neural audio codecs, Mel-spectrogram vocoders, VAE latent models, self-supervised representations, or semantic distillation.
- The authors scaled WaveGen from 0.1B to 0.6B parameters using open-source data, one architecture, and one-stage training.
- WaveGen achieved a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark; source code and checkpoints are available.
#text-to-speech #diffusion models #audio generation #spectral learning
Generated by gpt-5.6-sol. Check the original for details.
Metric trends
Last 30 days · Up to 120 recorded samples per source and metric, with reposts kept separate · First and latest retained
Points
Score reported by the source.
- Hugging Face Papers: 18, latest observation ; +5 over observed period since
Comments
Comment count reported by the source.
- Hugging Face Papers: 3, latest observation ; +2 over observed period since
Aliases
| Source | External ID | URL |
|---|---|---|
| Hugging Face Papers | 2611.12527 | https://arxiv.org/abs/2611.12527 Read discussion |