FastDiff: A Fast Conditional Diffusion Model for High-Quality Speech Synthesis

1 Introduction

With the recent development of deep generative models, speech synthesis has seen an extraordinary progress. Among the conventional speech synthesis methods, WaveNets [Oord et al., 2016] were demonstrated to generate high-fidelity audio samples in an autoregressive manner yet suffering from prohibitively expensive computational costs. In contrast, non-autoregressive approaches such as flow-based and GAN-based models [Prenger et al., 2019; Jang et al., 2021;

∗Work done during internship at Tencent AI Lab

†Equal contribution

‡Corresponding author

Kong et al., 2020a; Huang et al., 2021] were also proposed to generate speech audios with satisfactory speed. However, these models were still criticized for other problems, e.g., the limited sample quality or sample diversity [Xiao et al., 2021]. In speech synthesis, our goal is mainly two-fold:

• High-quality: generating high-quality speech is a challenging problem especially when the sampling rate of an audio is high. It is vital to reconstruct details at different timescales for waveforms of highly variable patterns.
• Fast: high generation speed is essential when considering real-time speech synthesis. This poses a challenge for all high-quality neural synthesizers.

As a blossoming class of generative models, denoising diffusion probabilistic models (DDPMs) [Ho et al., 2020; Song et al., 2020; Lam et al., 2022] has emerged to prove its capability to achieve leading performances in both image and audio syntheses [Dhariwal and Nichol, 2021; Kong et al., 2020b; Chen et al., 2020; Lam et al., 2022]. However, current development of DDPMs in speech synthesis was hampered by two major challenges:

• Different from other existing generative models, diffusion models are not trained to directly minimize the difference between the generated audio and the reference audio, but to de-noise a noisy sample given an optimal gradient. This in practice could lead to overly de-noised speech after a large number of sampling steps, in which natural voice characteristics including breathiness and vocal fold closure are removed.
• While DDPMs inherently are gradient-based models, a guarantee of high sample quality typically comes at a cost of hundreds to thousands of de-noising steps. When reducing the sampling steps, an apparent degradation in quality due to perceivable background noise is observed.

In this work, we propose FastDiff, a fast conditional diffusion model for high-quality speech synthesis. To improve audio quality, FastDiff adopts a stack of time-aware locationvariable convolutions of diverse receptive field patterns to efficiently model long-term time dependencies with adaptive conditions. To accelerate the inference procedure, FastDiff also includes a noise schedule predictor, which derives a short and effective noise schedule and significantly reduces the denoising steps. Based on FastDiff, we also introduce an end-toend phoneme-to-waveform synthesizer FastDiff-TTS, which simplifies the text-to-speech generation pipeline and does not require intermediate features or specialized loss functions to enjoy low inference latency.

Experimental results demonstrated that FastDiff achieved a higher MOS score than the best publicly available models and outperformed the strong WaveNet vocoder (MOS: 4.28 vs. 4.20). FastDiff further enjoys an effective sampling process and only needs 4 iterations to synthesize high-fidelity speech, 58x faster than real-time on a V100 GPU without engineered kernels. To the best of our knowledge, FastDiff is the first diffusion model with a sampling speed comparable to previous for the first time applicable to interactive, realworld speech synthesis applications at a low computational cost. FastDiff-TTS successfully simplify the text-to-speech generation pipeline and outperform competing architectures.

2 Background: Denoising Diffusion Probabilistic Models

Denoising diffusion probabilistic models (DDPMs) [Ho et al., 2020; Song et al., 2020; Lam et al., 2022] are likelihoodbased generative models that have recently succeeded to advance the state-of-the-art results in benchmark generative tasks [Dhariwal and Nichol, 2021] and have proved its capability to produce high-quality samples. The basic idea of DDPMs is to train a gradient neural network for reversing a diffusion process. Given i.i.d. samples {x0 ∈ RD } from an unknown data distribution pdata(x0), DDPMs try to approximate pdata(x0) by a marginal distribution pθ(x0) = ∫ · · · ∫ p(xT ) ∏Tt=1 pθ(xt-1|xt)dx1 . . . dxT .

In data distribution as q(x0), the diffusion process is defined by a fixed Markov chain from data x0 to the latent variable xT . For a small positive constant βt, a small Gaussian noise is added from xt to the distribution of xt-1 under the function of q(xt|xt-1). The whole process gradually converts data x0 to whitened latents xT according to the fixed noise schedule β1, · · · , βT . The reverse process is a Markov chain from xT to x0 parameterized by a shared θ, which aims to recover samples from Gaussian noises though eliminating the Gaussian noise added in the diffusion process in each iteration.

It has been demonstrated that diffusion probabilistic models [Dhariwal and Nichol, 2021; Xiao et al., 2021] can learn diverse data distribution in multiple domains, such as images and time series. While the main issue with the proposed neural diffusion process is that it requires up to thousands of iterative steps to reconstruct the target distribution during reverse sampling. In this work, we offer a fast conditional diffusion model to reduce reverse iterations and improve computational efficiency.

3 FastDiff

This section presents our proposed FastDiff, a fast conditional diffusion model for high-quality speech synthesis. We first describe the motivation of the design in FastDiff. Secondly, we introduce the iterative refinement model θ for highquality speech synthesis and the noise predictor ϕ for accelerated sampling. Furthermore, we describe the training and inference procedures in detail. At last, we extend FastDiff to FastDiff-TTS for fully end-to-end text-to-speech syntheses.

3.1 Motivation

While denoising diffusion probabilistic models have shown high potential in synthesizing high-quality speech samples [Chen et al., 2020; Kong et al., 2020b; Liu et al., 2021], several challenges remain for industrial deployment: 1) Different from the traditional generative models, diffusion models catch dynamic dependencies from noisy audio instead of clean ones, which introduce more variation information (i.e, noise levels) in addition to the spectrogram fluctuation. 2) With limited receptive field patterns, a distinct degradation could emerge when reducing the reverse iterations, making diffusion models difficult to get accelerated. As a result, hundred or thousand orders of iterations prevent existing diffusion models from real-world deployment.

In FastDiff, we propose two key components to complement the above issues: 1) FastDiff adopts a time-aware location-variable convolution to catch the details of noisy samples at dynamic dependencies. The convolution operations are conditioned on dynamic variations in speech including diffusion steps and spectrogram fluctuations, equipping the model with diverse receptive field patterns and promoting the robustness of diffusion models during reverse acceleration. 2) To accelerate the inference procedure, FastDiff adopts a noise schedule predictor to reduce the number of reverse iterations, frees diffusion models from hundreds or thousands of refinement iterations. This makes FastDiff for the first time applicable to interactive, real-world applications at a low computational cost.

3.2 Time-Aware Location-Variable Convolution

In comparison with traditional convolution networks, location-variable convolution [Zeng et al., 2021] shows efficiency in modeling the long-term dependency of audio and gets neural network free from a significant number of dilated convolution layers. Inspired by this, we introduce the TimeAware Location-Variable Convolution, which is sensitive to time steps in diffusion probabilistic models. At time step t, we follow [Vaswani et al., 2017] to embed the step index into an 128-dimensional positional encoding (PE) vector et:

In time-aware location-variable convolution, FastDiff requires multiple predicted variation-sensitive kernels to perform convolutional operations on the associated intervals of input sequence. These kernels should be time-aware and sensitive to variations of noisy audio including diffusion steps and acoustic features (i.e., Mel-spectrogram). Therefore, we propose a time-aware location-variable convolution (LVC) module, which is coupled with a kernel predictor as shown in Figure 1(b) and Figure 1(c). We describe the overall calculations below.

For the q-th time-aware LVC layer, we split the input xt ∈ RD using a M-length window with 3q dilations to produce K

segments with each xkt ∈ RM :

Next, we perform convolutional operations on the associated intervals of input sequence using the kernels generated by a kernel predictor α:

where Ft,Gt denote the filter and the gate kernels for xit , respectively, ∗ denotes the 1d convolution, ⊙ denotes the element-wise product and concat(·) denotes the concatenation between vectors. Since the time-aware kernels are adaptive to the noise-level and dependent to the acoustic features, FastDiff is capable of precisely estimating de-noising gradient with a superior speed given a noisy signal input.

3.3 Accelerated Sampling

Noise Predictor

To avoid sampling with hundreds to thousands steps, FastDiff adopts the noise scheduling algorithm in the bilateral denoising diffusion models (BDDMs) [Lam et al., 2022] to predict a sampling noise schedule much shorter than the noise schedule used in training. This scheduling method has been revealed to be superior than other sampling acceleration methods, e.g., the grid search algorithm in WaveGrad [Chen et al., 2020] and the fast sampling algorithm in DiffWave [Kong et al., 2020b]. The noise predictor iteratively derives a continuous noise schedule ˆ β ∈ RTm . We attach the learning objective and corresponding likelihood in Appendix.

Schedule Alignment

In FastDiff, similar to DDPMs, during training we use T = 1000 discrete time steps. Therefore, when needed to condition on t during sampling, we also need to approximate Tm discrete time indices by aligning the Tm-step sampling noise schedule ˆ β to the T -step training noise schedule β, with N << T. We have attached the detailed algorithms in Appendix.

3.4 Training, Noise Scheduling and Sampling

All illustrated in Algorithm 1, we separately parameterize FastDiff by two modules: 1) a iterative refinement model θ that minimizes a variational bound of the score function, and 2) a noise predictor ϕ that optimizes the noise schedule for a tighter evidence lower bound. For inference, we first derive the tighter and more efficient noise schedules ˆ β via an one-shot noise scheduling procedure, which makes FastDiff achieve orders of magnitude faster at sampling. It has been demonstrated [Lam et al., 2022] that the noise schedule searched for as few as 1 sample could be robust enough to maintain a high-quality generation among all samples in testing set. Secondly, we map the continuous noise schedules to discrete time indexes Tm using schedule alignment. Finally, FastDiff iteratively refines gaussian noise to generate high-quality samples with computational efficiency. The detailed information on training, noise scheduling and inference procedures has been presented in Appendix.

3.5 FastDiff-TTS

Existing text-to-speech methods usually adopt a two-stage pipeline: 1) A text-to-spectrogram generation module (a.k.a. acoustic model) aims to generate prosodic attributes according to variance prediction; 2) A conditional waveform generation module (a.k.a. vocoder) adds the phase information and synthesizes a detailed waveform. To further simplify the text-to-speech synthesis pipeline, we propose a fully end-to-end model FastDiff-TTS, which does not require intermediate features or specialized loss functions. FastDiffTTS is designed to be a fully differentiable and efficient architecture that directly produces waveforms from contexts (e.g. phonemes) without needing to generate acoustic features (e.g., Mel-spectrograms) explicitly.

Architecture

The architecture design of FastDiff-TTS refers to a convectional non-autoregressive TTS model - FastSpeech 2 [Ren et al., 2020] as the backbone. The architecture of FastDiff-TTS is illustrated in Figure 1(d). In FastDiff-TTS, the encoder first converts the phoneme embedding sequence into the phoneme

Algorithm 1 Training refinement network θ

• 1: Input: Pre-defined noise schedule β
• 2: repeat
• 3: Sample x0 ∼ qdata, ϵ ∼ N(0, I), and t ∼ Unif({1, · · · , T})
• 4: xt = αtx0 + √1-α2 t ϵ
• 5: Take gradient descent steps on ∇θ ∥ϵ - ϵθ (xt|c, t)∥22
• 6: until refinement model θ converged

Algorithm 2 Training noise predictor ϕ

• 1: Input: Pre-defined discrete β, trained refinement network θ, hyperparameter τ .
• 2: repeat
• 3: Sample x0 ∼ qdata, ϵ ∼ N(0, I), and t ∼ Unif({τ, · · · , T - τ})
• 4: xt = αtx0 + √1-α2 t ϵ
• 5: ˆ βt = min {1 - α2 t , 1 - α2t+τ α2t } ϕ(xt)
• 6: Take gradient descent steps on ∇ϕ { δ2 t 2(δ2 t -ˆ βt) ∥∥∥ϵ - ˆ βt δ2 t ϵθ (xt|c, t)∥∥∥2 2 }
• 7: until noise predictor ϕ converged

hidden sequence. Then, the duration predictor expands the encoder output to match the length of the desired waveform output. Given the aligned sequence, the variance adaptor adds pitch information into the hidden sequence. Note that it is difficult to use the full audio corresponding to the full text sequence for training due to the typically high sampling rate for high-fidelity waveform and the limited GPU memory. Therefore, we sample a small segment to synthesize the waveform before passing to the FastDiff model. Finally, the FastDiff model decodes the adapted hidden sequence into a speech waveform as in the vocoder task.

Training Loss

FastDiff-TTS does not require specialized loss functions and adversarial training to improve sample quality as suggested by the previous works [Ren et al., 2020; Donahue et al., 2020; Kim et al., 2021]. This, to a large extend, simplifies the textto-speech generation. The final training loss consists of the following terms: 1) a duration prediction loss Ldur: the mean squared error between the predicted and the ground-truth word-level duration in log-scale, 2) a diffusion loss Ldiff: the mean squared error between the estimated and gaussian noise, and 3) a pitch reconstruction loss Lpitch: the mean squared error between the predicted and the ground-truth pitch sequences. We empirically found that the pitch reconstruction loss Lpitch is helpful for handling the one-to-many mapping issue in text-to-speech synthesis.

4 RelatedWorks

Text-to-speech (TTS) systems aim to synthesize raw speech waveforms from given text. In recent years, Neural network based TTS [Ren et al., 2020; Kim et al., 2020; Liu et al.,

Algorithm 3 Sampling

• 1: Input: Searched ˆ β in noise scheduling process.
• 2: Compute discrete steps Tm sequences via schedule alignment in Section 3.3.
• 3: Sample xTm ∼N(0, I)
• 4: for t = Tm, · · · , 1 do
• 5: Sample xt-1 ∼ pθ(xt-1|xt; ˆ β)
• 6: end for
• 7: return x0

2021] has made huge progress and attracted a lot of attention in the machine learning and speech community.

Neural vocoder plays the most important role in the recent success of speech synthesis, which require diverse receptive field patterns to catch audio dependencies: 1) autoregressive model WaveNet [Oord et al., 2016] requires causal convolutions layers to increase the receptive field while suffering from slow inference speed. 2) Flow-based generative models [Prenger et al., 2019] fully utilize modern parallel computing processors to broaden receptive fields and speed-up sampling, while they usually achieve a limited sample quality. 3) Generative adversarial networks (GANs) [Jang et al., 2021; Kong et al., 2020a] are one of the most dominant deep generative models in audio generation. UnivNet [Jang et al., 2021] has demonstrated its success in using local-variable convolution on different waveform intervals, and HIFI-GAN [Kong et al., 2020a] proposes multi-receptive field fusion (MRF) to model the periodic patterns matters. However, GAN-based models are often difficult to train, collapsing [Creswell et al., 2018] without carefully selected hyperparameters and regularizers, and showing less sample diversity. 4) Recently proposed diffusion models Diffwave [Kong et al., 2020b] and WaveGrad [Chen et al., 2020] could generate high-quality speech samples, while suffering from a distinct degradation when reducing reverse iterations, making diffusion models difficult to get accelerated.

Another important line of work covers directly waveform generation from text: FastSpeech 2s [Ren et al., 2020] and VITS [Kim et al., 2021] adopt adversarial training process and spectral losses for improving audio quality, while they do not take full advantage of end-to-end training. Recently proposed WaveGrad 2 [Chen et al., 2021] estimates the gradient of the log conditional density of the waveform given a phoneme sequence, but suffers from a large model footprint and slow inference. Unlike all of the aforementioned methods, as highlighted in section 3.5, FastDiff-TTS is a fully differentiable and efficient architecture that produces waveforms directly without generating middle features (e.g., spectrograms) explicitly. In additional, our diffuion probabilistic model gets free from hundred or thousands of iterations and enjoy computational efficiency.

5 Experiments

5.1 Setup

Dataset

For a fair and reproducible comparison against other competing methods, we used the benchmark LJSpeech dataset [Ito and Johnson, 2017]. LJSpeech consists of 13,100 audio clips of 22050 Hz from a Female speaker with about 24 hours in total. To evaluate the generalization ability of our model over unseen speakers in multi-speaker scenarios, we also used the VCTK dataset [Yamagishi et al., 2019], which was downsampled to 22050 Hz to match the sampling rate with the LJSpeech datset. VCTK consists of approximately 44,200 audio clips uttered by 109 native English speakers with various accents. For both datasets, we used 80-band Mel-spectrograms as the condition for the vocoding task. The FFT size, window size, and hop size were, respectively, set to 1024, 1024, and 256.

Model Configurations

FastDiff mainly consists of the refinement model θ and noise schedule predictor ϕ. The refinement model θ comprises three Diffusion-UBlock and DBlock with the upsample or downsample rate of [8, 8, 4], respectively. We adopt a lightweight GALR network effective in separating the added gaussian noise from audio as the noise schedule predictor ϕ. For end-to-end text-to-speech generation, FastDiff-TTS follows the basic structure in FastSpeech 2 [Ren et al., 2020], which consists of 4 feed-forward transformer blocks in the phoneme encoder. More details have been attached in the Appendix.

Training and Evaluation

The complete training pipeline has been illustrated in Algorithm 1: FastDiff was trained with constant learning rate lr = 2 × 10-4 . The refinement model θ and noise predictor ϕ were trained for 1M and 10K steps until convergence, respectively. FastDiff-TTS was trained until 500k steps using the AdamW optimizer with β1 = 0.9, β2 = 0.98, ϵ = 10-9 . Both models were trained on 4 NVIDIA V100 GPUs using random short audio clips of 16,000 samples from each utterance with a batch size of 16 each GPU. More details have been attached in the Appendix.

We crowd-sourced 5-scale MOS tests via Amazon Mechanical Turk to evaluate the audio quality. The MOS scores were recorded with 95% confidence intervals (CI). Raters listened to the test samples randomly, where they were allowed to evaluate each audio sample once. We further include additional objective evaluation metrics including STOI [Taal et al., 2010] and PESQ [Rix et al., 2001] to test sample quality. To evaluate the sampling speed, we implemented real-time factor (RTF) accessment on a single NVIDIA V100 GPU. In addition, we employed two metrics NDB and JSD to explore the diversity of generated mel-spectrograms. More information about both objective and subjective evaluation has been attached in Appendix.

5.2 ComparsionWith Other Models

We compared our FastDiff in audio quality, diversity, and sampling speed with competing models, including 1) WaveNet [Oord et al., 2016], the autoregressive generative model for raw audio. 2) WaveGlow [Prenger et al., 2019], non-autoregressive flow-based model. 3) HIFI-GAN V1 [Kong et al., 2020a] and UnivNet [Jang et al., 2021], the most dominant and popular GAN-based models. 4) Diffwave [Kong et al., 2020b] and WaveGrad [Chen et al., 2020], recently proposed diffusion probabilistic models which achieve state-of-the-art in speech synthesis. For easy comparison, the results are compiled and presented in Table 1, and we have the following observations:

In terms of audio quality, FastDiff achieved the highest MOS with a gap of 0.24 compared to the ground truth audio, and it matched the performance of the autoregressive WaveNet baseline and outperformed the non-autoregressive baselines. For objective evaluation, FastDiff also demonstrated a large improvement in PESQ and STOI. For inference speed, with the efficient noise schedules searched by noise predictor, FastDiff could generate high-quality speech samples within as few as 4 reverse steps, significantly reducing the inference time compared with competing diffusion architectures. To the best of our knowledge, FastDiff makes diffusion models for the first time applicable to interactive, high-quality real-world speech synthesis at a low computational cost. In terms of sample diversity, we can see that FastDiff still witnessed a gap from autoregressive WaveNet, but it achieve a higher variety for generated speeches than nonautoregressive baselines. More detailed evaluation on sample diversity has been attached in Appendix.

5.3 Ablation Study

We conducted ablation studies to demonstrate the effectiveness of several designs in FastDiff, including the time-aware location variable convolution and noise predictor in neural vocoding. The results of both subjective and objective evaluations have been presented in Table 2, and we have the following observations: 1) Replacing time-aware location-variable convolution by traditional convolutional operations causes a distinct degradation in sampling speed and perceptual quality. 2) Using grid search instead of the noise predictor to search schedules had witnessed the decreased audio quality, demonstrating that the noise schedule prediction process provides more efficient reverse sampling without sacrificing quality.

Further, we compare two variants of FastDiff to test the modality differences of diffusion condition (i.e., continuous noise-level or discrete time-step). Note that the former model does not require the schedule alignment process mentioned in Section 3.3. We empirically find that the FastDiff model conditioned on discrete time steps could synthesize samples with higher quality, demonstrating that learning proposed FastDiff with discrete diffusion times could be a better choice. More information on the variant of FastDiff extended to continuous noise schedules has been attached in Appendix.

5.4 Generalization To Unseen Speakers

We used 50 randomly selected utterances of 5 unseen speakers in the VCTK dataset that were excluded from the training set for the MOS test. Table 3 shows the experimental results for the mel-spectrogram inversion of the unseen speakers: In summary, we noticed that FastDiff achieved state-of-the-art in terms of audio quality for out-of-domain generalization, indicating that FastDiff could universally generate high-fidelity audio from entirely new speakers outside the train set.

5.5 End-to-End Text-to-Speech

To demonstrate the robustness of the proposed model in end-to-end text-to-speech synthesis, we compare FastDiffTTS with other neural TTS systems, including 1) GT, the ground truth audio; 2) GT (voc.), where we first convert the ground truth audio into mel-spectrograms, and then convert the mel-spectrograms back to audio using FastDiff; 3) PortaSpeech [Ren et al., 2021] + FastDiff: vocoder cascaded with mel-spectrogram generation using the most popular nonautoregressive TTS models; 4) FastSpeech 2s [Ren et al., 2020]: the extension of FastSpeech 2 to fully end-to-end textto-waveform generation with multi-task learning; 5) WaveGrad 2 [Chen et al., 2021]: a diffusion probabilistic model to generate waveforms via gradient estimation.

The results are shown in Table 4: FastDiff-TTS could surpass competing end-to-end speech synthesis models and match the voice quality of the state-of-the-art cascaded TTS

systems, demonstrating that FastDiff-TTS is efficient in simplifying the overall text-to-speech synthesis pipeline.

6 Conclusion

In this work, we proposed FastDiff, a fast conditional diffusion model for high-quality speech synthesis. FastDiff employed a stack of time-aware location-variable convolutions with diverse receptive field patterns to model long-term time dependencies with adaptive conditions. A noise predictor was further adopted to derive tighter schedules for reducing reverse iterations without distinct quality degradation. The extension model FastDiff-TTS discarded intermediate features (e.g., spectrograms) and simplified the end-to-end text-towaveform syntheses pipeline. Experimental results demonstrated that our proposed model outperformed the best publicly available models in terms of synthesis quality, even comparable to the human level. Moreover, FastDiff showed a significant improvement in synthesis speed, which required as few as 4 iterations to generate high-quality samples. To the best of our knowledge, FastDiff made diffusion models for the first time applicable to interactive, real-world speech generation with a low computational cost. In addition, FastDiff performed strong robustness and enjoyed high-quality synthesis in out-of-domain generalization to unseen speakers. We envisage that our work could serve as a basis for future speech synthesis studies.

Acknowledgements

This work was supported in part by the National Key R&D Program of China under Grant No.2020YFC0832505, National Natural Science Foundation of China under Grant No.61836002, No.62072397, Zhejiang Natural Science Foundation under Grant LR19F020006, information Technology Center and State Key Lab of CAD & CG, Zhejiang University, and Tencent AI Lab.