BridgeVoC: Neural Vocoder with Schrödinger Bridge

1 Introduction

Neural vocoders are essential for generating high-quality waveforms from acoustic features, playing a crucial role in speech and audio generation tasks such as text-to-speech [Wang et al., 2017; Ren et al., 2019; Tan et al., 2024], textto-audio [Huang et al., 2023; Majumder et al., 2024], singing voice synthesis [Liu et al., 2022c; Hwang et al., 2025], voice conversion [Qian et al., 2019; Choi et al., 2021], audio editing [Wang et al., 2023], and speech enhancement (SE) [Liu et al., 2022a; Liu et al., 2022b].

In recent years, significant improvements in vocoding quality have been achieved because of the application of deep neural networks (DNNs). Auto-regressive (AR) methods such as WaveNet [Dieleman et al., 2016; Oord et al., 2018], SampleRNN [Mehri et al., 2022], and LPCNet [Valin and Skoglund, 2019] often face challenges with slow generation speeds due to their sequential nature. Flow-based vocoder methods, such as WaveGlow [Prenger and Valle, 2019], FlowWaveNet [Kim et al., 2019], and RealNVP [Laurent et al., 2017], address these issues by enabling faster generation speeds and improved performance through bijective mappings between a normalized probability distribution and the target data distribution using stacked invertible modules. Additionally, non-autoregressive (NAR) methods like HiFiGAN [Kong et al., 2020] have emerged, offering parallel processing and enhanced efficiency.

∗Andong Li is the corresponding author.

Most recently, time-frequency (T-F) domain-based neural vocoders have gained prominence. In these methods, the network estimates the spectral magnitude and phase in the Short-Time Fourier Transform (STFT) domain, and the inverse STFT (iSTFT) operation is then utilized to generate waveforms. These T-F methods have demonstrated competitive performance and faster inference speeds compared to time-domain methods [Lee et al., 2023; Hubert, 2024; Du et al., 2024]. Typically, these non-diffusion methods generate waveforms by taking acoustic features, such as the melspectrogram, as input. They employ various generators to estimate the spectral magnitude and phase or directly produce the waveform, as illustrated in Figure 1(a).

Diffusion-based methods typically have slower inference speeds and lower objective metrics, but offer greater flexibility, diversity, and more natural-sounding audio than nondiffusion vocoders. For example, WaveGrad refines white Gaussian noise into high-fidelity audio via a gradient-based sampler conditioned on the mel-spectrogram, balancing inference speed and quality [Chen et al., 2021]. DiffWave, a non-autoregressive diffusion model, efficiently generates high-fidelity audio through a Markov chain by optimizing a variational bound, requiring less computation and a smaller model size than WaveGrad, and excelling at unconditional generation [Kong et al., 2021]. PriorGrad replaces DiffWave's standard Gaussian prior with a data-driven adaptive prior, enabling faster convergence and improved perceptual quality [Lee et al., 2022]. Compared to PriorGrad and DiffWave, FreGrad achieves much faster training and inference, and a smaller model size, by operating in a simplified feature space and using frequency-aware components [Nguyen et al., 2024]. As shown in Figure 1(b), diffusion vocoders start from random Gaussian noise and iteratively denoise it, conditioned on mel-spectrograms or other features, following a noise-to-data pipeline.

In this work, we revisit neural vocoding task and introduce the Schr¨odinger Bridge to establish a data-to-data process between target and corrupted spectrograms in the TF domain from a restoration perspective rather than simple generation, as shown in Figure 1(c). Mel-spectrograms, derived from a linear-to-mel transform, can be projected back to the linear-scale domain using its pseudo-inverse [Lv et al., 2024] based on the range-null decomposition (RND) theory, which provides strong structural information of the target. Our vocoding goal is to reconstruct ground-truth spectrograms from mel-spectrograms, addressing both the spectral compression and phase information problems. According to our rank analysis, the mel-domain conversion and reversion process tends to decrease the spectral rank, necessitating that the neural vocoding task increases the spectral rank to restore clean speech. In contrast, the speech denoising task exhibits an opposite trend. Therefore, this work offers a novel perspective to bridge the connection between waveform generation and the commonly used restoration techniques in speech enhancement[Lei et al., 2025b]. Additionally, the multiperiod discriminator [Kong et al., 2020] and multi-resolution spectrogram discriminator [Won et al., 2021] are employed to further improve the generation quality.

The contributions of this paper are summarized as follows:

• BridgeVoC is the first T-F domain-based vocoder with the Schr¨odinger Bridge (SB) framework, exploring a data-to-data process rather than the conventional noiseto-data process in the previous literature.
• BridgeVoC introduces a novel perspective on bridging waveform generation and restoration, a connection not investigated in the preliminary literature.
• By integrating the SB framework with multi-mel losses

and a generative adversarial network (GAN), BridgeVoC achieves performance comparable to the state-of-the-art model BigVGAN, addressing the limitations of diffusion models in achieving excellent objective metrics.

2 Motivation

In this section, we start with the fundamental signal models to elucidate the transition from the conditional mel-to-waveform paradigm to the spectrum-to-spectrum restoration paradigm. Firstly, through the RND theory, a novel insight is provided to convert the mel-spectrogram back to degraded counterpart in the linear-scale spectrogram. Subsequently, rank analysis reveal contrasting rank trends between vocoding and denoising tasks. This observation inspired us to apply restoration methods commonly used in SE to the vocoding task.

2.1 Signal Models

The signal model of the speech denoising task in the T-F domain is represented as:

where {X, S,N} ∈ CT×F are the mixture, target, and noise signals; t and f index time and frequency.

For the vocoding task, mel-spectrograms Y mel ∈ RT×Fmel are obtained through the following signal model

where A ∈ RF×Fmel is the linear mel filter, with Fmel ≪ F for compression. This transform discards phase and linearly compresses the frequency dimension.

2.2 Range-Null Space Decomposition

For a classical signal compression physical model in the noise-free scenario, the target x ∈ RD and the observed signals y ∈ Rd can be simplified into y = Ax. If the pseudoinverse of A ∈ Rd×D is defined as A† ∈ RD×d , which satisfies AA†A ≡ A and d ≪ D, then the signal x can be decomposed into two orthogonal sub-spaces:

where A†Ax is the range-space component and (I-A†A)x is the null-space component. Comparing Eq. (2) and Eq. (3), we notice the mel-spectrogram can be converted into the range space, i.e., the first term on the right-hand side of the equal sign in Eq. (3), by left-multiplying the pseudo-inverse of A, i.e., A† . Since the null-space component is unknown in practice, the vocoding task can be formulated into the target estimation problem given the range-space component as the prior input, which is actually a classical signal recovery problem. Thanks to the powerful capability of the generative approach, we can effectively recover the remaining null-space component. Therefore, the RND theory provides us a different perspective to rethink the vocoding task. Recall that in the classical compressive sensing (CS) field [Zhang and Ghanem, 2018], a similar target is shared, where the target signal can be recovered from a linearly-compressed representation with the help of the structural sparseness prior. Next, we delve into the analysis from the perspective of the matrix rank.

2.3 Rank Analysis

Following the RND, we use the pseudo-inverse to map melspectrograms back to the original linear-scale domain, despite imperfections due to information loss, non-unique inverse mapping, approximation limitations, and lack of phase information [Meinard, 2015]. This process is formulated as:

where A† ∈ RFm×F is the pseudo-inverse transform matrix satisfying AA†A ≡ A. The linear-scale representation ˆ Y ∈ RT×F matches the feature dimensions of the target signals S. By appending a zero-phase component to ˆ Y , we can obtain its complex form S† ∈ CT×F :

where 0 ∈ RT×F is the zero matrix. Mapping S† to S is a restoration task similar to speech denoising, but while denoising (additive degradation) may increase spectral rank, vocoding (compression) reduces it. We illustrate these spectral rank changes below, defining R(·) : RT×F → Z as the matrix rank operation. By basic rank properties, we have

In Eqs. (6)-(7), the phase component is omitted, as the rank is associated with eigenvalues, which are more closely related to signal energy. Eq. (6) provides an upper bound on the rank of the mixture spectrum X. This implies that after adding noise N, the upper bound of the matrix rank tends to increase, and the stronger the noise, the higher the upper bound. For Eq. (7), it is deduced that with the decrease in the number of mel bands, i.e., R(AA† ) decreases, the rank R(ˆ Y ) tends to decrease. These two disparities in the rank distribution between noise-induced and mel-oriented degradations are visualized in Figure 2, where we calculate the rank difference between the degraded and target spectrum, defined as:

The noise degradation employs three levels: "mild", "moderate", and "heavy" with decreasing signal-to-noise ratios

(SNRs). For vocoding, we use three mel-band configurations (40, 80, and 100) to represent varying spectral compression. An STFT operation results in 257-dimensional features. Higher noise level has higher spectral rank and hinders sparsity, while higher mel-band compression leads to a negative rank difference. Therefore, from the perspective of the matrix rank, the vocoder and speech enhancement can share a similar goal, i.e., decrease the rank difference between the degraded and target spectra, further motivating us to address the vocoding task with the restoration paradigm.

3 BridgeVoC

In this section, we introduce BridgeVoC, an SB-based T-F domain vocoder. We begin with a brief overview of commonly used diffusion models, specifically score-based generative models (SGMs), including the forward and reverse stochastic differential equations (SDE) and the score matching objective of the score network. Then we define the paired data for the restoration task based on the signal model described in Section 2.3. Next, we detail the operations of SB and the model's training objectives. Finally, we describe the loss functions used in training.

3.1 Score-Based Generative Models

Given a data distribution pdata(x), x ∈ Rd , SGMs [Song et al., 2021] are built on a continuous-time diffusion process defined by a forward SDE:

where t ∈ [0, T] is a finite time index, xt ∈ Rd is the state of the process, f is a vector-valued drift term, g is a scalarvalued diffusion term, and wt ∈ Rd is a standard Wiener process. To ensure that the boundary distribution is a Gaussian prior distribution pprior = N(0, σ2T I), we construct the drift term f and the diffusion term g accordingly. This construction guarantees that the forward SDE has a corresponding reverse SDE:

where ¯wt is the reverse-time Wiener process, and ∇ log pt(xt) is the score function of the marginal distribution pt. To enable inference generated data samples at t = 0, we can replace the score function with a score network sθ(xt, t) and solve it reversely from pprior at t = T. A score network is usually learned by the denoising score matching objective [Song et al., 2021] :

where t ∼ U(0, T ) and pt|0 is the conditional transition distribution from x0 to xt, determined by the pre-defined forward SDE and analytical for a linear drift f(xt, t) = f(t)xt.

3.2 Schr¨odinger Bridge

The SB problem [Schr¨odinger, 1932; Bortoli et al., 2021] originates from the optimization of path measures with constrained boundaries. For vocoding task, we define the target distribution pS to be equal to the data distribution pdata, and we consider the distribution of S† , denoted as pS†, to be the prior distribution. Considering p0, pT the marginal distributions of p at boundaries, SB is defined as minimization of the Kullback-Leibler (KL) divergence:

where P[0,T] is the space of path measures on a finite time index [0, T] with pref the reference path measure. When pref is defined by the same form of forward SDE as SGMs in Eq. (10), the SB problem is equivalent to a couple of forwardbackward SDEs [Wang et al., 2021; Chen et al., 2022]:

where f , g and wt are from the forward SDE in Eq. (10). With Ψt and ̂Ψt the optimal forward and reverse drifts, the marginal distribution of the SB state xt can be expressed as pt = ̂ΨtΨt. Typically, SB is not fully tractable; closed-form solutions exist only when the families of pref are strictly limited [Bunne et al., 2023; Chen et al., 2023].

3.3 Schr¨odinger Bridge between Paired Data

We assume the maximum time T = 1 for convenience. Exploring the tractable SB between Gaussian-smoothed paired data with linear drift in SDE, we consider Gaussian boundary conditions pS = NC(x0, ϵ20 I) and pS† = NC(x1, e2 ∫ 1 0 f(τ)dτ ϵ20I). As ϵ0 → 0, ̂Ψt and Ψt converge to the tractable solution between the target data x0 and the corrupted data x1:

and ¯σ2 t = ∫ 1 t g2 (τ ) α2τ dτ are determined by f and g in the reference SDE, which are analogous to the noise schedule in SGMs [Kingma et al., 2021]. The marginal distribution of the SB also has a tractable form:

Several noise schedules [Chen et al., 2023; Ante et al., 2024], such as variance-preserving (VP), variance-exploding (VE) and gmax, are listed in Table 1 with ∆β = β1 - β0.

3.4 Loss Function

Following the approach in [Ante et al., 2024], we let the neural model Bθ directly predict the target data, using both reconstruction and adversarial losses as the training criteria, where S denotes the target signal and ˜ S = Bθ(xt, xT , t) represents the current estimate produced by the neural network. We empirically observe that the introduction of adversarial loss can effectively improve the generation quality.

Given that we employ the pseudo-inverse to map melspectrograms back to the original uncompressed linear-scale spectrogram, the extraction of amplitude information in the mel domain can assist the model in better reconstructing the original linear-scale information. Therefore, the reconstruction losses include both the mean-square error (MSE) loss Lmse and the mel loss Lmel following the settings in [Ai and Ling, 2023; Du et al., 2024]. The former is defined as the MSE between ˜ S and S in the STFT domain:

The adversarial losses includes the hinge GANs of discriminators Dm and generator Bθ, denoted as Ld and Lg, respectively:

where ˜s = iSTFT(˜S) ∈ RL denotes the reconstructed waveforms, iSTFT(·) refers to the iSTFT operation, and M is the number of sub-discriminators. Discriminators includes multi-period discriminator [Kong et al., 2020] and multiresolution spectrogram discriminator [Won et al., 2021; Lei et al., 2025a]. Besides, the feature matching loss is also utilized:

where f m l (·) denotes the l-th layer feature for the m-th subdiscriminator. Finally, the loss for the neural model is

where λmel, λg, and λfm are the are the weight hyperparameters of corresponding loss. Detailed settings can be found in [Lei et al., 2025c].

4 Experiments

4.1 Datasets

Two benchmarks are used in this study: LJSpeech [Keith and Linda, 2017] and LibriTTS [Heiga et al., 2019]. LJSpeech contains 13,100 clean speech clips from a single female speaker at 22.05 kHz, partitioned into 12,500/100/500 clips for training, validation, and testing, following the VITS

repository. LibriTTS, sampled at 24 kHz, includes diverse recording conditions; we use the {train-clean-100, train-clean-300, train-other-500} subsets for training, devclean+dev-other for objective evaluation, and test-clean+testother for subjective evaluation, as in [Lee et al., 2023].

To evaluate the generalization capability of neural vocoders, the VCTK dataset [Yamagishi, 2012] is utilized for out-of-distribution evaluations, where around 200 clips are randomly selected from the dataset for evaluations.

4.2 Configurations

Since the bridge between the target data S and the corrupted data S† can be viewed as a restoration task, it is intuitive to choose the noise-conditional score network (NCSN++) [Song et al., 2021] as the backbone neural model. Our ablation study experimented with three sizes of NCSN++, with trainable parameter counts of 16.2M, 36.5M, and 64.9M, respectively. The number of the sampling in the reverse process is empirically set to 10. In terms of noise schedulers, β0 = 0.01 and β1 = 20 are set for both gmax and scaled VP types. For VE type, we use k = 2.6 and c = 0.40, and for scaled VP type, we use c = 0.30. The processing time for the proposed SB is set to T = 1 with tmin = 10-4 . The reverse SDE and the probability flow Ordinary Differential Equation (ODE) [Chen et al., 2022] samplers are chosen in the inference stage. More ablation studies are conducted and can be found in the supplementary material.

For the weight hyperparameters in Eq. (22), λmel, λg and λfm are 0.1, 10.0 and 10.0, respectively. "+GAN" refers to the inclusion of the loss terms Lg and Lfm in Eq. (22).

We train all models for 1 million steps, except for BigVGAN, which is trained for 5 million steps. The training configurations for the T-F domain SE models are aligned with those of APNet2 and BigVGAN. For feature extraction, we employ a 1024-point FFT, a Hann window of length 1024, and a hop size of 256. For the LJSpeech dataset, we utilize 80 mel-bands with the upper-bound frequency fmax set to 8 kHz, meaning the model is required to conduct a super-resolution task to generate the spectral component over 8 kHz. For LibriTTS, the mel-bands and upper-bound frequency are set to 100 and 12 kHz, respectively.

4.3 Results and Analysis

For vocoding performance comparisons, we select popular vocoding models as baselines, including time-domain methods (BigVGAN [Lee et al., 2023], HiFiGAN [Kong et al., 2020]), T-F domain methods (Vocos [Hubert, 2024], FreeV [Lv et al., 2024], APNet2 [Du et al., 2024]), and diffusion-based methods (DiffWave [Kong et al., 2021], PriorGrad [Lee et al., 2022], and FreGrad [Nguyen et al., 2024]). To compare the model efficiency, we calculate the number of model parameters (#Params) and real-time factor (RTF) which is measured on a single Tesla V100 GPU.

Eight metrics are involved in the objective evaluations: (1) Wide-band version of Perceptual evaluation of speech quality (PESQ) [Rec, 2005] serves to assess the objective speech quality. (2) Extended Short-Time Objective Intelligibility (ESTOI) [Taal et al., 2011] measures the intelligibility of speech. (3) Periodicity RMSE, V/UV F1 score, F0, and pitch RMSE [Morrison et al., 2022; Kawahara et al., 1999] are regarded as major artifacts for nonautoregressive neural vocoders. (4) Virtual Speech Quality Objective Listener (VISQOL) [Hines et al., 2015] predicts the Mean Opinion Score-Listening Quality Objective (MOSLQO) score by evaluating the spectro-temporal similarity. (5) UTMOS [Saeki et al., 2022] is used to obtain subjective scores related to the perceived quality of speech, providing an objective approximation of human judgment.

For subjective evaluations, we employ the MUSHRA and ABX testing methodologies based on the BeaqleJS platform [Kraft and Z¨olzer, 2014]. A total of 19 participants, all specializing in audio signal processing, are involved in the testing. In the MUSHRA test, each participant is required to rate the speech processed by various algorithms on a scale from 0 to 100, based on the overall similarity to a reference. In the ABX test, participants are asked to select the clip they prefer in terms of overall speech quality, or choose "equal" if no preference can be given.

Ablation studies

To determine the optimal configuration of diffusion hyperparameters and network settings for BridgeVoC, we conducted ablation experiments on the LJSpeech benchmark.

Table 2 presents the test performance with various combinations of losses and noise schedules when the network parameter count is 16.2M. From the experimental results, it is evident that the introduction of auxiliary losses, single mel loss "+mel" and multi-mel loss "+mmel", can significantly enhance the model's performance. Furthermore, adding GAN on top of "+mmel" further improves the WB-PESQ score by 0.016. Correspondingly, other metrics also show certain improvements. When comparing Scaled VP and VE under the "+mmel+GAN" condition, gmax emerges as the optimal choice for the majority of indicators. Additionally, when the sampler is switched from the reverse SDE to the probability flow ODE, there is a slight degradation in performance.

Table 3 lists the results for the methods of reconstructing the signal from the network output and varying the network size under the settings of "gmax", "+mmel+GAN", and "SDE". "map" and "crm" denote that the network output is the complex spectrum mapping and the complex mask, respectively. "decouple" indicates that the network outputs the amplitude and phase of the signal separately, which are then coupled to form the output signal. The results indicate that the "crm"configuration is optimal for our task, rather than the previously default "map" form used in the NCSN++ network. In addition, increasing the size of the network also improves the final output scores.

For the case of "gmax" / "+mmel" / "map" / "16.2M", Figure 3 shows the results of the number of reverse sampling steps ablations. We observe that increasing the number of steps improves some metrics, while others peak at a specific step count, consistent with findings in other diffusion-based studies [Ho et al., 2020]. This phenomenon maybe due to the trade-off between the granularity of the sampling process and the accumulation of numerical errors. As the number of sampling steps increases, the model can more accurately capture the underlying data distribution, leading to improved performance for some metrics. However, beyond a certain point, the benefits of additional steps may be outweighed by the increased potential for error accumulation, resulting in a decline in performance for other metrics. This finding also implies that 10 steps are adequate for BridgeVoC, while reducing the number of steps to 7 does not lead to a substantial performance decline, suggesting that BridgeVoC can further lower computational cost and speed up inference.

Comparisons with SoTA methods

Tables 4 and 5 present objective comparisons on the LJSpeech and LibriTTS datasets, revealing key observations. First, the T-F domain-based methods exhibit faster inference speeds compared to the time-domain methods, primarily due to the use of STFT and its inverse transform, iSTFT, which eliminate the need for upsampling operations. Second, the T-F domain-based methods have significantly lower computational complexity, e.g., 5.8 GMACs for Vocos versus 152.9 GMACs for HiFiGAN, making them increasingly attractive. Third, despite these advantages, the speech quality of these existing T-F domain-based neural vocoders remains inferior to that of BigVGAN. Fourth, previous diffusion-based methods start from noise in the time domain and use the melspectrogram as a diffusion condition, failing to leverage the prior information of the mel-spectrogram. The proposed BridgeVoc, however, benefits from the prior structural information provided by the pseudo-inverse operation and the combination of the T-F domain-based Schr ¨odinger bridge and auxiliary losses. This allows BridgeVoc to achieve both fast inference speeds and promising performance. Notably, even when compared to BigVGAN trained for 5 million steps on the LibriTTS benchmark, our method remains competitive, validating the effectiveness of the proposed approach.

Table 6 presents the results on the out-of-domain test set. Compared to Table 5, the relative advantage of BridgeVoc over BridgeVoc-base in objective metrics slightly decreases. This is because the amount of data in LibriTTS is probably insufficient for a large NCSN++ network. The MUSHRA results on the test set of the VCTK dataset reveal that our BridgeVoc is statistically superior to BigVGAN (p < 0.05), further demonstrating the advantage of our method in achieving subjective quality close to the ground truth signal.

The preference scores are shown in Figure 4. For both nature and synthesized mel cases, the preference performance of the BridgeVoC-base is significantly better over FreGrad (p < 0.001), and is not significantly different from BigVGAN and Vocos (p > 0.05). Note that we choose PriorGrad as the baseline diffusion model because the Mean Opinion Score (MOS) experiments in [Nguyen et al., 2024] indicate that PriorGrad achieves higher subjective scores compared to FreGrad.

Figure 5 presents spectral visualizations of different models for a vocal clip from the out-of-distribution MUSDB18 [Rafii et al., 2017] test set. Our approach more effectively recovers harmonic details and avoids artificial harmonic fluctuations compared to other baselines, particularly BigVGAN-base. Subjective experiments revealed that some listeners reported "strange pitch shifts" relative to the ground truth in the MUSHRA experiments, with most instances traced back to BigVGAN-base. While BigVGAN also shows some "artificial generation" artifacts, their extent is significantly reduced.

5 Conclusions

In this paper, we introduce a novel time-frequency (TF) domain-based diffusion neural vocoder that effectively bridges the gap between the data-to-data Schr¨odinger Bridge framework and range-null decomposition theory. Our approach involves converting the original acoustic features from the mel-scale domain to the target linear-scale domain using the range-space component, while the null-space component reconstructs the remaining spectral details through a diffusion generation process. By incorporating generative adversarial networks and optimizing various hyperparameters, our method achieves promising results in both objective and subjective evaluations. Extensive experiments on the LJSpeech and LibriTTS benchmarks demonstrate the efficacy and superiority of the proposed approach.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant No. 12274221) and the Yangtze River Delta Science and Technology Innovation Community Joint Research Project (Grant No. 2024CSJGG1103).