A Multi-view Fusion Approach for Enhancing Speech Signals via Short-time Fractional Fourier Transform

1 Introduction

Speech enhancement (SE), commonly applied in various scenarios, aims to improve the intelligibility and quality of speech signals degraded by noise or other forms of distortion. It is a crucial task in modern communication and accessibility technologies, helping people with hearing impairments by filtering background noise in hearing aids, improving speech clarity for accurate recognition in voice-controlled systems, and improving call quality by reducing ambient noise in telecommunications [Li et al., 2022].

∗Corresponding author: jinchengqyh@126.com

The speech representation space plays a key role in SE base on deep learning [Yu et al., 2020; Tian et al., 2024]. The existing methods mainly depend on time-domain and frequency-domain. (1) Time domain methods take raw speech signals as input and apply deep neural network architectures for processing. Some approaches [D´efossez et al., 2020; Strauss and Edler, 2021; Kong et al., 2022] leverage UNet architectures to extract multiscale features, while RNNs or LSTMs capture long-term dependencies in the time-domain signals. These methods preserve the time-domain characteristics of speech, which is beneficial for enhancing the speech SNR [Luo and Mesgarani, 2019]. (2) Frequencydomain methods, such as the Short-Time Fourier Transform (STFT), provide flexible signal representations for speech and noise separation. The models learn how to extract speech components from the noise spectrum while suppressing the noise. The enhanced spectrum is converted back to a time domain signal by the Inverse STFT (ISTFT) [Hu et al., 2020; Fu et al., 2021; Shin et al., 2023]. These methods can suppress noise in a finer space than the time domain, which is advantageous for improving speech intelligibility.

Recent works argue that relying solely on time or frequency domain information only captures partial features of the audio, thus limiting the comprehensiveness of the data analysis [Tang et al., 2021; Lin et al., 2024]. To take full advantage of the information in both domains and to integrate their representation spaces, recent approaches have focused on joint optimization of the time and frequency domains as multi-view inputs. These methods [Nareddula et al., 2021; Dang et al., 2023] utilize different networks to map time and frequency domain views to the same underlying representation thus fusing them, extending the representation space of speech and noise to improve the performance of speech enhancement. Some recent work such as [Liang et al., 2024; Guo et al., 2024a] have carried out some architectural innovations for multi-view convergence. There are also works such as [Liang et al., 2022; Liang et al., 2025] that utilize associative relationships and trusted fusion to enhance the interpretability of multi-view fusion.

However, these approaches still focus on analyzing signals from only two perspectives, time and frequency, and they have a fixed resolution for time and frequency, making it difficult to achieve a fine-grained representation of speech and noise. Coupled with the fact that the STFT originally contained time variations, the effect of optimizing the two jointly is not obvious. The key to improving SE is to find a more flexible auxiliary view for STFT that extends the representation space to better decouple speech and noise. The short-time fractional order Fourier transform (STFrFT), on the other hand, allows the adjustment of time and frequency resolution by introducing a fractional order parameter, resulting in better time resolution in the high-frequency bands and better frequency resolution in the low-frequency bands, which can be adapted to the characteristics of different signals. This flexible resolution adjustment capability allows the model to capture richer time-frequency information, thus improving the effectiveness of SE. As shown in Figure 1, STFrFT spectrum is capable of obtaining a more focused spectral representation, which serves as an auxiliary view to direct the STFT spectrum to focus on important information at the same time frame (red and orange box).

In this paper, we present a novel multi-view single-channel SE method that employs the short-time fractional Fourier transform (STFrFT), which provides a richer representation space than methods that use only the time and frequency domains.

The main contributions of this work include:

• We extend speech to the dynamic domain by STFrFT. Determining the best auxiliary STFrFT view from the dynamic domain using spectral centroids makes STFT more focused on useful frequency information.
• We propose a lightweight framework containing a feature extraction module and an attention module for better separation of speech and noise.
• In VoiceBank+Demand dataset, our proposed algorithm has significant advantages over other advanced algorithms.

2 RelatedWork

The current inputs for deep learning-based single-channel SE are mainly classified into three cases: (i) Time domain based inputs. (ii) Frequency domain based inputs. (iii) Multi-view inputs optimized jointly in both the time and frequency domains. Inputs based on STFT variations are commonly referred to as the time-frequency (T-F) domain in the field of SE, which is abbreviated as the frequency domain in this paper to better distinguish it frommulti-view inputs.

Time domain based SE models operate directly on time domain waveforms without domain transformation. SEGAN [Pascual et al., 2017] is a model that transforms noisy speech into clear speech, with its generator handling the transformation and its discriminator distinguishing between synthetic and real clear speech. Building on this idea, DSEGAN [Phan et al., 2020] introduces multiple generator chains to perform multi-stage augmented mapping, which enhances the coherence of the representation space compared to SEGAN. In contrast, SEflow [Strauss and Edler, 2021] proposes a normalized flow framework that directly models the enhancement process by estimating the density of clean speech utterances. Additionally, DEMUCS [D´efossez et al., 2020] and CleanUNet [Kong et al., 2022] both introduce causal SE models that operate directly on raw waveforms, further advancing the capabilities of speech enhancement technologies.

STFT domain based SE methods process speech signals in the frequency domain, where the input signals are typically amplitude spectrum, complex spectrum, and phase spectrum. MMSEGAN [Soni et al., 2018] integrates GAN within an augmented framework, using STFT domain masks to implicitly learn and predict clean STFT domain representations. In a similar vein, PSMGAN [Routray and Mao, 2022] tackles the challenge of phase information by designing a phase-sensitive speech enhancement technique, employing conditionally generated adversarial networks. Meanwhile, PR-WaveGlow [Maiti and Mandel, 2020] and Glance [Li et al., 2022] take a different approach, using the UNet architecture to enhance the representation space of the information flow through specialized modules developed in each of their frameworks. On the other hand, S4ND [P-J et al., 2023] and Dual-S4D [Sun et al., 2024] employ Structured State Spaces for Sequences, a linear time-invariant system, to provide more granular continuous characterizations of long sequences. In contrast, these works [Lu et al., 2021; Tai et al., 2023a; Tai et al., 2023b; Guo et al., 2024b] propose a different approach by modeling natural images and raw audio waveforms through paired diffusion and inversion processes, offering a novel method for both image and audio enhancement.

Multi-view based SE models typically combine time and frequency domain features to expand the representation space. In an effort to unify the feature spaces in both the time and frequency domains, [Wang et al., 2024] introduces a feature-fused two-branch SE method, which leverages the encoder from [Luo and Mesgarani, 2019]. Building on this idea, [Tang et al., 2021] proposes a two-channel attention spanning framework called TFTNet, designed to fully exploit the correlation between the time and frequency axes. TFTNet takes time-frequency spectrograms as inputs and produces timedomain waveforms as outputs, enhancing the overall performance of speech enhancement tasks by more effectively capturing cross-domain dependencies.

3 Methodology

This section describes how to convert speech signals into STFrFT spectrum and how to pick the optimal fractional counterpart STFrFT spectrum for constructing speechenhanced multiviews. Finally, we illustrate the design principles and loss functions of the network architecture. The MFSE is described in Figure 2.

3.1 Short-time Fractional Fourier Transform

The Fractional Fourier Transform (FrFT) generalizes the classical Fourier transform by introducing a fractional parameter that allows continuous interpolation between time and frequency domains, offering greater flexibility in adjusting the time-frequency balance and making it especially effective for analyzing signals with complex, non-stationary behaviors.

The STFrFT extends this concept by combining the advantages of the FrFT with the short-time analysis of the signal, allowing for a localized time-frequency representation. In the STFrFT, the signal is segmented into short-time intervals, with the fractional transform applied to each segment, allowing for more precise and adaptive feature extraction from nonstationary signals. This enables SE models to better capture the intricate, non-stationary characteristics of speech, enhancing performance in complex acoustic environments.

FRFT is based on the mathematical properties of the classical Fourier transform, and the fractional rotation of a signal in the time-frequency domain is achieved by introducing fractional powers of order. The definition of the FRFT for a signal x(t) is given by

where α is the angle, and when the α ̸= cπ (c ∈ Z), for the transformation kernel Kα(u, t) is defined as follows

In this paper, the proposed method uses an order range of α ∈ [0, π2 ], and we utilize a order p ∈ [0, 1] to control the size of the α so that pπ 2 = α ∈ [0, π2 ]. FFT is a special form of FrFT when p = 1 i.e. α = π. For the characterization of Kα(u, t) in FrFT, IFrFT can be obtained by transforming Fα by F -α again, and the definition is given below

Analogous to the STFT, the mathematical form of STFrFT is given by

where g(τ) denotes the chosen window function centered around the origin, e.g., Hamming Window or Hanning Window.

As can be visualized in Figure 3, u is able to dynamically display the entire time-frequency space by changing the size of alpha by adjusting the order p, which is STFT when p = π2 .

In practice, it is also necessary to interpolate discrete signals to ensure accurate computation, and in this paper, sinc interpolation is used to recover continuous signals from discrete samples. It has the unique property of being the "perfect" interpolation kernel, as it perfectly reconstructs a bandlimited signal from its samples in the context of the Shannon-Nyquist theorem. The sinc function is given by:

The following section describes the process of how to select an STFrFT spectrum corresponding to the optimal order from the candidate views as an auxiliary view with the STFT to form the multi-view input.

3.2 Order Selection

Selecting the optimal fractional order in SE algorithms is crucial because it allows for a precise balance between time and frequency domain representation, enhancing the signal's clarity and intelligibility. By adjusting the fractional order, the algorithm can better suppress noise while preserving important speech features such as pitch and formants, thus improving the SNR and overall speech quality. The optimal fractional order adapts to different noise types, environmental conditions, and speech signal characteristics, ensuring both effective noise reduction and the preservation of natural speech. This selection also ensures computational efficiency by avoiding unnecessary complexity, leading to a more robust and effective enhancement process. In order to select the optimal order, we devised a method of selection using the spectral centroid. The calculating spectral centroid is defined as follows:

where C is the spectral centroid, N is the number of discrete frequency points in the spectrum, fk is the k - th frequency point, X(fk) is the complex spectrum of the signal at the frequency point fk (usually the result of the Fourier transform), and |X(fk)| is its modulus, which represents the amplitude at that frequency point.

We calculated the spectral centers C T p=0 , · · · C T p=0.9 for all candidate STFrFTs time lengths of T .

Algorithm 1 Compute the spectral centroid

Output: cent

• 1: Initialize set cent.
• 2: for i = 0 to 0.9 do
• 3: Initialize set centroids, freq = FFTfrequencies
• 4: for t = 1 to T do
• 5: Initialize centroid, mag = ∣∣xt×F α=i ∣∣
• 6: centroid = sum(freq ×mag) / sum(mag)
• 7: centroids ← centroid
• 8: end for
• 9: cent ← sum(centroids)
• 10: end for
• 11: return cent

where M is the number of order and C ′T p is the distance from C T p to average spectral centroid. Finally, based on the C ′T p of each order, the order p corresponding to the maximum C ′T max , average C ′T avg and minimum C ′T min are obtained.

After selecting the STFrFT spectrum corresponding to the completed optimal fractional order, we connect it to the STFT spectrum in the channel dimension as a multiview input, and the advantages of this to construct a multiview will be discussed later.

3.3 Multi-viewConstruction

The most direct way to utilize the multi-view idea for SE is to utilize the combination of time domain and frequency domain. However, the time-domain and frequency-domain features are heterogeneous, requiring fusion through feature mapping that projects both into the same high-dimensional space. This introduces a non-consistency factor between the views, making the fusion dependent on the quality of the feature mapping and the design of the loss function [Xue et al., 2024; Zheng et al., 2024]. In multi-view fusion, keeping consistency between views helps to describe the consistent target more accurately [Son et al., 2024; Wu et al., 2025], and if the consistency between two views is low, it often can broaden the feature space to some extent, but it does not help much in feature extraction of the target itself. The STFT is a special case of the STFrFT, differing only in the order p of the transform kernel, while remaining consistent in the time dimension. This consistency allows both to simultaneously characterize the distributions of speech and noise within the same time frame, enabling them to be input into the network as a whole and mitigating, to some extent, the conflicting nature of the viewpoints.

In deep learning, especially in SE networks, the instability of spectral resolution affect the performance of the SE model, and retaining fixed input features helps to maintain stability during the training process [Shi et al., 2023]. STFrFT dynamically adjusts the degree of time-frequency localization of the signal by introducing a fractional parameter p, which allows STFrFT to provide more flexible time-frequency resolution theoretically, but also increases the computational complexity. Selecting different fractional order parameters will have different effects on the time-frequency localization of the signal, which may lead to fluctuations in the time-frequency resolution, especially when the selected fractional order is large or small. The instability of the spectral resolution may affect the performance of the enhanced model. STFT as a fixed mode involved in training can provide stable time-frequency representation, thus yes the model can be maintained to a certain extent stable training, the introduction of the optimal short-time fractional view can be utilized to exploit the complementary nature of the multiview network.

3.4 Multi-view Fusion SE Framework

In this paper, we design the Multi-view Fusion SE Framework (MFSE) architecture and the overall architecture is shown in Figure 2, and we follow [Sun et al., 2024; Chen et al., 2024] using a UNet architecture. The encoder consists of three convolutional layers, two downsampling layers, and three DownUp Blocks. The first convolution fuses multi-view information and expands it to a specified dimension, while the next two convolutions use depth-separable convolutions to integrate downsampled information for better compatibility with the Down-Up Block module. The decoder includes a UNet upsampling module, a mask deconstructor, and a complex domain deconstructor. The UNet upsampling module mirrors the encoder, with three upsampling layers and three Down-Up Blocks. The UNet encoder and decoder parts of the corresponding layers utilize the fusion module for fusing the conduction information flow. The mask deconstructor is used to decode the features into a frequency domain mask, which is used to identify the noise components in the frequency domain and mask them out, and then combine them with the phase information to transform them into the real and imaginary parts of the frequency domain, where MASK is the output of the mask deconstructor and Ym and Yp represent the amplitude spectrum and phase of the input main view STFT, respectively.

The complex domain deconstructor is used to decode the features into real and imaginary parts in the frequency domain and finally the final output is weighted and fused with the real and imaginary parts converted by the mask deconstructor as shown by

Through experimentation, we finally set a to 0.75.

Down-Up Block, the main feature extraction module of MFSE, consists of three parts: (i) TF-DenseBlock (ii) TFConvBlock (iii) Low-High Frequency Fusion (LHFF) based on multi-granularity attention block. The details of the three modules are shown in Figure 2(c).

TF-DenseBlock uses depth-separable expansion convolution combined with the residual structure of DenseNet, where larger sensory fields tend to give better results in speech tasks. Under the premise of ensuring lightweight, TF-DenseBlock performs feature extraction and fusion of features at different scales, expanding the sensory ability of the extractor on the features.

TF-ConvBlock utilizes depth-separable convolution for feature extraction in the time and frequency dimensions respectively after TF-DenseBlock extracts multiscale features, keeping the extractor's ability in the time and frequency dimensions dynamically balanced to improve the stability and performance of model training.

The multi-granularity attention block leverages multigranularity features to perform stagewise fusion of symmetric UNet features at the same stage. It employs channel and spatial attention mechanisms for coarse-grained feature extraction, followed by a 7×7 convolution and sigmoid activation to compute fine-grained attention for each spectral feature. The symmetric features are then weighted and fused. To address the neglect of the correlation between low-frequency and high-frequency information in previous SE methods, we designed the LHFF module. The LHFF module divides the spectral features into four equal parts along the frequency axis. It passes the low-frequency features in sequence with other frequency bands through a multi-granularity attention block and then splices them in the frequency dimension. LHFF progressively maps from coarse-grained to finegrained features, using the well-recovered low-frequency information to guide the optimization of high-frequency components, which are strongly affected by noise.

3.5 Loss Function

The loss function is used in this paper as a weighted fusion of the magnitude spectrum with three weighted fusions of the complex spectrum and the time-domain loss. The magnitude spectrum is defined as follows:

where ˆ Y ′′m = √ ˆ Y ′′ r 2 + ˆ Y ′′ i 2 . In order to more accurately portray the degree of spectral reduction, we optimize using the complex spectral loss, which is defined by

Utilizing time loss enables a greater focus on the metric SSNR is given by

where ˆx = ISTFT(ˆ Y ′′ r , ˆ Y ′′ i ), x is clean speech lable. The final loss is formulated as follows:

where we assign the value of α to 0.9 and β to 0.2.

4 Experiments

In order to verify the noise reduction performance and generalization ability of the proposed method, we conducted a series of experiments.

4.1 Datasets

VoiceBank+DEMAND is widely used for SE benchmarking. It includes recordings from 30 speakers and 10 noise types, divided into training and test sets with 28 and 2 speakers, respectively. Training uses four signal-to-noise ratios (SNRs) of [0, 5, 10, 15] dB, while testing uses [2.5, 7.5, 12.5, 17.5] dB. The pristine audio is sourced from the VoiceBank corpus, consisting of 11,572 speech instances from 28 speakers in the training set, and 824 utterances from 2 unseen speakers in the test set. The dataset covers a wide range of noisy environments, including public spaces (cafeterias, restaurants, offices), domestic settings (kitchens, living rooms), and urban transport hubs (cars, metros, buses, and subway stations). For our experiments, all speech samples were down-sampled to 16 kHz.

DNS Challenge corpus consists of over 500 hours of clean audio data recorded by 2150 distinct speakers, alongside more than 180 hours of diverse noise recordings. We synthesize 30s long clips by augmenting clean speech utterances and noise. Four signal-to-noise level bands were used to generate the generalization performance test samples, namely [- 5-0dB,0-5dB, 5-10dB, 10-15dB] to establish 10,000 speech samples each, and after that, 100 random samples of noisy each were used to evaluate the model's ability to generalize.

We follow [D´efossez et al., 2020; Lu et al., 2021; Lu et al., 2022; Sun et al., 2024; Guo et al., 2024b] using Perceptual Evaluation of Speech Quality (PSEQ), prediction of the signal distortion (CSIG), prediction of the background intrusiveness (CBAK), prediction of the overall speech quality (COVL), segmental signal-to-noisy ratio (SSNR) and ShortTime Objective Intelligibility (STOI) these indicators to evaluate our methodology.

4.2 Trainning Setup

Throughout the training process, the speech data was uniformly partitioned into 51,040 points, using an FFT size of 510, a window length of 510, a hop length of 160, and a sampling rate of 16kHz. The training configuration included a batch size of 4, a learning rate of 5e-4, with learning rate decay occurring every 30 epochs and a decay coefficient of 0.5. The model architecture used a base channel size of 64. All models were trained for 120 epochs using the AdamW optimizer. Training was performed on a single 46GB NVIDIA L40 GPU.

4.3 Performance on VoiceBank+Demand

Compared with other comparative methods on the Voice Bank+Demand dataset, the algorithm proposed in this paper has a good performance on all the metrics, as shown in Table 1, where it can be seen that the MFSE improves on CISG, CBAK, COVL and SSNR by 5%-11.7% over the second place metrics in the table.

In order to analyze the effect of the optimal order on the model performance, we also performed the experiments in Table 2 according to from the max spectral centroid (SCmax), the min spectral centroid (SC-min), and the average spectral centroid (SC-avg), and the experiments found that the SC-avg chosen to have the best metrics of the order voice enhancement.

4.4 Generalizability Performance on DNS Challenge

After the model training was completed, to test the difference in generalization ability using the single view model (BASE) and the multi-view model (MFSE), we tested it using the DNS challenge dataset. We evaluated the model trained using the BASE model and the model trained using SC-avg on four different signal-to-noise ratio distributions and showed the improvement in various metrics as shown in the figure 4, which suggests that the approach using multiple views has better generalization ability.

4.5 Ablation Study

We conducted ablation experiments by separately using only the STFT instead of the multi-view method

(Single-Spectrum), without the TF-DenseBlock (w/o TF-DenseBlock), without the TF-ConvBlock (w/o TFConvBlock), and without the LHFF (w/o LHFF). The experimental results after ablation are shown in Table 3. The analysis of the ablation experiments yielded that the number of model parameters remained basically unchanged after the addition of STFrFT as the multiview input, and PESQ, CISG, CBAK, and COVL were improved by 0.12, 0.07, 0.07, and 0.1, respectively, which verified the validity of the multiview input based on STFrFT as SE at the experimental level. When the TF-DenseBlock is omitted, the number of parameters remains largely unaffected, but all metrics show a degradation, with PESQ experiencing the most significant decline. When the TF-ConvBlock is not used, both PESQ and COVL metrics deteriorate more noticeably. The absence of LHFF leads to an even greater reduction in the PESQ and COVL scores.

We visualized the spectra obtained from the ablation experiments, as shown in Figure 5. In the figure, the yellow boxed region in "w/o TF-ConvBlock" shows poorer recovery compared to MFSE, the green boxed region in "w/o TFDenseBlock" is less well-recovered than MFSE, and the blue boxed region in "w/o LHFF" also exhibits worse recovery than MFSE.

5 Conclusion

We propose a multi-view SE approach based on short-time fractional Fourier transform. Specifically, it is divided into two parts: multi-view construction and enhanced network. The former uses STFrFT and STFT to construct multi-view input to expand the representation space for speech and noise separation, while the latter uses the UNet architecture to combine the correlation between the low-frequency and highfrequency information of speech to reconstruct speech. Experiments show that our proposed method performs well in the benchmark test VoiceBank+Demand and has some generalization ability under multiple signal-to-noise ratios in another benchmark test DNS challenge.

Acknowledgements

This work was supported by National Natural Science Foundation of China (Nos. T2495251, 62306171, 62406218, 62472267), the Science and Technology Major Project of Shanxi (No. 202201020101006), and Fundamental Research Program of Shanxi Province (Nos. 202303021211023, 202303021221075, 202203021222183).