An Association-based Fusion Method for Speech Enhancement

1 Introduction

Speech enhancement (SE) aims to improve the quality of a speech signal by reducing noise and other distortions, thereby making it more suitable for further analysis or utilization [O'Shaughnessy, 2024]. SE has become a critical process in various applications including communications, hearing aids, and medical applications [Chhetri et al., 2023].

Recent advancements in deep learning (DL) have brought significant improvements to SE. DL-based approaches leverage the power of neural networks to model the complex relations between noisy and clean speech signals, showing remarkable success in various scenarios [Tai et al., 2021]. These methods typically focus on minimizing the overall difference between the target speech and its denoised speech using Lp-norm distance [Li et al., 2021]. However, these discriminative models inherently lack the ability to capture the underlying structure of speech signals and their variability in real-world environments [Phan et al., 2020]. Specifically, discriminative models primarily focus on feature mapping between noisy and clean signals, ignoring the higher-order relations within the signal where such relations are critical. As a result, they often struggle to generalize to unseen noise conditions or complex acoustic scenarios.

∗Corresponding author

Generative models have been introduced to improve SE by learning the distribution of clean speech signals and generating plausible denoised outputs [Welker et al., 2022]. The type of methods are mainly based on two architectural frameworks: generative adversarial networks (GANs) and denoising diffusion probabilistic models (DDPMs). It is well known that capturing the local and global relations between signal frames is crucial for the success of these methods. These frameworks typically employ a UNet as their foundational backbone, integrating Long Short-Term Memory (LSTM) networks or attention mechanisms within the backbone to effectively model both local and global signal relations. However, the coupled relation modeling way may not fully harness the potential of these relations.

Association is often used to characterize the relation strength between multiple variables in data. Its ability for relation modeling has been proved in multi-modal classification [Liang et al., 2022], representation learning [Liang et al., 2025]. The integration of association-based methods with advanced DL algorithms has the potential to significantly improve the performance of signal enhancement systems. In this paper, we propose a decoupled relationmodeling method, called Association-based Fusion Speech Enhancement (AFSE) method. Specifically, AFSE first constructs a graph that encapsulates the association between each time window of the speech signal, and then models the global relations between frames by fusing the features of these time windows in a manner akin to graph neural networks. Furthermore, AFSE leverages a UNet with dilated convolutions to model the local relations, enabling the network to maintain a high-resolution representation while benefiting from a wider receptive field. The contributions of our work are as follows:

• We propose an effective method for modeling global relation between frames by combining the association with graph neural network.
• An association-based fusion speech enhancement (AFSE) method is proposed, which achieves a better usage of relation between signal frames in an decoupled relation modeling strategy.
• The extensive comparison experiments on two public datasets show that AFSE achieves competitive performance with fewer model parameters compared to the state-of-the-art speech enhancement methods.

2 RelatedWork

2.1 Speech Enhancement (SE)

Recently, SE methods driven by deep learning are mainly based on the generative adversarial networks (GANs) and diffusion models.

GANs-based SE: GAN is a representative generative method and has been adopted in various domains. In the field of SE, GANs have received widespread attention in recent years and have achieved remarkable results. SEGAN [Pascual et al., 2017] is one of the earliest GAN models applied to speech enhancement. It improves the clarity of speech by using binary labels (such as real or fake) in the discriminator to distinguish enhanced speech from original speech. Subsequent work, such as FSEGAN [Donahue et al., 2018], further optimized this structure and proposed an improved generative model for speech enhancement tasks. However, these early methods rely on discrete labels for training and may not be able to directly optimize speech quality evaluation indicators related to human perception. To this end, MetricGAN [Fu et al., 2019] introduced continuous labels based on evaluation indicators and improved the training method of GAN. MetricGAN-OKD [Shin et al., 2023] further optimizes this process and achieves multi-indicator optimization through online knowledge distillation, which improves the effect of speech enhancement.

Diffusion-based SE: In the field of SE, DDPMs [Ho et al., 2020] as a class of generative model have made significant progress in recent years. The diffusion model gradually adds noise to the data through a fixed forward process, and iteratively de-noises through a parameterized reverse process to generate samples from the noise. Compared to other generative models, diffusion models demonstrate excellent sample quality and a simpler training process, while placing few restrictions on the model architecture. The application of specific diffusion models to SE includes a variety of methods. For example, DiffuSE [Lu et al., 2021] is an SE method based on a diffusion model that generates high-quality speech samples by step-by-step de-noising. CDiffuSE [Lu et al., 2022] further optimizes this process by combining conditional generation techniques to improve model generalization and enhancement. In addition, DR-DiffuSE [Tai et al., 2023b] significantly improves noise reduction and voice quality by introducing fast sampling technology and refining networks.

In summary, GANs often suffer from training instability and mode collapse. DDPMs are computationally expensive due to the iterative denoising steps. Furthermore, they model the local and global relationships of data in a coupled manner, which may not fully capture the intrinsic relationships within the data. By adopting a decoupled modeling strategy and explicitly modeling the relationships between global frames, we can better capture the intrinsic relationships within the data.

2.2 Graph Neural Networks (GNNs)

We utilize the message-passing mechanism of GNNs to achieve global frame fusion after obtaining the global association between frames. GNNs directly operate on the graph structure and aggregate information via a message-passing mechanism [Zhou et al., 2020].

The k-th layer of the GNN message-passing scheme is defined as:

where Cat(·), Agg(·) and Nv denote the concatenation, aggregate functions, and immediate neighbors of node v, respectively; h(k) v is the representation vector of node v in the k-th layer, and euv is the edge vector between nodes u and v.

3 The Proposed AFSE

The whole framework of AFSE is shown in Fig. 1, which consists of (1) frame representation learning, (2) global frame fusion with association between frames, (3) local frame fusion with dilated convolution, (4) loss function.

3.1 Frame Representation Learning

To improve the effectiveness of modeling relationships, we first extract feature representations for each frame. This way enhances the ability to capture detailed characteristics. Specifically, the mixture signal in the time domain is first transformed into the time-frequency (T-F) domain using the Short-Time Fourier Transform (STFT), and its formulation in T-F domain is written as:

where X ∈ R2×L×K , Y ∈ R2×L×K , and N ∈ R2×L×K denote the complex-valued noisy signal, clean signal, and noise, respectively. 2 is the number of channels, L is the number of time steps, and K is the number of frequency bins. k ∈ {1, · · · ,K} and l ∈ {1, · · · ,L} is the index of frequency bin and time, respectively.

Since the global distribution of speech amplitudes obtained with STFT is typically heavy-tailed, the information visible in untransformed spectrograms is dominated by only a small portion of bins. Inspired by recent results from [Ju et al., 2022] that power compression can decrease the dynamic range of the spectrum and improve the significance of lowenergy regions with more informative speech components, we apply an amplitude compression to adjust the energy distribution to achieve approximate normalization. The transformation [Tai et al., 2023b] is defined as follows:

where ∠X denotes the phase, i denotes imaginary number, and e is a natural constant. Next, the noisy spectrum ˜X is passed into an encoder Enc to obtain an intermediate feature representation H:

where C is the number of channels and M denotes the encoded feature dimension. To model the relationship between frames, we convert the features H into features F :

where Reshape(H, size) denotes the shape of tensor H is changed size. F contains L frame feature vectors, where each ft is defined as:

3.2 Global Frame Fusion with Association

To enhance the signal, we model the relationships within the signal across different time windows. The association-based method leverages these relationships to improve the signal quality by reducing noise while preserving the essential characteristics of the original signal. This section mainly introduces how to perform global frame fusion with association in a GNN-like work manner. To this end, we model the association between frames as graph.

Modeling Association Graph Between Frames. We get the representation of each frame by the frame representation learning, and we use association analysis to get the relationship graph between frames. Specifically, we construct a dense graph for each signal, where each frame is represented as a node, and any two nodes are connected with an edge based on their association. Thus, the graph can be formulated as G(V,E), where V is the set of vertices and E is the set of edges. Here, H(0) = Reshape(H, (L,C,M)) ∈ RL×C×M is the embeddings denoting the nodes in the graph, A ∈ RL×L be the adjacency matrix of the graph G(V,E). Each of its elements represents an edge which is defined as:

where a greater value means a stronger correlation, and vice versa means a weaker correlation. The constructed graph has a global perception and can mine the relationship between frames because Aij measures the connection weight between any two frames in the signals. Therefore, we named the graph G(V,E) as the global graph.

Fusion with Association. After constructing this densely connected graph, we update the node embedding by correlation fusion in a residual way, formulated as:

where ˜ A = D- 12AD- 12 is the normalized adjacency matrix. D denote the diagonal degree matrix, where Dii = ∑nj=1 Aij. α is the mixing coefficient, and H(k) denotes the output embedding of the k-th graph convolution layer.

Next we combine embeddings from different layers to capture multi-level information. After k graph convolution layers, the concatenated embeddings are:

where Cat denotes the tensor concatenation operation. Then we transforms the concatenated embedding to produce the final output, incorporating non-linear relationships. The final output is obtained through a Multi-Layer Perceptron (MLP):

where σ is an activation function. Finally, we restores the transformed output to its original dimensions through a decoder Dec for subsequent local relation fusion.

3.3 Multi-scale Local Frame Fusion

In this section, we focus on the fusion of local frames at multiple scales using a specialized module that can leverage the relation among different temporal scales to boost the speech recovery. The model uses the complex spectrum as input and follows a standard encoder-decoder U-Net architecture.

Encoder. Instead of regular convolutional layers, the encoder and decoder employ five BiConvGLU (Bi-directional Convolutional Gated Linear Units) layers. These BiConvGLUs are designed to capture local spectral-temporal patterns while compressing the spectral features. The use of BiConvGLU enhances the model's capacity to handle complex features in both time and frequency domains.

Each convolutional block within the model uses a kernel size of (2, 3) along the time and frequency axes, except for the first block, which uses a kernel size of (2, 5). The stride is set to (1, 2), meaning that while the frequency size is gradually halved, the time size remains unchanged to ensure real-time processing capability. After each convolution, the model applies Instance Normalization (InstanceNorm) and a Parametric ReLU (PReLU) activation function, which help to stabilize and accelerate the training process.

Bottleneck Layer. In the bottleneck layer, which serves as the transition between the encoder and decoder, the model incorporates Stacked Temporal Convolution Modules (STCMs) as proposed by [Li et al., 2021]. These S-TCMs are crucial for capturing long-range dependencies in the time domain, which is essential for tasks that require understanding temporal dynamics, such as speech enhancement.

The model integrates three groups of S-TCMs as the sequential module, each group stacks six S-TCM units with exponentially increasing dilation rates (1, 2, 4, 8, 16, 32). This design allows the network to cover a large temporal receptive field, which is vital for effectively leveraging relations across different temporal scales. This approach significantly boosts the network's ability to recover detailed speech information across various time scales.

Decoder. The decoder mirrors the encoder's structure but operates in reverse. It gradually restores the compressed features to their original size using five BiConvGLU layers with a fixed channel dimension of 64. The stride remains to be set to (1, 2), and the kernel size matches that of the encoder blocks, except for the last BiConvGLU, which uses a kernel size of (2, 5). The model also incorporates skip connections between corresponding encoder and decoder layers, which

Algorithm 1 AFSE Training Procedure

Input: the number of samples M, total epochs N, loss functions Q, hops K, activation function σ, concatenation operation Cat, mixing coefficient α

Parameter: W: a trainable weight matrix , U: Unet, Enc: Encoder, Dec: Decoder

• 1: for epoch = 1 to Total epochs N do
• 2: for m = 1 to M do
• 3: Sample a pair of noisy and clean speeches (X, Y )
• 4: H0 = Enc(X)
• 5: The adjacency matrix A is obtained by Eq. 7
• 6: ˜ A = D- 12 AD- 1 2
• 7: Hc =H0
• 8: for k = 1 to K do
• 9: H(k) = (1 - α) ˜ AH(k-1) + αH(k-1)
• 10: Hc = Cat(Hc, Hk )
• 11: end for
• 12: XG = Dec(σ(HcW))
• 13: ˆX = U(XG)
• 14: loss = Q( ˆX, Y )
• 15: Update the learnable parameters W, U, Enc, Dec
• 16: end for
• 17: end for

help to mitigate information loss and enhance the quality of the reconstructed output.

where U represents Unet. ˆX is the enhanced spectrum. Finally, we use the Inverse Short-Time Fourier Transform (ISTFT) to convert the enhanced speech from the timefrequency domain back into the time domain.

The whole algorithm is shown in Algorithm 1.

3.4 Loss Function

The loss function used to train the network aims to minimize the difference between the RI (real and imaginary) of the estimated spectrum and the target spectrum, as well as the magnitude of the estimated spectrum and the target spectrum.

RI components loss is defined as:

This term measures the Frobenius norm of the difference between the RI ˆXr, ˆXi components of the estimated spectrum and the target spectrum Yr and Yi.

Magnitude loss is defined as:

This term measures the difference in magnitude between the estimated and target spectra. By focusing on the magnitude, this loss helps the model to better capture the overall energy distribution of the signal.

The total loss is a weighted sum of the RI components loss and the magnitude loss, and is defined as:

where λRI and λMag are weighting hyper-parameters and are set to 0.5 and 0.5 with empirical trials, respectively. This combined loss ensures that the network learns to accurately predict both the phase and magnitude of the signal, leading to more effective speech enhancement.

4 Experiment

To evaluate the proposed AFSE, we use two datasets: VoiceBank-DEMAND (VBD) and DNS Challenge.

Training Configuration. We sample all the utterances at 16 kHz. The window size is set as 20 ms, with 50% overlap between adjacent frames. 320-point FFT is utilized, leading to 161-D in the feature axis. The model is trained on Pytorch platform with a NVIDIA RTX 4090 GPU. We use the Adam optimizer with a batch size of 8 to train the proposed model, and the learning rate is initialized as 1e-3 . Moreover, we train the model by 60 epochs for two datasets.

EvaluationMetrics. We use the following metrics to evaluate SE performance: narrow-band (NB) [Rix et al., 2001] and wide-band (WB) perceptual evaluation of speech quality (PESQ) [Rec, 2005] for speech quality, short-time objective intelligibility (STOI) [Taal et al., 2010] for intelligibility, segmental signal-to-noise ratio(SSNR), scale-invariant signal-to-noise ratio (SISNR) [Roux et al., 2019] and the mean opinion score (MOS) prediction of the speech signal distortion (CSIG) [Hu and Loizou, 2008] for speech distortion, the MOS prediction of the intrusiveness of background noise (CBAK) [Hu and Loizou, 2008] for noise intrusion, the MOS prediction of the overall effect (COVL) [Hu and Loizou, 2008] for overall signal quality.

4.1 VoiceBank-DEMAND

The VoiceBank-DEMAND [Valentini-Botinhao et al., 2016] consists of 30 speakers, with 28 used for the training/validation dataset and the remaining two for the test dataset. The training/validation and test datasets contain 11,572 utterances with four signal-to-noise ratio (SNR) (15, 10, 5, and 0 dB) levels and 824 utterances with four SNR (17.5, 12.5, 7.5, and 2.5 dB) levels, respectively. In the dataset, we compare our model with seventeen state-of-theart methods which are classified into following three categories:

• GAN-based methods: SEGAN [Pascual et al., 2017], MMSEGAN [Soni et al., 2018], MetricGAN [Fu et al., 2019], MetricGAN+ [Fu et al., 2021], DSEGAN [Phan et al., 2020] and MGAN-OKDv2 [Shin et al., 2023].
• Diffusion-based methods: DiffuSE [Lu et al., 2021], CDiffuSE [Lu et al., 2022], DR-DiffuSE [Tai et al., 2023b], DOSE [Tai et al., 2023a] and VPIDM [Guo et al., 2024].
• Other methods: DEMUCS [D´efossez et al., 2020], CleanUNet [Kong et al., 2022], GaGNet [Li et al., 2022b], TaylorSENet [Li et al., 2022a], S4ND U-Net [Ku et al., 2023] and Dual-S4D [Sun et al., 2024].

We report two types of metrics including model parameters and enhancing performance the latter one contains CSIG, CBAK, COVL, PESQ, SSNR, STOI. The experiment results are reported in Table 1, where "T" and "F" denote time domain and spectrum domain, respectively; "G" and "L" denote global relation and local relation, respectively. From Table 1, the following conclusions can be drawn.

• AFSE achieves the best results in terms of all enhancing performance except for STOI. For model parameters, the S4ND U-Net ranks the first, followed by AFSE. However, S4ND U-Net ranks only the third, third, fourth, fifth in CSIG, CBAK, COVL, PESQ, respectively.

• On the one hand, compared with DR-DiffuSE, our method has achieved comprehensive advantages. The reason may be that the core units of DR-DiffuSE is convolution, cannot model the global relation between frames. On the other hand, the enhanced performance of S4ND U-Net is inferior to our method, possibly because of the lack of local relationship modeling. This indicates that both of the local and global relation are important for SE.
• Compared to DEMUCS and CleanUNet, which model both local and global relationships in a coulped manner, our superior performance may be attributed to the decoupled fusion of global and local relationships.

In summary, our method combines high performance with a low number of parameters. It achieves excellent results in speech quality and noise suppression. By maintaining a lower parameter count while delivering superior performance, our method is suited to practical applications where both quality and resource constraints are critical.

4.2 DNS Challenge

The Interspeech 2020 DNS challenge dataset [Reddy et al., 2020] is a large speech enhancement dataset. The clean speechs are collected from Librivox and totally includes 500 hours utterances from 2150 speakers. The noise clips are from Audioset and Freesound, including 60000 noise clips with 150 classes. Following [Zheng et al., 2021], we synthesize 500 hours noisy clips with SNR levels of -5 dB, 0 dB, 5 dB, 10 dB and 15 dB for training. For evaluation, we use another 150 noisy clips from the test set without reverberation. The testing SNR levels are randomly distributed in the range from 0 dB to 20 dB.

For this dataset, we compare against nine state-of-the-art methods: LSTM based methods FullSubNet [Hao et al., 2021] and DTLN [Westhausen and Meyer, 2020], U-Net based methods DCCRN [Hu et al., 2020] and PocoNet [Isik et al., 2020]. NSNet [Xia et al., 2020] utilizes weighted speech distortion losses, and CTS-Net [Li et al., 2021] is a two-stage complex spectral mapping method. FAF-Net [Yue et al., 2022] leverages a reference-based feature alignment and fusion strategy to enhance speech quality. GaGNet [Li et al., 2022b] utilizes a dual-path glance-and-gaze structure for collaborative spectrum estimation. TaylorSENet [Li et al., 2022a] employs a Taylor-unfolding framework for decoupled magnitude and residual modeling. The results of NSNet are quoted from FullSubNet, and others are directly quoted from their respective papers.

To verify the superiority of the proposed SE system in more complex acoustic scenarios, we present the results on the Interspeech 2020 DNS-Challenge corpus, as shown in Table 2. The evaluation metrics include Params, WB-PESQ, NBPESQ, STOI, and SISNR. From the results in Table 2, we can draw the following conclusions: (1) When the training data is 3000 hours, AFSE achieves the best results in terms of all evaluation metrics. With the same amount of training data, our performance performs better than TaylorSENet that ranks the first place among the comparison methods, but the number of parameters of AFSE is less than half of ones of TaylorSENet; (2) The gradual improvement in performance with increased training set size (500h, 800h, 3000h) indicates that AFSE benefits more from larger datasets, further showing the scalability and effectiveness of our AFSE; (3) Compared with the local fusion method such as FAF-Net, TaylorSENet, AFSE achieves better overall performance, possibly due to its more comprehensive integration of global and local information, reinforcing the importance of this dual-fusion strategy.

These results furthermore validate the effectiveness and superiority of our approach, showcasing its potential to set a new benchmark in the field of speech enhancement.

4.3 Further Analysis

In the part, we further analysis our model from six aspects: each module effectiveness, hyper-parameter sensibility, impact of different SNR values and effect of different lengths.

Ablation Study. This subsection is to investigate the effect of different components in AFSE via ablation experiments. The components include frame representation, Graph and Unet. In the ablation study, we compare AFSE with three its degeneration models: Case 1: Remove the frame representation module; Case 2: Remove the Graph module; and Case 3: Remove the Unet module.

The results are shown in Table 3. From the results, we can observe that (1) Modeling the global relationships based on representations is better than that based on raw features; (2) Removing the graph structure leads to a performance decline, highlighting the importance of learning global associations between frames in speech enhancement. (3) Based on the results of removing the Unet module, it can be concluded that learning local correlation significantly contributes to improving model performance. In summary, each component in AFSE plays a key role.

Hyper-Parameters. We conducted an in-depth analysis of key hyper-parameters within our model. Fig. 2 visually demonstrates the model's performance variations across distinct k orders, ranging from 1 to 5. Our assessment encompasses the VoiceBank-DEMAND, evaluating the PESQ, CSIG, CBAK, COVL as the metric. From Fig. 2, we draw the following key observations: In general, opting for a relatively smaller hop order, such as 3, yields improved performance.

Analysis on Different SNR Values. We evaluate the performance of the model at different SNR levels on the DNS dataset. Since the speeches in different groups are different, we further generate a new DNS test set by adding noise with different SNR values to all the clean speeches in the test set. We classify its noise level (SNR ranging from -5 to 20 dB) into five groups and give the average WBPESQ, NBPESQ, CSIG, CBAK, COVL result for each group. The results in Fig. 3 are produced on the new DNS test set. This is evident from the upward trends across all evaluation metrics (WB-PESQ, NB-PESQ, CSIG, CBAK, COVL) that the performance of the AFSE improves as the SNR increases.

Analysis on Different Lengths. We evaluate the enhancement results using references with different lengths. The result is shown in Fig. 4. The result indicates that there is a noticeable improvement in all metrics as the reference length increases from 3s to 10s. This suggests that longer reference speech provides more contextual information, enabling better enhancement. However, the performance gains seem to plateau after a certain length (10s to 20s), indicating that beyond a certain point, increasing the reference length does not significantly boost enhancement quality. Considering the trade off between computing complexity and performance, we utilize the reference with 10s in our experiments.

5 Conclusions

In this paper, we have proposed an efficient association-based fusion method for speech signal enhancement that integrates global and local fusion techniques. This method efficiently captures both broad contextual relationships and detailed local patterns, achieving a better enhanced speech signal quality. Extensive experiments have demonstrated the superiority of this dual-fusion approach in improving speech enhancement performance. Our results suggest that combining global and local fusion strategies can be particularly effective in addressing the challenges of complex auditory environments. In the future, we plan to explore more refined strategies for balancing and optimizing these fusion techniques to further advance the field of speech enhancement.

Acknowledgements

This work was supported by National Natural Science Foundation of China (Nos. 62306171, T2495251, 62406218, T2495253, 62373233, 62376146), the Science and Technology Major Project of Shanxi (No. 202201020101006), and Fundamental Research Program of Shanxi Province (No. 202203021222183).