Innovative Directional Encoding in Speech Processing: Leveraging Spherical Harmonics Injection for Multi-Channel Speech Enhancement

1 Introduction

Multi-channel speech enhancement involves extracting a desired speech signal from noisy environments using data captured by multiple microphones. This technique is critical for applications including, but not limited to, video conferencing systems [Mehrotra et al., 2011; Rao et al., 2021], distant communication [Nguyen et al., 2014; Tan et al., 2019], and hearing aids [Doclo et al., 2010; Nossier et al., 2019]. Unlike single-channel methods that rely solely on spectral and temporal properties, multi-channel enhancement uniquely capitalizes on spatial information. By exploiting spatial cues like inter-channel differences, multi-channel systems can substantially improve speech clarity, background noise reduction, and overall listening experience compared to single-channel techniques. However, effectively integrating and processing spatial cues remains an open challenge.

The evolution of multi-channel speech enhancement techniques can be broadly categorized into two main approaches: traditional spatial filtering methods and deep learning techniques. Traditional methods focus on extracting signals based on differences in phase and timing across microphones, which struggle in noisy environments due to difficulties accurately estimating spatial information [Hoshuyama et al., 1999; Souden et al., 2009]. The second approach involves deep learning methods, which initially processed each channel independently [Wang and Wang, 2018] and later evolved into more advanced techniques such as merging deep neural networks with traditional beamforming [Li et al., 2024] to combine the strengths of deep learning and array processing. However, the short-time Fourier transform (STFT) representation has difficulty expressing the spatial information of sound sources. Effectively incorporating spatial information remains a challenge. Independent per-channel processing fails to capture inter-channel dependencies and spatial relationships, which provide valuable context. Directly modeling full multi-channel spatial correlations is computationally infeasible. More efficient spatial feature extraction is needed to incorporate spatial information without excessive complexity and fully capitalize on spatial diversity in multichannel scenarios.

Fortunately, spherical harmonic coefficients(SHCs) obtained via SHT provide a comprehensive spatial representation of soundfields [Kumar and Hegde, 2016; Varanasi et al., 2020]. This spherical harmonic representation offers two key advantages for multi-channel speech enhancement: Effective capture of spatial information [Lugasi and Rafaely, 2020]: Unlike the STFT, SHT primarily captures the spatial distribution characteristics of soundfields. Grounded in spherical harmonics theory, the SHT discerns the spatial attributes of signals arriving from various directions and their inter-relations across microphone channels. Such spatial capture is crucial for multi-microphone speech enhancement. In a microphone array, this transform adeptly captures the spatial orientation of sounds, enabling more precise differentiation between target speech and background noise. Enhanced spatial resolution [Rafaely, 2015]: Spherical harmonics constitute a complete basis for functions defined on the spherical surface. Thus, any spherical function can be represented precisely as a linear combination of spherical harmonics. The SHT facilitates an accurate depiction of a sound field's spatial distribution. The order of the spherical harmonics determines the granularity of spatial feature capture. While lower orders delineate broad spatial patterns, higher orders characterize finer spatial nuances. By selecting a suitable order, the desired spatial resolution can be achieved. If the spatial information in these SHCs can be fully utilized, this may help compensate for the lack of descriptive spatial modeling in current mainstream approaches. Exploiting this could greatly improve the performance and robustness of multi-channel speech enhancement. Furthermore, SHT has the property of converting signals from varying numbers of microphones into coefficients of consistent dimension. This property can be utilized to address the problem of generalizing to varying numbers of microphones by formulating multi-channel speech enhancement independently of the number of input microphones. As a result, deeper neural networks can be employed for the task.

In this work, we propose a method to fully leverage spatial information using SHCs as auxiliary model inputs. We design two network variants that incorporate SHT as auxiliary inputs: parallel and serial approaches. Specifically, the parallel model contains two separate encoders - one that takes the STFT of the input signals, and another encoder that processes the SHCs obtained by applying SHT to the inputs. By fusing both encoders in the decoder to estimate the enhanced STFT, we effectively incorporate spatial context. For the serial approach, SHT is first applied to transform the multi-channel inputs into the spherical harmonic domain. Then, STFT is taken of the transformed signals represented as SHCs to obtain features. Unlike the parallel approach which processes STFT and SHT separately, the serial approach applies SHT before STFT so the network inputs encompass both spectral and spatial information from the start. This serial use of SHT then STFT aims to more deeply integrate spatial cues into feature extraction versus direct STFT features. Our contributions can be summarized as three-fold:

• We integrate SHT into deep learning methods to enable improved spatial processing for multi-channel speech enhancement.
• We introduce two novel network architectures: a parallel model that handles STFT coefficients and SHCs independently, and a serial model for jointly processing the combined spatial-spectral data.
• We demonstrate the proposed model achieves superior performance across various conditions, effectively generalizes to different microphone array configurations

2 RelatedWork

Traditional spatial filtering methods. Traditional approaches include spatial filtering methods such as delayand-sum beamformer [Klemm et al., 2008], minimum variance distortionless response (MVDR) beamformer [Souden et al., 2009], super-directivity beamformer [Bitzer and Simmer, 2001], and others. These leverage phase and timing differences between microphones to preferentially extract signals from certain directions. Although these approaches can perform well, their performance depends on reliable estimation of spatial information, which can be challenging to accurately estimate in noisy conditions.

Deep learning techniques. Deep learning techniques have made significant progress in multi-channel speech enhancement recently. Earlier deep learning methods such as GCRN [Tan and Wang, 2019], DPCRN [Le et al., 2021] focused on spectral mapping, processing each channel independently. To better preserve spatial cues, [Liu and Zhang, 2021] proposed the inplace gated convolutional recurrent neural network (IGCRN) which efficiently retains spatial information in each frequency bin without the downsampling and upsampling used in conventional CRNs. Later, [Tan et al., 2022] introduced the concept of neural spectro-spatial filtering which jointly optimizes spectral and spatial filtering using a convolutional neural network with densely-connected blocks. This achieves significant gains over prior approaches for multimicrophone speech enhancement. More recent methods combine DNNs with traditional beamforming to better utilize spatial information. Examples like FasNet [Luo et al., 2019], MIMO-Unet [Ren et al., 2021], and EabNet [Li et al., 2022] exploit complementary strengths of deep learning and array processing for state-of-the-art performance.

Spherical harmonic transformation. The spherical harmonic basis functions constitute a comprehensive set of basis functions in spherical coordinates, enabling the expression of any square-integrable function or star-shaped object in a mathematical closed form. Consequently, it is possible to decompose any sound field on the sphere into spherical harmonic components using the spherical harmonic transform, also known as the spherical Fourier transform, which employs the spherical harmonics. The primary applications in the field of speech involve sound field reconstruction and direction-of-arrival (DOA) estimation. [Xu et al., 2021] utilized spherical harmonic decomposition and headrelated impulse responses (HRIR) to establish a correlation between spatial locations and received binaural audio signals. [Dwivedi et al., 2022] introduced a low-complexity convolutional neural network-based method for DOA estimation in the spherical harmonic domain. [Weng et al., 2023] put forward a solution to the problem of multi-source DOA estimation in challenging environments characterized by reverberation and noise, involving the utilization of a spherical harmonic domain beam-space MUSIC algorithm.

3 Methodology

We consider a set of microphone arrays of arbitrary configuration located at the origin of the Cartesian coordinates and composed of II omnidirectional microphones. Let ri = (ri cosϕi sinθi, ri sinϕi sinθi, ri cos θi)T denote the position of the i-th microphone of the array, where ri represents the distance of the i-th microphone to the center of the array. The azimuth ϕi is measured counterclockwise from the x-axis, and the elevation angle θi is measured downward from the z-axis. The adopted spherical coordinate system is illustrated in Fig.1.

The array is assumed to be positioned in a reverberant sound field. According to the image method [Allen and Berkley, 1979], the sound pressure in a reverberant environment, generated by a single source in the far field, can be modeled as a sum of L significant plane waves produced by L image sources under free-field conditions. This can be assumed equivalently as L far-field sound sources generating plane waves propagating through space and picked up by the microphones. Let Ψl = (θl, ϕl) denote the direction of propagation of the l-th sound source, and let kl = -(k cosϕl sinθl, k sinϕl sinθl, k cos θl)T represent the wave number vector of the l-th plane wave.

3.1 Space Domain SystemModel

The signal received by the i-th microphone in the frequency domain can then be expressed as:

where vi (k, Ψl) denotes the steering vector of the i-th microphone associated with the l-th plane wave. sl(k) is the complex amplitude of the l-th plane wave, and ni(k) is the noise received by the i-th microphone. The frequency domain received sound pressure model can be expressed in matrix form as:

where V (k,Ψ) is the I × L dimensional direction matrix. s(k) = [s1(k), s2(k), . . . , sL(k)]T is the L dimensional source signal vector, n(k) = [n1(k), n2(k), . . . , nI(k)]T is the I dimensional zero-mean Gaussian white noise vector, and n(k) is assumed to be uncorrelated with s(k). By Npoint STFT, the p(k) in the T-F domain can be recorded as Pi(t, f ):

where t represents frame index and f represents frequency bin obtained from STFT. Xi(t, f) and Vi(t, f) represent the target and noise component, respectively.

3.2 Spherical Harmonic Domain SystemModel

In this section, we describe the proposed method for modeling acoustic signals in the spherical harmonic domain through the utilization of the SHT. By calculating the coefficients of spherical harmonics, the received speech signal at a specific point on the sphere surface can be estimated. The spherical harmonics Y m n (θ, ϕ) of order n (n ∈ N) and degree m (m ∈ Z and -n ≤ m ≤ n) are defined as [Rafaely, 2015]:

where N is the truncated order of the soundfield, (.)! is the factorial function, and P m n is the normalized associated Legendre polynomial. The spherical harmonic function P m n (cos θ) captures the dependency on the elevation angle θ, while the complex exponential term eimϕ captures the dependency on the azimuth angle ϕ.

According to the Fourier acoustic principle, the sampled sound pressure p(k, r) and its spherical harmonic domain representation pnm(k, r) at frequency k and angle (θ, ϕ) can be expressed as:

As explained in [Rafaely, 2015], the coefficients pnm diminish for k, r in a range smaller than N and can therefore be neglected for n > N. Hence, Eq.5 can be approximated for an appropriate finite order N:

where N is the truncation order, p(k, r) denotes the timedependent amplitude of the sound pressure in free threedimensional space, pnm(k, r) are the weights known as coefficients of the SHT, k = 2πf/c is the wave number, f is the frequency, and c is the speed of sound in air. The coefficients pnm(k, r) are defined as [Rafaely, 2015]:

where (.)∗ denotes complex conjugation. To satisfy the farfield condition, the distance d between the sound source and the center of the microphone array must exceed 8r2 f/c [Meyer, 2001], where r is the array radius. This ensures negligible wavefront curvature effects. For n ≤ N, pnm(k, r) can be obtained as:

where ri = (r, θi, ϕi) is the location of the i-th physical microphone and I is the number of physical microphones.

3.3 Two variants of SH Injection

We propose a novel architecture that fully leverages spatial information by using SHCs as auxiliary inputs to the model. The key innovation is to introduce SHT for multi-channel speech enhancement to explore spatial clues. The microphone array signals are transformed into the spherical harmonic domain to obtain SHCs, pnm(k, r), up to order N, which compactly encode the spatial distributions.

In proposed-parallel system, as is done in Fig. 2, the coefficients form SHT are then provided as auxiliary inputs to a dedicated spatial encoder, along with the spectrograms from the STFT fed to the main spectro-temporal encoder (as in IGCRN). Finally, the enhanced embeddings from both encoders are concatenated and passed to the decoder, which reconstructs the clean speech spectrogram. Using SHT provides orientation-invariant coefficients that concisely capture useful spatial properties and inject global contextual information about the soundfield to guide the model. The dualencoder architecture enables joint modeling of spectral and spatial cues for improved speech enhancement.

In proposed-serial system, as is done in Fig. 3, the coefficients obtained from applying the SHT to the input audio signals are used as inputs to the STFT. This provides STFT coefficients that contain both spectral and spatial information from the beginning of the processing pipeline. The sequential application of SHT followed by STFT aims to deeply integrate spatial cues into the feature extraction process compared to directly using STFT features alone. By applying SHT first, spatial information from the spherical harmonics domain is incorporated into the spectral features computed by the subsequent STFT. This is intended to provide the neural network with inputs that integrate spatial cues more deeply than conventional STFT features. The algorithm outlining the proposed system is presented in Algorithm 1.

4 Experiments

4.1 Datasets

To evaluate the performance and generalization of the proposed method, sound-field data was simulated using two datasets: TIMIT [Zue et al., 1990] and MS-SNSD [Reddy et al., 2019]. The simulation and evaluation process is described as follows:

TIMIT database. To evaluate the performance of the

Algorithm 1 Algorithm for the proposed method.

Require:

A minibatch data {X(t, f)mix, pnm(k, r)mix, s(k)}. Where X(t, f)mix is the result of STFT of mixed speech. pnm(k, r)mix denote the SHCs of order n and degree m, which are obtained by applying the SHT to the multichannel mixed speech signal. s(k) is the target speech. learning rate is µd.

Ensure:

The optimized proposed model

• 1: for number of training iterations do

2: for m-th minibatch do 3: if proposed-parallel then 4: STFTout = Encoderstft(X(t, f)mix) 5: SHTout = Encodersht(pnm(k, r)mix) 6: LSTMout = BiLSTM(STFTout, SHTout) 7: else if proposed- serial then 8: STFTout = Encoder(STFT (pnm(k, r)mix)) 9: LSTMout = BiLSTM(STFTout) 10: end if 11: estout = Decoder(LSTMout) 12: out = ISTFT (estout) 13: Loss =MSE(out, s(k)) 14: θd ← θd - µd ∂Loss ∂θd 15: end for 16: end for 17: return θd

proposed system, the training data is generated by convolving multi-channel room impulse responses (RIRs) [Allen and Berkley, 1979] with speech signals from the TIMIT database. The TIMIT clips are divided into non-overlapping training, validation, and testing sets. Noise clips from the DNSChallenge corpus are used for training and validation, while NOISEX-92 [Varga and Steeneken, 1993] and cafe noises from CHiME3 [Barker et al., 2015] comprise the testing set. RIRs are generated using the image method based on a 9microphone uniform circular array with radius 0.035 m, randomly positioned inside a 6 × 5 × 4 m3 room. The sourcearray distance is 1 m. RIRs are simulated with varying SNR (-6 to 6 dB) and RT60 (0.2 to 1 s) values. For the purpose of evaluation, we define three distinct SNR levels: -5dB, 0dB and 5dB. In addition, we explore five distinct T60 values, ranging from 0.2s to 0.6s, with intervals of 0.1s. This comprehensive configuration results in the generation of 350 pairs for each specific case.

MS-SNSD database. To assess the generalizability of the proposed system, we modified the MS-SNSD to employ gpuRIR's[Diaz-Guerra et al., 2021] implementation of the image source method for room impulse response simulation. To simulate acoustic scenes, we selected a speech sample and a noise sample which were rescaled to random SNRs within the range of [-10, 10] dB and RT60 [0.2, 1.0]s. Sources were positioned randomly inside a room measuring 6 × 5 × 4 m3 . The distance between sources and array was set to 1 m. The radius of the microphone array was 0.035 m. Microphone locations within the room were distributed uniformly at random. Distinct testing and training sets were generated without overlap of speech or noise samples between sets. The number of microphones in the training and verification sets was 8, and the number of microphones in the test set was [4, 8, 12, 16]. The amount of data in the training set was 42,000 samples, the amount of data in the verification set was 4,200 samples, and the test set in each case contained 7,000 pieces of test data. Each scene simulation lasted two seconds.

4.2 Experiment Setup

Network Configuration

The proposed parallel and serial methods build upon the baseline IGCRN model [Liu and Zhang, 2021], which utilizes an encoder-decoder structure for multi-channel speech enhancement. For the Proposed-parallel system, the input features of the STFT and SHT are fed into two independent encoders. Each encoder consists of six cascaded 5 × 1 kernels inplace gated linear units (GLU), which are constructed using inplace convolutions as follows:

where ELU(.) and Sigmoid(.) are activation functions, BN(.) is batch normalization, iConv is inplace convolution, and ⊗ denotes element-wise multiplication. To achieve a similar computational cost and number of parameters as IGCRN, which has 64 GLUs, we set the number of input channels for each GLU here to 32. After the encoders, we concatenate their outputs along the channel dimension and feed them into a channel-wise LSTM to refine the spatial information. The decoder consists of six cascaded inplace transpose GLUs with 128 input channels per transpose GLU.

For the Proposed-serial systems, the coefficients obtained from applying the SHT to the input audio signals were used as inputs to the STFT. We set the number of input channels for each gated linear unit (GLU) here to 64.

Training Configuration

For the SHT, N = 4, resulting in 25 spherical harmonic functions: Y 0 0 (θ, ϕ), Y -1 1 (θ, ϕ), Y 0 1 (θ, ϕ), Y 1 1 (θ, ϕ), · · · , Y 4 4 (θ, ϕ). Then the complex value of each Y m n (θi, ϕi) for the i-th microphone is specified. By employing (8) a set of pnm(k, r) is calculated which consists of 25 signals in the spherical harmonics domain.

All utterances had a frame length of 32 ms and a frame shift of 16 ms. The square-root Hann window was used as the analysis window. The sampling rate was 16 kHz. A 512-point discrete Fourier transform was used to extract complex STFT spectrograms. All models are trained using Adam optimizer with a fixed learning rate of 1e-3. If validation loss does not decrease for consecutive two epochs, the learning rate will be halved. All models are trained for 100 epochs.

4.3 EvaluationMetrics

In this study, the perceptual evaluation of speech quality (PESQ)[Rix et al., 2001] and short-time objective intelligibility (STOI)[Taal et al., 2011] metrics are used to objectively evaluate the enhancement performance of different models. PESQ assesses speech quality on a scale from -0.5 to 4.5, with higher scores indicating better quality. STOI gauges speech intelligibility on a scale of 0% to 100%, where higher scores reflect better intelligibility. Improved scores in both metrics reflect better performance.

4.4 Results and Discussions

In our experimental section, we carried out four main experiments. The first tested the overall performance of our method. The second, an ablation study, compared two model variants to identify the more effective one. The third assessed how the method performed with various microphone setups, checking its adaptability. The fourth extended these variants to more complex network architectures to show the method's generalizability. Together, these experiments comprehensively evaluated the method in different settings.

Performance of Proposed Structure

We compare against two baseline models: GCRN, which uses convolutional recurrent networks for complex spectral mapping, and IGCRN, which extends GCRN with inplace convolutions. We propose two variants of IGCRN:

• Proposed-serial: The coefficients obtained from applying the SHT to the input audio signals were used as inputs to the STFT, and then the STFT coefficients were fed into a single IGCRN model.
• Proposed-parallel: The SHT and STFT features are fed in parallel into two separate encoder branches of a dualencoder IGCRN.

In Proposed-serial, the spatial and spectral features are combined into a single stream input to IGCRN. In Proposedparallel, the SHT and STFT features are processed independently in dual encoder pathways before fusion. Both architectures augment the baseline with additional spherical harmonic spatial cues to enhance separation performance.

Experiments were conducted at -5dB, 0dB, and 5dB SNR levels, with 0.2s to 0.6s reverberation times. As shown in Figures 4 and 5, the results demonstrate the superiority of the two proposed models utilizing SHT over the baseline model without SHT. Specifically, at -5dB SNR, Proposed-parallel achieved an average PESQ score of 1.89, while Proposedserial scored 1.79, compared to 1.74 for the baseline IGCRN. The STOI results followed a similar trend, with both proposed models surpassing the baseline STOI score. Overall, the results indicate that the proposed models can effectively incorporate and leverage the spatial cues from the SHT to achieve marked improvements in speech quality and intelligibility over the baseline model. By capturing the spatial information in the multi-channel input via spherical harmonics, the proposed models significantly outperform the baseline lacking this capability.

Ablation Study for Performance

We further analyze the performance difference between our Proposed-serial and Proposed-parallel models. At an SNR of -5dB, the mean gains observed are 0.1 for PESQ and 4.41% for STOI. As the SNR increases to 0dB, more substantial improvements are attained, with gains of 0.13 in PESQ and 2.69% in STOI. Finally, at the highest tested SNR level of 5dB, the enhancements over the unprocessed signals are 0.14 for PESQ and 1.41% for STOI, as shown in Figures

4 and 5. The gains from parallel encoders remain consistent as we vary the noise and reverberation levels. This indicates that modeling the SHT and STFT features separately enables better representations to be learned, compared to serially processing the concatenated features. The network can train more specialized encoders when the inputs are independent. In contrast, the serial design forces the model to process the entire concatenated input in one encoder. This makes disentangling the spatial and spectrotemporal characteristics more challenging. The parallel approach does not have this constraint, allowing more robust joint representations to be formed after encoding. In summary, our ablation study demonstrates the superior performance achieved by modeling the SHT and STFT features in parallel architecture for multi-channel speech enhancement.

Performance of Generalization

By utilizing SHCs as auxiliary inputs, multi-channel speech signals can be transformed into a fixed dimension. This formulation allows designing multi-channel speech enhancement techniques independently of the microphone array size, thereby addressing the challenge of generalizing to different microphone configurations. This paper evaluates the proposed method's generalization ability under different numbers of microphones (specifically 4, 8, 12, and 16 microphones). Among these configurations, 8 microphones represent a matching condition, while the other three (4, 12, and 16 microphones) represent non-matching conditions. We use the CCAMS model [Casebeer et al., 2021] as the baseline. The CCAMS model jointly trains a microphone selection mechanism and a speech enhancement network to perform multi-channel speech enhancement using an ad-hoc microphone array, without applying SHT to explore more comprehensive spatial representations. As shown in Figures 6 and 7, comparisons were made using the PESQ and STOI measures across four different microphone configurations. For PESQ, which reflects the perceived speech quality, the proposed method consistently outperformed both the unprocessed signals and the CCAMS model across all microphone configurations in both its serial and parallel formulations. Notably, the parallel version of the proposed method achieved a substantial increase in PESQ scores, particularly for MIC 12, indicating a significant improvement in perceived speech quality. Regarding STOI, which is indicative of speech intelligibility, the proposed method again demonstrated superior performance over the unprocessed signals and the CCAMS model. The parallel version of the proposed method achieved the highest intelligibility scores, peaking at MIC 12. These results underscore the proposed method's ability to effectively enhance speech quality and intelligibility across varying numbers of microphones, highlighting its potential for robust application in diverse multi-channel speech processing scenarios.

Ablation Study for Generalization

In our research, we conducted ablation experiments to delve deeper into the potential of spherical harmonic information across different network architectures. We aimed to test two hypotheses: 1) Whether standardizing input dimensions through spherical harmonics would enable the use of more complex, deeper networks, thereby improving generalization, and 2) Whether our method could be effectively integrated with these deeper networks without affecting performance. Our baseline for comparison was the CCAMS model, which does not incorporate spherical harmonic information. We examined the performance of three distinct networks - IGCRN, ICCRN, and TFGrid [Wang et al., 2023] - in serial and parallel configurations. As shown in Tables 1 and 2, our meticulously detailed PESQ and STOI scores for arrays of 4, 8, 12, and 16 microphones clearly showed that models using spherical harmonic information outperformed the baseline. Notably, the TFGrid network, especially in parallel, displayed a remarkable advantage over other models in all tested microphone configurations. This finding validated our first hypothesis by demonstrating the benefits of deeper networks when enabled by standardized spherical harmonic inputs. Furthermore, our second hypothesis was confirmed - the robust integration of our method with various network architectures was evidenced by consistent, notable performance improvements across models. This proves the flexibility and robustness of our approach, even when combined with complex networks.

5 Conclusion

In this work, we propose a method to effectively utilize spatial information by incorporating SHT as an auxiliary input to models. We design two variants for incorporating SHT: parallel and serial approaches. Spherical harmonics provide a compact representation of spatial cues between microphones, improving multi-channel speech enhancement performance. By mapping speech signals to a fixed dimensional space via SHT, deeper networks can be employed to generalize to varying microphone numbers. Experiments demonstrate that both variants introducing SHT outperform established baselines with significantly reduced computations and parameters. By incorporating directional cues through SHT, our model effectively enhances speech. Ablation studies validate superior performance from modeling SHT and STFT features in parallel encoders rather than sequentially, highlighting benefits of joint spatial-spectral modeling. Moreover, our methodology demonstrates enhanced generalization across diverse microphone array configurations.

Acknowledgments

This work was partially supported by the China National Nature Science Foundation (No. 61876214, No. 61866030), the Innovation Research Project for Postgraduate Students at Inner Mongolia University, and the Fund for Supporting the Reform and Development of Local Universities (Disciplinary Construction).