M3ANet: Multi-scale and Multi-Modal Alignment Network for Brain-Assisted Target Speaker Extraction

1 Introduction

The cocktail party problem [Haykin and Chen, 2005] refers to the ability of humans to track a particular conversation in a multi-speaker social setting, known as selective auditory attention. Target speaker extraction (TSE) is a fundamental task in speech signal processing that aims to extract the target speech using auxiliary cues, thus effectively addressing the cocktail party problem.

∗Corresponding author

Recent advances in deep learning have significantly improved TSE performance [Ge et al., 2020; Hao et al., 2024]. Traditional TSE approaches often rely on prerecorded speech from the target speaker as a cue [Xu et al., 2020], but this method requires prior knowledge of the target speaker. Other studies have explored alternative cues, such as spatial cues (e.g., the direction of the target speaker) [Wang et al., 2024; Ge et al., 2022] or visual cues (e.g., facial video) [Lin et al., 2023; Li et al., 2024]. However, in practical applications, visual cues can be hindered by obstacles in the environment, and spatial cues are inherently limited to multi-microphone setups [Zmolikova et al., 2023]. These challenges, in turn, degrade the performance of TSE in complex, real-world environments.

Recent studies on auditory attention decoding (AAD) have revealed the correlation between brain neural activities and speech stimuli [Yan et al., 2024; Ni et al., 2024], opening a new avenue for TSE techniques. Researchers are increasingly exploring how to leverage speech-related information embedded in Electroencephalography (EEG) to guide target speech extraction, thereby eliminating the need for prior speaker knowledge. Early methods focus on reconstructing speech envelope from EEG data and matching it with sound sources to identify target speech [Han et al., 2019; O'Sullivan et al., 2017], but this approach is computationally intensive due to the need to separate all sources. The brain-informed speech separation (BISS) network [Ceolini et al., 2020] innovatively integrates the reconstructed envelope into the TSE task, using it as an audio cue to avoid separating all sources. However, stimulus reconstruction requires a distinct network, which still entails considerable computational costs. More recent studies [Qiu et al., 2023; Pan et al., 2024b] have further streamlined this process by directly using EEG signals as the reference cue, and these methods have since become the standard in the field.

However, EEG and speech data are simultaneously fed into the models during training, yet they are not perfectly synchronized. This is because sound stimuli must first pass through the ear and auditory pathways before reaching the auditory cortex, where EEG signals are generated [Litovsky, 2015]. Previous works [De Silva et al., 2024; Zhang et al., 2023] only focus on the effectiveness of modality fusion but neglect the temporal alignment between speech and EEG, which increases the difficulty of efficient multi-modal fusion and negatively impacts overall performance. Additionally, convolutional neural networks (CNNs) used in most TSE speech encoders [Hao et al., 2023] often fail to effectively capture the latent features of mixed speech, leading to reduced model performance. While transformer-based encoders [Fan et al., 2025a; Heo et al., 2024] offer more power, they tend to introduce significant computational complexity.

In this paper, we propose a multi-scale and multi-modal alignment network (M3ANet) for brain-assisted TSE, which facilitates temporal alignment between EEG and speech, while extracting more comprehensive and deeper speech features from both the breadth and depth. To address the misalignment caused by the temporal asynchrony between EEG and speech, this paper introduces a modal alignment module into the brain-assisted TSE task for the first time. It employs a contrastive learning strategy to minimize the distance between EEG and speech features at the same time steps, thereby achieving precise synchronization of cross-modal data. Additionally, to fully capture the latent features in speech, the speech encoder first uses multi-scale convolutions to extract short- and long-term features, obtaining local- and global-level temporal representations. Next, GroupMamba (GM) modules are introduced to efficiently model the channel and feature dimensions of speech from different directions, capturing deeper features at each level with linear complexity. Experimental results show that the M3ANet model achieves state-of-the-art (SOTA) performance on the Cocktail Party, AVED, and MM-AAD datasets, with relative improvements of 8.2%, 12.8%, and 10.4% in scale-invariant source to distortion ratio (SI-SDR), respectively, compared to the best current baselines.

The main contributions of this paper can be summarized as follows:

• This paper proposes a new brain-assisted target speaker extraction model, introducing an innovative modal alignment module that leverages contrastive learning to align EEG and speech temporally.
• This paper proposes a multi-scale encoding approach combined with the GM module, which effectively extracts latent speech features in both breadth and depth.
• Experimental results on the Cocktail Party, AVED, and MM-AAD datasets show significant improvements over baseline models, demonstrating the effectiveness of our proposed model.

2 RelatedWorks

Neuroscience research has revealed that humans possess selective auditory attention, wherein target speech signals are closely linked to neural activity in the brain. This makes it possible to utilize EEG signals as references for target speaker extraction. O'sullivan et al. [2015] proposed that auditory stimuli can be decoded from neural responses (i.e., EEG) through a process called stimulus reconstruction [Vanthornhout et al., 2018]. BISS [Ceolini et al., 2020] is a pioneering brain-assisted TSE model that jointly trains a stimulus reconstruction network and a speaker extraction network to estimate the target speech. BESD [Hosseini et al., 2021] is the first model to directly use EEG signals as references in this field, enabling training with a single extraction network and reducing computational complexity. BESD employs a speech encoder consisting of two convolutional layers. The improved version, U-BESD [Hosseini et al., 2022], enhances feature extraction by introducing skip connections and a Ushaped structure in the encoder-decoder. However, the performance of these models remains limited. BASEN [Zhang et al., 2023], building upon ConvTasNet [Luo and Mesgarani, 2019], uses 8 layers of depthwise convolutions to better capture speech information and incorporates an effective multi-layer cross-attention fusion module, leading to substantial performance improvements. MSFNet [Fan et al., 2024], which employs a multi-scale parallel extraction network, applies convolutions at different scales to extract speech features more effectively and currently holds the SOTA performance on the Cocktail Party dataset. NeuroHeed [Pan et al., 2024a], based on the Spex [Xu et al., 2020] architecture, introduces an online self-registration auditory cue network and is the first to consider real-time processing.

However, these networks do not account for the inherent temporal discrepancies between EEG and speech, which limits further performance improvements. Additionally, the simple convolution operations in their speech encoders result in inadequate feature representation.

3 Model Architecture

The proposed model primarily consists of four components: the speech encoder, EEG encoder, modal alignment module, and speaker extractor. The system workflow is as shown in Figure 1. The speech and EEG encoders extract feature representations of the input signals, which are then fed into the fusion module for effective information integration, while also being passed through the alignment module for modal alignment. The fused features are then passed through a speaker extractor to generate an estimated mask, which is multiplied by the encoded speech features to obtain the masked speech. Finally, the target speaker's speech is decoded through a 1D transposed convolution. Each part is described in detail below.

3.1 EEGEncoder

The structure of the whole EEG encoder is shown in Figure 1. In this paper, we use the same EEG encoder with graph convolutional network (GCN) [Ren et al., 2024] as in [Fan et al., 2024]. If the EEG data are recorded with N electrodes, it can be abstracted as a graph G = (V,E) (Here, E denotes the set of edges, while elsewhere in the paper, it represents the EEG data), where V is the set of |V | = N channels, and E ⊂ V × V is the set of undirected connections between channels.

Initially, a three-layer graph convolution operation is used on each EEG segment to generate graph-level embedding, and the chebyshev polynomial is used as the convolution kernel of GCN to simplify the laplacian matrix computation. Next, we normalize the data along the channel dimension using channel-wise normalization to achieve a more uniform feature distribution. The data then sequentially pass through a 1×1 Conv1D layer, followed by three ResBlocks, and are finally processed by another 1×1 Conv1D layer to produce the encoded EEG embedding vector E′ ∈ RB×Ne×Te . The ResBlock consists of two convolution layers, each of them followed by a batch normalization (BN) layer and a PReLU activation function, with residual connections introduced after the second BN layer. Additionally, a MaxPooling layer is included to capture more representative temporal features. Now let R denote the input features and R′ the output of the first convolution module. The ResBlock can be represented as follows:

where Nin, Nout represent the number of channels.

3.2 Speech Encoder

While most time-domain TSE models typically use the convolution layers to extract speech embeddings, M3ANet adopts a more comprehensive approach by first capturing broad temporal features from the speech, followed by modeling long-range dependencies in each time window to extract deeper, more intricate speech characteristics. The proposed speech encoder, depicted in Figure 1, comprises multi-scale encoding and GMmodules [Shaker et al., 2024]. Firstly, four Conv1D layers with different kernel sizes are applied to the mixed speech waveforms, each followed by a ReLU activation function, respectively, mapping the number of channels to Ns to obtain richer speech information. Given mixture X ∈ RB×1×T , where B represents the batch size, and T is the length of the time-domain sequence, let Li represent the filter length at the i-th scale for i ∈ {1, 2, 3, 4}. The process is described as:

where Xi denotes the encoded features at the i-th scale, and the stride of all parallel convolutions is set to L1/2. By defining different Li, these layers extract features at different temporal resolutions, capturing both short- and long-term dependencies within the speech signals.

To further extract the deeper information contained in speech with linear complexity, N layers of GM modules are incorporated after multi-scale encoding for long sequence modeling. The GMmodule divides the 2D input features into four groups along the channel dimension. Each group is independently processed by a visual single selective scanning block (VSSSBlock), which integrates a mamba block operating in a specific direction-left-to-right, right-to-left, topto-bottom, or bottom-to-top-as illustrated in Figure 2. The full GM module also includes a channel affinity modulation (CAM) block and a feedforward layer.

M3ANet treats speech features at different granularities as channel-grouped 2D data. Specifically, Xi is first expanded by adding a new dimension at the end and concatenated to form ˆX ∈ RB×Ns×Ts×4 , which serves as the input to the GM block. Taking the first-layer GM block as an example, ˆX is then split along the last dimension to recover each Xi, which are individually processed by the corresponding VSSSBlocks. The process can be described as follows:

where CrossScani represents the i-th scanning direction. Then, X′ i is also concatenated along the last expanded dimension to obtain Xc. At this point, Xc contains information from four scales and feature representations from four scanning perspectives.

Subsequently, the CAM block is used to enhance crossscale feature interaction by reweighting the scale information. The structure of CAM can be represented as:

where δ and α denote the ReLU and sigmoid functions, respectively, and W1, W2 are the learnable weights of fully connected layers. Next, XCAM is passed through a normalization layer and a feedforward layer, and then a residual connection is applied with the initial input ˆX, producing the output of the current GM module. To achieve optimal feature extraction, the GM module is repeated N times. In summary, the speech encoder in this paper ultimately generates speech embeddings ˜X ∈ RB×Ns×Ts×4 .

3.3 Modal Alignment Module

Existing brain-assisted target speaker models focus on efficiently fusing EEG and speech information, yet they overlook the latency issue inherent in the data acquisition process, leading to misalignment of time frames. Inspired by the concept of "align before fusion" in [Li et al., 2021], this paper proposes a multi-modal alignment framework based on contrast learning between speech and EEG embeddings, utilizing InfoNCE loss to optimize the alignment process. As shown in the red arrows of Figure 1, the speech features ˜X are first reshaped to (B × Ns × Ts × 1) and then squeezed along the last dimension. Convolutional layers are applied to reconfigure the channels to match the EEG features E′ in dimension. To satisfy the computation requirements of InfoNCE, padding operation was applied to make E′ equal in temporal dimension with the speech sequence ˜X, and the new vector obtained is ˜E ∈ RB×Ne×Ts , and then the EEG and speech embeddings are unified to the shape (B, Ne × Ts).

The strategy for constructing sample pairs is also illustrated in Figure 1. For the input EEG sequence {e1, e2, ..., en} and speech sequence {x1, x2, ..., xn}, where n represents the batch size here, each EEG segment ei is treated as a query vector qi. The corresponding speech frame xi in the time series is considered the positive sample, and both of them form a positive sample pair (qi, x+i ). Meanwhile, other speech frames xj(j ̸= i) from the same mini-batch are treated as negative samples, resulting in n - 1 negative sample pairs (qi, x-j ). To reduce time-level modality inconsistencies and achieve contrastive learning, we use the InfoNCE loss function for optimization, aiming to maximize the similarity between positive samples and minimize the similarity between negative samples. For each sample pair (qi, x+i ), the InfoNCE function is defined as:

where xk represents speech segments in the current batch, sim(·, ·) represents the cosine similarity calculation between two samples, and τ is the temperature coefficient, set to 0.1.

Based on this contrastive learning strategy, the modal alignment module minimizes the distance between the EEG and the corresponding speech during training, achieving alignment between the two on a temporal frame level. This kind of cross-modal temporal alignment not only captures additional contextual information to enhance temporal consistency but also generates optimized multi-modal representations for downstream extraction tasks [Hu et al., 2023].

3.4 Speaker Extractor

The speaker extractor network, which serves as the core of the system, estimates a mask to extract the target speaker's speech. To leverage cross-modal information, speech and EEG embeddings are fused prior to mask estimation. The proposed extractor comprises a modality fusion module implemented using convolutional multi-layer cross attention (CMCA) [Zhang et al., 2023], and four layers of dual-path recurrent neural networks (DPRNN) [Luo et al., 2020].

CMCA is an efficient information fusion method. In this module, EEG and speech features are processed separately through information processing flows that include cross-attention, residual connection, and group normalization, enabling bidirectional feature coupling. CMCA employs a three-layer structure, where the output vector of the last layer is concatenated with the input vector of the first layer and processed with a 1D convolution to obtain the fused feature Y ∈ RB×Ne×Ts .

DPRNN, as a classical speech separation network, performs chunking processing on Y with each chunk having a length of L and a 50% overlap, applying RNN operations in both intra-chunks and inter-chunks. Compared to CNNs, RNNs are more suitable for processing sequential data, as their recurrent structure effectively captures temporal dependencies. After processing through four layers of DPRNN, the network outputs the target speech mask M ∈ RB×Ns×Ts , which is element-wise multiplied with the encoded speech Xen to obtain the masked speech ˆ S. Xen ∈ RB×Ne×Ts is obtained by concatenating the multi scale features Xi along the channel dimension, followed by a convolution with Ne output channels. The process is as follows:

Finally, the one-dimensional transposed convolution within the speech decoder reconstructs the estimated target speech ˆ s from ˆ S.

3.5 Objective Functions

The objective function used in this study consists of two components: the SI-SDR loss function and the InfoNCE loss function. The SI-SDR loss is commonly employed in speech processing tasks and is defined as follows:

where s represents the ground truth speech and ˆ s represents the predicted target speaker speech extracted from the M3ANet. We combine the LSI-SDR and the contrastive learning loss LInfoNCE to form the overall objective function, defined as follows:

where λ is the weighting factor and the value has been determined by experiment to be 3.

4 Experimental Setup

4.1 Datasets

Cocktail Party. For this dataset [Broderick et al., 2018], 33 subjects (aged 23-38 years) perform 30 trials, each lasting 60 seconds. During each trial, they listen to two classic works of fiction: one in the left ear and the other in the right ear. The subjects are divided into two groups, each instructed to focus on either the left or right ear. EEG data are recorded with 128 channels (plus two mastoid channels) at a rate of 512 Hz. For each subject, five trials are randomly selected as the test set, two for validation, and the remaining trials are used for training.

AVED. The audio-video EEG dataset (AVED) [ZHANG et al., 2024] includes EEG signals and corresponding speech from 20 normal-hearing students (14 males, 6 females) in a cocktail party scenario. Each completes 16 trials, and each trial lasts 152 seconds. EEG signals are recorded with 32 electrodes at a 1k Hz sampling rate. As in [Fan et al., 2024], the speaker-specific setting is applied to the AVED, analyzing only the trials where subjects focus on the left ear. For each subject, one trial is randomly selected as the test set, one as the validation set, and the remaining chosen trials are used for training.

MM-AAD. This dataset is amulti-modal AADdataset [Fan et al., 2025b], consisting of EEG recordings from 50 normalhearing subjects (34 males, 16 females). Each subject completes approximately 165 seconds of 20 trials. EEG data are captured from 32 channels at a 2048 Hz sampling rate. The attention direction used in AVED is also selected for MMAAD. Similarly, for each subject, two trials are randomly selected as the test set, one as the validation set, and the remaining trials are used for training.

4.2 Data Processing

Raw EEG signals often contain a lot of noise, requiring preprocessing before they can be used. For the Cocktail Party dataset, EEG signals are downsampled to 128 Hz, filtered between 0.1-45 Hz, and re-referenced to the mastoid average. Independent component analysis (ICA) is then applied to remove artifacts from blinks and muscle movements. For the AVED and MM-AAD datasets, we apply a notch filter to remove 50 Hz frequency interference, followed by band-pass filtering (1-50 Hz) and downsampling the EEG signals to 128 Hz. ICA is used to separate mixed signal components and remove interference. Finally, EEG signals are re-referenced using a whole-brain average reference.

For all three datasets, the audio stimuli are downsampled to 14.7 kHz. EEG and speech signals are cut into 2-second segments for training and validation sets, while testing segments are 20 seconds long. No overlap occurs between data in different sets or segments.

4.3 Training Details

In the training process, the model was trained for 60 epochs using the Adam optimizer with a weight decay of 1 × 10-3 and momentum of 0.9. The learning rate scheduler used in this study combines a linear warm-up phase (with a warm-up ratio of 5%) followed by a cosine decay phase. The learning rates are tailored for each dataset to suit them better. Specifically, the initial learning rate is set to 6e-4 for the Cocktail Party dataset, 2e-3 for AVED and 3e-3 for MM-AAD. The batch size was set to 8, and all experiments were accelerated by a single NVIDIA GeForce RTX 4090 GPU.

For our network, we set the speech feature dimension Ns to 128 and EEG feature dimension Ne to 64. The speech encoder uses filter lengths of 2.5ms, 5ms, 10ms, and 20ms (L1 to L4). In the DPRNN, the size L of each block is set to 250.

4.4 EvaluationMetrics

We focus on using five objective evaluation metrics to measure the quality of predicted speech. The source to distortion ratio (SDR) [Le Roux et al., 2019] and scale-invariant source to distortion ratio (SI-SDR) measure the quality of a signal, with SI-SDR being more robust to scaling differences. Shorttime objective intelligibility (STOI) and extended short-time objective intelligibility (ESTOI) [Taal et al., 2011] assess speech intelligibility, with ESTOI offering more accurate predictions by considering nonlinear hearing properties, with values ranging from 0 to 1. Perceptual evaluation of speech quality (PESQ) [Rix et al., 2001] compares the loudness spectra of desired and separated speech signals to evaluate speech quality. The PESQ scores range from [-0.5, 4.5]. These metrics are widely used in speech applications due to their objective nature and ability to accurately reflect signal quality, intelligibility, and perceptual performance. Higher values are better for all metrics. Among them, SDR and SI-SDR are measured in dB, while the other metrics are unitless.

5 Results

5.1 Comparison with Baseline Methods

We compare M3ANet with current brain-assisted target speaker extraction models, including BESD, UBESD, BASEN, NeuroHeed, and MSFNet. The experimental results on three datasets, presented as the median of all test segments from the subjects, are summarized in Table 1. As shown, our M3ANet outperforms all other models across all three datasets.

On the Cocktail Party dataset, M3ANet outperforms the current SOTA model, MSFNet, with significant improvements across all metrics. Specifically, M3ANet achieves improvements of 1.08 dB in SDR, 1.06 dB in SI-SDR, 1.23% in STOI, 1.36% in ESTOI, and 0.07 in PESQ. Moreover, with an SDR of 14.11 dB and an SI-SDR of 13.95 dB, the results demonstrate that our proposed model achieves competitive performance in speaker extraction.

On the AVED dataset, M3ANet also shows notable improvements over MSFNet. It achieves 1.3 dB in SDR, 1.24 dB in SI-SDR, 1.6% in STOI, 3.06% in ESTOI, and 0.14 in PESQ improvements. Notably, the significant improvements in SDR and SI-SDR highlight M3ANet's effectiveness in enhancing speech quality and reducing distortion. Similarly, on the MM-AAD dataset, M3ANet delivers impressive performance, surpassing the current best performance by 1.02 dB in SDR, 1 dB in SI-SDR, 1.49% in STOI, 3.6% in ESTOI, and 0.15 in PESQ, which further demonstrates the outstanding generalizability and efficiency of the proposed model. The notable ESTOI gains on both datasets indicate that M3ANet also effectively improves speech intelligibility.

Additionally, we present the combined boxplot and halfviolin plot for the extracted speech segments from BASEN, MSFNet, and M3ANet models on the Cocktail Party dataset, as shown in Figure 3. The experimental results of all models are obtained from our own reproductions. The SI-SDR results of BASEN are widely distributed, accompanied by high variance and many low or even negative values, suggesting unstable performance and weak generalization. MSFNet shows more stability, with a narrower distribution and fewer outliers. In contrast, M3ANet outperforms both BASEN and MSFNet, with a concentrated distribution around its high median value of 13.95 dB and almost no outliers, highlighting its superior and stable performance.

These results underscore the importance of maintaining temporal consistency between EEG and speech signals, while also highlighting the significance of fully extracting features from mixed speech. Furthermore, the results highlight the model's potential for real-world applications in brain-assisted speech extraction, offering a promising solution to the challenges posed by multi-modal data integration.

5.2 Ablation Study

To investigate the effectiveness of the GM layer and modal alignment module, we conducted ablation experiments on all three datasets, with the results shown in Table 2. We primarily present three key speech quality evaluation metrics: SDR, SISDR and STOI.

Effectiveness with GM Block. In the w/o GM model, we removed the GM module from the speech encoder but still used the four-scale Conv1D for initial feature extraction. To be consistent with the structure of the proposed model, we use a simple addition operation to fuse features of different granularities and input them into the speaker extractor. The experimental results demonstrate that removing the GMmodule led to a decrease in SDR, SI-SDR, and STOI across all three datasets, especially in the AVED and MM-AAD datasets, where SDR values decreased by 0.97 dB and 0.67 dB, SISDR values dropped by 0.93 dB and 0.7 dB, and STOI declined by 1.67% and 1.43%, respectively. These findings not only highlight the critical role of the GM module in maintaining model performance but also demonstrate its profound capability to extract latent speech features in depth. Thanks to the powerful long sequence modeling capability of the mamba module, the GM module can extract the speech embeddings more adequately and efficiently. Meanwhile, the CAM structure within the GM module acilitates information interaction across temporal scales, thereby enhancing both separation performance and speech extraction quality.

Effectiveness withAlignmentModule. To explore the impact of the modal alignment module on overall performance, we removed it between EEG and speech embeddings, while keeping the rest of the model architecture consistent with M3ANet. Compared to the final results, removing the alignment module led to a decrease in all performance metrics. On the three datasets, the SDR values dropped by 0.33 dB, 0.89 dB, and 0.59 dB, respectively, and the SI-SDR values decreased by 0.32 dB, 0.85 dB, and 0.58 dB. Moreover, the STOI scores show varying degrees of degradation across different datasets. These results highlight the critical role of the alignment module in enhancing model performance for target speaker extraction. The performance degradation without this module can be attributed to the lack of feature alignment between the EEG and speech modalities. By optimizing the InfoNCE loss, the module effectively reduces the distance between positive pairs and aligns features at corresponding time steps across modalities, thereby improving separation quality through more accurate temporal correspondence between EEG and speech signals.

5.3 Impact of GMLayers onModel Performance

Additionally, we investigated the impact of the number of GM layers on overall performance. As shown in Table 3, we explored the model results with 1 to 5 GM layers. The results indicate that the model with 2 layers of GM achieved the best performance across SDR, SI-SDR, and STOI. As the number of GM layers increased, the overall performance of the model deteriorated. This experiment demonstrates that the GM module is a simple and efficient feature extraction method, which can significantly improve model performance with only a small increase in model complexity while maintaining good training efficiency and generalization ability.

6 Conclusion

This paper proposes a novel time-domain, multi-scale and multi-modal alignment network for brain-assisted target speaker extraction. To align EEG and speech temporally, we introduce a modal alignment module based on contrastive learning, which reduces temporal discrepancies between the two embeddings. Additionally, to fully capture speech features, we employ multi-scale encoding to extract broad temporal features and use GM modules to extract more detailed and deeper speech information. Experimental results on three datasets demonstrate that the proposed model achieves stateof-the-art performance and strong generalization capabilities. In future work, we will explore advanced contrastive learning strategies and more efficient feature fusion techniques to further enhance the alignment and complementarity between the two modalities.

Acknowledgments

This work is supported by the STI 2030-Major Projects (No. 2021ZD0201500), the National Natural Science Foundation of China (NSFC) (No.62201002, 6247077204), Excellent Youth Foundation of Anhui Scientific Committee (No. 2408085Y034), Distinguished Youth Foundation of Anhui Scientific Committee (No. 2208085J05), Special Fund for Key Program of Science and Technology of Anhui Province (No. 202203a07020008), Cloud Ginger XR-1.