MHANet: Multi-scale Hybrid Attention Network for Auditory Attention Detection

1 Introduction

In a complex auditory scene where multiple people speak simultaneously, commonly referred to as the cocktail party problem [Haykin and Chen, 2005], humans are capable of directing their auditory attention on one particular speaker. However, individuals with hearing impairments often face significant challenges in identifying and focusing on the attended speaker. Previous neuroscience studies have demonstrated that there exists a connection between brain activity and auditory attention [Mesgarani and Chang, 2012]. The auditory attention detection (AAD) task aims to detect auditory attention from neural activities. Inspired by these findings, researchers have proposed various neurorecording modalities to address this issue, including electroencephalography (EEG) [Choi et al., 2013; O'sullivan et al., 2015], electrocorticography (ECoG) [Mesgarani and Chang, 2012], and magnetoencephalography (MEG) [Ding and Simon, 2012; Akram et al., 2016]. Among these, EEG-based methods stand out as premier solutions due to the non-invasive nature, wearability, and relative affordability of EEG. Therefore, in this paper, we concentrate on utilizing EEG signals for AAD.

∗Corresponding author

According to studies that the cortical responses to an attended speaker are encoded within EEG signals, which correlate with auditory stimulus [W¨ostmann et al., 2016; Bednar and Lalor, 2020], the linear methods reconstruct the stimulus from EEG signals to detect the correlation between the reconstructed stimulus and the attended speech envelopes [Biesmans et al., 2017; Katthi et al., 2020]. However, in most real-world scenarios, obtaining clean auditory stimuli is challenging, on which these methods heavily depend. Meantime, the cortical responses have a nonlinear relationship with the acoustic stimuli [Keshishian et al., 2020], and these linear methods have difficulty in capturing this complexity, thus leading to a longer decision window.

Therefore, some studies attempt to employ nonlinear neural networks to directly relate raw EEG signals to the attention detection decision [Ciccarelli et al., 2019; Monesi et al., 2020; Vandecappelle et al., 2021]. For instance, the long short term memory (LSTM) layer is used to capture temporal patterns of EEG signals [Monesi et al., 2020]. Then, a convolutional neural network (CNN) is proposed to extract the locus of auditory attention [Vandecappelle et al., 2021]. Recently, some research has demonstrated that the latent spatial distribution of different EEG channels improves AAD performance. Consequently, some studies project the extracted differential entropy (DE) values on two-dimensional (2D) topological maps [Jiang et al., 2022; Ni et al., 2024; Fan et al., 2025] or transform the original EEG channels into a 2D spatial topological map [Xu et al., 2024]. To capture the dynamic auditory attention activity sensitive to the temporal patterns [Zion Golumbic et al., 2013], the self-attention mechanism, which is proposed in [Vaswani et al., 2017] and widely used in numerous vision tasks [Yan et al., 2023], is also introduced into AAD to learn mutual temporal relationships or spatial distribution features within EEG signals [Su et al., 2022; Pahuja et al., 2023]. However, most methods usually have limited capabilities in comprehensively considering spatiotemporal representations of EEG signals.

To solve the problem, some studies adopt sequential approaches by applying the spatial attention mechanism followed by the temporal attention mechanism [Su et al., 2022]. Nevertheless, EEG signals inherently possess spatiotemporal characteristics, and the separate consideration of spatial and temporal dimensions hinders these models from capturing spatiotemporal dependencies in an integrated manner. Simultaneously, the fusion of representations across multiple temporal and spatiotemporal scales is critical for uncovering latent brain activities related to auditory attention. Unfortunately, most existing models fail to adequately consider multi-scale features of EEG signals, leading to suboptimal feature representation and sensitivity to noise. This oversight significantly restricts their performance of AAD.

To address these issues, this paper proposes a novel multiscale hybrid attention network for AAD, named MHANet, which effectively captures spatiotemporal features at multiple scales and uncovers the latent dependencies between spatial distribution features and temporal features from a global perspective. Specifically, our model comprises two modules: (1) Multi-scale Hybrid Attention Module. This module integrates a multi-scale temporal attention (MTA) block with the channel attention (CA) mechanism to combine expressive multiscale temporal information and spatial distribution features, enhancing the representation capacity of EEG signals. Additionally, we introduce a multi-scale global attention (MGA) block, which treats EEG signals as Euclidean feature maps to simultaneously capture spatial distribution features and temporal patterns at multiple scales. This allows our model to focus on key channels while capturing long-short range temporal contextual information globally. (2) Spatiotemporal Convolution Module. In this module, multi-scale spatiotemporal features are aggregated through spatial and temporal layers, followed by global average pooling, to consolidate comprehensive representations. We evaluate the AAD performance of our MHANet on three datasets: KUL, DTU, and AVED. The results demonstrate that MHANet achieves state-of-theart (SOTA) performance across three datasets. The major contributions of this paper are as follows:

• A novel AAD network, named MHANet, is proposed in this paper. This architecture combines multi-scale temporal features and spatial distribution features to capture long-short range spatiotemporal dependencies simultaneously.
• We introduce MTA to extract temporal information of EEG signals at multiple scales, enabling the construction of comprehensive temporal representations. Moreover, we propose MGA to capture multi-scale spatiotemporal dependencies globally, effectively identifying key channels and temporal patterns within EEG signals.
• The MHANet achieves SOTA decoding accuracy within an extremely short 0.1-second decision window on the KUL dataset, with an accuracy of 95.6%. It outperforms the best model by 6.4%. Moreover, our model is highly efficient with only 0.02M parameters, 3 times fewer than the leading model.

2 The ProposedMHANet

2.1 Overall Architecture

The previous AAD models typically apply attention mechanisms sequentially and neglect long-short range dependencies of spatiotemporal features of EEG signals at multiple scales.

To address these issues, we propose MHANet, as shown in Figure 1, which consists of MHA and STC. Specifically, MHA includes CA with a MTA and a MGA. Our MHANet extracts key spatial distribution features and long-short range temporal information at multiple scales, thus establishing a strong relationship between them.

Firstly, the processed data E is fed into MHA to extract comprehensive spatiotemporal features. Then, the output data is aggregated through STC. Finally, a fully connected (FC) layer is applied to generate the final prediction p.

where w and b are the weight and bias parameters of the FC layer. We apply the cross-entropy loss function to supervise the network's training.

In the following, we introduce the proposed MHA, MTA, MGA and STC.

2.2 Multi-scale Hybrid AttentionModule

As demonstrated in previous studies [Arvaneh et al., 2011], EEG signals capture the brain's neuronal electrical activity, which varies over time and reflects activity patterns and connectivity across different brain regions. This temporal and spatial variability makes it possible to analyze the brain's response patterns to auditory stimuli by deeply extracting spatiotemporal features from EEG signals.

However, previous studies solely apply sequential attention mechanisms and focus on isolated spatial or temporal information. To address these shortcomings, we combine a MTA with CA [Zamir et al., 2022] to comprehensively extract multi-scale spatiotemporal features from EEG signals. This architecture ensures that both temporal dynamics and spatial relationships are robustly captured, enhancing the decoding of auditory attention.

Firstly, we employ a convolutional layer to increase the channel resolution, enhancing the representation of spatial relationships of EEG signals. Following this, a depth-wise convolution is applied to extract temporal features E′ within each individual channel, focusing on the temporal dynamics within EEG signals. This can be formulated as follows:

where Conv(·) represents a 1 × 1 convolutional layer and DWConv(·) denotes a depth-wise convolutional layer. This two-step convolution ensures that the spatial and temporal features of EEG signals are effectively preserved and processed for the next steps.

Subsequently, we split the tensor E′ along the channel dimension to obtain the query Q, key K and value V components used for the self-attention mechanism [Vaswani et al., 2017].

where Split(·) denotes the operation of splitting E′ along the channel dimension.

Then, we introduce a MTA to comprehensively extract latent long-short range information and construct expressive temporal features V ′ with richer contextual details. The purpose of MTA is to further refine the value V representation by capturing diverse temporal dependencies at multiple scales.

The detailed design and functionality of MTA will be elaborated upon in the following subsection.

Then, by employing the effective self-attention operation across all channels, our method extracts the key spatial distribution features and multi-scale temporal features. Meantime, we apply a 1×1 convolutional layer to further improve the nonlinear expression ability and obtain more robust spatiotemporal features H for the next block. The process can be formalized as:

where Conv(·) represents a 1×1 convolutional layer without bias. Softmax(·) denotes the softmax function and t is a learnable scaling parameter.

Finally, the EEG data H is passed into MGA for further processing, constructing the refined spatiotemporal features F .

The MGA will be discussed in detail in the following subsection.

2.3 Multi-scale Temporal Attention Block

Previous studies have demonstrated that the use of temporal attention mechanisms can effectively capture the timevarying nature of EEG signals [Su et al., 2022; Ni et al., 2024]. However, they usually overlook the importance of multi-scale information and fail to fully consider long-short range dependencies inherent in temporal features. To address this limitation, we propose MTA to fully leverage time-series information by incorporating multiple temporal scales.

Firstly, a spatial filter is used to reduce the channel dimension to prevent an excessive number of trainable parameters. Next, we use a 1×1 convolutional layer to increase the channel again and split EEG signals into three parts X, Y,Z for operations of different scales. It can be formally described as

where SpatialConv(·) denotes a C × 1 convolutional layer. UpConv(·) denotes a 1 × 1 convolutional layer with 3 output channels. Split(·) performs the split operation along the channel dimension.

Next, X, Y and Z are passed through different multi-scale temporal layers to compute multiple attention values α, β, γ.

where Convi(·) refers to a convolutional layer with a kernel size of i ∈ {2, 4, 6}. LN(·) denotes the layer normalization technique [Ba et al., 2016]. ELU(·) is the exponential linear unit (ELU) activation function [Clevert et al., 2016] and AAP(·) denotes an adaptive average pooling layer.

Then, we multiply each attention value with X, Y, Z separately and add them up to obtain the final fused temporal features V ′ with multi-scale temporal patterns. Finally, it is put into a 1 × 1 convolutional layer to recover the number of channels and reshaped for further processing in the next stage.

where ⊙ denotes the element-wise multiplication and Conv(·) represents a 1×1 convolutional layer with C output channels. α, β, γ are the learnable attention weight values.

2.4 Multi-scale Global Attention Block

Most previous studies usually overlook the dynamic relationship between spatial distribution features and time-series data of EEG signals. Enlightened by [Wang et al., 2024], we propose MGA to further extract latent spatiotemporal dependencies within EEG signals from a global perspective.

Firstly, a 1×1 convolutional layer is applied to increase the channel dimension of the input data, producing H′ . This is followed by cloning the data to facilitate the final multi-scale fusion. Then, we split the data into three distinct parts P, S, R for further processing.

where UpConv(·) denotes a 1 × 1 convolutional layer that outputs 3 channels. Split(·) refers to a split operation along the channel dimension.

Subsequently, we separately compute the spatiotemporal attention maps. To capture multi-level spatiotemporal dependencies, we utilize dilated convolutional layers with varying kernel sizes and dilation rates. Each attention map is multiplied with its corresponding original features to obtain local refined attention maps δ, φ, µ.

where DConvi(·) represents i-th dilated convolutional layer with varying kernel size (3×3, 5×5 and 7×7). The dilation rate of each layer is determined by the length of the decision window.

Next, we concatenate the resulting feature maps to obtain a multi-scale global attention map. It is then multiplied with H′ to comprehensively capture spatiotemporal dependencies. Finally, a 1 × 1 convolutional layer is employed to restore its original shape. Simultaneously, a residual connection [He et al., 2016] is introduced to enhance the performance and robustness of the block. It can be formulated as follows:

where [·] denotes the concatenate operation, which combines multiple attention maps along the channel dimension. DwConv(·) represents a 1 × 1 convolutional layer with 1 output channel.

2.5 Spatiotemporal ConvolutionModule

Through the comprehensive extraction of spatiotemporal features performed by the previous module, we obtain expressive information from EEG signals. To further improve the performance of AAD, STC is introduced at the end of our model to aggregate the extracted features. In STC, we apply a temporal layer and a spatial layer, followed by an adaptive average pooling layer to further process and downsample the data.

where TemporalConv(·) denotes a 2D convolutional layer with a 1 × 2 kernel size, and SpatialConv(·) represents a 2D convolutional layer with a C ×1 kernel size across all the channels. BN(·) is the batch normalization layer [Ioffe and Szegedy, 2015] and AdaptiveAvgPool(·) denotes an adaptive average pooling layer.

3 Experiments

3.1 Datasets

In this paper, we conduct extensive experiments on three publicly available datasets, namely KUL [Das et al., 2016; Das et al., 2019], DTU [Fuglsang et al., 2017; Fuglsang et al., 2018] and AVED [Fan et al., 2024b], as shown in Table 1. The KUL and DTU datasets are among the most commonly used for AAD with the audio-only scene. The AVED dataset, provided for the ISCSLP Chinese AAD Challenge 2024, includes both audio-only and audio-visual scenes.

1. KUL Dataset: In this dataset, 64-channel EEG data were collected from 16 normal-hearing subjects (8 males and 8 females) at a sampling rate of 8192 Hz. The stimuli comprised four Dutch short stories narrated by three male speakers from 90° to the left or right. Each subject completed 8 trials, with each trial lasting 6 minutes.
2. DTU Dataset: In this dataset, 64-channel EEG data were collected from 20 normal-hearing subjects at a sampling rate of 512 Hz. The auditory stimuli consisted of Danish audiobooks narrated by a male and a female speaker. Each subject completed 60 trials, with each trial lasting 50 seconds.
3. AVED Dataset: In this dataset, 32-channel EEG data were collected from 20 normal-hearing subjects (14 males and 6 females) at a sampling rate of 1 kHz. The subjects were divided into two groups of 10. One group underwent audio-only experiments, and the other group underwent audio-video ones. All auditory stimuli were derived from 16 stories selected from a collection of Chinese short stories narrated by a male and a female speaker. Each subject completed 16 trials, with each trial lasting 152 seconds.

3.2 Data Processing

To ensure a fair comparison of the performance of our MHANet, specific preprocessing steps are applied to each dataset. For the KUL dataset, EEG data are initially rereferenced to the average response of mastoid electrodes, followed by bandpass filtering between 0.1 Hz and 50 Hz. The data are then down-sampled to 128 Hz. For the DTU dataset, EEG data are processed to remove 50 Hz linear noise and its harmonics. Eye artefacts are removed through joint decorrelation, and then the data are re-referenced to the average response of mastoid electrodes. Finally, the data are downsampled to 64 Hz. For the AVED dataset, a notch filter is applied first to eliminate powerline interference at 50 Hz. Next, a finite impulse response (FIR) filter is used for high-pass and low-pass filtering to remove the noise. Subsequently, the EEG data are downsampled to 128 Hz, followed by independent component analysis (ICA) for further noise removal. Finally, a re-referencing process is performed across all EEG channels to ensure consistency and comparability.

3.3 Implementation Details

In previous research on AAD, classification accuracy has been utilized as the standard metric for evaluating model performance. Following this convention, we assess our proposed MHANet using the KUL, DTU, and AVED datasets. To illustrate implementation details, including training settings and network configuration, we use the KUL dataset as an example, with a 1-second decision window.

The dataset is initially divided into training, validation, and test sets in a ratio of 8:1:1. For each subject in the KUL dataset, we allocate 4,600 decision windows for training, 576 for validation, and 576 for testing. The training process uses a batch size of 32, with a maximum of 100 epochs. An early stopping strategy is employed, halting training if there is no decrease in the validation set's loss function value for 15 consecutive epochs. The model is trained using the AdamW optimizer with a learning rate of 5e-3 and weight decay of 3e-4.

Initially, we employ the common spatial patterns (CSP) algorithm [Ramoser et al., 2000; Blankertz et al., 2007] to extract raw features from the EEG signals and rearrange them into E. Then, through MHA, we get the refined spatiotemporal features F. It is then sent to STC. After convolutional layers and global average pooling, we obtain O. Finally, the final binary AAD classification result p is achieved through a fully connected layer (input: 5, output: 2).

4 Results on AAD

4.1 Performance of MHANet

To comprehensively evaluate our proposed MHANet, we assess its AAD performance under different decision windows and compare our model with other outstanding methods, as shown in Table 2. For open-access models, we replicate their architectures, while results from other studies are cited accordingly.

Our MHANet demonstrates significant improvement and superior performance on KUL, DTU, and AVED datasets.

On the DTU dataset, the accuracies are 75.5% (SD: 5.68%), 82.2% (SD: 8.13%), 83.0% (SD: 7.14%) under the 0.1second, 1-second and 2-second decision windows, respectively. Similar accuracy trends are observed on the KUL and AVED datasets at different lengths of the decision window.

Overall, our model's performance decreases as the decision window length shortens. However, we observe that on the KUL dataset, the performance under 0.1-second is nearly as good as under 1-second. This could be because, under a 0.1-second decision window, the number of windows significantly increases, providing the model with more training samples, which aids in its learning. At the same time, the performance on the DTU dataset is generally lower compared to the KUL dataset. This could be attributed to differences in stimulus source locations, the number of speakers, their genders, and variations in data processing.

4.2 Ablation Experiment

To thoroughly analyze our model, we conduct extensive ablation experiments by removing the CA mechanism, MTA, MGA, both MTA and CA, and STC. All experiments are conducted under the same conditions as the previous settings. The results of these ablation experiments are presented in Table 3.

Experimental results show that on the DTU dataset, removing CA from MHANet leads to a decrease in average accuracy by 8.6% under the 1-second decision window. Removing the MTA also causes a similar accuracy drop: 3.7% for the 1second decision window. When the MGA is removed, the accuracy decreases by 0.5% for the 1-second decision window. After removing both MTA and CA, the average accuracy decreases by 11.2% under the 1-second decision window. The removal of STC results in accuracy drops of 2.5% for the 1-second decision window. Similar trends of decreased accuracy are observed on the KUL and AVED datasets and at different lengths of the decision window after removing the aforementioned modules or blocks.

Overall, the complete MHANet demonstrates the best performance compared to versions with individual modules or blocks removed. On the DTU dataset with a 1-second decision window, MHANet outperforms the version without CA by 8.6%, highlighting the importance of focusing on spatial distribution features within EEG signals. It also achieves a 3.7% improvement over the version without MTA, indicating the effectiveness of capturing multi-scale temporal patterns. Additionally, MHANet shows a 0.5% improvement compared to the version without MGA, demonstrating the value of considering spatiotemporal dependencies from a global perspective. Finally, it outperforms the version without STC by 2.5%, emphasizing the necessity of integrating and refining spatiotemporal features of EEG signals from different dimensions.

5 Analysis and Discussion

5.1 Comparison with the SOTAModels

We compare the performance of our MHANet with other advanced AAD models, as presented in Table 2. The results demonstrate that MHANet achieves significant improvements over the current SOTAmethods.

On the DTU dataset, our MHANet achieves relative improvements of 13.0%, 8.6%, 3.0%, 1.5%, and 0.9% under the 0.1-second decision window compared to the SSFCNN, MBSSFCC, EEG-Graph Net, DBPNet and DARNet models, respectively. For the 1-second decision window, MHANet achieves relative improvements of 12.4%, 6.6%, 2.6%, 3.5%, 2.4%, and 2.1% compared to the SSF-CNN, MBSSFCC, DGSD, EEG-Graph Net, DBPNet, and DARNet models. Similarly, the relative improvements under the 2second decision window are 9.7%, 4.3%, 0.6%, 3.6%, 2.8%, and 1.8%, respectively.

On both the KUL and AVED datasets, MHANet achieves comparable improvements over other models. Notably, on the KUL dataset, MHANet achieves a 6.4% relative improvement under the 0.1-second decision window compared to the current SOTA method. At the same time, on the AVED dataset, the accuracies of other models are generally lower under the 0.1-second decision window, while our model shows a significant improvement. These all suggest that our model has an advantage in tasks that require adaptation to extremely short time windows.

Overall, the outstanding performance of MHANet across different datasets and decision windows highlights its strong real-world practicality and applicability, making it a promising solution for realistic hearing aids.

5.2 Ablation Analysis

As shown in Table 3, the results demonstrate that removing the CA, MTA, MGA, MTA and CA or STC leads to performance degradation. This highlights the effectiveness of each block and module in contributing to the advanced performance of our MHANet.

Effectiveness of CA

The distribution of EEG channels reflects the spatial relationships between them. Therefore, it is crucial to exploit the spatial coherence across different EEG channels [Geirnaert et al., 2021]. The incorporation of the channel attention mechanism enables our model to focus on key spatial information within EEG signals, thus capturing the activities of different brain regions, which is essential for effectively analyzing EEG signals.

Effectiveness of MTA

Our MTA effectively captures multi-scale temporal information and generates more expressive and robust temporal features through convolutional attention at different scales. Therefore, it further improves the model's ability to understand latent temporal context at different levels and identify long-short range temporal patterns within EEG signals.

Effectiveness of MGA

Our MGA fully leverages the spatiotemporal feature maps to extract more valuable spatial distribution features and temporal patterns at multiple scales. By treating the EEG data as Euclidean data, it can capture spatiotemporal dependencies of EEG signals from a global perspective. The results show that our MGA further enhances performance to some extent, demonstrating the effectiveness of simultaneous spatiotemporal consideration.

Effectiveness of STC

The STC effectively aggregates spatiotemporal features along the temporal and channel dimensions through convolutional operations, respectively, resulting in more robust features with reduced noise and outliers. Meanwhile, the integration enhances our model's ability to understand brain activity, further improving its overall robustness and generalization ability.

5.3 Comparison of Computational Cost

We compare the training parameter counts of our MHANet, SSF-CNN [Cai et al., 2021], MBSSFCC [Jiang et al., 2022], DBPNet [Ni et al., 2024] and DARNet [Yan et al., 2024]. The results are shown in Table 4. The parameter count of MHANet is 209.5 times lower than that of SSF-CNN, 4194.5 times lower than MBSSFCC, 44.5 times lower than DBPNet, and 3 times lower than that of DARNet. MHANet showcases excellent parameter efficiency by achieving competitive performance with a significantly reduced number of parameters. This suggests our model's strong real-world applicability and practicality, making it well-suited for use in real-world lowresource hearing aids.

6 Conclusion

This paper proposes MHANet, a novel multi-scale hybrid attention network, to address the oversight of multi-scale spatiotemporal dependencies in AAD. We introduce MHA to capture long-short range spatiotemporal dependencies of EEG signals. The MHA combines CA with MTA to construct comprehensive and robust EEG features. Then, the MGA extracts the dynamic relationship between key spatial distribution features and temporal patterns. Subsequently, STC aggregates expressive EEG signals, improving our model's overall robustness and generalization ability. We evaluate the performance of the proposed MHANet on three datasets: KUL, DTU, and AVED, which demonstrate that MHANet achieves SOTA performance with fewer trainable parameters across all three datasets. This highlights its strong real-world practicality and applicability for realistic hearing aids. For future research, we plan to incorporate time-frequency analysis to further explore the spatiotemporal dependencies within EEG signals and enhance the performance of AAD.

Acknowledgments

This work is supported by the STI 2030-Major Projects (No. 2021ZD0201500), the National Natural Science Foundation of China (NSFC) (No.62201002, 6247077204), Excellent Youth Foundation of Anhui Scientific Committee (No. 2408085Y034), Distinguished Youth Foundation of Anhui Scientific Committee (No. 2208085J05), Special Fund for Key Program of Science and Technology of Anhui Province (No. 202203a07020008), Cloud Ginger XR-1.

Contribution Statement

Lu Li and Cunhang Fan contributed equally to this work.