Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction

1 Introduction

Speech serves as the primary means of exchanging and communicating information among humans. However, in realworld settings, speech signals often encounter contamination from various interfering factors, such as noise and irrelevant speakers. This scenario is commonly known as the "cocktail party problem" [Bronkhorst, 2000]. During actual conversations, perceivers possess the ability to selectively focus on

auditory attention. A long-term objective of speech processing research is to develop machines capable of emulating human auditory capabilities. These machines should be able to selectively extract the speech of a target speaker by utilizing auxiliary cues, such as reference speech [Xu et al., 2020;

Ge et al., 2020; Mu et al., 2023], static facial images [Gao and Grauman, 2021], and lip movements [Wu et al., 2019; Gao and Grauman, 2021; Pan et al., 2021].

Neuroscientific research has revealed that human perception of speech is inherently multi-modal, with lip movements being observed to enhance auditory perception during conversations [Li et al., 2018; Stenzel et al., 2019]. Visual cues have proven to be robust auxiliary references for speaker extraction algorithms at the frame level, yielding remarkable outcomes in separating target speakers, particularly in the presence of highly overlapping speech [Ephrat et al., 2018; Ochiai et al., 2019]. However, the utilization of static facial images has demonstrated limited improvement in the performance of target speech extraction (TSE) [Gao and Grauman, 2021] and may raise privacy concerns. Hence, this paper focuses on leveraging lip movements as the primary cue to guide TSE.

In previous studies on audio-visual target speech extraction (AV-TSE), it has been observed that visual information is often underutilized, resulting in performance comparable to ignoring visual cues. This situation arises due to the audiofocused nature of AV-TSE, where the richness of visual feature information is less substantial compared to audio feature information. Consequently, the importance of visual cues has been insufficiently considered, creating an imbalance between audio and visual modalities. To address this issue, we draw inspiration from the concept of the speech chain [Denes and Pinson, 1993] in speech science and propose the audio-visual speech separation chain (AVSepChain). The speech chain depicts the flow of information between speakers and perceivers in speech communication, encompassing two stages: speech production and speech perception [Deller Jr, 1993]. The information flow within the speech chain includes auditory feedback from the speaker's mouth to the ears, the perceiver receiving acoustic information from the speaker, and the perceiver receiving visual information regarding the speaker's lip movements, as shown in Figure 1.

In our approach, we employ a two-stage method to replicate the inverse process of the two stages of the speech chain. In the first stage, we utilize the audio-visual target speech extraction network to simulate speech perception. This network extracts the target speaker's speech from the mixed audio, leveraging the target speaker's lip movements as a guiding condition. In the second stage, we employ the lip-to-speech synthesis network to simulate speech production. This network generates speech based on lip movements, utilizing the speech extracted in the first stage as a condition. Unlike previous AV-TSE approaches that solely relied on the video modality for conditional guidance, our approach alleviates the modality imbalance issue by incorporating both audio and video modalities as mutual conditions.

To ensure the consistency between the speech content conveyed through the voice and lip movements of the target speaker, as depicted in the two channels of speech perception shown in Figure 1, we propose a contrastive semantic matching loss. Specifically, we employ audio and audio-visual selfsupervised pre-trained models to extract frame-level quantized representations of speech content and lip movement content separately. These representations serve as pseudophonemes and pseudo-visemes, respectively, enabling us to align cross-modal semantics. A viseme represents a basic unit of visual speech, corresponding to a group of phonemes present in acoustic speech [Massaro and Simpson, 2014; Bear and Harvey, 2017]. By capitalizing on the consistency between pseudo-visemes and pseudo-phonemes, we establish that the generated speech of the target speaker possesses the same semantic information as its lip movements.

In general, this paper presents three main contributions. (i) We introduce the concept of speech chain into AV-TSE, which, to the best of our knowledge, has not been explored previously. By leveraging the audio and video modalities as conditional information for each other, we simulate the speech perception and production processes in the speech chain, thereby alleviating the modality imbalance issue in AVTSE. (ii) We propose a contrastive semantic matching loss to regulate the generation of more precise audio signals in the lip-to-speech synthesis network. This loss conducts contrastive matching training on the content information of the audio and video modalities, which is extracted using pretrained models. (iii) We perform comprehensive experiments to validate the effectiveness of our approach. The results demonstrate that our proposed AVSepChain significantly enhances the state-of-the-art performance of AV-TSE.

2 RelatedWork

2.1 Audio-Visual Target Speech Extraction

Recent neuroscience research has demonstrated that the human brain effectively addresses the "cocktail party problem"

by incorporating visual cues [Li et al., 2018; Stenzel et al., 2019]. Drawing on this understanding, researchers have integrated visual information into the paradigm of speech separation to enhance performance in noisy and challenging environments. Prior studies by Lin et al. [2023] and Lee et al. [2023] employed cross-modal attention for feature fusion, while Zhou et al. [2022] designed an adaptive multi-modal fusion framework. Additionally, Li et al. [2022] proposed a multi-scale fusion framework inspired by cortical-thalamic circuits. However, these methods typically prioritize audio as the dominant modality, resulting in an imbalanced treatment of audio and visual modalities.

2.2 Lip-to-Speech Synthesis

The tongue and lips assume a crucial role in controlling the shape of the vocal tract and modifying its resonance characteristics to convey distinct phonemes. Perceivers can infer a speaker's speech content by observing their lip movements, providing opportunities for integrating video into speech synthesis. Lip-to-speech synthesis (LTS) aims to convert visual information derived from the analysis of lip movements into corresponding speech signals. Similar to text-to-speech synthesis (TTS) approaches [Mu et al., 2021], most LTS methods follow a two-stage procedure [Prajwal et al., 2020; Kim et al., 2021; de Mira et al., 2022; Kim et al., 2023], involving the generation of mel-spectrograms followed by their transformation into audio signals using vocoders. In this paper, we propose predicting the residual signal of the preliminary separated speech to alleviate the difficulty of the LTS task. Moreover, LTS inherently poses an underdetermined problem, as a single lip movement video can correspond to multiple speech possibilities. To impose constraints, previous research commonly introduced reference speech from the same speaker and employed a speaker encoder to obtain speaker embeddings as guiding information [de Mira et al., 2022; Choi et al., 2023b; Kim et al., 2023]. However, in this study, we utilize the target speaker's speech extracted during the speech perception stage as the registration, eliminating the need for additional speaker reference speech. Furthermore, we implicitly constrain the synthesis process by predicting the residual signal.

2.3 Modality Imbalance Problem

When dealing with audio-visual data, which involves multiple modalities, a common challenge is modality imbalance.

Modality imbalance occurs when one modality dominates over others in a specific task or scenario. To alleviate this issue, a common approach is to adjust the training strategy of the model to balance the learning process for each modality. For instance, Fan et al. [2023] introduced modalityspecific loss functions and regularization terms to enhance the learning of non-dominant modalities. Similarly, Fu et al. [2023] and Wu et al. [2022] balanced the learning process of different modalities by dynamically adjusting gradients and learning rates, enabling the model to allocate more attention to non-dominant modalities. Another approach, as proposed by Du et al. [2021], involves leveraging pre-trained single-modal models to tackle the modality imbalance problem. They extracted features from the single-modal model and distilled them into the multi-modal model, thereby improving the learning of non-dominant modalities. In this study, we alleviate the issue of modality imbalance by shifting the dominant role between two modalities at different stages.

3 Methodology

3.1 Overview

When presented with a mixed speech signal x and a lip movement video v of the target speaker, the proposed AVSepChain framework aims to extract clean speech s corresponding to the target speaker from the speech of interfering speakers and environmental noise by leveraging the visual information. The framework consists of two stages: speech perception and speech production, as illustrated in Figure 2. In the speech perception stage, we employ AV-HuBERT [Shi et al., 2022a], a pre-trained audio-visual model, to extract the visual hidden unit fv from the lip movement video v. Subsequently, the audio-visual speech separator (AV-Separator) is utilized to preliminarily separate the target speech spre from the mixed speech x, using fv as a condition.

In the speech production stage, the mel-spectrogram Spre of the preliminary separated target speech spre obtained in the first stage serves as a conditional input. The audio-visual speech synthesizer (AV-Synthesizer) predicts the residual signal sres of the target speech based on the visual hidden unit fv. The final target speech sfin is obtained by adding spre and sres. Moreover, we utilize HuBERT [Hsu et al., 2021], a pre-trained audio model, to extract the audio hidden unit fa from sfin. This unit is matched with fv to ensure semantic feature alignment, thus guaranteeing that the generated target speech captures the same semantic information as the lip movement. Each section will be further elaborated in the subsequent sections.

3.2 Speech Perception

AV-HuBERT. In the speech perception stage, AV-HuBERT functions as the encoder for lip movement videos. AVHuBERT is a self-supervised model designed for audio-visual learning, which predicts clustering assignments in masked regions based on speech features and lip movement sequences. Previous studies have demonstrated the efficacy of this approach in capturing meaningful semantic information conveyed by lip movements. Consequently, it offers significant benefits for downstream tasks such as audio-visual speech recognition [Shi et al., 2022a; Shi et al., 2022b].

In our approach, for the lip movement video of the target speaker, represented as v ∈ RH×W×Tv , where H, W, and Tv denote the height, width, and number of image frames respectively, we input it into the pre-trained AV-HuBERT. This results in a condensed frame-level visual representation, denoted as fv ∈ RNfv ×Tfv , where Nfv signifies the number of channels of fv, and Tfv indicates its length.

AV-Separator. In our approach, the AV-Separator utilizes the widely adopted AV-Sepformer [Lin et al., 2023] as its backbone. Sepformer [Subakan et al., 2021] is a timedomain monaural speech separation model that leverages the dual-path Transformer architecture. AV-Sepformer extends Sepformer by incorporating the cross-attention mechanism [Vaswani et al., 2017] to integrate both audio and visual modalities. In this study, similar to AV-Sepformer, we employ cross-modal attention to fuse audio and video features. In this case, considering that the visual information can be regarded as cues for separating the target speech, we adopt an approach where the audio modality is treated as the dominant modality while the visual modality acts as the conditional modality.

Specifically, we initially encode the mixed audio x ∈ RTa through a 1-D convolutional encoder, resulting in an audio feature representation X ∈ RNa×TX . Here, Na represents the number of convolutional kernels, and TX denotes the length of the audio feature. Subsequently, we divide the 2D audio feature X into chunks of length KX, with a stride size of KX 2 . These chunks are then concatenated to form a 3D audio feature Xc ∈ RNa×KX×SX , where SX refers to the number of chunks. To match the dimensions of the video feature fv and the audio feature Xc, we replicate fv KX times, adjust the length of fv to SX using interpolation, and project the channel dimension of fv to Na, resulting in the adjusted video feature ˜ fv ∈ RNa×KX×SX .

Next, we utilize the visual feature ˜ fv as the query and the audio feature Xc as the key and value for cross-modal attention fusion. This fusion yields an audio-dominant multimodal feature denoted as Hav ∈ RNa×KX×SX , which can be computed as:

Here, W Q v , W K a , and W V a ∈ RNa×Na are learnable parameters. Subsequently, the AV-Separator extracts the speech of the target speaker spre ∈ RTa from the audio-dominant crossmodal feature Hav. To accomplish this, we utilize the scaleinvariant signal-to-noise ratio (SI-SNR) [Roux et al., 2019] between spre and the true signal as the loss function Lper during the speech perception stage. This loss function aims to separate the speech of the target speaker preliminarily.

3.3 Speech Production

AV-Synthesizer. During the speech production stage, it is crucial to handle the intrinsic one-to-many relationship between visemes and phonemes, as well as accommodate speech-related variations such as voice, accents, and prosody.

To tackle these challenges, we leverage the target speech that was extracted in the previous stage as a registration to facilitate the synthesis of speech from lip movements. In contrast to the previous stage, we assign vision as the dominant modality and audio as the conditional modality in this stage, aiming to mitigate modality imbalance issues.

Specifically, we utilize the mel-spectrogram Spre ∈ RNmel×Tmel of the output spre from the first stage, along with the lip movement feature sequence fv extracted by AVHuBERT, as the audio-visual input to the AV-Synthesizer. Here, Nmel denotes the channel number of Spre, and Tmel represents its length. Since the video stream and the audio stream are inherently synchronized, the visual features and the mel-spectrogram are naturally aligned. Consequently, we upsample the video frames to match the sampling rate of the mel-spectrogram based on the ratio between the sampling rates of the mel-spectrogram and the video frames. Additionally, we project the video feature fv and audio feature Spre to the same channel dimension Npro, resulting in ˆ fv and ˆ Spre ∈ RNpro×Tmel .

Next, we employ the audio feature ˆ Spre as the query and the visual feature ˆ fv as the key and value for cross-modal attention fusion. This fusion yields a visual-dominant multimodal feature, denoted as Hva ∈ RNpro×Tmel , which can be expressed as:

Here, W Q a , W K v , andW V v ∈ RNpro×Npro represent the learnable parameters. Subsequently, the AV-Synthesizer utilizes the visual-dominant multi-modal feature Hva to predict the residual signal sres ∈ RTa of the target speech spre, which was extracted in the first stage. The final generated speech of the target speaker sfin ∈ RTa is obtained by adding the residual signal sres to the preliminary result spre, as shown in the equation below:

During the speech production stage, our approach involves predicting the residual signal rather than the complete speech waveform. This strategy aims to provide a preliminary candidate for the AV-Synthesizer, thus alleviating the challenge of modeling the entire speech signal. By focusing on the residuals, the AV-Synthesizer can effectively capture the remaining information necessary for synthesis and refine the preliminary results. This strategy also incorporates an implicit constraint into the synthesis process. To ensure that the generated speech accurately corresponds to lip movements, we also employ the SI-SNR between sfin and the true signal as the loss function Lsyn during the speech production stage.

Semantic matching. To ensure that the speech generated by the AV-Synthesizer conveys the same semantic information as the corresponding lip movements, we propose a contrastive semantic matching loss. Initially, we employ a pretrained HuBERT to extract the frame-level speech representation fa ∈ RNfa ×Tfa from the generated speech sfin. Here, Nfa and Tfa denote the channel number and length of fa, respectively. It is worth mentioning that this representation has demonstrated effectiveness in downstream tasks like automatic speech recognition (ASR) [Hsu et al., 2021; Shi et al., 2022c]. Therefore, fa can be considered as a pseudo-phoneme representation. Similarly, the representation fv extracted by AV-HuBERT can be regarded as a pseudo-viseme representation. To align the cross-modal semantic representations, we introduce a contrastive semantic matching loss, denoted as Lmat, which is defined as follows: Lmat = max{d(n(fv), n(fa))-d(n(fv), n( ¯ fa))+m, 0} (4)

Here, d(·, ·) represents the L2 distance, n denotes L2 normalization, ¯ fa is the embedding obtained by applying HuBERT to the interfering audio x - sfin, and m represents the margin. To compute Lmat, we index each frame of the pseudo-phoneme representations to their corresponding pseudo-viseme representations based on the ratio of the sampling rates of the audio and video streams. This ensures that the lengths of fa, ¯ fa, and fv are matched. The gradient of Lmat is then back-propagated to the AV-Synthesizer to update its parameters.

3.4 Training

The overall training objective of our method can be defined as follows:

Here, λ is a weighting factor. The loss function SI-SNR, which is used to calculate Lper and Lsyn, is defined as:

In the above equations, u and ˆu represent the ground truth and predicted speech of the target speaker, respectively. ∥u∥2 = ⟨u, u⟩ denotes the signal power. During training, we optimize the parameters of the AV-Separator and AV-Synthesizer while keeping the parameters of AV-HuBERT and HuBERT fixed.

4 Experiments

4.1 Datasets

We conducted experiments on two audio-visual datasets, namely LRS2-2Mix and VoxCeleb2-2Mix [Li et al., 2022], derived from the LRS2 [Afouras et al., 2022] and VoxCeleb2 [Chung et al., 2018] datasets, respectively. Both LRS2-2Mix and VoxCeleb2-2Mix consist of mixtures of two speakers. We randomly selected two distinct speakers and mixed their speech with a signal-to-noise ratio ranging from -5 dB to 5 dB. These datasets contain multilingual speech corrupted by noise. The speaker identities in the training and test sets do not overlap. Each speech segment has a duration of 2 seconds and a sampling rate of 16 kHz. The video frames are synchronized with the speech at a frame rate of 25 FPS. In our experiments, we cropped the lip region and resized the cropped frames to 88 × 88 pixels, resulting in a height and width of H = W = 88.

4.2 Implementation Details

For both AV-HuBERT and HuBERT, we employ a 12-layer BASE pre-trained model with a feature dimension of 768, denoted as Nfv = Nfa = 768, and extract the features from the last layer as embeddings. The AV-Separator is built on the same hyperparameters as the AV-Sepformer [Lin et al., 2023], comprising two repetitions of 8 Intra-Transformers, 7 InterTransformers, and 1 Cross-Modal Transformer. The values for Na and KX are set to 256 and 160, respectively. We employ a logarithmic mel-spectrogram with 80 mel bands, a filter length of 1024, a hop size of 10 ms, a window length of 40 ms, and a Hann window to capture the spectro-temporal features Spre of the output audio spre from the AV-Separator. In other words, Nmel is 80. The intermediate feature dimension Npro is set to 256. The AV-Synthesizer consists of a cross-modal attention layer, followed by three 1-D convolution layers. The hidden dimensions for the convolution layers are 256, 128, and 160, respectively, with a kernel size of 7. Overall, our model encompasses a total of 33.1M trainable parameters, with 31.3M parameters in the AV-Separator and

1.8M parameters in the AV-Synthesizer. When calculating the contrastive semantic matching loss, we set the margin m to 0.5 and λ to 1. The model is trained using Adam optimization, starting with an initial learning rate of 1.5 × 10-4 . If the loss does not decrease on the validation set for three consecutive epochs, the learning rate is halved. The training process is terminated if there is no decrease in the loss for five consecutive epochs.

4.3 EvaluationMetrics

We evaluate the quality of separated speech using scaleinvariant signal-to-noise ratio improvement (SI-SNRi) [Roux et al., 2019] and signal-to-noise ratio improvement (SDRi) [Vincent et al., 2006]. The overall perceptual quality is measured using the perceptual evaluation of speech quality (PESQ) [Rix et al., 2001]. Moreover, to assess whether the proposed method, as a speech enhancement front-end, improves the accuracy of downstream tasks such as ASR, we utilize the publicly available Google speech-to-text API to obtain recognition results. Our primary focus lies on measuring the word error rate (WER), where a lower error rate indicates better preservation and restoration of the content information in the speech.

4.4 Comparison with the State-of-the-Art

We comprehensively compared AVSepChain and existing AV-TSE methods on the LRS2-2Mix and VoxCeleb2-2Mix datasets. The results, presented in Table 1, demonstrate that AVSepChain achieves state-of-the-art performance on both datasets. Notably, our proposed AVSepChain significantly outperforms other methods in terms of PESQ and WER metrics. These findings suggest that our method, designed to simulate the perception and production processes of the speech chain, enhances the perceptual quality of separated speech and can be effectively integrated into downstream ASR tasks.

Cross-domain performance comparison. To assess the generalization performance of our approach, we conducted cross-domain performance tests on additional audio-visual datasets. Specifically, we trained our model using the VoxCeleb2-2Mix dataset and evaluated its performance on the LRS3 [Afouras et al., 2018b] and TCD-TIMIT [Harte and Gillen, 2015] datasets, derived from TED videos and studio recordings, respectively. For these datasets, we randomly selected speech segments from two distinct speakers and mixed them, following a similar approach as in [Pan et al., 2021; Lin et al., 2023]. The results, presented in Table 2, demonstrate that our proposed method exhibits robust generalization capabilities despite potential distribution shifts, which are of utmost importance in real-world applications.

4.5 Ablation Study

In this section, we performed ablation experiments to validate the effectiveness of each key design proposed in AVSepChain. All experiments were conducted on the LRS22Mix dataset.

Ablation study on speech production. To investigate the impact of the speech production process on the separation performance in AVSepChain, we excluded the AVSynthesizer and utilized the output of the AV-Separator as the final separated speech. The results are presented in the "w/o AV-Synthesizer" row of Table 3. These findings reveal that the speech production stage significantly enhances the performance of the AV-TSE task by compensating for the limitations of speech perception.

Ablation study on semantic matching. To examine the impact of the contrastive semantic matching loss, we excluded this component during training and solely utilized the SI-SNR loss to train AVSepChain. The findings are presented in the "w/o Lmat" row of Table 3. The notable enhancements observed in the PESQ and WER metrics emphasize the vital role played by the contrastive semantic matching loss in facilitating the AV-Synthesizer to learn accurate pronunciation and speech content.

Ablation study on prediction of residual signal. During the speech production stage, we compensate for the target speech, extracted during the speech perception stage, by predicting the residual signal. To emphasize the advantage of this strategy, we compare the results achieved by predicting the complete speech signal, as indicated in the "Predict complete signal" row of Table 3. The findings demonstrate a significant improvement in speech production through the prediction of the residual signal. This improvement can be attributed to the ill-posed nature of the lip-to-speech synthesis task, which encounters a one-to-many problem. By focusing on predicting the residual signal, we effectively reduce the complexity of the task and introduce implicit constraints into the synthesis process, leading to enhanced generation results of the AV-Synthesizer with a smaller scale. In future research, we aim to explore the utilization of powerful generative models, such as diffusion models [Luo et al., 2023; Choi et al., 2023a], to directly generate complete speech signals during the speech production stage and improve the upper limit of our method's performance [Lutati et al., 2023].

Comparison of visual front-ends. To validate the superiority of AV-HuBERT as the visual front-end, we compared it with several popular pre-trained lip-reading models, including ResNet-18 [Afouras et al., 2018a], MS-TCN [Mart´ınez et al., 2020], and CTCNet-Lip [Li et al., 2022]. In this experiment, we employed these models to extract visual embeddings, which were then utilized as inputs for the AVSeparator and AV-Synthesizer, as well as for calculating the contrastive semantic matching loss. The experimental results, presented in Table 4, consistently reveal the superior performance of AV-HuBERT over other models, both as input to the network and for computing the loss. Notably, AV-HuBERT significantly outperforms other pre-trained lip-reading models as a pseudo-viseme extractor in estimating the semantic matching loss. This superior performance can be attributed to the integration of visual (lip movement) and auditory (audio signals) information in AV-HuBERT, which enables the model to grasp the relationship between lip movement and speech, thereby enhancing lip-reading accuracy. Moreover, AV-HuBERT shares a similar self-supervised learning framework with HuBERT, involving feature clustering and masked prediction. Consequently, utilizing AV-HuBERT as a pseudoviseme extractor leads to semantic feature representations that better align with those of HuBERT, resulting in improved effectiveness of the contrastive semantic matching loss.

Comparison of modulation strategies. To validate the effectiveness of cross-modal modulation using cross-modal attention, we compared various modulation strategies, including concatenation and summation. The results are presented in Table 5. The findings demonstrate a significant superiority of the employed cross-modal attention method compared to other approaches. This superiority stems from the adaptive learning capability of cross-modal attention, which enables the determination of weights for each modality rather than simply feature combination or addition. Consequently, the model can accurately control the contribution of each modality by dynamically adjusting the dominant and conditional status of different modalities according to the specific task requirements, achieved by modifying the modalities of the query, key, and value. Furthermore, the model can selectively focus on the most pertinent and valuable information within each modality based on the specific task, thereby reducing the impact of redundant information. This refinement enhances the robustness and generalization ability of our model.

The impact of changing dominant and conditional modalities. In AVSepChain, we alleviate the issue of modality imbalance by dynamically switching between dominant and conditional modalities. This is accomplished by modifying the queries, keys, and values of the cross-modal attention in various tasks. To assess the effectiveness of our approach, we conducted comparative experiments by altering the order of queries, keys, and values. The results of these experiments are presented in Table 6. As the keys and values in the cross-modal attention should originate from the same feature, there are four potential settings. The outcomes demonstrate that our configuration consistently achieves the most favorable performance. Additionally, the table reveals that using either audio or visual modalities as the dominant modality for both the speech perception and speech production stages results in a significant modality imbalance problem, potentially leading to failure in the other conditional modality. When the positions of the audio and visual modalities are swapped, the negative impact increases considerably, suggesting that the dominant modality should be assigned as the keys and values, while the conditional modality should serve as the query.

5 Conclusion

From the perspective of the speech chain, this paper proposes AVSepChain, which simulates the process of speech perception and speech production in the speech chain. Its objective is to alleviate the modality imbalance problem in AV-TSE by considering audio and visual modalities as conditional modalities for each other. This is achieved by assigning different modalities as queries, keys, and values for cross-modal attention. Furthermore, to ensure that the synthesized speech preserves the same semantic information as the lip movement video during the speech production process, a contrastive semantic matching loss is proposed. Extensive experiments confirm the effectiveness of our approach, indicating that our method defines state-of-the-art performance on the AV-TSE task by mitigating modality imbalance.