Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition

1 Introduction

With recent advancement of deep learning techniques, automatic speech recognition (ASR) has achieved quite good performance [Graves, 2012; Vaswani et al., 2017]. However, ASR systems are usually vulnerable to noise and would degrade significantly when put under noisy conditions [Sumby and Pollack, 1954]. To improve their performance under various scenarios, recent works on noise-robust speech recognition have made some progress [Wang et al., 2020].

A currently popular research direction on robustness combines audio (A) and visual (V) features to benefit from the noise-invariant lip movement information. With use of two modalities, audio-visual speech recognition (AVSR) systems move one step closer to how human perceives speech [Sumby and Pollack, 1954] and achieve better performance in many application scenarios [Biswas et al., 2016; Koguchi et al.,

2018]. Thanks to recent advance of neural network, AVSR has achieved a remarkable success [Afouras et al., 2018a; Makino et al., 2019; Xu et al., 2020; Ma et al., 2021; Pan et al., 2022; Shi et al., 2022a; Shi et al., 2022b; Hsu and Shi, 2022]. However, most existing AVSR works simply employ feature concatenation for audio-visual (A-V) fusion, without explicit interactions to capture deep correlations between them [Raij et al., 2000]: 1) From global perspective, they may not capture the complementary relationship between A-V modalities. Such relationship means when one modality is missing or corrupted, the other modality can supply valid information for downstream task [Wang et al., 2022]. Failure to capture it would make the system confused about the significance of each modality and thus degrade the performance [Hori et al., 2017; Tao and Busso, 2018]. 2) From local perspective, they may ignore the temporal alignment between A-V frames, which could be a problem due to the ambiguity of homophenes [Kim et al., 2022] where same lip shape could produce different sounds. Such misalignment between lip and audio sequences would increase the difficulty of efficient multimodal fusion and affect final performance [Tsai et al., 2019; Lv et al., 2021].

To capture the global complementary relationship between different modalities, cross-attention has been widely investigated in recent multimodal studies to learn the inter-modal correspondence [Lee et al., 2020; Li et al., 2021; Goncalves and Busso, 2022; Mercea et al., 2022]. Despite the effectiveness, it fails to simultaneously preserve the intra-modal correspondence that could adaptively select the information of each individual modality for the inter-modal correspondence modeling [Wang et al., 2022], which thus results in sub-optimal complementary relationship between modalities.

From the local perspective, contrastive learning has been popular for cross-modal temporal alignment to model the frame-level consistency [Korbar et al., 2018; Morgado et al., 2021; Yang et al., 2022], but they seem to only align the multimodal features within same model layer, ignoring the alignment across different layers. Since different-layer features contain semantic representations of different granularities [Gu et al., 2021], we argue that the alignment between them could capture extra contextual information to improve the modeled temporal consistency.

In this paper, we propose a cross-modal global interaction and local alignment (GILA) approach to effectively capture the deep audio-visual correlations from both global and local perspectives. Specifically, we propose an attention-based global interaction (GI) model to capture the A-V complementary relationship on modality level. On top of the vanilla cross-attention, we propose a novel iterative refinement module to jointly model the A-V inter- and intra-modal correspondence. It could adaptively leverage the information within each individual modality to capture the inter-modal correspondence, which thus results in better complementary relationship between A-V modalities. With global knowledge of A-V correlations, the system may still be less aware of the local details. To this end, we further design a crossmodal local alignment (LA) approach via contrastive learning to model the A-V temporal consistency on frame level. Based on the vanilla within-layer alignment, we propose a novel cross-layer contrastive learning approach to align A-V features across different GI model layers. Such design could capture extra contextual information between the differentgranularity semantic representations, which enables more informative temporal consistency between A-V frames. As a result, our proposed GILA can capture deep holistic correlations between A-V features and finally generate better multimodal representations for downstream recognition task.

To the best of our knowledge, this is the first AVSR work to model deep A-V correlations from both global and local perspectives. Our main contributions are summarized as:

• We present GILA, a novel approach to capture deep audio-visual correlations for AVSR task, from both global and local perspectives.
• We propose a cross-modal global interaction (GI) model to capture A-V complementary relationship on modality level, as well as a local alignment (LA) approach to model the A-V temporal consistency on frame level.
• Experimental results on two public benchmarks demonstrate the effectiveness of our GILA against the state-ofthe-art (SOTA) supervised learning baseline, with up to 16.2% relative WER improvement.

2 RelatedWork

Audio-Visual Speech Recognition. Most existing AVSR works focus on novel architectures and supervised learning methods, investigating how to effectively model and fuse the audio-visual modalities. TM-seq2seq [Afouras et al., 2018a] proposes a Transformer-based [Vaswani et al., 2017] AVSR system with sequence-to-sequence loss. HybRNN [Petridis et al., 2018] proposes a RNN-based AVSR system with hybrid seq2seq/CTC loss [Watanabe et al., 2017]. RNN-T [Makino et al., 2019] employs recurrent neural network transducer [Graves, 2012] for AVSR task. EG-seq2seq [Xu et al., 2020] builds a joint audio enhancement and multimodal speech recognition system based on RNN. LF-MMI TDNN [Yu et al., 2020] proposes a joint audio-visual speech separation and recognition system based on TDNN. Hyb-Conformer [Ma et al., 2021] proposes a Conformer-based [Gulati et al., 2020] AVSR system with hybrid seq2seq/CTC loss, where the audio-visual streams are encoded separately and then concatenated for decoding, which has achieved the supervised learning SOTA on both LRS3 and LRS2 datasets. MoCo+wav2vec [Pan et al., 2022] employs self-supervised pre-trained audio/visual frontends to improve AVSR performance, which has achieved the SOTA on LRS2 dataset. However, these studies simply concatenate the audio and visual features for multimodal fusion, without explicit interactions to capture their deep correlations. Recently proposed AV-HuBERT [Shi et al., 2022a; Shi et al., 2022b] employs self-supervised learning to capture contextual correlations between audio-visual features, and the latest u-HuBERT [Hsu and Shi, 2022] extends it to a unified framework of multimodal and unimodal pre-training, which has achieved the SOTA on LRS3 dataset. However, they require a large amount of unlabeled data and computing resources. In this work, we propose a novel supervised learning approach called GILA to efficiently capture deep A-V correlations from both global and local perspectives.

Cross-Modal Modality-Level Interaction. Attention methods have been widely investigated to interact between different modalities to capture their complementary relationship, in various multimodal applications such as A-V emotion recognition [Goncalves and Busso, 2022], A-V action localization [Lee et al., 2020], etc. Recent works employ crossattention to enable extracted features of different modalities to attend to each other [Lee et al., 2020; Li et al., 2021; Goncalves and Busso, 2022; Mercea et al., 2022], which is found effective to capture the inter-modal correspondence and significantly improves the system performance. However, they may not simultaneously preserve the intra-modal correspondence that could adaptively select the unimodal information for inter-modal correspondence modeling [Wang et al., 2022]. To this end, we propose a novel iterative refinement module to jointly model the inter- and intra-modal correspondence, where the key idea is introducing a bottleneck feature to recurrently collect multimodal information.

Cross-Modal Frame-Level Alignment. Cross-modal alignment aims to model the temporal consistency between sequences of different modalities, and alleviate the frame-level misalignment problem in some scenarios [Tsai et al., 2019; Lv et al., 2021; Kim et al., 2022]. This is typically done by contrastive learning where the correspondence between positive pairs is trained to be stronger than those of negative pairs [Chopra et al., 2005]. Recently, contrastive learning is popular for cross-modal temporal alignment, which has achieved significant improvement on various tasks [Korbar et al., 2018; Morgado et al., 2021; Yang et al., 2022]. However, they seem to only align features of multiple modalities within same model layer, ignoring the alignment across different layers that could learn extra contextual information between different-granularity semantic representations. In this work, we propose a cross-layer contrastive learning approach for holistic A-V alignments.

3 Methodology

In this part, we first introduce the overall architecture of proposed GILA in Section 3.1. Then, we describe its two main components, i.e., the cross-modal global interaction model in Section 3.2 and local alignment approach in Section 3.3. Finally, we explain the training objective in Section 3.4.

3.1 Overall Architecture

As illustrated in Figure 1(a), the proposed GILA system consists of front-end module, fusion module and recognition module. We first introduce a front-end module to pre-process the synchronized audio-video input streams, which employs a linear projection layer for audio front-end and a modified ResNet-18 [Shi et al., 2022a] for visual front-end. We also concatenate the processed A-V features to build a bottleneck feature XBN to collect multimodal information. Then, we propose a fusion module for audio-visual fusion. Specifically, we propose a global interaction model and a local alignment approach to capture deep A-V correlations. The resulted audio, visual and bottleneck features are then concatenated to generate the multimodal feature XMM. Finally, we introduce a Transformer-based recognition module to encode the multimodal feature and predict the output tokens. The overall training objective consists of the speech recognition loss LASR and the local alignment loss LLA.

3.2 Cross-Modal Global Interaction (GI)

As shown in Figure 1(b), we propose a cross-modal global interaction model to capture the complementary relationship between A-V modalities. Specifically, we first introduce cross-attention to interact audio-visual features to capture inter-modal correspondence. On top of that, we further propose a novel iterative refinement (IR) module to jointly model the inter- and intra-modal correspondence, aiming to better capture the complementary relationship on modality level.

Cross-Attention aims to capture the A-V inter-modal correspondence. As illustrated in Figure 1(b), the input audiovisual features of i-th GI model layer (i.e., Xi-1 A , Xi-1 V , i ∈ {1, 2, 3}) are first sent into two separate self-attention modules [Vaswani et al., 2017] for modeling, which generates two intermediate features, F iA and F iV :

where "LN" denotes layer normalization [Ba et al., 2016], "MHA" denotes multi-head scaled dot-product attention [Vaswani et al., 2017].

Then, we introduce cross-attention to enable audio-visual features to attend to each other for complementation, in order to capture the inter-modal correspondence:

After that, we utilize position-wise feed-forward network (FFN) [Vaswani et al., 2017] to generate outputs:

where FFN consists of two linear layers with a ReLU [Glorot et al., 2011] activation in between.

Iterative Refinement (IR) aims to jointly model the A-V inter- and intra-modal correspondence, where the bottleneck feature plays a key role. As shown in Figure 1(c), the input bottleneck feature Xi-1 BN first attends to the A/V feature from cross-attention (i.e., XiA, XiV ) respectively, followed by convolution to generate two residual features RiA and RiV :

where "Conv" denotes a 1 × 1 convolution layer followed by batch normalization (BN) [Ioffe and Szegedy, 2015] and parametric ReLU (PReLU) activation.

The attention blocks aim to build interactions between individual audio/visual feature and the bottleneck feature that contains multimodal information. Therefore, the individual A/V modality can not only attend to the other modality, but also attend to itself simultaneously. As a result, we can jointly model the inter- and intra-modal correspondence, which helps extract the useful information in A/V modality.

Finally, we add the two generated residual features to input bottleneck feature, in order to refine more informative multimodal representations:

With increasing multimodal information in the bottleneck feature, the IR module in next GI model layer can better capture the A-V correspondences by Equation 4, and so on. Such refining mechanism enables IR module to effectively model the inter- and intra-modal correspondence.

3.3 Cross-Modal Local Alignment (LA)

In order to learn more local details of A-V correlations, we further propose a cross-modal local alignment approach to model the temporal consistency between A-V frames, as presented in Figure 2. Specifically, we first introduce withinlayer contrastive learning to align the A-V features within same GI model layer. Based on that, we propose a novel cross-layer contrastive learning method for A-V alignment across different GI model layers, aiming to learn more informative A-V temporal consistency on frame level.

Within-Layer (WL) Contrastive Learning aims to align the A-V features within same GI model layer. As illustrated by Figure 2(a)(b), we select the i-th layer's intermediate features F iA and F iV for alignment. Denote that F iA = {F iAt |Tt=1 }, F iV = {F iVt |Tt=1 }, i ∈ {1, 2, 3}, T is number of frames. Given each audio frame F iAt , the model needs to identify its corresponding visual frame F iVt from the entire visual sequence, and vice versa. In this sense, the A-V sequences can get well aligned to each other.

The within-layer contrastive loss is defined as:

where ⟨·, ·⟩ denotes cosine similarity, τ is temperature parameter. The two alignment directions (i.e., a2v, v2a) are averaged to obtain the final WL contrastive loss.

Cross-Layer (CL) Contrastive Learning aims to align the A-V features across different GI model layers. As presented in Figure 2(a)(c), we select the j-th layer's output audio feature XjA and k-th layer's output visual feature Xk V for alignment, where j, k ∈ {0, 1, 2, 3}, j ̸= k. Particularly, in this work we select (j, k) ∈ {(0, 3), (3, 0)} to align the input and output A-V features of entire GI model, where more selections are discussed in ablation study (See Section 4.3).

Denote that XjA = {XjAt |Tt=1 },Xk V = {Xk Vt |Tt=1 }, where T is number of frames. First, we randomly sample T ′ A-V frame pairs from them for alignment, as a dropout to prevent over-fitting. Therefore, we can write the sampled frames as {(XjAt , Xk Vt )|t ∈ I}, where I ⊂ {1, 2, ..., T}, |I| = T ′ .

Then, we introduce a vector-quantization (VQ) module [Baevski et al., 2019] to discretize the sampled audiovisual frames to a finite set of representations, which results in quantized targets to enable more effective contrastive learning, especially between different-layer features that usually locate in distant domains [Baevski et al., 2020]:

Finally, we calculate cross-layer contrastive loss to align the audio/visual frames to the quantized visual/audio representations respectively, similar toWL contrastive loss:

where It contains the index t and another 100 randomlyselected indexes from I, for positive and negative samples respectively [Baevski et al., 2020]. The two alignment directions are averaged to obtain the final CL contrastive loss.

3.4 Training Objective

We first calculate cross-entropy based sequence-to-sequence loss [Watanabe et al., 2017] for speech recognition, as indicated by LASR in Figure 1(a). Then, we build the local alignment loss LLA fromWL and CL contrastive learning:

where M = {1, 2, 3}, N = {(0, 3), (3, 0)}, λiWL and λj,k CL are weighting parameters for different training objectives.

We combine them to form the final training objective and train the entire GILA system in an end-to-end manner:

4 Experiments

4.1 Experimental Setup

Datasets. We conduct experiments on two large-scale publicly available datasets, LRS3 [Afouras et al., 2018b] and LRS2 [Chung et al., 2017]. LRS3 dataset collects 433 hours of transcribed English videos from TED and TEDx talks. LRS2 dataset contains 224 hours of video speech from BBC programs. More details can be found in [Shi et al., 2022a].

Baselines. We employ AV-HuBERT 1 [Shi et al., 2022a] as our baseline, but for fair comparison we discard the pre-training stage. To evaluate our GILA, we select some popular AVSR methods for comparison: TM-seq2seq, TMCTC, Hyb-RNN, EG-seq2seq, RNN-T, LF-MMI TDNN, Hyb-Conformer, MoCo+wav2vec, AV-HuBERT (LARGE), u-HuBERT (LARGE), which are introduced in Section 2.

Implementation Details. For model configurations, our baseline follows AV-HuBERT LARGE [Shi et al., 2022a] with 24 Transformer encoder layers and 9 decoder layers. For fair comparison, we build the GILA with 3 GI model layers, 12 Transformer encoder layers and 9 decoder layers. All other model configurations are same as AV-HuBERT LARGE. The number of parameters in our baseline and GILA are 476M and 465M respectively. We also use Conformer as our backbone, with the convolution kernel size of 31.

1 https://github.com/facebookresearch/av hubert

The system inputs are log filterbank features for audio stream and lip regions-of-interest (ROIs) for video stream. To sample A-V frame pairs in CL contrastive learning, we first sample starting indexes from (X0A,X3 V ) with probability of 0.4 and from (X3A,X0 V ) with 0.45 respectively, and then cut out 10 consecutive frames after each sampled index. To calculate contrastive loss, we use the same VQ module in wav2vec2.0 [Baevski et al., 2020], and set the temperature parameter τ to 0.1. We further use data augmentation to improve noise robustness, where we add MUSAN noise [Snyder et al., 2015] following prior work [Shi et al., 2022b], and report WER results on both clean and noisy test sets. The weighting parameters λiWL(i ∈ {1, 2, 3})/λ0,3 CL/λ3,0 CL are set to 0.001/0.08/0.01 respectively. All hyper-parameters are tuned on validation set. Our training follows the finetuning configurations in [Shi et al., 2022a] and takes ∼ 1.3 days on 4 V100-32GB GPUs, which is much more efficient than AV- HuBERT pre-training (∼ 15.6 days on 64 V100-GPUs). The details of data augmentation, model and training configurations follow previous work [Shi et al., 2022b].

4.2 Main Results

Results on LRS3. Table 1 compares the performance of our proposed GILA with existing methods on LRS3 benchmark. Under clean test set, our best model outperforms the supervised learning SOTA by 14.8% relatively (2.3%→1.96%), while without the CTC training loss, external LM rescoring and extra A/V front-end pre-training that their method uses. Moreover, the proposed GILA has also achieved significant WER improvements over our baseline (3.75%→2.61%, 2.64%→1.96%). Specifically, its two main components, i.e., GI model and LA method, both contribute a lot to the improvements, and the data augmentation also yields better results. We can also observe similar improvements on noisy test set. In addition, the Conformer backbone significantly outperforms Transformer (2.61%→1.96%).

Results on LRS2. Table 2 compares the performance of our GILA with existing AVSR methods on LRS2 benchmark. Under clean test set, our best model achieves 16.2% relative WER improvement over the supervised learning SOTA (3.7%→3.10%). Moreover, the GILA has also achieved significant improvements over our baseline (5.79%→4.02%, 4.09%→3.10%), where the GI model, LA method and data augmentation all yield positive contributions.

Therefore, our GILA has achieved new supervised learning SOTA on both LRS3 and LRS2 benchmarks, with up to 16.2% relative WER improvement over the best baseline. It also moves closer to the self-supervised learning SOTA (1.96% vs. 1.2%, 3.10% vs. 2.6%) while costs no unlabeled data and much less computing resources (See Section 4.1).

4.3 Ablation Study

Effect of Global Interaction Model. Table 3 summarizes the effect of proposed GI model and its two sub-modules, i.e., cross-attention and IR modules. We first observe that using cross-attention to capture inter-modal correspondence can improve the WER results (3.75%→3.50%, 2.64%→2.45%). Further improvements can be achieved by adding IR module to jointly model the inter- and intra-modal correspondence

(3.50%→3.29%, 2.45%→2.31%), where using it separately can also improve. Similar improvements can be observed on the noisy test set. Therefore, these results verify the effectiveness of our proposed GI model.

Visualizations of Inter- and Intra-Modal Correspondence. Figure 3 visualizes the captured inter- and intra-modal correspondence by our GI model, using similarity matrixes where the diagonal elements denote cosine similarity between true A-V, A-A or V-V pairs. We first observe chaotic mappings between A-V embeddings in baseline from Figure 3(a). After introducing cross-attention to interact A-V features, we can capture some inter-modal correspondence between true A-V pairs, i.e., (b) vs. (a). However, it fails to capture the A/V intra-modal correspondence, i.e., (f) vs. (e), (j) vs. (i). Thus, we further propose an iterative refinement module to jointly model the inter- and intra-modal correspondence, which improves significantly as indicated by the clearer diagonals in column 4. As a result, our GI model can effectively capture both inter- and intra-modal correspondence.

We further investigate the relationship between these two correspondences. When compared to baseline, using crossattention can learn better inter-modal correspondence, i.e., (b) vs. (a), while using it on top of IR module achieves significantly more improvements, i.e., (d) vs. (c). Similar phenomenon can be observed on WER results in Table 3. It indicates that the proposed IR could be beneficial to cross-attention, where its captured intra-modal correspondence could help to model the inter-modal correspondence, thus results in better A-V complementary relationship.

Effect of Local Alignment Approach. Table 4 summarizes the effect of proposed LA method and its two components, i.e., within-layer and cross-layer contrastive learning. We first introduce WL contrastive learning for audio-visual alignment within same GI model layer, which can improve the WER performance (3.29%→3.03%, 2.31%→2.13%). Further improvements can be achieved by adding CL contrastive learning to align the A-V features across different layers (3.03%→2.88%, 2.13%→2.04%), where using it separately can also improve. Similar improvements can be observed on noisy test set. Therefore, these results validate the effectiveness of our proposed LA method.

Effect of Cross-Layer Contrastive Learning. Table 5 further analyzes the effect of cross-layer contrastive learning, where we report WER results of alignment between different A-V feature pairs (XjA,Xk V ). We observe that the more layers our A-V alignment across (i.e., larger |j -k|), the better performance we can achieve, where the best two results (2.07%, 2.06%) are achieved by aligning the input and output A-V features of entire GI model. After combining them, we can achieve even better WER result, as indicated in Table 4 (2.04%). The reason could be that, the higher-layer features contain semantic representations of larger granularity, or larger receptive field. Therefore, the A-V alignment across more layers also means across larger granularity gap, which could learn richer cross-modal contextual information and results in more informative A-V temporal consistency.

Visualizations of Audio-Visual Temporal Consistency. Figure 4 visualizes the A-V temporal consistency modeled by within-layer and cross-layer contrastive learning, using attention map where the diagonal elements indicate the attention weights between corresponding A-V frames. We first observe misalignment between A-V sequences in GI model, such as the one-to-many lip-audio mappings shown in Figure 4(a). Our proposed WL contrastive learning can help model the temporal consistency between A-V sequences, as indicated by the clearer diagonal in (b). Similar improvements can be observed on cross-layer temporal consistency, i.e., (f)/(j) vs. (e)/(i), while we also observe some vertical and horizontal stripes near the diagonal, which indicate the granularity gap between different-layer features.

Then in the proposed CL contrastive learning that consists of two alignment directions (See Equation 8), the low-layer features first learn rich A-V contextual correlations from the high-layer features that with large receptive field, which alleviates the granularity gap between them, i.e., (g)/(k) vs. (e)/(i), (h)/(l) vs. (f)/(j). Meanwhile, the high-layer features can learn clearer A-V contextual mappings by aligned to the low-layer features that with small granularity, as indicated by the brighter diagonals in Figure 4 (column 3 vs. column 1, column 4 vs. column 2). As a result, the proposed cross-layer alignment can capture rich cross-modal contextual information to learn better A-V temporal consistency.

5 Conclusion

In this paper, we propose a cross-modal global interaction and local alignment (GILA) approach for audio-visual speech recognition, in order to capture the deep audio-visual correlations from both global and local perspectives. In particular, we first propose a global interaction model to capture the A-V complementary relationship on modality level. Furthermore, we design a cross-modal local alignment approach to model the A-V temporal consistency on frame level. Such a holistic view of cross-modal correlations enable better multimodal representations for AVSR. Experimental results on two public benchmarks demonstrate that our approach has achieved the state-of-the-art in supervised learning methods.

Acknowledgments

This research is supported by ST Engineering Mission Software & Services Pte. Ltd under collaboration programme (Research Collaboration No.: REQ0149132). The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore (https://www.nscc.sg).