Balancing Multimodal Learning via Online Logit Modulation

1 Introduction

Intuitively, multimodal models that fuse different modality data are expected to outperform unimodal models due to the richer information they provide. However, a counterintuitive finding is often observed in practice, where the bestperforming unimodal networks outperform jointly trained multimodal networks, especially in coarse-grained multimodal classification tasks [Wang et al., 2020; Peng et al., 2022]. This phenomenon can be attributed to the fact that different modalities learn representations at different rates of convergence and generalization. Current mainstream multimodal joint training frameworks, which incorporate late fusion [Xu et al., 2023a] to encode different modalities' features into a shared latent space and map them to the task space, may lead to inconsistent final convergence states among modalities, where some may be overfitted while others remain underfitted. In such cases, the multimodal model experiences degradation, relying excessively on a specific modality and failing to effectively utilize information from other modalities.

∗Corresponding author

Existing research on balancing multimodal learning can be categorized into two approaches: i) overcoming architectural deficiencies and ii) harmonizing multimodal training schemes. The former involves designing modality-specific encoders to learn the unique features of each modality and modality-invariant encoders to learn shared representations across modalities [Hazarika et al., 2020], or bridging the gap between modalities by employing modality-aware encoders [Xiao et al., 2020] (e.g., SlowFast [Feichtenhofer et al., 2019], comprising a slow pathway operating at a low frame rate to capture spatial semantics and a Fast pathway operating at a high frame rate to capture fine temporal motion). The latter includes methods such as G-blend [Wang et al., 2020], which computes an optimal blending of modalities based on their overfitting behaviors; CUR [Wu et al., 2022], which balances the conditional learning rates between modalities using the condition utilization ratio to measure the performance difference between unimodal and bimodal models;

and OGM-GE [Peng et al., 2022], which achieves dynamic and differentiated parameter updates through on-the-fly gradient modulation. Despite the modest success of these methods, they either require additional complex gradient computations and modifications to standard backpropagation or involve multiple training runs to estimate imbalanced proxies.

We postulate that joint multimodal training can only benefit from multimodal interactions when sufficient learning of unimodal features is ensured. To achieve this, we first decouple the learning of unimodal features and cross-modal feature interactions through an Intermediate Representation Fusion Block (IRFB). Next, to coordinate the optimization processes of different modalities, we propose Online Logit Modulation (OLM). Our motivation arises from observations in Fig. 1 (a) and (b), where the audio modality contributes significantly to overall performance. This dominance of the audio modality is evident from the average unimodal logit scores within each batch, while the visual modality persistently remains underoptimized throughout training. Further observations in Fig. 1 (c) and (d) demonstrate that the logit norms corresponding to each category continuously increase during training, and the logit norms of the audio and visual modalities gradually diverge as training proceeds. Guided by these observations, we propose to use OLM to regulate the magnitudes of logit vectors across diverse modalities. The core principle of OLM involves the adaptive adjustment of logit norms for each modality during training. This is achieved by applying adaptive logit coefficients to the modality-wise logit vectors. The intention is to align the magnitude of the logit vector with its predetermined target. Consequently, this approach attenuates logit norms of swiftly converging modalities while amplifying those of modalities that converge at a slower pace. The cumulative effect engenders a more harmonized optimization process. To sum up, the contributions of this work are as follows:

• We decouple the learning of unimodal features and multimodal interactions via an intermediate representation fusion block (IRFB), thereby enabling sufficient training of unimodal features.
• We introduce OLM, an online logit calibration strategy that scales each modality logit vector during training, thus aligning their magnitudes with the target logit norms. Notably, OLM is model-agnostic and harmonizes seamlessly with most multimodal training architectures.
• Empirical findings substantiate that our approach significantly enhances the performance of baseline models across diverse multimodal tasks, including human action recognition, scene categorization, audiovisual event localization, and multimodal sentiment analysis.

2 RelatedWork

Multimodal Alignment and Fusion Features fromdistinct modalities typically inhabit separate embedding spaces [Li et al., 2021; 2022]. The primary goal of multimodal alignment is to project these diverse modalities onto a shared representation space, thereby facilitating the modeling of subsequent cross-modal fusion/interaction. Contrastive learning [Chen et al., 2020b] has extensively been employed to train transformer-based multimodal models for achieving modality alignment [Jia et al., 2021; Radford et al., 2021; Yang et al., 2021; Li et al., 2022; Shen et al., 2023]. Moreover, beyond contrastive learning, the moment-based maximum mean discrepancy [Gretton et al., 2012], and the optimal transport dataset distance [Alvarez-Melis and Fusi, 2020] have also been explored for cross-modal distributional alignment. Multimodal interaction can occur at three levels: input (a.k.a, early fusion), intermediate representation (a.k.a, middle fusion), and prediction or decision level (a.k.a, late fusion) [Xu et al., 2023a]. Early fusion immediately combines features right after their extraction, often achieved through concatenation or summation of diverse representations. In contrast, in a late fusion setting, all modalities are trained independently and merged right before the model makes a decision. Middle fusion, which typically employs cross-modal attention or co-attention [Lu et al., 2019] and its variants, such as 'attention bottleneck' [Nagrani et al., 2021], enable more fine-grained modal interactions and yield more robust multimodal contextual representations.

Imbalanced Multimodal Learning Simple multimodal joint training can lead to the optimization of only one dominant modality, while other modalities suffer from underfitting [Wang et al., 2020; Peng et al., 2022; Huang et al., 2022; Wu et al., 2022; Fan et al., 2023]. In such cases, the multimodal model excessively relies on a single modality, compromising its generalization performance. To mitigate such imbalanced optimization, a series of multimodal calibration training algorithms have been proposed. For instance, Wang et al. present G-Blend, which utilizes five-fold crossvalidation to estimate the overfitting-to-generalization ratio and re-weights the training losses accordingly. Despite its effectiveness, this approach requires additional data splitting and training of individual unimodal models, resulting in an increased computational burden. Peng et al. propose onthe-fly gradient modulation (OGM-GE), which adaptively controls the optimization for each modality by monitoring their contributions to the learning objective. Moreover, it incorporates additional Gaussian noise to mitigate potential generalization degradation resulting from gradient modulation. Nevertheless, this method requires sampling from the distribution of gradient variances, hindering its training efficiency. Wu et al. introduce the concept of the conditional utilization rate (CUR), which is defined as the accuracy gain when merging one modality with another into a model and is applied to update one of the unimodal branches intentionally. However, it complicates the training protocol due to its iterative loops between standard training steps and rebalancing steps. PMR [Fan et al., 2023] leverages the prototypes, namely the centroids of each modal in representation space, to adjust the learning direction of each modal towards its prototypes. Different from these methods, OLM accelerates the slow-learning modality and alleviates the suppression from the dominant modality by adaptively modulating the logit magnitude (while maintaining the learning directions unchanged) of each modality during the entire training stage.

3 Method

Problem Definition and Notation

Empirically, we consider two modalities e.g., visual and audio modality, as an exemplar to illustrate our core idea. Given a labeled dataset D = {xi, yi}Ni=1 , where yi ∈ {1, 2, . . . , C} represents the category associated with xi. C is the number of classes and N is the number of samples. For each sample xi, we extract visual feature sequence z[0] v ∈RTv×dv and acoustic feature sequence z[0] a ∈RTa×da , where {Tm}m∈{v,a} is the sequence length and {dm}m∈{v,a} is the feature dimension of each modality. Let zm denote the sequence of token representations generated by the multimodal interactions. Tf(·) indicates the processing of Transformer layers (blocks). Our goal is to learn a robust model that can efficiently integrate all channels of multimodal information to predict y.

3.1 Multimodal Fusion Framework

Fig. 2 illustrates two distinct multimodal fusion frameworks, namely, late fusion and mid fusion. Subsequently, we provide an in-depth exposition of the proposed Intermediate Representation Fusion Block (IRFB).

Unimodal Encoder

Analogous to [Devlin et al., 2018; Dosovitskiy et al., 2020], we first append an additional learnable [CLS] token to each modality input, and then employ the standard Transformer [Vaswani et al., 2017] as a unimodal encoder to obtain token embeddings for each modality at each layer:

where ℓ indexes the layer number of the transformer models. Since the appended [CLS] token aggregates the information from all tokens, we use its embedding as the utterance-level representation for each modality.

Intermediate Representation Fusion Block

To effectively model the interactions among the utterancelevel intermediate representations, we introduce IRFB (see Fig. 3). This block not only facilitates capturing interactions among the multimodal intermediate representations within the same layer but also enables capturing interactions among different layers. It is worth noting that directly fusing multiple modalities in a one-to-one manner can be inefficient, particularly when dealing with multiple modalities simultaneously [Sun et al., 2023]. To overcome this inefficiency, we use a set of bottleneck tokens, denoted as zb, as a central message hub to facilitate communication with each modality, drawing inspiration from [Nagrani et al., 2021]. The multimodal bottleneck token set zb is randomly initialized and the number of tokens is set to Tb, i.e., zb = {zi}Tb i=1 . Tb is typically much smaller than Tv or Ta. To interact with unimodal features, we employ the multi-head attention mechanism [Vaswani et al., 2017] as follows:

where Qmb = zbWQ bm,Km = zmWKm and Vm = zmWVm are linear transformations of the bottleneck tokens and unimodal input sequences, m ∈ {v, a}. To model interactions across different layers, we adopt a tanh-gating mechanism [Hochreiter and Schmidhuber, 1997], which effectively filters out irrelevant information while retaining valuable information flow by:

where [, ] denotes the concatenation along the feature dimension, Wℓmb ∈ R(dv+da)×d are the layer-specific weight matrices, and bℓmb ∈ Rd is the bias and m ∈ {v, a}. Denote by zg =Concat(z[ℓ] v→b , z[ℓ] a→b ) the temporal concatenation of cross-attended feature sequences. To aggregate refined information from different modalities, the update rule of zb is finally defined as:

where v, W, and b are layer-specific learnable parameters. We omit the layer indexes of these parameters for brevity.

Remarks. A multimodal transformer equipped with IRFB effectively disentangles the process of unimodal feature acquisition from that of multimodal interactions, thus allowing for the decoupled learning of inter-modal features through modal fusion while ensuring sufficient and independent learning of unimodal features. Theoretically, our unimodal encoders can achieve the same performance as the unimodal baselines when trained long enough.

3.2 Online Logit Modulation

After the encoding of a multimodal transformer, we obtain a set of token representations {zv, za, zb} for the visual modality, audio modality, and fused modality, respectively. These token representations will be pooled along the temporal dimension and fed into three independent classifiers to generate the logits (also named logit vectors). That is, the pre-softmax logits can be calculated as the following:

where W ⊤ m ∈Rd×C ,m∈ {v, a, b} denote the three separate linear classifiers. Here, the SeqPooling indicates the simple sequence pooling, where the average of the output vectors is taken as the summary representation.

Without loss of generality, a logit vector sm can be decomposed as sm = ∥sm∥2 ˆ sm, where ∥·∥ denotes the L2 norm and ˆ sm is the unit vector in the same direction as sm. In other words, ∥sm∥2 and ˆ sm indicate the magnitude and the direction of the logit vector sm, respectively. Previous studies have elucidated that distinct modalities exhibit varying levels of convergence and generalization rates [Wang et al., 2020; Nagrani et al., 2021]. Our objective is to devise a set of dynamic blending weights {wm}, ∀m ∈ {v, a, b} to improve joint multimodal training, which ensures similar rates of parameter updates across different modalities during training. To achieve this, we impose constraints on the magnitude of logit vectors corresponding to each modality. In order to determine the optimal magnitude value for each modality, we take into account both the model's convergence rate and generalization rate, similar to [Wang et al., 2020]. Concretely, we gauge the generalization rate at the n-th iteration step using:

where Ldev represents the validation loss, and Θ denotes network parameters. Similarly, we measure the convergence rate C at the n-th iteration step as follows:

where Ltrain denotes the training loss. We then compute the generalization and convergence rates for each modality m, denoted as {Gn,m} and {Cn,m}. A higher convergence rate indicates a higher risk of overfitting for a modality.We encourage the use of smaller logit magnitudes to counteract overfitting and larger logit magnitudes to mitigate underfitting. To this end, we can derive a set of modality-aware rebalancing factors by:

Building upon these modal rebalancing factors, we first outline a baseline for different modality logit magnitudes, which can be an average logit norm of weighted modality logit vectors: ¯ sn = Em∈{v,a,b}(||wm,n · sm,n||2). Then we construct the target logit magnitude for each modality via λn,m · sn. Our goal is to find an optimal set of blending logit coefficients that modulate the magnitude of the logit vector per modality to align the target logit magnitude. Therefore, the logit modulation loss can be formalized as follows:

Note that the optimization of Eq. 9 is nonlinear due to the L2 norms involved in the expression. As a result, it is unlikely to find a global analytical solution for the optimal values of {wv,wa,wb}. Instead, we employ the gradient descent algorithm to search for the optimal coefficients that lead to the desired minima. Algorithm 1 describes a pipeline for solving a set of logit coefficients using the SGD optimizer.

4 Experiment

4.1 Datasets andMetrics

Kinetics-Sounds (KS) [Arandjelovic and Zisserman, 2017] is a subset of 36 human action classes selected from the Kinetics dataset [Kay et al., 2017], comprising 10-second videos sampled at 25fps from YouTube. In line with [Peng et

Algorithm 1: Online Logit Modulation (OLM)

Input : Logit vectors {sv, sa, sb} output by unimodal and cross-modal encoders; the training loss Ltrain(m, n) and validation loss Ldev(m, n) for each modality m at the nth iteration step; and the learning rate η Output: Logit Coefficients {wv,n, wa,n, wb,n} 1 Initialize the learning rate and the logit coefficients {wv , wa, wb}, wv (0) ← 1, wa(0) ← 1, wb(0) ← 1; 2 Compute the rebalancing fators {λv, λa, λb} according to Eq. 8; 3 for i ← 1 to max iteration step do // compute an average ℓ2-norm of different logit vectors 4 ¯ s ← Em∈{v,a,b}(||wm · sm||2); // compute the logit modulation loss between the current weighted logit norms and target ones 5 Llogit ←Eq. 9; // update the coefficients using stochastic gradient descent 6 wv(i) ←wv(i -1) -η ∗ ∂Llogit ∂wv ; 7 wa(i) ←wa(i -1) -η ∗ ∂Llogit ∂wa ; 8 wb(i) ←wb(i -1) -η ∗ ∂Llogit ∂wb ; 9 Return: {wv,n,wa,n,wb,n}={wv(i),wa(i),wb(i)};

al., 2022], we focus on 31 action categories that can be recognized visually and auditorily, including actions like playing various musical instruments. This dataset contains 19,000 10second video clips, with 15,000 clips used for training, 1,900 for validation, and 1,900 for testing.

VGGSound [Chen et al., 2020a] is a large-scale video dataset comprising 309 classes, with nearly 200K 10-second video clips capturing a diverse range of audio events in everyday life. Each clip's sound source is visually presented in the video, demonstrating clear audio-visual correspondence. After filtering out unavailable videos, we obtained 168,618 videos for training and validation, and 13,954 for testing.

AVE [Tian et al., 2018] is a subset of the AudioSet dataset [Gemmeke et al., 2017] designed for audio-visual event localization. It comprises 28 event categories, consisting of 4,143 10-second videos. This dataset encompasses a diverse range of audio-visual events from various domains, with each video containing at least one 2-second long audiovisual event, annotated with frame-level boundaries. The training, validation, and test sets are divided into 3,339, 402, and 402 samples, respectively.

CREMA-D [Cao et al., 2014] is a multimodal dataset designed for speech emotion recognition. It comprises 7,442 video clips, each lasting 2 to 3 seconds, featuring 91 actors delivering concise utterances. The dataset encompasses six of the most prevalent emotions: anger, happiness, sadness, neutral, disgust, and fear. It consists of 6,698 samples for training and validation, with 744 samples for testing.

MER-MULTI is a subchallenge of the MER2023 [Lian et al., 2023], aiming to simultaneously recognize discrete emotions and valence in given raw video clips. The discrete emotion categories include happiness, neutral, anger, sadness, worry, and surprise. Valence is an emotional dimension with values ranging from -5 to 5, reflecting the degree of emotional pleasantness. This dataset comprises 3,373 video clips for training and 411 video clips for testing.

SUNRGBD V1 [Song et al., 2015] comprises 10,335 RGBD images collected from different sensors. We evaluate OLM on the scene classification task, which entails categorizing a given RGB-D image into one of the predefined 19 scene categories. We partition the data into training and testing sets, ensuring that approximately half of the data from each sensor goes into two subsets. Given that some images were captured from the same building or house with similar furniture styles, we ensure that images from the same building are either entirely within the training set or exclusively in the testing set.

The evaluation metrics employed in our experiments include the commonly used top-1 accuracy, F1-score, mean average precision (mAP), and mean squared error (MSE).

4.2 Experimental Settings

We evaluate two model variants. One variant is implemented with traditional CNN-based encoders combined with late fusion (refer to Fig. 2), termed OLM-Conv. The other variant uses transformer-based encoders equipped with the Intermediate Representation Fusion Module (IRFM), referred to as OLM-Trans. Both variants of our model employ the online logit modulation strategy. Specifically, for OLM-Conv, we use ResNet18 [He et al., 2016] as the encoders following previous works [Zhao et al., 2018; Peng et al., 2022]. AVE, Kinetics-Sounds, and VGGSound datasets consist of videos with a duration of 10 seconds each. To process these videos, we extract frames at a rate of 1fps and uniformly sample 3 frames from each clip, which serve as the visual input for our model. For the audio data, we utilize a window of length 512

with an overlap of 353 to transform the raw audio data into spectrograms of size 257×1004 using the librosa [McFee et al., 2015] library. As for SUNRGBD, we adopt PlacesCNN [Zhou et al., 2014], which reaches the optimal performance for color-based scene classification on the SUN database [Xiao et al., 2010], for feature extraction of both RGB and depth images. Regarding CREMA-D, its video clips last from 2 to 3 seconds. From each clip in CREMAD, we extract 1 frame and use a window of length 512 with an overlap of 353 to convert the audio data into spectrograms of size 257×299. For MER-MULTI, we first extract human face images using the OpenFace toolkit. The pre-trained MANet [Zhao et al., 2021], HuBERT [Hsu et al., 2021], and MacBERT [Cui et al., 2020] models were employed for the extraction of visual, audio, and textual features, respectively. For OLM-Trans, we stack six standard transformer blocks and IRFB blocks (cf. Sup. D for implementation details).

4.3 Comparison with State-of-the-Arts

We compared OLM with a wide range of baseline methods, including two unimodal baselines and four simple bimodal baselines: concatenation, summation, gated, and attention. Additionally, we explored several advanced techniques for multimodal fusion, alignment, and training, encompassing:

• FiLM [Perez et al., 2018] performs a simple featurewise affine transformation on the intermediate features

of a neural network based on conditional information.

• GradNorm [Chen et al., 2018] proposes gradient normalization to balance training in deep multitask models by dynamically tuning gradient magnitudes.
• MMCosine [Xu et al., 2023b] imposes modality-wise L2 normalization to features and weights by cosine similarity towards balanced multi-modal learning.
• AVSlowFast [Xiao et al., 2020] is a multimodal extension of SlowFast [Feichtenhofer et al., 2019], which incorporates a faster audio pathway and deeply fuses audio and visual features at multiple levels.

The remaining comparison methods for modulating the pace of multimodal training, including G-Blend [Wang et al., 2020], CUR [Wu et al., 2022], OGM-GE [Peng et al., 2022] and PMR [Fan et al., 2023], are introduced in § 2. Table 1 reveals several intriguing observations: i) a performance imbalance among modalities, with the audio modality exhibiting dominance. For instance, the performance of audio-only baselines significantly surpasses that of visual-only baselines across the three datasets; ii) Occasionally, the performance of unimodal baseline surpasses that of simple multimodal fusion baselines, as witnessed in the case of CREMA-D where the audio-only baseline outperforms all bimodal fusion baselines. This indicates potential under-optimization of multimodal models due to naive multimodal joint training; iii) The proposed OLM strategy exhibits advantages compared to the other competitors. Specifically, OLM-Conv attains superior or comparable accuracy and mAP scores across the three datasets. To further validate the versatility of our strategy, we apply OLM to the AVE localization task. Table 2 compares OLM-Conv with several competitive counterparts on this task. As seen, when combined with diverse AVE backbone networks, OLM-Conv yields the most significant improvement over the baseline in most cases, confirming the effectiveness of logit modulation in bolstering multimodal training. Worth noting is that unlike G-Blend and OGM-GE, our OLM neither necessitates additional unimodal networks to compute per-modality weights nor involves intricate gradient calibrations, therefore boasting greater out-of-the-box applicability. Furthermore, we present the confusion matrices generated by models with and without the OLM strategy applied in Fig. 4. The OLM-trained model exhibits pronounced enhancement over the non-OLM-trained model (w.r.t. acc. and recall), highlighting the OLM's generalization ability.

4.4 Ablation Study

Impact of Different Modalities and IRFB

Table 3 presents the unimodal baseline results and attention fusion-based multimodal baseline results on MER-MULTI. Particularly, we investigate two variants of our model, namely OLM-T-MBT and OLM-T-IRFB. The former employs a nondecoupled multimodal attention bottleneck (MBT) [Nagrani et al., 2021] for cross-modal mid fusion, while the latter employs IRFB for mid fusion. Both model variants share identical transformer-based unimodal encoders and use the same attention late fusion to generate unimodal and cross-modal outputs. In Table 3, several observations merit attention: a) Disparities in performance among different modalities are evident. The text modality performs worst, while the audio modality performs best; b) OLM substantially boosts the performance of any modality combination baseline, showcasing its versatility; c) OMT-T-IRFB outperforms OMT-T-MBT, with an average improvement of approximately 2%.

Visualization of Logit Modulation

Fig. 5 illustrates the class-specific L2 norm of logit vectors for the visual and audio modalities. It is evident that, as training progresses, the magnitude of logit vectors per class continues to increase. However, the rate of logit norm growth varies across modalities, with the logit norm for the audio modality increasing more rapidly than that for the visual modality.

This suggests a faster learning pace for audio modality features and potential underfitting in the visual modality. OLM operates on the logit magnitudes of diverse modalities. Analysis of Fig. 5 (C) and (D) reveals that the rate of logit norm growth in the audio modality decelerates, whereas it accelerates in the visual modality, thus achieving modality-aware balanced training. The same trend is also discernible from the logit score curves in Fig. 6 (B) and (D). Fig. 6 (A) and (B) aptly demonstrate that OLM mitigates the underfitting issue of visual modality in the context of multimodal joint training. These figures intuitively underscore the benefits of OLM in the realm of multimodal training: the ability to foster more robust multimodal features while ensuring comprehensive unimodal feature training, thereby unlocking the genuine potential of multimodal models.

5 Conclusion

In this work, we present two core components, IRFB and OLM, tailored to enhance multimodal training. Specifically, IRFB efficiently alleviates potential underfitting concerns of unimodal networks in imbalanced multimodal learning by disentangling the learning of unimodal features from multimodal interactions. OLM, unrestricted by model architectures and fusion methods, mitigates the suppression of dominant modalities on other modalities. This is achieved by regulating the magnitude of the logit vector for each modality, aligning it with its modality-aware target. Empirical studies have fully unveiled the efficacy of IRFB and the superiority of OLM over prevailing multimodal training calibration alternatives.

Contribution Statement

Daoming Zong and Chaoyue Ding contributed equally to this paper.