Towards Equilibrium: An Instantaneous Probe-and-Rebalance Multimodal Learning Approach

1 Introduction

Inspired by the human ability to perceive and understand the world through multiple senses, multimodal learning (MML) [Zhao et al., 2016; Yang et al., 2019; Baltrusaitis et al., 2019; Sun et al., 2023] has emerged as a highly popular research field in recent years. In the era of deep learning, leveraging the powerful representational capabilities of deep neural networks, deep learning-based multimodal learning [Sun et al., 2023] has achieved significant advancements. In real-world applications, multimodal learning has also been successfully applied across a wide range of fields [Chang et al., 2015; Wang et al., 2018; Zhu et al., 2023; Yang et al., 2024b].

∗Corresponding author.

Despite the abundant information provided by multimodal data, multimodal learning is expected to deliver superior performance compared to unimodal approaches. However, recent studies have revealed that multimodal models sometimes underperform compared to unimodal models in specific scenarios [Wang et al., 2020]. Since different modalities vary in the amount of information they contain and their representation capabilities, their performances also tend to be inconsistent. Generally, strong modalities contain richer information and exhibit higher performance, resulting in superior outcomes, whereas weak modalities lack such advantages and perform worse [Yang et al., 2015]. This phenomenon is commonly referred to as modality imbalance [Wang et al., 2020; Huang et al., 2022; Du et al., 2022].

To deal with this problem, many efforts [Wang et al., 2020; Huang et al., 2022; Wu et al., 2022; Zong et al., 2024] have been made in recent years. Some scholars have approached this topic from a theoretical perspective, investigating issues such as generalization [Huang et al., 2021], training competitiveness [Huang et al., 2022], and the greedy nature [Wu et al., 2022] of multimodal learning. Simultaneously, numerous multimodal learning methods have been proposed to address modality imbalance, including gradient-based approaches [Peng et al., 2022; Fan et al., 2023], learning rate modulation techniques [Yao and Mihalcea, 2022], and alternating optimization paradigms designed to enhance interaction [Zhang et al., 2024; Fan et al., 2024; Hua et al., 2024]. These efforts lay a theoretical and methodological foundation for a comprehensive understanding of the nature of multimodal learning and, to some extent, mitigate performance degradation caused by modality imbalance.

Although numerous methods have been proposed to address the problem of modality imbalance, existing methods [Wang et al., 2020; Li et al., 2023] predominantly adopt a deferred strategy for modality rebalancing, i.e., probing the strength at first and performing multimodal learning under the imbalanced status. Among existing methods, some of them, e.g., OGR-GB [Wang et al., 2020] and OGM [Peng et al., 2022], first explicitly calculate multimodal imbalance-related metrics like OGM score, and then make adjustments based on metrics. While others [Li et al., 2023; Zhang et al., 2024; Zong et al., 2024] leverage key quantities like gradients or logits that implicitly reflect the learning state of one modality to guide interventions in another. Regardless of the method, the rebalancing learning process is based on the current imbalanced status, rather than the balanced status after instantaneous adjustment. This deferred rebalancing strategy has an inherent limitation: it addresses modal imbalance only after it has occurred, mitigating its impact but failing to prevent the problem from arising in the first place.

Can we design an instantaneous modality rebalancing method? The answer is yes. We propose a simple twopass forward strategy: the first forward pass probes the modality strength and recalibrates the status weights instantaneously, while the second forward pass applies these calibrated weights to perform model learning under the balanced status. Specifically, we first utilize geodesic multimodal mixup [Oh et al., 2023], a powerful tool to capture heterogeneous representation within hypersphere, to establish the relationship across modalities with fusion representations. With this architecture, we can effortlessly adjust the modality strength between different modalities. Subsequently, we perform the first forward pass to probe the strength of each modality based on information entropy. Using this strength, we adjust the intervention intensity for different modalities in the mixup process. With this adjustment, the fusion of different modalities achieves a more balanced status. After that, the second forward pass is conducted to finalize the forward computation, loss calculation, gradient backpropagation, and parameter updates, all under a relatively balanced status. This two-pass forward process, involving evaluation followed by balanced learning, is consistently applied throughout the entire training procedure. To this end, this approach enables instantaneous multimodal rebalancing, maximizing the potential for improving the model's overall performance. Our contributions are listed as follows:

• A novel fusion representation strategy using geodesic multimodal mixup is proposed, enabling the dynamic adjustment of fusion weights based on modality strength.
• A novel two-pass forward strategy is proposed, which first probes the strength of modality imbalance and then facilitates multimodal learning under the balanced status in the second forward pass.
• A novel learning method that incorporates a two-pass forward approach into the entire training process is proposed, dynamically performing both the evaluation and balanced multimodal learning stages.
• Experiments reveal that our IPRM can achieve the best performance on various datasets by comparing with SOTA baselines.

2 RelatedWork

2.1 RebalancedMultimodal Learning

Recently, multimodal learning has been observed to be less effective than unimodal models for certain tasks such as multimodal classification [Wang et al., 2020; Peng et al., 2022; Zong et al., 2024]. To mitigate this gap and fully leverage complementary representations of different modalities, various approaches [Wang et al., 2020; Peng et al., 2022; Li et al., 2023; Hua et al., 2024; Zhang et al., 2024] have been proposed.

Several approaches attempt to address this problem from a theoretical standpoint. [Huang et al., 2021] demonstrates that leveraging multiple modalities leads to a lower population risk compared to utilizing only a subset of those modalities. [Huang et al., 2022] examines this issue through the lens of joint training. In multimodal learning, models corresponding to different modalities tend to compete during training, causing the encoder network to focus on learning only a subset of the modalities. [Wu et al., 2022] investigates the inherent greediness of deep models in multimodal learning scenarios and introduces an algorithm designed to balance the conditional learning rates across modalities during training.

Other attempts [Peng et al., 2022; Li et al., 2023; Hua et al., 2024; Zhang et al., 2024] focus on designing specific algorithms to address this issue. [Peng et al., 2022; Fan et al., 2023; Li et al., 2023] propose algorithms based on gradient modulation to balance the learning performance across different modalities. [Yao and Mihalcea, 2022] seeks to achieve consistency in learning speeds by dynamically adjusting the learning rates of models across different modalities. [Yang et al., 2025] defines an adaptive element-wise score function for parameter updates based on modal significance to rebalance optimization across modalities. [Zhang et al., 2024; Hua et al., 2024; Fan et al., 2024] design an alternating learning paradigm to enhance the interaction across different modalities during training.

2.2 Mixup

Mixup is a widely used data augmentation strategy proposed by [Zhang et al., 2018], aiming at improving model generalization and robustness. It generates virtual training samples by linearly interpolating between pairs of examples and their corresponding labels. Some variants of mixup have been proposed to improve the original mixup strategy. For instance, CutMix [Yun et al., 2019] proposes to cut and paste the patches into images to improve the robustness of model against input corruptions. Manifold mixup [Verma et al., 2019] design to improve the generalization of deep neural networks by applying linear interpolation not in the input space, but in the hidden representations of the model. Additionally, several mixup strategies [Fang et al., 2022; Oh et al., 2023] for multimodal data have also been proposed. Representative geodesic multimodal mixup [Oh et al., 2023] provides a powerful tool to establish the correlation relationship across modalities, capturing the heterogeneous representation within hypersphere.

3 Multimodal Learning

Without loss of generality, we use audio and video modalities as illustrative examples. Notably, IPRM can be readily extended to scenarios involving more than two modalities. We suppose that the multimodal data training set with K category labels is defined as: T = {(xi, yi)}Nt i=1 , where x = (xa , xv ) respectively denote the audio and video data points, Nt is the number of training data, and yi ∈ {1, · · · ,K} denotes the i-th category label.

Multimodal learning aims to train a model that maps data points into label space, and approximates its label as accurately as possible. In deep learning-based multimodal learning, deep neural networks serve as the foundational components of the model. Concretely, we employ ga and gv to denote the encoders for the audio and video modalities. Given xai and xvi , the representations can be calculated by:

where Θo denotes the parameters. Then we take a fusion function to obtain fusion representation:

where f denotes the fusion function like sum or averaging.

Following [Fan et al., 2023] and [Wei and Hu, 2024], we consider to minimize both unimodal and multimodal losses. Hence, we define the classifiers for unimodal and multimodal branches as ha, hv and h, respectively. The predictions can be calculated by:

where Φo and Φ denote the parameters of the classifier.

Then the unimodal and multimodal loss functions can be formed as:

where pik denotes the k-th element of pi, and 1condition denotes the indicator function, i.e., 1true = 1 and 1false = 0. The overall loss can be formed as:

Then, the gradient is computed through the backward process, and the parameters are updated accordingly.

4 Methodology

The architecture of IPRM is presented in Figure 1. We first introduce geodesic multimodal mixup to bridge the fusion representation. Then, we design a two-pass forward strategy to probe the strength of modality imbalance first and then perform rebalanced learning. Finally, a dynamic learning algorithm is proposed to impose the two-pass forward strategy into the entire training process.

4.1 Multimodal Fusion with GMM

Instead of linear mixup, we utilize geodesic multimodal mixup strategy to construct heterogeneous representations across modalities. Specifically, we first generate normalized representation for both audio and video modalities:

Then the geodesic multimodal mixup strategy is imposed on these vectors, and we can redefine the fusion function as:

where θ = arccos(⟨¯zai , ¯zvi ⟩) and λ is a parameter. Symbol ⟨·, ·⟩ denotes the inner product of two vectors.

After obtaining the fusion representation with geodesic multimodal mixup, we can further feed them into classification networks and calculate the prediction. Based on prediction, the multimodal loss is calculated correspondingly.

We use λ to control the influence strength of different modalities, thereby achieving the goal of characterizing intervention based on the strength of each modality.

4.2 Instantaneous Probe-and-Rebalance for MML

We then present the two-pass forward strategy in detail. This strategy can be divided into instantaneous probing phase and rebalanced learning phase.

Instantaneous Probing Phase: At the t-th iteration, given a batch of data points, we first utilize encoders to extract features. Then, during the instantaneous probing phase, we can probe the strength of modality imbalance based on multimodal and unimodal predictions after we obtain the fusion representation ¯zi and prediction ¯pi:

where λat denotes the weight controlling the strength of the modality before learning for audio, and λvt = 1 - λat .

Then, we utilize a widely used measure Kullback-Leibler (KL) divergence to evaluate the strength of each modality:

where P o and ¯ P denote the corresponding prediction distributions. KL divergence quantifies the distance between different distributions. A smaller value indicates that the two distributions are closer. Therefore, a modality with a large KL divergence is considered a weak modality, and its weight should be reduced during the hybrid enhancement process. Conversely, a modality with a small KL divergence is a strong modality, and its weight should be increased. Hence, we define the instantaneous strength weight of a specific modality based on the proportion of the KL divergence from another modality:

Algorithm 1: The IPRM learning algorithm.

Input : Training set T .

Output: Learned parameters of all models.

INIT Initialize parameters {Θa, Θv , Φ, Φa, Φv}, maximum iterations Mt, learning rate ηα, modality weight ωa 0 = ωv 0 = 0.5, λa1 and λv1 based on Eq. (4).

for t = 1 to Mt do

Sample a mini-batch data samples Tt. Calculate features based on ga(·) and gv(·).

/* The first forward phase. */

Calculate fused feature ¯z the by the first forward phase. Calculate the instantaneous strength score based on

Eq. (2).

Calculate the balanced weight based on Eq. (3).

/* The second forward phase. */

Calculate the prediction ¯p by the second forward phase. Calculate the gradients based on backward phase. Update the network parameters based on SGD. Update λat and λvt based on Eq. (4).

end

Rebalanced Learning Phase: As the instantaneous strength score accurately captures the imbalance degree among different modalities, we directly use this parameter to update the balanced weights for each modality at t-th iteration:

Then, ˆλat , ˆλvt are used to perform the second forward phase to obtain fusion representation under the balanced status:

The multimodal loss can be calculated by:

Accordingly, we derive the gradients of the parameters and update them to finish the model training.

After completing the learning in t-th iteration, we update the initial weights for the next iteration to adjust the intervention intensity between modalities accordingly. We utilize the exponential moving average (EMA) to update the weight:

Here, α is a hyper-parameter to tune the weight, and ωo 0 is initially set to be 0.5 in practice.

Learning Algorithm: Finally, we briefly summarize our algorithm. We train the model using the entire training dataset T in a mini-batch style. At t-th iteration, we first utilize encoders to extract features. And then we perform the first forward phase to obtain the instantaneous strength weight ωa t and ωv t . Based on instantaneous strength weight ωa t and ωv t , we can update the modality weight λat and λvt as ˆλat and ˆλvt . In the second forward phase, the models are updated under balanced status by using ˆλat and ˆλvt . The whole algorithm is summarized in Algorithm (1).

5 Experiments

5.1 Experimental Setup

Datasets: We utilize five datasets for experiments, i.e., CREMA-D [Cao et al., 2014], KSounds [Arandjelovic and Zisserman, 2017], NVGesture [Molchanov et al., 2016], IEMOCAP [Busso et al., 2008], and Sarcasm [Cai et al., 2019] datasets. Among these datasets, the CREMAD and KSounds datasets involve audio and video modalities. NVGesture consists of RGB, optical flow (OF), and Depth modalities. IEMOCAP is also a trimodal dataset, which contains audio, video and text modalities. Sarcasm dataset is an image-text dataset.

The CREMA-D dataset contains 7,442 clips from 91 actors. And it is divided into a training set with 6,698 samples and a testing set with 744 samples. The KSounds dataset is divided into a training set with 15K samples, a validation set with 1.9K samples, and a testing set with 1.9K samples. For NVGesture dataset, it is split as 1,050 data points for training and 482 for testing. And IEMOCAP dataset is split as a training set with 3,318 samples and a testing set with 1,107 samples. Sarcasm dataset consists of 24,635 and is split as a training set with 19,816 samples, a testing set with 2,409, and a validation set with 2,410 samples.

Baselines: Considering that geodesic multimodal mixup can be treated as a modified fusion function, we first utilize unimodal approaches and naive fusion strategies including Concat, Sum and Weight for comparison. Furthermore, a wide range of SOTA rebalanced multimodal learning approaches are used for comparison. They are OGR-GB [Wang et al., 2020], MSLR [Yao and Mihalcea, 2022], OGM [Peng et al., 2022], PMR [Fan et al., 2023], AGM [Li et al., 2023], MMPareto [Wei and Hu, 2024], Reconboost [Hua et al., 2024], MLA [Zhang et al., 2024], and LFM [Yang et al., 2024a].

Evaluation Metrics: Following [Zhang et al., 2024; Yang et al., 2024a], the accuracy and mean average precision (MAP) are used to evaluate the performance on CREMA-D and KSounds datasets. Furthermore, for the remaining datasets, we utilize accuracy and macro F1 (Macro-F1) for evaluation. The accuracy evaluates the degree of agreement between predictions and ground-truth labels. The MAP is obtained by averaging the average precision across all categories, while the macro-F1 is determined by computing the mean of the F1 scores for each category.

Implementation Details: Following [Peng et al., 2022], we adopt ResNet18 [He et al., 2016] as backbones for both audio and video modalities on CREMA-D and KSounds datasets. And the models are trained from scratch. For NVGesture dataset, we use I3D [Carreira and Zisserman, 2017] as unimodal backbone, as described in [Wu et al., 2022]. Following [Zhang et al., 2024], we respectively employ large pretrained model M3AE [Geng et al., 2022] and CAVMAE [Gong et al., 2023] as encoders for image/text and audio modalities on IEMOCAP dataset. For Sarcasm, ResNet50 is utilized for the image modality and BERT [Devlin et al., 2019] for the text modality, which is consistent with [Yang et al., 2024a]. The optimization algorithm for the audio-video and trimodal datasets is stochastic gradient descent (SGD), while Adam is employed for the image-text dataset. The learning rate is set to 10-2 for the audio-video datasets and NVGesture, 10-3 for IEMOCAP, and 10-4 for Sarcasm, respectively. It is then reduced by a factor of 10 when the loss saturates. The batch size is set to be 64 for CREMA-D, KSounds and Sarcasm, while is respectively set to be 2 and 16 for NVGesture and IEMOCAP due to out-of-memory issue. Furthermore, the hyper-parameter α is set to 0.8 for audiovideo datasets and 0.7 for trimodal and image-text datasets, based on cross-validation strategy. Unless otherwise specified, the mixup in our experiments all uses the paired sample method, and we will discuss the impact of this strategy in subsection 5.5. All experiments are conducted on GeForce RTX 4090 NVIDIA card.

5.2 Main Results

We first compare our method with unimodal approaches and naive multimodal learning with different fusion functions, i.e., concat, sum, and weight [Yang et al., 2019]. The results are reported in Table 1, where "A/A/R/A/I" and "V/V/O/V/T" denote the abbreviations of the modalities used for evaluation across all datasets, and "D/T" respectively denote the abbreviations of depth (for NVGesture dataset) and text (for IEMOCAP dataset). The results in Table 1 demonstrate that: (1). The multimodal learning approach outperforms the singlemodal approach in most cases but performs worse in certain scenarios, which is indicated by the symbol "↓"; (2). Our IPRM can achieve the best results in all cases by a large margin, demonstrating that it is not merely a simple fusion variant but an algorithm specifically designed for the practical requirements of multimodal learning.

Then we compare our IPRM with various existing SOTA rebalanced multimodal learning approaches. Table 2 provides the corresponding results, where the results of OGM and PMR on NVGesture and IEMOCAP datasets are denoted as "N/A" as these approaches cannot be applied to the case with three modalities. The results in Table 2 demonstrate that our IPRM surpasses the existing state-of-the-art algorithm and achieves superior performance in almost all cases, demonstrating that the module we designed effectively addresses the challenge of modality imbalance.

5.3 Ablation Study

We study the impact of key components of our IPRM, including geodesic multimodal mixup, EMA strategy, and two-pass forward phase. Specifically, we report the accuracy to compare the IPRMwith its variants. The variants include the vari-

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Figure 2: Sensitivity to α.

Figure 3: Unpaired GMM.

ant using linear mixup, the variant without EMA strategy, and the variant with one-pass forward phase, which are denoted as "w/ L-Mixup", "w/o EMA", and "One-Pass", respectively. For One-Pass variant, the loss calculation, gradient calculation by backward, and parameter update are performed in the first forward phase, and the ωa t and ωv t are used to update λat+1 and λvt+1 at the next iteration.

The accuracy on all datasets is reported in Table 3. The results reveal that both geodesic multimodal mixup, EMA strategy and two-pass forward phase can boost performance. On CREMA-D, KSounds, and IEMOCAP datasets, the most significant performance improvement is attributed to the use of geodesic multimodal mixup. In contrast, for the remaining datasets, the module contributing the greatest impact on performance is two-pass forward phase.

5.4 Sensitivity to Hyper-Parameter

We further explore the sensitivity to the hyper-parameter α. Parameter α ∈ (0, 1) is the weight of the EMA strategy in Equation (4). The accuracy on all datasets is reported in Figure 2, where α is chosen from the list [0.1, 0.2, · · · , 0.9]. The accuracy in Figure 2 demonstrates that IPRM is not sensitive to the hyper-parameter α in a large range.

5.5 Further Analysis

GMM with Unpaired Sampling: In IPRM, we utilize paired multimodal data to perform model training. Thus we do not discuss the fusion strategy for category label when we apply the geodesic multimodal mixup [Oh et al., 2023] in our method. In practical scenarios, unpaired multimodal data may often be used for training. Therefore, this section discusses the impact of paired and unpaired sampling strategies on the overall performance. For geodesic multimodal mixup, given an unpaired multimodal data {(xi, yi), (xj, yj)}, the augmented ground-truth can be formulated by linear combination, i.e., y′ = λyi + (1 - λ)yj. We exploit the influence of geodesic multimodal mixup with unpaired sampling. The accuracy on all datasets is presented in Figure 3. From Figure 3, we can see that across all datasets, the overall performance of the unpaired sampling method is slightly inferior to that of the paired sampling method. This performance gap can be attributed to three factors: (1). Unpaired sampling disrupts the exploration of complementary multimodal information; (2). Mixing labels introduces additional noise into the training process; (3) Unlike standard mixup, IPRM's λ is not randomly generated from a distribution, which may shift the distribution of the generated labels towards the stronger modality.

Computation Cost of Two-Pass Forward: As IPRM adopts a two-pass forward strategy, we compare the computation cost to illustrate the influence of training time. We report the accuracy and computation cost on CREMA-D for naive MML, MLA, LFM, and IPRM. The results in Table 4 show that IPRM exhibits slightly slower training compared to naive MML. However, it achieves the best performance among all methods. Notably, the IPRM's algorithmic complexity remains linear with respect to the sample size N, i.e., O(N), reflecting its linear scalability. During inference, IPRM follows the standard MML inference procedure and does not introduce any additional computational overhead.

GMM Strategy for Trimodal Dataset: For trimodal datasets, we employ the GMM strategy between any two modalities to achieve modal fusion. Consequently, our structure incorporates three fused classification heads. For instance, in NVGesture dataset, the classification heads correspond to RGB and OF, RGB and depth, as well as OF and depth. However, this structure is somewhat cumbersome. Therefore, we explore an alternative strategy to simplify the design of classification heads for trimodal settings.

Specifically, we design random modality sampling strategy for trimodal dataset. In the architecture, we only utilize one classifier for all three modalities. During the forward phase, we randomly select two modalities to perform modality fusion. We compare these two strategies on trimodal datasets, i.e., NVGesture and IEMOCAP datasets. The results in Table 5 demonstrate that the method based on the random modality sampling strategy achieves comparable results, where the random modality sampling strategy and the strategy adopted in the paper are respectively denoted as "SingleCLS" and "Tri-CLS". Therefore, in practice, a specific architecture or strategy can be selected based on the requirements. Fine-Tuning with Pretrained CLIP: We further study the applicability of IPRM for large-scale language vision pretrained models like CLIP [Radford et al., 2021] on Sarcasm dataset. The encoders for image and text modalities are replaced by the ViT-B/16 and Transformer from the CLIP

model. Then the fine-tuning process is performed to fit the downstream task. The accuracy performance is reported in Table 6, where we utilize "CLIP+Method" to denote the specific Method to train the models with replaced encoders. According to the results in Table 6, we can see that: (1). Comparing the results of fine-tuning downstream tasks using the rebalanced MML method with those based on the CLIP model reveals that fine-tuning on downstream tasks leads to improved model performance; (2). IPRM can achieve the best accuracy in almost all cases by comparing IPRM with LFM, MLA, and naive CLIP.

Visualization: To study the properties of the learned embeddings for different methods, we present the visualization results by using t-SNE [Van der Maaten and Hinton, 2008] on CREMA-D dataset. We select three methods for comparison, including the method with linear mixup (denoted as LMixup), MLA, LFM and IPRM.

The t-SNE visualzation results on CREMA-D dataset are presented in Figure 4, where we also present the average distance of each multimodal data sample to its respective cluster center in each sub-figure. We can find that our IPRM can achieve smaller distance compared with the remaining approaches, demonstrating that IPRM effectively learns discriminative representations. This effect is particularly pronounced in the video modality, i.e., the weak modality for CREMA-D dataset, demonstrating that IPRM more effectively addresses the modality imbalance issue.

6 Conclusion

In this paper, we propose a novel instantaneous probe-andrebalance multimodal learning approach, namely IPRM, by employing two-forward phase strategy to capture the learning status instantaneously. We utilize geodesic multimodal mixup to probe the strength using dynamic weight. The weights are subsequently recalibrated instantaneously based on the evaluated strength, enabling balanced training through an additional forward pass. This dynamic adjustment is performed continuously throughout the training process. Experiments demonstrate that our IPRM can achieve the best performance on widely used datasets comparing numerous state-ofthe-art baselines. In future reseach, related extensions, such as integrating IPRM with attention mechanisms and adapting IPRM to streaming multimodal data scenarios, remain promising for further improving the efficiency and scalability of the approach.

Acknowledgments

This work is supported by the National Key RD Program of China (2022YFF0712100), NSFC (62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20240081).