Balance-Aware Sequence Sampling Makes Multi-Modal Learning Better

1 Introduction

Multi-modal learning has emerged as a prominent research area in artificial intelligence across various scenarios [Yin et al., 2021; Xu et al., 2023; Yang et al., 2021], including speech recognition [Hu et al., 2023], information retrieval [Yang et al., 2024b], and recommender systems [Ye et al., 2025]. By integrating information from diverse sensors, MML has become a driving force in improving performance across these applications. Despite these promising outcomes, MML faces a significant challenge: modality imbalance. Specifically, the inherent heterogeneity of data endows each modality with distinct properties, such as convergence speed [Peng et al.,

∗Corresponding authors.

2022]. As a result, the learning process tends to be dominated by the stronger modality (i.e., the one that converges faster) during joint training, which can lead to insufficient learning of other modalities. In extreme cases, this imbalance may even cause the multi-modal model to perform worse than its best unimodal counterpart [Wang et al., 2020].

Recently, many impressive studies have been proposed to address the modality imbalance problem from various perspectives [Du et al., 2021; Peng et al., 2022; Li et al., 2023; Yang et al., 2024a; Yang et al., 2025]. Considering the inherent modal differences, a straightforward idea is to manually control the optimization process between strong and weak modalities to achieve rebalancing, such as learning rate adjustment [Yao and Mihalcea, 2022] and gradient modulation [Fan et al., 2023; Peng et al., 2022]. Other approaches attempt to facilitate multi-modal learning through neural architecture design [Du et al., 2021; Xiao et al., 2020]. Although these optimization- and architecture-based methods have shown promising results, they generally overlook an important aspect: MML can be highly sensitive to the sequence in which training samples are presented at different stages. This motivates us to investigate the role of sample sequences in addressing modality imbalance.

Since the standard training paradigm is characterized by random data shuffling, this process inevitably introduces imbalanced samples into the early training stages (Figure 1 (a1)), which may further exacerbate modality imbalance and ultimately degrade overall performance. To support our viewpoint, we conduct a toy experiment on the Twitter2015 dataset to investigate the relationship between different training sequences and MML performance. Inspired by curriculum learning (CL) [Wang et al., 2022; Soviany et al., 2022], we first evaluate the balance degree of sample pairs based on both correlation criteria (e.g., prediction similarity) and information criteria (e.g., training loss), and then rank them to construct new training sequences (Figure 1 (a2)). The comparison results in Figure 1 (b) reveal an interesting phenomenon: CL effectively boosts MML performance, while anti-CL (i.e., learning from imbalanced to balanced samples) leads to performance degradation across all criteria. This experiment suggests that introducing balanced samples in the early training stages can guide the model toward a more stable and robust optimization path, thereby enhancing overall performance.

Based on our findings, in this paper, we attempt to address the modality imbalance by adjusting the sample sequences, a training paradigm that provides appropriate training samples to the model at different stages. Concretely, we first design a multi-perspective measurer from both correlation and information criteria to evaluate the balance degree of each sample. Via sample evaluation, we propose a heuristic scheduler that progressively constructs training sequences in a balanced-toimbalanced manner. Moreover, considering that the heuristic scheduler is relatively coarse and may neglect feedback from the current model, we propose a learning-based scheduler that dynamically reconstructs training sequences by assigning sampling probabilities to each data point (Figure 1 (a3)), further enhancing MML performance as shown in Figure 1 (c). To sum up, our contributions are outlined as follows:

• We highlight the critical role of training sequences in addressing modality imbalance, and show that wellstructured sequences can significantly improve MML performance.
• We define a multi-perspective measurer to quantify the balance degree of each sample. Based on the resulting balance scores, we then propose both a heuristic and a learning-based sampling method to adjust the training sequences.
• Extensive experiments demonstrate that our proposed method outperforms existing baselines and achieves SOTA performance across widely used datasets.

2 RelatedWork

2.1 ImbalancedMulti-Modal Learning

Recent research [Peng et al., 2022; Huang et al., 2022] has shown that many multi-modal models fail to outperform the best unimodal counterpart. This phenomenon is attributed to modality imbalance [Fan et al., 2024; Wei et al., 2024b], where each modality cannot be fully learned due to inhibition between them. Considering the existence of both strong and weak modalities, several representative [Wang et al., 2020; Fan et al., 2023; Zong et al., 2024] methods focus on balancing the optimization of individual modalities. In particular, OGM [Peng et al., 2022] introduces an on-the-fly gradient modulation technique, which adaptively adjusts the optimization process for each modality by monitoring the discrepancy in their contributions to the learning objective. PMR [Fan et al., 2023] uses prototypes to control the update direction for improved unimodal performance. Other studies [Du et al., 2021; Wu et al., 2022] attempt to boost MML performance by introducing supplementary modules. For instance, UMT [Du et al., 2021] trains the multi-modal model with knowledge distillation [Gou et al., 2021] from well-learned teacher encoders to obtain richer unimodal representations. However, these methods increase model complexity and training costs. In this paper, from the perspective of sample sequences, we address modality imbalance by guiding the model to progressively learn training samples in a balanced-to-imbalanced manner, without the need for additional modules.

2.2 Sequence-orientedMulti-modal Learning

Sequence-oriented MML is crucial in machine learning, enabling models to train on a meaningful subset derived from the original dataset distribution. These strategies are primarily applied in two areas: curriculum learning (CL) [Soviany et al., 2022] and noisy label learning (NLL) [Patel and Sastry, 2023]. CL is a training paradigm that progresses from easier samples to harder ones. By guiding the model toward a better parameter space, CL has been widely adopted across various fields, including large language models [Wang et al., 2024], action recognition [Tong et al., 2023], and reinforcement learning [Narvekar et al., 2020]. A typical curriculum system consists of two main components: a difficulty measurer to evaluate the learning difficulty of samples and a scheduler to manage the assignment of training subsets. On the other hand, sequence-oriented MML in NLL focuses on selecting clean samples from a noisy training set for model learning. Commonly used criteria for identifying noisy labels, such as training loss [Wei et al., 2020], Jensen-Shannon divergence [Xu et al., 2025], and representation similarity [Ortego et al., 2021], facilitate reliable data selection and ultimately enhance model robustness. Inspired by the core idea of sequence-oriented MML, we prioritize balanced samples to address modality imbalance. This strategy helps rebalance the training process, enabling our method to learn robust feature representations while avoiding early-stage optimization dilemmas.

3 Methodology

In this section, we present our proposed method in detail. The overall architecture is shown in Figure 2, which consists of two main components: a multi-modal training framework for learning representations, and a Balance-aware Sequence Sampling (BSS) module for rebalancing MML via a multi-perspective measurer and two optional schedulers (one heuristic and the other learning-based).

3.1 Preliminary

Without loss of generality, we consider a multi-modal sample with u and v modalities. Formally, let D = {X, Y } denote the dataset, where X = {x(u) i , x(v) i }ni=1 represents n training samples and Y = {yi | yi ∈ {0, 1}c }ni=1 is the corresponding category labels with a total of c categories. MML aims to train a model to predict the category label of a given multimodal sample.

For the MML prediction task, we typically employ deep neural networks to learn feature embeddings of each modality from the original space. We use f (j) (·) as the feature extractor for j-th modality, j ∈ {u, v}. Given a sample x(j) i , the feature extraction can be expressed as:

where e(j) i ∈ Rd denotes the d-dimension feature embedding of x(j) i , the θ(j) denotes the learnable parameters of j-th encoder. After extracting feature embeddings for all modalities, we adopt a fusion function g(·) to fuse them, followed by a classifier (e.g., a fully connected layer) to map the feature embedding to Rc . This procedure can be formulated as:

Here, W ∈ Rc×D , b ∈ Rc denote the weights and bias of the classifier, respectively, and D denotes the dimension of ei. Then, the objective function of multi-modal learning can be formulated as:

Considering that MML can benefit from the supervision of unimodal predictions [Zou et al., 2023], we jointly optimize both multi-modal and unimodal objectives. Therefore, the final objective function can be reformulated as:

where α denotes the weighted parameter between the losses.

3.2 Multi-perspective Measurer

To construct well-structured training sequences that address modality imbalance, we next introduce how to measure the balance degree of a multi-modal sample from the perspectives of correlation and information criteria.

Correlation Criterion: Different modalities exhibit inherent cross-modal correlation, as they describe the same concept via diverse representations, capturing complementary information. Although cross-modal correlation can be measured from various aspects, we focus on the most commonly used and critical criterion, i.e., prediction similarity. The reason is that prediction similarity directly measures output consistency. Formally, given a sample xi = {x(u) i , x(v) i }, the prediction similarity is defined as:

Here, ∥ · ∥2 denotes L2 norm of the unimodal predictions. Information Criterion: While prediction similarity reflects the balance between modalities, it does not verify whether the predictions of each modality are correct. In other words, high prediction similarity may still occur even when all modalities produce incorrect predictions. Therefore, we further introduce label-related training loss as an intuitive metric to evaluate the learning difficulty of each sample.

Balance Score: We denote the sets of prediction similarity and training loss computed over all n training samples as S = [sim(x(u) 1 , x(v) 1 ), . . . , sim(x(u) n , x(v) n )] and L = [ℓtotal(x(u) 1 , x(v) 1 , y1), . . . , ℓtotal(x(u) n , x(v) n , yn)], respectively. Hence, the balance score of sample xi = {x(u) i , x(v) i } can be formulated as the combination of correlation criterion and information criterion:

This normalization ensures that both terms lie on a comparable scale. A higher score s(xi) ∈ [-1, 1] indicates a better balance degree.

3.3 Training Scheduler

After evaluating the balance score of each sample, we proceed to control the presentation order of training data from balanced to imbalanced samples, i.e., the sample sequence for each training epoch.

Similar to human education, if teachers impart knowledge all at once, students may become overwhelmed and fail to learn effectively. On the other hand, if teachers focus too much on basic knowledge, students may lose motivation. In the following, we introduce a coarse but efficient heuristic scheduler and a more refined, effective learning-based scheduler for constructing sample sequences.

Heuristic Scheduler: Inspired by curriculum learning [Wang et al., 2022; Soviany et al., 2022], we rank the training samples from balanced to imbalanced according to the defined balance score, and then employ a pace function [Hacohen and Weinshall, 2019] to determine the number of samples included in the training set at each epoch. In practice, various pacing functions exist, such as the baby step [Bengio et al., 2009], linear function [Wang et al., 2022], and root function [Platanios et al., 2019]. However, the impact of existing pacing functions on modality imbalance is not the focus of our work. Here, we adopt a widely used root function λ(t) to achieve this:

where Tgrow represents the training epoch when this function first reaches 1, and λ0 ∈ (0, 1] is the initial proportion of the training samples. λ(t) maps the training epoch t to an interval λ ∈ (0, 1], which means λ proportion of the most balanced samples are available at t-th epoch. This function starts at λ(0) > 0 and ends at λ(Tgrow) = 1.

From Equation 7, the pace function serves as a threshold that progressively expands the sampling space during training. At each epoch t, the current batch data Xbatch is randomly sampled from the top λ proportion of training data in the entire ranked sequence Xrank:

where n denotes the number of training samples. Thus, the heuristic scheduler allows the model to focus on balanced samples during the early training stages and gradually broaden the learning scope by incorporating those imbalanced ones. Please note that the sample evaluation is performed only once before model training, which means Xrank is a fixed sequence.

Learning-based Scheduler: Despite the simplicity and efficiency of the heuristic scheduler in practice, it has one main limitation: the fixed training sequence is coarse-grained, which may neglect feedback from the current model and potentially lead to inaccurate sample evaluation. Therefore, we further propose a learning-based scheduler that flexibly addresses the above limitation. This scheduler reconstructs the dynamic sequence by learning a sampling probability for each data point, considering both the balance of past and current samples in a more fine-grained manner.

Specifically, we refer to the balance score s(xi) in Equation 6 and update it in a certain epoch interval, using E for short. Subsequently, the k + 1-th balance score can be expressed as:

where k = ⌊t/E⌋, t denotes the t-th epoch, β is an adjustment parameter, and s1 is the balance score obtained before model training, i.e., the initial evaluation results.

According to the updated balance scores, the learned sampling probability p for each data point xi in the t-th epoch can be computed using the softmax operation:

Finally, in the t-th epoch, each data point xi is sampled with probability p(xi) to construct the current batch data Xbatch, without replacement. This process is formulated as:

Hence, training data with higher sampling probabilities (i.e., more balanced ones) are preferentially selected for the mini-batch in each epoch.

Discussion: Our proposed method aims to address the modality imbalance problem through sequence sampling in a balanced-to-imbalanced learning manner. Thus, our BSS can be integrated as a model-independent plugin into most existing MML approaches.

3.4 Model Inference

After training, the learned model can be applied for prediction during the inference stage. Following [Fan et al., 2024; Zhang et al., 2024], we adopt a simple weighted combination of logits output from each modality and their fusion, represented as ztotal = zmulti + ∑j∈{u,v} z(j) uni . Subsequently,

Input : Training set Xtrain, category labels Y train. Output: Learned parameters θ of all models. INIT Initialize parameters θ0 , maximum epochs T , training set for ranking Xrank = ∅, curriculum period Tgrow, initial proportion λ0, epoch interval E. /* Calculate the balance score via measurer. */ for each sample xi in Xtrain do Obtain balance score s(xi) based on Equation 6. Add xi toXrank in descending order of s(xi). end /* Train model using sample sequences from scheduler. */ for t = 0 to T - 1 do if scheduler == 'heuristic' then Calculate the proportion of the training samples λ(t) with Equation 7. Obtain current batch data Xbatch fromXrank with Equation 8. else if scheduler == 'learning-based' then Update s(xi) every E epochs with Equation 9. Assign sampling probability p(xi) based on s(xi) with Equation 10. Obtain current batch data Xbatch with Equation 11. end Train model with Xbatch and update parameters θ. Update t = t + 1. end

the predicted category ˆ y for a given unseen multi-modal sample can be denoted as:

4 Experiments

4.1 Experimental Setup

Datasets: We validate our proposed method on six widely used datasets, including CREMA-D [Cao et al., 2014], Kinetics-Sounds [Arandjelovic and Zisserman, 2017], VGGSound [Chen et al., 2020], Twitter2015 [Yu and Jiang, 2019], Sarcasm [Cai et al., 2019], and NVGesture [Molchanov et al., 2016]. Among them, CREMA-D, Kinetics-Sounds, and VGGSound contain both audio and video modalities. CREMA-D includes 7,442 video clips across six emotional categories, with 6,698 clips for training and 744 for testing. Kinetics-Sounds is categorized into 31 distinct actions, split into 15,000 for training, 1,900 for validation, and 1,900 for testing. VGGSound provides 168,618 videos for training and validation, along with 13,954 videos for testing. Moreover, Twitter2015 and Sarcasm datasets involve both image and text modalities. Twitter2015 comprises 5,338 text-image pairs, divided into 3,179 for training, 1,122 for validation, and 1,037 for testing. Sarcasm contains 24,635 text-image pairs, allocated as 19,816 for training, 2,410 for validation, and 2,409 for testing. Lastly, NVGesture features three modalities, i.e., RGB, optical flow (OF), and Depth, with 1,050 samples for training and 482

samples for testing.

Baselines and Evaluation Metrics: We conduct a comprehensive comparison of BSS with two types of baselines: vanilla fusion methods and multi-modal rebalance approaches. The former includes Concat, Affine [Perez et al., 2018], Channel, ML-LSTM [Nie et al., 2021], Sum, Weight, and ETMC [Han et al., 2023]. The latter comprises MSES [Fujimori et al., 2019], OGR-GB [Wang et al., 2020], DOMFN [Yang et al., 2022], OGM [Peng et al., 2022], MSLR [Yao and Mihalcea, 2022], AGM [Li et al., 2023], PMR [Fan et al., 2023], ReconBoost [Hua et al., 2024], MMPareto [Wei and Hu, 2024], SMV [Wei et al., 2024a], MLA [Zhang et al., 2024], and AMSS [Yang et al., 2025].

Following [Peng et al., 2022; Hua et al., 2024], we utilize accuracy (ACC), mean average precision (MAP), and Macro F1-score (Mac-F1) as evaluation metrics. ACC measures the ratio of correct predictions to total predictions. MAP reflects the average precision across all samples, while Mac-F1 computes the average of F1 scores across all categories.

Implementation Details: Following [Fan et al., 2023], for audio-video datasets, we use ResNet18 [He et al., 2016] as the backbone to encode each modality. For text-image datasets, we employ ResNet50 for images and BERT [Devlin et al., 2019] for text processing. For the trimodal dataset NVGesture, we follow the setup of [Wu et al., 2022] and adopt the I3D [Carreira and Zisserman, 2017] as the unimodal backbone. To ensure fairness, all methods use the same backbone during training. The optimizer for audio-video datasets is stochastic gradient descent (SGD) with a momentum of 0.9 and weight decay of 10-4 . The initial learning rate is set to 10-2 and is reduced by a factor of 10 when the loss saturates. The batch size is set to 64 for CREMA-D and KineticsSounds, 16 for VGGSound, and 2 for NVGesture. For textimage datasets, we employ the Adam optimizer starting with a learning rate of 10-5 , with a batch size of 64. Furthermore, the hyperparameters α and β are set to 0.2 and 0.6, respectively. For the training scheduler, the curriculum period Tgrow and the initial proportion λ0 are configured as 40 and 0.1 under the heuristic setting, while the epoch interval E is configured as 5 under the learning-based setting. All models are trained on an NVIDIA GeForce RTX 3090 GPU.

4.2 Comparison with SOTAMML Baselines

We conduct comprehensive comparisons to assess the superiority of our proposed method in addressing the imbalanced MML problem. The classification performance across all datasets is reported in Table 1 and Table 2, where "BSS-H" and "BSS-L" denote the proposed method with the heuristic scheduler and learning-based scheduler, respectively. Please note that "-" in Table 1 denotes that the corresponding methods are not applicable to the respective datasets.

Results on Bimodal Dataset: Referring to the first four datasets in Table 1, we derive the following key observations: (1). Unimodal performance may outperform multimodal joint training. For instance, the text-modal performance on the Twitter2015 dataset is obviously better than most vanilla fusion methods, indicating an inhibitory relationship between different modalities. (2). Most multi-modal rebalance approaches demonstrate significant improvements over vanilla fusion methods. This phenomenon not only underscores the adverse impact of modality imbalance on performance but also validates the effectiveness of the multimodal rebalance approach. (3). Compared to all baselines, including both vanilla fusion methods and multi-modal rebalance approaches, our proposed method achieves the best performance by a large margin across all metrics. It can be observed that BSS-L delivers significant performance improvements on both the CREMA-D and Kinetics-Sounds datasets. After sequence sampling, our method surpasses the best baseline (MLA) [Zhang et al., 2024] with gains of 3.37%/2.89% and 3.91%/5.30% in ACC and MAP metrics, respectively.

Results on Trimodal Dataset: In addition, we present a comparison with SOTA baselines on the NVGesture dataset. As shown in the last dataset of Table 1, unlike multi-modal rebalance approaches limited to scenarios with only two modalities (e.g., OGM [Peng et al., 2022] and PMR [Fan et al., 2023]), our method effectively tackles the challenges in scenarios involving more than two modalities and achieves the best performance.

Results on Large-scale Dataset: To further evaluate the generality of our method, we conduct experiments on the largescale VGGSound dataset. Given the size of the dataset, we select a few representative baselines for comparison, including OGM, AGM [Li et al., 2023], ReconBoost [Hua et al., 2024], MMPareto [Wei and Hu, 2024], SMV [Wei et al., 2024a], and MLA. The results in Table 2 consistently demonstrate that our BSS-L achieves superior performance.

4.3 Ablation Study

We conduct ablation studies to verify the effectiveness of using different criteria for sample evaluation, namely unimodal prediction similarity (PreSim) and training loss (Loss). Table 3 presents the results under the learning-based setting, which reveal that: (1). Vanilla training may exacerbate modality imbalance. For instance, when the video modality converges, the audio modality remains insufficiently trained, leading to a significant gap between the two modalities (4.66%/6.41% in ACC/MAP). (2). Both "PreSim" and "Loss", when employed, can boost classification performance. (3). By integrating "PreSim" and "Loss", BSS-L achieves the best performance. This is predictable, as prioritizing balanced samples based on correlation and information criteria helps narrow the gap between modalities, facilitating both unimodal and multi-modal learning processes.

4.4 Further Analysis

Sensitivity to Hyperparameters: In calibrating our proposed method, we identify two hyperparameters: α in Equation 4 and β in Equation 9, determining the strength for balancing classification loss and regulating the balance score, respectively. Figure 3 (a) depicts the performance of different α. As α increases, the accuracy of our method first increases and then decreases. This shows that proper unimodal learning has a promoting effect, but over-considering unimodal optimization may hinder multi-modal interactions. From Figure 3 (b), we can find that the performance is marginally affected by β, highlighting the insensitivity of our method to hyperparameters. Despite some fluctuations, our method still demonstrates excellent effectiveness, i.e., being consistently better than baseline vanilla MML.

Robustness of the Pre-trained Model: We further explore the robustness of the large pre-trained model on text-image datasets. We replace each modality encoder with the corresponding encoder pre-trained by CLIP [Radford et al., 2021] and fine-tune the model. The results are shown in Figures 3 (c) and (d), where "CLIP+MLA" and "CLIP+Ours" represent the use of MLA and our approach, respectively. From the results, we can draw the following observations: (1). Both "CLIP+MLA" and "CLIP+Ours" outperform CLIP in all cases. (2). Via sequence sampling, our method achieves better performance than MLA.

Case Study: We investigate whether our method can effectively distinguish between balanced and imbalanced samples in a randomly ordered sequence. From the representative samples in Figure 4, we observe that balanced samples exhibit strong semantic consistency between modalities, as indicated by high balance scores, while imbalanced samples typically display weak semantic connections or irrelevant information.

(c). Twi�er2015.

(d). Sarcasm.

Figure 4: Qualitative results of sample evaluation. We present some representative samples selected from different segments (early, middle, and late) of the training sequence after evaluation under the heuristic setting.

5 Conclusion

In this paper, we propose a novel multi-modal learning method called Balance-aware Sequence Sampling (BSS). By defining a multi-perspective measurer, we evaluate the balance score of each sample. Via this evaluation, we design a heuristic and a learning-based scheduler to construct sample sequences for the model at different training stages. As a result, BSS addresses modality imbalance through a balancedto-imbalanced learning strategy, thereby enhancing MML performance. Furthermore, BSS can be integrated as a modelindependent plugin into most existing MML approaches. Extensive experiments on widely used datasets demonstrate the superiority of BSS over SOTA baselines.

Acknowledgments

This work is supported by the National Key RD Program of China (2022YFF0712100), NSFC (62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20240081), National Key Laboratory of Information Systems Engineering (05202403), Fundamental Research Funds for the Central Universities (No. 30925010205).