Disentangling Multi-view Representations via Curriculum Learning with Learnable Prior

1 Introduction

Multi-view or multi-modal data such as image, text, and audio are appealing yet challenging for real-world applications. The data often contains consistent information across all views and complementary information for others compared to single-view data. Multi-view representation learning aims at learning latent representations from multi-view data and later uses them for downstream tasks, e.g., text generation [Ju et al., 2021], anomaly detection [Wang et al., 2023], and cross-modal retrieval [Ma et al., 2024].

Over the years, various multi-view representation learning methods have been proposed. These methods generally fall into two categories: model-based multi-view methods [Xu et al., 2022; Yan et al., 2023; Zou et al., 2024] and informationtheory-based methods [Shi et al., 2019; Federici et al., 2020; Sutter et al., 2024]. Among them, Federici et al. [2020] extended the information bottleneck principle to multi-view representation learning. Xu et al. [2022] proposed a multilevel (low-level features, high-level features, and semantic labels) representation learning framework for multi-view data. Yan et al. [2023] presented a cross-sample and cross-view aggregation method for representations of multi-view data. Sutter et al. [2024] designed a data-dependent multimodal variational prior for guiding the latent representation of each modal towards a shared aggregate posterior. However, despite the advances in these methods, there is a commonly overlooked issue to be resolved for multi-view representation learning, i.e., the intrinsic differences between viewconsistency learning and view-specificity learning.

∗Corresponding author.

Multi-view data usually come from diverse sources or views. View-consistent representation mainly extracts common correlations across all views, and view-specific representation mines exclusive information within an independent view. Therefore, there are intrinsic differences between viewconsistent and view-specific representation learning, i.e., the difficulty of view-consistency learning is lower than that of view-specificity. Due to the intrinsic differences, the same learning strategy may not work simultaneously for viewconsistency and view-specificity learning. However, most existing methods ignore such intrinsic differences and adopt the same learning strategy for them, and finally result in a secondbest performance. Moreover, neural networks have an important property that neural networks initially memorize simple patterns and then gradually memorize hard patterns. That is, neural networks initially focus on learning simple features of the data. As training progresses, the networks start to gradually model hard and complex features. Applying a simultaneously uniform learning strategy to both view-consistency (i.e., simple patterns) and view-specificity (i.e., hard patterns) would conflict with the inherent property of neural networks and then degrade the efficacy of data on downstream tasks.

To deal with the aforementioned issues, we propose to disentangle view-consistency and view-specificity, and devise an innovative Curriculum Learning model with Learnable Prior (denoted as CL2P). As shown in Figure 1, curriculum learning has scheduled parameters over epochs to adjust the model first to learn simple view-consistent representations and then to learn hard view-specific representations progressively, which we call progressive curriculum learning. Such a learning manner exactly fits this property of neural networks. Meanwhile, to enhance view-consistency, we saddle the model with a novel Mixture-of-Experts (MoE) layer. The MoE layer boosts the fusion of multi-view data. To enhance view-specificity, we further introduce a learnable prior. The learnable prior allows view-specific representation from each view to fit its optimal distribution. Moreover, to avoid interference between view-consistency and view-specificity, we delve into mutual information and propose to minimize the upper bound of it to disentangle view-consistent and viewspecific representations.

In summary, we make the following contributions:

• We explore a well-known property of neural networks that neural networks fit simple patterns first and then hard ones. To our knowledge, this is the first work to investigate this property of neural networks in the context of multi-view representation learning.
• We propose a novel curriculum learning model called CL2P. The CL2P exhibits an innovative amalgamation of curriculum learning, mixture-of-experts learning, learnable prior, representation disentangling, and a joint loss function in wrapping up all components.
• We evaluate our CL2P using five real-world multi-view datasets. Extensive experimental results demonstrate the effectiveness of the proposed method and its superior performance in comparison to baselines.

2 RelatedWork

Multi-view Autoencoders. Multi-view autoencoders (MVAE) are employed to model datasets that originate from multiple sources or views. MVAEs are particularly useful for representation learning, understanding the relationships between different views, and generating missing data. Existing MVAE-based methods can be classified into two categories: model-based MVAE [Xu et al., 2022; Li et al., 2023; Hu et al., 2023; Lu et al., 2024], and information-theorybased MVAE [Shi et al., 2019; Federici et al., 2020; He et al., 2024; Sutter et al., 2024].

Our method falls into the category of information-theorybased MVAE. We leverage the principles of information theory to address the issue of optimal distributions for viewspecific representations from different views, enhancing representation performance and decreasing the learning complexity of view-specificity. In addition, we propose a progressive curriculum learning strategy to tackle the heterogeneity between view-consistent and view-specific representations.

Curriculum Learning. Curriculum learning (CL) is a training strategy that trains a machine learning model from easier data to harder data, which imitates the meaningful learning order in human curricula [Bengio et al., 2009; Soviany et al., 2022]. To date, most CL methods are designed by following the pipeline of a difficulty measurer and a training scheduler. The difficulty measurer is used to measure the hard level of each data to decide learning priority. The training scheduler determines the timing for feeding hard data into the training process. Regarding whether these two components are designed in a data-driven automated manner, CL can be broadly categorized into two types: predefined CL and automatic CL [Wang et al., 2021].

Our method proposes a progressive curriculum learning strategy that extends the concept and applications of predefined CL. In our method, the intrinsic differences (i.e., learning priority) between view-consistency and view-specificity are determined based on human prior knowledge.The training scheduler is an adaptive trade-off parameter over epochs.

Prior Learning. Prior learning (PL) is a critical part of variational inference. PL has emerged as an approach to refining variational inference models [Xu et al., 2019]. It enhances the quality of learned representations by improving the evidence lower bound (ELBO). For example, Bauer et al. [2019] designed a learned acceptance function to reweight the proposal. Similarly, Takahashi et al. [2019] presented an implicit optimal prior for variational autoencoders. This implicit prior uses the density ratio trick. Building upon prior learning ideas, we novelly use a learnable prior based on pseudoinputs. Our method is the first to extend the concept of prior learning to multi-view settings. The prior learning enables each view-specificity to discover its optimal distribution.

3 Methodology

Definition 1 (Problem Statement). Given a set of multi-view data with n samples and m views D = {xi|x1i , ..., xmi }ni=1 , the dataset is used to train a model (e.g., our CL2P). The trained model is then employed to derive high-quality viewconsistent representation and view-specific representation for downstream tasks (e.g., clustering and classification).

Overall Architecture. Figure 2 illustrates the architecture of the proposed CL2P, comprising four novel components: Curriculum Learning, Mixture-of-Experts (MoE), Prior Learning, and Disentangling module. Specifically, curriculum learning guides the whole model to first learn the simple consistent representations and then pay attention to the hard view-specific ones progressively. In practice, the losses of view-consistency and view-specificity are adjusted by curriculum learning using an adaptive trade-off parameter λ, where λ is automatically generated by the "Scheduler" according to the training progress. In view-consistent representation learning, we use a shared encoder Ec to extract viewconsistent representations from all views. Then, we use a MoE layer to fuse these representations by adaptively weighting each expert's output. Finally, the fused consistent representation c serves as input for decoders {Dv c }mv=1 , generating reconstructed views. In view-specific representation learning, we leverage a set of view-specific encoders {Ev s }mv=1 to extract view-specific representations {sv }mv=1 . Meanwhile, we allocate the pseudo-inputs for view data xv . The pseudoinputs serve as auxiliary elements for learning an optimal prior for the view-specific representation sv . For the v-th view-specificity, sv and c are concatenated as the input to the v-th view-specific decoder Dv s . Moreover, to prevent the mutual dilution between view-consistency and view-specificity, we use a disentangling module to minimize the redundancy between these two aspects. For efficiency, we adopt variational autoencoders [Kingma, 2013] to construct the model. Next, we elaborate on our solutions for each component.

Property 1. DNNs tend to prioritize memorization of simple instances first and then gradually memorize hard instances [Zhang et al., 2017; Arpit et al., 2017; Kumar et al., 2024].

3.1 Progressive CurriculumLearning

According to Property 1, DNNs tend to initially focus on learning simple patterns from the data, and then to fit more complex features within the data. We revisit this property in the context of multi-view representation learning. As aforementioned, there are intrinsic differences in the learning of view-consistency and view-specificity. In this work, we propose a progressive curriculum learning method to shift the learning attention of the networks from view-consistency to view-specificity. That is, the progressive curriculum learning method aims at first learning view-consistent representations and then gradually learning view-specific representations. To this end, we design an adaptive trade-off parameter λ to schedule the network parameters and losses produced by these two parts. During the training stage, the Lossc on learning view-consistent representations is weighted by λ, while the Losss on learning view-specific representations is weighted by 1 - λ. Thus, the overall loss of model is λLossc + (1-λ)Losss. The scheduler of λ is formulated as follows:

where Tmax is the number of total training epochs and T is the current epoch. λ gradually decreases as the training epochs increase. The changes of λ ultimately affect the attention on network parameter updates.

The proposed curriculum learning shifts the learning "attention" between simple view-consistent representations and hard view-specific representations. This approach, which is closely aligned with the Property 1 of DNNs, is beneficial for fully extracting patterns from multi-view data. Furthermore, λ controls the parameter updating for each part, thereby preventing damage to the consistent representations while emphasizing view-specific representations at the later stages of training. Next, we delve into the details of view-consistent and view-specific representation learning.

3.2 Mixture of Consistent Posteriors

For view-consistency learning, we assign each view an "expert" to model its posterior distribution, and then combine these individual posteriors as follows

where ϕc are the trainable parameters of consistent encoder Ec(·). Using a Gaussian distribution as the probability distribution for view-consistency c, we refine Eq. (2) as

where [µc , σv ] is the mixture-of-posteriors ("experts"). [µcv , σv v ] is the v-th posterior ("expert") and [µcv , σv v ] = E(xv ). W = [w1 , · · · , wm] is the parameters of the router in the mixture-of-posteriors. The weights W are obtained using a softmax function over the outputs of a gating network:

The mixture-of-posteriors assigns posterior distributions of different views to multiple experts. Each expert focuses on a corresponding view, and the gating network integrates these learned representations. In such a manner, the approach extracts consistent information across all views.

To infer view-consistent representation c frommixture-ofposteriors, we utilize reparameterization trick

where ϵc ∼ N(0, 1). After obtaining the view-consistent representations c, we adopt a series of decoders {Dv c (·)}mv=1 to reconstruct view-specific content. This design allows the model to preserve the view-specific detail in the reconstructions when aligning the view-consistency across views.

Then, we can derive the ELBO of view-consistency as shown below

where θc are the trainable parameters of view-consistent decoders {Dv c (·)}mv=1 . KL(·) denotes the KL divergence. During the inference stage, we discard the view-consistent decoder and solely utilize the consistent encoder Ec(·) to obtain the consistent representations, i.e., c = Ec({xv }mv=1 ).

3.3 View-specific Prior Learning

For view-specificity learning, we employ a set of viewspecific encoders {Ev s (·)}mv=1 to obtain coarse view-specific representations sv for each view. Then we concatenate the above-mentioned view-consistent representations and these view-specific representations, formulated as zv = [c, sv ], which is the input to the view-specific decoders {Dv s (·)}mv=1 . The loss function for this process is shown as follows

We rewrite the KL divergence term in Eq. (7) to provide a more interpretable form, introducing two regularization terms instead of a single KL divergence term:

where the first is the cross-entropy between the aggregated posterior and the prior p(sv ), and the second is the entropy of the variational posterior. Here, q(sv ) = 1n ∑ni=1 qϕs (sv |xvi ) [Makhzani et al., 2015]. Typically, the prior is predefined, often chosen as a standard normal distribution. Notably, we find an optimal prior that optimizes the ELBO by minimizing the following Lagrange function:

where β is the Lagrange multiplier ensuring the proper normalization p(sv ). The solution to this problem is the aggregated posterior p∗ (sv ) = 1n ∑ni=1 qϕs (sv |xvi ), indicating that the optimal prior aligns with the aggregated posterior.

Based on the p∗ (sv ), we can get a new prior for viewspecific representations sv . However, directly using the aggregated posterior as a prior can lead to challenges such as overfitting, high computational complexity, and unlearnability. To mitigate these challenges, the optimal solution can be further approximated by a series of pseudo-inputs:

where K is the number of pseudo-inputs, and uvk is a pseudoinput with the same shape as the real input xvi . These pseudoinputs are learnable parameters, optimized via backpropagation. The learned pseudo-inputs represent typical patterns in the data distribution. So they effectively act as hyperparameters for the prior distribution. Overall, by substituting Eq. (10) into Eq. (7), we derive the loss function of the v-th view-specificity. This loss harmonizes prior learning and representation learning. Learnable priors facilitate the quality of view-specific representations. During the inference stage, we solely utilize the view-specific encoder Ev s (·) to obtain the v-th view-specific representations, i.e, sv = Ev s (xv ).

3.4 Representation Disentangling

For disentangling view-consistent and view-specific representations, we propose to minimize the upper bound of mutual information (MI) between sv and c. The MI between two variables is typically formulated as

However, the conditional probability q(sv |c) is unknown. To address this issue, we adopt contrastive log-ratio upper bound (CLUB) [Cheng et al., 2020], which uses a variational distribution qθ(sv |c) to approximate q(sv |c), as shown below

Given Eq. (12), we then define disentangling loss Ld as

Objective Function. Finally, merging the Eq. (6), Eq. (7) and Eq. (13) via the adaptive parameter λ, we have the joint loss function of our CL2P as follows

We denote that the joint loss function leverages an adaptive parameter λ to make the learning of c and {sv }mv=1 conform to the Property 1 of neural networks. In addition, the training pseudo-code of the proposed CL2P is outlined in Algorithm 1, which describes the steps for updating the model parameters by optimizing the total loss function.

Algorithm 1 Training of the proposed CL2P.

Input: Multi-view dataset D = {x1 , · · · , xm}.

Parameter: Total training epochs Tmax, current training epoch T , K pseudo-inputs of each view, parameters θc, ϕc, θs, ϕs of encoders and decoders.

Output: View-consistent representations c, and view-specific representations {sv }mv=1 .

• 1: Initial the K pseudo-inputs for each view
• 2: while T ≤ Tmax do
• 3: c ←Ec({xv }mv=1 ), and {sv }mv=1 ←{Ev s (xv )}mv=1
• 4: Compute the consistency-loss Lc using Eq. (6)
• 5: Compute the specificity-loss Lvs using Eq. (7)
• 6: Compute the disentangling-loss Lvd using Eq. (13)
• 7: Update T ← T +1, λ←1- ( T Tmax )2
• 8: Compute the total loss Lmodel using Eq. (14) via λ
• 9: Update θc, ϕc, θs, ϕs←∆Lmodel(θc, ϕc, θs, ϕs)
• 10: end while
• 11: return c and {sv }mv=1

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate our CL2P and other competitive methods using five real-world datasets, including: (1) EdgeMNIST [LeCun et al., 1998], which consists of 0-9 grayscale digit images. The views contain the original digit images and the edge-detected version. (2) Edge-Fashion [Xiao et al., 2017], containing grayscale images of various fashion clothing. We generate the second view by applying the same edgedetection technique used in Edge-MNIST; (3) Multi-COIL20 [Nene et al., 1996b], which contains grayscale images of 20 distinct objects, captured from multiple angles. We construct a three-view dataset by randomly grouping the images of each object into three groups. (4) Multi-COIL-100 [Nene et al., 1996a] which depicts from different angles containing RGB images of 100 items. Similar to Multi-COIL-20, we create a three-view dataset by randomly grouping the images of an object into three different groups; (5) Multi-Office31 [Saenko et al., 2010], which consists of objects commonly encountered in office settings, from three distinct domains

(Amazon, DSLR, and Webcam). We construct a three-view dataset with each domain serving as a view. A summary of the statistics for these datasets is shown in Table 1.

Baselines. We compare CL2P against three categories of baseline methods, namely, i) single-view learning methods: Beta-VAE [Higgins et al., 2017] and Joint-VAE [Dupont, 2018]; ii) model-based multi-view methods: MFLVC [Xu et al., 2022], GCFAgg [Yan et al., 2023], SCM [Luo et al., 2024], and CSOT [Zhang et al., 2024]; and iii) informationtheory-based methods: MVAE [Wu and Goodman, 2018], DVIB [Bao, 2021], Multi-VAE [Xu et al., 2021], and MRDD [Ke et al., 2024]. Our method falls into the third category. Additionally, we evaluate two variants of our method: CL2P-C, which uses only view-consistent representations c, and CL2P-CS, which incorporates both view-consistent representations c and view-specific representations {sv }mv=1 .

Implementation Details. We implement the proposed method and other comparison methods on PyTorch 2.1.0, utilizing one NVIDIA A10 GPU (24 GB). We employ convolutional networks [He et al., 2016] to construct our model, with a uniform encoder-decoder architecture across all datasets. Both view-consistency and view-specificity dimensions are set to 20. The number of pseudo-inputs is fixed at 250, initialized with randomly selected training data. We train our model for 200 epochs using the AdamW optimizer with the learning rate of 1 × 10-4 and a weight decay of 1 × 10-4 . We set a batch size of 128 for Edge-MNIST and Edge-Fashion, and 32 for Multi-COIL-20, Multi-COIL-100, and Multi-Office31. For all baseline models, we use the optimal settings as recommended in the original paper for fair comparison.

4.2 Experimental Results

Task Settings. We evaluate all baseline models across five datasets for both clustering and classification tasks. For clustering, we use the k-means algorithm [Hartigan and Wong, 1979]. For classification, we apply support vector classification (SVC) [Hsu, 2003] with an 80:20 train-test split ratio. We use Accuracy (ACC) as the metric to evaluate the performance of each model for both clustering and classification tasks on five datasets. Each model is evaluated over 10 runs, and we report the average values along with the variances. Notably, for single-viewmethods, we select the results from the best view as the evaluation outcome. For multi-view methods limited to two views, we choose the optimal pair of views as inputs for the models.

Overall Evaluation. Tables 2 and 3 show the performance of clustering and classification tasks, respectively. From the results, we have the following observations: (1) The proposed CL2P is superior to all baseline methods. The results show the effectiveness of the proposed method. (2) CL2PC outperforms CL2P-CS on the Multi-COIL-20 and MultiCOIL-100 datasets, which indicates that naive concatenation may degrade the overall quality of representation. (3) Singleview methods are inferior to multi-view methods in most cases. On the contrary, single-view methods sometimes outperform certain multi-view methods. The reason might be due to that some specific views are dominant and those multiview methods struggle to handle data with significant viewspecific differences. (4) Information-theory-based methods generally outperform model-based methods, indicating that information-theoretic constraints guide the model to learn higher-quality representations. Nevertheless, those methods overlook the inherent properties of neural networks and thus fail to surpass our method.

Ablation Study. We conduct an ablation study to measure the contribution of the four key components, i.e., curriculum learning (CL), prior learning (PL), disentangling learning (DL), and Mixture-of-Experts (MoE), in our CL2P-CS method. We remove each component from CL2P-CS and then evaluate the performance of each resulting variant on five datasets using the clustering performance. The ablation results are shown in Table 4. From the results, we can see that every component in our model plays an important role. This shows that all components in our model are essential. For the datasets Multi-COIL-100 and Multi-Office-31, the ablation results show slight differences. This indicates that more advanced encoders and decoders may be required to enhance the model's capability for information extraction.

Scheduler Ablation. We explore several different strategies to generate the adaptive trade-off parameter λ, including line decay (λ = 1 - T Tmax ), cosine decay (λ = 0.5 · cos(π ·

dulers.

(b) Clustering accuracy (%) of different sche-

dulers on five datasets.

(a) ¸ curves against T=Tmax on different sche-

(c) Classification accuracy (%) of different schedulers on five datasets.

Table 4: Component ablation on the proposed model. Clustering accuracy scores (%) on five datasets. Best results are in bold.

T Tmax ) + 0.5), exponential decay (λ = e-2π·( T Tmax ) ), and identity (λ = 0.5), in our CL2P-CS. The results are shown in Figure 3. Figure 3(a) illustrates the variation of λ with respect to different schedulers. Figures 3(b) and 3(c) present the performance for each scheduler. From the results, we observe that a slower decay rate for parameter λ yields better performance. When λ decays too rapidly, it prematurely shifts the attention of networks away from view-consistency. The premature shift reduces the quality and effectiveness of consistency, leading to a decline in overall representations.

Study on the number of pseudo-inputs K. We investigate the effect of parameter K by conducting a grid search of the settings {0, 50, 100, 150, 200, 250, 300, 350, 400}, in the CL2P-CS method. The results are shown in Figure 4. We find that the model remains robust with a wide range of parameter values, e.g., [150, 350]. However, both excessive and insufficient numbers of pseudo-inputs lead to suboptimal performance. Additionally, certain settings may introduce side effects on the performance. Therefore, the selection of the K value is crucial to the overall performance of the model.

5 Conclusion

In this paper, we proposed a novel multi-view representation learning framework called CL2P. Specifically, we extend curriculum learning to tackle the intrinsic differences between view-consistency learning and view-specificity learning. Moreover, we devise a learnable prior for view-specific representations from different views. Additionally, we saddle the model with a mixture-of-experts layer and a disentangling module to enhance the representation quality. We evaluated CL2P and baseline methods on clustering and classification tasks. The results show that CL2P outperforms baselines markedly on five real-world datasets. In the future, we plan to integrate more advanced encoders-decoders into CL2P to enhance representational capability, particularly for multi-modal data. We also intend to explore CL2P in incomplete multi-view and semi-supervised learning settings.

Acknowledgments

This work was supported in part by the National Key R&D Program of China (grant no. 2024YFB4710604), NSFC (grant nos. 62406209, 62472295, U24B20174, and U21B2040), Sichuan Science and Technology Planning Project (grant nos. 2024NSFTD0130, 2024NSFTD0038, and 2025ZNSFSC1486), and the Fundamental Research Funds for the Central Universities (grant nos. CJ202303, CJ202403, and YT202421).