Learning Robust Multi-view Representation Using Dual-masked VAEs

1 Introduction

Many applications face the situation where each data instance in a set X = {𝑥1, ..., 𝑥𝑛} is sampled from multiple views or even multiple modalities. Here each 𝑥𝑖 | 𝑁 𝑖=1 is denoted by multiple views, e.g., 𝑚 views {𝑥 1 𝑖 , ..., 𝑥 𝑚 𝑖 }. Such forms of data are referred to as multi-view data. Multiview data provide richer and more comprehensive information from raw features within data objectives compared to single-view data [Liang et al., 2024; Yang and Wang, 2018]. As a cutting-edge research topic, multi-view representation learning (MvRL) addresses the motivation of discovering a shared representation from different views with the complex underlying correlation [Zheng et al., 2023; Wang et al., 2015] and later adapting it for downstream tasks, such as human activity recognition [Yadav et al., 2021], 3D reconstruction [Xie et al., 2019] and anomaly detection [Wang et al., 2023].

∗ Corresponding author (H. Wang).

Over the years, MvRL has showcased promising performance, along with its potential to inspire research in the realm of multi-view or multi-modal AI. MvRL methods primarily focus on learning shared information (i.e., view consistency) among views and distinctive information (i.e., view specificity) within each view [Xu et al., 2021; Ke et al., 2024]. Most of them assume that all views are complete and data are noise-free, involving a conditional scenario that we call closed multi-view setting. However, the closed multi-view setting is too limited for real-world applications as it frequently happens that some data instances are not sampled in certain views and data features are corrupted by noise, which refers to a more realistic scenario that we call generic multiview setting. This paper is concerned with view-missing and sample-noise problems in such a generic setting.

View-missing and sample-noise would disturb multi-view data, and sequentially impact the data's benefit for downstream tasks. To date, some efforts have been devoted to solving view-missing issues [Zhang et al., 2018; Wen et al., 2019; Zhang et al., 2020] and devising noise-tolerance multiview models [Yue et al., 2019]. It is worth noting that the existing methods can only tackle view missingness or noise robustness, rather than addressing both of them within a unified framework. That is, the robustness of a unified model for both cases is still challenging and non-trivial in practice. Specifically, there are three major challenges: 1) view representation, 2) view fusion, and 3) view disentangle, due to the problems of view-missing and noisy data.

As expatiated above, there is no existing work that can address all issues holistically using a single model in the generic multi-view setting. In this work, we aim to deal with view-missing and sample-noise problems using a holistic model. To this end, we propose Dual-masked Variational Autoencoders (DualVAE), which jointly aim at learning robust view-consistent and view-specific representations. Our main ideas here are three-fold: dual-masked prediction, mixture-ofexperts learning, and representation disentangling. We will expatiate on each of them shortly. In a nutshell, the pipeline of the proposed DualVAE is as follows: i) view-specific encoders take the input of each view and learn view-specific representation against view-specific noise within each view, ii) a consistent encoder with dual-masked prediction and mixtureof-experts learning for processing all views, yielding robust view-consistent representations, and iii) a disentangled module that minimizes the upper-bound of mutual information to disentangle view-consistent and view-specific representations such that the distribution of view-consistency can be maximized to differ from the view-specific information of each view, thereby extracting robust shared information.

In summary, we make the following contributions:

• We delve into multi-view representation learning in a generic multi-view setting, and propose a robust multiview representation learning method called DualVAE to solve both view-missing and sample-noise problems.
• We devise a novel dual-masked mechanism, together with a Mixture-of-Experts layer and a disentangle learning module in addressing the distinct challenges posed by view-missing and sample-noise in multi-view data.
• Experimental results using four real-world datasets demonstrate the proposed DualVAE outperforms baseline methods dramatically.

2 RelatedWork

Multi-view Representation Learning. The objective of multi-view representation learning (MvRL) is to extract highquality feature representations from diverse sources of data for downstream tasks [Wang et al., 2015; Chen et al., 2022; Yacobi et al., 2024]. In recent years, exciting progress has been achieved in MvRL by assuming view-completeness and noise-free multi-view data [Hwang et al., 2021; Xu et al., 2022; Ke et al., 2024]. However, MvRL encounters the challenge of poor quality on input data such as view-missing and sample-noise [Li et al., 2018]. As for their specific objectives, the methods dedicated to improving model robustness can be generally categorized into the following two types: 1) incomplete MvRL methods that aim at learning view-common information across observable views to handle incomplete information such as partially aligned or partially missingness in multi-view data [Zhang et al., 2018; Wen et al., 2019; Zhang et al., 2020], and 2) noise-robust MvRL methods that focus on employing specific methods like filtering noises and disentangling noises to address data noises in different views [Yue et al., 2019; Fan et al., 2023; Wang et al., 2025].

It is worth noting that existing MvRL methods rarely consider a generic multi-view setting where the data encounter both view-missingness and sample-noise problems. Our approach is a holistic method that can handle both of them.

Multi-view VAE. With the advancement of generative models, variational autoencoders (VAE) technologies have been introduced into the realm of multi-view research, called multi-view VAE. Multi-view VAE typically extracts generalizable representations by learning a joint distribution of multiview data [Daunhawer et al., 2021; Sutter et al., 2021]. Following Wu and Goodman [2018], existing multi-view VAE methods show promising results for learning high-quality representations across multiple views, which is beneficial for downstream tasks such as clustering and classification [Shi et al., 2019; Sutter et al., 2020; Sutter et al., 2021].

Some efforts have also been devoted to learning latent representations from the subsets of observable views to address missing views [Wu and Goodman, 2018; Vedantam et al., 2017; Shi et al., 2019; Sutter et al., 2021]. However, their computational cost increases significantly as the combinatorial number of views increases, which leads to a decrease in efficiency. Our approach adopts a dual-masked mechanism, which deals with both view-missing and sample-noise problems without increasing such combinatorial computing cost.

Masked VAE. Masking technique is a self-supervised learning (SSL) method that randomly masks some partial information (e.g., words in a sentence or patches in an image) and then prompts the model to predict invisible blocks, which receives a lot of attention in NLP and CV community [Vaswani, 2017; Devlin, 2018; Bao et al., 2021; He et al., 2022]. The masking mechanism has also been incorporated into VAE, resulting in the development of masked VAE [Xia et al., 2023; Li et al., 2023; Xu et al., 2023] and multi-view masked VAE [Ke et al., 2024].

Multi-view masked VAE [Ke et al., 2024] uses pixelmasking for input images and does not consider view-missing and sample-noise problems in the generic multi-view setting. Our approach is a dual-masked VAE, which consists of viewmask and patch-mask, along with mixture-of-experts learning and representation disentangling, an innovative amalgamation to enhance model robustness against both view-missing and sample-noise problems.

3 Methodology

As aforementioned, we propose a DualVAE for multi-view representation learning. Prior to introducing the details of the model, we first clarify the problem studied in this paper.

Definition 1 (Problem Statement). Given a multi-view dataset with 𝑚 views and 𝑛 samples X = {𝑥𝑖 |𝑥 1 𝑖 , ..., 𝑥 𝑚 𝑖 } 𝑛 𝑖=1 in a generic setting, where some data might have unavailable views (e.g., 𝑗-view) and corrupted parts (e.g., 𝑥 𝑗 𝑖,𝑘 where 𝑘 denotes dimension), referred to as missing views and noisy samples respectively, the dataset is used to train a model. The trained model is then employed to drive multi-view representations for downstream tasks after deployment, where the test may also contain view-missing and sample-noisy data.

3.1 Overall Architecture

Figure 1 illustrates the architecture of our DaulVAE, comprising three novel components: i) Dual-masked Prediction (DMP), which processes multi-view data using view-mask and patch-mask, ii) Mixture-of-Experts (MoE), which models view-consistent representation, and iii) Representation Disentangling, which disentangles view-consistent and viewspecific representations. Specifically, we first employ a set of view-specific encoders {𝐸 𝑖 𝑠 } 𝑚 𝑖=1 to extract view-specific representation {s 𝑖 } 𝑚 𝑖=1 which contains noises from corresponding views. We also perform data augmentation on the original multi-view data by cropping images into blocks. Then we use the DMP to mask views and feature patches. Next, those visible blocks and views are fed into a view-shared encoder 𝐸𝑐 to learn a posterior distribution of the latent space {𝑞(c|𝑥 𝑖 )} from different views. To mitigate the impact of missing views, we aggregate these distributions by using a MoE layer. In practice, we reparameterize from the aggregated distribution to obtain view-consistent representation c. Subsequently, we use a disentangling module by minimizing the mutual information (MI) between c and {s 𝑖 } 𝑚 𝑖=1 to separate representations from noise and enhance the quality of c. Finally, we concatenate c and {s 𝑖 } 𝑚 𝑖=1 and feed the results into decoders to reconstruct multi-view data.

Roadmap to Our Model. Given the problem and architecture, our primary objectives for designing DualVAE are (1) processing multi-view data using DMP (Section 3.2), (2) modeling view-consistent representation with MoE (Section 3.3), and (3) learning disentangled representations using upper-bound MI (Section 3.4). We now elaborate on our solutions to each component.

3.2 Dual-masked Prediction

In this work, we aim to extract robust multi-view representations in the generic settings. We denote that the denoising autoencoders in the domain of computer vision apply different levels of distortion to the input images and then prompt the model to recover the degraded areas of the images [Vincent et al., 2008]. Following [Vincent et al., 2008], we devise a view-mask that corrupts multi-view data by discarding one or more views of the data to enhance robustness against viewmissing. Moreover, noisy samples in the data would degrade a model in extracting shared information among multi-view data. To address this challenge, we use a patch-mask, which has demonstrated its robustness against sample-noise in [He et al., 2022; Xia et al., 2023] by randomly masking tokens and reconstructing them later. That is, our masking approach is a dual-masked model of view-mask and patch-mask.

Specifically, view-mask focuses on the adaptability to view-missing data. In practice, we randomly select samples {X𝑘1 , X𝑘2 , . . . , X𝑘𝑛 } ⊆ {Xi} 𝑛 𝑖=1 . For each sample, we next randomly select one view (e.g., the 𝑣-th view) and then mask its data 𝑥 𝑣 𝑘𝑖 . An intuition of our view-mask is that it simulates view-missing scenarios and prompts the model to predict invisible views so as to enhance robustness against viewmissing. Patch-mask focuses on the noise tolerance of the model. Since noise-corrupted views share a common latent space, our motivation here is that the patch-mask has capacity to shield most noise and force the model to extract more information gains from data. By leveraging these visible blocks as inputs to the view-shared encoder 𝐸𝑐, the model then predicts the original information of incomplete data. We call such a method Dual-masked Prediction (DMP), formulated as ˇX = 𝐷𝑀𝑃(X). By training the view-shared encoder, the data of each view are mapped to the shared latent space, denoted as posterior distributions {𝑞𝜃 (c| ˇ 𝑥 𝑖 )}, where ˇ 𝑥 𝑖 is the output of dual-masked module on 𝑥 𝑖 and 𝜃 is trainable parameters of the view-shared encoder. Moreover, our dual-masked model processes multi-view data without incurring additional computational overhead.

3.3 Mixture-of-Experts Learning

To extract view-consistent representation, a common solution is to aggregate information from multiple views, referring to view-fusion. Since the existence of missing views and noisy samples in multi-view data, view fusion is challenging in generic settings. Aiming to extract coherent representations in terms of the shared latent spaces, we propose to mitigate the interference caused by missing views. Our main idea is to reduce the weight coefficient of the distribution from missing views. To this end, we saddle the model with a Mixture-ofExperts (MoE) layer.

Specifically, as the missingness of samples and nonidentically distributed noises from multiple views, the posterior 𝑞(c| ˇ 𝑥 𝑖 ) generated by the view-shared encoder might not be optimal. To address this challenge, we assign each view an expert (as shown in Figure 1), to process the posterior from each view, namely, transforming the posterior and further learning on the transition space to the shared latent space. In our framework, each expert is implemented by a Multilayer Perceptron (MLP) parameterized by 𝜙 𝑖 . The transformed posteriors are expressed as 𝑞𝜙 𝑖 (c| ˇ 𝑥 𝑖 ) = 𝑀𝐿𝑃𝜙 𝑖 (𝑞𝜃 (c| ˇ 𝑥 𝑖 )). Then we have

where ˇX = 𝐷𝑀𝑃(X). 𝜶 = {𝛼 𝑖 } 𝑚 𝑖=1 are gating scores generated by a gating network as below

where 𝛾 denotes the parameters of the gating network. The intuition here is that by dynamically reducing gating scores 𝛼𝑖 of the missing views, the model can mitigate their impact to improve the quality of view-consistent representations for view-missing and sample-noise data. In addition, to further enhance the model's adaptability, we use a normal Gaussian distribution to model the prior of view-consistent representation, formulated as 𝑝(c) = N(0, I). Moreover, as the random sampling from posterior 𝑞𝜙(c| ˇX) cannot directly propagate gradients, we reparameterize this distribution to obtain viewconsistent representation c, formulated as follows

where 𝜇𝑐 and 𝜎𝑐 are trainable parameters and 𝜖𝑐 ∼ N(0, I).

3.4 Representation Disentangling

Due to the diversity of multi-view data, the noise distributions in different views are also distinct. As the noises are often irregular and random, enhancing the noise-tolerance of MvRL is challenging and non-trivial. To address this challenge, our insight is that view-specific representation might contain both view-specific features and sample noise, as shown below

where 𝑠 𝑖 denotes view-specific information, 𝜖 𝑖 denotes sample-noise, and 𝜉 denotes a joint distribution of them. That is, multi-view data contains three types of information: view-consistent information, view-specific information, and distinct noise. As shown in Section 3.2, we proposed a novel dual-masked prediction to improve the quality of viewconsist representation, where the masked content might be view-specific information or noisy data. We now introduce how to disentangle them.

As mentioned above, suppose we have extracted a viewconsistent representation, we then propose to explore the remaining information across views. Our motivation is that view-consistency and view-specificity should be independent. We design a disentangle module by minimizing the upper-bound of mutual information between them. Specifically, as illustrated in Figure 1, we deploy a set of viewspecific encoders {𝐸 𝑖 𝑠 } 𝑚 𝑖=1 to tackle each views, which extract view-specific representation {s 𝑖 } 𝑚 𝑖=1 . We formulate the posterior distributions of s 𝑖 as 𝑞𝜑 𝑖 (s 𝑖 |𝑥 𝑖 ), where 𝜑 𝑖 denotes trainable parameters of 𝐸 𝑖 𝑠. Similarly, we employ a Gaussian distribution to model view-specific representation, formulated as 𝑝(s 𝑖 ) ∼ N(0, I).

Then, the view-consistent representation c and viewspecific representation {s 𝑖 } 𝑚 𝑖=1 are the inputs of disentangling module. After that, we concatenate view-consistent representation and view-specific representation, denoted as z 𝑖 = [c, s 𝑖 ]. Finally, we deploy a set of decoders {𝐷 𝑖 } 𝑚 𝑖=1 to generate data, where z 𝑖 is the input of the decoder 𝐷 𝑖 .

Based on Eq. (4) and the chain rule for mutual information, we have

which shows that we can disentangle c and 𝜖 𝑖 by minimizing the upper bound of mutual information between c and s 𝑖 to address noise data. The mutual information between s 𝑖 and c is formulated as follows

where ℎ(s 𝑖 ) denotes the variational marginal approximation of s 𝑖 . However, the upper bound of Eq. (6) is difficult to estimate. Following CLUB [Cheng et al., 2020], we use an approximation without assuming a prior. CLUB approximates 𝑝(s 𝑖 |c) using a variational distribution 𝑞𝜓 𝑖 (s 𝑖 |c), where 𝜓 𝑖 denotes the parameters of the 𝑖-th estimator. Then, we define our disentangling loss function as

where L 𝑖 𝑑 is the loss of the 𝑖-th estimator, formulated as

where 𝐼𝐶𝐿𝑈𝐵(s 𝑖 ; c) ≥ 𝐼 (s 𝑖 , c) as proved in [Cheng et al., 2020].

We then concatenate view-consistent representation and view-specific representations to generate z 𝑖 , i.e., z 𝑖 = [c, s 𝑖 ], whose posterior is defined as 𝑞𝜙,𝜑 𝑖 (z i |{𝑥 𝑖 }). Similar to viewconsistent representation, we yet reparameterize the posteriors of view-specific representation as follows

where {𝜇𝑠 𝑖 }, and {𝜎𝑠 𝑖 } are trainable parameters in neutral networks and 𝜖 𝑖 𝑠 ∼ N(0, I). The sampling results from the latent distribution are then fed into decoders to reconstruct samples denoted as { ˆ 𝑥 𝑖 } 𝑚 𝑖=1 . The likelihood of the reconstructed samples of 𝑖-th view is shown as follows

In practice, we adopt vanilla VAE to build a base model [Kingma, 2013]. The reconstruction loss L𝑔 with the evidence lower bound (ELBO) can be expressed as:

Suppose c and s 𝑖 are conditionally independent, then we have 𝑞(z i |{𝑥 𝑖 }) = 𝑞(c, s 𝑖 |{𝑥 𝑖 }) = 𝑞(c|{𝑥 𝑖 })𝑞(s 𝑖 |𝑥 𝑖 ). The KL divergence in Eq. (11) is then formulated below

However, KL[𝑞(c|{ ˇ 𝑥 𝑖 })| |𝑝(c)] cannot be computed directly. By Eq. (1) and dropping 𝜙 𝑖 for simplicity, we have

By Jensen's inequality, we can derive

Then, the ELBO of the model can be reformulated as

Objection Function. Formally, we have the total loss function of our DualVAE:

4 Experiments

4.1 Experimental Setup

Datasets. To evaluate the proposed model, we conduct experiments on four publicly available multi-view datasets. The statistics of each dataset are shown in Table 1.

Baselines. We compare DualVAE against the following three types of baseline methods:

• Single-view VAE: 𝛽-VAE [Higgins et al., 2017], and Joint-VAE [Dupont, 2018].
• Multi-view non-VAE: SCM [Luo et al., 2024], and MFLVC [Xu et al., 2022].
• Multi-view VAE: MVAE [Wu and Goodman, 2018], Multi-VAE [Xu et al., 2021], MIB [Federici et al., 2020], and MRDD [Ke et al., 2024].

For those sing-view VAE methods, we choose the best results among all views. For the multi-view VAE methods that are tailored for two views, we select the first two views as inputs. For other baseline methods, we ran their original systems with the settings as they suggested.

ImplementationDetails. We deployed the proposedmodel and baselines using Pytorch 2.3.1 and ran experiments on NVIDIA TITAN GPUs with 24GB of memory in Ubuntu 20.04.1 LTS. We utilize Adam optimizer with weight decay and set training epochs as 200. Moreover, we set the initial learning rate to 1 × 10 -4 and adopt cosine annealing during the training. Following [He et al., 2022; Ke et al., 2024], the default patch-mask ratio is 0.7. We then initially set view-mask ratio as 0.3. Both the dimensions of view-consistent representation and view-specific representation are 10. We ran 10 times on each evaluation and recorded the average value and standard deviation.

4.2 Experimental Results

Task Settings. We evaluate each model with clustering and classification tasks. For clustering, we implement 𝐾-means algorithm on the learnt representations. For classification, we employ support vector classification (SVC). To have a comprehensive evaluation, we evaluate each model under different settings of View-Missing Ratio (VMR) and Sample-Noise Ratio (SNR). The VMR is defined as 𝑉𝑀𝑅 = 𝑘/𝑁, where 𝑁 is the size of all samples and 𝑘 denotes the number of samples randomly selected to remove one of the views. For SNR, we added salt-and-pepper noise into four datasets with varying degrees of intensity, denoting the ratio of pixels that are corrupted by noises. we vary the values of VMR and SNR from 0% to 90%, with an interval of 10%.

We use Accuracy (ACC) as the metric to evaluate the performance of each model for both clustering and classification tasks on four datasets.

Overall Evaluation. The evaluation results in terms of ACC on each dataset with different VMR and SNR settings are shown in Figure 2 and Figure 3. From the results, we have the following observations:

• The proposed DualVAE is superior to baseline methods and achieves promising performance even on the heavy missing settings. DualVAE performs well with the increasing of VMR. The results demonstrate that our DualVAE has capability to handle view missingness.
• All the baseline models suffer performance degradation as the SNR increases. In contrast, DualVAE exhibits significant robustness and achieves superior performance.
• The results in the above two experiments clearly show that the proposed DualVAE model is effective against view-missing and sample-noise problems.

In addition, to further demonstrate the performance of the proposed DualVAE in a generic setting, we examine the model under a mixed setting of VMR and SNR, where both variables range from 0% to 80% with an interval of 20%. The results are shown in Figure 4. From the figure, we can see that for each fixed SNR, the model has small variation as VMR increases, which demonstrates its robustness against view-missing problems. The model also exhibits robust performance against noise. The results show that our DualVAE is robust in generic multi-view settings.

(a) COIL-20

(b) COIL-100

(d) PolyMNIST

Figure 4: Clustering performance with different View-Missing Ratio (VMR) and Sample-Noise Ratio (SNR) settings on four datasets.

Ablation Study. To explore the framework of DualVAE, we conduct an ablation study by analyzing the effectiveness of the devised DMP (Section 3.2), MoE (Section 3.3), and disentanglement (Section 3.4). We remove each of them from the framework and conduct clustering tasks on four datasets with the setting of VMR=0.5 and SNR=0.1. Particularly, we exclude MoE by replacing linear layers with it. Experimental results of the ablation study are shown in Table 2. From the results, we see that the removal of any component leads to a degradation in performance. This indicates that all components in our framework are essential. As for the three components, DMP contributes more than the other two.

Parameter Study. Since the proposed dual-masked model contains two hyperparameters, i.e., view-mask ratio and patch-mask ratio, we now explore how they influence the performance of the model. We conduct grid search experiments in the setting of VMR=0.5 and SNR=0.1, where view-mask ratio and patch-mask ratio vary from 0% to 90% with a 10% interval. First, we conduct classification on four datasets with different view-mask ratios by fixing the patch-mask ratio as 70%. The results are shown in Figure 5. It is worth noting that a proper view-mask ratio can significantly enhance the performance of the model, such as 30%. Then we fix the view-mask ratio as 0.3, and vary the patch-mask ratio for further exploration. The results are shown in Figure 6. From this figure, we can see that a higher patch-mask ratio may achieve better performance. In addition, for small datasets such as E-MNIST and COIL-20, a high-intensity patch-mask would lead to the loss of representative information, resulting in performance degradation. That is, both view-mask and patch-mask contribute to the robustness of multi-view representations learned by the proposed model.

5 Conclusion

In this paper, we are concerned with the robustness of multiview representation learning in a generic multi-view setting, particularly for view-missing and sample-noise problems. To this end, we proposed a novel Dual-masked Variational AutoEncoders (denoted as DualVAE), which is a unified framework to extract robust representations for multi-view data. The DualVAE exhibits an innovative amalgamation of dualmasked prediction, mixture-of-experts learning, representation disentangling, and a joint loss function in wrapping up all components. Extensive experimental results demonstrate the superior robustness of the proposed method compared to baseline methods in the settings of different view-missing ratios and sample-noise ratios.

Acknowledgments

This work was supported in part by the National Key R&D Program of China (grant no. 2024YFB4710604), NSFC (nos. 62406209, 62472295, U24B20174, and U21B2040), Sichuan Science and Technology Program (nos. 2024NSFTD0130, 2024NSFTD0038, and 2025ZNSFSC1486), and the Fundamental Research Funds for the Central Universities (nos. CJ202303, CJ202403, and YT202421).