View-Association-Guided Dynamic Multi-View Classification

1 Introduction

Rapid advancement in technology has led to an exponential increase in the generation of multi-view data, including images, text, audio, and video [Fu et al., 2025; Jiang et al., 2021; Li et al., 2023; Wen et al., 2023; Zhang et al., 2024a]. This abundance of data presents opportunities and challenges for researchers and practitioners in diverse fields, such as computer vision and natural language processing. Integrating multiple views information from can significantly enhance the performance of machine learning models, improving their ability to understand complex, real-world scenarios [Jiang et al., 2024; Liang et al., 2024; Guo et al., 2024; Fu et al., 2024b].

Most of multi-view classification algorithms exploits two fundamental principles which ensures their success: (i) consensus, and (ii) diversity principles. The consensus principle seeks to maximize the agreement between multiple representations of the data. The diversity principle demonstrates that in a multiview learning problem, each representation or view of the data may contain some information which other views do not have. Based on these two principles, many fusion algorithms for multi-view classification have been proposed. For example, [Liang et al., 2022] introduced the association information between modality features into multi-modal data fusion and proposed an association-based fusion strategy for multi-modal classification (MMC) in an interpretable manner. [Chen et al., 2023] proposed a joint deep learning framework to learn an underlying feature representation from heterogeneous views.

∗Corresponding author.

We analyze fusion methods within kernel-based, graphical, encoder-decoder, and attention-based fusion frameworks. Fig. 1 illustrates three typical structures of fusion models [Li and Tang, 2024]. In Fig. 1(a), text and images are processed through simple operations (dot product, multiplication, and addition). In Fig. 1(b), the Fusion Network is designed to combine the individual image and text embeddings. In Fig. 1(c), models use an integrated encoding-decoding process to handle multi-view inputs simultaneously. Most existing fusion methods implicitly exploit the relationships between views during the fusion process, where the interactions among views are typically captured without explicit modeling. This implicit approach, while effective in many cases, may not fully capture the nuanced relationships between the different views, leading to suboptimal fusion performance in certain complex multi-view scenarios. The lack of explicit modeling of view relationships could potentially limit the ability of the model to explore consensus or diversity more effectively. To address this limitation, it is crucial to consider explicitly modeling the relationships between views in multiview fusion frameworks. As shown in Fig. 1(d), we propose an explicit modeling step for the relationships between views before the fusion of the view features.

Moreover, traditional fusion methods have largely overlooked the dynamic variation in the quality of multi-view data. The oversight often results in fusion methods that fail to adapt to the evolving nature of multi-view data, leading to performance degradation in real-world applications. Therefore, there is a need for fusion techniques that not only integrate information across views but also dynamically adjust to the varying quality of each view.

To address these issues, we propose an end-to-end framework called a view-association-guided dynamic multi-view classification method (AssoDMVC). AssoDMVC is comprised of two fundamental components: a view association encoding module, and a view association-guided dynamic weighting fusion module.

Our main contributions are summarized as follows:

• A view association encoding module is designed to explicitly model the relationships between views, offering guidance for the interaction between different views in downstream tasks, thereby enhancing the effectiveness of the fusion process.
• A novel end-to-end framework, the view-associationguided dynamic multi-view classification method (AssoDMVC), is proposed to address the challenges of multi-view data fusion, providing a more effective and comprehensive solution for integrating diverse views.
• The extensive comparison experiments on six public datasets show that AssoDMVC achieves competitive performance compared to the the state-of-the-art MVC methods.

2 RelatedWork

Multi-view classification (MVC)is a powerful approach that leverages multiple sources of information (views) to improve the accuracy and robustness of classification tasks. In multiview classification, each view provides a different perspective or representation of the same underlying data, and combining these views effectively can lead to better performance than relying on a single view[Fu et al., 2024a].

In recent years, many multi-view classification models with complex objectives were further proposed. [Chen et al., 2021] proposed a joint framework for multi-view spectral clustering by learning an adaptive transition probability matrix. The nuclearnorm-based optimization method was proposed to conduct multi-view image data fusion via a joint learning framework[Huang et al., 2020].Sparsitybased optimization methods are also essential in multi-view classification[Wang et al., 2022a].

Fusion methods play a important role in MVC. The quality among multi-view features is different, lots of works consider their contribution to the final tasks[Liang et al., 2025]. Multi-view classification methods can be roughly divided into the feature level fusion-based and decision level fusion-based. Based on where the view contribution are considered, these methods can be grouped into feature level weighting-based method (FW) and decision level weightingbased method (DW). FW learns the contribution weight of each feature[Jiang et al., 2022]. For example, EmbraceNet [Choi and Lee, 2019] assigns 1 to the weight value of one view while 0 to others for each example according to a multinomial distribution. An adaptive-weighting discriminative regression approach (AWDR) [Yang et al., 2019] adopts the square root form of view weight to distinguish features from different views. [Zhang et al., 2024b] proposed discriminative multi-view fusion via adaptive regression (DMVF), it simultaneously discriminates the contribution diversity of different views and samples in an adaptive weighting manner, reducing the influence of low-quality views and outliers for classification. DW learns to assign weights at the decision level. For example, [Han et al., 2022] proposed a trusted multi-view classification (TMC), which models the confidence of each view at an evidence level using the DempsterShafer theory.

3 The ProposedMethod

In this section, we first introduce a view association encoding module, followed by the design of a view association-guided dynamic weighting fusion module. Fig. 2 provides an overall illustration of the proposed approach.

3.1 Basic Setting

The remainder of this section uses the following notation. Let X = Rm1 ×Rm2 × · · · ×Rm|V | represent the instance space

(or feature space) of representations from |V | views, where mi (1 ≤ i ≤ |V |) denotes the feature dimension of the i-th view, and let Y = {l1, l2, . . . , lq} represent the label space with q class labels. Let D denote an unknown distribution over X ×Y. A training set D = {(xvi , yi), |, 1 ≤ v ≤ |V |, 1 ≤ i ≤ n} ∈ (X × Y)n is drawn independently and identically from D, where xvi = (xvi1 , xvi2 , . . . , xvimv ) ∈ Rmv is the feature vector of the v-th view with dimension mv, and yi ∈ Y is the known label associated with xvi . The task of multi-view classification is to learn a prediction function f : X → Y from D, which can assign an appropriate label f(x) ∈ Y to an unseen instance x.

3.2 ViewAssociation EncodingModule

The module aims to explicitly model the association between different views and learn the corresponding view association embeddings. By capturing the complex between views, the learned view association embeddings enable the model to better understand how each view influences the others, thereby optimizing the fusion process and facilitating more accurate predictions.

First, we construct the view association graph based on the similarity relationships between views. Let G = (V, E) represent the view association graph, where V denotes the set of views and E denotes the set of edges. The adjacency matrix A stores the weights associated with each edge, representing the similarity between views. Since the feature dimensions of samples in each view are not consistent, the similarity relationships between views cannot be directly calculated. Therefore, modeling the relationships between views is converted to examining the relationships between the features of the same sample across different views. To reduce computational cost, the following strategy is adopted for calculating the similarity between views:

• First, from the same view, select k samples. Let the sample set X v = {xv1 , xv2 , . . . , xvn } be the set of all samples in the v-th view, where each xvi ∈ Rmv is the feature vector of the i-th sample. Select k samples from these n samples to form a sample set Sv k = {xvi1 , xvi2 , . . . , xvik }, where i1, i2, . . . , ik ∈ {1, 2, . . . , n}.
• Next, calculate the Euclidean distance between each of these k samples and every other sample, resulting in an n × k Euclidean distance matrix Dv , where the element Dv ij represents the distance between the i-th and j-th samples:

This matrix Dv is of size n × k, representing the Euclidean distances between each sample and the k selected samples. The obtained Euclidean distance matrix D is subtracted pairwise to get the relative distance matrix R between views:

where i, j ∈ {1, 2, . . . , |V |}. A smaller relative distance indicates a higher similarity between the views. Finally, we perform a normalization operation on R to obtain the adjacency matrix A.

Subsequently, a graph autoencoder is applied to learn the view association embeddings. The encoder in the graph autoencoder is instantiated by a Graph Isomorphism Network (GIN) . Given a feature matrix H(t) ∈ R|V |×d(t) for a node, where each row represents a view association embedding and d(t) denotes the dimension of the node features, the GIN layer updates the node as follows:

where H(t+1) ∈ R|V |×d(t+1) is the updated feature matrix of the nodes, and f (t+1) represents a neural network consisting of two fully connected layers, followed by Batch Normalization and a LeakyReLU activation function. ϵ(t+1) is a learnable parameter that controls the importance of a node's own features in neighborhood aggregation.

After passing through the GIN layer, H(T) ∈ R|V |×d(T) is taken as the final view association embedding E ∈ R|V |×de , i.e., E = H(T) . The embedding loss function is:

where cos(ei, ej) represents the cosine similarity between the embeddings of views Vi and Vj, and ˆ A = A+ I, where I is the identity matrix.

3.3 ViewAssociation-Guided DynamicWeighting Fusion Module

View Association-Guided Representation Learning. In multi-view data, each sample in the same dataset has multiple related feature representations which leads to differences in the dimensionality of the feature vectors of samples across different views. This requires unifying the dimensionality of feature vectors from different views to facilitate subsequent computations. Therefore, the first step is to map instances from different views in the original feature space to the same latent space:

where W ∈ Rd×mv and b ∈ Rd are learnable parameters.

Then, to enable view association embeddings to better adapt to downstream tasks, an attention-like mechanism, called the Adjustment Module(AM), is designed to guide the sample in focusing on distinct features across different views. Specifically, a two-layer fully connected network is used to transform the view association embeddings into guiding vectors:

where αv ∈ Rd is the guiding vectors of the v-th view, and σ is the sigmoid function. Successively, the guiding vector is element-wise multiplied (Hadamard product) with the vector after the sample is unified to the same dimension. Subsequently, the selected features are input into their respective learners.

where fv represents the learner of the v-th view consisting of one fully connected layer, followed by Batch Normalization and a ReLU activation function.

DynamicWeighting Fusion. After the feature disentanglement process, the features from each view are adjusted, and their quality may vary depending on the specific characteristics of each view. As a result, the importance and relevance of these features differ across views. Therefore, we introduce Dynamic Uncertainty(DU) to measure the uncertainty of each view [Cao et al., 2024]. The dynamic uncertainty for i-th view can be formulated as follows:

where C is the class number, µ is the mean of probability, and it holds µ = 1C . The distribution of probabilities after softmax offers critical insights into a view's uncertainty: A uniform distribution typically suggests high uncertainty, whereas a peaked distribution implies low uncertainty in predictions. In the multi-view fusion process, one view should dynamically perceive the changes of other views and modify its relative contribution to the multi-view system. Therefore, we introduce relative calibration as the weight for multi-view fusion:

Next, the fused features are obtained as follows

Finally, the fused features are passed to a softmax function for computing the class probability as follows:

3.4 Overall Objective Function

The AssoDMVC is trained with the following objective function in an end-to-end fashion:

where Lle is the view association embedding loss, λ is a tradeoff parameter, and Lce denotes the cross entropy loss, formulated as

4 Experiments

4.1 Datasets

Our experiments are conducted on six challenging multi-view classification datasets which include image, text, audio, depth and video datasets. (1) Animals with Attributes (AWA)[Lampert et al., 2013] dataset, which includes 30,475 images from 50 categories with seven view features. (2) NUS-WIDE128 (NUS)[Tang et al., 2016] dataset, which includes 43,800 samples from 128 categories with seven view features. (3) Reuters [Amini et al., 2009] dataset, which includes 111,740

samples from six categories with five multilingual view features. To enable the model to process this dataset, PCA is used to reduce the dimensions of all views to 1000. According to [Han et al., 2022; Liu et al., 2021], Gaussian noise is added to either 5-view or 3-view datasets, resulting in two versions named Reuters5 and Reuters3, respectively.(4) VoxCeleb [Nagrani et al., 2020] dataset, which includes 153,516 samples from 1,251 categories with five audio view features. (5) YoutubeFace dataset, which includes 3,425 videos from 1,595 different people with five view features. According to [Wang et al., 2022b], we use a subset of 31 categories from this dataset, with a total of 101,499 frames.

We employ three measures to evaluate the performance of each method, which are accuracy, precision and F1 score [Liang et al., 2024].

4.2 Experimental Results with Other Methods

To validate the effectiveness of the our method, comprehensive comparison experiments are conduced with eight relatedweighting-basedmulti-viewclassificationmethods. The compared methods can be classified into the following two groups according to the level of weighting:

1. The first category is the feature level including EmbraceNet, AWDR and RAMC [Jiang et al., 2022]. EmbraceNet assigns 1 to the weight value of one view while 0 to others for each example according to a multinomial distribution. AWDR is an adaptive-weighting discriminative regression approach. Following [Yang et al., 2019], the parameter λ is chosen from the set {10-3 , 10-2 , · · · , 103 }, while k varies within the range {1, 3, · · · , 9}. RAMC employs an L2,1-norm loss function to acquire a joint weighted projection space across all views. This method preserves the correlation and diversity among views through a self-supervised weighting strategy. Similarly, the parameter λ is chosen from the set {10-3 , 10-2 , · · · , 103 } and k ranges from the range {1, 3, · · · , 9}.
2. The second category is the decision level including BV, SSV, MR [Liang et al., 2021], TMC [Han et al., 2022], TMOA [Liu et al., 2022], ETMC [Han et al., 2023], ECML [Xu et al., 2024], RMVC [Yue et al., 2025] and TUNED [Huang et al., 2025]. BV assigns 1 to the weight value of the view with the best performance while 0 to others according to whole classification performance of each view. MR assigns 1 to the weight value of the view with the best performance while 0 to others for each example according to the classification performance of each view of each example. SSV assigns the same values to all views. TMC, TMOA, ETMC, ECML and RMVC are trusted fusion methods.

For the proposed AssoDMVC, in order to make the model elegant and lightweight, we set each module to contain one or two fully connected layer, and the number of neurons in the fully connected layer is selected from [128, 256]. We initialize the learnable parameters ϵ in GIN by 0. The results in Tables 1, are presented through the mean metric value and the standard deviation obtained from 5-fold cross-validation. From Table 1, the following observations can be made:

1. The AssoDMVC method shows remarkable performance in accuracy, especially on the VoxCeleb and YoutubeFace datasets, achieving 93.85% and 86.21%, respectively. Additionally, AssoDMVC consistently maintained top or near-top performance across most datasets, including AWA, NUS, Reuters5, and Reuters3, achieving 90.86%, 74.62%, 81.79%, and 86.04%, respectively. The F1 score, which balances precision and recall, provides a comprehensive measure of classification performance. The AssoDMVC shows significant advantages in F1 scores as well, especially on the AWA, NUS, and Reuters5 datasets, where it achieves 88.38%, 74.50%, and 81.77%, respectively, reflecting a good balance between precision and recall.
2. The AssoDMVC shows stable performance across multiple datasets, not only outperforming SOTA methods on some datasets but also matching or coming close to the best performing methods on others. This demonstrates its broad applicability in multi-view learning tasks.

Overall, the AssoDMVC method outperforms existing SOTA methods across several key performance metrics. By introducing new strategies for view relation and dynamic fusion, AssoDMVC handles the complexity and diversity of multiview data more effectively, demonstrating stronger adaptability and higher model performance.

4.3 Further Analysis

In the part, we further analysis our model from three aspects: each module effectiveness, hyper-parameter sensibility, impact of different weight strategy.

Ablation Experiments. To validate the effectiveness of the components in AssoDMVC, we performed experiments to assess the impact of the view association encoding module (RM), adjustment module (AM), and dynamic weighting (DW) on the experimental results. Results of the ablation study are presented in Table 2.

As shown in the third and fourth rows of the table, adding the RM results in a significant improvement in performance across all datasets. Specifically, the model performs better with the RM, achieving 90.49% on the AWA dataset, which is significantly higher than the 82.27% without it. The AM also has a significant impact on the experimental results. Specifically, the accuracy on the AWA dataset increases from 90.49% to 90.76% when the AM is added. As seen in the first and second rows of the table, adding the DW leads to significant gains. For example, the accuracy on NUS increases from 69.84% to 73.17%. The last row of the table presents the results when all three modules are combined. It is evident that the combination of all three modules yields the superior performance across all datasets.

Parameter Sensitive Analysis. In the View Association Encoding Module, to reduce computational cost, we select k samples for graph construction. To assess the impact of different k values on the experimental accuracy, we perform a comparison using various values of k. The results are shown in the Fig. 3. The accuracy remains relatively consistent across different k values, suggesting that we can select an appropriate k without being overly concerned about its impact on model performance.

k

k

Figure 3: Impact of k values on performance

DifferentWeight Strategies. We introduced three different weighting strategies for the multi-view fusion process: Average Weighting (Avg), Learnable Weighting (LW), and No Weighting (NW), and compared them with Dynamic Weighting (DW).

1. Average Weighting (Avg): In this strategy, equal weights are assigned to all views, meaning no differentiation is made between the views.
2. Learnable Weighting (LW): This strategy allows the model to learn the optimal weights for each view during the training process.
3. No Weighting (NW): In the No Weighting strategy, no explicit weights are applied to the views.

In the Fig. 4, Dynamic Weighting (DW) demonstrates a clear advantage. Specifically, on the Reuters3 dataset, DW achieved the highest performance at 86.04%, outperforming other weighting strategies such as Average Weighting (Avg), Learnable Weighting (LW), and No Weighting (NW), highlighting its effectiveness and superiority on this dataset. Similarly, on the VoxCeleb dataset, DW also performed well, reaching 93.85%, showing a slight improvement over other methods.

Visualization. To provide deeper insights into AssoDMVC, we visualize the learned view association embeddings on the AWA dataset. As shown in Fig. 5, views 0, 2, 3, and 4 exhibit notably high correlations, suggesting strong mutual dependencies among these views. These results indicate that AssoDMVC effectively guides multi-view fusion by modeling the associations between views.

5 Conclusion

In this paper, we proposed a view-association-guided dynamic multi-view classification method, which effectively models the relationships between views and dynamically adjusts their contributions. By introducing a view association encoding module and a relative calibration mechanism, our method enhances view interaction and improves fusion accuracy. Experimental results show that our approach outperforms traditional fusion techniques, providing a more robust and flexible solution for multi-view learning tasks. Future work could explore more effective view interaction mechanisms to further enhance the information fusion and collaboration between views.

Acknowledgements

This work was supported by National Natural Science Foundation of China (Nos. 62306171, 62406218, T2495251, T2495253, 62476160, 62376146, 62373233), the Science and Technology Major Project of Shanxi (No. 202201020101006), and Fundamental Research Program of Shanxi Province (No. 202203021222183).