Integrating Vision-Language Semantic Graphs in Multi-View Clustering

1 Introduction

In recent years, multi-view clustering (MVC) has emerged as a pivotal component in the realms of cross-modal representation learning and data-driven decision-making. This technique has been shown to be highly effective in various domains, such as image [Guan et al., 2024b; Guan et al., 2024a] and video analysis [Xu and Wei, 2021]. MVC methods capitalize on exploiting the consistency and complementary information of multiple views to group samples into distinct clusters [Ren et al., 2022; Yan et al., 2024; Wang et al., 2021; Wang et al., 2023; Zhang et al., 2018a; Ling et al., 2023]. Currently, self-supervised learning has achieved significant improvements in multi-view clustering [Zhou et al., 2023; Zhou et al., 2024b]. At the core of this approach lies the extraction and utilization of the intrinsic representational attributes of the data to further enhance clustering performance.

∗Corresponding author.

Nevertheless, previous MVC methods still encounter challenges when it comes to extracting latent features from realworld data, especially in scenarios involving high-resolution color images and high-dimensional features.

One potential resolution to these problems involves the incorporation of pre-trained models within the encoding process. Many approaches have used pre-trained models to explore the limits of clustering in single-view data [Adaloglou et al., 2023; Shen et al., 2023; Cai et al., 2023]. However, there still exist challenges in the unsupervised tuning of pretrained models for multi-view data. First, for k-means, it usually leads to unbalanced clustering [Yang et al., 2017; Cui et al., 2023] and is mainly applicable to data samples that are uniformly dispersed around the center [Van Gansbeke et al., 2020]. On the other hand, when it comes to multi-view image data, that are similar in feature space do not always have the same semantic category [Van Gansbeke et al., 2020] and therefore must be treated as noisy pairs, which will result in noisy spatial relations in images embedded.

In this paper, combining the classic image-text dual encoders model CLIP (Contrastive Language-Image Pretraining [Radford et al., 2021]), we propose a novel multiview clustering method using pre-trained models. In the face of the two previous limitations, we introduce the filter-based graph to utilize filters compatible with homogeneous and heterogeneous graphs to combat embedding noise and spatial imbalance, resulting in representations that balance spatially homophilous and heterophilous relationships. In addition, we use a joint graph to combine node features and edge relationships, where embeddings can be used to efficiently combine global feature relationships among multi-view data. To the best of our knowledge, our method is the first to apply largescale pre-trained models multi-view data clustering.

Our main contributions can be summarized as follows:

• We introduce large-scale vision-language pre-trained models to multi-view clustering by proposing a method

to counteract the embedded spatial noise and imbalance of the pre-trained encoder with graph filters and a graph joint process.

• Our method efficiently applies multi-view image clustering to large-scale multi-view image datasets, including the high-resolution multi-channel multi-view image dataset MVImgNet, achieving an accuracy of 82%.
• Our method achieves good performance for the clustering task on top of its untrained dataset, extending the zero-shot performance of CLIP.

2 RelatedWork

2.1 Multi-ViewClustering

The collection of data from multiple perspectives and sources has become commonplace with the development of multimedia technology. Multi-view clustering (MVC) methods leverage complementary information from different views of the same instance to address the limitations of traditional clustering methods [Wu et al., 2024; Ren et al., 2024]. Several algorithms and techniques have been proposed for MVC, including classic methods like spectral clustering, as well as more recent developments in deep learning. Recently, deep learning has emerged as a powerful tool in the field of MVC, with numerous proposed architectures of deep neural networks tailored for this endeavor, including the introduction of new constraints, feature learning techniques, and graph filtering frameworks. LT-MSC [Zhang et al., 2015] introduces a low-rank tensor constraint to explore the complementary information from multiple views. [Xu et al., 2021b] proposes a framework for contrastive multi-view clustering (MFLVC) that utilizes multi-level feature learning to improve clustering effectiveness. [Wen et al., 2024] suggests an adaptive hybrid graph filter based on homophily degree, which dynamically captures both low and high-frequency information to improve graph clustering. These methods have advanced the field of multi-view clustering significantly. However, there has been no exploration of incorporating the recently popular visionlanguage pre-training models into MVC, which has the potential to elevate multi-view clustering to unprecedented heights.

2.2 Vision-Language Pre-trainingModels

Vision-Language Pre-training (VLP) models aligning multimodal data in a common feature space have been applied in various areas such as large vision language model [Zhou et al., 2024a] and adversarial attack [Dong et al., 2023]. They can be categorized into two main groups: those that use language-based training strategies, including Mask Language Modeling, such as mask language/region modeling VisualBert [Li et al., 2019a], or autoregressive language modeling, such as image captioning and text-based image generation DALL-E [Ramesh et al., 2021]. The other category is to utilize cross-modal contrastive learning to align the visual and textual information into a unified semantic space, e.g. CLIP [Radford et al., 2021]. VLP aims to model the interaction between images and texts. Unicoder-VL [Li et al., 2020] combines visual and textual embeddings, feeding them into a single encoder. In contrast, CLIP [Radford et al., 2021] obtain visual and textual embeddings with separate encoders.

3 Method

As shown in Figures 1 and 2, the proposed method consists of two steps: 1) Semantic Graph Construction: Based on nodes characterized by images, we propose a method for constructing graph data, incorporating prior knowledge from the design of the CLIP model, we further construct edge relationship. This includes an unsupervised step of meaningful noun selection we refer to as word filtering, aiming to identify better nouns that accurately convey the overall details depicted in the image; 2) Adaptive Hybrid Graph Filtering: Herein, we design a filtering method that further processes based on the previously presented images and node-edge relationships. This method accounts for the heterogeneity arising from inconsistencies in their representations, resulting in an adaptive filter of Nodes and Edges that learns both consistency and complementarity, containing a graph joint process that combines features of nodes and edges, maximizes the use of consistency and complementarity across different views, while also leveraging that of node features and edge features as much as possible.

3.1 Semantic Graph Construction

Since the node feature of our graph to process is the embeddings of raw feature, it would be less meaningful when we directly use the similarity matrix between CLIP encoding results or original images as the adjacency matrix: the former does not take full advantage of the core of CLIP dual encoder design and training for computing image-text similarity, and the latter introduces noise in the edge relations because the original images are not encoded.

Given the application scenario of CLIP's design, a natural approach is to unsupervisedly select some words and then refer to the way CLIP predicts between them. By adding prefix words, we would calculate the cosine similarity between images and words to form the image-text bipartite graph.

In terms of word selection, to maximize the effectiveness of the final matrix representation, the meanings of the words should represent and distinguish the embeddings of the image dataset as much as possible without being overly generalized. In other words, this requires the identification of nouns that accurately convey the overall details depicted in the image. So first and foremost, we design a noun filter for selecting nouns unsupervisedly in the WordNet dataset [Miller, 1995] as raw noun set Nr to select a suitable noun set N.

Noun Filter

Firstly, to exclude some nouns overly generalized, i.e., Thing, Object, Item, Matter, Entity, we initially exclude nouns from the WordNet that are close to the centers of almost all words. More precisely, we use cosine similarity between embeddings to measure the similarity of objects, as used in CLIP training. This selection can be formulated as selecting set Nd follows:

Let t be an element of the set Nd. This membership is defined such that t ∈ Nd if and only if the Cosine Distance CD between the text encoder output f(t, θt) and the centroid CN is greater than or equal to a threshold ε. Formally, we first select nouns t satisfied:

we use set Nd to mark the selected nouns t. Where, ft(·) refers to the text encoder parameterized by θt, Cosine Distance between any two vectors a and b is defined as CD(a, b) = 1 - a·b ∥a∥∥b∥ , ε refers to the threshold hyperparameter, in the following process and experimental setup, we maintain the threshold hyperparameter ε at 0.05. The centroid CN is calculated as the mean of normalized text embeddings Um for each noun m in the raw WordNet noun set Nr, expressed as:

where k is the number of nouns in Nr.

To further select an appropriate subset of nouns, considering other priors that can be integrated into the noun filter process that is merely the number of categories and the image embeddings themselves, it is noteworthy that experiments demonstrate over 70% accuracy in direct clustering of the embeddings of images from common datasets. Under these circumstances, for the given multi-view image datasets I subjected to clustering, we cluster the directly concatenated encoded results. Based on the clustering result, we select the final nouns N that satisfied:

where normalized centroids CIi are computed based on the embeddings of each dataset and their respective clustering labels as:

(4)

where for any specific sample m, CLm denotes the label assigned to the sample by the k-means clustering algorithm. Here, wi represents the number of samples in I with CLm = i, mj refers to the j-th view of the sample m, and v is the number of views in I, fi(·) refers to image encoder parameterized by θi. Additionally, εth is defined as a dynamic threshold that is set such that the number of nouns meeting the criterion outlined in Eq. (3) is equal to the hyperparameter β (i.e. the total number of nouns selected equal to βNClass, where NClassrefers to the classes number of image dataset) for each cluster label i.

In the following content, we call the selected nouns set as filtered nouns set N.

Graph Construction with Filtered Nouns

Drawing upon the methodology of CLIP, particularly its approach to calculating result probabilities, we proceed to construct a comprehensive graph dataset utilizing the Filtered nouns obtained after the initial step. This is achieved by first creating a cosine similarity matrix derived from the embeddings of both the images and the Filtered nouns. Notably, diverging from the standard application of CLIP, in our approach, the constructed graph-text bipartite graph utilizes the cosine similarity matrix directly as the adjacency matrix A, as opposed to the softmax operation typically employed in CLIP. This process is described by the following equation:

where Bv represents the adjacency matrix of image-nouns bipartite graph based on cosine similarity in the v-th view. ZvI j refers to the j-th row of ZI in v-th view, ZNk refers to the k-th row of ZN , mϵI v , refer to the j-th image of dataset I 's v-th view, tϵI v , refer to the k-th nouns of filtered nouns set N, fi(·) refers to image encoder parameterized by θi, ft(·) refers to text encoder parameterized by θt. N refers to row normalization.

Conclusively, within the computed image-text bipartite graph, since the image is our only clustering target, we have chosen an approach that involves multiplying the image-text similarity matrix with its transpose. This operation produces the final adjacency matrix Av of v-th view, that is,

Av would then be utilized for graph clustering. This strategy allows the clustering to also take into account the intrinsic connection between the image and its corresponding text.

For the above process, in the case of using CLIP, we use the pre-trained image encoder as fi(·) and text encoder with the prompt template A photo of a nouns as ft(·). We construct the adjacency matrix of the semantic graph to characterize edge relationships based on the multi-view dataset and unsupervised selection of nouns fromWordNet.

3.2 Spectral Self-Supervised Learning with Adaptive Hybrid Graph Filter

Graph Joint Process

Considering that the semantic graph constructed in Section 3.1 may have some noisy edges, i.e., there may be inconsistencies between image embedding and word. It is feasible to use it directly for filtering, but its dominance in the filtering operation may lead to excessive loss of node feature information. Thus we try to utilize image embedding to further correct the semantic graph.

Specifically, we implement a graph joint process, which injects the nodes' original image features into the semantic graph. We first explore A from Eq. (6) as follows:

where fa(·) represents deep auto-encoder, θa are learning parameters of the autoencoder. ZA are the encoded outputs of A. Since the graph data Av of each view is a semantic graph generated by the same method, in order to motivate further extraction of inter-view consistency of the model, we adopt the practice of using a shared deep auto-encoder parameter θa for the semantic graphs of all views. We actually consider here the adjacency matrix A of the semantic graph being constructed as a kind of feature information, i.e., an edge feature, and thus we adopt a similar encoding operation as for image features.

Next, the image feature information is injected into the semantic graph. Consensus information between the image embedding and the edge features is then utilized to correct the semantic graph. To be more specific, we perform the following:

where fi(·) represents pre-trained image encoder with parameters θi, the dimension of S is the same as the original adjacency matrix A of semantic graph, we regard graph joint matrix S as the modified graph. ZI is obtained through fiθ, fiθ is the pre-trained image encoder. As S joins the image feature information of the nodes and the edge feature information of the semantic graph, it is more reliable than the adjacency matrix of the original semantic graph.

Additionally, we propose discretizing and sparsifying S. This approach can save memory and accelerate computation, while also removing weakly correlated edge noise. We perform the following operation to remove the weakly correlated edge noise:

where N is the number of nodes and Sdis is the discretized S. All S below denote the discretized S, i.e., Sdis if not otherwise specified.

Adaptive Hybrid Graph Filter

After completing the graph joint process, a natural idea is to leverage popular graph neural networks [Zhou et al., 2020] to explore structural information and feature information to enhance clustering performance. In terms of filter selection for graph filtering of the resulting semantic graph, the most common choice is to use the widely used Graph Convolutional Neural Network (GCN) to achieve this goal. Previous studies have pointed out that GCN is essentially a low-pass filter [Nt and Maehara, 2019]. From a spectral perspective, GCN only captures low-frequency information on the graph, completely losing high-frequency information. However, Bo et al. point out that both low-frequency and high-frequency information are equally crucial for learning representations of nodes on the graph [Bo et al., 2021]. Completely discarding high-frequency information would lead to significant information loss, as in the constructed semantic graph, information is present both in the homogeneous and heterogeneous components.

A better idea is to use weighted high-pass and low-pass filters for obtaining high and low frequencies on the graph respectively to retain as much information as possible, and we design an adaptive hybrid graph filter as follows:

where Hhybrid represents the output of the adaptive hybrid graph filter. hr is a learnable parameter that measures the homophily degree and is used to control the adaptive process of the hybrid graph filter, which will be calculated in Eq. (11). The diagonal matrix Dii = ∑j aij represents the degree ma-

trix. ˜L is the normalized Laplace matrix. k is the order of the filter.

In Eq. (10), (˜A)kZI represents the low-pass filter and (˜L)kZI represents the high-pass filter.

Instead of manually setting the weights for the low-pass and high-pass filters, to further enhance the universality of the filtering mechanism, we have implemented an adaptive mechanism for the hybrid graph filter based on the homophily ratio. Homophily edges are edges on a graph that connect two similar nodes, and homophily ratio is a measure of the proportion of homophily edges on a graph. If a graph has a high homophily ratio, there will be more homophily edges, and from a graph signal processing perspective, low-frequency signals dominate the graph, and conversely, high-frequency signals dominate the graph. In other words, the homophily ratio of a graph can reflect the frequency distribution and the proportion of signals on the graph. Therefore, we considered assigning weights to hybrid graph filters using the homophily ratio and designing adaptive mechanisms. As the real label information is unavailable in the unsupervised setting, we estimate the homophily ratio (hr) using pseudo-labels:

where, SUM(·) denotes the summation operation, ⊙ denotes the Hadamard product, and P ∈ {0, 1}n×c is the one-hot encoding of the pseudo label. Specifically, for the obtained semantic graph, if it has a high homophily ratio, then the low-pass filter in the adaptive hybrid filter will play a major role, and conversely, the high-pass filter will play a major role, which just matches the frequency distribution on the graph, thus extracting of information effectively. The final learned consensus embedding of n views is set as an adaptive weighted sum of the individual view embeddings after the graph filter: H = ∑nv=1 ωvhHv

3.3 Model Optimization

In IVSGMV, learning from previous multi-view clustering studies [Zhao et al., 2021; Ren et al., 2022], in order to guide deep auto-encoder fa(·) and learn consistency and complementarity between views, we apply reconstruction loss on A as follows:

where LCE(·; ·) denotes cross-entropy loss.

We obtain the LKL from the soft clustering distribution Q and the target distribution P as follows:

where LKL(·; ·) denotes Kullback-Leibler (KL) divergence loss [Kullback and Leibler, 1951], qv ij ∈ Qv describes the probability that node i in the v-th view belongs to the center of cluster j. Pv represents the target distribution of nodes embedding Hv in the v-th view. Q and P denote the soft and target distributions, respectively, of the consensus embedding H. LKL encourages the soft distribution of each view to match the target distribution of the final consensus embedding H. Additionally, it enhances the consistency between the soft distribution and the target distribution of the consensus embedding.

Eventually, the loss of IVSGMV is defined as:

4 Experiments

4.1 Datasets

As shown in Table 1, we use the following four real-world multi-view datasets in our study. MNIST [LeCun et al., 1998] is a widely used dataset of handwritten digits from 0 to 9. The Fashion dataset [Xiao et al., 2017] comprises images of various fashion items, including T-shirts, dresses, coats, etc. The COIL dataset [Nene et al., 1996] contains images of various objects, such as cups, ducks, and blocks, shown in different poses. We use multi-view datasets derived from origin datasets: Multi-COIL-10, Multi-COIL-20, Multi-MNIST, and Multi-Fashion. Each dataset includes multiple views of each example, all randomly sampled from the same category. In Multi-COIL-10 (K = 10) and Multi-COIL-20 (K = 20), different views of an object correspond to various poses, but retain the same label. In Multi-MNIST, different views of a digit represent the same digit written by different individuals. In Multi-Fashion, different views of a product category signify different fashionable designs for that category. MVImgNet [Yu et al., 2023] is a multi-view image dataset presented with a large scale, high accuracy, and large diversity, providing an average of 30 image views for each sample, based on its "MVlmgNet by categories" subset that is available in 2023, we select the most differentiated 3 views to build a 3-views images dataset in 14 categories.

4.2 Comparison with State-of-the-Art Methods

Comparison Methods The comparison methods include three single-view clustering methods: K-means [MacQueen, 1967], β-VAE (β-VAE: learning basic visual concepts with a constrained variational framework [Higgins et al., 2017]), and VaDE (variational deep embedding: an unsupervised and generative approach to clustering [Jiang et al., 2017]), the input of which is the concatenation of all views, and five stateof-the-art MVC methods: BMVC (binary multi-view clustering [Zhang et al., 2018b]), SAMVC (self-paced and autoweighted multi-view clustering [Ren et al., 2020]), RMSL (reciprocal multi-layer subspace learning for multi-view clustering [Li et al., 2019b]), DEMVC (deep embedded multiview clustering with collaborative training [Xu et al., 2021a]), FMVACC (fast multi-view anchor-correspondence clustering [Wang et al., 2022]), GCFAggMVC (Global and Cross-view Feature Aggregation for Multi-view Clustering [Yan et al., 2023]), MFLVC (Multi-level feature learning for contrastive multi-view clustering [Xu et al., 2022c]), DIMVC (Deep incomplete multi-view clustering via mining cluster complementarity [Xu et al., 2022a]), SDMVC (Self-supervised discriminative feature learning for deep multi-view clustering [Xu et al., 2022b])

Evaluation Metrics

We evaluate the effectiveness of clustering by three commonly used metrics, i.e., clustering accuracy (ACC), normalized mutual information (NMI), and adjusted rand index (ARI). A higher value of each evaluation metric indicates a better clustering performance.

Results

Tables 2 and 3 shows the quantitative comparison between the proposed method and baseline models for several datasets. IVSGMV achieved superior performance compared to baselines on all datasets, and the state-of-theart comparison method still mighty underperforms our proposed method on the multichannel high-resolution multiview dataset MVImgNet even when CLIP encoding embeddings are used as inputs. This reflects our full application of large-scale pre-trained model CLIP.

Vision-Language Model Implementation

For the pre-trained encoder, we used the CLIP ViTL/14@336px model, whose visual and text backbones are ViT [Dosovitskiy et al., 2020].

4.3 Ablation Studies

Effect of Semantic Graph Construction and Pre-trained Encoder

To understand the importance of our graph construction method and pre-trained encoder. We conducted three ablation experiments on both to analyze their impact on the performance of the IVSGMV in the Multi-Fashion dataset, where fiθ represents the use of pre-trained self-encoder as node feature and w/ofiθ represents the direct use of the original image features without encoder, A represents the use of the proposed adjacency matrix construction method. As alternative constructions of A, Ae and Ar refer to graph construction method accord to the spectral clustering, representing the use of affinity matrix based on cosine similarity constructed using the raw features (Ar) and image embeddings (Ae) of the pre-trained encoder, respectively.

Effect of Graph Process Components

Graph joint aggregation matrix S and adaptive hybrid graph filter are important components of the IVSGMV. We conducted three more ablation experiments. w/o S represents the alternative implementation of graph joint aggregation matrix S with adjacency matrix A, w/o Hhybrid represents the alternative implementation of adaptive hybrid graph filter with a common GCN low-pass filter Hlp and w/o S&Hhybrid represents the combination of the above two. As Table 4 demonstrates, the performance of the model is greatly and adversely affected whether the graph joint aggregation matrix or adaptive hybrid graph filter, or both are eliminated.

The results of ablation Table 4 show that compared to Ae and Ar, the semantic graph construction method of A leads to 1.5% and 3.8% ACC improvement, and the pre-trained encoder fiθ brings about 18.7% accuracy improvement. Also, the graph joint aggregation and hybrid graph filter lead to 0.8% and 2.5% ACC improvement.

Notably, the semantic graph construction method of A also delivers 1.3% and 7.3% ACC improvement compared to Ae and Ar without the use of fiθ, respectively, which demonstrates the methodological superiority of semantic graph construction.

Parameter Sensitivity Analysis

The sensitivity analysis for order is on Figure 3. From the spatial perspective, order controls the aggregation order of the graph filter. The higher order enables nodes to aggregate information from more distant ones, while nodes can only access feature information of closer nodes in lower order.

Visualization of Consensus Embedding H

Figure 4 visualizes the consensus embedding H of our model on Multi-COIL-10, Multi-COIL-20, and MVImgNet.

5 Conclusion

In this study, we introduce large-scale pre-trained models into the field of multi-view clustering. We analyze the challenges of unsupervised fine-tuning pre-trained models and combat imbalance and noise relations in the embedding space with the help of building semantic graphs and graph filters. For semantic graph building, we propose an unsupervised construction process based on the CLIP model priori, which builds a graph-word bipartite graph by adaptively selecting words from the WordNet, and further obtains the adjacency matrices that imply semantic relations. On the graph clustering model, we apply an adaptive hybrid graph filter for multi-view clustering to adaptively mine the low and highfrequency information in the graph to learn distinguishable node embeddings, which in turn mines the homogeneous and heterogeneous information among embedding relations of the pre-trained model. In addition, the joint graph process is used to construct filters to enhance the distinguishability between low and high-frequency signals, where threshold discretization is applied to combat noise. Our IVSGMV has good performance on several multi-view datasets, extending the application of multi-view clustering to realistic image datasets while maintaining competitive performance on other datasets.

Acknowledgements

This work is supported in part by Shenzhen Science and Technology Program under grants JCYJ20230807115959041 and JCYJ20230807120010021. Lifang He is partially supported by the NSF grants (MRI-2215789, IIS-1909879, IIS2319451), NIH grant under R21EY034179, and Lehigh's grants under Accelerator and CORE.