Resisting Label Drift: Real-Time Multi-View Clustering with Semantic Consistency

1 Introduction

In practical applications, data is routinely gathered from various independent sources, resulting in multi-view data which inherently exhibits heterogeneity across different perspectives or views [Liang et al., 2025; Liu et al., 2025]. Due to the increasing prevalence of open-world problems involving such data, especially in the unsupervised domain, learning methodologies tailored for this data format have become a prominent area of research. Clustering, a foundational unsupervised task, aims to partition data into non-overlapping groups [Liu et al., 2023; Xiao et al., 2023]. In the context of multiple views, Multi-view clustering (MVC) approaches are designed to effectively utilize the data by uncovering both the consistency and complementarity that exist across the views. A wide array of MVC techniques has been developed, encompassing subspace-based methods [Wei et al., 2023; Peng et al., 2022; Qin et al., 2025], kernel-based methods

∗Corresponding author.

[Wu et al., 2025; Zhang et al., 2025b], graph-based methods [Pan and Kang, 2021; Xia et al., 2022; He et al., 2025], and deep learning-based architectures [Cui et al., 2025; Wang et al., 2025], among others. Despite their achievements, most of these fall under the category of static MVC, meaning they are restricted to scenarios where the dataset is fixed. This condition is frequently violated in real-world settings.

A more challenging yet common scenario involves data that is continuously accumulated and updated, forming a dynamic data stream [De Lange et al., 2021]. For instance, social media platforms like YouTube continuously ingest multiview streams (images, texts, videos) generated by users every second. Similarly, financial markets rely on high-frequency feeds of stock prices, news headlines, and trading volumes. Analyzing and clustering such continuously evolving streams presents significantly greater challenges compared to static scenarios.

A naive approach to address this challenge is to accumulate all observed data and repeatedly apply conventional MVC techniques to the aggregated dataset. However, this strategy is highly inefficient, incurring prohibitive storage costs and computational overhead. Moreover, it necessitates repeated access to historical data, which may be restricted due to privacy or storage constraints. To mitigate these issues, several incremental multi-view clustering algorithms have been proposed. Nevertheless, they suffer from two critical limitations. First, most methods lack real-time capability, as they fail to output instant clustering results for the current batch, often requiring offline post-processing. Second, and more critically, they overlook semantic consistency across data batches. For instance, a cluster representing the semantic concept of "Bird" might be assigned the label "ID-1" in the first batch, but the exact same object type could be randomly reassigned to "ID-5" in the next batch simply due to initialization differences. Without a mechanism to map cluster labels across time steps, these methods cannot consistently track evolving clusters, failing to provide interpretable, semantically continuous results.

To address these challenges, we propose a real-time semantic consistency incremental multi-view clustering (RTSC) framework. Our method integrates incoming data with a compact historical knowledge base via a consensus bipartite graph under a k-connectivity constraint, enabling real-time clustering without the need for post-processing. Simultaneously, we employ an adaptive diversity-aware selection mechanism to dynamically update the knowledge base. Furthermore, we introduce a semantic alignment strategy based on consensus centers to ensure label consistency over time.

The following three points detail the primary contributions of this paper:

1. We introduce an adaptive, diversity-aware sample selection mechanism that dynamically updates the knowledge base by balancing representativeness and redundancy to efficiently retain valuable historical structure.
2. We develop a semantic alignment technique to ensure that cluster labels maintain consistent meaning over time. To our knowledge, this is the first work to explicitly address the semantic consistency of cluster labels in incremental multi-view clustering.
3. We introduce a multi-view clustering framework for incremental instance scenarios. Experimental results across multiple datasets demonstrate the effectiveness and efficiency of the proposed framework.

2 RelatedWork

2.1 Static Multi-viewClustering

In recent years, diverse methodologies have emerged to address MVC, encompassing techniques grounded in matrix factorization, graph-based strategies, and deep learning frameworks. Methods utilizing matrix factorization typically leverage factorization techniques to derive a shared latent representation for clustering purposes. For instance, [Yang et al., 2024] proposed an efficient and scalable anchor-based multiview clustering method that directly obtains low-dimensional consensus embedding by factorizing the large-scale consensus graph. In a similar vein, [Liang et al., 2020] introduced a non-negative matrix factorization model with co-orthogonal constraints for multi-view clustering, which enhances the orthogonality of the basis and representation matrices to improve clustering performance. Graph-based approaches construct individual graph embedding matrices for each view by leveraging intrinsic data structural information, followed by fusing them into a unified graph matrix through learned weights. For example, [Tan et al., 2023] proposed a unified multi-view clustering framework that learns the manifold topological structure of data and explores sample-level crossview consistency to better preserve consistent local structures across views. [Liu et al., 2024] learned a consistent anchor graph with a k-connectivity constraint to directly generate cluster labels while filtering out view-specific noise. In recent years, following the advancements in deep learning, several studies have adopted deep neural networks to learn high-level feature representations for this task [Xu et al., 2025], [Bian et al., 2025], [Cui et al., 2026]. While these varied methods have demonstrated promising results, most are tailored for static MVC configurations, operating under the assumption that all data instances and views are fully available in advance. This premise, however, is frequently unrealistic in open-world environments.

2.2 Dynamic Multi-viewClustering

In practical applications, data are often collected continuously, resulting in evolving data streams. Conventional static MVC approaches are generally ineffective in handling such dynamic data, as they typically require storing and reprocessing all available data whenever new samples arrive. In the instance incremental learning scenario, researchers conducted preliminary explorations. For instance, [Shao et al., 2016] laid foundational work by adapting the conventional non-negative matrix factorization into an online framework by introducing a regularization term that enforces a consistent base matrix. Building on matrix factorization techniques, [Hu and Chen, 2019] subsequently leveraged matrix factorization to simultaneously learn cluster indicators and a unified base matrix, thereby facilitating effective knowledge transfer. Taking a different approach, [Huang et al., 2019] tackled this problem by developing a multi-view support vector domain description model coupled with a multi-view cluster labeling model. Further diversifying the methodologies, [Zhang et al., 2024] utilized multi-level semantic anchor information as prior knowledge to steer the representation learning of incoming data. Most recently, [Zhang et al., 2025a] addressed a more complex challenge by employing reusable embedding and centroid modules with consistency regularization to efficiently handle the simultaneous arrival of new instances and views. Although the methods mentioned above have made some progress, they all perform clustering only after processing all the data. In practical applications, the subsequent data in the current batch is unpredictable, and it is often necessary to provide clustering results in real time after a batch of data arrives. Furthermore, they do not consider the semantic drift problem between clustering results of different batches of data.

3 The Proposed Framework

This section introduces a real-time semantic consistency incremental multi-view clustering framework designed for the incremental instance scenario in which samples arrive in sequence and cannot be fully revisited.

·

3.1 Notations and Problem Statement

Table 1 provides an overview of the main symbols employed in this article. This work focuses on multi-view clustering in the incremental instance scenario. In each iteration, a batch of multi-view observation samples arrives. The goal is to obtain k semantic labels through real-time clustering for each batch of data, with the clustering labels having the same semantic meaning across different batches. The main challenges in achieving this goal are as follows: 1) How to perform realtime clustering after a new batch of data arrives. 2) How to retain valuable historical information with limited computing resources. 3) How to achieve label semantic alignment in clustering across different batches. To address the challenges mentioned above, the proposed method designs three core components to tackle these problems, respectively.

3.2 One-step Incremental Multi-viewClustering with Memory Replay

Upon the arrival of the t-th batch data Xt = {Xtv }Vv=1 , we integrate the historical knowledge base with the new data for joint clustering. The clustering process of newly arriving data is softly guided by utilizing the implicit clustering structure within the historical knowledge base. Simultaneously, the historical knowledge base is integrated with the newly arriving data for unified graph learning, in order to present a clear clustering structure and achieve real-time, one-step clustering. Based on the above ideas, we define the following optimization objectives:

where the weight α is used to balance the effects of different views, Kt-1 v represents the historical knowledge base, and its construction method is described in section 3.3, Ctv and Ht represent the anchors matrix and anchor graph matrix at time t, respectively. The augmented graph matrix St is defined as a block matrix consisting of four blocks. The two diagonal blocks are zero matrices, the upper-right block is the transpose of Ht , and the lower-left block is Ht . LtS is the normalized Laplace matrix of St .

The number of connected components in St matches the number of zero eigenvalues of LtS . Since Ht is constructed from St , both share identical connectivity. Consequently, enforcing a rank condition on LtS is essentially equivalent to ensuring a specific connectivity structure in Ht . With Ht designed to contain exactly k connected components, the clustering assignments can be obtained directly, without any additional post-processing [Nie et al., 2016]. Each component corresponds to a group of nodes that are mutually reachable, forming the real-time cluster partitions.

3.3 Historical Knowledge Base Information Selection

The constructed historical knowledge base should implicitly contain the real-time clustering structure of the data stream while maintaining the representativeness of historical information. Based on this, the historical knowledge base Ktv should be adaptively updated with each new batch of data to adapt to the new data distribution. To enable the knowledge base to naturally embed the graph learning process, it should store a compact and easily learnable representative sample set. To ensure the knowledge base meets these characteristics, the specific process for constructing and updating the knowledge base information is designed as follows.

When t = 1, no prior information is available, and the historical knowledge base Kt-1 v is initialized as an empty set.

When t > 1, the process of building and updating the knowledge base is as follows. First, the current data Xt and the existing historical knowledge base Kt-1 are merged to form unified set Rt = [Xt ,Kt-1 ]. Then, each sample xi in Rt is evaluated using the reconstruction loss: loss(xi) =

∑Vv=1 αv∥Rtv(:,i) -CtvHt:,i ∥22 , where Ctv and Ht:,i are learned from objective function 1 at time t. A smaller reconstruction loss indicates that the sample is more easily learned as an anchor representation, thus carrying more stable and explicit structural information. The samples are ranked according to the magnitude of the reconstruction loss.

Simply selecting samples based on the magnitude of the reconstruction loss can lead to significant redundancy and class imbalance, as samples with smaller reconstruction losses are often concentrated in a few narrow regions of the feature space. To overcome this problem, we employ a diversityaware selection mechanism. In each iteration, the sample with lower reconstruction loss is prioritized as a candidate sample. If its similarity to any selected sample exceeds a given threshold θ, it is discarded; otherwise, it is accepted and added to the updated knowledge base. This process is repeated iteratively until no other samples can be selected. The similarity threshold θ plays a crucial role in balancing diversity and representativeness. To determine an appropriate value corresponding to a given sampling rate, we employ a binary search strategy to automatically adjust θ. The detailed algorithm for building and updating the knowledge base is shown in Algorithm 1.

Algorithm 1 Knowledge base construction and update algorithm

Input: Multi-view data Xt at time t, anchors Ct = {Ctv }Vv=1 and anchor graph Ht learned at time t, historical knowledge base Kt-1 , knowledge base size l.

Output:Updated knowledge base Kt at time t.

• 1: Initialize the knowledge base as an empty set;
• 2: Add anchors Ct learned at time t to the new knowledge base Kt = [Ct ];
• 3: Calculate the reconstruction loss for each sample in Xt and Kt-1 based on Ct and Ht and sort them;
• 4: Calculate the similarity between Xt , Kt-1 , and Ct ;
• 5: Adaptively adjust similarity threshold θ using a binarysearch strategy until knowledge base reaches size l;
• 6: Add samples in Xt and Kt-1 to update knowledge base Kt in sequence based on their ranking and θ.

3.4 Label Semantic Alignment

To address the issue of semantic drift and ensure that cluster labels across successive data batches maintain a consistent meaning, we introduce a semantic alignment mechanism based on consensus cluster center matching. The core idea is to establish an optimal one-to-one mapping between the clusters identified in the previous batch and those in the current batch.

First, for the current batch Xt and its initial clustering result Yt , we compute the cluster centers for each view by taking the mean of samples belonging to each cluster. These view-specific centers are then concatenated vertically to form a unified multi-view center matrix Ztcombined ∈ R∑ dv×k .

Then, we use the consensus aligned multi-view centers Zconsensus and the newly computed centers Ztcombined to determine the optimal correspondence. Specifically, this paper uses the cluster centers of the first batch of data as the consensus cluster centers. The gaussian kernel [Hu et al., 2010] is applied to compute the similarity matrix A ∈ Rk×k between all pairs of consensus centers and current centers. This similarity matrix measures how closely the i-th consensus cluster center semantically matches the j-th current cluster center.

Then, the alignment problem is then converted into a maximum weight bipartite matching problem, which is efficiently solved using the Hungarian Algorithm on the cost matrix M = -A. The resulting optimal pairing yields a label mapping M : {1, . . . , k} → {1, . . . , k}.

Finally, the initial cluster labels Y t are globally remapped using the derived mapping M to generate the semantically aligned labels Yt aligned . This procedure ensures that when a cluster from the previous batch corresponds to a cluster in the current batch, all samples currently assigned to that cluster are consistently reassigned the label of the matching cluster from the previous batch.

4 Optimization Strategy

To solve the above optimization problem, we propose an alternative iterative algorithm that updates one variable while keeping the other variables fixed and thus updates all variables alternately.

4.1 Solve Ctv by Fixing Other Variables

Let Rtv = [Xtv ,Kt-1 v ]. To solve for Ctv while keeping all other variables fixed, we consider the following subproblem:

Rewriting the Frobenius norm in Eq. (2) into the form of trace yields the following equivalent formula about Ctv :

The above formula can be transformed into the following equivalent problem:

where Btv = RtvHtT . Assuming that the singular value decomposition of Btv is given by Btv = UvΣVT v , the optimal solution of Ctv can then be derived as Ctv = UvVT v .

4.2 Solve Ht by Fixing Other Variables

When all other variables are held constant, finding Ht becomes equivalent to solving the following problem:

By choosing a sufficiently large value for ϕ, Eq. (5) can be reformulated as the following optimization problem [Fan, 1949]:

where Ft ∈ R(nt+m+l)×k denotes the indicator matrix. Following reference [Liu et al., 2022], we alternately optimize Ft and Ht . After obtaining the optimal Ft , we proceed to update Ht in Eq. (6). Following reference [Liu et al., 2022], we optimize each column of Ht separately, transforming optimization Eq. (5) into the following form:

ement. Reference [Nie et al., 2014] shows that Eq. (7) can be solved in closed form. More detailed optimization procedures for Ctv are provided in the Supplementary materials.

Algorithm 2 Algorithm for RTSC

Input Multi-view data stream {X1 , X2 , . . . XT }; the number of clusters k.

Output Real-time clustering results with semantic alignment.

• 1: for t = 1 : T do
• 2: Initialize Ctv , α and Ht ;
• 3: repeat
• 4: Update Cv by solving Eq. (2);
• 5: Update Ht by solving Eq. (7);
• 6: Update α by via Eq. (9);
• 7: until convergence
• 8: Output the cluster label of Xt partitioned by Ht ;
• 9: Update historical knowledge baseKt at time t according to Algorithm 1;
• 10: Semantic alignment of cluster label according to clustering centers alignment.
• 11: end for

4.3 Solve α by Fixing Other Variables

When all other variables are held constant, finding α becomes equivalent to solving the following problem:

where Ptv = ||Rtv -CtvHt ||2F . Based on the Cauchy-Schwarz inequality, the value of α can be derived directly as

4.4 Complexity Analysis

The calculation process of the proposed method is shown in Algorithm 2. The time complexity of the algorithm is mainly composed of the following aspects. The time complexity of updating Ctv is O(mntdv). The complexity of updating Ht is O(m2nt +m3 + ntmd). The time complexity of updating α is O(1). The time complexity of updating the historical knowledge base Kt is O((nt )2d). The time complexity of label semantic alignment is O(k2d + k3 + ntd). In general, the overall time complexity of the proposed algorithm is O(∑Tt=1 (nt )2d + m2n + k2d+mnd), where n = ∑Tt=1 nt .

5 Experimental Analysis

In this section, we present a comprehensive set of experiments to evaluate the effectiveness of our proposed approach. Due to space constraints, we have placed the convergence and complexity analysis and parameters analysis in the Supplementary materials.

5.1 Datasets

To ensure a thorough evaluation, our experiments utilize six widely recognized multi-view benchmark datasets: Animal 1 , NUSWIDEOBJ [Chua et al., 2009], YoutubeFace sel, YTF10 2 , YTF20 3 , and YTF50 4 . Detailed statistics of these real-world datasets are summarized in Table 2. To construct a streaming environment, the entire dataset is partitioned into multiple batches, with one batch being processed in each iteration. For each dataset, the number of data blocks is set to the number of samples divided by 2000 and rounded up.

5.2 Experimental Design

We evaluated our proposed method by comparing it with seven other multi-view clustering methods proposed in recent years. These include three dynamic multi-view clustering algorithms: MVC-IIV [Zhang et al., 2025a], ACMVC [Zhang et al., 2024], and OMVC [Shao et al., 2016], and four static multi-view clustering algorithms: FMGM [Lu et al., 2024], NpGC [Yu et al., 2024], LMVSC [Kang et al., 2020], and FPMVS [Wang et al., 2022].

Our method does not require adjusting coefficients to balance the effects of individual components. However, the number of anchors and the size of the historical knowledge base must be specified. In our experiments, the number of anchors is selected from the set [k, 2k, 3k], and the size of the historical knowledge base is selected from the set [m, n/4, n/2, 3n/4, n]. The comparison method uses the same number of anchor points as the proposed method. The comparison methods adjust and select their optimal parameters for fairness. To assess clustering performance, we adopt four widely used metrics: Accuracy (ACC), Purity (PUR), F-score and Adjusted Rand Index (ARI).

1 https://github.com/wangsiwei2010/large scale multi-view clustering datasets.

2 https://www.micc.unifi.it/resources/datasets/e-ytf/

3 https://www.micc.unifi.it/resources/datasets/e-ytf/

4 https://www.micc.unifi.it/resources/datasets/e-ytf/

5.3 Clustering Results with Batch Average

Table 3 shows the clustering performance of different clustering algorithms on six multi-view datasets. We performed clustering on each batch of data separately and then averaged the clustering results across all batches.

The experimental evaluation demonstrates that the proposed method consistently outperforms baseline approaches across diverse datasets. Our method achieves the highest accuracy on all six datasets, demonstrating its clustering performance in streaming data environments. For example, on the YTF10 dataset, the framework achieves the highest ACC, marking a significant improvement of over 10% compared to algorithms such as OMVC and FMCM. This superiority extends to the Animal datasets, where the method records the top ACC, F-score and ARI across seven different comparison methods, as well as the NUSWIDEOBJ dataset. Furthermore, the proposed method outperforms the comparative methods on all clustering accuracy metrics across the YTF10, YTF20, and YTF50 datasets. The results presented in the table indicate that the incremental multi-view clustering algorithm generally surpasses its static counterpart, thereby validating the efficacy of methods specifically engineered for streaming data. Furthermore, the consistent performance observed across diverse datasets, including object and animal imagery as well as video-based facial data, highlights the broad generalizability of our framework. The integration of adaptive knowledge base updating enables the model to retain essential historical information without being overwhelmed by obsolete patterns. The consistent gains over strong baselines, including state-of-the-art incremental and non-incremental multi-view clustering methods, confirm that our proposed framework can effectively handle incremental multi-view scenarios and retain valuable historical information.

5.4 Overall Clustering Results with Semantic Alignment

To evaluate the effectiveness of the proposed semantic alignment mechanism in mitigating label semantic drift across data batches, we conducted a semantic alignment experiment. For each batch of data, we obtain real-time cluster labels and align them with the previous labels. Once all batches of data have arrived, we combine all the real-time cluster labels from all batches to perform an overall clustering performance evaluation. For the comparison method, they are also implemented using the same real-time semantic alignment approach as proposed in this paper. We recorded the overall semantic alignment clustering results after all batches of data arrived, as shown in Table 4. As can be seen from the table, our method achieves the best clustering performance compared to other methods on most datasets and most metrics.

Furthermore, we demonstrate how the aligned clustering accuracy changes for four metrics with data arrives sequentially in batches, as shown in Figure 2. Due to space limitations, we present the semantic alignment results of the proposed method on four datasets: Animal, NUSWIDEOBJ, YTF10 and YTF20. As can be seen from the figure, the clustering accuracy after semantic alignment inevitably decreases slightly in the initial stage. However, as the number of data batches gradually increases, the accuracy tends to level off or even slightly improve, rather than decreasing monotonically. This demonstrates the effectiveness of the proposed semantic alignment method.

5.5 Ablation Study

To evaluate the effectiveness of the proposed method, we conducted ablation experiments to assess the performance of the proposed historical knowledge base construction and update method. Specifically, we designed three variant methods to simplify the construction of the knowledge base. Var-1 eliminates the historical knowledge base construction process and uses only the current batch of data for clustering. Var-2 randomly selects historical samples as the historical knowledge base. Var-3 incorporates only historical sample information without adding learned anchor or centroid information when constructing the historical knowledge base.

The performance of the proposed method and its variants is evaluated using the ACC metric on the Animal, NUSWIDEOBJ, YTF10 and YTF20 datasets. As illustrated in Figure 3, our proposed method consistently achieves the highest accuracy, outperforming all three variants on every dataset. These results confirm that each proposed component of the historical knowledge base construction and update is essential to the overall effectiveness of the framework.

6 Conclusion

In this paper, we proposed a novel a real-time semantic consistency incremental multi-view clustering framework to address the challenges of efficiency and label inconsistency in dynamic data streams. By employing a consensus bipartite graph with a k-connectivity constraint, our method enables real-time clustering without additional post-processing. Key to our approach is an adaptive diversity-aware selection mechanism that maintains a compact and representative knowledge base, alongside a semantic alignment strategy that prevents label drift across sequential batches. Future research will explore adapting this framework to handle data streams with an evolving number of clusters and enhancing its robustness against missing views.

Acknowledgments

This work is supported by the National Science Fund for Distinguished Young Scholars of China (No. 62325604) and the National Natural Science Foundation of China (No. 62276271, 62441618, 625B2182).