Continual Multi-View Clustering with Consistent Anchor Guidance

1 Introduction

In real-world scenarios, data are usually gathered from diverse sources, which form multi-view data and show the nature heterogeneity across views. Learning from such data has become a hot research topic due to the surge of openworld problems entailing multi-view data [Sun et al., 2024b; Jiang et al., 2023; Jiang et al., 2024; Li et al., 2023a; Zhang et al., 2023a; Sun et al., 2024a; Sun et al., 2023], especially in unsupervised cases. Clustering aims to divide data into disjoint groups, which is a fundamental unsupervised task in many applications. Focusing on discovering the cross-view consistency and complementarity for comprehensive data utilization, many multi-view clustering (MVC) approaches are developed, such as subspace based methods [Kang et al., 2020; Chen et al., 2022; Qin et al., 2024], kernel based methods [Liu et al., 2021b; Ren et al., 2021], graph based methods [Zhang et al., 2023b; Li et al., 2023b; Li et al., 2022a], deep learning based methods [Huang et al.,

∗Corresponding author.

2019; Trosten et al., 2021; Lin et al., 2021; Cai et al., 2022; Cai et al., 2024], etc. In spite of the success of these approaches, most of them fall within static MVC approaches, i.e., they can only handle the problem where the data are fixed, which is usually not satisfied in practice.

One more generalized issue is that data are continually collected and accumulated over time, constituting a dynamic data stream [Wang et al., 2023; Lange et al., 2022]. For example, on social media platforms like Twitter, thousands of images, texts and videos are posted every second from users, which form continuous multi-view data flow. In an industrial setting, the continuous collection of different sensors also represents a dynamic stream of multi-view data. Apparently, analyzing and clustering such dynamic data poses greater challenges than static data.

A straightforward approach to tackle this challenge is to store all historical and incoming data, and then apply traditional MVC methods on entire data for clustering. However, it will incur substantial storage and time overheads, and also waste the model knowledge obtained from previous data. Additionally, the gathered data may only be available temporarily due to privacy concerns. Continual learning (CL) [Lange et al., 2022] provides an effective solution to learn from dynamic streaming data, which aims to gradually expand acquired knowledge and use it for future learning without re-training all previous data. Concentrating on the catastrophic forgetting issue, many CL methods are designed, which can be roughly divided into five categories [Wang et al., 2023], including regularization-based, replay-based, optimization-based, representation-based, and architecturebased approaches. These various methods attempt to integrate new knowledge while retaining previous knowledge, enabling the model continually adapt and evolve over time. While CL has been widely studied [Li and Hoiem, 2018; Zhou et al., 2022; Hou et al., 2023; Li and Gu, 2023], its exploration of MVC remains relatively limited, leaving a vast space for further research and investigation.

In this paper, we propose a new consistent Anchor guided Continual MVC (ACMVC) method for streaming multi-view data clustering. ACMVC consists of two stages, Initial learning stage (I-stage) and Continual learning stage (C-stage). In I-stage, the model learns from existing data, which is the same as static MVC. In C-stage, a chunk of data arrives per round, and the model adapts by transferring historical knowledge to explore new data. Figure 1 provides the framework of our method, showcasing the process using two data chunks. In I-stage, ACMVC adaptively learns some anchors for each view, and a shared non-negative sample-anchor similarity matrix. The matrix not only exploits the consistent and complementary information, but also aligns the multi-view anchors. The similarity matrix is further factorized into a low-dimensional sample embedding and an anchor embedding which preserves the similarity relationships and conversely enhances the low-rank property of similarity matrix. In C-stage, rather than re-learning all previous data, we refine the model by leveraging transferred anchor knowledge. We achieve it by imposing a regularization term on original anchors and latent anchor embeddings, which helps to capture cross-chunk consistency information and enhances the model's adaptability and performance. The main contributions of this work are summarized as follow:

• We propose a new continual multi-view clustering method, integrating multi-level anchor knowledge reuse. It can efficiently learn from streaming multi-view data with linear space and time complexity.
• We solve the problem via a two-stage strategy. In Istage, a basic model is built with multi-level anchors and similarity learning. In C-stage, the model is refined by adding regularization to reuse anchor knowledge.
• We develop an efficient alternate algorithm to solve the model. Comparative experiments on popular datasets demonstrate the effectiveness and efficiency of our method in clustering streaming multi-view data.

2 RelatedWork

In this section, we present a brief overview of related research concerning multi-view clustering and continual learning.

Multi-view Clustering. During the past decade, numerous methods for MVC have been proposed, including graphbased, subspace-based, kernel-based, and deep learningbased approaches. Graph-based methods have gained significant popularity among these techniques [Huang et al., 2022; Zhang et al., 2023b; Zhang et al., 2022a]. They typically involve constructing a similarity graph to delineate pairwise relationships between data points and subsequently applying classical spectral clustering framework [Tang et al., 2019; Wang et al., 2020]. Nevertheless, constructing a complete n × n graph makes them unapplicable to large-scale data. Thus, anchor graph-based technique is developed, focusing on constructing a concise sample-anchor bipartite graph [Li et al., 2015; Zhang et al., 2024]. This approach greatly reduces the time and space consumption while upholding considerable clustering performance. For example, [Kang et al., 2020] adopt k-means to select representative anchors. [Li et al., 2022b] propose a directly alternate sampling scheme by measuring feature score to select anchors. These methods use heuristic strategies to determine anchors. In contrast, some researchers take an adaptive approach to learn anchors and integrate it with anchor graph learning in a unified model. For example, [Sun et al., 2021] learn a consensus anchor matrix and a non-negative graph for all views in a latent subspace. Although these methods have attained significant success, they generally belong to static MVC approaches, and cannot handle streaming data.

Continual Learning. CL studies the problem of learning from an infinite stream of data, aiming to progressively expanding existing knowledge and using it to facilitate future learning. It is also referred to as incremental learning, lifelong learning in most cases, without a strict distinction [Wang et al., 2023]. CL encompasses a variety of dynamic scenarios such as instance-incremental, class-incremental, domainincremental, task-incremental and more, most of which belong to supervised learning. Recently, some researchers introduced CL into unsupervised MVC, focusing primarily on two dynamic scenarios, view-incremental and instanceincremental. To tackle the view-incremental problem, [Yin et al., 2021] learn a consensus low-rank and sparse similarity graph for existing data, and use it to guide new graph learning upon the arrival of additional views. [Wan et al., 2022] keep a consensus partition matrix. To tackle the instanceincremental problem, [Shao et al., 2016] extends the classical non-negative matrix factorization (NMF) to an online version with a consistent base matrix regularization. [Hu and Chen, 2019] apply matrix factorization to learn cluster indicators and a unified base matrix for knowledge reuse.

3 ProposedMethod

In this section, we first give the notations and problem definition, and then delve into the details of I-stage and C-stage within our proposed ACMVC framework.

3.1 Notations and Problem Statement

Scalar, vector and matrix are denoted by lowercase, lowercase bold and uppercase bold letters (e.g., b, b, B), respectively. Tr(·) returns the matrix trace. 1 is a vector whose all elements are 1, and I is an identity matrix.

Our method addresses the streaming multi-view data clustering problem. We assume a chunk of multi-view data arrives per round. Let Xt = {X1t ,X2t , ...,Xmt } denote the received data in the t-th round with total m views, and Xvt = [xvt,1 ; xvt,2 ; ...; xvt,nt ] ∈ Rnt×dv denotes the v-th view feature matrix, nt and dv represent the sample number and feature dimension, respectively. In I-stage (round 1), we obtain data X1 and construct an initial model H1 for clustering. In C-stage (round t, t ≥ 2), we receive new data Xt and adapt Ht-1 to Ht by jointly acquiring new knowledge and transferring previous knowledge. The goal is to use the evolving model to partition all data into c clusters.

3.2 I-stage

During the I-Stage, we need to construct a basic model for X1, which not only effectively explores the relations among data for cluster structure discovery, but also offers easy adaptability to process streaming data. Inspired by the efficiency of anchor based approaches, we first develop the following primary model:

where Av1 ∈ Rk×dv denotes the anchor matrix containing k anchors, Z1 ∈ Rn1×k is a shared sample-anchor similarity matrix indicating the similarity relationship between each sample and anchors, w1 = [w1 1 , w2 1 , ..., wm 1 ]T ∈ Rm is a view weight vector. We employ orthogonal constraint on Av1 to ensure diverse anchors and avoid arbitrary scaling. The nonnegative constraint on Z makes the learned similarity matrix physically meaningful.

For a clustering-friendly similarity learning, it is expected that the similarity matrix Z is low-rank with rank(Z) = c, since the data are approximately drawn from c subspaces [Liu et al., 2013; Zhang et al., 2022b]. However, it is hard to optimize by explicitly enforcing the the rank to be c. Inspired by matrix factorization, we factorize the assymetric similarity graph Z into two latent factor embeddings, and address the following problem:

where F1 ∈ Rn1×c and V1 ∈ Rm×c can be viewed as sample embedding and anchor embedding, respectively, which encode the latent information of samples and anchors. The sample-anchor similarities are preserved into latent embeddings by inner product. Eq. (2) embodies an adaptive learning process that dynamically seeks representative anchors and constructs a low-rank similarity graph. The similarities are preserved into latent representation, where the sample embeddings can be used for k-means clustering.

3.3 C-stage

When a new chunk of data Xt arrives at the t-th (t ≥ 2) round, it is essential to learn the latent sample embedding Ft like that in I-stage. Moreover, Ft should not only discern within-chunk similarities but also capture the inter-chunk data similarities. Based on the fundamental model in I-stage, a straightforward way is to use the same anchors and latent anchor embeddings to guide new similarity construction and sample embedding learning, whose objective function can be described as

where {Avt-1 }mv=1 and Vt-1 are directly inherited from the historical model Ht-1, and it has Avt-1 = Avt-2 = ... = Av1 and Vt-1 = Vt-2 = ... = V1.

However, with the continual increase and distribution shift of data, the historical anchor information may be sub-optimal to guide the similarity graph and sample embeddings learning of new data. Thus, we plan to refine the anchor information while maintaining their semantic consistency across chunks, which can be achieved by

where ε1, ε2 ≥ 0 are tolerance error. For ease of optimization and avoid intricate parameter adjustment, we formulate the objective function in C-stage as follows:

When β →∞, Eq. (5) is equivalent to Eq. (3). When β = 0, Eq. (5) is independent with historical models and no previous knowledge is used. Eq. (5) extends Eq. (2) by adding regularization on multi-view anchors {Avt }mv=1 and anchor embedding Vt. With the guidance of consistent multi-level anchor information, on the one hand, our model leverages historical knowledge to enhance the exploration on new data. On the other hand, the similarities of data in different chunks are also mined to derive homogeneous latent embeddings, enabling a comprehensive exploitation for streaming data.

4 Algorithm

In this section, we present the optimization algorithm to solve the proposed model, and analyze the algorithm complexity.

4.1 Optimization Algorithm

We can observe that, when β = 0, Eq. (5) is exactly Eq. (2), and thus, we only need to optimize the former. We use alternate search strategy to solve it.

Avt Step: Because the anchors of different views are independent, we can obtain each view-specific anchor matrix by solving the following problem:

By utilizing Avt Avt T = Avt-1 (Avt-1 )T = I, we can convert Eq. (6) as the following maximum optimization problem:

whereQvt = 1 wv t ZTt Xvt +βAvt-1 . This problem can be solved by the following theorem.

Theorem 1. Suppose the singular value decomposition (SVD) of P = UΣVT , the optimal solution to

is given by G∗ = VUT .

Proof. By introducing P =UΣVT into Eq. (8), we have

whereM= VT GU. It can be easily verified that MT M= I. Thus, we have -1 ≤ mii ≤ 1, and then

Without of generality, let M = VT GU = I, i.e., G∗ = VUT , Tr (GP) gets the maximum. This completes the proof.

Zt Step: When other variables are fixed, the optimization problme of Zt becomes

Algorithm 1 ACMVC algorithm

Input: Multi-view data stream {X1,X2, ...,Xt, ...}, cluster number c, anchor number k, model parameters α, β. Output: Apply k-means on {F1, F2, ..., Ft, ...}.

• 1: for t = 1; t ++ do
• 2: Initialize Avt , Zt, Ft, Vt by 0, and wv t = 1/m.
• 3: while not converged do
• 4: Update {Avt }mv=1 by Thm. 1; ▷ β = 0 if t = 1
• 5: Update Zt by Eq. (14);
• 6: Update Ft by solving (15);
• 7:
• 8: Update {wv t }mv=1 by Eq. (19);
• Update Vt by solving (16); ▷ β = 0 if t = 1
• 9: end while
• 10: end for

Because Avt Avt T = I, with some simple operation, we can rewrite the problem as follows:

It can be further simplified as

where Pt = αFtVT t +∑mv=1 Xvt Avt T /wv t α+∑mv=1 1/wv t . Obviously, the optimal solution of Zt is given by

Ft Step: With some manipulation, we can rewrite the problemw.r.t. Ft as follows:

Similar to Avt , it can also be solved by Theorem 1.

Vt Step: The problemw.r.t. Vt is

We also use SVD to resolve it.

wt Step: Defining dvt = ∥Xvt -ZtAvt ∥F , the optimization problemw.r.t. wt is

According to Cauchy-Schwartz inequality, we get

where (i) holds because ∑v wv t = 1, and (ii) holds because √wv t ∝ dvt √wv t . The optimal viewweight is

By iteratively executing the above update steps, the objective function can be minimized until convergence. For convenience, we summarize the entire optimization process in Algorithm 1.

4.2 Complexity Analysis

Time Complexity. When updatingAvt , the main time costs come from the matrix multiplication and SVD operation, which take O(kdvnt) and O(k2nt) complexity, respectively. When updating Zt, its complexity is O((c+d)knt) with d = ∑mv=1 dv. For Ft and Vt, the complexity is O(cknt + c2nt) and O(cknt + c2k), respectively. The time complexity for updating wt is O(kdnt). Because k, c ≪ nt, the overall time complexity of our algorithm is O(dnt), which is linear to the number of samples.

Space Complexity. The major memory costs of our algorithm are variables Avt ∈ Rk×dv , Zt ∈ Rnt×k , Ft ∈ Rnt×c , Vt ∈ Rk×c . Thus, the space complexity of our method is also linear to the number of samples.

5 Experiment

In this section, we conduct experiments on several popular multi-view datasets to investigate the effectiveness of our method and compare it with some state-of-the-art approaches. We also evaluate the convergence property, running time, parameter sensitivity, and ablation study of our model. The computing platform is MATLAB R2019b with Win10 System, Intel Core i7-8700 CPU@3.2GHz and 16GB RAM.

5.1 Experimental Setup

Datasets. We adopt seven popular multi-view datasets for experiments, including BBCSport 1 , Mfeat 2 , Wiki 3 , MITIndoor 4 , Caltech101 5 , Fashion 6 , and VGGFace 7 datasets. Table 1 lists the general statistics.

Baselines. We compare ACMVC with following baselines:

• SMVSC [Sun et al., 2021] adaptively learns unified latent anchors, and a consensus subspace representation.
• OPMC [Liu et al., 2021a] integrates matrix factorization and discrete cluster indicator matrix learning into a unified model.
• OMSC [Chen et al., 2022] jointly considers anchor learning, anchor graph construction and discrete partition for efficient MVC.
• EOMSC [Liu et al., 2022] jointly learns anchors and low-rank anchor graph with a connectivity constraint.
• AWMVC [Wan et al., 2023] learns multiple subspace representations in different dimensions for each view, and then fuses them in an auto-weighted ensemble way.
• OMVC [Shao et al., 2016] is an online MVC method, which learns the latent representations by weighted NMF and pushes them towards a consensus one.

1 http://mlg.ucd.ie/datasets/bbc.html

2 https://archive.ics.uci.edu/dataset/72/multiple+features

3 http://www.svcl.ucsd.edu/projects/crossmodal/

4 https://web.mit.edu/torralba/www/indoor.html

5 https://www.vision.caltech.edu/datasets/

6 https://www.kaggle.com/datasets/paramaggarwal/

fashion-product-images-dataset

7 https://www.robots.ox.ac.uk/∼vgg/data/vgg face2/

• OPIMC [Hu and Chen, 2019] is also an online method, which directly learns a consensus discrete partition matrix by weighted matrix factorization.

Implementation Details. Our method addresses the streaming multi-view data clustering problem. To construct the streaming data, we split the whole dataset into several chunks, and a chunk of data arrives in each round. The chunk size is set to 100 for BBCSport, 1000 for Mfeat, Wiki, MITIndoor, Caltech101, and 5000 for Fashion and VGGFace. The source codes of baselines are publicly available, and we search their optimal hyper-parameters following the suggestions of original papers. For our ACMVC, we search the parameters α and β in the range of {100 , 101 , ..., 106 }, and fix the number of anchors k = 5c in all experiments for simplicity, where c is the number of clusters. We adopt five popular clustering evaluation metrics, i.e., accuracy (ACC), normalized mutual information (NMI), purity (PUR), adjusted rand index (ARI) and Fscore. The larger values indicate the better clustering performance. We repeat each experiment 10 times to report the mean values as well as standard deviation.

5.2 Clustering Results

Table 2 lists the clustering results of our method and baselines on seven datasets w.r.t. five metrics, where the best results and second-best are highlighted in red and blue, respectively. We have the following observations:

• Our method generally achieves the best or second-best results on all datasets. EOMSC and AWMVC are two advanced offline MVC algorithm. They first accumulate each data chunk, and then perform clustering on the accumulated data, which can comprehensively explore the similarity relations among all data. Thus, they also obtain the best and competitive performance in some cases. Although our method sequentially learns from each data chunk in an online manner, it still gets excellent results and outperforms the offline methods like SMVSC, OPMC, OMSC and EOMSC in most cases, demonstrating the effectiveness of our consistent anchor based continual learning mechanism.
• OMVC and OPIMC are two continual learning methods based on multi-view matrix factorization. The performance of OMVC is not desirable, and in most cases, it is inferior to some offline MVC approaches like OMSC, EOMSC, and AWMVC. OPIMC generally achieves better results than OMVC, and it obtains competitive performance on Wiki, Caltech101 and Fashion datasets.

Compared with the two methods, our approach further improves the clustering performance, implying that the adaptive sample-anchor similarity learning and latent embeddings learning can obtain discriminative representation for clustering.

5.3 Parameter Analysis

In this subsection, we analyze the influence of model parameters α and β on clustering performance. Figure 2 shows the change in ACC values with different combination of α and β on BBCSport and VGGFace datasets. The horizontal axes are scaled using log10. We can see that the parameters influence the algorithm performance. Generally, when α and β are selected from [102 , 105 ] and [100 , 102 ], the results are not sensitive to the parameters and satisfactory performance is obtained. Moreover, the number of anchors is fixed as k = 5c in all experiments. For a comprehensive evaluation, we also investigate the influence of k and display the results in Figure 2. In general, when m is selected from [4, 6], the clustering results are desirable.

5.4 Convergence and Time Comparison

An alternate optimization algorithm is developed to solve the proposed model. Since each sub-problem is solved with a closed-form solution, the algorithm can converge to a local minimum finally. In this subsection, we experimentally illustrate the algorithm convergence characteristic by showing the change of objective function value. Figure 3 plots the convergence curves on BBCSport and VGGFace datasets. The curve of I-stage is obtained from the initial data chunk (round 1), while the curve of C-stage is derived from the subsequent data chunk (round 2). As can be observed, the objective function value declines swiftly on two datasets, allowing the algorithm to converge within only 10 iterations.

We also compare the running time (second) of different methods. SMVSC, OPMC, OMSC, EOMSC, and AWMVC belong to static MVC algorithm, and they have to undergo a complete re-learning process when new data arrives, while OMVC, OPIMC and our ACMVC are in an online learning manner. Figure 4 shows the running time comparison results, with the y-axis rescaled using log2. OMVC is not efficient because it adopts a Hessian matrix based projected gradient descent algorithm to solve the non-negative matrix factorization problem. OPIMC and our method stand out as the two fastest algorithms in terms of speed. For example, on VGGFace dataset, the running time of our method is less than 50 seconds. These results demonstrate the efficiency our proposed method.

5.5 Ablation Study

To efficiently handle streaming data, our ACMVC transfers the historical multi-level anchor information, i.e., {Avt }mv=1

and Vt. In this subsection, we conduct ablation study to evaluate the effectiveness of the two knowledge reuse regularization. In specific, based on Eq. (5), we establish two variants Ours-A and Ours-V. Ours-A excludes the ∑mv=1 ∥Avt - Avt-1 ∥2F term, and Ours-V excludes the ∥Vt-Vt-1∥2F term. Besides, we also evaluate the performance of Eq. (3), which adopts fixed anchor information for knowledge transfer without regularization, and we denote it as Ours-F. Table 3 reports the clustering results w.r.t. ACC of the three variants and our original method on seven datasets. It should be noted that the model parameters of all variants are also tuned by grid search to report the best performance for a fair comparison. From Table 3, we can see that our proposed model achieves the best performance in all scenarios. Ours-V performs the worst, because the latent anchor embedding is strongly associated with sample embedding. Without the guidance of historical anchor embedding, the inter-chunk sample relationships cannot be captured. Ours-F gets competitive results because it also uses the historical multi-level anchor knowledge. Our method still outperforms Ours-F, proving the effectiveness of anchor consistency regularization.

6 Conclusion

In this paper, we propose a consistent anchor guided continual multi-view clustering method called ACMVC to deal with streaming multi-view data. ACMVC consists of two stages. In initial learning stage, an efficient low-rank anchor graph based model is constructed. In continual learning stage, the historical model is refined by adding consistency regularization to transfer the learned multi-level anchor information. Experiments demonstrate the superiority and efficiency of our method compared with state-of-the-art approaches, and ablation studies prove the effectiveness of our designed knowledge reuse mechanism. In the future, we will consider other advanced knowledge transfer strategies.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China under Grants 62176116, 62073160, 62276136, and the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under Grant 20KJA520006.