Distributed Cascaded Manifold Hashing Network for Compact Image Set Representation

1 Introduction

In recent years, the rapid development of multimedia technology has led to an explosion in the number of images captured by cameras and surveillance videos. As a result, many classification and retrieval tasks now focus on image sets, which consist of multiple images of a specific object [Zhu et al., 2014; Huang et al., 2015a; Huang et al., 2017; Pigou et al., 2018]. Image sets provide richer variability than single images, but due to factors like lighting, angle, posture, and background, the image sets from the same object often exhibit large intra-class variations, making image set classification a challenging task.

Image set modeling is crucial for image set tasks, and various effective methods have been proposed. Affine/convex hull [Cevikalp and Triggs, 2010] and linear subspace methods [Yang et al., 2013; Kim et al., 2007] model image sets but struggle to capture nonlinear structures. Manifold-based methods have gained attention for their strong nonlinear modeling capabilities. These methods can be categorized into three types: Riemannian kernel methods [Wang et al., 2012b; Hamm and Lee, 2008; Wang et al., 2018b; Shen et al., 2024], Riemannian metric learning methods [Huang et al., 2015b; Huang et al., 2015a; Zhu et al., 2018; Lu et al., 2015], and deep manifold networks [Huang and Van Gool, 2017; Huang et al., 2018; Chakraborty et al., 2020; Wang et al., 2021]. Riemannian kernel methods use kernel functions to embed manifolds into a Reproducing Kernel Hilbert Space (RKHS), but they ignore Riemannian geometry. Riemannian metric learning addresses this by using manifold similarity as the Riemannian metric. To enhance modeling, deep manifold networks construct neural networks directly on manifolds. However, these methods learn in continuous space, leading to high computational and storage costs on large-scale datasets, making learning of compact image set representations a continuing challenge.

∗Corresponding author.

Conventional image set methods primarily learn from image sets collected at a single location. However, in realworld applications, such as person re-identification [Cao et al., 2022], image sets are often collected across multiple locations. One approach is to centralize the image sets before applying a centralized learning method. However, this incurs high communication costs and may exceed the computational capacity of a single machine [Zhai et al., 2017].

Hashing [Wang et al., 2018a] has attracted extensive interest in large-scale image retrieval due to its advantage of computation and storage. Hashing maps high-dimensional data into low-dimensional hash codes while preserving similarity structure among data. The advantage of hashing motivates us to extend to the application of image retrieval to more challenging image set classification and retrieval, and thus proposes Distributed Cascaded Manifold Hashing Network (DCMHN) to significantly reduce computation and storage costs. The illustration of the proposed DCMHN is shown in Figure 1. The main contributions of this work can be summarized as follows:

• We propose the Distributed Cascaded Manifold Hashing Network (DCMHN) to learn compact image set representations using hash codes, enabling efficient classifi-

cation and retrieval. DCMHN is the first deep neural network approach designed for compact representation learning on distributed image sets.

• The manifold hashing network is used to encode the manifold and is trained in a cascaded manner to reduce training time. DCMHN enforces local consistency across neighboring nodes, enabling parallel optimization.
• We evaluate DCMHN on three benchmark image set datasets, demonstrating competitive accuracy in distributed settings and superior performance in terms of computation and storage efficiency compared to stateof-the-art methods.

2 RelatedWork

Image Set Classification Conventional image set methods can be mainly divided into five types: affine/convex hull methods, linear subspace methods, Riemannian kernel methods, Riemannian metric learning methods, and deep manifold networks.

Affine/Convex hull methods, e.g., Affine/Convex Hull Based Image Set Distance (AHISD/CHISD) [Cevikalp and Triggs, 2010] characterize an image set by the convex geometric region, i.e., affine or convex hull, and measure the distance between two image sets by geometric distance, i.e., distance of closest approach between convex models. Linear subspace methods, e.g., Mutual Subspace Method (MSM) [Yang et al., 2013] and Discriminant Canonical Correlations (DCC) [Kim et al., 2007] represent an image set as a linear subspace. MSM uses the cosine of the minimum angle between two subspaces as the distance measurement, and DCC aims to seek the projection metric such that canonical correlations in the learned subspace are maximized. Riemannian kernel methods, e.g., Covariance Discriminative Learning (CDL) [Wang et al., 2012b] and Grassmann Discriminant Analysis (GDA) [Hamm and Lee, 2008] map the manifold into a reproducing kernel Hilbert space(RKHS), and then employ the kernel discriminant analysis for discriminant learning. Riemannian metric learning methods, e.g., LogEuclidean Metric Learning (LEML) [Huang et al., 2015b]

and Projection Metric Learning (PML) [Huang et al., 2015a] map the manifold to lower-dimensional discriminative manifold through metric learning. Deep manifold networks, e.g., Symmetric Positive Definite Network (SPDNet) [Huang and Van Gool, 2017] and Grassmann Network (GrNet) [Huang et al., 2018] generalize Euclidean network paradigm to manifold space and perform end-to-end training through matrix backpropagation. Later, [Wang et al., 2022] propose two Riemannian operation modules, i.e., Riemannian batch regularization layer, Riemannian pooling layer to improve performance. In addition, SymNet [Wang et al., 2021] proposes a simple SPD manifold deep learning network that learns multilevel connection matrices and reduces computational complexity.

Hashing Hashing [Liu et al., 2011; Gionis et al., 1999; Wang et al., 2012a; Liu et al., 2012; Norouzi and Fleet, 2011] projects the original high-dimensional data into a compact Hamming space while preserving the similarity structure. Locality Sensitive Hashing (LSH) [Datar et al., 2004] generates hash functions in a random manner. Iterative Quantization (ITQ) [Gong et al., 2012] uses Principal Component Analysis (PCA) to reduce dimension, and then employs orthogonal rotation to reduce quantization error. Supervised Discrete Hashing (SDH) [Shen et al., 2015] assumes that high-quality hash codes are ideal for classification, and then are learned jointly by minimizing quantization losses under a linear classifier framework. Deep Hashing Network (DHN) [Zhu et al., 2016] captures the image representation by a convolutional layer and a pooling layer, and uses cross-entropy loss and quantization loss supervision to generate a compact binary code, which addresses the quantization error issue.

3 Distributed CascadedManifold Hashing Network

3.1 Problem Setup

Suppose that we are given a distributed image set dataset {Si,Yi}Pi=1 that has N samples distributed in P nodes of a network, as shown in Figure 1. In the i-th node, image set features and labels are denoted as Si = {Sij ∈ Rl×nij }Ni j=1

and Yi ∈ RC×Ni respectively, where Sij and nij denote feature and number of images of the j-th image set in the i-th node respectively, l is the dimension of image feature, C is the number of classes, Ni denotes the number of image sets in the i-th node, and we have N = ∑Pi=1 Ni. The goal is to learn distributed hashing network and hash code B = {Bi ∈ Rd×Ni }Pi=1 from such distributed image set to perform efficient image set classification and retrieval, where d is the dimension of hash code.

3.2 Network Structure

SPDmanifold Given an image set S, its covariance matrix is defined as:

where m, si, and n denote the mean, the i-th image, and the number of images in S respectively. To ensure positive, we add a small regularization matrix Tr(X) α I, where I denotes the identity matrix, and parameter α is empirically set to 103 .

SPD Manifold and Hash Layer As shown in Figure 1, the proposed DCMHN constructs a manifold hashing network to encode manifold to generate hash code. Specifically, DCMHN first employs three SPD manifold layers [Huang and Van Gool, 2017], i.e., BiMap, ReEig, and LogEig layers to encode manifold, and further employs a hash layer to transform the continuous feature into hash code. We assume Xki ∈ Rdk×dk denotes the input of the k-th layer of the i-th node, and define the layers [Huang and Van Gool, 2017]:

BiMap Layer projects each input SPD matrix into a new one through a bilinear mapping Xk+1 i = Wk⊤ i Xki Wk i , where Wk i ∈ Rdk×dk+1 denotes transformation matrix, and we have dk+1 < dk.

ReEig Layer injects nonlinearity by tuning-up the small positive eigenvalues of each input SPD matrix via Xk+1 i = Umax(ϵI,Σ)U⊤, where U and Σ are eigenvectors and eigenvalues of Xki respectively, ϵ is a predefined small activation threshold.

LogEig Layer embeds each input SPD matrix into a flat space via Xk+1 i =Ulog(Σ)U⊤.

Hash Layer transforms the input using a linear mapping matrix and a nonlinear sign function via Xk+1 i = sign(Wk⊤ i Xki ), where sign denotes the sign function.

3.3 Formulation

As illustrated in Figure 1, the proposed DCMHN uses BiMap, ReEig, and LogEig layers for manifold encoding, with a Hash layer to generate hash codes. The ReEig and LogEig layers are parameter-free, while the BiMap and Hash layers contain learnable parameters. This section details the parameter learning process, as summarized in Algorithm 1.

Distributed Bilinear Mapping Learning

Conventional deep network often uses stochastic gradient descent, and further employs chain rule as backpropagation (BP). However, compared to calculating Euclidean gradient, calculating Riemannian gradients in deep manifold network is challenging. As such, a variant of SGD on Stiefel manifold [Huang and Van Gool, 2017] has been developed to train such network, however it leads to very high computational cost, and usually requires hundreds or even thousands iterations to converge. In this work, instead of using BP, we propose to learn bilinear mapping in BiMap layer from the perspective of effective preservation of SPD structure [Wang et al., 2024b].

We consider the i-th node, and denote Xn,k i ∈ Rdk×dk as the input SPD matrix of the k-level BiMap layer of the n-th image set, where dk denotes feature dimension. As defined earlier, BiMap layer transforms the input SPD matrix into a new SPD matrix via the following function via a projection matrix Wk i ∈ Rdk×dk+1 , and it is defined as follows:

We have the following theorem related to property of Wk.

Theorem 1. Given the input SPD matrix Xn,k i , the output matrix Xn,k+1 i is a SPD matrix if Wk i is required to be a row full-rank matrix, i.e.,Wk⊤ i Wk i = I.

Proof. It is easy to prove the output matrix is satisfied with the definition of SPD matrix if transformation matrix is fullrank using simple matrix computation.

Motivated by the popular idea of Principle Component Analysis (PCA), we propose a simple yet effective solution to learnWk i . Specifically, we aim to preserve SPD structure across BiMap layer [Wang et al., 2024b], and employ the following reconstruction loss:

where Xn,k i is first required to be centralized. The above problem can be equivalently transformed to the eigenvalue problem, andWk i is formed by the eigenvectors corresponding to the largest dk+1 eigenvalues of sample covariance matrix Cki = ∑Ni n=1 Xn,k⊤ i Xn,k i .

Note that we are given a distributed task, and we need to consider all the P nodes. To make full of all the distributed manifolds on the P nodes, we propose to calculate the global sample covariance matrix Ck = ∑Pi=1 Cki , and perform its eigenvalue decomposition. To achieve this, we introduce a parameter server to first receive the local covariance matrices on the P nodes, calculate the global covariance matrix by averaging the local ones. The parameter server finally returns the learned bilinear mapping matrices to the P nodes.

Distributed Hash Code Learning

On the i-th node, each SPD manifold is processed through BiMap, ReEig, and LogEig layers, and we have vectorized Euclidian features Xi ∈ Rm×Ni , where m denotes the feature dimension. The feature is then feed into hash layer to obtain hash code. Existing researches [Shen et al., 2015] show that optimal hash codes are expected to have good classification performance. In addition, quantization errors between continuous features and hash codes affect quality of hash codes. Therefore, we consider all the P nodes, and propose to learn hash codes by jointly minimizing the following simple yet linear classification loss and quantization loss:

where U ∈ Rd×C denotes the weighted matrix related to the linear classifier, V ∈ Rm×d denotes transformation matrix in hash layer, λ and µ denote regularization parameters. The first term is least square classification loss to improve discrimination of hash code, the second term is regularization term, and the third term is quantization loss between continuous feature and hash code. The binary constraint enforces the continuous feature to approximate hash code, which can handle large-scale applications efficiently.

Inspired by the brain's distributed memory processing [Wen et al., 2018] and brain-inspired coordination strategies [Han et al., 2025; Jia et al., 2025; Wang et al., 2024a], we formulate a distributed optimization framework. The local variable Bi can be optimized independently in each node, while the global variables U and V are shared among all the nodes, making them difficult to optimize. Motivated by block splitting strategy, we introduce a new set of local auxiliary variable in each node, i.e., Ui, Vi, and optimize these new local variables in parallel. To the end, (4) can be rewritten as the following distributed objective function:

where N(i) represents the neighbors of the i-th node. In (5), we impose consistency constraint on global variables in neighbor nodes rather than all the nodes by utilizing transitivity property of a connected graph. This optimization problem is non-convex with respect to the variables Bi, Ui, and Vi. Therefore, we divide it into multiple sub-problems and optimize the variables in one node while fixing the variables in other nodes. We use alternating direction multiplier method (ADMM) to solve (5). To the end, the augmented Lagrangian function of (5) can be defined as follows:

where Λi,j and Γi,j are Lagrangian multipliers for the constraints Vi = Vj and Ui = Uj respectively, α and β are penalty parameters corresponding to augmented Lagrangian.

Update Vi By fixing the other variables and removing irrelevant terms, the subproblem related to Vi is reduced as follows:

ADMM is applied to solve (7) by repeating the following two steps:

where t and t - 1 represent the t and t - 1 iterations respectively. We provide the following theorem to simplify the Lagrangian multipliers.

Lemma 1. The augmented Lagrangian function on Vi, i.e., (7) is equivalent to the following form:

where Λi = ∑j∈N(i) (Λi,j - Λj,i) is the new Lagrangian multiplier.

Proof. The lemma can be proven with straightforward matrix computations.

As can be seen, (9) has fewer Lagrange multipliers than (7), and computational complexity of ADMM can be greatly reduced. Based on Lemma 1, we have the following theorem to update Vi:

Theorem 2. For the new augmented Lagrangian, i.e., (9), we can update Vi using the following rule:

Proof. The theorem can be proven with straightforward matrix computations.

Update Bi By fixing other variables, the subproblem with respect to Bi is defined as:

It is challenging to directly optimize Bi as discrete constraint is NP hard. Following the widely-used coordinate descent in optimization, discrete cyclic coordinate descent (DCC) [Shen et al., 2015] is employed to optimize Bi bit by bit, and each bit admits a closed-form solution.

Algorithm 1 Distributed Cascade Manifold Hashing Network (DCMHN)

Input: Image sets S = {S1,S2, . . . ,SP , }; Labels Y = {Y1, Y2, . . . , YP }; Hash code length d; parameter α, β; Output: Hash code B = {B1, B2, . . . , BP };

• 1: Obtain SPD manifold of each image set via (1);
• 2: Calculate global covariance matrices of all the nodes and perform eigenvalue decomposition, and return the learned bilinear mapping matrix in BiMap layer on each node;
• 3: Perform eigenvalue correction to introduce non-linearity in ReEig layer;
• 4: PerformRiemannian computation in LogEig layer;
• 5: Randomly initialize Lagrangian multipliers {Λi}Pi=1 , {Γi}Pi=1 , {Ui}Pi=1 , {Vi}Pi=1 , and penalty parameters α, β;
• 6: for All nodes simultaneously do
• 7: Update {Bi}Pi=1 using DCC algorithm;
• 8: Update {Vi}Pi=1 , {Γi}Pi=1 using (10);
• 9: Update {Ui}Pi=1 , {Λ}Pi=1 using (13);
• 10: end for

Update Ui By fixing the other variables and removing irrelevant terms, the subproblem related to Ui is reduced as follows:

Similar to optimizing Vi, we repeat the following steps to optimize Ui:

4 Experiments

4.1 Experiment Setting

Datasets Three benchmark image set datasets, i.e., FPHA [Garcia-Hernando et al., 2018], AFEW [Wang et al., 2012b], BBT [Li et al., 2015] are used for experiment. The statistics of the three datasets are summarized in Table 1. For FPHA, a 63-dimensional image feature is extracted for each sequence [Wang et al., 2021]. For AFEW, a 400-dimensional feature is extracted for each frame [Wang et al., 2012b]. For BBT, a 512-dimensional deep feature for each frame is extracted by CNN [Schroff et al., 2015]. The training set of each image set benchmark is equally distributed across all the nodes in the network to construct distributed data. The network includes 3 nodes for FPHA and AFEW, and 4 nodes for BBT.

Comparison Methods We compare the proposed method with various advanced image set classification methods, which can be divided into the following four categories: Affine/Convex hull methods including AHISD/CHISD [Cevikalp and Triggs, 2010], Linear subspace methods including DCC [Kim et al., 2007], Nonlinear manifold methods including CDL [Wang et al., 2012b], LEML [Huang et al., 2015b], RMML [Zhu et al., 2018], SPDNet [Huang and Van Gool, 2017], SymNet [Wang et al., 2021], Hashing methods including ITQ [Gong et al., 2012], SDH [Shen et al., 2015]. Specifically, ITQ-SPD and SDH-SPD apply ITQ and SDH to the SPD manifold respectively. The implementations of the baseline methods are kindly provided by the authors. The manifold network in the proposed DCMHN is constructed as fb, fr, fb, fr, fl, fh, where fb, fr, fl, fh denote BiMap, ReEig, LogEig, Hash layers respectively. The dimensions of the two BiMap layers are set to [20, 10], [70, 35], [80, 40] for FPHA, AFEW, and BBT respectively.

Evaluation Metrics Accuracy (Acc) is used as evaluation metric for classification, and mean Average Precision (mAP) and Precision-Recall (PR) curves are used as evaluation metrics for retrieval.

4.2 Performance Evaluation

Acc and mAP The classification and retrieval performances of all the methods on the three datasets are summarized in Table 2. From Table 2, we have the following observations:

• The proposed DCMHN generally achieves the best performance among all the cases, and improves the best

baseline averagely by 2% and 7% in terms of Acc and mAP respectively. The empirical results demonstrate the effectiveness of the proposed method on image set modeling.

• Among two deep Riemannian manifold baselines, SymNet outperforms SPDNet. The deep Riemannian manifold baselines outperform the shallow baselines, due to superior capability of the deep architecture. Among hashing baselines, SDH-SPD outperforms ITQ-SPD, as SDH-SPD is supervised.
• Some baselines, e.g., AHISD, CHISHandLEML cannot be well scaled to large-scale datasets due to high computational complexity.

Storage The storage usages for SPD manifold, kernel matrix, continuous feature, and hash code of three datasets are reported in Table 3. As can be observed, the hash code only requires less than 300 KB of storage for the three datasets, while manifold feature requires significantly expensive storage, thus manifold methods cannot be applied to large-scale image set datasets. Particularly, the storage required using hash code is 64 times less than that required for continuous feature, and considerably less compared to other features. The empirical studies demonstrate the significant advantage of hashing in terms of efficient storage usage.

Time The running time associated with distance calculation of different methods on three benchmarks is reported in Table 4, and time of embedding generation is not included. We can find that the running time of hashing is significantly lower than that of the other methods. Specifically, on the AFEW dataset, hashing methods are nearly 280 times faster than those of continuous feature, and significantly faster than manifold methods. Hashing methods, including the proposed DCMHN, which calculate Hamming distance, are theoretically faster than conventional image set methods that calculate Euclidean distance. The empirical studies verify the superiority of the proposed method in terms of efficient distance calculation.

4.3 Further Analysis

Discrete versus Continuous This section compares the performance of the proposed method with its continuous variant that removes binary constraint. The variant is denoted by adding the suffix '-C'. The Acc and mAP of the two methods on the three datasets are shown in Table 5, and the PR curves are shown in Figure 2. We can clearly see that even with binary constraint, the proposed DCMHN outperforms its continuous variant in all the cases, indicating that binary constraint does not degrade performance image set tasks. The empirical studies demonstrate superiority of hash code on modeling image set, and verify the effectiveness of the proposed optimization strategy.

Code Length Analysis This section analyzes the impact of hash code length on the performance of the proposed method. Specifically, the Acc and mAP of the proposed method with respect to different code lengths varying from 16 to 1024 are illustrated in Figure 3. As can be seen, the performance of the proposed method improves as code length increases, and the best performance is generally achieved when code length is set to 256. The above empirical results suggest the optimal hash code length that can achieve good performance.

Node Number Analysis This section analyzes the impact of number of nodes in the network on the performance of the proposed method. The Acc, mAP, and running time of the proposed method with 32 bits on AFEW varying from 2 to 8 nodes are illustrated in Figure 4. As observed, increasing the number of nodes significantly reduces training time, with only a slight decline in accuracy and mAP. Using approximately 3 nodes strikes a balance between performance and training time. The empirical studies confirm that parallel optimization can improve training efficiency.

Trade-off Parameter Analysis This section empirically analyzes the sensitivity of a trade-off parameter, i.e., µ in the proposed DCMHN, where µ is used to balance the relative importance of quantization loss compared to classification loss. The Acc and mAP of the proposed method on AFEWwith respect to varying from [10-10 , 10-4 ] are shown in Figure 5. It can be observed that with the decrease of µ, the performance remains stable. The good performance can be achieved when µ is suggested to be less than 10-6 .

Visualization This section conducts qualitative empirical study and performs visualization. The visualization results of six representative methods on BBT are shown in Figure 6, and The widely-used t-SNE is applied to the learned representations. It can be observed that SDH-SPD, SPDNet, and the proposed DCMHN can separate multiple clusters better than the other methods. These qualitative results are consistent with previous quantitative results, intuitively reinforcing the effectiveness of the proposed DCMHN.

5 Conclusion

This paper introduces, for the first time, a Distributed Cascade Manifold Hashing Network (DCMHN) designed to construct a manifold-based network for learning hash codes in a distributed manner. The core idea is to model distributed image sets as a connected graph and build a distributed deep manifold hashing network on this graph, with the network trained in a cascaded fashion. Compared to existing image set methods, the proposed approach offers significant advantages in handling distributed image sets while enhancing computational and storage efficiency. The empirical studies validate our theoretical findings.

Acknowledgments

This work was supported by the National Science Fund for Distinguished Young Scholars of China under Grant No. 62325601, the National Natural Science Foundation of China under Grant No. 62472226, 62176126, the Natural Science Foundation of Jiangsu Province, China under Grant No. BK20230095.