AdaptPFL: Unlocking Cross-Device Palmprint Recognition via Adaptive Personalized Federated Learning with Feature Decoupling

1 Introduction

Over the past two decades, biometric identification technologies have garnered significant attention and have been widely applied across various domains. Among these, face recognition stands out for its convenience and widespread use. However, the ability to capture facial images without the user's consent has raised growing concerns regarding potential privacy breaches. In contrast, palmprint recognition typically requires active user cooperation to capture palmprint images. As a result, palmprint recognition is considered a more privacy-preserving biometric technology than face recogni- Although palmprint recognition offers better privacy protection than face recognition, traditional methods still pose significant privacy and security risks in many mobile application scenarios requiring personal data sharing. As a result, researchers have been increasing interest in leveraging federated learning [Li et al., 2020a] to safeguard user privacy and security. The fundamental principle of federated learning is to decentralize the training of machine learning models by shifting the computation from a central server to various devices or data centres. Each data holder trains the model locally and shares only the model parameters or updates rather than raw data. This approach enables the participation of sensitive data in the training process without leaving the local domain, thus ensuring user data security and privacy protection.

tion.∗ Corresponding authors

Despite traditional federated learning methods offering a degree of privacy and security for users, complex application scenarios in palmprint recognition continue to pose significant challenges. As illustrated on the left side of Fig. 1, the same people in the traditional method produce obvious feature drift between different clients. There is still some drift even within the same client. These discrepancies lead to aggregation challenges at the server, ultimately reducing recognition accuracy. To address these issues, we propose an adaptive personalized federated learning framework based on feature decoupling for cross-device palmprint recognition. This method decouples palmprint features into identity and contextual information, as shown on the right side of Fig. 1. An intra-local adaptive aggregation strategy is introduced to correct intra-client drift, effectively minimizing the impact of external environmental factors. Furthermore, an adaptive global-local aggregation strategy addresses inter-client drift by integrating global information essential for client models.

The contribution of this paper can be summarised as follows:

• (i) We design a de-entanglement mechanism using a deep edge detection model and a mutual information method. The influence of external factors on identification is isolated by decoupling the palmprint information.
• (ii) Two adaptive aggregation strategies, ILAA and GLAA, are introduced to correct intra-client drift and inter-client drift, respectively.
• (iii) We conducted extensive experiments on three contactless palmprint databases. The results demonstrate the method's higher accuracy than the SOTA methods and effectively mitigate the client-side drift problem.

2 RelatedWork

The section focuses on the current work on palmprint recognition and federated learning in detail.

2.1 Palmprint Recognition

In recent years, deep learning-based methods have been applied to palmprint recognition. Genovese et al. [Genovese et al., 2019] proposed PalmNet for unsupervised palmprint recognition. It includes a convolutional neural network, Gabor filter, and principal component analysis. Zhong et al. [Zhong and Zhu, 2019] proposed an end-to-end palmprint recognition method, which applies a new loss function of CNN to enhance the inner class compactness of palmprint features. Matkowski et al. [Matkowski et al., 2019] proposed the palmprint recognition network EE-PRnet, which consists of two main networks that can solve the palmprint recognition problem in uncontrollable and non-collaborative environments. Zhao et al. [Zhao et al., 2019] proposed a joint deep convolutional feature representation applied to feature extraction of hyperspectral palmprint images. Jia et al. [Jia et al., 2022] improved MobileNet-V3 and proposed a lightweight convolutional neural network for palmprint recognition. Zhao et al. [Zhao and Zhang, 2020] used deep convolutional neural networks to learn complete and discriminative convolutional features and proposed a jointly constrained least squares regression framework. This framework combined deep local convolutional features to solve the undersampling classification problem in palmprint recognition. Shao et al. [Shao and Zhong, 2021] introduced a deep hash network of MoblieFaceNets to extract discriminative features. It improves the matching efficiency and allows the target network to adapt to unlabelled target palmprint images. Fan et al. [Fan et al., 2022] proposed using deep learning architecture to extract the main lines of palmprints to reduce the effect of fine lines and further classify the palmprint phenotypes from the 2D palmprint images. Samai et al. [Samai et al., 2018] proposed using DCTNet for 3D palmprint recognition. Chaa et al. [Chaa et al., 2019] first used a single-scale retina algorithm to enhance the depth image of 3D palmprints and then used PCANet for recognition.

However, most contactless palmprint recognition methods ignore user privacy issues. The huge security risk of sharing personal data among different users is also an urgent problem to solve.

2.2 Federated Learning

Currently, the federated learning technique is a distributed machine learning framework that effectively solves the data silo problem. McMahan et al. [McMahan et al., 2017] proposed a FedAvg algorithm, an important milestone in developing federated learning. Most of the current federated learning methods are based on the idea of the FedAvg algorithm and continuously optimize and upgrade the federated learning algorithm. Li et al. [Li et al., 2020b] proposed the FedProx algorithm, which adds a proximal term to the client's original loss function. This maintains the similarity between each client's local and global model parameters in federated learning. The robustness of the model is improved by balancing the differences between the global and local models. With the continuous development of federated learning, techniques such as comparative learning, prototype learning, and metalearning have been introduced into the research and practice of federated learning. Li et al. [Li et al., 2021] proposed the MOON algorithm, which performs comparative learning at the model level by comparing representations learned from different models. The aim is to reduce the distance between the local and central weight representations. Thereby increasing the distance to the local weight representation of the previous round. The similarity of model representations is used to correct the bias in local training and alleviate the data heterogeneity problem. Mu et al. [Mu et al., 2023] introduced prototype learning based on the MOON algorithm, which uses prototypes as global knowledge to correct the bias in local training for each client. Decrease the distance between the prototype of a local class and the global prototype of the same class. Thus increasing the distance from the global prototype of a different class.

The above approaches aim to train a global model with generalization performance for all clients. However, when the data distribution of cross-device clients varies significantly by conditions such as lighting and brand, a single global model may not be able to meet the diverse needs of each client [Huang et al., 2021]. Therefore, this paper presents two adaptive aggregation strategies for intra-client and interclient drift phenomena.

3 Methodology

The section describes each module in the model in detail. It mainly contains a feature decoupling module, a classifier alignment module, an intra-local adaptive module, and a global-local adaptive module. Fig. 2 shows the architecture of the federated model AdaptPFL.

3.1 Feature DecouplingModule

We decouple palmprint features into two non-overlapping and independent parts: identity and contextual information. The identity information feature contains the unique feature identifier of the personal information in the data, while contextual information features contain other features besides personal information. The next part of the section describes in detail how these two types of features are extracted.

Identity-Related Extraction

This paper focuses on extracting identity-related features from palmprint images. The key identity information is primarily contained within the three main palm lines. We employ edge detection methods to extract these features that effectively capture identity information from the primary palm lines. Additionally, the resulting grayscale image helps mitigate the influence of lighting conditions, further enhancing the robustness of the feature extraction process.

We use a deep edge detection algorithm to accomplish feature extraction of identity information. A deep convolutional neural network consists of many layers, and different layers can learn various levels of features. Shallow layers can extract basic physical features, and advanced features, such as trunk lines, can be extracted as the layer depth increases. Since training an edge detection model requires a large amount of labelled data, we first train the model on a publicly available benchmark edge detection dataset. The best learning parameters are saved after the training is completed, and then the parameters are introduced for training using the migration learning method. The main palm line feature is extracted by fine-tuning it, and this feature is used to contain identity information.

Inspired by the HED [Xie and Tu, 2015] method, the architecture of the edge detection algorithm chosen is shown in the identity-related extraction module on the top right of Fig. 2. The model uses the VGG16 network as the backbone network, and the specific training process contains two phases. The first phase receives information through five different-sized receiving fields. The second stage connects a side-output layer i to the last convolutional layer of each receptive field to learn multi-scale features. Finally, the outputs of the five side-output layers are connected as inputs to the final output layer to achieve edge detection.

For simplicity, we set of all network layer parameters is denoted as W. The network has five side output layers, each of which is also associated with a classifier. The corresponding weights are denoted as w = (w(1) , · · · ,w(5) ), and the corresponding fusion weights are denoted as h = (h1, · · · , h5). Its specific computational procedure is shown below:

Contextual-Related Extraction

This paper introduces a division strategy to extract the features of contextual information. The strategy divides the data feature X into two parts in the same metric space, making X = Xi + Xc. Specifically, the data feature X ∈ Rd with dimension d is decomposed into the identity information feature Xi ∈ Rd and the contextual information feature Xc ∈ Rd . Intuitively, the identity information feature contains all the labelled information. In contrast, the contextual information feature contains all the data except the labelled information.

As shown in the contextual-related extraction module in the lower right of Fig. 2, the edge detection method has obtained the initial features of personal identity information. We use mutual information to generate contextual features Xc for the data X. This paper aims to ensure that the features after removing the personal identification information are close to the original features. More specifically, the contextual information features are obtained with the goal of:

where Y is the label of X, and Xi should contain necessary information about the label Y . X-Xi denotes the contextual information feature Xc, and IFF is a constant. With this strategy, we can obtain both contextual information features Xc and optimized identity information features Xi at the same time.

3.2 Classifier Alignment Module

We propose a classifier module that aligns identity and contextual information. It also designs a maximum mean difference (mmd) loss function.

Classifier Structure Design

The classifier alignment module is a multi-output network consisting of two specific predictors for predicting the outcome of identity information feature Xi and contextual information feature Xc, respectively. A ResNet18 network forms each predictor. Each predictor is a softmax classifier that makes a classification judgment after the two feature extractors extract the features to obtain Pi and Pc, respectively. Its specific computation is shown below:

mmd Loss Function Design

Since the classifiers are trained independently, their predictions of target samples may diverge, especially for target samples near class boundaries. Intuitively, different classifiers should get the same prediction when receiving different features from the same sample. Therefore, this paper aims to minimize the differences between all classifiers.

To align the distribution of predicted outcomes for identity and contextual information features, we choose mmd as an estimate of the difference in the distribution of predicted outcomes for the two features. mmd is based on the idea that all statistical data are the same if the generated distributions are the same. Formally, mmd defines the following measure of

where H is the reproducing kernel Hillbert space (RKHS) with feature kernel k, where ϕ(·) denotes some feature mapping that maps the original sample to the RKHS. k denotes k(Xi, Xc) =< ϕ(Xi), ϕ(Xc) >, where < ·, · > denotes the inner product of the vectors. We use Eq. (6) to estimate the difference between the predictions of the two classifiers. The formula for the Lmmd loss is shown below:

Each classifier can reduce the variance between classifier predictions by minimizing Eq. (7).

3.3 Intra-Local Adaptive Module

Consider that Xi contains important identity information and Xc contains rich auxiliary information. Therefore, we use the weighted average of the predictions Pi and Pc generated by Xi and Xc for inference. We introduce a personalization weight λi ∈ [0, 1] for the i - th client, where the weight difference increases personalization. The client uses P for integrated prediction, which is computed as shown below:

However, the optimal value of λ may differ for each client due to client drift issues. Therefore, we propose to learn an appropriate value of λi for each client instead of setting a common value for it centrally. We randomly split a small subset from the training set DT to train the i - th client for λi, name it the adaptation set and denote it by DλA. We learn λi by training on the adaptive set, which is computed using the cross-entropy loss function. Given the predictions generated by the identity information model and the contextual information model, the learning process of the loss function Ladapt and the learning rate η on the adaptive set are denoted as:

The algorithm uses stochastic gradient descent (SGD) for small batch iterations while sandwiching λ in [0,1].

3.4 Global-Local Adaptive Module

While traditional aggregation methods can generate a global model, the global model performs poorly in generalization on a per-client basis. Inspired by FedALA [Zhang et al., 2023], this paper proposes an adaptive aggregation method between global-local models. The method judiciously aggregates global and local models to fit local goals. We show the adaptive aggregation method's learning process in Fig. 3.

In the r - th iteration of traditional federated learning, the server sends the old global model θr-1 to the client k. θr-1 overwrites the old local model θr-1 k to get the initialized local model ˆ θrk for local model training, i.e., ˆ θrk = θr-1 . In contrast, we propose an adaptive method to aggregate the global and local models instead of overwriting. The specific calculation method is shown as follows:

where ⊙ is the Hadamard product, Wk,1 and Wk,2 are the learnable weight matrices and Wk,1 +Wk,2 = 1. For coverage, Wk,1 = 0 and Wk,2 = 1.

However, learning Wk,1 and Wk,2 with constraints by the gradient-based learning method is difficult. Therefore, we merge Wk,1 and Wk,2, and the modification to Eq. (11) is shown below:ˆ

where Wk is the learnable weight matrix.

3.5 Overall Loss Function Design

The loss function Ltotal of the method consists of three parts: classification loss Lcls i and Lcls c for the identity information module and contextual information module, Lmmd loss, and Ladapt loss. We use the cross-entropy loss as the classification loss for the two classifiers, as shown in Eq. (13) and Eq. (14). By minimizing the classification loss, the network can classify accurately. By minimizing the mmd loss, the network can reduce the differences between the classifiers. The network can better integrate the identity and contextual information modules by minimizing the Ladapt loss. The formula for the total loss Ltotal is shown below:

where β is a hyperparameter, training is mainly done using the standard SGD algorithm.

4 Experiment

We evaluate the method's effectiveness on three public noncontact palmprint datasets. The method is compared with SOTA models, and the basic components of the models are further analyzed.

4.1 Datasets

To evaluate the method's recognition accuracy, extensive experiments and analyses were conducted on three contactless palmprint databases, XJTU-UP [Shao et al., 2019], MPD [Zhang et al., 2020], and Multi-Spectral [Zhang et al., 2009]. Table 1 summarises the details of the three datasets.

XJTU-UP: An unconstrained palmprint database collected from two smartphones, iPhone 6S and Huawei Mate8. A total of four datasets are included, which are noted as HuaWei Nature (HWN), HuaWei Flash (HWF), IPhone Nature (IPN) and IPhone Flash (IPF).

MPD: Palm images are collected by two smartphones, HuaWei and XiaoMi, denoted as HW and XM, respectively.

Multi-Spectral: Palmprint images are taken from different spectral bands, including blue, green, red, and near-infrared.

4.2 Comparison of Other Methods

We validate the effectiveness of the methods through several experiments. Three baseline methods are included: FedAvg [McMahan et al., 2017], FedProx [Li et al., 2020b], and SCAFFOLD [Karimireddy et al., 2020]. Three SOTA methods are included: FedAPEN [Qin et al., 2023], FedFed [Yang et al., 2024], and FedML [Shao et al., 2024].

As can be seen from the experimental results in Table 2, our method produces excellent performance on all three datasets. In particular, on the XJTU-UP dataset, which is poorly recognized by the SOTA method, our method's F1 value can still achieve excellent performance of 97.17%. Some SOTA methods have also produced better performance on the multispectral dataset. This is because Multi-Spectral images capture different bands of light, some of which (e.g., nearinfrared light) are less sensitive to changes in ambient light. Therefore, changes in the external environment have less effect on them and produce better performance. However, the AdaptPFL method still shows the best performance on this dataset.

the Global-Local Adaptive Aggregation (-GLAA). The experimental results show that the method's performance is degraded to different degrees when the core modules are removed. In particular, the method's performance degrades most severely when we remove the identity information module after detangling. This indicates that the decoupling mechanism we designed effectively separates the identity information from the palmprint features. The performance degradation is more severe after GLAA removal compared to ILAA. This is because the inter-client drift phenomenon is much larger than the intra-client drift phenomenon. The results of the ablation studies show that all our core modules are effective.

4.4 HistogramAnalysis of Feature Distribution in Different Conditions

The section verifies the method's validity by randomly selecting the distribution of palmprint features generated by a batch of volunteers under different scenes. Specifically, by

4.3 Ablation Study

We conduct ablation studies on three contactless palmprint datasets to validate the effectiveness of the method's core modules. The results are shown in Table 3.

This paper conducts four types of ablation studies: removal of the Identity Information Module (-IIM), removal of the Contextual Information Module (-CIM), removal of the Intra-Local Adaptive Aggregation (-ILAA), and removal of

(b) AdaptPFL method

summing and averaging the palmprint features of each volunteer in various scenarios as the current valid features, the palmprint feature distribution of the same batch of volunteers in different scenarios is formed. We mainly verify the effect of the drift of two scenes, mobile phone brand and light, on the feature distribution. The specific experimental results are presented in Figs. 4 and 5.

The experimental results show that the traditional federated learning method using different mobile phone brands or light conditions produced two different palmprint feature distributions for the same group of volunteers. This indicates that traditional federated learning is susceptible to the influence of external conditions to produce client drift. The present method produces similar feature distributions in different scenarios, which indicates that it can effectively mitigate the client-side drift phenomenon produced by external factors.

5 Conclusion

This paper proposes an adaptive personalized federated learning method based on feature decoupling for cross-device palmprint recognition. The method designs a decoupling mechanism to decompose palmprint features into identity and contextual information, effectively isolating the mutual influence between external factors and identification. Meanwhile, we present two adaptive aggregation strategies for intra-client and inter-client drift phenomena, respectively. They used an intra-local adaptive module and a global-local adaptive module to correct the drift phenomenon further. We verify the method's effectiveness on three public non-contact palmprint datasets. The results show that the present model performs better than SOTA methods.

Acknowledgments

This work was supported by National Natural Science Foundation of China (Nos. 62472220, 62371234, 62076129), Jiangsu Province 100 Foreign Experts Introduction Plan (BX2022012), Natural Science Foundation of Jiangsu Province (No. BK20231438), and Key Research and Development Plan of Jiangsu Province (No. BE2022842).