Federated Adaptation for Foundation Model-based Recommendations

1 Introduction

Recently, the Foundation Models (FMs) [Radford et al., 2019; Bommasani et al., 2021; Achiam et al., 2023] emerge rapidly and have made breakthroughs in various AI applications, ranging from language [Alayrac et al., 2022], vision [Saharia et al., 2022], reasoning [Kojima et al., 2022] and recommendation [Geng et al., 2022]. FMs are typically trained on extensive data sources, allowing them to capture and utilize inherent common knowledge. This capability empowers FMs to achieve outstanding performance in various downstream tasks. Applying foundation models to recommendation systems is considered a highly promising direction, which has significantly propelled the state-ofthe-art in recommendation system studies [Harte et al., 2023; Liu et al., 2023a; Lin et al., 2023].

∗This works was done during Chunxu Zhang was an intern at Institute for AI Industry Research, Tsinghua University.

†Corresponding authors.

Two new open challenges have been encountered when we introduce the foundation models into a practical recommendation system. First, given the fast changes in user preference, how to timely update the foundation models-based recommendation system with reasonable cost on communication and computation. An on-device parameter-efficient fine-tuning mechanism is desired to be incorporated with the foundation models. The second challenge is how to tackle the privacy-sensitive data of users that is needed to train or fine-tune the foundation model in a recommendation system.

Federated learning trains the global model by iterative model parameters transmission between server and clients without accessing private client data [McMahan et al., 2017; Miao et al., 2023; Zhong et al., 2023; Liu et al., 2024]. Due to its excellent privacy-preserving properties, federated learning has become a popular privacy protection enhanced scheme for recommendation, named federated recommendation systems [Chai et al., 2020; Yang et al., 2020; Wu et al., 2022; Sun et al., 2022; Zhang et al., 2023a; Li et al., 2023]. Trained on extensive data, the foundation model's attribute embedding and prediction function retain valuable general domain knowledge and user decision logic. Integrating federated learning into foundation model-based recommendation systems can benefit from this shared knowledge while ensuring privacy preservation. However, effectively addressing this integration remains unsettled.

Several challenges must be addressed urgently to adapt foundation model-based recommendations with federated learning frameworks. Challenge 1: efficient user personalization modeling. Given the substantial preference differences across isolated clients, the uppermost goal for this recommendation system is effective personalization modeling of diverse users under the privacy protection limitation.

At the same time, there are usually concerns about efficiency when using large-scale foundation models, which is more urgent in federated learning that requires end-device optimization. Challenge 2: common knowledge and user personalization fusion. Common knowledge maintained in the pretrained foundation model can incorporate insights from collective user behavior. By integrating it with user personalization, the federated recommendation system can leverage the collective intelligence of the user community, leading to improved decision-making. However, the importance of common knowledge and user personalization varies across users, and ineffective knowledge fusion can lead to information confusion and misleading recommendations. Hence, balancing general knowledge and individual user preference emerges as a main challenge.

In this paper, we present a novel method, named Federated recommendation with Personalized Adapter (FedPA), to explore the federated foundation model for recommendation. Our method leverages the pre-trained model as the foundation, allowing us to incorporate common knowledge and optimize the federated recommendation system from a wellestablished starting point. To capture user personalization efficiently, we propose a personalized adapter to deploy on the client, which can learn individual user preferences in a lightweight manner. Then, we learn these adapters with the pre-trained model in an adaptive fusion manner to balance the collaboration of common knowledge and user personalization in federated optimization.

Motivated by research about neural network optimization [Li et al., 2018; Aghajanyan et al., 2020], the model parameter solution for the target task usually resides within an intrinsic dimension. In the federated recommendation system, each user has task-specific intrinsic parameter space that can be learned from personal data. To this end, we design a low-rank adapter to learn user personalization in a lightweight manner. Particularly, we develop two levels of personalization to accommodate the recommendation scenario, including user-level and user-group-level. They cater to the unique preferences of individual users while also capturing and leveraging shared patterns and preferences within specific user groups. Furthermore, we design an adaptive gate learning mechanism that dynamically learns the weights for common knowledge and user personalization, enabling effective knowledge fusion. During federated optimization, our FedPA focuses on updating only the parameters relevant to user-specific modeling and the others are frozen and exempted from optimization, leading to a significant reduction in communication cost and achieving faster convergence.

We assess the performance of FedPA on four benchmark datasets and compare it with various advanced baselines. Experimental results consistently demonstrate that our method outperforms the baselines by a significant margin. In addition, we conduct comprehensive experiments to analyze FedPA's ability to capture user personalization and the impact of common knowledge in the pre-trained model on federated recommendation systems. Furthermore, we validate the model's feasibility in real-world applications. By distilling the pre-trained model into a smaller size, we address the computational and storage challenges of deploying the pretrained model on edge devices. Additionally, we enhance privacy protection by leveraging the Local Differential Privacy technique. Experimental results demonstrate FedPA's stable performance with distilled smaller models and privacy preservation, affirming its practical applicability. To summarize, the main contributions are listed as follows,

• For the first time, we investigate the federated adaption paradigm for foundation model-based recommendation, named FedPA. It enables the integration of the rich knowledge encapsulated within pre-trained models while upholding privacy protection for users.
• We present a personalized low-rank adapter to learn user personalization from user-level and user-group-level in a lightweight manner. Furthermore, we design an adaptive gate learning mechanism to dynamically learn weights, allowing for the effective fusion of common knowledge with user personalization.
• Extensive experiments on four benchmark datasets demonstrate the superior performance of FedPA against advanced baselines. Additionally, FedPA also shows excellent feasibility in deploying on clients with limited computation capability and strengthening user privacy protection in federated recommendation systems.

2 RelatedWork

2.1 FoundationModels for Recommendation

Pre-training in Natural Language Processing (NLP) [Qiu et al., 2020] has witnessed significant progress, with language models like GPT [Brown et al., 2020] and BERT [Devlin et al., 2018] achieving state-of-the-art results. The pre-training and fine-tuning paradigm allows for the extraction of valuable knowledge and eliminates the need to train new models from scratch. Given its remarkable benefits, increasing research has been on developing foundation models for recommendation systems [Liu et al., 2023a; Wu et al., 2023]. The essential learning objective of recommendation is to estimate the user's preferences for a certain item set. By incorporating the foundation models into the recommendation system, it can absorb valuable knowledge and improve the system's ability for characteristics extraction and user decision pattern learning. However, existing foundation models for recommendation rely on collecting personal data from users to optimize, which poses severe risks to protecting user privacy.

2.2 Federated Recommendation System

Federated recommendation system is a rising service paradigm that can learn the model in a privacy-preserving manner [Lin et al., 2020; Perifanis and Efraimidis, 2022; Qu et al., 2023; Zhang et al., 2023b]. Existing studies focus on developing federated recommendation models with mainstream recommendation architectures, e.g., matrix factorization [Chai et al., 2020] and neural collaborative filtering [Perifanis and Efraimidis, 2022], popular recommendation tasks, e.g., POI prediction [Zhang et al., 2023c] and multi-domain recommendation [Liu et al., 2023c], and user privacy protection enhancement [Liu et al., 2023b; Huang et al., 2023]. In this paper, we investigate the integration of foundation model into federated recommendation system, which can utilize the inherent common knowledge in the pre-trained model and prosper the powerful federated recommendation system.

3 Methodology

We present a novel federated adaption paradigm for foundation model-based recommendation, named Federated recommendation with Personalized Adaptor (FedPA). In this section, we first provide an overview of the framework architecture. Then, we delve into the components to elucidate the details and summarize the workflow into an optimization algorithm. Following that, we engage in a discussion on the feasibility of deploying our FedPA in physical applications. Finally, we develop a privacy-protection enhanced FedPA that can strengthen the system's protection for user privacy.

3.1 Framework Overview

The pre-trained recommendation model is learned from a large amount of publicly available data, embodying rich knowledge. It can effectively characterize user and item attributes and possesses strong predictive capabilities for user decision patterns. Effective utilization of this common knowledge contributes to building a more powerful federated recommendation model. To achieve this, we take the pretrained model as the foundation model for the federated recommendation system, enabling federated optimization from a favorable starting point.

To efficiently capture user-specific preferences, we propose a low-rank adapter that models user personalization from both the user-level and user-group-level perspectives. Additionally, we design an adaptive gate learning mechanism that effectively integrates common knowledge and personalized knowledge for better user modeling. The model architecture is illustrated in Figure 1. In this paper, we utilize an existing recommendation architecture as the base model, and it can be easily extended to other popular architectures.

3.2 Base Model

Given user attributes, item attributes, and user interaction records, we adopt a widely used two-tower recommendation model architecture. Specifically, the model consists of two input branches, one for learning user representations based on user attributes and the other for learning item representations based on item attributes. These representations are then fed into a prediction function to estimate user preferences for items. The user interaction records serve as supervision information to guide the model in updating its parameters.

User Embedding Module. There are certain attribute information available for the users, e.g., user active degree. For each attribute i, we construct a learnable embedding table Ei ∈ Rp×d , where p is the total attribute categories and d is the embedding dimension. Then, for each user u with attributes Au, we retrieve the embedding vectors from all the attribute embedding tables based on attribute values and obtain the user representation ur by concatenating them,

where |Au| is the total attribute number of user u. Item Embedding Module. For each item v, we adopt a similar approach as the user embedding module to construct the item embedding tables and obtain item representation vr,

and |Av| is the total attribute number of item v.

Prediction Function Module. Given user representation ur and item representation vr, we use a simple MLP (MultiLayer Perceptron) as the prediction function to estimate user preferences for items,

Loss Function. To update the model parameters, we construct a loss function that encourages the model's predictions to be as close as possible to the true labels. For common implicit feedback recommendation tasks, where the label value is Yuv = 1 when a user u interacts with an item v and Yuv = 0 otherwise, we use binary cross-entropy as the loss function,

(4)

where θbase is the model parameter, including user embedding tables θue, item embedding tables θie and the MLP parameters θmlp. D is the user-item interaction record set, and |D| is the total interactions number.

3.3 Personalized Adapter

Existing federated recommendation methods typically model user personalization by preserving partial model parameters locally. However, it limits direct access from other clients and potentially hinders the collaborative context utilization. Moreover, the personal data on each client is generally limited, which can introduce biases and compromise model performance. To overcome the challenge, we propose to learn user personalization from two perspectives, i.e., user-level and user-group-level. Particularly, we devise a personalized adapter applied to the prediction function module due to its crucial role in predicting user preferences.

Drawing inspiration from research on neural network optimization, model parameters can be embedded within an intrinsic dimension [Li et al., 2018; Aghajanyan et al., 2020]. To this end, we propose a low-rank adapter that leverages low-rank matrices to model user-specific knowledge on each client. This approach offers two prominent advantages: First, it can learn user personalization from user-specific and user groups with similar characteristics, which enhances the system's ability to model individual user preferences. Besides, the low-rank matrices introduce only a small number of parameters, making it a parameter-efficient solution.

Low-Rank adapter. For each layer l of MLP in the prediction function module, it maps the input x ∈ Rd into a new space x ∈ Rk with the weight matrix Wl ∈ Rk×d . We intensify the personalization learning by adding a low-rank decomposition matrix Wlr = WaWb, where Wa ∈ Rk×r and Wb ∈ Rr×d , and the rank r ≪min(d, k). Then, the forward pass of each layer can be modified as follows,

whereWlr is responsible for learning user personalization. In the context of recommendation tasks, we develop two levels of personalization: user-level personalization and user-grouplevel personalization. They cater to individual user preferences as well as capturing patterns and preferences shared within specific user groups.

User-Level Personalization. For the user-level personalization, we aim to learn a specific low-rank adapter for each user so the parameters W u a and W u b would be preserved locally and not be shared globally. For the user u, we formulate the user-specific low-rank adapter as follows,

User-Group-Level Personalization. For the user-grouplevel personalization, we aim to learn the same low-rank adapter for users in a specific group. In recommendation systems, users with similar characteristics tend to share similar preferences. To fully leverage this information, we learn multiple groups of low-rank adapters. For each user group g, we formulate the group-specific low-rank adapter as follows,

Users within the same group share parameter W g a and W g b . It is worth noting that users in the system can be grouped in multiple ways, meaning each user u can belong to multiple groups {gi}total i=1 . For example, a user can belong to the "young adults" group while also to the "highly active degree" group. By categorizing users from multiple orthometric perspectives, our model can learn more detailed personalized parameters and enhance the user preference capture.

3.4 Adaptive Gate LearningMechanism

By incorporating the low-rank adapter, the prediction function module can learn both the shared decision patterns among users and the personalized decision logic at two granularities, i.e., user-level and user-group-level. To effectively combine common knowledge and personalized knowledge, we propose an adaptive gate learning mechanism that dynamically assigns weights to each decision branch. Specifically, we utilize a two-layer non-linear mapping to learn the weights for the branches based on the input. The fusion process can be formulated as follows,

where W1 and W2 are the parameters of the adaptive gate learning mechanism, xc is the output of the MLP layer in the prediction function module. ⊙ and Sum(·) denote elementwise multiplication and summation calculation, respectively. The parameters of the adaptive gate learning mechanism can be updated based on gradients along with other model parameters, eliminating the cumbersome manual setting of hyperparameters. This approach enhances the flexibility of multi-branch fusion, allowing for adaptive adjustment of the weights for all branches during the model's training stages.

3.5 Optimization andModel Scalability

Optimization Objective

In the federated recommendation system, each user u acts as a client and trains the local recommendation model according to the personal dataset Du. Let Lu(θu all ) denote the local model's loss function, where θu all contains of base model parameter θbase, low-rank adapter parameter θu lr and {θgi lr }total i=1 and adaptive gate learning mechanism parameter θgate. The overall optimization of the federated recommendation model can be formulated as follows,

where n is the total number of clients in the federated recommendation system. Here, we employ a naive average aggregation approach to optimize system parameters. It is also possible to use a more flexible weighted aggregation approach further to enhance the optimization objective [McMahan et al., 2017; Wang et al., 2020].

Efficient Parameter Update

To leverage the knowledge inherent in the pre-trained model effectively, we propose to freeze the item embedding module and prediction function module during federated optimization. However, since the federated recommendation system operates on different user sets, we continue to update the user embedding module to adapt to the specific characteristics. To summarize, during the federated optimization process, we focus solely on updating the parameters related to personalized user modeling, e.g., user embedding, low-rank adapter, and adaptive gate learning mechanism parameters. It effectively saves computational and communication costs.

Discussion about Model Scalability

The pre-trained recommendation models learned with abundant computational resources often have complex structures and large sizes in real-world scenarios. Deploying such models directly to the clients poses significant challenges in terms of limited storage and computational capability. To address this issue, we propose to leverage the knowledge distillation (KD) technique [Gou et al., 2021] to distill the pre-trained model into a smaller-sized recommendation model that can be accommodated by client devices. We then utilize this distilled model to warm up the federated recommendation system. This approach effectively enhances the scalability of the proposed framework in real-world applications.

3.6 Privacy-Preserving Enhanced FedPA

The distributed training nature of federated learning avoids direct exposure to private user data. To further mitigate the risk of the server inferring user privacy through model parameter reverse engineering, we integrate the Local Differential Privacy technique [Choi et al., 2018] into our method, whose basic insight is to add a zero-mean Laplacian noise to the shared model parameters before uploaded to the server. Our method's shared model parameters include user grouplevel low-rank adapter and the adaptive gate learning mechanism parameters. By adjusting the intensity of noise, we can control the privacy protection capability of the system, that is, increasing the intensity of the noise enhances the effectiveness of privacy protection.

4 Experiment

In this section, we conduct comprehensive experiments to verify the efficacy and illustrate a deep analysis of various aspects of our proposed FedPA. Implementation code is available to ease reproducibility 1 .

4.1 Experimental Setup

Datasets

We evaluate FedPA on four practical industrial recommendation datasets collected from the short video platform Kuaishou 2 , i.e., KuaiRand 3 (KuaiRand-Pure and KuaiRandsmall) and KuaiSAR 4 (KuaiSAR-S and KuaiSAR-R). For dataset split, we first divide each dataset into two subsets: one for training the foundation model and the other for training the federated recommendation system. The dataset for the federated recommendation system is further split into train, validation, and test sets for each user based on interaction timestamps, with a ratio of 6:2:2.

Evaluation Protocols

We evaluate the model performance by calculating the evaluation metrics on the test set of private users. Specifically, we take widely used AUC (Area Under Curve) and Precision as two evaluation metrics. All experimental results are in units of 1e-2 and the average values of five individual runs. For the user group level low-rank personalization, we group users based on their attribute values. It is important to note that the clients can update the corresponding user group low-rank adaptor parameters based on their own attribute values locally without exposing the attribute values to the server. Additionally, we have incorporated Local Differential Privacy technique to further protect user privacy, hence user privacy protection can be ensured in our method.

Baselines

This paper concentrates on developing a personalized federated recommendation system and investigating the assistance provided by the common knowledge contained in pretrained models. To assess the feasibility and effectiveness of our proposed FedPA, we compare it with two types of baseline models: (1) Train and evaluate the model on the private user dataset without warm-staring from the pre-trained model, i.e., w/o Warm. (2) Warm-start the federated recommendation models with the pre-trained model, i.e., w/ Warm. Specifically, we select four representative personalized federated recommendation models, including FedNCF [Perifanis and Efraimidis, 2022], PFedNCF [Pillutla et al., 2022], FedRecon [Singhal et al., 2021] and PFedRec [Zhang et al., 2023a], and the corresponding warm-starting variants as baselines. Besides, we remove the warm-starting strategy from our method, named FedPA w/o Warm, to assess the contribution of our personalization modeling insight for the federated recommendation system.

1 Code: https://github.com/Zhangcx19/IJCAI-24-FedPA

2 https://www.kuaishou.com/cn

3 https://kuairand.com/

4 https://kuaisar.github.io/

4.2 Comparison with Baselines

Table 1 shows the performance of AUC and Precision on four datasets. We provide a summary of the experimental results and discuss noteworthy observations as follows,

1. Compared to the naive FedNCF, existing user personalization modeling techniques provide modest performance improvements and, in some cases, can even degrade the model performance. Existing federated recommendation models focus on modeling user preferences based on user (item) ID attributes. The key idea behind these personalization techniques is to learn user-specific model parameters (e.g., prediction function module) via personal data. However, when the system can leverage additional attributes, it provides mode auxiliary information for model optimization and increases learning difficulty. Consequently, selecting specific model parameters exclusively for modeling user personalization loses effectiveness. In contrast, our method introduces low-rank matrices to modeling personalization from both user-level and user-group-level, which ensures common knowledge sharing and enhances model performance.
2. Warm-starting the federated recommendation systems with pre-trained models can improve their performances in most cases. For instance, when incorporating the warm-start strategy, FedNCF achieves 2.54% and 8.62% performance gains on KuaiRand-Pure and KuaiSAR-S, respectively. The pre-trained model has already learned general user preferences. By transferring the common knowledge to the federated recommendation system, it is able to supplement the beneficial information and alleviate the performance bottleneck caused by distributed data storage. In our method, we devise an adaptive gate learning mechanism. It can dynamically learn the weights for an effective blend of common knowledge and personalized knowledge and hence achieves state-of-the-art performance.
3. Our FedPA is a communication-efficient federated foundation model for recommendation. In the federated

optimization phase, our method only updates the parameters relevant to user personalization modeling, leading to significant savings in communication overhead. For example, on the KuaiRand dataset, the number of trainable parameters of baseline models is 13,649, while only 8,189 for our method, resulting in a 40% reduction. In federated recommendation systems, the number of clients is typically extensive, and frequent parameter communication between clients and servers presents a substantial challenge to system optimization. Our method effectively reduces communication overhead, making it suitable for deployment in real-world environments.

4.3 Low-Rank Personalization Analysis

Our FedPA enables the learning of low-rank personalization at both the user and user-group levels, providing comprehensive modeling of user preferences from multiple perspectives. To make an in-depth analysis of the efficacy of the two forms of personalization, we conduct two model variants: one focusing on user-level personalization and the other on usergroup-level personalization. Specifically, we take FedNCF as the baseline and incorporate two forms of personalization, denoted as w/ UP (with user-level) and w/ GP (with user-grouplevel), respectively. Given the user grouping based on their attributes, we conduct the experiments according to multiple user attributes to make a comprehensive analysis. As shown in Table 2, incorporating either user-level or user-group-level low-rank personalization into FedNCF can improve its performance. Therefore, combining these two forms of personalization allows them to complement each other and achieve superior performance.

4.4 Effect of Common Knowledge on Federated Optimization

In the recommendation model, different modules have specific roles. User embedding and item embedding focus on attribute information learning, while the prediction function captures user decision patterns. To further explore the impact of the common knowledge inherent in the pre-trained model on federated optimization, we assess the specific effects of each module in the recommendation model on model performance. Specifically, we conduct warm-start experiments using pre-trained models, freezing user embedding, item embedding, and the prediction function separately, and analyze the experimental results.

From Table 3, we can summarize two conclusions: (1) Fixing user embedding during the optimization process of federated recommendation system leads to a significant decline in model performance. This is because the pre-trained model and the federated recommendation system are trained on different user sets, and the variations between user sets make it challenging for the pre-trained user embedding to adapt well to the federated recommendation system, and consequently resulting in a degradation of performance. (2) Updating item embedding or the prediction function in the federated recommendation system leads to further performance improvement. This finding aligns with our expectations, as updating a larger number of parameters facilitates the model in learning intricate user preferences. Our FedPA, which simultaneously fixes item embedding and the prediction function, greatly alleviates communication overhead, presenting an effective compromise to balance the performance and cost of federated recommendation systems.

4.5 Lightweight FedPAwith KD

In the physical setting, the service provider generally learns a large-scale pre-trained model, which poses challenges in terms of computational and storage capabilities when deployed directly to clients. To fill the gap, we develop a lightweight FedPA with knowledge distillation technique. Specifically, we first distill a small-scale model from the pretrained large-scale model and then deploy it on each client as the base model. For a comprehensive investigation, we distill three different sizes, i.e., 8-(8, 1), 4-(32, 8, 1) and 4-(8, 1), from the original model whose size is 8-(32, 8, 1) by adjusting the embedding dimension or the prediction function architecture. As shown in Table 4, employing the distilled smallscale models for warm-starting the federated recommendation system not only preserves the model's performance but also yields performance improvements. This finding further strengthens our method's viability in real-world applications.

4.6 Privacy-Enhanced FedPA

We conduct experiments to evaluate the privacy-protection enhanced FedPA by integrating the Local Differential Privacy technique. Particularly, we set the Laplacian noise with different intensities, e.g., from 0.1 to 0.5 with an interval of 0.1, to observe the effect, and the results are summarized in Table 5. As the noise intensity increases, the model performance deteriorates, while the decline is slight when the noise is not excessively large. Therefore, a moderate noise intensity, e.g., 0.2 is desirable to strike a balance between model performance and system privacy protection capability.

5 Conclusion

In this paper, we develop FedPA, the first federated adaption paradigm for foundation model-based recommendation. It optimizes from a solid starting point, leveraging a pre-trained model as the backbone. Our method involves learning personalized adapters for each client based on local data, focusing on lightweight low-rank adapters at user and user-group levels to learn detailed personalization with complementary perspectives. In addition, we design an adaptive gate learning mechanism to effectively blend common knowledge and user personalization with dynamic weights. Our method significantly reduces computational and communication costs by updating only user-specific parameters during federated optimization. Extensive experiments demonstrate superior performance compared to advanced baselines. We address the challenge of deploying models on resource-constrained devices by distilling a compact model from the original pretrained model. Additionally, we enhance privacy protection in FedPA by incorporating Local Differential Privacy, achieving a solid balance between recommendation performance and privacy preservation.

Acknowledgments

Chunxu Zhang and Bo Yang are supported by the National Key R&D Program of China under Grant No. 2021ZD0112500; the National Natural Science Foundation of China under Grant Nos. U22A2098, 62172185, 62206105 and 62202200; the Fundamental Research Funds for the Central Universities, JLU; the Key Science and Technology Development Plan of Jilin Province under Grant No. 20240302078GX; Jilin Province Capital Construction Fund Industry Technology Research and Development Project No. 2022C047-1. Yang Liu and Chunxu Zhang are supported by the Tsinghua-Kuaishou Joint Research Program.