Federated Probabilistic Preference Distribution Modelling with Compactness Co-Clustering for Privacy-Preserving Multi-Domain Recommendation

1 Introduction

Cross-domain recommendation (CDR) systems have shown great success in alleviating the serious data sparsity problem [Guo et al., 2021; Li et al., 2022; Zang et al., 2021a; Zheng et al., 2022]. It is well acknowledged that CDR models always share the user-item rating information across domains for joint collaborative filtering. Such methods can provide better performance on both source and target domains. However, with the increasing law on privacy protection, the user-item rating information are not accessible among different domains [Chen and et al, 2022]. How to provide highquality cross- and multi-domain recommendations under the demand of privacy protection has become an urgent problem.

∗Chaochao Chen is the corresponding author.

In this paper, we focus on a more general problem of Privacy-Preserving Multi-Domain Recommendation (PPMDR). That is, user-item rating interactions are regarded as private information and other domains (participants) cannot directly obtain them. Such demand makes it difficult for knowledge transfer across domains to extract users' domaininvariant preferences. Meanwhile, these multiple domains all suffer from the data sparsity problem. It further hurdles the model to learning reliable representations and limits the model's performance.

Existing research on CDR cannot solve the PPMDR problem well. Firstly, most conventional CDR models need to access the whole user-item rating interactions among different domains for modelling [Hu et al., 2018; Chen et al., 2020]. While it is not suitable under the settings of PPMDR, since the user-item ratings should be protected in each domain. Recently, FedMF [Chai et al., 2020] first adopts federated learning strategy for ensuring data privacy protection. However, FedMF only adopted a shallow model and thus cannot depict the complex user-item relationship well. Secondly, most of current CDR models only utilize embeddings rather than distributions to represent users and items, which leads to an inaccurate understanding of user preference, especially under the data sparsity scenario [Jiang et al., 2020; Ma and et al, 2020]. Specifically, we provide an example in Fig.1(a)-(b) where the user Mary prefers romantic and historical items while dislikes horror items. If we only consider the user/item embeddings, the horror items Suspect X or Kill Game will be recommended to Mary since they are closer than Woman in Love or Flipped. While when we model the users/items with Gaussian distributions, the users/items can represent more richer meanings. As depicted in Fig.1(a), Mary's distribution overlaps her prefer books (e.g., Woman in Love and Global History) rather than horror items to avoid inaccurate results. While only using single domain information cannot better model users' preference distribution, as shown in Fig.1(b), Mary's local distribution with dashed eclipse does not overlap with the romantic movie Titanic that she likes. Then we should adopt the global server to aggregate the user distributions for knowledge sharing in Fig.1(c). Meanwhile, different domains are always heterogeneous with diverse kinds of items. Hence, it could be rather difficult to directly apply the commonly-used federated learning methods for information gathering. What is more, most of the CDR models only concentrate on user-item interactions and neglect potential semantic relations. As shown in Fig.1(d)- (e), the user preference distributions are scattered and users with different tastes still have overlapped space. In conclusion, how to properly model and aggregate the user/item preference distribution is still a challenging problem.

To address the aforementioned issues, in this paper, we propose Federated Probabilistic Preference Distribution Modelling (FPPDM) for solving the PPMDR problem. FPPDM involves two main components, i.e., local domain modelling component and global server aggregation component where user-item rating interactions are stored in the local domain due to the demand for privacy protection. We further propose neural networks to capture user/item preference distribution via modelling the corresponding mean and covariance in the local domain modelling component. Then we aggregate these users' preference distributions in the global server aggregation component and send them back to local domains. To further leverage the user similarity relationship, we propose a compactness co-clustering method in FPPDM++. The compactness co-clustering method can gather users based on their tastes or characteristics to provide more satisfactory results. We summarize our main contributions as follows: (1) We propose a novel federal learning framework, i.e., FPPDM, for solving the PPMDR problem and protecting the privacy of user-item rating information. FPPDM involves multiple local domains and a global server to model and aggregate user/item preference distributions. (2) To achieve better results, we further propose a compactness co-clustering method to leverage similar users' information in FPPDM++. (3) Extensive empirical studies on Douban and Amazon datasets demonstrate that both FPPDM and FPPDM++ significantly improve the stateof-the-art models, especially under the PPMDR setting.

2 RelatedWork

Cross- and Multi-Domain Recommendation. The existing CDR models have two main types, i.e., single-target CDR models and dual-target CDR models [Zang et al., 2021b;

Zhu et al., 2021; Zhang et al., 2022b; Yuan et al., 2019; Wang and et al, 2022; Sun and et al, 2022]. Single-target CDR models leverage the information from the relevant source domain with rich data to the sparse target domain. Conventional single-target CDR models aims to adopt matrix factorization mechanism for collaborative filtering on overlapped users/items [Lian et al., 2017]. More recently, some single-target CDR models also adopt the deep neural networks to users/items preferences [Elkahky et al., 2015; Hu et al., 2018]. Latest PriCDR [Chen and et al, 2022] further considers the privacy protection when distilling the information from rich to sparse domains. Dual-target CDR models aims to simultaneously share and transfer information when both source and target domains are sparse. DOML [Li and Tuzhilin, 2021] first tried attempt to share dual knowledge transfer based on orthogonal mapping. Recently, researchers also utilized graph-based approaches [Zhao et al., 2019; Liu et al., 2020; Cao et al., 2022] for solving dual-target CDR problem. Zhao et.al. [Zhao et al., 2022] further enhanced the method with gate mechanism into multi-domain recommendation. While the above methods should distribute the local user-item ratings across different domains which could raise the concern on privacy protection. Hence they cannot be suitable for solving the PPMDR problem.

Federated Learning. With the increasing emphasis on protecting personal privacy, federated learning is widely used in diverse applications [Zhang et al., 2022a; Meihan et al., 2022]. FedAvg [McMahan et al., 2017] was first be proposed for protecting the clients' privacy in distributed learning. To better satisfy the non-identical independent fashion, proximal regularization term [T Dinh et al., 2020] or primal-dual algorithm [Zhang et al., 2021; Acar et al., 2021] have been added in the local clients. Furthermore, shared representations have been exploited [Pillutla et al., 2022] to provide personal solutions. Meanwhile knowledge distill method [Lin et al., 2020] and prototype alignment [Tan and et al, 2021; Dong et al., 2022] have been proposed with the consideration of heterogeneous data distribution or network architecture. More recently, FedMF [Chai et al., 2020] successfully adopted the federated learning into the field of cross domain recommendation. Latest FedCDR [Meihan et al., 2022] also adopted federated learning into solving cold-start CDR problem. However, FedMF only adopted shallow model which cannot capture and depict the complex and complicate useritem relations. While FedCDR separated the rating prediction stage and transfer stage which limits the model performance. As a comparison, in this paper, we extend the federated learning with PPMDR problem by adopting distribution aggregation method with compactness co-clustering strategy.

3 Methodology

First, we describe notations. We assume there are K (K ≥ 2) domains (clients) as {D(1) ,D(2) , · · · ,D(K) } where D(i) denotes as the i-th domain (client). We assume each domain share the same set of NU users. Meanwhile each domain has N (i) V different types of items. Let R(i) ∈ RNU×N (i) V be the observed rating matrices in i-th domain D(i) and users in different domains are overlapped. To simplify the problem, in this paper, we assume both domains have no other auxiliary information. Meanwhile, the rating matrix R(i) is the private data which cannot be directly shared in the public space.

Then, we introduce the overview of our proposed FPPDM framework, as is illustrated in Fig. 2. FPPDM mainly involves two components, i.e., local domain modelling component and global server aggregation component. The i-th local domain stores the user-item rating R(i) in i-th domain D(i) . The local domain aims to model user/item probabilistic preference distribution based on the rating interactions. The global server aggregation component aims to gather the overlapped user information for knowledge sharing. To achieve more better recommendation results, we further provide FPPDM++ with compactness co-clustering method. This method tends to exploit and cluster users with similar tastes and characteristics.

3.1 The Framework of FPPDM

Local Domain Modelling Component

Firstly, we provide the details of the local domain module in FPPDM. For the i-th user and the j-th item in the D(k) domain, we define their corresponding one-hot ID vectors as XU(k) i and XV(k) j , respectively. We adopt a trainable lookup table to exploit the user/item one-hot ID embedding as EU(k) i = LookUp(XU(k) i ) and EV(k) j = LookUp(XV(k) j ) respectively. To better aggregate useful information among the user-item interactions, we further adopt the commonlyused graph neural network in modelling. Then we first build up the corresponding interaction A(k) graph among users and items as A(k) = [ 0 R(k) ]. Then we can conduct

the graph convolution neural network for modelling user/item preference distribution:

where E(k) = {EU(k) ,EV(k) } denotes the set of users and items in D(k) . The W (k) µ and W (k) σ denote the trainable weights. GCN(·) denotes the graph convolution network operation which can be computed as:

where ˜D(k) = diag( ˜ A(k)1) denotes the degree matrix for the graph ˜ A(k) and ˜ A(k) = A(k) + I. Specifically, we adopt ℓ-th layers of graph convolution network layers to achieve the users/items' mean µ and covariance σ2 of their embeddings distribution as follows:

where P(U (k) ) and P(V (k) ) denote the local user and item distributions respectively. Since using the single user or item embeddings cannot precisely depict user-item relationship, we adopt the Gaussian distribution to parameterize user and item distribution. Specifically, the Gaussian distribution can capture the complex and complicated user/item preference via measuring the covariance. After that we should train the model to fit the observed user-item ratings. To better achieve this goal, we propose the distribution-based metric learning loss as given below:

where O(k) denotes the positive user-item pairs. V (k) neg,(i) denotes the negative items for the i-th user. The D(·) denotes Wasserstein distance among different Gaussian distributions which can be calculated as:

After adopting the metric-based rating prediction loss, we can pull the positive user-item pairs while push away the negative user-item pairs. Meanwhile, the local and global overlapped user distributions should be consistent for knowledge sharing. To this end, we propose a regularization term to reduce the distance among these corresponding local and global user distributions as follows:

where N denotes as the batchsize. P( ̂Ui) = N(̂µi, ̂σ2 i ) denotes the i-th global user distribution. By combining the rating prediction loss L(k) R with the regularization termL(k) P , we can obtain the total loss in local domain modelling component:

where λP denotes as the balanced hyper parameter. After the training in local domains, we can obtain the local user distribution P(U (k) ) and send them to the global server.

Global Server Aggregation Component

Since the global server have received the local user distributions from multiple different domains, the global server should aggregate and update the global user distributions P( ̂U). This process is important since the data sparsity problem will deteriorate the user-item modelling in local domain. As an example in Fig.1(b), the local user distribution of Mary will not fully cover her interests since it still away from the romantic movie Titanic. Then we adopt the server aggregation

Algorithm 1 FPPDM / FPPDM++

• 1: Server executes:
• 2: Initialize global user distribution as P( ̂U).
• 3: for round = 1 to T do
• 4: for each domain = k to K in parallel do
• 5: Obtain P(U (k) i ) ←LocalUpdate(i, P( ̂U))
• 6: end for
• 7: Update P( ̂U) via Eq.(7) and send to local domain.
• 8: end for
• 9: LocalUpdate(i, P( ̂U)):
• 10: for epoch = 1 to t do
• 11: Calculate the rating prediction loss in Eq.(4).
• 12: Calculate the regularization loss in Eq.(5).
• 13: if The model is FPPDM++ then
• 14: Optimize the compactness similarity in Eq.(8).
• 15: Optimize the assign indication in Eq.(10).
• 16: Calculate compactness co-clustering loss inEq.(11).
• 17: end if
• 18: Update the local model.
• 19: end for
• 20: Return: Well-trained local model.

Then we introduce the details of our proposed compactness co-clustering method. To start with, we tend to figure out the similarity s(k) ij between the i-th and j-th users in the D(k) domain. Meanwhile we suppose to cluster the users into M groups to achieve more compact user representations. Specifically, we present the compactness similarity optimization problem as follows:

where L(k) = diag(S(k)1) - S(k) denotes the Laplacian similarity matrix. ϵ denotes the hyper parameter to balance the similarity matching and the entropy regularization term s(k) ij log s(k) ij . Meanwhile the entropy regularization term is set to obtain the nonnegative and nontrivial solution [Bai and Liang, 2020; Nie et al., 2014]. Furthermore, we add the rank constraint, i.e., rank(L(k) ) = N -M, on Laplacian similarity matrix L(k) to obtain more compact results

to obtain the global user preference distribution in Fig.1(c) and after that it will enhance the model performance in the local movie domain in Fig.1(b). To fulfill this task, we tend to exploit the new global user distributions P( ̂U) which has the smaller Wasserstein distance among these local user distributions as [minP( ̂U) 1K ∑Ni=1 ∑Kk=1 D(P( ̂Ui), P(U (k) i ))]. By taking the differentiation w.r.t. on ̂µi and ̂σi, we can easily obtain the optimal results as:

After we achieve the aggregated global user distributions, we further send them back to the local domains. Meanwhile only mean and covariance of user preference distributions will be transmitted to the global server which makes it different from conventional federated learning methods (e.g., FedAvg [McMahan et al., 2017]). Note that it is difficult to reconstruct the raw user-item rating interactions from user preference distributions. Furthermore, one can even adopt encryption transportation method (e.g., homomorphic encryption [Gentry, 2009; Aono et al., 2017]) on mean and covariance to strengthen security level.

3.2 The Framework of FPPDM++

Although FPPDM can share and transfer useful knowledge via the overlapped users across domains, the user preference distributions will become scattered in the latent space as shown in Fig.1(d). Specifically, the purple and green dots represent the users with different types of preferences in Fig.1(d). Some users (e.g., purple #7 and green #2) with different preferences still have overlapped distributions. The scattered latent distributions will hurdle the model performance by providing inaccurate results. The main reason behind is FPPDM cannot fully exploit and incorporate the semantic neighbors to enhance the model performance. That is, the users with similar tastes or preferences are their semantic neighbors who should be clustered. Meanwhile utilizing the co-clustering strategy can even filter out inherit data noise to further reduce data sparsity problem [Lin et al., 2022b; Lin et al., 2022a]. However, previous methods mainly focus on the simple embedding situation and they cannot be directly applied for distribution clustering. To resolve this issue, we first propose compactness co-clustering method in FPPDM++ to enhance the model performance by extracting useful semantic information among these preference distributions.

by avoiding the situation that most users clustered in one group. However, it is difficult to directly optimize the rank constraint. Here, we make the approximation by calculating the minimal eigenvalues of L(k) to replace the rank constraint. In other words, rank(L(k) ) = N-M is equivalent to ∑Mm=1 ϕm(L(k) ) where ϕm(L(k) ) denotes the m-th smallest eigenvalue of L(k) . According to the Ky Fan's theorem [Fan, 1949], ϕm(L(k) ) can be calculated as:

where F (k) ∈ RN×M is the indicator matrix. We provide the Lagrange multipliers for the original problem as follows:

where η is the hyper parameter and γ (k) i is the multiplier. We further make simplification for rank constraint since ∑Ni=1 ∑Nj=1 s(k) ij ||f (k) i -f (k) j ||22 = 2Tr((F (k) )⊤L(k)F (k) ). We first fix and F (k) and update the S(k) as:

Then we fix S(k) and update F (k) . The optimal solution F (k) can be obtained by the M eigenvectors of L(k) corresponding to the M smallest eigenvalues. We can alternatively update S(k) and F (k) to achieve the optimal solution.

Once we obtain the indicator matrix F (k) , we tend to figure out the relationship between each data and the corresponding clusters. To achieve this goal, we propose entropy-based KMeans method as follows:

where Z(k) denotes the cluster centers of assign indicator matrix F (k) . That problem can be solved via iteratively optimize ζ (k) and Z(k) . Due to space limits, please kindly refer to [Bai and Liang, 2020; Liu and et al, 2023] for more details on the optimization process. After we obtain the optimal solution of ζ (k) , we can further calculate the results of the corresponding cluster distribution P(C(k) j ) =N(˜µ(k) j , (˜σ(k) j )2 ) as:

After we obtain these results, we tend to narrow the distance between the local user distribution and these cluster distributions. Therefore, we further provide the compactness coclustering loss as follows:

Using the compactness co-clustering, the users with similar tastes or preferences will have more compact representations as shown in Fig.1(e). Finally, we can combine the rating prediction loss ℓ(k) R , local-global regularization loss ℓ(k) P , and compactness co-clustering loss ℓ(k) C for the total loss of proposed FPPDM++:

where λP and λC denote the balance hyper parameters. Note that the FPPDM and FPPDM++ have the same global server aggregation component during training process. The training algorithm details have been provided in Algo.1.

4 Experiments

4.1 Experimental Setup

Datasets. We conduct experiments on two popuarly used real-world datasets, i.e., Douban and Amazon. First, the Douban dataset [Zhu et al., 2019; Zhu and et al, 2021] has three domains, i.e., Book, Music, and Movie. Second, the Amazon dataset [Zhao et al., 2020; Ni et al., 2019] has six domains, i.e., Phone, Electronics (Elec), Cloth, Sport, Game, and Video. The detailed statistics of these datasets are given in Table 1. For each dataset, we binarize the ratings higher or equal to 4 as positive. We also filter the users and items with less than 5 interactions, following existing research [Zhu et al., 2019; Liu and et al, 2022]. We conduct three crossdomain recommendation tasks in Amazon when only two domains (clients) involve (K = 2), i.e., Phone and Elec, Cloth and Sport, and Game and Video. Meanwhile we conduct multi-domain recommendation on the whole Douban datasets (K = 3). We assume that users in different domains are overlapped in the same task, and there are two/three clients involves in Amazon/Douban tasks respectively.

Baseline. We compare our proposed FPPDM with the following state-of-the-art models. (1) NeuMF [He et al., 2017] first adopts deep neural network for collaborative filtering in single domain. (2) PMLAM [Ma and et al, 2020] first adopts probabilistic user/item modelling with metric-based loss in single domain. (3) LightGCN [He et al., 2020] adopts the graph neural network to model user/item interactions in single domain. (4) FedMF [Chai et al., 2020] is the first attempt to adopt federal learning framework across domains. (5) FedGNN [Wu et al., 2021] further adopts the graph neural network into federated recommendation. (6) PriCDR [Chen and et al, 2022] is the single-target CDR model while sharing the information from the rich domain to the sparse domain.

(7) DOML [Li and Tuzhilin, 2021] adopts the dual deep transfer module with orthogonal constraints across domains. (8) ETL [Chen et al., 2020] utilizes equivalent transformation with autoencoder framework for cross-domain modelling. (9) CDRIB [Cao et al., 2022] is the state-of-the-art graph-based cross domain model with information bottleneck theorem. (10) MSCDR [Zhao et al., 2022] is the state-of-the-art graphbased multi domain model with attention mechanism. We adopt the same user-item ratings without other auxiliary information for our propose methods and these baselines.

Implemented details. We set batch size N = 256 and embedding dimension as D = 128 across different domains. We set the hyper parameter λP = 0.05 for local client modelling component in both FPPDM and FPPDM++. The number of graph encoder layer is set to 3 in each clients, following [He et al., 2020; Wang et al., 2019b]. Meanwhile set the hyper parameter λC = 0.07 and the number of clusters as M = 15 for compactness co-clustering loss in FPPDM++. Besides, we set the balance hyper parameter ϵ = 0.05 and

η = 0.1. We conduct training for FedCDR, FPPDM and FPPDM++ with 5 local epochs per round until converge. We choose Adam [Kingma and Ba, 2014] as optimizer, and adopt HR and NDCG [Wang et al., 2019a] as the evaluation metrics. For all the experiments, we perform five random experiments and report the average results. We report the results measured by the commonly used metrics as Top@5 and Top@10 in Douban and Amazon datasets, respectively.

4.2 Recommendation Performance

The comparison results on Douban and Amazon datasets are shown in Table 2. Note that conventional CDR models (e.g., DARec, DOML, and CDRIB) cannot directly adjust to the multi-domain recommendation in Douban. Therefore, we set Douban Movie as the source domain and Douban Book, Douban Music as the target domain for dealing with these CDR models. From that we can conclude: (1) Modelling the user preference distribution can achieve much better results by comparing PMLAM with NeuMF. However, single domain methods cannot fully utilize the knowledge across domains which make them hard to tackle the sparsity problem. (2) Combining multiple domains information can enhance the model performance by comparing NeuMF with DOML. Meanwhile, recent models (e.g., CDRIB) which adopt graphbased framework even achieve much more promising results than single domain graph-based model (e.g., LightGCN). (3) The federal learning framework (e.g., FedMF) can even exceed the performance than single domain methods (e.g., NeuMF) by combing knowledge and protecting privacy. Nevertheless, the non-deep-learning based FedMF cannot handle the complex and complicated user-item relationship. Meanwhile, single-target based PriCDR simply transferred knowledge from rich to the sparse domain. Thus it cannot better handle the situation when different domains are both sparse. Latest FedCDR model also separates the training and mapping process limits the model performance. (4) Although latent models (e.g., MetaDPA and MSCDR) reach better performance, they still only extract user/item embeddings which limit their model potentials. These deterministic manner cannot capture and depict complicated user-item relationships. (5) Our proposed FPPDM achieves much satisfied results which indicates the efficacy of FPPDM by integrating preference distribution across different domains. Furthermore, FPPDM++ via compactness co-clustering can gather users with similar characteristics which improve the model performance. Hence FPPDM++ even achieve much better results than proposed FPPDM.

4.3 Analysis

Ablation. We compare FPPDM/FPPDM++ with its several variants, including FPPDM++(B) and FPPDM++(P) to study how does each component of FPPDM++ contribute to the final performance. FPPDM++(B) and FPPDM++(P) both adopt the deterministic training manners without estimating the user covariance. While FPPDM++(B) excludes compactness co-clustering module and set λC = 0. We conduct the FPPDM++(B) and FPPDM++(P) on Amazon Phone and Elec and report the results (HR and NDCG) in Fig. 3. By comparing the FPPDM and FPPDM++(B) we can conclude that modelling user preference distribution can better capture diverse and complex relationships. However, they cannot gather similar users in the latent space limits the models' accuracy. We can further observe that FPPDM++(P) exceeds the FPPDM++(B) which indicates that clustering users similar tastes or characteristics will boost the model potentials.

Moreover, combing distribution modelling and compactness clustering can greatest improve the model performance.

Method Extension. We further analyse the general extension of our proposed federated framework. To validate this, we apply our proposed FPPDM and FPPDM++ into the MLPbased dual autoencoder model ETL as ETL-F and ETL-FC on conducting the experiments on Douban datasets respectively. We report the results HR and NDCG on Table 3. From that we can conclude our proposed federated framework is model-agnostic. That is it can be applied in both graph-based or MLP-based recommendation approaches. Meanwhile utilizing the federated learning strategy by safely sharing knowledge can even enhance the model performance.

Effect of hyper-parameters. We finally study the effects of hyper-parameters on model performance. We first conduct experiments to study the effects of λP by varying them in {0.01, 0.02, 0.05, 0.07, 0.1, 0.5} for FPPDM++ on Amazon Cloth and Sport and report the results in Fig.4. It is difficult to apply local-global regularization alignment for knowledge sharing on each clients when λP is too small. While when λP is too large, it will slightly hurdle the rating prediction modelling. Therefore we set λP = 0.05 empirically. We further vary λC in {0.01, 0.02, 0.05, 0.07, 0.1, 0.5} and report the results in Fig.4. Similarly, the bell-shaped curve of λC indicated that too large or too small value of λC will not suitable in training the model and we set λC = 0.07. Finally, we conduct the experiments on varying the number of cluster M in {5, 10, 15, 20} and report the results on Table 4. When M is smaller, it cannot better depict the users with similar tastes. By comparing M = 20 and M = 15, the performance improvement is rather marginal by consuming more time and space. Hence we set M = 15 empirically.

5 Conclusion

In this paper, we investigate the Privacy-Preserving MultiDomain Recommendation (PPMDR) problem. To tackle this problem, we first propose Probabilistic Preference Distribution Modelling (FPPDM), which includes the local domain modelling component and global server aggregation component. FPPDM can model and share user/item preference distribution across different domains with federated learning strategy. To better exploit useful semantic information by clustering users with similar characteristics, we further propose FPPDM++ with compactness co-clustering method. We also conduct extensive experiments to demonstrate the superior performance of our proposed FPPDM models.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (No.62172362) and CCF-AFSG Research Fund.