Self-supervised Graph Disentangled Networks for Review-based Recommendation

1 Introduction

Review-based recommendation aims to alleviate the data sparsity problem [Mao et al., 2016] in collaborative filtering methods [Koren et al., 2022] by using the reviews of users to items as auxiliary information. Textual reviews contain useful semantic information that can be associated with the basis of users for their ratings, thereby leading to several investigations [Zheng et al., 2017; Chen et al., 2018] aimed to evaluate these user reviews to improve user preference modeling and rating predictions.

Early review-based recommendation methods usually employ deep neural networks such as Convolutional Neural Networks to model the review data for recommendation [Zheng et al., 2017; Catherine and Cohen, 2017]. In addition, motivated by the attention mechanism [Vaswani et al., 2017], many attention-based methods [Chen et al., 2018; Wu et al., 2019a] are proposed to identify the different importance of components, such as sentences, reviews, and users/items for better recommendation. More recently, the success of graph neural works [Kipf and Welling, 2016] in modeling the graph data also inspires its application in review-based recommender systems [Shuai et al., 2022].

Despite their success, existing methods typically employ a holistic approach to leverage review data for user preference or user-item interaction modeling, i.e., either aggregating user/item reviews for user/item embedding learning [Chen et al., 2018; Wu et al., 2019b] or approximating the review of each user-item interaction based on learned user/item embeddings [Catherine and Cohen, 2017; Sun et al., 2020]. Recently proposed RGCL [Shuai et al., 2022] moves them forward by modeling the reviews as edge features in the useritem graph and incorporating them into the message passing process. However, user ratings of items are typically influenced by various complex latent intents, such as price, quality, appearance, etc. As shown in Figure 1, User 1 likes the printer because of its excellent quality, whereas User 2 dislikes it for its cheap cost performance. When it comes to predicting User 3's rating to the printer, we notice that User 3 is price sensitive based on his/her interactions with other items, so we anticipate a low rating score. Therefore, the complex latent factors underlying user-item interactions highlight a desire to disentangle these factors in the review-based recommendation, which is still unexplored. As a result, the representations learned by existing methods contain a jumble of entangled factors, reducing interpretability and leading to suboptimal recommendation performance.

In this paper, we propose to learn disentangled user/item and interaction representations for better and more explainable review-based recommendation. To this end, we borrow the idea from disentangled representation learning (DRL) [Higgins et al., 2016], which learns factorized representations to characterize the latent factors hidden in the data. Although introduced for some other recommendation tasks [Wang et al., 2020; Ma et al., 2020], DRL in review-based recommendation faces the following challenges.

• How to accurately identify the distribution of latent factors in the user-item interactions based on review and graph information and model the interactions at a finer granularity?
• How to design a proper self-supervised task based on the learned interaction representations to alleviate the sparsity issue and encourage disentanglement?

To tackle these challenges, we propose a novel Selfsupervised Graph Disentangled Network (SGDN) for reviewbased recommendation. In particular, we first design a disentangled graph learning module equipped with graph disentangling and factorized message passing mechanisms. The former models the distribution of latent factors in each useritem interaction jointly from semantic information in the review and structure information in the user-item graph. The latter characterizes the user preferences from various aspects based on the generated factor graphs by accumulating factorrelevant information from neighborhoods. Furthermore, we present an intent-aware contrastive learning (CL) task that can alleviate the sparsity issue. Specifically, we dynamically select positive and negative samples for each interaction based on the learned intent distributions and maximize the agreement between positive pairs compared to negative ones. In comparison to existing methods, SGDN learns disentangled representations for users, items, and interactions, allowing it to investigate the meaning of each latent factor, resulting in greater explainability for predicting user ratings. Our contributions can be mainly summarized as:

• Based on the review and graph information in review-based recommendation scenario, we make the first attempt to learn disentangled representations for users/items and interactions at a finer granularity.
• We propose a novel SGDN framework based on a contrastive learning task that alleviates the data sparsity issue as well as encourages disentanglement.
• We conduct extensive experiments on five datasets to validate the effectiveness and interpretability of SGDN.

2 RelatedWork

With the development of deep learning, many advanced text methods are used to extract semantic information from the review data. For instance, DeepCoNN [Zheng et al., 2017] models user behaviors and item properties from review data using two parallel networks. TransNet [Catherine and Cohen,

2017] extends DeepCoNN by adding an additional layer for learning the target review features. In addition, inspired by attention mechanism [Vaswani et al., 2017], DAML [Liu et al., 2019] used the local and mutual attention of CNN to learn the user/item and interaction representations. DRRNN [Xi et al., 2021] uses both target ratings and reviews for backpropagation to retain more semantic review information.

Recently, inspired by the success of Graph neural networks (GNNs) [Kipf and Welling, 2016], RMG [Wu et al., 2019b] applied a three-level attention network to learn representations of sentence, review, and user/item and a graph attention network to model interactions. RGCL [Shuai et al., 2022] incorporated review information as edge features into user/item embedding learning and designed two contrastive learning tasks as additional self-supervised signals. Summarizing existing review-based recommendation methods, they do not consider disentangling the hidden factors of user ratings, and the problem of data sparsity still exists.

Meanwhile, many contrastive learning (CL) paradigms have been designed to alleviate the data sparsity issue in recommender systems. For instance, SGL [Wu et al., 2021] utilized an auxiliary GCL task to enhance user/item representation learning via self-discrimination. HCCF [Xia et al., 2022b] conducted cross-view contrastive learning between the explicit interaction graph and the learned implicit hypergraph structure. While these works simply regard the same data instance and others as positive and negative samples, respectively, our intent-aware CL module selects proper positive and negative samples based on the learned intent distributions and further enhances the disentanglement.

Disentangled representation learning aims to learn factorized representations that reveal and disentangle the underlying latent factors hidden in the observed data [Ma et al., 2019]. When it comes to recommendation, DGCF [Wang et al., 2020] factorized the user-item graph into several intentaware interaction graphs and iteratively update them based on user-item interaction. [Ma et al., 2020] proposed a sequenceto-sequence training strategy based on latent self-supervision and disentanglement for sequential recommendation. Despite the promising performance, existing methods do not fit our task since they ignore the fruitful semantic information hidden in the review texts.

3 ProblemDefinition

In the task of review-based recommendation, we denote U (|U| = M) as the user set and I (|I| = N) as the item set. The rating record is formulated as a user-item rating matrix R ∈ RM×N , where Rij denotes the rating score of user i to item j and R denotes the set of all the possible ratings in the dataset (e.g., R = {1, 2, 3, 4, 5} in Amazon). Meanwhile, the review texts are pre-processed to a fixed-length tensor E ∈ RM×N×d , where ei,j denotes the feature of review text user i comments on item j. Then the user-item interactions E can be represented by the combination of the rating matrix and the review tensor, i.e., E = (R,E). Finally, the input data can be formulated as a user-item bipartite graph G = (U ∪ I, E). The task is to predict the values of the full rating matrix ˆR ∈ RM×N based on the graph G.

4 ProposedModel

This section gives a detailed introduction to our proposed SGDN. The overview of SGDN is shown in Figure 2, which is composed of two parts: 1) Disentangled Graph Learning (DGL) Module: factorizing the input graph based on the useritem interactions and learning disentangled representations for users/items and interactions. 2) Intent-aware Contrastive Learning (ICL) Module: introducing an auxiliary CL task to alleviate the sparsity issue and encourage disentanglement. Our demonstration is unfolded as follows.

4.1 Disentangled Graph LearningModule

To model user/item's attributes pertinent to latent factors, we design a GNN model that learns disentangled user/item and interaction representations. Each GNN layer consists of a graph disentangling mechanism which identifies the latent factors in interactions to form multiple factor graphs, and a factorized message passing mechanism that performs multichannel message passing on the factor graphs. Finally, multiple GNN layers are stacked to gather useful information from higher-order neighborhoods and make rating predictions.

Initialization

We parameterize user/item ID embeddings as free embedding matrices U ∈ RM×d and V ∈ RN×d . Then we further divide the ID embedding into K chunks for separate user/item representation learning in each channel. Specifically, the ID embedding for user i is represented as:

where uk,(0) i ∈ R dK is the chunked embedding of user i on the k-th latent factor. Analogously, v(0) j = (v1,(0) j , v2,(0) j , . . . , vK,(0) j ) is initialized as the ID embedding for item j.

For review representations, we follow [Shuai et al., 2022] to encode user i's review on item j into the vector ei,j with BERT-Whitening [Su et al., 2021], whose parameters are frozen during training for time efficiency consideration.

Graph Disentangling

Considering the fruitful semantic information, we propose to mine the distribution of latent factors from review texts. Specifically, given the review embedding eki,j of user i to item j in channel k, we present a prototype-based method to obtain the semantic score seki,j that indicates how relevant is the review (i, j) to factor k. We introduce K latent factor prototypes {ck}Kk=1 and the score seki,j is calculated as:

where Wk ∈ Rd×d is the transformation parameter matrix, ϕ(·, ·) denotes the cosine similarity and τ is the temperature hyperparameter. c ∈ RK×d is learnable parameters and initialized by the K-means clustering on the review features e.

Although the reviews can provide us some hints as to which factor user-item interactions fall into, there might be some missing information. Recalling the example in Figure 1, the review text (d) is general and cannot explicitly reflect the reason for user's rating. To tackle this problem, we propose to infer it from the neighborhood of user/item. Generally, if user i/item j frequently interacts with its neighbors based on factor k, we can draw the inference that user i might also rating item j based on factor k with high probability. On the basis of this insight, we further introduce the similarity between user i and item j in terms of aspect k to assist in judging the latent factors of interactions, which can be formulated as:

where stk,(l) i,j denotes the structural score of user i and item j on the k-th factor at the l-th layer and uk,(l-1) i /vk,(l-1) j denotes the learned embedding of user i/item j at the (l - 1)-th layer in the k-th channel. When l = 1, stk,(l) reflects the matching degree of user and item's own attributes on factor k; When l > 1, stk,(l) can integrate the information on factor k from a larger receptive field due to the iterative accumulation of factor-relevant information from neighborhoods via factorized message passing.

Having modeled the distributions of latent factors from both semantic and structural perspectives, we then combine them into the final score sk,(l) i,j representing the coefficient of the edge between user i and item j in the k-th factor graph:

where ηi,j ∈ R is a learnable parameter to balance the weights of semantic score and structural score.

Factorized Message Passing

Given the learned factor graphs, we aim to leverage message passing to accumulate factor-relevant information for user/item representation learning. Specifically, we perform embedding propagation [Kipf and Welling, 2016] in each channel, such that the information of reviews and neighboring items/users, which are relevant to the factor, are integrated into the learned user/item representations. Following [Berg et al., 2017], we treat rating score as edge type. Then for rating r, the factorized message passing from item j to user i in the l-th layer is formulated as:

where W k,(l) r,1 ∈ Rd× dK and W k,(l) r,1 ∈ R dK × dK are the parameter matrices to project the review embedding and user/item embedding to the same space in the k-th channel. Dk,(l) i = ∑p∈N(i) sk,(l) i,p and Dk,(l) j = ∑p∈N(j) sk,(l) p,j denote the degrees of user i and item j in the l-th layer of channel k.

To intuitively figure out the essence of Equation (5), we hypothesize that factor k represents price. Then the interpretation is three-fold: 1) the coefficient sk,(l) i,j is capable of filtering out the noise information of reviews and items with which user i do not interact due to price. 2) The review information eki,j of user i is collected to characterize his/her reviewing behaviors based on price. 3) The neighboring item feature vk,(l-1) j of user i is accumulated to depict his/her price-sensitive preference on items.

After message passing in each channel, we then aggregate all the factor-relevant messages as follows:

where W(l) ∈ R dK × dK is the parameter matrix and Ni,r is the set of items that user i rates with rating r. Similarly, we can obtain the aggregated feature vk,(l) j of item j.

Layer Combination and Prediction

To capture the useful information from higher-order neighbors, we further stack L graph disentangling layers to form the final representations for users/items in each channel:

Then we model the interactions between users and items from each latent factor using a Multi-Layer Perceptron (MLP) to obtain the factorized interaction feature hki,j and merging representations under all factors:

where hi,j ∈ Rd denotes the merged interaction feature between user i and item j. The predicted rating score of user i to item j is calculated as:

where w ∈ Rd is a parameter vector. We employ Mean Square Error (MSE) loss as the supervision signal:

where T denotes the observed user-item interactions in the training set.

4.2 Intent-aware Contrastive LearningModule

In this section, we design a contrastive learning task to alleviate the sparsity issue of user-item interactions. So far, we have obtained the interaction features coupling with the latent factor distributions sLi,j = (s1,L i,j , s2,L i,j , . . . , sK,L i,j ) from the disentangled graph learning module. A very important step in contrastive learning is to select reasonable positive and negative examples. Previous practice [Shuai et al., 2022] considers the embeddings of the same node/interaction from different views as positive pairs and that of different nodes/interactions as negative pairs. However, we argue that different interactions with the same rating and similar intents should also be treated as positive pairs. Therefore, for each interaction (i, j), we calculate the similarity of latent factor distribution between it and other interactions with the same rating and select the top-k ones as positive samples of (i,j), which is formulated as follows:

where P(i,j) denotes the interaction set with the same rating as anchor and P(i,j)+ denotes the set of positive samples. Rank(yi′ ,j ′ ,Kp) returns the interaction indices of the Kplargest values in y.

As revealed in a recent study [Xia et al., 2022a], CL benefits from hard negatives which are similar to the anchor. In our rating prediction task, interactions with the same rating tend to have similar embeddings under the supervision by the MSE loss. Thus to mine the hard negatives, we select the remaining interactions in P(i,j) as negative examples. Specifically, we randomly discard edges on graph G with probability p to generate two different views G1 and G2. Then we adopt the InfoNCE loss [Gutmann and Hyvärinen, 2010] to maximize the agreement of positive pairs compared to negative pairs:

where h1i,j and h2i,j are the interaction features from the two views, respectively. By dynamically generating positive and negative pseudo-labels based on the intent similarities, SGDN provides a self-supervised signal to refine the interaction features and further encourage disentanglement.

4.3 Model Optimization

We jointly optimize the recommendation model by combining the above two losses:

where λ is a hyperparameter to control the contribution of CL task towards the overall objective.

4.4 Model Complexity Analysis

For the memory cost, it is notable that we divide the ID embedding into K chunks to keep that the same as previous works [Liu et al., 2021; Shuai et al., 2022]. The extra parameters involved in the DGL module are O(|E| + K × d × d). For the time cost, the complexity of the DGL module is O(L × K × |E| × dK ) and the complexity of calculating the CL loss in Eq. (12) is O(B × |E| × d). An alternative to reduce the time complexity is selecting positive and negative samples within the batch, reducing the time complexity to O(B2 × d). Thus the overall time complexity of SGDN is O((L × |E| + B2 ) × d), which is comparable with many GNN-based recommendation methods [Shuai et al., 2022; Xia et al., 2022b].

5 Experiments

5.1 Experimental Settings

Datasets

Following [Shuai et al., 2022], we evaluate SGDN on the Amazon review dataset [He and McAuley, 2016]. Toys and Games, Office Products, Clothing, Home and Kitchen, and Tools and Home Improvement are the five 5-core subsets selected (shortened as Toys, Office, Clothing, Kitchen, and Tools, respectively). The rating scores for all the five datasets range from 1 to 5. Each dataset is randomly split into training, validation, and test sets with a ratio of 8:1:1. The details of these datasets are summarized in Table 1.

Baselines

We compare SGDN with the SOTA methods, including CNNbased methods (DeepCoNN [Zheng et al., 2017], TransNet [Catherine and Cohen, 2017] and DRRNN [Xi et al., 2021]), attention-based methods (DAML [Liu et al., 2019] and DRRNN [Xi et al., 2021]), disentanglement-based method (DGCF [Wang et al., 2020]), and graph-based methods (RMG [Wu et al., 2019b], and RGCL [Shuai et al., 2022]).

Parameter Settings

The hyperparameters for the baseline models are tuned according to the original paper. It is notable that we reimplement DGCF by replacing the BPR loss [Rendle et al., 2012] with MSE loss to accommodate the rating prediction task. For SGDN, we use Adam to optimize the parameters with a learning rate of 0.01. The size of embeddings d for users/items and reviews is set as 64. We choose the number of message passing layers L from {1, 2, 3}, the number of latent factors from {2, 4, 8}, and the dropout ratio from {0.7, 0.8, 0.9}. The temperature hyperparameter τ is tuned from {0.2, 0.5, 1}. The hyperparameter λ is searched from {0.01, 0.05, 0.1, 0.5}.

Evaluation Metric

Following [Liu et al., 2019], we evaluate the performance by MSE. Each experiment is repeated five times and the average performance is reported for each dataset.

5.2 Performance Comparison

Overall Performance Comparison

The comparison results of all methods are presented in Table 2 and the following observations can be made:

• SGDN achieves the best results on every dataset tested and significantly outperforms the strongest baseline, RGCL, by an average of 2.5%. The improvements of SGDN relative to all other baselines can be attributed to: 1) By disentangling the graph from semantic and structural perspectives, SGDN is able to model user preferences based on multiple latent factors more accurately. 2) The intent-aware CL task can assist SGDN in disentangling the factors and alleviating the problem of the sparsity interactions.
• DGCF achieves comparable or superior performance in comparison to many CNN-based or attention-based baselines despite its ignoring review information, which demonstrates the efficacy of disentangling in review-based recommender systems. Meanwhile, SGDN outperforms DGCF by a large margin, which validates the efficacy of the graph

disentangling mechanism in SGDN and highlights the importance of review data in disentangling.

Performance Comparison in Alleviating Data Sparsity

To verify the robustness of SGDN against sparsity issue, we partition users into distinct groups according to their interaction numbers in the training set (e.g., 5-10). Then we report the MSE of DGL, SGDN in comparison to the SOTA models RGCL and RG (RGCL minus the CL tasks) for each group. Figure 3 demonstrates that, compared to RG, DGL is more robust to the sparsity issue, allowing for more effective use of review information to disentangle latent factors in user-item interactions. In addition, SGDN improves upon RGCL by selecting proper positive and negative samples based on user's interaction intents in the CL task. Consequently, SGDN achieves the highest performance across all groups, demonstrating the efficacy of our proposed CL task.

5.3 Ablation Studies

In this section, we conduct ablation research on the two modules in SGDN to comprehend their functions more deeply.

Impact of the DGL module. To validate the efficacy of DGL, we compare it with RG, the SOTA graph learning model. In addition to RG, we also compare DGL to its three variants: 1) Variant 1 calculates the coefficient sk,(l) i,j in a uniform manner, i.e., sk,(l) i,j = 1K . 2) Variant 2 calculates sk,(l) i,j based on the semantic information, i.e., sk,(l) i,j = seki,j . 3) Variant 3 calculates sk,(l) i,j based on the structural information, i.e., sk,(l) i,j = stk,(l) i,j . The results are shown on the top of Table 3. The key observations are as follows:

First, Variant 1 almost has no performance improvements compared to RG because it fails to model the intent distributions among interactions. Second, we observe a decrease in the performance of Variant 2 and 3 compared to DGL, demonstrating that integrating semantic and structural information allows for a comprehensive exploration of the distributions of latent factors in interactions. Third, DGL has a significant improvement over RG, validating the efficacy of its disentanglement in user's intents.

Impact of the ICL module. To validate the efficacy of the proposed CL task, we remove the positive and negative sampling strategy in the ICL module (named ICL-NP) and compare it with SGDN. The results are summarized at the bottom of Table 3. We observe that adding self-supervised task can alleviate the sparsity issue and improve the overall effect of DGL. Moreover, SGDN consistently outperforms DGL+ICLNP, highlighting the efficacy of learned intent distributions in identifying the positive and negative samples.

5.4 Hyperparameter Studies

In this section, we evaluate the effect of key hyperparameters (# of latent factors K, # of positive samples Kp, # of message passing layers L, and importance hyperparameter of CL task λ) in SGDN and show the evaluation results in Figure 4.

• The value of K is examined in {1, 2, 4, 8}. We find that SGDN performs the worst when K = 1, indicating that modeling the user characteristics as a whole is insufficient to capture user behavioral patterns. Increasing the factor number from 1 to 4 significantly enhances the model performance. However, excessive disentanglement leads to

small performance degradation, which might be attributed to too fine-grained factors and limited expressiveness.

• The value of Kp is tuned in the range from 1 to 100. Figure 4(b) show the results. With the increase of the positive examples, the performance of SGDN almost remains stable on the three datasets when Kp < 50. However, too large KP might lead to the quality drop due to misidentifying the interactions with dissimilar intents as positive samples.
• We investigate the impact of message passing layer number by setting L ∈ {1, 2, 3, 4} and present the results in Figure 4(c). Clearly, SGDN is able to distill useful higherorder information from multi-hop neighbours according to the significant improvements by increasing L from 1 to 2. While continuing the model depth, the improvements are not that obvious or even slightly degrade. This indicates that second-order connectivity might be sufficient to extract factor-relevant information.
• We also study the impact of λ ∈ {0.01, 0.05, 0.1, 0.5, 1} that weighs the contribution of CL task. As shown in Figure 4(d), the performances of SGDN first increase and then decrease on the three datasets, following a under- to overfitting pattern. Generally, SGDN is not very sensitive when λ is tuned in a reasonable range (e.g. 0.05-0.1).

5.5 Explainability Studies

To better interpret the latent semantics of the learned factor graphs, we explain the reasons behind user ratings by presenting the reviews of high-confidence interactions. In particular, we conduct experiments on Office with factor number K = 4 and layer number L = 2. For each factor k, we randomly select one review with score sk,L i,j > 0.5 for rating 1,

• 3, 5, respectively. This indicates that factor k dominates user i's rating on item j. We present the reviews and associated scores in Table 4 and have observations as follows:
• By jointly analyzing the reviews of the same latent factor, we find that, despite being written for different items and ratings, they all have inherent semantic connections. For example, the reviews of factor k1 reflect user ratings based on how well the product meets their needs, such as good for me and beneficial in my business. The reviews for factor k2 are all about the quality and durability of the products, such as quality seems to be OK and solidly made.
• By jointly analyzing the reviews across multiple factors, we find that, SGDN is capable of modeling the interactions frommultiple perspectives. In general, we character k1, k2, k3, and k4 as demand-supply match, quality, integrity and cost performance, respectively. This verifies our hypothesis that different uses' ratings on different items are driven by distinct latent factors.

6 Conclusion

This paper proposes a novel framework, SGDN, which focuses on exploring and disentangling the latent factors behind user-item interactions for better and more explainable review-based recommendation. Specifically, we design a disentangled graph learning module to factorize the user-item rating graph and learn disentangled user/item representations. Then an intent-aware contrastive learning task is designed to alleviate the sparsity issue and encourage disentanglement. Experiments on five benchmark datasets validate the superior performance and interpretability of SGDN.

Acknowledgments

This work was supported by NSF China (No. 42050105, 62020106005, 62061146002, 61960206002), Shanghai Pilot Program for Basic Research - Shanghai Jiao Tong University.