KDDC: Knowledge-Driven Disentangled Causal Metric Learning for Pre-Travel Out-of-Town Recommendation

1 Introduction

With the rapid growth of location-based social networks such as Foursquare and Yelp, Point-of-Interest (POI) recommendation [Wang et al., 2023a] has become increasingly prevalent in our daily lives. Compared to conventional regionspecific POI recommendation, out-of-town POI recommendation serves users who want to travel out of their hometowns. Most out-of-town POI recommender systems [Wang et al., 2017; Ding et al., 2019; Li and Gong, 2020; Sun et al., 2021; Xin et al., 2021] aim to solve issues such as data sparsity and interest drift when users have ascertained the travel locations. Recently, a new task (i.e. pre-travel out-of-town recommendation) [Xin et al., 2022] has been proposed for providing POIs to the users who plan to out-of-town travel yet have not decided on the destination region. To deal with user interest drift, the pre-travel recommendation requires modeling users' behavior preferences and inferring their travel intentions to out-of-town POIs spanning diverse regions. However, existing out-of-town recommendation methods still face two significant challenges, which may fail to provide satisfying results for pre-travel recommendation:

∗Corresponding author.

1. The sparsity of check-in data and diversity of POIs. Due to the sparsity of interactive data (whether hometown check-ins or out-of-town check-ins) and the diversity of candidate POIs, relying solely on the sequence of visited POIs and their spatial relationships for modeling user preference is insufficient. The semantic information of a POI, such as its category, type, rating, and other details, similarly plays a prominent role in shaping user preference [Yang et al., 2022; Zhang et al., 2023]. For instance, users are likely to consult the evaluations of a POI before making a visitation decision. Thus, it is wise to learn semantic knowledge of POIs.
2. ii) The causes of a check-in are easily bundled together. In terms of cause, user interest is undoubtedly a significant

driver for out-of-town checking-in, but users may also visit popular POIs owing to crowd behavior [Zheng et al., 2021; Zhao et al., 2023]. As shown in Figure 1, a user residing in Hangzhou, whose primary interest lies in gourmet food rather than history, also intends to visit the popular American Museum of Natural History. Therefore, observed out-of-town check-ins can be attributed to both interest and conformity. Despite [Xin et al., 2022] utilizes other travelers' behavior memory to take conformity into account, the crowd memory is generally similar for all users, which does not reflect conformity's personalization. Figure 1 also exhibits distinct conformity and interest distributions in a user's each travel intention. This demonstrates the necessity of disentangling the causes (conformity and interest) from diverse POIs for the effect (hometown and out-of-town check-ins).

In light of the above limitations and challenges, we propose a Knowledge-Driven Disentangled Causal metric learning framework (KDDC) for pre-travel out-of-town recommendation. Specifically, to learn the semantic features of POIs, we first construct an attribute knowledge graph encompassing all POIs and employ a segment-wise pre-training strategy to enhance information interaction between knowledge graph embedding. Then, given the relation heterogeneity in knowledge graph, we introduce a relation-aware knowledge aggregation mechanism to capture dependent contextual signals from both entities and relations. In this way, the users' hometown and out-of-town POI embeddings can be enhanced. By using designed disentangled causal metric learning that considers both interest and conformity, we improve the representation of users' behavioral preferences. Meanwhile, we develop a travel intent inference network (TIIN) to facilitate the inferring of travel intentions from the perspectives of both interest and conformity. Ultimately, we conduct recommendations within the detailed metric space, partitioned into various subspaces w.r.t. the users' hometowns using dynamic mapping.

The following are our summarized contributions:

• We propose a knowledge-driven approach, namely KDDC, to jointly consider the semantic knowledge of POIs and two different reasons why users check-in (i.e. conformity and interest) for pre-travel recommendation.
• To alleviate the sparsity in check-in data while exploring potential similarities among diverse POIs, we develop a segmented interaction pre-training method for enhancing knowledge semantic representation and utilize relation heterogeneity to aggregate knowledge for enriching POI embedding. Besides, we design a disentangled causal metric learning to improve users' behavior preferences modeling and travel intentions inference.
• To demonstrate the effectiveness of KDDC, we conduct diverse experiments on two real-world datasets showing that the proposed model consistently outperforms others.

2 Preliminary

Definition 1 (POI Attribute Knowledge Graph). The POI attribute knowledge graph is defined as Gk = {(v, r, e)}, which organizes external POI attributes with diverse types of entities and corresponding relationships. Specifically, each POI-relation-entity triplet (v, r, e) characterizes the semantic relatedness between the POI and entity v and e with the relation r, such as (The Great Wall, Located in, Beijing).

Definition 2 (User Out-of-town Travel Behavior). Given a user u, his/her out-of-town travel behavior is denoted by a five-tuple τ = (u,⃗ch,⃗co, ah, ao), indicating that the user u travels from his/her hometown ah to out-of-town region ao and leaves check-in records in both regions, which are represented by ⃗ch and ⃗co, respectively.

Formally, given a set of users U = {ui}|U| 1 , a set of POIs V = {vi}|V| 1 , a POI attribute knowledge graph Gk, a set of regions A = {ai}|A| 1 , and users' out-of-town travel behavior records O = {τi}|O| 1 , our studied problem is stated as follow. Problem Formulation. Given the historical out-of-town travel behavior records O left by U and the POI attribute knowledge graph Gk, learn a recommender F. Then, a new user u∗ /∈ U shows up with his/her hometown check-in ⃗ch observed in a∗h , F recommends a list of out-of-town POIs V ∗ ⊂ V which not in ah to u∗ .

3 Methodology

We present the overall framework of KDDC in Figure 2. This framework comprises three main stages: Knowledge Graph Segmented Pre-Training, POI Semantical Knowledge Aggregation, and Disentangled Causal Metric Learning for PreTravel Recommendation:

1) We perform fine-grained partitioning of multiple embeddings in a constructed attribute knowledge graph for pretraining to achieve sufficient information interaction. 2) To consolidate semantic knowledge across various relationships, a relation-aware knowledge aggregation layer is introduced to update POI embeddings. 3) We utilize a disentangled causal representation manner to attribute users' behavioral preferences to interest and conformity and hence infer the travel intentions of users by TIIN. Moreover, metric learning is adopted to assist model training for recommendation. Technical details are introduced in the following subsections.

3.1 Knowledge Graph Segmented Pre-Training

Knowledge graph pre-training aims to enhance models' understanding and inference capabilities by improving entity and relation embeddings. Inspired by [Liu et al., 2022], lowdimensional embeddings can be partitioned into several segments and the interactions among these segments foster the exchange of richer information. Thereby, we first performed a fine-grained embedding segmentation of POIs, relations and entities. The number of their segments is equal and the set of segmented embedding vectors is defined as:

where n is the number of segments, m ∈ {v, r, e} represents POI, relation and entity respectively. Then, in accordance with [Xu et al., 2020], we introduce a scoring function fS(·) to implement embedding chiasma interactions among distinct segments and evaluate the association strength among them:

where 〈·〉 means the dot product sum of vector elements. {x, y, z} represent segmented ordinal numbers and

controls its sign to support both symmetric and antisymmetric relations [Xu et al., 2020]. Simultaneously, we employ another variable zx,y to prevent over-embedding for computing overhead reduction:

As a result, the even-numbered segments can capture the symmetry of the relationship and the odd-numbered segments capture antisymmetry information. Formally, the segmentation-based pre-training loss LS is shown below:

where e′ is the negative sample generated by randomly replacing the entity e for the observed triplets (v, r, e).

3.2 POI Semantical Knowledge Aggregation

The semantics conveyed by different relationships are distinct. The embeddings of POIs that share similar semantics (e.g., POIs that belong to the same category) should be in close proximity to each other in the latent space. Given the success of graph attention mechanisms in [Wang et al., 2019a; Xia et al., 2021; Yang et al., 2022], we utilize the relation heterogeneity between POI and entities over the knowledge graph to update POI embeddings as illustrated in Figure 2b. In order to avoid the manual design of path generation on knowledge graphs, entity- and relation-dependent context is projected into specific representations by parameterized attention matrix. To this end, we construct our relation-aware message aggregation mechanism between the POI and its connected entities in GK. This mechanism is used to generate knowledge-aware POI embeddings, relying on the heterogeneous attentive aggregator described as follows:

where Nv is the neighboring entities of POI v under different types of relations re,v in GK, v ∈ Rd and e ∈ Rd represent the embedding of POI and entity, respectively. The assessed entity- and relation-specific attentive relevance during the knowledge aggregation process is denoted as φ(e, re,v, v), which encodes the distinct semantics of relationships between POI v and entity e. ‖ means the concatenation of two embeddings. W ∈ Rd×2d denotes the weight matrix that is tailored to the input POI and entity representations. σ is the activation function LeakyReLU for non-linear transformation. Through the aforementioned mechanism, the underlying distances between embeddings of diverse POIs infused with similar semantic knowledge will be brought closer together.

3.3 Disentangled Causal Metric Learning for Pre-Travel Recommendation

User behavior preference modeling. With the aim of strengthening recommender systems, several causal-related methods such as inverse propensity scoring (IPS) [Gruson et al., 2019], counterfactual inference [Wang et al., 2021a] and causal embedding [Zheng et al., 2021] have been proposed. In this work, we use causal embedding in the pre-travel recommendation because of the following advantages: 1) from a check-in generation perspective, causal embedding accurately models users' hometown and out-of-town preferences for different reasons; 2) causal modeling helps to improve model generalization capabilities. Following previous works [Zheng et al., 2021; Zhao et al., 2023], we focus on the two main causes: interest and conformity to model users' behavior preferences. One can see the process of disentangling each cause into causal embedding in Figure 2c. We first partition the users' hometown and out-of-town context embeddings ˜vh and ˜vo into corresponding interest and conformity embeddings with two distinct linear transformations:

where W{conf,int} ∈ Rd×d and b{conf,int} ∈ Rd are trainable parameters. In addition, we partition the embedding space by users' hometowns with a dynamic mapping mechanism for making the following metric more distinguishable [Xin et al., 2022]. In particular, given a user u, his/her related check-in embeddings are encoded as

where Wd ∈ Rd×d and bd ∈ Rd are trainable parameters. R ∈ Rd×|A| is a learnable partition matrix and Rah represents the column vector in R w.r.t. region ah. The user ultimate hometown and out-of-town check-in embeddings ¯vh and ¯vo , considering both interest and conformity are separately generated through the concatenation operation. Since the user's hometown behavior is relatively long-term and could reveal the user's inherent preference, we model his/her behavior preference based on his/her hometown check-in embeddings:

where AGG(·) denotes an arbitrary aggregation operation, such as max pooling, average pooling or attention pooling. User travel intention inference. As a user's hometown behavior is different from that outside the city, comprehending travel intention plays a crucial role in out-of-town recommendation. Toward that, we develop a travel intention inference network (TIIN) to infer users' intentions from different reasons. In particular, given a user's mapped check-in embeddings in Eq. (8), we first employ the aggregator to summarize his/her hometown and out-of-town preferences under two causes:

Then, two nonlinear transformation layers: conformity inference layer and interest inference layer are adopted to infer the user's conformity and interest travel intentions:

where WC ∈ Rd×d , WI ∈ Rd×d , bC ∈ Rd and bI ∈ Rd are trainable parameters. σ denotes the activation function SiLU. Metric Learning. Inspired by the great performance of metric learning in recommendation [Hsieh et al., 2017; Zhou et al., 2019; Xin et al., 2022], we apply it in disentangled causal intention inference and implement a metric-based outof-town recommender. Aiming at the inference task, we first devise two supervised popularity-based causal metric losses to bridge the user's inferred embedding and factual out-oftown embedding from two causes as follows,

where Ppop represents the normalized popularity of the interacted POIs, which is defined as the ratio of the number of the POIs the user interacted with vs. the max POI interacted number [Zhao et al., 2023]. For the highly popular POIs, the popularity weight exp (-Ppop) is close to exp (-1) rather than 0, which allows the task to learn interest preferences to a certain extent. Oppositely, 1-exp (-Ppop) ensures that interactions with more popular POIs are more likely to be attributed to conformity. The total loss of the task is expressed as LT .

Then we integrate the user's inferred intention Io into his/her preference and employ the triangle inequality in the embedding space to calculate the distance between such fortified preference and a POI vi as follows,

Given a positive out-of-town POI vi ∈ ⃗co visited by u, we sample a negative POI vj /∈ ⃗co. From pairwise comparisons, the main disentangled causal metric loss under the Bayesian personalized ranking (BPR) criterion for pre-travel out-oftown recommendation is formally defined as,

where δ is the margin size and [·, 0]+ = max(·, 0) represents the standard hinge loss.

Joint Optimization and Recommendation. According to Eqs. (12) and (14), the integrative optimization loss of our KDDC is:

where λ1, λ2 are the hyper-parameters to balance the effects of the two losses in KDDC.

Here, we formally give the process of the pre-travel outof-town recommendation. Given a new coming user u∗ , we first gain his/her hometown behavior preference embedding uh∗ and inferred travel intention embedding Io∗ through Eqs. (7), (8), (9) and (11). Then we select top-k out-of-town POIs whose mapped embeddings ¯vo i are closest to the concatenation of uh∗ and Io∗ as the recommendation list for u∗ , where we formulate the distance as below:

4 Experiments

4.1 Experimental Settings

Datasets. We chose two nationwide travel behavior datasets, Foursquare 1 and Yelp 2 , to evaluate our framework. The statistical information of our experimented datasets with different check-in records and knowledge graph characteristics is given in Table 1. For check-in records, following [Xin et al., 2022], we picked out the users who left check-ins in both hometown and out-of-town regions from these two datasets and the check-in sequence of each user was reformed as the homologous out-of-town travel record τ = (u,⃗ch,⃗co, ah, ao) (Definition 2). Besides, for the quality of two datasets, the POIs visited less than 2 times were filtered out and a user was dropped if his/her travel behavior τ = (u,⃗ch,⃗co, ah, ao) fails to fulfill the following conditions: 1) ⃗ch ≥ 4; 2) ⃗co ≥ 2; 3) the frequency of (ah, ao) ≥ 10. The two datasets are randomly partitioned based on users into three sets for training, validation, and testing following the proportions: 80%, 10% and 10%. To ensure the fairness of the assessments, all of the users on the two datasets are anonymization. For the knowledge graphs, we generate entity-dependent relations with various types of entities (e.g., category, star, and location).

Baseline. We compare KDDC with: 1) Three general recommendation methods, BPR [Rendle et al., 2012], TEMN [Zhou et al., 2019] and STAN [Luo et al., 2021]; 2) A knowledge graph-enhance method KGCL [Yang et al., 2022]; 3) Two disentangled recommendation methods, DCCL [Zhao et al., 2023] and DisenPOI [Qin et al., 2023]; 4) Two out-of-town recommendation methods, TRAINOR [Xin et al., 2021] and CAPTOR [Xin et al., 2022]. Note that since there is no explicit user-item interaction graph in the task of pre-travel recommendation, for fairness, we model implicit user preference through the same sequence aggregation operation as Eq. (9) for some methods like DCCL and KGCL.

Evaluation Metrics. Three widely used metrics for recommendation are adopted to appraise method performances, i.e., Hit Rate, Precision, and NDCG at a cutoff k, abbreviated as HR@K, Pre@K, and NDCG@K respectively. Following previous works [Xin et al., 2021; Xin et al., 2022], a fixed number of negative POIs, randomly sampled from the candidate set, are taken together with the positive out-of-town POIs as final candidates for each user. In the experiments, the number of negative POIs is set to 100.

1 https://sites.google.com/site/yangdingqi/home/foursquaredataset

2 https://www.yelp.com.tw/dataset

Implementation details. The baselines and their parameter settings were implemented based on the original papers and official public code. In addition, the parameters of the KDDC were sampled as follows: the number of dimensions of all latent representations was set to 128. In the Knowledge Graph Segmented Pre-Training, n was 8 for Foursquare and 4 was for Yelp. In the optimization stage, λ1 and λ2 were set as 1, the optimizer was chosen as Adam with an initial learning rate of 0.001 and an L2 regularization with a weight of 10-5 . We implemented our KDDC and experimented with Pytorch. Note that the evaluation was conducted three times with varying initial parameters, and the reported results represent the average. The code for the implementation of KDDC is available for reproducibility 3 .

4.2 Performance Comparison

Table 2 summarizes the performance of KDDC and baselines on the two datasets. We carry out the following observations: 1) Compared with all baselines, KDDC achieves optimal performance in terms of all metrics across two datasets. This illustrates that our method considering POI semantic knowledge and analyzing two main causes of user interaction is effective and generalizable for pre-travel recommendation. In particular, the improvements in contrast to the corresponding runner-up baselines, 13.53% ∼ 19.11% in HR@K, 11.33% ∼ 16.57% in Pre@K and 15.36% ∼ 16.18% in NDCG@K on Foursquare, and 5.56% ∼ 12.88% in HR@K, 6.71% ∼ 10.41% in Pre@K and 3.91% ∼ 9.18% in NDCG@K on Yelp for K = 5, 10, and 15; 2) Among these baselines, the two methods including DCCL and DisenPOI achieve relatively satisfactory results in most cases, which suggests the dominance of disentangling on recommendation. Meanwhile, KGCL and CAPTOR also produce sub-optimal performances in some cases, which proves the necessity of paying attention to semantic knowledge and spatial correlation. The spatial correlation will be discussed in Subsection 4.5.

4.3 Ablation Study

To evaluate the effectiveness of each component of KDDC on two datasets w.r.t all metrics, we compare it with its variants, including three large modules: 1) -PT: KDDC removes Knowledge Graph Segmented Pre-Training; 2) -KA: KDDC without POI Semantical Knowledge Aggregation; 3) -DC: KDDC discards Disentangled Causal Metric Learning, and two small modules: 4) -DM: KDDC removes the dynamic mapping mechanism and performs recommendation in a universal metric space; 5) -II: KDDC removes the travel intention inference network (TIIN). The histogram in Figure 3 illustrates the comparative results. Evidently, KDDC demonstrates superior performance compared to -KA, -DC, and - DM across all metrics on both datasets, which proves the critical roles of the aggregation of semantic knowledge and the application of metric learning within separated mapping spaces in disentangled causal embedding training. Besides, KDDC consistently outperforms -PT on Yelp but has a minor decrease on Foursquare w.r.t HR@15 and Pre@15, while KDDC persistently beats -II on Foursquare and achieves similar results on Yelp w.r.t all metrics. This is probably related to the size of the knowledge graph of the two datasets. When the scale is relatively small, pre-training has negligible impact on long sequence recommendation. However, as the scale increases, modeling users' out-of-town preferences can be achieved solely based on the enhanced user hometown preferences derived from a substantial amount of knowledge. But still, the segmented pre-training and travel intention inference are beneficial to the recommendation when there is no sufficient data.

3 https://github.com/Yinghui-Liu/KDDC

4.4 Parameter Sensitivity

We examine the sensitivity results across all metrics concerning the number of embedding segments, denoted as n in Eq. (1). The search for an optimal value is conducted within the range of {2, 4, 8, 16}. As displayed in Figure 4, the perfor- HR@5 HR@10 HR@15

Pre@5 Pre@10 Pre@15

Figure 5: The evaluations of w and w/o SA-CRF on Foursquare.

mance is the best when n equals 8 on Foursquare and when n is 4 or 8 on Yelp, but the result in the case of n = 16 is not as good as n = 2. It may be due to the overfitting of the model with excess interaction information. Besides, to justify the validity of our embedding method separately, we compare it with other classical methods including TransE and TransR, and report the result under K = 10 and 15 in Table 3. One can see its performance improvement.

4.5 Discussion on Spatial Correlation

In most recent POI recommender systems, the spatial correlation among POIs is undoubtedly a research hotspot. For instance, CAPTOR [Xin et al., 2022] captured the spatial affinity by a structured spatial affined conditional random field (SA-CRF) and DisenPOI specially designed a geographical graph to explore spatial dependence. For that purpose, we also try to consider the spatial correlation of POIs. Specifically, we adopt the SA-CRF same as [Xin et al., 2022] in our framework, and it is directly inserted after Figure 2b and before Figure 2c. The comparison with the original framework w.r.t all metrics is reported in Figure 5, which presents a certain degree of performance degradation. We argue that incorporating spatial relation learning directly after knowledge aggregation will lead to contamination of the learned semantic features. There may be a demand to disentangle the two types of learning and we will consider this in our future work.

5 RelatedWork

Out-of-Town Recommendation. Out-of-town recommendation attempts to provide a list of new POIs for out-of-town users to visit. It is more intractable than the general POI recommendation due to problems like cold start and interest drift. [Ference et al., 2013] was the first work to solve the problem with social influence. Next, some works based on Latent Dirichlet Allocation (LDA) [Wang et al., 2017; Yin et al., 2016] have been proposed to model the interest drift by user preferences, spatiotemporal effects and POI contents. In addition, there have been deep-learning methods for addressing this problem. [Li and Gong, 2020] learned the user preference from both hometown and out-of-town cities with transfer learning. [Xin et al., 2021] used the neural topic model to analyze the user's out-of-town travel intentions. [Xin et al., 2022] settled the more intractable pre-travel recommendation by capturing the crowd behavior memory. However, the above methods still face the challenge of data scarcity, especially in the pre-travel recommendation scenarios. In this paper, we are committed to alleviating this problem through the proposed knowledge-driven method.

Knowledge Graph-enhanced Recommendation. Existing Knowledge Graph (KG)-enhanced methods in the general recommendation task can be broadly grouped into three categories: embedding-based, path-based and graph neural network (GNN)-based. Embedding-based methods [Zhang et al., 2016; Wang et al., 2018] adopted the transition-based entity embedding schemes (e.g., TransE, TransR) to generate prior item embeddings. Path-based methods [Wang et al., 2019b; Xia et al., 2021] aim to build the user-item meta path to improve capturing high-order KG connectivity but they highly depend on the design of meta-paths needing domain knowledge and human efforts. Recent research directions have focused on the usage of GNNs to recursively perform information propagation between multi-hop nodes and incorporate long-range relational structures, such as KGIN [Wang et al., 2021b] and KHGT [Xia et al., 2021]. KGCL [Yang et al., 2022] employed a joint self-supervised learning paradigm to alleviate the data noise issue but will cause extra time consumption. Accordingly, we combine a devised knowledge graph embedding segmented pre-training with a lightweight knowledge graph relational aggregation mechanism for pretravel recommendation.

Disentangled Representation Learning for Recommendation. Disentangled representation learning aims to learn independent explanatory representations from disparate underlying factors behind data. For the temporal correlation, CLSR [Zheng et al., 2022] tried to differentiate users' long-term and short-term interests from their clicking sequences. There also have been some GNN-based models to learn disentangled representation from various graph structures, including IDCL [Wang et al., 2023b] (user-item graph), DisenHAN [Wang et al., 2020] (heterogeneous graph) and DisenPOI [Zhao et al., 2023] (multitype graph). Furthermore, for causality, DICE [Zheng et al., 2021] and DCCL [Zhao et al., 2023] disentangled conformity and interest influences by learning causal embedding. Analogously, we focus on such two causes to construct users' behavior preferences and infer their travel intentions with metric learning for pre-travel recommendation.

6 Conclusion

In this paper, our proposed KDDC framework performs pretravel out-of-town recommendation by taking into account the semantic information of POI and the two causes of user interactions: conformity and interest. Three phases including Knowledge Graph Segmented Pre-Training, POI Semantical Knowledge Aggregation, and Disentangled Causal Metric Learning for Pre-Travel Recommendation make up our framework. The first phase uses a segmented pre-training strategy to improve the representations of knowledge graph elements. Then, the second phase aggregates semantics from different relationships to further ameliorate POI representation. After that, the third phase applies metric learning in disentangled causal representations for the more accurate construction of a user's preference and intention. Extensive experiments on two real-world datasets have demonstrated the superiority of KDDC over state-of-the-art methods for pretravel recommendation.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62073295 and Grant 62072409.