RecipeRec: A Heterogeneous Graph Learning Model for Recipe Recommendation

1 Introduction

Large-scale food data offers rich knowledge about the food landscape and can help address many important issues of today's society. Recipe websites are a popular source of food inspiration and enable users to rate and comment on recipes as well as receive recipe recommendation [Tian et al., 2022]. In particular, www.food.com, one of the largest recipe-sharing websites in the world, consists of over half a million recipes - reflecting the popularity of online recipe portals and the great demand for recipe recommendation services. However, digging into this overwhelming amount of online recipe resources to find a satisfying recipe is very difficult [Britto et al., 2020]. Thus, recipe recommendation systems are proposed to help users identify recipes that align with users' preferences and needs among numerous online resources [Khan et al., 2019].

Existing recipe recommendation approaches typically rely on computing similarities between recipes [Chen et al., 2020]. A few approaches attempt to take user information into account [Khan et al., 2019; Gao et al., 2020], but they only define similar users based on the overlapping rated recipes between users. This overlooks the higher-order collaborative signal (e.g., relational structure information) [Wang et al., 2019a] present among ingredients, recipes, and users. However, individual preference for foods is complex. A user may decide to try a new recipe, for example, because of its ingredients, its cuisine type, or perhaps based on a friend's recommendation. Thus, a comprehensive recipe recommendation system should consider all of these factors. In particular, it is important to encode the higher-order collaborative signal into recipe recommendation and model the relationship among ingredients, recipes and users.

Therefore, we propose the problem of recipe recommendation with graphs, which leverages the advances of graph modeling to address recipe recommendation. With the higher-order collaborative signal encoded by a graph learning model, the recommendation system can successfully capture the hidden collaborative filtering effect. Due to the complex relationship among ingredients, recipes and users (e.g., complementary ingredients, similar recipes), solving recipe recommendation with graphs requires an extensive and comprehensive recipe graph. Hence, it should not be surprising that the lack of research on this topic could be due to the lack of such a dataset. With the aim of making better recipe recommendation, we create URI-Graph, a new and large-scale user-recipe-ingredient graph for recipe recommendation. As shown in Fig.1, we extract ingredients, instructions, and user interactions of each recipe to build URI-Graph, which consists of three types of nodes and four types of relations.

Based on the created URI-Graph, we propose RecipeRec, a novel heterogeneous graph learning model for recipe recommendation. In particular, we first design a heterogeneous GNN to capture both recipe content and the relational structure information in URI-Graph. To fully encode this information, we propose an adaptive nodelevel attention module to distinguish the subtle difference of neighboring nodes under a specific relation. We also introduce a relation-level attention module to select the most meaningful relations surrounding a node and automatically fuse them with proper weights. Then, we apply a set transformer [Lee et al., 2019] to encode the ingredient information and model the pairwise interactions and higher-order interactions among ingredients. Next, we introduce a graph contrastive augmentation strategy to extract informative graph knowledge in a self-supervised manner. Finally, we design a novel combined objective function of recommendation and graph contrastive learning to optimize the model. To summarize, our major contributions in this paper are as follows:

• To the best of our knowledge, this is the first attempt to study recipe recommendation with graphs. We create URI-Graph, a new and large-scale recipe graph dataset, that can facilitate recipe recommendation research and graph-based food studies.
• We propose RecipeRec, a novel heterogeneous graph learning model for recipe recommendation. RecipeRec can capture recipe content and relational structure information through various modules, including GNN with hierarchical attention, ingredient set transformer and graph contrastive augmentation. We also design a novel objective function to optimize the model.
• We conduct extensive experiments to evaluate the performance of our model. The results demonstrate the superiority of RecipeRec by comparing it with state-ofthe-art baselines for recipe recommendation.

2 Preliminary

In this section, we describe the concept of user-recipeingredient graph, and formally define the problem of recipe recommendation with graphs. We further introduce the URIGraph data that we create and use in this paper.

Definition 2.1. User-Recipe-Ingredient Graph. A userrecipe-ingredient graph is defined as a heterogeneous graph G = (V,E,C) with three types of nodes V (i.e., user, recipe, ingredient) and four types of edges E (i.e., ingredientingredient, recipe-ingredient, recipe-recipe, and user-recipe relations). In addition, nodes are associated with contents C, (e.g., instructions, nutrients, or ratings).

Problem 1. Recipe Recommendation with Graphs. Given a user-recipe-ingredient graph G = (V,E,C), the task is to design a machine learning model FΘ with parameters Θ to recommend recipes to users. In particular, the task is to return a ranked list of recipes for each user, such that its top ranked recipes are true favorite recipes of users. The recommender is supposed to take advantage of the associated recipe contents C and relational structure information in G.

URI-Graph Data. To solve the problem, we first build URIGraph, as shown in Fig.1. We start by collecting recipes from Reciptor [Li and Zaki, 2020] and crawl the user ratings for each recipe from food.com as the user-recipe interactions. We then match each ingredient to the USDA nutritional database [USDA, 2019] to get the nutritional information. Next, we build URI-Graph by representing the users, recipes and ingredients as nodes, and constructing four types of relations to connect them. Specifically, we first connect each recipe and its ingredients with an edge, denoting the recipeingredient relation, while the weight of each ingredient is used as the edge weight. We then connect recipe nodes by the similarity determined by FoodKG [Haussmann et al., 2019] and the score is used as the edge weight, as shown in Reciptor [Li and Zaki, 2020]. We further connect ingredient nodes by the co-occurring probabilities using Normalized Pointwise Mutual Information [Tian et al., 2021]. We also connect users and recipes based on the interactions, with the rating scores as the edge weight. In the end, we build the graph with 7,958 user nodes, 68,794 recipe nodes and 8,847 ingredient nodes. Moreover, the graph contains four types of edges, including 135,353 user-recipe, 647,146 recipe-recipe, 463,485 recipeingredient and 146,188 ingredient-ingredient relations.

3 RecipeRec

To address recipe recommendation over the graph, we develop RecipeRec, a heterogeneous graph learning model for recipe recommendation. As illustrated in Fig.2, RecipeRec contains a heterogeneous GNN with hierarchical attention and an ingredient set transformer, with a graph contrastive augmentation strategy and a joint objective function.

3.1 GNNwith Hierarchical Attention

We first develop a heterogeneous GNN with hierarchical attention to obtain effective user and recipe embeddings. Specifically, for a node vi ∈ G, we use a type-specific input projection to transform the input feature, and leverage the adaptive node-level attention module to obtain a relationspecific embedding hi,r for each relation r that connects to vi. Then, we propose the relation-level attention module to fuse all relation-specific embeddings and obtain the final GNNbased node embedding hGi .

Type-specific Input Projection. For the input features of each node type, we use nutrient vectors to represent the ingredients (xing), the average of the pretrained skipinstruction embeddings [Marin et al., 2019] to represent the recipes (xins) and random initialized vectors to represent the users (xuser). Due to the heterogeneity of the associated information, given a node vi of type ϕi, we introduce a typespecific input projection Wϕi ∈ Rdϕi ×d to project the input features into a shared embedding space:

where xi is the input feature of vi with dimension dϕi , and hi is the projected feature of vi with dimension d.

Adaptive Node-level Attention. To compute the relationspecific embedding hl+1 i,r for vi in layer l+1 of GNN, we first apply a shared weight matrix W l r ∈ Rdl×dl+1 to transform the input features from layer l, where dl and dl+1 are the dimensions of embeddings in layer l, l + 1 respectively:

where zl i ∈ Rdl+1 is the intermediary embedding of vi. We then calculate the unnormalized attention score elij between vi and vj, and normalize the elij using softmax function:

where ∥ is concatenation, W l ij ∈ R2dl+1×dl+1 is a weight vector, Ni,r denotes the index set of neighboring nodes that connect to vi through relation r and αlij is the attention weight between vi and vj. We use αlij as coefficients to linearly combine the neighboring nodes features. We further make it adaptive to recommendation task by incorporating the interaction between vi and vj so that the interaction between users and recipes can be formulated attentively [Wang et al., 2019a]. We also extend it to multi-head attention to make the training process stable. The process is formulated as follows:

where hl+1 i,r is the encoded embedding of layer l + 1 for vi through relation r, ⊙ denotes the element-wise product, W lh ∈ Rdl×dl+1 andWa ∈ RMdm×dl+1 are trainable weights, M is the number of attention heads, and dm is the dimension of attention heads which satisfies dm = dl+1/M.

Relation-level Attention. We further introduce the relationlevel attention to learn the importance of each relation and fuse all relation-specific node embeddings. In particular, we first apply a shared non-linear weight matrix WR ∈ Rdl+1×dl+1 to transform the relation-specific node embeddings. Then we use a vector q ∈ Rdl+1 and calculate the similarities and average it for all node embeddings of relation r to obtain the importance score wi,r for node vi:

where Vr denotes the set of nodes that are correlated with relation r and b ∈ Rdl+1 is the bias vector. We then normalize the wi,r to get the final relation-level attention weight βi,r:

where Ri indicates the associated relations of vi and βi,r can be explained as the contribution of relation r to vi. After that, we fuse the relation-specific node embeddings hl+1 i,r with βi,r to obtain the node embedding hl+1 i :

We denote the embedding obtained in the last propagation layer as the final GNN-based node embedding hGi such that hGi = hLi , where L is the number of layers.

3.2 Ingredient Set Transformer

To better encode the ingredients in a recipe, we propose an ingredient set transformer to incorporate the ingredient interactions. Specifically, given a recipe with n ingredients, we have a matrix Q ∈ Rn×d of n query vectors with dimension d. We define an attention function Att maps Q to outputs using p key-value pairs with K ∈ Rp×d , V ∈ Rp×d :

where ω is the scaled activation function (we use softmax(·/√d) in our model) and Att(Q,K, V ) is the weighted sum of V . We further incorporate the multihead attention by projectingQ,K, V into different subspaces. Considering two ingredient sets X1,X2 ∈ Rn×d , the multihead attention for X1 on X2 is defined as:

where WQ,WK,WV ∈ Rd×do are the trainable weights and do is the output dimension. Then we feed this self-attention into a Multi-head Attention Block (MAB) and a Set Attention Block (SAB) [Lee et al., 2019] to get the transformer-based recipe embeddings hTF i of recipe vi:

where FFN is a feed-forward neural network and Wm ∈ Rd×do is a trainable weight. We then combine the generated hT F i with the GNN-based learned embedding hGi to obtain the final embedding for each recipe node, denote as hi = WO · (hGi + hTF i ), where WO ∈ Rdo×do is a weight for output projection. For each user node, we have hi = hGi .

3.3 Objective Function for Recommendation

We employ a ranking-based objective function to train the model. In particular, the objective involves comparing the preference scores between nodes connected by a user-recipe relation against the scores between an arbitrary pair of user and recipe nodes. For example, given a connected user node vi and a recipe node vj, we encourage the score si,j between vi and vj to be higher than the score si,j ′ between vi and a random negative recipe node v′j . Accordingly, we use a predictor (e.g., inner product) to calculate the score and formulate the recommendation loss Lrec as follows:

where hi, hj, h′j are the embeddings of nodes vi, vj, v′j respectively, U is the set of users, and Ni indicates the recipe neighbors of vi.

3.4 Graph Contrastive Augmentation

Since the above objective function only considers explicit and supervised information in the URI-Graph, we further introduce a graph contrastive augmentation strategy to capture graph knowledge in a self-supervised manner. In particular, the input graph G will be augmented with two correlated views ˆ G ′ , ˆ G ′′ , where ˆ G ′ ∼ q′ (·|G), ˆ G ′′ ∼ q′′ (·|G), and q′ (·|G), q′′ (·|G) are two augmentation functions composed of node dropout and edge dropout [Wang et al., 2019a]. We apply these augmentations each training epoch and define a contrastive loss to maximize the agreement between different views of the same node, compared to that of other nodes. Specifically, we treat the views of the same node as positive pairs (i.e., {(h′i , h′′ i )|i ∈ YT}) and those of different nodes that appear in the same batch as negative pairs (i.e., {(h′i , h′′ j )|i ∈ YT , j ∈ BT , i ̸= j, ϕi = ϕj}). Here YT is the set of users and recipes that appear in the training set, and BT ⊆ YT is a random batch of users and recipes in the training set. We use InfoNCE [Gutmann and Hyvärinen, 2010] to model the graph contrastive learning loss Lcon:

where τ is the temperature. The final objective function L is defined as the weighted combination of Lrec and Lcon:

where λ is a trade-off weight for balancing two losses.

4 Experiments

In this section, we conduct extensive experiments to compare the performances of different models. We also show ablation studies, parameter sensitivity, case studies and embedding visualization to demonstrate the superiority of RecipeRec.

4.1 Experimental Setup

We employ the classic leave-one-out method to evaluate the model performance [He et al., 2017]. During the testing, we randomly sample 100 negative recipes for each user and evaluate the performance using Precision (Pre), Hit Rate (HR), Normalized Discounted Cumulative Gain (NDCG), and Mean Average Precision (MAP). We report the performance under top@K, where K ranges from 1 to 10.

4.2 Baseline Methods

We compare our model with 14 baselines, including traditional recommendation approaches BPR [Rendle et al., 2009] and NeuMF [He et al., 2017], recipe representation learning and recommendation methods Nutrition [Teng et al., 2012], PopIngre [Teng et al., 2012], IngreNet [Teng et al., 2012] and Reciptor [Li and Zaki, 2020], homogeneous GNNs GraphSAGE [Hamilton et al., 2017] and GAT [Veliˇckovi´c et al., 2018], heterogeneous GNNs RGCN

[Schlichtkrull et al., 2018], mp2vec [Dong et al., 2017], HAN [Wang et al., 2019b] and HGT [Hu et al., 2020], and graph-based recommendation methods PinSage [Ying et al., 2018] and NGCF [Wang et al., 2019a].

4.3 Implementation Details

For the proposed RecipeRec, we set the learning rate to 0.005, the number of attention heads to 4, the hidden size to 128, the temperature τ to 0.07, λ to 0.1, node and edge dropout ratio to 0.1, batch size to 1024 and the training epochs to 100.

4.4 Performance Comparison

We report the performances of all models in Figure 3. As shown in the figure, RecipeRec outperforms all the baselines for all metrics over different Ks. Specifically, traditional recommendation approaches (i.e., BPR, NeuMF) perform poorly because they neither consider the higher-order collaborative signal nor the associated ingredients. Recipe recommendation approaches (e.g., PopIngre, IngreNet) obtain decent performance after incorporating the relational information, while Reciptor performs poorly since it is designed to learn recipe representations only, which ignores the user information. Homogeneous GNNs (i.e., GraphSage, GAT) behave poorly because of the neglect of node type information. Heterogeneous GNNs (e.g., HAN, HGT) and graph-based recommendation approaches (i.e., PinSage, NGCF) achieve satisfactory results, but fail to encode the influence of different nodes and relations as well as the complex interactions among nodes. Finally, RecipeRec achieves the best performance compared to all the baselines, demonstrating the effectiveness of the proposed model.

4.5 Ablation Study

RecipeRec is a joint learning framework composed of several neural network modules and optimized by a joint loss. We further conduct ablation studies to evaluate the performance of different model variants including: (a) RecipeRec-NA that only uses adaptive node-level attention and recommendation loss; (b) RecipeRec-RA that utilizes adaptive node-level attention, relation-level attention and recommendation loss; (c) RecipeRec-IT that combines all neural network modules but only uses recommendation loss. We report the performances of these model variants under different Ks in Figure 4. From the figure, we can find that the performance increases when we incorporate more modules and optimize the model with both losses. Among these variants, the full model RecipeRec achieves the best result in all cases. This demonstrates the effectiveness of adaptive node-level attention, relation-level attention, ingredient set transformer and graph contrastive augmentation in our model.

4.6 Parameter Sensitivity

We perform parameters sensitivity analyses, including the impact of dropout ratio, trade-off weight, and score predictor. Impact of Dropout Ratio in Contrastive Learning. We conduct experiments to compare the influence of node dropout and edge dropout in Table 1. We can find that in most cases, the model performs the best when the ratio is 0.1. In general, the performance decreases when further increasing the dropout ratio, which indicates the suitable node/edge dropout ratio will lead to the best model performance.

Impact of Trade-off Weight in Joint Loss. To investigate the influence of graph contrastive learning, we study the impact of trade-off weight λ in Table 2. We can find that the model achieves the highest performance when λ is 0.1. Further increment of the value will decrease the performance. This demonstrates the incorporation of suitable graph contrastive augmentation can lead to the best recommendation results.

Impact of Score Predictor. To show the effectiveness of the score predictor used in our model, we examine the effect of other predictors (i.e., cosine similarity, multilayer perception)

in Fig.5. From this figure, we conclude that inner product performs the best since it considers both the angle difference and the magnitude between two embeddings.

4.7 Case Study: Embedding Visualization

To show a more intuitive understanding and comparison, we randomly select 8 user-recipe interaction pairs and visualize their embeddings using t-SNE. As shown in Fig.6, NGCF can roughly separate the users and recipes into two clusters. There are gaps within each cluster and the lines between clusters are disordered. Ideally, there should be a clear mapping between each user-recipe pair (similar to the 'kingman=queen-woman' relationship in Mikolov et al. (2013)) and the lines should be parallel to each other. However, RecipeRec can easily divide the users and recipes into two condense clusters and obtain parallel lines for most pairs, which further demonstrates the superiority of our model.

5 RelatedWork

This work is closely related to the studies of recipe recommendation and graph neural networks (GNNs).

Recipe Recommendation. Existing recipe recommendation approaches are mostly content-based [Chen et al., 2020] or collaborative filtering-based [Khan et al., 2019] or hybrid [Gao et al., 2020]. However, none of these approaches consider the higher-order collaborative signal such as relations. In our work, we propose a novel heterogeneous graph learning model to encode both recipe content and relational structure information. Additionally, existing recipe recommendation systems usually focus on the rating prediction task, which minimizes the errors between predictions and real ratings. This is concerned with only the observed recipes with explicit ratings, while our work uses a pairwise ranking strategy to examine the recommendation, which accounts for all the recipes in the collection.

Graph Neural Networks. Many graph neural networks [Veliˇckovi´c et al., 2018; Wang et al., 2019b; Zhang et al., 2019] were proposed to encode the graph-structure data. They take advantage of content information and relation information in graphs to learn node embeddings for various downstream tasks such as recommendation. Recently, several graph-based models have been proposed for recommendation [Zhang et al., 2017; Ying et al., 2018; Wang et al., 2019a]. However, most of them either fail to consider the collaborative signal. In this work, we propose a novel heterogeneous GNN to consider both collaborative signal and recipe information. In addition, contrastive learning has shown great performance in pre-training GNN [Qiu et al., 2020]. We thus introduce a graph contrastive augmentation strategy to capture informative graph knowledge and improve the model.

6 Conclusion

In this paper, we propose and formalize the problem of recipe recommendation with graphs. To solve this problem, we create URI-Graph, a new and large-scale recipe graph data to facilitate graph-based food studies. Furthermore, we develop RecipeRec, a novel heterogeneous graph learning model for recipe recommendation. RecipeRec is able to capture both recipe content and the relational structure information through a graph neural network and a transformer. A novel combined objective function of recommendation loss and graph contrastive learning loss is employed to optimize the model. Extensive experiments demonstrate that RecipeRec outperforms state-of-the-art baselines.

Acknowledgements

This work is supported by the Agriculture and Food Research Initiative grant no. 2021-67022-33447/project accession no.1024822 from the USDA National Institute of Food and Agriculture.