Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks

1 Introduction

Large-scale food data provides a wealth of information about food and can assist in resolving many critical societal issues [Min et al., 2019]. Recipe representation learning, in particular, embeds recipes in a latent space, allowing for the extraction of hidden information from massive food datasets and facilitating various application tasks that rely on a robust food space representation, such as culinary habits analysis [Min et al., 2018], recipe healthiness estimation [Rokicki et al., 2018], recipe recommendation [Tian et al., 2022b], and recipe classification [Li and Zaki, 2020].

Existing approaches to recipe representation learning rely entirely on textual content [Li and Zaki, 2020]. They take advantage of the recipe ingredients and instructions but overlook the relationship between recipes and food items. Besides leveraging the textual information, rn2vec [Tian et al., 2021] proposes a recipe network embedding model to capture the relational information. However, rn2vec ignores the large number of images associated with each recipe and thus results in suboptimal performance. While some studies consider recipe images, they typically focus on image-based applications such as recognizing and retrieving recipes from images [Pandey et al., 2017; Chen et al., 2018]. These works attempt to align the images and the text in a shared embedding space for specific downstream tasks, but ignore the unique information contained in each modality.

* Corresponding authors

In this paper, we propose the problem of multi-modal recipe representation learning, which leverages different modalities such as images, text, and relations to learn recipe embeddings. To solve this problem, we first create and release a new recipe graph, Large-RG, which contains over half a million nodes and is the largest recipe graph to date. As shown in Fig. 1, we extract visual, textual, and relational information of each recipe to build Large-RG, which consists of three types of nodes and four types of edges. We then propose Recipe2Vec, a novel graph neural network (GNN) based model for multi-modal recipe representation learning. Specifically, we first encode the node attributes using various pre-trained neural networks, including a twostage LSTM [Marin et al., 2019] for instructions and a ResNet [He et al., 2016] for images. Next, we introduce a multi-view neighbor sampler to capture both local and high-order information from graph schema-based and metapath-based neighbors, respectively. We also design several neural network based modules to fuse the information from different nodes, modalities, and relations. In addition, we introduce a feature-based adversarial attack strategy to ensure stable learning and improve model performance. Finally, we design a novel combined objective function of node classification and adversarial learning to optimize the model. To summarize, our major contributions are as follows:

• As the first attempt to study the problem of multi-modal recipe representation learning, we create and release Large-RG, a new recipe graph data with over half a million nodes, which is the largest recipe graph to date.
• We propose a novel GNN-based recipe embedding model Recipe2Vec to solve the problem. Recipe2Vec is able to capture visual, textual, and relational information and learn effective representations through several neural network modules. We further introduce a novel objective function to optimize the model.
• We conduct extensive experiments to evaluate the performance of our model. The results show the superiority of Recipe2Vec by comparing it with state-ofthe-art baselines on two classic food study tasks: cuisine category classification and region prediction.

2 RelatedWork

This work is closely related to recipe datasets, recipe representation learning, and graph neural networks.

Recipe Datasets. Existing datasets focus exclusively on either recipe images, text, or the simple relationship between ingredients [Marin et al., 2019; Haussmann et al., 2019; Tian et al., 2021; Tian et al., 2022a], but fail to contain all of these information and further neglect the complex structure or user information, resulting in poor generality and integrity. Different from existing works, we create and release a recipe graph data which contains extensive relational information and multi-modal information including images and text.

Recipe Representation Learning. Existing works focus on using textual content to learn recipe representations [Li and Zaki, 2020; Tian et al., 2021]. They take advantage of the instructions and ingredients associated with recipes but ignore the large number of recipe images. Several other works consider the images, [Carvalho et al., 2018; Marin et al., 2019; Wang et al., 2019a; Salvador et al., 2021] but they typically concentrate on the recipe-image retrieval task and attempt to align images and text together in a shared space, resulting in information loss for both modalities. Instead, we focus on learning recipe representations using multi-modal information extracted from images, text, and relations.

Graph Neural Networks. Many GNNs [Veliˇckovi´c et al., 2018; Schlichtkrull et al., 2018; Wang et al., 2019b; Zhang et al., 2019a; Zhang et al., 2019b] were proposed to learn vectorized node embeddings. They take advantage of both node attributes and relational information to learn node embeddings for different graph mining tasks. For example, GAT [Veliˇckovi´c et al., 2018] employs self-attention mechanisms to measure the impacts of different neighbors and combine their effects to obtain node embeddings. HAN [Wang et al., 2019b] uses hierarchical attentions to learn embeddings by aggregating information from both node-level and semantic-level structures. Inspired by these studies, we build a GNN-based model to learn recipe embeddings. In addition, adversarial learning has shown great performance in training neural networks [Jiang et al., 2020; Xie et al., 2020]. We thus introduce a feature-based adversarial attack strategy to ensure stable learning and further improve performance.

3 Preliminary

In this section, we describe the concept of heterogeneous recipe graph and formally define the problem of multi-modal recipe representation learning. We further introduce the Large-RG data that we create in this work.

Definition 3.1. Heterogeneous Recipe Graph. A heterogeneous recipe graph is defined as a graph G = (V,E,X) with multiple types of nodes V (i.e., User, Recipe, Ingredient) and edges E (e.g., U-R, R-R, R-I, I-I in Fig. 1). In addition, nodes are associated with attributes X, e.g., images, instructions, and nutrients.

Problem 1. Multi-modal Recipe Representation Learning. Given a recipe graph G = (V,E,X), the task is to design a learning model FΘ with parameters Θ to learn d-dimensional recipe embeddings E ∈ Rd , while encoding the multi-modal information (i.e., visual, textual, and relational information). The learned recipe representations can be utilized in various downstream tasks such as cuisine category classification and region prediction.

Large-RG Data. To solve the problem, we first create Large-RG data. The statistics of Large-RG are reported in Tab. 1. Specifically, we first collect recipes from Recipe1M [Marin et al., 2019] and crawl the user ratings for each recipe from food.com. We then match each ingredient to the USDA nutritional database [USDA, 2019] to determine its nutritional value. After that, we build Large-RG by representing users, recipes, and ingredients as nodes and connecting them with four types of relations. In particular, we build the recipe-ingredient relation by connecting each recipe and its ingredients. We construct the user-recipe relation based on the user ratings. We also connect recipe nodes by the similarity determined by FoodKG [Haussmann et al., 2019] and connect ingredient nodes based on the cooccurring probabilities. Additionally, we crawl recipe tags from food.com to create two categories and assign each recipe a cuisine class and a region class [Tian et al., 2021].

4 Model

In this section, we present the details of Recipe2Vec. As illustrated in Fig. 2, we first encode the node attributes and develop a multi-modal heterogeneous GNN to learn recipe embeddings. The proposed GNN contains several modules including a multi-view neighbor sampler, an adaptive node aggregator (Fig. 2(b)), a cross-modal aggregator (Fig. 2(c)), and an attentional relation aggregator (Fig. 2(d)). We also introduce a feature-based adversarial attack strategy (Fig. 2(e)) to ensure stable learning. Finally, we design a novel objective function to optimize the model.

4.1 Encoding Nodes Attributes

We pretrain a ResNet [He et al., 2016] to encode the recipe images and a two-stage LSTM [Marin et al., 2019] to encode the instructions. Next, we associate the pre-trained image embeddings (ximg) and instruction embeddings (xins) with recipe nodes as the node attributes. Similarly, we consider nutrients as the attributes of ingredient nodes (xing) and random vectors as the attributes of user nodes (xuser). Due to the various modality of attributes, we introduce a modalityspecific input projection to project the node attributes into a shared embedding space:

where Wm ∈ Rdm×d is the projection for modality m, xi,m is the associated attribute of node vi with dimension dm, and hi,m is the projected feature of node vi with dimension d.

4.2 Multi-modal Heterogeneous GNN

For each recipe node vi with input feature hi,m, we first introduce a multi-view neighbor sampler to sample nodes from graph schema view and meta-path view. We then use an adaptive node aggregator to encode the information from nodes under a specific relation. Next, we propose a cross-modal aggregator to fuse the information from different modalities. After that, we design an attentional relation aggregator to combine the information from different relations and obtain the final recipe embedding.

Multi-view Neighbor Sampler. We introduce the multi-view neighbor sampler to simultaneously capture local information through graph schema view and high-order information through meta-path view [Dong et al., 2017]. In the graph schema view, the node aggregates the information from its direct neighbors, which we denote as the graph schema-based neighbors N S i . In the meta-path view, the neighbors are determined by meta-path walks for the highorder information. In particular, given a meta-path P that starts from vi, instead of taking all the nodes that appear in the meta-path as neighbors, we define an importance pooling to select the most important p nodes. We start by simulating multiple meta-path walks from vi and compute the L1 normalized visit count for each node. We then determine the important neighbors by choosing the top p nodes with the highest normalized visit counts. We denoted these selected important neighbors as the meta-path-based neighbors N P i . Finally, the neighbor nodes of vi is the combined set of N S i and N P i , denoted as Ni. In addition, since the recipe nodes are associated with different modalities (e.g., visual and textual), we split the graph into modality-specific graphs to account for each modality individually.

Adaptive Node Aggregator. Given a node vi and its neighbor nodes Ni, we first sample the relation-specific neighbors Ni,r ⊂ Ni by choosing the nodes that connect to vi through relation r. Then, we calculate the unnormalized attention score eij,m between vi and vj ∈ Ni,r given the modality m, and normalize the eij,m using sof tmax function to make the coefficients easily comparable:

where ∥ indicates concatenation, Wij ∈ R2d×d is a shared weight vector, and αij,m is the attention weight between vi and vj for modality m. We then use the αij,m as coefficients to linearly combine the interaction and yield the united interaction-dependent affinity Am for each modality m:

where Wa ∈ Rd×d is a shared trainable weight and ⊙ denotes the element-wise product. We then aggregate the neighbor node features following GIN [Xu et al., 2019] while including two shared transformation matrix Wi ∈ Rd×d and Wj ∈ Rd×d to enhance learning capacity. We further make it adaptive to specific downstream tasks by incorporating the interaction-dependent affinity Am, which makes the message dependent on the affinity between vi and vj:

where hi,r,m is the encoded embedding of vi through relation r for modality m, and Wo ∈ Rd×d is a shared weight vector. Cross-modal Aggregator. To better exploit the information in different modalities and learn a joint embedding, we design the cross-modal aggregator to fuse embeddings from visual and textual modalities. Specifically, we first concatenate the hi,r,image and hi,r,text, and then introduce a node type-specific projection Wϕi ∈ R2d×d to transform the concatenated embedding, with ϕi are the node type of vi:

where hi,r is the learned cross-modal embedding of node vi through relation r.

Attentional Relation Aggregator. We further introduce the attentional relation aggregator to learn the importance of each relation and fuse them accordingly. Specifically, we first apply a weight matrix WR ∈ Rd×d to transform the hi,r and use a relation-level vector q ∈ Rd to calculate the similarity. We then average the similarity scores of all nodes that connected to relation r to obtain the importance score wi,r for node vi. The process is formulated as follows:

where Vr denotes the set of nodes that are connected to relation r, and b ∈ Rd is the bias vector. Next, we normalize wi,r to get the final relation-level attention weight βi,r:

where Ri indicates the associated relations of vi. After that, we fuse the cross-modal embedding hi,r with βi,r to obtain the final recipe embedding hi:

4.3 Objective Function

The learned recipe embedding hi can be used for various downstream tasks such as cuisine category classification, region prediction, or recipe recommendation. Specifically, in this work, we can introduce a supervised node classification loss (e.g., cross-entropy) to classify the cuisine categories (or predict the recipe regions):

where YT is the set of training data, FC denotes the fully connected layer, and Yi is the one-hot label of vi.

4.4 Feature-based Adversarial Learning

Since the above objective function only considers the explicit feature information associated with each node, we further introduce an adversarial attack strategy to leverage the regularization power of adversarial features. Specifically, we choose Projected Gradient Descent [Madry et al., 2018] as the default attacker to generate adversarial features on the fly. Compared to the vanilla training that we send original input features hi,m (Eq. 1) into GNN to obtain the learned recipe embedding hi (Eq. 8), adversarial training takes the maliciously perturbed features h′i,m as input and obtains the perturbed recipe embedding h′i , which is further used to calculate the adversarial learning loss Ladv:

where ϵm is the adversarial perturbation for modality m and S is the allowed perturbation range. The final objective function L is defined as the weighted combination of Lsup and Ladv:

where λ is a trade-off weight for balancing two losses.

5 Experiments

In this section, we conduct extensive experiments to evaluate the performance of Recipe2Vec and show related analyses.

5.1 Baseline Methods

We compare with 10 baselines including the classic classifiers TextCNN [Kim, 2014], ResNet [He et al., 2016], homogeneous graph embedding models GAT [Veliˇckovi´c et al., 2018], GIN [Xu et al., 2019], heterogeneous graph embedding models mp2vec [Dong et al., 2017], RGCN [Schlichtkrull et al., 2018], HAN [Wang et al., 2019b], HGT [Hu et al., 2020], and recipe representation learning models Reciptor [Li and Zaki, 2020] and rn2vec [Tian et al., 2021].

5.2 Implementation Details

We split the data into train/validation/test set by 70/15/15. For the proposed Recipe2Vec, we set the learning rate to 0.005, the hidden size to 128, the input dimension of instruction and image embeddings to 512, the input dimension of ingredient embeddings to 46, batch size to 4096, meta-path P to recipeuser-recipe, the number of meta-path neighbors p to 10, training epochs to 100, the trade-off factor λ to 0.1, the perturbation range S to 0.02, number of iterations for attack to 5, and the attack step size to 0.005.

5.3 Performance Comparison

We use Micro-F1 and Accuracy (Acc) as the evaluation metrics and report the performances on cuisine category classification and region prediction tasks in Tab. 2 and Tab. 3, respectively. The best and second-best values are highlighted by bold and underline. According to these tables, we can find that Recipe2Vec outperforms all the baselines for both tasks in all cases. Specifically, classic classifiers (i.e., TextCNN and ResNet) perform poorly because of the neglect of both relational information and multi-modal information. Graph embedding methods (e.g., GIN and HAN) obtain decent performance after incorporating the complex relational information. Similarly, recipe representation learning models (e.g., rn2vec) achieve satisfactory results, but they fail to encode the influence of different modalities. Finally, Recipe2Vec achieves the best performance compared to all the baselines, by improving +3.5% (F1) and +3.6% (Acc) in cuisine category classification, and +4.5% (F1 and Acc) in region prediction. This demonstrates that Recipe2Vec can obtain better recipe embeddings compared to other models.

5.4 Ablation Study

Since Recipe2Vec contains various essential components (i.e., multi-view neighbor sampler (NS), adaptive node aggregator (NA), cross-modal aggregator (CA), attentional relation aggregator (RA), and feature-based adversarial learning (AL)), we conduct ablation studies to analyze the contributions of different components by removing each of them independently (see Tab. 4). Specifically, removing NA and CA significantly affects the performance, showing that both NA and CA have large contributions to Recipe2Vec. In addition, we remove NS, RA, and AL from our model, respectively. The decreasing performance of these model variants demonstrates the effectiveness of NS, RA, and AL in enhancing the model. Finally, Recipe2Vec achieves the best results in all cases, indicating the strong capability of different components in our model.

5.5 Case Study

To show the performance of different models with concrete examples, we analyze the misclassified cases in the cuisine category classification task, as shown in Tab. 5. Specifically, TextCNN misclassifies all of these four recipes, indicating that the textual features cannot fully represent the recipes and that ignoring relational information may lead to suboptimal performance. HAN successfully classifies the recipe Corn and Coriander Soup, but fails to classify the others. One potential reason is that relying solely on meta-path-based neighbors may result in information loss. rn2vec successfully classifies the recipes Corn and Coriander Soup and Herbed Tomatoes, but fails to classify the other two. This may be because the model cannot fully capture the multi-modal information by only using a simple GNN-based structure. However, our model Recipe2Vec takes into account the visual, textual, and relational information through several neural network modules, which clearly distinguishes the difference among categories and correctly classifies these recipes.

5.6 Embedding Visualization

For a more intuitive understanding and comparison, we visualize embeddings of different models using t-SNE. As shown in Fig. 3, TextCNN does not perform well. Only Beverage, Salads, Desserts, and Bread are separated apart while other categories are mixed. HAN can separate most categories but fails to distinguish the Appetizer, Vegetables, and Main-dish. While rn2vec can successfully separate all categories, their distinctions are obscure, i.e., points from different categories are close to each other. However, our model Recipe2Vec can clearly identify each category and distinctly separate all of them. This again demonstrates that Recipe2Vec can learn discriminative recipe embeddings.

6 Conclusion

In this paper, we propose and formalize the problem of multi-modal recipe representation learning. To solve this problem, we create and release Large-RG, a new and largescale recipe graph data to facilitate graph-based food studies. Furthermore, we develop Recipe2Vec, a novel GNN-based recipe embedding model. Recipe2Vec is able to capture visual, textual, and relational information through various carefully designed neural network modules. We also design a joint objective function of node classification and adversarial learning to optimize the model. Extensive experiments show that Recipe2Vec outperforms state-of-the-art baselines on two classic food study tasks.

Acknowledgements

This work is supported by the Agriculture and Food Research Initiative grant no. 2021-67022-33447/project accession no.1024822 from the USDA National Institute of Food and Agriculture.