Balancing User-Item Structure and Interaction with Large Language Models and Optimal Transport for Multimedia Recommendation

1 Introduction

In the context of the internet age and the resulting information overload, the importance of recommendation systems is becoming progressively more evident. The emergence of multimodal data (e.g., text, images, audio, video, etc.) provides new views to understand user preferences and behavior patterns. Multimodal recommendation addresses the limitations of single-modal information, offering more efficient solutions for applications such as e-commerce and social media.

Graph-based recommendation methods have shown significant performance by modeling higher-order relationships between users and items [Wang et al., 2019; He et al., 2020; Liu et al., 2022a]. In light of these studies, some methods attempted to explicitly learn the latent item-item semantic graphs [Zhang et al., 2021; Zhou and Shen, 2023]. However, user-side information is difficult to access due to privacy policies, which makes the modeling of semantic relationships between users a challenging task, leading to Structure imbalance, as illustrated in Figure 1 (a). To address this issue, some studies have introduced Large Language Models (LLMs) to generate user profiles [Ren et al., 2024; Ma et al., 2024], while others have integrated social graphs from content platforms into graph-based recommendation models to enrich interest patterns [Wu et al., 2022; Wei et al., 2024]. However, simply incorporating user profiles generated by LLMs into ID embeddings fails to capture higherorder user-user relationships. Moreover, inconsistencies between social and behavioral representations may lead to noise propagation [Xiao et al., 2023].

Besides structure imbalance, recommendation systems also suffer from the Interaction imbalance [Liu et al.,

2022b; Wahab et al., 2022; Liu et al., 2023c], which refers to the uneven distribution of user activity and item popularity within interaction data. Here, activity indicates the number of items a user interacts with, while popularity denotes the number of users interacting with an item. As shown in Figure 1 (b), the disparity in user activity and item popularity is so pronounced that many users and items suffer from data sparsity, hindering the learning of high-quality user and item representations [Liu et al., 2021; Xu et al., 2024]. To enrich the behavioral history of sparsely-interacted entities (users or items), more potential items should be uncovered. However, they are often obscured by false negative noise [Li et al., 2021; Ren et al., 2024]. Suppose a female user has not been exposed to an item n that she might like, which is similar to a previously purchased item m, as illustrated in Figure 1 (c). During training, item n may be treated as a negative sample, which can lead the recommender to suppress content the user is actually interested in. This not only wastes potential items but also misguides the modeling of user preferences.

In light of the above limitations and challenges, we propose a novel framework called BLAST that BaLances structures and interActions via large language modelS and optimal Transport for multimodal recommendation. To rectify structure imbalance, we first generate user profiles using large language models. Then, we design an intra- and inter-entity structure balancing module, which converts multimodal features into user-user and item-item graphs, bridging the semantic gap between multimodal information and collaborative signals. Finally, we perform graph convolution on three graphs (the user-item graph, the user-user graph, and the item-item graph) and fuse their outputs to derive a comprehensive representation.

To mitigate interaction imbalance, we first constrain the selection of negative samples when generating the training set to avoid interference from false negatives. Furthermore, to enrich entities with few interactions, we augment the training data with false negative items, then refine the augmented dataset by solving an Optimal Transport (OT) problem, thus providing a solid foundation for efficient representation learning. Note that our OT Augmentation is model-agnostic, which means that it can serve as a plug-and-play tool for a wide range of recommendation models to boost performance. Our main contributions is summarized as follows:

• We propose a novel framework called BLAST to rectify imbalances in structure and interaction for multimedia recommendation, which can effectively fuse multimodal information and collaborative signals to improve item and user representations.
• We highlight the importance of false negative items and propose an effective data augmentation strategy that can be seamlessly incorporated into different recommendation models to enhance their performance.
• Extensive experiments on three real-world datasets demonstrate the superiority and effectiveness of BLAST in multimedia recommendation.

To foster reproducible research, our code is made publicly available at: https://github.com/UPCLHD/BLAST-master.

2 RelatedWork

2.1 Multimodal Recommendation

Multimodal recommendation leverages multimedia content to model user preferences and item attributes. Previous work tends to treat multimedia features as side information to enrich item representations [He and McAuley, 2016; Liu et al., 2017]. However, due to the semantic gap between multimodal information and collaborative signals, directly fusing the two may not be optimal. To this end, several graph-based recommendation methods [Zhang et al., 2021; Guo et al., 2024] have attempted to model item-item relationships by converting multimodal features into bipartite graphs. Nevertheless, the user-item interaction graphs in these methods are susceptible to noise introduced by popularity bias or accidental clicks. To address this issue, recent research has explored cropping redundant edges [Rong et al., 2020; Zhou et al., 2023b; Liu et al., 2023a]. In this work, we argue that user-user relationships are as important as item-item ones, and thus we devise a user-item balanced structure.

2.2 Large Language Models for Recommendation

Recently, several studies have attempted to introduce LLMs into recommendation tasks as inference models [Bao et al.,

2023]. Leveraging the powerful text comprehension and inference capabilities of LLMs, the cross-domain recommendation task has been extended [Vajjala et al., 2024], where the behavior history of the same user in one domain can be applied to other domains to alleviate the cold start problem. RLMRec [Ren et al., 2024] utilizes LLMs to refine valuable text to learn informative representations. XRec [Ma et al., 2024] builds an explainable recommendation system and generates data by LLMs for supervised training. To enhance interactivity, Chat-Rec [Gao et al., 2023] designs a conversational system which generates recommendations by LLMs. However, there are some disadvantages of using LLMs directly as recommenders, such as the illusion issue and slow inference speed. Therefore, in this paper, we only employ LLMs as a preprocessor to help us refine the textual information of items as well as to summarize user preferences.

3 Preliminaries

In our recommendation scenario, we denote the set of users to be U(|U| = M), the set of items to be I(|I| = N), and the user-item interaction matrix to be R ∈ RM×N , where Rui = 1 if user u ∈ U has interacted with item i ∈ I, otherwise Rui = 0 . zu, zi ∈ Rd is the ID embedding of u and i, respectively, where d is the embedding dimension. Nu indicates the set of items that user u has interacted with (positive items). We define the triple (u, i, j) for training, where i ∈ Nu, j ∈ I, but j /∈ Nu. j is selected from negative items at random. Each element of R with a value of 1 (Rui = 1) corresponds to a triple (u, i, j), all of which make up the entire training set D. We adopt the BPR loss to train models:

where yui denotes the predicted preference score of user u for item i. The purpose of the recommendation system is to offer the top-k uninteracted items with the highest score for user u.

In this paper, we set the multimodal features of item i as xmi ∈ Rdm , where m ∈ {v, t} is the modality and dm is the dimension of the features over modality m. v and t denote the visual and textual modalities, respectively.

4 Method

In this section, we describe the core components of BLAST: User and Item Profile Generator, Intra- and Inter-entity Structure Balancing Module, and OT Augmentation. Figure 2 shows the overall architecture.

4.1 User and ItemProfile Generator

Following [Ren et al., 2024; Ma et al., 2024], We extracted user and item profiles by feeding prompts and raw data into the LLM. Specifically, the item profile Pi generated by LLM can be described as follows:

where Mi denotes the pre-defined item prompt, li, di, and ci are title, description and categories of item i, respectively, and ru,i represents the review of user u on item i. To extract deep contextual insights from LLMs, Mi is designed to illustrate the data format, task details, and expected output.

Sampled from items that user u has interacted with, the user profile is generated as follows:

where Mu represents the pre-defined user prompt. Mu elaborates the input and output formats for the LLM and asks it to analyze user preferences. Finally, combining the side information with collaborative signals, we obtain the brief but informative item profile Pi and user profile Pu.

To capture the semantic relationships in the text, Pu and Pi need to be encoded to fixed-length representations:

where T denotes the text encoder. xti ∈ Rd and yu ∈ Rd are the text-modality (t) feature of item i and user u, respectively.

4.2 Intra- and Inter-entity Structure Balancing Module

Item-Item and User-User Graph Constructing

Following [Zhou and Shen, 2023], we construct an initial modality-aware item-item graph Sm by features of each modalitym. Sm ij is the element of the matrix Sm ∈ RN×N in row i, column j, which is calculated by the cosine similarity between the features of item i and item j in modality m:

Then, kNN sparsification [Chen et al., 2009] is conducted. For each item i, only the top-k similar edges are retained:

We set k to 10, and ̂ Sm ij is normalized to ˜ Sm, following [He et al., 2020]. Then, information from all modalities are aggregated into the final item-item graph SI = ∑m∈M αm ˜ Sm.

Beyond the item-item graph SI, the user-user graph SU is constructed from textual features yu extracted from user profiles. Similarly, after getting the user similarity matrix, we perform kNN sparsification:

̂ SU ij is then normalized to SU. Since there is no image for users, SU is the final user-user graph.

Item and User Representation Constructing

Graph-based recommendation methods usually aggregate and propagate information between nodes to model higher-order relationships in user interactions [Zhang et al., 2021; Liu et al., 2023b; Guo et al., 2024]. Inspired by these studies, we conduct graph convolutions on the item-item graph SI, the user-user graph SU , and the user-item graph A, respectively. User-Item graph convolution. Following [Zhou and Shen, 2023], we conduct the edge pruning and normalization on A = ( 0 R R⊤ 0 ) to get A′ . Then, we perform the graph convolution:

where H(l) UI ∈ R(M+N)×d is the l-th layer embedding matrix, H(0) UI is initialized by ID embedding zu, zi, and LUI is the number of layers of the user-item graph convolution. The embedding matrices obtained at each layer are combined to form the representation:

Item-Item and User-User graph convolution. In addition to modeling the relationship between users and items, we also mine the relevance within items and users. Therefore, we conduct graph convolution on the item-item graph SI :

where H(l) I ∈ RN×d is the l-th layer item embedding matrix, H(0) I is initialized by item ID embedding zi(i ∈ I), and LII is the number of layers of the item-item graph convolution. Analogously, the user embedding matrix H(l) U ∈ RM×d is obtained by graph convolution on the user-user graph SU :

Embedding fusion. To integrate relationships from multiple graph convolutions, we fuse the user-item, item-item, and user-user embeddings to obtain the final representation:

where f I i and hUu are the embedding in H(LII+1) I and H(LUU+1) U , respectively, f UI i and hUI u are the embedding in HUI. Then, the predicted preference scores are obtained by calculating the inner product of user and item representations:

4.3 OT Augmentation

As illustrated in Figure 1 (c), given a user u, the items which have interacted with u are called positive items (Nu), otherwise negative items. BPRloss (Eqs. 1) encourages the score of a positive pair (user u, positive itemm) to be higher than a negative pair (user u, negative item n), where m and n sampled from positive items and negative items, respectively. A further distinction can be made among negative items: true negative items Tu, which are those that user u indeed dislikes, such as o, and false negative items Fu, which are those that user u has not seen but is interested in, such as n. In our study, false negative items are defined based on multimodal features. Specifically, for a positive itemm, items connected to it in either the visual item-item graph ( ̂ Svmn = 1) or the textual item-item graph ( ̂ Stmn = 1) are regarded as its false negative items.

Negative Pairs Sampling Restriction

If user u has purchased item m, she may also be interested in item n because of their similar appearance or function. However, if item n is mistakenly included in a negative pair, the model will be trained to assign it a lower prediction score, which may lead the model to misrepresent the user's true preferences. Therefore, false negative items should be excluded from negative pairs. To this end, we impose a constraint on negative pair sampling in the BPR loss. Specifically, we restrict negative samples to be drawn only from true negative items Tu, as shown in Figure 1 (c).

Effect of False Positive Items

Furthermore, we believe that positive samples can also be taken from false negative items. To verify this, we conduct an experiment. Specifically, we build positive pairs using false negative items to augment the Clothing dataset and train a classical model FREEDOM under different ugmentation strategies. The results are demonstrated in Table 1 (higher is better). The experimental settings are as follows: (1) The original dataset without any augmentation. (2) Augment positive pairs by sampling from the false negative items until the number of training data doubles. (3) Augment positive pairs by sampling from the false negative items until each user interacts with 10 items at least. (4) Augment positive pairs with guidance of OT. We can clearly observe that false negative items can be used to supplement samples for items and users that lack interactions, thus compensating for the imbalance of interaction data and improving recommendation accuracy.

Data Augmentation via Sinkhorn

In this work, we reformulate the task of sampling positive pairs from false negative items for data augmentation as an OT problem. The objective of OT is to find an assignment matrix P that minimizes the cost T = ∑i,j CijPij, while satisfying the constraint ∑j Pij = ai and ∑i Pij = bj . C is the cost matrix. By converting C into -C, the cost becomes T = - ∑i,j (-Cij)Pij. The new objective is to maximize ∑i,j (-Cij )Pij , which is equivalent to finding a P that maximizes the inner product with -C. As illustrated in Figure 2, we set ai, bj as the average user activity and item popularity of the dataset, respectively, and fill C with the user-item interaction matrix R and the item-item graph ̂Smij :

where Gi denotes the set of items that item i has interacted with in graph ̂ Sm ij (m ∈ {v, t}), which includes items most visually or textually similar to item i. To maximize the inner product, the elements of P and -C tend to align. Therefore, at positions where Rij = 1 (-Cij = -1), Pij is encouraged to take a negative value, thereby reducing redundancy between the augmented and original data. In contrast, when j is a false negative item (-Cij = 1), Pij tends to be assigned a positive value, thus increasing its likelihood of being selected as part of a positive pair. Furthermore, the row sum (ai) and column sum (bj ) of P are restricted to match the average user activity and item popularity of the dataset, respectively. This ensures that users and items with fewer interactions are provided with more augmented data.

Following [Sarlin et al., 2020; Izquierdo and Civera, 2024], we use the Sinkhorn algorithm to get P. Finally, we construct a set H = {(u, i)|Pui ∈ top - q(P)} by collecting the indices of the top q largest elements in the assignment matrix P, where q = r × |D|. r is the augmentation ratio, and |D| is the size of the original training set. We add H to the original dataset to obtain the augmented training set DH = {(u, i, j)|(u, i) ∈ H or Rui = 1, j ∈ Tu}, which is used to train recommendation models later.

The computational complexity of OT Augmentation is O(MN). Both OT Augmentation and LLM-based profile generation are conducted as preprocessing steps before training, and thus only need to be executed once per dataset, with the results reused throughout the training process. Moreover, BLAST performs LightGCN-style graph convolutions on three graphs using PyTorch's sparse matrix operations for efficiency. Its computational complexity is O(d · (∥R∥0 + M +N)), where ∥R∥0 denotes the number of non-zero entries in the user-item interaction matrix R.

5 Experiment

To validate the effectiveness of our proposed method, we conduct experiments on three widely used datasets to answer the following research questions:

• RQ1: How does BLAST perform compared with stateof-the-art multimodal recommendation models?
• RQ2: How does OT Augmentation perform when applied to other models?
• RQ3: How do different components of BLAST contribute to the overall performance?
• RQ4: How sensitive is BLAST to perturbations in hyperparameters?

5.1 Experiments Settings

Datasets. Following [Zhang et al., 2021; Zhou and Shen, 2023], we conduct experiments on three subsets of the Amazon dataset [McAuley et al., 2015]: (1) Baby, (2) Sports and Outdoors, (3) Clothing, Shoes and Jewelry. We refer to them simply as Baby, Sports and Clothing. Each dataset provides both visual and textual information. The visual information consists of 4096-dimensional features extracted from item images. The textual information includes the title, description, categories, and brand of the items. Following [Ren et al., 2024; Zhang et al., 2023], we encode all textual data (including user and item profiles) by a pre-trained text encoder [Izacard et al., 2022].

Baselines. We compare BLAST with representative stateof-the-art models, including collaborative filters (BPR [Rendle et al., 2009], LightGCN [He et al., 2020], and LayerGCN [Zhou et al., 2023b]) and multimodal recommenders (VBPR [He and McAuley, 2016], LATTICE [Zhang et al., 2021], SLMRec [Tao et al., 2023], BM3 [Zhou et al., 2023c], FREEDOM [Zhou and Shen, 2023], MGCN [Yu et al., 2023], and LGMRec [Guo et al., 2024]).

Evaluation Protocols. To ensure fair comparisons, we adopt the same evaluation settings as those used in [Zhang et al., 2021; Zhou and Shen, 2023]. Specifically, we randomly split all user-item interactions into training, validation and testing sets with a ratio of 8:1:1, and evaluate model performance on the testing set using two widely used metrics: Recall@K and NDCG@K (K = 10 or 20).

Implementation details. We adopt Llama 3.1 1 to generate item and user profiles. The profiles are encoded into 768-dimensional textual features xti and yu using Contriever [Izacard et al., 2022]. We train BLAST using the Adam optimizer [Kingma and Ba, 2015], with a batch size of 2048 and a learning rate of 1e-3. Following [Zhou and Shen, 2023], we perform edge pruning on the user-item graph with pruning ratios of 0.8 or 0.9. The optimal hyperparameters, such as LUI, LUU, and LUU, are selected via grid search. The embedding parameters are initialized using the Xavier method [Glorot and Bengio, 2010]. Early stopping is applied when no improvement is observed on the validation set for 20 consecutive epochs. Visual feature ratio αv is set to 0.1.

5.2 Performance Comparison

Performance of BLAST (RQ1). To demonstrate the effectiveness of our proposed method, we compare BLAST with several state-of-the-art models. The results are presented in Table 2. All baseline results are directly cited from their original papers. We make the following key observations:

1 https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

(1) BLAST significantly outperforms other models, including both traditional collaborative filtering and graph-based approaches, demonstrating the effectiveness of our method. Specifically, compared with the strongest baseline, BLAST achieves improvements of 22.06%, 6.30%, and 7.20% on NDCG@10 for the Baby, Sport, and Clothing datasets, respectively. These results suggest that complementing the graph structure with user affinities and balancing interactions through data augmentation are beneficial for enhancing recommendation performance.

(2) BLAST achieves the largest improvement on the Baby dataset, outperforming LGMRec by 19.10% and 22.06% in Recall@10 and NDCG@10, respectively. This may be attributed to the Baby dataset containing more comprehensive attributes and richer item descriptions, resulting in higherquality item profiles. In BLAST, the item-item graph SI is constructed from the item profiles Pi. Hence, the item text not only directly affects the graph convolution operation but also indirectly influences OT Augmentation through the selection of false negative items.

Performance of OT Augmentation (RQ2). To demonstrate the effectiveness of OT Augmentation, we integrate it into several recommendation models. Table 3 shows the consistent performance gains achieved by augmented interaction data. Note that all results reported in Table 3 are obtained fromMMRec [Zhou et al., 2023a], and may differ from those in Table 2. To ensure a fair comparison, we keep the basic hyperparameter settings consistent for both baselines and their augmented variants. We can observed that OT Augmentation, as a model-agnostic method, significantly enhances the performance of baselines. This highlight both the potential of false negative items and the effectiveness of complementing entities with fewer interactions.

5.3 Ablation Studies (RQ3)

In this section, we evaluate the contribution of each component in BLAST to the overall effectiveness. We choose FREEDOM as the baseline and design several variants:

• BLASTLLM: The LLM summarizes item texts to generate item profiles and learn item-item relationships.
• BLASTFN: Negative sample selection is restricted, and false negative items are utilized to augment the data.
• BLASTUSER: User-user relationships are modeled with user profiles to enhance user representations.

The results are shown in Table 4, from which we have the following observations:

1. Each component of BLAST has a positive impact on the improvement of performance. The most significant gain in performance comes from BLASTUSER, highlighting the effectiveness of incorporating user-user relationships to balance the user and item structure. Notably, these relationships are derived from the LLM, demonstrating its strong contextual understanding and reasoning abilities.
2. On the Clothing dataset, item profiles generated by the LLM result in decreased performance. We attribute this to the limited availability of item descriptions in the dataset. An observation of the profiles suggests that the lack of sufficient information may lead the LLM to make inaccurate inferences.
3. BLASTFN gains consistent improvements over baseline on three datasets, which validates the effectiveness of restrictions and augmentations for false negative items. Constraining negative sampling helps prevent false negatives from disrupting the modeling of user interests. Mining potential interactions balances the training data and supports the representation learning of low-interaction entities.

5.4 Sensitivity Analysis (RQ4)

Augmentation Ratio. To explore the optimal augmentation ratio r, We conduct experiments on three datasets, As illustrated in Figure 3 (a). We observe that BLAST generally achieves optimal performance when r is small (0.1-0.3), and its performance declines as r increases. We infer that with the continuous expansion of the training data, some true negative items are introduced, which encourages BLAST to increase the prediction scores on items that users dislikes, thus decreasing the recommendation accuracy.

Visual Feature Ratio & Edge Pruning Ratio. As illustrated in Figure 3 (b), we evaluate the performance of BLAST under different modality ratios and pruning rates. We observe that BLAST achieves the best performance when the visual feature ratio is low and the edge pruning ratio is high. We speculate that, compared to visual features, textual features are more informative and thus more effective in capturing latent item-item relationships.

Assignment Matrix Constraints. Figure 3 (c) illustrates the performance of BLAST under different OT constraints (ai, bj) on the Sports dataset. We observe that BLAST benefits from appropriate constraint values, particularly when ai and bj are close to the average user activity and item popularity (Sports: 6.14, 11.90). By taking average activity as the objective, OT Augmentation encourages low-interaction entities to engage with more entities, thereby facilitating better representation learning.

Number of Graph Convolutional Layers. As illustrated in Figure 3 (d), we evaluate the performance of BLAST under different combinations of LUI, LUU, and LII on the Baby dataset. We observe that BLAST achieves optimal performance with the following number of layers: LUI = 5, LUU = 2, and LII = 2. This indicates that graph structures built on semantic similarity can effectively model high-order relationships, highlighting the strength of BLAST.

6 Conclusion

In this paper, we propose a model called BLAST that balance user-item structure and interaction for multimedia recommendation. Specifically, we utilize LLMs to generate user profiles, and devise an intra- and inter-entity structure balancing module that model the item-item and user-user relationship simultaneously, integrating their affinities into final representations. Afterwards, to compensate for the user or item with few interactions, we augment the training data guided by the optimal transport and restrict negative sampling to exclude the interference of false negative items. Finally, extensive experiments are conducted on three real-world datasets to demonstrate the effectiveness of our proposed method.

Acknowledgements

This work is partially supported by the Natural Science Foundation of Shandong Province (ZR2023MF007), State Key Lab. for Novel Software Technology (KFKT2024B50, KFKT2024A03).