Towards Automatic Sampling of User Behaviors for Sequential Recommender Systems

1 Introduction

Recommender systems [Koren et al., 2009; Liu et al., 2018; Zhang et al., 2025; Zhang et al., 2024a; Cheng et al., 2021] have become crucial tools for information filtering in various online applications, such as e-commerce, advertising, and online videos. Among them, sequential recommender systems (SRS) [Li et al., 2017; Liu et al., 2022] have become increasingly prevalent due to their ability to capture long- and short-term user interests.

∗Corresponding author

So far, many efforts have been devoted to sequential recommendation, ranging from early matrix factorization and Markov chain based method [Rendle et al., 2010] to stateof-the-art deep neural network models [Cheng et al., 2022], including recurrent neural networks (RNNs) [Hidasi et al., 2015; Hidasi et al., 2016], convolutional neural networks (CNNs) [Tang and Wang, 2018; Yuan et al., 2019], and selfattentive models [Kang and McAuley, 2018; Sun et al., 2019; Li et al., 2021]. Meanwhile, the effectiveness and efficiency of the generative loss (i.e., auto-regressive loss) [Yuan et al., 2019] in SRS tasks have been widely demonstrated in these approaches. Despite their remarkable success, these methods often consider from a model perspective while treating all historical behaviors as uniformly positive. In fact, this may lead to sub-optimal performance as items in the sequences are usually of unequal importance on shaping user interests [Zhang et al., 2013]. For instance, purchased items should hold more significance than merely clicked ones. Moreover, the motivations behind the same action (e.g. clicking) can also be very complex and multifaceted. Interactions on online platforms may sometimes be driven by a herd mentality or could even be the result of mis-clicks, while in other cases, they may accurately represent a user's genuine interests.

Recently, there have also been some works which attempt to improve the recommender systems from a data perspective [He et al., 2023; Qin et al., 2023]. SIM [Pi et al., 2020] and UBR4CTR [Qin et al., 2020] extracts user interests with search or retrieve units to capture the diverse user's longterm interest with target item. Similarly, SDIM [Cao et al., 2022] samples from multiple hash functions to gather similar behavior items to the target for CTR prediction. Though the aforementioned methods have proven effective, they may suffer from the following limitations. Firstly, these methods often select items according to the target, which brings challenges to compute scores across all candidate items in parallel and only suitable to fine-ranking stage [Covington et al., 2016]. And such setup may be too strict. Secondly, these sampling processes are mainly designed for CTR prediction tasks [Guo et al., 2017; Wang et al., 2017], which are often typically optimized as binary classification problems. As a result, these approaches often cannot be equipped with generative loss in SRS tasks, whose effectiveness have been demonstrated in many previous works [Kang and McAuley, 2018; Yuan et al., 2019].

We hold that an ideal solution should be adaptive while maintaining the core training principles of SRS. Hence, to achieve this goal, we propose a general automatic sampling framework, named AutoSAM, to non-uniformly treat historical behaviors for sequential recommendation. To be concrete, an additional sampler layer is first employed to adaptively explore the skew distribution of raw input. Then, informative sub-sets are sampled from the distribution to build more generalizable sequential recommenders. In order to overcome the challenges of non-differentiable discrete sampling actions, and to introduce multiple decision factors for sampling, we further introduce a novel reinforcement learning based method to guide the training of the sampler due to its flexibility. We believe that a proper decision should not only focus on the target item in the future, but also consider the coherence with the previous context. Along this line, we incorporate the major factors into the reward estimation, including Future Prediction and Sequence Perplexity. Finally, both the SRS and the sampler can be jointly optimized in an end-to-end manner by combining the policy gradient. As a result, our method exhibits more accurate recommendations than previous approaches, and is generally effective for various sequential recommenders with different backbones. We summarize the contributions as follows:

• We propose to sample user behaviors from a nonuniform distribution for SRS task. We highlight that the main challenge is to sample historical behaviors dynamically while leveraging the benefits of generative loss.
• We propose a general automatic sampling framework for sequential recommendation, named AutoSAM, to build generalizable SRS with an additional sample layer. We further introduce a reinforcement learning based method to solve the challenges of non-differentiable actions and design multi-objective rewards to optimize the sampler.
• We conduct extensive experiments on public datasets to show superior recommendation results compared to previous competitive baselines. We also validate the generality of the proposed method and additionally analyze the effectiveness of the sampler.

2 RelatedWork

2.1 Sequential Recommendation

Sequential recommendation aims to predict users' future behaviors given their historical interaction data [Luo et al., 2024; Cheng et al., 2024; Zhang et al., 2024b]. Early approaches mainly fuse Markov Chain and matrix factorization to capture both long- and short-term item-item transitions [Rendle et al., 2010]. Latter, with the success of neural network, recurrent neural network (RNN) methods are widely conducted in sequential recommendation [Hidasi et al., 2015; Hidasi et al., 2016]. Besides, convolutional-based models [Tang and Wang, 2018; Yuan et al., 2019] can also be very effective in modeling sequential behaviors. In addition, we notice that graph neural networks [Wu et al., 2019; Xu et al., 2019] have become increasingly prevalent by constructing graph structures from session sequences. And in recent years, self-attention models [Kang and McAuley, 2018;

Sun et al., 2019; de Souza Pereira Moreira et al., 2021] have shown their promising strengths in the capacity of long-term dependence modeling. However, these methods focus on the model architectures and treat all behaviors as uniformly positive, may lead to the sub-optimal performance.

2.2 User Behavior Sampling

As a data-driven technology, recommendation systems have attracted a series of works that consider from the data perspective in recent years [Qin et al., 2021]. To be specific, in SIM [Pi et al., 2020], a two-stage method with a general search unit (GSU) and an exact search unit is proposed to model long-term user behaviors better. Similarly, UBR4CTR [Qin et al., 2020] conducts retrieval-based method to achieve the goal. Besides, SDIM [Cao et al., 2022] proposed a sampling-based approach which samples similar items to the target by multiple hash functions. And some other works [Wang et al., 2021; Lin et al., 2023] mainly formulate a denoising task with the aims of filtering irrelevant or noise items. Despite effectiveness, these methods are mainly designed for CTR prediction, which may be not suitable to conduct the left-to-right generative loss and sample items dynamically in the SRS task. We also notice some works which attempt to capture more informative patterns between user behaviors sequence by designing advance model architectures. RETR [Yao et al., 2022] build recommender transformer for sampling the user behavior pathway. And an all-MLP based model is proposed [Zhou et al., 2022] which adopts Fourier transform with learnable filters to alleviate the influence of the noise item. Differently, in this paper, we present a general framework to learn the distributions of the raw inputs adaptively with carefully designed multi-objectives, so as to enhance the SRS with more informative training data.

3 Preliminaries

3.1 ProblemDefinition

Assume that there are item set I = {i1, i2, ..., i|I|} and user set U = {u1, u2, ..., u|U|}, we denote each behavior sequence of user u ∈ U as Xu = [xu1 , xu2 , ..., xun ], where xut ∈ I is the item that u interacted at step t and n is the sequence length. Sequential recommender systems (SRS) aim to predict the item that users might interact with at the next time step. Different from traditional SRS, the main idea behind this work is to sample historical behaviors with a policy π(Au |Xu ), where Au ∈ {select = 1, discard = 0}n , and then utilize ˆ Xu = [xuk1 , xuk2 , ..., xukm ] to train more generalizable SRS, in which k is the set of time steps about sampled items, m is the length of ˆ Xu .

3.2 Sequential Recommender System

As shown in Figure 1(a), modern SRS contain the components as following. First, each input item xut is mapped into an embedding vector by an embedding lookup operation as: Et = ζxt , where ζ ∈ R|I|×d is item embedding matrix. Then, the sequential embedding will be fed into stacked hidden layers to capture the long- and short-term dependence.

where L stands for the number of layers, H(l) is the output of the l-th hidden layer F (l) while H(0) = E. Next, a prediction layer is adopted to generate the distribution P of the user's next preferences for each time step t:

in which W ∈ Rd×|I| , b ∈ R|I| is the learnable parameters. Finally, to train the model, the auto-regressive generative loss are employed as optimization objectives, i.e., left-toright supervision signals. Combining the cross-entropy loss with such an objective, the loss function can be written as:

where θ is the parameters of the model, yt = xt+1 is the item expect to be predicted at step t, and n is the input length.

4 AutoSAM: the ProposedMethod 4.1 Overview of the Framework

The framework of our proposed AutoSAM is depicted in Figure 1(b). First, we employ a light-weighted sampler to adaptively learn the non-uniform distribution of the raw input. After that, the sequential recommender systems (SRS) are trained with informative sub-sets sampled from the whole sequence to gain stronger generalizations. Considering the challenges of non-differentiable sampling actions and the necessity to introduce multiple decision factors, we further introduce a novel reinforcement learning based method to optimize the sampler. Specifically, we treat the sampler as the agent. For each time step t of the user behavior sequence X, the sampler takes X≤t as the current state st, and outputs an action at ∈ {0, 1} of the t-th item. Then, it observes the carefully designed multi-objective rewards from the environment to update the model via the policy gradient. Finally, the sampler could make optimal decisions by continuously interacting with the environment. And it is worth mentioning that both the sampler and the SRS can be jointly optimized in an end-to-end manner in our proposed framework.

4.2 User Behavior Sampler

We first transform sequence X into embedding EX = ζX + PX, where ζ and P are the embedding matrix and position embedding, respectively. We suggest that the sample decisions should be based on both local and global information. Hence, the sampler leverages a Transformer Block with triangular mask matrix to aggregate the global information X≤t at each step t, and then concatenate it with the local item embedding to generate the sample policy π through an MLP layer. The above procedure can be formulated as:

where || means concatenation, W1 ∈ R2d×d , b1 ∈ Rd ,W2 ∈ Rd×2 , b2 ∈ R2 are learnable parameters. After that, binary decisions A ∈ {0, 1}n are sampled from the Bernoulli distribution as following:

(5) where τ is the temperature of the Softmax. Finally, we can build the sampled sub-set as ˆX = {xt|at = 1}.

4.3 Multi-Objective Rewards

Now we discuss how to design our reward, which plays an important role in learning the optimal π. As mentioned before, traditional SRS often follow the next item prediction task, which can neither update sampler network directly due to the non-differentiable challenge of discrete actions nor control the sample rate. Besides, such single objective may be too strict and one-sided. We argue that the proper decisions should not only focus on the target item but also consider the coherence with preceding context. Along this line, we incorporate the major factors into the reward estimation, including Future Prediction and Sequence Perplexity.

Future Prediction

Generally, items that contribute to predicting the future interactions should be of more opportunities to be selected. Accordingly, we theoretically propose our rewards with consideration of the recommendation loss in Eq. 3. Since we train the SRS with the sampled sub-set ˆX, the objective can be written as:

where θ and ϕ are the parameters of SRS and sampler, respectively. To optimize this objective, we first transform the prediction loss to an equivalent form:

where m, n is the length of ˆX,X respectively. And then derive the gradients with respect to ϕ, we have:

where rpre = ∑n-1 t=1 -Lt(X,A; θ) means the rewards associated to the sampler. In practice, wemaymake a fewchanges to it. First, we add a baseline Lb computed on raw input to L as it is always positive. Second, we discount the future rewards by a factor γ ∈ [0, 1] and ignore the targets whose sample probabilities are lower than ψ to increase stability. Then, the future prediction reward at each time step t can be defined as follows:

in which I[·] is the indicator function, γ is set to 0.9 across all the experiments in this work.

Sequence Perplexity

In addition to sampling based on future predictions, it should also be attached great importance to considering the coherence with previous context. Inspired by perplexity (PP), which is widely used to evaluate the quality of sentences in Natural Language Processing (NLP) [Dathathri et al., 2019] with the definition as:

As words causing high perplexity tend to stray from the context [Lin et al., 2024], similarly, we insist that the items causing high perplexity may lack representativeness. A natural idea is to assign lower sampling probabilities to the items whose predicted probabilities according to historical behaviors are lower than 1/PP(X), so as to reduce the perplexity and obtain informative sub-sets in some way. To this end, the sampler should be encouraged to drop xt (at = 0) while P(xt|X<t) < 1/PP(X), and select xt (at = 1) while P (xt|X<t) > 1/P P (X).

In practice, we employ the SRS to compute the approximate objective at each time step t. Besides, a relax factor is further conducted to control the strictness of the sampler. Above all, the reward can be defined as follow:

where b is the relax factor, by which we can indirectly control the global sampling ratio. Generally, the smaller b will lead the sampler to be stricter.

4.4 Optimization

We summarize the rewards with trade-off parameter λ and scaling factors k as:

To further learn the sequential recommendation task, we also derive the gradients according to Eq. 6 with respect to θ:

Finally, both the SRS and sampler can be optimized jointly in an end-to-end manner by combining the policy gradient:

where α1, α2 are the learning rates of SRS and sampler. Overall, we present our algorithm in Algorithm 1.

4.5 Time and Computation Complexity Analysis

To evaluate the efficiency of our proposed method for online services, we analyze the time complexity of AutoSAM during inference in this part. Denote L as the number of layers and N as the sequence length. Since we conduct Transformerbased SRS whose time complexity is O(LN 2 ), the sampling processing takes O(N 2 ) and the sequence modeling can be done within O(L(E[(∑Ni=1 ai)2 ])) = O(L(µ2N 2 + Nσ2 )), in which µ ∈ {0, 1} denotes the average sampling rate while σ is the variance. Thus, AutoSAM could reduce the time complexity of about O((L - µ2L - 1)N 2 ) by shorten the behavior sequence.

Algorithm 1 Learning the AutoSAM framework

1: Initial SRS parameters θ and sampler parameters ϕ. 2: for Epoch = 1, 2, ...T do 3: for u = 1, 2, ...|U| do 4: X ← [xu1 , xu2 , ..., xun ]; 5: ˆX ← [ ]; 6: Compute Sampling policy πϕ(A|X) ∈ [0, 1]n×2 ; 7: for t = 1, 2, ...n do 8: Sample at from πt; 9: if at = 1 then Append xt to ˆX; 10: end for 11: Compute the recommendation loss L( ˆX; θ) = - ∑m-1 t=1 logP(ˆ yt| ˆX≤t; θ) 12: Compute reward r = k(λrpre + (1 -λ)rpp ) 13: θ ← θ - α1∇θL( ˆX; θ) 14: ϕ ← ϕ + α2∇ϕ ∑nt=1 logπϕ(at|X)rt 15: end for 16: end for

5 Experiments

5.1 Experimental Setup

Datasets

We conduct four real-world datasets from different online platforms as following: Tmall 1 mainly contains anonymized users' shopping logs of Tmall platform in the past 6 months before and on the "Double 11" day. Alipay 2 collects huge amount of user data of Alibaba Group and Koubei between July 1st , 2015 and November 30th , 2015. Yelp 3 is a public dataset where we obtain each user's sequential behaviors by ranking its rated items in time order. Amazon Book 4 (abbreviated as Amazon) is selected from Amazon review data. We construct sequential behaviors by using user's rating history. Similar to some previous works [Zhou et al., 2022], we filter inactive users and items with fewer than c interactions, where c is set to 10 for Tmall, Alipay and Amazon while adjusted to 5 for Yelp due to the smaller data scale. The statics are summarized in Table 1.

Compared Methods

We first compare our method 5 from data perspective with following approaches: (1) FullSAM trains normally with full historical behaviors. (2) FMLP-Rec [Zhou et al., 2022] conducts Fourier transform with learnable filters to alleviate the influence of noise. (3) RETR [Yao et al., 2022] builds recommender transformer to sample the user behavior pathway for sequential recommendation. (4) SDIM [Cao et al., 2022] replaces the attention mechanism with multiple hash functions to sample relevant historical items for modeling longterm preference. (5) RanSAM, (6) LastSAM, (7) PopSAM samples random, last or most popular behaviors respectively. Note that we adopt SASRec [Kang and McAuley, 2018]

1 https://tianchi.aliyun.com/dataset/dataDetail?dataId=42

2 https://tianchi.aliyun.com/dataset/dataDetail?dataId=53

3 https://www.yelp.com/dataset

4 http://deepyeti.ucsd.edu/jianmo/amazon/index.html

5 https://github.com/zh-ustc/AutoSAM

as the sequential recommender system for all sample-based methods due to its effectiveness. It is worth mentioning that some other sampling based methods e.g. UBR4CTR, SIM are mainly designed for CTR tasks and typically optimized as a binary classification problem. While they sampling behaviors using target item, the sampled user behaviors often depend on what the candidate item is [Cao et al., 2022]. As a result, these methods are hard to compute the scores of large candidate set as the million-level setting in this paper.

Besides, We also compare the models with other architectures: (8) PopRec is a popularity-based method, in which each user is recommended according to the popularity. (9) BPR-MF [Rendle et al., 2012] is a well-known matrix factorization-based method optimized by Bayesian personalized ranking loss. (10) FPMC [Rendle et al., 2010] combines matrix factorization with the Markov chain to predict the next interaction. (11) GRU4Rec [Hidasi et al., 2015] is a pioneering attempts by employing recurrent neural network for next item recommendation. (12) NextitNet [Yuan et al., 2019] is a CNN-based method to capture long-term and short-term interests. (13) BERT4Rec [Sun et al., 2019] utilizes a bidirectional self-attention network to model user sequence. (14) SR-GNN [Wu et al., 2019] applies GNN with attention network to model each session. (15) CL4SRec [Xie et al., 2022] utilizes contrastive learning to extract the discrimination information in sequential recommendation. (16) DuoRec [Qiu et al., 2022] is proposed with a contrastive objective serving as the regularization over sequence representations.

Hyper-parameter Settings

We set the batch size of 128 with 10, 000 random negative items per batch, and the embedding size is set to 128 for all methods. In all sampling based methods which employ twolayer SASRec as the backbone, we conduct 4 multi-head selfattention and 2 FFN layers, while the hidden size is set to 256. The sample rates of RanSAM, LastSAM and PopSAM are searched from {0.5, 0.6, 0.7, 0.8, 0.9}. We consistently employ Autom as the default optimizer for all recommenders, combined with a learning rate α1 of 1×e-3 . As for our sampler, we conduct SGD with a learning rate α2 of 1×e-1 . We use grid search to find the best group of AutoSAM's hyperparameters as shown in Table 2.

Evaluation Metrics

We evaluate the performance over all items with Recall and Normalized Discounted Cumulative Gain (NDCG). The first one is an evaluation of unranked retrieval sets while the other reflects the order of ranked lists. We consider top-k items for recommendations where k ∈ {10, 20}. We split the dataset into training, validation and testing sets following the leaveone-out strategy [Cheng et al., 2022; Zhao et al., 2022].

5.2 Experimental Performance Analysis

Overall Performance

The result of different models on datasets are shown in Table 3. First of all, we can find that considering from the data perspective instead of treating all items as uniformly positive can benefit the performance. However, SDIM achieves limited improvement compared to FullSAM. A possible reason is that SDIM samples according to the last item maybe too strict to capture the user's rich interests. Besides, FMLPRec achieves much better performance than traditional SRS probably due to its stronger ability to filter noise information. And we surprisingly find that RanSAM outperforms FullSAM, this suggests that sampling can be regarded as a method of data augmentation in some way, which lead to significant enhancement in generalization capability by increasing the diversity of the training sequence. Unfortunately, RanSAM treats each interaction uniformly and may also drop lots of high-quality behaviors, making the improvement limited. Different from these baselines, the proposed AutoSAM adaptively explore the skew distribution of the raw input, and then enhance the sequntial recommender with informative samples. Consequently, our method performs best on all datasets, which gains an average improvement over the best baseline of 7.40%, 7.99% on Recall@10, NDCG@10, and 6.47%, 7.60% on Recall@20, NDCG@20 respectively.

Generality Analysis

To investigate the general applicability of AutoSAM to other models, we conduct a performance comparison between models using different sampling or filtering methods on various architectures, including RNN-based GRU4Rec [Hidasi et al., 2015], CNN-based NextitNet [Yuan et al., 2019] and Transformer-based SASRec [Kang and McAuley, 2018]. It should be noted that since RETR and SDIM are improvements based on attention mechanisms, they are not universally applicable to other architectures. Figure 2 presents the comparison results in terms of test NDCG@10 on Yelp and Tmall datasets. Note that "+ MLP Filter" corresponds to FMLP-Rec. It is evident that all the sequential recommenders demonstrate the most significant improvements through the integration of automatic sampling, highlighting the generality of our AutoSAM for different backbones.

Ablation Study of Sampling Reward

In the left column of Table 4, we analyze the efficacy of each component of the reward. From the results, we can find that removing either of the components decreases the performance. Besides, rpp plays a more important role than rpre . A possible reason is that rpp could more directly reflect representativeness of a single item. We further show the impact of scaling factor k and trade-off parameter λ by performing a grid search in Table 4. The model achieves the best performance at (k, λ) = (2e-3 , 0.5). The results demonstrate the effectiveness of the incorporation of these two aspects, and it also exhibits the stability of AutoSAM while the different settings of k within the appropriate range bring few influence.

Efficiency and Effectiveness of Different Relax Factors

In practice, it is necessary to balance efficiency and effectiveness when adapting to online services. Thus, we control the relax factors b in Eq. 11 to solve this problem. The performance and the computation cost of the well-trained model in inference w.r.t. average sample rate by setting relax factor from {-0.5, 0, ...2} are shown in Figure 3. Note that the computation cost is measured with million floating-point operations (MFLOPs). We surprisingly observe that even preserving 50% interactions can achieve the comparable performance as baseline. However, while the sample rate reaches 80%, the performance becomes to drop which probably because the sampler leads to a uniform distribution. Besides, the shortened sequences save lots computations especially with small b, highlighting the value in practical applications.

Impact of Sequence Length

AutoSAM can model users' non-uniform interest over interacted items, so the method should compatible with different sequence length. To verify such effectiveness, we analyze the performance improvement of AutoSAM on sequences of different lengths. As shown in Table 5. We can find that (1) AutoSAM is also effective for users with short sequences. (2) Longer sequences do maximize the effectiveness of sampling, This is probably because longer sequences not only contain richer information but also tend to include noise, offering greater potential for optimization.

Sampling Quality Evaluation

Next, we assess the sample quality by comparing the proportion of purchasing behaviors on sampled and discarded items across different sampling strategies. Conventionally, items with purchase or favorite actions are deemed more critical than clicked ones, as they reflect stronger user intent. As illustrated in Figure 4, AutoSAM preserves the highest ratio of purchasing behaviors, indicating that the learned probability distributions better capture user preferences. Notably, AutoSAM retains a substantial number of clicked items. A plausible explanation is that clicked items can still significantly reflect user interests, though the underlying mechanisms governing such behaviors are often complex and multifaceted.

6 Conclusion

In this work, we proposed a general automatic sampling framework, named AutoSAM, to non-uniformly treat historical behaviors. Specifically, a light-weighted sampler was first leveraged to adaptively explore the distribution of raw input, so that the sequential recommender systems (SRS) could be trained with more informative and diverse samples. Considering the challenges of non-differentiable actions and the necessity to introduce multiple decision factors for sampling, we further introduced a novel reinforcement learning-based method to guide the training of the sampler in an end-to-end manner. We conducted extensive experiments on four public datasets. The experimental results showed that the AutoSAM could obtain a higher performance gain by adaptively sampling informative items. We hope this paper could inspire more works to be proposed from data perspective for SRS.

Acknowledgements

This research was supported by grants from the grants of Provincial Natural Science Foundation of Anhui Province (No.2408085QF193), USTC Research Funds of the Double First-Class Initiative (No. YD2150002501), the National Natural Science Foundation of China (62337001), the Key Technologies R & D Program of Anhui Province (No. 202423k09020039) and the Fundamental Research Funds for the Central Universities (No. WK2150110032).