Probabilistic Masked Attention Networks for Explainable Sequential Recommendation

1 Introduction

Transformer [Vaswani et al., 2017] and its variants have become the dominant architectures for language modeling tasks, due to their efficient parallel training and good ability of modeling long-range dependencies within sequences [Chen et al., 2022b; Yeh et al., 2022]. In light of this, many researchers have applied Transformers to understand the item-item dependencies within users' sequential actions, which has recently achieved remarkable performance in sequential recommendation. Some popular Transformer-based sequential models include SASRec [Kang and McAuley, 2018], BERT4Rec [Sun et al., 2019], TiSASRec [Li et al., 2020], Transformers4Rec [de Souza Pereira Moreira et al., 2021], and STOSA [Fan et al., 2022].

At the heart of the Transformer is the self-attention mechanism [Vaswani et al., 2017], which offers insights of how a user makes decisions by inspecting the attention distribution to determine the amount of influence of each item in the decision-making process. However, the existing Transformer-based models often rely on dense and fullyconnected attention maps: the attention probabilities are computed by the Softmax function, which always returns positive values [Kang and McAuley, 2018; Sun et al., 2019; Li et al., 2020]. As a result, dense attention maps inevitably assign certain credits to every item in the sequence for nextitem recommendation, causing misleading explainability.

Recent studies show that the dense attentions are not necessary for real-world scenarios, where users' logged data is often noisy, containing a mixture of true-positive and falsepositive interactions [Wang et al., 2021b; Chen and Li, 2019a; Wang et al., 2021c; Chen and Li, 2019b; Chen et al., 2022a; Wang et al., 2022]. For instance, as shown in Figure 1, a user's click logs may contain Headphones, T-shirts, Keyboards, Shoes, and Macbooks in a chronological order. Nevertheless, T-shirts and Shoes are irrelevant to Macbooks, leading to poor explanation in sequential recommendation. To address this issue, ADT [Wang et al., 2021a] discards the large-loss samples with a dynamic threshold to identify and prune the noisy interactions. Rec-denoiser [Chen et al., 2022a] learns a sparse attention to remove the redundant attentions. Nevertheless, Those frameworks suffers from large computational bottleneck, i.e, reweighting loss, multi-head attention, and double forward computation.

In this work, we propose a Probabilistic Masked Attention Network (PMAN) that is more effective for pruning noisy items in sequential recommendation. Our idea is rooted in the principle that explicitly sparsifying attention maps is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments [Child et al., 2019; Correia et al., 2019; Zaheer et al., 2020; Beltagy et al., 2020]. This phenomenon can be also verified by the recent Lottery Ticket Hypothesis [Frankle and Carbin, 2018; Chen et al., 2020], showing that a sparse sub-network is enough to achieve good performance without training a dense network. To identify the sparse pattern of attentions, PMAN adopts a probabilistic mask to select task-specific items in a data-driven fashion, which provides better accuracy. In addition, PMAN uses the squared ReLU rather than Softmax as the activation function to compute attention distributions, abandoning the probabilistic constraints. This allows to generate exactly zero attention scores for irrelevant items, mitigating the impact of falsepositive interactions.

Our PMAN can be formulated as a constrained optimization problem, which can be efficiently solved by standard projected gradient descent. More importantly, our PMAN is an easy-to-implement drop-in replacement for existing selfattention layers with no specialized operations. Remarkably, the sparse attentions are enough to obtain better accuracy than the standard Transformer with a fewer number of model parameters. Experimental studies on real-world benchmark datasets show that PMAN can significantly improve the performance of Transformers, and the performance gain becomes larger for more noisy sequences.

Overall, our contributions are summarized as follows:

• We propose a novel Probabilistic Masked Attention Network (PMAN), which simplifies the design of Transformers and is able to reduce the negative impacts of noisy items in sequential recommendation.
• We introduce a probabilistic masked mechanism to achieve sparse attention distributions, resulting in better performance. Our probabilistic mask can be learned automatically via a constrained optimization framework.
• We empirically demonstrate the effectiveness and efficiency of PMAN in real-world datasets. Besides the superior performance, our PMAN could provide a certain level of explainability, i.e., eliminating the redundant item-item dependencies.

2 RelatedWork

2.1 Sequential Recommendation

Sequential recommendation aims to predict the next item based on a historical sequence of users' actions. Earlier studies mainly adopt Markov Chain models to learn local itemitem transition patterns [Rendle et al., 2010]. Deep sequential models have received much attention owing to their efficiency. Current efficient architectures include Recurrent Neural Networks [Hidasi et al., 2016], Convolutional Neural Networks [Tang and Wang, 2018], and Graph Neural Networks [Wu et al., 2019]. Recently, Transformer-based models have shown promising potential by using the self-attention mechanism to learn pairwise item-item relationships in the sequence. For example, one can adopt either left-to-right unidirectional attentions (e.g., SASRec [Kang and McAuley, 2018] and TiSASRec [Li et al., 2020]) or bidirectional attentions (e.g., BERT4Rec [Sun et al., 2019]) to predict the next item. SSE-PT [Wu et al., 2020] is a personalized Transformer model that further incorporates user embeddings to enhance performance. Recently, LSAN [Li et al., 2021] proposes a twin-attention network to learn both long- and short-term user preference via a dedicated self-attention operation.

However, vanilla self-attention mechanism uses the Softmax function to compute dense attention distributions, which is not Lipschitz continuous [Kim et al., 2021], and is thus sensitive to noisy sequences. Rec-denoiser [Chen et al., 2022a] learns a sparse attention to remove the redundant attentions but with expensive computational efforts due to the Softmax operator. Inspired by recent Softmax-free attention mechanisms [Hua et al., 2022; Shazeer, 2020], we put forward a probabilistic attention network that consists of a probabilistic mask and the ReLU normalization, to prune the noisy items within a sequence. As such, the redundancy item-item dependencies can be removed efficiently, providing clean and sparse attention maps with better explainability.

2.2 Sparse Transformer

Learning sparse and efficient attention mechanisms has recently garnered considerable interest in language modeling [Child et al., 2019; Correia et al., 2019; Zaheer et al., 2020; Beltagy et al., 2020]. The key idea of these sparse Transformers is to sparsify the attention maps by fixing the field of view with pre-defined patterns. For example, Sparse Transformer [Child et al., 2019] uses a fixed attention patterns, where specific cells summarize previous locations and propagate the information to all future cells. Longformer [Beltagy et al., 2020] further increases the receptive field by employing dilated sliding window.

However, these sparse models highly depend on predefined attention schemes, which require domain-specific knowledge and lack flexibility [Yun et al., 2020]. Also, the original purpose of these models is to capture long-range dependencies for long sequences, not to prune noise within sequences. Despite the extension is conceptually straightforward, this direction is less-explored for sequential recommendation. It is thus unclear whether these fixed sparse patterns could generalize well for noisy item sequences [Child et al., 2019; Beltagy et al., 2020]. In this paper, we follow a different route with the aim of learning sparse attentions via a probabilistic mask. As such, the mask can be simultaneously optimized with the downstream objective, which is able to remove the task-irrelevant items in sequential recommendation.

3 Background

3.1 Problem Setup

In the sequential recommendation tasks, let U be a set of users, V be a set of items, and S = {S 1 , S 2 , . . . , S |U| } a collection of users' actions. Each user u ∈ U is associated with a sequence of items S u = (Su 1 , Su 2 , . . . , S u |Su | ) in a chronological order, where |S u | is the length of the sequence, and Su t ∈ V is the item that user u has interacted with at time t. The sequential recommendation is commonly evaluated as next-item prediction [Hidasi et al., 2016; Kang and McAuley, 2018]. For each user u, we seek to predict the next item Su |Su |+1 at time step |Su | + 1 based on the interaction history S u .

3.2 Self-attention Network

Owing to the efficient parallel training, Transformers have been widely used in sequential recommendation [Kang and McAuley, 2018; Sun et al., 2019; Li et al., 2020; de Souza Pereira Moreira et al., 2021]. Here we briefly introduce the design of self-attention mechanism.

Embedding Layer. Transformer-based recommenders maintain an item embedding table T ∈ R|V|×d , where d is the size of embedding vector. For each sequence S u , it can be converted into a fixed-length sequence (Su |Su |-n+1 , . . . , S u |Su | ), where n is the maximum length, such as keeping the most recent n items by truncating or padding items as needed [Kang and McAuley, 2018]). The embedding for (Su |Su |-n+1 , . . . , S u |Su | ) is denoted as E ∈ Rn×d , which can be retrieved from the embedding table T. To preserve the time information, a learnable positional embedding P ∈ Rn×d is further constructed. Usually, the item embedding and the positional embedding are added up:

where the composited embedding X ∈ Rn×d can be directly fed to any sequential recommenders.

Self-Attention Layer. The self-attention layer is critical to learn long-range dependencies within a sequence [Vaswani et al., 2017]. The scaled dot-product attention is widely used as:

where Q = XWQ, K = XWK , and V = XWV are the queries, keys and values, respectively; {WQ,WK ,WV } ∈ Rd×d are weights. Also, one can include other components like point-wise feed-forward layer, residual connection, and layer normalization. We skip the details as these parts are the same as the original Transformer [Vaswani et al., 2017].

Limitations. The scaled dot-product attention in Eq. (2) always generates dense attention distributions, which complicates the item-item dependencies, and even degrades the performance. Moreover, recent studies show that Eq. (2) is not Lipschitz continuous [Kim et al., 2021], implying that small perturbations (e.g., noise) on input sequences are likely to cause large variances of attention distributions. This usually increases the training difficulty.

4 Our Proposed Framework

In this section, we present our Probabilistic Masked Attention Network (PMAN), a simpler yet more desirable for detecting noisy sequences in sequential recommendation.

4.1 Probabilistic Masked Attention Network

Sparse Attention. Let X ∈ Rn×d (from Eq. (1)) be the representations over n items, we make the use of a much simpler attention mechanism to compute the attention distri-

bution [Hua et al., 2022]:

where A ∈ Rn×n contains dense item-item attentions, Z ∈ Rn×d is a shared representation for both query Q(Z) and key Q(Z), where Q(·) and K(·) are two simple affine functions that apply per-dim scalars and offsets to Z, Wz ∈ Rd×d is the weight, and ϕz(·) is a nonlinear function.

To address the issue of noise, PMAN introduces masked mechanism that allows to control what information should be retained/dropped. Specifically, we have:

where H ∈ Rn×d is the output item representations; ⊙ stands for element-wise product; V is the value with weight Wv ∈ Rd×d and nonlinear function ϕv(·); M ∈ Rn×n is a trainable binary mask that sparsifies the attention maps, i.e., Mij = 0 means that the attention Aij is pruned, otherwise Aij is kept. Empirically, we choose the squared ReLU rather than Softmax as activation function. As such, the combination of the binary mask with squared ReLU normalization is able to yield exactly zero probabilities for irrelevant items. We will show how to jointly optimize the discrete variable M with model parameters later.

Loss Function. One can predict the next item (given the first t items) based on Ht (Eq. (4)). An inner product is used to predict users' preference score of item i as:

where Ti ∈ Rd is the embedding of item i. Typically, Transformer feeds a sequence (Su |Su |-n , . . . , S u |Su |-1 ) and its desired output is a shifted version of the same sequence o = (Su |Su |-n+1 , . . . , S u |Su | ). Finally, one can adopt the binary cross-entropy loss as objective function:

where Θ is the model parameters, o′t ̸∈ S u is a negative sample corresponding to ot, and σ(·) is the sigmoid function.

4.2 Probabilistic Sparsification Framework

Constrained Optimization. We jointly train the model parameters Θ and the binary mask M in a unified optimization framework. The empirical risk minimization is:

where L1 norm is applied to the mask to control the sparse degree of the attentions, with the upper bound B. Nevertheless, the objective is discrete with respect to M, which is

Algorithm 1 PMAN

Input: The training sequence set S, attention capacity B, embedding size d.

• 1: Initialize model parameters Θ, mask parameters s;
• 2: for epoch t = 1, 2, . . . , T do
• 3: Schedule the temperature parameter by τ = 0.97(1-t/T)+ 0.03;
• 4: for each mini-batch do
• 5: Compute the input embedding X in Eq. (2);
• 6: Compute dense attentions A in Eq. (3);
• 7: Compute sparse attentions H in Eq. (4);
• 8: Compute the entropy loss L in Problem (8);
• 9: Draw samples {g(i) 1 , g(i) 0 }Ii=1 ∼ Gumbel(0, 1);
• 10: Compute gradient ∇sEp(m|s)L(Θ,m) in Eq. (9);
• 11: Update the mask variables s via PGD in Eq. (12);
• 12: Update the model parameters Θ via SGD;
• 13: end for
• 14: end for
• 15: return 1) The model parameters Θ;
• The trained mask m that can be sampled from the distribution p(m|s).

computationally intractable. To overcome this issue, we view each element of mask M as a binary random variable, then Problem (7) can be relaxed into an excepted loss minimization problem over the probability spaces, which is continuous and differentiable for gradient computation.

Formally, we flatten the mask M as a vector m ∈ Rn2 , and view each element mi as a Bernoulli random variable with probability si to be 1 and (1 - si) to be 0, that is mi ∼ Bern(si), where si ∈ [0, 1]. Assuming the elements of variable m are independent, its distribution becomes p(m|s) = Πi(si)mi (1 - si)(1-mi) . Then we have Em∼p(m|s)∥m∥1 = ∑n2 i=1 si. As such, Problem (7) can be relaxed into the following excepted loss minimization:

where 1 is the all-one vector.

For above problem, we use an alternating optimization schema to iteratively update Θ and s. For model parameters Θ, it can be learned via standard Stochastic Gradient Descent. For variable s, it involves a constrained optimization problem. Next we show how to efficiently update s by using Projected Gradient Descent (PGD).

Projected Gradient Descent. We adopt Gumbel Softmax trick [Jang et al., 2017] to calculate the gradients of the binary variable s as:

where I[·] is the indicator function; g0 and g1 are two random variables, with each element being i.i.d sampled from Gumbel (0, 1) distribution. We use I pairs of Monte Carlo samples (g(i) 1 , g(i) 0 ) to approximate the expectation; σ(·) is the Sigmoid function; τ is the temperature that can be decreased linearly during training [Jang et al., 2017].

Besides, we denote C = {s|1⊤s ≤ B and si ∈ [0, 1]} as the feasible region of Problem (8). For any vector y, Proposition 1 shows that its projection on the set C has a closed-form solution. More details can be found in Appendix 1 .

Proposition 1. Given any vector y, its projection on the set C = {s|1⊤s ≤ B, si ∈ [0, 1]} is computed as:

where α = max(0, β), and β is the solution of the equation:

In addition, let f(β) = 1⊤[min(1,max(0, y - β1))] - B, and it can be verified that f(β) is a monotone decreasing function with respect to β. Thus, the equation f(β) = 0 can be efficiently solved by using the bisection method that converges in the logarithmic rate. After obtaining β∗ , we can get α∗ = max(0, β∗ ), which can be then used to compute the projection in Eq. (10).

To this end, we can apply PGD to update s for Problem (8) by jointly considering Eq. (9) and Eq. (10):

where η denotes the learning rate. We briefly summarize the overall training procedure of PMAN in Algorithm 1.

Complexity Analysis. From Eq. (3) and Eq. (4), PMAN has quadratic complexity over the sequence length as the Transformers, but with a cheaper architecture. More importantly, we empirically observe that PMAN only requires one head, without the need of point-wise feed-forward layer, and layer normalization as in the original Transformer, which enhances computing efficiency. The running time of different Transformer-based models can be found in Appendix. Additionally, it is worth noting that our PMAN can easily achieve linear complexity by using low-rank matrix decomposition [Chen et al., 2021] or Nystr¨om approximation [Xiong et al., 2021]. We leave the extension of linearized attentions in the future.

5 Experiment

5.1 Experimental Setup

Dataset. We consider five benchmark datasets: AmazonBeauty, Amazon-Sports 2 , Yelp 3 , MovieLens1M 4 , and Steam 5 . For each dataset, we group the interactions by users, and sort their items by the timestamps ascendingly. Following [Fan et al., 2022], we adopt 5-core setting to filter out unpopular items and inactive users with fewer than five interaction records. Their statistics are listed in Table 1.

Baseline. We compare our PMANwith the followingmethods: 1) BPRMF [Rendle et al., 2009] is a matrix factorization model with Bayesian personalized ranking loss; 2) FPMC [Rendle et al., 2010] utilizes Markov Chains to learn item transitions; 3) GRU4Rec [Hidasi et al., 2016] adopts GRU to learn the item sequences; 4) Caser [Tang and Wang, 2018] is a CNN-based sequential model; 5) SR-GNN [Wu et al., 2019] employs graph neural network to capture complex item transitions. 6) SASRec [Kang and McAuley, 2018], TiSASRec [Li et al., 2020], BERT4Rec [Sun et al., 2019] are all Transformer-based sequential models; 7) S3 -Rec [Zhou et al., 2020] applies self-supervised learning for sequential recommendation; 8) DSAN [Yuan et al., 2021] is a sparse attention network by replacing softmax with α-entmax. 9) STOSA [Fan et al., 2022] is a recent Transformer model that uses Wasserstein distance mechanism. Our proposed probabilistic mask can generally plug in the Transformer-based models during training process. For our PMAN, we choose SASRec, TiSASRec, and BERT4Rec as its backbones by simply replacing the Transformer blocks (e.g., self-attention layer, point-wise feed-forward layer, and layer normalization etc.) with our sparse masked attention block. Note that we do not apply our mask mechanism on STOSA since it adopts Wasserstein distance attention, rather than the standard scaled dot-product attention.

1 https://www.dropbox.com/s/7namvdjm0jy4cmj/Appendix.pdf?dl=0

2 https://jmcauley.ucsd.edu/data/amazon/

3 https://www.yelp.com/dataset

4 https://grouplens.org/datasets/movielens/1m/

5 https://cseweb.ucsd.edu/ jmcauley/datasets.html#steam data

Parameter Settings. The parameters for the baselines are initialized as their original settings and are then carefully tuned to obtain optimal performance. We adopt Adam as optimizer and search embedding dimension d in Eq. (2) within {32, 64, 128}, the length of item sequence n within {25, 50}. For the attention capacity B in Problem (8), we vary the ratio r in {0.3, 0.5, 0.7, 0.9}, such that B = r · n2 . Moreover, all of our PMANs only use single-head attention in the experiments. The impact of different number of heads will be discussed later.

Evaluation. Following the procedure [Kang and McAuley, 2018; Li et al., 2020; Fan et al., 2022], we use the last item of each user's sequence for testing, the second-to-last for validation, and the remaining items for training. We adopt the widely-used Hit@k and NDCG@k as the evaluation metrics (k = 10 by default). Instead of the biased sampling evaluation, we compute Hit@10 and NDCG@10 by the all-ranking protocol in the experiments [Krichene and Rendle, 2020].

5.2 Overall Performance

Table 2 presents the overall recommendation performance of all methods on the five datasets. All the simulated experiments are repeated five times independently and the average results are reported in the table.

From Table 2, we have the following observations:

• All Transformer-based models (e.g., S3 -Rec, STOSA, SASRec, BERT4Rec, TiSASRec, and PMAN) generally outperform BPRMF, FPMC, GRU4Rec, Caser, and SRGNN with a large margin, implying that the attention

(a) SASRec

mechanism is able to capture long-range item dependencies for sequential recommendation.

• The relative improvements of PMANs over their corresponding backbones are significant for all datasets. For example, SASRec+PMAN outperforms the vanilla SASRec by 7.97% and 9.06% on average in terms of Hit@10 and NDCG@10, respectively. This is mainly attributed to the ability of pruning noisy items via learnable mask in PMANs.
• STOSA and PMANs generally perform better than other Transformer-based models. The reason is that the vanilla self-attention mechanism is not Lipschitz continuous and is vulnerable to small perturbations in real-world applications. STOSA and PMANs address this issue by modeling uncertain noise within sequence.
• TiSASRec+PMAN consistently obtains the best performance for all datasets, which indicates the benefit of considering both temporal information and sparse attentions simultaneously. STOSA models dynamic uncertainty by using stochastic Gaussian distribution, but still generating dense attention maps, which cannot exactly remove the negative impacts of noisy items. Moreover, DSAN performs worse than our PMANs since its αentmax function is relatively sensitive to the hyperparameters, which may cause overfitting issues.

In terms of running time, we empirically observe that the training time of the PMANs is around 0.81 times that of their backbones with the same hardware. That is because PMANs have a much simpler attention module with fewer model parameters. For example, we only use per-dim scalars and offsets to compute the query and key. Also, our single-head attention is sufficient to achieve the best performance without the requirements of point-wise feed-forward layer and layer normalization as in the original Transformer.

Overall, the experimental results demonstrate the superiority of our PMANs. Specifically, the proposed methods outperform all baselines with less time complexity.

5.3 Explainability

In addition to the good performance of our PMANs, we also visualize the attention maps to provide model's interpretability. In Table 3, a random user's engagement history in the Movielens1M dataset is given in a chronological order (column-wise). One can observe that the user tends to watch various types of movies, including Comedy, Drama, Animation, Action, Romance, etc. Specially, the drama and romance are the two types that the user likes to watch most. In this work, we regard the last item (e.g., 26::Othello::Drama) as test set, and aim to investigate the relationships between the last item and the previous engagement history. This can be achieved by inspecting the attention distribution to determine the amount of influence of each item in the sequence.

From humans, it is natural to reason that not all of historical movies are related to the target movie. For example, there is not obvious relationship between 38::It Takes Two::Comedy and 26::Othello::Drama. To see the models' reasoning, we provide attention distribution for both SASRec and SASRec+PMAN in Figure 2. For SASRec, all context movies have non-zero attention weights due to the Softmax attention mechanism. As a result, the dense attention maps cause misleading explainability. In contrast, our SASRec+PMAN can yield exactly zero probabilities for irrelevant movies, such as 38::It Takes Two::Comedy and 29::City of Lost Children::Adventure|Sci-Fi. This is attributed to our probabilistic mask's ability of pruning task-irrelevant items in sequential recommendation. As such, our SASRec+PMAN can pay more attentions on the relevant drama movies, such as 31::Dangerous Minds::Drama and 36::Dead Man Walking::Drama. We observe the similar phenomenon for both TiSASRec+PMAN and BERT4Rec+PMAN, and their attention distribution for both can be found in Appendix.

Concisely, the visualization of spare attentions strongly shows that our PMANs can reduce the negative impacts of noisy items, which highly improves model interpretability.

5.4 Further Probe

Robustness. To further evaluate the robustness of our models, we follow the strategy [Ma et al., 2020] to corrupt the training sequences by randomly replacing a portion of the observed items with uniformly sampled items that are not in the valid/test set. We vary the corrupted ratio δ from 0% to 20%. We report the results of SASRec and SASRec+PMAN in terms of Hit@10 on Beauty and Sports datasets, and we observe similar trends in other datasets. Figure 3 shows that the performance gain becomes larger for more noisy sequences, i.e., the relative improvement ranges from 8.57% to 36.49% for different values of δ. This demonstrates the better robustness of our PMAN than its backbone in the task of sequential recommendation.

Sparsity. One important hyperparameter of our PMANs is the attention capacity B in Problem (8). A small B may lead to aggressive pruning, i.e., B = 0 will mask out all of attentions. Here we set B = r · n2 , and vary the sparsity ratio r within {0.3, 0.5, 0.7, 0.9}. Figure 4 shows the impact of the attention capacity B for Beauty and Sports datasets. We observe that the performance of SASRec+PMAN is consistently better than SASRec, indicating the benefits of pruning techniques. In practice, it is reasonable to set the sparsity ratio within [0.5, 0.7] in our experiments.

Learnable Mask vs. Dropout. Our probabilistic mask in Eq. (4) is an elegant extension to Binary Dropout [Srivastava et al., 2014]. Dropout drops neurons randomly, whereas our probabilistic mask is trainable with the model parameters. We compare our PMAN with Dropout in Table 4. We observe that our model consistently performs better than Dropout across different dropping ratios. Dropout is known to be susceptible to bias: the fact that attentions can be dropped randomly does not mean that the model allows them to be dropped. In contrast, our probabilistic mask would become close to a deterministic sparse mask as the optimizer goes on. Thus, a full trained mask would have much lower variance with better interpretability.

Single Head vs. Multi-head. For Transformer [Vaswani et al., 2017], it is often useful to use multi-head mechanism, which applies self-attention operator in h subspaces, where h denotes the number of heads. Our Eq. (4) can be easily extended to multi-head attentions by following the similar procedure as in Transformer (e.g., projecting X in more subspaces.). As shown in Table 5, the performance of single head roughly achieves the similar results as two heads. Nevertheless, the performance slightly drops with four heads, which may owe to overfitting issue. The appealing feature of our single-head method allows to greatly enhance computing efficiency for large-scale datasets.

6 Conclusion

Here we propose a Probabilistic Masked Attention Networks (PMAN) to filter out irrelevant item-item dependencies to enhance the robustness of Transformer-based recommender systems. We design a probabilistic masked mechanism to sparsify the attention distribution, and jointly train the mask with model parameters. PMAN is compatible with various Transformers, such as SASRec, TiSASRec, and BERT4Rec. Our experiments demonstrate the effectiveness of the proposed PMANs on benchmark datasets. Also, our models are able to provide a certain level of explainability by pruning irrelevant items in the sequences.