UNBERT: User-News Matching BERT for News Recommendation

1 Introduction

Online news platforms such as Google News 1 and MSN News 2 have become a prevalent way for users to access news information [Das et al., 2007; Lavie et al., 2010]. The large quantity of news articles generated everyday makes it hard for users to hunt for their interested news contents. Therefore, recommendation systems become necessary to provide personalized news recommendations to users according to their preferences [Phelan et al., 2011; IJntema et al., 2010; De Francisci Morales et al., 2012].

∗Corresponding Author

1 https://news.google.com/

2 https://www.msn.com/en-us/news

for various NLP tasks. We argue that BERT-based models could enhance the deep semantic modeling of news to better mitigate the cold-start problem, since both word embeddings and model parameters are pre-trained on the Web-scale textual data full of general-domain knowledge. Meanwhile, we adopt the pretain-finetune strategy to fine-tune the entire model in our in-domain news data for recommendation.

How to precisely match users' interests and candidate news is another key factor for news recommendation [Wang et al., 2020]. A lot of research has been done toward this goal. For instance, DKN [Wang et al., 2018] and NPA [Wu et al., 2019b] learn user representations based on the similarity between candidate news and previously clicked news. LSTUR [An et al., 2019] models short-term and long-term user interests from the clicked news using GRUs. NAML [Wu et al., 2019a] and NRMS [Wu et al., 2019c] apply the attention networks for learning user representations. After obtaining each user vector and item vector separately, these models then match these two vectors to predict click probability. However, treating each news as a whole vector and encoding users and items separately may ignore some low-level matching signals (e.g., word-level relations) between users' interests and candidate news. As shown in Figure 1, the candidate news has a strong semantic similarity to the 1st and 3rd browsed news because all of them are related to movies at the news level. But at the words level, some words such as "Florida" and "American" in the browsed news may mismatch the "Japan" in candidate news, and the user actually does not click the news. As such, the word-level matching signal which tells a user's location is not well utilized for news recommendation. In this paper, we propose a User-News matching BERT (namely UNBERT) model for news recommendation, which not only leverages the powerful pre-trained BERT model but also captures user-news matching signals at both news-level and word-level. The contributions of our work are summarized as follows:

• To the best of our knowledge, UNBERT is the first work to introduce the pre-trained BERT to capture user-news matching signals for news recommendation that takes full advantage of the out-domain knowledge.
• UNBERT proposes the idea of representing user by raw text of the browsed news directly, and learns user-news matching representation at both word-level and newslevel to capture multi-grained user-news matching signals through two matching modules.
• Extensive experiments on the real-world dataset show that our approach can effectively improve the performance of news recommendation.

2 RelatedWork

Traditional methods utilize manual feature engineering to represent news and user for matching [Son et al., 2013; Bansal et al., 2015]. For example, CCTM [Bansal et al., 2015] leverages article and comment content through topic modeling and the co-commenting pattern of users through collaborative filtering to build news and user representations. In recent years, several deep learning methods are proposed for news and user representations, and achieve better performance than traditional methods. For example, DKN [Wang et al., 2018] and NPA [Wu et al., 2019b] use CNN and personalized attention mechanism respectively to learn news representations, LSTUR [An et al., 2019], NAML [Wu et al., 2019a] and NRMS [Wu et al., 2019c] apply the attention networks in news representation, and these methods then learn the user representation by aggregating user's browsed news. In addition, matching-based method FIM [Wang et al., 2020] performs fine-grained matching between segment pairs of each browsed news and the candidate news at each semantic level via stacked dilated convolutions.

Different from these methods, our method introduces a capable pre-trained BERT model to alleviate the cold-start problem by introducing out-domain knowledge, learns the matching representations at both the word-level and news-level via self-attention to capture the multi-grained user-news matching signals.

3 Our Approach

In this section, we first formulate the problem of news recommendation, and then elaborate on our UNBERT including the input and output as well as the model architecture.

3.3 Model Architecture

In this section, we present the architecture details of our UNBERT approach as shown in Figure 3, including Embedding Layer, Word-Level Module, News-Level Module and Click Predictor.

Embedding Layer

For a given input token, the corresponding embeddings including token, segment, position and news segment are generated and the input representation is constructed by summing these embeddings in our study. A visualization of this construction can be seen in Figure 2.

The token, segment and position embeddings are pretrained using masked LM by masking 15% of all word-piece tokens in each sequence at random and predicting the masked words. The news segment embedding is randomly initialized and further updated in our fine-tuning task. The final input token representation Et in UNBERT is constructed by summing its corresponding token, segment, position and news segment embedding 4 :

Word-Level Module

As illustrated in Figure 3, the Word-Level Module (WLM) applies multiple Transformer Layers (TL) iteratively to compute the hidden representations at each layer for each word and propagate the matching signal at word-level simultaneously. Each TL contains two sub-layers: a Multi-Head SelfAttention sub-layer and a Position-wise Feed-Forward network [Vaswani et al., 2017].

4 The position embeddings is not used in our approach because the performance will be worse when using it.

3.1 Problem Statement

Given a user u and a set of candidate news Vu = {v1, v2, . . . , v|Vu|}, where u ∈ U, v ∈ V, U and V denote the set of user and news. Formally, our model aims to generate scores for all candidate news by predicting their click probability of u. The score of the i-th candidate news vi for user u is denoted by ˆ yi = f(u, vi).

3.2 Input and Output

For a given user u and a candidate news v, we denote clicked news of user u as [nu1 , nu2 , . . . , nu|nu| ], where nuj (j = 1, . . . , |nu|) is the j-th clicked news by user u, and nv represents the candidate news v. In this study, each news is represented by the title 3 which is composed of a sequence of words, i.e., nu = [w1,w2, . . .], and wi represent the i-th word.

For news representation, its sequence of words is used as "News Sentence" as shown in Figure 2. And for user representation, we simply concatenate the words of user's clicked news into a whole sequence as "User Sentence", where a segment token [NSEP] is added at begin of each news as the separated signal. As shown in Figure 2, we add the special classification token [CLS] followed by these two sentences and a special token [SEP] to separate them. Finally, the "Input" token sequence to UNBERT is a combination of "News Sentence" and "User Sentence" with a set of special symbols.

The output of UNBERT is a matching score that stands for the probability that the user u will click the candidate news v.

3 In this paper, we only adopt news titles as input, since title is the decisive factor affecting users' choice of reading. But note that our approach can be easily generalized to any sort of news-related texts like the abstract.

Multi-Head Self-Attention applies Scaled Dot-Product Attention as the attention function to learn the combination weights of the output:

where Q, K and V represents the query, key and value matrix correspondingly, which are projected from Et matrix with different learned projection matrices as in Eq.(3), and 1 √dk is a scaling factor to avoid extremely small gradients by producing a softer attention distribution. Multi-Head Self-Attention applies h attention functions in parallel to produce the output representations which are concatenated:

where W O , W Q i , W K i and W V i are learnable parameters, i represents i-th head. With the multi-head, the information from different representation subspaces at different positions of word-level is jointly learnable which would be very helpful for capturing the matching signal between different words.

Position-Wise Feed-Forward network is a fully connected feed-forward network (FFN) applied to each position separately and identically in order to endow the model with nonlinearity and interactions between different dimensions. This consists of two linear transformations with a ReLU activation

where W1, W2 and b1, b2 are learnable parameters and shared across all positions.

In addition, the residual connection is introduced into each of the two sub-layers, as shown in Figure 3, in order to make the model deeper and reduce training difficulty. Moreover, the layer normalization is applied to each sub-layer to normalize the inputs over all the hidden units in the same layer for stabilizing and accelerating the network training.

News-Level Module

We aggregate the word's hidden representation of each news from WLM to the news representation, and then implement the other multiple Transformer Layers to capture the news-level matching signal in News-Level Module (NLM).

Denote the hidden representation for the i-th word obtained from WLM as wi, nj is the j-th news representation aggregated from its sequence of words Sj where i ∈ Sj . Here, we obtain the news representation nj from the word representation wi using the following three types of aggregators:

• NSEP Aggregator directly uses the embedding of the special token [NSEP] for news representation, due to the final hidden state corresponding to this token is designed to represent the token sequence [Devlin et al., 2018]:

• Mean Aggregator averages the words' embedding directly to form the news embedding:

• Attention Aggregator uses a light-weight attention network [Bakhtin et al., 2018] to learn the combination weights of the word embedding matrix w. It applies a fully connected neural network with Wh and bh, tanh as hidden layer activation function, and then another fully connected neural network with Wo and bo to learn combination weights f defined in Eq.(8) and finally computes the linear combination of words embedding in Eq.(9).

We then stack the same transformer layers as WLM to gather the information propagated from these news embeddings, and explore the news-level matching representation in NLM,

Click Predictor

The click predictor module is used to predict the probability of a user clicking a candidate news. Denote the word-level matching representation ew and the news-level matching representation en as shown in Figure 3, we concatenate these two representation vectors as multi-grained matching representation before applying a full connection layer:

where ew and en refer to the word-level and new-level matching representation correspondingly, W c and bc are the learnable parameters.

4 Experiments

4.1 Datasets

We conduct experiments on a real-world news recommendation dataset MIND 5 [Wu et al., 2020] collected from MSN

5 https://msnews.github.io

Boldface indicates the best results (the higher, the better), while the second best is underlined. UNBERT-en△ represents the ensemble score based on UNBERT which is at the top of https://msnews.github.io/#leaderboard.

Table 2: The overall performance of different methods on MIND.

News 6 logs. There are two versions of the MIND datasets named MIND-large and MIND-small, where the MINDsmall is a small version of the MIND-large by randomly sampling the daily behavior logs with equal probability from the MIND-large. The detailed statistics of the datasets are shown in Table 1.

4.2 EvaluationMetrics

Several standard evaluation metrics in the recommendation field are used including: AUC, MRR and nDCG@K with K = 5, 10. The performance is the average of these metrics on all impression logs. Since test set labels of MIND-large are not provided, the test performance is obtained through submission on the MIND News Recommendation Competition 7 .

4.3 Models and Training Details

In our experiments, MIND-small dataset is used to determine the parameter settings, then we train and evaluate on both small and large dataset. The bert-base-uncased is used as the pre-trained model to initialize the word-level module. We apply negative sampling with ratio 4 in consideration of being consistent with other baselines as well as the training efficiency, Adam [Kingma and Ba, 2014] is used for model optimization. The batch size is set to 128, the learning rate is set to 2e-5 , and 2 epochs are trained. All the hyper-parameters are tuned on the validation set.

4.4 Performance Evaluation

The performance of our approach is evaluated by comparing with the following methods: (1) LibFM [Rendle, 2012], which extracts TF-IDF [Beel et al., 2016] features from users' browsed news and candidate news, and concatenates them as the input; (2) DeepFM [Guo et al., 2017]: a deep factorization machine with the same features as LibFM; (3) DKN [Wang et al., 2018], a deep news recommendation method based on knowledge-aware CNN; (4) NPA [Wu et al., 2019b], a neural news recommendation method with personalized attention mechanism; (5) NAML [Wu et al., 2019a], a neural news recommendation approach with attentive multiview learning; (6) LSTUR [An et al., 2019], a neural news recommendation method using GRU to learn user representations; (7) NRMS [Wu et al., 2019c], a neural news recommendation method using multi-head self-attention to learn user and news representations; (8) FIM [Wang et al., 2020], a fine-grained interest matching method for neural news recommendation; (9) UNBERT, our approach. For fair comparison, only title of news is adopted for all methods, in particular, we only use the TF-IDF features extracted from all news titles for LibFM and DeepFM. The results of these methods on two datasets are summarized in Table 2.

6 https://www.msn.com/en-us/news

7 https://competitions.codalab.org/competitions/24122# participate

We have several observations from Table 2. First, the methods which learn word representations in an end-to-end manner such as DKN, NPA, NAML, LSTUR and NRMS outperform the neural recommendation methods that craft the features manually such as LibFM and DeepFM, which tells that these end-to-end methods are more suitable for news representation learning than the methods based on manual feature engineering.

Second, the methods of learning news representation using the attention mechanism (e.g., NPA, NAML, LSTUR and NRMS) outperform DKN. This is probably because the attention mechanism can model the interaction between the words, and learn the news representations more accurately by capturing relative importance of the interaction. Particularly, NRMS outperforms other attention-based methods because it adopts the useful multi-head self-attention to capture the relatedness between users' browsed news and the candidate news in news-level.

Third, FIM outperforms other methods because its pairwise multi-level matching architecture can detect fine-grained matching signals not just news-level matching information.

Finally, our approach can consistently outperform other baseline methods in terms of all metrics. The significant improvement indicates that the leverage of out-domain data through the pre-trained model can introduce rich language knowledge and enhance the textual representation. Moreover, this also validates advantage of the multi-grained user-news matching signals at both word-level and news-level via selfattention to predict the probability of a user clicking a candidate news.

4.5 Ablation Study of WLM and NLM

In order to study the effectiveness of word-level module and news-level module which aims to capture the multi-grained user-news matching signal, we test our model by disabling one part to evaluate another, yielding UNBERT with wordlevel signal (UNBERTword), UNBERT with news-level signal (UNBERTnews) and the complete UNBERT, as shown in Table 3. The experiment is conducted on the MIND-small due to the submission limit on the MIND-large.

Firstly, we observe that UNBERTword and UNBERTnews achieve a pretty good performance close to the full version of UNBERT, which confirms the effectiveness of these two level matching signals. Secondly, UNBERTnews outperforms UNBERTword, which proves that the word-level is insufficient for its weakness on capturing news structure. Finally, the full version of UNBERT performs best, which tells that the multigrained matching signal is necessary for news recommendation.

4.6 Impact of Aggregator Type

We also study the impact of different aggregator type in NLM of UNBERT, on the final ranking performance in Table 4 .

We observe that Attention Aggregator achieves the best performance, Mean Aggregator follows and NSEG Aggregator is the worst. Since NSEG Aggregator uses the embedding of a special token that can represent news from word embeddings to some extent, while Mean Aggregator aggregates word representation to news representation by mean reducing explicitly, which leads to a better performance of Mean Aggregator. Attention Aggregator applies attention mechanism and further improves the performance, which indicates that the relatedness between the words is a vital factor for news representation.

4.7 Effectiveness on Cold Start

This experiment studies the performance of different news recommendation methods on unseen news. Recall that the large quantity of unseen news are generated everyday, and effective method should be able to alleviate this problem. Therefore, we evaluate UNBERT on the test set of different days, and compare it with other models. We divide the MIND-small dataset into 7 groups by day, train and validate models on the first 3 days, and then test models on each day of the remaining 4 days.

As shown in figure 4, our method consistently outperforms other methods in a large margin on different days, which proves the effectiveness of UNBERT. In addition, with the cold news growing over time, UNBERT performs stably, while the performances of other methods drop significantly, especially from 11/13/2019 to 11/14/2019. This illustrate the advantage of out-domain knowledge on alleviating the coldstart problem. The abnormal that the overall performance increases while the percentage of unseen news increases from 11/12/2019 to 11/13/2019 is due to the token distribution has not changed much between these two days.

5 Conclusion

In this paper, we propose a novel method named User-News matching BERT (UNBERT) for news recommendation. Our method introduces a pre-trained model on the out-domain data to alleviate the cold-start problem, and captures multigrained user-news matching signals at both word-level and news-level through WLM and NLM. The experimental results show that UNBERT outperforms the state-of-the-art methods on the real-world dataset in terms of AUC, MRR, nDCG@5 and nDCG@10, and UNBERT ranks the first in the leaderboard of MIND competition.