Neural News Recommendation with Attentive Multi-View Learning

1 Introduction

Online news services such as Google News 1 and MSN News 2 which collect and aggregate news from many sources have become popular platforms for news reading [Das et al., 2007; Lavie et al., 2010]. Tens of thousands of news are generated everyday, and it is impossible for users to read all these news articles due to time limit [Phelan et al., 2011]. Therefore, personalized news recommendation is very important for online news platforms to help users find their interested news and alleviate information overload [IJntema et al., 2010; De Francisci Morales et al., 2012].

1 https://news.google.com/

2 https://www.msn.com/en-us/news

Many deep learning based news recommendation methods have been proposed [Okura et al., 2017; Wang et al., 2018]. A core problem in these methods is how to learn representations of news and users. Existing methods usually learn these representations based on single news information. For example, Okura et al. [2017] proposed to learn news representations from the body of news articles via auto-encoders, and then learn users representations from news representations by applying GRU to their browsed news. Wang et al. [2018] proposed to learn news representations via a knowledge-aware CNN from the titles of news articles, and then learn user representations from news representations based on the similarity between candidate news and each browsed news. The single kind of news information may be insufficient for learning accurate news and user representations, and may limit the performance of news recommendation.

Our work is motivated by the following observations. First, a news article usually contains different kinds of information, such as title, body and topic category, which are all useful for representing news. For example, in Fig. 1 the title of the first news indicates that it is about a deal between a baseball star and a team, and the body can provide more detailed information of this deal. In addition, the topic category of this news is also informative, since if a user clicks this news and many other news articles with the same category, then we can infer she is very interested in sports. Thus, incorporating different kinds of news information has the potential to improve news and user representations for news recommendation. Second, different kinds of news information have different characteristics. For example, the titles are usually very short and concise, while the bodies are much longer and more detailed. The topic categories are usually labels with very few words. Thus, different kinds of news information should be processed differently. Third, different news information may have different informativeness for different news. For example, the title of the first news in Fig. 1 is precise and important for representing this news, while the title of the second news is short, vague, and less informative. Besides, different words in the same news may have different importance. For example, in Fig. 1, "outfileder" is more important than "familiar" for representing the first news. In addition, different news browsed by the same user may also have different informativeness for learning user representations. For example, if a user browses a news of "Super Bowl" which is extremely popular and a news of "International Mathematical Olympiad" which is less popular, then the second news may be more informative for inferring her interests than the first one.

In this paper, we propose a neural news recommendation approach with attentive multi-view learning (NAML) to learn informative news and user representations by exploiting different types of news information. Our approach contains two core components, i.e., a news encoder and a user encoder. In the news encoder, we propose an attentive multi-view learning framework to learn unified news representations from titles, bodies and categories by incorporating them as different views of news. In addition, since different words and views may have different informativeness for news representation, we apply word-level and view-level attention networks to select important words and views for learning informative news representations. In the user encoder, we learn the representations of users from the representations of their clicked news. Since different news may also have different informativeness for user representations, we apply attention mechanism to news encoder to select informative news for user representation learning. We conduct extensive experiments on a realworld dataset. The results show our approach can effectively improve the performance of news recommendation.

2 RelatedWork

News recommendation is an important task in natural language processing and data mining fields, and has wide applications [Zheng et al., 2018; Wu et al., 2019a]. Learning accurate news and user representations is critical for news recommendation. Many existing news recommendation methods rely on manual feature engineering for news and user representation learning [Liu et al., 2010; Son et al., 2013; Bansal et al., 2015]. For example, Liu et al. [2010] proposed to use topic categories and interest features generated by a Bayesian model as news representations. Son et al. [2013] proposed an Explicit Localized Semantic Analysis (ELSA) model for location-based news recommendation. They proposed to extract topic and location features from Wikipedia pages as news representations. Lian et al. [2018] proposed a deep fusion model (DMF) to learn news representations from various handcrafted features such as title length and news categories. However, these methods rely on manual feature engineering, which needs a large amount of domain knowledge and effort to craft. In addition, these methods cannot capture contexts and orders of words in news, which are important for learning news and user representations.

In recent years, several deep learning based news recommendation methods are proposed [Okura et al., 2017;

Kumar et al., 2017; Khattar et al., 2018; Wang et al., 2018; Wu et al., 2019c]. For example, Okura et al. [2017] proposed to learn news representations from news bodies using denoising autoencoders. In addition, they proposed to learn user representations from their browsed news using GRU network. Wang et al. [2018] proposed to learn news representations from news titles via a knowledge-aware CNN network which can incorporate the information in knowledge graphs. However, these methods can only exploit a single kind of news information, which may be insufficient for learning accurate representations of news and users. Different from these methods, our approach can learn representations of news and users by incorporating different kinds of news information such as title, body and topic category via an attentive multi-view learning framework. Extensive experiments on real-world dataset validate our approach can learn better news and user representations, and achieve better performance on news recommendation than existing methods.

3 Our Approach

In this section, we introduce our NAML approach for neural news recommendation. There are three major modules in our approach, i.e., a news encoder with attentive multi-view learning to learn representations of news, a user encoder with attention mechanism to learn representations of users from their browsed news, and a click predictor to predict the probability of a user browsing a candidate news article. The architecture of our approach is shown in Fig. 2.

3.1 News Encoder

The news encoder module is used to learn representations of news from different kinds of news information, such as titles, bodies and topic categories. Since different kinds of news information have different characteristics, instead of simply merging them into a long text for news representation, we propose an attentive multi-view learning framework to learn unified news representations by regarding each kind of news information as a specific view of news. As shown in Fig. 2, there are four major components in news encoder.

The first component is title encoder, which is used to learn news representations from their titles. There are three layers in the title encoder. The first layer is word embedding, which is used to convert a news title from a word sequence into a sequence of low-dimensional semantic vectors. Denote the word sequence of a news title as [wt1 ,wt2 , ...,wtM ], where M is the length of this title. It is converted into a sequence of word vectors [et1 , et2 , ..., etM ] via a word embedding look-up table We ∈ RV ×D , where V and D are vocabulary size and word embedding dimension respectively.

The second layer is a convolutional neural network (CNN) [Kim, 2014]. Local contexts of words in news titles are important for learning their representations. For example, in the news title "Xbox One On Sale This Week", the local contexts of "One" such as "Xbox" and "On Sale" are useful for understanding that it belongs to a game console name. Thus, we use CNNs to learn contextual word representations by capturing their local contexts. Denote the contextual representation of the i-th word as cti , which is calculated by:

where et(i-K):(i+K) is the concatenation of word embeddings from position (i -K) to (i +K). Ft ∈ RNf×(2K+1)D and bt ∈ RNf are the kernel and bias parameters of the CNN filters, Nf is the number of CNN filters and 2K + 1 is their window size. ReLU [Glorot et al., 2011] is the non-linear activation function. The output of this layer is a sequence of contextual word representations, i.e., [ct1 , ct2 , ..., ctM ].

The third layer is a word-level attention network [Wu et al., 2019b]. Different words in the same news title usually have different informativeness for learning news representations. For example, in the news title "Rockets End 2018 With A Win" the word "Rockets" is more informative than "With" in representing this news. Thus, recognizing the important words in different new titles has the potential to learn more informative news representations. We propose to use a wordlevel attention network to select important words within the context of each news title. Denote the attention weight of the ith word in a news title as αt i , which is formulated as:

where Vt and vt are the projection parameters, qt denotes the attention query vector.

The final representation of a news title is the summation of the contextual representations of its words weighted by their attention weights, i.e., rt = ∑Mj=1 αtj ctj .

The second component in news encoder module is body encoder, which is used to learn news representations from their bodies. Similar with the title encoder, there are also three layers in the body encoder.

The first layer is word embedding, which is shared with the title encoder. Denote the word sequence of a news body as

[wb1 , wb2 , ..., wbP ], where P is the length of the body. Through the word embedding layer, it is transformed into a sequence of word vectors [eb1 , eb2 , ..., ebP ]. The second layer is a CNN network, which takes the word vector sequence as input, and learns the contextual word representations [cb1 , cb2 , ..., cbP ] by capturing the local contexts. The third layer is an attention network. Different words in the same news body may have different informativeness for news representations. For example, in Fig. 1 the word "outfielder" is more informative than "familiar" in representing this news. Thus, we apply an attention network to the body encoder to select important words in the context of each news body for learning more informative news representations. Denote the attention weight of the ith word in a news body as αb i , which is calculated by:

where Vb and vb are projection parameters, and qb is the attention query vector. The final representation of a news body is the summation of the contextual word representations weighted by their attention weights, i.e., rb = ∑Pj=1 αb j cbj .

The third component in the news encoder module is category encoder, which is used to learn news representations from their categories. On many online news platforms such as MSN News, news articles are usually labeled with topic categories (e.g., "sports" and "entertainment") and subcategories (e.g., "basketball nba" and "gaming") for targeting user interests which contain important information of these news. For example, if a user clicked many news articles with the category of "sports", then we can infer that this user may be interested in sports news. Thus, we propose to incorporate both the category and subcategory information for news representation learning. The inputs of the category encoder are the ID of the category vc and the ID of the subcategory vsc. There are two layers in the category encoder. The first layer is a category ID embedding layer. It converts the discrete IDs of categories and subcategories into low-dimensional dense representations (denoted as ec and esc respectively). The second layer is a dense layer. It is used to learn the hidden category representations by transforming the category embeddings as:

where Vc, vc, Vs and vs are parameters in dense layers.

The fourth component in the news encoder module is attentive pooling. Different kinds of news information may have different informativeness for learning the representations of different news. For example, in Fig. 1, the title of the first news is precise and contains rich information. Thus, it should have a high weight in representing this news. However, the title of the second news is short and vague. Thus, the body and the topic category should have higher weights than the title for representing this news. Motivated by these observations, we propose a view-level attention network to model the informativeness of different kinds of news information for learning news representations. Denote the attention weights of title, body, category and subcategory as αt, αb, αc and αsc respectively. The attention weight of the title view is calculated by:

where Uv and uv are projection parameters, qv is the attention query vector. The attention weights of other news information such as body, category and subcategory can be computed in a similar way.

The final unified news representations learned by the news encoder module is the summation of the news representations from different views weighted by their attention weights:

In our NAML approach the news encoder module is used to learn the representations of both historical news browsed by users and the candidate news to be recommended.

3.2 User Encoder

The user encoder module in our approach is used to learn the representations of users from the representations of their browsed news, as shown in Fig. 2. Different news browsed by the same user have different informativeness for representing this user. For example, the news "10 best NBA moments" is very informative for modeling user preferences since it is usually browsed by users who are interested in sports, but the news "The Weather Next Week" is less informative since it is clicked by massive users. Therefore, in user encoder module we apply a news attention network to learn more informative user representations by selecting important news. We denote the attention weight of the ith news browsed by a user as αn i , which is calculated as follows:

where qn, Wn and bn are the parameters in the news attention network. The final representation of a user is the summation of the representations of news browsed by this user weighted by their attention weights: u = ∑Ni=1 αn i ri, where N is the number of browsed news.

3.3 Click Predictor

The click predictor module is used to predict the probability of a user browsing a candidate news based on their representations. Denote the representation of a candidate news Dc as rc and the representation of user u as u. Following [Okura et al., 2017], the click probability score ˆ y is calculated by the inner product of the representation vectors of user u and the candidate news Dc , i.e., ˆ y = uT rc. We also explored other probability computation methods such as multi-layer neural networks [Wang et al., 2018]. We find that the inner product is not only the one with the best time efficiency but also the one with the best performance, which is consistent with [Okura et al., 2017].

3.4 Model Training

Motivated by [Huang et al., 2013] and [Zhai et al., 2016], we propose to use negative sampling techniques for model training. For each news browsed by a user which is regarded as a positive sample, we randomly sample K news which are presented in the same session but are not clicked by this user as negative samples. We then jointly predict the click probability scores of the positive news ˆ y+ and the K negative news [ˆ y1 , ˆ y2 , ..., ˆ y-K ]. In this way, we formulate the news click prediction problem as a pseudo K+1-way classification task. We normalize these click probability scores using softmax to compute the posterior click probability of a positive sample as follows:

where ˆ y+ i is the click probability score of the i-th positive news, and ˆ yi,j is the click probability score of the j-th negative news in the same session with the i-th positive news. The loss function L in our approach for model training is the negative log-likelihood of all positive samples, which can be formulated as follows:L = - ∑

where S is the set of the positive training samples.

4 Experiments

4.1 Datasets and Experimental Settings

Our experiments were conducted on a real-world dataset, which was constructed by randomly sampling user logs from MSN News 3 in one month, i.e., from December 13, 2018 to January 12, 2019. The detailed statistics of the dataset are shown in Table 1. We use the logs in the last week as the test set, and the rest logs are used for training. In addition, we randomly sampled 10% of training samples for validation.

3 https://www.msn.com/en-us/news

In our experiments, the dimensions of word embeddings and category embeddings were set to 300 and 100 respectively. We used the pre-trained Glove embedding [Pennington et al., 2014] in our approach. The number of CNN filters was set to 400, and the window size was 3. The dimension of the dense layers in the category view was set to 400. The sizes of attention queries was set to 200. The negative sampling ratio K was set to 4. We applied 20% dropout [Srivastava et al., 2014] to each layer in our approach to mitigate overfitting. Adam [Kingma and Ba, 2014] was used as the optimization algorithm. The batch size was set to 100. These hyperparameters were selected according to the validation set. The metrics in our experiments include the average AUC, MRR, nDCG@5 and nDCG@10 scores over all impressions. We independently repeated each experiment for 10 times and reported the average performance.

4.2 Performance Evaluation

First, we will evaluate the performance of our approach by comparing it with several baseline methods. The methods to be compared include: (1) LibFM [Rendle, 2012], which is a state-of-the-art feature-based matrix factorization method; (2) CNN [Kim, 2014], applying CNN to the concatenation of news titles, bodies and categories to learn news representations; (3) DSSM [Huang et al., 2013], a deep structured semantic model with word hashing via character trigram and multiple dense layers; (4) Wide&Deep [Cheng et al., 2016], using the combination of a wide linear channel and a deep neural network channel; (5) DeepFM [Guo et al., 2017], using a combination with factorization machines and neural networks; (6) DFM [Lian et al., 2018], a deep fusion model by using combinations of dense layers with different depths; (7) DKN [Wang et al., 2018], a deep news recommendation method based on knowledge-aware CNN; (8) NAML, our neural news recommendation approach with attentive multiview learning. The experimental results of different methods are summarized in Table 2.

According to Table 2, We have several observations. First, the methods based on neural networks (e.g., CNN, DSSM and NAML) outperform traditional matrix factorization methods such as LibFM. This is probably because neural networks can learn better news and user representations than traditional matrix factorization methods.

Second, the methods using negative sampling (DSSM and NAML) outperform the methods without negative sampling (e.g., CNN, DFM and DKN). This is probably because the methods without negative sampling are trained on a balanced dataset, which is not consistent with real-word news recommendation scenarios. DSSM and our NAML approach can utilize the information from much more negative samples than other baseline methods, which is useful for learning more accurate news representations.

Third, our approach can consistently outperform other baseline methods. Since the characteristics of news titles,

• (a) Multi-view learning.

(b) Attention networks.

Figure 3: The effectiveness of the multi-view learning framework and the attention networks in our approach.

bodies and categories are usually divergent, simply concatenating the features extracted from these sources may be suboptimal. Different from baseline methods, our approach uses a multi-view framework to incorporate different information as different views. In addition, different words, news and views may have different informativeness for representing news and users. Our approach can simultaneously select important words, news and views, which can build more informative news and user representations.

4.3 Effectiveness of AttentiveMulti-ViewLearning

In this section, we conducted several experiments to validate the effectiveness of attentive multi-view learning framework in our model. First, we explore the effectiveness of multiview framework in our NAML approach. The performance of NAML and its variants with different combinations of views is shown in Fig. 3(a). According to Fig. 3(a), we have several observations. First, the model with the body view achieves better performance than those with the title or category view only. This is intuitive because the bodies of news usually contain the original information of news, and can provide rich information for modeling news topics. Second, the title and category views are also informative for news recommendation. This is probably because titles usually have decisive influence on users' reading behaviors. Thus, incorporating news titles is useful for modeling the characteristics of news and users. In addition, since categories of news are important clues of news topics, incorporating the category view is also useful for recommendation. Third, combine all three views can further improve the performance of our approach. These results validate the effectiveness of our multi-view framework.

Next, we conducted experiments to validate the effectiveness of the attention mechanism at word-, news- and viewlevel. The performance of NAML and its variant with differ-

(a) The mean and standard deviation of attention weights of views. (b) Distributions of attention weights of views.

Figure 4: Visualization of the view-level attention weights.

ent combinations of attention networks is shown in Fig. 3(b). According to Fig. 3(b), we have several observations. First, the word-level attention network can effectively improve the performance of our approach. This is probably because words are basic units in titles and bodies to convey their meanings, and different words usually have different informativeness for learning news representations. Our approach can recognize and highlight the important words, which is useful for learning more informative news representations. Second, the news-level attention is also important to improve the performance of our approach. This is probably because news browsed by the same user usually have different informativeness for learning the representations of this user, and recognizing important news is useful for learning high-quality user representations. Third, using view-level attention network can also improve the performance of our approach. Since the three views can also have different informativeness for modeling news and users, evaluating the importance of views may be useful for improving the performance of our approach. Fourth, combining all attention networks can further improve the performance of our approach. These results validate the effectiveness of attention mechanism in our approach.

4.4 Visualization of AttentionWeights

In this section, we will visually explore the effectiveness of the attention mechanism in our approach. First, we want to visualize the view-level attention weights of the three views. The results are shown in Fig. 4. From Fig. 4(a), we find the average attention weights of the body view are higher than those of the title view. This may be because bodies usually convey the original meanings of news and contain richer information than titles. Thus, the attention weights of the body view is higher. Second, we find it is interesting that the category view gains the highest attention weights. This may be because categories are very important clues for inferring news topics, which are very informative for learning accurate news representations. The visualization results of distributions of the view-level attention weights are shown in Fig. 4(b). We find that the attention weights on the title and body views are small for many samples. This may be because some news titles and bodies are vague and uninformative for learning news representations. In these cases, categories are can provide rich complementary information for recommendation, which validate the effectiveness of our multi-view framework.

Then, we want to visually explore the effectiveness of the word- and news-level attention networks. For the word-level attention network, we only visualize the attention weights of the words in news titles since news bodies are usually too long. The visualization results of several clicked news from a randomly selected users are shown in Fig. 5. From Fig. 5, we find the attention network can effectively recognize important words within news articles. For example, the word "NBA" and "Rockets" are highlighted, since these words are very informative for representing news topics, while the word "December" is assigned a low attention weight since it is less informative for learning news topics. These results show that our approach can learn informative news representations by recognizing important words. From Fig. 5, we also find our approach can also effectively recognize important news of a user. For example, the news "Rockets set NBA 3 - point record , blast Wizards" gains high attention weight because it is very informative for modeling user preferences, while the news "Today in History : December 22" gains low attention weight, since it is not very informative for representing users. These results show that our model can effectively evaluate the different importance of words and news articles for learning informative news representations.

5 Conclusion

In this paper, we propose a neural news recommendation approach with attentive multi-view learning (NAML). The core of our approach is a news encoder and a user encoder. In the news encoder, we propose a multi-view learning framework to learn unified news representations by incorporating titles, bodies and categories as different views of news. We also apply attention mechanism to news encoder to select important words and views for learning informative news representations. In the user encoder, we learn representations of users from their browsed news, and apply a news attention network to select important news for learning informative user representations. Extensive experiments on real-world dataset show our approach can improve the performance of news recommendation and outperform many baseline methods.

Acknowledgments

The authors would like to thank Microsoft News for providing technical support and data in the experiments, and Jiun-Hung Chen (Microsoft News) and Ying Qiao (Microsoft News) for their support and discussions. This work was supported by the National Key Research and Development Program of China under Grant number 2018YFC1604002, the National Natural Science Foundation of China under Grant numbers U1836204, U1705261, U1636113, U1536201, and U1536207, and the Tsinghua University Initiative Scientific Research Program under Grant number 20181080368.