Hashtag Recommendation Using Attention-Based Convolutional Neural Network

Introduction

On social networks and microblogging services, users usually use hashtags, which are types of labels or metadata tags, to make it easier to find messages with a specific theme or content. In most microblogging services, users can place the hash character # in front of words or unspaced phrases to create and use hashtags. Hashtags can occur anywhere in a microblog: at the beginning, middle, or end. Moreover, along with the increase in social media services, microblogs have also been widely used as data sources for public opinion analyses [Bermingham and Smeaton, 2010; Jiang et al., ], prediction [Asur and Huberman, 2010; Bollen et al., 2011], reputation management [Pang and Lee, 2008; Otsuka et al., 2012], and many other applications [Sakaki et al., 2010; Becker et al., 2010; Guy et al., 2010; 2013]. Many approaches for various applications have also demonstrated the usefulness of hashtags, including microblog retrieval [Efron, 2010], query expansion [A.Bandyopadhyay et al., 2011], and sentiment analysis [Davidov et al., 2010; Wang et al., 2011].

However, only a portion of microblogs contain hashtags created by their authors. Hence, the task of automatically recommending hashtags for microblogs has received considerable attention in recent years. Existing works have studied discriminative models with various kinds of features and models [Heymann et al., 2008; Liu et al., ], collaborative filtering [Kywe et al., 2012], and generative models [Krestel et al., 2009; Ding et al., 2013; Godin et al., ] based on textual and social information. Most of these methods are commonly based on lexical level features, including the bagof-words (BoW) model and exquisitely designed patterns. In addition to these methods, the effectiveness of word trigger assumption [Liu et al., ; Ding et al., 2013] has also been demonstrated. This means that substance of a given sentence can be realized by some important words in it.

In recent years, the rapid development of deep neural networks with word embedding has made it possible to perform various NLP tasks and achieve remarkable results. Among these methods, convolutional neural networks (CNNs)[LeCun et al., 1998], which were originally invented for computer vision, have shown their effectiveness for various NLP tasks, including semantic parsing [Yih et al., 2014], machine translation [Meng et al., ], sentence modeling [Kalchbrenner et al., ], and a variety of traditional NLP tasks [Dos Santos et al., 2015; Chen et al., ]. Instead of building hand-crafted features, these methods utilize layers with convolving filters that are applied on top of pre-trained word embeddings. Moreover, compared to standard feedforward neural networks, CNNs have far fewer parameters and thus are easier to train.

Inspired by the advantages of CNNs in processing NLP tasks, we propose to use it to solve the hashtag recommendation task. Previous methods for hashtag and keyphrase recommendation [Liu et al., ; Ding et al., 2013] have also demonstrated the effectiveness of the trigger word mechanism, for example, in the tweet "#ipad #iphone How to share calendar events on iPhone and iPad", the trigger words are iPhone and iPad. However, standard CNNs cannot handle it. Motivated by the attention mechanism, which has been used in speech recognition [Chorowski et al., 2014] and machine translation [Luong et al., 2015], in this work, we introduce a novel CNN that incorporates an attention mechanism. The hashtag recommendation task is modelled as a classification problem. We employ a attention layer to produce a weight for a word with its surrounding context. Then, trigger words selected based on the attention layer and whole microblogs are transformed into fixed length vectors respectively. In the final step, a fully connected layer with softmax outputs is constructed. Through experiments using a dataset collected from real online services, we demonstrated the effectiveness of the CNNs and attention mechanism. The CNN method could achieve a performance that was better than those of stateof-the-arts methods. The proposed attention-based CNNs yielded additional improvement compared to standard CNNs.

The main contributions of this work can be summarized as follows:

• ÷ To take advantages of deep neural networks, we adopted CNNs to perform the hashtag recommendation task.
• ÷ To incorporate a trigger word mechanism, we proposed a novel attention-based CNN architecture, which incorporates a local attention channel and global channel.
• ÷ Experimental results using a dataset collected from a real microblogging service showed that the proposed method can achieve significantly better performance than the state-of-the-arts methods.

Related Works

Hashtag recommendation

Due to the requirements of hashtag recommendation, a variety of methods have been proposed from different perspectives [Zangerle et al., 2011; Ding et al., 2013; Sedhai and Sun, 2014; Gong et al., ]. Zangerle et al. [2011] introduced a similarity based method to achieve the task. Firstly, they tried to retrieve a set of hashtags used within these most similar messages. Then, heuristics ranking methods are used to select hashtags from these selected candidates. Ding et al. [2013] proposed to convert the hashtag recommendation task as a translation process. They assume that hashtag and the trigger words of the tweets are two different language and have the same meaning. They integrate topical based translation model to perform this task.

Most of the works mentioned above are based on textual information. Besides these methods, Zhang et al. [2014] observed that when picking hashtags users have different perspectives, which are impacted by their own interests or the global topic trend. To model these factors, they proposed a topical model based method to incorporate the temporal and personal information. Since may tweets contain not only textual information but also hyperlinks, Sedhai and Sun [2014] studied the problem of hashtag recommendation for hyperlinked tweets.

From the brief descriptions given above, we can observe that deep neural networks have not been implemented to perform hashtag suggestion tasks. In this work, we propose to use convolution network with attentional mechanism to achieve the hashtag recommendation task.

Attention Mechanism

In recent years, attention-based neural network architectures, which learn to focus their "attention" to specific parts of the input, have shown promising results on various tasks, such as speech recognition [Bahdanau et al., 2015b; Chorowski et al.,

2015], machine translation [Bahdanau et al., 2015a], visual object classification [Mnih et al., 2014], caption generation [Xu et al., 2015] and so on.

Bahdanau et al. [2015b] proposed an attention-based recurrent sequence generators for sequence prediction. They performed the method on speech recognition task at the character level. The attention mechanism scanned the input sequence and chooses relevant frames. Chorowski [2015] also used attention-based RNN on a phoneme recognition task. On image classification tasks, Mnih [2014] introduced a recurrent neural network model, which can select a sequence of regions or locations from an image or video. Only the selected regions were incorporated into further processing.

Motived from these successful usages in these tasks, in this work, we adopt attentional mechanism to scan input microblogs and select trigger words. We further combine both the selected words and the whole microblog together to achieve the task.

The Proposed Models

In this work, we formulate a hashtag recommendation task as a multi-class classification problem. Networks handle input microblogs of varying length. Each dimension of the output layer represents the probability of a hashtag recommended. As discussed in the introduction section, we also follow the trigger words assumption, which has been successfully used in previous studies [Liu et al., 2012; Ding et al., 2013]. Hence, the proposed model incorporates two channels: a local attention channel and global channel. Figure 1 shows the architecture of the proposed model.

In the global channel, all the words will be encoded, while the local attention channel will only encode a few trigger words. Common to these two types of channels is the fact that for each input microblog, both channels first operate the input. The goal is then to derive a feature vector that captures relevant information. However, these channels differ in how the feature vector is derived. The global channel has to attend to all the words for each tag, while the tag may only have relations with some trigger words. A local attention mechanism chooses to focus only on a small subset of the words for each tag which is depended on gate scores. Specifically, we employ a simple convolutional layer to combine the information from both vectors as follows:

where hg is the feature vector extracted from global channel, hl is the feature vector of the local attention channel. M is a filter matrix for the convolutional operation. b is a bias. Each filter will produce one feature, we use multiple filters to produce a feature vector.

In our model, we use CNN to encode a sentence for the global channel, whereas we construct a local attention network for the local attention channel. The parameters in the global and local attention channels are learned jointly with our final objective function instead of being trained separately.

Local Attention Channel

In the local attention channel, we consider the attention problem as a decision process. Given an input microblog m, we take the embeddings wi 2 Rd for each word in the microblog to obtain the first layer, where d is the dimension of the word vector. A microblog of length n is represented with w1:n which is the concatenation of words w1↪ w2↪ ▷▷▷↪ wn. In general, let wi:i+j refer to the concatenation of words wi↪ wi+1↪ ▷▷▷↪ wi+j .

After converting words into embeddings, the next step is the local attention layer. Given a threshold value ⌘ and an input microblog m, the attention layer generates a sequence of trigger words (wi↪ ▷▷▷↪wj). Each word is extracted from a small window. In general, given an attention window size h, we define Ml 2 Rh⇥d to be the parameter matrix. At the i-th step, the local attention layer generates the score of the i-th word in the microblog by focusing on the words in the window. The score of the central word in the window is obtained as follow:

where s(2i+h-1)◁2 is the score of the word w(2i+h-1)◁2. b is a bias, g is a non-linear function. We extract the words depend on their scores, if the score of the word greater than the threshold ⌘, it will be extracted as a trigger word. The operation has been defined as follows:

where ⌘ is the threshold, we let ⌘ to be a function of the length of the microblog. Although many functions are possible, we simply model the threshold as follows:

s is a sequence of scores for each word in the microblog. min¶s♦ is the minimum score of the words, and max¶s♦ is the maximum score. δ 2 [0↪ 1] is a parameter to balance the minimum and maximum.

The local attention layer makes it possible to extract the most important words in the microblog. We apply the local attention layer as the first layer in the attention channel which insure that the following layer will only operate on the trigger words.

After local attention layer, the trigger words extracted will input the folding layer which can operate on different number of words. The folding layer is to abstract the features of the trigger words by the following operation.

z = g(Ml ⇤ folding( bw) + b)↪ (5) where bw are the trigger words. Ml 2 Rd⇥r is the parameter matrix, d is the dimension of the word vector, r is dimension of the output vector. b 2 Rr is a bias, g is a non-linear function. folding is the sum operation for each dimension of all the trigger words, fi = Pj b wj↪i, where bwj↪i is the value in the ith position of the embedding of the jth trigger word.

The final output of the local attention channel is a fixedlength vector, which represents the embeddings of the trigger

words bw. Global Channel

In the global channel, we use a CNN architecture to model the whole microblog. In the convolutional layer, we use a filter which is a weight matrix Mg 2 Rl⇥d to produce a feature. l is the window size which means the filter operates on l words, d is the dimension of the word. For example, the feature zi generated from a filter operated on l words wi:i+l-1 can be calculated as follow:

where g is a non-linear function and b 2 R is a bias term. We operate this filter on all the combinations of the words in the microblog ¶w1:l↪w2:l+1↪ ▷▷▷↪wn-l+1:n♦ to produce a map of feature as follow:

In the pooling layer, a max-overtime pooling operation is applied over the feature map z to produce a feature for this filter. Using this operation we can extract the most important feature for each feature map. And it can deal with different microblog lengths.

From the process described above, we can see that one feature is extracted from one filter. To obtain multiple features, we use multiple filters with varying window sizes in the model. After max-overtime pooling operation, a bias and a non-linear function tanh are applied to the pooled matrix.

The final output of our global channel is also a fixedlength vector, which represents the embeddings of the input microblog m.

After the local attention channel and global channel, we use a convolutional layer with multiple feature maps to combine the outputs of the local attention channel and the global channel. We regard the output from the convolutional layer as the embedding of the microblog from our deep neural network.

Training

In this work, we joined learning the parameters ⇥ in local attention and global channels.

where W are words embeddings; Ml and Mg are the parameters of the local attention channel and global channel respectively; the rest parameters belong to the fully connected layer. Our training objective function is formulated as follows:

where D is the training corpus, a is the hashtag for microblog m.

To minimize the objective function, we use the robust learning method AdaDelta [Zeiler, 2012].

Hashtag Recommendation

We perform hashtag recommendation as follows. Suppose given an unlabelled dataset, we first train our model on training data, and save the model which has the best performance on the validate dataset. For the microblog of the unlabelled data, we will encode the microblog through the local attention channel and global channel by the saved model.

After the encoded processes, we combine the features generated from both channels. Then we get the scores of the hashtags for the dth microblog in unlabelled data by the fully connected layer:

where β are the parameters. h is the feature vector connected from the channels. A is a set of candidate hashtags.

According to the scores output from fully connected layer, we can rank the hashtags for each microblog and recommending the top-ranked hashtags to users.

Experiments

Dataset and Setup

We use the microblog dataset collected by Ding et al. [2013] for evaluation. There are 110,000 microblogs in the dataset which contain hashtags annotated by users. The vocabulary of words is 106,323 in the dataset, the vocabulary of hashtags is 37,224, and the average number of words and hashtags in each microblog is 20.45 and 1.20 respectively. The dataset has been splitted into training set (100,000 microblogs) and test set (10,000 microblogs). In our experiment, we randomly select 10% of the training set as the development set.

To evaluate the performance, we use Precision(P), Recall(R), and F-score(F1):

Nr is the right number recommended, Ns is the total number recommended by system, Nm is the total number of the hashtags assigned in the corpus.

The parameters in our model include both of parameters in global channel and local attention channel. For the global channel, after trying different single window sizes and multi window sizes, we empirically set filter windows to multi window 1, 2, 3, and we use 100 feature maps for each window size. For the local attention channel, we set the width of attention matrix to 5, and δ to 0.8. Both the global and local attention channel we use hyperbolic tangent as non-linear function and set mini-batch size to 200.

We trained the word vectors on 10 million words from Sina Weibo. The vectors have dimensionality of 100 and were trained using the architecture proposed in [Mikolov et al., 2013]. New words are initialized randomly.

Baseline

In this section, we compare our attention-based model with some baseline models and two degeneration models. We consider the following methods

• ÷ Naive Bayes (NB): We used NB to model the hashtag recommendation as a classification task. Given the textual of the microblogs, we can estimate the posterior probability of each hashtag.
• ÷ LDA: we used the LDA based method proposed in [Krestel et al., 2009] to recommend hashtags.
• ÷ Translation model (IBM-1): IBMmodel 1 is proposed by [Liu et al., ] which is directly applied to obtain the alignment probability between the word and the hashtag.

• ÷ TopicWA: TopicWA is a topical word alignment model proposed in [Ding et al., ]. In this method, the standard LDA is employed to discover the latent topic and combine the alignment method for this task.
• ÷ TTM: TTM was proposed by [Ding et al., 2013] for hashtag recommendation.
• ÷ CNN: CNN was proposed by [Kim, 2014] for sentence classification. We use the publish code to accomplish the hashtag recommendation task.
• ÷ Attention-5: Attention-5 was the local attention channel in our model with the window size equals to 5.

The first method NB is traditional method. LDA, IBM-1, TopicWA and TTM are the methods based on topic model and translation model which are attracted a lot of attention in recent years. CNN and Attention-5 are the variant methods of our method.

Results and Discussion

Table 1 shows comparisons of the results of the proposed method with those of the state-of-the-art discriminative and generative methods on the evaluation dataset. "CNN+Attention-5" denotes the method proposed in this paper. "CNN" and "Attention-5" represent the convolutional neural network and attention model, respectively. The results show that the proposed method is significantly better than the other methods. "TTM", "TopicWA", and "IBM1" are the models based on trigger words. From the results of "TTM", "TopicWA", "IBM1", and "NB", we can observe that the trigger words methods could improve the performance of the hashtag recommendation task. A comparison of the "CNN" and "TTM" results shows that "CNN" achieves a significantly better F1 than "TTM". The results demonstrate that the neural network can achieve better performance on this task. From the results of "Attention-5", "TTM", "TopicWA", and "IBM1", we can observe that our local attention channel can improve the performance of the trigger words for this task. From the results of CNN and CNN+Attention-5, we observe that the attention model can benefit the task, and the multiple channels can achieve better performance than a single channel. The relative improvement of the proposed CNN-Attention-5 over TTM is around 9.4% in the F1 score, which demonstrates the effectiveness of our method.

Figure 2 shows the precision, recall, and F1 curves of NB, LDA, IBM1, TopicWA, TTM, and CNN+Attention-5 on the test data. Each point of a curve represents the extraction of a different number of hashtags, ranging from 1 to 5. The curve that is the highest of the graphs indicates the best performance. Based on the results, we can observe that the performance of CNN+Attention-5 is the highest in all the curves. This indicates that the proposed method was significantly better than the other methods, and when we recommended the top 1 hashtag for each microblog, we obtained the highest F1 score.

Table 2 lists the static and non-static results. These results show that the non-static model could achieve better results for all the models. The non-static method tuned the word vector more specifically to the task-at-hand. For randomly initialized tokens which are not in the set of pre-trained vectors, we can learn more meaningful representations from fine-tuning.

Parameters sensitive analysis

From the description of the proposed model, we can know that there are several hyperparameters in the proposed model. To evaluate their impacts, we evaluated two crucial ones, window size l of the global channel and the width of the attention size h.

Table 3 lists the results of different window sizes for the global channel. We modeled the global channel on window sizes of 1, 2, and 3. To show the performance of the model with multiple window sizes, we use the window size with the the following combinations: (1,2), (1,2,3), and we obtained the best performance when the window sizes were (1,2,3). From the results of window sizes equal to 1, 2, or 3, we observe that the best window size is 2 on hashtag recommendation. The reason is that when the window size equals to 1, the convolutional operation will extract the unigram information while ignore the context information. The results of window sizes equal to 2 and 3 show that the bigram information is more important than the trigram for our task. From the results of multiple window sizes, we can observe that the multiple can achieve better performance. This demonstrated the advantages of the models with multiple window sizes over the single window size models. Comparing the results of window sizes equal to (1,2) and (1,2,3), we can observe that the performance between the multiple window sizes were similar, so we can choose the multiple window size conveniently.

Table 4 lists the comparisons of different attention sizes on the constructed evaluation dataset. "Attention-h" represent the methods that only considered the local attention channel. "CNN+Attention-h" represent the methods proposed with the multiple window sizes (1,2,3) in the global channel (h 2 ¶1↪ 3↪ 5↪ 7♦ is the attention size). Based on the results, we can see that the performance will increase with the number of words considered, and we will obtain the best performance when the window size is set to 5. "Attention-1" achieved a bad performance. Because in this model, the importance of the central word only depended on itself while ignoring the context. From the results of "Attention-5" and "Attention-7", we can observe that when the distance between the words and the central word greater than 2, we can ignore the influence of these words to the central word.

Conclusions

In this paper, we investigated a novel attention based CNNs for performing the hashtag recommendation task. We adopted the architecture of CNNs to avoid hand-craft features and take other advantages of CNNs. To incorporate a trigger word mechanism, we proposed a novel attention-based CNN architecture, which consists of a local attention channel and global channel. Experimental results on the data collected from a real world microblogging service demonstrated that the proposed method outperforms the methods which take only global or local information into consideration and stateof-the-art methods.

Acknowledgments

The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by National Natural Science Foundation of China (No. 61532011, 61473092, and 61472088), the National High Technology Research and Development Program of China (No. 2015AA015408).