Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning

1 Introduction

Document classification is a fundamental task in Natural Language Processing (NLP). The goal is to assign proper class labels to documents. It is essential in many real world applications, such as sentiment analysis [Maas et al., 2011], topic labeling [Wang and Manning, 2012] and financial analysis [Luo et al., 2018].

Recently deep neural networks have shown great success in learning distributed representation of texts. Given a varilength text, they represent it as a fixed-length vector using different approaches, such as Convolutional Neural Network (CNN) [Wang et al., 2017], Recurrent Neural Network (RNN) [Sutskever et al., 2014] and syntactic composition models [Socher et al., 2012]. There are also some studies utilizing deep neural networks for document classification [Yang et al., 2016]. Since deep learning models always contain huge number of parameters, they require a large volume of labeled corpus to learn a good document representation. However, in many situations, it is difficult to construct large training sets because acquiring manually labeled documents is very expensive. In this case, the performance of such models would be limited due to the lack of training data.

Multi-Task Learning (MTL) [Caruana, 1997] can jointly train several tasks with the goal of mutual benefits. It is inattention weights merely from a document itself, we design a global attention layer which can learn the attention weights from other tasks. This idea comes from the observation that some semantic patterns are very rare in one task. In this case, the intra-attention mechanism would fail to recognize such patterns. But if similar semantic patterns appear in other tasks, we can recognize them and make the correct predication with the help of multi-task learning techniques. An example is shown in Figure 1. Case 1 to 4 are Amazon product reviews coming from different domains. For the Magazine review "There's really great fashion coverage that is not at all cookie-cutter and formulaic like some of the other women's books. " in Case 1, it is essential to understand the informative word cookie-cutter for sentiment classification. However, it is hard for intra-attention based model to capture this kind of expression since it is rare in this corpus. Nevertheless, as this kind of expression is common in datasets related to movies such as Video and IMDB as shown in Case 2 to 4, our inter-attention based model could take advantage of inter-attention layer to recognize it and finally obtain the right prediction. Moreover, we also propose a global shared layer with another LSTM network to utilize the global information among all tasks to enhance each task-specific representation. We conduct a comprehensive evaluation on 15 real world datasets. Experimental results show that our model outperforms previous approaches by an obvious margin on most datasets.

Contributions of this paper are summarized as following:

• We propose MT-HIA, a hierarchical model with multitask learning for document classification. Compared with previous studies, our proposed model can take advantage of both inherent document structure and the common knowledge from multi-tasks.
• We introduce a hierarchical inter-attention approach to enhance the document representation of each single task. Moreover, we also devise a global sharing mechanism to utilize the knowledge frommultiple tasks.
• We conduct extensive experiments on 15 real world datasets. The results demonstrate the effectiveness of our proposed methods.

2 RelatedWork

2.1 Document Classification

Recently neural network models have been widely applied in the task of document classification. Wang et al. [2017] integrated information from knowledge bases into CNN models to improve the performance. Sutskever et al. [2014] and Liu et al. [2015] utilized variants of RNN to capture the sequence information in the texts. Socher et al. [2012] proposed Recursive Neural Network by leveraging the syntax information. Zhang et al. [2015] adopted a CNN model in character level for text classification. Yang et al. [2016] proposed HAN, a hierarchical model to learn the document structure and enrich the representation.

2.2 Sentiment Analysis

Sentiment analysis is an important task in text classification which focuses on inferring the sentiment polarity of the text.

Recently neural network has been widely applied in the task of sentiment analysis. Tang et al. [2015] proposed a neural network based framework which can make use of extra information, i.e., product and user to improve sentiment classification. Johnson et al. [2016] explored region embeddings via one-hot LSTM to improve the performance. McCann et al. [2017] introduced an approach for transferring knowledge from an encoder pretrained on machine translation to sentiment analysis.

2.3 Multi-Task Learning

Multi-task learning [Caruana, 1997] is an approach to learn multiple related tasks simultaneously, aiming at yielding performance gains by leveraging potential correlations and common features among related tasks. It has been widely adopted in many applications, such as speech recognition [Rao et al., 2018], computer vision [Zhang et al., 2012] and natural language processing [Sutskever et al., 2014]. Multi-task learning techniques have already been widely adopted in NLP tasks. For example, Collobert et al. [2008] utilized the word inputs across multiple tasks to improve POS tagging. Firat et al. [2016] adopted multi-task learning in the task of machine translation. Liu et al. [2019] proposed a graph based model to capture the relations between different tasks.

Recently there have been some studies using multi-task neural network for text classification. Liu et al. [2016a; 2016b] proposed a multi-task RNN model for text classification with 3 mechanisms for sharing information. Liu et al. [2017] further employed the adversarial training mechanism to reduce the noise from different tasks. Chen et al. [2018] improved the multi-task RNN models by extracting meta-knowledge from different tasks. These methods just focus on general text classification tasks and cannot make use of the hierarchical structure of documents.

3 Methodology

Figure 2 displays the overall architecture of our proposed model. Here, we take the case of two tasks as an example. It consists of two components: the task specific encoder and the global shared layer. The task specific encoder is a hierarchical network which involves multi-task driven inter-attention mechanism from multiple granularities, i.e. both in wordlevel and sentence-level, to enhance the representation of documents. The global shared layer enables multi-task learning: it adopts a global shared memory to learn the common knowledge from all tasks. Next we will first introduce the task specific encoder and then the global shared layer in details.

3.1 Task Specific Encoder: Hierarchical Inter-Attention Network

The task specific encoder is a hierarchical attention network shown in Figure 3. It consists of three components: a BiLSTM layer, a multi-task driven inter-attention layer and an output layer. The Bi-LSTM and attention layers are applied in both word and sentence levels. Next we will describe the details of different components.

Bi-LSTM Based Sequence Encoder

We adopt LSTM as the basic building block for sequence encoder. Given input xt and previous hidden state ht-1, the current hidden state ht can be updated by:

Here we choose the LSTM architecture devised in [J´ozefowicz et al., 2015] as our encoder and use the function LSTM(·, ·, ·) as a shorthand for the encoding process. And θ refers to all parameters of LSTM. To better utilize the contextual information, we use Bi-LSTM to learn the hidden states.

Based on Bi-LSTM, we construct the hierarchical model as following. Given a sentence i with words wit, t ∈ [0, T] (T is the number of words in the sentence), we first embed the words to low-dimensional vectors through an embedding matrix We, i.e. xit = Wewit. And then we construct the word-level encoder by feeding xit into a Bi-LSTM network to obtain the hidden state of each word. The final representation of the tth word in ith sentence is the concatenation of output in both directions:

In sentence level, we feed the output of word-level encoders [⃗s1,⃗s2, ...,⃗sL] (L is the number of sentences in the document) into the Bi-LSTM and concatenate -→hi and ←- hi , i ∈ [1, L] to get a representation of ith sentence hi in a similar way.

Multi-task Driven Inter-Attention

As different words and sentences make different contributions to the composition of a document, it is necessary to assign different importance to them. To reach this goal, we design an inter-attention mechanism driven by multi-task learning to select the informative words and sentences. The intuition is that as external knowledge, information from the global shared component can provide common and taskinvariant knowledge, which can help disambiguate the semantics of words and sentences in the document of each single task. Specifically, we utilize a multi-task based MultiLayer Perceptron (MLP), which is a global layer shared by all tasks, to help compute attention weights. And we then involve information from this layer into both word-level and sentence-level encoder to help assign weights for each word and sentence.

That is, we first feed the word vector xit into a Word-level Shared MLP layer (Word Shared MLP) which consists of a MLP to get the shared representation x(s) it for computing attention weights in Eq. 3. Then we get a hidden representation u(s) it of x(s) it containing common and task-invariant knowledge in Eq. 4. We measure the importance of tth word in ith sentence by an attention weight αit which can be computed as the inner product of u(s) it and aword level weight vector uw followed by a softmax layer in Eq. 5. The word level weight vector uw is randomly initialized and jointly learned during the training process. Finally, we obtain the sentence representation by computing the weighted sum of the word hidden states based on the weights in Eq. 6.

Similarly, in order to obtain a better document representation with selected information, we also adopt multi-task driven inter-attention mechanism and propose a Sentencelevel Shared MLP layer (Sentence Shared MLP). We first feed the sentence vector si into the Sentence Shared MLP to get the representation s(s) i . Then we can correspondingly get the attention weights and the document representation d following the same route with the word-level encoder.

Output Layer

Since the document vector d is the high level representation of the document, it can be then fed into a softmax layer for classification:

3.2 Global Shared Layer for Multi-Task Learning

Based on the task-specific model, we then describe our multitask learning based architecture.

The task of document level classification is to assign a label Y to a document X. Supposing there are K related tasks, we denote the corpus of the kth task with Nk samples as:

where X(k) i and Y (k) i are ith sample and its label in the kth task.

The key factor of multi-task learning is the sharing scheme among different tasks. Considering the hierarchical structure of documents, we design the global shared layer to determine the common features in both word-level and sentence-level. Figure 4 is the illustration of word-level shared layer. Specifically, for each word x(k) t at time step t in task k, we first compute its shared representation h(s) t by an LSTMnetwork:

And then we utilize the gating mechanism to control the portion of information flows from the global shared layer to each task:

where g(k) controls the portion of information flows from the word-level global shared layer to task k, based on the correlation strength between x(k) t and h(s) t at the current time step.

Similarly, in sentence-level encoder, there also exists a global shared LSTM layer (sentence shared LSTM) which captures the shared information for all the tasks and can be conveyed to each task. In this way, the hidden states and memory cells of each single task can benefit from extra information from all other tasks.

3.3 Training

The objective of training process is to minimize the crossentropy of the predicted and true distributions for all tasks. For a single task k, we adopt cross-entropy to compute its loss L( ˆ y(k) , y(k) ) as shown in Eq. 14 where Nk refers to the number of training samples and C is the class number. y(k) ij is the ground-truth label and ˆ y(k) ij is the predicted probability.

The overall loss of all tasks is shown in Eq. 13, where λk is the weight for the kth task and Θ is the set of all trainable parameters.

For multi-task learning, the labeled data for training each task can come from completely different datasets. Following the previous study [Collobert and Weston, 2008], the training process is conducted in a stochastic manner by looping over the tasks:

1. Select a random task.
2. Select a mini-batch of examples from this task.
3. Update the parameters for this task and the global shared layer with respect to this mini-batch.
4. Go to 1.

After the joint learning phase, we can use a fine tuning strategy to further optimize the performance for each task.

4 Evaluation

In this section, we introduce the empirical results of our MTHIA model. We use the accuracy of classification as the evaluation metrics.

4.1 Experiment Setup

Datasets

We evaluate the effectiveness of our model on 15 documentlevel corpus. The first 14 datasets are Amazon product reviews coming from different domains such as Books, Music, Baby, etc. These datasets are collected based on the dataset 1 provided by Blitzer et al. [2007]. The last IMDB dataset contains movie reviews with binary classes [Maas et al., 2011].

All these datasets are document-level in which each review consists of several sentences. And the goal is to assign the positive or negative label to each document. Following previous studies, we randomly split these datasets into training sets, development sets and testing sets with the proportion of 70%, 10% and 20% respectively. The detailed statistics about these datasets are displayed in Table 1. To enable multi-task learning, we jointly train all 15 datasets simultaneously.

Competitor Methods

We compare our model with several state-of-the-art methods, including single-task learning based models and multi-task learning based models. For single-task learning models, we compare with four methods:

• LSTM: the standard LSTM [Hochreiter and Schmidhuber, 1997].
• HyperLSTM: a model which uses a small network to generate weights for a larger network [Ha et al., 2016].

1 https://www.cs.jhu.edu/∼mdredze/datasets/sentiment/

• MetaLSTM: a model which uses a meta-network to capture the meta-knowledge of semantic composition [Chen et al., 2018].
• HAN: a hierarchical attention network for single task document classification [Yang et al., 2016].

For multi-task learning, we compare our model with five models.

• ASP-MTL: an adversarial MTL framework, alleviating the shared and private latent feature spaces from interfering with each other [Liu et al., 2017].
• Meta-MTL: a function-level sharing scheme for MTL with a shared meta-network proposed in [Chen et al., 2018].
• DA-MTL: a MTL framework with a shared sentence encoding layer and a dynamic task-attentive mechanism [Zheng et al., 2018].
• CG-MTL: a graph based MTL framework, which is the up-to-date method [Liu et al., 2019].
• MT-HA: a method proposed by ourselves. It is implemented by just replacing the inter-attention with intraattention mechanism.

Although there are some other related approaches, such as single-task ones [Tang et al., 2015; Zhang et al., 2015] and multi-task ones [Liu et al., 2016a; Liu et al., 2016b], previous studies have shown that they cannot outperform the above selected methods. Therefore, we do not report the results of them here due to the space limitation.

Hyper-Parameter Settings

We initialize the word embeddings with pre-trained GloVe vectors [Pennington et al., 2014]. For out of box words, we randomly initialize their embeddings from uniform distribution in (-0.01, 0.01). The dimension of word embeddings is 200. And the models are trained with backpropagation using Adam optimizer with mini-batch size 30. The detailed settings of hyper-parameters are shown in Table 2.

4.2 Analysis of Results

Table 3 shows the classification accuracies on 15 datasets. The column "Single Task" refers to the results of 4 single task models. The column of "Multiple Tasks" shows the results achieved by six multi-task models including ours. Besides MT-HIA, we also report the results obtained by MT-HA. It stands for the multi-task hierarchical intra-attention network which assigns the attention weights merely from the document itself like attention mechanism in HAN. From the results, we have the following observations. Firstly, the overall performance of multi-task learning based models is much better than single-task ones. This result is consistent with previous studies [Chen et al., 2018] and demonstrates the benefits of multi-task learning techniques. Among them our MT-HIA obtains the best performance. Specifically, it outperforms HAN, the most competitive single-task model by 4.3% on average. Secondly, our model also achieves a better performance than competitors in most cases for multi-task learning. For example, our method shows an average improvement of accuracy 1.8% to CG-MTL, the up-to-date multi-task model and 0.4% to DA-MTL, the most competitive multi-task model. Lastly, we can see that MT-HIA outperforms MT-HA in 13 out of 15 datasets. This demonstrates the advantage of our inter-attention mechanism over the intra-attention one.

Next we look at the specific performance gain of MT-HIA against single-task methods. The column "Single Avg" shows the average accuracy of 4 single-task based models. From this table, we can see that the performance of most tasks can be improved with a large margin with the help of MTL. And our MT-HIA is able to achieve the state-of-the-art performances on each single task in most cases, indicating the effectiveness and robustness of our model. Among all single-task learning methods, HAN performs best since it takes hierarchical structure of documents into consideration and proposes a hierarchical attention architecture. However, as the attention weights of HAN only comes from the document itself, it might be difficult for HAN to assign proper weights for infrequent semantic patterns. As our proposed model utilizes multi-task learning, each task can extract both common and task-invariant knowledge from global shared layers. And we adopts inter-attention mechanism, which can learn the attention weights from the global sharing information contributed by multiple tasks. Therefore, our MT-HIA is more powerful in discovering informative sentences and words in a document and thus achieves better results.

Finally, we make detailed analysis on the results. Compared with other MTL methods, our MT-HIA model can better capture the hierarchical structure of documents. Other latest multi-task learning methods, such as Meta-MTL and DA-MTL focus on how to make good use of the shared information from text corpus. However, they all ignore the hierarchical structure of documents and will definitely fail to capture sufficient signals for document classification. Thus their performance are not as good as ours. Among all baseline methods, DA-MTL performs the best. The reason is that it proposes a new scheme of information sharing for MTL: all tasks share the same sentence representation and each task can select the task-specific information from the shared representation with attention mechanism. Compared with DAMTL, our model can also reach the same goal with the help of inter-attention mechanism. As our method can also capture the hierarchical structure, it can clearly beat DA-MTL.

4.3 Case Study

In order to illustrate that our model is capable of better discovering informative sentences and words in a document, we visualize the weights of sentences and words in an hierarchical manner in Figure 5. In this example, we compare MT-HIA with inter attention mechanism with MT-HA that only assigns the attention weights merely from the document itself.

The red color refers to the attention weights over sentences, blue denotes attention weights over words and the depth of color represents the size of the weights. From Figure 5, we can see that for the negative review, both models can select the words carrying strong sentiment like "attractive", "disappointed". However the intra-attention mechanism fails to capture the semantic patten "although ... I was..." since similar patterns are rare in this corpus. Therefore, it puts more attention on the fourth sentence and gives a wrong prediction. Nevertheless, as our MT-HIA is able to take advantage of common knowledge from other tasks, it will be easier to capture this pattern once it appears in other tasks. As a result, MT-HIA successfully captures this kind of pattern and chooses to ignore the fourth sentence and finally gets the right prediction.

5 Conclusion

In this paper, we propose the Multi-Task Hierarchical InterAttention Network model for document classification. We improve the task-specific document representation by proposing an inter-attention mechanism. We further devise a global shared mechanism to smartly utilize the knowledge shared by multiple tasks. Experimental results on 15 real world datasets show that our proposed model outperforms state-of-the-art methods by a substantial margin. For the future work, we would like to explore our model on more different types of NLP tasks.

Acknowledgements

This work was supported by NSFC(91646202), National Key R&D Program of China(SQ2018YFB140235).