Beyond Polarity: Interpretable Financial Sentiment Analysis with Hierarchical Query-driven Attention

1 Introduction

Sentiment analysis has been widely applied in financial applications since Robert Engle [Engle and Ng, 1993] suggested the asymmetric and affective impact of news on volatility. In recent years, researchers exploited various text resources e.g., news, microblogs, reviews, disclosures of companies to analyze the effects on markets in multifarious manners: impacting on price trends [Kazemian et al., 2016], volume of trade [Engelberg and Parsons, 2011], volatilities [Rekabsaz et al., 2017] and even potential risks [Nopp and Hanbury, 2015].

As a consequence, sentiment analysis in financial economics is considered different to that in user-product scenario [Liu, 2012; Ma et al., 2017] of opinion mining. In particular, financial sentiment analysis aims to identify market confidence indicators of text from proxies, which are derived from human judgments or movements of the market. For example, a financial retrospective that contains positive sentiment is optimistic about the future financial prospects of a company. A news article is regarded to have negative sentiment if it reveals scandal of executives of a company since it increases the risks in the future of that company. The "Enron scandal" could be a more specific example 1 . Compared with users' reviews on products, finance related documents have these characteristics: verbose, redundant, and written in a style that makes them complex to process. Thus, it calls for specified framework to capture key information from financial documents.

Moreover, it is a challenging task to analyze sentiment in financial documents since the implicit sentiment in financial domain is derived from miscellaneous perspectives such as macroeconomic information, microstructure factors, eventoriented, company-specific. The sentiment derived from different perspectives could be rather different even if the representations are under similar lexicons and language structures. For example, in Figure 1 the word "rise" indicates a positive sentiment in the movement of prices which belongs to microstructure factors in Case 1. However, in Case 2, the same word contributes much less than the previous case as the phrase "non-performing loan ratio" dominates the negative sentiment from the event-oriented viewpoint.

Traditional approaches for financial sentiment analysis fail to model the complex structure of representations of financial texts very well because they are mainly based on domainspecific lexicons [Tsai and Wang, 2014; Nopp and Hanbury,

1 https://en.wikipedia.org/wiki/Enron scandal

Case 1. Since the end of Aug, the stock price has risen about 30%

Positive

Case 2. The non-performing loan ratio has risen sharply to 2.35% Negative

Figure 1: An example of sentences in financial articles.

2015] or complicated feature engineering [Devitt and Ahmad, 2007; Schumaker and Chen, 2009; Rekabsaz et al., 2017]. Recently, the deep neural network based methods [Ding et al., 2015; Akhtar et al., 2017] have also been applied in this field so as to learn the representations of documents. However, they still suffer from the defects in the following aspects:

Firstly, most of existing approaches only provide plausible polarity identifications (e.g. positive, negative or neutral). But in the practical applications, when people make financial decisions, the result of sentiment analysis is only one of the building blocks. In order to obtain a comprehensive analysis, explainable justifications rather than polarity labels are required from the model. Although deep learning based methods can do a good job in polarity prediction, they produce few clues of explanation.

Secondly, the sentiment of a same financial document can be varied for tasks from different departments. For example, an aggregated news bulletins of listed companies may be negative for company A while positive for company B and neutral for the market. It makes the overall polarity identification from document-level inapplicable for real-world application. Meanwhile, different users may concern about diverse aspects even for the same task. For instance, an accounting department will pay more attention to financing risks, e.g. debts, cash flow, etc., while a legal department will focus on legal risks, e.g. litigation, illegality, etc. Even if they simultaneously perform risk predictions for quoted companies, they may expect different results from the same document. Hence it is necessary to take diversity and personality of users' preference into account for financial sentiment analysis.

To address above problems, in this paper we propose an interpretable framework FISHQA (FInancial Sentiment analysis network with Hierarchical Query-driven Attention) for financial sentiment analysis. First, we devise a hierarchical network structure to model the documents from multiple granularities: we first build representations of sentences from individual words and then aggregate those into document representations. Second, we equip FISHQA with a novel querydriven attention mechanism to meet the requirements from different aspects on one document. This mechanism is based on the idea that different parts of a document have varied weights in deciding the sentiment. To improve the flexibility of our model, we assign such weights according to the provided queries. Here the queries are natural language specified by the financial analysts from various departments. In this way, they can specify analysis results by deploying diverse queries for different purposes or tasks. With such mechanism, With such mechanism, our model is able to discover spotlights beyond the polarity of a document according to different users' requirements. Thus, compared with previous studies, our proposed method is more flexible and explainable. We conduct experiments on a real-world dataset related to bond default risk. The results demonstrate that FISHQA

significantly outperforms the compared methods in polarity identifications as well as produce meaningful evidences for the prediction results. It can also pick out better representative information of the whole documents.

The remainder of this paper is organized as follows. Section 2 introduces the related work. We present the proposed FISHQA in Section 3. Experimental results are discussed in Section 4, the paper is concluded in Section 5.

2 RelatedWork

Evaluating sentiments in financial articles. There are basically two categories of evaluating sentiments in financial related articles.

First, the sentiment indicator is from proxies of financial subjects that can be derived or computed from the market, e.g. prices, volume of trade, volatilities and so on. [Ding et al., 2015] proposed a neural network based framework to predict the stock price by measuring sentiment of events from financial news. [Nguyen and Shirai, 2015] predicted stock price movement by simultaneously analyzing topics and sentiments of social media. [Kazemian et al., 2016] made some in-depth analysis on how to evaluate specific tasks of financial sentiment analysis. Their conclusion is consistent with our motivation. The intrinsic sentiments of financial documents can be diverse for different tasks, and even for the same task, they might be various among distinct perspectives.

Our work belongs to the second category, in which the sentiment is indicated by human specifications [Devitt and Ahmad, 2007; Akhtar et al., 2017; Tetlock, 2007]. Here a sentiment analyzer is typically regarded as a potentially useful component to support either financial decision making tasks or specialized tasks where the indicators are difficult to be quantified, such as risk exposure management, bond rating. Lexicon-based and feature engineering approaches dominate the existing researches, which derives the incapable of modeling the semantic representations of text. Recently, [Akhtar et al., 2017] proposed an ensemble learning approach and achieved state-of-the-art performance on fine-grained financial sentiment analysis. However, they are not able to produce explainable clues for either prediction or users' preferences.

Sentiment analysis based on neural network. [Kim, 2014] was the first to explore CNN for sentence-level text categorization tasks including sentiment classification. [Johnson and Zhang, 2017] utilized very deep word-level CNNs to capture global representations of texts. Different from those word-level models, [Zhang et al., 2015] proposed a character-level CNN that achieved competitive results. [Tang et al., 2015] performed document-level sentiment classification by leveraging user and product information.

The attention mechanism was firstly utilized in machine translation [Bahdanau et al., 2015]. It then was adapted to both document-level [Yang et al., 2016] and aspect-level [Ma et al., 2017; Wang et al., 2016] sentiment classification on user-product comments. Compared with them, our framework adopts more flexible attention mechanisms which are interactive with users and adjustable for different tasks.

Recently, memory network [Sukhbaatar et al., 2015] was utilized to facilitate aspect-level sentiment analysis of userproduct comments [Chen et al., 2017; Tang et al., 2016]. However, their methods cannot be adopted to our problem because it is difficult to find such a high-level abstract of "aspects" in financial domain. Actually, how to define a broadly accepted aspect in financial sentiment analysis still remains an open problem.

3 The FISHQAModel

In this section, we introduce our proposed model. The overall architecture of FISHQA is shown in Figure 2. It adopts a hierarchical structure that involves attention mechanism from multiple granularities: both in word level and sentence level.

In the real-world applications, analysts usually care about different aspects of financial articles for different purposes. For instance, when we concern more about uncertainty in personnel, "suicide" is more important than "decrease" though "decrease" may be usually informative to financial sentiment analysis. Hence, we may expect the words like "executives change" and "layoff" to be highly-weighted. Motivated by such observation, we devise the query-driven attention mechanism to highlight such preference-driven words in a sentence (and preference-driven sentences in document) by assigning higher attention weights to them. Next, we introduce the details of the FISHQA model and it consists of three layers: embedding layer, representation layer and output layer.

3.1 Embedding Layer

We represent a document with n sentences as d = {s1, s2, ..., sn} ∈ D, where D is the document set and st is the embedded representation. In the document, the tth sentence consists of l words {wt,1,wt,2, ...,wt,l} ∈ W, where t ∈ [1, . . . , n] and W is the set of word vocabulary. Each word wt,i is embedded into a real-value wordvector xt,i ∈ RD from embedding matrix Ma ∈ RV×D , where D is the dimension of word vectors and V is vocabulary size. We set two sets of queries for word level and sentence level. We denote the i-th query for word level as {w′i,1 , w′i,2 , ..., w′i,l } ∈ W, where i ∈ [1, . . . , k], and we embed it into a vector qi ∈ RD via the same matrix Ma. We generate the embedding vector of j-th query pj for sentence level in the same way and j ∈ [1, . . . ,m]. Here k and m represent the query numbers for word and sentence level, respectively. We represent {q1, q2, ..., qk} and {p1, p2, ..., pm} as two sets of embedded queries.

3.2 Sequence Encoder

In our model, we adopt Gated Recurrent Unit (GRU) [Bahdanau et al., 2015] as the basic building block for sequence encoder. At step t, given input xt and previous hidden state ht-1, the current hidden state ht can be updated by,

where rt represents the reset gate while zt is the update gate. σ is the sigmoid activation function, ⊙ stands for elementwise product. Ur, Uz, Uh ∈ RK×D , Wr,Wz,Wh ∈ RK×K , br, bz, bh ∈ RK×1 are parameters to be learned. Here K is the dimension of hidden states, and D is the input size. We argue that the sequence encoder in FISHQA is not limited by GRU, and other variations of RNN can also be utilized.

3.3 Hierarchical Query-driven Attention

Given a word wt,i from the t-th sentence, we embed it into a real-value vector xt,i with:

Then we start from the word embedding and assign attention weights to enhance the representation from different levels.

Word-level Query-driven Attention

To better utilize the contextual information, we use bidirectional GRU to learn the hidden states, containing the forward representation -→f and backward ←-f respectively,

where ht,i concatenates hidden states of the i-th word in the t-th sentence from both directions.

In our model, we adopt query-driven attention mechanism to select related words. Different queries are employed to consider various aspects of financial news. Following the previous work [Sukhbaatar et al., 2015], for the i-th query {w′i,1 , w′i,2 , ..., w′i,l }, we represent it with the vector qi:

Then we construct the representation of st as:

where ht = (ht,1, ht,2, ..., ht,l)T denotes the combination of all the hidden states of the words in the t-th sentence, ut,i refers to the hidden representation of state ht through a fullyconnected layer corresponding to query qi as shown in Eq. 7.

Each query is answered by selecting informative words in each sentence, that is, we aim to allocate high weights for related words. Given a query qi, we measure the importance of words in sentence st by an attention weight vector at,i, which can be computed as the inner product of query qi and ut,i followed by a softmax layer in Eq. 8. ˜at stands for the average of all the attention weight vectors (Eq. 9). We finally construct the t-th sentence st with the sum of hidden states, weighted by ˜at in Eq. 9.

Sentence-Level Query-driven Attention

In sentence level, we learn the hidden states to form a document vector in the same way as the word-level. In particular,

where hst represents concatenated hidden states of GRU network from both directions in sentence st.

In sentence level, we also use query-driven attention mechanism to extract important sentences as well as form a better document representation with selected information.

where h = (hs1 , hs2 , ..., hsn )T stands for the combination of all the hidden states of sentences, pj is the embedding vector of the j-th query raised for sentence level, uj is the hidden representation of state h via a fully-connected layer for query pj, and aj = (aj,1, aj,2, ..., aj,n)T represents the attention weight vector corresponding to pj .

Similar to the word-level query-driven attention, each query is answered by selecting informative sentences in the document. Given a query pj, we measure the importance of each sentence by the attention vector aj, which is computed by the inner product of query pj and uj (Eq. 11), and allocate higher weights to related sentences. After averaging the attention weights for all the queries in sentence level (Eq.12), we can construct an overall representation of document v by computing the weighted sum of hidden state h (Eq. 12).

3.4 Output Layer

In the output layer, we feed the representative vector of the document into a fully-connected layer and predict the sentiment polarity. Here we use the softmax mechanism to get the probability distribution of the sentiment labels. The training objective is to minimize the cross-entropy loss as following:

4 Experiments

4.1 Data Description

Our dataset combines a collection of 30, 000 documents, which were extracted from various Chinese mainstream financial websites over 30 days, ranging fromMay 26 to June 25, 2017. Among them, 7, 648 documents were annotated by three domain experts in the perspective that whether the corresponding bonds of the companies mentioned in the document will encounter the risk of default in the future. If the judgment is yes, the document is labeled as negative, otherwise as non-negative. Such manually labeled documents form our experimental set. The details are shown in Table 1.

4.2 ComparedMethods

We compare our model with several baselines including feature-based methods and neural network based state-ofthe-art approaches. Lexicon-based methods are not included in our experiments because few available sentiment lexicon suits our task and dataset. In addition, some previous work already demonstrated that deep learning-based methods outperforms [Yang et al., 2016] lexicon-based methods.

1. SVM+BoW adopts SVM classifier using bag-of-word with frequency of each word as features.
2. SVM+BoW TFIDF adopts SVM classifier with TF-IDF of words as features under bag-of-word representation.
3. CNN-word is an available implementation 2 of the approach in [Kim, 2014].
4. Bi-LSTM learns document representation via a bidirectional LSTM network [Graves and Schmidhuber, 2005] by averaging all the hidden states of each word.
5. LSTM-GRNN was proposed by [Tang et al., 2015] with hierarchical network structures.
6. HAN reported in [Yang et al., 2016] adopts hierarchical networks with randomly initialized attention mechanism.
7. FISHQA-Q1 represents FISHQA with a set of userspecific queries Q1, which are raised by domain experts from the view of financial-related risk. We will detail the query construction later.
8. FISHQA-Q2 is FISHQA with another user-specific query set Q2 representing general focuses of financial documents.

2 https://github.com/dennybritz/cnn-text-classification-tf

4.3 Query Construction

The queries in FISHQA are manually created. Either sentences (e.g. "is there any non-performing loan?") or lists of concerned words (e.g. "arrears, debt, indebtedness") can be utilized as the queries.

Two groups of queries, namely Q1 and Q2, are compared in our experiments. As shown in Table 2, each query set contains three detailed queries and we denote them as q1, q2 and q3 hereafter. The maximum and average query length are 45 and 20, respectively. The main difference between Q1 and Q2 is that Q1 consists of customized risk-oriented queries from user-specific perspectives while Q2 contains more general queries in financial domain. Due to the space limitation, we here show two small query sets as running examples. In real-world applications, however, we can construct question sets from different aspects upon users' needs and add any number of queries into each set. While we notice that our model also supports different query sets, in our experiments, we adopt the same query set for both word-level and sentence-level attention to magnify the effects of queries.

4.4 Experimental Settings

We perform the following preprocessing on the experimental dataset. First, we remove inserted advertising of each article. Then we use public NLP tools to split documents into sentences and tokenize each sentence into words with an extra financial exclusive lexicon. The lexicon mainly contains the names of listed-companies and the listed-bond names.

We set the dimension of word embedding as 200 and hidden size of GRU as 100. We optimize the training process using Adam [Kinga and Adam, 2015] with a mini-batch of size 64 and a learning rate 0.001. The number of words for each sentence and sentences for each document are set to be 45 and 30 by grid search, respectively. The word embeddings are trained with the models simultaneously, and all the other parameters of compared methods are fine-tuned to achieve the best results.

4.5 Experimental Results

We perform 5-fold cross-validation on the experimental dataset for all the methods. Table 3 reports the average accuracy and F1-score of the compared methods on the five folds. We observe that FISHQA-Q1 reaches an accuracy of 94.46% and an F1-score of 94.49%, which outperforms HAN, the most competitive method in baselines, by 2.69% and 2.83%, respectively. The results of FISHQA-Q2 also show similar trends. HAN performs better than other baselines since it picks out important information with the embedded hierarchical attentions. However, compared with our methods, HAN assigns the attention weights merely from the document itself. While FISHQA obtains the information of attention from queries provided by users, it can better decide the weights of words and sentences according to users' preferences. For instance, since the queries in Q1 are related to potential risks of financial issues, FISHQA-Q1 learns better document representations for predicting polarities of the task and thus achieves the best results.

Next, we evaluate the effectiveness of FISHQA on discovering informative sentences for a document d. We design a quantified measure as follows. For a document d, we choose the top K representative sentences by the attention weights produced by the network, then we concatenate the extracted sentences as a new document d′ and feed it into the trained model to get sentiment prediction. Such derived prediction accuracy is denoted as Acc-reduced. Specifically, for FISHQA, we select the top K sentences by concatenating the top Ki weighted sentences associated to every query. It's worth mentioning that the number of selected sentences Ki is allowed to be different when K cannot be divisible by m. For HAN, we directly extract top K highly-weighted sentences.

The fourth column of Table 3 demonstrates the average results on 5-fold cross-validation when K=2. We observe that FISHQA-Q1 is much better at picking out more representative sentences with the domain-specific query-driven attentions. In more detail, FISHQA-Q1 reaches 86.65% in Acc-reduced, and outperforms HAN by 15.63%. Though FISHQA-Q2 takes a more general query set, it still has clear advantages by achieving 83.80% in Acc-reduced and outperforming HAN by 12.78%.

We further demonstrate the performances of FISHQA-Q1, FISHQA-Q2 and HAN on test sets at every 100 training steps as Figure 3. It shows that all the results are stable at around 0.9 after 300 steps. FISHQA-Q1 starts to beat the others from step 400 while FISHQA-Q2 begins to perform better than HAN from step 500. For the measure of Acc-reduced, both FISHQAs outperform HAN at almost each demonstrated step and gradually enlarge the gap. Moreover, it presents obvious and stable upward tendency for FISHQAs while HAN takes on fluctuation. These observations confirm that the proposed model can stably select better informative sentences to represent original documents.

We conduct another experiment to test the performances on Acc-reduced by varying the number of selected sentences K. Additionally, we compare the contributions of highly weighted sentences with randomly selected ones for both FISHQA and HAN. In Figure 4, we name the average accuracy on Acc-reduced of FISHQA-Q1 as FISHQA-Reduced and that of randomly selected sentences as FISHQARandom. The same notation also works for HAN. For each method, we conduct 50 times on each K, where K=[1, . . . , 10], and we visualize the average trends as well as the standard deviation, which reflects the distributions of Acc-reduced on the 50 rounds, in Figure 4. The results indicate that:

First, with the growth of the number of selected sentences, Acc-reduced of FISHQA takes on a stable trend of rising and a small variance. Additionally, the results are comparable with the accuracy on the full documents when selecting more than 5 sentences. Yet Acc-reduced of HAN shows fluctuated rising and larger variance. Because the uncertainty of randomly initialized vector used by HAN determines that it cannot always find the representative information.

Second, the selected highly-weighted sentences of both FISHQA and HAN perform better than randomly chosen ones. Meanwhile, FISHQA-Random takes on a trend of rising with a smaller variance and higher accuracy than HANRandom at every K. We conjecture the reason is that querydriven attention mechanism can deal with broader information as various queries are able to learn multiple aspects of a document. Hence, FISHQA can obtain more effective information even from randomly selected sentences.

4.6 Case Study on Interpretability

Finally, we compare the results of the attention mechanisms of FISHQA-Q1, FISHQA-Q2 and HAN by visualizing the weights of sentences and words in the same documents as shown in Table 4 and 5. For each table, the depth of color for each column aside illustrates the sentence-level distributions of attention weights related to each query. The colored words represent highly-weighted ones through wordlevel query-driven attention mechanism for the model in the left column, while the underlined words are for the model in the right column. For both word and sentence, the darker the color is, the higher attention weight it has.

First, we compare the differences of attention mechanisms between FISHQA-Q1 and HAN in Table 4. For example, the first and third sentence have high attention weights for query q2 of FISHQA-Q1. They all tell information about "bribery cases", which are clearly related to personnel perspective. While for HAN, its attention weights on sentences are evenly distributed. HAN tends to focus on sentences containing emotive words, such as "worries" and "suspended". Yet the financial sentiments are task-specific and more than only subjective emotive languages. On the other hand, unlike our method, HAN cannot provide any explainable clues according to user's preference.

Second, we visualize the attention results of FISHQA with query set Q1 and Q2 as Table 5. There are huge differences among the sentences and words that are paid attention to. Recall that the queries in Q1 are customized risk-oriented ones, while those in Q2 are general concepts in financial domain. The FISHQA equipped with Q1 pays more attention to sentences containing words like "debt" (replying to financing, i.e. q1 ∈ Q1), "suicide attempt" (replying to personnel, i.e. q2 ∈ Q1), and "investigation" (replying to litigation, i.e. q3 ∈ Q1). However, the model with Q2 pays more attention to company names, e.g. "Liren Group", and title of executives, e.g. "CEO" and "chairman", which are informative indicators to the viewpoints that user concerns. Such results demonstrate that our FISHQA model can effectively find the important parts within a document according to users' queries.

5 Conclusion

In this paper, we proposed an attention driven model FISHQA to conduct document-level financial sentiment analysis. We design a hierarchical structure that learns the representation of document from multiple granularities. We also propose a query-driven attention mechanism in different levels to discover spotlights from texts according to user-specified queries and provide explainable results. Experimental results on a real-world financial dataset demonstrate the effectiveness of FISHQA. It also suggests significant superiority in selecting informative and explainable information.

Acknowledgements

This research is supported by National key R&D program of China (No.2017YFB1002104), the National Natural Science Foundation of China (No.61602438, 91546122, 61573335), and Guangdong provincial science and technology plan projects (No. 2015B010109005).