``The Squawk Bot'': Joint Learning of Time Series and Text Data Modalities for Automated Financial Information Filtering

1 Introduction

Current multimodal analysis that combines time series with text data often focuses on extracting features from a text corpus and incorporating them into a forecasting model to enhance its predictive power [Ardia et al., 2019; Fraiberger et al., 2018]. At the same time, the ability to use text as a means of explanation [Akita et al., 2016; Weng et al., 2017] for the time series performance is an equally important requirement, as it helps to understand the possible reasons behind the patterns observed in the series. In many emerging financial applications such as "quantamental investing" [Wigglesworth, 2018], given a time series of an asset's performance, one often asks to retrieve a small set of text articles that can provide insights into the time series' performance. Also, investors do not solely base their decisions on historical prices when trading an asset, yet their decisions are often made with a careful consideration of macro-/micro-economic events and the most relevant news timely collected from the markets.

∗The name of this work is inspired by the popular financial commentary show called "Squawk Box" on CNBC (Squawk Box: Business, Politics, Investors and Traders, https://www.cnbc.com/ squawk-box-us , accessed April 28th, 2020)

However, with the increasing amount of available textual information nowadays [Schumaker et al., 2012], the challenge of finding the most relevant text articles associated with a particular asset series becomes more acute. From Figure 1, given recent m historical values of Apple's stock and today's n news (text) stories collected from the public media as two inputs, "Steve Jobs threatened patent suit" would be identified as one of the most relevant news articles associated with Apple. Certainly, for different stock time series, the set of relevant associated news documents would be different. One can find this important problem in other applications as well, such as cloud monitoring time series in combination with client support tickets and business performance metrics (e.g., sales) associated with the effectiveness of marketing campaigns, etc.

It is also worth mentioning that although filtering relevant text documents can rely on keywords (learning-to-match paradigm), we postulate that accurately identifying keywords for a specific time series is not an easy task, since it requires extensive domain knowledge and expertise. Moreover, for applications involving thousands of time series (e.g. stock markets), it is highly labor-intensive and error-prone in determining accurate keywords associated with each of them. More critically, the nature of both time series and textual contents is time-varying, filtering information based on fixed keywords would potentially lose essential and timely information. Hence an automated and data-driven system is highly desirable and becomes more relevant. In this work, we address the challenging problem of discovering relevant text articles associated with a numerical time series through developing a novel, multimodal, neural network that jointly learns both data modalities. The discovered articles are returned as a means of recommended documents for describing the current state of the time series. To the best of our knowledge, this is the first study that follows a deep learning approach to address this problem in

the financial domain.

We propose a novel neural model called Multi-Step Interrelation Network or MSIN (shown in Fig.1), which allows the incorporation of semantic information learnt in the text modality to every time step in the modeling of the time series. We argue that multiple data interrelations are important and compelling in order to discover the association between the categorical text and numerical time series. This is because not only are they of different data types, but also no prior knowledge is given on guiding the neural model to look at which text clues are relevant for a given time series. Hence, compared to existing multimodal approaches that learn two data modalities either in parallel or sequentially, our proposed MSIN network effectively leverages the mutual information impact between them through multiple steps of data convolution. During such process, MSIN gradually assigns large attention weights to "important" text clues, while ruling out less relevant ones, and finally converges to an attention distribution over the text articles that best aligns with the current state of the time series. MSIN also technically relaxes the strict alignment at the timestamp level between the two data modalities, allowing it to concurrently deal with two input data sequences of different lengths, which currently is beyond the capability of most recurrent neural networks [Cho et al., 2014; Chung et al., 2014].

We perform empirical analysis over large-scale financial news and stock prices datasets spanning over seven years. The model was trained on data of the first six years and evaluated on the last year. We show that MSIN achieves strong performance of 84.9% & 87.2% in recalling ground-truth relevant news for two specific time series of Apple (AAPL) and Google (GOOG) stock prices respectively, exhibiting superior performance compared to state-of-the-art, deep learning models relying on conventional attention mechanisms.

2 Learning Text Articles Representation

Our overall model is illustrated in Fig.1 which consists of (1) a Textual Module that learns to represent each input text article as a numerical vector so that it is comparable to the time series, (2) the MSIN (Multi-Step Interrelation Network) that takes as input both sequences of time series and of textual vectors to learn the association between them and hence to discover relevant text documents, (3) a Concatenation and Output layer that aggregates information from the two data modalities to output the next value in the time series. This section presents our first Textual Module. We start modeling from the level of words as their order within each text document is important in learning semantic dependencies. To this end, our network exploits the long-short term memory (LSTM) [Hochreiter and Schmidhuber, 1997] to learn word dependencies and aggregates them into a single representative vector for each individual text article.

Specifically, at each timestamp t, an input sample to the textual module involves a sequence of n text documents {doc1, . . . , docn} (e.g., n news articles released on day t-th). And it learns to output a sequence of n corresponding representative vectors {s1, . . . , sn} (t is omitted in sj's and docj's for simplicity). Each docj in turn is a sequence of words represented as docj = {xtxt 1 , . . . , xtxt K } (superscript txt denotes text modality). Each xtxt ℓ ∈ RV is a one-hot-vector of the ℓ-th word in docj, with V as the vocabulary size. We use an embedding layer to transform each xtxt ℓ into a lower dimensional dense vector eℓ ∈ Rdw via a transformation:

Often, dw ≪ V andWe can be trained from scratch; however, using or initializing it with pre-trained vectors from GloVe [Pennington et al., 2014] can render more stable results. In our examined datasets (Sec.4), we found that setting dw = 50 is sufficient given V = 5000. The sequence of embedded words {e1, e2, . . . , eK} for a document article is then fed into an LSTM that learns to produce their corresponding encoded contextual vector. The key components of an LSTM unit are the memory cell which preserves essential information of the input sequence through time, and the non-linear gating units that regulate the information flow in and out of the cell. At each step ℓ (corresponding to ℓ-th word) in the input sequence, LSTM takes in the embedding eℓ, its previous cell state ctxt ℓ-1 , and the previous output vector htxt ℓ-1 , to update the memory cell ctxt ℓ , and subsequently outputs the hidden representation htxt ℓ for eℓ. From this view, we can briefly represent LSTM as a recurrent function f as follows:

in which the memory cell ctxt ℓ is updated internally. Both ctxt ℓ , htxt ℓ ∈ Rdh with dh is the number of hidden neurons. Our implementation of LSTM closely follows the one presented in [Zaremba et al., 2014] with two extensions. First, to better exploit the semantic dependencies of ℓ-th word on its both preceding and following contexts, we implement two LSTMs respectively taking the sequence in the forward and backward directions (denoted by head arrows in Eq.(3)). This results in a bidirectional LSTM (BiLSTM):

The concatenated vector htxt ℓ leverages the context surrounding the ℓ-th word and hence better characterizes its semantic as compared to the embedding vector eℓ which ignores the local context in the input sequence. Second, we extend the model by exploiting the weighted mean pooling from all vectors {htxt 1 , htxt 2 , . . . , htxt K } to form the overall representation sj of the entire j-th document docj:

in which u ∈ R2dh ,Wβ ∈ R2dh×2dh are respectively referred to as the parameterized context vector and matrix whose values are jointly learnt with the BiLSTM. This pooling technique resembles the successful one presented in [Conneau et al., 2017] that learns multiple views over each input sequence. Our implementation yet simplifies them by adopting only a single view (u vector) with the assumption that each document (e.g., a news article) contains only one topic relevant to the time series. Note that, similar to CNNs [Kim, 2014], a max pooling can also be used in replacement for the mean pooling in defining sj . We, however, attempt to keep the module simple enough since the max function is non-smooth and generally requires more training data to learn. We apply our text module BiLSTM to every article collected at time period t and its output is a sequence of representative vectors {s1, s2, . . . , sn}. Each sj corresponds to one text document docj at input.

3 Multi-Step Interrelation Of DataModalities

Our next task is to model the time series taking immediately into account the information learnt from the textual data modality in order to discover a small set of relevant text articles well aligning with the current state of the time series. An approach of using a single alignment between the two data modalities (empirically evaluated in Section 4) generally does not work effectively since the text articles are highly general, contain noise, and especially are not step-wise synchronized with the time series. The number of look-back time points m in the time series can be much different from the number of text documents n collected at the time moment, i.e., n varies from time to time. To tackle these challenges, we propose the novel MSIN network that broadens the scope of recurrent neural networks so that it can handle concurrently two input sequences of different lengths. More profoundly, we develop in MSIN a neural mechanism that integrates information captured in the representative textual vectors learnt in the previous textual module to every step reasoning in the time series sequence. Doing so allows MSIN leverage the mutual information between the two data modalities through multi-steps of data interrelation.

Consequently, it gradually filters out irrelevant text articles while focuses only on those that correlate with the current patterns learnt from the time series, as it advances in the time series sequence. The chosen text articles are hence captured in the form of a probability mass attended on the input sequence of textual representative vectors.

In particular, inputs to MSIN at each timestamp t are two sequences: (1) a sequence of last m time steps in the time series modality {xts t-m, xts t-m-1 , . . . , xts t-1 } (superscript ts denotes the time series modality) 1 ; (2) a sequence of n text representative vectors learnt by the above textual module {s1, s2, . . . , sn}. Its outputs are the set of hidden state vectors (described below) and the probability mass vector pm outputted at the last state of the series sequence. The number of entries in pm equal to the number of text vectors at input. Its values are non-negative encoding which text documents relevant to the time series sequence. A large value at j-th entry of pm reveals a high relevance of docj to the time series.

The memory cell of our MSIN is illustrated in Fig.2. As observed, we augment the information flow within the cell by the information learnt in the text modality. To be concrete, MSIN starts with the initialization of the initial cell state cts 0 and hidden state hts 0 by using two separate single-layer neural networks applied on the average state of the text sequence:

where s = 1/n ∑j sj ; and Uc0 , Uh0 ∈ R2dh×ds , bc0 , bh0 ∈ Rds , with ds as the number of neural units. These are parameters jointly trained with our entire model.

MSIN then incorporates information learnt from the text articles into every step it performs modelling on the time series in a selective manner. Specifically, at each timestep ℓ in the time series sequence 2 , MSIN searches through the text representative vectors to assign higher probability mass to those that better align with the signal it discovers so far in the time series sequence, captured in the state vector hts ℓ-1 . In particular, the attention mass associated with each text representative vector sj is computed at ℓ-th timestep as follows:

where Wa ∈ Rds×ds , Ua ∈ R2dh×ds , ba ∈ Rds and va ∈ Rn . The parametric vector va is learnt to transform each alignment vector aℓ,j to a scalar and hence, by passing through the softmax function, pℓ is the probability mass distribution over the text representative sequence. We would like the information from these vectors, scaled proportionally by their probability mass, to immediately impact the learning process over the time series. This is made possible through generating a context vector vℓ:

1 We denote xts as vectors since an input time series can be multivariate in general (as illustrated in Fig. 1).

2 We re-use ℓ to index timestep in a time series sequence with similar meaning in Eq.(2)-(4). Yet, note here that: ℓ = 1, . . . ,m.

in which v0 is initialized as a zero vector. As observed, MSIN constructs the latest context vector vℓ as the average information between the current representation of relevant text article (1st term on the RHS of Eq.(9)) and the previous context vector vℓ-1. By induction, influence of context vectors in the early time steps is fading out as MSIN advances in the time series sequence. MSIN uses this aggregated vector to regulate the information flow to all its input, forget and output gates:

and the candidate cell state:

where U·x ∈ RD×ds ,U·h ∈ R2dh×ds ,U·v ∈ Rn×ds and b· ∈ Rds . Let ⊙ denote the Hadamard product, the current cell and hidden states are then updated in the following order:

By tightly integrating the information learnt in the textual modality to every step in modeling the time series, our network distributes burden of work in discovering relevant text articles throughout the course of the time series sequence. The selected relevant documents are also immediately exploited to better learn behaviors in the time series.

Concatenation and Output layer: Given representative vectors learnt from the text domain and the hidden state vectors from the time series, we use a concatenation layer to aggregate and pass them through an output dense layer. The entire model is trained with the output as the next value in the time series. As ablation study, we consider a variant of our model in which we exclude the multistep of interrelation between the two data modalities. A conventional LSTM is used to model the time series and subsequently align its last state with the representative textual vectors to find the information clues. This simplified model has fewer parameters yet the interaction between two data modalities is limited to only the last state of the time series sequence. We name it LSTMw/o abbreviated for LSTMwithout interaction.

4 Experiment

Datasets: We analyze dataset consisting of news headlines (text articles) daily collected from Thomson Reuters in seven consecutive years from 2006 to 2013 [Ding et al., 2014], and daily stock-price series of companies collected from Yahoo! Finance for the same time period. We construct each data sample from two data modalities as follows: (i) stock values in the last m = 5 days (one week trading) to form a time series sequence; (ii) all n news headlines (ordered by their released time) in the last 24 hours to form a sequence of text documents. A trained model hence aims at discovering top relevant news articles associated with the time series in a daily basis 3 .

3 To identify "top" relevant articles, we set 0.5 as a threshold of accumulated probability mass from pm, which is explained shortly.

#Top relevant news headlines - Precison

1

2

3

4

5

1

2

3

4

5

#Top relevant news headlines - Recall

Figure 3: (a) Precision and (b) Recall computed w.r.t. ground truth headlines annotated by Reuters for "Apple" stock time series.

Methods: We evaluate our model with each of the two company-specific stock time series: (1) Apple (AAPL), and (2) Google (GOOG). Each dataset (text and time series) is split into training, validation and test sets respectively to the year-periods of 2006-2011, 2012, and 2013. We use the validation set to tune model's parameters, while utilize the independent test set for a fair evaluation. As mentioned at the beginning, neither time series identity nor pre-specified keywords have been used to train the model. For baselines, we implement LSTMw/o as described above, the GRUtxt deep learning network based on [Yang et al., 2016]. In both models, their conventional attention mechanisms can reveal attentive documents. To emphasize the key contributions, we mainly discuss the performance of our model and these techniques on the important task of discovering relevant news articles associated to a given time series. For other predictive models, we further implemented LSTMpar [Akita et al., 2016] (analyzing two data modalities in parallel), CNNtxt [Kim, 2014] (using text modality), GRUts (using time series modality), SVM [Weng et al., 2017] (taking in both time series and unigrams of text news). A general observation is that models exploiting both data modalities tend to perform better than those utilizing a single data modality. For all models, we use 50-dim GloVe word embedding [Pennington et al., 2014], while other parameters are tuned from: neural units ds, dh ∈ {16, 32, 64, 128}, regularization L1, L2 ∈ {0.1, . . . , 0.001}, dropout rate ∈ {0.1, 0.2, 0.4}.

4.1 Relevant Text Articles Discovery

The Thomson Reuters corpus provides meta-data indicating whether a news article is reported about a specific company and we use such information as the ground truth relevant news, denoted by GTn. Just to clarify, such information is never used to train our model. Instead, we let MSIN learn itself the association between the textual news and the stock series via jointly analyzing the two data modalities concurrently and through multiple steps of data alignment. MSIN is hence completely data-driven and is straightforward to be applied to other corpus such as Bloomberg news source (or other applications like cloud business combined with client support tickets) where ground-truth articles are not available.

The set of GTn headlines allows us to compute the rate of discovering relevant news stories in association with a stock series through the precision and recall metrics. Higher ranking (based on attention mass vector pm as in MSIN's Eq.(8)) of these GTn headlines on top of each day signifies better performance of an examined model. Fig.3(a-b) and Fig.4(a-b) plot

#Top relevant news headlines - Precison

#Top relevant news headlines - Recall

Figure 4: (a) Precision and (b) Recall computed w.r.t. ground truth headlines annotated by Reuters for "Google".

these concrete evaluations over the AAPL and GOOG time series respectively, when we vary the number of returned daily top relevant headlines k between 1 and 5 (shown in x-axis). For example, at k = 5, MSIN achieves up to 84.9% and 87.2% in recall while retains the precision at 46.8% and 59.6% respectively to the GTn sets of AAPL and GOOG. Other settings of k also show that MSIN's performance is far better than those of two competitive models GRUtxt and LSTMw/o. Our novel neural technique of fusing two data modalities through multiplestep data interrelation allows MSIN to effectively discover the complex hidden correlation between the time-varying patterns in the time series and a small set of corresponding information clues in the categorical text data. Its precision and recall values significantly outperform those of LSTMw/o that utilizes only a single step of alignment between the two data modalities, and of GRUtxt which solely explores the text data with the attention mechanism in deep learning.

4.2 Explanation On Discovered Textual News

As concrete examples for qualitative evaluation, we show in Table 1 all news headlines from three specific examined days when our MSIN model is trained with the AAPL stock time series. We highlight the top relevant news headlines discovered by our MSIN model in green by setting the cumulative probability mass of attention ≥ 50%4 . As clearly seen on date 2013-01-22, MSIN assigns the highest attention mass to "steve jobs threatened patent suit to enforce no-hire policy" though none of the words mentioned the Apple company. Likewise, on 2013-09-06, "china unicom, telecom to sell latest iphone shortly after u.s. launch" received the 2nd highest attention mass (21%), in addition to the 1st one "apple hit with u.s. injunction in e-books antitrust case" (37%). These news headlines obviously demonstrate the success of MSIN in discovering relevant news stories associated with the stock time series, as our model is capable of unearthing the news contents that never explicitly mention the specific company name. For comparison, we show the performance of two other competing methods, GRUtxt and LSTMw/o using single-step attention mechanism, in the 1st and 2nd columns of Table 1 (respectively annotated in the purple and orange colors). It is seen that the top relevant news articles outputted by MSIN is more specific and far better than those highlighted by GRUtxt and LSTMw/o, which consolidates our quantitative evaluation reported in Fig.3 and 4.

In Table 2, we report the results of all three models when they are trained with the GOOG stock time series (input of the text corpus remains the same). One can observe similar results in which the quality of top news articles highlighted by our MSIN outperforms those discovered by GRUtxt and LSTMw/o using conventional attention mechanisms. Note that news articles on date 2013-08-14 are deliberately shown in both Tables 1 and 2 to demonstrate the reality that the set of relevant news stories are dependent on which time series has been used to train the model. As observed, they are different between AAPL and GOOG stock time series, yet their relevance to each of the stock is intuitively interpretable. The automate discovery of these relevant news articles w.r.t. each time series is desirable, and MSIN performs comparably well on both.

4 That means the cumulative summation from largest to smallest on the MSIN's column (pm) in Table 1 is greater than 0.5.

5 RelatedWork

A large number of single-modality studies analyzing either time series data or unstructured text documents have been proposed in the literature. Some of these works are based on classical statistical methods [Wu et al., 2016; Kristjanpoller and Michell, 2018] while others relying on neural networks [Qiu et al., 2016; Zhong and Enke, 2017]. Recent studies from the financial domain explore both time series of asset prices and the news articles [Schumaker et al., 2012; Weng et al., 2017; Akita et al., 2016] which are related to our work. Typically, these studies attempt to convert financial news into various numerical forms including news sentiment, subjective polarity, n-grams and then attach them as extra features of the numerical stock data. These handcrafted features require extensive pre-processing while being extracted independently from the time series data. [Akita et al., 2016] relies on recurrent neural networks that enable it to model stock series in their sequential form, which are then merged with the vector representation of the text news prior to making the market prediction. The text news must be chosen manually to ensure their relevancy to the given stock series. This technique and the aforementioned ones hence lack the capability of automatically discovering news articles relevant to a given time series and are limited in explanation.

Our work is also related to multi-modal deep learning studies [Baltruˇsaitis et al., 2019] which generally can be classified into three categories: early, late, and hybrid, depending on how and at which level the data from multiple modalities are fused together. In early fusion [Valada et al., 2016; Zadeh et al., 2016], multi-modal data sources are concatenated into a single feature vector prior to being used as the input to a learning model, while in late fusion, data are aggregated from the outputs of multiple models, each trained on a separate modality. The data can be fused based on aggregation rules such as, averaged-fusion [Nojavanasghari et al., 2016], tensor products [Zadeh et al., 2017], or a meta-model like gated memory [Zadeh et al., 2018]. The hybrid (in-between) fusion is the trade-off between the early and late paradigms, that allows the data to be aggregated at different scales, yet often requiring the strict synchronization among involved data modalities, such as in the gesture recognition [Neverova et al., 2015; Rajagopalan et al., 2016]. Our model is related to the third category; yet, we deal with asynchronous multimodals of numerical time series and unstructured text and relax the constraints on the time-step synchronization between modalities. More significantly, we perform data fusion through multiple steps and at the low-level features which strengthens our model in learning associated patterns across data modalities.

6 Conclusion

Jointly learning both numerical time series and unstructured text data is an important research endeavor to enhance our understanding of time series performance. In this work, we presented a novel neural model that is capable of discovering the top relevant textual information associated with a time series. In dealing with the complexity of relationship between the two data modalities, we develop MSIN that allows the direct incorporation of information learnt in the text modality to every time step modeling on the behavior of time series, considerably leveraging their mutual association through time. Through multi-steps of data interrelation, our model can learn to focus on a small subset of textual documents that best align with the time series. MSIN is completely data-driven and can act as an automated news filtering system for time series. We evaluate the performance of our model in the financial domain using stock time series, which are trained along with the common news headlines collected from Thompson Reuters. The empirical results well demonstrate the capability of MSIN in discovering the association between the two data modalities by highlighting top news articles relevant to the given stock series. Its performance is superior to state-of-the-art models relying on the conventional attention mechanisms.