Spotlight News Driven Quantitative Trading Based on Trajectory Optimization

1 Introduction

News-driven quantitative trading (NQT) has attracted great attention in recent years, which leverages news to capture the influence over market dynamics for making profitable quantitative trading (QT) decisions. Most existing NQT methods perform trading in a two-step paradigm: they first leverage news to analyze the market dynamics with some auxiliary financial prediction task, e.g., price regression [Schumaker and Chen, 2009], movement classification [Du and TanakaIshii, 2020], and stock ranking [Sawhney et al., 2021b]; and then make trading decisions based on their analyze. The trading performance of the two-step paradigm methods often relies on the predictive performance, however, which may hardly be accurate enough in practice [Millea, 2021; An et al., 2022]. In this paper, we focus on a more practical NQT paradigm, i.e., reinforcement learning (RL) based NQT paradigm, which bypasses the financial prediction and optimizes trading decisions directly, as shown in Figure 1.

∗Corresponding Author

The RL-based NQT paradigm models the complex sequential decision-making processes from the experience of interaction with the market to maximize overall profit directly [Ye et al., 2020; Sawhney et al., 2021c]. Although existing RL-based NQT methods produce remarkable results, they focus on learning effective news representations to enhance the market state rather than improving the learnability of RL to generate a series of coherent and profitable trading actions under the non-stationary financial markets, making them inapplicable in the online trading environment. Compared to the aforementioned approaches, we focus on addressing these prevalent issues in the RL-based NQT paradigm, which corresponds to the following three questions.

• (Q1) How can we construct an effective RL-based NQT method to make continuous and flexible trading actions?
• (Q2) How can we design an efficient mechanism that can leverage genuinely valuable information from a large amount of chaotic news to guide trading actions?
• (Q3) How can we adapt the RL-based NQT method to the non-stationary online trading environment?

Keeping this in mind, our works in this paper are all around answering the above three questions.

(A1) Novel RL-based NQT Framework. In quantitative practice, a multi-step profitable trading strategy is often a flexible action combination with several holding periods, e.g., short-term, medium-term, or long-term hold between consecutive buy and sell actions. Forming such a strategy requires ensuring trading action continuity, i.e., shifting trading actions frequently are not allowed to break the consistency of judgments and objectives of previous actions. To ensure action continuity, existing RL-based QT methods [Liu et al., 2020] introduce behavior cloning to learn from expert experience, which, however, may not be able to handle situations beyond the scope of the domain knowledge, especially in non-stationary markets. In addition, it is necessary for the multi-step profitable trading strategy to comprehensively consider the actions in the long history, so as to flexibly adjust the actions in different periods and achieve the longterm goal. Recently, in the RL community, a trajectory optimization framework originating from Decision Transformer [Chen et al., 2021], treats the RL problem as a sequence modeling problem under conditional behavior cloning, and applies transformer architecture to model a decision trajectory. Such a framework learns offline decision trajectories at a cheap knowledge cost and releases the powerful longsequence modeling capability of the transformer architecture, which inspires our work. Therefore, we propose an RL-based NQT method named SpotlightTrader inheriting the trajectory optimization framework, which models the trading decision trajectory and captures the multi-scale intrinsic relations between trading actions, market states, and rewards to form a continuous and flexible sequence of trading actions.

(A2) Novel spotlight-driven state trajectory modeling. For NQT, a common approach to leverage news is fusing the information from all news no matter whether or not the news is influential to stocks [Ye et al., 2020]. In reality, only a fraction of the massive and chaotic news stream holds the potential to influence stock trends [Sawhney et al., 2021c; Sawhney et al., 2021a], which we term as spotlight news. Considering all news equally not only dilutes the influence of spotlight news but also introduces noise to fool trading actions. Meanwhile, the occurrence of spotlight news will fleetingly and unanticipated break the stock trends that originally has the characteristics of random walk, and cause some irregular jumps in stock trends (cf. Figure 3). For example, once the spotlight news occurs, keen speculators will seize the profitable opportunity and quickly enter the market, resulting in abrupt jumps in the stock trend. Inspired by the above analysis, we innovatively propose a spotlight-driven state trajectory module, which first filters spotlight news by a sparse attention mechanism and then models a state trajectory following a stochastic process with irregular abrupt jumps caused by spotlight news to guide the trading actions.

(A3) Effective training pipeline for online adaption. The online RL-based models improve the capability of decisionmaking by interacting with the environment, which may not be suitable for trading, a scenario where online trial and error are expensive. By contrast, the offline RL-based models may be more practical [Levine et al., 2020], which leverages previously collected offline datasets to learn optimal policies without accessing the real market. However, the non-stationary markets face the severe distribution shift problem [Sun et al., 2021], which may render previously effective strategies ineffective if only exploiting historical data without any new exploration. Therefore, in order to adapt to the online environment, we propose a novel training pipeline, which blends offline pretraining for exploitation with online finetuning for exploration to enhance SpotlightTrader. In our training pipeline, we customize effective components, e.g., trading rollout mechanism, trajectory-level replay buffer, prioritized sampling policy, and hindsight return recalculation, to efficiently generate, store, and sample data in a trajectory-level way. The training pipeline prolongs SpotlightTrader's lifelong experience, i.e., new rollout decision trajectory immediately becomes part of the growing training set, to further improve our model's behavior in a continual online fashion.

In conclusion, we have proposed a novel RL-based NQT framework for making profitable trading decisions, followed by an effective spotlight-driven state trajectory modeling to enhance our proposed framework and an effective training pipeline for adapting the framework to online environments. To the best of our knowledge, this is the first work to plug the RL-based NQT model into the trajectory optimization framework. To verify the effectiveness of SpotlightTrader, we have conducted comprehensive experimental evaluations on three real-world datasets to demonstrate our model's superiority over the state-of-the-art NQT methods in terms of profitability and drawdown risk control.

2 RelatedWork

2.1 News-driven Quantitative Trading

NQT has attracted extensive research focus from the AI community, which incorporates natural language processing (NLP) techniques to translate raw unstructured textual data into meaningful insights for supporting NQT model. Early works [Ding et al., 2014; Hu et al., 2018] formulate NQT as a text classification problem by directly predicting the rise or fall of stock prices based on the extracted features, which hardly considers how to execute the trading decision. In recent years, sentiment-based and event-based NQT methods model the impact of news on trading. Sentimentbased NQT methods [Nan et al., 2022; Chen and Huang, 2021] regard the news articles' sentiments as the trading signal, which would be affected by the author's personal standpoints or writing styles, thus too subjective to assist trading decision making. Event-based NQT methods [Wu, 2020; Zhou et al., 2021] extract events from news articles and then regard events as the trading signal, which heavily relies on the knowledge extracted from specific events. Nevertheless, their works are orthogonal to our work, because instead of focusing on how to model complex news structure or content, we pay more attention to how news, as an important type of market observation, participates in and guides trading.

2.2 RL based Quantitative Trading

In recent years, RL has gained huge attention in the field of quantitative trading (QT) because of its superiority in complex sequential decision making. RL-based QT methods can be mainly categorized into two kinds in terms of the model design target. The first kind of method focus on designing various formulation of MDP to adapt to complex practical trading rules, e.g., reducing transaction cost [Zhang et al., 2022], refined trading operations [Wang et al., 2021], and

(a) The architecture of SpotlightTrader.

s1

o

m

1

k1

Spotlight-driven State Trajectory Module

s2

j

s3

f

( )

( )

s

( )

SDESolve

st-1

st

m

o

2

……

m

o

3

kt-1

o

m

1

t -

o

m

t

Spotlight

Embedding

Spotlight

News

sparsemax

Candidate

News Spotlight News Filtering Spotlight k sum Sparse Attention State As Query

s

(b) Spotlight-driven state trajectory module.

Figure 2: Overview of the proposed SpotlightTrader.

allowing short [Asodekar et al., 2022], and then directly apply classical RL model, e.g., DPG-based [Jiang et al., 2017], DDPG-based [Sawhney et al., 2021c], PPO-based [Yang et al., 2020] method, which spends little effort in modeling the complex relationship between financial environment and trading actions. The second kind of method spends a lot of effort on modeling complex states, i.e., extracting effective feature representations from financial environments by applying powerful sequential models, under the traditional RL framework [Xu et al., 2021; Sawhney et al., 2021c]. However, without careful consideration of the optimization of the RL model itself, these methods suffer from the instability issue [Hu et al., 2022] empirically when directly applying the modeled state to the decision process. Besides, Liu et al. [2020] formulate the QT process as a POMDP and learn expert behaviors based on behavioral cloning model, which is affected by the quality of the expert strategy to be imitated.

3 ProblemFormulation

3.1 NQT: News-driven Quantitative Trading

NQT refers to the process of constantly buying, holding, or selling N stocks during T trading rounds for profit by analyzing recent historical price features and news information. At each round t, we denote pct = [pct,1 , , · · · , pct,N ] as the close price of all stocks; ht ∈ RN as holding amount vector of all stocks; bt ∈ R+ as account balance; and rt as account profit that corresponds to reward in the RL term.

Initially, before actual trading, we set b0 to be the initial fund available for trading. At each trading round, a trading agent looks back at the stock price features and the stockrelated news of the past period for making trading decisions, aiming to maximize the cumulative account profits ∑Tt=1 rt at the end of the trading period.

3.2 POMDP for News-driven Quantitative Trading

According to the NQT process and the fact that the states of a financial market can only be observed partially, it is suitable to model the whole NQT process as a partially observable Markov Decision Process (POMDP). At each trading round t, a trading agent leverages a trading policy πθ(·) to take an action at ∈ A in a particular market state st ∈ S, which results in reward rt and the transition of state changes to state st+1. However, only part of the state, i.e., observation ot ∈ O, can be obtained by the agent. Hence, the key components of the NQT problem formulating as a POMDP are as follows:

Observation. At a round t, ot = [oat , omt ] is an accountmarket observation. The account observation oat comprises the account balance bt and the amount of holdings ht at round t. The market observation omt comprises the Opening-High-Low-Closing (OHLC) price features pohlc t-1 = [pot-1 , pht-1 , plt-1 , pct-1 ] of all stocks and stock-relevant news embeddings nt,1:nnews t in [t-1, t) of nnews t pieces of news.

Trading actions. At a round t, at ∈ [-1, 1]N is an action vector on all stocks. We re-scale this action using a constrain Kmax as described in [Kabbani and Duman, 2022], which represents the maximum allocation (buy/sell shares) and we denote the re-scaled action as ˆat,i. The available trading action ˆat,i of each stock i includes selling, buying, and holding, which results in decreasing, increasing, and no change of the holdings h, respectively, as follows: (1) Buying: when ˆat,i ∈ [-bt/pct,i ,-1], buy ˆait shares, and it leads to ht+1,i = ht,i + ˆat,i; (2) Selling: when ˆat,i ∈ [1, ht,i], sell ˆat,i shares, and it leads to ht+1,i = ht,i + ˆat,i; (3) Holding: when ˆat,i ∈ (-1, 1) shares, hold the shares, and it leads to no change in hit . It should be noted that all bought stocks should not result in a negative balance on the portfolio value. That is, without loss of generality, we assume that selling orders are made on the first d1 stocks and the buying orders are made on the last d2 ones, and that at should satisfy pct,[1:d1]⊤ˆat,[1:d1] + bt + pct,[N-d2:N]⊤ˆat,[N-d2:N] ≥ 0. The remaining balance is updated as bt+1 = bt + pct ⊤ˆat .

Reward. We define the reward rt as the change in the value when an action at is taken as:

where ct denotes the transaction costs incurred at round t.

Return-to-go (RTG). The returns-to-go refers to the future desired returns, which is an important component of the trajectory optimization framework. We calculate the returns-togo by gt = ∑Tt′=t rt′ . In order to generate actions based on future desired returns, we model the returns-to-go instead of directly modeling the rewards.

4 Method

The architecture of SpotlightTrader is presented in Figure 2(a). In this section, we first define the decision trajectory, i.e., a sequence of return-to-go, observation, and action, which is a basic target in decision autoregressive training and generation (§4.1). We then present the spotlight-driven state trajectory (SST) module, which recovers the state trajectory from the partial observation trajectory (§4.2). With the state trajectory, we present how to make trading decisions in the trading decision (TD) module (§4.3). After that, to adapt to an online trading environment, we present a pretrain-finetune pipeline (§4.4). Finally, we present the model optimization for model training and finetuning (§4.5).

4.1 Decision Trajectory

In order to keep the continuity of trading actions, SpotlightTrader inherits Decision Transformer [Chen et al., 2021] to learn the policy under the conditional behavior cloning framework. Here, we term a decision trajectory τ as a sequence of (RTG, observation, action) tuples collected from any period of consecutive trading rounds, which are amenable to autoregressive training and generation:

where L is the length of the decision trajectory.

We introduce the collection methods of decision trajectories in the offline training phase and online trading phase. In the training phase, we collect expert decision trajectories from existing strategies or replayed trajectories generated via the partially-trained SpotlightTrader to form trajectory-wise offline data Jtrain = {τ i }|Jtrain| i=1 , where τ i is a decision trajectory with L consecutive (RTG, observation, action) tuples. In the trading phase, we rollout trading actions to generate a brand new trajectory with every L trading round, which will be elaborated in §4.4.

4.2 SST: Spotlight-driven State TrajectoryModule

To precisely characterize state trajectories, SST models the dynamics of state trajectory, which consists of the stochastic observation-controlled continuous state trajectory with perturbation by the inherent spotlight-driven jump as shown in Figure 3. With the jump driven by spotlight news, we term the state trajectory as a spotlight-driven state trajectory.

Dynamics of state trajectory. In SST, each stock has its own spotlight-driven state trajectory, which will be merged together as the final state of the trading decision. SST model spotlight-driven state trajectory as a hybrid system including a stochastic process with jumps, which extends the neural ordinary differential equations (Neural ODEs) [Chen et al., 2018] with two terms: (1) a stochastic latent dynamics term that models the stochastic of observation om and (2) a spotlight jump term that models the discrete spotlight news.

For each stock's spotlight-driven state trajectory, a Neural ODE defines a continuous-time transformation of variables, and we specify the transformation of states at round t by a stochastic differential equation (SDE) as follows:

where fψ and σϕ are the drift and diffusion functions, respectively, and jφ is the jump function. To overcome the discontinuity of the spotlight jumps, we set an indicator function ω(t), where ω(t) = 1 when the spotlight jump exists at t.

Starting from an initial state s0, the transformed state at any time t is given by integrating an SDE forward by time:

We elaborate the neural networks in dst, which defines the stochastic latent dynamics and spotlight jump as follows.

Stochastic latent dynamics. We elaborate on the two terms in stochastic latent dynamics, i.e., dynamics drift function and dynamics diffusion function as follows.

Dynamics drift function fψ(t, st, omt ) model the deterministic of stochastic process. Given that the observation trajectory reflects part of the state trajectory, it is natural to learn the drift function based on the observation trajectory. We parameterize the internal state dynamics based on observation omt by a multi-layer perceptron (MLP). This constrains the internal state dynamics to the observation space and improves the stability of the state trajectory.

Dynamics diffusion function σϕ(t, st) is necessary to incorporate stochasticity into the stochastic latent dynamics, which we parameterize by an additional MLP.

Spotlight jump. The spotlight jump is modeled by first filtering the spotlight news from the noisy news stream and then applying it as the control point of the jump function.

Firstly, we introduce the spotlight news filtering mechanism. Here, we develop a novel spotlight news filtering layer based on a sparse attention mechanism [Martins and Astudillo, 2016], which is beneficial for focusing on the spotlight news driving market state changes. As shown in Figure 2(b), at each round t, we obtain a set of candidate news nt,1:nnews t , in which each piece of news i is embed by BERT [Kenton and Toutanova, 2019] into a news embedding nt,i. The sparse attention mechanism is adopted to filter spotlight news and aggregate the representations of all spotlight news to obtain one spotlight representation kt. Specifically, we first generate query Qt, keys Kt, and values Vt from the state and candidate news representations as:

Then, we leverage the query controlled by the current state to guide the spotlight representation learning as follows:

where the sparsemax(·) function retains most of the important properties of softmax(·) function and automatically prune the large volume of news.

Secondly, we introduce the spotlight jump function jφ(t, st, kt). The spotlight news introduces a jump ∆st to the spotlight-driven state trajectory. The jump is parameterized by an MLP that takes the spotlight representation kt and internal state st as input. Our architecture also assumes that the spotlight representation does not directly interrupt the internal state.

Final state embedding. For final state embedding at round t, we first apply a CNN layer for fusing all stocks' state embeddings learned by SST to obtain the spotlight-driven state st, then concatenate the trading-account state oat , to form an overall state representation sfinal t = [st, oat ].

4.3 TD: Trading DecisionModule

In our POMDP setting for NQT, the trading agent needs to generate a trading action at at each round t. Similar to Decision Transformer, we feed a decision trajectory into GPT-2 Transformer [Radford et al., 2019], which applies a causal mask to enforce the autoregressive structure of decision trajectory. To obtain token embeddings, we embed each element ai, gi, sfinal i in the decision trajectory by a linear layer. Additionally, a position embedding for each trading round is learned and added to each token, as one timestep corresponds to three tokens. The token embeddings are then processed by a GPT-2 Transformer, which predicts the latest action token embedding via autoregressive modeling. Finally, we added an action prediction layer, followed by a tanh activation function to make the latest action fall within the specified range, which outputs actions to buy, hold or sell the shares of each stock. With the latest action from the TD, SpotlightTrader makes a trading decision and obtains the reward according to Eq. (1).

4.4 Pretain-Finetune Pipeline

RL policies trained on purely offline datasets are typically sub-optimal because the offline trajectories may cover only a limited part of the state space under a non-stationary financial market. Therefore, it is in high demand to finetune the pretrained RL policy based on the newly collected trajectories during online trading. The key property of such pretrainfinetune pipeline is to balance the exploration-exploitation trade-off, so we devise a sample-efficient online finetune mechanism, which takes exploration into consideration and includes four key components introduced as follows.

Algorithm 1: SpotlightTrader's pipline

Input: Trajectories in training data Jtrain, trading rounds T, exploration RTG gonline, trajectory length L, replay buffer size K, mini-batch size B. Output: Trading actions a1:T . /* Pretrain stage */ 1 while not converged do 2 Sample B trajectories τ i1,...,iB out of Jtrain. 3 Update the policy πθ by τ i1,...,iB . /* Finetune stage */ 4 Initialize trajectory-level replay buffer: Jreplay ← top K trajectories in Jtrain. 5 for t = 1,L+1, 2L+1, ..., T do 6 Trading rollout a new trajectory τt. 7 Replace the oldest trajectory by τt in Jreplay.

• 8 PS: Compute sampling probability P(τ) and sample B trajectories out of Jreplay by P(τ).
• 9 for each sampled trajectory τ do

HRR: Recalculate the RTG sequence g1:L by gi = ∑Lj=i rj , 1 ≤ i ≤ L and assemble the new trajectory ˆ τ with the hindsight RTG sequence.

• 10 11 Update the policy π by ˆ τ .

Trading rollout. Firstly, we need to collect online trajectories during actual trading, which depends on the trading rollout of a learned policy. Specifically, given a pre-defined desired RTG gt and an initial observation ot at round t, TD generates the action at = πθ(ot, gt) for trading. Then, we obtain a reward rt and the next observation ot+1, which gives us the next RTG as gt+1 = gt - rt. As before, TD generates at+1 based on ot:t+1, gt:t+1, and at. This process is repeated to generate an online trajectory until a specified length of the trajectory is reached.

Trajectory-level replay buffer (TRB). Secondly, we need a replay buffer to store the trajectories that are used for online finetuning. Initially, we set a prioritized replay buffer Jreplay, which is filled with trajectory-level data with the highest returns from offline data, see Line 4 in Algorithm 1. Every time we completely rollout a trajectory with the current policy, we refresh the replay buffer in a first-in-first-out manner periodically. Afterward, we update the policy and rollout again.

Prioritized sampling (PS). Thirdly, we need to select the data from TRB to form the mini-batch used for online finetuning. Specifically, we enable each mini-batch to consist of decision trajectories from offline data and rollout data, which are sampled by a prioritized sampling mechanism [Schaul et al., 2016], see Line 8 in Algorithm 1. The prioritized sampling mechanism encourages replaying trajectory with higher expected learning progress more frequently. For each trajectory τ i ∈ Jreplay, we calculate it's trajectory priority as pτ i = Lτ i +ϵD, where Lτ i is loss calculated by Eq. (7), and ϵD is a positive constant for trajectory from offline data to increase the probability of getting sampled. The probability P(τ i ) of the trajectory τ i is proportional to its priority pτ i , namely, P(τ i ) = pτi ∑j pτj . To consider changes in the sample distribution, which may affect model optimization, trajectories for the network updates are weighted with importance sampling weights, i.e., wτ i = ( 1K · 1 P(τ i ) )β , where β is an annealing parameter [Schaul et al., 2016]. In this way, the prioritized replay buffer controls the ratio of data between the offline data and new rolling data.

Hindsight return recalculation (HRR). The return achieved during a policy rollout and the induced RTG can differ from the intended RTG, which harms the model learning. To make each sampled trajectory τ more similar to the expert trajectories and thus boost learning, we apply HRR to generate new trajectories ˆ τ with the achieved RTG, as opposed to the intended RTG with a manually set initial RTG gonline, see Line 10-11 of Algorithm 1.

4.5 Model Optimization

In this subsection, we aim to train our model to make profitable trading decisions, a prime purpose of the model, which is equivalent to maximizing the high-return stochastic optimal policy distribution π∗ θ (at|g≤t, o≤t, a<t). Further, in order to equip the ability of SpotlightTrader to balance the exploration-exploitation trade-off in online trading, we apply the negative log-likelihood (NLL) loss with a policy entropy [Zheng et al., 2022]. Thus, the loss function based on the decision trajectories' distribution T is calculated by:

where λ serves the role of temperature variable and HT θ is the policy entropy. Note that in the offline training phase, the temperature λ = 0. The loss of mini-batch can be calculated by L = ∑Bi=1 wτ iLτ i , where B is the mini-batch size.

5 Experiments

We conduct extensive experiments to answer the following questions:

• Q1: How does SpotlightTrader perform on profitability and risk over the trading period?
• Q2: How effective are constructing state trajectories with different information sources?
• Q3: How do the key components contribute to the performance of SpotlightTrader?
• Q4: How do the major parameters affect the performance and how to choose optimal values?

Q5: How does SpotlightTrader makes decision?

5.1 Experimental Settings

Dataset Descriptions. We conduct experiments on three real-world datasets 1 . Twitter-SP500 is a public dataset consist stocks in S&P 500 index from U.S. market and tweets from Twitter. THS-CSI300 is our collected China market dataset covering stocks from the CSI 300 Index and news from iFinD. COVID-SP500 is our collected U.S. stock market dataset covering stocks in the S&P 500 index and news from Aylien during the COVID-19 pandemic period. Following [Jiang et al., 2017; Sawhney et al., 2021c], we preprocess the price data by missing values filling and normalization, and we embed the English and Chinese news text by pre-trained FinBERT [Araci, 2019] and Chinese-FinBERT 2 . We divide each dataset into non-overlapping offline and online datasets, and the statistics of the datasets are presented in Table 1. We elaborate on the generation of trajectory-level offline data and provide the offline dataset in our GitHub repository 3 . The parameter setting and implement details will be presented in a longer version of this paper.

1 Data is collected from https://github.com/yumoxu/stocknetdataset, http://www.51ifind.com.cn/, and https://aylien.com/.

Baselines. We compare SpolightTrader with nine NQT methods in four groups: (1) Regression (REG) based method (i.e., AZFinText [Schumaker and Chen, 2009] and Game [Ang and Lim, 2022]) predicts asset returns and trades assets that are greater than or equal to 1%; (2) Classification (CLF) based method (i.e.,StockNet [Xu and Cohen, 2018], StockEmb [Du and Tanaka-Ishii, 2020], and HTLSTM [Sawhney et al., 2021a]) classifies prices trends and trades assets where prices are expected to rise; (3) Ranking(RAN) based method (i.e., FAST [Sawhney et al., 2021b] and HISN [Wang et al., 2022]) ranks assets and trades the most profitable assets; (4) Reinforcement Learning (RL) based method (i.e., PG-based SARL [Ye et al., 2020] and DDPG-based Profit [Sawhney et al., 2021c]) directly makes news-driven trading decisions.

Evaluation Metrics. Following [Sawhney et al., 2021c], we adopt four widely used metrics. Annualized Percentage Yield (APY) and Maximum Drawdown (MDD) measure the returns and risks, respectively. Sharpe Ratio (SR) and Sortino Ratio (StR) measure the trader's risk-adjusted returns.

5.2 Performance Comparison and Analysis

To answer Q1, in Table 2, we compare SpotlightTrader with methods from four different NQT paradigms.

Result 1: Performance on Profitability. In general, we observe that RL-based methods are typically more profitable, i.e., achieve higher APY values, compared to the two-step methods. This is because the RL trading agents optimize every trading action for profit generation directly, unlike twostep methods, where the trading mechanism is only to select the predicted profitable stocks to trade and the overall profits are not optimized as a reward. These observations validate the necessity of formulating quantitative trading as an THS-CSI300 COVID-SP500

2 https://github.com/valuesimplex/FinBERT

3 https://github.com/Yangmy412/SpotlightTraderOfflineDataset

0.35 0.10

0.64 0.06

0.50 0.10

1.01 0.07

1.09 0.05

RL problem. Furthermore, among the RL-based methods, our method achieves the highest APY values with an average improvement of 36.14% over the best-performing baselines. Our method benefits from not only its novel decision trajectory optimization framework to make flexible long-term and short-term profitable trading decisions, but also its effective spotlight news filtering and modeling mechanisms to enhance the decision trajectory optimization framework.

Result 2: Performance on Risk. From the results in Table 2, we observe that our model achieves the lowest MDD values with an average improvement of 13.55%, indicating its ability to respond to bearish markets. In addition to the superior RL-based trading framework in our model over the two-step strategies, such performance is attributable to our model's novel training pipeline design, which adapts the RLbased framework to online trading environments for avoiding volatile stocks timely during such turbulent environments. Our proposed strategy also achieves the best performance on the SR and StR metric, which indicates its superiority in balancing return and risk.

4 The index is S&P 500 (ˆGSPC) in Twitter-SP500 and COVIDSP500 datasets, and CSI 300 (000300.SH) in THS-CSI300.

5 The improvement over the best-performing baselines.

6 The improvement is significant based on paired t-test at the significance level of 0.05.

5.3 Ablation Study

Result 3: Effect of Absorbed Information. To answerQ2, we evaluate the effectiveness of different information sources on stock profiling. We report SpotlightTrader's performance under five cases, which cover all possible combinations of the price feature and news information, in Table 3, from which we have two observations. Firstly, significant APY and MDD improvements are noted by blending price features with news information, indicating the contributions of both news and price features. This is mainly because price features and news are indispensable stochastic processes and jumps, respectively, in the state trajectory, which can well support the generation of profitable trading actions. Secondly, we find that considering all pieces of news from the news stream unexpectedly hurts the model's performance, which demonstrates that the spotlight news filtering mechanism is very necessary to reduce the noise from irrelevant news.

Result 4: Effect of Model Variants. To answer Q3, we conduct an ablation analysis on the adaptability of our model to online environments. We build three variants of SpotlightTrader, including w/o-TRB, w/o-PS, and w/o-HRR, which corresponds to our model without TRB, PS, and HRR, respectively. From Figure 4, we can conclude that interactively finetuning using online trading decision trajectories is crucial to our model, and the priority sampling mechanism clearly benefits the model's adaptability. Besides, failure to perform hindsight return recalculation will confuse the model's estimation of the cumulative return of the trajectory, making it impossible to seek the optimal trajectory.

5.4 Parameter Analysis: Probing Sensitivity

To answer Q4, we conduct sensitivity experiments on two important parameters, i.e., trajectory length and online RTG.

Result 5: Effect of trajectory length. The trajectory length L determines how long the agent's current action de-

(a) Sparse-attention weights and corresponding Spotlight News.

(b) Trading actions from SpotlightTrader .

Figure 6: Case study on Boeing: the filtered spotlight news on May 26, 2020 and the corresponding trading actions afterwards.

pends on the past. We depict SpotlightTrader's performance on APY under different trajectory lengths L varying in {1, 5, 10, 15, 20, 25} in Figure 5. When the trajectory length is 1, an edge case where each action only depends on the current RTG and state and our POMDP formulation can be reduced to a Markov decision process (MDP), our model performs the worst, indicating that the MDP formulation of NQT is detrimental to trading performance. With the increase in trajectory length, our model achieves better performance, because considering long-term historical actions can contribute to achieving the long-term goal by ensuring trading action continuity, which supports the analysis as those in the main text. In addition, it is worth mentioning that a relatively large trajectory length, i.e., L = 15, is enough to gather enough information to make a trading decision, and a too-large trajectory length is unnecessary and oversaturated for modeling.

Result 6: Effect of online RTG. The online RTG gonline is used to set g1 during every trading rollout. Because gonline may seriously affect the stitching ability of our model on the trajectory during trading rollout, we discuss two mechanisms for pre-defined gonline [Zheng et al., 2022], i.e., fixed and dynamic RTG. Firstly, in the fixed RTG mechanism, we set the RTG gonline as the fixed RTG gfix, which is the scale of the maximum RTG obtained in the pretraining stage. The scale varies in {0.5, 1, 2}. Secondly, in the dynamic RTG mechanism, we adjust the online exploration RTG gonline to be a dynamic RTG gdynamic, which is calculated based on the cumulative rewards of the trajectories, i.e., achieved RTG in each trajectory, stored in the replay buffer. Specifically, we set dynamic RTG as the q-th percentile of these non-stationary achieved RTGs. The percentile q varies in {85%, 90%, 95%}. We report SpotlightTrader's performance on APY under different gonline in Figure 7, which shows that the dynamic RTG mechanism achieves higher returns compared to the fixed RTG mechanism, demonstrating the superiority of our model in adapting to non-stationary financial markets.

5.5 Case Study

We conduct an extended analysis in Figure 6 to elucidate on SpotlightTrader's spotlight news filtering result and its practical applicability to real-world quantitative trading for answering Q5. Here, we study the U.S. market from May 20, 2020 to June 20, 2020. We visualize the sparse-attention weights of Boeing (BA), which is an aircraft manufacturer, and excerpt the briefing of spotlight news to analyze how our proposed model trade BA on May 26.

Result 7: Analyzing trading actions. In order to explore how our model forms trading actions, we report BA's trading actions for 20 consecutive days from May 26, 2020. On each day, the sparse attention mechanism in our SST module filters out a large amount of news and only retains a handful of more influential ones. For example, on May 26, 2020, we found that the top 4 news with the highest weights were all about the loosening of flying bans during COVID-19, which is very likely to have a positive impact on BA, leading to BA's price rises afterward. This case study indicates that SpotlightTrader has the ability to filter spotlight news, that is, to find news that can drive price changes to guide transactions. Under the positive influence of the spotlight news, our model chose the Buy signal for 3 consecutive days after May 26, continuously expanding BA's position. Afterward, it chose to hold BA for a period of time, and our model did not sell the position of BA until June 10, 2020, when there was an obvious downward trend. The above continuous and explainable trading behavior indicates our model's powerful modeling ability for action trajectory, which ensures the consistency and coherence of trading behavior.

6 Conclusions and FutureWork

In this paper, we study the NQT task and propose a novel NQT framework SpotlightTrader, which can release the powerful long-sequence modeling capability of the transformer and effectively stitch together a continuous and flexible sequence of trading decisions. Extensive experiments demonstrate our model's superiority in terms of both profitability and drawdown risk control. In the future, we plan to extend our work into two potential directions, i.e., modeling complex relations between stocks to coordinate stock trading decisions and leveraging more multi-modal information, e.g., earning calls, to capture the market state more completely.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No. 72192823), the "Ten Thousand Talents Program" of Zhejiang Province for Leading Experts (No. 2021R52001), and the Cooperation Project of MYbank, Ant Group.