StockFormer: Learning Hybrid Trading Machines with Predictive Coding
Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. Main Track. Pages 4766-4774.
https://doi.org/10.24963/ijcai.2023/530
Keywords
- Machine Learning: ML: Deep reinforcement learning
- Multidisciplinary Topics and Applications: MDA: Finance
Abstract
Typical RL-for-finance solutions directly optimize trading policies over the noisy market data, such as stock prices and trading volumes, without explicitly considering the future trends and correlations of different investment assets as we humans do. In this paper, we present StockFormer, a hybrid trading machine that integrates the forward modeling capabilities of predictive coding with the advantages of RL agents in policy flexibility. The predictive coding part consists of three Transformer branches with modified structures, which respectively extract effective latent states of long-/short-term future dynamics and asset relations. The RL agent adaptively fuses these states and then executes an actor-critic algorithm in the unified state space. The entire model is jointly trained by propagating the critic's gradients back to the predictive coding module. StockFormer significantly outperforms existing approaches across three publicly available financial datasets in terms of portfolio returns and Sharpe ratios.
- [Basak and Chabakauri, 2010] Suleyman Basak and Georgy Chabakauri. Dynamic mean-variance asset allocation. The Review of Financial Studies, 23(8):2970-3016, 2010.
- [Benhamou et al., 2020] Eric Benhamou, David Saltiel, Jean Jacques Ohana, Jamal Atif, and Rida Laraki. Deep reinforcement learning (DRL) for portfolio allocation. In ECMLPKDD, 2020.
- [Cho et al., 2019] Chun-Hung Cho, Guan-Yi Lee, Yueh-Lin Tsai, and Kun-Chan Lan. Toward stock price prediction using deep learning. In UCC, 2019.
- [Duan et al., 2022] Yitong Duan, Lei Wang, Qizhong Zhang, and Jian Li. FactorVAE: A probabilistic dynamic fac- tor model based on variational autoencoder for predicting cross-sectional stock returns. In AAAI, 2022.
- [Elias, 1955] Peter Elias. Predictive coding-I. IRE transac- tions on information theory, 1(1):16-24, 1955.
- [Fedus et al., 2021] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021.
- [Feng et al., 2019] Fuli Feng, Xiangnan He, Xiang Wang, Cheng Luo, Yiqun Liu, and Tat-Seng Chua. Temporal relational ranking for stock prediction. ACM Transactions on Information Systems, 37(2):1-30, 2019.
- [Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Kris- tian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.
- [Hoseinzade and Haratizadeh, 2019] Ehsan Hoseinzade and Saman Haratizadeh. CNNpred: CNN-based stock market prediction using a diverse set of variables. Expert Systems with Applications, 129:273-285, 2019.
- [Hu and Lin, 2019] Yuh-Jong Hu and Shang-Jen Lin. Deep reinforcement learning for optimizing finance portfolio management. In AICAI, 2019.
- [Hu et al., 2018] Ziniu Hu, Weiqing Liu, Jiang Bian, Xu- anzhe Liu, and Tie-Yan Liu. Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction. In WSDM, 2018.
- [Huotari et al., 2020] Tommi Huotari, Jyrki Savolainen, and Mikael Collan. Deep reinforcement learning agent for S&P 500 stock selection. Axioms, 9(4):130, 2020.
- [Kitaev et al., 2020] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
- [Lee et al., 2020] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. In NeurIPS, 2020.
- [Li et al., 2018] Hao Li, Yanyan Shen, and Yanmin Zhu. Stock price prediction using attention-based multi-input LSTM. In ACML, 2018.
- [Liang et al., 2018] Zhipeng Liang, Hao Chen, Junhao Zhu, Kangkang Jiang, and Yanran Li. Adversarial deep rein- forcement learning in portfolio management. arXiv preprint arXiv:1808.09940, 2018.
- [Lillicrap et al., 2016] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
- [Liu et al., 2021] Xiao-Yang Liu, Hongyang Yang, Jiechao Gao, and Christina Dan Wang. FinRL: Deep reinforcement learning framework to automate trading in quantitative fi- nance. In ICAIF, 2021.
- [Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word repre- sentations in vector space. In ICLR, 2013.
- [Oord et al., 2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
- [Patil et al., 2020] Pratik Patil, Ching-Seh Mike Wu, Kate- rina Potika, and Marjan Orang. Stock market prediction using ensemble of graph theory, machine learning and deep learning models. In ICSIM, 2020.
- [Puterman, 2014] Martin L Puterman. Markov decision pro- cesses: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
- [Qin et al., 2017] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. arXiv preprint arXiv:1704.02971, 2017. IJCAI
- [Rao and Ballard, 1999] Rajesh PN Rao and Dana H Ballard. Predictive coding in the visual cortex: A functional interpre- tation of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79-87, 1999.
- [Spratling, 2017] Michael W Spratling. A review of predic- tive coding algorithms. Brain and cognition, 112:92-97, 2017.
- [Suri et al., 2021] Karush Suri, Xiao Qi Shi, Konstantinos Plataniotis, and Yuri Lawryshyn. TradeR: Practical deep hi- erarchical reinforcement learning for trade execution. arXiv preprint arXiv:2104.00620, 2021.
- [Théate and Ernst, 2021] Thibaut Théate and Damien Ernst. An application of deep reinforcement learning to algorith- mic trading. Expert Systems with Applications, 173:114632, 2021.
- [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
- [Wang et al., 2021] Heyuan Wang, Shun Li, Tengjiao Wang, and Jiayi Zheng. Hierarchical adaptive temporal-relational modeling for stock trend prediction. In IJCAI, 2021. IJCAI
- [Wen et al., 2019] Min Wen, Ping Li, Lingfei Zhang, and Yan Chen. Stock market trend prediction using high-order information of time series. IEEE Access, 7:28299-28308, 2019.
- [Weng et al., 2020] Liguo Weng, Xudong Sun, Min Xia, Jia Liu, and Yiqing Xu. Portfolio trading system of digital currencies: A deep reinforcement learning with multidi- mensional attention gating mechanism. Neurocomputing, 402:171-182, 2020.
- [Wu et al., 2019] Jheng-Long Wu, Chi-Sheng Yang, Kai- Hsuan Liu, and Min-Tzu Huang. A deep learning model for dimensional valencearousal intensity prediction in stock market. In iCAST, 2019.
- [Wu et al., 2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. In NeurIPS, 2021.
- [Yarats et al., 2021] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI, 2021.
- [Ye et al., 2020] Yunan Ye, Hengzhi Pei, Boxin Wang, Pin- Yu Chen, Yada Zhu, Ju Xiao, and Bo Li. Reinforcement- learning based portfolio management with augmented asset movement prediction states. In AAAI, 2020.
- [Zhang et al., 2017] Liheng Zhang, Charu Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequency trading patterns. In KDD, 2017.
- [Zhong et al., 2020] Yueyang Zhong, YeeMan Bergstrom, and Amy R Ward. Data-driven market-making via model- free learning. In IJCAI, 2020.
- [Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, 2021.
References were extracted automatically from the PDF and may contain errors.
Full text extracted automatically from the PDF; figures, tables and equations may differ. The PDF is the version of record.
1 Introduction
Reinforcement learning (RL) has shown promising results in practical applications of financial decision-making problems, such as improving stock trading strategies by identifying promising buying and selling points [Liu et al., 2021; Zhong et al., 2020]. A common practice is to formulate the portfolio optimization problem as a Markov decision process (MDP) [Puterman, 2014] and directly perform model-free RL algorithms in the state space represented by the observed data (e.g., stock prices, trading volumes, and technical indicators). However, it makes an excessively strong assumption that the observed data is sufficiently informative and can well represent (i) the correlations between hundreds (or thousands) of stocks and (ii) the underlying (or even future) dynamics in the rapidly changing financial markets.
To address this issue, we consider the way in which humans make investment decisions. There are two factors that require special consideration, that is, the dynamic stock correlations and the expected returns of each stock in both longand short-term horizons. We present StockFormer, a novel RL agent that learns to adaptively discover and capitalize on promising trading opportunities. It is a hybrid trading machine that integrates the forward modeling capabilities of predictive coding with the advantages of actor-critic methods for trading flexibility. Predictive coding [Elias, 1955; Spratling, 2017; Rao and Ballard, 1999] is one of the most successful selfsupervised learning methods in natural language processing [Mikolov et al., 2013] and computer vision [Oord et al., 2018], whose core idea is to extract useful latent states from noisy market data that can maximally benefit the prediction of future or missing contextual information.
∗Corresponding author
Specifically, we leverage three Transformer-like networks to respectively learn long-horizon, short-horizon, and relational latent representations from the observed market data. To ease the difficulty of concurrent time series modeling, StockFormer employs multi-head feed-forward networks in the attention block, which can maintain the diversity of temporal patterns learned from multiple concurrent market asset series (e.g., trading records of hundreds of stocks in the same time period). For policy optimization, the three types of latent states are adaptively and progressively combined through a series of multi-head attention structures. StockFormer exploits the actor-critic method but only propagates the critic's analytic gradients back into the relational state encoder.
Notably, there exists another line of work that aims to improve future stock prediction accuracy with powerful time series modeling networks [Li et al., 2018; Feng et al., 2019; Wang et al., 2021; Duan et al., 2022]. They use fixed trading rules, such as "buy-and-hold", which recycles capital from unsuccessful stock bets to average in on stocks that have promising future returns and hold them for a fixed period of time. As shown in Table 1, despite recent success, these models are not directly designed to maximize investment returns and cannot provide flexible trading decisions.
We empirically observe that StockFormer remarkably outperforms existing stock prediction and RL-for-finance approaches across three public datasets from NASDAQ and Chinese stock markets as well as the cryptocurrency market.
2 RelatedWork
RL for finance. There have been many attempts to use RL methods to make trading decisions [Th´eate and Ernst, 2021; Weng et al., 2020; Liang et al., 2018; Benhamou et al., 2020].
| Model | Category | Trading policy | RL state space |
|---|---|---|---|
| FactorVAE [Duan et al., 2022] SARL [Ye et al., 2020] StockFormer | Stock prediction RL-for-finance Hybrid | Fixed (e.g., buy-and-hold) Learned Learned | n/a Observed asset prices + Asset movement signal Temporal + Relational predictive coding (via SSL) |
Main differences of these models include: the definition of the input states [Zhong et al., 2020; Liu et al., 2021; Weng et al., 2020], the engineering of reward functions [Liang et al., 2018; Hu and Lin, 2019], and the RL algorithms [Benhamou et al., 2020; Suri et al., 2021; Huotari et al., 2020]. SARL [Ye et al., 2020] proposed to learn policy in the noisy observation space and expand the space with additional asset movement signals such as financial news embeddings. FinRL [Liu et al., 2021] integrates multiple off-the-shelf RL algorithms such as Soft Actor-Critic (SAC) [Haarnoja et al., 2018] and DDPG [Lillicrap et al., 2016]. It defines the states as a combination of the covariance matrix of the close prices of all stocks and the MACD indicators, whose dimension will sharply increase with the growth of the number of stocks. We use its SAC implementation as the baseline of StockFormer.
Stock prediction. Mainstream stock prediction approaches can be divided into three categories. CNN-based models [Hoseinzade and Haratizadeh, 2019; Wen et al., 2019] take the historical data of different stocks as a set of input feature maps of a convolutional neural network. RNN-based models [Li et al., 2018; Zhang et al., 2017; Qin et al., 2017] are better at capturing the underlying sequential trends in stocks. Other network architectures use attention mechanisms [Hu et al., 2018; Li et al., 2018; Wu et al., 2019], dilated convolutions [Wang et al., 2021; Cho et al., 2019], or graph neural networks [Feng et al., 2019; Wang et al., 2021; Patil et al., 2020] to jointly model the long-term dynamics and the relations of trading assets. Inspired by previous Transformer-based architectures [Vaswani et al., 2017; Kitaev et al., 2020; Zhou et al., 2021; Wu et al., 2021; Fedus et al., 2021], StockFormer integrates Transformer in the RL-for-finance framework. It borrows the idea of selfsupervised predictive coding as a representation learning method to extract useful and compact latent states from noisy and high-dimensional market data.
3 Portfolio Optimization as POMDPs
We believe that: One of the key challenges of RL-for-finance is to extract useful states that can reflect the essential dynamics of the market from noisy, high-dimensional raw transaction records. Therefore, unlike previous work that commonly treats portfolio optimization as an MDP problem with 5-tuples (S,A, T ,R, γ), in this paper, we take it as partially observable Markov decision processes (POMDPs) with 7-tuples (O, S,A, T ,Z,R, γ), where O is the observation space of the noisy market data, S is the state space, A is the action space, T (st+1|st, at) denotes the state transition probability, Z(Ot|st) is the prior distribution of the observed data Ot given latent state st. Assuming a fixed length of P steps in each episode (e.g., we use 1,000 trading days for the stock trading experiments), the goal is to learn the optimal policy π∗ that can maximize the total payoff: Gt = Eτ∼π[∑Pt=1 γ t-1Rt], where the Rt is the immediate reward at each time step drew from the reward function R(s, a) and γ ∈ (0, 1) is the discount factor of future rewards. We here give detailed definitions of O, S, A, and R as follows.
Noisy observation space (O). Let us take the stock trading problem for instance. The observed data includes:
- Raw trading records Oprice t ∈ RT×N×5 that involve daily opening, close, highest, lowest prices, and trading volume during the past T days, where N is the number of stocks.
- Technical indicators Ostat t ∈ RN×K , where K is the number of indicators that capture the dynamics of the time series.
- A covariance matrix Ocov t ∈ RN×N between sequences of daily close prices of all stocks over a fixed period before t.
In the following sections, we use Orelat t ∈ RN×(K+N) to denote the concatenation of Ostat t and Ocov t .
Compositional state space (S). The state space of StockFormer is composed of three types of latent states and the explicit account state samount t ∈ RN+1 that represents the total account balance and the holding amount of each trading asset.
Action space (A). We have a continuous action space such that at ∈ RN , indicating the amount of buying, holding, or selling shares on each trading asset. To simulate real-world trading scenarios, we discretize at into multiple intervals of daily trading signals when interacting with the environment.
Reward function (R). The reward Rt ∼ R(st, at) is defined as the daily portfolio return ratios.
4 StockFormer
As aforementioned, we have highlighted the challenge of RLfor-finance methods, that is, learning flexible policies aware of the dynamic relations between a batch of trading targets in the financial market and the future trends of each individual of them. To tackle this challenge, we introduce StockFormer, an RL agent that extracts disentangled latent states from noisy time series data through predictive coding, and then optimizes the trading decisions in the compositional state space. Therefore, StockFormer consists of two training phases: predictive coding and policy learning.
4.1 Predictive Coding
In this phase, three Transformer-like network branches are trained in a self-supervised manner to respectively learn the
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
N
S
Z
8
f
P
k
Y
u
g
C
/
5
w
L
M
q
p
>
A
B
H
c
V
D
F
3
W
r
2
K
d
0
7
U
T
y
m
Q
O
I
o
R
X
+
n
z
v
G
E
j
9
J
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
N
S
Z
8
f
P
k
Y
u
g
C
/
5
w
L
M
q
p
>
A
B
H
c
V
D
F
3
W
r
2
K
d
0
7
U
T
y
m
Q
O
I
o
R
X
+
n
z
v
G
E
j
9
J
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
A
g
W
L
m
T
Z
N
S
M
V
I
5
2
U
B
o
r
3
J
v
Q
>
+
X
c
8
E
K
O
k
9
F
j
0
P
D
w
7
d
C
/
z
u
G
H
f
q
y
p
R
n
Y
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
D
G
p
w
z
9
0
S
L
k
y
V
n
g
8
U
>
A
B
+
X
c
N
E
3
r
q
/
o
I
v
R
P
2
Q
u
7
J
d
C
j
W
M
Y
m
O
H
Z
K
f
F
5
T
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
5
+
C
X
P
j
m
W
3
8
I
c
H
D
N
/
f
Z
A
>
B
7
n
V
9
S
w
E
J
2
L
z
F
R
o
g
G
Q
v
k
y
d
T
M
Y
q
r
u
O
p
U
0
K
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
L
I
X
Q
f
N
F
k
Z
R
Y
U
w
g
P
0
A
>
B
/
c
V
D
S
J
3
2
v
+
5
C
m
q
o
M
u
G
K
H
p
O
T
z
d
7
E
9
W
r
j
8
n
y
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
y
M
Z
J
E
H
X
q
I
O
K
j
r
w
m
/
U
F
Y
>
A
B
7
n
c
V
N
S
8
3
9
L
0
o
C
f
d
u
Q
+
P
G
z
p
R
T
W
g
5
k
D
2
v
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
8
v
+
5
H
D
j
m
9
d
7
r
z
Z
F
k
R
3
g
o
>
A
B
/
X
c
V
J
S
N
E
O
G
L
y
Y
Q
w
K
n
p
I
0
u
P
W
T
f
U
q
C
M
2
relational, long-horizon, and short-horizon hidden representations. The key insight of so-called predictive coding [Elias, 1955; Spratling, 2017; Rao and Ballard, 1999] is to extract useful representations that are maximally beneficial for predicting future, missing or contextual information. These representations jointly form the compositional state space in the next training phase for learning the investment policy. Next, we first introduce an improved Transformer architecture and then discuss the predictive coding methods for the relational and predictive representations respectively.
Building block: Diversified multi-head attention. The diversity of temporal patterns among the concurrent sequences of multiple trading assets in financial markets (e.g., hundreds of stocks) greatly increases the difficulty of learning effective representations from raw data. To tackle this difficulty, as shown in Figure 1 (Left), we renovate the multi-head attention block in the original Transformer with a group of feed-forward networks (FFNs) rather than a single FFN, in which each FFN individually responds to a single head in the output of the multi-head attention layer. Without changing the overall number of parameters, such a mechanism strengthens multihead attention's original feature decoupling ability, which facilitates modeling the diverse temporal patterns in different subspaces. We thus refer to the modified attention block as diversified multi-head attention (DMH-Attn). For a set of dmodel-dimensional keys (K), values (V ), and queries (Q), the process in a diversified multi-head attention block can be represented as follows. We split the output features Z of the multi-head attention layer by h along the channel dimension, where h is the number of parallel attention heads, and then apply a separate FFN to each group of separated features in Z:
where "MH-Attn" denotes multi-head attention, fi denotes the output features of each FFN head, which contains two linear transformations with the ReLU activation in between.
General predictive coding architecture. Each Transformer branch in StockFormer can be divided into encoder and decoder layers, as shown in Figure 1 (Right). Both parts are used in the predictive coding phase with different training objectives, but only the encoder layers are used during policy optimization. We have L encoder layers and M decoder layers. The representations XL enc of the final encoder layer is used as one of the inputs of each decoder layer. The computation in the lth encoder layer and the mth decoder layer can be presented as follows:
Encoder Layer:
where Xl enc and Xm dec are the output of the encoder and decoder layer respectively. Specifically, X0 enc and X0 dec are the inputs of the the first encoder and decoder layers, which are the positional embedding [Vaswani et al., 2017] of the raw input data Oenc and Odec. XM dec by the last decoder layer are then fed into a projection layer to generate the final output in each predictive coding task, that is, predicting future returns or reasoning about the missing (masked) information in the financial market at this moment.
Relation inference module (1 st Transformer branch). As shown in Figure 2, the relation inference module is used to capture the dynamic correlations among concurrent time series, e.g., different stocks. At time step t, we use the same input for the encoder and the decoder, Orelat enc,t = Orelat dec,t ∈ RN×(K+N) , where N is the number of concurrent trading assets and K is the number of statistics that capture the dynamics of the time series data. Particularly for the stock market datasets, we use common technical indicators as the statistics, such as MACD, RSI, and SMA. During the training phase, Orelat enc,t can be divided into two parts:
- The covariance matrix Ocov t ∈ RN×N between sequences of daily close prices of all concurrent trading assets over a fixed period of time before t.
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
3
d
V
k
L
H
f
w
D
U
J
z
y
5
c
g
p
M
2
v
>
A
B
j
F
N
T
9
E
0
Q
7
G
I
K
o
R
S
C
W
q
/
r
X
m
n
P
Y
u
8
+
Z
O
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
O
V
M
B
v
+
L
T
Y
2
Q
d
D
0
R
N
P
g
>
A
C
3
c
j
H
J
F
U
K
r
I
z
q
o
G
p
7
9
f
/
Z
w
5
8
m
S
X
y
W
u
E
n
k
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
B
8
5
W
M
d
S
k
I
u
J
O
R
v
P
C
>
A
H
c
j
V
L
N
F
D
3
G
X
o
w
Q
U
q
p
0
K
g
E
Z
7
y
/
r
+
f
9
z
m
n
Y
T
2
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
O
j
0
T
k
A
C
G
Y
7
X
+
d
I
J
g
c
v
w
>
D
B
V
H
L
F
E
K
S
M
N
R
o
r
y
f
u
U
8
2
/
3
n
p
P
9
Z
5
q
z
W
Q
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
Z
J
V
A
d
C
o
r
w
Y
5
y
c
E
m
R
K
H
F
>
3
j
L
T
D
U
+
N
I
z
Q
q
G
0
p
B
k
v
g
7
9
f
/
P
2
8
S
X
W
u
M
n
O
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
w
o
J
T
L
c
k
F
N
r
Z
O
g
+
8
3
Y
U
>
A
C
H
V
7
S
B
G
d
p
z
j
m
M
I
9
2
X
W
D
5
E
n
P
v
0
K
q
Q
f
R
/
u
y
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
Z
y
K
9
L
N
u
8
/
U
C
r
+
I
2
g
j
3
V
A
>
n
c
H
S
F
D
p
f
q
7
d
B
v
R
0
G
X
k
5
z
E
O
P
o
W
T
Q
m
J
w
Y
M
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
m
B
H
9
O
T
j
W
g
f
D
w
+
n
X
S
d
K
o
0
>
A
/
c
V
L
N
F
J
3
U
2
v
u
k
Z
I
q
M
R
C
8
G
z
P
7
Y
p
E
Q
5
y
r
<
l
a
t
e
x
i
s
h
1
_
b
6
4
=
"
Q
B
S
f
Y
n
m
k
7
o
+
v
J
N
w
3
c
G
8
>
A
C
K
X
Z
D
L
F
I
9
V
r
z
W
R
p
q
M
2
0
y
E
u
H
O
d
5
j
U
g
/
T
P
- The masked statistics Ostat t ∈ RN×K , in which we randomly select half of the time series and mask their input statistics by zero. It is worth noting that in the test phase, we use complete data without masks as input to the module.
The task of the relation inference module is to reconstruct the masked statistics (i.e., technical indicators) given Ocov t and the remaining visible statistics. The self-supervised loss function in this module can be represented as:
where ̂Ostat t and ¯Ostat t are the reconstructed values of the masked statistics and their ground truth. Intuitively, such a predictive coding method drives the Transformer encoder to capture the dependencies among the concurrent time series (i.e., stocks). The relation inference module provides its final encoding features (i.e., XL enc in the above descriptions of the general architecture) for the subsequent decision module, as parts of the state space of the RL agent, denoted by srelat t .
Future prediction modules (2 nd &3 rd branches). In the short-term prediction module, the task is to predict the return ratio of each stock 1 for the next day (H = 1). In this module, we use different inputs for the encoder and the decoder. The inputs of the encoder are Oprice enc,t ∈ RT×5 , which involve daily opening, close, highest, lowest prices, and trading volumes during the past T days. We feed the transaction records at step t to the decoder, i.e., Oprice dec,t ∈ R1×5 . Similar to the shortterm prediction module, the long-term prediction module is to predict the return ratios in a longer future horizon (e.g., H = 5 for daily stock trading). The long-term prediction task encourages the Transformer branch to capture future dynamics from a longer perspective. We adopt the regression loss and stock ranking loss from [Feng et al., 2019]:
1 We here take daily stock trading as an example.
where rt+H = [rt+H 1 , . . . , rt+H N ] denotes the true future return ratios of all stocks, rt+H i = (pt+H i - pti )/pti , pti is the closing price of stock i on day t, ˆrt+H = [ˆ rt+H 1 , . . . , ˆ rt+H N ] is the predicted return ratios of all stocks, and α is a hyperparameter used to balance the weight of the two loss terms. The second term encourages the predicted ranking of any pair of stocks to maintain the same order as the ground truth. We here use the output features of the decoders (i.e., XM dec before the projection layer) in the future prediction modules as parts of the states of the RL agent (i.e., slong t and sshort t ).
4.2 Policy Optimization
In the second training phase, StockFormer integrates the three types of latent representations into a unified state space S ⊆ RN×(D+1) through a series of multi-head attention layers and then exploits an actor-critic method to learn the trading policy. StockFormer learns flexible trading strategies by taking full advantage of the future trends of each time series in the long/short time horizon, as well as the dynamic relationships between dozens of concurrent trading assets.
Latent states integration. With predictive coding, we obtain informative latent representations from three Transformer branches, denoted by srelat t ∈ RN×D , slong t ∈ RN×D , and sshort t ∈ RN×D . Therefore, the first question in the decisionmaking module is how to integrate different types of predictive embeddings into a unified state space. We use a cascaded multi-head attention mechanism as follows so that the decision module can be informed with the predicted future information in the financial market, as well as the dynamic relations between different trading targets.
where sfuture t and sht are the outputs from each multi-head attention layer, slong t is used as the query in the first MH-Attn unit as it is less vulnerable to short-term noise than sshort t and sfuture t is used as the query to extract more effective relational features. We combine sht with the shares of all trading targets that we hold at a certain time step, shold t ∈ RN×1 , to form the final states st ∈ RN×(D+1) , and then feed st into the actor network and the critic network. In the experiments, we empirically show the effectiveness of the proposed architecture of the decision module: Exploiting a series of multi-head attention layers better integrates different sources of latent representations into a unified state space, which can eventually benefit policy optimization.
Joint learning with RL objectives. We use the soft actorcritic (SAC) [Haarnoja et al., 2018] to learn the trading policy in the unified state space. The critic network learns the parametric soft Q-function by minimizing the following soft Bellman residual and propagates the analytic gradients of state values back into the relation inference module which is adopted from previous actor-critic approaches with visual inputs [Lee et al., 2020; Yarats et al., 2021]. In other words, we allow for the joint training of the two training phases of predictive coding and policy learning in this stage, so that the critic's evaluation of the state values can further help to mine the correlations between the trading assets from noisy and high-dimensional observation data. The advantage of learning a hybrid trading machine is that the two training phases can be deeply bound, which integrates the forward modeling capabilities of predictive coding with the advantages of Rl-for-finance methods for the flexibility of the trading policy. Specifically, we optimize the critic loss J (Q) by
where θ, ϕ, and ψ represent the parameters from the actor network, the critic network, and the relation inference module respectively, D is the replay buffer, Qϕ′ is the target Q network and at+1 ∼ πθ (· | st+1). For the actor loss J (πθ), we have
where Zϕ is a normalization factor.
5 Experiments
5.1 ComparedMethods
We compare StockFormer with the following methods:
- The "market benchmarks" in terms of CSI-300 Index and NASDAQ Composite Index.
- Min-variance portfolio allocation strategy (Min-Var) [Basak and Chabakauri, 2010] that balances returns and risks.
- Recent advances for stock prediction [Wang et al., 2021; Feng et al., 2019; Duan et al., 2022] and a cutting-edge Transformer model for general time series prediction [Wu et al., 2021]. We here use the "buy-and-hold" strategy, in which we buy a stock each day that has the highest estimated return in the next 5 days and sell it 5 days later .
- RL methods, including SAC [Haarnoja et al., 2018], DDPG [Lillicrap et al., 2016], and SARL [Ye et al., 2020]. The first two models are implemented on the FinRL platform [Liu et al., 2021] and directly perform policy learning on the technical indicators and the covariance matrix between the trading targets. We reproduce the SARL and feed the current asset prices (rather than the financial news) to its external state encoder for a fair comparison.
| Market | # Assets | # Train Days | # Test Days |
|---|---|---|---|
| CSI-300 | 88 | 1935 | 728 |
| NASDAQ-100 | 86 | 2000 | 756 |
| Cryptocurrency | 27 | 1108 | 258 |
All models are experimented with the transaction costs and each RL method is performed three times with different seeds.
5.2 CSI-300 Stock Dataset
This dataset contains stock data from the CSI-300 Composite Index from 01/17/2011 to 12/30/2021. As shown in Table 2, the dataset is divided into the training and test splits containing the basic stock price-volume information in 1,935 days and 728 trading days respectively. Furthermore, we follow [Feng et al., 2019] to retain the stocks that have been traded on more than 98% training days since 01/17/2011. Our investment pool contains 88 stocks. If a stock in the training set is suspended from trading, we interpolate the missing data in using the daily rate of change of the CSI-300 Composite Index.
Figure 3(Left) shows qualitative results of the accumulated portfolio return on the CSI-300 test set. It shows that StockFormer outperforms other compared models by large margins, including both stock prediction models trading with the "buy-and-hold" strategy, as well as the RL-for-finance methods. Notably, the red curve presents the results of the proposed long-term prediction module without involving any other branches or the decision module in StockFormer, which is trained to predict future returns in 5 days and works with the "buy-and-hold" strategy as other stock prediction models. The remarkable advantage of this baseline model compared with existing approaches, including HATR [Wang et al., 2021], Relational Ranking [Feng et al., 2019], and AutoFormer [Wu et al., 2021], demonstrates the powerful dynamics modeling capability of the proposed Transformer architecture with diversified multi-head attention.
Table 3 provides corresponding quantitative results. Besides the total portfolio return (PR), we follow the work of FactorVAE [Duan et al., 2022] to include annual return (AR), Sharpe ratio (SR), and maximum drawdown (MDD) as the evaluation metrics. It is worth noting that the final StockFormer further improves the future prediction baseline (Row 7) and significantly outperforms all of the stock prediction models by learning flexible RL strategies. StockFormer also outperforms the vanilla SAC by 40.3% (1.76 → 2.47) in the portfolio return and by 22.7% (1.41 → 1.73) in the Sharpe ratio. Such a profit boost comes from executing policy optimization over the extracted relational and predictive states.
| Method | CSI-300 NASDAQ-100 | Cryptocurrency | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| PR↑ | AR↑ | SR↑ | MDD↓ | PR↑ | AR↑ | SR↑ | MDD↓ | PR↑ | SR↑ | |
| Market benchmark | 0.24 | 0.08 | 0.51 | 0.18 | 1.05 | 0.30 | 1.16 | 0.30 | - | - |
| Min-Var [Basak and Chabakauri, 2010] | 0.11 | 0.04 | 0.38 | 0.13 | 0.59 | 0.18 | 0.99 | 0.26 | -0.09 | 0.02 |
| HATR [Wang et al., 2021] | -0.02 | -0.01 | 0.14 | 0.45 | 0.08 | 0.28 | 0.25 | 0.41 | -0.65 | -0.66 |
| Relational Ranking [Feng et al., 2019] | -0.01 | -0.01 | 0.17 | 0.45 | 0.96 | 0.28 | 0.95 | 0.30 | - | - |
| FactorVAE [Duan et al., 2022] | 1.37 | 0.38 | 1.27 | 0.17 | 1.20 | 0.33 | 0.99 | 0.24 | - | - |
| AutoFormer [Wu et al., 2021] | 0.08 | 0.03 | 0.25 | 0.40 | -0.25 | -0.10 | -0.21 | 0.42 | -0.27 | -0.16 |
| Our future prediction module (t + 5) | 0.89 | 0.27 | 0.73 | 0.49 | 0.66 | 0.20 | 0.64 | 0.44 | -0.40 | -0.71 |
| SARL [Ye et al., 2020] | 1.59 | 0.39 | 1.38 | 0.31 | 1.22 | 0.30 | 0.99 | 0.34 | 0.06 | 0.43 |
| FinRL-SAC [Liu et al., 2021] | 1.76 | 0.42 | 1.41 | 0.34 | 1.38 | 0.34 | 1.24 | 0.33 | 0.10 | 0.55 |
| FinRL-DDPG [Liu et al., 2021] | 1.43 | 0.36 | 1.23 | 0.39 | 0.83 | 0.22 | 0.87 | 0.33 | 0.15 | 0.60 |
| StockFormer | 2.47 | 0.54 | 1.73 | 0.31 | 1.71 | 0.40 | 1.39 | 0.31 | 0.24 | 0.75 |
5.3 NASDAQ-100 Stock Dataset
For the NASDAQ stock dataset, we collect the daily pricevolume records and related technical indicators between 01/17/2011 and 12/30/2021 from Yahoo Finance. Like in CSI300, we use the 98% criteria to filter stocks, which derives an investment pool of 86 stocks, and then fill in the missing data based on the daily rate of change of the NASDAQ 100 Index. We construct the training and test datasets that involve 2,000 and 756 trading days respectively. As shown in Table 3 and Figure 3(Right), on the NASDAQ-100 test set, due to the effectiveness of predictive coding, StockFormer improves the vanilla SAC by 23.9% (1.38 → 1.71) in the portfolio return and by 12.1% (1.24 → 1.39) in Sharpe ratio.
5.4 Cryptocurrency Dataset
We further evaluate StockFormer by using it to make trading decisions in the cryptocurrency market. With the 98% filtering criteria, we select 27 cryptocurrencies and collect their daily records between 11/01/2017 and 05/01/2022 from Yahoo Finance. We split the data into a training set of 1,108 trading days and a test set of 258 trading days.
An interesting result in Table 3 is that most stock prediction models present negative return ratios on the test set, even worse than the Min-Var baseline. We suspect that this is due to the complexity and the non-stationarity of the cryptocurrency data. In other words, it is difficult to model the temporal dynamics accurately in such data scenarios, which further highlights the advantage of the model-free RL methods. For FinRL, SARL, and StockFormer, they do not explicitly model the transition function in the state space, and can flexibly produce more conservative trading policies (as shown by the Sharpe ratios).
| Input latent states of RL agents | PR | SR |
|---|---|---|
| W/o relational states | 1.42 | 1.43 |
| W/o short-term predictive states | 1.53 | 1.43 |
| W/o long-term predictive states | 1.45 | 1.34 |
| StockFormer (Final) | 2.47 | 1.73 |
| Relation module | Future prediction modules | PR | SR |
|---|---|---|---|
| FFN | FFN | 1.29 | 1.27 |
| Multi-head FFN | FFN | 1.48 | 1.29 |
| FFN | Multi-head FFN | 1.52 | 1.31 |
| Multi-head FFN | Multi-head FFN (StockFormer) | 1.71 | 1.39 |
Another interesting fact in Table 3 is that while the future prediction module in StockFormer results in a negative portfolio return when working alone, its predictive coding states can still benefit the hybrid trading machine when integrated into StockFormer. We can see that StockFormer achieves positive and the best performance, which has a 140.0% (0.10 → 0.24) improvement over the vanilla SAC method in the portfolio return and a 36.4% (0.55 → 0.75) improvement in Sharpe ratio, showing the ability to control the investment risks.
5.5 Ablation Study
Necessity of individual Transformer branches. To verify the performance of the relation inference module and future prediction modules, we compare three variants of StockFormer on the CSI dataset. From Table 4, we can see that removing any encoding branch leads to a remarkable performance drop, and each branch makes essential contributions to the final performance of our final approach. The results demonstrate that the relation module and prediction modules provide complementary representations to the decision module from different perspectives of concurrent time series modeling.
Performance gain of the modified attention block. Table 5 compares the results of three StockFormer variants, in which we may use the original Transformer architecture without the proposed multi-head FFNs in any one of the three predictive coding branches. These results show that multi-head FFNs can effectively improve the attention block in Transformer by mitigating the difficulty of concurrent time series modeling.
Design of the decision module. An important question is how to integrate different types of predictive embeddings into a unified state space. To answer this question, we study different network structures for fusing the relational and long-/shortterm predictive states in the decision module and show the results in Table 6. By replacing the multi-head attention layers in StockFormer with two fully connected layers (denoted by "2-layer FFN"), we observe that the portfolio return drops from 1.71 to 1.62 and the Sharpe ratio drops from 1.39 to 1.31. The results validate the effectiveness of the proposed architecture of the decision module: Exploiting a series of multi-head attention layers better integrates different sources of latent representations into a unified state space, which can eventually benefit policy optimization.
| State fusion in decision module | PR | SR |
|---|---|---|
| 2-layer FFN | 1.62 | 1.31 |
| 2-layer multi-head attention (StockFormer) | 1.71 | 1.39 |
| Actor gradient | Critic gradient | PR | SR |
|---|---|---|---|
| ✗ | ✗ | 0.03 | 0.35 |
| ✓ | ✗ | 0.02 | 0.35 |
| ✗ | ✓(StockFormer) | 0.24 | 0.75 |
Joint training vs. Two separate training stages. The two training phases, predictive coding and policy learning, are closely related. In Table 7, we experiment with different gradient back-propagation solutions in the policy learning phase of the actor-critic method. We observe that StockFormer achieves the best results by passing the gradients of the critic loss back into the relation inference module. It significantly improves the baseline model with two separate training phases (PR: 0.03 → 0.24), in which the RL agent does not propagate any gradients back to the predictive coding modules. We believe that the critic's evaluation of the state values can further help to mine the correlations between the trading assets from noisy and high-dimensional observation data. These results show the superiority of the joint learning mechanism.
6 Conclusions
This paper presents StockFormer, an RL approach for financial market decision-making. It has two main contributions. First, it incorporates predictive coding in the actor-critic RL framework, which integrates the advantages of existing stock prediction models in learning future dynamics and the advantages of RL-for-finance methods in learning more flexible polices. Second, we proposed to use three branches of the modified Transformers to learn a mixture of long-/short-term predictive states and relational states. These three types of states are then combined adaptively and progressively through a series of attention structures in the decision-making module, such that the RL agent can optimize the decisions in a unified and meaningful state space. StockFormer shows competitive results on three finance datasets compared with a wide range of deep learning approaches, including both stock prediction and RL-for-finance models.
Acknowledgements
This work was supported by NSFC (62250062, U19B2035, 62106144), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), the Fundamental Research Funds for the Central Universities, and Shanghai Sailing Program (21Z510202133) from the Science and Technology Commission of Shanghai Municipality.
@inproceedings{ijcai2023p530,
title = {StockFormer: Learning Hybrid Trading Machines with Predictive Coding},
author = {Gao, Siyu and Wang, Yunbo and Yang, Xiaokang},
booktitle = {Proceedings of the Thirty-Second International Joint Conference on
Artificial Intelligence, {IJCAI-23}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Edith Elkind},
pages = {4766--4774},
year = {2023},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2023/530},
url = {https://doi.org/10.24963/ijcai.2023/530},
}
TY - CPAPER TI - StockFormer: Learning Hybrid Trading Machines with Predictive Coding AU - Gao, Siyu AU - Wang, Yunbo AU - Yang, Xiaokang T2 - Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 PB - International Joint Conferences on Artificial Intelligence Organization PY - 2023 SP - 4766 EP - 4774 DO - 10.24963/ijcai.2023/530 UR - https://doi.org/10.24963/ijcai.2023/530 ER -Copied
Machine-readable versions of this paper, for tools and AI agents:
