Enhancing Portfolio Optimization via Heuristic-Guided Inverse Reinforcement Learning with Multi-Objective Reward and Graph-based Policy Learning

1 Introduction

Portfolio optimization in dynamic financial markets presents the following three fundamental challenges: balancing riskreturn trade-offs under uncertainty [Markowitz, 1952], integrating domain-specific knowledge into algorithmic frameworks [Brandt, 2010], and modeling nonlinear asset interdependencies [Pflug et al., 2012].

∗Corresponding author.

As illustrated in Figure 1, effective risk management distinguishes between reducible factors (e.g., asset selection, industry spread, and correlation control) and irreducible ones. Modern portfolio theory establishes that the reduction of unsystematic risk follows from the following equation:

where σ2 p denotes the total portfolio variance, N represents the number of constituent assets, σ2 indicates the average idiosyncratic variance (i.e., asset-specific risks), ρ signifies the mean correlation coefficient between assets, and σ2 denotes the systematic market variance. As such, the first term 1N σ2 quantifies the reducible idiosyncratic risk that diminishes with increasing asset quantity. Consequently, we can see that (1) a higher number of assets dilutes idiosyncratic risk, (2) industry diversification mitigates sector-specific volatility [Ang et al., 2009], and (3) low asset correlation amplifies diversification efficacy [Choueifaty and Coignard, 2008], isolating non-systematic risk for reduction.

Traditional reinforcement learning (RL) approaches, while adaptive to market dynamics, suffer from three critical limitations in financial applications. First, their manual reward engineering often fails to capture essential market patterns, such as momentum effects [Jegadeesh and Titman, 1993] and regime-dependent correlations [Longin and Solnik, 2001]. Second, standard RL architectures struggle to meet the requirements for sector diversification [Roncalli, 2013]. Third, their fully-connected networks do not adequately model market microstructure relationships [Aste et al., 2010]. These limitations often lead to unstable performance during black swan events [Taleb, 2010], as evidenced by the 2020 COVID19 market crash [Baker, 2020]. To address these gaps, we propose a heuristic-guided inverse reinforcement learning (IRL) framework for portfolio optimization, with the following three main components.

• The optimization module formalizes domain knowledge (sector weight caps and momentum-correlation control from [Moskowitz et al., 2012]) into synthetic expert trajectories. This overcomes data scarcity in conventional IRL methods [Ng and Russell, 2000] while maintaining financial logic consistency [Buehler et al., 2019].
• The multi-objective reward learning mechanism dynamically balances four competing objectives, namely return maximization, sector diversification [Koumou, 2020], momentum alignment [Geczy and Samonov, 2016], and correlation penalization [Buraschi et al., 2010]. The adaptive weight adjustment mechanism enables automatic rebalancing during shifts in market regimes.
• The graph-based policy network advances prior financial models through three structural innovations: explicit encoding of sector affiliation graphs, correlation graphs derived from tail dependence coefficients, and hierarchical attention mechanisms combining local asset features with asset indicators. This design enhances microstructure modeling while improving generalization ability through adaptive information fusion.

Extensive experiments and case studies on real-world financial market data demonstrate that our framework consistently outperforms several state-of-the-art baselines in terms of risk-return trade-offs and robustness to extreme events, offering a novel technical pathway for adaptive portfolio management.

2 RelatedWork

Portfolio optimization has evolved through three research paradigms. The foundational work of Markowitz [1952] established mean-variance optimization but suffered from static assumptions and sensitivity to estimation errors [Choueifaty et al., 2013]. Subsequently, some dynamic extensions, such as stochastic portfolio theory [Fernholz and Karatzas, 2005], improved market adaptability but grappled with the curse of dimensionality in multi-asset scenarios. Then, reinforcement learning (RL) emerged as a promising alternative for dynamic portfolio management. Pioneering work by Moody et al. [1998] demonstrated the potential of RL in financial time series prediction. At the same time, recent deep RL approaches [Deng et al., 2016] achieved superior risk-adjusted returns through neural policy networks. However, these methods are critically dependent on manually designed reward functions that oversimplify market complexity, a limitation extensively documented by Hambly et al. [2023] in their survey on RL-based trading systems. Inverse RL (IRL) attempted to address reward engineering challenges by learning

Algorithm 1 Greedy Expert Strategy Generation

Input: Historical returns rt, industry relation matrix I, correlation matrix C, total number of stocks N, number of stocks to select K Parameter: max industry ratio α ∈ (0, 1), threshold γ ∈ (0, 1) Output: Expert action vector at 1: Ranking all stocks based on rt in descending order as C 2: Calculate K′ ← ⌊αK⌋ 3: Initialize at ← 0 ∈ {0, 1}N and S ← ∅ 4: while |S| < K and C ̸= ∅ do 5: i ← C.pop 6: Ii ← {j | I[i, j] > 0} ∪ {i} 7: ki ← ∑j∈Ii at[j] 8: if ki ≥K′ then 9: continue 10: if S ̸= ∅ then 11: ρi ← 1 |S| ∑j∈S C[i, j] 12: if ρi ≥ γ then 13: continue 14: at[i] ← 1 and S.add(i) 15: return at

implicit objectives from expert demonstrations. The foundational IRL framework [Ng and Russell, 2000] has inspired financial applications to systematically incorporate domain knowledge through heuristic-guided reward learning.

Recently, graph-based approaches have demonstrated potential in modeling market structures. In [Chen et al., 2018], graph neural networks are employed to capture stock correlations, while temporal graphs are integrated for financial time series prediction in [Xiang et al., 2022]. Some pioneering works [Wang et al., 2019; Wang et al., 2021] utilized graph neural networks as the RL decision model to enhance the understanding of cross-asset interdependencies in portfolio management. Our heterogeneous graph attention mechanism advances these works by explicitly modeling sector hierarchies and correlation dependencies in a unified architecture, which is particularly crucial for handling cross-market portfolio optimization. The incorporation of financial domain knowledge into adaptive learning mechanisms is based on hybrid approaches such as those in [Xiong et al., 2018; Liu et al., 2021] but significantly extends these paradigms through the systematic integration of interpretable heuristics, multi-objective reward optimization, and structured market graph representation learning.

3 Methodology

In this section, we present our proposed framework in detail. The architecture of the framework is illustrated in Figure 2.

3.1 Greedy Expert Strategy Generation with Heuristic Rules

We utilize a greedy algorithm to generate high-quality portfolio trajectories through iterative stock selection, where sector diversification and correlation constraints are incorporated. The detailed expert strategy generation procedure is described in Algorithm 1.

The process begins with ranking stocks based on their historical returns in descending order, ensuring that the most promising stocks are considered first. To manage the risk of sector concentration, we impose sector diversification constraints, limiting the maximum ratio per industry cluster to α ∈ (0, 1). That is, the number of selected stocks in each industry sector cannot exceed ⌊αK⌋, where K is the total number of stocks to select. Additionally, the strategy dynamically excludes candidates with an excessive correlation with selected stocks. If the average correlation of the stock i with the selected set S of stocks exceeds a threshold γ, given by 1 |S| ∑j∈S ρij ≥ γ, the stock will be excluded from consideration. Finally, the expert strategy outputs a binary action vector at ∈ {0, 1}N , indicating which stocks are selected for the portfolio.

3.2 Multi-Objective Reward Learning

The multi-objective reward learning component enhances the strategy by integrating four objectives: maximizing returns, diversifying across sectors, penalizing positive correlations, and incentivizing negative correlations.

Return Maximization

The return reward is based on the portfolio log-return, which is calculated as:

Sector Diversification

To promote diversification, the reward function maximizes the entropy of sector weights, defined as

Correlation Management

The positive correlation penalty discourages positive correlations among low-momentum assets, given by:

Conversely, the negative correlation incentive promotes negative correlations with high-momentum assets, represented as:

Adaptive Reward Function

The composite reward is a weighted sum of these components

with weights λi dynamically adjusted via Lagrangian duality.

3.3 MaximumEntropy Inverse Reinforcement Learning

The Maximum Entropy Inverse Reinforcement Learning (i.e., MaxEntIRL) framework, originally proposed in [Ziebart et al., 2008], addresses the ambiguity problem in estimating the reward function by preferring the reward function that maximizes entropy over the demonstrated trajectories. Our implementation extends this principle to portfolio optimization, adapting the formulation to handle non-stationary market conditions and partial observability.

Expert-Agent Reward Gap

We model the expert-agent reward loss function as:

which encourages it to learn from the expert's behavior.

We model the likelihood of expert trajectories τE under the learned reward function Rθ as:

This leads to the negative log-likelihood loss function:

where the first term encourages high rewards for expert trajectories, while the second term penalizes arbitrary scaling of rewards for agent trajectories πA, as discussed in [Finn et al., 2016]. The gradient of this loss with respect to reward parameters θ becomes

This formulation ensures that the agent policy πA covers the expert's behavior distribution while maintaining maximum entropy, crucial for handling the stochastic nature of financial markets.

Reward Network Optimization

The reward network implements a modular architecture that dynamically combines multiple financial factors through parameterized weighting. Formally, the network processes state-action pairs (st, at) through parallel encoder streams as follows:

where K = {base, ind, pos, neg} denotes the active feature modalities, ⊕ represents feature-action concatenation, and βk are learnable weights with ∑ βk = 1 through softmax normalization. The reward network parameters θ are updated via gradient clipping:

Policy Optimization

The agent policy πA is then trained using Proximal Policy Optimization (PPO) to maximize cumulative rewards under the updated reward function:

3.4 Graph-Based Policy Network

Finally, the graph-based policy learning component leverages a heterogeneous graph attention network (HGAT) to dynamically model industry relations and correlations among stocks.

Graph Construction

The industry graph Aind ∈ RN×N is constructed so that Aind,ij = 1 if stocks i and j belong to the same sector and 0 otherwise. The positive correlation graph Apos ∈ RN×N is defined as Apos,ij = 1 if correlation ρij > ρthreshold, and 0 otherwise. The correlation values are discretized as:

Multi-Head Graph Attention Encoding

Multi-head graph attention encoding is applied to the input features X ∈ RN×d , computing the attention embeddings for each graph g as H(l) g = MH-GAT(X,Ag), where the attention mechanism is calculated as:

where Q,K,V are linear transformations of the input features, and d is the feature dimension.

Heterogeneous Fusion Attention

The heterogeneous fusion attention stage concatenates the multi-head output Hind,Hpos,Hneg with raw features, calculated as:

where βk are adaptively learned weights:

Policy Generation

The fused embeddings are then fed into a generator to output normalized portfolio weights as:

where Wg is a fully-connected layer and Flatten(·) denotes the vectorization operation.

4 Experiments

In this section, we present our experimental setup and results.

4.1 Experimental Setup

To evaluate the performance of our method, we conduct extensive experiments on real-world data across the Chinese and US markets. Specifically, our evaluation is performed on the constituent stocks of the CSI 300, CSI 500, NASDAQ 100, and S&P 500 indices, covering the period from January 2018 to December 2024. Each dataset is partitioned into three subsets: a training set (from 2018 to 2022), a validation set (2023), and a test set (2024). About 5% of the stocks were discarded due to incomplete data. Raw financial data includes daily open, close, high, low, previous close prices, and trading volume. These features are normalized using rolling-window standardization (the window size is set to 20 days) and further grouped via k-means clustering for intra-group normalization. Monthly correlation matrices are computed using Pearson's correlation coefficients, with positive correlation graphs (edges for ρij > 0.2) and negative correlation graphs (edges for ρij < -0.2) generated to capture dynamic market interactions.

We compare our method with state-of-the-art deep learning (DL) models (including LSTM [Hochreiter and Schmidhuber, 1997], GRU [Chung et al., 2014], ALSTM [Feng et al., 2019], Transformer [Vaswani et al., 2017], PatchTST [Nie et al., 2023], DLinear [Zeng et al., 2023], iTransformer [Liu et al., 2024], Crossformer [Zhang and Yan, 2023], and MASTER [Li et al., 2024]), reinforcement learning (RL) models (including AlphaStock [Wang et al., 2019], DeepTrader [Wang et al., 2021], and DeepPocket [Soleymani and Paquet, 2021]), and large language model (LLM) models for time series (including GPT4TS [Zhou et al., 2023], TIME-LLM [Jin et al., 2024], and aLLM4TS [Bian et al., 2024]).

We use the following six metrics for performance evaluation: Annualized Return Rate (ARR), Annualized Volatility (AVol), Maximum Drawdown (MDD), Sharpe Ratio (SR), Calmar Ratio (CR), and Information Ratio (IR). To eliminate fluctuations, we average the metrics over three repeated tests for each model. Hyperparameters are configured with a learning rate of 10-4 , a batch size of 128, and HGAT policy networks featuring 128-dimensional hidden layers, 8 attention heads, and 200 training epochs.

Correlation matrices and graph structures are generated monthly, producing industry relation graphs (Aind), positive correlation graphs (Apos) and negative correlation graphs (Aneg). Synthetic expert trajectories are created to enforce sector diversification constraints and correlation control rules. We construct training and prediction datasets by encapsulating time-series features, graph structures, and labels into PyTorch Geometric Data objects. The IRL training loop initializes a multi-objective reward network and an HGAT policy network, alternately optimizing the reward function via MaxEntIRL and the actor-critic policy via Stable-Baselines3 PPO.

4.2 Performance Analysis

The overall results of all methods across four datasets are presented in Table 1. These results demonstrate the superior performance of our proposed method in different scenarios. Our method consistently outperforms the state-of-theart baseline models in terms of key return metrics, especially ARR, SR, and CR. In addition to superior returns, our method also demonstrates competitive performance in terms of risk metrics such as AVol and MDD. On the CSI 300 dataset, our method significantly outperforms the baseline methods with an ARR of 0.491, surpassing the closest competitor (iTransformer: 0.372) by 31.9%. This superiority extends to riskadjusted returns, as evidenced by the highest SR (1.777) and CR (5.134), indicating enhanced returns per unit of risk and drawdown. In particular, while our method exhibits moderate volatility (AVol: 0.225), its MDD (-0.095) remains competitive, slightly under-performing ALSTM (-0.073) but demonstrating improved resilience compared to other models. On the NASDAQ 100 dataset, our method achieves the highest ARR (0.432), outperforming Transformer (0.258) by substantial margins. However, its IR (0.433) lags behind other models, suggesting potential opportunities to refine risk-adjusted performance in highly volatile markets. On the CSI 500 dataset, our method achieves a remarkable Annualized Return Rate (ARR) of 0.710, outperforming all baseline models by a significant margin (+36.8% over GPT4TS: 0.519 and +28.1% over TIME-LLM: 0.554). This superior return generation is complemented by competitive risk management, as evidenced by an Annualized Volatility (AVol) of 0.290, which is lower than most models, except DeepPocket (0.260). While the MDD (-0.161) is slightly higher than PatchTST's (-0.129), our method demonstrates exceptional risk-adjusted performance with the highest Sharpe Ratio (SR: 1.847) and Calmar Ratio (CR: 4.406), indicating superior returns per unit of risk and drawdown. However, the Information Ratio (IR: 1.058) trails TIME-LLM (1.470), suggesting potential enhancements in aligning returns with benchmark consistency. For the S&P 500 dataset, our method achieves an ARR of 0.250, which, although lower than GPT4TS (0.321), is accompanied by the lowest AVol (0.117) and a competitive MDD (-0.058), outperforming even DeepTrader (-0.049) when normalized against return metrics. Its SR (1.906) and CR (4.293) rank second only to GPT4TS (2.034 and 4.346), highlighting a robust risk-return trade-off. In particular, the model's IR (1.184) remains moderate compared to GPT4TS (1.872), indicating room for improvement in benchmarkrelative efficiency. Comparative analysis reveals distinct strengths across models. GPT4TS dominates absolute returns on the S&P 500 dataset but exhibits higher volatility (AVol: 0.157), while DeepTrader achieves the lowest MDD (-0.049) at the cost of suboptimal returns. PatchTST excels in minimizing drawdowns on the CSI 500 dataset (-0.129) but lags in ARR. Our method's hybrid architecture-likely integrating temporal feature extraction with dynamic risk mitigationenables balanced performance, prioritizing high returns and stability.

In summary, our method establishes state-of-the-art performance on the CSI 300, NASDAQ 100, and CSI 500 datasets and delivers competitive, low-volatility results on the S&P 500 dataset, indicating that it effectively leads to more robust and adaptive portfolio management.

4.3 Ablation Study

To validate the effectiveness of key components in our proposed model, we perform an ablation study by sequentially removing specific modules while keeping other modules and parameters unchanged. The quantitative results are shown in Table 2, which are further supplemented by cumulative return curves in Figure 3.

The quantitative results and cumulative return curves confirm the following findings. The full model maintains superior risk-adjusted growth with smoother equity trajectories, while ablated variants exhibit either lower terminal returns or higher volatility. They collectively demonstrate that each proposed component makes a unique contribution to the model's robustness and profitability.

Necessity of Reward Network (RN) Learning

Removing the dedicated reward network and directly using portfolio returns severely degrades the performance of the framework. The 38.9% decrease in the Annualized Return Rate (ARR) and 34.4% reduction in the Sharpe Ratio (SR) demonstrate the critical role of RN. This aligns with our conjecture that the handcrafted reward mechanism better captures nonlinear market dynamics than raw returns.

Effectiveness of Industry Diversification

Although removing industry constraints can slightly improve ARR (+2.7%), it causes significantly deeper drawdowns (- 15.0% vs. -10.1%) and lower CR. This indicates a risk-return trade-off: Industry concentration may boost short-term returns but increases portfolio volatility, which justifies the role of diversification constraints in our framework.

Effectiveness of Correlation Management

The 12.9% reduction of ARR and SR when turning off correlation control highlights its importance in stabilizing returns. This module appears particularly effective in avoiding simultaneous drawdowns of highly correlated assets.

Effectiveness of Heterogeneous Graph Attention Network (HGAT)

Replacing the HGAT with MLP degrades all evaluation metrics, particularly with ARR decreasing by 19.1%. This validates our design choice to leverage graph-structured market data through attention mechanisms, capturing complex asset relationships.

4.4 Case Study: Market Regime Analysis

To validate the role of heuristic rules in our framework, we present regime-specific case studies by visualizing the portfolio sector distributions and stock correlation heatmaps (where darker red indicates stronger positive correlation and darker blue indicates stronger negative correlation) across two contrasting market conditions in Figures 4 (bull market: 2024.9) and 5 (bear market: 2024.10). These visualized results reveal the fundamental behavioral patterns in our framework. The portfolio provided by the full model shows balanced exposure across defensive/cyclical sectors in both market regimes, while the ablated model exhibits excessive concentration in dominant sectors. In the bull market dynamic, the full model prioritizes momentum continuation through sustained exposure to growth sectors. While in a bear market dynamic, it activates mean-reversion patterns by shifting to undervalued (negative-related) assets, whereas the ablated model maintains momentum-chasing in declines. These heuristic rules for adaptation enable dynamic balancing between maximizing returns and containing risk across market cycles.

5 Conclusion

This paper proposes a novel heuristic-guided inverse reinforcement learning (IRL) framework for portfolio optimization that effectively integrates financial domain expertise with advanced AI techniques. Our experimental results demonstrate that our proposed framework achieves significant improvements in performance and versatility across both Chinese and US markets, enhancing Sharpe and Calmar ratios considerably compared to existing DL, RL, and LLM-based methods and exhibiting robustness through controlled drawdowns during periods of market volatility. These promising results suggest the potential of our framework to bridge traditional portfolio management with modern AI techniques, thereby contributing to the development of more effective investment strategies in an increasingly complex financial landscape. Future work will explore further the applications of our framework in real-world trading scenarios.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant Number 62202169 and the Shanghai Sailing Program 23YF1410600.