Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance

Multimodal Cancer Survival Analysis via Hypergraph Learning with Cross-Modality Rebalance

Mingcheng Qu1 , Guang Yang1 , Donglin Di2 , Tonghua Su1 , Yue Gao2 , Yang Song3 and Lei Fan3∗

1Faculty of Computing, Harbin Institute of Technology

2School of Software, Tsinghua University

3School of Computer Science and Engineering, UNSW Sydney

lei.fan1@unsw.edu.au

and genomics leads to a significant modality imbalance. The high spatial resolution inherent in pathology data renders it a dominant role while overshadowing genomics in multimodal integration. In this paper, we propose a multimodal survival prediction framework that incorporates hypergraph learning to effectively capture both contextual and hierarchical 0.0 0 50 100 150 200 250 0 50 100 150 200 feature dim feature dim 250 0.0 Feature Heatmap

details from pathology images. Moreover, it employs a modality rebalance mechanism and an interactive alignment fusion strategy to dynamically reweight the contributions of the two modalities, thereby mitigating the pathology-genomics imbalance. Quantitative and qualitative experiments are conducted on five TCGA datasets, demonstrating 4.0 3.0 2.0 1.0 classifier result 4.0 3.0 2.0 1.0 0.0

that our model outperforms advanced methods by over 3.4% in C-Index performance. Code: https: //github.com/MCPathology/MRePath. 0.0 0 50 100 150 200 feature dim

has gained increasing prominence, exemplified by the integration of whole slide images (WSIs) and genomic profiles in clinical oncology [Chen et al., 2021b; Xu and Chen, 2023]. The rationale behind this approach lies in the complementary strengths of each data modality: WSIs, regarded as the gold standard for cancer prognosis, provide cellular-level morphology and histopathological biomarkers [Fan et al., 2022b; Tang et al., 2025], while genomic data yield essential molec0.2 0.4 0.6 0.8

∗Corresponding author: Lei Fan

High ular signatures and mutation profiles for accurate diagnosis [Andre et al., 2022; Qu et al., 2024].

Low

Weights Weights Original Pathology Weight High Low Genomic Weight Heatmap Original Genomic Weight Rebalanced Genomic Weight Weights Rebalanced Pathology Weight Weights In multimodal survival analysis, WSIs with giga-level resolution are typically segmented into multiple smaller patches to facilitate computationally efficient analysis [Ilse et al., 2018]. Then, the multi-instance learning (MIL) technique [Dietterich et al., 1997] is widely employed to aggregate patch-level features, operating under the assumption that a bag is labeled positive if it contains at least one positive instance; otherwise, it is considered negative. Although MILbased methods [Hou et al., 2016; Campanella et al., 2019] excel in WSI classification tasks by effectively distinguishing tumor from non-tumor regions, they remain suboptimal for survival analysis. In many cases, they fail to capture the crucial contextual and hierarchical information required for precise prognostic assessment, leading to the inherent limitation of information loss in survival prediction. Specifically, contextual information [Kapse et al., 2024] includes spatial relationships and the micro-environment surrounding tumor cells, while hierarchical information [Hou et al., 2022] involves structural organization across various scales, e.g., cellular, tissue, and organ levels. Both types of information are essential for accurate prognostic evaluations [Zheng et al., 2018; Jin et al., 2024].

On the other hand, integrating information from different modalities is a key research direction [Lipkova et al., 2022]. Existing methods often employ late fusion by simply combining two modalities [Mobadersany et al., 2018; Cheerla and Gevaert, 2019; Chen et al., 2020], but they overlook the interconnections between genetic and pathologic data. Recent studies [Jaume et al., 2024; Zhang et al., 2024] explored early or mid fusion strategies, such as leveraging cross-attention mechanisms to better capture interactions across the modalities. However, these strategies still struggle to address issues related to modality imbalance. Specifically, a WSI can be represented through numerous patches, whereas only a few hundred genes have been identified for common cancers due to a higher signal-to-noise ratio and lower information density [Raser and O'shea, 2005]. Consequently, this discrepancy can cause pathological data to dominate the fusion process and overshadow genomic data, especially when cross-attention mechanisms are utilized [Chen et al., 2021b; Jaume et al., 2024]. As illustrated in Figure 1, we conduct heuristic survival prediction experiments, where pathology modalities dominate the prediction task, highlighting the pathology-genomics imbalance.

In this paper, to address the information loss challenge, we aim to leverage graph learning as an alternative to MIL for aggregating patch-level features. Unlike existing graph-based approaches [Shao et al., 2024; Chan et al., 2023] that reduce complex relationships to binary connections with limited expressiveness, we utilize hypergraphs [Feng et al., 2019] to model higher-order interactions among pathological features through hyperedges. This approach enables a more sophisticated representation of hierarchical and contextual details, enhancing the model's capability to capture intricate relationships. To tackle the challenge of pathology-genomics modality imbalance, we propose a dynamic weighting mechanism that adjusts the contribution of each modality based on its inherent reliability and synergy. By evaluating and aligning their interconnections, this approach ensures that each modality contributes optimally to the fusion process, effectively leveraging the strengths of both pathology and genomic data for more accurate survival predictions.

We propose a multimodal learning framework, MRePath (Multi-Modal Rebalance for Pathology-Genomic Survival Prediction), that leverages hypergraph learning and modality rebalance for survival analysis in WSIs. Specifically, in contrast to previous hypergraph studies [Di et al., 2022a; Li et al., 2023; Jing et al., 2025] that focus only on spatial relationships, we designate patches as nodes and construct hyperedges in both topological and feature spaces to capture spatial interactions and model structural hierarchies. We further introduce sheaf hypergraphs to promote information exchange between nodes and hyperedges, effectively preserving contextual details within individual patches and hierarchical relationships across the entire WSI. For rebalancing pathologygenomics modalities, we employ a dynamic weighting mechanism that first assesses the mono-confidence of each modality's reliability and then calculates holo-confidence by evaluating their interactions. These assessments are integrated to determine the final weight, which rebalances the contribution of each modality. Subsequently, an interactive alignment fusion is introduced, leveraging cross-attention to enable mutual guidance between modalities to produce the final hazard prediction. The contributions can be summarized as below:

• A multimodal framework, MRePath, is proposed to address both the MIL-based information loss and the pathology-genomics modality imbalance challenges in survival analysis for WSIs.
• A hypergraph learning framework incorporating sheaf hypergraphs is constructed over both topological and feature spaces to capture contextual and hierarchical details, while enhancing the model's ability to differentiate between various types of information.
• A modality rebalance method, consisting of a dynamic weighting mechanism and an interactive alignment fusion, is introduced to adjust the contributions of the two modalities for the final hazard prediction.
• Qualitative and quantitative experiments on five public datasets demonstrate the superiority of our model, achieving a 3.4% improvement over advanced methods.

2 RelatedWork

Survival analysis on WSIs. Early studies typically rely on MIL [Ilse et al., 2018] to aggregate patch-level features for representing WSIs [Hou et al., 2016; Campanella et al., 2019]. Various methods have been used to extract global features, such as embeddings [Yao et al., 2021], attention weights [Li et al., 2021], and graph-based modeling [Guan et al., 2022; Di et al., 2022a; Di et al., 2022b]. Recently, multimodal approaches integrating WSIs with genomic data for survival analysis have gained popularity [Lipkova et al., 2022]. Most research has focused on late fusion, such as vector concatenation [Mobadersany et al., 2018], modality-level alignment [Cheerla and Gevaert, 2019], and bilinear pooling [Chen et al., 2020]. Additionally, some studies have explored early or middle fusion approaches, leveraging crossattention mechanisms for cross-modal interactions [Chen et al., 2021b; Zhou and Chen, 2023; Jaume et al., 2024].

Despite their promising performance, these approaches often overlook the modality imbalance. We propose a plugand-play adjustment to evaluate each modality's reliability and connections, dynamically optimizing their contributions. This enhances multimodal fusion and improves performance.

Graph-based pathology analysis. Graph learning methods are widely adopted to capture intricate relationships in pathology-related tasks [Guan et al., 2022]. Early studies represented patches as graph nodes, constructing either adjacency-based [Chen et al., 2021a] or fully-connected graphs [Adnan et al., 2020]. Recent studies have employed cellular graphs to reveal spatial relationships among specific biomarkers (e.g., Ki67) [Nakhli et al., 2023] or cell types (e.g., tumor and stromal cells) [Shao et al., 2024] within WSI. Additionally, some studies [Di et al., 2022a; Di et al., 2022b] have utilized hypergraphs in WSIs for survival analysis, leveraging their representational power to capture complex interactions.

In our work, to tackle MIL-based information loss, we construct a hypergraph from both topological and feature spaces to capture contextual and hierarchical details. Additionally, we incorporate a sheaf hypergraph to adeptly differentiate and manage information from various types of hyperedges.

3 Method

3.1 Overview

Preliminary. Given X = {X1, . . . ,Xn} is the cohort of n subjects, each subject Xi can be represented as a tuple Xi = {Hi, yi}. Here, Hi = {Pi,Gi} denotes a pair of pathology-genomics features, where Pi represents the WSI, and Gi represents the genomic profiles. Meanwhile, yi = {ci, ti} represents the label of the i-th subject, comprising an event status ci ∈ {0, 1} (ci = 0 indicates that the event has occurred) and the subject's overall survival time ti. The goal of survival prediction is to estimate the hazard function ϕh(t), which predicts the instantaneous incidence rate of the interest event at a specific time point t. Instead of estimating a patient's survival time, we aim to train a model F to predict the probability that a patient's survival exceeds t using the survival function ϕs(t). This process is supervised by the negative log-likelihood (NLL) [Yao et al., 2020], defined as:

MRePath. It comprises three stages: feature extraction, hypergraph learning, and modality rebalance, as illustrated in Figure 2. Initially, features P and G are extracted from paired pathology and genomics data using different encoders. Hypergraph learning refines P to produce Ph with more contextual and hierarchical details. Modality rebalance includes a dynamic weighting module to obtain balanced features Pw and Gw, and an interactive alignment fusion to produce integrated features Pf and Gf using cross-attention operations, predicting the final risk outcomes.

Pathology feature extraction. Following previous studies [Chen et al., 2021b; Xu and Chen, 2023], we partition each WSI into multiple 256×256 pixel patches at 20× magnification. A pretrained encoder model (e.g., ResNet50) is used to extract d-dimensional features from these patches. Each WSI is then represented as P ∈ RN×d = {p1, p2, . . . , pN}, where N is the number of patches, and the coordinates of the i-th patch pi are c(i) p = (x(i) , y(i) ).

Genomic feature extraction. To process genomic data, including RNA-seq, Copy Number Variation (CNV), and mutation status, which typically exhibit a high signal-to-noise ratio, a selection process is applied to enhance data quality [Chen et al., 2021b]. The selected genes are categorized into six functional groups: Tumor Suppression, Oncogenesis, Protein Kinases, Cellular Differentiation, Transcription, and Cytokines and Growth. These data are then embedded using a multilayer perceptron (MLP) to generate G ∈ RM×d = {g1, g2, . . . , gM}, where M represents the number of gene categories.

3.2 Hypergraph Learning

We aim to leverage hypergraph learning to capture contextual and hierarchical details from patch-level features in WSIs. Our approach consists of two key components: hypergraph construction, which encodes spatial and structural relationships within WSIs, and the sheaf hypergraph, which enhances representations through higher-order structures. Together, these components form a robust framework for representing spatial and structural patterns in WSIs, enabling more effective analysis of their complex relationships.

Hypergraph construction. A hypergraph G = {V, E} is defined by a set of vertices V and a set of hyperedges E. For a WSI P, each patch is treated as a vertex, such that V = {v1, v2, . . . , vN}, where N is the total number of patches. The feature f (i) v of each vertex vi corresponds to the extracted features of its associated patch.

Hyperedges E are generated using two complementary methods: topological-based and feature-based approaches, capturing spatial and feature-level relationships, respectively. In the topological space, hyperedges are formed by grouping each patch with its neighboring patches based on the Euclidean distance in the spatial domain. Specifically, for a given patch vi, its neighbors are determined as:

where c(i) p and c(j) p denote the coordinates of patches vi and vj, respectively, and δ is a distance threshold. This process results in the set of topological-based hyperedges ET = {{vi, vj1 , vj2 , . . . } | ∀ vj ∈ NT (vi)}.

In the feature space, hyperedges are created based on the similarity between patch features. For a given patch vi, its feature-based neighbors are identified as:

where sim(·, ·) is a similarity function, such as cosine similarity, and γ is a similarity threshold. The values of δ and γ are determined by the hyperedge construction threshold k. Using these neighbors, the feature-based hyperedges are defined as EF = {{vi, vj1 , vj2 , . . . } | ∀ vj ∈ NF (vi)}.

The final hyperedge set E is constructed by merging the topological-based and feature-based hyperedges, such that E = ET ∪ EF where ∪ represents the take union.

Sheaf hypergraph. Given the constructed hypergraph G, the pathology feature P undergoes vanilla hypergraph convolution at the l-th layer, expressed as:

where ∆ represents the Laplacian operator, IN is the identity matrix, Θ(l) is a learnable weight matrix, and σ is a nonlinear activation function. The sheaf hypergraph [Duta et al., 2024] enhances this process by substituting the standard Laplacian operator ∆ in Eq. 4 with the sheaf Laplacian ∆F , allowing for data processing within a structured space over hyperedges, producing high-order features Ph = P(L) after L layers. The sheaf Laplacian is defined as follows:

where Dv is the degree matrix for vertices, and LF is the sheaf Laplacian matrix, given by:

Here, Fv⊥e represents the linear maps that facilitate the flow of information from a vertex v to a hyperedge e. It is achieved by averaging the features of the nodes within each hyperedge and then further aggregating these averaged features based on their association with each vertex.

The sheaf hypergraph creates an information flow space for nodes and hyperedges, enabling more expressive and structured dependencies for fine-grained information propagation. This approach effectively captures both local contextual features within individual patches and global hierarchical relationships across the entire WSI, resulting in a more robust representation. This framework is well-suited for survival prediction tasks requiring diverse information integration.

3.3 Modality Rebalance

We aim to achieve modality rebalance and interactive alignment fusion to recalibrate the contributions of each modality. This stage comprises two key modules: dynamic weighting, which calculates modality-specific weights to rebalance each modality, and interactive alignment fusion, which integrates the balanced features for final prediction.

Dynamic weighting. It dynamically regulates the contributions of pathologic and genomic data by computing the weights wp and wg for their respective representations P and G. This process leverages two confidence measures: mono-confidence, which evaluates the reliability of individual modalities, and holo-confidence, which captures their interactions. Together, these measures ensure effective modality fusion for accurate predictions.

Mono-confidence assesses the reliability of each modality by estimating the probability of accurately identifying the true class label within the respective dataset, thereby reflecting the modality's confidence [Corbi`ere et al., 2019]. The mono-confidence scores for pathology and genomic modalities are computed as:

where Φp and Φg are learnable parameters implemented with MLPs. A higher mono-confidence value indicates greater reliability of the corresponding modality [Cao et al., 2024].

Holo-confidence extends mono-confidence by incorporating cross-modal interactions, reflecting the overall coordination and complementarity between pathology and genomics. The holo-confidence for each modality is computed as:

These measures quantify how effectively each modality interacts with the other, providing a more holistic evaluation of their contributions.

To obtain the final weights, mono-confidence and holoconfidence are combined through a linear operation followed by a softmax to convert probabilities, defined as:

where ϕ denotes the softmax, wp and wg are the final weights for pathology and genomic features, respectively. These weights are used to adjust the pathology and genomic features, resulting in weighted representations Pw = wpPh and Gw = wgG. This dynamic weighting mechanism enables the model to adaptively balance the contributions of the two modalities, leveraging both their individual reliability and their interactions to achieve optimal fusion.

Interactive alignment fusion. The integration of pathology and genomic features is further enhanced by an interactive alignment strategy, which captures the internal relationships between the two modalities and facilitates their effective fusion. This process involves modality-specific co-attention mechanisms to refine feature representations.

To enhance the selection of pathological features based on genomic information, a gene-guided co-attention layer AG→P is employed. This mechanism results in the PC, as:

where ϕ(·) refers to the softmax function. wk p Gf , wqpPw, and wv pGf are keys, queries, and values, respectively. This co-attention mechanism aligns genomic features Gf with pathology features Pw, highlighting the most relevant pathological features. PC is then combined with Pw via a residual connection to produce the fused pathological features Pf .

For genomic features, a self-attention layer is first applied to capture intra-modal relationships, producing GS. Subsequently, a pathology-guided co-attention layer AP→G is employed to enhance genomic feature selection based on pathological information. The co-attention GC are computed as:

where wk g Pw, wq gGS, and wv g Pw are keys, queries, and values in the co-attention process. The co-attended genomic features GC are then combined with GS via residual connections to produce the refined genomic features Gf .

The interactive alignment strategy ensures that both modalities contribute complementary information while maintaining internal consistency. The fused pathology features Pf and refined genomic features Gf are enriched with modalityspecific and cross-modality interactions, offering a robust and comprehensive representation for survival prediction.

4 Experiments

4.1 Datasets and Settings

Datasets. We followed previous studies [Jaume et al., 2024; Zhang et al., 2024] and selected five datasets from The Cancer Genome Atlas (TCGA) to evaluate the performance of our model. The datasets include: Bladder Urothelial Carcinoma (BLCA) (n=384), Breast Invasive Carcinoma (BRCA) (n=968), Colon and Rectum Adenocarcinoma (COREAD) (n=298), Head and Neck Squamous Cell Carcinoma (HNSC) (n=392), and Stomach Adenocarcinoma (STAD) (n=317). The detailed distribution of survival times within these datasets is provided in the supplementary materials.

Evaluation metric. We used the Concordance Index (CIndex) to assess the predictive accuracy of our model. For each cancer type, we conducted 5-fold cross-validation, splitting the data into training and validation sets with a 4:1 ratio. Results are reported as the mean C-Index ± standard deviation (STD) across the five datasets.

Implementation details. To ensure a fair comparison, we adopted similar settings as previous studies [Chen et al., 2021b; Jaume et al., 2024; Zhang et al., 2024], using identical dataset splits and employing the Adam optimizer with a learning rate of 1 × 10-4 , a weight decay of 1 × 10-5 , and 30 training epochs.

4.2 Comparisons and Results

We classified the advanced methods into three distinct groups according to the modalities they used: pathology only (ABMIL, AMISL, TransMIL, and CLAM), genomic only (MLP, SNN, and SNNTrans), and multimodal approaches (Porpoise, MCAT, MOTCat, CMTA, SurvPath, and PIBD).

As shown in Table 1, unimodal methods such as MLP and SNN achieved average C-Index scores of 57.6% and 57.0%, respectively. These scores were significantly lower than those of multimodal methods such as MOTCat (68.1%), SurvPath (65.6%) and PIBD (68.5%), underscoring the limitations of unimodal approaches in capturing complex biological information. Among multimodal strategies, early fusion methods (e.g., MCAT, MOTCat, CMTA, SurPath, PIBD, and our approach) generally outperformed late fusion methods including SNN + CLAM (66.1%) and Porpoise (65.9%). This highlights the importance of effectively capturing interactions between modalities. Our method consistently outperformed both unimodal and other multimodal approaches across all cancer datasets, achieving the highest overall CIndex of 71.5% and an improvement of 3% over the secondhighest method PIBD. These results demonstrate the superior integration of pathology and genomics in our framework.

4.3 Ablation Studies

Hypergraph learning. We evaluated various graph architectures and hypergraph configurations, as summarized in Table 2. The baseline model with only MLP for feature aggregation achieved a C-Index of 65.9%. By employing GNN structures, GAT and GCN enhanced the C-Index to 67.1% and 67.6%, respectively, showing the effectiveness of graph structure in capturing relational interactions. Leveraging HGNN further elevated the mean C-Index to 70.5%, highlighting the utility of hyperedges in capturing high-order relationships compared to standard graph structures.

Incorporating a sheaf hypergraph with only topologicalbased hyperedges ET or feature-based hyperedges EF yielded scores of 68.7% and 68.8%, respectively, while combining both hyperedges achieved a superior C-Index of 71.5%. This indicates that integrating topological and feature-based connections significantly enhances the feature representations of WSIs, and that sheaf hypergraphs encourage local and global interactions to capture richer contextual and hierarchical details. These results highlight the efficacy of hypergraphs in capturing intricate relationships within the pathology data.

Hyperedge construction threshold. We evaluated various hyperedge construction threshold k, as illustrated in Figure 3.

The baseline model without hyperedges (k = 0) showed moderate performance across all five datasets. Increasing k enhanced performance, with the best results achieved at k = 9. However, further increasing k led to a decline in performance and more computational costs. Although there was a slight improvement in CO-READ at k = 49, the other datasets failed to regain their peak performance. The threshold k determines the range of node connections, balancing local and global information. Proper tuning of k enhances model performance by capturing meaningful relationships while avoiding information homogenization (k = 49) or oversimplification (k = 0).

Modality rebalance. We evaluated the modality weighting and Interactive Alignment Fusion (IFA), as shown in Table 3. We first manually assigned equal weights to pathology and genomics (wp=0.5,wg=0.5) with IFA produced a mean C-Index of 67.8%. By using different weights to favor pathology (wp=0.7,wg=0.3), the mean C-Index increased to 69.3%, which can be attributed to the effective extraction of pathology features through the sheaf hypergraph. However, it also exhibited higher instability, with the largest standard deviation observed in the BRCA, CO-READ, HNSC, and STAD datasets. Weighting the higher value to genomics (wp=0.3, wg=0.7) yielded a slightly higher score of 69.5%, verifying the effectiveness of modality rebalance in modality fusion. We observed that introducing a dynamic weighting mechanism with IFA showed the best performance of 71.5%, demonstrating the advantages of dynamically recalibrating each modality's contributions.

When using dynamic weighting, models with different co-attention layers produced varied outcomes. Specifically, pathology-guided and gene-guided co-attention (P.G+G.P) achieved a mean C-Index of 66.9%, while self-attention for genomics combined with either P.G or G.P yielded scores of 68.9% and 68.5%, respectively. Compared to these settings, using IFA resulted in an improvement of 2.6% in the mean CIndex, highlighting the importance of capturing both internal interactions within each modality and their mutual influence during the modality fusion process.

Pathology feature encoder. We evaluated various pathology feature encoders including five pretrained models: UNI [Chen et al., 2024], Conch [Lu et al., 2024], PhiKon2 [Filiot et al., 2023], CTransPath [Wang et al., 2022], and ResNet50 [He et al., 2016], as detailed in Table 4. The UNI and CTransPath achieved the highest values of 69.6% and 76.9% on STAD and BRCA respectively. PhiKon2 performed best on BLCA (72.5%) and HNSC (67.4%). However, these encoders underperformed on other cohorts with TCGA-5T-A9Q9

suboptimal average scores. Conch demonstrated robust overall performance with leading scores in CO-READ (83.2%), consistently performing well across all datasets and emerging as the most effective patch feature encoder in this study.

4.4 Visualization

Cross-modality interaction. The pathology-genomics interactions and the impact of genomics in high- and lowrisk BRCA cases are visualized in Figure 4. In the highrisk patient, increased expression of genes associated with cell proliferation and differentiation, such as HNF1A, CDX2, TGFBR1, ATP1B3, HNF4A, and CGA, suggests a tumorpromoting environment, while the elevated expression levels of DNA repair genes, including RECQL4, JAK1, GRK1, FCGR3A, BSX, and LECT2, indicate protective effects for the low-risk patient, enhancing the body's resistance to cancer.

Kaplan-Meier analysis. To validate our model's discriminative ability, we presented Kaplan-Meier curves in Figure 4. Patients are split into high-risk and low-risk groups based on median risk scores predicted by our model, with p-values calculated via the log-rank test. In all five datasets, p-values below 0.05 indicate that our model effectively distinguishes between high-risk and low-risk populations.

5 Conclusion

In this paper, we propose a multimodal framework MRePath for cancer survival prediction. It addresses MIL-based information loss by using sheaf hypergraphs in WSIs, and pathology-genomics imbalance by employing a dynamic rebalance. Experiments on five datasets demonstrate the superior performance and effectiveness of our framework.

Limitations. Considering WSIs with varying numbers of patches, the same k-value in the hypergraph leads to different scopes, imposing a potential inconsistency in pathology slides. In clinical applications, due to various factors, it may be challenging to obtain complete paired pathology and genomic data. In some cases, low-quality data or even missing modalities can significantly impact our modality rebalancing process, highlighting the need for greater robustness of our framework. We only considered a single WSI or a set of genomic information. However, there may be multiple WSIs and more extensive genomic data, which introduces the need for more sophisticated rebalancing and fusion strategies.