KG-MultiMS: a Knowledge Graph-Enhanced Multimodal Framework for Multiple Sclerosis
KG-MultiMS: a Knowledge Graph-Enhanced Multimodal Framework for Multiple Sclerosis
Riccardo Francia
Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence
Doctoral Consortium. Pages 8329-8330.
https://doi.org/10.24963/ijcai.2026/943
Despite advances in the computational analysis of multiple sclerosis, the ability to accurately predict disease progression remains limited due to unimodal approaches in the context of deep learning. KG-MultiMS proposes as a multimodal deep-learning framework that integrates data from three heterogeneous sources: (1) MRI volumes using the Swin-UNETR architecture to learn both semantic embeddings and segmentation masks of brain lesions; (2) tabular clinical variables using the FT-Tokenizer architecture to learn clinical embeddings; (3) medical knowledge graph based on the Harvard-Oxford atlas using a Heterogeneous Graph Transformer to learn contextual embeddings from patient-specific subgraphs. A cross-attention mechanism will allow for leveraging multimodal contributions by dynamically weighting them based on the anatomical-functional concordance represented in the contextual embeddings.
Keywords:
Machine Learning: Multi-modal learning
Machine Learning: Knowledge-aided learning
Data Mining: Knowledge graphs and knowledge base completion
Computer Vision: Biomedical image analysis
