Human-Centred Multimodal Deep Learning Models for Chest X-Ray Diagnosis

1 Research Problem

Deep Learning (DL) technologies are widely used in medical imaging to automatically diagnose and detect subtle differences that may be difficult for human radiologists to detect. However, most DL applications in medical imaging only use image data due to limited literature on medical datasets combining different data modalities. While DL models can outperform humans in diagnosing chest X-rays (CXRs), many times these models get the predictions correct, but not for the right clinical reasons [Saporta et al., 2022]. Current state-of-the-art DL models often neglect the multimodal nature of radiologists' perception when examining medical images. This is a crucial limitation since the combination of visual cues, such as shapes, textures, and objects, allows radiologists to form mental models that generalize better with less data. Hence, there is a need to integrate the human component in DL models to improve the accuracy and robustness of medical image diagnosis.

To better understand how humans operate in the radiology reading room, I interviewed a team of expert radiologists about their approach to assessing X-ray images. They emphasized the importance of having access to clinical information about the patient to make an accurate diagnosis. Without this information, they noted that their diagnoses can be more prone to biases and errors. These findings highlight the importance of integrating clinical data in medical image analysis. However, there is a lack of consideration in existing DL architectures in using clinical data for prediction in CXRs. Some recent studies also confirm using accurate patients' clinical data together with medical images can improve the performance of human radiologists in terms of accuracy [Castillo et al., 2021].

Recently, two novel datasets (REFLACX [Lanfredi et al., 2022] and EyeGaze [Karargyris et al., 2021]) have been introduced that combine CXRs with radiologists' eye-gaze movements. While previous studies suggest that integrating radiologists' eye-gaze data can aid in detecting potential lesions in CXRs [Ganesan et al., 2018], there remains a lack of research exploring the incorporation of this modality in DL models for medical image analysis. Hence, further research is needed to investigate and leverage the potential of radiologists' eye-gaze data in DL to enhance diagnostic accuracy and mitigate errors.

The aim of this thesis is to address the aforementioned research gaps by investigating various types of multimodal DL architectures and assessing their effectiveness in conjunction with expert radiologists. Specifically, this research endeavors to explore the integration of different data modalities such as patients' clinical data and eye tracking data, to develop robust DL models for medical image analysis.

2 Current Research Contributions

So far, I have interviewed expert radiologists to understand how they operate in their radiology reading room, and how they assess CXRs. I also contributed to a public dataset that integrates several different datasets of the literature into a single multimodal dataset that contains clinical data, CXRs, radiologists' audio transcripts, and their eye movements. Using this dataset and information provided by radiologists, I proposed a multimodal DL model that combines clinical data with CXRs.

2.1 The MIMIC-EYEMultimodal Medical Dataset

In this thesis, I created MIMIC-EYE [Hsieh et al., 2023], a comprehensive dataset that integrates several MIMIC datasets, including medical images, reports, clinical data, and eye tracking data with gaze and pupil dilation information. The integration of eye tracking data with MIMIC modalities may enhance the understanding of radiologists' visual search patterns and improve DL model development. MIMIC-Eye contains 3,689 tuples of chest X-ray images, eye-gaze data, and radiologists' audio transcripts, of which 1,683 tuples include clinical data.

2.2 MDF-Net: Multimodal Dual-Fusion Network

After creating the MIMIC-Eye dataset, we developed a new architecture called multimodal Dual-Fusion Network (MDFNet), which uses both CXR images and clinical data for lesion detection. The challenge in using these modalities is that they have different dimensions, with images typically represented as 3-dimensional tensors and clinical data as 1-dimensional tensors. To address this, we introduced a strategy called spatialisation, which elevates 1-D clinical data into 3-D space (Figure 1). Our spatialisation module contains multiple deconvolutional layers that upsample the pixel to the desired size for fusion, allowing us to fuse images and clinical data in 3-D space. Through MDF-Net, we demonstrated that incorporating clinical data improved the generalization of the model and lesion detection performance by +12% on Average Precision (AP). Our novel architecture and spatialisation strategy provide a promising approach for using multimodal data in deep learning for medical image analysis (Table 1).

3 Future Research Directions

My current research investigates the integration of eye tracking data into DL architectures for lesion detection in CXRs. Initial results showed that directly providing fixation masks of radiologists' gaze patterns as input did not improve the baseline Mask RCNN. Future work will address two research challenges: (1) DL approaches that use raw eye gaze data without considering the intricacies of its human generation process, leading to noisy data that does not effectively contribute to the supervised learning process, and (2) the human-centric nature of eye tracking data, requiring the consideration of human factors in its generation, including the investigation of how radiologists examine CXRs and their visual search patterns.

Acknowledgements

My PhD research is sponsored by a full scholarship offered by the Centre for Data Science (CDS) at Queensland University of Technology. The material is based on the work supported by the UNESCO Chair on AI&XR; and the Portuguese Fundac¸ ˜ao para a Ciˆencia e a Tecnologia (FCT) under grants no. 2022.09212.PTDC (XAVIER) and no. UIDB/50021/2020.