My thesis consists of investigating how chest X-ray images, radiologists' eye movements and patients' clinical data can be used to teach a machine how radiologists read and classify images with the goal of creating human-centric AI architectures that can (1) capture radiologists' search behavioural patterns using their eye-movements in order to improve classification in DL systems, and (2) automatically detect lesions in medical images using clinical data and eye tracking data. Heterogeneous data sources such as chest X-rays, radiologists' eye movements, and patients' clinical data can contribute to novel multimodal DL architectures that, instead of learning directly from images' pixels, will learn human classification patterns encoded in both the eye movements of the images' regions and patients' medical history. In addition to a quantitative evaluation, I plan to conduct questionnaires with expert radiologists to understand the effectiveness of the proposed multimodal DL architecture.