Hybrid Learning System for Large-scale Medical Image Analysis

1 Introduction

Reliable annotation is a critical necessity for building a deep learning pipeline. However, due to human resources constraints, high-quality annotation cannot always be accommodated. Such constraints have become obstacles to applying deep learning for large-scale medical image analysis. To solve such a challenge, we need to lower the annotation quality and quantity requirements for the existing deep learning models in medical image analysis.

We will progressively reduce annotation requirements to achieve large-scale medical imaging analysis under clinical conditions. In this case, we would develop a hybrid learning system to utilize the large-scale unlabelled medical images and labelled medical images with unknown annotation quality. The annotation could be fine-grained annotation like key point and segmentation mask or coarse annotation like classification label or inexact bounding box or segmentation mask.

2 Current Works

2.1 Robust Medical Detection with Insufficient Annotations

Manual annotations are not always exact due to time, cost, and experience constraints in clinical practice. Such cases would produce inexact bounding box annotations where the bounding box is either too big for the target or partially covered. We evaluated different detection models over the different medical detection benchmarks with different levels of detection annotation quality. We present part of the experimental results in Table 1. We found performance and robustness degradation in all benchmarks when the bounding box annotation is not strictly covered the target. We developed a regularization policy to filter the positive samples using the centroid feature maps of the different levels of Feature pyramid networks (FPNs) [Lin et al., 2017]. We further constructed an adversarial loss to utilize the side-product of the regularization to improve the model robustness.

We believe this is a good start to lower the annotation requirements in medical image analysis. Since FPNs are also applied in segmentation tasks, we can also apply such a method over the segmentation tasks and integrate to a wide range of applications.

2.2 Hybrid Learning System for Large-scale Medical Image Analysis

Currently, there are several new training paradigms for deep learning and unsupervised and supervised learning. Selfsupervised learning aim to obtain a powerful pre-trained model by training pretext tasks and applying transfer learning to downstream tasks. Typically, pretext tasks do not require extensive human efforts, and the majority of tasks aim to design a reconstruction pipeline to extract the associated features. [Jing and Tian, 2020]. Semi-supervised learning refers to learning with limited annotated in conjunction with many unlabelled data [Van Engelen and Hoos, 2020]. Weak-supervised learning tried to learn a model with coarsegrained annotations [Zhou, 2018]. Such learning paradigms reduce the annotation requirement from a model training perspective. Each learning paradigm, however, has its own limitations. Self-supervised learning may require additional time to train a pre-trained model, making it inefficient for large-scale applications. A detailed guidelines [Oliver et al., 2018] has presented theoretical and empirical proof to support the claim that unlabelled data can always improve the performance of the deep networks in image classification when the annotation is correctly assigned. Large-scale weaksupervised learning may harmed localization performance when transferring to downstream tasks [Mahajan, 2018].

In this case, we would like to develop a system that merges the existing training pipeline to solve the large-scale medical image analysis with different types of annotations under unknown annotation quality.

Primary attempt apply self-supervised learning on largescale medical images shows large-scale medical images can improve the model performance and robustness [Ghesu, 2022]. We would extend such framework by merging different types of learning system. During our primary exploration on the medical images, we have found semi-supervised learning could serve as a bridge to self-supervised learning and weak-supervised learning. Our preliminary experiments found that applying semi-supervised learning to the downstream task of self-supervised learning improves overall performance and mitigates the sacrifice of localization performance in large-scale weakly supervised learning.

2.3 Improved Hybrid Learning Systemwith Graph Constraints

We would introduce graph constraints to learning pipeline to improve the hybrid learning system that discussed in the Section 2.2. Graph constrain has widely applied to learn the imbalanced data [Du et al., 2021]. A self-supervised learning pipeline could also benefit from the graph constraints [Li et al., 2021] to improve the robustness in learning fine-grained visual features. Thus, by introducing graphical constraints, information can be utilized in a more organized way, rather than being driven entirely by gradients.

Meanwhile, we found that knowledge graphs are heavily used in the field of medical diagnosis [Li et al., 2019]. Introducing knowledge graph could also improve the annotation utilization and the model performance. [Xie et al., 2022] introduced the graph neural networks to process the relationship of the predicted mask for instance segmentation task. Therefore, applying knowledge graph to the hybrid learning system is a prominent option. Knowledge graphs not only model the domain knowledge explicitly, but also incorporate prior exact expert rules or constraints. In particular, we would investigate using symbolic rule learner [Wu et al., 2022] to conduct inductive relational learning, in order to enhance the precision effectively and efficiently.

3 FutureWorks

Based on our primary exploration, while solutions in healthcare have some domain-specific requirements, the solution for challenges mentioned in the above section are broadspectrum and not limited to the medical domain. We would further explore the potential of our hybrid learning system over the large-scale data and cooperate with medical institutions to further verify our design.

Acknowledgements

Zehua Cheng is under supervision of Prof. Thomas Lukasiewicz. Lianlong Wu is under supervision of Prof. Emanuel Sallinger and Prof. Georg Gottlob.