OUCopula: Bi-Channel Multi-Label Copula-Enhanced Adapter-Based CNN for Myopia Screening Based on OU-UWF Images

1 Introduction

Global ocular pathologies, a major cause of visual impairment, impact at least 2.2 billion people, with myopia affecting around 130 million individuals globally [Holden et al., 2016]. Notably, high myopia is prevalent in Asian countries like China, affecting about 600 million people, including over 100 million students [Wang et al., 2022]. Since high myopia can lead to complications like cataracts and retinal issues, improvements to myopia screening techniques can have a substantial impact on public health. To aid in screening and detecting ocular diseases, different fundus imaging techniques have been developed. There are two primary types: conventional fundus images and ultra-widefield (UWF) fundus images. Conventional fundus images exhibit certain constraints including a restricted visual range of a mere 30◦ - 75◦ and the requirement of substantial photographic expertise and patient cooperation [Aggarwal et al., 2017]. In contrast, UWF imaging provides an expansive 200◦ field of view, and does not require highly specialized ophthalmologists, making it suitable for telemedicine applications in regions lacking medical resources [Ohsugi et al., 2017]. Although currently UWF imaging is widely used in developed countries [Nagiel et al., 2016], the limited availability of retinal specialists and adequately trained UWF interpreters poses significant challenges to the implementation of UWF imaging within China and other developing countries. Consequently, there is a pronounced lack of high-quality annotated data and intelligent screening models for both medical applications and scientific research in these regions.

Recently, deep learning (DL) has made important contributions to the field of medical imaging for detecting myopiarelated diseases and other retinal diseases [Devda and Eswari, 2019; Rauf et al., 2021]. Especially, different kinds of convolutional neural networks (CNN) have been developed, including ResNet, Vision Transformer, U-net, and Mask-RCNN, which have shown promising performance in various aspects of disease prediction, diagnosis, and segmentation [Yan et al.,

2019; Li et al., 2021; Mohan et al., 2022].

However, there is a paucity of research that considers the complex relationship between Oculus Uterque (OU, both eyes). The majority of existing studies treat the two images obtained from the left (OS) and right eye (OD) of the same patient as distinct entities, neglecting the inherent relationship between them [He et al., 2021]. One study has shown a significant correlation in the progression of ophthalmic diseases between bilateral eyes [Ferris et al., 2005]. This implies that integrating information from bilateral eyes for myopia prediction would be a more effective approach. Hence, in this work, we will model OU simultaneously, taking images of both eyes as inputs to the network, thereby transforming the model into a bi-channel CNN.

However, heterogeneity also exists between OU; the socalled "interocular asymmetries" refer to asymmetrical or unilateral features between two eyes [Lu et al., 2022]. In our study, the interocular asymmetry includes the scenario where the status of myopia in the two eyes of myopic patients is different, and it may even be that one eye is myopic while the other eye is not myopic. As Fig. 1 shows, the two eyes of the same patient can have very different myopia statuses: OS of this person is highly myopic, while OD is only mildly myopic. We have also done a Student's t-test, which shows that the difference in myopia status between OS and OD is statistically significant. Researchers suggest that consideration of interocular asymmetries could reduce statistical bias and offer additional information about retinal diseases [Sankaridurg et al., 2013; Henriquez et al., 2015]. This presents a challenge in the modeling process because both the strong correlation and the interocular asymmetry should be considered at the same time. Specifically, most parameters in the network should be shared to preserve the common features between two eyes, while the unique heterogeneous information contained in each eye also needs to be retained by distinct parts of the network. Inspired by the residual adapter introduced in [Rebuffi et al., 2018] from the field of multi-domain learning, we thought of using adapter modules to explain the small but significant asymmetry information between OU. In the multidomain context, the parameters of the larger backbone network are often shared across different domains, while adapter modules with fewer parameters are used to adapt different information patterns between domains [Lee et al., 2021; Li et al., 2022]. This aligns closely with the situation we are dealing with. The specific idea is that the strong correlation between OU implies that the input of OU should share most of the parameters for feature extraction and prediction. However, a small portion of distinct parameters is also needed to extract the asymmetric information. Therefore, the concept of the adapter model can be applied to this situation.

Based on the characteristics of the input images described above, we formulate our network as a bi-channel CNN, where our CNN has a shared backbone for the inputs of the two channels, and distinct adapter modules corresponding to each input channel. Currently, in the field of ophthalmic imaging, multi-channel network models commonly take different types or modalities of images of the same eye [Ma et al., 2021; Yi et al., 2023], which differs from our research setup. Moreover, the majority of them generate separate neural networks for each input channel, with fusion of the multiple channels occurring only at the end of fitting the whole network [Gour and Khanna, 2021; Xu and Fan, 2022]. To our best knowledge, only one article has considered the correlation between fundus image inputs and added a spatial correlation model in their proposed bi-channel network [He et al., 2021].

The assessment of myopia has predominantly relied on the Spherical Equivalent (SE), neglecting the potential of Axial Length (AL) as a significant outcome measure [Zadnik et al., 2015]. Existing literature suggests the predictive prowess of AL in anticipating myopia onset [Mutti et al., 2007]. In addition, as shown in the supplementary material, SE and AL of OS and OD are strongly correlated. These characteristics motivate us to use the interrelationship and take the correlation information between SE and AL into account to potentially further enhance the predictive capability of our models. To our best knowledge, only one work has utilized both SE and AL to predict myopia status based on fundus images and deep learning models [Zhong et al., 2023]. However, this work only focuses on left eyes. In this study, since we have SE and AL for both eyes, we formulate the prediction of four outcomes as a multi-label learning problem.

Regarding the multi-label learning problem from fundus images, most existing literature focuses on the classification of various retinal diseases [Sun et al., 2022; Al-Fahdawi et al., 2024]. If we extend the study of the correlation among labels to other fields, the most extensive one is multi-label classification [Wu et al., 2023; Lai et al., 2023]. Although there is abundant research on the correlation between labels in this area, what distinguishes our study is that these investigations primarily focus on the correlation between binary labels in the context of multiple classification tasks while our work is based on modeling the correlation between multiple continuous output variables.

Starting from these preliminary considerations, we propose to treat the prediction of SE and AL of both eyes as a multilabel learning problem (with continuous outcomes). We propose a framework of multi-label copula-enhanced CNN Learning with Oculus Uterque (OUCopula) for joint prediction of multiple clinical scores, using UWF fundus images on both eyes. Compared with previous studies, OUCopula can incorporate correlation information between SE and AL, the two outcome variables, into the neural network models. It can simultaneously address interocular asymmetries and the strong correlation between the two eyes, thereby enhancing the predictive capability of our model. Below we highlight our main contributions:

1. We establish a high-quality annotated dataset containing 5228 OU UWF fundus images. All the images have been desensitized, marked, and graded by experienced doctors.
2. We are the first to jointly model OU with pairwise UWF fundus image inputs that characterize the strong correlation and "interocular asymmetry" between OU. Our method improves the predictive power of different backbone models that only take single-eye inputs on myopia screening.
3. We construct a novel bi-channel multi-label architecture with adapter modules and copula-likelihood loss. Our novel strategy of building separate adapters for OS and OD and sharing the backbone models addresses the issue of interocular asymmetry and high correlation between OU by allowing the majority of parameters to be shared between both eyes while preserving their unique heterogenous characteristics. Also the copula-likelihood loss provides a new way to model the correlation between multiple continuous labels and perform joint regressions.

2 RelatedWork

The advent of UWF fundus imaging has revolutionized the field of ophthalmology, offering a broader perspective on retinal pathologies compared to traditional fundus images. Concurrently, the integration of DL techniques has emerged as a powerful tool for the analysis and interpretation of UWF fundus images. Several deep learning approaches have been proposed to predict and detect retinal diseases from either regular fundus images [Hemelings et al., 2021; Dong et al., 2022] or UWF fundus images [Li et al., 2021; Engelmann et al., 2022]. Despite the widespread application of DL to analysing fundus images, the majority of research has focused on the diagnosis, classification, and segmentation of diseases. Several myopia screening models have been developed but they mainly focus on binary classifications [Yang et al., 2020; Choi et al., 2021]. There has been relatively little direct prediction of continuous scores of myopia.

Recently, bi-channel and multi-channel learning have become common in the medical imaging field, due to the increasing availability of medical data from multiple sources and modalities [Liu et al., 2018]. In the field of ophthalmic imaging, the input channels for most multi-channel models commonly consist of different modalities of images of one eye [Ma et al., 2021; Yi et al., 2023]. Most researchers assign distinct CNNs for each input channel, and fuse the multiple channels at the end of fitting the network [Gour and Khanna, 2021; Xu and Fan, 2022]. Furthermore, most current CNNbased fundus imaging studies directly utilize the classic network models developed from other computer vision applications without special optimization of the architecture for medical scenarios. Regarding the case of OU, we find only one article that has considered the correlation between fundus image inputs and adds a spatial correlation model in their proposed bi-channel network [He et al., 2021]. Furthermore, there is a paucity of research considering bilateral asymmetry. To our knowledge, there are only very few studies that use CNN to determine the presence of bilateral asymmetry in fundus images [Kang et al., 2022]. Otherwise, there is no literature that considers bilateral asymmetry in retinal disease prediction.

Multi-label learning research from fundus images mostly focuses on classifying various retinal diseases [Al-Fahdawi et al., 2024; Sun et al., 2022], and based on our investigation, no DL research in the field of ophthalmology has considered the correlation between labels. More generally, for multi-label learning in the computer vision field, most existing literature also considers the multiple classification problem [Wu et al., 2023; Lai et al., 2023; Song et al., 2018; Chen et al., 2021], and some of them consider the spatial correlation across labels determined by small patches/instances of the image. In contrast, OUCopula employs a copula model to incorporate the conditional Pearson correlation across the labels, given the image covariates. Based on this copula model, we equip the backbone CNN with a new copulalikelihood loss to enhance the prediction capability.

3 Methods

3.1 Residual Adapter Modules

In this study, we use the residual adapter to adapt to the interocular asymmetry between the two eyes. The residual adapter was introduced in [Rebuffi et al., 2018], and it was initially applied in the field of multi-domain learning. In recent years, it has also been widely used in the Natural Language Processing domain for fine-tuning models and transfer learning. The conventional use of adapters in multidomain learning involves training different adapters based on the same large pre-trained backbone while freezing all the parameters in the backbone model.

Instead, in this work, we adopted a slightly different architecture to accommodate the presence of high correlation and interocular asymmetries. In our framework, we want to share most parameters in the network to preserve the common features between OU, while allowing small but distinct parts in the network to contain the heterogeneous information for OS and OD individually.

Hence, we constructed adapter modules for both OS and OD respectively and let them share the backbone model. During training, adapters are updated simultaneously with the backbone model. Specifically, the OS input passes through the backbone model and the OS's adapter, while the OD input goes through the backbone model and the OD's adapter. Each training iteration involves inputting images from both eyes of the same individual. Subsequently, the outputs from both eyes are used together in the computation of the copulalikelihood loss, followed by backward propagation to complete a single training iteration. In order to show the structure of the backbone model and adapter modules, and to explain the training process, Fig. 2b illustrates the detailed architecture when using ResNet as the backbone.

3.2 CopulaModeling

Multi-label Regression

Suppose we have n subjects with J different tasks and have at most K labels for each task. Let j = 1, . . . , J and k = 1, . . . ,K be the task and label indices respectively. For the ith subject, the covariates are given as {Xij}Jj=1 , where Xij denotes the image corresponding to the jth task. The corresponding labels Y are, therefore expressed as {yijk}, j = 1, . . . , J, k = 1, . . . ,K. Under the OU UWF data setting, J = 2 represents the OS and OD task, K = 2 represents the AL and SE labels, and Xij denotes the UWF fundus image of either the OS (j = 1) or the OD (j = 2). We consider the following multi-label mean regression model

where gjk denotes the unknown regression function associating the covariate Xij with yijk, and ϵijk represents the corresponding noise. Without loss of generality, we assume that Eϵijk = 0 and ϵi ⊥ ϵt for i ̸= t, where ϵi = (ϵi11, . . . , ϵiJK)T .

Gaussian Copula

Let ϵ = (ϵ11, . . . , ϵJK)T be the noise vector and its joint density be fϵ. As shown in supplementary, the labels are indeed dependent. Hence, it is unsuitable to assume that fϵ has a product form. To incorporate the dependence structure across the labels, we consider the copula model [Sklar, 1959]. Let C be a p-dimensional distribution function (CDF) defined on the rectangle [0, 1]p . Any p-dimensional joint continuous distribution function F can be completely specified by its marginal distributions and a copula C. That is, there exists a copula C such that

where Fj denotes the marginal CDF of yj for j = 1, . . . , p. Since all labels studied in this paper are continuous, we consider the Gaussian copula. Let Φ be the CDF of the standard normal distribution and Γ ≡ (γtj )p×p be a correlation matrix, where the elements γtj are defined as

representing the linear correlation between the two Gaussian scores (Φ-1 {Ft(yt)},Φ-1 {Fj(yj)}). Equipped with the correlation matrix Γ, a Gaussian copula is given as

where Φp(·|Γ) denotes the p-dimensional standard Gaussian CDF with correlation matrix Γ.

Copula Loss

By taking the derivative of (3), we obtain the copula density as

where q = {(Φ-1 (F11(t11)), . . . ,Φ-1 (FJK(tJK))}T . The copula loss OUCopula model is defined as minus the loglikelihood based on (4).

We often consider the Gaussian case where ϵjk ∼ N(0, σ2 jk ). In this case, the copula density (4) reduces to a multivariate Gaussian density, so the copula loss is given by

where eijk = yijk - gjk(Xi) represent the residuals, and Σ ≡ (Σ)(jk)=JK (jk)=11 = σj1k1 σj2k2 γj1k1,j2k2 represents the covariance matrix.

3.3 Choice of Backbone Models

The copula loss is flexible and generalizable in the sense that it does not impose specific requirements on the choice of the backbone model. We opted to use ResNet and DenseNet as the backbone models. Applying copula-likelihood loss should avoid overfitting during training; otherwise, it would fail to extract effective dependence information from overfitted residuals. Due to the relatively small size of our UWF dataset (a few thousand images), we followed the approach applied in [Zhong et al., 2023]: using a simplified ResNet-18 and DenseNet-121 (last two CNN blocks removed) as backbone models.

3.4 End-to-end OUCopula

In this section, we describe our OUCopula framework, which consists of three modules: (1) a warm-up module with adapter modules that train the backbone CNN under empirical losses; (2) a copula estimation module that estimates the parameters in the Gaussian copula based on the results of the warm-up module; (3) the OUCopula module that trains the backbone CNN using the derived copula-likelihood loss that can efficiently capture the dependence structure across multiple labels through a Gaussian copula model. We also summarize the OUCopula in Algorithm 1 and show an overview of the OUCopula architecture in Fig. 2a.

In detail, Module 1 includes the basic construction of backbone CNNs under empirical losses. Residual adapters are applied in the backbone CNN as described in 3.1. Then all the numerous parameters included in the CNN layers are updated by the Adam algorithm [Kingma and Ba, 2014] to optimize the losses. In Module 2, we estimate the two copula parameters including the marginal sample standard deviation (SDs) σjk and the Pearson correlation γj1k1,j2k2 by the sample SD and sample Pearson correlation respectively. Finally, in Module 3, we determine the copula-likelihood loss based on the estimated copula parameters and use the backbone CNNs fromModule 1 to train the OUCopula under the copula-likelihood loss. Note that in this Module, OUCopula does not rely on any specific architecture in the backbone CNN.

4 Experiments

In this section we evaluate our method quantitatively against the simplified ResNet backbone model, and investigate whether the proposed framework can improve the backbone model by extracting correlation information for the outcomes and addressing the issue of interocular asymmetry. We will test and compare the results between the sole backbone model, backbone + Adapters, and backbone + Adapters + copula model (OUCopula). Due to the limited space, we only present the result on the ResNet as the backbone. For results on the DenseNet, see supplementary materials.

4.1 Dataset

The data collection process involved capturing 5228 UWF fundus images from the eyes of 2614 patients using the Optomap Daytona scanning laser ophthalmoscope (Daytona, Optos, UK). All enrolled patients sought refractive surgery

Algorithm 1 End-to-end OUCopula

Input: Training data {Xi, Yi = (yi11, . . . , yiJK)}ni=1 .

Output: Trained OUCopula (ˆ g11, . . . , ˆ gJK).

Module 1: Warm-up CNN

• 1: for j = 1, . . . , J, k = 1, . . . ,K do
• 2: gjk = Backbone CNN with adapter modules.
• 3: ˆ g0 jk = arg min n-1 ∑ni=1 (yijk - gjk(Xi))2 .
• 4: end for

Module 2: Copula parameters estimation

• 5: Obtain residuals eijk = yijk - ˆ g0 jk (Xi).
• 6: ˆσjk = sd(eijk), ˆ γj1k1,j2k2 = corr(eij1k1 , eij2k2 ).

Module 3: OUCopula

• 7: gjk = Trained backbone CNN in Module 1.
• 8: Define the loss L2 the same as (5).
• 9: (ˆ g11, . . . , ˆ gJK) = arg min L2.

treatment and were exclusively myopia patients. The data collection period extended from December 2014 to June 2020, and was conducted at The Eye and ENT Hospital of Fudan University. For this study, only central gazed images were included and blurred or eccentric images were excluded. The UWF fundus images obtained during the study were exported in JPEG format and compressed to a resolution of 224 x 224 pixels to facilitate subsequent analysis.

4.2 Experiment Setup

The dataset of 5228 fundus images was partitioned into the training data set, the validation set, and the testing set, with a ratio of 6:2:2. To mitigate bias in model evaluation and obtain more reliable estimates of the results, 5-fold cross-validation (CV) was employed. Since the outcome variables are all continuous, we calculated the Mean Square Error (MSE) for the results. In addition to the total MSE loss of OU, we also performed various sub-analyses to study the detailed pattern of predictive performance. Specifically, the following MSE losses were compared:

1. MSE loss for SE and AL of each eye (OS SE, OS AL, OD SE, and OD AL);
2. Total loss for each eye (OS SE+AL, OD SE+AL);
3. Total loss for SE and AL separately (OU SE, OU AL);
4. Total loss for OU (OU SE+AL).

We compared the predictive performance between the backbone model, the backbone model with adapter modules, and the backbone model with both the adapter modules and copula-likelihood loss (OUCopula), aiming to show that the proposed framework could enhance the performance of the backbone models. The simplified ResNet model mentioned in subsection 3.3 served as the backbone model for our experiment. For the single backbone model of ResNet, since there is no adapter module, the bi-channel input was simplified to a single-channel input. This is equivalent to treating the left and right eyes independently as different individual input samples, which is the common approach of using single eye inputs in the majority of the literature dealing with fundus images. Therefore, comparing the backbone + adapters with the single backbone model can be considered as a comparison between the joint modeling of OU and the traditional modeling of single eyes.

4.3 Results

Fig. 3 and Fig. 4 compare OUCopula and baseline models. In these figures, the charts with a green background represent the results for a single label in one eye, those with a light blue background represent the total results for a single label, those with a light purple background represent the total results for one eye, and those with a red background represent the overall results for both eyes. In these two 3 × 3 charts, each third chart in every row or column is the sum of the preceding two charts.

According to Fig. 3 and Fig. 4, on average OUCopula surpasses both pure ResNet and ResNet + Adapters on almost all sub-results. Whether it is the MSE loss for the SE or AL of a single eye, or the total loss for both the left and right eyes, OUCopula achieves the best performance. We also have the following observations:

1. If we compare ResNet alone with ResNet + Adapters, we see an average improvement in total loss of 3.69%. Comparing ResNet alone with OUCopula, the total loss shows an average improvement of 7.18%. It can be roughly observed that the contributions of the residual adapters and copula loss to the prediction improvement are of a similar level, with adapter modules contributing slightly more.
2. Among all five folds, ResNet + Adapters shows improvements in 3 out of 5 folds, but not in every fold; on the other hand, OUCopula performed better than ResNet in 4 out of 5 folds. Additionally, OUCopula consistently outperforms ResNet + Adapters in most sub-results.
3. When comparing the results for SE or AL separately or evaluating the overall results for OU, OUCopula shows a relatively balanced improvement of around 6%-8% on average over ResNet. Specifically, utilizing OUCopula improves the MSE by 8.42% for OS SE, 6.69% for OS AL, 6.41% for OD SE, 5.54% for OD AL, 7.43% for OU SE, 6.12% for OU AL, and 7.18% for OU SE+AL. The improvements on SE are slightly better than those on AL. This phenomenon differs from previous literature where copula-likelihood loss in [Zhong et al., 2023] tended to optimize outcomes with smaller variances, and the RMSE of SE was improved by 2.232% and the RMSE of AL was improved by 3.683% on average using the same ResNet backbone.

The results on the DenseNet backbone in the supplementary materials show similar improvements compared to that on the ResNet backbone.

5 Discussion

Overall, OUCopula consistently outperforms both the backbone model and backbone + Adapter across various subresults, indicating that OUCopula exhibits remarkable efficacy in enhancing the predictive accuracy of backbone models. Observations presented in subsection 4.3 also provide valuable insights into the effectiveness of OUCopula.

• The improvement of backbone + Adapters over single backbone models shows that our joint modeling under the bi-channel architecture with residual adapter modules does capture the interocular asymmetries (the heterogeneity of OU inputs), which was often ignored in the literature. Meanwhile, OUCopula consistently surpasses backbone + Adapters across all sub-results among most folds, indicating that our copula modeling correctly characterizes the conditional correlation between the labels (given the image covariates).
• Compared to the backbone + Adapters, OUCopula shows improvement across all sub-results among most folds, indicating the robustness and the stable performance of OUCopula in cross validation.
• An interesting finding is that OUCopula can improve the prediction of multiple labels in a balanced way regardless of their marginal SDs; say, the improvements on SE tend to be slightly more pronounced than those on AL, though the marginal SD of SE is significantly larger than that of AL. This phenomenon is different from the common results in the modeling of task-dependent uncertainty (e.q. (7) of [Kendall et al., 2018] for instance). We conjecture that the more complex framework of jointly modeling OU and multiple labels allows our model to extract a broader range of information from input images and mitigate the square loss caused by larger marginal SD, leading to more balanced multi-label predictions.

Last but not least, we want to discuss the adaptability of OUCopula. Subsection 3.2 indicates that the copulalikelihood loss can be applied to any dimensions of outcome labels. This scalability is particularly relevant in medical applications where comprehensive assessments involving multiple clinical parameters are common. Researchers can extend the model to handle more than four outcomes for more comprehensive prediction and complex clinical practices. Furthermore, it is essential to highlight that our bi-channel architecture is also designed for flexibility and can readily be extended to a multi-channel model. This adaptability is crucial, especially in scenarios where additional imaging modalities or channels may provide complementary information. Future work could explore the integration of diverse data sources to further enhance the model's predictive capacity. Additionally, since the dataset is relatively small (only a few thousand input images), we adopted the simple solution for overfitting (simplifying the backbone network) by [Zhong et al., 2023]. The characteristics of the copula-likelihood loss mean that we can apply various kinds of backbone models in the OUCopula framework. Hence, exploring how to apply OUCopula to much larger models while avoiding overfitting is a valuable direction for future work.

6 Conclusion

Our study has demonstrated the effectiveness of the OUCopula model in leveraging UWF fundus images for myopia screening. The bi-channel input structure and the integration of adapter modules allowed for simultaneous modeling of OU, addressing both the complex interocular asymmetry and correlation between them. The ability of the copula-likelihood loss to incorporate the dependence structure between multiple continuous outcome labels is a notable strength. Furthermore, the adaptability of OUCopula means that it can be generalized to different kinds of backbone models, and extended to multi-channel, multi-label problems, and the generalizability on different kinds of backbone models.

Acknowledgements

YL and BF's research are supported by National Natural Science Foundation of China (71991471, 12071089). QH is supported by General Research Fund 15301123, RGC, HKSAR. CZ is supported by the Postdoc Fellowship of CAS AMSS-PolyU Joint Laboratory of Applied Mathematics and PolyU Research Grant P0045497. Research of DY and ML are supported by the Shanghai Rising-Star Program (21QA1401500). ML's research is also supported by National Natural Science Foundation of China (82371091). Research of AW is supported by the Australian Research Council Discovery Project (DP230101908). Research of AL is supported by the Intramural Research Program of the National Institute of Child Health and Human Development. CL's research is supported by General Research Fund 15327216 and 15301123, RGC, HKSAR and National Natural Science Foundation of China (12271060).

Contribution Statement

Y.L., Q.H., and C.Z are lead authors and equally contributed in computation implementation, methodology, investigation, analysis, and original draft. D.Y., M.L, and X.Z. were responsible for data collection and curation, and result interpretation. A.W. and A.L. reviewed and edited the manuscript. B.F.

• and C.L. supervised throughout the whole project.