Revealing Hierarchical Structure of Leaf Venations in Plant Science via Label-Efficient Segmentation: Dataset and Method

1 Introduction

Analyzing the detailed structure at different hierarchical levels of leaf venation is a fundamental step for botanists, crop breeders, and ecologists to understand the impact of these intricate structures on important physiological functions of leaves, such as photosynthesis, transpiration, respiration, and transportation. This understanding can facilitate the breeding of plant species with high yield, quality, or economic value [Sack et al., 2013], thus not only targeting the United Nations' Sustainable Development Goals of No Poverty and Zero Hunger (SDG1 & SDG2) [United Nations, 2023a] but also fulfilling the Leave No One Behind Principle (LNOB) [United Nations, 2023b]. To achieve this task, a non-negligible process is to segment the leaf venation. With the development of imaging technology and computer vision techniques, researchers have started to utilize image analysis methods to analyze leaf venation structures. But most studies focus on segmenting the entire leaf venation network without differentiating the order of veins [Price et al., 2011; Dhondt et al., 2012; B¨uhler et al., 2015; Salima et al., 2015; Grinblat et al., 2016; Lasser and Katifori, 2017; Xu et al., 2021; Li et al., 2022; Liu et al., 2022; Iwamasa and Noshita, 2023]. The task of interest, hierarchical vein segmentation from leaf images, involves not only isolating and identifying the venation network from other leaf components but also classifying each order of veins into distinct semantic categories. It is a challenging computer vision task due to the complexity of vein structures and the high similarity in color and texture among veins of different orders [Liu et al., 2022]. The first three orders are the primary (1°) vein or veins emerging from the leaf base to the apex, the smaller secondary (2°) veins branching at intervals from the 1° veins towards the leaf margin, and even finer tertiary (3°) veins [Ellis et al., 2009], as shown in Fig. 1. Until now, only two studies have attempted hierarchical vein segmentation [Gan et al., 2019; Jin and Han, 2020], which used directional morphological filtering and region-growing-based methods, respectively. But these traditional digital image processing techniques cannot extract rich semantic information from the highly dense leaf venation networks and have not yet addressed the semantic segmentation of 3° veins.

∗Corresponding authors.

Deep learning seems to be a promising technique for hierarchical leaf vein segmentation. However, as a data-driven method, the biggest obstacle is the lack of labeled training data. The existing public datasets have limitations in leaf image acquisition methods and vein labeling. For data collection, chemical cleaning and vein staining are commonly used practices for treating leaves before imaging to increase the color contrast between leaf veins and lamina [PerezHarguindeguy et al., 2016; Xu et al., 2021; Iwamasa and Noshita, 2023]. But it is quite time-consuming and can easily damage the leaf blade. X-ray imaging makes it possible without chemical treatment but requires expensive specialized facilities [Schneider et al., 2018; Zhu et al., 2020]. More importantly, the existing public datasets annotate all the veins in a leaf as one semantic class without recognizing vein orders, which cannot serve as the ground truth for deep learningbased hierarchical leaf vein segmentation.

To address the aforementioned limitations, in this work, we construct and release a new dataset for HierArchical Leaf Vein Segmentation (HALVS). To acquire high-resolution and high-contrast leaf venation images from the raw leaf samples, we adopt a flat-bed scanner in transmission scanning mode, which can provide backlight illumination throughout the scanning process. To the best of our knowledge, HALVS is the first public dataset that leverages this efficient and costeffective imaging system. The HALVS comprises 5,057 leaf images from three representative species: soybean, sweet cherry, and London planetree (see Fig. 1). Each species was selected for its unique significance in diverse contexts. Soybean is widely acknowledged as a fundamental global provider of protein and oil [Vollmann, 2016] with both ecological and economic values worldwide. Sweet cherry represents a high-class fruit with significant economic value, and London planetree was selected for its relevance in urban landscaping, complementing the dataset by providing diversity in leaf venation patterns. Furthermore, we have also provided detailed pixel-level annotations for the first three orders of veins of these plants, considering that their vein traits are of significance for biological and ecological research. It is worth noting that hierarchical annotated leaf veins are extremely time-consuming, as shown in Tab. 1. The dataset contribution will be described in detail in Sec. HALVS Dataset.

This paper also makes a methodological advancement in label-efficient learning. Given the label scarcity challenge, fully supervised deep learning methods are not optimal for hierarchical leaf vein segmentation tasks. Instead, semisupervised learning (SSL) [Chapelle et al., 2009; Papandreou et al., 2015], which learns a model with only a few labeled data and numerous unlabeled ones, looks like a feasible solution. We notice that annotations of 3° veins take much longer than those of the 1° and 2° veins (see Tab. 1). It is more practical to prepare partially labeled data with only annotations of 1° and 2° veins when the fully labeled data with annotations of veins of all three orders are unavailable. This has led to an increasing interest in learning from partially labeled data. Several attempts have been made towards partially supervised learning (PSL) in medical image analysis [Dong et al., 2022a], where human annotation costs are also high because annotators are required to have clinical expertise. For the first time, we study PSL in the domain of agricultural science. In this paper, we propose a label-efficient learning paradigm with partial supervision to adequately benefit from learning unlabeled, partially labeled, and fully labeled data, simultaneously. This method guides the segmentation of 3° veins based on the existing annotations of 1° and 2° veins. We conduct extensive ablation studies to understand the proposed method and HALVS. The empirical results also provide insights for future research on the task of interest under label scarcity. For example, we notice that the annotation efficiency varies significantly among different species. We are the first to investigate cross-species learning in the leaf vein segmentation task, where the segmentation model is trained by leaf venation data of species that are relatively more amenable to be annotated and makes predictions on the other species whose venation data are challenging to be annotated. The experiments show that cross-species learning is a challenging task under label scarcity and the annotations in HALVS can benefit future studies.

We summarize our contributions as follows:

• To the best of our knowledge, we release the first hierarchical leaf venation segmentation dataset, namely, HALVS, thereby establishing a benchmark for subsequent work and benefiting future studies.
• This is the first study that leverages partially labeled data in the plant phenomics domain and presents a new practical task domain for partially supervised learning.
• We evaluate the proposed method extensively on HALVS and achieve superior performance against stateof-the-art baselines. The experimental results also present initial empirical findings on hierarchical vein segmentation, paving the way for future investigations on the task of interest in the context of data scarcity.

2 RelatedWork

2.1 Leaf Vein Datasets

Due to the high human cost of pixel-level annotation, there are only a few public leaf vein datasets. To our best knowledge, none of these datasets offer annotations for leaf veins with hierarchical orders. To differentiate HALVS from the existing datasets, the limitations of these datasets are summarized in Tab. 2 in the aspects of image clarity, completeness of leaf blade, whether the raw leaf is chemically treated, and hierarchy of vein annotations. For example, LVD2021 dataset [Li et al., 2022] contains 4,977 low-contrast and lowresolution images captured by smartphone cameras and pixelwise annotations do not provide detailed structural information. Additionally, only 20 high-quality images have been annotated with non-human techniques using histogram equalization and binarization algorithms in [Iwamasa and Noshita, 2023]. HALVS is the first large-scale annotated dataset for high-quality images.

2.2 Partially Supervised Learning

An important research field of PSL is medical image analysis, where partially labeled datasets of different classes of interest are collected from different data sources due to high annotation cost [Dong et al., 2022b]. Many efforts have been paid into partially supervised medical image segmentation [Zhou et al., 2019; Fang and Yan, 2020; Shi et al., 2021; Dong et al., 2022a], a similar yet different task to leaf vein segmentation. First, in contrast to medical image segmentation, leaf vein segmentation is still an under-explored task. For medical images, transfer learning has been proven as an efficient solution for cross-sites and cross-modality setups, because of human structural similarity [Dong et al., 2022a]. However, this remains unclear in the leaf vein segmentation. Second, a direct application of PSL methods designed for medical images is infeasible for leaf images because of different image characteristics and task setups. For example, compared with leaf images in HALVS, medical images in previous PSL studies are "low-quality" in terms of resolution. Besides, the pixels of leaf veins are scarce, in contrast to most organs or human structures. The desired algorithms should be sensitive to pixel-level information. In this work, we propose the first PSL solution to leaf vein segmentation.

3 The HALVS Dataset

3.1 Data Collection

We collect 5,057 high-quality leaf images with precise venation details from three characteristic species to represent distinct venation characters, including 2,610 images of soybean, 1,947 images of sweet cherry, and 500 images of London planetree (see Fig. 1). The leaf images are acquired using a simple but efficient transmission scanning method, proposed by [Gan et al., 2019]. After being freshly picked from

(a) Soybean

(b) Sweet cherry (c) London planetree

Figure 2: Visualizations of leaves and corresponding vein annotations in HALVS (red: 1°, yellow: 2°, white 3°). Leaf patches are cropped in size of 256 × 256 for illustration purposes.

the tree and simply wiped clean, leaves are placed in the flatbed scanner (Epson Perfection V850 pro scanner) with the transmission scan mode. Images with 4800 × 6000 pixels at a resolution of 600 DPI are captured and saved in the 48-bit RGB PNG format. The leaf petiole can interfere with the segmentation of the primary vein, so in each leaf image, we crop it out and only retain the leaf blade. Then, we apply the minimum bounding rectangle algorithm to eliminate redundant background pixels and generate the final leaf image data.

3.2 Classes and Annotations

Determining the categories, or orders, of veins is the first and the most important step during the process of vein annotations. We recognize 1°, 2°, and 3° veins starting at the widest 1° vein and progressing to the finest 3° veins, following the roles and vein characteristics described in [Hickey, 1973] and [Ellis et al., 2009]. Generally, it is relatively easier to recognize the 1° and 3° veins, but sometimes the 2° veins are more complex as they may comprise several subsets with varying widths and courses. Nevertheless, all the vein subsets located between the 1° and 3° veins are considered to be 2° veins. Fig. 2 displays examples of our annotations. For each image, all three orders of veins are traced manually at full width. Despite the efforts, the annotation process for the HALVS dataset remains notably time-consuming. As detailed in Tab. 1, the average time for fully labeling the three orders of veins in a leaf blade image is about 2.1, 3.5, and 7.8 hours for soybean, sweet cherry, and London planetree, respectively. The significantly longer annotation time for the London planetree compared to the other two species is due to the larger leaf size. Labeling the 3° veins consumes considerably more time than labeling the 1° and 2° veins. As a result, 150 images (50 images per species) are densely annotated with fine-grained semantic segmentation labels, i.e. 7,831,792 pixels for 1° veins, 10,459,325 pixels for 2° veins, and 16,075,087 pixels for 3° veins. This is equivalent to around 13,000 non-overlapping patches in a size of 256 × 256. The whole annotation process takes up to 83.8 person-days by four experienced annotators following biological instructions.

4 Method

4.1 ProblemDefinition

In the task setup of interest, the dataset D = Dl ∪ Du ∪ Dp consists of fully labeled, unlabeled, and partially labeled sets. Dl = {(xi, yi)}Nl i=0 is the fully labeled set, where xi represents an image and each pixel belongs to one of four classes (i.e. the background, 1°, 2°, and 3° veins). The corresponding yi provides pixel-wise labels for xi. Du = {xj}Nu j=0 is the unlabeled set. Dp = {(xk, yk)}Np k=0 is the partially labeled set where yk only contains label information regarding 1° and 2° veins, missing the background and 3° veins, due to extremely high annotation cost of 3° vein. That is to say, we are only certain whether the pixel belongs to either 1° or 2° veins.

As shown in Tab. 1, annotating veins are time-consuming and expertise-demanding, and particularly, obtaining 3° vein labels is more challenging. We simulate a practical situation: Dl and Dp are small-scale datasets and Du is a large-scale dataset, i.e. Nl ≪ Nu and Np ≪ Nu. The learning objective is to train a leaf vein segmentation model with D. In contrast to standard SSL, we conjecture that the existence of Dp plays an important role in enhancing the model performance across all four classes.

4.2 Partially Supervised Semantic Segmentation

We propose a pluggable module for partially supervised semantic segmentation, which we denote as PSSS in the following context. The PSSS module can be easily integrated with state-of-the-art SSL frameworks [Wang et al., 2022; Yang et al., 2023]. The overall framework is illustrated in Fig. 3. The training on Dl and Du follows standard SSL

methods, where we use LS and LU to denote the supervised loss and unsupervised loss, respectively.

For simplicity, we assume that f is a neural network and (x, y) ∈ Dp is a sample-label pair where y is a partial label with respect to 1° and 2° veins. Following FixMatch [Sohn et al., 2020], there are a strong augmentation As and a weak augmentation Aw. We use p = f(As(x)) as the prediction output and ˆ y = argmax(f(Aw(x))) as the pseudo-label. Let S denote the set of pixels. S can be split into three sets, i.e. S = S1 ∪S2 ∪S3. S1 consists of the pixels corresponding to the 1° and 2° veins in the partial label y, i.e.

where m is the pixel of interest. S2 contains the pixels corresponding to the background and 3° veins with confidence higher than a threshold τ in the pseudo-label ˆ y, i.e.

where τ ∈ (0, 1) is the hyperparameter. S3 is the complement of S1 ∪ S2 under S.

For S1, the ground truth labels are available. The standard supervised learning is applied.

In Eq. (3), |·| denotes the cardinality of the set (i.e. the number of pixels) and H(·, ·) denotes the cross-entropy.

For S2, an unsupervised loss is computed using the pseudolabel ˆ y.

LuP utilizes consistency regularization to improve the model's generalization on unlabeled pixels and mitigate the overfitting. Due to class imbalance, the learned pseudo-labels might be dominated by the majority classes, thus exacerbating the overfitting. Note, in Eq. (2), only those pseudo-labels with high confidence participate in LuP . This excludes unreliable pseudo-labels that might sabotage the training. We will further discuss the impact of τ in the experiments.

The pixels in S3 are uncertain pixels, i.e. pixels with missing ground truth labels and with low confidence of the pseudo-labels. As the classes of interest are mutually exclusive, a pixel can only belong to one class. Though the remaining pixels in S3 can be either the background or 3° vein, they are highly possible to be 3° vein. The motivation here is that if the model can reliably predict the pixels belonging to the background, 1°, and 2° veins, the remaining pixels shall be the 3° vein. We hypothesize that the background class is much easier to predict in contrast to the vein classes. With LS and LU, the model can efficiently learn the patterns of the background as there are more non-vein pixels than vein pixels. Utilizing this prior knowledge allows the model to focus more on predicting the most challenging 3° vein class, making the segmentation task straightforward. We design a class-specific exclusion loss [Shi et al., 2021] to leverage these uncertain pixels:

where e = [1, 1, 1, 0]. As the labels for the 3° vein is scarce, LcP also regularizes the class imbalance.

To summarize, the loss for Dp is the sum of the three component losses above, i.e. LP = LsP +LuP +LcP . And the total loss for the optimization process is L = LS + LU + λLP , where λ is a hyperparameter controlling the weight of LP .

5 Experiments

The purpose of experimental design is twofold. First, we aim to evaluate the efficiency of PSSS on HALVS under label scarcity. Second, we aim to leverage HALVS to understand the challenges in cross-species transfer learning.

5.1 Experimental Setup

Baselines. The PSSS module can easily integrate into semisupervised semantic segmentation methods without changing the training pipeline, making it convenient for a comprehensive and practical evaluation. We choose three representative approaches as our baselines, namely: FixMatch [Sohn et al., 2020], U2PL [Wang et al., 2022], and UniMatch [Yang et al., 2023]. FixMatch is a seminal semi-supervised classification method that combines consistency regularization and pseudo-labeling. We adapt FixMatch to semantic segmentation as a strong baseline. U2PL and UniMatch are two recent state-of-the-art approaches in semi-supervised semantic segmentation. Additionally, we provide an "Oracle", where the training data D is completely labeled. "Oracle" is considered as the performance upper bound that a semi-supervised method can achieve. Similarly, we provide a performance lower bound, only using the labeled information to train a supervised model. We denote this baseline as "Supervised". Implementation. All experiments are implemented by PyTorch [Paszke et al., 2019] on two NVIDIA 4090 GPUs with 24G memory. The main purpose of the experiments is to evaluate the efficiency of PSSS in the default segmentation network DeepLabv3+ [Chen et al., 2018] with ResNet-101 [He et al., 2016] backbone, a strong semantic segmentation architecture. Following U2PL and UniMatch, we use a standard stochastic gradient descent optimizer with a batch size of 4. For a fair comparison, we adopt the poly learning strategy [Chen et al., 2017] for all methods, where the initial learning rate is 10-3 and the weight decay is 10-4 . We train all methods for 80 epochs and report the best performance measured in mean Intersection-over-Union (mIoU). By default, we set λ = 1 and τ = 0.95. To retain image details in our high-resolution data, we partition the original images into patches of size 256 × 256. The default data ratio of fully labeled, partially labeled, and unlabeled data is 1 : 1 : 10, where one basic unit denotes 6 images with full leaf blades (around 600 patches). For the class-balance purpose, the default class ratio among three species is 1 : 1 : 1.

5.2 Efficiency of PSSS

The evaluation results of PSSS with three seminal SSL methods are shown in Tab. 3. While the numerical results between different SSL baselines vary, the pattern is clear: adding PSSS to the SSL training can significantly improve the 3° vein IoU by 14.37%, 3.89%, and 11% on FixMatch, U2PL, and UniMatch, respectively. It also improves the mIoU of FixMatch, U2PL, and UniMatch by 8.55%, 2.57%, and 6.82%, respectively. It is worth mentioning that with PSSS, SSL baselines can achieve competitive performance with "Oracle", especially on the most challenging 3° vein.

In addition to quantitative evaluation, we perform qualitative comparison in Fig. 4. We use UniMatch as the baseline SSL method and visualize the segmentation results of "Supervised", "UniMatch" (Semi-Supervised), and "UniMatch + PSSS" (Ours) from the Tab. 3. It can be clearly seen that with PSSS, the segmentation performance is significantly improved across all three categories compared to the supervised and semi-supervised baselines, especially on the 3° vein. Meanwhile, we should also realize that hierarchical leaf vein segmentation is a challenging task and there are still lots of space for improvement.

5.3 Ablation Studies

In this section, we use UniMatch as the baseline SSL method to study the robustness of PSSS under the following setups. Effect of Component Losses in PSSS. As shown in Tab. 4, all three loss components of PSSS can improve the overall performance. It is interesting that adding our designed LcP not only significantly improves the performance of 3° vein, but also benefits the other two classes (2nd row vs 4th row, 3rd row vs 5th row). We conclude that LcP plays an important role in PSSS.

Sensitivity to τ and λ. As shown in Fig. 5, the model performance is unsatisfactory when the weight λ is too high or too low. This might be because such values disrupt the balance of the overall loss. As expected, higher confidence of τ is preferable, aiding in filtering out unreliable pseudo-labels. The optimal performance is observed with λ = 1 and τ = 0.95, adopted as the default setting for all experiments.

Sensitivity to Labeled Data Ratio. We study the impact of the relative ratio of partially labeled data and unlabeled data to fully labeled data. As shown in Tab. 5, the model's performance can be improved when only raising the ratio of partially labeled or unlabeled data. Also, only a slight increase in the ratio of partially labeled data can result in a non-trivial performance gain.

Effect of Dataset Size. We increase the volume of training data three times to study the scaling effect of dataset. As shown in Fig. 6, increasing data volume can achieve better performance under all three setups. However, there are two important findings. First, this scaling effect is not linear. There seems to be a diminishing return when the size of dataset becomes larger. Second, while increasing data volume results in some enhancement, the performance gain on segmenting the 3° vein remains limited. Importantly, though segmenting the 3° vein is difficult, this suggests that PSSS can use only 13 of training data to reach the performance upper bound while significantly outperforming supervised and semi-supervised baselines.

Effect of Network Backbones. We employ DeepLabV3+ as the architecture with ResNet101 and Xception [Chollet, 2017] serving as the backbones. Additionally, we select the Swin-Unet [Cao et al., 2022] architecture that employs Swin Transformer [Liu et al., 2021] as its backbone. As shown in Tab. 6, the PSSS module can improve the model's performance on all three representative backbones, proving its superior generalization capabilities.

5.4 Analysis of Cross-species Learning

We conduct the first study of cross-species learning for leaf vein segmentation. Based on Tab. 1, we find that soybean leaves are relatively labor-sparse and easy to be annotated in contrast to the other two species. Thus, soybean is considered the source species, and the other two species are considered the target species. In the first scenario, we assume that only soybean data are available and we perform direct transfer learning to two other species unseen in the training. We use the default ratio 1:1:10 where the basic unit contains 20 images with full blades. The results are shown in Tab. 7.

Without seeing any samples from the target species, simply adding more partially labeled and unlabeled data from the source species can improve the overall model generalization on the target species, especially on the 3° vein.

In the second scenario, we consider an extreme label scarcity case: in addition to small-scale fully labeled source data, the target species can have access to unlabeled data and small-scale partially labeled data in the training. With the default ratio, there are 6 fully labeled source leaves, 6 partially labeled, and 60 unlabeled target leaves. The results are shown in Tab. 8. We also include an "Oracle" for supervised training on fully labeled target species. We notice that though our method significantly outperforms the semi-supervised counterpart, the supervised Oracle can easily achieve better overall performance. We also conclude that, due to dataset shift [Qui˜nonero-Candela et al., 2008] between species, human annotations are still the key to successful applications. This further supports that HALVS is important for future research on the task of interest.

6 Conclusion

In this work, we explore the novel task of leaf hierarchical venation segmentation. We provide a finely annotated HALVS dataset for the first time and propose a label-efficient learning paradigm by considering the practical difficulties in annotating leaf veins. The empirical studies not only reveal new observations and challenges but also pose an insight into future research directions.

Acknowledgements

This study was partially supported by the National Science Foundation of China (No.32090061, No.32200331), the Major Science and Technology Research Project of Hubei Province (No.2021AFB002), the Start-Up Grant fromWuhan University of Technology of China (No.104-40120526), and Shanghai Artificial Intelligence Laboratory.