CLE-ViT: Contrastive Learning Encoded Transformer for Ultra-Fine-Grained Visual Categorization

1 Introduction

Ultra-fine-grained visual categorization (ultra-FGVC) distinguishes a sub-category of images from a single fine-grained category. As an emerging topic, ultra-FGVC demonstrates potential in artificial intelligence agriculture and smart farming e.g., automatic crop cultivar classification and plant disease classification [Yu et al., 2020; Larese et al., 2014; Yu et al., 2023]. The intrinsic challenge of ultra-FGVC lies in that very limited samples are provided due to the granularity further moving down to a sub-category level [Huang and Li, 2020]. Another key observation is that the visual variances are difficult to distinguish among different classes, while intra-class variances can be very large (see Figure 2 as an example). Thus how to learn discriminative representation within limited training samples becomes a core question in ultra-FGVC.

Recent progress has been made by incorporating selfsupervised learning to jointly optimize the objectives of representation learning. This is achieved by training with some predefined pretext tasks to drive the model to better understand the intrinsic feature of the data itself. For instance, learning a representation by training a model to predict the rotation [Gidaris et al., 2018] or spatial context [Doersch et al., 2015] of input images. Several works [Yu et al., 2023; Yu et al., 2022] have demonstrated such a strategy can lead to significant performance gain. However, none of these works consider explicit supervision for learning mutual information at instance level. This motivates us to introduce selfsupervised instance-level contrastive learning to gain a more discriminative representation via understanding mutual information between augmented views of a single image.

In this paper, we introduce CLE-ViT, a novel contrastive learning encoded transformer, to address the intrinsic challenges of ultra-FGVC. The core design is a self-supervised module that performs self-shuffling and masking and then distinguishes these altered image views from other images. This drives the model to learn an optimized feature space that has a large inter-class distance while remaining tolerant to intra-class variations. By incorporating this self-supervised module, the network acquires more knowledge from the intrinsic structure of the input data, thus improving the generalization ability with limited training samples.

CLE-ViT achieves superior (or comparable) performance on five ultra-fine-grained datasets, one plant disease dataset, and the CUB-200-2011 dataset. With promising performance on crop cultivar classification and plant disease classification, the proposed CLE-ViT may provide a promising solution to pushing forward the progress in smart farming. The main contributions of this paper are summarized as follows:

1. We introduce CLE-ViT, a novel vision transformer model that performs instance-level contrastive learning towards discriminative classification with limited training data.
2. CLE-ViT learns an optimized feature space that has a large inter-class distance while remaining tolerant to intraclass variance.
3. CLE-ViT achieves strong performance on five ultra-finegrained datasets and two fine-grained datasets, demonstrating its effectiveness for ultra-FGVC.

2 RelatedWork

Ultra-fine-grained visual categorization (ultra-FGVC) identifies objects at a very fine granularity where even humans feel difficult to accurately describe the visual difference. In comparison with fine-grained visual categorization (FGVC), ultra-FGVC has two unique properties/challenges: 1) the annotations are not labeled by human experts or volunteers but obtained from genetic source bank [Yu et al., 2021b]; 2) the classification granularity has moved from species level (FGVC) to a subordinate level, i.e., cultivar level [Yu et al., 2020]. An example of comparing ultra-FGVC and FGVC is shown in Figure 2. FGVC aims to distinguish between images from the top row and those from the bottom row, while ultra-FGVC classifies different images from a single row. This challenging research topic is attracting increasing attention for its significant potential in artificial intelligence agriculture and smart farming.

Earlier exploration started with a small ultra-fine-grained image dataset, which contains 600 images of 100 soybean cultivars [Yu et al., 2020]. Despite encouraging performance on this challenging dataset, their proposed feature modeling method requires a manually segmented vein structure that is inherently difficult to use for practical applications. Recently, [Yu et al., 2021b] released a benchmark platform with baseline performances of 13 state-of-the-art CNN methods on a large-scale ultra-fine-grained cultivar leaf dataset including in total of 47,114 leaf images from two plant species and 3,526 different cultivars. [Yu et al., 2021a] proposed a randommask covariance network (MaskCOV) to learn discriminative representation for ultra-FGVC. The MaskCOV randomly shuffles and masks out image patches, and then predicts the original position of each patch via a self-learning module. Despite its state-of-the-art performance on the ultra-FGVC tasks, the MaskCOV together with all the baseline methods in [Yu et al., 2021b] are all CNNs.

Transformer [Vaswani et al., 2017] was first developed on natural language processing and is now gaining increasing attention due to its effectiveness in extensive computer-vision tasks. Especially the Vision transformer (ViT) [Dosovitskiy et al., 2020] which adopted a pure transformer directly to deal with sequences of image patches, has now demonstrated very competitive performance in image classification. Yet ViTbased methods require a large-scale dataset for model pretraining. To that end, [Touvron et al., 2021] introduced DeiT, that employed a teacher-student strategy to speed up ViT training. Transformer models were further applied to other popular computer vision tasks. TransFG and FFVT [He et al., 2022; Wang et al., 2021a] proposed to use a few important tokens for final classification and explored ViT in the context of fine-grained visual classification. More recently, Mix-ViT [Yu et al., 2023] and SPARE [Yu et al., 2022] both introduce predefined pretext tasks as a supervision signal for implicit self-supervised learning in ultra-FGVC. Mix-ViT predicts the position of mixed tokens. SPARE classifies masked semantic part regions. None of them consider explicit supervision for learning mutual information at the stance level. In contrast, we develop explicit supervision for learning mutual information at the instance level. Our proposed self-supervised instance-level contrastive learning enables a desirable feature space that has a large inter-class distance while remaining tolerant to intra-class variance.

3 Methods

3.1 Overview&Motivation

Overview. Figure 4 illustrates the overall framework of the proposed CLE-ViT. A given image is first projected into two different views via the following operations: 1) standard augmentation and 2) random shuffling and masking. Then the two views are sent to the backbone network for feature extraction. In addition to the standard classification optimization, their output features form a positive pair, while remaining images from the same batch form negative samples for self-supervised contrastive learning. The whole network is trained in an end-to-end manner. The self-supervised module is detached in the inference stage.

Motivation. A desirable learned feature space should be tolerant to intra-class variations. Once the model is overfitting the training samples from the same category, the associated feature space of this class may be closely clustered (as demonstrated in Figure 1). This may hurt the generalization capability as the new (testing) samples may not fall into such a concentrated cluster due to the intrinsic intra-class variations in ultra-fine-grained samples (see Figure 2). To that end, we propose to conduct the contrastive learning on instance-level, instead of the class level in previous works [He et al., 2022; Wang et al., 2021b]. On one hand, this creates more diversified training samples to enable a more intra-class variance-tolerant feature space such that the training process becomes less likely to overfit. On the other hand, the diversified samples may also enlarge the feature space distance between different categories (see Figure 3).

3.2 Image Classification

Given an input image I ∈ RH×W×3 and its associated onehot category label y ∈ R1×Nc , we firstly employ a feature extractor, e.g., ResNet50 [Simonyan and Zisserman, 2015] and Swin Transformer [Liu et al., 2021], to obtain its patch features V ∈ RNp×D and global feature representation u ∈ R1×D which is used to undertake the category categorization via the classification head. Note that the H, W, Nc, Np, D are the height, and width of the image, the number of total classes in the datasets, the number of patches in the final stage, and the feature dimension of the final global features respectively. This process can be expressed as:

Where σ denotes the softmax function and p is the probability distribution given by the backbone model. fife refers to the image feature extractor. After obtaining the category prediction, the model is normally optimized by a Cross-Entropy loss in an end-to-end manner:

3.3 Instance-level Contrastive Learning

The proposed instance-level contrastive learning module is trained in a self-supervised manner as it is formed without the need for any extra manual label information. When there are only limited training samples, incorporating such self-supervised tasks can improve the representation learning without requiring extra annotations.

Positive Pair Construction

The anchor image is obtained by applying the standard data augmentation methods, e.g., Random Horizontal Flip, and Random Rotation to the input image. To form the positive sample for the anchor image, we first follow the same standard data augmentation methods as the anchor image to generate the base image, and perform a strong data augmentation, i.e., randomly mask out a proportion of pixels. Specifically, given the anchor image Ia ∈ RH×W×3 , the process of obtaining the positive sample Ip∗ ∈ RH×W×3 can be formulated by:

, where (i, j) denotes the position index in the images and H, is the set containing all the pixel positions to be masked.

k and t determine the masked size and are randomly selected. These parameters are controlled by a term α = k∗t H∗W , the proportion of masked region.

However, the model can easily infer the original information of masked pixels from the anchor image containing the masked regions within minor transformation in standard augmentation. To this end, we propose to randomly shuffle the positive image to enhance the difficulty in not only acquiring the masked information from the anchor image but also recognizing the category of the image. In particular, we evenly split the image into n = s ∗ s part where s is an integer larger than 1, then the masked image can be reformulated as:

where we represent the P(i, j) as the image patch in ith row and j th column. The final positive sample Ip is obtained by randomly shuffling the n parts in Ip∗ . An example of generating Ip is shown in Figure 5.

Negative Pair Construction

Since we aim to establish instance-level contrastive learning, the negative sample can be any other sample in the same batch. In detail, for each sample, we randomly pick up one anchor image (excluding itself) from the batch as its negative sample In to form the negative pair < Ia, In >.

Instance-level Contrastive Learning

After obtaining the positive pair < Ia, Ip > and negative pair < Ia, In >, the next problem is how to conduct the contrastive learning. Here, we employ the triplet loss as our contrastive learning metric. Specifically, let's denote the global features of the anchor image, its positive sample and negative sample as ua, up and un respectively, our instance-level triplet loss is formulated as:

where B is the number of samples in one batch and i denotes the ith sample in the batch. β refers to the margin to control the difficulty in contrastive learning. σ is the L2 normalization operation. Note that our proposed self-supervised approach can be added to any triple losses.

3.4 Objective Function

The model is jointly optimized by the Cross-Entropy Loss (Equation 3) and the proposed instance-level contrastive loss (Equation 7). We denote the Cross-Entropy Loss on anchor images as Lclsa and its positive images as Lclsp. Then, the final objective function can be expressed as:

where λ and γ are the hyper-parameters to control the contribution among the classification loss on anchor and positive images, and the self-supervised loss.

3.5 Discussion

Why instance-level contrastive learning? To enable instance-level contrastive learning, the first step is to perform in-image augmentation to generate instance-level positive pairs. Such in-image augmentation creates more diversified samples such that the training process is less likely to overfit. More importantly, distinguishing different views of the same sample from other samples can reshape the learned feature space to have a larger distance between different categories and within the same category. As such, the learned feature space is more generalized to adapt to new samples. Moreover, we also perform class-level contrastive learning as an ablation study for a comprehensive evaluation (will be discussed in Section 4.4).

Why triplet loss? Triplet loss and infoNCE loss [Oord et al., 2018] share a similar spirit of separating positive samples from negative samples. Yet infoNCE loss treats each negative sample as a unique category which is less applicable in instance-level setting given a number of more than 10K instances (categories) for some benchmarks. In addition, applying triplet loss focuses on optimizing the distance between positive samples and negative samples rather than urging the model to map the positive pairs to the same points in InfoNCE, thus can avoid overfitting and enhance generalization capability.

4 Experiments

4.1 Datasets &BenchemarkMethods

Following [Yu et al., 2023], five ultra-fine-grained image datasets are adopted for evaluation including Cotton80, SoyLocal, SoyGene, SoyAgeing and SoyGlobal. Moreover, two fine-grained datasets, Apple Foliar disease dataset [Thapa et al., 2020] and CUB-200-2011 (CUB) [Wah et al., 2011] are also used to further verify the effectiveness of the proposed method. Table 1 summarizes the statistics of benchmark datasets, i.e., the numbers of classes, training images, and testing images. For fair comparisons, the proposed CLE-ViT is compared with the same 17 benchmark methods as adopted in [Yu et al., 2023] for comprehensive evaluations.

4.2 Implementation

Model Details. We adopt the Swin Transformer Base (SwinB) [Liu et al., 2021] as our backbone model by taking both precision and efficiency into consideration. The same as the most transformer-based works [Yu et al., 2023; Touvron et al., 2021], our backbone is initialized by the ImageNet21K [Deng et al., 2009] pre-trained model. The proportion of the masked region and the number of parts n and are set to [0.15, 0.45] and 4 respectively. The margin β in Equation 7 is 1. λ and γ are both set to 1 for all datasets except 0.3 and 0.5 for CUB dataset.

Training and inference. Following the [He et al., 2022; Touvron et al., 2021; Wang et al., 2021a], input images are first resized to 600×600 for all datasets. Random (Center) cropping is then applied to crop the images into 448×448 during the training (inference) phase. After that, we adopt random horizontal flipping, color jitter, and random rotation during the training. The standard augmentation consists of the aforementioned transformations. The whole architecture is optimized by AdamW optimizer. In our experiment settings, the batch size and the learning rate are set to 12 and 1e-3 for all the datasets.

4.3 Comparison to The State-of-The-Arts

Evaluation on ultra-fine-grained image datasets. Table 2 lists the classification accuracy of all the competing methods and their backbone networks on five ultra-fine-grained datasets. The proposed CLE-ViT achieves 75.21% classification accuracy on the SoyGlobal dataset, outperforming other competing methods with a significant margin (more than 19%). We also observe a similar trend in SoyAgeing, where the proposed method surpasses other competing methods with a margin of more than 5% of classification accuracy. Among the ultra-fine-grained image datasets, the SoyAgeing dataset covers five subsets and each subset contains images collected from a specific cultivating stage. The comparison results of all competing methods on the five subsets are summarized in Table 3. The proposed method achieves strong performance compared with other competing methods, demonstrating its effectiveness in ultra-FGVC tasks.

Evaluation on fine-grained datasets. we present evaluation results in Table 4 of the proposed CLE-ViT on the widely used fine-grained image dataset, CUB-200-2011 [Wah et al., 2011]. CLE-ViT achieves competitive performance (ranked 2nd) among the state-of-the-art methods on CUB-200-2011, demonstrating its effectiveness and generalization capability in fine-grained visual classification. To further verify the effectiveness of the proposed method, we evaluate a plant disease classification image dataset, the Apple Foliar disease dataset. The comparison results are summarized in Table 2. The proposed method achieves the best classification accuracy of 97.58% on the Apple Foliar disease dataset.

Visualization. We visualize the attention maps of Mix-ViT [Yu et al., 2023] and the proposed CLE-ViT on the CUB dataset [Wah et al., 2011] in Figure 6, where highlighted regions contribute significantly to the visual categorization. We observe that the attentive regions of birds consistently cover unique patterns, verifying the reliability of the effectiveness. This is consistent with human observations and common sense that visual features from those unique patterns are vital for the determination of bird species.

4.4 Ablation Study &Analysis

Role of the self-supervised contrastive learning module. To further verify the contribution of the proposed selfsupervised contrastive learning module, we present a comprehensive ablation on 4 benchmark datasets. The baseline removes the self-supervised module from the CLE-ViT. In addition, contrastive learning can also be used at class level, i.e., all images from the same category are used to form positive pairs while remaining images from other categories are negative samples. For a comprehensive evaluation, we replace the instance-level contrastive learning with a standard classlevel contrastive learning, denoted as Baseline+Class Contrastive (Baseline+CC). The comparison results of baseline, baseline+CC, and CLE-ViT are shown in Figure 8. We observe that CLE-ViT consistently improves the performance over the baseline method and Baseline+CC, verifying the effectiveness of the proposed self-supervised learning module. Feature space. Figure 7 shows tSNE visualization of learned features from baseline, baseline+class contrastive and ours. Here the samples are randomly selected from 10 categories (testing set). We observe that the clusters from the baseline are not well separable. The tSNE result from Baseline+Class Contrastive shows a larger inter-class distance compared with baseline while the intra-class distance remains small. This may hurt the generalization ability given that samples in ultrafine-grained image datasets often have large intra-class variance, e.g., the point in the top left corner 7 (b). In contrast, the proposed CLE-ViT shows both large inter-class distance and intra-class distance, indicating a better generalization capability.

Ablation study on Lclsp. Table 5 shows an ablation study of Lclsp on Soy.Loc dataset by varying the weight from 0 to 1.5 at a step size of 0.5. Here λ partially balances the contribution from instance-level contrastive learning and categorylevel learning. λ = 0 means positive samples are unable to receive category-level supervision. Without category-level supervision, contrastive learning can still optimize the distance at the instance-level but might be unable to predict the category label of positive samples. As λ increases, the contribution from category-level supervision becomes larger and achieves the best performance when λ = 1. But when λ becomes too large, the category-level supervision will completely dominate the training while ignoring the contribution from contrastive learning, thus may lead to overfitting and performance drop.

5 Conclusion

This paper introduced a novel contrastive learning encoded vision transformer, CLE-ViT, to address intrinsic challenges in ultra-fine-grained visual categorization. A new selfsupervised learning module has been proposed, which drives the model to learn an optimized feature space that has a large inter-class distance while remaining tolerant to intra-class variations. By incorporating this self-supervised module, the network acquires more knowledge from the intrinsic structure of the input data, which improves the generalization ability of limited training samples. CLE-ViT has achieved competitive performance on seven public datasets, demonstrating its effectiveness in ultra-FGVC, birds, and plant disease classification tasks.

We also observe that negative samples are randomly sampled in batches for better efficiency, thus potentially introducing easy pairs. A promising direction in future work is to explore efficient hard sample mining approaches when forming negative pairs.

Contribution Statement

Xiaohan Yu and Jun Wang contributed equally to this paper.