Open-World Semi-Supervised Learning with Class Semantic Correlations

1 Introduction

In the field of machine learning [LeCun et al., 2015; Tu et al., 2024] , traditional semi-supervised learning (SSL) frameworks [Sohn et al., 2020] have provided an effective solution for scenarios with limited labeled samples. However, these frameworks typically operate under a strict assumption that the labeled samples encompass all classes within the application context. In real-world scenarios, especially in open environments where samples continuously arrive and unknown classes emerge, this assumption often does not hold [Li et al., 2021] . This challenge gives rise to the concept of open-world semi-supervised learning (OWSSL) [Cao et al., 2022] , which transcends the constraints of traditional SSL frameworks and aligns more closely with the complexities of the real-world. It requires models to be capable of effectively recognizing both known and unknown classes in unlabeled samples, even when the labeled samples only cover the known classes. Figure 1 illustrates the scenario encountered by OWSSL.

∗Corresponding author

OWSSL is currently receiving widespread attention with a range of methods being proposed. For instance, ORCA [Cao et al., 2022] introduces an adaptive margin mechanism to mitigate the model bias towards known classes, OpenLDN [Rizve et al., 2022] proposes a bi-level optimization rule to generate relatively reliable pseudo-labels for unknown classes, and PromptCAL [Zhang et al., 2023] proposes a twostage framework to tackle the class collision issue caused by false negatives. The methods described above make significant progress, but they generally face a common issue: they primarily represent classes as symbolic variables. This representation can only indicate a sample's membership in a certain class and thus ignores the rich internal semantic information associated with the classes. This simplification limits the model to learning only the simple correspondence between samples and class symbols, resulting in weakly discriminative image representations, thereby impairing its ability to recognize unknown classes.

To address these issues, recent studies assume knowledge of all class names and provide corresponding textual descriptions to fine-tune both image and text encoders, aiming to better understand the correspondence among visual features of samples and textual descriptions of classes. CLIP-GCD [Ouldnoughi et al., 2023] first introduces textual descriptions and uses image-text contrastive loss to fine-tune the pretrained CLIP model [Radford et al., 2021]; TextGCD [Zheng et al., 2024] then introduces a teacher model to mitigate the image-text matching noise in CLIP-GCD and has become the state-of-the-art in OWSSL. However, these methods do not consider the class semantic correlations when aligning the visual features of samples with their corresponding textual descriptions, resulting in insufficiently discriminative image representations. Moreover, fine-tuning the text encoder may disrupt the rich semantic correlations learned during pretraining phase, potentially leading to overfitting to the provided textual descriptions [Shu et al., 2023]. Both of these issues consequently affect the model's ability in recognizing unknown classes.

In light of the issues of existing methods, we propose a novel OWSSL method. This method fine-tunes only the image encoder during training while freezing the text encoder, thereby preserving the rich semantic correlations learned during the pre-training phase. Furthermore, our method employs a semantic margin to extract class semantic correlations from textual descriptions, which are then used in enhancing image representation discriminability. Experimental results demonstrate that our method achieves a significant performance improvement compared to representative OWSSL methods. The main contributions of this paper are as follows:

• We identify the key issue in existing OWSSL methods that utilize textual descriptions: they do not consider the class semantic correlations when aligning the visual features of samples with their corresponding textual descriptions, resulting in insufficiently discriminative image representations, thereby affecting their ability to recognize unknown classes.
• We propose a novel OWSSL method named CSCOWSSL, which freezes the text encoder, thereby preserving the rich semantic correlations learned during the pre-training phase and employs a semantic margin to extract class semantic correlations from textual descriptions, which are then used in enhancing image representation discriminability.
• We conduct comprehensive experiments on multiple fine-grained datasets. The experimental results demonstrate that our method achieves a significant performance improvement compared to representative OWSSL methods, fully illustrating the effectiveness of our method.

2 RelatedWork

2.1 Open-WorldMachine Learning

Most machine learning methods operate under the closedworld assumption, which frequently proves inadequate in real-world scenarios. To address this, methods such as openset recognition (OSR) [Scheirer et al., 2013] , robust semisupervised learning (Robust SSL) [Oliver et al., 2018] , and novel class discovery (NCD) [Han et al., 2019] have been developed for open-world adaptation. OSR requires models to recognize unseen classes during testing without compromising known class accuracy. Robust SSL assumes that unlabeled samples may contain classes not represented in the labeled samples, and the goal is to minimize any negative impact of unknown classes on the performance of known class classification. NCD assumes that the unlabeled samples consist solely of unknown classes, requiring the model to recognize these novel classes. Although these methods make progress, they do not fully capture the complexity of openworld scenarios. To effectively address open-world machine learning challenges, OWSSL is proposed to better align with real-world complexities.

2.2 Open-World Semi-Supervised Learning

[Cao et al., 2022] first proposes the concept of OWSSL, which requires the model to recognize both known and unknown classes simultaneously, thereby significantly enhancing the flexibility and generalization capabilities of SSL. [Vaze et al., 2022] introduces the concept of Generalized Category Discovery (GCD), which is similar to the concept of OWSSL and is often discussed alongside it. Existing OWSSL methods can primarily be categorized into two types: one type represents classes as symbolic variables, with representative methods such as SimGCD [Wen et al., 2023] and GPC [Zhao et al., 2023]; the other type incorporates textual descriptions of classes, with representative methods such as CLIP-GCD [Ouldnoughi et al., 2023] and TextGCD [Zheng et al., 2024]. To broaden the application scope of these methods, some researchers extend them to various fields such as point cloud segmentation [Riz et al., 2023] and intent classification [Shi et al., 2024; An et al., 2024] . Nevertheless, most research still focuses on image classification tasks, as they are crucial scenarios for testing the model's generalization ability and its capability to handle unknown classes. Therefore, in this paper, we choose to apply and validate the effectiveness of our proposed methods on image classification tasks.

2.3 Vision-Language Models

In recent years, Visual-Language Models (VLMs) have made significant advancements. These models are designed to process and understand visual and textual information. The main objective in this field is to train VLMs with a vast number of image-text pairs, allowing them to capture the correlation between visual content and textual descriptions. These models exhibit exceptional performance across various visuallanguage tasks [Zang et al., 2024]. Specifically, CLIP [Radford et al., 2021] attains impressive capabilities in understanding cross-modal concepts and correlations by aligning image and text features in a shared latent space, after being trained on extensive datasets. This robust generalization ability, along with the model's capability to handle multi-modal information, makes CLIP suitable for OWSSL scenarios, as it can adapt to and manage a broad spectrum of visual and textual inputs, which is essential for OWSSL methods.

3 ProposedMethod

In this section, we first describe the OWSSL setting. Subsequently, we introduce each part of our method. The overall framework of our method is depicted in Figure 2 .

3.1 Problem Setting

Given a labeled dataset Dl containing n samples, Dl = {(x1, y1) , (x2, y2) , · · · , (xn, yn)}, and an unlabeled dataset Du containing m samples, Du = {xn+1, xn+2, · · · , xn+m}, it is usually assumed that m ≫ n. Here, x ∈ RD , y ∈ Cl, where D represents the feature dimensions of the samples. Cl denotes the set of classes in Dl and Cu denotes the set of classes in Du, C = Cl ∪ Cu. In OWSSL, Cknown = Cl represents the set of known classes and Cunknown = Cu\Cl represents the set of unknown classes. Our goal is to learn a classification model f to classify samples into known and unknown classes. The model is trained on Dl and Du, and then evaluated on Du. Knowing the number of unknown classes is a fundamental assumption in most OWSSL methods.

In this paper, we utilize deep neural networks to build our classification model f(x; θf ), which consists of an image encoder I(x; θI) : RD → Rd , a classifier h(I(x; θI); θh) : Rd → R|C| , and a text encoder T(t; θT ) : RD → Rd . Here, D denotes the feature dimension of x, d represents the feature dimension following the processing by the image encoder, and t represents the textual descriptions. The dataset is processed in batches on the model, where B indexes all samples in a given batch, Bl indexes all labeled samples within that batch.

3.2 Baseline

We first establish a baseline model, which consist of the cross-entropy loss for labeled samples and the self-distillation loss [Assran et al., 2022] for all samples. These loss functions are widely used in OWSSL methods [Cao et al., 2022; Wen et al., 2023]. The cross-entropy loss on labeled samples is as follows:

where H denotes the cross-entropy function, p(xi ) = h (I(x; θI); θh), σ(·) is the softmax function, and τs is a temperature value. The self-distillation loss on all samples is as follows:

where τt is a temperature value. R(p) is a mean-entropy maximization regularization term designed to prevent trivial solutions during initial training processing [Cao et al., 2022], and p = 1 2|B| ∑i∈B σ(p(xi ), τs ) + σ(p(x′i ), τs ) is the mean softmax probability of a batch. Finally, the baseline loss is as follows:

where α is a hyper-parameter, which is set to 0.35 by default. However, Lbase represents classes as symbolic variables, which can only indicate a sample's membership in a certain class and thus ignore the rich internal semantic information associated with the classes. This simplification limits the model to learning the simple correspondence between samples and class symbols, which hampers its ability to recognize unknown classes. Therefore, we need to incorporate corresponding textual descriptions of classes to facilitate training.

3.3 Semantic Margin Contrastive Loss

Recent OWSSL methods [Ouldnoughi et al., 2023; Zheng et al., 2024] assume knowledge of all class names and provide corresponding textual descriptions to fine-tune both image and text encoders, aiming to better understand the correspondence among visual features of samples and textual descriptions of classes. These methods fine-tune both image and text encoders using the image-text contrastive loss, denoted as Litc, which is defined as follows:

where S is a similarity metric, specifically the cosine similarity, τk denotes the temperature value, and t represents the textual descriptions generated by a large language model (LLM). tyi represents the textual description associated with the sample xi. It is retrieved by an auxiliary VLM from a knowledge base that encompasses textual descriptions for all classes, and it is the description that, after being processed by the text encoder, exhibits the highest similarity to xi as processed by the image encoder. For more detailed information on the implementation, please refer to TextGCD [Zheng et al., 2024]. Moreover, the class label yi corresponding to tyi is also used as supervisory information within Lbase to guide the training process of the model. However, these methods only align the visual features of samples with their corresponding textual descriptions, yet treat all non-corresponding textual descriptions equally, neglecting the class semantic correlations. Furthermore, fine-tuning the text encoder may disrupt the rich semantic correlations learned during pre-training phase, potentially leading to overfitting to the provided textual descriptions. Both of these issues consequently affect the model's generalization ability in recognizing unknown classes. To address these issues, we take the following steps:

First, we fine-tune only the image encoder during training while keeping the text encoder frozen. This approach preserves the rich semantic correlations that were learned during the pre-training phase. Furthermore, we employ a semantic margin to extract class semantic correlations from these descriptions [Shu et al., 2023] and incorporate them into the image-text contrastive loss to fine-tune the image encoder, thereby enhancing image representation discriminability. The semantic margin is as follows:

where i, j ∈ {1, 2, · · · |C|}.

Based on the semantic margin, the semantic margin contrastive loss is as follows:

where β is a hyper-parameter, which is set to 0.3 by default. Unlike Litc, which treats all non-corresponding textual descriptions equally, Lsmc takes into account the class semantic correlations. When the semantic similarity between classes yi and k is small, indicating a distant semantic correlation, the value of the semantic margin Myi,k is larger. This larger margin effectively distances the image features of xi from the non-corresponding textual descriptions associated with class k. On the other hand, when the semantic similarity between yi and k is significant, suggesting a closer semantic correlation, the semantic margin Myi,k is reduced. This reduced margin allows the image features to be closer to the textual descriptions of classes that are semantically similar to yi, thus enhancing the precision of distinguishing visual features from non-corresponding textual descriptions and facilitating the extraction of class semantic correlations.

Additionally, we refine the approach for obtaining textual descriptions. In the existing OWSSL methods, textual descriptions are typically acquired by querying the LLM directly, without sufficient consideration of the significance of prompt templates. It is recognized that effective prompt templates can significantly enhance the quality of the LLM's output. Therefore, we incorporate the prompt templates proposed in [Pratt et al., 2023] to enhance the quality of the textual descriptions generated by the LLM.

Finally, the overall loss function of our proposed method is given as follows:

During the testing phase, the classification model f(x; θf) uses the classifier h(I(x; θI); θh) : Rd → R|C| to assign the corresponding class labels to each input sample xi ∈ Du, thereby completing the classification of the samples.

4 Experiments

In this section, we conduct a comprehensive evaluation of our method. The experimental results and detailed analysis demonstrate the superiority of our method.

4.1 Experimental Setup

Datasets

We conduct experiments on five fine-grained datasets: CUB [Wah et al., 2011] , Stanford Cars [Krause et al., 2013] , Flowers102 [Nilsback and Zisserman, 2008], and ImageNet100 [Deng et al., 2009], which contain 200, 196, 102, and 100 classes, respectively. To ensure the fairness of the experiment, we conduct our experiments using the data partitioning method described in [Zheng et al., 2024]. We adopt the same approach to divide the classes into known and unknown, considering 50% of the classes as known and the remaining 50% as unknown. Consequently, we construct the datasets Dl and Du accordingly. We train our method on Dl and Du, and subsequently evaluate its performance on Du. This procedure is applied consistently across all compared methods. Detailed dataset information is provided in Table 1 .

Compared Methods

For OWSSL methods represent classes as symbolic variables, we select ORCA [Cao et al., 2022], GCD [Vaze et al., 2022], SimGCD [Wen et al., 2023], GPC [Zhao et al., 2023], DCCL [Pu et al., 2023], and PromptCAL [Zhang et al., 2023] as representative methods. For OWSSL methods that incorporate textual descriptions of classes, we choose CLIP-GCD [Ouldnoughi et al., 2023] and TextGCD [Zheng et al., 2024] as representative methods. These selected methods collectively showcase the latest advancements in the field of OWSSL.

Evaluation Protocol

Following the approach in [Cao et al., 2022], we evaluate the performance of our method using accuracy (ACC) and clustering accuracy (CACC). Specifically, ACC and CACC are calculated on dataset Du, as illustrated in the following equations:

where labeli represents the ground-truth class label of xi ∈ Du, which is provided only during the testing phase. OP stands for optimal permutation, which aligns Resulti with labeli. Our evaluation includes calculating ACC for samples from known classes (K) in Du, as well as calculating CACC for all samples (A) and samples from unknown classes (U) in Du.

Implementation Details

We use a ViT-B/16 [Dosovitskiy et al., 2021] pre-trained with CLIP as the backbone of the image and text encoder, setting the output dimension of the backbone to 512. During training process, we adjust only the last block of the image encoder and freeze the text encoder. To ensure consistency with the experimental setup of other comparison methods, we follow the protocol described in TextGCD [Zheng et al., 2024]. We use a batch size of 128 and train for 200 epochs with an initial learning rate of 0.1. We adjust the learning rate using a cosine annealing schedule. The parameters are set as follows: α to 0.35, λ1 to 1, τs to 0.1, and τk to 0.01. τt is initialized to 0.07 and gradually increased to 0.04 using a cosine annealing schedule during the first 30 epochs of training. We choose GPT-3 [Brown et al., 2020] as our LLM and utilize the prompt templates proposed by [Pratt et al., 2023] to generate textual descriptions, and choose ViT-H based CLIP model as the auxiliary VLM. To validate the generalization capability of our method across different datasets, we pretrain the model using [Vaze et al., 2022], uniformly setting λ2 to 0.2 and β to 0.3. All our experiments are conducted on a single NVIDIA 3090 GPU.

4.2 Main Results

The classification accuracy of different methods across various datasets is provided in Table 2. We report the average maximum classification accuracy from three runs for our method, whereas the remaining results are obtained from [Wen et al., 2023; Zheng et al., 2024]. The experimental results show that our method significantly outperforms previous methods across various different datasets.

Compared to OWSSL methods that represent classes as symbolic variables, which can only indicate a sample's membership in a certain class and thus ignore the rich internal semantic information associated with the classes, this simplification causes the model to learn only a simple correspondence between samples and class symbols, resulting in weakly discriminative image representations, thereby limiting its ability to recognize unknown classes. Our method incorporates textual descriptions and aims to better understand the correspondence between the visual features of samples and the textual descriptions of classes. As a result, it outperforms methods that rely solely on symbolic variable representations.

Compared to OWSSL methods that incorporate textual descriptions of classes, these methods do not consider the class semantic correlations when aligning the visual features of samples with their corresponding textual descriptions, resulting in insufficiently discriminative image representations. Moreover, fine-tuning the text encoder may disrupt the rich semantic correlations learned during pre-training phase, potentially leading to overfitting to the provided textual descriptions. Both of these issues consequently affect the model's ability in recognizing unknown classes. In contrast, our method fine-tunes only the image encoder while keeping the text encoder frozen, thereby maintaining the semantic correlations established during pre-training. Additionally, we introduce a semantic margin contrastive loss to utilize class semantic correlations from textual descriptions. These combined strategies address the issues of the aforementioned methods and, as evidenced by our experimental results, significantly enhance performance.

4.3 Analyses and Discussions

Ablation Experiments

We conduct ablation studies on each component of the loss function. The results indicate that each component of the loss function contributes to the final performance. The specific results are presented in Figure 3.

Analysis of the Effect on Semantic Margin Contrastive Loss

To fully leverage the class semantic correlations, we freeze the text encoder to preserve the rich semantic correlations learned during the pre-training phase and employ a semantic margin to extract class semantic correlations from textual descriptions. This strategy allows us to utilize the class semantic correlations from textual descriptions, thereby effectively enhancing image representation discriminability and the model's ability to recognize unknown classes. To demonstrate the effectiveness of our strategy, we conduct experiments comparing a basic method that uses the loss function Lbase+Litc and fine-tunes the text encoder (which essentially is TextGCD [Zheng et al., 2024]) with our method, which employs Lbase + Lsmc. The experimental results, presented in Figure 4, show that our strategy significantly outperforms the basic method.

Analysis of the Sensitivity of Hyper-Parameters

We introduce an additional hyper-parameter β into the Lsmc, while all other hyper-parameters remain consistent with the references and are maintained at the same values across all experiments, as detailed in the experimental section. Sensitivity analyses for β are presented in Figure 5. These analyses demonstrate that the performance of our method is robust to the choice of the hyper-parameter β.

Analysis of the Performance Improvement of Semantic Margin for OWSSL Frameworks

In the field of OWSSL, several frameworks are incorporated within the existing methods. For instance, GCD [Vaze et al., 2022] constructs a discriminative representation space through semi-supervised contrastive learning, and SimGCD [Wen et al., 2023] enhances performance via a selfdistillation strategy. Most subsequent methods are improvements based on these two frameworks [Zhao et al., 2023; Pu et al., 2023; Zhang et al., 2023]. However, these frameworks commonly simplify classes into symbolic variables, which can determine the classification of samples but ignore the rich semantic information of the classes. Therefore, TextGCD [Zheng et al., 2024] attempts to address this issue by incorporating textual information and employing an image-text contrastive loss to align visual features with textual descriptions. As a result, it is considered an OWSSL framework. Nevertheless, it still fails to consider the class semantic correlations. Our method keeps the text encoder frozen, thus preserving the rich semantic correlations learned during the pre-training phase, and introduces a semantic margin contrastive loss to utilize class semantic correlations from textual descriptions. These combined strategies maintain the semantic correlations established during pre-training and collectively address the issues of these frameworks. Across multiple experimental metrics, our method demonstrates performance that surpasses the current frameworks, with the comparative experimental results detailed in Figure 6 .

Analysis of Performance in More Realistic Scenarios

Existing OWSSL methods that utilize textual descriptions typically assume that the names of all classes are pre-known. However, this assumption often does not hold in real-world scenarios, as in practical applications, we usually only have access to information about a subset of classes. To address this shortcoming, in this section, we investigate a scenario that is closer to reality: validating the performance of our proposed method using only the textual descriptions of known classes. Furthermore, we do not employ any auxiliary VLMs, which might not be available or practical in real-world settings. All other conditions remain the same. This allows us to more accurately evaluate the generalization capability of our method when dealing with real-world problems. The relevant experimental results are presented in Figure 7. The results show that even when using only the textual descriptions of known classes, our method still demonstrates competitive performance compared to OWSSL methods that use textual descriptions for all classes.

5 Conclusion

In this paper, we propose a novel OWSSL method. This method fine-tunes only the image encoder during training while keeping the text encoder frozen, thereby preserving the rich semantic correlations learned during the pretraining phase. Furthermore, our method introduces a semantic margin to extract class semantic correlations from textual descriptions. These semantic correlations are then utilized to enhance the discriminability of image representations. By focusing on these aspects, our method significantly improves model performance compared to representative OWSSL methods. Future research aims to explore and address the challenges posed by the continuous emergence of unknown classes, a scenario commonly encountered in practical applications, while investigating the capabilities of our method in such environments.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Nos. 62376141, U21A20473, 62376142, 62276158).