Dynamic against Dynamic: An Open-Set Self-Learning Framework

1 Introduction

Traditional supervised classification has achieved great success in past decades, partly due to the closed-set assumption that the training and test samples share the same feature and label spaces [Fang et al., 2021; Zhou, 2022]. However, this

usually know nothing about the incoming test samples, meaning they could potentially come from unknown classes unseen during training, thus drastically weakening the model performance. To meet this challenge, open-set recognition (OSR) is proposed [Scheirer et al., 2013], which aims to construct the models that not only accurately identify known class samples but also effectively reject unknown class samples.

With the unremitting efforts of the researchers, OSR has achieved significant progress [Neal et al., 2018; Oza and Patel, 2019; Chen et al., 2021; Fang et al., 2021; Guo et al., 2021; Kong and Ramanan, 2021; Zhou et al., 2021; Xu et al., 2023]. For example, [Chen et al., 2021] introduced the reciprocal point technique to model the unexploited extraclass space while [Zhou et al., 2021] adopted the placeholder technique reserve space for unknown classes, thus gaining the more accurate decision boundaries. [Xu et al., 2023] recently introduced the popular supervised contrastive learning technique to aim at learning more effective representations for OSR and established a new performance milestone.

Despite the encouraging advance, the ability of existing methods to deal with agnostic unknown classes actually remains limited due to the following issues:

• Static and Fixed Decision Boundaries. Existing methods generally learn statically fixed decision boundaries using known class samples (as shown in Figure 1) for the universal unknown classes in dynamic and open scenarios, which is evidently insufficient as these classes can emerge unexpectedly at any position in the feature space.

• Knowledge Waste of Rejected Test Samples. Existing methods just simply reject unknown class samples without any additional operations to effectively utilize them. In fact, such samples completely can constitute the true instantiated representation of the unknown class space to further enhance the model's performance.

To address these unsolved issues in open-set recognition, this paper proposes a novel dynamic against dynamic idea, i.e., dynamic methods against dynamic changing open-set world. Therefore, an open-set self-learning (OSSL) framework is correspondingly developed, which starts with a good closed-set classifier trained by known classes and then selftrains with the available tested yet commonly deprecated samples for the model adaptation during testing. Specifically, OSSL first leverages the well-trained closed-set classifier to perform the identity inference of all the test samples, where it divides the entire test set into three subsets according to the logit scores of samples obtained by the closed-set classifier. Then these subsets will be submitted to a novel selfmatching module which adaptively implements the classifier update. Afterwards, the updated classifier serves as the new inferrer, repeating the process until the model converges. In summary, our contributions can be highlighted as follows:

• To our best knowledge, it is the first time that the dynamic against dynamic idea, i.e., dynamic methods against dynamic changing open-set world, is introduced in the modeling of the OSR problem, where an open-set self-learning framework is correspondingly developed.
• A novel self-matching module is design for OSSL, which can achieve the adaptation in automatically identifying known class samples while rejecting unknown samples which are further utilized to enhance the discriminability of the model as the true instantiated representation of the unknown class space.
• Extensive experiments verify the effectiveness of our OSSL, establishing new performance milestones respectively in almost all standard and cross-data benchmark datasets.

2 RelatedWork

2.1 Open Set Recognition

The OSR problem is initially formalized in [Scheirer et al., 2013] and had achieved notable advancements in the past decade, where numerous OSR methods have been developed, which can be mainly divided into two categories: discriminative-based and generative-based methods [Geng et al., 2021]. Discriminative-based methods aim to accurately model known classes using various techniques in order to effectively reject unknown classes [Geng et al., 2020; Chen et al., 2020; Chen et al., 2021; Zhou et al., 2021; Huang et al., 2022; Wang et al., 2022; Cevikalp et al., 2023; Xu et al., 2023]. Generative-based methods usually employ adversarial generation techniques to generate pseudounknown class samples, then transforming the original Kclass OSR problem into a K + 1-class closed-set multi-class classification problem [Neal et al., 2018; Perera and Patel, 2021; Kong and Ramanan, 2021; Geng and Chen, 2022].

As mentioned previously, though these methods have shown promising results, their decision boundaries are all static and fixed, which evidently confines the further performance improvement when facing dynamic and open scenarios. This motivates us to propose the dynamic against dynamic idea, constructing dynamic changeable models to face the dynamic open-set world.

2.2 Self-Training

Self-training [Scudder, 1965; Lee, 2013] is a commonly used semi-supervised learning paradigm that allows the model to leverage the abundant unlabeled data to enhance its learning capabilities. The main focus of self-training lies in that limited labeled data often make the learned model unreliable, resulting in significant noise in pseudo-labeled data, which further induces the model to update in the wrong direction. To address this issue, large numbers of methods have been proposed [Zou et al., 2019; Mukherjee and Awadallah, 2020; Xie et al., 2020; Wei et al., 2021; Karisani, 2023; Garg et al., 2023]. For example, [Zou et al., 2019] proposed a confidence regularized self-training framework, treating pseudolabels as continuous latent variables jointly optimized by alternating optimization. Recently, some researchers have attempted to extend self-training to open scenarios, i.e., there are new classes in unlabeled data that do not appear in labeled data [Cao et al., 2022; Zhuang et al., 2024].

Similarly, we also explore the application of self-traininig in open-set recognition. Additionally, we have relatively sufficient labeled known class samples here, which to some extent ensures the reliability of the initial classifier inference.

2.3 Test-Time Adaptation

Test-time adaptation (TTA) aims to improve the model performance through model adaptation with the changing test data distributions. One pipeline of this study is to jointly train a source model via both supervised and self-supervised objectives, then adapting the model through self-supervised objectives during testing [Sun et al., 2020; Liu et al., 2021; Bartler et al., 2022]. The other pipeline is the full test-time adaptation method, which adapts a model with only test data [Schneider et al., 2020; Wang et al., 2020; Zhang et al., 2022; Iwasawa and Matsuo, 2021; Niu et al., 2022]. For example, [Wang et al., 2020] proposed to adapt by test entropy minimization while [Zhang et al., 2022] adopted the prediction consistency maximization over different augmentations.

Our work falls into the latter pipeline. In particular, to our best knowledge, it is the first time to apply the TTA paradigm to the OSR scenario, further enriching the research on TTA.

3 Methodology

3.1 Preliminary

Let Dtr = {(xi, yi)}Ntr i=1 denote the training set, where xi is the i-th sample and yi ∈ Ctr = {1, 2, 3, ...,K} represents the corresponding known class label. Similarly, let Dte = {(xj, yj)}Nte j=1 be the test set, where yi ∈ Cte = {1, 2, 3, .., K, K + 1}. Note that K + 1 demonstrates a group of unknown classes, which may contain more than one class

[Zhou et al., 2021]. Open-set recognition aims to learn a robust classifier that only accurately classify the known classes but also effectively reject unknown classes.

3.2 Open-Set Self-learning Framework

To face the challenge from the universal unknown classes in dynamic and open scenario, we propose the dynamic against dynamic idea and develop an open-set self-learning (OSSL) framework, which consists of two key components: one is a well-trained closed-set classifier as its starting point, and the other is a novel self-matching module. The former aims to ensure the reliability of pseudo labeling of test data to some extent, while the latter aims to achieve the model adaptation on test data. In specific, OSSL first leverages the well-trained closed-set classifier to perform the identity inference of all the test samples, where it divides the entire test set into three parts, namely known-label set, uncertainty set and unknownlabel set according to the logit scores of samples obtained by the closed-set classifier. Then these data sets will be submitted to a novel self-matching module, which contains three parts: a classifier part C ∗ applied to the known-label set, an adversarial matching part D applied to the known-label and uncertainty sets, and a detection part D′ applied to the known-label and unknown-label sets. These parts work collaboratively to facilitate the enhancement and update of the classifier C ∗ . Afterwards, the updated classifier serves as the new inferrer, repeating the process until the model converges. Figure 2 shows the overview of our OSSL framework. Next, we will elaborate on these two key components of OSSL, i.e., the good starting point and the self-matching module.

The Good Starting Point of OSSL

The core of this component is that the inference of initial classifier should be reliable to some extent, especially for unknown classes. Fortunately, the recent study [Vaze et al., 2022] has shown that a well-trained closed-set classifier can achieve this, where they just train a closed-set classifier using various tricks such as data augmentation, label smoothing, etc, achieving at least comparable performance to SOTA baselines [Chen et al., 2021; Xu et al., 2023]. Therefore, we here employ the network architecture in [Vaze et al., 2022] as our model's backbone, the trained closed-set classifier as our initial classifier C0 whose outputs are logit vectors.

Algorithm 1 Training Procedure of OSSL Framework

Input: Data from training set Dtr; Data from testing set Dte; Parameter: Starting point classifier C0(·); Adversarial matching part D(·); Detection part D′ (·);

• 1: Initialize model parameters D(·), D′ (·) .
• 2: for epoch =1 in epochmax do
• 3: DivideDte based on probability scores fromclassifier C∗ (its initial one is C0) into three parts: T1 , T2 and T3.
• 4: Sample data from training set to get Trs ⊂ Dtr.
• 5: Based on Eq.(6) , using data from Trs and known-label set T1 to calculate L′C∗ .
• 6: Obtain data from known-label set T1, unknown-label set T3 and Trs to compute L′D′ based on Eq.(7) .
• 7: Calculate ωs and ωt in the method of Eq.(3) .
• 8: Based on Eq.(8) , utilize calculated ωs and ωt to compute L′D for our adversarial matching part.
• 9: Based Eq.(9) , use data from unknown-label set T3 to calculate LMar .
• 10: Calculate the total loss of our OSSL framework based on Eq.(10).
• 11: Based on the loss function, update parameters of our framework.
• 12: end for

The role of the initial classifier C0 is to provide the first identity inference for the test data, where it divides the test set into three parts according the logit scores of samples obtained by C0, namely, known-label set T1 = {xj|max(C0(xj)) > µ}, unknown-label set T3 = {xj|max(C0(xj)) - min(C0(xj)) < γ}, and the remaining data set, i.e., uncertainty set T2. Note that considering the characteristic of the unknown class logit vectors, which tends to be uniformly distributed, we here adopt the relative differences of logit values rather than their magnitudes. Then these preliminarily inferred samples will be submitted to the self-matching module of OSSL for further processing.

Remark. In fact, all existing OSR approaches can serve as our initial classifiers. This implies that our OSSL is orthogonal to existing methods and can leverage them as starting points to achieve even better performance.

The Self-Matching Module of OSSL

The self-matching module aims to efficiently update the classifier C ∗ to adapt to the changing test distribution. In this module, the known-label set T1 is directly used to update C ∗ , however, the number of its samples may not be sufficient, just about 28% of the batch-size (256) in our framework. Thus, the adversarial matching part D is proposed to fully utilize the uncertainty set T2 to update C ∗ , whose core is a sample-level weighted mechanism, mainly drawing on the idea of distribution alignment in [You et al., 2019]. D aims to maximize the alignment/matching of known-class data distribution in T1 and T2 while automatically detecting unknown class samples in T2. Further, the key to the samplelevel weighted mechanism here is to quantify the similarity between each sample and the known class distribution, which can be achieved by the detection part D′ as its objective is to predict samples from known class distribution as 1 and samples from unknown class distribution as 0. In summary, these three parts work collaboratively to facilitate the enhancement and update of C ∗ . Their specific details will be elaborated in the followings.

For the classifier part, it mainly applies the cross-entropy loss to known-label set to gain the classifier update,

where F(·) denotes the feature extractor network, C ∗ is a linear fully connected layer classifier with only one layer, whose initialization parameters come from C0 and outputs are logit vectors. ̂ y denotes the pseudo-known label obtained by the previous classifier (the initial one is C0).

For the adversarial matching part, it operates on knownlabel and uncertainty sets, defined as follows:

Its core is the sample-level weighted mechanism, i.e., ωs and ωt , which represent the probabilities of corresponding samples belonging to known classes, and their values can be obtained by the following equations,

where H denotes the entropy of a vector, which is normalized by its maximum value (log|Ctr|) so that ωs and ωt are restricted into [0, 1]. |Ctr| represents the number of known classes. d′ quantifies the similarity of x to known classes and can be achieved by the detection part D′ (d′ = D′ (F((x)))), defined as follows:

In summary, the total loss function of the self-matching module can be written as:

Some Enhancement Strategies

To further improve the reliability of model inference, we enhance the model from the following two strategies.

Injecting a Small Amount of Ground-Truth Data. We sample a small amount of labeled data of known classes from the training set, i.e. Trs ⊂ Dtr, and add it to the self-learning process of the model to enhance the reliability of model inference. As a result, the loss functions of the self-matching module respectively become:

Marginal Logit Loss for Unknown Classes. In fact, we usually expect the logits of unknown class samples to tend towards the uniform distribution. However, such uniformlydistributed logits do not necessarily lead to sufficiently small values of them. To address this issue, we here introduce a marginal logit loss for the unknown-label set T3 inspired by [Deng et al., 2022], defined as follows,

where vi,k denotes the element value to the corresponding position of the logit vector. Therefore, the ultimately total loss of our OSSL can be summarized as follows:

The overall training procedure of OSSL is detailed in Algorithm 1.

4 Experiments

4.1 Implementation Details

We employ the network architecture in [Vaze et al., 2022] as the backbone of our learning framework 1 , and choose the Stochastic Gradient Descent (SGD) technique as the optimizer. The same data augmentations in [Vaze et al., 2022], like random horizontal flip, random cropping, etc., are also adopted here. Considering that the feature extractor F(·) is a already well-trained network, we here set its learning rate to 10-4 for TinyImageNet while 10-5 for other datasets. Furthermore, the learning rates of other parts except for F(·) are uniformly set to 0.01. For the threshold parameters used in the partition of test set, we set µ = 0.3, γ = 0.03 for TinyImageNet, µ = 0.5, γ = 0.03 for Cifar10, Cifar+10, Cifar+50, while µ = 0.8, γ = 0.02 for MNIST and SVHN. In addition, the number of samples from Trs in each batch is set to 16 (the batch-size is 256), while the hyper-parameter λ in LMar is set to 2 for all benchmark datasets.

4.2 Experiments in Standard-Dataset Benchmarks Datastes

We here follow the protocol defined in [Neal et al., 2018], and provide six standard OSR benchmarks:

• MNIST, SVHN, Cifar10. For MNIST [Lake et al., 2015], SVHN [Netzer et al., 2011], and CIFAR10 [Krizhevsky, 2009], each dataset comprises 10 distinct classes, in which 6 classes are randomly selected as known classes and the other 4 classes as unknown.
• Cifar+10, Cifar+50. In this series of experiments, we select 4 classes from the CIFAR10 dataset to serve as known classes for training and 10 or 50 classes from the CIFAR100 dataset as unknown classes in specific experiment.

1 https://github.com/ChuanxingGeng/OSSL

• TinyImageNet. TinyImageNet, a derived subset of the larger ImageNet [Russakovsky et al., 2014] dataset, encompasses 200 classes, where 20 classes are selected as known classes, while the remaining 180 classes as unknown classes.

Evaluation Metrics

Following the mainstream works [Geng et al., 2021], the area under ROC curve (AUROC) is used to evaluate the model's ability to detect unknown classes shown in Table 1, while the accuracy (ACC) is used to evaluate the performance of closed-set classification detailed in the supplementary materials.

Results Comparison

We compare OSSL with 14 classical and leading OSR methods including Softmax, OpenMax [Bendale and Boult, 2015], OSRCI [Neal et al., 2018], CROSR [Yoshihashi et al., 2018], C2AE [Oza and Patel, 2019], OpenHybrid [Zhang et al., 2020], PROSER [Zhou et al., 2021], EGT [Perera and Patel, 2021], ARPL [Chen et al., 2021], DIAS [Moon et al., 2022], PMAL [Lu et al., 2022] , All U NEED [Vaze et al., 2022] and [Xu et al., 2023] ConOSR.

Table 1 reports the AUROC results. Thanks to the selfmatching module in testing endowing the model with the adaptability to the changing data distributions, our OSSL establishes new performance milestone on almost all benchmark datasets. In particular, in some relatively difficult classification tasks, like TinyImageNet, it significantly outperforms the current two SOTA baselines, i.e., ALL U NEED [Vaze et al., 2022] and ConOSR [Xu et al., 2023], respectively achieving performance improvements of 1.2 and 3.3 percentage points.

To further demonstrate the effectiveness of our learning framework, we also employ a popular OSR method, i.e., ARPL [Chen et al., 2021] as the starting point of our OSSL (APRL+Ours). As shown in Table 1, APRL+Ours achieve performance gains across all benchmark datasets. For instance, it achieves an average improvement of 0.35% on MNIST and SVHN whose performance is almost saturated for APRL, while an average improvement of about 1.1% on the remaining benchmark datasets. This also implies that our OSSL is orthogonal to existing methods and can assist them in breaking through existing limitations, achieving further performance improvement.

4.3 Experiments in Cross-Dataset Benchmarks

In this series of experiments, we adhere to the established protocol in [Yoshihashi et al., 2018] to execute our experiments. Within this protocol, all the classes designated for training in the original dataset are employed as in-distribution (ID) data. Meanwhile, samples from an auxiliary dataset are incorporated into the test set, serving as out-of-distribution (OOD) data.

Datasets

• ID: MNIST; OOD: Omniglot, MNIST-Noise, Noise. MNIST is selected as ID data, while samples from Omniglot, MNIST-Noise, Noise are selected as OOD data. In specific, the Omniglot dataset consists of a diverse collection of hand-written characters from various alphabets. The Noise dataset is synthesized with each image generated by sampling pixel values from a uniform distribution in the range [0, 1]. Additionally, the MNIST-Noise dataset is created by superimposing the test images from the MNIST dataset onto the Noise images. The OOD data is quantified at 10,000 instances, which equals to the number of test samples in MNIST.
• ID: Cifar10; OOD: ImageNet, LSUN. Similarly, Cifar10 here is selected as ID data, while samples from ImageNet and LSUN are selected as OOD data. Since the image size of ImageNet and LSUN [Yu et al., 2015] is different from Cifar10, two different ways, i.e., crop

and resize operations are used to adjust their image sizes to match those in Cifar10. In addition, the numbers of test samples for Cifar10, ImageNet and LSUN are all 10, 000.

Evaluation Metrics

The performance evaluation is conducted using the macroaveraged F1-score (%), which provides a comprehensive measure of the model's effectiveness across the diverse class types.

Results Comparison

We compare our OSSL with 7 classical and leading methods. Table 2 and Table 3 respectively report the results on the MNIST and Cifar10 tasks. Compared with the SOTA baseline, i.e., ConOSR [Xu et al., 2023], our OSSL creates new performance milestones with significant advantages on all cross-dataset benchmarks. In particular, for the relative challenging Cifar10 task, it beats ConOSR with an average performance gain of about 6.6%, while an average performance gain of about 1.2% for the MNIST task. This once again proves the effectiveness of such an dynamic changeable learning framework for changing data distributions.

4.4 Effectiveness of Different Enhancements

Injecting a Small Amount of Ground-Truth Data

To verify the effectiveness of injecting a small amount of labeled samples, we here conduct the experiments on Cifar10 and TinyImageNet via changing the number of labeled samples from the training set in each batch (the batch size is 256 in all experiments here) during test-time training. Figure 3(a) shows the results. Our first observation is that whether injecting a small number of labeled samples to the Cifar10 task or the TinyImageNet task, the corresponding performances are significantly improved with about 0.4% gains for Cifar10 while about 0.35% gains for TinyImageNet. This verifies that adding a small number of labeled samples can improve the inference performance of the model. Our second observation is that as the number of labeled samples increases, there is no obvious trend of performance growth, which can be mainly attributed to the fact that the initial classifier has been welltrained on these samples. Please note that this may not necessarily be a bad thing. In fact, it provides evidence (at least to some extent) in such a scenario where we only obtain a well-trained black-box model and a small amount of authorized training samples, we can still train a model that performs comparably to models trained with large numbers of authorized training samples.

Marginal Logit Loss for Unknown Classes

To verify the effectiveness of the introduced marginal logit loss for unknown classes, we also conduct the relevant experiments. Due to the limited space, we here only present the results on Cifar10, Cifar+50, and TinyImageNet (Figure 3(b)), while the results on other datasets can be found in the supplementary materials. As shown in Figure 3(b), the introduce of LMar enables the model to achieve consistent performance improvement on all benchmark datasets.

4.5 Threshold Parameter Sensitivity Analysis

In this section, we undertake a specific sensitivity analysis focusing on the threshold parameter associated with the partition of the test set. We have two threshold parameters, i.e., γ and µ. Limited by the space, we here take the experiments on Cifar10 and TinyImageNet as the examples. In specific, For the parameter γ, we change its value in the range of {0.01, 0.02, 0.03, 0.04, 0.05} both for Cifar10 and TinyImageNet. For the parameter µ, we change its value in the range of {0.3, 0.4, 0.5, 0.6, 0.7} for Cifar10 while {0.1, 0.2, 0.3, 0.4, 0.5} for TinyImageNet. Figure 4 shows the results.

As shown in Figure 4, our model is relatively insensitive to the choice of γ and µ whether on Cifar10 or TinyImageNet. On one hand, it indicates that the selected closed-set classifier in our OSSL has already possessed a significant level of inference capability to some extent. On the other hand, it also provides evidence that we can select an effective threshold without the need for excessive parameter searching. Still, careful selection of these parameters would result in better performance.

4.6 Comparison with Self-Supervised Learning

To further demonstrate the advantages of our learning framework, we here also compare our OSSL with the selfsupervised learning method which can use the unlabeled test data as well. In specific, two self-supervised learning methods for utilizing unlabeled test data are provided. One adopts a joint training way (Self-sup1 ), where the cross-entropy (CE) loss is applied to the labeled training set, and the labeled training set is combined with the unlabeled test set to form a unified unlabeled dataset that is fed into the self-supervised loss (the contrastive loss here). Then these two losses work together to guide the learning of the network. The other one employs a two-stage training way (Self-sup2 ). In the first stage, the unlabeled dataset composed of the training and test sets is fed into the self-supervised loss to train the network. In the second stage, building upon the model trained in the first stage, the entire network is fine-tuned using the labeled training set with the cross-entropy loss. For the sake of fairness, all these methods, like ours, employ the network architecture in [Vaze et al., 2022]. Limited by the space, we here just provide the results in Cifar10, Cifar+50, TinyImageNet, while for the results on the remaining datasets, please refer to the supplementary materials.

As shown in Figure 5, though the performances of these two self-supervised methods have significantly improved compared to the baseline method, i.e., Softmax, on all benchmark datasets, there is still a significant gap between their performance and our OSSL. We believe that this is mainly attributed to the following reasons: i) There may be some conflicts between CE loss and self-supervised loss during training, thus interfering the learning of the network; ii) The selfsupervised proxy tasks may mismatch with the downstream classification task to some extent; iii) The learned features may be too generalized, which is not conducive to the detection of unknown classes.

4.7 Experiment on Frozen Feature Extractor

Despite significant performance breakthroughs achieved by our OSSL, it is worth noting that its training process still requires fine-tuning of the feature extraction network F(·), which can significantly decrease the model's efficiency. One direct way to alleviate this issue is to freeze F(·). To evaluate the performance of OSSL in this situation, we conduct the related experiments and Figure 6 reports the results. We can find that the OSSL with frozen F(·) achieves comparable performance compared to the unfrozen counterpart on all benchmarks. This indicates that in the pursuit of efficiency but with less stringent performance requirements, the model can be trained by directly freezing the feature extractor welltrained on known class data.

5 Conclusion

In this paper, we rethink the OSR problem from the dynamic against dynamic perspective, where an open-set self-learning framework is proposed to construct the dynamic and changeable decision boundaries to adapt changing data distributions rather than a static and fixed ones in existing methods. Extensive experiments verify the effectiveness of our framework, refreshing the new performance records on almost all standard and cross-dataset benchmarks.

Acknowledgments

This research was supported in part by the National Natural Science Foundation of China (62106102, 62076124, 62376126), in part by the Natural Science Foundation of Jiangsu Province (BK20210292), in part by the Hong Kong Scholars Program under Grant XJ2023035, in part by the RGC-GRF project RGC/HKBU12200820.

Contribution Statement

Haifeng Yang and Chuanxing Geng are Equal Contributions. Songcan Chen is the Corresponding Author.