Class-Specific Semantic Generation and Reconstruction Learning for Open Set Recognition

1 Introduction

In recent years, artificial intelligence models have achieved human-like performance in numerous settings, most of which are based on a closed assumption. This assumption implies that the model must be provided with knowledge of all classes during the training phase. However, the real world is full of unknowns, and the testing phase may present unknown classes [Yoshihashi et al., 2019]. This is a key issue that has prevented artificial intelligence models from reaching humanlevel performance in areas such as autonomous driving and medical diagnostics. This issue has drawn significant attention from researchers and engineers in diverse fields such as data mining [Lv et al., 2022; Wu et al., 2020; Zhu et al., 2023; Liu et al., 2023a], computer vision [Hu et al., 2024; Gao et al., 2023], and multimedia processing [Zhang et al., 2020; Qian et al., 2023; Cai et al., 2023].

∗Corresponding author.

To address classification and recognition tasks with openness, Open Set Recognition (OSR) defines a realistic task objective, which is to accurately classify samples from known classes and reject samples from unknown classes [Geng et al., 2020]. The main challenge of OSR is that the model has not received any information about unknown classes, which makes it impossible to directly divide the feature space of the unknown classes [Scheirer et al., 2013]. Therefore, many existing works have been proposed to learn compact representations of known classes, which can isolate the unknown space and thus reduce the open space risk.

Prototype learning and reconstruction learning show great power in the representation of known classes. Prototype learning represents each class by learning class-specific prototype points based on the Gaussian distribution assumption [Yang et al., 2020], which encourages known samples to approach prototype points, thus excluding unknown samples. Unfortunately, the assumption can easily be violated in practice, which limits the fitting ability of the method. Compared to the prototype representation, reconstruction learning can learn a more compact feature representation based on the Auto-Encoder (AE). These methods reconstruct the input through a compression-recovery process to learn latent representations of the original features, which can also be viewed as learning a low-dimensional manifold to fit the distribution of known samples [Huang et al., 2023]. Therefore, when the input comes from an unknown sample outside the manifold, the AE outputs a higher reconstruction error, which can be used as a basis for recognition [Oza and Patel, 2019a]. However, representing all known classes with a single continuous manifold is detrimental to the learning of class-specific features and potentially devours inter-class regions, thus introducing the open space risk.

Figure 1: Comparison of reconstruction learning methods. Method (a) have a limited ability to fit and compact representations, and for unknown samples on the margins of the manifolds, the low reconstruction error of known classes is not sufficient to reject them. Method (b) can obtain a compact representation. In addition, it can identify confusable samples that are still located at the margins of the manifolds based on known and unknown reconstruction errors.

Combining the respective advantages of prototype learning and AE is an effective solution for OSR, i.e., modeling each known class by class-specific manifolds. This method incorporates the class-specificity of prototype learning and the compactness of AE. However, the model is based on a closed paradigm during the training phase, so it is only allowed to fit the known classes while completely ignoring the unknown feature space. As illustrated in Figure 1 (a), this gives rise to two problems: (1) without the characterization of the unknown classes, the manifolds of the known classes may still not be completely compact, and (2) it is risky to rely exclusively on the distinction from the known classes to recognize the unknown classes.

In light of the above observations, we explore (1) a way to characterize the unknown feature space and (2) a more reliable method for distinguishing between known and unknown classes. As any samples from the unknown classes are invisible during the training phase, characterizing the global unknown space is unrealistic. As depicted in Figure 1(b), we propose that an alternative solution is to learn the boundaries of known classes to approximately characterize the unknown space. Once the model learns the union boundary region of the manifolds of the known classes, similar to the reconstruction error of the known classes, it can also be used as a basis for identifying the unknown classes. It is easier to find a threshold to distinguish the known space from the unknown space.

In this paper, we propose a Class-specific Semantic Generation and Reconstruction Learning (CSGRL) method that involves a reconstruction module and a generation module. In CSGRL, in order to characterize the unknown space, the generation module is adversarially constrained to generate samples around the boundary of the manifolds of known classes. Then, the reconstruction module fits the known classes with class-specific AEs and fits the unknown space with an additional AE. The joint optimization of all AEs keeps the unknown regions out of known samples and known manifolds out of unknown space, resulting in more compact representations of known classes. Finally, integrating the reconstruction errors of the AEs representing the known classes and the AE representing the unknown classes, CSGRL makes a decision of classification or rejection. Our contributions are the following:

1. We propose a training framework with openness, which learns the union boundary region of known classes by an independent AE to approximately characterize the unknown space. This reduces the open space risk and facilitates the fitting of known classes.
2. We define class-specific generators that can generate slack or tight boundary samples as required, which serve as inputs to augment the reconstruction module.
3. We recognize the testing samples from two perspectives, the reconstruction error of known classes and unknown classes, which effectively improves the recognition accuracy.
4. Experimental results under different protocols demonstrate CSGRL outperforms baseline methods and achieves state-of-the-art performance on several public datasets.

2 RelatedWork

We propose CSGRL to address the OSR problem, which is naturally related to Out-Of-Distribution (OOD) detection, and we will discuss their existing work in this section.

2.1 Open Set Recognition

Initially, solutions for OSR were based on traditional machine learning methods [Scheirer et al., 2014; Mendes J´unior et al., 2017; Zhu et al., 2018]. However, the performance of these methods was limited by the technology available at the time. With the development of deep neural networks, which have achieved excellent performance in a closed setting, more attention has been paid to their application in open scenarios. Deploying deep neural networks to solve OSR problems, prototype-like and reconstruction learning methods are the two feasible research paths.

Prototype-like learning methods. [Yang et al., 2020] presented the first convolutional prototype network for the openworld, which aimed to represent each specific class abstractly. As a variant of the prototype network, reciprocal point learning (RPL) [Chen et al., 2020] utilizes the differences between samples and the reciprocal points rather than their commonalities for classification and recognition. In order to enhance the recognition of unknown classes, RPL further integrated the adversarial training strategy, named ARPL [Chen et al., 2021]. [Liu et al., 2023b] derived a multiple Gaussian prototype framework from Bayesian inference, thus preserving some space for unknown classes. Constrained by Gaussian distribution assumptions, these methods fail to represent realworld complexities.

Reconstruction learning methods. [Yoshihashi et al., 2019] proposed the first Classification Reconstruction learning for Open Set Recognition (CROSR) based on deep representation, which inspired subsequent such methods. C2AE [Oza and Patel, 2019a] was constructed as a two-stage method, where a closed-set classifier is first trained, and then an open set recognition model was trained by reconstructing conditional on class identity. Based on the unknown sample detection capability of the variational AE, [Sun et al., 2020] proposed a conditional Gaussian distribution learning for open set recognition (CGDL), which classifies known samples by forcing latent features to approximate different Gaussian models. To reduce the class activation value of unknown samples and maximize their difference from known samples, [Perera et al., 2020] extended the optimal closed set features and learned both the original and reconstructed images. Further, [Huang et al., 2023] proposed the CSSR model that replaces prototype points with manifolds represented by class-specific AEs, which enhances the representation of the individual classes and releases the inter-class space. However, the above work neglects the characterization of unknown classes, which may impair the fitting ability of the model. In addition, reconstructions available only for known classes are not conducive to finding distinguishable thresholds.

2.2 Out-of-Distribution Detection

A subtask of OSR is to reject unknown classes, which shares the same target as some problems. For reasons of AI system safety, [Hendrycks and Gimpel, 2017] advocated reliable methods for detecting samples in the test set that come from outside the distribution of the training set. Without changing the pre-trained neural network, [Liang et al., 2018] used temperature scaling and added small perturbations to improve the separability of in- and OOD images. [Lee et al., 2018] minimized the Kullback-Leibler divergence from the predictive distribution of out-of-distribution samples to the uniform distribution. To this end, they introduced an OOD dataset for training and augmented it with boundary samples generated from low-density regions of known classes. Outlier exposure [Hendrycks et al., 2019] generalizes to unseen anomalies by feeding the anomaly detectors with auxiliary data. Similarly, outlier detection [Zhou and Paffenroth, 2017] and novelty detection [Perera et al., 2019] have been extensively studied to improve the robustness of models. Most of these works focus on rejecting OOD (unknown) samples without considering the ability of the model to classify in-distribution (known) samples. In addition, some methods introduce OOD data, which violates the OSR protocol. Obviously, an OSR setup without any unknown data is more general and realistic in practice.

3 Preliminaries

In this section, we define the research problem and briefly describe the relevant research components.

Problem definition. Given a training set of n labeled samples X = {(x1, y1), (x2, y2), ..., (xn, yn)}, where yi ∈ {1, 2, ..., k} is the label of xi. In the testing phase, any one sample xt corresponds to the label yt ∈ {1, 2, ..., k, k + 1}, where k + 1 is a uniform label of all unknown classes. The goal of OSR is to learn a model f from X that establishes a one-to-one mapping of xt to yt.

A good OSR model should be able to correctly classify known classes, i.e., minimize the empirical classification risk RE of labeled known data. In addition, unlike traditional classification models, OSR models also need to have the ability to reject unknown classes. As defined in [Scheirer et al., 2013], open space risk measures the uncertainty that f recognizes sample x as known or unknown classes, which is calculated as a nonzero integral function

where O is the open (unknown) space and g is a binary function whose output is

Therefore, the OSR problem can be transformed into an optimization objective as follows:

where λ is a positive regularization parameter.

Class-specific semantic reconstruction learning. Classspecific Semantic Reconstruction (CSSR) learning utilizes a set of AEs to learn class-specific low-dimensional manifolds to fit the distribution of the data, where each AE is a bottleneck structure consisting of an encoder E and decoder D. The encoder E learns a mapping of the original data from the low-dimensional embedding space, and then the decoder D recovers the low-dimensional vectors to the original feature space. In this process, the loss of compression and the difference in recovery need to be reduced as much as possible, i.e., minimizing the reconstruction error. CSSR considers the reconstruction error as the distance between the sample and a low-dimensional manifold, and it assigns the test sample to the nearest manifold. For OSR, samples far from all manifolds are regarded as unknown samples. These concepts can be formalized as follows:

where f = B(x) is the embedding feature of input x extracted by the backbone network B. AE corresponding to class c is denoted as Ac. During training, this method ignores modeling the unknown class and only maximizes the probability of the ground-truth class.

4 The ProposedMethod

In order to address the reported problems, we propose a Class-specific Semantic Generation and Reconstruction Learning (CSGRL) method. Figure 2 illustrates the overall architecture of the proposed method. The key theme is to characterize the unknown space. We first present the overall architecture of the fitting class. Then, a method for characterizing unknown space is introduced. Finally, we describe the classification of known classes and the recognition of unknown classes.

4.1 Learning Class-Specific Manifolds

To make decisions about sample (x, c) ∈ X, we define a reconstruction module R. The module consists of a set of AEs A = {A1,A2, ...,Ak+1}. Different from the existing methods, an additional AE Ak+1 is used to represent the unknown classes. Sample x is output by the backbone network as f , the reconstruction error specific to class i can be obtained by A as d(f ,Ai) = ∥f -Ai(f)∥1 . Similar to Eq. (4), the classification probability of sample x is related to the distance of the manifolds. Applying the SoftMax activation function, the reconstruction error can be normalized to probability as follows:

where γ is a hyperparameter used to control the hardness of the error-probability transformation, and it is set to 0.1.

We hope that the probability that sample x belongs to the ground truth class c should be infinitely close to 1, while the probability that it belongs to the other classes is approximately equal to 0. Therefore, the model optimization is guided by a loss function as follows:

To minimize the loss, the reconstruction module should guarantee a minimum distance mapping from the features to the manifolds of the true class, based on which the distance to other manifolds is maximized.

4.2 Characterizing Unknown Space

Due to there being no unknown samples available for training, the model can only fit known classes. The latent unknown sample space can be infinite and thus impossible to cover. We propose to use Ak+1 to fit the union boundary region. This intuitively approximately characterizes the unknown space, because most unknown samples are always closer to their boundary region than the manifolds of known classes. For this purpose, we need to build a generation module G = {G1,G2, ...,Gk} to generate boundary samples for each known class. As an effective generative model,

Algorithm 1 The training procedure of CSGRL

Input:X: Training data; ε: Hyperparameter.

Output:

• B: Backbone network; R: Reconstruction module;
• G: Generation module.
• 1: Initialize B, R and G. /** Step-I: Update proposed R **/
• 2: repeat
• 3: Update B, R by descending their stochastic gradient according to Eq. (6)
• 4: until convergence; /** Step-II: Alternate update proposed G and R **/
• 5: repeat
• 6: UpdateGby descending its stochastic gradient according to Eq. (9)
• 7: Update B, R by descending their stochastic gradient according to Eq. (10)
• 8: until convergence;
• 9: return B, R and G;

Generative Adversarial Networks (GAN) [Goodfellow et al., 2014] consists of a generator G′ and a binary discriminator D′ . The generator synthesizes pseudo-samples to augment the discriminator, while the discriminator guides the generator to obtain confusable samples. GAN performs a minimax optimization over D′ and G′ as

where the input of G′ is a Gaussian noise ˜x ∼ N(0, 1).

However, we need boundary samples to approximate the unknown space, instead of samples within manifolds of known classes. Obviously, the boundary samples should not be too far away from the known samples, but they should be located near the extreme points. The current maximum point for class c is collected in each mini-batch {(x1, c), (x2, c), ..., (xm, c)} as

where dc is set to 0 at the beginning of each epoch. Generating high-dimensional realistic images is challenging [Pidhorskyi et al., 2018; Zenati et al., 2018], and existing work [Kong and Ramanan, 2021] has demonstrated that generating samples at the feature-level works better than at the pixellevel. Therefore, G directly generates feature-level samples, and we impose constraints on G to obtain class-specific boundary samples

where ε is a slackness, the larger its value, the looser the resulting boundary. Conversely, the tighter the resulting boundary. To minimize the second term of LG, the generator Gc is forced to generate samples with reconstruction errors higher than dc - ε. However, if these samples are far from Ac, the first term of LG should be high. Thus, one can expect that LG can encourage G to generate valid boundary samples for each class.

Finally, input real and generated samples to update the reconstruction module, and Eq. (6) is expanded as

To minimize the loss, LR encourages the separation of known and unknown space.

Due to LG involves the computation of the maximum value of the reconstruction errors for known classes, R is preferentially trained to obtain a stable representation of the known classes. Then, G and D are alternately optimized. The complete training procedure for the proposed method is summarized in Algorithm 1.

4.3 Recognition and Classification

In order to obtain a reliable score to reject unknown classes, CSGRL integrates the reconstruction errors relative to known and unknown classes. OSR methods typically quantify the difference (or similarity) between a test sample and the known samples as a score S, and classify the sample into an unknown class if S exceeds (or is below) than a threshold. Similarly, referring to Eq. 4, we can utilize the reconstruction errors of the known classes to recognize the unknown samples. Known classes should have low relative reconstruction errors, which can be formulated as

For unknown classes that are vastly different from all known classes, e.g., far OOD datasets [Chen et al., 2021], their samples can lead to significant reconstruction errors. Therefore, they can be easily recognized based on Sk. However, there are also unknown samples in the unknown space with small differences from the known classes, e.g., near OOD datasets [Chen et al., 2021]. These data may be located at the margins of the manifolds of known classes, resulting in lower reconstruction errors. In CSGRL, because the Ak+1 learns the union boundary region of known classes, a natural idea is to make inferences with the help of reconstruction errors fromAk+1. This is given by

For a sample located at the margin of known manifolds, although it has a low score |Sk|, it also has a low score |Su|. Therefore, it is probably from an unknown class. A normalization is applied to Sk and Su to unify the scales:

where E(∗) and Std(∗) denote the mean and standard deviation of the scores, respectively. Finally, they are integrated into a final score:

where α is a hyperparameter that controls the weight of the two reconstruction errors on the recognition result. The transformation from "Is it close to known classes?" to "Is it closer to known classes or to their boundaries?" may be more favorable for selecting a threshold in practical scenarios.

A test sample is recognized as an unknown class when Sc is below a threshold δ, otherwise it is assigned to the known class with the highest probability. The process can be formulated as

5 Experiment

5.1 Implementation Details

The proposed method CGSRL does not involve feature extraction, so it can be plugged into any backbone network. To be fair, we use the same backbone network and data augmentation technique with [Huang et al., 2023]. In the training phase, the loss is minimized using the stochastic gradient descent optimizer with momentum = 0.9. We train the network for 250 epochs with a mini-batch size of 128, including 200 epochs for Step-I and 50 epochs for Step-II in Algorithm 1. The learning rate was initially set to 0.4

5.2 Experiments for Unknown Detection

The evaluation protocol defined in [Neal et al., 2018] is widely used for unknown detection.

Datasets. We construct experiments on three image datasets, including Cifar10 [Krizhevsky, 2009], SVHN [Netzer et al., 2011], and Tinyimagenet [Le and Yang, 2015]. For Cifar10 and SVHN, 6 classes are randomly sampled as known classes and the remaining 4 classes are set as unknown classes. For Tinyimagenet with 200 classes, the ratio of known to unknown classes is 20:180. To avoid chance, the splits of known and unknown classes are randomized five times, and then their average results are reported. In this experiment, we directly use the same data split with [Huang et al., 2023].

Evaluation metrics. For most OSR methods, selecting the appropriate threshold is crucial. Therefore, a thresholdindependent metric, the Area Under the Receiver Operating Characteristic (AUROC) curve is often used for evaluation. It reflects the true positive rate against the false positive rate at different thresholds. In our research problem, AUROC can be interpreted as the probability that a known sample is assigned a higher recognition score than an unknown sample.

Baselines. CSGRL was compared to advanced related methods, including prototype-like learning methods [Yang et al., 2020], [Chen et al., 2021], [Liu et al., 2023b] and reconstruction learning methods [Yoshihashi et al., 2019], [Oza and Patel, 2019a], [Oza and Patel, 2019b], [Sun et al., 2020],

[Perera et al., 2020], [Huang et al., 2023]. Among these methods, [Chen et al., 2021] uses GAN for OSR.

Result comparison. Table 1 shows the AUROC results for the thirteen methods on the three benchmark datasets. In all the tables presenting the experimental results, the results are presented as percentages and the best results are in bold. The results show that CSGRL outperforms other methods on all three datasets. This is attributed to the reasonable characterization of the unknown space, thus enabling the model to separate the known and unknown samples more efficiently.

5.3 Experiments for Open Set Recognition

An effective OSR method not only rejects unknown samples, but also categorizes known samples. [Yoshihashi et al., 2019] provided a setup for OSR experiments, and we apply it in this section.

Experimental setup. Cifar10 is used as the training set, and samples from other datasets are collected into the test set. We collect 10,000 unknown samples from ImageNet and LSUN respectively, which is the same number of samples as the Cifar10 test set. Then, we crop or resize them to ensure that their image sizes are consistent with the known samples. In this experiment, we use the version of the four datasets released by Liang et al. [Liang et al., 2018], i.e., ImageNetCrop (IN-C), ImageNet-Resize (IN-R), LSUN-Crop (LS-C), and LSUN-Resize (LS-R). Since AUROC can only evaluate the recognition of unknown classes and not the classification performance of the method on known classes, we use the macro-averaged F1-scores to evaluate the overall performance of the method on 10 known classes and 1 unknown class. Furthermore, except for prototype-like and reconstruction learning methods, other types of OSR methods [Zhou et al., 2021], [Liu et al., 2022], [Xu et al., 2023], [Wang et al., 2024] are introduced for comparison.

Results comparison. As shown in Table 2, CSGRL outperforms all methods on four datasets. This indicates that the proposed method is generally more in line with the goal of OSR, i.e., CSGRL possesses higher classification accuracy for known samples and stronger capability for unknown detection. In addition, CSSR and its reciprocal version RCSSR have mutual advantages in different setups. In order to reject unknown classes, instead of considering whether to represent the known classes by prototype or reciprocal points, CSGRL characterizes both known and unknown classes to separate them directly.

5.4 Experiments for Out-of-Distribution Detection

OOD detection aims to detect samples that come from outdistribution, which may be similar to, or very different from, in-distribution samples. The setup of the OOD experiment in [Chen et al., 2021] is adopted in this section.

Experimental setup. We conduct two sets of experiments where the models are trained on Cifar10 as the in-distribution dataset. Then, they are tested on Cifar100 or SVHN as the near OOD dataset and far OOD dataset, respectively. In addition to the previously introduced methods, we compare CSI [Tack et al., 2020] and OpenGAN [Kong and Ramanan, 2021], which enhance the representation of known classes from different perspectives. Following [Chen et al., 2021], we use DTACC, AUROC, AUIN, and AUOUT as evaluation metrics. DTACC calculates the maximum value of classification accuracy at different thresholds. AUPR measures the area of graph plotting precision against recall at different thresholds. When in-distribution (or out-distribution) samples are specified as positive, AUPR is further represented by AUIN (or AUOUT).

Results comparison. Based on the results in Table 3, we can summarize the following conclusions:

1. Under different OOD settings, CSGRL outperformed OpenGAN on four metrics, which indicates that the conventional GAN has limited improvement in unknown detection.
2. It is easier to recognize far OOD data than near OOD data, so its performance is close to saturation on most methods. Despite this, CSGRL improves performance over other methods.
3. For near OOD data, CSGRL is ranked first. This indicates that CSGRL is better able to distinguish unknown samples that are similar to known samples.

5.5 Ablation Study

In this section, we construct ablation experiments to investigate the effectiveness of two key designs for CSGRL. As in the dataset setup in Section 5.2 and Section 5.4, the experiment includes three datasets commonly used in OSR: Cifar10, SVHN, and Tinyimagenet.

Ablation terms. (1) Generation Module G: the model learns the union boundary region to approximately characterize the unknown space. (2) Reconstruction Score of Unknown Classes Sun: the model introduces the reconstruction error of the unknown classes as the recognition score.

Result comparison. Figure 3 shows the AUROC results of the ablation experiments. From these results, we can obtain the following three observations:

1. Learning a union boundary region benefits more compact representations of known classes.
2. The reconstruction score of unknown classes can effectively improve the recognition accuracy of unknown classes.
3. As more CSGRL components are integrated into the model, the performance of the model gradually improves. This indicates the necessity of the components for OSR.

6 Conclusion

In this paper, we proposed CSGRL, a class-specific semantic generation and reconstruction learning. In contrast to previous reconstruction learning for OSR, this method follows an open paradigm in the training phase, where it learns a union boundary region to approximately characterize the unknown space. In addition, the reconstruction error of the unknown classes is introduced as a decision score, which further separates the confusing unknown samples from the known space. Experimental results under multiple protocols demonstrated that this method outperforms other state-of-the-art methods.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants No. 62376085, 62076085, 62120106008, 62076116 and 62106262, and the Natural Science Foundation of Fujian Province through Grant No. 2021J02049.