Revealing the Two Sides of Data Augmentation: An Asymmetric Distillation-based Win-Win Solution for Open-Set Recognition

1 Introduction

The utilization of data augmentation (DA) strategies in training neural networks have been proven effective in expanding the training dataset [Yang et al., 2022] and have become widespread in many applications [Chen et al., 2021a; Xu et al., 2022; Chen et al., 2023; Hou et al., 2024; Wang et al., 2024]. As the simplest implementation, the base manipulation-based DA is the most common strategy and can be divided into two categories: single-sample-based augmentation (SSA) and multiple-sample-based augmentation (MSA). SSA creates new samples by conducting basic operations on a single sample, including rotation, flipping, blurring, or their combinations [DeVries and Taylor, 2017; Cubuk et al., 2018; Hendrycks et al., 2020]. Meanwhile, MSA further increases the diversity by involving more than one sample to generate the convex combination of them, i.e., cut-and-paste or addition [Zhang et al., 2017; Yun et al., 2019] and hence remarkably boosts the closed-set recognition ability as shown in Figure 1.

∗Corresponding author

Efficient and effective as MSA is, some research found it affects the performance of recognition tasks to some extent. Balestriero et al. [2022] demonstrate that MSA caused a drop in some classes, and Choi et al. [2023] argue that MSA disperses features in similar classes. However, compared with closed-set recognition, open-set recognition (OSR) is actually the biggest victim of this problem because it has no access to open-set data and hence heavily relies on the discriminative feature. As shown in Figure 1, we reveal that the significant improvement of MSA on closed-set recognition sacrifices the performance of OSR, and as closed-set recognition improves, the corresponding decline in OSR becomes more pronounced, dubbed as the two sides of MSA.

To mitigate the degradation of open-set performance caused by MSA, Roady et al. [2020] tempered the outputs of the models in a label-smoothing way to increase the entropies of the model's outputs. Xu et al. [2023] implemented the InfoNCE loss and used MixUp to enlarge the inter-class margins. However, these extra constraints alleviate the dilemma of MSA on OSR while ceiling the improvement on closed-set recognition. We argue that an ideal solution should be winwin for both closed- and open-set samples.

Based on preliminary experiments on the interplay of DA and OSR, we have two key observations: 1) MSA performs worse than SSA on OSR since it would disperse the features; 2) Knowledge distillation benefits OSR but goes back to decline when MSA joins in. Digging deeper into these observations, we found that MSA diminishes the criteria of OSR in two aspects. First, MSA degrades the magnitude of the activation of features and logits, which leads to great uncertainty in selecting unknown samples via the logits threshold. Distillation mitigates this problem somewhat by forcing the student network to mimic the activation magnitude of the teacher network. Secondly, low-discriminative features of MSA samples remain uncertain; merely distillating them still suffers OSR criteria diminution.

Motivated by the above observation and findings, we propose an asymmetric distillation framework, a win-win solution for both close-set and open-set performance. Concretely, in addition to the same MSA samples fed to the teacher and student in symmetric distillation, we introduce extra raw samples to the teacher and exert extra mutual information objectively to enlarge the teacher's benefit. The introduced objective enables the student to focus more on the class-specific features within the mixed samples. Moreover, since some hard mixed samples provide ambiguous semantic information, we filter them out by relaxedly checking the teacher's predictions and assigning them an unknown-like target to encourage the model to decrease its activation for the non-salient features of the known classes. Within this framework, the model can leverage the advantages of MSA on closed-set performance and better discriminate the novels under extra supervision.

The main contributions in this paper are as follows:

• We revealed the two sides of DA leading to the degeneration of OSR and conducted experiments to analyze how the augmented samples undermine the model.
• We introduce an asymmetric distillation framework with a cross-mutual information maximization and a twohot label smoothing to eliminate the effect and further improve the model's open-set performance.
• We perform extensive experiments and prove the effectiveness of our proposed method on various benchmarks.

2 Reveal the Two Sides of MSA

Despite MSA achieves significant improvement in closed-set recognition, we reveal two sides of MSA on closed-set and OSR in this section. We first present two distinct observations concerning DA and OSR in Section 2.1, followed by an indepth analysis expounded in Section 2.2. Finally, we elucidate the mechanisms through which knowledge distillation can alleviate the degradation in OSR performance induced by MSA, and we highlight inherent issues within existing symmetric distillation frameworks in Section 2.3.

2.1 Key Findings fromDA and OSR Interplay

Without loss of generality, we experiment on both SSA and MSA methods on accuracy and Area Under the Receiver Operating Characteristic curve (AUROC) on Tiny-ImageNet dataset. As shown in Table 1, taking the vanilla model as the baseline, we can make the following two observations:

Observation 1) on SSA vs MSA. Both SSA and MSA exhibit efficacy in enhancing the closed-set accuracy of the model, attributed to their capabilities in expanding the dataset. Notably, MSA demonstrates superior performance in this regard, as it effectively enlarges the diversity within the training data. Nevertheless, when evaluating AUROC, SSA modestly enhances performance, while the incorporation of MSA significantly undermines OSR capabilities.

Observation 2) Distillation Benefits OSR. To verify the influence of distillation [Hinton et al., 2015] on MSA, we compare the MSA-sample based distillation with the vanilla distillation framework. Following Wang et al. [2022], we use a non-MSA-trained network as the teacher model. As shown in Table 1, with distillation only, both accuracy and AUROC gain an improvement. Furthermore, the integration of MSA significantly enhances accuracy by a substantial margin. This implies that the dataset expanded through MSA contributes significantly to the augmentation of the model's representational capacity. Notwithstanding the attainment of more expressive and generalized features, MSA persists in yielding a decrement rather than an amelioration in performance on OSR.

2.2 MSADiminishes the Criteria of OSR

Choi et al. [2023] argue that the MixUp-trained model disperses features in closed-set classes. To quantify such a phenomenon, we visualize the discrepancy among all class pairings of the vanilla model and the CutMix-trained model in Figure 2 (a). At first glance, the heatmap of the CutMixtrained model is darker than the vanilla one, which indicates the degradation of the model's activation magnitude and the lower margins among all the classes. Specifically, the drastic decrease of the gap among the similar classes 'k 2 - k 3' and 'k 2 - k 5' in Figure 2 (a) indicates that the model tends to learn an obscure boundary among these classes. In contrast, the degradation of the distinct classes such as 'k 4 - k 5' and 'k 4 - k 3' is slighter. The broken boundaries among the similar classes are vulnerable to the unknowns which have similar features with these classes. For OSR, the darker colors in the intersect regions of the known classes and the unknown classes suggest the model's degeneration of discriminating them from each other. To dig deeper into the observation, we draw a theoretical analysis in the following.

Denoting D = {Dk,Dunk} as all the inputs the model may encounter during deployment, Dtrain = {xi, yi }ni=1 ⊆ Dk represents the training dataset and xi and yi are the image and the corresponding label. Given input image xi, model's C-classes prediction ˆyi can be obtained via ˆyi = softmax(WΦθ(xi)), where Φθ(·) is the feature extractor and

Φθ(xi) ∈ RD . W = [w1, w2, . . . , wC] ∈ RC×D is the linear classification matrix and WΦθ(xi) = [ˆ yi,1, ˆ yi,2, . . . , ˆ yi,C] is the logits. The training is based on the cross-entropy loss LCE :

Vaze et al. [2022] investigated how LCE influences OSR. The model initially embeds all the classes with a similar magnitude and gradually activates more for the known classes by increasing ||WΦθ(x)|| to better distinguish the unknowns. The final maximum logit score is used to provide the openset score in their conclusion. Additionally, the model's wrong predictions during training tend to reduce wk ·Φθ(xi) ∀k ̸= c. We use CutMix as an example to study the impact of MSA:

where M is a mask and λ is sampled from Beta distribution β(α, α). With xm, Eq. 1 can be rewritten as:

where c1 and c2 are the ground-truth classes of xi and xj .

We display the comparison of ||Φθ(x)|| and ||WΦθ(x)|| in Figure 2 (b) to explore how Eq. 3 influences the model's behavior. It is straightforward that the MSA-trained model suffers from a degradation of feature norm which have the direct bearing on the model's criteria of OSR. Consequently, in Figure 2 (b), ||WΦθ(x)|| of the MSA-trained model decreases drastically, thus harming the open-set score. The discrepancy between the known classes and the unknown classes is also reduced as can be seen in Figure 2 (b). Through the above analysis, we conclude that MSA diminishes the criteria of OSR.

2.3 Retrieve the Discrepancy by Distillation

The distillation experiments in Table 1 indicate that KD benefits OSR. However, distillation with CutMix brings a greater improvement on the model's accuracy while impairing the gain of OSR performance. We investigate how the teacher works by analyzing the distillation loss LDistill = DKL(ˆys ||ˆyt ), where the superscripts s and t denote the student and the teacher model. It encourages ˆys to minimize its divergence with ˆyt , which implicitly leads to an alignment of the magnitude of the activation. This can be concluded in Figure 2 (b) by comparing the ||Φθ(x)|| of the KD-trained model and the teacher. In addition, the comparison of ||WΦθ(x)|| between the MSA-trained model and the MSAdistilled model indicates that distillation with MSA helps the model retrieve the decreased discrepancy between the known classes and the unknown classes to a certain degree. However, the MSA is still harmful to the distilled model, which suggests that the vanilla symmetric distillation framework can not mitigate this issue.

To investigate why CutMix influences the benefit of distillation, we calculate the teacher's over-confident predictions (the maximum probability is greater than 95%) and wrong predictions (the predicted class is out of c1 and c2) over 10000 mixed samples on Tiny-ImageNet dataset as shown in Figure 2 (c). The statistical results suggest that the teacher makes amount of unreasonable predictions on MSA samples. For example, the mixtures of the similar classes will easily be over-confidently predicted because of the redundant activation of their similar features. To solve this problem, We regularize the teacher's redundant activation by an asymmetric distillation framework with an extra mutual information supervision and a re-label mechanism in Section 3. And the wrong prediction indicates that the mixed sample does not include the discriminative features so that the model should be encouraged to decrease its activation. To achieve this, we re-label the wrong predicted samples with smoothed twohot labels to make the model put less attention on the classagnostic features within the mixed samples.

3 Method

3.1 Overview

The overall pipeline of the proposed asymmetric distillation framework is outlined in Figure 3. Based on the vanilla symmetric distillation in which the student and the teacher are fed with the same inputs, we introduce extra initial samples xi and xj to the teacher while training with the augmented input xm to perform an asymmetric data flow between the student and the teacher. We utilize the teacher's output of xi and xj to exert the student's output of xm a cross mutual information objective which forces the student to concentrate more on the class-specific features within xm. In addition, for the confusing mixtures that are wrongly predicted by the teacher, we pick them out and re-label them with a smoothed two-hot label to decrease the student's activation of them. Achieving this can make the student less active in the classagnostic features.

3.2 Asymmetric Distillation Framework

The asymmetric distillation framework is specially designed for the training of MSA. In our experiments, the model is randomly trained using either the original or mixed samples, with a probability of 0.5. We control different data flows with different objects for the student and the teacher during distillation to leverage the teacher's prior knowledge.

Training with the Initial Samples. We inherit the advantages of KD for the initial samples by training with the symmetric distillation framework similar to Wang et al. [2022]. The loss is computed by:

Asymmetric Inputs of Training MSA. The unreasonable output of the teacher emphasizes the non-salient features within the mixture. To enable the student to concentrate more on the class-specific features, we propose extra supervision to amplify the teacher's optimization of ||Φθ(x)||. Concretely, we build an asymmetric distillation framework upon the vanilla symmetric distillation framework by introducing the initial samples to the teacher. The additional initial samples offer the mixed samples an extra mutual information maximization objective to amplify the teacher's impact.

Cross Mutual Information. Mutual Information (MI) is a fundamental measurement to quantify the relationship between random variables [Hou et al., 2021; Feng et al., 2023] denoting by I (v1, v2) where v1 and v2 are two random variables. Initially, the primary objective for MSA training can be understood as maximizing I (Φsθ (xm), ym). As the mixed label ym does not reflect the amount of the label information included in xm well, the powerful teacher is introduced to produce the embedding Φtθ (xm) which is considered as a proper and model-friendly target of optimization. The objective of distillation can be abstracted into the maximization of I (Φtθ (xm), Φsθ (xm)) which encourages the student to align its representation with the teacher.

However, since Φtθ (xm) may include class-agnostic information, especially the mixture of similar classes, we argue that merely imitating the teacher's output is not an optimal solution for OSR. An ideal objective encourages the student to maximize the discriminative features while discarding the common ones within the mixed samples. We achieve this by excluding their shared label information to amplify the teacher's impact on the class-specific features. The term I (Φsθ (xm), Φtθ (xm)|yj ) rigorously quantifies the amount of information of the c1-th class shared between Φtθ (xm) and Φsθ (xm) where '|' is an excluding operation. The maximization of this term forces the student to attend more on the characteristic features of the c1-th class in xm. We maximize this term for both of the two classes in xm by a mutual information loss:

Revisiting the terms in Eq. 5, we find that excluding the class-agnostic information of the c1-th class in xm, i.e. (Φtθ (xm)|yj ), can be easily achieved by replacing it with Φtθ (xi) in consideration of xi shares the same pure information of the c1-th class with xm. So we maximize the mutual information of xm, xi and xj in a cross manner and simplify Eq. 5 to a Cross Mutual Information loss:

where λ is determined by Eq. 2 to weight the contribution of xi and xj.

3.3 Sample Verify and Two-Hot Label Smoothing

Eq. 6 enables the model to discard the class-agnostic features in xm. However, in Figure 2 (c), we point out that the teacher makes mistakes for some corner cases, which may be suboptimal to the model's optimization. We revise the teacher's wrong predictions with a smoothed two-hot label to help the model learn more uncertainties within the confusing samples.

Relaxed Sample Verify. For a mixed sample xm, we argue that it contains the non-salient parts of both the c1-th and the c2-th class when the teacher predicts it to the third class. We utilize a relaxed verification that checks the top-2 accuracy of ˆytm to filter the teacher's wrong predictions out and assign them a revised target to optimize.

Two-Hot Label Smoothing. We aim to optimize the wrongly predicted mixed samples to decrease their activation so that the model can discard the non-discriminative features within the mixtures. The cross-entropy loss we discussed above can naturally degrade the activation for the wrong predictions by Eq. 1. In addition, we want the model to learn more uncertainties among the confusing mixtures, so we manually set a revised target for these samples. Concretely, we mix a uniform label ¯y ∈ RC whose elements are 1/C and ym by a ratio of 0.5 to generate the target of the wrongly predicted xm which we name it yu . The loss is computed by:

where 1(·) is an indicator function whose value is 1 when the following expression in the brackets is true and 0 vise versa. This object encourages the model to embed the uncertain samples xm to the origin of the feature space.

The overall loss can be denoted as:

where µ and η are hyper-parameters we set to 1.0.

4 Experiments

4.1 Experimental Settings

Datasets. We evaluate the performance of our model on three benchmarks: the OSR benchmark, semantic shift benchmark, and large-scale benchmark.

• OSR Benchmark: within this benchmark, the method is evaluated on five datasets, including SVHN [Netzer et al., 2011], CIFAR-10 [Krizhevsky et al., 2009], CIFAR+10, CIFAR+50, and Tiny-ImageNet [Le and Yang, 2015]. All settings align with those of AGC.
• Semantic Shift Benchmark: this evaluation protocol includes three datasets: Caltech-UCSD-Birds (CUB)[Wah et al., 2011], Stanford Cars[Krause et al., 2013], and FGVC-Aircraft [Maji et al., 2013]. The presence of specific attributes distinguishes different classes, and the difficulty of recognition is calculated based on the differences in the number of attributes. Consequently, the open-set classes of the FGVC datasets are divided into 'Easy', 'Medium', and 'Hard' levels to denote their similarities with close-set classes.
• Large-Scale Benchmark: Within this protocol, 200 classes from Tiny-ImageNet are used for training. Subsequently, non-overlapping 'Easy' and 'Hard' splits from Imagenet-21k are selected for evaluation, following the approach outlined by Ren et al. [2023].

Evaluation Metrics. In the OSR benchmark, AUROC serves as a threshold-independent metric [Davis and Goadrich, 2006]. It quantifies the probability that a positive example possesses a higher detector score or value compared to a negative example. OSCR is a metric that gauges the trade-off between accuracy and open-set detection rate by adjusting the threshold on the confidence of the predicted class.

Implementation Details. We utilize DIST [Huang et al., 2022a] as the foundational distillation method. The default teacher-student pair consists of ResNet-50 [He et al., 2016] and ResNet-18. The training duration spans 200 epochs, employing a batch size of 32. The initial learning rate is 0.1, subsequently reduced by a factor of 5 at the 60th, 120th, and 160th epoch. The optimization employs the SGD optimizer with a momentum of 0.9, and the weight decay is set to 5e-4.

4.2 Comparison on OSR Benchmark

To assess the recognition capability of our proposed asymmetric distillation framework in open-set scenarios, we compare it not only with traditional state-of-the-art open-set recognition methods (denoted as ARPL [Chen et al., 2021b], RCSSR [Huang et al., 2022a] and AGC [Vaze et al., 2022], etc.) but also with method CMPKD [Wang et al., 2022], which incorporates both MSA and distillation. As shown in Table 2, our method exhibits a significant improvement in open-set performance compared to traditional open-set recognition methods, achieving a gain of 0.4% on CIFAR10 and 2.6% on TinyImageNet. In comparison to method CMPKD, our model substantially enhances the model's open-set performance while maintaining closed-set accuracy, achieving improvements of 4.9% on SVHN and 2.8% on TinyImageNet. This demonstrates that through the use of the asymmetric distillation framework and the constraints of mutual information object engineering, we have indeed succeeded in enhancing the focus of the student model on class-specific features, and finally got a win-win solution for open-set recognition.

4.3 Comparison on Semantic Shift Benchmark

To further explore the discriminative ability of our model for feature extraction, we conduct experiments on the semantic shift benchmark following AGC [2022]. The results, as shown in Table 3, reveal that our method consistently outperforms the AUROC metric of the state-of-the-art method AGC by a margin of 1%∼2% in both 'Easy' and 'Hard' splits while maintaining closed-set accuracy on three fine-grained datasets (CUB, SCars, and FGVC), a slightly declined by less than 1% in the 'Easy' split of the Aircraft, possibly due to the invariable backgrounds (either the sky or the runway) among the dataset. Notably, our model's performance excels in the hybrid scenario with an OSCR metric exceeding 2.7% for the SOTA method. This suggests that, by employing MSA samples through an asymmetric distillation framework, our model can discard class-agnostic representations and focus more on class-specific representations, thereby enhancing recognition performance across various scenarios, even in challenging fine-grained classification scenarios.

4.4 Comparison on Large-Scale Benchmark

To further explore the effectiveness of our method in realworld scenarios, we conducted experiments on a large-scale dataset. Specifically, we trained on TinyImageNet with only 200 classes and tested on ImageNet-21k with 2100 classes. In our evaluation, we compare our method not only with conventional OSR methods, ARPL [Chen et al., 2021b] and AGC [Vaze et al., 2022], but also with additional multimodal methods such as CLIP [Radford et al., 2021], CoOp [Zhou et al., 2022], and A2Pt. The results, as presented in Table 4, showcase the performance of our method in both the openset recognition metric AUROC and the hybrid recognition metric OSCR. Remarkably, our method outperforms conventional methods by approximately 5%, as well as multimodal methods by 1%. This demonstrates that even in complex realworld scenarios, the features learned through the asymmetric distillation framework remain highly discriminative. Importantly, these features are not significantly disturbed by the increase in unknown novel classes, showcasing the ability of our method to stabilize the open-set recognition performance of the model. Additionally, we replace our ResNet-18 backbone with ViT and report the results. Compared to A2Pt and CLIP, the performance on the ImageNet-21k dataset shows the superiority of our method on the ViT backbone. And the comparison with our ResNet-18 proves that using a more powerful backbone model can reap better benefits.

4.5 Comparison on the Light-weight Model

We conduct additional experiments to assess the effectiveness of our model on lightweight networks in Table 5. In comparison to the state-of-the-art (SOTA) method AGC [Vaze et al., 2022] implemented on MobileNet-V2 [Sandler et al., 2018], our proposed asymmetric distillation model demonstrates superior performance. Our model exhibits improvements in both closed-set accuracy (Acc.) with a margin of +3.24% and open-set recognition AUROC with a margin of +2.64% on the TinyImageNet dataset. These results indicate that our method is not constrained by the network parameters and remains effective even in lightweight networks.

4.6 Ablation Study

In Table 6, we conduct an ablation analysis on the TinyImageNet dataset to delve deeper into the effectiveness of different elements of our method. The comparison between ResNet-18 with CutMix (a) highlights the significant positive impact of multiple samples-based augmentations on improving closed-set classification (+3.1%). However, this improvement comes at the expense of a substantial reduction in the model's open-set recognition performance (-1.8%). The introduction of distillation methods partially mitigates the degradation of open-set performance (from 81.0% to 82.5%), but does not lead to improvement, less than 82.8%. However, when our Contrastive Mutual Information (CMI) objective and Smoothed Two-Hot Label method are introduced, the open-set recognition metrics AUROC of the model sequentially increase from 82.5% to 84.8% and 85.3%. Although the closed-set classification metric (Acc.) slightly decreased by 1% point and 1.5% points, respectively, there are still +3% improvements over the original ResNet. This demonstrates that our proposed CMI objective and Smoothed Two-Hot Label method significantly enhance the model's open-set recognition ability. The empirical evidence highlights the effectiveness of our additional supervision methods in facilitating the learning of class-specific features and decreasing the activation of the non-salient features of the known classes.

To validate the robustness of our method to the hyperparameters, we test different combinations of hyperparameters µ and η including 0.5, 1.0, and 2.0. The result in Table 7 fluctuating around 0.5% under different combinations shows that our method is insensitive to hyper-parameter settings. And the optimal result appears when µ and η are set as 1.0.

4.7 Results on Other Tasks

Our proposed method ensures the backbone model extracts features with discrimination and hence further promotes downstream task performance like OSR task. In Table 8, we evaluate our method on uncertainty-related task Outof-Distribution (OoD) to verify our method as a general feature-strengthening tool. We equipped the maximum logit score (MLS) baseline with our method on Cifar-10/Cifar-100 and Cifar-10/Tiny-ImageNet and the improvements show the effectiveness on OoD task.

Furthermore, we focus on the fundamental and practically applicable recognition task. As a win-win solution for closeand open-set tasks, our proposed method can be regarded as an effective feature extractor enhancement strategy. Taking as an example, we verify the representation ability enhancement of our method on medical image analysis (on MedMNIST v2 dataset [Yang et al., 2023]) in Table 9.

5 Conclusion

In this paper, we start by revealing the two sides of the datamix augmentation by investigating how MSA interplays with open-set recognition. Our experiments and visualizations suggest that MSA diminishes the criteria of OSR and leads to confusion among similar classes. Based on the observations of how knowledge distillation works on OSR, we propose a win-win solution that leverages MSA to boost both the close-set and the open-set performance. The outstanding performance of our method conducted on multiple datasets demonstrates the effectiveness of our approach. It also demonstrates the potential of the known classes can help to detect novels.

Acknowledgments

This work is supported in part by the Beijing Natural Science Foundation under No. L221013, the National Natural Science Foundation of China under Grant Nos. 62102162 and 62203184, and the CCF-DiDi GAIA Collaborative Research Funds for Young Scholars.