Diffuse&Refine: Intrinsic Knowledge Generation and Aggregation for Incremental Object Detection

1 Introduction

Humans expect learning models to continually handle new tasks without forgetting old ones. However, models suffer from catastrophic forgetting in incremental learning, facing representation confusion between old and new classes (see Fig. 1(a)). Due to the independent and identical input of new classes [Gurbuz et al., 2024], feature representations of old classes would vary to mis-match the correct ones, causing semantic drift to exacerbate inter-class confusion (see Fig. 1(b)). These problems have sparked Incremental Object Detection(IOD), requiring to acquire knowledge on extracting representations of new classes and maintaining old ones.

∗Corresponding author.

Existing methods address problems with either Knowledge Distillation(KD) or Replay [Li and Hoiem, 2016]. KD stores old-class knowledge by incorporating parametric neurons with low-level semantics, working as a implicit knowledge transfer and requiring further abstraction [Kang et al., 2023]. Meanwhile, Replay covers massive old data with few key exemplars, only memorizing salient part with losing oldclass knowledge details. Despite achieving notable advances, they struggle to simultaneously keep old-class knowledge and update new-class one regarding incremental data stream. We argue their respective drawbacks can be alleviated by introducing knowledge to generate rigid boundary between oldand new-class representations. Such knowledge not only resolves inter-class confusion with explicit representation partition among different classes, but also mitigates semantic drift by maintaining details of old-class knowledge with anchorlike representations corresponding to rigid class boundary.

With the emergence of advanced multi-modal large language models(MLLMs), there has been a further exploration by importing external knowledge [Junsu et al., 2024]. However, by gradually injecting knowledge accompanying with incremental stream, the retrieving knowledge from external databases may not have strong semantic relevance with either old or new classes, progressively misleading into areas that MLLMs originally excelled. Therefore, involving intrinsic knowledge learned from incremental data itself should be emphasized, establishing rigid class boundary to boost performance with respect to catastrophic forgetting.

We thus propose DiffKA, an intrinsic knowledge generation and aggregation method, which gradually establishes rigid class boundary via diffusion to alleviate catastrophic forgetting. Inspired by high-quality reconstruction and strong associative ability [Rombach et al., 2022], we develop a diffusion-based view for generation and aggregation of intrinsic knowledge, representing them as forward and backward process respectively. Therefore, we first simulate forward diffusion to spread information gained from streaming data, generating potential inter-class associations as wide and deep as possible(see Fig. 1(c)). Specifically, forward diffusion integrates representations of old- and new-class prototypes regarding different learning phases and semantic levels, building hierarchical correlations with an organized tree named as Intrinsic Correlation Tree(ICT). Based on ICT storing intrinsic knowledge, backward diffusion is simulated by refining and aggregating knowledge, explicitly establishing rigid class boundaries in semantic space(see Fig. 1(d)). Specifically, extra supervised information generated by forward diffusion, such as labels, correlations and so on, assists class representations to progressively and optimally converge in semantic space as detailed knowledge. By aggregating fine-grained knowledge in different learning phases and different semantic levels, DiffKA stabilizes convinced correlations among both old and new classes for IOD task.

We further enhance the updating of ICT considering dynamic environment of IOD. ICT still requires emphasis on that intrinsic knowledge should focus on cross-phase and cross-level semantic consistency, in case that imbalanced, few or extreme inputs cause disorders of semantic space. So we leverage rearranging semantic relevance in paradigms to keep semantic consistency. Regarding cross-phase inconsistency, we adaptively enhance representation capability of new-class prototypes to adjust forward diffusion, thus decreasing uncertainty of defining class boundary with few or imbalanced input. Regarding cross-level inconsistency, we employ different types of tree structure adjustment in backward diffusion based on variant scenarios, thus avoiding potential conflicts in intrinsic knowledge. With updating promotion, DiffKA ensures rigid class boundary with excellent reliability and generality, even under extreme settings of IOD. Overall, the main contributions are three-fold:

• To avoid catastrophic forgetting in IOD, we propose DiffKA, with intrinsic knowledge generation and ag-
• gregation performed by forward and backward diffusion, building Intrinsic Correlation Tree(ICT) to establish rigid boundary between old and new classes.
• Regarding cross-phase and cross-level inconsistency within ICT, we enhance its adaptively and effectively updating via rearranging semantic relevance in paradigms.
• Experimental results demonstrate DiffKA achieves state-of-the-art performance with significant advantages.

2 RelatedWork

2.1 Incremental Object Detection

To retain knowledge, IOD methods utilize either knowledge distillation [Feng et al., 2022; Kang et al., 2023; Liu et al., 2023a] or replay [Liu et al., 2023a; Liu et al., 2023b; Junsu et al., 2024]. ERD [Feng et al., 2022] performs elastic distillation on the response of classification and regression heads to address class imbalance. Regarding destruction of semantic space as cause of catastrophic forgetting, [Kang et al., 2023] dynamically distills between-class and within-class semantics to prevent forgetting.

For replay-based methods, CL-DETR [Liu et al., 2023a] selectively stores exemplars of old data to memorize previous learned samples. ABR [Liu et al., 2023b] embeds old instances into new backgrounds to reduce shift. SDDGR [Junsu et al., 2024] generates virtual images via controllable generators, eliminating original-data reliance.

2.2 DiffusionModel

Diffusion models [Ho et al., 2020; Nichol and Dhariwal, 2021; Rombach et al., 2022] are famous for impressive generative capability. LDM [Rombach et al., 2022] diffuses in latent space for high-quality outputs. DiffusionDet [Chen et al., 2023] reformulates box prediction as a denoising process, while DFDD [Wu et al., 2023] uses forward diffusion of OOD features and reverse recovery to sharpen feature discrimination. DiffKA leverages diffusion to generate and aggregate intrinsic knowledge on rigid class boundary, expanding diffusion to aid abstraction of knowledge.

3 Methodology

3.1 Task Description

We follow the stricter IOD protocol of [Liu et al., 2023a]. Formally, given a categories set C = {1, 2, . . . ,C}, we represent a dataset as D = {(x, y)}, where x denotes an image sample, and y = {(b, c)} is the annotation set to indicate the bounding boxes b and the set of category labels c ∈ C corresponding to objects within the image. D and C are then divided into M disjoint subsets based on the number of training phases, where D = D1∪· · ·∪DM and C = C1∪· · ·∪CM. For phase m, we focus on samples in Dm, where y only contains annotations for objects of Class Cm and discard the others. In the mth training phase, the model need recognize the objects in Cm with Dm, while retaining the capability to recognize all previously learned categories C1:m-1. Notably, the model can observe all samples of categories in former phases, while samples of specified categories are annotated in the ith phase.

3.2 Overview

As shown in Fig. 2, we present the structure overview of DiffKA. Inspired by the end-to-end detector, i.e., DeformableDETR [Zhu et al., 2021], we first feed images into ResNet-50 backbone network to extract feature maps, which are then fed into encoder for position encoding. After encoding, the detector would predict a set of object queries. Finally, class and bounding box head would predict classes and positions of all objects based on queries. It's noted that detector is pre-trained with old classes, while we perform incremental learning to detect new classes with the proposed intrinsic knowledge diffusing module and enhanced updating scheme.

In intrinsic knowledge diffusing module(see Fig.2(a)), we simulate diffusion to gradually build Intrinsic Correlation Tree(ICT), with forward and backward diffusion generating and aggregating intrinsic knowledge, i.e., inter-class associations in semantic space. During forward diffusion, at Forward Timestep 0, we adopt detector to predict old-class instances Sold = {Si|i = 1, ...,K} with K classes of pre-learned semantic information. Based on Sold, we further extract set of informative K prototypes P0 = {Pi,0 | i = 1, ...,K}, which are fed to build ICT as initialization of its leave nodes. As shown in Fig. 2(b) to illustrate the establishment of ICT, P0 gradually merged to form its N parent nodes Pt = {Pi,t | i = 1, ...,N; t = 1, ..., T - 1} by step-wisely adding noise for simulation of forward diffusion, which explores potential inter-class associations as wide and deep as possible. Meanwhile, we progressively optimize towards non-overlap and geometrically distinct properties of Pt with reasonable prototype merging rules. During merging, we also generate label set Lt = {Li,t | i = 1, ...,N; t = 1, ..., T - 1} for each parent node by containing labels of all leave nodes. It's noted that we use the joint representation of labels other than individual category label, which exhibits the enhanced semantics of multi-label prototypes stored in ICT. Finally, we could abstract all old-class semantic representations as root of ICT PT , aggregating intrinsic knowledge in a progressive optimizing manner.

During backward diffusion, we refine and aggregate knowledge provided by ICT. At Backward Timestep T, we first sample sets of object queries QT from Gaussian noise. Based on QT , we adopt decoders to step-wisely refine QT with the intrinsic semantic knowledge generated in forward diffusion, i.e., additional supervised information referring to L0:t-1 and P0:t-1 stored in ICT, progressively generating denoised and semantic enriched object queries Qt.

Considering incremental streaming, we propose a general updating scheme to deal with four scenarios (see Fig. 2(c)). Facing old-class samples with representation confusion, we would abandon them due to lack of ground-truth labels. Facing old-class samples without confusion, we would update their corresponding old-class prototypes P0 with the inputting samples. Facing new-class samples without confusion, we would build a new prototype Pnew to update ICT as a leaf node for further forward diffusion. Facing new-class samples with confusion to cause catastrophic forgetting as shown in Fig. 1(a), we gradually move semantic representations of new-class samples via backward diffusion, generating rigid class boundary to avoid confusion.

In enhanced updating scheme, we solve cross-phase and cross-level semantic inconsistency with different strategies. Due to rich intrinsic knowledge of old-class learned and stored in previous phases, the detector would generate imbalanced number of new-class and old-class instances to cause semantic disorders. By calculating weight based on newclass object queries Qnew and old-class ones Qold, we adaptively adjust forward diffusion to generate balanced prototype representation Pt. Meanwhile, cross-level semantic inconsistency leads to conflicting confusions between new-class object query Qnew and old-class prototypes Pt-1, Pt generated by decoders at Timesteps t - 1, t. Such confusions result in unconvinced and terribly organized intrinsic knowledge stored in ICT. By determining whether Pt and Pt-1 exists parent-son relationship, we would either update ICT with general updating scheme if existing, or build new-class prototypes Pnew from the conflicting level to higher levels as well as from Timestep t + 1 to T if not.

3.3 Intrinsic Knowledge DiffusionModule

In this subsection, we focus on generating the ICT within Intrinsic Knowledge Diffusion module. Specifically, we employ forward diffusion to extract intrinsic knowledge from the incremental data stream to construct ICT. Within the ICT, the detector utilizes backward diffusion to refine information and establish rigid boundaries among all classes.

Forward Diffusion. At Forward Timestep 0, we feed detector Φ with new samples Dm, to predict all instances of the learned old classes Sold = Φ(Dm). Based on Sold, we extract set of informative K prototypes P0 = {Pi,0 | i = 1, ...,K}, where we define Pi,0 as the mean distribution center of all instances belonging to class i, i.e., Pi,0 = 1 |Si | ∑ Si. As demonstrate by [Kothapalli, 2023], all instances of the same class tend to form cluster-like distribution, leading Pi,0 to be efficient in representing semantic information. Afterwards, P0 are used to build ICT with each prototype initialized as its leaf node.

In Forward Diffusion period t ∈ {0, . . . , T -1}, step-wise Gaussian noise ϵt ∼ N(0, σ2 t ) is added to Pt, progressively blurring semantic information of prototypes over time:

where ¯αt = ∏ti=0 αi controls strength of the added noise. αi = 1 - βi, where βi represents the variable referring to noise variance gradient illustrated in [Ho et al., 2020]. With noise increasing, class prototypes overlap at first, and then the most similar prototypes start to merge. To achieve their nonoverlap and geometrically distinct properties, we propose to optimize Pt with merging rules, which is defined to search two prototypes Pi,t and Pj,t for merging with the minimum Kullback-Leibler Divergence:

To prevent over or insufficient merging, we further define the number of parent nodes Nt = ⌈|C| × ηt ⌉, where ⌈·⌉ donates the rounding up function, and η ∈ [0, 1] is a hyperparameter to control the rate of prototype merging. It's noted that we adopt a higher η, since it result in fewer fusions at each timestep, thus forcing DiffKA to focus on the easily confused classes in forward diffusion.

With P0 iteratively merging to less prototypes, we achieve an organized and hierarchical tree named as ICT, which captures intrinsic knowledge with semantic correlation established during forward diffusion. Similar with other diffusion models, the forward diffusion involves none of learnable parameters.

Backward Diffusion. We refine knowledge provided by ICT iteratively in backward diffusion. At Backward Timestep T, a fixed number of object queries QT are sampled from Gaussian noise. Based on QT , we adopt decoder ϕt to stepwisely refine Qt at Backward Timestep t with the corresponding intrinsic semantic knowledge stored in ICT, i.e., Lt and Pt, and previous Qt+1:

where F is the feature encoding of the inputting image. Based on Qt, we calculate the unified loss function LCID for refinement:

where 1i∈Lt donates that we only compute LCID for object queries if its predicted class exists in set Lt. Unlike [Liu et al., 2023a] which treats background class as negative instances for supervision, we filter instances with labels not existing in Lt, thus emphasizing to retain the missing annotations as well as additional information in incremental environment. Furthermore, we achieve LCID with tree-structured labels Lt rather than individual class labels, ensuring the detector to focus on the annotated classes and simultaneously avoiding to misclassify unannotated objects as background.

Specifically, classification loss Lcls, bounding box prediction lossLbox and prototype constraint loss Lpc are defined as:

where pi(c) denotes one-hot label encoding for class c, ˆ pˆσi (c) represents the confidence of prediction, γ1 and γ2 are hyperparameters, function fIoU() calculates intersection of union for the generated bounding boxes, ˆb ˆ σi and bi represent predicted and annotated bound boxes respectively, ∥·, ·∥1 refers to L1 Normalization to measure Euclidean distance between vectors, and ∥·, ·∥ computes the Kullback-Leibler Divergence of two distributions. Note that we obtain ˆb ˆ σi via binary matching [Carion et al., 2020] to search for the best annotations for object queries.

Specifically, Lcls encourages the detector to make high confidence predictions with multiple class labels for a single object, thus representing inter-class semantics within one object. By designing classification loss with multiple labels in backward diffusion, we facilitate the aggregation of shared intrinsic knowledge corresponding to single object instance. Lbox evaluates the accuracy of bounding box prediction. Lpc refers to the additional supervised information extracted from prototypes P0:t-1, thus keeping refined queries Qt to be consistent with the stored prototypes in ICT.

3.4 Enhanced Updating Scheme

In this subsection, we first introduce general updating scheme of ICT with incremental streaming data. Afterwards, we propose enhanced updating scheme to adaptively and efficiently update ICT, facing cross-phase and cross-level inconsistencies induced by extreme settings of IOD.

General Updating Scheme. As shown in Fig. 2(c), the general updating scheme is designed to deal with four scenarios. Focusing on representation confusion between new and old classes to cause catastrophic forgetting, at Backward Timestep 0 of the 1st training epoch for diffusion, we create a new prototype without the confusing samples to update ICT as a leaf node, maintaining non-overlap and geometrically distinct properties of all nodes.

At Backward Timestep 0 of the 2nd training epoch, we use the updated ICT to supervise the generation of newclass prototype with the confusing samples. Since its theory of Markov Chain ensures to effectively perform neighbor searching [Ho et al., 2020], diffusion owns capability to generate the cross samples on class boundary. Therefore, its generated supervision would gradually pull confusing samples in semantic space, until their detection results fits ground-truth labels in training epoch T , thus generating new-class prototype Pnew with rigid class boundary among old and new classes. In other words, the intrinsic knowledge embedded in ICT guides the way of boundary shifting, gradually moving confusing samples to locate near the boundary with supervised information provided via diffusion.

Enhanced Updating Scheme. Since IOD tasks often involve unstable incremental data streams, DiffKA may encounter extreme inputs with imbalanced class samples across different learning phases, thus causing cross-phase inconsistency in generating intrinsic knowledge during forward diffusion. Meanwhile, semantic information of new-class prototypes may vary greatly at different backward timesteps, leading to cross-level inconsistency. Both inconsistencies disorder the semantic space to exacerbate catastrophic forgetting.

Due to rich intrinsic knowledge of old-class learned and stored in previous phases, the detector would generate imbalanced number of new-class and old-class instances to cause cross-phase inconsistency (see Fig. 3(a)). Since prototype is defined as mean center of instances, few new-class instances rarely have impacts on prototypes stored in ICT Pt, which leads DiffKA to struggle in learning new-class semantic information. We thus rearrange semantic correlations between old and new classes, which weights new-class prototypes to not only boost the efficiency of its knowledge aggregation, but also strengths semantic associations between new-class and old-class prototypes. Given the number of old- and newclass instances, i.e., |Sold| and |Snew|, the total number of old and new class, i.e., |Cold| and |Cold|, we thus weight new-class instances with the old ones to adjust their impacts on forward diffusion:

where ⊕ represents the operation of element-wise sum.

Due to the small number of new-class samples, DiffKA obtains unstable representation in semantical space during backward diffusion, which might cause confusions with different levels of old-class prototypes stored in hierarchical ICT (see Fig. 3(b)). To resolve cross-level inconsistency, we define confusion predictions with two criteria: 1) not background class, where the predicted bounding box should coincide with the ground-truth, 2) the confusion occurs between new-class and old-class predictions with high confidence. At Backward Timestep t, We thus achieve the confused predictions Ut form object queries Qt :

where fh(·) is the classification and regression head mapping Qt to predicted class label ˆ c and bounding box ˆb, function fGIoU() measures the overlap or distance between bounding boxes [Rezatofighi et al., 2019], p(ˆ c) is the prediction confidence of object query for class c, and λ1 and λ2 are hyperparameters.

Afterwards, we detect the conflicting confusion by determining whether Pt and Pt-1 exists parent-son relationship. If exists referring to no conflicting confusion, DiffKA would perform the general updating scheme. If not exists referring to occurring of conflicting confusion, we adaptively adjust ICT to prevent the conflict. More precisely, if a conflict is detected at timestep t, DiffKA would create new prototypes for Pt, which are first connected as children to Pt-1, and then created as nodes at higher semantic levels as well as later backward timesteps t ∈ {t + 1, t + 2, . . . , T}. Therefore, such adjustment would create a new branch in ICT for crosslevel inconsistent classes.

4 Experiments

4.1 Experimental Settings

Datasets and Comparisons. We evaluate DiffKA with the widely-used COCO 2017 dataset [Lin et al., 2014]. For comparisons, we adopt six famous IOD methods, i.e., LwF [Li and Hoiem, 2016], RILOD [Li et al., 2019], iCaRL [Rebuffi et al., 2017], ERD [Feng et al., 2022], SID [Peng et al., 2021], and CL-DETR [Liu et al., 2023a]. Results of comparative methods are from [Liu et al., 2023a] and [Junsu et al., 2024].

Metrics. Following [Feng et al., 2022], we use six standard COCO evaluation metrics, i.e., mean Average Precision (mAP), AP50, AP75, APS, APM, and APL. After the first phase of training, we introduce Forgetting Percentage Points (FPP) metric [Liu et al., 2023a] to measure the decline in mAP for the previously learned classes.

Scenario setup. Following [Liu et al., 2023a], we consider two scenarios, i.e., the two-phase and multi-phase setting. In two-phase setting, the model is trained on A class using A A+B training samples, and the remaining B A+B samples are used to train the model on the new B class during the incremental learning. In experiments, we define two-phase settings, i.e., A = 40,B = 40, and A = 70,B = 10, resulting in total training phases M = 2, the number of classes in the first learning phase |C1| = A, and the number of classes in the second learning phase |C2| = B. In multi-phase setting, the model requires to recognize P +X × Y classes, where the model is trained with P classes in the first learning phase, and then incrementally learns X new classes in each learning phase. Therefore, we could define M = Y + 1, |C1| = P, and |C2| = · · · = |CM| = X. In our experiments, we set 40 + 20 × 2 and 40 + 10 × 4 for multi-phase experiments.

Implementation Details. DiffKA is bulit based on Deformable-DETR [Zhu et al., 2021] with its original settings. In incremental phases, we freeze the coarse Deformable-DETR and initialize a new detector with its parameters to train for 50 epochs in each phase.

4.2 Comparisons with Other Methods

Two-phase Setting. Table 1 presents the experimental results with the two-phase setting, where DiffKA achieves SOTA results on APm with both settings, proving its effectiveness in knowledge generation and aggregation across different learning phases. In 70+10 setting, DiffKA outperforms replay-based methods (e.g., CL-DETR) with no access to oldclass samples, indicating that DiffKA greatly reduces semantical dependence on old-class samples through knowledge generating and storing in ICT. Meanwhile, DiffKA ranks second in APS, where its performance is slightly smaller, i.e., 0.3%, than best performance achieved by CL-DETR. Since DiffKA is built based on Deformable-DETR for knowledge generation, DiffKA is less effective in small object detection.

Multi-phase Setting. Table 2 represents the comparison results with the multi-phase setting, where DiffKA achieves SOTA results with various settings. It's noted that errors accumulates and catastrophic forgetting exacerbates with more phases of learning. However, DiffKA exhibits a more significant improvement, i.e., 1.5 % in mAP, with more challenging 40+10x4 settings, comparing with 0.8 % gained with 40+20x2 setting. Such phenomenon proves the effectiveness of DiffKA in mitigating severe catastrophic forgetting, since intrinsic knowledge within ICT greatly alleviates the forgetting with imbalanced, few, or extreme setting of IOD.

4.3 Ablation Study

Table 3 shows the ablation results of ICT, EUS, and CID Loss. Without EUS and CID loss, ICT is still capable to generate intrinsic knowledge for old-class prototypes, maintaining oldclass knowledge to deal with IOD task. Such phenomenon indicates class prototypes and inter-class relations plays a critical role in mitigating forgetting. when CID Loss is removed, EUS causes a slight performance drop, since EUS fails in distinguishing and processing confused instances without proper supervision. With the combination of EUS and CID, ICT achieves optimized and gradually knowledge updating, which mitigates forgetting and promotes the learning of new-class prototypes.

Parameter Setting. Table 4 presents results of DiffKAwith different parameters η and λ1, λ2. η = 0.9 ensures the best performance of DiffKA with the most reasonable prototype merging rate, forcing DiffKA to focus on the easily confused classes in forward diffusion. Meanwhile, DiffKA is not sensitive to the varying of λ1 and λ2 with high robustness.

Strategy for Class Label Set. In Table 5, we evaluate different strategies for class labels set, where Random and Fixed refer to randomly select or select the first individual label from Lt for supervision respectively, Joint refers to contain multiple labels of all leave node as supervised information. Random strategy fails to capture intrinsic knowledge with a significant performance drop. Fixed strategy achieves worse results than joint strategy, since multiple and joint labels is efficient in describing inter-class associations stored in ICT.

4.4 Qualitative Analysis

In Fig. 4, we show the visualization of semantic space at different forward timesteps, which proves that DiffKA effectively simulates the diffusion process to hierarchically establish inter-class relationships for IOD task. Fig. 5 shows predictions on samples from COCO 2017 dataset. As shown in Fig. 5(a) and (b), DiffKA preserves predictions from previous learning phases, even enhancing the detection confidences of closely related classes in ICT. Several challenging cases containing both old and new-class instances are represented in Fig. 5(c) and (d), where we can find DiffKA accurately predicts the new-class instances (e.g., potted plant and horse), and occasionally struggles with difficult instances, such as blurred or small objects.

5 Conclusion

DiffKA generates and aggregates intrinsic knowledge with forward and backward diffusion, establishing rigid class boundaries and maintaining semantic consistency across training phases and semantic levels.

Acknowledgements

This work was supported in part by the National Key R&D Program of China under Grants 2023YFC3006501 and 2021YFB3900601, the Natural Science Foundation of Jiangsu Province of China under Grant BK20242050, the Major Science and Technology Program of the Ministry of Water Resources of China under Grant SKS2022072, and the Fundamental Research Funds for the Central Universities under Grants B250201042 and B250201046. It was also supported by the High Performance Computing Platform, Hohai University.