Sharp-Wave Ripples Learning: A Bio-Inspired Incremental Learning Method

1 Introduction

Over the past decades, Deep Neural Networks (DNNs) have become fundamental technologies across various research domains such as autonomous driving, biomedical engineering [Notin et al., 2024] and social simulation [Gao et al., 2024a]. Despite existing remarkable achievements, DNNs currently exhibit limited life-long learning capabilities in incremental learning scenarios [Kudithipudi et al., 2022].

Specifically, training datasets are presented to DNNs sequentially in the incremental learning settings, demanding that DNNs dynamically adapt to non-stationary data streams while retaining previously acquired knowledge [Wang et al., 2024]. However, existing commonly used optimization methods typically update synaptic weights greedily based on newly available data, resulting in severe performance degradation on prior tasks known as catastrophic forgetting (CF) [Wang et al., 2024].

To tackle with the catastrophic forgetting problem, series of incremental learning methods have been proposed, including replay-based methods [Rebuffi et al., 2017], which store and subsequently replay subsets of historical data during new learning sessions; architecture-based methods, which design specific architectures to minimize the interference between new and previously learned knowledge [Wang et al., 2022b]; and optimization-based methods, which regulate the magnitude [Kao et al., 2021] or direction [Zhang et al., 2023] of weight updates. Nevertheless, when limiting the accessibility of old datasets on account of the learning efficiency, memory costs, privacy policies or other factors [Kudithipudi et al., 2022], existing approaches struggle to balance the memory stability and learning plasticity of DNNs.

In contrast, the complementary learning system formed by the neocortex and hippocampus [Kudithipudi et al., 2022] endows the brain with a superior capacity to learn continuously without forgetting. Recent advances in neuroscience reveal that memory integration and consolidation, within this system, are closely associated with hippocampal sharp-wave ripples [Norman et al., 2019]. Specifically, after a learning episode concludes and the brain enters a wakeful rest state without external stimuli, bursts of hippocampal SWRs trigger spontaneous memory reactivations and synchronize the reinstatement of cortical neural circuits. During this process, neural circuits encoding long-term memories are consolidated, while new circuits are formed to integrate newly acquired knowledge from hippocampus.

Inspired by this biological mechanism, we propose an incremental learning method called Sharp-Wave Ripples Learning (SWRL) as illustrated in Figure 2. SWRL comprises two functionally distinct yet complementary modules: a neocortex-like model for storing long-term memory and a hippocampus-like model for rapidly acquiring new knowledge. The SWRL framework is designed to mimic the complementary interaction observed between the neocortex and hippocampus, involving two main stages: a fast learning stage and a memory integration stage. In the fast learning stage, the neocortex-like model is optimized to reduce the fusion gap between newly acquired and previously learned knowledge, serving as a "preview" to facilitate subsequent memory integration, while the hippocampus-like model is trained to fully learn new knowledge without concern of forgetting. Subsequently, during the memory integration stage, SWRL generates SWR-like signals to reactivate memoryassociated neural circuits, and updates the synaptic connections of the neocortex-like model to integrate newly learned knowledge in the hippocampus-like model, employing a novel meta-learning strategy. In contrast to rehearsal exemplars of existing replay-based methods, the SWR-like signals are generated internally by the neural network itself. The contributions are summarized as follows:

• We propose a brain-inspired incremental learning method named SWRL, which simulates the collaborative mechanism between the hippocampus and neocortex. It achieves superior incremental learning performance without accessing prior datasets or modifying network architectures.
• We design a self-triggered meta-weighted memory integration algorithm within SWRL, that utilizes internally generated SWR-like signals to update neuron weights in a meta-weighted manner, effectively balancing memory stability and learning plasticity.
• SWRL is applicable to various incremental learning scenarios across diverse DNN architectures.

2 RelatedWork

2.1 Replay-based Incremental Learning

Replay-based methods alleviate CF by replaying datasets associated with previously learned knowledge during incremental learning [Rebuffi et al., 2017; Chen et al., 2024]. To enhance replay efficiency under constrained memory budgets, recent approaches leverage heuristic informations such as saliency information [Bellitto et al., 2024] to select more representative rehearsal exemplars. Instead of storing raw training datasets, generative replay methods train generative models to synthesize pseudo-exemplars for rehearsal [Maracani et al., 2021]. Advanced generative models, such as diffusion models [Kim et al., 2024], have been employed to generate higher-fidelity rehearsal exemplars. Another complementary direction involves exploiting large-scale natural images to regularize representation drift via knowledge distillation [Feng et al., 2023] or gradient de-conflicting strategies.

2.2 Optimization-based Incremental Learning

Optimization-based methods mainly include weight regularization, gradient projection, and sharpness-aware approaches. Weight regularization approaches stabilize old knowledge by explicitly restricting changes to weights that are important to previous tasks [Kirkpatrick et al., 2017]. The importance of weights can be estimated using criteria such as gradient statistics or Fisher information [Kao et al., 2021]. In contrast, gradient projection approaches stabilize memory by reprojecting gradients away from directions that harm previously learned knowledge [Lin et al., 2022]. For instance, null-space gradient projection methods [Wang et al., 2021] project gradients into the null space of previous feature space, effectively reducing hidden feature shifts. To generate better gradient subspaces that balance memory stability and learning plasticity of DNNs, recent methods have been developed to adopt adaptive orthogonal subspaces [Apolinario et al., 2025] or synaptic meta-plasticity [Xie et al., 2025]. Finally, sharpness-aware approaches aim to enhance the stability of DNNs by finding optimization solutions with flatter loss landscapes [Bian et al., 2024], thereby explicitly reducing model sensitivity to weight perturbations.

2.3 Architecture-based Incremental Learning

Architecture-based methods require modifications to the network structure or depend on specific architectures. Early works, such as XdG [Masse et al., 2018], introduced gating mechanisms to separate the networks into task-specific sub-networks using task embeddings, effectively alleviating inter-task interferences. Recently, with the rise of foundation models, MoE-style adapters have been widely employed to enhance the incremental learning performance of large language models or vision-language models [Yu et al., 2024]. In contrast, model expansion approaches like NI-SSCL [Xie et al., 2025], are proposed to learn both task-specific and task-shared modules through continuous genesis of model weights. Finally, prompt tuning approaches have become increasingly popular for transformer-based models. Instead of updating all backbone parameters, these approaches learn a small set of task-related prompt parameters to steer the frozen backbone for acquiring new knowledge [Wang et al., 2022b]. Recent works have further improved prompt learning from different perspectives, such as tuning prompts along directions orthogonal to previously learned feature subspaces [Lu et al., 2024], or combing prompt learning with replay-based methods [Ke et al., 2025].

3 Method

In incremental learning, the training process of a DNNmodel is split into several sequential stages, where datasets in each stage contain specific learning objectives, such as adapting to new domains or acquiring task-specific skills. The proposed SWRL enables DNNs to learn without forgetting by mimicking the SWR-driven episodic replay phenomenon in the brain. As shown in Figure 2, SWRL mainly contains two stage, i.e., the fast learning stage and memory integration stage. Notably, it does not require access to prior datasets nor modifications to the model architecture.

3.1 Preliminary

Defining the sequential datasets as {D1, ...,Dt+1} and the DNN model weights as Θ, the objective and optimization process at the t +1-th incremental learning stage are formulated as:

where the model is supposed to achieve well performance both on the new dataset Dt+1 and previous datasets after training, yet the optimization process can only access Dt+1 as follows:

3.2 Fast Learning of Neocortex-like Model and Hippocampus-like Model

During the fast learning stage, a neocortex-like model Mnt and a hippocampus-like modelMht are trained independently on the new dataset Dt+1. Mnt is trained to establish a more favorable neural circuitry foundation, aiming to reduce the integration gap between new and prior knowledge while prioritizing the maintenance of memory stability. In contrast, Mht is dedicated to acquiring new knowledge with a focus on learning plasticity.

Stability-oriented Preview Process of Mnt

Given a neocortex-like model Mnt that have been trained on the previous t datasets, we employ a modified weighted null space projector [Xie et al., 2025] to optimize Mnt . This stability-oriented preview process aims to slightly enhance Mnt on Dt+1 while mitigating the forgetting of previously learned knowledge.

Consider a certain layer in Mnt whose basic operator is a linear transformation, e.g., convolution layer and linear projection layer. We denote the weight matrix of this layer as θn t→t+1 ∈ Rm×n , initialized by θn t . The update rule for it is formulated as:

where Pnull is projection matrix designed to constrain the update, allowing learning on Dt+1 while minimizing the change of prediction loss Lt on previous datasets Dt. The calculation of Pnull is detailed below.

Let the output of this layer as Zt on Dt, which can be calculated as:

where Xt ∈ Rn×s is the input feature for this layer corresponding to Dt, and s denotes the number of spatial elements in the feature map. Then, the change in Lt, due to a weight update ∆θ, can be approximated via a first-order Taylor expansion:

substituting ∆Zt = ∆θXt yields:

Let M = Xt(∇Zt Lt)T , and performing singular value decomposition on M gives M = UΣV T . To ensure ∆Lt ≈ 0, ∆θ should be constrained to the null space of M T as expressed in Eq.(3). Then the corresponding projection matrix Pnull is given as follows:

where ̂ U ∈ Rn×k is formed by the last k left singular vectors {Un-k+1, ...,Un} of U, which correspond to the smallest k singular values in Σ. This yields a new neocortex-like model M n t→t+1 with weights θn t→t+1 , which establishes a more favorable neuron circuitry foundation for memory integration.

Plasticity-oriented Learning of Mht

During this process, SWRL initializes the weights of the hippocampus-like model Mht with those of the neocortexlike model Mnt . It then employs common optimizers, such as SGD or Adam, to trainMht on the new datasets Dt+1. The primary focus of SWRL in this plasticity-oriented learning stage is to update the network weights to minimize its loss on Dt+1. Taking the weights θh of a certain layer as an example, this plasticity-oriented learning process is as follows:

yielding a new hippocampus-like model Mht+1 with weights θh t+1 which encodes more comprehensive new knowledge.

3.3 Meta-weightedMemory Integration Guided by SWR-like Signals

The fast learning stage yields a new neocortex-like model M n t→t+1 and a hippocampus-like model Mht+1 . To integrate the knowledge of Mht+1 into Mnt→t+1 , SWRL employs a novel meta-weighted memory integration algorithm, guided by SWR-like signals, to updateMnt→t+1 .

Mathematical Formulation of Memory Integration

For brief illustration, we take a certain layer as an example to introduce the memory integration. The integration process for the weight matrix θ can be initially formulated as:

where ⊙ denotes the Hadamard product, and 1n represents an n-dimensional all-ones column vector. The proposed SWRL will adaptively access the importance (π) of neurons to different memory, and accordingly adjust the update policy for each neuron by optimizing π for balancing learning plasticity and memory stability.

Meta-weighted Memory Integration by SWR-like Signals

Neuron circuit reactivations in the neocortex are closely coordinated with hippocampal sharp-wave ripples (SWRs) [Norman et al., 2019]. This cortico-hippocampal dialogue, is believed to be crucial for integrating newly acquired information from the hippocampus into long-term storage within the neocortex [Kudithipudi et al., 2022]. Inspired by this mechanism, we design a meta-weighted integration policy guided by SWR-like signals.

Specifically, SWR-like signals are tensors generated by the DNN models themselves, which can reactivate the models to reproduce the neuron activation patterns associated with specific memory of knowledge without accessing the original training datasets. We denote the signals that reactivate neuron circuits encoding previous knowledge as Xn , and those for newly learned knowledge as Xh . Details of generating SWR-like signals are provided in the next subsection.

As shown in Eq.(10), SWRL optimizes π to preserve the activation patterns of Mnt→t+1 on Xn , while synchronizing its activation patterns with Mht+1 on Xh . The optimization of π is a bi-level meta learning process, consisting of an inner optimization (InnerOpt) and an outer optimization (OuterOpt), described as follows:

︸

︷︷

OuterOpt

︸

where Csyn(a, b) measures the alignment of activation patterns using cosine similarity. SWRL adaptively evaluates the sensitivity of different knowledge to changes in neuron

(b)

Figure 3: Generating SWR-like signals using neuron activationconsistency prior. (a) Signal generation. (b) Model reactivation. Each circle represents a real neuron in Resnet-18 trained on CIFAR100, and the darker colors indicate higher levels of activation.

weights, and accordingly adjust the fusion weights for each neuron. The fusion results are given by:

Generating SWR-like Signals Using Activation Consistency

The SWR-like signals {Xn ,Xh } are applied to the evaluation of forgetting and synchronization level during memory integration. The proposed SWRL optimizes randomly initialized tensors X to obtain such signals, each associated with specific knowledge. Taking the generation of Xh for Mht+1 as an example, and assuming that produce SWR-like signals that reactivate the model for recalling newly learned knowledge about "what is yt+1?", then this is formulated as:

where Mh,i t+1 (X) denotes the activation of the i-th hidden (convolutional or linear projection) layer given input X, and {µi} represents the recorded historical activation levels of this layer. As shown in Figure 3, to generate signals that reactivate the circuits encoding knowledge of a "Gold fish" in ResNet-18, µi is the running mean of the corresponding layer, which can be obtained from its cascaded batch normalization layer. Here, yt+1 corresponds to the prediction logit vector in which the entry associated with "Gold fish" has the highest value. As shown in Figure 3(b), the activation patterns elicited by the generated signals closely resemble those produced by the original relevant inputs, demonstrating successful knowledge recall.

4 Experiments

We compare the proposed SWRL with state-of-the-art methods under three typical incremental learning settings, includ-

Algorithm 1 SWRL

Require: Initial neocortex-like model:Mn1 , training dataset:{Dt}t≤T . The number of incremental learning stage is T .

Ensure: neocortex-like model:MnT

• 1: Generating SWR-like signals of learned knowledge according to Eq. (12): (Xn 1 )
• 2: for t ∈ [2, T] do
• 3: ▼ Fast learning stage
• 4: Stability-oriented preview process of neocortex-like model Mnt-1 :Mnt-1 Eq.(3) =⇒ Mnt-1→t
• 5: Initializing hippocampus-like modelMht byMnt-1→t
• 6: Plasticity-oriented learning of hippocampus-like modelMht :Mht Eq.(8) =⇒ Mht
• 7: ▼Memory integration stage
• 8: Generating SWR-like signals on datasets Dt by Eq.(12)
• 9: Meta-weighted memory integration guided by SWRlike signals by Eq.(10): π∗
• 10: Calculating new neocortex-like modelMnt by Eq.(11)
• 11: end for

ing task incremental learning, class incremental learning, and domain incremental learning [van de Ven et al., 2022].

Datasets

This evaluation utilizes datasets from vision tasks, and employs DNN architectures including various convolutional neural networks and transformers. The employed datasets are described as follows:

• In task-incremental scenarios, we incrementally train Resnet-18 [He et al., 2016] on CIFAR100 [Krizhevsky et al., 2009] and TinyImageNet [Wu et al., 2017]. Following the experimental setup in [Lin et al., 2022], CIFAR100 dataset is split into 10 non-overlapping subsets, resulting in the task incremental learning dataset CIFAR100-TIL-s10. TinyImageNet is similarly divided into 25 non-overlapping subsets, forming TinyImageNet-TIL-s25.
• In class incremental learning, we incrementally train ViT-B/16 [He et al., 2016] on CIFAR100 [Krizhevsky et al., 2009]. Following the settings in [Gao et al., 2024b], CIFAR100 dataset is split into 10 non-overlapping subsets, resulting in the class incremental learning dataset CIFAR100-CIL-s10. Different from the setup in taskincremental scenarios, there is no task identity provided in this setting, which makes the class incremental learning is more challenging than task incremental learning.
• In domain-incremental scenarios, we follow the experimental settings of [Shi andWang, 2024], where Sequential CORe50 [Lomonaco and Maltoni, 2017] is used to construct domain incremental learning dataset. CORe50 is a real-world incremental dataset that includes 50 do-

mestic objects collected from 11 domains (image size: 128×128; total of 120,000 images).

Evaluation Metrics

We use the average classification accuracy (Avg. Acc) to evaluate the overall incremental learning performance. Let Am,t represents the classification accuracy of the neural network on the t-th datasets after incrementally learning on m datasets, and let K be the total number of learning sessions. The Avg. Acc can be calculated as follows:

In addition, we employ the backward transfer metric (BWT) and forgetting measure (FM) [Wang et al., 2024] to evaluate the degree of forgetting during the incremental learning process. BWT and FM can be calculated as:

4.1 Comparison of SWRL with ExistingWorks

The proposed SWRL achieves state-of-the-art performance across various incremental learning scenarios. Specifically, in the task-incremental learning setting, we compare SWRL with Adam-NSCL [Wang et al., 2021],Connector [Lin et al., 2022], GPCNS [Yang et al., 2024], NI-SSCL [Xie et al., 2025], HINT [Krukowski et al., 2025], and BECAME [Li et al., 2025]. SWRL attains a remarkable average accuracy of 83.35% on CIFAR100-TIL-s10, while exhibiting minimal forgetting (as reflected in its superior BWT score -0.76%), thereby outperforming existing methods. Moreover, on the more challenging TinyImageNet-TIL-s25 dataset, although its BWT is not the highest, it achieves the best overall performance with average accuracy of approximately 68.52%, substantially outperforming other approaches. Additional results are reported in Appendix, where SWRL achieves 94.12% average accuracy using ViT-B/16 on TinyImageNet-TIL-s25.

Class incremental learning is more challenging than task incremental learning, as task identities are not provided. In our CIFAR100 class-incremental learning experiment, we employed a commonly used Vision Transformer, ViT-B/16

[Dosovitskiy et al., 2021], as the base model and applied SWRL for training. The results in Table 2 show that SWRL not only enhances the performance of convolutional networks in task incremental learning but is also effective for transformer-based models, even in the more challenging class incremental setting. Compared with state-of-the-art methods such as C-prompt [Gao et al., 2024b] and DMNSP [Kang et al., 2025], SWRL achieves a better balance between stability and plasticity, attaining an overall accuracy of 89.15%. Crucially, owing to its architecture-agnostic and task-agnostic design, SWRL is more generalizable than existing leading prompt-tuning methods, which are often effective only for transformer-based models.

In domain-incremental learning settings, models are required to continuously adapt to new data distributions. A Vision Transformer ViT-B/16 pre-trained on ImageNet-21K is used as the base model. The results, presented in Table 3, show that SWRL achieves an overall accuracy of 98.29%, outperforming both replay-based methods using additional data buffers and exemplar-free prompt-based methods such as DUCT [Zhou et al., 2025]. In summary, the results in Tables 1 to 3 demonstrate that SWRL is a general incremental learning method applicable to various incremental learning scenarios across diverse deep neural network architectures.

4.2 Analysis of SWRL

Minimization of the Integration Gap

The designed fast learning process, consisting of stabilityoriented preview process of Mnt and plasticity-oriented learning of Mht , effectively closes the integration gap between the new-learned memory and previous long-term memory in Mnt . We visualize the loss landscape of models on the 1st and 2nd task in Figure 4 by training Resnet-18 on CIFAR100-TIL-s10. It can be seen that the preview process optimizeMnt to move to the low loss region of the 2nd task, while keeping its location at the low loss region of the 1st task. Results reported in Table 4 verify the effectiveness of the designed fast learning process.

Balancing Memory Stability and Learning Plasticity

The neuron circuits of the neocortex-like model will be changed when integrating new learned knowledge from the hippocampus-like model, inevitably disturbing the memory stability. To balance the learning plasticity and memory stability, the proposed SWRL adaptively optimizes the integration policy, as described in Eq.(10).

We take memory integration comparison experiments on Resnet-18 using CIFAR100-TIL-s10. Specifically, we compare the performance when using meta-weighted memory integration guided by SWR-like signals, and adopting linear fusion with constant fusion coefficient matrices, respectively. As illustrated in Table 4, the designed memory integration mechanism effectively enhances the overall accuracy while alleviating the forgetting. Figure 5 further presents the comparison results on ResNet-18 using TinyImageNet-TIL-s25, showing that the meta-weighted memory integration achieves better incremental learning performance.

For more intuitively demonstrating the changes of neocortex-like model after memory integration, we further visualize the neuron-circuits activation pattern in Figure 6. For clarity, we highlight partial neurons with higher activation levels. Figure 6 demonstrates that, after memory integration, the neocortex-like model reproduce a similar neuronactivation pattern which is obtained in the hippocampus-like model, and still keeping the previous activation pattern when processing previous SWR-like signals.

5 Conclusion

In this work, we propose a general and bio-inspired incremental learning framework named SWRL. The proposed SWRL simulates the collaborative mechanism between the hippocampus and neocortex by employing a two stage learning framework, including the fast learning stage and metaweighted memory integration guided by SWR-like signals. SWRL is a task-agnostic, architecture-agnostic, and bufferfree methods. Comprehensive experiments demonstrate its superior performance in different incremental learning scenarios, across various deep neural network architectures.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (T2521006 and 61973311).

Contribution Statement

Yi Sun and Xiaochang Hu contributed equally to this work. Jian Li and Jiting Li are corresponding authors.