Tackling Long-Tailed Data Challenges in Spiking Neural Networks via Heterogeneous Knowledge Distillation

1 Introduction

Spiking Neural Networks (SNNs) [Maass, 1997], modeled after biological neurons, have garnered considerable research attention because they use binary spike signals for communication between units. Unlike conventional Artificial Neural Networks (ANNs), which rely on continuous activation functions, spiking neurons encode continuous input values into spike trains using models like the Leaky Integrate-andFire (LIF) neuron model and its variants, such as the PLIF. This spike-based computation allows for information encoding in both spatial and temporal domains, potentially improving computational efficiency and reducing energy consumption. As a result, SNNs [Zhao et al., 2024; Shen et al., 2025b;

∗Corresponding Author

Medium

Class Id

(a) Long-tailed distribution.

Shen et al., 2025a] are well-suited for real-time processing tasks and neuromorphic hardware applications.

Two primary methodologies for constructing spiking neural networks with standard spiking behavior exist. The first approach is ANN-to-SNN conversion [Diehl et al., 2015; Sengupta et al., 2019], where spiking neurons replace ReLU activation layers in ANNs. This conversion method, however, often necessitates a large number of time steps to approximate the ReLU function accurately, leading to increased latency. The second approach, direct training of SNNs [Mostafa, 2017; Neftci et al., 2019], a widely utilized method, involves unfolding the network across simulation time steps and applying backpropagation through time. Due to the non-differentiable nature of the spike-generation process, this backpropagation typically relies on surrogate gradients. While numerous ANN models have been successfully adapted to SNNs, their application to long-tailed data distributions, which are common in real-world tasks, remains underexplored.

Long-tailed distribution is prevalent in real-world applications. As shown in Fig. 1a, such datasets exhibit highly imbalanced sample distributions, posing significant challenges. While numerous approaches have been developed to address this problem in ANNs, little exploration has been done into the promising, efficient, and robust SNN architecture. This absence of research has significantly hindered the deployment and effectiveness of SNNs in practical scenarios. Moreover, we found that methods that aim to solve the long-tailed problem faced by ANNs are challenging to adapt to the unique

Tail architecture of SNNs. To address this limitation, our paper introduces an efficient knowledge distillation framework designed to optimize SNN performance in long-tailed data settings, providing a straightforward and effective approach for high-performance model adaptation.

This work proposes a novel direct training framework for SNNs that handles diverse, long-tailed datasets. The proposed framework, LT-SpikingFormer, integrates a selfattention mechanism to enable dynamic interactions among spiking features. To bridge the impact of long-tailed distribution on the model, we construct a new CNN-SNN heterogeneous knowledge distillation paradigm to extract robust knowledge from widely trained CNNs to alleviate overfitting in most categories and enhance balanced feature learning across categories. Toward this heterogeneous knowledge distillation optimization, we propose a multi-granularity hierarchical feature distillation objective that leverages crosslayer local features and network global predictions to facilitate refined information distillation to optimize the network, specifically for the performance of the tailed classes. Extensive experimental results [Cui et al., 2019; Liu et al., 2024; Wang et al., 2024; Ma et al., 2024] demonstrate that our method can achieve state-of-the-art results, as shown in Fig. 1b.

Contributions. The highlights of the paper are threefold: 1) By analyzing and summarizing the existing work, we propose the first spiking neural network for long-tailed data and employ the heterogeneous distillation paradigm of CNN-SNN to alleviate the imbalance of samples, filling the research gap of long-tailed spiking learning; 2) For the proposed heterogeneous distillation framework, we constructed a joint optimization manner consisting of global and local distillations, in which the local one with norm guided can effectively alleviate the long-tailed distribution in different latent space, while the global knowledge distillation based on finally predictions can significantly improve the overall recognition performance; 3) Extensive experimental results demonstrate that our method achieves state-of-the-art performance on several datasets. The ablation experiments are conducted to verify the effectiveness of each module.

2 RelatedWork

Long-tailed Visual Recognition. Long-tailed distribution is a significant challenge in machine learning and visual recognition. In image classification, common object categories dominate the dataset, while rare categories are underrepresented, leading to model bias toward the "head" categories during training, which causes overfitting, while generalization to the "tail" categories with fewer samples is limited, degrading overall performance.

Traditional methods to mitigate the long-tailed problem involve resampling and reweighting. Resampling methods aim to balance sample sizes across categories by oversampling tail categories [Li et al., 2022] or undersampling head categories [He and Garcia, 2009], though these can lead to overfitting or underfitting. Reweighting [Khan et al., 2017] adjusts the learning rates for categories based on sample sizes, though such methods can improve performance on tail categories at the expense of head categories. Two-stage training has been proposed to address this: the first stage focuses on feature representation learning on the original long-tailed distribution, while the second applies resampling or reweighting for fine-tuning, optimizing both feature learning and classifier learning.

Transfer learning [Long et al., 2022] and knowledge distillation [Zhao et al., 2023; Jin et al., 2023] also offer solutions. Transfer learning facilitates knowledge transfer from majority classes to minority classes, while knowledge distillation uses soft labels from a large teacher model to transfer knowledge to a smaller student model. Recent methods like the mixture of experts (MoE) model, an ensemble approach, assigns experts to focus on different categories or dynamically adjusts based on sample sizes.

Spiking Vision Transformers. Spikformer [Zhou et al., 2022] is the first hybrid architecture to integrate spiking neural networks with Transformers. It introduces a spiking selfattention mechanism that eliminates traditional multiplication operations by activating the query, key, and value with spiking neurons and replacing the softmax function with spiking neurons. Furthermore, it substitutes the layer normalization and GELU activation used in Transformers with batch normalization and spiking neurons. The Spike-driven Transformer [Yao et al., 2024] proposes a linear-complex peakdriven self-attention mechanism designed to improve spatiotemporal information processing and significantly reduce energy consumption. SpikingResformer [Shi et al., 2024] is a spiking neural network architecture that combines the strengths of ResNet and Vision Transformer, enhancing performance while reducing parameter count and energy consumption through the introduction of a double spiking selfattention mechanism.

Knowledge Distillation. Knowledge distillation (KD) enhances the performance of a student model by guiding it to "imitate" the more complex and higher-performing teacher model. In KD, the soft labels produced by the teacher model serve as the target to transfer knowledge from the large teacher model to the smaller student model. There are two primary approaches to knowledge distillation: logitbased KD and feature-based KD. Logit-based KD methods [Cho and Hariharan, 2019; Chen et al., 2024] leverage the output of the teacher model as supervision to direct the learning of the student model, focusing on mimicking the teacher's decision-making process. In contrast, featurebased KD methods [Ahn et al., 2019; Tung and Mori, 2019; Liang et al., 2024] aim to align teacher and student models at the feature representation level by minimizing the distance or discrepancy between their intermediate feature layers. More recently, the Heterogeneous Bridge Distillation [Hao et al., 2023] method has been introduced, which addresses the disparity between heterogeneous features by projecting the intermediate features of the student model into a latent space that aligns with the output of the teacher model.

3 Methodology

This section proposes a novel framework, LT-SpikingFormer, which trains Spiking Neural Networks (SNNs) from scratch with a heterogeneous distillation learning paradigm, addressing the challenges of long-tailed data distributions. LTSpikingFormer adopts a compact ResNet-32 model with minimal augmentation as the teacher to explore the synergistic potential of combining SNNs and CNNs and enhance the recognition capabilities of minority classes through distinctive spiking characteristics. Considering the massive difference between convolutional and spiking networks, we construct a heterogeneous distillation optimization strategy from local to global modules, which is particularly important for fine-grained recognition of underrepresented categories in long-tailed datasets. The framework is shown in Fig. 2.

3.1 Framework of LT-SpikingFormer

As the basic unit of the SNN, the spiking neuron receives the generated current and accumulates the membrane potential to compare with the threshold to determine whether to create a spike. We describe the dynamics of the LIF (Leaky Integrateand-Fire) neuron by the following discrete-time model:

where U[t] and V [t] are the membrane potentials of neurons before and after charging, X[t] is the input current at time step t, and τ is the membrane time constant. Eq. (2) describes the firing process, where H(·) is the Heaviside step function. When the membrane potential U[t] exceeds the firing threshold Vth, the spike neuron will trigger a spike S[t]. Eq. (3) describes the resetting process. The membrane potential V [t] after the trigger event is equal to U[t] if no spike is generated, otherwise it is equal to the reset potential Vreset.

The Spike Neurons Layer comprises multiple LIF neurons, a key component for information encoding and transmission in SNN. In SNN, the input signal enters the SN layer for processing, where each LIF neuron decides when to emit a pulse based on the input current and its dynamic characteristics (such as membrane time constant, threshold, etc.). These pulses can then be further propagated and processed in the SNN to complete complex computational tasks. For simplicity and clarity in subsequent chapters, we use SN(·) to represent the spiking neuron layer, omitting the dynamic process inside neurons.

Given 2D images I ∈ RN×H×W×3 , a Spiking Feature Pre-Extractor G(·) is used to extract local features X = [x1, x2, · · · , xN] ∈ RN×T×D , where T, N, H, W, D denote time step, sample number, height, weight, and embedding dimension, respectively. We aim to learn a set of mappings X → Y = [y1, y2, · · · , yN] ∈ RN×C that enables accurate sample classification, where C is the number of classes.

It is crucial to adopt strategies that ensure the model's robustness and generalization to address the challenges of longtailed data distributions. An approach is to incorporate a balanced sampling technique alongside advanced image enhancement methods. Specifically, we utilize the imbalanced data sampler strategy to ensure adequate representation of minority classes during training. This sampler assigns sample weights based on the effective number of samples per class, promoting balanced class exposure and mitigating the risks of overfitting and under-sampling. In addition to conventional data augmentation strategies, such as rotation, flipping, and cropping, we also employ advanced methods like Mixup and CutMix, which have been proven to be very effective in alleviating the data imbalance problem.

On the other hand, to fully explore the association relationship in long-tailed data using Spike Neurons, we stack multiple Spiking Transformer (ST) blocks and represent them as F(·). As illustrated in Fig. 3, each ST block is composed of a Spike-based Self-Attention (SSA) block and a Spike MLP (SMLP) block, and residual connections are applied to each SSA block and SMLP block. The ST block can be formulated as follows:

where Xi is the output feature of i-th ST block. The SMLP consists of two feed-forward layers (FFL), and FFL(·) = BN(Conv(SN (·))).

For the Spike-based Self-Attention (SSA), we introduce a fully spike-driven self-attention mechanism, which is defined as follows:

where E is a linear dual-spike transformation function. σ1 and σ2 are the scaling factors. The key lies in the design of the scaling factors σ1 and σ2, which are adjusted based on the average firing rates of the input features and the attention map.

Finally, we perform global average pooling (GAP) and full connection classification (FC) on the features processed by the stacked Spiking Transformer encoders to obtain the final output prediction Y . SpikingFormer is defined as follows:

3.2 Heterogeneous Knowledge Distillation

To harness the extensive knowledge embedded in resourcerich CNNs, we introduce a novel heterogeneous teacherstudent knowledge distillation framework based on LTSpikingFormer. We select the classic ResNet-32 as the teacher model and train it with weak data augmentation. This selection is guided by the unique computational attributes of LT-SpikingFormer, enabling more flexible and efficient adaptation and optimization of the knowledge distillation process. The student model aims to achieve enhanced performance while maintaining lower complexity and fewer parameters.

For the heterogeneous architecture distillation between CNNs and SNNs, we emphasize extracting more profound and more effective information from the teacher model to improve the spiking neural network. We utilize two primary distillation strategies from global to local perspectives: Logitsbased Global Distillation and Norm-reduced Local Distillation.

Logits-based Global Distillation. In the knowledge distillation training framework, logits function as the essential knowledge representation, encapsulating the prediction distribution of the teacher model. The Logits-based Global Distillation aims to align the prediction of the teacher and student models. The output distribution of the teacher model across different classes guides the student model's predictions, allowing the student to approximate the teacher's performance in the output space progressively. This approach is formalized as:

where DKL is the Kullback-Leibler divergence function, N is the total number of samples, ysn and ytn represent the outputs of the student and teacher models on the n-th sample, respectively. This approach enhances the student network's predictive accuracy, particularly for minority class samples, as the teacher model provides a more refined and comprehensive category distribution.

Norm-guided Local Distillation. To leverage the deep feature information from the teacher model more effectively, we aim to guide the student model in learning the feature representations of the teacher model across different network layers via multi-granularity distillation. However, due to the heterogeneous nature of the teacher and student models, their feature representations exhibit differences. Specifically, the teacher network produces dense, continuous-valued features, while the student network generates sparse, pulsed features. Direct alignment of these disparate features using convolutional operations could introduce superfluous information, leading the student model to learn irrelevant or secondary features and failing to capture the essential features of the teacher model. This misalignment may negatively impact the efficiency and generalization performance of the model.

To address this challenge, we propose a Norm-reduced Local Distillation module. The core idea is to project heterogeneous features from the student and teacher networks into a shared latent space using norm constraints to facilitate the alignment of heterogeneous features. We first extract the mean feature of each category from the teacher network, which can eliminate the impact of long-tailed data on feature distribution. For any sample xn, the mean feature p(xn) of its category can be expressed as:

where T (·) denotes the final output of the teacher network.

C(xn) indicates the category to which xn belongs. Based on the above mean features, we use norm reduction to achieve mapping alignment of heterogeneous features. The implementation is defined as follows:

where F l (·) represents the feature output of the l-th latent layer of the student network, and ∆Θ is the adjustment term for enhancement. Since the teacher network is pre-trained, its feature distribution is known and relatively stable, allowing us to set a benchmark value Θ(p(xn)). The norm constraint minimizes the difference between each sample's mapped feature and the benchmark value, ensuring that the teacher's feature remains stable after being mapped to the latent space by the functions Θ and Φ.

For student loss Lls , the primary objective of the norm constraint is to make the intermediate layer output of the student network closely approximate the mapped features of the teacher network. This is achieved by minimizing the difference between the norm of the student features after mapping and the teacher benchmark value. By jointly optimizing these loss functions, the student network is trained to project its features into the same latent space as the teacher network, thereby aligning heterogeneous feature representations. Of course, we can capture different latent features to achieve a heterogeneous alignment. The combined local distillation loss Lloc is expressed as:

where L indicates the number of selected latent layer features. While local distillation shares some similarities with featurebased knowledge distillation, the norm-guided distillation in our LT-SpikingFormer is unique. Traditional feature-based distillation mainly aims to minimize the distance between intermediate feature layers. In contrast, our norm-guided local distillation uses norm constraints to project heterogeneous features into a shared latent space. This is new considering the differences between SNNs and CNNs. As equations show, it lessens the impact of long-tailed data on feature distribution and aligns student and teacher network features well.

Finally, the backbone network is trained using crossentropy loss, which is defined as follows:

where N and C represent the number of samples and the number of classes, respectively. ¯ yij is the actual label for the i-th sample belonging to the j-th class. yij is the prediction made by the student model that the i-th sample belongs to the j-th class.

Overall, the above three loss functions during the representation learning stage are assembled as a whole optimization objective:

where λglo and λloc are the adjustable loss weight coefficients. This unified loss function integrates the contributions from both distillation strategies, promoting the effective training of the LT-SpikingFormer in the context of long-tailed data challenges.

4 Experiments

4.1 Experimental Setup

In this section, we conduct comprehensive experiments to evaluate the proposed LT-SpikingFormer framework and validate its efficacy in the context of long-tailed learning tasks. Initially, a series of analytical experiments were undertaken to substantiate our hypotheses and provide an in-depth examination of the framework's constituent components. Specifically, these analyses encompassed: 1) an overall performance evaluation of the model, 2) a comparative analysis of two distinct distillation strategies, 3) a sensitivity analysis of hyperparameters, and 4) an assessment of the impact of data augmentation strategies. Subsequently, we performed a thorough comparison of our proposed method with prevailing supervised learning approaches on benchmark long-tailed datasets, including CIFAR10/100-LT and ImageNet-LT.

Model. We propose the LT-SpikingFormer framework, which utilizes SpikingFormer, a spiking neural network based on the self-attention mechanism, as the backbone for the student model. For small-scale datasets like CIFAR10-LT and CIFAR100-LT, we use ResNet-32 as the teacher model, and for larger datasets like ImageNet-LT, we employ ResNet-50. During the training of the student model, we apply crossentropy (CE) loss for the independent classifier and weighted Kullback-Leibler (KL) divergence for the distillation classifier to optimize the knowledge distillation process.

To further improve the student model, we propose a novel feature mapping strategy that uses norm constraints. This strategy maps the output of the teacher model and the intermediate-level information from the student model into a shared latent space. The student model, based on the SpikeTransformer framework, is divided into three equal parts, and the output of each segment is used as the exit point for feature mapping. Each exit point is made up of spike blocks, which are essential for transferring information from the teacher to the student.

Datasets. For the datasets, we assess our method using two small-scale datasets, CIFAR10-LT and CIFAR100-LT, and one large-scale dataset, ImageNet-LT. Long-tailed versions of CIFAR-10 and CIFAR-100 were generated using the approach described in [Cao et al., 2019], where the imbalance factor β is the ratio of the sample size of the most frequent class to the least frequent class. In our experiments, β was set to 100, 50, and 10, respectively. The original validation sets of CIFAR10 and CIFAR100 were used directly for testing. For ImageNet, we generated the long-tailed variant by sampling according to a Pareto distribution with a power value αp

= 6. The classes were categorized into Head, Medium, and Tail groups, and classification results were computed separately for each group.

For data augmentation, CIFAR10/100-LT datasets used weak augmentation techniques such as cropping, horizontal flipping, and rotation. Strong augmentation included these basic methods along with CIFAR10Policy and mixup [Zhang, 2017]. For ImageNet-LT, weak augmentation consisted of cropping, horizontal flipping, rotation, and ColorJitter. Strong augmentation for ImageNet-LT included mixup and cutmix [Yun et al., 2019] to ensure a fair comparison.

Baselines. In our experiments, we established multiple baselines to assess the performance of LT-SpikingFormer. The primary baseline was the SpikingFormer network itself, which served as the backbone of the student model across all experiments. This allowed for a consistent comparison of improvements introduced by LT-SpikingFormer. Additionally, we integrate the backbone model with state-of-the-art methods designed for long-tailed data distributions and compare their performance with that of our proposed method.

Optimization. For optimization, different optimizers were used for the student and teacher models. The student model employed AdamW with a weight decay of 0.01, a batch size of 128, and a cosine annealing learning rate starting at 0.0005 for 600 epochs on the CIFAR10/100-LT dataset. For ImageNet-1K, the learning rate was 0.001, and training lasted 320 epochs. The teacher model, based on CNN, was optimized using Sharpness-Aware Minimization (SAM) to improve generalization.

4.2 Result

We evaluate the proposed LT-SpikingFormer method against both single-stage and multi-stage training approaches using the CIFAR10-LT, CIFAR100-LT, and ImageNet-LT datasets.

All the methods examined are specifically designed to handle long-tailed data distributions.

Training results on small-scale. The table 1 present the training outcomes for CIFAR10-LT and CIFAR100-LT with imbalance factors of 10, 50, and 100, respectively. The proposed LT-SpikingFormer method achieves substantial performance improvements over competing approaches. Specifically, the LT-SpikingFormer model outperforms the backbone model by 14.67% on CIFAR10-LT and 16.86% on CIFAR100-LT. When compared to the distillation method BKD, our results show a 5%-10% improvement on both datasets. Notably, the common multi-stage training strategy in long-tailed learning typically involves the use of expert models. Our method demonstrates approximately 1% higher performance than MDCS, which is based on a self-distillation expert model, on CIFAR10-LT, with similar improvements observed on CIFAR100-LT.

Training results on large-scale. In table 2, we present the overall Top-1 accuracy for ImageNet-LT, along with the Top-1 accuracy for head, medium and tail categories. The LTSpikingFormer outperforms other competing methods on this dataset. To ensure a fair comparison, we re-implemented certain multi-stage networks under a consistent training environment. On the ImageNet-LT dataset, our LT-SpikingFormer model demonstrates a 7.52% improvement over the backbone model. Compared to existing methods such as DiVE and BKD, our LT-SpikingFormer achieves state-of-the-art (SOTA) performance under identical conditions.

4.3 Ablation Study and Further Analysis

Ablation studies on all components. As detailed in table 2, we evaluate the proposed components, including the Logitsbased Global loss (Lglo), Norm-reduced Local loss (Lloc), and weak-strong augmentation (w/mixup). When both Lglo and Lloc are marked as ✗ and Lce is ✓, it indicates that only the backbone model, SpikingFormer, is used for training. The ✗ in w/MixUp signifies that the data augmentation applied is limited to weak augmentation. As shown in table 3, implementing the LT-SpikingFormer with a combination of strong and weak augmentations increases performance from 72.04% to 76.54%, demonstrating the efficacy of mixup in handling long-tailed distributions. When the teacher model is introduced and the Lglo and Lloc losses are applied during training, the overall model performance improves by 4.45% and 7.08%, respectively. Lastly, applying the full distillation loss results in a further significant performance boost, raising accuracy to 86.71%.

Overview of the Imbalanced Classification Performance. Fig. 4 presents three confusion matrices for our method applied to the CIFAR-LT dataset, with imbalance factors of 100, 50, and 10, respectively. As depicted in the figure, our method performs effectively, particularly when the imbalance factor is large. As the imbalance factor increases, the method maintains high classification accuracy, with prominent values along the diagonal indicating robust prediction accuracy across all classes. Even at an imbalance factor of 100, the misclassification rate remains low, and the offdiagonal values are minimal. These results demonstrate that our method achieves excellent performance even in the presence of highly imbalanced datasets, highlighting its strong robustness and applicability.

Influence of loss weight λglo and λloc. λglo and λloc represent the weights of the two loss functions, and are used to control their relative contributions to the total loss. To determine suitable values for λglo and λloc, we performed a series of experiments on the CIFAR10-LT dataset (β=100). As illustrated in Fig. 5, the optimal performance was achieved when λglo = 0.5 and λloc = 0.05. This result indicates that our model has effectively balanced the use of external guidance for knowledge transfer, the refinement of internal feature representations, and its ability to learn directly from the data, thereby achieving the best overall performance.

Acknowledgements

This work is supported in part by the National Key Research and Development Program of China (No.2023YFC3305600), Joint Fund of Ministry of Education of China (8091B02072404), National Natural Science Foundation of China (62132016, 62171343, and 62201436), Key Research and Development Program of Shaanxi (2024GX-YBXM-127), Natural Science Basic Research Program of Shaanxi (2020JC-23) and National Key Laboratory Foundation of China (Grant No. HTKJ2024KL504011).