SAFformer: Improving Spiking Transformer via Active Predictive Filtering

1 Introduction

Spiking neural networks (SNNs) are regarded as the third generation neural network models[Maass, 1997]. Their event-driven and sparse computing characteristics have demonstrated great potential in the field of artificial intelligence, especially in the small-scale edge computing scenarios[Guo et al., 2022]. Unlike artificial neural networks (ANNs) that perform intensive computations at each inference step, SNNs transmit information asynchronously, with spiking neurons consuming energy only when receiving or emitting spikes[Eshraghian et al., 2023]. In recent years, the integration of SNNs with the powerful Transformer architecture has shown great promise, paving the way for a new generation of visual models that achieve high performance with improved energy efficiency.

Despite this promise, a fundamental limitation persists in the design of existing Spiking Transformers. Whether in early direct-conversion attention mechanisms or later linearcomplexity variants like QK-Attention[Zhang et al., 2024], the information selection process universally follows a "Passive Reactive" paradigm. This model involves indiscriminate and costly processing of all information, followed by a post-hoc filtering step that relies solely on the fixed threshold of spiking neurons[Guo et al., 2025; Chen et al., 2024]. Crucially, it lacks the ability to dynamically adjust its computational scope based on the input global context. Consequently, when faced with simple images containing large amounts of redundant background, the model still wastes significant computational resources on irrelevant regions. This paradigm runs counter to the highly efficient information processing mechanisms of the biological brain.

∗Corresponding authors.

In contrast, the predictive filtering mechanism inherent in the visual system enables the brain to utilize top-down feedback signals to modulate the activity of lower-level neurons[Crassidis and Markley, 1997; Rao and Ballard, 1999]. It actively suppresses predictable, redundant information and only forwards significant prediction errors for subsequent processing, thereby achieving extreme energy efficiency. Specifically, higher-level cortical areas (e.g. V4, IT)

form a holistic prediction of the scene based on prior experience and send these predictive signals down to lower-level areas (e.g. V1, V2)[Friston, 2009]. The lower-level cortex then compares this prediction with the actual visual input, and only the significant discrepancies are propagated upward as feedback signals, as illustrated in Fig. 3(b).

Beyond the self-attention module, the MLP layer in the Transformer architecture, as an important part of network feature enhancement, is also a centralized area for computation and parameters. Vanilla Spiking MLPs, typically direct conversions of their ANN counterparts, suffer from two major flaws. First, they are spatially blind. Their 1×1 convolutional structure can only mix information along the channel dimension[Wang et al., 2024] and is entirely incapable of perceiving or utilizing the local spatial context of tokens. Second, their feature transformation is static and homogeneous, applying an identical, unconditional transformation to all spatial locations[Chen et al., 2026b]. This inefficient processing model contradicts the principle of the visual cortex, which adjusts its processing intensity based on predictive information.

To systematically address these challenges, we draw inspiration from the brain's active prediction mechanism to propose SAFformer, a novel Spiking Transformer architecture that operates on the principle of Active Predictive Filtering. By incorporating two synergistic, bio-inspired innovations, SAFformer shifts the paradigm from passive reaction to active prediction. Our contributions are as follows:

1) We propose the Spiking Active Predictive Filtering Attention(SAF), a new sparse spiking self-attention mechanism, as shown in Fig. 2. It introduces a lightweight Spiking Sparsity Guidance Module (SGM) to actively predict the saliency distribution of the input, thereby prospectively focusing attention resources on key spatial locations and realizing active predictive filtering.

2) We propose the Spiking Multi-scale Adaptive Gated (SMAG) network to replace the vanilla SMLP. SMAG perceives local spatial context through a decoupled, multi-scale gating mechanism, achieving more refined extraction and selection of predictive features while significantly reducing the number of parameters.

3) Comprehensive experiments demonstrate that our architecture outperforms or matches SOTA Spiking Transformers on various datasets, with an excellent balance of efficiency and performance, as shown in Fig. 1. Notably, we achieve SOTA accuracies of 97.50% on CIFAR-10 and 83.38% on CIFAR-100, setting a new benchmark for the SNN field. 1

2 RelatedWork

2.1 Spiking Neural Networks

Spiking Neural Networks (SNNs) are widely recognized as the third generation of neural network models, with their design inspired by the information processing mechanisms of the biological brain. Unlike traditional ANNs, SNNs communicate by transmitting binary spikes, mimicking the firing of biological neurons[Duan et al., 2023]. When the membrane potential of a spiking neuron exceeds a certain threshold, it fires a spike, represented as '1'; otherwise, it remains silent, represented as '0'[Yao et al., 2024b]. This eventdriven mechanism facilitates sparse synaptic operations and avoids costly multiply-accumulate (MAC) operations[Chen et al., 2026a], thereby significantly enhancing the energy efficiency of these models. SNNs are primarily trained using two methods: ANN-to-SNN conversion and direct training with surrogate gradients[Di Yu et al., 2024]. ANN-to-SNN conversion involves training an ANN and then converting it to a corresponding quantized SNN version. Direct training, on the other hand, employs a surrogate function to approximate the spike firing process during backpropagation[Yao et al., 2024b]. The effectiveness of direct training in handling temporal data has made it a preferred choice in SNN research.

1 The full version is available at https://arxiv.org/abs/2605.08270.

However, conventional direct training methods require substantial GPU memory due to the simulation of multiple timesteps[Luo et al., 2024; Qiu et al., 2025]. To address the challenge of long training times, Yao introduced a novel direct training method called Spike Firing Approximation (SFA)[Yao et al., 2025]. This method uses quantized membrane potentials as activation values during the training phase and converts these values into spike sequences during inference, thereby reducing training duration. It has been demonstrated that SFA can reduce training time by at least 4.1×.

2.2 Spiking Vision Transformers

The Transformer, with its powerful sequence modeling capabilities and self-attention mechanism, has become a mainstream backbone in the field of computer vision[Dosovitskiy et al., 2020]. However, its quadratic computational complexity and significant memory footprint limit its application in power-sensitive and resource-constrained scenarios[Xiao et al., 2026]. Combining the powerful capabilities of the Transformer with the low-power and event-driven characteristics of SNNs has become an active research direction.

Spikformer[Zhou et al., 2023] pioneered spiking selfattention computation, establishing the first spiking ViT. Spikingformer[Zhou et al., 2026] further enhanced the SNN compatibility of the model by designing a preactivation residual connection. To achieve a fully spikedriven architecture, Spike-driven Transformer[Yao et al., 2023] implemented the Hadamard product in its self-attention module, becoming the first directly trained, fully spikedriven transformer model. To reduce temporal complexity, QKFormer[Zhang et al., 2024] proposed using only Q and K for self-attention computation, becoming the first spiking Transformer with a linear-complexity self-attention module. Additionally, OST[Song et al., 2024] compresses the time domain into 1 to enhance the training speed. To fully address the issue of over-allocating attention, SpiLiFormer[Zheng et al., 2025] introduced a lateral inhibition mechanism to simulate the inhibitory-excitation interaction process.

3 Methods

In this section, we first introduce the spiking neuron model we used. Next, we present the SAFformer architecture, which includes a novel spiking active predictive filtering attention mechanism and a multi-scale gated feedforward network.

3.1 Preliminary

In this paper, we use the widely adopted LIF model [Guo et al., 2024], which can be described by the following formula:

where t is the time step, U[t] is the membrane potential, the spatial input X[t] is extracted from the original spike sequence via Conv or Linear layers, and the temporal input βH[t - 1] comes from the decayed membrane potential of the previous time step, β is the leak factor. S[t] is the output spike, and H[t] is the membrane potential after firing. Hea(·) is the Heaviside step function, where Hea(x) = 1 if x ≥ 0. When U[t] exceeds the threshold Vth, the spiking neuron fires a spike, and the membrane potential is reset to Vreset.

3.2 Overall Architecture

The overall architecture of SAFformer is shown in Fig. 3(a), which consists of three stages. The number of blocks (SAFformer blocks or Spikformer blocks) in each stage is represented as N1, N2, and N3, respectively. The input data is represented as I ∈ RT×Cin×H×W , where Cin = 3 for static image data and Cin = 2 for neuromorphic data. In the first stage, a patch size of 4 × 4 is utilized. A Patch Embedding module converts the input into a spike sequence and maps the feature of each patch to an initial channel dimension C, reducing the number of tokens to H4 × W4 .The transformed features are then processed through SAF and SMAG. The processing mechanism in the second stage is similar, but with a patch size of 2 × 2, an expanded channel dimension of 2C, and a reduced token count of H8 × W8 . The third stage is the deep stage, where we continue to use a 2 × 2 patch size, expand the channel dimension to 4C, and reduce the token count to H 16 × W16 .

3.3 Spiking Active Predictive Filtering(SAF)

We propose Spiking Active Predictive Filtering Attention (SAF), whose detailed structure is shown in Fig. 3(c). The core idea of SAF lies in that it no longer calculates the pairwise relationships between tokens. Instead, it employs a lightweight guidance module to predict the most informationdense regions and dynamically focuses computational resources on them. Considering that the intrinsic low-pass filtering characteristics of spiking neurons can attenuate highfrequency signals[Yao et al., 2024a; Fang et al., 2025], we introduce a depthwise separable convolution (DWConv) into the generation paths of the spike matrices Q and K. This is to better preserve local details crucial in the shallow stages of the network. For a given input spike sequence X ∈ RN×C , where N is the number of tokens and C is the channel dimension, the generation is formulated as:

where WQ and WK represent the linear projections of the input spike sequence X, and SN(·) is the spiking activation function. Then, we obtain the initial saliency score for each token by summing over the channel dimension of Q:

To actively filter redundant information and adaptively adjust the computational load based on content complexity, we develop a lightweight and learnable module, named Spiking

Algorithm 1 Spiking Sparsity Guidance Module (SGM)

Sparsity Guidance Module (SGM). It dynamically predict a suitable attention matrix sparsity level, k, based on the global information of the input. In Algorithm 1, we detail the full process of generatingdynamic k using SGM.

With the dynamic k generated by SGM, a masking operation is performed on At to create a binary mask matrix Mk:

where Top-K(·, k) denotes the selection of the k largest elements from the vector.

Finally, the mask Mk is applied to the saliency scores, and the result is multiplied element-wise with K in a Hadamard product:

Theoretical Analysis

To formally analyze the fundamental differences in information processing across attention mechanisms, we introduce information flow graph[Ahlswede et al., 2000] as a unified analytical framework.

An information flow graph is a weighted directed graph G = (V, E,W), where the vertex set V = {v1, . . . , vN} represents the input tokens, a directed edge (vj , vi) ∈ E signifies an information flow from token j to token i, and the weight function W : E → R assigns a weight wij to each edge. The output representation yi for token i is a weighted combination of information from its inbound neighbors:

where In(vi) is the set of inbound neighbors of vi, Info(vj) is the information carried by token vj, and f(·) is a non-linear activation function.

Proposition 1. The computational paradigm of SSA is equivalent to performing a relational aggregation operation on a dense, fully-connected graph.

Proof. The output yi of SSA is determined by the equation:

where Qi,Kj, Vj are the Query,Key,Value vectors derived from tokens i and j. The summation ∑Nj=1 iterates over all input tokens vj for any given output yi. This implies that for any pair of vertices (vi, vj), an information flow path, i.e., a directed edge (vj, vi), exists. Therefore, the graph is a complete graph where the edge set is E = V × V .

And the weight of an edge (vj, vi), denoted wi, j, is given by the inner product 〈Qi,Kj〉. This weight is a binary relational function R(Qi,Kj) that quantifies the pairwise relation between vertices vi and vj. The final output yi is a weighted sum of the Value information vj from all vertices in the graph. This process, which gathers information from the entire graph, is a quintessential aggregation operation.

Proposition 2. The computational paradigm of SAF is equivalent to performing a node-wise filtering operation on a dynamically sparse subgraph.

Proof. The output yi of SAF can be represented as:

where qi, ki are derived from token i, and (Mk)i is a dynamic mask. The computation of yi depends solely on its own derivatives, implying that no cross-vertex edges exist. The information flow graph thus consists only of self-loops.

However, not all vertices participate in the computation. A global selection function Ψ(X, k), realized by the SGM, first identifies an active vertex subset Vactive ⊆ V .

where |Vactive| = k. The effective computation graph G′ is thus a dynamic sparse subgraph consisting only of the vertices in Vactive.

For any active vertex vi ∈ Vactive, the weight of its selfloop, wii, is determined by a unary saliency function S(vi) which is computed as ∑Cc=1 Qc,i. This weight is a function of the vertex's intrinsic properties, not its relation to others. The output yi is a scaled version of the vertex's own information Ki, which constitutes a filtering operation. Since the selection of the subgraph is dynamically determined by the global input X, the process is an active adaptive filtering.

SAF facilitates a paradigm shift in spiking self-attention, from relational aggregation to dynamic adaptive filtering, offering a new perspective for building efficient and powerful attention mechanisms in Spiking Neural Networks.

3.4 SpikingMulti-scale Adaptive Gated Network(SMAG)

Spiking MLP is a key part of the Spiking Transformer for improving feature representation. However, the commonly used SMLP with wide intermediate channels has the disadvantages of excessive parameter quantity and spatial blind viewing[Qiu et al., 2024], which violates the principle of active predictive filtering that we pursue.

Therefore, we propose a novel feedforward network, the Spiking Multi-scale Adaptive Gated Network (SMAG), to replace the vanilla SMLP, as illustrated in Fig. 3(d). SMAG performs feature transformation in a refine-and-gate paradigm, enhancing performance while reducing the burden of processing redundant information. Specifically, we first introduce a partial convolution[Chen et al., 2023] to a fraction of the channels of the input spike features X ∈ RT×B×C×H×W . This operation injects valuable local spatial priors into the feature maps at a minimal computational cost. Subsequently, the features are decoupled into three independent functional paths as follows:

where PConv(·) is the partial convolution operation, and [, ] denotes the channel-wise splitting, and W1 represents a pointwise convolution. PConv effectively introduces local context, which provides a foundation for generating more context-aware gating signals by initially filtering and strengthening a subset of features. Then, X′1 and X′2 serve as gating signals, while X′3 acts as the main path. A multi-scale dynamic gating mechanism is introduced to capture spatial patterns at multiple scales:

where DWConv3×3 and DWConv7×7 are depthwise convolutions with kernel sizes of 3 and 7, respectively. They learn spatial filters independently for each channel, significantly reducing parameters and computational complexity compared to standard convolutions, while effectively capturing channel-specific spatial patterns. The parallel design of 3x3 and 7x7 kernels enables the gating signals to perceive both fine-grained details and broader regional context simultaneously. Finally, through the element-wise gating multiplication, active suppression and enhancement of the content stream are achieved.

Theoretically, SMAG has significantly fewer parameters than vanilla SMLP. Moreover, compared to the two identical linear transformations in SMLP, the dynamic gating mechanism of SMAG endows the network with the ability to adaptively select and modulate feature channels based on the input content. This allows the model to focus more on critical information and suppress redundancy, thereby improving the discriminability of features.

4 Experiments

In this section, we conduct a comprehensive set of experiments to evaluate our proposed method, comparing it with other recent state-of-the-art architectures on several widely-used benchmarks. We evaluate our model on various datasets, including the static datasets including CIFAR10/100[Krizhevsky et al., 2009] and ImageNet[Deng et al., 2009], as well as neuromorphic datasets CIFAR10-DVS[Li et al., 2017] and DVS128Gesture[Amir et al., 2017].

4.1 Results on ImageNet-1KClassification

Experimental Setup on ImageNet-1K. The input image size is set to 224 × 224. For model training, we employ the SFA direct training method[Yao et al., 2025] with the virtual timestep D set to 4. We use the AdamW optimizer with a base learning rate of 6×10-4 , where the actual learning rate is calculated by multiplying the base rate by the batch size divided by 256. Training is conducted for 200 epochs with a batch size of 512, distributed across 6 NVIDIA RTX 3090

GPUs. Throughout the process, we apply standard data augmentation techniques, including RandAugment[Cubuk et al., 2020], mixup, and random erasing. The number of blocks in the three stages is set to {1, 2, 7}, respectively.

Primary Results on ImageNet-1K. The experimental results are shown in Table 1. Compared with the direct training method of vanilla time steps, our SAFformer (26.58M, 80.44%) achieves a 2.24% higher accuracy than STransformer v2 (31.30M, 77.20%) while reducing the number of parameters by approximately 15.1%. Compared with the direct training method of SFA, under the same virtual time step D = 4, our model has a 0.64% higher accuracy rate and similar energy consumption compared with E-SpikeFormer. Compared to the latest Max-Former (28.65M, 79.86%), SAFformer achieves a 0.58% performance improvement with fewer parameters and lower energy consumption. Without considering knowledge distillation, our model achieves comparable accuracy and efficiency to SOTA models in the SNN domain on ImageNet-1K.

4.2 Results on CIFAR and Neuromorphic Datasets

CIFAR Classification. On CIFAR datasets, SAFformer is trained for 400 epochs with a batch size of 64, and the virtual timestep D is set to 4. We use the AdamW optimizer with a learning rate of 1 × 10-3 . The SAFformer consists of 4 blocks, distributed across the three stages as {1, 1, 2}. The results are shown in Table 2. On CIFAR10, our model achieves 97.50% accuracy with 6.31M parameters, surpassing all previous models including Max-Former (97.04%) and STAA-SNN (97.14%). On the more challenging CIFAR100 dataset, SAFformer achieves an accuracy of 83.38%, which is also significantly higher than the previous SOTA models Max-Former (82.65%) and STAA-SNN (82.05%).

Neuromorphic Classification. The neuromorphic datasets use a SAFformer with 1.50M parameters, configured with a {0, 1, 1} block distribution across its three stages and a timestep of 16. The training process for DVS128Gesture involves 200 epochs, while for CIFAR10-DVS it is 106 epochs. On CIFAR10-DVS, our model achieves an accuracy of 85.0%, significantly outperforming QKFormer (84.0%) and Max-Former (84.2%) to establish a new state-of-theart. On the DVS128 Gesture recognition task, SAFformer achieves an accuracy of 99.0%, demonstrating highly competitive performance.

4.3 Ablation Study

We conduct ablation studies on CIFAR100 to analyze the effectiveness of each module, with all experiments following the training details described in Section 4.2 unless otherwise specified. The experimental results are presented in Table 3. The results show that each component contributes positively to the final performance. Specifically, SAF has improved its accuracy by 0.63% with almost no increase in the number of parameters by dynamically predict the tokens with the richest information. Furthermore, substituting the SMLP with our SMAG yields a substantial 3.28% accuracy gain while simultaneously reducing the parameter count by 0.42M. The best performance is achieved when both SAF and SMAG modules are combined, demonstrating that they are effective and complementary.

Effectiveness of the SGMModule in SAF. The effectiveness of the SGM module is validated in Fig. 4. We use a SAFformer without any filtering mechanism as the baseline. The "w/o SGM" version removes SGM and uses a fixed TopK ratio instead. For the 27M model, SGM reduces energy consumption by approximately 27% compared to the baseline and accelerates training time by 8%. Our SAFformer (15.12M, 3.75mJ,79.19%) outperforms the Fixed Top-k baseline (15.08M, 4.12mJ, 78.71%), which demonstrates SGM's effective efficiency and accuracy. While the training time is slightly longer for the smaller 15M model due to relative overhead, the substantial energy savings in all scenarios confirm the superior efficiency of our active predictive approach.

Component Analysis of the SMAGModule. We analyzed the contribution of each key design in SMAG through a stepby-step ablation, with results in Table 4. When we remove the multi-scale design of SMAG, the model's performance drops to 83.15%. Further removing the partial convolution reduces accuracy to 82.51%. Completely eliminating spatial awareness by replacing all depthwise convolutions with 1×1 convolutions causes the most significant performance degradation, with accuracy falling to 80.42%. These results clearly demonstrate that the superiority of SMAG stems from the synergistic combination of its spatial awareness, local prior injection, and multi-scale gating.

5 Conclusion

In this paper, we have delved into the inherent Passive Reactive limitation in existing Spiking Transformers. Drawing inspiration from the brain's predictive coding theory, we proposed SAFformer. It actively predicts and filters redundant information, guiding the model to focus its computational resources on salient features while preserving more highfrequency information. We introduced a new self-attention mechanism, SAF, which actively predicts global saliency via a lightweight guidance module to dynamically focus attention resources on key regions. Furthermore, we proposed SMAG to replace the vanilla SMLP. It leverages a decoupled, multiscale gating mechanism to overcome homogeneous transformation and parameter redundancy. Experimental results demonstrate that SAFformer achieves state-of-the-art performance on CIFAR-10/100 and CIFAR10-DVS datasets while maintaining low energy consumption.

Acknowledgments

This work was supported by the Natural Science Foundation of Guangdong Province, China (No.2025A1515012243) and the Guangdong S&T programme, China (No.2025A1010010002).