MSVIT: Improving Spiking Vision Transformer Using Multi-scale Attention Fusion

1 Introduction

Spiking Neural Networks (SNNs), referred to as thirdgeneration neural networks [Maass, 1997], have garnered significant attention attributed to their biological plausibility, event-driven processing, and potential for high energy efficiency [Roy et al., 2019; Pei et al., 2019]. Despite these advantages, SNNs have yet to achieve performance levels comparable to traditional Artificial Neural Networks (ANNs), particularly in complex vision tasks. This performance gap presents a major obstacle to the widespread adoption of SNNs in practical applications.

Initially developed for natural language processing tasks, Transformers [Vaswani et al., 2017] have been extensively explored and extended to various computer vision applications, including image classification [Dosovitskiy, 2020], object detection [Liu et al., 2021], and semantic segmentation [Wang et al., 2021]. Notably, the self-attention mechanism-the core component of Transformers enables the models to focus on salient input information selectively, bears a striking resemblance to the selective attention processes discovered in the human biological system [Whittington et al., 2018].

∗Corresponding authors.

Given the biological plausibility of self-attention mechanisms and their alignment with human cognitive processes, it is intuitive to investigate their integration within Spiking Neural Networks (SNNs) to enhance deep learning models. The combination of the powerful representational capabilities of Transformers with the energy-efficient nature of SNNs presents an exciting research direction. In recent years, several studies have explored incorporating spiking neurons into Transformer-based architectures. For instance, Zhou et al. [Zhou et al., 2023] introduced "Spikformer," a spike-driven self-attention mechanism, marking the first attempt to integrate spike-driven neurons into the Transformer framework. Shi et al. [Shi et al., 2024] proposed SpikingResformer, which combines ResNet-inspired architecture with self-attention computation to achieve competitive performance in image classification tasks while maintaining low energy consumption. Yao et al. [Yao et al., 2024b; Yao et al., 2024a] developed two versions of Spike-driven Transformer trained using sparse AND-ACcumulate (AC) operations, achieving state-of-the-art (SOTA) results on the ImageNet-1K dataset among SNN-based architectures in the same terms.

Despite these advancements, there remains a performance gap between SNN-based models and traditional Artificial Neural Network (ANN) counterparts. While spiking neurons offer energy-efficient processing, their binary nature (using only 0 and 1 spikes) poses significant challenges in training larger and deeper networks. Unlike ANNs, which rely on floating-point matrix multiplication and softmax operations, the binary representation in SNNs struggles to capture the intrinsic and diverse information present in input data, often resulting in suboptimal accuracy for downstream tasks. Addressing these limitations is critical to exploring the full potential of SNN-based Transformer architectures.

Multi-scale structures are extensively utilized across computer vision (CV) [Fan et al., 2021], natural language processing (NLP) [Guo et al., 2020], and signal processing domains [Park et al., 2019] due to their effectiveness in capturing patterns at varying scales. Studies on the visual cortex of cats and monkeys suggest that increasing the number of distinct channels, with each channel corresponding to progressively specialized features, enhances the model's ability to extract structured information from input images [Fan et al., 2021; Koenderink, 1984]. These findings demonstrate that multi-scale feature extraction improves the robustness and generalizability of learned representations in CV tasks.

Motivated by these insights, we propose a novel spiking attention mechanism, MSSA, which incorporates MultiScale Spike Attention (MSSA) blocks into transformer architecture. Each attention head within the MSSA block operates at varying scales via multiple inputs, thereby enriching the perceptual field of the spiking self-attention mechanism. Larger scales capture more global and smoother features, while smaller scales focus on local details, enhancing the sharpness and distinctiveness of feature representations. The balance between global and local information of inputs is crucial for spiking neural networks (SNNs), where it is essential to develop innovative methods that expand the representational capacity of spike-based models.

Building on MSSA, we further introduce the Multi-Scale Spike-driven Transformer, comprising hierarchical layers with MSSAs. Each layer is designed with attention heads that process input images at different scales, enabling the network to capture multi-scale features effectively. This architecture bridges the seminal concept of multi-scale feature hierarchies with spike-driven Transformer models, leveraging principles of resolution and channel scaling to improve performance in various visual recognition tasks. We hypothesize that integrating multi-scale attention mechanisms into spikedriven Transformers will significantly enhance their capability to extract diverse and robust features, thereby advancing the state of spiking models in computer vision applications.

The contributions of this work are summarized as follows:

1. We develop a novel multi-scale spiking attention module, tailor-made for the SNN's attributes, which enables the spiking transformer to extract features from different scales of inputs by spikes with low energy costs, improving the performance of the spiking transformer.
2. We develop a direct-training hierarchical spiking transformer, namely MSVIT, incorporating MSSA into a vision transformer. This design marks the effective exploration of multi-scale spiking representation in Transformer-based SNNs.
3. We conduct extensive experiments on mainstream static and neuromorphic datasets, achieving state-of-the-art performance compared to the latest SNN-based models. Notably, MSVIT has surpassed QKFormer, which has achieved 84.22% top-1 accuracy on ImageNet-1K with 2242 input size and 4-time steps using the direct training from scratch by 85.06%, positioning itself as a state-ofthe-art solution among SNN-transformer architectures.

Finally, based on the aforementioned experimental results, we conduct the ablation study to discuss and analyze MSVIT.

The source code is open-sourced and available at https:// github.com/Nanhu-AI-Lab/MSViT.

2 RelatedWork

2.1 Vision Transformers

ViTs segment images into patches and apply self-attention [Vaswani et al., 2017; Devlin, 2018] to learn contextual relationships, effectively reducing inductive bias [Neil and Dirk, 2020; Neil and Dirk, 2020] and outperforming CNNs across multiple vision tasks [Mei et al., 2021; Bertasius et al., 2021; Guo et al., 2021]. Nevertheless, ViTs face challenges like high parameter counts [Guo et al., 2021], and increased computational complexity proportional to token length [Pan et al., 2020; Liu et al., 2022]. To enhance the computational efficiency of ViTs, many researchers [Jie and Deng, 2023; Li et al., 2023] focused on exploring lightweight improvement methods from transformer architectures. For example, LeViT [Graham et al., 2021] incorporates convolutional elements to expedite processing, and MobileViT [Mehta and Rastegari, 2021] combines lightweight MobileNet blocks with MHSA, achieving lightweight ViTs successfully. However, these enhancements still rely on expensive MAC computations, which are not suitable for Edge devices. This highlights the need for investigating more energy-efficient ViT solutions. Involving SNNs in Transformer architectures is one of the approaches.

2.2 Transformer Architecture in Spiking Neural Networks

Transformer-based models have demonstrated remarkable capabilities in human-like text generation, natural language understanding, and text-to-image [Vaswani et al., 2017; Devlin, 2018]. As transformer-based networks have predominated in various tasks, researchers believe that the transformer architecture can also replicate the success when applied in spiking neural networks.

Zhou et al. proposed a Spiking Transformer model, namely Spikformer, which models images into sparse visual features by using spike-form information without using a softmax operation for the first time [Zhou et al., 2023]. This form of spiking-transformer conducts bio-inspired spatio-temporal dynamics and spike (0/1) activations to obtain high energy efficiency from spiking self-attention. Subsequently, Zhou et al. proposed an SNN-based variant to improve Spikformer to Spikformer-v2 [Zhou et al., 2024b].

The authors developed a Spiking Convolutional Stem (SCS) with supplementary convolutional layers and connected SCS to Spikformer to enhance the image representation by spikes. Additionally, Spikformer-v2 introduced selfsupervised learning for directly training SNNs.

Man et al. proposed a Spike-driven Transformer (SDT) [Yao et al., 2024b], which exploited only mask and addition operations without any multiplication and made up to 87.2× lower computation energy than vanilla self-attention. However, the experiment results in this work show that the top-1 accuracy on ImageNet-1K was only 77.1% with 66.34 (M) parameters, which still has a large gap between ANNbased transformers. Therefore, the authors improved SDT to SDT-v2 [Yao et al., 2024a], which extended the Spike-driven Transformer into a meta form. On ImageNet-1K, SDT-v2 achieved top-1 accuracy up to 80.0% with 55 (M) parameters, surpassing SDT-v1 by 3.7%. Zhou et al. proposed QKFormer [Zhou et al., 2024a], which enhanced the spiking transformer architecture by using a spiking Q-K attention module. Unlike the comment form of self-attention with Query (Q), Key (K), and Value (V), Q-K attention only adopts Q and K to implement the self-attention mechanism. The authors reported that QKFormer, significantly improved performance compared to Spikformer on ImageNet-1K.

3 Methods

In this section, we first present the overall architecture of MSVIT. Secondly, we introduce the important components of MSVIT, including hybrid spiking attention integration, Spiking Patch Embedding with Multi-scale Feature Fusion (SPEMSF) which serves as the tokenization method for MSVIT, and MSFormer Blocks. Finally, we introduce MSSA and SSA in detail.

3.1 Model Architecture

Overall of Architecture

The overview of MSVIT is illustrated in Figure 1. The input I of MSVIT is represented as T ×C0 ×H×W, where T denotes the timesteps, C0 denotes the number of channels, and H and W denote the height and width of the inputs, respectively.

When training MSVIT on static RGB image datasets, T is 1 and C0 is set to 3. For neuromorphic datasets, T = T0, C0 = 2. In the encoder, A patch size of 4 × 4 is used, and the input feature (4 × 4 × C0) is transformed into a spikeform representation with C channels (note C distinguishes from C0) using Spiking Patch Embedding with Multi-scale Feature Fusion-1 (SPEMSF-1).

Given an input I ∈ RT×C0×H×W , SPEMSF-1 first transforms I into a series of tokens xn (n ∈ N), where x ∈ RT×H×W×C and N = H4 × W4 . Along with the subsequent MSFormer block, this constitutes "Stage 1". To construct a hierarchical spiking transformer, the number of tokens n is further reduced in SPEMSF-2 and SPEMSF-3 to H8 × W8 and H 16 × W16 , respectively. These transformations are handled in "Stage 2" and "Stage 3", where each stage reduces the token dimensions by a 2 × 2 patch.

In the first and the second stages, the number of channels C is set to 2, while it is increased to 4 in Stage 3. The number of spiking transformer layers L in each stage is configured as l1, l2, and l3, based on the model size and the dataset being trained (detailed settings are provided in Section 4).

Together, the three stages collectively implement the hierarchical spiking-transformer architecture for MSVIT.

Hybrid Spiking Attention Block Integration

While Spiking Neural Networks (SNNs) are inherently energy-efficient, the amount of information that can be processed through spikes in transformer architectures remains limited compared to float values [Zhang et al., 2022; Zhou et al., 2023], resulting in suboptimal performance.

To address this limitation, we integrate multi-scale spiking attention (MSSA) into MSVIT. However, applying a single type of MSSA across all stages of MSVIT leads to an increase in model size, which is not ideal. To break the tradeoff between performance and efficiency, we revisit the model design and propose a hybrid spiking attention mechanism for MSVIT. In the first and second stages, we adopt MSSA within the MSFormer block to enhance feature extraction. In the final stage, we employ the standard spiking self-attention (SSA), as used in Spikformer [Zhou et al., 2023], to process the deeper layers of MSVIT. This hybrid design strikes a balance between model size and performance, enabling MSVIT to achieve improved accuracy while maintaining a relatively low energy cost during training.

MSFormer Blocks

Meta-former [Yu et al., 2022] established that the general structure of transformers can be characterized by two key components: a token mixer and a channel mixer. In conventional transformer architectures, the token mixer is often implemented by an attention block, while an MLP block implements the channel mixer.

Similarly, MSVIT adopts this structure and is referred to as MSFormer. Each MSFormer block comprises two modules: a multi-scale spiking attention (MSSA) module and a Spiking MLP (SMLP) module. The formulation of the block can be expressed as follows:

where N is the length of patches and D is the embedding size.

Spiking Patch Embedding with Multi-scale Feature Fusion

In this section, we describe the proposed Spiking Patch Embedding with Multi-Scale Feature Fusion (SPEMSF) in detail. For vision tasks, higher-level feature maps often preserve rich semantic information but suffer from lower resolution, whereas lower-level feature maps capture raw input details with higher resolution. To address this trade-off, some ANN-based methods [Chen et al., 2018; Badrinarayanan et al., 2017] combine feature maps across layers. However, the improvements in these works are limited due to the semantic gap between feature maps at different scales.

Inspired by these works, we propose a multi-scale feature fusion approach to enhance the spike representation in patch embedding for MSVIT. Our approach leverages convolutional layers with different kernel sizes to process input features at varying semantic levels, effectively treating them as multi-scale features. The fused features, represented as spikes, preserve both low-level and high-level semantic information, reducing information loss during embedding.

The core idea is to use lightweight convolutional operations to project input spiking maps into multiple feature channels. These projections are fused to generate a richer representation for subsequent processing. Specifically, we utilize a combination of linear projections Wd to achieve this.

Given the input spiking map X, the patch embedding process is formulated as:

where F and G are functions representing different multi-

scale transformations, and ⊕ denotes the fusion operation.

In our implementation:

• The linear projection Wd in function F is defined as a lightweight convolutional layer with a 1 × 1 kernel and stride ≥ 1, which focuses on channel-wise transformations,
• function G uses a 3 × 3 convolutional layer with stride = 2, incorporating more spatial context while reducing resolution.

The implementation for function F is a simple pipeline of Conv2D-BN-SNN, while G is designed with one of the following configurations: (1) Conv2D-BN-MaxPoolingSN-Conv2D-BN-SNN, or (2) Conv2D-BN-SN-Conv2D-BNMaxPooling-SNN. This multi-scale feature fusion not only enriches the spike representation for inputs but also ensures compatibility with the channel and token requirements of the patch embedding block. The figure of Spiking Patch Embedding is further illustrated in Appendix B.

3.2 Multi-scale Spiking Attention (MSSA)

We propose a novel Spike-driven Transformer that maintains the spike-driven nature of Spiking Neural Networks (SNNs) throughout the network while achieving strong task performance by incorporating multi-scale features into the attention module.

The overview of Multi-Scale Spiking Attention (MSSA) is presented in Figure 2 (c). For comparison, traditional Vanilla Self-Attention (VSA) and Spiking Self-Attention (SSA) which is the core component of Spikformer, are shown in Figure. 2 (a) and (b). Both VSA and SSA use three components (Q, K, V ) and have computational complexity of O(N 2d) or O(Nd2 ). In contrast, MSSA achieves linear complexity of O(Nd). The initialization of the three components in MSSA is defined as follows:

where X ∈ {0, 1}T×N×D is the spiking map of the input, N represents the number of patches, and D is the feature dimension. Q and P represent lower and higher-level spiking feature maps of X, generated by convolutional layers. All three components (Q, P, V ) are produced through learnable linear matrices. Here, SN denotes the spiking neuron layer, and BN represents the batch normalization layer.

Unlike VSA (a) and SSA (b), MSSA replaces matrix multiplication with column summation to compute the attention interaction between the components (Q, P, V ). The attention mechanism in MSSA is defined as:

where ⊕ represents element-wise addition, and SUMc(·) performs column-wise summation, resulting in an N × 1 vector

α, which encodes self-attention scores in spike form:

To implement multi-scale feature fusion, αq and αp are mixed by element-wise addition (⊕), fusing the lower-level feature Q with the higher-level feature P via the attention scores. The resulting spike-based attention vector is then applied to the V spike matrix to generate the final spike representation of X using MSSA.

3.3 Spiking Self Attention

Spikformer [Zhou et al., 2023] introduced a novel spikebased self-attention mechanism, termed Spiking SelfAttention (SSA). Unlike traditional self-attention, SSA leverages sparse spike-form representations (Q,K, V ) and eliminates the need for softmax operations and floating-point matrix multiplications. The computational process of SSA is formulated as:

where Q,K, V ∈ RT×N×D are spike-form representations computed through learnable linear transformations. Here, s is a scaling factor. SN denotes the spiking neuron layer and BN represents the batch normalization layer.

In stage 3 of MSVIT, we adopt SSA to perform spiking attention in the deeper layers, leveraging its efficiency and alignment with the spike-driven computation paradigm.

4 Experiments

This section introduces the details of the experiment, including data collection, implementation, and evaluation methods.

4.1 Experimental Setup

We evaluate MSVIT on both static image classification and neuromorphic classification tasks. For static image classification, we use ImageNet-1K [Deng et al., 2009] and CIFAR10/100 [Krizhevsky et al., 2009]. For neuromorphic classification, we employ the CIFAR10-DVS [Li et al., 2017] and DVS128 Gesture [Amir et al., 2017] datasets. Appendix D introduces the datasets in detail.

4.2 Results on ImageNet-1KClassification

Experimental Setup on ImageNet-1K. We adopt a training recipe similar to that proposed in [Zhou et al., 2024a] and detail the configurations in this section. First, the model is trained in a distributed manner for 200 epochs on an 8A100 GPU server. We employ several data augmentation techniques, including RandAugment [Cubuk et al., 2020], random erasing [Zhong et al., 2020], and stochastic depth [Huang et al., 2016], with a batch size of 512. Additionally, gradient accumulation is utilized to stabilize training, as suggested in [He et al., 2022]. Second, the optimization process leverages synchronized AdamW with a base learning rate of 6 × 10-4 per batch size of 512. The learning rate is linearly warmed up at the initial stage and subsequently decays following a half-period cosine schedule. The effective runtime learning rate is scaled proportionally to the batch size, calculated as BatchSize/256 multiplied by the base learning rate. Finally, the architecture is designed with three stages (as illustrated in Figure 1), where the number of layers in each stage is configured as {l1 = 1, l2 = 2, l3 = 7}, respectively. These configurations collectively ensure robust and efficient training of the proposed model.

Primary Results on ImageNet-1K. The experimental results demonstrate the superior performance of our proposed MSVIT, surpassing previous works' performance. Generally, MSVIT (69.80 M) with input size of 2242 achieves 85.06% top-1 accuracy and 97.58% top-5 accuracy on ImageNet-1K, which performs the best in Table 1. To start with the experiments, we first compare MSVIT with Spikformer which is the first version of the spikeform transformer[Yao et al., 2022]. Our MSVIT (69.80 M, 85.06%) significantly outperforms Spikformer (66.34 M, 74.81%, by 10.25% with the input size of 2242 . In Table 1, we can see that MSVIT achieves the best performance on accuracy, utilizing slightly more parameters. Additionally, compared to SDSA, MSSA has lower computational complexity. Meanwhile, MSVIT outperforms Spikedriven Transformer [Yao et al., 2024b] (SDT, built by SDSA) by 7.81%, 8.39%, and 8.58% respectively at three model size levels (17.69M, 30.23M, 69.80M), and surpasses QKFormer by 1.29%, 0.92%, and 0.84 % by comparable parameters. Surprisingly, MSVIT obtains significant improvement, outperforming MST-T [Wang et al., 2023b] by 4.45% with 30.23M parameters, and 45.88 mJ energy cost.

Comparing with ANN Models on ImageNet. MSVIT is designed as an event-driven SNN model, in which the outputs of the embedding layers, the matrix calculation in the attention blocks, and information transmission are binary spikes {0, 1}. As a result, the multiplications of the weight matrix and activations, such as RELUs, which are important, can be replaced by AND-ACcumulate (AC) operations, which benefit model training with high energy efficiency. Meanwhile, to achieve a competitive performance for ANN-based models, we apply the hierarchical transformer architecture to MSVIT. Although the implementation of hierarchical architectures is relatively more complex than those using the same attention blocks throughout the networks ([Zhou et al., 2023; Yao et al., 2024b]), it is still worth using a hierarchical architecture to reduce the performance gap between ANNs and SNNs. This is mainly because the hierarchical architecture naturally has the flexibility to model at various scales and has linear computational complexity with respect to image input size [Liu et al., 2021]. For instance, our MSVIT outperforms the most well-known Transformer-based ANNs in performance with high energy efficiency under the same experiment conditions without pretraining or extra training data, among MSVIT (69.80M, 85.06%, SNN, 45.88mJ), Swin Transformer (88M, 84.5%, ANN, 216.20mJ) [Liu et al.,

2021], DeiT-B (86M, 83.1%, ANN, 254.84mJ) [Touvron et al., 2021] and ViT (85.59M, 77.9%, ANN, 254.84mJ) [Dosovitskiy, 2020] ( the detailed results are in Table 1).

4.3 Results on CIFAR and Neuromorphic Datasets

CIFAR Classification. We conduct experiments on smaller datasets and configure the training process to ensure sufficient model optimization. Specifically, we train the model for 400 epochs with a batch size of 128, following the setup of Spikformer [Zhou et al., 2023]. For the three-stage training of MSVIT, we utilize a total of 4 blocks distributed as {1, 1, 2} across the stages. Thanks to the hierarchical architectural design, MSVIT comprises 7.59M parameters, which is slightly larger than QKFormer (6.74M) but smaller than Spikformer (9.32M). The performance results on the CIFAR datasets are summarized in Table 2. On CIFAR10, MSVIT achieves an accuracy of 96.53%, outperforming Spikformer by 1.02% and QKFormer [Zhou et al., 2024a] by 0.35%. For CIFAR100, MSVIT achieves an accuracy of 81.98%, exceeding Spikformer (78.21%) by 3.77% and QKFormer by 0.86%. Notably, MSVIT surpasses Vision Transformer (ViT), an ANN-based model, on CIFAR100 by 0.93%. This represents a relatively significant improvement over other SNN-based Transformer architectures, demonstrating the efficacy of MSVIT in classification tasks on various datasets. Neuromorphic Classification. To evaluate MSVIT on neuromorphic tasks, we compare our model with the state-ofthe-art models using the CIFAR10-DVS and DVS-Gesture datasets. Unlike conventional static image datasets, neuromorphic datasets comprise event streams instead of RGB images. This introduces a significant domain shift between the source (static image datasets) and target (neuromorphic datasets) domains for models pre-trained on static images. We aggregate events over specific time intervals to address this discrepancy to construct frames. The RGB channels are replaced with positive, negative, and the sum of events as input features, respectively. For this experiment, we implement a lightweight version of MSVIT with only 1.67M parameters, utilizing a block configuration of {0, 1, 1} across the three stages. The maximum patch embedding dimension is set to 256. The model is trained for 200 epochs on the DVS128-Gesture dataset and 106 epochs on the CIFAR10DVS dataset. The number of time steps for the spiking neurons is set to either 10 or 16. The experimental results for temporal neuromorphic classification are summarized in Table 2. On the DVS128-Gesture dataset, MSVIT with 1.67M parameters achieves an accuracy of 98.80% using 16-time steps and 98.37% using 10-time steps. For the CIFAR10-DVS dataset, MSVIT achieves an accuracy of 84.30% using 16-time steps, significantly outperforming Spikformer by 3.4%. Moreover, with 10-time steps, MSVIT achieves an accuracy of 83.80%, surpassing Spikformer by 4.20% and QKFormer by 0.3%. These results highlight the efficiency and performance gains of MSVIT, with a minimal parameter count.

5 Ablation Study

Hybrid Spiking Attention Integration. We test MSVIT on CIFAR100 and use the MSVIT equipped with MSSA (on stage 1,2) and SSA (on stage 3) as the baseline. The results show that using the same MSSA at each stage achieves relatively high performance with 81.65% on Top-1 accuracy among the study cases in Table 3. However, the number of parameters of MSVIT increases too much, which may incur Unworthy computational consumption, especially in the case we conduct experiments on large-scale datasets, such as the ImageNet-1K. This is mainly because a (3 × 3) kernel in convolution layers consumes more parameters than a pointwise convolution layer (with (1 × 1) kernels) to extract features from a larger perception field. Most of the baselines, including Spikformer [Zhou et al., 2023], QKFormer [Zhou et al., 2024a], and SDT [Yao et al., 2024a], solely use the PointWise Convolution layer (PWConv) to conduct information extractions. Although PWConv decreases the number of parameters throughout the model, it may cause information loss of high-level features at the shallow layers of spike-form transformer architectures, which are stage 1 and stage 2 in our MSVIT. Particularly, using the binary spikes that only use {0, 1} to transfer information is difficult. We addressed this issue and conducted the fusion of the low-level feature Q, and the high-level feature P to transmit more features to the model. Finally, we adopted the hybrid spiking attentions (MSSA + SSA) as the final version of MSVIT to strike the trade-off between the computational efficiency and performance of the model. The experimental results also show that MSVIT gains great performance with 81.98% on CIFAR100 using only 7.59M parameters.

The effectiveness of MSSA. Lines 1- 4 in Table 3 illustrate the effectiveness of feature fusion at stages 1 and 2 of MSVIT. Line 1 shows that MSSA (P+P) performs 81.25, which is almost the same as the result of MSSA (P) (Line 3). This indicates that the fusion for the same feature sizes can not improve its performance for MSVIT. Line 2 shows MSSA(Q+Q) obtains only 81.15, which is similar to QKformer's ( only using 1 × 1 kernels ) experimental result. In conclusion, adopting only low-level/local feature maps may still incur semantic information loss, damaging the spiking representation of input images.

6 Conclusion

In this work, we design a novel spike-driven multi-scale attention (MSSA), which involves a multi-scale feature fusion mechanism in the spiking hierarchical transformer architecture to improve the model's performance. MSSA fuses lowlevel and high-level information of inputs, significantly improving the model performance with limited parameter increases. Furthermore, MSSA replaces the dot-product or Hadamard product existing in VSA or SSA with a column sum, maintaining the computation in linear complexity to the tokens of the inputs. Correspondingly, we also utilize Spiking Patch Embedding with Multi-scale Feature Fusion (SPEMSF), which enhances spiking representation for both high-level and low-level information of inputs, thereby promoting model improvement. Finally, we implement a hierarchical spiking transformer architecture equipped with the aforementioned MMSA and SPEMSF, namely MSVIT. We have conducted extensive experiments, and the results show that our model achieves state-of-the-art performance on both static and neuromorphic datasets. Notably, MSVIT achieved top-1 accuracy on ImageNet-1K over 85% with 69.80M parameters and image input of size 2242 by direct training from scratch. Leveraging the MSVIT's superior capabilities, we strive to inspire confidence in the application of Spiking Neural Networks through our work.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (Grant No. 2024YDLN0011, 2024YDLN0006) and the Research Grants Council of the Hong Kong SAR (Grant No. PolyU25216423).