Efficient and Accurate Conversion of Spiking Neural Network with Burst Spikes

1 Introduction

As a representative of artificial intelligence, deep learning has surpassed human performance in many fields. However, the artificial neural network (ANN) training procedure requires enormous energy consumption and massive memory, which is challenging to apply to lightweight devices and limited storage scenarios. Recently, neuromorphic hardware such as TrueNorth [Akopyan et al., 2015] and Loihi [Davies et al., 2018] has attracted the attention of researchers due to their low computational burden, and fast processing [Roy et al., 2019]. This kind of hardware does not run artificial neural networks but spiking neural networks (SNNs) [Maass, 1997], which is considered the third generation neural network. It uses discrete spikes for information transmission and inference based on the spatio-temporal paradigm, so the calculation is more efficient and biologically plausible.

∗Corresponding author

Although the spiking neural network has the above advantages, the backpropagation algorithm widely used in ANN cannot be used for directly training the spiking neural networks due to the non-differentiable character of the spikes. So far, training highly efficient spiking neural networks remains an open problem. Some progress has been made in training networks directly using synaptic plasticity rules [Zeng et al., 2017], or surrogate gradient methods [Wu et al., 2018]. The former draws on the brain's nervous system rules to make spiking neural networks more explicable. The efficient selection and combination of these learning rules can help us explore the nature of intelligence [Zeng et al., 2017]. The latter uses mature backpropagation technology to smooth the spike propagation process. The gradient optimization-based SNN effectively combines their advantages, which helps explore effective spatio-temporal feature representation [Wu et al., 2018]. However, these methods show poor scalability on large networks and datasets. The training process needs enormous computing resources and simulation time.

To obtain efficient inference ability and demonstrate the low energy consumption characteristics of SNN, the researchers convert well-trained ANN to SNN [Cao et al., 2015; Diehl et al., 2015], which significantly reduces the training time and calculation consumption. Fig. 1 shows the ANN-SNN conversion process. So far, the converted SNN with Integrate-and-Fire neurons (IF) has demonstrated great advantages in reinforcement learning, object detection, and tracking [Tan et al., 2020; Kim et al., 2020; Luo et al., 2020].

However, the conversion process often suffers from performance losses and time delays. Methods such as soft-reset, weight normalization, and spike-norm [Han et al., 2020; Diehl et al., 2015; Sengupta et al., 2019] have been proposed to address these problems but still require long simulation times (usually greater than 256 or even thousands) to achieve acceptable performance. It greatly limits the use of SNN on mobile devices and in specific scenarios, such as mobile phones and target tracking. As far as we know, it is unclear how the converted weights affect the performance of SNN. Except for the neuron differences between ANN and SNN, the temporal dimension of SNN and other operations such as the pooling layer receive little attention.

In order to obtain excellent performance in dozens of simulation times, we theoretically analyze the error of the conversion process from the perspective of SNN structure and propose a conversion strategy using burst spikes to make the firing rate in SNN approximate to the activation value in ANN. Meanwhile, Lateral Inhibition Pooling (LIPooling) based on lateral inhibition mechanism [Koyama and Pujala, 2018] is proposed to replace MaxPooling. To our best knowledge, this is the first time to analyze the error from the whole conversion process, including differences between ReLU and IF, time dimension, and pooling opration perspectives. Moreover, the above method does not need additional layers in the ANN training process. Our main contributions can be summarized as follows:

• We formulate the conversion process and divide the conversion errors into residual information, spikes of inactivated neurons, and pooling errors. Then, we analyze their influence on the conversion process.
• We propose a neuron model for releasing burst spikes under the clock-driven simulation framework and propose the LIPooling to make information transfer in SNN more accurate.
• The experimental results demonstrate the accuracy and high efficiency of the proposed algorithm. Our method achieves state-of-the-art performance in less than 100 time steps.

2 RelatedWork

The synaptic plasticity and surrogate gradient methods often require more computing resources and are ineffective. While the ANN-SNN conversion method proposed by Cao et al. [2015] can adapt to more complex situations. Diehl et al. [2015] attribute the conversion error to inaccurate activation values of neurons and propose data-based weight normalization and threshold balancing methods. Spike-Norm [Sengupta et al., 2019] also tries threshold balancing from the SNN perspective. In order to balance the simulation time and accuracy and make the conversion process more robust, Rueckauer et al. [2017] propose the p-Norm method. They also extend bias and Batch Normalization (BN) to the conversion process. The soft-reset method is proposed to further reduce the information loss in the generation of spikes. RMP method [Han et al., 2020]with soft reset achieves nearlossless conversion through threshold balancing. Spiking- YOLO [Kim et al., 2020] uses channel-wise weight normalization to achieve object detection but requires more than 2k simulation steps. Zhang et al. [2019] and Han et al. [2020] try to use temporal coding for conversion. They only use one spike or two spikes to further reduce the network's energy consumption. Kim et al. [2018] and Li et al.[2021a] try to use phase coding to pack more information to the spikes but still need a long simulation time. The hybrid method [Rathi et al., 2020] achieves good performance by using the synaptic plasticity or surrogate gradient method to finetune the converted SNN. Ding et al. [2021] make the output of ReLU more convenient for IF approximation by adding a limiting layer in the training process of ANN, but the problems in the conversion process are still not solved. As far as we know, it is still tough to get high-performance SNN with low time delay. In addition, most of the work tends to use average pooling without addressing inaccuracies of MaxPooling in the conversion process.

3 Method

In this section, we formulate the conversion process. Based on the analysis of conversion error, we propose the neuron model with burst spikes and LIPooling to solve the problems of residual information and pooling error, respectively. Then, experiments are carried out to prove the efficiency and accuracy of the proposed method.

3.1 Conversion fromANN to SNN

Neuron Model for ANN. For the ℓ-th convolutional layer or fully connected layer in ANN, its forward propagation process can be described as follows:

where a(ℓ) and o(ℓ) represent the input and the activation value in ℓ-th layer respectively. g(x) is the ReLU activation function. w(ℓ) denotes the weight, and bias b(ℓ) is omitted from the pre-activation values for ease of representation. L is the number of layers of the network.

Neuron Model for SNN. We use Integrate-and-Fire (IF) neuron for conversion. At the time step t, the IF neuron integrates the binary spike information of the previous layer. w(ℓ) i s(ℓ-1) can be regarded as the synaptic current I (ℓ) i of the neuron i. The membrane potential changes after receiving the current as follows:

where V (ℓ) i means the membrane potential at time step t. When the neuron membrane potential exceeds the threshold V (ℓ) th , which is usually set to 1, the neuron fires and its membrane potential is updated by soft-reset [Rueckauer et al., 2017], i.e., the threshold is subtracted. Conversely, the membrane potential remains constant, and s(ℓ) i (t) = 0.

ANN-SNN Conversion. The principle of conversion is to use the firing rate in SNN to approximate the activation value in ANN. Ideally, the two are approximately equal after T time steps, as shown below.

The maximum firing rate in SNN is one under the time-driven simulation framework. It requires that the activation values in ANN be distributed between 0 and 1. In order to solve this problem, data-based weight normalization [Diehl et al., 2015] is proposed. We use the p-th percentile of the activation values in the ℓ-th layer λℓ = max(ℓ) p to scale the corresponding weight. Since the bias is additive, there is no need to multiply the maximum value of the previous layer.

As for batch normalization, we follow [Rueckauer et al., 2017] and merge the convolutional layer and the subsequent BN layer to form a new convolutional layer.

An input a is transformed into BN[a] = γθ (a-µ)+β, where µ and θ are mean and variance of batch, β and γ are two learned parameters during training.

3.2 Dividing the Conversion Loss

Consider the case where weights are normalized using the parameter p, then the neuron in ℓ-th can be divided into R1 = {j | o(ℓ) j max(ℓ) p ≤ 1} which represents the neuron set whose activation value of layer ℓ-th is less than 1 after data normalization and R2 = {j | o(ℓ) j max(ℓ) p > 1} whose activation value is more than 1. The total received synaptic current in ℓ-th layer at time step t can be written as

It can be seen from Eq. (8) that clip operation for activation values greater than 1 causes residual information in the membrane potential of neurons at the previous layer, affecting the accuracy of information transmission, as shown in the red box in Fig. 2. In addition, the choice of p will make the conversion suffer from the dilemma of accuracy and speed. When p is larger, there are more neurons in R1 set, but less information of current I can be integrated at each time step.

Previous works do not take the influence of time dimension on conversion into account, and consider T∑ t=0 s(ℓ) i (t) = ⌊ ∑Tt=0 I (ℓ) i (t) Vth ⌋ when o(ℓ) j max(ℓ) p ∈ {0, 1}, where ⌊x⌋ returns the largest integer smaller than x . Ideally, we hope I (ℓ) i (t) = a(ℓ) i , then Eq. (5) will follow. However, as the synapse current changes from the weighted sum to the integration of discrete spikes in SNN, t∑ τ=0 I (ℓ) i (τ) fluctuates around t·a(ℓ) i with time t. When t∑ τ=0 I (ℓ) i (τ) ≥ V (ℓ) th , neurons with o(ℓ) i = 0 can also emit spikes, called spikes of inactivated neurons (SIN) as shown in the blue box in Fig. 2. Suppose M is the number of spikes from these inactivated neurons and V (ℓ) th = 1. Then we can get

M = 0, if a(ℓ) i > 0. SIN arises from the fact that the sum of partial weights reaches the threshold in a short time, which can be reduced to some extent by increasing the parameter p. The problem is exacerbated during phase coding because more information is packed in one spike. Not that SIN doesn't matter very much in classification problems, because we only need to get a maximum for classification. The approximation degree of ANN activation value is required to be higher for detection or segmentation application.

In addition, most work uses average pooling, because it is equivalent to convolution operations with the same kernel size and step size. Some works with MaxPooling select spikes from neurons with maximum firing rates to pass through. This not only requires extra calculation of firing rate, but also causes pooling errors in the output as shown in the green box in Fig. 2. A key problem is that the neurons selected for this method are changed, namely, arg max i ∑tτ=0 s(ℓ) i (τ) is not a constant. Then we get

where O is the set of input values for the MaxPooling layer. Due to the instability of synaptic current, the neuron with the maximum firing rate may vary from neuron to neuron, resulting in the output tending to be larger than the actual activation value and smaller than the sum of the first two large values.

3.3 Transmit with Burst Spikes

In the conversion process, no matter the value of p, the converted SNN will be caught in a dilemma of speed and precision. As shown in Eq. (5), under the clock-driven simulation framework, the maximum firing rate r(ℓ) is 1. However, in biological nervous systems and neuromorphic hardware, neurons process information asynchronously, updating

Algorithm 1 Transmit with Burst Spikes

between observable time steps. Burst neurons continuously emit spikes when stimulated by input current, increasing the reliability of information transmission.

Inspired by the neuron burst mechanism, we introduce burst neurons into converted SNN. It can emit spikes between two simulation steps. Then the neuron of the next layer processes these spikes the next time step, as shown in Fig. 3. Burst spikes can make the residual information in the membrane potential released between simulation steps. In contrast, the regular spikes can only release one spike at most at a limited simulation step. Of course, considering the time needed for spike issuance in the actual situation, we limit the maximum burst spikes number Γ to prevent interference to the next simulation step. The pseudo-code is shown in Algorithm 1. The introduction of burst can make R1 set larger, that

no change to p. Thus SNN can deliver the same amount of information at a faster way.

3.4 LIPooling

It can be seen from Eq. (11) that the problem of using the spikes of the neuron with the maximum firing rate to carry out the MaxPooling is that the indexed neurons are changing, and there is no interaction between neurons. When multiple neurons have similar, even equal firing rates, previous work randomly selects the output of one neuron or uses the output of the neuron with the smallest index. These neurons will play the role of winners in different degrees for some time steps. Therefore, the output of the Maxpooling layer of converted SNN is usually greater than the actual maximum value. So most works in the past use Average pooling instead.

Inspired by the lateral inhibition mechanism [Koyama and Pujala, 2018] of biological neurons, we propose LIPooling method for converted SNN, as shown in Fig. 4. The number of hidden layer neurons of LIPooling is the same as the input of MaxPooling, and the updating of membrane potential and spike is shown in the Eq. (3) and (4). The excitatory connection receives the spikes s(ℓ) i (t) from the previous layer one to one, and the inhibitory connection influences the membrane potential of other neurons. Then the output neurons sum up the spikes of the hidden layer neurons. In fact, due to historical inhibition, only one neuron spikes at most at any time step, that is, the neuron with the maximum firing rate is selected to pass through, and its membrane potential update can be described as follows:

where V (ℓ) hi and h(ℓ) represent potential and spikes of hidden layers respectively, wexc and winh represent excitatory and inhibitory connections respectively. Here we set wexc = 1, winh = -1, Vth = 1. Then we get the output s(ℓ+1) i (t) = ∑j h(ℓ) j (t). It should be noted that, by adding the lateral in-

hibition mechanism, the spikes emitted by neurons with the maximum firing rate are suppressed by LIPooling to offset the spikes emitted by other neurons in the previous simulation

4 Experiments

4.1 Experimental Setup

To demonstrate the effectiveness and efficiency of our algorithm, we conduct experiments on CIFAR and ImageNet. For CIFAR, we use stochastic gradient descent with 0.9 momentum for weight optimization. The cosine learning rate decay strategy with an initial value of 0.1 is used to change the learning rate dynamically. The network is optimized for 300 iterations with a batch size of 128. We use data augmentations for high performance. For ImageNet, we use a pre-trained model from pytorchcv 1 . We follow the previous work and use the same network structure to ensure a fair comparison. For CIFAR, we use VGG16 and ResNet20 models, and for ImageNet, we use VGG16 model. As section 3.2 described, SIN has less effect on the classification problem. Future work will explore the influence of the SIN problem on detection tasks.

1 https://pypi.org/project/pytorchcv/

• (a) Accuracy curve compared to the baseline.

4.2 Ablation Study

To verify the effectiveness of the proposed method, we perform ablation experiments on CIFAR100 with VGG16 and ResNet20. All nets are trained with BN layers.

We first verify the effect of burst spikes on conversion. In Tab. 1, our method has the lowest reasoning time and achieves higher performance compared to the regular spike based method. For example, when converting VGG16, our method requires only 32 time steps to get 71.41% accuracy. The network has different performances under different Γ, and the number of Γ is limited by the processing capacity of the neuromorphic hardware. The accuracy curve of the whole conversion process is shown in Fig. 5(a). Burst significantly improves the accuracy and speed of conversion compared to baseline.

We also verify the influence of LIPooling on conversion. To make the comparison fair, p-Norm with MaxPooling is used. LIPooling perfectly solves the problem of MaxPooling in SNN, thus significantly improving the accuracy and speed as shown in Tab. 1. We calculate the error between the output firing rate and the value that directly pool the firing rate of input layer, and count the ratio of positive activation values correctly issued when the allowable error e < 2 T . Fig. 5(b) demonstrates that LIPooling can accurately pass the maximum value according to the spikes after LIPooling.

4.3 Comparison with OtherWorks

In this section, we compare our proposed method with other existing SOTA conversion works. Our work shows higher and faster performance, as shown in the Tab. 2. First, we focus on the SNN performance of the conversion. For VGG16 and ResNet, our conversion method achieves a performance loss of less than 1% over 100 time steps. For CIFAR10 and CIFAR100, we achieve 96.59% (ResNet20) and 80.72% (ResNet20) performance, respectively, which is the best performance of SNN on these datasets. For ImageNet, the accuracy of 74.26% has obvious advantages over RMP [Han et al., 2020] and other methods. It is also close to the SNN calibration [Li et al., 2021b] method. In addition to the superior performance, our method has outstanding advantages in simulation time. For CIFAR10, we achieve 96.11% performance with only 32 time steps. We also need dozens of time steps to achieve acceptable performance (78.26% on VGG16 with T = 64) on CIFAR100. For ImageNet, our method is twice as fast as the SNN calibration method, achieving a performance of 73.99% with 128 time steps.

4.4 Energy Efficiency

In SNN, neurons that do not emit spikes do not participate in calculations, so the implementation of SNN on hardware has the characteristics of energy efficiency, especially on neuromorphic hardware. We select VGG16 to verify the energy efficiency of the proposed method on CIFAR100 and use the energy estimation method in [Rathi and Roy, 2021]. Generally, a 32-bit floating-point MAC operation consumes 4.6J, while the AC operation only consumes 0.9J. We calculate the mean firing rate of each layer on the whole validation set. Although burst spikes are used to improve the firing capacity of neurons, we can see from Fig. 5(c) that the maximum and minimum firing rates of each layer are 0.097 and 0.014, respectively, which reflects the sparsity of spikes activity. Our spiking VGG16 only costs 69.30% energy of ANN's consumption while getting SOTA performance with extra short time, which exhibits the properties of low time latency, high inference performance, and energy efficiency.

5 Conclusion

In this paper, we firstly theoretically analyze the nodes that may produce errors in conversion. Furthermore, we propose the neuron model for releasing burst spikes to prevent information residue and replacing the MaxPooling with LIPooling to ensure the accurate transmission of information. To the best of our knowledge, this is the first time to analyze the error from the whole conversion process, including the differences between ReLU and IF, time dimension, and pooling operation. Our method can achieve comparable accuracy with ANN in a short (< 100) simulation time. It can successfully convert VGGG16 on ImageNet, and our method also achieves the best performance on networks using BN and MaxPooling. Hopefully, our work can be combined with neuromorphic hardware to demonstrate the advantages of SNNs.

Acknowledgements

This study is supported by National Key Research and Development Program (2020AAA0107800), the Strategic Priority Research Program of the Chinese Academy of Sciences (Grant No. XDB32070100).