A New ANN-SNN Conversion Method with High Accuracy, Low Latency and Good Robustness

1 Introduction

With the rapid development of artificial intelligence, deep learning has played a more and more important role in many fields as a representative of AI algorithms. However, Artificial Neural Networks (ANNs) also face many challenges, such as poor robustness and high energy consumption. In recent years, due to the increasing demand for edge computing and energy-saving applications, Spiking Neural Networks (SNNs) have also attracted great attention because of their distinctive advantages from ANNs [Roy et al., 2019]. As a representative of the third-generation neural networks

∗Corresponding Author

[Maass, 1997], SNN is inspired by neurons in the biological nervous system, whose cells use discrete spike signals to encode and transmit information. It is precisely because the spike signal is asynchronous, event-based and nondifferentiable that SNN has excellent robustness and energy efficiency [Diehl and Cook, 2015], but it also faces challenges such as training difficulties [Tavanaei et al., 2019].

Although training the SNN network is very difficult (due to the inability to directly use the backpropagation algorithm), many excellent works have made a lot of contributions to the improvement of SNN's performance [Zhang et al., 2019; Wu et al., 2018a]. In general, there are two mainstream implementation methods of SNNs: ANN-SNN Conversion and direct training SNN.

ANN-SNN conversion. It can be proved mathematically that, when the inference time is long enough, ANN-SNN conversion can obtain the same accuracy as ANN [Sengupta et al., 2019a], which is based on the equivalence of Integrateand-Fire (IF) neuron and ReLU activation function [Cao et al., 2015a] under firing rate encoding. However, for finite time steps, there will always be errors in the ANN-SNN conversion, such as clipping error, quantization error (flooring error) and unevenness error [Bu et al., 2021a]. Among these three errors, unevenness error is caused by the unevenness of input spikes, while the clipping error and the quantization error are caused by firing rate encoding. In addition to firing rate encoding, we also often use membrane potential encoding [Li et al., 2022a], time encoding and sequence encoding [Jeffares et al., 2021] in SNNs.

Direct training. Due to the great success of gradient descent algorithm and backpropagation algorithm in training ANNs, some works regard SNNs as Recurrent Neural Networks (RNNs) and use Back-Propagation Through Time (BPTT) algorithm to train it. In these works, surrogate gradient methods are used to map discrete spike signals into differentiable mathematical formulas, so that the backpropagation algorithm can be used [Neftci et al., 2019; Wu et al., 2018b; Wunderlich and Pehle, 2021; Mostafa, 2017]. Although surrogate gradient methods can lead to very few time steps, the performance of SNNs trained in this way cannot reach a level comparable to that of ANNs. In addition to the surrogate gradient approach, there are some works using synaptic plasticity [Kheradpisheh et al., 2018] for network training, which is also considered to be more biologically interpretable.

Difficulties. Generally speaking, no matter which method of building SNNs has its own difficulties and defects. Although the ANN-SNN conversion method can obtain better network performance and can also be used in deep networks, it will cause a relatively high time latency because this method requires more time steps [Sengupta et al., 2019b]. In addition, while the direct training method has the advantages of fast inference speed and good robustness, it also has problems such as difficult training and insufficient performance [Zenke and Vogels, 2021]. That is to say, if we use existing methods, in order to obtain a spiking neural network with better performance, we must make a trade-off between inference speed, network accuracy and energy efficiency, which is also an open hot issue.

Contributions. In this work, we address two core challenges by combining the ANN-SNN conversion with the RNNs network structure, resulting in high-accuracy and lowlatency SNNs with improved robustness. Specifically, we carefully analyzed the causes of conversion errors in the ANN-SNN conversion method, replaced firing rate encoding with membrane potential encoding, and proposed a new activation function StepReLU which is equivalent to IF neurons. This enables our SNNs to achieve a high accuracy close to that of ANNs at only one time step. In addition, we also use the network structure of RNNs and introduce a leakage mechanism, which makes our SNNs highly robust while maintaining high accuracy. The contributions of this work are summarized as follows:

• We reanalyzed the equivalence between IF neurons and ReLU activation function, and proposed a StepReLU activation function more suitable for spiking neural networks on the basis of discrete spike signals and membrane potential encoding. This allows SNNs to achieve state-of-the-art accuracy of ANNs over only one time step.
• We use the RNNs' network structure and introduce a leakage mechanism, so that the IF neurons in the last layer of the network have the same effect as the Leaky Integrate-and-Fire (LIF) neurons. This makes our model more robust and more tolerant to noisy signals.
• We conduct experiments on large datasets such as CIFAR and ImageNet, and the experimental results show that our method is effective on different network structures. The results show that our models are more robust while achieving state-of-the-art accuracy in a very short inference time.

2 RelatedWork

ANN-SNN conversion is a relatively mature method at present, which can obtain SNNs with the same accuracy as ANNs, and some of the latest works have also obtained models with less inference time than previous works. The ANNSNN conversion method was first proposed and used in practical work by Cao et al. [2015b]. Then data-based and modelbased normalization was proposed by Diehl et al. [2015].

Under firing rate coding, soft-reset mechanism (also called reset-by-subtraction mechanism) is proposed to solve the information loss problem caused by hard-reset mechanism [Rueckauer et al., 2016; Diehl et al., 2016]. Layer-based and neuron-based parameter normalization are proposed to solve the problem of different thresholds of neurons in different layers of SNNs, so that deeper networks can be realized [Sengupta et al., 2019b; Ding et al., 2021; Li et al., 2021]. Rueckauer et al. [2017] indicated that, after selecting appropriate parameters, we can essentially eliminate clipping errors in ANN-SNN conversion through parameter normalization. After that, a lot of following works are studying how to choose a more appropriate normalization scale. SpikingYOLO with channel-wise data-based normalization was proposed by Kim et al. [2020] to solve the problem of fast object detection in SNNs.

There are also some new neuron models proposed. In order to eliminate the quantization error in ANN-SNN conversion, Bu et al. [2021b] proposed the quantization clip-floorshift activation function under average membrane potential encoding. Wang et al. [2022] proposed a neuron-wise parameter normalization method and a signed neuron with memory function under membrane potential encoding to eliminate ANN-SNN conversion errors, and their method achieved state-of-the-art results on multiple datasets. In order to solve the problems of high energy consumption and large time latency of traditional methods, Li and Zeng [2022] proposed a neuron model for releasing burst spikes.

In recent years, people have begun to pay more attention to the inference time of SNNs, and some works have made considerable progress. Deng and Gu [2021], Li et al. [2021], Meng et al. [2022] and Li and Zeng [2022] all achieved excellent results with 16 and 32 timesteps. Bu et al. [2021b] achieved excellent inference accuracy within 10 time steps or even 2 time steps. Besides, Massa et al. [2020] and Singh et al. [2021] deployed converted SNNs on the Loihi Neuromorphic Processor [Davies et al., 2018] and evaluated the performance. Back-Propagation Through Time (BPTT) typically achieves very short inference time [Wu et al., 2019; Lee et al., 2016; Lee et al., 2020], but results in high computational consumption. On the other hand, BPTT is only suitable for simple datasets such as MNIST and CIFAR10 [Kheradpisheh and Masquelier, 2020; Zhang and Li, 2020]. Rathi et al. [2020] initialized SNNs with conversion method and tuned SNNs with STDP to get a shorter inference time. Li et al. [2022b] proposed a new neuromorphic data augmentation method to improve the inference speed of SNNs.

3 Method

In this section, we use mathematical formulations to describe the ANN-SNN conversion process of our method. Based on the reanalysis of ANN-SNN conversion errors, we propose a StepReLU activation function to train ANNs with membrane potential encoding. Then we solve the neurodynamic equations of LIF neuron model to obtain the conditions that the leakage mechanism needs to fulfill, and introduce the leakage mechanism into SNNs. Further, we propose a new ANNSNN conversion method based on the StepReLU function.

3.1 NeuronModel

Neuron model for ANNs. For ANNs, the activity of neurons can be simplified to the following formula:

where the vector al represents the outputs of all neurons in l-th layer, W l is the weight matrix between layer l and layer l - 1, the vector bl is a bias item, which also represents the thresholds of each neuron in layer l, and ReLU is a common activation function in ANNs.

Neuron model for SNNs. Similar to previous works [Han et al., 2020], we use the Integrate-and-Fire (IF) neuron model to describe the ANN-SNN conversion process, because the IF neuron is mathematically equivalent to the ReLU activation function. At each time step t, the IF neurons in each layer receive the inputs from the neurons in the previous layer and emit spikes as outputs according to certain conditions. This process can be expressed by Eq. (2):

where the vector vl (t) is the membrane potential of layer l at time step t, vector ml (t) is an intermediate variable for recording the membrane potential and W lxl-1 can be regarded as the inputs received by neurons in layer l. For IF neurons, when the membrane potential of the i-th neuron in layer l exceeds the threshold θl , the neuron will fire a spike sli (t) to neurons in the next layer. Since we used membrane potential encoding instead of firing rate encoding, sli (t) can be expressed as Eq. (3):

here γ l is a spike parameter that we can set, which is caused by different excitation thresholds of neurons in different layers. In ANNs, the minimum of γ l can be 0, then Eq. (3) becomes the ReLU activation function. However, in SNNs, we noticed that as long as a neuron fires a spike, the output (membrane potential) of the neuron must be a positive number, and it cannot be equal to 0. Furthermore, unlike firing rate encoding, the output of neurons under membrane potential encoding can exceed 1. That is to say, γ l > 0. Based on this, we propose the StepReLU activation function, as shown in the following Eq. (4):

In this way, we can rewrite the spike firing process of a neuron in SNNs as Eq. (5):

The difference between the StepReLU activation function and the ReLU activation function is shown in Fig. 1. After firing a spike, the membrane potential of the neuron is reset. In order to preserve more information of a spike, we choose the hardreset mechanism, in which we reset the membrane potential to 0. This process can be represented by Eq. (6).

The difference between hard-reset and soft-reset is shown in Fig. 2, and the figure was made with the help of the snntorch package [Eshraghian et al., 2021]. The short black vertical lines in the upper and lower parts of the figure represent the input and output spike trains of neurons, respectively. The blue line in the middle of the figure shows how the neuron's membrane potential changes over time. The only difference between hard-reset mechanism and soft-reset mechanism is that, when the membrane potential of a neuron reaches the threshold and fires a spike, hard-reset will cause the membrane potential to return to 0 directly, while soft-reset will only reduce the membrane potential by a specific value.

Membrane potential encoding. The reasonwhy firing rate encoding is currently used more frequently is mainly because people generally regard the SNNs as binary networks, in which neurons' firing a spike represents 1, and non-firing represents 0. In order to obtain excellent performance without sacrificing inference time, we no longer use binary firing rate encoding but full-precision membrane potential encoding. Under membrane potential encoding, a spike fired by a neuron no longer represents 0 or 1, but represents the membrane potential of the neuron when the spike is fired. Then we consider the case that neuron i sends a spike to neuron j. If we take the synaptic weight between i and j as 1, the spike sent by i is consistent with the spike received by j. Although the duration of the spike signal is very short, it is still the transmission of electrical signals in essence. In other words, when the connection weight between i and j is 1, the charge of the spike sent by i is equal to the charge of the spike received by j. We know that an IF neuron is physically equivalent to a Resistance-Capacitance (R-C) circuit with a battery. Due to the difference between the circuit constants R and C, the spike signals of the same charge represent different values for different neurons under membrane potential encoding, as shown in Eq. (7):

where C represents the capacitance constant of the neurons, si represents the value of the spike signal for neuron i, xj represents the value of the spike signal for neuron j and q represents the charge of the spike. Wij is the conversion weight between neuron i and neuron j (the connection weight is 1), and is also an element of the weight matrix W in Eq. (2). In this way, we have clarified the definition of the weights between neurons under membrane potential encoding.

3.2 Conversion fromANN to SNN

Conversion error analysis. Different from the conversion error under firing rate encoding and soft reset mechanism, the conversion error generated by our method is much simpler. As can be seen from Fig. 1, in theory, when we use the StepReLU activation function to train the network, the output value of StepReLU and the output value of ReLU only differ by a constant γ l . In other words, we can use the StepReLU activation function in the training of ANNs, in which we will not introduce additional conversion errors as long as we make reasonable adjustments to the bias term. The key points are around 0. We artificially introduced and amplified the difference between spike firing and non-spike firing, which will increase the tolerance of the network to noise interference, and may also cause some activated points near 0 to not be activated. This is the only source of ANN-SNN conversion error in our proposed method. For the trade-off between conversion error and robustness, we will show it in the Experiments part. Overall, if we choose appropriate parameters, we can achieve a robustness improvement with little drop in network inference accuracy.

ANN-SNNconversion. According to ourmethod, the cores of ANN-SNN conversion are using StepReLU function to train ANNs and mapping the outputs of ANNs' neurons to the membrane potential of SNNs' neurons. The ANN-SNN conversion in one time step t can be expressed by Eq. (8)-(9):

The specific process of ANN-SNN conversion is shown in Fig. 3. In the figure, red circles and blue circles in the middle represent the neurons in ANNs and SNNs respectively, and the horizontal arrows indicate that the corresponding neurons are equivalent. Semi-transparent circles along the edges of the figure indicate other non-target neurons in the network.

Leakagemechanism. We usually think that the reasonwhy SNNs have good robustness and sparsity is that there is a leakage mechanism between different time steps. The leakage mechanism is derived from the Leaky Integrate-and-Fire (LIF) neuron model, which means that the membrane potential of neurons that do not fire a spike at the current time step will gradually decrease over time. In neurobiology, the LIF neuron model is an approximation to the Hodgkin-Huxley (HH) model, and its dynamic process can be expressed by the differential equation shown in Eq. (10):

where u(t) represents the membrane potential, Ri(t) represents the input signals, τm is the temporal constant and τm = RC. Here we take urest = 0 because the value of urest does not affect the result under the hard-reset mechanism. In addition to the differential equation in Eq. (10), the LIF neuron also satisfies the hard-reset mechanism shown in Eq. (11):

here tf represents the moment when the LIF neuron fires a spike signal, which can be determined by tf = {t|u(t) = ϑ}. And limt→t+f represents the right limit to tf . ϑ is the membrane potential threshold of the LIF neuron. When the membrane potential u(t) of the LIF neuron reaches the threshold ϑ, the neuron will fire a spike s(t), and s(t) satisfies Eq. (12):

here we have considered the fact that the membrane potential may exceed the threshold in the actual solution, and we have already used the membrane potential code to describe the spike. The above three equations completely describe all the activities of a LIF neuron, and the solution to the differential equation in Eq. (10)-(12) can be written as:

where S(t) = sf1(t) + sf2(t) + · · · represents the output sequence and i(t) represents the input sequence, κ(s) represents the parameter related to leakage and η(s) represents the parameter related to firing sequence. Since the leakage only occurs in the input sequence, we neglect the firing item ∫ ∞ 0 η(s)S(t-s)ds and discretize the convolution of i(s) and κ(s) in Eq. (13) to get:

In Eq. (14), we have assumed that the neurons in the l-th layer do not send spikes to the next layer within T time steps. Obviously, this assumption is true for the last layer of neurons in the network. Based on this, we introduce a leakage mechanism on the neurons of the last layer (also called the inference layer) of the Spiking Neural Networks, hoping to make the entire networks more robust in a few time steps. In this way, the SNNs network structure we finally converted from ANNs is shown in Fig. 4. In the figure, similar to the structure of RNNs, the vertical arrows indicate the transmission direction of spike signals, and the horizontal arrows indicate the time steps. At each time step from 1 to T, the spike signals are transmitted along the direction of the network and the outputs are obtained according to Eq. (8) and Eq. (9), and then these outputs are summed along the time direction according to Eq. (14) to obtain the final output.

4 Experiments

In order to verify the effectiveness of our method, we use VGG and ResNet network structures to conduct experiments on CIFAR10 [LeCun et al., 1998], CIFAR100 1 [Krizhevsky, 2009] and ImageNet 2 [Deng et al., 2009] datasets, and compare with other state-of-the-art results on image classification tasks. In the experiment, we use the Kaiming normal initialization method to initialize the ANNs network parameters, and use the SGD optimizer with 0.9 momentum. The L2 penalty with a value of 5e-4 is also added. In addition, we set an initial learning rate of 0.1 and update the learning rate with the CosineAnnealingLR strategy. The training epoch is 300 and the batch size is 128. Specific to a single experiment, we will make targeted adjustments to various parameters.

In this part, we verify our method from two aspects, one is the inference accuracy of ANN-SNN conversion, and the other is the robustness of the converted SNNs, which is brought by StepReLU activation function and leakage mechanism respectively. Our code can be seen at https://github. com/QuelThalasGrace/aNewANN-SNNConversionMethod.

4.1 ANN-SNNConversion in One Time Step

Following the aforementioned method, we use the StepReLU activation function to train an ANN and convert it to an SNN at one time step. We test the performance of the converted SNN on different datasets, and the experimental results are shown in Table 1. Here, we compare our method with the state-of-the-art ANN-SNN conversion methods, including Quantization Clip-floor-shift Activation(QCA) from Bu et al. [2021b], Burst+LIPooling(BLIP) from Li and Zeng [2022], SNN Conversion with Advanced Pipeline(SNNC-AP) from Li et al. [2021], Signed Neuron with Memory and NeuronWise Normalization(SNMNN) fromWang et al. [2022] and Differentiation on Spike Representation(DSR) from Meng et al. [2022]. Experimental results show that our method can make the performance of SNNs close to that of ANNs in only one time step, and the optimal accuracy of transformed SNNs depends on the accuracy of ANNs. In other words, when using our method for ANN-SNN conversion, the better the initial performance of the ANNs, the better the performance that our resulting SNNs can achieve at one time step.

In addition, we also verify the robustness improvement of the network brought by the proposed StepReLU activation function. Specifically, we feed ANNs and converted SNNs with noisy inputs without introducing a leakage mechanism (in one time step), and observe the change of network's inference accuracy. Since SNNs use spike signals to transmit information, we choose to use salt and pepper noise, which is more similar to spike signals, as the experimental noise. We use salt and pepper noise signals with different signal-tonoise ratios (p=0.5) to process the input images and send them to ANNs and SNNs respectively, and get inference results. We have done the experiments on CIFAR10, CIFAR100 and ImageNet datasets, and the experimental results are shown in Fig. 5(a)-(c). The results show that as the input signal becomes more and more noisy, the inference accuracy at one time step decreases for both ANNs and converted SNNs. However, the performance decrease of converted SNNs is less than that of ANNs. Overall, the proposed StepReLU activation function can bring about 1% to 3% improvement in inference accuracy for SNNs compared to ANNs in noisy environments, when the parameter γ l is properly chosen.

1 https://www.cs.toronto.edu/∼kriz/cifar.html

2 https://image-net.org/challenges/LSVRC/2012/

4.2 ANN-SNNConversion with Leakage Mechanism

According to our derivation and analysis, both StepReLU activation function and the introduction of leakage mechanism will improve the robustness of the network. However, in this part, we pay more attention to the robustness improvement brought by introducing leakage mechanism to SNNs. We still use the salt and pepper noise signal as our experimental noise, and in order to maintain a fast inference speed, we set the time step of SNNs with the leakage mechanism to be 10. In experiments, we still feed the inputs processed with noisy signal directly into ANNs, however this is different in SNNs. Since the inference process of SNNs has ten time steps, we will feed the same input signal to the SNNs every time step, ten times in total. Considering a more realistic situation, we randomly select one of the ten identical input signals, process it with noise signal, and sequentially feed these new input signals into SNNs. Then we select an appropriate leakage parameter κ, and sum the ten output values as shown in Eq. (14) as the final output value, and the category corresponding to the neuron with the largest output value is the inference result. In this way, we observe the inference accuracy of SNNs with leakage mechanism over ten time steps, and the experimental results are shown in Fig. 5(d)-(f). The curves in these figures show the accuracy changes of ANNs and SNNs with leakage mechanism as the proportion of noise signals increases on CIFAR10, CIFAR100 and ImageNet. The results show that the leakage mechanism can greatly improve the robustness of SNNs, and the leakage mechanism makes the inference accuracy of SNNs about 31% higher than that of ANNs under noisy conditions.

4.3 Sources of Robustness Brought by Our Method

As we have verified before, both StepReLU activation function and leaky mechanism can bring robustness improvement to SNNs. Combining the derivation and experimental results, we believe that the StepReLU activation function artificially sets the difference between the activation and inactivation of neurons through the parameter γ l to achieve robustness improvement, while the leakage mechanism improves robustness by improving the fault tolerance of SNNs through the parameter κ.

For StepReLU activation function, it is more suitable for SNNs under the membrane potential encoding while retaining the characteristics of ReLU activation function to the greatest extent. As we discussed in Section 3.2, the StepReLU activation function has a jump at zero point, which of course makes some neurons that can be activated in ANNs (the membrane potential just reaches the threshold) cannot be activated in SNNs, resulting in a decrease in accuracy. But more importantly, this will also make the SNNs more tolerant to the noise signals near the zero point, because some weaker noise signals will not exceed the difference of the StepReLU function at the zero point. This point has also been verified in experiments. When we choose the appropriate parameters, the StepReLU function can make SNNs achieve the accuracy close to ANNs (this usually comes down a little bit) in one time step, and the robustness will also increase by 1%-3%, which is mainly produced under weak noise signals.

For leakage mechanism, the reason why it can bring about a huge improvement (about up to 31%) in the robustness of the network is its tolerance to noisy signals. Taking the 10 time steps we chose as an example, since the influence of the output of each time step on the final result will gradually decrease over time (leakage), this leads to the fact that only the output of the last time step will have a substantial impact on the final result. In other words, under the condition of ten time steps, when we assume that the noise signal randomly affects the input of one of the ten time steps, it will only affect the final output when the noise signal affects the input of the last time step. Theoretically, other situations will not have a great impact on the final output, which is the core reason why the leakage mechanism can bring great improvement on robustness of SNNs. If we set a larger time step, theoretically we can get more robust SNNs, but this requires a trade-off between inference time and robustness. This point has also been well verified in experiments.

5 Conclusion

In this paper, we propose a new ANN-SNN conversion method, derived from the StepReLU activation function, membrane potential encoding and leakage mechanism, which can achieve accurate, robust and low-latency inference. we reanalyze the relationship between Integrate-and-Fire (IF) neuron model and ReLU activation function, then we propose a StepReLU activation function more suitable for SNNs, and use it to train ANNs. After finishing the training, we convert the ANNs to SNNs and introduce the leakage mechanism to obtain a model with excellent performance. Our method can make SNNs achieve the inference accuracy close to ANNs in one time step and obtain a robustness improvement (about 1%-3%), and also enable SNNs to achieve a much higher robustness than ANNs (up to about 30%) in a very short inference time (10 time steps).

There are two core foundations in our method, the equivalence between the StepReLU activation function and the Integrate-and-Fire (IF) neuron model under membrane potential encoding, and the solution of the dynamic equations of the Leaky Integrate-and-Fire (LIF) neuron model. We have proved and solved the above two conclusions mathematically, and verified through experiments that our proposed method can effectively improve the performance of SNNs. Since the neurons in the inference layer of the network do not fire spikes, we ignore the output term in Eq. (13) and get Eq. (14), and only introduce a leakage mechanism in the last layer of neurons in the SNNs. In fact, Eq. (13) is valid for any LIF neuron, but since different neurons fire spikes at different moments, it is very difficult to simplify Eq. (13). This is also the core difficulty of reducing the energy consumption of SNNs and realizing asynchronous SNNs in the true sense.

Although our method has achieved excellent performance on different network structures and datasets, there are still some shortcomings that need our improvement. Essentially, the SNNs model we obtained in this work is still synchronous networks, not real asynchronous, event-driven networks, and these points are just considered to be the key to SNNs' low energy consumption. In the follow-up research, we will try to build a truly asynchronous SNNs' model, which can achieve the advantages of high accuracy, low latency, high robustness and low energy consumption at the same time.

Acknowledgements

This work is supported by the Joint Funds of the National Natural Science Foundation of China (U20A20204).

Contribution Statement

The first author and the second author contributed equally to this paper.