Are Spiking Neural Networks Useful for Classifying and Early Recognition of Spatio-Temporal Patterns?

1 Introduction

This research is motivated by two artificial intelligence problems that rely on an autonomous system's ability to encode and recognize spatio-temporal patterns: intent recognition and imitation learning. In both domains, the activities observed by the autonomous system contain both spatial and temporal information. Spike timing neural networks (SNNs) are suitable to model spatio-temporal patterns. In this work we propose a method based on SNNs that addresses the problem of learning, recognizing and early classifying spatiotemporal patterns, which are typically encountered when representing gestures or other human actions.

⇤Thanks to my supervisor Dr. Monica Nicolescu for all her guidance and support.

2 Approach and Results

We use a spiking network which consists of cortical spiking neurons with axonal conduction delays [Izhikevich, 2006]. Neurons in this network are connected to 10% other neurons with a 2D Guassian distribution. The first step in our proposed approach is mapping spatio-temporal patterns (such as hand-drawn digits (images) or gestures) to the neurons in our spiking network. This is achievable by encoding spatiotemporal patterns into neural spike trains. In the training phase, the spike trains correspond to training data that are being used for stimulating the neural network. Given that our dataset is a set of images, in order to map a pattern into a spike train, we assign one neuron to each pixel. The order of firing the neurons is based on the temporal information of the pattern being drawn. Given that our dataset is a set of human hand's gestures, the spatio-temporal patterns are mapped to the spike trains by assigning groups of 5 neurons to different orientations, which correspond to movements of a human hand while showing a pattern. The neurons are fired in the order of observed orientations within the pattern. No labels for training samples have been provided to the network, as our approach is unsupervised. The synaptic weights in our spiking network are updated during training using the spiketiming dependent plasticity rule (STDP). Second, after the network is fully trained, each training pattern is presented to the network and network's response is used to create a model that corresponds to that pattern. In this way we have one model corresponding to each training sample. These models are being used in the classification phase later on. Third, in the classification phase, for classifying an unseen spatiotemporal pattern, we map it into a spike train and then create a model that represents the network's response to that pattern. The classification decision is made based on the comparison between the testing model and all the training models and selecting the closest training sample model to the input model.

In contrast with all other types of neural networks, a spiking neural network does not have any output layer. After presenting one input pattern (training or testing) to the network, the network's overall behaviour is considered as the output. We considered three different responses of spiking neural networks to the patterns. These responses are 1) an unordered set of polychronous neural groups (PNG), which encode stereotypical time-locked firing patterns, 2) a string of all PNGs being activated by the pattern (sequence of PNGs - order is important) and 3) a string of "characters", in which each character is a set of neurons that fired at a particular time step. For classification, the model for each input pattern is compared to all training models. We defined different comparison methods for models generated from spiking networks. For unordered set of PNGs we use Jaccard index (Unordered PNGs-Jaccard), for sequence of PNGs we use longest common subsequence (LCS) (ordered PNGs-LCS), for string of characters we use a specified combination of Jaccard and LCS (characters-Jaccard-LCS). All approaches enable early detection of patterns.

The classifier approach for unordered PNGs-Jaccard models consists of a parallelized CUDA implementation of a spike-timing neural network. This work provides a realtime classification of spatio-temporal patterns with the ability to provide early predictions of the patterns before their completion. This approach is very well suited for real time and human-robot interaction applications. We also obtained promising results with testing our approach on a PR2 robot.

We evaluated the performance of our approaches by applying them on two data sets of hand-drawn digits and human hand gestures. In the hand gesture dataset there are a

set of variable sized gestures representing the digits from 0 to 9, extracted from video data of a human drawing the corresponding digits. There are 7 and 21 samples per each digit for training and testing phase respectively. As shown in the left part of Figure 1 although there was relatively low variation in both scale and translation in the training data, our approach was able to identify patterns correctly at different scales and locations in the field of view (middle part of Figure 1).

The hand-written digit dataset contains grayscale 16 by 16 images of hand-written digits from 0 to 9. As the right part of Figure 1 shows, the pixels with intensity values different than white are a part of the written digit. Pixel intensity values represent the relative temporal information: pixels with lower intensities are drawn before pixels with higher intensity values. There are 5 and 50 samples per each digit for the training and the testing phase respectively. For the dataset of hand-drawn digits the unordered-PNGs-Jaccard [Rekabdar et al., 2015a] and the characters-Jaccard-LCS [Rekabdar et al., 2016] approaches work very well and for the dataset of hand gesture, the unordered-PNGs-Jaccard [Rekabdar et al., 2015c] and the ordered-PNGs-Jaccard [Rekabdar et al., 2015b] methods are performing better. Table 2 shows the overall accuracy of aforementioned approaches for both datasets. We also tested three supervised standard pattern classification methods including, SVM, LR and ENN for both datasets for comparison with our unsupervised approaches. Table 1 shows that our system outperformed and in some cases, provided comparable results.

3 Future Direction

In the future I would like to apply my approach on more HRI applications using PR2 robot and focus on on-line learning. 1