Decoding EEG by Visual-guided Deep Neural Networks

1 Introduction

Vision is one of the most important components in the human perception system. When eyes receive visual stimulation, neural spikes are produced in brain [Jacobs et al., 2009]. Decoding neural spikes produced in the brain due to visual stimuli is instrumental to the exploration of visual information processing mechanism of humans and to development of computer vision [Nestor et al., 2016]. With the development of contemporary machine learning methods, it becomes possible to obtain high-quality decoding results from functional magnetic resonance imaging (fMRI) and EEG.

fMRI sequences provide information from several visual cerebral areas, and play an important role in visual decoding tasks varying from categorizing to reconstruction [Haxby et al., 2001; Haynes and Rees, 2005; Miyawaki et al., 2008; Naselaris et al., 2009; Cowen et al., 2014]. In previous works, machine learning models are proposed to capture the relationship between visual stimuli and fMRI. More recently, deep learning methods have also been proposed to reconstruct images from fMRI [Horikawa and Kamitani, 2017; G¨uc¸l¨ut¨urk et al., 2017; Du et al., 2017].

∗Corresponding Author

The huge volume and high cost render fMRI based neural decoding impossible in daily life [Poldrack and Farah, 2015]. On the contrary, EEG has significant advantages in aspects of portability and price which promote the application of EEG decoding systems. For example, EEG devices with integrated machine learning algorithms are the key components in braincontrolled systems for disabled persons [Liu et al., 2017; Park et al., 2011]. In addition, deep neural networks have also been reported to perform competitively on decoding visual related EEG trials [Adamos et al., 2016; Spampinato et al., 2017; Palazzo et al., 2017]. Assuming that EEG signal exists in a cognitive domain, the workflow of existing EEG decoding methods is summarized by Fig 1., i.e.s (1) Visual stimuli are shown to subjects; (2) EEG trails are recorded; (3) EEG trials are represented by handcrafted features or neural networks for decoding. Importantly, this decoding pipeline only utilizes information in cognitive domain. However, there is an inevitable fact that state-of-the-art deep learning models outperform humans on representing and classifying images [He et al., 2016]. So, information in the related visual domain can assist to improve the performance of EEG decoding.

In this paper, we propose using visual representation to guide the decoding of EEG. Our decoding framework contains two stages: (1) Visual-guided EEG classification stage; (2) Visual-guided stimuli generation stage. In the first stage, a classification network is guided by visual representation to category EEG into the classes of related visual stimuli. Then, in the generation stage, an improved generative adversarial network (GAN) [Goodfellow et al., 2014] produces the corresponding visual stimuli from visual-guided EEG representations. Compared with state-of-the-art neural decoding models, our method achieves superior performance.

2 RelatedWork

Spatial-temporal features of EEG are used to decode stimulus-related signals into two event-related potentials (ERPs) for capturing the distribution of signal with a Gaussian mixture model (GMM) [Tzovara et al., 2012]. In a recent study [Kaneshiro et al., 2015], principal component analysis

(PCA) and linear discriminant analysis (LDA) are combined to recover fine-grained object category from EEG recordings.

Deep learning methods have become dominant in various traditional and bio-inspired computer vision tasks and bio [LeCun et al., 2015; Zhang et al., 2019]. Variations of CNN models are applied to represent EEG for detecting particular signal components [Cecotti and Graser, 2011; Plis et al., 2014]. More recently, face image sequences and EEG data are analyzed jointly for emotion detection [Soleymani et al., 2016]. Frameworks combing CNN and other deep learning structures are proposed [Bashivan et al., 2015; Jiao et al., 2018] to decode mental loads from EEG. Novel EEG-driven automated visual classification and generation methods have also been proposed [Spampinato et al., 2017; Palazzo et al., 2017]. Generally, these studies are based on the same assumption that EEG can guide deep learning models to achieve better results on image classification and generation. To the best of our knowledge, this work is the first to use visual representation to guide the decoding of visual stimulated EEG. Our study is based on the proven fact that state-of-the-art deep neural networks perform better than humans on visual representation tasks [He et al., 2016].

3 Our Method

Since our decoding framework contains an EEG classification stage and a stimuli generation stage, they will be described respectively in the following subsections.

3.1 Visual-guided EEGClassification

In the cognitive domain, EEG signals are first organized to EEG maps Xcog as [Jiao et al., 2018], axes of which stand for EEG channels and recordings of each channel (Fig 2).

As shown in Fig 3., Xcog and Xvis stand for input to representation layers in these two domains. Our classification model contains: (1) Feature representation layers (Cognitive CNN and Visual CNN) in both cognitive domain and visual domain; (2) Classification layer (Class) in cognitive domain.

The loss function of our classification model is a modified softmax loss as shown in Equation (1).

In this equation, the first term is a standard softmax loss: Sj is the softmax value which stands for probability of one instance belonging to category j, yj is the label of input, and N is the number of classes. The second term is a squared error to make the normalized cognitive representation Fcog (output of feature representation layers in the cognitive domain) be close to the normalized visual representation Fvis in the feature space. In the training process, parameters of pretrained visual CNN are fixed to render the cognitive representations more discriminative, assisting to obtain superior performance in the classification task.

3.2 Visual-guided Stimuli Generation

Since our visual generation stage is in the form of an improved GAN model, we briefly introduce the basic GAN method first. Then, we show how our improved GAN model for decoding EEG data is constructed.

GAN models use a minimax game which guides two networks (a generative net and a discriminative net) to be trained in opposite directions. In the adversarial training process, the generative net tries to map a latent space to particular data distribution, while the discriminative net discriminates between real data and generated instances. In our EEG decoding task, the main role of GAN is learning to generate stimuli from corresponding EEG representations.

Our improved GAN contains a generative net G , a discriminative net D , and two representative networks R, which are shown as Fig 4. Both G and D are convolutional neural networks which share the same structures as ones proposed in [Palazzo et al., 2017]. R are the fully convolutional networks (FCN) [Long et al., 2015] that share the same structures and parameters. Goals of each network (G, D, and R) can be categorized as follow:

(1) G takes Fcog as input for generating plausible images Xfake;

(2) D tries to distinguish Xfake from visual stimuli Xvis;

1. R assists to maintain consistency of visual components between Xfake and Xvis via lower-level visual perception
2. and higher-level visual semantics.

The discriminative loss LD is computed as Equation (2), where Fcog-w is wrongly corresponding EEG representation, randomly chosen from different classes of Xvis. The generator loss LG is shown as Equation (3). Where λ (Lper+Lsem ) is the visual-consistent term which is achieved by visual representation networks R. In this visual-consistent term, Lper is in the form of ‖ f (Xvis)-f (Xfake) ‖22 , which stands for perceptual loss [Johnson et al., 2016] between FCN feature maps (f in the first FCN layer for representing more lowlevel visual details) of Xfake and Xvis. Lsem (containing more semantic information) is the softmax cross entropy between semantic segmentation results of Xfake and Xvis after going through R. This visual-consistent term acts as auxiliary for generating higher-quality images.

Methods which are the most similar with our decoding framework are [Spampinato et al., 2017; Palazzo et al., 2017]. The main differences between our method and theirs: (1) The representation Fcog is visual-guided, whereas previous works only utilize cognitive domain information in their generation stage. Our visual-guided Fcog is more discriminative in classification stage and more diverse in generation stage, contributing to better decoding performance; (2) The visualconsistent term in our generative loss helps to maintain Xfake with more qualitative consistence when observed Xvis .

4 Datasets

Performance of our framework is evaluated and compared with state-of-the-art methods on two public datasets: (1) ImageNet subset [Spampinato et al., 2017; Kavasidis et al., 2017]; (2) Face and object [Kaneshiro et al., 2015]. Details of datasets are described in the subsections.

4.1 ImageNet Subset

This dataset is a 40-class subset of ImageNet [Deng et al., 2009]. The related classes: dog, cat, butterfly, sorrel, capuchin, elephant, panda, fish, airliner, broom, canoe, phone, mug, convertible, computer, watch, guitar, locomotive, espresso, chair, golf, piano, iron, jack, mailbag, missile, mitten, bike, tent, pajama, parachute, pool, radio, camera, gun, shoe, banana, pizza, daisy, and bolete (fungus). During the EEG acquisition experiments, 2,000 images (50 images in each class) are shown to 6 subjects. A 128-channel Brainvision EEG system is applied to record related neural signal during the process described above, and totally 12,000 visual-evoked EEG sequences are acquired. More details can be found in the related reference [Spampinato et al., 2017].

4.2 Face and Object

Visual stimuli applied in this dataset are categorized into two classes: face and object (12 images in each class). EEG trials are recorded by a 128-channel EGI system when 10 subjects viewed a sequence of images. Details of stimulating experiments and subjects are described in [Kaneshiro et al., 2015]. Meanwhile, data acquisition and preprocessing are also detailed in the related paper. Finally, a dataset of 17,281 sequenced EEG recordings is formed, of which 8,641 samples are evoked by stimuli of faces and 8,640 samples are evoked by visual stimuli of objects. Formulation of this dataset is similar to that of ImageNet subset as mentioned above.

5 Experiments and Analysis

Our decoding method can not only classify but also reconstruct visual stimuli from single-trial EEG data. So, the first goal of our experiments is to investigate whether EEG data which represent visual stimuli of different categories can be classified accurately. The other goal is to evaluate whether our method can reconstruct plausible images from related EEG recordings. The performance of classification and generation are detailed in the following subsections respectively. Our models are based on the deep learning toolkit of TensorFlow [Abadi et al., 2016].

5.1 Performance of Classification

Both objective and subjective evaluations are applied to compare classification performance of our method with stateof-the-art ones. The objective evaluation criterion is classification accuracy, while the subjective one is t-distributed stochastic neighbor embedding (t-SNE) [Maaten and Hinton, 2008]. t-SNE can project high-dimensional data into a 2-D scatter plot which is widely used for evaluating discriminative ability of feature vectors. For t-SNE visualization, the evaluation criterion is: the more instances of one class can be separated from ones of other classes, the better the related features perform.

We performed classification experiments under the same condition as described in related references [Spampinato et al., 2017; Kaneshiro et al., 2015]. Training strategy of our classification network is Adam. The independent separations of EEG signal datasets are 80% for training, 10% for validation, and 10 % for testing. Structure of our EEG classification net in cognitive domain is the same form as that of AlexNet [Krizhevsky et al., 2012], except for numbers of neurons in the classification layer (40 for ImageNet subset dataset, and 2 for face and object dataset). Data augmentation methods and network training strategies are drawn from [Jiao et al., 2018]. Pretrained networks in the visual domain are AlexNet [Krizhevsky et al., 2012], VGG16 [Simonyan and Zisserman, 2014], ResNet50, and ResNet101 [He et al., 2016], which are used to obtain visual representation to guide EEG decoding.

For ImageNet subset dataset, classification accuracy of different methods are listed in Table 1. LDA and long short term memory (LSTM) are state-of-the-art methods [Kaneshiro et al., 2015; Spampinato et al., 2017]. CNN is our EEG classification net which is not guided by visual representation. CNNAlex, CNN-VGG16, CNN-ResNet50, and CNN-RestNet101 stand for classification performance guided by different pretrained visual networks. Results listed in this table show that our visual-guided frameworks outperform LDA and LSTM, among which the ResNet101 guided classification method achieves a new state-of-the-art result, with our method improving the performance of the EEG classification stage. EEG features extracted by CNN in the cognitive domain and EEG features guided by visual net are visualized by t-SNE

in Fig 5., data points with different colors representing categories of EEG features. It is obvious that the visual-guided EEG representations of different visual classes are more separable in the corresponding feature space, while instances in the same class are more uniformly distributed. These properties demonstrate superior performance of our method in handling diversity (more separable data points in feature space assist to generate diverse visual stimuli) in the generation stage. Besides, performance of visual guided model (CNNResNet101) and no visual guided CNN are compared by category in Fig 6., a positive number or negative number (superiority of classification accuracy (%)) representing respectively superior or inferior performance of our strategy compared to traditional strategies in a given category. Our visual guided strategy achieves superior classification performance among most of the categories, except for Airliner, Folding chair, Mailbag, Radio telescope, Revolver, and Running shoe, which are attributable to the same synsets (categories of some image classes) of Artifact in ImageNet dataset. We hypothesize that this illustrates that human knowledge in cognitive domain keeps the advantage on classifying visual stimuli of man-made objects.

We also compare the performance of our CNN and ResNet101 guided CNN with LDA and LSTM on face and object dataset. The related objective results are shown in Table 2, and the subjective results are illustrated in Fig 7. It is obvious to see that our method also performs better than the non-visual guided models on this dataset.

5.2 Performance of Generation

The subjective visual quality of generated images is usually chosen as an evaluation criterion for generative models. In addition, inception score and inception classification accuracy [Palazzo et al., 2017; Kavasidis et al., 2017] are widely used quantitative evaluations for GANs (higher values of inception score and inception classification accuracy stand for a superior performance of generation). We also use these methods to evaluate performance of our generation stage. Since the number of images in face and object dataset is very small, it does not meet the requirement of diversity for training an effective GAN model [Goodfellow et al., 2014]. So, generation performance of both previous works [Spampinato et al., 2017; Palazzo et al., 2017; Kavasidis et al., 2017] and our method are discussed just on ImageNet subset dataset.

Training strategies for our improved GAN model mainly follow those in [Palazzo et al., 2017]. Parameters of D and G are listed in Table 3 and Table 4, in which Conv and Deconv stand for convolution and deconvolution blocks with ReLU

EEG-Res152

activation and batch normalization operation, and Fc is fullyconnected layer. We choose the best-performed visual-guided representation (guided by ResNet101) as Fcog in generation stage. FCN for the visual-consistent term λ (Lper + Lsem) is pretrained on VOC2012 dataset [Everingham et al., 2010], and λ is set to 0.5. Since only limited images in the dataset are labeled by EEG, previous works initialize training of GAN with random noise Z and unlabeled images (50,000 unlabeled images in ImageNet). In our work, visual features Fvis of 50,000 images can be obtained from visual networks, and they are applied in the pretrain stage of GAN model. Then, training of GAN is refined by the visual guided representation Fcog .

In previous papers, high-quality results of three classes (Airliner, Jack-oa-Lantern, and Panda) and low-quality results of other three classes (Banana, Capuchin, and Bolete) are listed for subjective evaluation. We follow this evaluation strategy and list generated instances of different methods in Fig 8 and Fig 9. In these figures, the top rows named EEGGAN are the results generated by state-of-the-art method [Palazzo et al., 2017; Kavasidis et al., 2017], while the bottom ones are generated by our visual-guided GAN with the visual-consistent term. Importantly, images generated by our method contain more visual details and they correspond better with related classes. The subjective visual quality of our results on all these classes are superior to the state-of-theart one. This phenomenon demonstrates the efficiency of our proposed decoding method in visual stimuli generation stage.

Inception scores and inception classification accuracy of different methods are listed in Table 5, where EEG-GAN stands for the method (state-of-the-art EEG decoding model) proposed in [Palazzo et al., 2017; Kavasidis et al., 2017], VG-GAN stands for our visual-guided GAN without visualconsistent term, VG-GAN-VC represents our visual-guided GAN with visual-consistent term. Results in this table demonstrate that our visual-guided method achieves better results in both these two aspects. In addition, the visualconsistent term is efficient to further improve the objective visual quality of generated images for obtaining better decoding performance.

6 Conclusion

Deep learning methods have achieved significant improvements on neural decoding tasks. Inspired by the representative studies of other researchers, we propose a visual-guided decoding framework for EEG data in this paper. To obtain superior decoding results, our work takes full advantage of visual representations which are obtained from state-of-theart deep learning models on computer vision tasks. The proposed framework contains a visual-guided EEG classification stage and a visual-guided generation stage. In the classification stage, visual-guided EEG representations bridge the gap between cognitive domain and visual domain for categorizing EEG recordings evoked by different visual stimuli more accurately. In the visual-guided visual stimuli generation stage, the visual-guide EEG representation can also improve the performance of generation. Besides, our improved GAN model can improve consistency between visual representation of real stimuli and generated instances to further improve the subjective and objective quality of generated images. However, as the per-class classification results shown in Fig 6., human can achieve higher classification accuracy in certain classes. This phenomenon shows opportunities to combine the advantages of both human brain and deep learning models to improve the performance. In the future, we will extend the simultaneous and synergistic utilization of both visual information and cognitive knowledge to further improving neural decoding tasks, thus augmenting the areas such as hybrid intelligence and human-AI collaboration.

Acknowledgments

In this research, we perform the deep learning experiments on a Titan V graphics card provided by the NVIDIA Academic Program of GPU Grant Program.