Faces, locations, and tools: a proposed two-stimulus P300 brain computer interface

Objective. Brain computer interface (BCI) technology can be important for those unable to communicate due to loss of muscle control. Given that the P300 Speller provides a relatively slow rate of communication, highly accurate classification is of great importance. Previous studies have shown that alternative stimuli (e.g. faces) can improve BCI speed and accuracy. The present study uses two new alternative stimuli, locations and graspable tools. Functional MRI studies have shown that images of familiar locations produce brain responses in the parahippocampal place area and graspable tools produce brain responses in premotor cortex. Approach. The current studies show that location and tool stimuli produce unique and discriminable brain responses that can be used to improve offline classification accuracy. Experiment 1 presented face stimuli and location stimuli and Experiment 2 presented location and tool stimuli. Main results. In both experiments, offline results showed that a stimulus specific classifier provided higher accuracy, speed, and bit rate. Significance. This study was used to provide preliminary offline support for using unique stimuli to improve speed and accuracy of the P300 Speller. Additional experiments should be conducted to examine the online efficacy of this novel paradigm.

In recent years there has been research investigating the flashing of alternative stimuli, such as images of familiar faces, instead of matrix characters themselves (Kaufmann et al 2011, Zhang et al 2012, Kaufmann et al 2013, Kaufmann and Kubler 2014, Geronimo and Simmons 2017, Kellicut-Jones and Sellers 2018. This method of altering stimulus presentation is used to evoke different ERPs in addition to the P300 component. For example, two negative components, N170 and N400, have been shown to occur when participants recognize and process facial information. The N170 occurs in response to observing faces at lateral temporal electrode positions and occurs approximately 170 ms following stimulus presentation (Bentin 1996, Eimer 2000. The N400 occurs approximately between 200 ms and 600 ms (Kutas and Federmeier 2011) over the right hemisphere electrode positions. Both of these components have been observed using unaltered facial images, inverted facial images, and even line drawings of faces (Jin et al 2014, Chen et al 2015, Geronimo and Simmons 2017. Kaufmann et al (2011) first implemented faces as P300 Speller stimuli. It was proposed that components elicited by facial stimuli provide additional ERP information to augment the P300 ERP. Thus, the paradigm should increase signal-tonoise ratio through the addition of the N170 and N400 ERP components. The additional information would create a more robust and detectable response, resulting in improved overall P300 Speller performance. Kaufmann et al (2011) superimposed the familiar, famous image of Albert Einstein sticking his tongue out over characters within the BCI matrix. In each sequence of character flashes, the image itself would flash over the characters in the matrix, as opposed to the matrix characters themselves flashing. Kaufmann and Kubler (2014) introduced a paradigm that implemented a simultaneous presentation of two very different stimuli in the four quadrants of the matrix. The image of Einstein was presented in the top left and bottom right quadrants, and a yin-yang symbol was presented in the top right and bottom left quadrants. The two-stimulus presentation was compared to the standard row-column. The results showed that the two-stimulus paradigm was able to make selections more quickly than the one stimulus paradigm, despite a decrease in accuracy. This suggests that a two-stimulus paradigm could increase speed compared to the more common single stimulus paradigm.

Facial fusiform area and parahippocampal place area (PPA)
In addition to EEG research, several different neuroimaging techniques, such as positron emission tomography (PET) and functional magnetic resonance imaging (fMRI), have supported the idea that recognition and perception of different types of stimuli elicit different cognitive responses. There has been sufficient evidence to indicate that the processing of facial stimuli and object stimuli, activate distinct brain regions (Kanwisher et al 1997). The fusiform face area (FFA), which is comprised of the region in the mid-fusiform gyrus, is shown to be strongly activated by the viewing of faces compared to the viewing of objects (Haxby et al 1991, Sergent et al 1992, Kanwisher et al 1997, McCarthy et al 1997, Tong et al 1998. While neuroimaging studies indicate that the FFA responds selectively to facial stimuli, research has also demonstrated activation to images of buildings and scenes depicting locations in space in the parahippocampal gyrus referred to as the PPA (Agguire et al 1998, Epstein et al 1999, Malach et al 2002. The PPA, located in the ventromedial surface of the temporal lobe, has been shown to respond selectively to houses and places, but not to objects or faces (Epstein and Kanwisher 1998). The strongest activation of the PPA was shown to occur in response to the viewing of complete images or photographs of scenes that depicted places, or even in images that showed empty landscapes with few discrete objects. Epstein et al (1999) suggests that the spatial layout information of a scene may be itself be enough to activate the PPA, as the PPA may play a role in perceptual coding.

Present study
The current study was used to determine if two-stimulus paradigms can increase P300 Speller performance. In a two -stimulus paradigm, it could be beneficial to utilize a classifier for each stimulus type. For example, one classifier would be specific to one type of stimulus (e.g. face) and another classifier would be specific to another type of stimulus (e.g. location). The classifiers would compete in a 'race' to determine which stimulus type is the desired choice. Having two stimulus-specific classifiers operating simultaneously could potentially discriminate the distinct features produced by each unique stimulus. Presently, a simultaneous two -classifier paradigm has not been developed and the current study uses an offline analysis conducted on data collected from an ablebodied sample to provide evidence to support the need for the development of a simultaneous two-classifier system.
The study consisted of two experiments. In Experiment 1, faces and locations were used as stimuli. We hypothesized that a facial classifier would produce higher performance when applied to the facial stimuli, and a location classifier would produce higher performance to the location stimuli. In contrast, when each classifier is applied to the different class of stimuli performance would be reduced. In Experiment 2, locations and graspable object stimuli were used as stimuli. In this case it was hypothesized that the unique spatially distant locations activated by the two types of stimuli would result in more distinct ERPs, which could further increase performance over the performance observed in the face-location stimuli used in Experiment 1.
These experiments may provide a rationale for how and why a two-stimulus paradigm may be effective, and may also provide further evidence that the P300 Speller may detect features that are specific to very different types of stimuli. Both experiments consisted of two phases. Phase I was used to obtain training data. Phase II was conducted online and the stimulus specific classifiers were applied. In other words, the classifier for the face stimuli was only used to classify face stimuli and the location classifier was only used to classify location stimuli. Subsequent offline analyses applied each classifier to the opposite type of stimulus (e.g. face classifier/location stimuli and location classifier/face stimuli). Accuracy, selections per minute, and bit rate were calculated to indicate whether each stimulus specific classifier would lead to increased performance when presenting corresponding stimuli on the P300 Speller matrix.

Experiment 1
Two stimuli producing distinct ERPs could be used to improve upon current BCI classification methods. For example, a matrix presenting two types of images, which convey different types of information could be created for each stimulus. If simultaneous dual-classifiers were created and implemented, unique classifiers could potentially be used to identify target characters, based on the notion that these images produce different enough ERPs for the BCI to detect. By superimposing an image of a face on half of the matrix characters, and an image of a location on the other half of the characters, two different classifiers could be made specifically for each image type. One classifier could detect the face-specific ERPs and the other classifier could detect location-specific ERPs. If such a simultaneous dual-classifier was developed, the BCI system could potentially discriminate targets from non-targets more quickly by eliminating half of the characters in the matrix as potential targets. This is the first step in providing a rationale for developing a simultaneous dual-classifier.

Participants
Ten able-bodied participants (four men, six women; age range 19-31) were recruited from East Tennessee State University. Four of the participants had prior BCI experience; all of the other participants were naïve to BCI use. The study was approved by the East Tennessee State Institutional Review Board and each participant gave informed consent.

Data acquisition and processing
Electroencephalograph (EEG) was recorded using a cap (Electro-Cap International, Inc.) embedded with 32 tin electrodes. Only eight electrodes were used for online classification. The eight electrodes were subject-specific and determined by the jumpwise algorithm (Colwell et al 2014). The EEG was digitized at 256 Hz and bandpass-filtered to [0.5 Hz, 30 Hz] by two 16-channel g.tec g.USBamp amplifiers, before the classification coefficients were derived the data were downsampled to 20 Hz. Data collection and stimulus presentation was performed by the BCI2000 open-source software suite (Schalk et al 2004). Before the session, the impedance of each channel was reduced to below 40 kΩ. Participants were seated approximately 90 cm away from a computer monitor that displayed an 8 × 9 matrix of letters and numbers.

Classification
The classification technique known as Stepwise Linear Discriminate Analysis (SWLDA) as described by Draper and Smith (1981) is a commonly used method to determine classification coefficients, which has been shown to be an efficient method of classification for BCI research (Farwell and Donchin 1988, Sellers and Donchin 2006, Colwell et al 2014. To improve upon classification performance, multiple electrodes at various locations distributed over the scalp are used. A filter method known as jumpwise selection is used to improve upon classification through optimal channel selection (Colwell et al 2014). Jumpwise selection uses a variant of SWLDA that selects electrodes instead of electrode specific features. The advantage of jumpwise selection is that it reduces to the feature space to a unique set of electrode locations that are optimized for each individual participant. Once the eight electrodes that account for the most variance are selected, a SWLDA analysis is conducted on the eight electrodes to determine the spatio-temporal features that account for the most unique variance.

Experiment stimuli, procedure, and design
Two types of images were used. The face stimulus was the famous image of Albert Einstein sticking out his tongue; the image has been used in previous BCI studies (Kaufmann et al 2011, 2013, Kaufmann and Kubler 2014. The location stimulus was an image of the White House. The White House image was used because it includes a familiar famous landmark and additional landscape information. Each participant completed one experimental session consisting of two calibration phases and one copy spelling phase. Participants were fitted for an electrode cap, then an 8 × 9 matrix of letters and characters was presented on the computer monitor. For the calibration phase, participants were asked to focus their attention on a specific character in the matrix and count how many times it changed to one of the two images. For example, as shown in figure 1, the top left side of the display would show a word (e.g. WORDS) and the letter they should attend to is shown in parentheses at the end of the word. After a predetermined amount of flashes of each character (in this case 14) the matrix would stop flashing. After a 4 s pause the letter in parentheses would change to the next letter in the word (e.g. (O)).
The session consisted of two calibration phases, counter-balanced, in which the participant made selections from a matrix presenting only the face image (figure 1(a)) or only the location image ( figure 1(b)). Each participant spelled three six-letter words, 18 total characters, for each calibration. Following each calibration phase (i.e. training data collection), the jumpwise channel selection algorithm and a subsequent SWLDA analysis were conducted to derive channel specific classification coefficients for each stimulus.
Following calibration, participants completed an online copy-spelling task. The matrix presented face images over half of the matrix characters and location images over the other half of the matrix characters (figure 1(c)). In the online phase, 14 flashes of each character in the matrix were presented, corre sponding to the calibration phase of the experiment. There were two conditions, counter-balanced. In each condition, 18 character selections were made. In one condition, the face classifier was applied and the 18 targets were facial stimuli. In the other condition, the location classifier was applied and the 18 targets were location stimuli. Although we were primarily interested in the offline performance, it was necessary to provide the participants with feedback; thus, we used the congruent stimulus classifier in phase II of the experiment. Afterward, offline analyses were conducted to examine how well the face classifier performed when applied to the location data, and how well the location classifier performed when applied to the face data.

Offline analysis
The offline analysis was used to determine how many flashes would be necessary for classification. In the online phase of the experiment 14 flashes of each target were presented before the classification decision was made. This number of flashes is sufficiently high to produce a ceiling effect for accuracy. The offline analysis simulated the number of flashes necessary to make an accurate response. The SWLDA classification coefficients were applied to every character after each flash of the matrix and the number of flashes necessary to make an accurate selection was calculated. If an accurate selection was not made after 14 flashes, the selection was marked as inaccurate. There was not a cross-validation procedure; in contrast to a cross-validation procedure, the coefficients were applied to each flash as it was presented in the online phase of the experiment. With each flash of a stimulus, the mean ERP for each specific stimulus was updated. Therefore, the number of flashes varied from one character selection to the next.

Statistical analyses
A two-way repeated measures analysis of variance (ANOVA) was used to examine the effects of classifier type (Face or Location) and the effects of stimulus type (Face or Location). Analyses were performed on predicted accuracy, target flashes, selections per minute, and bitrate. Offline accuracy is expressed as the percentage of correctly selected characters. Offline selections per minute are the estimated number of correct character selections made in 1 min. Offline, predicted bitrate is calculated using the formula described by Wolpaw et al (2002): Paired sampled t-tests were used to examine the differences in waveforms between the face and location stimuli. To control the false positive error rate for performing multiple compariso ns, the Benjamini-Hochberg (B-H) procedure was used to determine the critical p value (Benjamini and Hochberg 1995). Waveform analyses were conducted on the calibration data to maintain a consistent amount of data in each condition.

Offline performance
The means and standard deviations examining offline accuracy, target flashes, selections per minute, and bitrate produced by each classifier type applied to each stimulus type are shown in table 1. Electrode locations used in the jumpwise-SWLDA classification algorithm are shown in table 2. The table shows the locations that were used by at least fifty percent of the participants. The ANOVA examining offline accuracy indicated no significant differences between the four conditions F (3, 27) = 1.48, p = .240. The interactions for the ANOVAs examining offline target flashes produced by each classifier type applied to each stimulus type, offline selections per minute, and bit rate were all significant (F (3, 27) = 3.059, p = .045; F (3, 27) = 3.619, p = .026; and, F (3, 27) = 3.992, p = .018, respectively). Post hoc tests indicated no significant simple effects in any of the three ANOVAs. Thus, indicating cross-over interactions where the face classifier performed better on face stimuli than it performed on location stimuli and the location classifier performed better on location stimuli than on face stimuli.

Waveforms
To determine whether differences in performance may be due to differences in ERPs produced by the two different stimulus types, a paired samples-t-test was used to examine ERPs. Figure 2 represents the waveforms averaged across all participants for each of the four conditions. The specified time windows examined for positive amplitudes and latencies were set to 150-320 ms and 350-550 ms, as well as 128-195 ms for the N170 component, and 191-300 ms for the N400 component. The time windows were determined by examination of the grand mean waveforms. Four electrode locations Cz, Pz, PO7, and PO8 were examined. Before producing the waveforms data were down-sampled to 20 Hz and a moving average of 12 samples was applied. The waveforms were not baseline corrected.
In the positive time window of 150-320 ms, the amplitude at electrode Cz was significantly higher in response to the face stimulus (M = 5.9, SD = 1.2) than to the location stimulus (M = 2.0, Sd = 1.6), t(9) = 7.642, p < .001 (B-H critical value 0.001 315 789). Comparison of responses at electrode location Pz also indicated significantly higher amplitude in response to the face stimulus (M = 5.9, SD = 1.6) than to the location stimulus (M = 2.2, SD = 0.3), t(9) = 6.059, Statistical analyses indicated no significant differences between amplitudes or latencies produced by either condition during time window 128-195 ms at any of the four electrode locations.

Experiment 1 discussion
Offline analyses were used to provide evidence that a stimulus specific classifier for each stimulus type results in better offline BCI performance. The interactions between classifier type (Location versus Face) and data type (Location versus Face) were significant for number of target flashes, number of selections, and bit rate. Thus, evidence indicates that using two independent classifiers, one for each stimulus type, could eliminate half of characters in the matrix as potential selections.
The ERPs produced by the two stimuli differed in the amplitudes at the first positive window for electrodes Pz and Cz. The only significant difference in negative amplitude was at the second negative time window at electrode location PO8. These findings are consistent with our previous findings (Kellicut-Jones et al 2018) and the findings of Kaufmann et al (2011Kaufmann et al ( , 2013Kaufmann et al ( , 2014.

Experiment 2
Experiment 1 examined the use of two types of stimuli, face and location, which have been shown to produce distinct ERPs. This was done with the intent to provide a rationale for the development of classifiers that identify specific stimuli in a two-stimulus matrix presentation. To further investigate two-stimulus paradigms, Experiment 2 examined a third type of stimuli (i.e. graspable objects) to determine if they would produce distinctly different ERPs from location stimuli.

Graspable objects as stimuli: tools
The results of Experiment 1 provide support for using a twostimulus, two-classifier paradigm. The main hypothesis was that these stimuli would produce significantly different N170 and N400 components. However, this result was not observed. The observed differences were in the P300 component. The rationale behind using face stimuli was based on previous EEG studies, as well as neuroimaging evidence identifying facial processing in the FFA. Similarly, location stimuli were chosen due to neuroimaging evidence showing distinct activation in the PPA. Functional MRI studies can discriminate the FFA from the PPA due to the high spatial resolution produced by MRI. In our study, we hypothesized that these differences would also be observed in the scalp recorded EEG. We expect this result was not observed due to the close proximity of the PPA and FFA. The PPA is located at the medial portion of the fusiform gyrus, whereas the FFA is located at a nearby cortical region in the mid-fusiform gyrus. Therefore, Experiment 2 examined another possible stimulus, images of tools, which activate more frontal areas such as the premotor and motor cortex.
Neuroimaging studies have shown a unique cognitive response to graspable objects such as tools (Creem-Regehr and Lee 2005). Tools are considered a unique class of objects, due to the relationship between object recognition as well as the potential actions that can be performed with the object (Handy et al 2003). Viewing images of tools has been shown to activate the premotor cortex, and research has suggested that the priming of visual systems by viewing tools also primes motor systems (Grafton et al 1997, Tucker andEllis 2004).

Participants
Twenty-four able-bodied participants (10 men, 14 women; age range 19-42) were recruited from East Tennessee State University. Seven of the participants had prior BCI experience, the remainder of participants were naïve to BCI use. The study was approved by the East Tennessee State Institutional Review Board and each participant gave informed consent. Data acquisition, processing, and classification Data acquisition, processing, and classification were identical to Experiment 1.

Experimental stimuli, procedure, and design
The experimental protocol used in Experiment 1 was also used in Experiment 2. The two experiments differed in the images presented to participants, and the instructions given to participants for 'attending' to the stimuli used. The images were those of the location image (i.e. the White House) and a hammer. The experimental task for the location image was to focus attention on a specific character in the matrix and to count how many times the image appeared, while ignoring the images flashing over the other characters in the matrix. For the tool stimuli, the experimental task was to focus attention on a specific character in the matrix and imagine themselves using the object each time the image of the object flashes (i.e. swing a hammer). This was done to elicit a stronger response in the premotor cortex than simply counting the number of character flashes. Following each calibration phase, a SWLDA analysis derived classification coefficients specific to each stimulus type. Following calibration, participants completed an online copy-spelling task similar to the online-copy spelling task in Experiment 1 using the stimuli shown in figure 3.

Statistical analyses
A two-way repeated measures analysis of variance (ANOVA) was used to examine the effects of classifier type (Location or Tool) and stimulus type (Location or Tool) on acc uracy, number of flashes, selections per minute, and bit rate. Waveform analyses were conducted on the calibration data to maintain a consistent amount of data in each condition. Paired sampled t-tests were used to examine the differences in waveforms between the tool and location stimuli. To control the false positive error rate for performing multiple comparisons, the Benjamini-Hochberg (B-H) procedure was used to determine the critical p value (Benjamini and Hochberg 1995).

Offline performance
The means and standard deviations examining offline accuracy, target flashes, selections per minute, and bitrate produced by each classifier type applied to each stimulus type are shown in table 3. Electrode locations used in the jumpwise-SWLDA classification algorithm are shown in table 4. The table shows the locations that were used by at least fifty percent of the participants. The ANOVAs examining offline accuracy, number of target flashes, selections per minute, and bit rate all yielded significant differences (F (3, 33) = 8.42, p < .001; F (3, 33) = 22.21, p < .001; F (3, 33) = 19.93, p < .001; and, F (3, 33) = 26.094, p < .001, respectively). Table 2 shows a summary of the means and standard deviations for each  measure. In all cases, the stimulus specific classifier provided higher performance. For example, the tool classifier applied to the tool data resulted in higher accuracy than the tool classifier applied to the location data. Pairwise comparisons indicated there was no significant difference in the comparisons of each stimulus specific classifier applied to the corresponding stimulus.

Waveform analysis
The specified time windows examined for positive amplitudes and latencies were 150-320 ms and 350-550 ms. The specified time windows to examine N170 and N400 amplitudes and latencies were sets to

Experiment 2 discussion
Experiment 2 provided offline evidence that a stimulusspecific classifier could produce superior BCI performance in terms of accuracy, selections per minute, and bitrate. The classifier applied to the same stimulus type (i.e. locationto-location or tool-to-tool) yielded better performance than either classifier applied to the other stimulus type. These findings support our hypothesis that a stimulus specific classifier applied to the corre sponding stimulus can result in improved BCI performance. By having two stimulus specific classifiers operating simultaneously, using a two stimulus paradigm could potentially lead to increases in online BCI performance. In addition, several differences in ERP components were observed (discussed below).

General discussion
Recent studies have shown that ERP components associated with facial stimuli can improve BCI performance in a two-stimulus presentation paradigm (Kaufmann et al 2014). The present work extends these findings and incorporated two additional types of novel stimuli, location and graspable objects. Prior to this study, ERPs produced by location and graspable objects have not been examined; however, fMRI data has provided evidence that these stimuli activate different brain regions. Thus, the purpose of the present study was to determine if location and graspable objects produce differential ERPs that can subsequently lead to an improvement in BCI speed and accuracy.
Experiment 1, using facial and location stimuli, showed no differences in face specific components; nonetheless, other ERP differences were observed and the location stimuli produced slightly better performance than facial stimuli. Kaufmann et al (2011) first examine facial stimuli in ablebodied subjects, based on their positive results, they extended the paradigm to people with severe speech and communication disorders and confirmed that facial stimuli produced higher speed and accuracy in this population as well (Kaufmann et al 2011). The findings of our project indicate that location stimuli produce comparable performance to facial stimuli. Thus, we suggest that location stimuli may be beneficial for people with severe speech and communication disorders. This hypothesis should be tested in future studies.
Functional MRI research indicating activation in the PPA in response to visual processing of location stimuli (Agguire et al 1998, Epstein et al 1999, Malach et al 2002, as well as activation in the premotor cortex in response to stimuli such as graspable objects (Tucker and Ellis 2004, Creem-Regehr and Lee 2005, Grafton et al 1997, prompted the examination of parietal locations (PO7 and PO8) and frontal locations (F3, F4, FC5, and FC6) in addition to locations Pz and Cz. Due to lack of EEG research on the ERPs produced by these stimulus types, exploratory analyses were conducted. In Experiment 2, location stimuli were compared to graspable object stimuli. Our working hypothesis was that higher P300 amplitudes would be observed for the location stimuli than for the tool stimuli. The rationale for this hypothesis was due to the fact that participants were instructed to imagine swinging a hammer each time the target item appeared. The added cognitive demands of the task were, therefore, expected to reduce P300 amplitude and maximize the difference between the ERPs produced by each stimulus type. Waveform analyses comparing the ERPs produced by the two stimulus types showed higher amplitude produced by the location image than the tool image at each of the examined electrode locations, except for PO7 in positive time window 350-550 ms. Similar to Experiment 1, BCI performance was higher in the location stimulus condition. These results support our hypothesis that stimulus-specific classifiers may provide higher performance, as compared to the current methodologies that rely on a single classifier. Future research will develop stimulus specific classifiers to be tested online in a two-stimulus presentation paradigm. The development of simultaneous, dual stimulus-specific classifiers could potentially allow the BCI to quickly eliminate half of the characters in the matrix as potential targets. Thus, having the potential to increase the speed with which selections can be made and decreasing the number of selection errors. The utility of two classifiers will be determined by the amount of variation in the ERPs produced by each class of stimuli. Thus, it is important to select stimuli that elicit significantly different ERPs.

Conclusion
P300 BCI technology has shown to be an effective method of communication; however, due to the relatively slow rate of communication improvements are necessary. The first online P300-based BCI study resulted in accuracy of 35 percent (Donchin et al 2000). Since this time, online accuracy is consistently near 100 percent. Nonetheless, further improvements are needed for the technology to rival assistive communication devices that rely on muscle movement. Therefore, novel classification techniques and paradigm modifications are necessary to provide people with severe speech and physical impairments more efficient BCI communication options after muscle control is lost. The goal of the current study, however, was to use offline analyses to investigate the efficacy of using unique categories of stimuli. Nonetheless, due to the early stages of this line of research, it would not be appropriate to test the current paradigm with people who have severe speech and physical impairments. As with most P300 BCI research conducted in laboratory settings, an inherent limitation to our study design is the use of able-bodied participants as our sample. Another limitation of the study design is that it did not afford us the opportunity to compare graspable object stimuli to facial stimuli. Further investigation comparing graspable object stimuli to facial stimuli may be beneficial to determine which would be more useful in the two-stimulus paradigm. In both experiments, the offline results suggest a two-stimulus dualclassifier paradigm can improve BCI performance. An online adaptation of the paradigm should be tested in future work.