The Effects of Working Memory on Brain-Computer Interface Performance

Abstract

Objective

The purpose of the present study is to evaluate the relationship between working memory and BCI performance.

Methods

Participants took part in two separate sessions. The first session consisted of three computerized tasks. The LSWM was used to measure working memory, the TPVT was used to measure general intelligence, and the DCCS was used to measure executive function, specifically cognitive flexibility. The second session consisted of a P300-based BCI copy-spelling task.

Results

The results indicate that both working memory and general intelligence are significant predictors of BCI performance.

Conclusions

This suggests that working memory training could be used to improve performance on a BCI task.

Significance

Working memory training may help to reduce a portion of the individual differences that exist in BCI performance allowing for a wider range of users to successfully operate the BCI system as well as increase the BCI performance of current users.

1. Introduction

Brain-computer interfaces (BCIs) are an alternative form of communication that, unlike the majority of augmentative and alternative communication (AAC) devices, require no muscular movement or control. BCIs use brain signals instead of overt movement for the purpose of controlling a computer. AAC devices include any method of communication that does not require speech (Brownlee and Palovcak, 2007). Examples of these include writing, typing, letter and picture boards, eye blinks, and eye tracking devices. A BCI can facilitate communication for people who have severe speech and physical impairments (SSPI; Akcakaya et al., 2014). More specifically, BCI devices can be a viable communication option for people who cannot adequately control augmentative and assistive communication (AAC) technology that depends on residual muscle control. This group of people includes those with classic locked-in syndrome (LIS), which refers to a condition in which an individual has lost all neuromuscular control except for eye movement (Bauer et al., 1979). Individuals with SSPI typically have more motor function than those with LIS; however, they are still unable to maintain a desirable level of communication through speech or writing. In some situations, people who have SSPI can benefit from the use of eye-tracking devices; however, as ocular muscle control deteriorates, eye-tracking devices can also fail and a BCI can restore effective communication (Sellers et al., 2010; Seller et al., 2014).

1.1 The P300 BCI

The P300 is often studied using what is known as an “oddball paradigm.” According to Donchin and Coles (1988), in order to elicit the classic oddball response, four conditions must be met. The first is that a participant must be presented with at least two categories of stimuli and each stimulus must fall into only one of the categories. The second is that one type of stimulus must be presented less frequently than other stimuli. The third requirement is that the participant is instructed to perform a task that entails placing each stimulus into one of the categories. The final requirement is that the stimuli must be presented in a random order (Donchin and Coles, 1988). Often times the task involves paying attention to the rare stimulus and ignoring the frequent stimulus (Fichtenholtz et al., 2004). This usually involves instructing the participant to silently count the number of times the rare stimulus is presented (Farwell and Donchin, 1988). The P300 is amiable to BCI use for a variety of reasons. It can be obtained through noninvasive means, it involuntarily occurs in response to an attended stimulus, it can be elicited without movement, calibration can be completed in approximately 10 minutes, and the majority of people are able to use it (Guger et al., 2009).

1.1.1 Psychological Predictors of BCI Performance

Signal processing and paradigm modifications have significantly improved BCI performance. In contrast, participant specific psychological factors have rarely been investigated. Many factors influence BCI performance (Kleih et al., 2010). Some people find the BCI very difficult to use and others are unable to use the system at all. For example, Kübler et al. (2001) found that restrictions to BCI use exist as a result of the patient’s psychological situation in addition to technological and physical restraints. Johnson (1986) developed a triarchic model of P300 amplitude in order to conceptualize variations in the P300-ERP response. Johnson’s model demonstrates that both subjective probability and stimulus meaning make equal but separate contributions to the P300 amplitude. Both subjective probability and stimulus meaning are modulated by a third dimension, information transmission (Johnson, 1986). Psychological variables have also been shown to have an influence on the P300 amplitude. Kleih et al. (2010) extended Johnson’s research and showed that motivation also impacts the P300 amplitude. Leeb et al. (2007) conducted an experiment that showed highly motivated participants were more successful at navigating a 3D environment than their counterparts. Hammer et al. (2012) conducted a study using an SMR-BCI that identified several psychological predictors of performance including accuracy of fine motor skills and degree of concentration. Hammer et al. (2014) carried out an additional study that showed visuo-motor coordination ability as well as the ability to concentrate on the task at hand are also significant predictors of SMR-BCI performance. Kübler et al. (2001) went so far as to predict the failure of the field of BCI if researchers do not begin to implement more psychological theory and experimentation into their research instead of focusing solely on the technological aspects of BCI. It has been suggested that additional research should examine the variability in BCI performance related to psychological factors and how these factors can be used to improve and predict subsequent performance (Nijboer et al., 2010).

1.1.2 Working Memory

Over the years, several models of memory have been proposed. One previously popular view was that of a dichotomous model with a long-term memory and a short-term memory component. Long-term memory is dependent on neuronal growth, whereas short-term memory is a result of brief electrical activation (Baddeley, 2003). The short-term memory component was meant to comprise working memory. As additional research examined the model, flaws began to arise within the idea that short-term memory included working memory (Baddeley, 1992). This gave rise to new theories that accounted for working memory being more independent from short-term memory.

The term “working memory” was first used more than 40 years ago by Miller et al. (1970). It was originally defined as a cognitive system designed to temporarily store and manipulate information (Baddeley and Hitch, 1975). Baddeley’s original model of working memory contained three subcomponents: the central executive, the visuospatial sketch pad, and the phonological loop. The central executive controls the way an individual’s attention is divided between the two subsystems. Experimental evidence has demonstrated that the subsystems are resource limited. For example, when dual tasks are performed in the same modality (i.e., both auditory or both visual) performance suffers. In contrast, when one task involves the auditory system and the other task involves the visual system performance is similar to that observed in single task conditions (Baddeley, 2000). Baddeley later added an additional component to working memory, the episodic buffer. This component is able to take information from the other three components as well as from long-term memory and create a single episodic representation that can be temporarily stored in the buffer (Baddeley, 2000). The original model continues to remain at the core of working memory theories today. Since the creation of the original model, theories thereafter have expanded the model to include multiple components such as attentional control (Cowan, 1988; Logie, 2011) and individual differences (Jarrold and Towse, 2006; Just and Carpenter, 1992; Unsworth and Engle, 2007).

Working memory differs between individuals and some individuals are able to retain more information and manipulate the information more effectively than others (Baddeley, 2003; Baddeley and Hitch, 1994). In a task designed to assess working memory span, participants are read a series of words or numbers and instructed to repeat as many words or numbers back as they can remember. The List Sorting Working Memory Task (LSWM) was used to measure working memory in the current study. In this task, individuals are required to remember a series of words and then recall them in a specific order (Weintraub et al., 2013). Further details of the LSWM are discussed in section 2.3.

Working memory, specifically working memory load, has been shown to modulate the amplitude of the P300 response. The greater the working memory load, the smaller the P300 amplitude (Morgan et al., 2008). Research has also shown that working memory capacity is greater for more simple stimuli (e.g., colored letters; Alvarez and Cavanagh, 2008). This means that working memory capacity should be greater for tasks such as the BCI task because the stimuli are made up of letters, numbers, and symbols. It has also been shown that individuals with greater working memory capacity have less load for any given task (Morgan et al., 2008), meaning that individuals with greater working memory capacity should perform better on the BCI task because it is less taxing cognitively. By examining the relationship between working memory and BCI performance in healthy participants, further insight can be obtained on ways to improve BCI performance in our target population. If working memory has a significantly large impact on accuracy, this information can be used to develop new methods to improve BCI performance. This would be particularly beneficial for individuals who have amyotrophic lateral sclerosis (ALS) because it has been shown that some people with ALS show deterioration in working memory (Hammer et al., 2011).

1.1.3 General Intelligence

Intelligence has been defined as the mental ability to reason, plan, solve problems, understand multifaceted concepts, and learn quickly and efficiently (Gottfredson, 1994, p. 13). Charles Spearman (1904) originally developed the concept of general intelligence, also known as “g” or the “g-factor.” He described it as representing a general measure of cognitive ability that could be applied to various kinds of cognitive tasks (Deary et al., 2010; Spearman, 1904). The “g-factor” is responsible for a large portion of the individual differences in cognitive ability and is a major source of the predictive power of cognitive measures (Deary et al., 2010). General intelligence is a component of executive function that has been found to be highly correlated with working memory (Ackerman et al., 2005). In particular, memory span, a component of working memory, has been shown to be a crucial component of intellectual functioning (Gignac and Weiss, 2015). Although the research supporting this relationship has continued to grow in recent years, there is some research that suggests the strength of this relationship is not quite as strong or non-existent (Matarazzo, 1972).This makes it an important construct to measure in the current study in order to ensure the individual differences in BCI performance are due to working memory and not general intelligence. The Picture Vocabulary Test (TPVT) was administered to measure general intelligence (Weintraub et al., 2013). The TPVT is a single-word vocabulary comprehension test used to measure the vocabulary portion of language. Additional aspects of the TPVT are discussed in section 2.3.

1.1.4 Executive Function

Executive function has been defined as a top-down process involving cognitive control over activities aimed to achieve goals (Weintraub et al., 2013, p. S55). Executive function changes throughout the life span; it gradually increases during childhood and begins to decline in old age (Zelazo et al., 2004). From a physiological standpoint, processes of executive function are primarily carried out in the frontal lobes of the brain (Craik and Bialystok, 2006). Executive function as a whole can be measured generally to obtain an average estimate of an individual’s ability to conduct high-level cognitive processes. There are several components within executive function that can be controlled and measured in order to obtain a more detailed analysis of an individual’s cognitive abilities. These include mechanisms such as cognitive flexibility, problem solving, working memory, general intelligence (“g”), and response inhibition and selection (Alvarez and Emory, 2006). The Dimensional Change Card Sort Test (DCCS) was administered to measure executive function (Weintraub et al., 2013). The DCCS is designed to measure cognitive flexibility by requiring participants to respond based on changing rules. Additional aspects of the DCCS are discussed in section 2.3.

1.1.5 Current Study

Working memory is one of the factors that may account for inter-individual differences in BCI performance. Conducting studies that compare working memory with performance on BCI tasks can help create methods that can assist in accounting for individual differences in BCI use, potentially leading to an increase in BCI implementation and performance. The current study examined executive function, general intelligence, and working memory prior to completion of a BCI task. These measurements were taken in order to compare the relationship between these constructs and an individual’s performance on a BCI task. The focus of the current study is working memory; however, executive function and general intelligence are correlated with working memory and could also affect BCI performance. Therefore, these measurements were taken in order to account for this possibility and to determine what portion of BCI task performance is due to working memory.

The main hypothesis of the study is that participants with high levels of working memory capacity will show higher accuracy on a BCI task because working memory is utilized during BCI use. When a participant is using the BCI, he or she must formulate a sentence, remember the sentence while spelling it, remember which character he or she is currently trying to select, evaluate feedback from the BCI, and make a decision regarding the next item to select. This entire process is aided by working memory and a participant with low working memory is more likely to make mistakes. Additionally, the authors hypothesize that participants with high levels of general intelligence and executive function, specifically cognitive flexibility, will also show higher accuracy on a BCI task. Furthermore, the authors hypothesize that working memory will account for the most variance in BCI performance, followed by executive function, and general intelligence.

2. Methods

2.1 Participants

The study involved a sample of 34 healthy participants obtained using the online participant pool at ETSU. The study was approved by the ETSU Institutional Review Board.

2.2 Stimuli and Materials

Each participant took part in two sessions conducted on different days. Upon arriving at the first session, the participant read and signed the informed consent document. The first session consisted of three computerized measures. The order in which the measures were completed was counter-balanced. In the second session, participants operated a BCI.

2.3 Measuring Psychological Factors

The three instruments used in this study were selected from the NIH (National Institutes of Health) Toolbox Cognition Battery, a component of the NIH Toolbox for the Assessment of Neurological and Behavioral Function (Gershon et al., 2013). The NIH consulted with 102 experts in the field of cognition in order to select or develop instruments to be included in the battery. The resulting collection of instruments was then tested using a sample of 476 participants ages 3 to 85. The instruments were validated in English and tested for age effects on performance, convergent and discriminant construct validity, and test-retest reliability. Gold standard measures of the constructs were used to test the convergent construct validity of each measure.

The measures selected for this study were the List Sorting Working Memory Test (LSWM), the Dimensional Change Card Sort Test (DCCS), and the Picture Vocabulary Test (TPVT). Once the test began, the participant responded verbally for the LSWM, used the left and right arrow keys for the DCCS, and used the mouse for the TPVT. Accuracy was recorded for all three measures and timing was recorded for the DCCS. The scoring for all three measures provides a computed score and an age-adjusted score for easier comparison.

The LSWM measures working memory by evaluating both information processing and storage. This computerized test takes seven minutes to administer and consists of two parts. For each trial in the first portion of the task, the participant was presented with an audio recording of a list of all animals or all food items. Participants were simultaneously shown pictures of the animals or food items on their monitor with the name of the animal or food item printed under each picture. Participants were then asked to repeat the list of animals or food items to the researcher in increasing size order. For trials in the second portion of the task, participants were presented with a list containing both food and animals. They were then instructed to repeat the list to the researcher in size order from smallest to largest, naming the food items first and then the animals. The dependent measure was accuracy.

The DCCS measures executive function by determining cognitive flexibility. The DCCS is also computerized and takes 4 minutes to administer. There are 40 trials and each trial takes approximately 6 seconds to complete. The task required the participant to make judgments regarding properties of objects. On each trial, the participant was instructed to fixate on a star. The star was then replaced by a cue word “shape” or “color” that indicated the property to be reported. A target picture then replaced the word in the center of the screen. Below the target picture were two more pictures, each one matching either the target picture’s color or shape. The participant’s task was to select whichever picture matched the target picture based on the property given by the cue word. The selection was made using the left or right arrow key.

All trials included in the first portion required the participant to select the picture that matched the shape of the target picture. All trials included in the second portion involved matching the color of the target picture. The third portion included a mix of the trials in the first two portions and participant was required to match either the shape or color of the stimulus based on a cue word (e.g., “shape,” “color”) that flashed before the two pictures were presented. The dependent measures were speed and accuracy.

The TPVT measures receptive vocabulary, which is considered to be a good representation of general intelligence (“g”). The test takes approximately 4 minutes to administer and is completely computerized. After the researcher read the participant the instructions, the test began by presenting the participant with an audio recording of a word paired with a display of four pictures on the monitor. For this test, the mouse was used to respond. In this test participants were presented with an auditory word and four pictures on the screen. They were required to click on the picture that best matched the meaning of the word specified by the experimenter. Accuracy was the dependent measure.

2.4 Brain-Computer Interface Task

The second session began with the participant being seated in front of a monitor to the right of the researcher. Each participant was given a series of surveys in order to measure his or her current levels of fatigue, hunger, caffeine, motivation, and mood. These were collected to be included in the statistical analyses as covariates. The Stanford Sleepiness Scale (SSS) was used to measure fatigue. Three visual analog scales were created for this study; measures of hunger, motivation, and mood. Caffeine use was also measured. A second measure of motivation was also used, the Questionnaire for Current Motivation for BCI2000 (QCM-BCI), which was modified by Nijboer et al. (2008) from the original QCM (Rheinberg et al., 2001) to measure motivation specific to the BCI task. All measures, except for the QCM-BCI, were given to the participants to fill out during the first session as well. Participants were then shown an informational PowerPoint about the BCI while they were measured and fitted with an EEG cap. The BCI task presented an 8x9 matrix and the checkerboard paradigm (CBP) was used to present the stimuli (Townsend et al., 2010). During the first portion of the session, participants completed a copy-spelling task that was used to calibrate the BCI. In copy-spelling, participants are presented with a word at the top of the screen and instructed to spell the word one letter at a time. Each participant copy-spelled three, six-letter words selected from a random word generator. An example of a calibration word is presented in figure 1. In this case, the participant is presented with the word DRAGON. The letter the participant is currently trying to select is presented at the end of the word in parentheses (figure 2). Once the BCI starts, the characters in the matrix begin to flash in groups (figure 2). The participant was instructed to focus on the letter they are currently trying to select within the matrix and count the flashes to themselves. Prior to beginning calibration, they were allowed to practice using the BCI by spelling out a three-letter word, DOG. Participants were not given feedback during calibration.

Figure 1.

Example of calibration word.

Figure 2.

Example of system flashing during calibration.

Each group of stimuli was presented for 62.5ms, followed by an inter-stimulus interval (ISI) of 62.5ms; thus, stimulus onset asynchrony (SOA) was 125ms. After spelling all three words, the data were processed offline using SWLDA (discussed in detail below) to produce a classifier that was used during the second portion of the session, which provided “online feedback” regarding the accuracy of the BCI selection.

For the online portion of the study, participants were required to spell out three predetermined sentences: THE_CAT_IN_THE_HAT, THE_QUICK_BROWN_FOX, and MARY_HAD_A_LITTLE_LAMB. During this portion of the session, the sentence was not displayed at the top of the screen. The researcher read a sentence to the participant and he or she was required to spell it, from memory, using the BCI. This was done to increase cognitive load during the task to make it more similar to how the BCI would be used in a practical application. An example of feedback provided by the BCI is shown in figure 3. Participants were instructed not to attempt to correct mistakes and to move on to the next letter in the sentence if a mistake was made.

Figure 3.

Example of feedback during online portion.

2.5 EEG Acquisition and Processing

A 32-channel EEG cap (tin electrodes; Electro-Cap International, Inc.) was used to record the EEG. The left mastoid electrode acted as the ground and the right mastoid electrode acted as the reference. Two 16-channel USB biosignal amplifiers from Guger Technologies (g.tec) were used in order to increase the amplitude of the electrical activity from the scalp being recorded. The electrical activity was then amplified (+/− 2V before ADC) before being digitized at 256 Hz. The data were filtered using a 0.50Hz to 30.0Hz bandpass filter. The researcher ensured impedance values were below 30.0 kΩ before proceeding with the session. BCI2000 software was used to control stimulus presentation, data collection, and online processing (Schalk et al., 2004).

2.6 Classification

Stepwise linear discriminate analysis (SWLDA) was used to create classifiers for this study. SWLDA has proven to be one of the more successful classification techniques, making it widely used across BCI studies (Krusienski et al., 2006; Krusienski et al., 2008). SWLDA weighs the input features using ordinary least-squares regression. The feature that accounts for the highest amount of unique variance to predict the target item is entered into the discriminant function. After a feature has been added to the model, forward and backward stepwise regression analyses are performed until no additional features meet the inclusion criteria (p < 0.10) or the exclusion criteria (p > 0.15), or the maximum number of features have been selected (maximum of 60; Krusienski et al., 2006). Although the EEG cap consists of 32 channels, only eight electrodes were used for BCI operation: Fz, Cz, Pz, P3, P4, P07, P08, and Oz. Previous research has shown that these eight electrodes perform as well as larger montages for accuracy (Krusienski et al., 2006).

2.7 Statistical Analysis

The means, standard errors, and standard deviations were calculated for all measures including the NIH toolbox tasks, BCI task, and additional measures (motivation, mood, hunger, caffeine, and fatigue). Pearson correlations were also calculated to evaluate the relationships between variables.

2.8 Regression Analysis

The goal of this analysis was to estimate the magnitudes of the influences of executive function, working memory, and general intelligence (g) on BCI performance. This goal is complicated by the interrelatedness between these three cognitive components. For example, working memory is considered to be a component of general intelligence, and intelligence tests such as the Weschler Intelligence Scale for Children (WISC) frequently include measures of working memory. Furthermore, even when these cognitive constructs are theoretically separable, it is extremely difficult to measure them independently. Because g is defined as the first common factor that creates the positive manifold (e.g., positive correlations) between performances on disparate cognitive tasks, it is inevitable that tasks designed to measure specific cognitive components (such as executive function) will also measure g to some extent. Although working memory and executive function are theoretically distinct, measuring executive function without also inadvertently measuring working memory is challenging. In the DCCS task used in this study, subjects must remember the cue long enough to respond, suggesting that working memory will “contaminate” the responses to the executive performance task to some extent. Similarly, in the LSWM working memory task, executive function is required in order to plan and execute rehearsal and recall.

3. Results

Data from 27 of the 34 participants were included in the statistical analyses. Participants who were excluded from the analyses were taken out for the following reasons: three participants failed to complete both sessions, two were not native English speakers, and two participants’ data were incomplete because of EEG recording problems. NIH toolbox scores and BCI performance accuracy for the participants that were included in the analysis are listed in table 1.

Table 1.

Data for NIH Toolbox tasks and BCI performance accuracy.

	NIH Toolbox Tasks			BCI Performance Accuracy
Subject ID	LSWM	DCCS	TPVT	Percent Correct
ExStudy_009	103.57	119.15	102.31	94.92
ExStudy_010	80.70	87.13	88.07	6.78
ExStudy_011	140.86	97.57	128.70	100.00
ExStudy_012	97.60	111.36	117.52	94.92
ExStudy_014	118.14	109.94	113.75	79.66
ExStudy_015	89.14	101.53	100.16	71.19
ExStudy_016	128.24	107.80	108.53	76.27
ExStudy_017	118.14	105.00	92.84	10.17
ExStudy_018	108.26	113.55	130.95	50.85
ExStudy_019	89.14	119.65	114.80	89.83
ExStudy_021	94.33	105.35	99.41	74.58
ExStudy_022	102.63	122.65	117.02	91.53
ExStudy_023	103.57	117.95	94.94	91.53
ExStudy_025	108.26	116.57	91.69	91.53
ExStudy_026	113.85	111.64	120.21	91.53
ExStudy_027	113.85	105.00	97.13	91.53
ExStudy_029	98.44	110.89	117.54	86.44
ExStudy_031	103.57	100.84	77.91	55.93
ExStudy_033	89.14	114.71	100.89	44.07
ExStudy_034	122.75	120.42	116.61	66.10
ExStudy_036	84.92	105.02	112.23	35.59
ExStudy_037	108.26	119.65	102.94	45.76
ExStudy_038	98.44	103.46	117.03	44.07
ExStudy_039	98.44	116.31	120.21	55.93

ExStudy_040	94.33	95.84	89.39	25.42
ExStudy_041	98.44	101.79	103.96	5.08
ExStudy_042	108.01	124.21	114.88	72.88

3.1 Descriptive Statistics

Table 2 shows mean scores on all three NIH toolbox tasks and the BCI task. The mean scores for all three NIH toolbox tasks were within one standard deviation of the normative data (mean=100; SD=15).

Table 2.

Means, standard deviations, and standard errors for first session.

	LSWM	DCCS	TPVT	VAS Mot	VAS Mood	Hunger	Caffeine	Fatigue
Mean	104.26	109.81	107.10	7.49	7.72	3.11	53.79	1.96
SD	13.71	9.14	13.14	1.86	1.52	2.45	89.66	0.76
SE	2.64	1.76	2.53	0.36	0.29	0.47	17.26	0.15

^aLSWM = List Sorting Working Memory Test

^bDCCS = Dimensional Change Card Sort Test

^cTPVT = Picture Vocabulary Test

^dVAS Mot = Visual Analogue Scale of Motivation

^eVAS Mood = Visual Analogue Scale of Mood

Similarly, table 3 presents measures collected in the second session. Correlations between the NIH Toolbox tasks and BCI performance accuracy are included in table 4. Correlations between all measures are presented in table 5 (first session) and table 6 (second session). Table 7 provides the significance of variables with mediators (i.e., hunger, caffeine, fatigue, mood, and motivation) excluded and included. Table 8 shows the model fit R² for mediators excluded and included.

Table 3.

Means, standard deviations, and standard errors for second session.

							QCM-BCI
	BCI Acc	VAS Mood	VAS Mot	Hunger	Caffeine	Fatigue	IF	I	C	MC
Mean	0.65	7.61	7.47	3.56	47.24	2.11	2.36	5.42	5.16	5.57
SD	0.29	2.06	2.37	2.75	69.73	0.89	1.13	1.21	0.68	0.99
SE	0.06	0.40	0.46	0.53	13.42	0.17	0.22	0.23	0.13	0.19

^aBCI Acc = Brain-Computer Interface Accuracy

^bVAS Mood = Visual Analogue Scale of Mood

^cVAS Mot = Visual Analogue Scale of Motivation

^dIF = Incompetence Fear

^eI = Interest

^fC = Challenge

^gMC = Mastery Confidence

Table 4.

Table of correlations between NIH Toolbox tasks and BCI performance accuracy.

	LSWM	DCCS	TPVT	BCI Performance
LSWM		.140	.304	.359
DCCS	.140		.366	.490**
TPVT	.304	.366		.347
BCI Performance	.359	.490**	.347

^**Correlation is significant at the 0.01 level (2-tailed).

^*Correlation is significant at the 0.05 level (2-tailed).

^aLSWM = List Sorting Working Memory Test

^bDCCS = Dimensional Change Card Sort Test

^cTPVT = Picture Vocabulary Test

^dBCI Performance = Brain-Computer Interface Performance

Table 5.

Table of correlations between all measures for the first session.

	LSWM	DCCS	TPVT	VAS Mot	VAS Mood	Hunger	Caffeine	Fatigue
LSWM			.304	.066	.192	.325	.242	.204
DCCS	.140			−.061	−.236	.152	.074	−.010
TPVT	.304	.366			.169	.089	.618**	−.323
VAS Mot	.066	−.061	.282			−.173	.255	−.363
VAS Mood	.192	−.236	.169	.389*			.281	−.111
Hunger	.325	.152	.089	−.173	.071			.130
Caffeine	.242	.074	.618**	.255	.281	−.170
Fatigue	.204	−.010	−.323	−.363	−.111	.130	−.082

^**Correlation is significant at the 0.01 level (2-tailed).

^*Correlation is significant at the 0.05 level (2-tailed).

^aLSWM = List Sorting Working Memory Test

^bDCCS = Dimensional Change Card Sort Test

^cTPVT = Picture Vocabulary Test

^dVAS Mot = Visual Analogue Scale of Motivation

^eVAS Mood = Visual Analogue Scale of Mood

Table 6.

Table of correlations between all measures for the second session.

	BCI Acc	MC	IF	I	C	VAS Mot	VAS Mood	Hunger	Caffeine	Fatigue
BCI Acc			−.299	.117	−.081	−.106	.212	.287	.021	.021
MC	.241			.657**	.220	.362	.448*	−.146	.323	−.281
IF	−.299	−.639**			.141	−.230	−.445*	−.084	−.134	.370
I	.117	.657**	−.391*			.651**	.636**	−.292	.476*	−.423*
C	−.081	.220	.141	.681**			.351	−.342	.259	−.030
Vas	−.106	.362	−.230	.651**	.486*			−.453*	.312	−.576**
Mot
Vas	.212	.448*	−.445*	.636**	.073	.729**			.341	−.627**
Mood
Hunger	.287	−.146	−.084	−.292	.081	−.453*	−.447*			.212
Caffeine	.021	.323	−.134	.476*	.192	.312	.341	−.131
Fatigue	.021	−.281	.370	−.423*	.883	−.576**	−.627**	.212	−.228

^**Correlation is significant at the 0.01 level (2-tailed).

^*Correlation is significant at the 0.05 level (2-tailed).

^aBCI Acc = Brain-Computer Interface Accuracy

^bMC = Mastery Confidence

^cIF = Incompetence Fear

^dI = Interest

^eC = Challenge

^fVAS Mot = Visual Analogue Scale of Motivation

^gVAS Mood = Visual Analogue Scale of Mood

Table 7.

Significance of variables with mediators excluded and included.

Model	Unstandardized Coefficients		Standard Coefficients
	B	SE B	β	t	p
(Constant)	−4.139	2.873		−1.440	.166
EF_Minus_WM_and_G	.071	.042	.300	1.695	.106
G_Minus_WM_and_EF	.110	.035	.643	3.168	.005
WM with G and EF	.069	.028	.444	2.446	.024
P2_Hunger	.355	.131	.483	2.708	.014
P2_Mood	.388	.242	.397	1.606	.125
P2_Caffeine	−.008	.005	−.280	−1.614	.123
P2_Fatigue	.590	.505	.260	1.169	.257
(Constant)	−4.906	4.848		−1.012	.329
EF_Minus_WM_and_G	.054	.072	.226	.744	.469
G_Minus_WM_and_EF	.108	.042	.627	2.590	.021
WM with G and EF	.070	.049	.454	1.442	.171
P2_Hunger	.349	.154	.476	2.265	.040
P2_Mood	.516	.418	.528	1.235	.237
P2_Caffeine	−.009	.006	−.317	−1.418	.178
P2_Fatigue	.519	.810	.229	.641	.532
P2_Mastery_Confidence	−.014	.598	−.007	−.023	.982
P2_Incompetence_Fear	.254	.530	.143	.480	.638
P2_Interest	.439	.735	.263	.598	.559
P2_Challenge	−.189	1.187	−.064	−.159	.876
P2_Motivation	−.257	.306	−.302	−.840	.415
Dependent Variable: logit_BCI

^aEF_Minus_WM_and_G = the residualized executive function variable

^bG_Minus_WM_and_EF = the residualized general intelligence variable

^cWM with G and EF = the residualized working memory variable

^dlogit_BCI = the logit transformation of the brain-computer interface accuracy outcome variable

Table 8.

Model fit R² for mediators excluded and included.

Model	R	R2	Adjusted R2	SE
1	.752a	.565	.405	1.558
2	.776b	.602	.261	1.736

Notes:

^aPredictors: (Constant), P2_Fatigue, WM with G and EF, P2_Hunger, P2_Caffeine,

EF_Minus_WM_and_G, G_Minus_WM_and_EF, P2_Mood

^bPredictors: (Constant), P2_Fatigue, WM with G and EF, P2_Hunger, P2_Caffeine,

EF_Minus_WM_and_G, G_Minus_WM_and_EF, P2_Mood, P2_Mastery_Confidence, P2_Challenge, P2_Incompetence_Fear, P2_Motivation, P2_Interest

3.2 Regression Analyses

This analysis proceeded in two stages. In the first stage, linear regression models were used to create “residualized” versions of the executive function and general intelligence variables. These residualized variables would be free of “cross-contamination” with the other cognitive constructs and would thereby represent purer independent measurements of their target constructs. The residualized executive function variable was created by regressing the executive function (DCCS) scores on general intelligence (TPVT) and working memory (LSWM). The residuals from this model thereby represented the unexplained variance in executive function after statistically removing the influences of general intelligence and working memory. These residuals were recorded in a variable that would be used in stage two of the analysis. Using a similar procedure, the residualized general intelligence variable was created by saving the residuals from a regression of general intelligence (TPVT) on executive function (DCCS) and working memory (LSWM); the residualized general intelligence variable would likewise be used in the next stage of the analysis.

The logit transformation of the BCI accuracy outcome variable was necessary because the outcome, being a proportion, has a ceiling at one and is not measured on an interval scale. However, the linear regression model assumes that the outcome variable is unconstrained with respect to its range and that the functional form of the relationship between the predictor and outcome variable is linear. Both of these assumptions are violated when the outcome variable is a percentage, the latter because the meaning of a unit gain in proportion correct does not represent equal improvement over the full range of the scale. Therefore, our outcome variable was transformed prior to analysis via the logit transformation. The logit transformation is defined as:

$$\text{logit}(p)=\mathit{log}\left(\frac{p}{1-p}\right)$$

where the natural log function is assumed. This transformation maps the BCI proportion correct outcome onto an unconstrained metric for which the linear functional form is reasonable.

In stage two, a stepwise linear regression model was fit to the data, regressing the logit-transformed BCI proportion correct on the working memory (LSWM) scores, the residualized general intelligence scores, the residualized executive function scores, and a set of covariates. Model 2A included as covariates measures of hunger, mood, caffeine, and fatigue. Model 2B retained added additional covariates including mastery confidence, incompetence fear, interest, challenge, and motivation. Results from these two models are summarized in tables 7 and 8. Results from model 2A revealed statistically significant effects of working memory (β = .444, p = .024) and the residualized general intelligence variables (β = .643, p = .005). In the covariate part of the model, only the hunger measure was significantly associated with logit BCI performance (β = .483, p = .014). The model explained 75.2% of the variance in logit BCI performance.

Model 2B retained all predictors from model 2A and added a set of psychological variables including mastery confidence, incompetence fear, interest, challenge, and motivation.

After adding those additional covariates, the effect of working memory became non-significant (β = 0.454, p =.171) while the effects of general intelligence (β = 0.627, p = .021) and hunger (β = 0.467, p = .040) remained statistically significant but slightly diminished in magnitude. Model 2B explained 77.6% of the variance in logit BCI performance, but its adjusted R² value is lower due to the increased number of predictor variables and the small sample size.

4. Discussion

Our analysis found evidence of an association between BCI accuracy and both general intelligence and working memory. The influence of general intelligence persisted across models with two varying sets of covariates, while the influence of working memory faded to non-significance when the psychological covariates were included in the model. We believe that the working memory effect disappeared because one or more of the psychological covariates (i.e., hunger, caffeine, fatigue, mood, and motivation) are likely mediators of the relationship between working memory and BCI accuracy. If working memory does exert a causal influence on BCI performance, and some of this influence is transmitted through psychological variables such as motivation and mood, then controlling for these mediating variables will reduce the magnitude of the regression coefficient for working memory on BCI performance. We note specifically that this “diminishing” of effect happened only for the working memory variable, which was not residualized prior to analysis. It is possible that psychological variables also mediate the effect of general intelligence but those mediational effects were diminished by the residualization process. Future research should explicitly investigate possible mediation; our sample size prohibited formal mediation analysis. We note that our sample size implies relatively poor precision of estimation and limited statistical power; larger samples may reveal additional relationships between these variables.

The majority of BCI research focuses on improving the hardware and signal processing methods. Although advancements in technology are necessary to continue progress in the field of BCI, additional research focusing on the users can provide further advancements where the technology falls short. Few studies have examined the causal mechanisms behind individual differences in BCI performance. By evaluating the impact of psychological factors such as executive function, general intelligence, and working memory on BCI performance, a better understanding of the causal factors behind variation in BCI performance can be achieved.

Both working memory and general intelligence were found to have significant effects on BCI performance; therefore, both can be measured in order to predict BCI performance. General intelligence tends to remain stable throughout the lifespan, whereas working memory is a more malleable construct. Working memory can be increased through training, which should lead to an increase in an individual’s BCI performance.

There are a number of possible explanations as to why working memory is related to BCI performance. When a user is using the system to spell out a sentence, he or she must formulate a sentence, remember the sentence while spelling it, and remember which character he or she is currently trying to select. This entire process is aided by working memory and a user with lower working memory is more likely to make mistakes due to forgetting where he or she is at in the sentence. It is well known that there is a strong attentional component in the BCI task and the central executive component of working memory assists with attentional control.

4.1 Working Memory Training

Working memory capacity has been previously thought to be a stable construct; however, recent research has indicated that there is plasticity in the neural systems underlying working memory that is training-induced (Olesen et al., 2003). Studies have shown that taking part in a training program containing working memory tasks can create increased activity in the prefrontal and parietal cortices, leading to improved performance on working memory tasks and increased working memory capacity (Akerlund et al., 2013; Klingberg, 2010; Olesen et al., 2003).

There are several different training procedures that can be implemented in order to improve working memory. The majority of training programs take place over a period of 5 weeks, 5 days per week, for under an hour (Klingberg, 2010; Morrison and Chein, 2010; Olesen et al., 2003). Training tasks involving variations of the n-back task have been shown to significantly increase working memory (Morrison and Chein, 2010; Verhaeghen et al., 2004). An n-back task with a four-back condition requires participants to indicate if each new stimulus is the same as the stimulus shown four items back. The n-back task can be distributed as a computerized measure for easy administration and data collection. The response system used to complete the n-back task can be modified to meet the needs of the user. For example, if the user has control over their eye movements, he or she may use eye blinks or looking to the left or the right to respond.

While several studies have demonstrated an increase in working memory as a result of working memory training, other studies have raised concerns related to the methodology used within these studies as well as the efficacy of working memory training in general (Shipstead et al., 2012). These concerns include the absence of a control group or the use of no-contact control groups, the use of subjective measures of working memory improvements (e.g., self-report measures), inconsistent use of working memory tasks, and researchers defining improvements in working memory through performance on a single task (Shipstead et al., 2012). Shipstead et al. (2012) provide an outline for working memory training in order to demonstrate that the training program successfully increased working memory capacity. Two groups should be included, a control group and an experimental group. The control group should take part in all components of the experiment; however, instead of participating in the working memory training program, they should participate in a similar training program designed to improve a different construct. Prior to the commencement of training, a pretest should be administered that incorporates a battery of tasks designed to measure working memory capacity. Approximately 20 training sessions are suggested, with each session lasting between 30 minutes to an hour. The training program should adapt to user performance through the modification of the length of the list of items to be remembered. Upon completion of the training program, participants should take part in a posttest that includes alternate versions of the pretest tasks. The tasks used in both the pretest and the posttest should differ from the tasks used in the training program in order to demonstrate that the working memory training has successfully transferred to tasks other than those used in the training program (Shipstead et al., 2012). Even with the above-mentioned procedures, further research is still needed to determine the true efficacy of working memory training.

Improvements on the training task are confounded because increased performance accuracy may be due to task-specific practice. Therefore, performance transfer must be demonstrated using untrained working memory tasks (Shipstead et al., 2012), such as BCI, for example. It is also possible that repeated use of the BCI leads to an increase in BCI performance by improving working memory over time. It may be the case that the BCI task itself is a better training method than tasks such as the n-back task for increasing working memory. In order to determine which method provides a larger and/or faster increase in BCI performance, a study including a BCI task group, a n-back task group (or other working memory training task), and a control group should be conducted. The current study was unable to test this hypothesis because it was limited to a single BCI session. Previous research conducted over many sessions has shown that performance does not improve with practice (e.g., Nijboer, Sellers, et al 2008; Sellers, Vaughan, Wolpaw 2010). The results suggest that practice alone will not necessarily improve BCI performance and incorporating a working memory training task in conjunction with BCI practice may be advantageous.

Recent research, such as that of Nijboer et al. (2010), has suggested that psychological factors may offer a significant contribution to the prediction of BCI performance. Understanding which factors contribute to an individual’s performance on a BCI task can help inform training procedures in order to allow a greater number of people to successfully operate the BCI as well as improve the BCI performance accuracy of current users. Studies such as that of Kleih et al. (2010) as well as the current study show promising results associated with psychological factors as predictors of BCI performance. Further research on these factors should lead to an overall improvement in BCI performance and allow more people to benefit from the technology.

5. Conclusion

Recently there has been an increased interest in researching psychological factors that have the potential to influence BCI performance. Through the examination of potential contributing factors such as executive function, general intelligence, and working memory, more can be learned about the causation of individual differences in BCI performance. The current study provides promising results indicating that there are additional psychological factors outside of motivation that contribute to a user’s BCI performance. Future research should focus on determining other potential psychological factors that are related to BCI performance such as anxiety, personality type, and self-esteem. This increase in knowledge will help to better inform BCI training procedures in order to allow a broader range of individuals to successfully operate and communicate using BCIs.

Highlights.

• Psychological factors such as working memory may help to account for individual differences in BCI performance.

• Working memory is a construct that can be increased through training, making it a potential avenue for increasing BCI performance.

• Reducing individual differences in BCI performance can lead to a wider range of users and higher BCI performance accuracy.