Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Oct 7.
Published in final edited form as: Brain Comput Interfaces (Abingdon). 2019 Aug 1;6(1-2):13–24. doi: 10.1080/2326263x.2019.1643203

Trends in research participant categories and descriptions in abstracts from the International BCI Meeting series, 1999 to 2016

Brandon S Eddy a,*, Sean C Garrett b, Sneha Rajen b, Betts Peters a, Jack Wiedrick c, Abigail O’Connor a, Ashley Renda a, Jane E Huggins b, Melanie Fried-Oken a
PMCID: PMC7540243  NIHMSID: NIHMS1535086  PMID: 33033728

Abstract

Much brain-computer interface (BCI) research is intended to benefit people with disabilities (PWD), but inclusion of these individuals as study participants remains relatively rare. When participants with disabilities are included, they are described with a range of clinical and non-clinical terms with varying degrees of specificity, often leading to difficulty in interpreting or replicating results. This study examined trends in inclusion and description of study participants with disabilities across six International BCI Meetings from 1999 to 2016. Abstracts from each Meeting were analyzed by two trained independent reviewers. Results suggested a decline in participation by PWD across Meetings until the 2016 Meeting. Increased diagnostic specificity was noted at the 2013 and 2016 Meetings. Fifty-eight percent of the abstracts identified PWD as being the target beneficiaries of BCI research, though only twenty-two percent included participants with disabilities, suggesting evidence of a persistent translational gap. Participants with disabilities were most commonly described as having physical and/or communication impairments compared to impairments in other areas. Implementing participatory action research principles and user-centered design strategies continues to be necessary within BCI research to bridge the translational gap and facilitate use of BCI systems within functional environments for PWD.

Keywords: Brain-Computer Interfaces, Rehabilitation, Disabled Persons, Biomedical Engineering, Biomedical Technology

Introduction

The International Brain-Computer Interface (BCI) Meeting is a multi-day retreat conference for multidisciplinary BCI research teams to present and discuss advances in research, innovation, and technology transfer in the field. Significant growth has occurred in the BCI field since the first International BCI Meeting in 1999. The first Meeting featured 22 different research groups from 6 different countries (Wolpaw et al., 2000) whereas the 2016 Meeting featured over 400 attendees, and included 188 research groups and organizations from 26 different countries (Huggins, Müller-Putz, & Wolpaw, 2017). Six International BCI Meetings took place within the period of 1999 to 2016 (Wolpaw et al., 2000; Vaughan et al., 2003; Vaughan & Wolpaw, 2006; Vaughan & Wolpaw, 2011; Huggins & Wolpaw; 2014; Müller-Putz, Huggins, & Steyrl, 2016). Research from the Meetings suggests technological growth and innovation, particularly in the areas of communication and control, which often aim to benefit people with disabilities (PWD) (Kübler et al., 2013).

BCI systems are designed to benefit PWD by facilitating increased independence in the functional areas such as communication, mobility, computer access and electronic aids to daily living (Wolpaw & Wolpaw, 2012). However, the extents to which PWD are involved as research participants in studies which aim to help them remains a question. There is now a significant body of literature documenting the ability of research participants without disabilities to effectively use a BCI system (e.g., Hwang et al., 2012; Akram, Han, & Kim., 2015; Baykara et al., 2016), and an increasing number of studies investigating BCI performance with PWD (e.g., Vaughan et al., 2006; Nijboer et al., 2008; Moghimi, Kushki, Guerguerian, & Chau, 2013; Monge-Pereira et al., 2017; Carelli et al., 2017; He et al., 2018). Studies comparing BCI performance for individuals without disabilities and PWD have produced inconsistent results, with some studies finding that control participants without disabilities perform better than PWD (i.e., Kaufmann et al., 2013; Ikegami, Takano, Kondo, Saeki, & Kansaku, 2014; Oken et al., 2014), and others finding no difference (Ikegami, Takano, Saeki, & Kansaku, 2011; Severens, Van der Waal, Farquhar, & Desain, 2014; McCane et al., 2015; Silvoni et al., 2016). Possible explanations for these inconsistent findings may include, but are not limited to, differences among participant diagnoses or functional severity, or variations among device interfaces and signal acquisition methods. Collectively, these findings suggest that results from control participants without disabilities may not generalize to PWD, and that testing the technology with PWD is a necessary research step.

Translational research involves studies which bridge basic science with clinical applications to end-users. Kübler and colleagues (2013, 2014) suggested that the BCI field faces a translational gap or a lack of studies investigating the problems and obstacles that emerge when BCI systems are used by PWD. Kübler (2013) indicated that 470 studies on BCI were published from 2008 to 2010, yet only 39 of those studies included people with severe motor impairments. There are many possible factors contributing to this translational gap in BCI research, such as signal reliability concerns for participants with disabilities resulting from multiple sources of signal artifact, as well as fluctuating health and variability in fatigue or medication use (Kübler et al., 2013; Kübler et al., 2014). Since these factors are likely to influence user performance, researchers may choose to study participants without disabilities to control for such variables. Kübler (2013) also described access to PWD, time requirements for data acquisition, costs, and vulnerability of the target group as additional potential barriers. While the presence of a translational gap was investigated by Kübler (2013) from 2008–2010, it is unclear whether the gap was only present for that time period, and further, if it was related to publication bias. No studies to date have yet examined if such a translational gap existed before or after 2008–2010, nor documented change in research participation of PWD over time. If a translational gap is present, quantifying and understanding the gap is the first step to reducing it.

The published abstracts of the International BCI Meeting series present snapshots of current innovations, applications, and research methods within the field for the interval preceding each Meeting, and provide a useful dataset to investigate the presence or absence of a translational gap and how it has changed over time. Here, abstracts for the six Meetings from 1999 to 2016 were analyzed. The purpose of this study is to report on the rate of inclusion of PWD as BCI research participants and on how those participants are described. Since the BCI field has moved beyond proof-of-concept and toward clinical application, an increase over time in the percentage of studies including PWD as participants, as well as an increase in the specificity with which such participants were described, was expected.

Methods

Primary data source:

The published abstracts from the first six International BCI Meetings, held in 1999, 2002, 2005, 2010, 2013, and 2016, served as primary data sources.

Variables:

Primary variables of interest included target user type, study participant type, diagnosis description, functional description, and impairment type. Variables are described in detail below. The research team established several exemplars and coding rules for each variable, and collected and managed data using REDCap electronic data capture tools hosted at the University of Michigan (Harris et al., 2009). REDCap features several data entry formats such as open-ended text boxes, radio-buttons (used for lists with two or more options and required the rater to select one choice), and check-boxes (used for lists of two or more options where the rater could select any number of options) which were used in this study.

Target user type was defined as the population which would experience an increase in function from use of the BCI system under investigation. For example, all studies which tested a BCI targeting communication or mobility were categorized as benefitting PWD, given that these systems would not currently benefit individuals without disabilities above their baseline abilities (i.e., verbal speech and walking). Radio-button options included (1) PWD, (2) people without disabilities, or (3) unspecified (used for studies aiming to acquire general knowledge about the brain or BCI technology).

Study participant type was defined as the population(s) from which study samples were drawn. Checkbox options included (1) PWD, (2) people without disabilities, (3) animal models, (4) perspective/theoretical, or (5) unspecified. Studies including human participants who were not described as PWD were assigned a rating of “people without disabilities.” Individuals with epilepsy who participated in invasive BCI studies were rated as “people without disabilities” given that a diagnosis of epilepsy does not include functional impairments that might be improved or restored by BCI use.

Diagnosis description rated the specificity with which participants’ medical diagnoses were described. Raters selected checkbox options including (1) specific (mention of a diagnostic label with location or type of onset, e.g., brainstem stroke or bulbar onset amyotrophic lateral sclerosis), (2) basic (mention of a diagnostic label without a location or type of onset, e.g., stroke or amyotrophic lateral sclerosis), or vague (no mention of a diagnostic label). The specific diagnostic labels present were inputted as text and used to explore the variety of diagnoses among study participants. Branching logic in the data collection tool ensured that these variables were rated only for studies in which study participant type was rated as PWD.

Functional description rated the specificity with which participants’ functional abilities were described. Checkbox options included (1) specific (mention of one or more areas of functional impact with a degree of impairment, e.g., severe speech impairment or total locked-in syndrome), (2) basic (mention of one or more areas of functional impact without a degree of impairment, e.g., speech impairment or locked-in syndrome), or (3) not reported. Branching logic was implemented such that selecting a functional description of “specific” or “basic” resulted in an opportunity to rate impairment type. Checkbox options were created to identify areas of functional impairment including (1) physical, (2) speech/voice, (3) cognitive, (4) sensory, and (5) consciousness. Examples of cognitive impairments included attention deficits, neglect, and traumatic brain injury.

The variables of interest currently reported were part of a larger database, not all of which were analyzed in this study due to incomplete data sets. Data were initially collected on these additional variables, but due to the extensive time required to analyze abstracts and record data, extraction of these variables was discontinued to prioritize the variables of interest.

Pilot testing:

Eleven published articles from special journal issues associated with the 1999 and 2013 Meetings (Birbaumer et al., 2000; Middendorf, McMillan, Calhoun, & Jones, 2000; Penny, Roberts, Curran, & Stokes, 2000; Pineda, Allison, & Vankov, 2000; Cohen, Koppel, Malach, & Friedman, 2014; Yang et al., 2014; Ang, et al., 2015; Hill, Kovacs, & Shin, 2015; Kübler, Holz, Sellers, & Vaughan, 2015; Peters, et al., 2015; Riccio et al., 2015) were selected for pilot testing and were coded independently by two trained researchers. A consensus process was completed, leading to iterative design of a rating manual that was used throughout the study. The extracted results of pilot testing articles were not included in the analyses described below.

Procedure:

After the rating manual was completed, one reviewer was assigned to evaluate the 1999 abstracts. A second reviewer was randomly assigned 25% of the 1999 abstracts to establish interrater agreement. For each subsequent Meeting, the two reviewers were randomly assigned half the set of abstracts, with 25% of the set randomly assigned for double-entry to establish interrater agreement.

Reviewers evaluated each abstract from a PDF document and coded the variables of interest directly into the REDCap database. Reviewers selected only one target user type and interrater agreement was calculated as binary (agree or disagree). Reviewers were allowed to select multiple participant types, as some studies included both PWD and people without disabilities. Interrater agreement was calculated as the proportion of cases which agreed divided by total opportunities for agreement. For studies including participants with disabilities, reviewers classified diagnosis description, participant diagnosis, functional description, and types of impairments. Reviewers could report more than one diagnostic label for participants.

Reviewers could select multiple impairment areas; for example, many participants presented with both communication and physical impairments. Reviewers could also select multiple levels of diagnosis description and functional description if there were two or more participants described with varying degrees of specificity. In the analyses of these two variables, a conservative approach was taken by using only the lowest-rated selection (e.g. if both “specific” and “basic” were selected for the same abstract, then only basic was used for analysis).

As abstract lengths varied for different Meetings, an independent reviewer identified the average abstract length for each Meeting using the Word Count feature in Microsoft Word 2010 (Microsoft, Redmond, WA). For Meetings in which the abstract format resembled one- or two-page articles that included an abstract element, text under the abstract subheading was kept as part of the overall word count. Titles, authors and affiliations, keywords, section titles, figure and table captions, acknowledgements, and reference lists (though not in-text citations) were removed from the word count.

Analysis:

Interrater agreement was calculated for each variable of interest. Interrater agreement for target user type, diagnosis description, and functional description was calculated as percent agreement using a binary method for each opportunity of agreement and averaging all opportunities. Interrater agreement for participant type and impairment type, which allowed multiple answers to be selected, was calculated as percent agreement, using a proportional overlap method for each abstract and averaging agreement over the sample. For example, there were five participant types and thus five opportunities for agreement per study.

It should be noted that, due to branching logic, diagnosis description and functional description were only available to rate if it was determined previously that PWD had participated in the study. Further, impairment area was only available to rate if the rater previously indicated a “specific” or “basic” functional description. For each of these subordinate variables, an additional calculated variable was created in the analysis phase to identify if the necessary superordinate variable was selected for the branching logic to become active. These calculated variables were included in their respective interrater agreement sample (i.e., a variable was created which identified if a specific or basic functional description was entered, which also was included in the proportional overlap interrater agreement calculation for impairment type). Interrater agreement was calculated after data entry was complete for the assigned Meeting. If interrater agreement was initially below 80% for the variables of interest, cases of disagreement were resolved by consensus by the two raters.

Only data from the assigned reviewer were included for analysis (the duplicate entry was ignored). Abstracts from each Meeting were filtered by study participant type, and those classified as “perspective/theoretical” (e.g. studies of electrode performance, tutorials, review articles, meta-analyses, or perspective/opinion/commentary), as well as studies with only animal subjects or pre-recorded data sets, were excluded. Remaining abstracts were included for analyses.

To compare each variable across each of the Meetings, simple linear regressions were used to estimate the rate of growth of each variable over time, where the variable was expressed as the fraction of matching abstracts among all abstracts for which there could have been a match. For example, every study could present PWD as the target user type, so this variable was expressed as the number of studies in each Meeting with PWD as the target user type divided by the total number of studies for the Meeting. Since diagnosis description was only assessed for studies that included PWD as participants, this variable was expressed as the number of studies in the Meeting with specific/basic/vague descriptions divided by the total number of studies in the Meeting including PWD as participants. The variables were regressed on the Meeting year to give an estimate of how the proportion of matching studies has changed over time; this estimate is the linear slope from the regression.

However, given the small sample size (number of Meetings considered), the highly variable counts (numerators), and numbers of abstracts (denominators) from Meeting to Meeting, it was necessary to assess the sensitivity of the linear slope, both to spikes of counts in individual Meetings and to the amount of information available in each Meeting for estimating proportions. Thus, jackknife analyses (Tukey, 1958; Efron, 1982) on the linear slope estimates along two axes of sensitivity were completed: (1) sensitivity to spikes in the counts (which may cause bias in the slope) and (2) sensitivity to the accuracy of the proportions (which may increase the variance of the slope estimate). The jackknife analysis systematically eliminated data from each Meeting (to assess (1)) and performed both unweighted and weighted regressions (to assess (2)) on each subset of data, where the weighted regressions used the square root of the total abstract count (roughly, the amount of information available for estimating the proportion) as the weight for the Meeting. The overall jackknife regression estimate was found by aggregating the estimates from each of these 12 five-sized subsamples (one for each Meeting being left out, in both unweighted and weighted versions). To the extent that all of the jackknife estimates for the slope are in good agreement, the original slope estimate can be trusted with higher confidence. Conversely, if some of the jackknife estimates are very different from others, there is evidence that the original slope estimate may be biased or associated with large variance, so confidence in that slope is correspondingly diminished. The confidence is based on augmenting the standard error of the original slope estimate with the variance observed in the jackknife estimates.

Results

A total of 761 abstracts were reviewed, and 134 perspective/theoretical studies were removed. Thus, 627 abstracts were included in the analyses. Interrater agreement was above 80% for all variables of interest.

Across the six Meetings, 365 studies aimed to benefit PWD. Of the 365 studies which aimed to benefit PWD, 132 included PWD as research participants. A total of 427 studies included people without disabilities as research participants. We observed 64 studies which included animals as research participants.

Word counts varied based on Meeting ([mean ± SD] 1999: 1453 ± 832; 2002: 466 ± 101; 2005: 358 ± 142; 2010: 461 ± 85; 2013: 896 ± 165; 2016: 489 ± 105; see Figure 2). There was greater variability and higher median word count at the 1999 Meeting compared to all other Meetings.

Figure 2.

Figure 2.

Open in a new tab

Box and whisker plots of abstract word counts by Meeting. The horizontal line within the box indicates the median; boundaries of the box indicate the 25th and 75th percentile; and the whiskers indicate the highest and lowest non-outlier counts. The dots represent outliers below and above the outer boundaries. Note the larger variation in word counts at the 1999 Meeting.

The fraction of abstracts with PWD as target user type appeared to decline somewhat over time (see Figure 3), and the slope was very similar in all jackknife estimates (all the dotted regression lines in the figure are in close proximity, with similar slopes). Although it is possible to draw alternative lines within the uncertainty bounds that are consistent with no change, the possible placement of such lines is very restricted compared to the relatively unrestricted possibilities for lines representing a declining trend, so the latter is the more plausible scenario. The percentage of abstracts targeting PWD as research beneficiaries is well over 50% for all Meetings, but the slight declining trend may indicate that studies considering applications to other user types are increasing in popularity.

Figure 3.

Figure 3.

Open in a new tab

Percentages of abstracts which identified PWD as beneficiaries of the reported research regressed on Meeting year using jackknife sensitivity analysis. Each individual regression is shown as a dotted line, and error bands are superimposed onto the plot to indicate uncertainty across all combinations. The solid line is the average slope which passes through the middle of the uncertainty. Individual dots are scaled to size to give an impression of the leverage each count exerts on the estimate of the linear slope, as well as the individual weight for estimating the variance of the slope.

A decline of studies including PWD participants from the 1999 Meeting to the 2013 Meeting was observed, though an increase occurred at the 2016 Meeting (see Figure 4). Note the two jackknife regression lines trending more negatively than the average; these regressions occurred when the 2016 BCI abstracts were eliminated, suggesting either a possible change in trend toward including more participants with disabilities compared to previous Meetings, or (perhaps less likely) an unusual spike of abstracts of this type in the 2016 Meeting that may not represent any systematic shift in research priorities. Due to the extra uncertainty created by the high-leverage 2016 point going against the historical trend, any plausible estimate of the linear slope based on these data alone cannot be made with confidence.

Figure 4.

Figure 4.

Open in a new tab

Percentages of studies in each Meeting which included PWD as research participants regressed on Meeting using a jackknife sensitivity analysis. Elements of the figure are as in Figure 3.

When considering participation of PWD in just the studies where PWD were identified as the target user type, the apparent disconnect between the historical declining trend and the 2016 proportion is even starker. As seen in Figure 5, notice the two regression lines trending more negatively than the average, representing the regressions when the 2016 BCI abstracts were eliminated; the plausible explanations of the more negatively trending regressions are the same: either there was a shift in trend toward including more PWD participants among studies aiming to benefit such users, or the large uptick in proportion for the 2016 abstracts represents an unusual spike of abstracts of this type compared to previous Meetings without representing a systematic change in research priorities. Based on these data alone, it is unable to be confidently determined whether the slope is trending upward or downward, or how likely it is that the average percentage will stabilize and remain flat in future Meetings.

Figure 5.

Figure 5.

Open in a new tab

Percentage of studies with PWD as target users which also included PWD as participants regressed on Meeting year using a jackknife sensitivity analysis. Elements of the figure are as in Figure 3.

In comparing the relative proportions of vague, basic, and specific descriptions of PWD participant diagnoses across Meetings (see Figure 6), it is important to remember that the total abstract counts are very small for early Meetings (prior to 2005), and for that reason the variance of the proportion estimates is likely to be quite large for those Meetings. Nevertheless, a persistent trend away from vague descriptions and toward more specific ones appears to emerge as time goes on. In particular, the 2013 and 2016 Meetings show a much higher proportion of specific descriptions compared to all previous Meetings, and these estimates are based on much larger (and more stable) counts.

Figure 6.

Figure 6.

Open in a new tab

Percentages of diagnosis description classifications among studies which included participants with disabilities by Meeting year. Note the general increase in more specific (i.e. less vague) diagnostic descriptions since the 2002 Meeting.

Proportions of various classifications of functional description for each Meeting year are presented in Figure 7. As in Figure 6, there appears to be a trend toward more specific descriptions of functional impairments as time goes on. For the 104 total studies which indicated a specific or basic description of function (1999: n = 6; 2002: n = 4; 2005: n = 8; 2010: n = 14; 2013: n = 26; 2016: n = 46), raters also indicated an area of functional impairment. Physical impairments were consistently the most frequently noted impairment area (total: n = 88; 1999: n = 5; 2002: n = 4; 2005: n = 8; 2010: n = 14; 2013: n = 20; 2016: n = 37) with communication disorders presenting as the second most frequent (total: n = 22; 1999: n = 3; 2002: n = 1; 2005: n = 2; 2010: n = 2; 2013: n = 7; 2016: n = 7). Participants with cognitive impairments were included in studies featured in the 2013 (n = 1) and 2016 Meetings (n = 4). Participants with disorders of consciousness were included in the 2002 (n = 1), 2013 (n = 4), and 2016 Meetings (n = 2). The 2016 Meeting was the first to include studies of participants with vision (sensory) impairments (n = 2).

Figure 7.

Figure 7.

Open in a new tab

Percentages of functional description classifications among studies which included participants with disabilities by Meeting year. Note the increase in specific functional descriptions since the 2002 Meeting.

Discussion

The purpose of this study was to explore trends in study participation by PWD across the six International BCI Meetings from 1999 to 2016, as well as trends in the language used to describe those participants.

Results indicate that 58% of the total studies from the six BCI Meetings aimed to benefit PWD. Although the results of this study suggest that this percentage did not significantly change across the Meetings from 1999 to 2016, a decreasing trend was noted, indicating that inclusion of PWD may be declining over time. One explanation of the slight negative trend is increased exploration of BCI applications in people without disabilities to enhance or improve normal functioning. For example, one of the U.S. Defense Advanced Research Project Agency (DARPA) research funding aims is to restore function after injury, and another is to improve the performance of individuals without disabilities (Miranda et al., 2015).

Despite the large number of studies focusing on BCI technology for PWD, only 22% of studies actually included them as research participants, compared to 68% that included people without disabilities. The level of inclusion of PWD in BCI research did not significantly change across Meetings in proportion to the number of abstracts published, although with just six Meetings to base conclusions on, it may yet be too soon to tell. Some studies include participants without disabilities for initial system testing or proof of concept before exploration of end user performance. Limited participation by PWD may result from challenges in recruiting or accessing populations with disabilities, or from researchers’ hesitancy to work with PWD due to reduced signal quality and artifacts (Kübler et al., 2013).

Notably, the jackknife regression used to explore trends in the participation of PWD suggests a potential shift at the 2016 Meeting to include more participants with disabilities. Change in level of participation of PWDs across Meetings may also reflect preference of funding agencies to issue awards to grant applications with PWD as participants. For example, the stated aims of DARPA’s Reorganization and Plasticity to Accelerate Injury Recovery (REPAIR) program suggest a recent shift to benefit PWD (Defense Advanced Research Projects Agency [DARPA], n.d.). As a second example, the National Institute on Deafness and other Communication Disorders (NIDCD), one U.S. funder within the National Institutes of Health (NIH), did not previously have goals specific to development of BCI, though the NIDCD 2012–2016 strategic plan related to assistive technologies included “…to enhance BCI technologies for communication” and to “promote community-based research and data collection” (National Institute on Deafness and Other Communication Disorders [NIDCD], 2016).

Changes in acceptance criteria for journals publishing BCI-related manuscripts also may have influenced levels of PWD participation in research studies. In previous years, journals did not require participation of PWD due to the then-explorative and iterative development of the BCI field. However, inclusion of study participants with disabilities in studies aiming to benefit them is now required by some journals (e.g. Clinical Neurophysiology and IEEE Transactions on Neural Engineering and Rehabilitation and Clinical Neurophysiology) and preferred by others (e.g. Brain-Computer Interfaces, Assistive Technology, ACM Transactions on Accessible Computing, and Journal of Neural Engineering) (Association for Computing Machinery Transactions on Accessible Computing [TACCESS], n.d.; Ferris, personal communication, July 30, 2018; Malloy, personal communication, July 31, 2018; Ziemann, personal communication, July 31, 2018). Still, there are other scholarly journals which do not have such preferences or requirements in place (Nam, personal communication, July 30, 2018), which supports why there remains limited participation of PWD even in studies which aim to benefit them.

Results suggested a trend toward greater specificity in diagnostic and functional descriptions across the six BCI Meetings. Notably, the 1999 BCI Meeting appeared to provide more detail regarding diagnostic and functional description than the 2002 BCI Meeting, though due to small sample sizes in both Meetings, this difference was not significant. The appearance of a difference is likely due to the lack of restrictions on word count in the 1999 Meeting, resulting in more space for authors to describe their participants in greater detail. In contrast, all later Meetings restricted words counts, which required authors to prioritize other aspects of the study rather than elaborating on participants.

The additional information provided by detailed diagnoses and functional abilities facilitates more accurate comparisons across studies to determine types of BCI systems that will benefit individual patient populations. In the future, journals and BCI Meeting organizers should stress participant diagnostic and functional descriptions in manuscript or abstract submissions.

Results of this study suggest BCI was applied to a greater variety of impairment areas across Meetings. This finding is complemented by the observation of an increase in applications to a variety of disabilities (e.g., locked-in syndrome, amyotrophic lateral sclerosis, primary lateral sclerosis, hemi- and tetraplegia, Duchenne muscular dystrophy, traumatic brain injury, stroke, cerebral palsy, multiple sclerosis, Parkinson’s disease, spinal cord injury, progressive supranuclear palsy, spinocerebellar ataxia, spinal muscular atrophy). These findings may be explained by the increase in number of studies across BCI Meetings and increased diversity of professions within the BCI field, including integration of rehabilitation medical professionals into BCI research teams. The number of research groups in BCI has grown from 22 research labs at the 1999 BCI Meeting to 188 labs at the 2016 meeting (Müller-Putz et al., 2016). It is likely that the number of research teams as well as the diversity of professions has contributed to BCI being tested with a greater variety of PWD.

The previously reported growth in BCI research groups (Huggins, Müller-Putz, & Wolpaw, 2017) and steady growth of total abstracts from the 1999 to 2016 Meeting observed in this study are likely related to increased funding for BCI-related research. For example, in 2002, DARPA launched the Brain Machine Interface program and the Human Assisted Neural Devices program (Miranda et al., 2015). DARPA-funded BCI research related programs also expanded, and now include programs such as Revolutionizing Prosthetics and Hand Proprioception and Touch Interfaces (Miranda et al., 2015). Other funding sources have also prioritized BCI research, as evidenced by President Barack Obama’s Brain Initiative (The White House Office of the Press Secretary, 2013). This growth in funding prioritization as well as popularization of BCI related research contributed to the increased number of studies following the 2002 BCI Meeting.

As the field continues to grow and develop, it is essential that research groups work with populations who will benefit from the research. PWD often present with a variety of impairments that will limit generalization of results from participants without disabilities. Engaging PWD in studies will result in improved ecological validity and lead to novel (and vital) research questions. Future research should include PWD as participants in studies when testing BCI tools. The International BCI Meetings are unique opportunities for BCI researchers to present their latest innovations and research. Future Meetings should consider encouraging researchers to include PWD in their research when appropriate, or to expand proof-of-concept studies to include testing with PWD after presentation at the Meeting. New standards will require research groups to partner with clinical teams or local community partners who have experience working with these populations.

Participatory action research (PAR) and user-centered design (UCD) represent approaches to research in which end-users work together with researchers as colleagues in a process of mutual learning, and where the end-user has influence on the research process (Cornwall & Jewkes, 1995; Abras, Maloney-Krichmar, & Preece, 2004; Baum, MacDougall, & Smith, 2006). These strategies have previously been applied to the field of BCI where research teams integrated PWD into study design, development, refinement, and implementation of BCI systems (Blaine-Moraes, Schaff, Gruis, Huggins, & Wren, 2012, Kübler et al., 2013; Kübler et al., 2014; Peters et al., 2015). Previous work has demonstrated that PWD offer unique insight and can be valuable members of the research team (Huggins, Wren, & Gruis, 2011; Blain-Moreas et al., 2012; Peters et al., 2015). These study design principles lend themselves well to narrowing translational gaps.

Research groups aiming to include PWD are encouraged to consider integration of principles of PAR and UCD, which lend themselves well to the current state of the BCI field in that many studies are moving from the bench to the bedside of PWD. Given that PWD are the target end-users of communication and control-based BCI, UCD principles recommend including these individuals in the design and testing of products (Abras et al., 2004). Kübler et al. (2014) outlined six principles and four stages of UCD as it applies to BCI, which can serve as a foundation for research teams seeking PWD involvement.

Limitations

Training and development of the rating manual was completed using only Meeting-related published journal articles from 1999 and 2013, which may not have reflected the diversity of study types or page length in the Meeting abstracts. Future studies should consider randomly sampling training abstracts across Meetings to better reflect the diversity of study types and detail provided.

Assignment of abstracts was completed Meeting-by-Meeting and required that the reviewers rate all abstracts within the assigned Meeting prior to being assigned abstracts for the following Meeting. While this allowed reviewers to familiarize themselves with the type and length of abstracts from an individual Meeting, it is possible that the reviewers’ schemata for rating were influenced by the Meetings they previously evaluated. Future studies should collect and randomize all abstracts into a single assignment and consider including an intrarater assessment and rating fidelity check. Additionally, it was noted that the consensus process for interrater agreement was only completed on the interrater agreement sample and that raters were not required to review other abstracts from the Meeting again unless a change was made to the rating manual.

Conclusion

There was no significant change in trends for participation or description of participants with disabilities across the six International BCI Meetings from 1999 to 2016. Fifty-eight percent of studies identified PWD as being the target beneficiaries of BCI research, though only 22% included participants with disabilities, suggesting evidence of a persistent translational gap. For participants with disabilities, there was no significant change in diagnostic or functional descriptions over the BCI Meeting series. Given studies documenting performance differences between PWD and controls without disabilities, researchers are encouraged to use PAR and UCD strategies to engage individuals with disabilities in their research, and to provide detailed participant descriptions.

Figure 1.

Figure 1.

Open in a new tab

Flow diagram for record inclusion. Records identified as perspective or theoretical were removed prior to analysis. Adapted from Moher, Liberati, Tetzlaff, Altman, and The PRISMA Group (2009).

Table 1:

Brain-Computer Interface Society Meetings 1999–2016

Meeting year Abstracts reviewed Abstracts analyzed Studies aiming to benefit PWD Studies with PWD as participants
1999 22 20 14 7
2002 36 18 15 5
2005 119 71 42 15
2010 165 130 69 18
2013 185 179 111 29
2016 234 209 114 58
Total 761 627 365 132
Open in a new tab

Note. Perspective/theoretical abstracts were removed prior to analysis.

Acknowledgements:

This work was supported by the National Institutes of Health, grant #2RO1DC009834; the National Institute on Disability, Independent Living, and Rehabilitation Research, grant #90RE5017; and the National Center for Advancing Translational Sciences of the National Institutes of Health under award number UL1TR0002369 and UL1TR002240. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We acknowledge Michelle Kinsella, Aimee Mooney, and Steven Bedrick for their support and theoretical considerations to this study.

Funding details: This work was supported by the National Institutes of Health National Institute on Deafness and Other Communication Disorders grant number 2RO1DC009834; the National Institute on Disability, Independent Living, and Rehabilitation Research grant number 90RE5017; and the National Center for Advancing Translational Sciences of the National Institutes of Health under award number UL1TR002240. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Data Availability: The data that support the findings of this study are available from the corresponding author, BE, upon reasonable request.

Footnotes

Disclosures: The authors report no conflict of interest.

References

  1. Abras C, Maloney-Krichmar D, & Preece J. (2004). User-centered design In Bainbridge W. (Ed.), Encyclopedia of Human-Computer Interaction (pp. 445–456). Thousand Oaks, CA: Sage Publications. doi: 10.1108/09504120610685373 [DOI] [Google Scholar]
  2. Akram F, Han SM, & Kim T-S (2015). An efficient word typing P300-BCI system using a modified T9 interface and random forest classifier. Computers in Biology and Medicine, 56, 30–36. doi: 10.1016/j.compbiomed.2014.10.021 [DOI] [PubMed] [Google Scholar]
  3. Ang KK, Guan C, Phua KS, Wang C, Zhao L, Teo WP, … Chew E. (2015). Facilitating effects of transcranial direct current stimulation on motor imagery brain-computer interface with robotic feedback for stroke rehabilitation. Archives of Physical Medicine and Rehabilitation, 96(3, Supplement), S79–S87. doi: 10.1016/j.apmr.2014.08.008 [DOI] [PubMed] [Google Scholar]
  4. Association for Computing Machinery Transactions on Accessible Computing (n.d.). Information for Authors. Retrieved October 11, 2018, from https://www.rit.edu/gccis/taccess/authors.html
  5. Baum F, MacDougall C, & Smith D. (2006). Participatory action research. Journal of Epidemiology and Community Health, 60(10), 854–857. doi: 10.1136/jech.2004.028662 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Baykara E, Ruf CA, Fioravanti C, Käthner I, Simon N, Kleih SC, … Halder S. (2016). Effects of training and motivation on auditory P300 brain–computer interface performance. Clinical Neurophysiology, 127(1), 379–387. 10.1016/j.clinph.2015.04.054 [DOI] [PubMed] [Google Scholar]
  7. Birbaumer N, Kübler A, Ghanayim N, Hinterberger T, Perelmouter J, Kaiser J, … Flor H. (2000). The thought translation device (TTD) for completely paralyzed patients. IEEE Transactions on Rehabilitation Engineering, 8(2), 190–193. doi: 10.1109/86.847812 [DOI] [PubMed] [Google Scholar]
  8. Blain-Moraes S, Schaff R, Gruis KL, Huggins JE, & Wren PA (2012). Barriers to and mediators of brain–computer interface user acceptance: Focus group findings. Ergonomics, 55(5), 516–525. doi: 10.1080/00140139.2012.661082 [DOI] [PubMed] [Google Scholar]
  9. Carelli L, Solca F, Faini A, Meriggi P, Sangalli D, Cipresso P, … Poletti B. (2017). Brain-computer interface for clinical purposes: Cognitive assessment and rehabilitation. Biomedical Research International, 2017, 1695290. doi: 10.1155/2017/1695290 [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cohen O, Koppel M, Malach R, & Friedman D. (2014). Controlling an avatar by thought using real-time fMRI. Journal of Neural Engineering, 11(3), 035006. doi: 10.1088/1741-2560/11/3/035006 [DOI] [PubMed] [Google Scholar]
  11. Cornwall A. & Jewkes R. (1996). What is participatory research? Social Science & Medicine, 41(12), 1667–1676. doi: 10.1016/0277-9536(95)00127-S [DOI] [PubMed] [Google Scholar]
  12. Defense Advanced Research Projects Agency (n.d.). Reorganization and Plasticity to Accelerate Injury Recovery (REPAIR) (Archived). Retrieved October 11, 2018, from https://www.darpa.mil/program/reorganization-and-plasticity-to-accelerate-injury-recovery [Google Scholar]
  13. Efron B. (1982). The jackknife, the bootstrap, and other resampling plans. Presentation at the Society for Industrial and Applied Mathematics; Philadelphia, PA. doi: 10.1137/1.9781611970319 [DOI] [Google Scholar]
  14. Ferris D, personal communication, July 30, 2018. [Google Scholar]
  15. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, & Conde JG (2009). Research electronic data capture (REDCap) – A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381. doi: 10.1016/j.jbi.2008.08.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. He Y, Eguren D, Azorin JM, Grossman RG, Luu TP, & Contreras-Vidal JL (2018). Brain-machine interfaces for controlling lower-limb powered robotic systems. Journal of Neural Engineering, 15(2), 021004. doi: 10.1088/1741-2552/aaa8c0 [DOI] [PubMed] [Google Scholar]
  17. Hill K, Kovacs T, & Shin S. (2015). Critical issues using brain-computer interfaces for augmentative and alternative communication. Archives of Physical Medicine and Rehabilitation, 96(3, Supplement), S8–S15. doi: 10.1016/j.apmr.2014.01.034 [DOI] [PubMed] [Google Scholar]
  18. Huggins JE, Müller-Putz G, & Wolpaw JR (2017). The sixth International Brain–Computer Interface Meeting: Advances in basic and clinical research. Brain Computer Interfaces, 4(1–2): 1–2. doi: 10.1080/2326263X.2017.1328211 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Huggins JE & Wolpaw JR (2014). Papers from the fifth International Brain-Computer Interface Meeting. Journal of Neural Engineering, 11(3). doi: 10.1088/1741-2560/11/3/030301 [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Huggins JE, Wren PA, & Gruis KL (2011). What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotrophic Lateral Sclerosis, 12(5), 318–324. doi: 10.3109/17482968.2011.572978 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Hwang H-J, Lim J-H, Jung Y-J, Choi H, Lee SW, & Im C-H (2012). Development of an SSVEP-based BCI spelling system adopting a QWERTY-style LED keyboard. Journal of Neuroscience Methods, 208(1), 59–65. doi: 10.1016/j.jneumeth.2012.04.011 [DOI] [PubMed] [Google Scholar]
  22. Ikegami S, Takano K, Kondo K, Saeki N, & Kansaku K. (2014). A region-based two-step P300-based brain–computer interface for patients with amyotrophic lateral sclerosis. Clinical Neurophysiology, 125(11), 2305–2312. doi: 10.1016/j.clinph.2014.03.013 [DOI] [PubMed] [Google Scholar]
  23. Ikegami S, Takano K, Saeki N, & Kansaku K. (2011). Operation of a P300-based brain–computer interface by individuals with cervical spinal cord injury. Clinical Neurophysiology, 122(5), 991–996. doi: 10.1016/j.clinph.2010.08.021 [DOI] [PubMed] [Google Scholar]
  24. Kaufmann T, Schulz SM, Köblitz A, Renner G, Wessig C, & Kübler A. (2013). Face stimuli effectively prevent brain–computer interface inefficiency in patients with neurodegenerative disease. Clinical Neurophysiology, 124(5), 893–900. doi: 10.1016/j.clinph.2012.11.006 [DOI] [PubMed] [Google Scholar]
  25. Kübler A. (2013). Brain-computer interfacing: Science fiction has come true. Brain, 136(6), 2001–2004. doi: 10.1093/brain/awt077 [DOI] [Google Scholar]
  26. Kübler A, Holz E, Kaufmann T & Zickler C. (2013). A user centred approach for bringing BCI controlled applications to end-users. In Fazel-Rezai R. (Ed.), Brain-Computer Interface Systems: Recent Progress and Future Prospects. doi: 10.5772/50696 [DOI] [Google Scholar]
  27. Kübler A, Holz EM, Riccio A, Zickler C, Kaufmann T, Kleih SC, … Mattia D. (2014). The user-centered design as novel perspective for evaluating the usability of BCI-controlled applications. PLoS One, 9(12), e112392. doi: 10.1371/journal.pone.0112392 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Kübler A, Holz EM, Sellers EW, & Vaughan TM (2015). Toward Independent Home Use of Brain-Computer Interfaces: A Decision Algorithm for Selection of Potential End-Users. Archives of Physical Medicine and Rehabilitation, 96(3, Supplement), S27–S32. doi: 10.1016/j.apmr.2014.03.036 [DOI] [PubMed] [Google Scholar]
  29. Malloy A, personal communication, July 31, 2018. [Google Scholar]
  30. McCane LM, Heckman SM, McFarland DJ, Townsend G, Mak JN, Sellers EW, … Vaughan TM (2015). P300-based brain-computer interface (BCI) event-related potentials (ERPs): People with amyotrophic lateral sclerosis (ALS) vs. age-matched controls. Clinical Neurophysiology, 126(11), 2124–2131. doi: 10.1016/j.clinph.2015.01.013 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Middendorf M, McMillan G, Calhoun G, & Jones KS (2000). Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Transactions on Rehabilitation Engineering, 8(2), 211–214. doi: 10.1109/86.847819 [DOI] [PubMed] [Google Scholar]
  32. Miranda RA, Casebeer WD, Hein AM, Judy JW, Krotkov EP, Laabs T, … Ling GSF (2015). DARPA-funded efforts in the development of novel brain-computer interface technologies. Journal of Neuroscience Methods, 244, 52–67. doi: 10.1016/j.jneumeth.2014.07.019 [DOI] [PubMed] [Google Scholar]
  33. Moghimi S, Kushki A, Guerguerian AM, & Chau T. (2013). A review of EEG-based brain-computer interfaces as access pathways for individuals with severe disabilities. Assistive Technology, 25(2), 99–110. doi: 10.1080/10400435.2012.723298 [DOI] [PubMed] [Google Scholar]
  34. Moher D, Liberati A, Tetzlaff J, Altman DG, & The PRISMA Group (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. doi: 10.1371/journal.pmed.1000097 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Monge-Pereira E, Ibanez-Pereda J, Alguacil-Diego IM, Serrano JI, Spottorno-Rubio MP, & Molina-Rueda F. (2017). Use of electroencephalography brain-computer interface systems as a rehabilitative approach for upper limb function after a stroke: A systematic review. PM&R, 9(9), 918–932. doi: 10.1016/j.pmrj.2017.04.016 [DOI] [PubMed] [Google Scholar]
  36. Müller-Putz G, Huggins J, & Steyrl D. (Eds.). (2016). Proceedings of the 6th International Brain-Computer Interface Meeting: BCI Past, Present, and Future. Verlag der Technischen Universität Graz. doi: 10.3217/978-3-85125-467-9 [DOI] [Google Scholar]
  37. Nam C, personal communication, July 30, 2018. [Google Scholar]
  38. National Institute on Deafness and Other Communication Disorders (2016, September 26). 2012–2016 NIDCD Strategic Plan. Retrieved October 11, 2018, from https://www.nidcd.nih.gov/about/strategic-plan/2012-2016/2012-2016-nidcd-strategic-plan
  39. Nijboer F, Sellers EW, Mellinger J, Jordan MA, Matuz T, Furdea A, … Kübler A. (2008). A P300-based brain–computer interface for people with amyotrophic lateral sclerosis. Clinical Neurophysiology, 119(8), 1909–1916. doi: 10.1016/j.clinph.2008.03.034 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Oken BS, Orhan U, Roark B, Erdogmus D, Fowler A, Mooney A, … Fried-Oken MB (2014). Brain–computer interface with language model–electroencephalography fusion for locked-in syndrome. Neurorehabilitation and Neural Repair, 28(4), 387–394. doi: 10.1177/1545968313516867 [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Penny WD, Roberts SJ, Curran EA, & Stokes MJ (2000). EEG-based communication: a pattern recognition approach. IEEE Transactions on Rehabilitation Engineering, 8(2), 214–215. doi: 10.1109/86.847820 [DOI] [PubMed] [Google Scholar]
  42. Peters B, Bieker G, Heckman SM, Huggins JE, Wolf C, Zeitlin D, & Fried-Oken M. (2015). Brain-computer interface users speak up: The Virtual Users’ Forum at the 2013 International Brain-Computer Interface Meeting. Archives of Physical Medicine and Rehabilitation, 96(3, Supplement), S33–S37. doi: 10.1016/j.apmr.2014.03.037 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Pineda JA, Allison BZ, & Vankov A. (2000). The effects of self-movement, observation, and imagination on mu rhythms and readiness potentials (RP’s): toward a brain-computer interface (BCI). IEEE Transactions on Rehabilitation Engineering, 8(2), 219–222. doi: 10.1109/86.847822 [DOI] [PubMed] [Google Scholar]
  44. Riccio A, Holz EM, Aricò P, Leotta F, Aloise F, Desideri L, … Cincotti F. (2015). Hybrid P300-based brain-computer interface to improve usability for people with severe motor disability: Electromyographic signals for error correction during a spelling task. Archives of Physical Medicine and Rehabilitation, 96(3, Supplement), S54–S61. doi: 10.1016/j.apmr.2014.05.029 [DOI] [PubMed] [Google Scholar]
  45. Severens M, Van der Waal M, Farquhar J, & Desain P. (2014). Comparing tactile and visual gaze-independent brain–computer interfaces in patients with amyotrophic lateral sclerosis and healthy users. Clinical Neurophysiology, 125(11), 2297–2304. doi: 10.1016/j.clinph.2014.03.005 [DOI] [PubMed] [Google Scholar]
  46. Silvoni S, Konicar L, Prats-Sedano MA, Garcia-Cossio E, Genna C, Volpato C, … Birbaumer N. (2016). Tactile event-related potentials in amyotrophic lateral sclerosis (ALS): Implications for brain-computer interface. Clinical Neurophysiology, 127(1), 936–945. doi: 10.1016/j.clinph.2015.06.029 [DOI] [PubMed] [Google Scholar]
  47. The White House Office of the Press Secretary (2013, April 02). Fact Sheet: BRAIN Initiative. Retrieved October 11, 2018, from https://obamawhitehouse.archives.gov/the-press-office/2013/04/02/fact-sheet-brain-initiative
  48. Tukey JW (1958). Bias and confidence in not quite large samples. Annals of Mathematical Statistics, 29, 614. doi: 10.1214/aoms/1177706647 [DOI] [Google Scholar]
  49. Vaughan TM & Wolpaw JR (2011). Special issue containing contributions from the fourth International Brain-Computer Interface Meeting. Journal of Neural Engineering, 8(2). doi: 10.1088/1741-2560/8/2/020201 [DOI] [PubMed] [Google Scholar]
  50. Vaughan TM, McFarland DJ, Schalk G, Sarnacki WA, Krusienski DJ, Sellers EW, & Wolpaw JR (2006). The Wadsworth BCI research and development program: At home with BCI. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 14(2), 229–233. doi: 10.1109/TNSRE.2006.875577 [DOI] [PubMed] [Google Scholar]
  51. Vaughan TM, Heetderks WJ, Trejo LJ, Rymer WZ, Weinrich M, Moore MM, … Wolpaw JR (2003). Brain-computer interface technology: A review of the second International Meeting. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11(2), 94–109. doi: 10.1109/TNSRE.2003.814799 [DOI] [PubMed] [Google Scholar]
  52. Vaughan TM & Wolpaw JR (2006). The third International Meeting on brain-computer interface technology: Making a difference. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 14(2), 126–127. doi: 10.1109/TNSRE.2006.875649 [DOI] [PubMed] [Google Scholar]
  53. Wolpaw JR & Wolpaw EW (2012). Brain-Computer Interfaces: Principles and Practice. Albany, NY: Oxford University Press. doi: 10.1093/acprof:oso/9780195388855.001.0001 [DOI] [Google Scholar]
  54. Wolpaw JR, Birbaumer N, Heetderks WJ, McFarland DJ, Peckham PH, Schalk G, … Vaughan TM (2000). Brain-computer interface technology: a review of the first international meeting. IEEE Transactions on Rehabilitation Engineering, 8(2), 164–173. doi: 10.1109/TRE.2000.847807 [DOI] [PubMed] [Google Scholar]
  55. Yang H, Guan C, Chua KS, Chok SS, Wang CC, Soon PK, … Ang KK (2014). Detection of motor imagery of swallow EEG signals based on the dual-tree complex wavelet transform and adaptive model selection. Journal of Neural Engineering, 11(3), 035016. doi: 10.1088/1741-2560/11/3/035016 [DOI] [PubMed] [Google Scholar]
  56. Ziemann U, personal communication, July 31, 2018. [Google Scholar]

RESOURCES