Effects of simulated visual acuity and ocular motility impairments on SSVEP brain-computer interface performance: An experiment with Shuffle Speller

Abstract

Individuals with severe speech and physical impairments may have concomitant visual acuity impairments (VAI) or ocular motility impairments (OMI) impacting visual BCI use. We report on the use of the Shuffle Speller typing interface for an SSVEP BCI copy-spelling task under three conditions: simulated VAI, simulated OMI, and unimpaired vision. To mitigate the effect of visual impairments, we introduce a method that adaptively selects a user-specific trial length to maximize expected information transfer rate (ITR); expected ITR is shown to closely approximate the rate of correct letter selections. All participants could type under the unimpaired and simulated VAI conditions, with no significant differences in typing accuracy or speed. Most participants (31 of 37) could not type under the simulated OMI condition; some achieved high accuracy but with slower typing speeds. Reported workload and discomfort were low, and satisfaction high, under the unimpaired and simulated VAI conditions. Implications and future directions to examine effect of visual impairment on BCI use is discussed.

1. Introduction

Brain-computer interface (BCI) technology has been implemented as an augmentative and alternative communication (AAC) access method for people with severe speech and physical impairment (SSPI), [1,2] often resulting from locked-in syndrome (LIS). [3,4] Many traditional or popular access methods are ineffective for this population due to their limited voluntary motor function. Even eye tracking, which requires only eye movement, does not work for some potential users. [5] BCI may offer a viable alternative. Researchers have investigated BCI spelling interfaces incorporating a variety of brain signals and stimulus presentation methods, but users with disabilities typically demonstrate weaker BCI typing performance than healthy controls. Individuals with SSPI may present with comorbid sensory deficits affecting their ability to perceive BCI stimuli and user interfaces. There has been relatively little exploration of these potentially confounding challenges.

BCI spelling systems often rely on visual stimuli and user interfaces presented via screens. [1] Although individual system requirements differ, use of visually-presented BCI typically requires skills including visual acuity to discriminate the desired target, ocular motility to direct gaze to a stimulus, and the ability to maintain fixation on that stimulus. [6–8] These requirements are critical, as users with SSPI frequently present with impairments that can degrade visual BCI performance. Visual acuity often decreases with age in the general population, [9] and may be higher among people with LIS. [10,11] Reduced ocular motility, as well as other symptoms including visual fixation impairment, nystagmus, diplopia, and gaze impersistence, have been observed in individuals with LIS [10] and advanced amyotrophic lateral sclerosis (ALS). [12]

Literature addressing the effects of reduced visual acuity or other ocular impairments specifically on BCI use is limited. Study participants’ visual skills are rarely assessed or reported, though some researchers have suggested such impairments as possible explanations for poor BCI spelling performance among users with SSPI. [13,14] The role of ocular motility in use of the popular P300 matrix speller has been investigated using eye tracking to monitor the eye movements of healthy participants. Such systems have been found to require overt visual attention, dependent on the user’s ability to direct her gaze with eye movement; selection accuracy is both lower and less consistent across participants when gaze is restricted. [6,15,16] Alternative P300 speller layouts which reduce visual crowding, arrange targets near a fixation point, and utilize visual cues (e.g. color, shape, or movement) demonstrate improved performance under covert attention (fixed-gaze) conditions. [16,17] Studies of individuals with SSPI using covert-attention or gaze-independent visual BCIs are limited, and typically report relatively poor performance. [18,19]

Steady state visually evoked potential (SSVEP) BCIs may offer an advantage over P300-based systems for some users with severe disabilities. [20] The SSVEP response is elicited by rapidly oscillating visual stimuli, such as flashing lights, and can be measured via electroencephalography (EEG). In a typical SSVEP-based BCI, multiple stimuli flicker at different frequencies. The user attends to a particular stimulus to select a target, producing an SSVEP response to the target stimulus frequency. By observing the EEG’s frequency response, a BCI can infer the user’s intent. Like the P300, SSVEP is more effective for BCI control when used with overt attention. [21,22] Allison and colleagues observed substantial inter-participant differences in SSVEP signals for healthy participants under covert attention conditions with a two-class system; only half of participants produced SSVEPs sufficient for potential control of a binarychoice BCI. [22] In another study, five individuals with advanced ALS and impaired eye movement used a four-stimulus SSVEP BCI as a simple, phrase-based communication system with varying degrees of success. [23] Brumberg and colleagues used a four-stimulus SSVEP BCI (without a communication task) with people with SSPI, some with reduced ocular motility. Two of their five participants achieved classification levels significantly better than chance. [24] Lesenfants and colleagues tested a covert-attention SSVEP-based system for binary-choice communication (e.g. yes/no) with healthy participants and people with LIS. Eight of 12 healthy users and two of four users with LIS achieved accuracy above chance levels when answering yes/no questions. [25]

In this article, the Shuffle Speller interface is explored as a potential means of mitigating the effects of visual impairments on SSVEP BCI typing. Shuffle Speller is a typing interface which makes the most of the control signals produced by individual users. [26] Its algorithm learns each user’s unique pattern of responses and errors, and adapts stimulus presentation accordingly. It aggregates results from multiple user selections before typing a letter, increasing the likelihood that the letter will match the user’s intent. Here we describe how error-prone SSVEP BCI classifications, such as those that might be seen with covert attention, can be combined to produce a single, accurate character selection. We also introduce a method of maximizing how quickly the system can infer a user’s intent by selecting a trial length tailored to a particular user and classifier. Shorter trial lengths offer more frequent opportunities for users to express their intent. On the other hand, longer trial lengths provide more physiological evidence to classify, typically yielding higher accuracies. Adaptive trial length selection leverages the trade-offs presented by a specific user’s calibration data to maximize user communication rates. Specifically, we optimize expected information transfer rate (ITR), which we demonstrate to be a strong proxy for the rate of correct character selections. Adaptive trial length selection may be useful in mitigating the effects of visual impairments.

Given the prevalence of visual acuity impairments (VAI) and ocular motility impairments (OMI) among people with SSPI, and the potential impacts of these impairments on visual BCI use, we investigate the effects of simulated VAI and OMI on Shuffle Speller SSVEP typing performance. VAI was simulated using blurred goggles. OMI was simulated by confining the user’s gaze within a circle, using eye tracking to ensure compliance, and discarding trials in which the user’s gaze wanders outside the circle. We hypothesized that simulated VAI would not significantly affect Shuffle Speller typing accuracy, as the interface had been modified for this experiment with VAI in mind. Based on previous reports in the literature and on preliminary, informal explorations of SSVEP typing with Shuffle Speller, we expected highly variable performance under the simulated OMI condition, with some participants able to type accurately and others unable to type at all. Finally, we expected generally positive user experience (UX) ratings (low discomfort, low workload, and high satisfaction) for Shuffle Speller under the unimpaired and simulated VAI conditions, with more negative ratings under the simulated OMI condition.

2. Methods

2.1. Shuffle Speller

Shuffle Speller is an algorithm that adapts to an individual user’s unique abilities to improve typing performance. While full technical details are reported in previous work, [26] we describe here the features of Shuffle Speller which are relevant to mitigating the effects of visual impairments, and how the user interface was configured for this study. See supplemental online materials for videos and additional information.

The Shuffle Speller interface includes a dashboard area which displays the typed string (and target string, for copy-spelling tasks) and messages about task status (e.g. complete, break), as well as a keyboard area with colorful boxes used for making letter selections (Figure 1). LED arrays are placed around the screen next to each box for use with SSVEP BCI. Typing begins with the 26 letters of the English alphabet, plus space and backspace characters (_ and <, respectively), arranged alphabetically in rows in the center of the keyboard area. The user finds her target letter and follows it as it moves to one of four boxes around the perimeter of the screen. To facilitate letter tracking, all boxes are given a unique color, and letters are displayed in the color of the next destination box. The user indicates which box contains her target character by attending to the LED array nearest that box. After SSVEP stimulation and classification, all boxes are shaded, with color intensity corresponding to the system’s confidence that a given box represents the user’s intended selection. If the probability of any character in a selected box exceeds a predetermined confidence threshold (e.g. 85%), it is typed in the dashboard area, and the user is notified via an audio cue (the letter name is spoken aloud in a synthesized voice). Otherwise, the system queries again by shuffling the characters into different groups, and the user selects another box. All 28 characters remain visible at all times during typing, allowing users the opportunity to recover and type their desired character even after an incorrect selection on a single query. Queries are repeated, and the system incorporates SSVEP evidence from each box selection, until the confidence threshold is reached for some letter.

Figure 1.:

Shuffle Speller user interface

Many Shuffle Speller interface features can be customized, including shuffle animation speed, pre-shuffle delay time, and the number, size, color, and position of boxes. The system automatically adjusts the number of queries per letter selection and the partitioning of letters into boxes to meet the needs of individual users, based on the varying strengths of their SSVEP responses. SSVEP response strengths vary both across and within users (as in [24]). Figure 2 displays confusion matrices representing SSVEP response patterns for two hypothetical users. The two users have identical average accuracy, but substantial differences in the reliability of SSVEP estimation for individual stimuli. Shuffle Speller learns the varying trustworthiness of each SSVEP response so that SSVEP classifications are incorporated into letter selection in a principled manner. Confusion matrix (a) represents a user with uniform reliability for all stimuli and uniform distribution of errors, an unlikely occurrence. Confusion matrix (b) demonstrates a more varied response pattern. An estimate indicating a response to the 9.7 Hz from user (b) should elicit some skepticism from the system, as it is known to be less accurate. In this case Shuffle Speller hedges by obtaining additional evidence, querying the user again and avoiding selecting a letter too hastily. Alternatively, Shuffle Speller puts its trust in an 8 Hz response from user (b) by advancing towards a letter selection with fewer queries, as excessive querying would only result in slower letter selection. The user-specific model of SSVEP response accuracy is constructed from calibration data. It describes the expected accuracy and errors associated with each SSVEP response for a given user. This model allows the system to trust responses that are associated with higher accuracies in the calibration. By doing so, Shuffle Speller adaptively paces letter selection. Users with error-prone SSVEP classifications type accurately by responding to additional queries, while users with accurate SSVEP classifications type quickly by avoiding unnecessary querying. Above some minimum SSVEP accuracy threshold, Shuffle Speller allows everyone to select letters accurately; only typing speed is impacted by poor SSVEP accuracy.

Figure 2.:

Hypothetical Confusion Matrices. Note that (a) and (b) have the same average accuracy despite their differences. As typically defined, ITR imposes symmetry assumptions which effectively approximate (b) as (a). Expected ITR, as defined in (1), does not impose these symmetry assumptions.

The partitioning of letters among boxes during typing is user-specific and determined by calibration data. Shuffle Speller distributes characters in such a way as to optimize expected ITR (see Sec 2.2) for an individual user; this feature is central to its performance. Shuffle Speller will avoid presenting characters in boxes associated with inaccurate SSVEP responses, instead preferring those associated with higher SSVEP accuracy. This allows the system to maximize the reliability and usefulness of the evidence it receives from each query. The character partitioning and selection algorithms utilize data from all previous user inputs, the user input error model (indicating the reliability of each SSVEP response), and an integrated 5-gram character-based language model. (The same language model was used in previous work [19]; see [27] for a review of language model use in BCI.) These data are aggregated to determine the probability of each character after each query [26].

2.2. Adaptive Trial Length Selection

We introduce a method of selecting a trial length that maximizes ITR:

$$ITR=\frac{1}{T}I(X,\widehat{X})=\frac{1}{T}\sum _{x\in X\widehat{x}\in \widehat{X}}\frac{P_{X,\widehat{X}}(x,\widehat{x})}{P_X(X)P_{\widehat{X}}(\widehat{x})}$$ (1)

where x is the target SSVEP frequency, $\widehat{x}$ is the system’s estimate of this frequency, I is the Mutual Information function, and T is the trial length (including both stimulation and inter-trial wait time). Please note that this definition is a generalization of the commonly used formula in BCI literature. To provide a complete motivation of our trial length selection procedure we give a brief description of its difference here. This information is not novel within the BCI field; we strongly encourage unfamiliar readers to review [28–34].

Observe that $P_{X,\widehat{X}}=P_{\widehat{X}|X}{P}_X$. ITR definitions vary in their estimation of these factors. We describe each term below and outline how our own assumptions differ from the popular definition.

The first term, $P_{\widehat{X}|X}(\widehat{x}|x)$, characterizes how often the system classifies in favor of frequency $\widehat{x}$ given that the user observed stimulation at frequency x. This conditional distribution defines a user-specific SSVEP generation model; it represents how often the system accurately identifies each SSVEP stimulation frequency (and, when a misclassification is made, which frequency is selected instead). $P_{\widehat{X}|X}(\widehat{x}|x_0)$ is estimated by normalizing a histogram of all calibration trials in which the user was stimulated at frequency x₀. The common definition of ITR assumes that each x is equally accurate and that errors are uniformly distributed among unintended SSVEP stimuli. This may not be the case as the positioning of the LED arrays or a user’s SSVEP responses may result in some errors occurring more frequently than others, as in confusion matrix (b) in Figure 2.¹

The second term, P_X, describes the prior belief (before EEG data is obtained) that each SSVEP stimulus will be selected. Commonly, this term is assumed to be uniform, though in practice this is rarely the case. Specifically, in the spelling task P_X(x₀) is the sum of the probabilities of all letters which are associated with SSVEP stimulus x₀. Shuffle Speller’s performance benefit is derived from the optimization of P_X (see [26]). It maps letters to X (i.e. animates letters to different boxes) to approximate:

$$P_X^{\prime }=\arg \underset{{}_{P_X}}{\max }\frac{1}{T}I(X,\widehat{X}).$$ (2)

Note that estimating P_X and $P_{\widehat{X}|X}$ in this way uses only calibration data. It yields a prediction of correct typing rate, not an observation. To remind ourselves of this fact (and distinguish it from other ITR definitions) we denote it as expected ITR. As will be shown later, expected ITR for Shuffle Speller is a strong predictor of the correct character selection rate (Figure 6). As such, it serves as a useful guide for the effect of parameter changes. We use it here to optimize the stimulation time of a single SSVEP trial. Intuitively, a longer SSVEP stimulus duration mitigates the effects of noise and artifacts to produce a more accurate SSVEP classification. However, excessively long trial lengths slow down the rate at which SSVEP queries can be presented. To select a trial length which balances these effects, the system calibrates on longer-than-necessary SSVEP stimulus durations. Next, we construct calibration datasets for different stimulation times by truncating to various durations. Finally, we select the trial length associated with the calibration dataset which has the largest expected ITR.

Figure 6.:

Expected ITR vs. correct typing speed

This method provides a principled way of tailoring the system around a user-specific model of SSVEP generation. Note that our selection of T seeks only to type correctly and quickly; it does not take into account user comfort. If such an innovation were to be used outside the research environment it is critical to provide users with as much control as possible in customizing the system to their needs. The quantification described above is still valuable: it allows users and caregivers to explicitly understand how much correct typing speed is traded away for their comfort.

2.3. Study design

This study followed a randomized, balanced crossover design in which each participant used Shuffle Speller with SSVEP BCI under three conditions: unimpaired vision, simulated VAI, and simulated OMI. Participants completed a total of four study visits, approximately one week apart, in a quiet office environment. The first visit included informed consent, vision screening, unimpaired SSVEP calibration and a one-word copy-spelling trial with Shuffle Speller. The initial copy-spelling trial was intended to introduce participants to the interface and exclude the effects of first time use in the experiment; these data were not included in analysis. During each of the remaining three visits, participants completed copy-spelling tasks with Shuffle Speller under one of the three experimental conditions (unimpaired, VAI, or OMI). Condition orders were randomly assigned in a counterbalanced manner.

2.4. Participants and screening procedures

Participants recruited for this study were healthy adults aged 21–80 years, able to read and communicate in English. Initial screening was done by telephone, and potential participants were excluded if they reported reduced visual acuity, retinal disease, or photosensitive seizure disorder, or if they scored lower than 32 on the Telephone Interview for Cognitive Status-Modified (TICS-m), an indication of possible cognitive impairment. [35]

During the first study visit, participants underwent a vision screening focused on skills identified as relevant to assistive technology use. [36] Acuity, motility, fixation, and visual field perception were deemed most important for use of Shuffle Speller, and participants were required to meet the following additional inclusion criteria: best corrected binocular vision of 20/40 or better, score of five on all aspects of the Northeastern State University College of Optometry (NSUCO) Oculomotor Test, [37] and lack of visual field impairment. [38] This study was approved by the Oregon Health & Science University (OHSU) Institutional Review Board (approval # 15331), and participants gave written informed consent.

2.5. System configuration and stimulus presentation

Shuffle Speller was constructed using MATLAB (MathWorks, Natick, MA) and Psychtoolbox [39]. A 21-inch monitor was positioned 55cm away from the participant’s face. Four boxes (with dimensions 6°×18°or 15°×7°visual angle) were arranged on the top, right, bottom, and left sides of the keyboard area (Figure 1). Stimulus colors (yellow, lilac, aqua, and red-orange) were chosen to be maximally dissimilar from one another while maintaining high visibility against a black background. Characters were presented in Arial font with visual angles ranging from 1°to 4°. Pre-shuffle delay time (i.e. the time allowed for the user to recognize her target character and note its color before shuffle animation) was five seconds for the initial alphabetical array and two seconds for each subsequent query. Shuffle animation duration (i.e. the time required for letters to move from the alphabetical array to a box or from one box to another) was 1 second. The dashboard area displayed the target word and indicated typed characters. SSVEP stimuli consisted of four LED arrays (1.6°×2.4°visual angle) attached to the monitor, one next to each box. The LED array 1) supports checkerboard stimulation, both on/off and alternating, and 2) has higher luminous flux with adjustable brightness to better tailor the stimulus to the user across distance from the monitor and spacing between targets. Each array included 25 surface-mount LEDs (Kingbright, City of Industry, CA) arranged in a 5×5 square, and was mounted on a 3D-printed car for easy positioning (Figure 3). [40] Stimulation frequencies were set at equally spaced intervals within the alpha band including 8, 9.67, 11.33, and 13 Hz [41].

Figure 3.:

Car and rail design for LED array mounting

2.6. Simulated Impairments

Modifications were made to simulate VAI or OMI during both the calibration and copy-spelling tasks for those conditions. During the VAI condition, participants wore goggles (Fork in the Road, Madison, WI) designed to simulate 20/200 visual acuity. This method of simulation is consistent with other recent studies involving participants without visual impairments. [42,43] During the OMI condition, a fixation circle with a diameter of 13° visual angle was centered in the keyboard area, and an EyeX eye tracker (Tobii, Danderyd, Sweden [44]) was used to ensure that participants kept their gaze within the circle. During pilot testing, a 13° circle was the smallest size that allowed some users to type successfully. If a participant looked outside the circle for more than 10% of the duration of a trial, she was notified by an alert sound and a change in the circle’s color, and was required to repeat that trial.

2.7. Tasks

Each Shuffle Speller session included a calibration and a copy-spelling task. Before each task, participants watched a brief demonstration video, with scripted explanation provided by a researcher. A calibration session consisted of 80 trials (20 per box) lasting six seconds each. During each trial, the participant was instructed to attend to one of the four flashing LEDs, indicated by a box appearing on the monitor next to the target LED. The participant could look directly at the LED during the unimpaired and VAI conditions, but was required to use covert attention during the OMI condition. A 10-second break occurred every 10 trials. Calibration data were used to train a classifier and determine an optimal trial length for copy-spelling, as described in Section 2.2.

Each copy-spelling session featured a different list of ten common five- to seven-letter words drawn from SUBTLEXus, a dataset of high-frequency American English words. [45] The total number of characters (53) was constant across the three word lists. Participants were instructed to select the backspace character (<) to correct any typing errors. Copy-spelling sessions were limited to 20 minutes total, but were ended after five minutes if no letters were selected during that time. For each session, participants completed a BCI-specific UX questionnaire modified from previous work, [46] including questions about workload, comfort, and satisfaction (questionnaire available in supplemental online materials).

2.8. Data acquisition

A non-slip elastic headband (Conair Corp., East Windsor, NJ) was used to position g.BUTTERFLY active electrodes at approximate locations over O1, Oz, and O2 (Figure 4), with Fpz ground and right earclip reference (g.tec, Schiedlberg, Austria). Oz was positioned approximately 3cm above the inion, with O1 and O2 approximately 5% of the head circumference to the left and right of Oz. EEG data were sampled at 256 Hz, recorded with a g.USBamp amplifier (g.tec, Schiedlberg, Austria), and were visually inspected for data quality before each calibration or copy-spelling session. EyeX data were collected at a sampling rate of 60 Hz, using custom MATLAB software and GazeSDK (Tobii, Danderyd, Sweden).

Figure 4.:

Shuffle Speller electrode placement

EEG features for calibration and copy-spelling were obtained using canonical correlation analysis (CCA). The standard CCA approach consists of computing the maximum Pearson’s correlation between a linear combination of all available channels of EEG data and a linear combination of template signals. [47] Our classifier offered confidence estimates alongside its classifications via a kernel density estimation [48] over the CCA features. A classifier was trained at the beginning of each typing session based on individual calibration data.

2.9. Data collection and analysis

The primary dependent variable was typing accuracy (percentage of correct character selections out of total selections; selection of backspace to correct an error was considered a correct response), as potential BCI users have indicated that they are willing to accept lower typing speeds as long as accuracy is high. [49,50] Additional dependent variables included typing speed (characters per minute), trial length (seconds), and UX questionnaire responses. Accuracy and speed were calculated using custom MAT-LAB functions. All data from paper forms and MATLAB-generated spreadsheets were stored using REDCap electronic data capture tools [51] hosted at OHSU.

Accuracy and speed for the unimpaired and VAI conditions were subjected to paired t tests of mean equivalence [52] using the tostt command [53] in Stata 13.1 (StataCorp, College Station, TX). Considering just the one-sided hypothesis of noninferiority, differences were coded so that negative differences would indicate inferior performance under the VAI condition. The combined test of noninferiority includes subtests for both mean difference of the conditions and mean separation between the conditions being smaller than a pre-specified negative value (delta). The delta values for accuracy (0.05) and speed (1 character per minute) were chosen based on pilot testing of the system, and reflect the smallest differences that we consider meaningful. Noninferiority is suggested by large difference p-values and small separation p-values. Due to the small number of participants who were successful under the OMI condition, only descriptive statistics were calculated for OMI accuracy and speed. A McNemar test was used to compare the ratio of successful copy-spellers under the unimpaired and OMI conditions.

3. Results

Thirty-eight participants attempted both the unimpaired and VAI conditions; their demographic information is presented in Table 1. Of those, 37 attempted the OMI condition, as one had to be excluded due to problems calibrating the eye tracker. See the participant flow chart in Figure 5 for additional details on eligibility assessment and study completion. Copy-spelling task results revealed similar letter selection accuracy, typing rate, and UX ratings for the unimpaired and VAI conditions, but significant differences with the OMI condition. Adaptive trial length proved effective for maximizing performance, with longer trials supporting accurate typing for several users under the OMI condition. Detailed results are presented below.

Table 1.:

Participant demographics

	Mean ± SD or n/38
Age (years)	37.2 ± 15.47
Women	26/38
Racial and/or ethnic minority	6/38
Education (years)	17.3 ± 2.41
Reported use of antidepressant, anticonvulsant, and/or drowsiness-inducing medications	5/38
Distance visual acuity (both eyes, corrected)
20/20	31/38
20/25	4/38
20/30	2/38
20/40	1/38
Near visual acuity (both eyes, corrected)
20/20	35/38
25/20	3/38

Figure 5.:

Participant flowchart

3.1. Typing Performance

Only six of the 37 participants who attempted the OMI condition successfully copied one or more words, while all participants did so under the unimpaired and VAI conditions. A McNemar test revealed that simulated OMI significantly affected whether a participant could type with Shuffle Speller (χ²(1,N = 37) = 31.00, p < 0.0001). Details on participants who were successful under the OMI condition are presented in Table 2.

Table 2.:

Successful OMI participant details

ID	Age	Distance visual acuity*	Near visual acuity*	Trial length	Words copied	Accuracy (%)	Speed (char/min)	Session number
A	46	20/20	20/20	8.27	1	100	0.67	1
B	24	20/20	20/20	6.18	4	88.9	1.96	2
C	23	20/20	20/20	7.48	3	85.7	1.52	2
D	62	20/20	20/20	8.01	7	97.3	2.65	2
E	21	20/25	20/20	8.01	3	100	1.10	1
F	23	20/20	20/20	6.70	1	100	0.86	2

^*Both eyes corrected

Performance results for all conditions are presented in Table 3, with OMI results separated into successful and unsuccessful copy-spelling groups. Paired t tests of mean equivalence indicated that performance under the VAI condition was not inferior to that under the unimpaired condition for either letter selection accuracy (delta=.05; p=.3229 for difference and p=.0003 for separation) or speed (delta=1.0; p=.6898 for difference and p < .0001 for separation). Mean accuracy for successful OMI copy-spelling sessions (n=6) was comparable to that of the other two conditions, though speed was greatly reduced. We used linear regression to examine whether the age of participants had an influence on BCI performance, and found no evidence to suggest that it did.

Table 3.:

Shuffle Speller performance results and trial length settings.

	Unimpaired (n=38)	VAI (n=38)	OMI: successful (n=6)	OMI: unsuccessful (n=31)
Accuracy (%)	96.2 ± 4.58	95.6 ± 5.70	95.3 ± 6.38	*
Speed (char/min)	4.39 ± 1.159	4.46 ± 1.072	1.47 ± .772	*
Trial length (sec)
Mean ± SD	5.26 ± 1.457	5.26 ± 1.391	7.44 ± .836	4.95 ± 1.493
Range	3.30–8.01	3.56–8.27	6.18–8.27	3.30–8.01

^*OMI accuracy and speed results are not reported for unsuccessful participants, as they were unable to type at all

Six participants demonstrated atypical performance in either the unimpaired or VAI condition, either due to very low accuracy (< 75%) or exceeding time limits for individual words or the entire task (see Outliers box in Figure 5 and Table S1 in supplemental online materials). As a sensitivity analysis, paired t tests of mean equivalence were repeated with these participants excluded (n = 32), with similar results: performance under the VAI condition was not inferior for either accuracy (δ = .05; p = .5384 for difference and p < .0001 for separation) or speed (δ = 1.0;p = .5660 for difference and p < .0001 for separation).

3.2. User Experience

Selected UX questionnaire responses are summarized in Table 4 (see Table S2 in supplemental online materials for complete UX results). Reported workload and discomfort were low, and satisfaction high, for the unimpaired and VAI conditions. Participants reported higher workload, more discomfort, and lower satisfaction for the OMI condition, regardless of their performance in the OMI trial.

Table 4.:

Selected UX questionnaire results.

	Scale	Unimpaired (n=38)	VAI (n=38)	OMI (n=37)
Overall workload	1 = not at all hard work, 7 = extremely hard work	2.5 ± 1.01 (reference)	2.4 ± 0.86 (p=.7365)	4.5 ± 1.45 (p<.0001)
Overall comfort	1 = not at all tired/uncomfortable, 7 = extremely tired/uncomfortable	2.1 ± 0.98 (reference)	1.9 ± 0.77 (p=.2791)	2.5 ± 1.04 (p=.0085)
Overall satisfaction	1 = extremely satisfied, 7 = extremely unsatisfied	2.0 ± 1.04 (reference) (n = 37)*	2.1 ± 1.18 (p=.7505)	4.9 ± 1.63 (p<.0001) (n = 36)*

^*Satisfaction data are missing for one participant each for the unimpaired and OMI conditions due to participants not completing the last page of the form

3.3. Adaptive Trial Length

Table 3 summarizes the trial lengths automatically chosen by Shuffle Speller for each condition and for successful and unsuccessful OMI participants. Trial lengths were similar for the unimpaired and VAI conditions and the unsuccessful OMI participants, but longer for successful OMI participants. In the unimpaired and VAI conditions, expected SSVEP selection accuracy almost always increased with trial length (see Figure 7). In the OMI condition, however, longer trial lengths rarely led to improvements in expected selection accuracy.

Figure 7.:

Stimulation time vs SSVEP trial accuracy

As described in Section 2.2, trial lengths were selected to maximize expected ITR based on user-specific calibration data. As expected, longer trials were associated with more accurate SSVEP selection (Figure 7). Across all conditions, expected ITR closely reflects the rate of correct character selection (see Figure 6), validating it as a worth-while objective to maximize.

4. Discussion

This experiment aimed to demonstrate that Shuffle Speller, with adaptive stimulus presentation and querying, is a viable interface for typing with SSVEP BCI, and to examine the effects of simulated visual impairment on Shuffle Speller use. Shuffle Speller was found to be a satisfactory spelling interface with high accuracy scores and low reported workload and discomfort. Healthy individuals with simulated VAI of 20/200 (meeting the U.S. definition of legal blindness [54]) could use Shuffle Speller for SSVEP BCI typing with accuracy and typing rates comparable to their performance with unimpaired vision, and were satisfied with system performance. Results indicate that individuals with VAI should not be ruled out as potential SSVEP BCI users. Shuffle Speller’s key algorithmic advantage is that it aggregates error-prone user evidence to make a single, accurate letter selection. However, the Shuffle Speller interface used in this experiment was specifically designed to be accessible to users with visual impairment, with features including large font sizes, high-contrast text, color and motion cues, and contextual cues from the alphabetical arrangement of characters. Visual BCI interfaces with other design characteristics, such as smaller font sizes, may pose more difficulty for users with VAI. Further investigation is needed to identify optimal interface characteristics for BCI users with reduced visual acuity. Web accessibility recommendations [55] may serve as a useful guide.

Some participants could use the system with high accuracy under a simulated OMI condition, though with slower typing speeds and increased workload. Most participants were unable to type under this condition. This variability in inter-participant performance is consistent with prior simulation results, including those of Allison and colleagues, who found that different participants often had very different SSVEP responses to the same stimuli, with highly variable levels of expected online control. [22] The same study found that users produced much stronger responses to non-overlapping than overlapping stimuli, underscoring the importance of gaze shifting and overt attention for many SSVEP BCI users.

Participants who were successful typing under the OMI condition had longer trial lengths than the full participant group for that condition, as well as for the unimpaired and VAI conditions, indicating that Shuffle Speller’s automatic trial length optimization feature appropriately adjusted for weak SSVEP signals in these users. Shuffle Speller’s adaptive querying feature also compensated for the weak EEG evidence obtained during the OMI condition, requiring more queries per letter selection. Participants who could type with reduced eye movements experienced reduced typing speeds compared to the other conditions, but similarly high typing accuracy.

Expected ITR was used as an objective function to optimize trial length for each participant. In practice this method will extend trial length for weak user responses, such those related to OMI, as long as the extension results in increased accuracy to justify the slowed typing rate. We expect the strong correspondence between expected ITR and the rate of correct character selection (see Figure 6) to be present in all systems which incorporate user evidence probabilistically. Some BCI systems utilize a decision tree in which characters not associated with the estimated SSVEP frequency are pruned away until only one is left. In these cases expected ITR is a less worthwhile predictor. There are many other advantages of probabilistic character inference, as discussed in previous work.[26]

Future work will investigate the use of the Shuffle Speller for SSVEP typing by people with SSPI and OMI. Additional analyses of eye tracking data for individuals who were successful in the OMI condition in this study might provide information about important characteristics of users with OMI. Clinically, this type of informtion might assist with the person-technology matching process for eventual home use of visual BCIs. Individuals with SSPI and OMI may be more successful with Shuffle Speller typing than our healthy participants under the OMI condition, as they may have developed strategies to compensate for their reduced motility, or be more skilled at attending to stimuli in their peripheral vision. They may also experience a different level of workload associated with system use, since participants in this study were required to consciously maintain central fixation during the calibration and spelling tasks, likely adding to their workload. Individuals with advanced ALS and reduced eye movement have used a four-class SSVEP BCI to select among predetermined phrases. [23] Shuffle Speller may provide such users with a means of typing novel messages with a similar four-class setup. Shuffle Speller’s adaptive stimulus presentation and querying methods are resilient to single-trial errors and designed to allow typing even with weak EEG evidence, and thus may support accurate (though slow) communication for some users with poor ocular motility.

Future work with Shuffle Speller may explore its suitability for use with access methods other than BCI, such as eye tracking, switches, or joysticks, which can produce weak, inconsistent control signals for users with unreliable motor function. Shuffle Speller employs fewer, larger targets than typical AAC typing layouts, which may increase selection accuracy, and its adaptive stimulus presentation, integrated language model support, and repeated querying may reduce typing errors even when inaccurate selections are made on individual queries. Movement-based alternative access methods may be preferable to BCI for users with some motor function; for example, eye tracking had higher ITR, better usability, and lower workload ratings compared to a P300-based BCI in users with motor impairments who were able to use eye tracking. [56] However, some individuals with SSPI cannot achieve accurate computer control with existing eye tracking keyboards. Interfaces that can compensate for inaccurate or inconsistent control signals are a valuable area for exploration.

Additionally, options could be explored for increasing Shuffle Speller’s capacity for adaptation to a user’s needs and abilities. For example, the number, size, position, or separation of boxes could be modified for individuals with OMI or visual field cuts. In the current system configuration, these characteristics would not be adaptive, as box placement is limited by the fixed locations of the LED panels. This could be addressed by presenting SSVEP stimuli directly on the screen instead of via LED. Customized animation speeds (i.e. the speeds at which letters travel from one box to another) or color cues might also be beneficial for some users, and could be determined during an initial calibration session.

Results of this study have implications for hardware configuration for SSVEP-based BCIs. Our electrode headband setup, which may be more convenient and esthetically pleasing than a traditional EEG cap, collected data of sufficient quality for control of an SSVEP BCI. Potential users have expressed concerns about the appearance, setup time, and messy gel associated with many current EEG-based BCI systems. [49,50] Until comfortable, effective dry electrodes are widely available, a system with a small number of electrodes (requiring fewer wires and less gel in the hair) and a headband (which alters the user’s appearance less than a cap) may be preferred by some SSVEP BCI users. However, further investigation should be conducted to compare this headband electrode setup to traditional EEG caps in terms of both data acquisition and user experience.

Shuffle Speller shows promise as a visual BCI interface for individuals with reduced visual acuity, and possibly for some with ocular motility. Future testing of this and other modified visual BCI interfaces with people with SSPI and comorbid visual impairments is recommended, and such interfaces should be compared with auditory and tactile BCIs. These explorations must be considered in the context of other potential comorbidities in the SSPI population. The impact of cognitive impairment on BCI performance has not been well explored, and is a crucial area for future exploration. The complicated profile of potential end users with both sensory and cognitive impairments underscores the need for BCI systems which are flexible and accommodating to a variety of specialized needs. [15]

5. Conclusion

Among healthy participants, simulated VAI did not cause inferior SSVEP BCI typing performance with Shuffle Speller. Some users could type accurately, though slowly, under a simulated OMI condition. Results suggest that individuals with impaired visual acuity or ocular motility should not be ruled out as potential visual BCI users, though interface designs must accommodate their particular needs. Future experiments should attempt to replicate these results among participants with real acuity or motility impairments, and to identify interface design features to increase BCI usability for these populations.