Eye‐controlled, power wheelchair performs well for ALS patients

Abstract

Background

Our pilot study tested the feasibility and performance of an eye‐controlled power wheelchair for amyotrophic lateral sclerosis (ALS) patients.

Methods

In this prospective pilot study, participants drove the wheelchair three times around an indoor course. We assessed the time to complete the course; starting and stopping on command; turning 90, 135, and 180 degrees; time to backup; and obstacle negotiation. Following their use of the wheelchair, subjects were given a questionnaire to assess user experience.

Results

Twelve patients participated, and all were able to complete three trials without difficulty. Eight participants completed all of the individual tasks (eg, turning, stopping, etc.) without any errors. Overall performance ratings were high across all participants (4.6/5‐excellent).

Conclusions

Our eye‐controlled power wheelchair prototype is feasible and has a very favorable user experience. This system has the potential to improve the mobility and independence of ALS patients, and other groups with motor impairments.

Short abstract

https://doi.org/10.1002/mus.26696.

Abbreviations

ALS

amyotrophic lateral sclerosis

ALSFRS‐R

ALS Functional Rating Scale Revised

AMB

ambulatory

EEG

electroencephalogram

FTD

frontotemporal dementia

FVC

forced vital capacity

mph

miles per hour

QOL

quality of life

UI

user interface

WCD

wheelchair‐dependent

1. INTRODUCTION

Power wheelchairs are useful for prolonging mobility in patients with motor diseases. Increasingly, software‐hardware integrations are improving the performance of power wheelchair systems and widening the population of patients that benefit from these devices. Systems that are currently available or in development include those controlled by speech; brain‐computer interfaces; movement of the tongue, hand, head, or foot; or facial expressions.1, 2, 3, 4, 5, 6, 7 These power control systems are increasingly adaptable, and may include multiple input interfaces, or autonomous navigation that helps users avoid obstacles.8, 9

Ocular movement‐controlled power wheelchair systems are a more nascent assistive technology. Limited data suggest that these systems provide an intuitive wheelchair control method for patients with severe disabilities, and previous studies have tested design components in healthy individuals.10, 11 Results from these studies suggest that eye gaze control may present a solution for wheelchair navigation for patients with severe motor deficits due to amyotrophic lateral sclerosis (ALS).

Our group has developed a prototype software‐hardware integration that uses eye tracking to control left, right, forward, and backward motion, as well as stopping, for a power wheelchair. This system offers an alternative method for power wheelchair navigation, and excluding oculomotor function, it does not depend on preserved motor function or speech. The primary objective of this study was to pilot the user interface (UI) and interface controls in a sample of ALS patients with a moderate level of disability.

In general, individuals with ALS report satisfaction with power wheelchair systems.9, 12 Specific wheelchair features that influence patient satisfaction include ergonomic components (e.g., arm and head rests), reclining and tilting capabilities, customizable controls, and controls that allow caregivers to operate wheelchairs.9 However, there is mixed evidence supporting the relationship between quality of life (QOL) and mobility in the ALS population, perhaps because a requirement for assistive technology is associated with reduced independence and more advanced disease.13, 14

While previous studies have shown a favorable effect of mobility on QOL in patients with ALS, others have not supported this relationship.15, 16, 17, 18 Studies that counter the relationship between mobility and QOL show that social interactions and leisure activities are more important determinants of QOL than mobility.18 Additionally, motor control systems in commercially available power wheelchairs may have sub‐optimal UIs, which may be nonintuitive, have delayed response times, or require motor input or complex abstract cognitive processing.10 Our goal in developing this prototype eye‐gaze controlled wheelchair was to develop an intuitive technology which would use preserved oculomotor control, enable basic mobility tasks, and yield a favorable user experience for ALS patients. Therefore, the secondary objective of this study was to evaluate the self‐reported user experience with the eye‐controlled UI, including responsiveness, performance, and ease of use.

2. METHODS

2.1. Participants

We conducted a pilot study to test an eye‐controlled wheelchair in patients with ALS. Inclusion criteria required subjects to be over 18 years of age and have a diagnosis of ALS as defined by the revised El Escorial criteria for Clinically Definite, Clinically Probable, or Clinically Probable‐Laboratory Supported ALS.19 All of the subjects that completed the study were patients in the Multidisciplinary ALS Clinic at Swedish Medical Center and had undergone comprehensive assessments before entering the study with a complete history and neurologic examination. This included mental status examinations and behavioral assessments to screen for any cognitive impairments or clinical features of frontotemporal dementia (FTD). Serial ALS Functional Rating Scale Revised (ALSFRS‐R) scores were tracked for each patient in the multidisciplinary clinic and the most recent scores within 1 month of screening were recorded.20 Duration of their disease (time since first onset of weakness) and region of onset were known for each subject. Both ambulatory (AMB) and wheelchair‐dependent (WCD) subjects were included.

Participants were excluded if they had prior use of eye‐controlled technology, significant respiratory compromise (forced vital capacity [FVC] < 50%), FTD, other cognitive impairments, or significant axial or neck extensor weakness that in the opinion of the investigator would potentially compromise the safety of the patient. The rationale for excluding participants who had previously used eye‐controlled technology was that a prior experience with similar software may have skewed individuals’ experience with the UI as well as their rating of the system's performance. At the screening visit, subjects were also tested with the eye tracking system to ensure they were able accurately move a curser with their eyes and that the software would respond to the subject's gaze. Subjects unable to use the eye tracking system were excluded.

Study procedures were approved by the Providence Health & Services Swedish Institutional Review Board. Written consent was obtained from all of the participants.

2.2. Technology

An eye‐controlled prototype system, which allowed the participant to direct their movements using their eye gaze, was mounted on the power wheelchair (Supplementary Figure S1A, which is available online). The eye‐controlled power wheelchair prototype was composed of the following elements:

1. An M300 electric power wheelchair (Permobil, Lebanon, TN).

2. A hardware interface module that converts the wheelchair electrical interface signals, available through an DB‐9 connector in the chair electrical control module, to USB interface signals (Curtiss‐Wright, Davidson, NC).

3. A tablet computer mounted on the wheelchair (Microsoft Surface Pro 3, Microsoft Corporation, Redmond, WA).

4. An eye‐tracking sensor (Tobii Eye Tracker 4C, Tobii AB, Danderyd, Sweden), which requires operators’ eyes to be within an 18 cubic inch area.

5. Control software that displays a UI with virtual control buttons, which can be selected by means of the user's eye movements (Supplementary Figure S1B). The software was a research prototype based on an eye‐tracking application interface that is currently used by Microsoft Windows applications, with a custom layer that supported the UI, video overlays, and other integration features.

6. Software safety features. Changes in user inputs are evaluated with watchdog routines in the interface modules, which check user signals 20 times per second. If a valid user signal (ie, continuous eye gaze in the direction of a UI direction control for 100 ms) is missing, these routines stop the wheelchair, thus ensuring that component failures do not indefinitely keep the wheelchair in motion.

The wheelchair moved in the direction of participants’ eye gaze. Participants directed their eye gaze at six transparent arrows to move the wheelchair, and at a central red square to stop the wheelchair (Supplementary Figure S1B). These controls were overlaid on the environment with semi‐transparent icons to permit participants to see and navigate their surroundings. The wheelchair prototype used was governed with a top speed of 2 miles per hour (mph) for all participants, and there was no variable speed control that would allow subjects to adjust the speed.

2.3. Test procedure

All wheelchair testing was performed in a large carpeted room (52 × 46 feet) with fluorescent lighting and no external windows (Supplementary Figure S2A). After a brief (~5 min) instruction session, participants drove the eye‐controlled wheelchair around a standard course, completing three trials. A red emergency override stop button was built into the prototype control system; a companion followed each participant through the course, and was always ready to press the stop button, if needed.

On the course, various tasks were assessed and videotaped: the ability to start and stop on command; turning 90, 135, and 180 degrees; backing up for short and longer distances; and negotiating an obstacle course, which was uniquely created to test the performance of this wheelchair system (Supplementary Figure S2B). For each trial, time needed to complete the individual tasks and a complete circuit of the course was recorded.

2.4. User experience

All participants completed a questionnaire regarding their self‐reported experience of the performance of the wheelchair system (Supplementary Table S1). Participants rated system performance in stopping, starting, negotiating obstacles, and backing up on a 5‐point Likert scale (5 = excellent, 1 = poor). Participants were also asked for open‐ended written feedback about the system. Questionnaire data were collected through oral interview by the study coordinator and were then transcribed.

2.5. Statistics and data analyses

Descriptive and inferential statistics were used to investigate the time it took participants to complete the overall test course and the individual tasks on that course. Self‐reported differences in wheelchair performance were also compared. Patient comments regarding the eye‐controlled wheelchair system were content analyzed, sorted into categories, and summarized. All statistics were performed with IBM SPSS 22, English Version (Armonk, New York). Significance was set at P < .05.

3. RESULTS

3.1. Participants and demographics

Fourteen participants were screened. Two participants were excluded, one due to strabismus that precluded accurate user input into the eye tracking system at screening, and another due to conflicting commitments. Twelve participants completed all study procedures.

Table 1 summarizes the demographics and clinical characteristics of the participants that completed the study. Of the 12 participants, four were completely AMB, three were AMB with occasional wheelchair or scooter use, and five were completely WCD.

Table 1.

Demographics and clinical characteristics of participants

Subject	Age (y)	Sex	Height (cm)	Weight (kg)	Disease duration (mos)	Region of onset	ALS FRS‐R	El Escorial Criteria	FVC (%)	Glasses	WCD vs AMB
1	75	M	180.3	79.8	35	RLE	33	CP	90	No	WCD
2	56	M	203.2	86.2	14	LUE	35	CD	75	No	AMB
3	57	M	189.2	85.7	84	RUE	32	CD	65	Yes	WCD
4	69	M	160.0	67.2	46	LLE	38	CD	71	Yes	WCD
5	68	F	162.6	68	34	BLE	21	CD	53	No	WCD
6	73	F	152.4	126	19	RLE	42	CP	84	Yes	AMB
7	35	F	160.0	61.7	22	RUE	35	CD	90	Yes	AMB
8	73	M	168.9	154	29	RLE	40	CP	74	Yes	AMB
9	63	M	165.1	55.2	16	LUE	25	CD	59	Yes	AMB
10	71	M	177.8	104.6	39	Bulbar	29	CP	74	No	AMB
11	63	M	177.8	59.9	48	BUE	34	CPLS	64	No	AMB
12	47	M	182.9	90.7	unk	RLE	29	CP	50.3	No	WCD
Means	62.5	75% M	173.3	86.6	35		32.8		70.8	50%

Abbreviations: CD, clinically definite; CP, clinically probable; CPLS, clinically probable laboratory‐supported; F, female; M, male; unk, unknown.

To compare the eye‐gaze wheelchair experience across different levels of ALS functionality, we grouped all of the participants who were primarily AMB into one group (N = 7) and those who were WCD into another (N = 5). There were no significant baseline differences between the AMB and the WCD groups in any demographic characteristic (age, sex, height, weight, disease duration, region of onset, ALSFRS‐R score, El Escorial criteria, FVC%, or frequency of prescription glasses use).

3.2. Course completion and task errors

Video recordings of all participants’ three course trials were collected. A single judge (L. Maassel) reviewed all of the trials to evaluate the participants for successful completion of the individual tasks involved in the course circuit. If the individual task was not attempted or not adequately completed, that was judged an error.

Eight patients completed all three course trials without any errors (failure to attempt or properly complete the given task) or difficulties. Four patients made at least one error. Three of five errors occurred in the 90‐degree turn, while 1 error occurred in the 180‐degree turn and another in the obstacle course. The rate of successful completion of all trial tasks was 98.3% (283/288). There were no accidents. The average age was higher in participants who made errors (70.3 ± 2.2 years) compared with those who did not (58.6 ± 13.2 years; P = .042). The average ALSFRS‐R score of participants who made errors (32.5 ± 9.4) did not differ from those who did not (32.9 ± 4.5), nor were there differences in the use of prescription glasses (50% for both groups) or in mean course completion times.

3.3. Course and task completion times

The means and SD for the individual tasks and the overall course completion time are shown in Table 2. No significant differences across the three trials were found for the average course completion times. The total course completion times (and standard deviations) for trials 1, 2, and 3 were 240 (45), 235 (46), and 231 (21) s, respectively. Comparing the AMB vs WCD groups completion times, there were no significant differences in total times, or in the individual tasks undertaken during the wheelchair course circuit (ie, stopping, starting, turns, obstacles, or backing up). Likewise, comparing subjects who wore glasses with those who did not, there were no significant differences in total course completion times or individual tasks undertaken during the wheelchair course. Using Pearson's correlation coefficient, no significant correlation was found between the age of each participant and their average course completion time over the three trials (R = 0.146). Similarly, no significant correlation was found between the ALSFRS‐R scores and the average course completion times (R = −0.453).

Table 2.

Participant task completion: Time in seconds

Tasks	Completion	Range	Tasks complete
Stop on command	1.1 (.3)	1‐2	12
Start on command	1.0 (0)	1‐1	12
Negotiate obstacles	97% N/A	0‐0	11
90° turn	6.1 (5.4)	2‐30	11
135° turn	7.2 (2.6)	4‐13	11
180° turn	13.8 (5.7)	5‐43	11
Backup short	13.9 (6.2)	6‐34	12
Backup long	23.6 (8.9)	12‐56	12
Overall time (s)	235.6 (38.6)	194‐373	N/A

Note: Mean (SD) time to successfully complete wheelchair tasks. Tasks complete indicates the number of participants who successfully completed the task across all three trials.

3.4. User experience

The participants rated the system's ability to complete each task and the overall performance of the system as excellent (mean: 4.6/5). Participants reported highest satisfaction with starting (mean: 4.7/5). However, the WCD group reported significantly less satisfaction with responsiveness than AMB patients for starting and all turns (all P < .02, see Table 3). The lowest satisfaction overall, for both WCB and AMB, was with backup. Comparing user experience across all measures with a repeated measures general linear model analysis of variance, the omnibus F = 7.9, P = .017. Following this with post‐hoc least significant difference comparisons, backup was the only measure found to be significantly lower than the overall experience (P = .011).

Table 3.

Self‐Reported Responsivity Experience

Experience measures	All participants	AMB	WCD	P Value
Stop	4.4	4.7	4.0	.064
Start	4.7	5.0	4.2	.016
Obstacles	4.3	4.6	4.0	.23
90° turn	4.2	4.7	3.4	.001
135° turn	4.3	4.7	3.6	.004
180° turn	4.3	4.7	3.8	.008
Backup	3.6	4.0	3.0	.085
Overall	4.6	4.7	4.4	.32

Note: Patient responses to wheelchair time trial. Data are presented as means.

3.5. Participant comments and feedback

The most frequent comments were: participants reported wanting more practice with the system (50%), a desire for a variety of control screen modifications (42%), and a desire for a backup camera (33%). There were no safety concerns raised in the comments by any of the participants. For a summary of the participant's comments, see Table 4.

Table 4.

Summary of participants comments from questionnaire

Comment category	N	(%)
Want/need more practice	6	(50%)
Want screen changes (total)*	5	(42%)
‐Can't see where going	2	(17%)
‐Want to lower or change screen angle	2	(17%)
‐Want smaller screen icons	2	(17%)
‐Hard to look where going	1	(8%)
‐Want screen controls centrally located	1	(8%)
Want backup camera/backup difficult	4	(33%)
System is fun, excellent	3	(25%)
Want more control	2	(17%)
Want shadow outline of head on screen	1	(8%)
Want chair change to improve vision	1	(8%)
Want visual & auditory obstacle cues	1	(8%)

Note: comments received from participant questionnaire.

^*While screen modifications were requested eight times, some patients suggested multiple changes, and the total number of patients asking for the specific screen changes listed here was five.

4. DISCUSSION

Results of this pilot study suggest that our eye tracking interface is a feasible power wheelchair control system for ALS patients. All participants were able to complete three runs around the trial course, and eight participants performed all tasks without any errors. Overall, 98% of individual tasks were completed successfully by the participants. Participants reported a favorable user experience with the device interface and provided suggestions for modifications and design improvements. Backing up was the most difficult task, and one third of the study group requested a backup camera as a design feature.

These results are similar to those reported previously in a study of power wheelchair use by three users with severe disabilities. Users reported that a gaze‐controlled interface was enjoyable to use, and provided a method to preserve independence.10 Previous studies have integrated other input mechanisms, such as breath or EEG, to augment the sensitivity and accuracy of gaze control.11, 21, 22 However, our results suggest that it may not be necessary to integrate other functional inputs. In terms of vision limitations, using our system was not possible for our patient with strabismus, but was possible for those with prescription glasses. Future studies with patients with a wider variety of disabilities will help clarify which patients are best suited to use this system, and which disabilities might preclude use.

While there were not overall satisfaction differences between the AMB and WCD groups, the WCD subjects reported lower satisfaction for most individual measures, particularly backing up and turning. Close inspection of individual comments revealed 3/5 (60%) WCD patients reported backup difficulty, while only 1/7 (14%) of AMB patients mentioned this. Although turning errors were very infrequent, 40% (2/5) of the WCD patients had at least one, while only 17% (1/7) of the AMB patients had any over three trials. The reason for these differences is unclear, but could be related to the need to of WCD patients to overcome prior learned use of a hand‐controlled joystick. No other significant differences were identified in the WCD group vs the AMB group to account for these observations.

Analyzing other factors that may have affected performance was challenging given the small number of subjects in this study. Comparing participants who wore glasses with those who did not, we did not find significant differences in course completion times or individual performance tasks. It was noted that the average age of subjects who made errors was greater than those who did not make any errors. However, greater age was not found to correlate with longer course completion times. Taken together, the effect of age on performance is uncertain. Looking at measures of overall disability, we did not see differences in the ALSFRS‐R scores in subjects who made errors compared with those who did not. Likewise, lower ALSFRS‐R scores did not correlate with longer course completion times. Hence, with our limited data set, we were not able to demonstrate that overall disability, as measured by the ALSFRS‐R, affected performance. Testing with larger numbers of patients would be necessary to determine with statistical certainty if age or overall disability affect subject performance with this system.

There are several limitations in this study. With this prototype, participants were not able to vary their speed, with the top speed held at 2 mph, and this may have hindered our ability to distinguish performance between the subjects. Another limitation is that the system was tested in only one type of environment: a large room in a commercial building with florescent lights. A similar system has previously been tested in a cluttered environment, and users reported satisfaction with the navigation system.10 However, performance of our system in the home environment or an outdoor setting under different ambient lighting conditions remains to be evaluated. In addition, we did not test patients with non‐ALS motor deficits, advanced ALS, impaired posture, or patients who were dependent on mechanical ventilation. Another limitation is that we did not use a standardized cognitive or behavior screen to more formally compare the participants, and it is possible that we may have missed some emerging impairments in our subjects affecting their performance. Lastly, in this pilot study we had a relatively small number of patients, which limited our ability to statistically confirm other factors that resulted in performance or user satisfaction differences within the group tested.

Subsequent iterations of the prototype power wheelchair system will focus on three major design improvements: sunlight tolerance, additional camera systems, and ergonomic positioning. Certain lighting conditions, such as bright sunlight or unfrosted incandescent bulbs, may interfere with ocular tracking sensors; consequently, future models will focus on improving tolerance to various ambient lighting conditions. Camera systems that facilitate backing up or alternate seating positions (i.e., leaning back at 45°), and controls that facilitate repositioning will be addressed to improve ease of use.

It is noteworthy to mention several safety features of this system, including those that were built into the control software, intended to improve safety in real‐world environments. Users can stop the wheelchair by one of two methods: directing eye gaze toward a central stop icon or looking away from the UI. Hence, a continuous eye gaze on a direction control icon is required to move the wheelchair in the corresponding direction. Additionally, speeds achieved by our prototype system are slower (2 mph) than those achieved with traditional joystick controls (~4 mph). Finally, an external emergency red stop button, which overrides user controls, is built in to permit a companion to stop the wheelchair.

This study is most accurately described as a feasibility and performance trial of the eye‐gaze system with ALS patients. We did not measure formal “safety” outcomes in this study. However, there were no accidents or mishaps in 36 course trials with 12 different ALS patients, there were no safety concerns raised in the comments, and 25% of the participants described the system as “excellent” and “fun.” While this does not constitute formal safety testing, it suggests that the system is relatively easy and comfortable for the ALS patients tested.

In this study, participants reported a high level of satisfaction with the power wheelchair control system. These results match those of previous reports, which have shown that patients with ALS are satisfied with power wheelchair devices and enjoy the independence that these devices provide.9, 12 Because eye movement is often preserved longer than other motor functions in patients with ALS, our system may provide a long‐term mobility option in patients with advanced ALS.23, 24 This is an advantage over power wheelchair control systems that require motor or respiratory function. Future studies in different environments and patient populations will help to refine our UI and patient satisfaction further, potentially allowing preserved mobility for ALS patients even into later stages of disease.

CONFLICT OF INTEREST

Michael A. Elliott, Lindsey L. Maassel, and Becky Wood received grant support from Microsoft Research to conduct this trial. Henrique Malvar, Jon Campbell, Harish Kulkarni, Irina Spiridonova, Noelle Sophy, Ann Paradiso, Chuck Needham, Jamie Rifley, Maggie Duffield, and Jeremy Crawford are all Microsoft employees. Jay Beavers is a Microsoft employee and currently is a Managing Member of Evergreen Circuits LLC. Emily J. Cox and James M. Scanlan report no disclosures.

ETHICAL PUBLICATION STATEMENT

We confirm that we have read the Journal's position on issues involved in ethical publication and affirm that this report is consistent with those guidelines.

Supporting information

Supplementary Figure 1 (A) Eye‐controlled system on power wheelchair (author‐J. Campbell). (B) Eye‐controlled system user interface. Gazing at the arrows allows user to direct the PWC while seeing their environment.

Supplementary Figure 2 Wheelchair Test Area (A) Wheelchair test area. (B) Map of course. 1 = Start on command; 2 = Drive forward to cone, getting as close as possible without touching the cone; 3 = Back an “L” through cones; 4 = Right angle, 90 degree turn; 5 = Weave through cones; 6 = Drive between narrow barricades; 7 = Make a sweeping 180 degree turn; 8 = Make a very sharp 135 degree turn; 9 = Cross Finish

Table 1 Hands‐Free Wheelchair User Experience Questionnaire

Elliott MA, Malvar H, Maassel LL, et al. Eye‐controlled, power wheelchair performs well for ALS patients. Muscle Nerve. 2019;60:513–519. 10.1002/mus.26655