New and Emerging Access Technologies for Adults with Complex Communication Needs and Severe Motor Impairments: State of the Science

Abstract

Individuals with complex communication needs often use alternative access technologies to control their augmentative and alternative communication (AAC) devices, their computers, and mobile technologies. While a range of access devices is available, many challenges continue to exist, particularly for those with severe motor-control limitations. For some, access options may not be readily available or access itself may be inaccurate and frustrating. For others, access may be available but only under optimal conditions and support. There is an urgent need to develop new options for individuals with severe motor impairments and to leverage existing technology to improve efficiency, increase accuracy, and decrease fatigue of access. This paper describes person-centered research and development activities related to new and emerging access technologies, with a particular focus on adults with acquired neurological conditions.

During the past several decades, many people with acquired neurological conditions who experience complex communication needs and severe motor impairments have become effective communicators because of evolving augmentative and alternative communication (AAC) strategies and technologies (Ball, Fager, & Fried-Oken, 2012; Beukelman, Fager, Ball, & Dietz, 2007; Fager, Bardach, Russell, & Higginbotham, 2012; Fager, Beukelman, Fried-Oken, Jakobs, & Baker, 2012; Fried-Oken, Beukelman, & Hux, 2012; Higginbotham, Shane, Russell, & Caves, 2007; Shane, Blackstone, Vanderheiden, Williams, & DeRuyter, 2012). Through the years, the AAC community has developed and implemented a range of motor access options for AAC technology that allow users to speak messages, write text, and participate with communication/social media. These options include adapted keyboards, modifications to touch screens on computers or mobile technologies (e.g., delayed activation, with touch activation on touch exit); single- or multiple-switch scanning; head tracking; and, most recently, eye tracking (Beukelman & Mirenda, 2013). These options, often referred to as access technologies, effectively address the motor access challenges of many people with complex communication needs. Unfortunately, some individuals still remain poorly served. They cannot accurately access AAC technology; have inconsistent control depending on changes to their physical condition, environments, or support; and often can only access AAC technology in optimal situations. For example:

Mandy is a university student with athetoid cerebral palsy who uses eye-gaze to access her computer, write papers, send e-mails and text messages, participate in communication/social media, and converse face-to-face with her caregivers and fellow students. She accomplishes this in her dormitory room, where her eye-gaze device is optimally situated. When she is out of her dormitory room, her eye-gaze access is inconsistent and relatively ineffective as she moves about the campus from class to class, the library, and social situations. Mandy would benefit from adaptive access methods that better meet her motor challenges as well as her location, communication partner, and message needs.

Individuals with other types of motor impairments meet similar challenges resulting in varying degrees of access limitations depending on their ability to compensate for access requirements either by using their motor capabilities or through the technical support of the people who provide care for them. These limitations range from inefficient access causing fatigue to a lack of access (and communication) in situations throughout the day.

The Challenges of Access

There are several ways to describe limitations to current AAC access technologies and opportunities to improve or enhance access options. For individuals with severe motor impairments (e.g., locked-in syndrome), an access option may not be readily available. This can occur for individuals with brainstem stroke, high-level spinal cord injury, or in the late stages of degenerative conditions such as amyotrophic lateral sclerosis (ALS) (Murguialday et al., 2011; Kruger, Teasell, Salter, Foley, & Hellings, 2007). For others, an access solution may be available but is unreliable, causing frustration and limited use of technology to support communication. For still others, the use of a single access method over time can become difficult due to over-use (Blackstone, 1995; Fager, Bardach et al. 2012) or age-related changes (e.g., individuals with spastic cerebral palsy experiencing increasing neck pain while using a head-activated switch for scanning access to their AAC device as they age).

There is an urgent need to improve available access technologies. Typical access challenges include the need for optimal positioning of equipment; and consistent, skilled support by caregivers throughout the day (Ball, Nordness, Fager, Kersch, Pattee, & Beukelman, 2010; Beukelman et al., 2007; Fager, Bardach et al. 2012). For those who are active in a wide range of communication environments (e.g., home, school, outdoors, work site, doctor’s office, restaurant etc.), variability in lighting and optimal positioning requirements often limits the use of AAC technology (Ball et al., 2010; Fager, Bardach et al. 2012; Fager, Beukelman et al., 2012). For individuals who find themselves in medical settings, where caregivers change frequently, the use of sophisticated access methods can be challenging if a consistent support system of caregivers trained in the set-up, positioning, calibration, and trouble-shooting of the technology to support access is not available. For example, precise positioning of switches may be required for successful use. This positioning may be compromised when individuals inadvertently shift position (e.g., due to a cough, sneeze, or muscle spasm) or as they are re-positioned throughout the day as part of their care (e.g., moved from a wheelchair to a recliner or bed).

Person-centered Design in Developing New Access Technologies

Developing technology to fill gaps in available access options requires the close involvement of individuals with complex communication needs and severe motor impairments, caregivers, clinical specialists (e.g., speech-language pathologists, occupational therapists), engineers, researchers, and manufacturers of AAC technology. Due to the complexity and uniqueness of the challenges that individuals with severe motor impairments exhibit, research and development activities often emerge out of individual case examples, which often illustrate a broader need. Technology is then developed based on the needs and skills of the target group and individualized incrementally, with ongoing feedback from all stakeholders to further refine and develop the access method.

This person-centered design process is consistent with theoretical frameworks employed in medical device and assistive technology development (Shah, Robinson, & Al Shawi, 2009). This process includes (a) early involvement of the individual who will be using the technology, research that investigates the human processes that impact access (e.g., motor performance, visual-cognitive processing, etc.), (b) development of prototypes with human processes in mind (e.g., solutions that reduce motor demands, consider cognitive demands, etc.), (c) iterative development and refinement of technology prototypes, and (d) outcome data directly related to individual performance, perceptions and preferences (Kübler et al., 2014; Shah et al., 2009; Fager et al., 2017; Peters, Mooney, Oken, & Fried-Oken, 2016; Wilkinson & De Angeli, 2014). Technology development typically occurs in stages, beginning with proof of concept or idea generation and prototype design, testing and evaluation or proof of product, and implementation or proof of adoption (NIDILRR Development Framework, 2017; Shah et al., 2009). Throughout all stages of the development framework, input from all key stakeholders is a critical component to successful development of complex devices such as access technologies (NIDILRR Development Framework 2017; Shah et al., 2009).

This paper describes the research and development of new and emerging access technologies that begin with a clinical challenge and lead to a research-driven engineering discovery. The access technology solutions described illustrate different stages of development. Following person-centered design frameworks, case illustrations and/or group outcome results are provided to describe the ongoing development process. New and emerging technologies are introduced, with case illustrations used to describe the need for each technology and associated research to date. Future research and development directions are discussed. The individuals described in the case illustrations were involved in the research of the technology solutions but all identifiable information has been changed to preserve anonymity.

Although many of the technologies described may be of benefit to children, this paper will not address challenges related to learning, cognition, or language development that may require different solutions and strategies. The focus is on the following new and emerging access technologies for adults with acquired neurological conditions: movement-sensing technologies, brain-computer interface, multi-input strategies, supplemented speech recognition, and supplementation of access with partner input.

Movement-Sensing Technologies

A range of movement-sensing devices exists to track fitness and activity (e.g., wearable fitness monitors that incorporate accelerometers and gyroscopes that track activities such as walking and running). The development of this technology has been driven primarily by a market of individuals without disabilities; there have been no reports to date of individuals with severe motor impairments using the technology as an access method to AAC. However, leveraging this technology to detect unique movement patterns of individuals with severe motor impairments may provide a new genre of access devices for AAC. For example, once movement signals are identified, they often can be categorized (Haghi, Thurow, & Stoll, 2017; Hasan & Hongnian, 2017). For individuals with severe motor impairment, movement-sensing technology provides opportunities to identify and categorize movements as intentional or unintentional (e.g., null-model detection). The following case example illustrates an early prototype of a movement-sensing system that utilizes null-model detection, developed for an individual with severe motor impairment who relies on AAC.

Andrea had a severe brainstem stroke that left her with limited motor capabilities that were restricted to small thumb and pinky finger movements on one hand. When her caregivers were by her side, they could watch her hand and identify these movements. When optimally positioned, she could access small micro-light switches using a custom switch mount and thereby control an alphabet array via step scanning (thumb movement to advance and pinky movement to select). However, if Andrea moved slightly out of position, she no longer had access to her switch system. Or if she coughed or laughed, she inadvertently activated the switch system causing errors in her message construction. If her caregivers were in another room or distracted with other activities, some time would pass before they might notice that she could no longer access her switch system. All of this was extremely frustrating for Andrea.

Technology Solution

Colleagues at InvoTek, Inc. developed a novel, wearable sensor system that incorporated an accelerometer, a gyroscope, and a magnetometer (Fried-Oken, Fager, & Jakobs, 2016). This sensor array was “trainable” in that, given several repetitions of intentional and unintentional movements, it could “learn” when Andrea was intentionally activating her device and when her movements were unintentional (e.g., during a cough). Housing for the sensors was 3-D printed for a customized fit to her hand (see Figure 1). The device constructed for Andrea leveraged sensor technology in a way that addressed her specific access challenges. In this early prototype, the system required recorded movement samples from Andrea and personalized support from the engineering team to calculate the equations needed to recognize her movements. With advances in machine learning, this process has the potential to be automated much like initial calibration and set up incorporated by other access devices today.

Figure 1:

Sensor switch incorporating accelerometer, gyroscope, and magnetometer for position-independent access.

The wearable nature as well as the 3-D sensor measurement properties of the wearable sensor device eliminated the need for Andrea to be precisely positioned to have access to her switch. Precise positioning is often a substantial challenge for individuals with severe motor impairment and their caregivers. The individuals with severe motor impairment often lack the motor ability to move back into position if they have shifted while using access technologies. Caregivers are often required to be highly trained to optimally position these devices because a slight deviation makes access impossible. The 3-axis sensors allowed the device to function regardless of position. This feature increased Andrea’s accurate access and alleviated the burden of optimal positioning that other access technologies often placed upon her caregivers. The sensor prototype developed for Andrea illustrates the potential to increase accuracy and independence of access for individuals with severe motor impairment and decrease the burden associated with optimal set-up for switch use by caregivers (Fried-Oken et al., 2016).

What Do We Know So Far?

There is limited information available about the use of wearable sensor technology to support access as described in this case illustration. The Invotek prototype met a specific need for an individual with severe motor impairment and complex communication needs. Given the ubiquity of sensor technology in mainstream devices, the exploration of the benefits of this technology as an access tool for individuals with severe motor impairments is emerging. For example, other studies have documented preliminary results of the use of capacitive-sensing technology for assistive technology and environmental control options (Fager et al., 2017; Singh, Nelson, Robucci, Patel, & Banerjee, 2015). These sensors have been embedded into fabrics that can be worn or placed within the environment. The sensors detect gestures or movements that an individual can make while near the sensor array (without having to touch the sensors). These gestures have been translated into environmental control command options (e.g., turn lights on or off) or used as a switch access option. Descriptions of the technology, stakeholder feedback in the early design process, and preliminary laboratory testing of the capacitive sensors have been reported (Fager et al., 2017; Singh et al., 2015). All stakeholders (individuals with motor impairments, family members, and caregivers) rated the technology as having high potential benefit as an access method for individuals with motor impairments and preliminary trials of the technology by individuals with motor impairments resulted in a high level of gesture recognition accuracy despite diversity of motor capabilities. Use of this technology as an AAC access method by individuals with severe motor impairments has yet to be explored.

Future Directions

Creating the movement recognition equations for the sensor system developed for Andrea must be automated. Machine learning algorithms hold promise for automating setup in a way that is reasonable and easily managed by caregivers and not excessively fatiguing for individuals with severe motor impairments. Additional research on how well sensor technology identifies and filters unintentional movements is needed. For example, it is unknown whether the technology can differentiate intentional from unintentional movements in individuals who exhibit extraneous movements (e.g., due to cerebral palsy). If this can be accomplished, sensor technology could vastly improve switch scanning for successful access to AAC devices and computers. There is a need for more extensive clinical evaluation to collect performance data using sensor technology under real life conditions with a range of individuals with severe motor impairments as well as person-centered data.

Brain-Computer Interface (BCI)

BCI is a promising, but somewhat elusive, access method for individuals with minimal movement capabilities who cannot rely on traditional methods for control of communication and computers. The role of BCI within AAC is becoming clearer, and the challenges for implementation of the technology better understood (Brumberg, Pitt, Mantie-Kozlowski, & Burnison, 2018). Adults with locked-in syndrome play a critical role as both participants and consultants within the research and development process because they have previously experienced communication competence and independent control over their environments, and can offer insights that drive person-centered designs, as in the following case example:

Kraig is a 37-year-old man who worked as a software engineer until he was diagnosed with ALS 4 years ago. He is now quadriplegic, relies on a ventilator, and cannot speak. Kraig has used many AAC access technologies, most recently single-switch access and an eye-tracking AAC device; however, he can no longer control the device due to inconsistent, unreliable ocular motility and loss of motor function for switch access. He uses small horizontal eye movements to answer yes/no and other binary-choice questions, but these subtle signals can be difficult for partners to recognize and interpret. Kraig lives in an adult care home. He is an avid sports fan and spends many hours sitting in a wheelchair in front of any televised competition. Kraig’s parents recently saw a television program about brain-computer interfaces and asked his physiatrist if BCI technology might help him regain reliable access to communication.

Technology Solution

There are a number of BCI systems that have been developed for communication (Akcakaya et al., 2014). Most are matrix spellers that rely on the P300 brain wave for intent selection (Wolpaw, Birbaumer, McFarland, Pfurtscheller, & Vaughan, 2002). In the matrix speller, letters are presented in an alphabetic grid with random flashes to letter boxes. Individuals select letters by focusing on the flashing letter of choice and their P300 waves are averaged for the selection (Sellers & Donchin, 2006). The P300 wave is elicited after a novel or surprise stimulus appears. The intended letter is the surprise or novel stimulus in a series of highlighted letters on a grid. The random flashing sequence on the grid is referred to as an oddball paradigm (Farwell & Donchin, 1988).

The RSVP Keyboard is a prototype BCI system that uses the P300 brain wave with a different selection paradigm: a Rapid Serial Visual Presentation (RSVP) of individual letters. One large letter at a time is presented on the screen for 400 milliseconds (or less), thus reducing the visual perceptual demands of BCI displays with a matrix with letters (Oken et al., 2014). When the intended letter appears, it is considered the surprise or novel stimulus and P300 waves are produced and averaged together so that a selection can be inferred. The same brainwave, the P300, is elicited for both the RSVP Keyboard and the matrix spellers but the end-user selects letters by looking at different letter displays. The keyboard is unique in that it fuses the P300 wave with a statistical language model (an algorithm that computes the probability of a sentence or sequence of words) for spelling by individuals with locked-in syndrome, thereby improving the probability of accurate letter selections (Moghadamfalahi et al., 2015).

The Shuffle Speller prototype is another spelling interface for BCI systems, which acquires and processes a different brain wave, the steady state visual evoked potential, to imply selection intent. Letters move (or shuffle) over a screen based on their probabilities. Color intensities and letter locations are animated until the system is confident of the speller’s intended selection (Higger et al., 2017). Yet another example of a BCI speller is the Berlin BCI which presents letters in a nested circular fashion, analyzing another brainwave, the sensory-motor rhythm, for selection intent (Blankertz et al., 2006). Finally, there are some spelling BCIs that use the more traditional AAC row-column scanning display and measure the sensory-motor rhythm as end-users are asked to imagine moving their limbs for selection (Brumberg, Burnison, & Pitt, 2016; Scherer et al., 2015).

What Do We Know So Far?

There are reports of BCI systems that have been used successfully by individuals with severe physical disabilities resulting from ALS (Sellers, Ryan, & Hauser, 2014; Sellers, Vaughn, & Wolpaw, 2010; Nijboer et al., 2008). There is a range of letter displays, different brain waves, and stimuli that are presented in various modalities that have been implemented with BCI systems. McCane et al. (2014) introduced a P300-BCI matrix speller to 25 individuals with ALS in their homes and demonstrated that most of the participants who had severe motor impairments could use a visual P300-BCI for communication. A current publication (Wolpaw et al., 2018) followed 39 individuals with ALS who met study criteria for BCI home implementation. From this initial cohort, 14 participants became home users of the noninvasive P300 Wadsworth BCI system and seven chose to continue using the system after the 18-month study ended.

Combaz and colleagues (2013) compared two spelling BCIs that analyzed different brainwaves (the P300 and the steady state visual evoked response) in seven patients with incomplete locked-in syndrome. They found that there was increased accuracy and satisfaction and reduced mental workload when typing was accomplished with a steady state visual evoked response compared to the P300-based system. Kaufmann, Holz, and Kübler (2013) described a case study with a patient with complete locked-in syndrome who tried spelling when letters were presented in different modalities. First, she tried spelling when letters were presented visually on different size letter grids; next she tried with an auditory BCI, when letters were presented one at a time through headphones to both ears with the same volume for all stimuli; and finally she tried spelling in tactile mode, which involved placing four sensors on her left arm, with six letters of the alphabet grouped together for each sensor. She attempted to spell in a partner-scanning approach, by responding to the tactor that produced a sensory stimulus and represented the letter she chose. Unfortunately, with event-related potentials for intent selection, the BCI system was not successful for communication in any modality. Oken et al. (2014) described performance with the previously mentioned RSVP Keyboard, where only one letter at a time is presented on the screen. Nine participants with locked-in syndrome and nine without disabilities (matched for age, gender, or years of education) used the keyboard. On average, as expected, the individuals without disabilities had substantially higher scores for a copy-spelling task, but all participants with locked-in syndrome could complete the copy-spelling task for predictable words.

Despite these positive preliminary results, enthusiasm for BCI implementation is dampened somewhat by the challenges faced when translating the technology from the laboratory to practical application (Chavarriago, Fried-Oken, Kleih, Lotte, & Scherer, 2017). Many of the hardware, software, and training obstacles that were discussed 5 years ago (Fager, Beukelman et al., 2012), remain today. Noninvasive EEG-based BCIs have not been readily implemented because it is still difficult to acquire a robust and reliable brain signal. There are long calibration and training sessions needed to teach the BCI to interpret brain signals and intent. The cognitive demands for attention, memory, and executive function are significant for extended use, though a comprehensive evaluation of the cognitive demands of BCI has yet to be completed. Currently there is interest in “human-in-the-loop” systems, where the machine learns an individual’s characteristics and adjusts signal acquisition and processing algorithms to optimize the system for each person (Lotte, Larrue, & Muhl, 2013). This is a closed-loop system between the computer and the human, where the computer relies on input from the human to improve its intelligence and decision making. Within BCI, human feedback in response to different tasks and training paradigms will improve the signal processing and robustness of the computer recognizing the correct intent for letter and word selection.

Future Directions

BCIs for communication are becoming a reality, and there is an international multidisciplinary research community that is dedicated to helping individuals with minimal movement capabilities utilize BCI to produce spoken output and control a computer cursor for written communication and social interaction. The ultimate goal of this technology is the transmission of language, either written or spoken, directly from brain signals, avoiding the need to operate an AAC device. The vision is that an individual would think of a word and the BCI would directly decode brain activity into spoken communication (Chakrabarti, Sandberg, Brumberg, & Krusienski, 2015). We are far from achieving that goal, though there has been research on decoding phonemes directly from intracortical microelectrodes (Herff et al., 2015; Kellis et al., 2010; Mugler et al., 2014) as well as from non-invasive approaches (Brumberg, Pitt, & Burnison, 2018). It is unclear at this time whether the non-invasive BCI (which requires sophisticated signal processing to detect the correct brain waves through the scalp) or the invasive BCI (which places electrodes right on the brain or dura) will optimize BCI access options. Initial trials of intracortical BCI for neural point-and-click communication by individuals with locked-in syndrome have been reported by a number of BCI labs (e.g., Bacher et al., 2015; Vansteelsel et al., 2016).

The integration of BCI with eye-tracking technologies is another promising direction for research that leverages current assistive technologies. It is possible that each person with locked-in syndrome will present with a constellation of sensory, motor, and cognitive skills that will define the type of BCI system (i.e., auditory or visual BCI), the type of letter display, and the choice of brainwave that maximizes function and success. A scientific discussion about success and priorities can only occur once there is consensus on the desired outcomes and how to measure them. For this population, there are many stakeholders who present with a range of needs and priorities, all of which must be met to ensure successful implementation so that practical usage of this new technology can become a reality (Wolpaw et al, 2018; Andresen, Fried-Oken, Peters, & Patrick, 2015).

Multi-Input Strategies: Eye Tracking Plus Switch Scanning

Eye tracking and switch scanning are currently available as access methods to AAC devices and computers with specialized software. Using a direct pointing method such as eye tracking can be an efficient way to get to a general location on an AAC device interface or computer. Using another access method such as switch scanning can be an efficient way to then make a discrete selection once the eye-tracker has narrowed the field to be scanned to a specific location. There are potential benefits to combining these two access methods. First, eye tracking can be challenging when selecting small targets. Lighting (e.g., bright lights, outdoor lighting), positioning (e.g., re-positioning requiring frequent re-calibration), and physical conditions of the eye (e.g., dry eyes) can pose challenges to continuous use of eye tracking and these problems often result in decreased accuracy of access (Ball et al., 2010; Fager, Bardach et al. 2012; Fager, Beukelman et al., 2012). Due to the severity of occulomotor control deficits, some individuals cannot rely on eye tracking alone for consistent access. Second, switch scanning can be slow, and the repetitive movements needed to activate a switch may be over-taxing. Technology that combines the best aspects of both methods could be beneficial for those who are challenged by using either access method individually, as illustrated in the following case example:

Lisa sustained a brainstem stroke and was undergoing rehabilitation at a long-term acute care hospital. Lisa had severe/profound dysarthria, limited movement in her left hand, and significant oculomotor control impairments and diplopia (double-vision). She applied spot patches to either of her glasses lenses periodically throughout the day to eliminate the impact of her double vision when reading and focusing on computer-based tasks. The severity of her oculomotor deficits and the periodic spot patching which obscured one of her eyes made eye-tracking access inconsistent, inaccurate, and frustrating. Switch-scanning with her left hand was possible, but she fatigued quickly due to her early stage of recovery and emerging motor capabilities, limiting her ability to use this access method to support her communication needs throughout the day.

Technology Solution

Lisa trialed a multi-input prototype that integrated eye tracking and switch scanning (Fager, Jakobs, & Sorenson, 2018). The prototype worked in the following manner: First, as she focused on her target (e.g., word in the word prediction list or a letter on an onscreen keyboard) with her eyes, small groupings of words or groups of letters were highlighted. Once the target word or letter was highlighted within the group, she activated her switch. Each word or letter within that grouping was then scanned. When the target letter or word was reached, Lisa selected it with her switch. The number of letters or words that were highlighted could be adjusted based on an individual’s eye-tracking capabilities. This approach was beneficial for Lisa because she could use gross eye tracking to narrow down the set of letters or words to be scanned which, in turn, decreased her physical fatigue. See https://rerc-aac.psu.edu/development/d1-developing-multimodal-technologies-to-improve-access/ for a brief video illustration of the multi-input prototype.

What Do We Know So Far?

Preliminary investigation of the multi-input (eye tracking plus switch scanning) approach has been initiated (Fager et al., 2018). A single-subject case study comparison of performance using eye tracking only and the multi-input approach was completed. Using an alternating treatment design, the participant in the case illustration completed a total of 10 sessions: five with each access option (single access using eye tracking, and multi-input using eye tracking plus switch scanning) randomly assigned. During the sessions, the participant spelled randomly selected sentences using an onscreen keyboard display (ABC layout) accessed via either eye tracking only or by using the multi-input prototype. Data were collected on accurate first attempts to target letters per sentence and the total number of errors per sentence. Results indicated that the participant averaged 63% first attempt accuracy across sentences using eye tracking only and 91% using the multi-input prototype. First-letter attempt accuracy (%) across sessions and access methods is displayed in Figure 2. Additionally, an average of eight errors per sentence occurred using eye tracking only and an average of two errors per sentence occurred using the multi-input prototype. The number of errors for each session and for each access method is displayed in Figure 3. Rate data were not collected for this participant because she struggled using the eye tracking only access method to make accurate selections and corrections and was therefore allowed to abandon attempts to correct errors and move onto the next target letter to reduce her fatigue and over-all frustration during the trials using eye tracking only.

Figure 2:

First attempt accuracy per sentence across alternating conditions (eye tracking only vs. eye-tracking plus switch scanning [multi-input]).

Figure 3:

Errors per sentence across alternating conditions (eye tracking only vs. eye-tracking plus switch scanning [multi-input]).

Other researchers are investigating the use of eye gaze coupled with scanning. Biswas and Langdon (2011, 2013) compared eye tracking alone to a combined eye tracking and switch- scanning access method for cursor control on a computer interface for navigation. The eye tracker moved the cursor to a general location and then the system scanned various mouse direction options to further move and refine the location of the cursor. Results of a study with individuals without disability indicated that the combined eye tracking and scanning access method was less strenuous than eye tracking alone and faster than scanning access only.

Research into combinations of other access methods is also emerging. Sahadat, Alreja, and Ghovanloo (2018) have preliminary results of a simultaneous, multimodal computer access method that uses tongue movement, head tracking, and speech recognition. Head tracking was used to move a computer cursor to a general location on a computer interface; tongue movement (captured with a head-mounted device that tracked a magnetic tracer attached to the tongue) was used to execute mouse functions such as click and hold, right click, left click; and speech recognition was used for text generation. Although this technology was described as being developed for individuals with tetraplegia, only participants without disabilities trialed the technology in the study and used this prototype to perform a computer search and send an email. Results indicated improvements in task completion rates with practice but overall slower task completion rates using the multimodal prototype compared to using a standard keyboard and mouse for individuals without physical impairments. Although these prototype examples demonstrate exploration into multi-input strategies for access, the inclusion of individuals with motor impairments in future research and development efforts will be essential to fully understand the benefits and challenges of these technology solutions.

Future Directions

Lisa’s case illustration demonstrates the use of a multi-input strategy for an individual who had challenges using each access method individually. Other studies examining a similar use of eye tracking and switch scanning (Biswas & Langdon, 2011, 2013) demonstrate promising results. However, further investigation with individuals who have severe motor impairments is warranted to fully understand when combinations of access methods may be most appropriate and beneficial. For example, an otherwise successful access method may become less accurate at times (e.g., during a hospitalization when positioning challenges are present, or when specific lighting environments interfere with successful calibration and use of an access method such as eye tracking). A multi-input approach may be useful to allow individuals to rely on their familiar access method (eye tracking) with additional support (switch scanning) to reduce new barriers in challenging conditions. Further examination by individuals with disabilities is critical at this point in development to understand the potential benefit of this approach for a wide range of individuals with severe motor impairments. Additional combinations of access methods (e.g., touch access plus switch scanning; eye tracking plus head tracking) need to be explored. The cognitive and learning demands of using multiple access options need to be investigated across different ages and with individuals who have cognitive challenges (e.g., severe traumatic brain injury impacting new learning, attention, and memory).

Supplemented Speech Recognition

Many individuals with severe motor impairment also have co-occurring speech impairments, such as dysarthria, that impact their ability to successfully use standard speech- recognition technologies (Young & Mihailidis, 2010; Hamidi, Baljko, Livingston, & Spalteholz, 2010; Christensen, Cunningham, Fox, Green, & Hain, 2012). Some researchers have attempted to design algorithms to recognize specific characteristics of dysarthric speech (Caballero-Morales & Trujillo-Romero, 2014; Mengitsu & Rudzicz, 2011). However, speech impairments can result from a wide range of conditions (e.g., cerebral palsy, traumatic brain injury, brainstem stroke, multiple sclerosis, Guillaine Barré), dysarthria types (e.g., flaccid, spastic, mixed, hyperkinetic) and severity levels, all of which are challenges for computer algorithms to manage effectively. High speech recognition rates (80% and above) have primarily been documented for those with mild dysarthria (Hux, Rankin-Erickson, Manasse, & Lauritzen, 2009) or for recognition of small vocabulary sets (Hawley et al., 2007; Judge, Robertson, Hawley, & Enderby, 2009). Although speech recognition use by individuals without speech impairments has grown as a result of consumer-based applications such as Alexa¹™ or Siri²™, the technology has been of limited benefit to those with severe speech impairments. Consider the following case illustration:

Mark has cerebral palsy and severe spastic dysarthria. He uses his natural speech to communicate with close friends and family and uses a text-to-speech AAC device to communicate with unfamiliar listeners. Due to Mark’s motor impairments, his AAC device requires a key guard and modifications to the touch screen to decrease accidental activation. Mark complains that the access method to his text-to-speech AAC device is slow and laborious. He often marvels at speech-recognition technology and states that he would love to be able to use his natural speech to decrease the motor demands of using his AAC device. However, Mark has tried many commercially available speech recognition systems with limited success due to the severity of his dysarthria.

Technology Solution

Leveraging other sources of information to supplement the speech signal of an AAC device may benefit individuals like Mark, who have severe speech impairments. One possibility is the Supplemented Speech Recognition (SSR)³ system (Hosom, Jakobs, Baker, & Fager, 2010). The SSR combines large vocabulary speech recognition algorithms designed for dysarthric speech, first letter supplementation, language models, and word prediction. Individuals customize the system to the specific characteristics of their speech capabilities prior to use through a training protocol. With the SSR, the user first types the first letter of the target word on a keyboard or accesses the first letter from an onscreen keyboard. The SSR system then provides an auditory prompt (e.g., a “beep” sound) that signals the individual using the system to speak the target word. After the target word is spoken, the SSR (a) types the word the speech-recognition algorithm recognizes into the line of text, and (b) provides the next five probable word choices in the word prediction boxes. The word prediction offers word choices based on the language model and information from the speech recognition algorithm. If the speech recognition algorithm cannot recognize the speech input, the system leaves the line of text blank and only provides word options in the word prediction boxes. If the speech-recognition algorithm accurately recognizes the target word the user then types the first letter of the next target word. If the word entered into the line of text is incorrect, or if the line of text is blank, the user either selects the target word from the word prediction boxes or spells the target word letter-by-letter. A full description of the SSR and details of the speech recognition algorithm development are available in Hosom et al. (2010) and Fager, Beukelman, Jakobs, and Hosom (2010). A video illustration of SSR can be viewed at https://www.invotek.org/collections/products/products/speech-recognition-software-with-desktop-mic.

What Do We Know So Far?

In Fager et al. (2010) the SSR’s effect on keystroke savings was examined because the system was originally designed to be an assistive writing tool for reducing physical access demands for individuals with motor impairments and severe dysarthria who used keyboards or touchscreens. There was a keystroke savings of 68% for typical sentences compared to standard text entry. Although the preliminary results demonstrated that SSR might reduce access demands for individuals with severe motor impairments and dysarthria through substantial keystroke savings, further investigations are needed to understand the performance of individuals with a range of dysarthria types and severities, the cognitive processing load required to use SSR, and the impact of using this technology in a range of communication settings.

Future Directions

Some individuals with complex communication needs prefer to use their natural speech. Coupling other input modalities (e.g., gesture cues, word and topic supplementation) with improved speech recognition may lead to more efficient and preferred access methods. Including individuals with severe speech impairments in the design of new speech-controlled technologies is important because they have been left behind in the advancements of current, commercially available technologies even though they could benefit extensively from improvements in speech recognition.

Supplementing Access with Partner Input

Literate adults who rely on single-switch scanning for text entry often experience slow message production rates. Some may use word prediction in an attempt to increase the efficiency of text entry (Lesher, Moulton, & Higginbotham, 1998). However, words that are infrequent or not within the language model’s word database, including proper names, acronyms or abbreviations, often do not show up in the system, requiring full typing (Roark, Fried-Oken, & Gibbons, 2015). In an effort to decrease effort and/or increase rate of communication, familiar conversational partners often provide spoken word choices to the person using the AAC device during message production, but doing so violates the production modality: While the user produces written words, the familiar partner suggests spoken words. The dyad must decide whether to take the time for the message to be written in its entirety or to stop, mid-typing. This approach creates more efficient but awkward text entries, with violations of both written language and spoken conversation rules. These conversational challenges are common for people who rely on AAC and their communication partners (Blackstone, Williams, & Wilkins, 2007).

Although word supplementation by partners might help with typing speed, people who rely on single-switch scanning are faced with additional challenges because partners often disengage from conversation during long, inactive waiting times. Lack of engagement is a serious obstacle to successful conversation using AAC (Hoag, Bedrosian, McCoy, & Johnson, 2004). Because of the slow interactions and timing violations, the conversation partner may lose focus and interest in the exchange, resulting in shorter or less satisfying conversations than those experienced by peers using spoken language (Higginbotham & Wilkins, 1999). Conversation management strategies are required so that communication partners can more effectively engage and participate in conversations with people who rely on switch-scanning communication devices. With word prediction and wireless connections, new forms of vocabulary supplementation are now possible to support communication for people who prepare messages with speech-generating devices, as in the following case example:

Pam is a 45-year old woman with severe dysarthria and quadriplegia secondary to spastic cerebral palsy. Pam relies on an AAC device with single-switch scanning for conversations when her speech is not understood. She has much to talk about, as she is employed in the universal design laboratory at the local university. Unfortunately, students often disengage from conversation because of the extended time it takes her to form messages. Even with the system’s word prediction, Pam is limited in her message length. She often complains that she cannot teach new techniques, ask questions, or give her opinion in a timely fashion in a group discussion.

Technology Solution

The Smart Predict³ app is a novel dual-AAC application designed to support spelling- based message generation. (see https://rerc-aac.psu.edu/development/d3-developing-a-smart-predictor-app-for-aac-conversation/ for a brief video illustration). The user’s app resembles a spelling-based AAC device. It includes a keyboard that is organized to meet the typist’s preferences (e.g., QWERTY, alphabetical, or frequency of occurrence letter display) and a horizontal word prediction list of five word choices that sits between the keyboard and a message bar. Instead of only relying on word predictions generated by the language model alone (e.g., words predicted based on rules of grammar, words used frequently, etc.), word suggestions from the communication partner are integrated into the five choices. There is a bidirectional wireless connection between the AAC app and a second partner app. The partner app displays a miniature copy of the AAC app’s interface display, enabling the partner to see messages as they are produced on the AAC app. Viewing the AAC app provides additional contextual information so that the communication partner can follow message formulation in real time. At any time during message production, he or she can type in a new word or phrase, which is immediately sent to the AAC app and subsequently appears in the five-word prediction line. If a whole phrase is sent, the first word of the phrase appears initially. If the person using the AAC app chooses to select this word, the remaining words within the phrase are presented in the word prediction area as a single, multiword prediction. The text chosen from the prediction line appends to the message bar. If the communication partner words are not chosen, the individual using the app simply continues to type, and the word prediction area returns to displaying five words based on the original language model and future partner supplementations.

Although an unchosen phrase predicted by the communication partner disappears from the user interface display, it is not discarded by the AAC app. Instead, the software compares the linguistic similarity of each word in the phrase predicted by the partner to the words typed on the AAC app. This enables “near miss” predictions to still have value to the individual using AAC. For example, if the communication partner predicts “Her child laughed and laughed” and the individual using the AAC device rejects the word “child” and instead writes “daughter,” the Smart Predict system can still offer “laughed and laughed” as a phrase after daughter.

Smart Predict builds on the concept of co-construction, the joint formulation of messages that is a natural part of many spoken interactions (Engelke & Higginbotham, 2013). Communication is viewed as a cooperative effort between the two partners, one with a communication impairment, the second without an impairment. Support provided by the unimpaired partner can be considered vocabulary supplementation, a strategy that improves message transmission (Hanson, Beukelman, & Yorkston, 2013). In the context of Smart Predict, the meaning or intent of a message is established by the person with complex communication needs and joint formulation occurs with vocabulary supplementation. Co-construction, with the dual-apps, occurs in the same written modality that is produced with the AAC device. The user maintains complete control over message production while knowledgeable partners augment the AAC app’s prediction capabilities by suggesting contextually relevant words and phrases, in real time, using a second app.

What Do We Know So Far?

Smart Predict has been tested with five literate adults with complex communication needs who rely on AAC, using an alternating treatment single subject research design (Fried-Oken, Jakobs, & Jakobs, 2018). Results demonstrated the potential value of adding a human vocabulary source to software-based language models to increase efficiency of message generation for some participants who used a keyboard-based AAC app. For one participant who relied on single-switch access for picture description, selections per word were reduced by 12% during 10 min of message generation when the communication partner offered word choices; another participant who relied on direct selection for picture description reduced selections per word by 21% during a 10-min interchange. The other three participants were challenged by the tablet screen’s access method and did not demonstrate significant efficiency savings. Additional evaluations are in process for individuals who rely on scanning for message generation. It is expected that Smart Predict will offer vocabulary support as well as increased partner engagement in conversations.

Technology-assisted word supplementation takes advantage of the communication partner’s knowledge of language, context, and personally relevant vocabulary. A familiar partner almost always will provide better predictions than a relatively uninformed (albeit state-of-the art) computational language model (Roark, Fowler, Sproat, Gibbons, & Fried-Oken, 2011). One benefit of the Smart Predict prototype is the level of engagement that is afforded during interactions, as the partner is not simply sitting and waiting for message completion and transmission. A system that leverages communication partner predictions may more fully engage those partners in the process rather than expecting them to wait their turn with nothing to do. Importantly, an application such as Smart Predict provides predictive, not direct compositional, input from the communication partner. The responsibility of selecting letters and words during text entry remains with the person with complex communication needs, as the sole author of the text (Roark et al. 2011).

Future Directions

Future directions should include work to measure the effect of Smart Predict on communication partner engagement during conversations with people who rely on AAC, especially for those who type slowly with scanning access. So far the effects of Smart Predict have only been evaluated with five participants and results are equivocal because of motor demands of the typing task. It will be important to investigate who benefits from this app (and who does not), in what ways, and under what conditions. This new technology, which really provides both communication partner assistance and technology assistance, has the potential to support language and literacy learning as well as message generation for those who are already literate. Within AAC technology, context-aware supplementation by partners or by computers, where the person who relies on AAC controls all message generation, remains an achievable goal with new artificial intelligence (AI) options, especially since they are appearing within general technologies at such a rapid rate.

Where Do We Go From Here?

Historically, access to AAC devices has focused on ensuring the independence of the individual using AAC. While independence, understood as giving the individual control over the exact message he or she produces, is non-negotiable, it has come at the price of very slow communication that likely is fatiguing for the individual using AAC, makes it difficult to maintain engagement with communication partners, and limits opportunities to engage and participate. The goal of the new and emerging access technologies described in this paper is to increase the ability to access technology, reduce fatigue, and improve satisfaction with communication, particularly in relation to individuals with acquired neurological conditions whose needs remain underserved.

Although the technologies presented in this paper are promising, there is more work to do. Access technologies are needed that (a) do not require precise setup, (b) support emerging motor abilities, (c) support message correction, (d) support control of multiple smart devices, and (e) reduce cognitive processing demands associated with alternative access solutions. Despite the advances in standard and assistive technologies, individuals with the most severe motor impairments are experiencing a growing digital divide as their needs continue to be unmet.

Most access methods still require precise positioning of individuals using AAC and their equipment for reliable access. New movement-sensing technologies such as the highly-integrated inertial measurement units (IMUs) presented in this paper hold promise for relaxing setup requirements and reducing access errors and fatigue resulting from unintentional movements, such as spasms or coughing. There is also a desperate need for access technologies that support emerging access skills. Some technologies, such as head-movement trackers (e.g., HeadMouse® Extreme⁴ or Glassouse⁵) and laser-pointing systems (e.g. Safe Laser System³), for example, enable an individual to transition from partner-interpreted pointing (i.e., a partner watches mouse or laser movements and judges the intent of those movements), to partner selection of targets (e.g. the partner uses a secondary switch to select items when the individual using assistive technology is on their target), to independent access via dwell. Also needed are new technologies that leverage artificial intelligence strategies or partner knowledge and abilities to reduce the effort required by individuals who use AAC and assist with message correction. In addition, many individuals who use technology, regardless of physical ability, need or want to use multiple smart devices; more technologies that support access to multiple devices, such as Tecla-e⁶, BlueHub³, and BlueSwitch³, are needed to support the access needs of all. Finally, continued research is needed to understand the human processes involved in using new access technologies, such as a BCI speller, to ensure that development efforts not only reduce motor demands but also consider cognitive processing demands associated with alternative access solutions. The cognitive demands associated with using a BCI speller have not been empirically examined yet, and only single case reports are available.

Most importantly, a person-centered design is essential to future access technology development, and models of person-centered design that identify real-life access challenges and closely integrate the experiences and feedback of all key stakeholders (e.g., individuals with complex communication needs, family, teachers, caregivers, AAC device manufacturers), must become standard requirements (Shah et al., 2009). Research and development efforts must incorporate evidence related to the human processes required for access; iteratively develop and evaluate prototype technologies; and work closely with key stakeholders for final development, evaluation, and implementation. This process is essential to ensure that the technology meets the needs of all individuals with complex communication needs in order to maximize opportunities to communicate and thereby increase participation and independence and reduce abandonment or disuse of technology. In addition, research efforts must be directly translated into clinical practice (McNaughton et al., 2019). Only then can future access technology development efforts begin to close the digital divide for all individuals with complex communication needs.