Access Interface Strategies

Abstract

Individuals who rely on augmentative and alternative communication (AAC) devices to support their communication often have physical movement challenges that require alternative methods of access. Technology that supports access, particularly for those with the most severe movement deficits, have expanded substantially over the years. The purposes of this article are to review the state of the science of access technologies that interface with augmentative and alternative communication devices and to propose a future research and development agenda that will enhance access options for people with limited movement capability due to developmental and acquired conditions.

INTRODUCTION

The augmentative and alternative communication (AAC) options are increasing dramatically for people unable to meet their communication needs through natural speech, handwriting, or typing. It is encouraging that proposed technical advances promise to provide even greater access to face-to-face and electronic communication options that will support social, recreational, educational, commercial, volunteer, and employment engagement (Shane, Blackstone, Vanderheiden, Williams, & DeRuyter, 2012). However, these levels of communication support require that those who rely on them can accurately and efficiently interact with these technologies.

The purposes of this article are to review the state of the science with regard to physical access of AAC technologies and to propose a future research and development agenda that will enhance access options for people with limited movement capability due to developmental and acquired conditions. This article focuses on direct AAC access strategies involving the tracking of head and eye movement, recognition of residual speech, recognition of gestures, and monitoring of the electrical activity of the brain.

MINIMAL MOVEMENT: HEAD, EYE

Many people with complex communication needs also experience such limited movement of their arms and hands that they must rely on head and/or eye movement to access their communication options. Through history, listeners (communication partners) have interpreted head or eye movements as they co-constructed messages with the person who experienced complex communication needs (CCN). During the last two decades steady progress has been made in providing alternative technical access using head and hand movement. For more detailed descriptions of access strategies discussed in this article, readers are referred to Beukelman and Mirenda (2005) and Cook and Polgar (2007).

LISTENER PERCEPTION OF HEAD OR EYE MOVEMENT

Augmentative and alternative communication access involving communication partner perception of residual eye or head movement is relatively common for people with minimal movement and AAC needs (Garrett, Happ, Costello, & Fried-Oken, 2007; Culp, Beukelman, & Fager, 2007; Hurtig & Downey, 2009). Examples of no-technology options include upward head or eye movement to signal “yes” and downward movement to signal “no,” or lateral movement to signal “more” or “less,” “slower” or “faster,” or “go” or “stop.”

Communication partner perception of eye gaze is also a common AAC strategy. The person with AAC needs looks toward an item (object, image, or printed message) in the environment or on a communication board, thereby directly selecting it, as the communication partner co-constructs meaning by confirming the meaning or message associated with the gaze.

Eye-linking is a low-tech communication option in which multiple images are placed or mounted on a transparent sheet. The communication partner visually focuses at the eyes of the person with AAC needs, who looks at the image of interest. The communication partner then moves the sheet until their eyes meet because they are both looking at a shared item. At this point, the communication partner confirms the meaning or message to be conveyed.

EYE TRACKING STRATEGIES

In the past decade, there have been significant advances in eye tracking using infrared technology. Eye tracking systems work by reflecting safe, invisible infrared light on the surface of the eye. This light causes a reflection on the user's pupil that the computer cameras can track. The software program correlates the reflected infrared light and the computer mouse with information gathered through a calibration routine. Symbols, pictures, messages, and/or letters are presented on the computer screen or speech generating device (SGD). Individual selections of this content are made through pausing (dwelling) on a location, blinking, or activating a switch when the cursor is located on the preferred location (item) on the screen. A wide range of SGDs now incorporate eye tracking as a potential access method through an accessory that can be added to the system.

Research into the use of eye gaze technology to access SGDs to support communication is beginning to emerge. Individuals with amyotrophic lateral sclerosis (ALS) have received the most attention in the research literature regarding the use of eye gaze technology. ALS is a degenerative condition that can leave the individual in a completely locked-in state with only eye movement preserved at the end stages of the disease process. Some have documented the relative level of ease of use of eye tracking technology experienced by individuals with ALS (Calvo et al., 2008; Harris & Goren, 2009; Gibbons & Beneteau, 2010). Ball and colleagues (2010) followed 15 individuals with ALS who used eye tracking technology to support communication. They found a wide range of communicative functions served using these systems (group communication, phone, e-mail, face-to-face interaction, and internet). Others have also reported a variety of communicative functions served by this technology for persons with ALS (Fried-Oken et al., 2006; Doyle & Phillips, 2001; McNaughton, Light, & Groszyk, 2001; and McNaughton & Bryen, 2002).

HEAD TRACKING STRATEGIES

Head tracking technologies use video or infrared cameras. Video camera-based systems track specific body features (e.g., the tip of the nose) and translate that into cursor control on a computer screen (Betke, Gips, & Fleming, 2002; Kim & Ryu, 2006). Infrared systems track a reflective dot or configuration of dots placed on the forehead, glasses, brim of a hat, or the individual's hand or finger. Through the years several variations of infrared head tracking applications have been developed. In general, some are relative strategies in which the cursor movement and head movement are calibrated by moving the head from side-to-side and up-and-down, thereby moving the cursor to the edges of the screen. Absolute strategies involve the calibration of the movement and cursor locations to specific locations on the screen. Communication content is represented on a computer or SGD screen similar to what has been described with eye tracking. Selections, similar to eye tracking, are often made by dwelling or can be made with the use of a switch activated by an additional movement (e.g., arm or hand movement). Case study illustrations are available that describe the use of this technology for individuals with disabilities (Fager, Bardach, Russell, & Higginbotham, 2011; Man & Wong, 2007; McKinley, Tewksbury, Sitter, Reed, & Floyd, 2004).

EYE AND HEAD TRACKING FUTURE RESEARCH

There are several important directions for future research into eye and head tracking technology. First, the clinical training and on-going support required of this sophisticated technology option to support communication needs over time is yet to be understood. Additionally how adverse environmental conditions affect technology stability and performance (natural light, changing positions over time due to medical needs, changing environments) needs to be documented. This knowledge not only drives technology development (making technology simpler and more robust in a wide range of settings), but drives models of clinical intervention to ensure successful implementation.

Relative and absolute head tracking each require different levels of head movement capability. Relative systems often require extremes in head movement with use in order to keep the cursor aligned on the computer or SGD screen. The physical demands of long-term use of these systems warrants further investigation as individuals who use these technologies may spend significant amounts of time using the technology to support their daily communication and computer access needs.

Finally, the use of eye and head tracking technology with other access methods needs to be explored. For example, the combined use of eye movement with other physical movements (i.e., head movement, gesture, speech recognition) may provide an added benefit or more efficient or intuitive access method. There is limited understanding of what combinations of access methods are best-matched for specific tasks.

HEAD POINTING STRATEGIES

Two different laser strategies have been utilized to access AAC technology or materials. A conventional laser pointer has been mounted on the head or, in some cases held in the hand or attached to a foot or hand. The laser beam is then directed toward an item, object, or image in the environment or on a communication board. The communication partner perceives the laser beam on the item (object, print, icon, or object) and confirms the message or meaning being communicated. A concern about this type of laser use is the eye safety of the communication partner or others in the environment. There is particular concern if young children or people with cognitive disability are not aware of the eye danger and may look directly into the laser device to investigate the light source.

The Safe-Laser is a second laser strategy. This laser system operates at a relatively low (eye-safe) level of intensity until it is pointed toward a laser-sensing surface at which it shifts to higher power. The Safe-Laser can be used to identify images positioned on the laser-sensing screen and the communication partner can confirm the message. The Safe-Laser can also be used to electronically activate a location on the laser-sensing screen, thereby accessing a stored spoken message, printing a message, or providing environmental control.

The authors have been involved in research of safe laser technologies. In an investigation of six individuals with locked-in syndrome (LIS) due to brainstem stroke, safe laser technology was used as a head movement training system (Fager, Beukelman, Karantounis, & Jakobs, 2006). Individuals with minimal head movement capabilities could use the safe laser technology if the laser sensing surface was moved a sufficient distance from the laser pointer. Three of the participants transitioned from using the safe laser to head tracking technologies. One individual continued to use the safe laser to support communication. Two of the participants discontinued use of the system due to ongoing medical setbacks.

Future research on head pointing technologies needs to further address the integration of this technology as a movement training device as well as a system to support communication. How this technology is used over time and the potential impact on range of movement as well as the ability to support independent communication needs to be explored.

Individuals who are early in recovery and who have minimal movement capabilities often have to rely on partner dependent and scanning strategies to support communication. Because safe laser technology can be used with minimal movement as a direct selection access strategy, the use of this technology in acute medical care settings needs to be explored. Additionally the use of this technology for environmental pointing (e.g., controlling the fan, call-light) and to support communication using items in the environment needs to be researched.

RECOGNITION OF RESIDUAL SPEECH

Automatic speech recognition (ASR) is rapidly being integrated into computers, gaming, and mobile device applications. ASR is standard on many computers (e.g., Microsoft Speech Recognition), and commercially available ASR has expanded to support a range of professions with specialized vocabularies (e.g., Dragon NaturallySpeaking). Mobile devices (iPad, Android, etc.) come equipped with voice command menus and specialized apps are available to assist in web searches and other applications. Simply speaking to your computer or mobile device is appealing, particularly for individuals with physical impairments impacting control of their arms and hands. However, this becomes challenging when the speaker has dysarthria, or impaired speech capabilities.

Limited reports exist of functional (sufficient to support writing and daily communication) use of ASR for individuals with moderate and severe speech impairments. Most reports of success are with individuals with mild impairments (Hux, Rankin-Erikson, Manasse, & Lauritzen, 2000). This can be attributed to several reasons. First, commercially available ASR has been developed for a market of typical (non-impaired) speakers. As such, the technology has been developed on models of non-impaired speech. The strategies that current ASR technology implement require the speaker to talk continuously making it difficult or impossible for some individuals with severe speech impairments to use. Additionally, individuals with dysarthric speech can demonstrate much inter- and intra-speaker variability making it difficult for the technology to reliably recognize and improve performance with use over time (Blaney & Wilson, 2000; Magnuson & Blomberg, 2000; Raghavendra, Rosengren, & Hunnicutt, 2001; Young & Mihailidis, 2010). Due to these challenges, some have focused on developing ASR technology to specifically recognize dysarthric speech. Many of these systems have focused on controlling specific software programs, environmental control, and recognition of other limited vocabulary sets (Hawley et al., 2007; Judge, Robertson, Hawley, & Enderby, 2009; Caves, Boemler, & Cope, 2007; Omar, Morales, & Cox, 2009; Hamidi, Baljko, Livingston, & Spalteholz, 2010).

Others are beginning to look at recognition of residual speech as a part of an overall access strategy for communication and assistive writing. Invotek, Inc., (Alma, AR, USA) a partner in the Rehabilitation Engineering Research Center for Communication Enhancement (AAC-RERC), has developed a Supplemented Speech Recognition (SSR) program with collaborators. This unique system incorporates ASR that has been developed on models of dysarthric speech with a method for individual speaker optimization (training), first letter cues or alphabet supplementation and word prediction (Fager, Beukelman, Jakobs, & Hosom, 2010). With multiple sources of information (audio signal, first letter cues, language model) as well as recognition that is based on dysarthric speech, the SSR gave speakers with a wide range of intelligibility (16.4%–89.1%) a high level of keystroke savings (64.8%–70.1%). For individuals with severe physical impairments, keystroke savings can be of particular benefit as it decreases the amount of precise and accurate movement control required to use a keyboard. Others are beginning to incorporate multiple sources of information that may be of benefit to dysarthric speakers. For example, SpeakQ by goQ utilizes speech recognition as well as word prediction. Deng and colleagues (2009) have investigated the use of speech recognition with sEMG signals for individuals with dysarthria.

There are several issues that arise from previous research on ASR with dysarthric speakers that warrant further investigation. First, variability of speech performance in dysarthria is a significant challenge for current commercially available ASR. Not only do individuals with dysarthria vary substantially in their speech performance across dysarthria types (flaccid, spastic, ataxic, hypokinetic, ataxia), but also across severity ranges (mild, moderate, severe) as dysarthria can occur from a wide range of etiologies. Personalization of speech recognition models as well as individualizing speaker templates for recognition appears to be a particularly important feature of dysarthric speech recognition. However, the amount and kind of personalization required for various types and severity levels of dysarthria is yet to be understood.

Commercially available ASR requires the speaker to produce speech in a relatively consistent pattern during use. Not only do the acoustic features of individuals with dysarthria vary between speakers, but significant variability of speech (the way a speaker produces words or specific sounds) has also been noted within some speakers. Additionally, this variability can change throughout the day due to fatigue, medications, and the nature of the disorders that cause the dysarthria. Research is needed to understand exactly how this variability impacts recognition performance and what strategies can improve performance. Additionally, with new ASR being developed that allows individuals with severe dysarthria to use their voices extensively throughout the day to support written and face-to-face communication, the impact of extensive voice use for these individuals is unknown.

Finally, the use of speech as an input method for AAC technology is becoming a possibility as strategies for dysarthric speech recognition continue to advance. However, the integration of natural speech via speech recognition technology during AAC interactions is yet to be understood. Specifically, the level of acceptance of using one's own speech, no matter how unintelligible, and having an AAC device recognize and speak the utterance intelligibly using synthesized speech, needs to be explored.

BRAIN COMPUTER INTERFACE

Twenty years ago, patients with Guillain-Barre Syndrome were asked to evaluate the AAC options they had used when they were locked in. At that time, one of the patients said, “Hey, what I really needed was a computer that could read my brain waves and speak out what I was thinking. Then none of this other stuff would be necessary” (Fried-Oken et al., 1991). Indeed, brain computer interface (BCI) technology as an AAC access method may now have the potential to meet these challenges (Kubler, Kotchoubey, Kaiser, Wolpaw, & Birbaumer, 2001).

Within the past two decades interest in the development of BCIs for function has been sparked by significant advances in computing capabilities, miniaturization, signal acquisition, and processing abilities. Also, we have a solid body of research from animal models that contributes to the rehabilitation technology efforts. We have learned so much about brain behavior connections and what the brain is capable of in the face of severe neuromotor impairment. And most importantly, from a clinical perspective, we have changed our perception of severe disability and the potential participation of individuals with severe disability. People who are locked-in and require access methods that do not rely on neuromuscular control are speaking up (Bauby, 1997; Bieker, Noethe, and Fried-Oken, 2011). It is time to examine the future of BCI as a plausible access method for AAC users. Our challenge is to find an independent, user-friendly means of expression that does not rely on neuromuscular control and produces written and spoken language that is fast, accurate, and non-fatiguing, with unlimited vocabulary.

What is a BCI? Brain-computer interface refers to technology whereby a computer detects a “selection” made by a person who does not rely on neuromuscular activity. Rather, the technology uses the person's changes in brain electricity as the intended execution. BCIs can substitute for the loss of typical neuromuscular outputs by enabling people to interact with their environments through brain signals rather than through muscles (Wolpaw, Birbaumer, McFarland, Pfurtscheller, & Vaughan, 2002). There are a number of different brain electrical signals that can serve as the intended selection method for the BCI. They are categorized as invasive and non-invasive techniques. The invasive BCI uses recordings of neuronal action potentials (spikes) or local field potentials (FLPS) where an electrode array is placed directly on to the cortex (Wolpaw & Birbaumer, 2006). The Braingate™ Neural Interface System is one example of an invasive BCI where intracortical microelectrode sensors read control signals directly from the motor cortex (Donoghue, Nurmikko, Black, & Hochberg, 2007). Non-invasive techniques use recording sites at the scalp (electroencephalographic activity or EEG) or rely on magnetic brain forces, magnetoencephalography (MEG and Magnetic Resonance Imaging (MRI) (Birbaumer & Cohen, 2007). There is a “partially invasive” technique called electrocorticography (ECoG) where sensors are placed within the skull but outside the gray matter of the brain. Non-invasive methods have the challenge of filtering out noise from a brain signal that is far from its source, but signal processing techniques are improving rapidly so that the advantages of lower cost, portability, lack of infection concerns, no surgery, and faster applications are become more appealing.

Presently, most human BCI systems used for communication and control rely on non-invasive EEG-based methodologies. The EEG is a popular recording method since it can measure spontaneous electrical activity of the cerebral cortex as well as cortical responses to external or internal events. Both the Visual Event Potential (VEP) and the Event-Related Potential (ERP) or P300 have been used for BCI signals (Kubler et al., 2001). The P300 produces an electrical response that is time-locked to a physical stimulus or behavior, and can be characterized as a positive peak in the EEG that occurs 300–600 ms after stimulus onset. The P300 is a well documented EEG response to a salient event, and has been used as a metric of cognitive function, attention and cognitive workload (Oken & Phillips, 2009). Consider the person who is looking at a series of numbers. All of a sudden, a letter appears. That letter is considered a salient and novel event and will cause an involuntary P300 response, which can be measured with surface brain electrodes. This response, when measured in multiple stimulus-response times, is considered the “virtual key press.”

Who can benefit from BCI? BCI as an alternative access method is often considered for individuals with LIS. Plum and Posner (1983) described the LIS as

“a state in which selective supranuclear motor de-efferentation produces paralysis of all four limbs and the last cranial nerves without interfering with consciousness. The voluntary motor paralysis prevents the subjects from communicating by word or body movement. Usually, but not always, the anatomy of the responsible lesion in the brainstem is such that locked-in patients are left with the capacity to use vertical eye movements and blinking to communicate their awareness of internal and external stimuli.”

Bauer, Gerstenbrand, & Rumpl (1979) subdivided the syndrome on the basis of the extent of motor impairment: (a) classical LIS is characterized by total immobility except for vertical eye movements or blinking; (b) incomplete LIS permits remnants of voluntary motion; and (c) total LIS consists of complete immobility including all eye movements combined with preserved consciousness. We propose to use the term in a functional framework. We refer to individuals who are “locked-in” as those who cannot rely on motor or speech skills to conduct activities that allow them to participate in the social, economic, and cultural aspects of their environments. We refer to potential users as individuals with severe speech and physical impairments (SSPI). This population of patients presents with severe dysarthria (Duffy, 2005) and severe motor impairments secondary to tetraplegia or quadriplegia. The most common etiologies of SSPI, or functional LIS, may include (but are not limited to): acquired neurological disease such as ALS and motor neuron disease; Parkinson's disease, Parkinsonian-plus syndromes, and other movement disorders; multiple sclerosis and neuroimmunologic disease; Guillain-Barre Syndrome; basilar artery strokes and other CVAs; muscular dystrophy; spinal cord injuries; traumatic brain injuries; as well as neurodevelopmental disorders such as cerebral palsy.

What BCIs do we have today? A number of noninvasive BCIs are available today, both for people with neurologic impairment, as well as for non-medical users. Blankertz and his colleagues (2010) discuss uses of the BCI for entertainment (such as the popular Emotiv™ for gamers) and for mental state monitoring (i.e., attention feedback and improvements for human performance). For communication and control access, the P300 Speller is a popular BCI that is used by people with LIS secondary to brainstem stroke, ALS, and spinal cord injury (Sellers, Krusienski, McFarland, Vaughan, & Wolpaw, 2006). The system relies on a P300 response when a 6 X6 matrix of characters randomly intensifies a row or column of letters (Farwell & Donchin, 1988). A number of trials must be averaged to classify the P300 as a reliable character selection, and the speed of the highlighting determines the number of characters processed per minute. The Berlin BCI or Hex-o-Spell relies on a circular array of letters and has been found to be more accurate for spelling than the standard letter matrix (Treder & Blankertz, 2010). The rapid serial visual presentation (RSVP Keyboard™) BCI, being developed at the Oregon Health & Science University (Orhan et al., 2011), uses individual letters presented singly on the screen with RSVP Keyboard™ for single-event classification of the P300. When a letter selection is validated, a language model works to strengthen future selections (Roark, de Villiers, Gibbons, & Fried-Oken, 2010; Roark, Gibbons, & Fried-Oken, 2010).

What should we expect from BCI in the future? Like with any assistive technology, the BCI of the future must be safe, convenient, and reliable for long term use. BCI is not presently a practical and available technology for people with LIS. Current efforts have focused primarily on the technical and electrophysiological aspects, and have not yet addressed the human interface and usage concerns. Many questions remain for clinical research, and many technical issues still need attention. We are challenged to explore the functions that BCI could provide for daily participation. We do not know what levels of cognitive communication skills are required, or what kind of training is needed for competent use. We do not yet know who will be successful potential users. The current BCI systems call for substantial concentration and motivation, with limited feedback or reward during automatic selection. The level of fatigue that is permissible is not yet understood. There are significant of obstacles for set up and calibration, problem solving, and reliability checks. A BCI system now must be administered by a technical expert. Ideally, it will not require continual technical support and can be used permanently in daily life with ease (Daly & Wolpaw, 2008). We predict that the BCI of the future will manage speed and accuracy tradeoffs for functional communication, but we have not yet examined these issues. Likewise, we need to compare this access method to available eye tracking and single switch devices for individuals with SSPI, and determine if there is a use benefit.

The development of functional BCIs to serve as alternative access methods for AAC technology is a multi-disciplinary, translational endeavor that will continue for many years before we reach our goal of finding an independent means of expression that does not rely on neuromuscular control. The task will require collaboration with experts in signal processing, computer miniaturization, computer language specialists, clinical teams of neuroscientists and communication specialists, rehabilitation experts, and users (Wolpaw et al., 2002). We, as AAC specialists and rehabilitation engineers, are new to this clinical research agenda and have much to offer as this promising access method is realized for people with significant motor impairments.

GESTURE RECOGNITION

Historically, word prediction, semantic compaction, and abbreviation expansion strategies for people with CCN have been used to reduce the amount of typing required to enter text; however, as with letter-by-letter typing all of these techniques require the user to precisely access each key or interface button involved in the access strategy. For head or eye tracking access strategies, the cursor or laser beam must be directed at an interface button and that button activated through dwell or acceptance switch.

Within the last few years gesture input strategies such as SWYPE and Shapewriter offer text input without requiring precise targeting of each letter; however, complete words must be gestured (with a few exceptions for double letters, apostrophes, etc.). These applications involve touch assess with the screen of a mobile device such as a smart phone, iPad, or tablet computer.

Recently, InvoTek, an AAC-RERC Partner, has developed a prototype gesture input method that only requires the first few letters of a word to be gestured either through touch access or head tracking. If the gesture points to a highly likely word, that word is automatically inserted into the sentence and alternatives are presented in a word prediction list. If multiple words are likely, the word prediction list is populated but the most likely word is not automatically inserted into the sentence. This predictive gesture prototype provides three important advantages over other AAC access strategies for people with disabilities. The number of precise activations required to enter text was 64% less than normal typing and 47% less than using word prediction.^1,2 Gesturing reduced the vigilance required to monitor the word prediction list and enabled the writer to maintain focus on composing their message. Gesture users accessed the word prediction list 78% fewer times, on average, than when using word prediction.

Future research is needed to study the use accuracy and efficiency of these new gestures technologies by people with a range of movement limitation patterns. Additionally, there has been essentially no research investigation instructional and learning needed to develop proficiencies with such technologies. Finally, there is an ongoing need to improve these access strategies to meet the needs of people with different types of movement patterns.

Given the above preliminary results, we believe that predictive gesturing promises to substantially reduce the workload (cognitive and physical) for people with severe disabilities when writing. The reduction in precise activations and word prediction vigilance could reduce the cognitive load of message preparation for people who presently have few choices.

CONCLUSION

The range of access technology options for individuals with severe physical impairments is rapidly expanding. Along with the new innovations, there is considerable need for research to understand the impact of these technologies on the communication of individuals with severe physical impairments.

While access technologies have developed over the past decade, most of these methods do not yet interface with mobile technologies (i.e., iPad, iPod, mobile tablets). Switch scanning applications are now beginning to emerge; however, touch-screen access to these devices (with no modifications for sensitivity) remain the primary access method to mobile technologies. For individuals with severe physical impairments, these devices remain largely inaccessible.

Additionally, there is limited information available about the use of language modeling techniques to provide necessary vocabulary to individuals using different access options as communication contexts change. Advances in how communication content is managed may have significant impact on the use of different access methods by facilitating more efficient and effective interaction.