Tongue-Operated Assistive Technology with Access to Common Smartphone Applications via Bluetooth Link

Abstract

Tongue Drive System (TDS) is a wireless and wearable assistive technology (AT) that enables people with severe disabilities to control their computers, wheelchairs, and electronic gadgets using their tongue motion. We developed the TDS to control smartphone's (iPhone/iPod Touch) built-in and downloadable apps with a customized Bluetooth mouse module by emulating finger taps on the touchscreen. The TDS-iPhone Bluetooth mouse interface was evaluated by four able-bodied subjects to complete a scenario consisting of seven tasks, which were randomly ordered by using touch on the iPhone screen with index finger, a computer mouse on iPhone, and TDS-iPhone Bluetooth mouse interface with tongue motion. Preliminary results show that the average completion times of a scenario with touch, mouse, and TDS are 165.6 ± 14.50 s, 186.1 ± 15.37 s, and 651.6 ± 113.4 s, respectively, showing that the TDS is 84.37% and 81.16% slower than touch and mouse for speed of typing with negligible errors. Overall, considering the limited number of commands and unfamiliarity of the subjects with the TDS, we achieved acceptable results for hands-free functionality.

I. INTRODUCTION

According to the National Spinal Cord Injury Statistical Center, the number of people having Spinal Cord Injury (SCI) in the United States has been estimated at approximately 265,000 persons in 2010. There is a growing interest in improving the quality of life of individuals with disability by helping them to become more independent [1]. Therefore, numerous assistive technologies (AT) have been developed, such as sip-and-puff, eye tracking system, electromyography (EMG) switches, voice recognition, brain-computer interfaces, and head arrays [2].

We have developed the Tongue Drive System (TDS) for people with severe disabilities to control computers and their environment using their tongue motion. TDS translates the user's volitional tongue movement into commands, which is then interfaced to external devices to give more freedom and independence in their daily lives. TDS has been evaluated with respect to its efficiency for computer access [3], [4]. We have developed iPhone apps as a key controlled device by TDS to dial phone numbers [5] and as interface to control the powered wheelchairs (PWC) using tongue motion [6].

According to ComScore report, released in Jan. 2012, there are more than 100 million smartphone subscribers in the U.S., and it has been reported that about 500,000 apps are available in iPhone's app store (Apple, Inc.) and about 400,000 apps in Android' market (Google, Inc.). However, there are only a few ATs that can interface with smartphones, most of which are still limited to specific applications. The first example is the sip-and-puff controlling iPod [7], which prompts the user to select from the iPod menu. Another example is a commercially available AT interface for the Android called Tecla [8], which uses customized hardware and applications that enable certain degree of control on the main menu of the smartphone. It is a scanning based system, which moves the cursor on a soft keypad on Android phone screen. Both systems provide more independence for the users, however, they are yet to become intuitive and freely navigable.

To implement the mouse function on the smartphone, we emulated a mouse sensor inside a commercial Bluetooth mouse using TDS commands. Therefore, the TDS control is no longer limited to a certain application, and users have access to all built-in and downloadable apps, along with full navigation capabilities of the iPhone touchscreen.

Fig. 1 shows that the TDS can be used for different tasks as a unified AT, with the next step being to access additional applications available in the market through a Bluetooth module. The new interface introduced in this paper allows the external TDS (eTDS) headset to communicate with the smartphone and relay the sensor data to the signal processing algorithm, running on the smartphone, and turn the raw data into control commands for computers, smartphones, and PWCs. Therefore, we have created a universal interface using TDS to infer user's intentions via their tongue movements.

Fig. 1.

Block diagram of the external Tongue Drive System (eTDS).

II. SYSTEM ARCHITECTURE

Fig. 2a shows the TDS-iPhone Bluetooth mouse interface test setup, being used by one of the subjects. This prototype consists of an ergonomically designed eTDS headset (Figs. 2a and 2c), a permanent magnetic tracer attached to the user's tongue (Fig. 2a inset), and an iPhone/iPod touch. The TDS-iPhone Bluetooth mouse interface connects to iPhone via a 30-pin connector, while being encased in a commercial Bluetooth mouse (Fig. 2a). When the Bluetooth connection is established, a custom-designed application running on the background of iPhone receives the wireless sensor data and classifies the raw data into TDS command that control the mouse cursor movements. The application can be expanded to be used for computer access and wheelchair navigation. However in this paper, we focus on accessing and usage of the smartphone itself via the TDS.

Fig. 2.

TDS-iPhone Bluetooth mouse interface testing setup: (a) A user wears eTDS headset while attaching a magnetic tracer near the tip of the tongue (inset). (b) TDS-iPhone Bluetooth mouse interface circuit board. (c) Complete eTDS prototype headset and magnetic sensor board.

A. External Tongue Drive System (eTDS) Headset

The eTDS prototype consists of four 3-axial magnetic sensors (Honeywell, Plymouth, MN) that are held bilaterally near the user's cheeks with a pair of adjustable poles to facilitate sensor positioning. The control unit is enclosed on the back of right ear and power management circuitry with a rechargeable lithium-ion battery (3.7 V, 130 mAh) is housed backside of the left ear (see Fig. 2c). The control unit includes sensor interfacing, which controls magnetic sensor duty cycling, amplifying, and digitizing, a low-power system-on- chip microcontroller (CC2510, Texas Instruments, Dallas TX) with 2.4 GHz RF transceiver, and a miniature sized 2.4 GHz chip antenna.

A small disk-shaped (⁄4.8 mm × 1.5 mm) rare earth permanent magnet (K&J Magnetics, Jamison, PA) was attached near the tip of the user's tongue using a drop of Cyanoacrylate adhesive (GluStitch Inc., Point Roberts, WA). A more detailed description of the eTDS headset prototype can be found in [9].

B. Sensor Signal Processing Algorithm

Sensor signal processing consists of two main parts: calibration and classification. Raw sensor data is collected while fixing the position of the sensors and determining the linear relationship between various sensors while reducing the effects of the earth's magnetic field (EMF) and external magnetic interference (EMI). The SSP algorithm, which is based on principal components analysis (PCA) and K-nearest neighbor (KNN) Classifier, runs on the iPhone.

Collected data from the training of six commands, plus the neutral position, by touching particular positions inside the mouth are reconstructed in the PCA space to generate a cluster for each command. During the operating period, the incoming data is transformed and reflected onto the PCA space to be classified into one of the commands using eight different classifiers. Majority voting method picks the most likely commands from the training data, and converts it to mouse cursor movements on the iPhone screen. A more detailed description of the TDS SSP algorithm can be found in [3].

C. Tongue Drive-iPhone Bluetooth Mouse Interface

TDS-iPhone Bluetooth mouse interface consists of an RF module, PWC interface circuitry, a microcontroller (CC2510) for emulating input from the mouse optical sensor, and a Bluetooth module (BCM2042, Broadcom, Irvine, CA), which is integrated with a human interface device (HID) profile for mouse and keyboard. The block diagram is shown in Fig 3.

Fig. 3.

Block diagram of the TDS-iPhone Bluetooth mouse interface. For details of the grayed area, see [6].

We have established the wireless connection between the eTDS headset and iPhone through an RF module placed in the TDS-iPhone Bluetooth mouse interface, which is powered by the iPhone battery. The received data is conveyed to the iPhone through serial port at a baud rate of 115.2 kbps. The PWC interface, shown in Fig. 3 but not used in this paper, enables driving a PWC via TDS as explained in [6].

The commercial Bluetooth mouse (Logitech V470 Cordless Mouse, Logitech International S.A.) that was selected for this interface has an optical sensor (ADNS-7550, Avago Technologies, San Jose, CA), which optically acquires surface images. It internally processes image frames with a digital signal processing (DSP) algorithm, and the result is a 2-axial relative displacement value. It communicates with Bluetooth module through Serial Peripheral Interface (SPI) to send the directional information along with the state of two mechanical switches for left and right click. The mouse emulator micro- controller (CC2510) receives the classified TDS commands from the iPhone, and either converts them to digital signals similar to the outputs of the mouse optical sensor for the directional commands or activates a pair of MOS switches that substitute mechanical mouse buttons.

D. Tongue Drive-iPhone Interfacing Software

The iPhone4 and iPod Touch 4G are equipped with ARM Cortex-A8 Apple A4 CPU (S5L8930, Samsung, Korea) running at 800 MHz, 512 MD DRAM, and 3.5” multi-touch display with 960 × 640 pixel resolution (326 ppi).

In the current prototype, the signal processing algorithm is programmed in Objective-C provided in the iPhone Software Development kit (SDK v3.2.5, Apple Inc., Cupertino, CA).

To minimize data packet loss, the packets are designed to contain the two previous data samples. If packet loss is detected, the buffered data is applied to substitute lost data and proceed with classifying commands. The iPhone can handle processing of three data packets within 20ms.

Four directional commands move the mouse cursor on the iPhone in the cardinal directions. The last two commands are dedicated to the left- and right-click switches, which have been dedicated to selection (tapping on the touchscreen) and pressing of the “Home” button of the iPhone, respectively.

Fig. 4b shows the developed GUI for the TDS-iPhone mouse control app that is running in the background, receiving and classifying magnetic sensor data, so that the user can continuously execute and control the mouse cursor, while browsing outside of this app.

Fig. 4.

(a) Recommended tongue positions for 6 TDS commands, (b) Mouse control application, which has receiving and classifying functions, and (c) the main menu screen to control the iPod touch to evaluate the system.

III. PERFORMANCE EVALUATION

Four able-bodied volunteers, two males and two females aged from 24 to 32 years old, participated in this trial to evaluate the functionality of the TDS-iPhone Bluetooth mouse interface, running on an iPod Touch 4G. Subjects wore the eTDS headset, as shown in Fig. 2a, and a permanent magnetic tracer was attached to their tongue, 1 cm anterior to the tip.

An auxiliary GUI was developed for calibrating and training the TDS either in LabVIEW or on iPhone. In order to provide some practicing opportunities with better visual feedback for the users, the initial TDS calibration and training procedures were completed on a PC, using a 22” LCD monitor, even though these steps could also be conducted on the smaller iPhone screen. Fig. 4a shows the recommended tongue position to train the TDS control associated with each individual command. Subjects cycled through the recommended positions, 10 times each, holding their tongue steady at each command position for a few seconds during the training procedure, which takes about five minutes.

After training the system, subjects were asked to practice the TDS commands with several video games on the computer for approximately 10 minutes. When the subjects were comfortable issuing the TDS commands, the trained data was transferred to the iPhone to be used in the SSP algorithm in real-time.

The iPod was placed 30 cm away from the user's face on a book holder with Sticky Pad^® (Fig. 2a). Subjects were required to fix their wrist on the table and only use their required to fix their wrist on the table and only use their dominant hand's index finger to touch the iPhone screen. The testing scenario was to complete seven tasks, described in detail in the following paragraph, while being recorded on a digital video camera. TDS was evaluated based in these parameters: completion time, typing speed, and the number of corrections. TDS was compared with touchscreen and Bluetooth mouse, which are the most common method to control the smartphone and computers, respectively. TDS-iPhone Bluetooth module operates by issuing TDS commands, which replaces the mouse commands with controlling only 4-directions (left, right, up, and down). The system also recognizes the left-cheek touch as selection and the right-cheek touch as the iPhone Home button.

Individual apps used for testing are shown in Fig. 4c.

• Phone call (Talkatone, Talkme.im, Inc.): press your phone number and when you hear the ring, press the hang-up button.

• Text messaging (Kakao talk, Kakao Co.): enter and send the given message to the registered number.

• Tic-Tac-Toe game (Optime Software LLC): play the game and complete it once.

• Facebook (Facebook, Inc.): find the “GTBionics” group page and leave a given message.

• Calendar: Find today and create a schedule.

• Mail: send an email to given address.

• iPod Music: find a given song on the list and play it.

Different tasks were randomly selected and each of them was evaluated by 4 rounds. Each subject went through 12 rounds of task scenarios. The first trial of each method was considered as the practice round. The entire session took ~60 min including the TDS calibration and training.

Experimental Results

All subjects successfully completed the assigned set of tasks. Two of the subjects had previous experience with the TDS, but not on a regular basis. Fig. 5a shows that completing the tasks with touchscreen, mouse, and TDS took 165.6 ± 14.5 s, 186.1 ± 15.4 s, and 651.5 ± 113.4 s, on average, respectively. In terms of characters typed per minute (CPM), these methods yielded 74.94, 62.15, and 11.71 CPM, respectively, as shown in Fig. 5b. Therefore, it is 6.39 and 5.31 times faster to type with touchscreen and mouse, respectively, than with the TDS. This is because the TDS had only 4 directional commands with no cursor acceleration, and the subjects were not as familiar with it as with the other two human interface devices.

Fig. 5.

(a) Average time to complete a task scenario. (b) Speed of typing in characters per min (CPM). (c) The number of corrections. Error bars on the graphs show the standard deviation.

Because of the small size of the iPhone keypad, the subjects’ fingers easily hit the nearby keys (See Fig. 5c). Thus the touch method showed the largest number of the corrections. Even though the typing speed of mouse was slower than touch, the users rarely hit the unintended keys. TDS was not as accurate as mouse, but fewer errors were recorded compared to the touchscreen.

IV. CONCLUSION

Tongue Drive System users can now access smartphone (iPhone/iPod Touch) universal applications by connecting a Bluetooth mouse module. It expands the TDS reach to numerous built-in and downloadable applications. TDS-iPhone Bluetooth mouse, which emulates the mouse sensor with TDS commands, can be used as an extension to replace the mouse controller. The significance of the TDS-smartphone Bluetooth mouse interface is that it enables the TDS user to fully operate the smartphone (as well as computer and PWC) without additional peripherals. We first tested the controllability of the iPhone. In the near future we will include further evaluation with PWC and other environmental controls. We chose iPhone as a popular smartphone to implement these functions. However, other smartphones can be used with minor modifications in the Bluetooth Mouse interface. The comparison between touchscreen and mouse control on smartphone showed better results than the TDS control. This is due to the lack of cursor motion profile (acceleration/deceleration) and proportionality in the current TDS system, and perhaps lack of subjects’ experience. Future development of the TDS-smartphone Bluetooth mouse interface and advanced signal processing algorithms are needed to add proportional control capability. More detailed evaluation of the TDS by end users with disabilities, not only in a controlled lab environment but also in real life situations from home to the workplace and the outdoor environments are among our future research directions.