Quantitative and Comparative Assessment of Learning in a Tongue-Operated Computer Input Device—Part II: Navigation Tasks

Abstract

Tongue drive system (TDS) is a novel tongue-operated assistive technology (AT) for the mobility impaired, to empower them to access computers and drive powered wheelchairs (PWC) using their free voluntary tongue motion. We have evaluated the TDS performance in five sessions over 5–8 weeks to study the learning process in different tasks of computer access and PWC navigation on nine able-bodied subjects who already had tongue piercing and used our magnetic tongue studs throughout the trial. Computer access tasks included on-screen maze navigation and issuing random commands to measure the TDS information transfer rate. PWC navigation included driving through a ~50-m obstacle course using three control strategies. Some of the qualitative aspects of using the TDS were also evaluated based on the two Likert scale questionnaires, one of which was short (eight questions) and asked at the end of each session and the other one (46 questions) at the end of the trial. Included in this study was also a task to measure the tongue fatigue as a result of using the TDS continuously for a few hours. All performance measures showed significant improvement from the first to the second session as well as further gradual improvements throughout the rest of the sessions, suggesting a rapid learning process.

I. Introduction

There has been considerable growth in technologies that assist people with functional disabilities over the last two decades, and the need to maintain this trend or even accelerate it becomes more important as the population ages [1], [2]. Three key areas where new technologies can offer assistance to people with severe physical disabilities in their daily lives are computer access, wheeled mobility, and environmental control [3]. Although complete functional recovery for the majority of disabilities is still beyond the reach of the most advanced assistive technologies (ATs), they can offer life changing aid in daily living activities as well as education, vocation, and social participation [1], [4]–[6]. Another important role that ATs play, which affects not only the individuals with severe disabilities but also the society as a whole, is to reduce the individuals’ dependence on caregivers and consequently decrease their healthcare and assisted living costs while improving their quality of life.

Main categories of existing ATs for the mobility impaired include the following [7]: 1) those that operate based on physiological signals, e.g., electroencephalogram [8]–[10], electromyogram [11]–[14]), and electro-oculogram [15], [16]; 2) those that track eyes, head, or other parts of the body [17]–[22]; and 3) those that are activated acoustically (e.g., speech), mechanically, or pneumatically (e.g., sip and puff) [23]–[27]. There also exist several ATs that can fit in more than one of the aforementioned categories [12], [20].

ATs in the first category are the most vulnerable to noise and interference, some offer limited degrees of freedom, require a high level of user concentration, and have a lengthy setup procedure. Most video-based trackers in the second category suffer from sensitivity to changes in lighting conditions and the Midas touch phenomenon (unintended commands). Moreover, there should always be a camera in front of the users that may obstruct their field of view, particularly in mobile applications. Sensor-based trackers, such as head trackers, or pneumatically/mechanically activated ATs, such as sip and puff and chin joysticks, are reliable to drive powered wheelchairs (PWC) due to their simplicity but have a high rate of exertion and may not be effective for computer access due to low number of options.

There are unique benefits in using the tongue as a manipulative appendage to operate an AT: tongue is inherently capable of sophisticated motor movements in the oral space due to the wide area of sensory and motor cortex that is dedicated to mouth and tongue [28]. Its muscles have low rate of perceived exertion, i.e., it does not fatigue easily as long as it can move freely [28]. It is easily accessible in the mouth, while being hidden from sight giving the user a certain degree of privacy. Tongue motion is not influenced by the position of the rest of the body, which can be adjusted for maximum user comfort. For instance, it does not matter whether the user is lying in bed or sitting on a wheelchair. These features have inspired researchers to develop several tongue-operated ATs [29]–[33]. These devices, however, often require bulky objects inside the mouth, which may interfere with speech or ingestion.

Tongue drive system (TDS) is a magnetic-sensor-based AT that can detect users’ voluntary tongue motion and translate them to user commands. It is wireless, wearable, and it has the capacity to provide a unified solution for computer access, PWC navigation, and environmental control [34]. TDS is designed for individuals with disabilities in their upper limbs due to amputation and neurological injuries or diseases, who have voluntary tongue motion. Every new AT for this vulnerable population needs to be quantitatively and comparatively assessed according to accepted performance measures to inform the stakeholders, such as clinicians, rehabilitation professionals, caregivers, payers, and potential end users on one hand, and to guide future improvements of that device and other similar technologies on the other hand.

One way of doing this assessment is a direct comparison between the new AT and well established commercial products by the end users. However, considering the wide range of disabilities and variety of ATs that are in the market, this approach will require recruiting a large number of subjects, which is not quite feasible. Even then, subjects’ prior exposure to one or more ATs may bias the outcomes. An alternative approach is to recruit able-bodied subjects that are naive with respect to all ATs and define tasks that are not affected by the end users’ disability, such as tongue motion. We adopted the latter approach with an additional goal of observing the learning effect in naive subjects by conducting the trials over five consecutive sessions in five weeks through a range of standard and novel tasks. The advantage of this approach is a certain level of homogeneity among subjects in terms of exposure to ATs [35]. However, there are also limitations that are discussed in Section IV.

Each session comprised of two parts: computer access (CA) and wheelchair drive (PWC). CA part comprised of four rapid tapping tasks based on the ISO9241-9 [36], which are common in evaluating nonkeyboard computer input devices, plus on-screen maze navigation, and random command selection tasks. The results of the rapid tapping tasks have been reported elsewhere [35]. PWC part consisted of navigating a PWC through an obstacle course with three control strategies, namely, unlatched, latched, and semiproportional (see Section II). In an attempt to explore the level of tongue fatigue after using the TDS for a few hours, we included a novel rapid tongue movement task at the beginning and at the end of the CA part. Moreover, to assess the qualitative aspects of the TDS functionality, subjects filled out a short questionnaire at the end of the first four sessions and an elaborate one at the end of the last session. Like other adaptive technologies, TDS needs to be initially trained by the user to know which tongue positions should be associated to which command, and the subsequent TDS performance depends on the quality of the initial training [34], [37], [38]. Thus, we also quantified the quality of the initial TDS training to explore the subjects’ acquired skills over the course of the trial. After a brief TDS overview in the following section, the experimental methodology has been described in Section III. Results are presented in Section IV, followed by a short discussion and the final remarks.

II. TDS Overview

The external TDS (eTDS) prototype, used in this study and shown in Fig. 1, consists of an array of three-axial magnetic sensors on a headgear, which are positioned symmetrical to the sagittal plane near the subjects’ cheeks to sense the magnetic field generated by a small permanent magnetic tracer fixed on the tongue via tongue piercing, implantation, or adhesives [34], [37]–[39]. In this study, we recruited subjects that had already received tongue piercing, and exchanged their tongue stud with a custom-made magnetic stud, shown in Fig. 1(f), which had an m&m shaped upper ball (ø8 mm × 3.5 mm) made of titanium, embedded with a small disk-shaped (ø4.8 mm × 1.5 mm) rare earth permanent magnet (K&J Magnetics, Jamison, PA) as the tracer. The top ball was laser welded to a 12-gauge post with the length of 12 or 15 mm, depending on the subjects’ tongue thickness. The post passed through the tongue and screwed tightly onto a spherical lower ball like a barbell. The magnetic field generated by the tracer was sampled at 50 Hz by a control unit equipped with a built-in 2.4-GHz RF transceiver (TI CC2510, Dallas, TX) on top of the headgear [see Fig. 1(c)]. The sensors’ raw data were delivered wirelessly to a PC through a wireless receiver USB dongle [see Fig. 1(d)], where the position of the magnetic tracer and the tongue was recognized in real time by a sensor-signal processing (SSP) algorithm and translated to a set of user-defined commands [34]. A custom-designed interface was used to deliver TDS commands from the PC to PWC through its standard nine-pin connector [see Fig. 1(e)].

Fig 1.

(a) eTDS prototype used in this study consists of a headgear (b) with an array of three-axial magnetic sensors (c) mounted on a pair of goosenecks and a wireless control unit. (d) Wireless receiver USB dongle is used for delivering data to a PC. (e) Custom-designed interface connects the PC to powered wheelchairs via a standard nine-pin connector. (f) eTDS detects the position of a small permanent magnetic tracer that is embedded in the upper ball of a titanium tongue stud.

TDS has six individual commands that are simultaneously available to the user [see Fig. 2(b)]: four directional commands (LEFT, RIGHT, UP, and DOWN) and two selection commands (LEFT- and RIGHT-SELECT). When using TDS for cursor control, the directional commands are used to move the mouse cursor on the screen in four cardinal directions and the selection commands are used for left- and double-click. When driving PWCs, UP, and DOWN are used to move the wheelchair forward (FD) and backward (BD), while LEFT and RIGHT are used to turn (TL and TR). The SSP algorithm ignores the tongue motion in the sagittal plane to eliminate respiratory and speech related tongue motions. To deactivate the system during eating, a specific tongue gesture (touching the left cheek with the tip of the tongue for 3 s) switches the TDS from active to standby mode, and vice versa.

Fig 2.

(a) On-screen maze navigation task. (b) Recommended tongue positions for six TDS tongue commands plus the tongue resting position, which is considered neutral. (c) Designated keys on the keypad to resemble the TDS commands positions.

III. Methods

All tasks were performed for four rounds, the first of which was considered for practice.

A. Tasks

1) Maze Navigation

In maze navigation task, subjects were instructed to move the cursor through an on-screen maze, shown in Fig. 2(a), as fast and accurately as possible. To minimize the memory effect, five different maze designs with equal number of segments and turns were used, one of which was randomly selected for each rounds. All designs were wider at the beginning (38 pixel) and became gradually narrower toward the end (15 pixels). Maze navigation task only used the four TDS directional commands. Cursor movement in each direction was unlatched, meaning that the cursor moved only as long as the directional command was being issued, in which case the cursor speed increased linearly at the rate of 500 pixels/s² until it saturated at 200 pixels/s. These values were chosen experimentally based on our pilot experiments in the development phase.

To compare the performance of tongue with that of index finger, subjects were also required to perform maze navigation with their right index finger pressing a subset of adjacent keys on a standard keypad that resembled TDS four-directional commands, as shown in Fig. 2(c). Keypad output was sampled at 50 Hz, similar to TDS, with the same velocity profile. Throughout the experiment, the order of TDS and keypad was randomized.

To quantify maze navigation performance, we calculated two indices: Task completion time (TCT) and sum of deviation (SoD) from the track. TCT, an indicator of navigation speed, was the average time that it took to complete each round over the three main rounds. SoD, an indicator of the navigation accuracy, was the area between all deviations of the actual traversed path from the edges of the track divided by 1000 pixels². Fig. 3 shows a typical corner of the maze where the subject is required to pass through segment 1 toward segment 2 in the direction of the arrows. As long as the cursor has not crossed the diagonal intersegment border, deviation is calculated with respect to segment 1 and after passing the borderline, it is calculated with respect to segment 2. Fig. 3 shows two sample paths, which go around the inner and outer corners of the track. In the latter case, deviations are calculated relative to the extensions of the two segments.

Fig 3.

Portion of the maze and two typical cursor paths from segment 1 to segment 2. SoD is the sum of all deviations from the track. When the path is around the outer corner of the track, deviation is also measured over the extensions of segments.

2) Timed Randomly Selected Commands

This task was designed to measure how quickly and accurately a random command can be issued on a visual cue. In this task, shown in Fig. 4, one out of six TDS commands was randomly selected and its indicator turned pink. At the same time, the center cue turned red, reading “Wait” for 1 s, during which subjects were required to decide the corresponding tongue position for the selected command without any tongue movements. As soon as the center light turned green, reading “Go!” subjects were asked to issue the TDS command as fast and accurately as possible by moving the tongue from resting position to issue the randomly selected command. They had to do this within a time interval of T when the center light was still green, during which the blue bar for the issued command was filled and it was registered. Three values of T = 2, 1.5, and 1 s, each including 20 random commands, were selected per round.

Fig 4.

GUI screen for timed randomly selected commands task from which the TDS ITR can be derived: (a) indicating the random command and being ready, (b) selecting the tongue command and staying there until the blue bar is filled before returning back to neutral.

From the random commands, we calculated two TDS performance indices: percentage of correctly completed commands (CCC%) and information transfer rate (ITR). ITR indicates the rate of information that can be transferred from the user to a computer, and can be calculated from [8],

$$\text{ITR}=\frac{1}{T}\left({log}_{2}N+P{log}_{2}P+(1-P){log}_2\frac{1-P}{N-1}\right)$$ (1)

where N = 6 (for TDS) is the number of simultaneously available commands, P is the ratio of CCC% based on 20 random commands in each round, and T is the system response time.

3) Tongue Rapid Movements

There is no consensus among speech-language pathologists and speech rehabilitation researchers as how to measure the tongue fatigue. The Iowa Oral Performance Instrument indicates the tongue and lips strength, by measuring the amount of pressure that patients can apply to a rubber balloon in their mouth [40]. However, TDS only requires tongue motion as opposed to tongue pressure. Thus, we came up with a simple task to measure tongue speed and range of motion, which we hypothesize that might be a more relevant measure of tongue fatigue with respect to using the TDS for a few hours. In this task, subjects were asked to protrude their tongue and move it horizontally from side to side as quickly as possible for 15 s. This task was conducted before and after the CA part of each session. The SSP algorithm calculated the difference between magnitudes of the left and right magnetic-field vectors from the three-axial sensor modules. Movement rate (MR) was defined as the frequency at which the fast Fourier transform (FFT) of the resulting signal peaked, and movement amplitude (MA) was the FFT peak value. Movement time (MT) was defined as the time interval between two successive peaks in the signal, i.e., the time it took for the tongue to move from one lip corner to the other. Movement variation (MV) was also defined as the coefficient of variation (CV) of all MTs. Fig. 5 shows a typical tongue rapid motion waveform and its corresponding FFT. Amplitude values are based on the sensor outputs in microtesla.

Fig 5.

(a) Typical tongue rapid movement waveform and (b) its spectrum derived by FFT. Units are in microtesla because the amplitudes are directly derived from magnetic sensor outputs.

4) PWC Navigation Tasks

PWC part in each session was conducted after the CA part to make sure subjects had gained enough experience with the TDS to transition from a stationary to a mobile platform, particularly in the earlier sessions. We used a custom-designed interface [see Fig. 6(a)] and GUI to operate a Q6000 (Pride Mobility Inc., Exeter, PA) PWC with the TDS through a universal PWC controller, called Q-Logic, that had two state vectors, one for linear movements and the other for rotations [41]. Absolute values and polarities of these two state vectors determined the linear speed and rotation of the PWC.

Fig 6.

(a) TDS-PWC interface attached to a Pride Q6000 enhanced display [also see Fig. 1(e)]. (b) Experimental setup for the PWC driving part of each session. An operator walked behind the subject with an emergency stop button in hand as a safety measure. (c) Plan of the obstacle course showing its dimensions, obstacle locations, and driving trajectory.

Three PWC control strategies were evaluated in this study. Unlatched and latched strategies used four TDS directional commands to modify the state vectors: FD and BD modified the linear, while TL and TR modified the rotation vector. For the third strategy, called semiproportional, the linear vector was controlled similar to unlatched, and the vector sum of the left and right three-axial sensor modules controlled the rotation vector. Each command increased or decreased its state vector at a certain rate until a predefined limit level was reached. Returning the tongue to its resting position either returned the state vectors back to zero (Unlat.), causing the PWC to stop, or kept it at the level reached by the last command (Latch and Semi). State vectors were sent to the interface circuit via a laptop USB port, which converted them to voltages in 4.8–7.2 V range. These voltage levels were then applied to the Q-Logic controller via its standard DB-9 connector.

PWC task was driving through a ~50-m obstacle course that had six turns and 24 obstacles [see Fig. 6(c)]. Subjects were asked to drive as fast as possible without hitting the obstacles or driving outside the track, either one of which was counted as a navigation error (NE). Driving through obstacle course required using all TDS commands and included making a U-turn, backing up, and fine tuning the direction in a loop. Subjects were also required to make an emergency stop as soon as they heard a randomly timed alarm, which was played once per round while the PWC was moving at its maximum speed. A laptop was placed on a tray in front of the subjects with its lid open in the practice round to provide visual feedback on the issued commands, but the lid was closed in the main three rounds to allow subjects to have a better field of view. The operator walked behind the PWC, while holding an emergency stop button as a safety measure, and recorded the completion time (CT) and NE parameters [see Fig. 6(b)].

The order of the three PWC driving strategies was randomized in each session for each subject: 1) Unlatched: The PWC motion continued only as long as a TDS command was being issued [38]. In the first session, subjects were recommended to stop the PWC prior to 90° or U-turns in order to have more control in making a sharp turn. To stop the PWC, subjects simply returned the tongue to its resting position. 2) Latched: Linear motion continued at the same speed even when subjects returned the tongue to its resting position. There were 5 speed levels in this strategy: backward (B, −0.8 km/h), neutral (N, 0 km/h), forward-1 (F1, 0.8 km/h), forward-2 (F2, 1.13 km/h), and forward-3 (F3, 1.45 km/h). Issuing the FD or BD commands, increased or decreased the speed by one level, respectively. PWC rotation was similar to the unlatched mode. Regardless of the speed level, continuously issuing BD command for 1 s brought the PWC to a standstill. 3) Semiproportional: PWC speed in TL/TR was proportional to the tongue proximity to the left or right corners of the lips when subjects slipped the tip of the tongue over their lips to left or right. Linear motion was controlled similar to the latched mode except for tapping the left or right chicks with the tip of the tongue to change the speed level. To stop, subjects could either lower the speed to neutral (N) or quickly tap the right-chick twice.

5) Questionnaire

To systematically record the subjective aspects of the TDS, we asked subjects to fill out a short questionnaire with eight questions related to CA, PWC, and fatigue at the end of the first four sessions (see Fig. 12), and an elaborate questionnaire with 46 questions at the end of the last session. The long questionnaire covered all ISO-9241-9 functionality assessment questions based on a five-point Lickert scale as well as a few modified questions about the level of tongue fatigue [36]. It also included several questions derived from the QUEST, a popular assessment tool to measure the level of user satisfaction with a specific AT [42].

Fig 12.

Results of the short questionnaire asked at the end of each session for subjective evaluation of the TDS.

B. TDS Training Quality

Good TDS performance can be achieved when subjects consistently position the tip of the tongue or the upper ball of the magnetic tongue stud at a certain position in their mouth over ten random repetitions of each of the six TDS tongue commands during training. Optimal performance is also dependent on whether the command clusters are far enough when they are mapped onto the PCA space, resulting in minimum SSP errors [39]. In order to assess the subjects’ quality of the TDS training over five sessions, we employed two figures of merit (FoM), which were originally used as criteria functions for clustering [43]:

$${\text{FoM}}_1=\text{trace}(S_W^{-1}{S}_B)=\sum _{i=1}^d{\lambda }_i$$ (2)

$${\text{FoM}}_2=-10{log}_{10}\left(\frac{\mid S_W\mid }{\mid S_T\mid }\right)=-10{log}_{10}\left(\prod _{i=1}^d\frac{1}{1+{\lambda }_i}\right)$$ (3)

where between-cluster scatter matrix S_B, within-cluster scatter matrix S_W, total scatter matrix S_T, and their associated parameters are defined in Table I. n is the total number of points, c is the number of clusters, n_i is the number of points in each cluster, D is the set of all points, and D_i is the set of points that belong to the ith cluster. d is the dimensionality of the points (d = 3 in this case) and λ_i is the ith eigenvalue of the matrix S_W⁻¹ S_B, which does not change under nonsingular linear transformations of the data. To improve classification, we would like |S_W | to decrease and |S_B | to increase, resulting in further compaction and separation of the command clusters, respectively.

TABLE I.

Components Involved in Calculation of the Training FoM

Mean vector for the ith cluster	$m_i=\frac{1}{n_i}{\displaystyle \sum _{x\in D_i}}x$
Total mean vector	$m_i=\frac{1}{n}{\displaystyle \sum _D}x$
Scatter matrix for the ith cluster	$S_i=\frac{1}{n_i}{\displaystyle \sum _{x\in D_i}}(x-m_i){(x-m_i)}^t$
Within-cluster scatter matrix	$S_W={\displaystyle \sum _{i=1}^c}{S}_i$
Between-cluster scatter matrix	$S_B={\displaystyle \sum _{i=1}^c}{n}_i(m_i-m){(m_i-m)}^t$
Total scatter matrix	ST = SW + SB

C. Human Subjects and Protocol

The necessary approval was obtained from the institutional review board (IRB) of the Georgia Institute of Technology. Nine able-bodied subjects, four male and five female, with the age of 19–28 years old completed this trial out of 14 who were initially recruited. Subjects had no previous experience with the TDS and had tongue piercing in the midline of their tongue between the tip and frenulum for more than three months. Subjects’ trial sessions were scheduled on a certain day of the week ±2 days. They were allowed to cancel their appointment for no more than two nonconsecutive sessions or they would have been considered dropped out.

At the beginning of each session, subjects conducted TDS calibration and pretraining [28]. To facilitate learning, they trained the TDS in four steps in the CA part from easy (two commands) to relatively more complex (six commands). Maze navigation and timed random command tasks were conducted immediately after four and six command trainings, respectively. On average, the CA and PWC parts took 5 and 1.5 h for the first session, which included detailed explanation of the tasks, and reduced to 2.5 h and 45 min for the following sessions, respectively. In the first session, subjects replaced their own tongue studs with our cold-sterilized magnetic tongue studs, which they wore throughout the 5–8 week duration of the trial.

D. Data Analysis

To measure the TDS learning effect for this group of subjects, parameters such as the initial performance level, rate of improvement, and whether the subjects’ performance reached a plateau during five sessions were considered. In addition, by contrasting the TDS performance in the fifth session, when subjects had gained maximum experience, with that of keypad, a discrete input device that subjects were quite familiar with, we compared the tongue-TDS performance versus finger-keypad in the maze navigation task. Maze navigation task was a 5 × 2 within-subject factorial design with two factors of session (five levels) and device (two levels). We used the Helmert contrast to find nonsignificance between performances of each session with the remaining sessions using one-way repeated measures analysis of variance (RM-ANOVA) with only TDS data, assuming that the tongue performance is independent of the index finger [44]. Random command task was also analyzed in the same fashion. The PWC part was a 5 × 3 within-subject factorial design with two factors of session (five levels) and strategy (three levels). Tongue rapid movement task was a 5 × 2 within-subject factorial design with two factors of session (five levels) and order (two levels), indicating whether the task was done before or after the CA part.

IV. Results

Table II summarizes the statistics of all tasks, including performance measures for the TDS first session, the first plateau session, and the fifth session. Typical paths of a subject navigating the cursor with TDS through one of the maze designs in the first and fifth sessions are shown in Fig. 7(a) and (b), respectively, where improvements in speed (TCT) and accuracy (SoD) are evident. Fig. 7(c) and (d) shows the maze navigation results throughout five sessions, where both TCT and SoD showed significant improvements from the first to the second session, indicating that a considerable amount of learning occurs early on, during the first session. Although the session effect became statistically nonsignificant from the second session on for both measures, it is evident from Table II and Fig. 7 that as sessions went by, the average performance and variability kept improving. Keypad TCT and SoD were both lower than that of TDS (p = 0.009 for the fifth session).

TABLE II.

Summary of the Experimental Result

Task	Performance measure		1st session	1st plateau (session #)	5th session
Maze Nav.		TCT(s)	23.3 ± 5.4	16.5 ± 4.5 (2)	14.6 ± 2.8
		SoD	12.7 ± 5.8	4.4 ± 3.7 (2)	1.9 ± 1.3
Rand. Comm.	2 (s)	CCC%	99.2 ± 1.3	99.2 ± 1.3 (1)	98.6 ± 2.3
		ITR	75.7 ± 3.1	75.7 ± 3.1 (1)	74.8 ± 4.1
	1.5 (s)	CCC%	95.5 ± 6.8	95.5 ± 6.8 (1)	99.4 ± 0.9
		ITR	93.2 ± 14.9	93.2 ± 14.9 (1)	101.5 ± 2.8
	1 (s)	CCC%	94.7 ± 3.4	94.7± 3.4 (1)	97.5 ± 3.8
		ITR	132.8 ± 13.9	132.8 ± 13.9 (1)	144.3 ± 15.2
PWC Drive	CT (s)	Unlat.	198.2 ± 34.0	169.2 ± 25.1 (2)	160.1 ± 18.2
		Latch.	230.3 ± 48.4	192.5 ± 27.5 (2)	179.1 ± 23.2
		Semi.	196.5 ± 16.5	178.5 ± 21.1 (2)	175.3 ± 24.6
	NE	Unlat.	2.7 ± 1.9	1.7 ± 1.3 (2)	0.8 ± 0.7
		Latch.	3.5 ± 2.1	1.8 ± 1.1 (2)	1.0 ± 1.4
		Semi.	2.1 ± 1.2	1.0 ± 0.9 (2)	0.9 ± 1.0
TDS Training	FoM1	4	82.8 ± 42.2	133.5 ± 61.3 (2)	148.1 ± 75.6
		6	61.7 ± 30.8	93.0 ± 59.4 (2)	108.3 ± 62.5
	FoM2	4	36.9 ± 6.6	41.7 ± 7.9 (2)	41.9 ± 7.2
		6	34.8 ± 7.2	38.2 ± 6.6 (2)	41.0± 7.1
Rapid Tongue Motion	MR (Hz)	Bf	1.8 ± 0.4		2.6 ± 0.7
		Af	2.1 ± 0.6		2.9 ± 0.8
	MV	Bf	0.1 ± 0.04	0.08 ± 0.03 (4)	0.07 ± 0.03
		Af	0.1 ± 0.03	0.07 ± 0.03 (4)	0.06 ± 0.02
	MA (μT)	Bf	13.75 ± 8.0	12.26 ± 7.2 (2)	9.77 ± 5.3
		Af	9.59 ± 6.8	9.42 ± 6.9 (2)	8.34 ± 5.6

Fig 7.

Cursor path of a subject navigating through one of the maze designs with TDS in the (a) first session (SoD = 8.54, TCT = 18.8 s) and (b) fifth session (SoD = 1.33, TCT = 13.3 s) (c) TCT and (d) SoD for all subjects in the maze navigation task.

Results of the timed random command selection task are shown in Fig. 8. Although RM-ANOVA found no significance in the CCC% and ITR throughout the five sessions for any of the time intervals, it is evident from these graphs that both measures improved throughout five sessions, particularly for the smaller intervals. Being close to maximum CCC% in the first session of 2-s time interval and the slight drop in the fourth session due to random variations suggest that perhaps this time interval was not challenging enough for the subjects, and their performances were almost saturated from early on.

Fig 8.

(a) Percentage of CCC% and (b) ITR using TDS in timed randomly selected commands task.

Fig. 9 shows the PWC results throughout five sessions, where all performance measures showed significant improvement from the first to the second session (we had an outlier in this task and the results are based on eight subjects). Similar to the maze navigation, there was no statistical significance between the second and remaining sessions for CT and NE, while averages were reduced and performances were improved. Fig. 9 also shows the minimum task CT range for each of the three strategies with a perfect performance, considering the PWC speed limit. The upper and lower bounds of this range correspond to navigation with and without stopping for 90° turns, respectively. It is interesting to note that by the second session, unlatched, and semiproportional strategies fell within perfect performance range (PPR), while with latched strategy subjects entered this range in the fourth session. Being in this range implies that CT was mainly limited by the top speed (1.45 km/h, 17°/s) rather than the subjects’ lack of control. Pairwise comparison with the Bonferroni adjustment applied to the CT of the last session showed that semiproportional and latched strategies were not significantly different (p = 0.333), however, unlatched was significantly superior to both of them (p = 0.038 between both corresponding pairs). The effect of strategy was not significant on the NE (F(2,14) = 1.19, p = 0.334).

Fig 9.

(a) Obstacle course PWC CT and the PPR for various PWC control strategies. (b) Sum of the two types of NEs (collisions and out-of-tracks).

FoM₁ and FoM₂ for TDS training of four and six commands in Table II indicate significant improvements from the first to the second session. Although session effect becomes nonsignificant from the second session, suggesting an early plateau, as sessions went by, the FoM averages increased and quality of training improved. It was also noted that the variances increased slightly because a few subjects became very good at training the TDS, while others did not improve as much. We observed that both clustering criteria, which were employed as FoMs, show similar trends, suggesting that each of them might be sufficient for assessing the quality of the TDS training. Fig. 10(a) and (b) shows a sample PCA space for four-command TDS training in the first and fifth sessions, respectively. The quality of training has been clearly improved for this subject by the command clusters becoming denser and more spaced, signifying more consistent and distinguishable tongue positioning while issuing TDS commands.

Fig 10.

(a) Sample of PCA space for four-command training in the first session with FOM₁ = 48.3 and FOM₂ = 32.1. (b) PCA space for the same subject in the fifth session with FOM₁ = 681.7 and FOM₂ = 67.1.

Fig. 11 shows the result of tongue rapid movement task. MR has increased while MA has decreased after several hours of using TDS for computer access. The same trend can be observed throughout five sessions. In addition, MV has decreased throughout the duration of experiment and over five sessions. Since these parameters have changed in opposite directions and were not significantly different before and after TDS usage, we cannot attribute these changes to fatigue. Our hypothesis is that over time subjects have learned how to perform this task more efficiently.

Fig 11.

(a) MR in Hertz. (b) CVs of movement times (MV). (c) Maximum amplitude of the signal FFT (MA).

A. Short Questionnaire

Fig. 12 shows the questions and average subjects’ ratings of eight items in the short questionnaire given at the end of each session. In all of these questions, the higher score is the better, except for Q5 and Q7, in which the middle score is the best. This chart clearly shows improvement in the TDS subjective scores as the sessions proceed. The only scores that have slightly degraded (Q7 and Q8) are related to the speed and smoothness of the PWC motion, which suggest that as the subjects became more skilled in navigating the PWC using TDS, they expected a faster and smoother PWC ride.

Fig. 13 shows the preferred PWC driving strategy over five sessions. Popularity of the unlatched strategy increased in the second session and remained constant afterward, while the latched strategy was popular at the beginning and at the end. Semiproportional was popular in the first four sessions, but dropped in the last one. It can be seen that the qualitative information that we collected on the subjects’ preferred PWC driving strategy was inconclusive.

Fig 13.

Preferred PWC driving strategy throughout the five sessions.

B. Final Questionnaire

All subjects thought that the setup and calibration of the TDS was very easy. Four of them thought the procedure to train the TDS to recognize tongue commands was easy, three thought it was fairly easy, and two thought it was neither difficult nor easy. Three subjects thought that TDS was very effective for computer access, four thought it was somewhat effective, and two of them thought it was neither effective nor ineffective. Seven subjects thought the TDS was very effective for driving the PWC, while one of them thought it was somewhat effective (one outlier was omitted from PWC). All subjects thought that while driving the PWC using TDS they felt very safe. Three of them thought they were completely in control, while five thought that they were somewhat in control. Six subjects felt that TDS became easier to use during the second session and the remaining three felt it became easy during the third session. Six subjects felt that they could use the TDS really well by the end of the third session, two felt that it was by the end of the fourth session, and one felt that it was by the end of the fifth session. Three subjects thought that, in general, TDS was very easy to use, while four of them thought it was somewhat easy and two thought it was ok.

V. Discussion

The main purpose of this study was to observe the TDS learning process, including the subjects’ initial performances, improvement rates, and overall achievements through five sessions. Moreover, maze navigation was performed with both index finger-keypad and tongue-TDS for benchmarking the TDS against a computer input device that able-bodied subjects used on a daily basis.

In the random commands task, the best ITR in the first session was 132.9 ± 13.9 bits/min with CCC = 94.7% and T = 1 s, which was slightly better than our earlier results [34]. Although statistically insignificant, the effect of session was to improve the average ITR and reduce its variability among subjects. Maximum achieved ITR was 150.3 ± 11.5 bits/min with CCC = 98.8% and T = 1 s, which occurred in the fourth session and was better than most evaluated devices [34]. The reason for using three different time intervals was to explore the drop in accuracy as T decreased, and find the maximum ITR corresponding to an acceptable CCC% to be regarded as the ITR for TDS [7], [8]. Since even with the shortest T = 1 s subjects were easily able to achieve CCC% ≥95%, and considering the agility and dexterity of the tongue motion along with the 50-Hz sampling rate of the current TDS, we hypothesize that the ITR of TDS could be significantly higher if the subjects were challenged with smaller time intervals.

By the second PWC session, unlatched and semiproportional CTs fell within the PPR [see Fig. 9(a)], indicating that the PWC speed was a limiting factor. The low PWC speed was also reflected in response to related questions and should be increased in future studies without compromising safety. These are promising results as they show the ease of use and confidence of subjects in navigating the PWC using TDS. PWC driving strategies did not differ much in terms of NE with an average of less than one event per round [see Fig. 9(b)], which was very close to the minimum expected level.

The rapid tongue movement task that we included before and after CA part of each session did not lead to any quantifiable measure of tongue fatigue (see Fig. 11). However, the questionnaire did capture data on the perceived levels of fatigue. Subjects reported low levels of perceived tongue (and jaw) fatigue particularly after the second session (see Q2 and Q3 in Fig. 12). These results are consistent with properties of the tongue muscle fibers, which are known to have a low rate of perceived exertion, particularly when the tongue moves freely without applying any pressure [29]. In order to reach a more accurate measure of tongue fatigue, our results suggest that both speed and accuracy should be incorporated in this task. For instance, instead of leaving the speed and range of motion to the subject, in our future studies, we plan to restrict the MA by specifying the range of motion between two landmarks such as the lip corners or two target bars on the computer screen, and ask subjects to focus on the speed of tongue movement (MR) while maintaining the range. Another possibility is to ask subjects to track a waveform or moving target with their tongues as accurately as possible.

One of the limitations of this study was recruiting able-bodied subjects with tongue piercing as opposed to the potential TDS end users. We believe that this limitation has a low impact on the quantitative aspects of our study because most physical disabilities, such as high-level spinal cord injuries (SCI), have little effect on the tongue motion or cognitive abilities of the individual, which were the basic requirements to accomplish our tasks. However, we expect the qualitative results from the questionnaires, particularly the long questionnaire used at the end of the trial, which involved usability and efficacy aspects of this new AT, to be quite different among able-bodied subjects, who might view their participation in this trial as being altruistic or entertaining, and those who deal with the realities of a severe physical disability on a daily basis. Therefore, we presented the results of the final questionnaire in a very brief format.

Another limitation of this trial was our inability to maintain a constant interval between every two consecutive sessions, which was imposed by the availability of our participants, the majority of whom were college students. The experimental procedure was planned for one week intervals, while there were cases in which the time between two consecutive trials was twice as long due to a cancelation. We did not notice degradation in the subjects’ skills following longer intervals. However, measuring the longevity of the learning effects and the role of memory are out of the scope of this study. Yet another limitation was the relatively narrow age span (19–28 years old) of our subjects compared to the potential end user population, which reflects the fact that body piercing is often practiced by the adolescence and younger adults. It is, however, worth noting that 55% of the individuals with SCI are between 16 and 30 years old, who currently need lifelong special care, and can immensely benefit from becoming more independent with modern ATs such as the TDS [45].

VI. Conclusion

We have evaluated the TDS performance in maze navigation, issuing random commands, and driving a powered wheelchair through five consecutive sessions to study the learning effect with able-bodied subjects who already had tongue piercing and wore our magnetic tongue studs during the 5–8-week period of this trial. We also explored the qualitative aspects of using the TDS with a short questionnaire asked at the end of each session and a long one asked at end of the trial. We quantitatively measured the level of tongue fatigue as the result of using TDS for a few hours and also quantified the quality of the TDS training during five sessions. All performance measures experienced significant improvements and some plateaued at an early stage, suggesting a rapid and easy learning process. The quantitative and qualitative results suggest that TDS is effective for computer access and PWC navigation. It is also easy to learn with low cognitive burden and physical fatigue. Further evaluation is, however, necessary by potential end users.

Biographies

Behnaz Yousefi (S’09) received the B.S. degree from the Khajeh Nasir University of Technology, Tehran, Iran, and the M.S. degree from the Sharif University of Technology, Tehran, Iran, both in electrical engineering, in 2004 and 2006, respectively. She is currently working toward the Ph.D. degree at the W.H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA.

She worked as an RF-Microwave Engineer for three years before joining the Ph.D. program at the Georgia Institute of Technology.

Xueliang Huo (S’07–M’12) was born in 1981. He received the B.S. and M.S. degrees in mechanical engineering from Tsinghua University, Beijing, China, in 2002 and 2005, respectively, and the Ph.D. degree in electrical and computer engineering from the Georgia Institute of Technology, Atlanta, in 2011. His Ph.D. research was on developing wearable and wireless assistive technologies as well as human computer interfaces for people with severe disabilities.

He is currently a Hardware Engineer at Microsoft Corporation, Redmond, WA.

Jeonghee Kim (S’11) was born in 1983. She received the B.S. degrees in electrical engineering from the Kyungpook National University, Sangju, Gyeongbuk, Korea, and the University of Texas at Dallas, Richardson, in 2007 and 2008, respectively, and the M.S. degree in electrical and computer engineering from the University of Michigan, Ann Arbor, in 2009. She is currently working toward the Ph.D. degree from the Georgia Institute of Technology, Atlanta, in GT-Bionics Lab.

Her research interests are system design for biomedical devices with embedded mobile application, human computer interaction, and assistive technologies.

Emir Veledar was born in 1954. He received the B.S. degree in economics from the University “Dzemal Bijedic” in Mostar, Bosnia and Herzegovina, in 1976, the M.Sc. degree in statistics from the Institute of Economic Sciences in Belgrade, Belgrade, Serbia, in 1985, and the Ph.D. degree in statistics from the University “Dzemal Bijedic” in Mostar in 1990. His Ph.D. research was on applying fuzzy logic to clustering.

From 1990 to 1993, he was an Assistant Professor at School of Economics at Mostar and the University of Sarajevo. From 1993 to 1994, he was a Visiting Research at Tilburg University, The Netherlands. From 1994 to 1998, he was an Instructor of Mathematics at the University of Georgia, Athens. In 1998, he was an Assistant professor of Mathematics at the “James Madison” University in Harrisonburg, VA. Since 1999, he has been with the Cardiology Division, Emory University School of Medicine, Atlanta, GA. He has contributed to more than 200 conference and journal publications, almost all of them in the field of medical outcomes and its consequences.

Maysam Ghovanloo (S’00–M’04–SM’10) was born in 1973 in Tehran, Iran. He received the B.S. degree in electrical engineering from the University of Tehran, Tehran, in 1994, the M.S. degree in biomedical engineering from the Amirkabir University of Technology, Tehran, in 1997, and the M.S. and Ph.D. degrees in electrical engineering from the University of Michigan, Ann Arbor, in 2003 and 2004, respectively.

From 2004 to 2007, he was an Assistant Professor in the Department of Electrical and Computer Engineering, North Carolina State University, Raleigh. Since June 2007, he has been an Associate Professor with the Faculty of Georgia Institute of Technology, Atlanta, and the Founding Director of the GT-Bionics Laboratory in the School of Electrical and Computer Engineering. He has authored or coauthored more than 100 peer-reviewed conferences and journal publications.

Dr. Ghovanloo is an Associate Editor of the IEEE Transactions on Biomedical Engineering, the IEEE Transactions on Biomedical Circuits and Systems, and a member of the Imagers, MEMS, Medical, and Displays subcommittee at the International Solid-State Circuits Conference(ISSCC). He received the 2010 CAREER award from the National Science Foundation. He has also received awards in the 40th and 41st Design Automation Conference/ISSCC Student Design Contest in 2003 and 2004, respectively. He has organized several special sessions and was a member of the Technical Review Committees for major conferences in the areas of circuits, systems, sensors, and biomedical engineering. He is a member of the Tau Beta Pi, the American Association for the Advancement of Science, the Sigma Xi, the IEEE Solid-State Circuits Society, the IEEE Circuits and Systems Society, and the IEEE Engineering in Medicine and Biology Society.