A Body Machine Interface based on Inertial Sensors

Abstract

Spinal cord injury (SCI) survivors generally retain residual motor and sensory functions, which provide them with the means to control assistive devices. A body-machine interface (BMI) establishes a mapping from these residual body movements to control commands for an external device. In this study, we designed a BMI to smooth the way for operating computers, powered wheelchairs and other assistive technologies after cervical spinal cord injuries. The interface design included a comprehensive training paradigm with a range of diverse functional activities to enhance motor learning and retention. Two groups of SCI survivors and healthy control subjects participated in the study. The results indicate the effectiveness of the developed system as an alternative pathway for individuals with motor disabilities to control assistive devices while engaging in functional motor activity.

I. Introduction

Spinal cord injury affects 12000 people every year in United States with the majority of these injuries occurring at the cervical level [1]. However, even in individuals with high-level injuries, some residual motor and sensory capacities continue to exist. These residual body movements can provide signals of different types to operate external devices. Hence, a body machine interface is a motor learning paradigm in which users reorganize their voluntary movements to accomplish new functional tasks. Controlling a prosthetic arm with EMG signals [2], driving a wheelchair using tongue motions [3], or compensatory strategies in stroke survivors [4] are a few examples of reorganization of motor coordination patterns.

Our approach is based on constructing a high dimensional signal space from the vocabulary of movements that paralyzed users can still comfortably generate, and mapping this space onto a lower- dimensional space of control variables. We consider the abundance of degrees of freedom to be a natural and fundamental resource in motor learning. This kinematic redundancy presents a distinct opportunity to identify and coordinate a convenient subset of degrees of freedom to achieve task objectives with a flexible and adaptable motor behavior [5].

The present study builds upon our previous BMI design that was based on tracking active or passive markers by infrared cameras [6, 7]. The new design is based instead on inertial measurement units (IMUs), thus suppressing all influence of variable ambient light conditions and reducing the equipment bulk of the system.

II. Methods

Here, we outline the hardware and software components that provide the requirements for the implementation of an IMU-based body machine interface.

A. Hardware

Four MTx (Xsense Technologies B.V., Netherlands) motion trackers were used to capture scapular retraction, protraction, elevation and depression. The orientation of each sensor was computed by a sensor-fusion algorithm through the combination of the output of 3-degree of freedom embedded accelerometers, gyroscopes and magnetometers. Orientation can be expressed in various formats, such as quaternions or rotation matrices. We used the Euler representation. In the Euler formalism by Xsens, the elemental rotations occur about the axis of a coordinate system that is aligned with an earth-fixed Cartesian frame and therefore is equivalent to ‘roll, pitch and yaw’ angles, which are rotations around global X, Y and Z-axis respectively.

In IMU design, the earth magnetic field is used to stabilize the heading angle (yaw). Therefore, the heading angle is unreliable when the earth’s magnetic field is disturbed. This happens when ferromagnetic materials come within proximity of the sensor or when the sensor is exposed to magnetic fields other than the earth magnetic field. For this reason, we discarded the heading angle from the orientation vector and used roll and pitch as input signals for the interface.

To track the upper body motions we mounted the sensors on an adjustable size vest with Velcro patches on the shoulder area.

B. Software

The signals of the IMUs were captured at the rate of 50 samples per second. The total signal space was an 8D vector of coordinates that defined the “body space”. The control space was defined by the two coordinates (x, y) of the cursor on the monitor, which was a Euclidean domain. The first step was to define a body-to-cursor map. Transformations from a high dimensional motor space to a 2D control space are inherently redundant: each body configuration corresponds to a unique point on the screen, while each screen location corresponds to a subspace of “equivalent” body postures. A simple way to construct a Euclidean space from body motions is by principal component analysis (PCA)[8]. This is a standard technique to transform a multidimensional signal to an orthogonal coordinate system, whose axes are ordered by decreasing amount of variance in the corresponding direction. A calibration phase was required to find the transformation matrix. To this end, we asked the participants to perform free upper body movements for 1 min, an activity that we called “the body dance”. The purpose of the dance was to evaluate how the subject distributed motor variance over the signal space. The task space was then defined by the first two principal eigenvectors of the calibration data [6]:

$$\mathrm{p}=\mathrm{Ah}+{\mathrm{p}}_0$$ (1)

Where p_[2×1] represents the coordinates of the cursor, h_[8×1] is the body vector generated by the IMUs, and A_[2×8] is the transformation matrix, whose rows are formed by the first two eigenvectors extracted by PCA. By definition, the first PC accounts for a larger variance of the data set than the second. Therefore, a gain adjustment was required to obtain the same amount of task space motion on each axis. For that reason, the gains associated with each principal direction were normalized by the standard deviation of the projected data along the corresponding axis. The components of the A matrix were then scaled to insure that every point in the task space could be comfortably reached. p₀ is an offset vector that sets the origin to be the centroid of the task space representation of the calibration data.

Although the two-dimensional subspace that is formed by the first two PCs captured a large fraction of the total variance, it did not necessarily reflect the natural up-down/left-right orientation of the display monitor. In fact, it was very likely for the angle between the two coordinate frames not to be zero. Therefore, immediately after the calibration, there was a customization phase in which users were allowed to set the origin and the orientation of the task space based on their preference.

Small involuntary movements of the shoulders, associated for example with breathing, caused the cursor to jitter on the screen. We filtered this unwanted jitter by simulating a virtual mass-spring-damper system with tuned parameters to stabilize the cursor position when the subjects were at rest. A block diagram of the mapping from IMU signals to cursor coordinates is shown in Fig. 1

Figure 1.

From IMU signals to cursor coordinates.

C. Interface Modules

The interface was developed to serve as a comprehensive motor training tool, with features that combine the learning process with useful and entertaining tasks. The user is engaged in a variety of tasks, each involving different movement sequences and strategies to fulfill its specific objectives. This approach is supported by recent studies [9] suggesting that the consolidation of learning and long term retention of the acquired skills is achieved to a greater extent by practicing multiple diverse tasks rather than by focusing on a single task. A family of diverse tasks with functionally relevant objectives, such as moving in the environment, navigating the internet and playing dexterity games provides the disabled users with a powerful framework for enhancing the maintenance and reorganization of body movements that were spared by the injury to the spinal cord.

In this first implementation, the interface was comprised of four modules:

C.1 Reaching

Subjects performed a center-out reaching task to eight peripheral targets (1.1 cm radius) placed at equal intervals on an 18-cm diameter imaginary circle. Targets were presented in a block structure, with randomized order within each block. Trials were initiated by moving the cursor (a magnified Windows mouse pointer) to the center target and holding it there for 500 ms. At this time, the target to reach appeared in green color. The task was to acquire the target within 1 second and to hold the cursor inside it for 500 ms. If the reaching time exceeded this given limit, the color of the target changed to red but subjects were still to reach the target and complete the trial.

The reaching task was added to the interface as a paradigm to specifically investigate the evolution of the motor learning using metrics adopted from classic human motor control studies. These metrics are widely adopted to evaluate performance in brain machine interfaces [10].

C.2 Typing

We implemented an on-screen virtual QWERTY keyboard (http://cnt.lakefolks.com/) that allows user to select a key by hovering the pointer over it. The dwelling time to select a key for display was set at about 300 ms.

Although the typing task might be considered as a sequence of reaching movements, it required higher levels of skill and control proficiency compared to the reaching task. The target keys were significantly smaller than the targets presented during the reaching session. Furthermore the requirement to stabilize the cursor over each key required a higher level of attention and posture control, since failure would result either in the inability to generate text or in the production of erroneous text (if the cursor rested over the wrong key) requiring corrective actions.

C.3 Gaming

To promote learning and retention and to make practice sessions pleasant and entertaining, we created a library of flash video games. Subjects engaged with a variety of activities with each one addressing a particular operational skill e.g. reaching (card games), trajectory control and tracking (snake game) and obstacle avoidance (car racing game). We also developed a 2-d virtual Ping-Pong game in Matlab and appended that to the library of games. We expected that playing Ping-Pong would improve the timing accuracy of the movements and besides by designing our own game, we would be able to manipulate the difficulty level of the game and track the performance across sessions.

Subject could make different types of clicks using Dwell clicker (http://www.naturalpoint.com), which is an application that allows user to operate a mouse or other pointing devices without pressing a button.

C.4 Virtual Navigation

The main rationale for the interface was to train SCI survivors to skillfully operate powered wheelchairs while maintaining and enhancing upper-body mobility. Earlier studies have demonstrated that the dexterity acquired while practicing the control of a virtual wheelchair are to some extent retained and generalized to the control of an actual wheelchair [11]. Furthermore, virtual reality allows users to safely experience various driving maneuvers. Thus, participants performed simulated driving tasks in a virtual environment prior to driving the actual wheelchair. The virtual environment was a slightly modified version of the miWe simulator described in [12]. Custom environments were also developed to encourage both free explorations as well as to practice the specific tasks tested using the real wheelchair. During virtual driving, the cursor was constantly present on the screen. The forward speed of the wheelchair was proportional to the displacement of the cursor from the center of the screen in the y direction and the turning speed was proportional to the deviation in the x direction.

D. Participants

Three spinal cord injury survivors: S1 (31 year old male, 9 years post injury at the C5 level), S2 (33 year old male, 15 years post injury at the C2/4 level), S3 (58 year old female, 1 year post injury at the C6 level) and three unimpaired age/gender-matched control subjects participated in this study. S1 participated in our previous optical BMI study [6]. No remaining subjects had any prior experience with BMI. All participants gave their informed, signed consent to participate in this experiment, which was approved by Northwestern University’s Institutional Review Board.

E. Experimental Protocol

All control subjects participated in five one-hour training sessions held across three weeks. The calibration was done in the first session and control subjects continued to use the same map during the remaining sessions. Each session initiated by performing three blocks of reaching movements (24 Targets) followed by a typing task. Subjects were instructed to type a pangram sentence containing every letter of the English alphabet: “The quick brown fox jumps over the lazy dog”. Subsequently, subjects played five pong games, each game lasting for two minutes. Then they were given ten minutes to play a flash game(s) based on their preference. Finally, they completed another set of reaching movements with the same number of trials. In the last two sessions, participants also completed virtual driving in an environment that reproduced a series of tasks features that mirrored those that the participants would need to perform in the actual driving scenarios.

SCI subjects participated in an extended training program with more than twenty sessions (twice a week). Before the official start of training, all subjects took part in a familiarization session, in which they learned about the interface design and performed the calibration followed by one or two blocks of reaching movements. S1 and S3 continued to use the same map during the remaining sessions. S2 asked for a recalibration for the first three sessions. Training sessions initiated with SCI subjects performing two blocks of reaching movements with 24 targets in each block. Other tasks e.g. typing, gaming and virtual driving were gradually added to the protocol until the list of activities for each session for the SCI group became identical with the control group. The protocol between the two groups initially varied to allow more time for the SCI group to become familiar with the experiment.

F. Data Analysis

Orientation data from all four IMU’s and the coordinates of the cursor were recorded at the rate of 50 Hz for offline analysis. Time to acquire the target and the Euclidean distance between the cursor position and the center of target after one second from the target appearance were used to quantify the reaching performance. Records with more than 1.5 interquartile ranges were removed from the reaching data. Characters per minute and hits per minute were the quantitative metrics used to evaluate the typing and the pong skill, respectively. The values obtained for each task from each subject were averaged across all trials and all subjects in each group.

SCI subjects did not accomplish the typing and the pong tasks from the very first session (see experimental protocol). For that reason, we report the analysis of the first five typing and pong executions by SCI subjects. Table.1 indicates the sessions from which the data have been analyzed.

TABLE I.

Data Extraction Sessions

Subject	Session Number
	Reaching	Typing	Pong
S1	1-5	2-6	1,5,7-9
S2	1-5	1-5	3-7
S3	1-5	2-6	1-5

III. Results

All subjects learned to control the cursor by reorganizing their shoulder movements. The performance of each task improved with training. Extended practice led to less variable and more rectilinear cursor movement. Both the SCI and the control subjects continuously reduced the reaching time and the Euclidean endpoint error (Fig.2). The result showed not only retention of the learned skills across multiple sessions but also a significant improvement within a single session. SCI subjects had slightly better performance in the first session that might be attributed to the fact that SCI subjects were acquainted with the interface in a separate session before starting the experiment. Results also showed that control subjects had a faster learning rate than SCI group, however at the end of fifth session there was no apparent difference between the two groups.

Figure 2.

Reaching performance for SCI (blue) and control (red) subjects across training sessions. A. Average time to reach the targets. B. Average Euclidean endpoint error after one second from initiating the movement. Error bars represent standard deviation.

All subjects were able to complete the typing task in each of the five sessions. In the first session, the average typing rate across all subjects was 6.6±2.8 character per minute (CPM). Typing efficiency continuously improved with training reaching to 12.2±4.3 CPM in the fifth session. Typing performance of the SCI subjects improved at the same rate as the control subjects (Fig. 3).

Figure 3.

Average Pong performance for SCI (blue) and control (red) subjects across training sessions. Error bars represent standard deviation.

The pong task specifies not only the position to be reached by the cursor, but also the time at which each position needs to be attained. Increased task performance across sessions (Fig.4) indicates that the subjects leaned how to effectively regulate the timing of their movements.

Figure 4.

Average typing speed for SCI (blue) and control (red) subjects across training sessions. Error bars represent standard deviation.

All subjects were able to drive the virtual wheelchair without difficulties and they successfully completed all the maneuvers in the structured virtual environment. Fig.5 shows the trajectories of a representative SCI subject (S2) in the first and the fourth sessions of virtual driving where he completed a variety of different maneuvers such as driving forward and backward, driving through doorways, parallel parking and obstacle avoidance.

Figure 5.

Top view of a representative SCI subject’s path in the virtual environment performing different maneuvers: diving through doorways and a slalom obstacle course, parallel parking and complete turn.

IV. Discussion

The present study indicates that high level spinal cord injury survivors can efficiency operate assistive devices by functional reorganization of their residual upper-body movement skills. The fact that the performance of SCI participants was comparable to able-bodied subjects across all different tasks suggests that despite the injury, these tetraplegic participants retain a high level of control of their shoulder motions.

Most importantly, operating a BMI engages the users in a new goal-directed and sustained physical form of activity that has a potential to prevent muscle atrophy and to maintain mobility. In this way, the approach has the potential to combine within a single framework both the assistive and the rehabilitation goals.