Remapping residual coordination for controlling assistive devices and recovering motor functions

Abstract

The concept of human motor redundancy attracted much attention since the early studies of motor control, as it highlights the ability of the motor system to generate a great variety of movements to achieve any single well-defined goal. The abundance of degrees of freedom in the human body may be a fundamental resource in the learning and remapping problems that are encountered in human–machine interfaces (HMIs) developments. The HMI can act at different levels decoding brain signals or body signals to control an external device. The transformation from neural signals to device commands is the core of research on brain-machine interfaces (BMIs). However, while BMIs bypass completely the final path of the motor system, body-machine interfaces (BoMIs) take advantage of motor skills that are still available to the user and have the potential to enhance these skills through their consistent use. BoMIs empower people with severe motor disabilities with the possibility to control external devices, and they concurrently offer the opportunity to focus on achieving rehabilitative goals. In this study we describe a theoretical paradigm for the use of a BoMI in rehabilitation. The proposed BoMI remaps the user’s residual upper body mobility to the two coordinates of a cursor on a computer screen. This mapping is obtained by principal component analysis (PCA). We hypothesize that the BoMI can be specifically programmed to engage the users in functional exercises aimed at partial recovery of motor skills, while simultaneously controlling the cursor and carrying out functional tasks, e.g. playing games. Specifically, PCA allows us to select not only the subspace that is most comfortable for the user to act upon, but also the degrees of freedom and coordination patterns that the user has more difficulty engaging. In this article, we describe a family of map modifications that can be made to change the motor behavior of the user. Depending on the characteristics of the impairment of each high-level spinal cord injury (SCI) survivor, we can make modifications to restore a higher level of symmetric mobility (left versus right), or to increase the strength and range of motion of the upper body that was spared by the injury. Results showed that this approach restored symmetry between left and right side of the body, with an increase of mobility and strength of all the degrees of freedom in the participants involved in the control of the interface. This is a proof of concept that our BoMI may be used concurrently to control assistive devices and reach specific rehabilitative goals. Engaging the users in functional and entertaining tasks while practicing the interface and changing the map in the proposed ways is a novel approach to rehabilitation treatments facilitated by portable and low-cost technologies.

1. INTRODUCTION

In the past three decades, there has been rapid progress in the development of human-machine interfaces. These interfaces can extend or replace human capabilities by facilitating the control and operation of assistive and prosthetic devices. HMIs come in different forms, acting at different levels of the sensory-motor system. Sensory interfaces can transform sounds into cochlear stimuli [1], images into stimuli to the visual cortex [2, 3], or stimulate the somatosensory cortex to generate an artificial proprioceptive sensation [4–6]. Motor interfaces may transform electromyographic (EMG) signals into commands for a prosthetic limb [7], electroencephalographic (EEG) signals into characters on a computer screen, multiunit recordings from cortical areas into a moving cursor [8, 9], or upper body movements into commands for a wheelchair [10–12]. HMIs implement novel transformations between internal neural representations and the external physical world. In a sensory interface, external states are transformed into stimuli, while in a motor interface, motor intentions are translated into commands and actions upon the environment. Since the turn of the millennium, decoding motor intention from neural recordings has been a major research goal for Brain-Machine interfaces [8, 13, 14]. In this article, we focus on the same goal in a different technological, clinical and scientific setting: the "Body-Machine Interface". Instead of decoding spike trains, BoMIs decode patterns of signals derived non-invasively from body motions. The problem of forming a functional map between neural or body signals and external environment is challenging. While some of the computational challenges of the BoMIs are similar to those of the BMIs, there are at least two most notable differences.

First, the BoMI exploits the natural kinematic redundancy of the sensory motor apparatus [15]. In contrast, brain based interfaces rely on signal redundancy of the recorded neural activities. While signal redundancy is typically larger than kinematic redundancy, the user of a BMI does not have a natural preexisting ability to control motion variables by modulating the firing rates of the specific neurons in the vicinity of the recording array. Nor there exist a natural pre-existing sensory feedback of the neurons' activities. Instead, the BoMI is based on the available skill to generate and sense the motions of body elements whose control has survived the injury. The interface allows us to identify the residual abilities of each subject and for detecting the embedded subspace that the users remain able to control with more ease and precision. Indeed, the residual mobility following injuries to the spinal cord, stroke and other motor impairments, even in the most severe cases, typically exceeds the number of variables that one needs to specify for controlling devices such as powered wheelchairs and computers.

The second, and perhaps most important difference between brain- and body-machine interfaces, is that the BoMIs being based on natural movements, facilitate the exploration of new motor patterns through the recognition of silent or weak abilities and targeting them with specific exercises. This is beneficial from a general clinical perspective and, specifically, for supporting the process of recovery for its users.

The assistive and rehabilitative functions of the BoMI are of mutual benefit: as the subject regains some of the lost motor control abilities through the practice with the BoMI, these abilities are included in the control of the interface to generate a more efficient and balanced operation of the external device [16].

A rich body of literature suggests that sustained sensory-motor practice promotes and facilitates plastic changes at different sites of the central nervous system [17–19] and, in particular after spinal cord injury (SCI) [20–22]. Anderson, [23] in a survey of 687 SCI survivors, underlined a diffuse awareness of the importance of exercise for recovery across this population. However, half of the respondents did not have access to exercise or to a trained therapist. The BoMI is likely to initiate a new field of research, where the development of devices supporting critical functional activities, such as ambulation and computer use, is combined with research of rehabilitation procedures aiming at the restoration of motor abilities.

Here we report the results from a study based on harnessing the residual mobility of the upper body in survivors of cervical spinal cord injury. We describe how the interface can be designed and modified for engaging the users in functional exercises aimed at enhancing their residual mobility while assisting them R2-1 in the performance of functional and entertaining activities.

2. METHODS

2.1 General Approach

The body-machine interface is a system that remaps the body motions that remain available to a disabled subject, to control an external device, e.g. a powered wheelchair or a personal computer. Typically, people with tetraplegia are able to control a number of degrees of freedom that are in excess of the number of independent signals needed for controlling the device. Thus, the BoMI establishes a mapping from a higher-dimensional body signal space to a lower-dimensional space of device-control variables. This mapping transforms any targeted residual movement capacity into a specific operational function, such as moving a cursor to a target or setting the speed and direction of a wheelchair. Unlike brain-computer interfaces, the body-machine interface engages its users in overt motor activities. It is therefore possible to design the interface with opposite but complementary goals: to exploit the motions that the user is capable to produce with greatest ease or to practice the motions that the user may be able to recover. The transition between assistive and rehabilitative goals is regulated by the mapping parameters. These can be set to enhance the contribution of the degrees of freedom that the subject control with greater ease, when the goal is to enable the operation of the assistive device. Alternatively, a greater role can be assigned to the degrees of freedom that the subject needs to exercise for promoting functional recovery. Here, we focus on the rehabilitative potential of the interface and on the possibility to switch from ease of control to exercise weak or silent abilities.

2.2 Experimental Setup

In this study, the BoMI's sensors are four wireless and low cost inertial measurement units (IMUs) (Yei Technology, 3-Space Sensor™ Wireless) placed on a garment attached by Velcro™ strips to the shoulders and upper arms [24]. The sensors were positioned as follows: sensor 1 on left arm, sensor 2 on left shoulder, sensor 3 on right shoulder and sensor 4 on right arm (Figure 1).

Figure 1.

Experimental Setup. The subject sits in front of a computer wearing four Inertial Measurement Units that communicate wireless with the computer. With his upper body movements the user is controlling the movements of a virtual cursor.

Each IMU outputs 3 signals in real time: pitch, roll and yaw angles. We decided not to use the yaw angle because it is based on measurements from a tri-axis magnetometer that tends to drift or provide unreliable measurements in presence of strong magnetic fields. For this reason the system generates, at every instant n, a 8-dimensional signal vector q̄⁽ⁿ⁾ = [q₁⁽ⁿ⁾, q₂⁽ⁿ⁾, …, q₈⁽ⁿ⁾]^T containing the output (pitch, roll) of all sensors. We use a linear transformation, Principal Component Analysis, to perform a dimensionality reduction mapping the body motion vector q̄ into a lower dimensional control vector p̄⁽ⁿ⁾ = [p₁⁽ⁿ⁾, q₂⁽ⁿ⁾]^T that is used to guide the movement of a computer cursor.

During the calibration phase, at the beginning of the first session, each participant executed free-style upper-body motions for 1 minute. The IMUs captured the user’s residual mobility and then PCA was performed on this data set to identify the covariance matrix of the IMU signals. We took the first two eigenvectors and h₁ = [h_1,1, h_1,2, ⋯, h_1,8]^T and h₂ = [h_2,1, h_2,2, ⋯, h_2,8]^T this matrix and combined them in a matrix H that generates the linear mapping from the body space to the cursor space:

$${\overline{p}}^{(n)}=\left[\begin{array}{ccc}{h}_{1,1}\hfill & \dots \hfill & h_{1,8}\hfill \\ h_{2,1}\hfill & \dots \hfill & h_{2,8}\hfill \end{array}\right]·{\overline{q}}^{(n)}=H·{\overline{q}}^{(n)}$$ (1)

The H-matrix represents the hyperplane that explains most of the variance of the movements the subject performed during the calibration. The components of the vector p̄⁽ⁿ⁾ are respectively the x and y coordinate of the cursor at every instant. Since the eigenvectors are organized by PCA in decreasing order of variance accounted for, we renormalized the H-matrix so as to equalize the total excursion of the two components of p̄. This calibration procedure effectively identified the plane of maximum mobility embedded within the space of the IMU signals. A complete technical description of the interface can be found in [25]. By establishing a correspondence of the task space with this plane we associated the control variables with the degrees of freedom that the impaired subjects spontaneously used most and with greatest ease. This constitutes an assistive approach to the control of the external device. Next, we considered how the interface (and the corresponding plane) may be modified to support a rehabilitative goal

2.3 Representing goals as map modifications

Our technique for transforming body movements into cursor movements allows us to select not only degrees of freedom and coordination patterns that are most comfortable for the user, but also those that the subject has more difficulty operating. One can modify the map to challenge the participants and push them to perform motions that the therapist would like to train and recover. To this end, one can alter the initial map H in two different ways, by using two additional matrices D and S:

$$H_{\mathit{\text{new}}}=H\circ (D·S)$$ (2)

Here, D is 2×8, S is 8×8 and ∘ is the Hadamard product obtained by multiplying pairwise the corresponding elements of the two matrices.

S = diag(s_i) is a diagonal matrix that acts on each IMU signal (q_i). All the elements s_i are initialized to 1. Then, if we set s_i > 1 the gain of the correspondent input signal q_i will increase. On the contrary 0 < s_i < 1 will decrease it. A second modification regards how much each IMU signal contributes to the cursor movement in the horizontal and vertical directions. The matrix D operates this transformation. Its first row d̄₁ = [d_1,1, d_1,2, ⋯ d_1,8] contains the contribution of each sensors’ channel in the horizontal cursor movement, while the second row d̄₂ = [d_2,1, d_2,2, ⋯ d_2,8] refers to the sensor contributions to the cursor movement along the vertical direction. Initially, all the elements d_i,j are set to 1. To increase the control authority of the j-th sensor on the cursor's horizontal direction one must set d_i,j > 1. Conversely, 0 < d_i,j < 1 will decrease the authority of the sensor on the horizontal direction. In the same manner, if one wishes to modify the contribution of the j-th sensor to the cursor movement along the vertical direction, one will change the correspondent element d_2,j in the matrix D. The change applied by the matrices S and D influence not only the relative contribution of each sensor, but also the total range of the movement of the cursor.

At this time, we limited the methodology to a heuristic approach based on the observation of specific deficits in two subjects. We considered the goal of recovering left/right symmetry in SCI participants that used one side of their body more than the other.

The calibration procedure assigns higher gain and therefore more control authority to the body's degrees of freedom that subjects tend to move more. Accordingly, the weaker body parts that have lower range of mobility and are controlled with more difficulty are associated with lower gains. This strongly limits their contribution to the BMI control. Therefore, the first step of the map modifications is to increase the weight of these sensors so that even a smaller motion of these sensors would have significant effect on the motion of the cursor. This can be done by increasing either the value of the corresponding elements of the S or D matrices. One acts on the S matrix when we want to change the control authority of a body part in all the workspace. One acts on the D matrix when we want specifically modify it in a given direction. It is also possible select a second action, i.e. to decrease the gains of the stronger degrees of freedom along one direction acting on the D matrix so that subjects cannot use them to reach their goal along this direction. Thus, subjects are induced to use the weak degrees of freedom to reach the exercises’ goals.

Once the subjects reach sufficient body symmetry in controlling the interface, it is possible to work on the range of motion by scaling all the sensor gains in the S matrix in order to encourage subjects to move more to reach the same goals. The amount of change for each gain depends both on the individual subject’s ability and on the initial gains of the map. The therapist sets the value according to the principles mentioned above, by verifying that the task is still feasible for each subject with a reasonable and sustainable effort.

We decided to operate the modifications in two separate days because we wanted to gradually challenge the users. We thought that abrupt changes could make the practice too hard and would demotivate the participants. On the contrary, in the way we chose to act the user was still challenged by the interface but he was able to practice and terminate every task proposed in the protocol.

2.4 Subjects

We validated this approach with two high level spinal cord injured subjects, 1 male and 1 female (see Table 1). They were medically stable and were recruited from the inpatient unit of the Rehabilitation Institute of Chicago after signing the informed consent approved by Northwestern University Institutional Review Board.

TABLE I.

Characteristics of the subjects that participated in the study. ASIA: American Spinal Injury Association impairment scale. ASIA A means that the lesion is complete, thus there are no motor and sensory functions below the lesion [26].

Subject	Gender	Age	Level of Injury	Time after Injury
SCI 1	Male	48	C4 ASIA A	90 days
SCI 2	Female	20	C4 ASIA A	45 days

As consequence of the lesion, Subject 1 was having a poor control of his right side, especially the arm, and a better use and movement of the left side. For this reason the goal was to recruit more the right side in the use of the interface. In contrast, Subject 2 preferred to use the right side over the left. In this case we wanted her to equally use both body sides to control the cursor movement.

2.5 Experimental Protocol

The participants were sitting in their own wheelchairs and placed in front of a computer screen.

They went through an intensive period of training, consisting of 11 practice sessions, each one of a maximal duration of 1 hour, across 2 weeks.

After the assessment tests of the first day, the participants performed the calibration procedure. They were asked to engage in a variety of self-directed, self-paced motions of the upper body. This "free dance" calibration procedure is also described in [27]. From the collected IMU data we calculated the map H and then we adjusted the map parameters to insure that the subjects could control without difficulty the cursor position across the entire domain of the monitor. As described in [25] after the calibration phase, there was a customization phase. In fact, even if the first two PCs captured the biggest amount of the total variance, they did not necessarily reflect the natural up-down/left-right orientation of the display monitor. During the customization phase we can change the origin and the orientation of the task space in accordance to the user’s preferences.

The calibration procedure was performed only in the first session. Each session consisted of three parts:

1. Reaching. The subjects performed the reaching task by moving the cursor over a set of targets. The targets were placed in a center-out pattern, with 8 directions, and the subjects were to perform three reaches for each direction. The peripheral target appeared with a green border, and turned red after one second. The goal was to acquire the external target before it changed color, and the cursor were to stay within the target for 500 ms.

2. Gaming practice. Generally, the daily exercise protocol was organized into different tasks: pong, and other computer games. Pong consisted in a horizontal handle controlled by the participant, which had to hit a ball bouncing off the 4 walls of a court. The player got a point every time he hits the ball bouncing off the top wall. Each pong epoch lasted 2 and half minutes, and was repeated 5 times. If there was enough time within the hour of practice, the participant could choose one or more computer games to play among different flash games available in the menu of the interface (e.g. Uno, Spider, Snake, Gems Swap).

3. Reaching. This was identical to part 1.

In [25] we observed that SCI subjects’ performance become stable after 4 sessions. Thus, we started with four sessions of familiarization, where we did not apply any changes to the calibration map. In the first day, subjects performed only the reaching task. On day 5 and day 7 of the training we introduced the modifications of the map in order to achieve the rehabilitative goal established for each participant.

Specifically, subject 1 at the end of the familiarization phase was controlling the cursor mostly with the left arm. So we first decided to act on the matrix S, trying to encourage the use of left and right shoulders (s₃, s₄ = 5 and s₅, s₆=10). Then, the second change was operated on the matrix D. The subject was responding well to the given challenge, and we decided to decrease the control authority of the left arm on the vertical direction and to increase the authority of right shoulder and arm. In this way the subject had to use his right body side to move the cursor vertically, while he could still control the movement of the cursor on the horizontal direction with the left side.

Compared to the first subject, subject 2 had a greater range of motion of both arms (Figure 8). However, subject 2 did not use the two arms equally well because of poor trunk control. The first change we operated with subject 2 was on the matrix D, so as to assign gradually in two steps - i.e in two subsequent changes of the D matrix - to each body side the control of one cursor's direction. When the symmetry and the trunk control were recovered, we could work on the range of motion, equally decreasing on the matrix S all the gains associated with all the IMUs.

Figure 8.

Results of the assessment tests for Subject 1, A, and Subject 2, B. Manual Muscle Test, top left, Isometric force, top right, and range of motion, bottom, before and after the training.

In table II we reported the modifications chosen for each participant.

TABLE I.

Map modifications details for each participant. The first modification chosen for Subject1 consisted in increasing the gains s₃, s₄, s₅, s₆ associated with the sensors placed on the shoulders. The second modification acted on the contribution of the body to the movement of the cursor in the vertical direction, thus d̄₂ was changed. With Subject 2 we acted from the beginning on the contribution of the body to the cursor movement in the horizontal and vertical direction, and we changed d̄₁ and d̄₂ respectively.

Subject	1st modification	2nd modification
SCI 1	s3 = s4 = 5 s5 = s6 = 10	d̄2 = [0.4 0.4 0.4 0.4 2 2 2 2] s3 = s4 = 5 s5 = s6 = 10
SCI 2	d̄1 = [1.2 1.2 1.2 1.2 0.4 0.4 0.4 0.4] d̄2 = [0.5 0.5 0.5 0.5 1 1 1 1]	d̄1 = [1.3 1.3 1.3 1.3 0.4 0.4 0.4 0.4] d̄2 = [0.4 0.4 0.4 0.4 1.3 1.3 1.3 1.3] S = diag(0.6)

2.6 Data Analysis

2.6.1 Learning Metrics

We investigated whether the subjects became skilled at controlling the cursor. The analysis focused on the center-out movements in the reaching tasks (beginning and end of every session) and on the pong. In particular we evaluated if the movements in the reaching tasks became faster, smoother and straighter by computing the following metrics:

• -Movement Time, time to reach the external target once the cursor exited the central target;
• -Normalized Path Length, length of the path covered by the cursor to reach the external target normalized by the nominal distance between the central and the peripheral targets. If this metric is equal to 1 the cursor controlled by the user moved along a straight line;
• -Jerk index, the norm of the jerk (the time derivative of the cursor acceleration), averaged over the entire movement duration and normalized with respect to duration (T) and path length (L) [28]. This index is sensitive to smoothness, larger jerk index corresponding to less smoothness;
• -Number of peaks in the velocity profile. We considered every peak larger than a threshold that was set to be 15% of the maximum speed of each trajectory This is effectively another measure of smoothness, based on how many movement segments were included in each reaching movement.

In the pong game, we calculate the mean values of the number of hits.

2.6.2 Body Contribution to Cursor Control

We were interested in isolating the contribution of each side of the body to the cursor movement. The first 4 elements of the body signal vector derived from IMUs on the left side of the body, while the last 4 derived from IMUs on the right side. Therefore we rewrite q̄ as the sum of left and right vectors

$$\overline{q}=\left[\begin{array}{c}\hfill q_1\hfill \\ \hfill ⋮\hfill \\ \hfill q_4\hfill \\ \hfill 0\hfill \\ \hfill ⋮\hfill \\ \hfill 0\hfill \end{array}\right]+\left[\begin{array}{c}\hfill 0\hfill \\ \hfill ⋮\hfill \\ \hfill 0\hfill \\ \hfill q_5\hfill \\ \hfill ⋮\hfill \\ \hfill q_8\hfill \end{array}\right]={\overline{q}}_{\mathit{\text{left}}}+{\overline{q}}_{\mathit{\text{right}}}.$$ (3)

Substituting this expression in Equation 1, we determine how each side of the body contributed to the total movement of the cursor:

$$\overline{p}=H·{\overline{q}}_{\mathit{\text{left}}}+H·{\overline{q}}_{\mathit{\text{right}}}={\overline{p}}_{\mathit{\text{left}}}+{\overline{p}}_{\mathit{\text{right}}}.$$ (4)

Symmetry Indices in Body Space and Task Space

In the task space, we used the following equation to calculate the percentage of the body contributions (c_left and c_right) to the cursor movement

$$\frac{{\overline{p}}_{\mathit{\text{left}}}}{{\overline{p}}_{\mathit{\text{left}}}+{\overline{p}}_{\mathit{\text{right}}}}+\frac{{\overline{p}}_{\mathit{\text{right}}}}{{\overline{p}}_{\mathit{\text{left}}}+{\overline{p}}_{\mathit{\text{right}}}}\equiv c_{\mathit{\text{left}}}+c_{\mathit{\text{right}}}=1$$ (5)

If c_left ~ 0.5 or c_right ~ 0.5 there is a symmetric condition in the cursor control. If c_left < 0.5 the right part is the one being used more, and vice versa.

In body space, we used the same approach, and we calculated the relative amount contributed by each side of the body (b_left and b_right) from the standard deviations of the 2 channels of each IMU:

$$\frac{{\mathit{\text{std}}}_{\mathit{\text{pitch}}1}+{\mathit{\text{std}}}_{\mathit{\text{roll}}1}+{\mathit{\text{std}}}_{\mathit{\text{pitch}}2}+{\mathit{\text{std}}}_{\mathit{\text{roll}}2}}{{\mathit{\text{std}}}_{\mathit{\text{tot}}}}+\frac{{\mathit{\text{std}}}_{\mathit{\text{pitch}}3}+{\mathit{\text{std}}}_{\mathit{\text{roll}}3}+{\mathit{\text{std}}}_{\mathit{\text{pitch}}4}+{\mathit{\text{std}}}_{\mathit{\text{roll}}4}}{{\mathit{\text{std}}}_{\mathit{\text{tot}}}}\equiv b_{\mathit{\text{left}}}+b_{\mathit{\text{right}}}=1$$ (6)

with

$${\mathit{\text{std}}}_{\mathit{\text{tot}}}={\mathit{\text{std}}}_{\mathit{\text{pitch}}1}+{\mathit{\text{std}}}_{\mathit{\text{roll}}1}+{\mathit{\text{std}}}_{\mathit{\text{pitch}}2}+{\mathit{\text{std}}}_{\mathit{\text{roll}}2}+{\mathit{\text{std}}}_{\mathit{\text{pitch}}3}+{\mathit{\text{std}}}_{\mathit{\text{roll}}3}+{\mathit{\text{std}}}_{\mathit{\text{pitch}}4}+{\mathit{\text{std}}}_{\mathit{\text{roll}}4}.$$

Also in this case in case of symmetry we will have b_left ~ b_right ~ 0.5.

To evaluate the movement of the upper body we calculated the standard deviation of the signals generated by each sensor during session four and session eleven. In particular we used the total standard deviation, std_tot, as an estimate of the subject's overall mobility.

We computed these indicators during two sessions: a) session four, the last session of the familiarization phase, and b) session eleven, the last session of the training phase.

2.7 Clinical evaluation

Before and after training, we executed different assessment tests to evaluate the upper body strength and mobility. To characterize the strength of the three upper body regions, scapulae, shoulders and arms, we used a modified Manual Muscle Test [29]. The test was performed as the participant was sitting in the wheelchair. The movements that were tested are reported in Table III. Each movement was evaluated with a number from 0 to 5, with 0 = no movement, 1 = trace, 2 = poor movement without gravity, 3 = fair movement against gravity, 4 = good, 5 = normal. The maximum score for the scapula is 15, for the shoulder is 20, and for the arm is 10. In addition, we used a force transducer (Mark-10, force gauge MG series) to measure the isometric forces of the shoulder during movement in the upward, backward and forward directions. We also measured the range of motion (ROM) of the shoulders in all the possible directions using a goniometer, see Table IV. We could not measure shoulder adduction and shoulder flexion because the participants were sitting in the wheelchair that makes impossible to perform those movements.

TABLE IIII.

Muscles tested with the modified manual muscle test (MMT).

Body Part	Movement - Muscle
Scapula	Elevation – Upper Trapezius
	Adduction – Rhomboids
	Abduction – Serratus Anterior
Shoulder	Flexion – Anterior Deltoid
	Abduction – Middle Deltoid
	Horizontal Adduction – Pectoralis Major-Clavicular
	Horizontal Abduction – Posterior Deltoid
Elbow	Flexion – Biceps Brachii
	Extension – Triceps Brachii

TABLE IV.

Movements evaluated with the goniometer to extract the range of motion of the upper body in the frontal, sagittal and transverse plane.

Plane	Movement
Frontal	Shoulder Elevation
	Shoulder Depression
	Shoulder Abduction
Sagittal	Shoulder Protraction
	Shoulder Retraction
	Shoulder Flexion
Transverse	Shoulder Horizontal Adduction
	Shoulder Horizontal Abduction

2.8 Statistical Analysis

We tested the performance of each SCI subject separately by comparing the indicator values between the first and last session using a paired t-test. Threshold for significance was set at p<0.05. Moreover, we tested if the changes in the assessment tests where significant running a paired t-test between the values pre and post treatment.

3. RESULTS

3.1 Participants learned to use the interface

During the initial phase of training (sessions 1 to 4), the interface operated in "assistive mode" and its parameters remained unchanged as established by the calibration. Both subjects displayed a significant learning, with improvement in all the metrics that we chose to assess the performance during the reaching task (Figure 2). All the metrics decreased, both between and within sessions. The reaching trajectories became straighter, as shown also in figure 3. In day 1, the cursor trajectories were quite erratic and tangled (Figure 3A,D). On the contrary, on day 4 they became straighter, as the subjects were now able to move more consistently toward each of the 8 targets. As the trajectories became straighter, the time to reach the target decreased (Figure 2B,F). The cursor movement became smoother, as the jerk index and the number of peaks in the speed profile decreased.

Figure 2.

Reaching performance of subject 1 (A–D) and subject 2 (E–H). Mean path length for subject 1 and 2 are respectively shown in panel A and E, mean time to target in B and F. Jerk in C and G, and number of submovements in D and H. In dark color are shown the metrics extracted from the data of the reaching executed at the beginning of each session, in light color those at the end of the session. The dashed lines indicate when the changes in the map happened. In session 1 there is only one bar because on that day the subjects practiced only one reaching task. The difference between all the indicators of session 1 and session 11 is significant.

Figure 3.

Reaching trajectories evolution of subject 1 (A–C) and subject 2 (D–F). Trajectories of the very first reaching (A,D), of the reaching at the end of day 4 (B,E) and at the end of the training (C,F) are shown. Only one trajectory for each direction is visualized.

The changes of all reaching metrics that occurred from the beginning to the end of the training period were significant, see Table V.

TABLE III.

P values resulting from the paired t test of the reaching metrics of the first and last session.

Reaching Metric	Subject 1	Subject 2
Normalized Path Length	< 0.001	0.019
Time to Target	< 0.001	0.009
Jerk Index	0.014	0.048
Number of peak in the velocity profile	< 0.001	0.002

3.2 Performance changes after modification of the map

On day 5, when we introduced the first map modification, the performance worsened and the values of all the indicators increased. They then got slightly better starting from the second reaching of the same session (day 5) until the introduction of the second map modification on session 7, when we observed a similar negative effect on the performance indicators for session 5. Both modifications were intended to challenge the subjects and induce them to increase the participation in cursor control by the less used side of their body. After day 7 all metrics started to decrease again, and on the last training day they became comparable with the values observed at the end of the familiarization phase. This is also evident from a visual inspection of the trajectories (Figure 3C,F).

A similar trend is evident in the number of hits by both subjects in the Pong game (Figure 4). The subjects improved their ability to play and intercept the ball during the first 4 days on familiarization. When we introduced the changes in the map their performance worsened, but with practice it improved and eventually exceeded the levels attained after the first phase of training with the assistive map.

Figure 4.

Pong Performance of Subject 1 (left panel) and Subject 2 (right panel). Mean number of hits per each session. The line is interrupted every time we made a change in the map (vertical dashed red lines).

3.3 Increased Symmetry in Cursor Control and Upper Body Movement

When we analyzed the body contribution to the cursor movement in the early training phase, it was evident that subject 1 used mostly the left side to complete the tasks, (Figure 5A). In contrast, subject 2 at the beginning (session 1) was using both arms to control the cursor (Figure 5D) with a predominance of the right. Then practicing with the assistive map led at the end of the familiarization (session 4) to an almost exclusive use of the right arm (Figure 5B).

Figure 5.

Body contributions to cursor trajectories in different phases of the training for Subject 1 (A–C) and Subject 2 (D–F). In red is shown the contribution of the right body side, while in green the contribution of the left one.

By changing the map to reach the rehabilitative goal, at the end of training we modified the behavior of both participants in the control of the cursor (Figure 5C,F). They were able to proficiently control the interface, guiding the cursor along straight, fast and smooth trajectories, while they recruiting right and left side almost symmetrically. Both subjects used the right side to move the cursor along the vertical direction and the left side along the horizontal direction.

At the end of the familiarization phase, in the last reaching, subject 1 using the assistive map was contributing to the cursor movement for 6% with his left side, while at the end of the training using the rehabilitative map the contribution increased up to 47% (Figure 6A). A corresponding outcome was observed for subject 2. At the end of the familiarization phase, only 30% of the cursor movement was due to the left side. At the end of training, the left increased the contribution to 56% (Figure 6B).

Figure 6.

Symmetry in task space (A, B) and body space (C, D) of subject 1 (A and C) and subject 2 (B and D). In each panel the variability or the contribution to cursor control due to the left and right body parts are shown for all the sessions of the familiarization phase and for the last two sessions of the training. In blue is represented the information regarding the left side while in red the one of the right side.

The same indices in body space exhibit a correlated change (Figure 6C,D). During the reaching task of session 4, there was a big asymmetry in the body movements. Subject 1 moved the right body side for 14% of the total body motion. Similarly, subject 2 used the left body side for a 27% of the total body motion. At the end of the training period, both subjects recovered a more symmetric body mobility with a 46% left body usage for subject 1 and 40% left body usage for subject 2 (Figure 6A,B). Furthermore, from session 4 to session 11 there was an increase of the total movement of all upper body parts for both subjects (Figure 7A, C). At the end of the training, the symmetry recovery did not suppress the movement of the side that at the beginning was moving more (Figure7B and 7D). On the contrary it promoted the movement of the weak side, right side for subject 1 and left side for subject 2, keeping stable (for subject 1) or even increasing (for subject 2) also the movement of the strong side.

Figure 7.

Total Mobility during the reaching exercise of session 4 and session 11, A, C and total mobility of left (blue) and right (red) body side during session 4 and session 11, B, D for subject 1 and 2 respectively.

3.4 Improvements in the Assessment Tests

Data from the assessment tests showed a positive effect of practicing with the interface on muscle strength and upper body mobility (Figure 8). The sum of the MMT values for each upper body district (scapula, shoulder and arm) after the training period increased for both subjects (Figure 8A,B). Consistently with the MMT scores, the isometric force value measured at the shoulders increased (figure 8A,B top right). With respect to the upper body mobility we measured an increase in the ROM of shoulders and arms in all the different planes (Figure 8A,B bottom). Subject 1 could not move the right arm against gravity, thus we could not measure arm flexion, arm abduction and arm horizontal abduction/adduction. The global changes in the MMT, isometric force test and ROM pre and post treatment are significant with exception of the isometric force test for Subject 1, see Table VI. Additionally, between pre and post treatment, we noticed an increase in the MMT and isometric force test of the symmetry of the scapular district, that was the one mostly targeted in the training (MMT scapulae S1: ΔLR_pre = 3 ΔLR_post = 0; MMT scapulae S2: ΔLR_pre = 2 ΔLR_post = 0; isometric force scapular district S1: ΔLR_pre = 24.4 ΔLR_post = 7.4; isometric force scapular district S2: ΔLR_pre = 15.8 ΔLR_post = 10.4). We do not see the same similarity left/right in the range of motion.

TABLE IVI.

Significance of the changes in the assessment tests performed pre and post treatment in Subject 1 and Subject 2. We considered significant a p value <0.05.

Paired t-test Pre and Post treatment	Subject 1	Subject 2
Isometric Force	0.09	0.02
Manual Muscle Test	0.02	0.01
Range of Motion	0.04	0.003

4. DISCUSSION

In this work we tested the feasibility of a new approach to the human-machine interface that combines assistive technology with rehabilitative goals. We developed a low-cost interface that we call the "Body-Machine Interface" (BoMI). It is based on overt body motions of its users and exploits the natural ability to reassign a motor goal to a component of the body that is not normally used for that goal. An example of this kind of adaptive skill is seen when people that lose their hands write holding a pencil in their mouth or using their feet [30].

Here, we considered a BoMI that maps motions of the upper body onto a two-dimensional point, corresponding to the location of a cursor on a computer screen or, equivalently, to the speed/direction control of a powered wheelchair. The key requirement of our approach is for the interface to be highly customizable to the user impairment and to the rehabilitative goal established for each subject. The BoMI can be used to evaluate and promote mobility and independence of the user. At the same time, it can be modified to address specific rehabilitative goals. Here, as a first example, we consider the goal of promoting in spinal cord injured participants a more symmetric and synergistic use of the two sides of the body that they are still able to control. We first initialized the interface to match the user's residual abilities so as to control a computer cursor with the greatest ease. Then, after the users familiarized with the task, we reprogrammed the interface so as to encourage the recovery of mobility on the side that was used less. As expected, immediately after the change of the interface map there was a transient degradation of performance. However, with practice, both subjects reestablished a high level of competence and, by the end of training, they attained the desired level of symmetry. The achievement of greater symmetry did not suppress the movement of the side that initially was most active. On the contrary it promoted the movement of the weaker side, keeping stable or even increasing the movement of the stronger side.

The outcomes of the assessment tests were encouraging as well. When we compared the values of the MMT before and after training we noted an increase of the indicator in all the body districts tested. These results are supported by the concurrent increase of the strength at the shoulders and of the range of motion of shoulders and arms, consistent with findings by Casadio et al. [31].

One limitation of our study is that both subjects had recent injuries. Therefore, one must also take into consideration the spontaneous recovery that may take place during the training (7 sessions). However, in our study the left/right asymmetry persisted stably throughout the first four sessions until the introduction of the map modifications.

A second limitation is that at this time, we are not yet able to offer an automated procedure for modifying the BoMI mapping, although we believe that such procedure may become feasible in the future, based on the possibility to use the interface both as a diagnostic and as a therapeutic tool. Currently, the interface can only be used based on the assessment of the therapists and on a heuristic procedure.

The findings described in this article are consistent with the general notion that the successful interaction with the interface is based on the brain’s ability to reorganize movement coordination. This reorganization corresponds to a central remapping of brain activities. In a landmark imaging study [32] Rijntjes and coworkers found that when handwriting motions are performed by the foot, activations appear in brain areas associated with the hand, the body part that usually performs this action. Other studies suggested that learning a new skill induces also structural changes in the brain[33–38], and that these changes increase with amount of practice [35]. There are now several examples of movement reorganization occurring after being exposed to force fields, visual distortions or other sensory-motor remapping processes. Flanagan and Rao [39] demonstrated that joint coordination of arm reaching movements is altered when the visual feedback is presented in terms of joint angles instead of hand coordinates. Danziger and Mussa-Ivaldi [40] obtained a similar result in another multi-joint coordination task.

Several studies have highlighted the ability of the human motor system to reorganize the coordination of multiple degrees of freedom in the human body for performing novel functional tasks [10–12, 27, 31, 41, 42]. Mosier et al. [42] found that through practice unimpaired participants were able to learn how to control the motion of a cursor by hand gestures. In this process, the subject effectively learned to remap hand control and to represent the inherent Euclidean properties of the space in which the reaching task was defined. Afterwards, Casadio et al. [10, 27, 31] extended this paradigm to the control of a cursor through movements of the upper body. Radhakrishnan and colleagues [43] performed similar experiments in which subjects were required to control a cursor by modulating the myoelectric activity of an immobilized arm. They found that subjects learned to move the cursor in directions that were not congruent with the directions that would have been obtained by a freely moving hand driven by the observed muscle activity patterns. The authors called these directions "unintuitive" and concluded that overt movement is not necessary for this type of motor remapping.

Our findings are consistent with the motor learning observed in monkeys whose motor cortical activities were controlling a cursor on a computer monitor through a brain-machine interface [44–46]. There, like in our case, the nervous system must learn to select the degrees of freedom that are most relevant to the desired movement. By controlling the amount of dimensionality reduction, our paradigm allows us to explore the mechanisms by which the brain reorganizes the control of a kinematically redundant system when presented with a novel mapping from natural body motion to their sensory and functional consequences.

The approach presented in this work is based on a dynamical interface, which changes its parameters and behavior depending on the state of the user and on his goals. The subject needs to learn how to redirect the residual functions to achieve alternative and potentially useful applications. At the same time the BoMI "understands" and facilitates the reorganization of residual motions for the control of any assistive device through a user specific design. The idea of adaptive interface is actively pursued in BMI research as well. The concepts of “dual learning” and coadaptation are prerogatives in the development of fast adaptive BMIs that take into account also the learning processes that happen in the subjects when they learn how to use the interface. For example, several investigators consider how to modify the decoding map based on the history of errors and performance of the users [47–49].

Here we also considered the case in which the interface must assume a role opposite to enhancing the ease of operation by its user. If the interface is used in the “rehabilitative mode”, it has to modify itself so as to make the exercise challenging but not too difficult [50] and this modification has to be coherent with the user’s deficit and the rehabilitative goal. Recently also Brain–machine interfaces were used for reaching rehabilitative goals with chronic stroke survivors [51–53]

We do not want to present the BoMI in alternative to the BMI. On the contrary our idea is that the two types of human-machine interface are potentially complementary. In this work, signals coming from IMUs are decoded to generate a control command to an external device. Other non-invasive approaches can be used as well, focusing more on EMG o EEG signals. The use of IMUs does not exclude the possibility to adopt hybrid methods that will combine different kind of signals to increase the dimensionality of the body space. However, it is only the emphasis on exploiting overt body motions that allows us to include functional rehabilitation as a collateral goal to providing assistance and substitution of natural skills.

As Krakauer et al. suggested [54, 55], neurorehabilitation plays an important role in movement recovery after injury, and is based on the assumption that motor learning contributes to that process. Thus, body-machine interfaces that exploit motor learning principles and are based on highly customizable and flexible approaches, could be potentially beneficial not only for SCI individuals, but also for different populations of impaired subjects that need both assistance and rehabilitation, such as stroke survivors or individuals affected by multiple sclerosis..

5. CONCLUSIONS

By mapping all the residual movement capacity into specific operational functions, we expected that paralyzed users find a natural balance between ease of device control and exercise of under-utilized muscles. The proposed body-machine interface is suited to exercise all of the available degrees of freedom in the upper body through targeted practice of control actions in virtual reality environments. Here, we showed that it is possible to define and implement transformations from body motions to cursor control space emphasizing the degrees of freedom that are more difficult to control. The proposed methods is promising for the development of low cost technologies and systems that can be used with two different modalities: assistive and rehabilitative. People with severe disabilities can use the BoMI on a regular basis, to control a powered wheelchair or a computer, checking emails, playing games and surfing the Internet. But by changing the interface’s parameters they can also do their workout and rehabilitation sessions, in full autonomy or with a physical therapist overlooking the training.

Highlights.

• A paradigm for the use of body-machine interfaces in rehabilitation is presented

• The BoMI engages the user in exercises aimed at motor recovery while playing games

• The interface is highly customizable to the users’ needs and deficits

• This approach restored symmetry between left and right side of the upper-body

• Mobility and strength of the body parts involved in the task increased