Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Feb 16.
Published in final edited form as: Lancet. 2013 Feb 16;381(9866):557–564. doi: 10.1016/S0140-6736(12)61816-9

7 degree-of-freedom neuroprosthetic control by an individual with tetraplegia

Jennifer L Collinger 1, Brian Wodlinger 2, John E Downey 3, Wei Wang 4, Elizabeth C Tyler-Kabara 5, Douglas J Weber 6, Angus JC McMorland 7, Meel Velliste 8, Michael L Boninger 9, Andrew B Schwartz 10
PMCID: PMC3641862  NIHMSID: NIHMS448153  PMID: 23253623

SUMMARY

Background

We use our arms to transport and orient the hand which is used to grasp and manipulate objects. Upper limb paralysis or amputation limits a person’s ability to interact with their environment to accomplish activities of daily living. Brain-machine interfaces (BMIs) may provide a solution to restoring much of this function.

Methods

Two 96-channel intracortical microelectrodes were implanted in the motor cortex of an individual with tetraplegia. Thirteen weeks of BMI training were conducted with the goal of controlling an anthropomorphic prosthetic limb with 7 degrees-of-freedom (3D translation, 3D orientation, 1D grasping). Clinical measures of upper-limb function were used to assess the participant’s ability to use the prosthetic limb.

Findings

The participant demonstrated the ability to move the device freely in the three-dimensional (3D) workspace on the second day of training. After 13 weeks, robust 7 degree-of-freedom movements were performed routinely. Over time, performance on target-based reaching tasks improved in terms of success rate, completion time, and path efficiency. The participant was also able to use the prosthetic limb to perform skillful and coordinated reach and grasp movements that resulted in clinically significant gains in tests of upper-limb function.

Interpretation

This study demonstrates that a person with chronic tetraplegia can perform consistent, natural, and complex movements with an anthropomorphic robotic arm to regain clinically significant function.

Funding

Defense Advanced Research Projects Agency, National Institutes of Health, Department of Veterans Affairs, and UPMC Rehabilitation Institute

INTRODUCTION

Brain-machine interfaces (BMIs) transform neural activity into control signals for an external device. Functional electrical stimulators, exoskeletons, and sophisticated prosthetic limbs are all being developed with the goal of restoring or replacing natural function to a paralyzed or lost limb. In order to perform activities of daily living, one needs to be able to position the hand in space, orient the palm, and grasp an object. Hand placement, orientation, and grasping are normally smoothly coordinated and follow general principles of natural movement.13 Ideally, a BMI will capture these features directly from neuronal activity and translate them to control of an external device with the capability of producing natural movements that follow these principles.

Animal studies using intracortical microelectrodes have shown that these natural features of movement can be captured in recordings of populations of motor cortical neurons4. These studies have demonstrated control of a robot arm in 4 dimensions to perform self feeding tasks5, 7-dimensional (7D) control for orientation and grasping6, and control over functional electrical stimulators of the upper limb.7, 8 To date, human studies have demonstrated 3D translational control911, plus control over a discrete (on/off) grasping dimension.12 Our results build on previously established methods5, 6 to show that a person with tetraplegia was able to rapidly achieve control of a state-of-the-art anthropomorphic prosthetic limb (MPL, Johns Hopkins University, Applied Physics Laboratory). After 13 weeks of training, robust 7D (3D translation, 3D orientation, 1D grasping) movements were performed routinely. These movements included reaching and grasping tasks, similar to many activities of daily living, which were carried out with coordination, skill, and speed approaching that of an able-bodied person.

METHODS

This study was conducted under an Investigational Device Exemption (IDE) granted by the US Food and Drug Administration and with approval from the Institutional Review Boards at the University of Pittsburgh and the Space and Naval Warfare Systems Center Pacific. This trial is registered on clinicaltrials.gov (http://clinicaltrials.gov/ct2/show/NCT01364480). No adverse events have been observed to date.

Participant

Informed consent was obtained from a 52-year-old woman who was diagnosed with spinocerebellar degeneration 13 years prior to participation in the study. Thorough chart review as well as discussions with the participant’s neurologist revealed no indication of cerebellar involvement. Her injury was motor complete with manual muscle test scores of 0/5 for the upper limb.13 Physical examination revealed limited sensory deficits and some hypersensitivity.

Array implantation

Two intracortical microelectrode arrays (4x4 mm, Blackrock Microsystems Salt Lake City, UT, USA), each with 96 electrode shanks (1·5 mm length) were implanted in the left motor cortex on February 10, 2012. Stereotactic image guidance (Brainlab, Inc., Westchester, IL, USA) with co-registered structural (MRI) and functional (fMRI) imaging was used to guide placement of the arrays (figure 1A).

Figure 1. Array location and experiment setup.

Figure 1

(A) Pre-operative functional magnetic resonance imaging (fMRI) activation maps on a subject-specific brain model during video-guided attempted movement. The colored activation maps represent blood-oxygenation level dependent (BOLD) activity during video-guided attempted movements: yellow=sequential finger flexion, red=hand grasping, blue=shoulder shrug, green=lip pursing. Approximate array locations are shown as black squares on the inset figure. The arrays were implanted over motor cortex anterior to the central sulcus (CS) approximately 14 mm apart. (B) MPL and 7D Sequence Task setup. The subject was not presented with physical targets. Instead, LEDs (indicated by the white arrow) were used to instruct the participant to hit the “near” (0·35 m from the shoulder) or “far” (0·52 m from the shoulder) translation target corresponding to one of the white circles on the board in front of the MPL. Orientation and grasp targets were presented by a computer-generated verbal command. (C) Diagram of the MPL and translation targets for the 7D Sequence Task. The MPL coordinate system is shown centered at the shoulder. Translation targets had an 8 cm radius and the MPL endpoint (center of the palm) had to be within this region for a successful trial. The MPL endpoint also had to be within the translation target success region in order to successfully achieve the orientation (± 15 degrees) and grasp the targets that were given as audio cues. The time-out period was set to 10 seconds.

Neural recording

Generally 3 sessions were conducted per week, each approximately 4 hours in duration. Neural signals were recorded using the NeuroPort data acquisition system (Blackrock Microsystems, Salt Lake City, UT). Single- and multi-unit events were classified using manually set thresholds and time-amplitude windows. Daily set-up time, including unit sorting, required approximately 30 minutes. Activity was converted to firing rate in 30 ms bins and low pass-filtered using an exponential smoothing function with a 450 ms window.

Observation-based calibration and neural decoding

BMI training progressed from 3D endpoint translation control (Weeks 2 and 3 post-implant), to 4D control of translation and grasp (Week 4), to 7D control of translation, orientation and grasp (Weeks 5–14). The joints of the fingers and thumb were combined into a single grasp dimension. The neural decoder, based on a model that linearly related neural firing rate to movement velocity (Eq. 1)6, 14, was calibrated daily using a two-phase calibration similar to methods used in non-human primate studies.5, 6

During the observation phase, the MPL moved automatically, driven by a proportional controller, to targets that spanned the reachable workspace of the arm as part of a 7D Sequence Task. In the 7D Sequence Task, a target was indicated by an LED on a board in the frontal plane (80x80 cm), 70 cm in front of the MPL shoulder (figure 1B). Once the MPL was moved to the correct 3D position, computer-generated verbal prompts described an orientation to be performed by rotating the palm in one of 6 directions, or a grasp target (open or closed hand). Possible orientations included pronation-supination (±45°), ulnar-radial deviation (±20°), and flexion-extension (±45°) of the wrist. The participant was instructed to carefully watch the limb as it moved automatically to the targets. Neural data and MPL state information including endpoint position, joint angles, and joint torque were recorded at a rate of 33 Hz. Approximately 80 trials (6 minutes) of 7D Sequence Task data were collected to train the initial observation-based 7D neural decoder. The following model was used to relate unit activity to MPL movement velocity6, 14:

f=b0+bxvx+byvy+bzvz+bθxvθx+bθyvθy+bθzvθz+bgvg Eq. 1

where f was the firing rate of a unit during movement described by 7D velocity vector V (vx,vy,vz,vθx,vθy,vθz,vg) and b0, bx, by, bz, bθx, bθy, bθz, bg were the coefficients for each unit. V includes coordinates for the 3D endpoint translation (x,y,z), 3D orientation (θx, θy, θz) and 1D grasp (g) velocities. Units that failed to fit the model (R2≤0.1) were excluded. Indirect optimal linear estimation (OLE)15 with ridge regression16 was used to solve for the coefficient matrix B (webappendix p1).

During a second phase of calibration, the participant controlled the MPL using the observation-based neural decoder while ortho-impedance was applied to the command signal.5 During this phase, ortho-impedance attenuated, by 100%, the brain-command component perpendicular to the ideal 7D trajectory, restricting movements to directions directly toward or away from the target. Data from 80 trials of the 7D Sequence Task collected during the second phase of calibration were used to build the final decoder using OLE as described above. This second round of calibration allowed the participant to actively engage in the task, while minimizing errors and corrective movements. Early in the experiment (Weeks 2–4), similar calibration procedures were used to generate lower-dimensional neural decoders. Calibration was conducted daily and took approximately 15 minutes.

Target-based brain-control task

After completing the calibration, the 7D Sequence Task was used for BMI testing. Targets were selected randomly by the computer from a possible combination of 10 translation targets, 7 orientation targets, and 2 grasp targets. Endpoint velocity derived from the recorded firing rates were sent to the MPL controller which converted them to joint motor commands every 20 ms. As in calibration, a translation target was specified by an LED and then an orientation or grasp target was announced by computer-generated voice. The participant had control of all 7 dimensions (translation, orientation, and grasp) throughout the task.

In the early phases of the experiment, an adjustable amount of ortho-impedance was used in the testing phase. In addition, computer assistance was also provided to stabilize the hand’s position or orientation to facilitate learning.5, 6 Stabilizing control blended an ideal endpoint velocity control signal determined by the proportional MPL controller with the control signal derived from the participant’s brain activity. The relative contributions of each signal were specified by the experimenter. Stabilizing control was only applied to domains that were not actively being controlled. In other words, position was stabilized at a specified percentage (10–100%) during the orientation phase and/or orientation and grasp posture were stabilized during translation. Similarly, the experimenter could also specify the percentage of ortho-impedance. Both forms of computer assistance were adjusted subjectively to keep the participant engaged and motivated with the aim of maintaining a 60–80% success rate. It should be emphasized that after week 10 (Day 66), the subject performed all tasks with no computer assistance. A 10-second maximum movement time was enforced for each trial phase (translation or orientation/grasp).

Performance metrics including success rate, completion time, and path efficiency were calculated during these testing sessions. Success was determined based on MPL kinematics recorded in real-time. All 7 degrees-of-freedom were controlled continually and had to be correct during all three phases of movement in order for a trial to be considered a success. The experimenter could also manually mark successful (or failed) trials that might occur if the subject rotated too far in the specified direction or if mechanical limits of the MPL prevented movement along a single dimension. To calculate performance, chance levels were determined using random walk-neural signals as input to a simulation while using the assist parameters (if any) and target sizes used during the experiment. Two hundred simulations were run per trial and the median daily chance levels from these simulations are reported (figure 3A). For each trial, path efficiency (PE) was computed for each control domain using the following equation:

PE=LOT/LBCT Eq. 2

where LOT is the length of the optimal trajectory from the starting position to the end target location and LBCT is the length of the brain-controlled trajectory for a given trial. LBCT was determined by numerically integrating the velocity vector magnitude, V, over the duration of a single trial.

Figure 3. Summary of 7D brain-control performance.

Figure 3

(A) The solid black dots indicate the subject’s success rate on the 7D Sequence Task for each block (20 trials) of 7D brain-control training. Blocks that the subject performed with various levels of ortho-impedance or stabilizing assist are shown as open circles and squares respectively. After Day 66 post-implant, all reported performance data were collected using full brain-control with no computer assistance. The red dots indicate the median chance level for each day with the 95% confidence interval (5th and 95th percentiles from 200 simulations per trial) shown as error bars. (B) Normalized performance index for each day of 7D brain control. For each block of 20 trials, the success rate was normalized to the median chance level. Performance increased exponentially over time as described by the equation y = 1·812e0·04(x-32) (R2 = 0·114, p=0·001), where x is the number of days post-implant. For subplots C-E, each dot represents the mean block time or path efficiency for one block of 20 trials of the 7D sequence task completed by the participant. Linear fits to the participant’s data are shown as a red line. The mean block time or path efficiency of the MPL under auto-control is shown as a solid horizontal black line. (C) Block completion time, not including the presentation phase time, decreased linearly over the testing period (y = −0·632x + 171·4, R2 = 0·142, p<0·001). (D) Average 7D path efficiency of the MPL under brain-control increased linearly over time (y = 0·001x + 0·228, R2 = 0·160, p<0·001). (E) The greatest improvement in path efficiency occurred in the translation dimensions. Path efficiency for 3D translation during the 7D sequence task increased linearly over time (y = 0·003x + 0·317, R2 = 0·165, p<0·001). Only successful trials were included in the calculation of path efficiency. For all equations, x is the number of days post-implant.

Functional brain-control tasks

The participant used the MPL under full brain-control to perform 9 tasks (selected from 19 possible) on the Action Research Arm Test (ARAT), which is an evaluation of unilateral upper limb function used commonly in patients who have had a stroke.17 The ARAT is psychometrically sound and tracks the ability to perform activities of daily living (ADLs). The selected tests include all 6 items from the grasp subscale and 3 of the 4 items from the grip subscale (listed in table 1). Standard administration procedures18 were followed with a few modifications. One of the items was a ball, which was positioned inside a loose coil of wire to keep it from rolling off the table, although it was still possible for the subject to bump the ball hard enough to roll outside of the wire coil. The MPL hand started approximately 12 cm from the table surface, with the palm facing to the left instead of on the table. Each test item was timed and scored as 0 (no movement possible), 1 (task partially performed), 2 (task performed, but abnormally), or 3 (task performed normally). Movements that required more than 5 seconds to complete were scored as a 2. The subject attempted each evaluation 3 times and was instructed that only the best score counted. This kept her motivated and open to trying new strategies.

Table 1.

Action Research Arm Test (ARAT): Completion Time and Total Score

Completion Time (sec)

Item Day 80 Day 81 Day 84 Day 87 Day 88 Day 95 Day 98
Block, 10 cm cube 17·9 18·4 12·1 10·9 15·6 7·3
Block, 2.5 cm cube 15·0 16·1 12·7 10·2 67·0 8·1
Block, 5 cm cube 18·3 17·8 10·4 7·7 10·5 8·5 14·0
Block, 7·5 cm cube 15·6 19·8 12·8 11·2 8·0 16·1 9·0
Ball, 7·5 cm diameter 18·3 9·4 8·4 19·7 26·8 9·9
Stone, 10 cm x 2·5 cm x 1 cm 21·4 12·7 50·9 10·2 28·0 56·2
Pour water from glass to glass*
Tube, 2·5 cm x 16 cm 29·4 50·1 8·2 17·1 10·4 26·2
Tube, 1 cm x 16 cm 16·3 7.8 9·5 9·0 9·3

Average Time/Item (sec) 19·6 16·8 23·5 9·5 21·3 18·8 12·6

Total ARAT Score (out of a possible 27) 15 15 16 17 17 17 15
*

During the scored sessions, the participant was not able to fully complete the task that required pouring water from one glass to another and setting the glass back down on the table. Typically the water (simulated as a paper ball) would miss the cup, or the cup would tip over when it was set back down. The subject was able to complete the task twice in practice trials.

The participant also performed a cone-stacking task, which involved moving three plastic cones (30 cm tall, 4·1–6·4 cm diameter) from a base and stacking them sequentially on a second base located 30 cm away. The time to complete the task and the number of drops were recorded for each attempt. This test measured the subject’s ability to grasp, transport, and position objects with precision.

Role of the funding source

The sponsors of this study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all of the data and had final responsibility for the decision to submit for publication.

RESULTS

Neural recordings

The number of units, including single and multi-unit activity, recorded daily ranged from 209 to 271 (figure 2). There appeared to be an initial settling period where the number of units started off high until Day 21 post-implant and dropped in Week 4 (Day 24). In Week 4, the number of units began to increase at a rate of 0·368 units/day or 2·58 units/week. Figure 2 also shows that the number of units tuned to movement velocity with an R2>0·1 (Eq. 1) increased linearly over time at a rate of 0·388 units/day or 2·72 units/week.

Figure 2. Number of units over time.

Figure 2

The top set of data points (blue dots) indicates the number of units recorded during BMI sessions conducted 10–98 days post-implant. Starting at Day 24, the number of units increased linearly (y = 0·368x + 210·0, R2 = 0·356, p<0·001). The bottom set of data points (red squares) indicates the number of units tuned to movement velocity (Eq. 1, R2>0·1), which increased linearly over the duration of recording (y = 0·438x + 24·3, R2 = 0·155, p<0·01). For reference, 4D training began on Day 24 and 7D training began on Day 32.

Performance on target-based reaching tasks

Figure 3A shows the participant’s success rate for both computer-assisted and full brain control trials on the 7D Sequence Task over all 22 sessions of 7D testing, as well as the chance level determined for each day. Ortho-impedance and stabilizing assist were only used until week 10 (Day 66). No testing with the MPL was conducted on Days 52–58. The participant’s performance improved over the duration of training. During the last two weeks of training, with no computer assistance, the subject achieved an average success rate of 91·6±4·4%, compared to a median chance level of 6·2% calculated using simulations with random walk neural data as an input signal. Figure 3B shows the normalized performance index (success rate/chance level) over the duration of training. The participant’s performance increased over time even as the difficulty increased.

The participant also showed improvements in MPL control as evidenced by a decrease in block completion time (figure 3C) and an increase in path efficiency (figure 3D-E). Block completion time was reduced from a mean of 148 seconds to 112 seconds over the course of 7D training and also became less variable. Average path efficiency (figure 3D) increased as the participant moved along a straighter, more coordinated path. Mechanical constraints sometimes forced the MPL to deviate from an ideal linear trajectory, causing the maximum achievable efficiency under auto-control to be less than 1 (figure 3D). The greatest improvement in control appeared to be in the translation domain which increased linearly over time (figure 3E). An MPL kinematics sample of four consecutive trials is shown in figure 4. These plots show that the participant was able to rapidly reach the target position in the specified control domain (translation, orientation, or grasp) while maintaining her position in the other two domains (webvideo 1).

Figure 4. MPL position, orientation, and grasp aperture during four 7D Sequence Task trials under full brain control.

Figure 4

A thick black horizontal bar denotes whether a translation, orientation, or grasp target was being attempted although the participant had control of all 7 dimensions at all times. Each new translation target indicates the start of a new trial, which is also marked with an arrow along the time axis. MPL kinematics as controlled by the participant are shown as solid lines. The target position for each dimension is shown as a dotted line. Grey shaded regions indicate presentation phases in which the MPL was paused and the subject was listening to a computer-generated verbal command. A grasp aperture of 1 indicates that the hand was fully closed. The participant was successful in maintaining position in one control domain while changing position in another, as instructed. Webvideo 1 shows the subject’s performance with the MPL in these 4 trials.

Neural changes were also observed during the 7D brain control testing period. The percentage of units tuned to 7D MPL velocity (Eq. 1) showed a linear increase (y = 0·002x + 0·100, R2 = 0·119, p=0·066) as shown in figure 5A. The percentage of units for R2 values >0·1 increased during the late training phase (Weeks 11–14) compared to the early phase (Weeks 5–8). The median R2 value was significantly larger during late training than during early training (p<0.001, Wilcoxon rank-sum test). The percentage of units with R2 values of 0·15–0·30 showed the greatest increase (figure 5B). This may suggest that the participant was learning to modulate her brain activity in order to better fit the neural encoding model to improve performance.19

Figure 5. Changes in neural tuning over time.

Figure 5

(A) Fraction of units whose firing rate predicted 7D MPL endpoint velocity (Eq. 1) with an R2>0·10 on each day of 7D brain-control training. Each dot represents data from a single decoder. On a single day decoders were trained using observation data and brain-control data with ortho-impedance. The percentage of units tuned to 7D MPL velocity trended towards a linear increase over the training period (y = 0·002x + 0·100, R2 = 0·119, p=0·066). (B) Percentage of units significantly tuned to MPL kinematics for early training (Weeks 5–8, Blue) of 7D BMI training compared to late training (Weeks 11–14, Red). Each bar is centered between the upper and lower bound of R2 values for a given bin. All R2>0·5 were combined into a single bin. Over time, the percentage of units with an R2<0·1 decreased, while the percentage of units tuned to kinematics with 0·1≤R2≤0·5 increased.

Clinical assessments

Table 1 shows the completion time, as well as the total score, for each test item on each day that the ARAT was completed. If no time is listed, the score for that item was a 1 since the task was partially completed. For items with completion times listed, the score was a 2 because it took longer than 5 seconds (equivalent to the approximate amount of time it would take an unimpaired adult to complete the task18). The participant demonstrated consistent performance as her total score ranged from 15–17 across 7 testing days. The subject was unable to move her own upper limbs volitionally and therefore scored a 0 without the use of the BMI or other assistive technology. Webvideo 2 shows her ARAT performance on Day 87. The movements were generally smooth, direct, and fast. The subject reported that she was thinking about the goal of the action, such as “grab the block,” rather than issuing specific kinematic commands to the MPL endpoint, suggesting that the control was intuitive.

Table 2 shows the cone-stacking task completion time and the number of drops for each day of testing (See also webvideo 3). Dropped cones were repositioned by the experimenter. The fastest performance was achieved on the last day of testing, although the participant had more drops than on Day 87. In this case, she knocked the first cone off the base 6 times before quickly moving all three cones, with only one additional drop. The participant was able to complete this task consistently on all 4 days of testing.

Table 2.

Performance on 3 Cone-stacking Task

Day 81 Day 87 Day 88 Day 95
Time (sec) Drops Time (sec) Drops Time (sec) Drops Time (sec) Drops
Right to left 214 5 140 1 223 11 126 7
Left to right 96 2 123 1 84 1 87 2

DISCUSSION

This study shows that an individual with tetraplegia rapidly learned to reach and grasp using coordinated 7D control of a high performance anthropomorphic prosthetic limb. Performance on reaching and grasping tasks consistently improved over the course of 34 training sessions completed over 13 weeks. The steady improvement demonstrates a robust approach and that the participant was consistently learning. In this context, learning is defined as the modification of the correspondence between neural activity and action. Although the mechanism of this modification is unknown, the clear and consistent change in neural tuning is a hallmark of the BMI paradigm1922 and our results show that the neural population became more tuned to MPL movement velocity over time. The use of neural extraction algorithms derived from natural movement along with an anthropomorphic effector likely promoted the acquisition of the skill necessary to achieve the performance demonstrated here. This type of BMI control is model-based, in contrast to classifiers which operate on pre-defined libraries of discrete input-output categories. With our approach, continuous 3D translation of the hand was achieved with a few hours of training, and 7D control was performed routinely in less than 4 months. The movements were smooth, coordinated and skillful. The subject had the flexibility to explore a spectrum of arm/hand configurations to complete tasks with different strategies (webvideo 4).

Non-human primate studies established the observation-based calibration, computer assist methods, and the extraction algorithms used in the current study.5, 6 Previous human studies have demonstrated that intracortical electrodes could be used to provide rudimentary control of computer cursors23 and robotic manipulators.12 Our study builds upon this work by adding degrees of freedom which allow for more natural and coordinated movements, a broader workspace, and the ability to interact with a variety of objects in different orientations (webvideo 4). Across multiple days of testing, the participant was able to improve her ARAT score from 0 to 15–17 (out of 27). This achievement far exceeds the threshold of 5·7 points for a clinically significant improvement in function21, demonstrating that use of a neuroprosthetic device has the potential to provide clinically significant functional benefit. As shown in webvideo 2, movements were smooth and coordinated, with speeds approaching that of able-bodied adults. Since we used a velocity-based decoding model, the MPL speed was controlled directly by the participant. The level of control achieved by the participant allowed for subtle corrective movements and object manipulation (webvideo 3). ARAT testing began once the participant had achieved consistent 7D control on the target-based task and her performance on clinical measures was consistent during the last three weeks of testing.

We expect a number of further developments in BMI technology in the near future. A wider range of tasks will become possible by adding hand shape to the repertoire of volitional control. Tactile feedback will be transduced by fingertip sensors and transmitted to sensory cortex. Telemetry will eliminate the need for transcutaneous leads and connectors. Paralyzed arms will be reanimated by activating the subject’s own muscles, which we know is a high priority for individuals with tetraplegia.6,7,24 Our current results and the very rapid developments in this field show the potential to restore much of the function lost to individuals with tetraplegia or upper limb amputation, allowing them to again perform natural behaviors to more fully interact with the world around them.

PANEL: RESEARCH IN CONTEXT

Systematic Review

A literature review was conducted in PubMed without date or language restriction using the search terms “neuroprosthetics,” “brain-machine interface,” or “brain-computer interface.” Since our goal is to restore upper limb function, we restricted our survey to studies using robotic technology or functional electrical stimulation to achieve this same goal. In addition, the authors drew from their own substantive reference libraries. Two published reports of robotic arm12 and hand25 control in human subjects with motor impairments were identified. One non-human primate study reported 4D control of a robotic arm to perform a self-feeding task5, while two have investigated functional electrical stimulation of the upper limb.7, 8 For some individuals with amputation or residual muscle activity, alternative prosthetic control techniques including non-invasive approaches26, targeted-reinnervation27, or direct peripheral neural control28, 29 may be appropriate.

Interpretation

Previous work has shown that it is possible to record neural activity from motor cortex and translate that activity to movement of an external device or one’s own muscles. However, to date, no human studies have shown that natural and complex movements can be performed consistently for a variety of tasks. Here, we have demonstrated that a person with chronic tetraplegia can perform complex and coordinated movements freely in 7D space consistently over multiple weeks of testing. This is novel because previous studies have limited control to translation dimensions, used staged control schemes, or limited the workspace to complete very structured tasks. Increasing control dimensionality allows our participant to fully explore the workspace by placing the hand in the desired 3D location and orienting the palm in three dimensions. This is also the first time that performance has been quantified using functional clinical evaluations. While most human studies rely on only a few days of performance data, we have shown that the participant learned to improve her performance consistently over many days using a variety of metrics. By using training methods and algorithms validated in non-human primate work, individuals with long term paralysis can recover the natural and intuitive command signals for hand placement, orientation, and reaching to move freely in space and interact with the environment.

Supplementary Material

01
02
Download video file (1.7MB, mp4)
03
Download video file (4.3MB, mp4)
04
Download video file (2.6MB, mp4)
05
Download video file (1.2MB, mp4)

Acknowledgments

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) Revolutionizing Prosthetics program contract number N66001–10-C-4056, the National Institutes of Health (NIH) grant 8KL2TR000146–07, the Office of Research and Development, Rehabilitation Research & Development Service, Department of Veterans Affairs (Grants # B6789C, B7143R, and RX720), and the UPMC Rehabilitation Institute. The views expressed herein are those of the authors and do not reflect the official policy or position of the Department of Veterans Affairs, Department of Defense, or the United States government.

We thank Jan Scheuermann for her extraordinary commitment and effort related to this study as well as insightful discussions with the study team. We also thank Karina Palko for her participation as an honorary research team member and support of the study. We thank clinicians and researchers at the University of Pittsburgh and UPMC including Dr. Joseph Ricker (Physical Medicine and Rehabilitation, PM&R) for conducting presurgical neuropsychological screening, Dr. Richard Barbara (PM&R) for providing psychological monitoring throughout the study, Dr. Ferenc Gyulai (Anesthesiology) for directing general anesthesia for the implantation surgery, and Dr. Elizabeth Skidmore (Occupational Therapy) for advice related to functional assessments using the prosthetic limb. We thank Drs. Ramana Vinjamuri and Robin Ashmore (PM&R) for technical and software development efforts, Dr. Stephen Foldes (PM&R) for assistance with presurgical neuroimaging, Dr. Elke Brown (PM&R) for assistance with BMI testing sessions, Dr. Samuel Clanton (School of Medicine) who developed a prototype-system for high-dimensional decoding and subject training that was adapted for this experiment, Ms. Elizabeth Harchick (PM&R) for assistance with presurgical neuroimaging, recruitment, and regulatory compliance, and Ms. Alisha Shane-Cuniff (PM&R) for coordination of the Data and Safety Monitoring Board (DSMB). We thank the University of Pittsburgh Clinical and Translational Science Institute as well as the Office of Investigator-Sponsored IND and IDE support for assistance with protocol development and regulatory reporting and compliance. We thank the volunteer members of the DSMB for their ongoing monitoring of this study. Finally we thank the Johns Hopkins University Applied Physics Laboratory and Blackrock Microsystems, Inc. for coordination efforts and technical support related to this project.

Footnotes

Contributors

JC, WW, ET-K, DW, MB, and AS designed the study. JC, BW, and JD completed most of the data collection and analysis with all authors contributing to development of analysis techniques and interpretation of results. ET-K developed and performed the surgical implantation procedures. BW, JD, AM, and MV developed the software required for the real-time experiments. JC, MB, and AS wrote the manuscript and all authors provided critical review and approval of the manuscript.

Conflict of interest statement

MV and AS have a patent application pending that covers some of the methodology used in this study. We declare that we have no other conflicts of interest.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Jennifer L Collinger, Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA. Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA.

Brian Wodlinger, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA.

John E Downey, Center for the Neural Basis of Cognition, Pittsburgh, PA, USA.

Wei Wang, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA. Clinical and Translational Neuroscience Institute, University of Pittsburgh, Pittsburgh, PA, USA.

Elizabeth C Tyler-Kabara, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. Department of Neurological Surgery, University of Pittsburgh, Pittsburgh, PA, USA. McGowan Institute for Regenerative Medicine, University of Pittsburgh, Pittsburgh, PA, USA.

Douglas J Weber, Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA. Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA.

Angus JC McMorland, Systems Neuroscience Institute, University of Pittsburgh, Pittsburgh, PA, USA.

Meel Velliste, Systems Neuroscience Institute, University of Pittsburgh, Pittsburgh, PA, USA. Department of Neurobiology, University of Pittsburgh, Pittsburgh, PA.

Michael L Boninger, Department of Veterans Affairs Medical Center, Pittsburgh, PA, USA. Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. McGowan Institute for Regenerative Medicine, University of Pittsburgh, Pittsburgh, PA, USA.

Andrew B Schwartz, Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, PA, USA. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA. McGowan Institute for Regenerative Medicine, University of Pittsburgh, Pittsburgh, PA, USA. Systems Neuroscience Institute, University of Pittsburgh, Pittsburgh, PA, USA. Department of Neurobiology, University of Pittsburgh, Pittsburgh, PA.

References

  • 1.Santello M, Flanders M, Soechting JF. Postural hand synergies for tool use. J Neurosci. 1998;18:10105–15. doi: 10.1523/JNEUROSCI.18-23-10105.1998. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Soechting JF, Ross B. Psychophysical determination of coordinate representation of human arm orientation. Neuroscience. 1984;13:595–604. doi: 10.1016/0306-4522(84)90252-5. [DOI] [PubMed] [Google Scholar]
  • 3.Desmurget M, Prablanc C, Rossetti Y, et al. Postural and synergic control for three-dimensional movements of reaching and grasping. J Neurophysiol. 1995;74:905–10. doi: 10.1152/jn.1995.74.2.905. [DOI] [PubMed] [Google Scholar]
  • 4.Moran DW, Schwartz AB. Motor cortical representation of speed and direction during reaching. J Neurophysiol. 1999;82:2676–92. doi: 10.1152/jn.1999.82.5.2676. [DOI] [PubMed] [Google Scholar]
  • 5.Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB. Cortical control of a prosthetic arm for self-feeding. Nature. 2008;453:1098–101. doi: 10.1038/nature06996. [DOI] [PubMed] [Google Scholar]
  • 6.Clanton ST. PhD Thesis. Pittsburgh: Carnegie Mellon University; 2011. Brain-computer interface control of an anthropomorphic robotic arm. [Google Scholar]
  • 7.Moritz CT, Perlmutter SI, Fetz EE. Direct control of paralysed muscles by cortical neurons. Nature. 2008;456:639–42. doi: 10.1038/nature07418. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Ethier C, Oby ER, Bauman MJ, Miller LE. Restoration of grasp following paralysis through brain-controlled stimulation of muscles. Nature. 2012:1–4. doi: 10.1038/nature10987. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Wang W, Collinger JL, Degenhart AD, et al. An electrocorticographic brain interface in an individual with tetraplegia. University of Pittsburgh; 2012. (unpublished work, in review) [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.McFarland DJ, Sarnacki WA, Wolpaw JR. Electroencephalographic (EEG) control of three-dimensional movement. J Neural Eng. 2010;7:036007. doi: 10.1088/1741-2560/7/3/036007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Doud AJ, Lucas JP, Pisansky MT, He B. Continuous three-dimensional control of a virtual helicopter using a motor imagery based brain-computer interface. PLoS One. 2011;6:e26322. doi: 10.1371/journal.pone.0026322. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Hochberg LR, Bacher D, Jarosiewicz B, et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature. 2012;485:372–5. doi: 10.1038/nature11076. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Kendall FP, McCreary EK, Provance PG. Muscles: Testing and Function. Philadelphia, PA: Lippencott Williams & Wilkins; 1993. [Google Scholar]
  • 14.Georgopoulos AP, Schwartz AB, Kettner RE. Neuronal population coding of movement direction. Science. 1986;233:1416–9. doi: 10.1126/science.3749885. [DOI] [PubMed] [Google Scholar]
  • 15.Wang W, Chan SS, Heldman DA, Moran DW. Motor cortical representation of position and velocity during reaching. J Neurophysiol. 2007;97:4258–70. doi: 10.1152/jn.01180.2006. [DOI] [PubMed] [Google Scholar]
  • 16.Marquardt DW. Generalized Inverses, Ridge Regression, Biased Linear Estimation, and Nonlinear Estimation. Technometrics. 1970;12:591–612. [Google Scholar]
  • 17.Lyle RC. A performance test for assessment of upper limb function in physical rehabilitation treatment and research. Int J Rehabil Res. 1981;4:483–92. doi: 10.1097/00004356-198112000-00001. [DOI] [PubMed] [Google Scholar]
  • 18.Yozbatiran N, Der-Yeghiaian L, Cramer SC. A standardized approach to performing the action research arm test. Neurorehab Neural Repair. 2008;22:78–90. doi: 10.1177/1545968307305353. [DOI] [PubMed] [Google Scholar]
  • 19.Taylor DM, Tillery SI, Schwartz AB. Direct cortical control of 3D neuroprosthetic devices. Science. 2002;296:1829–32. doi: 10.1126/science.1070291. [DOI] [PubMed] [Google Scholar]
  • 20.Jarosiewicz B, Chase SM, Fraser GW, Velliste M, Kass RE, Schwartz AB. Functional network reorganization during learning in a brain-computer interface paradigm. Proc Natl Acad Sci U S A. 2008;105:19486–91. doi: 10.1073/pnas.0808113105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Ganguly K, Carmena JM. Neural correlates of skill acquisition with a cortical brain-machine interface. J Mot Behav. 2010;42:355–60. doi: 10.1080/00222895.2010.526457. [DOI] [PubMed] [Google Scholar]
  • 22.Ganguly K, Dimitrov DF, Wallis JD, Carmena JM. Reversible large-scale modification of cortical networks during neuroprosthetic control. Nat Neurosci. 2011;14:662–7. doi: 10.1038/nn.2797. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Simeral JD, Kim S-P, Black MJ, Donoghue JP, Hochberg LR. Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. J Neural Eng. 2011;8:025027. doi: 10.1088/1741-2560/8/2/025027. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Collinger JL, Boninger ML, Bruns TM, Curley K, Wang W, Weber DJ. Functional Priorities, Assistive Technology, and Brain-Computer Interfaces after Spinal Cord Injury. J Rehabil Res Dev. doi: 10.1682/jrrd.2011.11.0213. In Press. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Yanagisawa T, Hirata M, Saitoh Y, et al. Real-time control of a prosthetic hand using human electrocorticography signals. J Neurosurg. 2011;114:1715–22. doi: 10.3171/2011.1.JNS101421. [DOI] [PubMed] [Google Scholar]
  • 26.Losier Y, Englehart K, Hudgins B. Evaluation of shoulder complex motion-based input strategies for endpoint prosthetic-limb control using dual-task paradigm. J Rehabil Res Dev. 2011;48:669–78. doi: 10.1682/jrrd.2010.08.0165. [DOI] [PubMed] [Google Scholar]
  • 27.Kuiken TA, Li G, Lock BA, et al. Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. JAMA. 2009;301:619–28. doi: 10.1001/jama.2009.116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Benvenuto A, Raspopovic S, Hoffmann KP, et al. Intrafascicular thin-film multichannel electrodes for sensory feedback: Evidences on a human amputee. Conf Proc IEEE Eng Med Biol Soc. 2010;2010:1800–3. doi: 10.1109/IEMBS.2010.5626401. [DOI] [PubMed] [Google Scholar]
  • 29.Dhillon GS, Horch KW. Direct neural sensory feedback and control of a prosthetic arm. IEEE Trans Neural Syst Rehabil Eng. 2005;13:468–72. doi: 10.1109/TNSRE.2005.856072. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

01
02
Download video file (1.7MB, mp4)
03
Download video file (4.3MB, mp4)
04
Download video file (2.6MB, mp4)
05
Download video file (1.2MB, mp4)

RESOURCES