Skip to main content
PMC home page
Clinical and Translational Science logoLink to Clinical and Translational Science
. 2013 Aug 27;7(1):52–59. doi: 10.1111/cts.12086

Collaborative Approach in the Development of High‐Performance Brain–Computer Interfaces for a Neuroprosthetic Arm: Translation from Animal Models to Human Control

Jennifer L Collinger 1,2,3, Michael A Kryger 2, Richard Barbara 2,†, Timothy Betler 4,†, Kristen Bowsher 5,†, Elke H P Brown 2,†, Samuel T Clanton 2,†, Alan D Degenhart 3,6,†, Stephen T Foldes 1,2,6,†, Robert A Gaunt 2,3,†, Ferenc E Gyulai 7,†, Elizabeth A Harchick 2,†, Deborah Harrington 2,†, John B Helder 8,†, Timothy Hemmes †, Matthew S Johannes 8,†, Kapil D Katyal 8,†, Geoffrey S F Ling 9,†, Angus J C McMorland 10,11,†, Karina Palko †, Matthew P Para 8,†, Janet Scheuermann †, Andrew B Schwartz 2,3,6,10,11,12,†, Elizabeth R Skidmore 2,13,†, Florian Solzbacher 14,15,16,†, Anita V Srikameswaran 4,†, Dennis P Swanson 17,†, Scott Swetz 8,†, Elizabeth C Tyler‐Kabara 2,3,12,18,†, Meel Velliste 10,11,†, Wei Wang 2,3,6,19,†, Douglas J Weber 1,2,3,6,†, Brian Wodlinger 2,6,†, Michael L Boninger 1,2,3,12,✉
PMCID: PMC3929226  NIHMSID: NIHMS502390  PMID: 24528900

Abstract

Our research group recently demonstrated that a person with tetraplegia could use a brain–computer interface (BCI) to control a sophisticated anthropomorphic robotic arm with skill and speed approaching that of an able‐bodied person. This multiyear study exemplifies important principles in translating research from foundational theory and animal experiments into a clinical study. We present a roadmap that may serve as an example for other areas of clinical device research as well as an update on study results. Prior to conducting a multiyear clinical trial, years of animal research preceded BCI testing in an epilepsy monitoring unit, and then in a short‐term (28 days) clinical investigation. Scientists and engineers developed the necessary robotic and surgical hardware, software environment, data analysis techniques, and training paradigms. Coordination among researchers, funding institutes, and regulatory bodies ensured that the study would provide valuable scientific information in a safe environment for the study participant. Finally, clinicians from neurosurgery, anesthesiology, physiatry, psychology, and occupational therapy all worked in a multidisciplinary team along with the other researchers to conduct a multiyear BCI clinical study. This teamwork and coordination can be used as a model for others attempting to translate basic science into real‐world clinical situations.

Keywords: brain, clinical trials, methodology, translational research

Introduction

Brain–computer interfaces (BCIs) are systems that allow signals from the cerebral cortex to be recorded and utilized by a computer system for the purpose of controlling assistive technology. More than 40 years of basic science research by neurophysiologists and biomedical engineers have advanced the understanding of motor cortex physiology and the idea of neural control in animal models.1, 2, 3, 4, 5, 6, 7, 8, 9 This has enabled nonhuman primates to control computer cursors,10 functional electrical stimulation of forearm muscles for grasping,11 and a robotic arm to perform self‐feeding.12 Motor BCIs have the potential to assist veterans arriving home with disabling injuries and lost limbs, as well as the thousands of individuals who acquire spinal cord injuries or lose limbs through accidents or neurologic disorders.13, 14, 15 However it is a complex and expensive mission to bring all of the tools and professionals together to translate BCI technology to clinical application. As a result, there have been very few clinical trials with implanted BCIs in people with motor impairments.16, 17, 18, 19, 20, 21, 22

At the University of Pittsburgh, our focus is on upper limb motor neuroprostheses, which aim to restore movement and function. We are interested in developing a BCI to provide intuitive and natural control of a robotic upper limb. We have followed a multistep approach to move from basic science research to long‐term clinical trials of implanted BCI technology. To date, we have demonstrated three degrees of freedom (DOF) control using electrocorticography (ECoG) in a short‐term study (<30 days) as well as seven DOF control of an anthropomorphic sophisticated prosthetic arm using intracortical microelectrode arrays (MEAs).19, 20 In this manuscript, we describe a roadmap of resources and procedures that we believe are pertinent to clinical trials of implanted neuroprosthetic devices. Larger, multisite investigations and improvements in this technology will be necessary in order to translate this technology to individuals with disabilities.

Summary of Translational Approach

In ECoG studies, initial human investigations, including real‐time BCI and neuroscience experiments, were conducted in patients being observed in an epilepsy monitoring unit (EMU).23, 24, 25 ECoG during epilepsy monitoring has been used by a number of research groups to enable the development of software and training techniques with human participants.26, 27, 28, 29, 30, 31 Following the studies in epilepsy patients, the next step in our BCI research pathway was to conduct a 28‐day study of ECoG‐based BCI in an individual with spinal cord injury.20 This represented a critical translation of BCI technology into the targeted patient population. For the MEA investigation, we obtained Investigational Device Exemption (IDE) approval from the Food and Drug Administration (FDA) to conduct a long‐term (<5 years) study in individuals with tetraplegia. This multiyear study required substantial collaboration with the government, that not only provided funding, but also launched a vast collaborative effort that brought together institutions across the country that have developed advanced robotic technology and neural recording devices.

This paper aims to describe key elements of our translational research project, specifically focused on the collaboration between various teams and disciplines. This will include a discussion of the foundational elements that led to the multiyear intracortical BCI study, including animal models, the Modular Prosthetic Limb (MPL), EMU research, and the 28‐day ECoG Study. We will then describe the various components necessary in translating this work for the MEA BCI study, including funding, IDE preparation, hurdles in translation, and participant recruitment. Next we detail clinical aspects of the study, highlighting considerations in imaging, surgery, anesthesia, and psychiatry. The paper then summarizes the experimental trials, including the virtual environment, training, and outcome measurements. We conclude with the participants’ perspectives, and results from the MEA BCI study.

Foundational Studies

Nonhuman primate investigations of neural control

The science behind prosthetic control originated from the seminal findings of directional tuning2 and population coding,3 which elucidated the relationship between neuronal firing and the direction of limb movements. Since then, three decades of monkey experiments have paved the way for human trials, including studies on continuous trajectory decoding,10, 32 brain‐controlled reaching in a virtual environment,4 and a self‐feeding experiment12 in which a monkey learned to reach and grasp using a robotic arm in four DOF. The specific methodology, including algorithms, real‐time software, and training protocols culminated in 2010 with a PhD thesis33 that demonstrated for the first time seven DOF robotic control in a nonhuman primate model by adding three DOF control of hand orientation. This work included a system for calibration and training of nonhuman primates using shared control, a training process by which the subject is progressively given more control of a robotic arm beginning with calibration from observed robotic arm movements.19, 33 These animal studies demonstrated that intuitive control using cortical signals was possible in real time, and formed the foundation for an effort to translate this technology into human subjects.

ECoG studies in the EMU

Developing experience with implanted neural recording in human subjects was crucial to improving our scientific understanding and refining our protocols. An ideal patient population is one that is undergoing ECoG monitoring for clinical brain mapping in EMUs. Our EMU studies set the foundation for the BCI work in many ways. Because of its multidisciplinary nature, the EMU work accelerated the integration of team members with diverse backgrounds, allowing us to design experiments and solve problems by considering future research questions, technical capabilities and constraints, and clinical practicality. This work guided development of subsequent human study protocols and IDE applications. Finally, the EMU study generated critical pilot data23, 24, 25, 34, 35 and initial funding from multiple agencies that made it possible for us to form and grow our clinical BCI research team with a demonstrated track record. The EMU research conducted in patients undergoing clinical brain mapping in itself has important scientific, engineering, and clinical value as exemplified by the ever‐increasing number of research groups and publications based on this study setting.23, 24, 25, 26, 27, 28, 29, 30, 31, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43

Short‐term ECoG study in spinal cord injury

ECoG studies in clinical patients,39, 44 as well as nonhuman primate investigations of ECoG,45, 46 provided a foundation for translating ECoG into the targeted patient population. In August 2011, with institutional support from UPMC and the University of Pittsburgh, and funding from National Institutes of Health (NIH) and the Craig Neilsen Foundation, a 30‐year‐old male with tetraplegia was implanted with a high‐density ECoG grid over sensorimotor cortex for 28 days.20 The participant was able to move a cursor toward a target in two dimensions with an 87% success rate. The participant also attempted 3D control, and steadily improved performance up to 80% after only a few days. While significant in its own right, this short‐term ECoG study also allowed us to pilot and refine a number of the principles and technologies required for longer term investigations, including presurgical mapping, software development, and training strategies. This stepping stone also provided evidence to encourage further funding support and attract future study participants. Most importantly, it provided the team with experience working with a participant with paralysis, which required us to make accommodations related to accessibility, necessitated the use of training strategies that did not require overt movement, and allowed us to experience the amount of motivation and dedication required from successful study participants.

Regulatory and technical preparations

DARPA revolutionizing prosthetics program

The long‐term MEA study was primarily funded by the Defense Advanced Research Projects Agency (DARPA), which through a primary contract with Johns Hopkins University Applied Physics Laboratory (JHU/APL), brought Blackrock Microsystems (Salt Lake City, UT, USA) and their relationship with the FDA to the team. As part of the Revolutionizing Prosthetics program, DARPA funded two research teams at the DEKA Research and Development Corporation and JHU/APL to design and build advanced upper extremity prostheses that enabled near‐natural movement.47, 48 The DARPA team brought together hundreds of professionals across the country. DARPA also assisted by encouraging support through the NIH and Veterans Administration, both of whom contributed to this study through resources and grants.

Modular prosthetic limb (MPL)

The training paradigm discovered in the animal work required observation of movement.12 However, available prosthetics were not capable of producing high‐dimensional, natural movements. The MPL (Figure 1) is an advanced upper‐limb prosthetic designed to mimic the capabilities, form factor, and function of the human arm and hand.49 The MPL emulates the weight, volumetric envelope, speed, torque, and range of motion abilities of its human counterpart, with 17‐controllable DOF and 26 articulating joints. This anthropomorphic design and appearance of the MPL provides lifelike movement, and may convey a feeling of embodiment to users.

Figure 1.

Figure 1

Open in a new tab

JHU/APL modular prosthetic limb (MPL).

The MPL team at JHU/APL collaborated with the University of Pittsburgh team through various developmental planning, system modification and updating, prototyping, and support activities. Initial collaboration focused on defining the user specifications for the MPL as well as system integration within the experimental framework. This primarily focused on physical, electrical, and logical integration of the system logistically within the clinical environment. Software and control parameters of the MPL system were optimized to participant‐decoded intent signals during the experiments. Much of this optimization was conducted through nonhuman primate testing, which facilitated quick implementation during the clinical trial. Safety features including limited velocity regions and customizable workspace exclusion were developed and rigorously tested to allow for experiments involving interaction between the participant and the MPL. Frequent practice sessions with the BCI are critical to experiment success, and thus minimizing hardware and software malfunctions was essential. This was achieved through quick turnaround repair efforts and hardware swapping with backup assemblies.

Virtual integration environment (VIE)

To allow offline training by the participant without the use of the physical MPL, a virtual reality framework called the VIE was developed by JHU/APL. The VIE visually and functionally simulates the MPL and provides a surrounding environment filled with everyday objects such as cups, books, and tables. This virtual MPL utilizes the same control interface as the physical MPL, allowing it to be used alongside or instead of the MPL during neural training.50, 51 Because the VIE system is executed on a standard personal computer and requires minimal support, it was usable both in the laboratory and in the participant's home.

The VIE (Figure 2 and Video 1) was built using Unity3D (Unity Technologies, San Francisco, CA, USA), a game development environment, which allowed for rapid development of new scenarios that incorporated virtual objects with properties similar to their real counterparts. Custom objects that closely match those in real life were developed including those used in tests of upper limb function.52 Participant‐requested games that were both engaging to the participant and complementary to the training paradigm were also developed. The games allowed for longitudinal “scoring” and evaluation of the subject's performance using quantitative metrics.

Figure 2.

Figure 2

Open in a new tab

Example scene in the Virtual Integration Environment (VIE).

Regulatory process

Clinical trials of significant risk devices not cleared for marketing require the submission of an IDE and prospective approval by the FDA.53 Institutional Review Board (IRB) approval must also be obtained for the corresponding clinical investigation. Many neural recording electrodes have premarket approval or 501 (k) clearance from the FDA with specific indications for use and allowable duration of use. While this process allows for leveraging existing FDA cleared devices without requiring as robust a statistical analysis of safety and efficacy data, an IDE must still be submitted in order to use devices for nonapproved indications. All IDEs have three main components: a report of prior research, an investigational plan, and a device description. Applications are evaluated to ensure that the device meets appropriate manufacturing specifications, the investigational plan is sound, the device has the potential to be effective, and that the risks of the study do not outweigh the potential benefit to participants or knowledge to be gained. In our case, prior research included laboratory, animal, and clinical testing of the intracortical MEA, which address safety and efficacy of the device. The investigational plan made up the majority of the document and included the purpose of the investigation, a detailed description of the device, the clinical protocol, informed consent documentation, and a risk analysis. Creating a clinical protocol and IDE that maximized the knowledge to be gained, while minimizing risks to participants required the team's diverse expertise. The device description detailed the safety measures associated with the manufacturing process, including methods, facilities, manufacturing, processing, packing, and storage. Detailing device information required close cooperation with an industry partner, Blackrock Microsystems, who supplied the 510k‐cleared MEA (Figure 3). They provided written permission to the FDA to access their 510 (k) application for manufacturing information in support of our IDE application. Without cooperation from the device company, independent bench and preclinical testing of the device would have been required.

Figure 3.

Figure 3

Open in a new tab

Neuroport intracortical microelectrode array (MEA). Used with permission from Blackrock Microsystems.

Important to our success was the close working relationship between the FDA and DARPA that began in 2006. In 2011, the FDA selected the Revolutionizing Prosthetics program as the pilot program for the FDA Innovation Initiative. The FDA established a transparent and open communication channel that encouraged discussion of requirements to gain approval for technologies and processes related to human trials. As a result of this initiative, our team was able to obtain regulatory approvals efficiently without the need for multiple rounds of submission. Further, the University of Pittsburgh Office for Investigator‐Sponsored Investigational New Drug and IDE Support provided presubmission reviews and assistance with the submission process. In parallel with FDA discussions, our team held regular meetings with the University of Pittsburgh IRB and members of an independent Data and Safety Monitoring Board. Thus, long before the documentation was submitted, the regulatory personnel provided feedback on the protocol, again reducing the number of submissions. This early involvement was facilitated by the Clinical and Translational Science Institute at the University of Pittsburgh, which is an NIH‐funded effort to improve the efficiency with which biomedical advances are translated to patients.

Leveraging basic science resources

Nonhuman primate studies of a BCI‐controlled MPL were conducted prior to beginning the clinical trial.12, 33 As previously mentioned, these studies formed the scientific basis for the training paradigms used in the human trials and also provided an estimate of the number of neural units needed for high‐dimensional control of the MPL. The human BCI software system utilizes the Real Time Messaging Architecture developed for earlier nonhuman primate experiments. The software was modified to provide extra flexibility in paradigm design allowing the participant to complete a larger variety of tasks with and without the MPL. This proved crucial as we routinely implemented new tests and training exercises throughout the study. We also used a flexible graphical user interface to tune computer‐assist parameters12, 19 throughout each test session in order to keep the participant motivated.

Translating this work to humans revealed several challenges not observed in the nonhuman primate work. First, it was discovered that the decoding algorithm needed to be more robust to provide good control over our longer testing sessions, especially after meals or other breaks. Most nonhuman primate experiments last 1–2 hours, while our human sessions were typically 4 hours long. Second, we found that BCI performance on the training task did not always generalize to good performance over a larger variety of tasks. These challenges were addressed by adding a regularization stage to the decoding algorithm54 and modifying our training paradigm to include object interactions. The regularization stage helped prevent over‐fitting to noise and other signals that improve performance only slightly but are likely to change over time. These changes not only improved the quality of our results but made testing sessions easier and more enjoyable for both participants and researchers.

Participant recruitment

At the core of any successful study are motivated individuals willing to serve as study participants. Given the very limited number of participants to be tested, their efforts have the power to determine the success of the study. Recruiting participants for clinical trials of neuroprosthetic devices can be challenging. Study participation requires a great commitment, as participants are asked to undergo voluntary neurosurgery and a demanding testing schedule. Strict inclusion/exclusion criteria as defined in the IDE, while necessary to ensure participant safety, limit the recruitment pool significantly. It is essential that the research team and potential participants have an open dialog and that the participants are fully informed about the potential risks associated with the study. For this particular study, participants were told that the BCI was a temporary implant and that they would derive no direct benefit from the study.

Three primary methods of recruitment were used. First, research registries allowed us to contact individuals who met our criteria and had expressed an interest in research. Second, clinicians were asked to refer participants. Finally, we encouraged media coverage of our studies, in particular the short‐term ECoG study, to educate and inform the public about the science and technology behind the BCI projects. Media Relations chronicled study events with video of the testing procedures and interviews of the participants and the research team. That video was used by reporters from national and local media outlets and was also used in a story package that was posted with other press materials on the UPMC Website (www.upmc.com/bci). Media outreach conducted in collaboration with the journal press office and a Webcast press conference led to global coverage of the participant and research team's achievements. Since then, multiple potential candidates have contacted the trial coordinator for follow‐up studies.

Clinical Considerations

Preoperative imaging

In order to account for functional reorganization secondary to chronic paralysis, it was important that electrode placement target cortical areas active for individual participants.55 We utilized functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). FMRI evaluates changes in cerebral perfusion with high spatial resolution. The blood‐oxygenation level dependent (BOLD) response is an indirect measurement of neural activity that has been widely used for mapping motor activity. MEG records changes in magnetic fields resulting from neuronal firing, with high temporal resolution. MEG mapping complemented the more standard fMRI results as it is a direct measurement of neural activity. The two imaging modalities were used to identify cortical areas that were active while the participants attempted to mimic videos of repeated hand, wrist, elbow, and shoulder movements, even though they were unable to move their own paralyzed arm. The neuroimaging results were used to guide electrode placement as described later.

Surgical approach

Forming the surgical plan involved collaboration with the basic science team. The neurosurgeon responsible for the human surgery scrubbed in on nonhuman primate operations. This approach allowed the surgeon to learn strategies related to placing the MEA, bending the interconnect, and positioning pedestals within the constraints of the craniotomy and exposed skull. It also provided an opportunity to develop ideas for identifying standard human instruments that could interface with the special instruments used for placing the MEAs for the BCI surgeries. In preparing for the human surgery, special considerations were made involving craniotomy size, external pedestal location, and internal MEA location. We attempted to minimize the size of the craniotomy to prevent infection. The pedestals were secured to the skull, and exit through the scalp to provide the physical interface between the data acquisition cable and the MEA. Pedestal locations were chosen for multiple reasons. It was important to place the pedestals far enough away from the entry site to be able to secure the pedestals to the skull, and far enough away from each other to be able to accommodate the cables that connect the pedestals to the recording hardware. Importantly, the pedestals could not interfere with the participant's ability to position her head on her chair's headrest or a pillow for sleeping. Internal MEA position on the cortex was decided based on the preoperative imaging. Neuroimaging information was made available during the surgery and co‐registered to the participant's anatomy in real‐time using a surgical navigation system (Brainlab, Westchester, IL, USA) to help guide electrode placement.56 The functional and anatomic images were reviewed by the entire team and targets were determined as a consensus. This consensus‐driven approach for targeted electrode placement was further refined by identifying alternative sites preoperatively in case the initial target site was found to be suboptimal intraoperatively.

The surgery required general anesthesia, and thus we collaborated with an anesthesiologist early in the protocol development process. Pathologies such as tetraplegia, spastic, or flaccid paralysis, autonomic hyperreflexia, receptor up‐or‐down regulation at the neuromuscular junction, cervical fusions, and tracheostomy57, 58, 59, 60, 61, 62, 63, 64, 65 were evaluated preoperatively and monitored closely intraoperatively, among other comorbidities that have a significant impact on anesthetic management. Collaboration with the anesthesiologist and surgeon allowed for the inclusion of appropriate risk descriptions and mitigation strategies in our clinical protocol.

Psychological support

An important component of the study process was providing the participant with support from a psychologist who specialized in spinal cord injury counseling. Prior to the surgery, the participant was screened using neuropsychological and psychosocial assessments. The goal of the screening was to ensure that the participant had normal cognitive function, realistic expectations about the study, and adequate family and caregiver support. Psychological support was made available during study participation with in‐person or phone meetings approximately every 2 months, or whenever the participant thought it was necessary. The participant's mood and psychological well‐being were assessed periodically, as well as how the participant handled the demands and expectations of the project. In the short term, the psychologist helped the participant adjust to the spotlight of a high‐profile research study. In the long term, the role of the psychologist is to help the participant transition from a demanding research schedule back to every‐day life.

BCI training and outcome measures

Frequent practice is key to facilitate learning of a BCI. For the intracortical study, the participant completed three 4‐hour sessions per week. We attempted to balance purely target‐based tasks that allowed for more quantitative assessments of progress, and activities of daily living (ADLs) tasks, such as feeding. If the participant was frustrated or tired, we could take a break, vary the difficulty of tasks, or switch from a target‐based task to a more enjoyable ADL task or game. All tasks were designed to use the same endpoint‐velocity control of the MPL or virtual MPL. We also kept track of success rates and scores to provide additional motivation.

An important component of the study was measurement of clinical outcomes associated with the BCI‐controlled MPL. In order to use validated metrics, we used tests designed to measure upper limb function. In upper limb rehabilitation, clinical outcomes are characterized in terms of reach, grasp, and manipulation, as observed during standardized laboratory tasks (e.g., picking up and placing an object on a surface) or practical “real‐world” tasks (e.g., brushing teeth, eating with a fork). Quality of movement (measured with ordinal scales or performance time) and the amount of use of the impaired limb (measured with ordinal scales or actigraphy) are frequent indicators of limb function. This study measured quality of movement with the Action Research Arm Test (ARAT),52 an assessment that is traditionally used to measure unilateral upper limb function following a stroke.

Study methods and results

The MEA BCI study is registered at clinicaltrials.gov (http://clinicaltrials.gov/ct2/show/NCT01364480) and was conducted under an IDE granted by the FDA and with approval from the IRB at the University of Pittsburgh and the Space and Naval Warfare Systems Center Pacific. In February 2012, two intracortical MEAs (100 electrode shanks 1.5 mm in length, 4 × 4 mm footprint, Blackrock Microsystems) were implanted in the motor cortex of a 52‐year‐old woman with spinocerebellar degeneration. As previously published,19 over 13 weeks of training, the participant learned to control the MPL in seven DOF. By modulating her neural activity, she was able to control the 3D endpoint velocity of the hand, 3D hand/wrist orientation, and 1D grasp, simultaneously. Steady improvements in performance were observed on tasks that required her to reach targets that sampled all seven dimensions. Performance was measured as success rate, completion time, and path efficiency. The participant also used the MPL to perform reach and grasp movements that resulted in clinically significant gains on the ARAT. Operation of the MPL under brain‐control resulted in smooth and coordinated movements with speeds approaching that of able‐bodied individuals.

Since the time of the original publication, we have worked to extend control to include a variety of hand shapes which would allow for different grasp types, such as a pinch or whole hand grasp (Video 2). The subject can play games involving several hand postures, such as rock‐paper‐scissors (Video 3). The participant has also used the MPL to accomplish a goal that she set at the beginning of the study: to feed herself chocolate. In the video, the participant is controlling the 3D endpoint velocity of the MPL and has continuous control over a single grasp dimension. Once she hits her target location, the MPL was paused to allow her to take a bite of the chocolate (Video 4). The patient has been implanted with the MEA for over 1 year, a testament to the robustness of the implantation and training system. No adverse events from MEA implantation have been observed to date.

Being a research participant: In their own words

Subjects require a strong support system from family, caregivers, and friends. In fact, such support was an important inclusion criterion of the study, and the neuropsychological assessment evaluated this support system. However, the traditional roles in the laboratory tended to shift; through many hours of cooperation between researchers and subjects, not only did the investigators become a part of the subjects’ support system, but it soon became clear that the research participants were as much team members as the study investigators. The participants offered their view of being collaborators in the studies below.

Tim Hemmes, short‐term ECoG study participant

My role in this study was to be completely focused and give the team as clean a signal as possible. At times this was easy, but not always. I have not been able to move my arms for 8 years so the saying “use it or lose it” was really affecting me. It was hard to focus on “moving” my right arm and hand. In my opinion this was the most difficult obstacle to overcome and I compare it to a baby learning to walk. But when the electrodes started lighting up as I was thinking “up, down, left or right,” it made all of my “mental thinking” and exhausting focus worthwhile. (Note: the participant observed a measure of signal strength for each electrode on a computer display during the early phases of training.) My initial fear going into the study was being unable to create the thoughts that we needed. So to see these electrodes light up knowing it was working took a lot of pressure off me, and allowed me to focus even more. During this study, the team and I really started understanding each other and becoming closer on a personal level. Finding the words to explain to the researchers how I was able to overcome every obstacle they asked of me and what I was feeling, I very quickly felt like a team member and not just their “lab monkey.”

Jan Scheuermann, the MEA BCI study participant, and Karina Palko, Jan's attendant

After seeing Tim Hemmes in a video moving the robotic arm, I had no doubts about volunteering. I was fully informed of all the risks, and my attitude was “full speed ahead!” I feel blessed and honored to have the role of the human subject in this study. I was hooked up to the computer that sent signals to the robotic hand, 3 days a week for 4‐hour sessions. Moving the hand was easier than I had thought it would be, and every success spurred me on for the next challenge. Karina, my attendant, went with me to feed me and take care of me. Not content to just sit there, she started taking notes that proved to be useful to the team. She also became an integral part of the process of cleaning my pedestals. We have both been very excited to see the triumphs of this project, and we are happy to be part of it.

Conclusions

The MEA BCI study brought together teams and individuals from multiple different disciplines including physiatry, bioengineering, neurosurgery, neurobiology, anesthesiology, psychology, and occupational therapy. The multidisciplinary effort and timeline is summarized in Figure 4. The team was able to work toward a common goal, and decide on achievable clinical research objectives and methods. Open communication and collaboration across groups was essential to the success of the project. Coordination between the University of Pittsburgh researchers, regulatory staff, Blackrock, JHU/APL, and DARPA allowed us to engage the FDA and IRB early, and thus quickly initiate the project. Technical and clinician teams worked with the participants allowing us to anticipate software and hardware problems early, resulting in long uninterrupted training periods. Finally, in working with the volunteers through our successes and failures, our participants became important members of the team. The BCI study can be an example of the multidisciplinary cooperation needed to bring a highly complex technology from animal models to a successful human trial.

Figure 4.

Figure 4

Open in a new tab

Roadmap toward the implementation of the multiyear MEA BCI clinical study.

Acknowledgments

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) Revolutionizing Prosthetics program contract number N66001‐10‐C‐4056, the National Institutes of Health (NIH) grant 8KL2TR000146‐07, the Office of Research and Development, Rehabilitation Research & Development Service, Department of Veterans Affairs (Grants # B6789C, B7143R, and RX720), and the UPMC Rehabilitation Institute. The views expressed herein are those of the authors and do not reflect the official policy or position of the Department of Veterans Affairs, Department of Defense, or the United States government.

References

  • 1. Fetz EE. Operant conditioning of cortical unit activity. Science 1969; 163(3870): 955–958. [DOI] [PubMed] [Google Scholar]
  • 2. Georgopoulos AP, Kalaska JF, Caminiti R, Massey JT. On the relations between the direction of two‐dimensional arm movements and cell discharge in primate motor cortex. J Neuroscience: Official J Soc Neurosci. 1982; 2(11): 1527–1537. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3. Kalaska JF, Caminiti R, Georgopoulos AP. Cortical mechanisms related to the direction of two‐dimensional arm movements: relations in parietal area 5 and comparison with motor cortex. Experimental brain research. Experimentelle Hirnforschung. Experimentation cerebrale 1983; 51(2): 247–260. [DOI] [PubMed] [Google Scholar]
  • 4. Taylor DM, Tillery SI, Schwartz AB. Direct cortical control of 3D neuroprosthetic devices. Science 2002; 296(5574): 1829–1832. [DOI] [PubMed] [Google Scholar]
  • 5. Schwartz AB, Kettner RE, Georgopoulos AP. Primate motor cortex and free arm movements to visual targets in three‐dimensional space. I. Relations between single cell discharge and direction of movement. J Neurosci: Offic J Soc Neurosci. 1988; 8(8): 2913–2927. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Lebedev MA, Carmena JM, O'Doherty JE, Zacksenhouse M, Henriquez CS, Principe JC, Nicolelis MA. Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain‐machine interface. J Neurosci: Offic J Soc Neurosci. 2005; 25(19): 4681–4693. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Serruya MD, Hatsopoulos NG, Paninski L, Fellows MR, Donoghue JP. Instant neural control of a movement signal. Nature 2002; 416(6877): 141–142. [DOI] [PubMed] [Google Scholar]
  • 8. Carmena JM, Lebedev MA, Crist RE, O'Doherty JE, Santucci DM, Dimitrov DF, Patil PG, Henriquez CS, Nicolelis MA. Learning to control a brain‐machine interface for reaching and grasping by primates. PLoS Biol. 2003; 1(2): E42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Georgopoulos AP, Schwartz AB, Kettner RE. Neuronal population coding of movement direction. Science 1986; 233(4771): 1416–1419. [DOI] [PubMed] [Google Scholar]
  • 10. Schwartz AB. Direct cortical representation of drawing. Science 1994; 265(5171): 540–542. [DOI] [PubMed] [Google Scholar]
  • 11. Ethier C, Oby ER, Bauman MJ, Miller LE. Restoration of grasp following paralysis through brain‐controlled stimulation of muscles. Nature 2012; 485(7398): 368–371. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12. Velliste M, Perel S, Spalding MC, Whitford AS, Schwartz AB. Cortical control of a prosthetic arm for self‐feeding. Nature 2008; 453(7198): 1098–1101. [DOI] [PubMed] [Google Scholar]
  • 13. DeVivo MJ, Chen Y. Trends in new injuries, prevalent cases, and aging with spinal cord injury. Archiv Phys Med Rehabilitat. 2011; 92(3): 332–338. [DOI] [PubMed] [Google Scholar]
  • 14. Lee BB, Cripps RA, Fitzharris M, Wing PC. The global map for traumatic spinal cord injury epidemiology: update 2011, global incidence rate. Spinal Cord 2013. Feb 26 [Epub ahead of print]. [DOI] [PubMed] [Google Scholar]
  • 15. Ziegler‐Graham K, MacKenzie EJ, Ephraim PL, Travison TG, Brookmeyer R. Estimating the prevalence of limb loss in the United States: 2005 to 2050. Arch Phys Med and Rehabilitat. 2008; 89(3): 422–429. [DOI] [PubMed] [Google Scholar]
  • 16. Kim SP, Simeral JD, Hochberg LR, Donoghue JP, Black MJ. Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. J Neural Eng. 2008; 5(4): 455–476. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Hochberg LR, Bacher D, Jarosiewicz B, Masse NY, Simeral JD, Vogel J, Haddadin S, Liu J, Cash SS, van der Smagt P, et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 2012; 485(7398): 372–375. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, Branner A, Chen D, Penn RD, Donoghue JP. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 2006; 442(7099): 164–171. [DOI] [PubMed] [Google Scholar]
  • 19. Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler‐Kabara EC, Weber DJ, McMorland AJ, Velliste M, Boninger ML, Schwartz AB. High‐performance neuroprosthetic control by an individual with tetraplegia. Lancet 2012; 381(9866): 557–564. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Wang W, Collinger JL, Degenhart AD, Tyler‐Kabara EC, Schwartz AB, Moran DW, Weber DJ, Wodlinger B, Vinjamuri RK, Ashmore RC, et al. An electrocorticographic brain interface in an individual with tetraplegia. PLoS One 2013; 8(2): e55344. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Hill NJ, Lal TN, Schroder M, Hinterberger T, Wilhelm B, Nijboer F, Mochty U, Widman G, Elger C, Scholkopf B, et al. Classifying EEG and ECoG signals without subject training for fast BCI implementation: comparison of nonparalyzed and completely paralyzed subjects. IEEE Transact Neural Syst Rehabilitat Eng: A Publicat IEEE Eng Med and Biol Soc. 2006; 14(2): 183–186. [DOI] [PubMed] [Google Scholar]
  • 22. Murguialday AR, Hill J, Bensch M, Martens S, Halder S, Nijboer F, Schoelkopf B, Birbaumer N, Gharabaghi A. Transition from the locked in to the completely locked‐in state: a physiological analysis. Clinic Neurophysiol: Off J Int Federat Clinical Neurophysiol. 2011; 122(5): 925–933. [DOI] [PubMed] [Google Scholar]
  • 23. Wang W, Degenhart AD, Sudre GP, Pomerleau DA, Tyler‐Kabara EC. Decoding semantic information from human electrocorticographic (ECoG) signals Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA: 2011: 6294–6298. [DOI] [PubMed] [Google Scholar]
  • 24. Degenhart AD, Collinger JL, Vinjamuri R, Kelly JW, Tyler‐Kabara EC, Wang W. Classification of hand posture from electrocorticographic signals recorded during varying force conditions Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Boston, MA: 2011: 5782–5785. [DOI] [PubMed] [Google Scholar]
  • 25. Wang W, Degenhart AD, Collinger JL, Vinjamuri R, Sudre GP, Adelson PD, Holder DL, Leuthardt EC, Moran DW, Boninger ML, et al. Human motor cortical activity recorded with Micro‐ECoG electrodes, during individual finger movements Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN: 2009: 586–589. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. Wang W, Collinger JL, Perez MA, Tyler‐Kabara EC, Cohen LG, Birbaumer N, Brose SW, Schwartz AB, Boninger ML, Weber DJ. Neural interface technology for rehabilitation: exploiting and promoting neuroplasticity. Phys Med Rehabilitat Clinics of North Am. 2010; 21(1): 157–178. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Crone NE, Miglioretti DL, Gordon B, Sieracki JM, Wilson MT, Uematsu S, Lesser RP. Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. I. Alpha and beta event‐related desynchronization. Brain : A J Neurol. 1998; 121 (Pt 12): 2271–2299. [DOI] [PubMed] [Google Scholar]
  • 28. Crone NE, Miglioretti DL, Gordon B, Lesser RP. Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event‐related synchronization in the gamma band. Brain: A J Neurol. 1998; 121 (Pt 12): 2301–2315. [DOI] [PubMed] [Google Scholar]
  • 29. Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW. A brain‐computer interface using electrocorticographic signals in humans. J Neural Eng. 2004; 1(2): 63–71. [DOI] [PubMed] [Google Scholar]
  • 30. Schalk G, Kubanek J, Miller KJ, Anderson NR, Leuthardt EC, Ojemann JG, Limbrick D, Moran D, Gerhardt LA, Wolpaw JR. Decoding two‐dimensional movement trajectories using electrocorticographic signals in humans. J Neural Eng. 2007; 4(3): 264–275. [DOI] [PubMed] [Google Scholar]
  • 31. Miller KJ, Schalk G, Fetz EE, den Nijs M, Ojemann JG, Rao RP. Cortical activity during motor execution, motor imagery, and imagery‐based online feedback. Proc Natl Acad Sci U S A 2010; 107(9): 4430–4435. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. Isaacs RE, Weber DJ, Schwartz AB. Work toward real‐time control of a cortical neural prothesis. IEEE Trans Rehab Eng: Publ IEEE Eng Med Biol Soc. 2000; 8(2): 196–198. [DOI] [PubMed] [Google Scholar]
  • 33. Clanton ST. Brain‐Computer Interface Control of an Anthropomorphic Robotic Arm. Pittsburgh, PA: Robotics Institute, Carnegie Mellon University; 2011. [Google Scholar]
  • 34. Kelly JW, Degenhart AD, Siewiorek DP, Smailagic A, Wang W. Sparse linear regression with elastic net regularization for brain‐computer interfaces Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Diego, CA: 2012: 4275–4278. [DOI] [PubMed] [Google Scholar]
  • 35. Vinjamuri R, Weber DJ, Degenhart AD, Collinger JL, Sudre GP, Adelson PD, Holder DL, Boninger ML, Schwartz AB, Crammond DJ, et al. A fuzzy logic model for hand posture control using human cortical activity recorded by micro‐ECog electrodes Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MN: 2009: 4339–4342. [DOI] [PubMed] [Google Scholar]
  • 36. Crone NE, Sinai A, Korzeniewska A. High‐frequency gamma oscillations and human brain mapping with electrocorticography. Progress Brain Res. 2006; 159: 275–295. [DOI] [PubMed] [Google Scholar]
  • 37. Mukamel R, Gelbard H, Arieli A, Hasson U, Fried I, Malach R. Coupling between neuronal firing, field potentials, and FMRI in human auditory cortex. Science 2005; 309(5736): 951–954. [DOI] [PubMed] [Google Scholar]
  • 38. Schalk G. BCIs that use electrocorticographic activity In Wolpaw JR, Winter‐Wolpaw E, eds. Brain‐Computer Interfaces: Principles and Practice. New York: Location is New York, NY: Also, to clarify, this is a chapter from a textbook, In: Brain‐Computer Interfaces: Principles and Practice. Eds: Wolpaw J.R. and Winter‐Wolpaw E. Oxford University Press; 2012: 251–264. [Google Scholar]
  • 39. Schalk G, Leuthardt EC. Brain‐computer interfaces using electrocorticographic signals. IEEE Rev Biomed Eng. 2011; 4: 140–154. [DOI] [PubMed] [Google Scholar]
  • 40. Degenhart AD, Kelly JW, Ashmore RC, Collinger JL, Tyler‐Kabara EC, Weber DJ, Wang W. Craniux: a LabVIEW‐based modular software framework for brain‐machine interface research. Computat Intel Neurosci. 2011; 2011: 363565. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41. Vinjamuri R, Weber DJ, Mao ZH, Collinger JL, Degenhart AD, Kelly JW, Boninger ML, Tyler‐Kabara EC, Wang W. Toward synergy‐based brain‐machine interfaces. IEEE Trans Inf Technol Biomed: Publ IEEE Eng MedBiol Soc. 2011; 15(5): 726–736. [DOI] [PubMed] [Google Scholar]
  • 42. Crone NE. Functional mapping with ECoG spectral analysis. Adv Neurol. 2000; 84: 343–351. [PubMed] [Google Scholar]
  • 43. Hill NJ, Gupta D, Brunner P, Gunduz A, Adamo MA, Ritaccio A, Schalk G. Recording human electrocorticographic (ECoG) signals for neuroscientific research and real‐time functional cortical mapping. J Vis Exp: JoVE. 2012; (64): e3993. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44. Yanagisawa T, Hirata M, Saitoh Y, Kishima H, Matsushita K, Goto T, Fukuma R, Yokoi H, Kamitani Y, Yoshimine T. Electrocorticographic control of a prosthetic arm in paralyzed patients. Ann Neurol. 2012; 71(3): 353–361. [DOI] [PubMed] [Google Scholar]
  • 45. Rouse AG, Williams JJ, Wheeler JJ, Moran DW. Cortical adaptation to a chronic micro‐electrocorticographic brain computer interface. J Neurosci: Official J Soc Neurosci. 2013; 33(4): 1326–1330. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Chao ZC, Nagasaka Y, Fujii N. Long‐term asynchronous decoding of arm motion using electrocorticographic signals in monkeys. Frontiers in Neuroeng. 2010; 3: 3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. Revolutionizing Prosthetics . http://www.darpa.mil/our_work/dso/programs/revolutionizing_prosthetics.aspx. Accessed February 1, 2013.
  • 48. Harris A, Katyal K, Para M, Thomas J. Revolutionizing prosthetics software technology Paper Presented at: Systems, Man, and Cybernetics (SMC), 2011 IEEE International Conference, Anchorage, AK: 9–12 October, 2011. [Google Scholar]
  • 49. Johannes MS, Bigelow JD, Burck JM, Harshbarger SD, Kozlowski MV, Van Doren T. An overview of the developmental process for the Modular Prosthetic Limb The Johns Hopkins University Applied Physics Laboratory Technical Digest. Vol 30 Laurel, MD: Johns Hopkins University; 2011. [Google Scholar]
  • 50. Armiger RS, Tenore FV, Bishop WE, Beaty JD, Bridges MM, Burck JM, Vogelstein RJ, Harshbarger SD. A real‐time virtual integration environment for neuroprosthetics and rehabilitation In The Johns Hopkins University Applied Physics Laboratory Technical Digest. Vol 30 Laurel, MD: Johns Hopkins University; 2011. [Google Scholar]
  • 51. Bishop W, Armiger R, Burck J, Bridges M, Hauschild M, Englehart K, Scheme E, Vogelstein RJ, Beaty J, Harshbarger S. A real‐time virtual integration environment for the design and development of neural prosthetic systems Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vancouver, BC: 2008: 615–619. [DOI] [PubMed] [Google Scholar]
  • 52. Lyle RC. A performance test for assessment of upper limb function in physical rehabilitation treatment and research. International journal of rehabilitation research. Internationale Zeitschrift fur Rehabilitationsforschung. Revue Internationale de Recherches de Readaptation 1981; 4(4): 483–492. [DOI] [PubMed] [Google Scholar]
  • 53. Investigational Device Exemption . Device Advice: Comprehensive Regulatory Assistance 2013; http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/HowtoMarketYourDevice/InvestigationalDeviceExemptionIDE/default.htm. Accessed February 1, 2013. [Google Scholar]
  • 54. Marquaridt DW. Generalized inverses, ridge regression, biased linear estimation, and nonlinear estimation. Technometrics 1970; 12(3): 591–612. [Google Scholar]
  • 55. Kokotilo KJ, Eng JJ, Curt A. Reorganization and preservation of motor control of the brain in spinal cord injury: a systematic review. J Neurotrauma. 2009; 26(11): 2113–2126. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56. Wang W, Sudre GP, Xu Y, Kass RE, Collinger JL, Degenhart AD, Bagic AI, Weber DJ. Decoding and cortical source localization for intended movement direction with MEG. J Neurophysiol. 2010; 104(5): 2451–2461. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 57. Martyn JA, White DA, Gronert GA, Jaffe RS, Ward JM. Up‐and‐down regulation of skeletal muscle acetylcholine receptors. Effects on neuromuscular blockers. Anesthesiol . 1992; 76(5): 822–843. [DOI] [PubMed] [Google Scholar]
  • 58. Rossman AC. The physiology of the nicotinic acetylcholine receptor and its importance in the administration of anesthesia. AANA J. 2011; 79(5): 433–440. [PubMed] [Google Scholar]
  • 59. Eltorai IM, Wong DH, Lacerna M, Comarr AE, Montroy R. Surgical aspects of autonomic dysreflexia. J Spinal Cord Med. 1997; 20(3): 361–364. [PubMed] [Google Scholar]
  • 60. Hambly PR, Martin B. Anaesthesia for chronic spinal cord lesions. Anaesthesia 1998; 53(3): 273–289. [DOI] [PubMed] [Google Scholar]
  • 61. Krassioukov A, Warburton DE, Teasell R, Eng JJ. A systematic review of the management of autonomic dysreflexia after spinal cord injury. Archiv Phys Med and Rehabilitat 2009; 90(4): 682–695. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62. Lambert DH, Deane RS, Mazuzan JE Jr. Anesthesia and the control of blood pressure in patients with spinal cord injury. Anesthesia Analgesia 1982; 61(4): 344–348. [PubMed] [Google Scholar]
  • 63. Schonwald G, Fish KJ, Perkash I. Cardiovascular complications during anesthesia in chronic spinal cord injured patients. Anesthesiology 1981; 55(5): 550–558. [DOI] [PubMed] [Google Scholar]
  • 64. Yoo KY, Jeong CW, Kim SJ, Chung ST, Bae HB, Oh KJ, Lee J. Sevoflurane concentrations required to block autonomic hyperreflexia during transurethral litholapaxy in patients with complete spinal cord injury. Anesthesiology 2008; 108(5): 858–863. [DOI] [PubMed] [Google Scholar]
  • 65. Yoo KY, Jeong CW, Kim SJ, Jeong ST, Kim WM, Lee HK, Oh KJ, Lee JU, Shin MH, Chung SS. Remifentanil decreases sevoflurane requirements to block autonomic hyperreflexia during transurethral litholapaxy in patients with high complete spinal cord injury. Anesthesia Analgesia 2011; 112(1): 191–197. [DOI] [PubMed] [Google Scholar]

Articles from Clinical and Translational Science are provided here courtesy of Wiley

RESOURCES