Developing a hippocampal neural prosthetic to facilitate human memory encoding and recall

Abstract

Objective

We demonstrate here the first successful implementation in humans of a proof-of-concept system for restoring and improving memory function via facilitation of memory encoding using the patient’s own hippocampal spatiotemporal neural codes for memory. Memory in humans is subject to disruption by drugs, disease and brain injury, yet previous attempts to restore or rescue memory function in humans typically involved only nonspecific, modulation of brain areas and neural systems related to memory retrieval.

Approach

We have constructed a model of processes by which the hippocampus encodes memory items via spatiotemporal firing of neural ensembles that underlie the successful encoding of short-term memory. A nonlinear multi-input, multi-output (MIMO) model of hippocampal CA3 and CA1 neural firing is computed that predicts activation patterns of CA1 neurons during the encoding (sample) phase of a delayed match-to-sample (DMS) human short-term memory task.

Main results

MIMO model-derived electrical stimulation delivered to the same CA1 locations during the sample phase of DMS trials facilitated short-term/working memory by 37% during the task. Longer term memory retention was also tested in the same human subjects with a delayed recognition (DR) task that utilized images from the DMS task, along with images that were not from the task. Across the subjects, the stimulated trials exhibited significant improvement (35%) in both short-term and long-term retention of visual information.

Significance

These results demonstrate the facilitation of memory encoding which is an important feature for the construction of an implantable neural prosthetic to improve human memory.

Introduction

We report here the first demonstration of a closed-loop neural prosthetic based on ‘writing’ information ‘codes’ for memory into the hippocampus of human subjects. Previous studies that focused on human memory restoration or enhancement implemented pharmaceutical facilitation (Epperly et al 2017), single electrode/fixed-frequency stimulation of brain regions other than the hippocampus (Suthana and Fried 2014, Jacobs et al 2016), or even fixed frequency stimulation of the hippocampus proper (Ezzyat et al 2017). The results reported here illustrate, for the first time, that multi-site spatiotemporal codes designed to mimic specific memory-related neural ensemble firing have been demonstrated to facilitate memory in humans. This demonstration is an important first step in the development of a neural prosthetic for memory which utilizes the information content of hippocampal neural ensembles in a manner similar to prosthetics developed for visual (Weiland et al 2011), auditory (Lim and Lenarz 2015) and neuromuscular (Chase et al 2012) recovery and facilitation.

Memory impairment due to synaptic dysfunction or neuron loss is one of the earliest and most distressing symptoms of Alzheimer’s disease and age-related dementia (Skaper et al 2017). The important role of the human hippocampus in the encoding and retrieval of retained information has been known since 1957, when Scoville and Milner (Scoville and Milner 1957) described memory impairment in ten patients who had undergone partial bilateral medial temporal lobe resections for seizures or psychosis. The classic situation of the patient, HM, revealed the critical importance of the hippocampus in episodic memory, evidenced by the inability to recall events (i.e. anterograde amnesia) following bilateral medial temporal lobe resection for medical refractory epilepsy. The same investigators and others (Zola-Morgan et al 1986, Dickson and Vanderwolf 1990) further noted that patients without a large degree of bilateral resection of the hippocampus did not display such significant memory deficits.

Over many years, the assessment of memory-based neural firing in animal models has been examined to determine how hippocampal CA1 and CA3 subfields are organized into hierarchical networks that underlie the encoding of information and memory retrieval (Squire et al 1987, Witter 1993). Furthermore, previous investigations have focused on the significance of memory-specific encoding patterns for behavioral task performance, and whether effective multineuron firing patterns could be reproduced via the delivery of electrical stimulation in the same pattern to hippocampal neuron ensembles from which the patterns were originally extracted (Hampson et al 1999, 2011). Thus, the development of a potential therapeutic device or system to enhance or restore damaged memory in the primate brain (Berger et al 2011, Hampson et al 2011), required: (i) localization of electrodes within hippocampal sub-layers, including CA1 and CA3 (Hampson et al 1999, Opris et al 2015) to allow (ii) recording of hippocampal neuronal firing patterns during successful information encoding (Berger et al 2011, Hampson et al 2013), and development of (iii) a nonlinear model of CA1 output firing patterns based on neural inputs to CA3, which provided the way to (iv) deliver model-based electrical micro stimulation to these same areas during the task. In previous studies, we successfully applied patterned CA1 stimulation (Song et al 2015) to restore and facilitate memory in rodent and nonhuman primate (NHP) functional models (Berger et al 2011, Hampson et al 2013) via procedures that can now be applied to the restoration of memory capability in humans.

In this study, hippocampal neural ensembles were recorded during a delayed match-to-sample (DMS) memory task performed by epilepsy patients surgically implanted with intracranial electrodes to localize clinically-based seizures. A nonlinear dynamic multi-input, multi-output (MIMO) model of hippocampal information processing was constructed for each subject from initial recordings of neuron firing in input (putative CA3) and output (putative CA1) regions of the hippocampus during the first session of the DMS human memory task. This allowed a derivation of specific MIMO model-based firing patterns during the sample (encoding) phase for each patient. These patterns were applied as stimulation in the sample phase in randomly selected trials in the second DMS task session, which occurred 2–5 d later. In the same subjects, electrical stimulation derived from real-time MIMO model-predicted input–output ensemble firing patterns was applied to hippocampal CA1 electrode sites during the encoding (sample) phase of the task. Trials with MIMO model-predicted stimulation were interleaved with randomized (control) stimulation and no-stimulation trials in the same session in order to compare the stimulation effects. Note that normal CA1 activity was also present during the DMS task in addition to the stimulated pattern. Memory retention was assessed in the DMS task in seven subjects, and in a DR test of information presented in the DMS task in five of the same subjects with (and one additional subject tested with DR only) and without stimulation. The results presented below demonstrate significant memory facilitation produced by closed-loop electrical stimulation with MIMO model-based ensemble ‘codes’ for effective memory encoding during these tasks.

Materials and methods

The 22 subjects that enrolled in the study (table 1) had medically-refractory focal epilepsy and underwent implantation of intracranial depth electrodes for seizure monitoring and localization. All subjects were selected via noninvasive monitoring of epileptogenic foci, in which further invasive monitoring with recording electrodes was deemed necessary prior to potential surgical treatment. Each subject underwent comprehensive pre-operative assessment, including long-term, non-invasive video-EEG analysis, pre-operative MRI, and neuropsychological assessment (see table 3 found in the supplementary material, available online at stacks.iop.org/JNE/15/036014/mmedia). The locations of the depth electrodes used in this study were selected on the basis of pre-surgical EEG and imaging, and were confirmed via post-implant MRI (figures 1 and 2) and electrophysiological analyses. Seventeen subjects were recorded during at least one memory test session, and provided data suitable for developing and testing the MIMO model (see below). Nine subjects were used strictly for model development (Song et al 2016), while eight subjects received stimulation to test a potential model for the facilitation of memory encoding and recall and are presented here.

Table 1.

Details of patients tested and hippocampal neural data recorded in this study. The top section (total # of subjects) provides details of the number of patients recruited and enrolled in the study—consented, recorded and stimulated. Note that one patient was initially enrolled, but elected not to provide consent for the memory testing. Of the remaining patient subjects, hippocampal recordings during the DMS task were obtained from 17. Complete recording data could not be obtained from the remaining four subjects due to factors unrelated to this study. The data from the initial nine subjects was used for model development and simulation of the MIMO model stimulation. Eight patients were tested with the MIMO-based stimulation once modeling was complete.

Patient recruitment and subject enrollment
	Total	# of probes	Total neurons	Total CA1	Total CA3
Total # of subjects:
Enrolled	22	—	—	—	—
Consented	21	—	—	—	—
Recorded	17	32	663	319	344
Stimulated	8	18	349	170	179
Implant types:
Unilateral anterior-only	7	7	165	76	89
Bilateral anterior-only	4	8	167	83	84
Unilateral anterior + posterior	3	6	81	42	39
Bilateral anterior + unilateral posterior	1	3	53	26	27
Bilateral anterior + bilateral posterior	2	8	128	66	62
Subjects reported in this study:	8	18	349	170	179
Unilateral anterior-only	1	1	19	10	9
Bilateral anterior-only	3	6	124	61	63
Unilateral anterior + posterior	2	4	55	28	27
Bilateral anterior + unilateral posterior	1	3	53	26	27
Bilateral anterior + bilateral posterior	1	4	85	44	41

The middle section (implant types) provides details of the electrode implants, number of electrodes recorded, and the overall number of neurons recorded from the CA3- and CA1-positioned sites on the electrodes. These details are provided for all subjects recorded (n = 17) in the middle section (implant types), as well as the details limited to the eight patients that received MIMO-based stimulation reported in this study.

Figure 1.

Coronal and axial 3-Tesla magnetic resonance imaging (MRI) showing macro–micro depth electrode placement in the hippocampus. T2-weighted (left column) pre-surgical, and T1-weighted (center and right columns) pre- and post-surgical coronal images from the level of anterior hippocampus (top row) and posterior hippocampus (middle row) depict the brain areas of the respective anterior and posterior bilateral hippocampal electrode placement. Axial views (bottom row) compare placement at a level that shows all four electrodes.

Figure 2.

Enlargement of left anterior hippocampus in the same patient as shown in figure 1. Pre-surgical T2-(A) and T1-weighted (B) MRI (0.8 × 0.8 × 0.8 mm voxels) show that the macro–micro depth electrode is placed in the anterior head of the hippocampus (C). A computed tomography x-ray (D) from the same patient shows the placement of all depth electrodes in the ‘stereo-EEG’ configuration for phase II invasive monitoring of epileptic seizures. Schematic (E) shows the structure and orientation of the micro-electrode single neuron recording sites relative to the hippocampal cell layers. The electrode is represented on the schematic emphasizing the macro (EEG) recording sites which are visible as ‘bead-like’ shadows superimposed on the linear electrode shadow on the MRI in C. (F) Schematic of the electrode illustrating the position of the 17 μm micro-electrode single neuron recording sites interspersed with the 2 mm macro-electrode EEG recording sites. A combination of stereotaxic placement, imaging and electrophysiological analysis in each subject was utilized to assign neurons recorded on each electrode (Wicks et al 2016) to putative CA3 and CA1 neuroanatomical placement (see table 2). The voxel size for the images was 0.8 mm × 0.8 mm × 0.8 mm.

All procedures were reviewed and approved by the Institutional Review Board of Wake Forest University, in accordance with the National Institutes of Health. The subjects provided voluntary written informed consent (separate from consent for surgery) prior to participation in this study. All surgical procedures, post-operative monitoring and neurocognitive testing were performed at the Comprehensive Epilepsy Center, Wake Forest Baptist Medical Center.

Electrode implantation and removal

All subjects underwent pre-operative imaging to determine intraoperative electrode placement for ‘stereo-EEG’ assessment of epileptic seizures. Epilepsy patients typically received 8–10 ‘macro’ style EEG probes, and 1–4 ‘macro–micro’ style EEG/single neuron probes in a single surgical procedure. All probes consisted of FDA-approved depth electrodes (Ad-Tech Medical Instrumentation Corporation, Racine, WI). Intraoperative placement of the electrodes was performed using either a stereotactic head-frame (CRW Precision Arc, Integra Life Sciences, Plainsboro, NJ) or a frameless stereotactic system (VarioGuide, Brainlab AG, Feldkirchen, Germany). Hippocampal electrode placement (figure 2(E)) was oriented to penetrate the head of the hippocampus perpend icular to the long axis in order to record neuronal activity within both the putative CA3 and CA1 subfields (Wicks et al 2016). Intraoperative monitoring of neuronal activity was performed during electrode positioning to confirm single unit and field potential recording. Electrodes were then secured and the craniotomies closed. After recovery from anesthesia, the subjects employed in this study were transferred to the Epilepsy Monitoring Unit, where all subsequent tests were performed. Post-operative localization of electrode placements was verified by a combination of MRI (figure 1) and electrophysiological activity consistent with putative hippocampal CA1 and CA3 principal cells (including the nonlinear MIMO model (Song et al 2016) and pairwise cross-correlations (Wicks et al 2016)). Electrodes in all subjects were explanted after seizure localization was confirmed or at a time designated by each subject’s care team after a sufficient period of invasive monitoring had been performed.

DMS memory testing

Neurocognitive experiments were performed on post-implantation days 3–10, depending on the clinical needs of each subject. In addition to undergoing the standard seizure localization protocol, all subjects agreed to participate in the approved neurocognitive tasks involving simultaneous recording and electrical stimulation of ensemble activity recorded from the hippocampal electrode arrays.

During the memory tasks, the subjects were seated in a hospital bed or in a bedside chair facing a touch-sensitive computer screen which displayed a custom-designed visual, DMS short-term memory task (see figure 1 of the supplementary material) utilizing visual stimuli based on either clip-art/photo images, or non-verbalizeable color-block images (modeled after the Cambridge Neuropsychological Test Automated Battery (CANTAB, Cambridge Cognition LTD, Cambridge, UK)). The details of the DMS task were identical with respect to timing and stimulus presentation to prior reports from NHPs (Hampson et al 2004), but differed in specific image sets, and use of a touch screen for humans. On the first (MIMO recording) day of testing, human subjects completed a single DMS session for each set of stimulus images. On subsequent sessions (i.e. recording + stimulation), seven subjects were tested with non-verbalizeable color block images in the DMS task, while six subjects were tested with the delayed recognition (DR) task, (see figure 2 of the supplementary material). The DR task was still in development when the first two subjects were tested, thus Px14 and Px15 received only the DMS test. Px16–Px20 were tested on both DMS and DR, with Px21 receiving only the DR test.

DMS trials were initiated by a touch screen which triggered the sample phase of the task (see figure 1 of the supplementary material) via presentation of a single sample image on the screen (sample presentation, SP), which required the subject to touch the image, which was designated as the sample response (SR). The SR initiated the ‘delay interval’ phase of the trial, in which the screen was blank for a period of 1–70 s randomly determined on a trial-to-trial basis. Termination of the delay interval was signaled at a marked screen location which had to be touched (figure 1 of the supplementary material) and was followed by the match presentation (MP) phase of the task. The MP phase consisted of the simultaneous display of 2–7 images at separate, randomly selected spatial locations on the screen, one of which ‘matched’ the previous sample image and the others were distractor images. Touching the sample image in the MP constituted a correct ‘match response’ (MR) and produced a reward tone followed by a blank screen. Touching a non-match-to-sample image on the screen (non-match response) led to an error tone followed by a blank screen. Individual trials were presented separated by 5 s from the previous trials. All the images presented (sample and non-match distractors) were unique for each of the 75–100 trials within a session.

DR memory testing

Long-term memory was tested via DR in a subset of patients (see figure 2 of the supplementary material). In this task, subjects were presented with a separate set of 100 DMS trials which utilized clip/art/photo image stimuli with short delays, and 2–4 images in the MP to ensure maximal correct response in the task. After completion of the 100 DMS trials, subjects rested for 10–15 min, then commenced the 100-trial recognition phase in which subjects were presented on each trial with three clip-art/photo images simultaneously (a DR trial) consisting of: (1) a sample/match image which had been delivered during the previous DMS session, (2) a nonmatch image from the same DMS trial, and (3) a novel image which had not been previously seen by the patient (see figure 2 (bottom) of thesupplementary material). The on-screen position of the three types of images was randomized for each DR trial, and the sequence in which the trials were tested was randomized with respect to the sequence of those images in the previous DMS trials.

For each image, the subject could indicate whether they had seen the picture before, and rank its familiarity on a scale of ‘5: definitely saw in DMS task’ to ‘1: seems familiar, but uncertain’. Subjects could rate any, all or none of the images presented for each DR trial. Once a trial was signaled as complete, the next (randomized) set of DR images was presented, for a total of 100 DR trials, corresponding to the 100 DMS trials. The DR session was concluded after completion of the recognition rating for all the sample/match images presented in the DMS task immediately prior to it. ‘Successful’ recognition was determined to occur in trials in which the match image was rated 4 or 5 and the nonmatch and/or novel images were either not ranked or both rated less than 4. Trials in which the match image was ranked 3 or lower, and/or nonmatch or novel images were ranked higher than the match image, were classified as unsuccessful recognition.

Neuronal recording

Two, three, or four hippocampal-positioned ‘macro–micro’ depth electrodes (Ad-Tech) were placed appropriately per subject. Each electrode contained 4–6 ‘macro’ EEG recording sites and 10–24 ‘micro’ extracellular neural recording sites. Probe and recording site placements were determined as cited above (Electrode Implantation and Removal). One day was allowed for recovery from anesthesia following surgery; therefore, on the second day after implantation at WFBMC, macro–micro electrodes were connected to a Blackrock Cervello neural recording system (Blackrock Neuromed, Salt Lake City, UT). Single neuron extracellular action potential waveforms were isolated and identified for online and offline sorting of single unit discharges (see figure 3 of the supplementary material). Continuous electrical digitized monitoring identified single unit action potential waveforms (bandpass filtered to 500–5000 Hz, 30 000 samples s⁻¹), and single unit spike events (i.e. timestamps, 200 μs resolution) during DMS task performance. Spike events were merged with DMS task markers from a single trial and retained for analysis. Following the first session of DMS testing, neural recordings from subjects participating in the study were sent to the USC team for nonlinear modeling.

Additional details of neural recording and analysis are provided in (supplementary material figures 3–6). Single neuron isolation from continuous waveform microelectrode recordings is shown in figure 3 (supplementary material). The cross-correlation analysis to confirm the putative neural location within hippocampal cell layers is shown in figure 4 (supplementary material). An example stripchart of neural firing synchronized to task events on non-stimulated and stimulated DMS trial stripcharts for the same patient (selected for four neurons each, CA3 and CA1) is shown in figure 5 (supplementary material). Finally, figure 6 of the supplementary material shows the continuous waveform (30 spikes s⁻¹), microelectrode EEG recordings of the stimulation artifact for a single stimulation trial. Additional analysis detail and discussion is provided in the captions to the supplementary figures.

MIMO model

Nonlinear dynamic MIMO models were built for each subject (n = 15) to capture the hippocampal CA3-CA1 input–output properties (Song et al 2016). The MIMO model took the form of the sparse generalized Laguerre–Volterra model, which is a combination of a Volterra model and a generalized linear model. The MIMO model identified the transformation from the CA3 spatiotemporal firing patterns to the CA1 spatiotemporal firing patterns (Song et al 2016). This identification was formulated as the estimation of the MIMO model that could be decomposed into a series of multi-input, single-output (MISO) models with physiologically plausible components expressed by the following equations:

$$w=u(k,x)+a(h,y)+\epsilon (\sigma ),$$

$$y=\{\begin{array}{ll}0& \text{when}w<\theta \\ 1& \text{when}w⩾\theta \end{array}$$

$$\begin{gathered} u(t)=k_0+\sum _{n=1}^N\sum _{\tau =0}^M{k}_1^{(n)}(\tau )x_n(t-\tau ) \\ +\sum _{n=1}^N\sum _{{\tau }_1=0}^M\sum _{{\tau }_2=0}^M{k}_2^{(n)}\left({\tau }_1,{\tau }_2\right)x_n\left(t-{\tau }_1\right)x_n\left(t-{\tau }_2\right) \end{gathered}$$

$$\begin{gathered} a(t)=\sum _{\tau =1}^M{h}_1(\tau )y(t-\tau ) \\ +\sum _{{\tau }_1=1}^M\sum _{{\tau }_2=1}^M{h}_2\left({\tau }_1,{\tau }_2\right)y\left(t-{\tau }_1\right)y\left(t-{\tau }_2\right). \end{gathered}$$

Variables x and y are CA3 (input) and CA1 (output) spike trains. k are feedforward Volterra kernels which describe the transformation from x to the post-synaptic potential u. h’s are feedback Volterra kernels which describe the transformation from preceding y to the output spike-triggered after-potential a. The zeroth-order kernel k₀ captures the input-independent baseline firing rate. First-order feedforward kernels $k_1^{(n)}$ describe the linear relation between the nth input x_n and u, dependent on the time intervals τ between the past and the present time. Second-order feedforward kernels $k_2^{(n)}$ describe the second-order nonlinear interaction between pairs of spikes in the nth input x_n as they jointly affect u, in addition to their individual first-order effects. First-order feedback kernel h₁ and second-order feedback kernel h₂ can be interpreted similarly by treating the preceding y as an extra input. N is the number of inputs. M is the system memory length. The total synaptic potential u is added with the feedback after-potential a, and a Gaussian noise ε with standard deviation σ, to form the pre-threshold potential w. When w crosses threshold θ, an action potential is generated.

Laguerre basis functions and regularized estimation were used to optimize the model complexity to avoid overfitting (Song et al 2013). All models were validated with a multifold cross-validation procedure. Model goodness-of-fit was evaluated with an out-of-sample Kolmogorov–Smirnov test based on the time-rescaling theorem. During the model estimation phase, MIMO model coefficients were estimated with the hippocampal CA3 (input) and CA1 (output) spike trains. During the MIMO stim phase, hippocampal CA1 spike trains were recursively predicted from the ongoing CA3 spike trains based on the estimated MIMO coefficients.

Neuronal stimulation

Upon completion of the MIMO model computation, parameters for online MIMO calculation were sent back to the WFBMC patient site for input to a Matlab (Mathworks, Natick, MA) script that continuously predicted output (putative CA1) spike discharges from online input (putative CA3) cell firing using the neural signals recorded from those same locations during the first DMS recording session. Note that data from nine previous subjects was used to develop and valid ate MIMO modeling (Song et al 2016) prior to application of the model to the data reported here. Once the model was confirmed, the seven DMS subjects progressed to closed-loop MIMO stimulation testing.

Approximately 8 d after surgery (after seizure focus identification procedures were complete—see table 3 of the supplementary material), the subjects (n =8) were once again connected to the Blackrock system, and recording templates from the first session (2–4 d previously) were used to isolate as many of the same CA1 and CA3 cells as possible. The input spike trains were analyzed online by the MATLAB MIMO script, with the output prediction binned by a recording channel at 50 ms intervals and sent to a Blackrock CereStim 96 channel microstimulator connected to the MIMO model output (putative CA1) sites on the macro–micro electrodes configured to deliver stimulation pulses with a minimum of 50 ms between pulses (i.e. ⩽20 stimulation pulses per second) per channel.

The online model prediction of output discharge patterns was continuous, but model predicted stimulation (MIMO stim) was gated to occur only within a 4.0 s interval after the SP on selected DMS trials. Once recording and online MIMO stim was set up, a new DMS test session (see below) was performed with MIMO model-controlled electrical stimulation (1 V, 150 μA, biphasic pulses 1 ms per phase in duration, 4 sec total pattern duration) delivered on 30–35 randomly-selected DMS trials within the session. A control stimulation pattern (rand stim) consisting of pulses of the same amplitude on the same channels, at the same frequency, in a random (non-MIMO) generated pattern (see figure 6), was also delivered on 30–35 other DMS trials intermixed with no stim and MIMO stim trials in the same DMS session. MIMO stimulation patterns and timings were logged for further analysis. DMS trial events, including stimulation occurrence and type of stimulation, were recorded for analysis of the effect of stimulation on behavioral performance.

Figure 6.

Operation of the multi-input, multi-output (MIMO) model during memory encoding in the DMS task. MIMO heat maps indicate input and output neural firing from a single subject organized by the neuron/recording channel versus time (s), from the sample presentation (SP) to sample response (SR). Input (CA3, left) and output (CA1, top center) recordings are shown along with output firing predicted by the MIMO model (bottom center) for the same recording interval within a single DMS trial. MIMO-based stimulation patterns (upper right) were derived from MIMO-predicted firing binned at 50 ms intervals per channel (Berger et al 2011, Hampson et al 2013). A heat map representing a typical random (RAND) control stimulation is shown for comparison (lower right). The color scale (inset) indicates the firing rate for neural recordings, and the probability of firing for output (CA1) prediction. Stimulation (2 ms biphasic, 1 V, 150 μA per pulse) is indicated in red on the stimulation heat maps on the right.

Closed-loop stimulation in the DMS/DR tasks

Subjects (n =2 DMS only, 5 DMS+DR, 1 DR only ) were tested with MIMO closed-loop micro-stimulation of hippocampal electrode sites during a second DMS test session (session 2). This session consisted of 75–100 DMS trials performed under the same conditions as in session 1. To test the MIMO model, the DMS task utilized non-verbalizeable color block images only. Session parameters consisting of trial delays and the number of images in the MP were the same as in session 1, and produced similar behavioral outcomes (see yellow and red bars, figure 7). Approximately 50% of the trials in the second DMS session received MIMO or random stimulation during the sample phase (see Neuronal Stimulation, above). Stimulation trials were distributed randomly with no more than two consecutive stimulation trials in the DMS session. The ratio of trials with MIMO versus random stimulation was 60% MIMO/40% random, with not more than 35 trials of each type per 100 trial DMS session.

Figure 7.

Bar graphs depict mean (±SEM) DMS task performance in the first recording session (yellow bars—rec only) and in the second recording session consisting of intermixed MIMO stimulated (blue bars—MIMO stim), nonstimulated (red bars—no stim) and random pattern stimulated (green bars—rand stim) trials. Top: results for overall DMS performance (i.e. across all delays) for each subject Px14-Px21. MIMO-based stimulation and random pattern stimulation were delivered for 4 s during the sample/encoding phase. The normal performance range computed for ten non-patient subjects is shown by the dashed horizontal lines. Note that rand stim trials were not tested (NT) in subject Px16 due to clinical time constraints. Bottom: mean (±SEM) DMS performance across subjects Px14–Px20 computed by normalizing performance across subject-specific delay intervals (1–20 s for subjects Px14 and Px15, 1–30 s for subjects Px18 and Px19, 1–50 s for subject Px17 and Px20, and 20–70 s for subject Px16. The performance is computed and plotted in 20% increments of the respective maximum delays. The color code is the same as above. Horizontal bars and asterisks on both graphs indicate significant pairwise comparisons between conditions (see methods and results). *p < 0.01; **p < 0.001.

The DR task utilized at WFBMC was developed after the DMS testing of patients was completed. For the DR task, five subjects (Px16-Px20) performed an additional 100 DMS ‘training’ trials with short 1–10 s delays utilizing clip-art/photo images with MIMO or random stimulation delivered during the sample phase of DMS trials in the same ratio as above. One subject—Px21—was tested solely with the DR task. DR ‘test’ trials commenced 15 min after completion of the second set of DMS trials. No stimulation was delivered during DR trials.

Statistical analysis

DMS and DR trial performance was initially analyzed via a repeated measures analysis of variance (RMANOVA) (PROC GLM, SAS, Research Triangle Park, NC). Each single trial outcome (correct versus error) in the DMS task was classified by the normalized maximum (1–20, 21–40, 41–60, 61–80, 81–100%) of DMS trial delay, ranging from 1–20 s to 1–75 s per subject; stimtype of trial (rec only (i.e. session 1), MIMO stim, no stim, rand stim), and subject. Initial analysis of the DMS data showed a significant between-subjects effect of stimtype: F_(2,16) =3.93, p =0.04, as was the within-subjects effect of delay: F_(4,64) =3.24, p =0.02 (p =0.04 when adjusted for multiple comparisons by the Greenhouse–Geisser epsilon). There was no significant within-subject interaction of delay and stimtype: F_(4,64) =0.82, p > 0.50. However, it should also be noted that the correlation matrix did not show uniformly high correlation of within-subject effects, and the sphericity tests were highly inconsistent with assumptions of repeated measures: $X_{(9)}^2=294$, p < 0.001.

Each single trial outcome in the DR test (correct versus error) was similarly classified by actual delay (1%–25%, 26%–50%, 51%–75%, 76%–100% of maximum DMS-DR trial interval, which ranged from 20–75 min per subject), stimtype within the initial DMS trial (MIMO stim, no stim, rand stim), and subject. As above, the data was initially analyzed via RMANOVA, yielding a significant between-subjects effect of stimtype: F_(2,9) =11.66, p =0.003; a trend toward within-subject subject effects of delay: F_(4,36) =2.24, p =0.08 (p =0.13 when adjusted for multiple comparisons by the Greenhouse–Geisser epsilon), and no significant within-subjects interaction of delay and stimtype: F_(4,36) =0.95, p > 0.45. Again, the correlation matrix did not show uniformly high correlation of within-subject effects, and the sphericity test yielded an abnormally large result: $X_{(9)}^2=294$, p < 0.001.

Given the fact that repeated measures analyses indicated that the assumptions of repeated measures were not warranted, conventional two-way ANOVA for stimtype, delay and interaction were also run and are reported in the next section. Error bars/statistical comparisons on bar graphs were computed using pairwise linear contrasts within the ANOVA models, corrected for multiple comparisons via Scheffe’s method.

Results

Figure 1 shows high resolution T2-weighted pre-surgical 3-Tesla MRIs (left column) and T1-weighted pre-surgical (center) and post-surgical (right) 3T MRIs for a representative patient in this study. Figure 2 enlarges the mesial temporal lobe regions from figure 1 to illustrate the depth electrode placement utilized for all subjects. A pre-implant MRI (figures 2(A) and (B)) was obtained approximately 3 months prior to surgery, while the post-implant MRI (figure 2(C)) was obtained 6 h after completion of surgery. Figure 2(D) shows the computed tomography (x-ray) of the multiple depth electrodes utilized for ‘stereo-EEG’ phase II seizure monitoring in the same patient, while figure 2(E) is a schematic of the macro–micro (AdTech Medical, Racine, MI) depth electrode positioning relative to the hippocampal cell layers for comparison with the appearance of ‘beads-on-a-string’ shadowing produced by the evenly spaced EEG ‘macro’ recording sites in the post-surgical MRI (figure 2(C)). ‘Micro’ recording sites for single neuron recording are not visible in the MRI but are illustrated in figure 2(F). Each ‘macro–micro’ depth electrode contained ten sites capable of isolating and differentiating individual hippocampal neuron firing at depths spanning the CA1 and CA3 hippocampal cell layers. The subject received bilateral depth electrodes in the anterior and posterior hippocampus, as shown in the post-implant axial MRI (figure 1). Table 1 provides details of the total number of patients enrolled, implanted and tested in this study.

Short-term memory: DMS task

To establish the MIMO model (figure 3) it was necessary to first record hippocampal neurons from patients with depth electrodes during the initial performance of a DMS short-term memory task. Neurons were recorded during the task, and then sorted according to firing rate, and DMS event firing correlates (figures 4 and 5). Neurons which met the criteria consistent with the hippocampal principal cell baseline firing rate (0.5–5.0 Hz) and showed transient increases up to 20 Hz synchronized to DMS events, were retained for analysis. Other neurons with slightly higher baseline firing rates (5–10 Hz) that also exhibited DMS-related firing correlates (figures 4 and 5), and neurons with a 0.5–5 Hz baseline firing rate and no DMS firing correlates were also analyzed. High baseline firing rate (>10 Hz) neurons, and neurons with baseline firing rates >5 Hz but with no DMS correlates were excluded from the task-related analysis. Table 2 lists the total, mean per patient, mean per electrode, and minimum/maximum neuron ratios, recorded from putative CA3 and CA1 microelectrode sites across subjects.

Figure 3.

Multi-input, multi-output (MIMO) model utilized for the simulation of a hippocampal neural prosthesis. (A) The MIMO model (left) is a series of multi-input single-output (MISO), physiologically plausible spiking neuron models (right). Each MISO model takes the form of the sparse generalized Laguerre–Volterra model that can be viewed as a combination of the Volterra model and the probit generalized linear model.

Figure 4.

Peri-event histograms (PEHs) of putative CA1 and CA3 neural firing in response to behavioral events in the DMS task. The mean firing rate within the sample (SP—sample presentation to SR—sample response) and match (MP—match presentation to MR—match response) phase across 167 putative CA1 and 191 putative CA3 neurons recorded from the anterior hippocampus across all subjects. Single DMS trials were sorted according to correct versus error outcome, and averaged to yield mean PEHs of the task-specific firing rate (FR). Differences between correct and error trials ±0.5 s around the SP, SR, MP and MR events were computed and analyzed using standard score (z =(peak correct FR—mean error FR) ÷ pooled standard deviation 1.5–0.5 s prior to SP) to identify significant differences between trial types within the sample and match phases of the task. Asterisks and horizontal bars: *z > 2.2, p < 0.01, **z > 3.1, p < 0.001 reflect the differences between peak neural firing on correct versus mean firing in error trials.

Figure 5.

Heat map plots depict the consistency between mean ensemble PEH firing correlated to the DMS task during the first recording session (typically 48–72 h after implant surgery), and recordings from the same electrode locations using the same neuron isolation parameters during the second recording session (typically 5–8 d after surgery). Output (putative CA3, top) and input (putative CA1, bottom) neurons were localized by electrode position and imaging. The vertical position on the axis indicates neuron order sorted according to the intensity of SR and MR firing in the first DMS recording session. The color scale (inset) indicates the firing rate. Neural firing was averaged over 100 DMS trials per session; all neurons shown were recorded from the same subject (subject Px17). The dashed white vertical bars indicate the occurrence of SR and MR, respectively; the solid white bar indicates discontinuity produced by the variable duration of delay between the sample and match phases across the trials.

Table 2.

Hippocampal neurons recorded from eight patients reported in this study were sorted into putative CA3 and CA1 neurons by location on the electrode (figure 2(F)) and further described by the firing rate. The neural firing rate was scored as a high probability of being a pyramidal neuron (0.5–5.0 Hz), moderate probability of being a pyramidal neuron (5.0–10.0 Hz) and low probability/unlikely to be a pyramidal neuron (>10 Hz). Activity recorded on the microelectrode sites that exceeded 20 Hz was rejected. The table provides the mean (±SEM) number of neurons recorded per putative hippocampal cell layer as well as the minimum and maximum number of cells recorded, and is further organized per subject and per electrode. The totals reflect the total number of cells recorded across all eight subjects reported in this study.

Neuron recording and identification
	Total	Mean ± SEM # recorded	Minimum # recorded	Maximum # recorded
Per subject (n = 8):
# Putative CA1 neurons	170	21.2 ± 2.8	10	45
0.5–5.0 Hz	128	16 ± 3.3	4	43
5.0–10.0 Hz	26	3.2 ± 0.7	0	8
10.0–20.0 Hz	16	2 ± 0.5	0	6
# Putative CA3 neurons	179	22.3 ± 3.5	9	53
0.5–5.0 Hz	137	17.1 ± 3.4	5	47
5.0–10.0 Hz	22	2.7 ± 0.6	0	8
10.0–20.0 Hz	20	2.5 ± 0.4	0	5
Per electrode (n = 18):
# Putative CA1 neurons	—	9.4 ± 0.5	6	13
0.5–5.0 Hz	—	7.1 ± 0.9	1	13
5.0–10.0 Hz	—	1.4 ± 0.3	0	4
10.0–20.0 Hz	—	0.8 ± 0.3	0	5
# Putative CA3 neurons	—	9.9 ± 0.6	5	15
0.5–5.0 Hz	—	7.6 ± 0.7	2	13
5.0–10.0 Hz	—	1.2 ± 0.3	0	5
10.0–20.0 Hz	—	1.1 ± 0.2	0	4

Hippocampal principal cells (i.e. CA3 and CA1 neurons) were utilized to compute a nonlinear MIMO model (Song et al 2016) of the feedforward CA3 to CA1 neural circuit to obtain predicted patterns of putative CA1 (output) neuron firing driven by putative CA3 (input) firing on correct DMS trials. During the second DMS test session, input neurons were recorded and, using the same neural isolation as session 1 (figure 5), an on-line computation of output neuron firing statistics was used to determine patterns of microstimulation applied to output sites during the same session (see figure 1 of the supplementary material for a schematic illustration). Session 1 DMS task recording sessions were performed on 15 subjects in order to develop and test the MIMO model. Neuron designation utilized the recording channel (i.e. position on the probe) as an identifier coupled with high-resolution post-implant MRI to assign putative input and output locations which were subsequently confirmed via feed-forward cross-correlation and nonlinear kernels in the MIMO model.

Each subject provided 80–100 DMS trials with hippocampal neural activity recorded from each microelectrode location. Recorded neuron data was sorted by location and firing rate (table 2) from the first recording session (figure 5, left), and was subject to analysis and characterization of task-related hippocampal cell firing via a customized MIMO nonlinear model (figures 3 and 5) developed by Drs Song and Berger et al USC. The MIMO model computed nonlinear firing relationships between input–output neuron pairs (Song et al 2016) that occurred during the sample phase of the DMS task associated with correct choices in the subsequent MP of the same trials (figure 6). The MIMO model parameters for each subject were derived and returned to the study team within 48 h, which allowed re-testing of subjects in a second recording session (figure 5, right) with online application of the derived MIMO model (figure 6) for electrical stimulation of the same neural substrates recorded in correct trials in the previous DMS memory task (see Berger et al (2011) and Hampson et al (2013)).

The second DMS test session for each subject (2–5 d after the first session) utilized the derived nonlinear MIMO model parameters for the online prediction of output (CA1) neuron firing from simultaneously recorded online input (CA3) neuron firing (figure 6). As previously demonstrated in rodents and NHPs (Berger et al 2011, Hampson et al 2013, Deadwyler et al 2017), electrical micro-stimulation pulses (150 μA, biphasic, 1.0 ms per phase) were delivered in the same spatiotemporal pattern to the output (CA1) electrode sites identified by the MIMO model that corresponded to the firing patterns of the same cells recorded in the sample phase in correct trials in the first session. In this second DMS session, stimulation was delivered during the sample phase of the task (figure 6), which allowed the effects of MIMO-derived stimulation to be compared directly to performance by the same subjects on trials without stimulation (figure 5 of the supplementary material illustrates neural firing in non-stimulated and stimulated DMS trials).

Figure 7 (top) shows the performance of all seven subjects tested in the second DMS test session in which MIMO stim was delivered in the sample phase on some trials, compared with trials in the same session with no stimulation (no stim). In addition, figure 7 also shows the effects of electrical stimulation that was not MIMO model-based, delivered randomly (rand stim) in different trials in the same period of the sample phase. Performance was enhanced maximally in trials with MIMO stimulation compared to the other three test conditions, including performance in the first DMS session without stimulation (figure 7, rec only). Thus, MIMO model-based stimulation was effective in activating a pattern of neuron firing which corresponded to correct performance and was less likely to occur in no stim (control) trials. The lower graph in figure 7 shows performance as a function of trial delays averaged across subjects Px14–Px20 (shown in the upper graph), and indicates that reduced performance on more difficult trials with increased delays was facilitated if that trial received MIMO stim.

An important control for the effectiveness of MIMO model-derived stimulation during the task was the direct comparison with trials in which stimulation was delivered randomly (rand stim) at the same intensity and frequency in a non-MIMO pattern across the same output electrode sites at the same time interval in the sample phase of the DMS trial (figure 7). Rand stim delivery to each of the output sites was therefore unrelated to the correlated input–output firing patterns extracted by the MIMO model during encoding in correct trials in the first DMS session. Figure 7 shows that performance with stimulation delivered in this random manner (rand stim) did not improve performance over control (no stim) levels, and in some cases even caused performance to decrease below nonstimulation control levels, suggesting that random stimulation may have actually impaired effective encoding when delivered to the same recording locations selected by the MIMO model. This result also confirms that simple electrical excitation was not a factor in improved memory encoding, since rand stim was delivered with the same net frequency and intensity as ‘pattern-specific’ MIMO stim. These results show that the stimulation of multiple output sites in the same pattern derived by the MIMO nonlinear model was the most effective process for encoding correct trials by the same human subjects in the second memory test. (Furthermore, facilitation of DMS performance was not limited to patients with either ‘normal’ or impaired memory function, nor localization of seizure foci, as indicated by the patient status provided in table 3 of the supplementary material.)

Statistical analysis via two-way ANOVA (overall: F_(14,97) =2.12, p < 0.02) yielded a significant main effect of the type of stimulation (rec only, no stim, MIMO stim, rand stim): F_(2,97) =10.50, p < 0.001, but no overall significant effect of delay: F_(4,97) =1.36, p =0.25 or interaction between stimtype and delay: F_(8,97) =0.41, p =0.91. (See Methods for the full statistical model.) The asterisks in figure 7 indicate significant pairwise comparisons via linear contrast. It is important to note that subject Px19, who showed little to no influence with MIMO stimulation (figure 7, top), was the only stimulated subject to have just a single unilateral macro–micro electrode, with only five output and ten input neurons (see table 3 of the supplementary material). In contrast, subjects Px14–Px18 and Px20 all had bilateral implants, with an average of 25 output and 27 input neurons per subject.

Enhanced retrieval: DR task

An additional method of assessing short-term memory was employed in which the retrieval of previous trial information was assessed over a much longer interposed delay interval. In this subsequent DR test session, the recall of sample images from trials in a recently performed and completed DMS task was assessed over a much longer time period (40–60 min) in a random manner. This DR paradigm (see methods) utilized a standard DMS task with verbalizeable clip-art and photographic images as a training set. The recognition of images within test sets consisting of one sample, one nonmatch and one novel image per trial started 15 min after completion of the DMS training task. The novel image was not presented in any of the previous DMS trials, training sets or test sets. All images from the DMS training trials were randomized with respect to the order in which they occurred in the previous DMS task. The delay from the DMS training trial to the respective recognition test for sample images varied from 20–75 min. Figure 8 shows the performance results of six subjects in which three DMS stim conditions (MIMO stim, no stim, rand stim) were delivered in the sample phase in previous DMS training trials. It is very clear in figure 8 that images presented in MIMO stim DMS trials were recognized more frequently and over longer ‘recognition-intervals’ (from 30 to more than 75 min) than images without MIMO stim (no stim or rand stim). Thus, MIMO stim also enhanced encoding for recognition extended over longer delays than in the initial DMS session, irrespective of patient characteristics (see table 3 of the supplementary material), or even though additional MIMO stim was not administered in the longer DR test.

Figure 8.

Bar graphs depict mean (±SEM) DR task performance in test sessions in which MIMO stim (blue bars—MIMO stim), no stimulation (red bars—no stim) and random pattern stimulation (green bars—rand stim) were presented during the DMS training session. Top: results for overall DR performance (i.e. across delays) for subjects Px16–Px21. Note that subjects Px14 and Px15 were not tested in DR due to low initial memory performance (figure 6) and rand stim trials were not tested (NT) in subject Px16 due to clinical time constraints. MIMO-based stimulation and random pattern stimulation were delivered for 4 s during the sample/encoding phase of the DMS training session, not during the DR recall session. Bottom: mean (±SEM) DR performance across all four subjects tested, sorted by interval between DMS training and DR test for a given trial/image. Mean Correct recognition (see Methods) is computed and plotted in 15 min increments of the respective DR delays. The color code is the same as above. Horizontal bars and asterisks on both graphs indicate significant pairwise comparisons between conditions (see methods). *p < 0.01; **p < 0.001.

Figure 9 provides a summary of the overall effects of stimulation on both DMS and DR memory tasks. A separate statistical analysis via two-way ANOVA of the DR data again yielded a significant overall effect (overall: F_(14,73) =3.05, p < 0.001), with significant main effects of type of stimulation: F_(2,73) =11.54, p < 0.001, and delay: F_(4,73) =2.88, p =0.03. There was no significant interaction between stimtype and delay: F_(8,73) =1.02, p =0.43. (See Methods for full statistical model.) The asterisks in figures 7–9 indicate significant pairwise comparisons via linear contrast. Note that subject Px19 showed a slightly significant (p < 0.01) effect of MIMO stimulation (figure 8, top) in DR, despite no significant effect of MIMO stimulation in the DMS task (figure 7).

Figure 9.

Summary bar graphs of overall DMS (left) and DR performance across all subjects (DMS: n =7, DR: n =6) and test conditions. The color code is the same as in figures 6 and 7. Horizontal bars and asterisks on both graphs indicate significant main effects comparisons between the ‘stim type’ levels in the respective ANOVAs (see Methods and Results). *p < 0.01; **p < 0.001.

Discussion

The results shown here represent the first demonstration in humans of nonlinear model (MIMO) derived electrical stimulation patterns successfully applied to hippocampal regions responsible for the encoding of task-specific information. Importantly, these results provide the means to apply the same procedure to counteract memory impairment in humans due to disease and aging. It is clear that the derived MIMO stim patterns obtained in these subjects have the required specificity to enhance encoding in difficult retention tasks; however, the fact that the same nonlinear MIMO model extracted functionally specific patterns across different human subjects provides unique evidence for the potential to detect and characterize an inherent human memory process. Therefore, it is possible that the same principles can be applied to the development of a memory prosthesis for individuals with different types of brain disorders, as long as the hippocampal circuitry underlying information encoding is operative and can be assessed via either (a) in vivo electrodes, or possibly in the future, (b) noninvasive detection methods (i.e. functional neuroimaging).

The marked effects of induced MIMO stim across all patients, even though there were considerable differences in background and clinical conditions under which MIMO stim was delivered, provides further support for the effective use of enhancing memory across a number of different medical conditions that involve hippocampal function. The patient characteristics (supplied as table 3 in supplementary material) shows that some patients were even classified clinically with memory deficits and yet these patients (Px14, Px15, Px21) were facilitated as much by MIMO stim as those that did not have the same degree of clinically evaluated memory impairment. This same distinction applied to the involvement of the hippocampus in seizure generation (again see table 3 in supplementary material). Finally, and possibly most importantly, even patients showing structural disruption with respect to the hippocampus in their respective MRIs (Px18, Px21) were also facilitated in the memory task performance by MIMO stim. Thus, the wide range of patient conditions that affected hippocampal function did not negate the ability of the MIMO model-based stimulation to enhance encoding and subsequent recall, even in patients previously classified as having memory deficits via clinical evaluation methods.

The results presented here contrast with several previously published attempts to alter hippocampal memory processing in humans using micro-electrical stimulation. Direct stimulation of the human hippocampus with a fixed frequency to all cells, similar to that employed with deep brain stimulation, typically produces memory impairment (Halgren et al 1985, Jacobs et al 2016), even though two studies demonstrated either facilitated memory (Suthana et al 2012), or reduction in the mnemonic decline associated with Alzheimer’s disease (Laxton and Lozano 2013). We note that a key distinction between the results presented here and previous attempts lies in the recording and integration of neural ensembles, not just the analysis of single neuron firing patterns or local field potentials.

Another major distinction is that randomized, non-MIMO stimulation of the same locations (rand stim) during the same memory task did not facilitate (and often impaired) DMS performance (figures 7–9), which served as an important negative control for any nonspecific effects of MIMO stim. However, MIMO stimulation patterned from the native spatiotemporal connectivity of input and output neurons extracted by the MIMO model facilitated short-term memory in the form of DMS performance (figure 7) as well as longer-term memory in the form of DR performance (figure 8). This result may explain why both the direct stimulation-induced impairments cited above, as well as similar effects shown in a recent study in humans (Jacobs et al 2016), are not effective, since hippocampal memory encoding requires specific spatiotemporal relationships with respect to within and between cell layer neural firing, i.e. neural ‘codes’ (Berger et al 2011, Song et al 2014, Deadwyler et al 2017). Hence, fixed frequency or randomized stimulation applied directly to the hippocampus likely disrupts (Jacobs et al 2016, Ezzyat et al 2017) the generation of the specific neural codes underlying particular memory representations, as has been recently shown with stimulation in the interval between encoding and retrieval (Merkow et al 2017). Consequently, it is possible that only stimulation patterns that are consistent with natural-occurring neural codes can successfully restore memory function in the hippocampus (Hampson et al 2011).

Continued testing and application of the closed-loop memory prosthetic proposed here also warrants sorting and testing of ‘strong’ versus ‘weak’ encoded DMS and DR trials. Figure 7 (bottom) shows that MIMO-stimulation produced facilitation even on short delay trials, which had the highest percentage of correct performance in most subjects. Hence, rand stim trials (as mentioned above) and short versus long delay trials do indeed provide these important ‘negative control’ conditions.

Recent studies have reported testing a different model for facilitating human memory (Jacobs et al 2016, Ezzyat et al 2017). Both studies tested stimulation in a subset of subjects and did not report consistent positive results. For example, Ezzyat et al (2017) showed significant effects—either positive or negative—in only eight subjects tested. We report here stimulation testing in a total of eight subjects, with all subjects showing significant positive results with closed-loop MIMO stimulation. Also, Px19 exhibited no significant effect of MIMO stim in the DMS task, nevertheless demonstrated significant facilitation in DR.

At present, there are no other reports of techniques or agents for humans that can be applied to facilitate specific neural codes for memory. In the past, drugs that were shown to facilitate memory in humans may have provoked a similar type of CA1 pattern-specific firing. Development of a neural prosthetic based on spatiotemporal modeling and hippocampal stimulation, as shown here, can provide a device adapted to restoring and facilitating memory across human patients. Development and use of the MIMO model recording and stimulation methods in the memory prosthesis device described here could allow application of the same procedures to effectively counter acute injury, or chronic disease-related, mnemonic dysfunction in humans.