A Comprehensive Model-Based Framework for Optimal Design of Biomimetic Patterns of Electrical Stimulation for Prosthetic Sensation

Abstract

Objective.

Touch and proprioception are essential to motor function as shown by the movement deficits that result from the loss of these senses, e.g., due to neuropathy of sensory nerves. To achieve a high-performance brain-controlled prosthetic arm/hand thus requires the restoration of somatosensation, perhaps through intracortical microstimulation (ICMS) of somatosensory cortex (S1). The challenge is to generate patterns of neuronal activation that evoke interpretable percepts. We present a framework to design optimal spatiotemporal patterns of ICMS that evoke naturalistic patterns of neuronal activity and demonstrate performance superior to four previous approaches.

Approach.

We recorded multiunit activity from S1 during a center-out reach task (from proprioceptive neurons in Brodmann’s area 2) and during application of skin indentations (from cutaneous neurons in Brodmann’s area 1). We implemented a computational model of a cortical hypercolumn and used a genetic algorithm to design spatiotemporal patterns of ICMS (STIM) that evoked patterns of model neuron activity that mimicked their experimentally-measured counterparts. Finally, from the ICMS patterns, the evoked neuronal activity, and the stimulus parameters that gave rise to it, we trained a recurrent neural network (RNN) to learn the mapping function between the physical stimulus and the biomimetic stimulation pattern, i.e., the sensory encoder to be integrated into a neuroprosthetic device.

Main results.

We identified ICMS patterns that evoked simulated responses that closely approximated the measured responses for neurons within 50 μm of the electrode tip. The RNN-based sensory encoder generalized well to untrained limb movements or skin indentations. STIM designed using the model-based optimization approach outperformed STIM designed using existing linear and nonlinear mappings.

Significance.

The proposed framework produces an encoder that converts limb state or patterns of pressure exerted onto the prosthetic hand into spatiotemporal patterns of ICMS that evoke naturalistic patterns of neuronal activation.

INTRODUCTION

Loss of touch and proprioception, e.g., through sensory neuropathy, causes significant movement deficits, and the absence of such sensory feedback, even in the presence of vision (Sainburg et al. 1993), limits the function of prosthetic devices (Bensmaia and Miller 2014). Stimulation of peripheral nerves or the spinal cord may be appropriate for restoring sensory feedback in amputees, i.e., individuals with an intact spinal cord (Chandrasekaran et al. 2019; D’Anna et al. 2019; George et al. 2019; Graczyk et al. 2016; Ortiz-Catalan et al. 2020; Petrini et al. 2019; Tan et al. 2014; Valle et al. 2018). However, in persons with spinal cord injury, intracortical microstimulation (ICMS) of the somatosensory cortex (S1) is one of the few viable options to provide sensory feedback (Flesher et al. 2016; Salas et al. 2018). ICMS of S1 evokes crude tactile sensations and can restore limited touch and proprioception (Flesher et al. 2016; O’Doherty et al. 2011; Salas et al. 2018; Tabot et al. 2013; Tomlinson and Miller 2016).

There are two broad approaches to generating artificial sensation via ICMS in S1 (Bensmaia and Miller 2014): learning-based and biomimetic. In learning-based approaches, arbitrary patterns of stimulation are used to generate artificial sensation with a goal of training the subject to learn the meaning of those patterns over time (Dadarlat et al. 2015; O’Doherty et al. 2011). Alternatively, patterns of stimulation in biomimetic approaches are intended to mimic the neural activity evoked during natural movements and interactions with objects (George et al. 2019; Valle et al. 2018), ideally requiring less learning on the part of the user. The biomimetic approach requires design of spatiotemporal patterns of ICMS that generate neural activity, and subsequently evoke percepts, as close as possible to the natural firing patterns (Berger et al. 2012; Choi et al. 2016; Dura-Bernal et al. 2016).

The biomimetic approach comes with several challenges (Bensmaia and Miller 2014; Choi et al. 2016; Gilja et al. 2011). First, the relationship between limb state or object interactions and neuronal responses needs to be characterized, which is not possible in deafferented subjects. Second, a mapping is required between the desired patterns of neuronal activation and the spatiotemporal patterns of ICMS. The latter is not expected to be a simple superposition of responses evoked by individual channel stimulation due to nonlinear interactions between stimulation-induced electric fields and neural and synaptic mechanisms (Berger et al. 2012; Hokanson et al. 2018). These nonlinear interactions result in an exceedingly large parameter space across multiple stimulation channels that cannot be searched by brute force

Our goal was to implement a mapping between limb state or object interactions and spatiotemporal patterns of ICMS (STIM) that elicit naturalistic patterns of neuronal activation (Fig. 1). We first recorded multiunit neural activity (SPIKES) in non-human primates that occurred in response to natural somatosensory inputs (PHYS). Second, we used a computational model of neural firing to compute the response to STIM such that the simulated response (EVOKED) approximated SPIKES. Finally, having obtained the desired STIM for many different PHYS conditions, we trained a recurrent neural network (RNN) to implement a mapping from PHYS to STIM. This method culminated in an encoder that compute the STIM to evoke patterns of neural activity that are appropriate to the limb state or contact events. The expectation is that the resulting naturalistic patterns will give rise to more verisimilar and intuitive sensory percepts than those evoked by simple static linear maps.

Figure 1:

Framework for design of optimal spatiotemporal patterns of intracortical microstimulation (ICMS) for biomimetic prosthetic sensation. The framework consisted of three parts. First, we recorded neural response (SPIKES) to naturalistic sensory inputs (PHYS) in non-human primates. Second, we used model-based optimization to design spatiotemporal patterns of ICMS (STIM) such that model-based response (EVOKED) matched SPIKES. Finally, we developed a mapping function between PHYS and STIM by training a recurrent neural network to predict optimal STIM during continuously varying PHYS.

METHODS

Overview

We recorded multiunit activity (SPIKES) from Brodman’s area 2 of S1 in rhesus macaques during a planar center-out reaching task while continuously tracking the hand position (PHYS prop). We also recorded SPIKES from Brodman’s area 1 of S1 while applying indentations to the distal pads of digits 3 and 4. We then implemented a computational model of a cortical hypercolumn (the slab model) comprising multicompartment cortical neurons organized in layers. To reduce the computational demand of the model for the purposes of optimization, we extracted the layer-4 neurons from the slab model and combined the resulting simplified sheet model with a genetic algorithm (GA) to design optimized spatiotemporal patterns of ICMS (STIM). We then identified STIM patterns such that model-based responses (EVOKED) matched the measured SPIKES (Fig. 1). Finally, we developed a mapping function between PHYS and STIM by training an RNN. We compared the performance of STIM designed using our optimization framework to STIM designed using four other methods: linear mapping of STIM from SPIKES (Weber et al. 2011), nonlinear mapping of STIM from PHYS (Berg et al. 2013), linear mapping of STIM from PHYS (Flesher et al. 2016), and linear mapping of STIM from $\frac{d}{dt}(PHYS)$ (George et al. 2019).

Neural Response to Naturalistic Proprioceptive inputs

All surgical and experimental procedures were approved by the Institutional Animal Care and Use Committee of Northwestern University. Two male rhesus macaque monkeys (Monkey C: 9.8 kg and Monkey H: 12.5 kg) were trained to sit in a primate chair and grasp the handle of a 2-D planar manipulandum that controlled a computer cursor on an LCD screen. The monkey’s hand was confined to a plane, but the posture of the arm was not constrained. A trial began when the monkey moved the cursor to a target in the center of the screen. After a center-hold period of random duration between 400-1200 ms, a 3 cm target was displayed in one of eight outer positions at a radial distance of 12 cm. The monkey was required to wait throughout the delay period (400-1200 ms), and when the go cue was presented, to reach to the outer target within one second, then hold there for 500 ms to receive a liquid reward. We computed the position of the handle from encoders mounted in the two joints of the manipulandum. Manipulandum position and target location were collected using a Cerebus system (Blackrock Microsystems, Inc.). The monkeys were very well trained on the center-out task and conducted this or related tasks intermittently for a couple of years before data was collected for this project. Figs. 2A, B show kinematic data (hand endpoint position and velocity) collected from center-out trials, with movements to targets in different directions displayed in different colors. The number of trials per reach direction across which neural data was collected are shown in Tables S1 and S2.

Figure 2:

Neural response to proprioceptive inputs. 2-dimensional hand endpoint (A) position and (B) velocity during center-out reach task. The task spanned eight reach directions (2 trials per direction) with each direction represented in a different color. (C) Sample neural response (multiunit activity) recorded from Area 2 of S1 using a 96 channel multi-electrode array (UEA) during the center out task with each color representing a different reach direction. The spiking data was binned with a width of 50 ms. The crossed boxes represent unconnected channels. Empty boxes indicate channels with no spiking activity.

Following training, each monkey was implanted with a 1 mm length, 96-electrode, sputtered iridium-oxide Utah microelectrode array (UEA) with an interelectrode spacing of 400 μm (Blackrock Microsystems, Inc.) in the proximal arm representation of somatosensory cortical area 2, near the posterior edge of the post-central gyrus, approximately 15 mm from the midline. Area 2 is a high-order region of S1 that receives a combination of cutaneous and muscle afferent input (Friedman and Jones 1981; Pons et al. 1985b). It is thought to provide a relatively high-level representation of limb state, and can be readily implanted with standard microelectrode arrays (London and Miller 2012; Prud'Homme and Kalaska 1994). Neural data was collected about 2 months after surgery in Monkey H and about 6 months after in Monkey C. Neural waveforms were collected using the Cerebus system. The continuous voltage signal from each electrode in the UEA was processed using 0.3 Hz first order high-pass and 7500 Hz third order low-pass analog filters and then sampled at 30 kHz. The sampled signal was passed through a 250 Hz fourth order high-pass digital filter. Thresholds were set at each channel using a multiple of the RMS amplitude (typically −4.5 to −6.0x) of the recorded signal, and all threshold-crossing events on each channel were recorded to obtain multiunit activity. Fig. 2C depicts the neural discharge during the center-out reach trials, which is distributed across a large fraction of the electrodes and is typical of the response to limb movement.

Neural Response to Naturalistic Tactile Inputs

All surgical and experimental procedures were approved by the Institutional Animal Care and Use Committee of the University of Chicago and are described in detail elsewhere (Tabot et al. 2013). Briefly, one rhesus macaque monkey (10 kg) was implanted with a 1.5 mm long UEA in the hand representation of area 1 in primary somatosensory cortex targeted based on anatomical landmarks. Given that the array was contiguous to the central sulcus and area 1 spans approximately 3-5 mm of cortical surface from the sulcus (Pons et al. 1985a), few if any electrodes were located in area 2. The electrode tips terminated in the infragranular layers of somatosensory cortex, as we have previously shown in postmortem histological analysis with other animals instrumented with identical arrays (Rajan et al. 2015).

Tactile stimuli were delivered to the hand using a stainless steel probe with a 4-mm tip diameter driven by a custom-made shaker motor capable of delivering a time-varying pattern of indentations with micrometer precision. The stimuli consisted of low-pass filtered wide-band noise with frequencies above 20 Hz filtered out, scaled to a root-mean-square amplitude of 367 μm, and lasting 60 s. Such mechanical noise spans the range of frequencies and amplitudes experienced by the skin during activities of daily living and was used successfully to train sensory encoding algorithms for peripheral nerve interfaces (Okorokova et al. 2018). Neural recordings were made during ten repetitions delivered to the distal pads of digits D3 and D4 whose representations were well represented on the array (Callier et al. 2019). Neural data were collected about eight months after surgery. We recorded simultaneously from UEA using a Cerebus system as for the proprioceptive experiments, using a 100 Hz fourth order high-pass digital filter. Fig. S1C shows the neural discharge across the 96 channels of UEA during application of the tactile stimulus onto digits D3 (Fig. S1A) and D4 (Fig. S1B).

Computational Model Design

We implemented a computational model of a population of biophysically-based Hodgkin-Huxley style multi-compartment cortical neurons, following an existing thalamocortical (TC) network model (Kumaravelu et al. 2018; Traub et al. 2005). The TC network model included 356 model neurons, each with 50-137 compartments. The cortex was composed of twelve different populations of model neurons, whereas the thalamus comprised two different populations of model neurons. The cortical neurons were arranged in a columnar fashion consisting of six layers with a width of ~400 μm and a depth of ~2000 μm (Fig. 3A): L1 mostly consisted of tufted dendrites arising from the L5 pyramidal neurons; L2/3 included 100 regular spiking (RS) pyramidal neurons, 5 fast rhythmic bursting (FRB) pyramidal neurons, 18 fast-spiking (FS) interneurons and 9 low-threshold spiking (LTS) interneurons; L4 included 24 RS spiny stellate cells; L5 included 80 intrinsic bursting tufted pyramidal cells and 20 RS pyramidal neurons; L6 included 50 RS non-tufted pyramidal neurons, 20 FS interneurons and 10 LTS interneurons. The thalamic population consisted of 10 thalamocortical relay (TCR) cells and 10 reticular nucleus (nRT) cells. Network interactions in the model were through chemical synaptic (AMPA, NMDA and GABA-A) connections (Fig. 3B). The biophysical properties of the cells and the network properties including synaptic connection strengths, type, density, delay, etc., were primarily based on in vivo and in vitro data from rats and are described in detail in the original publication (Traub et al. 2005).

Figure 3:

Biophysically-based computational model of sensory cortex used for model-based design of biomimetic patterns of ICMS. (A) Schematic of the cortical column showing the different populations of neurons across layers 1-6. Each color denotes a specific cell type in the column characterized based on firing behavior. The cortical column model comprised 105 E2/3 pyramidal neurons, 27 I2/3 inhibitory interneurons, 24 E4 spiny stellate neurons, 100 E5 pyramidal neurons, 50 E6 pyramidal neurons, 30 I6 inhibitory interneurons. The thalamus model comprised 10 thalamocortical relay neurons (TCR) and 10 reticular nucleus neurons (nRT). ‘E’ and ‘I’ indicate excitatory and inhibitory neurons, respectively. Neural density of neurons in column was reduced to enhance visibility of individual cell type. (B) Intra-columnar synaptic connectivity showing both excitatory (glutamate – NMDA and AMPA) and inhibitory (GABA-A) connections (Traub et al. 2005). (C) Schematic of slab model comprising 100 cortical columns placed in a 10*10 grid. (D) Schematic of sheet model including only the L4 spiny stellate neurons extracted from the slab model. Each tile in the sheet model (indicated by dashed arrows) included nine L4 neurons placed a 3*3 grid with a spacing of 30 μm.

The single cortical column model was extended to 100 cortical columns placed in a 10*10 grid to accommodate a 10*10 UEA with interelectrode distance of 400 μm (slab model) (Fig. 3C). The slab model comprised a total of 100 * 356 cortical neurons/module = 35,600 neurons with no inter-columnar synaptic connections. Although it may have been desirable to conduct the model-based optimization on the biologically inspired slab model, the process would be computationally prohibitive. Hence, for optimization, but not for evaluation, we reduced complexity by extracting the L4 neurons (59 compartments each – one soma, six axon and 52 dendritic compartments) from the slab model. The resulting sheet model, comprising 900 L4 neurons (nine neurons/column), was used for model-based optimization of STIM (Fig. 3D). The nine neurons within a column were positioned in a 3*3 grid, with the soma of each neuron separated by 30 μm from the nearest off-diagonal neighboring neurons and by ($30\times \sqrt{2}$) μm from the diagonal neighboring neurons. We used the sheet model to optimize STIM since S1 lamina 4 is the most likely location of the electrode tips, and it is the major recipient of peripheral sensory inputs via the thalamus (Mountcastle 1997).

The stimulating tip of each channel of the UEA was approximated as a point current source (McIntyre and Grill 2001). Extracellular potential V_e(i,j) due to the point current source in each compartment (j) of each model cortical neuron (i) was computed using the equation,

$$V_e(i,j)=\frac{1}{4\pi {\sigma }_e}\sum \frac{Isti{m}_1}{r_{1,(i,j)}}+\frac{Isti{m}_2}{r_{2,(i,j)}}+\frac{Isti{m}_3}{r_{3,(i,j)}}+\cdots +\frac{Isti{m}_{96}}{r_{96,(i,j)}}$$

where Istim_k is the STIM across each channel of the UEA, r_i,j is the distance from the stimulating electrode to each compartment of the model cortical neuron, and σ_e is the isotropic and homogeneous conductivity of the extracellular medium (0.3 S/m) (Ranck 1963).

Simulations were implemented in NEURON 7.4 with equations solved using the backward Euler method with a time step of 0.025 ms (Hines and Carnevale 2001) and a total simulation time based on the duration of the relevant modulation of PHYS. The values of Ve were coupled to each neuronal compartment using the e_extracellular mechanism. The network simulation (slab model) was parallelized by dividing the total number of neurons across 100 processors in a round-robin fashion (356 neurons/processor) and by setting up communication of spike times between processors through the Message Passing Interface (MPI) protocol (Hines and Carnevale 2008).

Genetic Algorithm to Design Optimized Patterns of STIM

We coupled a GA with the sheet model to design optimized multichannel patterns of STIM such that responses generated in model neurons (EVOKED) were equivalent to those recorded experimentally (SPIKES) (Fig. 4A). The GA is an evolutionary search heuristic that works well when the input-output (STIM-EVOKED) relationship is highly nonlinear (Davis 1991). The GA operates by emulating natural selection of possible solutions (organisms) using a cost function, and each organism is composed of genes. Here, we implemented a GA where each organism corresponded to a time-varying pattern of ICMS pulses (biphasic symmetric pulses with a fixed width of 200 μs/phase, an interphase interval of 50 μs and pulse repetition rate of 50 Hz) across each channel of the UEA, and each gene in the organism was the amplitude of the ICMS pulse (Fig. 4B). The total number of genes in a single organism ($\frac{lengthofneuraldata}{binwidth}$) was 34 for proprioceptive inputs and 40 for tactile inputs. We chose to modulate the amplitude of ICMS rather than frequency or pulse width, since ICMS amplitude conveys most effectively information about contact pressure (Bensmaia 2015; Choi et al. 2016; Tabot et al. 2013). Further, we chose a low ICMS frequency, in contrast to the high frequencies (>200 Hz) used in current brain-machine interface (BMI) studies, as higher frequencies result in greater number of stimulus artifacts and might destabilize primary motor cortex (M1) recordings in a bidirectional BMI (Kim et al. 2015).

Figure 4:

A genetic algorithm (GA) was coupled to the sheet model of ICMS for model-based design of optimized spatiotemporal patterns of ICMS. (A) The process flow of the GA. 24 randomly initialized spatiotemporal patterns of ICMS were tested on the sheet model. The most fit patterns were retained for the next generation (elites), random patterns (immigrants) were introduced in each generation, and the remaining patterns were replaced with “offspring” from the most fit patterns of the prior generations, subject to random mutations. The GA iterated through successive generations until the cost function was reduced. (B) Sample organism representing an ICMS train with each gene of the organism representing the amplitude of the ICMS pulse. (C) Mating through uniform crossover with ~50% of genes from each of the parent organisms. Parents for mating were chosen based on fitness using proportionate roulette wheel selection, i.e., parents with higher fitness levels had a greater chance of being chosen. Randomly chosen genes from each of the offspring organisms were subject to Gaussian mutation. (D) Evolution of cost function of one channel in the UEA over generations. The GA continued through 25 generations until the cost function was substantially reduced. Blue and red traces show the minimum and maximum costs, respectively, and the black trace represents the median across the population comprising of 24 organisms. (E) Evolution of cost function across all 96 channels in the array over generations. A cost function was associated with each channel of the array. The plot shows the minimum cost for each channel in the array. At the end of maximum allotted 25 generations, the ICMS patterns with lowest cost across the 96 channels in the array was deduced as the optimal spatiotemporal STIM. (F) Recurrent neural network used to learn the mapping from PHYS to STIM for PHYS prop. There were two neurons in the input layer, one each for the 2D hand endpoint position coordinates. The hidden and output layers were comprised of 96 neurons with each neuron in the output layer representing a channel of the array. The output of the hidden layer was fed back as input to hidden layer using two-tap delay units. Hyperbolic tangent sigmoid and linear units were used as activation functions for hidden and output layers, respectively. The same RNN structure was used for PHYS tact except there was only one neuron in the input layer representing indentation onto a digit.

One requirement of the GA is to choose an appropriate bin width to quantify the neural data without excessive loss in the resolution of spike timing information, as spike timing plays a prominent role in tactile perception (Saal et al. 2015; Saal et al. 2016). We posited that any change in neural firing rate should correlate with the dynamics of PHYS and $\frac{d}{dt}$(PHYS) (Callier et al. 2019; Daly et al. 2012; Okorokova et al. 2018; Weber et al. 2011). The spectrum of hand endpoint position and velocity had no power at frequencies > 5 Hz, and similarly PHYS tact and $\frac{d}{dt}$(PHYS tact) did not exhibit any power at frequencies > 20 Hz. Therefore, we chose a width of 50 ms to bin SPIKES, bearing in mind that a very short bin width would increase the computational demands of the optimization process. A bin width of 50 ms results in 2.5 stimulation pulses per bin ($binwidth\ast stimulationfrequency={}^{50ms}∕_{20ms}=2.5pulses∕bin$) and the amplitude of ICMS across each channel was therefore modulated every 2.5 pulses, i.e., 3 pulses at one amplitude, followed by two pulses at the new amplitude, followed by three pulses at the new amplitude, etc. (Fig. 4B).

To optimize STIM for a given reach trial (PHYS prop), the GA was initialized with 24 randomly seeded spatiotemporal patterns of ICMS with pulse amplitudes chosen from a uniform distribution ranging from 0 to 15 μA. The population size of 24 STIM patterns was determined by the number of central processing units (CPUs) available in a single compute node. We had 15 compute nodes with 24 CPUs/node available on the Duke Compute Cluster (DCC). The total number of trials across reach directions for which we could compute STIM was limited by the number of available compute nodes. At any given time, we had access to 15 compute nodes, and we ran 15 separate GA optimizations on the 15 nodes, with a GA for one trial for the reach direction of 315° and two trials for each of the other seven reach directions for Monkey-C. Similarly, for Monkey-H, we ran one trial for the reach direction of 90° and two trials for each of the other seven reach directions. Each trial in a given reach direction lasted for ~1.7 s. We simulated 7 epochs of 2 s data for PHYS tact onto digits D3 and D4 on 14 compute nodes. We limited the range of amplitudes of the STIM patterns to <15 μA to avoid potential side effects of multichannel stimulation using large current amplitudes (Zaaimi et al. 2013).

For each channel in the UEA, 20% of ICMS patterns from the current generation were carried forward to the next generation (elite patterns), and 5% of patterns in each generated were new randomly generated ICMS pulse trains (immigrants). The remaining 75% of patterns in each generation were offspring resulting from mating between pairs of patterns from the prior generation. Mating was performed through uniform cross-over (mixing ratio of 50%), with parent ICMS patterns selected in proportion to their fitness using roulette wheel selection (Fig. 4C). Amplitudes of eight randomly chosen pulses from each time-varying ICMS pattern across each channel of the UEA were subjected to a Gaussian mutation using a normal distribution with mean equal to the original magnitude and standard deviation of 2 μA (Fig. 4C).

The fitness of STIM patterns was evaluated using the sheet model. The cost function of each channel in the UEA was

$${\text{Cost}}_{\text{electrode}}=\sum _{i=1}^{\#ofbins}\mid SPIKE{S}_i-EVOKE{D}_i\mid$$

where SPIKES_i is the firing rate recorded in vivo to naturalistic sensory inputs (PHYS), EVOKED_i is the total firing rate of the 9 neurons (recorded at soma), and number of bins is the total simulation time divided by the bin width. Each channel of the UEA was associated with an individual cost function, and the GA continued through successive generations until each of the cost functions for the 96 channels converged to its minimum value (Fig. 4D,E) resulting in an optimized spatiotemporal pattern of STIM.

To evaluate the efficacy of the model-optimized STIM pattern, we computed mean-squared error (MSE) between SPIKES and EVOKED for each channel in the UEA,

$$\text{Mean squared error (MSE)}=\frac{1}{\#ofbins}\sum _{i=1}^{\#ofbins}(SPIKES[i]-EVOKED[i]{)}^2$$

Unless otherwise noted, we present MSE data only for the ten most modulated channels (i.e., 10 channels with maximal SPIKES response) to PHYS inputs. Statistical inferences were made on the effect of PHYS prop reach directions on mean-squared error using a one-way analysis of variance (ANOVA). When the omnibus test statistic did not reveal significance at p<0.05, the MSE data across reach directions were pooled together. We also computed mutual information (MI) between SPIKES and EVOKED for each channel in the UEA,

$$\text{MI}(SPIKES,EVOKED)=\mathrm{H}(SPIKES)+\mathrm{H}(EVOKED)-\mathrm{H}(SPIKES,EVOKED)$$

where H(SPIKES) is entropy of SPIKES, H(EVOKED) is the entropy of EVOKED and H(SPIKES,EVOKED) is the joint entropy between SPIKES and EVOKED. Five bins were used for the entropy calculations.

Testing Performance of Optimized STIM on Slab Model

We tested the performance of the sheet model-optimized STIM on the more biologically faithful slab model. The UEA was placed in the slab model such that the tips of the stimulating electrodes were in L4 of the cortical columns, and recordings of model neuron activity were performed through the stimulating electrodes. The slab model comprised 336 neurons/electrode module and the recording electrode is not expected to sample activity from all of those neurons. Although the recording range of a microelectrode is not precisely known, it is theorized to be a spherical volume (Buzsáki 2004). Hence, we computed EVOKED by sampling activity from all neurons that were within the spherical volume around the recording electrode at three different radii: 50, 100, and 150 μm (Fig. S2), and this resulted in 39, 103, and 191 neurons, respectively. We computed normalized MSE between SPIKES and sheet model based EVOKED and compared it to normalized MSE between SPIKES and slab model based EVOKED.

Recurrent Neural Network Design

We developed a mapping function between PHYS and STIM – the sensory encoder to be integrated into a neuroprosthetic device – by training two RNNs (Fig. 4F) using the optimized STIM patterns. The Elman’s layer recurrent neural network was implemented in MATLAB R2015a using the Deep Learning Toolbox (Elman 1990). The input layers of the RNNs included two units for PHYS prop (one each for the 2D hand position coordinates) and one neuron for PHYS tact representing the indentation amplitude. The hidden and output layers in both RNNs each comprised 96 units with each unit in the output layer representing one channel of the UEA. Hyperbolic tangent (sigmoid) or linear activation functions were assigned for the units of the hidden and output layers, respectively. The output of the hidden layer was fed back as an input to the hidden layer via two-tap delay lines, i.e., input to the hidden layer units depended on the present state and two most recent past values. The synaptic weights of the RNNs were iteratively refined using the Levenberg-Marquardt backpropagation algorithm to minimize the mean square error between actual and RNN predicted STIM,

$$\text{Mean square error}=\frac{1}{N}\sum _{i=1}^N(Predicte{d}_i-Actua{l}_i{)}^2$$

where N is the sample size of STIM. The training dataset, which came from the GA based model optimization, consisted of one trial for the reach direction of 315° and two trials for each of the other seven reach directions for Monkey-C. Similarly, for Monkey-H, it was one trial for the reach direction of 90° and two trials for each of the other seven reach directions. For PHYS tact, we trained the RNN using 14 s of data from digits D3 and D4. The network was trained for 100 epochs. Pearson linear regression analysis was performed between actual and RNN predicted STIM values to evaluate the performance of the training phase.

Testing Performance of RNN

One common issue with training RNNs is overfitting to the training data, and consequently the network does not generalize well and fails to predict output to untrained inputs. Thus, the RNN was cross-validated using PHYS that were not included in training, and the output STIM was evaluated on the slab model. The cross-validation dataset comprised 6 trials across each reach direction for PHYS prop. For PHYS tact, it comprised 14 s of mechanical stimulus applied onto digits D3 and D4. To evaluate the performance of the RNN, we quantified MSE between SPIKES and EVOKED computed using the slab model. We then compared the MSE between SPIKES and EVOKED to the averaged MSE between all possible combinations across ten trials of SPIKES (trials: 1-2, 1-3, 1-4…1-9, 1-10, 2-3, 2-4, 2-5…8-9, 8-10, 9-10, i.e., the trial to trial variability in SPIKES). Further, we compared the SPIKES-EVOKED MSE to shuffled MSE computed between SPIKES and EVOKED. The shuffled MSE between SPIKES and EVOKED was constructed by reordering the spike times in EVOKED. We created a distribution of MSE values from 5000 such shufflings, and the 95^th percentile of this distribution was considered to be the shuffled MSE. Shuffling EVOKED introduced a temporal dissociation between EVOKED and SPIKES, and if the MSE between SPIKES and shuffled EVOKED was lower than the MSE between SPIKES and EVOKED predicted by RNN, then the RNN was not well trained. The MSE between SPIKES-SPIKES and shuffled SPIKES-EVOKED served as the lower and upper control limits, respectively, to assess the performance of the RNN predicted STIM. All source code files for the model-based framework are available on the NEURON ModelDB.

Comparison of the Optimized Biomimetic STIM to Existing Mappings

We compared the performance of STIM designed using our framework to STIM designed using four existing methods to generate sensory feedback: (1) linear mapping of STIM from SPIKES (Weber et al. 2011), (2) nonlinear mapping of STIM from PHYS (Berg et al. 2013), (3) linear mapping of STIM from PHYS (Flesher et al. 2016), and (4) linear mapping of STIM from $\frac{d}{dt}(PHYS)$ (George et al. 2019). We computed EVOKED response to STIM patterns obtained through these methods on the slab model and compared the performance to STIM by computing MSE between SPIKES-EVOKED.

(1). Linear Mapping of STIM from SPIKES

STIM was designed by linearly mapping from SPIKES for each channel in the UEA using the equation (Weber et al. 2011),

$${\text{STIM}}_{\text{electrode}}=\frac{SPIKE{S}_{electrode}-\text{min}(SPIKES)}{\max (SPIKES)-\text{min}(SPIKES)}(\max (STIM)-\text{min}(STIM))+\text{min}(STIM)$$

where SPIKES_electrode is the time-varying SPIKES at each channel in the UEA, min(SPIKES) and max(SPIKES) are the minimum and maximum values of SPIKES across channels, respectively, and min(STIM) and max(STIM) are the range of stimulation amplitudes set to 0 μA and 15 μA, respectively.

(2). Nonlinear Mapping of STIM from PHYS

Stimulation was designed by computing psychometric equivalence functions between perceptions generated by corresponding mechanical and electrical stimuli (Berg et al. 2013). This mapping function between electrical and mechanical stimulation was used to obtain the amplitude of electrical stimulation for any arbitrary amplitude of mechanical stimulus. We obtained a similar mapping function between electrical and mechanical stimulation using the evoked response obtained from the slab model. First, we computed S1 neural response to different depths of mechanical indentations applied onto the distal portion of digit D5. The mechanical stimulus consisted of 1 s long trapezoidal indentations delivered at a rate of 10 mm/s and depths ranging from 25-2000 μm. The firing rate was computed at the most active recording channel during the 1 s long indentation pulse (Callier et al. 2019). Second, we applied 1 s long electrical stimulation trains at different amplitudes (5-50 μA) at a frequency of 300 Hz to a model comprising a single cortical column and calculated the firing rates. Finally, we mapped the electrical stimulation amplitude onto mechanical stimulation amplitude that evoked the same firing rate. This mapping function was smoothened and then rescaled to have lower and upper limits of 0 and 10 μA. Rescaling was done to keep the mean stimulus amplitude consistent across different mapping methods. This rescaled mapping function was then used to obtain electrical stimulation amplitude (i.e., STIM) for any arbitrary PHYS tact.

(3). Linear Mapping of STIM from PHYS

STIM was computed through linear mapping from PHYS using the equation (Flesher et al. 2016),

$${\text{STIM}}_{\text{electrode}}=\frac{PHY{S}_{tact}-\text{min}(PHY{S}_{tact})}{\max (PHY{S}_{tact})-\text{min}(PHY{S}_{tact})}(\max (STIM)-\text{min}(STIM))+\text{min}(STIM)$$

where PHYS_tact is the time-varying mechanical stimulation, min(PHYS_tact) and max(PHYS_tact) are the minimum and maximum values of PHYS_tact, respectively, and min(STIM) and max(STIM) are the range of stimulus amplitudes set to 0 μA and 10.5 μA, respectively.

(4). Linear Mapping of STIM from $\frac{d}{dt}$ PHYS

STIM was obtained by linear mapping from $\frac{d}{dt}$(PHYS) using the equation (George et al. 2019),

$${\text{STIM}}_{\text{electrode}}=\frac{\frac{d}{dt}(PHY{S}_{tact})-\text{min}(\frac{d}{dt}PHY{S}_{tact})}{\max (\frac{d}{dt}PHY{S}_{tact})-\text{min}(\frac{d}{dt}PHY{S}_{tact})}(\max (STIM)-\text{min}(STIM))+\text{min}(STIM)$$

where PHYS_tact is the time-varying mechanical stimulation, $\text{min}(\frac{d}{dt}{\text{PHYS}}_{\text{tact}})$ and $\max \left(\frac{d}{dt}{\text{PHYS}}_{\text{tact}}\right)$ are the minimum and maximum values of $\frac{d}{dt}{\text{PHYS}}_{\text{tact}}$, respectively, and min(STIM) and max(STIM) are the range of stimulus amplitudes set to 0 μA and 11.5 μA, respectively.

(5). Random mapping of STIM from PHYS

We randomly assigned STIM magnitude for any given PHYS using a uniform distribution between 0 μA and 10 μA. Random mapping was used to assess the performance of a worst-case biomimetic encoding scheme.

RESULTS

Model-optimized STIM Patterns

The GA converged to spatiotemporal patterns of STIM that generated model neural responses (EVOKED) closely resembling SPIKES (Fig. 5). We compared the MSE between EVOKED and SPIKES to the MSE obtained before the optimization of STIM (i.e., from the 1^st generation of GA, which was randomly initialized with pulse amplitudes distributed uniformly between 0 to 15 μA). The MSE across the eight reach directions were pooled together for PHYS prop as there were no differences between group means across reach directions (p=0.60, one-way ANOVA, F(7,591)=0.79). The median MSE between SPIKES and EVOKED, across channels, was 373 Hz² for PHYS prop (Monkey H) (Fig. 6A1) and 400 Hz² for PHYS tact (D4) (Fig. 6B1) on the 25^th generation compared to 7785 Hz² for PHYS prop (Monkey H) (Fig. 6A1) and 6080 Hz² for PHYS tact (D4) (Fig. 6B1) in the first generation. We quantified information present in SPIKES and EVOKED by computing entropy (PHYS prop: H_SPIKES = 1.83 bits, H_EVOKED = 1.87 bits and PHYS tact: H_SPIKES = 1.81 bits, H_EVOKED = 2.10 bits). Further, mutual information between SPIKES and EVOKED was higher at the 25^th generation of GA compared to 1^st generation for both PHYS prop and PHYS tact (Fig. 6A2,B2). There was a strong correlation between SPIKES and EVOKED at the 25^th generation (PHYS prop: R² = 0.88, p < 0.001, PHYS tact: R² = 0.69, p < 0.001) compared to the 1^st generation (PHYS prop: R² < 0.001, p = 0.60, PHYS tact: R² < 0.001, p = 0.98).

Figure 5:

Spatiotemporal patterns of ICMS (STIM) designed by model-based optimization. (A) Proprioceptive inputs (PHYS prop) in Monkey-H. (A1) 2-dimensional hand position and (A3) firing rate of recorded neurons (SPIKES) as monkey reached target direction of 0° (two trials). Each trial is separated by vertical dashed line. (A2) Model-optimized patterns of ICMS delivered to each electrode (STIM) produced patterns of activity in the simplified sheet model (A4, EVOKED sheet model) that matched well with the experimentally recorded patterns of activity (A3). (B) Tactile inputs onto digit D4 (PHYS tact). (B1) The position of the cutaneous indenter and (B3) firing rate of recorded neurons (SPIKES) as a function of time. (B2) Model-optimized patterns of ICMS delivered to each electrode as a function of time (STIM) produced patterns of activity in the simplified sheet model (B4, EVOKED sheet model) that matched well with the experimentally recorded patterns of activity (B3).

Figure 6:

Performance of ICMS (STIM) designed by model-based optimization. (A1) MSE between SPIKES and EVOKED from the 1^st and 25^th generations of the genetic algorithm for proprioceptive inputs (PHYS prop). (A2) Mutual information between SPIKES and EVOKED for PHYS prop. (B1) MSE between SPIKES and EVOKED for tactile inputs onto digit D4 (PHYS tact). (B2) Mutual information between SPIKES and EVOKED for PHYS tact. For each box, the central mark indicates the median MSE/mutual information across the ten most modulated UEA channels, the bottom and top edges of the box indicate the 25th and 75th percentiles, whiskers extend to 1.5 times the interquartile range with the plus sign indicating an outlier. There was a strong correlation between SPIKES and EVOKED at the 25^th generation (PHYS prop: R² = 0.8822, p < 0.001, PHYS tact: R² = 0.6865, p < 0.001) compared to the 1^st generation (PHYS prop: R² < 0.001, p = 0.5987, PHYS tact: R² < 0.001, p = 0.9780).

Performance of Sheet Model-optimized STIM in the Slab Model

Having optimized STIM using the simplified sheet model, we tested the optimized STIM on the slab model. We also evaluated how the performance varied as a function of recording sensitivity by sampling activity from model neurons within 50, 100, 150 μm of the electrode tip (Fig. S2A). EVOKED matched SPIKES well for a recording radius of 50 μm (Fig. S2B1,B2), yielding comparable MSE between SPIKES and EVOKED as in the sheet model (Fig. S2C). The median MSE between SPIKES and EVOKED across channels was 1423 Hz² for the slab model with recording radius of 50 μm compared to 525 Hz² for the sheet model. However, for radii of 100 μm and 150 μm, the firing rates of EVOKED were higher than SPIKES (Fig. S2B1,B3,B4), and median MSE increased to 4877 Hz² and 9650 Hz², respectively (Fig. S2C). Correlation between SPIKES and EVOKED decreased with increasing recording radius (50 μm: R² = 0.73, p < 0.001, 100 μm: R² = 0.65, p < 0.001, 150 μm: R² = 0.62, p < 0.001). While STIM influenced neurons farther than 50 μm from the stimulating electrode, optimization using the sheet model did not capture this phenomenon.

Comparison of GA-optimized STIM to Linearly Mapped STIM

To evaluate the necessity of the GA-based optimization approach for calculating STIM, we also used a simple mapping technique and compared performance to the GA-optimized STIM. The mapping function consisted of firing rates computed from a single cortical column for different amplitudes of 1 s ICMS pulse trains delivered at 50 Hz. Fig. 7A shows the mapping between ICMS amplitude and firing rates computed by summing the activity (spike counts) of neurons that were closer than 50 μm from the electrode tip. We calculated the amplitudes of the time-varying ICMS patterns for the 96 channels in the UEA using the mapping function such that the EVOKED derived by linear interpolation from the mapping function approximated SPIKES in each time bin. Fig. 7C shows the STIM across 96 channels obtained using the GA optimization approach, and Fig. 7D shows the STIM amplitudes across 96 channels computed using the mapping function. The simple mapping function yielded substantially poorer performance than did the GA in the slab model (Fig. 7E and Fig. 7F). The median MSE between SPIKES and EVOKED, across channels, was 1718 Hz² for the simple mapping function compared to 1423 Hz² for GA-optimized STIM (Fig. 7G). Further, correlation was the same across the two STIM types (GA stim: R² = 0.73, p < 0.001, Mapping stim: R² = 0.73, p < 0.001). Mutual information was comparable for the GA-optimized STIM and mapping function STIM tested on the slab model (Fig. 7H). However, the MSE across channels ranged from 934 Hz² to 3469 Hz² for the mapping function, but only from 829 Hz² to 2486 Hz² for GA-optimized STIM. This difference may be in part because the GA-optimized STIM accounts for the effects of cross-channel interactions and the carryover effects of pulse-induced polarization, and these effects are neglected in the mapping function.

Figure 7:

Comparison of performance of GA-optimized STIM to STIM optimized using simple mapping function. (A) Mapping function: response of single cortical column (firing rate) to different amplitudes of 1s STIM trains delivered through a single channel at 50 Hz. (B) SPIKES in response to PHYS prop. One trial each in reach directions of 225° and 270°, recorded in Monkey-H. Trials are separated by vertical dashed line. STIM obtained using (C) GA-based optimization and (D) mapping function shown in (A). EVOKED response computed using slab model to (E) GA-optimized STIM and (F) mapping function based STIM. (G) Comparison of MSE between SPIKES and EVOKED response to GA-optimized STIM to MSE between SPIKES and EVOKED response to mapping function based STIM. GA STIM1 and GA STIM2 indicate EVOKED computed using sheet and slab models, respectively. (H) Comparison of mutual information between SPIKES and EVOKED. For each box, the central mark indicates the median MSE/mutual information across the ten most modulated UEA channels, the bottom and top edges of the box indicate the 25th and 75th percentiles, whiskers extend to 1.5 times the interquartile range and the plus signs indicate outliers.

Neural Network Predicted STIM Patterns with Trained PHYS

We trained two RNNs, one for PHYS prop and one for PHYS tact, to generate STIM outputs for any arbitrary PHYS input. We trained the RNN for 100 epochs and compared the STIM optimized using the GA to that predicted by the RNN. The RNN output closely matched the STIM for both PHYS prop (Monkey C: R² = 0.85, p < 0.001, Monkey H: R² = 0.85, p < 0.001) and PHYS tact (D3: R² = 0.90, p < 0.001, D4: R² = 0.89, p < 0.001) (Fig. S3).

Performance of STIM Predicted Using Neural Network with Untrained PHYS

We evaluated the trained RNN by computing STIM using untrained PHYS. Fig. 8A1 depicts one trial (blue) each for PHYS prop in reach directions of 0° and 90°. PHYS prop that were part of the RNN training are shown in red. The 2D hand position coordinates were slightly different even during reaches made in the same directions (Fig. 8A1). Similarly, a subset of the untrained (blue) and trained (red) PHYS tact are shown in Fig. 8B1. We tested the performance of STIM predicted by the RNNs to these untrained PHYS inputs by comparing the EVOKED using the slab model to SPIKES. The EVOKED response (Fig. 8A3) matched SPIKES (Fig. 8A2) for PHYS prop (R² = 0.25, p < 0.001). Similarly, the EVOKED response (Fig. 8B3) matched SPIKES (Fig. 8B2) for PHYS tact, as well (R² = 0.22, p < 0.001). We quantified the performance by computing MSE between EVOKED and SPIKES, compared to the mean trial-to-trial MSE of SPIKES and shuffled MSE (Fig. 9A1,B1,C1,D1). The MSE data across reach directions were pooled together for PHYS prop as there were no differences between group means across reach directions (Monkey C: p=0.92, one-way ANOVA, F(7,416)=0.37 and Monkey H: p=0.25, one-way ANOVA, F(7,586)=1.3). The median SPIKES-EVOKED MSE, across channels, were 4364 Hz² and 4218 Hz² for Monkeys C and H, respectively (Fig. 9A1,B1). These MSE were higher than the MSE between SPIKES across repeated trials, which were 2899 Hz² for Monkey C and 3207 Hz² for Monkey H but lower than the shuffled MSE between SPIKES and EVOKED: 5928 Hz² for Monkey C and 5465 Hz² for Monkey H (Fig. 9A1,B1). Similarly, for tactile inputs, the SPIKES-EVOKED MSE was higher than the MSE between SPIKES across repeated trials but lower than the shuffled MSE (Fig. 9C1,D1). Further, mutual information between SPIKES and EVOKED was lower compared to mutual information across SPIKES trials for both PHYS prop and PHYS tact (Fig. 9A2,B2,C2,D2). The sensory encoding algorithms developed using the framework described above are demonstrated in videos Vid1 (PHYS prop) and Vid2 (PHYS tact).

Figure 8:

Spatiotemporal patterns of ICMS predicted by the RNN for untrained PHYS inputs. PHYS prop (hand endpoint position coordinates in target directions of 0° and 90°) that were not part of training from (A1) Monkey-H were used as inputs to the RNN. Red indicates PHYS prop that were part of training and untrained PHYS prop are shown in blue. (A2) SPIKES to the untrained PHYS prop recorded in Monkey-H. Trials are demarcated by vertical dashed line. (A3) EVOKED in the slab model by RNN predicted STIM for the untrained PHYS prop shown in A1. (B1) Untrained PHYS tact onto digit D4. Red indicates PHYS tact that were part of training and untrained PHYS tact is shown in blue. (B2) SPIKES to untrained PHYS tact onto digit D4. (B3) EVOKED in the slab model by RNN predicted STIM for untrained PHYS tact shown in B1.

Figure 9:

Performance of recurrent neural network predicted STIM with untrained PHYS. MSE between SPIKES and EVOKED, MSE between SPIKES and SPIKES across all possible combinations of ten repeated experimental trials and shuffled MSE for (A1) PHYS prop in Monkey-H, (B1) PHYS prop in Monkey-C, (C1) PHYS tact onto digit D3 and (D1) PHYS tact onto digit D4. Mutual information between SPIKES and EVOKED, mutual information between SPIKES trials for (A2) PHYS prop in Monkey-H, (B2) PHYS prop in Monkey-C, (C2) PHYS tact onto digit D3 and (D2) PHYS tact onto digit D4. MSE and mutual information are shown across 7 most modulated channels for Monkey-C and 14 channels for Monkey-H. For PHYS tact , MSE and mutual information are shown across ten channels. For each box, the central mark indicates the median MSE/mutual information across UEA channels, the bottom and top edges of the box indicate the 25th and 75th percentiles, and the whiskers extend to the absolute minimum and maximum.

Comparison of performance of model-optimized STIM to STIM designed using existing methods

We evaluated the performance of GA-optimized STIM to STIM designed using four existing approaches at the electrode that showed maximum sensitivity to PHYS for PHYS tact onto D4. Fig. 10 shows (A) GA-optimized STIM and STIM designed by (B) linear mapping of STIM from SPIKES (Weber et al. 2011), (C) nonlinear mapping of STIM from PHYS tact (Berg et al. 2013), (D) linear mapping of STIM from PHYS tact (Flesher et al. 2016), and (E) linear mapping of STIM from d/dt (PHYS tact) (George et al. 2019). All STIM had the same mean amplitude of approximately 5 μA. We computed MSE between SPIKES and EVOKED in the slab model (Fig. S4) and compared it to MSE of randomly generated STIM. The model-optimized STIM outperformed STIM designed using all other techniques (Fig. 10G), and all STIM performed better than random STIM (Fig. 10G).

Figure 10:

Comparison of performance of model-optimized STIM to STIM patterns designed using existing techniques at the most active electrode for PHYS tact. (A) GA-optimized STIM, (B) linear mapping of STIM from SPIKES (Weber et al. 2011), (C) nonlinear mapping of STIM from PHYS tact (Berg et al. 2013), (D) linear mapping of STIM from PHYS tact (Flesher et al. 2016), (E) linear mapping of STIM from d/dt (PHYS tact) (George et al. 2019), (F) STIM randomly assigned using uniform distribution ranging from 0-10 μA. STIM patterns designed using all techniques had a mean amplitude of approximately 5 μA. (G) MSE between SPIKES and EVOKED across STIM patterns. EVOKED response was computed using the slab model. STIM obtained through GA based model optimization outperformed STIM designed using all other methods. (H) MSE between SPIKES and EVOKED for multi-channel STIM designed using model-based optimization and linear mapping of SPIKES. MSE are shown across ten most modulated channels in the UEA. Mutual information for model-optimized and linear mapping STIM are shown in the inset box. For each box, the central mark indicates the median MSE across UEA channels, the bottom and top edges of the box indicate the 25th and 75th percentiles, and the whiskers extend to 1.5 times the interquartile range.

We extended the STIM designed for the most active channel to all 96 channels in the UEA by linear mapping from SPIKES and compared performance to multichannel STIM designed using model-based optimization. It should be noted that it is infeasible to design unique STIM at each channel of the multi-channel array using other techniques. The median amplitude of mean STIM across channels was same for linearly mapped STIM and for the GA based STIM (~1.5 μA). The model-optimized multichannel STIM performed better than multi-channel STIM designed through linear mapping from SPIKES (Fig. 10H), especially on the channel that was most modulated to PHYS. However, linear mapped STIM showed comparable performance or in some cases slightly better performance on channels that did not exhibit substantial modulation to PHYS (Fig. 10H).

DISCUSSION

We developed and validated a computational framework for the systematic design of spatiotemporal patterns of ICMS optimized to evoke biomimetic neural activity in S1 and thereby evoke naturalistic percepts. First, we recorded multi-unit neural responses (SPIKES) to sensory inputs (PHYS) in non-human primates using a 96 channel multi-electrode array. Second, we implemented a biophysically-based computational model of cortical columns and optimized spatiotemporal patterns of ICMS (STIM) such that model-based responses (EVOKED) matched SPIKES. Finally, we developed a mapping function between PHYS and STIM by training a recurrent neural network using STIM obtained from the model-based optimization. The model-optimized STIM outperformed STIM generated from four other existing approaches: (1) linear mapping of STIM from SPIKES, (2) nonlinear mapping of STIM from PHYS, (3) linear mapping of STIM from PHYS, and (4) linear mapping of STIM from $\frac{d}{dt}(PHYS)$. Although our model-based framework is a substantial advance for designing biomimetic STIM patterns for prosthetic sensory feedback, it comes with several important limitations.

Experimental Limitations

Current multi-electrode arrays (like the UEA) are capable of sampling activity with high fidelity within a radius of at least 30-50 μm from the recording tip (Buzsáki 2004), More distant neurons can be recorded less reliably and glial scaring around the electrode and further limits the maximum number of neurons that can be captured per channel (Pedreira et al. 2012). Despite the fact that decoding neural activity from ~100 neurons in M1 is sufficient to predict limb kinematics (Serruya et al. 2002; Wessberg et al. 2000), it remains to be determined whether replicating the activity patterns in a relatively small number of S1 neurons is sufficient to induce naturalistic sensory perception. As the accuracy of decoding improves with numbers of recorded neurons (Carmena et al. 2003), scalable electrode technology capable of recording large-scale neural activity and stimulating through a very large number of channels with small amplitude currents might benefit future biomimetic approaches (Viventi et al. 2011).

Another challenge with the biomimetic approach is the inability to measure neural responses to naturalistic sensory inputs in persons with spinal cord injury, peripheral neuropathy, or limb amputation. Decoding limb kinematics from M1 of persons with paralysis presents a similar problem, but one that can be circumvented in experiments with intact non-human primates (Carmena et al. 2003; Ethier et al. 2012; Gilja et al. 2012; Santhanam et al. 2006; Taylor et al. 2002) or through the use of imagined movements (Gilja et al. 2012). In experiments with humans, subjects are asked to imagine controlling the movement of a prosthetic arm through a specified trajectory within the 3D workspace, and decoders are built based on M1 recordings obtained during the task together with the kinematics of the imagined trajectory (Ajiboye et al. 2017; Collinger et al. 2013; Hochberg et al. 2012; Hochberg et al. 2006). Such decoders yield efficient performance without the need for training data collected during actual movement. Although sensory representations of the body remain stable after deafferentation, subtle differences in somatotopy and electrode locations will likely make generalization between subjects challenging (Makin and Bensmaia 2017). We found that the RNN generalized poorly to untrained reach directions, and the lack of a clear somatotopic arrangement of proprioceptive information in S1 might have contributed to the model not learning a mapping for data to which it was not exposed (Lucas et al. 2019).

An additional experimental challenge is that artifacts from ICMS may obscure the signals in M1 needed for BMI control (Gilja et al. 2011). Although several groups have developed different techniques such as amplifier blanking (O’Doherty et al. 2011; Weiss et al. 2018) and offline estimation of artifacts (O’Shea and Shenoy 2018), the challenge is exacerbated when using continuous stimulation such as emerges from our model-based mapping function. The issue of stimulation artifact can be obviated when stimulating targets upstream from S1 (thalamus) to restore sensory feedback (Heming et al. 2010; Swan et al. 2018). Further, optical, rather than electrical stimulation, may be one approach to overcome the hurdle of stimulation artifacts in a bidirectional BMI design (Diester et al. 2011; Gilja et al. 2011), although neither its safety nor its efficacy for ICMS has been established in primates even for small numbers of optrodes, let alone the number necessary to convey detailed limb-state information.

Model Limitations

The sheet and slab models lacked several features. First, both models lacked horizontal connections between cortical columns, and such connections might mediate distant activation during ICMS (Butovas and Schwarz 2003; Hao et al. 2016). However, ICMS in area 1 induces focal sensory percepts (Flesher et al. 2016; Salas et al. 2018; Tabot et al. 2013), implying that local activation by ICMS dominates the elicitation of sensory percepts. Further, the axon morphology of neurons in both the sheet and slab models was simplified – with only single straight axon with two branches – and this can influence patterns of activation, as it is primarily axons (terminals, branches, bends) that are activated by ICMS (Aberra et al. 2018; Histed et al. 2009; Hussin et al. 2015; McIntyre and Grill 2000; Nowak and Bullier 1998).

Second, the mode of activation by ICMS of local cells remains unclear. Although a consensus has emerged that ICMS activates axon terminals at stimulation threshold (Aberra et al. 2018; Jankowska et al. 1975; McIntyre and Grill 1999), it is not known to what degree activation of axon terminals results in trans-synaptic activation of local cells via pre-synaptic inputs (Hussin et al. 2015) and or direct activation of the local cells at their nearby axon terminals (Histed et al. 2009). The sheet model does not account for the first mode of activation by ICMS, while the slab model accounts for both.

Third, the neuron density in the slab model was not comparable to histological measurements in macaque brain. The density of L4 neurons in macaques is ~160,000/mm³ (Hilgetag et al. 2016). Thus, spherical volumes of radii 50, 100, 150 μm would contain ~83, 670, 2240 neurons, respectively, while the slab model included only 39, 103, 191 neurons within the respective spherical volumes. A model representing realistic neural density would further reduce the amplitudes required for optimized STIM due to the presence of more neurons within the spherical volume activated at smaller currents but would place substantial demands on the required computation.

Finally, the slab model did not include GABA-B synaptic receptors. The original thalamocortical network model developed by Traub et al. lacked GABA-B synapses and included only AMPA, NMDA and GABA-A receptors (Traub et al., 2005). GABA-B synaptic connections appear to play a prominent role in mediating the responses evoked by ICMS. Single-pulse ICMS generate short latency excitation followed by a period of long-lasting inhibition (Butovas and Schwarz 2003; Hao et al. 2016). This inhibition is believed to be generated by activation of GABA-B synaptic connections (Butovas et al. 2006). However, paired pulse experiments revealed that the excitatory response to the second pulse was largely unaffected by the inhibition evoked by the first pulse (Butovas and Schwarz 2003).

Running the model-based optimization for ~25 s of PHYS prop data in one monkey took ~37 days to complete on a 360 CPU high-performance compute cluster. Simulating an equivalent model-based optimization on a standard desktop computer with Intel quadcore processor (2.40 GHz) would take ~9 years. Running the optimization on more complex models with more neurons, realistic morphologies, and horizontal connections between columns would increase the computational cost several-fold. Further, we chose to modulate the amplitude of ICMS rather than parameters such as frequency, pulse width, train duration etc. However, recent experimental evidence has shown ICMS frequency can play a prominent role in shaping artificial tactile sensation independently of amplitude (Callier et al. 2020; Salas et al. 2018). Modulating other stimulus parameters simultaneously would make the search space complex and the solutions might take more time to converge. Massively parallel systems incorporating diverse computing entities would need to be exploited in the future to handle such complexity (Jovanov et al. 2018; Peters et al. 2010). An alternative solution to overcome the challenge of computational demand is to optimize STIM across a reduced number of channels (e.g., a 3x3 portion of the electrode array) spanning a volume of cortex that shares common receptive fields.

Status and Future Directions of Model-based Biomimetic Framework

Compared to the effort to develop optimal efferent interfaces (Dyer et al. 2017; Sussillo et al. 2015; Sussillo et al. 2012; Sussillo et al. 2016), there has been relatively little systematic work on a model-based framework for the design of biomimetic afferent neural prostheses. Choi and colleagues developed such a framework to design patterns of thalamic microstimulation for somatosensory feedback (Choi et al. 2016). First, they recorded responses (local field potentials) from S1 in rats during the application of tactile input to the forepaw. Second, they recorded evoked response from S1 (output) during application of spatiotemporal patterns of thalamic microstimulation at different amplitudes (input). Finally, they trained a state-space model using the input-output dataset and used the model to predict optimal spatiotemporal stimulus patterns such that the response of the state-space model equaled the experimentally measured response to the tactile inputs onto the forepaw. The correlation coefficient between the responses in S1 evoked by optimized thalamic microstimulation and the response to naturalistic sensory input was ~80%.

One of the major hurdles of our biomimetic framework is the lack of an experimental ground truth against which the model-based ICMS patterns can be validated. As described above, ICMS can activate local cells through either direct activation of local axons and/or trans-synaptic activation through presynaptic axon terminals. Electrical recordings made directly from the stimulated electrode exhibit artifacts that might extend to at least 2 ms beyond the stimulus pulse (Butovas and Schwarz 2003; Hao et al. 2016). Thus, it will be challenging to record spikes evoked by direct activation of axons of local cells that backpropagate to the soma (Aberra et al. 2018; Overstreet et al. 2013). Hence, specialized hardware mitigating stimulation artifacts should be developed and might hold the key to validate such model-based biomimetic frameworks (Mueller et al. 2014).