Neural correlates of phosphene perception in blind individuals: A step toward a bidirectional cortical visual prosthesis

Abstract

Blindness is one of the most impactful disabilities in human lives. Cortical prostheses could one day restore functional vision in some blind subjects, but their success will depend on integrating advanced technologies to realize the therapeutic benefits they promise. Most previous studies in humans used electrodes only for stimulation, which has made it challenging to precisely control the appearance of individual phosphenes. Herein, we implanted an intracortical microelectrode array of 100 electrodes in the visual cortex of two blind volunteers. We recorded the neural activity around the electrodes while performing electrical stimulation to induce visual perceptions. Besides showing how stimulation parameters influence perceptual thresholds, perceived brightness, and the minimum interval required to distinguish separate stimuli, our results indicate that subjective visual experience can be accurately predicted from the recorded neural activity. These results highlight the potential for using the neural activity of neighboring electrodes to accurately infer and control visual perceptions in cortical visual prostheses.

Visual prostheses are improved by decoding neural signals to predict and control visual perceptions in blind individuals.

INTRODUCTION

The restoration of a functional visual sense in profoundly blind individuals has long been a driving force in the fields of neural engineering and neuroprosthetics. Electrical stimulation of the visual cortex has been proposed as an approach to transfer information from the outside world to blind individuals (1–3). As our understanding of the intricate neural circuits underlying visual perception advances, we are closer to the development of innovative solutions that bridge the gap between visual stimuli and neural responses. However, up to now it is not possible to control the appearance of individual phosphenes; therefore, many efforts are still needed to achieve the therapeutic benefits envisioned by this technology (4).

Previous works in different animal models (5–14) and humans (15–25) have shown that microelectrodes implanted in the visual cortex allow the recognition of simple shapes, motion, and letters. However, most of these studies have been proof-of-concept experiments and involved only a small number of electrodes. To enable visually impaired individuals to perform visual tasks such as recognize objects or navigate in complex environments, it is clear that prostheses must incorporate a substantial number of electrodes, generally exceeding 625 (26, 27). This is not too difficult from a technical point of view, but the manual fine-tuning of the stimulation parameters for each electrode to achieve the desired perception in a clinical context can be a very time-consuming process. Moreover, the features of the perceived phosphenes and the required current levels for perception may vary between different days (28), and the induced brightness could be affected by the sequence of stimulation used (29). These issues pose many challenges and highlight the necessity for a user feedback-independent method to automatically adjust stimulation parameters in future clinical visual prostheses (30).

Here, we investigate the role that bidirectional communication with the brain may play in the development of next-generation cortical visual prostheses. We demonstrate that neural activity induced in V1 by electrical stimulation correlates with the probability of phosphene perception, the perceived brightness, and the minimum interstimulus interval required for separate perceptions. Previous studies have demonstrated that activity in the area V4 of the visual cortex in monkeys can predict phosphene perceptions elicited in V1 (14). However, this approach implies inserting microelectrodes into multiple brain areas, leading to a more invasive procedure with substantial clinical drawbacks. A more direct strategy is to perform simultaneous stimulation and recordings on the same set of electrodes. This approach can enhance the effectiveness, safety, and long-term stability of visual cortex stimulation. Furthermore, it holds the potential not only to allow for the fine-tuning of stimulation parameters but also to actively engage in bidirectional communication with the occipital cortex, thereby providing more precise and personalized visual percepts for the user.

RESULTS

Human studies in blind volunteers

Traditionally, cortical neuroprostheses have relied on open-loop mechanisms, where information from the outside world is relayed directly to the brain without real-time adaptation based on neural feedback. However, this approach falls short in capturing the dynamic nature of neural circuits, leading to limitations in perceptual richness and adaptability. Moreover, this approach does not allow a precise control of the population of neurons surrounding the electrodes. In contrast, closed-loop systems introduce a paradigm shift by establishing a continuous dialogue between the artificial sensory input and the recipient’s brain activity (31). This interaction allows for real-time adjustments in the delivered visual cues, thereby aligning the prosthesis output with the brain’s ongoing neural dynamics.

To explore this approach and assess its potential for enhancing visual perception, we implanted an array of 100 penetrating intracortical microelectrodes [Utah Electrode Array (UEA)] into the visual cortex of two blind volunteers. This device has been previously implanted for extended periods of time in the motor and sensory cortex of human subjects, allowing for both neural recordings and intracortical microstimulation (15, 32, 33). The surgical approach has been described elsewhere (34) and is briefly outlined in the Materials and Methods section. The locations of the UEA in both subjects, along with the predicted retinotopic map organization, are presented in Fig. 1. The study protocol was approved by the Hospital General Universitario de Elche Clinical Research Committee (see Materials and Methods) and registered at ClinicalTrials.gov (NCT02983370).

Fig. 1. Location of the UEA implantation site on the right occipital cortex of the two participants.

Predicted retinotopic map organization overlaid on the 3D brain reconstruction.

Both blind volunteers were implanted for a period of 6 months, during which we performed daily stimulation and recording experiments. As with our first blind volunteer (15), we consistently obtained high quality recordings and most of the electrodes yielded reliable spikes over the whole study period. Spike amplitudes of about 1000 μV were frequently observed (noise around 100 μV), and occasionally we recorded amplitudes >1500 μV (120-μV noise). Furthermore, both participants were able to reliably perceive phosphenes induced by electrical stimulation throughout the entire implantation period.

Using neural activity to measure thresholds

Phosphene thresholds were first determined by presenting electrical stimuli of varying current intensities and recording the subject’s perceptual responses (i.e., whether a phosphene is perceived) using standard psychometric curves (see Materials and Methods for more details). Briefly, stimuli characterized by different currents (or pulse widths) were randomly presented until a total of 10 responses (indicating either perception or no perception) were obtained for each unique current (or pulse width) level. In addition, 10 “sham” stimuli (no stimulation) were also randomly interleaved as controls (Fig. 2A). Both subjects performed well in the psychometric experiments, reporting perception in only 7.3% (subject #1) and 2.1% (subject #2) of the total number of sham trials.

Fig. 2. Prediction of thresholds from neural activity.

(A) Schematic of the protocol. I_i represents current amplitude values. (B) Distribution of charge at perceptual threshold, derived from participants’ responses in the psychometric threshold experiments for each subject. (C to E) Representative experiment on subject #2, stimulation from electrode 23. (C) Location of stimulation electrodes (yellow) and recording electrodes (blue) included in the threshold estimation. (D) Psychometric curve: Proportion of perceived stimuli (blue dots) as a function of the charge of a pulse and sigmoidal fit (purple line) used to extract the behavioral threshold for phosphene perception. (E) Population neural response from included electrodes used to estimate the neural threshold. For each charge, light blue “×” marks indicate trial-level z-scored responses (0 to 167 ms); blue circles and error bars show mean and SD across the 10 repetitions for all the included electrodes. The purple sigmoid fit is used to extract the neural threshold for phosphene perception. (F) Correlation between the charge at threshold for phosphene perception extracted from the neural activity (neural threshold) and from the participant’s answers (behavioral threshold) in all the experiments performed by subject #2. (G to I) Same as (C) to (E) for another experiment performed on subject #1 stimulating from electrodes 30, 31, 40, and 41 (yellow electrodes in G). Data shown in (I) are collected from the blue electrodes. (J) Correlation between the charge at threshold for phosphene perception extracted from the neural activity (neural threshold) and from the participant’s answers (behavioral threshold) in all the experiments performed by subject #1.

The probability of phosphene perception was analyzed as a function of the stimulation train’s charge per phase, with the frequency, number of pulses, and train duration maintained at constant values. A total of 49 experiments were performed with subject #1 and 28 experiments with subject #2 (a comprehensive list of all experiments is provided in table S1). These experiments spanned different days and months over the entire 6-month period and included both single electrodes and combinations of electrodes. For this task, most experiments used stimulation trains at 300 Hz, lasting 167 ms with 50 biphasic, cathodic-first pulses. In experiments with a constant pulse width, the current was randomly varied from 1 μA up to a maximum value, which depended on the number of stimulation electrodes used. Increments were either 5 or 10 μA (see Materials and Methods for more details). When the current was held constant, pulse width was randomly adjusted. For subject #1, the pulse width ranged from 50 to 230 μs in 20-μs increments, whereas for subject #2, it ranged from 10 to 110 μs, also with 20-μs step increments. These variations in pulse widths were implemented to account for intersubject variability.

Behavioral thresholds for this task were defined as the charge at which the participants reported visual perceptions 50% of the time. Averaged thresholds were 3.45 ± 1.27 nC per phase (means ± SD) for subject #1 and 2.42 ± 0.99 nC per phase (means ± SD) for subject #2 (Fig. 2B). Figure S1 presents the current thresholds, assuming a 170 μs pulse width. For subject #1, the average threshold was 20.3 ± 7.5 μA (means ± SD), and for subject #2, it was 14.2 ± 5.8 μA (means ± SD).

To monitor the effects of electrical stimulation on neuronal activity and investigate whether the activity of neurons surrounding the electrodes could offer insights into perceived visual sensations, we conducted simultaneous electrophysiological recordings during electrical stimulation. The procedures for removing electrical artifacts and extracting multiunit activity (MUA) during microstimulation are detailed in the Materials and Methods section (see also fig. S2). We observed an increase in MUA neural activity, particularly in electrodes near the stimulation site, which saturated at the highest current levels (see fig. S3 and the spatial activity maps at different currents in fig. S4). Figure S3B illustrates this behavior with a representative recording from electrode 34 (dark blue in the fig. S3A map) during an experiment conducted in subject #2, wherein electrode 23 served as the stimulation source (yellow in the fig. S3A map). Calculating the average MUA throughout the stimulation train (from t = 0 ms to t = 167 ms) for each stimulation allowed us to derive the curve presented in fig. S3C. This MUA-derived curve exhibits close agreement with the psychometric curve generated from the responses of the blind volunteer, as shown in Fig. 2D.

Given that electrodes near the stimulation site exhibit similar and stronger responses than those farther away (fig. S4), we quantified neural thresholds for perception using the neural responses from all electrodes surrounding the stimulation sites that yielded reliable MUA recordings. Neural current thresholds were defined as the currents at which the averaged activity across all included electrodes (z-score normalized) reached 50% of its range. Figure 2E displays the neural psychometric curve obtained for the same experiment mentioned above, aggregating the neural responses from the electrodes surrounding the stimulation electrode (blue electrodes in Fig. 2C). Thus, we were able to successfully derive the thresholds by analyzing the neural activity of electrodes surrounding the stimulated sites.

Similar results were obtained when stimulating groups of electrodes rather than individual ones. Figure 2 (G to I) presents a representative example from an experiment conducted on subject #1, where a group of four electrodes (30, 31, 40, and 41) was stimulated. Consistently, the activity of electrodes adjacent to the stimulated group was modulated by the applied current. For instance, fig. S3 (E and F) shows the MUA recorded from electrode 39 (highlighted in dark blue in fig. S3D), which is adjacent to the stimulated group. This modulation enabled us to use the activity of surrounding electrodes to estimate the neural current threshold for perception (Fig. 2I).

We then calculated neural thresholds for perception for every experiment where participants completed the psychometric task, analyzing the neural activity of electrodes surrounding the stimulated sites. A highly significant correlation was observed between behavioral and neural thresholds for both subjects. Specifically, for subject #1, we found R = 0.71 (N = 49, P < 0.001) (Fig. 2J), and for subject #2, R = 0.71 (N = 28, P < 0.001) (Fig. 2F). The Supplementary Materials provide further examples of the method’s application to experiments involving variations in current (fig. S5) and pulse width (fig. S6).

Assessing perception brightness from neural activity

Next, we investigated whether neural recordings could also estimate the brightness of perceptions. To achieve this, we first examined how stimulation parameters affected phosphene brightness. In these experiments, we randomly varied the stimulation parameters (frequency, number of pulses, and train duration) while maintaining a constant charge for each pulse. These experiments were performed only with subject #2 because the advanced implementation of our stimulator software was unavailable during the first subject’s implantation period. Each stimulation train with its unique set of parameters was repeated five times, with the order randomized. Five interleaved randomized sham trials (no stimulation) were also included. The participant reported perception in only 0.9% of the total sham trial stimulations during this task.

A total of 22 experiments were conducted using different electrodes, and the subject was asked to rate the brightness of the perceived phosphene on a scale from 0 to 5 (Fig. 3A). The stimulation frequency (f) ranged from 50 to 400 Hz, and the train duration (TD) from 100 to 500 ms (the number of pulses varied accordingly: pulses = f × TD). In each experiment, the pulse width and current were fixed at values that consistently produced perception with a 300 Hz train lasting 167 ms. Specifically, the current was 60 μA, and the pulse width was either 100 or 170 μs. Table S2 provides a complete list of all stimulation electrodes and parameters used for each experiment.

Fig. 3. Perceived brightness and neural activity.

(A) Schematic of the protocol. TD_i, f_i, and pulses_i are the train duration, frequency, and number of pulses. (B) Average z-scored brightness reported for each stimulation parameter combination across all brightness experiments. White spaces indicate untested parameters. a.u., arbitrary units. (C) Location of stimulation and recording electrodes for the data shown in (D) to (H). (D) Brightness and neural responses (recording electrode 26, stimulation electrode 16) at different frequencies (top: 100 Hz, middle: 200 Hz, and bottom: 300 Hz). Left: Average brightness reported as a function of train duration for different stimulation frequencies. Middle: Average neural responses as a function of the train duration (the stimulation train ends at time = 0 ms). Each trace averages the five repetitions with the same train duration. Right: Light red “×” marks represent neural responses averaged across 20 to 300 ms after train end for each stimulation trial. Red circles with error bars indicate mean and SD across five repetitions for every train duration. (E) Brightness and neural responses (recording electrode 26, stimulation electrode 16) at different train durations (top: 100 ms, middle: 300 ms, and bottom: 500 ms). Left: Average brightness reported as a function of the stimulation frequency for different train durations. Middle: Average neural responses as a function of the stimulation frequency. Each trace averages the five repetitions with the same stimulation frequency. Right: Light blue “×” marks represent neural responses averaged across 20 to 300 ms after train end for each trial. Blue circles with error bars represent mean and SD across five repetitions for every stimulation frequency. (F) Average brightness reported by the subject for each frequency/train duration combination after stimulation from electrode 16. (G) Average MUA in electrode 26 (averaged 20 to 300 ms after train end and across the five repetitions) for each frequency/train duration combination. (H) Correlation between neural activity recorded at electrode 26 and perceived brightness.

When averaging the reported brightness (after z-score normalization within each session) across different experiments, we observed that brightness increased with both the duration of the stimulation train and the stimulation frequency (Fig. 3B). To examine the separate effects of frequency, train duration, and number of pulses on brightness, as well as their potential interaction, we performed a regression analysis between each individual variable (frequency, train duration, and number of pulses) and their combinations to predict the perceived brightness. Each regression model was evaluated on the basis of R², Akaike Information Criterion (AIC), and Bayesian Information Criterion (BIC). The results, summarized in table S3, indicate that the model incorporating all three parameters provided the optimal fit. The combination of frequency and train duration ranked as the second-best predictor. Although the number of pulses alone also yielded good results, its predictive capability was inferior to that of frequency and train duration combined. This suggests that the brightness of perception is more accurately predicted by how pulses are delivered (i.e., their frequency and train duration), not simply by their total count. Although the number of pulses is the product of frequency and pulse train duration, the model’s performance is substantially improved when all three variables are included. This finding implies a nonlinear interaction between these parameters and the perception of brightness, which is more accurately captured by a linear regression model that considers all three variables.

Our initial step to assess perceived brightness from neural responses involved measuring neural activity across all 96 electrodes following the termination of each stimulation train. The decision to exclude neural activity recorded during stimulation stemmed from the observation that varying stimulation frequencies produced distinct fast settle blanking durations in our stimulator, thereby affecting the recorded neural activity. Thus, to enable comparisons across different stimulation parameters, we focused on the activity recorded within the 300 ms period immediately after the end of each stimulation train (see Materials and Methods).

Representative examples of these experiments appear in Figs. 3 and 4. Specifically, Fig. 3 (C to H) illustrates how the neural activity recorded in electrode 26, following stimulation from electrode 16, correlates with the brightness perceived by the subject. Figure 3D illustrates the effects of train duration for each stimulation frequency. The first column displays the participant’s average reported brightness as a function of train duration (for constant stimulation frequencies: top, 100 Hz; middle, 200 Hz; bottom, 300 Hz). The time course of the neural response after each stimulation train is depicted in the second column (each trace represents the average across five repetitions at the same train duration). By considering the average value within the 300 ms period following each stimulation train, we obtained the trends shown in the third column, which closely resemble the average brightness trend reported by the participant. Figure 3E shows comparable neural response behavior when examining different stimulation frequencies at each train duration.

Fig. 4. Spatial distribution of the correlation between neural activity and brightness.

(A) Spatial distribution of electrodes in the UEA showing correlations between MUA and brightness of the perception. Representative stimulation sites on the UEA are marked by a cross: electrode 24 (left), electrode 16 (middle), and electrode 34 (right). Colors represent the correlation (R) values. (B) MUA averaged 20 to 300 ms poststimulation train for each electrode presented in (A). Data correspond to the highest applied stimulation parameters for each experiment. (C) Average correlation between neural activity and perceived brightness as a function of distance from the stimulation electrode (all experiments included). (D) Relationship between correlation with perceived brightness and z-score of MUA activity under maximal stimulation conditions (all experiments included).

Figure 3 (F and G) illustrates the combined impact of stimulation frequency and train duration on both the brightness evoked by electrode 16 stimulation and the neural activity at electrode 26. There is a high positive correlation which is shown in Fig. 3H (R = 0.871, P < 0.001). Figure S7 offers an additional illustration of a neural response and its correlation with perceived brightness, stemming from an experiment stimulating electrode 33. In this case, the activity recorded in electrode 34 also shows a strong correlation with the brightness reported by the subject (R = 0.897, P < 0.001).

Next, we investigated the correlation between perceived brightness and the neural activity across all electrodes in the UEA for every experiment. Figure 4 presents representative examples from three different electrodes. The Pearson correlation coefficients (Fig. 4A) show that the highest correlation coefficients are consistently found in electrodes near the stimulation sites. A similar trend was observed for the averaged MUA neural activity (Fig. 4B). This behavior was consistent across all 22 experiments, as detailed in fig. S8.

When we plotted the averaged correlation between neural activity and brightness perception as a function of the distance from the stimulated electrode (Fig. 4C), we observed a decrease in correlation for distances greater than 1 mm. Notably, no correlation was found for the stimulated electrodes themselves (distance = 0 mm). Figure 4D displays the averaged magnitude of the neural response for each channel and its correlation with brightness, pooling all experiments after z-score normalization of the neural activity. Although electrodes with the strongest responses generally showed a high correlation with brightness (points in the top-right corner of Fig. 4D), some electrodes with weaker responses also exhibited a good correlation (see also Fig. 4, A and B).

We also tested whether the correlation with brightness depended on the stimulated electrode. Our findings indicate that the correlation observed at neighboring electrodes was generally high, with the exceptions of electrodes 7 and 96 (see fig. S9). These two electrodes are positioned at the periphery of the UEA, potentially eliciting neural activity beyond the UEA, which we cannot measure.

Estimating the number of phosphenes from neural activity

To assess whether we could differentiate the number of perceived phosphenes from neural recordings, we first determined the minimum intertrain interval that led to the perception of two distinct phosphenes. For these experiments, we used a psychometric task, asking participants to report the number of perceived phosphenes (0, 1, or 2) as we randomly varied the time between two consecutive stimulation trains. We stimulated single electrodes and groups of two or four electrodes. Each stimulation consisted of trains of 50 biphasic, cathodic-first pulses (170 μs pulse width, 300 Hz, and 167 ms total duration). The current was always above the perception threshold and ranged from 30 to 70 μA depending on the electrodes and number of electrodes tested (see table S4). To account for potential biases, we also included 20 interleaved sham trials: 10 with no stimulation and 10 with a single stimulation train (Fig. 5A). In total, subject #1 completed 11 psychometric experiments, and subject #2 completed 12. Both subjects performed well on this task, with an overall rate of incorrect reports in sham trials of only 5.2% for subject #1 and 5.5% for subject #2.

Fig. 5. Prediction of the number of perceived phosphenes from neural activity.

(A) Schematic of the protocol. Delay_i are intertrain delays. (B) Distribution of intertrain delays at threshold for separate phosphene perceptions per subject. (C to E) Example experiment: Subject #2, stimulation from electrode 54. (C) Location of stimulation electrodes (yellow) and recording electrodes (blue) included in the threshold estimation. (D) Reported 1-phosphene (blue dots) and 2-phosphene (orange dots) perception rates per intertrain delay, with sigmoidal fit (purple line) used to derive the behavioral threshold for separate phosphene perception. (E) Population response from the included electrodes used to determine the neural threshold for separate phosphene perception for this experiment. For each intertrain delay, light blue “×” marks represent z-scored neural responses averaged during the second stimulation train for each trial and included electrode. Blue circles with error bars show the mean and SD. The purple line represents the sigmoidal fit used to extract the neural threshold for separate phosphene perceptions. (F) Correlation between the intertrain delay at threshold for separate phosphene perception extracted from the neural activity (neural threshold) and from the participant’s answers (behavioral threshold) in all the experiments performed by subject #2. (G to I) Example experiment: Subject #1, stimulation from electrodes 49, 50, 51, and 52. These panels represent the same as (C) to (E). (J) Correlation between the intertrain delay at threshold for separate phosphene perception extracted from the neural activity (neural threshold) and from the participant’s answers (behavioral threshold) in all the experiments performed by subject #1.

The threshold intertrain interval, defined as the temporal separation at which participants reported perceiving two distinct phosphenes 50% of the time, was 0.30 ± 0.07 s (means ± SD) for subject #1 and 0.33 ± 0.04 s (means ± SD) for subject #2 (Fig. 5B). Our observations indicate that the intertrain distance necessary for perceiving clearly separated phosphenes significantly increases with the number of stimulating electrodes (P = 0.04, Welch t test comparing the threshold for separate perceptions when stimulating with one or two electrodes against four electrodes; fig. S10, top left for subject #2). Conversely, the Pearson correlation coefficient does not show a significant correlation between applied current and the perception of two distinct phosphenes (R = 0.022, P = 0.949 for subject #1; R = −0.412, P = 0.183 for subject #2; fig. S10, right).

All experiments were conducted while simultaneously recording neural activity from all 96 microelectrodes of the UEA. Then, our analysis focused particularly on the MUA during the first and second stimulation trains of each trial. We found that only the MUA during the second stimulation train was significantly modulated by the intertrain delay, with the effect being especially pronounced in electrodes located near the stimulation sites. Moreover, this activity increased proportionally with longer intertrain delays, eventually reaching levels comparable to those observed during the first train. This pattern is illustrated in fig. S11 (B and C), which show the MUA recorded from electrode 44 (dark blue in fig. S11A) in response to stimulation of electrode 54 (highlighted in yellow in fig. S11A) in an experiment performed on subject #2. To quantify the responses, we averaged the MUA across the stimulation period (0 to 167 ms) for each train in every trial. These averages were used to generate the curves shown in fig. S11D (red for the first stimulation train and blue for the second). Notably, the curves representing MUA from the second stimulation train (blue) closely aligned with the psychometric function of perceived phosphene separation probability, derived from participants’ behavioral responses (see Fig. 5D for a representative example).

On the basis of these neural activity patterns observed during the second stimulation train, we analyzed the responses from all electrodes surrounding the stimulation sites to estimate the minimum temporal separation required for stimuli to be perceived as distinct sensations. We defined the neural threshold for perceiving two separate phosphenes as the intertrain delay at which the average activity during the second stimulation train (across all included electrodes and normalized using a z-score) reached 50% of its dynamic range. This procedure is illustrated in Fig. 5E for the previously described experiment (stimulation of electrode 54 in subject #2), using data from all the electrodes surrounding the stimulated one (blue electrodes in Fig. 5C). The same analysis is presented in Fig. 5 (G to I) and fig. S11 (E to H) for other representative experiment conducted in subject #1, in which a group of four electrodes (49, 50, 51, and 52) was stimulated. Figure S12 shows more examples of recordings and how neural thresholds for seeing two separate phosphenes are determined.

When this procedure was applied to all experiments conducted in both subjects, a significant correlation emerged between the intertrain delay associated with the subjective perception of two separate stimuli and the delay extracted from the neural activity. Specifically, for each experiment, an intertrain delay threshold for separate phosphene perception was extracted from the participant’s responses and a corresponding threshold was obtained from the neural data; these paired values were then used to compute the correlation across experiments in each subject (evaluated with the Pearson correlation coefficient). For subject #1 (Fig. 5J), this correlation was R = 0.74 (N = 11, P < 0.01), whereas for subject #2 (Fig. 5F), it was R = 0.72 (N = 12, P < 0.01). These findings suggest a direct link between conscious perception of two different phosphenes and the underlying brain responses.

DISCUSSION

Our results show that the neural activity induced by intracortical electrical stimulation of the human occipital cortex can be used to gain valuable insights into the perception experience, providing proof of concept for the use of bidirectional communication with the brain to create an advanced form of artificial vision in the blind.

A previous study in monkeys has shown that V4 neural activity can be used to determine phosphene thresholds in V1 (14). However, this approach requires implanting microelectrode arrays into multiple brain areas, posing substantial technical and clinical challenges. Our results go a step further and indicate that we can accurately predict not only phosphene thresholds but also brightness levels and the number of perceived phosphenes on the basis of the activity of neighboring electrodes in V1. This offers several advantages, particularly in terms of safety and clinical implications. Thus, implanting only in V1 reduces the invasiveness of the procedure, thereby minimizing the potential risks associated with implantation into several cortical areas. This, in turn, reduces the likelihood of complications such as infections, bleeding, and tissue damage. In addition, having fewer arrays leads to shorter surgical times, a reduction in the complexity of electrode placement, and simplifies the processing and analysis of neural signals.

These results have the potential to drive several key developments. First, these findings can contribute to the development of improved methods for automating the calibration procedures currently used for phosphene perceptions. This is a laborious and time-consuming process, requiring the prosthesis user to report whether currents are above or below the perceptual threshold on hundreds of electrodes. Using the correlation between neural responses and perception threshold can substantially streamline the setup process, thus facilitating the adaptation and acceptance of the device (31). In this framework, we should be aware that similar procedures have also been used to optimize stimulation parameters in response to specific visual stimuli in retinal prostheses (35, 36), as well as for enabling a better control of stimulating electrodes in the management of pain (37), epilepsy (38, 39), and various neurodegenerative diseases (40, 41).

Second, we found that it is feasible to estimate the time resolution of separate phosphene perception without relying on user feedback. Thus, we observed that the neural activity during the second stimulation of two consecutive trains is modulated by the intertrain delay. This modulation enabled us to know whether the user perceives a single continuous sensation or two distinct perceptions. In addition, although additional data are needed to fully understand how different stimulation parameters influence the temporal resolution of phosphene perception, the modulated neural responses to the second stimulation trains suggest the presence of adaptation effects in the primary visual cortex. Future studies should investigate how sequential electrical stimulation affects the perceived brightness of each phosphene and explore the role of higher visual and prefrontal cortical areas in processing intracortical stimulation in V1. Nevertheless, our results suggest that understanding the time resolution of phosphene perception is key for developing complex stimulation patterns. This is crucial for enabling more useful visual perceptions in future users of cortical prostheses.

Third, our results demonstrate that neural activity evoked by electrical stimulation can be used to assess the brightness of the perception. This finding suggests the possibility of dynamically adjusting stimulation parameters in real time to achieve the desired perceptual brightness, thereby compensating for fluctuations that may occur during prolonged prosthetic use. Such adaptability could help maintain an optimized and consistent visual experience for users over time. We acknowledge that poststimulation neural activity may also be influenced by decision-making processes related to the participant’s perceptual response. However, the observed neural response patterns indicate that this activity is primarily driven by the stimulation itself. Nonetheless, future experiments comparing trials with and without participant responses could aid in isolating the effects of decision-making.

In addition to revealing a correlation between neural activity and stimulation-induced percepts, this study provides several perceptual measures that can inform models of phosphenes appearance. Notably, the temporal resolution of separate phosphene perceptions and the effect of stimulation parameters (frequency, train duration, and number of pulses) on perceived brightness are aspects that have been largely underexplored in cortical visual prostheses. However, a recent study reported that the brightness perceived by users of visual prostheses is affected by the sequence of stimulation (29). To mitigate this effect, we introduced a sufficient interstimulus interval between subsequent stimuli. Nevertheless, additional research is required to fully characterize the underlying mechanisms and optimize future stimulation protocols.

We analyzed neural activity during stimulation for the perception threshold and separate perceptions tasks. In contrast, for the brightness task, our analysis focused on the activity after stimulation due to varying fast settle blanking times across different frequency conditions. Although a previous study has shown inhibition following intracortical electrical stimulation (42), our data revealed a more complex relationship. We found a correlation between neural activity during and after stimulation for electrodes neighboring the stimulated sites, with an anticorrelation only appearing for train durations longer than 250 ms (more details can be found in Supplementary Text and fig. S13). This strong relationship between activity recorded during and after stimulation suggests that either metric may be used interchangeably to assess the cortical response.

Combined with the demonstration of artificial vision by electrical stimulation of occipital cortex in blind subjects, the results presented in this study underscore the significance of incorporating bidirectional communication with the brain in the next generation of cortical visual prostheses (28). This approach allows for dynamic adjustments, leading to better adaptation and potentially faster learning curves for users. Moreover, monitoring the brain’s responses can help to detect subtle indicators of declining performance, allowing for timely adjustments, and can also help to identify early warning signs of abnormal brain activity. However, the capability to automatically adjust thresholds, time resolution, and stimulation parameters to achieve the desired perceptions requires enhanced computational power in both stimulation and recordings systems. We hope that, thanks to the advancements in this field, future users will no longer need to manually adjust parameters. Instead, an automated procedure involving stimulation and real-time analysis of neural responses will be seamlessly executed by the prosthesis system, even when the user is not actively engaged in tasks requiring the device (31). Such a system could continually refine its output on the basis of the brain’s evolving responses, leading to better adaptation and providing more precise and personalized visual percepts for the users.

MATERIALS AND METHODS

Experimental procedures in blind human subjects

All the studies were conducted in accordance with a protocol approved by the Hospital General Universitario de Elche Clinical Research Committee and registered at ClinicalTrials.gov (NCT02983370). We observed and complied with all pertinent ethical guidelines associated with clinical trials regulation [EU no. 536/2014 (repealing Directive 2001/20/EC), the Declaration of Helsinki, and the EU Commission Directives 2005/28/EC and 2003/94/EC]. Before participating in the study, all risks and procedures were thoroughly explained to the participants, emphasizing the investigational nature of the study. The participants were fully aware that the main purpose of the study was to gain knowledge essential for the future development of a cortical visual prosthesis for the blind and provided informed consent.

The experiments were conducted with two male blind subjects with irreversible bilateral optic neuritis and complete lack of light perception. Subject #1, aged 64 at the time of implantation, had lost his vision 2.5 years before the procedure. Subject #2, aged 61 at the time of implantation, had been blind for 16 years before implantation. Both subjects exhibited good cognitive function and functional abilities, with no underlying health issues. Before their involvement in the study, both participants underwent a thorough medical and psychological evaluation. This assessment included a detailed review of their medical history, comprehensive physiological and neurological examinations, and an assessment of their psychological state. In addition, we conducted a systematic mapping of the visual sensations induced by noninvasive transcranial magnetic stimulation (TMS) of the occipital cortex to help identify the optimal implantation sites, as previously reported (43).

Surgery

A detailed description of the surgical procedures has been provided elsewhere (15, 34). Briefly, the surgical procedure began with the sterilization of the scalp using an antiseptic solution. Then, a linear incision was made over the entry point, guided by a neuronavigation system, followed by a 10-mm burr hole targeting the intended area for array implantation. Upon opening the dura, the brain’s surface was exposed, and the UEA was inserted using a pneumatic inserter manufactured by Blackrock Microsystems Inc. (Salt Lake City, UT). The external connector was affixed to the skull using titanium microscrews. After 6 months, both subjects underwent another surgery to explant the UEA.

Implant location

The UEA was implanted in the right occipital pole of both subjects (see Fig. 1 for more information). The specific implant location in the occipital cortex for each subject was selected for easy access during surgery while avoiding major blood vessels. Anatomical images of the occipital cortex and other areas of the visual system were acquired on a 3-T magnetic resonance scanner (Siemens MAGNETON Skyra) using the MPRAGE protocol (192 sagittal slices, 256 by 256 matrix, 1 mm by 1 mm by 1 mm, repetition time (TR) = 1900, echo time (TE) = 2.49, and fat saturation (FS) = 3). We used Horos 4.0 (https://horosproject.org/) to create three-dimensional (3D) reconstructions of the cerebral surface showing cortical anatomy and the major cortical vessels and the FreeSurfer image analysis suite v6.0 (https://surfer.nmr.mgh.harvard.edu/) for cortical reconstruction and volumetric segmentation. We used the method described by Benson et al. (44, 45) to estimate the retinotopic organization of the occipital cortex of the blind subjects using cortical surface anatomy to predict visual areas V1, V2, and V3.

The UEA was positioned in between V1 and V2 for subject #1 and in V1 for subject #2. For subject #1, the electrodes in V2 were those on the right side of the maps shown in the manuscript. We did not observe any difference in the neural responses or in the phosphenes produced using electrodes in V1 and in V2; thus, on the basis of the electrode neighbors of those being stimulated, in some cases, we pooled the data collected from electrodes in V1 and V2.

Neural data recording and visual cortex stimulation

We used the Ripple Neuromed Summit Explorer processor (Ripple Neuromed, Salt Lake City) for recording and stimulation. Three Micro2+Stim front ends (32 channels each) were connected to the processor and to the UEA’s connector. Neural signals were recorded at 30 kHz from 96 channels applying a 0.1- to 7.5-kHz analog filter. The resolution of the analog-to-digital conversion was set to 0.25 μV (16 bit). The stimulation step size was set to 1 μA.

External events were sent through coaxial cables and recorded through the Grapevine Digital I/O front end and the Grapevine Analog I/O front end. The timing of electrical stimulation was recorded as a digital event in the stimulation data stream.

During each stimulation pulse, the recording was blanked in all the channels using the fast settle option of the processor. Fast settle disables the recording for the duration of the pulse plus 0.5 ms and increases the quality of the recording during stimulation. In particular, the fast settling enables measurement of neural signals between stimulation pulses and allows quicker recovery of recordings on the stimulation channels immediately after the stimulation train ends.

From the raw data we extracted the enveloped MUA using previously described procedures (14). Briefly, the raw signals were filtered between 500 and 9000 Hz, full-wave rectified, low-pass filtered at 200 Hz, and downsampled at 1 kHz. The multiunit signal offers an averaged representation of the spiking activity from several neurons near the electrode tip. Consequently, the population response obtained through this approach is expected to be similar to the population response achieved by pooling data from numerous single units (46, 47).

Artifact removal

To extract the MUA from the data during stimulation, we applied an artifact cleaning procedure similar to the one proposed by Chen et al. (14). An example of raw signal during stimulation, with the blanking periods produced by the fast settle option, is shown in fig. S2B. Initially, we removed 0.5 ms after the end of the blanking period to remove the steep decay of the signal. This step was particularly important for the stimulation channels and their neighboring electrodes. We then fitted a third-order polynomial in each of the unblanked signal window to remove slow trends and maintain only the high-frequency components of the signal (see fig. S2, C and D). Last, to eliminate the high frequencies introduced by the transition from blanked signal to neural signal, we replaced the zeros with a linear interpolation between the last neural signal time point and the first after the blanking. Following this step, the MUA extraction procedure remains as previously stated. Figure S2 (E to H) shows the process applied to four different electrodes at several distances from the stimulation electrode.

In general, the MUA extracted from the signal portion affected by artifacts tends to be smaller compared to the MUA from a segment without artifacts due to the blanking periods during which high-frequency activity is absent (see the MUA of electrodes 18 and 87 in fig. S2H).

The efficacy of the method to give a measure proportional to the neural activity during stimulation is proven by the neural activity induced in the intertrain protocol (fig. S11, B, C, F, and G). The neural activity in the first and second stimulation train is significantly different (especially for low intertrain delays) even if the stimulation current (and therefore the artifact) is the same in the two trains.

Behavioral experiments

We performed standard psychometric experiments to assess the perception thresholds, defined as the minimum current needed to elicit a perceptual response. A predefined set of stimulation parameters, including frequency, number of pulses, and train duration, was selected (usually 300 Hz, 50 pulses, 167 ms; see table S1 for a complete summary of all the parameters). During the experiments, participants received an auditory cue followed by electrical stimulation (see Fig. 2A). The current amplitude varied randomly from 1 μA to a maximum value, ranging from 61 to 91 μA, depending on the number of electrodes used for stimulation, with step increments of 5 or 10 μA. Each current amplitude was repeated 10 times. In addition, 10 sham trials were included, wherein only the auditory cue was presented without electrical stimulation. We also conducted another set of experiments in which we kept the current constant while varying the pulse width from 50 to 230 μs in 20-μs increments (subject #1) and from 10 to 110 μs in 20-μs increments (subject #2). Both variations in current amplitude and pulse width modify the charge delivered by each pulse, which is the variable directly related to the threshold for neural activation (48).

The participants were requested to report whether they had perception or not using a keyboard. Following their response, a 1-s pause preceded the next stimulation. The probability of perception was determined on the basis of participant’s responses as a function of the stimulation current amplitude (or pulse width).

To investigate brightness perception, we selected a specific electrode at a given current amplitude and pulse width and varied the frequency (f) and the train duration (TD), with the number of pulses adjusted accordingly (pulses = f × TD). The frequency values used were 50, 80, 100, 150, 200, 250, 300, and 400 Hz; the train duration values were 100, 150, 200, 250, 300, 350, 400, 450, and 500 ms. Not all the combinations were present in every experiment (see table S2 for a complete list of the parameters of each experiment). Each combination of frequency and train duration was repeated five times within the same trial. During the experiments, participants were prompted by an auditory cue followed by electrical stimulation. They were then asked to rate the brightness of the perception on a scale from 0 to 5. Following the participant’s response, a 1-s pause preceded the next stimulation. The sequence of frequency and train duration parameters was randomized, and five sham trials were included. The sham trials consisted of the auditory cue without electrical stimulation (Fig. 3A). From the participant’s responses, we obtained the relationships between stimulation parameters and perceived brightness. The effects of frequency, train duration, number of pulses and their interaction on the perceived brightness were assessed using regression analysis. We evaluated which parameter (frequency, train duration, and number of pulses) and combination of parameters best predicted perceived brightness, comparing models using R², AIC, and BIC (see table S3).

We used a psychometric intertrain task to investigate the minimum time interval between two stimulation trains required by the participants to perceive two separate phosphenes. We selected either a single electrode or a group of electrodes and defined a set of stimulation parameters (including frequency, number of pulses, train duration, pulse width, and current amplitude; see table S4 for a complete list of the parameters for each experiment). We then varied the time between the two stimulation trains. Depending on the experiment, the intertrain delay varied between a minimum between 0.05 and 0.1 s to a maximum between 0.55 and 1 s in increments of 0.1 s. Each intertrain interval was repeated 10 times. During the experiment, participants were presented with an auditory cue followed by two electrical stimulations separated by a specific intertrain time. Participants were instructed to report whether they perceived zero, one, or two separate perceptions using a keyboard. After the participants’ response, a 1-s pause preceded the next pair of stimulations. The sequence of intertrain delays was randomized, and 20 sham trials were included: 10 sham trials with only the auditory cue and no electrical stimulation, and 10 sham trials with the auditory cue and only one stimulation train (see Fig. 5A). From the participants’ responses, we derived the probability of perceiving two separate phosphenes as a function of the time between the stimulations.

Correlation between behavioral measures and neural activity

To establish the relationship between behavioral thresholds and neural activity, we fitted sigmoidal functions to the psychometric curves. The current amplitude at which the sigmoidal fit reached 50% was deemed the behavioral current threshold. Then, for each repetition of the stimulation, we computed the MUA during the stimulation train and determined the mean MUA after applying a low-pass filter at 17 Hz (second-order Butterworth filter). Subsequently, we identified all electrodes neighboring the stimulated ones, with an average MUA during stimulation (0 to 167 ms from the beginning of the stimulation train) exceeding 5.5 μV (twice the average value obtained during stimulation from electrodes without neural activity). These average MUA values were then normalized with z-score and grouped by stimulation current amplitude. The z-score normalization allows for the grouping of electrodes that show the same trend for different stimulation parameters but have different activity ranges. We then fitted a sigmoidal function to the mean across channels included and repetitions of the z-score average MUA per current intensity and considered as neural current threshold the current at which the sigmoid reached its half range value. In experiments where pulse width was varied, we applied the same procedure to the pulse width instead of the current. To combine the results from the two types of experiments, we converted the threshold into charge values by multiplying by the constant pulse width in the current experiments and by the constant current in the pulse width experiments.

The same procedure was applied to the psychometric intertrain experiments. We considered the intertrain delay at which the sigmoidal fit of the probability of separate perceptions on the basis of the subjects’ answers reached 50% as the behavioral threshold. We applied the same processing to the neural signals during the second stimulation of each repetition (0 to 167 ms from the beginning of the second stimulation train of each trial). We fitted a sigmoidal function to the mean across channels included (electrodes neighboring the stimulated ones, with a mean MUA during stimulation exceeding 5.5 μV) and repetitions of the z-score average MUA per intertrain delay and considered as neural threshold the intertrain delay value at which the sigmoid reached its half range value.

For both types of psychometric experiments (current threshold and intertrain), we used the linregress function from the python scipy.stats package to calculate the correlation between behavioral and neural measures across all the experiments performed. The function calculates the linear least-squares regression and returns the Pearson correlation coefficient (R), the slope and intercept of the line which best represents the data and the significance of the slope being different from 0 (Wald test).

In the experiments for investigating the brightness of the perception, we extracted the average MUA in a time window going from 20 to 300 ms after the end of each stimulation train. We did not use the activity inside the stimulation train because with different frequency/train duration combinations, the amount of fast settle blanking was different so that the neural data were not comparable inside the stimulation for different parameters. Using linregress function, we then calculated the correlation between the average brightness reported by the participant for each frequency/train duration combination and the MUA activity induced for each channel. We then analyzed the overall trend across all experiments, grouping the correlation with brightness for electrodes at the same distance from the stimulation sites.

Supplementary Materials

This PDF file includes:

Supplementary Text

Figs. S1 to S13

Tables S1 to S4