Image-dependence of the detectability of optogenetic stimulation in macaque inferotemporal cortex

Summary

Artificial activation of neurons in early visual areas induces perception of simple visual flashes^1,2. Accordingly, stimulation in high-level visual cortices is expected to induce perception of complex features^3,4. However, results from studies in human patients challenge this expectation. Stimulation rarely induces any detectable visual event, and never a complex one, in human subjects with closed eyes². Stimulation of the face-selective cortex in a human patient led to remarkable hallucinations only while the subject was looking at faces⁵. In contrast, stimulations of color- and face-selective sites evoke notable hallucinations independent of the object being viewed⁶. These anecdotal observations suggest that stimulation of high-level visual cortex can evoke perception of complex visual features, but these effects depend on the availability and content of visual input. In this study, we introduce a novel psychophysical task to systematically investigate characteristics of the perceptual events evoked by optogenetic stimulation of macaque inferior temporal (IT) cortex. We trained macaque monkeys to detect and report optogenetic impulses delivered to their IT cortices^7–9 while holding fixation on object images. In a series of experiments, we show that detection of cortical stimulation is highly dependent on the choice of images presented to the eyes and it is most difficult when fixating on a blank screen. These findings suggest that optogenetic stimulation of high-level visual cortex results in easily detectable distortions of the concurrent contents of vision.

eTOC Blurb:

Azadi et al. find that perception of cortical stimulation in inferior temporal cortex depends on the choice and visibility of images presented to the eyes at the time of stimulation, and it is the least perceivable when no image was presented. This suggests that the physiological effect of local stimulation varies with the state of vision.

Results

We used the Opto-Array^10,11, a chronically implantable array of LEDs, to stimulate the same cortical sites expressing excitatory opsin C1V1 across hundreds of sessions. This was done in the context of a highly sensitive cortical perturbation detection (CPD) task^7–9 unrestricted by prior assumptions about function of the stimulated neurons. We trained two monkeys to behaviorally detect a short optogenetic stimulation impulse delivered to their IT cortex while fixating on images of various objects and scenes (Figure 1A). In each trial, following fixation on a central cue, an image was displayed on the screen for 1 s. In half of the trials, randomly selected, a 200 ms illumination impulse was delivered to IT cortex halfway through the image presentation, and the animal was rewarded for correctly identifying whether the trial did or did not contain cortical stimulation. The image content was independent of whether brain stimulation would or would not occur and the animals’ exclusive behavioral task was to detect if brain stimulation occurred in a given trial. We reasoned that if different visual inputs affect the magnitude of the perceptual events evoked by stimulation, varying the visual input will directly affect the animals’ ability to detect the cortical stimulation. This approach enables us to systematically assess the effect of the visual input characteristics on the perceptual events evoked by cortical stimulation.

Figure 1. Behavioral task, surgical procedure and training phase results.

(A) Behavioral task: in each behavioral trial following fixation an image was displayed on the screen for 1 s. In half of the trials, randomly selected, a 200 ms illumination impulse was delivered to IT cortex halfway through image presentation. The animal was rewarded for correctly identifying whether the trial did or did not contain cortical stimulation by looking at one of the two subsequently presented targets at the end of trial.

(B) Schematic illustration of the procedure for chronic optogenetic stimulation of IT cortex. Left, injection of AAV5 expressing the excitatory opsin C1V1. Right, Opto-Array implantation: in a separate surgery, we visually confirmed the expression of the excitatory virus and implanted an Opto-Array over the expression zone. We implanted a second array in the corresponding region of the opposite hemisphere where no virus was injected (control site, not shown).

(C) Behavioral performance of monkey Ph as a function of session number during the training phase. The y-axis indicates the proportion of the trials reported as stimulated. Red, blue and yellow colors represent data from the stimulation, non-stimulation, and catch trials respectively. Error bars represent bootstrapped 95% confidence intervals. Note that the y-axis does not directly represent performance here, instead the separation between the red and blue lines illustrate the difference between stimulation and nonstimulation trials implying the performance. This difference became significant at session 4 (arrow) and remained so through the training. Fluctuations of performance in time represent usage of different visual stimuli and stimulation intensities throughout the training. No significant difference was found between the catch and non-stimulation trials. The violin plots on the right side illustrate the mean and bootstrapped 95% confidence interval of stimulation report rate for each trial type in the last 3 sessions, between the dashed lines.

See also Figure S1

To stimulate the cortex, for each animal, we implanted an Opto-Array over the central IT cortex where we had previously injected Adeno Associated Virus 5 (AAV5) expressing excitatory opsin C1V1 under the CaMKIIa promoter (right and left hemispheres in monkeys Ph and Sp, respectively; Figure 1B). We also implanted an array in the corresponding region of IT in the opposite hemisphere where no virus was injected. This array served to control for nonspecific effects of LED illumination such as heat or light.

Training phase: Monkeys robustly learned to detect optogenetic cortical stimulation of IT cortex

The animals learned to accurately detect and report cortical illumination over the opsin-expressing site in only a few sessions, yet they were not able to detect illumination over the control site throughout training and Experiment 1 (catch trials). Figure 1C shows the performance of monkey Ph as a function of session number during the training phase (monkey Sp performed similarly). The difference between stimulation report rate in stimulation and non-stimulation trials became and stayed statistically significant after 4 and 11 sessions, respectively, in monkeys Ph and Sp (Figure 1C, red and blue lines. arrow: Ph: X² (1, N = 1337) = 6.7, p = 0.010 and Sp: 30.4 X² (1, N = 1337) = 30.3, p < 0.001). This difference remained significant throughout the training phase (Ph: p = 0.010, Sp: p < 0.001 for each session). In contrast, performance on catch trials - which were rewarded with the same contingency as stimulation trials (Figure 1C, yellow line) - did not differ from non-stimulation trials (Ph: p > 0.142, Sp: p > 0.054 for each session), implying that on stimulation trials the animal is reporting detection of cortical stimulation, rather than some other artifact of LED illumination, such as heat or light. These results show that excitatory optogenetic stimulation in IT cortex is behaviorally detectable in monkeys.

Experiment 1: Detection of IT stimulation depends on the content of the visual input

To determine if the perceptual event evoked by optogenetic cortical stimulation is affected by the visual input, monkeys performed the CPD task with an image set that consisted of 40 novel images, including a condition with no image in which the subject viewed only the uniform gray background (Figure S1A). In the stimulation trials (50% of trials), we randomly interleaved stimulations of two cortical sites (~3 mm apart). We found that the performance in detection of cortical illumination systematically varies by site while the animals fixate on the same set of images, creating a unique array of performances for each cortical position that we refer to as the ‘detection profile’. Figure 2A shows a detection profile for one stimulation site in monkey Ph (see Figure S2 for more). Performance levels were significantly different across the images (permutation test of randomly selected images per trial Ph: p < 0.001, Sp: p < 0.001 for both stimulation sites). The animals’ performance for the ‘no image’ condition was the lowest in all detection profiles obtained (permutation test of randomly selected images versus no image for each trial Ph: p < 0.001, Sp: p < 0.001 for both stimulation sites). To test the consistency of the detection profiles across the sessions, we created two sets of data by separating the trials from odd and even sessions. Then we computed the correlation between the detection profiles for each set. The results showed a strong and significant correlation between the odd and even sessions for both stimulation sites (Pearson’s r(39) = 0.79, 0.76 and 0.82, 0.72; p < 0.001 Benjamini-Hochberg corrected, for each stimulation site, respectively for monkey Ph and Sp).

Figure 2. Stimulation detection performance is modulated by visual input, cortical location, and illumination power.

(A) Left, detection profile: the behavioral performance (d’) on the CPD task for 40 images. The black dots represent d’ for each image and the violin plots represent bootstrapped 95% confidence intervals. Right, permutation test: the blue line indicates the standard deviation of d’s across images, and the red histogram represents results from a permutation test with 10,000 times randomly assigned images on trials revealing the statistical significance of the effect of image on performance.

(B) left, correlation between detection profiles within each cortical stimulation site and between them. The violin plots represent 95% confidence intervals of the bootstrapped distribution of the correlations with 10,000 resamples, and the horizontal lines indicate their medians. Right, permutation test: the blue line indicates the observed correlation between the sites. The red histogram represents results from a permutation test with 10,000 times random assignment of stimulation condition over the trials. This shows that the correlation of detection profile patterns between the sites is significantly lower than the null distribution.

(C) left, detection performance (d’), as a function of illumination power. Each line represents data from 1 image (5 images in total including the no image condition). Right, permutation test, the standard deviation of the coefficients for each image, derived from fitting of the psychometric curves. The blue line indicates the observed value, and the red distribution represents the null distribution generated by 10,000 times randomly assigning the image indexes to the trials. This confirms the coefficients are significantly different from each other.

See also Figures S2 and S3

Counterintuitively, but consistent with the human reports, these results show that the content of the visual input alters the effect of optogenetic stimulation. Moreover, these results suggest that the perceptual event evoked by stimulation is visual in nature as its detectability interacts with the visual input.

Could the observed variation in performance across images be explained by image-specific choice biases, irrespective of brain stimulation? If not, we would expect the detection profile to change when stimulating a different site in IT cortex. To test this, we measured detection profiles at two distinct cortical sites, though the distance between sites was limited by the span of the Opto-Array. To assess the difference between the detection profiles obtained from the two stimulation sites, we trained a Support Vector Machine (SVM) to discriminate the stimulation sites from their detection profiles. The classifier was able to predict the stimulation sites with a performance above chance level (5-fold cross-validated performance = 81.0% and 78.5%; permutation test p = 0.010 and 0.013 respectively for monkey Ph and Sp), meaning the detection profiles are significantly different from each other. Moreover, a correlation analysis also showed a significant difference between the detection profiles. The detection profiles of the two sites in each animal were correlated (Pearson’s r(39) = 0.91 and 0.82 respectively in Ph and SP; p < 0.001 for both subjects) yet significantly different from each other (Figure 2B for Ph and Figure S3A for Sp). A Pearson’s correlation analysis of the hit rates derived from the two distinct stimulation sites revealed that the correlation between patterns of performance across the image set was significantly larger within a stimulation site than across sites ( Ph: p = 0.009, Sp: p = 0.002). This difference indicates that the detection profile changes with cortical position and does not exclusively result from image specific choice biases. The high baseline correlation between sites may result from potential overlap of dendritic branches or similarity of their neural characteristics (e.g. object selectivity) as the two sites were only ~3 mm apart. Detection profiles were uncorrelated across the two subjects (Pearson’s r(36) > 0.07 and < 0.29, p > 0.077 for all four comparisons).

Experiment 2: Detection profiles are stable across different illumination powers

Experiment 1 reveals that looking at some images helps the animals detect IT stimulation, and that the rank of images for doing so varies across cortical sites and animals. The ability to detect cortical stimulation also likely depends on nonspecific factors such as virus expression heterogeneity and potential tissue build up under the array that may vary the effective cortical illumination power. In Experiment 2 we tested whether the rank order of the images remained constant across different illumination powers (7 different power levels and 5 visual stimuli including the ‘no image’ condition). Figure 2C shows the psychometric functions obtained from one stimulation site in Experiment 2 for Monkey Ph (see Figure S3B-D for more). Illumination power had a significant effect on performance (Pearson’s correlation: Ph: r(33) > 0.78, p < 0.001, Sp: r(53) > 0.79, p < 0.001 for both stimulation sites) as did the choice of image (permutation test of randomly selected images per trial Ph: p < 0.001, Sp: p < 0.001 for both stimulation sites). The no image condition led to the poorest performance in both animals (permutation test of randomly selected images versus no image per trial Ph: p < 0.001, Sp: p < 0.001 for both stimulation sites).

This reveals the robustness of the effect of the visual input on the detection of cortical stimulation. We have used special precautions to achieve homogeneous expression of the virus across the tissue¹², as well as the accurate alignment of the array with the cortex, however, it is still possible to attribute the differences of detection profiles across cortical sites to minor heterogeneities in virus expression or effective illumination. The robust preservation of image ranks across a large range of illumination powers makes it difficult to explain the variation of image rank across cortical positions by nonspecific factors that may influence effective illumination.

Experiment 3: Detection of IT stimulation depends on the visibility of the visual input

Human studies show that patients rarely detect stimulation in high level visual areas when they are blindfolded in a dark room². We find this observation extremely counterintuitive and interesting. One might think that eliminating the visual input would help the subject perceive the pure effect of stimulation, but the report suggests the opposite. This has important potential theoretical implications (see Discussion) but the study does not provide a positive control with open eyes in the light. As the authors suggested, had the patients been viewing real stimuli, they might have experienced the effect of stimulation. Another consideration here is that closing the eyes changes the brain state as evident from the EEG literature¹³. In the ideal scenario the visibility of the visual input should vary within the experimental procedure without closing the eyes.

Experiments 1 and 2 revealed low performances for detecting stimulation in the “no image” condition, which is consistent with results from cortical stimulation in humans². But the “no image” condition was an unusual condition among multiple visual stimuli and odd-ball psychophysical effects may have contaminated this finding, a shortcoming that inspired Experiment 3.

In order to systematically address these important considerations, we tested how the attenuation of visibility of screen images affects detection of the cortical event. The animals performed the CPD task while fixating on randomly presented images of five objects at four visibility levels in addition to a no image condition (Figure S1). Visibility was degraded by reducing the saturation, spatial frequency and contrast of the images.

Visibility of the image had a strong effect on stimulation detectability (Figure 3 and Figure S3E; one-way ANOVA Ph: F(4,16) = 5.24, p = 0.006; Sp: F(4,16) = 22.33, p < 0.001). Spearman’s Ph: r(20) = 0.71, p < 0.001; Sp: r(20) = 0.78, p < 0.001). The monkeys’ performance increased with the visibility of the visual input; the fully visible stimuli produced significantly higher performance compared with the two lower levels of visibility and with the “no image” condition (p < 0.001 and < 0.001 for all comparisons for monkeys Ph and Sp). The animals’ performance in the “no image” condition was still significantly above the chance level (p < 0.001 and < 0.001 for monkeys Ph and Sp) which could be due to the fact that we did not completely eliminate the visual input (e.g. the monitor was still visible) or a potential residual effect that persists even in the absence of the visual stimulus. Even though a fixation point was presented throughout the trials, visibility of visual content may modify fixational eye movements, which can potentially be the cause of the observed changes in the performance. To control this possibility we calculated the variance of gaze position from the start until the end of optogenetic stimulation time (first 700ms). The results showed no significant difference in variance of gaze position as a function of image visibility (one-way ANOVA Ph: F(4,16) = 1.96, p = 0.148; Sp: F(4,16) =0.389, p < 0.812; Spearman’s Ph: r(20) = 0.19, p = 0.400; Sp: r(20) = 0.14, p = 0.537).

Figure 3. Stimulation detection performance is modulated by image visibility.

The x-axis represents 4 levels of image visibility and the gray background, used in experiment 3. The y-axis is the detection performance (d’) on the cortical stimulation detection task. The thin lines represent data from 5 different images and the thick line illustrates the overall averages. Error bars represent 95% confidence intervals. There is a significant correlation between the image visibility and performance (r = 0.7). The p-values for pairwise comparisons are from post-hoc tests of ANOVA (Benjamini-Hochberg corrected).

See also Figures S3

Discussion

These results reveal that optogenetic stimulation of a small subregion of IT cortex evokes visual events that are easily detectable by the subjects. Detectability of these events depends on cortical location, illumination power, and the content and visibility of the visual input. In particular, these events are strongly and selectively enhanced by concurrent object-related activity in the visual system and are not psychophysically isolated from the ongoing visual perception. These results reconcile the conflicting accounts of stimulation of high level visual areas in humans ^2,5,6,14. Specifically, the fact that detection of cortical stimulation highly depends on the visual input explains why stimulation in the high level visual cortex does not always lead to detectable perceptual events in human patients.

Previous studies in nonhuman primates have provided evidence for stimulation induced perceptual effects, but were agnostic to the question of whether these events depend on the concurrent visual input. Stimulation of the primary visual cortex in macaque monkeys has been shown to induce changes in the apparent brightness of specific locations in the visual field, also known as “phosphenes”¹⁵. Electrical stimulation of macaque middle temporal (MT) cortex biases the animals’ perceptual judgments in direction detection¹⁶, and depth discrimination tasks¹⁷. Cortical stimulation of disparity-tuned neurons in area V4¹⁸ and IT cortex¹⁹ biases depth perception. Stimulation of IT gloss-selective neurons induces corresponding biases in a gloss discrimination task²⁰. Stimulation of face-selective subregions of IT cortex decreases the threshold for detecting faces²¹ and optogenetic silencing of small clusters of face-selective neurons takes a toll on the ability to discriminate faces²². These studies suggest that stimulation of neurons tuned to a given visual dimension (e.g. face neurons) induces perceptual biases along that dimension but remain agnostic to the question of whether the perceptual events induced by cortical stimulation depend on the concurrent visual input. An observation by Murasugi et al. provides a tantalizing clue: stimulation of a direction selective column in MT cortex induces little perceptual biases when the animal fixates at a blank screen²³. These biases are larger when static white noise is presented to the animal, and are largest when a dynamic noise pattern is used. These results show that the magnitude of the perceptual events caused by stimulation varies depending on the concurrent visual state. A more recent study found that stimulation of face-selective parts of IT cortex strongly affects match-to-sample performance for faces but not other stimuli²⁴; this report suggests that microstimulation of face selective neurons leads to a more dramatic perceptual event when the animal is looking at faces compared to objects. However, the match-to-sample task structure invites alternative explanations, given that the animal was incentivized to ignore the effect of stimulation. For instance, if stimulation of face selective cells induces a hallucinatory face when the monkey is looking at faces and objects, this event might be easier for the animal to ignore in the object matching condition but not in the face matching condition. The present report avoids this potential issue by not requiring the animal to perform any task related to the visually presented stimuli and directly measuring detectability of cortical stimulation. This, in practice, allows measurement of the isolated effect of the visual input on behavioral detection of cortical stimulation.

Optogenetic manipulations in the visual system have struggled to induce large behavioral effects in nonhuman primates²⁵. However, the present results showed robust behavioral effects using cortical optogenetic stimulation despite using minimal power of illumination. We attribute this to the sensitivity of our psychophysical task that was not biased to any assumptions about the neural properties of the stimulated cortex. For example, Afraz 2015 found that optogenetic perturbation affected performance on a face gender discrimination task by a small amount. However, this perturbation may have caused a large effect on other perceptual dimensions, but a small effect on the tested task. In contrast to that highly specific task, the detection task used here allows any possible effect to be reported. This is consistent with previous work in nonhuman primate somatosensory cortex using a similar optogenetic CPD task which also induced a large behavioral effect⁸.

Optogenetics provides several advantages over electrical stimulation in this study. Optogenetic illuminations only change the activity of neurons expressing the opsin. This rules out off-target stimulation such as subcortical axons-of-passage, and allows cell type specific targeting. In our study, we aimed to target excitatory neurons by using the CaMKIIa promoter, which causes preferential expression in excitatory cortical neurons^26,27. Moreover, chronically implanting Opto-Arrays avoids the tissue damage associated with both acute optogenetic preparations and chronic electrical stimulation methods, and permits collection of thousands of trials over dozens of sessions at one cortical site.

The current Opto-Array technology does not allow recording of neural activity, limiting the scope of this study to phenomenology of the stimulation-induced events. Yet it is hard not to speculate about the neural underpinnings of the observed effects. Stimulation of ~1 cubic millimeter of tissue by activation of 1 LED on the array¹⁰ is expected to engage IT cortex at a scale that still preserves object category selectivity^28–31. The fact that behavioral detection of local cortical perturbation of this scale interacts differentially with various objects is consistent with the heterogeneity of object responses across IT cortex.

At the neural level, two categories of mechanisms potentially explain the behavioral variance for the different images: first, mechanisms that modulate the input of the targeted neurons, second, downstream mechanisms that decode their output. The first explanation suggests that the excitability of the stimulated neurons vary with the choice of the image³². The net effect of excitatory/inhibitory input to the neuron is different for various images, thus the same illumination level drives the neuron differently. While we do not know how the targeted neurons responded to different visual objects at the stimulated site, we know that diminishing visibility of the retinal input reduces neural responses in IT³³. Dependence of cortical stimulation detectability on the visibility of the retinal input is consistent with the idea that it is harder to artificially drive IT neurons in the absence of bottom-up input. Although we stimulated the cortex with equal optical power in all trials, the neurons might be vulnerable to activation or resist stimulation depending on the input they receive from other neurons. The second possible mechanism implies that optical stimulation activates the neurons independent of the visual input, but the downstream read-out mechanisms filter the effects of stimulation differently depending on the state of the rest of the visual system. Further modeling and physiology work is required to understand the underlying neural mechanics of a psychophysical landscape that looks promising.

STAR★Methods

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by Reza Azadi (reza.azadi@nih.gov).

Materials availability

This study did not generate new unique reagents.

Data and Code Availability

All the data collected during the experiments from both animals has been deposited at Figshare. DOIs are listed in the key resources table.

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Bacterial and virus strains
Adeno-associated virus	UNC Gene Therapy Center- Vector Core	AAV-CaMKIIa-C1V1(E122T/E162T) -TSEYFP
Deposited data
Raw and analyzed data	Figshare	https://doi.org/10.6084/m9.figshare.21685595
Analyzed data and codes	This paper; GitHub; Zenodo	https://doi.org/10.5281/zenodo.7407390
Experimental models: Organisms/strains
Macaca mulatta	National Institute of Mental Health Animal Study Protocol LN-28	Monkey Ph, Monkey Sp
Software and algorithms
MWorks	Release 0.9	https://mworks.github.io
macOS	macOS 10.12	https://www.apple.com
MATLAB	MATLAB 2020a	http://www.mathworks.com
Other
Opto-Array	N/A	https://blackrockneurotech.com
EyeLink 1000	EyeLink 1000	https://www.sr-research.com

All original code has been deposited to Zenodo. DOIs are listed in the key resources table.

Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.

Experimental model and subject details

In this study, we performed three experiments and collected data from two adult male rhesus monkeys (Macaca mulatta), referred to as Sp and Ph. All procedures were conducted in accordance with the guidelines of the National Institute of Mental Health Animal Use and Care Committee.

Method details

Surgical procedure

In a sterile surgery under general anesthesia we performed a craniotomy and opened the dura to access the surface of IT cortex (left hemisphere in Sp, right hemisphere in Ph). We then injected AAV5-CaMKIIa-C1V1(t/t)-EYFP (nominal titer: 8×10¹² particles/ml) into the cortex. To ensure uniform viral expression and reduce anesthesia-controlled time, we used an injection array¹² including four 31-gauge needles arranged in a 2×2 mm square. We placed the injection array four times, tiling central IT cortex with sixteen evenly spaced injection sites, resulting in a region of ~ 6mm x 6mm viral expression. Each needle was connected with flexible tubing to a 100 μl Hamilton syringe, and injection was controlled by a microinjection pump (Harvard Apparatus Pump 11 Elite). At each injection site, 10 μl of virus was injected at a 0.5 μl/min rate, for a total volume of injection of 160 uL. Ten minutes were allowed to elapse after each injection before removing the array to allow the virus to diffuse into the cortical tissue.

Several weeks later (12 and 4 weeks in Sp and Ph respectively), in a second surgery, we confirmed the virus expression and implanted an Opto-Array (Blackrock Microsystems) on the injection site. To confirm the viral expression, we used the fluorescent signature of the enhanced yellow fluorescent protein (EYFP) coexpressed with the opsin in transduced cells, by shining a 490–515 nm wavelength light (with a NIGHTSEA Dual Fluorescent Protein Flashlight) and viewing the cortex through 550 nm longpass filter-goggles (NIGHTSEA). This fluorescent signature was confirmed in Monkey Sp, but not in Monkey Ph. Therefore, in Monkey Ph, before proceeding with the array implantation, we performed a second virus injection similar to the first injection procedure (3 injection array placements, yielding 12 injection sites in a region of 4 × 6 mm; 10 μl of virus injected into each site at 0.5 μl/min rate). Then we implanted an Opto-Array over the injection sites. The Opto-Array was placed directly on the pia mater and sutured to the neighboring dura. Following this, in the same surgery, we implanted a second Opto-Array on a similar area of the IT cortex in the opposite hemisphere (control site: right hemisphere in Sp, and left hemisphere in Ph) where no virus injection was performed. The detailed surgical procedure is explained in¹¹

Apparatus

The experiment was carried out with the monkey head fixed, positioned 57 cm from a 27 in, 3840×2160 pixel, 60 Hz, Dell P2715Qt monitor. Fluorescent room lights were turned on to avoid dark adaptation of the retinae. This was done to minimize the possibility that the monkey would detect the light from the Opto-Array through the skull. To guard against heating cortical tissue by LED activation, temperature on the LED die was monitored by a thermistor inside the Opto-Array at the beginning of each trial and trial delivery was paused if the temperature on the LED die rose more than 3° C above the baseline temperature, and restarted once they were less than 1° C above the baseline. A 3° C change at the LED die translates to approximately 0.5° C temperature change on the cortical surface; this temperature management regime is detailed in Rajalingham et al. 2021¹⁰. The experiment was controlled with a custom MWorks script (The MWorks Project), running on a Mac Pro 2018. Opto-Arrays were controlled by a Blackrock LED Driver (Blackrock Microsystems) running a custom firmware version for compatibility with MWorks. Gaze was tracked with an Eyelink 1000 Plus (SR Research). Animals were water-restricted in their cages and received liquid rewards for successfully completing trials.

Behavioral task

Monkeys were trained to perform a detection task in which they were rewarded if they correctly identified whether a trial did or did not contain an optogenetic stimulation impulse. The subject started a trial by fixating on a central fixation point (black-on-white bullseye, 0.4° outer diameter and 0.2° inner diameter) for 500 ms on a gray background. Then, an image (scaled so the largest dimension spanned 8° for most images and 30° for four scenes during training and two scenes in experiment 1) appeared on the screen for 1000 ms while the animal held fixation on a central target. In half of the trials (randomly selected) 500 ms from the image onset, an LED on one of the Opto-Arrays was activated for 200ms. Then the image and central fixation point disappeared and two response targets appeared on the vertical midline (white, 0.4° diameter, 5° above and below center). The subject reported the existence or absence of cortical stimulation by fixating for 100 ms on the corresponding response target to the condition. Then, the response targets disappeared and a unique sound was played for correct and incorrect responses. The subject received a juice reward for a correct response or a punishment of 3.5 s delay for an incorrect response before starting the next trial. Trials with broken fixations or a latency of more than 3 s for choosing a response target were considered as an incorrect response during the experiment and excluded from further analysis. A ~300 ms tone played at the same time the image appeared to indicate that a trial had started.

Throughout the training phase and all the experiments, 50% of the trials were ‘no-stimulation.’ The other 50% were trials in which an opto-array was activated. In ‘stimulation’ trials (40%−50% of all trials depending on the experiment and monkey, see experimental conditions for details), the opto-array on the virus-expressed site was activated and in ‘catch’ trials (0%−10% of all trials) the opto-array on the control site was activated. The catch trials used the same stimulation parameters and were rewarded and punished the same as stimulation trials. Performance above chance level on the catch trials would indicate that the subjects did not truly perform the task by detecting the optogenetic activation of IT neurons. This controlled for the possibility that the subjects might be glimpsing light through the skull, or be detecting a potential perturbation of the neural activities caused by the heat³⁴.

Behavioral Training

Both monkeys were operantly trained on the experimental task using a different set of images than would be used in the subsequent experiments (Figure S1). To maximize the signal that the monkey was learning to detect, we began training by activating five LEDs simultaneously with power of 10.6 mW and 12.1 mW per LED for Ph and Sp respectively in stimulation trials. To reduce choice bias, we employed a ‘correction loop’ procedure³⁵. Under this protocol, if the monkey chose the same incorrect response target more than three times in a row, every subsequently presented trial would be the opposite type until the monkey selected the correct response target. Data collected in correction loops were excluded from analysis. Ph. started the training phase with 2 images and the number of images was gradually increased to 22. Sp. started training with all 22 images, but we eventually reduced the number of images to 1 and slowly reintroduced the full training set like in Ph. Then, in both monkeys we gradually reduced the number of activated LEDs to one, and illumination power to 4.5 mW in Ph and 9.1 mW in Sp. We introduced catch trials to Ph. after 17 sessions at an initial rate of 5% of all trials, then after 23 sessions increased the rate to 10% of all trials which continued for the rest of training. Catch trials comprising 10% of all trials were included for Sp. in all training sessions. In total, the subjects performed 42 and 48 sessions in the training phase, with 67,115 trials and 41,409 trials respectively for Ph and Sp. Part of the training data is reported in Rajalingham et al., 2021¹⁰.

Experimental conditions and visual stimulus

Experiment 1 contained 40 images and 2 illumination sites for stimulation trials (see Figure S1A for image set and inset in Figure 2B and Figure S3A for schematic of illumination locations) with illumination power of 3.6 mW and 5.4 mW respectively for Ph and Sp. Catch trials were included at a rate of 10% and 2%, and 10 and 13 sessions were performed with a total of 17,033 trials and 16,125 trials, and an overall performance of 84.6% and 84.9% correct (catch trials excluded), respectively for Ph and Sp. Ph only received catch trials to one site on the control array while Sp received catch trials to two sites, randomly interleaved. The performances for detecting cortical stimulation were statistically significant for stimulation trials (Ph: X² (1, N = 15320) = 7295.1, p < 0.001 and Sp: X² (1, N = 15794) = 7714.7, p < 0.001) but not for catch trials (Ph: X² (1, N = 10370) = 0.02, p = 0.879 and Sp: X² (1, N = 8428) = 1.7, p = 0.190).

Experiment 2 contained 5 images, 2 stimulation sites and 7 intensity conditions. The stimulation sites were the same as in experiment 1. The images used in this experiment were a subset of the images used in experiment 1, with the two highest and two lowest d’ image conditions selected (average of the two cortical locations). The fifth image was chosen by calculating which image had the greatest difference in d’ between cortical location conditions in experiment 1. Illumination power for “stimulation” trials ranged from 0.4 mW to 5.4 mW for both monkeys and 9 and 12 sessions were performed with 14,941 and 14,056 trials collected with overall performance of 79.6% and 74.7% correct, respectively for Ph and Sp. This experiment included no catch trials.

Experiment 3 contained 5 images and 4 image visibility conditions, plus one “no image” (uniform gray) condition (see Figure S1C for this image set). The “no image” condition occurred as often as any one degraded image condition, creating 21 total conditions. One cortical site was used for this experiment (Site 1 for both monkeys). We selected the top 5 highest d’ images from experiment 1 at that cortical site for this imageset and degraded their visibility by reducing their contrast, saturation, and spatial frequency to near gray. To do this, the mean luminance of each image was adjusted to match that of the gray display background. Then, saturation was reduced by multiplying each pixel’s chromaticity coordinates (a* and b*, CIELAB 1976) by a scale factor of ⅓, 1/9, and 1/27 for the decreasing visibility levels. Image contrast was reduced by the same operation on the L* dimension (lightness) of the CIELAB color profile, but first the mean L* of the distribution was subtracted from each pixel, then re-added after multiplication by the scale factor, ensuring that the mean luminance of the distribution was unchanged. Finally, the spatial frequency of the Lab-scaled images was reduced by convolving each image with a 2D gaussian smoothing kernel with standard deviations of 0.39, 0.78, and 1.56° for the different visibility level. To ensure that the filtered images blended evenly into the background, padding was added to the edges of the images but care was taken to ensure the presented size was the same 8° as experiment 1 and 2. Each visibility condition was a combination of one CIELAB scaling factor and one gaussian filter. Illumination power was 3.5 mW and 5.4 mW, and 1 session was performed for each monkey with 2030 and 3193 trials collected with overall performance of 77.8% and 90.9% correct trials, respectively for Ph and Sp.

Quantification and Statistical Analysis

Detection performance:

we used d’ as a bias-free measure of performance for detecting cortical stimulation³⁶ which is estimated by the following equation:

$$d^{\prime }=Z(H)-Z(F)$$

Where Z is Z-transform, H is the animal’s hit rate for detecting stimulation, and F is the false alarm rate representing the proportion of trials where no stimulation was applied but the animals reported the trial as stimulated.

Effect of image on detectability (Experiment 1):

first, we calculated a d’ for each image, indicating the detectability of cortical stimulation. This creates a ‘detection profile’ shown in Figure 2A and Figure S1. The 95% confidence intervals are estimated for each image by bootstrapping the data, resampling 10,000 times with replacement³⁷ and the violin plots represent the distribution of the bootstrapped data. To statistically test the effect of image on detectability of cortical stimulation, we ran a permutation test in which first we calculated the standard deviation of the d’s across images (observed standard deviation). Then, we generated the null distribution by randomly assigning the images to the trials with 10,000 replications and compared the observed standard deviation to the distribution of standard deviations generated from the null model. The permutation tests showed that the effect of images on detection of cortical stimulation is statistically significant (p < 0.001 for all the detection profiles). Moreover, we ran the same permutation tests after excluding the no image trials from the data and the result remained statistically significant (p < 0.001 for all the detection profiles).

Effect of cortical stimulation location on image detection profile, SVM classifier analysis (Experiment 1):

we trained a classifier on the detection profiles resulting from stimulation of two neighboring sites. First we made the detection profile for each stimulation site by using the hit rates of images. Then we used an SVM classifier to predict the stimulation site from the detection profiles. Any significant performance from this classifier would imply a significant difference between these two detection profiles. However, the significant performance of the classifiers could simply be due to higher overall hit rates for one of the sites. To avoid such a problem, we first centered the hit rates for each stimulation site by subtracting the average of hit rates across all images. Then we trained an SVM classifier with L1QP solver with 5-fold cross validation, by using four fifths of the trial to train the classifier and measuring the performance on the one fifth that has not been used for the training. The average performance of the SVM classifier was 81.0%, and 78.5%, respectively for monkey Ph and Sp, distinctly above the chance level (50%). Then we used a permutation test to estimate the statistical significance of these performances, by randomly assigning ‘stimulation sites’ to stimulation trials with 1,000 repeat without replacement. The p-values were 0.010 and 0.013 respectively for monkey Ph and Sp indicating statistical significance from chance level. These results imply a systematic difference between the detection profiles obtained from each stimulation site.

Effect of cortical stimulation location on image detection profile, correlation analysis (Experiment 1):

we used Pearson’s correlation to evaluate the similarity between the detection profiles derived from two neighboring stimulation sites. The correlations between hit rates for each image (detection profiles) at neighboring sites were statistically significant (Pearson’s r(39) = 0.91 and 0.82 respectively in Ph and SP; p < 0.001 for both subjects). To determine if there was a difference between the sites, we followed up these results with bootstrapped estimates of the correlations within each site and between them, resampled 10,000 times with replacement. The median correlation coefficients were 0.95 and 0.95 within the sites, and 0.86 between the sites for Ph (Figure 2B) and 0.89 and 0.95 within and 0.75 between for Sp (Figure S3A). These results show that the detection profiles are more correlated within the sites compared with between sites in both subjects. To test if this is a statistically significant difference, we generated a null distribution by randomly assigning sites to the stimulation trials with 10,000 replications; the results of this permutation test show that the observed correlation between the sites is smaller than the correlation between sites in the null distribution derived by randomly assigning sites to the trials (Ph: p =< 0.010, Sp: p < 0.001; Figure 2B and Figure S3A).

Effect of image on psychometric curves (Experiment 2):

in Figure 2C and Figure S3B-D, we plotted the animals’ psychometric curves derived from data collected in Experiment 2, which are performances (d’) as a function of intensity of stimulation (illumination power) for each image separately. This data can be fitted by an exponential function with an exponent between 0 and 1. In order to statistically test the effect of choice of image on the performance, first we fitted the average performances across all images on the following equation:

$$d_i^{\prime }=\alpha P_i^{\gamma }$$

Where d’_i is average performance across all image conditions for illumination power i. P_i is illumination power in mW. α and γ are the fit coefficients. The results showed very reliable goodness-of-fits (Ph: R² = 0.98, 0.98, α = 10.4, 7.4; γ = 0.52, 0.38; Sp: R² = 0.96, 0.96, α = 8.5, 7.1, γ = 0.61, 0.43, for each stimulation site for each monkey). Then we used the following regression formula to estimate the effect of each individual image on performance:

$$d_{i,j}^{\prime }={\beta }_j\left(\alpha P_i^{\gamma }\right)+{ϵ}_j$$

Where d’_i,j is the performance for power i and image j. P_i is the illumination power in mW. α and γ are constants calculated in the previous step for average performances across all images. β_j is the regression coefficient and ε_j is the error term for image j. In this way, β_j represents the effect of image j on performance. Here, the null hypothesis (H0) is that the choice of images did not have any systematic effect on the performance. In this case, all the βs should be equal to 1 and their variance should be 0. The alternative hypothesis (H1) is that performance as a function of illumination power changes with different rates depending on the choice of image, thus βs should be different from each other and their variance should be significantly larger than 0. Therefore, first we calculated standard deviation of βs for each stimulation site and each monkey (Ph: 0.27, 0.25; Sp: 0.23, 0.22). To test if these variances are statistically different from zero, we ran permutation tests in which the regression coefficients β_j was calculated after randomly assigning ‘image’ to the trials for 10,000 times with no replacement. The results from the permutation tests showed a strong level of significance (p ≤ 0.001 for all the tests in both monkeys and both stimulation sites).

Highlights:

• Monkeys learn to reliably detect optogenetic stimulation in inferotemporal cortex

• Behavioral detection of cortical stimulation depends on the image being viewed

• Cortical stimulation is the least perceivable in the absence of visual stimulus