The eyes reflect an internal cognitive state hidden in the population activity of cortical neurons

Abstract

Decades of research have shown that global brain states such as arousal can be indexed by measuring the properties of the eyes. The spiking responses of neurons throughout the brain have been associated with the pupil, small fixational saccades, and vigor in eye movements, but it has been difficult to isolate how internal states affect the eyes, and vice versa. While recording from populations of neurons in the visual and prefrontal cortex (PFC), we recently identified a latent dimension of neural activity called “slow drift,” which appears to reflect a shift in a global brain state. Here, we asked if slow drift is correlated with the action of the eyes in distinct behavioral tasks. We recorded from visual cortex (V4) while monkeys performed a change detection task, and PFC, while they performed a memory-guided saccade task. In both tasks, slow drift was associated with the size of the pupil and the microsaccade rate, two external indicators of the internal state of the animal. These results show that metrics related to the action of the eyes are associated with a dominant and task-independent mode of neural activity that can be accessed in the population activity of neurons across the cortex.

Introduction

In the fields of psychology and neuroscience, the eyes are often viewed as a window to the brain. Much has been learned about cognitive processes, and their development, from studying the action of the eyes (Hannula et al. 2010; Aslin 2012; König et al. 2016; Eckstein et al. 2017; Hessels and Hooge 2019; Ryan and Shen 2020). In addition, a large body of research has shown that properties related to the eyes can be used to index global brain states such as arousal, motivation, and cognitive effort (Di Stasi et al. 2013; Binda and Murray 2015; Mathôt and Van der Stigchel 2015; Wang and Munoz 2015; Mathôt 2018; Becket Ebitz and Moore 2019; Joshi and Gold 2020). This use of the term “states” might be taken to imply a dichotomy (e.g., arousal could be described as high or low), but here we refer to a more continuously graded process (perhaps due to fluctuating levels of neuromodulators and their effects). The action of the eyes can be considered broadly in two distinct contexts—the action of the pupil when the eyes are relatively stable, and the action of the eyes when they move, be it voluntarily, in response to novel objects in the visual field, or involuntarily during periods of steady fixation. In each context, technological advancements in infrared eye-tracking have allowed rich insight about a subject’s global brain state to be surmised in a rapid, accurate, and noninvasive manner (Kimmel et al. 2012).

When the eyes are relatively stable, the size of the pupil changes in response to the amount of light hitting the retina (Campbell and Gregory 1960). However, the pupil does not merely reflect accommodation. Even when luminance levels are held constant, changes in the size of the pupil have been found to occur due to a range of cognitive factors such as attention, working memory, and arousal (Binda and Murray 2015; Mathôt and Van der Stigchel 2015; Wang and Munoz 2015; Mathôt 2018; Becket Ebitz and Moore 2019; Joshi and Gold 2020). In the mammalian brain, arousal has been largely associated with the activity of the locus coeruleus (LC) (Aston-Jones and Cohen 2005; Sara 2009; van den Brink et al. 2019). This small structure in the pons contains a dense population of noradrenergic neurons and is the primary source of norepinephrine (NE) to the central nervous system. Recent neurophysiological work carried out in rodents and nonhuman primates has shown that pupil size is significantly associated with the spiking responses of LC neurons (Varazzani et al. 2015; Joshi et al. 2016; Reimer et al. 2016; Breton-Provencher and Sur 2019). Note that this is true for both raw pupil size and evoked (baseline-corrected) changes in pupil size that occur during task-relevant events (Binda and Murray 2015; Mathôt and Van der Stigchel 2015; Wang and Munoz 2015; Mathôt 2018; Becket Ebitz and Moore 2019). These two metrics are negatively correlated such that larger evoked dilations occur when the pupil is more constricted and vice versa (Gilzenrat et al. 2010; Murphy et al. 2011; Eldar et al. 2013; Joshi et al. 2016). Under conditions of heightened arousal, increases in raw pupil size are accompanied by decreases in evoked pupil size.

Voluntary saccades occur approximately three times per second to bring novel objects onto the high-resolution fovea (Kowler 2011). The characteristics of these saccades, such as the reaction time to initiate the saccade, and the velocity reached during the saccade, have similarly been used to index global changes in arousal (Di Stasi et al. 2013). For example, when arousal is increased by delivering a startling auditory stimulus prior to the execution of a saccade, reaction time decreases, and saccade velocity increases (Kristjánsson et al. 2004; Castellote et al. 2007; Deuter et al. 2013; DiGirolamo et al. 2016). Another metric that has been linked to global brain states, although to a lesser extent than reaction time and saccade velocity, is microsaccade rate. These small involuntary saccades are generated at a rate of 1–2 Hz through the activity of neurons in the superior colliculus (SC) (Rolfs 2009). Evidence suggests that microsaccade rate decreases with increased cognitive effort on a range of behavioral tasks (Valsecchi et al. 2007; Valsecchi and Turatto 2009; Siegenthaler et al. 2014; Gao et al. 2015). Taken together, these results suggest that the action of the eyes, be it when they are relatively stable or when they move, can be used to index global brain states.

A number of studies have related the activity of single neurons in many regions of the brain to raw pupil size (Reimer et al. 2014; Joshi et al. 2016), microsaccade rate (Bair and O’Keefe 1998; Leopold and Logothetis 1998; Martinez-Conde et al. 2000; Snodderly et al. 2001; Herrington et al. 2009; Chen et al. 2015; Lowet et al. 2018), reaction time (Hanes and Schall 1996; Cook and Maunsell 2002; Roitman and Shadlen 2002; Supèr and Lamme 2007; Khanna et al. 2019; Steinmetz and Moore 2010), and saccade velocity (Huang and Lisberger 2009; O’Leary and Lisberger 2012). However, if changes in these eye metrics are driven by a shift in an underlying internal state, then one might expect them all to be related to a common underlying neural activity pattern. Recent work in our laboratory used dimensionality reduction to identify a dominant mode of neural activity called slow drift that was: (1) present in visual and prefrontal cortex (PFC) of the macaque; (2) related to behavior on a change detection task; and (3) correlated with eye metrics such as raw pupil size (Cowley et al. 2020). However, this study consisted of data recorded on a single task (change detection), and primarily focused on behavioral metrics related to that task (such as false alarm rate). From that work, it remained unclear how general this slow drift is, and what signatures might be associated with it across tasks. This motivated us to ask if slow drift is present in different task contexts and if it is related to a constellation of eye metrics that are task-independent. We recorded the spiking responses of populations of neurons in V4 while monkeys performed a change detection task, and PFC, while the same subjects performed a memory-guided saccade task. On the change detection task, slow drift was significantly associated with raw pupil size, evoked pupil size, microsaccade rate, and saccade velocity, whereas on the memory-guided saccade task it was correlated with evoked pupil size and microsaccade rate. These results show that noninvasive metrics related to the eyes can be used to index a dimension of neural activity that is pervasive and task-independent. They suggest that slow drift represents a global brain state that manifests in the movement of the eyes and the size of the pupil.

Materials and Methods

Subjects

Two adult rhesus macaque monkeys (Macaca mulatta) were used in this study. Surgical procedures to chronically implant a titanium head post (to immobilize the subjects’ heads during experiments) and microelectrode arrays were conducted in aseptic conditions under isoflurane anesthesia, as described in detail by Smith and Sommer (2013). Opiate analgesics were used to minimize pain and discomfort during the perioperative period. Neural activity was recorded using 100-channel “Utah” arrays (Blackrock Microsystems) in V4 (Monkey Pe = right hemisphere; Monkey Wa = left hemisphere) and PFC (Monkey Pe = right hemisphere; Monkey Wa = left hemisphere) while the subjects performed the change detection task (Fig. 1A). Note that this is the same dataset used by Snyder et al. (2018) and Cowley et al. (2020). On the memory-guided saccade task (Fig. 1B), neural activity was only recorded in PFC (Monkey Pe = left hemisphere; Monkey Wa = left hemisphere). Note that the data presented here are a superset of the data presented in Khanna et al. (2019). The only difference between the memory-guided saccade data presented here and that previous study is that here we also analyzed neural activity from additional sessions in Monkey Pe after a new array was implanted in left PFC. The sessions were also longer, and particularly well suited to analyze slow fluctuations in neural activity. The arrays comprised a 10 x 10 grid of silicon microelectrodes (1 mm in length) spaced 400 μm apart. Experimental procedures were approved by the Institutional Animal Care and Use Committee of the University of Pittsburgh and were performed in accordance with the United States National Research Council’s Guide for the Care and Use of Laboratory Animals.

Figure 1.

Experimental methods. (A) Change detection task. After an initial fixation period, a sequence of orientated Gabor stimuli was presented. The subject's task was to detect an orientation change in one of the stimuli and make a saccade to the changed stimulus. We recorded neural activity from V4 while the animals performed the change detection task using 100-channel ``Utah'' arrays (inset). (B) Memory-guided saccade task. After an initial fixation period, a target stimulus was presented at 1 of 40 locations followed by a delay period. The central point was then extinguished prompting the subjects to make a saccade to the remembered target location. Neural activity was recorded in PFC while the animals performed the memory-guided saccade task using ``Utah'' arrays (inset). (C) Measuring eye metrics on the change detection task. Raw pupil size was recorded during stimulus periods (indicated by gray shaded regions), whereas microsaccade rate was measured during periods of steady fixation (except for the initial fixation period, see Methods). Microsaccades (indicated by crosses in 1D space/emboldened lines in 2D space) were detected using a velocity-based algorithm (Engbert and Kliegl 2003). Reaction time was the time taken to make a saccade from when the orientation change occurred (indicated by the dashed gray line). Saccade velocity was the peak velocity of the saccade. (D) Measuring eye metrics on the memory-guided saccade task. Raw pupil size was recorded during the presentation of the target stimulus (indicated by the gray shaded region), whereas microsaccade rate was measured during the delay period. Reaction time was the time taken to make a saccade to the remembered target location from when the central point was extinguished (indicated by the dashed gray line). Saccade velocity was the peak velocity of the saccade. RT = reaction time.

Microelectrode Array Recordings

Signals from each microelectrode in the array were amplified and band-pass filtered (0.3–7500 Hz) by a Grapevine system (Ripple). Waveform segments crossing a threshold (set as a multiple of the root mean square noise on each channel) were digitized (30 kHz) and stored for offline analysis and sorting. First, waveforms were automatically sorted using a competitive mixture decomposition method (Shoham et al. 2003). They were then manually refined using custom time amplitude window discrimination software (Kelly et al. 2007), code available at https://github.com/smithlabvision/spikesort, which takes into account metrics including (but not limited to) waveform shape and the distribution of interspike intervals. A mixture of single and multiunit activity was recorded, but we refer here to all units as “neurons.” On the change detection task, the mean number of V4 neurons across sessions was 41 (SD = 10) for Monkey Pe and 21 (SD = 10) for Monkey Wa, whereas the mean number of PFC neurons across sessions was 65 (SD = 13) for Monkey Pe and 60 (SD = 12) for Monkey Wa. On the memory-guided saccade task, the mean number of PFC neurons across sessions was 38 (SD = 11) for Monkey Pe and 19 (SD = 19) for Monkey Wa.

Visual Stimuli

Visual stimuli were generated using a combination of custom software written in MATLAB (The MathWorks) and Psychophysics Toolbox extensions (Brainard 1997; Pelli 1997; Kleiner et al. 2007). They were displayed on a CRT monitor (resolution = 1024 × 768 pixels; refresh rate = 100 Hz), which was viewed at a distance of 36 cm and gamma-corrected to linearize the relationship between input voltage and output luminance using a photometer and look-up-tables.

Behavioral Tasks

Orientation-Change Detection Task

Subjects fixated a central point (diameter = 0.6°) on the monitor to initiate a trial (Fig. 1A). Each trial comprised a sequence of stimulus periods (400 ms) separated by fixation periods (duration drawn at random from a uniform distribution spanning 300–500 ms). The 400-ms stimulus periods comprised pairs of drifting full-contrast Gabor stimuli. One stimulus was presented in the aggregate receptive field (RF) of the recorded V4 neurons, whereas the other stimulus was presented in the mirror-symmetric location in the opposite hemifield. Although the spatial (Monkey Pe = 0.85 cycles/°; Monkey Wa = 0.85 cycles/°) and temporal frequencies (Monkey Pe = 8 cycles/s; Monkey Wa = 7 cycles/s) of the stimuli were not optimized for each individual V4 neuron they did evoke a strong response from the population. The orientation of the stimulus in the aggregate RF was chosen at random to be 45 or 135°, and the stimulus in the opposite hemifield was assigned the other orientation. There was a fixed probability (Monkey Pe = 30%; Monkey Wa = 40%) that one of the Gabors would change orientation by ±1, ±3, ±6, or ± 15° on each stimulus presentation. The sequence continued until the subject (1) made a saccade to the changed stimulus within 700 ms (“hit”); (2) made a saccade to an unchanged stimulus (“false alarm”); or (3) remained fixating for more than 700 ms after a change occurred (“miss”). If the subject correctly detected an orientation change, they received a liquid reward. In contrast, a time-out occurred if the subject made a saccade to an unchanged stimulus delaying the beginning of the next trial by 1 s. It is important to note that the effects of spatial attention were also investigated (although not analyzed in this study) by cueing blocks of trials such that the orientation change was 90% more likely to occur within the aggregate V4 RF than the opposite hemifield.

Memory-Guided Saccade Task

Subjects fixated a central point (diameter = 0.6°) on the monitor to initiate a trial (Fig. 1B). After fixating within a circular window (diameter = 2.4° and 1.8° for Monkey Pe and Monkey Wa, respectively) for 200 ms, a target stimulus (diameter = 0.8°) was presented. For Monkey Pe, the duration of the target stimulus was 400 ms (except for 1 session in which it was presented for 50 ms), whereas for Monkey Wa the duration of the target stimulus was 50 ms. The target stimulus appeared at 1 of 8 angles separated by 45°, and 1 of 5 eccentricities, yielding 40 conditions in total. After the target stimulus had been presented, subjects were required to maintain fixation for a delay period. For Monkey Pe, the duration of the delay period was either (1) drawn at random from a distribution spanning 1200–3750 ms; or (2) fixed at 1400 (1 session) or 2000 ms (11 sessions). For Monkey Wa, the duration of the delay period was 500 ms. If steady fixation was maintained throughout the delay period, the central point was extinguished prompting the subjects to make a saccade to the remembered target location. The subjects had 500 ms to initiate a saccade. Once the saccade had been initiated, they had a further 200 ms to reach the remembered target location. To receive a liquid reward, the subject's gaze had to be maintained within a circular window centered on the target location (diameter = 4 and 2.7° for Monkey Pe and Monkey Wa, respectively) for 150 ms. In a subset of sessions, the target was briefly reilluminated, after the fixation point was extinguished and the saccade had been initiated, to aid in saccade completion.

Eye Tracking

Eye position and pupil diameter were recorded monocularly at a rate of 1000 Hz using an infrared eye tracker (EyeLink 1000, SR Research).

Microsaccade Detection

Microsaccades were defined as eye movements that exceeded a velocity threshold of six times the standard deviation of the median velocity for at least 6 ms (Engbert and Kliegl 2003). They were required to be separated in time by at least 100 ms. In addition, we removed microsaccades with an amplitude greater than 1° and a velocity greater than 100°/s. To assess the validity of our microsaccade detection algorithm, correlations (Pearson product–moment correlation coefficient) were computed between: (1) the amplitude and peak velocity of microsaccades and correct saccades made to the changed stimulus (change detection task) or remembered target location (memory-guided saccade task); and (2) the amplitude and peak velocity of microsaccades only (see Supplementary Fig. S5). The mean correlation across sessions between saccade amplitude and peak saccade velocity for all eye movements (microsaccades and correct saccades combined) was 0.98 (SD = 0.02) for the change detection task (see Supplementary Fig. S5A) and 0.98 (SD = 0.01) for the memory-guided saccade task (see Supplementary Fig. S5B). On the change detection task, the mean correlation between microsaccade amplitude and peak microsaccade velocity was 0.84 (SD = 0.06). On the memory-guided saccade task, it was also 0.84 (SD = 0.03). These findings validate our detection algorithm as they show that detected microsaccades fell on the main sequence (Zuber et al. 1965).

Eye Metrics

Change Detection Task

Mean raw pupil size was measured during stimulus periods, whereas microsaccade rate was measured during fixation periods (Fig. 1C). We did not include the first fixation period when measuring microsaccade rate in the change detection task. As can be seen in Figure 1C, there was an increase in eye position variability during this period resulting from fixation having been established a short time earlier (300–500 ms). Such variability was not present in subsequent fixation periods. Reaction time and saccade velocity were measured on trials in which the subjects were rewarded for correctly detecting an orientation change. Reaction time was defined as the time from when the change occurred to the time at which the saccade exceeded a velocity threshold of 150°/s. Saccade velocity was the peak velocity of the saccade to the changed stimulus. To isolate slow changes in the eye metrics over time, the data for each session were binned using a 30-min sliding window stepped every 6 min (Figure 2A and B). If necessary, sessions were truncated to ensure that each bin had a full duration of 30 min.

Figure 2.

Isolating slow fluctuations in the eye metrics. (A) Change detection task. Each scatter point corresponds to raw (unbinned) measurements except plots showing slow fluctuations in microsaccade rate over time. Because microsaccades can only occur an integer number of times (and often that number is zero), for visualization purposes each scatter point represents the mean rate during a 5-min sliding window stepped every 1 min. In all plots (including microsaccade rate plots), solid lines correspond to raw measurements binned using a 30-min sliding window stepped every 6 min (Cowley et al. 2020). (B) Same as (A) but for the memory-guided saccade task. Note that data points with a SD two times greater or less than the mean are not shown in (A) and (B). This was done for visualization purposes only, that is, so that slow fluctuations in the data can be more easily identified by eye. In addition, bins were computed in a manner such that the last bin always had a full duration of 30 min. This explains why the binned data (solid lines) in (A) and (B) are truncated. (C) Example session from Monkey Pe on the change detection task. (D) Example session from the same subject on the memory-guided saccade task. In (C) and (D), the binned data were z-scored for visualization purposes only, that is, so that eye metrics with different units could be shown on the same plot.

Memory-Guided Saccade Task

Eye metrics were only measured on trials in which the subjects received a liquid reward for making a correct saccade to the remembered target location. Mean raw pupil size was measured during the presentation of the target stimulus, whereas microsaccade rate was measured during the delay period (Fig. 1D). Reaction time was defined as the time from when the fixation point was extinguished to the time at which the saccade reached a threshold of 150°/s. Saccade velocity was the peak velocity of the saccade to the remembered target location. As in the change detection task, the data for each session were binned using the same 30-min sliding window stepped every 6 min (Fig. 2B).

Calculating Slow Drift

Orientation-Change Detection Task

The spiking responses of populations of neurons in V4 were measured during a 400 ms period that began 50 ms after stimulus presentation. Research has shown that neurons in V4 are tuned for stimulus orientation (Desimone and Schein 1987). To prevent the principal component analysis (PCA) identifying components related to stimulus tuning, residual spike counts were computed by subtracting the mean response for a given orientation (45 or 135°) across the entire session from individual responses to that orientation. To isolate slow changes in neural activity over time, residual spike counts for each V4 neuron were binned using a 30-min sliding window stepped every 6 min (Cowley et al. 2020). PCA was then performed to reduce the high-dimensional residual data to a smaller number of variables (Cunningham and Yu 2014). Slow drift in V4 was estimated by projecting the binned residual spike counts for each neuron along the first principal component (Cowley et al. 2020). As described above, the spiking responses of neurons in PFC were simultaneously recorded on the change detection task. When PFC slow drift was calculated using the method described above, we found it to be significantly associated with V4 slow drift (median r = 0.95, P < 0.001), consistent with previous results (Cowley et al. 2020). On the change detection task, we investigated the relationships between the eye metrics and V4 slow drift. However, a very similar pattern of results was found when slow drift was calculated using simultaneously recorded PFC data (see Supplementary Fig. S6).

Memory-Guided Saccade Task

The spiking responses of populations of neurons in PFC were measured during the delay period. Neurons in PFC exhibit a range of properties including selectivity for different spatial locations (Funahashi et al. 1989, 1991; Khanna et al. 2020). As in the change detection task, we wanted to ensure that signals related to tuning preferences were not present in the slow drift. To control for the fact that some neurons might have preferred one target location over another, residual spike counts were calculated. We subtracted the mean response to a given target location across the entire session from individual responses to that location. To isolate slow changes in neural activity over time, residual spike counts for each PFC neuron were binned using a 30-min sliding window stepped every 6 min. PCA was then performed, and slow drift was estimated by projecting the binned residual spike counts for each neuron along the first principal component.

Controlling for Neural Recording Instabilities

To rule out the possibility that slow drift arose due to recording instabilities (e.g., the distance between the neuron and the microelectrodes changing slowly over time), we only included neurons with stable waveform shapes throughout a session. This was quantified by calculating percent waveform variance for each neuron (see Supplementary Fig. S7). First, the session was divided into 10 nonoverlapping time bins. A residual waveform was then computed for each time bin by subtracting the mean waveform across time bins. The variance of each residual waveform was divided by the variance of the mean waveform across time bins yielding 10 values (one for each time bin). Percent waveform variance was defined as the maximum value across time bins. Neurons with a percent waveform variance greater than 10% were deemed as having unstable waveform shapes throughout a session. They were excluded from all analyses, consistent with previous research (Cowley et al. 2020).

Aligning Slow Drift Across Sessions

As described above, slow drift was calculated by projecting binned residual spike counts along the first principal component (Cowley et al. 2020). The weights in PCA can be positive or negative (Jolliffe and Cadima 2016), which meant the sign of the correlation between slow drift and a given eye metric was arbitrary. Preserving the sign of the correlations was particularly important in this study because we were interested in whether slow drift was associated with a pattern that is indicative of changes in the subjects’ arousal levels over time, that is, increased raw pupil size and saccade velocity, and decreased microsaccade rate and reaction time. Thus, we had to devise a method to align slow drift across sessions. Cowley et al. (2020) devised an alignment method based on responses to the stimuli on the change detection task, but we wanted to establish a routine that was independent of the task performed (and thus suitable for areas like PFC, where the stimulus tuning can be quite broad and flipping based on that tuning can be unreliable). To achieve this goal, the sign of the slow drift was flipped if the majority of neurons had negative weights (Hennig et al. 2021). Forcing the majority of neurons to have positive weights established a common reference frame, in which an increase in the value of slow drift was associated with higher firing rates among the majority of neurons.

Results

To determine if observation of the eyes could provide insight into the internal state associated with slow drift, we recorded the spiking responses of populations of neurons in two macaque monkeys using 100-channel “Utah” arrays. We recorded from neurons in (1) V4 while the subjects performed an orientation-change detection task (Fig. 1A); and (2) PFC while they performed a memory-guided saccade task (Fig. 1B). Behavioral data for both subjects on the change detection task (Snyder et al. 2018) and the memory-guided saccade task (Khanna et al. 2019) have been published before in reports analyzing distinct aspects of the experiments described in this study. Here, the primary goal was to determine whether the neural population activity was related to eye metrics in a predictable manner across tasks. We analyzed data recorded in V4 on the change detection task because neural activity in midlevel visual areas has long been associated with performance on perceptual decision-making tasks (Shadlen et al. 1996). Similarly, neural activity in PFC is correlated with performance on memory-guided saccade tasks (Funahashi et al. 1989). Four eye metrics were recorded during each session: pupil size, microsaccade rate, reaction time, and saccade velocity. These metrics were chosen because they have been used extensively to index global changes in brain state (see Introduction), and can be measured easily and accurately with an infrared eye tracker (Kimmel et al. 2012). We made each of these four measurements on every trial of both behavioral tasks (Fig. 1).

Correlations Between the Eye Metrics Over Time

First, we investigated if the different measures of the eyes were themselves correlated over time during performance of the behavioral task. Taken together, these results suggest that the action of the eyes, both when they are relatively stable and when they move, can be used to index global brain states. As described above, increases in arousal are typically accompanied by increases in raw pupil size and saccade velocity, and concomitant decreases in microsaccade rate and reaction time. Given that arousal is a domain-general phenomenon, one might expect a similar pattern to emerge on different behavioral tasks. To explore whether or not this was the case, we binned our eye data using a 30-min sliding window stepped every 6 min (Fig. 2A and B). The width of the window, and the step size, were chosen to isolate slow changes over time based on previous research. They were the same as those used by Cowley et al. (2020), which meant direct comparisons could be made across studies. An example session from the same subject on the change detection task and the memory-guided saccade task is shown in Figure 2C and D, respectively. In the example sessions shown across both tasks, a characteristic pattern was observed that was indicative of slow changes in the subject’s arousal level over time. Specifically, a large raw pupil size (measured during stimulus periods on the change detection task and target presentations on the memory-guided saccade task) was associated with high saccade velocity, shorter reaction times, and a lower rate of microsaccades.

The example sessions in Figure 2 exhibit trends that are consistent with changes in the subject’s arousal level over time. However, it is difficult to determine if slowly changing variables are correlated over the course of a single session. Standard correlation analyses assume that all samples are independent, but smoothed variables that fluctuate slowly over time can violate this assumption leading to “nonsense correlations” between variables that are unrelated (Harris 2020). One way to overcome this problem is to record data from multiple sessions, compute a correlation coefficient for each session, and then perform a statistical test to investigate if the distribution of coefficients is centered on zero. Hence, we computed correlations (Pearson product–moment correlation coefficient) between all combinations of the eye metrics for each session. Results showed that null distributions (generated by computing correlations between sessions recorded on different days) were centered on zero (see Supplementary Figs2 and 3). Therefore, Wilcoxon signed rank tests were used to test the null hypothesis that the median correlation across sessions was equal to zero.

Before computing correlations between all combinations of the eye metrics, the data for Monkey Pe and Monkey Wa were combined to enhance statistical power (change detection task: Monkey Pe = 20 sessions, Monkey Wa = 16 sessions; memory-guided saccade task: Monkey Pe = 31 sessions, Monkey Wa = 22 sessions). This was justified because similar across session trends were observed for both subjects, that is, the overall pattern of results was the same (see Supplementary Fig. S1). Consistent with the trends observed in several individual sessions, we found significant interactions among the eye metrics (Fig. 3). Raw pupil size was significantly and negatively correlated with microsaccade rate (median r = −0.45; P = 0.001) and reaction time (median r = −0.17; P = 0.033) on the change detection task (Fig. 3A and Supplementary Fig. S2) and memory-guided saccade task (Fig. 3B and Supplementary Fig. S3, median r = −0.43; P = 0.001 for microsaccade rate and median r = −0.32; P < 0.001 for reaction time). In the change detection task, we did not observe significant across session trends in the correlation between raw pupil size and saccade velocity (r = 0.08, P = 0.307). However, there was a significant positive correlation between these two metrics in the memory-guided saccade task (r = 0.14, P = 0.004). In both tasks, saccade velocity was negatively correlated with reaction time (change detection task: median r = −0.67, P < 0.001; memory-guided saccade task: median r = −0.70, P < 0.001). Furthermore, on the change detection task, reaction time was significantly correlated with hit rate (median r = −0.42, P = 0.011) and false alarm rate (median r = −0.30, P < 0.001). Reaction times were shorter when hit rate/false alarm rate was high. Taken together, these results suggest that changes in the subject’s arousal levels were accompanied by changes in the action of the eyes. This was true of raw pupil size, microsaccade rate, reaction time, and to a lesser extent, saccade velocity, and suggests that these eye metrics may be used to index global changes in brain state. This motivated us to ask next whether there were neural correlates of these behavioral signatures.

Figure 3.

Correlations between the eye metrics over time. (A) Change detection task. Correlation matrix showing median r values across sessions between the four eye metrics. (B) Same as (A) but for the memory-guided saccade task. In (A) and (B), P-values were computed using two-sided Wilcoxon signed rank tests. Note that histograms showing distributions of r values are shown in Supplementary Figures 2 and 3. P < 0.05^*, P < 0.01^**, P < 0.001^***.

Correlation Between the Eye Metrics and Slow Drift Over Time

Previously we reported a slow fluctuation in neural activity in V4 and PFC that we termed “slow drift” (Cowley et al. 2020). We found that this neural signature was related to the subject’s tendency to make impulsive decisions in a change detection task (false alarms), ignoring sensory evidence. Here, we wanted to investigate if the constellation of eye metrics we observed were associated with the neural signature of internal state that we termed “slow drift.” To calculate slow drift, residual spike counts were computed by subtracting the mean response for a given orientation (change detection task) or target location (memory-guided saccade task) across the entire session from individual responses. This was an important first step, as it ensured that signals related to stimulus tuning were not present in the slow drift. Residual spike counts in V4 (change detection task) and PFC (memory-guided saccade task) were then binned using the same 30-min sliding window that had been used to bin the eye metric data (Fig. 4A and B, see Methods). We then applied PCA to the data and estimated slow drift by projecting the binned residual spike counts along the first principal component (i.e., the loading vector that explained the most variance in the data). Because the sign of the weights in PCA is arbitrary (Jolliffe and Cadima 2016), the correlation between slow drift and a given eye metric in any session was equally likely to be positive or negative. This was problematic since we were interested in whether slow drift was associated with a characteristic pattern that is indicative of changes in the subject’s arousal levels over time, that is, increased raw pupil size and saccade velocity, and decreased microsaccade rate and reaction time. For simplicity, we flipped the sign of the slow drift such that the majority of neurons had positive weights (Hennig et al. 2021). This established a common reference frame in which an increase in the value of the drift was associated with higher firing rates among the majority of neurons (see Methods).

Figure 4.

Calculating slow drift. (A) Change detection task. Three example neurons from a single session (Monkey Pe). Each point represents the mean residual spike count during a 400 ms stimulus period. The data were then binned using a 30-min sliding window stepped every 6 min (solid line), so that direct comparisons could be made with the eye metrics. PCA was used to reduce the dimensionality of the data and slow drift was calculated by projecting binned residual spike counts along the first principal component. (B) Same as (A) but for the memory-guided saccade task (same subject). (C) Example session from Monkey Pe on the change detection task. Each metric has been z-scored for illustration purposes. (D) Example session from the same subject on the memory-guided saccade task. In (C) and (D), the binned data were z-scored for visualization purposes only, that is so that eye metrics with different units could be shown on the same plot.

We computed the slow drift of the neuronal population in each session using this method, and then compared it with the four eye measures. An example session is shown in Figure 4C and D for the same subject on the change detection task and the memory-guided saccade task, respectively (same sessions as in Fig. 2). On both tasks, we found a characteristic pattern in which slow drift was positively associated with pupil diameter and saccade velocity, and negatively associated with microsaccade rate and reaction time.

Next, we explored if a similar pattern was found across sessions. We computed correlations (Pearson product–moment correlation coefficient) between slow drift and the eye metrics. Wilcoxon signed rank tests were then used to test the null hypothesis that the median correlation coefficient across sessions was equal to zero.

Consistent with the pattern of results observed in several individual sessions, we found that slow drift in V4 on the change detection task (Fig. 5A) was positively correlated with pupil diameter (median r = 0.56, P < 0.001) and saccade velocity (median r = 0.35, P = 0.038), and negatively correlated with microsaccade rate (median r = −0.47, P = 0.010). Interestingly, no significant correlation was found between slow drift and microsaccade rate when the latter was recorded during stimulus periods as opposed to prestimulus periods (median r = −0.05, P = 0.520). This finding might reflect biphasic changes in microsaccade rate that occur following the presentation of a visual stimulus. Microsaccade rate decreases shortly after stimulus presentation and then increases later (Engbert and Kliegl 2003; Rolfs et al. 2008; Hafed and Ignashchenkova 2013). Indeed, we found that the mean microsaccade rate was significantly higher during stimulus periods (M = 1.52, SD = 0.45) than pretimulus periods (M = 1.17, SD = 0.68) (t(70) = 2.52, P = 0.014). Finally, no significant correlation was found between slow drift and reaction time in the data pooled across subjects (median r = −0.22; P = 0.265). A similar pattern of results was found when a range of different bin widths/step sizes were used (see Supplementary Fig. S4) and when a partial correlation analysis was performed on the five variables of interest: raw pupil size, microsaccade rate, reaction time, saccade velocity, and slow drift. Results of this partial correlation analysis showed that slow drift was positively correlated with raw pupil size (median partial r = 0.33; P = 0.03) and negatively correlated with microsaccade rate (median partial r = −0.24; P = 0.003). However, no significant correlation was found between slow drift and reaction time (median partial r = 0.06; P = 0.767) or saccade velocity (median partial r = 0.17; P = 0.498). Hence, the only difference between the results of the correlation analysis and the partial correlation analysis was that, in the latter, the correlation between slow drift and saccade velocity was not statistically significant. Together, these analyses demonstrate that raw pupil size and microsaccade rate make unique contributions to the total variation in slow drift on the change detection task.

Figure 5.

Correlation between the eye metrics and slow drift over time. (A) Change detection task. Histograms showing distributions of r values across sessions. (B) Same as (A) but for the memory-guided saccade task. In (A) and (B), median r values across sessions are indicated by dashed lines. We computed P-values using two-sided Wilcoxon signed rank tests. P < 0.05^*, P < 0.01^**, P < 0.001^***.

On the memory-guided saccade task (Fig. 5B), no significant correlation was found between PFC slow drift and raw pupil size (median r = −0.16, P = 0.753), reaction time (median r = 0.05, P = 0.774), or saccade velocity (median r = −0.16, P = 0.198). However, consistent with the pattern of results for the change detection task, microsaccade rate was significantly negatively correlated with PFC slow drift on the memory-guided saccade task (median r = −0.70, P < 0.001). As in the change detection task, this was true for a range of different bin widths/step sizes (see Supplementary Fig. S4) and when a partial correlation analysis was performed. Raw pupil size (median partial r = −0.16; P = 0.252), reaction time (median partial r = −0.01; P = 0.835) and saccade velocity (median partial r = 0.02; P = 0.449) were not significantly correlated with slow drift, but there was a significant negative correlation between slow drift and microsaccade rate (median partial r = −0.48; P < 0.001). These findings show that microsaccade rate can be used to index slow drift across tasks with differing cognitive demands.

Correlation Between Evoked Pupil Size and Slow Drift Over Time

In the analyses described above, raw pupil size was defined as the mean raw value recorded during stimulus periods (change detection task) and the presentation of the target stimulus (memory-guided saccade task). However, it is also possible to investigate the relationship between slow drift and evoked (baseline-corrected) pupil size, a common metric in many studies of the pupil (Binda et al. 2013; Mathôt et al. 2013; Bombeke et al. 2016; Ebitz and Moore 2017; Wang and Munoz 2018; Van Kempen et al. 2019; Zokaei et al. 2019). To do so, we recorded evoked pupil responses following stimulus periods on both tasks. The data were then binned using the same 30-min sliding window stepped every 6 min and baseline corrected. Previous work has shown that raw pupil size is negatively correlated with evoked pupil size, a finding that might reflect physical limits imposed by iris musculature (Gilzenrat et al. 2010; Murphy et al. 2011; Eldar et al. 2013; Joshi et al. 2016). Hence, one would expect slow drift to be negatively correlated with the mean amplitude of the evoked pupil response on both tasks. Note that evoked pupil responses were only recorded for one Monkey (Pe) on the memory-guided saccade task. This was because in the shorter target stimulus and delay period duration used for Monkey Wa, the effects of eye position/luminance changes resulting from the establishment of fixation were still present in the pupil trace.

Results showed that the mean amplitude of the evoked pupil response (gray shaded regions in Fig. 6A and B) changed slowly over time on both tasks. As in our previous analysis, a combination of correlations and Wilcoxon signed rank tests were used to investigate if slow drift was associated with the mean amplitude of the evoked pupil response. On both tasks, slow drift was negatively correlated with the mean amplitude of the evoked pupil response (change detection task: r = −0.59, P = 0.008; memory-guided saccade task: r = −0.55, P < 0.001). These results support previous research showing that larger evoked dilations occur when the pupil is more constricted and vice versa (Gilzenrat et al. 2010; Murphy et al. 2011; Eldar et al. 2013; Joshi et al. 2016). They suggest that evoked pupil size, in addition to microsaccade rate, can be used to index global fluctuations of brain activity across different cognitive tasks.

Figure 6.

Correlation between evoked pupil size and slow drift over time. (A) Change detection task. An example session in which the mean amplitude of the evoked pupil response (gray shaded region) changed slowly over time. (B) Same as (A) but for the memory-guided saccade task. In (A) and (B), the data were binned using a 30-min sliding window stepped every 6 min and baseline corrected by subtracting the mean value prior to the onset of the evoked response (white regions). Each bin is represented by a gray shaded line. (C) Change detection task. Histogram showing the distribution of r values computed to investigate the relationship between the mean amplitude of the evoked response and slow drift. (D) Same as (C) but for the memory-guided saccade task. In (C) and (D), median r values across sessions are indicated by dashed lines. We computed P-values using two-sided Wilcoxon signed rank tests. P < 0.05^*, P < 0.01^**, P < 0.001^***.

Discussion

In this study, we investigated if the size of the pupil and the movement of the eyes could be taken as an external signature of an internal brain state, a low-dimensional neural activity pattern called slow drift (Cowley et al. 2020). There is strong evidence that internal brain states such as slow drift can be measured in the population spiking activity of neurons, and that measurements of the eyes can provide important context into the behavior of subjects on perceptual and decision-making tasks. Hence, we were keen to determine whether we could directly link a neural measure of internal brain state acquired from the spiking activity of a population of neurons with external features of behavior. On two types of perceptual tasks, we found that slow drift was significantly correlated with two metrics that can be accurately measured using readily available infrared eye tracking technology: microsaccade rate and evoked pupil size. Our results show that the action of the eyes, be it when they are relatively stable, as is the case for evoked pupil size, or when small movements are made, is associated with a latent dimension of neural activity that is pervasive and task-independent.

As described above, decades of research have shown that eye metrics are related to task performance in a variety of contexts (Di Stasi et al. 2013; Binda and Murray 2015; Mathôt and Van der Stigchel 2015; Wang and Munoz 2015; Mathôt 2018; Becket Ebitz and Moore 2019; Joshi and Gold 2020). Heightened levels of arousal have been associated with increased raw pupil size and saccade velocity as well as decreased reaction time and microsaccade rate (Castellote et al. 2007; Valsecchi et al. 2007; Valsecchi and Turatto 2009; Deuter et al. 2013; Siegenthaler et al. 2014; Gao et al. 2015; DiGirolamo et al. 2016; Joshi et al. 2016). Given that arousal is a global phenomenon, one might expect a common pattern to emerge between these different eye metrics across multiple behavioral tasks. In the present study, we explored if this was the case by investigating the relationships between eye metrics on tasks designed to probe the mechanisms underlying perceptual decision-making (change detection task) and working memory (memory-guided saccade task). Results showed that raw pupil size was negatively correlated with microsaccade rate and reaction time on both tasks. Reaction time was also negatively correlated with saccade velocity in a task-independent manner, and there was a significant positive correlation between raw pupil size and saccade velocity on the memory-guided saccade task. Taken together, these results suggest that each subject’s arousal level was changing over time irrespective of the task performed. These findings support the view that measuring properties related to the eyes can provide a noninvasive index of global brain states or fluctuations (Di Stasi et al. 2013; Binda and Murray 2015; Wang and Munoz 2015; Mathôt 2018; Becket Ebitz and Moore 2019; Joshi and Gold 2020). This motivated us to ask if they are also associated with slow drift.

Most studies that have explored the relationship between eye metrics and neural activity have used single-neuron (spike count, Fano factor) and pairwise statistics (spike count correlation, or r_sc). However, numerous recent studies have shown that rich insight about cognitive processes (e.g., learning, decision-making, working memory, time perception) can be gained from analysis of the simultaneous activity of populations of neurons (Harvey et al. 2012; Mante et al. 2013; Sadtler et al. 2014; Murray et al. 2017; Remington et al. 2018; Musall et al. 2019). In addition, recent work has shown that a low-dimensional representation of neural activity in the mouse can be used to index global changes in brain state. Stringer et al. (2019) applied PCA to data recorded from more than 10,000 neurons and found that fluctuations in the first principal component were significantly associated with whisking, raw pupil size, and running speed. In this study, we investigated if slow drift, a dominant mode of neural activity in macaque cortex is associated with the action of the eyes. On both tasks, we found that it was correlated with two eye metrics: microsaccade rate and evoked pupil size. Our results, coupled with those of Stringer et al. (2019), suggest that much can be learned about global brain states, as well as cognitive processes, when high-dimensional population activity is reduced to a low-dimensional subspace (Cunningham and Yu 2014). A key question for future research is whether latent dimensions of neural activity in the cortex are associated with activity in subcortical brain regions? One might expect this to be the case given that slow drift was significantly correlated with evoked pupil size on both tasks.

Evidence suggests that raw and task-evoked pupil size is associated with activity in the LC, a subcortical structure that regulates arousal by releasing NE in a diverse manner throughout much of the brain (Aston-Jones and Cohen 2005; Sara 2009; van den Brink et al. 2019). That slow drift can be used to index global brain states suggests that it might be associated with LC activity. Although the LC is limited in size (~3 mm rostrocaudally) and buried deep within the brainstem (German and Bowden 1975; Sharma et al. 2010), several studies have successfully recorded from single neurons in LC of the macaque (Aston-Jones et al. 1994; Clayton et al. 2004; Kalwani et al. 2014; Varazzani et al. 2015; Joshi et al. 2016). In addition, LC can be activated using optogenetics (Carter et al. 2010; Li et al. 2016; Quinlan et al. 2019; Hayat et al. 2020), electrical microstimulation (Joshi et al. 2016; Reimer et al. 2016; Liu et al. 2017), and pharmacological manipulations (Vazey and Aston-Jones 2014; Liu et al. 2017). Thus, it should be possible to alter the course of slow drift in the cortex using some, if not all, of these methods. In the present study, there was no experimental manipulation of arousal, and thus any changes we observed were the type of naturally occurring fluctuation that would occur in most existing data in a variety of task contexts. If LC is the source of some of these fluctuations in arousal, then activating the LC, directly or indirectly, may lead to task-independent changes in microsaccade rate, reaction time, and saccade velocity in addition to its well-known association with the pupil.

Evidence suggests that microsaccades, reaction time, and saccade velocity are associated with neural activity in the SC (Gandhi and Katnani 2011). This layered structure, located at the roof of the brain stem, plays a critical role in transforming sensory information into eye-movement commands. Population recordings have been successfully performed in the SC using linear probes (Massot et al. 2019), and it thus might be possible to identify dominant patterns of neural activity in SC using dimensionality reduction (Cunningham and Yu 2014). Our results predict that slow drift in the cortex should be significantly correlated with slow drift in the SC. However, this might depend upon the mixture of SC neurons in the recorded population. We previously suggested that slow drift must be removed at some stage before motor commands are issued to prevent unwanted eye movements (Cowley et al. 2020). Thus, one might not expect a correlation between slow drift in the cortex and slow drift in deep-layer SC neurons that fire vigorously prior to the execution of a saccade, and relay motor commands to downstream nuclei innervating the oculomotor muscles (Sparks and Hartwich-Young 1989). An alternative possibility is that slow drift is not removed, but instead occupies an orthogonal subspace in the SC that is not read out by downstream nuclei. This is not beyond the realm of possibility given that an identical scheme appears to exist in the skeletomotor system to stop preparatory signals reaching the muscles (Kaufman et al. 2014; Elsayed et al. 2016; Stavisky et al. 2017; Ames and Churchland 2019). Further research is needed to disentangle these possibilities.

Studies in the fields of psychology and neuroscience have mainly used raw/evoked pupil size to index arousal, but metrics such as heart rate (HR) and galvanic skin response (GSR) are also associated with global brain states. For example, Wang et al. (2018) measured raw pupil size, HR and GSR while human subjects viewed emotional face stimuli specifically designed to evoke fluctuations in arousal. Results showed that all three metrics were positively correlated prior to the presentation of the stimuli. That is, when raw pupil size was large, and the subjects were in heightened state of arousal, HR and GSR increased. In addition, it has been suggested that prestimulus oscillations in the alpha band can be used to index global brain states as they are inversely related to performance on visual detection tasks. Several studies using electroencephalography (EEG) have shown that the probability of detecting near-threshold stimuli increases when prestimulus power in the alpha band is low (Van Dijk et al. 2008; Benwell et al. 2017; Samaha et al. 2017). Simultaneous recordings of spiking activity and EEG in awake behaving monkeys are rare. However, our results predict that slow drift should be associated with prestimulus alpha oscillations.

Research has also uncovered a link between alpha oscillations and microsaccades (Bellet et al. 2017). This effect has been attributed to changes in spatial attention, which have a profound effect on microsaccade direction. For example, Lowet et al. (2018) found that attention-related modulation of spiking responses, Fano factor, and trial-to-trial response variability (measured by r_sc) only occur following a microsaccade in the direction of an attended stimulus. In the present study, we found that slow drift was significantly correlated with microsaccade rate on both tasks. This is unlikely to be explained by changes in spatial attention as both Cowley et al. (2020) and Rabinowitz et al. (2015) found that slow drift on a change detection task was not associated with the blocks of trials used to cue spatial attention inside and outside the RF. In addition, the correlation between slow drift and microsaccade rate in the present study was lower on the change detection task (r = −0.27) than the memory-guided saccade task (r = −0.37). One would not have expected this to be the case if slow drift was mediated by changes in spatial attention. These results raise the possibility that spatial attention and arousal have differential effects on microsaccades. Attention might specifically affect microsaccade direction, whereas arousal might affect microsaccade rate irrespective of direction. Further research is needed to test this hypothesis.

In summary, we investigated if properties related to the eyes are associated with slow drift: a low-dimensional pattern of neural activity that was recently identified in the macaque cortex by Cowley et al. (2020). On both tasks, we found that slow drift was significantly associated with microsaccade rate and evoked pupil size. These results demonstrate that the action of the eyes is associated with a latent dimension of neural activity that can be observed irrespective of the task performed. They suggest that slow drift can be used to index global changes in brain state over time. Further research is necessary to determine the origins of this slow drift in population activity. A key question for future work will be to determine the mechanisms by which slow drift influences behavior in some instances (e.g., when arousal level drives an urgent response) and is circumvented in others (e.g., when an accurate perceptual judgment must be made regardless of arousal level).

Funding

National Institutes of Health (NIH) (Grants R01 EY-029250, R01 MH-118929, R01 EB-026953, and P30 EY-008098) and National Science Foundation (NSF) Grant NCS 1734901 to M.A.S.; NSF Grant NCS 1734901; a career development grant and an unrestricted award from Research to Prevent Blindness; and the Eye and Ear Foundation of Pittsburgh). NIH (Grant K99 EY-025768 to A.C.S.). NIH (Grant T32 EY-017271 to S.B.K.).

Notes

The authors would like to thank Samantha Schmitt for assistance with surgery and data collection, and Ben Cowley for helpful advice and discussion. Conflict of Interest: None declared.