When Attention Wanders: How Uncontrolled Fluctuations in Attention Affect Performance

Abstract

No matter how hard subjects concentrate on a task, their minds wander (Raichle et al., 2001; Buckner et al., 2008; Christoff et al., 2009; Killingsworth and Gilbert, 2010). Internal fluctuations cannot be measured behaviorally or from conventional neurophysiological measures, so their effects on performance have been difficult to study. Previously, we measured fluctuations in visual attention using the responses of populations of simultaneously recorded neurons in macaque visual cortex (Cohen and Maunsell, 2010). Here, we use this ability to investigate how attentional fluctuations affect performance. We found that attentional fluctuations have large and complex effects on performance, the sign of which depends on the difficulty of the perceptual judgment. As expected, attention greatly improves the detection of subtle changes in a stimulus. Surprisingly, we found that attending too strongly to a particular stimulus impairs the ability to notice when that stimulus changes dramatically. Our results suggest that all previously reported measures of behavioral performance should be viewed as amalgamations of different attentional states, whether or not those studies specifically addressed attention.

Introduction

It is well known that our internal state affects our ability to perform even basic tasks. Laboratory studies in psychology, economics, and behavioral neuroscience typically measure the effects of internal state on perception by manipulating specific cognitive factors such as arousal, attention, or motivation and comparing behavioral performance between blocks of behavioral trials. Despite the best efforts of experimenters and subjects, however, a subject's internal state is likely to vary from trial to trial. Indeed, behavioral studies indicate that our minds wander away from the present task as much as half the time (Killingsworth and Gilbert, 2010). Because fluctuations in cognitive factors such as attention, arousal, or motivation are difficult to control or even measure, their effects on behavior have been difficult to assess. The results of all psychophysical studies therefore reflect average performance over a range of cognitive and behavioral states, and uncontrolled variability could potentially have large and complex effects on performance.

Perceptual abilities are typically assessed using psychometric functions, which plot average behavioral performance (e.g., percentage of stimuli detected) as a function of task difficulty (e.g., stimulus intensity) in a given condition. Fluctuations in internal state can in principle affect psychometric functions, and therefore assessment of behavioral performance, in systematic ways. For example, if internal fluctuations did nothing but change psychophysical thresholds, the average psychometric curve would be an average of several identical curves that have been shifted left or right. This average curve would have a lower slope than its constituent curves, providing a distorted picture of the underlying behavioral sensitivity.

We investigated the impact of uncontrolled variability on behavior by examining visual attention, which is a well studied cognitive factor that affects subjects' perceptual abilities. Attention allows observers to focus on a subset of a complex visual scene and improves perception of attended objects or features. In addition to improving behavior, attention modulates the responses of sensory neurons, typically by increasing responses to attended stimuli (Assad, 2003; Yantis and Serences, 2003; Reynolds and Chelazzi, 2004; Maunsell and Treue, 2006).

We used the fact that attention modulates neuronal responses to measure uncontrolled fluctuations in attention. The responses of individual sensory neurons are themselves too variable to provide a precise measure of attention on individual trials, but we have shown that the responses of dozens of simultaneously recorded sensory neurons can be used to measure attention nearly instantaneously. Attentional variability can reliably predict large changes in an animal's ability to perform a difficult psychophysical task (Cohen and Maunsell, 2010).

Here, we use the ability to measure attentional state on individual trials to investigate how attentional fluctuations affect performance on a visual orientation change detection task of varying difficulty. We found that fluctuations in attention can affect psychometric functions in diverse and unexpected ways. Our results suggest that all previously reported measures of behavioral performance should be viewed as amalgamations of behavioral performance associated with different attentional states, whether or not those studies specifically addressed attention.

Materials and Methods

Subjects and electrophysiological recordings.

Different aspects of the data presented here have been described previously (Cohen and Maunsell, 2010). All procedures were approved by the Institutional Animal Care and Use Committee of Harvard Medical School. Our subjects were two male rhesus monkeys (Macaca mulatta), weighing 9 and 12 kg. Before behavioral training, each animal was implanted with a head post and a scleral search coil for monitoring eye movements. After the animal learned the behavioral task (3–4 months), we implanted a 6-by-8 array of microelectrodes (Blackrock Microsystems) in V4 in each cerebral hemisphere (96 electrodes per animal). Each electrode was 1 mm long, and the distance between adjacent electrode centers was 400 μm. The two arrays were connected to a percutaneous connector that allowed simultaneous recordings from all 96 electrodes.

We implanted the arrays between the lunate and superior temporal sulci, which were visible during surgery. The centers of the spatial receptive fields for both monkeys were in the lower hemifield (eccentricities for Monkey 1: left hemifield, 3–5°; right hemifield, 5–8°; eccentricities for Monkey 2: left hemifield, 10–15°; right hemifield, 15–30°). The receptive fields of most neurons within a hemisphere overlapped at least partially (Cohen and Maunsell, 2009). Monkey 2 underwent an unscheduled explantation of both arrays before recordings began, so we implanted new arrays several millimeters dorsal to the sites of the original implants. Consequently, Monkey 2 had receptive fields that were larger and more eccentric than Monkey 1. Otherwise, the physiological results were indistinguishable between the two monkeys.

We recorded from 461 single neurons (235 from Monkey 1 and 226 from Monkey 2) and 4413 multiunits (1721 from Monkey 1 and 2692 from Monkey 2) over 49 recording sessions. All spike sorting was done manually off-line using spike-sorting software (Plexon Inc.). Previously, we found that the physiological effects of attention in single units were indistinguishable from multiunits (Cohen and Maunsell, 2009), and the population analyses presented here required large populations of neurons recorded simultaneously, so all analyses include both single units and multiunits. On average, we recorded 46 single units and multiunits per hemisphere during each session (range, 14–74).

Behavioral task and visual stimuli.

The monkeys performed the orientation change detection task depicted in Figure 1A. A trial began when the monkey fixated a small white spot within a 1.5° square fixation window in the center of a video display (85 Hz frame rate, 1024 × 768 pixels, gamma corrected). Two achromatic, 100% contrast, odd-symmetric Gabor stimuli whose size, location, spatial frequency, and orientation were optimized for one single unit in each hemisphere (new units and stimuli in each session) flashed on for 200 ms and off for a randomized period (200–400 ms picked from a uniform distribution between each stimulus presentation). In one unsignaled presentation, the orientation of one of the stimuli was different from preceding stimuli, and the monkey was given a liquid reward for making a saccade to the stimulus that changed within 100–500 ms of its appearance. The smallest orientation change was always 2° and increased in five logarithmic steps to either 45° or 60°. To encourage the monkey to maintain attention throughout the trial and to discourage guessing, the time of the orientation change was drawn from an exponential distribution (minimum 1000 ms, mean 3000 ms, maximum 5000 ms). If no orientation change occurred within 5000 ms, the monkey was rewarded for simply maintaining fixation, and the trial was excluded from analysis. We found no dependence of behavioral performance on trial length.

Figure 1.

Fluctuations in attention change both the thresholds and slopes of psychometric curves. A, Schematic of the orientation change detection task. Two Gabor stimuli synchronously flashed on for 200 ms and off for a randomized 200–400 ms period. At an unsignaled time picked from an exponential distribution (minimum 1000 ms, mean 3000 ms, maximum 5000 ms), one of the stimuli was presented in a different orientation, and the monkey was rewarded for making a saccade to the stimulus that changed. Attention was cued in blocks, and the cue was valid on 80% of trials, meaning that on an “attend-left” block of trials (depicted here), 80% of orientation changes were to the left stimulus. The monkey was rewarded for correctly detecting any change, even on the unattended side. All analyses were performed on responses to the stimulus before the orientation change (black outlined panel). B, Fitted psychometric curves sorted by position on attention axis. Strong attention (red lines and large positive values) improves performance on difficult trials relative to weak attention (blue lines). Arrows indicate the two orientation change bins plotted in C. C, Proportion correct as a function of position on the attention axis for the largest and smallest orientation change bins. Error bars are bootstrapped 95% confidence intervals. D, E, Fitted threshold (D) and slope (E) as a function of attention axis position. Error bars are bootstrapped 95% confidence intervals.

Because we observed some adaptation of neuronal responses between the first and second stimulus presentations, the changed stimulus never occurred before the third stimulus presentation. The mean neuronal response was not significantly correlated with stimulus presentation number for the second through last stimuli (p = 0.21), and neuronal responses to the second stimulus and the stimulus immediately preceding the changed stimulus were not significantly different (paired t test on all 4874 single and multiunits, p = 0.13).

We manipulated attention in blocks of trials by presenting 10 instruction trials before the start of each block. Instruction trials (which were not analyzed) consisted of a single flashing Gabor stimulus. The stimulus that appeared in the instruction trials was the one that changed orientation in 80% of trials in the upcoming block. We tested five different orientation changes in each block. To obtain reliable neuronal and psychometric data for the orientation changes of interest and also to keep the monkey's reward rate sufficiently high, we presented an unequal number of trials at each difficulty level. Of 125 trials per block, 100 contained changes in the attended stimulus. The block also contained 25 changes in the unattended stimulus, which were all a single, intermediate orientation change. These invalidly cued trials provided an important behavioral means of assessing whether the monkey attended differentially in the two cued-attention conditions, but all of the analyses focus on the 80% of trials in which the change occurred at the cued location. Within each block of trials, the trial types (valid or invalid cues, and size of the orientation change) were randomly interleaved. Only one stimulus change occurred in each trial, and the monkey was rewarded for correctly detecting a change in either stimulus, regardless of cued location. Each analyzed recording session comprised at least four blocks of trials in each attention condition (at least 1000 total trials).

Single-trial measure of attention.

We were interested in a measure of attention on individual trials that was independent of the visual stimulus or motor preparation. We therefore focused all analyses on the stimulus presentation immediately before the orientation change (Fig. 1A). Importantly, this pair of stimuli was the same on every trial throughout an experimental session, regardless of attentional location or the size of the eventual orientation change.

We extracted spike counts from the period between 60 and 260 ms after stimulus onset, which allowed for the latency of visual responses in area V4. We included only trials in which a change occurred in the cued location and the animal either correctly detected the change or missed the change. We excluded catch trials (in which no change occurred within 5000 ms and the monkey was rewarded for maintaining fixation; 12% of trials), false alarms (eye movements to one of the stimuli before an orientation change occurred; 4% of trials), or trials in which the monkey broke fixation by making an eye movement to a location other than one of our two stimuli (5% of trials).

We estimated the monkey's attentional state on each trial by quantifying the similarity of the population response (generated from all simultaneously recorded neurons from both cerebral hemispheres) on a given trial to the mean response in each attention condition. This process has been described in detail previously (Cohen and Maunsell, 2010). We expressed the population response on each trial as a point in an n-dimensional space in which the response of each simultaneously recorded neuron represented one dimension. Therefore, if a recording session contained 79 simultaneously recorded neurons in the two hemispheres combined, the population response on each trial would be a point in a 79-dimensional space.

We then constructed a putative “attention axis” connecting mean responses before correct detections in each of the two cued attention conditions. The population response on each trial was then projected onto this axis. We normalized the scalar projections for each recording session so that a projection of 1 was equal to the mean response before correct detections of stimulus changes in the same condition as the current trial, and a projection of −1 was equal to the mean response before a correctly detected change in the opposite condition. Projections onto the attention axis are not bounded: the mean of the distribution of projections before correct detections is defined as 1, so many trials will have projections that lie beyond this value. The attention axis was constructed based only on data from correct trials, so missed trials provide an important test of whether position on the attention axis correlates with behavioral performance. On average, projections for trials in which the animal missed the orientation change had smaller values than for correct detections, meaning that attention was shifted toward the mean of the opposite attention condition (see also Cohen and Maunsell, 2010). The average attention axis projection for all correct and missed trials was 0.41.

To compute the effect of fluctuations in attention on psychometric curves, we combined the data from all recording sessions and discarded the outlying 1% of trials (0.5% of trials with the largest and 0.5% with the smallest values on the attention axis). We assigned the remaining trials to seven bins based on attention axis projection such that there were an equal number of trials in each bin, and then computed a psychometric curve for each of the two stimulus locations in each session. The monkeys' overall performance varied from session to session (due in part to the different stimulus eccentricities in different sessions), so we shifted the psychometric curves for each session left or right so that the average psychometric thresholds for all sessions were aligned (median threshold = 9.3°, 25th percentile threshold = 4.5°, 75th percentile threshold = 14.8°). We then combined the data from across all recording sessions, fit the combined psychometric curves with Weibull functions, and extracted slopes and thresholds for the function for each attention bin.

Analysis of psychophysical data.

We defined percentage correct as the number of correct detections divided by the number of correct detections plus the number of missed changes. We fit behavioral data for stimulus changes in the attended location (Fig. 1) using a Weibull function:

where p is the proportion of correct responses and c is the orientation change in degrees. The parameter α represents the orientation change at which performance is 63% correct, which we used for threshold. The parameter β controls the slope of the curve. We computed confidence intervals for each proportion correct using a bootstrap test in which we picked, with replacement, random sets of trials within an attention bin, and orientation change and recomputed proportion correct for the resulting datasets. We repeated this process 100,000 times and defined the confidence interval as the middle 95% of proportion correct values (Figs. 1C, 2B,D, error bars). We also fit each randomly created psychometric curve with a Weibull function to create distributions of the parameters α and β and computed 95% confidence intervals for these fitted parameters (Fig. 1D,E, error bars).

Figure 2.

When the task involves differences in orientation as well as location, strong attention impairs subjects' ability to detect large orientation changes. A, Proportion correct as a function of normalized orientation change sorted by attention axis position as in Figure 1B, for recording sessions in which the orientation of the two stimuli differed by <45° (28/49 recording sessions). B, Proportion correct as a function of position on the attention axis for the largest and smallest orientation change bins. Conventions as in Figure 1C. C, D, Same as A and B for recording sessions in which the orientation of the two stimuli differed by >45° (21/49 recording sessions).

Results

Our primary goal was to assess how psychophysical performance depends on fluctuating attentional state. We sorted all trials by attention axis position and constructed a psychometric curve using trials in each attention axis bin (see Materials and Methods).

Overall, the best performance was observed for trials occupying positions on the attention axis beyond the average position for attention to the cued stimulus (values >1; Fig. 1B, red curves). However, the influence of attention depended on the difficulty of the perceptual judgment: stronger attention (larger, positive projections on the attention axis) leads to markedly improved performance on the most difficult trials (Fig. 1C, smallest orientation change). In contrast, attention has little effect on performance on the easiest trials (Fig. 1C, largest orientation change).

As is standard in psychophysical studies, we summarized each psychometric curve by extracting values of psychometric threshold and slope from the Weibull fits (Fig. 1D,E) (see Materials and Methods). As expected, we found that attending strongly to the cued stimulus is associated with better (smaller) psychometric thresholds (Fig. 1D). Thresholds were poorest when the attention axis position approached the average position for attention to the other, uncued stimulus (values near −1). Although increased attention is often viewed as simply shifting psychometric functions, thereby changing the threshold without changing the slope (Lu and Dosher, 1998; Lee et al., 1999; Carrasco et al., 2000; Cameron et al., 2002), we found that it also made the functions shallower by preferentially enhancing detection of the smallest orientation changes than on detection of larger changes (Fig. 1C).

Unexpectedly, our data show that attending strongly to a stimulus has little effect or can even impair performance on the easiest trials (Fig. 1C, largest orientation change). On trials occupying positions on the attention axis near or beyond 1, detection of the largest orientation changes was poorer than in other attention states (93% correct for the largest attention axis position vs 98% for other attention axis positions; binomial test, p < 0.01). This is surprising because the largest orientation change was usually 60°, which might be expected to be easily detected under any circumstances. Instead, our results suggest that attending to a stimulus made it more difficult to see this very large orientation change. We hypothesized that this impairment arose from feature attention. Attending to a particular value of a feature (e.g., orientation, direction, or color) can reduce neuronal responses to different values of that feature (Haenny et al., 1988; Motter, 1994; Treue and Martínez Trujillo, 1999; McAdams and Maunsell, 2000; Martinez-Trujillo and Treue, 2004; Maunsell and Treue, 2006; Hayden and Gallant, 2009; Khayat et al., 2010; Cohen and Maunsell, 2011). In our task, focusing intently on orientations close to the base orientation might impair detection of orientations that differ greatly from that orientation.

Because the attention axis compares the population response on the current trial to the average response in the two cued attention conditions, it measures attention along all of the stimulus dimensions along which the two stimuli differ. The two stimuli used on a given session often had different orientations (Fig. 1A), so the attention axis typically captures not only whether spatial attention is directed to one location or the other, but also whether feature attention is directed to one orientation or the other. However, feature attention will have little impact on attention axis projections in recording sessions when the two repeating stimuli happened to have similar orientations (Fig. 2A), because the attention axis measures primarily spatial attention. In sessions when the two stimuli have very different orientations (Fig. 2C), projections onto the attention axis capture both spatial and feature attention.

We tested the prediction that feature attention impairs the monkey's ability to detect the largest orientation changes by dividing sessions into two groups depending on whether the orientation of the two repeating stimuli differed by more or less than 45°. Consistent with the feature attention hypothesis, we found that strong attention impairs performance on the easiest trials when the attention axis captures fluctuations in both feature and spatial attention (Fig. 2D, largest orientation changes) but not when the attention axis measures primarily spatial attention (Fig. 2B). The combination of feature and spatial attention has a larger effect on performance on difficult trials than spatial attention alone: the monkeys showed a greater range of performance as a function of attention axis position when the axis measured both types of attention (Fig. 2D, smallest orientation change) than when the attention axis measured primarily spatial attention (Fig. 2B).

Discussion

Even under the most elaborate laboratory conditions, it is impossible to control a subject's internal state completely, so fluctuations in cognitive states like attention must occur in all behavioral experiments. Our results show that variability in attention can be identified using the responses of a few dozen simultaneously recorded neurons and that attentional fluctuations affect psychometric curves in complicated ways.

The observation that strong attention can impair perception of very suprathreshold stimuli is revealing for two reasons. First, it informs us about how feature attention affects perception. That attention improves perception of small but not large orientation changes suggests that, consistent with the predictions of feature-similarity gain model for sensory responses (Treue and Martínez Trujillo, 1999; Martinez-Trujillo and Treue, 2004), feature attention enhances perception of orientations near that of the repeating stimulus rather than amplifying the difference between successive stimuli.

Second, these results provide a novel explanation for the common observation that subjects usually do not achieve perfect performance even on easy tasks. These mistakes are usually attributed to sloppiness, and this “lapse rate” is often used to quantify a subject's motivation or arousal. However, in our dataset, the animals performed nearly perfectly on easy trials in some attentional states but less well in others, suggesting that lapses on easy trials may be explained in part by extremely high levels of feature attention or similar factors, rather than a lack of motivation.

Our results indicate that inevitable fluctuations in subjects' attentional states have a large and complex effect on psychometric curves, but they offer an encouraging view for interpreting the results of behavioral studies. If attention had, as previously hypothesized (Lu and Dosher, 1998; Lee et al., 1999; Carrasco et al., 2000; Cameron et al., 2002), simply changed psychophysical thresholds without changing the slope of the psychometric curve, the average curve would have had a very different slope than its constituent curves corresponding to different attentional states. Because attention changes slopes as well as thresholds (Fig. 1), our results indicate that the average performance measured in behavioral experiments is in effect an unbiased estimate of the animal's psychometric curve. That is, although the animal's attentional state, and therefore the true psychometric function, is constantly changing, the measured curve will be, on average, very similar to the average underlying curve. Future work will be needed to determine whether fluctuations in other internal states share this property.

More generally, our results suggest that fluctuations in internal state affect neurons throughout cortex in a coordinated way, meaning that the responses of small groups of neurons in many areas can likely be used to measure fluctuations in attention and other cognitive factors. There is no reason that the neurons we recorded should have been particularly well suited for detecting fluctuations in attention. The tuning properties of these neurons were not precisely matched to the visual stimuli we presented. Neurons throughout visual and frontal cortex respond to visual stimuli and are modulated by attention. Many of those neurons have tuning that is better matched to the stimulus or show stronger attentional modulation than the neurons we recorded. That the responses of a small population of somewhat randomly sampled neurons in a single area correlate so well with behavior suggests that the responses of a wide variety of cortical neurons reflect fluctuations in attention. This observation is promising for the future of relating fluctuations in attention and other cognitive states to different aspects of behavior because it suggests that internal state can be measured using an experimentally tractable number of neurons in a variety of cortical areas.

Further work is needed to identify fluctuations in other cognitive factors and their effects on behavior, but the current results show specific ways in which a wandering mind affects behavioral performance. Grasping the implications of variability in cognitive states on basic perceptual and motor abilities may be an important step toward understanding the scope of impairments in cognitive functions such as attention and the neuronal mechanisms that underlie them.