Cortical Information Flow During Flexible Sensorimotor Decisions

Abstract

During flexible behavior, multiple brain regions encode sensory inputs, the current task, and choices. It remains unclear how these signals evolve. We simultaneously recorded neuronal activity from six cortical regions (MT, V4, IT, LIP, PFC and FEF) of monkeys reporting the color or motion of stimuli. Following a transient bottom-up sweep, there was a top-down flow of sustained task information from frontoparietal to visual cortex. Sensory information flowed from visual to parietal and prefrontal cortex. Choice signals developed simultaneously in frontoparietal regions and travelled to FEF and sensory cortex. This suggests that flexible sensorimotor choices emerge in a frontoparietal network from the integration of opposite flows of sensory and task information.

Our reactions are not always the same to the same sensory input. Depending on context, we can map the same input onto different actions. This involves a distributed network of brain regions. During visuomotor decisions, choice predictive activity has been found in frontoparietal regions, including the lateral intraparietal area (LIP) (1–4), prefrontal cortex (PFC) (1, 5–9), and frontal eye fields (FEF) (7), and motor and sensory cortex (10–13). However, it remains unclear how choice signals evolve. Do they flow bottom-up, top-down, or evolve concurrently across brain regions? Do choice signals in sensory regions reflect their causal effect on the decisions or feedback from decision stages (12)? Similarly, little is known about the flow of task signals. Neuronal activity encodes task rules in prefrontal (6, 8, 14, 15), parietal (2), and visual (16) cortices. Task dependent attention modulates neuronal activity throughout sensory cortices (17–19). It remains unknown how task signals evolve across these regions.

We trained two monkeys on a flexible visuomotor task (Fig. 1, Methods). They categorized either the color (red vs. green) or direction (up vs. down) of a colored visual motion stimulus, reporting it with a left or right saccade (Fig. 1A). A visual cue instructed animals about the task (motion or color, Fig. 1C). Each task was indicated by two different visual cues, to dissociate cue and task-related activity. Color and motion spanned a broad range around the category boundaries (yellow and horizontal) (Fig. 1B and Fig. S1). Both monkeys were proficient at categorizing the cued feature (Fig. 1D) (94% and 89% correct for motion and color tasks, respectively, excluding ambiguous trials with stimuli on the category boundary).

Fig. 1. Task, behavior, and neuronal information.

(A) Monkeys categorized the motion direction or color of centrally presented, colored random dot stimuli. Before stimulus onset, a central cue indicated which feature to categorize. Monkeys indicated their choice with a leftward or rightward saccade and held central fixation throughout each trial until their response. Monkeys were free to respond any time up to 3 s past stimulus onset. (B) Stimuli systematically covered motion direction and color space between opposite motion directions (up/down) and opposite colors (red/green; Lab space). All stimuli were 100% coherent, iso-speed, iso-luminant, and iso-saturated. (C) Two different cue shapes cued each task. (D) Responses were strongly modulated by motion and color for the motion and color task, respectively. (E) Time courses of neuronal information in spiking activity about five different task variables averaged across all units and brain regions. Information is measured as percent variance of spiking explained by the variable of interest, independent of all other variables (%EV). (F) Percentage of units per region significantly encoding each type of information (P < 0.05). Dashed lines indicate chance level. (G) Average information encoded for each region and type of information. (H) Schematic display of the recorded brain regions. (I) Time course of average motion, color and choice information analyzed separately for motion and color categorization tasks. Note information is log-scaled to facilitate comparison between tasks. All error bars denote SEM.

We recorded multi-unit activity (MUA) from up to 108 electrodes simultaneously implanted in six cortical regions acutely each day (Fig. 1H and Methods): FEF (532), dorsolateral PFC (1020), LIP (807), IT (57), V4 (155) and MT (123) (total: 2694 multi-units). For each multi- unit, we quantified how neural activity encoded cue identity, task (motion vs. color), stimulus motion direction, stimulus color, and motor choice. Information was quantified as spiking variance across trials explained by each factor. All five types of information were quantified independently, e.g. choice measured only information about the choice that was not explained by cue, task, color or motion (see Methods). To rule out activity due to the saccade itself, we included neuronal activity up to 5 ms before saccade onset.

Averaging across all units revealed temporal dynamics of information (Fig. 1E). Cue information peaked directly after cue onset and stayed tonically elevated during cue presentation (latency to reach half maximum: 74 ± 1 ms). Task information showed a bimodal dynamic. A transient peak shortly after cue onset had a similar latency as cue information (100 ± 25 ms). This transient peak was followed by a dip and later rise of sustained task information (333 ± 15 ms). In contrast to cue information, task information increased during stimulus presentation. Motion and color information rose following stimulus onset with a significantly shorter latency for color (98 ± 2 ms) as compared to motion (108 ± 2 ms) information (P < 0.001). Finally, choice information rose (193 ± 1 ms) before the motor responses (270 ms ± 3 ms) and significantly later than motion and color information (both P < 0.0001).

We quantified for each type of information, the percentage of units with significant effects (Fig. 1F) and the average amount of information (Fig. 1G). We used the second half of the cue interval (0.5 s to 1 s) for cue and task information, the interval from stimulus onset to the average response latency (1 s to 1.270 s) for motion and color information, and the 200 ms interval preceding the saccade for choice information. We found significant encoding of each type of information in each region (P < 0.05 for all regions and information), but the regional profiles differed. In accordance with shape selectivity of V4 and IT, we found the most frequent and strongest cue information there. Task selectivity was frequent in all regions and strongest in V4 and IT. Motion and color information were strongest in MT and V4, respectively. Choice information was most frequent and strongest in LIP, FEF, and PFC. Task (motion vs. color) had little effect on strength and dynamics of motion, color and choice information (Fig. 1I) (20, 21). There was no evidence that only task-relevant sensory information was routed to frontoparietal stages and no evidence that choice information was present only in the task-relevant sensory region. In sum, all types of information were encoded across the entire visuomotor pathway, albeit with different incidences and strength.

Next, we investigated the temporal dynamics of information across regions. Cue information flowed bottom-up, rising first in MT, followed by LIP, V4, IT, FEF and PFC (Fig. 2A). Most of the pairwise comparisons revealed significant latency differences between regions (Fig. 2B, P < 0.001). Task information showed very different dynamics (Fig. 2C). There was a significant early transient peak of task information (<150 ms) in IT and V4 only, without a latency difference between V4 and IT (P > 0.05). The latency of this peak in IT (72 ms) was not different (P > 0.05) from the latency of cue information in IT (also 72 ms). In V4, the transient peak of task information was slightly later (96 ms) than cue information (73 ms) (P < 0.05). Directly following this transient peak, task information was low in the PFC, but then appeared there first and flowed from PFC to LIP, MT, FEF, V4 and IT. Many pairwise latency comparisons were significant according to this pattern (Fig. 2D, P < 0.01). In particular, task information rose earlier in PFC and LIP than in FEF, V4 and IT (all P < 0.01). In summary, IT and V4 first extracted task information from the cues along with the encoding of cue identity. After this transient burst, there was a flow of sustained task-information from PFC and LIP across the entire sensorimotor hierarchy.

Fig. 2. Dynamics of cue and task information.

(A) Each row displays for one brain region the average time-course of neuronal information about cue identity. Left panels display raw information (%EV, same scale for all regions). To support comparison across regions, right panels display time-courses normalized by maximum information for the interval of interest. The bottom right panel shows an overlay of all regions’ information time-courses. Cue and stimulus onset are at Time = 0 s and Time = 1 s, respectively. (B) Comparison of cue information latencies between regions. Latencies are quantified as the time to reach half maximum information. Black dots in the right panel indicate significant latency differences between regions. (C) Time-courses of task information across regions. Same conventions as in (A). (D) Comparison of task information latencies between regions. Latencies were separately analyzed for the early transient peak around 100 ms and for the later sustained increase of task information after 200 ms. Early peak latencies were only estimated for regions that showed a significant effect (V4 and IT, P < 0.05). Same conventions as in (B). All error bars denote SEM.

Motion information rose first in MT followed by LIP, V4, IT, FEF and PFC (Fig. 3A). Color information rose first in MT followed by V4, LIP, FEF, IT and PFC (Fig. 3C). Most pairwise comparisons revealed latency differences between regions according to these sequences (Fig. 3, B and D, P < 0.05). Furthermore, color information appeared significantly earlier than motion information in V4, MT, PFC and FEF (all P < 0.001). Analyzing motion and color tasks individually confirmed these results and showed that neuronal latencies for motion and color information were almost identical for both tasks (Fig. S2).

Fig. 3. Dynamics of motion and color information.

Time-courses and latencies of neuronal information about (A and B) motion direction and (C and D) color of the categorized stimulus. Stimulus onset is at Time = 1 s. All other conventions as in Figure 2.

Choice signals had a different dynamic. If spontaneous fluctuations of activity influenced animals’ choices, activity would predict the choice even before presentation of the motion/color stimulus. Indeed, for all regions except IT, significant choice information preceded stimulus onset (-0.5 to 1 s, P < 0.01). We ruled out that this pre-stimulus choice information merely reflected an effect of the previous trial (see Methods). We next investigated the build-up of choice information during decisions (Fig. 4). Because this reflects the forthcoming behavioral response, we time-locked analysis to the saccade. Choice information increased in LIP and PFC before FEF (Fig. 4B, P < 0.05), but there was no latency difference between LIP and PFC. Choice information increased later in V4 and MT than in LIP and PFC. (Fig. 4B, all P < 0.05), suggesting feedback of choices from frontoparietal stages. Analyzing choice information for motion and color tasks individually confirmed the above results (Fig. S3).

Fig. 4. Dynamics of choice information.

Response-locked (A) time-courses and (B) latencies of neuronal information about the animals’ choice. Responses are at Time = 0 s. Latency was not estimated for IT because there was no significant increase of choice information in IT in the analyzed interval (linear regression, p > 0.05). All other conventions as in Figures 2 and 3.

Our results provide insights into the neuronal mechanisms underlying sensorimotor choices (summarized in Fig. S4). First, sensory (cue/motion/color), cognitive (task), and behavioral (choice) information was not confined to specific cortical regions, but instead broadly distributed. This is incompatible with models of compartmentalized cortical function. Our results instead suggest a graded functional specialization of cortical regions with information shared between regions (22). Second, sensory information flowed feed-forward from sensory cortex. Third, task information was first extracted in an early, transient burst in higher sensory cortex (V4, IT). This early transient may reflect the learned cue associations, i.e. the grouping of the two cues for each task into one representation that is then fed forward to PFC and LIP. After the early transient, sustained task information appeared first in PFC and LIP and then spread to other regions. Thus, task information may need to reach PFC and LIP before being broadcasted across the sensorimotor pathway (23). Fourth, choice predictive activity was present in sensory (V4, MT) frontoparietal (LIP, PFC) and premotor (FEF) cortex before onset of the decision process. This suggests a link between spontaneous fluctuations of neuronal activity along the entire sensorimotor pathway and subsequent decisions. Fifth, choice signals first and simultaneously built up in PFC and LIP and then followed in FEF. Our findings accord with previous reports of ramping choice predictive activity in LIP (3), PFC (7), and FEF (7), but shed light on how choices are made in this network. Our results suggest that, although sensory information reaches LIP and FEF before PFC, the accumulation of sensory evidence occurs first and jointly in LIP and PFC before decision signals are relayed to FEF. Similar dynamics in PFC and LIP could indicate that accumulation of sensory evidence depends on their recurrent interactions (24, 25). The delayed choice signals in FEF may reflect the transformation of accumulated evidence into a discrete choice (26). Sixth, we found an increase of choice signals in LIP and PFC before MT and V4. This is consistent with feedback of choice signals from frontoparietal to sensory cortex (12, 13, 27). This may support cooperative computations between different hierarchical stages (28) and perceptual stability (27). In sum, flexible sensorimotor decisions are not a simple feed-forward process, but result from complex temporal dynamics including feed-forward and feedback interactions between frontal and posterior cortex.

One Sentence Summary.

Sensorimotor decisions result from feed-forward and feedback interactions between frontal and posterior cortex.