Task context impacts visual object processing differentially across the cortex

Significance

Visual recognition is often thought to depend on neural representations that primarily reflect the physical properties of the environment. However, in this study we demonstrate that the intent of the observer fundamentally perturbs cortical representations of visual objects. Using functional MRI we measured the patterns of response to identical objects under six different tasks. In any given task, these patterns could be used to distinguish which object was being viewed. However, this ability was disrupted when the task changed, indicating that object representations reflect not only the physical properties of the stimulus, but also the internal state of the observer.

Abstract

Perception reflects an integration of “bottom-up” (sensory-driven) and “top-down” (internally generated) signals. Although models of visual processing often emphasize the central role of feed-forward hierarchical processing, less is known about the impact of top-down signals on complex visual representations. Here, we investigated whether and how the observer’s goals modulate object processing across the cortex. We examined responses elicited by a diverse set of objects under six distinct tasks, focusing on either physical (e.g., color) or conceptual properties (e.g., man-made). Critically, the same stimuli were presented in all tasks, allowing us to investigate how task impacts the neural representations of identical visual input. We found that task has an extensive and differential impact on object processing across the cortex. First, we found task-dependent representations in the ventral temporal and prefrontal cortex. In particular, although object identity could be decoded from the multivoxel response within task, there was a significant reduction in decoding across tasks. In contrast, the early visual cortex evidenced equivalent decoding within and across tasks, indicating task-independent representations. Second, task information was pervasive and present from the earliest stages of object processing. However, although the responses of the ventral temporal, prefrontal, and parietal cortex enabled decoding of both the type of task (physical/conceptual) and the specific task (e.g., color), the early visual cortex was not sensitive to type of task and could only be used to decode individual physical tasks. Thus, object processing is highly influenced by the behavioral goal of the observer, highlighting how top-down signals constrain and inform the formation of visual representations.

Perception reflects not only the external world but also our internal goals and biases. Even the simplest actions and decisions about visual objects require a complex integration between “top-down” (internally generated) and “bottom-up” (sensory-driven) signals (1). For example, the information used for object categorization depends on top-down signals arising from the spatial (2) or conceptual (3) context in which the object appears, the prior experience of the observers (4, 5), and the specific task (6, 7). Despite such strong behavioral evidence, the neural correlates of this integration remain unclear, both in terms of the cortical regions involved and the extent of the integration within those regions. Here, we investigate the impact of diverse behavioral goals on the neural architecture that supports object processing.

Object recognition is known to depend on the ventral visual pathway, a set of interconnected cortical regions extending from early visual areas (e.g., V1/V2) into the anterior inferotemporal cortex (8). It has been argued that object processing along this pathway can largely be captured in feed-forward hierarchical frameworks without the need for top-down signals (9–12). For example, in the HMAX model (10), the integration of top-down signals is largely constrained to the extrinsic targets of the pathway (8), and in particular the lateral prefrontal cortex (LPFC) (13–17). However, there is strong evidence that top-down signals, such as attention and task, modulate the magnitude of response to simple visual stimuli (e.g., gratings) in early visual areas (18–23) and the response to objects in extrastriate regions (24–30).

Although these prior studies provide evidence of an effect of top-down signals on object processing, they afford only limited insight because they tested only the modulation of overall activity and not the impact of top-down signals on fine-grained object information available in the response. The importance of this distinction between gross modulation versus fine-grained information is apparent in functional MRI (fMRI) investigations of working memory, where not all regions that evidence activity modulations contain information about the maintained objects (31, 32). Crucially, quantifying object information allows for a direct test of whether object representations are task-independent (equivalent information within and across tasks) or task-dependent (reduced information across compared with within tasks). Without this test, it remains unclear whether top-down signals, such as task, fundamentally alter the representations of objects or simply scale the response to them.

To investigate the full range of task effects, we presented a broad set of objects in six separate tasks, half of which probed physical properties of the stimulus (e.g., color: red/blue) and half its conceptual properties (e.g., content: manmade/natural). This paradigm overcomes a limitation of previous studies, which often treated task and stimulus as simple dichotomous variables (26–30), making it difficult to generalize beyond the limited range of tasks and objects tested. Furthermore, previous studies often manipulated only whether an object was attended or not, and therefore could not establish how different types of information are extracted from the same attended stimuli. In contrast, by presenting an identical set of object images under multiple tasks, all requiring attention to the images, and extracting the response to each combination of task and object, we were able to directly test the effect of task on object responses.

Our results revealed that task context has a pervasive effect on visual representations throughout the early visual cortex (EVC), the ventral visual pathway, and the LPFC. Task modulated both response magnitude and multivariate response patterns throughout these regions. Critically, responses in the ventral object-selective as well as the LPFC, were task-dependent, with reduced object information across tasks compared with within task. In contrast, object information in the EVC was task-independent, despite large task-related activity modulations. Together, these findings demonstrate that top-down signals directly contribute to and constrain visual object representations in the ventral object-selective and LPFC. Such effects strongly support a recurrent, highly interactive view of visual object processing within the ventral visual pathway that contrasts with many primarily bottom-up frameworks (9, 10).

Results

Twenty-five participants (10 males) viewed images of everyday objects (Fig. 1A) while they performed six different tasks, requiring judgments about either a physical (fixation, color, tilt) or a conceptual property (content, movement, size). We used a fully interleaved event-related fMRI design that allowed us to simultaneously and independently manipulate both the object presented and the task performed on each trial (Fig. 1B).

Fig. 1.

Experimental paradigm. (A) Examples of each of the eight objects presented (butterfly, cow, dresser, flower, motorcycle, roller skates, tree, vase). Six unique exemplars of each object were used, totaling 48 individual stimuli. Each stimulus was presented in each of the six tasks allowing us to compare the effect of task when the physical stimuli are held constant. (B) Sequence of events in a single trial. Each trial commenced with a cue specifying the task (physical tasks: fixation, color, tilt; conceptual tasks: content, movement, size), followed by a jittered ISI and then the presentation of the object stimulus. After another jittered ISI, a response screen was presented indicating the two response alternatives for that task and which button (left or right) was associated with each (response mapping).

Our primary question was whether the neural representations of objects vary as a function of task. Thus, we independently localized two object-selective regions-of-interest (ROIs) in the occipito-temporal cortex: lateral occipital (LO) and posterior fusiform (pFs). To compare the effect of top-down signals on these areas with the earliest and latest stages of visual processing, we also localized the central-field representation of the EVC and LPFC, respectively (Materials and Methods). We performed two types of analyses: First, we used multivariate pattern analysis to investigate the effects of task on object information present in the distributed pattern of response. Second, we examined the overall effect of task on both the multivoxel and average response in each ROI.

Average Activation.

In each ROI (EVC, LO, pFs, LPFC), we first extracted the magnitude of response for each of the six tasks averaged across objects (Fig. 2). A three-way ANOVA with Task, ROI, and Hemisphere (left, right) as factors, revealed a significant ROI × Task × Hemisphere interaction [F(15, 270) = 2.64, P < 0.03]. Separate two-way ANOVAs in each ROI with Task and Hemisphere as a factor showed that the EVC and LO evidenced significant task modulations [EVC, F(5, 120) = 4.77, P < 0.001; LO, F(5, 105) = 6.76, P < 0.0001]; other ROIs (all F < 1.14, P < 0.35), largely because of a stronger response in the fixation, compared with all other the tasks (paired t tests; all t > 2.55, P < 0.02). This modulation of activation in the EVC is consistent with prior studies (e.g., refs. 18 and 21). Only the LPFC showed any effect of Hemisphere, manifested in a significant Task × Hemisphere interaction [F(5,120) 11.55, P < 0.00], largely because of differing preferences for the conceptual and physical tasks (see Task Type, below). The differences in task effects between ROIs argue against a general arousal or difficulty account of the modulation. Furthermore, there was no evident relationship between the observed activations and behavioral performance measured outside the scanner. Briefly, across all ROIs, none of the reaction times for the different task conditions were significantly correlated with the response magnitude for the individual tasks (SI Text and Table S1). These findings strongly suggest that the task effects cannot be simply explained by differences in performance across tasks.

Fig. 2.

Task effects on magnitude of response. Average response magnitude in each of the ROIs for the six tasks (physical tasks, blue; conceptual tasks, red) averaged across hemispheres. The EVC and LO showed a significant main effect of task, driven primarily by higher magnitude of response to the fixation task relative to all other tasks. *P < 0.05, Main effect of Task, assessed within each ROI by a Hemisphere × ROI ANOVA. Note that although the LPFC did not show a significant main effect, it displayed a significant Task × Hemisphere interaction (see text for details). n.s., not significant.

Multivariate Information Analysis.

To quantify the precise effect of task on object information, we used multivariate pattern analysis (Materials and Methods) to quantify the relationship between every possible pairing of object and task. Specifically, we cross-correlated the voxel response patterns for every possible pair of the 48 conditions (8 objects × 6 tasks) across independent halves of the data. This split-half analysis yielded a 48 × 48 similarity matrix for each ROI in which each point represents the response similarity between a pair of conditions (Fig. 3). These matrices provide a very rich description of the strong and differential integration of task and object in the response patterns of the ROIs. Here we will focus on three particular aspects of the information available in these matrices. First, and critically, the relative strength of object decoding both within and across tasks (Fig. 3C, Upper) as a measure of the impact of task on object information; second, task type decoding (physical/conceptual) (Fig. 3C, Lower Left) as a measure of the grouping of object responses by the type of high-level judgment; finally, task decoding within each task type (Fig. 3C, Lower Right) as a measure of the grouping of object responses by individual task.

Fig. 3.

Comparison of multivoxel object and task responses. (A) Raw similarity matrices for the EVC, LO, pFs, and LPFC ROIs averaged across all participants. Each matrix is 48 × 48 cells (6 tasks × 8 objects), with each cell reflecting the correlation between a pair of conditions across two independent halves of the data. Solid lines denote borders between the two Task Types (physical, conceptual). Dashed lines denote borders between individual tasks. Note that the positive and negative correlations are generally well grouped by task, suggesting that information about task is manifest in the neural pattern of response of all visual ROIs as well as in LPFC. The colors are scaled from the highest (yellow) to lowest (cyan) correlation value in each matrix. (B) Organization of the similarity matrices. (C) Schematic matrices indicating how different effects would be manifest in the similarity matrices and how the decoding indices were calculated. Object information within task (Upper Left) is indicated by stronger correlations for the same object (black) than for different objects (gray). Object information across tasks (Upper Right) is indicated by stronger correlations for the same object in different tasks (black) compared with different objects (gray). Task type information (Lower Left) is indicated by stronger correlations between all conditions of the same task type (black), compared with those across task type (gray). Finally, task information (Lower Right) for each task type is indicated by stronger correlations between objects in the same task (black) compared with different tasks (gray).

Task-Dependence of Object Information.

We started by investigating the critical question of whether object decoding is equivalent within and across tasks (task-independent) or reduced across tasks (task-dependent) (Figs. 3C and 4A).

Fig. 4.

Object decoding within and across tasks. (A) Examples of within-task object decoding (Left) and across-task object decoding (Right) averaged across tasks in the pFs. For each individual task, we focused on the 8 × 8 object similarity matrix and calculated within-task object decoding indices by subtracting the average between-object correlations (off-diagonals) from the within-object correlations (diagonal). Across-task object-decoding indices were calculated in a similar manner but focusing on the object similarity matrices comparing object response patterns in two different tasks and averaging across all possible pairwise comparisons of tasks. In this example for the pFs, strong object decoding within task is abolished when comparing across tasks. (B) Within-task object-decoding indices (green bars) and across-task indices (purple bars). Although within- and across-task object-decoding indices were significantly above chance in all ROIs except LPFC, the relative levels of within- and across-task decoding varied. In the EVC and LO, there was no difference in decoding, suggesting task-independent object representations. In contrast, in both the pFs and LPFC across-task decoding was significantly weaker then within-task decoding, suggesting task-dependent object representations. All error bars in this and every other plot indicate the between-subjects SE. *P < 0.05.

First, to compute within-task object decoding, for each separate task we subtracted the average between-object correlations (off-diagonals) from the average within-object correlation (main diagonal; schematically illustrated in Fig. 3C, Upper Left, black minus gray squares; e.g., Fig. 4A, Left). This results in an index of object decoding, where any value significantly greater than zero reflects the presence of significant object information. The strength of within-task object decoding did not vary as a function of Task [F(5, 90) < 1.00] or Task Type [F(1, 18) < 1.00], as assessed by a Hemisphere × ROI × Task (or Task Type) ANOVA. Therefore, the within-task object decoding indices were averaged across all tasks, revealing significant object decoding in all visual areas (see also refs. 33–35) (all t > 3.73, P < 0.001) (Fig. 4B; see also Table S2 for analogous results in terms of percent correct) as well as the LPFC (t > 2.55, P < 0.008). An ANOVA with ROI and Hemisphere as factors revealed that within-task object decoding varied across ROIs [F(3, 54) = 5.86, P < 0.003] and was stronger in visual areas (EVC, LO, pFs) compared with the LPFC (all t > 3.15, P < 0.004).

We next calculated across-task object information by conducting a similar subtraction across each pairwise comparison of tasks (schematically illustrated in Fig. 3C, Upper Right) and then averaged across all possible task pairs (e.g., Fig. 4A, Right; see also Table S2), revealing significant across-task object decoding in all visual areas (all t > 3.31, P < 0.003), but not the LPFC (t = 1.26, P > 0.10). An ANOVA with ROI and Hemisphere as factors revealed that across-task object decoding also varied across ROIs [F(3, 54) = 16.80, P < 0.0001] with stronger decoding in visual areas (EVC, LO, pFs) than in the LPFC (all t > 3.43, P < 0.001).

Although object decoding was significantly above chance both within and across tasks in all visual ROIs, the critical test is the relative level of object decoding. In the EVC and LO, within- and across-task object decoding was similar, suggesting task- independent object representations. However, in both the pFs and LPFC, within-task object decoding was greater than across-task object decoding, suggesting task-dependent object representations (Fig. 4B). A three-way ANOVA with ROI, Hemisphere (left, right), and Task (within task, across tasks) as within-subject factors revealed a significant main effect of ROI [F(3, 54) = 11.01, P < 0.0001], reflecting the weaker decoding in the LPFC than the visual regions, no effects of Hemisphere and, critically, a significant Task × ROI interaction [F(3, 54) = 5.12, P < 0.005]. A series of t tests demonstrated stronger within- than across-task object decoding in the pFs [t(24) = 2.91 P < 0.004] and LPFC [t(24) = 2.07, P < 0.02], but equivalent within- and across-task object decoding in the EVC and LO (both t < 1.00, P > 0.20). This pattern of results was also found when the fixation task, which differed from the other tasks in response magnitude in some ROIs, was excluded from the decoding analysis (Fig. S1).

Importantly, variations in noise across individual tasks cannot account for the reduction in decoding, because there were no significant differences in within-task object decoding (see above). Thus, the present results demonstrate that object representations in the pFs and LPFC are task-dependent (changing systematically and significantly across tasks).

Task Information.

We next investigated whether task in general is reflected in the response patterns of the ROIs. The full similarity matrices suggest that task is a strong and differential driver of the variance in correlations in all ROIs (Fig. 3A). To better visualize this effect, we used multidimensional scaling (MDS) to produce plots wherein the distance between points reflects the dissimilarity in their patterns of response (Fig. 5). There was clear separation by task in all ROIs, but the exact nature of the separation varied. In particular, the LO, pFs, and LPFC primarily showed a separation by Task Type (physical, conceptual). This separation was most pronounced in the LPFC, which also showed a strong segregation of individual physical tasks. In contrast to the other ROIs, the EVC did not show a distinction between the two Task Types, but did evidence a strong distinction between the fixation and other tasks (Fig. 5). To quantify these observations we used two measures of task information: decoding of Task Type (i.e., how distinguishable are the response patterns to physical and the conceptual tasks), and decoding of individual tasks within each Task Type (i.e., how distinguishable from one another are the response patterns to individual tasks).

Fig. 5.

Task information. MDS plots highlighting the relationship between responses for each task in each ROI. The distance between points represents the similarity in the response patterns for the different conditions, with closer distance representing greater similarity. In the EVC and LO the structure strongly reflects the distinctiveness of the fixation task, with little grouping by Task Type. In contrast, the pFs and LPFC display a weaker separation of the fixation task, but also a strong grouping by Task Type, particularly in the LPFC. Response patterns in the LPFC also display a strong separation of the individual physical tasks.

Task Type.

We calculated Task Type decoding indices (Materials and Methods, and Figs. 3C, Lower Left, and 6A; see Table S3 for analogous results in terms of percent correct) and entered them into a two-way ANOVA with ROI and Hemisphere as within-subject factors. Confirming the patterns apparent in the MDS plots, this ANOVA revealed only a significant main effect of ROI [F(3, 54) = 6.09, P < 0.004], reflecting greater Task Type decoding in all regions relative to the EVC (all t > 2.52, P < 0.01). Both the pFs and LPFC showed significant Task Type decoding (all t > 2.26, P < 0.02), whereas the LO and EVC carried no significant information [EVC: t(24) = −2.18, P < 0.02; note that negative decoding values reflect lack of a consistent response across tasks within a Task Type].

Fig. 6.

Task Type information. (A) Task Type (physical, conceptual) decoding across the ROIs. Any decoding index significantly greater than zero indicates that the Task Type could be decoded in the region. Significant decoding was found in all regions, except for the EVC. *P < 0.05. (B) Comparison of response magnitude for physical and conceptual tasks averaged across the individual tasks. Bars show the difference in magnitude for conceptual and physical tasks with blue bars indicating stronger responses to physical tasks and red bars stronger response to conceptual tasks. In the EVC and LO, the physical tasks elicited a higher response relative to the conceptual tasks, but in pFs the reverse was true. In the LPFC, the effect of Task Type interacted with Hemisphere: Left hemisphere evidenced a conceptual task advantage, whereas the right hemisphere displayed a physical advantage. **P < 0.01.

Given the strong effect of Task Type in the response patterns, we next examined modulations in response magnitude by Task Type. Although the response in the EVC was stronger during the physical than conceptual tasks, the opposite was true in the pFs (Fig. 6B). Furthermore, the LPFC incorporated both effects, with the left hemisphere showing a conceptual advantage and the right hemisphere showing a physical advantage. A three-way ANOVA with ROI, Hemisphere, and Task Type as within-subject factors revealed a significant ROI × Hemisphere × Task Type interaction [F(3, 54) = 9.93, P < 0.0001].

To establish the source of this three-way interaction, we then ran separate two-way ANOVAs for each ROI with Task Type and Hemisphere as factors. These analyses revealed a main effect of Task Type in the EVC and LO, with greater response in the physical relative to the conceptual tasks [EVC: F(1, 24) = 10.85, P < 0.003, LO: F(1, 21) = 4.30, P < 0.05]. Object-selective pFs showed an opposite pattern with greater response for the conceptual than physical tasks, [F(1, 21) = 7.20, P < 0.02]. Notably, there were no significant interactions with Hemisphere (all F < 4.02, P > 0.06) in any visual area. In contrast to the visual areas, the LPFC showed a Task Type × Hemisphere interaction [F(1, 24) = 29.23, P < 0.0001], reflecting a stronger conceptual then physical response in the left hemisphere [t(24) = 2.70, P < 0.01] with the opposite pattern in the right [t(24) = 2.40, P < 0.02]. This differential effect of Task Type across the ROIs argues against any general effect of difficulty or attention between the physical and conceptual tasks.

Having shown that Task Type plays a key role in driving the responses to visual stimuli, we next evaluated whether the patterns of response within each ROI contained information about the specific task.

Individual Task.

Given the effects of Task Type, we evaluated decoding within each Task Type separately (i.e., physical: fixation vs. color vs. tilt; conceptual: content vs. size vs. movement). Specifically, we calculated individual task-decoding indices by subtracting across-task from within-task correlations (Fig. 3C, Lower Right) separately for each Task Type (Fig. 7) (see Materials and Methods for further details).

Fig. 7.

Individual task decoding. Decoding of individual tasks within each Task Type. Any decoding index significantly greater than zero indicates that the task could be decoded from the others within the same Task Type. Schematic matrices illustrate the specific comparisons made to compute the indices. (A) Decoding of individual physical tasks. (B) Decoding of individual physical tasks (with fixation task removed from analysis). (C) Decoding of individual conceptual tasks. *P < 0.05.

Individual physical tasks could be decoded from one another in all ROIs (all t > 3.49, P < 0.002), with strongest decoding in the EVC and LO (Fig. 7A). A two-way ANOVA with ROI and Hemisphere as within-subject factors revealed a significant main effect of ROI [F(3, 54) = 13.20, P < 0.0001] and a significant ROI × Hemisphere interaction. Pairwise comparisons between ROIs revealed a posterior-anterior gradient: physical task decoding was stronger in the EVC than the other ROIs (all t > 4.05, P < 0.0001), and the LO was stronger than the other pFs and LPFC (all t > 2.75, P < 0.01), which, in turn, did not differ from each other (P < 0.21). Whereas there was equivalent decoding in the EVC and LPFC across hemispheres (all t < 1.64, P > 0.11), in the object-selective cortex (LO and pFs) decoding was stronger in the left (all t > 2.08, P < 0.05).

To evaluate physical task decoding absent the strong effects of the fixation task in the early visual areas (Fig. 5), we recalculated the physical task-decoding indices excluding this task. This process results in significant (all t > 2.60, P < 0.02) and equivalent decoding in all ROIs (Fig. 7B), as shown by a two-way ANOVA with ROI and Hemisphere as within-subject factors revealing no significant main effect of ROI [F(3, 54) < 1.00]. There was a significant ROI × Hemisphere interaction [F(3, 54) = P < 0.02], stemming from stronger decoding in the left hemisphere in the pFs and LPFC (all t > 2.74, P < 0.01), but not in the EVC and LO (all t < 1.76, P > 0.09). The lack of a gradient with the removal of the fixation task highlights the potential involvement of spatial attention in the fixation task, consistent with previous reports of strong spatial attention effects in early visual areas (for reviews, see refs. 36 and 37).

Finally, decoding of individual conceptual tasks was significant only in the LPFC [t(24) = 3.14, P < 0.004; all other t < 0.76, P > 0.45]. A two-way ANOVA with ROI and Hemisphere as factors showed no effects of either factor on the strength of conceptual task decoding (all F < 1.07).

To assess whether the strength of task decoding reflected individual differences in task performed, we compared the mean pair-wise decoding of tasks with the corresponding mean pair-wise differences in reaction time (SI Text). There were no significant correlations within any of the ROIs, suggesting that decoding does not simply reflect individual differences in task difficulty.

In sum, these analyses of task information demonstrate that task plays a major role in driving the response patterns within the EVC, the ventral visual pathway, and the LPFC. Task Type could be decoded from response patterns in the LPFC and pFs, but not the LO or EVC. Furthermore, individual tasks could be decoded, and the physical tasks could be decoded even with the fixation task excluded.

Parietal Cortex.

The main goal of the current work was to determine how top-down signals impact object representations along the ventral visual pathway. However, the parietal cortex is also known to flexibly change its activity according to task demands (38–40), and thus it has been suggested to represent the behavioral relevance or prioritization of planned actions and goals (41–44). To evaluate how the parietal cortex represents the current task manipulations, we localized the parietal cortex (similar to the LPFC; see Materials and Methods) and conducted the same analyses reported above. Multivariate information analysis revealed that task has a profound effect on the patterns of response in the parietal cortex. In particular, both the type of task and the individual task could be decoded from the patterns of response in the parietal cortex (Fig. 8A). The MDS showed a clear separation between physical and conceptual tasks, as well as between individual physical and conceptual tasks (Fig. 8B). Task Type decoding was highly significant [mean = 0.018, SEM = 0.004; t(24) = 4.26, P < 0.0001]. Furthermore, decoding of individual physical tasks (with and without the fixation task: mean = 0.036, SEM = 0.007; mean = 0.036, SEM = 0.007, respectively, all ts > 4.93, P < 0.0001) and individual conceptual tasks [mean = 0.004, SEM = 0.001; t(24) = 3.69, P < 0.001] were both highly significant. Task also had a significant effect on the response magnitude of the parietal cortex [F(5, 120) = 10.17, P < 0.0001)] (Fig. 8C), with the fixation task eliciting higher activation than all other tasks (paired t tests; all t > 2.49, P < 0.02) and the tilt task eliciting significantly higher response relative to the conceptual tasks (paired t tests; all t > 2.63, P < 0.02). Collapsing across Task Type, the physical tasks elicited significantly higher activations than the conceptual tasks [F(1, 24) = 20.22, P < 0.0001], with no significant interaction with hemisphere [F(1, 24) = 3.22, P > 0.08].

Fig. 8.

Task and object representations in the parietal cortex. (A) Raw similarity matrix for the parietal ROI averaged across all participants. As in Fig. 4, the matrix is 48 × 48 cells (6 tasks × 8 objects), with each cell reflecting the correlation between a pair of conditions across two independent halves of the data (color-coded according the normalized correlation values). Solid lines denote borders between the two Task Types (physical, conceptual) and dashed lines denote borders between individual tasks. Note the strong grouping by both Task Type and by individual tasks within each Task Type. (B) MDS plot highlighting the relationship between responses pattern for each task. Note the clear separation between the physical (blue) and conceptual (red) tasks as a whole, as well as a clear separation within each type of task between the individual tasks. (C) Average response magnitude for the six tasks (physical tasks, blue; conceptual tasks, red) averaged across hemispheres. *P < 0.05, Main effect of Task, assessed by a Hemisphere × ROI ANOVA. (D) Object decoding indices within- and across-task (green and purple bars, respectively). Within- and across-task object-decoding indices were significantly above chance but there was no difference in decoding, suggesting task-independent object representations in parietal cortex, even though it contains rich task information. *P < 0.05.

Critically, despite its rich task representations, object representations in the parietal cortex were found to be task-independent (Fig. 8D), as there was no significant difference in the strength of object decoding within and between task [t(24) = 0.63, P > 0.50]. Consistent with prior studies highlighting the presence of object representations in the parietal cortex (45–47), significant object information was present both within and between task (all t > 2.29, P < 0.03). Thus, the present results are consistent with prior work highlighting the role of the parietal cortex in representing behavioral relevance. Notably, however, task does not impact the object information contained in this region.

Discussion

In the present study we demonstrated that object information in the ventral object-selective and LFPC is significantly reduced across tasks compared with within task, indicating task-dependent neural representations of objects. In contrast, object information in earlier visual areas was task-independent. Furthermore, the high-level distinction between the physical and conceptual tasks could be decoded from the object response of the ventral object-selective and prefrontal cortex. The specific type of physical task being performed could be decoded in all regions, including the EVC, but the specific conceptual task in the LPFC only. These findings suggest that object processing is highly influenced by the behavioral goal of the observer, to the extent that the response to identical objects is strongly determined by the task context under which those objects appear.

Our results contrast with the common characterization of the ventral visual pathway as a feed-forward hierarchical processing stream that builds complex representations of visual objects in a primarily bottom-up fashion (9, 48). Although these frameworks do not reject the integration of top-down and bottom-up signals, they propose that the primary neural locus of the integration is extrinsic to the pathway [e.g., the LPFC (13–17)], with largely task-independent representations within the pathway itself. However, our findings demonstrate that the neural representations of objects in the ventral temporal cortex are strongly influenced by the behavioral goals of the observer. Outside the EVC and LO, these representations reflect a constant and ongoing interaction between sensory input and high-level behavioral goals (see also refs. 1, 8, and 49–52 for related proposals).

Within the ventral visual pathway, we found that the pFs evidenced stronger effects of behavioral goal on object representations than the LO, although both were defined by the same contrast. This pattern of results is consistent with the strong engagement of the pFs cortex in the processing of high-level visual object properties (53, 54), visual imagery (35, 55), and visual working memory (32, 56), as well as with the characterization of the LO as representing lower-level visual object properties (48, 57). Furthermore, this functional distinction may reflect the greater density of connections in the monkey between the ventral than lateral surfaces of inferotemporal cortex and medial temporal lobe structures involved in long-term memory (8). Further research is needed to elaborate the differential mechanisms underlying object representation in the lateral and ventral object-selective cortex.

Our results highlight the need to understand the exact contribution of the LPFC to visual object representations. In the present study, the LPFC evidenced object decoding, but only within-task, and a strong general sensitivity to behavioral goal. Furthermore, only the LPFC but not visual regions evidenced significant decoding of the individual conceptual tasks. These results suggest that there are selective visual object representations in the LPFC that are very strongly constrained by task (17, 58). Interestingly, the LPFC also evidenced a laterality difference in magnitude with greater responses for physical than conceptual tasks in the right hemisphere and the opposite pattern in the left (Fig. 6B). These findings support previous reports of hemispheric specialization in the LPFC (59–64). The conceptual bias we observed in the left LPFC is consistent with its characterization as a semantic executive system, accessing long-term conceptual knowledge in a task-specific manner (65, 66; for a review, see ref. 67). The physical advantage in the right resonates well with its proposed role in processing of nonverbal content (68–70).

To reach the present conclusions we relied on a design that allowed a direct quantification of the impact of task on object information that goes beyond the previous studies (for a related discussion, see refs. 31 and 32). Specifically, many studies (24–30) treated task or object to a simple dichotomous variable (e.g., attended vs. unattended, faces vs. scenes) as well as averaging over the pattern of response, making it impossible to assess object information. In our study, we treated each combination of task and object individually and measured information to precisely quantify the interaction between bottom-up and top-down signals within each region. The differential sensitivity of this approach compared with a standard approach is apparent within our own results: whereas the pFs showed little modulation of average activation by task but large effects on object information, the EVC evidenced the opposite pattern. Although average activation and multivariate measures of information are certainly not orthogonal, these results indicate that they can reveal complementary aspects of the neural response (see also ref. 71).

Our demonstration of a fundamental impact of task on object representations even within the ventral visual pathway raises important issues for the design of future studies. In particular, it is clear that generalizing findings from one task to another is not straightforward, although it is an implicit assumption in the literature. Great care must be taken in the interpretation of results from any single task, especially complex tasks that require a very specialized type of information to be extracted from the stimuli. At the same time, it is important to note that in all visual regions it was still possible to decode object identity across tasks, suggesting that although representations are perturbed, they are not completely changed. Future work, involving a wider array of objects and tasks, is needed to establish exactly how much the representations are perturbed within these regions and what generalization is reasonable between tasks. This work could also more firmly establish the exact perturbation caused by each task: for example, by demonstrating predictable clustering of the stimuli based on the features of interest.

Conclusion

In sum, we demonstrate that behavioral goals strongly perturb object representations in the ventral object-selective and prefrontal cortex, producing reduced object decoding across tasks. Moreover, both the individual task and type of task can be decoded from visual response patterns in these regions. Even within the EVC, the initial stage of visual processing, the task modulates the response. Taken together, these findings suggest that behavioral goals directly impact visual representations within the ventral visual pathway, strongly supporting a recurrent interactive view of visual object processing, and suggest that top-down signals play a major role at all levels of visual processing culminating in our complex subjective perceptual experience.

Materials and Methods

Participants.

Twenty-five participants (10 males) aged 19–37 y participated in the experiment. All participants had normal or corrected-to-normal vision and gave written informed consent. The consent and protocol were approved by the National Institutes of Health Institutional Review Board.

Stimuli and Design.

To test the influence of task on cortical object processing, we designed an experimental paradigm that would enable the simultaneous and independent manipulation of both task and object information on a trial-by-trial basis. In every trial, participants saw an image containing an object, and were required to make a judgment regarding a certain attribute of the image. Before the presentation of the object, the specific task-relevant attribute was presented in an instruction screen (Fig. 1A). Forty-eight unique objects were presented in the experiment. These objects consisted of six exemplars from eight object categories (butterflies, cows, dressers, flowers, motorbikes, roller skates, trees, and vases). These object categories were selected to span three conceptual dimensions: (i) Naturalness; is the object manmade or natural? (ii) Movement: is the object mobile or static? (iii) Size: is the object bigger or smaller than a stove? In addition, the objects were systematically manipulated to vary in their physical appearance: the objects were slightly tilted from their principal axis to the right or to the left, either had a blue or a red outline, and the fixation cross changed either in the length or height of the horizontal and vertical bars, respectively. The eight combinations of these three physical dimensions were equally distributed across the stimuli, so that each object category contained an even number of each one of the physical attributes. In sum, each presentation of an object contained three conceptual dimensions and the three physical dimensions. These six dimensions formed the basis for the six tasks used in the experiment: (i) Fixation: is the fixation cross changing in height or width? (ii) Color: is the outline of the object colored red or blue? (iii) Tilt: is the object tilted clockwise or counterclockwise? (iv) Content: is the object natural or man-made? (v) Movement: is the object typically mobile or stationary? (vi) Size: would the object fit in an oven or not? Each of the forty-eight objects was presented in each task, allowing us to assess how the representations of the same set of stimuli vary under different tasks.

Each trial had three components: (i) cue, indicating the task to be performed; (ii) object image; and (iii) response screen. To estimate the neural response to the object distinct from the activity driven by the cue and response, these components were separated by variable interstimulus intervals (ISIs) (Fig. 1B). Furthermore, to avoid effects of response planning on the object response, participants did not know which button was associated with the possible responses until the onset of the response screen. The experiment consisted of six runs, each run composed of 48 unique (exemplar-by-task) trials, giving a total of 288 individual trials. The order of the trials was pseudorandomized across participants.

Event-Related fMRI Experiment.

A trial commenced with the presentation of a cue that specified the task the participant was to perform on the object in the form of one of the six task names presented for 1 s at the center of the screen. The cue was replaced by a uniform black screen, presented for a variable ISI ranging from 3,000 to 5,000 ms, followed by the presentation of the stimulus for 500 ms. All objects were presented in grayscale on a black background at the center of the screen (subtending ∼5° of visual angle) with a superimposed white fixation cross. Following the offset of the stimulus, a uniform black screen appeared again for a variable ISI of 3,000–5,000 ms. Next, a response screen appeared, indicating the two response alternatives relevant for the particular task (color: red/blue; fixation: wide/tall; tilt: positive/negative; content: man-made/natural; movement: still/moves; size: big/small). The response alternatives were presented simultaneously, each alternative presented on a separate half of the screen, indicating which button on the response box (left or right) needed to be pressed. The button associated with a particular response appeared was counter-balanced across the trials. The response alternatives appeared on the screen for 500 ms, followed by a blank gray screen presented for an intertrial interval of 1,500 ms. The overall length of each trial was 12 s. A 12-s fixation block was added to the beginning and the end of each run, culminating in a total run length of 10 min.

fMRI Localizer Experiment.

Independent block-design scans were collected in each participant to localize object-selective regions (LO and pFs) and the EVC. Each of these scans was an on/off design with alternating blocks of different types of stimuli presented while participants performed a one-back task. The LO and pFs were defined by the contrast of objects versus retinotopically matched scrambled objects (see also ref. 71). Object images were grayscale photographs (5 × 3°) of objects from the same categories used in the main experiment. Importantly, all images used in the localizer runs were from an independent stimulus set not used in the event-related experiment. The EVC was localized using a simple contrast of central and peripheral flickering (8 Hz) checkerboards sized less or more than 5° of visual angle, respectively, allowing us to identify retinotopic voxels that showed a greater preference for the center relative to the periphery of the visual field. We also defined a V1 ROI using the method developed by Hinds et al. (72) to identify V1, producing qualitatively similar results.

fMRI Scanning Parameters.

Participants were scanned on a research dedicated GE 3-Tesla Signa scanner located in the Clinical Research Center on the National Institutes of Health campus in Bethesda, MD. Whole-brain volumes were acquired using an eight-channel head coil (30 axial slices, 3 × 3 × 3 mm, 0.3-mm interslice gap, TR for the event-related scans = 1.5 s, TR for the localizer scans = 2 s, TE = 30 ms, matrix size = 64 × 64, FOV = 192 mm). Six event-related runs (400 TRs each) and a minimum of two localizer scans (144 TRs) were acquired in each session. In addition, high-resolution MPRAGE (magnetization prepared rapid gradient echo) anatomical volumes were collected for each participant at the beginning of each scan session.

fMRI Preprocessing.

Data were analyzed using the AFNI software package (http://afni.nimh.nih.gov/afni) and custom Matlab (2007, Mathworks) scripts. Before statistical analysis, all of the images for each participant were motion-corrected to the eighth image of their first run. Following motion correction, the event-related and the localizer runs were smoothed with a 5-mm full-width half-maximum Gaussian kernel.

fMRI Statistical Analysis.

Functional ROIs were created for each participant from the two localizer runs. Significance maps of the brain were computed by performing a correlation analysis between the assumed hemodynamic response function and the activation time courses thresholded at P < 0.0001 (uncorrected). ROIs were generated from these maps by taking the contiguous clusters of voxels that exceeded threshold and occupied the appropriate anatomical location based on previous work (33, 34). To more precisely define the LO cortex, we excluded retinotopic voxels from the ROIs (in five of the participants, one of the ROIs could not be localized: the right LO in three participants, left LO and right pFs in two different participants).

To localize the LPFC we used a joint anatomical-functional ROI defined as follows. Cortical reconstruction was performed with Freesurfer (http://surfer.nmr.mgh.harvard.edu/) and SUMA (http://afni.nimh.nih.gov/afni/suma/) software packages. Using FreeSurfer’s automatic cortical parcellation algorithm (72, 73), we identified in each participant a region corresponding to the inferior frontal gyrus pars opercularis, and rostral and caudal middle frontal in the LPFC. To restrict this large anatomical ROI, we constrained the ROI to only those voxels that showed significant activation during the presentation of the objects in the event-related experiment. To do this, we calculated which voxels within the anatomical LPFC ROI that showed significant average activation during the object presentations in each possible split of the event-related data. We then extracted data from those voxels in the other half of the data to assure independence. A similar procedure was also applied to localize a parietal ROI. We identified in each participant a region encompassing the supramarginal gyrus, inferior- and superior- parietal cortex, using Freesurfer’s cortical parcellation, and then further restricted this large ROI by selecting only the visually active voxels.

We conducted a standard general linear model using the AFNI software package with a γ-fit to extract the event-related responses for each voxel within the predefined ROI. β-Parameters were extracted for all individual voxels within a given ROI and used as estimates of the magnitude of the ROI’s response to the different conditions. The β-estimates were averaged across the ROI and the 10 splits of the data (see below), and were later subjected to omnibus ANOVAs (all reported P values throughout the article are Greenhouse–Giesser corrected. For simplicity, we present the uncorrected degrees of freedom). We report only the significant effects, unless the absence of the effect is particularly relevant for the theoretical question asked.

Multivoxel response patterns across each ROI from the six event-related runs were analyzed using an iterative version of the split-half analysis method, a form of linear classifier (74). Specifically, the six runs were divided into two separate datasets of three runs in all 10 possible ways (₆C₃/2). For each half of the data in each of the 10 splits, significance maps were created by performing t tests between each condition and baseline. The t values for each condition were then extracted from the voxels within each ROI and the mean t value subtracted from the resulting vector of condition responses in each voxel (33, 34, 74). Note that the subtraction of the mean had no qualitative effect on either the object of task decoding indices (Fig. S2). We then cross-correlated across the halves of each split. The correlation values were averaged across the 10 splits, resulting in a single 48 × 48 (6 tasks × 8 object categories) similarity matrix for each ROI, wherein each datapoint in the matrix represented the correlation of the pattern of response for a pair of conditions across the two halves of the data (Fig. 3). These correlation values were either within a given condition, reflecting the consistency of response across splits of the data (e.g., butterfly in the size task vs. butterfly in the size task) or between a pair of different conditions (e.g., butterfly in the size task vs. dresser in the movement task), reflecting the similarity between the patterns of response to the two conditions.

Object decoding.

We investigated object decoding in each ROI both within and across tasks. Comparing the strength of object decoding within and across tasks provides a measure of the dependence of object representations on task. Within-task object decoding was computed from the full matrices by subtracting the average correlation between the different objects within each task from the average correlation between objects and themselves (main diagonal) (Fig. 3C, Upper Left). This subtraction resulted in an index of within-task object identity decoding, representing the amount of object information available in a particular task context. To measure the across-task object decoding we computed the average correlations between the patterns of response to objects and themselves across tasks and subtracted the average correlation between different objects across tasks (Fig. 3, Upper Right). The resulting between-task object-decoding indices represent the amount of object information available across task context: that is, object information that is task-independent within a particular ROI.

Task Type decoding.

To quantify how well can individual tasks be decoded both within each type of task and between the two types, the full similarity matrices in each ROI were averaged by task to produce a 6 × 6 matrix (Fig. 3B). Importantly, to remove any effect of the individual objects, the correlations between an object and itself in the same task (main diagonal) and across tasks were removed before averaging. The main diagonal of these averaged task matrices represent the average within-task correlation between different objects. The off-diagonals are the average between-tasks correlation between different objects. To formally assess how well distinguished the physical tasks are from the conceptual tasks, we derived a Task Type decoding index by calculating the coherence within each Task Type (the average between-task correlation within each Task Type) and subtracting the average correlation between tasks of different types. A value significantly greater than zero indicates that the response patterns in the region can be reliably used to decode the type of task.

Individual task decoding.

To establish whether individual tasks can be discriminated from the response patterns within the physical and conceptual tasks, we computed a task-decoding index for each of the three tasks within each Task Type by subtracting the average between-task correlations from the within-task correlations within each Task Type (Fig. 3C, Lower Right). This process resulted in task-decoding indices for each of the six tasks, calculated for each region across all participants.