Understanding Human Original Actions Directed at Real-World Goals: The Role of the Lateral Prefrontal Cortex

Abstract

Adaptive, original actions, which can succeed in multiple contextual situations, require understanding of what is relevant to a goal. Recognizing what is relevant may also help in predicting kinematics of observed, original actions. During action observation, comparisons between sensory input and expected action kinematics have been argued critical to accurate goal inference. Experimental studies with laboratory tasks, both in humans and nonhuman primates, demonstrated that the lateral prefrontal cortex (LPFC) can learn, hierarchically organize, and use goal-relevant information. To determine whether this LPFC capacity is generalizable to real-world cognition, we recorded functional magnetic resonance imaging (fMRI) data in the human brain during comprehension of original and usual object-directed actions embedded in video-depictions of real-life behaviors. We hypothesized that LPFC will contribute to forming goal-relevant representations necessary for kinematic predictions of original actions. Additionally, resting-state fMRI was employed to examine functional connectivity between the brain regions delineated in the video fMRI experiment. According to behavioral data, original videos could be understood by identifying elements relevant to real-life goals at different levels of abstraction. Patterns of enhanced activity in four regions in the left LPFC, evoked by original, relative to usual, video scenes, were consistent with previous neuroimaging findings on representing abstract and concrete stimuli dimensions relevant to laboratory goals. In the anterior left LPFC, the activity increased selectively when representations of broad classes of objects and actions, which could achieve the perceived overall behavioral goal, were likely to bias kinematic predictions of original actions. In contrast, in the more posterior regions, the activity increased even when concrete properties of the target object were more likely to bias the kinematic prediction. Functional connectivity was observed between contiguous regions along the rostro-caudal LPFC axis, but not between the regions that were not immediately adjacent. These findings generalize the representational hierarchy account of LPFC function to diverse core principles that can govern both production and comprehension of flexible real-life behavior.

1. Introduction

In a scene from the motion picture, ‘the Da Vinci Code’, the cargo hold door of an armored truck opens and the protagonists, Langdon and Neveu, find themselves facing a gunpoint (Howard, 2006). The gun is fired, and the spent shell of a bullet lands on the floor. Langdon deliberately kicks this small piece of steel into the door frame. The captor got what he wanted, but is having difficulties resealing the truck doors. Seizing the opportunity, Langdon and Neveu escape.

A keyword string, “original action sequences”, typed into the Google search engine, brings up dozens of motion picture reviews attesting that moviegoers value “brain-teaser” experiences evoked when they attempt to comprehend the adaptive purpose of actions in the real-world context. In the example above, viewers might appreciate that the shell of a bullet, which was cleverly placed into the door frame, interfered with alignment and sealing of the doors. The present study investigated the neurocognitive mechanisms engaged by viewing “brain teaser” events. The implications of this research go beyond simulated real-life experiences in motion pictures, extending to a range of social contexts, including national security. On 9/11/2001, hijacked airplanes were used as suicide bombs. Optimizing the performance of intelligence analysts may ensure that any original type of attack is foreseen.

1.1 Predictive Coding Hypothesis

In part, a theoretical framework for the current study comes from prior research on recognition of everyday actions. Based on evidence that a visual representation of the target movement is formed in motor control, and that some neurons, termed 'mirror' neurons, are engaged during both action execution and observation, it was suggested that action recognition may be possible by inverting neural computations of motor control (Wolpert et al., 1995; Gallese et al., 1996; Umilta et al., 2001; Rizzolatti and Craighero, 2004; Kilner et al., 2007b; Kilner et al., 2009; Mukamel et al., 2010). Because the same kinematic pattern might be used to accomplish more than one goal, it was argued that observers must use preexisting expectations about likely goals (Kilner et al., 2007b). The mirror neurons may encode concrete attributes of actions constrained by goal priors generated elsewhere, for instance, via process of semantic retrieval of usual goals associated with objects or contextual situations (Iacoboni et al., 2005; Grafton and Hamilton, 2007; Ortigue et al., 2010; Kilner, 2011; Press et al., 2012). Self-organizing process of updating goal priors to minimize the error between the expected and perceived kinematics may lead to action comprehension (Kilner et al., 2007a).

The logic of this ‘predictive coding’ hypothesis is applicable to original actions. In Langdon’s escape example, the kinematic pattern of kicking was consistent with several intentions, such as to hide the shell out of sight, or remove refuse from the cargo hold. It was the preceding movie context that, by communicating Langdon’s overall goal to exit the cargo hold before the doors are locked, helped the audience to rapidly select the appropriate interpretation. Nonetheless, this account leaves gaps in our understanding. Knowing which actions and objects could usually accomplish real-life goals may be insufficient for kinematic predictions of the original actions that are constructed by the actor to adaptively achieve goals. For instance, a bullet shell is not associated with jamming a door, or exiting a truck cargo hold. If actions are original, how could they be predicted?

1.2 Goal Relevance and the Lateral Prefrontal Cortex

Two additional lines of research give clues with regard to the above question. First, extensive literature has demonstrated that, during language comprehension, the verbal input is transformed into spatiotemporal mental models of perceived events (Johnson-Laird, 1983; Fischer and Zwaan, 2008; Barsalou, 2009). Several experiments suggested that such representations for original object-directed actions could be generated by integrating likely goals with the goal-relevant properties of target objects (Glenberg and Robertson, 2000; Kaschak and Glenberg, 2000).

The other line of investigation has documented that the lateral prefrontal cortex (LPFC) can selectively encode goal-relevant features of stimuli, and apply these representations to novel contexts. Single cell recordings in non-human primates and the multivariate pattern analysis of functional magnetic resonance imaging (fMRI) data in the human brain showed that, following training in categorical decisions, activity in the LPFC reflected task-relevant properties of untrained stimuli, and was predicted by behavioral expertise gained in a multi-day training (Freedman et al., 2001, 2002, 2003; Moore et al., 2006; Li et al., 2007; Cole et al., 2011). Computational network models of the LPFC circuits demonstrated that goal-relevant information could be encoded by selectively updating active representations, based on success or failure in a task (O'Reilly et al., 2002; O'Reilly et al., 2010). Interestingly, distinct regions in the human LPFC may encode different levels of hierarchically-organized, goal-relevant information. FMRI activations were anatomically organized during reinforcement-based training to respond to hierarchically-structured stimulus contingencies (Badre et al., 2010). When task goals were defined at multiple nested levels, progressively more anterior regions along the LPFC rostro-caudal axis were activated as increasingly more global goals were controlled by stimulus contingencies (Koechlin et al., 2003; this activity was left lateralized in Badre and D'Esposito, 2007). Self-organization of goal representations into functional hierarchies was documented in a neural model that modulated speed of updating activity of its units (Yamashita and Tani, 2008).

Importantly, in the above studies, the LPFC capacity to encode goal relevance was observed when the performed task involved complex visual shapes, real-life objects, spatiotemporal kinematic patterns, and/or more than one relevant stimulus dimensions (e.g., spatial, temporal;Freedman et al., 2001, 2002, 2003; Moore et al., 2006; Li et al., 2007; Cole et al., 2011). Taken together, these findings suggested that neurobiological mechanisms involving LPFC may be capable of learning and utilizing hierarchically-organized, goal-relevant dimensions of multifaceted real-world behaviors. Through experience with diverse real-world behaviors, which succeeded or failed in achieving their goals, these mechanisms may identify goal-relevant elements. More abstract representations relevant for larger goals would not be updated at shifts between smaller sub-goals (see Reynolds and O'Reilly, 2009), or different experiences with the same global goal. The resulting hierarchies of goal-relevant representations would facilitate design of novel combinations between actions and objects that are able to adaptively reach behavioral goals.

The goal-relevant representations could guide both motor output during execution and kinematic prediction during comprehension of original actions. In Langdon’s example, concrete representations for guiding his motor output could be selected based on the abstract goal constraint (an entity with the sufficient volume, secured inside a door frame, would keep the door unlocked) and the relevant features of the bullet shell (solid, easy to manipulate). s Langdon’s foot approached the bullet shell on the movie screen, analogous representations could be constructed by the audience to guide the kinematic prediction for Langdon’s unfolding action.

1.3 Current Event-Related fMRI Study

Thus, prior evidence suggested that, during observation of real-world behaviors, hierarchical representations, formed within LPFC, and integrating a likely overall goal with relevant properties of engaged objects, may guide kinematic predictions of original object-directed actions. When the observed kinematic pattern unfolds, updating of the a priori goal hierarchies at all levels to minimize error between the predictions and sensory input would lead to action comprehension. Despite such intriguing suppositions, the previous experimental findings allow limited generalization to neurocognitive processing of original actions in real life. This research focused either on cognitive mechanisms during language comprehension or neural mechanisms encoding stimulus properties relevant to laboratory goals (e.g., classifying objects with regard to a single set of rules). The current study employed event-related fMRI to directly examine the LPFC function during comprehension of naturalistic video-depictions of real-life events.

Goal-directed behavioral sequences were designed to culminate in an object-directed action that would achieve their overall goal. The video-depictions of these behaviors were produced in pairs that shared the same context, but had either usual or original ending. The usual videos were included to confirm that mere content differences between the conveyed situations could not account for results. Kinematic predictions of the usual completions were possible by retrieving the action associated with both the context and the target object presented in the final scene. In contrast, the target object of the original final action was not associated with the scenario context. We hypothesized that lack of convergence between the actions associated with the context and those associated with the target object would trigger LPFC computations to identify alternative action representations necessary for a kinematic prediction. The formed goal-relevant representations were predicted to remain active, while they were selected and fine-tuned, leading to an enhanced LPFC activity during observation of original, relative to usual, actions.

Properties of the target object presented in an original movie completion were potentially relevant to multiple goals. The abstract representations, constrained by the overall goal of the contextual behavior, were expected to facilitate comprehension of the original endings by limiting the relevant properties of objects, and giving rise to a unique kinematic prediction. To test the hypothesis that more abstract goal representations will be supported within the anterior LPFC, we ensured that original actions varied in their interpretability as targeting the overall goal conveyed in the context. We reasoned that, if the abstract properties selected by the overall goal mismatched the object or kinematics observed in the original ending, the initial interpretation would have to be rejected. Importantly, the contextual behavior of actors clearly communicated a single overall goal, and was unlikely to be construed as aimed at an alternative overall goal. Therefore, abstract representations were more likely to remain active and be fine-tuned, leading to a larger activity increase in the anterior LPFC, when viewing the original actions that were more rather than less interpretable as completing the contextual goal.

In the posterior LPFC, an enhanced activity was predicted to all original actions. When the abstract constraint limited concrete representations within this region, an additional activity increase was expected due to processing demand of signal arriving from the anterior LPFC, as the representational levels interacted during fine-tuning. On the other hand, if the abstract representation of the initially perceived overall goal was rejected, an activity increase in the posterior LPFC was expected due to selection between multiple sets of relevant concrete representations, giving rise to multiple kinematic predictions (for similar logic, see Badre and D'Esposito, 2007).

1.4 Functional Networks Involving LPFC and Comprehension of Original Actions

Neuroanatomical evidence in the rhesus monkey is consistent with organization of functional neural networks involving LPFC according to two principles (Badre and D'Esposito, 2009). First, adjacent areas along the LPFC rostro-caudal axis might be interconnected, but no projections might be present to the more rostral regions that are not immediately adjacent. Second, sensory inputs from the regions outside of LPFC may be more integrative to the anterior than posterior LPFC. Such neuroanatomical architecture would be optimal for coding hierarchically-organized, goal-relevant information. General features of situations and unfolding actions, relayed to the anterior LPFC, would be sufficient for updating abstract representations, applicable to a broad range of contexts (Badre et al., 2010). Progressively more concrete representations within the more posterior LPFC would be constrained by both abstract representations via rostro-caudal contiguous connections, and detailed inputs about objects, actions, and contextual situations from outside of LPFC.

In the human brain, fMRI data recorded during ‘task-free’, resting state has been used to identify discrete functional networks, indicative of information-encapsulated, specialized processes, that may interact with each other via a superordinate network of profusely connected regions (“a rich club of hubs” -- Biswal et al., 1995; Fox and Raichle, 2007; Bullmore and Sporns, 2009; Yeo et al., 2011; Crossley et al., 2013; van den Heuvel and Sporns, 2013). The predictive coding mechanism, hypothesized to mediate comprehension of original actions, arguably demands engaging conscious effort to form a globally integrated system, sharing information across several specialized neural modules. Nonetheless, patterns of inter-modular functional connectivity in the human brain that would afford such processing have not yet been characterized.

1.5 Current Functional Connectivity Study

The present study employed resting-state fMRI data to test two hypotheses. First, we focused on the LPFC network activated during comprehension of original actions in videos. Within this network, functional connections, identified based on the resting-state data, were expected between contiguous regions along the rostro-caudal gradient, but not between the regions that are not immediately adjacent. Second, we aimed to identify an ensemble of regions outside of LPFC that supported comprehension of different types of original actions in videos. We predicted that functional connections between these regions and LPFC will reflect the level of the hierarchically-organized goal-relevant code that they contribute to; more integrated inputs from outside of LPFC will correspond to more anterior LPFC regions.

2. Materials and Methods

2.1 Participants

Sixteen participants (seven male; aged 19-28, mean 21.5) took part in the fMRI experiment that examined brain activations during comprehension of video-depictions of goal-directed activities. Pretest behavioral studies enrolled an additional group of 164 participants, who did not take part in the fMRI study (75 male; aged 18-28, mean 22). Finally, a separate group of 46 participants (23 male; aged 18-28, mean 22) underwent resting-state fMRI scans used in the functional connectivity analysis. All participants were recruited from the Boston area community, were right-handed, and had normal or corrected to normal vision. Participants gave informed consent before enrolling.

2.2 Event-related fMRI: Video comprehension experiment

2.2.1 Materials

Stimuli in this study consisted of 160 paired, color video clips depicting real-world activities. Details regarding production of videos have been previously described (Sitnikova et al., 2008). Still frames from sample videos are shown in Figure 1. All videos were 8 sec long. In each video, a 6-sec-long context showed consecutive events from a common, goal-directed activity (e.g., slicing bread, washing hands). These behavioral sequences were designed to culminate in a specific final action that would achieve the overall goal of the activity (e.g., “grabbing a cutting board and placing it on a kitchen counter, and then taking a loaf of bread out of a bread box and placing it on the cutting board” is likely to culminate in “cutting the bread with a knife”). Videos in each pair had the same contextual events but different final scenes, conveying either the expected completion of the behavioral sequence or an original action. All final scenes were 2 sec long. Each final event included an object (e.g., knife) that was not present in the movie context and was involved in an action (e.g., cutting). Novel final events included unusual object-action combinations (e.g., moving a clothes iron across the bread loaf). Segments conveying separate events were spliced together using a cinematographic ‘cut’ technique (Bobker and Marinis, 1973).

Figure 1.

Frames taken from movie clips used in the study. A pair of videos from the abstract goal-constraint condition had the same context (a person stands in front of a bathroom sink, turns on the faucet, uses soap, and rinses his hands), which was followed either by a usual ending (the person dries his hands on a towel – [A]) or an original ending (the person pours coffee grounds on his hand – [B]). Similarly, a pair of videos from the concrete goal-constraint condition had the same context (a person stands in front of a kitchen counter, grabs a cutting board and places it on the counter, takes a loaf of bread from a bread box, and places it onto the cutting board), which was followed either by a usual ending (the person cuts the bread with a knife – [C]) or an original ending (the person moves a clothes iron across the bread – [D]).

Several behavioral pretest studies were conducted to characterize cognitive processing that was likely evoked during comprehension of the video materials. Importantly, in these behavioral studies, stimulus presentation parameters as well as instructions given to participants, encouraging them to comprehend videos naturally and postpone the task decisions until after viewing each video clip, were comparable to those used in the fMRI experiment (see section 2.2.2 below). We confirmed that viewing the contextual videos was sufficient to accurately identify the likely overall goal and the expected final event in each goal-directed activity. 20 participants (behavioral study #1) were asked to view the 6-sec-long contextual videos and predict what the actor might do in the video ending to accomplish his or her overall goal. The majority of this participant sample guessed the same, unique overall goal, and predicted the corresponding final object-directed action for each context (percentage of participants: mean 89.44% +/− s.e.m. 10.63%). For example, after viewing an actor stand in front of a bathroom sink, turn on the faucet, scrub his hands with soap, and rinse them, most participants guessed that, to accomplish his overall goal of ‘getting his hands clean and dry’, he will next dry his hands with a towel.

For our main analysis, we aimed to divide the original actions into two conditions based on their interpretability as targeting the initially perceived overall goal of the video scenario. A group of 20 participants (behavioral study #2), who viewed the 8-sec-long videos with original endings, were asked to attribute a realistic goal to each scenario and to convey how the final action would achieve it. The videos were then subdivided with a median split on the proportion of interpretations offered for the original scenarios that matched the overall activity goal constrained by the contextual video clip (as determined in behavioral study #1). To assure test-retest reliability, we replicated this classification in a separate sample of 20 participants (behavioral study #2a). The proportion of matching goal interpretations was highly consistent across the test and retest studies (between studies #2 and #2a: pearman’s correlation = 0.950, p < 0.01; any differences between the studies were comparable between high and low goal-agreement subsets of videos: t(78) = .972, p > 0.05).

We reasoned that in the subset of videos with higher goal agreement (mean 57.91 % +/− s.e.m. 3.61% in study #2 and mean 63.56 % +/− s.e.m. 4.20 % in study #2a), the abstract constraints posed by the overall activity goal, together with the corresponding set of concrete features, must have matched the properties of the target object and kinematics of the original actions. For example, in one video from this category, an actor, standing in front of a bathroom sink, turned on the faucet, scrubbed his hands with soap, rinsed them, and then poured coffee grounds on his wet hands (see Figure 1B). Recall that the behavioral study #1 showed that the contextual events in this video communicated the overall goal of ‘getting hands clean and dry’. Participants often interpreted the original ending ‘pouring coffee grounds on the hands’ as an effort to accomplish the overall goal of ‘getting hands clean and dry’ because coffee grounds absorb water. Arguably, participants arrived at this interpretation because properties of coffee grounds (ability to absorb) and the observed kinematics matched the concrete features that were selected by the abstract constraint of the overall goal (need for any entity with the ability to move water molecules and kinematics that would allow physical contact between this entity and water molecules).

In contrast, in the subset of videos with lower goal agreement (mean 3.93 % +/− s.e.m. 0.97% in study #2 and mean 7.06 % +/− s.e.m. 1.61 % in study #2a), the abstract constraints posed by the initially perceived goal of the overall activity must have been less compatible with the original object-directed actions. For example, in the behavioral study # 1, “grabbing a cutting board and placing it on a kitchen counter, and then retrieving a loaf of bread out of a bread box and placing it on the cutting board” was consistently interpreted as directed at “cutting the bread”. Nonetheless, the abstract constraint of this overall goal (e.g., need for any object with a relatively sturdy and sharp edge and the kinematics permitting physical contact between the object’s edge and the bread) mismatched the original ending, showing the flat surface of a clothes iron being moved across the loaf (see Figure 1D). Interpretations given for these scenarios seemed driven by the concrete properties of the objects which the original actions were directed at. For example, in the behavioral studies #2 and #2a, the video ending with ‘the actor moving a clothes iron across the bread’ was frequently interpreted as efforts to heat up the bread or to make ‘flat’ bread. The concrete properties of a clothes iron, including <emits heat>, <is heavy>, <has a flat surface>, were sufficient for heating or flattening.

The two experimental conditions formed on the basis of interpretability of the original scenes as aimed at the contextual overall goal were labeled as “abstract” and “concrete” types of goal constraint. This is because the abstract constraint of the overall goal was arguably salient for comprehension of the more-interpretable, higher-agreement videos, whereas the concrete constraint of the target objects was salient for comprehension of the less-interpretable, lower-agreement videos. Video clips with usual endings (see Figure 1A and 1C) were split into two corresponding control conditions based on whether they shared the context with a novel video from the abstract or concrete goal-constraint condition. For example, the video clip in Figure 1A was assigned to the ‘control abstract” condition because it was a counterpart of the novel video in Figure 1B, which was assigned to the abstract condition.

We ensured that several potentially confounding variables, which could have influenced general comprehension difficulty, were comparable across our comparisons of interest. We obtained evidence that there were no differences in the amount of contextual integration in general between the two types of original video endings. A group of 20 participants (behavioral study #3) watched 2-sec-long videos containing only the original final events, and were asked to interpret goals of the conveyed activities. The most frequent (dominant) goal interpretations given in the behavioral study #2 based on watching the 8-sec-long videos were provided less often in response to the 2-sec videos, and such differences were comparable between the two original conditions (reductions: abstract goal constraint, mean 19.34% +/− s.e.m. 4.18%, concrete goal constraint, mean 20.16% +/− s.e.m. 2.90%; main effect of reduction: F(1,156) = 29.975, p < 0.01; reduction by condition interaction: F(1,156) = 0.013, p > 0.05). For example, the response ‘heating the bread’ was dominant for the 8-sec version of the video featured in Figure 1D. This interpretation was offered less frequently for the 2-sec version (the right pane in Figure 1D; the video ending presented with no context). Instead, interpretations such as “defrosting the bread” were offered more frequently. Thus, we found no evidence that the extent to which contextual influences shaped the interpretations of final video scenes was different between the two original conditions (the contextual information was maintained in memory and taken into consideration in both conditions – e.g., the room-temperature bread would not need to be defrosted). This helped to rule out a possibility that variables related to general contextual processing, such as working memory load, could account for fMRI results.

Furthermore, we matched the number of events included in the contextual videos (abstract, mean 3.73+/− s.e.m. 0.13, concrete, mean 3.78 +/− s.e.m. 0.17; t(78) = −0.238, p > 0.05). Accuracy in identifying the likely overall goal and the expected final event based on viewing the contextual videos was matched based on the data collected in the behavioral study #1 (percentage of responses stating the same unique overall goal and predicting the corresponding final action: abstract, mean 89.01 +/− s.e.m. 1.72%, concrete, mean 89.86 +/− s.e.m. 1.66%; t(78) = −0.355, p > 0.05). Note that similar predictability of the final scenes suggested that participants were similarly familiar with real-world situations used in different experimental conditions. In addition, we confirmed that comparable target objects were included in the abstract and concrete goal-constraint conditions in a group of 20 participants (behavioral study #5), who viewed 2-sec long novel endings. We ensured that the target object in each video could be accurately recognized (less than 2% of participants failed to correctly identify the target objects). Furthermore, there were no significant differences between the two original conditions in ratings of the target objects (on a scale 1-7) with respect to their familiarity (abstract, mean 5.97 +/− s.e.m. 0.09, concrete, mean 5.97 +/− s.e.m. 0.11; t(78) = −0.014, p > 0.05) or visual complexity (abstract, mean 3.82 +/− s.e.m. 0.14, concrete, mean 3.75 +/− s.e.m. 0.13; t(78) = 0.340, p > 0.05). Finally, to quantify comprehensibility of the final scenes, a group of 32 participants (behavioral study #4) viewed the entire 8-sec-long videos with usual and original endings (counterbalanced as described below) and rated each video (on a scale 1-5) based on how easy it was to make sense of the final video scene. Comprehensibility ratings were matched between the two types of videos with original endings (abstract, mean 1.46 +/− s.e.m. 0.06, concrete, mean 1.45 +/− s.e.m. 0.05; t(78) = .178, p > 0.05), as well as between the two corresponding subgroups of control usual videos (abstract, mean 4.81 +/− s.e.m. 0.02, concrete, mean 4.80 +/− s.e.m. 0.03; t(78) = .174, p > 0.05).

In addition to the two types of goal constraint, we used a supplementary independent variable quantifying the fit of the final original action with the initially perceived overall goal. To assess the fit, we asked participants who took part in behavioral studies #2 and #2a to also rate (on a scale 1-7) how easy it was to reconcile the video input and the goal interpretation they had selected in each scenario. Then, for each goal interpretation that matched behavioral study #1, we computed an average rating that was weighted by the percentage of participants who provided this interpretation. By examining the effects of a linear increase in this directly assessed agreement between the sensory input and the goal priors, we hoped to construct an additional, and possibly, more stringent test of the hypothesized effects of the variable engagement in abstract information processing. We reasoned that the better the fit, the longer the abstract representations relevant to the initial overall goal should be active during comprehension of the original actions. Because these weighted ratings were stable across the behavioral studies #2 and #2a (Spearman's correlation = 0.949, p < 0.01), the data was combined across the studies. We divided the videos with original endings into three subgroups based on the weighted fit (low, medium, high) for examining the effect of incremental modulation of this parameter. The usual counterparts of each original video were again used in the control condition and were divided into three corresponding subgroups (based on whether they shared the contextual situation with a novel video from the low, medium, or high fit subgroup). To rule out a possibility that potentially confounding variables, described above, might influence this analysis, we confirmed that none of them covaried significantly with the weighted fit variable (absolute values of Spearman’s correlations < 0.064; p values > .05).

Stimuli were counterbalanced using the Latin square design. All target objects (e.g., a clothes iron) used in the original video endings were also featured in the usual ending of another video (e.g., ironing pants). The contextual and final video segments were counterbalanced such that none of the participants viewed any segment more than once, but each context segment and each target object appeared in the usual and original conditions the same number of times across participants.

2.2.2 Procedure

Event-related fMRI data were recorded while participants viewed video clips. Movie clips were shown at a rate of 30 frames per second with no sound. All frames subtended approximately 4° of visual angle, and were centered on a black background. In each trial, the presentation of a movie, which lasted for 8 sec, was followed by a blank screen for 100 ms, and then a '?' prompt, which was presented in the center of the screen for 900 msec. Participants were instructed to keep their eyes in the center of the screen, but otherwise view the videos naturally (as they would watch TV at home). They were asked to watch each video carefully, and try to understand what the character(s) were doing. To ensure that participants paid attention, after viewing each video, they were asked to judge whether the presented events would commonly be witnessed in everyday life. To further encourage naturalistic comprehension, we asked them to make their decision at the '?' prompt but not sooner. Participants recorded their decisions by pressing a 'Yes' or 'No' button. Assignment of right index and middle fingers to response buttons was counterbalanced across participants. Each trial ended with a blank screen presented for 1 sec.

Before the fMRI experiment, we confirmed that difficulty of judging whether the events were common in everyday life was comparable between the goal-constraint conditions. A group of 32 participants (behavioral study #6) viewed 8-sec-long videos with usual and original endings (counterbalanced as described above) and responded as quickly as possible according to whether each video showed events that would commonly be witnessed in everyday life. Accuracy and reaction times to distinguish between original and usual video endings were matched between the two types of novel videos (accuracy: abstract, mean 92.34% +/− s.e.m. 1.62%, concrete, 89.06% +/− s.e.m. 1.99%, t(78) = 1.280, p > 0.05; reaction time: abstract, mean 1008.50 msec +/− s.e.m. 20.32 msec, concrete, mean 1043.33 msec +/− s.e.m. 23.91 msec, t(78) = −1.110, p > 0.05), as well as between the two corresponding subgroups of usual videos (accuracy: abstract, mean 88.91% +/− s.e.m. 1.64%, concrete, mean 90.62% +/− s.e.m. 1.70%, t(78) = −0.727, p > 0.05; reaction time: abstract, mean 987.14 msec +/− s.e.m. 25.04 msec, concrete, mean 963.91 msec +/− s.e.m. 24.85 msec, t(78) = 0.658, p > 0.05).

Each participant viewed 40 videos with original endings and 40 videos with usual endings. Video clips were pseudorandomly ordered and intermixed with fixation trials (presentation of a "+" symbol in the center of the screen during 33% of the session time for variable durations, ranging between 3-17sec). The random interleaving of these "fixation" or "null-events" among the video clips is critical for the efficient deconvolution of the entire event-related hemodynamic response in fMRI experiments (Burock et al., 1998). Prior to the experimental run, 10 additional movie trials intermixed with fixation periods were used in a practice block.

2.2.3 MRI data acquisition and preprocessing

MRI data were acquired on a 3T Tim Trio scanner (Siemens, Erlangen, Germany) at the Martinos Center for Biomedical Imaging. Functional scanning, sensitive to the blood oxygenation level-dependent (BOLD) changes, was divided into four runs during which T2*-weighted echo-planar images (EPI; 30 axial slices parallel to the intercommissural plane, 203 images/slice, 3-mm thickness, skip 1 mm, in-plane resolution of 3.125 mm) were acquired with a gradient echo sequence (repetition time [TR], 2 sec; echo time [TE], 25 msec; flip angle, 90°; matrix size, 64 × 64). High-resolution, three-dimensional (3D) structural images were acquired using a spoiled gradient-recalled acquisition in the steady-state (GRASS) sequence (TR, 2530 msec; TE, 3.3 msec; flip angle, 7°; matrix size, 256 × 256).

Preprocessing was conducted using standard procedures in the FreeSurfer structural and functional analysis streams (https://surfer.nmr.mgh.harvard.edu). The first four volumes of each functional run were discarded to allow for T1-equilibration effects. To correct for any motion all volumes in each run were aligned to a single volume chosen from the middle of the run using the AFNI algorithm (www.afni.nimh.nih.gov). For each participant, these middle-run volumes were then aligned to the 3D structural images using a combination of automated procedures from FSL/FLIRT (www.fmrib.ox.ac.uk/fsl) and FreeSurfer bbregister. Brain mask was created using FSL’s BET. The structural images were also used to construct two-dimensional (2D) models of individual cortical surfaces, using the FreeSurfer algorithms. Both 3D and 2D structural models were registered to an average template -- a model of the brain volume and surface based on an independent sample of 40 adults, which is registered to the Montreal Neurological Institute atlas (MNI305). Intensity-normalized functional time series were resampled to the surface-based average template, and smoothed with a 5mm full width half-maximum (FWHM) Gaussian kernel.

2.2.4 Analysis of event-related fMRI data

The functional data were analyzed under assumptions of the general linear model using the FreeSurfer functional analysis stream. Our aim was to identify voxels where [1] the hemodynamic response was different between original and usual video endings, [2] there were differences in this novelty effect between the abstract and concrete goal-constraint conditions; [3] this novelty effect was influenced by parametric modulation of the fit between the original video ending and the initially perceived overall goal. The BOLD response within a peristimulus time window (during processing of the final video scenes) at each voxel was modeled using a gamma response function. Participant-specific statistical models removed low frequency drifts in the data, and estimated a main effect of novelty (original vs. usual), an interaction between novelty (original vs. usual) and goal constraint (abstract vs. concrete), and an interaction between novelty (original vs. usual) and the fit between the original video endings and the initially perceived overall goals (parametric modulation: low, medium, high). Group statistics were computed while treating participants as a random effect and correcting for multiple comparisons using simulation testing. We determined the likelihood that an activation cluster of the observed size (p < 0.05, uncorrected, in each adjacent voxel) could occur by chance by running 10,000 Monte Carlo simulations of synthesized white Gaussian noise using the parameters of the functional analysis. To test our a priori hypotheses concerning the functional properties of LPFC, we restricted the simulations to bilateral sets of adjoining areas that were localized using the FreeSurfer parcellation algorithms and corresponded to loci of interest identified in prior studies on processing goal-relevant information (Koechlin et al., 2003; Moore et al., 2006; Badre and D'Esposito, 2007; Cole et al., 2011). These broad areas spanned over the inferior, middle, and superior frontal sulci, the middle frontal gyrus, the lateral portion of the superior frontal gyrus, dorsal premotor cortex, and a posterior portion of the inferior frontal gyrus with the neighboring ventral premotor cortex. Significant clusters (p < 0.05, corrected) were displayed on the inflated average template of the cortical surface, and the X, Y, and Z coordinates of peak activations in each cluster were determined in the MNI space.

In a secondary analysis, we aimed to characterize contributions to goal-relevant analyses from cortical regions outside of LPFC that previously were implicated in processing of goal-directed actions (Beauchamp et al., 2003; Majdandzic et al., 2007; Hamilton and Grafton, 2008; Lestou et al., 2008; Hesse et al., 2009). To determine which regions disproportionately contributed to processing of either abstract or concrete goal constraints, we ran simulation testing for clusters with novelty by goal constraint interactions on the entire cortical surface.

In addition to the voxelwise, gamma-function-based analysis, we planned to conduct a region of interest (ROI) analysis employing a finite impulse response (FIR) model. The goal of this analysis was three-pronged: [1] to evaluate sources of the interaction between novelty and goal constraint, if any was observed in the voxelwise analysis, by testing simple effects of novelty (original vs. usual) at each level of goal constraint (abstract and concrete); [2] to directly compare differences in the novelty effect amplitude due to type of goal constraint between the anterior and posterior LPFC regions; [3] to examine time-courses of the differences in activity. For each participant, unbiased estimates of BOLD signal intensity at 40 consecutive 2-sec-long intervals (MR frames) during peri-stimulus time were obtained without making a priori assumptions about the shape of the hemodynamic response. These estimates at each time-point were selectively averaged across trials from each condition. ROIs were defined based on the clusters of differences found significant in the voxelwise analyses. The same set of ROIs was used to quantify the activity in all conditions in a given ROI analysis. For each ROI, we calculated average percent signal change, relative to prestimulus baseline, across the included voxels at each MR frame and condition. To quantify the signal change associated with each video condition, we computed the integral of the average percent signal change (iPSC) within the time-window of 4-12 sec after presentation of final video scenes. In the cortical regions that displayed an interaction between novelty and goal constraint in the voxelwise analysis, we determined the source of this interaction by subjecting the iPSC values to two repeated-measures analyses of variance (ANOVAs) that examined simple effects of novelty at each level of goal constraint. The direct comparison between the anterior and posterior LPFC activations included regions, which according to the above ROI analysis and/or the voxelwise analysis, displayed modulation of activity amplitude due to novelty in any goal-constraint condition. IPSC values in these LPFC regions were subjected to a repeated-measures ANOVA that examined an interaction between the anatomical location along the rostrocaudal LPFC axis, novelty, and goal constraint. The latency of the peak of activity differences was examined in the regions that showed a main effect of novelty or an interaction between novelty and goal constraint in the voxelwise analysis. To get a robust measure of peak latency (expressed in MR frames), we computed the weighted average of the peak frame and the frames just before and after the peak, weighting each of the three frame numbers by the average percent signal change at that frame. The peak latency values were subjected to repeated-measures ANOVAs, testing the effects of novelty and goal constraint.

2.3 Resting-state fMRI data

2.3.1 MRI data acquisition and preprocessing

MRI data were acquired on the same 3T Tim Trio scanner as the data for the video comprehension task. Resting-state activity was studied during a 6.2-min scan, during which T2*-weighted echo-planar images (124 consecutive volumes: 47 axial slices parallel to the intercommissural plane with no gap in between, 3-mm thickness, in-plane resolution of 3 mm) were acquired with a gradient-echo sequence (TR, 3 sec; TE, 30 msec; flip angle, 85 degrees, matrix size 72 × 72). Participants were instructed to stay awake and keep their eyes open. In addition, high-resolution 3D structural images were acquired using a multi-echo T1-weighted magnetization-prepared gradient-echo (MPRAGE) sequence (TR, 2530 msec, TE, 1.64/3.5/5.36/7.22 msec for four consecutive images, flip angle, 7°; matrix size, 256 × 256).

Data were preprocessed according to the steps described previously (Buckner et al., 2009). The first four functional volumes were discarded. Slice-time correction employed SPM2 (www.fil.ion.ucl.ac.uk/spm). Correcting for any head motion was accomplished with rigid body translation and rotation using FSL/MCFLIRT. The functional data of each participant was registered to an average brain template, corresponding to the MNI305 atlas, via structural MPRAGE images and using FSL/FLIRT, and was resampled to 2 mm cubic voxels. Following intensity normalization and smoothing with a 4-mm FWHM Gaussian kernel, constant offset and linear trend were removed, and a 0.8 Hz low-pass temporal filter was applied. Variance due to spurious or regionally nonspecific sources (quantified by computing six parameters of head motion, the whole-brain signal, a ventricular signal, and a white matter signal) was removed using linear regression.

2.3.2 Analysis of resting-state fMRI data

Intrinsic BOLD signal correlations between brain regions during rest are thought to reflect monosynaptic and/or polysynaptic connectivity (Biswal et al., 1995; for review, see Fox and Raichle, 2007). The present study used resting-state functional connectivity measurements to obtain indirect evidence for presence of synaptic connections (mono- or poly-) enabling information flow between groups of cortical regions activated during comprehension of original actions. Previous investigations of the functional connectivity in the entire cortex have consistently described distinct functional modules, such as dorsal attention system, frontoparietal executive system, or default network (Vincent et al., 2008; Yeo et al., 2011). Because we hypothesized that comprehension of original actions will depend on the engagement of multiple neural modules, we were interested in how interaction between the modules may be supported. We aimed to identify functional connections between neural modules, represented by the regions identified in our video-fMRI experiment, that due to conscious effort during video comprehension could support formation of globally integrated systems (Meunier et al., 2010). To capture variance patterns within each region we used relatively large cortical areas (approx. 300 mm²). The regions included activation peaks and adjacent voxels that were significantly activated in the voxelwise analysis of the event-related fMRI recordings during comprehension of original actions. Because we aimed to represent each region with approximately equal number of voxels, the regions’ size was chosen based on the smallest cluster of activity identified in the voxelwise analysis. For each region, the average BOLD signal, aggregated across the included voxels, was computed for each of the 120 consecutive 3-sec-long intervals (MR frames). In the first-level analysis, functional connectivity among regions was quantified for each participant. Partial correlations between regional time-series, taken pair wise, were examined while controlling for the signal variance over time in other regions in the entire network under investigation, thus decreasing probability of global or third-party contributions to the functional connectivity estimates (Hampson et al., 2002; Salvador et al., 2005). Correlation coefficients were then converted to normally distributed z-scores by performing Fisher’s transform. In the second-level analysis, a significance level of association between regions in each pair was determined using the Hochberg procedure while participants were treated as a random factor (Turkheimer et al., 2001). The Hochberg procedure offers a sensitive test for studying systems such as functional brain networks that are expected to yield larger proportions of non-null findings, even if the number of brain regions under investigation is relatively large in relation to the number of observations (Turkheimer et al., 2001; Hwang, 2010). To adjust for multiple comparisons, this method estimated the number of “true” null hypotheses in the dataset by means of the P plot. The computations were supported by the HamNet software (Imperial College London; coded as a toolbox for Matlab -- the Mathworks, Natick, MA). Fisher’s z-scores were also averaged across participants, and then converted back to correlations (r-values) for reporting.

First, we aimed to test the hypothesis that the LPFC network involved in constructing representations of goal-relevant features during video comprehension will display functional connections between the contiguous regions along the rostro-caudal gradient, but not between the regions that are not immediately adjacent. Therefore, we initially restricted our analysis to the LPFC regions that, in the video fMRI experiment, displayed modulation of activity amplitude due to novelty in any goal-constraint condition. To evaluate our second hypothesis about the sensory inputs to the LPFC regions, we conducted the functional connectivity analysis in the network including all regions delineated in the video fMRI experiment, including additional regions outside of LPFC where activity differences due to novelty were modulated by the goal constraint type. We predicted that among brain regions that process specific domains of real-world behaviors, such as motion or object visual features, regions that conduct more abstract analyses will be functionally connected to more anterior LPFC regions.

3. Results

3.1 Video comprehension experiment: Behavioral and Event-related fMRI data

The analysis of behavioral responses obtained during the fMRI session confirmed that difficulty of the task (determining whether the conveyed events would often be witnessed in real life) did not differ between the goal-constraint conditions. Accuracy in distinguishing between original and usual videos was not significantly different between the two types of goal constraint. Original endings were correctly identified in most trials (abstract goal-constraint, mean 92.81% +/− s.e.m. 1.71%; concrete goal-constraint, mean 95.00% +/− s.e.m.1.65%; t(15) = −1.199, p > 0.05). Usual videos in the two corresponding control conditions were also accurately identified (control-abstract, mean 94.69% +/− s.e.m.1.33%; control-concrete, mean 95.62% +/− s.e.m.1.28%; t(15) = −0.643, p > 0.05).

Our main goal was to test a priori hypotheses about LPFC activation during comprehension of original actions. Therefore, our primary voxelwise analysis targeted LPFC. Figure 2A shows clusters of voxels in the left LPFC that were significantly more active during comprehension of original, relative to usual, video endings. The peaks of these differences in the BOLD signal were located in the dorsal premotor cortex (dPM; BA 6, middle frontal gyrus: MNI, X Y Z, −42 0 45), the caudal LPFC (BA 44, pars opecularis: −46 6 14), and the mid-LPFC (BA 9, middle frontal gyrus: −31 24 40). In addition, a significant interaction between novelty and goal constraint was observed in clusters of voxels in more anterior LPFC regions, bilaterally. These regions, shown in Figure 2B, had peaks of activity differences in the rostral LPFC (BA 10, middle frontal gyrus/sulcus: left −32 48 6; right 39 49 5), the left mid-dorsal LPFC (BA 9, middle frontal gyrus: −40 34 27), and the right rostro-dorsal LPFC (BA 10; superior frontal gyrus: 23 45 31). The regions are shown in yellow to indicate that the difference in the amplitude of novelty effects between abstract and concrete goal-constraint conditions (calculated by subtracting values in the concrete condition from the values in the abstract condition) had a positive value. Finally, a region, which was almost identical to the above left rostral LPFC, displayed an interaction between novelty and parametrically modulated fit (low, medium, high) of the original action with the initially perceived overall goal. This cluster showed peak differences in the left rostral LPFC (BA 10, middle frontal gyrus/sulcus: −34 47 14), and is shown in Figure 2C in yellow to indicate that the novelty effect was enhanced as the fit increased from low to medium to high. To illustrate this progressive increase in the novelty effect, we included an insert in Figure 2C, showing the mean effect size at each level of fit in this region, computed by subtracting the integrated percent signal change (iPSC) evoked by the usual videos from iPSC evoked by the original videos.

Figure 2.

Cortical regions that were differentially activated by different types of movie endings. The central sulcus is identified with dashed white lines. Shown are region(s) identified during testing of a priori hypotheses about the function of LPFC, which were more activated by the original, relative to usual, movie endings: 1, the left dorsal pre-motor cortex (dPM); 2, the left caudal LPFC; 3, the left mid-LPFC [A]; displayed an interaction between novelty and goal-constraint type: 4, the left mid-dorsal LPFC; 5, the left rostral LPFC; 6, the right rostral LPFC; 7, the right rostro-dorsal LPFC [B]; displayed an interaction between novelty and parametrically modulated fit between the original action and the overall goal: 5a, the left rostral LPFC -- the mean novelty effects at each level of fit in this region, computed by subtracting the integrated percent signal change (iPSC) evoked by the usual videos from iPSC evoked by the original videos shown in the insert [C]. Also shown are regions identified in a whole-cortex analysis that showed an interaction between novelty and goal-constraint type: 8, the left posterior inferior parietal cortex (IPC); 9, the left anterior IPC; 10, the right superior prefrontal cortex (SPFC); 11, the right posterior IPC; 12, the left lateral occipital-temporal cortex (LOTC); 13, the right temporal cortex (TC); 14, the right superior parietal cortex (SPC), 15, the left ITC; 16, the left precuneus; 17, the left posterior cingulate cortex (PCC); 18, the right IPC [D]. In B, C, and D yellow shows regions with positive values and blue shows regions with negative values of the differences in the amplitude of the novelty effects between abstract and concrete goal-constraint conditions (calculated by subtracting values in the concrete condition from the values in the abstract condition). All clusters were significant (p < 0.05, corrected). L - left, R - right.

The aim of our secondary voxelwise analysis was to characterize contributions to comprehension of original actions from the cortical regions outside of LPFC believed to be involved in processing of goal-directed actions. This analysis of the entire cortex examined which regions contributed disproportionately to processing of each type of goal constraint (regions where the novelty effect was sensitive to the type of goal constraint). A significant interaction between novelty and goal constraint was observed in several regions that were previously implicated in processing of object-directed actions (Beauchamp et al., 2003; Majdandzic et al., 2007; Hamilton and Grafton, 2008; Lestou et al., 2008; Hesse et al., 2009). Among these regions, clusters of voxels where differences in the novelty effect between the abstract and concrete goal constraints were positive are shown in yellow in Figure 2D. These differences peaked in the posterior inferior parietal cortex (IPC; BA 7/40, near the junction of the intraparietal sulcus with supramarginal and angular gyri; left −39 −54 34; right 48 −59 41), the left anterior IPC (BA 40, supramarginal gyrus: −57 −44 34), and the right superior prefrontal cortex (SPFC; BA 8, superior frontal gyrus: 20 16 56). In contrast, clusters of voxels where differences in the novelty effect between the abstract and concrete goal constraints had a negative value are shown in blue in Figure 2D. These clusters were extensive and showed peak differences in the left lateral occipital-temporal junction (LOTC; BA 19: −42 −77 −7), the left inferior temporal cortex (ITC; BA 37, fusiform gyrus: −41 −47 −15), the right temporal cortex (TC; BA 37, fusiform gyrus: 43 −43 −16; this cluster extended to the right LOTC, BA 19), and the right superior parietal cortex (SPC; BA 7, posterior intraparietal sulcus: 25 −56 51).

The voxelwise results were supplemented with an ROI analysis. The first aim of the ROI analysis was to evaluate sources of the interactions observed both within and outside of LPFC in the voxelwise analysis. Planned tests of simple effects of novelty (original vs. usual) were conducted at each level of the goal-constraint factor (abstract and concrete). When constructing simple-effects ANOVAs, we divided the regions that displayed an interaction between novelty and goal constraint in the voxelwise analysis into three subgroups on the basis of their correspondence to prior findings in the literature. The regions in the lateral prefrontal and inferior parietal cortices mapped well on the neural network previously implicated in processing of goal-relevant information (Koechlin et al., 2003; Sakai and Passingham, 2006; Hamilton and Grafton, 2008; Hesse et al., 2009). Nonetheless, little consensus has been reached regarding the factors that determined hemispheric lateralization of such activations (Sakai and Passingham, 2006; Badre and D'Esposito, 2007; Charron and Koechlin, 2010). Therefore, these ROIs formed two separate subgroups: the left lateral fronto-parietal subgroup included the left rostral LPFC, the left mid-dorsal LPFC, the left posterior IPC, and the left anterior IPC, and the right lateral fronto-parietal subgroup included the right rostral LPFC, the right rostro-dorsal LPFC, the right SPFC, and the right posterior IPC. In addition, the loci in the temporal, occipital, and superior parietal cortices corresponded to the regions within the ventral and dorsal visual streams that in prior studies have been suggested to represent detailed information about real-world objects, including their visual features, motion, and ways they can be manipulated (Martin and Chao, 2001; Beauchamp et al., 2003). These ROIs, including the left LOTC, the left ITC, the right TC, and the right SPC, were included in the third, visual stream subgroup.

For each level of goal constraint, a three-way repeated-measures ANOVA was conducted on iPSC values, with novelty, ROI subgroup (three levels), and region (four levels) as factors. These analyses revealed significant interactions between novelty and ROI subgroup (abstract goal constraint: F(2, 30) = 5.065; p < 0.05; concrete goal constraint: F(2, 30) = 9.009; p < 0.005). Further, planned simple-effects ANOVAs for each level of the ROI subgroup factor revealed two highly significant main effects of novelty. The iPSC values were enhanced during processing of original, relative to usual, video scenes in the left lateral fronto-parietal ROI subgroup in the abstract goal-constraint condition (F(1, 15) = 11.133; p < 0.005; interaction between novelty and ROI was not significant: F(1, 15) = 0.202; p > 0.05), and in the visual stream ROI subgroup in the concrete goal-constraint condition (F(1, 15) = 18.484; p < 0.001; interaction between novelty and ROI was also significant: F(3, 45) = 4.700; p < 0.05). In the visual stream subgroup, the planned tests of the novelty effect in each ROI in the concrete goal-constraint condition revealed the source of the observed novelty by ROI interaction: the novelty effect was significant in all (Fs(1, 15) > 18; ps < 0.001) but the right SPC (F(1, 15) = 1.695; p > 0.05). Figure 3A shows mean values of the iPSC novelty effects at each level of goal constraint for each of the three ROI subgroups (the visual stream subgroup excludes the right SPC).

Figure 3.

Results of the region of interest (ROI) analysis. A. The novelty effect in the integrated percent signal change (iPSC), computed by subtracting iPSC evoked by the usual videos from iPSC evoked by the original videos, averaged across each ROI subgroup in the abstract goal-constraint condition (shown in orange) and the concrete goal-constraint condition (shown in blue). B. Modulation of the iPSC novelty effect by goal-constraint type in the left LPFC ROIs. C. Differences in the time-course of BOLD response, computed by subtracting weighted peak latency during viewing the usual video endings from weighted peak latency during viewing the original video endings, averaged across ROIs within each cortical ROI subgroup in the abstract goal-constraint condition (shown in orange) and the concrete goal-constraint condition (shown in blue). D. Time-course of peri-stimulus BOLD response in the right rostral LPFC in the usual-abstract (shown in solid) and original abstract (shown in dashed) conditions; note, video context was presented during the first 6 sec, and was followed by final scenes presented during the next 2 sec.

* p < 0.01, error bars represent standard error of the mean (s.e.m.); L – left; R – right; MPC – medial parietal cortex; dPM – dorsal premotor cortex; MR frames – magnetic resonance frames (1 MR frame corresponds to 2 sec).

The second goal of the ROI analysis was to directly compare the modulation of the novelty effect due to goal-constraint type between the anterior and posterior LPFC regions. This analysis included the left posterior LPFC regions (dPM and caudal LPFC), which showed the main effect of novelty in the voxelwise analysis, and the left anterior LPFC regions (rostral LPFC and mid-dorsal LPFC), which, according to the voxelwise analysis showed a novelty by goal-constraint interaction, and according to the first ROI analysis, showed a novelty effect in the abstract but not concrete condition. A four-way repeated-measures ANOVA was conducted on iPSC values, with novelty, goal constraint, anatomical location (anterior vs. posterior), and region (two levels) as factors. This analysis revealed a significant interaction between novelty, goal constraint, and anatomical location (F(1, 15) = 10.369; p < 0.01). Figure 3B shows mean differences between the iPSC novelty effects due to goal-constraint type at each of these LPFC ROIs.

The final goal of the ROI analysis was to examine time-courses of the activations. We reasoned that activity involved in direct matching between the perceptual input and the representations formed based on goal priors would be more likely to show comparable peak latencies between the original and usual conditions. However, a delayed peak in the novel, relative to usual, condition would likely correspond to secondary integration/evaluation processing. Four repeated-measures ANOVAs were conducted on a weighted measure of peak latency, including factors of novelty and goal constraint. For the three ROI subgroups described above (left and right lateral fronto-parietal; visual stream), these ANOVAs also included a factor of region (4 levels). An additional ANOVA with a factor of region (2 levels) was used to test differences in the latency of the activity peaks in the dPM and caudal LPFC regions, which showed a main effect of novelty in the voxelwise analysis. Results revealed a significant interaction between novelty and goal constraint in the right lateral fronto-parietal subgroup (F(1, 15) = 7.654; Bonferroni-corrected significance threshold, p < 0.05). Planned simple-effects ANOVAs in this subgroup at each level of goal constraint revealed a highly significant main effect of novelty in the abstract goal-constraint condition (F(1, 15) = 21.851; p < 0.0005; interaction between novelty and region was not significant (F(3, 45) = 1.363; p > 0.05). There was a reliable delay (on the order of 1.5 sec) in peak latency in the original, relative to usual, condition. The novelty-related differences in the time-course of activity in the ROI subgroups at each level of goal constraint are shown in Figure 3C. Furthermore, Figure 3D shows the pattern of time-course differences observed in the right rostral LPFC, which was representative of the right lateral fronto-parietal regions. The delay in the activity peak in the right lateral fronto-parietal cortex in the abstract, but not concrete, condition was consistent with a possibility that these regions might mediate secondary evaluation of the overall meaning of video scenarios. Inconsistency between the initial overall goal and the final scene in the concrete original condition made it less likely that a global meaning representation would be formed. In contrast, successful contextual integration would be expected in the abstract original condition.

Activity in the right LPFC has been previously implicated in rule-governed evaluation of information recalled from episodic memory (Dobbins and Han, 2006). Interestingly, the whole-cortex voxelwise analysis in the present study revealed a significant interaction between novelty and goal constraint in the bilateral medial parietal cortex (MPC), which is thought to support retrieval from episodic memory (Wagner et al., 2005; Cavanna and Trimble, 2006; Spreng et al., 2009; Ranganath and Ritchey, 2012). Figure 2D shows clusters of the MPC activity differences in the present study (p < 0.05, corrected; yellow color represents positive abstract-concrete difference between the novelty effects), which peaked in the left posterior cingulate cortex (PCC; BA 23: −5 −20 32), the left precuneus (BA 31/7: −10 −64 26), and the right MPC (BA 23, PCC: 5 −20 38; this cluster extended to the right precuneus, BA 31/7). To evaluate the source of the observed interaction, an ROI analysis included simple-effects ANOVAs on average iPSC at each level of goal constraint, with factors of novelty and region (three levels). These analyses revealed that the MPC activity in the abstract goal-constraint condition was enhanced during viewing of the original, relative to usual, video endings (main effect of novelty: F(1, 15) = 9.673; p < 0.01; interaction between novelty and region was not significant: F(2, 30) = 2.119; p > 0.05). In contrast, an analysis of time-course differences, including an ANOVA on a weighted measure of peak latency with factors of goal constraint, novelty, and region (3 levels), revealed no significant differences. These data are graphically represented in Figure 3A and 3C.

Noteworthy, some cortical regions showed a significant novelty by goal constraint interaction in the whole-cortex voxelwise analysis, but no reliable differences (p > 0.05) in simple-effects ANOVAs at each level of goal constraint and ANOVAs on weighted peak latencies. Peak locations of such BOLD differences are reported but are not further discussed. These included the cuneus (left −6 −85 20; right 8 −83 16), the left insula (−36 −17 10), the motor cortex (left −24 −31 57; right 25 −31 57), and the right lingual gyrus (8 −91 −4).

3.2 Interim Summary: Rostro-caudal LPFC Organization and Comprehension of Real-world Original Actions

It is remarkable that the four left LPFC regions involved in comprehension of real-world original actions were closely proximal to the regions that previously were implicated in representing stimulus dimensions relevant for performing laboratory tasks (Koechlin et al., 2003; Badre and D'Esposito, 2007; cf. Figure 6 in Badre and D'Esposito, 2007). In the present study, the mid-dorsal and rostral left LPFC displayed a larger novelty effect in the abstract than concrete goal-constraint condition. What's more, the rostral left LPFC showed a linear increase in activity with improved fit between the initially perceived overall goal and the final original action. In the prior studies, comparable regions were found to guide task performance based on abstract stimulus attributes. In contrast, left-lateralized areas in the caudal LPFC and dPM, which in the present study showed the novelty effect regardless of whether the abstract goal constraint was relevant to comprehension, were similar to the regions that previously were linked to task control based on more concrete stimulus features. These results were consistent with our prediction that the LPFC regions along the rostro-caudal gradient would be involved in forming representations of abstract and concrete attributes relevant to real-life goals.

3.3 Resting-state fMRI data

The regions within the left LPFC, as well as other brain regions, which in our event-related fMRI study were engaged by comprehension of the original actions, mapped on several resting-state neural networks, identified in previous studies of intrinsic brain activity (Vincent et al., 2008; Yeo et al., 2011). To test our hypotheses about monosynaptic and/or polysynaptic connections that would enable information flow across such previously described neural modules, which would be critical for comprehension of the original actions, two functional connectivity analyses were performed on the fMRI data that we recorded during task-free, resting state.

The first analysis of the resting-state fMRI data tested the hypothesis about functional connections within the left LPFC network, including the rostral LPFC, mid-dorsal LPFC, caudal LPFC, and dPM regions. Figure 4A displays partial correlation coefficients obtained between time-series in each region pair, while controlling for the variance in time-series of the remaining two regions. The pattern of correlations suggested contiguous functional connections along the rostro-caudal gradient: rostral LPFC to mid-dorsal LPFC to caudal LPFC to dPM. The correlation coefficients between these contiguous regions were highly significant (p < 10^-9), whereas the rest of the correlations were not significant at the p = 0.05 level. This result is graphically represented in Figure 4B (see region numbers 1, 2, 4, and 5). Note that the contiguous relationships did not appear to reflect merely anatomical proximity. For instance, there was no evidence of functional connectivity between the mid-dorsal LPFC and dPM regions, even though – as can be seen in Figure 4B – the distances between these two regions and between the mid-dorsal LPFC and the caudal LPFC are approximately equal.

Figure 4.

Functional connections identified based on resting-state fMRI data between regions selected based on the activity during video comprehension. A. Partial cross-correlation matrix obtained in the analysis of the left LPFC regions. Shaded cells, p < 10^-9, non-shaded cells, p > 0.05. B. Functional connections (p < 10^-9), observed in the left LPFC and/or all-region analyses, plotted on the left lateral view of the cortex. Numbers correspond to brain region labels as stated in Figure 2. Pink color represents the connection between regions, which showed the main effect of novelty during video comprehension. Orange and blue colors represent connections with regions whose activity during video comprehension yielded positive and negative values, respectively, of the differences in the novelty effects between abstract and concrete goal-constraint conditions. C. Partial cross-correlation matrix obtained in the all-region analysis. Different colors represent the values of the partial correlation coefficients, p < 0.05. L – left, R – right.

The second analysis of the resting state data aimed to test the hypothesis about interactions of the left LPFC network with other brain regions engaged by comprehension of the original actions. This analysis included the four left LPFC regions and 12 additional regions, identified in the event-related fMRI experiment, where the amplitude or timing of the novelty effect was sensitive to goal-constraint type. We were particularly interested in the functional connections that the left LPFC might form with the left parietal regions that in the event-related fMRI experiment showed an increased novelty effect in the abstract condition, and previously were implicated in processing of goal-relevant information (Cattaneo et al.; Majdandzic et al., 2007; Lestou et al., 2008; Hesse et al., 2009). We also focused on the left LPFC connections with the visual stream regions that in our study showed an increased novelty effect in the concrete condition, and previously were suggested to represent detailed perceptual information (Beauchamp et al., 2003). Figure 4C shows the matrix of partial correlation coefficients between regions, taken pairwise, controlling for signal variance in the remaining 14 time-series. Based on previously outlined considerations for resting-state fMRI data that has undergone removal of global signal, we conservatively focused on positive correlations (see Buckner et al., 2009). Shown are correlations, corresponding to the obtained mean Fisher’s Z scores, which were significantly higher than 0 (t(45) > 2.090, ps < 0.05, uncorrected) and remained significant after correction for multiple comparisons using the Hochberg procedure (p < 0.05). The main results are graphically represented in Figure 4B; the correlations between these regions were highly significant (p < 10^-9). It is noteworthy that this analysis replicated the contiguous functional connections in the left LPFC. Crucially, there was specialization in the way the left LPFC was functionally connected to the left IPC: the anterior LPFC (rostral and mid-dorsal LPFC regions) was linked to the posterior IPC, and the more posterior, caudal LPFC region was linked to the anterior IPC. Furthermore, the caudal LPFC and the visual stream subsystem were functionally linked (via the left ITC). Finally, note that the right LPFC had functional connections with MPC and the left LPFC (Figure 4C).

4. Discussion

Neuroimaging in humans and neurophysiology in non-human primates have obtained converging evidence that stimulus dimensions relevant to performing laboratory tasks might be represented in LPFC (Freedman et al., 2001; Moore et al., 2006; Sakai and Passingham, 2006; Li et al., 2007; Cole et al., 2011). Within LPFC, rostral regions may subserve abstract information relevant to global goals, whereas more caudal regions may support more concrete stimulus characteristics pertinent to smaller goals (Koechlin et al., 2003; Badre and D'Esposito, 2007). In a tightly controlled study, we examined whether these functional properties of LPFC, which in real life could enable both production and comprehension of adaptive, original actions, are indeed generalizable to diverse goals of human behavior. During comprehension of videos, kinematics of original object-directed actions was predictable based on abstract and/or concrete information relevant to goals of depicted real-world activities. When the original actions unfolded, the attributes at each level, which were consistent with the observed target object and kinematics, were hypothesized to evoke enhanced response in the corresponding LPFC regions. Peri-stimulus activity in the left LPFC increased during comprehension of original, relative to usual, video endings. In the more anterior regions, this novelty effect was enhanced when original actions matched the abstract goal constraints, as was indicated by their interpretability as targeting the overall goal of the conveyed activity. Thus, we obtained evidence for regionally different activations in the left LPFC that were evoked when hierarchically-organized core principles of a broad range of real-world behaviors were relevant to naturalistic comprehension of original video scenes. Crucially, each behavioral sequence was guided by a distinct set of core principles.

Differences in the fMRI BOLD response observed in this study are unlikely to be accounted for by mere variations in general task demands, such as on working memory, memory retrieval, or sequential processing. The Latin square counterbalancing design, modulation of novelty at each level of goal constraint, and extensive behavioral pretesting of the video materials helped to rule out a possibility that superficial differences between video types might explain the results. Linking of specific LPFC regions with processing of different levels of goal constraint depended on activation increases due to novelty between otherwise physically comparable visual scenes within counterbalanced video sets. Moreover, the effects of interpretability of original actions as targeting the overall activity goal were established while carefully controlling for potentially confounding variables, such as contextual integration and the number, complexity, familiarity, comprehensibility, and predictability of sequentially presented real-world actions and objects. During the scanning session, participants were encouraged to watch the videos naturally. A simple task given to them, deciding whether events are often witnessed in real life, ensured that they paid attention. Nonetheless, their decisions were delayed until after watching each clip, and therefore, were likely independent from naturalistic comprehension processes¹. Difficulty of these judgments, quantified by response speed and accuracy, was comparable across the goal-constraint types.

Previously, strong evidence for a representational hierarchy along the LPFC rostro-caudal axis came from parametric modulation of processing at different levels of a rule hierarchy guiding laboratory task goals (Badre and D'Esposito, 2007). Here, within similar anatomic locations, we report a pattern of parametrically-modulated, peri-stimulus activity, consistent with hierarchical goal representations, during real-world comprehension. In the most anterior region within the left LPFC, activity was predicted by a linear increase in the fit between the initially inferred goal and the relevant attributes of objects and kinematics observed in the original ending. This region might form representations at the highest level of abstraction that were likely evaluated based on highly variable amounts of sensory input. On one end of a spectrum, these representations could be immediately rejected due to mismatching features of the target object, and on the other, they could be fine-tuned based on viewing the entire video scenario. In the posterior LPFC regions, which may form more concrete representations, activity patterns were consistent with less variability in the processing demand, likely due to a tradeoff between the number of active goal-relevant representations and the signal arriving from the higher representational levels. Selecting and fine-tuning hierarchical goal representations would require flow of information between the abstract and concrete levels of representation within LPFC. Using resting-state fMRI, we obtained evidence that, consistent with prior neuroanatomical findings in the monkey, the left LPFC regions recruited during comprehension of original actions form contiguous functional connections along the rostro-caudal gradient (see Badre and D'Esposito, 2009).

Importance of “core principles” of actions in comprehension of real-world events has long been noted in linguistics (termed “selection restrictions” of verbs; Fillmore, 1968) and cognitive psychology (termed “action affordances” of objects; Gibson, 1979; Kaschak and Glenberg, 2000). Interestingly, in comprehension of both verbal and video depictions of real-world behaviors, electrophysiological data suggested neuroanatomical distinctions between goal-relevant and associative representations (Kuperberg et al., 2003; Kim and Osterhout, 2005; Sitnikova et al., 2008). The current study significantly extended this prior work by implicating LPFC in processing of representations relevant to real-life goals. Intriguingly, consistency exists in loci of LPFC activations between our study and previous fMRI studies of real-world perception that also might have tapped into goal-relevant representations. Combining individual actions into sequentially-structured behaviors may involve computations of how outcomes of one action are relevant to goals of others (Botvinick and Plaut, 2004). When more abstract information was relevant to such computations during comprehension of long-term behaviors (e.g., raising a crop), activity was enhanced in the rostral LPFC, whereas when more concrete aspects were relevant in short-term scenarios (e.g., brushing teeth), activity was increased in the mid-dorsal LPFC (Ruby et al., 2002; Speer et al., 2009). Furthermore, the posterior LPFC showed sensitivity when action goals were evaluated based on unusual combinations between objects and actions presented with no global context (de Lange et al., 2008; Bach et al., 2010). Taken together, these data suggest that there may be an intrinsic hierarchical organization in real-world, goal-directed behaviors that can be discovered in diverse experiences over individuals’ lifetimes. Distinct LPFC regions may be involved in learning and later using specific levels of goal-relevant information (see Wood and Grafman, 2003, for a review of similar perspectives).

The present study suggests that activity in the anterior LPFC during viewing videos may be contingent on the fit between the initial goal inference and the original video completion. The disruption in the LPFC hierarchical processing for the mismatching actions occurred, possibly, because any alternative interpretation of the contextual behavior would have been overly broad to effectively bias a kinematic prediction. This result underscores importance of conceiving adequate goal priors for effective comprehension – a cognitive skill that may be crucial in optimizing performance of intelligence officers and detectives in the national security or police forces. Poor intertwining between the formulated goals and the observed original actions, such as that exemplified in another episode from the motion picture, ‘the Da Vinci Code’ (Howard, 2006), may lead to comprehension lapses. The British police were informed that fugitives, Langdon and Neveu, were on board of a landing plane. The plane slowed down, Langdon and Neveu exited and hid in a limousine. As the plane taxied to a complete stop, the police – who will never realize what had happened -- approached. Perhaps, the police failed to grasp that a plane slowing down is relevant to “exiting the plane”, because they contemplated an overly broad goal, such as “the fugitives aimed to escape”.

Growing evidence suggests that distributed representations within domain-specific posterior brain regions may also contribute to processing task-relevant stimulus dimensions (Li et al., 2007). In the present study, the left IPC that may mediate motion processing showed an increased novelty effect during comprehension of videos from the abstract, relative to concrete, goal-constraint condition. The posterior left IPC might process abstract motion representations relevant to behavioral goals. In prior studies, this region was active when motion representations were part of instructional sets during mental simulation and task preparation (Peran et al.; Luks and Simpson, 2004; Hesse et al., 2006). Furthermore, during execution and comprehension of object-directed actions, activity in this area was effector and kinematics independent (Cattaneo et al.; Majdandzic et al., 2007; Hesse et al., 2009). In contrast, the anterior left IPC might mediate more concrete goal-relevant aspects of motion because it previously showed sensitivity to changes in both goals and kinematics of movements (Lestou et al., 2008). Modulation of the video novelty effect in the opposite direction, a larger effect in the concrete than abstract goal-constraint condition, was within the bilateral ventral and dorsal visual streams, believed to subserve detailed representations of object visual features, motion, and manipulations (Beauchamp et al., 2003). Importantly, resting-state fMRI suggested that the functional connections of the domain-specific posterior regions might afford more integrative communications with the anterior than posterior LPFC regions, which again was consistent with the prior neuroanatomical evidence in the monkey (see Badre and D'Esposito, 2009). The posterior and anterior left IPC regions may be functionally connected with the anterior and caudal left LPFC, respectively. In agreement with the levels of the LPFC goal representations, inputs about perceived motion to the anterior LPFC might convey more abstract information than inputs to the caudal LPFC. Similarly, the visual streams may be functionally connected with the caudal LPFC, and thus, might facilitate goal interpretations based on detailed situational constraints of perceived video scenes.

Prior findings on hemispheric lateralization of the activity devoted to processing task goals have been inconsistent (Badre and D'Esposito, 2007). We observed that right-lateralized activity in LPFC and IPC in the global goal-constraint condition was delayed in novel, relative to usual, videos. Resting-state fMRI suggested functional connectivity between these regions and the bilateral MPC, which showed an enhanced novelty effect in the global, relative to current, goal-constraint condition. It is possible that during real-world comprehension, MPC supports episodic recall of prior experiences, which are then evaluated in relation to global goals of the perceived activity within the right lateral fronto-parietal network (Wagner et al., 2005; Dobbins and Han, 2006; Zacks et al., 2010; Swallow et al., 2011; Ranganath and Ritchey, 2012).

5. Conclusions

In summary, our findings provide novel evidence for hierarchically-organized neural processing of goal-relevant information during naturalistic comprehension of real-world behaviors. Crucially, the spatial pattern of the region-specific activity within the left LPFC was consistent with earlier findings when participants followed hierarchically-organized instructions of experimental tasks. This result generalizes the representational hierarchy account of the LPFC function to diverse core principles that can govern both comprehension and production of adaptive real-life behavior. Our results suggest that quality of comprehension, and perhaps, subjective experiences of being engaged, arising from the effortful LPFC-mediated cognition, might depend on the extent of interleaving between presumed behavioral goals and relevant features of observed original actions.

Highlights.

• Humans offered consistent interpretations for original real-world actions viewed in videos

• Some, but not all, original interpretations were biased by the abstract goal of overall behavior

• Viewing original real-world actions evoked greater left LPFC activity, relative to usual actions

• Activity in the anterior left LPFC reflected processing of the abstract bias

• Identified LPFC regions had resting-state functional connectivity along rostro-caudal axis