Neurocognitive Mechanisms of Conceptual Processing in Healthy Adults and Patients with Schizophrenia

Abstract

This overview outlines findings of cognitive and neurocognitive studies on comprehension of verbal, pictorial, and video stimuli in healthy participants and patients with schizophrenia. We present evidence for a distinction between two complementary neurocognitive streams of conceptual analysis during comprehension. In familiar situations, adequate understanding of events may be achieved by mapping the perceived information on the associative and similarity-based connections between concepts in semantic memory – a process reflected by an N400 waveform of event-related electrophysiological potentials (ERPs). However, in less conventional contexts, a more flexible mechanism may be needed. We suggest that this alternative processing stream, reflected by a P600 ERP waveform, may use discrete, rule-like goal-related requirements of real-world actions to comprehend relationships between the perceived people, objects, and actions. This neurocognitive model of comprehension is used as a basis in discussing relevant studies in schizophrenia. These studies suggest an imbalanced engagement of the two neurocognitive streams in schizophrenia, whereby patients may rely on the associative and similarity-based networks in semantic memory even when it would be more adaptive to recruit mechanisms that draw upon goal-related requirements. Finally, we consider the roles that these conceptual mechanisms may play in real-life behavior, and the consequences that their dysfunction may have for disorganized behavior and inability to plan actions to achieve behavioral goals in schizophrenia. Imbalanced

1. Introduction

Deficits in goal-directed behavior contribute to disability in schizophrenia (Poole et al., 1999; Velligan et al., 1997). Behavioral abnormalities may include context-inappropriate commission errors – actions that appear bizarre and are out of place, which constitute part of the disorganization syndrome of schizophrenia (American Psychiatric Association, 1994; Andreasen, 1984b; Andreasen et al., 1995a; Andreasen et al., 1995b; John et al., 2003; Liddle, 1987). Schizophrenia patients may also manifest global reduction in goal-directed behavior, which constitutes part of negative symptoms of avolitionapathy (Andreasen, 1984b; Kiang et al., 2003; van Reekum et al., 2005). In the paragraphs below, we outline neurocognitive mechanisms that may play an important role in normal goal-directed behavior, and consider how disease-related alterations in these mechanisms may contribute to the symptoms of schizophrenia.

The ability to find means to achieve behavioral goals is critical for goal-directed behavior. Traditionally, much research has focused on how humans are able to facilitate this process by mapping goals on semantic memory templates, or schemas, of common combinations between actions and entities (Bower et al., 1979; Fischler and Bloom, 1985; Hutchison, 2003; Zacks and Tversky, 2001). For instance, serving a cake may be effortlessly planned by accessing knowledge that cutting is usually done with a knife, and that this activity usually involves steps such as placing a piece of cake on a plate, passing the plate to a guest, etc.

Even though using such associative semantic memory is effective in familiar situations, it may be counterproductive in non-routine contexts when common connections between concepts must be overridden and the available objects and actions may need to be combined in an unusual way to achieve behavioral goals. For example, at an office birthday party, if a knife is unavailable, a reasonable substitute might be a stretched piece of dental floss. Recent research suggests that adaptive goal-directed behavior may be supported by a distinct type of real-world knowledge, coding goal-related requirements of actions, which can be applied to novel combinations between actions and entities (Sitnikova et al., 2008b). Goal-related requirements are properties that are necessary for the successful completion of an action, and exclude properties that are commonly present but are not necessary. In the above office party scenario, a ‘cutting’ implement must have <a sharp and sturdy enough edge>, but other properties such as <has handle> are not relevant. One can acquire an instrument with this goal-relevant element by pulling a few inches of dental floss out of a dispenser and stretching it tight between two hands. This knowledge of goal-related requirements of actions may also contribute to combining individual actions into goal-directed sequences. At each stage of a goal-directed activity, only a small subset of actions is possible given the current state of environment. For example, serving a piece of a cake is possible only after it has been cut.

In schizophrenia, abnormalities in neural activity mediating associative semantic memory have been previously linked to conceptual disorganization, which is often evident in the patients’ speech (Aloia et al., 1996; Goldberg and Weinberger, 1995; Kuperberg et al., 2007a; Maher, 1983; Maher et al., 2005; Spitzer et al., 1994; Titone et al., 2002; Titone et al., 2000). We propose that a similar neurocognitive dysfunction may contribute to the behavioral disorganization in schizophrenia. However, a distinct neurocognitive deficit in operations necessary to construct plans for goal-directed actions may underlie the avolition-apathy symptoms (Godbout et al., 2007; Gold et al., 2008; Levy and Dubois, 2006; Rempfer et al., 2003). We suggest that schizophrenia patients may under-recruit conceptual knowledge encoding goal-related requirements of behavioral actions, particularly when planning more complex, non-routine tasks (Sitnikova et al., 2008b; Sitnikova et al., 2008c).

Growing evidence suggests similarities in neurocognitive systems mediating real-world knowledge that is recruited in execution and comprehension of behavioral actions (Humphreys and Forde, 1998; Raposo et al., 2009; Rizzolatti et al., 2001; Ruby et al., 2002; Sokolov et al., 2009; Tettamanti et al., 2005). Experimental paradigms in which participants are asked to comprehend presented stimuli are ideal for examining specific conceptual processes while controlling for confounding variables. In this overview, we discuss findings from comprehension studies both in healthy and schizophrenia participants, and consider their implications for models of goal-directed behavior and its abnormalities in schizophrenia. We also describe our own work that assayed real-world knowledge mechanisms during comprehension of goal-directed activities conveyed in video clips. This video paradigm is especially attractive for studies of psychiatric populations as it taps into naturalistic cognitive processing of real-world behaviors, while spontaneously engaging patients’ attention (Levin and Simons, 2000).

Throughout this overview, we highlight experiments that aimed to characterize rapid conceptual processes of interest by recording event-related potentials (ERPs), which measure electrophysiological brain activity with the millisecond temporal resolution (Cohen et al., 1980; Williamson et al., 1978). ERPs are recorded at electrode sensors placed on participants’ scalp and are time-locked to the onset of experimental trials of interest (e.g., presentation of target words, object pictures, or visual scenes). In a typical ERP study, electrophysiological data is collected in 30–50 trials per experimental condition, and is selectively averaged to obtain a single waveform for each condition. The changes in the neurophysiological activity that give rise to ERPs appear as positive-going or negative-going deflections in the recorded waveform, often referred to as ERP components. These components vary in their distribution across the scalp. Usually, differences in the polarity and/or topography of ERP components between experimental conditions or study groups are interpreted as reflecting distinctions in the underlying neuronal sources of these components (Holcomb et al., 1999; Kutas, 1993). In the studies described below, such polarity and topography information is used to distinguish between the neural mechanisms mediating different conceptual processes. In contrast, changes merely in the amplitude or timing of an ERP component are usually interpreted as indexing modulation of the same neurocognitive processes (Holcomb et al., 1999; Kutas, 1993). In the studies described below, the onset, peak latency, and duration of the amplitude changes are used to characterize the time-course of the corresponding neurocognitive processes.

2. Graded semantic representations

Semantic memory has been proposed to store information about a person’s previous real-world experiences in a structured fashion (Bower et al., 1979; Brewer and Dupree, 1983; Fischler and Bloom, 1985; Hutchison, 2003; Lichtenstein and Brewer, 1980; Sitnikova et al., 2008b; Zacks et al., 2007; Zacks et al., 2001). According to this view, semantic representations of individual concepts have connections of varying strength, depending on factors such as their feature similarity or how often they have been experienced in the same context. These graded semantic relationships are believed to be accessed and used in comprehension and behavior, particularly in familiar situations. By finding correspondence between a perceived stimulus and the semantic memory network that represents a given real-world situation – i.e., by mapping the stimulus on the corresponding memory network – a comprehender can anticipate and prepare for the stimuli that would likely come next. Similarly by accessing a semantic memory network corresponding to an individual’s goal, this goal may be achieved by following a usual plan of actions.

2.1 Cognitive studies in healthy participants

An ability to map perceived stimuli on the graded semantic memory networks may account for the finding that comprehension is facilitated for concepts preceded by related context. The time to comprehend a target word (e.g., “restaurant”) is often the longest when it is preceded by an unrelated prime word (e.g., “bed”); it is intermediate when the target word is preceded by a moderately related prime (e.g., “ham”); and it is the shortest when the target is preceded by a strongly related prime (e.g., “wine”; for a review see Hutchison, 2003). Similar semantic priming has been found for picture targets following either word or picture primes (Bajo, 1988; Carr et al., 1982; McCauley et al., 1980; McEvoy, 1988; Sperber et al., 1979; Theios and Amrhein, 1989). Semantic priming is observed even when asynchronies between the prime and target onsets are very short (less than 300ms), suggesting that the facilitation may occur, at least in part, due to automatic access to graded semantic representations (Hutchison, 2003).

A series of recall paradigms suggested that common relationships within visual scenes may be stored in semantic memory (Intraub et al., 1992; Intraub and Bodamer, 1993; Intraub et al., 1996; Intraub and Richardson, 1989). Viewers were found to incorporate into their recollections of visual scenes expectations of what must have been present but was not actually perceived. More specifically, drawings of previously viewed visual scenes often incorporated added elements (e.g., a tree branch over the yard fence that was not present in the original picture). On recognition tests, originally seen visual scenes were often reported as their close-up views, and wide-angle foils were frequently reported as old pictures. This boundary extension effect occurred even when pictures were presented for only 250ms at a rate of three stimuli per second, suggesting that common scene representations may be activated automatically (Intraub et al., 1996).

There is also evidence that common relationships between actions and entities may be represented in semantic memory. In a set of sentence processing studies, verbs were found to be recognized faster following the context describing typical participants or locations of their actions (McRae et al., 2001; McRae et al., 2005b). Similarly, comprehenders processed nouns faster and preferentially looked at real-world objects when the target entity was customarily involved into an action conveyed by a preceding verb (Ferretti et al., 2001; Kamide et al., 2004; McRae et al., 2005a). Conceptual processing was especially facilitated when the context constrained a specific role that the entity usually plays in the action. For example, in sentences such as “She was arrested by a cop/crook”, expected words such as “cop” were processed faster than unexpected words such as “crook”, even though both types of target words were semantically associated with the described action (e.g., “arresting”). Interestingly, verbs may access typical spatial properties of their corresponding actions (Richardson et al., 2003). When listening to verbs that commonly refer to vertical actions (e.g., “smash”) or horizontal actions (e.g., “point”), participants were slower to detect visually presented stimuli of the corresponding relative to different spatial orientation (e.g., processing a vertical action “smash” interfered with discriminating targets on the top and bottom of the screen, but not on the left or right).

At a more global level, representations of individual events in semantic memory appear to be linked to the events that usually precede or follow the target event. When presented with common sequences of real-life events (e.g., eating in a restaurant), which were described in text passages or depicted in video clips, participants were quite inaccurate in distinguishing between the actions that were presented and lures that were plausible elements of the described activities (Bower et al., 1979; Brewer and Dupree, 1983; Lichtenstein and Brewer, 1980). Furthermore, common event sequences that were verbally described or viewed in an atypical, scrambled order tended to be falsely remembered in a typical order (Bower et al., 1979; Brewer and Dupree, 1983; Lichtenstein and Brewer, 1980). Finally, common, consecutive events were processed faster when they appeared in a chronological order than in the reverse order. For example, combinations such as “The boy bites off a juicy apple – chew” were processed faster than combinations such as “The stomach digests the food – swallow” (van der Meer et al., 2002).

2.2 Neurocognitive studies using ERPs in healthy participants

The excellent temporal resolution of the ERP technique has been used to gain insights into the time-course of the semantic relatedness effects on comprehension. Electrophysiological recordings have identified a negative-going waveform, peaking at approximately 400 ms after stimulus presentation (Barrett and Rugg, 1990; Kutas and Hillyard, 1980a; Kutas and Hillyard, 1980b), which may reflect mapping of perceptual input on graded semantic memory networks (Sitnikova et al., 2008a; Sitnikova et al., 2006). The magnitude of this waveform, labeled the N400, was found to be inversely correlated with the strength of the prime-target relationship in word or picture pairs as well as with the predictability of target words or pictures within the preceding sentence context (Federmeier and Kutas, 2001; Ganis et al., 1996; Grose-Fifer and Deacon, 2004; Holcomb, 1988; Holcomb, 1993; Kutas and Hillyard, 1980b; Kutas and Hillyard, 1984; Kutas and Hillyard, 1989; McPherson and Holcomb, 1999). These results suggested that the difficulty of mapping the stimulus on graded semantic representations is indexed by the N400: the closer the meaning of the eliciting item to the specific semantic-memory field activated by the preceding context, the less demanding this mapping process, and the smaller the amplitude of the N400.

The N400 has been found to be sensitive to relationships within visual scenes and between real-world events. An N400 attenuation was reported to individual objects presented in congruous relative to incongruous visual scenes (Ganis and Kutas, 2003). For example, a soccer ball presented as a part of a congruous scene depicting a soccer match evoked a smaller N400 than a toilet paper roll presented in place of the soccer ball in the same visual scene. Furthermore, the N400 was modulated by the global story context provided by successively presented groups of sentences or visual scenes (Van Berkum et al., 2005; van Berkum et al., 1999; van Berkum et al., 2003; West and Holcomb, 2002). For example, following a series of pictures showing a girl run a race and then fall down, a congruous final scene showing the girl watch her competitors cross the finish line elicited an attenuated N400 relative to the incongruous final scene showing a girl carrying a pot (West and Holcomb, 2002).

Even though similar in the time-course and sensitivity to experimental variables, the N400 effects evoked to words and pictures have been reported to differ in their distribution across the surface of the scalp, suggesting distinct underlying neuronal sources. Whereas the N400 evoked by verbal stimuli is characterized by a parietal-occipital scalp topography (Friederici et al., 1993; Hagoort and Brown, 2000; Holcomb et al., 1999; Kutas and Van Petten, 1994; van Berkum et al., 1999), the negativities elicited by pictures are typically distributed over more anterior electrode sites (Barrett and Rugg, 1990; Hamm et al., 2002; Holcomb and McPherson, 1994; McPherson and Holcomb, 1999; West and Holcomb, 2002). Thus, the N400 component may be comprised of at least two separable negativities, each reflecting access to similar but non-identical graded semantic networks (Holcomb et al., 1999; Holcomb and McPherson, 1994; Kellenbach et al., 2002; McPherson and Holcomb, 1999; Sitnikova et al., 2003b; Sitnikova et al., 2006; West and Holcomb, 2002). These findings are consistent with the multiple-code theory of semantic memory that postulates several forms of conceptual knowledge (e.g., visual, auditory, abstract, etc.) stored within distinct brain regions (Paivio, 1971; 1986; 1991; Shallice, 1988; 1993).

2.3 Cognitive and neurocognitive studies in schizophrenia

A loosening of thought whereby consecutive ideas, even though seemingly related to each other, fail to follow a coherent progression is considered a core symptom of schizophrenia (American Psychiatric Association, 1994; Andreasen et al., 1995a; Andreasen et al., 1994; Maher, 1983). In the clinical setting, this so-called positive thought disorder has been traditionally diagnosed based on patients’ speech. More recently, computerized analysis methods have also characterized speech of schizophrenia patients as incoherent across the levels of individual sentences and broader discourse (Elvevag et al., 2007; Maher et al., 2005). Studies that used comprehension paradigms have led to important insights into the underlying mechanisms of such language disturbances.

One influential theory of positive thought disorder in schizophrenia posits that it arises from a hyperactivity within semantic memory networks. Support for this hypothesis comes from several studies that examined semantic priming in word pairs in schizophrenia (Kwapil et al., 1990; Manschreck et al., 1988; Moritz et al., 2001; Moritz et al., 2003; Spitzer et al., 1993a; Spitzer et al., 1993b; Spitzer et al., 1994). Especially in schizophrenia patients with positive thought disorder, under automatic processing conditions (e.g., short prime-target onset asynchronies), reaction time attenuation for the targets following semantically related primes was exaggerated, and robust priming was even observed between words that were indirectly related (e.g., ‘lemon’ is related to ‘sour’, and hence indirectly related to ‘sweet’). This result has been confirmed in a meta-analysis of 36 studies that documented augmented reaction time priming in schizophrenia patients with thought disorder relative to healthy controls (Pomarol-Clotet et al., 2008). The inverse relationship between the priming effect size in these patients and the inter-stimulus asynchrony was also confirmed. The main criticism of the semantic hyperactivity hypothesis of the positive thought disorder has been that the increased semantic priming may be an artifact of general slowing down of responses to both related and unrelated targets in schizophrenia patients (e.g., discussed in Barch et al., 1996). However, this explanation is unlikely given that the ERP recordings in schizophrenia have also reported an abnormally enhanced N400 attenuation to directly and indirectly related targets, particularly in patients with positive thought disorder, in the absence of general N400 size abnormalities (Kreher et al., 2007; Mathalon et al., 2002).

Words that have multiple unrelated meanings, called homographs, have been used to demonstrate how excessive activity within the semantic memory may disrupt sentence processing in schizophrenia. To comprehend a sentence that biases an infrequently used (subordinate) meaning of a homograph (e.g., “The skyscraper had ninety stories”), the homograph’s frequently used (dominant) meaning (e.g., ‘tale’ meaning of ‘story’) needs to be inhibited. Patients with schizophrenia were found to have deficits in inhibiting the context-inappropriate dominant meanings of homographs. When the context moderately biased subordinate meanings of a sentence-final homograph, recognition of target words associated with the homograph’s dominant meaning, which were presented after the sentence, was excessively primed in schizophrenia patients (Titone et al., 2000). Similarly, the N400 in schizophrenia patients was abnormally attenuated to target words that were incongruous with the global preceding sentence context but were semantically related to a dominant meaning of a homograph embedded within the context (e.g., the target word ‘author’ in a sentence “The skyscraper had ninety stories because the author won an award for it.” -- Sitnikova et al., 2002). One interpretation of these findings is that hyper-activation of the dominant word meanings in schizophrenia led to difficulties in inhibiting these representations, even when they were context-irrelevant.

Unlike the automatic spread of activation within semantic memory networks, strategic effects on contextual integration may be reduced in schizophrenia (Titone et al., 2002). When prime-target asynchronies in word pairs were long, facilitation in recognition of semantically related compared to unrelated targets was abnormally decreased in schizophrenia patients (Barch et al., 1996; Vinogradov et al., 1992). Similarly, several ERP studies have reported reductions in modulation of the N400 by context in schizophrenia (Adams et al., 1993; Condray et al., 2003; Condray et al., 1999; Ditman and Kuperberg, 2007; Grillon et al., 1991; Kiang et al., 2007; Kiang et al., 2008; Kostova et al., 2003; Kostova et al., 2005; Ohta et al., 1999; Salisbury et al., 2000; Salisbury et al., 2002). One possibility is that strategic activation of semantic memory networks may be generally more effortful in schizophrenia patients. In several studies with long context-target asynchronies, the amplitude of the N400 to target words was abnormally increased in schizophrenia, independent of the congruence of the preceding sentence context (Iakimova et al., 2005; Nestor et al., 1997; Niznikiewicz et al., 1997). Moreover, this enhancement, especially for context-appropriate targets, and the reduced N400 priming effect were associated with thought disorder and more generally, with psychotic symptoms (Andrews et al., 1993; Condray et al., 2008; Kiang et al., 2007; Kiang et al., 2008).

Overall, studies that have examined graded semantic memory representations in schizophrenia suggest lack of balance between an appropriate use of these representations, controlling their automatic activation, and supplying alternative strategies to meet the demands of a specific cognitive task. The challenge of delineating the precise conditions under which this lack of balance leads to deficits in comprehension and/or speech has yet to be met. Importantly, a number of behavioral and electrophysiological studies reported that some markers of the semantic memory processes were relatively intact in schizophrenia (Andrews et al., 1993; Barch et al., 1996; Iakimova et al., 2005; Kiang et al., 2007; Kuperberg et al., 2006c; Nestor et al., 1997; Niznikiewicz et al., 1997; Olichney et al., 1997; Ruchsow et al., 2003; Sitnikova et al., 2002; Vinogradov et al., 1992). Another outstanding question, which we have started to address in our research described below, is whether abnormalities in activation of graded semantic memory networks may contribute not only to the thought disorder but also to the behavioral disorganization in schizophrenia.

3. Goal-related requirements of real-world actions

Even though graded semantic networks may facilitate comprehension and behavior in familiar circumstances, this form of knowledge representation may be too rigid to account for humans’ remarkable ability to adaptively plan and interpret actions in less conventional contexts. Thus, the concept of a cutting implement, such as a knife, may be represented in graded semantic networks together with several of its common properties including <has handle>. This representation would have a limited value for planning a cutting action if the only implement available was dental floss that does not have properties such as <has handle>. This representation may also be insufficient to comprehend such unconventional combinations between actions and objects.

We have previously suggested that the ability to flexibly plan and comprehend actions, especially in nonroutine situations, may depend on conceptual knowledge that selectively encodes information of what is required to achieve a specific goal (Sitnikova, 2003; Sitnikova et al., 2008b; Sitnikova et al., 2008c). A set of requirements including the semantic properties of entities and the spatiotemporal relationships between them can uniquely constrain specific actions. For example, a cutting action requires that an agent who is doing the cutting be able to execute this action (e.g., <have ability for volitional actions>), an instrument that is used to cut have the necessary physical properties (e.g., <have a sturdy sharp edge>), and the entity that is being cut be cuttable (e.g., <unsturdy>). In addition, there are minimal spatiotemporal requirements for the cutting action (e.g., <the involved entities must come in physical contact>). In comprehension of visual events, the correspondence between the perceptual input and the requirements of a given real-world action would allow viewers to identify the action and to assign roles to the involved entities. In planning one’s own behavior, accessing this knowledge may be used for flexible selection of the means to achieve a given goal.

Employing these discrete, rule-like semantic representations of goal-related requirements is fundamentally different from accessing graded connections between concepts in semantic memory. This analysis takes into account only a subset of the semantic properties of a given behavior – those that are necessary to achieve its goal. As a result, rather than providing a map of what can be commonly expected in a given context, this analysis offers great flexibility in coping with novel situations. It can be applied to combinations of entities and actions that have not previously been encountered. Thus, the goal-related requirements of the cutting action can be applied to determine that objects such as tightly stretched dental floss, which has a relatively sharp sturdy edge, can be used to cut objects such as a cake or a lasagna, which are unsturdy.

3.1 Cognitive studies in healthy participants

Violations of goal-related requirements of actions are rapidly detected during language comprehension, but this processing may be distinct in its temporal course from mapping of the perceived information on graded semantic representations (Caplan et al., 1994; Marslen-Wilson et al., 1988; McElree and Griffith, 1995; McElree and Griffith, 1998). For example, Marslen-Wilson and Tyler (1988) found that, when monitoring for target words in sentences containing different types of violations, participants were slower to recognize words that violated goal-related requirements of actions (when a noun argument of a verb was incompatible with the verb’s action) than to recognize non-violated words. In their experiment, subjects took longer to respond to target words such as “guitar” in a sentence “The crowd was waiting eagerly. John drank the guitar…” relative to the target words in control sentences such as “The crowd was waiting eagerly. John grabbed the guitar…”. Importantly, the time to recognize the target words in sentences with violations of goal-related action requirements was also longer than to recognize target words that were merely unexpected in the preceding context such as in “The crowd was waiting eagerly. John buried the guitar…”. Thus, difficulties in processing goal-related action requirements and mapping the target words on graded semantic representations influenced word recognition on different time scales.

The distinction between goal-related requirements of actions and graded semantic representations was more carefully examined in a series of elegant experiments by Glenberg and colleagues (Glenberg and Robertson, 2000; Kaschak and Glenberg, 2000). In these studies, participants were asked to make judgments after reading short text passages (e.g., a story about a girl who wanted to prove that she could hit well in baseball; she borrowed a crutch from a person recovering from a twisted ankle and used the crutch to hit an apple). After reading each scenario, participants might be asked to verify the truth value of a probe statement that was either: [1] highly relevant for the goal of the central action described in the passage (e.g., “the crutch is sturdy” – the crutch sturdiness was necessary for it to be used as a baseball bat), [2] was relatively unimportant for the central action (e.g., “the crutch is long”), or [3] was irrelevant for the central action but described a scenario that is frequently associated with the critical object (e.g., “the crutch can help with injuries”). Even though the first two probe types were similar in their degree of semantic relatedness to the contextual passages, participants were faster to verify the probes that were highly-relevant for the goal of the described central action (sturdy) than the less relevant probes (long). Critically, the highly-relevant probes (sturdy) were responded to even faster than the frequent associate probes (injuries). This and other similar results demonstrated that comprehenders not simply accessed common roles played by the target object in real life, but rather used the knowledge of goal-related action requirements to integrate the target items with the passage context.

Recent studies provided evidence for rapid integration between properties of visually-inspected real objects and goal-related requirements of actions conveyed in sentences. Chambers et al. (2004) tracked the eye-movements of participants who listened to spoken instructions about visual displays of real objects. On a given trial, participants were presented with a set of items such as: an empty bowl, some flour on a cutting board, and two eggs. One of the eggs was in a bowl and another in a different container. While viewing the object display, participants heard an instruction such as: “Pour the egg in the bowl over the flour”. The critical manipulation involved semantic properties of the viewed target objects (eggs). In the control condition, the display provided no information that would help to disambiguate the target noun (“egg”) in the verbal instruction: both of the eggs were liquid. In contrast, in a second condition the target noun was disambiguated: only the egg in the bowl was liquid and therefore could be poured. This study found that participants used such action affordances, perceived in the visual displays, to interpret relationships between constituents in the verbal instructions. In the visually disambiguated but not in the control condition, the prepositional phrase “in the bowl” was misinterpreted as a location where the egg had to be poured, as was evident by participants’ anticipatory eye-movements toward the empty bowl in the visual display. Similar results were also obtained when the critical variable was the size of objects in visual displays (Chambers et al., 2002). Thus, comprehenders were able to rapidly evaluate whether properties of the real objects meet the goal-related requirements of the linguistically described action.

3.2 Neurocognitive studies using ERPs in healthy participants

Several ERP investigations of language comprehension have examined temporal differences between processing of merely unexpected target words and incompatible verb-argument combinations in sentences (Hoeks et al., 2004; Kim and Osterhout, 2005; Kuperberg et al., 2003, for review see Kuperberg, 2007). In one of these studies,Kuperberg et al. (2003) asked participants to read three types of sentences such as the ones below, while ERPs were recorded to the target verbs (underlined):

• [1]At a basketball game the players would throw …
• [2]At a basketball game the players would sew …
• [3]At a basketball game the ball would throw …

In sentences like [3], the target verb “throw” was semantically associated with the preceding context (throwing is related to the basketball game context), but its preceding noun argument “the ball” violated the goal-related requirements of the action conveyed by this verb (balls do not have semantic properties necessary to execute the throwing action). In contrast, in sentences like [2], the goal-related requirements of the action conveyed by the target verb “sew” were not violated by the preceding noun argument “the players” (players can sew). In this case, comprehension difficulties arose at the level of relating these sentences to what commonly happens in the real world (sewing is unlikely at a basketball game). The results are shown in Figure 1. Confirming prior research, verbs that were merely unexpected in their preceding context evoked an increased N400 effect relative to the predictable verbs (see Figure 1A). However, a different pattern of the brain electrophysiological response was evoked by the verb-argument violations. Relative to predictable verbs, these violations evoked a P600 – a later, positive-going ERP wave that started at approximately 500ms after target verb onset and peaked between 600–700ms over the parietal scalp region (see Figure 1B).

Figure 1.

ERPs time-locked to unexpected target words in sentences, compared to ERPs time-locked to predictable target words (A), and ERPs time-locked to verb-argument violations, compared to ERPs time-locked to predictable target words (B). Note: negative voltages are plotted upward. Shown are waveforms at frontal, central, and parietal electrode sites whose relative locations on the scalp are indicated on the head diagram. Adapted from Kuperberg et al. (2003).

The P600 evoked by verb-argument violations might reflect attempts to make sense of the sentences by considering whether the noun argument matches goal-related requirements for some other role in the action of the target verb (e.g., the ball cannot throw but it can be thrown). In other words, participants might have ignored the syntax of the sentences, and attempted to use the goal-related requirements of actions conveyed by verbs to comprehend the statements. Nonetheless, there were some interesting similarities in the time-course and scalp topography between this effect to verb-argument violations and the P600 effect previously evoked by difficulties in the syntactic analysis of sentences. In prior studies, a P600 was elicited to syntactic ambiguities in sentences such as “The lawyer charged the defendant was lying.” – this sentence would often be initially misinterpreted as “The lawyer laid blame on the defendant”, and eventually would need to be re-interpreted as “The lawyer who charged the defendant was lying” (Osterhout et al., 1994). A P600 was also evoked to frank syntactic errors such as in the sentence “The elected officials hopes to succeed” (Osterhout et al., 1997). In sentences like [3] above, the semantic association between the target verb and the context might have suggested a likely interpretation (e.g., “At a basketball game the ball would be thrown”) and therefore, the verb-argument violations might have been perceived as syntactic anomalies (Kim and Osterhout, 2005).

These alternative explanations were evaluated by comparing ERPs between three different types of verb-argument violations (see Kuperberg et al., 2006a; Kuperberg et al., 2007b). In two of the violation types, the target verb was semantically associated with the context. The difference between these conditions was in the difficulty of integration based on goal-related action requirements. In sentences such as [3] above, the noun argument “ball” of the target verb “throw” had semantic properties required for the role of an undergoer of the conveyed action (the ball can be thrown). In the second condition, such as [4] below, the noun argument “gym” did not have the semantic properties required to be an undergoer of the target action “throw” (it is not possible that “the gym would be thrown”). Finally, in the third type of violations, such as [5] below, the noun argument “gym” of the target verb “sew” did not have the semantic properties required to be an undergoer of the action, and the target verb was not semantically related to the context (it is not possible that “the gym would be sewn” and sewing is not related to playing basketball in a gym).

• [4]At a basketball game the gym would throw …
• [5]At a basketball game the gym would sew …

If participants were attempting to integrate the noun arguments with the verbs based on goal-related action requirements, the P600 was expected to reflect the difficulty of this integration that increased from condition [3] to conditions [4] and [5]. Alternatively, if semantic association between the target verb and the context was the critical factor in evoking the P600, then the magnitude of this ERP was expected to be reduced in condition [5] relative to conditions [3] and [4]. The results are shown in Figure 2. Relative to the predictable verbs, all three violation conditions evoked a P600 effect that increased in amplitude as the integration between the verb and its noun argument based on goal-related action requirements became more effortful. Thus, the P600 is likely to reflect a mental effort engaged by the attempts to combine actions and entities based on goal-related requirements of actions.

Figure 2.

ERPs time-locked to different types of verb-argument violations in sentences, compared to ERPs time-locked to predictable target words. Violated target verbs were semantically related to the preceding sentence context, and were preceded by noun arguments that were either compatible (the ball would throw) or incompatible (the gym would throw) with the role of an undergoer of the action conveyed by the verb (A), or were not semantically related to the preceding sentence context and were incompatible with the role of an undergoer of the action conveyed by the verb (B). Shown are ERPs at a parietal electrode site. Adapted from Kuperberg et al. (2006a; 2007b)

Taken together, these findings suggest that during sentence comprehension, the process of evaluating entities conveyed by nouns against goal-related requirements of actions conveyed by verbs is reflected by the P600 – an ERP component that is different from the N400 not only in its timing but also in its polarity and scalp topography. These ERP data demonstrate that, during comprehension, there are not only temporal but also neuroanatomical distinctions between the mapping of the perceptual input on graded semantic representations and evaluating the input against the goal-related action requirements.

3.3 Cognitive and neurocognitive studies in schizophrenia

Recent research from our laboratory has compared processing of incompatible verb-argument combinations between healthy controls and patients with schizophrenia (Kuperberg et al., 2006b; Kuperberg et al., 2006c). When asked to judge acceptability of sentences, healthy control participants were considerably more accurate in judging incompatible combinations between verbs and their arguments (conditions [3] and [4] above) than when making judgments about predictable sentences (condition [1] above), expectancy violations (conditions [2] above), or sentences with syntactic errors. However, this selective improvement in the response accuracy for verb-argument violations was not observed among schizophrenia patients. Furthermore, abnormalities were detected in an ERP study using the same sentences, and are shown in Figure 3 (Kuperberg et al., 2006c). The P600 effect to verbs violating goal-related action requirements ([3] and [4] above) relative to predictable control verbs ([1] above) was found to be attenuated in patients relative to healthy controls. These findings are consistent with the hypothesis that schizophrenia patients have deficits in using goal-related requirements to flexibly integrate objects and actions during language comprehension.

Figure 3.

ERPs time-locked to verb-argument violations in sentences, compared to ERPs time-locked to predictable target words, in healthy control participants (A), and in patients with schizophrenia (B). Shown are ERPs at a parietal electrode site. Adapted from Kuperberg et al. (2006c).

This result is potentially very important because ability to combine actions and entities based on goal-related action requirements may lie at the core of flexible planning of goal-directed behavior (Sitnikova et al., 2008b; Sitnikova et al., 2008c). In schizophrenia, negative symptoms of avolitionapathy, including deficits in goal-directed behavior are often treatment-refractory, and lead to real-world disability (Andreasen, 1984b; Kiang et al., 2003; van Reekum et al., 2005). A subset of avolition-apathy symptoms have been linked to abnormalities in cognitive operations necessary to construct the plan of actions (Godbout et al., 2007; Gold et al., 2008; Levy and Dubois, 2006; Rempfer et al., 2003).

Nonetheless, language function may be impaired at many levels in schizophrenia (Condray et al., 2002). Indeed, the P600 evoked by syntactic errors in sentences was reported to be attenuated in schizophrenia (Kuperberg et al., 2006c; Ruchsow et al., 2003). Therefore, using linguistic descriptions of real-world actions may not be optimal for isolating the cognitive operations that support adaptive goal-directed behavior. To address this challenge, we designed an experimental paradigm employing short, silent video clips conveying real-world activities. Video stimuli have multiple advantages: they can eliminate the effects of certain non-semantic deficits (e.g., in syntactic analysis), they can engage patients’ attention while preserving naturalistic conceptual processing (Levin and Simons, 2000), and they can present events in a short time frame avoiding confounding influences from impaired context maintenance (see Barch et al., 2001; Barch et al., 2003; Cohen et al., 1999; Cohen and Servan-Schreiber, 1992; Condray et al., 1996; Holmes et al., 2005; Salisbury et al., 2002 for previous reports of impaired context maintenance in schizophrenia).

4. Neurocognitive studies using video clips

In our laboratory, we have recently employed ERPs to determine whether analyses based on graded semantic representations and goal-related action requirements are neurophysiologically dissociable during visual real-world comprehension (Sitnikova, 2003; Sitnikova et al., 2008b; Sitnikova et al., 2003a). Based on the research in language comprehension, we expected that mapping of visual events on graded semantic representations would evoke an N400, while evaluating the events against goal-related action requirements would elicit a P600.

To explore this hypothesis, we produced silent, color video clips depicting real-world events. All video clips were about 10 sec long, and were structured in a similar way: a common real-world activity was depicted in a lead-up context and was followed by a predictable or incongruous final scene. ERP recordings were time-locked to the onset of the final scenes.

We used several variations of this video paradigm. In one version, we aimed to modulate the difficulty of mapping the final event on graded semantic representations, and in other versions, we aimed to modulate the effort needed for the analysis based on goal-related action requirements. The final scene in all predictable video clips introduced a target object that was not seen in the lead-up context. For example, in one clip, the lead-up context depicted a man place a pair of pants on an ironing board; in the final scene, the man used an electric iron (the target object) to press wrinkles out of the pants (see Figure 4: [1]). As described below, the incongruous versions of the scenarios differed depending on whether we aimed to examine the processing based on graded semantic representations or goal-related action requirements.

Figure 4.

Frames taken from video clips (produced using Canon-GL1 digital video camcorder and Adobe digital-editing software) used in our video comprehension paradigm. For each video scenario, shown are two frames illustrating real-world events depicted as a context, followed by a single frame illustrating the predictable final scene [1], the unexpected final scene [2], and the less comprehensible [3] or more comprehensible [4] final scenes that were both unexpected and showed unconventional object-action combinations. The corresponding video clips may be viewed at http://www.nmr.mgh.harvard.edu/~tatiana/IJP. Adapted from Sitnikova et al. (2008b).

4.1 Mapping visual events on graded semantic representations and the N400 in healthy participants

Our initial goal was to extend previous findings of the N400 contextual priming evoked to static pictures, which were described above (see section 2.1), to the video stimuli. Therefore, our first experiment manipulated the predictability of the final scenes in the preceding video context (Sitnikova et al., 2008b). The unexpected video endings were created by replacing the original context shot in each video clip with a context shot from another semantically unrelated scenario. For example, a context shot might show a man place a loaf of bread on a cutting board; in the unexpected final scene, the man used an electric iron to press wrinkles out of a pair of pants (see Figure 4: [2]). We predicted that the N400 would be attenuated to the predictable relative to unexpected final scenes, because mapping of the context-appropriate scenes on the semantic representation of the preceding contextual events would be less effortful.

Our results are shown in Figure 5. The final scenes in video depictions of real-world events evoked a robust N400 that was attenuated by the congruency of the preceding video context. The onset of this effect at approximately 200ms after target scene presentation was comparable to the onset time of the N400 previously observed to words and static pictures, suggesting that mapping on graded semantic representations is initiated within a similar time frame across these types of stimuli (cf. Figure 1, also see Ganis et al., 1996; Kutas and Hillyard, 1980b; West and Holcomb, 2002). The overall time-course of the N400 effect to videos was prolonged, which might be accounted for by the extended presentation of video scenes, which unfolded over several hundreds of milliseconds. In line with this explanation, similar prolonged N400s have been observed during comprehension of spoken words, which also unfold phoneme by phoneme over time (Holcomb and Neville, 1991a; Holcomb and Neville, 1991b). Just as in the previous studies that used static pictures (McPherson and Holcomb, 1999; West and Holcomb, 2002), the N400 effect to dynamic visual events presented in video clips was evident predominantly over more anterior electrode sites (see Fz electrode site in Figure 5). This scalp topography was distinct from that of the N400 effect to words (cf. Figure 1, also see Friederici et al., 1993; Hagoort and Brown, 2000; Holcomb et al., 1999; Kutas and Van Petten, 1994; van Berkum et al., 1999). Thus, our study provided further support for the multiple code theory of semantic memory: comprehending the visual world and language might preferentially access graded semantic representations supported by distinct neurocognitive systems (Paivio, 1971; 1986; 1991; Shallice, 1988; 1993).

Figure 5.

ERPs time-locked to unexpected final scenes in video scenarios, compared to ERPs time-locked to predictable final scenes. Shown are waveforms at frontal, central, and parietal electrode sites whose relative locations on the scalp are indicated on the head diagram. Adapted from Sitnikova et al. (2008b).

4.2 Evaluating visual events against goal-related action requirements and the P600 in healthy participants

The main objective of our research with videos was to examine object-action integration based on goal-related requirements of real-world actions. In our second study, incongruous scenario endings depicted unconventional object-action combinations: they introduced a single context-inappropriate object into the activity conveyed in the video clip (Sitnikova, 2003; Sitnikova et al., 2008b; Sitnikova et al., 2003a). For example, after a man placed a pair of pants on an ironing board, in the final scene, he moved a dinner fork (the target object) across the pants (Figure 4: [3]). These scenes were not only unexpected in their context but also violated the goal-related requirements of the action constrained by the preceding context (e.g., the dinner fork does not have a flat hot surface necessary for pressing wrinkles out of pants). We predicted that these target scenes would evoke not only an N400, but also a large P600 similar to the one evoked by verb-argument violations in language.

Our results are shown in Figure 6. The magnitude of the N400 was reduced to predictable video endings relative to unconventional scenes (see Fz electrode site), indicating that participants attempted to map the perceived events on graded semantic representations, but this process was less effortful in the predictable condition. In addition, over more posterior scalp regions, the unconventional object-action combinations evoked an enhanced P600 (see Pz electrode site). Again, these ERP differences to video materials had a prolonged time-course, possibly, due to extended presentation of the video scenes. However, the onset latency and overall temporal pattern of these effects were similar to the language studies: the N400 started at approximately 250 ms, whereas the P600 was delayed until approximately 500 ms after target scene presentation. Interestingly, the P600 effect evoked in video clips resembled in its scalp topography the linguistic P600 evoked to incompatible combinations between the verbs and their noun arguments in sentences – both these effects had broad posterior distributions peaking over the parietal sites (cf. the P600 to words in Figures 1). These results suggest that similar neural mechanisms may be recruited by the analysis of goal-related action requirements during comprehension of both real-world events and their verbal descriptions.

Figure 6.

ERPs time-locked to the final video scenes that were both unexpected and showed unconventional object-action combinations, compared to ERPs time-locked to predictable final scenes. Shown are waveforms at frontal, central, and parietal electrode sites whose relative locations on the scalp are indicated on the head diagram. Sitnikova et al. (2008b).

In addition, we directly tested our hypothesis that the video P600 reflected attempts to integrate between the perceived objects and actions based on goal-related action requirements by manipulating the difficulty of this integration process. We compared ERPs between the more comprehensible scenes -- when the properties of engaged entities and their spatiotemporal relationships matched the goal-related requirements of a real-world action (e.g., see Figure 4: [4] a man squeezed the tire of his bicycle, which appeared soft, and then unscrewed the valve cap on the tire; in the final scene, he twirled a corner of a framed painting against the opening of the bike tire valve – the pointy and sturdy corner of the frame fulfills goal-related requirements for the goal “to scrape off mud”, which might have been stuck to the opening in the tire valve) than when the goals were less comprehensible (e.g., see Figure 4: [3] the dinner fork does not fulfill goal-related requirements for any conceivable goal-directed action). Our results are shown in Figure 7. The P600 effect to the less comprehensible unconventional video scenes was larger than the P600 effect to the more comprehensible unconventional video scenes. Taken together with our previous finding that the linguistic P600 was modulated by a similar manipulation (see section 3.2 above), this result provided further evidence that a neurocognitive process integrating entities and actions based on goal-related requirements may be reflected by the P600.

Figure 7.

ERPs time-locked to the less comprehensible or more comprehensible final video scenes that were unexpected and showed unconventional object-action combinations, compared to ERPs time-locked to predictable final scenes.

It is important that the P600 effect in video clips was selectively evoked to violations of goal-related action requirements, rather than any type of task-relevant semantic anomaly (this effect was not evoked to anomalies described in section 4.1). This result suggested a clear distinction of the video P600 from another positive-going waveform, the P300, which is generally evoked by ‘oddball’ stimuli and is thought to index domain-general strategic processes (Donchin and Coles, 1988). To further support this conclusion, we demonstrated that the P600 effect evoked to video violations of goal-related action requirements was not modulated by the behavioral task performed by participants (e.g., passive viewing vs. classifying the videos into congruous/incongruous -- Sitnikova, 2003; Sitnikova et al., 2003b). This lack of the P600 modulation by the behavioral task performed by participants was inconsistent with the pattern of modulation expected for the P300 (Picton, 1992; Polich, 1986).

All in all, our research with video stimuli supported the hypothesis that, just as language comprehension, the ability to understand the visual real world might be mediated by two complementary semantic integration mechanisms. The first mechanism, reflected by the N400, maps incoming information onto graded connections between concepts in semantic memory. The second mechanism, reflected by the P600, evaluates the incoming information against the discrete goal-related requirements of real-world actions. There may be a tradeoff between these mechanisms in their utility for integrating the people, objects, and actions during event comprehension, in which the first mechanism is better suited for familiar situations, and the second mechanism is better suited for novel situations.

4.3 A neurocognitive study using video clips in schizophrenia

Is the balance between utilization of graded semantic memory networks and goal-related action requirements disrupted during real-world comprehension in schizophrenia? What consequences may abnormalities in the function of these two complementary processes have for the real-life behavior in schizophrenia? Our next study addressed these questions by recording ERPs while patients with schizophrenia and healthy controls viewed videos that ended either with predictable final scenes (Figure 4: [1]) or scenes showing unconventional object-action combinations (Figure 4: [3] & [4]), and by examining the relationships of the evoked N400 and P600 to schizophrenia symptoms (Sitnikova et al., 2009).

Similar to the priming paradigms that presented words with short context-target asynchronies, videos that deliver rapid and naturalistic sequences of images may automatically access graded semantic memory networks. We predicted that hyperactivation of these representations during viewing videos in schizophrenia patients would be evidenced by an abnormally enhanced N400 attenuation to predictable video endings. Furthermore, we hypothesized that the semantic hyperactivity, as indexed by the N400 priming effect, would be proportional to the severity of disorganization symptoms in patients. To assess this symptomatology we used a disorganization factor score obtained based on the Scales for the Assessment of Positive and Negative Symptoms (SAPS/SANS) and the Brief Psychiatric Rating Scale (BPRS -- Andreasen, 1984a; Andreasen, 1984b; Overall, 1974).

To examine patients’ ability to recruit goal-related requirements when integrating between actions and objects in real-world activities, we measured the P600 enhancement to the unconventional object-action combinations relative to predictable scenes. Inadequate recruitment of this mechanism was expected to lead to an abnormally reduced P600 effect to these goal-related action violations. Moreover, we expected that if this neurocognitive deficit contributes to a real-life dysfunction in goal-directed behavior in schizophrenia, the magnitude of the P600 effect would be negatively correlated with the severity of this symptom. To quantify patients’ disability in goal-directed behavior, we used the SANS score on impersistence at work or school (Andreasen, 1984a). In addition, we aimed to examine whether schizophrenia patients would show more impairments during video comprehension when object-action integration based on goal-related action requirements was more demanding -- during viewing less comprehensible unconventional scenes (see Figure 4: [3] moving a dinner fork across pants) relative to viewing more comprehensible unconventional scenes (see Figure 4: [4] twirling a framed picture against the opening of a bike tire valve).

The results of this study are shown in Figure 8 – Figure 10. Our N400 findings obtained during real-world comprehension extended previous reports linking positive thought disorder in schizophrenia with abnormally enhanced reaction time and N400 modulation to verbal targets in priming paradigms (Kreher et al., 2007; Kwapil et al., 1990; Mathalon et al., 2002; Moritz et al., 2001; Moritz et al., 2003; Spitzer et al., 1993a; Spitzer et al., 1993b; Spitzer et al., 1994). Consistent with many of these previous reports of hyperpriming, the N400 attenuation to the predictable video endings did not significantly differ between the patient group as a whole and healthy controls, but rather was linked specifically to disorganization symptoms. This is illustrated in Figure 8A. Within the schizophrenia group, the magnitude of the N400 priming was correlated with the severity of disorganization: the more severe the disorganization, the smaller (less negative) the N400 to predictable scenes, the larger the N400 priming effect to predictable relative to unconventional scenes. Thus, in disorganized patients, increased activity within semantic memory networks may abnormally influence conceptual processing which is critical both for language and real-world behavior. In the non-verbal domain, this hyperactivity might lead to behavioral disorganization due to intrusions of goal-inappropriate but semantically related actions or entities into ongoing behavior (Andreasen, 1984b).

Figure 8.

Scatter plots showing relationships in patients with schizophrenia between the N400 ERP priming effect to predictable relative to unexpected/unconventional final scenes in video scenarios and the disorganization symptoms (A), and between the P600 ERP effect to unexpected/unconventional relative to predictable final scenes in video scenarios and the impersistence at work or school (B). Mean absolute value of voltage differences in each time window of interest were averaged across three electrode sites in a frontal midline region to quantify the N400 effect and in a parietal midline region to quantify the P600 effect. Adapted from Sitnikova et al. (2009).

Figure 10.

ERPs time-locked to the less comprehensible final video scenes that were unexpected and showed unconventional object-action combinations, compared to ERPs time-locked to predictable final scenes, in healthy control participants (A), and in patients with schizophrenia (B). Adapted from Sitnikova et al. (2009).

Our P600 results supported the hypothesis that goal-directed behavior deficits in schizophrenia may stem from the inability to use goal-related requirements for adaptive integration between objects and actions to achieve behavioral goals. As can be seen in Figure 9, the P600 effect to the unconventional object-action combinations relative to predictable endings was reduced in patients compared to controls. Moreover, the amplitude of the P600 effect was inversely correlated with patients’ impersistence at work or school: the more impaired the goal-directed behavior, the smaller (less positive) the P600 to unconventional scenes, the smaller the P600 effect to the unconventional relative to predictable video endings. This relationship is illustrated in Figure 8B. The processing abnormalities in schizophrenia were most evident during the viewing of the unconventional scenes that were more difficult to comprehend based on goal-related action requirements. In patients, these less comprehensible unconventional scenes evoked a dramatically reduced P600, but an abnormally increased N400 (see Figure 10). This finding suggested that schizophrenia patients under-recruited the knowledge of goal-related requirements, and instead, they inappropriately relied on a more rigid comprehension mechanism of mapping the stimuli onto the graded semantic memory networks.

Figure 9.

ERPs time-locked to the final video scenes that were unexpected and showed unconventional object-action combinations, compared to ERPs time-locked to predictable final scenes, in healthy control participants (A), and in patients with schizophrenia (B). Adapted from Sitnikova et al. (2009).

It is noteworthy that in our study, schizophrenia patients were encouraged to watch video clips naturally as they would watch a movie or a show on TV. Previous studies in schizophrenia suggested that, even though patients often fail to generate effective cognitive strategies (e.g., on episodic memory tasks), they were able to recruit appropriate neurocognitive mechanisms when such strategies are encouraged by a structured task (Bonner-Jackson et al., 2005; Ragland et al., 2005). In healthy participants, the recruitment of knowledge of goal-related action requirements may be influenced by the performed task. The P600 effect to verbal descriptions of incompatible action-entity combinations was larger when participants judged sentence acceptability as opposed to passively reading sentences (Kolk et al., 2003). Therefore, in the future work it will be important to examine the effects of task on the video P600 in schizophrenia. If the P600 effect increases when patients are asked to perform a task designed to facilitate the recruitment of goal-related action knowledge, this would suggest an inability to spontaneously engage the appropriate mechanism during naturalistic event comprehension. If the P600 remains strongly attenuated, this would suggest a loss of the knowledge representations of goal-related action requirements in schizophrenia.

5. Conclusions: Two semantic neurocognitive mechanisms and the real-life disability in schizophrenia

In this overview, we have outlined evidence for a distinction between two complementary neurocognitive mechanisms that may support comprehension of real-world events and their verbal descriptions. We also discussed experimental results suggesting that these mechanisms may not be engaged optimally during comprehension in schizophrenia, and that such abnormalities can be linked to specific symptoms of this disorder.

The first neurocognitive mechanism, reflected by the N400 ERP component, appears to access associative and similarity-based relationships between concepts stored in graded semantic memory networks. Elements of common real-world situations can be effortlessly mapped on strong connections within these semantic networks, thus facilitating comprehension in familiar situations. At least under some experimental conditions, patients with schizophrenia are able to engage this mechanism both during verbal and visual-world comprehension. However, in patients with disorganization symptoms, this process might be hyperactive, and may account for the language and behavior that include semantically associated elements that, nonetheless, fail to follow a coherent goal-directed course.

The second mechanism, reflected by the P600 ERP component, appears to access a distinct type of conceptual knowledge encoding discrete goal-related requirements of real-world actions. This mechanism is suggested to be effective in novel situations, as it evaluates whether the perceptual input meets certain core requirements, disregarding any irrelevant information. Our ERP studies provided evidence that this form of conceptual processing may be impaired in schizophrenia, and may track with deficits in real-life goal-directed behavior. This suggests that real-world disability in schizophrenia may, at least in part, arise from an impaired use of goal-related requirements to flexibly combine objects and actions into effective behaviors.

The knowledge of goal-relevant elements of actions may be essential in combining individual actions into goal-directed sequences. The sequential order of events in goal-directed activities is generally not random but is defined by which actions are possible given the current state of environment. Comprehension studies suggested that enabling and preventing relationships between events are established by calculating how each event influences the current state of the environment in relation to the requirements of the central actions in other events (Wolff, 2003). Similar mental calculations have been simulated in computational modeling of neural networks that map environmental inputs onto actions (Ballard et al., 1997; Botvinick and Plaut, 2004). In schizophrenia, an underutilization of goal-related action requirements may lead to an inability to structure individual actions into multi-step, goal-directed behaviors. Consistent with this possibility, a number of studies showed that schizophrenia patients are impaired in comprehending relationships between goal-directed events and ordering events in goal-directed sequences (Chan et al., 1999; Corrigan and Addis, 1995; Corrigan et al., 1992; Matsui et al., 2006). In these studies, associative knowledge of real-world activities was reported to be relatively intact: patients were able to recall frequent components of real-world activities, and distinguish between components that were and were not associated with a given activity.

Recent research in computational neuroscience suggested that the neurobiological mechanisms unique to the prefrontal cortex can support updating of maintenance contingent on the presence of a reward (Rougier et al., 2005). These mechanisms can mediate self-organization of discrete, rule-like representations coded by patterns of activity -- distinct sets of units with high synaptic weights (Rougier et al., 2005), rather than by changes in synaptic weights, which have been used to simulate graded connections within semantic memory networks (Hutchison, 2003). Such a prefrontal system might support self-organization of patterns of activity coding minimal goal-related requirements of real-world actions. Specifically, through breadth of learning experience with actions that either achieved or failed to achieve their goal (i.e., either coupled or not with a “reward” signal), this system may identify patterns of activity that were consistent across all instances of achieving specific goals (Sitnikova et al., 2008a; Sitnikova et al., 2008c). This learning mechanism would have great power to support the complexity of goal-relevant elements of diverse real-world actions, accounting both for explicit (which individuals are consciously aware of) and implicit (less conscious) knowledge of goal-related action requirements. Consistent with this possibility, a recent neuroimaging study revealed that the left dorsolateral prefrontal cortex (DLPFC) was activated when healthy adults decided whether two verbally described objects fit the requirements of a given action – when they were likely to utilize explicit knowledge of goal-related action requirements (Murray and Ranganath, 2007). Furthermore, other neuroimaging studies in healthy participants indicated that DLPFC was activated when outcome feedback was used to acquire non-verbal knowledge required to control a complex and dynamic real-world system (the sugar production task -- Rostami et al., 2009); and activity selectively within the DLPFC was sensitive to expertise in retrieving previously learned functionally-relevant, hard-to-verbalize attributes of novel complex shapes (Moore et al., 2006).

Negative symptoms, such as goal-directed behavior deficits, are related to poor treatment success, posing a considerable burden to affected individuals and society at large (Poole et al., 1999; Sharma and Antonova, 2003; Velligan et al., 1997). Our findings, linking these deficits to a specific abnormality in utilizing goal-related requirements of real-world actions, give some insights into the neurocognitive mechanism that may contribute to this poor outcome. Abnormalities of the prefrontal function that are a cardinal feature of schizophrenia (Barch, 2005; Braver et al., 1999; Manoach, 2003; Sitnikova and Kuperberg, 2007; Thompson et al., 2005; Weinberger et al., 2001) may prevent the very formation of memory representations coding goal-related action requirements. As a result, improving the prefrontal function in treatment may not be sufficient but might need to be complemented by cognitive remediation covering a broad range of real-world activities so as to form the conceptual memory representations required for successful goal-directed behavior.