Connecting and considering: Electrophysiology provides insights into comprehension

Abstract

The ability to rapidly and systematically access knowledge stored in long-term memory in response to incoming sensory information—that is, to derive meaning from the world—lies at the core of human cognition. Research using methods that can precisely track brain activity over time has begun to reveal the multiple cognitive and neural mechanisms that make this possible. In this article, I delineate how a process of connecting affords an effortless, continuous infusion of meaning into human perception. In a relatively invariant time window, uncovered through studies using the N400 component of the event-related potential, incoming sensory information naturally induces a graded landscape of activation across long-term semantic memory, creating what might be called “proto-concepts”. Connecting can be (but is not always) followed by a process of further considering those activations, wherein a set of more attentionally demanding “active comprehension” mechanisms mediate the selection, augmentation, and transformation of the initial semantic representations. The result is a limited set of more stable bindings that can be arranged in time or space, revised as needed, and brought to awareness. With this research, we are coming closer to understanding how the human brain is able to fluidly link sensation to experience, to appreciate language sequences and event structures, and, sometimes, to even predict what might be coming up next.

1 |. INTRODUCTION

As human beings, we experience a world that is infused with meaning. Our everyday environment teems with complex, often ambiguous sensory information, in the form of spoken, signed, and written words, numbers, visual objects, scenes, faces, environmental sounds, odors, and tastes. We respond to such information by rapidly and relatively effortlessly bringing to mind a rich array of knowledge that constitutes our understanding of these inputs and that drives our interactions with them. Uncovering the mechanisms that afford such pervasive and adept linking between perceptual stimuli and distributed, multi-modal information stored in long-term memory lies at the core of building an understanding of human cognition.

Within cognitive psychology, meaning processing has often been viewed as contingent on the outcome of a discrete access event, constituting what is termed “recognition” of an input. In classic psycholinguistic models, for example, word recognition is theorized to occur when the system detects a sufficient level of match between incoming word form information and representations stored in some kind of mental lexicon (e.g., Coltheart et al., 2001; Forster, 1976). This so-called “magic moment of recognition” (Balota, 1990) then serves as a gateway to the access of meaning information associated with the recognized word. Views of meaning processing across multiple domains, including word, object, and face processing, have been importantly and persistently shaped by variants of the hypothesis that a relatively punctate recognition event affords a transition between perceptual and conceptual processing. Notably, however, as I will discuss in detail, data accumulated across a large literature built from studies that track processing through time using measures of electrical brain activity (EEG) reveal an importantly different picture of the relationship among perception, meaning processing, and (something akin to) recognition. These studies (along with complementary work from other methods; for example, in language, Allopenna et al., 1998; Rodd, 2004; Tabossi et al., 1995) show that meaning is constructed incrementally from noisy, distributed representations, using a multifaceted set of temporally extended, dynamic processes.

Strikingly, across different types of perceptual inputs (including words in all modalities, as well as complex nonverbal stimuli), there is a time window around 400 ms after stimulus onset during which semantic information, broadly construed, impacts processing (for a review, see Federmeier et al., 2016). This time window encompasses what has come to be known as the N400 component of the event-related potential. In this article, I will first describe what the N400 seems to index and overview what work using this component has revealed about some fundamental features of how the brain perceives the world and links it to long-term memory, creating the effortless sense of meaningfulness we experience as we look around, read, and listen. However, what ultimately constitutes successful comprehension of a stimulus’ meaning varies across tasks and goals. Thus, I will next describe a set of other effects that are also sometimes—but, critically, not always—observed when people comprehend. I will discuss how the nature of these effects reflects the divergent constraints imposed by understanding the world versus acting in it (as, for example, during language production). Therefore, across this article, it will become evident that there are multiple “kinds” of comprehension, with differing processing requirements, yielding different representations, and producing diverse patterns of concomitant benefits and costs for various cognitive and behavioral outcomes. I will conclude by situating the resulting view of comprehension in our broader understanding of cognition and neural functioning.

2 |. THE N400

The first description of the N400 came from a series of experiments in which Marta Kutas, working with Steve Hillyard, compared responses to congruent and semantically anomalous endings of sentences. The sentences were presented one word at a time using so-called Rapid Serial Visual Presentation (RSVP), allowing participants to read without moving their eyes (since eye movements yield a substantial change in electrical potential that can contaminate the EEG signal of interest). One, now famous, example sentence began: “I take coffee with cream and …” The expected, congruous ending of this sentence is, of course, the word “sugar”; the anomalous ending was “dog”. Kutas and Hillyard (1980b) examined ERP responses time-locked to the onset of these sentence final words. Given what was known at the time, it was expected that the anomalous endings, which were also infrequent in the stimulus set, would elicit a P300—specifically, a P3b, a domain-general response associated with the processing of unexpected, improbable, “surprising” events (e.g., Donchin, 1981). Words that were congruous but unexpectedly larger in physical size indeed elicited enhanced positivity peaking just before 600 ms. The effect pattern seen when comparing semantic anomalies to congruent words, however, was notably different, with the anomalous words eliciting a larger negativity, peaking around 400 ms, compared to their more expected counterparts. (Anomalous words also elicited a small degree of P3b-like activity after that). Based on its polarity (negative-going) and peak latency (just before 400 ms), Kutas and Hillyard labeled this waveform feature “the N400”. Kutas and Hillyard replicated the N400 effect within their seminal paper, showing that the size of the effect also varied with the degree of semantic deviation and, in a follow-up paper, established that this response pattern was particularly associated with semantics, as other types of language-related anomalies (such as those arising at the level of language structure) yielded different (P3b-like) effects (Kutas & Hillyard, 1980a).

The N400 effect to semantic anomalies is robust at the individual subject level (see, e.g., fig. 2 in Kutas & Hillyard, 1980b) and has become one of the most well-replicated findings in the cognitive electrophysiology literature. However, it is important to note that much about the overall nature and behavior of the N400 is not well-represented by this specific effect, especially as it is sometimes described. Although there is a tendency for the N400 to be portrayed as if it were a brain response that arises when people encounter semantically anomalous words, subsequent work has made clear that semantic anomalies are neither necessary nor sufficient for the elicitation of N400 activity (for a review, see Kutas & Federmeier, 2011). Instead, the N400 is part of the normal response to all complex perceptual stimuli—basically, any type of input that will produce some activity in long-term semantic memory. The effect observed in the N400 anomaly paradigm arises, not because something particular is happening when people encounter anomalous words, but rather because something particular is happening when people process words that have contextual support. That is, being more expected reduces the size of the N400 to a particular word.¹ Moreover, although it was first discovered in the context of language, the N400 is not a “language component”. Like the P3b, the N400 is domain-general, elicited not only to words in all modalities, but, among other things, to meaningless letter strings, to gestures and environmental sounds, and to pictures (of objects, scenes, and faces) in both static and dynamic formats. Thus, although presenting participants with semantically anomalous words is a good way to elicit a robust N400 effect, the most robust understanding of the N400, and of the neural and cognitive processing that creates this effect, does not come from studies of semantic anomalies.

FIGURE 2.

Plotted (negative voltage up) at a channel corresponding to Pz on the 10–20 system are ERP responses in the related anomaly paradigm, for a selection of experiment types and participant groups. Solid lines depict responses to expected sentence endings, dashed lines to anomalous endings that come from the same semantic category as the expected endings (“within category violations”), and dotted lines to anomalous endings from a different semantic category (“between category violations”). In all cases, N400 amplitudes are sensitive to cloze probability, with smaller N400s to expected exemplars than to between category violations. The top row shows cases in which there is additional, prediction-related facilitation for the within compared to the between category violations: from left to right, young adults listening for comprehension to sentences presented as natural speech (Federmeier et al., 2002, strongly constraining condition), young adults reading word by word at a comfortable pace (Federmeier & Kutas, 1999a, strongly constraining condition), young adults viewing sentence-ending line drawings (Federmeier & Kutas, 2001, Expt. 2), and young adults reading words presented in the right visual field, biasing processing toward the left hemisphere (Federmeier & Kutas, 1999b). The bottom row shows cases in which prediction-related facilitations were absent in this same paradigm: from left to right, older adults listening for comprehension to sentences presented as natural speech (Federmeier et al., 2002, strongly constraining condition), older adults reading word-by-word (Federmeier & Kutas, 2019, Expt. 1, strongly constraining condition), young adults reading word-by-word at a fast pace (Wlotko & Federmeier, 2015), and young adults reading words presented in the left visual field, biasing processing toward the right hemisphere (Federmeier & Kutas, 1999b)

A more apt impression of the N400 can be found in the word position effect, first characterized by Van Petten and Kutas (1990, 1991). Participants read for comprehension, and N400 amplitudes are measured for all content words across the course of a normal, congruent sentence, such as this example from Payne et al. (2015): “The battery of my phone was dying, so I had to quickly plug it in to charge.” ERPs to the first content word of the sentence (here, “battery”) are characterized by a prominent N400—as is also characteristic of words encountered out of context, such as in a list. N400 amplitudes then incrementally decrease over the course of the sentence (i.e., show the pattern “battery” > “phone” > “dying” > “quickly” > “plug” > “charge”); Figure 1a shows the effect pattern, averaged over several word positions. This contrasts with the pattern observed in what are known as “syntactic prose” sentences, which were created by swapping out the content words, yielding a sentence that is grammatical but that does not afford a coherent representation at the message level: “The employee of my birthday was looking, so I had to happily put it in to charge.” Random sequences of words, which were thus not even grammatical, were also used as a comparison condition. Since participants never knew what type of sentence they would be reading, N400 amplitudes at the first content word were identical for all conditions. Notably, in syntactic prose sentences and random sequences, N400 amplitudes did not then increase as the sentences unfolded in an implausible manner. Instead, amplitudes in these conditions stayed consistent across sentence position, remaining at levels similar to that observed for the first content word. This pattern illustrates several core features of the N400: (1) eliciting it requires nothing more than passive perception of a word (or other stimulus), (2) its “baseline” state is large, and (3) its amplitude can be reduced, in a graded manner, by factors that facilitate processing, such as supportive context information.

FIGURE 1.

Plotted (negative voltage up) at a channel corresponding to Cz on the 10–20 system are ERP responses showing three basic types of N400 effects. In (a) is the effect pattern due to the position of a word in a sentence (data from Payne et al., 2015). Depicted with a dotted line is the average ERP to content words early in a sentence; note the prominent N400 response around 400 ms. If the sentence unfolds in a coherent manner, N400 amplitudes are reduced to words in the middle of the sentence (dashed line) and further reduced in late sentence positions (near sentence end; solid line). In (b) is the effect pattern due to cloze probability, in this case of sentence-final words (data from Payne & Federmeier, 2019). A large N400 is observed for words that are plausible but not predictable (cloze probability under 5%; dotted line). As cloze probability increases, N400 amplitudes show a graded decrease: The dashed line depicts the average response to words of moderate cloze (~25%) and the solid line to words with high cloze probability (~85%). In (c) is the effect pattern due to orthographic neighborhood size (data from Payne & Federmeier, 2019). Words (as well as nonword strings) that have many orthographic neighbors (dotted line) elicit a larger N400 than those with fewer neighbors (dashed line = moderate N; solid line = low N)

The activity measured in the N400 is thus something that routinely occurs when meaningful stimuli are encountered. As described in prior reviews (e.g., Federmeier & Laszlo, 2009; Lau et al., 2008), data from studies using magnetoencephalography (MEG), the event-related optical signal (EROS), and human intracranial recordings have delineated activity in a wide-spread brain network in both cerebral hemispheres (although often more prominent in the left) as contributing to the scalp-recorded EEG signal in the N400 time window. MEG and EROS measures, which are biased toward more surface sources, have consistently found a pattern of activity beginning in the middle and superior temporal gyrus and the superior temporal sulcus, rapidly spreading anteriorly along the temporal lobe, and eventually incorporating waves of activity in frontal areas, including the inferior frontal gyrus and dorsolateral prefrontal cortex (Halgren et al., 2002; Tse et al., 2007). Intracranial studies, which have mostly sampled inferior and medial temporal areas, have observed activity in anterior parts of the medial temporal cortex (Nobre et al., 1994). Intracranial studies have also found that during the time window encompassed by the scalp-recorded N400, there are waves of activity in higher-order sensory areas and in emotion- and motivation-related areas, such as the amygdala (Halgren, Baudena, Heit, Clarke, & Marinkovic, 1994; Halgren, Baudena, Heit, Clarke, Marinkovic, & Chauvel, 1994). The N400 therefore seems to reflect a coordinated, network-level activity pattern that yokes sensory processing and multimodal, associative processing. In cognitive terms, such activity likely corresponds with what might be described as semantic access, a linking between the current input and long-term stores of experience and knowledge. N400s to stimuli that are encountered out of context or in weak contexts (such as at the beginning of an isolated sentence or the end of a non-constraining one), or that mismatch their contexts, are large because these are cases in which a lot of new information is coming online as the stimulus contacts long-term memory. If, however, some or all of the information that would normally be evoked by the stimulus has already recently been activated, then the N400 to that stimulus will be correspondingly reduced. Consequently, N400s are smaller to stimuli that are repeated, primed by prior stimuli, or that fit with incrementally accrued information from context—as seen, for example, in the word position effect.

3 |. CORE PROPERTIES OF SEMANTIC ACCESS

The N400 is therefore a measure that can provide incisive evidence about how the brain retrieves knowledge from long-term memory in response to perceptual stimulation. Here, I will overview properties of the N400 that seem to divulge key features of the mechanisms involved in comprehension. I will begin with robust functional characteristics of the N400 response—that is, properties that have been shown to hold true for the N400 across different types of stimuli and various task conditions, over adult development (in healthy older adults as well as college students), and for both cerebral hemispheres when tested using visual half-field presentation methods.² In particular, I will present evidence that semantic activation is graded across a distributed network that allows relatively independent modulation of activation states and that time, rather than recognition, seems to be critical for triggering semantic processing.

3.1 |. Semantic activation is graded

As attested by the N400 amplitude decreases seen across the course of a congruent sentence, context information can accrue, and the influence of context on word processing is not all-or-nothing. Contexts vary in how much they narrow down the space of likely upcoming information (i.e., their “constraint” or entropy) and stimuli vary in their probability in any given context (i.e., their “expectancy” or surprisal). In language, these parameters have often been estimated using a procedure known as cloze probability norming. Cloze probabilities are typically calculated by giving a large group of people some context information, such as a sentence missing its final word, and asking them to produce the word they think is most likely to come next. A word’s cloze probability in a given context is then the proportion of people that produce that particular word in that particular context. Correspondingly, the constraint of a context can be approximated by the cloze probability of its most frequent completion. A highly constraining sentence is continued by most people with the same (high cloze probability) word, whereas a weakly constraining sentence yields a larger number of lower probability completions. Kutas and Hillyard (1984) first showed that there is an inverse relationship between cloze probability and N400 amplitude, such that N400 amplitudes are smaller to higher cloze probability words; see Figure 1b. The graded reduction of N400 amplitude as a function of cloze probability (or, more recently, estimates of word probability derived from language models) is a strong, highly-replicable effect, and is observed across the entire range of word probabilities (Szewczyk & Federmeier, submitted). When measured over scalp sites where N400 activity tends to be largest, the subject-level correlation between cloze probability and N400 amplitude is as high as 0.9 (e.g., Wlotko & Federmeier, 2012b) and this relationship remains robust even when assessed at the level of single items within single participants (Payne & Federmeier, 2019; Wlotko & Federmeier, 2012a).

Graded context effects on the N400 are not only strong, but they are also robust, generalizing across stimulus type and task conditions. For example, in experiments asking people to judge the fit of exemplars to category cues, N400 amplitudes pattern with typicality (e.g., cued with “A type of fruit”, N400 responses are smaller to “apple” (typical) than to “kiwi” (atypical); Federmeier et al., 2010; Heinze et al., 1998). In word priming experiments, across a wide variety of task conditions, N400 amplitudes are graded by factors such as association strength and corpus-based word co-occurrence/similarity statistics (Smith & Federmeier, 2019; Van Petten, 2014). Although less well-studied, N400 responses to pictures and line drawings are also influenced in a graded fashion by their fit to verbal and nonverbal context information (e.g., Cohn et al., 2012; Geukes et al., 2013; McPherson & Holcomb, 1999). Moreover, N400 effects of cloze probability, typicality, and semantic priming can be observed across the lifespan—that is, not only in college students but also in children (e.g., Byrne et al., 2007; Friedrich & Friederici, 2006; Stites & Laszlo, 2017), as well as healthy older adults ages 55–80 (e.g., Federmeier et al., 2003, 2010; Wlotko et al., 2012)—and, in visual half-field designs, for processing biased to both the left and the right hemisphere (Kandhadai & Federmeier, 2010a; Mech et al., submitted; Wlotko & Federmeier, 2013). Priming effects, in particular, have been attested across a wide range of task conditions, including manipulations of stimulus-onset asynchrony (SOA), relatedness proportion, and levels of processing (reviewed in Holcomb, 1988). The overall magnitude of such effects and details of, for example, the function relating cloze probability to N400 amplitude can vary across task parameters or participant groups and across the cerebral hemispheres, but, in all cases, N400 amplitudes are monotonically reduced as the probability of the item appearing in a context increases. Thus, one ubiquitous feature of comprehension is that stimulus-elicited activity associated with semantic processing is systematically reduced when inputs are encountered in “supportive” contexts of a variety of types.

3.2 |. Activation states across the network can be independently altered

Across the literatures of many methodologies, contextual probability effects are often examined using comparisons between conditions that differ both in entropy and surprisal. For example, the high cloze probability expected completion of a strongly constraining sentence might be compared to the same word in a weakly constraining sentence, wherein no word, including the critical word, has a high cloze probability. In those cases, one cannot determine whether effects arise because processing is different for stimuli (of any kind, high or low in surprisal) that are encountered in a more versus less constraining context or because stimuli, once encountered, are processed differently based on their fit to the context. However, it is possible to examine these factors separately. For instance, a word might have a low cloze probability because its sentence context doesn’t provide strong support for any particular word; for example, “She looked up and saw the word …” In other cases, however, a particular word is unpredictable in its context because the sentence creates an expectation for a different word. Given the sentence, “When the two met, one of them held out his …” most people think the ending will be “hand”. The word “badge” would then be an unexpected, albeit plausible, completion of this sentence.

On many theoretical accounts, building support for “hand” in the sentence above should impact the processing of other words like “badge”, making them less accessible, either via active suppression or because of competitive changes in activity levels across a lexical or semantic network. Comparing the response to “badge” when it occurs as this kind of “prediction violation” versus when it is matched in cloze probability but occurs in a weakly constraining context (e.g., “Sandy always wished that she’d had a badge”) tests this hypothesis, allowing an assessment of the effects of sentential constraint separable from cloze probability. Kutas and Hillyard (1984) were the first to compare N400 responses to items with similar cloze probability but in contexts of differing constraint. The pattern was clear and striking and has now been replicated many times: N400 amplitudes are correlated with cloze probability, independent of constraint. Thus, N400 responses are identical (and large) to low cloze probability completions like “badge” in strongly and weakly constraining sentences (e.g., Federmeier et al., 2007). N400 amplitudes are also the same (and of intermediate size) for words of moderate cloze probability (e.g., ~40%), whether they are the best completion of their sentence context or competitors to a slightly more probable completion (of, say, ~60% cloze; Kutas & Hillyard, 1984). Relatedly, N400s are equally small for high cloze probability endings as a function of whether there are only a few likely alternatives or, instead, a large set of other possible endings (Wlotko & Federmeier, 2012b).

The size of the N400 thus reflects only the extent to which the system was prepared for the stimulus that was actually presented. The state of activation of other parts of the system does not seem to make a difference. Accordingly, the processing reflected in the N400 does not appear to be inherently competitive in nature: Activating some information does not entail suppressing or “down-weighting” other information. This is consistent with accounts that model the N400 as arising from a wave of stimulus-driven activation, with inhibition coming online (if at all) only after a delay (Cheyette & Plaut, 2017; Laszlo & Armstrong, 2014; Laszlo & Plaut, 2012). As will be discussed later, this feature of the system shows up in other ways as well, including activation of the contextually irrelevant features of ambiguous words (Lee & Federmeier, 2009) and lingering activation of words that were predicted even when those predictions are ultimately disconfirmed (Rommers & Federmeier, 2018a). Although it goes against (some) deeply-engrained theoretical perspectives, I call this relative independence of activation states across the semantic system a “feature” because I believe it critically affords the flexibility that characterizes comprehension. It allows the system to remain open to processing whatever might come; after all, an item that was a competitor in one moment may actually be encountered in the next. Competitive interactions could be important for narrowing down an activation space to afford explicit recognition—but, as will become increasingly clear, this is something that may not play as much of a role in basic comprehension as has typically been postulated.

3.3 |. Semantic access does not depend on recognition

Thus far, I have overviewed some core factors that do, and do not, modulate the amplitude of the N400 to a given input. What about N400 latency? One oft-noted, but perhaps underappreciated, characteristic of the N400 is that it has a strikingly stable peak latency (reviewed in Federmeier & Laszlo, 2009). In the college student population that has been most extensively studied, N400s to visual words tend to peak a bit before 400 ms after stimulus onset, and responses to static visual objects and faces manifest a similar timing (reviewed in Federmeier et al., 2016). Auditory words in natural speech and dynamic visual stimuli also elicit N400 effects within the same basic time window, although precisely delineating the timing of the N400 to such stimuli is complicated by the fact that, in these cases, sensory stimulation itself is distributed across time. N400 latency varies to some extent across individuals as a function of childhood development and healthy aging (e.g., Kutas & Iragui, 1998; Sheehan & Mills, 2008). However, very few manipulations change the within-subject latency of the N400. As described already, contextual predictability, priming, and repetition all modulate the amplitude of the N400—but not its latency. The same is true for manipulations of word frequency (Fischer-Baum et al., 2014; Rugg, 1990; Van Petten & Kutas, 1990), perceptual noise (Aydelott et al., 2006; Vogel et al., 1998), and task demands/attention (Erlbeck, et al., 2014; Laszlo et al., 2012; Relander et al., 2009). This is a very striking pattern. If the N400 reflects semantic access, then what the lack of latency effects on the N400 reveals is that semantic access does not vary (much) in time with any of these variables. Yet, these same kinds of factors both theoretically should—and empirically do—impact the timing of measures associated with the recognition of words, faces, and objects. Thus, what the N400 reveals is that semantic access seems to be yoked to time, rather than being triggered by a particular functional outcome such as a recognition process.

To understand what semantic access is like if it can occur before stimuli are even fully recognized, it is useful to discuss the effect of another variable that has a strong, persistent effect on N400 amplitude: neighborhood size. In language, neighborhood size is a measure of how much match there is between a given input and all the words a person is likely to know. For example, one measure of orthographic neighborhood size, known as Coltheart’s N, is calculated by counting the number of words in a language that can be made by replacing any one letter of a given letter string (Coltheart et al., 1977). On this metric, the English word “MAP” has a fairly high neighborhood size: There are 23 words in English that are one letter away, including “CAP”, “MOP”, and “MAT”. In contrast, a word like “SKI” and many familiar acronyms, such as “BBQ”, have low neighborhood sizes; “SKI” has just two words in its neighborhood. Nonword strings like “LAR” and “NLR” also vary in neighborhood size. “LAR” is not a word in English but is similar to many English words, such as “FAR” and “LAP”, whereas “NLR” (like “BBQ”) has few neighbors. Thus, it is possible to match neighborhood size across strings that are and are not individually meaningful.

There is a strong, graded effect of neighborhood size on the N400, such that items with more neighbors elicit larger N400s—that is, more activity in long term memory (see Figure 1c). This pattern was first shown (to single word stimuli) by Holcomb et al. (2002) and has now been well-replicated (see, e.g., Laszlo & Federmeier, 2007). Orthographic neighborhood effects are attested for processing biased to both visual fields (Dickson & Federmeier, 2014) and are observed in (and similar in size for) both younger and older adults (Jongman & Federmeier, submitted; Payne & Federmeier, 2018); they are also seen in school-aged children and are predictive of later vocabulary development (Stites & Laszlo, 2017). Effects of neighborhood size are observed not only for isolated words but also for words in sentences (Laszlo & Federmeier, 2008). Indeed, whereas effects of other lexical variables, like frequency, generally decrease as local contextual information accrues, neighborhood effects persist undiminished across word position (Payne & Federmeier, 2018; Payne et al., 2015) and across all levels of contextual predictability (Payne & Federmeier, 2019). Sensitivity to neighborhood thus seems to be a core property of the network activity reflected in the N400, and what drives the amount of semantic activation observed to a given input is how many things in the larger network this stimulus is similar to and thus coming into contact with.

A particularly striking pattern is seen when effects of neighborhood size are compared for strings that are meaningful (words, acronyms) and those that are not (pronounceable pseudowords, illegal letter strings). Laszlo and Federmeier (2011) showed that the functions relating N400 amplitude to neighborhood size are the same for these classes of stimuli and, in fact, when neighborhood density is controlled, there are no reliable differences in N400 amplitude as a function of whether or not a stimulus has a learned meaning (see also Jongman & Federmeier, submitted). More generally, this is consistent with the observation that N400s are robustly elicited to even unfamiliar versions of all types of complex perceptual stimuli—not only pseudowords/letter strings, but also pseudo-objects (Holcomb & McPherson, 1994) and unfamiliar faces (Paller et al., 2000). Thus, the N400 doesn’t index activity associated with semantic access for a single, recognized percept; it instead reflects activity across the network with which the products of high level, but often still incomplete, perceptual processing make contact. That includes activation of not only information associated with the stimulus itself (including, if the stimulus is ambiguous, its multiple possible meanings; Lee & Federmeier, 2009, 2011), but also parallel activation of information related to a stimulus’ orthographic neighbors, as well as its phonological neighbors, its lexical associates, and so forth (Carrasco-Ortiz et al., 2017; Laszlo & Federmeier, 2011, 2014; Winsler et al., 2018). In the N400, then, we see semantic access arising directly from the products of still noisy, “pre-recognition” aspects of perception, and doing so in a relatively stable time window relative to sensory stimulation.

3.4 |. Some aspects of semantic processing are not attentionally demanding

As described in Box 1, the N400 seems to reflect neural activity that is critical for creating an initial, stable representation of semantics—that is, long-term, multi-modal information linked to an incoming word, object, scene, face, etc. Studies investigating the relationship between the N400 and attention or awareness have indicated that the establishment of this type of network dynamics is relatively “automatic” and is not sufficient for awareness. The kind of N400 effects described thus far are seen even when people cannot explicitly report the manipulations or the stimuli themselves. For example, N400 repetition effects are observed in amnesic patients, who cannot identify those repetitions and who do not show other ERP effects associated with explicit recollection (Olichney et al., 2000). N400 priming effects have been found in some stages of sleep and in other states of reduced consciousness (Brualla et al., 1998; Ibáñez et al., 2006; Rämä et al., 2010). They can also be observed during the attentional blink, which is a brief refractory period occurring about 300–600 ms after a target is detected in a stream of rapidly presented stimuli; during the attentional blink, subsequent targets are missed (i.e., cannot be explicitly reported). The attentional blink does not impact sensory processing but, during this period, stimuli are not effectively encoded into working memory, as evidenced by the diminution of P3b effects. Yet, N400 semantic priming effects have been reported, both from primes and onto targets that are presented during the attentional blink period (Rolke et al., 2001; Vogel et al., 1998).

BOX 1. Semantics before the N400?

I have argued that the extant literature supports a link between the activity measured in the N400 and a process that might be termed semantic access and that such access reflects contact between high level perception and long-term memory, without filtering based on recognition, beginning somewhere around 250 to 300 ms after stimulus onset. This view raises several important questions. First, how does one reconcile the (stable) timing of the N400 with the time course of other measures that have been used to study comprehension, such as eye fixation measures? And, if the N400 is taken to reflect semantic access, how should we interpret what appear to be effects of semantics that have been reported in time windows earlier than the N400? Here, I consider each of these questions in turn.

RECONCILING THE N400 WITH EYE FIXATION MEASURES DURING NATURAL READING

The average eye fixation duration for literate young adults naturally reading text is ~250 ms. In that time, readers must take in visual information and do whatever decision making and planning is necessary to execute their next eye movement. Several prominent accounts of reading, such as EZ Reader, postulate that some level of lexical access must be completed before the decision can be made to move the eyes (Reichle et al., 2006). If so, there would seem to be a critical discrepancy between the time course of lexical processing implied by eye fixation data versus the N400 as a measure of semantic access, as the fixation data seem to require lexical access to take place before about 200 ms—and thus before N400 effects would typically even begin (see discussion by Sereno & Rayner, 2003). However, there are also important differences between natural reading and the task conditions generally used to assess the time course of semantic processing with ERPs in terms of what information is available and when. Once those differences are taken into account, the timelines suggested by the two measures may not be at odds.

The N400 has been characterized both for isolated stimuli and stimuli in context, and, as discussed, its timing doesn’t change across these two conditions. Eye fixations during reading, however, are, by their nature, measured for words in context (sentences and larger texts). The context available during natural reading—but, critically, not during the typical RSVP paradigm used in ERP studies—also includes information about upcoming words, available through parafoveal preview. The region across which information can be processed in a single fixation is known as the perceptual span. In English, this span ranges from ~3–4 character spaces to the left of a fixation to upwards of 15 characters to the right of fixation (McConkie and Rayner, 1976). The perceptual span is thus asymmetric, showing an attentional skew in the direction of reading. Within any given fixation, therefore, a reader gets information not only about the word(s) in foveal vision, but, often, also about upcoming words that have not yet been fixated. It is well-established that readers can obtain orthographic and phonological information about these upcoming words (reviewed in Rayner, 2009). Whether or not semantic information is also extracted parafoveally has been more controversial, although some recent eye fixation data have shown parafoveal semantic effects in English, at least for words embedded in constraining contexts (e.g., Schotter & Jia, 2016).

One approach to studying parafoveal processing with ERPs is to use the flanker-RSVP paradigm, in which readers maintain central eye fixation and sequentially read centrally presented words that are flanked by the just previously seen word on the left and the upcoming word on the right (2 degrees from fixation), thus making the upcoming word first available to the reader parafoveally. Barber et al. (2011) used this paradigm to demonstrate that N400 expectancy effects are obtained for words in parafoveal preview, and this finding has been both replicated and extended. For example, Stites et al. (2017) observed graded N400 effects of cloze probability and plausibility for words in parafoveal vision. Moreover, these effects were functional, as, in young adults, they did not recur when the words moved into foveal vision. That is, the pattern showed that words first activated their associated semantics during parafoveal processing, and that information was then maintained, so that foveating the word did not yield (much) additional semantic activation (but see, Payne & Federmeier, 2017a, for age-related changes in this pattern). If, as these data indicate, initial semantic access can take place in parafoveal vision, and readers can retain that information into the subsequent fixation, then the timeline of processing delineated by ERPs and eye fixation data converges. On average, semantic information from parafoveal vision would begin to become available around the onset of subsequent foveation of that word. N400 responses associated with semantic access from the parafoveal preview would then peak during the critical window for saccade planning while that word was being fixated. Of course, eye fixation times vary, and the amount of information extracted from parafoveal vision does as well, such as with the difficulty of processing and concomitant amount of attention required by any given fixated word (“foveal load effects”; Henderson & Ferreira, 1990; see Payne et al., 2016, for an example of parallel eye movement and ERP foveal load effects). Thus, effects observed on the N400 would likely often be distributed in the eye movement record across multiple fixations and fixation patterns—e.g., sometimes seen on the target word, sometimes spilling over into subsequent fixations, and, in some cases, manifesting as regressions; indeed, this seems to be the case (see discussion in Stites & Federmeier, 2015). Finally, some (post-N400) semantic effects do not show up for words outside of fixation; for example, they are not seen for words in parafoveal preview but only obtain once the word has moved into foveal vision (Payne et al., 2019; Wlotko & Federmeier, 2007). Thus, as will be discussed in more detail later, some aspects of comprehension seem to require additional attentional resources beyond those required for the processing that drives the N400. As such, not all effects observable on the N400 will penetrate into later aspects of processing, including downstream eye movements and other forms of overt behavior.

UNDERSTANDING EARLY ERP EFFECTS OF SEMANTIC MANIPULATIONS

The timing of semantic access indicated by N400 data may also seem at odds with semantic (or other higher-level) effects that have sometimes been reported in pre-N400 time windows in ERP studies. Many such effects remain controversial, as they have not been replicated or extensively characterized (see Nieuwland, 2019). However, to the extent that some early effects are reproduceable, one important consideration is again the role of context in shaping processing. Context can shift the informativeness of different stimulus features and, accordingly, the time course with which certain types of effects might occur. One example of this phenomenon can be found in the field’s emerging understanding of what is known as the ELAN (Early Left Anterior Negativity; e.g., Friederici, 2002), an ERP response that has been found to certain types of syntactic violations and that occurs as early as 120 ms after word presentation—thus, in the same time frame as effects typically linked to relatively basic aspects of sensory processing. In a series of MEG experiments, Dikker et al. (2009) showed that early responses to word category violations (1) depend upon those violations being overtly marked (and thus perceptually salient) and (2) arise from activity in visual cortex that, in other paradigms, has been shown to primarily be sensitive to manipulations of low-level visual features. Thus, although syntax is not computed in visual areas, and word category information, as such, is unlikely to be available by 120 ms, when syntactic information allows advance predictions about visual features, visual areas are correspondingly able to provide information with utility for assessing whether incoming information fits those context-based expectations (see discussion in Lau et al., 2006).

However, it is notable that some early effects have been reported for words and other stimulus types encountered in the absence of this kind of predictive context information. For example, when participants are asked to categorize natural scenes as to whether or not they contain an animal, ERP responses differentiating targets from nontargets have been observed as early as 150 ms after stimulus presentation (Thorpe et al., 1996). Similarly early effects have been found for the categorization of words as referring to living/nonliving entities (but not other properties, such as object graspability: Amsel et al., 2013; Hauk et al., 2012). These effects obtain on a frontal negativity known as the N2, which has been linked to inhibition of motor processing and thus tends to be larger in go/nogo tasks for stimuli associated with the nogo decision (Simson et al., 1977). To the extent that these studies were able to successfully control for low-level stimulus properties that could be correlated with the higher-level categorical decision (see, e.g., Johnson & Olshausen, 2003, for evidence that the earliest effects reported for categorization of natural scenes arise from low-level stimulus differences), such results suggest that some aspects of semantics may reach decision and/or motor systems notably earlier than the time window of the N400.

Findings like these highlight that there is not a single time course of processing, but rather multiple processing streams, involving different kinds of neural activity that afford different kinds of representations and behaviors. Indeed, N400 responses are later than the time frame of the earliest feedforward pass of neural activity elicited by stimulus presentation. Using vision as an example, feedforward information propagated along the ventral visual stream reaches anterior temporal areas within ~150 ms; early activity in these areas seems to provide a coarse representation of superordinate (although not basic) category information (Clarke et al., 2013, 2015; Ganis et al., 2012). Such signals can then cascade into motor systems and provide evidence weighted toward one (of a few) options in a pre-established task set. Thus, under the right circumstances, coarse semantic information can be rapidly mapped onto a limited set of decision categories affording basic responses in constrained tasks (e.g., right/left hand button press, direction of saccade). Nevertheless, this kind of feedforward neural processing and the information it provides is insufficient to yield most of the cognitive phenomena and behaviors that are typically being referenced by the term “comprehension”.

Appreciation of finer-grained (basic and subordinate) aspects of object category and associated semantics, as well as the ability to become aware of that information and use it flexibly to drive behavior, requires more complex interactions between regions, involving feedback signals and the establishment of recurrent activity patterns (e.g., Lamme & Roelfsema, 2000; Wyatte et al., 2014). As overviewed by Schendan (2019), fast, largely feedforward processing before 200 ms supports the detection of salient shapes against a background, appreciation of superordinate object category information (e.g., faces vs. other object types), and perceptual implicit learning and memory; such processing can allow for simple, category-based decision-making under optimal, unambiguous conditions. Subsequently, a sustained state of iterative activation is established, including recurrent and feedback interactions among areas in the ventral visual stream and between those areas and prefrontal and posterior parietal areas. This second state of network dynamics, unfolding between about 200 and 500 ms post-stimulus-onset, is critical for most higher-level cognition, including the formation of complex, multimodal representations of a type that can eventually be brought to awareness and used for decision-making under ambiguity. Indeed, modeling supports the hypothesis that a combination of feedforward and feedback connections drives the development of network activity seen in potentials after about 200 ms post-stimulus-onset (David et al., 2006). Thus, the N400 likely indexes the initial establishment of this more stable, detailed representation that links perceptual information to associated knowledge stored in long-term memory—a type of binding, albeit one that, as will be delineated further, may not depend on attention.

N400 priming effects have also been investigated using paradigms wherein either the primes or the targets are processed outside of selective attention or in the presence of masking (see review by Deacon & Shelley-Tremblay, 2000). N400 priming effects are observed (although may be attenuated) when targets are encountered outside of attention and/or are masked (Kellenbach & Michie, 1996; Stenberg et al., 2000). When, instead, manipulations of attention or masking are done on the primes, N400 effects are more often eliminated (e.g., McCarthy & Nobre, 1993). Taken together, these results suggest that initially accessing semantic information associated with an incoming stimulus may not require attention, but that some degree of attention could be necessary to create a representation stable enough to linger and have downstream effects on subsequent words … a point that I will expand upon later.

3.5 |. Summary: Comprehension universals

As a set, the features of the N400 discussed thus far provide insight into what might be described as comprehension universals, the foundation by which the brain is able to habitually connect sensory input with memory, affording us an expeditious, multimodal, knowledge-infused representation of our world. Studies using the N400 have revealed that within a stable time window after new stimulus information is apprehended, high level perception, in a still noisy form, makes contact with long term memory in a manner that is relatively obligatory. Given that sensory information itself is coming in dynamically over time, the stable timing of this contact helps to organize the stream of incoming information. Time creates a link between each stimulus and the distributed information it activates across long-term memory stores. This link is noisy and transient but provides a ready and efficient means for the brain to track the relationship between external stimuli and the associations they evoke. Different from the fast, feed-forward pass of activation discussed in Box 1, the neural activity reflected in the N400 thus reflects an initial implicit binding between the eliciting stimulus and its semantics.

The goal of this initial phase of comprehension primarily seems to be getting information active—a form of pattern completion, which unfolds locally across the widely distributed network that is long-term semantic memory. Activation states within this network are modulated relatively independently, such that the brain can effectively be considering multiple, even contradictory, interpretations of the stimulus stream at once; it doesn’t (yet) have to choose. The information activated in long term memory then lingers for some time, such that, when another stimulus comes in, some of the associations it would normally evoke may not (need to) be activated again. As experimenters measuring N400 activity, therefore, we observe graded amounts of new information coming online in response to the same stimulus in different contexts. More importantly, this lingering of information allows the system to naturally build up an emergent representation of meaning from a multi-stimulus stream over time.

The neural activity that underlies the N400 thus supplies perception with a continuous infusion of meaning. This type of processing, however, will not afford success on all types of comprehension tasks, as such mechanisms are susceptible to a variety of errors. For example, N400 responses can be insensitive to certain kinds of semantic information and structures, including negation and quantification. N400 amplitudes to the final word of sentences such as “A robin is (not) a bird/vegetable” are unaffected by the presence versus absence of “not” and the corresponding difference in plausibility that creates; that is, in this example, N400s are equally large to “vegetable” and equally small to “bird” in sentences with versus without “not” (Fischler et al., 1983; Kounios & Holcomb, 1992). This pattern holds even if comprehenders are given extra time to process the negation (Dudschig et al., 2019), although, when it is encountered in richer pragmatic contexts, negation can contribute to N400 amplitude reductions by increasing expectations for a particular word/concept (e.g., “With proper equipment, scuba diving isn’t very dangerous;” Nieuwland & Kuperberg, 2008). A similar pattern is observed for quantifiers (e.g., “most/few”, “often/rarely”; Urbach et al., 2015; Urbach & Kutas, 2010). N400 amplitude patterns are also not consistently sensitive to thematic role assignments. For example, Kuperberg et al. (2003) found equal N400 facilitation for “eat” in the context of “For breakfast, the boys/eggs would only eat …” even though “eggs” are implausible agents of an eating event. Findings like these make clear that not all of the information important for determining plausibility or truth value becomes available in a manner and/or timeframe that can influence the type of comprehension reflected in the N400.

Indeed, there is more to the story of comprehension than the fundamental effects discussed thus far. In the next part of this article, I will overview some of the other mechanisms brought to bear during comprehension. However, one thing that is important to emphasize is that all of these other processing mechanisms are not obligatory or universal. As I will describe, their influence varies with task demands and goals and is subject to age-related, individual, and hemispheric differences. In turn, what this means is that, as comprehenders, our actual understanding of language and the world is routinely characterized primarily or exclusively by the comprehension universals discussed thus far. As also revealed in behavior and emphasized by accounts such as the “good enough” view of language processing (e.g., Ferreira et al., 2002), comprehension is often noisier and more incomplete than many theories and models—and our subjective impressions—have typically suggested.

4 |. ACTIVE COMPREHENSION

Over the past couple of decades, beginning with some of my doctoral dissertation work (Federmeier & Kutas, 1999a; see also Altman & Kamide, 1999), language research has amassed evidence for additional, more “active” processes in comprehension, including prediction. Because not everyone uses the term “prediction” to mean the same thing in terms of either mechanisms or consequences, I will begin with a characterization of what I would define as prediction in language and some empirical illustrations of its impact.

In my view, prediction arises from a change in the processing dynamics of the comprehension network, in which the activation states created by passive comprehension (which I will call “connecting”) come to be transformed (through a process I will label “considering”). Among other things, this change allows the system to generate and preactivate likely upcoming information, such as a probable next word in a sentence. As a result, the system can come to behave as if it has encountered a stimulus that it has not (yet) gotten as input and that it may never encounter, since predictions can, of course, be wrong. To illustrate this, consider the findings in Rommers and Federmeier (2018a). As mentioned previously, N400 amplitudes are reduced by repetition, and such repetition effects linger, even across intervening stimuli (e.g., Van Petten et al., 1991). In this study, we asked college students to read sentences for comprehension. We embedded a target word like “hot” in a very open-ended sentence: “It was strange to hear someone call this hot.” Since this (critical) sentence provides no support for the word “hot,” it elicits a large N400 response. However, in a second condition, participants had already read the word “hot” a few sentences prior, in a different open-ended sentence, such as “He was surprised when he found out that it was hot.” In that condition, we observed, as expected, an N400 amplitude reduction to the word “hot” in the critical sentence, due to repetition. The key manipulation took place in a third condition, in which the target word and critical sentence had been preceded (again, with several other intervening sentences) by a sentence that began “Be careful, because the top of the stove is very ….” If readers are predicting, it is very likely that they will predict that this sentence will end with the word “hot”. Crucially, however, participants did not see the word “hot”; instead, they were given an unexpected (but still sensible) ending, such as “dirty.” The key question, then, is whether, when readers later get to “hot” in the critical sentence, they will behave as if they had previously encountered it—even though they didn’t.

Indeed, comprehenders’ responses attest that prediction can have a similar impact as actually encountering a concept or word. Rommers and Federmeier (2018a) found that the N400 to “hot” in the critical sentence was reliably facilitated by previous prediction (although the effect of overt repetition was larger). Prediction is thus causing the brain to react as if it had previously encountered the word “hot”, and its impact is potent enough to be observed several sentences later. In fact, even later in time, if participants are tested on their memory for words encountered during a previous sentence reading task, they are likely to erroneously say they remember reading words that they had only predicted (Hubbard et al., 2019). In these cases, the lingering effects of prediction occurred even though participants not only didn’t see the predicted word, but instead encountered something unexpected that signaled that their original prediction, at both the level of form and meaning, had been incorrect. This lingering effect of information that was predicted is consistent with the comprehension universals described previously. Prediction is causing conceptual and word information to become active, but those activation states need not interact with the different semantic activations elicited by the unexpected word. Accordingly, repetition effects for words that violate predictions are not different from effects observed for equally unexpected words that are not prediction violations—both types of words simply elicit new semantic activity, independent of whether or not additional words and/or concepts were also active (Lai et al., 2021).

Prediction thus involves an active generation process that instantiates robust representations of likely stimuli, which can be like representations created directly through sensory stimulation. In other words, prediction is a type of (internal) production. Indeed, as I point out in the sections below, there are several lines of evidence linking predictive processes in comprehension to mechanisms used during overt language production.³ Thus, in trying to build an understanding of what prediction in language might be like, as well as how prediction is linked to the other processes that comprise what I will call “active comprehension,” it is useful to consider how language production—and possibly action more generally—differs from passive perception and comprehension. In particular, as discussed in the following sections, production places specific demands on the brain’s ability to actively bind information into representational “units” that can be propagated through the system, to select among those representations, to revise selections when needed, and to spatially arrange and/or temporally sequence the resulting representations.

4.1 |. Binding and propagating

To produce language, one must activate representations and propagate them in the system—moving, for example, from a conceptual representation to a specific word and a specific set of phonemes. In turn, this requires a particular kind of binding mechanism, which can bring together the distributed semantic features that comprise a concept, link the concept with the word or words that can express it, and/or combine the various phonemes that make up a word. This type of binding must have some degree of temporal stability, allowing utterances to be planned and prepared over time, and must be robust enough to allow the simultaneous tracking of multiple, bound representations, so that, for example, multiple words and phonemes can be ordered.

Indeed, when people are predicting during comprehension, we can see that they are activating conceptual “units” that go beyond the features directly evoked by the context and that they, at least sometimes, propagate that activation to specific words. Consider a pair of sentences like, “They wanted to make the hotel look more like a tropical resort. So, along the driveway, they planted rows of ….” N400 amplitudes should be reduced if this sentence ends with the expected word “palms” because some of the features of that concept have likely already become activated as comprehenders moved through the preceding context, encountering related words like “tropical” and accessing their schemas about resorts. The context, however, would activate very few features associated with either “pines” (another tree; thus, a “within category violation”) or “tulips” (a “between category violation”). Both words have a measured cloze probability of 0, and the context provides no differential support for these two concepts, both of which are items that can be “planted” but that are otherwise implausible in the described scenario. However, if one predicted palms—and thereby preactivated the whole concept—then features of palms that were not elicited directly by the context, such as knowledge that they are trees and are evergreen, would also become active. In this case, then, more of the features of pines than of tulips would be active when those stimuli were encountered, and N400 responses should be corresponding smaller to the within category violations. Indeed, when comprehenders are predicting, N400 responses are facilitated to unexpected (even anomalous) stimuli that share non-contextually-relevant features with expected stimuli. This is true for young adults reading word by word (Federmeier & Kutas, 1999a), listening to natural speech (Federmeier et al., 2002), and viewing line drawings in sentence contexts (Federmeier & Kutas, 2001); see Figure 2 (top). Further supporting that this pattern arises from prediction, the amount of facilitation observed for the within category violations is graded by contextual constraint: More facilitation is observed when the contexts are more constraining, meaning that those contexts allow stronger predictions for the expected concept. Notably, the strongly constraining contexts that afford the most facilitation for the within category violations are also those in which the direct fit of the violations to the context is the worst (i.e., the violations are rated as most implausible); thus, this pattern cannot be explained by the ease with which these words can be integrated into the message conveyed by the sentences.

Prediction thus brings into play additional conceptual information associated with likely upcoming stimuli. Moreover, we can see a similar effect pattern for unexpected words that share orthographic, rather than semantic, features with expected words (Laszlo & Federmeier, 2009). For example, given a sentence like, “The genie was ready to grant his third and final …,” where the word “wish” might be predicted, N400 amplitudes to semantically anomalous words are reduced in amplitude if they share many letters in common with the expected word (e.g., “dish”) versus if they do not (e.g., “clam”). Thus, at least in some cases, predictive processing also leads to the activation of specific words and their features, including grammatical gender (e.g., Wicha et al., 2003) and orthographic or phonological form information (e.g., whether a word starts with a vowel; Urbach et al., 2020). Effects of matching versus mismatching the gender or phonological features of the predicted word have been seen as soon as those features are encountered, at articles or adjectives that precede the anticipated noun, attesting that this information was, in fact, activated in advance. To even more directly measure predictive preactivation, Hubbard and Federmeier (2021b) used Representational Similarity Analysis, showing that EEG activity patterns for expected words, especially in strongly constraining sentences, can already be detected at the immediately preceding word. This signature of prediction was seen in an early time window, suggestive of visual feature activation (cf., Dikker & Pylkkänen, 2013). Taken together, these results support the idea that prediction leads to the prospective activation of whole concepts, which, in turn, can propagate activation to specific likely words and even their perceptual instantiations.

However, different from the N400 effects discussed in Sections 2 and 3, predictive processing effects are not ubiquitous. Influences from predictive processing on comprehension vary across task circumstances and are subject to group and individual differences. For example, prediction-based similarity effects (facilitation for within compared to between category violations) are not observed when young adults must read at a fast pace (Wlotko & Federmeier, 2015). Comprehenders with lower literacy are less likely to show prediction effects, not only when reading (Ng et al., 2017) but even when listening to natural speech (Ng et al., 2018; see also review by Huettig & Pickering, 2019). These effects are also absent at the group level for healthy older adults, in both listening (Federmeier et al., 2002) and reading tasks (Federmeier & Kutas, 2019); see Figure 2 (bottom). However, older adults with higher levels of verbal fluency are more likely to elicit ERP signatures of prediction (Federmeier et al., 2002, 2010). Verbal fluency is assessed by asking people to name as many exemplars of a category or as many words beginning with a particular letter as they can within a short period of time. The fact that the efficacy of cued language production is linked to the likelihood of showing predictive processing effects during comprehension is one piece of evidence linking prediction and other active comprehension processes to language production mechanisms. As I will discuss in more detail later, these kind of prediction-based effects in language comprehension are also asymmetric, as they have been associated with the left hemisphere and do not seem to characterize right hemisphere comprehension patterns. The fact that the left hemisphere is the one that, in most people, controls speech, further suggests a link between prediction and language production mechanisms (see discussion in Federmeier, 2007).

Note that in all these cases we continue to observe the direct, context-based facilitation of the expected item—that is, the basic word probability effect on the N400 that seems to reflect a core property of semantic access. That is, the high cloze probability expected exemplar is always facilitated compared to the low cloze probability violations (as can be seen in Figure 2). The difference across groups, individuals, and task parameters arises in whether there are additional effects, such as, in the related anomaly paradigm, a facilitation for within category violations compared to between category violations, despite their similar (near zero) cloze probabilities. Thus, prediction, as defined here, is a separate, non-obligatory factor that can shape activation states in semantic memory and thereby affect the N400, as well as many other aspects of the ERP signal (see review in Kutas et al., 2014). As discussed in more detail in Box 2, prediction, as a facet of active comprehension more generally, isn’t specifically tied to the N400 (or any other ERP component) and isn’t localized in either time or space. Instead, it reflects a large-scale change in how processing unfolds over time and across brain networks.

BOX 2. Is the N400 a prediction error signal?

I have made the case that N400 amplitudes can be sensitive to predictive processes in language. This raises the question of whether, as suggested by several models (Fitz & Chang, 2019; Rabovsky et al., 2018; Robovsky & McRae, 2014) and theoretical accounts (Bornkessel-Schlesewsky & Schlesewsky, 2019), the N400 should be thought of as a prediction error signal—that is, as reflecting the brain’s computation of the degree of mismatch between incoming information and a prior expectation. Although a detailed discussion of the strengths and weaknesses of these specific accounts is beyond the scope of this article, several general aspects of the nature of the N400, and of ERPs more generally, are important to highlight when thinking about the relationship between the N400 and prediction error.

To begin, how do the known response properties of the N400 align with an understanding of this activity as prediction error? I have made the case that N400 amplitudes closely pattern with the amount of new semantic activity elicited by a given stimulus. This view of the N400 creates a natural overlap with patterns expected on many prediction error accounts: To the extent that information can be predicted, it will both not create prediction error and not be new, and, to the extent that stimulus features are unpredictable, they will both create prediction error and engender activation of new information. Thus, many N400 effect patterns can be successfully modeled by mapping the N400 onto some kind of prediction error (e.g., Fitz & Chang, 2019; Rabovsky et al., 2018), but can also be modeled by accounts that do not build in any form of prediction and instead map the N400 onto newly arising semantic activation (e.g., Cheyette & Plaut, 2017). Differentiating these accounts, therefore, requires finding cases wherein the amount of new semantic information carried by a word can be dissociated from the prediction error it is posited to create.

Some models (e.g., Rabovsky et al., 2018) have specifically linked the N400 to prediction error at the level of the message, assuming that the processing system maintains and updates an implicit, probabilistic representation of the unfolding meaning of the sentence. In this case, the amount of new semantic feature information provided by a word can be separated from the amount of message-level updating it should induce. We did this in a study looking at the response to adjectives that, to varying degrees, enhanced or diminished the predictability of likely upcoming nouns (Szewczyk et al., in press). For example, the unfolding sentence, “His skin was red from spending the day at the …” could continue in different ways. Most people (76% in cloze probability norms) expect the word “beach” to come next. “Pool” is another plausible continuation, but one that is provided by only 2% of people. However, preceding the noun with an adjective (or adjectival noun), such as “neighborhood,” notably changes these expectations. After “neighborhood,” the cloze probability of “pool” increases to 59% and the likelihood of “beach” is correspondingly reduced (whereas the adjective “sandy” increases the cloze probability of “beach” to 98%). The degree to which predictions for likely nouns are being updated when the adjective is encountered can be quantified (e.g., using Kullback-Leibler divergence), allowing an examination of the extent to which the N400 elicited by the adjective varies with the prediction error (about the noun) that it gives rise to.

The amplitude of the N400 elicited by the nouns, once they occurred, patterned with the (off-line) post-adjective cloze probabilities, showing that information from the adjective indeed affected online semantic processing. However, notably, N400 responses at the adjectives themselves were not sensitive to either the direction or amount of updating in the probabilities of the upcoming noun. Although measures of (message-level) prediction error did not correlate with N400 amplitudes, there was an impact of the semantic information directly provided by the adjectives, with smaller N400s observed for adjectives with higher (model-estimated) cloze probabilities. Thus, adjectives show the pattern attested for all words—arising from basic comprehension mechanisms—with N400 amplitudes graded by the amount of new semantic information those words directly activate, without the need to assume prediction at all. Szewczyk and Wodniecka (2020) similarly found that N400 amplitudes are not sensitive to the amount of prediction error (about upcoming nouns) caused by grammatical markers on adjectives. Findings like these cast doubt on the hypothesis that the N400 arises from prediction error, at least if that error is presumed to be computed at the level of the unfolding message.

Moreover, recall that factors beyond contextual predictability importantly modulate the amplitude of the N400. Different inputs elicit different baseline N400 amplitudes; e.g., words with larger orthographic neighborhoods elicit larger N400s whether or not they are predictable in their contexts (Payne & Federmeier, 2019). One could perhaps argue that neighborhood size effects on the N400 also reflect a kind of error, with larger responses indexing greater uncertainty about the form and/or meaning of the input. However, at minimum, this would seem to be a different kind of error from that arising because of incorrect context-based expectations, creating a complex attribution problem: An accurate context-based prediction for a word from a high density neighborhood can yield an N400 of similar amplitude as that observed for a contextually unexpected word from a low density neighborhood. The multiplicity of factors (e.g., neighborhood density, repetition, contextual fit of varying types and levels) that are known to affect N400 amplitude thus raises questions about the feasibility of the N400 serving as a univariate error signal that can shape processing and learning (as in, e.g., Fitz & Chang, 2019).

Finally, when considering any possible mechanistic role for the N400, it is important to note that ERPs provide a specific view of neural activity, which—however useful to us as experimenters—may not necessarily always reflect what is most critical to the processing system itself. By their nature, ERPs reveal only changes in neural activity levels, and, further, only those changes that are aligned in time and phase to the events that we examine. Therefore, the fact that N400 amplitudes correlate with the amount of new semantic information becoming active, rather than with something like absolute levels of activity in the semantic system, is a property of the measure and doesn’t necessarily mean that it is these new semantic activations that matter most for neural processing at any given time. After all, the activations revealed in the N400, especially to a stimulus occurring in a rich context, may reflect a relatively small modification to a complex landscape of ongoing activity. Indeed, it is unclear whether the view of processing afforded by the scalp-recorded N400 is even available within the system itself. As I already discussed, the N400 has been linked to activity changes across a broadly distributed brain network. The large size of the network might be an important part of the reason that this activity is visible in the scalp-recorded EEG. However, by the same token, it may be unlikely that this distributed change can be monitored, and hence appreciated, from anywhere within the brain itself. Thus, it would be a mistake to assume that the view of semantic processing that is obtained at the scalp must highlight those factors with the most significance for underlying processing mechanisms or with the largest implications for downstream neural processing or behavior.

In summary, although N400 amplitudes do pattern with stimulus predictability, there is not strong evidence for the claim that it is prediction error as such that generates the N400 nor that the activity measured in the N400 can or does serve the system as a signal of prediction error. Instead, prediction is but one of many factors that can—but does not always—affect semantic access and, hence, the N400.

4.2 |. Selecting

One impact of active comprehension, as just discussed, is that it brings online additional information associated with anticipated upcoming stimuli. Importantly, though, active comprehension isn’t just about adding activation; it also entails enhanced selectivity. Again, it is useful to consider the differing constraints imposed by comprehension (perception) and production (action). As we saw, at its core, initial semantic access is relatively non-selective, allowing many pieces of information to become active in parallel, including multiple associations with the same percept. This allows the comprehension system to be resilient to noise and flexible in the face of uncertainty. However, this approach is ill-suited to production, which requires the creation of a targeted action plan to say a particular word (e.g., “cup” or “mug”, but not both) at a particular time. In the context of production—and thus, I would argue, also active comprehension—the brain needs to use the landscape of activation across semantic memory to derive representations that are more discrete and then select among those active representations to meet task goals.

During comprehension, mechanisms that enact selection might particularly tend to be invoked when incoming stimuli activate multiple associations, especially if those associations are in some way contradictory (i.e., have different implications for interpretation or for other task goals)—for example, in the case of words that are semantically ambiguous. Decades of studies using manual response and eye-movement measures have established that semantically ambiguous words tend to activate their multiple meanings, even when disambiguating context information is available (e.g., Duffy et al., 2001). In ERP measures, words that are class ambiguous (can be either nouns or verbs) have been found to elicit a frontally-distributed negativity, beginning around the same time as the N400 (Federmeier et al., 2000). The frontal negativity arises only when these class-ambiguous words are semantically distinct across their noun and verb senses (e.g., “duck”; Lee & Federmeier, 2006) and when meaning selection is hard because it must be done based only on syntactic cues (e.g., “to/the duck”; Lee & Federmeier, 2009). This effect seems to reflect the recruitment of mechanisms important for meaning selection (see also similar effects elicited by referential ambiguity—NRef effects—reviewed in Van Berkum et al., 2007), perhaps including the maintenance of multiple candidates in working memory to allow selection (Chou et al., 2014; Kluender & Kutas, 1993; Ruchkin et al., 1992; Wlotko & Federmeier, 2012b).

The presence and size of the frontal negativity predicts how successfully the contextually irrelevant meaning of the ambiguous word is ultimately suppressed, as measured by ERPs downstream of the ambiguity. For example, having read, “Ben tried to duck in the …,” correct disambiguation of the ambiguous word will render “alley” more plausible than “dish” as a continuation (and vice versa, after instead reading “the duck” in the same sentence context), which should yield an N400 difference. Indeed, the size of the frontal negativity to “duck” predicts the size of the subsequent N400 plausibility effect at the noun in the prepositional phrase (Lee & Federmeier, 2012). In particular, when the frontal negativity is smaller (and, presumably, meaning selection is less successful), the downstream response to the noun that should be implausible (and thus should elicit a large N400) instead elicits a facilitated N400, similar to that elicited by the contextually plausible noun. Therefore, the activity measured in the frontal negativity is part of a mechanism that is recruited to help select among possible meanings, allowing the suppression of those that are contextually inappropriate.

This mechanism has impact beyond meaning selection. When people are asked to read the same materials naturally, they fixate for a longer time on the ambiguous words compared to matched, unambiguous ones. This slowing of eye movement behavior occurs in a similar time frame as the frontal negativity (on readers’ first fixations onto the class ambiguous words) and similarly predicts the size of the downstream plausibility effect in readers’ gaze durations on the subsequent nouns (Stites & Federmeier, 2015; Stites et al., 2013). Thus, the mechanism that is recruited for meaning selection (which, in both language comprehension and production tasks, has been associated with a brain network that includes the left inferior prefrontal cortex; see, e.g., Schnur et al., 2009; Thompson-Schill et al., 1997), also seems to cause inhibition in the oculomotor system, rapidly altering eye movement patterns during natural reading.

Notably, the recruitment of these mechanisms is not a pervasive characteristic of comprehension, even in connection with ambiguity resolution. When semantic context information (e.g., from the preceding sentence) is available to aid meaning disambiguation, neither the frontal negativity nor the corresponding slowing of first fixations is observed, even for the same ambiguous words under the same task conditions (Lee & Federmeier, 2009; Stites et al., 2013). It is clear that the contextually irrelevant meaning of the ambiguous word is still activated to some degree in this case, because N400s are larger to ambiguous compared to matched unambiguous words, especially in contexts biased toward the words’ subordinate meaning. This pattern is consistent with the general finding of larger N400 amplitudes to wordforms that engender broader activation in memory because they have, e.g., more neighbors or, as here, more meaning features. But, as we have seen in other cases, the fact that multiple things are active at once does not, in and of itself, entail competitive interactions between them. When the semantic bias provided by the context is strong enough, the comprehension of even ambiguous words can proceed without the need to recruit additional mechanisms (i.e., with contextual support, semantic activations may naturally settle into a stable state, even for ambiguous words; see, e.g., Rodd et al., 2004).

The tendency to engage selection also varies with age and individual difference factors in a pattern strikingly like that seen for the engagement of prediction. As a group, older adults are less likely to successfully recruit these mechanisms. They elicit reduced or absent frontal negativity and first fixation slow-down effects (Lee & Federmeier, 2011; Stites et al., 2013) and show corresponding reductions in the downstream plausibility effects that depend upon successful selection (Lee & Federmeier, 2012). However, these patterns are again modulated by verbal fluency: Individuals with higher fluency show larger frontal negativity and first fixation effects and correspondingly increased evidence of selection. Moreover, the natural reading data show that when comprehenders fail to engage selection mechanisms upon initially encountering an ambiguous word, they are more likely to come back and reread it later (Stites & Federmeier, 2015; Stites et al., 2013). Thus, there are multiple possible pathways to successfully building an understanding of an unfolding sentence, and a key part of active comprehension is likely to be strategically choosing among available approaches and flexibly adapting comprehension mechanisms in the face of resource limitations or errors.

4.3 |. Revising

In comprehension, patterns of co-activation with potentially conflicting implications can also arise when context-based activations, including information that has been predicted, diverge from those elicited by an incoming word. Consider the pair of example sentences I used in Section 3.2:

Strongly constraining:

When the two met, one of them held out his badge. (The most expected completion is “hand” with 91% cloze probability.)

Weakly constraining:

Sandy always wished that she’d had a badge. (The most expected completion is “dog” with 24% cloze probability.)

In the case of the strongly constraining sentence, we know that predicting “hand” will create activations robust enough to linger for several more sentences, even when, as here, that prediction is disconfirmed (cf., Rommers & Federmeier, 2018a). Information associated with both “badge” and “hand” would then be active in parallel, and, importantly, these words/concepts have differing implications for an understanding of the event being described. Thus, we might expect that the system will need to somehow resolve this discrepancy. As already discussed, the N400 to “badge” in the strongly constraining sentence is not different from that in the weakly constraining one (which does not promote the prediction of something different)—that is, these activations can co-exist mostly independently in semantic memory. However, other aspects of processing do differ. One such difference is that the unexpected words in the strongly constraining contexts yield a frontally distributed post-N400 positivity compared to when those same words end weakly constraining sentences (Federmeier et al., 2007; Figure 3a). Given that the words have the same cloze probability in the two contexts, the difference between them must be related to whether something else had been predicted instead. Thus, although the precise functional correlates of the anterior positivity remain unclear, it seems to reflect activity associated with some kind of revision process.

FIGURE 3.

Plotted (negative voltage up) at a left, medial, prefrontal channel are ERP responses to prediction violations (plausible but unexpected endings of strongly constraining sentences; solid lines) and to those same words as unexpected endings of weakly constraining sentences (dotted lines). In both cases, the unexpected words were not semantically related to the expected completions (not plotted). In (a), young adults read the sentences for comprehension (Federmeier et al., 2007). Prediction violations elicited a post-N400 anterior positivity. Older adults (d) fail to show this effect (Wlotko et al., 2012). In (b), young adults read the same sentences with the same endings, interspersed with a third condition in which sentences ended with synonyms of the expected completion. In the presence of unexpected form information that did not signal a corresponding change in sentence meaning, young adult comprehenders did not elicit the anterior positivity to the prediction violations (unpublished data). However, in (c), when a lexical decision task was used, creating a focus on word processing as opposed to sentence comprehension, young adults again elicited the anterior positivity, even with the synonyms in the stimulus set (unpublished data). The difference between the patterns in (a) and (b) was replicated in a within-subjects version of the task, in which, in (e), the synonym endings were present only in the second half of the experiment, so that the first half was a replication of Federmeier et al. (2007) or, in (f), in which the same blocks of trials were instead preceded by the blocks with the synonym endings, replicating the lack of anterior positivity seen in (b)

The anterior positivity has been observed in a variety of situations that have in common (1) that a semantic context has been established and (2) that the incoming information is plausible in that context (implausible information is instead associated with post-N400 positivities with a posterior distribution; see review by Van Petten & Luka, 2012) but (3) that there is a strong competitor. In addition to being elicited in response to plausible prediction violations in sentences, as described already (see also, e.g., Delong et al., 2011; Kuperberg et al., 2020; Moreno et al., 2002), the anterior positivity has been observed in more impoverished contexts, such as when a category cue (e.g., “a type of fruit”) is followed by an exemplar that is atypical (“kiwi”; Federmeier et al., 2010) or that is generally typical (“apple”) but is unexpected for a particular speaker, given what the comprehender knows about that speaker (e.g., she works at a kiwi farm; Ryskin et al., 2020). It has also been observed to words that are unexpected because social norms make it unacceptable for someone to tell a hurtful “blunt truth” as opposed to making up a white lie (Moreno et al., 2016). Importantly, the N400 response to the anterior-positivity-eliciting words is quite different across these examples: N400 responses are large to plausible, unexpected words in sentences, are intermediate in size to atypical category exemplars, and are small to typical category exemplars or the information conveying a blunt truth. Thus, it is not the degree to which features of the incoming word are or are not already active in semantic memory that matters for eliciting this effect. Instead, the anterior positivity seems to arise when, whether because of long-term knowledge structures or because of the local linguistic or social context, a different representation is also highly active, and, thus, successfully incorporating the incoming word into one’s mental model requires the reprioritization of meaning-based information.

Like other effects associated with active comprehension, the anterior positivity is not ubiquitous. Older adults as a group do not show the effect, either to prediction violations in sentences (Figure 3d) or to atypical category exemplars, but, again, those with higher verbal fluency do (Federmeier et al., 2010; Wlotko et al., 2012). For young adults, the anterior positivity effect is sensitive to the task environment (e.g., Brothers et al., 2020; Federmeier & Wlotko, 2014). Figure 3b shows responses from an experiment in which strongly and weakly constraining sentence contexts were not only completed with expected and unexpected-but-plausible endings (as in the example above), but sometimes also with (near) synonyms of the most expected word (e.g., “When the two met, one of them held out his palm”). Strikingly, in this case, young adults stopped eliciting the anterior positivity to the prediction violations like “badge”, presumably because prediction of particular words was discouraged when the task prioritized comprehension but readers could not use the presence of unexpected form information as a reliable signal about whether or not their semantic expectations were met. We replicated this cross-experiment pattern in a follow-up study in which the unrelated unexpected (prediction violation) and synonym endings were blocked. When comprehenders were first given blocks with prediction violations but no synonyms, they elicited an anterior positivity, replicating the Federmeier et al. (2007) study (Figure 3e). However, when comprehenders read those same prediction violations after first having encountered blocks with synonyms, they (again) did not elicit an anterior positivity (Figure 3f). In another line of work, we changed the task demands, shifting them away from understanding the meaning of the sentence and toward word-level processing by asking participants to make speeded lexical decision judgments on the sentence-final words. Under these conditions, young adults elicited the anterior positivity to the unexpected words, even with the synonyms in the stimulus set (Figure 3c). In this case, being able to sometimes predict the final word was likely advantageous for succeeding at the lexical decision task. The pattern across these studies provides a striking example of the flexibility with which comprehension strategies can be adapted to the stimulus and task environment.

Even beyond the level of task, the anterior positivity varies across people and trials. Comprehenders who are allowed to self-pace their word-by-word reading tend to slow down when they encounter the unexpected but plausible words in strongly constraining sentences. This slow-down manifests on the tau parameter of the ex-Gaussian distribution, meaning that the prediction violation condition has a larger number of trials with very long response times. The ERP response to trials in that right tail of the distribution is characterized by the presence of an N2, peaking between 250 and 300 ms, which, as mentioned in Box 1, has been linked to cognitive control processes including response inhibition. Thus, on a subset of trials, young adults seem to quickly appreciate that the incoming word is not what they predicted, and they exert control to overcome the prepotent button press response so that they can spend more time reading that word. Strikingly, the anterior positivity is instead elicited to the prediction violation trials that did not manifest a slowed behavioral response (or an N2), suggesting that this effect is part of a more reactive strategy supporting revision (Payne & Federmeier, 2017b). In passive RSVP reading tasks a similar trade-off between reactive and proactive strategies can be seen across individual participants. Some participants elicit the anterior positivity as well as responses associated with conflict monitoring (increases in frontal theta activity) when encountering a prediction violation. Other participants are less likely to show these effects and instead seem to use predictive context information to proactively adjust attention, indexed as changes in alpha activity prior to the sentence-final word (Rommers et al., 2017).

Indeed, the anterior positivity shows a complex and intriguing relationship to attention. In young adults, anterior positivity responses are not diminished when attention is divided between reading for comprehension and monitoring a visual display (Hubbard & Federmeier, 2021a). Similarly, instructing young adults to try to explicitly predict the sentence-final word and to indicate with a button press whether or not their predictions were met does not augment the anterior positivity (Lai et al., submitted). Thus, when predictive strategies are being used, the size of the anterior positivity effect does not seem to be modulated by resource availability or explicit “effort” invested in prediction. However, the anterior positivity is not observed to stimuli presented outside the focus of attention. It is not seen to prediction violations when these are in parafoveal preview (Payne et al., 2019; Stites et al., 2017). Similarity, in visual half-field designs, prediction violations do not elicit anterior positivities with presentation to either the right or the left visual field (Wlotko & Federmeier, 2007); in unpublished data (described in Wlotko & Federmeier, 2008), we also showed that when trials presenting the sentence-final word either centrally or to both visual fields simultaneously were intermingled, the anterior positivity was only observed for central presentation and not when the words were lateralized (albeit going to both hemispheres at once). Thus, the anterior positivity indexes processing that seems to require central/foveal attention.

4.4 |. Sequencing and enriching

At its core, producing language—and, correspondingly, making predictions during comprehension—involves not only activating and selecting information but also sequencing it, so that the system can prepare the appropriate series of actions. This is a general property of action planning: To be effective, actions have to be spatially and temporally coordinated with one another and with objects in the world. This kind of sequencing is different from more generally activating information related to the overall schema or event evoked by a context. The difference between planning a sequence and activating an event representation can be seen very strikingly in patterns of hemispheric differences.

In the related anomaly paradigm discussed earlier, comprehenders encounter items that do not fit the context overall but that share semantic features with a stimulus that was likely to appear at that position in the sequence. We saw already that, when comprehenders are predicting, these within category violations elicit facilitated N400s compared to the equally unexpected between category violations, which do not share as many features with the expected word. Figure 2 (right-most column) shows young adults’ responses to the three types of sentence endings in this paradigm when those endings were presented in the right visual field (to bias processing toward the left hemisphere) or in the left visual field (to bias toward right hemisphere processing) (Federmeier & Kutas, 1999b). In all cases, the context words were presented in central vision. As already discussed, both hemispheres manifest the comprehension universals and thus show the basic cloze probability effect: Expected items are facilitated compared to between category violations and, in fact, the size of this effect is similar across visual field. However, the pattern of response to the within category violations notably differs in the two visual fields. When processing is biased toward the left hemisphere, we see the prediction-based similarity effect, with facilitated N400s to within compared to between category violations. Instead, with presentation biased to the right hemisphere, N400 amplitudes pattern entirely with the direct fit of the item to the context; i.e., the right hemisphere shows the pattern seen for groups or task contexts wherein prediction is less likely. Thus, prediction-based facilitation for the related anomalies seems to arise from left hemisphere processing mechanisms. Indeed, older adults, who are less likely to elicit prediction-based effects overall, also do not exhibit this hemispheric asymmetry, instead showing a simple effect of cloze probability for stimuli presented in either visual field (Federmeier & Kutas, 2019). In young adults, asymmetric prediction-based effects are also observed when the target concepts are instantiated as pictures rather than words (Federmeier & Kutas, 2002), revealing that this is not a language-specific hemispheric difference but a more general processing asymmetry. This line of work thus demonstrates that pre-activation of the semantic features of a likely upcoming item is accomplished through the engagement of left hemisphere processing mechanisms—arguably, mechanisms important for language production.

Now, consider a different kind of unexpected word. Metusalem et al. (2012, 2016) presented comprehenders with short vignettes, such as “A huge blizzard swept through town last night. My kids ended up getting the day off from school. They spent the whole day outside building a big ….” Here, “snowman” is the expected completion. The word “jacket” is obviously anomalous in this position in the sentence—and, different from the within category violations, it also isn’t related to snowman. Instead, jacket is a word that fits with the general event evoked by the context but that isn’t occurring in a sensible part of the unfolding language sequence. Presented centrally, event-consistent words like jacket elicit facilitated N400 responses compared to anomalous words that aren’t related to the event (e.g., “towel”; Metusalem et al., 2012). However, lateralizing the critical words reveals that the mechanisms responsible for this event-related facilitation are different from those causing facilitation in the related anomaly paradigm. In this case, the two types of anomalous words elicit identical N400 responses when processing is biased toward the left hemisphere. Instead, it is the right hemisphere that shows facilitation for a word that is related to the larger event but that is occurring at an infelicitous position within the unfolding language sequence (Metusalem et al., 2016).

Taken together, these differing patterns clearly illustrate that comprehension networks can have different processing dynamics—even within the same brain! Both the left and the right hemispheres build a meaning representation from language input (and more generally). Processing in both of these networks is characterized by the comprehension universals that I discussed in Section 3, such that both hemispheres consistently differentiate stimuli that fit the current context from those that do not (see also Coulson et al., 2005; Federmeier et al., 2005; Kandhadai & Federmeier, 2008; reviewed in Federmeier et al., 2008). However, in predicting and pre-activating the features of items likely to come next, the left hemisphere network prioritizes the input sequence. Correspondingly, it will show facilitation for—in a sense, be “lured by”—event-inconsistent information if that item shares features with the immediately-anticipated stimulus. At the same time, this kind of network is less attuned to event-congruent information if it is encountered out of sequence. In contrast, in the right hemisphere network, information associated with the larger event is active even when that information does not fit well in a particular part of the language sequence.

Note that, although sequential prediction in particular is linked to left hemisphere processing strategies, I am not claiming that the left hemisphere mediates all forms of active comprehension. The right hemisphere engages other kinds of non-obligatory processes that serve to enrich comprehension. For example, the ERP response to concrete words or phrases, such as “green book,” differs from that to abstract words or phrases, even for just a different sense of the same noun: “interesting book”. Part of this ERP concreteness effect is a sustained, frontally distributed negativity beginning around 300 ms post-stimulus-onset that has been linked to the formation of mental images in response to the concrete expressions (West & Holcomb, 2000; cf., Farah et al., 1989). Visual half-field studies have revealed that this effect is linked to right hemisphere-biased processing (Huang et al., 2010). Like other forms of active comprehension, the concreteness effect is non-obligatory, varying with stimulus type (Lee & Federmeier, 2008) and task demands related to imagery (Gullick et al., 2013), and, again, is reduced with normal aging (Huang et al., 2012; Lucas et al., 2017, 2019). Thus, both hemispheres augment basic comprehension with more active processes that, on the one hand (hemisphere!), enable perceptual simulation to enrich the developing situation/event model, or, on the other, serve to sequence and more generally structure the input to afford prediction.

5 |. MULTIPLE MODES OF COMPREHENSION

Patterns of hemispheric differences emphasize how active comprehension entails a network-wide change in processing dynamics (compared to those arising from passive comprehension alone). In young adults, who show lateralized patterns like those just described,⁴ left and right hemisphere comprehension networks diverge in how they activate information over time and across a context. Consequently, they deal differently with both expected and unexpected incoming information, and they will make different kinds of errors. Moreover, these processing differences can have long-lasting consequences in terms of downstream memory and learning.

For example, predicting affects memory in multiple ways. I opened Section 4 of this article by describing an experiment in which comprehenders read sentences like “Be careful, because the top of the stove is very ….,” which led them to predict the word “hot”. The results showed that prediction-related activation for “hot” lingered, even though participants had read a different sentence ending instead (Rommers & Federmeier, 2018a). Not surprisingly, this lingering prediction-related activation can also cause people to make memory judgment errors, such that they misclassify something that they only predicted as something they actually encountered (Hubbard et al., 2019). However, importantly, the impact of prediction on memory goes beyond these kinds of confusions. Consider what happens in a different version of this paradigm (Rommers & Federmeier, 2018b), in which comprehenders’ predictions are confirmed, rather than violated—that is, in which the sentence above ends with the predicted word “hot” and that critical word is then repeated several sentences later, as the ending of a weakly constraining sentence. As before, we used as a comparison the case in which “hot” initially appears in a weakly constraining context and is then repeated, as well as a case in which “hot” is only presented once.

First, when we look at the response to the initial presentation of the critical words, we replicate the well-established cloze probability effect: When comprehenders encounter a high cloze probability ending of a strongly constraining sentence (e.g., “hot” in the sentence above), they elicit reduced N400s compared to when that same word initially appears as the (low cloze probability) completion of a weakly constraining sentence. Of critical interest, then, is what happens in these two cases when the word “hot” is seen again, as the ending of a (different) weakly constraining sentence. As before, repetitions of initially unpredictable words show a basic N400 repetition effect, with smaller N400s on second presentation compared to when those words were seen only once. What about the condition in which the word being repeated was previously predictable? Given that prediction alone can cause a (pseudo-) repetition effect, it seems logical to hypothesize that there might be a particularly large facilitation for a word that was predicted, was then actually seen, and now is being seen again. However, notably, that is not what the data show. Instead, although previously predicted words do show a repetition effect compared to words seen only once, the impact of repetition is reduced compared to that found for repeated words that were not predictable when they were first encountered (Rommers & Federmeier, 2018b). That is, activations for words that were initially predictable are not as robust downstream … but why?

As I have overviewed throughout this section, one consequence of predicting is that it tends to ease stimulus processing when a predictable stimulus is actually encountered; that is part of the reason for the large N400 facilitation at the first presentation of “hot”. Stimulus processing is eased because a system that is predicting is essentially shifting its priorities away from the analysis of incoming sensory information and toward information that its current model of the world makes probable. Accordingly, when the word “hot” is presented for the first time in a predictive context, the system need only process the incoming word enough to confirm that the sentence seems to be unfolding as expected. Indeed, although a direct experimental comparison would be necessary to confirm this, it is striking that the size of the N400 repetition effect for words that confirm a prediction (0.62 μV in Rommers & Federmeier, 2018b) is very similar to that for that for the (pseudo)repetition effect obtained for words that were only predicted and never actually viewed (0.7 μV in Rommers & Federmeier, 2018a; in both studies, basic repetition effects for unpredictable words were the same size: 1.3 μV). It seems that, when a comprehender is predicting, actually encountering a predictable word may not augment the activation of its representation beyond what had already been achieved by predicting it. And this is true despite the fact that prediction alone does not activate the word’s representation to the same degree as perceiving it in the absence of prediction. Thus, in reducing the “work” associated with stimulus processing, prediction also reduces the strength of encoding and affects later memory.

Studies of hemispheric biases in memory provide corroborating evidence for this tradeoff. In one series of experiments, we presented single words to the left or right visual field and then tested recognition memory for those words (presented centrally) at varying lags, ranging from immediate repetition to repetitions after 50 intervening words (Evans & Federmeier, 2007; Federmeier & Benjamin, 2005). Overall hit rates were higher for words that had initially been presented in the right visual field, consistent with general left hemisphere advantages for word recognition (Jordan et al., 2003). However, for correctly recognized words, hemispheric biases in behavioral and ERP indices of memory interacted with lag. At short and medium lags (under 20 intervening words), ERP repetition effects were of similar size for words studied in the two visual fields, although reaction times were shorter for words that had initially been presented to the left hemisphere. At long lags, instead, ERP repetition effects were larger for words initially presented to the right hemisphere, and reaction times to these items were also shorter. Thus, right hemisphere processing mechanisms seem to be especially advantageous for maintaining word information over time. Moreover, even at short lags, other work has found right hemisphere advantages for retaining form-specific information about words and other stimuli (Marsolek & Burgund, 1997) and distinguishing items that were experienced from semantically similar lures (Fabiani et al., 2000). Thus, in patterns of hemispheric asymmetry, we again observe the differing strengths and weaknesses of networks that prioritize the use of a mental model to make goal-directed predictions, enhancing the efficiency of stimulus processing in context (e.g., left hemisphere comprehension: Federmeier, 2007; Kandhadai & Federmeier, 2010b; Wlotko & Federmeier, 2013) versus networks that, instead, prioritize the accurate analysis of incoming stimuli (e.g., right hemisphere comprehension: reviewed in Evans & Federmeier, 2008), with corresponding benefits for veridical memory (see Avery et al., 2012 for an example of a neural model that instantiates this kind of tradeoff).

Overall, then, hemispheric asymmetries, aging effects, and variability observed across individuals, tasks, and stimulus configurations provide compelling evidence that there are multiple processing modes that can underlie comprehension. These different kinds of comprehension arise from different types of network dynamics and yield different processing outcomes at multiple time scales and levels of analysis. In particular, across this article I have focused on the distinction between a more passive form of comprehension that I refer to as connecting and comprehension characterized by a set of more active processes involved in what I call considering.

5.1 |. Connecting

On the one hand, as explicated in Section 3, through studies using the N400 we have begun to build an understanding of fundamental mechanisms that allow the brain to rapidly and efficaciously map between sensory information and knowledge, allowing the language we take in and the world we experience to be imbued with meaning. The view provided by the N400 is particularly valuable because the N400 is one of few measures we can use to examine comprehension when there are no button presses to be executed, no verbal responses or eye movements to be planned, and no explicit judgments to be made. With ERPs, we have been able to follow the progression of perceptual processing to memory access when the system does not need to do all the things it would have to do to act in the world … when simple, “passive” comprehension is sufficient.

Under these conditions, the brain’s priority is to activate information associated with incoming stimuli. Given that the environment is dynamic and complex, the brain needs to make these connections in a way that is robust to noise and the possibility of misperception. To do this, it naturally takes advantage of context, experience, and statistical regularities in its environment. Even passive comprehension is not strictly or even primarily “bottom-up” in nature; top-down and recurrent connectivity are central to these network dynamics. However, importantly, this mode of comprehension prioritizes graded, parallel processing. In doing so, the brain can avoid over-commitment and thereby remain flexible enough to deal with uncertain environments, low probability events, and errors.

Passive comprehension takes advantage of time, using it to create a link between a given sensory input and its distributed associations in memory—a type of implicit, stimulus-anchored binding. The time-based binding we can observe in the N400 is heterogeneous and ephemeral but nevertheless functional and, importantly, not resource-demanding. Moreover, the resulting activation states linger long enough to allow for the accumulation of information as people look around, read, or listen to an unfolding sentence, affording meaning-based representations that encompass more than just a single input. Therefore, this process creates what we might think of as a dynamic spatial and temporal window of emergent meaning, providing us with a basic representation of semantics from sentences and scenes, continuously and relatively effortlessly.

Vision scientists have talked about related processes unfolding in perception. Over the first few hundred milliseconds after the apprehension of a sensory stimulus, the pre-attentive processing of perceptual information, which Rensink (2000) refers to as “sensing,” creates a distributed landscape of activation across visual processing areas. Shaped by top-down and recurrent connectivity, these activations coalesce into representational structures that have sometimes been termed “proto-objects” (reviewed in, e.g., von der Heydt, 2015). Similarly, the processing reflected in the N400 creates massively parallel, graded activation states in semantic memory, yielding what we might then describe as “proto-concepts”. In some ways, this insight follows directly from the claim that semantic activations do not wait for some discrete outcome in perception to be put into play—that is, that semantic access does not depend upon recognition. Instead, the activation states that make up visual proto-objects (and their equivalents in other sensory modalities) necessarily propagate to multimodal/abstract knowledge stores and create a linked field of structured activations there. Thus, connecting arises naturally from sensing. The precise nature of the functional networks that establish proto-objects and proto-concepts (when they are formed, how long they last, what “kind” of information they contain, and the exact neural firing patterns that afford them) shifts as sensing develops into connecting (see, e.g., Federmeier et al., 2016; Schendan, 2019). Ultimately, however, both sensing and connecting are part of a sensory-evoked processing cascade that reflexively links perception to memory and yields dynamic representations of the world around us. Importantly, the circuits that create proto-objects and proto-concepts also provide a ready interface for attention (Kastner & McMains, 2007; Qiu et al., 2007), thereby permitting processing to be further shaped in task- and goal-specific ways.

5.2 |. Considering

To allow action in the world, the massively parallel and graded activation states afforded by connecting must be pared down, so that particular words can be prepared for production in a specific sequence or particular objects reached for at specific locations. To do this, the brain needs to reshape activations in order to resolve competition and select, and, ultimately, to create different kinds of bindings—of the features that make up a concept, of form to concept, and of information to time and space. The neural processes involved in resolving competition, effecting selection, and creating more stable bindings are what we would likely broadly describe as attentional mechanisms.

In his account of visual processing, Rensink (2000) distinguishes “sensing” from what he refers to as “scrutinizing,” which involves deploying attention on the information that has been sensed in order to create stable perceptual representations. Similarly, I am proposing that comprehending the world involves not just the relatively effortless and unconscious process of connecting, but also non-obligatory, active comprehension mechanisms that I am grouping under the label “considering”. The active comprehension processes that I described in Section 4 are notable for being facultative—employed strategically and in a manner that seems to require resources that are not readily available to all comprehenders or under all circumstances. Moreover, the kinds of prediction, selection, revision, and sequencing processes that characterize active comprehension are associated with effects that also arise in other domains and that have been linked to general attentional and control systems (see reviews by Aydelott et al., 2005; Federmeier et al., 2020). For example, predictive language contexts yield decreases in alpha power leading up to a potentially predictable stimulus (Rommers et al., 2017)—the same kind of effects that have been linked more generally to attentional allocation (Haegens et al., 2012). Moreover, when people are predicting, stimuli (both words and pictures) in strongly compared with weakly constraining contexts yield differences on the P2 (Federmeier & Kutas, 2001; Federmeier et al., 2005; Huang et al., 2010; Kandhadai & Federmeier, 2010a; Wlotko & Federmeier, 2007); similar effects are seen during overt language production (Strijkers et al., 2010), and, more generally, P2 effects have long been associated with selective attention and target detection in visual search tasks (Luck & Hillyard, 1994). When predictions turn out to be wrong, there is an increase in frontal theta, which, again, across many non-language tasks has been associated with the need for control and shifts of attention (Cavanagh & Frank, 2014). Prediction violations are also one of the conditions that elicit the anterior positivity ERP effect already described, which, along with several other post-N400 effects (including posterior positivities associated with the appreciation of semantic anomalies and thematic role violations; see Leckey & Federmeier, 2020; Leckey & Federmeier, submitted; Payne et al., 2019), seems to only occur for stimuli within the central focus of attention.

One important component of active comprehension, therefore, is the use of attention to bind semantic information into a more stable unit—a conceptual variant of Rensink’s scrutinizing. However, the representations created by “considering” are not merely more stable. Critically, they are representations that have been shaped to afford action, allowing semantic information to be brought to awareness, ordered and structured, and, if desired, used to produce language or to control behavior. When it goes from connecting to considering, therefore, we observe the brain transitioning, perhaps through evoked or intrinsic fluctuations in functional connectivity networks (e.g., Cole et al., 2014; Gratton et al., 2018; Sadaghiani et al., 2010), from processes and representations that are centered on an apprehended stimulus to those that are centered on the current task situation and goal states. Through the engagement of attentional and control systems, the brain dynamically creates a representation that brings together curated, task-appropriate, goal-oriented information about form, semantics, and the kind of temporal, spatial, and syntactic (role-related) properties that are critical for sequencing and directed action.

6 |. CONCLUSION

Across this article, I have laid out my claim that the brain has multiple ways of making meaning. It leverages timing to connect—to create large-scale, temporary bindings that permit a continuous, flexible interplay between the world and our stores of knowledge. With the engagement of active comprehension mechanisms, the brain can then further consider these activation states, transforming them into a more restricted set of stable, multidimensional, spatially- and/or temporally-grounded representations … the kinds of representations that are necessary for action. Thus, the critical difference between comprehension and production is not, as we often tend to describe it, just about the direction of information flow; it is as much about the types of representations that the brain is creating and using to accomplish these goals. Production requires that we use attention in particular ways to create new kinds of representations. Active comprehension, which includes prediction but also much more, arises when those same attentional mechanisms are deployed in the service of comprehending.

In the end, then, what I hope to have conveyed is the idea that what we ultimately experience as words or other objects is an attentionally-dependent binding derived from a vast, spatio-temporal array of knowledge-infused activations. Our understanding of the world goes beyond what can be captured in that attentionally reshaped representation. But the stable binding is what allows those objects or words to be manipulated in the world or in our minds … to be grasped, literally or figuratively.