Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2016 Mar 19.
Published in final edited form as: Brain Res. 2015 Jan 16;1601:73–84. doi: 10.1016/j.brainres.2015.01.018

The notion of the motion: The neurocognition of motion lines in visual narratives

Neil Cohn 1,2, Stephen Maher 1,3
PMCID: PMC4376351  NIHMSID: NIHMS656601  PMID: 25601006

Abstract

Motion lines appear ubiquitously in graphic representation to depict the path of a moving object, most popularly in comics. Some researchers have argued that these graphic signs directly tie to the “streaks” appearing in the visual system when a viewer tracks an object (Burr, 2000), despite the fact that previous studies have been limited to offline measurements. Here, we directly examine the cognition of motion lines by comparing images in comic strips that depicted normal motion lines with those that either had no lines or anomalous, reversed lines. In Experiment 1, shorter viewing times appeared to images with normal lines than those with no lines, which were shorter than those with anomalous lines. In Experiment 2, measurements of event-related potentials (ERPs) showed that, compared to normal lines, panels with no lines elicited a posterior positivity that was distinct from the frontal positivity evoked by anomalous lines. These results suggested that motion lines aid in the comprehension of depicted events. LORETA source localization implicated greater activation of visual and language areas when understanding was made more difficult by anomalous lines. Furthermore, in both experiments, participants' experience reading comics modulated these effects, suggesting motion lines are not tied to aspects of the visual system, but rather are conventionalized parts of the “vocabulary” of the visual language of comics.

Keywords: Motion lines, comics, visual language, events, motion, P600, P300, Broca's area

1. Introduction

The depiction of motion poses a challenge for static images. Motion lines (also called action or speed lines) offer a solution to this issue by attaching lines to a moving object to show the path of an action (as in Figure 1). While motion lines are especially pervasive in the visual vocabulary used in comics across the world (Cohn, 2013a; McCloud, 1993), recent theories have hypothesized that their comprehension originates in the biological foundations of vision. Since moving objects leave behind “streaks” in the visual system when a viewer tracks an object (Geisler, 1999)—similar to a slow shutter speed of a camera—this residual is argued to form the basis of our understanding about motion lines (Burr, 2000; Burr & Ross, 2002). Research following this idea has stressed that participants can better understand or remember the direction of moving objects when they have motion lines than when they do not (Burr & Ross, 2002; Kawabe & Miura, 2006; Kawabe, Yamada, & Miura, 2007; Kim & Francis, 1998). Thus, under this interpretation, motion lines are an iconic depiction of a basic aspect of human perception, rooted directly in the visual system.

Figure 1.

Figure 1

Open in a new tab

Different uses of lines to depict motion.

Nevertheless, this “biological” origin for motion lines has several limitations. First, motion lines are understood by blind people comparably to sighted people when presented using raised-line pictures (Kennedy, Gabias, & Piertantoni, 1990). Second, people of cultures unfamiliar with this style of drawing have trouble understanding that these lines depict motion, though they do understand iconic representations (Duncan, Gourlay, & Hudson, 1973; Kennedy & Ross, 1975; Winter, 1963). Third, the interpretation of motion lines changes as people age (Carello, Rosenblum, & Grosofsky, 1986; Friedman & Stevenson, 1975; Gross et al., 1991; Mori, 1995; Nakazawa, 1998). Younger children often interpret motion lines as invisible yet iconic physical forces, such as wind or air moving, but only recognize them as symbolic conventions as they grow older (Gross et al., 1991). As children accept this symbolic meaning, they also rely less on postural cues to signify movement, which they do understand even at younger ages.

Fourth, motion lines vary in their representations cross-culturally, both in contexts like comics (Cohn, 2013a; McCloud, 1993), and in other drawing systems, like sand drawings created by Australian Aboriginals (Green, 2014; Munn, 1962). Fifth, motion lines in comics use a wide range of shapes, not only trailing laterally moving objects, but also showing manner of motion like bouncing or spinning (Figures 1c and 1d), which cannot resemble lateral motion streaks (Cohn, 2013a; McCloud, 1993). Lines can also be placed behind a moving figure to converge onto a single point (as in Figure 1b). These type of lines cannot appear in vision, but have been shown to be more effective at conveying motion than parallel lines (as in Figure 1a), which do occur in vision (Ito, Seno, & Yamanaka, 2010). Altogether, these reasons provide a strong argument against the view of motion lines being tied to basic aspects of the biological visual system.

Additional research has studied motion lines in relation to their depiction of events. In general, images with motion lines are thought to be better at depicting the idea of motion than those that rely on only postural cues (Brooks, 1977; Friedman & Stevenson, 1975; Gross et al., 1991; Ito et al., 2010; Kawabe & Miura, 2006), though together motion lines and postural cues clarify an expected path of an action more than each component alone (Kawabe & Miura, 2006). Also, the number and length of lines used in a representation may influence the perceived speed that they convey: more lines and longer lines lead to participants interpreting faster movement (Hayashi, Matsuda, Tamamiya, & Hiraki, 2012). Furthermore, motion lines that trail an object have also been rated as more effective at depicting motion than a lack of lines, background lines, or lines moving in the wrong direction (Ito et al., 2010). Motion lines may facilitate comprehension and memory of depicted events more than when images lack motion lines because they help clarify the interaction between entities that otherwise may remain underspecified (Brooks, 1977).

Altogether, these findings point towards an alternative account for the understanding of motion lines that is not tied to the visual system alone. Rather, drawings are written in a “visual language” similar in underlying cognitive structure to spoken languages (Cohn, 2013a). Just like different languages have words for expressing certain conceptualized meanings, visual languages use diverse ways to map graphic representations onto the same conceptualization of paths, a basic ontological category of the human conceptual system (Jackendoff, 1983; Mandler, 1992; Talmy, 2000). A “visual vocabulary” view helps account for why motion lines differ across cultures, need to be learned, and can be understood by blind individuals in raised relief images (Kennedy et al., 1990), who would still have basic knowledge of paths despite having no access to a visual vocabulary or vision.

1.1. The present study

Despite this substantive body of research, most prior studies of motion lines remain fairly limited. Even though motion lines appear ubiquitously in comics—and most studies make reference to this fact—few studies actually use this context. Stimuli typically have abstract circles or squares with a trailing motion line, focus specifically on the action of running, and/or only use straight lateral motions. However, in comics, motion lines accompany nearly any type of depicted actions, not just a prototypical lateral running figure (as in Figure 1). Also, contrary to studies using only straight lines for lateral motions, motion lines in comics may be curved, may spiral around in a circle, or may depict points along a path (such as when an object bounces or spins, as in Figure 1c and 1d). No work has previously studied the understanding of motion lines within the naturalistic context of comics—i.e., by examining comprehension of motion lines directly in a visual narrative sequence as opposed to an individual image.

Previous studies have also been limited in their experimental measurements and methodologies. Most studies of motion lines have focused on recall tasks and/or subjective ratings. Thus, despite the claims that motion lines may connect to basic perceptual processes (e.g., Burr, 2000; Burr & Ross, 2002; Kim & Francis, 1998), no studies directly examine the online comprehension of motion lines, whether with behavioral or neurocognitive measures.

In order to overcome these limitations, we carried out two experiments that analyzed participants' comprehension of motion lines embedded in the naturalistic context of comic strips. Our analysis focused on panels with 1) normal motion lines from the original panels of the comic, 2) panels with no motion lines, and 3) panels with anomalous motion lines where the direction of the original lines is reversed to be incongruous to their context (see Figure 2).

Figure 2.

Figure 2

Open in a new tab

Manipulation of normal depictions of motion lines to show either no lines or anomalous lines.

Experiment 1 used a “self-paced viewing” paradigm (Cohn, 2014; Cohn & Paczynski, 2013) that examined the viewing times to images as participants progressed through a visual narrative sequence frame-by-frame at their own pace. If motion lines facilitate event comprehension, panels with anomalous lines or no lines should have longer viewing times than those with normal lines, consistent with previous offline measurements (e.g., Brooks, 1977; Ito et al., 2010). Specifically, if longer viewing times appear for panels with no lines than those with normal lines, it would support that motion lines aid the comprehension of depicted events beyond postural cues and spatial relationships between characters and objects.

In Experiment 2, we measured event-related brain potentials (ERPs) to panels in these same sequences. If motion lines index biological aspects of perception, we might expect ERP effects on early components thought to index visual perception such as the P1 and N1, which have been shown to be modulated by motion stimuli (Anllo-Vento & Hillyard, 1996; Coch, Skendzel, Grossi, & Neville, 2005). However, if motion lines are connected to language processing, we might expect ERP effects in later time windows where manipulations to visual narratives and real-world events often evoke the same ERP effects as in language. For example, the “P600” or “Late Positivity” was originally thought to index syntactic processing in language, appearing as a posteriorly distributed positivity peaking around 600ms to words violating the grammar of sentences (Kuperberg, 2007; Osterhout & Holcomb, 1992). However, similar positivities appear when real-world visual actions are not carried out correctly (de Bruijn, Schubotz, & Ullsperger, 2007), or when objects and/or hand position mismatch their corresponding actions (Bach, Gunter, Knoblich, Prinz, & Friederici, 2009; Sitnikova, Holcomb, & Kuperberg, 2008b; van Elk, Bousardt, Bekkering, & van Schie, 2012). Posterior positivities also appear to incongruities of structure in scene processing of individual images (Võ & Wolfe, 2013). Such results are important for understanding motion lines because, as argued previously, they bind together the elements in an action to better clarify an event (Brooks, 1977). If P600 effects appear to incongruities of actions, altering the binding between objects created by motion liens may elicit a similar brain response.

Because P600 effects appear across domains, several researchers have posited that it reflects a domain-general process of attempting to integrate an input into a structure in the face of a prediction error, possibly causing an update or revision of the wider representation of context (Christiansen, Conway, & Onnis, 2011; Cohn, Jackendoff, Holcomb, & Kuperberg, 2014; Kuperberg, 2013; Sitnikova, Holcomb, & Kuperberg, 2008a; Võ & Wolfe, 2013). This general “prediction error” may also be connected to a more frontally distributed positivity appearing after 500ms from the stimulus onset (Kuperberg, 2013; Van Petten & Luka, 2012). This “frontal positivity” appears to violated lexical predictions—when a reader expects a specific word and gets another, related or unrelated word (Delong, Urbach, Groppe, & Kutas, 2011; Federmeier, Wlotko, De Ochoa-Dewald, & Kutas, 2007; Thornhill & Van Petten, 2012)—but has also been tied to failed semantic predictions more generally (Van Petten & Luka, 2012). If we think of motion lines as part of a “visual lexicon” (Cohn, 2013a), then violating the expectations of their appearance may evince a frontal positivity as well, as in our anomalous lines.

The notion of the P600 indexing general neurocognitive processing has been debated for some time, specifically in connection with the well-studied “P300” associated with unpredictable stimuli across modalities (Coulson, King, & Kutas, 1998; Friederici, Mecklinger, Spencer, Steinhauer, & Donchin, 2001; Osterhout & Hagoort, 1999; Van Petten & Luka, 2012). Indeed, many of the positivities discussed above have been interpreted as a late onset P300 (de Bruijn et al., 2007; van Elk et al., 2012). The P300 is the oldest ERP component recognized as relevant to cognition (Chapman & Bragdon, 1964), and is commonly divided into two types. The P300a typically has a more anterior distribution and is thought to be sensitive to task irrelevant “novelty,” possibly connected to top-down monitoring of attentional mechanisms (Polich, 2007) or the attentional demands initiated by aspects of a task (Pardo, Fox, & Raichle, 1991; Posner, 1992). The more posteriorly distributed P300b is sensitive to task relevant “oddballs,” that is, stimuli with relatively low probabilistic likelihood compared to more frequent stimuli, which may cause disconfirmation of expectations (Donchin & Coles, 1988). This effect may reflect attempts to update a mental model in memory (Polich, 2007), particularly with regard to task demands (Katayama & Polich, 1996).

Given this literature, we predicted modulation of this family of late positivities (P600/P300) to the violation of motion lines, reflecting the violation of event understanding and “visual lexical” predictions. In particular, an attenuated response to normal lines compared to no lines or anomalous lines would suggest that they facilitate event understanding. However, we did not necessarily predict the same pattern of responses to no lines and a reversal of lines, since these representations should differ in their congruity. We predicted that panels with anomalous lines should require additional processing compared to those with no lines, because anomalous lines would disrupt the event knowledge and should be overtly incongruous, which may be manifest as an “oddball” P300a. In contrast, the absence of motion lines should be less incongruous (no “oddball” response), but should still impact the understanding of the depicted events since no lines would bind together the component elements of the action.

Finally, because prior studies have indicated that motion line understanding requires some degree of learning (Carello et al., 1986; Friedman & Stevenson, 1975; Gross et al., 1991; Mori, 1995; Nakazawa, 1998), we reasoned that participants' experience with comics might affect their comprehension. Indeed, in previous work we found that “visual language fluency” (measured via a “VLFI score”, see the Methods section) correlates with both response times and ERP amplitudes to manipulations of visual narratives (Cohn, Paczynski, Jackendoff, Holcomb, & Kuperberg, 2012), and thus similar findings here would provide evidence against the view that motion lines tie directly to perception. Rather, modulation with VLFI score would support that motion lines are part of the conventionalized vocabulary of the visual language of comics.

2. Results

2.1. Experiment 1

2.1.1. Viewing times

On the whole, viewing times differed significantly between critical panels of all motion line types, F1(2,118)=11.30, p<.001, F2(2,178)=5.63, p<.005. As depicted in Figure 3, critical panels with Normal Lines were viewed significantly faster than those with No Lines, which in turn were faster than those with Anomalous Lines (all ts < -2.4, all ps < .05).

Figure 3.

Figure 3

Open in a new tab

Viewing times to critical panels containing either 1) normal motion lines, 2) no motion lines, or 3) anomalous motion lines. Error bars represent standard error.

2.1.2. VLFI score

A near significant positive correlation between viewing times to panels with Anomalous Lines and participants' VLFI score, r(58)=.252, p=.053, suggested that the more experience participants had reading comics, the slower they viewed panels with Anomalous Lines.

2.1.3. Ratings

While all sequences were rated as highly coherent (all greater than 4.0 out of 5), ratings for sequence types significantly differed from each other in the subjects analysis, F1(2,118)=4.57, p<.05, but not the items analysis, F2(2,178)=2.08, p=.129. Sequences with Normal Lines were rated as more coherent (M= 4.18, SD=0.58) than those with No Lines (M=4.06, SD=0.55) and Anomalous Lines (M=4.03, SD=.66), all ts > 2.5, all ps < .05. However, ratings to sequences with No Lines and Anomalous Lines in critical panels did not differ, t1(59)=.328, p=.744, t2(89)=.447, p=.656.

2.2. Experiment 2

2.2.1. Event-Related Potentials

Between 200 and 400ms, interactions appeared between Line Type and Region in the midline and peripheral regions. Follow up ANOVAs showed main effects of Line Type in the Occipital and Left Posterior regions (all Fs > 3.02, all ps < .05), and a trending main effect of Line Type in the Parietal region, F(2,46)=3.02, p=.059. Overall, these findings suggested positive deflections between panels with Normal Lines and those with No Lines or Anomalous Lines (see Table 1 and Figures 4 and 5).

Table 1.

Results of ANOVAs comparing each sequence type at the Critical Panel.

200-400 400-600 600-900
Midline Peripheral Midline Peripheral Midline Peripheral
All Sequence Types
 LT 1.77 1.24 2.19 3.6* 1.68 2.63˄
 LT × R 2.57* 2.73˄ 2.38* 2.17 2.27* 2.6˄
 LT × H 0.61 0.227 0.203
 LT × R × H 0.008 0.253 0.64
Normal - No Lines
 LT 2.51 1.53 0.803 0.78 0.302 0.148
 LT × R 2.49* 3.23˄ 2.17˄ 0.92 1.83 2.27
 LT × H 0.067 0.29 0.23
 LT × R × H 0.009 0.052 0.058
Normal - Anomalous
 LT 1.82 1.44 3.22˄ 5.26* 2.33 3.25˄
 LT × R 0.616 0.004 1.02 0.92 1.07 0.581
 LT × H 0.814 0.49 0.069
 LT × R × H 0.002 0.25 1.34
No Lines - Anomalous
 LT 0.06 0.003 2.12 4.23˄ 2.47 5.19*
 LT × R 5.38** 4.59* 3.87** 6.83* 4.33** 6.34*
 LT × H 0.929 0.008 0.237
 LT × R × H 0.014 0.38 0.56
Open in a new tab

Note. LT = Line Type, R = Region, H = Hemisphere. F-values are given.

All Sequence Types df = 2,46 except Midline LT × R = 8,184.

All pairwise df = 1,23 except Midline LT × R = 4,92.

˄

p<.1,

*

p <.05,

**

p<.01,

***

p<.001.

Figure 4.

Figure 4

Open in a new tab

Waveforms from electrode sites contrasting the difference between critical panels with Normal Lines, No Lines, and Anomalous Lines.

Figure 5.

Figure 5

Open in a new tab

Voltage maps for the differences across the scalp surface of ERPs evoked by contrasts between panels with Normal Lines, No Lines, and Anomalous Lines for the 200-400 ms, 400-600 ms, and 600-900ms time windows.

First, a larger leftward posterior positivity appeared to panels with Normal Lines than those with No Lines. Following significant and/or trending interactions between Line Type and Region at both midline and peripheral regions, this positivity localized to Occipital, Parietal, and Left Posterior regions (all Fs > 4.36, all ps < .05). Interactions were also found between panels with No Lines and Anomalous Lines in both midline and peripheral regions. This revealed an anteriorly distributed positivity that was greater to Anomalous Lines than No Lines in the Prefrontal region, F(1,23)=4.56, p<.05. Normal Lines and Anomalous Lines did not differ.

Within the 400 to 600ms epoch, an interaction appeared between Line Type and Region in midline regions, and a main effect of Line Type in Peripheral Regions. Main effects of Line Type appeared at the Prefrontal, Left Anterior, and Left Posterior regions (all Fs > 3.62, all ps < .05), along with a trending main effect in the Left Posterior region, F(2,46)=3.02, p=.059. These effects indexed a larger positivity for panels with Anomalous Lines than those with Normal or No Lines (see Table 1 and Figures 5 and 6).

Figure 6.

Figure 6

Open in a new tab

Electrode montage, illustrating midline and peripheral regions for analysis of ERP data along with placement of left and right eye electrodes (LE, HE) as well as left mastoid reference (A1) and right mastoid for differential activity (A2).

Closer analysis showed only a trending interaction for panels with Normal and No Lines in the midline regions, but no significant main effects or interactions at individual regions. In contrast, panels with Normal Lines and Anomalous Lines differed between Line Type at the midline and peripheral regions. A frontal positivity was suggested by main effects at the Prefrontal and Left and Right Anterior regions, with a posterior positivity in the Left Posterior region (all Fs > 4.6, all ps < .05) and trending differences in the Parietal region F(1,23)=3.07, p=.093. An interaction also appeared at the midline and peripheral regions between No Lines and Anomalous Lines. Follow up analyses showed a frontal positivity at the Prefrontal, and both Left and Right Anterior regions (all Fs > 4.5, all ps < .05). Altogether, these findings suggested that panels with Anomalous Lines evoked a bilateral, frontally distributed positivity that was greater than both Normal and No Lines.

Finally, within the 600 to 900ms epoch we found trending main effects of Line Type at midline and peripheral regions and a trending interaction between Line Type and Region at peripheral regions. This effect originated anteriorly from the Prefrontal region, F(2,46)=2.77, p=.073, and the Left Anterior region, F(1,23)=3.81, p<.05 (see Table 1 and Figures 4 and 5). These effects were motivated by a greater positivity for panels with Anomalous Lines than those with No Lines in Prefrontal, Central, Left Anterior, and Right Anterior regions (all Fs > 3.8, all ps < .065). No significant differences appeared between panels with Normal Lines and No Lines or Anomalous Lines.

2.2.2. Localization

Because the pattern of our results suggested an anterior positivity to the presence of Anomalous Lines, and a posterior positivity to the absence of Normal Lines, we sought to determine whether these effects originated in different or similar neural generators using a source localization algorithm (LORETA, (Pascual-Marqui, Esslen, Kochi, & Lehmann, 2002; Pascual-Marqui, Michel, & Lehmann, 1994). Generators in frontal regions were implicated for all subtractions, consistent with the need for working memory and executive control in gleaning narrative meaning (Whitney, Ritchie, & Clark, 1991), here from sequentially presented panels (Cohn, 2013b). Brodmann areas (BA) 10 (anterior prefrontal cortex) and 11 (orbitofrontal cortex) were implicated for all panels, subtracted conditions and time windows, being no lower than the fourth highest average voxel intensity level across comparisons. BAs 9 or 46 (dorsolateral prefrontal cortex) were found for all panels, subtracted conditions and time windows except for No Lines and Normal Lines for critical panels.

At the critical panels, early visual areas (BAs 17, 18 and 19) were activated for the contrasts between Anomalous Lines with Normal Lines or No Lines, but not between No Lines and Normal Lines in the 400-600 ms time window. This suggested reactivation of early visual area when it is more difficult to extract the meaning from the input. That is, when meaning cannot be easily gleaned from visual stimuli, top-down facilitative processes (Bar, 2003) may reactivate visual areas in an attempt to disambiguate visual input.

BA 47 was also active for the contrast of Anomalous and Normal Lines in the 200-400 ms window. Given that previous research has implicated BA 47 with sentence processing in language (Dapretto & Bookheimer, 1999; Poldrack et al., 1999) and syntactic processing of music (Levitin & Menon, 2003), we followed up our analysis at the subsequent-to-critical image to see if a violation of motion lines influenced the processing of subsequent narrative information (Cohn et al., 2012). We found no activation of early visual areas during this time window at the subsequent panel. However, areas associated with language were more robust. BAs 44 and 45 (Broca's Area) were active for the contrast between Anomalous with Normal Lines (200-400 ms) and Anomalous with No Lines (200-600 ms), but not No Lines with Normal Lines. A portion of Wernike's Area (BA 39), however, was active for No Lines and Normal Lines only from 400-600 ms. These findings suggest that greater activation of Broca's Area is associated with greater effort in assimilating incongruous lines with the overall narrative structure, while Wernike's Area is associated with integrating the meaning in a sequence with prior information where the depiction of events may have been less explicit due to the absence of motion lines.

2.2.3. VLFI score

We found significant correlations between our ERP effects and VLFI scores in the 200-400ms epoch. First, a trending negative correlation suggested that the difference in magnitude between panels with Normal Lines and No Lines was smaller for participants with higher VLFI scores, r(24)=-.368, p=.077. Second, a positive correlation indicated that greater fluency was associated with a larger positivity effect for Anomalous Lines compared to No Lines, r(24)=.683, p<.001. No significant correlation was found between VLFI scores and the contrast between Normal Lines and Anomalous Lines, nor were any contrasts significantly correlated at other epochs.

2.2.4. Ratings

Finally, the manipulation of motion lines appeared to have little effect on the offline judgments of the broader narrative sequence. All sequences were rated as moderately coherent—Normal Lines: 3.8 (.35); No Lines: 3.9 (.44); Anomalous Lines: 3.8 (.28)—and did not significantly differ from each other, F(2,48)=.889, p<.410.

3. Discussion

In this study, we examined motion line comprehension within the naturalistic context of comic strip panels. By measuring self-paced viewing times and ERPs, we compared the online comprehension of panels that either had normal motion lines, no lines, or anomalous (i.e., reversed direction) lines. In Experiment 1, panels with normal lines were viewed faster than those with no lines, which were in turn faster than those with anomalous lines. In Experiment 2, we found two distinct patterns of ERP effects: a posterior positivity peaking between 200-400ms appeared to panels with no lines compared to normal lines, and a bilateral anterior positivity peaking around 500ms for panels with anomalous lines compared to those with both normal and no lines (see Figure 5). Together, these results suggest that the presence of motion lines in comic panels further aids the comprehension of events, and that this comprehension is not motivated by biological aspects of the visual system, but rather to a conventionalized mapping of conceptual understanding to graphic representation.

In Experiment 1, the longer viewing times for panels with no lines than normal lines showed greater ease of comprehension when motion lines are present. These results suggest that motion lines provided additional understanding to motion beyond just postural cues (Brooks, 1977; Friedman & Stevenson, 1975; Gross et al., 1991; Ito et al., 2010; Kawabe & Miura, 2006), and is consistent with findings of lower ratings of comprehension to no lines or reversed lines than normal lines when interpreted as conveying motion (Ito et al., 2010). This evidence supports that normal motion lines aid in the comprehension of events more than images without lines or incongruous lines.

Our ERP results from Experiment 2 further clarified these behavioral measures. Overall, we found no evidence of early ERP components associated with perceptual processing (i.e., P1 or N1). Rather, our findings suggested two overall patterns: a posterior positivity modulated by the absence of normal lines, and a frontal positivity modulated by the presence of anomalous lines. These later effects suggest that comprehenders require additional processing to images with no lines or with anomalous lines that is above and beyond those with normal motion lines.

Our most noteworthy finding was a posterior positivity modulated by panels with no lines compared to normal motion lines. This positivity appeared to be greater to panels with no lines, rather than an attenuated negativity to panels with normal lines. These results imply that, to experienced comic readers, the absence of lines actually impairs a reader's comprehension of an event, rather than the presence of motion lines facilitating event information. One possibility is that this effect is tied to the posteriorly distributed “oddball” P300b, typically elicited by low probability or unexpected stimuli (Polich, 2007). Under this view, comic readers are habituated to motion lines in the depiction of events, and their absence are inconsistent with these expectations. Thus, the presence of lines would be “higher probability” than the absence of lines, which would be “lower probability.” However, the correlation between the VLFI scores and the magnitude of the difference between normal lines and no lines suggested that greater expertise predicted a smaller effect—meaning that this effect may not involve probabilistic expectations for motion lines. In addition, there was no task while viewing these panels, and the P300b is often related to task effects (Katayama & Polich, 1996).

An alternative interpretation could connect this positivity with the P600 found in studies of language and visual events (de Bruijn et al., 2007; Kuperberg, 2007, 2013; Osterhout & Holcomb, 1992; Sitnikova et al., 2008b; van Elk et al., 2012). In this view, events become harder to understand because there are no lines to bind together the component elements. For images like Figure 2, the motion lines simply clarify the direction of motion. However, in Figure 1c the lines specify the manner of motion (bouncing) and nature of its path. Omitting these lines may impair a comprehender's ability to connect the component parts of the full event, thus eliciting a P600 effect comparable to the alteration of an element involved in an event (Sitnikova et al., 2008b). Again, less experienced readers (as measured by VLFI score) showed larger effects, suggesting that they receive more benefit from motion lines in clarifying depicted actions, while more experienced readers do not require as much support.

Our second finding was that the presence of anomalous lines, when compared to both normal and no lines, evoked a frontal positivity with bilateral distribution. This effect may be connected to the typically anterior P300a, and which commonly appears as a “novelty” response to surprising or unexpected stimuli (Polich, 2007). Anomalous lines, which reverse the expected direction of motion, should clearly be considered as “novel” or unexpected given their context. Nevertheless, this frontal positivity peaked between 400-600ms, and between panels with anomalous lines and no lines, lasted until 900ms, well after the typical peak of a P300a (Polich, 2007). Such longer latency may be attributable to the greater complexity in these visual representations than simple tones or lights. Relatedly, this frontal positivity may also connect with effects shown in language studies to predictable words in highly constraining contexts (Delong et al., 2011; Federmeier et al., 2007; Kuperberg, 2013; Thornhill & Van Petten, 2012; Van Petten & Luka, 2012). Under this interpretation, this frontal positivity could index the violated expectation of viewing a specific type of motion line and instead getting a reversed result. Yet, unlike the fully anomalous stimuli here, the frontal positivities shown to language usually occur in highly constraining contexts to unpredictable congruous words. Thus, clear differences arise in the contexts between these frontal positivities that can only be addressed by future work.

3.1. Impact on sequential image comprehension

Our source localization of these effects further supported that greater neural resources were required for experienced comic readers to glean meaning from anomalous motion lines. Subtracting normal lines or no lines from anomalous lines resulted in a similar pattern for the critical and subsequent-to-critical panel, and this pattern differed from that found by subtracting normal lines from no lines. During the critical (but not subsequent-to-critical) panels, both subtractions from the anomalous lines produced reactivation of early visual areas after 400 ms. This finding suggests that when participants viewed anomalous (but not normal or absent) motion lines, an additional volley of visual cortex activation—a sort of “neural double-take”—was initiated to disambiguate this input. This is consistent with previous work showing an iterative process between frontal areas and visual cortex in deriving meaning from visual stimuli (Bar, 2003). Furthermore, while language areas were found to be active in frontal regions at the critical panel (BA 47), they became more active for viewing anomalous lines during the subsequent-to-critical panel (discussed below), consistent with the idea that greater effort was required to incorporate the meaning of the anomalous lines. Taken together, this pattern of localization shows that the anomalous motion lines were more difficult to integrate into a cohesive narrative, and that the “bilateral” distribution to the “frontal positivity” may have originated from two different frontal areas.

An intriguing finding of our source localization found activation for all contrasts with anomalous lines in the frontal region of BA 47 (pars orbitalis)—an area typically associated with the processing of sentence structure in language (Dapretto & Bookheimer, 1999; Poldrack et al., 1999) and musical syntax (Levitin & Menon, 2003). In previous research, we have argued that sequential images use a “narrative grammar” that operates with the same architectural principles as linguistic syntax (Cohn, 2013b). Manipulation of this narrative grammar has evoked similar ERP responses as those found in studies of language, including negativities localized to left anterior regions of the scalp (Cohn et al., 2014; Cohn et al., 2012), while earlier research has suggested a role of frontal brain regions in the comprehension of comic strips (Bihrle, Brownell, Powelson, & Gardner, 1986; Nagai, Endo, & Takatsune, 2007).

Given that we embedded motion lines within a broader sequence, we hypothesized that the activation of BA47 related to their impact on the broader narrative sequence. We thus carried out follow up analyses at the subsequent panel, where we found more robust activation of areas related to language processing, specifically Broca's Area (BA 44/45) and portions of Wernicke's Area (BA 39). These results suggested that anomalous lines impacted the understanding of the narrative sequence both at the panel they appeared and at subsequent panels. This activation of Broca's Area by anomalous lines implies that cues from the “morphology” of individual images can influence the understanding of subsequent panels (Cohn & Paczynski, 2013). In this case, “illegal” visual morphological information within single images impairs the processing of the broader narrative structure.

In contrast, we found no activation of Broca's Area in the contrast between no lines with normal lines, which did localize to areas of Wernicke's Area. We hypothesize that this contrast yielded no activation of Broca's Area because the absence of lines does not explicitly violate morphological information that might be relevant for a narrative. However, the absence of normal lines may still strain the understanding of event information across the images, reflected in the activation of Wernicke's Area. This is again consistent with previous work showing that less “active” figures incur greater costs at subsequent panels than those with more explicit event information (Cohn & Paczynski, 2013), and with work showing posterior positivities to violations of visual events (Amoruso et al., 2013; Sitnikova et al., 2008b). Nevertheless, given that these findings used source localization of EEG, our findings can hopefully be confirmed by subsequent studies using measures with better spatial resolution (fMRI, MEG, etc.).

3.2. Visual language fluency

We also found a relationship between comic reading expertise and the observed effects across both experiments. In Experiment 1, more experienced comic readers viewed panels with anomalous lines slower than participants who had less experience. In Experiment 2, the magnitude of the posterior positivity to panels with no lines versus normal lines was larger for participants with less experience. Meanwhile, the frontal positivity effect between panels with anomalous lines versus no lines was larger for participants who had more expertise. In all, these results suggest that more experience with reading comics—and thus familiarity with their visual vocabulary—correlated with understanding motion lines.

Both sets of results indicate that motion lines—used in diverse and varied contexts—are conventionalized signs understood and expected by comic readers to appear in the depiction of motions and events. This connection to “visual language fluency” supports that motion lines do not simply originate in basic perceptual processes of the streaks left behind in the visual system (Burr, 2000; Burr & Ross, 2002). Rather, motion lines are conventionalized—yet not symbolic (Cohn, 2013a)—representations understood through experience with a visual vocabulary, which is sensitive to both cultural (Duncan et al., 1973; Kennedy & Ross, 1975; Winter, 1963) and developmental knowledge (Carello et al., 1986; Friedman & Stevenson, 1975; Gross et al., 1991; Mori, 1995; Nakazawa, 1998).

4. Conclusion

Our results provide substantial evidence against the idea that the understanding of motion lines relates to basic biological aspects of the visual system. Rather, these conventionalized representations of paths are tied to experience with a cultural graphic system. Altogether, this study shows that research of phenomena in visual narrative can be useful to the study of a broad range of issues related to visual cognition, language, and the comprehension of events.

5. Methods and Materials

5.1. Stimuli

We created 90 6-panel long comic strips using explicit motion lines from panels in The Complete Peanuts volumes 1 through 6 (1950-1962) by Charles Schulz. Critical panels depicted several different events including running, jumping, throwing or kicking various objects (sports balls, sticks, paper, etc.), hitting balls (golf balls, croquet balls, baseballs), falling off of objects, moving down a slide, punching or running into objects, among several others actions. The actual depiction of motion lines varied based on their natural original context: ranging from just one or two lines to many lines. They also varied in shape: straight and lateral, angled, vertical, curved, circular, etc. Such variety was chosen to cover motion lines wide range of naturalistic contexts. The nature of the event, and number and shape of lines were based solely on the characteristics of the original panels in our corpus of Peanuts comic strips.

We then created three types of critical panels (see Figure 2). Panels with “normal lines” used the original motion lines trailing behind a moving object. Panels with “no lines” omitted the motion lines from the object. In these cases, lines were broken up and distributed to other parts of a frame in order to retain the same visual complexity as the original panel. Finally, “anomalous lines” reversed the lines from their original position to make the object appear to move in an incongruous motion to the action. For example, in Figure 2, the anomalous line makes the baseball appear to be going towards the bat that is known to have struck the ball. These three sequence types were counterbalanced into three lists in a Latin Square design such that each list contained an equal number of each type and no sequences repeating across lists. Each list also included 120 filler sequences without manipulations to motion lines. The same stimuli appeared in Experiments 1 and 2.

5.2. Experiment 1: Participants

Sixty-two experienced comic readers (35 male, 27 female, mean age: 24.03) from the Tufts University student population and surrounding neighborhoods were paid for their participation. All participants gave informed written consent according to Tufts University's Human Subjects Review Board guidelines. We sought self-defined “comic readers” to ensure “fluency” in understanding visual narratives (Cohn, 2013a; Nakazawa, 2005). This proficiency was assessed using the “Visual Language Fluency Index” (VLFI) questionnaire that asked participants to rate their habits for reading and drawing various types of visual narratives (comic books, comic strips, graphic novels, Japanese manga), and their comic reading “expertise.” From these ratings we generated a “VLFI score” for each participant (for details, see Cohn et al., 2012). An idealized average along this metric would be a score of 12, with low being below 7 and high above 20. Included participants had a mean comic-reading fluency of 13.89 (SD=6.81), while data from two participants were excluded from analysis because they had difficulty understanding the task.

5.3. Experiment 1: Procedure

Sequences were presented frame-by-frame on a desktop computer screen. Participants controlled the pace of reading with a button press at each panel, and we measured how long each frame stayed on the screen. Trials began with a screen reading READY, followed by a fixation-cross (+). Then, each panel appeared on the screen one at a time until the end of the sequence. Here, a question mark appeared where they rated how easy the strip was to understand on a 1 to 5 scale (1=hard, 5=easy), at which point the next trial appeared. Prior to the experimental session, participants completed a practice list of five strips to orient them to the procedure.

5.4. Experiment 1: Data analysis

Viewing times and ratings were analyzed using a one-way repeated-measures Analysis of Variance (ANOVA) with three levels (Normal Lines, No Lines, Anomalous Lines) for both subjects (F1) and items (F2) analyses. Significant main effects were followed up by pairwise t-tests of individual relationships. An additional Pearson's correlation with alpha set to .05 was carried out between participants' VLFI scores and their viewing times.

5.5. Experiment 2: Participants

Twenty-five comic readers from Tufts University and the surrounding area (8 male, 17 female, mean age: 19.9) participated in the study for compensation. All participants gave their informed written consent, and none had participated in Experiment 1. Participants were pre-screened to be right-handed English speaking comic readers with normal vision, no history of head trauma, and taking no neuropsychiatric drugs. They overall had a high-average mean comic reading fluency of 16.12 (SD=6.7). Data from one participant was excluded due to artifact rejection exceeding 15%.

5.6. Experiment 2: Procedure

Participants sat in a comfortable chair facing a computer screen in a room separate from the experimenter and computers. Participants began a trial by viewing the word READY, where they pressed a button on a keypad. A fixation-cross then appeared in the center of the screen, after which each panel of the sequence automatically appeared one at a time in that location, out of control of the participant. Each panel was onscreen for 1350ms, and an ISI of 300ms prevented a “flipbook” effect that made overlapping panels appear animated. Stimuli durations were based on the average viewing times in Experiment 1. After the final panel, a question mark cued participants to rate the sequence for how easy it was to understand (1=difficult to 5=easy). Ten practice sequences acclimated participants to the procedure and stimuli prior to the experiment.

5.7. Experiment 2: ERP Recordings

ERPs were measured using an elastic cap with 29 tin electrodes distributed along the scalp according to the International 10-20 system, plus additional sites over the left and right hemisphere. Electrodes were also placed below the left eye and next to the right eye to record blinks and vertical and horizontal eye movements. Electrodes were placed along the five midline sites (FPz, Fz, Cz, Pz, Oz), three pairs of medial sites (FC1/FC2, C3/C4, CP1/CP2), four pairs of lateral sites, (F3/F4, FC5/FC6, CP5/CP6, P3/P4), and five pairs of peripheral sites (FP1/FP2, F7/F8, T3/T4, T5/T6, O1/O2) on each hemisphere. All sites were referenced to an electrode placed on the left mastoid, while differential activity was monitored in the right mastoid (see Figure 6).

An SA Bioamplifier amplified the electroencephalogram (EEG) using a bandpass of 0.01 to 40 Hz and continuously sampled at a rate of 200 Hz. Electrode impedances were kept below 10 kΩ for the eyes and below 5 kΩ at all other sites.

5.8. Experiment 2: Data Analysis

Coherence ratings in Experiment 2 were analyzed using the same methods as in Experiment 1. ERPs were time-locked to the onset of each critical panel. Mean voltages were analyzed within the windows of 200-400ms, 400-600ms, and 600-900ms. ANOVAs were carried out along five midline regions and four peripheral regions, with each region comprised of three electrodes (see Figure 6). The midline regions consisted of Prefrontal (FPz, FP1, FP2), Frontal (Fz, FC1, FC2), Central (Cz, C3, C4), Parietal (CP1, CP2, Pz), and Occipital (O1, O2, Oz) regions. Peripheral regions included the Left Frontal (F3, F7, FC5), Right Frontal (F4, F8, FC6), Left Posterior (CP5, T5, P3), and Right Posterior (CP6, T6, P4) regions. Within-subject factors used the three levels of Line Type (Normal, No Lines, Anomalous) and five levels of Region for midline. Analyses of peripheral regions used four levels of Region as well as two levels of Hemisphere. Main effects and interactions were followed by individual comparisons between Line Types at each Region.

Cortical sources of grand average ERPs were estimated using low-resolution brain electromagnetic tomography (LORETA) (Pascual-Marqui et al., 2002; Pascual-Marqui et al., 1994). LORETA was used to estimate the cortical source on difference waves created by subtractions of: No Lines - Normal Lines, No Lines - Anomalous Lines, and Anomalous Lines - No Lines for ERPs time locked to critical panels. This initial analysis warranted investigation beyond the critical panel; thus the same cortical source analysis was also performed for ERPs time-locked to the panel immediately following the critical panel. Voxel intensity levels were averaged for each Brodmann Area (BA) reported as a source by LORETA, and BAs with the ten greatest average intensities were compared across subtractions.

Finally, as in Experiment 1, participants' VLFI scores were correlated with ERP amplitude differences between Line Types. These ERP differences scores were averaged across all electrode sites in each individual, and were then correlated with each individual's VLFI score using a Pearson's correlation set to an alpha level of .05.

Acknowledgments

Gina Kuperberg is thanked for funding this work through NIMH (R01 MH071635), NICHD (HD25889) and NARSAD (with the Sidney Baer Trust). Additional funding was provided by the Tufts Center for Cognitive Studies. Chelsey Ott and Patrick Bender are thanked for aiding with data acquisition and Priya Mitra for help with the figures. Fantagraphics Books is thanked for their generous donation of The Complete Peanuts.

Footnotes

Statement of contribution: Dr. Cohn conceived and ran these studies, while Dr. Maher contributed to the analysis. Both authors contributed to the writing of the paper and approve the final manuscript.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Amoruso L, Gelormini C, Aboitiz F, Alvarez González M, Manes F, Cardona J, Ibanez A. N400 ERPs for actions: Building meaning in context. Frontiers in Human Neuroscience. 2013;7 doi: 10.3389/fnhum.2013.00057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Anllo-Vento L, Hillyard S. Selective attention to the color and direction of moving stimuli: Electrophysiological correlates of hierarchical feature selection. Perception & Psychophysics. 1996;58(2):191–206. doi: 10.3758/BF03211875. [DOI] [PubMed] [Google Scholar]
  3. Bach P, Gunter TC, Knoblich G, Prinz W, Friederici AD. N400-like negativities in action perception reflect the activation of two components of an action representation. Social Neuroscience. 2009;4(3):212–232. doi: 10.1080/17470910802362546. [DOI] [PubMed] [Google Scholar]
  4. Bar M. A Cortical Mechanism for Triggering Top-Down Facilitation in Visual Object Recognition. Journal of Cognitive Neuroscience. 2003;15(4):600–609. doi: 10.1162/089892903321662976. [DOI] [PubMed] [Google Scholar]
  5. Bihrle AM, Brownell HH, Powelson JA, Gardner H. Comprehension of humorous and nonhumorous materials by left and right brain-damaged patients. Brain and Cognition. 1986;5:399–411. doi: 10.1016/0278-2626(86)90042-4. [DOI] [PubMed] [Google Scholar]
  6. Brooks PH. The Role of Action Lines in Children's Memory for Pictures. Journal of Experimental Child Psychology. 1977;23:93–107. [Google Scholar]
  7. Burr DC. Motion vision: Are “speed lines” used in human visual motion? Current Biology. 2000;10(12):R440–R443. doi: 10.1016/s0960-9822(00)00545-5. [DOI] [PubMed] [Google Scholar]
  8. Burr DC, Ross J. Direct evidence that “speedlines” influence motion mechanisms. The Journal of Neuroscience. 2002;22(19):8661–8664. doi: 10.1523/JNEUROSCI.22-19-08661.2002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Carello C, Rosenblum LD, Grosofsky A. Static depiction of movement. Perception. 1986;15(1):41–58. doi: 10.1068/p150041. [DOI] [PubMed] [Google Scholar]
  10. Chapman RM, Bragdon HR. Evoked Responses to Numerical and Non-Numerical Visual Stimuli while Problem Solving. Nature. 1964;203(4950):1155–1157. doi: 10.1038/2031155a0. [DOI] [PubMed] [Google Scholar]
  11. Christiansen MH, Conway CM, Onnis L. Similar neural correlates for language and sequential learning: Evidence from event-related brain potentials. Language and Cognitive Processes. 2011;27(2):231–256. doi: 10.1080/01690965.2011.606666. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Coch D, Skendzel W, Grossi G, Neville H. Motion and color processing in school-age children and adults: an ERP study. Developmental Science. 2005;8(4):372–386. doi: 10.1111/j.1467-7687.2005.00425.x. [DOI] [PubMed] [Google Scholar]
  13. Cohn N. The visual language of comics: Introduction to the structure and cognition of sequential images. London, UK: Bloomsbury; 2013a. [Google Scholar]
  14. Cohn N. Visual narrative structure. Cognitive Science. 2013b;37(3):413–452. doi: 10.1111/cogs.12016. [DOI] [PubMed] [Google Scholar]
  15. Cohn N. You're a good structure, Charlie Brown: The distribution of narrative categories in comic strips. Cognitive Science. 2014;38(7):1317–1359. doi: 10.1111/cogs.12116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Cohn N, Jackendoff R, Holcomb PJ, Kuperberg GR. The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension. Neuropsychologia. 2014;64:63–70. doi: 10.1016/j.neuropsychologia.2014.09.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Cohn N, Paczynski N. Prediction, events, and the advantage of Agents: The processing of semantic roles in visual narrative. Cognitive Psychology. 2013;67(3):73–97. doi: 10.1016/j.cogpsych.2013.07.002. http://dx.doi.org/10.1016/j.cogpsych.2013.07.002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Cohn N, Paczynski M, Jackendoff R, Holcomb PJ, Kuperberg GR. (Pea)nuts and bolts of visual narrative: Structure and meaning in sequential image comprehension. Cognitive Psychology. 2012;65(1):1–38. doi: 10.1016/j.cogpsych.2012.01.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Coulson S, King J, Kutas M. Expect the unexpected: Event-related brain responses to morphosyntactic violations. Language and Cognitive Processes. 1998;13(71-74):71. [Google Scholar]
  20. Dapretto M, Bookheimer SY. Form and Content: Dissociating Syntax and Semantics in Sentence Comprehension. Neuron. 1999;24(2):427–432. doi: 10.1016/s0896-6273(00)80855-7. http://dx.doi.org/10.1016/S0896-6273(00)80855-7. [DOI] [PubMed] [Google Scholar]
  21. de Bruijn ERA, Schubotz RI, Ullsperger M. An event-related potential study on the observation of erroneous everyday actions. Cognitive, Affective, & Behavioral Neuroscience. 2007;7(4):278–285. doi: 10.3758/cabn.7.4.278. [DOI] [PubMed] [Google Scholar]
  22. Delong KA, Urbach TP, Groppe DM, Kutas M. Overlapping dual ERP responses to low cloze probability sentence continuations. Psychophysiology. 2011;48(9):1203–1207. doi: 10.1111/j.1469-8986.2011.01199.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Donchin E, Coles MGH. Is the P300 component a manifestation of context updating? Behavioral and Brain Sciences. 1988;11(03):357–374. doi: 10.1017/S0140525X00058027. [DOI] [Google Scholar]
  24. Duncan HF, Gourlay N, Hudson W. A Study of Pictorial Perception Among Bantu and White School Children. Johannesburg: Witwaterstrand University Press; 1973. [Google Scholar]
  25. Federmeier KD, Wlotko EW, De Ochoa-Dewald E, Kutas M. Multiple effects of sentential constraint on word processing. Brain Res. 2007;1146:75–84. doi: 10.1016/j.brainres.2006.06.101. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Friederici AD, Mecklinger A, Spencer KM, Steinhauer K, Donchin E. Syntactic parsing preferences and their on-line revisions: a spatio-temporal analysis of event-related brain potentials. Cognitive Brain Research. 2001;11(2):305–323. doi: 10.1016/s0926-6410(00)00065-3. http://dx.doi.org/10.1016/S0926-6410(00)00065-3. [DOI] [PubMed] [Google Scholar]
  27. Friedman SL, Stevenson MB. Developmental Changes in the Understanding of Implied Motion in Two-dimensional Pictures. Child Development. 1975;46:773–778. [PubMed] [Google Scholar]
  28. Geisler WS. Motion streaks provide a spatial code for motion direction. Nature. 1999;400(6739):65–69. doi: 10.1038/21886. [DOI] [PubMed] [Google Scholar]
  29. Green J. Drawn from the ground: Sound, sign and inscription in Central Australian sand stories. Cambridge, UK: Cambridge University Press; 2014. [Google Scholar]
  30. Gross D, Soken N, Rosengren KS, Pick AD, Pillow BH, Melendez P. Children's Understanding of Action Lines and the Static Representation of Speed of Locomotion. Child Development. 1991;62:1124–1141. [PubMed] [Google Scholar]
  31. Hayashi H, Matsuda G, Tamamiya Y, Hiraki K. Visual cognition of “speed lines” in comics: Experimental study on speed perception; Poster preseneted at the 34th Annual Conference of the Cognitive Science Society; Sapporo, Japan. 2012. [Google Scholar]
  32. Ito H, Seno T, Yamanaka M. Motion impressions enhanced by converging motion lines. Perception. 2010;39(11):1555–1561. doi: 10.1068/p6729. [DOI] [PubMed] [Google Scholar]
  33. Jackendoff R. Semantics and Cognition. Cambridge, MA: MIT Press; 1983. [Google Scholar]
  34. Katayama Ji, Polich J. P300 from one-, two-, and three-stimulus auditory paradigms. International Journal of Psychophysiology. 1996;23(1–2):33–40. doi: 10.1016/0167-8760(96)00030-x. http://dx.doi.org/10.1016/0167-8760(96)00030-X. [DOI] [PubMed] [Google Scholar]
  35. Kawabe T, Miura K. Representation of dynamic events triggered by motion lines and static human postures. Experimental Brain Research. 2006;175(2):372–375. doi: 10.1007/s00221-006-0673-6. [DOI] [PubMed] [Google Scholar]
  36. Kawabe T, Yamada Y, Miura K. Memory displacement of an object with motion lines. Visual Cognition. 2007;15(3):305–321. doi: 10.1080/13506280600591036. [DOI] [Google Scholar]
  37. Kennedy JM, Gabias P, Piertantoni R. Meaning, presence and absence in pictures. In: Landwehr K, editor. Ecologicalperception research, visual communication and esthetics. New York: Springer-Verlag; 1990. pp. 43–56. [Google Scholar]
  38. Kennedy JM, Ross A. Outline picture perception by the Songe of Paua. Perception. 1975;4:391–406. [Google Scholar]
  39. Kim H, Francis G. A computational and perceptual account of motion lines. Perception. 1998;27(7):785–797. doi: 10.1068/p270785. [DOI] [PubMed] [Google Scholar]
  40. Kuperberg GR. Neural mechanisms of language comprehension: Challenges to syntax. Brain Research. 2007;1146:23–49. doi: 10.1016/j.brainres.2006.12.063. [DOI] [PubMed] [Google Scholar]
  41. Kuperberg GR. The pro-active comprehender: What event-related potentials tell us about the dynamics of reading comprehension. In: Miller B, Cutting L, McCardle P, editors. Unraveling the Behavioral, Neurobiological, and Genetic Components of Reading Comprehension. Baltimore: Paul Brookes Publishing; 2013. [Google Scholar]
  42. Levitin DJ, Menon V. Musical structure is processed in “language” areas of the brain: a possible role for Brodmann Area 47 in temporal coherence. NeuroImage. 2003;20(4):2142–2152. doi: 10.1016/j.neuroimage.2003.08.016. http://dx.doi.org/10.1016/j.neuroimage.2003.08.016. [DOI] [PubMed] [Google Scholar]
  43. Mandler JM. How to build a baby: II. Conceptual primitives. Psychological Review. 1992;99(4):587–604. doi: 10.1037/0033-295x.99.4.587. [DOI] [PubMed] [Google Scholar]
  44. McCloud S. Understanding Comics: The Invisible Art. New York, NY: Harper Collins; 1993. [Google Scholar]
  45. Mori K. The influence of action lines on pictorial movement perception in pre-school children. Japanese Psychological Research. 1995;27(3):183–187. [Google Scholar]
  46. Munn ND. Walbiri Graphic Signs: An Analysis. American Anthropologist. 1962;64(5):972–984. [Google Scholar]
  47. Nagai M, Endo N, Takatsune K. Measuring Brain Activities Related to Understanding Using Near-Infrared Spectroscopy (NIRS) In: Smith MJ, S G, editors. Human Interface and the Management of Information: Methods, Techniques and Tools in Information Design. Vol. 4557. Heidelberg: Springer Berlin; 2007. pp. 884–893. [Google Scholar]
  48. Nakazawa J. Development of manga notation understanding. Proceedings of the 9th Annual Conference of Japan Society of Developmental Psychology. 1998:182. [Google Scholar]
  49. Nakazawa J. Development of manga (comic book) literacy in children. In: Shwalb DW, Nakazawa J, Shwalb BJ, editors. Applied Developmental Psychology: Theory, Practice, and Research from Japan. Greenwich, CT: Information Age Publishing; 2005. pp. 23–42. [Google Scholar]
  50. Osterhout L, Hagoort P. A superficial resemblence does not necessarily mean you are part of the family: conterarguments to Couson, King and Kutas (1998) in the P600/SPS=P300 debate. Language and Cognitive Processes. 1999;14:1–14. [Google Scholar]
  51. Osterhout L, Holcomb P. Event-related potentials elicited by syntactic anomaly. Journal of Memory and Language. 1992;31:758–806. [Google Scholar]
  52. Pardo JV, Fox PT, Raichle ME. Localization of a human system for sustained attention by positron emission tomography. Nature. 1991;349(6304):61–64. doi: 10.1038/349061a0. [DOI] [PubMed] [Google Scholar]
  53. Pascual-Marqui RD, Esslen M, Kochi K, Lehmann D. Functional imaging with low resolution brain electromagnetic tomography (LORETA): review, new comparisons, and new validation. Japanese Journal of Clinical Neurophysiology. 2002;30:81–94. [PubMed] [Google Scholar]
  54. Pascual-Marqui RD, Michel CM, Lehmann D. Low resolution electromagnetic tomography: a new method for localizing electrical activity in the brain. International Journal of Psychophysiology. 1994;18(1):49–65. doi: 10.1016/0167-8760(84)90014-x. http://dx.doi.org/10.1016/0167-8760(84)90014-X. [DOI] [PubMed] [Google Scholar]
  55. Poldrack RA, Wagner AD, Prull MW, Desmond JE, Glover GH, Gabrieli JDE. Functional Specialization for Semantic and Phonological Processing in the Left Inferior Prefrontal Cortex. NeuroImage. 1999;10(1):15–35. doi: 10.1006/nimg.1999.0441. http://dx.doi.org/10.1006/nimg.1999.0441. [DOI] [PubMed] [Google Scholar]
  56. Polich J. Updating P300: An integrative theory of P3a and P3b. Clinical neurophysiology : official journal of the International Federation of Clinical Neurophysiology. 2007;118(10):2128–2148. doi: 10.1016/j.clinph.2007.04.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Posner MI. Attention as a cognitive and neural system. Current Directions in Psychological Science. 1992;1(1):11–14. [Google Scholar]
  58. Sitnikova T, Holcomb PJ, Kuperberg GR. Neurocognitive mechanisms of human comprehension. In: Shipley TF, Zacks JM, editors. Understanding Events: How Humans See, Represent, and Act on Events. Oxford University Press; 2008a. pp. 639–683. [Google Scholar]
  59. Sitnikova T, Holcomb PJ, Kuperberg GR. Two neurocognitive mechanisms of semantic integration during the comprehension of visual real-world events. Journal of Cognitive Neuroscience. 2008b;20(11):1–21. doi: 10.1162/jocn.2008.20143. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Talmy L. Toward a Cognitive Semantics. Vol. 1. Cambridge, MA: MIT Press; 2000. [Google Scholar]
  61. Thornhill DE, Van Petten C. Lexical versus conceptual anticipation during sentence processing: Frontal positivity and N400 ERP components. International Journal of Psychophysiology. 2012;83(3):382–392. doi: 10.1016/j.ijpsycho.2011.12.007. http://dx.doi.org/10.1016/j.ijpsycho.2011.12.007. [DOI] [PubMed] [Google Scholar]
  62. van Elk M, Bousardt R, Bekkering H, van Schie HT. Using Goal- and Grip-Related Information for Understanding the Correctness of Other's Actions: An ERP Study. PLoS ONE. 2012;7(5):e36450. doi: 10.1371/journal.pone.0036450. [DOI] [PMC free article] [PubMed] [Google Scholar]
  63. Van Petten C, Luka BJ. Prediction during language comprehension: Benefits, costs, and ERP components. International Journal of Psychophysiology. 2012;83(2):176–190. doi: 10.1016/j.ijpsycho.2011.09.015. http://dx.doi.org/10.1016/j.ijpsycho.2011.09.015. [DOI] [PubMed] [Google Scholar]
  64. Võ MLH, Wolfe JM. Differential Electrophysiological Signatures of Semantic and Syntactic Scene Processing. Psychological Science. 2013 doi: 10.1177/0956797613476955. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Whitney P, Ritchie BG, Clark MB. Working-memory capacity and the use of elaborative inferences in text comprehension. Discourse Processes. 1991;14(2):133–145. doi: 10.1080/01638539109544779. [DOI] [Google Scholar]
  66. Winter W. The perception of safety posters by Bantu industrial workers. Psychological Africana. 1963;10(2):127–135. [Google Scholar]

RESOURCES