Abstract
Speakers access information from objects they will name but have not looked at yet, indexed by preview benefit—faster processing of the target when a preview object previously occupying its location was related compared to unrelated. This suggests that speakers distribute attention over multiple objects, but does not reveal the time course of processing of a current and to-be-named object. Is preview benefit a consequence of attention shifting to the next-to-be-named object shortly before the eyes move to that location, or does preview benefit reflect a more unconstrained deployment of attention to upcoming objects? Using the multiple object-naming paradigm with a gaze-contingent display change manipulation, we addressed this issue by manipulating the latency of the onset of the preview (SOA) and whether the preview represented the same concept as (but a different visual token of) the target or an unrelated concept. Results revealed that preview benefit was robust, regardless of the latency of the preview onset (SOA) or the latency of the saccade to the target (the lag between preview offset and fixation on the target). Together, these data suggest that preview benefit is not restricted to the time during an attention shift preceding an eye movement, and that speakers are able to take advantage of information from non-foveal objects whenever they are visually available.
Keywords: Parallel Processing, Object Naming, Eye Movements, Attention
Part of what makes speaking so efficient is our ability to plan ahead. When speaking about objects in our environment, planning ahead can be measured by a subject’s ability to pre-process upcoming, to-be-named objects before looking at them (Meyer & Dobel, 2003; Meyer, Ouellet, & Hacker, 2008; Morgan & Meyer, 2005; Morgan, van Elswijk & Meyer, 2008; Pollatsek, Rayner, & Collins, 1984; Schotter, Ferreira & Rayner, 2013; for reviews see Meyer, 2004; Schotter, 2011). This ability for simultaneous processing raises the issue of how processing and management of information from multiple objects is achieved. Do speakers process all visible objects whenever available, or is pre-processing of the upcoming object restricted to a brief amount of time, before the speaker has moved his/her eyes to that object but when attention is shifted to its location?
Because a shift in attention generally precedes an eye movement to a particular location (Deubel & Schneider, 1996; Henderson, 1993; Hoffman & Subramaniam, 1995; see Rayner, 1998, 2009 for reviews), it may seem advantageous for speakers describing their environment to restrict pre-processing of an upcoming object to just before the eyes move to it (Henderson, Pollatsek, & Rayner, 1989); otherwise they may risk prematurely accessing and saying the wrong word. However, such narrow deployment of attention may be difficult because the order in which speakers inspect their environment is generally unconstrained (in comparison to reading, in which the sequence of fixations is guided by the order of the words in the text; see Rayner, Angele, Schotter, & Bicknell, 2013; Reichle, Liversedge, Pollatsek, & Rayner, 2009; Schotter, Angele, & Rayner, 2012). Therefore, a process whereby speakers fully process one object and then shift attention to pre-process the next object before an eye movement may not be viable. Rather, because speakers explore their visual environment with the task of speaking about it (Griffin & Bock, 2000), it may be that they distribute attention and processing resources over the entire scene. We note that how visual attention is allocated (e.g., sequentially versus in a more distributed fashion) likely depends on the nature of the task (e.g., reading, visual search, scene perception, etc.). Therefore, the focus of this article and the interpretation of our data reflect only how multiple objects are processed during a task in which subjects name several in succession.
In the present study, we investigated how speakers distribute attention over multiple objects using the multiple object-naming paradigm (Morgan & Meyer, 2005) with a gaze-contingent display change manipulation (Pollatsek et al., 1984; Rayner, 1975). In this paradigm, speakers see three objects on the screen and name them in a prescribed order. During the trial, the target (the object in the second tobe- named location) is replaced with a preview (a different object) while the speaker looks at and names the first object. When the speaker makes an eye movement to the target location, the preview is replaced by the target. Preview benefit—faster processing of (shorter gaze durations on1) the target when the preview was related to it compared to when it was unrelated—suggests that speakers had processed the preview and used that information to facilitate processing of the target. Preview benefit in object naming has been demonstrated for visually similar objects (Henderson & Seifert, 2001; Pollatsek et al., 1984; Schotter et al., 2013), objects that represent the same concept (Meyer & Dobel, 2003; Pollatsek et al., 1984; Pollatsek, Rayner, & Henderson, 1990), and homophones (Meyer, Ouellet, & Hacker, 2008; Morgan & Meyer, 2005; Pollatsek et al., 1984; see Schotter, 2011 for a review). In all these studies, however, the timing of the preview was not manipulated. Therefore, it is unclear when the preview benefit arises.
We manipulated the timing of the preview (i.e., its onset relative to the start of fixation on the preceding object) to assess whether preview benefit diminishes as the preview is presented earlier, relative to the saccade to the target. If so, it would suggest that speakers only process the upcoming object during an attention shift that precedes a saccade to it. Conversely, if the preview benefit remains relatively stable regardless of when the preview was available, it would suggest that allocation of attention to the upcoming object does not only occur during this attention shift. Using a preview that is identical to the target would yield the largest preview benefit (because the preview and target match on all dimensions: visual, conceptual and phonological). However, smaller but significant preview benefit is still observed for previews that represent the same concept as the target with a different visual token (Meyer & Dobel, 2003; Pollatsek et al., 1984, 1990). To ensure that the preview benefit we observe is due to post-perceptual processing, we followed Pollatsek et al. (1984, 1990) and Meyer and Dobel (2003) in using previews that represented the same concept as the target, but with a different visual token (Figure 1).
Figure 1.
Example stimuli used in the experiment.
Method
Subjects
Fifty-two students from the University of California, San Diego (ages 18–24) participated in this experiment for course credit or $10. Five subjects were excluded because blinking led to excessive track loss. All subjects were native speakers of English with normal or corrected-to-normal vision and were naïve concerning the purpose of the experiment.
Apparatus
Eye movements were recorded via an SR Research Ltd. Eyelink 1000 eye tracker in remote setup (no head restraint, but head position was monitored) with a sampling rate of 500 Hz. Viewing was binocular, but only the right eye was monitored. Following calibration, eye position errors were less than 1°. Subjects were seated approximately 60 cm away from a 20” Sony Trinitron CRT monitor with 1280×1024 pixel resolution and an 85 Hz refresh rate.
Materials and Design
One hundred and forty-four line drawings of objects (48 target pictures, 48 pictures for the first location, and 48 pictures for the third location) were selected from the International Picture Naming Project database (IPNP; E. Bates et al., 2003) with similar selection criteria as used in Schotter et al. (2013; monosyllabic names, a large proportion of usable responses, high name agreement, and fast response times). Forty-eight other line drawings were taken from web searches to create same-concept previews. For each target, one picture was selected that represented the same concept as the target but with a different visual token of that concept (see Figure 1). To create the different-concept preview, the same-concept previews were shuffled across targets so that the target and preview were semantically or phonologically unrelated. Because the same items were used both as same-concept and different-concept previews, any idiosyncratic effects of those objects was controlled across conditions.
There were three object locations on the monitor (Figure 2), and subjects were instructed to name the image on the top left (the first object), then the image on the top right (the target object), and then the bottom image (the third object). Objects were black line drawings on a white background (on average, they subtended approximately 5 ) and arranged so that the distance between the midpoints of any two objects was 21 . The experiment used a 2×2 design with preview type (same vs. different concept) and stimulus onset asynchrony (SOA; 50 vs. 250 ms) as crossed within subject and item factors. We chose these timings to ensure that the preview would likely be fully presented by the time the subject moved his/her eyes to the target (to avoid data loss due to the exclusion of trials in which the display change to the target started before the offset of the preview)2.
Figure 2.
The display during a trial in the experiment. Subjects are to name the top-left object, then the top-right object, then the bottom object. On a correct trial, a subject would say “book, arm, map.” During fixation on the first object (the book), the second object location appears as a gray box. After either 50 or 250 ms (SOA) the preview appears in the second location. The preview is either the same concept (with a different visual token) or a different concept as the target. Once the subject makes a saccade to the second object location the target appears there.
Procedure
To ensure subjects used the correct names, the experiment began with a training session in which all line drawings, including the preview objects, were presented individually at the center of the monitor. Subjects were instructed to name each object as it appeared, and naming errors were corrected (all intended object names were monosyllabic and common terms for the objects). The objects were displayed until the correct response was made or until the experimenter corrected the subject.
After training, the subject put on a headset microphone and a target sticker was placed on his/her forehead to monitor head movements. To record movements of the eyes in both the x- and y-axes, the tracker was calibrated using a 9-point calibration. At the start of each trial, the subject saw a fixation point in the center of the monitor. If the tracker was accurately calibrated, the experimenter pressed a button, causing the fixation point to disappear and a black box to appear in the top left quadrant of the monitor (the location of the first object to ensure that the subject was looking in that location when the trial started). Once a fixation was detected in this region, images appeared in all locations, with objects appearing in the first and third locations and a gray box appeared in the second (target) location.
Depending on the SOA condition, the preview appeared in the second location either 50 or 250 ms after trial onset (fixation on the black box that triggered the objects to appear). In all conditions, the preview was displayed for 200 ms before reverting to a gray box. When the subject’s eyes crossed an invisible boundary located to the right of the first object, the target appeared in place of the gray box. This occurred regardless of whether the display changes of the preview objects were complete. If the display change to the target occurred before the preview offset (< 1% of the time), the trial was excluded. The display change (which took, on average, 15 ms) occurred during a saccade (when vision is suppressed) that took approximately 50–60 ms to complete. The three objects remained in view until the subject named them all. The experimenter then pressed a button to end the trial and code naming accuracy of the first and second objects. Subjects were not informed of the display changes in the instructions. However, at the end of the experiment, they were asked if they were aware of the display changes and to report the percentage of the time they could identify what the preview object was. The experiment lasted 30 minutes.
Results and Discussion
Data were excluded if subjects (a) misnamed or disfluently named the first or target object (i.e., the first word in the utterance was not the target name; 2% of the data), (b) there was track loss on the first or target object (15% of the data), (c) the target object was not fixated (<1% of the data), or (d) the display causing the target to appear was triggered before the offset of the preview (< 1% of the data). There were 7436 observations remaining after these exclusions (82% of the original data).
We report inferential statistics based on generalized linear mixed-effects models (LMMs). In the LMMs, preview type (same vs. different concept) and SOA (50 vs. 250 ms) were centered and entered as fixed effects, and subjects and items were entered as crossed random effects (Baayen, Davidson, & Bates, 2008) with the maximal random effects structure (Barr, Levy, Scheepers, & Tily, 2013). To fit the LMMs, the lmer function from the lme4 package (Bates, Maechler, & Bolker, 2011) was used within the R Environment for Statistical Computing (R Development Core Team, 2012). We report regression coefficients (b) that estimate the effect size (in milliseconds) of the reported comparison, standard errors, and the t-value of the effect coefficient (for which a value greater than or equal to 1.96 indicates an effect that is significant at approximately the .05 alpha level).
Gaze Duration on the First Object
Gaze duration on the first (pre-target) object was unaffected by preview type (t < .49) because, at this time, the subject did not know whether the preview was related or unrelated to the target (which had not yet appeared). There was a significant effect of SOA (b = 22.96, SE = 8.33, t = 2.75), with longer gaze durations on the pre-target picture when the preview appeared at the 250 ms SOA (Msame concept = 864 ms, Mdifferent concept = 866 ms) than when it appeared at the 50 ms SOA (Msame concept = 867 ms, Mdifferent concept = 871 ms). There was no interaction between preview type and SOA (t < .09).
Gaze Duration on the Target Object
There was a significant effect of preview type (b = 20.37, SE = 5.29, t = 3.86) with shorter gaze durations on the target when the preview was the same concept as the target than when it was unrelated. There was also a significant effect of SOA (b = 12.13, SE = 6.12, t = 1.98) with longer gaze durations on the target when the preview appeared at the later SOA3. There was no interaction between preview type and SOA (t < 0.31) suggesting that the magnitude of the preview benefit was unaffected by when, relative to trial onset, the preview appeared (Figure 3).
Figure 3.
Gaze duration on the target object as a function of preview type (same vs. different concept) and SOA (50 vs. 250 ms after trial onset).
The Effect of Saccade Latency on Preview Benefit
If preview benefit only accompanies an attention shift preceding an eye movement, one would expect that it would be larger when the preview appeared shortly before the subject moved his/her eyes to the target. If so, we would expect to see an interaction between preview type and preview offset-to-saccade latency such that preview benefit is larger at shorter latencies (when the preview would coincide with an attention shift to that location). If an interaction were not observed it would suggest that preview benefit during multiple object naming does not only occur during an attention shift preceding a saccade. To test this, we ran an additional model on all data in which, instead of SOA, the continuous variable of the latency from preview offset to the saccade to the target (binned in 100 ms intervals)4 was centered and entered as a predictor (both in the fixed effects and random effects). In this model, the effect of preview was significant (b = 18.08, SE = 5.38, t = 3.36) as was the effect of latency (b = .05, SE = .02, t = 2.50). Again, there was no interaction between preview type and preview offset-to-saccade latency (t < .77), suggesting that the magnitude of the preview benefit was the same, regardless of saccade latency.
Together, these data suggest that preview benefit during multiple object naming is fairly stable, and does not depend on the timing of the preview relative to the saccade to the target. This indicates that speakers do not restrict pre-processing of the upcoming object to an attention shift before the saccade. Neither the experimental manipulation of preview SOA nor the latency between preview offset and the saccade to the target modulated the magnitude of preview benefit in our task (preview benefits were a fairly stable, regardless of these variables; Figure 4).
Figure 4.
Gaze duration on the target as a function of preview type and preview offset-to-saccade latency (binned by 100 ms). Only saccade latencies between 200 and 1200 ms are included in the figure for clarity, because data at more extreme latencies are much noisier (note the greater variability of the data points above 800ms in the figure). All data points were used in the statistical model.
The Effect of Preview Awareness on Preview Benefit
It is also possible that, rather than pre-processing the upcoming object continuously, subjects processed the preview because the sudden visual onset attracted their attention (although, in a previous study with display changes during fixations (Schotter et al., 2013) preview benefit was not obtained from objects that subjects were told to ignore, suggesting that display changes during fixation per se are not sufficient to cause preview benefit). If this were the case, we would expect to see that subjects who were more aware of the display changes (i.e., those who reported identifying the preview a large proportion of the time) would exhibit larger preview benefit than subjects who were unaware or less aware. Thus, we categorized subjects into three groups: unaware (reported identifying less than 2% of the previews- 11 subjects), less aware (reported identifying 2–65% of the previews- 25 subjects) and more aware (reported identifying more than 65% of the previews- 10 subjects). We entered this factor in the analyses (both as a fixed and a random effect for items with interactions with the other factors). There was a main effect of preview (b = 30.12, SE = 12.18, t = 2.47), but neither the effects of SOA (t = 1.31) nor awareness group, nor any of the interactions were significant (all ts < 1; Figure 5). Although self-reported post-experimental awareness may not be the most accurate measure of awareness during an experiment, this analysis suggests that preview benefit is, at least, independent of remembered awareness of the preview.
Figure 5.
Gaze duration on the target as a function of preview type and subjects’ awareness of the display changes.
Conclusion
The data reported here add to a growing body of research suggesting that speakers access information from to-be-named objects concurrently as they fixate preceding objects (see also Henderson et al., 1989; Meyer & Dobel, 2003; Meyer et al., 2008; Morgan & Meyer, 2005; Pollatsek et al., 1984, 1990; Schotter et al., 2013; for a review see Schotter, 2011). Crucially, though, the present study provides evidence that preview benefit occurs regardless of the latency of the preview relative to the onset of processing the prior object (our manipulation of SOA) or the latency of the saccade to the target location. Under the assumption that preprocessing only occurs during an attention shift preceding an eye movement (Henderson et al., 1989), preview benefit should show a negative relationship with saccade latency, but this was not observed. This finding suggests that speakers access information from objects they intend to name whenever the information is available. This may be a natural part of the speaking process; In general, speakers combine speaking about their environment with visual inspection of it (Griffin & Bock, 2000). For this reason, it may be advantageous for speakers to distribute attention broadly in order to allow apprehension of the scene.
One consequence of the fact that speakers promiscuously process multiple objects simultaneously is that the linguistic processing system requires a way to regulate the potentially overwhelming amount of incoming information. Other research of ours (Schotter et al., 2013) reveals that speakers restrict this processing to only those objects they intend to process (name), such that only to-be-named objects provide preview benefit and objects that are to-be-ignored do not affect processing. This work suggests that the results of the current study are not purely driven by the display changes capturing attention because a sudden onset is not sufficient to cause preview benefit; in that study, no preview benefit was observed when the subjects were told to ignore the object in that location (despite the sudden onset of the preview during fixation). This suggests that one way speakers avoid potentially accessing and naming the wrong object is by restricting the set of processed objects to only those that are intended.
Acknowledgments
This research was supported by grants DC000041, HD065829, and HD051030 from the National Institutes of Health. This paper was prepared in partial fulfillment of the requirements for the PhD in the Psychology Department at the University of California, San Diego. We thank Jane Morgan and an anonymous reviewer for helpful comments on an earlier version of the paper and Sarah D’Angelo for help with data collection
Footnotes
As noted by Schotter et al. (2013; see also Henderson et al., 1989), gaze duration (the sum of all fixations on the object before moving to another) is the dependent measure because it is sensitive to many properties of objects in production studies (visual degradation, word frequency, word length, codability, phonological properties, and preview). Naming latencies are not as diagnostic because they are more sensitive to idiosyncrasies of the words spoken before the target word and measurement of onset latency is less precise than measurement of gaze duration.
We chose these fixed timings rather than tailoring the SOAs to each subject because that would lead to lack of equivalency across subjects and because subjects tend to speed up or slow down across the experiment.
This is may be due to a psychological refractory period effect (Pashler, 1994; Welford, 1952). Later previews appear closer in time to the onset of target fixation. If there is a refractory period due to processing of the preview, this will be more likely for later than for earlier previews to extend into the period of the target gaze duration.
The pattern of data and significance tests show the same pattern when the data are not binned and raw latency values are used.
References
- Baayen R, Davidson D, Bates D. Mixed-effects modeling with crossed and random effects for subjects and items. Journal of Memory and Language. 2008;59:390–412. [Google Scholar]
- Barr DJ, Levy R, Scheepers C, Tily HJ. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language. 2013;68:255–278. doi: 10.1016/j.jml.2012.11.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bates DM, Maechler M, Bolker B. Lme4: Linear mixed-effects models using S4classes. R package version 0.999375-42. 2011 http://CRAN.R-project.org/package.lme4.
- Bates E, D'Amico S, Jacobsen T, Szekely A, Andonova E, Devescovi A, Herron D, Lu CC, Pechmann T, Pleh C, Wicha N, Federmeier K, Gerdjikova I, Gutierrez G, Hung D, Hsu J, Iyer G, Kohnert K, Mehotcheva T, Orozco-Figueroa A, Tzeng A, Tzeng O. Timed picture naming in seven languages. Psychonomic Bulletin & Review. 2003;10:344–380. doi: 10.3758/bf03196494. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Deubel H, Schneider WX. Saccade target selection and object recognition: Evidence for a common attentional mechanism. Vision Research. 1996;36:1827–1837. doi: 10.1016/0042-6989(95)00294-4. [DOI] [PubMed] [Google Scholar]
- Griffin Z, Bock JK. What they eyes say about speaking. Psychological Science. 2000;11:274–279. doi: 10.1111/1467-9280.00255. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Henderson JM. Eye movement control during visual object processing: Effects of initial fixation position and semantic constraint. Canadian Journal of Experimental Psychology. 1993;47:79–98. doi: 10.1037/h0078776. [DOI] [PubMed] [Google Scholar]
- Henderson JM, Pollatsek A, Rayner K. Covert visual attention and extrafoveal information use during object identification. Perception & Psychophysics. 1989;45:196–208. doi: 10.3758/bf03210697. [DOI] [PubMed] [Google Scholar]
- Henderson JM, Siefert A. Types and tokens in transsaccadic object identification: Effects of spatial position and left-right orientation. Psychonomic Bulletin & Review. 2001;8:753–760. doi: 10.3758/bf03196214. [DOI] [PubMed] [Google Scholar]
- Hoffman JE, Subramaniam B. The role of visual attention in saccadic eye movements. Perception & Psychophysics. 1995;57:787–795. doi: 10.3758/bf03206794. [DOI] [PubMed] [Google Scholar]
- Meyer A. The use of eye tracking in studies of sentence generation. In: Henderson JM, Ferreira F, editors. The interface of language, vision, and action: Eye movements and the visual world. New York: Psychology Press; 2004. pp. 191–211. [Google Scholar]
- Meyer A, Dobel C. Application of eye tracking in speech production research. In: Hyönä J, Radach R, Deubel H, editors. The mind’s eye: Cognitive and applied aspects of eye movement research. North Holland: Amsterdam; 2003. pp. 253–272. [Google Scholar]
- Meyer A, Ouellet M, Hacker C. Parallel processing of objects in a naming task. Journal of Experimental Psychology: Learning, Memory and Cognition. 2008;34:982–987. doi: 10.1037/0278-7393.34.4.982. [DOI] [PubMed] [Google Scholar]
- Morgan J, Meyer A. Processing of extrafoveal objects during multiple-object naming. Journal of Experimental Psychology: Learning Memory and Cognition. 2005;31:428–442. doi: 10.1037/0278-7393.31.3.428. [DOI] [PubMed] [Google Scholar]
- Morgan J, van Elswijk G, Meyer A. Extrafoveal processing of objects in a naming task: Evidence from word probe experiments. Psychonomic Bulletin & Review. 2008;15:561–565. doi: 10.3758/pbr.15.3.561. [DOI] [PubMed] [Google Scholar]
- Pashler H. Dual-task interference in simple tasks: Data and theory. Psychological Bulletin. 1994;116:220–244. doi: 10.1037/0033-2909.116.2.220. [DOI] [PubMed] [Google Scholar]
- Pollatsek A, Rayner K, Collins W. Integrating pictorial information across eye movements. Journal of Experimental Psychology: General. 1984;113:426–442. doi: 10.1037//0096-3445.113.3.426. [DOI] [PubMed] [Google Scholar]
- Pollatsek A, Rayner K, Henderson JM. Role of spatial location in integration of pictorial information across saccades. Journal of Experimental Psychology: Human Perception and Performance. 1990;16:199–210. doi: 10.1037//0096-1523.16.1.199. [DOI] [PubMed] [Google Scholar]
- R Development Core Team. R: A language and environment for statistical computing [Computer software] Austria: Vienna; 2012. [Google Scholar]
- R Foundation for Statistical Computing. Rayner K. The perceptual span and peripheral cues in reading. Cognitive Psychology. 1975;7:65–81. [Google Scholar]
- Rayner K. Eye movements in reading and information processing: 20 years of research. Psychological bulletin. 1998;124:372–422. doi: 10.1037/0033-2909.124.3.372. [DOI] [PubMed] [Google Scholar]
- Rayner K. The Thirty-fifth Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology. 2009;62:1457–1506. doi: 10.1080/17470210902816461. [DOI] [PubMed] [Google Scholar]
- Rayner K, Angele B, Schotter ER, Bicknell K. On the processing of canonical word order during eye fixations in reading: Do readers process transposed word previews? Visual Cognition. 2013 doi: 10.1080/13506285.2013.791739. (online first) [DOI] [PMC free article] [PubMed] [Google Scholar]
- Reichle ED, Liversedge SP, Pollatsek A, Rayner K. Encoding multiple words simultaneously in reading is implausible. Trends in Cognitive Science. 2009;13:115–119. doi: 10.1016/j.tics.2008.12.002. [DOI] [PubMed] [Google Scholar]
- Schotter E. Eye movements as an index of linguistic processing in language production. Journal of Psychology and Behavior. 2011;9:16–23. [Google Scholar]
- Schotter ER, Angele B, Rayner K. Parafoveal processing in reading. Attention, Perception, & Psychophysics. 2012;74:5–35. doi: 10.3758/s13414-011-0219-2. [DOI] [PubMed] [Google Scholar]
- Schotter ER, Ferreira VS, Rayner K. Parallel object activation and attentional gating of information: Evidence from eye movements in the multiple object naming paradigm. Journal of Experimental Psychology: Learning, Memory and Cognition. 2013;39:365–374. doi: 10.1037/a0028646. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Welford AT. The ‘Psychological Refractory Period’ and the timing of high-speed performance: A review and a theory. British Journal of Psychology. 1952;43:2–19. [Google Scholar]





