Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 Mar 28.
Published in final edited form as: Atten Percept Psychophys. 2010 May;72(4):1120–1129. doi: 10.3758/APP.72.4.1120

Video Game Players Show More Precise Multisensory Temporal Processing Abilities

Sarah E Donohue 1,2, Marty G Woldorff 1,2,3,4, Stephen R Mitroff 1,4
PMCID: PMC3314265  NIHMSID: NIHMS362383  PMID: 20436205

Abstract

Recent research has demonstrated enhanced visual attention and visual perception in individuals with extensive experience playing action video games. These benefits manifest in several realms, but much remains unknown about the ways in which video game experience alters perception and cognition. The current study examined whether video game players’ benefits generalize beyond vision to multisensory processing by presenting video game players and non-video game players auditory and visual stimuli within a short temporal window. Participants performed two discrimination tasks, both of which revealed benefits for video game players: In a simultaneity judgment task, video game players were better able to distinguish whether simple visual and auditory stimuli occurred at the same moment or slightly offset in time, and in a temporal-order judgment task, they revealed an enhanced ability to determine the temporal sequence of multisensory stimuli. These results suggest that people with extensive experience playing video games display benefits that extend beyond the visual modality to also impact multisensory processing.

Introduction

We are constantly bombarded with both visual and auditory stimuli, all of which must be rapidly processed to construct a veridical representation of our environment. While this is seemingly accomplished with ease, a series of complicated processes and principles have been shown to underlie multisensory perception (see Driver & Noesselt, 2008; Stein & Stanford, 2008 for reviews). For example, one fundamental principle of multisensory processing is that input from different modalities must be perceived in temporal synchrony to be bound into a single multisensory object (e.g., Meredith, Nemitz, & Stein, 1987; Stein and Meredith, 1993). This can be easily evidenced by focusing on the effects of temporal asynchrony, when the stimulus components from different modalities occur separated by too large of a temporal gap, such as in a badly-dubbed movie or when computer processors temporarily freeze resulting in where what is typed (i.e., the tactile input) is delayed with respect to what appears (i.e., the visual display). It certainly can be distracting when stimuli from multiple modalities are temporally misaligned; however, how far apart in time do these stimuli need to be before we begin to notice such temporal discrepancies? Or, conversely, how far apart in time can multisensory information be while still being bound into one perceptual representation? Moreover, in a complex multisensory world where it is of fundamental importance to be able to accurately link corresponding cross-modal inputs and separate non-corresponding ones, how might the ability to accurately make such links be affected by an individual’s prior perceptual experiences?

While presumably not done so explicitly, most prior work has appeared to assume that there is relatively little difference across individuals in their temporal window of integration (i.e., how close together in time stimuli must occur in order to be perceptually integrated into a single, multisensory object). This is manifested in that individual-participant data are rarely reported (but see Stone, et al., 2001; Vatakis, Navarra, Soto-Faraco, & Spence, 2007), with most studies not examining individual or group differences (e.g., Spence, Shore, & Klein, 2001; van Wassenhove, Grant, & Poeppel, 2007; Zampini, Guest, & Shore, 2005). Might each individual’s prior experiences and life history influence his or her perceptual processing? Extreme cases suggest yes, revealing for example altered behavior (Putzar, Goerendt, Lange, Rosler, & Roder, 2007) and differential neural processing in auditory-attention tasks in early-blind participants (Liotti, Ryder, & Woldorff, 1998) and differential neural firing patterns (for sensory integration) in animals deprived of early sensory input (Carriere, et al., 2007; Ghoshal, Pouget, Popescu, & Ebner, 2009). Given that differential life-related perceptual experiences can lead to altered neural activity and perceptual abilities, here we ask whether certain individuals with a specific type of experience (extensive video game playing) have a more fine-tuned sense of temporal synchrony that enables a greater ability to notice slight asynchronies.

Tools to distinguish multisensory temporal processing

To delineate individuals’ temporal window of integration, we used two standard well-established tasks: the simultaneity judgment task and the temporal-order judgment task. In multisensory simultaneity judgment tasks, participants are presented with stimuli from two different modalities that can occur either simultaneously or with various temporal offsets. The temporal offsets typically occur in both directions (i.e., a visual stimulus could proceed or follow an auditory stimulus) at varying stimulus onset asynchronies (SOAs). Participants are simply to report if the stimuli appeared simultaneously or asynchronously (e.g., Stone, et al., 2001). Likewise, in multisensory temporal-order judgment tasks (e.g., Zampini, Shore, & Spence, 2003), participants are presented with stimuli that occur either simultaneously or are offset by various SOAs and are to judge which modality came first (e.g., was the auditory stimulus presented first, or was the visual stimulus presented first?). There is clearly some redundancy between the temporal-order and simultaneity judgment tasks, but subtle differences in their effects suggest they may operate by somewhat different mechanisms. As such it is beneficial to employ both of them to assess crossmodal processing (see van Eijk, Kohlrausch, Juola, & van de Par, 2008; Vatakis, Navarra, et al., 2007, for discussion).

In simultaneity judgment tasks, when stimuli in different modalities (e.g., auditory and visual) are presented at offsets at or close to physical simultaneity, participants typically judge these stimuli to be simultaneous. As the SOAs between the stimuli increase, the reports of perceptual simultaneity gradually decrease, falling off more and more as the stimuli get farther away from physical simultaneity. This task is particularly sensitive to temporal offsets at longer SOAs, highlighting those SOAs at which participants determine the stimuli to be temporally disparate. Based on this task, auditory and visual stimuli appear to be integrated into a single perceptual representation at SOAs from physical simultaneity (i.e., 0 ms apart) up to ~150 to 200 ms, after which the two stimuli are perceived as distinct (Schneider & Bavelier, 2003; Zampini, Guest, et al., 2005). Accordingly, this temporal window of around 150 ms has been viewed as reflecting the typical temporal window of multisensory integration.

Temporal-order judgment tasks, in contrast to simultaneity judgment tasks, are most informative at short SOAs (e.g., 50 ms) where it is difficult to distinguish which stimulus came first (e.g., Zampini, et al., 2003). Performance is typically very good at longer SOAs (where it is obvious which stimulus appeared first) but individual differences can potentially arise at the more difficult, shorter SOAs. Together, simultaneity and temporal-order judgment tasks provide a complete picture of the temporal intervals at which information can be integrated or discriminated and we implemented both here to best assess an individuals’ temporal window of integration.

Finding individual differences in multisensory temporal integration

Although the SOAs over which crossmodal stimuli are integrated into one perceptual representation differ some across tasks and modalities (e.g., Zampini, et al., 2003 vs. Zampini, Brown, et al., 2005), the temporal window of multisensory integration is generally a reliable and replicable effect, with the likelihood of integration decreasing with increasing SOA (e.g., Spence, et al., 2001; Stone, et al., 2001; Vatakis, Bayliss, Zampini, & Spence, 2007; Vatakis & Spence, 2006). However, there is some evidence that temporal integration can be altered – in one study, when participants were repeatedly exposed to audio-visual stimuli with a given temporal offset, their subsequent simultaneity judgments were biased toward that offset (Vroomen, Keetels, de Gelder, & Bertelson, 2004). Additionally, a recent study has demonstrated that repeated pre-exposure is not modality specific in that pre-exposure to temporally offset stimuli (by 100 ms) in either the auditory/visual, visual/tactile or auditory/tactile modalities can shift the perception of simultaneity for audio-visual stimuli in the direction of that temporal offset (Harrar & Harris, 2008). These two examples offer an intriguing suggestion that temporal integration can be manipulated by prior experiences.

Beyond effects of immediate malleability of sensory perception, longer-term malleability has also been demonstrated. For example, extensive experience with a specific set of frequencies in an auditory discrimination task has been shown to shape neuronal responses and cortical organization in non-human primates (Recanzone, Schreiner, & Merzenich, 1993). More recently, striking evidence of long-term perceptual malleability in healthy adult humans has come from studies examining the effects of extensive action video game experiences (e.g., Castel, Pratt, & Drummond, 2005; Feng, Spence, & Pratt, 2007; Green & Bavelier, 2003, 2006a, 2006b, 2007; Quaiser-Pohl, Geiser, & Lehmann, 2006). Action video game players (VGPs) have been shown to have, among other benefits, heightened visual acuity (Green & Bavelier, 2007), enhanced contrast sensitivity (Li, Polat, Makous, & Bavelier, 2009), an improved ability to simultaneously track multiple moving visual items (Green & Bavelier, 2006b), better spatial abilities (e.g., Quaiser-Pohl, et al., 2006), enhanced divided attention abilities (e.g., Greenfield, et al., 1994), and improved eye-hand motor coordination (e.g., Griffith, Voloschin, Gibb, & Bailey, 1983). Non-video game players (NVGPs) who are trained on action video games for a relatively short time period reveal some VGP-like benefits, supporting the claim that the observed benefits arises from experience and not from pre-existing predilections (e.g., De Lisi & Wolford, 2002; DeLisi & Cammarano, 1996; Dorval & Pepin, 1986; Green & Bavelier, 2003, 2006a, 2006b, 2007; McClurg & Chaille, 1987; Okagaki & Frensch, 1994; but see Boot, Kramer, Simons, Fabiani, & Gratton, 2008; Gagnon, 1985; Rosenberg, Landsittel, & Averch, 2005; Sims & Mayer, 2002).

Video games are inherently multisensory, with first-person shooter and other action games often having both auditory and visual cues that are relevant to an appropriate behavioral response. High-action first-person shooter games combine intense visual graphics with corresponding and informative auditory cues and feedback, and can involve multiplayer interactions wherein players communicate with each other via auditory conversations. Given this multisensory nature of the games, it seems quite possible that the VGPs’ benefits would extend to multisensory processing, such as by affecting individuals’ window of multisensory temporal integration. Specifically, given that action video games bombard the players with multisensory stimuli that must be processed rapidly and accurately, it seems reasonable to hypothesize that VGPs would be better able to parse audio-visual information when they occur closely together in time.

In the current experiment we tested VGPs and NVGPs on both a simultaneity judgment task and a temporal-order judgment task, which allowed us to simultaneously ask two novel questions: (1) Do individual differences exist for multisensory temporal integration windows; and (2) Do video game playing benefits extend beyond vision to the realm of multisensory processing? In the simultaneity judgment task, we hypothesized that VGPs would show a narrower window of integration, being less likely to judge stimuli as occurring simultaneously when they were indeed physically asynchronous. That is, we predicted that VGPs would be more accurately able to distinguish asynchronies between closely occurring visual and auditory stimuli. Likewise, for the temporal-order judgment task, we predicted that VGPs would be better able to distinguish which stimulus (auditory or visual) occurred first at small SOAs, thereby also revealing enhanced perceptual discrimination abilities. On the other hand, we live in a complex world that is inherently multisensory, and so NVGPs should have abundant exposure to integrating visual and auditory information. Thus, another possibility was that there would be little-to-no benefit from playing action video games to multisensory temporal integration.

Methods

Participants

Forty-five male members of the Duke University community participated. Based upon assessments of their prior gaming experiences, we categorized these participants into 18 VGPs (mean age = 20, SD = 2.5), 18 NVGPs (mean age = 20.6, SD = 3.5), and 9 other participants whose experience with video games fell between these two categorical levels of gaming experience, which are described below (mean age = 19.2, SD = 1.4). Six additional participants (1 VGP, 4 NVGPs, and 1 other) were excluded due to poor behavioral performance, indicated by having points of subjective simultaneity (see Results Sections 1.2 & 2.2) in either task that exceeded the range of stimulus onset asynchronies tested. Similar to previous experiments (e.g., Green & Bavelier 2006a), no female participants were included due to difficulty in finding sufficient numbers of females with extensive gaming experience. Participants received either course credit or monetary compensation.

Video game experiences were assessed via a post-experiment questionnaire that asked about the length and amount of experience within several video game genres, as well as via a self-report of level of expertise with each genre. The questionnaire served two purposes. First, it provided a means to classify participants as a VGP, NVGP, or “other.” NVGPs were defined as those participants who had zero hours per week of first-person shooter experience in the past six months, as well as having less than 1.5 hours per week within the past 6 months of real time strategy and sports games (NVGP mean = 45 minutes per week). VGPs were defined as having at least 2 hours per week of first-person shooter experience in the past six months, as well as playing any type of action game (including first-person shooter, as well as including real-time strategy and sports games) for a minimum of 4.5 hours per week within the past 6 months (VGP mean = 11 hours per week). Additionally, VGPs had all played first-person shooter games for at least five hours per week at some point in their lives. The additional nine participants were excluded from the majority of the analyses, which were categorical VGP/NVGP comparisons, as their video game experiences fell between these two criteria; however, their data were included in correlational analyses (see below and Results section 3).

The second purpose of the video game questionnaire was to provide a means to quantify a participant’s amount of gaming experience on a continuous scale. Based upon the answers for each video game genre, we calculated for each participant an overall gaming score (i.e., a number between 0 and 317) that accounted for general gaming experience and expertise across all genres of video games. This score was used in the correlational analyses (see Results section 3).

Apparatus

Participants sat approximately 57 cm from a 19in CRT monitor in a quiet testing room. The auditory stimuli were presented centrally through two speakers evenly spaced to the left and right of the monitor, and the presentation of the visual and auditory stimuli was controlled by Presentation (Neurobehavioral Systems) on a Dell PC.

Stimuli

Each trial was comprised of a visual black-and-white square checkerboard pattern (5°×5°, 33 ms duration) and an auditory tone (33 ms duration; 60 dBSL; 5 ms rise-and-fall time; 1200Hz). Across trials, the visual and auditory stimuli appeared equally often with the following SOAs, in milliseconds, where negative represents auditory first, positive indicates auditory second (i.e, visual first), and 0 represents physical simultaneity: −300, −250, −200, −150, −100, −50, 0, 50, 100, 150, 200, 250, 300. The visual stimuli were either presented in the midline for a given block (see Figure 1), with the visual stimulus appearing centered 3.4° below a fixation cross, or presented laterally, with the visual stimulus appearing 12.3° to the left or right of midline and 3.4° below the level of the fixation cross. The auditory stimulus was always presented centrally regardless of the position of the visual stimulus. The variation of the spatial location of the visual stimulus was done for two reasons. First, it has been previously shown that the spatial position of the multisensory stimuli can influence the judgments of simultaneity and temporal order, with increased spatial separation yielding a decreased perception of simultaneity (e.g., Zampini, Guest, et al., 2005). Since VGPs had not previously been tested in multisensory paradigms of this nature, we wished to determine if the spatial separation between the stimuli would have more of an effect on the judgments for one group of participants than the other. Second, as VGPs have been previously shown to have particularly enhanced visual resolution and attention in the periphery (e.g., Green & Bavelier, 2003, 2007), it is possible that differences between VGPs and NVGPs would occur mainly or even only for stimuli presented in the periphery. In a given block, the spatial position of the visual stimuli was kept constant (e.g., presented only on the left for a given block) so that participants did not have to spatially shift attention across sides from trial to trial.

Figure 1. Experimental Task.

Figure 1

Open in a new tab

Depiction of experimental stimuli in the central and lateral conditions (left-side stimulus shown here). The auditory stimulus (represented here by a the symbol for a musical note) was presented centrally, and the visual stimulus’ location varied by block.

Procedure

Each participant completed both a simultaneity judgment task and a temporal-order judgment task, with the task order counterbalanced across participants. In the simultaneity judgment task, participants were asked to judge whether the auditory and visual stimuli were presented simultaneously or asynchronously, and to indicate their response with a keypress (‘1’ for simultaneous, ‘2’ for non-simultaneous; using a standard keyboard number pad). In the temporal-order judgment task, participants were asked to judge whether the auditory or the visual stimulus was presented first in time, again indicated with a keypress (‘1’ for auditory first, ‘2’ for visual first). Participants were instructed to be as accurate as possible and there was no response time limit. After each trial, participants pressed ‘0’ on the number pad to advance to the next trial. Each block was comprised of 12 trials at each SOA for a total of 156 randomly presented trials per block. There were 4 blocks per task (two with central, one with left, and one with right visual presentation) resulting in 624 total trials per task. Block order was randomized for each participant. Prior to the start of each task, participants completed a practice block of 12 trials.

Results

1.Simultaneity Judgment Task

The primary measure of interest was the proportion of ‘simultaneous’ responses at each audiovisual SOA. These proportion values were calculated for each SOA for each participant, separately for the central and lateral conditions. Preliminary analyses revealed no differences between the left and right lateral visual presentation trials, and so all lateral data were collapsed over left-right position. In that participants were instructed to prioritize accuracy over response time and no response time limits were employed, no response-time effects were found for either task and will not be discussed further.

1.1. Response Distributions

To examine the effect of lateralization on simultaneity judgments, we conducted a 2×2×13 mixed-design ANOVA on the ‘percent simultaneous’ responses, with VGP status (VGP vs. NVGP) as a between-subjects factor and stimulus position (central vs. lateral) and SOA (each of the 13 intervals) as within-subject factors. These analyses revealed main effects of SOA (F(1,12) = 105.33, p < 0.001) and an interaction of SOA x VGP status (F(1,12) = 3.84, p < 0.001), with only a trend toward significance for the effect of stimulus position (F(1,34) = 3.13, p = 0.09). Further, the interaction of VGP status and position was not significant. Because the position of stimuli had no significant effect on the response pattern, subsequent analyses were collapsed across central and lateral conditions (Figure 2A). As seen in Figure 2 and discussed below, the primary differences between VGPs and NVGPs occurred when the visual stimulus came first. Overall, however, VGPs showed a more narrow perceptual distribution function with more precise judgments at the various SOAs.

Figure 2. Results by task and data type.

Figure 2

Open in a new tab

The raw and Gaussian-fitted conditions for the simultaneity judgment task (panels A&B) are plotted as the ‘proportion simultaneous’ responses, and the corresponding conditions for the raw and Sigmoid-fitted temporal order judgment task (panels C&D) are plotted as the ‘proportion auditory first’ responses. SOAs represent the temporal asynchrony between the visual and auditory stimuli on a given trials, with negative values indicating the auditory stimulus preceded the visual, positive values indicating the auditory following, and 0 representing physical simultaneity. Asterisks denote significant differences between VGPs and NVGPs. Compared to NVGPs, VGPs were more accurate when the visual stimulus came before the auditory stimulus in the simultaneity judgment task and were more accurate at SOAs close to physical simultaneity in the temporal-order judgment task.

Planned post-hoc t-tests revealed that the VGPs differed significantly from the NVGPs at the visual-first SOAs of +150 ms (t(34) = 2.50, p = 0.02), +200 ms (t(34) = 2.23, p = 0.03), +250 ms (t(34) = 2.93, p = 0.006), and +300 ms (t(34) = 2.52, p = 0.02), and marginally differed from NVGPs at +100 (t(34) = 1.96, p = 0.06). For each of these SOAs, the VGPs more accurately reported the trials as ‘not simultaneous’ compared to the NVGPs.

1.2. Gaussian Fitting

To further characterize potential differences between VGPs and NVGPs we fit each participant’s data to a Gaussian function. The results of this fitting (and subsequent averaging for the VGP and NVGP groups) are shown in Figure 2B. As had been done with the raw data, above, the fitted data for each participant were analyzed in a 2×13 (VGP status x SOA) ANOVA. This analysis revealed a main effect of SOA (F(1,12) = 136.62, p < 0.001), and a SOA by VGP status interaction (F(1,12) = 5.08, p < 0.001). Subsequent planned t-test revealed that, as above, VGPs were more accurate than NVGPs (i.e., they were more likely to correctly judge the SOAs as non-simultaneous) at the SOAs of +100 ms (t(34) = 2.38, p = 0.02), +150 ms (t(34) = 2.62, p = 0.01), +200 ms (t(34) = 2.71, p = 0.01), +250 ms (t(34) = 2.79, p = 0.009), and +300 ms (t(34) = 2.82, p = 0.008).

1.3. Point of Subjective Simultaneity

For each participant we calculated their ‘point of subjective simultaneity:’ the SOA at which the participant was the most likely to judge the auditory and visual stimuli as occurring simultaneously. An ideal observer would have a point of subjective simultaneity at an SOA of 0, and an auditory-first biased observer would have a negative value on our scale used here. That is, a point of subjective simultaneity of −50 ms would mean that the observer would be most likely to judge the auditory and visual stimuli as occurring simultaneously when the auditory stimulus preceded the visual by 50 ms. Using each participant’s data that had been fit to a Gaussian function, we calculated the mean and the standard deviation of the distribution (Zampini, Shore, & Spence, 2005). The resulting mean gave the point of subjective simultaneity as it occurs at the SOA with the most ‘simultaneous’ responses, and the resulting standard deviation indicated the spread of the participant’s responses. This spread of responses served as a proxy for how difficult the participant found the task – the narrower their curve (i.e., the smaller the standard deviation), the easier the task was for them.

VGPs and NVGPs produced significantly different group averages for their points of subjective simultaneity (VGP M = −15.1 ms; NVGP M = +26.6 ms; t(34) = 3.09, p < 0.005), such that VGPs were biased towards perceiving auditory stimuli coming first as simultaneous, and NVGPs were biased towards perceiving visual stimuli coming first as simultaneous. Additionally, the point of subjective simultaneity for VGPs was closer to the veridical SOA of 0 (i.e., physical simultaneity) than that of NVGPS; moreover, the VGPs’ point of subjective simultaneity did not differ from 0 (t(17) = 1.68, p = 0.11), while NVGPs’ did (t(17) = 2.64, p = 0.02). VGPs also had a smaller within-subject standard deviation (VGP M = 127.0 ms; NVGP M = 160.3 ms; t(34) = 2.32, p = 0.03).

2. Temporal-Order Judgment Task

The proportion of ‘auditory first’ judgments were calculated for each participant at each SOA in the central and lateral conditions. Preliminary analyses for the lateral visual presentation trials revealed no differences between the left and right locations, so all lateral data were collapsed over position. Here we present analogous analyses as to those conducted for the simultaneity judgment task as well as additional analyses that reveal several nuanced differences.

2.1. Response Distributions

A 2×2×13 ANOVA was conducted with VGP status (VGP vs. NVGP) as a between-subjects factor, and Position (central vs. lateral, collapsed across left and right) and SOA (each of the 13 intervals) as within-subject factors. A significant main effect was observed for the SOA (F(1,12) = 260.45, p < 0.001), as would be expected, with both VGPs and NVGPs indicating that their perception of temporal order differed as a function of the SOA.. There was also a SOA x VGP interaction (F(1,12) = 2.56, p < 0.005), with VGPs and NVGPs showing somewhat different response patterns, as described below. There was no main effect of Position, however, nor any interaction of Position with any of the other factors. Because the position of the stimuli did not have a significant effect on the judgments, the central and lateral conditions were collapsed for all subsequent analyses (Figure 2C).

To determine which specific SOAs were driving the SOA x VGP interaction, post-hoc t-tests were conducted. These revealed that VGPs and NVGPs significantly differed at 0 ms (t(34) = 2.46, p = 0.02), with VGPs being closer to chance at this point, as should be the case for the forced-choice temporal order judgment of two stimuli that actually were simultaneous. Additional marginally significant effects were found at +200 ms (t(34) = 1.98, p = 0.056), +250 ms (t(34) = 1.73, p = 0.09), and +300 ms (t(34) = 1.73, p = 0.09), with VGPs being more likely to correctly report the auditory stimulus as coming last.

2.2. Sigmoid fitting

To further characterize the differences between groups, we fit the data from each participant to a sigmoid function. The averaged fitted data are shown in Figure 2D. We ran a 2×13 (gamer status x SOA) ANOVA for these data. This revealed a main effect of SOA (F(1,12) = 455.55, p < 0.001), again confirming that participants were distinguishing the stimuli at the various SOAs, and an SOA x VGP interaction (F(1,12) =1.85, p = 0.04). This interaction derived from VGPs trending to be more accurate with their judgments than NVGPs, and therefore having a higher percentage of ‘auditory first’ responses when the auditory stimulus physically came first and likewise, a higher percentage of ‘visual first’ responses when the visual stimulus physically came first. None of the post hoc t-tests revealed that VGPs differed from NVGPs at any particular SOA; however at −50 ms there was a trend for VGPs to be more accurate at judging the auditory stimuli as coming before the visual (t(34) = 1.89, p = 0.07). Additionally, no differences in the slope were observed between groups (p > 0.05).

2.3. Point of subjective simultaneity and measure of just noticeable difference

As for section 1.3, here we calculate the point of subjective simultaneity for each participant – the point at which participants were most likely to report the stimuli as being simultaneous (here the point at which participants were least able to discriminate which stimulus came first). In addition, we also calculated a just noticeable difference measure – the smallest SOA at which participants are able to accurately distinguish which stimulus came first in time (e.g., Coren, Ward, & Enns, 2004; Poliakoff, Shore, Lowe, & Spence, 2006; Spence, et al., 2001; Vatakis, Navarra, et al., 2007; Zampini, Brown, et al., 2005).

The differences between the VGPs and the NVGPs in these analyses tended to mirror those in the simultaneity judgment task, although they did not reach significance. More specifically, compared to the NVGPs, the VGPs had a point of subjective simultaneity that appeared to be slightly closer to veridical physical simultaneity, although the groups did not differ significantly from one another (VGP M = −1.47 ms, NVGP M = −4.67 ms). There was a slight trend for the just-noticeable difference to be smaller (closer to physical simultaneity) for the VGPs compared to the NVGPs, but this did not reach significance (VGP M = 120.00 ms, NVGP M = 140.84 ms; t(34) = 1.24, p = 0.22).

2.4 Other assessments of accuracy on the temporal-order judgment task

Our initial hypotheses had been that VGPs would be better than NVGPs at those specific SOAs at which the task was particularly difficult (i.e., at those SOAs close to physical simultaneity). To determine if this was indeed the case, we calculated the overall accuracy from −50 ms to + 50 ms for each participant. Comparing the data between groups revealed that at these specific SOAs, VGPs were indeed more accurate than NVGPs (VGP M = 61.30 % correct, NVGP M = 53.66 % correct; t(34) = 2.72, p = 0.01). Further, NVGPs were biased toward reporting that the visual stimulus came first at the SOA of 0 ms; their average response significantly differed from chance (t(17) = 2.40, p = 0.03), whereas the VGPs’ responses did not (t(17) = 1.00, p = 0.33).

The simultaneity judgment task revealed that VGPs differed from NVGPs when the visual stimulus came before the auditory at the larger SOAs. To examine possible similar effects here, we collapsed the proportion of ‘auditory first’ responses across SOAs from +200 to +300 ms and −200 to −300 ms. Doing this revealed that when the auditory stimulus came before the visual (i.e., from −300 to −200 ms) VPGs did not differ from NVGPs in their judgments (t(34) = .616, p = 0.54); however, when the auditory stimulus came after the visual (i.e., from +200 to +300 ms), VGPs and NVGPs did differ on their judgments, with VGPs being more likely to accurately report that the auditory stimulus came second (t(34) = 2.012, p = 0.05).

3. Correlation between VGP Status and Temporal Processing

Our continuous measure of video game experience calculated from the post-experiment questionnaire provides an additional means to assess the relationship between video game experiences and the temporal processing of multisensory stimuli. To do so we examined correlations between the participants’ amount of video game experience and their point of subjective simultaneity in the simultaneity judgment task. For this analysis, all the participants (n = 45) were included so that the amount of video game experience value was continuous, rather than just at the extremes (that is, we included those participants that were not classified as either VGPs or NVGPs). Our video game questionnaire score was directly related to the amount of experience a participant had had with playing video games such that a higher score equated to more experience (see Methods). The analysis indicated that the video game score significantly correlated with the point of subjective simultaneity (r = −0.39, p = 0.008, with a higher video game score correlating with a shift of the point of subjectivity towards an auditory-first bias (Figure 3A). Participants with more video game experience were more likely to perceive auditory stimuli that preceded visual stimuli as occurring simultaneously.

Figure 3. VGP Experience Correlates with Judgment.

Figure 3

Open in a new tab

Correlations between the amount of video game experience (score on our video game experiences questionnaire with a higher score signifying more experience) and point of subjective simultaneity (A) and the standard deviation (B) for the simultaneity judgment task across VGPs (N=18; closed circles), NVGPs (N=18, open circles), and participants whose gaming experience fell between these two categories (N=9; asterisks). A. As experience with video games increased, the point at which participants were likely to report the stimuli as appearing simultaneously shifted toward the SOAs where the auditory stimulus came before the visual stimulus. The confidence interval for the slope is −0,57 to −0.09. B. As experience with video games increased, the participants had a smaller standard deviation (i.e., they were more likely to correctly identify the stimuli as not occurring simultaneously). The confidence interval for the slope is −0.59 to −0.05.

As well, the standard deviation from the simultaneity judgment task (i.e., how great the spread was after the data had been fitted to a Gaussian distribution) correlated with the amount of gaming experience (r = −0.34, p = 0.02), showing that participants with more gaming experience had a smaller standard deviation (Figure 3B). Participants with increased gaming experience were accordingly more likely to correctly assess stimuli that were separated in time as occurring at different times than those participants with little gaming experience.

General Discussion

Summary

The present study had two main goals. First, we sought to examine if there were individual differences present in the temporal perception of auditory and visual information that were modulated by action video game experience. Second, we sought to determine if the visual benefits previously observed as the result of video game playing would translate to other modalities. Using two perceptual tasks (a simultaneity judgment task and a temporal-order judgment task), we found evidence that VGPs were able to distinguish auditory and visual stimuli as being temporally distinct at closer temporal intervals than NVGPs.

Simultaneity judgment task summary

Simultaneity judgment tasks are generally considered good indicators of determining when stimuli that are physically separated in time become perceptually separated. Effects are best observed at the larger SOAs, where temporal distinctness is more apparent (e.g., Schneider & Bavelier, 2003; Zampini, Guest, et al., 2005). In the current experiment, VGPs were generally more accurate at discriminating the non-simultaneity of the auditory and visual stimuli at smaller intervals compared to NVGPs. VGPs had a point of subjective simultaneity that did not differ from physical simultaneity (0 ms), while NVGPs had a point of subjective simultaneity that was shifted toward conditions in which the visual stimulus preceded the auditory stimulus, significantly differing from physical simultaneity.

Notably, significant differences between VGPs and NVGPs arose primarily when the visual stimulus preceded the auditory stimulus. One possible explanation for this is that VGPs may have heightened sustained visual attention (e.g., Green & Bavelier, 2006a), allowing them to focus their attention to the spatial position of the visual stimulus more quickly and accurately, which in turn allows them to distinguish that the subsequent auditory input did not occur simultaneously with the visual input. However, as the central and lateral conditions did not differ or interact with gamer status, it seems unlikely that this explanation alone could account for the observed differences. Another possibility that is consistent with our data is that VGPs may be able to more rapidly process visual stimuli (e.g., Green & Bavelier, 2003), thereby allowing them to more quickly have attentional and perceptual resources available to distinguish the subsequent auditory input from the visual, rather than needing to devote continued resources toward processing the visual. Action video games require the rapid processing of vast amounts visual information, and it is highly possible that extensive experience with these games would lead to more efficient visual processing.

Temporal-order judgment task summary

The temporal-order judgment task revealed that the VGPs were generally better than the NVGPs at being able to distinguish which stimulus came first, showing a more ideal behavioral pattern around SOAs close to physical simultaneity. Here, too the VGPs were better than the NVGPs when the auditory stimulus came after the visual, showing more accurate judgments at the largest positive SOAs (+200 to +300 ms). Interestingly, NVGPs had a bias toward reporting the visual stimulus coming first at the SOA of 0 ms, while VGPs were at chance at this SOA, showing more precision in their judgments. This bias for the NVGPs to report the visual stimulus first could be the result of a form of attentional capture. While it has been shown that NVPGs are less able to spread their attention throughout space or time in within-modality tasks (Green & Bavalier, 2003, 2006a), it is equally possible that they are unable to spread their attention across modalities as well. The result of this could be that their attention is pulled, or captured, by the most salient stimulus, which in this case may be the visual stimulus as it has more features in its pattern than the simple auditory tone. If their attention were pulled toward this visual stimulus, then NVGPs might be more likely to judge it as occurring first since it was the first to capture their attention.

Task differences and biases

Simultaneity judgment and temporal order judgment tasks are thought to tap into somewhat different underlying mechanisms (van Eijk, et al., 2008). This may explain some of the subtle differences revealed in the current study. For example, when the visual stimulus preceded the auditory stimulus, VGPs revealed robust differences in the simultaneity judgment task relative to NVGPs, with smaller, but still significant, differences in the temporal-order judgment task. Another potential difference between these two tasks that may introduce a bias into the simultaneity judgment task is the nature of the response requirement. Given that there was an equal probability in both tasks of one stimulus coming before the other (i.e., auditory or visual first), in the temporal-order judgment task both responses of ‘auditory first’ or ‘visual first’ would occur equally likely. In the simultaneity judgment task, however, only 1 of out 13 SOAs was physically simultaneous (0 ms SOA), and thus one could argue that this task creates an artificial bias toward responding ‘non-simultaneous’. Importantly, however, auditory and visual information separated by SOAs ranging from approximately +150 ms to −150 ms are typically reported as occurring simultaneously (e.g., Zampini et al., 2005), thus resulting in a much more balanced distribution of perceptually simultaneous and asynchronous trials, relative to the actual physical distribution. In addition, single-unit recording in the superior colliculus in animals has indicated that stimuli occurring within this temporal window of +150 ms to −150 ms are integrated into a single representation (e.g., Meredith et al., 1987). Therefore while the absolute physical stimuli presented may be biased toward non-simultaneity, the behavioral and neural responses suggest a more even balance.

Possible mechanisms underlying VGPs’ benefits

While much of our discussion has focused on attention, prior research suggests that the performance differences between VGPs and NVGPs may be due to other underlying elements. Much of the current evidence for VGPs’ benefits suggests that action video game playing alters both attentional and perceptual abilities (e.g., Green & Bavelier, 2003, 2007; Li, et al., 2009). For example, VGPs (and NVGPs exposed to a video game training regimen) reveal enhanced visual acuity (Green & Bavelier, 2007) and contrast sensitivity (Li, et al., 2009). Likewise, it has been suggested that VGPs and NVGPs may employ similar cognitive strategies, but that VGPs do so with an added benefit of enhanced response-mapping abilities (Castel, et al., 2005).

Beyond such visual and attentional benefits, there have also been discussions of motivational or strategic benefits that can arise from extensive video game playing (e.g., Fleck & Mitroff, 2008; see also Green & Bavelier, 2006b). It is certainly possible that the differences between VGPs and NVGPs in these and other tasks were the result of more global strategic differences rather than, or perhaps in addition to, differences in attentional and perceptual abilities. Because this task was done on a computer, VGPs may have been more motivated to perform well since many games use a computer (or similar) interface, and VGPs may have been more “in their element”. However, if increased motivation to perform well were underlying the VGPs’ improved performance here, we should expect to see uniform improvements for VGPs over NVGPs. That is, regardless if the auditory stimulus preceded the visual or followed it, VGPs should differ from (i.e., be better than) NVGPs. However, our effects revealed a clear asymmetry in the simultaneity judgment task, as well as weaker but corroborating effects in the temporal-order judgment task, wherein VGPs showed significant improvements only when the auditory stimulus followed the visual. Thus, while more work needs to be done to fully determine motivational differences between VGPs and NVGPs, it seems unlikely that the currently observed effects were due to differences in motivation alone.

Causal effect of video game playing?

An important question concerns whether the multisensory benefits observed here were caused by extensive action video game play, or if people with a priori enhanced abilities were just more likely to have engaged in action video game play in their lives. Previous studies have trained NVGPs with video games (i.e., having them play action video games for 10 to 50 hours over the course of a training regimen) and have found they subsequently reveal effects typical of VGPs (e.g., De Lisi & Wolford, 2002; Dorval & Pepin, 1986; Green & Bavelier, 2003, 2006b, 2007; Okagaki & Frensch, 1994). On the other hand, some other studies have not revealed such training benefits (e.g., Boot, et al., 2008; Gagnon, 1985; Rosenberg, et al., 2005; Sims & Mayer, 2002).

Although a training component was not included in the current study, the data acquired here, in particular the asymmetry of the effects, can provide some insight on this issue. More specifically, the amount of our participants’ video game playing experience correlated with their point of subjective simultaneity and the associated standard deviation (Results Section 3; Figure 3). In the simultaneity judgment task, participants with more video game experience were more likely to perceive the stimuli as occurring simultaneously when the visual stimulus followed the auditory, while participants with less experience were more likely to perceive the stimuli as occurring simultaneously when the visual stimulus preceded the auditory. Thus, although we cannot infer causation from this correlation, this observed relationship between the amount of experience and subjective perception, together with the previous training studies, suggests that extensive video game experience may in fact lead to altered multisensory perception.

Conclusions

In a world where humans are constantly facing a rapid barrage of stimuli from multiple modalities, it is of fundamental importance to be able to accurately integrate corresponding information and parse non-corresponding information. We found that participants with extensive action video game experience are better able to distinguish events that occur close together in time, revealing enhanced multisensory perception and integration. These findings shed new light on individual differences in temporal aspects of multisensory integration and add to the growing body of evidence that suggests the importance of individual experience on perception.

Acknowledgments

For helpful conversation, we thank the other members of the Mitroff and Woldorff Labs. We also thank Melissa Bulkin and Ricky Green for assistance with data collection and Kenneth Roberts for technical assistance. We also thank Dr. Jeremy Wolfe and three anonymous reviewers for their helpful suggestions. This research was supported by an NIMH grant (R03-MH080849) awarded to S.R.M., an NINDS grant (R01-NS051048) awarded to M.G.W., and an NSF Graduate Research Fellowship awarded to S.E.D.

References

  1. Boot WR, Kramer AF, Simons DJ, Fabiani M, Gratton G. The effects of video game playing on attention, memory, and executive control. Acta Psychologica. 2008;129(3):387–398. doi: 10.1016/j.actpsy.2008.09.005. [DOI] [PubMed] [Google Scholar]
  2. Carriere BN, Royal DW, Perrault TJ, Morrison SP, Vaughan JW, Stein BE, et al. Visual deprivation alters the development of cortical multisensory integration. Journal of Neurophysiology. 2007;98(5):2858–2867. doi: 10.1152/jn.00587.2007. [DOI] [PubMed] [Google Scholar]
  3. Castel AD, Pratt J, Drummond E. The effects of action video game experience on the time course of inhibition of return and the efficiency of visual search. Acta Psychologica. 2005;119(2):217–230. doi: 10.1016/j.actpsy.2005.02.004. [DOI] [PubMed] [Google Scholar]
  4. Coren S, Ward LM, Enns JT. Sensation and perception. 6th ed. Wiley; Hoboken, NJ: 2004. [Google Scholar]
  5. De Lisi R, Wolford JL. Improving children’s mental rotation accuracy with computer game playing. Journal of Genetic Psychology. 2002;163(3):272–282. doi: 10.1080/00221320209598683. [DOI] [PubMed] [Google Scholar]
  6. DeLisi R, Cammarano DM. Computer experience and gender differences in undergraduate mental rotation performance. Computers in Human Behavior. 1996;12(3):351–361. [Google Scholar]
  7. Dorval M, Pepin M. Effect of playing a video game on a measure of spatial visualization. Perceptual and Motor Skills. 1986;62(1):159–162. doi: 10.2466/pms.1986.62.1.159. [DOI] [PubMed] [Google Scholar]
  8. Driver J, Noesselt T. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron. 2008;57(1):11–23. doi: 10.1016/j.neuron.2007.12.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Feng J, Spence I, Pratt J. Playing an action video game reduces gender differences in spatial cognition. Psychological Science. 2007;18(10):850–855. doi: 10.1111/j.1467-9280.2007.01990.x. [DOI] [PubMed] [Google Scholar]
  10. Fleck M, Mitroff SR. Videogamers excel at finding rare targets [Abstract] Journal of Vision. 2008;8(6):313, 313a. http://journalofvision.org/8/6/313/, doi:10.1167/8.6.313. [Google Scholar]
  11. Gagnon D. Videogames and spatial skills - an exploratory study. Educational Communication and Technology Journal. 1985;33(4):263–275. [Google Scholar]
  12. Ghoshal A, Pouget P, Popescu M, Ebner F. Early Bilateral Sensory Deprivation Blocks the Development of Coincident Discharge in Rat Barrel Cortex. Journal of Neuroscience. 2009;29(8):2384–2392. doi: 10.1523/JNEUROSCI.4427-08.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Green CS, Bavelier D. Action video game modifies visual selective attention. Nature. 2003;423(6939):534–537. doi: 10.1038/nature01647. [DOI] [PubMed] [Google Scholar]
  14. Green CS, Bavelier D. Effect of action video games on the spatial distribution of visuospatial attention. Journal of Experimental Psychology: Human Perception and Performance. 2006a;32(6):1465–1478. doi: 10.1037/0096-1523.32.6.1465. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Green CS, Bavelier D. Enumeration versus multiple object tracking: the case of action video game players. Cognition. 2006b;101(1):217–245. doi: 10.1016/j.cognition.2005.10.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Green CS, Bavelier D. Action-video-game experience alters the spatial resolution of vision. Psychological Science. 2007;18(1):88–94. doi: 10.1111/j.1467-9280.2007.01853.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Greenfield PM, DeWinstanley P, Kilpatrick H, Kaye D. Action video games and informal education: Effects on strategies for dividing visual attention. Journal of Applied Developmental Psychology. 1994;15:105–123. [Google Scholar]
  18. Griffith JL, Voloschin P, Gibb GD, Bailey JR. Differences in eye-hand motor coordination of video-game users and non-users. Perceptual and Motor Skills. 1983;57(1):155–158. doi: 10.2466/pms.1983.57.1.155. [DOI] [PubMed] [Google Scholar]
  19. Harrar V, Harris LR. The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Experimental Brain Research. 2008;186(4):517–524. doi: 10.1007/s00221-007-1253-0. [DOI] [PubMed] [Google Scholar]
  20. Li R, Polat U, Makous W, Bavelier D. Enhancing the contrast sensitivity function through action video game training. Nat Neurosci. 2009;12(5):549–551. doi: 10.1038/nn.2296. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Liotti M, Ryder K, Woldorff MG. Auditory attention in the congenitally blind: where, when and what gets reorganized? Neuroreport. 1998;9(6):1007–12. doi: 10.1097/00001756-199804200-00010. [DOI] [PubMed] [Google Scholar]
  22. McClurg PA, Chaille C. Computer games - environments for developing spatial cognition. Journal of Educational Computing Research. 1987;3(1):95–111. [Google Scholar]
  23. Meredith MA, Nemitz JW, Stein BE. Determinants of multisensory integration in superior colliculus neurons .1. Temporal factors. Journal of Neuroscience. 1987;7(10):3215–3229. doi: 10.1523/JNEUROSCI.07-10-03215.1987. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Okagaki L, Frensch PA. Effects of video game playing on measures of spatial performance: Gender effects in late adolescence. Journal of Applied Developmental Psychology. 1994;15:33–58. [Google Scholar]
  25. Poliakoff E, Shore DI, Lowe C, Spence C. Visuotactile temporal order judgments in ageing. Neuroscience Letters. 2006;396(3):207–211. doi: 10.1016/j.neulet.2005.11.034. [DOI] [PubMed] [Google Scholar]
  26. Putzar L, Goerendt I, Lange K, Rosler F, Roder B. Early visual deprivation impairs multisensory interactions in humans. Nature Neuroscience. 2007;10(10):1243–1245. doi: 10.1038/nn1978. [DOI] [PubMed] [Google Scholar]
  27. Quaiser-Pohl C, Geiser C, Lehmann W. The relationship between computer-game preference, gender, and mental-rotation ability. Personality and Individual Differences. 2006;40(3):609–619. [Google Scholar]
  28. Recanzone GH, Schreiner CE, Merzenich MM. Plasticity in the frequency representation of primary auditory-cortex following discrimination-training in adult owl monkeys. Journal of Neuroscience. 1993;13(1):87–103. doi: 10.1523/JNEUROSCI.13-01-00087.1993. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Rosenberg BH, Landsittel D, Averch TD. Can video games be used to predict or improve laparoscopic skills? Journal of Endourology. 2005;19(3):372–376. doi: 10.1089/end.2005.19.372. [DOI] [PubMed] [Google Scholar]
  30. Schneider KA, Bavelier D. Components of visual prior entry. 2003 doi: 10.1016/s0010-0285(03)00035-5. [DOI] [PubMed] [Google Scholar]
  31. Sims VK, Mayer RE. Domain specificity of spatial expertise: The case of video game players. Applied Cognitive Psychology. 2002;16(1):97–115. [Google Scholar]
  32. Spence C, Shore DI, Klein RM. Multisensory prior entry. Journal of Experimental Psychology: General. 2001;130(4):799–832. doi: 10.1037//0096-3445.130.4.799. [DOI] [PubMed] [Google Scholar]
  33. Stein BE, Stanford TR. Multisensory integration: current issues from the perspective of the single neuron. Nature Reviews Neuroscience. 2008;9(4):255–266. doi: 10.1038/nrn2331. [DOI] [PubMed] [Google Scholar]
  34. Stone JV, Hunkin NM, Porrill J, Wood R, Keeler V, Beanland M, et al. When is now? Perception of simultaneity. Proceedings of the Royal Society of London Series B-Biological Sciences. 2001;268(1462):31–38. doi: 10.1098/rspb.2000.1326. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. van Eijk RLJ, Kohlrausch A, Juola JF, van de Par S. Audiovisual synchrony and temporal order judgments: Effects of experimental method and stimulus type. Perception & Psychophysics. 2008;70(6):955–968. doi: 10.3758/pp.70.6.955. [DOI] [PubMed] [Google Scholar]
  36. van Wassenhove V, Grant KW, Poeppel D. Temporal window of integration in auditory-visual speech perception. Neuropsychologia. 2007;45(3):598–607. doi: 10.1016/j.neuropsychologia.2006.01.001. [DOI] [PubMed] [Google Scholar]
  37. Vatakis A, Bayliss L, Zampini M, Spence C. The influence of synchronous audiovisual distractors on audiovisual temporal order judgments. Perception & Psychophysics. 2007;69(2):298–309. doi: 10.3758/bf03193751. [DOI] [PubMed] [Google Scholar]
  38. Vatakis A, Navarra J, Soto-Faraco S, Spence C. Temporal recalibration during asynchronous audiovisual speech perception. Experimental Brain Research. 2007;181(1):173–181. doi: 10.1007/s00221-007-0918-z. [DOI] [PubMed] [Google Scholar]
  39. Vatakis A, Spence C. Temporal order judgments for audiovisual targets embedded in unimodal and bimodal distractor streams. Neuroscience Letters. 2006;408(1):5–9. doi: 10.1016/j.neulet.2006.06.017. [DOI] [PubMed] [Google Scholar]
  40. Vroomen J, Keetels M, de Gelder B, Bertelson P. Recalibration of temporal order perception by exposure to audio-visual asynchrony. Cognitive Brain Research. 2004;22(1):32–35. doi: 10.1016/j.cogbrainres.2004.07.003. [DOI] [PubMed] [Google Scholar]
  41. Zampini M, Brown T, Shore DI, Maravita A, Roder B, Spence C. Audiotactile temporal order judgments. Acta Psychologica. 2005;118(3):277–291. doi: 10.1016/j.actpsy.2004.10.017. [DOI] [PubMed] [Google Scholar]
  42. Zampini M, Guest S, Shore DI. Audio-visual simultaneity judgments. Perception & Psychophysics. 2005;67(3):531–544. doi: 10.3758/BF03193329. [DOI] [PubMed] [Google Scholar]
  43. Zampini M, Shore DI, Spence C. Audiovisual temporal order judgments. Experimental Brain Research. 2003;152(2):198–210. doi: 10.1007/s00221-003-1536-z. [DOI] [PubMed] [Google Scholar]
  44. Zampini M, Shore DI, Spence C. Audiovisual prior entry. Neuroscience Letters. 2005;381(3):217–222. doi: 10.1016/j.neulet.2005.01.085. [DOI] [PubMed] [Google Scholar]

RESOURCES