Workload assessment of computer gaming using a single-stimulus event-related potential paradigm

Abstract

Behavioral and event-related potential (ERP) measures were used to assess cognitive workload from expert computer gamers playing a “first-person shooter” video game. Game difficulty level was manipulated in separate conditions by adjusting the number of enemies (view, easy, medium, hard). Infrequently presented single-stimulus tones were either ignored or counted across difficulty conditions. Game performance and tone-counting accuracy declined as game difficulty increased. ERP component amplitudes diminished for both the tone ignore and counting conditions as game difficulty increased. The findings suggest that cognitive workload induced by video gaming can be reliably assessed through behavioral and neuroelectric means, and that the single-stimulus paradigm can be a useful tool for evaluating workload in an immersive stimulus environment with less distraction than conventional tools.

1. Introduction

1.1. P300 and workload

Assessment of cognitive workload has employed event-related potential (ERP) components, which evince reliable changes as primary and/or secondary task difficulty are manipulated (Dyson et al., 2005; Humphrey and Kramer, 1994; Parasuraman, 1980; Senkowski and Hermann, 2002; Singhal and Fowler, 2004; Sirevaag et al., 1984; Trejo et al., 1995b). For example, the P300 elicited by background tones decreases in amplitude and increases in latency as primary task difficulty is increased (Isreal et al., 1980b; Kramer et al., 1983). These findings have been interpreted as reflecting variation in the amount of attentional resources engaged by the primary task relative to a secondary task (Fournier et al., 1999; Gopher and Donchin, 1986; Kok, 2001; Wickens et al., 1983). Such ERP effects occur irrespective of changes in motor requirements (Isreal et al., 1980a; Makeig et al., 2004). Thus, mental activity from task situations that challenge attention resource capacities rather than response activities appear to govern P300 measures of cognitive workload.

Although these laboratory outcomes are encouraging, practical workload monitoring tools need to function well under realistic and immersive stimulus conditions. ERP studies have explored workload changes while subjects perform ecologically valid tasks in field settings such as flight simulators (Fowler, 1994; Kramer et al., 1987; Sirevaag et al., 1993) or advanced military monitoring and control settings (Humphrey and Kramer, 1994; St. John et al., 2003; Trejo et al., 1995a; Ullsperger et al., 2001). Brain-computer interface systems also have allowed communication via P300 components obtained while subjects wore a virtual reality helmet that simulated a car-driving game or virtual apartment (Bayliss, 2003; Bayliss and Ballard, 2000). Such interface measurements appear promising for clinical and other applications (Allison et al., 2007; Sellers and Donchin, 2006).

ERP workload studies typically present a sequence of low- and high-pitch tones in an oddball procedure while subjects perform a primary task that varies in difficulty (Kramer et al., 1995; Sirevaag et al., 1993; Trejo et al., 1995a). Although this method provides a reasonable workload monitoring capability, the typical two-stimulus oddball task with different auditory tones may not be optimal. Secondary cognitive tasks such as recognizing, discriminating, and disregarding irrelevant auditory probes produce at least three unwanted effects: (1) Such tasks could require mental resources that detract from primary task performance. (2) These tasks could impair effective workload assessment by altering cognitive load in intractable ways. (3) The tasks might annoy or fatigue the subject (Kramer and Spinks, 1991; Sirevaag et al., 1993; Trejo et al., 1995a; Ullsperger et al., 2001). Given these potential difficulties, a means of eliciting ERPs sensitive to variation in workload demands but without irrelevant probes and produced automatically could be practically quite useful.

1.2. Single-stimulus paradigm

The single-stimulus paradigm is a variant of the oddball task in which all nontarget tones are replaced with silence and produces ERPs comparable to those obtained with the two-stimulus oddball task (Cass and Polich, 1997; Pan et al., 2000; Polich et al., 1994; Polich and Margala, 1997) across auditory, olfactory, and visual modalities (Polich and Heine, 1996; Wetter et al., 2004). A key variable is the inter-target stimulus interval, with 8-20 s typically employed to maximize P300 activity (e.g., Strüber and Polich, 2002; Gonsalvez et al., 2007). The single-stimulus procedure can produce substantial P300 components even when stimuli are ignored (Mertens and Polich, 1997). Although the P300 from an oddball task is generally stable within a trial block (Polich and McIsaac, 1994; Ravden and Polich, 1999; Sellers and Donchin, 2006), component amplitudes from the single-stimulus task appear even more robust than from an oddball task as virtually little change or habituation effects have been found over target stimulus repetitions (cf. Pan et al., 2000; Wetter et al., 2004). Thus, the single-stimulus method may provide a more reliable and less intrusive tool for cognitive workload assessment than the conventional oddball task.

1.3. Present study

The present study used the single-stimulus paradigm to elicit ERPs to assess cognitive workload variation in video gaming environment. Subjects experienced with video games played a very immersive “first-person shooter” computer game in which workload can be directly manipulated by varying the number of enemy soldiers (cf. Green and Bavelier, 2003, 2006). These highly practiced subjects and engaging gamer task environment were employed to provide a challenging visual workload environment that could systematically maximize attentional requirements in order to ascertain sensitivity of the single-stimulus probe task. ERPs were elicited using an auditory single-stimulus paradigm while subjects either ignored or counted tones across difficulty conditions.

Application of the single-stimulus technique in this fashion has not yet been reported, so that the major goal of this study was therefore to evaluate the single-stimulus paradigm as a workload assessment tool by determining: (1) how task difficulty manipulations in a gaming environment affect early and late ERPs, and (2) whether ignore and counting tone procedures demonstrate differences across game difficulty levels. If successful for well practice subjects in an intensive and demanding workload environment, the single-stimulus approach may provide an appreciably sensitive method that is operationally easy to implement.

2. Method

2.1. Participants

A total of 14 young adult males were recruited via questionnaires distributed in psychology classes and by word-of-mouth (M=23.5, SD=5.1 years). All subjects played first-person shooter games for 10 or more hours per week (M=13.6, SD=7.7 hours/week) and had played frequently for many months (M=31.8, SD=22.5 months). All subjects reported that they slept between 6-8 hours the preceding night, were alert when assessed, and free of neurological or psychiatric disorders. Each provided informed written consent and received $20/hour for participation.

2.2. Procedure

Subjects were seated 75 cm in front of a computer monitor and encouraged to adjust the screen, chair, keyboard, key mappings, and mouse for comfort. Subjects played on a Dell Dimension 9100 with a 21 inch Sony ViewSonic PF775 CRT monitor and Harman Karden HK 195 speakers at a constant volume setting that attained a maximum of 75 dB SPL during the study. The game was Tom Clancy's Rainbow Six: Rogue Spear Black Thorn (Ubi Soft, 2001), using the “lone wolf” custom mission, the Alaska Depot map, and the default character at the “veteran” level (intermediate). In this mode, the character moves freely through a complex virtual environment and engages armed enemies trying to kill him.

Game difficulty was manipulated by varying the number of enemies in four different conditions: view game only (zero), easy (four), medium (ten), and hard (twenty), which were developed with similarly experienced pilot subjects. In the view only condition, the subject remained still and observed the game setup screen. In the other three conditions, the subject manipulated the game character to kill as many enemies as possible while avoiding being shot. If a character was shot, one or more wounds caused death depending on injury location. The character could only be wounded if shot by an enemy rather than from falling, traps, artillery, grenades, knives, vehicles, or random events; injuries could not be healed.

In addition to game stimuli, 1000 Hz tones (100 ms duration, 10 ms rise/fall time, 95 dB SPL) were presented over Koss speakers about 15 cm behind and above the subject's head. The inter-stimulus interval varied randomly between 6 to 30 s, with 30 stimuli occurring in each run. During the first four “ignoring” runs, the subject ignored these tones. During the last four “counting” runs, the subject silently counted tones and reported this count at the end of that run. Counting accuracy equaled the absolute value of the percent difference between the correct number of tones per run and the subject's reported count. Difficulty was counterbalanced across subjects within the ignoring and counting conditions, but the four counting runs always followed the four ignoring runs.

If the character was killed or killed all enemies, neuroelectric recording and tone presentation were paused. The experimenter recorded game performance measures, allowed the subject to restart the mission, and waited five seconds to allow the subject to become engaged in the game. If the character was killed within that period, game performance was not recorded, and both EEG activity and tones remained paused until the player survived more than five seconds. All runs continued until ten minutes of neuroelectric activity were recorded. Due to pauses, all runs except the view only condition took about twelve minutes.

2.3. Electrophysiological recordings

Electroencephalographic (EEG) activity was recorded with an electrode cap from 19 sites, Fz, Cz, Pz, FP1/2, F3/4, F7/8, C3/4, T7/8, P3/4, P7/8, O1/2, referenced to linked earlobes, with a forehead ground and impedances at 30 kOhms or less. Additional bipolar electrodes were placed at the outer left and right canthi and above and below the left eye to measure ocular activity. The band pass was 0.02-50 Hz (6 dB octave/slope), with the EEG digitized at 256 Hz. Data were recorded and analyzed using laboratory software developed locally. ERP averages were derived by extracting the period from 100 ms prior to 900 ms following tone onset, relative to a 100 ms prestimulus baseline. ERP averages were obtained such that single trials in which the amplitude exceeded ±150 μV on any channel were excluded from the average for that condition and subject. Mean area amplitude was obtained using an automated algorithm for the following latency windows: N1=80-150; P2=180-270; N2=220-340; P3=270-400; Slow Wave 1=400-650; Slow Wave 2=650-900 ms. Geisser–Greenhouse corrections and probability values are reported.

3. Results

3.1. Game performance

Several measures of game performance were evaluated (e.g., kills, wounds, shots fired, or mission success or failure). The kill-to-wound ratio, defined as the number of enemies killed divided by the number of times the character was shot, was the most effective measure of gaming performance. A high kill-to-wound ratio reflects superior game performance.

The mean (SD) kill-to-wound ratios for the easy, medium, and hard task for the ignore condition were 4.3 (3.4), 3.8 (4.0), 2.5 (1.0), and for the count condition 7.6 (5.3), 4.4 (2.7), 3.6 (2.3). The data were assessed by applying a 3 difficulty (easy, medium, hard) × 2 count (ignore vs. count) repeated measures analysis of variance. The kill-to-wound ratio decreased overall (5.9, 4.1, 3.0) as game difficulty increased (F(2,26)=5.02; p<0.03). The ignore condition demonstrated somewhat overall poorer performance than the counting (3.5 vs. 5.2) condition (F(1,13)=5.92; p=0.03), and no interaction was observed (p>0.16). Thus, game performance decreased as task difficulty increased, and increased with a secondary counting task that always occurred after the ignore condition trials.

3.2. Tone task performance

Tone counting error rate increased with task difficulty (view=1.0%, easy=8.3%, medium=10.0%, hard=15.5%). A one-factor repeated measures analysis of variance found that these differences were highly significant (F(3,39)=12.24, p<0.00002). Newman-Keuls comparison found that all pair-wise comparisons except easy vs. medium were significantly different from each other (p<0.05 in all cases). Counting performance was affected only by the hard condition.

3.3. Trial analysis

Figure 1 illustrates the grand averages from midline electrodes for ignore (left) and count (right) conditions with the four difficulty conditions overlapped. Preliminary assessment of the lateral electrodes did not reveal any hemispheric differences, so that only midline sites were evaluated. As the ERP data were obtained under atypical neuroelectric recording conditions that produced considerable movement, the number of acceptable trials for difficulty and count condition was assessed by manipulating the artifact rejection window. These assessments demonstrated that the ±150 μV criterion produced sufficient numbers of trials without significant artifact. A two-way (4 difficulty × 2 count) analysis of variance was used to evaluate the number of trials after rejection. Number of trials did not vary reliably overall for task difficulty level (F(3,39)=1.33, p>0.25). The ignore (M=19.6, SD=1.0) condition yielded slightly more overall trials compared to the count (M=17.2, SD=1.4) condition (F(1,13)=6.46, p<.025), but difficulty and count condition did not interact (F(3,39)=1.01, p>0.35). Thus, the number of trials for the ERP averages was sufficient and stable across task conditions (cf. Cohen and Polich, 1997).

Figure 1.

Grand average waveforms from the view, easy, medium, and hard task conditions for each of the midline electrodes (N=14).

3.4. ERP data

Figure 2 illustrates the mean amplitude area for the six latency windows defined above from each midline electrode for the ignore and count conditions as a function of task difficulty level. A three-factor repeated measures analysis of variance was applied to the data from each component (4 difficulty × 2 count × 3 electrodes). Table 1 summarizes the statistical outcomes. In addition, planned comparisons using Scheffé's contrast procedure were conducted on the data from each component to assess task difficulty effects in detail. Comparisons included: view vs. the three difficulty levels, easy vs. medium, easy vs. hard, medium vs. hard.

Figure 2.

N1, P2, N2, P3, SW1, and SW2 mean amplitude as a function of the view, easy, medium, and hard task conditions from the Fz, Cz, and Pz electrodes.

Table 1.

Summary of the three-factor (4 difficulty × 2 count conditions × 3 electrodes) repeated measures analyses of variance applied to the mean amplitude (latency window in ms) for each component and electrode.

	N1 (80-150)	P2 (180-270)	N2 (220-340)	P3 (270-400)	SW1 (400-650)	SW2 (650-900)
Difficulty (3,39)	---	13.21***	29.44***	20.70***	2.44+	---
Count (1,13)	---	17.47**	14.30***	5.10*	---	---
Electrode (2,26)	28.21***	13.23**	3.31+	10.77**	36.47***	14.10 ***
Count × Diff (3,39)	2.54+	---	---	---	---	2.98+
Count × Elec (2,26)	---	---	---	---	6.30*	9.90 ***
Diff × Elec (6,78)	5.60**	---	3.15*	10.18***	12.60***	2.10+
Count × Diff × Elec (6,78)	---	---	---	---	---	---

⁺p < .10

^*p < .05

^**p < .01

^***p < .001

N1 component

The N1 generally was less negative as game difficulty increased—an outcome that was more pronounced for the ignore compared to count condition. N1 area was most negative at Cz and least prominent over Pz.

P2 component

Increasing game difficulty and presenting a secondary counting task reduced P2 amplitude. In the counting condition, P2 amplitude was slightly larger during the medium difficulty condition than the other two game playing conditions. P2 area was maximal over Cz, and larger over Fz than Pz except in the view only condition. Planned comparisons yielded a difference between medium vs. hard (F(1,13)=4.6; p<.06) and among view vs. all gaming conditions (F(1,13)=22.3; p<.0005).

N2 component

As difficult increased and for the count conditions, N2 amplitude generally diminished. The N2 was largest over site Cz. These effects appeared to reflect activity from the surrounding P2 and P3 components. Planned comparisons found a difference between medium vs. hard (F(1,13)=8.8, p<.015) and between view vs. all gaming conditions (F(1,13)=40.2, p<.00003).

P3 component

As game difficulty increased and for the counting task P3 amplitude decreased and was reduced for all game difficulty conditions for both the ignore and count task procedures. The planned comparison between view vs. all gaming conditions was significant (F(1,13)=34.1, p<.00006). P3 amplitude was generally was largest over Pz and smallest over Fz.

Early slow wave component (SW1)

Although the early slow wave also declined in amplitude with increasing game difficulty and the addition of a counting task, neither of these effects was reliable. During the count conditions, slow wave activity was more negative over site Fz and during the hard difficulty setting.

Late slow wave component (SW2)

Late slow wave amplitude was most negative over Fz and most positive over Pz in the count condition only. No other effects were obtained.

3.5. Association between task performance and ERP outcomes

The relationship between task performance and ERP measures were assessed by correlating the mean kill-to-wound ratio and count task performance with mean component amplitudes (using Cz for N1, P2, N2 and Pz for P3, SW1, SW2). No meaningful association between kill-to-wound ratio and amplitude for either the ignore or count conditions was found, primarily because of the limited amplitude spread for the easy, medium, and hard conditions. However, mean count performance error rate was negatively if marginally correlated with component amplitude for the P2, N2, P3 and SW1 components, r = −0.88 to −0.92, p<0.15, for the view, easy, medium, and hard conditions. Thus, increased errors in tone-counting performance were associated with smaller component amplitudes.

4. Discussion

4.1. Task performance

Behavioral game performance and counting accuracy indicated that increases in game difficulty affected workload. As the single-stimulus tones were louder than game events, failure to count tones effectively when engaging more enemies likely occurred because of increased workload. The counterintuitive finding of improved game performance during counting conditions may have resulted from task practice, and demonstrates that the counting task did not substantially distract subjects. Practice, distraction, fatigue, order effects[A1], and other factors should be assessed systematically in future work.

4.2 ERP results

N1 exhibited a fronto-central distribution, whereas the P3 and later components were larger over parietal sites (Kutas and Dale, 1997; Polich, 2004). The spatial distribution did not change substantially as workload increased, whether by increases in primary task difficulty or the addition of a secondary tone counting task (Mertens and Polich, 1997; Zenker and Barajas, 1999). This outcome implies that ERP workload assessment yields a relatively general rather than component-specific measure.

Most ERP components became smaller as gaming workload increased (Humphrey and Kramer, 1994; Kramer et al., 1995; Mertens and Polich, 1997; Ullsperger et al., 2001; Watter et al., 2001). This effect was most prominent for the P2, N2, and P3 and when comparing the view condition to the three gaming conditions. These outcomes were statistically strong and similar, although it is unclear if multiple ERP components will assist in identifying variations in level of mental work (cf. Fournier et al., 1999; Smith et al., 2001; St. John et al., 2003). ERP measures of primary task difficulty are most apparent when tones were ignored, which indicates that workload monitoring devices based on single stimulus tones should not require that subjects shift resources away from primary task performance to count tones. This finding suggests for the first time that the single-stimulus procedure could be used to develop an objective workload assessment tool that is likely less distracting and psychologically invasive than conventional approaches (Berka et al., 2004; Lal and Craig, 2001; Prinzel et al., 2003; Smith et al., 2001; St. John et al., 2003; Strayer et al., 2006; Trejo et al., 1995a). With appropriate analysis methods it may even be possible to obtain measures from single-trials that could provide close to real-time index of cognitive workload.

4.3. Noise and artifact rejection

Subjects produced greater movement artifact when gaming than found in conventional EEG studies. Approximately one third of all ERP trials were rejected despite a rejection threshold of ±150 μV. However, the mean number of accepted trials in each condition was sufficient to produce stable average ERPs (Cohen and Polich, 1997). These findings are consistent with earlier studies demonstrating that EEG measures index workload even though subjects are engaged in a realistic game or simulation (Sirevaag et al., 1993; Smith et al., 2001; Trejo et al., 1995a). Hence, ERPs can be recorded and analyzed effectively despite greater motion artifact, distraction, and external noise than conventional laboratory ERP studies. P300 activity has been recorded and utilized in real-time under more adverse field conditions using conventional simple signal processing and artifact rejection (Allison and Moore, 2004; Sellers and Donchin, 2006). Thus, computationally sophisticated tools may not be needed.

4.4. Implications

The use of video games as a method to engage attentional mechanisms is not conceptually different from standard laboratory tasks such that engage different cognitive operations (e.g., working memory, visual tracking, etc.). Theoretical as well as practical implications, however, may be considered with respect to using primary tasks that can be termed “violent” rather than cognitively challenging but non-aggressive laboratory procedures (Green and Bavelier, 2003, 2006). As the present data suggest the personal involvement of such tasks when played by experienced gamers is striking. Whether the attentional engagement for “first-person shooter” games depicting humans is different from similar task demands involving target balloons is unknown. Evaluation of the neurocognitive consequences of violent stimulus conditions compared to task similar but non-violet stimulus conditions would be revealing.

Indeed, this particular violet video task environment may engage much stronger affective neural activity than more mundane laboratory based counterparts. The present study was designed to provide an initial foundation toward addressing these issues by evaluating ignore/count conditions, a control view compared to increasing levels of task difficulty, and characterization of all major ERP components. Application of similar methods that evaluate different aspects of the remarkably growing video culture should prove informative for understanding affective as well as cognitive workload engagement.

4.4. Summary

The auditory single-stimulus paradigm and contemporary first-person shooter games are effective means to assay cognitive workload. Changes in both primary and secondary task demands are apparent in auditory single-stimulus ERPs elicited during noisy real-world conditions. ERP differences are most apparent when comparing view only to playing conditions, and relatively small when comparing different game difficulties to each other. The approach described here may be adapted to automated real-time workload assessment systems.