Multimodal Virtual Reality-Based Assessment of Adult ADHD: A Feasibility Study in Healthy Subjects

Abstract

Neuropsychological assessments are often surprisingly inaccurate in mapping clinically-reported attention-deficit hyperactivity disorder (ADHD) symptoms, presumably due to their low ecological validity. Virtual reality (VR) might offer a potential solution for this problem, given its capability to generate standardized and yet highly realistic virtual environments. As the first adaptation of existing virtual classroom scenarios to an adult population, we developed a Virtual Seminar Room (VSR) for multimodal characterization of ADHD symptoms. To test its feasibility, N = 35 healthy participants were immersed into the VSR via a head-mounted display and carried out a VR-embedded continuous performance task (CPT) under varying levels of distractions in two experimental blocks (24 min each). CPT performance, electroencephalography (EEG) measures, and head movements (actigraphy) were simultaneously recorded and analyzed offline. Although CPT performance remained constant throughout the task, head movements increased significantly from Block 1 to Block 2. In addition, EEG theta (4–7 Hz) and beta (13–30 Hz) power was higher during Block 1 than Block 2, and during distractor-present than distractor-absent phases. Moreover, P300 amplitudes were higher during Block 1 than Block 2, and P300 latencies were prolonged in distractor-absent compared with distractor-present phases. Although the paradigm awaits further improvements, this study confirms the general feasibility of the VSR and provides a first step toward a multimodal, ecologically valid, and reliable VR-based adult ADHD assessment.

Attention-deficit hyperactivity disorder (ADHD) is a childhood-onset developmental disorder that manifests in symptoms of inattention, impulsivity, and/or hyperactivity (American Psychiatric Association, 2013). While symptoms of impulsivity and hyperactivity often diminish with age, inattention symptoms frequently persist across the whole lifespan (Francx et al., 2015; Franke et al., 2018; Willcutt et al., 2012). Therefore, ADHD is not only a disease of childhood and adolescence but often also of adulthood. On a neuropsychological level, adults with ADHD show deficits in a variety of cognitive domains, including sustained attention, interference control, behavioral inhibition, and perceptual speed (Chamorro et al., 2021; Hervey et al., 2004; Woods et al., 2002). Among the neuropsychological tests most commonly employed, is the continuous performance task (CPT; Rosvold et al., 1956). In this task, participants are presented with a series of stimuli and instructed to press a response key as soon as a certain, infrequent target stimulus appears and to suppress responses to any other, nontarget stimuli. For reaching optimal task performance, participants, thus, need to concomitantly sustain their attention and control their impulsive behavior throughout the task, which is why the CPT theoretically appears well-suited for assessing inattention and impulsivity.

Although the CPT is often employed in the assessment of ADHD, it has been of surprisingly limited diagnostic utility so far. In fact, although numerous variants of the CPT have been developed, with modifications in form, number, and frequency of stimuli, correlations between clinically reported ADHD symptoms and CPT performance are typically only low to moderate (Barkley, 1991; Lange et al., 2014). Likewise, although at group level some CPT differences between ADHD patients and healthy controls have been observed, a precise, CPT-based single-subject classification between ADHD patients and healthy controls is not yet possible (Barkley, 2019; Lange et al., 2014).

So far, the reasons for the CPT’s low diagnostic utility are not sufficiently understood. However, several causes are conceivable. First, ADHD is a highly heterogeneous disorder. The pattern and severity of cognitive deficits differ greatly between patients, with some individuals even scoring in the normal range on neuropsychological tasks, therefore not showing any impairment at all (Mostert et al., 2015; Nigg et al., 2005; Willcutt et al., 2005). Second, most CPT evaluations focus on markers of inattention (e.g., omission errors) and impulsivity (e.g., commission errors), while markers of hyperactivity are usually not evaluated. A CPT-based evaluation of this ADHD core symptom, however, appears possible by acquiring additional levels of analysis, for example, recording the participant’s motor activity during the CPT. Hall et al. (2016), for instance, demonstrated that by combining CPT performances with actigraphy, classification rates can be improved. And third, ecological validity of most CPT implementations appears to be rather low. Given the need for a highly standardized and reproducible test environment, most existing CPT implementations confine themselves to the presentation of simple, two-dimensional stimuli (e.g., letters or numbers) via computer screens. Such test implementations, however, raise the question, as to how far such a stimulus presentation may reliably mimic everyday life challenges, where environments are substantially more complex and the individual is surrounded by various distracting external stimuli (Varao-Sousa et al., 2018).

A solution for creating more reality-close test situations may be offered by virtual reality (VR) technology. Through creating three-dimensional (3D), immersive, and interactive virtual environments which allow to reliably mimic everyday life demands, ecological validity can be increased while still maintaining a high level of standardization (Parsons, 2015).

Regarding ADHD assessment during childhood and adolescence, two similar, but independently-developed virtual test environments have been investigated over the last years: the Virtual Classroom by Rizzo et al. (2006) and the AULA Nesplora by Iriarte et al. (2016). In the Virtual Classroom, children with ADHD are immersed into a virtual environment that resembles an ordinary classroom. Sitting at a desk surrounded by virtual classmates, the children are instructed to follow a classical visual CPT that is presented on the blackboard. To enhance reality closeness and incorporate a measure of distractibility and external interference control, different visual, auditory, and audio-visual distractors inside the virtual environment (e.g., a paper-plane flying through the room) can be presented during the task (Parsons & Rizzo, 2019). The design of the AULA Nesplora is similar, except that it contains both visual and auditory CPT stimuli (Iriarte et al., 2016). Both virtual test environments have been shown to differentiate between ADHD children/adolescents and healthy controls, with ADHD patients committing more overall errors in the CPT (Areces et al., 2018; Mühlberger et al., 2016; Neguț et al., 2017; Parsons et al., 2007; Rizzo et al., 2006) and displaying a larger amount of head- and overall body-movements during task completion (Areces et al., 2018; Parsons et al., 2007; Rizzo et al., 2006). In the Virtual Classroom, ADHD patients were additionally more affected by the insertion of distractors than healthy controls (Neguț et al., 2017; Parsons et al., 2007; Rizzo et al., 2006). Moreover, Adams et al. (2009) found a higher classification rate for discriminating ADHD patients from healthy controls if the classifier was trained on a VR-based CPT compared with a traditional CPT.

Although the two virtual test environments have demonstrated their potential utility in assessing ADHD during childhood and adolescence, no similar VR scenarios have yet been developed for adult ADHD patients. Moreover, besides behavioral assessments and actigraphy analyses, no other variables of interest have been investigated yet. To gain further insights into possible neuromarkers of ADHD, it would, however, be beneficial to additionally examine task-dependent brain activity.

Regarding electroencephalography (EEG), one oscillation of interest might, for instance, be the theta rhythm (4–8 Hz), which has been reported to be abnormally elevated in ADHD patients (see, for example, Adamou et al., 2020). More specifically, it has been suggested that the increased theta power in ADHD children and adolescents declines with age but remains enhanced during adulthood (Bresnahan & Barry, 2002; Koehler et al., 2009; Picken et al., 2020).

Another interesting EEG parameter is the theta–beta ratio (TBR), which reflects the ratio between absolute theta power and absolute beta power (12–40 Hz) and has been associated with attentional control (e.g., Angelidis et al., 2016). Although for several years the TBR was considered a robust neuromarker for ADHD (see, for example, Arns et al., 2013; Barry et al., 2003), more recent studies found only low diagnostic utility (e.g., Loo & Makeig, 2012) and qualified the TBR-hypotheses: Although TBR differences between children with ADHD and healthy controls appear to exist (Monastra et al., 2001; Snyder & Hall, 2006; Zhang et al., 2017), a significant TBR difference between adult ADHD patients and healthy controls could not be consistently found (Kiiski et al., 2020; Saad et al., 2015; van Dijk et al., 2020).

A third EEG parameter of interest relating to event-related potential (ERP) analyses is the P300 component, a positive voltage deflection ~300 ms after the target stimulus, which has been associated with stimulus evaluation (Sutton et al., 1965). Evidence from numerous studies suggests reduced amplitude (Grane et al., 2016; Hasler et al., 2016; Marquardt et al., 2018; Prox et al., 2007; Szuromi et al., 2011; Wiersema et al., 2006; Woltering et al., 2013) and prolonged latency (Idiazábal et al., 2002; Lazzaro et al., 2001; Tsai et al., 2012; Yamamuro et al., 2016) of this ERP component in ADHD patients compared with healthy controls.

The aim of the present, preregistered feasibility study was to complement existing VR research by undertaking a first step toward a VR-assisted, ecologically valid, and multimodal assessment procedure for adult ADHD patients. As a first step, we developed a new Virtual Seminar Room (VSR) scenario that resembles the already existing virtual classroom paradigms but is specifically tailored to adults. Moreover, our VSR not only enables CPT performance and actigraphy analyses but also ecological momentary assessment and EEG analyses. To demonstrate the general feasibility of our newly developed scenario, we applied our VSR to a sample of N = 35 healthy adults. Our main objectives were, first, to ensure that the VR scenario is feasible and does not induce discomfort in participants (see, for example, Barrett, 2004), and second, to test whether the simultaneous assessment of the different measures in VR is possible. Here, our main focus was on the combination of VR and mobile EEG, considering that EEG signals are easily distorted by head movements or pressure on the electrodes (Tauscher et al., 2019). In the VSR, both of those confounders are difficult to avoid. On one hand, a head-mounted display (HMD) on top of an EEG cap may induce strain on the electrodes, which might cause artifacts interfering with EEG signals. On the other hand, an immersive VR experience can only be created if participants can freely move their heads and look around, which may lead to an increased amount of motion artifacts.

Therefore, to test whether our setup allows us to derive plausible data, we analyzed participants’ CPT performance, EEG data, and head actigraphy over time and during distractor-present and distractor-absent task phases. Regarding CPT performance over time, previous VR classroom studies did not find a performance drop in healthy participants (see, for example, Bioulac et al., 2012). However, in the current VSR paradigm, the CPT blocks are substantially longer, and therefore we expect to observe a similar increase in error rates over time, as observed in traditional computer-based CPTs (Ballard, 1996a; Grier et al., 2003). Regarding the influence of distractions on CPT performance, previous VR classroom studies yielded mixed results. Although Parsons et al. (2007) found distractor-induced increases in error rates in healthy controls, Neguț et al. (2017) did not. Therefore, considering the length of our task and the comparatively high number of distractor-present and distractor-absent phases, we expect to see a distractor-induced performance decline in our present sample. Regarding ERP analyses, we expect to see a target P300 as in previous CPT studies (Fallgatter et al., 2000; Kirmizi-Alsan et al., 2006). Moreover, regarding TBR analyses, we expect an increased TBR over time which has been attributed to mind-wandering in the past (van Son et al., 2018). With regards to head actigraphy, we hypothesize that, similar to previous VR classroom studies, head movements will increase over time (Mühlberger et al., 2016) and in distractor phases (DP) compared with non-distractor phases (NDP; Parsons et al., 2007).

Method

Participants

Thirty-five healthy volunteers (M_age = 23.43; SD = 2.87; 14 males) were recruited for the study via mailing lists, direct advertisements, and social media. Eligibility criteria were normal or corrected-to-normal vision, no history of severe psychiatric or neurological disease and sufficient knowledge of the German language. Participants filled in a demographic questionnaire in which they had to inform the experimenter about current medication and whether they received any neurological, psychiatric or psychotherapeutic treatment. All participants gave written informed consent and received an expense allowance of 20 € for their participation. The study was approved by the University of Bonn’s medical ethics committee (protocol number: 011/20) and preregistered at the German Clinical Trials Register (https://www.drks.de/, Trial-ID: DRKS00021495).

General Procedure

The experiment lasted approximately 2 hr and was conducted in the VR laboratory of the University Hospital of Bonn. Upon arrival, participants were first informed about the study procedure and then signed the consent sheet. Next, they filled in three digital questionnaires via a computer, using the online survey tool SoSci-Survey (https://www.soscisurvey.de/). The three questionnaires administered were a demographical questionnaire, the Scale for the Evaluation of Attention Deficits (revised version, SEA-R, Volz-Sidiropoulou et al., 2007) and the Scale of Impulsive Behavior 8 (I-8, Kovaleva et al., 2012). After these questionnaires were filled in, the participants were prepared for the EEG measurement. Next, they became equipped with a HMD and immersed into the VSR, in which they first underwent a 60 s familiarization phase and then the actual CPT. In total, the CPT lasted ~48 min and took place directly within the VSR (details in “Continuous Performance Task” section). After the CPT was finished, the participants remained in the VSR to document their momentary level of cybersickness. To this end, they completed a subset of the Virtual Reality Sickness Questionnaire (VRSQ, Kim et al., 2018) by means of a VR-embedded gesture-based user interface (UI, see “Experience Sampling” section). Finally, the participants left the virtual environment and completed a recognition test (see “Recognition test” section) and an expense allowance sheet.

Apparatus and Virtual Environment Implementation

The experimental apparatus and VSR are displayed in Figure 1. Participants sat at a 1 m × 1 m table (cf. Figure 1A) within a 3.70 m × 2.65 m VR-play area (cf. Figure 1B). The VSR was presented via the HMD HTC VIVE Pro (HTC Corporation, Taoyuan City, Taiwan). This HMD has a 110-degree field of view, 90 Hz screen refresh rate and 1,440 × 1,600 per eye image resolution. The VSR was self-assembled under Unity 3D 2019.1.10f1 (Unity Technologies, San Francisco, CA, USA) and C#. When immersed into the VSR, participants found themselves sitting at a 1 m × 1 m virtual table, whose position matched the position of the 1 m × 1 m table in the real world. The virtual table was located in the back of the VSR, so that the participants had a good overview over the entire VSR. The VSR contained the typical furniture found in a seminar room, including a canvas right at the front wall of the VSR. Moreover, the VSR contained virtual classmates that performed unobtrusive idle movements during NDP and, if applicable, more complex actions during DP (details in “Continuous Performance Task” section). The 3D objects, sounds, and animations used for implementing the VSR were obtained from different commercial and non-commercial asset sources (i.e. Mixamo, Unity Asset Store, Renderpeople).

Figure 1.

The Virtual Seminar Room.

(A) Real-world third-person perspective and (B) first person perspective in the virtual environment. Participants were immersed into the Virtual Seminar Room (VSR), in which the continuous performance task (CPT) was presented at the canvas in front of the room.

Both the physical and virtual environments were spatially mapped by positional tracking, such that whenever the participants changed their head position in the real world, the HMD position in the virtual world adjusted accordingly. Using the Leap Motion system (Leap Motion Incorporation, San Francisco, CA, USA) together with a Unity SDK (https://developer.leapmotion.com/unity; accessed 07.01.21), the participants’ biological hand movements were real-time tracked and translated to two virtual hands shown in the VSR. The 3D hand models used for that were obtained from the “Leap Motion Realistic Hands” collection (downloadable over Unity’s asset store) and represented white-colored, average-sized human hands. The virtual hands were animated in such a way, that whenever the participants moved one of their biological hands, the respecting virtual hand moved correspondingly and without noticeable delay at the respecting position in virtual space. The virtual hands were used to amplify the level of embodiment and to enable gesture-based experience sampling (see “Experience Sampling” section).

Continuous Performance Task

The implementation of the CPT is illustrated in Figure 2. Directly implemented into the VSR, the CPT was realized by a series of single letters that were iteratively presented at the middle of the virtual canvas. As soon as a target letter sequence appeared, participants had to press the spacebar on a keyboard in front of them, following the second of the two letters in the sequence, while for any other letter sequence they had to withhold any button presses. The two target letter sequences defined were “A—K” and “H—F.” Whereas the sequence “A—K” was derived from other VR CPT studies (Mühlberger et al., 2016; Neguț et al., 2017), the sequence “H—F” was added to further increase task difficulty. Each letter was shown for 100 ms, with an interstimulus interval of 1900 ms (Mühlberger et al., 2016). After a practice run of 20 trials, the actual CPT began, which was split into two blocks (Block 1, Block 2) with a duration of 24 min each. In each block, 360 letter pairs were presented, out of which 108 (~30%) letter pairs represented a target sequence and 252 a non-target sequence (~70%), a ratio similar to the one used by Neguț et al. (2017). To increase task difficulty, the non-target pairs entailed 126 pseudo-target sequences, in which the first letter was either an “A” or “H,” but the second was not “K” or “F,” or the second letter was a “K” or “F,” but the first was not “A” or “H.”

Figure 2.

Experimental Design.

(A) Time course of the experiment. After being acquainted with the general procedure of the experiment and continuous performance test (CPT), the participants had to perform two CPT blocks (24 min each) and to undergo an experience sampling (ES) after each block. While the two CPT blocks were running, distracting events were concomitantly happening in the Virtual Seminar Room (VSR) during distractor phases (DP), but not during non-distractor phases (NDP). Within each CPT block, four DP and four NDP were alternatingly run, whereby each of these phases lasted 3 min. (B) Implementation of the CPT. Participants performed the CPT with an interstimulus interval of 1900 ms and a stimulus interval of 100 ms. Upon each target letter sequence (“A—K” or “H—F”), participants had to react with a spacebar press, while for any other letter sequence they had to withhold any button presses. (C) DP and NDP. During DP, distracting events were presented every 45 s in the VSR. 15 of these distractors were solely visual (e.g., a paper plane), 15 solely auditory (e.g., phone ringing) and 15 audiovisual mixed (e.g., an ambulance driving by).

While the two CPT blocks were running, intermittently distracting events were played in the VSR (cf. Figure 2). More specifically, within each CPT block, four DP and four NDP were alternatingly run, whereby each of these phases lasted 3 min. Whereas in a DP, different distracting events occurred every 45 s, no distractors were played during an NDP. Among the 32 distractors presented in total, 10 were solely visual (e.g., a classmate waving), 11 solely auditory (e.g., a dog barking), and 11 audiovisual (e.g., an ambulance driving by; for a complete list of all distracting events presented, see SM1). Although the distractor order was completely randomized, the phase order was counterbalanced across participants, in that for even participant numbers, the experiment started with a DP and for odd numbers, it started with an NDP.

For assessing CPT performance, three parameters of interest were extracted for each participant: the rate of omission errors (i.e., the percentage of nonresponses to target stimuli), the rate of commission errors (i.e., the percentage of responses to nontarget stimuli) and reaction time variability (RTV), which was defined as the standard deviation of reaction times toward correct hit trials divided by the mean reaction time (Kofler et al., 2013; Levy et al., 2018). Omission error rates are considered a measure of inattention, whereas commission error rates are thought to reflect impulsivity (Nichols & Waschbusch, 2004). RTV is considered a measure of vigilance, as lapses in attention lead to temporary slowing of responses, resulting in overall more variable reaction times (Levy et al., 2018).

Experience Sampling

Assessment of the participants’ subjective performances was carried out by a gesture-controlled UI (cf. Figure 3). After each block, this UI appeared as a VR-embedded, semi-transparent overlay in front of the participants. The UI iteratively surveyed the participants about three typical ADHD symptoms: inattention (“I had difficulty concentrating during this block.”), impulsivity (“I often had to stop myself from giving a wrong answer.”), and hyperactivity (“I moved a lot during this block.”). For each statement, the participants had to indicate their momentary level of agreement on a 7-point Likert-type scale, ranging from −3 (“totally disagree”) to +3 (“totally agree”). The VRSQ, which was assessed at the end of the VR experiment, was also presented via this UI.

Figure 3.

Experience Sampling.

(A) For an immediate assessment of the participants’ experiences during continuous performance task (CPT) performances, a virtual user interface showed up after each CPT block and surveyed the participants about their momentary subjective levels of inattention, impulsivity and hyperactivity, (B) real-world third-person perspective, and (C) real-world first person view.

Recognition Test

To assess the extent to which the participants noticed the presented distractors during the CPT, a recognition test was administered at the end of the experiment. The recognition test surveyed the participants about 64 distracting events that might potentially have happened during the CPT. For each of these potential events at issue, participants were presented a “reminder” picture and/or sound file of the respective event and were asked whether they recognized the event or not (e.g., “Did you notice that this person yawned?”). To control for false-positive answers, only 32 of the 64 suggested events represented an event that actually happened. For the statistical analysis, recognition sensitivity (d’) was separately calculated for each participant. To adjust for extreme values (i.e., hit or false alarm rate of 0 or 1), the loglinear approach was used (Hautus, 1995; Stanislaw & Todorov, 1999).

EEG Recording and Analyses

EEG was acquired via a wireless EEG system (Smarting®, mBrainTrain®, Belgrade, Serbia). The electrode montage represented a subset of the 10–20 system and consisted of 24 Ag/AgCl sintered ring electrodes: Fp1, Fp2, AFz, F3, Fz, F4, T7, C3, Cz, C4, T8, CPz, P7, P3, Pz, P4, P8, POz, O1, O2, M1 and M2. The ground electrode (DRL) was placed at FPz, while FCz served as reference (CMS). The amplifier was attached to the back of the EEG cap (EASYCAP, Herrsching, Germany) and communicated wirelessly with the computer via Bluetooth. All impedances were kept below 10 kΩ. The EEG signal was recorded via Lab Streaming Layer (https://github.com/sccn/labstreaminglayer) with a 500 Hz sampling rate and 24-bit step-size resolution. Data analysis was performed using Matlab 2018b (The MathWorks Inc., Natick, MA, USA) and EEGLAB 2019 (Delorme & Makeig, 2004).

Pre-Processing and Data Cleaning

For offline analyses, EEG data were first low-pass filtered with a cut-off frequency of 40 Hz and high-pass filtered with a cut-off frequency of 1 Hz (Hamming windowed finite impulse response filter of order 1,650, transition bandwidth 1 Hz) and then detrended. No rereferencing was applied. Next, data were screened for noisy EEG channels. In four datasets, channel Fz had to be replaced via spherical interpolation using EEGLAB’s in-built function pop_interp (Perrin et al., 1989). Moreover, all datasets were screened for missing data segments due to Bluetooth connection losses. In three datasets, missing data segments ranging from 48 to 132 s were found. In these cases, the entire DP or NPD in which the respecting corrupted sequence occurred, was removed, before all further EEG analyses were performed. As a next step, all EEG datasets were cleaned from artifacts. To this end, the continuous EEG data were first epoched into 2-second time windows and nonstereotypic artifacts were removed by the built-in EEGLAB function pop_jointprob with a threshold of 1.7 standard deviations. Next, an independent component analysis (ICA) using pop_runica (extended version) was computed on the epoched EEG data and components containing stereotypical artifacts, like for example ocular, cardiac or muscle activity, were identified by visual inspection. The ICA demixing matrix was then applied to the original continuous dataset (1–40 Hz filtered) and the previously identified artifactual components were rejected before back-projecting them onto the source space.

Frequency Analyses

Time-frequency analyses focused on TBR differences between phases (DP vs. NDP) and blocks (Block 1 vs. Block 2). Therefore, the ICA-corrected continuous EEG data were first cut into four separate epoched subsets (one for each condition): One subset for DP segments from Block 1, another subset for NDP segments from Block 1, a third subset for DP segments from Block 2, and a fourth subset for NDP segments from Block 2. To investigate stimulus-independent changes in the frequency bands, epochs for each subset were obtained by cutting all belonging DP or NDP into as many non-overlapping 5 s segments as possible. Next, the following identical preprocessing and analysis steps were undertaken on every subset: First, all segments of the respecting subset were baseline corrected (0–5 s). Second, using EEGLAB functions, all segments containing obvious, nonstereotyped artifacts exceeding 2 standard deviations were rejected. On average, M = 83.75 segments (SD = 2.33) remained within each subset. Third, a time-frequency analysis on channel Fz was performed on each remaining segment using Matlab’s pspectrum function. Frequencies ranged between 0 and 35 Hz, while the frequency resolution amounted to 0.034 Hz. Fourth, all derived power spectra were averaged to obtain one mean power spectrum. Fifth, the mean theta (4–7 Hz) and beta (13–30 Hz) power of the respecting subset (condition) was derived by taking the average power across all frequency bins that fell into the respecting frequency range and laid within 0.5 to 4.5 s. Finally, TBR values for the statistical analyses were calculated by dividing the theta power values by the beta power values.

ERP Analyses

ERP analyses focused on differences in the target P300 between phases and blocks. Therefore, the ICA-corrected continuous EEG data were first low-pass filtered at 15 Hz (Hamming windowed FIR filter of order 440, transition bandwidth 3.75 Hz) and separated into DP and NDP. Here, each subset was derived by aggregating all available segments within each pertaining phase from −2,200 to +2,000 ms (4.2 s), relative to each available correctly identified target stimuli (i.e., each detected “K” that followed an “A,” respectively, each detected “F” that followed an “H”). Next, for each segmented subset, the same preprocessing and analysis steps were carried out: First, using EEGLAB functions, all derived segments were baseline corrected from −2,200 to −2,000 ms relative to target onset. Second, segments containing residual artifacts were identified and rejected using the pop_jointprob function with a threshold of 3 SDs. On average, this resulted in M = 5.81 (SD = 0.72) segments for each subset. Third, ERPs were computed by averaging the segments for channel CPz, since P300 activity regarding target detection is expected to be mainly elicited in centro-parietal regions (Duncan et al., 2009; Polich, 2007). Finally, for statistical analyses, the maximum P300 peak and its corresponding latency were identified for each participant within the time range of +200 to +500 ms relative to target onset (automatic detection). To compare blocks, all DP and NDP were allocated to the first or second block. For creating topographic maps, a grand average over all conditions was calculated.

Actigraphy Recording and Analyses

Actigraphy analyses focused on differences in head position shifts and head rotations between phase types and blocks. Both actigraphy parameters were inferred from the built-in positional tracking of the Vive system, by means of which the HMDs momentary positions and rotations during the experiment were each recorded with a ~90 Hz sampling rate and in 3D Euclidean space coordinates.

For later offline analyses, the actigraphy data were first down-sampled to ~10 Hz and then the Euclidean distance between each sample point (3D position or rotation vector) and its preceding sample point was separately calculated for the HMD position data and HMD rotation data. Next, to statistically compare the amount of head position shifts and rotations between conditions, the mean Euclidean distance in respect to head position shifts and head rotations was derived for each type of phase and each block.

Statistical Analyses

Eleven main dependent variables were in the focus of this study: commission error rates, omission error rates, RTVs, TBRs, P300 latencies and amplitudes, head position shifts, head rotations, self-rated inattention, self-rated impulsivity, and self-rated hyperactivity. Using graphical inspection and skewness values, all main dependent variables were first checked for normality before any further statistical analyses were conducted. If a variable was highly skewed, data transformation was applied. That is, commission error rates, omission error rates, and RTV were square-root transformed and head position shift and head rotation were log-transformed. After transformation, skewness of these variables was acceptable (between −1 and 1). Since analyses of variance (ANOVAs) are, however, considered to be sufficiently robust against normality violations (Blanca-Mena et al., 2017; Schmider et al., 2010), ANOVAs were applied also for these variables.

For each variable of interest, except for the self-rating variables, a separate 2 × 2 repeated-measures ANOVA with the within-factors “Block” (Block 1 vs. Block 2) and “Phase” (DP vs. NDP) was conducted. For reporting ANOVA effect sizes, partial eta squared (η_p²) was used. According to Cohen (1988), η_p² = .01 indicates a small effect, η_p² = .06 a medium effect and η_p² = .14 a large effect. For t-test effect sizes, Cohen’s d was used, whereby d = .20 indicates a small effect, d = .50 a medium effect and d = .80 a large effect (Cohen, 1988).

Moreover, to identify potential interrelations between objective and subjective measures of inattention, impulsivity, and activity, exploratory correlation analyses were conducted: First, all three CPT variables were correlated with recognition d’. Second, omission error rates were correlated with self-rated inattention, commission error rates with self-rated impulsivity, and head position shifts/head rotations with self-rated activity. Third, the inattention and impulsivity measures were correlated with each other. All correlations were tested for significance and Bonferroni–Holm correction was applied to correct for multiple comparisons.

Three participants were not included in the statistical analyses, two due to technical failures and one, because after the experiment they admitted having taken antidepressant medication which led to meeting an exclusion criterion for our healthy sample. Therefore, the final sample analyzed comprised n = 32 individuals (11 male, 21 female), aged between 19 and 29 years (M = 23.03, SD = 2.52). For all statistical analyses, Matlab R2018b was used and the α-level was set to .05.

Results

CPT Performance

On average, the total commission error rate amounted to M = 0.53% (SD = 0.58%) and the omission error rate to M = 2.89% (SD = 3.74%). The average reaction time (RT) amounted to M = 0.41 s (SD = 0.05 s), whereas the average RTV, in turn, was M = 0.23 (SD = 0.08). Pearson correlations between Block 1 and 2 performances yielded internal consistencies of r = .744 (p < .001) for omission errors, r = .733 (p < .001) for commission errors, r = .814 (p < .001) for RT and r = .507 (p = .003) for RTV. CPT descriptive statistics for each experimental condition can be found in Supplemental Table S2. ANOVAs did not yield any significant effects for any of the three CPT outcome parameters (cf. Figure 4, for statistical details, see Table 1).

Figure 4.

Continuous Performance Task (CPT) Results.

Note. (A) Percentage of commission errors, (B) percentage of omission errors, and (C) reaction time variability (RTV) in distractor phases (DP) and non-distractor phases (NDP) of Blocks 1 and 2. All barplots depict square root transformed data. Analyses of variance (ANOVAs) did not yield significant effects for any of the three parameters.

Table 1.

ANOVA Results of CPT Performances.

CPT parameter	Predictor	df	F	p	ηp2
Omission error rate	Block	1, 31	1.33	.257	.04
	Phase	1, 31	0.61	.441	.02
	Block × Phase	1, 31	1.39	.247	.04
Commission error rate	Block	1, 31	0.60	.446	.02
	Phase	1, 31	0.12	.735	.00
	Block × Phase	1, 31	2.06	.161	.06
RTV	Block	1, 31	0.41	.529	.01
	Phase	1, 31	0.51	.480	.02
	Block × Phase	1, 31	0.03	.871	.00

Note. CPT = continuous performance task; RTV = reaction time variability.

Electrophysiological Analyses

Frequency Analyses

Results of the time-frequency analyses are shown in Figure 5. In line with the literature (Ishihara & Yoshii, 1972) and across conditions, theta power prominently showed up over frontal-midline and occipital electrodes, whereas beta power was more broadly distributed over the whole scalp (cf. Figure 5B). Regarding TBRs (cf. Figure 5C), the ANOVA neither revealed a significant main effect of “Block”, F(1, 31) = 0.00, p = .960, η_p² = .00, nor of “Phase”, F(1, 31) = 0.47, p =.500, η_p² = .01, nor an interaction effect, F(1, 31) = 0.04, p = .850, η_p² = .00.

Figure 5.

Results of the Time-Frequency Analyses.

Note. (A) Time-frequency spectrum across conditions between 0 and 30 Hz at electrode Fz (grand average of 5 s segments). (B) Corresponding topographic maps for analyzed theta power (4–7 Hz, lower plot) and beta power (13–30 Hz, upper plot). Electrode Fz is white circled. (C) Theta-beta-ratio (TBR), (D) theta power, and (E) beta power distributions during the different continuous performance task blocks (Block 1 vs. Block 2) and phases (distractor phases (DP) versus non-distractor phases (NDP)). Barplots for theta and beta power depict square root transformed data.

As an exploratory follow-up analysis, theta and beta power at electrode Fz were also evaluated individually, using the same 2 × 2 repeated-measures ANOVA design. Beforehand, both variables were square-root transformed to reduce skewness to an acceptable level $(\le \pm$ 1). Regarding theta power (cf. Figure 5D), the ANOVA revealed significant main effects of “Block”, F(1, 31) = 22.51, p ≤ .001, η_p² = .42, and “Phase”, F(1, 31) = 9.89, p = .004, η_p² = .24. Whereas the effect of “Block” was due to a higher theta power in Block 2 than Block 1, the effect of “Phase” was due to a higher theta power under NDP than DP. No interaction effect was found by the ANOVA, F(1, 31) = 0.02, p = .885, η_p² = .00.

Concerning beta power (cf. Figure 5E), the ANOVA revealed the same pattern: Also here significant main effects of “Block”, F(1, 31) = 20.17, p ≤ .001, η_p² = .39, and of “Phase”, F(1, 31) = 13.28, p ≤ .001, η_p² = .30, were found, but no interaction effect, F(1, 31) = 0.73, p = .398, η_p² = .02. And again, the “Block” effect was due to a higher beta power in Block 2 than 1, whereas the “Phase” effect consisted in a higher beta power under NDP than DP. Frequency descriptive statistics for each experimental condition can be found in Supplemental Table S2.

ERP Analyses

One dataset was identified as an outlier and therefore excluded from further ERP analyses. Waveforms and topographies of the analyzed ERPs are depicted in Figure 6A. In line with the literature, the extracted ERPs showed the typical waveform and topography of a target P300 (e.g., Polich, 2007), with a maximum peak at around 330 to 347 ms over centro-parietal electrodes. The ANOVA on the target P300 amplitudes (Figure 6B, left panel) revealed a significant block effect, F(1, 30) = 4.71, p = .038, η² = .14, indicating that amplitudes were higher in the first compared with the second block. The ANOVA on the target P300 latencies (Figure 6B, right panel), in turn, revealed a significant phase effect, F(1, 30) = 5.15, p = .031, η² = .15, indicating prolonged latencies in NDP compared with DP. There were no other significant effects (for statistical details, see Table 2). P300 descriptive statistics for each experimental condition can be found in Supplemental Table S2.

Figure 6.

Results of the Target P300 Analyses.

(A) Waveform and topography of the target P300 at channel CPz (white marked electrode) across distractor phases (DP) and non-distractor phases (NDP) for each block. The red waveforms depict the target P300 DP and the blue waveforms the target P300 during NDP. Black dotted lines indicate the interval used for the statistical analyses and topography compilations. (B) Target P300 peak amplitudes and (C) corresponding latencies for both blocks and phases.

Table 2.

ANOVA Results of P300 Amplitudes and Latencies.

Parameter	Predictor	df	F	p	ηp2
P300 amplitude	Block	1, 30	4.71	.038	.14
	Phase	1, 30	1.04	.316	.03
	Block × Phase	1, 30	2.15	.153	.07
P300 latency	Block	1, 30	1.10	.303	.04
	Phase	1, 30	5.15	.031	.15
	Block × Phase	1, 30	0.01	.920	.00

Actigraphy Analyses

The ANOVA for head position shifts (Figure 7A) yielded a significant main effect of “Block”, F(1, 31) = 24.34, p < .001, η_p² = .44, but no main effect of “Phase”, F(1, 31) = 1.41, p = .244, η_p² = .04, and no significant interaction, F(1, 31) = 0.43, p = .518, η_p² = .01. The effect of “Block” revealed that stronger head position shifts were conducted during Block 2 than Block 1.

Figure 7.

Results of the Actigraphy Analyses.

(A) Head rotations and (B) head position shifts in distractor phases (DP) and non-distractor phases (NDP) of Blocks 1 and 2. Barplots depict log-transformed data. Analyses of variance (ANOVAs) for head position shifts and rotation yielded that participants conducted stronger head position shifts and head rotations during Block 2 than Block 1.

Head rotation findings were in line with these findings (cf. Figure 7B). The ANOVA yielded a significant main effect of “Block”, F(1, 31) = 9.14, p = .005, η_p² = .23, in that stronger head rotations were executed under Block 2 than Block 1. Likewise, the ANOVA did not reveal a main effect of “Phase”, F(1, 31) = 0.06, p = .813, η_p² = .00, nor a significant interaction, F(1, 31) = 0.07, p = .798, η_p² = .00. Actigraphy descriptive statistics for each experimental condition can be found in Supplemental Table S2.

Experience Sampling and Recognition Test

Self-rated levels of inattention, hyperactivity, and impulsivity during the two blocks are depicted in Figure 8. Self-reported inattention and hyperactivity were significantly higher in the second experimental block compared with the first experimental block, inattention: t(31) = −5.17, p < .001, d = −.91, hyperactivity: t(31) = −3.73, p < .001, d = −.66. There was no significant difference in self-reported impulsivity between the two experimental blocks, t(31) = −1.36, p = .184, d = −.24. Across participants, the mean cybersickness score was M = −0.37 (SD = 0.93), indicating that participants experienced little or no symptoms of discomfort in the VR environment. In the recognition test, d’ was on average M = 1.32 (SD = 0.58).

Figure 8.

Self-Ratings of Inattention, Impulsivity, and Hyperactivity.

Subjective inattention, impulsivity, and hyperactivity ratings for Blocks 1 and 2. T-tests yielded significantly higher subjective inattention and hyperactivity during Block 2 than Block 1.

Correlation Analyses

The correlation analyses revealed a significant positive correlation between head position shifts and subjective hyperactivity, r(30) = .63, p < .001. Moreover, an additional moderate negative correlation was found between recognition test score and RTV, r(30) = −.38, p = .032, which, however, did not remain significant after Bonferroni–Holm correction (adjusted p = .310). All other correlations were between r = ±.40 and nonsignificant, even without correction. For the exact correlation results with both uncorrected and Bonferroni–Holm corrected p values, see Supplemental Table S3.

Discussion

In this feasibility study, we examined the viability of a VSR as a potential assessment tool for identifying and multimodally characterizing inattention, impulsivity, and hyperactivity symptoms in adult ADHD patients. As a first step toward such a tool, we immersed N = 35 healthy adults into our VSR and let them perform a CPT under varying levels of distractions. Although during distractor phases (DP), distracting events regularly occurred every 45 s, no distracting events occurred during non-distractor phases (NDP).

With regards to the general feasibility of the VSR, our study yielded promising results in terms of both tolerability and data plausibility. All included participants were able to undergo the whole experiment from start to finish, and no session had to be paused due to physical discomfort or any other reason. In fact, self-reported cybersickness scores indicated little to no discomfort during immersion. Furthermore, we succeeded to simultaneously record both behavioral and neurophysiological data. Our concerns about artifacts due to participants’ movements or due to the HMD were justified, but we were able to sufficiently clean up the EEG signal, in order to enable physiologically plausible ERP and frequency analyses.

For CPT outcome measures, we neither found differences between Block 1 and Block 2 nor between DP and NDP. That is, CPT performance neither declined over the course of the experiment nor was it significantly influenced by the occurrence of distracting events. Although the first null finding complies with Bioulac et al.’s (2012) virtual classroom study, which found that CPT performance deteriorated over time in children with ADHD, but not in healthy controls, the latter null finding converges with the two Virtual Classroom studies by Rizzo et al. (2006) and Neguț et al. (2017), which revealed that, unlike children with ADHD, healthy controls did not show performance differences between DP and NDP. Given that all these previous studies were conducted with children and varying task designs, such comparisons should, however, be made with caution. Although Bioulac et al.’s (2012) CPT, for instance, only lasted 14 min, our own CPT endured 48 min. Consequently, we expected fatigue-induced performance deteriorations in our healthy participants, too. A better explanation for the present null findings, therefore, might be that our CPT was not sufficiently sensitive for detecting small performance drops, possibly due to a ceiling effect: On average, participants committed only 6.24 omission errors (SD = 8.08) and 6.53 commission errors (SD = 7.03) over the whole 48 min of the task, and in at least one of the two blocks, 50% of participants made fewer than two commission errors and over two thirds (69%) made fewer than two omission errors. In sum, based on these results, we cannot unambiguously conclude that our CPT maps change in attention better than traditional, computer-based CPTs.

Regarding our TBR evaluations, we neither found any Block 1 versus Block 2 nor DP versus NDP differences. Thus, contrary to our expectations, we did not find evidence for attention-related TBR changes. One possible explanation could be that the TBR is not a sensitive marker of attentional control. In fact, the TBR has so far almost exclusively been studied as a potential discriminating feature between ADHD populations and healthy populations (for a review, see Arns et al., 2013), but only little as a general EEG measure of attention per se. Hence, although the few studies conducted indicate an association between TBR and attention (Angelidis et al., 2016; Putman et al., 2014; van Son et al., 2018), further confirmatory studies are necessary. An alternative explanation could be that participants’ attention levels simply stayed stable throughout the task. It is to be noted, however, that our relatively small sample of N = 35 might not have been sufficient to detect intra-individual TBR differences. Still, it can be observed that although the TBR itself remained stable, both beta power and theta power significantly increased from Block 1 to Block 2 and were also higher during NDP than DP. Both the long CPT duration and variation of distractor levels thus clearly induced oscillatory changes in the EEG.

Furthermore, it could be reasoned that the theta power increase from Block 1 to Block 2 was due to a drop in sustained attention over time. Given previous evidence for a positive association between frontal theta power and higher mental effort (Sauseng et al., 2006), this line of reasoning conflicts with our result of lower theta power during DP than NDP as we expected distractions to increase cognitive demands. Hence, if theta power increased as a function of mental effort, we would have expected higher theta power during DP than NDP instead. However, there is also evidence suggesting that in repetitive tasks with low difficulty, task-irrelevant stimuli can facilitate attention performance by increasing arousal and therefore counteracting task-induced fatigue (Olivers & Nieuwenhuis, 2005; Smucny et al., 2013; Zentall & Zentall, 1983). Considering that participants’ performance was overall very high in our study, thus indicating low task difficulty, it could be assumed that arousal levels were higher during DP than NDP, leading to the observed decline in theta power.

The increase in beta power from Block 1 to Block 2, in turn, complies with a study by Boksem et al. (2005), who also reported an increase over time in theta and beta power in a visual attention task. Although an increase in theta power is considered to reflect mental fatigue due to attentional demands, increasing beta power might reflect compensatory attentional efforts to counteract time-on task-related fatigue and maintain cognitive control (Boksem et al., 2005; see also Stoll et al., 2016). This interpretation also appears applicable to the present EEG results, especially if one also considers the present behavioral and subjective results: Although subjectively, participants clearly reported an attention decrease from Block 1 to Block 2, their CPT performance remained unaffected by this subjective attention decrease. That is, they were still able to compensate for their increasing mental fatigue. The effect of higher beta power during NDP than DP, in turn, could potentially reflect distraction-induced lapses in task engagement, since previous literature has associated task-related beta power increases with increasing task engagement and alertness (Coelli et al., 2015; Kamiński et al., 2012). Hence, it might be speculated that the distractors played during the DP temporarily interrupted the participants’ task engagement in the CPT.

As regards ERP analyses, we successfully extracted the expected topography and waveform of a target P300 with an averaged peak from 330 to 347 ms for phases and blocks. This confirms that not only frequency analyses but also ERP analyses can be reliably conducted with our VSR. As pertains statistics, we found a reduced P300 amplitude in Block 2 as compared with Block 1 and a prolonged latency in NDP as compared with DP. Previous studies have associated a reduced P300 amplitude and prolonged latency with mental fatigue and higher cognitive workload, for example, during driving simulation paradigms (Coleman et al., 2015; Zhao et al., 2012). Although our result of a reduced P300 amplitude over time is in line with these findings, indicating an increase in mental fatigue over time, the prolonged latency in NDP compared with DP is surprising, as we would have expected DP to be more cognitively demanding than NDP. Perhaps, however, this finding can be explained in a similar way as our finding of reduced theta power in DP: Due to the distracting events, arousal levels may have increased in DP, leading to a reduction in fatigue, which then resulted in shorter P300 latencies.

Regarding actigraphy and self-rating measures, we found three indications that participants increased their body activity from Block 1 to Block 2. Not only did participants self-report higher activity levels in Block 2 but they also conducted more head position shifts and head rotations during Block 2. Our assumption is that this increase in body activity can be attributed to increasing fatigue and impatience over the course of the CPT and, therefore, an increasing difficulty in sitting still. The current VSR scenario thus appears capable of inducing hyperactivity, and this effect should become even more pronounced, if the scenario will be applied to ADHD patients. Regarding the comparison between DP and NDP, no significant actigraphy differences were found. This corresponds to the Virtual Classroom study by Rizzo et al. (2006), who also did not find differences in head, arm, and leg activity between distractor and non-distractor conditions in healthy children. Parsons et al. (2007), on the other hand, reported higher means of body movement in distractor than in non-distractor conditions in healthy children, but this difference was not inference-statistically analyzed.

There were no correlations between SEA-R attention score and omission error rate, nor between I8 impulsivity score and commission error rate. A possible explanation for this repeatedly observed lack of convergent validity (Aichert et al., 2012; Cyders & Coskunpinar, 2011; Gomide Vasconcelos et al., 2014; Solanto et al., 2004; but see Asbjørnsen et al., 2010; Epstein et al., 2003) is that computerized tasks and self-report scales reflect different facets of behavior. Although the CPT objectively measures attention performances during a single experimental session in a specific, laboratory setting, self-report scales typically summarize subjective experiences over much longer periods of time and across a variety of situations (Barkley, 1991; Meyer et al., 2001; Slobodin & Davidovitch, 2019). Another reason might be that in the present study the CPT was not sensitive enough to detect small differences in inattention and impulsivity between participants, and therefore CPT parameters did not correlate with subjective scores. However, due to this lack of correlation, the results of the current study provide no evidence that the VSR can map individual attention differences in everyday life and is more ecologically valid than traditional CPTs.

Limitations and Future Directions

One important limitation of the present study is that the CPT was not particularly difficult for healthy participants. Consequently, ceiling effects may have resulted in an insufficient depiction of the true variance between participants’ individual attention and impulsivity capacities. That we did not find influences of time and varying distraction levels on CPT performances, as well as no correlations between CPT performances and subjective levels of inattention and impulsivity, should therefore not be over-interpreted. Instead, these null results might potentially just be attributable to variance restrictions, due to a too low task difficulty.

Low CPT error rates are also suboptimal for EEG analyses. Besides analyzing the target P300, it would be, for instance, also interesting to analyze error-related potentials, like the error-related negativity (ERN). The ERN negatively peaks between 50 and 100 ms following an error response and has repeatedly been found to be attenuated in amplitude and shortened in latency in ADHD patients compared with healthy controls (for a meta-analysis, see Geburek et al., 2013). For analyzing the ERN, the EEG signal, must, however, be aggregated across several trials, which is not possible, if the error rate is as low as in the present study.

Consequently, to ensure sufficiently high error rates and performance variance in both ADHD patients and healthy controls, it is crucial to increase the CPT’s difficulty for future studies. One way to achieve a higher difficulty would be to modify the CPT parameters, for instance, by increasing the task speed, lowering the ratio of targets to non-targets, or reducing stimulus salience (Ballard, 1996a, 1996b). Another possibility would be to increase the level of distraction. In the present study, the number of distractors per DP was relatively low, since distracting events were played only every 45 s. This led to long periods in which no distraction occurred. In addition, some distracting events turned out to be not salient enough. As revealed by our recognition task, 12 of the distractors (1 auditory, 6 visual, 5 audio-visual distractors) were not noticed by over 50% of our participants. Thus, for further studies, both distraction frequency and the salience of distractors might also be increased.

Another possibility for improvement of the present VSR implementation is to complement the existing measurement methods with additional ones. Further insights into the participant’s distractibility could, for instance, be gained by eye-tracking recordings, which would allow to track at which distractors participants look closely and which they ignore, especially given the observation of impaired oculomotor inhibition in ADHD (Chamorro et al., 2021).

Conclusion

This study set out to test the feasibility of a VR paradigm to investigate attention, impulsivity, and hyperactivity in a multi-modal assessment procedure. Our results confirm that it is possible to simultaneously analyze these symptoms by several neuropsychological, phenomenological, electrophysiological, and actigraphic measures. Although in our healthy participants, no CPT performance differences were observed, presumably due to ceiling effects, we found various time on tasks effects in respect to electrophysiological, phenomenological, and actigraphical measures. In the next step, to further prove the validity of our multimodal VSR, a sample of adult ADHD patients will be investigated.