Fluctuations of Attention During Self-paced Naturalistic Goal-Directed Behavior in Attention-Deficit/Hyperactivity Disorder

Abstract

Objective

Temporal fluctuations of attention detected with strictly controlled neuropsychological tests is an important objective behavioral marker for attention-deficit/hyperactivity disorder (ADHD). This study examined whether intraindividual variability in response latencies is also detectable in more realistic open-ended virtual contexts where the participants can freely interact with the surroundings when performing instructed everyday tasks from memory.

Method

Three ex-Gaussian parameters, μ, σ, and τ, were derived from response latencies in 2 tasks obtained from 2 datasets comprising 9- to 13-year-old children (72 with ADHD and 71 typically developing controls). In the Executive Performance in Everyday LIving (EPELI) task, participants performed instructed household chores in a virtual apartment. In the other task, a continuous performance test (CPT), was used to examine whether previous findings were replicated in this sample.

Results

Children with ADHD had shorter response latencies than controls in the EPELI task, while group differences in τ reflecting occasional sluggish responses depended on whether the trials were task-relevant (smaller τ in children with ADHD) or task-irrelevant (larger τ in children with ADHD). CPT results replicated previous observations of longer response latencies and larger τ in children with ADHD compared with control children. Intraindividual variability in the naturalistic EPELI task, however, explained more of the symptom variability than the CPT.

Conclusion

This study demonstrates that task context and stimulus relevance considerably influence how intraindividual variability in attention is manifested in children with ADHD. Virtual reality tasks provide a promising avenue for ecologically relevant quantification of this common cognitive deficit in neuropsychiatric disorders.

Plain language summary

Temporal fluctuation of cognitive functioning is a behavioral marker for attention-deficit/hyperactivity disorder (ADHD). This study examined data from 72 children with ADHD and 71 children without ADHD, to identify whether cognitive fluctuation can be detected in both a classical experimental task and a novel naturalistic virtual reality task. Results showed that while cognitive fluctuations were observed in both tasks, intra-individual variability was dependent on task context, with ADHD children responding faster in the naturalistic task than the experimental task. Importantly, temporal fluctuation of cognitive function from the naturalistic task explained more symptom variability than those from the experimental task. These results suggest that virtual reality tasks may help identify ecologically relevant markers for ADHD.

Various problems in attention and executive function are common in many neuropsychiatric disorders, but their objective measurement has been limited to strictly controlled, artificial tasks. Such tasks are very different from complex real-life situations in which symptoms critical to diagnosis occur. As objective markers for real-life cognitive problems that could support diagnostics are lacking, a clinician has to rely mostly on interviews and self-ratings that are imprecise and prone to subjective bias.¹ Here, we focused on a key symptom, abnormal fluctuation of cognitive processing, that is present in a wide variety of disorders, such as attention-deficit/hyperactivity disorder (ADHD),^2,3 schizophrenia,⁴ mood disorders,⁵ anxiety disorders,⁶ and neurodegenerative illnesses.⁷ Using a virtual reality simulation, we examined intraindividual variability in children with ADHD to obtain an objective measure of temporal fluctuations in cognitive processing in lifelike situations. In the everyday life of a child with ADHD, fluctuations of cognitive processing take place in complex, changing environments. For example, this could mean mind wandering or getting distracted during a school class or homework or digressing to irrelevant actions while performing daily routines (eg, preparing to go to school or completing morning activities).

Intraindividual variability in response latencies is typically observed as occasional unusually slow response times reflecting temporary shifting of attentional focus away from the main cognitive task. For instance, with the widely used continuous performance test (CPT), participants’ attentiveness is evaluated based on hundreds of button presses to simple stimuli (eg, an isolated stream of letters) during a 14-minute consecutive task presentation.^2,8,9As the intervals between the stimuli are fixed by the experimenter, the pace of the task is strictly externally determined (ie, externally paced task). Highly structured and simplified tasks such as the CPT have been developed to establish reliable measures that can be interpreted straightforwardly. However, the context in which such symptoms appear is often fundamentally important to determine whether or not the problems relate to a neuropsychiatric condition.¹⁰ For example, ADHD symptoms in everyday life encompass various closely related cognitive processes (eg, attention, memory, planning, inhibition, reward processing, executive function) operating together in open-ended naturalistic contexts filled with meaningful information. Manifestation of ADHD symptoms strongly depends on the features of the naturalistic context, for instance, whether the required actions are pleasurable or not¹¹ and which kind of temporal dynamics or cognitive demands the task in question involves.12, 13, 14 However, ADHD-related deviations in response latencies and fluctuations of cognitive processing in such complex conditions remain largely unclear.

In the present study, we employed a recently developed function-led virtual reality game that allows studying real-world goal-directed behavior under naturalistic conditions, tapping multiple cognitive processes that are considered to underlie ADHD symptoms.^15,16 In the game, Executive Performance in Everyday LIving (EPELI), participants perform everyday household chores from memory in a virtual home. Each scenario has a time limit, but the participant is free to navigate in the stimulus-rich surroundings and decide which actions to take to perform the set of tasks. In that sense, EPELI can be considered as a self-paced task, as opposed to the CPT. In EPELI, the interaction with the environment and moving around take place using a hand controller, making it possible to extract response latencies in a situation in which the participants pace the dynamics of their actions by themselves, not as simple responses to restricted external stimuli. The virtual apartment contains various attractive objects provoking irrelevant impulsive behaviors, while task-relevant behavior requires keeping the instructions in memory and focusing at executing the tasks.

Here, we examined response latencies in 2 datasets with EPELI and CPT data collected in children with ADHD and typically developing (TD) controls.^15,16 We focused on the late childhood (age range 9-12 years old) period, which is a critical time for the development of executive functions supporting independent goal-directed behavior and occurs at the same time when environmental demands start to increase.¹⁷ Our first hypothesis was that because EPELI is a self-paced task, participants with ADHD would show faster interacting with the environment in this task due to their hyperactive-impulsive behavioral characteristics. Our second hypothesis was that in EPELI the response latencies of children with ADHD would also be more variable compared with TD participants due to occasional fluctuations of cognitive processing even when the task is open-ended and naturalistic by nature. We also aimed to test how strongly the intraindividual variability measures in the two different tests that both successfully capture ADHD relate to each other and to interindividual variability in ADHD symptoms. In EPELI, we separately examined the latencies for the relevant (button presses to target objects) and irrelevant (button presses to nontarget objects) responses. Relevant responses were expected to mostly reflect attentiveness, and irrelevant responses were interpreted to mostly relate to hyperactive-impulsive behaviors.¹⁶

For response latencies in both tasks, we separated unusually slow responses from the reaction time distribution via ex-Gaussian modeling.¹⁸ In this approach, the mean (μ) and standard deviation (σ) are derived from the normally distributed component and decay parameter (τ), which is the tail of the distribution representing the slow response latencies, from the exponential component. Larger τ has been mostly associated with inattention because occasional lengthy response latencies could reflect focus of attention going away from the task. Impulsive-hyperactive features, in turn, were expected to be specifically associated with smaller μ and σ. The ex-Gaussian approach allowed applying similar modeling for response latency data in 2 different conditions, CPT and EPELI, which are used for detecting ADHD and predicting the related symptoms. Hence, the rationale for choosing these tasks was related to their common purpose of use and link to the complex clinical construct of ADHD, rather than testing how the tasks capture a specific cognitive construct.

Method

Participants

The participants in the 2 datasets (dataset 1 comes from a study by Seesjärvi et al.,¹⁶ and dataset 2 is from a study by Merzon et al.¹⁵) were 143 native Finnish-speaking children 9 to 13 years old (mean [SD] age = 10.5 [1.1] years; 109 boys, 34 girls). An ADHD diagnosis was present in 72 children, and 71 children were TD controls. The groups did not significantly differ in age and sex; however, there was a difference in socioeconomic factors (Table 1). The ADHD group also demonstrated lower scores on the two subtests (Similarities and Matrix Reasoning) of the WISC-IV¹⁹ than the TD group (Table 1). As expected, the ADHD group showed higher scores in ADHD Rating Scale-IV, parent form (ADHD-RS) and lower performance in the CPT and EPELI (Table 1). Children with comorbid disorders in neurological (section G in ICD-10) or psychiatric (section F) domain except for ICD-10 code F93 (Emotional disorder with onset specific to childhood) or F98 (Unspecified behavioral and emotional disorder) were excluded. The TD group included children within the same age range, without any neurological (section G) or psychiatric disorder (section F), and not receiving special assistance at school. Detailed criteria and characteristics of both samples can be found in the previous publications.^15,16 All children participated in the experiment either by visiting the laboratory at Aalto University (especially during COVID-19) or at their school facilities. The study was reviewed and approved by the Ethics Committee of the Helsinki University Hospital. All participants gave their informed consent according to the Declaration of Helsinki and were compensated with 2 movie tickets.

Table 1.

Background Characteristics of Attention-Deficit/Hyperactivity Disorder (ADHD) and Typically Developing (TD) Groups

Variable	ADHD (n =72)		TD (n = 71)		95% CI	Cohen d	p
	Mean	(SD)	Mean	(SD)
Age, y	10.37	(1.11)	10.71	(1.0)	−0.68 to 0.02	0.32	n.s.
	n		n
Sex					0.95 to 5.48	2.23a	n.s.
Boys	60		49
Girls	12		22
	Mean	(SD)	Mean	(SD)
Parental incomeb	3.75	(1.05)	4.27	(0.88)	−0.85 to −0.20	0.53	.002∗∗
Parental educationc	2.51	(0.59)	2.77	(0.42)	−0.44 to −0.09	0.51	.0034∗∗
Similarities test	22.39	(5.30)	24.71	(4.42)	−3.94 to −0.70	0.50	.005∗∗
Matrix reasoning test	19.63	(4.88)	21.72	(5.27)	−3.76 to −0.41	0.39	.02∗
ADHD-RS	31.69	(8.86)	7.21	(6.44)	21.90 to 27.05	3.10	< .001∗∗∗
CPT accuracy	0.83	(0.11)	0.88	(0.11)	−0.09 to −0.01	0.48	.007∗∗
EPELId
Total score	46.43	(8.28)	51.80	(6.71)	−7.09 to −2.83	0.71	< .001∗∗∗
Task efficacy	0.20	(0.08)	0.32	(0.15)	−5.72 to −0.08	1.00	< .001∗∗∗
Total actions	874.37	(361.33)	581.17	(218.71)	192.92 to 393.48	0.97	< .001∗∗∗

Note: ADHD-RS = ADHD Rating Scale-IV, parent form; CPT = continuous performance test; EPELI = Executive Performance in Everyday LIving task; n.s. = nonsignificant.

∗p < .05; ∗∗p < .01; ∗∗∗p < .001.

^aOdds ratio for Fisher exact test.

^bBefore tax per adult; 1 = < €1,500/mo (equivalent to <$1,642/mo); 2 = €1,500-2,200/mo (equivalent to $1,642-2,408/mo); 3 = €2,200–3,000/mo (equivalent to $2,408-3,284/mo); 4 = €3,000-4,000/mo (equivalent to $3,284-4,379/mo); 5 = > €4,000/mo (equivalent to $4,379/mo).

^c1 = comprehensive school; 2 = high school/vocational school; 4 = university degree or equivalent.

^dSee Seesjärvi et al.¹⁶ for description of the performance measures.

Tasks and Questionnaires

EPELI mimics demanding open-ended real-world situations in which participants perform everyday household chores under high attentional-executive demands in a home environment (see the original studies^15,16 for details) (Figure 1). The game is implemented by Peili Vision company (http://www.peilivision.fi/). An Oculus Go (Meta Reality Labs, Menlo Park, California) (2560 × 1440 resolution, 60/72 Hz refresh rate, and 101° field of view)¹⁶ or Pico Neo 2 Eye (Pico Immersive Pte. Ltd, Singapore)¹⁵ head-mounted display and a hand controller were used for playing EPELI. Navigating in the environment is implemented via teleporting, which can be done by pointing at a waypoint circle on the floor with a hand controller and simultaneously pressing a button. The same response button is used for interacting with the objects. In the game, children perform 13 short everyday scenarios that are given orally by a cartoon dragon character. Each task scenario includes 4 to 6 subtasks (eg, put your clothes on, eat breakfast, brush your teeth). In total, there are 70 tasks. Only a fraction of objects in the virtual home is relevant to the instructed tasks, and about half of the task scenarios are embedded with naturalistic auditory or audiovisual distractors. Total score (ie, correctly performed subtasks), task efficacy (ie, proportion of relevant actions out of all actions), and number of irrelevant actions were used to analyze speed-accuracy trade-off (see original study¹⁶ for detailed description of the measures).

Figure 1.

Illustration of the Task Performance in Executive Performance in Everyday LIving (EPELI) Task

Note:(A) The participant approaches a target object (a plate with breakfast) by moving next to the table via teleporting. (B) The participant points the target object to interact with it (eat the breakfast). (C) This subtask is now done. (D) The participant receives feedback (sees stars and hears a sound indicating that the performance was successful) and proceeds to the next stage (to put the empty plate in the sink). (E, F) The participant teleports next to the washing machine by pointing the ray to the closest waypoint and pressing the response button to perform a subtask requiring filling up the machine. (G) The participant opens the washing machine by pointing at it with the controller and clicking the response button. (H) The participant points to the clothes to grasp them, presses the response button to take them up, and points the ray to the washing machine and presses the response button again to move the clothes to the washing machine.

In the CPT, participants were presented with a series of either letters or images with fixed alternating intervals (1 second, 2 seconds, and 4 seconds) and instructed to press the space bar button as fast as possible when they see any stimulus except the predefined no-go stimulus. The CPT version with letters (dataset 1)¹⁶ resembles the Conners Continuous Performance Test Third Edition (Conners CPT 3). The participants were presented with one letter at a time and instructed to press the space bar as fast as possible when they saw any letter except the letter “X.” For the letter “X,” which was presented for 10% of the trials, the participants were instructed to withhold their response. The stimulus was presented for 250 ms. In the other CPT version (dataset 2),¹⁵ images were used as stimuli instead of letters. The participants were instructed to press a button as fast as possible when they saw any picture of a cartoon animal but withhold a response when they saw a cartoon image of a person (demonstrated to the participant during the instructions). The picture stimuli were presented for 350 ms, and this version had 20% trials targeting inhibition of the response. In both CPT versions, there was a 30-trial practice session that was repeated twice if the participants achieved fewer than 20 correct responses. The actual task included 180 trials, and the task duration was approximately 7 minutes.

The severity of participants’ ADHD symptoms was assessed with ADHD-RS.²⁰ Also, several other questionnaires and a Mini-International Neuropsychiatric Interview for Children and Adolescents (MINI-KID) diagnostic interview were administered as reported in the original studies.

Ex-Gaussian Modeling

The ex-Gaussian model was fitted to each participant’s response latency distribution with R package retimes v. 0.1.2.²¹ For EPELI, response latencies were defined as the time between 2 consecutive button presses. EPELI data were fitted with 3 models, one based on all response latencies, another for task-relevant clicks (response latencies where the second click was on an object relevant to the task), and the third for task-irrelevant clicks (response latencies where the second click was on an object irrelevant for completing the task). For the CPT, response latency was defined as the time between the stimulus onset and the button press. Only trials with correct response were included. Response latency data was square root transformed before the ex-Gaussian fitting.

Outlier Exclusion

The outliers were excluded in 2 steps. First, to achieve a good fit of the ex-Gaussian model, abnormally long or short response latencies were excluded for each participant separately based on the 1.5 interquartile range. Second, to avoid bias in statistical inference, participants with mean reaction time lying outside the 3 σ interval were excluded. The proportion of excluded EPELI data at the first step was 6.9% for dataset 1 and 7.3% for dataset 2. The second step resulted in exclusion of 1 participant (ADHD group) from dataset 1 and 1 participant (TD group) from dataset 2. For the CPT, 4.9% of data points in dataset 1 and 3.5% of data points in dataset 2 were excluded at the first step. At step 2, CPT data from 2 participants in the ADHD group were removed from dataset 1, while no participants were removed from dataset 2. Finally, 2 participants were excluded from dataset 1 (1 participant from the TD group and 1 participant from the ADHD group) due to CPT accuracy below 33%. In dataset 2, no participants were excluded due to low CPT accuracy. In addition to task performance exclusions, 3 participants were excluded from the original samples to balance the mean age between the groups (the youngest participant from the ADHD group and the 2 oldest participants from the TD group).

Goodness of Ex-Gaussian Model Fit

Outliers with worst fit lying outside of 1.5 interquartile range in log likelihood were excluded separately for each type of the clicks for each dataset separately. In EPELI, the fit of the model measured by log likelihood did not differ between the groups (all response latencies: 16.86, 95% CI −1.61 to 35.32, Cohen d = 0.31, p = .07; task-relevant response latencies: 1.05, 95% CI −2.33 to 4.22, d = 0.10, p = .57; task-irrelevant response latencies: 9.2, 95% CI −31.64 to 12.74, d = 0.14, p = .41) or between the 2 datasets (all response latencies: 4.21, 95% CI −14.39 to 22.81, d = 0.08, p = .66; task-relevant response latencies: 0.88, 95% CI −2.38 to 4.16, d = 0.09, p = .59; task-irrelevant response latencies: 17.39, 95% CI −4.53 to 39.32, d = 0.26, p = .12). In the CPT, there was a significant difference in the model fit between the data in the ADHD group and TD group (44.55, 95% CI 30.24 to 58.85, d = 1.09, p < .001). To accommodate for that, the mixed models used to compare the parameters between the ADHD and TD groups included log likelihood as a random factor. Also, the dataset was included as a random factor in the models.

Statistical Analysis

The statistical analysis was performed with R statistical computing software v. 4.2.0, using additional packages retimes,²¹ lme4,²² and car.²³ Fisher exact test, t test, and linear mixed effect models were used in the analysis. All the statistical tests were two-sided with a significance level of .05. False discovery rate correction was applied to correct p values for multiple comparisons.

Four models were tested to see which one would explain the most of variablity in ADHD symptoms' severity (see Supplement 1, available online). The models included ex-Gaussian parameters of response latency in EPELI and in the CPT. Candidate parameters were selected based on group differences. Highly intercorrelated parameters (r > 0.8) were excluded. Akaike information criterion was used to compare the quality of the models.²⁴

To test if the response latency variability can reliably predict participant group, a support vector machine classifier was trained with 10-fold cross-validation on ex-Gaussian parameters obtained in both tasks. The training and evaluation were implemented with the Python package sklearn v.1.0.242. Statistical difference in classifiers’ performance was assessed with two-sided t test on 30 iteration bootstrap cross-validation using 10 folds.

Results

EPELI Task

The 2 datasets did not differ significantly in mean and SD of participants’ response latencies either across all participants or within the groups (see Supplement 1, available online), and they were hence combined for further analyses. Overall, children with ADHD had faster response latencies (0.4 second, 95% CI 0.2 to 0.5, φ = 0.38, p < .001) with less variability (0.21 second, 95% CI 0.08 to 0.27, φ = 0.30, p < .001) compared with the TD group as indexed by SD (Figure 2). A further analysis indicated that shorter mean response latency was associated with a higher number of irrelevant actions (r = −0.79, 95% CI −0.85 to −0.72, p < .001) and lower task efficacy (r = 0.72, 95% CI 0.63 to 0.79, p < .001), while there was no correlation between participant’s total score and the mean response latency.

Figure 2.

Response Latency Distributions in Executive Performance in Everyday LIving (EPELI) Game

Note:ADHD = attention-deficit/hyperactivity disorder; TD = typically developing.

An ex-Gaussian model fitted to response latencies resulted in lower μ and σ in the ADHD group vs TD group (Table 2), indicating shorter intervals between controller clicks and lower intraindividual variability in the ADHD group. A further analysis examining task relevance revealed that for task-relevant clicks, σ was higher and τ was smaller in the ADHD group vs TD group, while for task-irrelevant clicks the group difference was in the opposite direction (Table 2). Task-irrelevant and task-relevant response latency measures also showed differential correlational patterns with the symptom scores. With respect to task-irrelevant responses, significant correlations with the symptoms were observed for μ and σ, while for task-relevant responses the only measure that correlated with the symptoms was τ (see Supplement 1, available online). An analysis testing the predictive power of the ex-Gaussian parameters derived from EPELI provided an area under the curve (AUC) of 0.76, suggesting fair classification accuracy for these parameters.

Table 2.

Results of Ex-Gaussian Modeling in Executive Performance in Everyday LIving (EPELI) and Continuous Performance Test (CPT) Data

Response latency measure	Parameter	ADHD	TD	95% CI	φ	p
		Mean	Mean
EPELI: all clicks	μ	0.74	0.92	0.10 to 0.24	0.40	< .001∗∗∗
	σ	0.19	0.25	0.03 to 0.10	0.29	.0015∗∗
	τ	0.38	0.34	−0.09 to 0.001	0.17	.052
EPELI: relevant clicks	μ	0.96	0.97	−0.05 to 0.08	0.04	.64
	σ	0.25	0.22	−0.07 to −0.007	0.21	.02∗
	τ	0.23	0.29	0.02 to 0.11	0.24	.0077∗∗
EPELI: irrelevant clicks	μ	0.72	0.91	0.12 to 0.27	0.42	< .001∗∗∗
	σ	0.17	0.26	0.05 to 0.13	0.37	< .001∗∗∗
	τ	0.39	0.33	−0.11 to −0.02	0.22	.012∗
CPT: correct responses	μ	0.34	0.35	−0.0005 to 0.03	0.17	.08
	σ	0.06	0.06	−0.006 to 0.007	0	.99
	τ	0.09	0.06	−0.035 to −0.014	0.41	< .001∗∗∗

Note: ADHD = attention-deficit/hyperactivity disorder; TD = typically developing.

∗p < .05; ∗∗p < .01; ∗∗∗p < .001.

Finally, to confirm that the obtained results are not explained by secondary factors, we examined intercorrelations between the potential extraneous variables (age, sex, WISC-IV score, and socioeconomic factors) and EPELI measures. Weak (r < 0.4) but significant correlations were observed between EPELI measures and other potential extraneous variables except parental income, which was not correlated with EPELI response latency measures (see Supplement 1, available online). In the multiple regression models where the demographic factors correlated with EPELI response latency measures were included as covariates, the results reported above remained essentially similar (see Supplement 1, available online).

CPT Task

In the CPT data, the ADHD group had longer mean response latency (0.02 second, 95% CI 0.003 to 0.039, φ = 0.20, p = .022) and higher response latency variability (0.02 second, 95% CI 0.016 to −0.039, φ = 0.42, p < .001) as indexed by SD (Figure 3). Ex-Gaussian analysis of CPT data showed that the only significant difference between the group was in the parameter τ, indicating a higher proportion of slow responses in the ADHD group (Table 2). An analysis testing the predictive power of the ex-Gaussian parameters derived from the CPT provided an AUC of 0.75, which did not differ significantly from the performance of the classifier with EPELI data.

Figure 3.

Response Latency Distributions in Continuous Performance Test (CPT)

Note:ADHD = attention-deficit/hyperactivity disorder; TD = typically developing.

The CPT variables were weakly correlated with age, sex, and WISC-IV score (see Supplement 1, available online). In the multiple regression analysis with the demographic factors associated with the CPT as covariates, mean CPT response latency no longer showed group differences, while in the SD of the response latency and τ parameter of the ex-Gaussian analysis the group differences remained significant.

Explained Variability in Symptoms

The linear mixed effects model explaining ADHD symptom variability with the ex-Gaussian parameters of response latencies included ADHD-RS total score as a dependent variable. The model based on EPELI included parameters μ and τ of all response latencies as a fixed effect and the dataset as a random effect, and both parameters were significant predictors of the symptom severity (μ: −31.19, 95% CI −43.87 to −18.50, φ = 0.41, p < .001; τ: −25.58, 95% CI −48.30 to −2.86, φ = 0.19, p = .028; the model's R² = 0.15). In the CPT model, only τ was a significant predictor of symptom severity in the similar model (165.98, 95% CI 82.70 to 249.26, φ = 0.34, p < .001; the model's R² = 0.11). The differences in Akaike information criterion of the models fit to the same sample was 3.26, which indicates considerably stronger support for the EPELI-based model than the CPT-based model for explaining symptom variability.²² The ex-Gaussian parameters of the 2 tasks used in the models were not correlated (see Supplement 1, available online). EPELI response latency measures were, however, correlated with other CPT measures. This correlational pattern was similar to the one for EPELI response latency measures and parent-rated ADHD symptoms: μ and σ for task-irrelevant responses and τ for task-relevant responses showed correlations with CPT accuracy as well as omission and commission errors in the CPT (see Supplement 1, available online).

Discussion

This study demonstrated how intraindividual variability in response latencies in a naturalistic task tapping complex goal-directed behavior can be quantified and used for child neuropsychiatric assessment. It also examined the related intraindividual variability measures obtained with a gold standard CPT that is widely used to complement ADHD assessment. Thus, we tested whether the 2 different tasks—a simple construct-based experimental task (CPT) and a function-led virtual reality task (EPELI) where real-life symptoms critical for diagnosis are expected to manifest themselves—were associated with similar or different characteristics of ADHD.^25,26 As our hypothesis predicted, we observed that response latencies in EPELI were shorter in children with ADHD than in TD controls, whereas in the CPT, children with ADHD were slower than TD children. Moreover, shorter response latencies in EPELI were associated with task-irrelevant behavior and ADHD symptom scores.

These findings provide supportive evidence for a novel objective indicator of hyperactive-impulsive characteristics of ADHD. Increased intraindividual variability of cognitive processing in ADHD was observed in both tasks. However, our novel findings indicate that besides response latencies as such, related intraindividual variability measures also are context-specific, as they were in opposite directions for responses to relevant vs irrelevant stimuli in EPELI, and the correlations between intraindividual variability in EPELI and the CPT were negligible. The lack of correlation between EPELI and the CPT could reflect either that fluctuation of sustained attention in a situation where the participant is a passive observer (eg, monitoring a school class) is different from a situation where the participant is an active agent (eg, executing any action sequences at own pace) or that intraindividual variability in response latencies reflects different cognitive processes in self-paced (eg, inhibition of impulsive reactions, response strategy adaptation) and strictly externally paced (eg, fluctuation of attention) tasks.²⁷ Nevertheless, we demonstrate that measuring intraindividual variability can also be used to develop behavioral markers for open-ended naturalistic tasks. Our findings advocate for the use of naturalistic paradigms to interpret how objective behavioral markers of psychiatric disorders connect to different types of real-life situations in which the symptoms occur. Such a cross-validation framework could also be used for testing the ecological relevance of other neuropsychological indicators developed to characterize cognitive deficits in psychiatric and neurological disorders.

Sitting still with eyes fixed and waiting to press a response button to isolated stimuli is an extremely simplified situation that is neither characteristic of human goal-directed behavior nor representative for cognitive deficits observed in many common child and adolescent psychiatric disorders.^1,28,29 While detection of problems with sustained attention with such methods is important, it represents a relatively narrow aspect out of the diverse problems associated with ADHD.30, 31, 32 Moreover, in real-world situations, there is often a considerable freedom to choose different actions, for instance, based on information searched by turning the head and eyes and moving around. Much of everyday goal-directed behavior manifests as self-paced interaction with rich contexts filled with meaningful stimuli.

EPELI was developed to emulate such everyday situations where sequences of multiple actions should be executed in a given time (eg, preparing to go to school or engage in a hobby, morning or evening routines). While there is increasing evidence that in individuals with ADHD the ability to concentrate is highly dependent on the context (eg, viewing movies,¹¹ playing games,¹³ doing physical activity,¹⁴ listening to music³³), function-led experimental approaches that allow reliable objective measurements of volitional real-world behavior have been lacking.^1,34 The present novel methodological framework allows study of whether and how the proposed endophenotype of neuropsychiatric disorders, intraindividual variability in cognitive performance, is exhibited in real-world situations.³⁵ Our results suggest that it does, but one should pay attention to the contextual factors influencing intraindividual variability. In EPELI, processing task-relevant stimuli was faster and less fluctuating in children with ADHD, meaning that every time that they knew what should be done, they reacted more quickly and more consistently than TD children. This was unexpected, as numerous CPT studies, including the present one, have shown that occasional sluggish task-relevant responses are more common in participants with ADHD.³ In both tasks, τ in relevant responses was correlated with symptoms, but as there was no correlation between τ in EPELI and τ in the CPT, we presume that this measure reflects different aspects of ADHD. These differences could possibly reflect whether the task is endogenously or exogenously controlled or may relate to different aspects in ADHD symptoms that task-specific τ measures capture (inattention symptoms for the CPT, hyperactive-impulsive symptoms for EPELI). In EPELI, σ of task-relvenat response latencies was larger in children with ADHD than in TD children, meaning that the clinical group did manifest greater intraindividual variability, but it was in a different latency range than typically observed in CPT studies. Further research is needed to clarify the cognitive functions related to response fluctuations at different timescales in the context of the novel EPELI task. It has been suggested that the flexibility to adjust the dynamics of a situation to sync with their natural pace (eg, a video game) could help children with ADHD to perform prolonged tasks.¹² Also, here, children with ADHD were able to perform the rather long and demanding EPELI task relatively well. However, the fast-paced children with ADHD were still more likely to be involved in irrelevant behaviors, and even though they responded more consistently to relevant stimuli, their error levels in regard to targets were higher in EPELI.

Occasional distractions to the irrelevant stimuli clearly attracted the attention of children with ADHD in EPELI as well as in the CPT, and this type of behavior was more fluctuating when the stimulus was naturalistic and the situation was dynamic. Regarding what cognitive processes specifically are behind this phenomenon, the present study does not give a clear-cut answer. In function-led tasks such as EPELI, the main advantage is interpretability in terms of ecological relevance. At the same time, in such a complex task the link between the measurement variable and the underlying cognitive construct necessarily becomes more vague.³⁶ Interestingly, however, the response latency measures for the task-irrelevant responses showed a different pattern than for the task-relevant responses. Interactions with nonrelevant objects occurred more quickly (μ) and less variably (σ) in children with ADHD compared with TD children. This is consistent with the immediate reward–seeking nature of impulsive behavior.³⁷ Larger τ for task-irrelevant responses in children with ADHD, in turn, could reflect several things. Their behavior may have been temporally variable at longer latencies, for instance, because they did not remember what to do and ended up testing object interactions that turned out incorrect or because mind wandering or attentional lapses may have been more frequent during episodes when the target action was not actively in mind. It is also noteworthy that difficulties with inhibition control (impulsivity) and focusing attention (inattention) could be closely related when it comes to dynamic processes related to rapid responding. In this study we observed speed accuracy trade-off, meaning that faster responding resulted in more mistakes, and in our previous study we showed that children with ADHD had difficulties in efficiently focusing attention to the objects with which they interacted in the game at the same time when they performed more impulsive actions. To give a practical example, in the school classroom, children with ADHD could have difficulties in inhibiting task-relevant behavior, for instance, waiting their turn when they know the answer to the teacher’s question, and in a similar manner they may fail to inhibit task-irrelevant behaviors.^2,3

A potential challenge for naturalistic paradigms where multiple features of attention control are needed simultaneously (eg, following the instructed task, regulating impulses to the attractive objects, executing goal-directed behaviors) relates to complex interactions that come to play. For instance, whether the person remembers, understands, or can hold attention to the goal could be associated with both with the person's state and the environmental characteristics such as the reward value, physical saliency or attractiveness of the available task-relevant and task-irrelevant objects. To be able to objectively capture the complex ADHD symptoms and their underlying mechanisms, the construct-based and function-led tasks are likely to complement each other.^38,39 In future studies, different types of naturalistic but contextually controlled virtual reality tasks could possibly be used to determine the contextual boundaries for intraindividual variability in psychiatric disorders and to apply this information also for transdiagnostic and differential diagnostic purposes.^40,41 We believe that the novel approach in which the measured behaviors are self-paced and more closely correspond to the real-life symptoms is important in the development of objective predictive measures of the common debilitating cognitive dysfunctions in psychiatric conditions.

Sustained attention that the CPT taps is required in real-world situations where the individual should continuously focus on the incoming input in a relatively static situation. The most popular ecologically valid variant of this task that emulates everyday life is the virtual classroom task, where task performance is correlated with the conventional CPT version employed in this study.⁴² Intraindividual variability in the CPT was also observed in the present study.^3,43 EPELI, in turn, was developed to detect various aspects of everyday life executive functions in dynamic situations involving interaction with the surroundings.¹⁶ With regard to real-life ADHD symptoms, the EPELI game emulates several factors that go beyond the concept of sustained attention (eg, “Often does not follow through on instructions and fails to finish schoolwork, chores, or duties in the workplace”; “Often has trouble organizing tasks and activities”; “Is often forgetful in daily activities”), and we have operationalized measures other than those related to response latencies to capture these characteristics.^15,16 Although EPELI explained variability in the ADHD symptoms slightly better than the CPT, it is noteworthy that intraindividual variability measures in both of the tasks were unable to explain a considerable amount of parent-evaluated symptom variability. Moreover, only moderate classification accuracy with respect to the group status was achieved with the measures derived from the 2 tasks. To obtain high discriminative validity, there are other EPELI measures, such as task efficacy (AUC = 0.83) or eye movement behavior (AUC = 0.92), that give better results.^15,16 Single-response latency measures are hence probably not sufficient to capture the widespread spectrum of symptoms. Perhaps a combination of naturalistic measures reflecting multiple carefully selected situations in which the symptoms are manifested (eg, home, school) would provide the best outcome with respect to objective measurements. The lack of correlation between EPELI and the CPT highlights the fact that intraindividual variability should not be considered as a unitary indicator, but rather a measure that is strongly modulated by the task context and stimulus contents.

The main limitations of the present study relate to sampling and characteristics of the population as well as inferences that can be made about the response latency effects in a complex task. Although it has been previously reported that children with ADHD are more likely to come from families with lower education and income,⁴⁴ future studies should pay more careful attention to balancing the groups in this regard. We also had a limited age range. Children younger than 9 years were not selected to make sure that even the rather short VR exposure time did not have negative effects considering participant safety.45, 46, 47 However, with younger children, it would be possible to use the flat-screen version of EPELI that is less immersive but taps largely the same constructs as the current head-mounted display version.⁴⁸ The 2 tasks selected for the study, EPELI and CPT, are quite different from each other, which limits the comparisons of the related results to the clinical domain. In our opinion, the CPT as a gold standard was a good choice as a comparison task even though it likely taps different cognitive constructs than EPELI: first, to demonstrate that the participants have problems with sustained attention; to compare a self-paced and externally paced task; and third, due to the lack of more complex and comparable tasks that would be able to capture cognitive fluctuations in ADHD. Finally, parent ratings obtained for inattention and hyperactivity/impulsivity domains in the present study were strongly correlated, meaning that the ADHD-RS questionnaire was not able to distinguish participants with respect to ADHD subtypes. Such a distinction could help in interpreting the clinical constructs that response latency variables in EPELI measure.

Our findings support the results of previous studies showing increased intraindividual variability that reflects a subset of unusually slow response latencies in children with ADHD compared with TD children. This was observed in the widely used CPT and for task-irrelevant stimuli in a naturalistic EPELI task. At the same time, in EPELI the children with ADHD were generally clearly faster than TD controls in their self-paced interaction with the environment, while in the CPT task children with ADHD were slower than TD controls. Interestingly, response latencies for task-relevant stimuli in the EPELI task were, in turn, less variable in children with ADHD than in the TD children, and correlations between intraindividual variability metrics in EPELI and the CPT were negligible. Hence, our results suggest that task context and type considerably influence how intraindividual variability manifests. Naturalistic virtual reality conditions such as EPELI allow quantification of attentional executive deficits in situations resembling real-life situations in which the symptoms leading to ADHD diagnosis occur. Although multiple different situations are probably needed to cover the symptoms hampering everyday life of individuals with ADHD, the present results provide a strong recommendation for novel research that pursues development of objective behavioral markers.