Skip to main content
PMC home page
JMIR Serious Games logoLink to JMIR Serious Games
. 2016 Sep 22;4(2):e15. doi: 10.2196/games.5997

A Study on the Validity of a Computer-Based Game to Assess Cognitive Processes, Reward Mechanisms, and Time Perception in Children Aged 4-8 Years

Janneke CAW Peijnenborgh 1,✉, Petra PM Hurks 2, Albert P Aldenkamp 3,4, Erik D van der Spek 5, GWM Rauterberg 5, Johan SH Vles 6, Jos GM Hendriksen 1
Editor: Mariano Alcañiz
Reviewed by: Jose Antonio Gil Gomez
PMCID: PMC5054232  PMID: 27658428

Abstract

Background

A computer-based game, named Timo’s Adventure, was developed to assess specific cognitive functions (eg, attention, planning, and working memory), time perception, and reward mechanisms in young school-aged children. The game consists of 6 mini-games embedded in a story line and includes fantasy elements to enhance motivation.

Objective

The aim of this study was to investigate the validity of Timo’s Adventure in normally developing children and in children with attention-deficit/hyperactivity disorder (ADHD).

Methods

A total of 96 normally developing children aged 4-8 years and 40 children with ADHD were assessed using the game. Clinical validity was investigated by examining the effects of age on performances within the normally developing children, as well as performance differences between the healthy controls and the ADHD group.

Results

Our analyses in the normally developing children showed developmental effects; that is, older children made fewer inhibition mistakes (r=−.33, P=.001), had faster (and therefore better) reaction times (r=−.49, P<.001), and were able to produce time intervals more accurately than younger children (ρ=.35, P<.001). Discriminant analysis showed that Timo’s Adventure was accurate in most classifications whether a child belonged to the ADHD group or the normally developing group: 78% (76/97) of the children were correctly classified as having ADHD or as being in the normally developing group. The classification results showed that 72% (41/57) children in the control group were correctly classified, and 88% (35/40) of the children in the ADHD group were correctly classified as having ADHD. Sensitivity (0.89) and specificity (0.69) of Timo’s Adventure were satisfying.

Conclusions

Computer-based games seem to be a valid tool to assess specific strengths and weaknesses in young children with ADHD.

Keywords: experimental games, ADHD, children, neuropsychological test

Introduction

Assessment of children’s cognitive strengths and weaknesses is an important focus of clinical child neuropsychological research and clinical care worldwide [1]. Cognitive abilities are quantified traditionally by use of, for example, paper-and-pencil performance tests and more recently with computer-assisted tools [2,3]. Performances on traditional cognitive tests are believed to be influenced significantly by noncognitive functions, such as motivation and perseverance [4]. Therefore, a lower score on, for instance, a working memory test might indicate a memory problem but also, for example, a decreased motivation. This overall performance score therefore only limitedly reflects the underlying “cause” in case of a decreased performance, which makes this score difficult to interpret. In order to test one’s cognitive abilities more purely, it would be preferable to (1) optimize motivation in the test situation and (2) assess motivation in a separate test additionally— unfortunately, tests measuring motivation in children are relatively scarcely used in the clinics.

On the basis of the literature, it is known that introducing immediate (vs delayed) rewards and the adaptation of item difficulty levels to the child’s abilities are likely to increase motivation in children and help the child to stay focused on the tasks that he or she needs to do [5]. Also, introducing a context (eg, by introducing cognitive tests in the context of a story) may increase motivation, although data on the effects of including a story line on motivation or engagement in games are so far inconclusive [6]. The introduction of a story line improves the child’s feeling of being part of a gaming environment [7]. In addition, the use of intrinsic fantasy elements has been found to improve motivation to conduct a specific task [8,9].

The above-mentioned increase in popularity of computer-assisted assessment tools is partly caused by the fact that it is relatively easy to implement these immediate rewards, to adapt difficulty levels, to implement a story, and to use intrinsic fantasy elements. This leads to a situation in which the child does not have the feeling of being assessed but instead thinks that he or she is playing a game [10]. This is especially important in tests that are designed for children because it is known that children can behave differently when they know they are being studied (also known as the Hawthorne effect [11]). For this purpose, we developed “Timo’s Adventure,” a computer-based game that consists of 6 mini-games. These mini-games aim to assess different cognitive processes, for example, attention, planning, and working memory; delay aversion, as a measure of motivation; and time perception (see method section). The aim of this study was to examine the validity of Timo’s Adventure in assessing strengths and weaknesses in the above-mentioned domains of young children, aged 4-8 years. Proof of validity was sought by studying group differences in performance on the mini-games. Two types of relevant group comparisons were made. First, the age of the child is believed to be a relevant variable and was therefore studied in relation to performances on the mini-games: we expected younger children (aged <6 years) to perform less well than older children on all mini-games (in line with studies on, eg, cognitive development [12] and time perception [13,14]). For this purpose, age-related differences in a group of normally developing children (N=96) were examined.

Next, the scores on the mini-games of normally developing children were compared with those of children with attention-deficit/hyperactivity disorder (ADHD; N=40). Attention-deficit/hyperactivity disorder is a developmental disorder that is associated with academic difficulties and social disadvantage [15]. According to the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition; DSM-V) [16], 2 main areas of impairment in children with ADHD exist: inattention (eg, difficulty in maintaining attention during a task or problems in dividing attention) and hyperactivity and impulsive behavior (eg, acting out before thinking about the consequences). Previous research has found that cognitive difficulties, more specifically in the domain of working memory and attention, occur in children with ADHD [17]. However, according to the triple-pathway model by Sonuga-Barke et al [18], not all children with ADHD have cognitive weaknesses. In this model 3 distinct patterns of ADHD deficits are distinguished. The first pathway is related to cognitive functions and is called the inhibitory-based executive dysfunction. This pathway views ADHD as a disorder of dysregulation of thought and action associated with diminished inhibitory control (ie, executive functions). In the second pathway, ADHD is explained as a motivational style associated with fundamental alterations in reward mechanisms. Children with ADHD are assumed to prefer small immediate rewards over large delayed rewards, which results in inattentive, overactive, and impulsive behaviors [19]. The third pathway states that deficits in time perception, for instance, deficits in distinguishing between two time intervals, producing time intervals, and estimating time, are another component of ADHD. Indeed, time perception deficits have been reported for children with ADHD [20]. However, the results are not consistent; that is, some authors report no ADHD-related deficits [21]. All 3 pathways are believed to have their own neural substrate [18]. Sonuga-Barke and colleagues [18] found in children with ADHD aged between 6 and 17 years that delay aversion, poor executive functions, and poor time perception are core, but unrelated and independent, characteristics of ADHD. A person with ADHD can have deficits in one of the pathways or a combination of pathways. Neuropsychological measurements (which are used to examine possible deficits in one of the pathways) usually focus primarily on just one of the pathways, whereas one can conclude from the model by Sonuga-Barke et al that it is necessary to gain information on possible deficits in all 3 pathways. Sonuga-Barke and colleagues used several distinctive computerized tasks to collect information about the 3 pathways. These were, however, not connected in a fantasy gaming environment or by a story line. In our game, a story line was included in order to immerse the player in an intrinsic fantasy and possibly improve the reliability of the diagnosis. To our knowledge no computerized diagnostic tools exist in which all 3 pathways are included in combination with all motivation-enhancing elements discussed above (including a story line or fantasy game elements), although some training tools with a story line exist (for instance, Braingame Brian [22]). In Timo’s Adventure story line, distracting factors are included to measure real-life distraction. Previous research found that distractors in a computerized continuous performance test resulted in more distractibility in children with ADHD than in their healthy peers [23].

In summary, the aim of this study was to investigate the clinical validity of Timo’s Adventure.

Methods

Participants

Normally Developing Children

Parents of all children enrolled in the first 4 grades of 4 Dutch elementary schools were informed by a letter about the study. Informed consent of 102 children was acquired. A total of 4 children were excluded because they were not native speakers and instructions in Timo’s Adventure were in the Dutch language. In addition, 2 children were excluded because they had a DSM-V diagnosis. The final dataset consisted of 96 children (43 boys), age ranging from 4 to 8 years. An overview of characteristics for this group can be found in Table 1.

Table 1.

Characteristics of all participants.

Characteristics Typically developing
children		 ADHDa group
Number of participants 96 40
Boys/girls 43/53 30/10
Age range, years 4-8 6-8
Mean age (SD), years 5.85 (1.33) 6.90 (0.74)
Verbal IQ (WPPSI-III-NLb Vocabulary), mean (SD) 94.74 (10.79) 87.92 (13.05)
Open in a new tab

aADHD: attention-deficit/hyperactivity disorder.

bWPPSI-III-NL: Dutch version of the Wechsler Preschool and Primary Scale of Intelligence [25].

All children were tested individually in a private room at their school. Approval for testing this sample was given by the Ethical Review Committee of the Faculty of Psychology and Neuroscience of Maastricht University, the Netherlands.

Children With Attention-Deficit/Hyperactivity Disorder

Parents of patients of the outpatient clinic Center for Neurological Learning Disabilities were asked to participate in this study by their medical specialist. In parallel, parents of children enrolled in a special needs program for children with behavioral problems were informed by a letter about the study and asked to participate, via the children’s school. Informed consent of 62 children with a diagnosis of ADHD was acquired. A total of 22 children did not meet the inclusion criteria because they had a comorbid DSM-V diagnosis (n=4), because they used medication for attentional problems and hyperactive behavior (stimulants, atomoxetine, tricyclic antidepressants, or clonidine; n=6), or because of a combination of these exclusion criteria (n=12). The final dataset of the ADHD group consisted of 40 children (30 boys), all with a diagnosis of ADHD according to DSM-V. These diagnoses were made on the basis of a protocol formulated by Goldman et al [24] that includes (1) extensive history taking, (2) cognitive testing, (3) general physical and neurological examination of the child, and (4) systematic assessment of ADHD characteristics by means of structured questions based on the most recent version of the DSM [16]. Age range of the clinical sample was 6-8 years. An overview of characteristics of the ADHD group can be found in Table 1.

Children enrolled in the special needs program for children with behavioral problems were tested individually in a private room at their school. The children who were patients of the Center for Neurological Learning Disabilities were seen for neuropsychological testing as part of clinical care.

Approval for testing this sample was given by the Medical Ethical Board of Kempenhaeghe.

Materials

Computer-Based Game Timo’s Adventure

Timo’s Adventure is a single-player game. All tasks are embedded in a story line: the main character in Timo’s Adventure is Timo, a friendly alien whose rocket has run out of fuel [26]. He asks the child to go on an adventure together to collect stars that can be used as fuel. To complete all tasks, it takes approximately 20 minutes. The game is categorized as a serious game: a game designed for specific purposes beyond entertainment [27]. The game has a first-person view to simulate the feeling of presence, to make the child feel like he or she is inside the game world.

Development of the game was divided into 3 stages: design, implementation, and evaluation, as based on the iterative software cyclic model and the spiral model [28-30]. Besides the designers of Eindhoven University of Technology, users were invited in the design process. Users in this case were children (who helped us by explaining what they would like and by making drawings) and psychologists from Kempenhaeghe (who participated in the development of the functionalities of the game, the visual graphics, and story line that needed to match the age of the children). In the implementation phase, the functionalities of the game were created (by engineers of Eindhoven University of Technology in close collaboration with the psychologists from Kempenhaeghe). During the evaluation phase (consisting of a user test with a paper prototype technique and an computer-based prototype), the game was played and evaluated by children and by psychologists who were not part of the development team. This feedback was used to improve the game and remove small bugs [31].

The computer-based game consists of 6 different tasks (ie, 6 mini-games), each of which measures different neurocognitive functions and thus gives information about possible deficits. The 6 tasks measure aspects of executive functions, time perception, and reward mechanisms to represent the 3 pathways of Sonuga-Barke’s model. All tasks were developed because of the need to measure the specific function and modeled after the theoretical background of the triple-pathway model.

Pathway 1: Executive Functions

All irrelevant mouse clicks are measured and thus give information on impulsivity and hyperactive behavior of the child. Furthermore, the following mini-games are included to assess impairments in this pathway: Dressing up (planning), Sandwich (working memory), Monkey (inhibition), and Magic Land (simple reaction time).

The first task, the mini-game Dressing up (planning), is set in the bedroom of the child. Timo tells the child that he or she needs to get dressed before the adventure can start. Several garments are spread throughout the room, which can be selected by clicking on them (see Figure 1). After clicking, the garment moves toward the reflection of the child in the mirror and the child gets dressed. This task gives information on the planning and organization skills of the child. The order in which the child selects the clothes is assessed to see whether the child is capable of planning his or her actions in the right order. The child can get a total score of 2 points: 1 point for being completely dressed and 1 point for using an executable and correct manner.

Figure 1.

Figure 1

Open in a new tab

Screenshot of the mini-game Dressing up (planning).

The second task, the mini-game Sandwich (working memory), is set in the kitchen (see Figure 2). Timo tells the child that it is necessary to eat something before starting the adventure, and he shows the child pictures of the ingredients. The child needs to remember these ingredients and select them in the same order as presented by Timo. The first sandwich starts with 2 ingredients, adding up to 5 ingredients in the last sandwich. This task is a measurement of the capacity of the visual working memory. According to Craeynest [32], the capacity of the working memory develops from remembering 2 targets when the child is 2.5 years old, to 3 targets when 3 years old, and 5 targets when the child is 7 years old. Martinussen et al [33] found that the capacity of the working memory in children with ADHD is markedly lower. In this task, the child can get a total score of 5 points: one for each correct sandwich.

Figure 2.

Figure 2

Open in a new tab

Screenshot of the mini-game Sandwich (working memory).

In the third task, the mini-game Monkey (inhibition), a monkey has thrown banana peels on the road (see Figure 3). To cross the road, the child needs to swipe the banana peels and clear the road. However, if the monkey sees the child swiping banana peels, it will undo the child’s actions. The monkey is playing hide-and-seek and appears suddenly. The child needs to wait for the moment the monkey disappears. This is a go or no-go task that gives information on the response inhibition of the child: is the child capable of inhibiting his or her response until the monkey hides? Children with ADHD have deficits in this response inhibition and they can be inclined to react impulsively [19]. The number of failures (ie, when the monkey sees the action of the child) is the outcome variable of this task; the higher this score, the worse the inhibition skills.

Figure 3.

Figure 3

Open in a new tab

Screenshot of the mini-game Monkey (inhibition).

In the mini-game Magic Land (simple reaction time), stars shoot upward from magic holes (see Figure 4). The child needs to collect these stars. If the child does not react within 2 seconds, the stars will disappear. The task ends after 50 stars shoot upward. Outcome variables in this task are the number of collected stars (with a maximum of 50) and average reaction time for collected stars. Slower and more variable reaction times have been found to be a characteristic of ADHD [34].

Figure 4.

Figure 4

Open in a new tab

Screenshot of the mini-game Magic Land (simple reaction time).

Pathway 2: Reward Mechanisms

In the task mini-game Rocket (delay aversion; see Figure 5), the child gets a choice between an immediate but small reward (ending of the task) or a delayed but bigger reward (a flight in the rocket, after 2 minutes of waiting). The child can end the task at any moment. Impulsive behavior occurs when responding produces more immediate, relatively smaller rewards at the cost of delayed, larger rewards [35]. Outcome variables are whether the child chooses a small or big reward and how long (in seconds) the child waited.

Figure 5.

Figure 5

Open in a new tab

Screenshot of the mini-game Rocket (reward mechanisms).

Pathway 3: Time Perception

The task mini-game Balloon (time production) is set at a river with a broken bridge (see Figure 6). To cross the river, the child needs to inflate a balloon with the balloon machine by producing a time interval of 10 seconds. A produced interval between 9 and 11 seconds results in a perfect balloon. When the produced interval is smaller than 9 seconds the balloon falls into the water, and a produced interval larger than 11 seconds results in a balloon that flies away. The child can make a maximum of 3 perfect balloons, or the task will end after 3 minutes with a perfect balloon (regardless of what the interval is). Barkley et al [36] suggested that the estimation of temporal intervals is atypical in children with ADHD. The number of correct balloons is an outcome variable. Furthermore, the average production interval for the first 3 balloons is measured by subtracting 10 seconds from each of the first 3 balloons, transforming these scores to absolute scores, adding these scores, and then dividing them by 3. The higher this score, the less precise the mean produced intervals are.

Figure 6.

Figure 6

Open in a new tab

Screenshot of the mini-game Balloon (time perception).

Vocabulary

This subscale of the Dutch version of the Wechsler Preschool and Primary Scale of Intelligence (WPPSI-III-NL [25]) was used to estimate verbal intelligence [37]. In this task, the child was asked to give definitions of words such as “umbrella” or “shoe.” The total score can be transformed to IQ scores, with 100 as the mean (SD 15).

Statistics

Data were analyzed with IBM SPSS Statistics version 21.0.0.0. All outliers (scores with z>3.29) within the concerned group (ie, normally developing children and the ADHD group) were replaced by the mean + 3 times its standard deviation as advised by Field [38]. Means and standard deviations of all variables were calculated.

Potential age-related differences on Timo’s Adventure within the sample of normally developing children were examined by conducting Pearson correlation (for scale outcome measurements; eg, number of irrelevant mouse clicks, time used to complete the task) and Spearman correlation analyses (for the ordinal outcome measurements; eg, correct or incorrect, did the child choose the large or small reward). Age was used as a continuous variable in these analyses.

Second, a discriminant analysis was performed to investigate to which level the variables of the game can discriminate between children belonging to the ADHD group and the normally developing children. Sensitivity and specificity were measured. All children aged <6 years in the sample of normally developing children were excluded from these analyses, in order to match with the age of the children in the ADHD group, resulting in a control group of 57 children. An independent samples t test revealed that children in the ADHD group and children in the normally developing group were equal in terms of Vocabulary scores (ADHD group: mean 87.92, SD 13.05; control group: mean 94.74, SD 10.79; t67=1.92, P=.11), therefore it was not necessary to correct for IQ in further analyses.

Finally, it was analyzed on which variables children with ADHD scored significantly different scores from normally developing children. Again, all children aged <6 years in the sample of normally developing children were excluded from these analyses, in order to match with the age of the children in the ADHD group. Three types of analyses were used. For the continuous variables, general linear model univariate analyses were used. In four variables, the assumption of homogeneity in variances was violated; therefore, nonparametric t tests (Mann-Whitney) were used in these variables. In the categorical variables, Pearson chi-square analyses were performed.

Results

Descriptive Statistics of All Variables

Means and standard deviations for all variables of both groups are reported together with the analyses of group differences. All reported results are after correction for outliers.

Developmental Effects

An overview of all correlation analyses between age and variables of Timo’s Adventure can be found in Table 2.

Table 2.

Correlation between results on Timo’s Adventure and age for normally developing children (N=96).

Variables Agea P value
Pathway 1: executive functions


Dressing up, total score ρ=−.08 .46

Dressing up, clicks r=−.05 .66

Sandwich, total score ρ=.21 .05

Sandwich, clicks r=.09 .71

Balloon, clicks r=.14 .18

Monkey, failures r=−.33 .001

Monkey, clicks r=.08 .45

Magic Land, number of collected stars r=.60 <.001

Magic Land, average time for collected stars r=−.49 <.001

Magic Land, clicks r=.16 .14
Pathway 2: reward mechanisms


Rocket, reward ρ=.05 .65

Rocket, time waited r=.07 .50
Pathway 3: time perception


Balloon, correct ρ=.35 <.001

Balloon, average time for attempts r=−.25 .01
Open in a new tab

ar: Pearson correlation; ρ: Spearman correlation.

Pathway 1: Executive Functions

Pearson correlation analyses with age as a continuous variable showed significant correlations on 2 tasks: in the inhibition task (Monkey), older children had significantly fewer inhibition failures (r=−.33, P=.001); and in the reaction time task (Magic Land), older children collected significantly more stars (r=.60, P<.001) and were faster in collecting these stars (r=−.49, P<.001).

Pathway 2: Reward Mechanisms

No significant correlations were found in this pathway, indicating that age does not influence reward mechanisms.

Pathway 3: Time Perception

In the time production task (Balloon), older children produced significantly more correct balloons (ρ= .35, P<.001) and had more precise time productions than younger children (r=−.25, P=.01).

Differences Between Children With ADHD and Healthy Controls

Discriminant Analysis

All variables were included in a discriminant analysis to investigate whether Timo’s Adventure can discriminate between the children with ADHD and the healthy controls. A significant difference between the groups was found: Wilks Λ=.51, χ214=50.8, P<.001. The structure matrix (see Table 3) revealed that especially the number of mouse clicks in several tasks and the mini-games on reaction time (Magic Land), inhibition (Monkey), and planning (Dressing up) were potential predictors. The classification results showed that 72% (41/57) children in the control group were correctly classified, and 88% (35/40) of the children in the ADHD group were correctly classified as having ADHD. Overall, 78% (76/97) of the children were correctly classified as being in the ADHD group or in the control group. Sensitivity of Timo’s Adventure was 0.89 and specificity was 0.69.

Table 3.

Structure matrix in discriminant analysis.

Output variable Pooled within-group correlations between discriminating variables and standardized canonical discriminant functions
Magic Land, number of clicks −.82
Magic Land, number of collected stars .40
Balloon, number of clicks .34
Monkey, number of failures .31
Dressing up, total score .29
Rocket, time waited −.27
Balloon, number of correct balloons −.18
Sandwich, total score .17
Sandwich, number of clicks .14
Magic Land, average reaction time for collected stars .11
Rocket, small (=0) or large (=1) reward −.10
Monkey, number of clicks .09
Balloon, average time taken to inflate balloons −.02
Dressing up, number of clicks .01
Open in a new tab

Group Differences on Individual Variables

Because the combination of all variables was useful in discriminating between children with ADHD and normally developing children, the specific variables for which children with ADHD had a different result from normally developing children were examined. All significant differences between children with ADHD and healthy controls are reported.

Pathway 1: Executive Functions

Results of this pathway can be found in Table 4. There was a significant association between the group (ADHD or control) and the score on the Dressing up task (planning; χ22=11.4, P=.003, V=.35), indicating that typically developing children had better scores than children with ADHD on a planning task. In the Magic Land task (simple reaction time), children with ADHD used more mouse clicks in collecting stars than children in the control group (U=389.50, P<.001, r=−.55).

Table 4.

Results on Timo’s Adventure for normally developing children in the control group (N=56) and children in the attention-deficit/hyperactivity disorder group (N=40) in pathway 1, executive functions.

Variable Group Mean SD Statistic P value Effect size
Dressing up, total score (0-2)

 Controls 1.18 0.56 χ22=11.41 .003 Va=.35

ADHDb 0.93 0.81


Dressing up, number of clicks (minimum for a satisfying result is 4 clicks)

 Controls 17.59 14.32 Uc=895.00 .09 rd=−.17

ADHD 26.14 22.61


Sandwich, total score (0-5)

 Controls 0.89 1.10 Ff1,95=0.47 .50 Ƞp2=.01e

ADHD 1.05 0.97


Sandwich, number of clicks (minimum for 5 correct sandwiches is 22 clicks)

 Controls 87.49 51.12 U=823.00 .25 r=−.12

ADHD 92.57 53.48


Balloon, number of clicks (minimum for 3 correct balloons is 6)

 Controls 108.52 93.91 U=945.50 .24 r=−.12

ADHD 206.87 227.87


Monkey, number of failures (minimum is 0)

 Controls 0.77 1.33 F1,96=2.97 .09 Ƞp2=.03

ADHD 1.26 1.48


Monkey, number of clicks (minimum to complete the task is 6)

 Controls 29.72 2.20 F1,96=0.82 .37 Ƞp2=.01

ADHD 33.83 1.62


Magic Land, number of collected stars (maximum is 50)

 Controls 37.61 10.87 F1,95=2.92 .09 Ƞp2=0.03

ADHD 41.41 8.33


Magic Land, average reaction time for collected stars

 Controls 2.25 0.55 F1,95=0.02 .88 Ƞp2=0.00

ADHD 2.22 0.48


Magic Land, number of clicks

 Controls 76.27 19.52 U=389.50 <.001 r=−.55

ADHD 155.65 89.68


Open in a new tab

aV: Cramer’s V.

bADHD: attention-deficit/hyperactivity disorder.

cU: Mann-Whitney test.

dr: Pearson correlation coefficient.

eȠp2: partial variance explained.

Pathway 2: Reward Mechanisms

Results of this pathway can be found in Table 5. In the Rocket task, a significant association between the group (ADHD or control) and whether or not a child chose the delayed reward was found (χ21=7.3, P=.01, V=.28). Also, the total time that the child waited before he or she ended the task was significantly different: children in the control group were able to wait longer for the delayed reward than the children in the ADHD group (F1,92 =5.52, P=.02, Ƞp2=.06).

Table 5.

Results on Timo’s Adventure for normally developing children in the control group (N=56) and children in the attention-deficit/hyperactivity disorder group (N=40) in pathway 2, reward mechanisms.

Variable Group Mean SD Statistic P value Effect size
Rocket, small (=0) or large (=1) reward

 Controls 0.55 0.50 χ21=7.3 .01 Va=.28

ADHDb 0.28 0.46


Rocket, time waited (minimum is 0 seconds, maximum is 120 seconds)

 Controls 80.98 50.05 F1,92=5.52 .02 Ƞp2=.06d

ADHD 56.26 50.13


Open in a new tab

aV: Cramer’s V.

bADHD: attention-deficit/hyperactivity disorder.

cȠp2: partial variance explained.

Pathway 3: Time Perception

Results of this pathway can be found in Table 6. No significant differences between children with ADHD and normally developing children were found in the time production task.

Table 6.

Results on Timo’s Adventure for normally developing children in the control group (N=56) and children in the attention-deficit/hyperactivity disorder group (N=40) in pathway 3, time perception.

Variable Group Mean SD Statistic P value Effect size
Balloon, number of correct balloons (minimum is 0, maximum is 3)

 Controls 2.25 1.09 F1,96=0.07 .80 Ƞp2=.00a

ADHDb 2.21 1.06


Balloon, average time taken to inflate balloons

 Controls 4.33 4.05 F1,96=1.69 .20 Ƞp2=.02

ADHD 3.38 2.01


Open in a new tab

bȠp2: partial variance explained.

cADHD: attention-deficit/hyperactivity disorder.

Discussion

Principal Findings

Recently, the development and use of computerized tasks in measuring, for example, neurocognitive abilities is increasing and results in promising effects in the field of interventions. For instance, children with ADHD benefit from game-based training tools on executive functions such as Braingame Brian and Cogmed, as was reported in a review by Peijnenborgh et al [39]. Important elements of these training tools are believed to be the use of fantasy, a story line, adaptation of the degree of difficulty, and the use of immediate rewards. However, most studies focus on training tools, whereas in our research a diagnostic tool was studied. This computerized diagnostic tool, named Timo’s Adventure, was developed for young children (between 4 and 8 years old) to investigate the presence of the 3 distinct patterns of possible deficits in ADHD as described in the triple-pathway model by Sonuga-Barke and colleagues. To our knowledge, this is the first computerized tool in which all 3 pathways are assessed. The aim of this study was to investigate the clinical validity of Timo’s Adventure.

The first proof of validity was found in the developmental effects of the game. In a population of 96 normally developing children between 4 and 8 years old, we found significant correlations with age in 2 of the 3 pathways. The older the child, the faster he or she is in completing the tasks. Furthermore, it was found that older children are better in inhibiting their response on a go or no-go task. This is in line with previous research (eg, [40,41]). Also, in a reaction time task, older children have better responses on alertness and have better reactions to visually presented stimuli after a visual warning signal. Again, this is in line with previous research (eg, [42]). Furthermore, age-related differences were found in the third pathway (ie, time perception), indicating that the older the child, the better he or she is in producing a predetermined time interval. This is in line with research by, for example, Friedman and Laycock [43] and Pouthas and Jacquet [13] stating that development of time perception skills increases sharply at an early age (ie, before the age of 7 years) and refines in the last grades of elementary school. Interestingly, no developmental effects were found in the second pathway (reward mechanisms), indicating that age does not influence the choice between immediate or delayed rewards. This might be caused by the fact that this aspect might be fully developed before the age of 4 years, as the findings by Mischel et al [44] suggest.

Further proof for clinical validity was found in satisfying results on the discriminant analysis, indicating that Timo’s Adventure was correct in most classifications. Sensitivity and specificity of the measurement were satisfying. Our results are similar to, and sometimes even more promising than, other diagnostic measurements. For instance, Williams and colleagues [45] could classify 68% of the children with ADHD correctly when using IntegNeuro. However, only 2 of the 3 pathways are included in IntegNeuro, and it is not suitable for young children.

When looking more closely at the individual variables that help to differentiate between children with and without ADHD, we found several significant differences between both groups. In the first pathway (executive functions), we found that children with ADHD have significantly more irrelevant mouse clicks on the Magic Land task (reaction time) than healthy peers. This indicates impulsiveness, motor restlessness, and hyperactive behavior, as is also suggested by Hervey and colleagues [46]. Also, children with ADHD had lower planning skills than healthy controls had, what might be expected because planning and organization are affected in children with ADHD [47]. In several tasks we found results that were encouraging but not statistically significant (eg, children with ADHD tend to have more inhibition mistakes in the Monkey task and collect more stars in the Magic Land than typically developing children). This seems promising for the future: maybe, with some adjustments to the tasks, sensitivity and specificity can even increase.

Evidence was also found for differences between the ADHD group and the controls in the second pathway (reward mechanisms): children with ADHD chose the large but delayed reward less often than the control group and did not wait as long as the control group before deciding to end the task. This is in line with previous research, which states that children with ADHD have an aversion for delay [48]. No significant differences were found in the third pathway of Sonuga-Barke’s model (time perception). Although timing deficits are known in children with ADHD, it is not uncommon that time production tasks do not result in significant effects [49]. Further research is necessary to gain more information on this aspect of possible ADHD-related deficits.

Finally, we found that user experiences were positive: when asked afterward, 81% of the children said they liked the game very much, and an extra 14% of the children said that they liked the game.

One limitation of this study is that information on reliability cannot be reported at this moment. Because Timo’s Adventure consists of several independent functions, analysis of Cronbach alpha would automatically result in low consistency between the items. It would be interesting to test children several times, to collect data for test-retest reliability analyses. Further research is necessary to examine the reliability of this instrument. Another interesting question for future research might be to investigate possible effects of use of medication. In our analysis all children with ADHD who were taking medication (18 in total) were excluded, but it might be possible that medication influences only one (or a combination) of the pathways. Finally, it would be interesting to determine which (or combination of) tasks and corresponding outcome measurements are especially sensitive for the diagnosis of ADHD. It is possible that a total score and normative data can be measured, which can be used to determine a profile of ADHD symptoms. Future research is necessary to determine such a profile or total score.

Conclusions

This is the first time that all 3 pathways of Sonuga-Barke’s model are included in one diagnostic computerized tool with a context of rewards and story line. In clinical care, diagnostic instruments on time perception and reward mechanisms are scarce, but it is necessary to gain information on these aspects to complete an analysis of strengths and weaknesses of the child. Proof for validity of Timo’s Adventure was found in developmental effects and group differences between normally developing children and children with ADHD, and Timo’s Adventure was satisfying accurately when classifying to which group (ie, the ADHD group or the healthy controls) the child belonged. This suggests that Timo’s Adventure can be of added value in the diagnosis of ADHD because it helps in formulating a profile of strengths and weaknesses. Further research is necessary to confirm these findings and to examine potential effects of medication.

Acknowledgments

The authors are grateful for the collaboration with Pongpanote Gongsook, PhD, Christian Sallustro, PDEng, and Jun Hu, PhD, from the Eindhoven University of Technology (the Netherlands) in the development of Timo’s Adventure.

Abbreviations

ADHD

attention-deficit/hyperactivity disorder

DSM-V

Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition)

WPPSI-III-NL

Wechsler Preschool and Primary Scale of Intelligence, Dutch version

Footnotes

Conflicts of Interest: None declared.

References

  • 1.Swaab H, Bouma A, Hendriksen J, König C. Klinische Kinderneuropsychologie. Amsterdam: Boom; 2016. [Google Scholar]
  • 2.Assef EC, Capovilla AG, Capovilla FC. Computerized stroop test to assess selective attention in children with attention deficit hyperactivity disorder. Span J Psychol. 2007 May;10(1):33–40. doi: 10.1017/s1138741600006296. [DOI] [PubMed] [Google Scholar]
  • 3.Messina F, Tiedemann KB, de Andrade ER, Primi R. Assessment of working memory in children with attention-deficit/hyperactivity disorder. J Atten Disord. 2006 Aug;10(1):28–35. doi: 10.1177/108705284299. [DOI] [PubMed] [Google Scholar]
  • 4.Luciana M, Nelson CA. Assessment of neuropsychological function through use of the Cambridge Neuropsychological Testing Automated Battery: performance in 4- to 12-year-old children. Dev Neuropsychol. 2002;22(3):595–624. doi: 10.1207/S15326942DN2203_3. [DOI] [PubMed] [Google Scholar]
  • 5.Bioulac S, Arfi L, Bouvard MP. Attention deficit/hyperactivity disorder and video games: a comparative study of hyperactive and control children. Eur Psychiatry. 2008 Mar;23(2):134–41. doi: 10.1016/j.eurpsy.2007.11.002. [DOI] [PubMed] [Google Scholar]
  • 6.Wouters P, van Nimwegen C, van Oostendorp H, van der Spek ED. A meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology. 2013;105(2):249–265. doi: 10.1037/a0031311. [DOI] [Google Scholar]
  • 7.Gorini A, Capideville CS, De LG, Mantovani F, Riva G. The role of immersion and narrative in mediated presence: the virtual hospital experience. Cyberpsychol Behav Soc Netw. 2011 Mar;14(3):99–105. doi: 10.1089/cyber.2010.0100. [DOI] [PubMed] [Google Scholar]
  • 8.Habgood MPJ, Ainsworth SE. Motivating Children to Learn Effectively: Exploring the Value of Intrinsic Integration in Educational Games. Journal of the Learning Sciences. 2011 Apr 19;20(2):169–206. doi: 10.1080/10508406.2010.508029. [DOI] [Google Scholar]
  • 9.Van der Spek ED, Sidorenkova T, Porskamp P, Rauterberg M. The effect of familiar and fantasy aesthetics on learning experience of serious games. In: Pisan Y, Sgouros NM, Marsh T, editors. Entertainment Computing – ICEC 2014. Berlin, Heidelberg: Springer; 2014. pp. 9783662452110–138. [Google Scholar]
  • 10.Gongsook P, Peijnenborgh J, Sallustro C, Van der Spek E, Hu J, Bellotti F, Rauterberg M, Hendriksen J. In: A diagnostic tool on time perception of children with ADHD. De Gloria A, editor. Paris: Springer; 2014. pp. 9783319121567–405. [Google Scholar]
  • 11.McCarney R, Warner J, Iliffe S, van HR, Griffin M, Fisher P. The Hawthorne Effect: a randomised, controlled trial. BMC Med Res Methodol. 2007;7:30. doi: 10.1186/1471-2288-7-30. http://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-7-30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Diamond A. The early development of executive functions. In: Bialystok E, Craik FIM, editors. Lifespan Cognition - Mechanisms of Change. New York: Oxford University Press; 2006. p. 9780195169539. [Google Scholar]
  • 13.Pouthas V, Jacquet AY. A developmental study of timing behavior in 4 1/2- and 7-year-old children. J Exp Child Psychol. 1987 Apr;43(2):282–99. doi: 10.1016/0022-0965(87)90064-6. [DOI] [PubMed] [Google Scholar]
  • 14.Friedman WJ, Lyon TD. Development of temporal-reconstructive abilities. Child Dev. 2005;76(6):1202–16. doi: 10.1111/j.1467-8624.2005.00845.x. [DOI] [PubMed] [Google Scholar]
  • 15.Swanson J, Sergeant J, Taylor E, Sonuga-Barke E, Jensen P, Cantwell D. Attention-deficit hyperactivity disorder and hyperkinetic disorder. The Lancet. 1998 Feb;351(9100):429–433. doi: 10.1016/S0140-6736(97)11450-7. [DOI] [PubMed] [Google Scholar]
  • 16.American Psychiatric Association . Diagnostic and statistical manual of mental disorders. 5th edition. Washington, DC: Author; 2013. p. A. [Google Scholar]
  • 17.González-Garrido AA, Gómez-Velázquez FR, Zarabozo D, López-Elizalde R, Ontiveros A, Madera-Carrillo H, Vega OL, De Alba JL, Tuya JM. Time reproduction disturbances in ADHD children: an ERP study. Int J Neurosci. 2008 Jan;118(1):119–35. doi: 10.1080/00207450601042177. [DOI] [PubMed] [Google Scholar]
  • 18.Sonuga-Barke E, Bitsakou P, Thompson M. Beyond the dual pathway model: evidence for the dissociation of timing, inhibitory, and delay-related impairments in attention-deficit/hyperactivity disorder. J Am Acad Child Adolesc Psychiatry. 2010 Apr;49(4):345–55. doi: 10.1016/j.jaac.2009.12.018. [DOI] [PubMed] [Google Scholar]
  • 19.Barkley RA. Behavioral inhibition, sustained attention, and executive functions: constructing a unifying theory of ADHD. Psychol Bull. 1997 Jan;121(1):65–94. doi: 10.1037/0033-2909.121.1.65. [DOI] [PubMed] [Google Scholar]
  • 20.Hurks PP, Hendriksen JG. Retrospective and prospective time deficits in childhood ADHD: The effects of task modality, duration, and symptom dimensions. Child Neuropsychol. 2011;17(1):34–50. doi: 10.1080/09297049.2010.514403. [DOI] [PubMed] [Google Scholar]
  • 21.Bauermeister JJ, Barkley RA, Martinez JV, Cumba E, Ramirez RR, Reina G, Matos M, Salas CC. Time estimation and performance on reproduction tasks in subtypes of children with attention deficit hyperactivity disorder. J Clin Child Adolesc Psychol. 2005 Mar;34(1):151–62. doi: 10.1207/s15374424jccp3401_14. [DOI] [PubMed] [Google Scholar]
  • 22.Prins P, Ten Brink EL, Dovis S, Ponsioen AJGB, Geurts H, De Groot H, De Vries M, Van der Oord S. Braingame Brian: Een Executieve Functie training met game-elementen voor kinderen met cognitieve controleproblemen. Amsterdam: Lucertis & Shosho; 2010. [Google Scholar]
  • 23.Cassuto H, Ben-Simon A, Berger I. Using environmental distractors in the diagnosis of ADHD. Front Hum Neurosci. 2013;7:805. doi: 10.3389/fnhum.2013.00805. http://dx.doi.org/10.3389/fnhum.2013.00805. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Goldman LS. Diagnosis and Treatment of Attention-Deficit/Hyperactivity Disorder in Children and Adolescents. JAMA. 1998 Apr 08;279(14):1100. doi: 10.1001/jama.279.14.1100. [DOI] [PubMed] [Google Scholar]
  • 25.Wechsler D. WPPSI-III-NL Nederlandse bewerking: Technische Handleiding 2nd edition. Amsterdam: Netherlands: Pearson Assessment and Information BV; 2010. [Google Scholar]
  • 26.Sallustro C. Timo&Friends: A diagnostic tool on the time perception of children with ADHD. Eindhoven: University of Technology; 2013. p. A. [Google Scholar]
  • 27.Bellotti F, Berta R, De Gloria A. Designing Effective Serious Games: Opportunities and Challenges for Research. Int. J. Emerg. Technol. Learn. 2010 Nov 24;5(SI3) doi: 10.3991/ijet.v5s3.1500. [DOI] [Google Scholar]
  • 28.Boehm BW. A spiral model of software development and enhancement. Computer. 1988 May;21(5):61–72. doi: 10.1109/2.59. [DOI] [Google Scholar]
  • 29.Rauterberg M, Strohm O, Kirsch C. Benefits of user-oriented software development based on an iterative cyclic process model for simultaneous engineering. International Journal of Industrial Ergonomics. 1995 Oct;16(4-6):391–409. doi: 10.1016/0169-8141(95)00021-8. [DOI] [Google Scholar]
  • 30.Rauterberg M. An iterative-cyclic software process model. Proceedings Fourth International Conference on Software Engineering and Knowledge Engineering; 1992; Zurich, Switzerland. 1992. pp. 600–607. [DOI] [Google Scholar]
  • 31.Gongsook P. Interactive diagnostic game for time perception. Eindhoven: Technische Universiteit; 2015. pp. 978–90. [Google Scholar]
  • 32.Craeynest P. Psychologie van de levensloop. Leuven: Acco; 2010. [Google Scholar]
  • 33.Martinussen R, Hayden J, Hogg-Johnson S, Tannock R. A meta-analysis of working memory impairments in children with attention-deficit/hyperactivity disorder. J Am Acad Child Adolesc Psychiatry. 2005 Apr;44(4):377–84. doi: 10.1097/01.chi.0000153228.72591.73. [DOI] [PubMed] [Google Scholar]
  • 34.Gooch D, Snowling MJ, Hulme C. Reaction Time Variability in Children With ADHD Symptoms and/or Dyslexia. Developmental Neuropsychology. 2012 Jul;37(5):453–472. doi: 10.1080/87565641.2011.650809. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Grey I, Healy O, Leader G, Hayes D. Using a Time Timer to increase appropriate waiting behavior in a child with developmental disabilities. Res Dev Disabil. 2009;30(2):359–66. doi: 10.1016/j.ridd.2008.07.001. [DOI] [PubMed] [Google Scholar]
  • 36.Barkley RA, Koplowitz S, Anderson T, McMurray MB. Sense of time in children with ADHD: effects of duration, distraction, and stimulant medication. J Int Neuropsychol Soc. 1997 Jul;3(4):359–69. [PubMed] [Google Scholar]
  • 37.Alloway T. Can interactive working memory training improve learning? Journal of Interactive Learning Research. 2002;23(3):197–207. [Google Scholar]
  • 38.Field A. Discovering Statistics Using IBM SPSS Statistics 3rd edition. London: Sage Publications Ltd; 2009. [Google Scholar]
  • 39.Peijnenborgh JC, Hurks PM, Aldenkamp AP, Vles JS, Hendriksen JG. Efficacy of working memory training in children and adolescents with learning disabilities: A review study and meta-analysis. Neuropsychol Rehabil. 2015 Apr 17;:1–28. doi: 10.1080/09602011.2015.1026356. [DOI] [PubMed] [Google Scholar]
  • 40.Gathercole SE, Pickering SJ, Ambridge B, Wearing H. The structure of working memory from 4 to 15 years of age. Dev Psychol. 2004 Mar;40(2):177–90. doi: 10.1037/0012-1649.40.2.177. [DOI] [PubMed] [Google Scholar]
  • 41.Davidson MC, Amso D, Anderson LC, Diamond A. Development of cognitive control and executive functions from 4 to 13 years: evidence from manipulations of memory, inhibition, and task switching. Neuropsychologia. 2006;44(11):2037–78. doi: 10.1016/j.neuropsychologia.2006.02.006. http://europepmc.org/abstract/MED/16580701. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Thomas KM, Nelson CA. Serial reaction time learning in preschool- and school-age children. J Exp Child Psychol. 2001 Aug;79(4):364–87. doi: 10.1006/jecp.2000.2613. [DOI] [PubMed] [Google Scholar]
  • 43.Friedman WJ, Laycock F. Children's Analog and Digital Clock Knowledge. Child Development. 1989 Apr;60(2):357. doi: 10.2307/1130982. [DOI] [Google Scholar]
  • 44.Mischel W, Shoda Y, Rodriguez M. Delay of gratification in children. Science. 1989 May 26;244(4907):933–938. doi: 10.1126/science.2658056. [DOI] [PubMed] [Google Scholar]
  • 45.Williams LM, Hermens DF, Thein T, Clark CR, Cooper NJ, Clarke SD, Lamb C, Gordon E, Kohn MR. Using brain-based cognitive measures to support clinical decisions in ADHD. Pediatr Neurol. 2010 Feb;42(2):118–26. doi: 10.1016/j.pediatrneurol.2009.08.010. [DOI] [PubMed] [Google Scholar]
  • 46.Hervey AS, Epstein JN, Curry JF, Tonev S, Eugene AL, Keith CC, Hinshaw SP, Swanson JM, Hechtman L. Reaction time distribution analysis of neuropsychological performance in an ADHD sample. Child Neuropsychol. 2006 Apr;12(2):125–40. doi: 10.1080/09297040500499081. [DOI] [PubMed] [Google Scholar]
  • 47.Gau SS, Chiang H. Association between early attention-deficit/hyperactivity symptoms and current verbal and visuo-spatial short-term memory. Res Dev Disabil. 2013 Jan;34(1):710–20. doi: 10.1016/j.ridd.2012.10.005. [DOI] [PubMed] [Google Scholar]
  • 48.Bitsakou P, Psychogiou L, Thompson M, Sonuga-Barke EJ. Delay Aversion in Attention Deficit/Hyperactivity Disorder: an empirical investigation of the broader phenotype. Neuropsychologia. 2009 Jan;47(2):446–56. doi: 10.1016/j.neuropsychologia.2008.09.015. [DOI] [PubMed] [Google Scholar]
  • 49.Zakay D. The evasive art of subjective time measurement: Some methodological dilemmas. In: Block RA, editor. Cognitive models of psychological time. Hillsdale: Lawrence Erlbaum Associates; 1990. pp. 59–84. [Google Scholar]

Articles from JMIR Serious Games are provided here courtesy of JMIR Publications Inc.

RESOURCES