The Groundskeeper Gaming Platform as a Diagnostic Tool for Attention-Deficit/Hyperactivity Disorder: Sensitivity, Specificity, and Relation to Other Measures

Abstract

Objective: The purpose of this study was to assess the relative accuracies of the Conners' Brief Rating Scale, Parent Version, the Conners' Continuous Performance Test II (CPT II), and a novel interactive game called “Groundskeeper” to discriminate child psychiatric patients with and without attention-deficit/hyperactivity disorder (ADHD).

Methods: We administered the three assessments to 113 clinically referred ADHD and non-ADHD patients who had been diagnosed with the Kiddie-Schedule of Affective Disorders and Schizophrenia- Present and Lifetime (K-SADS-PL), Version 19.

Results: As measured by the area under the curve (AUC) statistic from receiver operating characteristic (ROC) analysis, the diagnostic accuracy of Groundskeeper (0.79) was as high as the accuracy of the Conners' parent rating of inattention (0.76) and better than the CPT II percent correct (0.62). Combining the three tests produced an AUC of 0.87. Correlations among the three measures were small and, mostly, not significant.

Conclusions: Our finding of similar diagnostic accuracies between Groundskeeper and the Conners' inattention scale is especially remarkable given that the Conners' inattention scale shares method variance with the diagnostic process. Although our work is preliminary, it suggests that computer games may be useful in the diagnostic process. This provides an important direction for research, given the objectivity of such measures and the fact that computer games are well tolerated by youth.

Introduction

Attention-deficit/hyperactivity disorder (ADHD) is diagnosed by evaluating symptoms of hyperactivity, impulsivity, and inattention, and impaired functioning across settings. The diagnosis of ADHD shows considerable levels of concurrent and predictive validity in its clinical features, course, neurobiology, and treatment response (Faraone et al. 2000; Faraone 2005). The diagnosis has high diagnostic reliability, one of the highest in the Diagnostic and Statistical Manual of Mental Disorders, 5th ed. (DSM-5) (American Psychiatric Association 2013; Regier et al. 2013). Nevertheless, concerns about diagnostic accuracy persist. Some suggest that the use of subjective diagnostic procedures may lead to the overdiagnosis of ADHD in the community (Bruchmuller et al. 2012; Visser et al. 2014) (although see Sciutto and Eisenberg [2007] for a contrary view). The diagnosis has been called “subjective” because it relies on clinician evaluation of responses from patients, parents, and/or informants. Other studies have raised concerns about the underdiagnosis of ADHD (Express Scripts Lab 2014; Ginsberg et al. 2014).

In response to such concerns, researchers have sought to develop objective measures to diagnose ADHD or to monitor the course of ADHD symptoms during treatment. The first approach to objectifying ADHD medication response, and eventually diagnosis, was via parent and teacher rating scales. Although ratings scales rely on parent, teacher, or self-reports of symptoms, they evaluate these reports in the context of large, normative databases, which allows for a more data-driven approach to interpreting information obtained in assessment. Other research has examined peripheral biochemical markers as objective measures. Meta-analyses of these studies indicate that five measures differentiated ADHD and control patients (norepinephrine [NE], 3-Methoxy-4-hydroxyphenyl ethylene glycol [MHPG], monoamine oxidase [MAO], zinc, and cortisol) (Faraone, et al. 2014, Scassellati, et al. 2012). Moreover, NE, MHPG, MAO, β-phenylethylamine, and cortisol were responsive to ADHD medications. Meta-analysis also shows that peripheral measures of oxidative stress differ between ADHD and control participants (Joseph et al. 2015). Other approaches to biomarker development for ADHD have used neuropsychological (Ritsner 2009), electroencephalographic (EEG) (Snyder et al. 2015), structural imaging (Silk et al. 2009), and functional imaging (Bush et al. 2005) methods, often with the application of machine learning approaches to optimize diagnostic accuracy (Mueller et al. 2010).

Of particular relevance are continuous performance tests (CPTs), of which many are available (e.g., Corkum and Siegel 1993; Riccio and Reynolds 2001; Homack and Riccio 2006), the Quotient ADHD system (Sumner et al. 2010), which pairs a type of CPT with recording of motor activity, and the Neuropsychiatric EEG-Based Assessment Aid (NEBA) quantitative EEG assessment (Snyder et al. 2015). CPTs are frequently used in neuropsychological testing, but have less than optimal sensitivity and specificity for diagnosing ADHD. Quotient and NEBA are cleared by the United States Food and Drug Administration for augmenting clinical assessments (Dolgin 2014; Snyder et al. 2015); they are not cleared for diagnosing ADHD in the absence of a full clinical assessment. Therefore, these tests can be used to collect adjunct information but do not provide objective diagnoses of ADHD. For example, Snyder et al. (2015) showed that NEBA is useful for clarifying diagnoses when clinicians are uncertain if the patient truly has ADHD. However, because of these limitations, none of the abovementioned tests should be used to diagnose the disorder in the absence of a systematic and comprehensive clinical assessment.

Thome et al. (2012) presented the results of the task force on biological markers of the World Federation of ADHD. They used the following criteria to define a “useful” biomarker: Sensitivity >80%; specificity >80%; and that the putative biomarker is reliable, reproducible, inexpensive, noninvasive, easy to use, and has been confirmed by at least two independent studies. Putative biomarkers examined by Thome et al. were: EEG-based event-related potentials; neuropsychological measures of executive functioning, attention, memory, spatial abilities, and language; olfactory functioning, structural, and functional magnetic resonance imaging; transcranial sonography; genetics; peripheral metabolites; and proteomics. None of these markers met their criteria for utility.

To develop an objective diagnosis for ADHD, we are studying a novel interactive game called Groundskeeper, which captures the ADHD symptoms of impulsivity and inattention, as well as associated features: Motor coordination (Fliers et al. 2008, 2009, 2010), reaction time variability (Perry et al. 2010; Frazier-Wood et al. 2012), impaired temporal processing (Toplak et al. 2006), and impaired decision making (Drechsler et al. 2008). Building diagnostics into game playing has the advantage of offering patients a rewarding, engaging procedure that avoids confounds associated with boredom and random responding (Van der Oord et al. 2012; Dovis et al. 2015). In a preliminary study of 52 outpatients 6–17 years of age, Groundskeeper data predicted inattentive ADHD with a sensitivity of 76.9% and a specificity of 80.7% (Heller 1993). It predicted combined type ADHD with a sensitivity of 58.8% and a specificity of 82.8%. Groundskeeper was a more accurate predictor of gold standard clinician diagnoses than the parent report version of the Conners' Brief Rating Scale. The present study sought to further characterize the diagnostic accuracy of Groundskeeper and to examine its relationship and comparative predictive accuracy with parent ratings of ADHD symptoms and patient performance on a CPT.

Methods

Recruitment of participants

This was a cohort study in which participants were recruited as consecutive referrals. We did not attempt to enrich the participant pool with ADHD patients. Instead, we invited for participation consecutive patients referred to a child psychiatrist. From this cohort of patients, we formed two groups: Patients who had received an ADHD diagnosis and those who had not. Participants were children and adolescents between 6 and 17 years of age (n = 113, see Table 1), recruited from two outpatient psychiatric clinics.

Table 1.

Psychiatric Disorders in the ADHD and Control Groups

Disorder	n in control group	n in ADHD group
Anxiety disorders	45	33
Mood disorder	36	27
Disruptive behavior disorders	8	11
Autism spectrum disorder	2	6
Reading disability	10	23

The total number of disorders is more than the total number of participants because each participant can have more than one disorder.

ADHD, attention-deficit/hyperactivity disorder.

Study participants were administered the Kiddie-Schedule of Affective Disorders and Schizophrenia- Present and Lifetime (K-SADS-PL), Version 19, a semistructured diagnostic interview conducted by a psychiatric nurse trained by a child/adolescent psychiatrist at a community based clinic (κ = 1.0) and reviewed by two independent child and adolescent psychiatrists. Patients were eligible to participate if they met criteria for ADHD, major depressive disorder, dysthymia, generalized anxiety disorder, anxiety disorder not otherwise specified (NOS), social phobia, oppositional defiant disorder, panic disorder, or eating disorders on the K-SADS-PL. We excluded participants with a history of psychosis or neurological disorder, low intellectual functioning (i.e., child was not in a mainstream academic class), substance use disorders, conduct disorder, tic disorders, or physical impairments precluding game play.

Fourteen participants were taking stimulant medications: Dextroamphetamine (n = 1), methylphenidate extended release (ER) (n = 1), lisdexamfetamine (n = 3), transdermal methylphenidate (n = 3), mixed amphetamine salts (n = 2), and osmotic controlled-release oral delivery system (OROS) methylphenidate (n = 4). These were withheld on the days of testing for Groundskeeper and Conners' Continuous Performance Test II (CPT II) but were not otherwise washed out. Eighty participants were on at least one nonstimulant, psychiatric medication: Citalopram (n = 12), bupropion (n = 8), aripiprazole (n = 16), quetiapine (n = 12), buspirone (n = 6), mirtazapine (n = 4), trazodone (n = 11), clonidine (n = 2), duloxetine (n = 1), venlafaxine (n = 3), guanfacine immediate release (IR) (n = 2), guanfacine extended release (XR) (n = 8), escitalopram (n = 2), fluvoxamine (n = 1), paroxetine (n = 3), atomoxetine HCl (n = 1), sertraline (n = 7), fluoxetine (n = 8), risperidone (n = 4), propranolol (n = 1), lamotrigine (n = 1), olanzapine (n = 2), and topiramate (n = 2). Twenty-nine percent (n = 33) of patients were medication free at study initiation. Among the 66 patients diagnosed with ADHD, 25 (38%) were on a medication not typically used to treat ADHD (i.e., medications other than stimulants, bupropion, clonidine, guanfacine, and atomoxetine). Four subjects were on both stimulant and nonstimulant medications.

Institutional review board approval was obtained from the University of Minnesota. Written permission and assent were obtained from one parent and each participant, respectively. Procedures were in accordance with the ethical standards of the responsible committee on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. For eligible families, a parent completed the Conners' Brief Rating Scale, Parent Version using the past month as the time frame for reporting. The scale was completed once, and we extracted five subscales for analysis: Hyperactivity/impulsivity, inattention, learning problems, aggression, and executive functions. The child played the Groundskeeper game and was administered the CPT on separate occasions within 1 week of each other.

Continuous Performance Task

The CPT II was administered by a technician who remained in the room while the test was completed. Following the standard protocol, after a practice session, the actual testing session began. CPT II respondents were asked to press the space bar whenever any letter except the letter “X” appeared on the computer screen. The inter-stimulus intervals (ISIs) were 1, 2, and 4 seconds, with a display time of 250 milliseconds. The CPT II comprises six blocks and three sub-blocks, each containing 20 trials. The presentation order of ISIs varied among blocks. For our analyses, we used the percent certainty that the CPT II results were in the clinical range.

Groundskeeper game

Groundskeeper (Fig. 1) is played using four cubes and a placement board. One cube is used as a “mallet” to hit targets that appear on other cubes. The mallet must be moved by the player. The other cubes are placed in a straight vertical line. These three cubes have an image of green grass and blue sky as a backdrop. Images of a rabbit, a “groundskeeper” (man with a lawn mower), a gopher, or a few small birds appear on these screens for 1, 1.5, or 3 seconds at random. The object of the game is to touch another cube when the gopher image appears. Successful hits are associated with a “bonk” noise. The other images are distractors to be avoided. Each of the 17 game sessions is 90 seconds long, with a 20 second interval in between each session. Game playing instructions are administered via a script read to each participant (see Appendix A). Summaries of each game session are in Appendix B. Variables derived from Groundskeeper are in Appendix C.

FIG. 1.

Sifteo cubes used for Groundskeeper.

The Groundskeeper protocol consists of 17 game sessions, numbered 0 through 16, each with different levels and types of distractions: Visual, auditory, and spatial. Low visual distraction consists of a bird appearing on the cube screens. High visual distraction adds large rabbits. Low auditory distraction consists of occasional tweeting noises; high auditory distraction increases tweet frequency. When there is no spatial distraction, the image cubes are in a vertical line. In low spatial distraction, they are set diagonally 5.08 cm apart (Fig. 1). High spatial distraction consists of each cube put 7.62 cm apart. Sessions 0, 1, and 16 have no distraction. Each session is 90 seconds long, consisting of a randomized number of trials and frequency of target stimulus presentations.

The game was designed to measure attentional capabilities on a go/no go task, with the addition of visual, auditory, and visuospatial distractions at various frequencies. Given that we were implementing multiple levels to the game, each level was restricted to 90 seconds in order to keep the full game time at a reasonable time length, while giving time between levels to arrange the cubes in a specific configuration if necessary. The first and the last levels were meant to be a baseline for comparison and to measure any effect of “learning” on performance. Next, we implemented visual distractors of birds and a groundskeeper to test the ability of a patient to alter their go/no go response when additional figures were presented. First, the visual distractors were presented at “low frequency,” meaning only the Groundskeeper was introduced as a distractor, in addition to the gopher, and then at a high frequency when a bird, Groundskeeper, and rabbit were presented. A rabbit was chosen because it most closely resembled the gopher pictorially.

Next we added auditory distractors of a bird tweeting at a low frequency (one tweet at various intervals) and then at a high frequency (multiple tweets at various intervals). Lastly, we spread the cubes out diagonally in an attempt to introduce a visuospatial element, thus adding a new distractor and trying to eliminate any habituation a person may have had to the cubes being at a vertical configuration for various levels. We then added low frequency distractors (visual and auditory), to see if the effect of these distractors while we had a visuospatial element had the same effect.

Statistical analyses

For Groundskeeper, we used logistic regression to assess the significance of factors derived from principal factors factor analysis as predictors of the ADHD diagnosis. We retained factors that significantly predicted ADHD at the Bonferroni-corrected α level of 0.0023 (i.e., 0.05/22). We also used logistic regression to assess the ability of the parent-rated Conners' subscales to predict ADHD diagnoses. For this model, the Bonferroni-corrected α level for selecting significant predictors was 0.01.

Receiver operating characteristic (ROC) curve analysis determined the accuracy of predictions. ROC analysis assesses the diagnostic efficiency of tests for diagnoses and allows for adjustment of cut-points for clinical or research purposes (McNeil and Hanley 1984); this approach has been widely applied to assess the accuracy of diagnostic tests across multiple disorders (Swets 1982; Swets and Pickett 1982; Swets 1986a,b). For each participant, we computed the predicted values, or logits, from the logistic regression models. For each successive point on the logit scale we computed a sensitivity and specificity of the logit as a predictor of ADHD diagnosis, by predicting those higher than the cut-point to have ADHD and the others not to have ADHD. These data were used to draw the ROC curve. ROC analysis summarizes diagnostic efficiency with the area under the curve (AUC) statistic. The AUC ranges from 0.5 (for a diagnostically useless test) to 1.0 (for a diagnostic test that is a perfect predictor). All analyses used STATA 13.1.

Results

The ADHD (n = 66) and psychiatric control groups (n = 47) did not differ significantly in sex (57% vs. 38% male, respectively; χ²[1] = 3.6, p = 0.06) or ethnicity (88% vs. 82% Caucasian, respectively; χ ²(3) = 4.0, p = 0.3). They differed significantly in age (12.3 vs. 13.6; t[105] = 2.5, p = 0.01), which was used as a covariate along with sex (marginally significant) in subsequent analyses. Although no participants were taking ADHD medications at the time of testing, we included medication status (yes/no) as a covariate, because the non-ADHD group was significantly more likely to be on other medications at the time of testing compared with the ADHD group (82% vs. 60%; χ ²[1] = 6.2, p = 0.01). Table 1 gives the distribution of psychiatric disorders.

Principal factors factor analysis with varimax rotation reduced the 106 Groundskeeper variables to 22 principal factors having eigenvalues >1.0. These factors accounted for 68% of the variance of Groundskeeper scores. We entered these factors into a logistic regression model to predict the “gold standard” K-SADS diagnoses of ADHD. The eighth (z = 3.7, p < 0.001) and tenth (z = 3.1, p = 0.002) factors remained statistically significant after correcting for age, sex, and medication status. The area under the ROC curve (AUC) was 0.78 (Fig. 2; χ²[2] = 28, p < 0.0001). Neither factor interacted with age or sex in predicting ADHD diagnoses (p's > 0.10). We used the rotated factor loadings to determine which Groundskeeper scores accounted for the two significant factors. Table 2 shows the scores with loadings >0.25 on one or both of the factors. Boldface highlights the highest loadings. When these raw scores were used to predict the K-SADS ADHD diagnosis, the AUC was 0.79 (χ²[12] = 29.0, p = 0.003).

FIG. 2.

Receiver operating characteristic curves.

Table 2.

Groundskeeper Variables Accounting for Factors 8 and 10

Variable	Factor 8	Factor 10
Movement, incorrect	−0.3002	0.0229
Omissions	0.2579	−0.0475
Reaction time, correct	0.3403	0.1154
Reaction rate, correct	0.1249	0.2645
Reaction rate, correct, 2–7	0.7368	0.1026
Reaction rate, correct, 8–11	0.0490	0.7961
Reaction rate, Incorrect, 2–7	0.3729	−0.0655
Incorrect, S, V, A distractors	0.2566	−0.0722
Performance	−0.2755	−0.4649
Reaction time, correct, 2–7	0.7544	0.1804
Reaction time, Correct, 8–11	0.2150	0.6856
Reaction time, Incorrect, 8–11	−0.0418	0.3437

Movement incorrect: Average amount of tilt/movement during incorrect response.

Omissions: No response when target stimulus presented.

Reaction time, correct: Average reaction time from point image is shown to response (correct).

Response rate, correct: Ratio of time to response divided by total time image is shown.

Correct reaction rate, 2–7: Average of correct reaction rate levels 2–7.

Correct reaction rate, 8–11: Average of correct reaction rate levels 8–11.

Incorrect reaction rate, 2–7: Average of incorrect reaction rate levels 2–7.

Incorrect, S, V, A distractors: Incorrect, effect of spatial distractor with low frequency visual and auditory distractors.

Performance: Difference in reaction time between first and last levels demonstrating change in performance.

Reaction time, correct, 2–7: Average of correct reaction time levels 2–7.

Reaction time, correct, 8–11: Average of correct reaction time levels 8–11.

Reaction time, incorrect, 8–11: Average of incorrect reaction time levels 8–11.

Boldface indicates the highest factor loadings.

For the logistic regression analysis of the parent-rated Conners' subscales as a predictor of ADHD diagnoses, only the inattention scale was significant (z = 3.25, p = 0.001) after controlling for age, sex, and medication status. The AUC was 0.76 (Fig. 2; χ²[1] = 8.1, p = 0.004). This did not differ significantly from the Groundskeeper AUC (χ²[1] = 0.1, p = 0.8). The subscales did not interact with age or sex in predicting ADHD diagnoses (p's > 0.10). In another logistic regression model, the CPT percent correct score significantly predicted K-SADS ADHD diagnoses after controlling for age, sex, and medication status. The AUC was 0.62 (Fig. 2; χ²[1] = 5.0, p = 0.03). This was significantly lower than the Groundskeeper AUC (χ²[1] = 4.6, p = 0.03) and the CPT AUC (χ²[1] = 5.8, p = 0.02). When we combined the significant Groundskeeper factors with the Conners' inattention subscale and the CPT percent correct in the same model, all terms remained significant after controlling for age, sex, and medication status (all p's < 0.04) and the AUC was 0.87 (Fig. 2; χ²[5] = 49.2, p < 0.0001). This AUC was significantly greater than the CPT AUC (χ²[1] = 15.0, p = 0.0001), but did not differ significantly from either the Groundskeeper (χ²[1] = 0.5, p = 0.5) or Conners AUCs (χ²[1] = 1.7, p = 0.19).

The correlations between Groundskeeper factor eight and the Conners' scores ranged from −0.02 to 0.11. None were statistically significant (all p's > 0.05). The correlation between factor eight and the CPT percent correct score was 0.13 (p = 0.18). The correlations between Groundskeeper factor ten and the Conners' scores ranged from −0.28 to 0.001. Only the correlations with the Executive Functioning score (r = −0.21, p = 0.03) and the Aggression subscale (r = −0.28, p = 0.003) were statistically significant. The correlation between factor ten and the CPT percent correct score was also significant (r = −0.30, p = 0.002).

To clarify the degree to which the three logistic models based on the Groundskeeper, Conners', and CPT identified the same cases, we used each of the corresponding logistic regression models to compute the probability of each participant having K-SADS-diagnosed ADHD. We took a median split of these predicted probabilities and classified participants above the median as ADHD and those below the median as not ADHD. Only 14% of participants were predicted to have ADHD by the three methods. In this group, 79% were diagnosed with ADHD. Only 18% of participants were predicted to not have ADHD by all three models. In this group, 16% were diagnosed with ADHD. The κ coefficients of agreement were 0.15 for Groundskeeper versus Conners' (z = 1.6, p = 0.06), 0.18 for Groundskeeper versus CPT (z = 1.9, p = 0.9), and 0.3 for Conners versus CPT (z = 3.2, p = 0.0007).

Discussion

We found that the Groundskeeper game can significantly discriminate ADHD patients from other psychiatric patients. As measured by the AUC statistic from ROC analysis, the diagnostic accuracy of Groundskeeper (0.79) was as high as the accuracy of the Conners' parent rating scale (0.76), which is used as a screening or adjunct diagnostic tool for the diagnosis of ADHD.

The Conners' and Groundskeeper models had similar levels of diagnostic accuracy. Both predicted ADHD diagnoses more accurately than the CPT. The similar diagnostic accuracies between Groundskeeper and the Conners is remarkable, given that the questions asked of parents during the diagnostic interview are similar to the questions asked of the parent by the Conners' form. Both require subjective reports of ADHD symptoms and share method variance. In contrast, Groundskeeper is performed by the child with no parent involvement. Therefore, it is encouraging that an objective measure (Groundskeeper) is as accurate as subjective assessments of symptom criteria.

Although the AUCs for Groundskeeper and the Conners' rating scales were of similar magnitude, Figure 2 shows that their tradeoffs between sensitivity and specificity differ in clinically important ways. Groundskeeper maintains a false positive rate of zero for a sensitivity of 37%, a positive predictive power of 100%, and a negative predictive power of 53%. For the Conners' to achieve a sensitivity of 37%, the false positive rate would rise to 9.1%, the positive predictive power would be 85%, and the negative predictive power would be 51%. In Figure 2, this difference is seen as the Groundskeeper ROC being skewed toward the left side of the graph, whereas the Conners' ROC is spread more evenly throughout the graph. This means that the Conners' will be a better test for ADHD if a higher false positive rate can be tolerated. For example, the graphs show that, at a false positive rate of 25%, the Conners' has a sensitivity of 75%, a positive predictive power of 81%, and a negative predictive power of 69%; Groundskeeper has a sensitivity of 65%, a positive predictive power of 75%, and a negative predictive power of 65%. For screening tests, a high false positive rate can be tolerated if the costs of a second stage confirmation test are low, but if the costs are high, then the low false positive rate of Groundskeeper is preferred. A low false positive rate is essential for clinicians seeking to confirm an ADHD diagnosis about which they are uncertain, suggesting that Groundskeeper could be used for diagnostic confirmation. For example, in settings such as college health clinics, where the misuse and diversion of ADHD medications is a major concern, having a means of eliminating false positives would be very important. These examples are only illustrative. It would be premature to suggest specific cut-points for clinical practice. Groundskeeper should not be used to diagnose ADHD independently from a clinical diagnosis.

The correlations among the Groundskeeper, Conners', and CPT scores were mostly low and not significant. Consistent with this, we found low kappa coefficients of agreement between each model's predictions of ADHD diagnoses. This lack of shared variance is probably because of the unique method variance for each method, the imperfect reliability of each method, and the possibility that each is sensitive to different components of the ADHD syndrome. Consistent with this latter interpretation, each score domain remained significant when all were included in the same model. Future work should address the possibility that multimodal assessments of ADHD are needed to create a highly accurate objective diagnostic tool.

Our work has limitations. We used a medicated, psychiatric control group, which we presumed would be relatively difficult to differentiate from ADHD compared with healthy controls. Use of the latter would likely lead to better diagnostic accuracy statistics. Because this was a preliminary study, we used a wide age range so that we could examine age effects. Although we found no effects of age on diagnostic accuracy, our sample was too small to detect small effects. The sample was also small relative to the number of variables analyzed. This required us to use factor analysis to limit our statistical tests. Because most research participants were taking medications of many different types, we cannot rule out medication effects completely and cannot be certain that our results will generalize to samples with a different profile of medication use. We also excluded youth with conduct disorder and tics, which further reduces the generalizability of our findings. Because the CPT and Groundskeeper tests were not counterbalanced, results could have been biased by sequence effects. Moreover, Groundskeeper benefited from the use of multiple variables derived from the test whereas for the CPT, we only used the percent correct, a commonly used index for the version we used. Our results may not generalize to other CPTs or to more complex analyses of CPT data. For these reasons, our results must be considered tentative until cross validated in an independent sample. We do not know if our results will generalize to adults with ADHD or if Groundskeeper will be useful for studying treatment effects over time. Future work must address these issues.

Conclusions

Despite these limitations, our results are encouraging. We demonstrated good discriminative ability when comparing ADHD patients with other psychiatric patients. The discriminative ability of Groundskeeper will likely be much better for discriminating ADHD patients from subjects not having psychiatric disorders. Future work should examine this possibility and should also determine if a revision to Groundskeeper or the application of different algorithms will be able to improve its diagnostic accuracy.

Clinical Significance

Although Groundskeeper is not ready to be used as a diagnostic tool, this work foreshadows to clinicians a new genre of clinical tool that is in development. For a diagnostic tool to be useful, it must be accepted by patients and must be engaging enough to assure that valid data are obtained. Clinicians also need to be aware that the ability of a tool to be successful depends on the diagnostic context. We have shown here that contrasting ADHD with other disorders is a difficult task, which suggests that future uses of Groundskeeper address a different diagnostic issue.

Appendix A

This script is read to each participant prior to game administration.

Today you will be playing a game called Groundskeeper. It will be used to measure your attention. The goal is to move the mallet cube to the left or right side of the cube showing an image of a gopher. When you have a correct hit, you will hear a “boink” noise. There will be no noise for an incorrect hit. Keep in mind you only want to hit the gopher! Do not let other images, like birds or rabbits, or noises distract you from this task.

There will be a short pause between each level. The first level that you play is for practice and will not be counted against your final score. After each level, a voice will instruct you to move the cubes to colored dots on the game board. Note that the mallet cube should not be moved to these dots.

Be sure to use two hands and hold the cubes together until you hear a sound. If you do not hear a boink sound, you do not get a point. Move as quickly as possible!

Appendix B

Summary of Groundskeeper Sessions

Each session is 1.5 minutes with 20 seconds between sessions. There will be a 10 second countdown on the screen prior to each session. Program will automatically progress to the next session without intervention by person conducting the test. Total run time is 24 minutes (plus 5 minutes for pauses).

Session 1: Screen shot of gopher, groundskeeper, or grass (neutral) presented for 1, 2, or 3 seconds at a random frequency.

Go/no go task.

Session 2: Gopher, groundskeeper, or grass screen shot with visual disturbance at a low degree (one bird showing up on screen) alternating with screen shot of grass presented for 1, 2, or 3 seconds at random frequency.

Visual disturbance, low frequency.

Session 3: Gopher, groundskeeper, or grass screenshot with visual disturbance at a high degree (bird and rabbit) alternating with screen shot of grass presented for 1, 2, or 3 seconds at random frequency.

Visual disturbance, high frequency.

Session 4: Screen shot of gopher, groundskeeper, or grass screen shot presented for 1, 2, or 3 seconds at a random frequency with auditory disturbance at a low degree (one bird chirping) occurring at random frequencies for 1, 2, or 3 seconds, not in concert with screen shot frequency.

Auditory disturbance, low frequency.

Session 5: Screen shot of gopher, groundskeeper, or grass screenshot presented for 1, 2, or 3 seconds at a random frequency with auditory disturbance at a high degree (multiple birds chirping) occurring at random frequencies for 1, 2, or 3 seconds, not in concert with screen shot frequency.

Auditory disturbance, high frequency.

Session 6: Gopher, groundskeeper, or grass screen shot with visual and auditory disturbances at a low degree (one bird chirping) occurring at random frequencies for 1, 2, or 3 seconds.

Visual and auditory disturbance, low frequency.

Session 7: Gopher, groundskeeper, or grass screen shot with visual and auditory disturbances at a high degree (bird and rabbit and chirping) occurring at random frequencies for 1, 2, or 3 seconds.

Visual and auditory disturbance, high frequency.

Session 8: Gopher, groundskeeper, or grass screen shot with spatial disturbance at a low degree. Cube set diagonally. Spaced at 5.08 cm apart and occurring at random frequencies for 1, 2, or 3 seconds.

Spatial disturbance, low frequency.

Session 9: Gopher, groundskeeper, or grass screen shot with spatial disturbance at a low degree combined with low frequency visual disturbance. Cube set diagonally. Spaced at 5.08 cm apart and occurring at random frequencies for 1, 2, or 3 seconds.

Spatial disturbance, low frequency, low frequency visual disturbance.

Session 10: Gopher, groundskeeper, or grass screen shot with spatial disturbance at a low degree combined with low frequency auditory disturbance. Cube set diagonally. Spaced at 5.08 cm apart and occurring at random frequencies for 1, 2, or 3 seconds.

Spatial disturbance, low frequency, low frequency auditory disturbance.

Session 11: Gopher, groundskeeper, or grass screen shot with spatial disturbance at a low degree combined with low frequency visual and auditory disturbance. Cube set diagonally. Spaced at 5.08 cm apart and occurring at random frequencies for 1, 2, or 3 seconds.

Spatial disturbance, low frequency, low frequency visual and auditory disturbance.

Session 12: Gopher, groundskeeper, or grass screen shot with spatial disturbance at a low degree. Cube set diagonally. Spaced at 7.62 cm apart and occurring at random frequencies for 1, 2, or 3 seconds.

Spatial disturbance, high frequency.

Session 13: Gopher, groundskeeper, or grass screen shot with spatial disturbance at a high degree combined with low frequency visual disturbance. Cube set diagonally. Spaced at 7.62 cm apart and occurring at random frequencies for 1, 2, or 3 seconds.

Spatial disturbance, high frequency, low frequency visual disturbance.

Session 14: Gopher, groundskeeper, or grass screen shot with spatial disturbance at a high degree combined with low frequency auditory disturbance. Cube set diagonally. Spaced at 7.62 cm apart and occurring at random frequencies for 1, 2, or 3 seconds.

Spatial disturbance, high frequency, low frequency auditory disturbance.

Session 15: Gopher, groundskeeper, or grass screen shot with spatial disturbance at a high degree combined with low frequency visual and auditory disturbance. Cube set diagonally. Spaced at 7.62 cm apart and occurring at random frequencies for 1, 2, or 3 seconds.

Spatial disturbance, high frequency, low frequency visual and auditory disturbance.

Session 16: Screen shot of gopher, groundskeeper, or grass (neutral) presented for 1, 2, or 3 seconds at a random frequency.

Go/no go task with learning curve.

Appendix C:

Variables Derived from Groundskeeper

Variable	Formula	Level	Category	Description
eval_id	NA	All	Identifier	Unique ID for each transformed evaluation
mCRR		All	Correct	Ratio of correct to incorrect responses
mIRR		All	Incorrect	Ratio of incorrect to correct responses
mEOM		All	Omission	Ratio of omissions to both correct and incorrect responses
TiltX	TiltX = TiltLeft+TiltRight	All	Movement	Total movement left/right as denoted by an acceleration state change
TiltY	TiltY = TiltUp+TiltDown	All	Movement	Total movement up/down as denoted by an acceleration state change
TiltZ	m = TiltFlip, n = TiltFlipBack	All	Movement	Total movement diagonal as denoted by an acceleration state change
Movement	TiltX+TiltY+TiltZ	All	Movement	Composite variable totaling accelerometer state changes
VirtualTicks	NA		Counter	Measureable time unit recorded as ticks, which operates within the state machine
correct		All	Correct	Normalization of correct response, whereby the participant has hit the gopher
incorrect		All	Incorrect	Normalization of incorrect response, whereby the participant has hit something other than the gopher
error_omission		All	Omission	Normalization of omission response, whereby the participant missed the gopher and hit nothing
correct_reaction		All	Reaction time	Average reaction time from point image is shown to correct response
incorrect_reaction		All	Reaction time	Average reaction time from point image is shown to incorrect response
offtilt_reaction		All	Movement	Average movement from point in time cubes are neighbored until mallet cube removed
ontilt_reaction		All	Movement	Average movement from point in time gopher image is displayed until neighbored with mallet
neighbor_reaction		All	Reaction time	Average amount of time cubes held together
correct_reaction_movement		All	Movement	Average amount of tilt during a correct response
incorrect_reaction_movement		All	Movement	Average amount of tilt during an incorrect response
omission_movement		All	Movement	Average amount of tilt during an omission
correct_speed_variance		All	Movement	Average amount of speed during a correct response
incorrect_speed_variance		All	Movement	Average amount of speed during an incorrect response
mCRRate2_7		2–7	Correct	Ratio of correct to incorrect responses
mICRRate2_7		2–7	Incorrect	Ratio of incorrect to correct responses
mNR2_7		2–7	Reaction time	Average of neighbor reaction
mCT2_7	TiltX+TiltY+TiltZ	2–7	Movement	Sum of all movement
mCR2_7		2–7	Reaction time	Average of correct reaction
mICR2_7		2–7	Reaction time	Average of incorrect reaction
mOTR2_7		2–7	Movement	Average of offtilt_reaction
mONTR2_7		2–7	Movement	Average of ontilt_reaction
mMMO2_7	TiltX+TiltY+TiltZ	2–7	Movement	Average mallet movement
mAOM2_7	TiltX+TiltY+TiltZ	2–7	Movement	Average all other movement
mCRRate8_11		8–11	Correct	Ratio of correct responses to incorrect
mICRRate8_11		8–11	Incorrect	Ratio of incorrect responses to correct
mNR8_11		8–11	Reaction time	Average of neighbor reaction
mCT8_11	TiltX+TiltY+TiltZ	8–11	Movement	Sum of all movement
mCR8_11		8–11	Reaction time	Average of correct reaction
mICR8_11		8–11	Reaction time	Average of incorrect reaction
mOTR8_11		8–11	Movement	Average of offtilt_reaction
mONTR8_11		8–11	Movement	Average of ontilt_reaction
mMMO8_11	TiltX+TiltY+TiltZ	8–11	Movement	Amount of mallet movement
mAOM8_11	TiltX+TiltY+TiltZ	8–11	Movement	Amount of all other movement excluding the mallet
mCRRate12_15		12–15	Correct	Ratio of correct to incorrect responses
mICRRate12_15		12–15	Incorrect	Ratio of incorrect to correct responses
mNR12_15		12–15	Reaction time	Average of neighbor reaction
mCT12_15	TiltX+TiltY+TiltZ	12–15	Movement	Sum of all movement
mCR12_15		12–15	Reaction time	Average of correct reaction
mICR12_15		12–15	Reaction time	Average of incorrect reaction
mOTR12_15		12–15	Movement	Average of offtilt_reaction
mONTR12_15		12–15	Movement	Average of ontilt_reaction
mMMO12_15	TiltX+TiltY+TiltZ	12–15	Movement	Amount of mallet movement
mAOM12_15	TiltX+TiltY+TiltZ	12–15	Movement	Amount of all other movement excluding the mallet
mMRATIO13_15		13–15	Movement	Ratio of mallet movement to all other cube movement
mMRATIO12_15		12–15	Movement	Ratio of mallet movement to all other cube movement
VSC_VFC		2–3	Distraction	Low/high visual distraction success/failure comparison
VSC_VFC_O		2–3	Distraction	Low/high visual distraction omission/correct comparison
ASC_AFC		4–5	Distraction	Low/high auditory distraction correct/failure comparison
ASC_AFC_O		4–5	Distraction	Low/high auditory distraction omission/correct comparison
LA2TSC_LA2TFC_HA2TSC_HA2TFC		2,3,6,7	Distraction	Low/high audio and low/high visual to total success/failure comparison
LA2TSC_LA2TFC_HA2TSC_HA2TFC_O		2,3,6,7	Distraction	Low/high audio and low/high visual to total correct/omission comparison
CVSC_CVFC		2,3,6,7	Distraction	Low visual to low/high visual and audio to total success/failure comparison
CVSC_CVFC_O		2,3,6,7	Distraction	Low visual to low/high visual and audio distraction to total omissions/correct comparison
CASC_CAFC		4,5,6,7	Distraction	Low audio to low/high visual and audio distraction success/failure comparison
CASC_CAFC_O		4,5,6,7	Distraction	Low audio to low/high visual and audio distraction omission/correct comparison
SSC_SFC		6,11	Distraction	Low spatial distraction success/failure comparison during low frequency visual and auditory disturbance
SSC_SFC_O		6,11	Distraction	Low spatial distraction omission/success comparison during low frequency visual and auditory disturbance
SSC_SFC_NR		6,11	Distraction	Low spatial distraction of neighbor reaction comparison during low frequency visual and auditory disturbance
SSC_SFC_CR		6,11	Distraction	Low spatial distraction of correct response comparison during low frequency visual and auditory disturbance
SSC_SFC_ICR		6,11	Distraction	Low spatial distraction of incorrect response comparison during low frequency visual and auditory disturbance
CSSC		11,15	Distraction	Low spatial, visual, auditory distraction correct/incorrect comparison to high spatial, visual, auditory distraction
CSSC_O		11,15	Distraction	Low spatial, visual, auditory distraction omission/correct comparison to high spatial, visual, auditory distraction
LearningCurve		1,16	Learning curve	Measure correct responses from first level compared with last level
LearningCurve_O		1,16	Learning curve	Measure omissions from first level compared with last level
LearningCurve_CT		1,16	Learning curve	Measure movement amount from first level compared with last level
LearningCurve_NR		1,16	Learning curve	Measure neighbor reaction amount from first level to last level
LearningCurve_CR		1,16	Learning curve	Measure correct reaction time from first level to last level
LearningCurve_ICR		1,16	Learning curve	Measure incorrect reaction time from first level to last level
mallet_movement_only	TiltX+TiltY+TiltZ	All	Movement	Total amount of mallet movement
all_other_movement	TiltX+TiltY+TiltZ	All	Movement	Total of all other movement excluding the mallet
subsecond_response_total	K	All	Reaction time	Total number of times when time to respond is ≤1 tick (1 tick = 10 Hz)
topbottomhit	1 or 0	All	Auxiliary movement	Total for device hits from top or bottom. Measured by response to top or bottom of device.
cubepressed	1 or 0	All	Auxiliary movement	Total number of times player pushes or presses cube screen - no real reason why player should do this
doublehit	K	All	Auxiliary movement	Total number of times cubes are neighbored together after a response has already been made
Gender	NA	NA	Demographic	M or F assignment
Age	NA	NA	Demographic	Discrete assignment of age
Inattention	NA	NA	Condition	Diagnosis made by clinicians
Hyperactivity	NA	NA	Condition	Diagnosis made by clinicians
Combined	NA	NA	Condition	Inattention and hyperactivity
Depression	NA	NA	Condition	Diagnosis made by clinicians
Autism	NA	NA	Condition	Diagnosis made by clinicians
Anxiety	NA	NA	Condition	Diagnosis made by clinicians

Disclosures

In the past year, Dr. Faraone received income, potential income, travel expenses, and/or research support from Akili Interactive Labs, Alcobra, Arbor, CogCubed, Impax, Ironshore, NeuroLifeSciences, Neurovance, Pfizer, Shire, and VAYA Pharma. With his institution, he has United States patent US20130217707 A1 for the use of sodium-hydrogen exchange inhibitors in the treatment of ADHD. In previous years, he received income or research support from Alcobra, Eli Lilly, Janssen, McNeil, Novartis, Otsuka, Pfizer, and Shire. Dr. Faraone receives royalties from the following books published by Guilford Press: Straight Talk about Your Child's Mental Health; Oxford University Press: Schizophrenia: The Facts; and Elsevier: ADHD: Non-Pharmacologic Interventions. In the past year, Dr. Newcorn is/has been an advisor/consultant to Alcobra, Biobehavioral Diagnostics, Ironshore, Neos, the National Football League (NFL), Rhodes, and Shire. He receives research support from Enzymotec and Shire, and serves on a data and safety monitoring board (DSMB) for Sunovion. In the previous 2 years he was also an advisor/consultant to GencoSciences, Lupin, and Neurovance. Dr. Heller is the co-founder and chief medical officer for CogCubed. Kurt Roots is the co-founder and chief executive officer of CogCubed. During the past 3 years, Dr. Adler has received grant support from the APSARD/Pound Foundation, Department of Veterans Affairs, Eli Lilly and Company, Enzymotec, Purdue Pharmaceuticals, Shire Pharmaceuticals, Sunovian Pharmaceuticals, and Theravance. He was also a consultant to Alcobra Pharmaceuticals, Enzymotec, Major League Baseball, NFL, Novartis Bioventures, Shire Pharmaceuticals, Sunovian Pharmaceuticals, and Theravance. He has received royalty payments (as inventor) from NYU for the license of adult ADHD scales and training materials since 2004. Dr. Antshel does not have any conflicts of interest.