Evaluating Vigilance Deficits in ADHD: A Meta-Analysis of CPT Performance

Abstract

We meta-analytically review 47 between-groups studies of continuous performance test (CPT) performance in children with attention-deficit/hyperactivity disorder (ADHD). Using a random effects model and correcting for both sampling error and measurement unreliability, we found large effect sizes (δ) for overall performance, but only small to moderate δ for performance over time in the handful of studies that reported that data. Smaller δs for performance over time are likely attributable, in part, to the extensive use of stimuli for which targets and distractors are quite easily differentiated. Artifacts accounted for a considerable proportion of variance among observed δs. Effect sizes reported in previous reviews were significantly attenuated because of the presence of uncorrected artifacts and highlight the necessity of accounting for artifactual variance in future work to determine the amount of true neurocognitive heterogeneity within ADHD. Signal detection theory and diffusion modeling analyses indicated that the ADHD-related deficits were because of decreased perceptual sensitivity (d’) and slower drift rates (v). Results are interpreted the context of several recent models of ADHD.

Interest in sustained attention as a cognitive process began during World War II, when it was observed that the ability of radar observers to detect enemy signals declined with the amount of time on task (Mackworth, 1948). Since Mackworth’s first description of the “vigilance decrement,” continuous performance tests (CPTs) have been among the most popular measures used to study sustained attention, which refers to the ability to maintain a tonic state of alertness over an extended period of time. Extensive variations of the paradigm exist (Riccio, Reynolds, Lowe, & Moore, 2002), but most require participants to detect a rare target among rapidly presented nontargets over the course of 10–30 min. In clinical research, CPTs have been used to study how attention is modulated by medication/substances (Koelega, 1993, 1995a; Riccio, Waldrop, Reynolds, & Lowe, 2001) and whether attention is disturbed in individuals with a range of neurologic and psychiatric conditions including the following: Alzheimers’ (Perry & Hodges, 1999), neurologic insult (Riccio et al., 2002), schizophrenia (Nieuwenstein, Aleman, & de Haan, 2001), bipolar disorder (Robinson et al., 2006), and childhood attention-deficit/hyperactivity disorder (ADHD; Corkum & Siegel, 1993; Losier, McGrath, & Klein, 1996; Nigg, 2005; Pennington & Ozonoff, 1996; Sonuga-Barke, Sergeant, Nigg, & Willcutt, 2008; Willcutt, Doyle, Nigg, Faraone, & Pennington, 2005).

The degree of interest in the ability of CPTs to assist in the search for cognitive mechanisms of neuropsychiatric disorders, and particularly ADHD, is unsurprising. Maintaining attentional focus is required to be successful in a range of daily activities, from learning fundamental academic skills in the classroom, to playing duck-duck-goose in the schoolyard. Across multiple settings, children with ADHD have difficulty sustaining attention to effortful tasks, are inattentive to environmental cues, and have concurrent impairments in academic and peer settings. Thus, identifying the mechanisms that are responsible for the regulation and maintenance of sustained attention and effort allocation have been key components of influential theories of ADHD (Douglas, 1999; Sergeant, Oosterlaan, & van der Meere, 1999b). Brain imaging data have consistently pointed to the role of the right fronto-parietal network in mediating sustained attention processes (Cabeza & Nyberg, 2000; Sarter, Givens, & Bruno, 2001), structures that have also been implicated in the disorder (Valera, Faraone, Murray, & Seidman, 2007). Despite this body of literature, there are several interrelated reasons why the presence or absence of a sustained attention deficit in children with ADHD remains unresolved.

The first is that the overall number of commission and omission errors, as well as the signal detection theory (SDT) variables d’ (referring to perceptual sensitivity, or the ease with which targets can be distinguished from nontargets) and β (referring to response bias, or whether the individual is biased toward responding “yes target present,” vs. “no, target absent”) that are derived from error rates, are the most commonly reported and examined dependent variables in studies of sustained attention in ADHD. However, such indices can only speak to overall levels performance. They are unable to address performance over time (POT), which is the defining feature of a sustained attention process, and which is ironically much less frequently reported. Because performance is expected to degrade with time, a sustained attention deficit could only be claimed if significant Diagnosis × Time interaction effects were found in which error rates, reaction time (RT), or SD of RTs (SDRT) increased over time to a greater extent for children with ADHD compared with controls (Huang-Pollock, Nigg, & Halperin, 2006; Sergeant et al., 1999b). Alternative definitions of sustained attention deficits exist that redefine sustained attention deficits to include deficits in phasic arousal. Though phasic arousal interacts with sustained attention to influence performance, the two are conceptually and neuroanatomically distinct (Huang-Pollock & Nigg, 2003; Posner & Petersen, 1990; Sarter et al., 2001). Thus, we rely on this more traditional definition of performance over time to evaluate the status of sustained attention in ADHD.

Second, the theories and methods of indexing the vigilance system first originated for situations in which the differences between targets and non targets were of degree rather than kind (e.g., apparent brightness of a visual signal, often referred to as “sensory” stimuli). This results in considerable overlap between signal and noise distributions, and therefore moderate error rates (Koelega, 1995b). In contrast, the CPTs most commonly used within the clinical literature often involve clearly distinctive target/nontarget stimuli (e.g., letters or numbers, often referred to as “cognitive” stimuli), which almost always results in high hit rates and low false alarm rates, or even perfect performance, because the discriminations are much less demanding (See, Howe, Warm, & Dember, 1995). When this occurs, SDT indices are less reliably interpreted because their calculation requires overlapping signal and noise distributions, so that RT and SDRT become critical dependent variables of performance (Sarter et al., 2001). Despite this, fewer studies report either RT or SDRT effects for the CPT, instead constraining their analyses to error rates even in the presence of highly accurate performance. Previous meta-analytic reviews of CPT performance in ADHD have in turn been unable to examine RT and SDRT variables in the aggregate (Corkum & Siegel, 1993; Losier et al., 1996; Nigg, 2005; Sonuga-Barke et al., 2008; Willcutt et al., 2005).

That being said, one obvious limitation to the use of RT data is that RTs produced from even the simplest of RT tasks are the culmination of multiple interacting processes, including speed–accuracy trade-off effects, rate of information processing, motor bias, and response criteria (Ratcliff, 2002). Modeling these multiple interacting processes accurately, and predicting how they ultimately produce a “RT” is therefore critical to the interpretability of RT data. The diffusion model (Ratcliff & Smith, 2004; Voss, Rothermund, & Voss, 2004) and the EZ-diffusion model technique (Wagenmakers, van der Maas, & Grasman, 2007) applied herein, is one way to do so.

Though an in-depth discussion of this process is beyond the scope of this paper, we briefly outline it for readers who may be unfamiliar with the model. The diffusion model assumes that simple forced choice decisions (i.e., those that can generally be made in <2000 ms) are made after a noisy information accumulation process. Simultaneously combining information from RT, RT variability, and accuracy, the diffusion model is able to extract information on three primary parameters: drift rate, boundary separation, and nondecision time. Drift rate is represented as v, which is the rate at which an individual is able to acquire information from an encoded stimulus to make a forced choice response (e.g., is this a target?). It is a fixed property of the stimulus (i.e., harder conditions, such as degraded stimuli, have slower drift rates) or of the participant (i.e., there are individual differences in drift rate). In this model, information accumulates until it reaches either the correct or incorrect decisional boundary, at which point a response is initiated. Because the accumulation of information is a noisy process, there is variability in the drift rate, which produces RT distributions. The degree of boundary separation is denoted as a. It represents how conservative an individual’s response criterion is, how certain s/he must be before making a response, or, the individual’s speed/accuracy trade off setting. The size of the separation is under the control of the individual (i.e., it can be increased or decreased depending on whether accuracy/speed is emphasized in the instructions). Even when drift rate is held constant, larger separations result in higher accuracy rates and slower RTs. Conversely, smaller separations result in faster RTs and lower accuracy rates. Ter, or nondecision time, represents the time it takes to complete all other information processes not involved in the stimulus discrimination process (e.g., encoding, memory access, motor preparation, etc.). Use of this model therefore allows a more precise estimation of the source of RT differences between groups of individuals or conditions. The model has been well validated, is robust to misspecifications (Wagenmakers, Ratcliff, Gomez, & McKoon, 2008), and has been used to identify the source of individual differences in complex cognitive processes in college-aged adults (Ratcliff, Schmiedek, & McKoon, 2008; Schmiedek, Oberauer, Wilhelm, Suss, & Wittmann, 2007), normal aging (Ratcliff, Thapar, & McKoon, 2004) and clinical populations (White, Ratcliff, Vasey, & McKoon, 2010). Using the EZ technique (Wagenmakers et al., 2007), we apply the diffusion model to account for ADHD-related differences in RT and error rate to complement the more traditional SDT indices, d’ and β.

Third, the complexity and interconnectedness of executive functions (EFs) have made it difficult to identify the optimal task parameters by which to measure sustained attention. In the absence of a consensus guiding principle, the number of variations in CPT paradigms, particularly in the clinical literature, is legion. Though effect sizes (ESs) are often aggregated without identifying potential moderating task variables (e.g., Corkum & Siegel, 1993; Nigg, 2005; Sonuga-Barke et al., 2008; Willcutt et al., 2005), variations in task parameters can have important ramifications on performance (Chee, Logan, Schachar, Lindsay, & Wachsmuth, 1989; Denney, Rapport, & Chung, 2005; Rose, Murphy, Schickedantz, & Tucci, 2001; See et al., 1995). One prominent difference that bears mentioning is the frequency with which a response is required. To evaluate the alerting response over time, traditional CPT tasks require participants to make a keypress to a rare signal. In contrast, inhibitory control paradigms require participants to make a frequent keypress to establish a prepotent motor response, which must then be inhibited on command. Because of the frequency with which a response is required and the requirement to inhibit that response, the Conners’ CPT (Conners, 2002) and the Sustained Attention to Response Task (SART: Robertson, Manly, Andrade, Baddeley, & Yiend, 1997) are better described as inhibitory paradigms despite their nomenclature, dominant presence within the sustained attention literature, and their wide clinical use for that purpose. Though deficits in inhibitory control are well documented in ADHD (Barkley, 1997; Lijffijt, Kenemans, Verbaten, & van Engeland, 2005; Nigg, 2001), recent reviews of EF in ADHD (Nigg, 2005; Sonuga-Barke et al., 2008; Willcutt et al., 2005) have included both types of paradigms when evaluating sustained attention deficits in ADHD. Thus, the degree to which impaired inhibitory control might have confounded ES estimates for sustained attention in ADHD is not known.

It has now been 16 years since Losier et al.’s (1996) meta-analysis of CPT performance in children with ADHD. Since that time, studies of sustained attention in ADHD have increasingly begun to report POT data, as well as RT and SDRTs, which allow for the computation of diffusion modeling parameters to better isolate the source of group differences. Although other reviews of cognitive processing in ADHD have included CPT studies (Corkum & Siegel, 1993; Nigg, 2005; Sonuga-Barke et al., 2008; Willcutt et al., 2005), those reviews were limited to aggregation of error rate data, and more recent reviews have further been contaminated by the inclusion of the Conners’ and SART go-no-go tasks. These changes, coupled with the importance of investigating attention as a critical cognitive mechanism of ADHD, argues for a comprehensive review of sustained attention processes in ADHD.

Method

Identification of Studies

ERIC, PsychInfo, PubMed, ISI Web of Knowledge, and Pro-Quest dissertation and theses databases were searched using the keyword “ADHD” combined with each of the following search terms: attention, vigil*, continuous performance, CPT, tests of variables of attention, TOVA, and sustained attention. We limited the search to articles that (a) were published between July 1995 (the last period of time included in Losier et al. [1996]) and January 2010, (b) primarily included children aged 6–12 (i.e., studies that predominantly consisted of preschool children, older adolescents, or adults, were excluded because of the comparatively limited number of studies in those age groups), and (c) were written in English. We also reviewed the bibliographies of included articles to identify any other potential missing articles. This returned a total of 3,248 unique articles.

Because of the extensive amount of variability in paradigm construction across the literature, we were forced to make some decisions regarding which paradigms could be included. First, the number of studies reporting complete data utilizing an auditory CPT task were rare, so we restricted our sample to studies examining visual paradigms. We only included paradigms that had <50% target probability. Thus, two commonly used CPTs, the Conners’ CPT (Conners, 2002) and the SART (Robertson et al., 1997) which have 90% target probabilities, were excluded. The Tests of Variables of Attention (TOVA; Greenberg & Waldman, 1993) includes separate go-no-go (i.e., target probability >50%) and vigilance (i.e., target probability <50%) blocks. If performance on these blocks was reported separately so that we could separate vigilance blocks from go-no-go type blocks, then those studies were included; otherwise, they were excluded.

We included studies that compared children with ADHD to typically developing controls, and which either directly reported an ES, data that allowed the calculation of an ES, and/or data that allowed for the calculation of SDT or diffusion analyses variables. Based on these criteria, we identified a total of 47 articles for inclusion. Of the studies that were excluded, 83% were reviews or studies that were clearly unrelated to the aims of the current article. Six percent did not include a typically developing control group (e.g., ADHD vs. bipolar, or a treatment effectiveness study utilizing an ADHD-only prepost design), 3% did not report minimally necessary information to calculate an ES (e.g., abstracts only), 5% were not CPT paradigms (e.g., Conners’, SART, flanker task, go-no-go or other inhibitory paradigms), and 3% were either not within the age group targeted (e.g., primarily adults) or were not written in English.

Calculation of Effect Sizes for Between-Group Studies of ADHD Versus Non-ADHD Controls

ESs (Cohen’s d) for errors of commissions, errors of omission, RT, and SDRT were calculated based on group means and pooled SD (when available), or were derived from reported F- or t-values. Because the number of omissions and hits are nearly perfectly inversely correlated (i.e., Omissions = 1 – Hits), we incorporated studies reporting hits but not omissions, into omissions (n = 15). Positive ESs indicated worse performance by the ADHD group. Not all studies reported data on all variables. ds for performance over time (POT) effects were derived either from F values for the Diagnosis × Block interaction or from reported means and SDs following Cohen (2002).

We next followed the meta-analytic procedures recommended by Hunter and Schmidt (2004) to correct the observed ds for both sampling and measurement error, and used the Hunter-Schmidt Meta-Analysis Programs Package v.1.1 (Schmidt & Le, 2005) to do so. Studies were weighted by sample size, which produces more accurate estimates of ES variability than does the more traditional inverse variance weight (Hunter & Schmidt, 2004). To most easily correct for measurement error, we algebraically transformed each d value to an r value following the equation for unequal sample sizes (Hunter & Schmidt, 2004).

Covered in much greater detail in Hunter and Schmidt (2004), measurement error has a systematic and multiplicative effect on the size of a correlation. Specifically, ρ_o = abρ, where the observed correlation is denoted by ρ_o, the population correlation is denoted as ρ, and the square root of reliability of the independent and dependent variables are denoted as a and b, respectively. Because the studies included in the current meta-analysis did not report the necessary information that would have allowed them to be individually corrected for measurement artifacts, corrections for measurement error were made using artifact distributions. That is, a distribution of reliability coefficients for the independent and dependent variables was obtained from published sources, and was used to calculate both mean reliability and variance in the population correlation that was because of variance in measurement unreliability.

Measurement unreliability in the independent variable (i.e., interrater reliability of ADHD symptomology and status) was corrected using published estimates of interrater reliability for the most commonly used and accepted behavioral rating scales and interviews as compiled in Pelham, Fabiano, and Massetti (2005). Twenty-eight percent of studies combined clinical interview and both parent and teacher questionnaires to diagnose children with ADHD; 8.5% used parent interview and parent or teacher questionnaires (but not both); 17% used clinical interview only; 28% used only parent and teacher questionnaires; 8.5% used chart review; and 10% did not report information on diagnostic process. Thus, our estimates of unreliability for the independent variable are imperfect corrections because they are based upon single measure reliabilities. However, even in this case, correcting for unreliability using imperfect estimates of reliability ultimately yields much more accurate estimates of ES than not correcting for measurement unreliability at all (Hunter & Schmidt, 2004).

Measurement error in the dependent variable (i.e., test–retest reliability) was corrected using published estimates of the test–retest reliability for commissions, omissions/hits, RT, and SDRT on the Gordon (Gordon & Mettelman, 1988) and TOVA (Greenberg & Waldman, 1993; Leark, Wallace, & Fitzgerald, 2004; Llorente et al., 2001).

Following the meta-analysis, we algebraically converted the corrected mean population correlation to a d value representing the population mean, δ, and report 80% credibility intervals (CV) around it (Hunter & Schmidt, 2004). CVs are based on the SD (rather than the SE, as in confidence intervals, CIs) and refer to the range of ESs which lie within a given interval (e.g., if an 80% CV is given as 0.60–0.90, it indicates that 80% of the values in the δ distribution lie between 0.60–0.90). They are distinct from CIs, which are centered on a single estimate of a (presumably) homogeneous δ, and refer to the amount of error attributable to sampling error. Because random effects models do not assume the presence of a homogeneous δ, and because sampling and measurement error has already been accounted for, the use of CVs versus CIs are more appropriate for meta-analyses (Hunter & Schmidt, 2004).

Moderator Variables

When the number of studies is small (i.e., when k = 40–100), the power to detect the presence of moderators is very low and the probability of capitalizing on sampling error and falsely identifying moderators when they are not present, is quite high (Hunter & Schmidt, 2004). This is particularly true when a significant proportion of variation in δ can be attributed to sampling or measurement error. However, in an exploratory fashion, we conducted meta regressions for major task characteristics (i.e., total number of trials, percentage of trials requiring a keypress, display time (ms), interstimulus interval (ISI, ms), and length of task (in minutes) as well as for prominent sample characteristics (i.e., proportion of males, average age, and average full scale IQ), noting that such results should be cautiously interpreted.

SDT Indices

Based on signal detection theory, d’, an index of perceptual sensitivity (i.e., the ability to discriminate between target and noise), was calculated for each study based on the mean hit and false alarm rate (Stanislaw & Todorov, 1999). An index of response bias, or the tendency to respond “yes, target present” versus “no, target absent,” was also calculated. Because β is based on a likelihood ratio, the natural log (lnβ) is often used instead in analyses, as we have also done herein. Negative values indicate a bias or predisposition to responding “yes, target present;” positive values indicate a bias or predisposition to responding “no, target absent,” and values of 0 indicate unbiased responses. Both d’ and lnâ were calculated with the formulas provided in Stanislaw and Todorov (1999). Only studies (n = 31) that reported raw hit (i.e., 1- proportion omission errors when the number of omissions was available; else, the raw hit rate was taken) and false alarm rate (i.e., proportion of commission errors) could be used to calculate SDT variables. Studies that reported hit and false alarm rate as standard scores (e.g., Z or T scores) were excluded from this set of analyses.

Diffusion Modeling

Using the downloadable Excel program for the EZ-diffusion modeling technique (Wagenmakers et al., 2007) from the author’s website: http://www.ejwagenmakers.com/papers.html, we estimated the drift rate (v), boundary separation (a), and nondecision time (Ter) variables based on mean RT, RT variance, and accuracy rate. Only studies that reported the mean RT, within subjects RT variance, and accuracy rate in raw (i.e. not standardized) format, could be included in the diffusion analyses (n = 12). Smaller values of v indicate slower drift rates. Smaller values of a indicate greater speed–accuracy trade-off, and Ter is reported in seconds.

Results

Meta-Analytic Results

The estimated population δ (corrected for both sampling and measurement error) for omissions, commissions, RT, and SDRT were large, and ranged from 0.61 (RT) to 1.34 (omissions), indicating that children with ADHD committed more errors and had slower/more variable RTs than non-ADHD controls (Table 1). When only corrected for sampling bias (i.e., bare bones meta-analysis), estimates were moderately sized, ranging from 0.37 (RT) to 0.62 (omissions), and were similar to earlier reviews (Losier et al., 1996; Willcutt et al., 2005). These differences are quite large, and suggest that previously reported ESs were greatly attenuated because of measurement unreliability. The amount of variance among ds because of measurement error was considerable, ranging from 32% (RT) to 90% (omissions). All CVs, with the exception of RT, exclude 0.

Table 1.

Summary of Meta-Analytic Results

	k	N	Mean d (SD)	Mean δ (SD)	80% CV	% var
Commissions	33	3,165	0.55 (0.18)	0.98 (0.27)	0.63 to 1.32	77.06
Omissions	39	3,192	0.62 (0.21)	1.34 (0.28)	0.98 to 1.69	90.30
RT	26	1,342	0.37 (0.44)	0.61 (0.73)	−0.33 to 1.55	32.10
SDRT	16	930	0.56 (0.23)	0.93 (0.38)	0.45 to 1.41	68.45
POT commissions	5	431	0.17 (0)	0.24 (0)		1474.86
POT omissions	7	488	0.38 (0.37)	0.54 (0.53)	−0.14 to 1.22	32.97
POT RT	5	338	0.19 (0)	0.27 (0)		206.01
POT SDRT	4	299	0.16 (0)	0.22 (0)		146.96

Note. k = number of studies; N = total number of participants; d = average uncorrected effect size where positive values mean worse performance by children with attention-deficit/hyperactivity disorder; δ = average population effect size; 80% CV = upper and lower limits of the 80% credibility interval; % var = percentage variance in observed ES explained by sampling and measurement error (when this value >100, it indicates the presence of second order sampling error); RT = reaction time; SDRT = SD of reaction time; POT = performance over time.

For POT effects, estimates of δ were small/moderate for all variables. The percentage variance accounted for by artifacts exceeded 100% (with co-occurring population SDs of 0) for all POT effects except omissions, indicating that for those variables, all the variance in ES between studies can be entirely attributed to the presence of sampling error and artifacts (Hunter & Schmidt, 2004). Obtaining a better estimate of the range of δ and the amount of variance that is attributed to second-order sampling error can only be resolved by the conduct of a second-order meta-analysis (i.e., a meta-analysis of meta-analyses).

We conducted two tests to evaluate for publication bias. A file drawer analysis (Orwin, 1983) indicated that the number of unidentified studies that would be required to be missing to reduce δ to negligible levels (i.e., 0.05) ranged from 281 (SDRT) to 1,006 (Omissions), suggesting that the δs are quite robust. The trim and fill procedure (Duval & Tweedie, 2000) estimates what δ might have been if publication bias did not exist. Following this procedure, we found a moderate degree of bias existed for Omissions (δ = 1.34 vs. 0.97 with 11 studies, or 28% imputed), Commissions (δ = 0.98 vs. 0.83 with 8 studies, or 24% imputed), RT (δ = 0.61vs. 0.29 with 7 studies, or 27% imputed), and SDRT (δ = 0.93 vs. 0.81 with 3 studies, or 19% imputed). All corrected δ’s remained large with the exception of RT. For the POT variables, the file drawer analysis indicated between 14 (POT SDRT) and 68 (POT Omissions) would need to be missing to reduce those ESs to negligible levels. The trim and fill procedure identified a larger degree of bias for POT Commissions (0.24 vs. 0.18 with 2 studies, or 40% imputed) and POT SDRT (0.22 vs. 0.14 with 1 study, or 25% imputed).

Moderators of Effect Size

All meta regressions for continuous task and sample characteristics were nonsignificant except that paradigms with more frequent targets resulted in larger δs for RT (r = .39, p = .03), samples with older children resulted in larger δs for Omissions (r = .35, p = .02), and samples with a larger proportion of males resulted in larger δs for SDRT (r = .44, p = .05).

Signal Detection Indices and Diffusion Analyses

Perceptual sensitivity (d’) was larger for non-ADHD controls than for children with ADHD, t(30) = 10.45, p < .001, d = 0.98, indicating that children with ADHD had greater difficulty distinguishing targets from nontargets (Table 2). The mean hit rate (and SD) for ADHD versus non-ADHD controls was 0.79 (0.17) versus 0.89 (0.12), and the mean false alarm rate (and SD) was 0.07 (0.07) versus 0.04 (0.06), respectively. There were no group differences for response bias (lnβ), t(30) = 0.49, p = .62, d = 0.04. As ISI increased, children were better able to distinguish between targets and nontargets (i.e., larger ISIs resulted in larger d’, r = .45, p = .02). As would be expected, as the percentage of targets increased, children showed a greater tendency to respond “target present.” There were no significant associations with sample characteristics (all r < .32, all p > .15).

Table 2.

Signal Detection Theory (SDT) and Diffusion Model Parameters

	ADHD	Control
SDT variables (k = 31)
d ’	2.68 (0.89)***	3.57 (0.93)
lnβ	0.99 (1.07)ns	1.04 (1.44)
Diffusion variables (k = 11)
v	0.18 (0.11)***	0.28 (0.16)
a	0.11 (0.03)ns	0.12 (0.04)
Ter	0.38 (0.11)ns	0.37 (0.09)

Note. ADHD = attention-deficit/hyperactivity disorder; ns = nonsignificant. Negative lnβ values indicate a tendency to respond “yes”; positive values indicate a tendency to respond “no.” Smaller values of v indicate slower drift rates, larger values of a indicate reduced speed accuracy trade off and larger values of Ter indicate slower nondecision times.

^***p ≤ .001.

Drift rate (v) was faster, t(11) = 4.55, p = 0.001, d = 0.75 for non-ADHD controls than children with ADHD (see Table 2). There were no group differences in boundary separation, t(11) = 1.71, p = 0.11, d = 0.16 or non-decision time (Ter), t(11) = −0.27, p = 0.79, d =−0.06. Longer ISIs were associated with faster v’s, r = 0.77, p < 0.01, and longer tasks were associated with slower v’s, r =−0.65, p = 0.04. Paradigms with a larger number of trials were associated with larger Ter, r = 0.70, p = 0.01. There were no other significant task or sample characteristics associated with v, a, or Ter (all r < 0.57, all p > 0.08).

Discussion

Meta-Analytic Results

We found that δ was large for the number commissions, omissions, and variability of RT. δ for overall RT was moderately sized, but was also sufficiently variable such that the 80% CV included 0, indicating that an ES of 0 is present within the δ distribution. When corrected for publication bias (Duval & Tweedie, 2000), δ for RT dropped to 0.29. Far fewer studies reported performance over time effects, with moderately sized effects found in POT for omissions, but small effects for Comissions, RT, and SDRT. That being said, vigilance decrements are more prominent in CPTs that use sensory (i.e., discriminations of physical characteristics that differ in degree) as opposed to cognitive stimuli (i.e., stimuli that differ in kind, such as alphanumeric symbols) because of the lower level of effort the latter type requires (See et al., 1995). Virtually all CPT paradigms currently in use in the clinical literature rely on cognitive stimuli. Thus, larger sustained attention deficits in ADHD might be observed under sensory discrimination conditions.

Previous reviews examining commission and omission errors reported only small to moderate ESs. The reason for the difference is methodological. Unlike previous reviews, and consistent with the contemporary recommendations for the conduct of meta-analyses (Council, 1992; Schmidt, 2010), we used a random effects model and corrected for both sampling and measurement error to provide the most accurate estimates of population-level δ and variance. When we only corrected for sampling error, estimates of δ were smaller and similar to earlier reviews, suggesting that previously reported ESs were greatly attenuated by measurement error.

Identifying cognitive endophenotypes is critical to improving the field’s understanding of the pathophysiology of ADHD. Initial findings in the search for endophenotypes have been encouraging (Nigg, Blaskey, Stawicki, & Sachek, 2004; Rommelse et al., 2008; Slaats-Willemse, Swaab-Barneveld, de Sonneville, & Buitelaar, 2007). But, significant interindividual variability in performance also suggests the presence of multiple causal pathways (Nigg, Willcutt, Doyle, & Sonuga-Barke, 2005), and has in turn spurred the search for and validation of etiologically homogenous subgroups of children with ADHD (Biederman et al., 2004; Lambek et al., 2010). However, our meta-analytic results provide clear evidence that measurement unreliability accounts for a significant proportion of the variability in effect sizes (ranging from 32%–100% in the current review). Concern regarding the psychometric properties of commonly administered tests of EF has long been noted (Doyle et al., 2005; Pennington & Ozonoff, 1996). This state of affairs is troubling, because it significantly hampers the ability to estimate the true level of population heterogeneity, and the search for factors that would assist in the creation of more homogeneous groupings. It also serves as a caution against any attempt to use CPTs in a clinical manner to assist in the diagnosis or treatment monitoring of ADHD.

It is interesting to note that our smallest δ was for RT, which contrasts with the large ES reported in a number of other cognitive paradigms (Douglas, 1999; Kuntsi, Oosterlaan, & Stevenson, 2001; Lijffijt et al., 2005). RT also had an exceptionally large CV, ranging from −0.33 to 1.55. Only ~30% of the variance for RT could be attributed to artifacts, with meta regression analyses indicating that paradigms with more frequent targets resulted in larger δ for RT. As a rule of thumb, if at least 75% of the variance in ESs can be attributed to measurement error, then it can be assumed that the remaining 25% is also attributable to uncorrected measurement artifacts (Hunter & Schmidt, 2004). Because this was not the case for RT, SDRT, or POT Omissions, it is possible that unidentified true moderators do exist that we did not examine and/or (more likely) that uncorrected measurement artifacts remain. We also found that samples with older children lead to larger δs for Omissions and samples with a larger proportion of males lead to larger δs for SDRT. Other task and sample specific moderators have previously been reported in other meta-analyses (Alderson, Rapport, & Kofler, 2007; Kofler, Rapport, & Alderson, 2008; Losier et al., 1996) but the presence of sampling error and uncorrected measurement artifacts present significant challenges to the identification of replicable moderators (Hunter & Schmidt, 2004).

SDT/Diffusion Analyses

Accurately estimating δ and population variance is only the first step to characterizing the cognitive mechanisms involved in childhood ADHD. We next applied signal detection and diffusion modeling parameters to better explain why children with ADHD performed so much more poorly than non-ADHD controls. We found that children with ADHD had more difficulty detecting targets from nontargets (i.e., smaller d’), but that they were no more biased than controls to responding “yes, target present” than “no, target absent.” (i.e., no group differences in lnβ). These results are consistent with earlier reviews and studies (Corkum & Siegel, 1993; Huang-Pollock et al., 2006; Losier et al., 1996; Nigg, 2005; Sergeant, 2000). Though use of SDT indices, which is based on the number of correct detections and false alarms, has become virtually synonymous with the concept of sustained attention, there remain significant concern regarding their proper use and interpretation, especially when accuracy rate is high (Koelega, 1995b; Sarter et al., 2001; See et al., 1995). Under those conditions, RT and RT variability, which are not utilized in SDT parameters, become particularly important aspects of performance.

In general, speeded RTs among children with ADHD tend to be slower and more variable than non-ADHD controls. Several different methods of examining patterns of RT data (e.g., ex Gaussian approaches, or analysis of modal and median RTs) have reported that this pattern is attributable to the presence of a large number of abnormally slow RTs in the tail of the distribution rather than slower/more variable RTs within the component of the distribution that represents the central tendency (Epstein et al., 2011; Hervey et al., 2006; Spencer et al., 2009; Williams, Strauss, Hultsch, Hunter, & Tannock, 2007). This in turn has lead to interpretations that the abnormally slow RTs are caused by cognitive processes that are distinct from those responsible for the production of central RTs (i.e., slow RTs are caused by periodic lapses of attention). However, those results can be more parsimoniously explained within the rubric of a diffusion model simply by proposing slower drift rates in ADHD (Figure 1). That is, if faster and slower drift rates are reduced by the same amount, the fast RTs are slowed by a much smaller degree than the slower RTs (Ratcliff & McKoon, 2008; Wagenmakers, Grasman, & Molenaar, 2005). Thus, if children with ADHD have slower drift rates, then it would explain why the RT distributions of children with ADHD are more skewed and also why measures of central tendencies may not differ as greatly between groups. Failure to consider speed–accuracy trade-off settings, which requires the simultaneous integration of error rate and RT data, could also account for findings that RT distributions of children with ADHD can also be marked by abnormally large numbers of fast RTs (Williams et al., 2007).

Figure 1.

Hypothetical RT distributions for children with and without attention-deficit/hyperactivity disorder (ADHD). If the drift rate for children with ADHD is “x” degrees slower than controls, the slowdown results in large changes in the tail of the distribution (Z), but only small changes at the leading edge (Y) (see Ratcliff & McKoon, 2008).

Our results indicated that the rate at which children with ADHD are able to accumulate information (i.e. their drift rate, v) is slower than that of non-ADHD controls, with an effect size of d = 0.75. We note here that the variance of the diffusion process for any given trial, or the change in the rate of the accumulation of information across time, is usually treated as a scaling parameter. This means that if that parameter is set to a particular value, then other parameters in the model also change by the same amount. Thus, the choice of the scaling parameter is entirely arbitrary, though by convention it is typically set to 0.1 (Wagenmakers et al., 2007). Though choice of scaling parameter has no impact on significance testing results, its arbitrary nature means that the absolute values of v, a, and Ter have little individual meaning. Thus, the interpretation of these values is dependent on the relative distance between groups.

In a manner that would be consistent with SDT and diffusion modeling expectations, task parameters were found to influence lnβ, d’, and a. Specifically, paradigms with a larger proportion of targets were associated with a greater tendency (as indexed by lnβ) to respond “yes, target present.” The ability to distinguish between targets and nontargets (d’) also increased as ISI increased; paradigms with slower pacing may be easier than paradigms with fast pacing. Similarly, longer ISIs and shorter task lengths were associated with faster v’s, and paradigms with a larger number of trials were associated with larger Ter. One of the strongest advantages of using the EZ diffusion technique is also one of its limitations. The EZ technique is able to estimate the main parameters of interest based just on average RT, RT variance, and accuracy. This lends itself quite well to the present purposes in which the full RT distributions for each study are not available. The weakness is that the extracted variables cannot be falsified by poor fit of the data to the model (Ratcliff, 2008; Wagenmakers et al., 2007), which is important here because the EZ technique was modeled for two-choice as opposed to one-choice RT data. But, we are reassured of the interpretability of our findings (to the extent possible) because the experimental manipulations had the specific and expected effects on the extracted variables of interest. That being said, the same cautions with respect to the reliability of moderator analyses in overall ES, should also be extended toward the SDT and diffusion model parameters.

Implications for Theories of ADHD

These results are relevant to several nonmutually exclusive etiologic theories of ADHD. In the cognitive energetic approach (Sergeant, 2000; Sergeant, Oosterlaan, & van der Meere, 1999a), Sergeant and colleagues have argued that the primary cognitive mechanisms in ADHD involve any one of three energetic pools: arousal, activation, effort, and the executive management of these pools. In this model, arousal refers to phasic alertness that influences the encoding of stimuli; effort refers to the necessary energy that must be expended to meet central task demands (e.g., search or decision processes); and activation refers to tonic changes in physiological activity that influences the speed of motor response/organization. From this perspective, the EF deficits observed in many children with ADHD are not because of deficits in the central processes themselves (e.g., inhibition and attention) but to the motivation or response to contingencies (i.e., effort), as well as level of arousal (phasic alertness), that power those processes (Sergeant, 2000). When restricted to standard SDT parameters, our results could be (and have been e.g., Douglas, 1999; Huang-Pollock et al., 2006) interpreted as evidence for a deficit in the arousal pool. However, as diffusion modeling in the context of the current meta-analysis makes clear, the problem appears to lie not so much in the arousal/encoding process (which is accounted for in Ter, nondecision time), or in the activation/motor step (also accounted for in Ter), but instead appears to implicate deficits in the effort pool and/or the central computational process that the effort pool feeds (i.e., processes directly related to information accumulation and decision making).

Altered motivational and reward contingencies that influence the effort pool have featured prominently within other etiological theories of ADHD (Sagvolden, Johansen, Aase, & Russell, 2005; Sonuga-Barke, 2002). Specifically, the dynamic developmental theory of ADHD (Sagvolden et al., 2005) proposes that EF deficits in inhibitory control and sustained attention can be attributed to a shortened delay of reinforcement gradient which itself is the result of altered dopaminergic functioning in the mesolimbic cortical branch. Though children with ADHD tended to prioritize speed over accuracy (smaller a), a finding which would be consistent with such a model, this difference was not statistically significant. Significant speed–accuracy trade-offs might be more prominent for tasks that require two choice decisions, or have a higher percentage of target, however.

And lastly, our findings for a slower drift rate among children with ADHD is consistent with recent work which suggests that an inability to suppress the spontaneous low-frequency natural rhythms of the resting brain state (i.e., the default mode) during active or effortful cognition (Fox, Snyder, Vincent, Corbetta, Van Essen, & Raichle, 2005; Fox, Snyder, Vincent, & Raichle, 2007) is not only responsible for increased RT variability (Castellanos, Kelly, & Milham, 2009; Sonuga-Barke & Castellanos, 2007), but may also be related to white matter abnormalities in ADHD (Valera et al., 2007). This model, known as the default mode hypothesis (Sonuga-Barke & Castellanos, 2007), suggests that ADHD-related weaknesses on tasks of sustained attention and other higher-order goal directed processes occur because of an inability to fully transition (or to maintain the transition) from a resting to active state of cognition. In light of the current meta-analytic results, this failure to transition and become fully engaged in the task results in slower rates of information uptake as reflected in slower drift rates. Recent work in adults (Schmiedek et al., 2007) and childhood ADHD (Karalunas & Huang-Pollock, 2012) have likewise found that drift rate predicts working memory capacity, a weakness which has been well documented in ADHD (Castellanos, Sonuga-Barke, Scheres, Di Martino, Hyde, & Walters, 2005; Diamond, 2005; Huang-Pollock, Karalunas, Tam, & Moore, 2012; Martinussen, Hayden, Hogg-Johnson, & Tannock, 2005; Willcutt et al., 2005). Thus, beyond indexing the speed at which information can be accrued in RT tasks, drift rate appears to represent an important link in explaining individual differences in higher order cognitive processes.

Summary and Conclusion

It is now widely accepted that the etiology of ADHD is complex. The search for the single gene or primary cognitive process that can account for the majority if not all of the variance between children, has long ceased to a primary pursuit of developmental psychopathology research. Instead, the field has moved toward a point of specifying how much neurocognitive heterogeneity exists among children with the same disorder, and what factors might moderate this heterogeneity (Doyle et al., 2005). However, achieving this goal requires that the amount of true heterogeneity in the population be as accurately estimated as possible, which in turn requires the use of meta-analytic techniques that correct not only for sampling error but also other artifacts that would otherwise obscure scientific reality (Schmidt, 2010).

We not only demonstrate large δs for overall performance on tasks of sustained attention in school-age children with ADHD, but also that a considerable proportion of the variability among studies is attributable to the presence of sampling error and artifacts. For some variables (i.e., RT, POT Omissions), only a small amount of variance could be attributed to artifacts, suggesting either the presence of uncorrected artifacts, or the presence of true moderators that encourages future work in this area. Our follow-up analyses suggest that group differences in performance are attributable to decreased perceptual sensitivity and slower drift rates. Our results are consistent with theories of ADHD that focus on the executive regulation of effort allocation (Sergeant, 2000) and with recent work identifying neurological causes of inter and intraindividual variability in RT for children with ADHD (Sonuga-Barke & Castellanos, 2007). Effect sizes were small to moderate for performance over time in the handful of studies reporting that data. When compared against large ESs in overall performance, this striking contrast is likely to be because of the almost exclusive use of cognitive versus sensory stimuli in the clinical literature. Distinguishing targets from non targets requires much greater effort when stimuli differ in degree rather than kind (i.e., when sensory stimuli are used), and are known to lead to more prominent vigilance decrements than when cognitive stimuli are used (See et al., 1995).

Though issues of measurement reliability are still critical, future work should consider whether, compared with more molar indices of processing speed and variability, drift rate may ultimately serve as a better endophenotype and be better suited to the identification of homogeneous cognitive subtypes in childhood ADHD. Future empirical studies examining sustained attention are encouraged to report performance over time data for all variables, to utilize sensory stimuli, and to avoid tasks like the Conners’ CPT (Conners, 2002) and the SART (Robertson et al., 1997), which are better described as inhibitory control paradigms. Future meta-analyses are also encouraged to correct not only for sampling error, but also measurement error and other artifacts that would allow for more precise estimation of population effect sizes and variance of effect sizes.