Using Meta-analysis to Compare the Efficacy of Medications for Attention-Deficit/Hyperactivity Disorder in Youths

Abstract

Medications used to treat attention-deficit/hyperactivity disorder (ADHD) have been well researched, but comparisons among agents are hindered by the absence of head-to-head clinical trials. By using meta-analysis, we sought to compare the efficacy of these medications for the symptoms of ADHD. We analyzed published literature on the pharmacotherapy of ADHD to describe the variability of drug–placebo effect sizes and conducted a literature search to identify double-blind, placebo-controlled studies of youths with ADHD that were published after 1979. Meta-analysis regression was used to assess the influence of the medication type on drug effects. We also assessed for publication bias.

Thirty-two trials met our criteria and were included in this meta-analysis. These trials involved 16 drugs using 20 different outcome measures of ADHD behaviors. The effect sizes for immediate-release stimulants and long-acting stimulants were similar and were greater than the effect sizes for non-stimulants. There was no evidence of publication bias.

Although nearly all of the ADHD medications had significant effects, we found substantial variability. When translated into the costs of treating large numbers of patients, these effect sizes have implications for formulary medication choices.

INTRODUCTION

Attention-deficit/hyperactivity disorder (ADHD), which affects 8% to 10% of youths and 4% to 5% of adults,^1–3 is a source of chronic disability that can lead to academic impairment, social dysfunction, substance-use disorders, and an increased risk of accidents and criminal behavior.⁴ The symptoms and associated impairments of ADHD result in high costs for clinical services and to society.^5–10

Fortunately, ADHD is a treatable disorder, and several classes of medications are available on many formularies. The stimulant medications comprise methylphenidate (MPH), dextroamphetamine (d-Amph), and mixed amphetamine salts (MAS). Stimulants have been used for decades to treat ADHD. They increase the availability of synaptic dopamine^11,12 and reduce the overactivity, impulsivity, and inattention that are characteristic of patients with ADHD. Stimulants also improve associated behaviors, including on-task activities, academic performance, and social functioning.¹³ Compared with short-acting formulations, which require dosing up to three times per day, long-acting (LA) formulations extend the action of these medications over 8 to 12 hours to allow once-daily dosing.^14–16 Although stimulants are the most widely used ADHD medications, several nonstimulant medications are also effective for treating the disorder: tricyclic antidepressants (TCAs),^17–19 bupropion (Wellbutrin, GlaxoSmithKline),^20–22 modafinil (Provigil, Cephalon),^23,24 monoamine oxidase (MAO) inhibitors,^25,26 extended-release guanfacine (Intuniv, Shire),^27,28 and atomoxetine (Strattera, Lilly).^29–31

ADHD is a costly disorder because (1) its onset is early in life, (2) it causes a wide range of disabilities, and (3) it is a chronic condition, with most cases persisting into adulthood.³² The medical utilization costs of ADHD have been quantified by several investigators.

Leibson et al.⁷ conducted a population-based cohort study of 4,119 children. Among these children, the 309 who met criteria for ADHD were more likely to also have nonpsychiatric medical diagnoses, including major injuries. Children with ADHD were more likely to use inpatient, outpatient, and emergency care services. The nine-year median costs for youths with ADHD, compared with those without ADHD, were more than double ($4,306 vs. $1,944). Similar results were seen by Chan et al.³³ In their study of 5,439 children, 165 had ADHD and 322 had asthma. Children with ADHD had higher mean total health care costs ($1,151) than those with asthma ($1,091) and children with neither disorder ($712).

Another cohort study of 100,000 youths reported that the annual average medical expenditure was $1,574 per ADHD patient compared with $541 among matched controls.⁹ In a cohort study of 2,252 adults, Secnik et al.⁸ found that individuals with ADHD had significantly higher outpatient costs ($3,009 vs. $1,492 for matched controls), inpatient costs ($1,259 vs. $514), prescription drug costs ($1,673 vs. $1,008), and total medical costs ($5,651 vs. $2,771). These studies were consistent in concluding that people with ADHD, compared with persons without ADHD, showed substantially greater use of medical care in multiple delivery settings.

Given the high human and financial costs of ADHD, treatment is clearly warranted. But how should physicians, managed care professionals, and P&T committees choose among the many medications known to be safe and effective?

Ideally, physicians would evaluate different treatments by referring to randomized, double-blind, placebo-controlled studies that directly compared the treatments. Although individual medications are typically investigated thoroughly during placebo-controlled studies, studies comparing different treatments are less common and often involve smaller sample sizes. In the absence of direct comparative head-to-head trials, the best evidence comes from comparing the individual randomized, double-blind, placebo-controlled studies of each treatment using the method of meta-analysis, which provides a systematic quantitative framework for assessing the effects of medications reported in different studies.

MATERIALS AND METHODS

A literature search was conducted to identify double-blind, placebo-controlled studies of ADHD in youths (6 to 18 years of age) that were published after 1979. We included both parallel and crossover designs. We searched for articles using the following search engines: PubMed, Ovid, ERIC, CINAHL, MEDLINE, PREMEDLINE, the Cochrane database, e-psyche, and social sciences abstracts. We also reviewed presentations from meetings of the American Psychiatric Association (APA) and the American Academy of Child and Adolescent Psychiatry (AACAP). We included only studies that used randomized, double-blind methodology with placebo controls; that defined ADHD using diagnostic criteria from the Diagnostic and Statistical Manual of Mental Disorders, 3rd revised edition (DSM-III-R) or 4th edition (DSM-IV); and that followed subjects for two weeks or more.

To be included, studies had to have presented the means and standard deviations (SDs) of either change in scores or endpoint scores for the drug and placebo groups. For studies presenting data for more than one fixed dose, we used the highest dose because it seemed likely that the inclusion of lower doses would bias the study toward less optimal outcomes compared with studies using a design that used titration to optimal doses. Ideally, we would have included an analysis of dose effects, but such data were not available in enough studies to make this feasible.

We excluded studies that rated behavior in laboratory environments, involved fewer than 20 subjects in either the drug or placebo groups, explored appropriate doses for future work, or selected ADHD samples for the presence of a comorbid condition (e.g., studies of ADHD among children with mental retardation).

Effect sizes for dependent measures in each study were expressed as standardized mean differences (SMDs). The SMD is computed by taking the mean of the active drug group minus the mean of the placebo group and dividing the result by the pooled SD of the groups. Studies reporting change scores provided endpoint minus baseline scores for drug and placebo groups. In this case, the SMD was computed as the difference between change scores. For studies reporting endpoint scores, the SMD was computed as the difference between endpoint scores.

Studies were weighted according to the number of participants included. Our meta-analysis used the random effects model of DerSimonian and Laird.³⁴ We used the I² index to assess the heterogeneity of effect sizes.³⁵ Its value lies between 0 and 100 and estimates the percentage of variation among effect sizes that can be attributed to heterogeneity. A significant I² suggests that the effect sizes analyzed are not estimating the same population effect size. We used metaanalytic regression to assess the degree to which the effect sizes varied with class of medication. Egger’s method was used to assess for publication biases.³⁶

For each study, we treated all dependent outcome measures reported as a separate data point for entry into the analysis. Several studies provided data on more than one measure. Because measures reported from the same study are not statistically independent, standard statistical procedures produce inaccurate P values. To address this intrastudy clustering, we adjusted variance estimates using Huber’s formula,³⁷ as implemented in Stata software.³⁸ This formula is a “theoretical bootstrap” that produces robust statistical tests. The cluster scores (i.e., the sum of scores within studies) are entered into the formula to estimate variance. The resulting P values are valid even when observations are not statistically independent.

RESULTS

Table 1 presents the 32 articles that met criteria for inclusion in the meta-analysis. Studies are listed more than once if more than one drug was assessed or if independent studies of the same drug were reported. The studies in the table evaluated 15 drugs using 20 different measures of ADHD symptoms to assess efficacy. These measures were the Inattention/Overactivity with Aggression (IOWA) Conners Index; the Swanson, Nolan, and Pelham (SNAP-IV) total score; inattentive and hyperactive-impulsive scales; the Clinical Global Impressions (CGI) scale (both overall ratings, ADHD specific ratings and improvement ratings); the Global Efficacy Scale; the ADHD Rating Scale total score; inattentive and hyperactive–impulsive scales; the Conners ADHD Rating Scale total score; inattentive and hyperactive–impulsive scales; the Children’s Scale; the Conners Abbreviated Symptom Questionnaire for Teachers; the ADHD Clinical Composite Rating Scale; and the Conners–Wells ADHD Adolescent Self-Report Scale.

TABLE 1.

Features of the Meta-analysis

Study	Drug	No. in Drug Group	No. in Placebo Group	Mean Age	% Male	DSM Edition
Conners, 1980	MPH	20	20	12	95
Taylor, 1987	MPH	37	37	9	100	3
Klorman, 1994	MPH	44	44	9	84	3R
Conners, 1996	Bupropion	61	31	9	90	3R
Schachar, 1997	MPH	37	29	8	77	3R
Manos, 1999	MAS	42	42	10	79	4
Manos, 1999	MPH	42	42	10	79	4
Zeiner, 1999	MPH	36	36	9	100	3R
Pliszka, 2000	MAS	20	18	8	NA	NA
Pliszka, 2000	MPH	20	18	8	NA	NA
James, 2001	MAS	35	35	9	60	4
James, 2001	d-Amph	35	35	9	60	4
James, 2001	d-Amph ER	35	35	9	60	4
Wolraich, 2001	MPH	97	90	9	87	4
Wolraich, 2001	OROS MPH	95	90	9	78	4
Michelson, 2001	Atomoxetine	82	83	11	72	4
Michelson, 2002	Atomoxetine	84	83	10	71	4
Biederman, 2002	Atomoxetine	31	21	10	0	4
Biederman, 2002	MAS-XR	120	203	9	80	4
Greenhill, 2002	MPH-MR	155	159	9	83	4
Spencer, 2002	Atomoxetine	49	47	10	81	4
Spencer, 2002	Atomoxetine	53	45	10	81	4
Biederman, 2003	MPH-LA	63	71	9	77	4
Wigal, 2004	MPH	41	41	10	88	4
Wigal, 2004	d-MPH	42	41	10	88	4
Kelsey, 2004	Atomoxetine	126	60	10	71	4
Biederman, 2005	Modafinil	163	81	10	71	4
Findling, 2005	OROS MPH	89	85	9	70	4
Weiss, 2005	Atomoxetine	99	51	NA	80	4
Swanson, 2006	Modafinil	120	63	10	74	4
Wilens, 2006	OROS MPH	87	90	15	80	4
Greenhill, 2006	Modafinil	127	65	10	73	4
Biederman, 2006	Modafinil	48	51	11	74	4
Spencer, 2006	MAS-XR	26	28	14	64	4
Spencer, 2006	MAS-XR	27	24	14	64	4
Findling, 2006	MPH	120	39	10	79	4
Findling, 2006	MPH-MR	120	39	10	81	4
Biederman, 2007	LDX	73	72	9	71	4
Palumbo, 2008	MPH	29	30	9	83	4
Palumbo, 2008	Clonidine	31	30	9	87	4
Biederman, 2008	GXR	54	61	10	66	4

Note: Studies are listed once for each drug evaluated.

d-Amph = dextroamphetamine; d-MPH = dexmethylphenidate; DSM = Diagnostic and Statistical Manual of Mental Disorders; ER = extended-release; GXR = guanfacine extended release; LA = long-acting; LDX = lisdexamfetamine dimesylate; MAS = mixed amphetamine salts; MPH = methylphenidate; MR = modified release; NA = not available; OROS = osmotic-release oral system; XR = extended release.

Each drug–placebo comparison provided information on more than one outcome score, which allowed us to compute 150 effect sizes. The drugs studied fell into three classes: non-stimulants, immediate-release (IR) stimulants, and LA stimulants.

The nonstimulants studied were atomoxetine, bupropion, modafinil, clonidine (Catapres, Boehringer Ingelheim) and extended-release guanfacine (GXR). The IR stimulants were methylphenidate (MPH), dextroamphetamine (d-Amph), mixed amphetamine salts (MAS), and dexmethylphenidate (d-MPH). The LA stimulants were MAS-extended release (MAS-XR), d-Amph extended release (d-Amph ER), MPH-modified release (MPH-MR), osmotic-release oral system MPH (OROS-MPH), MPH-long acting (MPH-LA), and lisdexamfetamine dimesylate (LDX).

A pooled analysis of all studies computed an effect size of 0.78 (95% confidence interval [CI], 0.72, 0.83). As Figure 1 shows (with diamonds representing each subclass of medication), the effect sizes for IR stimulants (0.99; 95% CI, 0.88, 1.1) and LA stimulants (0.95; 95% CI, 0.85, 1.1) were similar and greater than the effect sizes for nonstimulants (0.57; 95% CI, 0.53, 0.62). There was no evidence of publication bias (t[146] = 1.1; P = 0.27) in the pooled analysis of all studies.

FIGURE 1.

Standardized mean differences and 95% confidence intervals (CIs) stratified by type of drug. The point indicates the effect size for each study. The horizontal line through each box gives the 95% CI. The diamonds give CIs for each type of drug. d-Amph = dextroamphetamine; ER = extended release; ES = effect size; GXR = guanfacine extended-release; IR = immediate release; LA = long-acting; LCI = lower 95% confidence interval; LDX = lisdexamfetamine dimesylate; MAS = mixed amphetamine salts; MPH = methylphenidate; MR = modified release; No. = number of observations; OROS = osmotic release oral system; UCI = upper 95% confidence interval; XR = extended release.

I² was significant (P < 0.0001) and indicated that 68% of the variability among effect sizes was a result of heterogeneity. One source of that heterogeneity was revealed by meta-analysis regression, which found a significant effect of drug type (F(2, 31) = 17; P < 0.0001). The effect sizes for nonstimulant medications were significantly less than those for IR stimulants (F[1, 31] = 25; P < 0.0001) or LA stimulants (F[1, 31] = 15; P = 0.001). The two classes of stimulant medications did not differ significantly (F[1, 31] = 0.3; P = 0.62).

We ran the heterogeneity analyses stratified by type of medication. For the LA stimulants, I² was significant (P < 0.0001) and indicated that 66% of the variability among effect sizes was a result of heterogeneity. For nonstimulants and short-acting stimulants, I² was low (0 and 2.7%, respectively) and was not significant (P = 0.8 and 0.5, respectively).

Figure 1 presents our results stratified by type of medication. The effect size for each medication was highly significant (P < 0.001) with the exception of two non-stimulants: bupropion (P = 0.2) and clonidine (P = 0.9). As the figure shows, both drugs had small effect sizes, but because the number of observations for these was small, we had low power to detect such small effects.

DISCUSSION

Our meta-analysis found significant differences between stimulants and nonstimulants. We found no evidence of publication bias. Although head-to-head trials are needed to make definitive statements about differences in efficacy, our results comparing stimulants and nonstimulants are compatible with the efficacy differences between atomoxetine and methylphenidate^39–41 and between atomoxetine and MAS^42,43 and the conclusions of a prior review limited to a smaller subset of studies that excluded short-acting stimulants.⁴⁴

The robust effects of most ADHD medications can be seen in Figure 1, with most of them having statistically significant effects on ADHD outcomes. It is also notable that for LA and IR stimulants, variability within each class was statistically significant. As described by Faraone,⁴⁵ the SMD effect size statistic can be translated into treatment costs of wasted treatment using simulations. In the absence of real-world comparative effectiveness trials, this is often the only way to model treatment costs.

For illustration, we can assume that placebo outcomes are distributed as a standard normal distribution and treatment outcomes are similarly distributed with three times the variability plus a constant that is equal to the SMD. When we simulate treatment of 100,000 patients, the SMD for nonstimulants (0.57) would yield 40,000 patients who do not respond (i.e., 40,000 wasted treatments). Each wasted treatment incurs unnecessary costs. These include the cost of the medication and the cost of the visits needed to titrate the medication. The SMD for both types of stimulants (1.0) would yield only 25,000 wasted treatments. If we consider the largest SMD in Figure 1 (1.52 for LDX), the number of wasted treatments would be only 12,000. This outlying observation might explain why we found significant heterogeneity for LA stimulants but not for the other classes of medication.

These results have implications for how physicians, other health care professionals, and P&T committees should choose among the many medications known to be safe and effective for ADHD. Currently, there are no strong criteria for deciding which medication should be tried first. Physicians must rely on clinical experience, the patient’s history of pharmacotherapy, the need for once-a-day dosing, and other clinical features that might influence their choice. For example, for patients with a history of substance abuse, prescribers would likely consider nonstimulants or stimulants that are more resistant to abuse.

Although reducing the costs of treatment should be secondary to finding the optimal treatment for each patient, the cost data reviewed previously suggest that improved efficacy and reduced costs are not necessarily at odds with one another. Cost does become an issue for P&T committees when IR stimulants (which are available as generic brands and are thus less expensive) are being compared with LA stimulants (most of which are not available as generics and hence are more expensive). Although the meta-analysis shows no differences in efficacy between these two types of stimulants, LA stimulants are associated with greater adherence⁴⁶ and a lower likelihood of being abused or diverted.⁴⁷ The strongest conclusion from the meta-analysis is that the nonstimulants are less efficacious than either type of stimulant. This suggests that a stimulant medication should be the treatment of choice in the absence of any clinical indicator that a nonstimulant is preferable.

It is difficult to compare the efficacy of ADHD agents with the efficacy of the most common nonmedical treatment for ADHD—behavior therapy; whereas there are many double-blind placebo-controlled studies of medication, behavior therapy studies tend to be uncontrolled. The meta-analysis of behavior therapy studies by Fabiano et al.⁴⁸ described 154 uncontrolled studies and only 20 controlled studies, and none of the latter studies were reported to have used double-blind methodology. The mean SMD for all of these studies was 0.67 (95% CI, 0.54–0.80). Thus, the SMD for behavior therapy is comparable to the SMD for nonstimulants and less than the SMD for stimulants.

STUDY LIMITATIONS

This work had several limitations. We relied on simulations to model costs of wasted treatment instead of using comparative-effectiveness trials, which were not available. Further, the interpretation of effect sizes from clinical trials might not be generalizable to the real-world clinical experience of patients, especially if trials use specially selected patients, such as those without other concurrent psychiatric disorders or concomitant medications. Effect sizes are sometimes derived from only single trials of a specific medication. Multiple trials of each medication yield more confidence in the estimates of effect size, as indicated by the CIs in Figure 1. Because we relied on data presented by authors, we were restricted by what investigators chose to present. For example, we could not compute the effect sizes at specific time points, because such data were rarely provided.

Future work should review time-course studies such as the analogue school laboratory paradigm. Although that work does not assess outcomes in the patient’s environment, it would provide useful data about efficacy peak or trough effect. Similarly, we did not assess differential duration of action between medication classes because this effect is rarely presented.

Although meta-analyses for indirect comparisons of treatments are useful, there are some disadvantages. All meta-analyses are limited by the quality of the studies analyzed. For that reason, we confined our review to double-blind, placebo-controlled studies.

The results of a meta-analysis may also differ from those of direct comparisons. For example, Bucher et al.⁴⁹ compared sulfamethoxazole–trimethoprim (e.g., Bactrim) with dapsone–pyrimethamine for the prevention of Pneumocystis carinii in HIV-infected patients. The meta-analysis suggested that the former treatment was much better.

In contrast, direct comparisons from randomized trials found a much smaller, nonsignificant difference. Song et al.⁵⁰ examined 44 published metaanalyses that used measure of effect magnitude to compare treatments indirectly. In most cases, the metaanalyses did not differ from the results of direct comparisons. However, for three of the 44 comparisons, there were significant differences between the direct and the meta-analysis estimates.

Chou et al.⁵¹ compared initial highly active antiretroviral therapy (HAART) plus a protease inhibitor (PI) against a non-nucleoside reverse transcriptase inhibitor (NNRTI). They conducted a direct meta-analysis of 12 head-to-head comparisons and an indirect meta-analysis of six trials of NNRTI-based HAART and eight trials of PI-based HAART. In the direct meta-analysis, NNRTI-based regimens were better than PI-based regimens for virological suppression. In contrast, the indirect meta-analyses showed that NNRTI-based HAART was worse than PI-based HAART for virological suppression.

Thus, although results of meta-analyses usually agree with those of head-to-head direct comparisons, the results of indirect comparisons using measures of effect magnitude should be viewed cautiously. Many variables can affect the apparent efficacy of treatments such as study quality, the nature of the population studied, the setting for the intervention, and the nature of the outcome measure. If these factors differ between studies, indirect comparisons can be misleading. For example, if all the stimulant studies used outcome measures that were more reliable than those used in the nonstimulant studies, the apparent difference between stimulants and nonstimulants would be an artifact.

CONCLUSION

Despite the limitations encountered, the findings highlight the variability of effect of ADHD medications. The efficacy of IR and LA stimulants was similar and significantly greater than that of nonstimulants. Moreover, for LA and IR stimulants, variability within each class was statistically significant. Because medication effect size differences translate into large cost differences when one is treating many patients, these data should be taken into account when P&T committees are making formulary decisions.