Meta-Analysis and Moderator Analysis: Can the Field Develop Further?

In this issue of the Journal, Hirota et al. and Schwartz et al. report the results of 2 clinically relevant and well-conducted meta-analyses. These reports examine the efficacy of atomoxetine and α-2 agonists for the treatment of children with attention-deficit/hyperactivity disorder (ADHD).^1,2 These studies represent an advance from previous meta-analytic work in the area for the following reasons: they use systematic search strategies to uncover unpublished trials; they examine not only the benefit but also the risks associated with these medications; they present both absolute and relative measures of benefit; and they conduct moderator analyses to examine characteristics of trials associated with the greatest measured treatment benefit of these medications.

In both meta-analyses, the authors report a medium-to-large benefit compared to that associated with placebo. The effect sizes of these nonstimulant medications are slightly more modest than those observed in similar meta-analyses of randomized controlled trials of psychostimulant medications.^3,4 α-2 Agonist and atomoxetine appeared to be similarly effective in targeting both the hyperactivity and inattention symptoms of ADHD. Both medications also demonstrated a statistically significant (but more modest) benefit in treating oppositional defiant disorder symptoms in children with ADHD. There was no difference in the rate of all-cause discontinuation between either medication and placebo. However, the dropout rate resulting from adverse effects was significantly higher for both α-2 agonists and atomoxetine compared to placebo. Atomoxetine demonstrated an increased rate of anorexia, fatigue, gastrointestinal, and central nervous system side effects. α-2 Agonists were associated with higher rates of fatigue, sedation and somnolence. α-2 Agonists also were associated with a modest decrease in blood pressure and heart rate, whereas atomoxetine was associated with a slight increase in blood pressure and heart rate.

These meta-analyses serve as excellent examples of what can be done when the data from many studies are combined and analyzed, including precise estimates of treatment effects (e.g., whether a treatment works and, if so, how well), and precise estimates of risk (e.g., what significant risks are associated with a medication and how common they are). On the other hand, these reports also provide a glimpse into the limitations of meta-analysis as a methodology. Most specifically, these meta-analyses demonstrate that even when many studies are combined, there may be insufficient power to detect potential moderating effects of treatments.

In their meta-analysis Schwartz et al.² demonstrated that children with ADHD experienced on average a 7-point improvement on the Attention-Deficit Hyperactivity Disorder–Rating Scale with atomoxetine compared to placebo. This reduction equates to a medium-to-large effect size of atomoxetine (0.64). However, 40% of children with ADHD failed to improve by even a mere 25% on atomoxetine. By contrast, 44% of children experienced a greater than 40% improvement on the drug. Would it not be useful to know which characteristics in children (if any) made them more likely to respond to atomoxetine? Similarly, Hirota et al.¹ report that α-2 agonist were associated with an average systolic blood pressure of 7 mm Hg. Would not it be useful to know what characteristics in children (if any) made them more likely to have hypotensive events on α-2 agonists? These are important issues that have the potential to guide clinical decision making. With information about treatment moderators, we could provide a more personalized, targeted treatment and perhaps avoid prescribing atomoxetine to many of the 40% of children for whom it would prove ineffective.

Important to the study of moderators of treatment in meta-analyses is to include study heterogeneity and the availability of patient-level data. Heterogeneity is a measure of the consistency of the results between trials in a meta-analysis. A large degree of heterogeneity occurs when there are large differences between trials in the measured efficacy of an intervention. Heterogenity is measured statistically in a meta-analysis using a χ² test. Furthermore, the I² statistic is used to report the degree of heterogeneity in a meta-analysis on a scale of 0 to 100. The I² statistic is independent from the number of trials included in a meta-analysis, whereas traditional statistical tests are not. Generally, I² values greater than 20% are considered to represent a large amount of heterogeneity worthy of additional investigation.

Hirota et al. and Schwartz et al.^1,2 report significant heterogeneity between trials in their measured efficacy of both α-2 agonists and atomoxetine. When confronted with heterogeneity in these meta-analyses, the authors did appropriately exhaustive work attempting to uncover sources of heterogeneity. They were able to identify some study-level factors associated with increased measured efficacy of atomoxetine in trials (e.g., having a higher proportion of treatment-naive patients). However, their findings in regard to potential moderating effects of these treatments were minimal. This result is not surprising, given the lack of access to individual patient data. Therefore, a major limitation of traditional meta-analysis is the inability to identify moderators when patient-level data are not available for analysis.

We actually do have some good examples of how moderator analysis can be useful for informing treatment decisions in ADHD when patient-level data are available. The Multi-modal Treatment Study for ADHD (MTA) randomized 579 children to 14 months of treatment with medication management (with psychostimulants), behavioral treatment, their combination, or community care.⁵ After 14 months of treatment, results in children assigned to medication management were statistically equivalent to those given combination treatment. Both medication management and combination treatment were superior to behavior or community treatment on most outcome measures. Behavioral treatment did not demonstrate significant incremental benefit over medication management alone on most outcome measures. Moderator analysis demonstrated, however, that children with comorbid anxiety disorders or significant comorbid anxiety symptoms (i.e., patient-level data) responded quite differently to treatment. Behavioral treatments were quite beneficial in this population and demonstrated significant benefit over medication management alone.⁶ Other moderator analyses suggested that significant maternal depression is associated with poor response in the combined and medication management arms of MTA.⁷ Thus, moderator analyses from the acute phase of the MTA informs current treatment of patients with ADHD.

There is only 1 extraordinary characteristic from the MTA trial that makes it particularly well suited to moderator analysis: namely, that all data (study- and patient-level data) were available to interested researchers. These MTA moderator analyses were conducted with less than 600 subjects across 4 treatments—powerful information with a relatively small sample size. There have been dozens of industry-funded ADHD medication trials with similar sample sizes and baseline assessments, but few informative moderator analyses from them, in part because of the lack of available patient-level data.

So, bravo to Hirota et al. and Schwartz et al.^1,2 for conducting excellent meta-analyses that clarify the beneficial and adverse effects of these medications. But imagine what a difference it could make for clinical care of patients if the individual patient data were available from the 4,000 subjects randomly assigned in atomoxetine randomized controlled trials; the 2,200 studied in α-2 agonist trials; and, importantly, the tens of thousands of children who have been studied in randomized controlled trials of psychostimulant medications.

Perhaps, most frustratingly and tantalizingly, these data actually exist. Moderator analyses identifying for which children our treatments are most likely to be effective (or harmful) are possible. All de-identified data from large National Institutes of Health–funded trials are made publically available.^8,9 Correll et al.^1,2 report significant heterogeneity between trials in their measured efficacy of both α-2 agonists and atomoxetine. The pharmaceutical industry currently makes the overall findings from clinical trials available, but does not easily make available individual subject data for these trials.

The American Academy of Child and Adolescent Psychiatry (AACAP) is at an important time in its history with the Food and Drug Administration (FDA) and the pharmaceutical industry. AACAP advocacy with the FDA and the pharmaceutical industry to make patient-level data available for meta-analyses could drastically improve the clinical outcomes of the children whom we treat. De-identified individual patient data should be made publically available from all pivotal FDA trials. Furthermore, efforts should be made by the FDA to encourage—or even require—the collection of potentially important baseline data (e.g., genetics, detailed clinical assessments) that can be used in moderator analyses. Imagine the new treatment potential that such cooperation could bring for our patients. Imagine how much faster we could start making personalized medicine a reality.