Methodological advantages and disadvantages of parallel and crossover randomised clinical trials on methylphenidate for attention deficit hyperactivity disorder: a systematic review and meta-analyses

Abstract

Objective

To assess the methodological advantages and disadvantages of parallel and crossover designs in randomised clinical trials on methylphenidate for children and adolescents with attention deficit hyperactivity disorder (ADHD).

Design

Secondary analyses of a Cochrane systematic review.

Setting and participants

We searched relevant databases up to March 2015 and included data from parallel and crossover randomised trials assessing children and adolescents up to 18 years with ADHD.

Interventions

Methylphenidate compared with placebo or no-treatment interventions.

Primary and secondary outcomes

The primary outcomes were teacher-rated ADHD symptoms and serious adverse events. The secondary outcomes were non-serious adverse events.

Results

We included 38 parallel trials (n=5111) and 147 crossover trials (n=7134). When comparing methylphenidate with placebo or no-treatment on ADHD symptoms, we found no differences between the end of parallel trials and the first-period from crossover trials (Χ²=1.06, df=1, p=0.30, I²=5.5%). We also found no differences when combining the end of first-period crossover trials with the end of parallel trials and comparing them to the end of last-period crossover trials (Χ²=3.25, df=1, p=0.07, I²=69.2%). We found no differences in serious and non-serious adverse events, and no risk of period and carryover effects. However, only two trials contributed data to the latter analyses.

Conclusions

Both parallel and crossover trials seem suitable for investigating methylphenidate in children and adolescents with ADHD, with comparable estimates on ADHD symptom severity and adverse events. However, parallel trials might still offer ethical and statistical advantages over crossover trials.

Strengths and limitations of this study.

• Study strengths include publishing a protocol in accordance with The Cochrane Collaboration guidelines, conducting a broad and comprehensive literature search, and assessing both published and unpublished data.

• The Food and Drug Administration and European Medicine Agency databases were not searched for unpublished data.

• The analysis comparing the end of first-period and last-period crossover trials included two trials only and might suffer from low power and spurious data.

• Methylphenidate gives rise to several easily recognisable adverse events, which may lead to loss of blinding and influence symptom ratings.

• A total of 179 trials (97%) were considered at high risk of bias, leading to decreased confidence in the effect estimates.

Introduction

Attention deficit-hyperactivity disorder (ADHD), one of the most common mental disorders among children and adolescents, is characterised by inattention, hyperactivity and impulsivity.^{1 2} The first-choice pharmaceutical treatment for ADHD is methylphenidate,³ a central nervous system stimulant exerting agonistic effects on dopamine and norepinephrine in the brain-stem and neocortex.⁴ We previously conducted a Cochrane systematic review on methylphenidate for children and adolescents with ADHD and found significant improvements on ADHD symptoms, general behaviour and quality of life, but the included randomised clinical trials (RCTs) suffered from low to very low evidence certainty.⁵

Crossover designs were about four times more frequent than parallel designs in the review.⁵ Clinical trials with crossover designs allocate participants to different interventions over two or more time periods, whereas in parallel trials, participants are randomised to the same intervention over a single period of time.⁶ Crossover trials may offer more precise estimates of intervention effects compared with a parallel trial because they would remove any biological and methodological variation. Hence, less participants need to be randomised to observe intervention effects.⁷ Importantly, the first period of a crossover trial can be viewed as independent and identical to a parallel trial.^7–9 Crossover trials typically last longer (which may inflate attrition rates) and involve fewer participants than parallel trials.^{7 8 10} However, since the participants are allocated to different interventions over time, it makes it potentially challenging to determine the causality.¹¹ Moreover, the crossover design is only suitable for a condition that is stable, when intervention effects are short-lived, and do not cure the condition.⁷ Parallel trials are versatile, simple and easy to incorporate into meta-analyses compared with crossover trials, but they usually require larger sample sizes.^{7 12}

The optimal way to incorporate data from crossover trials into meta-analyses is to retrieve data from all time periods,^{13 14} but such data are often unavailable. Another option is to impute missing data and adjust for ‘unit of analysis errors’ by conducting covariate analyses (for continuous data) or adjust for the variance to account for the correlation coefficient (dichotomous data).^{6 13 14} A third option is to ignore the crossover design, and treat all time periods as parallel designs. However, this introduces the possibility for ‘unit of analysis errors’ as the same patients receive both interventions.¹⁵ The data are therefore not independent, which is typically required for standard statistical methods.¹³

Crossover trials may also be biased by carryover and period effects. Carryover effects occur when an intervention effect during one time period is carried over and interferes with a subsequent period.6 7 9 12 14 This can be controlled for by introducing washout periods between intervention periods, causing the effects of the initial intervention to dissipate.^{9 16 17} Period effects occur when treatment effects differ between time periods in crossover trials,^{16 17} but because randomisation procedures are thought to diminish period effects, they are usually not as serious as carryover effects.^{7 17} Carryover effects are generally less suspected in crossover trials on methylphenidate because of its short pharmacokinetic half-life of 1.5 to 3 hours, depending on the drug formulation.^18–21 However, poor reporting of carryover and period effects and the lack of rationale for using crossover designs, is widespread and problematic.10 14 17 22

The aim of this study is to investigate the advantages and disadvantages of parallel and crossover trials on methylphenidate for ADHD, and to assess the risk of carryover and period effects when using crossover designs. The study rationale was to determine whether crossover trials are suitable for assessing the benefits and harms of methylphenidate for ADHD relative to parallel trials, and whether authors in the field should be more cognisant of potential differences between the two trial designs.

Methods

Data source

The data was derived from our Cochrane systematic review on the efficacy and adverse events of methylphenidate in children and adolescents with ADHD.⁵ The methods and design adhered to The Cochrane Collaboration standards for systematic review production.²³ The protocol was published in The Cochrane Library.²⁴

Patient involvement

No patients were involved in conducting this study.

Study selection

We included parallel and crossover RCTs on methylphenidate for children and adolescents with ADHD. Methylphenidate compared with placebo or no-treatment were to be administered regardless of dose and medical treatment regimen. We allowed co-interventions if the comparison groups received them simultaneously. All trials were included irrespective of language, publication type and publication status.

Inclusion criteria

We included children and adolescents, aged 18 or less, with a formal diagnosis of ADHD according to the Diagnostic and Statistical Manual of Mental Disorders (DSM-III, DSM-III-R and DSM-IV, DSM-V)^{1 25–28} or the International Classification of Diseases (ICD-9, ICD-10).² The participants were included regardless of comorbid conditions. At least 75% were required to have normal intellectual capacity (IQ>70), and 75% had to be under 18 years of age, with a mean of 18 or less.

We contacted authors of crossover trials to obtain data for all time periods if the trial report lacked information. We assessed study quality and risk of bias according to Grades of Recommendation, Assessment, Development and Evaluation standards and the Cochrane guidelines.^{5 23 29} For risk of bias, each study was assigned one of three categories: low risk, unclear risk or high risk of bias. Data was entered into Review Manager, V.5.3.³⁰

Search strategy

We searched CENTRAL, MEDLINE, Embase, CINAHL, PsycINFO, ClinicalTrials.gov, iCTRP and ISI CPCI up to March 2015. For CENTRAL, MEDLINE, Embase and PsycINFO, two different search strategies were used: one for efficacy and one for adverse events. To overcome the risk of poor indexing and abstracting, individual brand names were listed within the search strategies. We also contacted experts in the field and pharmaceutical companies for published and unpublished data, and we checked review articles for additional relevant studies. Relevant articles in non-English languages were translated. Further details on the data sources and search criteria are available in the Cochrane systematic review.⁵

Outcomes and comparisons

The primary outcomes were ADHD symptoms (ie, teacher, parent and observer-rated) and serious adverse events. We chose the teacher ratings as our primary outcome since ADHD symptoms are more readily detectable in the school setting.³¹ The secondary outcomes were non-serious adverse events.⁵ We first compared the end of parallel trials to the end of first-period of crossover trials. If no significant differences were found, we pooled the first-period of crossover trials with the parallel trials and compared these data to the end-period of the crossover trials. First- period and end-period crossover trials were compared to assess the risk of carryover and period effects.

Statistical analyses

We summarised dichotomous data as risk ratios (RR) with 95% CI, and if trials used the same continuous outcome, we summarised the data as mean group differences with 95% CI. For different continuous outcome measures, we used standardised mean differences with 95% CI. We applied both fixed effects and random effects models. If we were not able to retrieve the end of the first-period crossover data, we created groups of crossover trials with the end of the last-period data only. We originally intended to adjust for unit-of-analysis errors in crossover trials by conducting a covariate analysis for the continuous data, but the data were insufficient for this. For the dichotomous data, we were unable to adjust for the variance to account for the correlation coefficient due to insufficient data. We could not estimate the RR using the marginal probabilities and we therefore used the end of the last-period data for calculating the RR.³²

We presented the results in forest plots, and χ² and I² were applied to test for statistical heterogeneity. χ² assesses whether observed differences in the results are compatible with chance alone. χ² may have low power in meta-analyses of a few studies or small sample sizes and should therefore be interpreted with caution. I² describes the percentage of variability in the results that is caused by heterogeneity rather than sampling error (chance). Roughly speaking, I² <40% might not be important, 30%–60% might represent moderate heterogeneity, 50%–90% might represent substantial heterogeneity and 75%–100%, considerable heterogeneity.²³

Results

The search up to March 2015 identified 14 431 initial records. After removal of duplicates and irrelevant articles by abstract screening, 1460 publications were assessed for full-text inclusion. Subsequently, 185 randomised trials from 449 reports were included, of which 38 were parallel trials (n=5111 participants) and 147 were crossover trials (n=7134 participants) (figure 1). Only five of our crossover trials provided data from the different periods of the trial, either described in the publication or acquired after our requests to the authors.^{21 33–36} One trial had insufficient data for use in the meta-analyses. Lehmkuhl³⁶ was part of a larger study,³⁷ and data were acquired directly from HB Pharma (part of Medici). Of the five eligible trials, only two had data on teacher-rated ADHD symptoms.^{34 35}

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowchart. ADHD, attention deficit hyperactivity disorder; RCT, randomised clinical trials.

Characteristics of included studies

We assessed 179 (97%) trials at high risk of bias and six (3%) at low risk of bias.⁵ However, these six could potentially have suffered from lack of blinding because of the common, easily recognisable adverse effects of methylphenidate.⁵ Most trials were from high-income countries, the boy to girl ratio was 5:1 and participant ages ranged from 3 to 21 years with an average of 9.7 years. The average trial duration was less than 2 months, with only four trials lasting longer than 6 months. The median duration of the 38 parallel-group trials was 49 days (range 1 to 425 days, mean 75) and for the 147 crossover trials it was a total 14 days (1 to 56 days, mean 16 days).

ADHD symptoms

Parallel trials compared with first-period crossover trials

The analysis on teacher-rated ADHD symptoms included only 19 trials (n=1601) (figure 2). No significant differences were found between the end of the parallel trials and the end of the first-period of crossover trials (χ²=1.06, df=1, p=0.30, I²=5.5%). Moreover, no differences were found for independent assessor-rated ADHD symptoms (χ²=0.30, df=1,(p=0.58, I²=0%) or parent-rated ADHD symptoms (χ²=0.00, df=1,p=0.96, I²=0%) in these two groups of trials.

Figure 2.

Data from the end of the parallel trials compared with the end of the first period of the crossover trials on the effects of methylphenidate compared with placebo or no intervention on teacher-rated attention deficit hyperactivity disorder symptoms.

Parallel trials plus first-period crossover trials compared with end-period crossover trials

The analysis of teacher-rated ADHD symptoms included 75 trials (n=6247) (figure 3). No significant differences were found between the end of the parallel group trials plus the end of the first period of the crossover trials and end of the last period of the crossover trials (Χ²=3.25, df=1, p=0.07, I²=69.2%). Also, no differences were found for parent ratings (Χ²=1.58, df=1, p=0.21, I²=36.6%) in these two groups of trials. However, these groups of trials differed regarding observer ratings (χ²=4.23, df=1, p=0.04, I²=76.4), with methylphenidate exerting larger effects in the end of the last period of the crossover trials.

Figure 3.

Data from the end of the parallel trials plus the end of the first period of the crossover trials compared with data from the end of the last period of the crossover trials on the effects of methylphenidate versus placebo or no intervention on teacher-rated attention deficit hyperactivity disorder symptoms.

First-period compared with end-period crossover trials

The analysis of teacher-rated ADHD symptoms included only two trials (n=95 participants) (figure 4). The effect estimates as well as the 95% CI are comparable and do not indicate any difference between subgroups. There were insufficient data on parent ratings and independent-assessor ratings to conduct independent analyses.

Figure 4.

Data from the end of the first period compared with data from the end of the last period of crossover trials on the effects of methylphenidate versus placebo or no intervention on teacher-rated attention deficit hyperactivity disorder symptoms.

Serious adverse events

Parallel trials compared with end-period crossover trials

One analysis compared the end of the parallel-group to the end of the last period of the crossover trials on serious adverse events (figure 5). The analysis with 17 trials found no significant differences between the two groups of trials (Χ²=0.31, df=1, p=0.58, I²=0%).

Figure 5.

Data from parallel trials compared with data from end of the last period of crossover trials on serious adverse events.

Non-serious adverse events

Parallel trials compared with the first-period crossover trials

One analysis compared parallel-group trials to first-period crossover trials. The analysis included 20 parallel-group trials and one crossover trial with data from first period (online supplementary file 1). No significant differences were found between the two groups of trials (χ²=1.45, df=1, p=0.23, I²=31.1%).

Parallel plus first-period crossover trials compared with end-period crossover trials

One analysis compared parallel-group plus end of first-period crossover trials to the end of last period of crossover trials for non-serious adverse events (online supplementary file 2). The analysis with data from 42 trials found no significant differences between the two groups of trials (χ²=0.06, df=1, p=0.80, I²=0%).

First-period compared with end-period crossover trials

One analysis compared end of first-period crossover trials to end of last period crossover trials – one study was included (online supplementary file 3). The effect estimates as well as the 95% are comparable and do not indicate any difference between subgroups.

Subgroup analysis

We conducted a subgroup analysis investigating the impact of bias. We found that the intervention effect did not vary according to risk of bias (test for subgroup differences: χ²=2.43, df=1, p=0.12, I²=58.9%). However, it is likely that the trials initially judged to be at low risk of bias are in fact trials at high risk of bias because methylphenidate gives rise to various prevalent and easily recognisable adverse events, leading to potential loss of blinding and biased symptom ratings. This may result in overestimations of benefits and underestimations of harms.^38–40

In the original review,⁵ we conducted several subgroup analyses, for example, types of scales, duration of treatment and medication status. These were not deemed relevant for this publication. For more information on these, we refer to the original review.

Discussion

We aimed to investigate the methodological advantages and disadvantages of parallel and crossover designs of RCTs on methylphenidate for children and adolescents with ADHD. The study is important because RCTs with children and adolescents are challenging to conduct and often raise ethical concerns. Moreover, both parallel and crossover designs are used to investigate methylphenidate for children and adolescents, without anyone, to our knowledge, having investigated specifically whether both types of designs are suitable. table 1 summarises potential advantages and disadvantages of parallel and crossover designs based on the findings from this study and previous literature.

Table 1.

Methodological advantages and disadvantages with crossover and parallel randomised designs when investigating methylphenidate for children and adolescents with attention deficit hyperactivity disorder. Based on previous literature and the present findings

	Advantages	Disadvantages
Crossover designs	Suitable for stable conditions, when interventions are short-lived and do not cure the condition (eg, methylphenidate for ADHD. Crossover trials did not alter effect estimates relative to parallel trials in the present study with the exception of one observer-rated analysis of ADHD symptoms. Commonly reported biases with crossover studies, such as period and carryover effects, were not documented. Crossover trials need fewer participants than parallel trials to achieve sufficient power, which may decrease the use of participant and resource consumption. This study did not document any unit of analysis errors from using end-of-trial data from crossover trials in meta-analyses even after ignoring the within-patient correlations.	The present study was constrained by low evidence quality. Therefore, one cannot state for certain whether crossover trials on methylphenidate for ADHD are free of methodological biases relative to parallel trials. Crossover trials often last longer than parallel trials, which may increase overall attrition rates. Data from all time periods in crossover trials are often unavailable, and because participants switch interventions, causality of adverse events and effects may be difficult to determine in some circumstances. There may be undocumented adverse events associated with exposing participants to multiple interventions in a trial.
Parallel designs	Parallel trials are versatile, simple and easy to incorporate into meta-analyses when compared with crossover trials. By including more participants, parallel trials will have higher external validity regarding benefits and harms compared with crossover trials. Researchers do not need to address issues concerning unit of analysis errors, period effects and carryover effects or washout periods between interventions.	Parallel designs usually require larger sample sizes than crossover studies, which may increase financial costs.

For teacher-rated ADHD symptoms, we found no significant differences when comparing the end of the first period of the crossover trials to that of the parallel trials, the end of the first period to the end of the last period of crossover trials, or when comparing the end of the first period of the crossover trials plus that of the parallel group trials to the end of the last period of the crossover trials. We also found no significant differences when looking at parent ratings for the abovementioned analyses. However, we found significant differences in the observer-rated end-of- first -period crossover trials compared with parallel trials, with methylphenidate exerting larger effects in the end of the last -period of crossover trials.

Regarding our analyses on serious adverse events, we found no significant differences when comparing parallel trials with the last period of crossover trials, and no indications of period and carryover effects were found.

Only 10 out of the 147 included crossover trials (7%) addressed or mentioned the issue of carryover effects, which is fewer than previous studies have reported.^{6 17} We found no risk of carryover and period effects when comparing the end of the first period to that of the last period of crossover trials (figure 4), but this analysis was based on two trials only. When comparing the end of the parallel trials plus the end of the first period of crossover trials to the end of the last period of crossover trials for ADHD symptoms, we found no significant subgroup differences either (figure 3) which further decreases the likelihood of carryover and period effects in the crossover trials. Notably, this analysis had heterogeneity of I²=69.2%, indicating substantial heterogeneity between the groups of trials.

When meta-analyses incorporate crossover data, the unit of analysis errors may bias the results.¹⁵ The insignificant subgroup differences between the end of the parallel trials and the last period of the crossover trials suggest, however, that such errors were not major. The majority of the crossover trials presented only end-of-last period data, which undermines their interpretability and generalisability. This also underlines a larger problem with incorporating crossover trials in meta-analyses: data on within-individual comparison of treatments from paired analyses and data from multiple time periods are often unavailable to review authors.¹³ We only received few positive responses when asking for missing data.

A washout period between interventions is recommended when using crossover designs to reduce period and carryover effects.⁹ However, only 27 out of the 147 crossover trials (19%) included a washout period, lasting between 1 to 14 days. In total, 79 (56%) of the crossover trials did not include a washout period, and in the remaining 41 (29%), it was unclear whether washout periods were used. Of the trials with a washout period, only 15 (10%) provided arguments for its necessity. However, the results from this study indicate that the lack of reporting may have contributed to less bias than one might expect, possibly because of the short half-life and brief effects of methylphenidate.

Strengths and limitations

This study has several strengths. A protocol was published following The Cochrane Collaboration guidelines,²³ a broad and comprehensive literature search was conducted, and we included both published and unpublished data. Review authors worked independently in pairs of two, and all data from both parallel and crossover trials were analysed in accordance with the Cochrane Handbook.²³ Furthermore, several of the analyses that compared parallel trials to crossover trials included over 1000 participants, which may have increased statistical power and decreased the risk of type II errors.

Some limitations should also be highlighted. First, the Food and Drug Administration and European Medicine Agency databases were not searched for unpublished data. Second, the analysis comparing first-period and last-period crossover trials included two trials only.

The best way to analyse intervention effects in crossover trials is to conduct paired analyses, and the best way to investigate the carryover effect and period effect is to look at the within-trial data. Unfortunately, the reporting of crossover trials were very variable, and the data required to include paired analysis in meta-analysis were not available to us. We believe that both period effect and carryover effect, as well as unit of analysis error (error due to using end of period data as these were independent data) would have affected the estimates and showed significant group differences between the parallel group and the crossover trials investigating the same interventions for children and adolescents with ADHD. When we, in our first analysis, compared the end of the parallel group trials and the end of the first period of crossover trials with the end of the last period of crossover trials (using end-of-period data as if these were independent data which they, of course, are not) one would expect significant subgroup differences between the groups due to unit of analysis error (error due to using end of period data as if these were independent data) as well as due to carryover effect. When we, in one of the other analysis, compared end-of-first period crossover trials with the end-of-the-last-period crossover trials, one would also expect differences had there been any carryover effect from the methylphenidate treatment during the first period to the placebo treatment during the second period affecting the effect of the placebo treatment.

We also carried out a search for additional articles comparing the use of the two designs, but no articles describing methylphenidate or similar drugs were found. Other areas where the choice of design has been discussed include antiepileptic drugs, sleep research and infertility trials.^{7 8 41}

Conclusions

This study mostly found no signs of period effects or carryover effects in crossover trials assessing methylphenidate for children and adolescents with ADHD. Accordingly, crossover trials and parallel trials on methylphenidate for children and adolescents with ADHD seem to offer the same advantages and pose similar problems as described for these two designs in other therapeutic areas. Despite the insignificant results, crossover trial suitability for methylphenidate trials should still be questioned. It can be harder to detect and analyse adverse events in crossover trials, especially the late onset adverse events, since the treatment periods are often short. The two types of designs also have clear methodological and practical differences concerning what statistical analyses to use, systematic bias, ethical concerns about exposing children or adolescents to multiple interventions, and financial costs.^{17 42} Authors of crossover trials ought to report more comprehensively the data for end of all trial periods. Review authors should know how to incorporate the widely used crossover design into meta-analyses, in order not to lose valuable data.