The Cochrane Collaboration withdraws a review on methylphenidate for adults with attention deficit hyperactivity disorder

Abstract

A Cochrane systematic review on immediate-release methylphenidate for adults with attention deficit hyperactivity disorder (ADHD) was withdrawn from the Cochrane Library on 26 May 2016 after substantial criticism of its methods and flawed conclusions. Retraction of scientific papers on this basis is unusual but can be necessary. We provide a summary of the criticism that led to the withdrawal. We detail the methodological flaws of the withdrawn Cochrane systematic review and general issues of bias and shortcomings of the included ADHD trials: cross-over designs compared with parallel-group designs, exclusion of participants with psychiatric comorbidity, absence of ‘functional outcomes’ and use of clinical outcomes with limited relevance, short trial duration and small trial populations, broken blinding caused by easily recognisable side effects, combining outcome assessments by trial investigators and participants, outcome reporting bias, poor evaluation of cardiovascular and psychiatric harms and conflicts of interest of trialists and systematic reviewers. The withdrawal of the Cochrane systematic review signals recognition of previous unreliable clinical ADHD research. We conclude that clinical trials of immediate-release methylphenidate in adults with ADHD are of very low quality. We urgently need well-conducted long-term trials free of bias to assess the benefits and harms of central stimulant treatment in adult ADHD.

Background

Attention deficit hyperactivity disorder (ADHD) in adults is one of the most controversial diagnoses in medicine,^{1 2} and the use of methylphenidate is sharply rising.³ The Cochrane review by Epstein and colleagues about immediate-release methylphenidate for adults with ADHD from September 2014⁴ was therefore long awaited, also because it took 9 years to appear after the protocol was published. However, four research groups independently submitted extensive criticisms of the review during the period October 2014 to November 2015. Three of the four groups coauthored the present paper and were in frequent correspondence with the editors and eventually Cochrane's editor-in-chief until the Cochrane editorial group decided to withdraw the review in May 2016.⁵ They issued a note outlining the reasons for their retraction.

"The Editorial Base of the DPLPG (Developmental, Psychosocial, and Learning Problems Group) appreciate the feedback received from the contributors named below, and their patience with our endeavours to secure a response from the author team. These endeavours have been persistent, but we were, until recently, unable to secure a response from the author team, by which time we had received guidance from Cochrane's Funding Arbiter as to how to manage their conflict of interest. Subsequently, one of the authors has expressed a wish to have his name removed from the review, and we have still to hear from a second author. The third author wishes to remain an author but given these, and subsequent criticisms, together with an unsatisfactory response to those criticisms from the author team, we have decided to withdraw this review. To make public comments for the benefits of readers, we will delay withdrawing the review until 26 May 2016". ⁴

The criticisms focused mainly on the review's flaws and misleading conclusions that gave a false sense of certainty of the benefits and the absence of harms, when in fact this could not be concluded. Fundamental methodological problems in the included trials that were not addressed in the review were also highlighted in the criticisms.⁴ The withdrawn Cochrane review reflects in many ways a general inadequacy of clinical ADHD research,^6–8 but the review authors failed to point out important flaws in the included trials. We provide here a summary of the criticism and discuss the consequences of the withdrawal.

The confidence in the evidence

Using the Grades of Recommendation Assessment Development and Evaluation (GRADE) approach,⁹ the review authors expressed ‘high’ confidence in the evidence for most outcomes, despite the fact that most domains in Cochrane's Risk of Bias assessment for the included trials were labelled as ‘unclear’. Furthermore, there was very high heterogeneity between results from the individual trials for ADHD symptoms (between 76% and 88%; the maximum is 100%), and the estimates were very imprecise. The confidence in the evidence should therefore have been ‘very low’. The authors did not explore the reasons for the heterogeneity, except for a subgroup analysis that compared low versus high dosage methylphenidate, arguing that they had too few studies for meta-regression. However, reasons for substantial heterogeneity should always be explored.

Cross-over studies

Nine of the included 11 trials had a cross-over design. The cross-over design is questionable to use in trials of central stimulants (such as methylphenidate or amphetamine) due to the risk of carry-over effects.¹⁰ It also increases the risk of unblinding and is therefore more susceptible to expectancy effects (see also Lack of blinding). According to the Cochrane Handbook, ¹¹ cross-over studies may be used for ‘stable’ conditions where ‘long-term follow-up is not required’, neither of which is the case for ADHD treatment. A previous meta-analysis that included trials of stimulants for adults with ADHD showed larger effects in cross-over studies than in parallel-group trials.¹² The failures to consider such fundamental elements of the trial design and explore its importance in sensitivity analyses raise major concern.

Psychiatric comorbidity

Most adults with ADHD suffer from psychiatric comorbidity, primarily anxiety and depression,¹³ and the exclusion of such participants from the trials greatly impairs their external validity.¹⁴ Only 2 of the 11 trials explicitly did not exclude participants with psychiatric comorbidity; 5 trials excluded them, and it was unclear in 4 trials. The review paid no attention to this, which should have led to a downgrading for ‘indirectness’ according to GRADE.⁹ Downgrading for indirectness should be applied when the trial conditions (population, intervention, outcome measures) differ considerably from clinical practice.

‘Functional’ outcomes

The authors acknowledged the negative impact of ADHD on ‘functional outcomes’, such as frequency of job change, work performance, traffic accidents, marital status and academic achievements. Yet, the treatment effect on functional outcomes was not quantified in the review, although these outcomes are arguably the most important ones. None of the included trials reported data on these outcomes specifically, but this should have been highlighted in the review as an important limitation.

Trial duration

The trials lasted only 1–7 weeks, and the long-term beneficial and harmful effects of methylphenidate are therefore unknown. Most adults with ADHD receive methylphenidate for substantially longer periods,¹⁵ and the beneficial effects may furthermore diminish over time. This was observed in the large Multimodal Treatment of Attention Deficit Hyperactivity Disorder Study in children.^{7 16–18} The short trial duration should therefore have led to additional downgrading for indirectness according to GRADE.⁹

Small trials

The total sample size in the review was only 474 participants, and the trial median was only 30. Small trials are underpowered to detect clinically relevant effects, and their results are generally misleading,¹⁹ particularly when they are biased, as is the case here. Furthermore, small trials cannot detect rare harms of drugs, a very important issue that was not mentioned in the review.⁴

Lack of blinding

Maintaining blinding in trials of central stimulant drugs with characteristic detectable (and for some people) pleasurable effects and also characteristic adverse effects is very unlikely without the use of an active placebo. This important issue was not considered in the review. Furthermore, 9 of the 11 studies used a titration scheme to achieve the highest tolerated dose, which virtually excludes the possibility that blinding was effective. Still, the trials were described as ‘double-blind’, and this was accepted by the review authors, the peer reviewers and the editors. The included trials did not formally assess whether the blinding was effective, but one trial²⁰ reported that all eight participants were able to guess their allocation. The inherent problem of maintaining blinded conditions in trials of central stimulants for ADHD was highlighted in the Cochrane reviews of methylphenidate for children with ADHD⁷ and of amphetamines for adults with ADHD.⁶

Pooling investigator-rated and self-rated ADHD symptom scores

The review reported the combined effect size for self-rated and investigator-rated ADHD symptom scores, where the latter was preferred when both were reported in the same study. However, previous meta-analyses have demonstrated substantial differences, with investigators estimating higher effects.^{12 21} Obviously, the participants’ perception of the effect is more important than that of the investigator. It has been argued that the investigator is more ‘useful’ to detect changes in ADHD behaviour,¹² but investigators may simply be more biased than participants in trials that have not been adequately blinded even though the detectable effects of central stimulants may bias the participants’ assessments. It is essential that both participant and investigator ratings are analysed and reported separately.

Outcome reporting bias

The authors of the retracted review wrote that prestated outcomes were reported in most studies and therefore judged the risk of outcome reporting bias as ‘low’.⁴ However, in direct contradiction to this, they wrote in the abstract that “We were unable to determine whether adverse effects were not discussed by study authors because none occurred or because no data on adverse effects were collected”.⁴ They also noted that this prevented them from conducting meta-analyses of adverse events and concluded that “Trial data suggest that adverse effects from immediate-release methylphenidate for adults with ADHD are not of serious clinical significance, although this conclusion may be limited, certainly in the case of weight loss, by the short duration of published studies”.⁴ It is misleading to conclude that trial data suggest that adverse effects are not of ‘serious clinical significance’ when important data are not available and follow-up is very short. The risk of outcome reporting bias in the included trials, most evident for the adverse events data, should have been stated clearly in the review.

Outcome reporting bias is not limited to old trials of immediate-release methylphenidate. We have pointed out²² that the most recent trial of extended-release methylphenidate from 2015²³ failed to report and specify crucial outcomes such as adverse events and quality of life.²² In addition, ‘absence from work’ and long-term outcomes after 2.5 years were omitted from the publication.

Cardiovascular and psychiatric harms

The cardiovascular harms of stimulants have long been in focus.²⁴ The review authors narratively described the cardiovascular effects in five studies, but they did not conduct meta-analyses for pulse or blood pressure changes, which are the only meaningful cardiovascular surrogate outcomes in small, short-term trials. Three of those five trials reported data for such an analysis. Stimulants increase blood pressure and pulse in adults²⁵ and may increase the risk of ischaemic heart disease beyond the short follow-up period of the trials, particularly in adults, where many have additional risk factors for cardiovascular disease. In children and adolescents with ADHD, there is an increased risk of arrhythmias and myocardial infarction during the initial phase of methylphenidate treatment,²⁶ and the risk of cardiovascular events with different levels of severity is doubled in stimulant users compared with non-users.²⁷ The risk of developing psychiatric adverse events such as psychosis, mania and aggression was not mentioned in the review,⁴ although a large Canadian population-based study²⁸ and an FDA review^{29 30} found that ADHD drug use, including methylphenidate, was associated with such events in children and adolescents.

Comparison with other research

The review authors compared their results with those of three meta-analyses but not with three newer systematic reviews,^{12 31 32} nor with a comprehensive drug class review.³³ These omissions are problematic as these reviews reported different effect sizes, considered several of the methodological limitations that we have outlined above and conducted subgroup and meta-regression analyses that Epstein and colleagues did not.⁴ The drug class review³³ even abstained from doing meta-analyses as the authors considered the trials to be too unreliable.

Conflicts of interests

We believe that the trialists’ conflicts of interests should have been discussed in the review. Only 2 of the 11 trials were explicitly funded by the drug industry, whereas 2 were unclear about this. Yet, two trials^{34 35} (one publicly funded³⁵ and one with no declaration of funding³⁴) were authored by three psychiatrists, which were investigated by US Senator Grassley in 2008 for financial ties to the pharmaceutical industry.³⁶ The investigation revealed that each researcher had received millions of dollars from the drug industry without disclosing it.³⁶ Two of the three authors of the Cochrane review declared financial conflicts of interests.⁴ It is very likely that authors of systematic reviews with conflicts of interests are influenced in the same way as trialists are.¹¹ We believe that the Cochrane Collaboration should introduce a policy that does not allow any financial conflicts of interest for review authors.

Discussion

The withdrawal sends a strong message to the field of clinical ADHD research and about the trustworthiness of the Cochrane Library. The decision to withdraw the review also provides indirect support to the Cochrane review on methylphenidate for children with ADHD, whose authors emphasised the ‘very low’ confidence they had in the evidence.⁷ This main conclusion has been criticised by the European Network for Hyperkinetic Disorders (EUNETHYDIS), a group of ADHD researchers. EUNETHYDIS has worked hard to discredit the Cochrane review of children^37–43 by publishing virtually the same criticisms in several papers in their own journals,37 38 41 43 despite the fact that their criticisms have been refuted repeatedly by the Cochrane review authors.^{7 44–49} EUNETHYDIS seems to defend a certain opinion rather than to consider the many flaws and low quality of the data outlined here and in the review of children.⁷ A systematic review on atomoxetine for adults with ADHD⁵⁰ concluded that the harms of treatment probably outweighed the benefits, but it was criticised in a similar manner by, among others, employees of Eli Lilly, the manufacturer of atomoxetine.^{51 52}

Scientific papers are rarely retracted, even when they are seriously flawed. Outright fraud is a reason for retraction, but even then it can take many years before retraction,⁵³ if it ever occurs. A policy of retracting papers that mislead readers importantly, with serious consequences for patients' and public health, should be introduced. Such initiatives could be taken by organisations like the World Association of Medical Editors or the International Committee of Medical Journal Editors. Appropriate measures should be taken, however, to avoid such policy being misused to get critical papers retracted.

The Cochrane editors should be praised that they made the difficult choice to retract the misleading Cochrane review, although we also note that it took 19 months from the first critical comments were submitted in October 2014 until these were published⁴ alongside the withdrawal in May 2016.⁵ During this time, the review was cited in influential publications such as the Canadian Agency for Drugs and Technologies in Health's review of high-dose stimulants for ADHD;⁵⁴ it was summarised in a ‘Cochrane for Clinicians’ article in the American Family Physician; ⁵⁵ and a recent JAMA editorial referred to the now withdrawn review in support of the view that methylphenidate trials are of high quality.⁵⁶ We urge Cochrane editors to speed up communications with readers and to introduce functions similar to the rapid responses in the BMJ or the response function in PubMed Commons as soon as it is technically possible.

Conclusions

The withdrawal of the Cochrane review on immediate-release methylphenidate for adults with ADHD was an important decision that underlines the fact that ADHD drug trials are unreliable. It is time to start doing them right, and we hope that the criticism we have listed above may serve as a checklist for future trials and also for systematic reviews of trials of central stimulants.