Defining the hidden evidence in autism research. Forty per cent of rigorously designed clinical trials remain unpublished ‐ a cross‐sectional analysis

Abstract

Autism spectrum disorders (ASD) have a prevalence of up to 2.7% and show significant rates of comorbidities. Pharmacological treatment can be difficult. New treatment options are needed, several are currently under investigation. Publication bias presents a major problem in current clinical research. This study was designed to quantify publication bias in rigorously designed ASD research. The database at http://ClinicalTrials.gov was searched for all completed randomized controlled clinical trials investigating interventions in ASD and their results made public. If results could neither be retrieved through search of the database, nor of scientific databases nor by enquiries of the responsible parties or sponsors listed, a trial was defined as not published. The search delivered N = 30 (60%) trials were published, N = 20 (40%) remained unpublished, N = 2,421 (59%) patients were enrolled in the published trials, N = 1,664 (41%) patients in the unpublished trials, time to publication was 21.4 months [standard deviation (SD) = 18.48; range = −5 to 80 months]. Results of N = 22 trials were available through http://ClinicalTrials.gov. Characteristics of published compared to unpublished trials did not show apparent differences. The majority of trials investigated drugs. The results emphasize the serious issue of publication bias. The large proportion of unpublished results precludes valuable information and has the potential to distort evidence for treatment approaches in ASD.

1. INTRODUCTION

Autism spectrum disorders (ASD) are characterized by persistent deficits in social communication and interaction across multiple contexts in combination with restricted, repetitive patterns of behaviour, interests or activities (American Psychiatric Association, 2013). Over the last decade, ASD have been an entity of particular interest and the number of recognized diagnoses has risen worldwide. ASD show a prevalence of up to 2.7% depending on the population investigated (Brugha et al., 2011; Kim et al., 2011). Recent data from the United States report the prevalence of ASD to be one out of every 68 children, or 14.7 per 1,000 (Centres for Disease Control and Prevention, 2014). Yet, these rates are discussed critically in the literature (Mandell & Lecavalier, 2014). Depending on the classification methodology they may differ and, specifically, be lower when strictly applying the Diagnostic and Statistical Manual of Mental Disorders (DSM‐5) criteria (Smith, Reichow, & Volkmar, 2015).

ASD have an early onset before the age of three years and are lifelong conditions with significant rates of comorbidities, e.g. major depression, attention deficit/hyperactivity disorder, anxiety disorders (Hofvander et al., 2009). In spite of extensive research, the etiology of ASD is still unknown (Ghosh, Michalon, Lindemann, Fontoura, & Santarelli, 2013). Treatment of ASD can be difficult, and the disease can be severely disabling thus resulting in multiple challenges for patients and afflicted families. Current pharmacological treatment strategies show comparably high rates of adverse events (especially for antipsychotic compounds) which can be a significant burden to patients and their families, and, thus, potentially lower medication adherence (Posey, Stigler, Erickson, & McDougle, 2008). New treatment options are needed which provide both clinical efficacy and tolerability/safety (Hampson, Gholizadeh, & Pacey, 2012; Maglione, Gans, Das, Timbie, & Kasari, 2012). From another perspective, this need also represents opportunities for researchers and pharmaceutical companies to develop and profit from new treatment approaches (Ghosh et al., 2013). A plethora of concepts is currently being considered as possible treatments for ASD. This includes (but is not limited to) pharmacological compounds, behavioural interventions, dietary supplements, medical devices, and others.

In current clinical research, federal agencies and ethical review boards require the registration of clinical trials in openly accessible (via Internet) databases. The largest clinical trials registry worldwide is http://ClinicalTrials.gov, operated by the United States National Library of Medicine at the National Institute of Health (NIH).

Publication bias describes the preferred publication of statistically significant or, in broader terms, positive results. This may be because authors or sponsors may have an interest in not publishing unfavourable results or because scientific journals prefer to publish articles that report significant results and thus increase the likely impact in the scientific community. In situations where new treatment options are being developed and evaluated, publication bias is an especially important factor as crucial evidence might not be made publically available. Publication bias presents as a major problem in clinical research as it may lead to either overestimation of treatment effects or neglect of important findings with likely negative consequences for patients, and higher socio‐economic costs (Easterbrook, Gopalan, Berlin, & Matthews, 1991; Hart, Lundh, & Bero, 2012; Dwan, Gamble, Williamson, & Kirkham, 2013; Wager & Williams, 2013).

Recently, 13,327 completed clinical trials registered at http://ClinicalTrials.gov from a five‐year‐period were evaluated for publication bias. Of these, 13.4% of trials reported results within 12 months after completion while 38.3% of trials reported results at any time after completion (Anderson et al., 2015). This estimates the publication bias for the investigated five‐year‐period at approximately 60%.

Anderson et al. (2015) comprehensively analysed all clinical trials regardless of the field of research, disease or drug investigated, and thus presented an overview of publication bias in current clinical research. Yet, clinical research and medicine in general is highly specialized today. In order to address the problem of publication bias, professionals need to be provided with figures relating to their fields of research or medical practice. The current publication bias in ASD is uncertain. Due to this lack of transparency, the practice of evidence‐based medicine in ASD can be distorted. We therefore directed our efforts to the quantification of publication bias in ASD research.

2. METHODS

2.1. Database search

From all clinical trials registered there, the database at http://www.clinicaltrials.gov was searched for completed randomized controlled clinical trials investigating therapeutic interventions in ASD. We focussed on randomized‐controlled (double blind) interventional clinical trials because they potentially represent the most robust level of evidence. Keywords used for the database search on http://ClinicalTrials.gov were: “double blind”, “placebo‐controlled study”, “randomized”, “efficacy”, “completed”, “autism”, and “child”. Data were downloaded on 22 January 2015 and manually checked independently by two authors for plausibility. To assess the status of publication of these trials' results, the respective entry of the database at http://www.clinicaltrials.gov was searched for results made public there or references/links to publications reporting results from the trial. If no results and no references were provided at the http://ClinicalTrials.gov database, a thorough search of the scientific databases of the US National Library of Medicine NIH (PubMed) at http://www.ncbi.nlm.nih.gov/pubmed (PubMed) and Google Scholar at http://www.scholar.google.com was performed using the http://ClinicalTrials.gov identification number (NCT), the intervention investigated, keywords, sponsor information, details of the study design and “autism” as search terms. If no publications could be found by these means, an enquiry was sent to the responsible parties and/or sponsors listed at the trial's respective entry at the http://ClinicalTrials.gov database via e‐mail.

2.2. Definition of publication bias

If no publications could be retrieved through the database search described earlier and by direct enquiry of the responsible party or sponsor, results from a clinical trial were defined as not published.

On the contrary, reports of trial results were defined as published if any peer‐reviewed manuscript published by a scientific journal could be identified. Results provided at http://www.clinicaltrials.gov were also considered peer‐reviewed since all entries are reviewed by an expert prior to being made publicly accessible. Research conference abstracts and press releases were not considered peer‐reviewed and consequently these trials' results were defined as not published.

2.3. Statistical analysis

The ratio as a percentage between published and unpublished results according to the definition used in this study represented the estimation of the publication bias for clinical trials in ASD. Trial characteristics were described by methods of standard descriptive statistics. Time to publication was computed as the time of publication minus the time of trial completion. All analyses were performed using SAS software (SAS Institute, Cary, NC, USA). The p‐values reported were two‐sided.

3. RESULTS

3.1. Trial characteristics

Fifty randomized controlled clinical trials investigating interventions in ASD completed between 2001 and 2014 were identified as completed on http://ClinicalTrials.gov by use of the methods and definitions described earlier. Figure 1 shows the details of the process and results of the study selection.

Figure 1.

Details of the study selection process

3.2. Trial characteristics

Years of completion of clinical trials ranged from 2001 to 2014, i.e. the observation period of this study covers the last 14 years. Table 1 summarizes key characteristics of the identified 50 interventional trials. Eighty‐six per cent of trials included paediatric populations exclusively, and 78% of trials investigated drugs. Thirty‐eight per cent were Phase 2 while 24% were Phase 3 trials. Funding was provided by industry for 42%, by the NIH and/or US Federal Funding for 20% of trials. Forty‐two per cent of trials were indicated as having received funding from other sources. Yet, the term “other” was not defined any further in the http://ClinicalTrials.gov database.

Table 1.

Characteristics of the N = 50 trials identified

		Trial results published N (%)	Trial results unpublished N (%)	Total N (%)
Overall		30 (60)	20 (40)	50 (100)
Trial type
	Interventional	30 (60)	20 (40)	50 (100)
Gender
	Male only	1 (100)	0 (0)	1 (2)
	Female only	0 (0)	1 (100)	1 (2)
	Mixed	29 (60.4)	19 (39.6)	48 (96)
Age groups
	Children only (< 18 years)	25 (58.1)	18 (41.9)	43 (86)
	Children and adults	5 (71.4)	2 (28.6)	7 (14)
Funding source
	Industry	8 (61.5)	5 (38.5)	13 (26)
	Industry/other	2 (33.3)	4 (66.7)	6 (12)
	NIH	3 (75)	1 (25)	4 (8)
	NIH/US Federal Funding	0 (0)	1 (100)	1 (2)
	NIH/other	4 (80)	1 (20)	5 (10)
	Other	13 (61.9)	8 (38.1)	21 (42)
Trial phases
	Phase 0	1 (100)	0 (0)	1 (2)
	Phase 1	0 (0)	2 (100)	2 (4)
	Phase 1/Phase 2	1 (50)	1 (50)	2 (4)
	Phase 2	13 (68.4)	6 (31.6)	19 (38)
	Phase 2/Phase 3	2 (66.7)	1 (33.3)	3 (6)
	Phase 3	6 (50)	6 (50)	12 (24)
	Phase 4	3 (100)	0 (0)	3 (6)
Intervention type	Drug	24 (61.5)	15 (38.5)	39 (78)
	Dietary supplement	3 (60)	2 (40)	5 (10)
	Behavioural	0 (0)	1 (100)	1 (2)
	Biological	0 (0)	1 (100)	1 (2)
	Device	1 (100)	0 (0)	1 (2)
	Procedure	1 (100)	0 (0)	1 (2)
	Other	1 (50)	1 (50)	2 (4)

3.3. Quantification of publication bias

Thirty trials (60%) were published and 20 trials (40%) were not published as defined by publication in a peer‐reviewed manuscript. Figure 2 shows the distribution of published versus unpublished trials by year.

Figure 2.

Distribution of published (N = 30) versus unpublished (N = 20) trials by year of completion

An overall number of N = 2,421 patients were enrolled in the published trials (N = 30 trials), whereas N = 1,664 patients were enrolled in the unpublished trials (N = 19 trials with available data). This means that 41% of the overall population participated in randomized controlled clinical trials without publication of the respective trials' results whereas 59% took part in trials whose results were published. The distribution of patient count stratified by publication status and year is depicted in Figure 3.

Figure 3.

Distribution of patient count stratified by publication status and year

3.4. Time to publication

The average time to publication, i.e. the delay from completion of the trial until public availability of the data was 21.4 months [standard deviation (SD) = 18.48; range = −5 to 80 months). One trial's results had been published before the official completion date listed at http://ClinicalTrials.gov. Figure 4 provides an illustration of time to publication, i.e. time between completion of trial and publication of results.

Figure 4.

Time to publication, i.e. time between completion of trial and publication of results, by year of completion of trial, in months (N = 30)

3.5. Publication medium

Results of N = 22 randomized controlled clinical trials were available through detailed listings of study results on http://ClinicalTrials.gov or links to peer‐reviewed papers from http://ClinicalTrials.gov to entries at PubMed. More trial results were found (N = 8) through additional manual searches on PubMed or Google Scholar. Two unpublished studies were mentioned in high level press releases, three studies were presented at meetings. However, for these five studies, no peer‐reviewed data were made available.

4. DISCUSSION

A substantial 40% (N = 20) of randomized controlled clinical trials, performed and completed with study designs that have the potential to deliver the highest level of scientific evidence, in ASD investigated in this study remained unpublished. This translates to 41% of trial participants (N = 1,664) whose data remained unpublished. Characteristics of published compared to unpublished trials did not show apparent differences regarding age groups, funding sources, trial phases, or types of intervention (cf. Table 1). No specific pattern discriminating published and unpublished trials could be identified.

The majority of trials investigated drugs but also dietary, behavioural, biological interventions, medical devices and procedures were studied. Drug development is a costly and time consuming process (Ciociola, Cohen, & Kulkarni, 2014). Recruiting paediatric patients with psychiatric disorders into interventional trials is particularly challenging (Bliznak, Berg, Häge, & Dittmann, 2013).

The largest of the unpublished randomized controlled clinical trials investigated drugs that have been granted marketing approvals for treatment of other disorders, e.g. memantine in Alzheimer's disease (480 patients), fluoxetine and sertraline in major depressive disorder (266 patients), and lurasidone (150 patients) in schizophrenia (cf. Table 2).

Table 2.

Characteristics of unpublished randomized controlled clinical trials (RCTs) (N = 20)

Designated type of intervention	Intervention	Study design	Trial phase	Enrolment (n)	Funded by
Behavioural	Gluten‐ and casein‐free diet	RCT	Phase 1	21	Other/NIH
Biological	Oral zinc, vitamin C	RCT	Phase 1	3	Other
Dietary supplement	Gluten and dry milk	RCT	‐	12	Other
Dietary supplement	Pyridoxine hydrochloride	RCT	‐	40	Industry/other
Drug	Arbaclofen	RCT	Phase 2	150	Industry
Drug	Aripiprazole	RCT		15	Other
Drug	CM‐AT	RCT	Phase 3	182	Industry
Drug	Fluoxetine	RCT	Phase 3	158	Industry/other
Drug	Fluvoxamine, Sertraline	RCT	Phase 3	108	Other
Drug	Galantamine	RCT	Phase 3	20	Other
Drug	Glutathione, vitamin C, cysteine	RCT		24	Other/industry
Drug	Lurasidone	RCT	Phase 3	150	Industry
Drug	Memantine	RCT	Phase 2	480	Industry
Drug	Methyl B12	RCT	Phase 2 / Phase 3	55	Other
Drug	N‐Acetylcysteine	RCT	Phase 2	32	Other
Drug	NNZ‐2566	RCT	Phase 2	67	Industry/other
Drug	Olanzapine	RCT	Phase 2	78	US Federal/NIH
Drug	Oxytocin	RCT	Phase 2	25	Other
Drug	Synthetic human secretin	RCT	Phase 3	0	Industry
Other	Cholesterol	RCT	Phase 1 / Phase 2	44	NIH

Most patients with ASD and their families use therapies which are not thoroughly investigated, e.g. off‐label drug treatment. Adverse events can be especially burdensome and often harmful, especially for dietary and drug treatments. Scientifically based and conducted trials are therefore of utmost importance and must of course include publication of results to be valuable.

Current pharmacological treatment options for ASD mostly target co‐occurring symptoms such as irritability and/or comorbidities and include behaviourally based interventions, e.g. applied behaviour analysis (ABA), and psychopharmacological treatments, e.g. antipsychotics or psychostimulants. While there are many new possible therapeutic approaches, their evaluation is mostly of pilot study character and therefore heavily restricted by methodological limitations such as small sample sizes and lacking control groups (Mechler et al., under review). This has also been shown for evidence of early intensive behavioural and developmental interventions in ASD (Warren et al., 2011). In 2012, a number of experts emphasized the existing research gap for evidence of efficacy of non‐pharmacological treatments of ASD (Maglione et al., 2012). The results presented in this study, despite focussing on randomized controlled trials, further underline this issue.

A recently published analysis of clinical trials in general used a very similar methodology as provided here, and reported comparable rates of publication bias: within five years after completion, results were only reported for 38.3% of clinical trials (Anderson et al., 2015). This figure is almost identical to the main outcome parameter in our study while trial characteristics are comparable as well [e.g. 77.3% drug trials in Anderson et al. (2015) and 78% in this study]. Another comparable study investigating clinical trials across disorders found 54.8% of trials registered at http://ClinicalTrials.gov not providing evidence of their results (Huser & Cimino, 2013).

In anxiety disorders, reporting biases have been analysed and found to lead to significant increases in the number of positive findings in the literature compared to reviews by the US Food and Drug Administration (FDA) (Roest et al., 2015). Pharmaceutical companies may have an interest in not publishing or delaying publication of unfavourable results from clinical trials. Yet, only 26% of trials included here were sponsored by a pharmaceutical company exclusively. Forty‐two per cent of trials were sponsored by “other” (not defined any further).

Non‐positive results (negative results or results showing inferiority or non‐superiority of an intervention) are considered to be more difficult to publish in journals. Results and experience from well planned and carried out trials may therefore not reach the scientific community and in the following not benefit the care of patients. An inadequate and incomplete picture of the respective field of research is the likely result. Non‐positive results are important, e.g. in order to avoid repetition of unnecessary research, for planning of further studies, including sample size estimations, as they provide quantitative as well as qualitative data, plus to prevent patients from being exposed to ineffective off‐label therapies. In addition, unpublished data cannot be considered in systematic reviews and meta‐analyses, and these missing trial findings may delay the recognition of important safety signals.

Also, it has to be considered unethical when recruiting patients into a clinical trial, under the premises that their participation contributes to research and clinical evidence, and in the following breaking this intention by not publishing the trials' results.

Various measures have recently been started to minimize publication bias. The Food and Drug Administration Amendments Act (FDAAA) of 2007 mandates results from a clinical trial to be published within one year after completion of the trial (FDA, 2007). Some online research journals have implemented open‐access publication of manuscripts that meet a set of qualitative criteria regardless of the outcome of the presented data, e.g. no statistically significant group differences (PLOS One 2015). In early 2013, the multinational campaign named “All Trials Registered – All Results Reported” was launched to promote reporting of all clinical trials' results (http://www.alltrials.net/). At the time of database closure for this study 83,178 people worldwide had signed the petition.

4.1. Limitations

As this analysis is by design and purpose limited to the most rigorously designed clinical research in autism, it may not represent the overall spectrum of autism research including scientifically more ambiguous work at a lower level of evidence. However, the lack of transparency of high‐evidence results has the most significant negative scientific impact.

Although http://ClinicalTrials.gov is currently the largest database worldwide for clinical trials it is possible that some clinical trials may not have been registered there. Although deemed unlikely by the authors, this may influence the real publication bias across all clinical trials being carried out worldwide. Additionally, for some of the more recently completed trials manuscripts reporting the trials' results may be in preparation or under peer‐review for publication and published soon or during the publication process of the present manuscript.

In this study, responsible parties or sponsors of the respective clinical trials were contacted if no publications could be found by the methods described. Although extensive efforts were made to identify current contact information, it is possible that contact information may have been outdated or incomplete. From some contacted individuals or companies no response was received. Reasons for this remain unknown.

5. CONCLUSION

Out of 50 randomized controlled clinical trials in ASD, 20 studies (40%) remained unpublished. This large proportion hampers information, especially on ineffective off‐label therapies, and has the potential to distort evidence‐based medicine for individuals suffering from ASD. The results presented here for research in ASD will serve as reference data for future studies reassessing publication bias and tracking potential progress resulting from recent and future advances and intentions to minimize publication bias.

DECLARATION OF INTEREST STATEMENT

Konstantin Mechler has served as an investigator in clinical trials funded by the European Union, Shire and Lundbeck. Georg F. Hoffmann has received compensation for serving as a speaker from Danone. Ralf W. Dittmann has received compensation for serving as consultant/speaker, or he or the institution he works for have received research support or royalties from the companies or organizations indicated: EU (FP7 Programme), US National Institute of Mental Health (NIMH), German Federal Ministry of Health/Regulatory Agency (BMG/BfArM), German Federal Ministry of Education and Research (BMBF), German Research Foundation (DFG), Volkswagen Foundation; Boehringer Ingelheim, Ferring, Janssen‐Cilag, Lilly, Lundbeck, Otsuka, Shire, Sunovion/Takeda, and Theravance. Dr Dittmann owns Eli Lilly stock. Markus Ries received consultancy fees or research grants from Alexion, GSK, Oxyrane and Shire.

The authors declare no financial interests in this manuscript.

Mechler K, Hoffmann GF, Dittmann RW, Ries M. Defining the hidden evidence in autism research. Forty per cent of rigorously designed clinical trials remain unpublished ‐ a cross‐sectional analysis. Int J Methods Psychiatr Res. 2017;26:e1546 10.1002/mpr.1546