Time to publication for results of clinical trials

Abstract

Background

It has been suggested that a time‐lag bias exists whereby research studies with striking results are more likely to be stopped earlier than originally planned, published quicker, or both. If time‐lag bias exists, new interventions might be mistakenly assumed to be effective.

Objectives

To study the extent to which time to publication of a clinical trial is influenced by the significance of its result.

Search methods

Studies were identified by searching the Cochrane Methodology Register (The Cochrane Library, Issue 3, 2005), MEDLINE (1966 to May 2005), EMBASE (1980 to May 2005), Science Citation Index (June 2005) and by handsearching journals and conference abstracts.

Selection criteria

Studies were eligible if they contained analyses of any aspect of the time to publication of clinical trials and tracked the publication of a cohort of clinical trials.

Data collection and analysis

Data extraction was performed independently by two authors. Data were extracted on the median time from the date the trial started to the date of publication. Data were also extracted on source of trials under investigation; source of funding; area of health care; means by which the publication status of these trials were sought; and methodological quality of the empirical study.

Main results

Two studies with a total of 196 trials met the inclusion criteria. In both studies just over half of all trials had been published in full. Trials with positive results (i.e. statistically significant in favour of the experimental arm) were published in approximately 4 to 5 years. Trials with null or negative results (i.e. not statistically significant or statistically significant in favour of the control arm) were published after about 6 to 8 years. One study suggested that this difference could, in part, be attributed to the length of time taken to publish the results of a trial once follow up has been completed. This study showed that trials with null or negative findings took, on average, just over a year longer to be published than those with positive results.

Authors' conclusions

Our review shows that trials with positive results are published sooner than other trials. This has important implications for the timing of the initiation and updating of a review, especially if there is an association between the inclusion of a trial in a review and its publication status. It is of particular concern when one considers reviews containing only a small number of studies.

Plain language summary

Time to publication for results of clinical trials

The aim of this methodology review was to assess whether the time taken to publish the results of clinical trials is influenced by the statistical significance of their results (time‐lag bias). If clinical trials with positive findings are stopped earlier than planned and published quicker than those trials with null or negative findings, then new interventions might be mistakenly assumed to be effective. Two studies with a total of 196 trials met the inclusion criteria for this review. In both studies just over half of the trials had been published in full. Trials with positive results (i.e. with statistically significant results in favour the experimental arm of the trial) tended to be published in approximately 4 to 5 years. Trials with null or negative results (i.e. not statistically significant or statistically significant in favour of the control arm) were published after about 6 to 8 years. One of the studies suggested that this difference could, in part, be attributed to the length of time taken to publish the results of a trial once follow up has been completed. This study showed that trials with null or negative findings took, on average, just over a year longer to be published than those with positive results. Our review shows that trials with positive results are published sooner than those with null or negative results. This has important implications for the timing of the initiation and updating of a systematic review, especially if there is an association between the inclusion of a trial in a review and its publication status. It is of particular concern when one considers reviews containing only a small number of studies.

Background

It has been known for some time that it is more likely that studies with striking results get published than those with less striking results (publication bias). For example, a systematic review of five studies confirmed that failure to publish is heavily influenced by the direction and strength of the research findings, and that studies with positive results are published preferentially over those reporting null or negative results (Dickersin 1997). It has also been suggested that a time‐lag bias exists whereby research studies with positive results are more likely to be stopped earlier than originally planned, published quicker, or both (Stern 1997; Ioannidis 1998; Misakian 1998). Preliminary work on the publication of meta‐analyses of individual patient data has also revealed that the time taken to publish such studies may be associated with the statistical significance of the findings (Tierney 2000).

If time‐lag bias exists, such that the results of studies with more striking findings become available (through publication) sooner that those with less striking findings, this could be harmful. If the time taken to publish trials with positive results is years shorter than it is for those with negative or null results, new interventions will be accepted as effective in the absence of evidence to the contrary, even though that evidence may already have been gathered (Jadad 1998). Time‐lag bias might also introduce bias into systematic reviews if a study's inclusion is related to the timing of the availability of its data (Clarke 1998). It could be a particular problem for Cochrane reviews that initially have very few or no trials, if the intention is that these will be updated to include new studies as soon as these become available.

Objectives

To systematically review research studies that have empirically investigated the possibility and effect of time‐lag bias. We aimed to determine the extent to which the time to publication of a clinical trial is influenced by the significance of its result by studing a series of trials, forwards from their inception, rather than backwards from their publication.

Methods

Criteria for considering studies for this review

Types of studies

A research study is considered eligible for inclusion in this review if it contains analyses of any aspect of the time to publication of clinical trials and tracks the publication of a cohort of registered clinical trials such as those submitted to an ethics committee, funding body or obtained from a clinical trials' register.

Types of data

Cohorts of clinical trials that have been tracked over time.

Types of methods

The time to publication of positive findings will be compared with the time to publication of null or negative findings. Positive findings are defined as those with statistically significant results (P < 0.05) in favour of the experimental arm of the trial. Null or negative findings are defined as those with results that are not statistically significant or statistically significant results in favour the control (non‐experimental) arm of the trial.

Types of outcome measures

The primary outcome measure is the median time taken to publish the results of a clinical trial. This could be measured by analysing the interval between the date the trial started and date of publication, the date the trial closed and date of publication, or the date follow up ceased and date of publication.

Search methods for identification of studies

Studies were sought from the Cochrane Methodology Register (The Cochrane Library, Issue 3, 2005), MEDLINE (1966 to May 2005) and EMBASE (1980 to May 2005). The searches were initially conducted in March 2001 and were last updated in May 2005.

The Cochrane Methodology Register (The Cochrane Library, Issue 3, 2005) was searched using the indexing term "Publication Bias".

Studies were also sought during the handsearching of selected journals, which is being carried out by the UK Cochrane Centre for all studies relevant to the methodology of systematic reviews. The abstracts presented at all Cochrane Colloquia (1993 to 2004), Systematic Reviews Symposia (1998 to 2002) and Society for Clinical Trials Meetings (1980 to 2004) (as published in Controlled Clinical Trials and, more recently, Clinical Trials) have also been handsearched as part of this activity.

The titles and abstracts retrieved were assessed for relevance to the review (see below, Identifying studies). References in relevant reports were checked to identify additional studies and the Science Citation Index (June 2005) was also used to identify articles that cite relevant reports. Finally, researchers who may have carried out relevant studies were contacted.

For the full search strategy, see Appendix 1; Appendix 2.

Data collection and analysis

Identifying studies

One author (SH) screened the titles and abstracts of all retrieved records to identify obvious exclusions. A second author (MC) checked less obvious records. Full text copies of the non‐rejected records were assessed by at least two authors to determine if they met the inclusion criteria for the review. Any disagreements were resolved through discussion.

Assessment of methodological quality

The methodological quality of the included studies was assessed by noting whether: explicit criteria were used to select the individual trials; attempts were made to control for important clinical differences between the participants or interventions in the included trials (since these may influence the time to publication regardless of the trial's result); data were complete for the sample of trials. The overall quality of each research study was summarised as (1) no important flaws, (2) possible important flaws or (3) major flaws.

Data extraction

Data extraction was performed independently by two authors using a paper data extraction form. Information was extracted on the methodological quality of the included research study, the time period covered by the study, the source of the clinical trials under investigation, how the trials were classified (i.e. positive, null or negative results), source of funding, area of health care, single‐ or multi‐centre trials, the means by which the publication status of these trials were sought, and the time to publication for the trials in the sample. Differences in data extraction were resolved through discussion.

Data analysis

Data were extracted on the median time to publication for all trials, including 95% confidence intervals and hazard ratios. The median time to publication was also obtained for positive findings as compared to null or negative findings. The results were not combined in a formal statistical meta‐analysis as the studies measured different times to publication intervals and possible heterogeneity was therefore perceived.

Results

Description of studies

Included studies

Two studies (Ioannidis 1998; Stern 1997) met the inclusion criteria for this review and both were published as full articles in journals. The first study (Ioannidis 1998), of AIDS trials, examined the time interval between the date of first patient enrolment in a trial to completion of follow up and the time interval between completion of follow up and the date of first publication in a peer reviewed journal. Randomized trials conducted between 1986 and 1996 were identified from multi‐centre trialist groups. Trials were classified as positive (P < 0.05, or favouring the experimental arm of the trial) and negative (P > 0.05, or favouring the control arm of the trial). For the purposes of this review trials classified by Ioannidis 1998 as positive will be called positive and trials classified as negative will be called null/negative. A later version of this study has been published, but the outcomes reported were not relevant for our review (Haidich 2001). The second study (Stern 1997) examined the time interval between the date a trial was approved by an ethics committee to the date of first publication in a peer reviewed journal. Trials were identified from a hospital ethics committee between 1979 and 1988. The principal investigator of each trial was sent a questionnaire to determine when or where the trial had been published, if at all, and the significance of its result. Trials were classified as significant (P < 0.05), as showing a non‐significant trend (0.05 ≤ P <0.10) or as non‐significant or null (P ≥0.10). For the purposes of our review trials classified by Stern 1997 as significant will be called positive and trials classified as showing a non‐significant trend or null result will be called null/negative. Quantitative and qualitative studies were analysed separately and only quantitative studies (including clinical trials) are included in our review (see Table of Included Studies).

Excluded studies

Five studies that were initially assessed as potentially eligible, were excluded from the review. One study (Misakian 1998) assessed time to publication for studies of the effects of passive smoking but did not include clinical trials. The second study (Liebeskind 1999) examined the time to publication of trials in acute ischaemic stroke. Trials were identified from international trial registries and by searching MEDLINE for published reports of trials. We excluded this study as we felt that there was potential for bias in the sample of trials identified as it only included published reports of trials and did not track the publication status of all the registered trials. The third study (Simes 1987) compared the results of registered trials and published trials evaluating the effectiveness of chemotherapy on survival in advanced breast cancer. This study was excluded because the analysis of time to publication was not available separately for the registered cohort of trials and the published cohort of trials. The fourth study (Cronin 2004) assessed factors influencing publication of health care research studies. This study was excluded because not all of the research studies were reports of clinical trials. The analysis of time to publication was also not available separately for positive and negative/null trials. The final study (Burrett 2003) examined the time to publication of trials in childhood acute lymphoblastic leukaemia. We excluded this study because there was a potential for bias in the sample of trials identified (the study included published reports of trials only and did not track the publication status of a set of registered trials). See Table of Excluded Studies for more information.

Ongoing studies

We are aware of one ongoing study (Clarke) which will be assessed for relevance for inclusion in this review when the results become available (see Table of Ongoing Studies).

Risk of bias in included studies

In both included studies (Ioannidis 1998; Stern 1997) explicit criteria were used to select the individual trials. In the study by Ioannidis 1998 attempts were made to control for important clinical differences between the participants or interventions in the included trials. Whether this was done in the other study (Stern 1997) was not clear. In the study by Ioannidis 1998 there were complete data for the sample of trials, but this was not the case in the study by Stern 1997 because a number of trialists failed to respond to a questionnaire which aimed to determine how many of the included trials had been published. Using these criteria, the overall methodological quality of the study by Ioannidis 1998 could be summarised as having no important flaws while the study by Stern 1997 may possibly have important flaws.

Effect of methods

The results have not been combined in a formal statistical meta‐analysis because they measured different time to publication intervals. Stern 1997 measured the interval from the date of ethics committee approval to publication and Ioannidis 1998 measured from the date of the start of patient enrolment to publication. In both of the included studies the median time to publication was calculated from a survival type analysis of all the eligible trials. In the Ioannidis 1998 study the median time from start of enrolment to first publication was 5.5 years. This was less for trials with positive results, with a median of 4.3 years as compared to 6.5 years for trials with null/negative results (P < 0.001; hazard ratio for time to publication for positive versus null/negative trials was 3.7, 95% confidence interval (CI) 1.8 ‐ 7.7). The median time from completion of follow up to first publication was 2.4 years, which was shorter for positive trials with a median of 1.7 years as compared to 3.0 years for null/negative trials (P < 0.001; hazard ratio 3.2; 95% CI 1.6 ‐ 6.2). Stern 1997 measured the time interval between approval by an ethics committee and first publication. This was less for trials with positive results with a median of 4.7 years (95% CI 3.8 ‐ 5.7) as compared to 8.0 years for trials with null/negative results (95% CI 7.0 ‐ infinity) (P < 0.001; hazard ratio 4.2; 95% CI 1.7 ‐ 10.3). (Table 1).

1. Time to publication.

Study	Time interval	Time to publication	Positive	Null/Negative
Ioannidis (1998)	Enrollment to publication	5.5 years (median)	4.3 years (median)	6.5 years (median)
Ioannidis (1998)	Completion to publication	2.4 years (median)	1.7 years (median)	3.0 years (median)
Stern (1997)	Ethics committee to publication		4.69 years (median)	7.99 years (median)

Discussion

It is not surprising that a trialist who obtains a positive result will wish to communicate this rapidly. Our review showed that trials with positive results tended to be published one to three years earlier than those with null or negative results. The study by Ioannidis 1998 suggests that this difference could in part be attributed to the length of time taken to publish the results of a trial when follow up has been completed. That study found that null or negative findings took just over a year longer to be published than those with positive results.

Our review also confirms that trials with positive results were more likely to be published than those with negative or null results. In the Ioannidis 1998 study, 74% (20/27) of trials with positive results were published in full compared to only 41% (16/39) of trials with null or negative results. Similarly, in the study by Stern 1997, 72% (55/76) of trials with positive results were published in full compared to only 33% (18/54) of trials with null or negative results. This is reassuring as it suggests that the results from these studies are not atypical. A number of other studies (Bardy 1998; Dickersin 1992; Dickersin 1993; Dickersin 1997; Easterbrook 1991; Song 2000; Wormald 1997) have measured the rate of publication and have demonstrated the existence of publication bias, but these were not eligible for our review as they did not assess time to publication.

Both Ioannidis 1998 and Stern 1997 measured the time to publication of a trial's results in a peer reviewed journal. However, information is also disseminated in other ways such as through conference proceedings, non‐peer reviewed journals and internal reports. Other methods of dissemination were not assessed in these studies, although it is likely that these are subject to their own forms of publication bias. For example, just over half of all conference abstracts describing the results of clinical trials are published in full, and those with significant results are published in full more frequently than those with non‐significant results (Scherer 2007). There is clearly a danger that information from trials which have positive findings will become available sooner and therefore influence practice sooner than information from trials which show little or no benefit or, indeed, which may show harm.

Authors' conclusions

Implication for methodological research.

Despite rigorous searching we found only two eligible studies. This raises the question as to whether our review could, in itself, be subject to time‐lag or publication bias related to the component, empirical studies. It is possible that similar studies have been carried out and have either not yet been published, or will never be published, because their results were not shown to be significant. Prospective registration of methodological research, as is being promoted within The Cochrane Collaboration, may help to overcome this potential problem.

Further research is required to determine whether there is time‐lag bias for trials in other areas of health care. Research is needed that is pre‐registered and inspects cohorts of trials from their inception, rather than tracking the time and rate of publication retrospectively. This may be helped, in part, by the introduction of an International Standard Randomized Controlled Trial Number (ISRCTN) which can be assigned to planned, ongoing, and completed trials with the aim of encouraging prospective registration of all clinical trials, and the increasing pressure for prospective registration of clinical trials. The interpretation of any future research would also be helped considerably by more consistent reporting of outcomes. As a minimum, we suggest that the following should be available: time from approval, start of enrolment, end of enrolment and completion of follow‐up to publication.

What's new

Date	Event	Description
27 December 2007	Amended	Converted to new review format.

History

Review first published: Issue 2, 2002

Date	Event	Description
20 February 2007	New citation required and conclusions have changed	Substantive amendment

Appendices

Appendix 1. MEDLINE search strategy

MEDLINE (1966 to May 2005) was searched using the following search terms in Ovid:

1. Time factors.mp. 
 2. Publications/ 
 3. Publishing/ 
 4. Periodicals/ 
 5. Bibliometrics/ 
 6. "Bias (Epidemiology)"/ 
 7. Research Design/ 
 8. Publication Bias/ 
 9. Bias$.mp. 
 10. 1 and (or/2‐9) 
 11. Clinical Trials/ or Controlled Clinical Trials/ or Randomized Controlled Trials/ or trial?.tw. 
 12. 10 and 11 
 13. (time adj5 (lag$ or complet$ or publish$ or publication$)).tw. 
 14. 13 and (or/2‐9) 
 15. ((stop$ or suspen$ or discontinu$ or delay$ or early or slow$) adj5 (trial$ or publish$ or publication$)).tw. 
 16. 15 and (or/2‐9) 
 17. 12 or 14 or 16

Appendix 2. EMBASE search strategy

EMBASE (1980 to May 2005) was searched using the following search terms in Ovid:

1. Time/ 
 2. Time factors.tw. 
 3. Publication/ 
 4. Publishing/ 
 5. Epidemiology/ 
 6. bias$.mp. 
 7. Methodology/ 
 8. (1 or 2) and (or/3‐7) 
 9. Clinical Trial/ or Randomized Controlled Trial/ or trial?.tw. 
 10. 8 and 9 
 11. (time adj5 (lag$ or complet$ or publish$ or publication$)).tw. 
 12. 11 and (or/3‐7) 
 13. ((stop$ or suspen$ or discontinu$ or delay$ or early or slow$) adj5 (trial$ or publish$ or publication$)).tw. 
 14. 13 and (or/3‐7) 
 15. 10 or 12 or 14

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Ioannidis 1998.

Methods	Multi‐centre trials groups in HIV/AIDS sponsored by the US National Institutes of Health (trials conducted between 1986 and 1996). Publication status was obtained from the trial registry which sponsored the trial.
Data	66 multi‐centre AIDS trials.
Comparisons	(1) Time from enrolment to publication in a peer reviewed journal. (2) Time from completion of follow up to publication in a peer reviewed journal.
Outcomes	(1) Median time to publication for all trials was 5.5 years, positive 4.3 years, negative 6.5 years (P < 0.001; HR positive vs negative 3.7; 95% CI 1.8‐7.7). (2) Median time to publication for all trials was 2.4 years, positive 1.7 years, negative 3.0 years (P < 0.001: HR positive vs negative 3.2; 95% CI 1.6‐6.2 ).
Notes	109 trials identified. 43 trials excluded: 8 closed as failed to accrue over 20 patients; 25 still open to accrual; 10 still open to follow up.

Stern 1997.

Methods	Studies submitted to the ethics committee at the Royal Prince Alfred Hospital, Sydney, Australia (between 1979 and 1988). Publication status was obtained from the principal investigator.
Data	130 clinical trials submitted to the hospital ethics committee.
Comparisons	Time from approval by ethics committee to publication in a peer reviewed journal.
Outcomes	Median time to publication for positive trials was 4.69 years, null/negative 7.99 years (P = 0.0004; HR positive vs null/negative 4.19; 95% CI 1.71‐10.32).
Notes	748 studies identified. 618 excluded: 228 no response to questionnaire; 199 analysis had not begun; 103 qualitative studies; 88 observational studies.

HR: hazard ratio

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Burrett 2003	149 trials in childhood acute lymphoblastic leukaemia were identified from international trial registries and by searching MEDLINE. This study was excluded because of the potential for bias in the sample of trials identified as it only included published reports of trials and did not track the publication status of all the registered trials. The analysis of time to publication was also not available separately for positive and negative/null trials.
Cronin 2004	101 research studies commissioned by the North Thames Region of the NHS R & D Programme in the UK between 1993 and 1998 were assessed. The median time to publication from completion of follow up to publication in a peer reviewed journal was 35 months. This study was excluded because not all of the research studies were reports of clinical trials. The analysis of time to publication was also not available separately for positive and negative/null trials.
Liebeskind 1999	127 published trials in acute ischaemic stroke were identified from international trials registries and by searching MEDLINE. The median time from enrolment of first patient to publication was 4.2 years (positive trials 3.5 years, null/negative trials 4.4 years). This study was excluded because of the potential for bias in the sample of trials identified as it only included published reports of trials.
Misakian 1998	84 studies into the effects of passive smoking were identified through organisations known to fund research into passive smoking (1981‐1995). The median time from the year the grant was obtained to first publication was 3 years for studies with positive results, 5 years for non‐significant results and 7 years for null studies. This study was excluded because it did not include clinical trials. Although studies involving humans were analysed separately to those involving animals it was anticipated that the factors influencing time to publication of non‐clinical research studies might be different to those influencing clinical trials.
Simes 1987	Registered trials and published trials evaluating the effectiveness of chemotherapy on survival in advanced breast cancer were assessed. The cohort of registered trials were identified from protocols registered with the International Cancer Data Bank before October 1987. The cohort of published trials was identified by searching MEDLINE and conference proceedings prior to October 1987. This study was excluded because the analysis of time to publication was not available separately for the registered cohort of trials and the published cohort of trials.

Characteristics of ongoing studies [ordered by study ID]

Clarke.

Trial name or title	Publication bias in randomized trials funded by the MRC.
Methods
Data	All randomized trials (in a range of health care areas) sponsored by the UK Medical Research Council.
Comparisons	Time of enrolment of first patient to first publication.
Outcomes	Median time to publication for positive versus null/negative trials.
Starting date	April 1999: Work has started on this study but is currently on hold due to resource issues.
Contact information	Dr Mike Clarke, UK Cochrane Centre, NHS R&D Programme, Summertown Pavilion, Middle Way, Oxford OX2 7LG, UK.
Notes

Contributions of authors

SH and MC were involved in all stages of the review process, LS and JT were involved in drafting the protocol and in commenting on the various stages of the review.

Sources of support

Internal sources

• NHS Research and Development Programme, UK.

• Medical Research Council, UK.

External sources

• No sources of support supplied

Declarations of interest

None.

Unchanged