Publication bias in clinical trials due to statistical significance or direction of trial results

Abstract

Background

The tendency for authors to submit, and of journals to accept, manuscripts for publication based on the direction or strength of the study findings has been termed publication bias.

Objectives

To assess the extent to which publication of a cohort of clinical trials is influenced by the statistical significance, perceived importance, or direction of their results.

Search methods

We searched the Cochrane Methodology Register (The Cochrane Library [Online] Issue 2, 2007), MEDLINE (1950 to March Week 2 2007), EMBASE (1980 to Week 11 2007) and Ovid MEDLINE In‐Process & Other Non‐Indexed Citations (March 21 2007). We also searched the Science Citation Index (April 2007), checked reference lists of relevant articles and contacted researchers to identify additional studies.

Selection criteria

Studies containing analyses of the association between publication and the statistical significance or direction of the results (trial findings), for a cohort of registered clinical trials.

Data collection and analysis

Two authors independently extracted data. We classified findings as either positive (defined as results classified by the investigators as statistically significant (P < 0.05), or perceived as striking or important, or showing a positive direction of effect) or negative (findings that were not statistically significant (P ≥ 0.05), or perceived as unimportant, or showing a negative or null direction in effect). We extracted information on other potential risk factors for failure to publish, when these data were available.

Main results

Five studies were included. Trials with positive findings were more likely to be published than trials with negative or null findings (odds ratio 3.90; 95% confidence interval 2.68 to 5.68). This corresponds to a risk ratio of 1.78 (95% CI 1.58 to 1.95), assuming that 41% of negative trials are published (the median among the included studies, range = 11% to 85%). In absolute terms, this means that if 41% of negative trials are published, we would expect that 73% of positive trials would be published.

Two studies assessed time to publication and showed that trials with positive findings tended to be published after four to five years compared to those with negative findings, which were published after six to eight years. Three studies found no statistically significant association between sample size and publication. One study found no significant association between either funding mechanism, investigator rank, or sex and publication.

Authors' conclusions

Trials with positive findings are published more often, and more quickly, than trials with negative findings.

Plain language summary

Publication bias in clinical trials due to statistical significance or direction of trial results

The validity of a systematic review depends on the methods used to conduct the review. If there is a systematic bias, such that studies with statistically significant or positive findings are more likely to be published and included in systematic reviews than trials with non‐significant findings, then the validity of a review's conclusions can be threatened.

This methodology review identified five studies that investigated the extent to which the publication of clinical trials (such as those approved by an ethics review board) is influenced by the statistical significance or direction of a trial's results. These studies showed that trials with positive findings (defined either as those that were statistically significant (P < 0.05), or those findings perceived to be important or striking, or those indicating a positive direction of treatment effect), had nearly four times the odds of being published compared to findings that were not statistically significant (P ≥ 0.05), or perceived as unimportant, or showing a negative or null direction of treatment effect. This corresponds to a risk ratio of 1.78 (95% CI 1.58 to 1.95), assuming that 41% of negative trials are published.Two studies found that trials with positive findings also tended to be published more quickly than trials with negative findings. The size of the trial (assessed in three studies) and the source of funding, academic rank, and sex of the principal investigator (assessed in one study) did not appear to influence whether a trial was published.

These results provide support for mandating that clinical trials are registered before recruiting participants so that review authors know about all potentially eligible studies, regardless of their findings. Those carrying out systematic reviews should ensure they assess the potential problems of publication bias in their review and consider methods for addressing this issue by ensuring a comprehensive search for both published and unpublished trials.

Background

Completed research is frequently left unpublished (Dickersin 1990). It has been suggested that in the case of research conducted on humans, failure to publish represents scientific misconduct, since individuals who consent to participate in research, and agencies that provide funding support for these investigations, do so with the understanding that the work will make a contribution to knowledge (Chalmers 1990). Clearly, knowledge that is not disseminated is not making a contribution.

Failure to publish is not only inappropriate scientific conduct, it also influences the information available for interpretation by the scientific community and by clinicians. If research is left randomly unpublished, there is less information available, but that information is not necessarily biased. The tendency for investigators to submit manuscripts and of editors and reviewers to accept them, based on the strength and direction of the research findings, has been defined as publication bias (Chalmers 1990; Dickersin 1990; Dickersin 1997).

The validity of a systematic review depends on the methods used to conduct the review and the ability to identify and include relevant studies. If there is a systematic bias, such that studies with statistically significant or positive findings are more likely to be published and included in systematic reviews than trials with non‐significant findings, then the validity of a review's conclusions may be threatened.

Objectives

To systematically review studies of cohorts of clinical trials that investigate the extent to which publication is influenced by the statistical significance, perceived importance, or direction of trial results.

Methods

Criteria for considering studies for this review

Types of studies

Studies were eligible if they assessed a cohort of trials registered at onset or while ongoing, but prior to the main results being known (e.g. trials entered into a formal database, or submitted to an ethics committee or a prospective trials register).

Types of data

Eligible studies included either a complete series of trials (e.g. all registered during a specified time period) or an unbiased sample (e.g. a random sample) of trials in a cohort. Studies were accepted as clinical trials if they were so defined by the study authors, and involved the testing of a health care intervention in humans. Studies that had also included other types of research were eligible for this review, if data specifically related to clinical trials were available and could be analysed separately. If these results could not be separated, attempts were made to obtain the data from the study investigators.

Types of methods

Eligible studies needed to compare the publication of trials with positive findings with the publication of those with either negative or null findings (collectively termed negative findings for the purposes of this review). Positive findings included (1) trials classified as having statistically significant results or P < 0.05, (2) when there was no statistical test done, findings classified by the study investigator as important or striking, and (3) those findings showing a positive direction of effect as defined by the authors of the included studies. Negative findings were defined as (1) those that were not statistically significant or P ≥ 0.05, (2) where there was no statistical test done, those findings classified by the study investigator as of moderate or little importance or not striking, and (3) those findings showing a negative or null direction of effect as defined by the authors of the included studies.

Types of outcome measures

To be included studies needed to report at least one of the two primary outcomes: publication or time to publication. Secondary outcomes included other potential risk factors possibly associated with failure to publish: source of funding, sample size, number of clinical centres, investigator rank and sex. If sufficient data were available we assessed whether or not the study results were written up, reasons for failure to publish, publication in English versus other languages, publication in a MEDLINE versus non‐MEDLINE‐indexed journal, and publication type (e.g. grey literature, including in‐house publications and theses).

Search methods for identification of studies

We searched the Cochrane Methodology Register (Issue 2, 2007 as published in The Cochrane Library [Online]) using the index term "publication bias" which includes "language bias" and "duplicate publication bias". In addition we searched MEDLINE (1950 to March Week 2 2007 Appendix 1), EMBASE (1980 to Week 11 2007 Appendix 2 ) and Ovid MEDLINE In‐Process & Other Non‐Indexed Citations (March 21 2007 Appendix 3).

We also searched the Science Citation Index (April 2007) to identify additional articles that cited any included studies. Finally we contacted authors of key studies on publication bias to try to identify further studies, and checked reference lists of any included studies to identify references to possible relevant citations.

Data collection and analysis

Selection of studies

Our searches identified over 5000 references. One author (SH) screened the titles and abstracts of all retrieved records to identify obvious exclusions. A second author (KL) checked all retrieved records once any obvious exclusions had been removed. Any disagreements were resolved through discussion. Each of the non‐rejected records were assessed by at least two authors to see if they were likely to meet the inclusion criteria and full copies of the reports were obtained. Each of the full reports were then assessed by at least two authors to determine if they met the inclusion criteria for the review. Any disagreements were resolved through discussion.

Data extraction and management

Two authors independently extracted data from each of the included studies. Differences in data extraction were resolved by discussion. We contacted the authors of the studies if information was either incomplete or missing, or to obtain data separately for reports of clinical trials.

We extracted the following data when available:

Relationship of publication to positive findings and magnitude of effect:

• Statistical significance or P value (< 0.05 versus ≥ 0.05)

• Perceived importance of the findings (clinically important, striking)

• Direction of results (positive or negative)

• Time to publication

Relationship of publication to other potential risk factors:

• Funding mechanism (grant, contract, other)

• Sample size (< 100, 100 to 999, > 999 or as defined in included studies)

• Number of centres

• Primary investigator (male versus female)

• Primary investigator academic rank (e.g. professor, associate professor, assistant professor, other)

Outcome measures:

• Publication

• Publication type (e.g. grey literature, abstract, presentation, language of publication)

Assessment of risk of bias in included studies

The following criteria, perceived as likely sources of bias, were used to assess the methodological quality of included studies:

1. Was there an inception cohort? 
 Yes = a sample of clinical trials registered at onset or on a roster (e.g. approved by an ethics committee) during a specified period of time 
 No = anything else 
 Unclear

2. Was there complete follow up (after data analysis) of all of the trials in the cohort? 
 Yes ≥ 90% 
 No < 90% 
 Unclear

3. Was publication ascertained through personal contact with the investigators or sponsor? 
 Yes = personal contact with investigators or sponsor, or searching the literature and personal contact with investigator or sponsor 
 No = searching the literature only 
 Unclear

4. Were positive and negative findings clearly defined? 
 Yes = clearly defined 
 No = not clearly defined 
 Unclear

5. Were other possible confounders examined in the analysis, for example: sample size, duration, multi‐centre versus single centre, funding (external versus internal, industry funded versus other), investigator academic rank or whether trials were grouped for common treatment comparisons? 
 Yes = two or more of the above. 
 No = one or none 
 Unclear

Data synthesis

The primary analysis was to compare publication and time to publication for trials with positive findings compared to those with negative or null findings. Studies used slightly different definitions for positive, negative and null findings and therefore we first analysed the data separately for studies using similar definitions. No statistical heterogeneity was observed among studies for the odds ratio estimate and we proceeded with combining results from the individual studies to produce an overall pooled effect estimate using the odds ratio (calculated using the Mantel‐Haenszel method) and a fixed‐effect model with 95% confidence intervals using The Cochrane Collaborations Review Manager software RevMan 5. In a sensitivity analysis, we converted the overall odds ratio (OR) and its 95% confidence interval to a risk ratio (RR) using the following formula: RR = OR/(1 ‐ (R_c x (1 ‐ OR)), where R_c (the control group risk) was the median proportion of negative trials that were published among the included studies. We also examined other factors potentially associated with publication, including funding mechanism, sample size, number of centres, and primary investigator rank and sex.

Results

Description of studies

Included studies

Five studies (Bardy 1998; Dickersin 1992; Dickersin 1993; Ioannidis 1998; Stern 1997) met the inclusion criteria and assessed the proportions of published trials in a cohort of  clinical trials. All of the studies were published as full articles in journals. Studies used slightly different definitions to define trials with positive findings and trials with negative findings. The study by Bardy 1998 assessed the publication of a cohort of clinical trials notified to the National Agency for Medicine in Finland (1987). Trials were classified by the authors as having either positive, negative or inconclusive findings (P values were not given). Dickersin 1992 assessed the publication of clinical trials approved by two institutional review boards (IRBs) in 1980 and Dickersin 1993 assessed the publication of clinical trials funded by the National Institutes of Health in 1979. In both of these studies, trial findings were classified by the primary investigator as either statistically significant, similar, or not statistically significant. When statistical tests were not used, investigators were asked to classify the findings as important or not. Ioannidis 1998 assessed the publication of AIDS trials funded by the National Institutes of Health (1986 to 1996). Trials were classified as either positive if the P value was < 0.05 or favoured the experimental arm of the trial, or as negative if the findings were associated with a P value ≥ 0.05 or favoured the control arm of the trial. Finally, the study by Stern 1997 assessed the publication of clinical trials approved by a local ethics committee (1979 to 1988). Trial findings were classified as statistically significant if the P value was < 0.05, as showing a non‐significant trend if 0.05 < P < 0.10, or as non‐significant or null if no difference was observed between the two groups (seeCharacteristics of included studies).

Excluded studies

Ten studies that were initially assessed as potentially eligible were later excluded from this review (Chan 2004A; Chan 2004B; Cronin 2004; Decullier 2005; Easterbrook 1991; Hahn 2002; Misakian 1998; Melander 2003; Pich 2003; Wormald 1997). Nine studies assessed factors influencing publication in a cohort of studies such as those approved by an ethics committee or local funding body. One study assessed factors influencing publication in a cohort of studies approved by the Swedish Drug Regulatory Authority (Melander 2003). In four studies (Cronin 2004; Decullier 2005; Easterbrook 1991; Misakian 1998) information on the rate of publication for positive versus negative findings was not available separately for reports of clinical trials. In three studies (Hahn 2002; Pich 2003; Wormald 1997) the association between publication and the statistical significance of trial results was not assessed and in two studies (Chan 2004A; Chan 2004B) no data were available on trial findings when trials were unpublished. In the final study (Melander 2003) some trials were registered after the main results had been published (seeCharacteristics of excluded studies).

Ongoing studies

We identified no ongoing studies.

Risk of bias in included studies

We assessed the methodological quality of the included studies using the criteria described in the Methods section. Full details of the methodological quality of the included studies are given in the 'Risk of bias tables'. In two studies (Dickersin 1993; Stern 1997), there was less than 90% follow up of all trials; in the study by Dickersin 1993 there was 86% follow up and in the study by Stern 1997 70% follow up. In the other study (Bardy 1998), the definition of positive and negative findings was unclear, as were control for possible confounders in the analysis.

Effect of methods

Publication and trial findings

(Table 1)

1. Publication and trial findings.

Study ID	Total published	Positive	Negative	Null
Bardy 1998	68/188 (36%)	52/111 (47%)	5/44 (11%)	11/33 (33%)
Dickersin 1992	136/168 (81%)	84/96 (87%)	52/72 (72%)
Dickersin 1993	184/198 (94%)	121/124 (98%)	63/74 (85%)
Ioannidis 1998	36/66 (54%)	20/27 (74%)	16/39 (41%)
Stern 1997	73/130 (56%)	55/76 (72%)	3/15 (20%)	15/39 (38%)

All five studies (Bardy 1998; Dickersin 1992; Dickersin 1993; Ioannidis 1998; Stern 1997) assessed the association between publication and trial findings. The percentage of published clinical trial findings varied greatly across the five studies and ranged from 93% in the study by Dickersin 1993 of National Institutes of Health trials, to 36% in the study by Bardy 1998 of Medicine Control Agency trials. Trials with positive findings were more likely to be published than trials with negative or null findings (odds ratio 3.90; 95% confidence interval 2.68 to 5.68) (Analysis 1.1; Figure 1). This corresponds to a risk ratio of 1.78 (95% CI 1.58 to 1.95), assuming that 41% of negative trials are published (the median among the included studies, range = 11% to 85%). In absolute terms, this means that if 41% of negative trials are published, we would expect that 73% of positive trials would be published. The risk ratio for each individual study is provided in Analysis 2.1 (Figure 2); individual risk ratios were not pooled due to high levels of heterogeneity (I² = 88%; Chi² = 32.32 (df = 4); P < 0.00001).

1.1. Analysis.

Comparison 1 Rate of publication and significance of trial result (pooled), Outcome 1 Total number of trials published.

1.

Forest plot of comparison: 1 Rate of publication and significance of trial result (pooled), outcome: 1.1 Total number of trials published.

2.1. Analysis.

Comparison 2 Rate of publication and significance of trial result (unpooled), Outcome 1 Total number of trials published.

2.

Forest plot of comparison: 2 Rate of publication and significance of trial result (unpooled), outcome: 2.1 Total number of trials published.

Only one of the five studies (Dickersin 1993) provided information separately for trials on why the investigators had not published the trial findings. Some of the reasons were that the trial results were not interesting or that the investigators had not had time (43%), that they had co‐investigator or other operational problems (38%), and that they had additional analyses to complete (14%). In some cases, they did not know the reason.

Time to publication and statistical significance or direction of trial results

(Table 2)

2. Time to publication and significance of results.

Study ID	Time interval	Time to publication	Positive	Negative
Ioannidis 1998	Enrolment to publication	5.5 years (median)	4.3 years (median)	6.5 years (median)
Ioannidis 1998	Completion to publication	2.4 years (median)	1.7 years (median)	3.0 years (median)
Stern 1997	Ethics committee to publication		4.69 years (median)	7.9 years (median)

Two studies (Ioannidis 1998; Stern 1997) assessed time to publication and statistical significance or direction of the trial findings. In both studies the median time to publication was calculated from a survival type analysis of all the eligible trials. In the study by Ioannidis 1998 the median time from start of enrolment to first publication was 5.5 years. This was less for trials with positive findings (P < 0.05, or as defined by the authors of the studies included in this review), with a median of 4.3 years as compared to 6.5 years for trials with negative or null findings (Hazard Ratio (HR) for time to publication for positive versus negative or null findings 3.7; 95% CI 1.8 to 7.7). The median time from completion of follow up to first publication was 2.4 years, which was shorter for trials with positive findings with a median of 1.7 years as compared to 3.0 years for trials with negative or null findings (HR 3.2; 95% CI 1.6 to 6.2). Stern 1997 measured the time interval between approval by the ethics committee and first publication. This was less for trials with positive findings with a median of 4.7 years (95% CI 3.8 to 5.7 as compared to 7.9 years for trials with negative or null findings (95% CI 7.2 ‐ infinity) (HR 4.2; 95% CI 1.7 to 10.3). Further details on these studies are included in a Cochrane methodology review of time to publication for clinical trials (Hopewell 2007B).

Other potential risk factors influencing publication

(Table 3)

3. Other potential risk factors influencing publication.

Study ID	Source of funding	Sample size	Academic rank	Sex
Dickersin 1992		< 100 participants = 86% published (110/128) ≥100 participants = 92% published (34/37)
Dickersin 1993	Grant = 91% published (92/101) Contract = 98% published (58/59) Other = 91% published (34/38)	< 100 participants = 91% published (76/84) ≥ 100 participants = 95% published (102/107)	Professor = 95% published (119/125) Associate / assistant professor = 91% published (20/22) Other = 88% published (45/51)	Female = 88% published (14/16) Male = 93% published (170/182)
Ioannidis 1998		< 200 participants = 51% published 200 ‐ 1000 participants = 79% published > 1000 participants = 67% published

A study of trials funded by the National Institutes of Health in the US (Dickersin 1993) found no significant difference between funding mechanism and publication (grants: 91% published; contracts: 98% published; other funding: 91% published). Studies by Stern 1997 and Dickersin 1992 also assessed the association between publication and source of funding, however, it was not possible to obtain information separately for clinical trials.

Three studies (Dickersin 1992; Dickersin 1993; Ioannidis 1998) assessed the association between publication and sample size. None of the studies had a statistically significant association between sample size and the proportion of trials published. In the study by Dickersin 1993, 91% of trials with a sample size of less than 100 participants were published compared to 95% of trials with a sample size of 100 participants or more. In the study by Dickersin 1992 86% of trials with a sample size of less than 100 participants were published compared to 92% of trials with a sample size of 100 participants or more. In the study by Ioannidis 1998, 51% of trials with a sample size of less than 200 participants were published compared to 79% of trials with a sample size of 200 to 1000 participants, and 67% of trials with a sample size of more than 1000 participants. The study by Stern 1997 also assessed the association between publication and sample size, however, it was not possible to obtain information separately for clinical trials.

One study assessed differences in the proportion of trials published for multi‐site versus single‐site trials and found a difference that was not statistically significant (91% published versus 96% published, respectively, (Dickersin 1993)).

One study (Dickersin 1993) assessed the association between the academic rank of the primary investigator and publication. In this study the differences were not statistically significant between rank and publication (95% of studies with a professor investigator were published; 91% with associate or assistant professor; 88% of other rank were published).

One study (Dickersin 1993) assessed the association between sex of the primary investigator and publication. It did not find a statistically significant association between the sex of the primary investigator and publication (93% for males and 88% for females).

No information was available in the included studies for the following secondary outcome measures: publication in English versus other languages; publications indexed in MEDLINE versus non‐MEDLINE; and type of publication. However, the study by Dickersin 1993 showed that 95% of published trials appeared in MEDLINE‐indexed journals.

Discussion

Despite rigorous searches, only five studies assessing the association between findings and publication in a cohort of clinical trials were identified. These studies showed that trials with positive findings are more likely to be published, and published more quickly, compared to trials with negative findings (odds ratio 3.90; 95% confidence interval 2.68 to 5.68). These findings support those of a closely related Cochrane review assessing full publication of findings initially presented as conference abstracts. Here abstracts of clinical trials with positive findings were also published more quickly and more frequently than those with negative findings (RR 1.18, 95% CI 1.07 to 1.30) (Scherer 2007).

For our primary analysis we used the odds ratio, as planned in the protocol. In a sensitivity analysis using the risk ratio we found substantial heterogeneity (I² = 88%, P < 0.00001) and elected not to report the overall risk ratio. There was no evidence of heterogeneity among the odds ratios (I² = 0%, P = 0.66). To reduce the chances of the odds ratio being misinterpreted (as a risk ratio), we converted the overall odds ratio (OR) and its 95% confidence interval to a risk ratio, assuming that 41% of negative trials were published (the median among the included studies) and found RR 1.78; 95% CI 1.58 to 1.95. It should be noted that the corresponding risk ratio would be larger when a smaller proportion of negative trials were published and smaller when a larger proportion of negative trials were published. This is consistent with the findings of the individual studies (Figure 2) and with what would be expected if the proportion of negative trials that are published is large (85% at the upper range of the included trials).

Data were available from three of the five studies included in our review assessing other risk factors potentially associated with failure to publish. Three showed no statistically significant association between sample size and publication and one study found no statistically significant association between funding mechanism, investigator academic rank, or sex and publication.

It would have been of interest to know whether positive findings are associated with abstract publication, publication in the grey literature, or full publication in indexed journals. However, this information was not available in the included studies.

Other studies not included in this review have assessed the association between publication and trial findings in a cohort of registered studies, but the subset of clinical trials was not available separately (seeCharacteristics of excluded studies). Attempts have been made to contact the authors of these studies but the other data were either no longer available, or the analysis was not carried out.

Only one of the five studies provided information specifically for trials on why the investigators had not published the trial findings. The most common reasons were because the investigators thought the trial findings were not interesting enough or had lack of time. These findings are supported by other studies assessing failure to publish the results of clinical research; here reasons included that the authors thought that a journal was unlikely to accept their study, or because the authors themselves perceived that the results were not important enough (Callaham 1998; Donaldson 1996; Easterbrook 1991; Weber 1998).

In an attempt to determine whether there was any evidence of publication bias occurring after manuscripts were submitted to a journal and during the editorial process, Olson and colleagues (Olson 2002) tracked manuscripts submitted to JAMA until their publication decision. They concluded that there was no statistically significant difference in the publication rates between studies with positive and negative results. There was also no difference in the time from manuscript submission to publication in the journal for studies with positive and negative results (Dickersin 2002).

One of the potential limitations of this review is that the trials included in the studies we have reviewed were undertaken one or more decades in the past. It is possible that publication practices may have changed over the last decade which could change the results of this review, although this is unlikely given the relatively short time span. Indeed a very recent study of Food and Drug Administration registered studies suggests that failure to publish based on the strength and direction of trial findings is still significant a problem. In this study 97% (n = 37/38) of FDA clinical trials with positive findings were published compared to 33% (n = 8/24) of studies with negative findings (Turner 2008). These findings will be incorporated when this Cochrane review is next updated and all searches have been rerun systematically.

This review focuses on one very important aspect of publication bias, that is, publication associated with the trial findings. Other factors are also associated with failure to publish, for example, there is evidence to show that published reports of clinical trials funded by industry are more likely to show positive results than those trials funded by other sources such as government (Bero 1996; Djulbegovic 2000; Kjaergard 2002; Lexchin 2003). Selective reporting of trial outcomes within studies is also a substantial problem, with trialists more likely to report and publish fully outcome measures that have positive results (Chan 2004A). In contrast, there is conflicting evidence as to whether positive findings are more likely than negative findings to be published in an English‐language journals compared to non‐English language journals (Egger 1997; Jüni 2002).

The findings of this review support the need for review authors to search for and include trials in both the published and unpublished literature (Hopewell 2007A) as there is strong evidence to show that there may be systematic differences in the results of these trials. One of the problems faced by those carrying out systematic reviews is how to identify all trials for a particular condition or health care intervention, irrespective of the statistical significance or direction of the trial's results.

Publication bias in health care has been examined over many years (Dickersin 1987; Simes 1986; Simes 1987). There is general agreement that those who carry out systematic reviews need to identify as unbiased and complete a set of relevant studies as possible for inclusion in their review, to minimize biased and misleading results. Statistical methods for detecting publication bias exist but their application can be problematic (Song 2000).

Over the last 25 years, there have been repeated calls to register clinical trials at their inception, to assign unique trial identification numbers, and to record other basic information about the trial so that essential details are made publicly available (Tonks 2002). In September 2004 members of the International Committee of Medical Journal Editors published a statement saying that they would only consider a trial for publication if it has been registered before the enrolment of the first patient (as of 1 July 2005) (De Angelis 2005). This is an important step forward for the prospective registration of clinical trials and one which we hope will be endorsed by other journals. Registration will aid those conducting systematic reviews as it will help protect against publication bias. The World Health Organisation is establishing an International Clinical Trials Registry Platform with the dual aims of improving access to information about clinical trials and their findings, and producing a single worldwide standard for information that trialists should disclose (Gulmezoglu 2005).

Authors' conclusions

Implication for methodological research.

This review focuses on one very important aspect of publication bias, that is, publication associated with the trial findings. A systematic investigation into other potential risk factors associated with publication, such as the selective reporting of outcomes would be warranted.

History

Protocol first published: Issue 3, 2001
 Review first published: Issue 1, 2009

Date	Event	Description
27 December 2007	Amended	Converted to new review format.
20 February 2007	New citation required and major changes	Substantive amendment

Appendices

Appendix 1. MEDLINE search strategy

We searched MEDLINE (1966 to March Week 2 2007) using OVID with the following terms: 
 1. Publication Bias/ 
 2. exp Publications/ 
 3. publication$.tw. 
 4. Publishing/ 
 5. publish$.tw. 
 6. exp Bias Epidemiology/ 
 7. (bias or biases).tw. 
 8. or/2‐5 
 9. or/6‐7 
 10. 8 and 9 
 11. 1 or 10

Appendix 2. EMBASE search strategy

We searched EMBASE (1980 to Week 11 2007) using OVID with the following terms: 
 1. Publishing/ 
 2. publishing.tw. 
 3. Publication/ 
 4. publication.tw. 
 5. (bias or biases).tw. 
 6. or/1‐4 
 7. 5 and 6

Appendix 3. MEDLINE In‐Process and Other Non‐Indexed Citations search strategy

We searched MEDLINE In‐Process and Other Non‐Indexed Citations (March 21 2007) using OVID with the following terms:

1. publication?.tw. 
 2. publish$.tw. 
 3. 1 or 2 
 4. (bias or biases).tw. 
 5. 3 and 4 
 6. (publication bias or publication biases).tw. 
 7. 5 or 6

Data and analyses

Comparison 1. Rate of publication and significance of trial result (pooled).

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Total number of trials published	5	750	Odds Ratio (M‐H, Fixed, 95% CI)	3.90 [2.68, 5.68]
1.1 Positive versus negative or null	5	750	Odds Ratio (M‐H, Fixed, 95% CI)	3.90 [2.68, 5.68]

Comparison 2. Rate of publication and significance of trial result (unpooled).

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Total number of trials published	5		Risk Ratio (M‐H, Fixed, 95% CI)	Totals not selected
1.1 Positive versus negative or null	5		Risk Ratio (M‐H, Fixed, 95% CI)	0.0 [0.0, 0.0]

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Bardy 1998.

Methods	Clinical drug trials notified to the National Agency for Medicine, Finland in 1987. Publication status was obtained by writing to the principal investigator or sponsor of each trial. A MEDLINE search was conducted (1987 ‐ 1995) to identify relevant publications.
Data	188 clinical trials.
Comparisons	Publication status of trials with positive findings compared with those with negative or inconclusive findings. Trials were classified as either positive findings, negative or inconclusive findings. P values were not given.
Outcomes	Trials with positive findings were more likely to be published than those with negative or null findings. Total published = 68/188 (36%) Positive = 52/111 (47%) Negative = 5/44 (11%) Inconclusive = 11/33 (33%)
Notes	274 trials were identified of which 188 were included in the analysis. Reasons for exclusion were: ongoing (n = 9); suspended (n = 64); not commenced (n = 17).
Risk of bias
Item	Authors' judgement	Description
Inception cohort?	Yes	"The material consisted of clinical trials on medicinal products notified to the National Agency for Medicines in 1987." (page 147)
Complete follow up of all trials?	Yes	"With a specific request the status of all but one trial was reported. Of all the 274 trials, 183 were completed, 9 remained ongoing, 64 were suspended and 17 had not commenced." (page 148)
Publication ascertained through personal contact with investigators or sponsor?	Yes	"The sponsors of non‐reported trials were requested by letter to report the outcome of specified trials" (page 147) ... "A Medline search for 1987‐1995 was conducted to identify any publication based on the trials." (page 148)
Definition of positive and negative findings clearly defined?	Unclear	"... the terms "better and inferior" refer to the opinion of the investigator, not to the statistical significance. The expression "not clinically significantly different" also refers to the opinion of the investigators, not to robust statistical evaluation of equivalence or non‐inferiority." (page 148)
Possible confounders controlled for in the analysis?	Unclear	No statistical methods are reported have been applied in order to control for possible confounders.

Dickersin 1992.

Methods	Studies submitted and approved by two institutional review boards (IRBs) which serve the John Hopkins Health Institutions prior to and during 1980. Publication status was obtained in 1988 by a telephone call to the principal investigator of each study.
Data	168 clinical trials.
Comparisons	Publication status of studies with significant findings compared with those with non‐significant findings. Studies were classified as either statistically significant if the P value was < 0.05, or as not significant. When statistical tests were not used, investigators were asked to classify the findings as "important" or not.
Outcomes	Trials with significant findings were more likely to be published than those showing non‐significant findings. Total published = 136/168 (81%) Significant = 84/96 (87%) Non significant = 52/72 (72%) Other variables assessed included sample size, primary funding source, sex and academic rank.
Notes	1048 applications were received by the IRBs of which 514 were included in the analysis. Reasons for exclusion were: applications withdrawn, not approved, not implemented, exempt, did not describe a study, or no humans (n = 311); data on both results and publication not available (n = 223). 273 were observational studies, 73 were experimental studies and 168 were clinical trials and included data on both study results and publication.
Risk of bias
Item	Authors' judgement	Description
Inception cohort?	Yes	"The studies that formed the basis for our research were those that appeared on the logs of the two institutional review boards (IRBS) that serve The Johns Hopkins Health Institutions and were approved in 1980 or prior to 1980 and were still ongoing in that year... the logs of the two institutions ... enumerated 1048 applications." (page 374)
Complete follow up of all trials?	Yes	1048 applications were received by the IRBs of which 514 were included in the analysis. Reasons for exclusion were: applications withdrawn, not approved, not implemented, exempt, did not describe a study, or no humans (n = 311); data on both results and publication not available (n = 223). 273 were observational studies, 73 were experimental studies and 168 were clinical trials.
Publication ascertained through personal contact with investigators or sponsor?	Yes	"The principal investigators associated with interview eligible studies were contacted for interviews in 1988 ... Publication status of a study was determined from responses provided to specific questions asked during the interview." (page 375)
Definition of positive and negative findings clearly defined?	Yes	"Studies reported to have statistically significant findings were combined with those reported to have findings of great importance. Together they are referred to as "significant" and are contrasted with the reminder, which are referred to as "not significant”. In this article we chose to use the term statistically significant to refer to P value less than 0.05." (page 375)
Possible confounders controlled for in the analysis?	Yes	" ... initially, unadjusted ORs for the association between variables listed in table 4 and publication were calculated for each IRB separately using SAS ... Subsequently, adjusted ORs for each IRB alone and for the two IRBs combined (by including a term in the model for the effect of IRB) were calculated using multiple logistic regression. The combined model included two‐way interaction terms between IRB and each of the other factors." (page 376).

Dickersin 1993.

Methods	Completed clinical trials funded by the National Institute of Health (NIH) in 1979 (excluding National Cancer Institute). Publication status was obtained in 1988 by a telephone call to the principal investigator of each study.
Data	198 completed trials funded by the NIH.
Comparisons	Publication status of studies with significant findings compared with those with non‐significant findings. Trials were classified by the primary investigator in terms of statistical significance or classified as not significant. When statistical tests were not used, investigators were asked to classify the findings as "important" or not.
Outcomes	Trials with significant findings were more likely to be published than those showing non‐significant findings. Total published = 184/198 (93%) Significant and published = 121/124 (98%) Non significant and published = 63/74 (85%) Other variables assessed included sample size, funding mechanism, sex and academic rank.
Notes	332 clinical trials were funded by the NIH in 1979 of which 198 were included in the analysis. Reasons for exclusion were: investigator refused to interview (n = 40); not a trial (n = 22); no patients (n = 17); analysis not completed (n = 55).
Risk of bias
Item	Authors' judgement	Description
Inception cohort?	Yes	"We obtained magnetic tapes of the 1979 Inventory of Clinical Trials from the National Institutes of Health. We elected to follow trials funded by all institutes except the National Cancer Institute." (pages 3,4)
Complete follow up of all trials?	No	332 clinical trials were funded by the NIH in 1979 of which 198 were included in the analysis. Reasons for exclusion were: investigator refused to interview (n = 40); not a trial (n = 22); no patients (n = 17); analysis not completed (n = 55). There was 86% of follow up.
Publication ascertained through personal contact with investigators or sponsor?	Yes	"Investigators were asked whether any abstracts, journal articles, book chapter proceedings, letters to the editor, or other material had been published from the trial. If there had been, they were asked for the number of publications and the references. If there had not been any publications, the investigators were asked why not. Publications were classified by whether or not they were in journals indexed by the 1988 Index Medicus." (page 5)
Definition of positive and negative findings clearly defined?	Yes	"Investigators were also asked to characterize the trial findings, either in terms of the results of statistical testing or in terms of the investigator's assessment of the relative importance of the results, when statistical tests were not used. For analysis purposes, responses were classified as falling into 1 of 2 groups: results reported to be statistically significant in either direction were grouped with those deemed to be of "great importance" and classified as "significant". Results showing a trend either direction, but not statistically significant, were grouped with those results designated by investigators to be of "moderate importance" with those results showing no difference, and those designated to be of "little importance". This 2nd group was classified as having "non‐significant" results." (page 4)
Possible confounders controlled for in the analysis?	Yes	"A forward, stepwise logistic regression procedure,17 BMDP LR (BMDP statistical software, Los Angeles, 1990), was used to compute the adjusted OR. The regression model tested the following variables: significance of results, funding, multicenter status, number of study groups, sample size, type of control, use of randomization, masking, type of analysis, PI rank in 1988, and PI sex. Missing value were imputed to the most frequent category." (page 5)

Ioannidis 1998.

Methods	Multi‐centre trials groups in HIV sponsored by the National Institute of Health (NIH) conducted between 1986 and 1996. Publication status was obtained from the trial registry that sponsored the trial.
Data	66 multi‐centre AIDS trials.
Comparisons	Publication status of trials with positive findings compared with those with negative findings. Trials were classified as either positive if P < 0.05 (or favoured the experimental arm of the trial), or as negative if the difference had a P value above 0.05 (or favoured the control arm of the trial).
Outcomes	Trials with positive results were more likely to be published than those with negative results. Total published = 36/66 (54%) Positive = 20/27 (74%) Negative = 16/39 (41%) Other variables assessed included time to publication and sample size.
Notes	109 trials were identified of which 66 were included in the analysis. Reasons for exclusion were: closed as failed to accrue (n = 8); still open to accrual (n = 25); still open to follow up (n = 10).
Risk of bias
Item	Authors' judgement	Description
Inception cohort?	Yes	"All efficacy clinical trials conducted from 1986 until 1996 by the AIDS Clinical Trial Group (ACTG) and by the Terry Beirn Community Programs for Clinical Research on AIDS (CPCRA) were considered in the analysis." (page 282)
Complete follow up of all trials?	Yes	109 trials were identified of which 66 were included in the analysis. Reasons for exclusion were: closed as failed to accrue (n = 8); still open to accrual (n = 25); still open to follow up (n = 10) (page 282)
Publication ascertained through personal contact with investigators or sponsor?	Yes	"Supplemental information about recently analysed trials and clarifications on unclear or missing data were obtained from investigators and medical officers and staff responsible for the protocols." (page 282)
Definition of positive and negative findings clearly defined?	Yes	"In this article, a trial is called "positive" if a statistically significant finding (denoted by P < 0.05) had been found in the analysis of the data for a main efficacy end point defined in the protocol in favour of an experimental therapy arm. Trials with non statistically significant findings or favouring the control arm are called "negative ..." (page 282)
Possible confounders controlled for in the analysis?	Yes	" .. the significance levels of the findings and other trial characteristics were used as covariates for the risk of publication in Cox proportional hazard regression. Trial characteristics included the actual sample size, the ratio of accrual compared with the originally anticipated (target) enrolment (typically based on power calculation), the trialist group, the age of the population (adult or paediatric), the trial domain (antiretroviral therapy vs complication of HIV), the presence or not of double blinding, and the place were data were managed (pharmaceutical industries or other)." (page 282)

Stern 1997.

Methods	Studies submitted and approved by the ethics committee of the Royal Alfred Hospital Sydney between 1979 and 1988. Publication status was obtained in July 1992 by a telephone call to the principal investigator of each study.
Data	130 completed trials.
Comparisons	Publication status of trials with significant findings compared with those with non‐significant or null findings. Trials were classified as either significant if P < 0.05, as showing a non‐significant trend if the difference had a P between 0.05 and 0.10, or as null if no difference was observed between the two groups.
Outcomes	Trials with significant findings were more likely to be published than those showing null findings. Total published = 73/130 (56%) Significant = 55/76 (72%) Non significant trend = 3/15 (20%) Null = 15/39 (38%) Other variables assessed included time to publication, funding and sample size.
Notes	748 studies were included by the ethics committee of which 130 were included in the analysis. Reasons for exclusion were: no response from investigators (n = 228); analysis not yet begun (n = 199); qualitative studies (n = 103); not clinical trials (n = 88).
Risk of bias
Item	Authors' judgement	Description
Inception cohort?	Yes	"Eligible studies were defined as single studies approved by the Royal Prince Alfred Hospital Ethics Committee between September 1979 and December 1988 with more than one patient and with protocol information available." (page 642)
Complete follow up of all trials?	No	748 studies were included by the ethics committee of which 130 were included in the analysis. Reasons for exclusion were: no response from investigators (n = 228); analysis not yet begun (n = 199); qualitative studies (n = 103); not clinical trials (n = 88). There was 70% of follow up.
Publication ascertained through personal contact with investigators or sponsor?	Yes	"In July 1992 the principal investigator for each study was asked to complete a questionnaire providing information on the current status; starting date, closure of recruitment, and finishing date; sample size reached; the nature of funding (none, pharmaceutical, government, other (external), or other (internal)); the rating of scientific importance of the study; the status and date of the most recent analysis; the main research questions posed by the study at the outset; the results for the main research questions; and the publication status and date of initial publication as an article in a peer reviewed journal." (page 642)
Definition of positive and negative findings clearly defined?	Yes	"For quantitative studies, in which the main study outcome was assessed by using statistical methods with tests of significance, outcome was classed as significant (P < 0.05), as showing a non‐significant trend (0.05 < P < 0.10), or as non‐significant or null (P > 0.10) ... For qualitative studies, in which the main study outcome was assessed subjectively by the principal investigator, the study was classed as showing striking, important and definite, or unimportant and negative findings." (page 642)
Possible confounders controlled for in the analysis?	Yes	"With the exception of the investigator's rating of scientific importance of the study, which we judged to be largely influenced by study results, all other factors were examined in a multivariate Cox regression to determine the relative importance of study results on time to publication adjusted for any other significant factors." (page 642)

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Chan 2004A	Assessed factors influencing publication of outcomes in randomised trials approved by the Scientific Committees for Copenhagen and Frederiksberg, Denmark (1994 ‐1995). Data for positive and negative findings were available only for published trials.
Chan 2004B	Assessed factors influencing publication of outcomes in randomised trials approved by the Canadian Institutes of Health (1990 ‐1998). Data for positive and negative findings were available only for published trials.
Cronin 2004	Assessed factors influencing publication of outcomes in 101 research studies commissioned by the North Thames Region of the NHS R & D Programme in the UK (1993 ‐1998). Data were not available separately for reports of clinical trials. The analysis of proportion published was also not available for positive and negative/null findings.
Decullier 2005	Assessed factors influencing publication in 649 research protocols approved by French Research Ethics Committee (1994). Data were not available separately for reports of clinical trials.
Easterbrook 1991	Assessed factors influencing publication in 285 studies approved by the Central Oxford Research Ethics Committee (1984 ‐1987). Data were not available separately for reports of clinical trials.
Hahn 2002	Assessed factors influencing publication of protocols of studies approved by a local ethics committee. This study did not assess the rate of publication for positive and negative findings.
Melander 2003	Assessed factors influencing publication of 42 clinical trials of selective serotonin reuptake inhibitors approved by the Swedish Drug Regulatory Authority (1989 ‐1994). Not all trials were registered prior to the main results becoming known.
Misakian 1998	Assessed factors influencing publication in 84 studies of the effects of passive smoking which were identified through organisations known to fund such research (1981‐1995). Data were not available for reports of clinical trials.
Pich 2003	Assessed factors influencing publication in 166 clinical trials submitted to the Hospital Clinic Ethics Committee, Spain (1997). This study did not assess the rate of publication for positive and negative findings.
Wormald 1997	Assessed factors influencing publication in 68 clinical trials registered with the pharmacy of Moorfields Eye Hospital, London (1963 ‐1993). Data were not available for the rate of publication for positive and negative findings.

Characteristics of studies awaiting assessment [ordered by study ID]

Menzel 2007.

Methods	Clinical trials approved by the Medical Association Westfalen‐Lippen Research Ethics Committee, Germany (1996).
Data
Comparisons
Outcomes
Notes	This study is published in German.

Turner 2008.

Methods	Clinical trials approved by the Food and Drug Administration (FDA) of antidepressant agents between 1987 and 2004. Publication status was obtained by contacting the drug sponsor and by conducting electronic searches.
Data	74 industry‐sponsored trials.
Comparisons	Publication status of trials with positive findings compared with those with negative findings.
Outcomes	Trials with positive results were more likely to be published than those with negative results. Total published = 51/74 (69%) Positive = 37/38 (97%) Negative = 8/24 (33%) Questionable = 6/12 (50%)
Notes

Contributions of authors

Sally Hopewell and Kirsty Loudon conducted the searches (with help from Marit Johansen), assessed trials for inclusion, extracted data, contacted authors for additional information, assessed study quality and wrote the review. Mike Clarke, Andy Oxman and Kay Dickersin contributed to the development of the protocol and commented on drafts of the review.

Sources of support

Internal sources

• NHS Research and Development Programme, UK.

• National Institute of Public Health, Norway.

• Johns Hopkins University, USA.

External sources

• No sources of support supplied

Declarations of interest

Kay Dickersin was the primary investigator of two of the included studies.

Unchanged