Bias in reporting clinical trials

Abstract

Aims

The primary aim of the present study was to identify possible occurrence of selective reporting of the results of clinical trials to the Finnish National Agency for Medicines. Selective reporting may lead to poorly informed action or inaction by regulatory authorities.

Methods

In 1987, 274 clinical drug trials were notified to the Finnish National Agency for Medicines. By December 1993, final reports had been received from 68 of these trials and statements that the trial had been suspended from 24 trials. The sponsors of the non-reported trials were requested to report the outcome. The outcomes, if any, of all reported and non-reported trials were classified as positive, inconclusive or negative.

Results

The total number of trials with positive, inconclusive or negative outcome were 111, 33 and 44, respectively; the outcomes of 86 trials could not be assessed. Final reports were received from 42/111 (38%) trials with positive, 6/33 (18%) with inconclusive and 9/44 (20%) with negative outcomes.

Conclusions

Substantial evidence of selective reporting was detected, since trials with positive outcome resulted more often in submission of final report to regulatory authority than those with inconclusive or negative outcomes.

Introduction

Publication bias results in overestimation of treatment effects arising from published work [1, 2]. Physicians should be familiar with this bias and not base their decisions on sporadic research reports claiming the benefits of new therapy. Regulatory authorities, however, are supposed to receive all relevant published and unpublished information on clinical trials on medicinal products. In Finland [3] the sponsor or the investigator is obliged to submit a final report on the results of the trial to the National Agency for Medicines. If the trial did not commence or has been suspended, the Agency should be notified of the decision with the reasons. Information on the extent of reporting of trial results to national authorities is scarce; it is incomplete at least in Norway [4] and in Finland [5]. Selective reporting of the results of clinical trials on medicinal products could result in inappropriate regulatory decisions. Such decisions include specification of the marketing authorisation and, post-authorisation, variation, revocation or suspension.

The primary aim of the present study was to estimate the possible occurrence of selective reporting of the results of clinical trials to the Finnish National Agency for Medicines. The possible association between outcome of the trials and reporting is here referred to as ‘reporting bias’. The secondary aim was to estimate association between outcome of the trials and publishing the results in medical journals, publication bias. The tertiary aim was to assess the quality of the trials.

Methods

The material consisted of 274 clinical trials on medicinal products notified to the National Agency for Medicines in 1987. Of these trials 199 were sponsored by foreign or multinational companies, 67 by Finnish companies and 8 by Finnish research groups. A preliminary report of this study has been published previously [5]. Until late 1993, final reports had been received from 68 trials, these were classified as reported trials. Statements that the trial had been suspended were received from 24 trials, these were classified as reported suspensions.

The sponsors of the 182 non-reported trials were requested by letter to report the outcome of those specified trials. The trial outcomes, if any, of all reported and non-reported trials were classified as positive, inconclusive or negative. The trial outcome as interpreted by the investigator (not by the present author), was regarded as positive if:

• assessing risks and benefits, the drug under investigation was better than its comparative, either placebo or an active drug, or

• assessing risks and benefits, the drug under investigation was not clinically significantly different from that of an established comparative, or

• the objective of the study was supported or confirmed.

• The trial outcome as interpreted by the investigator (not by the present author), was classified as inconclusive if:

• the primary aim of the study was exploratory, e.g. to determine the pharmacokinetic characteristics of the drug, or

• the risk-benefit assessment was inconclusive, or

• the study was non-comparative.

The trial outcome as interpreted by the investigator (not by the present author), was classified as negative if:

• assessing risks and benefits, the drug under investigation was inferior to its comparative, either placebo or active drug, or

• assessing risks and benefits, the drug under investigation was not clinically significantly different from placebo, or

• the objective of the study was not supported or was rejected, or

• the trial was suspended due to lack of efficacy of the drug under investigation, or

• the trial was suspended due to lack of safety of the drug under investigation.

In these rules, the terms ‘better’ and ‘inferior’ refer to the opinion of the investigator, not to statistical significance. The expression ‘not clinically significantly different’ also refers to the opinion of the investigator, not to robust statistical evaluation of equivalence or non-inferiority.

A Medline search for 1987–1995 was conducted to identify any publications based on the trials. The searches were conducted using the generic name of the drug (if available), the code name of the drug (if no generic name was available), the name of the disease and the name of the principal investigator. The publications were identified by comparing them with trial protocols. Trials resulting in publications in journals included in Medline were classified as published, while those not included were classified as unpublished.

The quality of the trial protocols was assessed by using a reduced list of quality assessment questions [6] in which the value 1.0 indicated highest and 0.0 lowest possible quality. Two modifications of the original method were used: (1) the trial protocols were assessed instead of the reports and (2) assessment was performed by one unblinded examiner.

The χ² test was used to calculate the association between trial outcome and reporting or publishing of the results. One-way analysis of variance and t-tests were used to compare the quality of the study protocols in various groups. Significance levels ≤0.05 were regarded as statistically significant.

Results

Before requests were sent from the national authority, final reports had been received from 68 trials only and statements that the trial had been suspended from 24 trials. With a specific request the status of all but one trial was reported. Of all the 274 trials, 183 were completed, 9 remained ongoing, 64 were suspended and 17 had not commenced. The reasons for suspension were adverse events in 17, scarcity of suitable patients in 19 and reasons associated with investigator in 11, trial site in 12 and sponsor in 5 trials. Only seven of the 17 suspensions resulting from adverse events were reported to the agency without a specific request.

One hundred and eighty-eight trials resulted in classifiable outcomes that were positive in 111, inconclusive in 33 and negative in 44 trials. The association between outcome and reporting to the national authority is given in Table 1. A significant reporting bias was detected. Trials with positive outcome were significantly more likely to be reported than those with inconclusive or negative outcomes.

Table 1.

The number (percentage of outcome group) of reported and non-reported trials, by outcome. There was a significant reporting bias, i.e. association between trial outcome and submission of final report: χ²=7.30, df=2, P=0.023. 86 trials not resulting in any outcome are excluded.

If reported suspensions were added to reported trials, no statistically significant association was present (Table 2). Thus, when considering both reported trials and reported suspensions, the information submitted to the agency was incomplete but not systematically biased. Final reports of 28 trials with negative outcome were not submitted without specified request, among them a trial indicating that treatment with a drug under investigation resulted in increased mortality in patients with ischaemic brain infarction.

Table 2.

The number (percentage of outcome group) of reported trials or reported suspensions and non-reported trials or non-reported suspensions, by outcome. There was no significant association between trial outcome and submission of final report or reported suspension: χ²=3.19, df=2, P=0.20. Eighty-six trials not resulting in any outcome are excluded.

The results of 68 trials were published in journals included in Medline. The association between outcome and publication is given in Table 3. There was a significant publication bias. Trials with positive outcome were more likely to be published than those with inconclusive or negative outcomes. Thirty-nine unpublished trials showed a negative outcome, among them one (the trial mentioned in the previous paragraph) indicating that treatment with a drug under investigation resulted in increased mortality in patients with ischaemic brain infarction.

Table 3.

The number (percentage of outcome group) of trials resulting and not resulting in published articles in the journals included in Medline, by outcome. There was a significant publication bias, association between trial outcome and publication: χ²=17.3, df=2, P<0.001. Studies not resulting in any outcome are excluded.

The mean quality (95% confidence interval) of all trials was 0.55 (0.52 to 0.58) and the quality of trials with positive, inconclusive or negative outcome were 0.56 (0.52 to 0.60), 0.53 (0.47 to 0.59) and 0.59 (0.54 to 0.64) respectively. The mean quality (95% confidence interval) of the trials resulting in either submitted final report or information on suspension was 0.57 (0.53 to 0.61) and of other trials 0.55 (0.52 to 0.58). The mean quality (95% confidence interval) of published trials was 0.57 (0.52 to 0.62) and of unpublished trials 0.55 (0.52 to 0.58). No statistically significant differences in quality were detected among any of the groups.

Discussion

The present study revealed the presence of a significant ‘reporting bias’ in the submission of final reports to the national authority. Trials with positive outcome were significantly more likely to be reported than those with inconclusive or negative outcome. If reported suspensions were added to reported trials, no statistically significant bias was present. However, the information submitted to the agency was incomplete. The outcomes of 182 (66% of total) trials were first reported with specified request from the national authority. These trials included 28 with negative outcome. When assessing the efficacy and safety of a medicinal product, it is imperative to consider the outcomes of all clinical trials, not only those reported to national authority without request. However, at present information on suspended trials is only available for regulatory authority of the country (or countries in multinational trials) where the trial was conducted. This could result in inappropriate decisions.

Was the material (i.e. all trials notified to the Finnish authority over one year) representative of trials conducted elsewhere? Similar studies have not been conducted in other countries, but reporting is also incomplete in Norway [4]. Of the 274 trials, 199 were sponsored by foreign or multinational companies who also conduct clinical trials in other countries, while 67 trials were sponsored by Finnish companies, some of whom also conduct clinical trials outside Finland. No obvious reason exists for assuming that the results obtained should not also be valid in other countries. The trials included in this study were notified 10 years ago. The number of final reports and notified suspensions submitted to Finnish national authority is still only one-third of the number of notified trials; thus the reporting has probably not changed since 1987.

The outcomes of the trials were classified as positive, inconclusive or negative according to the statements of the investigators, using simple rules; thus the classification was not influenced by the subjective opinion of the present author. This was regarded as important when considering the objective of this study. If the outcomes had been classified using robust statistical rules, the number of trials with inconclusive or negative outcomes would have probably been increased. For example, in equivalence or non-inferiority trials a clinically acceptable difference was often not defined before performing the analysis of the results.

The occurrence of significant publication bias in the journals included in Medline was substantiated. This result is in line with earlier studies [2]. Some trials were published elsewhere, in journals published by medical companies or in journals with limited readership and often only in Finnish. These journals are seldom referred to, are difficult to locate and have a minor impact on medical practice. When performing meta-analysis efforts should be made to find all relevant publications, not only those readily available.

The assessment of quality of was in part subjective. The duties of the author include control of clinical trials conducted in Finland; effective blinding was thus not possible. Assessment was modified to include only trial protocol, not the final report. This method was chosen to enable the assessment of trials that did not result in reports. The quality of the trial protocols, a tertiary variable in the present study, was not significantly different between reported and non-reported or published and unpublished trials, nor were differences present among trials with positive, inconclusive or negative outcomes. Despite the sub-optimal method of quality assessment, the occurrence of major differences was unlikely, thus the reporting and publishing were probably not influenced by the quality of trial design.

What are the consequences of reporting bias and publication bias? An overly positive risk-benefit assessment can result in inappropriate regulatory decisions. New and expensive medicinal products may be used instead of older, cheaper and thoroughly investigated products. Inefficient or unsafe experimental therapies may be retried by other investigators, unaware of the outcome of previous trials. Treatment of ischaemic brain infarction with one product resulted in increased mortality; this outcome was neither reported to the Finnish authority nor published in any journal. The failure of such information to reach the medical community is a serious hazard. The publication of research results is an ethical imperative [7, 8].

If the selection of final research reports submitted to national authorities and articles published in medical journals are found to be biased, the question should be asked whether unbiased data on clinical drug trials are available elsewhere. At present, however, it seems that such data are not readily accessible. Meta-analyses performed by pooling results of clinical trials may suffer from incomplete reporting. If meta-analysis is needed to demonstrate efficacy, the treatment may be ineffective [9, 10]. National authorities should actively request the outcome of all trials, and especially the reasons why trials have been suspended. Marketing authorisation applications should include information on all trials with the product in any country. Efforts should be made to establish multinational databases that should include pertinent information on all clinical trials, including those notified, ongoing, suspended and completed [11–13]. Such databases should be accessible to regulatory authorities and preferably to the entire medical community.