Undisclosed Changes in Outcomes in Randomized Controlled Trials: An Observational Study

Abstract

PURPOSE We wanted to investigate the frequency of undisclosed changes in the outcomes of randomized controlled trials (RCTs) between trial registration and publication.

METHODS Using a retrospective, nonrandom, cross-sectional study design, we investigated RCTs published in consecutive issues of 5 major medical journals during a 6-month period and their associated trials registry entries. Articles were excluded if they did not have an available trial registry entry, did not have analyzable outcomes, or were secondary publications. The primary outcome was the proportion of publications in which the primary outcome of the trial was, without disclosure, changed between that recorded in the trial registry and that reported in the final publication. The secondary outcome was the proportion of publications in which the secondary outcome was changed without disclosure.

RESULTS We reviewed 158 reports of RCTs and included 110 in the analysis. In 34 (31%), a primary outcome had been changed, and in 77 (70%), a secondary outcome had been changed.

CONCLUSIONS There are substantial and important undisclosed changes made to the outcomes of published RCTs between trial registration and publication. This finding has important implications for the interpretation of trial results. Disclosure and discussion of changes would improve transparency in the performance and reporting of trials.

You want more scepticism at the very foundation of your work. Scepticism, the tonic of minds, the tonic of life, the agent of truth, the way of art and salvation.

Joseph Conrad, in a letter to John Galsworthy, 1901

INTRODUCTION

The primary outcome (or endpoint) is an essential element of a randomized controlled trial (RCT), and specification of outcomes is central to the design and reporting of these trials. Changing the outcomes of a clinical trial without explanation calls its validity into question. To allow authors to change their outcomes without reason or disclosure threatens the underpinnings of the scientific enterprise and threatens the ability of practicing physicians to apply published research to patient care. Acceptable reasons to change outcomes include results of other trials, intertrial comparisons, theoretical advances in the relevant field, and technical advances (for example, in measurement or treatment). These changes present few problems if they are reported and interpreted appropriately. Unfortunately, outcomes may be changed in ways that violate the fundamental principles of trial design and interpretation¹ and, therefore, raise doubts about the validity of the conclusions. These changes, if made, should be carefully reasoned, limited in number, and clearly labeled as hypothesis generating rather than hypothesis supporting.

During the past several years efforts have been made to encourage “full transparency with respect to performance and reporting of clinical trials,”² because the “credibility of research depends on a critical assessment by others of the strengths and weaknesses in study design, conduct, and analysis.”³ As one part of this larger endeavor, most major medical journals and many funding agencies now require that trial details be registered. Registration includes the “definitions of the primary and secondary outcome measures.”² The purpose of our current research is to estimate the extent to which outcomes are changed, without disclosure in the final report, between trial registration and publication.

How prevalent is this problem of changing outcomes? Chan and colleagues^4,5 and Hahn and colleagues⁶ independently have looked at changes in outcomes that occur between trial protocols and publication (or nonpublication). This approach is likely to give an accurate estimate of outcome changes, because researchers are required to submit amendments to trial protocols to their institutional review boards. Protocols, however, are not usually available (although Hawkey⁷ has argued that they should be), making it nearly impossible for a typical practicing physician to critique trials using this information. Trial registries are an almost ideal solution. They are easy to find on the Internet, and the information is well-organized and concise. This system provides easier access and better transparency for the typical journal reader. With this background in mind, we set out to examine how often outcomes are changed between trial registration and final publication.

METHODS

Our primary outcome was the proportion of trials in which the primary outcome of the trial under consideration had been changed without disclosure from registration to publication. Our secondary outcome was the proportion of trials in which the secondary outcome of the trial under consideration had been changed without disclosure from registration to final publication.

To determine the frequency of changes in outcomes, we assembled consecutive issues of 5 major medical journals. These 5 journals were chosen because they have high impact factors and are widely considered of high quality. We identified all reports of randomized controlled trials for a 6-month period. We searched each report for a reference to a trials registry. We then excluded trials for the following reasons: there was no publicly accessible trial registration recorded in the report; the registration database did not record a primary outcome or the outcome recorded was too vague to make any judgments; and the trial publication was not the main report of the trial results. If there were multiple reports from the same trial, we used what appeared to be the main one. If multiple reports considered different outcomes of the same study, we either combined them and considered them together as one study or considered them as separate studies depending on the circumstances. We kept a database of all excluded trials.

For each included report we extracted the primary and secondary outcomes as listed in the registration database and in the final publication (or, if necessary, in the associated design and methods publication). We did not search the archives of the trials registries to see whether the outcomes had been changed since registration. Rather, we took them as is on the day we searched. For quality assurance, each of the 3 authors reviewed all the articles independently. The few disagreements were resolved easily by consensus after reviewing the articles a second time. The authors were not masked to the identity of the research reports.

We considered outcomes to have been changed only if there were substantive differences, rather than minor differences in wording that did not appear to change the intent of the trial. We counted outcomes as unchanged if the authors acknowledged the change and made any statement indicating that the changes were made before any analyses were done. Secondary outcomes were counted as unchanged if the authors said they would be published separately (although we made no attempt to confirm event).

As this sample of articles was not generated randomly, the assumptions underlying the calculation of confidence intervals do not hold; therefore, the confidence intervals presented here (calculated by the modified Wald method and at the 95% level⁸) should be interpreted cautiously.

RESULTS

We found a total of 158 RCTs in the 5 journals (Table 1▶). Forty-eight were excluded, and 110 were included in the analysis. Trials were registered in 5 databases. Table 2▶ gives further details of the included and excluded trials, and Table 3▶ gives further details on the registry databases.

Table 1.

Study Flow (N = 158)

Characteristics	No.
Excluded trialsa	48
No, incorrect, or not publicly accessible trial registration database recorded in the main publication	11
Registry did not record a primary outcome, or the outcome recorded was too vague to use	20
Publication was not the main report of the trial results	23
Included trials
Information extracted: study reference and copy of study, trials registration reference and copy of registration, primary and secondary outcomes in the final report, primary and secondary outcomes in the trials registration database	110

Note: Randomized controlled trials reported in Annals of Internal Medicine, British Medical Journal, The Journal of the American Medical Association, The New England Journal of Medicine, and The Lancet between September 2006 and February 2007.

^a Results total to more than 48, as several trials were excluded for more than 1 reason.

Table 2.

Journals Publishing Trials Reports Reviewed in the Study

Journal	Included Trials	Excluded Trials	Total
Annals of Internal Medicine	12	3	15
British Medical Journal	20	11	31
The Journal of the American Medical Association	15	7	22
The Lancet	19	12	31
The New England Journal of Medicine	44	15	59
Total	110	48	158

Table 3.

Databases of Reported Trials

Database or Registry	No. of Reports	Comments
Australian Clinical Trials Registry	2	No longer active. Merged into Australian New Zealand Clinical Trials Registry (ANZCTR: http://www.anzctr.org.au) Managed by NHMRC Clinical Trials Centre, University of Sydney
Clinicaltrials.gov (http://www.clinicaltrials.gov)	112	Managed by US National Institutes of Health
European Clinical Trials Database (http://eudract.emea.europa.eu)	1	Not publicly searchable. Managed on behalf of the European Commission
International Standard Randomised Controlled Trial Number Register (http://www.isrctn.org)	30	Managed by Current Controlled Trials, Ltd (http://www.controlled-trials.com). Also contains a searchable meta-register of other trials databases
National Research Register	1	No longer active. Archive available at: http://portal.nihr.ac.uk/Pages/NRRArchive.aspx. Managed by UK National Health Service – National Institute for Health Research

Note: Eleven reports did not list a registration database.

Thirty-four reports (31%; 95% confidence interval [CI], 23%–40%) had a change in one or more primary outcomes. Seventy-seven reports (70%; 95% CI, 61%–78%) had a change in one or more secondary outcomes.

For the primary outcomes, most changes were by deletion. For the secondary outcomes, the changes were more evenly split between addition of outcomes and deletion. Table 4▶ expands on these results. Although not part of our research question, we noted that there were almost no differences in outcomes when comparing trials funded by pharmaceutical companies with those that had noncommercial sponsorship. In Table 5▶ are 3 examples of changes made between registration and publication. These were representative of the different types of changes we encountered.

Table 4.

Summary of Outcome Changes of Included Studies (n = 110)

Study Characteristic	No.	% (95% CI)
Primary outcomes
No change	76	69 (60–77)
Change	34	31 (23–40)
Change by addition	10	–
Change by promotion from secondary	3	–
Change by deletion	20	–
Change by demotion to secondary	6	–
Secondary outcomes
No change	33	30 (22–39)
Change	77	70 (61–78)
Change by addition	54	–
Change by demotion from primary	5	–
Change by deletion	48	–
Change by promotion to primary	3	–

Note: Studies could have multiple changes.

Table 5.

Examples of Changes Made Between Trial Registration and Publication of Findings

	Primary Outcomes		Secondary Outcomes
Study Description	Registry	Publication	Registry	Publication	Notes
Kaplan et al9,10	1. Patient perception of treatment benefit at week 12	1. Reported	24 outcomes grouped in several domains	7 outcomes, 2 not mentioned as out- comes in registry	Several clinically important secondary outcome measures listed in the registry (continence, overactive bladder, erectile function) are not mentioned in the publication
Safety and efficacy of tolterodine ER, tamsulosin, or the combination, against placebo in men with lower urinary symptoms	2. Treatment benefit, treatment satisfaction, and willingness to continue treatment questions at week 12	2. Not reported as a primary outcome but some measures reported as secondary outcomes
Bosset et al11,12	1. Disease-free survival	1. Not reported	None listed	None listed	Disease-free and progression-free survival both discussed in the publication, but not as outcomes
Treatment of rectal cancer with radio- therapy and 5-FU	2. Overall survival	2. Reported
Michalowicz et al13,14	1. Gestational age at birth	1. Reported	None listed	1. Birth weight	Demotion of 1 primary outcome and addition of multiple secondary outcomes
Effect of treatment of periodontal disease on the risk of pregnancy complications	2. Birth weight	2. Reported as a secondary outcome		2. Infants small for gestational age
				3. Apgar scores
				4. Admissions to NICU

ER=extended release; 5-FU=5-fluorouracil; NICU = neonatal intensive care unit.

DISCUSSION

Several biases could have effected our results. It is possible that the outcomes in the trial registry were incorrect. For example, difficulty in database formats, errors in data entry, misperception of registry importance, or deliberate falsification could all invalidate the entries. Trials registries are not useful, however, if they do not provide an accurate representation as to what was intended.

It is also possible that the authors of the articles had made changes in their registry entries after data analysis but before publication, and our search would not have uncovered this change. During our data-checking process, we found 3 instances in which authors had changed their registry entries after trial publication. In all 3 cases, the original registry entry had not listed any outcomes,^15–17 but the subsequent changes listed outcomes identical to the final publications.^18–20 If these sorts of changes were common, we would have underestimated the frequency of outcome changes.

Further, our selection of articles was not random. We selected articles from what are often considered better journals. This focus could have resulted in a lower percentage of changed outcomes (as these articles are presumably reviewed more closely before publication) or a higher percentage (as the financial stakes may be higher with a correspondingly higher incentive to produce good results).

Finally, our exclusion criteria were subjective, and other researchers reviewing our database might make slightly different decisions. Although almost all of the decisions on changes in outcomes were straightforward, other reviewers of these same data might arrive at slightly different results. We doubt, however, that the conclusions would change.

Outcomes can be changed either by addition, deletion, or by some combination of the 2. The issue with addition of outcomes is that it asks the question after the answer is known. Deletion, while more subtle, may also turn a statistically insignificant result into a significant one. When a study has multiple outcomes, one may be significant by chance alone. In the past, authors have been tempted to mine data for statistically significant outcomes after a study was completed. It is well-understood that such activity is unacceptable.^21,22 Instead, authors may be tempted to list multiple outcomes in advance and then eliminate those that were not significant—in essence, data mining in reverse. Any use of multiple outcomes requires P value adjustments to account for chance findings of statistical significance. Reducing the number of outcomes can, therefore, make an outcome appear significant where it would not have been before the reduction. Although there is considerable disagreement in the statistical literature about the best approach to doing these adjustments, there is no argument about the necessity.²³

We suggest several potential remedies for the issues discussed here. (1) The International Committee of Medical Journal Editors²⁴ requirement that authors register the World Health Organization Minimal Registration Data Set²⁵ should be enforced by all databases (or all journals). For example, the American registry does not require the registration of any outcomes,²⁶ an omission that is easily remedied. (2) Trial registration databases should have the archive of each trial linked directly to the trial registration, which would make it easier to follow changes. (3) Our investigation was about transparency, not misrepresentation. We believe most changes were made in good faith and without an understanding of the issues involved. To aid authors in avoiding these problems, the next revision of the CONSORT statement²⁷ should require reporting of more details about trial registration and changes that have been made therein.²⁸ (4) Journal editors should expect that trials be registered at the outset and certainly before any interim statistical analyses are performed. (5) Reviewers should routinely check the trial protocol, the registry, and any quoted design and methods publications to ensure that the authors have not altered their outcomes without appropriate comment and justification.

Although our results must be regarded as only approximate, they nevertheless indicate a substantial and important lack of transparency in the reporting of clinical trials.