Key Points
Question
Does primary outcome change between trial registration and publication alter a randomized clinical trial’s reported intervention effect size?
Findings
In this cross-sectional study that included 389 trials, 130 of them had at least 1 primary outcome change between registration and publication. This significantly overestimated the reported intervention effect size by 16% compared with those without primary outcome change.
Meaning
Inconsistencies between registered and published primary outcomes in clinical trials are common, with a larger reported intervention effect size among those with primary outcome change than those without.
This cross-sectional study examines the status of randomized clinical trials (RCTs) whose primary outcome changed between trial registration and publication and quantifies the association of this change with the reported intervention effect size.
Abstract
Importance
Primary outcome change could threaten the validity of a clinical trial; however, evidence about the consequences on the reported intervention effect size is unclear.
Objectives
To examine the status of randomized clinical trials whose primary outcome changed between trial registration and publication and to quantify the association of this change with the reported intervention effect size.
Design, Setting, and Participants
In this cross-sectional study on the primary report of randomized clinical trials with clear prospectively registered primary outcomes, PubMed and Embase were searched for articles published between January 1, 2011, and December 31, 2015. The search was conducted in January 2016, identifying randomized clinical trials and the combination of keywords and text words related to registry.
Main Outcomes and Measures
Based on the developed approach, trials were classified as having primary outcome change when there was a major discrepancy between the registered and published primary outcomes. Intervention effect was estimated or recalculated using the odds ratio (OR) for each comparison. Each component OR is structured so that an OR is less than 1 if the intervention group has a more favorable result than the control group. The ratio of ORs (ROR), which is the summary OR for trials with primary outcome change divided by those without, and its 95% CI were calculated, with a value less than 1 indicating a larger reported intervention effect size in trials with primary outcome change than those without.
Results
Among 29 749 searched articles (28 810 MEDLINE and 939 Embase), 1488 articles were randomly selected for review. Of 389 trials with clear primary outcomes prospectively described in the registry (416 outcomes reported), 33.4% (130 of 389) of trials had at least 1 primary outcome change. Most (66 of 130) of the changes were either not reporting or omitting the primary outcome. In total, 338 trials (365 outcomes and 487 comparisons) were available for quantitative analysis on the reported intervention effect size bias assessment. Compared with those without primary outcome change, trials with primary outcome change showed a 16% (pooled ROR, 0.84; 95% CI, 0.73-0.96) larger reported intervention effect size. The result persisted after adjustment for potential confounders (ROR, 0.81; 95% CI, 0.71-0.93) and other sensitivity and subgroup analyses.
Conclusions and Relevance
Results of this study suggest that inconsistencies between registered and published primary outcomes of clinical trials are common, and trials with primary outcome change are likely to have a larger intervention effect than those without.
Introduction
Randomized clinical trials (RCTs) have a crucial role in assessing the efficacy and safety of a treatment and in advancing medical knowledge.1 Clinical trials of investigational medicinal products have been legally required to be registered before participant enrollment since January 5, 2004; furthermore, to improve transparency of results, the International Committee of Medical Journal Editors member journals have required since January 7, 2005, that for publication clinical trials of any intervention should be preregistered.2 Registering a trial is mostly free, and options include ClinicalTrials.gov, EU Clinical Trials Register, and International Standard Randomized Controlled Trial Number (ISRCTN) register. Although the reported information differs between registries, a clearly defined and prespecified primary outcome is an important element.3,4 Discrepancies between registered and published outcomes may imply selective outcome reporting based on significant P values.5,6 This practice could threaten the validity of clinical trials by producing conclusions that may mislead physicians and policy makers.6,7,8
Although it is well recognized that registries are important tools to reduce the risk of selective reporting of outcomes, Jones et al7 found that consistency between planned and published outcomes varied substantially among 27 eligible studies, with a median consistency proportion of 31% (interquartile range, 17%-45%). Similarly, another study9 analyzing all interventional trials registered on ClinicalTrials.gov from 1999 to 2012 showed that 32% of trials had their primary outcomes altered between the listed study start and completion dates. However, previous studies in this area have focused on specific design characteristics (eg, pain or continuous outcomes) and have not attempted to quantify to what extent a primary outcome change will alter the invention size that is being estimated.
The objectives of this study were 2-fold. The first objective was to estimate the proportion of RCTs that had a primary outcome change, without restriction by journals, diseases, or registry entries. A second objective was to quantify the consequences associated with primary outcome change on the reported intervention effect size.
Methods
This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for cross-sectional studies. The study was not submitted for institutional review board approval because all data are publicly available.
Search Strategy
In this cross-sectional study, we studied the primary report of registered RCTs published between January 1, 2011, and December 31, 2015. A search was conducted in January 2016 using the suggested filters from the Cochrane Collaboration (MEDLINE via PubMed) and the BMJ Evidence Centre (Embase via Ovid) for identifying RCTs and the combination of keywords and text words related to registry. No restrictions by journal, disease, or outcome type (ie, continuous or dichotomous outcome) were applied except that reports had to be in English.
We then randomly selected 5% of the retrieved records from each year. Details of the search strategy and sampling method are listed in eTables 1, 2, and 3 in the Supplement, and our study protocol was registered on PROSPERO.10
Study Selection and Eligibility
Articles were screened for relevance by title and abstract and then by the full text to identify primary trial reports. During this process, one of us (T.C.) excluded duplicate publications, protocol studies and analysis plans, system reviews or meta-analyses, and feasibility, pilot, or phase 1 studies, as well as ancillary studies (eg, subgroup analyses, exploratory analyses, secondary outcome analyses, preliminary results, interim analyses, post hoc analyses, pooled analyses, cost-effectiveness analyses, and mechanism research). Next, for each published trial report, we identified the registration number via published articles or clinical trial registries (ClinicalTrials.gov, ISRCTN register, or country-specific registries). We only included trials that were registered before study completion and gave a clear description of the primary outcome in the registry. Two of us (T.C. and R.Q.) checked the full-text articles in the next 2 processes.
Data Extraction and Risk of Bias (Quality) Assessment
We assigned a unique identification number to each trial included in this study. Data were extracted using a standardized extraction form. For published articles, we extracted the publication information (eg, author name and year of publication) and study characteristics, including study design (eg, noninferiority, superiority, or equivalence), sample size for each group, randomization method (eg, cluster or individual), and type of outcome (eg, time to event, binary, or continuous). For registered information, we extracted the following information: start and end dates of participant enrollment, registration date, the last amendment date, originally registered primary outcomes, and amended outcomes (if applicable).
The risk of bias (RoB) for each trial was assessed by the RoB tool as recommended by the Cochrane Collaboration. Overall RoB was assessed as low risk (low for all Cochrane Collaboration components), high risk (high for ≥1 Cochrane Collaboration component), or unclear risk (unclear for ≥1 Cochrane Collaboration component). Other RoB items included intent-to-treat (ITT) status, trial centers, and source of funding. We defined and classified these items according to published references as follows: deviation from ITT principle (ie, ITT, modified ITT, or no ITT/unknown),11 trial centers (ie, multiple centers or single center),12 and source of funding (ie, public funding, cofinanced, for-profit funding, not funded, or not reported) (with not-for-profit funding and not funded considered high risk).13
Outcome of Interest
A major discrepancy was defined if the registered and published primary outcomes were different or were assessed at a different time point. This definition is according to a modified classification by Chan et al.6 Specifically, the following were considered major discrepancies: (1) a prespecified primary outcome in the trial registration protocol was subsequently reported as a secondary outcome in the final published article; (2) The published primary outcome was described as a secondary outcome in the registry; (3) The prespecified primary outcomes in the trial registration were either omitted or not reported or labeled from the published article; (4) a new primary outcome was introduced in the published article (eg, an outcome that does not appear at all in the registry but is introduced as primary in the article, or one of the components changed among a composite outcome); and (5) the timing of assessment of the primary outcome in the registered protocol and published article differed.
Inconsistencies were independently identified by 2 of us (T.C. and C.L.), and disagreements were resolved by discussion until consensus (κ coefficient, 0.92; 95% CI, 0.88-0.96). If changes to the registered primary outcome were made, they were further reviewed by another one of us (V.C.), and disagreements were resolved by consensus.
Statistical Analysis
Categorical variables were described by frequencies and percentages and quantitative variables by medians and ranges. We used the κ coefficient to determine the degree of agreement between reviewers.
To quantify overall the consequences associated with primary outcome change on the reported intervention effect size across different types of outcomes, we assumed relative risk, hazard ratio, and odds ratio (OR) to be the same measure. This strategy has been used in published meta-analyses of observational studies.14,15 In our study, we considered ORs to be a common estimate, but heterogeneity by different types of outcomes was explored in subgroup analyses. For continuous outcomes, we converted them to ORs according to the method by Hasselblad and Hedges by multiplying the standardized mean differences and their SEs by 1.81 to calculate the log ORs and the corresponding SEs.16
For each comparison, we estimated the OR between the intervention group and the control group. Where necessary, we inverted the effect size so that each comparison was indicated by an OR less than 1 if the intervention group had a more favorable result than the control group.
Because an enrolled trial may contribute 2 or more comparisons due to multiple groups and/or outcomes, we used a linear mixed model with the log ORs of each comparison as the dependent variable, primary outcome change as a fixed effect, and study identification number as a random effect after weighting the inverse variation of the log OR of each comparison. Differences are presented by estimating the ratio of ORs (ROR) after anti–log transformation. The ROR is the summary OR for trials with primary outcome change divided by those without, with a value less than 1 indicating a larger reported intervention effect size in trials with primary outcome change than those without. To test the robustness of our study, we conducted 4 sensitivity analyses. First was a mixed model with adjustments for trial characteristics (ie, deviation from ITT principle, study design, trial centers, type of comparator, randomization method, type of outcome, source of funding, and RoB). Second was a mixed model using primary outcome change based on reviewer 1 assessment only or reviewer 2 assessment only. Third, we repeated the mixed model but excluded trials in turn with (1) different levels of RoB (low risk, high risk, or unclear risk), (2) different type of outcomes (time to event, binary, or continuous), and (3) multiple outcomes and/or multiple groups. Fourth was a mixed model with the inverse variation of the log OR as additional covariates rather than weights.
We also carried out subgroup analyses according to prespecified characteristics. These included deviation from ITT principle, study design, trial centers, randomization method, type of outcome, source of funding, and overall RoB.
All data analyses were performed using statistical software (SAS, version 9.4; SAS Institute Inc). P values were 2 tailed, and P < .05 was considered statistically significant.
Results
Among 29 749 searched articles (28 810 MEDLINE and 939 Embase), 1488 articles were randomly selected for review. After excluding 864 reports by reviewing the titles and abstracts, we identified 624 potentially eligible publications. Of them, 65 were further excluded after reading through the full text. After comparing the remaining 559 publications with online registry, we found that 4 trials (0.7%) were not registered, 92 trials (16.5%) trials were registered after completion of the study, and 74 trials (13.2%) were registered with no description or unclear description of the primary outcome. Figure 1 shows the screening process for both published RCTs and registry records.
Figure 1. Flowchart of Article Selection.
RCTs indicates randomized clinical trials; SDs, standard deviations.
Prevalence of Primary Outcome Change and Its Type
Of 389 trials with clear primary outcomes prospectively described in the registry (416 outcomes reported), 33.4% (130 of 389) of trials had at least 1 primary outcome change. Among those studies with primary outcome change, we found that the most common discrepancy was either omission or not reporting or labeling a registered primary outcome (66 of 130). This was followed by publication of a new outcome (40 of 130), which included 9 composite outcomes with component changes; different timing of assessment in the article and the registry (17 of 130); a registered primary outcome reported as a secondary outcome in the article (6 of 130); and the published primary outcome registered as a secondary outcome (1 of 130). The detailed classification for those trials with primary outcome change and the type of change are listed in eTable 4 in the Supplement.
Association of Primary Outcome Change With Intervention Effect
To quantify the consequences of the change in primary outcome on the reported intervention effect size, we calculated the OR for each trial. Of the 389 trials, 22 did not report a primary outcome in the publication, and 29 did not have a reproducible way to calculate an OR. As a result, we included 338 trials (365 outcomes and 487 comparisons) for quantitative analysis on the reported intervention effect size bias assessment. The characteristics of trials with and without primary outcome change are listed in Table 1.
Table 1. Characteristics of Included Randomized Clinical Trials With and Without Primary Outcome Change With Available Effect Sizea.
| Characteristic | With Change (n = 100) | Without Change (n = 238) | P Value |
|---|---|---|---|
| Year of publication, No. (%) | |||
| 2011-2012 | 18 (18.0) | 42 (17.6) | .93 |
| 2012-2013 | 20 (20.0) | 49 (20.6) | |
| 2013-2014 | 23 (23.0) | 57 (23.9) | |
| 2014-2015 | 27 (27.0) | 55 (23.1) | |
| 2015-2016 | 12 (12.0) | 35 (14.7) | |
| ITT status, No. (%) | |||
| ITT | 44 (44.0) | 104 (43.7) | .02 |
| mITT | 26 (26.0) | 91 (38.2) | |
| No ITT/unknown | 30 (30.0) | 43 (18.1) | |
| Study design, No. (%) | |||
| Noninferiority | 5 (5.0) | 26 (10.9) | .09 |
| Superiority | 95 (95.0) | 212 (89.1) | |
| Use of placebo, No. (%) | |||
| Yes | 29 (29.0) | 83 (34.9) | .30 |
| No | 71 (71.0) | 155 (65.1) | |
| Sample size calculation, No. (%) | |||
| Not reported | 5 (5.0) | 14 (5.9) | .75 |
| Reported | 95 (95.0) | 224 (94.1) | |
| Trial centers, No. (%) | |||
| Multiple centers | 71 (71.0) | 185 (77.7) | .19 |
| Single center | 29 (29.0) | 53 (22.3) | |
| Randomization method, No. (%) | |||
| Cluster | 9 (9.0) | 8 (3.4) | .03 |
| Individual | 91 (91.0) | 230 (96.6) | |
| Comparison, No. (%) | |||
| 2 Groups and single outcome | 66 (66.0) | 183 (76.9) | .05 |
| 2 Groups but multiple outcomes | 3 (3.0) | 12 (5.0) | |
| Multiple groups and single outcome | 28 (28.0) | 36 (15.1) | |
| Multiple groups and multiple outcomes | 3 (3.0) | 7 (2.9) | |
| No. of outcomes, median (range) | 1 (1-3) | 1 (1-3) | .96 |
| No. of comparisons, median (range) | 1 (1-8) | 1 (1-10) | .12 |
| Sequence generation, No. (%) | |||
| Low risk | 74 (74.0) | 155 (65.1) | .11 |
| Unclear risk | 26 (26.0) | 83 (34.9) | |
| Allocation concealment, No. (%) | |||
| Low risk | 62 (62.0) | 128 (53.8) | .03 |
| High risk | 2 (2.0) | 0 | |
| Unclear risk | 36 (36.0) | 110 (46.2) | |
| Masking of patients and personnel, No. (%) | |||
| Low risk | 73 (73.0) | 186 (78.2) | .15 |
| High risk | 17 (17.0) | 23 (9.7) | |
| Unclear risk | 10 (10.0) | 29 (12.2) | |
| Masking of outcome assessor, No./total No. (%)b | |||
| Low risk | 92/108 (85.2) | 208/257 (80.9) | .26 |
| High risk | 9/108 (8.3) | 13/257 (5.1) | |
| Unclear risk | 7/108 (6.5) | 36/257 (14.0) | |
| Incomplete outcome data, No./total No. (%)b | |||
| Low risk | 65/108 (60.2) | 179/257 (69.6) | .08 |
| High risk | 12/108 (11.1) | 29/257 (11.3) | |
| Unclear risk | 31/108 (28.7) | 49/257 (19.1) | |
| Type of outcome, No./total No. (%)b | |||
| Time to event | 11/108 (10.2) | 54/257 (21.0) | .94 |
| Binary | 51/108 (47.2) | 107/257 (41.6) | |
| Continuous | 46/108 (42.6) | 96/257 (37.4) | |
| Source of funding, No. (%) | |||
| Public funding | 49 (49.0) | 83 (34.9) | .03 |
| Cofinanced | 17 (17.0) | 35 (14.7) | |
| For-profit funding | 26 (26.0) | 106 (44.5) | |
| Not funded | 1 (1.0) | 2 (0.8) | |
| Not reported | 7 (7.0) | 12 (5.0) | |
| Overall risk of bias, No. (%) | |||
| Low risk | 33 (33.0) | 68 (28.6) | .08 |
| High risk | 27 (27.0) | 45 (18.9) | |
| Unclear risk | 40 (40.0) | 125 (52.5) | |
| Odds ratio, median (range)c | 0.57 (0.00-2.25) | 0.79 (0.01-5.54) | .01 |
Abbreviations: ITT, intent to treat; mITT, modified intent to treat.
Studies with no primary outcome and/or recalculable data for treatment effect estimation are excluded.
Numbers and percentages are based on number of outcomes (n = 365).
Based on number of comparisons (n = 487).
On average, the reported intervention effect size in trials with primary outcome change was found to be larger by 16% (pooled ROR, 0.84; 95% CI, 0.73-0.96) compared with those without change. This result persisted after adjustment for potential confounders (ROR, 0.81; 95% CI, 0.71-0.93) and using the classification of the primary outcome change from reviewer 1 (unadjusted ROR, 0.83; 95% CI, 0.73-0.95) and reviewer 2 (unadjusted ROR, 0.85; 95% CI, 0.75-0.97). Similarly, we did not find material changes in other sensitivity analyses, with RORs ranging from 0.73 after regarding the study weight as additional covariates rather than the weights in the mixed model to 0.96 after excluding studies with multiple groups (Table 2).
Table 2. Main and Sensitivity Analyses With ROR Between Randomized Clinical Trials With and Without Primary Outcome Changea.
| Analysis | No. of Trials | No. of Comparisons | ROR (95% CI) | P Value |
|---|---|---|---|---|
| Main Analysis | ||||
| Unadjusted | 338 | 487 | 0.84 (0.73-0.96) | .01 |
| Sensitivity Analysis | ||||
| Adjustedb | 338 | 487 | 0.81 (0.71-0.93) | .002 |
| Primary outcome change based on reviewer 1 assessment only | 338 | 487 | 0.83 (0.73-0.95) | .007 |
| Primary outcome change based on reviewer 2 assessment only | 338 | 487 | 0.85 (0.75-0.97) | .02 |
| Adjustedc | 338 | 487 | 0.73 (0.60-0.89) | .002 |
| Exclusion of studies with time-to-event outcome | 279 | 409 | 0.81 (0.69-0.95) | .01 |
| Exclusion of studies with binary outcomes | 194 | 275 | 0.78 (0.65-0.93) | .007 |
| Exclusion of studies with continuous outcomes | 209 | 290 | 0.88 (0.74-1.05) | .16 |
| Exclusion of low risk | 238 | 348 | 0.77 (0.65-0.92) | .005 |
| Exclusion of high risk | 266 | 393 | 0.81 (0.69-0.95) | .008 |
| Exclusion of unclear risk | 173 | 233 | 0.95 (0.81-1.11) | .59 |
| Exclusion of multiple outcome | 313 | 411 | 0.80 (0.70-0.92) | .003 |
| Exclusion of multiple groups | 264 | 280 | 0.96 (0.83-1.11) | .55 |
| Exclusion of multiple outcome or multiple groups | 249 | 250 | 0.94 (0.82-1.07) | .36 |
Abbreviation: ROR, ratio of odds ratios.
Analyses were based on 338 studies with available effect size.
Based on weighted mixed model with covariates of deviation from intent-to-treat principle, study design, trial centers, type of comparator, randomization method, type of outcome, source of funding, and overall risk of bias.
Based on mixed model with the same covariates as in footnote b, plus inverse of variance.
Subgroup Analyses
For multicenter trials, the ROR between changed and unchanged primary outcomes was 0.83 (95% CI, 0.72-0.96). For studies assessing continuous outcomes, the corresponding result was 0.74 (95% CI, 0.57-0.94). For trials using a superiority study design, the ROR between changed and unchanged primary outcomes was 0.82 (95% CI, 0.71-0.94). Overestimation of the reported intervention effect size among trials with primary outcome change could be observed among other subgroups, although they did not all reach statistical significance. In addition, there was no evidence of an interaction between different trial characteristics (eg, study design, multiple centers or single center, and type of outcome) and the estimated intervention effects (Figure 2).
Figure 2. Subgroup Analyses by Various Study Characteristics.
ITT indicates intent to treat; mITT, modified intent to treat; and ROR, ratio of odds ratios.
aDifferent outcomes could be observed within the same trial.
Discussion
Our cross-sectional study was a survey of contemporary trials that included a broad range of medical conditions and interventions. We found that 33.4% of the sample had at least 1 primary outcome inconsistency between registration and publication. Among studies for which we could calculate an intervention effect, we demonstrated that trials with primary outcome change reported larger intervention effect sizes. This finding remained even after adjustment for RoB items and other potential bias (eg, deviation from ITT principle, multiple centers, and source of funding) and a series of sensitivity analyses (eg, exclusion of studies with binary outcomes). Because we were unable to include trials that either did not declare a primary outcome in the registry or did not register their protocol, this study may underestimate the true consequences of the practice of primary outcome change.
Several studies have assessed the discrepancy rates between registered and published clinical trial outcomes among specific clinical areas (eg, pain),17,18,19,20,21,22 journals (eg, general medical journals and high-impact journals),5,23,24,25 or registry entries (eg, ClinicalTrials.gov).5,9,21,26 A systematic review of studies up to 2014 that compared registered with reported primary outcomes demonstrated a median 31% rate of discrepancies.7 We found a similar rate in our study (33.4%), although it was lower than the 60% in recent study27 among 192 trials. These findings highlight this prevalent issue after publication of the Consolidated Standards of Reporting Trials (CONSORT) 2010 Statement.28 Notably, our analysis found that the 2 most common discrepancies were omission of registered primary outcomes and inclusion of new unregistered outcomes, which have been repeatedly reported in other studies.5,21,22,24,29,30,31 Unlike other studies that compared published outcomes with those in the registry at the time of the search18,22,25,32,33 or used a specific function (eg, “History of Changes” in the ClinicalTrials.gov archive site),9 our study attempted to identify all outcome changes after the initial registration and excluded those registered after the end of the trial. This is because comparison with the original registered outcome is more relevant in understanding the true consequences of outcome switching on the validity of a clinical trial. However, we still observed high rates of trial registration after study completion (92 of 559) and no primary outcome or unclear primary outcome in the registry (74 of 559) in this survey of contemporary trials after the CONSORT 2010 Statement. Such poor quality of trial registration has been highlighted in previous studies.24,27,32,34,35,36 Although due to our study design we were unable to address the reasons for primary outcome change, some possibilities need to be assessed in future studies, such as pressure to publish positive results with public funding or high rates of nonpublication among industry-sponsored trials with primary outcome change.
Our study is the first to date to quantify the consequences of primary outcome change on the reported intervention effect size in individual RCTs. Some specific characteristics of a trial, such as deviation from ITT principle,11,37 small sample size,38,39 concealment of allocation,40,41 and single center,12 have been assessed in several meta-epidemiological studies. In general, various components of inadequate trial methods are associated with imprecision in the estimated intervention effects, but the magnitude and direction of the bias may vary depending on the medical conditions examined, the definition of inadequate methods, and analytic methods.11,12,37,38,39,40,41 In our study, we found that trials with discrepancies between registry and publication show more beneficial treatments than those without. Our results were robust to a series of supplementary adjusted analyses to adjust for potential confounding factors that may contribute to statistical precision in RCTs, as well as to sensitivity analyses to account for the classification of primary outcome change.
Our study is based on a large sample of individual RCTs rather than a meta-epidemiological approach. The study was performed across a range of medical disciplines, registries, and types of outcomes. Therefore, the trials included in our study are likely representative of a cross-section of the general population, and we believe that our results are generalizable to multiple settings. To explore the association of primary outcome change with the reported intervention effect size, we used several analytic approaches, which gave consistent results.
Limitations
Some caveats should be recognized in our study. First, similar to other meta-epidemiological studies, our study mainly used published information and compared it with records in registries or protocols if necessary. Consequently, our results largely depended on the quality of reporting, which is often unsatisfactory.42 Second, almost 40% (221 of 559) of trials could not be assessed due to unavailable or insufficient information on primary outcomes from both articles and registries (n = 170) and unavailability of treatment effect size estimates (n = 51). Together with the reported publication bias (eg, trials sponsored by industry are less likely to be published), this probably led to an underestimation of the proportion of trials with primary outcome change and their association with the reported intervention effect size. Third, although the potential for selective reporting of primary outcomes was discernible in published trials, we first extracted the adjusted treatment differences and their SEs and then the unadjusted ones if there was no clear statement to specify which was the primary analysis result. This is because few trials released their original protocol and statistical analysis plan.
Conclusions
Results of this study suggest that primary outcome change in RCTs is common and likely overestimates intervention effects. Trial sponsors and investigators should register the primary outcomes, justify changes (if they occur), and report the results accordingly. This will allow the reader to critically appraise and interpret the trial results without bias. Reviewers and editors should routinely use prospectively registered data to avoid changes in primary outcomes during peer review, a practice that has been adopted by leading journals (eg, JAMA and BMJ). Readers and clinicians must be cautious about interpreting trial results and should be aware that trials with primary outcome change could lead to an overestimation of intervention effects.
eTable 1. Search History in PubMed
eTable 2. Search History in Embase via Ovid After Excluding Medline Journals
eTable 3. Number of Hits by Year
eTable 4. List of Included Randomized Clinical Trials With Classification and Type of Primary Outcome Change
eReferences
References
- 1.Kane RL, Wang J, Garrard J. Reporting in randomized clinical trials improved after adoption of the CONSORT Statement. J Clin Epidemiol. 2007;60(3):-. doi: 10.1016/j.jclinepi.2006.06.016 [DOI] [PubMed] [Google Scholar]
- 2.De Angelis C, Drazen JM, Frizelle FA, et al. ; International Committee of Medical Journal Editors . Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med. 2004;141(6):477-478. doi: 10.7326/0003-4819-141-6-200409210-00109 [DOI] [PubMed] [Google Scholar]
- 3.Zarin DA, Ide NC, Tse T, Harlan WR, West JC, Lindberg DA. Issues in the registration of clinical trials. JAMA. 2007;297(19):2112-2120. doi: 10.1001/jama.297.19.2112 [DOI] [PubMed] [Google Scholar]
- 4.Kahan BC, Jairath V. Outcome pre-specification requires sufficient detail to guard against outcome switching in clinical trials: a case study. Trials. 2018;19(1):265. doi: 10.1186/s13063-018-2654-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302(9):977-984. doi: 10.1001/jama.2009.1242 [DOI] [PubMed] [Google Scholar]
- 6.Chan AW, Hróbjartsson A, Haahr MT, Gøtzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457-2465. doi: 10.1001/jama.291.20.2457 [DOI] [PubMed] [Google Scholar]
- 7.Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:282. doi: 10.1186/s12916-015-0520-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Ioannidis JP, Caplan AL, Dal-Ré R. Outcome reporting bias in clinical trials: why monitoring matters. BMJ. 2017;356:j408. doi: 10.1136/bmj.j408 [DOI] [PubMed] [Google Scholar]
- 9.Ramagopalan S, Skingsley AP, Handunnetthi L, et al. Prevalence of primary outcome changes in clinical trials registered on ClinicalTrials.gov: a cross-sectional study. F1000Res. 2014;3:77. doi: 10.12688/f1000research.3784.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.PROSPERO Influence of primary outcome change on treatment effect estimates in clinical trials: cross-sectional study of registered clinical trials. CRD42017058054. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=58054. Accessed June 7, 2019.
- 11.Abraha I, Cherubini A, Cozzolino F, et al. Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study. BMJ. 2015;350:h2445. doi: 10.1136/bmj.h2445 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Bafeta A, Dechartres A, Trinquart L, Yavchitz A, Boutron I, Ravaud P. Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study. BMJ. 2012;344:e813. doi: 10.1136/bmj.e813 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Als-Nielsen B, Chen W, Gluud C, Kjaergard LL. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? JAMA. 2003;290(7):921-928. doi: 10.1001/jama.290.7.921 [DOI] [PubMed] [Google Scholar]
- 14.Kivimäki M, Jokela M, Nyberg ST, et al. ; IPD-Work Consortium . Long working hours and risk of coronary heart disease and stroke: a systematic review and meta-analysis of published and unpublished data for 603,838 individuals. Lancet. 2015;386(10005):1739-1746. doi: 10.1016/S0140-6736(15)60295-1 [DOI] [PubMed] [Google Scholar]
- 15.Fang X, Han D, Cheng Q, et al. Association of levels of physical activity with risk of Parkinson disease: a systematic review and meta-analysis. JAMA Netw Open. 2018;1(5):e182421. doi: 10.1001/jamanetworkopen.2018.2421 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med. 2000;19(22):3127-3131. doi: 10.1002/1097-0258(20001130)19:22<3127::AID-SIM784>3.0.CO;2-M [DOI] [PubMed] [Google Scholar]
- 17.Killeen S, Sourallous P, Hunter IA, Hartley JE, Grady HL. Registration rates, adequacy of registration, and a comparison of registered and published primary outcomes in randomized controlled trials published in surgery journals. Ann Surg. 2014;259(1):193-196. doi: 10.1097/SLA.0b013e318299d00b [DOI] [PubMed] [Google Scholar]
- 18.Li XQ, Yang GL, Tao KM, Zhang HQ, Zhou QH, Ling CQ. Comparison of registered and published primary outcomes in randomized controlled trials of gastroenterology and hepatology. Scand J Gastroenterol. 2013;48(12):1474-1483. doi: 10.3109/00365521.2013.845909 [DOI] [PubMed] [Google Scholar]
- 19.Rongen JJ, Hannink G. Comparison of registered and published primary outcomes in randomized controlled trials of orthopaedic surgical interventions. J Bone Joint Surg Am. 2016;98(5):403-409. doi: 10.2106/JBJS.15.00400 [DOI] [PubMed] [Google Scholar]
- 20.Anand V, Scales DC, Parshuram CS, Kavanagh BP. Registration and design alterations of clinical trials in critical care: a cross-sectional observational study. Intensive Care Med. 2014;40(5):700-722. doi: 10.1007/s00134-014-3250-7 [DOI] [PubMed] [Google Scholar]
- 21.Smith HN, Bhandari M, Mahomed NN, Jan M, Gandhi R. Comparison of arthroplasty trial publications after registration in ClinicalTrials.gov. J Arthroplasty. 2012;27(7):1283-1288. doi: 10.1016/j.arth.2011.11.005 [DOI] [PubMed] [Google Scholar]
- 22.Rosenthal R, Dwan K. Comparison of randomized controlled trial registry entries and content of reports in surgery journals. Ann Surg. 2013;257(6):1007-1015. doi: 10.1097/SLA.0b013e318283cf7f [DOI] [PubMed] [Google Scholar]
- 23.Fleming PS, Koletsi D, Dwan K, Pandis N. Outcome discrepancies and selective reporting: impacting the leading journals? PLoS One. 2015;10(5):e0127495. doi: 10.1371/journal.pone.0127495 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Milette K, Roseman M, Thombs BD. Transparency of outcome reporting and trial registration of randomized controlled trials in top psychosomatic and behavioral health journals: a systematic review. J Psychosom Res. 2011;70(3):205-217. doi: 10.1016/j.jpsychores.2010.09.015 [DOI] [PubMed] [Google Scholar]
- 25.Jones CW, Platts-Mills TF. Quality of registration for clinical trials published in emergency medicine journals. Ann Emerg Med. 2012;60(4):458-64.e1. [DOI] [PubMed] [Google Scholar]
- 26.Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.gov: a cross-sectional analysis. PLoS Med. 2009;6(9):e1000144. doi: 10.1371/journal.pmed.1000144 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Jones CW, Misemer BS, Platts-Mills TF, et al. Primary outcome switching among drug trials with and without principal investigator financial ties to industry: a cross-sectional study. BMJ Open. 2018;8(2):e019831. doi: 10.1136/bmjopen-2017-019831 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Schulz KF, Altman DG, Moher D; CONSORT Group . CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332. doi: 10.1136/bmj.c332 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Gandhi R, Jan M, Smith HN, Mahomed NN, Bhandari M. Comparison of published orthopaedic trauma trials following registration in Clinicaltrials.gov. BMC Musculoskelet Disord. 2011;12:278. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Nankervis H, Baibergenova A, Williams HC, Thomas KS. Prospective registration and outcome-reporting bias in randomized controlled trials of eczema treatments: a systematic review. J Invest Dermatol. 2012;132(12):2727-2734. doi: 10.1038/jid.2012.231 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Ewart R, Lausen H, Millian N. Undisclosed changes in outcomes in randomized controlled trials: an observational study. Ann Fam Med. 2009;7(6):542-546. doi: 10.1370/afm.1017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.You B, Gan HK, Pond G, Chen EX. Consistency in the analysis and reporting of primary end points in oncology randomized controlled trials from registration to publication: a systematic review. J Clin Oncol. 2012;30(2):210-216. doi: 10.1200/JCO.2011.37.0890 [DOI] [PubMed] [Google Scholar]
- 33.Vera-Badillo FE, Shapiro R, Ocana A, Amir E, Tannock IF. Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer. Ann Oncol. 2013;24(5):1238-1244. doi: 10.1093/annonc/mds636 [DOI] [PubMed] [Google Scholar]
- 34.Saquib N, Saquib J, Ioannidis JP. Practices and impact of primary outcome adjustment in randomized controlled trials: meta-epidemiologic study. BMJ. 2013;347:f4313. doi: 10.1136/bmj.f4313 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Wildt S, Krag A, Gluud L. Characteristics of randomised trials on diseases in the digestive system registered in ClinicalTrials.gov: a retrospective analysis. BMJ Open. 2011;1(2):e000309. doi: 10.1136/bmjopen-2011-000309 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Liu JP, Han M, Li XX, et al. Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study. BMJ Open. 2013;3(7):e002968. doi: 10.1136/bmjopen-2013-002968 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Nüesch E, Trelle S, Reichenbach S, et al. The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ. 2009;339:b3244. doi: 10.1136/bmj.b3244 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Nüesch E, Trelle S, Reichenbach S, et al. Small study effects in meta-analyses of osteoarthritis trials: meta-epidemiological study. BMJ. 2010;341:c3515. doi: 10.1136/bmj.c3515 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Dechartres A, Trinquart L, Boutron I, Ravaud P. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ. 2013;346:f2304. doi: 10.1136/bmj.f2304 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Pildal J, Hróbjartsson A, Jørgensen KJ, Hilden J, Altman DG, Gøtzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol. 2007;36(4):847-857. doi: 10.1093/ije/dym087 [DOI] [PubMed] [Google Scholar]
- 41.Sterne JA, Jüni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in “meta-epidemiological” research. Stat Med. 2002;21(11):1513-1524. doi: 10.1002/sim.1184 [DOI] [PubMed] [Google Scholar]
- 42.Huwiler-Müntener K, Jüni P, Junker C, Egger M. Quality of reporting of randomized trials as a measure of methodologic quality. JAMA. 2002;287(21):2801-2804. doi: 10.1001/jama.287.21.2801 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eTable 1. Search History in PubMed
eTable 2. Search History in Embase via Ovid After Excluding Medline Journals
eTable 3. Number of Hits by Year
eTable 4. List of Included Randomized Clinical Trials With Classification and Type of Primary Outcome Change
eReferences
