Evaluating Clinical Trial Outcome Reporting Practices

INTRODUCTION

The Consolidated Standards of Reporting Clinical Trials (CONSORT) guidelines set the framework for the ethical standards of research and are widely accepted.

To satisfy the established ethical standards for research, investigators are required to register trials with a clinical trial registry and list the primary and secondary outcomes. This requirement prevents the manipulation of data and omission of negative results.¹ This is important as evidence suggests an association between statistically significant results and increased rates of article publication.^{2, 3}

Previous research provides evidence to suggest that many publications are not properly reporting primary and secondary outcomes.^{4, 5} Our study aimed to evaluate how often clinical trials were properly reporting primary and secondary outcomes

METHODS

We selected the most recent twenty-five consecutive randomized controlled trials from December 2019 from four major scientific journals: Journal of the American Medical Association (JAMA), New England Journal of Medicine (NEJM), British Medical Journal (BMJ), and The Lancet. The published primary and secondary outcomes were then compared to the pre-trial registrations (www.clinicaltrials.gov).

Outcomes were judged as correctly reported if they contained the same number of, and content as the original outcomes listed in the pre-trial registration. Trials that listed reasons for changes in trial outcomes were considered to be correctly reported.⁶

The trial information was independently analyzed by ZD and MOK and subsequent discrepancies were discussed with DM. The inter-rater reliabilities were 88.1% for primary outcomes and 96.5% for secondary outcomes.

This study was granted exemption status by Georgetown University’s IRB.

RESULTS

Outcome Reporting

Among 100 published trials, 73% had correct reporting of their primary outcome and 23% of secondary outcomes were properly reported.

Of the 27 trials found to have inconsistencies in reported primary outcomes, 25 of those trials also had inconsistencies in reported secondary outcomes.

Breakdown of Discrepancies

Outcome discrepancies were identified as including more outcomes than originally pre-specified, omitting pre-specified outcomes, or changing pre-specified primary outcomes to secondary outcomes, and listed in Figure 1. Of the 27 trials with primary outcome inconsistencies, 5 listed additional outcomes, 14 listed fewer outcomes, and 8 changed the outcomes such that the total number remained the same. Of the 77 trials with secondary outcome inconsistencies, 55 listed additional outcomes, 16 listed fewer outcomes, and 6 changed the outcomes such that the total number remained the same.

Figure 1.

Primary and secondary outcome matching by journal. The percentage of primary and secondary outcomes that were correctly reported across four major medical journals: Journal of the American Medical Association (JAMA), New England Journal of Medicine (NEJM), The BMJ, and The Lancet. Followed by the breakdown of outcome discrepancies.

Quantifying Protocol Changes

Of the 100 trials analyzed, 64 trials recorded changes to their pre-specified research outcomes between trial commencement and publication (Table 1). For those 64 trials, researchers recorded changes to their outcomes an average of 1.68 times (range, 1–6), with an average elapsed time between trial commencement and the final time that outcomes were changed of 48.9 months (range, 1–135).

Table 1.

Comparison of Registered Outcome Changes Across Journals

	JAMA	NEJM	BMJ	Lancet	Total
Trials that changed outcomes on registry—no. (%)	15 (60)	19 (76)	13 (52)	17 (68)	64 (64)
Average number of times outcomes were changed—no. [range]	1.81 [1–6]	2 [1–3]	1.5 [1–4]	1.35 [1–3]	1.68 [1–6]
Average time elapsed—months [range]	47.56 [1–123]	56.47 [15–113]	48.07 [4–135]	42.24 [2–122]	48.90 [1–135]

DISCUSSION

Our study found that nearly a quarter of primary outcomes and the majority of secondary outcomes are being reported incorrectly. These findings are similar to measures in previously conducted research.⁵ The lower rates of primary outcome inconsistencies compared to secondary outcomes are encouraging since primary outcomes are the key inquiries of clinical trials.

Secondary outcome discrepancies most commonly resulted from the addition of new outcomes while primary outcome discrepancies more often occurred when researchers omitted pre-specified outcomes. This is concerning as omitting primary outcomes has shown to significantly change trial results.¹ This may favor statistically significant clinical trial results, which are shown in increased rates of article publication.^{2, 3} Furthermore, the addition of secondary outcomes after trial commencement represents a concerning finding that suggests researchers are gathering and reporting data that was not specified in the original pre-trial registration.

Our analysis revealed that changes to outcomes were common and occurred on average 48 months between trial commencement and publication. The fact that alterations were made to trial outcomes 4 years post-trial onset is potentially troublesome and may mean researchers are changing outcomes after reviewing data.

This survey had limitations. We only reviewed four journals and it is possible that other journals would have different results. However, we chose journals that are widely regarded as academically rigorous and represent a reasonable standard for ethical publishing. It is possible that researchers planned to include omitted outcomes in future publications. These outcomes could not be included in our review and might account for some of the discrepancies in outcome reporting.

Our data highlights that the system is not working to its full potential. Journal editors should require researchers to provide this information with justification, in the manuscript. Incomplete data reporting can lead to inaccurate impressions about treatment efficacy. Complete reporting of all outcomes is necessary for reviewers and readers to judge the impact of clinical trials. Furthermore, transparency regarding outcome changes must be enforced, and a future discussion regarding what is deemed an acceptable justification for such changes is warranted.

Author Contribution

All those who contributed to the manuscript meet the criteria for authorship.

Data Availability

The datasets analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Conflict of Interest

The authors declare that they do not have a conflict of interest.