Table 1.
Researchers’ inaccurate statements about outcome reporting
| Theme | Trialists’ response | Issue |
|---|---|---|
| Timing of pre-specification | ||
| Stating or implying that pre-specification after trial commencement is acceptable | “The prespecified analysis of PATHWAY-2 precisely followed a detailed statistical analysis plan (SAP) that was published in BMJ Open before data lock and unblinding of data (4), and was provided in full to The Lancet, dated and signed before any unblinding or analysis. All primary and secondary endpoints reported in The Lancet were listed at ClinicalTrials.gov before data lock and unblinding” (Trial 57, Lancet, 02/04/16). | The authors suggest that pre-specification should happen before “data lock and unblinding”. However, CONSORT item 6b requires that trial reports declare and explain “any changes to trial outcomes after the trial commenced, with reasons” in the paper reporting the results of the trial. |
| Failure to report changes to pre-specified outcomes in paper | ||
| Failure to recognise that post-commencement changes are acceptable but should be declared in the paper reporting the results of the trial | “Length of stay in survivors and days to death (the fifth so-called new endpoint) are components of length of hospital stay, but they were presented separately to prevent bias from higher mortality in either group that resulted in a difference in length of stay between groups” (Trial 27, Lancet, 30/01/16). | This change from protocol was not mentioned or explained in the paper. CONSORT item 6b requires that trial reports declare and explain “any changes to trial outcomes after the trial commenced, with reasons” in the paper reporting the results of the trial. |
| Stating that pre-specified outcomes missing from the trial report, or declarations of changes, will be reported elsewhere but failing to declare this in the trial report | “With regards to cost outcomes, both the primary and four of the eight so-called missing secondary outcomes (therapy costs, quality of life, institutionalisation, and cost-effectiveness ratios) will be presented in a later publication as stated in the headline paper” (Trial 27, Lancet, 30/01/16). | Changes from pre-commencement outcomes should be declared in the paper reporting the results, as above. Note that while the authors state the additional outcomes “will be presented in a later publication as stated in the headline paper”, there is no such disclosure in the paper; we asked the authors to identify it in our follow-up letter; this was not published and we received no reply. |
| Registries | ||
| Incorrect statements about registries | “Trial registries often do not request or have space for sufficient detail about secondary outcomes” (Trial 10, Lancet, 16/04/16) | There are no restrictions on posting secondary outcomes to registers. |
| Multiple sets of discrepant pre-specified outcomes | ||
| Making reference to protocols that are publicly inaccessible, or were published after trial commencement, which allegedly contain outcomes that are discrepant with registry entries but consistent with the published report | “The trial was registered at ClinicalTrials.gov where we indicated that the primary outcome was... The protocol was also sent to The Lancet shortly after the study started and a summary of the protocol was published. The submitted protocol clearly indicated that the primary outcome was… all outcomes at 2 years prespecified in the submitted protocol were reported but were not included in the published protocol” (Trial 10, Annals, 16/04/16). | The argument appears to be that there is a publicly inaccessible pre-commencement protocol that contains pre-specified outcomes different from those in the contemporaneous pre-commencement registry entry. There is no methodological justification for discrepant outcomes between registry entry and protocol for the same trial at the same time point: the two should be the same, and changes after trial commencement should be discussed in the results paper. Registries were devised as a publicly accessible location for trial information specifically to prevent selective outcome reporting. Having multiple discrepant sets of pre-specified outcomes, with the option to choose between multiple discrepant documents, undermines the purpose of pre-specifying outcomes. |
| Making reference to multiple discrepant sets of pre-specified outcomes | “All primary and secondary endpoints reported in The Lancet were listed at ClinicalTrials.gov before data lock and unblinding. The protocol… also posted on the public domain, EudraCT, before patient recruitment, correctly identified the primary objective… The primary outcome measure was correctly stated on EudraCT” and so on (Trial 57, Lancet, 02/04/16). | This trial had multiple different sets of conflicting “prespecified” outcomes in different locations at similar dates. For example, different outcomes are registered on ClinicalTrials.gov in February and July 2015, and both sets of outcomes in turn are inconsistent with those in the protocol of June 2015. |
| Issues with time points | ||
| Incorrect statements around issue of multiple time points | “We do not see how these multiple measurement time points should be counted as separate outcomes, as the procedure of the COMPare team seems to propose. We think this leads to misuse of overall statistics on their website and exaggerated conclusions about the magnitude of outcome switching in RCTs” (Trial 70, BMJ, 04/02/16). | The trial report states “Secondary outcome measure were symptoms of depression and anxiety measured with the CES-D and HADS-A at baseline and at 3, 6, 9, 12, 18 and 24 months”; and all of these time points are then separately reported in Table 5 of the trial report as a mean with a standard deviation. These are all outcomes, according to the CONSORT guidelines. None of these time points was pre-specified before trial commencement; therefore, 21 non-pre-specified secondary outcomes were reported. |
References throughout are to the correspondence archive at http://COMPare-trials.org/data containing the full public correspondence on all trials, and all correspondence with editors, organised by trial ID and date, or journal name for general correspondence. Abbreviations: BMJ British Medical Journal, CES-D Center for Epidemiologic Studies - Depression, COMPare Centre for Evidence-Based Medicine Outcome Monitoring Project, CONSORT Consolidated Standards of Reporting Trials, HADS-A Hospital Anxiety and Depression Scale - Anxiety, PATHWAY Prevention And Treatment of Hypertension With Algorithm-based therapy, RCT randomised controlled trial