INTRODUCTION
Randomized controlled trials (RCTs) are the most reliable form of evidence for evaluating drug safety and efficacy. Because clinicians rely on RCTs to inform clinical practice, accurate representation of clinical trial results is important to patient health and safety. Spin, defined as reporting that distorts results or misleads the reader,1 threatens accurate evidence interpretation and application by clinicians. Given that many clinicians obtain information from the study abstract only, spin in the abstract is concerning.2 We examined the prevalence of spin among abstracts in a random sample of trials focused on the efficacy of drugs and examined the association of spin with study characteristics.
METHODS
We searched Medline for all English-language RCTs examining drug efficacy published in 2013. Our search identified 2851 potential studies. After screening titles and abstracts, 1101 potential studies remained. We randomly selected 646 of this sample, of which 190 studies were focused on drug efficacy and met inclusion criteria. For each study, we extracted information on outcome, funding source, financial ties to the manufacturer, and design characteristics (Table 1). The methods are described in full in a prior publication.3
Table 1.
Prevalence of Spin by Study Characteristics (N = 59)
| All studies (n = 59) N |
Spin present (n = 25) N (%) |
Spin absent (n = 34) N (%) |
p-value | |
|---|---|---|---|---|
| Funding source | 0.89 | |||
| Industry involvement | 36 | 15 (42) | 21 (58) | |
| No industry involvement | 23 | 10 (43) | 13 (57) | |
| Financial ties of PI present | 0.08 | |||
| Yes | 29 | 9 (31) | 20 (69) | |
| No | 30 | 16 (53) | 14 (47) | |
| Any relationship with manufacturer (funding and/or financial ties) | ||||
| Yes | 41 | 17 (42) | 24 (58) | 0.83 |
| No | 18 | 8 (44) | 10 (56) | |
| Sample size | 0.01* | |||
| Q1 (24–109) | 15 | 10 (67) | 5 (33) | |
| Q2 (110–270) | 15 | 5 (33) | 10 (67) | |
| Q3 (271–410) | 14 | 8 (57) | 6 (43) | |
| Q4 (411–13,229) | 15 | 2 (13) | 13 (87) | |
| First author affiliation by continent | 0.17 | |||
| Europe | 22 | 10 (45) | 12 (55) | |
| North America | 27 | 8 (30) | 19 (70) | |
| Asia | 7 | 5 (71) | 2 (29) | |
| Other | 3 | 2 (67) | 1 (33) | |
| First author affiliation by country | 0.35 | |||
| USA | 23 | 8 (35) | 15 (65) | |
| Other | 36 | 17 (47) | 19 (53) | |
| Specialty | 0.63 | |||
| Cardiology | 9 | 3 (33) | 6 (67) | |
| Oncology | 7 | 4 (57) | 3 (43) | |
| Other | 43 | 18 (42) | 25 (58) | |
| Trial registration | 0.69 | |||
| Yes | 53 | 22 (42) | 31(58) | |
| No | 6 | 3 (50) | 3 (50) | |
| RCT type | 0.93 | |||
| Phase 2 | 21 | 9 (43) | 12 (57) | |
| Phase 3 | 26 | 10 (38) | 16 (62) | |
| Phase 4 | 4 | 2 (50) | 2 (50) | |
| Other | 8 | 4 (50) | 4 (50) | |
| Type of analysis | 0.24 | |||
| Superiority | 58 | 24 (41) | 34 (59) | |
| Non-inferiority | 1 | 1 (100) | 0 (0) | |
| Comparator | 0.59 | |||
| Placebo | 49 | 20 (41) | 29 (59) | |
| Active | 10 | 5 (50) | 5 (50) | |
| Outcome measure | 0.20 | |||
| Clinical | 45 | 17 (38) | 28 (62) | |
| Surrogate | 14 | 8 (57) | 6 (43) | |
| Blinding | 0.40 | |||
| Double-blinded | 46 | 18 (39) | 28 (61) | |
| Open-label | 12 | 6 (50) | 6 (50) | |
| Single-blinded | 1 | 1 (100) | 0 (0) | |
*Kruskal–Wallis p-value of 0.03 based on medians for studies with spin (201) vs. those without (352)
PI principal investigator
Identification of RCT Outcome
Trained abstractors used the results section of each study to determine whether the study reported positive or negative primary efficacy outcome. For superiority studies, if the drug of interest was statistically superior to the control (e.g., p-value < 0.05), the study outcome was defined as positive. For non-inferiority studies, if the drug of interest and the control had no significant difference, the study outcome was defined as positive. Study outcomes were assessed independently and in duplicate. Any disagreement on study outcome was resolved by discussion among the research team.
Outcome Measure
The main outcome variable considered was the presence or absence of spin in the abstract of the RCT. We considered spin present if the abstract outcome was positive or mixed and the study reported a negative primary efficacy outcome in the results section of the manuscript.1 Two clinician reviewers (SK, DK) evaluated all abstract conclusion sections, rating each conclusion as positive (in favor of study drug), negative (neutral or in favor of control), or mixed. A mixed rating meant that it was unclear in the abstract whether the study drug or the control was favored (i.e., if a subgroup analysis was emphasized over the primary outcome). The two clinicians remained blinded to the study outcome during this discussion.
Analysis
We report the prevalence of spin in a sample of RCTs. We examined the association between spin and study characteristics using a two-sided, 0.05-level χ2 test of significance. Statistical analysis was performed using SAS version 9 statistical software (SAS Institute Inc., Cary, NC).
RESULTS
Of the 190 RCTs identified, 59 had a negative primary outcome in the results. These 59 studies were evaluated for the presence of spin. Among the 59 studies, clinician reviewers rated 8 abstracts as having a positive outcome and 17 as having a mixed outcome, for a total of 25 (42%) abstracts with spin. Study characteristics were largely similar across studies with and without spin (Table 1). Overall, studies with spin had smaller samples (median: 201) than studies without spin (median: 352; p = 0.03). There was no relationship between any financial tie to the manufacturer and presence of spin in the abstract (p = 0.83).
DISCUSSION
Nearly half of abstracts of RCTs focused on drug efficacy that report negative results contain spin. We did not find an association between spin and financial ties to industry, but our study may be underpowered to detect this association. Many clinicians do not read beyond the abstract, and many readers of the literature may not have the skill to critically analyze a trial themselves to combat spin or other bias in the report.4–6 Given the widespread reliance on the abstract, the peer review process needs to be improved to reduce spin in abstracts. A simple prompt asking reviewers to comment on the presentation of the results of the study with a specific question about spin may help focus reviewer attention on this issue. Editors can also review for spin in the editorial decision process. These simple steps that allow more scrutiny of the abstract and provide feedback to authors may reduce, if not eliminate, spin in the literature.
Funding
This project was not directly supported by any research funds. Dr. Keyhani is funded by grants from the NIH (grants RO1 HL116522-01A1, RO1 HL114563-01A1) and VA HSR&D (1IP1HX001994). Dr. Korenstein’s work on this paper was supported by a Cancer Center Support Grant from the National Cancer Institute to Memorial Sloan Kettering Cancer Center (award number P30 CA008748).
Prior Presentations
SGIM meeting April 21, 2017.
Conflict of Interest
All authors declare that they have no conflict of interest.
Ethical approval
Not needed.
Data sharing
Data set available from corresponding author on request.
References
- 1.Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and Interpretation of Randomized Controlled Trials With Statistically Nonsignificant Results for Primary Outcomes. JAMA. 2010;303(20):2058–2064. doi: 10.1001/jama.2010.651. [DOI] [PubMed] [Google Scholar]
- 2.Saint S, Christakis DA, Saha S, et al. Journal reading habits of internists. J Gen Intern Med. 2000;15(12):881–4. doi: 10.1046/j.1525-1497.2000.00202.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Ahn R, Woodbridge A, Abraham A, et al. Financial ties of principal investigators and randomized controlled trial outcomes: cross sectional study. BMJ. 2017;356:i6770. doi: 10.1136/bmj.i6770. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Caverly TJ, Matlock DD, Prochazka AV, Lucas BP, Hayward RA. Interpreting Clinical Trial Outcomes for Optimal Patient Care: A Survey of Clinicians and Trainees. J Grad Med Educ. 2016;8(1):57–62. doi: 10.4300/JGME-D-15-00137.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Johnson TV, Abbasi A, Schoenberg ED, et al. Numeracy among trainees: are we preparing physicians for evidence-based medicine? J Surg Educ. 2014;71(2):211–5. doi: 10.1016/j.jsurg.2013.07.013. [DOI] [PubMed] [Google Scholar]
- 6.Korenstein D. Blinding Them With Science? Evidence-Based Medicine as a Barrier to Health Care Value. J Grad Med Educ. 2016;8(1):106–8. doi: 10.4300/JGME-D-15-00570.1. [DOI] [PMC free article] [PubMed] [Google Scholar]