Evaluation of Spin in Abstracts of Systematic Reviews and Meta-analyses Focused on Treatments of Erectile Dysfunction: A Cross-sectional Analysis

Abstract

Introduction

It is predicted that erectile dysfunction will affect around 322 million men worldwide by 2025. Because of the large volume of literature on the topic, physicians often turn to systematic reviews and meta-analyses—and particularly abstracts of such articles—for clinical guidance. Thus, it is crucial that findings are not misrepresented in abstracts. In this study, we evaluated the use of spin (ie, the misreporting of study findings by overstating or selectively reporting efficacy results, minimizing harms, or making unwarranted clinical recommendations) in the abstracts of systematic reviews on erectile dysfunction.

Methods

A search strategy was developed using the MEDLINE and Embase databases to retrieve systematic reviews focused on treatments for erectile dysfunction. 2 investigators independently screened the titles and abstracts from the reviews for study inclusion. Investigators analyzed the included systematic reviews for 9 of the most severe types of spin using a previously developed classification scheme and rated them for methodological quality using the revised A MeaSurement Tool to Assess systematic Reviews (AMSTAR) in a masked, duplicate manner. Study characteristics for each review were also extracted in duplicate.

Results

Our search returned 2,224 articles, of which 102 systematic reviews and meta-analyses were included in the final analysis. A total of 31.4% (32/102) of systematic reviews contained spin. 8 types of spin were identified in our sample. Type 3 (selective reporting of or overemphasis on efficacy outcomes) and type 5 (conclusion claims beneficial effect despite high risk of bias) were the most common types of spin, each occurring in 10.8% (11/102) of abstracts. There was no significant association between the presence of spin and the extracted study characteristics or methodological quality.

Conclusion

Spin was present in systematic reviews and meta-analyses covering erectile dysfunction treatments. Steps should be taken to improve the reporting quality of abstracts on erectile dysfunction treatment.

Reddy AK, Lulkovich K, Ottwell R, et al. Evaluation of Spin in Abstracts of Systematic Reviews and Meta-analyses Focused on Treatments of Erectile Dysfunction: A Cross-sectional Analysis. Sex Med 2021;9:100284.

Introduction

“Spin” refers to misleading readers (however unintentionally) by overstating or beautifying results.¹ There are many reasons why authors attempt to spin their journal article—whether full-text, abstract, or both—including real or imagined pressures from funders, justifying their time and effort, or the hope that positive results will improve the quality of the journal the study is published in.2, 3, 4 Regardless of the reasons, recent studies indicate that spin is rampant in the abstracts of clinical trial articles,5, 6, 7, 8, 9, 10, 11 leaving readers with a questionable understanding of their results. This is further complicated by the fact that most clinicians only read an article's abstract, often due to time constraints, paywalls, and other factors.^12,13

In 2016, Yavchitz et al¹⁴ developed a classification and ranking system for the type of spin that is contained in the abstracts of systematic reviews. This is valuable, considering systematic reviews and meta-analyses are the backbone of the clinical practice guidelines that influence medical practices and treatment decisions worldwide. As a synthesis of all the available evidence on a topic, the authors of systematic reviews attempt to offer an accurate conclusion on the associated benefits and risks of a given treatment.¹⁵ Importantly, the reporting and structure of systematic reviews was the subject for the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).¹⁶ Using this checklist helps to ensure that all pertinent outcome data are included in the full-text manuscript, but authors still have a good deal of leeway in the way they report their findings in abstracts.

Our study sought to identify and classify spin in the abstracts of systematic reviews and meta-analyses focused on the treatment of erectile dysfunction, a medical condition that could affect 322 million men worldwide by 2025.¹⁷ Considering that the causes and treatments of erectile dysfunction are multifactorial and that there is a high volume of research being conducted on this condition, physicians must navigate the available evidence to determine the best treatment options. The outcomes of these treatments can significantly impact a patient's quality of life and self-image.¹⁸ Knowing this, our study explored whether the 9 most severe types of spin, as outlined by Yavchitz et al, occur in the abstracts of erectile dysfunction systematic reviews and meta-analyses. We hypothesize that there will be spin present in abstracts of erectile dysfunction treatment systematic reviews. Furthermore, we discuss opportunities to educate authors, editors, reviewers, and readers in recognizing spin and its consequences in scientific and medical literature and in eliminating the presence of spin in scholarly articles.

Methods

Oversight, Transparency, Reproducibility, and Reporting

This study did not meet the regulatory definition for human subjects research per the U.S. Code of Federal Regulations because it did not contain human subjects. Therefore, it was not subject to institutional review board oversight. The protocol, data, analysis scripts, extraction forms, and other study artifacts are available on Open Science Framework, which ensured transparency and reproducibility for our study.¹⁹ Our data and analysis scripts were also provided to an independent laboratory and reanalyzed in a masked fashion, ensuring analytic reproducibility. We conducted this study in tandem with other studies that evaluated spin in systematic reviews for other medical conditions. Therefore, these methods are being used elsewhere because of the standard methodology. The PRISMA²⁰ and Murad and Wang's²¹ guideline were among the relevant reporting guidelines used for writing this article.

Search Strategy

A systematic review librarian developed our search strategies for the MEDLINE (Ovid) and Embase (Ovid) databases, which allowed us to identify systematic reviews and meta-analyses centered on treatment for erectile dysfunction (Figure 1). The systematic review librarian carried out the searches on June 2, 2020, and uploaded the resulting records to Rayyan, a systematic review screening platform.¹¹ Records were deduplicated and screened by 2 investigators for inclusion criteria. We conducted the deduplication and screening using the titles and abstracts of the studies in a masked, duplicate fashion. Disparities in screening findings were resolved between the 2 investigators by discussion.

Figure 1.

Search strategies.

Eligibility Criteria

To be included, an article was required to meet the following criteria¹: The article must be a systematic review with or without a meta-analysis.² The article must pertain to the treatment of erectile dysfunction.³ The abstract and full article must be available in English.⁴ The article must contain only human subjects. We used the PRISMA-Protocols definition of systematic reviews and meta-analyses.²²

Training

The 2 investigators completed an online training course on systematic reviews and meta-analyses by Li and Dickersin.²³ These investigators also underwent 2 days of online and in-person training on the topic of spin. The training consisted of learning the definition and interpretation of the 9 most severe types of spin that occurs in abstracts of systematic reviews.¹⁴ To characterize and determine whether or not these 9 types of spin were present in abstracts of erectile dysfunction treatment systematic reviews, we used the classification system developed by Yavchitz et al.¹⁴ The 9 most severe types of spin are defined in Table 1. Finally, investigators were trained to assess the methodological quality of each article using the revised A MeaSurement Tool to Assess systematic Reviews (AMSTAR 2). A detailed outline of training modules may be found in our study protocol.

Table 1.

Spin types and frequencies (%) in abstracts (N = 102)

9 most severe types of spin∗	No. (%) of abstracts containing spin
1) Conclusion contains recommendations for clinical practice not supported by the findings.	5 (4.9)
2) Title claims or suggests a beneficial effect of the experimental intervention not supported by the findings.	1 (1)
3) Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention.	11 (10.8)
4) Conclusion claims safety based on non-statistically significant results with a wide confidence interval.	0 (0)
5) Conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in primary studies.	11 (10.8)
6) Selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention.	10 (9.8)
7) Conclusion extrapolates the review's findings to a different intervention (ie, claiming efficacy of one specific intervention although the review covers a class of several interventions).	5 (4.9)
8) Conclusion extrapolates the review's findings from a surrogate marker or a specific outcome to the global improvement of the disease.	1 (1)
9) Conclusion claims the beneficial effect of the experimental treatment despite reporting bias.	8 (7.8)

^∗Spin types as defined by Yavchitz et al.¹⁴

Data Extraction

The investigators extracted data in a masked, duplicate fashion using a pilot-tested Google form. Each systematic review and meta-analysis was thoroughly analyzed for the 9 most severe types of spin found in their abstracts. Table 1 contains the definitions for the 9 types of spin used in this study. We then evaluated the methodological quality of each systematic review and meta-analysis using AMSTAR 2 (https://amstar.ca/). AMSTAR 2 is a 16-item scale commonly used to measure the quality of a systematic review or meta-analysis methodology.²⁴ Construct validity coefficients have been high with the original AMSTAR instrument (r = 0.91) and the Risk of Bias in Systematic Reviews (ROBIS) instrument (r = 0.84). Inter-rater reliability of AMSTAR 2 scores has also been found to be moderate to high across studies.²⁵ The rating classification for the methodological quality of the review was of “high,” “moderate,” “low,” or “critically low” quality. It has been shown that AMSTAR scores for erectile dysfunction treatment systematic reviews have a “fair to good” quality.²⁶ The AMSTAR 2 items can be found in Table 2.

Table 2.

AMSTAR 2 items and frequency of responses (N = 102)

AMSTAR 2 item	Response, no. (%)
	Yes	No	Partial yes
1) Did the research questions and inclusion criteria for the review include the elements of PICO?	101 (99%)	1 (1%)	0 (0)
2) Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?	11 (10.8%)	79 (77.5%)	12 (11.7%)
3) Did the review authors explain their selection of the study designs for inclusion in the review?	88 (86.3%)	14 (13.7%)	0 (0)
4) Did the review authors use a comprehensive literature search strategy?	11 (10.8%)	12 (11.8%)	79 (77.4%)
5) Did the review authors perform study selection in duplicate?	65 (63.7%)	37 (36.3%)	0 (0)
6) Did the review authors perform data extraction in duplicate?	69 (67.6%)	33 (32.4%)	0 (0)
7) Did the review authors provide a list of excluded studies and justify the exclusions?	33 (32.4%)	24 (23.5%)	45 (44.1%)
8) Did the review authors describe the included studies in adequate detail?	44 (43.1%)	8 (7.8%)	50 (49.0%)
9) Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?	50 (49.0%)	39 (38.2%)	13 (12.7%)
10) Did the review authors report on the sources of funding for the studies included in the review?	7 (6.9%)	95 (93.1%)	0 (0)
11) If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results?∗	68 (88.3%)	9 (11.7%)	0 (0)
12) If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?∗	61 (79.2%)	16 (20.8%)	0 (0)
13) Did the review authors account for RoB in primary studies when interpreting/discussing the results of the review?	66 (64.7%)	36 (35.3%)	0 (0)
14) Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?	68 (66.7%)	34 (33.3%)	0 (0)
15) If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?∗	40 (51.9%)	37 (48.1%)	0 (0)
16) Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?	79 (77.5%)	23 (22.5%)	0 (0)

AMSTAR 2 = revised A MeaSurement Tool to Assess systematic Reviews; PICO = patient/population, intervention, comparison, outcomes.

^∗25 articles did not perform a meta-analysis.

The investigators also extracted the following additional items from each systematic review and meta-analysis: (1) intervention type (pharmacologic, nonpharmacologic, surgery, combination, other); (2) the date the review was received by the journal; (3) the funding source(s) for each systematic review (industry, private, public, none, not mentioned, hospital, combination of funding not including industry, combination of funding including industry, other); (4) whether the review discussed adherence to PRISMA²⁷ or PRISMA for abstracts²⁸; (5) whether the journal submission guidelines required adherence to PRISMA and (6) the publishing journal's 5-year impact factor. Upon completion of data extraction, the investigators were unmasked and met to resolve any discrepancies by discussion. If an agreement in discrepancies could not be reached, additional authors served as arbiters.

Statistical Analysis

We used descriptive statistics to describe the overall frequency of spin and its subtypes. We also reported results as frequency counts and percentages. We prespecified the potential use of binary logistic regression in our protocol. We then calculated a power analysis to define our sample size. We did not perform a multivariable logistic regression because our final sample size of 102 was not sufficiently powered. Therefore, we calculated unadjusted odds ratios to determine any relationships between study features and the presence of spin. Our protocol documents the analytical decisions made in the study. Stata 16.1 (StataCorp, LLC, College Station, TX) was used for all analyses.

Results

Study Characteristics

Database search queries retrieved 2,224 studies. After the removal of 616 duplicates, 1,608 studies underwent the initial screening process, from which 1,454 articles were excluded. An additional 50 articles were excluded during full-text screening. In total, 102 systematic reviews and meta-analysis were included for data extraction. Figure 2 illustrates our screening process accompanied by a rationale for all exclusions. Of the 102 systematic reviews and meta-analyses, the most common intervention type was pharmacologic (68/102, 66.7%) (Table 3). 61 studies (61/102, 59.8%) were published in journals whose submission guidelines recommend adherence to PRISMA. In regards to funding, 46 studies did not mention a source of funding (46/102, 45.1%), 24 studies were not funded (24/102, 23.5%), and 32 studies (32/102, 31.4%) were funded with the most common funding source being public (15/102, 14.7%). The average 5-year impact factor for our included journals was 4.04 (SD 4.48).

Figure 2.

Flow diagram of inclusion and exclusion criteria.

Table 3.

General characteristics of systematic reviews and meta-analyses

Study characteristics	No. (%) of articles (N = 102)
	Total	Abstract without spin	Abstract with spin	Odds ratio (95% CI)
Intervention type
Mixed	8	6	2	1 [Ref]
Nonpharmacologic	18	14	4	0.86 (0.12–6.01)
Pharmacologic	68	42	26	1.86 (0.35–9.9)
Surgery	8	8	0	1
Article mentions adherence to PRISMA
No	67	48	19	1 [Ref]
Yes	35	22	13	1.49 (0.63–3.55)
Publishing journal recommends adherence to PRISMA
No	41	29	12	1 [Ref]
Yes	61	41	20	1.18 (0.50–2.78)
Funding source
Not funded	24	16	8	1 [Ref]
Industry	11	10	1	0.20 (0.02–1.85)
Not mentioned	46	31	15	0.97 (0.34–2.76)
Private	6	3	3	2.00 (0.33–12.34)
Public	15	10	5	1.00 (0.25–3.93)
AMSTAR 2 rating
Critically low	41	32	9	1 [Ref]
Low	8	5	3	2.13 (0.43–10.68)
Moderate	51	31	20	2.29 (0.91–5.81)
High	2	2	0	1
Journal 5-year impact factor	4.04 (4.48)	3.93 (4.11)	4.26 (5.16)	1.02 (0.92–1.12)
Year review was conducted (1997–2020)				1.02 (0.95–1.11)

AMSTAR 2 = revised A MeaSurement Tool to Assess systematic Reviews; PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Spin in Abstracts

Spin was present in 31.4% of the abstracts from the included 102 systematic reviews and meta-analyses focused on the treatment of erectile dysfunction (32/102, 31.4%). However, a total of 52 different instances of spin were identified as several abstracts contained more than one type of spin (Table 1). Spin type 3 (selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention) and type 5 (conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in primary studies) were the most common spin types, both of which were found in 11 abstracts (11/102, 10.8%). There were no instances of spin type 4 occurring in abstracts (Table 1). In regard to abstracts containing spin and particular study characteristics, there was no association between abstracts containing spin and the review's intervention type, funding source, whether the publishing journal recommended adherence to PRISMA or the systematic review mentioned PRISMA adherence, or the journal's 5-year impact factor (Table 3).

Methodological Quality

In regard to methodological quality, AMSTAR 2 scores indicated 2 studies were of high quality (2/102, 1.9%), 51 studies were moderate quality (51/102, 50%), 8 studies were low quality (8/102, 7.8%), and 41 studies were critically low quality (41/102, 40.2%). We found no statistical significance between the methodological quality and the types.

Discussion

Our study found that spin was present in 31.4% of the abstracts in our sample. These findings suggest that spin frequently occurs within abstracts of systematic reviews regarding erectile dysfunction therapies. These findings are consistent with the larger body of literature on spin in randomized trials. Kinder et al¹⁰ performed a methodological review of 138 randomized trials published in anesthesia journals and found that 23.2% contained some form of spin in their abstracts. Similarly, Wayant et al²⁹ found spin in 37.1% of abstracts published in oncology journals. In our sample, the most common spin types were types 3 (selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention.), 5 (conclusion claims the beneficial effect of the experimental treatment despite the high risk of bias in primary studies), and 6 (selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention).

It is important to consider these forms of spin and how they manifest to emphasize their influence on readers’ perceptions about a study and conclusion regarding clinical benefits and harms. For example, Wang et al (2014)³⁰ conducted a systematic review of the effects of phosphodiesterase type 5 inhibitors on the treatment of erectile dysfunction after bilateral nerve-sparing radical prostatectomy. The abstract states, “The subgroup analysis could find a trend that [emphasis added] longer treatment duration, higher dosage, on-demand dosing, sildenafil, and mild ED are associated with more responsiveness to PDE5-Is.” However, when reviewing the full text, there were statistically nonsignificant results for treatment duration (Table 2). Thus, this form of spin (type 3, selective reporting), in which information is omitted from the abstract, can be misleading for readers and distort important perspectives for clinical practice.

In another example, Wang et al (2020)³¹ conducted a systematic review and meta-analysis on the effects of Chinese herbal medicine combined with tadalafil on erectile dysfunction. The abstract states, “All included studies were tested for publication bias, and results indicated that there was no significant bias… Traditional Chinese medicine combined with tadalafil has significant efficacy in the treatment of ED with no increase in side effects.” However, while reviewing the full article, the authors failed to mention in their abstract that there was a high risk of detection bias in 10 of the 11 included studies and a high risk of performance bias in all 11. Not placing the results within the context of the risk of bias (which falls under spin type 5) limits the reader's ability to make sound judgments regarding the nature of the overall results.

In considering these examples from our findings, spin is alarming because it may affect study interpretation and, thus, clinical decision-making. In one study, Boutron et al³² performed a randomized experiment in which 300 oncologists were asked to read and respond to questions about a clinical trial abstract. This clinical trial reported statistically nonsignificant results in the abstract; however, one-half of the oncologists were presented with an abstract containing spin while the other half were presented with an accurately written abstract. The oncologists reviewing the spun abstract were more likely to find a paper beneficial and less rigorous. Physicians would also be more interested in reading the full-text article. It was concluded that spin in abstracts has an influence on clinician interpretation of trial results.³²

Moving forward, we believe that a number of stakeholders can help deter spin. Primarily, spin is incorporated by authors as they write abstracts. While this investigation did not assess for motive or whether incorporating spin into abstracts was intentional or not, we believe that training opportunities and published guidance could help. Training opportunities could be in the form of continuing education modules, conference workshops, or webinars. A greater awareness of spin could be accomplished by journal editorials on the topic. Journals are also an important player to reduce spin. We recommend that journals issue guidance directly on minimizing spin, which could easily be incorporated into their existing instructions for the authors' page. We do not believe that requiring PRISMA for reporting is sufficient because it offers no specific guidance on spin. Even PRISMA for abstracts provides limited guidance to reduce spin in systematic review abstracts. However, adding guidance on spin would be a welcomed addition to PRISMA and PRISMA for abstracts. Finally, peer reviewers may play an important role in detecting spin. If peer reviewer guidance from journals specifically addressed spin, it may be detected with higher frequency. Collectively, such efforts would likely contribute to reductions in spin.

There are both strengths and limitations for the present study. Regarding its strengths, we wanted to increase the likelihood of our study being reproducible and wanted to ensure that we were transparent throughout. Thus, we placed our protocol, extraction forms, data, analysis scripts, and other study documents on the Open Science Framework. Any deviations to our protocol were recorded, and subsequent protocol versions were also placed to the Open Science Framework. All screening, evaluations of spin, data extraction, and AMSTAR 2 assessments were conducted in a masked, duplicate manner. This process was carried out to reduce error and bias and is recommended by The Cochrane Collaboration.³³ Finally, our study results were verified by an independent group to ensure the reproducibility of our findings.

This study also had some limitations. First, spin is a subjective topic, which is open to interpretation. To minimize this limitation, each investigator underwent standardized training. Second, this study is a cross-sectional study; therefore, our findings should not be generalized broadly. Third, although a systematic review librarian created the search strategies for the study and we searched the most widely used biomedical literature databases, there is a possibility that some systematic reviews pertaining to erectile dysfunction treatment were left out of the study. Finally, to our knowledge, no studies have sought to identify whether the presence of spin is changing overtime. Thus, future research should assess whether the incidence of spin is increasing or decreasing in more recent literature.

Conclusion

Our study found that approximately 30% of the abstracts of systematic reviews contained spin. We advance that such reporting may have downstream effects on clinical decision-making. Practically speaking, when a busy physician in office sees multiple patients per day, they may look to abstracts to provide pertinent study information. Thus, it is crucial for abstracts of systematic reviews to not be misleading or contain information that can change perception based on overemphasis, selective reporting, or disregarding risks of bias in the studies. Within the context of ED, reviewing an abstract with spin might misguide clinical decision-making, such as prescribing sildenafil in higher dosage with longer duration in the setting of bilateral nerve-sparing radical prostatectomy where treatment duration was not discussed or recommending herbal medicine in combination with tadalafil considering the high risk of bias of the studies in the systematic review. Thus, we argue that abstracts of systematic reviews contain objective, complete information without spin.

Statement of authorship

Arjun K. Reddy: Methodology, Investigation, Writing-Reviewing and Editing; Kaley Lulkovich: Methodology, Investigation, Writing-Reviewing and Editing; Ryan Ottwell: Supervision, Methodology, Writing-Reviewing Editing; Wade Arthur: Supervision, Methodology, Writing- Reviewing and Editing; Aaron Bowers: Writing, Methodology; Shafiq Al-Rifai: Writing, Supervision; Katherine Cook: Supervision, Visualization; Drew N. Wright: Methodology; Micah Hartwell: Software, Investigation, Data Curation; Matt Vassar: Supervision, Visulaization, Project administration, Funding Acquisition.