Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-analyses Covering Treatments for Achilles Tendon Ruptures

Abstract

Background:

Approximately 18 in every 100 000 people have experienced a ruptured Achilles tendon. Despite the prevalence of this condition, treatment options remain contested.

Hypothesis/purpose:

The objective of this study was to evaluate the use of spin—reporting practices that may exaggerate benefit or minimize harm—in abstracts of systematic reviews related to Achilles tendon repair. We also evaluated whether particular study characteristics were associated with spin.

Study design:

Cross-sectional.

Methods:

We developed a search strategy for Ovid MEDLINE and Ovid Embase for systematic reviews focused on Achilles tendon treatment. Following title and abstract screening of these search returns, these reviews were evaluated for spin (according to a previously developed classification scheme) and received AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews–2) appraisals by 2 investigators in a masked, duplicate manner. Study characteristics for each review were also extracted in duplicate.

Results:

Our systematic search returned 251 articles of which 43 systematic reviews and meta-analyses were eligible for data extraction. We found that 65.1% of included studies contained spin (28/43). Spin type 3 was the most common type, occurring in 53.5% (23/43) of abstracts. Spin types 5, 6, 1, and 4 occurred in 16.3% (7/43), 9.3% (4/43), 7% (3/43), and 5.3% (1/43), respectively. Spin types 2, 7, 8, and 9 did not occur. AMSTAR-2 appraised 32.6% (14/43) of the studies as “moderate” quality, 32.6% (14/43) as “low” quality, and 34.9% (15/43) as “critically low” quality. No systematic reviews were rated as “high” quality. There was no significant association between the presence of spin and the following study characteristics: intervention type, article discussing Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) adherence, journal recommending PRISMA adherence, funding sources, journal 5-year impact factor, year the review was received for publication, or AMSTAR-2 critical appraisals.

Conclusion:

Spin was present in abstracts of systematic reviews and meta-analyses—covering Achilles tendon tear treatment. Steps should be taken to improve the reporting quality of abstracts on Achilles tendon treatment as well as other common orthopedic conditions.

Clinical relevance:

In order to avoid negative patient outcomes, articles should be free of spin within the abstract.

Introduction

Approximately 18 in every 100 000 people have experienced a ruptured Achilles tendon. The injury is often seen in middle-aged men who engage in athletic activity recreationally and inconsistently. Ruptures are also common among professional athletes, especially members of the National Football League and National Basketball Association, as these sports involve explosive motions and rapid directional changes. ¹¹ Despite the prevalence of this condition, treatment options are contested. The injury itself can result in career-altering changes in a professional athlete’s ability to return to play, ²⁰ rates of which range from 61% to 65% among professional basketball, football, and baseball players. ²² With that in mind, most physicians consider nonoperative vs open repairs in the context of their patient’s level of activity. ²¹ Retear rates and posttreatment plantarflexion strength are also factors, with evidence demonstrating improved outcomes from operative treatments. ¹¹ Given the prevalence of the injuries and, regarding athletes, the high stakes of recovery, evidence on the best clinical practices to treat this injury is crucial, particularly for orthopedic surgeons.

Meta-analyses and systematic reviews offer collections of rich data that practitioners can use across specialties, including orthopedics. In order to avoid misinterpretation—which could result in suboptimal patient care and outcomes—the findings in these reviews must be articulated clearly, precisely, and objectively. Framing data with an overemphasis on positive results or an underemphasis on negative results has been described in literature across specialties, and is often referred to as “spin.” ²⁴ In terms of knowledge transfer, clinicians often rely on information that is immediately available to them, because of time or resource constraints, including glancing through the results section of an article or simply reading the conclusion of an abstract. ³ Although it is the duty of clinicians to ensure the evidence they employ in practice is sound, systematic reviewers are also obligated to present their findings accurately and without spin.

Nevertheless, spin has been identified in various fields in randomized controlled trials, and more recently it has been documented in systematic reviews. ^2,13,15,16 Yavchitz and colleagues ²⁴ published a comprehensive article delineating the 9 types of spin that they regard as most significant in the context of medical decision making. Because of the frequency of Achilles tendon rupture and the impact it can have on young athletes’ careers, we believe that systematic reviews and meta-analyses on repairing the tears are important and that their results should be reported clearly. The aim of our study is to apply the parameters designed by Yavchitz et al ²⁴ to systematic reviews and meta-analyses of Achilles tendon repairs. The objectives of the study included evaluating whether particular study characteristics are associated with spin in systematic reviews and meta-analyses abstracts related to Achilles tendon rupture, as well as evaluating the prevalence of spin and its subtypes in the abstracts of systematic reviews and meta-analyses related to Achilles tendon rupture.

Materials and Methods

Search Strategy

A systematic review librarian (investigator) developed the search strategy for the MEDLINE (Ovid) and Embase (Ovid) databases. The goal of the search was to identify systematic reviews and meta-analyses that focused on the treatment of or quality of life after an Achilles tendon rupture. The search strategies are shown in Figure 1. The search was conducted in June 2020. The results of the search were uploaded into Rayyan, which is a systematic review screening platform. ¹⁷ At that point, (investigator) and (investigator) removed any duplicate articles and screened each article for inclusion eligibility. Screening was performed in a masked and duplicated fashion. Any screening disagreements were resolved through discussion between (investigator) and (investigator).

Figure 1.

Search strategies to obtain systematic reviews.

Eligibility Criteria

An article had to meet the following criteria to be included in our sample: (1) the article must be a systematic review or meta-analysis; (2) the article must discuss the treatment of, or quality of life after, an Achilles tendon rupture; (3) the article must be in English; (4) the article must contain only human subjects; and (5) the article must have an abstract.

Training

The authors tasked with data extraction (investigator and investigator) participated in training prior to starting the study. The training consisted of an online training course on systematic reviews and meta-analyses. ¹⁰ Following the online training, (investigator) and (investigator) participated in 2 days of in-person and online training. During the 2 days, the 9 most severe forms of spin defined by Yavchitz et al ²⁴ were assessed, and examples of each form of spin were discussed in detail. Following training on spin identification, (investigator) and (investigator) were trained to analyze a systematic review using AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews–2). A detailed outline of the training can be found in the study’s protocol.

Data Extraction

(Investigator) and (investigator) extracted data using a pilot-tested Google form, in a masked, duplicate fashion. They resolved any disagreements between them through discussion. If an agreement could not be reached, (investigator) and (investigator) served as arbiters.

The following general characteristics were extracted from each review: the intervention type (surgery, pharmacologic, nonpharmacologic, and education); whether the review discussed adherence to Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) ¹⁴ or PRISMA for abstracts (PRISMA-A) ⁴ ; whether the journal recommended adherence to PRISMA; the funding source (industry, private, public, none, not mentioned, hospital, a combination of funding not including industry, a combination of funding not including industry); the journal’s 5-year impact factor; and the year in which the review was received by the journal. Following the extraction of general characteristics, each article was analyzed for the 9 most severe forms of spin previously identified. ²⁴ Each of the 9 types of spin extracted is shown in Table 1. After spin extraction, the quality of each review was analyzed using AMSTAR-2. ¹⁸ Based on the 16-item appraisal tool, each review was rated as high, moderate, low, or critically low quality. A CONSORT flow diagram representing our articles is shown in Figure 2.

Table 1.

Spin Types and Frequencies (%) in Abstracts (n = 43).

Nine Most Severe Types of Spin	No. (%) of Abstracts Containing Spin
1. Conclusion contains recommendations for clinical practice not supported by the findings.	3 (7.0)
2. Title claims or suggests a beneficial effect of the experimental intervention not supported by the findings.	0 (0)
3. Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention.	23 (53.5)
4. Conclusion claims safety based on non–statistically significant results with a wide confidence interval.	1 (5.3)a
5. Conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in primary studies.	7 (16.3)
6. Selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention.	4 (9.3)
7. Conclusion extrapolates the review’s findings to a different intervention (ie, claiming efficacy of 1 specific intervention although the review covers a class of several interventions).	0 (0)
8. Conclusion extrapolates the review’s findings from a surrogate marker or a specific outcome to the global improvement of the disease.	0 (0)
9. Conclusion claims the beneficial effect of the experimental treatment despite reporting bias.	0 (0)

^a As a result of 24 studies not investigating interventions necessitating safety outcomes or measures, n = 19.

Figure 2.

Flow diagram.

Statistical Analysis

The overall frequency of spin, as well as its subtypes, were analyzed using descriptive statistics. All of the results are reported both as frequency counts and percentages. Following the screening we included 43 systematic reviews in our analysis. Since the number of included reviews was short of the number of reviews suggested by the power analysis, we used a Fisher exact test to evaluate the relationship between study characteristics and presence of spin. All our analytic decisions are outlined in our protocol. Stata 16.1 was used for all analyses.

Transparency, Reproducibility, and Reporting

Because our study did not include human subjects, it did not meet the definition of human subjects research of the US Code of Federal Regulations and was not subject to review by the institutional review board. To promote reproducibility and transparency, we put our protocol, training material, and extraction forms on the Open Science Framework. This study was conducted in tandem with other studies that also evaluated spin in systematic reviews across medical conditions. Because these used a common methodology, these methods have been described elsewhere.

Results

General Characteristics

Our search strategy produced 251 articles for review. Of the procured articles, 79 were duplicates; these were removed. (investigator) and (investigator) then screened the remaining 172 abstracts according to our predefined inclusion/exclusion criteria. Of these remaining abstracts, 110 were excluded and 62 were included for full-text extraction. During full-text data extraction, we excluded another 19 articles. The rationale for all exclusions is shown in Figure 2. We finished with a total of 43 systematic reviews and meta-analyses in our sample. The most commonly evaluated intervention was operative, which was the intervention in 76.7% (33/43) of the included articles. Of the 43 studies, 46.5% (20/43) stated adherence to PRISMA guidelines, and 41.9% (18/43) of the included journals were published in PRISMA-endorsing journals. Of the 43 articles, it was found that 41.9% (18/43) of the studies reported no source of funding, with the most common funding source being public (14.0%, 6/43). All of the characteristics of the included systematic reviews and meta-analyses are given in Table 2.

Table 2.

General Characteristics of Systematic Reviews and Meta-analyses.

Characteristics	No. (%) of Articles (n = 43)
	Total (%)	Abstract Without Spin	Abstract With Spin	P Value
Intervention type				.45a
Mixed	0 (0)	0 (0)	0 (0)
Nonpharmacologic	10 (23.3)	2 (4.7)	8 (18.6)
Pharmacologic	0 (0)	0 (0)	0 (0)
Surgery	33 (76.7)	13 (30.2)	20 (46.5)
Article reports adherence to PRISMA				.51b
No	23 (53.5)	7 (16.3)	16 (37.2)
Yes	20 (46.5)	8 (18.6)	12 (27.9)
Publishing journal recommends adherence to PRISMA				.64b
No	25 (58.1)	8 (18.6)	17 (39.5)
Yes	18 (41.9)	7 (16.3)	11 (25.6)
Funding source				.20a
Not funded	18 (41.9)	4 (9.3)	14 (32.6)
Private	0 (0)	0 (0)	0 (0)
Combination of funding including industry	1 (2.3)	1 (2.3)	0 (0)
Combination of funding not including industry	0 (0)	0 (0)	0 (0)
Public	6 (14.0)	3 (7.0)	3 (7.0)
Not mentioned	17 (39.5)	6 (14.0)	11 (25.6)
Industry	1 (2.3)	1 (2.3)	0 (0)
AMSTAR-2 rating				.79a
High	0 (0)	0 (0)	0 (0)
Moderate	14 (32.6)	6 (14.0)	8 (18.6)
Low	14 (32.6)	4 (9.3)	10 (23.3)
Critically low	15 (34.9)	5 (11.6)	10 (23.3)
Journal Impact Factor, M (SD)	4.09 (4.66)	3.95 (2.75)	4.18 (5.58)	.88c (OR: 1.01; 95% CI: 0.88-1.17)
Year study was received (1997-2020)				.67c (OR: 0.97; 95% CI: 0.84-1.11)

Abbreviations: AMSTAR-2, A Measurement Tool to Assess Systematic Reviews–2; CI, confidence interval; M, mean; OR, odds ratio; PRISMA, Preferred Reporting Items for Systematic reviews and Meta-Analyses.

^a Fisher exact test.

^b Pearson χ².

^c Logistic regression

Spin in Abstracts of Systematic Reviews and Meta-analyses

Of the 43 systematic reviews in our investigation, 65.1% contained spin (28/43). However, several abstracts that contained spin contained more than 1 type of spin. We found a total of 38 different spin occurrences. The most common form of spin, type 3 (Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention), was found in 23 of the 43 abstracts (53.5%). The second most common form of spin, type 5 (Conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in primary studies), was found in 7 of the 43 abstracts (16.3%). No examples of spin type 2, 7, 8, or 9 were identified (Table 1). There was no significant association between the presence of spin and any particular characteristics (Table 2). The presence of spin was not related to the journal’s 5-year impact factor (OR: 1.01; 95% CI: 0.88-1.17) or the year the systematic review was received (OR: 0.97; 95% CI: 0.84-1.11).

AMSTAR-2

AMSTAR-2 appraised 14 (32.6%) studies as “moderate” quality, 14 (32.6%) as “low” quality, and 15 (34.9%) as “critically low” quality. None of the studies was rated as “high” quality. We found that 42 of 43 (97.7%) of the systematic reviews formulated their research question using the Population, Intervention, Comparator group, Outcome (PICO) method. Only 1 study adhered to criterion 7 (Review authors provided a list of excluded studies with justification for exclusion 1/43, 2.3%). Additionally, 17 studies did not perform a meta-analysis; thus, these studies could not be assessed by criterion 11, 12, or 15. There was no significant association between the methodological quality of a study and the presence of spin in the abstract (Table 2). All AMSTAR-2 criteria and the frequency of responses are illustrated in Table 3.

Table 3.

AMSTAR-2 Items and Frequency of Responses.

AMSTAR-2 Item	Response, n (%)
	Yes	No	Partial Yes
1. Did the research questions and inclusion criteria for the review include the elements of PICO?	42 (97.7)	1 (2.3)	0 (0)
2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?	7 (16.3)	31 (72.1)	5 (11.6)
3. Did the review authors explain their selection of the study designs for inclusion in the review?	8 (18.6)	35 (81.4)	0 (0)
4. Did the review authors use a comprehensive literature search strategy?	3 (7.0)	13 (30.2)	27 (62.8)
5. Did the review authors perform study selection in duplicate?	28 (65.1)	15 (34.9)	0 (0)
6. Did the review authors perform data extraction in duplicate?	24 (55.8)	19 (44.2)	0 (0)
7. Did the review authors provide a list of excluded studies and justify the exclusions?	1 (2.3)	42 (97.7)	0 (0)
8. Did the review authors describe the included studies in adequate detail?	5 (11.6)	1 (2.3)	37 (86.0)
9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?	26 (60.5)	15 (34.9)	2 (4.7)
10. Did the review authors report on the sources of funding for the studies included in the review?	22 (51.2)	21 (48.8)	0 (0)
11. If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results?a	24 (55.8)	2 (4.7)	0 (0)
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?a	20 (46.5)	6 (14.0)	0 (0)
13. Did the review authors account for RoB in primary studies when interpreting/discussing the results of the review?	25 (58.1)	18 (41.9)	0 (0)
14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?	24 (55.8)	19 (44.2)	0 (0)
15. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?a	10 (23.3)	16 (37.2)	0 (0)
16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review	30 (69.8)	13 (30.2)	0 (0)

Abbreviations: AMSTAR-2, A Measurement Tool to Assess Systematic Reviews–2; PICO, Population, Intervention, Comparator group, Outcome (PICO) method.

^a Seventeen articles did not perform a meta-analysis.

Discussion

Our study found that spin was present in about 65% of systematic review abstracts on Achilles tendon injuries. This amount of spin corresponds to previous published studies analyzing spin in abstracts in orthopedic randomized controlled trials. For example, Arthur et al ¹ reviewed 250 randomized controlled trials published in orthopedic surgery journals and found spin in 44% of abstracts. Another study of spin in lower extremity joint trials found spin in 58.7% of 46 abstracts. ⁶ Outside of orthopedic surgery, spin was identified in more than 50% of trial abstracts published in psychiatry journals and 70% of the trial abstracts in otolaryngology journals. ^8,12 We also evaluated whether particular study characteristics were associated with spin but found no statistically significant relationships. Other findings have shown that reporting quality is associated with higher journal impact factors and adherence to PRISMA-A. ¹⁵ The discrepancy between the findings of O’Donohoe ¹⁵ and our study could be due to the difference in sample sizes: 257 reviews in the O’Donohoe study and 43 reviews in our study. It is equally possible that the discrepancy was due to the overall poor reporting quality of orthopedic literature. Of the 9 forms of spin, the third “most severe” form found by Yavchitz et al ²⁴ (Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention) as the most common in our sample. In the paragraphs that follow, we will highlight this specific type of spin and its potential effects on Achilles tendon rupture treatment.

Overemphasis on efficacy was identified in a systematic review by Song and Hua. ¹⁹ In the conclusion of this abstract, the authors state, “tendon allograft offers favorable outcomes in patients with chronic Achilles tendon rupture.” In the abstract, the authors did not mention that 3 of the included primary studies reported complications with tendon allograft, including delayed healing of the incision, infection, fragmented calcaneal tuberosity, and interosseous ossification proximal to the insertion. The failure to report such adverse effects with efficacy outcomes may lead to an overemphasis on efficacy, with orthopedic surgeons being led to believe that using a tendon allograft for the repair of an Achilles tendon rupture is safer than it is. Reporting the benefit of the allograft as well as the complications would provide a more complete picture to the readers and allow for surgeons to make a complete decision when choosing an operative technique.

Our study used AMSTAR-2 to appraise the methodological quality of the systematic reviews in our sample. Our study found that the most frequent rating for Achilles tendon systematic reviews and meta-analyses was “critically low” and that none of the included studies received a rating of “high.” The presence of spin was equally distributed across study quality, with the exception of high quality, of which we had none. This finding, although based on a small sample size, suggests that the methodologic quality and the use of spin are not related. It would have been interesting to observe whether this pattern exists for high-quality reviews and may be an interesting avenue for future research based on more robust sample sizes, which may be available for other orthopedic-related conditions. Also of concern is the dearth of high-quality systematic reviews that we found, and although not the primary focus of this investigation, perhaps future studies should further evaluate the methodologic quality or risk of bias in systematic reviews across other conditions. Doing so would shed additional light on this issue.

Currently, several available treatment options (ie, open surgery, percutaneous surgery, nonoperative) exist for Achilles tendon ruptures. Therefore, determining which intervention is best suited for a particular patient is often determined through a shared decision-making model. Using this model allows a physician and patient to determine the best treatment option based on a balance between desired patient outcomes and risks of the intervention. ⁹ When placed within the context of sports medicine, determining the best treatment option becomes of utmost importance because of the vast array of different sports injuries, athlete age groups, and desired performance outcomes. Our study found that several abstracts contained spin type 6 (Selective reporting of or overemphasis on harm outcomes, or analysis favoring the safety of the experimental intervention). An example of spin type 6 occurred in a study by Yang et al, ²³ which claimed the superiority of percutaneous repair for Achilles tendon ruptures but failed to report all harm outcomes studied. Thus, a systematic review that selectively reports harm outcomes has the potential to misguide, especially in situations when the clinician only has access to an abstract. This notion of misguiding is supported by previous research, which found that spin does influence a physician’s interpretation of results, but even when an abstract did contain spin, physicians often ranked the experimental treatment as more beneficial than physicians who were given an abstract without spin. ⁵ Thus, it is imperative that abstracts of systematic reviews remain void of spin and that all outcomes of interest are reported, because these evidence-based reviews are used to determine treatment options.

Recommendations

Efforts to reduce spin will require action from multiple stakeholders. For authors, awareness of spin—and how to avoid using it—could be accomplished in current reporting guidelines for systematic reviews, namely, PRISMA and PRISMA-A, if they incorporate spin as a reporting problem. Continuing education sessions, conferences, webinars, and other events that teach about spin would also help raise awareness. For journals, providing specific guidance to authors and peer reviewers about spin would be helpful. The endorsement of reporting guidelines, should spin be incorporated, would likely lessen the prevalence of spin as well; previous studies have found that PRISMA-endorsing journals tend to publish systematic reviews that are more completely reported than non-PRISMA-endorsing journals. Journal editors could also consider expanding the number of words allowed in abstracts, which would allow authors more space to write about adverse effects and other important findings. Further, journals could revise their current instructions to discourage spin.

Strengths and Limitations

Our study had both strengths and limitations. With respect to its strengths, our team included a systematic review librarian to develop our search strategies and perform our searches. All screening was performed and all data were extracted by 2 authors (team members) independently and in a masked fashion — currently the method suggested by the Cochrane Collaboration. ⁷ We placed our protocol, extraction forms, data, analysis scripts, and other documents on the Open Science Framework in an effort to increase the reproducibility of our study. With respect to its limitations, our study was cross-sectional; thus, our results should not be generalized beyond our sample. Also, our sample size was small. While we deployed searches of the most common bibliographic databases used in systematic reviews, it is possible that other databases may have returned additional systematic reviews on Achilles tendon repair. Finally, the classification of spin is a subjective process. Others may not fully agree with our assessments.

Conclusion

Spin was present in the abstracts of systematic reviews and meta-analyses covering Achilles tendon rupture treatment, and many systematic reviews received an AMSTAR-2 rating of “low” or “critically low.” As initial evidence suggests that spin may influence perceptions about a study—which could have downstream effects on clinical decision making—steps should be taken to improve the reporting quality of abstracts on Achilles tendon systematic reviews and meta-analyses, as well as other common orthopedic conditions.