Evaluation of spin in the abstracts of systematic reviews and meta-analyses covering surgical management, or quality of life after surgical management, of osteoarthritis of the knee

Abstract

Objective

Our primary objective was to identify the prevalence of spin — misleading reporting practices that overemphasize benefit or underemphasize harm — within the abstracts of systematic reviews and meta-analyses focused on surgical management of osteoarthritis of the knee.

Methods

A search string was developed to search Ovid MEDLINE and Ovid Embase for articles pertaining to surgical management, or quality of life after surgical management, of osteoarthritis of the knee. Titles and abstracts were screened according to our protocol, developed a priori, followed by full-text evaluation for spin in included articles. Study characteristics were simultaneously extracted with spin data and each included study received an AMSTAR-2 quality appraisal. All procedures were performed by two examiners in a masked, duplicate fashion.

Results

Of the 1419 articles returned, 96 systematic reviews qualified for inclusion. 35.4% of the included abstracts (34/96) contained at least one type of spin with a total of 36 occurrences (two abstracts contained two types of spin). Selective reporting favoring benefit (type 3; 15/36, 41.7%) was the most prevalent followed by selective reporting of harms (type 6; 7/36, 19.4%). None of the abstracts contained spin types 2, 4, or 8. We found no significant association between spin and either AMSTAR-2 rating or extracted study characteristics.

Conclusion

Of the included systematic reviews and meta-analyses, 35.4% contained spin in their abstract. To improve the reliability of systematic reviews and meta-analyses, researchers should act to minimize spin in future abstracts.

1. Introduction

As the baby boomer population continues to age, cases of osteoarthritis will increase [1]. In fact, the prevalence of knee osteoarthritis has risen nearly two-fold since the mid-twentieth century, affecting nearly 52.5 million American adults [2,3]. With osteoarthritis being the leading cause of disability in adults [1,3], this will increase the burden on both patients and global healthcare systems. Accordingly, it is essential that those in the orthopedic community not only critically analyze the available treatment options for this condition, but also the quality of the research that supports these treatment approaches.

Systematic reviews and meta-analyses have long been considered the gold standard in guiding clinical practice, guidelines, and patient outcomes [4,5]. It is therefore vital that these studies be critically analyzed and that any discovered gaps or biases be addressed in future work. Doing so will ensure that patients receive the best possible care that is based on the highest levels of evidence and derived from studies that meet the highest methodological standards.

Unfortunately, this is complicated by recent discoveries of “spin” within studies in which the data and/or study results are misrepresented. The most widely recognized definition of spin is “a specific way of reporting, intentional or not, to highlight that the beneficial effect of the experimental treatment, in terms of efficacy or safety, is greater than that shown by the results [6].” Spin can take a variety of forms, including writing about the secondary outcomes before or instead of the primary outcome, only reporting one favorable outcome, changing the prospectively registered primary outcomes, or failing to mention any unfavorable outcomes [7]. Spin can also be located in multiple portions of a manuscript – the most concerning (and most common) of which is in the abstract [[8], [9], [10], [11]]. It is deeply problematic when spin is contained in the abstracts of published studies because physicians often only have the time and resources to view the abstract section [12]. Against this backdrop, the ever-growing incidence of osteoarthritis makes it imperative to evaluate spin in the abstracts of osteoarthritis systematic reviews.

While spin has been observed in orthopedic randomized controlled trials (RCTs) [13,14], to our knowledge, no appraisal of spin in orthopedic systematic reviews and meta-analyses has been conducted. Thus, the primary objective of this study was to evaluate the presence or absence of spin in the abstracts of systematic reviews on surgical management of osteoarthritis of the knee by using the classification system created by Yavchitz and colleagues [6]. Specifically, we aimed to evaluate the top nine most severe examples of spin as described by Yavchitz et al. [6]. Our secondary objective assessed study characteristics associated with spin in the abstracts of systematic reviews of the surgical management of osteoarthritis of the knee. Due to previously published findings regarding spin within the abstracts of RCTs [[8], [9], [10], [11],15], and considering the overall poor reporting quality in systematic reviews [[16], [17], [18], [19]], we hypothesize that spin will exist in the abstracts of knee osteoarthritis systematic reviews.

2. Methods

2.1. Oversight, transparency, reproducibility, and reporting

Because no humans were involved in this study, it does not meet the regulatory definition of human subject research per the U.S. Code of Federal Regulations. Therefore, it was not subject to institutional review board oversight. The transparency and reproducibility of this study are supported through the public posting of the associated protocol, data analysis scripts, data extraction forms, and other study artifacts to the Open Science Framework (OSF) [20]. Furthermore, to enhance inferential reproducibility [21], the data were re-analyzed by an independent investigation team using the original analysis scripts. Due to the simultaneous conduction of this study with other studies evaluating the presence of spin in systematic reviews in different medical fields, similar methods have been described elsewhere.

2.2. Search strategy

A systematic review librarian (DW) developed search strategies for both MEDLINE (Ovid) and Embase (Ovid) databases to obtain systematic reviews and meta-analyses on the surgical management of osteoarthritis of the knee (Fig. 1). The search was conducted by DW in June 2020, and the articles returned by these searches were uploaded to the systematic review screening platform Rayyan [22]. To determine eligibility, two of us (WN and PS) removed duplicates and screened the remaining records by title and abstract in a masked, duplicate manner. All screening discrepancies were resolved by WN and PS through consensus agreement.

Fig. 1.

Search string.

2.3. Eligibility criteria

Articles were included if they met the following criteria: (1) the article must be a systematic review or a meta-analysis as defined by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA-P) [23], (2) the article must be focused on the surgical management of osteoarthritis of the knee or the quality of life after the surgical management of osteoarthritis of the knee, (3) the article must be conducted only on human subjects, (4) the article must be available in English due to the literacy of the authors, and (5) the article must contain an abstract.

2.4. Training

WN and PS completed an online training course covering systematic reviews and meta-analyses by Li and Dickersin [24] before beginning title and abstract screening. These examiners then completed two days of training both online and in-person as to the definition and interpretation of the nine most severe types of spin in systematic review abstracts as defined by Yavchitz et al. [6]. Additionally, the authors were trained in A MeaSurement Tool to Assess Systematic Reviews (AMSTAR-2), a tool we used to determine the methodological quality of each included systematic review and meta-analysis. The details of all completed training are located in the protocol of this study.

2.5. Data extraction

Examiners WN and PS thoroughly assessed the abstracts of all included systematic reviews and meta-analyses using a pilot-tested Google form to extract the desired data in a masked, duplicate fashion. The primary data extracted was the presence of the nine most severe types of spin located within the abstracts of the included studies. The definition of each of these spin types is found in Table 1. The following are additional study characteristics extracted from each included study: (1) the year the associated journal received the review, (2) the review's source of funding (public, private, industry, hospital, a combination of funding including industry, a combination of funding not including industry, none, not mentioned, or other), (3) whether or not the review discussed adherence to PRISMA or PRISMA for abstracts, (4) whether or not the associated journal required or recommended adherence to PRISMA, and (5) the journal's five-year impact factor. After completing the data extraction, examiners (WN and PS) were unmasked, and discrepancies were resolved. If an agreement could not be reached, additional authors (MV and RO) acted as mediators. Following data extraction, WN and PS graded the methodological quality of each review as critically low, low, moderate or high using the 16-item AMSTAR-2 scale [25]. This process was also performed in a masked and duplicate manner with all discrepancies resolved as previously described above.

Table 1.

Spin types and frequencies (%) in abstracts.

Nine most severe types of spin	No. (%) of Abstracts With Spin (n = 96)
1) Conclusion contains recommendations for clinical practice not supported by the findings.	3 (3.1)
2) Title claims or suggests a beneficial effect of the experimental intervention not supported by the findings.	0 (0)
3) Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention.	15 (15.6)
4) Conclusion claims safety based on non-statistically significant results with a wide confidence interval.	0 (0)
5) Conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in primary studies.	5 (5.2)
6) Selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention.	7 (7.3)
7) Conclusion extrapolates the review's findings to a different intervention (i.e., claiming efficacy of one specific intervention although the review covers a class of several interventions).	5 (5.2)
8) Conclusion extrapolates the review's findings from a surrogate marker or a specific outcome to the global improvement of the disease.	0 (0)
9) Conclusion claims the beneficial effect of the experimental treatment despite reporting bias.	1 (1.0)

2.6. Statistical analysis

All statistical analyses were performed using Stata 16.1 (Stata Corp, LLC, College Station, TX), and results are reported as frequency counts followed by percentages (count/total,%). Our sample size of 96 did not reach the minimum sample size of 185 that was calculated and pre-specified in our protocol. Therefore, we could not perform a multivariate logistic regression. As an alternative, we evaluated the relationships between the presence of spin and study characteristics through the use of univariate logistic regression analysis. These analytic decisions are congruent with those discussed in our protocol.

3. Results

After removing duplicates, our searches retrieved 1419 articles, of which 1174 were excluded following abstract and title screening. An additional 149 articles were excluded during full-text analysis; thus, 96 systematic reviews were retained for data extraction. Fig. 2 demonstrates our screening process with the rationale for all excluded articles. Of the 96 studies, 67 reported adherence to PRISMA guidelines (out of 96, 69.8%), and 62 of the publishing journals recommended a PRISMA commitment to their author guidelines (out of 96, 64.6%). Twenty-three studies were funded (23/96, 24.0%), with the most common funding sources being public (17/96, 17.7%). Forty-four systematic reviews in our sample did not provide a funding statement (out of 96, 45.8%), and 29 reported that their study received no funding (out of 96, 30.2%). The mean 5-year impact factor was 3.35 (standard deviation: 4.58). All systematic reviews were received between the years 2000–2019 (Table 2).

Fig. 2.

Flow diagram of study selection.

Table 2.

General characteristics of systematic reviews and meta-analyses.

Characteristics	No. (%) of Articles (n = 96)
	Total (%)	Abstract Contains Spin	Odds Ratio (95% CI)
Article mentions adherence to PRISMA
No	29 (30.2)	9 (9.4)	1 [Reference]
Yes	67 (69.8)	25 (26.0)	1.32 (0.52–3.35)
Publishing journal recommends adherence to PRISMA
No	34 (35.4)	16 (16.7)	1 [Reference]
Yes	62 (64.6)	18 (18.8)	0.46 (0.19–1.10)
Funding Source
Not Funded	29 (30.2)	8 (8.3)	1 [Reference]
Private	6 (6.3)	0 (0)	1 -
Public	17 (17.7)	6 (6.3)	1.43 (0.40–5.18)
Not Mentioned	44 (45.8)	20 (20.8)	2.19 (0.80–5.99)
AMSTAR-2 Rating
High	3 (3.1)	1 (1.0)	1 [Reference]
Moderate	14 (14.6)	8 (8.3)	2.67 (0.19–36.76)
Low	18 (18.8)	7 (7.3)	1.27 (0.10–16.81)
Critically low	61 (63.5)	18 (18.8)	0.84 (0.07–9.83)
Journal Impact Factor, M (SD)	3.35 (4.58)	3.67 (5.45)	1.02 (0.92–1.14)
Publication Year (2000–2019)			0.99 (0.89–1.10)

3.1. Spin in abstracts

In total, 34 abstracts contained at least one type of spin (out of 96, 35.4%). Two abstracts had more than one spin type, resulting in 36 occurrences of spin in our sample. Spin type 3 (“selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention [6]”) occurred most frequently, accounting for 41.7% of all the spin occurrences (15/36). Spin type 6 (“selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention [6]”) was the second most common type, being identified in 7 different abstracts (out of 96, 7.3%). No abstracts contained spin types 2,4, or 8 (Table 1). Regression models found no significant association between the presence of spin and whether the systematic review mentioned or the journal recommends adherence to PRISMA. Additionally, no significant association was found between the presence of spin and the systematic review's funding source, the journal's 5-year impact factor, or the year the systematic review was received by the publishing journal (Table 2).

3.2. AMSTAR-2 rating

Analysis of the AMSTAR-2 quality assessment data for each of the 96 systematic reviews and meta-analyses rated 3 as “high” (3.1%), 14 as “moderate” (14.6%), 18 as “low” (18.8%), and 61 as “critically low” (63.5%; Table 2) Furthermore, 61 of the 96 included systematic reviews and meta-analyses did not account for the risk of bias when interpreting or discussing their results (63.5%; Table 3). No significant association could be drawn between the presence of spin and the methodological quality of the systematic review.

Table 3.

AMSTAR-2 items and frequency of responses.

AMSTAR-2 Item	Response, n (%)
	Yes	No	Partial Yes
1) Did the research questions and inclusion criteria for the review include the elements of PICO?	95 (99.0)	1 (1.0)	0 (0)
2) Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?	22 (22.9)	56 (58.3)	18 (18.8)
3) Did the review authors explain their selection of the study designs for inclusion in the review?	14 (14.6)	82 (85.4)	0 (0)
4) Did the review authors use a comprehensive literature search strategy?	14 (14.6)	44 (45.8)	38 (39.6)
5) Did the review authors perform study selection in duplicate?	73 (76.0)	23 (24.0)	0 (0)
6) Did the review authors perform data extraction in duplicate?	62 (64.6)	34 (35.4)	0 (0)
7) Did the review authors provide a list of excluded studies and justify the exclusions?	14 (14.6)	28 (29.2)	54 (56.3)
8) Did the review authors describe the included studies in adequate detail?	68 (70.8)	10 (10.4)	18 (18.8)
9) Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?	36 (37.5)	28 (29.2)	7 (7.3)
10) Did the review authors report on the sources of funding for the studies included in the review?	9 (9.4)	87 (90.6)	0 (0)
11) If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results?a	34 (72.3)	13 (27.7)	0 (0)
12) If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?a	28 (59.6)	19 (40.4)	0 (0)
13) Did the review authors account for RoB in primary studies when interpreting/discussing the results of the review?	35 (36.5)	61 (63.5)	0 (0)
14) Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?	55 (57.3)	41 (42.7)	0 (0)
15) If they performed quantitative synthesis did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?a	25 (53.2)	22 (46.8)	0 (0)
16) Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review	79 (82.3)	17 (17.7)	0 (0)

^a49 articles did not perform a meta-analysis, so N = 47.

4. Discussion

Our study found that over one-third of systematic reviews focused on the surgical management of osteoarthritis of the knee contained spin within their abstracts. Our findings are consistent with other spin studies in orthopedic surgery. For example, similar studies by Checketts and colleagues [14] and Arthur and colleagues [13] identified spin in 58.7% of lower extremity randomized control trials and 44.8% of orthopedic randomized control trials, respectively.

Spin type 3 — “selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention [6]” — was the most common type of spin in our study. An example of type 3 spin occurred in a systematic review performed by Wang and colleagues [26], which investigated pain control after total knee arthroplasty. The secondary outcomes identified for investigation were the length of hospital stay and postoperative complications; however, in the abstract, the authors selectively reported only one of these outcomes, postoperative complications. With this example, it is worth mentioning that abstracts are often used by clinicians when formulating treatment plans [12,27,28]; thus, selectively reporting only one of the secondary outcomes does not provide the clinician with all the benefits and risks associated with the procedure.

Our second most identified spin type was type 6 — “selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention [6]” — which is similar to type 3, as both types are associated with authors selectively reporting findings. An example of spin type 6 can be found in the abstract of a systematic review performed by Li and colleagues [29] in which they selectively reported deep vein thrombosis as the most prevalent complication of simultaneous high tibial osteotomy and anterior cruciate ligament reconstruction procedures at a rate of 7.7%. However, when examining the full-text it becomes apparent that the authors did not include the total complication rate of 24.3% in the abstract, instead, selectively reporting a fraction of this total. As stated previously, abstracts are used by clinicians in their clinical decision-making processes; therefore, failing to report all outcomes has the potential to affect patient care.

4.1. Recommendations

Studies have shown that peer reviewers often fail to identify spin and, in a few instances, actually recommend that authors add a form of spin [30]. Knowing this, it would seem logical to provide education and training to peer reviewers and editors to combat spin's presence. However, we believe that the best initial approach in preventing spin is to provide resources for the people who create it: the authors. We recommend that journals include examples and explanations of spin within their instructions for authors. This education would allow authors unfamiliar with spin to introspectively analyze the work they are submitting for spin. Interestingly, our results show no difference between the presence of spin and whether the authors mentioned or the journal recommended PRISMA adherence. We did not detect decreases in spin's frequency following the 2013 release of the PRISMA extension for abstracts (PRISMA-A) [31] that was developed to improve abstract reporting quality. This suggests that either the PRISMA-A checklist needs to be reformed, journals should mandate its use, or both. Currently, the 12-item PRISMA-A checklist does not address spin. Therefore, we recommend specific amendments regarding spin be added to the PRISMA-A checklist. Although not exhibited in this study, studies show that PRISMA does improve the quality of reporting [16,32]. Therefore, we recommend that journals enforce strict adherence to PRISMA rather than recommend its use. The above recommendations would also increase awareness of spin. Increasing awareness of spin is important as it is a relatively newly recognized type of reporting bias. Thus, it is reasonable to conclude that increased awareness would decrease the prevalence of spin in future scientific literature.

4.2. Strengths and limitations

Both strengths and limitations exist in this study. With regards to its strengths, first, we took efforts to ensure that our study is reproducible by making our protocol, data, analysis scripts, data extraction forms, and other study materials freely available online. In addition, an independent group verified our results. Second, our protocol was strictly adhered to and developed a priori; any deviations from the protocol were recorded in an update and made public in the Open Science Framework. Third, screening and data extraction were performed in a masked and duplicate fashion to limit bias. This is considered by the Cochrane Collaboration to be the gold standard [33]. With regards to our study's limitations, first, the classification of spin is inherently subjective, which may lead to disagreement among readers and reporting bias. We acted to reduce this subjectivity with careful training, calibration exercises, and considerable evaluation of the spin definitions explained by Yavchitz and colleagues [6]. However, additional training and education regarding spin identification and its definitions may be required for future studies. Second, our study was cross-sectional in design; therefore, the results should not be generalized to systematic reviews published in other journals or time periods. Third, our data took into account various surgical methods for the treatment of osteoarthritis of the knee. More studies would be needed to further investigate the presence of spin as it pertains to specific therapies and treatment modalities. Finally, while we searched MEDLINE and Embase, the two largest bibliographic databases, it is possible that other databases may have cataloged additional systematic reviews pertaining to the surgical management of osteoarthritis of the knee.

5. Conclusion

Our data indicate that systematic reviews and meta-analyses regarding surgical management of osteoarthritis of the knee contain a large degree of the reporting bias described as “spin”. Furthermore, a majority of these studies were qualified as “critically low” using the AMSTAR-2 criteria. Despite this study's limitations, knowledge regarding biases such as spin can increase awareness and reduce its prevalence in future orthopedic surgery systematic reviews.

Author contributions

MV, RO, and WA were responsible for the conception and design of this study. DNW, JC, MV, MH, RO, PS, WA, and WN contributed to the development and writing of the corresponding protocol. DNW developed the search strategy and performed the initial database search of MEDLINE and Embase. PS and WN performed title and abstract screening, extracted data, and performed AMSTAR assessment of included studies. MH, SC, and ZM performed the data analysis. All authors contributed to the interpretation of the data and to drafting and revising the manuscript. The final version of this manuscript was approved by all authors.

Funding

Development of this protocol and study was funded by the Oklahoma State University Center for Health Sciences Presidential Mentor-Mentee Research Fellowship Grant.

Declaration of competing interest

The authors of this study declare no competing conflicts of interest.