Evaluation of spin in systematic reviews and meta-analyses involving glenoid augmentation in total shoulder arthroplasty

Abstract

Background

Spin in scientific literature is defined as bias that overstates efficacy and/or underestimates harms of an experimental treatment. Glenoid augmentation is a technique used to correct glenoid bone deficiency during shoulder surgery. Literature on the outcomes of glenoid augmentation in total shoulder arthroplasty (TSA) is limited, which may lead to a risk of bias in systematic reviews and meta-analyses in this topic. The goal of this study is to identify the quantity and types of spin in systematic reviews and meta-analyses of glenoid augmentation in TSA.

Methods

A search was conducted on the PubMed, Scopus, and SPORTDiscus databases for systematic reviews and meta-analyses evaluating outcomes of glenoid augmentation in TSA. The abstract of each study was assessed for the 15 most common types of spin using the method proposed by Yavchitz et al Study quality was assessed per A Measurement Tool to Assess Systematic Reviews 2. Fisher’s exact tests were performed to investigate relationships between general characteristics and spin types.

Results

Thirteen (81.3%) of the 16 included studies were found to have at least one form of spin. The most common type of spin was type 5 (“the conclusion claims the beneficial effect of the experimental treatment despite a high risk of bias in primary studies”), which appeared in 7 studies (43.8%). There was a statistically significant association between the presence of a conflict of interest and the lack of reporting funding sources.

Conclusion

Spin is highly prevalent in the abstracts of systematic reviews and meta-analyses studying glenoid augmentation with TSA. Misleading reporting is the most common category of spin.

Shoulder arthroplasty is an accepted technique for the management of a variety of conditions, including end-stage arthritis, traumatic injuries, and other degenerative conditions. Anatomic total shoulder arthroplasty (aTSA) and reverse total shoulder arthroplasty (rTSA) have a history of significant success in restoring function, decreasing pain, and reported 10- and 20-year survival rates ranging from 93% to 95% and 70% to 85%, respectively.^7,13,35, 36, 37^,47 However, many of these diagnoses may lead to challenges due to changes in bony composition.^18,32 Glenoid bone deficiency and severe deformities, such as increased retroversion pose challenges in component positioning, and may lead to future instability, implant loosening, and failure of the prothesis if not properly addressed.^10,17,43

A proposed solution to this problem is glenoid augmentation, which encompasses a spectrum of surgical approaches, including bone grafting and augmented implants, with the goal of restoring glenoid version and inclination.^30,46 The use of glenoid augmentation in shoulder arthroplasty has been shown to have low rates of complication and revision surgery.⁴⁶ More specifically, various polyethylene and metal component designs can be utilized in aTSA and rTSA to address these bony defects and improve long-term outcomes.^22,23 Bone grafting has also been utilized to provide additional support for implant fixation. Autograft, allograft, and synthetic grafts have been employed for reconstruction of the glenoid surface with positive long-term patient satisfaction.^4,5,24,48,54 Collectively, current glenoid augmentation techniques have been shown to play a role in enhancing implant stability and favorable outcomes in the setting of glenoid wear.

Despite these notable positive results, glenoid augmentation is not free from complications. Reported complications of glenoid augmentation include implant loosening, fracture of augmented bone, infection, and screw-related complications.^9,12,29 Revision rates due to symptomatic failure have been reported at 2%-21% and can lead to significant loss of bone stock.^12,26,31,33 Revision rates have been found to be similar between augmented and nonaugmented glenoid components.^21,34,52 Previous research has also found high rates of allograft failure due to resorption when used alone.^8,40 Given these reported complications, it is crucial to critically assess existing research when determining clinical implementation.

Previous literature on this topic often reports small sample sizes leading to lower levels of evidence. When these studies are then compiled into systematic reviews and meta-analysis, although stronger evidence, the risk of bias (RoB) is increased. This paper looks to uncover spin in such studies. Spin is defined as “specific way of reporting, intentional or not, to highlight that the beneficial effect of the experimental treatment in terms of efficacy or safety is greater than that shown by the results.”⁵³ Yavchitz et al, identified 39 different types of spin with three main categories as follows: misleading reporting, misleading interpretation, and inappropriate extrapolation.⁵³ Spin commonly occurs in the abstract of studies and is an important factor for clinicians to identify when reading current literature.⁴⁹ The purpose of this paper was to identify and summarize incidence of spin in the abstracts of systematic reviews and meta-analyses of glenoid augmentation in aTSA and rTSA. Our secondary purpose was to detect patterns in papers with spin regarding general study characteristics.

Methods

This systematic review was performed according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) guidelines. Search criteria and the methods of this study were determined a priori. A single author (C.L.) searched three databases (Pubmed, Scopus, and Sport Discus) for studies using the following terms: “glenoid augmentation,” “glenoid component augmentation,” “glenoid bone grafting,” “glenoid reconstruction,” “shoulder arthroplasty with glenoid augmentation,” “glenoid implant augmentation,” “half-wedged glenoid augmentation,” “full-wedge glenoid augmentation,” “stepped glenoid augmentation,” “glenoid prosthesis augmentation,” and “glenoid deficiency management.” The search was conducted in February 2024. The inclusion criteria included systematic reviews and meta-analyses, which examined outcomes of glenoid augmentation surgery. The exclusion criteria included narrative reviews, primary studies, and studies that did not have a full text available. Search results were aggregated in EndNote X9 (Clarivate, Philidelphia, PA, USA), and duplicates were removed. Two authors (C.L. and B.B.) independently screened the articles by title, abstract, and then full-text.

Three authors (C.L., B.B., and A.F.) were then trained to identify and characterize the 15 most commonly found types of spin in abstracts using the method proposed by Yavchitz et al,⁵³ which is summarized in Table I. The same 3 authors were also trained to assess the full text of the included studies using version 2 of A Measurement Tool to Assess Systematic Reviews (AMSTAR 2). AMSTAR 2 is a 16-point questionnaire that quantifies the quality of a systematic review based on criteria, such as whether authors report presence of bias, impact of bias, the use of a predetermined protocol, funding sources, and conflicts of interest, and/or adequately characterize studies included in the review.⁴⁵ AMSTAR 2 has undergone rigorous assessment itself and has demonstrated high inter-rater reliability and construct validity.²⁵

Table I.

Assessment of spin in articles involving glenoid augmentation.

Category	Type	Description	Abstracts with spin (n)	Abstracts without spin (n)
Misleading interpretation	1	The conclusion formulates recommendations for clinical practice not supported by the findings	0/16	16/16
	2	The title claims or suggests a beneficial effect of the experimental intervention not supported by the findings	0/16	16/16
	4	The conclusion claims safety based on nonstatistically significant results with a wide confidence interval	2/16	14/16
	9	Conclusion claims the beneficial effect of the experimental treatment despite reporting bias	6/16	10/16
	12	Conclusion claims equivalence or comparable effectiveness for nonstatistically significant results with a wide confidence interval	3/16	13/16
Misleading reporting	3	Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention	1/16	15/16
	5	The conclusion claims the beneficial effect of the experimental treatment despite a high RoB in primary studies	7/16	9/16
	6	Selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention	0/16	16/16
	10	Authors hide or do not present any conflict of interest	0/16	16/16
	11	Conclusion focuses selectively on statistically significant efficacy outcome	1/16	15/16
	13	Failure to specify the direction of the effect when it favors the control intervention	2/16	14/16
	14	Failure to report a wide confidence interval of estimates	3/16	13/16
Inappropriate extrapolation	7	The conclusion extrapolates the review findings to a different intervention (eg, claiming efficacy of one specific intervention although the review covered a class of several interventions)	0/16	16/16
	8	Conclusion extrapolates the review's findings from a surrogate marker or a specific outcome to the global improvement of the disease	0/16	16/16
	15	Conclusion extrapolates the review's findings to a different population or setting	2/16	14/16

RoB, risk of bias.

Data extraction

Data extraction was performed independently by C.L. and B.B. A third author (A.F.) resolved any disputes. Aside from spin data and AMSTAR data, data collected included journal, publication year, level of evidence, study design, reported adherence to PRISMA guidelines, preregistration of the study protocol with International Prospective Register of Systematic Reviews (PROSPERO), primary and secondary outcome measures, declaration of funding sources, presence of a conflict of interest, and whether or not the article conclusion was in favor of the experimental intervention. The 2022 Scopus CiteScore (Elsevier) and 2022 Clarivate impact factor were obtained for each journal.

Statistical analysis

The frequency of spin occurring in the abstracts of included studies was characterized using descriptive statistics. Study characteristics and the presence of spin were individually analyzed for an association using the Fisher exact test. RStudio (RStudio version 2022.7.1.554; Posit, Boston, MA, USA) was used for all analyses.

Results

Nine hundred and sixty-four articles were obtained from searching PubMed, Scopus, and Sport Discus databases. After removing duplicates, 673 articles were screened by title to assess for the inclusion criteria. Following title screening, 135 articles were screened by abstract, leaving 58 articles, which were screened by reading the full text. After all screening steps, 16 articles from 1 unique journals were included for the final analysis (Fig. 1).^12,14, 15, 16^{,21,27,28,33,38,41,42,44,46,50,51,54}

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) flowchart. TSA, total shoulder arthroplasty.

Of the 16 studies, 3 (18.8%) included meta-analysis. Three studies (18.8%) did not mention whether or not there were funding sources. Fourteen studies (87.5%) reported adherence to the PRISMA guidelines. Only 2 studies (12.5%) were registered with PROSPERO. From the included journals in this analysis, the average Scopus CiteScore was 4.6 with a range of 0.2-9.1. The average 2022 Clarivate Impact Factor was 2.1 with a range of 0.6-4.6. Nine (56.3%) of the studies had a conclusion, which was in favor of the experimental intervention. The included reviews contained a total of 206 primary studies. In regards to these primary studies, 3 (1.46%) were level 1 evidence, 8 (3.88%) were level 2 evidence, 50 (24.3%) were level 3 evidence, and 145 (70.4%) were level 4 evidence.

At least one form of spin was identified in the abstracts of 13 (81.3%) of the 16 studies. The median number of spin categories present in the studies was 1 (range 0-4). The most frequent type of spin was type 5 (“the conclusion claims the beneficial effect of the experimental treatment despite a high RoB in primary studies”), which appeared in 7 (43.8%) of the studies. The second most common spin type was spin type 9 (“conclusion claims the beneficial effect of the experimental treatment despite reporting bias”), which appeared in 6 (37.5%) studies. Spin types 1, 2, 6, 7, 8, and 10 did not appear in any of the reviewed studies. A full assessment of spin type frequencies is included in Table I.

There was was a statistically significant association between the presence of a conflict of interest with an author and the omission of a statement addressing the funding sources of the article (P value .0357). All 16 studies included a conflict of interest statement, and 6 (37.5%) of these had a documented conflict of interest. Thirteen studies (81.3%) included a statement addressing funding sources, and all 13 reported receiving no external funding. The 3 studies which did not disclose their funding sources all had conflicts of interest. These conflicts included examples of all of the following: authors who reported receiving grants, personal fees, royalties, and research fees from orthopedic device manufacturers, as well as authors who were investors, presenters, or consultants for orthopedic device manufacturers.

There were no statistically significant associations between any of the spin characteristics and adherence to PRISMA, adherence to PROSPERO, declaration of funding sources, Clarivate Impact Factor, Scopus CiteScore, and having a conclusion in favor of the intervention. There were no statistically significant associations between a study's adherence to AMSTAR requirements and adherence to PRISMA, adherence to PROSPERO, declaration of funding sources, Clarivate Impact Factor, Scopus CiteScore, and having a conclusion in favor of the experimental intervention.

Based on AMSTAR 2 assessment, 3 (18.8%) studies were rated as “moderate quality,” and 13 (81.3%) studies were rated as “low quality.” No studies met the criteria for “high quality” or “critically low quality.” A full 16-point AMSTAR 2 assessment is included in Table II.

Table II.

AMSTAR 2 assessment of articles involving glenoid augmentation.

AMSTAR 2 criteria	Yes	No
1. Did the research questions and inclusion criteria for the review include the elements of PICO?	16/16	0/16
2. Did the report of the review contain an explicit statement that the review methods were established before the conduct of the review, and did the report justify any significant deviations from the protocol?	3/16	13/16
3. Did the review authors explain their selection of the study designs for inclusion in the review?	12/16	4/16
4. Did the review authors use a comprehensive literature search strategy?	16/16	0/16
5. Did the review authors perform study selection in duplicate?	13/16	3/16
6. Did the review authors perform data extraction in duplicate?	12/16	4/16
7. Did the review authors provide a list of excluded studies and justify the exclusions?	0/16	16/16
8. Did the review authors describe the included studies in adequate detail?	15/16	1/16
9. Did the review authors use a satisfactory technique for assessing the RoB in individual studies that were included in the review?	11/16	5/16
10. Did the review authors report on the sources of funding for the studies included in the review?	1/16	15/16
11. If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results?∗	3/3	0/3
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?∗	1/3	2/3
13. Did the review authors account for RoB in primary studies when interpreting or discussing the results of the review?	11/16	5/16
14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?	11/16	5/16
15. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?∗	1/3	2/3
16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?	15/16	1/16

RoB, risk of bias; PICO, Population, Intervention, Control Group, Outcome.

^∗Only applicable to studies including meta-analysis.

Discussion

The principal findings of our study demonstrated that the majority (81.3%) of systematic reviews and meta-analyses on glenoid augmentation in TSA contained at least one form of spin in the abstract. The most frequent type of spin was type 5 (“the conclusion claims the beneficial effect of the experimental treatment despite a high RoB in primary studies”), which appeared in 7 (43.8%) of the studies. The second most common spin type was spin type 9 (“conclusion claims the beneficial effect of the experimental treatment despite reporting bias”), which appeared in 6 (37.5%) studies. These findings indicate that spin is common in the literature, and much of this bias results from limitations in the primary studies in addition to bias when reporting results in secondary studies.

The presence of spin in orthopedic research is not a novel concept and has been the focus of multiple other studies. A recently published spin analysis on biodegradable balloon spacers found that 93.1% of the 29 included studies had at least one type of spin.¹¹ Another analysis on studies evaluating treatments for Achilles tendon rupture reported that 65.1% of the 43 included studies had spin.⁶ Our review further contributed to these findings, showing 81.3% of studies involving glenoid augmentation have spin. However, it is also worth noting that not all areas of the orthopedic literature have such a high prevalence of spin. Reddy et al found that only 28.4% of the 176 total systematic reviews and meta-analyses involving platelet-rich plasma treatments had spin.³⁹ Although the amount of spin varies between studies, numerous analyses have revealed that bias is pervasive throughout the orthopedic literature. In the topic of glenoid augmentation, we found that spin is present in the majority of systematic reviews and meta-analyses.

The fact that type 5 spin (“the conclusion claims the beneficial effect of the experimental treatment despite a high RoB in primary studies”) was the most commonly occurring spin in our review is not surprising, as it has been found in many previous studies in other orthopedic literature, ranging from 23.1% to 65%.^1,3,20 A common weakness contributing to type 5 spin was drawing conclusions based on primary studies of low levels of evidence. Of the 206 primary studies, the majority (70.4%) were level 4 evidence. Studies with level 1 or 2 evidence only accounted for 5.30% of the total. Wilcox et al and Gates et al exclusively included studies of level 4 evidence.^14,51 Sheth et al concluded that “early-to mid-term outcomes following augmented a aTSA for posterior glenoid deficiency demonstrate good-to-excellent overall clinical results.”⁴⁶ However, the authors note later in their discussion that their review “is comprised of levels III and IV studies, which makes it difficult to make any definitive conclusions.” While the RoB was acknowledged in the discussion, it was not addressed in the abstract. A common underlying reason for the low levels of evidence seen in these reviews was the overreliance on retrospective studies. In the case of Velasquez Garcia et al, the number of clinical studies meeting the inclusion criteria was so limited that authors decided to include 2 cadaveric studies in their analysis.⁵⁰ In contrast, an example of appropriately avoiding spin type 5 can be seen in Shanmugaraj et al The high RoB in primary studies is directly addressed in their abstract conclusion, which stated that “the optimal treatment strategy remains elusive due to a lack of high-quality, comparative studies with long-term surveillance.”⁴⁴ Clearly, there is a need for more prospective studies with large sample sizes, such as randomized controlled trials, evaluating outcomes of glenoid augmentation. Until that level of evidence is available, authors of systematic reviews and meta analyses need to be careful when writing their abstract conclusions to avoid overrepresenting their results.

Another commonly occurring spin was type 9 (‘‘conclusion claims the beneficial effect of the experimental treatment despite reporting bias’’), which may mislead readers through the selective inclusion and omission of results in the abstract. Reporting bias results from the tendency to overreport or selectively publish positive results. One example of reporting bias can be seen in Heifner et al, where the abstract conluded that the reverse shoulder arthroplasty “provide[d] highly favorable results” but only reported the statistically significant improvement in Constant scores and omitted the lack of statistically significant improvements in Visual Analogue Scale, American Shoulder and Elbow Surgeons, Simple Shoulder Test and functional range of motion measurements.¹⁶ Conversely, Velasquez Garcia et al avoided reporting bias in their abstract by including both results favorable to the use of computer-assisted navigation (a decreased rate of deviation and inclination of the glenoid components) as well as results that were not in favor of the experimental intervention (increased surgical time).⁵⁰ Including both favorable and unfavorable results in the abstract is important to adequately represent all of the available results, as many clinicians look solely at the abstract when incorporating research into their clinical decision-making.²

Although there were no statistically significant associations between spin types and study characteristics, we did find a statistically significant association between the presence of a conflict of interest and the lack of a statement addressing funding sources (P value .0357). Financial interests are an important source of potential bias, so it is notable that the studies who reported these conflicts of interest were less likely to disclose funding sources. It is also worth remembering that our analysis only examined review articles; there may be further evidence of financial incentives in the primary studies contained in these reviews. Research on financial incentives in glenoid augmentation studies is limited; however, broader studies have indicated that such conflicts of interest are present in orthopedics. For instance, an analysis of 100 randomized controlled trials published in 5 leading orthopedic journals concluded that studies funded by industry were 10.9 times more likely to report favorable outcomes compared to those without industry funding.¹⁹ Thus, we recommend that journals strive to always include a funding statement in their published articles to ensure maximum transparency.

There are several limitations to our study. Notably, the relatively small sample size of 16 studies was a limitation of this review. While we performed searches on three different databases, our inclusion and exclusion criteria ruled out many articles. This resulted in low statistical power for our correlation analyses assessing for relationships between general characteristics and the presence of spin. This may be the reason we did not find many significant associations, so it is possible that more patterns would emerge when examining larger samples of studies. Another limitation was the subjectivity of identifying spin. Although Yavchitz et al's method allows for quantification of predefined spin types, different reviewers may disagree on the specific interpretation of these questions. AMSTAR analysis was affected by the same issue of subjectivity. We attempted to address this by having 2 reviewers (C.L. and B.B.) evaluate spin independently and a third author (A.F.) provide guidance in the case of disagreements.

Conclusion

Spin is highly prevalent in the abstracts of systematic reviews and meta-analyses studying glenoid augmentation with TSA. Misleading reporting is the most common category of spin.

Disclaimers

Funding: This project received no funding.

Conflicts of interest: Frank A Petrigliano reports a relationship with Exactech Inc. that includes consulting or advisory and a relationship with Stryker Orthopaedics that includes: consulting or advisory. Joseph N Liu reports a relationship with Stryker Orthopaedics that includes speaking and lecture fees, a relationship with Innocoll Biotherapeutics NA Inc. that includes travel reimbursement, a relationship with Arthroscopy Association of North America as a board or committee member, and a relationship with American Shoulder and Elbow Surgeons as a board or committee member. The other authors, their immediate families, and any research foundation with which they are affiliated have not received any financial payments or other benefits from any commercial entity related to the subject of this article. The authors have no relevant conflicts of interest to declare.