Abstract
Background
The present study aimed to evaluate the methodological quality and determine the quality of reporting of randomized controlled trials (RCTs) that assess surgical treatment for shoulder instability.
Methods
A Cochrane, Pubmed, EMBASE and Trip database search was performed, including the relevant literature, regarding RCTs that report on shoulder instability published between January 1994 and January 2017. Methodological quality was assessed with a modification of the Checklist to Evaluate A Report of a Nonpharmacologic Trial (CLEAR-NPT). Points were assigned based on 18 items regarding patient characteristics, randomization, care provider characteristics, surgical details and blinding, with a total score ranging from 0 points to 18 points. Missing items were verified with the corresponding authors of the studies. Quality of reporting corresponds to the total scores including the items that were additionally provided by the authors.
Results
We included 22 studies. Of these, nine corresponding authors provided additional information. The average methodological quality was 16.9 points (11 studies) and the average quality of reporting was 9.5 points (22 studies). Items scoring worst included information regarding the surgeon’s experience, the patients’ level of activity, comorbidities, analyzing according to ‘intention-to-treat’ principles, and blinding of care providers, participants and assessors.
Conclusions
RCTs reporting on shoulder instability surgery are well performed but poorly reported.
Keywords: glenohumeral, instability, quality, reporting, shoulder
Introduction
Randomized controlled trials (RCTs) provide the highest level of evidence.1 Although having a high level of evidence, these studies are susceptible to potential bias.2 For example, the absence of allocation concealment can lead to selection bias, whereas ascertainment bias occurs when the patients or outcome assessors are not blinded.
It is important that healthcare providers are aware of the study details, quality and potential biases, to enable adequate interpretation of the results on which they base their treatment strategies. Moreover, the quality of reporting can influence the overall quality of a scientific article dramatically (e.g. a high quality study that is reported poorly can give the impression that the study is conducted poorly). Therefore, checklists have been developed to help authors to evaluate the quality of reporting of their studies. One of those checklists is the Checklist to Evaluate A Report of a Nonpharmacologic Trial (CLEAR-NPT).3
Because none of the available checklists is specified for surgical interventions, they do not take into account the surgeon’s experience, the patient’s level of activity and characteristics, details of the surgical techniques, and the post-treatment protocol. These aspects, however, strongly relate to clinical outcome parameters.4–13
Previous studies have modified the CLEAR-NPT score so that they are able to assess orthopaedic papers.14,15 With regard to trials on surgical interventions in other joints, such as in the elbow, it has been demonstrated that the quality of RCTs is disappointing.14 To our knowledge, the general quality of RCTs on surgical interventions for shoulder instability is still unclear. Therefore, the present study aimed (i) to evaluate the overall quality of RCTs that report on surgical treatment for shoulder instability and (ii) to assess the quality of reporting of these studies.
Materials and Methods
A systematic review was conducted on RCTs that involve surgical interventions in shoulder instability treatment aiming to determine the overall quality of these studies and their quality of reporting. To assess the actual overall quality of the selected studies, a follow-up survey was conducted to reveal any information that may have been available but was not reported. The study was performed using the PRISMA guidelines.
Inclusion criteria and study identification
The inclusion and exclusion criteria are summarized in Table 1. The study had to be a RCT describing two or more (variations of) surgical interventions to treat anterior shoulder instability (e.g. open versus closed operation), published between 1 January 1994 and 16 January 2017 in the English or Dutch language. We have limited the search to 1994 because of studies resulting in a rise in procedures for shoulder instability.16,17 Animal, cadaveric, biomechanical and/or in vitro studies were excluded. Studies with any other designs, studies summarizing previously published results and studies assessing the effects of nonsurgical treatment were also excluded.
Table 1.
Inclusion criteria | Exclusion criteria |
---|---|
(Pseudo) randomized controlled trial | Study design other than (pseudo)randomized controlled trials |
Shoulder joint | Inclusion of other joints than the shoulder in the trial |
Shoulder instability | Shoulder pathology other than shoulder instability (e.g. stroke, rheumatoid arthritis, or after bone marrow transplantation) |
Invasive intervention | No invasive intervention (e.g. physiotherapy, slings/braces, etc.) |
Human | Animal or cadaveric |
Published between 1994 and 20 March 2016 | Published before 1994 |
Letters to the editor | |
Anesthesia techniques | |
In vitro studies | |
Pilot studies (i.e. preliminary studies. These are not hypothesis testing studies and therefore safety and efficacy are not evaluated)15 |
The Cochrane Library, PubMed, EMBASE and Trip Database were searched for relevant RCTs indexed between 1 January 1994 and 16 January 2017. A medical librarian was consulted to help construct an appropriate search strategy and to ensure that all randomized trials published during the study period were identified (Appendix 1).
Titles and abstracts were screened by two independent assessors (H.A. and A.S.). Any discrepancy was resolved by reading the full-text and discussion between the two reviewers.
Data abstraction
We used the CLEAR-NPT to determine the quality of the included studies because of its ability to evaluate the quality of reporting and the fact that it is a widely used scoring system in the literature. Moreover, it is a checklist designed specifically to be able to grade the methodological quality in a short time and is used in previous studies assessing the methodological quality of RCTs.14 Because the CLEAR-NPT was not specialized for surgical interventions, we added six modifications (Appendix 2). Sub-items of questions 6 and 7 of the original CLEAR-NPT (blinding of participants and care providers) were no longer optional but embedded in the list for each study. Moreover, questions were added regarding the surgeon’s experience, the patient’s activity level, any comorbidities, the details of the surgical techniques and the post-treatment protocol, resulting in a 18-item list. We considered surgeon’s experience appropriate if the surgeon had at least 5 years of experience in shoulder instability surgery. The intention-to-treat principle means that, in contrast to per-protocol analysis, all of the participants including drop-outs are analyzed in the arm they were originally randomized to, aiming to avoid overestimating the effect size of the treatments. The sub-items were not answered separately but were used to clarify the associated main items. To be able to rate the quality of the studies over the years, we have divided the studies in different time intervals (1990 to 2000, 2001 to 2005, 2006 to 2010 and 2011 to 2016) and assessed the methodological quality of the studies in these groups.
Each item of the list can be answered with ‘yes’, ‘no’, ‘not-applicable’ (N/A) or ‘not reported’ (NR). An item answered with ‘yes’ or ‘not-applicable’ scored 1 point (to be able to compare the studies at the same time as maintaining an 18-point scoring system) and ‘no’ or ‘not reported’ scored 0 points. The subsequent total score ranges from 0 points to 18 points.
The same assessors performed data abstraction. The discrepancies were resolved by a discussion.
Author’s survey
When items of the modified CLEAR-NPT were ‘not reported’ in the original publication, the authors were contacted to verify whether the information was not reported or actually missing. This additional author’s survey was performed to differentiate between the quality of the research performed and the quality of reporting in these articles. This regarded, for example, the surgeons’ experience (how many years?) and the patients’ level of activity. For each study, the total number of items answered with ‘yes’, ‘no’, ‘N/A’ or ‘NR’ For each item of the CLEAR-NPT, the quality of reporting was calculated by subtracting the percentage of studies in which the data was not collected from the percentage of studies in which the data was collected but not reported.
Statistical analysis
All data were summarized descriptively. Categorical variables are described with frequencies and percentages. The p-values for the difference between the studies before/after contacting the authors are calculated using the Mann–Whitney U-test and an unpaired t-test. The Kruskal–Wallis test was used to calculate significance between the different time points in which the studies are published and the results based on country of origin. All analyses were performed using SPSS, version 18.0 (IBM Corp., Armonk, NY, USA).
Results
Study selection
A total of 647 studies were screened for title and abstract. In total, 624 studies did not meet the inclusion criteria. From the remaining 23 full-text articles, two additional studies were excluded because these did not report on a surgical intervention. Cross-referencing resulted in one additional study. This resulted in a total of 22 RCTs that compare surgical interventions for shoulder instability.18–39 (Fig. 1) The studies originate from Sweden (n = 5),21,25,31,35,36 Denmark (n = 1),24 the USA (n = 3),19,32,38 Italy (n = 3),20,22,28 the UK (n = 2),34,37, Korea (n = 1),33 Canada (n = 3),23,27,29 Brazil (n = 2),18,30 Turkey (n = 1)26 and Iran (n = 1).39 The studies compare techniques regarding different sutures/anchors,25,28,30,31,36–38 open/closed stabilization techniques,18,19,22–24,26,29,32,33,35,39 arthroscopic Bankart repair versus joint lavage34 and variations of arthroscopic Bankart repairs.20,27 The study characteristics are summarized in Appendix 3.
Methodological quality per study
Based on information provided in the paper
Based on the provided data in the included 22 studies, the average score is 9.5 points, ranging from 4 points39 to 14 points,19,24,27,29 corresponding to the poor quality of the studies (Tables 2 and 3). Almost half of the items of the modified CLEAR-NPT are not addressed in the studies.
Table 2.
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | Total score | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Archetti Netto 15 | Yes | Yes | Yes | Yes | No | NR | Yes | N/A | No | No | NR | Yes | Yes | NR | NR | Yes | No | Yes | 10 |
Bottoni 16 | Yes | Yes | Yes | Yes | No | NR | Yes | N/A | No | Yes | NR | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 14 |
Castagna 17 | Yes | Yes | Yes | Yes | No | NR | Yes | NR | No | No | NR | NR | Yes | Yes | N/A | Yes | Yes | Yes | 11 |
Elmlund 18 | Yes | Yes | Yes | Yes | No | NR | Yes | NR | Yes | No | NR | Yes | Yes | Yes | N/A | Yes | NR | Yes | 12 |
Fabbriciani 19 | Yes | Yes | Yes | Yes | No | NR | Yes | NR | No | No | No | Yes | Yes | NR | NR | Yes | Yes | Yes | 10 |
Hiemstra 20 | No | NR | Yes | Yes | Yes | yes | Yes | NR | No | No | NR | Yes | Yes | Yes | N/A | Yes | Yes | No | 11 |
Jørgensen 21 | No | Yes | Yes | Yes | Yes | yes | Yes | NR | No | Yes | NR | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 14 |
Magnusson 22 | Yes | Yes | Yes | Yes | No | NR | No | NR | No | No | NR | Yes | No | Yes | N/A | Yes | NR | Yes | 9 |
Mahiroğulları 23 | NR | NR | No | Yes | No | NR | No | N/A | Yes | No | NR | Nr | No | NR | NR | Yes | NR | Yes | 5 |
McRae 24 | Yes | Yes | Yes | Yes | No | yes | Yes | yes | No | No | Yes | Yes | Yes | Yes | N/A | Yes | No | Yes | 14 |
Milano25 | Yes | Yes | Yes | Yes | No | NR | No | NR | Yes | Yes | NR | Yes | No | NR | NR | Yes | No | Yes | 9 |
Mohtadi 26 | Yes | Yes | Yes | Yes | Yes | yes | Yes | N/A | Yes | Yes | No | No | Yes | No | NR | Yes | Yes | Yes | 14 |
Monteiro 27 | Yes | Yes | Yes | Yes | No | NR | Yes | NR | Yes | No | NR | Yes | Yes | NR | NR | Yes | No | No | 9 |
Norlin28 | No | No | Yes | Yes | No | NR | No | NR | No | Yes | no | Yes | No | NR | NR | Yes | NR | Yes | 6 |
Owens 29 | Yes | Yes | Yes | Yes | No | NR | Yes | NR | No | No | NR | Yes | Yes | NR | NR | Yes | No | No | 8 |
Rhee 30 | Yes | Yes | Yes | Yes | No | NR | No | NR | No | No | NR | Yes | No | NR | NR | Yes | No | No | 6 |
Robinson 31 | Yes | Yes | Yes | Yes | No | NR | Yes | NR | Yes | Yes | NR | No | Yes | NR | NR | Yes | Yes | Yes | 11 |
Salomonsson32 | Yes | Yes | No | Yes | No | NR | Yes | NR | No | Yes | NR | Yes | Yes | NR | NR | Yes | Yes | Yes | 10 |
Sperber 33 | Yes | Yes | No | Yes | No | NR | No | NR | No | No | NR | Yes | No | NR | NR | Yes | NR | No | 5 |
Tan 34 | Yes | Yes | No | Yes | No | NR | Yes | NR | Yes | No | NR | Yes | Yes | NR | NR | Yes | No | Yes | 9 |
Warme 35 | Yes | Yes | Yes | Yes | No | NR | Yes | NR | No | No | NR | Yes | Yes | NR | NR | Yes | Yes | No | 9 |
Zarezade 36 | NR | NR | No | No | No | NR | Yes | NR | Yes | No | NR | NR | Yes | NR | NR | Yes | No | No | 4 |
The result of each study is shown regarding each item of the CLEAR-NPT scoring. The rows represent the items of the modified CLEAR-NPT as stated in Appendix 2.
Table 3.
After contacting the authors |
|||||||||
---|---|---|---|---|---|---|---|---|---|
Question | Yes (N) | No (N) | N/A (N) | NR (N) | Scored Yes or N/A (%) | Yes (N) | No (N) | N/A (N) | Scored Yes or N/A (%) |
1. Was the generation of allocation adequate? | 17 | 3 | 0 | 1 | 77 | 11 | 0 | 0 | 100 |
2. Was the treatment allocation concealed? | 18 | 1 | 0 | 2 | 82 | 11 | 0 | 0 | 100 |
3. Were details of the intervention administered to each group presented? | 17 | 4 | 0 | 0 | 77 | 11 | 0 | 0 | 100 |
4. Were details of the rehabilitation/follow-up process presented? | 21 | 0 | 0 | 0 | 95 | 11 | 0 | 0 | 100 |
5. Was the experience or skill of the care providers in each arm presented? | 3 | 18 | 0 | 0 | 14 | 10 | 1 | 0 | 91 |
6. Was the experience or skill of the care providers in each arm appropriate? | 4 | 0 | 0 | 17 | 18 | 11 | 0 | 0 | 100 |
7. Was participant adherence assessed quantitatively? | 15 | 6 | 0 | 0 | 68 | 10 | 1 | 0 | 91 |
8. Were participants adequately blinded? | 1 | 0 | 4 | 16 | 23 | 7 | 0 | 4 | 100 |
9. Was the level of activity of the patients adequately presented? | 7 | 14 | 0 | 0 | 32 | 9 | 2 | 0 | 82 |
10 Were details of the comorbidity/associated lesions presented? | 7 | 14 | 0 | 0 | 32 | 10 | 1 | 0 | 91 |
11. Were care providers or persons caring for the participants adequately blinded? | 1 | 3 | 0 | 17 | 5 | 6 | 4 | 1 | 64 |
12. Were all other aspects of treatment and care identical for each arm? | 17 | 2 | 0 | 2 | 77 | 11 | 0 | 0 | 100 |
13. Were numbers of patients who were screened but found to be not eligible mentioned? | 15 | 6 | 0 | 0 | 68 | 10 | 1 | 0 | 91 |
14. Were outcome assessors adequately blinded to assess the primary outcomes? | 7 | 1 | 0 | 13 | 32 | 10 | 1 | 0 | 91 |
15. If outcome assessors were not adequately blinded, were specific methods used to avoid ascertainment bias? | 0 | 0 | 7 | 14 | 32 | 1 | 0 | 10 | 100 |
16. Was the follow-up schedule the same in each group? | 21 | 0 | 0 | 0 | 95 | 11 | 0 | 0 | 100 |
17. Were the main outcomes analyzed according to the intention-to-treat principle? | 9 | 7 | 0 | 5 | 41 | 11 | 0 | 0 | 100 |
18. Was the amount of complications presented? | 15 | 6 | 0 | 0 | 68 | 11 | 0 | 0 | 100 |
We found no difference in quality over the different time intervals (p = 0.544) and per country of origin (p = 0.178). The average scores for time intervals between 1990 to 2000,24,31,38 2001 to 2005,22,36 2006 to 201019–21,23,25,26,28,30,33–35,37 and 2011 to 201618,27,29,32,39 is 9.7, 7.5, 9.7 and 10, respectively.
There is a difference in the quality between studies that are published in different journals (p < 0.001). The average scores on the CLEAR-NPT for the studies of the journals Acta Orthopaedica; Advanced Biomedical Research; The American Journal of Sports Medicine; Arthroscopy; The Journal of Bone & Joint Surgery;29,34 Journal of Shoulder and Elbow Surgery; Knee Surgery, Sports Traumatology, Arthroscopy; and Orthopaedic Journal of Sports Medicine are 7.5, 4, 10.4, 9.4, 12.5, 5.5, 12 and 8, respectively. When analyzing the difference in journal impact factor and CLEAR-NPT score using the Spearman correlation test, the CLEAR-NPT score was not correlated with journal impact factor (ρ = 0.6826, p = 0.0621).
Based on information provided by the authors (reporting quality)
All 22 authors were contacted and 11 authors responded. One author could not retrieve the specifications of his study because his study was performed more than 20 years ago.31 The other 11 authors could provide the missing information regarding items that were answered with NR/no. Tables 3 and 4 provide a sub-analysis of these 11 studies. It can be seen that the highest discrepancy occurs in items regarding the experience of the surgeon, blinding of the participants, level of activity of the patients, details about the comorbidity and blinding of care providers.
Table 4.
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | Total score | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Archetti Netto15 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | N/A | Yes | Yes | Yes | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 18 |
Bottoni et al.16 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | N/A | Yes | Yes | N/A | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 18 |
Castagna Poehling and Freehill17 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 18 |
Elmlund et al.18 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 18 |
Hiemstra et al.20 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | N/A | No | Yes | No | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 16 |
Magnusson et al.22 | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | No | Yes | N/A | Yes | Yes | Yes | 16 |
McRae et al.24 | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 17 |
Monteiro et al.27 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 17 |
Mohtadi et al.26 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | N/A | No | Yes | No | Yes | Yes | No | NR | Yes | Yes | Yes | 14 |
Salomonsson et al.32 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | yes | Yes | Yes | Yes | 17 |
Warme et al.35 | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes | Yes | N/A | Yes | Yes | Yes | 17 |
The result of each study is shown regarding each item of the CLEAR-NPT scoring. The rows represent the items of the modified CLEAR-NPT as stated in Appendix 2.
The CLEAR-NPT scores improved to good after the remaining information provided by the authors was added; the average CLEAR-NPT score improved to 16.9 points (94.8% of items adequately answered) compared to 11.2 points (with 62.2% of the items adequately answered) per study based on the information provided in their papers. The average difference is 5.7 points for each study (an improvement of 31.6%; p < 0.001).
Moreover, 11 of the 18 items were adequately reported by 100% of the corresponding authors; the worst reported item was adequately reported by 64% of the authors and regarded the blinding of the care providers (Fig. 2 and Table 3).
When analyzing the difference between the studies in which the authors have responded and the studies in which the authors have not responded, the first ones had a higher mean CLEAR-NPT score (11.2 points versus 7.9 points; p < 0.012) based on scoring the data in the manuscript.
Methodological quality per CLEAR-NPT item
Based on information provided in the paper
We assessed how well the items are addressed by counting the number of studies that answered a specific item with ‘yes’ or ‘not applicable’ (i.e. adequately assessed). Nine of the items are inadequately assessed by more than 50% of the authors (items 5, 6, 8, 9, 10, 11, 14, 15 and 17), three adequately assessed by 50% to 75% of the authors (items 7, 13 and 18) and six by 75% to 100% of the authors (items 1, 2, 3, 4, 12 and 16) (Fig. 3 and Table 3). The items that were least adequately assessed are the experience or skill of the care provider (adequately assessed in 14%; i.e. three studies), blinding of participants (adequately assessed in 23%; i.e. five studies), the level of activity of the patients (adequately assessed in 32%; i.e. seven studies), the associated comorbidity (adequately assessed in 32%; i.e. seven studies), blinding of outcome assessors, (adequately scored in 32%; i.e. seven studies) blinding of care providers (adequately assessed in 5%; i.e. one study) and analyzing the results according to the intention-to-treat principles (adequately assessed in 41%; i.e. nine studies). The best-scored items consisted of the rehabilitation (adequately assessed in 95%; i.e. 21 studies), the follow-up schedule (adequately assessed in 95%; i.e. 21 studies) and concealment of treatment allocation (82%; i.e. 18 studies).
Based on information provided by the authors (reporting quality)
Based on the information provided in the papers from the above-mentioned 11 studies, an average of 6.8 (61.8%) studies reported on each item adequately. Based on the authors’ information, on average, 10.4 (94.5%) of the studies reported adequately (32.7% improvement; p < 0.001).
Discussion
The findings of the present study show that the RCTs regarding shoulder instability surgery have a good quality (average score of the modified CLEAR-NPT of 16.9 points with a maximum of 18 points), whereas the quality of reporting is low (average score of the modified CLEAR-NPT of 9.5 points with a maximum of 14 points). This means that the RCTs have a good study design but that important study methodology issues are poorly reported, which makes the quality of reporting low. Important items that have been reported poorest concern the care providers’ experience and their blinding. There is no difference in quality of reporting through the years.
There are several explanations for the poor quality of reporting. First, surgeons could be focused on the results and forget to pay attention to or report other details (for example blinding of care providers). Another possible explanation is that the current quality assessment tools (checklists such as the CLEAR-NPT, etcetera) are not specific for surgical trials. Important factors that are specific for surgical studies, including the surgeons’ experience and the patients’ level of activity, are insufficiently addressed. These factors are however also not assessed in the other checklist, such as the CONSORT checklist40–42 or the Rob 2.0 tool.43
Our findings are consistent with previous studies in other surgical fields.14,44–46 These studies also found under-reporting of the ‘experience or skill of the care provider’, ‘blinding of participants’, ‘blinding of outcome assessors’, ‘blinding of care providers’ and ‘analyzing the results according to intention to the intention-to-treat principles’. From these studies Somford et al.,14 van Oldenrijk et al.,15 and Jacquier et al.,44 have used the CLEAR-NPT scoring system; Balasubramanian et al.,45 used the CONSORT checklist; and Poolman et al.46 assessed outcome reporting by identifying outcome measures, the quality of reporting by analysing the methodological safeguard of blinding and the effect of blinding on the reported treatment effects. Somford et al.14 and van Oldenrijk et al.15 have modified the CLEAR-NPT studies to be able to rate the quality of elbow and spine surgery studies, respectively. We have chosen the CLEAR-NPT over the CONSORT because the CLEAR-NPT has been modified previously to rate the methodological quality of surgical trials, such as spine and elbow surgery.
Nevertheless, our results have to be interpreted with caution because only 11 out of the 22 authors have responded to the e-mails requesting additional information regarding their study (overall response rate of 50%). This could lead to bias when authors who have performed a better study could be more inclined to provide additional information, leading to an overestimation of the quality of the studies in general. Although it is impossible to determine whether the studies for which the authors did not answer our questions had a bad study design, these studies had a lower CLEAR-NPT score based on the published paper. Moreover, it is almost impossible to adequately blind the surgeons performing the operations (i.e. care providers).
Notwithstanding these limitations, the results of the present study are highly important, because they address the poor quality of reporting of the current RCTs regarding shoulder stability surgery, which could lead to misinterpretation of the results as a result of missing information. We hope to highlight the importance of good quality reporting. Further research could include the validation of quality reporting measurement scales specifically designed for surgical trials
Conclusions
The present study shows that the RCTs reporting on shoulder instability surgery are well performed but poorly reported. The items that were scored the worst consisted of the experience or skill of the care provider, blinding of participants, the level of activity of the patients, the associated comorbidity, blinding of outcome assessors, blinding of care providers and analyzing the results according to intention to intention-to-treat principles.
Supplementary Material
Acknowledgements
HA participated in data collection, design of the study, editing of the manuscript and drafting the manuscript. AS participated in data collection. JA, NW, DD and MB participated in the design of the study and editing the manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Ethical Review and Patient Consent
Not applicable to this article.
Level of evidence
Level I: Systematic review of randomized controlled trials
REFERENCES
- 1.Poojary SA, Bagadia JD. Reviewing literature for research: doing it the right way. Indian J Sex Transm Dis 2014; 35: 85–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Viera AJ, Bangdiwala SI. Eliminating bias in randomized controlled trials: importance of allocation concealment and masking. Fam Med 2007; 39: 132–137. [PubMed] [Google Scholar]
- 3.Boutron I, Moher D, Tugwell P, et al. A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. J Clin Epidemiol 2005; 58: 1233–1240. [DOI] [PubMed] [Google Scholar]
- 4.Hammond JW, Queale WS, Kim TK, et al. Surgeon experience and clinical and economic outcomes for shoulder arthroplasty. J Bone Joint Surg Am 2003; 85A: 2318–2324. [DOI] [PubMed] [Google Scholar]
- 5.Hoppe DJ, de Sa D, Simunovic N, et al. The learning curve for hip arthroscopy: a systematic review. Arthroscopy 2014; 30: 389–397. [DOI] [PubMed] [Google Scholar]
- 6.Jain N, Pietrobon R, Hocker S, et al. The relationship between surgeon and hospital volume and outcomes for shoulder arthroplasty. J Bone Joint Surg Am 2004; 86A: 496–505. [DOI] [PubMed] [Google Scholar]
- 7.Nunley RM, Zhu J, Brooks PJ, et al. The learning curve for adopting hip resurfacing among hip specialists. Clin Orthop Relat Res 2010; 468: 382–391. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Park Y, Lee S, Bin, Seok SO, et al. Perioperative surgical complications and learning curve associated with minimally invasive transforaminal lumbar interbody fusion: a single-institute experience. CiOS Clin Orthop Surg 2015; 7: 91–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Spaans AJ, van den Hout JAAM, Bolder SBT. High complication rate in the early experience of minimally invasive total hip arthroplasty by the direct anterior approach. Acta Orthop 2012; 83: 342–346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.O’Neill DB. Arthroscopic Bankart repair of anterior detachments of the glenoid labrum. A prospective study. J Bone Joint Surg Am 1999; 81: 1357–1366. [PubMed] [Google Scholar]
- 11.Cho NS, Hwang JC, Rhee YG. Arthroscopic stabilization in anterior shoulder instability: collision athletes versus noncollision athletes. Arthrosc – J Arthrosc Relat Surg 2006; 22: 947–953. [DOI] [PubMed] [Google Scholar]
- 12.Hayashida K, Yoneda M, Nakagawa S, et al. Arthroscopic Bankart suture repair for traumatic anterior shoulder instability: Analysis of the causes of a recurrence. Arthroscopy 1998; 14: 295–301. [DOI] [PubMed] [Google Scholar]
- 13.Kim SH, Ha KI, Jung MW, et al. Accelerated rehabilitation after arthroscopic Bankart repair for selected cases: a prospective randomized clinical study. Arthrosc – J Arthrosc Relat Surg 2003; 19: 722–731. [DOI] [PubMed] [Google Scholar]
- 14.Somford MP, Deurzen DFP van, Ostendorf M, et al. Quality of research and quality of reporting in elbow surgery trials. J Shoulder Elb Surg 2015; 24: 1619–1626. [DOI] [PubMed] [Google Scholar]
- 15.van Oldenrijk J, van Berkel Y, Kerkhoffs GMMJ, et al. Do authors report surgical expertise in open spine surgery related randomized controlled trials? A systematic review on quality of reporting. Spine (Phila Pa 1976) 2013; 38: 857–864. [DOI] [PubMed] [Google Scholar]
- 16.Arciero RA, Wheeler JH, Ryan JB, et al. Arthroscopic Bankart repair versus nonoperative treatment for acute, initial anterior shoulder dislocations. Am J Sports Med 1994; 22: 589–594. [DOI] [PubMed] [Google Scholar]
- 17.Luo TD, Poehling GG, Freehill MT. Review of Arciero’s article (1994) on arthroscopic Bankart repair versus non-operative treatment for acute, initial anterior shoulder dislocations: does the same hold true in 2016? J ISAKOS Jt Disord Orthop Sport Med 2016; 1: 358–364. [Google Scholar]
- 18.Netto NA, Tamaoki MJS, Lenza M, et al. Treatment of Bankart lesions in traumatic anterior instability of the shoulder: a randomized controlled trial comparing arthroscopy and open techniques. Arthrosc – J Arthrosc Relat Surg 2012; 28: 900–908. [DOI] [PubMed] [Google Scholar]
- 19.Bottoni CR, Smith EL, Berkowitz MJ, et al. Arthroscopic versus open shoulder stabilization for recurrent anterior instability: a prospective randomized clinical trial. Am J Sports Med 2006; 34: 1730–1737. [DOI] [PubMed] [Google Scholar]
- 20.Castagna A, Borroni M, Delle Rose G, et al. Effects of posterior-inferior capsular plications in range of motion in arthroscopic anterior Bankart repair: a prospective randomized clinical study. Knee Surg Sports Traumatol Arthrosc 2009; 17: 188–194. [DOI] [PubMed] [Google Scholar]
- 21.Elmlund AO, Kartus J, Rostgård-Christensen L, et al. A 7-year prospective, randomized, clinical, and radiographic study after arthroscopic Bankart reconstruction using 2 different types of absorbable tack. Am J Sports Med 2009; 37: 1930–1937. [DOI] [PubMed] [Google Scholar]
- 22.Fabbriciani C, Milano G, Demontis A, et al. Arthroscopic versus open treatment of bankart lesion of the shoulder: a prospective randomized study. Arthrosc – J Arthrosc Relat Surg 2004; 20: 456–462. [DOI] [PubMed] [Google Scholar]
- 23.Hiemstra LA, Sasyniuk TM, Mohtadi NGH, et al. Shoulder strength after open versus arthroscopic stabilization. Am J Sports Med 2008; 36: 861–867. [DOI] [PubMed] [Google Scholar]
- 24.Jorgensen U, Svend-Hansen H, Bak K, et al. Recurrent post-traumatic anterior shoulder dislocation – open versus arthroscopic repair. Knee Surg Sports Traumatol Arthrosc 1999; 7: 118–124. [DOI] [PubMed] [Google Scholar]
- 25.Magnusson L, Ejerhed L, Rostgard-Christensen L, et al. A prospective, randomized, clinical and radiographic study after arthroscopic Bankart reconstruction using 2 different types of absorbable tacks. Arthroscopy 2006; 22: 143–151. [DOI] [PubMed] [Google Scholar]
- 26.Mahiroğulları M, Ozkan H, Akyüz M, et al. Comparison between the results of open and arthroscopic repair of isolated traumatic anterior instability of the shoulder. Acta Orthop Traumatol Turc 2010; 44: 180–185. [DOI] [PubMed] [Google Scholar]
- 27.McRae S, Leiter J, Subramanian K, et al. Randomized controlled trial of arthroscopic electrothermal capsulorrhaphy with Bankart repair and isolated arthroscopic Bankart repair. Knee Surg Sports Traumatol Arthrosc 2015; 24: 414–421. [DOI] [PubMed] [Google Scholar]
- 28.Milano G, Grasso A, Santagada DA, et al. Comparison between metal and biodegradable suture anchors in the arthroscopic treatment of traumatic anterior shoulder instability: a prospective randomized study. Knee Surg Sports Traumatol Arthrosc 2010; 18: 1785–1791. [DOI] [PubMed] [Google Scholar]
- 29.Mohtadi NGH, Chan DS, Hollinshead RM, et al. A randomized clinical trial comparing open and arthroscopic stabilization for recurrent traumatic anterior shoulder instability: two-year follow-up with disease-specific quality-of-life outcomes. J Bone Joint Surg Am 2014; 96: 353–360. [DOI] [PubMed] [Google Scholar]
- 30.Monteiro GC, Ejnisman B, Andreoli CV, et al. Absorbable versus nonabsorbable sutures for the arthroscopic treatment of anterior shoulder instability in athletes: a prospective randomized study. Arthroscopy 2008; 24: 697–703. [DOI] [PubMed] [Google Scholar]
- 31.Norlin R, Bankart A, Bankart A, et al. Use of Mitek anchoring for Bankart repair: a comparative, randomized, prospective study with traditional bone sutures. J Shoulder Elbow Surg 1994; 3: 381–385. [DOI] [PubMed] [Google Scholar]
- 32.Owens BD, Cameron KL, Peck KY, et al. Arthroscopic versus open stabilization for anterior shoulder subluxations. Orthop J Sport Med 2015; 3: 1–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Rhee YG, Lim CT, Cho NS. Muscle strength after anterior shoulder stabilization: arthroscopic versus open Bankart repair. Am J Sports Med 2007; 35: 1859–1864. [DOI] [PubMed] [Google Scholar]
- 34.Robinson CM. Primary arthroscopic stabilization for a first-time anterior dislocation of the shoulder. A randomized, double-blind trial. J Bone Jt Surg 2008; 90: 708–708. [DOI] [PubMed] [Google Scholar]
- 35.Salomonsson B, Abbaszadegan H, Revay S, et al. The Bankart repair versus the Putti-Platt procedure: a randomized study with WOSI score at 10-year follow-up in 62 patients. Acta Orthop 2009; 80: 351–356. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Sperber A, Hamberg P, Karlsson J, et al. Comparison of an arthroscopic and an open procedure for posttraumatic instability of the shoulder: a prospective, randomized multicenter study. J Shoulder Elb Surg 2001; 10: 105–108. [DOI] [PubMed] [Google Scholar]
- 37.Tan CK, Guisasola I, Machani B, et al. Arthroscopic stabilization of the shoulder: a prospective randomized study of absorbable versus nonabsorbable suture anchors. Arthrosc – J Arthrosc Relat Surg 2006; 22: 716–720. [DOI] [PubMed] [Google Scholar]
- 38.Warme WJ, Arciero RA, Savoie FH, III, et al. Nonabsorbable versus absorbable suture anchors for open Bankart repair. A prospective, randomized comparison. Am J Sports Med 1999; 27: 742–746. [DOI] [PubMed] [Google Scholar]
- 39.Zarezade A, Dehghani M, Rozati AR, et al. Comparison of Bristow procedure and Bankart arthroscopic method as the treatment of recurrent shoulder instability. Adv Biomed Res 2014; 12: 256–256. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Altman DG. Better reporting of randomised controlled trials: the CONSORT statement. Bmj 1996; 313: 570–571. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001; 357: 1191–1194. [PubMed] [Google Scholar]
- 42.Plint AC, Moher D, Morrison A, et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 2006; 185: 263–267. [DOI] [PubMed] [Google Scholar]
- 43.Higgins JPT, Sterne JA., Savović J, et al. A revised tool for assessing risk of bias in randomized trials. Cochrane Database Syst Rev 2016; 10: CD201601–CD201601. [Google Scholar]
- 44.Jacquier I, Boutron I, Moher D, et al. The reporting of randomized clinical trials using a surgical intervention is in need of immediate improvement: a systematic review. Ann Surg 2006; 244: 677–683. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Balasubramanian SP, Wiener M, Alshameeri Z, et al. Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg 2006; 244: 663–667. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46.Poolman RW, Struijs PAA, Krips R, et al. Reporting of outcomes in orthopaedic randomized trials: does blinding of outcome assessors matter? J Bone Joint Surg Am 2007; 89: 550–558. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.