Abstract
Background:
Statistical analysis of systematic reviews allows the results of previous studies to be combined and synthesized to assess the overall health effect of the intervention in question. Systematic reviews can also be used to guide the creation of clinical practice guidelines and are considered to have a high level of evidence. Thus, it is important that their methodological quality is of the highest standard. Publication bias presents 2 problems: (1) studies with significant results may be overrepresented in systematic reviews and meta-analyses (“false positives”) and (2) studies without significant results may not be included in systematic reviews and meta-analyses (“false negatives”) because each study, on its own, was underpowered, meaning that some treatment options that may have clinical benefit will not be adopted.
Methods:
We performed a study to evaluate the techniques used by authors to report and evaluate publication bias in the top 10 orthopaedic journals as well as 3 orthopaedic-related Cochrane groups. Two authors independently screened the titles and abstracts to identify systematic reviews and meta-analyses. We assessed publication bias in the systematic reviews that did not assess publication bias themselves.
Results:
Our final sample included 694 systematic reviews or meta-analyses that met our inclusion criteria. Our review included 502 studies (72%) that focused on clinical outcomes, with the majority of the remaining studies focused on predictive and prognostic accuracy (20%) or diagnostic accuracy (5%). Publication bias was discussed in 295 (42.5%) of the included studies and was assessed in 135 (19.5%). Of the studies that assessed publication bias, 31.9% demonstrated evidence of publication bias. Only 43% and 22% of studies that involved use of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines discussed and assessed publication bias, respectively.
Conclusions:
Publication bias is infrequently discussed and assessed in the high-impact orthopaedic literature. Furthermore, nearly one-third of the studies that assessed for publication bias demonstrated evidence of publication bias. In addition to these shortcomings, fewer than half of these studies involved use of the PRISMA guidelines and yet only one-fourth of the studies assessed for publication bias.
Clinical Relevance:
By understanding the degree to which publication bias is discussed and presented in high-impact orthopaedic literature, changes can be made by journals and researchers alike to improve the overall quality of research produced and reported.
The quality of a systematic review and meta-analysis is directly proportional to its methodological quality and the quality of the individual studies it includes1,2. A potential shortcoming in systematic reviews and meta-analyses is the overrepresentation of studies with significant results. This overrepresentation is a remnant of publication bias in clinical trials. Publication bias results from the non-publication of studies that do not demonstrate significant results or studies that demonstrate negative results. However, even in the most methodologically sound systematic review, low levels of evidence and underpowered studies in the literature can still lead to distorted conclusions3 because of the overrepresentation of studies with significant results (“false positives”). Furthermore, treatment options that may have clinical benefit will not be adopted because studies without statistical significance may not be included in systematic reviews and meta-analyses because each study on its own was underpowered (“false negatives”). It has been demonstrated that, despite a large increase in the quality and quantity of orthopaedic studies over time, the levels of evidence still remain low4.
Evaluation of publication bias within systematic reviews is mentioned in many pertinent reporting guidelines5-7. For instance, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines clearly state that authors should “specify any assessment of risk of bias that may affect the cumulative evidence (e.g., publication bias, selective reporting within studies)”7. Adherence to the PRISMA guidelines is recommended by many of the journals included in the present study8. Despite this recommendation, it has been found that publication bias is underreported in systematic reviews published in high-impact journals and that there are high levels of publication bias in those journals9. Studies have shown that, in systematic reviews published in anesthesiology and dermatology journals, assessment for publication bias is rare10,11. The consequences of not evaluating for publication bias can lead to treatments based on only part of the available evidence and can potentially lead to serious clinical consequences.
A powerful example of the ramifications of unchecked publication bias is the use of reboxetine for the treatment of depression. As a result of the positive findings of initial research, mainly sponsored by the pharmaceutical industry, reboxetine was approved for the acute treatment of major depression in the United Kingdom and parts of Europe. However, a systematic review that included more unpublished data regarding the safety and efficacy of the drug found that it was ineffective and in fact was harmful as an antidepressant12. This is a striking example of the potential harm that publication bias can cause and illustrates the urgent need for more transparency and accountability in published research.
Given the prevalence of publication bias in other medical specialties in addition to findings of publication bias in orthopaedic and surgical research13, the primary objective of the present study was to assess the method by which publication bias is evaluated within high-ranking orthopaedic journals. In instances in which publication bias had been evaluated, our secondary objective was to perform post-hoc publication bias assessments of the systematic reviews that met certain inclusion criteria.
Materials and Methods
We performed a study to evaluate the techniques used by authors to report and evaluate publication bias in the top 10 orthopaedic journals as well as 3 orthopaedic-related Cochrane groups: (1) the Bone, Joint and Muscle Trauma group, (2) the Back and Neck group, and (3) the Musculoskeletal group. The top 10 orthopaedic journals, which were identified by consulting Google Scholar’s H5-Index scores for “Orthopedic Medicine & Surgery”, included The American Journal of Sports Medicine; The Journal of Bone & Joint Surgery; Spine; Clinical Orthopaedics and Related Research; Arthroscopy: The Journal of Arthroscopic and Related Surgery; Knee Surgery, Sports Traumatology, Arthroscopy; The Journal of Arthroplasty; European Spine Journal; Journal of Shoulder and Elbow Surgery; and International Orthopaedics. The present study includes only previously published research and therefore is not subject to institutional review board oversight. We applied Statistical Analyses and Methods in the Published Literature (SAMPL) guidelines for reporting descriptive statistics when necessary.
Search Strategy
On August 11, 2017, we conducted a search of PubMed for systematic reviews and meta-analyses that had been published between 2013 and 2016. The search string is available in the Appendix.
We located relevant Cochrane reviews from the selected groups by accessing the associated Cochrane Library web page for the group and manually extracting references for those reviews that met our inclusion criteria.
Using Rayyan (Qatar Computing Research Institute), a tool for optimizing work flow of systematic reviews, 2 authors independently screened the titles and abstracts to identify systematic reviews and meta-analyses. We determined an article to be a systematic review if the methods involved summarizing evidence across multiple studies that had been identified through a comprehensive, well-described literature search. Meta-analyses were defined as any quantitative synthesis of data from multiple studies. We did not include any primary studies (i.e., randomized controlled trials, cohort studies, case-control studies), reviews of other reviews, letters to the editor, or commentaries in our review. We used Paperpile software to retrieve and access full texts of the studies meeting our inclusion criteria.
Data Extraction and Analysis
Using a pilot-tested Google Form, we extracted elements from each systematic review and/or meta-analysis. We stored the results in Microsoft Excel. Data were analyzed with use of Microsoft Excel and Comprehensive Meta-Analysis software (Biostat).
Assessing for Publication Bias
We assessed publication bias in the systematic reviews from our sample that did not assess publication bias themselves and that met the following eligibility criteria: (1) the systematic review had to include a meta-analysis, (2) the meta-analysis had to include ≥10 primary studies, and (3) sufficient data from the primary studies (e.g., odds ratios, risk ratios, mean differences, or other effect size measurements with confidence intervals) had to be included in the systematic review. Our minimum required size of 10 primary studies was based on previous research in The BMJ14 indicating that statistical power is too low to distinguish chance from actual asymmetry in <10 studies. In the event that a systematic review had >1 meta-analysis that met our inclusion criteria, we assessed publication bias of the meta-analysis with the most primary studies. When necessary in order to replicate the analyses and perform publication bias assessments, we made minor adjustments to the upper confidence interval until symmetry was reached. We used funnel plots, the Egger regression test, and the Duval and Tweedie trim-and-fill method to assess publication bias in those meta-analysis that met inclusion criteria. We set p < 0.05 as the significance level for the Egger regression test. All publication bias assessments were performed with use of Comprehensive Meta-Analysis software.
Search Results
Our PubMed and Cochrane Library searches returned 848 and 74 studies, respectively (Fig. 1). We uploaded our search results to Rayyan and screened the returned studies by title and abstract, yielding 717 studies. The retained studies were uploaded to Paperpile to assist in full-text retrieval and data extraction. Our final sample included 694 systematic reviews or meta-analyses that met our inclusion criteria.
Fig. 1.
PRISMA flowchart demonstrating the study acquisition process. MA = meta-analysis, and PB = publication bias.
Results
Characteristics of Included Studies
Our review includes 502 studies (72%) that focused on clinical outcomes, with the majority of the remaining studies focused on predictive and prognostic accuracy (20%) or diagnostic accuracy (5%). The majority of studies included >1 study type, with 56% of the systematic reviews including randomized controlled trials. Studies from 32 countries, most frequently the United States (206 studies; 30%) and China (138 studies; 20%) were included. Fifty percent (346) of the included studies mentioned the use of the PRISMA reporting guidelines. Grey literature searches (171 studies; 24.6%) and bibliography or hand searches (559 studies; 80.5%) were common methods used to reduce publication bias.
Publication Bias Reporting
Publication bias was discussed in 295 (42.5%) of the included studies and was assessed in 135 (19.5%) (Table I). The studies by the 3 Cochrane groups discussed publication bias 90% of the time, whereas the studies in Arthroscopy discussed publication bias only 18.8% of the time. Publication bias was assessed most often in The Journal of Arthroplasty (36.4%) and least often in Arthroscopy (6.0%) (Table II). Construction and visual analysis of funnel plots was the most common method for assessing publication bias (107 studies; 79.3%). Other methods for assessing publication bias included the Begg rank correlation test, Egger regression, Duval and Tweedie trim-and-fill test, and Rosenthal failsafe-N test. Of the studies that assessed publication bias, 31.9% reported evidence of publication bias and another 3.7% did not specify or were inconclusive as to whether publication bias was present. Only 43% and 22% of studies that were performed with use of the PRISMA guidelines discussed and assessed publication bias, respectively.
TABLE I.
Publication Bias Variables and Counts
| No. of Studies | |||
| Variable | Yes | No | Unspecified |
| Publication bias discussed (n = 694) | 295 (42.5%) | 399 (57.5%) | — |
| Publication bias evaluated (n = 694) | 135 (19.5%) | 559 (80.5%) | — |
| Funnel plots presented (n = 135) | 107 (79.3%) | 28 (20.7%) | — |
| Publication bias present (n =135) | 43 (31.9%) | 87 (64.4%) | 5 (3.7%) |
| Grey literature search (n = 694) | 171 (24.6%) | 523 (75.4%) | — |
| Bibliography or hand search (n = 694) | 559 (80.5%) | 133 (19.2%) | 2 (0.3%) |
TABLE II.
Publication Bias Results by Journal
| No. of Studies | ||
| Journal | Yes | No |
| Publication bias discussed | ||
| The American Journal of Sports Medicine (n = 72) | 16 (22.2%) | 56 (77.8%) |
| The Journal of Bone & Joint Surgery (n = 17) | 8 (47.1%) | 9 (52.9%) |
| Spine (n = 49) | 25 (51.0%) | 24 (49.0%) |
| Clinical Orthopaedics and Related Research (n = 56) | 26 (46.4%) | 30 (53.6%) |
| Arthroscopy: The Journal of Arthroscopic and Related Surgery (n = 117) | 22 (18.8%) | 95 (81.2%) |
| Knee Surgery, Sports Traumatology, Arthroscopy (n = 114) | 35 (30.7%) | 79 (69.3%) |
| The Journal of Arthroplasty (n = 55) | 27 (49.1%) | 28 (50.9%) |
| European Spine Journal (n = 82) | 47 (57.3%) | 35 (42.7%) |
| Journal of Shoulder and Elbow Surgery (n= 18) | 8 (44.4%) | 10 (55.6%) |
| International Orthopaedics (n = 44) | 18 (40.9%) | 26 (59.1%) |
| Cochrane groups (n = 70) | 63 (90.0%) | 7 (10.0%) |
| Publication bias assessed | ||
| The American Journal of Sports Medicine (n = 72) | 12 (16.7%) | 60 (83.3%) |
| The Journal of Bone & Joint Surgery (n = 17) | 5 (29.4%) | 12 (70.6%) |
| Spine (n = 49) | 17 (34.7%) | 32 (65.3%) |
| Clinical Orthopaedics and Related Research (n = 56) | 15 (26.8%) | 41 (73.2%) |
| Arthroscopy: The Journal of Arthroscopic and Related Surgery (n = 117) | 7 (6.0%) | 110 (94.0%) |
| Knee Surgery, Sports Traumatology, Arthroscopy (n = 114) | 16 (14.0%) | 98 (86.0%) |
| The Journal of Arthroplasty (n = 55) | 20 (36.4%) | 35 (63.6%) |
| European Spine Journal (n = 82) | 21 (25.6%) | 61 (74.4%) |
| Journal of Shoulder and Elbow Surgery (n = 18) | 5 (27.8%) | 13 (72.2%) |
| International Orthopaedics (n = 44) | 11 (25.0%) | 33 (75.0%) |
| Cochrane groups (n = 70) | 6 (8.6%) | 64 (91.4%) |
Publication Bias Assessments
We found 52 studies that met our inclusion criteria for publication bias assessment. Sufficient information was available in 41 systematic reviews to perform publication bias assessment (see Appendix). Funnel plot asymmetry was present in 36 analyses (88%). Studies were most often missing from the left side of the pooled analysis (54%). The Duval and Tweedie trim-and-fill method indicated that a mean of 3.5 added studies (range, 0 to 12 added studies) would be needed to produce symmetry of the funnel plot. Fourteen meta-analyses (34%) had evidence of publication bias when assessed with Egger regression, whereas 9 (22%) had evidence when assessed with Begg rank correlation. Evidence of publication bias from all 3 methods was noted in 7 studies (17%). No evidence of publication bias from any of the 3 methods was found in 3 studies (7%). The mean absolute percent difference between adjusted and observed point estimates was 17.2% (median, 9.2%; range, 0% to 123.1%) (Table III).
TABLE III.
Publication Bias Assessment Results*
| Article† | Begg P Value | No. of Studies Trimmed | Side of Mean Missing | Model | Observed Point Estimate (95% CI) | Adjusted Point Estimate (95% CI) | Egger Intercept | T Value |
| Andriolo et al. | 0.06 | 2 | Left | Random | 78.93 (71.13 to 86.74) | 77.15 (69.63 to 84.67) | 3.98 | 2.75† |
| Avenell et al. | 0.36 | 1 | Right | Random | 0.74 (0.49 to 1.11) | 0.77 (0.51 to 1.17) | −0.17 | 0.28 |
| Beckmann et al. | 0.0001 | 5 | Left | Fixed | 0.0019 (−0.006 to 0.0099) | 0.0014 (−0.0064 to 0.0093) | 0.47 | 5.19† |
| Beckmann et al. | 0.02 | 8 | Left | Random | 0.0076 (0.0047 to 0.01) | 0.0069 (0.004 to 0.0098) | 1.02 | 5.54† |
| Beckwee et al. | 0.27 | 2 | Left | Random | 1.09 (1.04 to 1.14) | 1.07 (1.01 to 1.13) | 1.30 | 1.75 |
| Buckland et al. | 0.31 | 12 | Left | Fixed | 53.71 (53.47 to 53.95) | 50.91 (50.68 to 51.15) | 21.99 | 2.99† |
| Chee et al. | 0.26 | 3 | Left | Random | 0.79 (0.46 to 1.39) | 0.70 (0.42 to 1.16) | −0.12 | 0.28 |
| Desai et al. | 0.47 | 0 | NA | Random | 0.36 (0.22 to 0.51) | 0.36 (0.22 to 0.51) | −0.37 | 0.36 |
| Hewison et al. | 0.15 | 1 | Right | Random | 0.49 (0.32 to 0.78) | 0.51 (0.33 to 0.80) | −0.93 | 1.02 |
| Higgins et al. | 0.33 | 0 | NA | Random | 0.43 (−0.04 to 0.90) | 0.43 (−0.04 to 0.90) | 0.66 | 0.19 |
| Houwert et al. | 0.20 | 0 | NA | Random | 1.32 (0.65 to 2.67) | 1.32 (0.65 to 2.67) | −0.62 | 0.58 |
| Kamper et al. | 0.24 | 3 | Left | Random | 1.02 (0.74 to 1.40) | 0.96 (0.70 to 1.32) | 0.54 | 1.14 |
| Kang et al. | 0.31 | 4 | Left | Random | 0.03 (0.02 to 0.04) | 0.02 (0.01 to 0.03) | 0.75 | 1.94 |
| Keurentjes et al. | 0.00006 | 3 | Left | Random | 91.53 (89.53 to 93.53) | 90.49 (87.63 to 93.36) | −6.11 | 3.29† |
| Kim et al. | 0.45 | 1 | Right | Random | 1.06 (0.98 to 1.15) | 1.07 (0.98 to 1.18) | 0.33 | 0.44 |
| Kim et al. | 0.31 | 6 | Right | Random | 0.32 (0.14 to 0.49) | 0.49 (0.31 to 0.67) | 1.97 | 0.95 |
| Li et al. | 0.35 | 4 | Left | Random | −0.13 (−0.51 to 0.25) | −0.29 (−0.62 to 0.03) | 5.41 | 1.43 |
| Lieberman et al. | 0.00001 | 0 | NA | Random | −0.91 (−1.21 to −0.62) | −0.91 (−1.21 to −0.62) | −4.53 | 6.35 |
| Liu et al. | 0.09 | 2 | Right | Fixed | 1.37 (0.99 to 1.88) | 1.43 (1.05 to 1.95) | −0.53 | 1.29 |
| Ma et al. | 0.21 | 3 | Right | Random | 0.36 (0.08 to 0.65) | 0.54 (0.24 to 0.85) | 1.74 | 1.11 |
| Medina et al. | 0.0004 | 7 | Left | Random | 17.94 (13.16 to 22.72) | 12.98 (7.72 to 18.24) | 2.68 | 3.98 |
| Meijer et al. | 0.24 | 2 | Right | Fixed | 1.07 (0.78 to 1.48) | 1.20 (0.88 to 1.64) | −1.38 | 1.06 |
| Negahban et al. | 0.003 | 7 | Left | Fixed | 0.36 (0.26 to 0.45) | 0.24 (0.14 to 0.33) | 3.69 | 2.72† |
| Pavon et al. | 0.33 | 4 | Right | Random | 0.93 (0.58 to 1.47) | 1.33 (0.81 to 2.19) | −1.01 | 1.39 |
| Peersman et al. | 0.02 | 4 | Left | Random | 0.0104 (0.0085 to 0.0123) | 0.0103 (0.0083 to 0.0122) | 0.51 | 1.71† |
| Rebal et al. | 0.15 | 2 | Left | Random | 0.14 (0.09 to 0.18) | 0.12 (0.07 to 0.17) | 1.66 | 0.83 |
| Riboh et al. | 0.44 | 2 | Left | Random | 1.28 (0.46 to 3.58) | 1.15 (0.43 to 3.12) | 0.29 | 1.01 |
| Santesso et al. | 0.07 | 6 | Left | Fixed | 0.91 (0.88 to 0.94) | 0.85 (0.83 to 0.88) | 2.58 | 1.62 |
| Saragiotto et al. | 0.07 | 4 | Right | Random | −0.49 (−0.70 to −0.29) | −0.34 (−0.56 to −0.12) | −1.69 | 1.79 |
| Schneider et al. | 0.00005 | 7 | Left | Random | 0.012 (0.0028 to 0.021) | 0.0085 (−0.000070 to 0.02) | 1.12 | 4.28 |
| Si et al. | 0.29 | 1 | Left | Fixed | 1.42 (1.07 to 1.88) | 1.39 (1.06 to 1.84) | 0.29 | 0.55 |
| Song et al. | 0.20 | 3 | Right | Fixed | 2.45 (2.40 to 2.50) | 2.48 (2.43 to 2.53) | −0.51 | 0.48 |
| Towle et al. | 0.21 | 5 | Left | Random | 1.29 (1.15 to 1.47) | 1.13 (0.98 to 1.29) | 0.48 | 0.35 |
| Van Bodegom Vos et al. | 0.17 | 2 | Right | Random | 0.49 (0.39 to 0.59) | 0.51 (0.41 to 0.62) | −1.06 | 1.46 |
| Wiggins et al. | 0.04 | 6 | Left | Random | 0.06 (0.05 to 0.07) | 0.05 (0.04 to 0.06) | 1.42 | 2.39 |
| Woodmass et al. | 0.11 | 4 | Right | Random | 0.41 (0.31 to 0.51) | 0.47 (0.37 to 0.57) | −1.39 | 1.98 |
| Wyles et al. | 0.16 | 6 | Left | Fixed | 0.12 (0.08 to 0.19) | 0.07 (0.05 to 0.10) | 2.51 | 6.10 |
| Xiao et al. | 0.26 | 1 | Right | Fixed | 0.56 (0.41 to 0.76) | 0.58 (0.43 to 0.78) | −1.56 | 1.36 |
| Yang et al. | 0.16 | 7 | Right | Fixed | −6.78 (−7.78 to −5.78) | −4.90 (−5.79 to −4.01) | −1.07 | 1.12 |
| Yun et al. | 0.21 | 0 | NA | Random | 4.40 (3.19 to 5.61) | 4.40 (3.19 to 5.61) | 0.77 | 0.87 |
| Zhang et al. | 0.31 | 2 | Left | Random | 0.24 (−0.11 to 0.59) | 0.15 (−0.23 to 0.53) | 1.87 | 1.22 |
CI = confidence interval, and NA = not applicable.
The references for these studies can be found in the Appendix.
Discussion
The primary objective of the present study was to assess the methods used to evaluate for publication bias in high-ranking orthopaedic journals as well as in journals from the orthopaedic-related Cochrane groups. We found that even though publication bias was discussed in 43% of the reviews published in these journals, a formal assessment for publication bias only occurred in roughly 20% of these reviews.
Our study demonstrated a higher percentage of reviews lacking formal publication bias assessment than the findings of other studies in anesthesiology11 and dermatology10, which demonstrated that publication bias was assessed in 43% and 21% of reviews, respectively. The same can be said in comparison with a similar study published in 20149 that investigated rates of publication bias assessments across multiple medical specialties. Of the 694 reviews that were included in our study, 559 gave no indication if publication bias affected their results. Of the studies that did formally evaluate for publication bias, 31.9% reported evidence of it. It appears that, compared with other fields of medical research, the orthopaedic literature is assessing for publication bias in systematic reviews at a lower rate. This finding is concerning given the disproportionately high levels of studies with positive findings in the surgical literature13. The effects of the overrepresentation of studies with positive findings as the only studies that should be investigated for clinical benefit leads us to the other shortcoming of publication bias: a drug or treatment that actually has substantial clinical benefit may not be adopted because, although several independent studies indicated a positive clinical effect, each study on its own lacked the power (did not include enough patients) to show this effect as being statistically significant (e.g., p < 0.05). For example, if 3 independent studies were to show a similar substantial improvement in patient outcome, but the p value in each study was p ≥ 0.05, the individual studies would be reported as demonstrating “no significant effect.” However, if the 3 studies were to be combined using meta-analysis, their combined p value may be significant.
There are many ways to prevent the effect of publication bias in medical literature. The driving force to implement these measures must come from the journals and publishers15-17. One method to increase the number of relevant studies included in systematic reviews is a search of the grey literature18. Grey literature has been defined as “that which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers,” as defined by the Luxembourg Convention on Grey Literature (i.e., conference proceedings, technical documents or government reports).19 Although searching for grey literature can be difficult, it provides a more complete view of the evidence. There are several guides that provide both reliable sources of grey literature and sound search methods18,20-23. Furthermore, publication bias also could be avoided if authors were required by journals to report the total sample size, the magnitude of the observed effect (mean and standard deviation), and the magnitude of the associated p value (and/or confidence interval), regardless of whether the p value was above or below an arbitrary value (e.g., p < 0.05) specified as corresponding to “statistical significance.” This would allow reviewers to more accurately assess the total magnitude of the observed effect and the significance of a study’s findings and would permit a meta-analysis that includes all of the studies, rather than just the ones with individually positive outcomes. In addition, we believe that if the medical community were to abandon the arbitrary use of the p value as a measure of “success,” the degree of publication bias would drastically decrease, as proposed in recent high-profile literature24-26. Such efforts have the power to mitigate the proven negative effects of publication bias and prevent an ineffective or harmful treatment from becoming the standard, or conversely, a potentially effective treatment from being dismissed as having “no significant effect” simply because the study or studies examining it individually lacked the power (i.e., sufficient numbers of patients) to show the effect at p < 0.05, or any other arbitrary value. As systematic reviews are considered high-level evidence in the creation of clinical practice guidelines, there is a need to address publication bias.
Reporting guidelines for systematic reviews also have the potential to address the issue of publication bias. However, these guidelines can be effective only if they are used properly and if adherence to the guideline is required by the journal. In our study, 50% of the reviews mentioned the use of the PRISMA guideline. Of those studies, 43% discussed publication bias and only 22% evaluated for it. The Cochrane group, well known for their support of reporting guidelines such as PRISMA27, discussed publication bias at the highest rate (90%).
One limitation of the present study is that other databases may have identified additional systematic reviews outside of PubMed and the Cochrane reviews. However, given that the journals included in our study are indexed in PubMed, this is a very low possibility. There are also other journal metrics, such as the impact factor, available to assess the rank of a journal. We chose Google Scholar’s H5-Index on the basis of its open availability. As the present study was limited to high-ranking journals, the results should not be generalized to lower-ranking journals.
In conclusion, the high level of publication bias found in the present study, coupled with the low levels of adherence to systematic review reporting guidelines, is of concern. Discussion and evaluation of publication bias in the development of systematic reviews should be standard practice. However, the quality of reporting for systematic reviews in high-ranking orthopaedic journals has been found to be poor28. Therefore, readers of systematic reviews should not assume that lack of assessment for publication bias indicates that publication bias was not present. The publication bias present in orthopaedic journals is worrisome, considering that randomized controlled trials published in orthopaedic journals likely have exaggerated treatment effects due to bias29. It has been shown that in high-ranking orthopaedic journals, such as The Journal of Bone & Joint Surgery, positive and nonpositive studies were accepted at similar rates30. These findings should encourage orthopaedic researchers to submit studies regardless of the direction of the results.
Appendix
Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A93).
Footnotes
Investigation performed at the Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma
Disclosure: The authors indicated that no external funding was received for any aspect of this work. The Disclosure of Potential Conflicts of Interest forms are provided with the online version of the article (http://links.lww.com/JBJSOA/A92).
References
- 1.Phan K, Mobbs RJ. Systematic reviews and meta-analyses in spine surgery, neurosurgery and orthopedics: guidelines for the surgeon scientist. J Spine Surg. 2015. December;1(1):19-27. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.DiSilvestro KJ, Tjoumakaris FP, Maltenfort MG, Spindler KP, Freedman KB. Systematic reviews in sports medicine. Am J Sports Med. 2016. February;44(2):533-8. Epub 2015 Apr 21. [DOI] [PubMed] [Google Scholar]
- 3.Provencher MT, Brand JC, Rossi MJ, Lubowitz JH. Are orthopaedic systematic reviews overly prevalent? Arthroscopy. 2016. June;32(6):955-6. [DOI] [PubMed] [Google Scholar]
- 4.Little Z, Newman S, Dodds A, Spicer D. Increase in quality and quantity of orthopaedic studies from 2002 to 2012. J Orthop Surg (Hong Kong). 2015. December;23(3):375-8. [DOI] [PubMed] [Google Scholar]
- 5.Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0. 2011. March http://handbook-5-1.cochrane.org. Accessed 2019 Apr 1. [Google Scholar]
- 6.Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. QUality Of Reporting Of Meta-analyses. Lancet. 1999. November 27;354(9193):1896-900. [DOI] [PubMed] [Google Scholar]
- 7.Moher D, Liberati A, Tetzlaff J, Altman DG. PRISMA Group. Preferred Reporting Items for Systematic reviews and Meta-Analyses: the PRISMA statement. PLoS Med. 2009. July 21;6(7):e1000097. Epub 2009 Jul 21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Checketts JX, Sims MT, Detweiler B, Middlemist K, Jones J, Vassar M. An evaluation of reporting guidelines and clinical trial registry requirements among orthopaedic surgery journals. J Bone Joint Surg Am. 2018. February 7;100(3):e15. [DOI] [PubMed] [Google Scholar]
- 9.Onishi A, Furukawa TA. Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study. J Clin Epidemiol. 2014. December;67(12):1320-6. Epub 2014 Sep 4. [DOI] [PubMed] [Google Scholar]
- 10.Atakpo P, Vassar M. Publication bias in dermatology systematic reviews and meta-analyses. J Dermatol Sci. 2016. May;82(2):69-74. Epub 2016 Feb 24. [DOI] [PubMed] [Google Scholar]
- 11.Hedin RJ, Umberham BA, Detweiler BN, Kollmorgen L, Vassar M. Publication bias and nonreporting found in majority of systematic reviews and meta-analyses in anesthesiology journals. Anesth Analg. 2016. October;123(4):1018-25. [DOI] [PubMed] [Google Scholar]
- 12.Eyding D, Lelgemann M, Grouven U, Härter M, Kromp M, Kaiser T, Kerekes MF, Gerken M, Wieseler B. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010. October 12;341:c4737. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Hasenboehler EA, Choudhry IK, Newman JT, Smith WR, Ziran BH, Stahel PF. Bias towards publishing positive results in orthopedic and general surgery: a patient safety issue? Patient Saf Surg. 2007. November 27;1(1):4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Sedgwick P, Marston L. How to read a funnel plot in a meta-analysis. BMJ. 2015. September 16;351:h4718. [DOI] [PubMed] [Google Scholar]
- 15.Dirnagl U, Lauritzen M. Fighting publication bias: introducing the negative results section. J Cereb Blood Flow Metab. 2010. July;30(7):1263-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Demaria AN. Publication bias and journals as policemen. J Am Coll Cardiol. 2004. October 19;44(8):1707-8. [DOI] [PubMed] [Google Scholar]
- 17.Diguet E, Gross CE, Tison F, Bezard E. Rise and fall of minocycline in neuroprotection: need to promote publication of negative results. Exp Neurol. 2004. September;189(1):1-4. [DOI] [PubMed] [Google Scholar]
- 18.Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods. 2014. September;5(3):221-34. Epub 2013 Dec 6. [DOI] [PubMed] [Google Scholar]
- 19.Boekhorst AK, Farace DJ, Frantzen J. Grey Literature Survey 2004: a research project tracking developments in the field of grey literature. Grey Journal-Amsterdam. 2005;1(1):41. [Google Scholar]
- 20.Auger CP. Information sources in grey literature. Berlin: Walter de Gruyter; 2017. https://trove.nla.gov.au/work/8702344?q&versionId=45642315. Accessed 2018 Jun 25. [Google Scholar]
- 21.Canadian Agency for Drugs and Technologies in Health (CADTH). Grey Matters: a practical tool for searching health-related grey literature. https://www.cadth.ca/resources/finding-evidence/grey-matters. Accessed 2018 Jun 25.
- 22.Saleh AA, Ratajeski MA, Bertolet M. Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. Evid Based Libr Inf Pract. 2014;9(3):28-50. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Giustini D, Thompson D. Finding the hard to finds: searching for grey literature. 2012. https://guides.mclibrary.duke.edu/sysreview/greylit [Google Scholar]
- 24.Baker M. Statisticians issue warning over misuse of p values. Nature. 2016. March 10;531(7593):151. [DOI] [PubMed] [Google Scholar]
- 25.Woolston C. Psychology journal bans p values. Nature. 2015;519(7541):9. [Google Scholar]
- 26.Wayant C, Scott J, Vassar M. Evaluation of lowering the p value threshold for statistical significance from. 05 to. 005 in previously published randomized clinical trials in major medical journals. JAMA. 2018. November 6;320(17):1813-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Mann M, Sampson M, Cooper C, Stanfield C, Rader T. PRISMA in practice: keeping track of the literature search documentation. What are the problems? Presented at the 19th Cochrane Colloquium; 2011. October 19-22; Madrid: http://2011.colloquium.cochrane.org/abstracts/b2o4-prisma-practice-keeping-track-literature-search-documentation-what-are-problems.html. Accessed 2018 Jun 28. [Google Scholar]
- 28.Gagnier JJ, Kellam PJ. Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Joint Surg Am. 2013. June 5;95(11):e771-7. [DOI] [PubMed] [Google Scholar]
- 29.Chess LE, Gagnier J. Risk of bias of randomized controlled trials published in orthopaedic journals. BMC Med Res Methodol. 2013. June 9;13:76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M. Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery (American Volume). J Bone Joint Surg Am. 2008. March;90(3):595-601. [DOI] [PubMed] [Google Scholar]