Analysis of Publication Bias in the literature for Distal Radius Fracture

Abstract

Purpose

Distal radius fractures are the most commonly treated fracture, and their management remains complex. We aim to evaluate the presence of publication bias in the literature on distal radius fracture management and identify specific study variables that may influence the reporting of positive outcomes.

Methods

We conducted a systematic review on all available journal articles to find primary articles reporting on the management of distal radius fractures. Data collected included the direction of study outcome (positive, neutral, and negative) and various study characteristics including sample size, geographic origin of the study, clinical setting, study design, type of treatment, analysis for statistical significance, evaluation of wrist function, presence of subjective outcome measures, mean follow-up time, adequacy of reduction, complications, mean patient age, and the presence of any extramural funding.

Results

We reviewed 215 journal articles and found that 70% of articles reported positive outcomes, 25% reported neutral outcomes, and 5% reported negative outcomes. Funnel plot analysis suggested the presence of publication bias due to the asymmetric distribution of studies. In addition, we found statistically significant differences between study outcomes with respects to treatment type, presence of external funding, reduction adequacy, hand/wrist functional assessment, and patient questionnaires for subjective assessment.

Conclusions

Publication bias likely exists in the literature for distal radius fracture management. Several study characteristics influence the reporting of positive outcomes, but whether or not the presence of these characteristics portends a greater chance of publication remains unclear. A standardized approach to measure and track results may improve evidence-based outcomes.

INTRODUCTION

Distal radius fractures are the most common fracture treated by physicians [1] and have a substantial impact on health care. [2]The annual incidence in the United States is more than 640,000 cases among all ages.[3] Management of distal radius fractures remains difficult and is complicated by varying fracture patterns, various treatment options, and the complex relationship between reduction and functional outcome. [4] Treatment selection relies on evidence-based literature, which depends on the availability of unbiased and objective data from published studies.

Publication bias refers to the tendency of researchers, peer reviewers, and journal editors to submit or accept manuscripts for publication based on the direction or strength of study findings. [5] In other words, publication of studies reporting statistically significant or positive findings is more likely than publication of those without. [6] The validity of literature and foundation for evidence-based practice may be compromised by publication bias because scientific publications are the source for systematic reviews and meta-analyses. [5] A serious potential consequence of publication bias is that it may overestimate treatment effects in published work that could lead to inappropriate or unjustified treatment methods. [7]

Publication bias has been recognized and described in the internal medicine literature; however, despite its potentially detrimental clinical impact, the prevalence of publication bias remains largely unexplored in surgery, particularly in hand surgery. [6]Determining the presence of publication bias in the distal radius facture literature and investigating factors that lead to unbalanced reporting may improve patient care and reduce unjustified treatments. The purpose of this study was to conduct a critical review of all available literature on the treatment of distal radius fractures to evaluate the presence of publication bias. We hypothesized that studies with positive (statistically significant) findings were published in greater numbers in comparison to those with either negative or neutral findings (non-significant findings) and that the reporting of positive outcomes was influenced by specific study variables.

METHODS

A systematic literature review was performed using MEDLINE, SCOPUS, and EMBASE databases to find primary articles reporting on treatments and outcomes of distal radius fractures (Figure 1). The search was performed using the key words distal radius, fracture, treatment, and reduction. Database limits were used to exclude non-human, pediatric, and non-English studies. After deleting duplicate studies, articles and abstracts were then screened to exclude technique papers, studies with concomitant fractures (with the exception of ulnar styloid fractures) or treatment for nonunion or other complications, and studies describing pharmacologic treatment for distal radius fractures. Furthermore, anatomic, biomedical, economic, and rehabilitation studies were excluded. Screening and analysis of articles was performed by 2 investigators (IS and SM) independently. Any discrepancy was discussed and resolved by consensus. All original, full-text research articles were included and were defined as studies that presented original data and were not review articles.

Figure 1.

Flowchart of database search.

Each article included in our investigation was then analyzed and outcome directions (positive, neutral, or negative) for each was categorized based on the methodology adopted from Hasenboehler et al. [6] Accordingly, a positive outcome article presented significant differences between comparative groups with positive conclusions and/or recommendations. A neutral outcome article showed no differences between comparative study groups and reached no clear conclusions or offered no specific recommendations. A negative outcome article depicted significant differences between comparative groups with negative conclusions and/or recommendations. After categorizing the articles, we calculated the percentage of each type of study published as a ratio of the total number of articles included in our review using the formula: (number of obtained positive or negative or neutral articles divided by total number of articles in review) x100. We then collected data for different study variables including study sample size, geographical origin of the study, clinical setting (academic vs. community), study design, type of treatment, quality of fracture reduction, complications, analysis for statistical significance, mean follow-up time, mean patient age, and the presence of any extramural funding. Articles were also evaluated for the presence of hand/wrist function assessment and for the presence of patient questionnaires for subjective assessment.

Publication bias analysis

Study variable and outcome direction data were compiled for each article. Chi-square tests and Fisher exact tests (for values less than 5) were used to analyze potential associations between study variables and outcome direction. Statistical significance was set at a P value of 0.05. Funnel plots were then used to evaluate for the presence of publication bias. Funnel plots are scatter plots in which treatment effects estimated from individual studies placed on the horizontal axis are plotted against a measure of study precision on the vertical access. [8] In the absence of publication bias, the graph resembles an inverted funnel as treatment effects for small studies scatter more widely toward the bottom, whereas they narrow toward the top with increasing precision among larger studies. [9, 10] However, when bias is present, the plot will be asymmetrical due to non-publication of studies that are small or do not have significant results. An illustration of this is provided in Figure 2. Furthermore, one should be cautious with regards to interpretation of the shape of funnel plot. The appearance of the plot is dependent upon the choice of metrics used on both axes, such as odds ratio, risk ratio on x-axis, and sample size, standard error, or inverse standard error (precision) on y-axis. [9, 11] Precision is the inverse of standard error which is calculated as S.E. = S.D. /√N where S.D. is the standard deviation and N is the sample size of the study. Precision and sample size are directly related. In other words, the greater the sample size, the smaller will be its standard error, and thus greater precision will be achieved for an effect estimate. Log of the effect estimate (odds ratio or risk ratio) for the x-axis and inverse standard error for the y-axis were found to be preferred measures for funnel plot. [9] In addition, heterogeneity of individual studies, reporting bias, and chance can lead to funnel plot asymmetry. [10] We extrapolated treatment effects from studies comparing 2 or more treatment methods. Common outcome measures between these studies included motion measurements of wrist flexion, extension, and radial and ulnar deviation and forearm pronation and supination. Mean values for the various range of motion measurements and Disabilities of the Arm, Shoulder and Hand (DASH) scores were plotted against their precision, and the shape and symmetry of the resulting funnel plots were evaluated for the presence of publication bias.

Figure 2.

Illustration of funnel plots, (A) A funnel plot with pooled estimate (solid line) and approximate point-wise 95% confidence limits showing no apparent publication bias. (B) A funnel plot suggesting publication bias in favor of larger studies and positive results.* * Adapted from Andrek JA, Keith SW, Leiby BE. Meta-analysis: A brief introduction. Clin Transl Sci. 2009; 2(5)374–378.

RESULTS

A total of 2459 articles was identified through our initial search of MEDLINE, SCOPUS, and EMBASE databases between 1979 and 2012 using the key words distal radius, fracture, treatment, and reduction. After excluding the 1424 duplicates, 1035 articles remained. Non-human, pediatric, and non-English studies were excluded using database limits; and of the remaining 492 articles, 277 were excluded based on the other criteria. We thus obtained 215 articles for our review. (Figure 1)

Our analysis showed that 70% of articles reporting on the treatment for distal radius fractures were associated with positive outcomes, 25% were associated with neutral outcomes, and 5% were associated with negative outcomes. In negative outcome studies, the authors did not recommend the proposed treatment in comparison to positive studies where the authors strongly recommended the treatment method of choice. In neutral studies, there were no specific recommendations for or against the proposed treatment method made. The frequency and distribution of studies based on outcome for various study variables are presented in Table 1. The vast majority of studies were from universities (88%), whereas only 12% were from community or private practice. A total of 88 (41%), 71 (33%), and 56 (26%) articles originated from Europe, North America, and other continents, respectively. Half of all articles were prospective studies (50%), nearly half were retrospective case reviews (48%), and about 2% were case reports. The majority of articles had sample sizes fewer than 50 (64%), and only 10% of articles had greater than 100 study subjects. Few studies had mean follow-up greater than 5 years (4%). A higher proportion of positive outcome studies were associated with adequate reduction compared with negative outcome studies (87% vs.43 %). The results of chi-square and Fisher exact tests are presented in Table 2. We found that the following studies variables had significant influence on the reporting of study outcome: treatment type, presence of external funding, reduction adequacy, hand/wrist functional assessment, and patient questionnaires for subjective assessment.

Table 1.

Distribution of positive, neutral, and negative outcome studies according to study variable

		Outcome
		Positive	Neutral	Negative
Setting	University	135	47	8
	Community-based	15	7	3
Origin	Europe	56	25	7
	North America	50	19	2
	Other	44	10	2
Sample Size	1 to 50	97	33	8
	51 to 100	40	12	3
	> 100	13	9	0
Study Design	Case Report	4	0	1
	Prospective	75	26	6
	Retrospective	71	28	4
Mean Patient Age	18–50	62	17	1
	> 50	79	32	9
	Not Available	9	5	1
Mean Follow Up	< 1 year	27	8	3
	1–5 years	110	40	4
	> 5, < 15 years	5	2	1
	Not Available	8	4	3
Treatment	ORIF	66	14	1
	Closed Reduction	14	6	2
	External Fixation or Percutaneous Pinning	49	17	4
	Other	21	17	4
Reduction	Adequate Reduction	130	29	3
	Poor Reduction	8	12	6
	Not Available	12	13	2
Subjective Assessment	Assessed	47	37	3
	Not Assessed	103	17	8
Assessment of Function	Assessed	141	47	7
	Not Assessed	9	7	4
Complications	Clinically Significant	33	15	5
	Insignificant	116	36	6
	Not Available	1	3	0
Statistical Significance	Present	69	19	7
	Absent	81	35	4
Funding	External Funding	18	6	5
	No External Funding	132	48	6

Table 2.

Chi-square test of independence between study variable and outcome

Study Variable	Chi-Square
Setting	χ2 = 3.10, P = 0.212
Origin	χ2 = 5.11, P = 0.276
Sample Size	χ2 = 4.20, P = 0.380
Study Design	χ2 = 4.07, P = 0.397
Mean Patient Age	χ2 = 5.89, P = 0.207
Mean Follow Up	χ2 = 10.85, P = 0.093
Treatment*	χ2 = 14.76, P = 0.022
Reduction*	χ2 = 36.76, P < 0.001
Subject Assessment*	χ2 = 12.34, P < 0.001
Assessment of Function*	χ2 = 23.63, P = 0.002
Complications	χ2 = 9.07, P = 0.059
Statistical Significance	χ2 = 3.66, P = 0.160
Funding*	χ2 = 10.18, P = 0.006

^*P < .05

Out of the 215 articles in our final review, 43 studies compared 2 or more treatment methods and formed the basis for our publication bias analysis. Data on the outcome measure, range of motion, were presented in 15 of these studies, of which only 8 presented relevant data such as mean and standard deviation essential to construct a funnel plot. Six studies presented data on DASH scores. Inspection of our funnel plots showed considerable asymmetry when mean flexion, extension, ulnar deviation, radial deviation, pronation, and supination measurements and DASH scores were plotted against their respective precision (inverse of standard error). Studies with large sample sizes (standard error and sample size are inversely related) are concentrated towards the upper region, whereas there are few or no studies with small sample sizes resulting in an overall non-uniform distribution of the plot. In other words, the studies are unequally distributed on either sides of the central line that represents the summary statistic or the pooled effect estimate. (Figure 3) Overall, results from our funnel plot analysis suggest the presence of publication bias and an overestimation of treatment effects in the literature of distal radius fracture management. Availability of large number of studies to construct funnel plots is always desirable and advantageous to ascertain the results. However, in this review, due to lack of such an advantage the shape of our plots provided us the results.

Figure 3.

Funnel plots. The mean values of various range of motion measurements and DASH vs. precision.

DISCUSSION

We present an analysis of all primary articles reporting on the treatment of distal radius fractures. Our funnel plot analysis suggested the presence of publication bias. Furthermore, in our review, 70% of published articles reported positive outcomes with distal radius fracture management, which is similar to the 74% of studies reporting positive outcomes for various conditions treated in the general surgery and orthopedic surgery literature. [6]

Although establishing and maintaining anatomic reduction has long been the goal in the treatment of distal radius fractures, the paradigm has shifted toward optimizing patient satisfaction through improvement in hand function regardless of reduction. [12] In a time of rising healthcare costs, [13] budget limitations, and a new healthcare model, physicians are under pressure to improve resource management. Studies have shown that radiographic outcomes do not always correlate with functional outcomes and that the cost burden associated with certain treatment modalities may be avoidable.[14, 15] Over reporting of positive outcomes in the literature may overestimate the clinical effects of certain treatment methods due to lack of information from unpublished negative studies. [16, 6

A recent Cochrane review showed that clinical trials with positive results are more likely to be submitted, expedited through review, and accepted for publication than those with negative or less favorable results. [17]Either negative or neutral outcome studies are 2.3 times less likely to reach publication than positive outcome studies. [18] This bias towards publication leads to an overestimation of treatment effects or risk-factor associations in publications that could lead to inappropriate decision-making in patient management. [19] Awareness about the prevalence of publication bias in the medical and surgical literature and its potential consequences in the practice of evidence-based medicine and patient care is increasing. [5, 7, 18] Surgical journals may be at even greater risk for publication bias given their high prevalence of observational and case series. [20]

It is important for the readers to be aware of other possible biases that result in publication bias while reviewing the literature. Author bias or submission bias refers to investigators’ tendency to submit studies with positive outcomes, whereas studies that do not have positive results may never be submitted for publication. [7]. Although there are no measures to detect this, possible reasons include either negative or neutral results being uninteresting to publishers and readers and publication not being the sole aim for conducting a study. Lack of time for investigators also prioritizes studies with positive results over studies with other results to be submitted. Editor or publisher bias refers to journal editors’ selective publication of studies with positive results. Space constraints, impact factor, and readership acquisition often result in editor’s bias. [7]

Previous studies have shown that certain study variables, such as a large sample size, the presence of commercial sponsorship, country of origin, and the presence of statistically significant results have been shown to increase the tendency toward publication. [18, 21–23] Overall, the majority of studies in our analysis were smaller case series and had follow-up period less than 5 years. Long term outcomes were therefore not reported. A high proportion of studies in our review were from academic institutions. As a result, outcomes from the private or community-based setting are underrepresented compared to academic institutions, and differences in treatment patterns and patient outcome are difficult to appreciate. Consequently, outcomes presented in the distal radius fracture literature may not accurately reflect general clinical practice. Our analysis also showed statistically significant differences between outcome groups with respect to treatment type, reduction adequacy, hand/wrist functional assessment, and patient questionnaires for subjective assessment, which may suggest that such variables are associated with positive outcomes. However, manuscripts that have not been published were unavailable to us, and thus, we were unable to evaluate whether the inclusion of certain study variables portend a greater chance of ultimate publication.

There are several limitations to our study. Our analysis included published literature only. Articles that were rejected or not submitted were not available for our review. However, we proceeded to do this analysis because investigations conducted earlier had used percentages of published articles to establish publication bias. [6, 20, 21] Our analysis however included funnel plot approach to confirm our preliminary results of publication bias existence in distal radius fracture literature. Similar to other systematic reviews, our study was limited by the quality of the studies we reviewed. Studies were difficult to standardize for analysis due to inconsistencies between study design and measurements of outcome. Study designs ranged from case reports to multicenter, randomized control trials; and tools to measure objective and subjective outcomes, such as active range of motion, grip strength, DASH score, patient-rated wrist evaluation score, visual analog scale, Green and O’Brien score, Michigan Hand Questionnaire, and Gartland Werley classification varied widely. Furthermore, several fracture classification schemes exist, which limited comparison between studies. Our study consisted only of studies in English, which limited its scope. Although we were able to identify associations between study variables and outcome direction, we were unable to determine whether the presence of certain variables in a study portend a higher likelihood of publication.

Our study suggests that publication bias exists in the literature for the treatment of distal radius fractures and that certain study characteristics influence the reporting of positive outcomes. Comparison of studies based of their outcome is nonetheless difficult due to inconsistencies between study designs and outcome measurements. In order to improve treatment algorithms and optimize evidence-based outcomes for the management of distal radius fractures, we recommend creating a standardized nationwide clinical registry to track and evaluate outcomes.