Quotation errors related to the Proximal Fracture of the Humerus Evaluation by Randomization (ProFHER) study

Abstract

Aim

The aim of this study was to assess the accuracy of quotations of the Proximal Fracture of the Humerus Evaluation by Randomization (ProFHER) study in the published literature.

Methods

A literature search was performed from March 2015 to November 2019 to identify all papers that reference ProFHER since its publication. Full text articles were reviewed by two independent reviewers using a validated framework of assessing quotation errors. A kappa co-efficient was calculated to assess interobserver reliability of the reviewers.

Results

There were 260 individual ProFHER quoted references within the 138 included articles. We identified 35/260 quotation errors (13%). Of these, 10/35 (29%) were major quotation errors and 25/35 (71%) minor quotation errors. There was substantial interobserver agreement when errors were classified. Of the 10 major errors, six quotations were not substantiated by the results of ProFHER and three were unrelated to ProFHER. One paper contained a quotation error that contradicted the results of ProFHER. Of the 25 minor errors, 19 oversimplified or generalised the conclusions of ProFHER and six contained numerical or grammatical errors.

Conclusion

The current study demonstrated substantial inaccuracies in quotations of the Proximal Fracture of the Humerus Evaluation by Randomization study. Vigilance is recommended when quoting the literature and reviewing submitted papers in order to prevent the perpetuation of misquoted data.

Introduction

Proximal humeral fractures (PHFs) are common, have a bimodal age distribution ¹ and account for 5–6% of all adult fractures. ² Consensus regarding the optimal management of PHFs is lacking, particularly for older patients. Contributing to this lack of consensus are the many types of treatment available (Figure 1). PHFs are commonly managed either nonoperatively (with slings, braces and physiotherapy) or operatively (fixation with screws, intramedullary nails, locking plates, percutaneous techniques, hemiarthroplasty, reverse arthroplasty).^1,3–5

Figure 1.

Modified axial (a) and AP (b) views of a 67-year-old female patient's shoulder, demonstrating a displaced surgical neck fracture of the proximal humerus. There are several ways to treat this fracture.

Numerous publications have attempted to guide PHF treatment. An influential recent study was the Proximal Fracture of the Humerus Evaluation by Randomization (ProFHER) article. ⁶ This was a pragmatic parallel-group multicentre randomised controlled trial. ProFHER addressed the treatment of displaced fractures of the proximal humerus involving the surgical neck in adults. The clinical outcomes of surgical and non-surgical treatment were compared, with follow-up over two years. PHFs with a clear indication for surgery were excluded.

ProFHER concluded that, at two years after injury, there was no clinical benefit to surgical treatment of displaced PHFs involving the surgical neck in adult patients. The results may not be generalisable to all displaced PHFs because of ProFHER's pragmatic nature and broad inclusion and exclusion criteria. ⁷ Nevertheless, since ProFHER's publication, its findings have guided treatment and further research.^8,9 Accurate quotation in the literature of ProFHER's findings is important. Inaccurate quotations might mislead readers, resulting in suboptimal treatment or poorly guided further research. ¹⁰

A quotation error is defined as a discrepancy between the author's assertion and information presented in referenced material. It is distinct from a citation error (an error occurring in the bibliography, for example the incorrect spelling of an author's name in the reference). A 2008 Cochrane review reported the findings of 66 studies concerning the quotation errors of 27,000 references. The median quotation error rate was 20% (range 0–50%). ¹¹

The aim of this study was to establish the rate of quotation error in literature quoting the ProFHER study.

Methods

Literature search

A literature review was conducted using EMBASE, Google Scholar, Medline, Ovid Cited Reference search and Web of Science databases. We included literature that referenced the ProFHER trial in their bibliography, using publications from March 2015 to November 2019. The final search was conducted on the 1st November 2019. We excluded duplicate articles, non-English language articles, textbooks, PhD theses and technical tip videos.

Literature evaluation

Evaluation of the remaining full text articles was independently conducted by two reviewers. Quotation errors were assessed and classified as either major or minor errors, according to a previously validated framework. ¹² All errors were then discussed and any disparities between the two datasets were reviewed and reclassified with agreement of both reviewers. The senior author was consulted when a consensus could not be met by the primary reviewers.

Major errors were defined as references that did not substantiate, were unrelated to or contraindicated the findings of the ProFHER trial. Minor errors were defined as either trivial numerical inaccuracies or statements that oversimplified or generalised the findings of ProFHER but did not considerably change the original assertion. Multi-referencing refers to the use of multiple references to support the same quotation.

Data analysis

Data were collected in a pre-designed Microsoft Excel sheet (Microsoft 2019). The paper's title, type and class of error were recorded. For each error, the rationale for error classification was noted, and whether the error statement was mono- or multi-referenced as well as the number of ProFHER citations in the paper.

A kappa co-efficient was calculated to measure interobserver reliability when assessing for quotation errors.

Results

A total of 187 items of academic literature were identified from the literature search. Full text articles were retrieved for all papers. Of the 187 items, 49 were excluded (Table 1) and the remaining 138 papers were reviewed in full. There were 260 separate ProFHER quotations.

Table 1.

Category and numbers of papers excluded.

Category for exclusion	Number of excluded papers
Non-English language papers	23
Textbooks	9
Duplicates	6
PhD theses	3
Technical tip videos	3
Unavailable	2
ProFHER not cited	2
Miscellaneous	1
Total	49

ProFHER: Proximal Fracture of the Humerus Evaluation by Randomization.

We found 34/138 (25%) papers reviewed included at least one quotation error. A total of 35/260 (13%) quotation errors were identified. One paper (3%) contained two minor errors. There were 10/35 (29%) major errors and 25/35 (71%) minor errors identified (Figure 2).

Figure 2.

Distribution of errors, including both major and minor.

Major errors

Of the 10 major errors (Table 2), 6/10 (60%) concerned statements not supported by the ProFHER paper.^13–18 For example one paper incorrectly cited ProFHER as reporting that certain fracture patterns had a worse prognosis. ¹⁵

Table 2.

Breakdown of major errors.

Error type	Quotation error	Correct finding from ProFHER	Reference
Unsubstantiated	70% of secondary surgery occurred within the first year	This was not mentioned in the ProFHER paper	Launonen et al. 13
Unsubstantiated	The literature supports the conservative management of non-displaced factures of the humerus	ProFHER included only displaced fractures	Antonios et al. 14
Unsubstantiated	Certain fracture patterns have worse prognosis	This was not mentioned in the ProFHER paper	Hudgens et al. 15
Unsubstantiated	Outcomes when comparing surgical versus non-surgical management are similar regardless of fracture pattern	ProFHER only included displaced fractures. Fractures that had a clear indication for surgery were excluded	Gracitelli et al. 16
Unsubstantiated	ProFHER is against surgical fixation	ProFHER did not demonstrate inferior outcomes with surgical treatment – they were not significantly different and a number of fractures that had a clear indication for surgery were excluded	Meunier et al. 17
Unsubstantiated	ProFHER is unclear if younger patients may benefit from surgical fixation	An age group subanalysis was performed that found no significant difference between >65 and <65 in either group	von Keudell et al. 18
Unrelated	Patients from nursing homes are treated conservatively with the primary aim being pain relief	This was not mentioned in the ProFHER paper	Spross et al. 19
Unrelated	A certain subset of patients with high grade acromioclavicular joint separation may benefit from conservative management	This was not mentioned in the ProFHER paper	Cooper et al. 20
Unrelated	Ankle injury management (AIM) trial reference	This was not mentioned in the ProFHER paper	Costa et al. 21
Contraindicated	There were significantly higher complications in the ProFHER surgical group	There were higher numbers of complications related to the shoulder fracture in the surgical group; however, this was not significant	Launonen et al. 22

ProFHER: Proximal Fracture of the Humerus Evaluation by Randomization.

Papers that cited ProFHER to support assertions that were unrelated to the topic of ProFHER accounted for 3/10 (30%) of major errors.^19–21 One paper cited ProFHER as supporting non-surgical management for patients with acromioclavicular joint separation. ²⁰ This was not relevant to the ProFHER trial.

The remaining 1/10 (10%) major error was a contradiction to the findings reported in ProFHER. The paper quoted a higher risk of complications and re-operation in the surgical group. ²² ProFHER reported a non-significant difference in complications related to the shoulder fracture or its treatment and 9% of patients in both groups required further surgery.

Minor errors

Of the 25 minor errors, 19/25 (76%) oversimplified or generalised the findings of the ProFHER trial. For example, one paper cited ProFHER as reporting no benefit to using operative locked plating to surgically correct displaced PHFs when compared to non-surgical management. ²³ ProFHER described both the use of locking plates and other modalities (such as hemiarthroplasty and intramedullary nails, which accounted for almost 18% of patients) and this was not made clear. The remaining 6/25 (24%) quoted minor numerical or grammatical/spelling inaccuracies that did not change the overall interpretation of ProFHER's results.

Multi-referencing

Of the identified errors 16/35 (46%) were part of groups of references used to support single assertions.

Interobserver agreement

The kappa co-efficient in relation to interobserver reliability was 0.789 indicating substantial agreement between the two independent observers.

Discussion

The current study found 13% of ProFHER quotations in the surgical literature to be inaccurate. Of these quotation errors 10/35 (29%) were major and 25/35 (71%) minor errors.

Quotation error rate

The quotation error rate (13%) was lower than has been previously shown in the general medical and orthopaedic literature. A systematic review and meta-analysis of quotation accuracy across the medical literature demonstrated an overall quotation error rate of 25.4%, ²⁴ similar to that reported in a 2008 Cochrane review of 20%. ¹¹ The range of quotation errors reported in purely orthopaedic literature was between 7.6 and 41%.^10,25–27

A possible explanation for the low ProFHER quotation error rate was that the ProFHER study is a large and well-known trial with 93% of orthopaedic surgeons who treat PHFs reportedly aware of ProFHER's findings. ⁹ This means that authors of other related research and peer-reviewers for journals were likely to be familiar with the results and could correctly identify most quotation errors.

Ratio of major and minor errors

In our study minor errors outnumbered major errors. The previous literature reported similar numbers of major and minor errors^10,24,25,28 or even higher numbers of major errors.^26,29

Our findings could be explained either by the subjective nature of error classification or by differences in major and minor error definition. Mogull ¹² re-evaluated quotation error rates with standardised set of definitions in 15 studies and demonstrated that six of these studies had similar ratios of major to minor errors while the remaining nine studies had a higher number of major errors than minor errors.

Causes of quotation error

Numerous studies have sought to identify factors that contribute to quotation errors. The following do not correlate with error rate: journal impact factor,^10,25,26,29 number of authors, word count, references per article^25,26,28 or study design.^26,28

A possible cause of quotation errors was multi-referencing. We found 16/35 (46%) identified errors involved multiple references to support a statement. This was lower than the previously reported 65% of errors associated with multi-referencing. ¹⁰ The same study also found major errors were significantly more common when multi-referencing was used. Our low multi-reference rate may have contributed to the observed lower major error rate.

Currently, it is unclear what factors could impact on quotation error rates but these inaccuracies are detrimental to the quality of the medical literature as their presence undermines the reliability of published work if readers are unable to trust cited statements. It is imperative that authors ensure the accuracy of their work and it is their onus to ensure they have not misinterpreted referenced material, ²⁴ especially when citing and quoting high quality evidence and data, such as the ProFHER trial. A number of methods have been advised to help authors reduce the number of quotation errors. These include reading any referenced material in its entirety to ensure isolated statements are not taken out of context, using primary references in preference to indirect references and if possible reducing the number of references used. ²⁴

We recommend that if an error is identified, the reader should contact the corresponding author of the article to inform them of this. Quotation errors are usually inadvertent and so this offers the author the opportunity to correct their work and improve the reliability of the literature. If the authors take no action or if serious errors are encountered with the potential to lead to patient harm then the editor of the journal should be requested to issue a correction or retraction. An alternative to these conventional methods of reporting is to use social media and online resources such as pubpeer.com that allow post publication independent scrutiny of scientific articles. ³⁰

Study limitations

First, our study reported only quotation errors regarding the ProFHER trial and did not address citation errors. Second, the findings regarding the ProFHER trial cannot be generalised to the wider published literature. Further work is necessary to investigate practices that increase the risk of quotation error. For example, no study has examined whether the quotation error rates for highly cited and quoted studies differ from studies that have fewer citations.

Conclusion

This study found a 13% quotation error rate regarding the ProFHER study. This error rate is lower than that reported by larger pooled datasets.^11,24 We suggest authors and reviewers remain vigilant to lower the quotation error rate of the published literature.