Improvement of research quality in the fields of orthopaedics and trauma—a global perspective

Abstract

The international orthopaedic community aims to achieve the best possible outcome for patient care by constantly modifying surgical techniques and expanding the surgeon’s knowledge. These efforts require proper reflection within a setting that necessitates a higher quality standard for global orthopaedic publication. Furthermore, these techniques demand that surgeons acquire information at a rapid rate while enforcing higher standards in research performance. An international consensus exists on how to perform research and what rules should be considered when publishing a scientific paper. Despite this global agreement, in today’s “Cross Check Era”, too many authors do not give attention to the current standards of systematic research. Thus, the purpose of this paper is to describe these performance standards, the available choices for orthopaedic surgeons and the current learning curve for seasoned teams of researchers and orthopaedic surgeons with more than three decades of experience. These lead to provide an accessible overview of all important aspects of the topics that will significantly influence the research development as we arrive at an important globalisation era in orthopaedics and trauma-related research.

Introduction

As surgical techniques and learning curves advance, the need to address their respective clinical outcomes has heightened, especially as the majority of studies overlap. Therefore, the following important questions must be considered: Why do we perform research? When should we stop addressing the same topic already explored in other studies? How do we perform meaningful research that fulfills the requirements of patient care while matching our rapidly advancing surgical technology, clinical demands and changing health delivery environment?

Sophisticated clinical research must be performed to offer the foundation for functioning novelties that will improve the care of musculoskeletal patients.

The important issue concerning the relevance of the global expansion of orthopaedics-related research must be strategised more comprehensively.

Global insight into the diversity of research standards should be addressed and resolved with a comprehensive plan that considers the contrasting cultures that exist and the regulations that are enforced by different countries. Consequently, there is an obvious need to expand awareness of this overwhelming discrepancy.

Another issue for analysis concerns the importance of open access publishing, including whether to filter manuscripts prior to their publication, and into what order. Future resolutions may induce readers to be critical of studies that are submitted for open access publishing.

It will also be important to determine whether the current standard of filtering before publishing outweighs the pros and cons of publishing before filtering, as is currently the case for open-journal publications [1].

Another aspect to consider is whether the reader is able to understand the results of the study, which are determined and presented using complex statistical techniques. In other words, is the orthopaedic research too complicated to understand, or are the researchers attempting to persuade the reader with these data?

Within the orthopaedic community, there is immense pressure for research teams to publish even if the value of the study is questionable because goals are set to attain a certain number of publications rather than to focus on the quality or validity of the studies. Furthermore, there is a clear need to reform this new culture that rewards the quantity of publications instead of the quality of the research. Therefore, the overarching theme of this paper is to acknowledge the gold standard in implementation and publication of a research study. The main prerequisite for success within the orthopaedic community is to identify these trends before they become problematic while emphasising an ongoing commitment to quality in the orthopaedic field.

The peer review process is broken—why do we need it?

The peer review process usually reflects the ability of journals and editors to attract the best papers available. We need independent and “brave” editors who are able to publish papers that they believe will trigger a field that appears to be innovative in the future. These individuals should therefore be more independent from the possible citation consequences on the journal, which is an incentive to publish mainstream articles.

In the future, the impact factor (IF) will become less relevant. During this conversion, the creation of “sister journals” will draw attention to the main publication without affecting the main journal’s IF, i.e., the Journal of Bone and Joint Surgery (JBJS) Case Connector offers access to case reports that are not pertinent to be cited often [2]. Based on the above-mentioned factors, the incentives must be reset to quality rather than to quantity. However, this factor will influence research funding, which is mainly based on quantitative analyses of successful publications (few highly rated vs. many low rated papers). Even highly rated journals, which are always preferred by the authors, cannot assure that a specialised “breakthrough” will be identified by their review process. Therefore, a specialty-based rating would support individual progress, but because of the diversity of research, such a rating would make a uniform type of grant distribution impossible. The present system is far from optimum but offers relatively easy and general rules that are transparent to politicians, funding agencies, grant donors and even the general public.

Even with “citation” indices in place (e.g., the “H-factor”), there is currently no system that allows rating of the “progress” that an individual publication has already triggered, not even considering a rating that, in the future, might be triggered within the orthopaedic community.

Rating and weighting of research “value” is a difficult topic and, as history shows, frequently what is supposed to be “modern, mainstream research” will produce highly rated papers but does not necessarily guarantee the breakthroughs and scientific progress that would be expected by the investments in personnel, structure and financial resources. Therefore, considering the balance between independent thinking and long-term independence, which also facilitates long-lasting projects, even if the outcome is frustrating over a recent period, a more competitive organisation of research must be considered. Regarding new or small specialties and entities, possible progress might often be hindered by the applied universal rating systems (e.g., “Impact Factors”).

Okike et al. [28] analysed 1,181 manuscripts submitted to the JBJS between 2004 and 2005. The direction of study findings was graded by three different blinded reviewers. They found no indication of publication bias in the review of manuscripts, as positive and non-positive studies were accepted at analogous rates. The researchers indicated that the shortage of non-positive studies in the orthopaedic literature is cause for concern. To publish an article in a prestigious journal, the study methodology and level of evidence are essential. However, evidence indicates that some non-scientific factors may render a paper more likely to be accepted for publication. Factors associated with acceptance include disclosure of a conflict of interest, involvement of a non-profit entity, performance of the study in the same country as the journal in question, prior publication in frequently cited orthopaedic journals, and performance of the study in Canada or the United States. No significant association was found between acceptance and conflict of interest disclosure associated with a profit company, the sex of the corresponding author, or the author’s native language.

Positive outcomes account for 66 % of 301 articles and 75 % of 559 articles published in leading orthopaedic journals [25, 29]. As suggested by Lynch et al. [26], an unconscious affection/bias towards American and Canadian research, which tends to be more familiar to U.S.-based reviewers and editors, may lead to the preference of American- and Canadian-based studies. The authors also indicated that studies with negative outcomes are not accepted compared with positive studies, even when the negative studies having a higher quality.

The apparently preferential publication of positive studies is the reason why authors might be discouraged from submitting negative and non-positive studies for publication. In the future, the submission of more negative and neutral studies must be encouraged while still acknowledging possible rejection because of the direction of the study findings.

To protect the merit of science, institutions have developed rules for the ethical conduct of research [7]. In 2000 and 2003, the International Committee of Medical Journal Editors, the Council of Science Editors and the Committee on Publications Ethics developed ethical guidelines addressing several relevant issues [4, 5, 22]. Research misconduct, redundant publication, plagiarism, conflicts of interest and ethical approval for research were the main focus areas. In our understanding, ensuring a high level of international ethical standards for research warrants high-quality research that provides better evidence for the care of musculoskeletal patients, especially in conjunction with a higher level of evidence.

A retrospective study may provide important information, such as the outcome regarding fracture union and maintenance of alignment using a new surgical technique; a gold standard can only be replaced by a new technique when the new technique has been opposed in a randomised trial [35]. However, a good prospective cohort study should be as acceptable as a randomised trial in this context. We often encounter new plates or techniques that appear to be very promising, but they cost more than the gold standard and are employed because of their novelty, even though the previous method is equally effective and less expensive.

By submitting a paper, the author must always complete a statement to the editor regarding all the submissions and meeting presentations; the author should also list the results in registries and send the complete set of data to the editor. The author should inform the editor if the manuscript contains subjects about which the author has published a previous report or if a related report has been submitted to another journal.

Until a consensus is reached regarding a universal standard for publication review, there will be an accumulation of culture-based educational errors that will continue to propagate throughout the literature. Therefore, there is a clear need to set a universal standard for publication review that addresses the deficiencies of the current publication culture: a precise trend towards a reduction in the publication volume, a reduction in the fabrication of redundant data, an adequate peer review process, the selection and identification of relevant papers, a diversion of resources that are currently assigned to “guaranteed” or “safe” publications to “unguaranteed” or “unsafe” research projects and a conceptualisation of reward strategies for taking the risk of research projects that fail.

Relevant parameters in publishing and accepting a paper

As long as a peer review process exists, the identification of something novel that does not yet fit into any scheme and the subsequent identification of its potential future value represent the fundamental challenges of journals or publications that are not biased by tradition.

This is certainly an unsolved problem, as the “research rating” is based upon the “impact” of the publication. This consequence again implicates the type of research and the probability of acceptance in a highly rated journal. Therefore, there is increasing concern that high-risk projects will not be pursued on a routine basis, and the required “specialist interaction” is focused on “generalist journals” that are not capable of identifying “innovative ideas”.

Prevention of the recycling culture of research performance

To advance their academic career, orthopaedic surgeons face the challenge to publish frequently. The tradition of submitting redundant publications is a result of this pressure. Redundant publications alter the assessment of treatment effects in meta-analysis. The transition toward evidence-based orthopaedics research has led to an explosive increase in the number of meta-analyses over the last decade, which is cause for increasing concern [39]. The publication of redundant data is not only unethical but also has a confounding effect on meta-analysis; furthermore, the practice also increases the costs of peer review. Gwilym et al. [19] performed a study that quantified the severity of duplicate publication in the orthopaedic literature. The grade of duplication was based on a scale presented by Schein and Paladugu [33]. The authors selected original articles that were published in British and American issues of the JBJS in 1999. Medline was accessed to identify all publications between January 1996 and July 2002 and to classify those papers that were identified as duplications of the original JBJS article. They concluded that 26 out of 343, or one in 13 (7.6 %), of the original articles published in JBJS in 1999 exhibited some amount of redundancy [19]. Eck et al. [8] identified the rate of redundant publications in the orthopaedic literature. They searched 619 index articles published during the year 2000 in the JBJS (American volume) (127 articles), Journal of Orthopaedic Trauma (70 articles), Journal of Spinal Disorders (69 articles), and Spine (353 articles) using PubMed. The redundancy rate was four (3.15 %) of 127 for the JBJS (American volume), zero (0 %) of 70 for Journal of Orthopaedic Trauma, two (2.90 %) of 69 for Journal of Spinal Disorders, and 11 (3.12 %) of 353 for Spine. They concluded that the criteria implemented to categorise a publication as redundant were subjective. In this study, a publication was considered redundant only if there was close resemblance in the methodology, results, and conclusions. The redundancy rate might have been higher if the criteria had been more lax and included cases of “salami slicing.” These results implicate the publishing of multiple articles produced from a common core of patients or data [8].

Indeed, there are no specifically-defined criteria regarding what constitutes a redundant publication. A consensus should be reached among the publishers, editors and reviewers of scientific journals to define adequate criteria. Furthermore, disciplinary actions should be considered for violators of these standards.

Impact factors

The original purpose for establishing the IF was to endorse that current contents concealed the most substantial journals. The first published report on journal impact factors was presented by E. Garfield and I.H. Sher, entitled “New factors in the evaluation of scientific literature through citation indexing”, American Documentation, 14(3):195–201, July 1963. Later, E. Garfield and I.H. Sher coined the term ‘impact factor’ to support assessment and to select journals for current contents [13]. The IF of a journal reveals the frequency with which the journal’s articles are cited in the scientific literature. As indicated by Garfield [14], the IF is obtained by dividing the number of citations in year three to any items published in the journal within the two previous years by the number of substantive articles published in that journal during years one and two. Theoretically established in the 1960s, the IF is extensively recognised as a quantitative measure of journal quality [12].

The Institute of Scientific Information (ISI) in Philadelphia reviews approximately 2,000 new journal titles annually. Each journal undergoes an extensive assessment process (the basic publishing standard, editorial content, internationality of authorship and citation data), and only 10–12 % of the new journals assessed are selected. Journal IFs are considered as “objective quantitative measures of quality of output of individuals, research groups or universities” [23].

However, due to the uneven spread of article citations, the IF is not effective for analysing individual papers, individual scientists or groups. Indeed, the variability is too large to make IF an objective parameter for quality assessment.

Citation analysis is predisposed to self-citation, consistent in-house citation, competition and several other factors that influence the number of citations but are not associated with the quality of the research [34].

Although the application of the IF is deemed invalid, it nonetheless endures to some extent because no superior tool has yet been established and agreed upon. Citation-based indicators should be considered solely as a part of the larger peer review process in which scientific quality, productivity and significance are assessed by a group of independent experts [23].

In conclusion, we would prefer to reform the criteria of the peer review process rather than choose bibliometric indices as the instrumentation for assessing the quality of a scientific publication. The more relevant question would be how to improve the peer review process so that it stimulates research performance at a higher quality standard to implement better strategies in minimizing the growing global burden in treatment of musculoskeletal injuries.

Level of evidence

Since the era of Galen, medical practice has been built on empirical comprehension. However, the use of evidence-based medicine is also longstanding and has been taught for hundreds of years.

The combination of clinical assessment, experience and application of the literature helps physicians arrive at clinically relevant decisions. The current problem is that most orthopaedic surgeons overlook the clinical aspect of this combination and now depend only on what is dictated in the literature. This practice has become very evident in the widespread application of guidelines and quality measures that bear little relationship to the provision of good clinical care.

The term evidence-based medicine is more recent, appearing for the first time in a 1992 article [10]. Initially, criteria based on levels of evidence were classified internationally by the US Preventative Services Task Force, the National Health Service in Great Britain, and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group [17].

Currently, the evidence ladder varies from meta-analyses of high-quality prospective, randomised, controlled trials (Level I) to case-control studies (Level III), case series and case reports (Level IV) to expert opinions (Level V) [10].

Indeed, observational studies indicated decreased rates of ischemic heart disease among women using hormone replacement therapy, and this study was designated as supporting hormone replacement for healthy women, women with confirmed ischemic heart disease, and women with risk factors for ischemic heart disease. However, randomised controlled trials performed by Rossouw et al. [31] concluded that hormone replacement therapy augmented the risk of ischemic heart disease, indicating that the deceptive protective effects observed in observational studies were due to bias.

Studies such as this demonstrate that medical interventions based merely on observational data should be judged carefully [30, 31]. Today, there is anticipation that a physician’s verdict should be based on evidence of the highest quality [17, 36].

Obremskey et al. [27] reviewed 382 clinical articles published over a six-month period in nine orthopaedic journals and classified them based on the JBJS (American volume) level of evidence method, ranging from I to IV. They found that 11.3 % of the sample were Level I studies, whereas 58.1 % were Level IV studies. They concluded that their article “exposes a well-known weakness in the orthopaedic literature, which is its tendency to contain retrospective studies at a lower level of evidence.” Unconditionally, Obremskey et al. argue that all studies should be Level I. However, most classic and frequently cited articles do not reflect the highest quality, and the level of evidence ratings are linked to the frequency of citations.

The level of evidence is only one aspect of study quality that necessitates a detailed critical assessment. As indicated by Dr. J. Ioannidis, there are significant errors in many published studies. He also noted that all level IV studies will eventually be proven incorrect, and almost 40 % of randomised clinical trials will be revealed as invalid [20, 21].

Meanwhile, the level of evidence terminology has become a broadly used metric to assess the quality of orthopaedic research. However, the status of orthopaedic studies is not precisely presented by the average level of evidence in the articles but rather by the deficit, if any, between the methods applied in the published literature and the methods that should have been applied. Based on this notion, Baldwin et al. [1] presented the concept of the level-of-evidence gap: “the difference between the level of evidence that should have been used to answer the primary research question of the study and the level of evidence that was actually used in the study.” The authors reviewed 64 JBJS-Am articles. The overall level of evidence was between II and III, indicating a deficit of approximately 2, relative to the assumed ideal of Level I.

The perception of evidence-based medicine is that scientific education regarding the disease and the treatment efficacy will facilitate high-quality and cost-effective treatment.

High-quality research performance is expensive, and creating quality evidence remains challenging. At the international level, the orthopaedic community is not opposed to evidence-based surgery; however, we are aware that the evidence in most cases is not yet obtainable and will require time to sufficiently accumulate.

A systematic review is a broad review of the literature, whereas a meta-analysis contains the supplementary step of statistically combining studies. Thus, it is imperative that only the highest-quality studies be integrated into meta-analyses. The quality of meta-analyses and systematic reviews has differed considerably, particularly in the orthopaedic literature [6]. Well-conducted meta-analyses provide a more objective assessment of the evidence than do conventional reviews; moreover, meta-analyses improve the accuracy of approximating a treatment’s effect and may help to explain differences between the results of the various studies (heterogeneity) [9].

Meta-analyses that merge the outcomes of high-quality randomised trials are considered to be Level-I evidence [15, 16, 18].

According to Wright et al. [37], many meta-analyses considerably overlap in content, including the articles that are reviewed and the conclusions that are presented. The authors stated that this overlap increases apprehension about redundant publication and is confusing for readers. Based on their findings, the authors generated a consensus, which includes identifying all meta-analyses or systemic reviews that are published on the same topic within the first five years, rejecting meta-analyses or systematic reviews on the same topic published within five years, rejecting papers that are used to achieve the same findings, accepting only summaries of randomised trials, and providing adequate homogeneity of the inclusion and exclusion criteria that are considered suitable for meta-analysis.

According to Bhandari et al. [3], meta-analyses in orthopaedic surgery have been determined to have inadequate scientific quality. In a review of 40 meta-analyses published between 1984 and 1999, 33 % had major methodological errors, and more than 50 % were published after 1994. The researchers concluded that 88 % of the meta-analyses had methodological errors that could diminish their validity. The main deficiency was the absence of information on the methods used to analyse the validity of the primary studies. Bhandari et al. concluded that the meta-analyses with lower scores for quality involved positive findings. Meta-analyses of fracture treatment and degenerative disease (hip, knee, or spine) had significantly lower mean quality scores than did meta-analyses that focused on thrombosis prevention and diagnostic tests (p < 0.05). The authors concluded that limiting bias and upgrading the validity of the meta-analyses might be achieved by devotion to strict scientific methodology.

However, the ultimate quality of a meta-analysis is based on the quality of the primary studies on which it is built. A meta-analysis is most credible when data from high-quality randomised trials are collected [3, 32].

Dijkman et al. [6] reported a dramatic rise in the number of orthopaedic-related meta-analyses since 1984, with fewer methodological errors in those meta-analyses that were published in 2005 and 2008. Their review was enhanced by a decisive primary question, stringent eligibility criteria, a systematic approach (including a comprehensive search strategy), duplicate assessment of the methodological quality, assessment of agreement among reviewers, and application of a validated measure to evaluate the “methodological rigour of the meta-analyses”. The researchers indicated that the significance of meta-analysis in orthopaedics can only be presented with high-quality methods focusing on relevant clinical questions. The scope that randomised trials maintain to expand recognition stimulates the on-going increase of meta-analysis as a decisive methodology in the practice of evidence-based orthopaedics [10, 38].

The globalisation of orthopaedics

Historically, the language barrier has hampered the communication between various orthopaedic communities around the globe. However, now that English is recognised as the international language of science, this barrier has been removed. Sir John Charnley’s innovation in total hip arthroplasty was initiated in Great Britain. A Japanese surgeon, Watanabe, presented a new technique to examine the inside of a knee with a video camera. Dubousset and Cottrell, French surgeons, introduced a modified instrumentation for spinal deformity.

In addition to the import of overseas innovations, international orthopaedic societies now exist. These societies function as an outstanding medium for the exchange of ideas between researchers and clinicians at an international level. In late 1940, the random development of fracture fixation and implants emerged. At that time, the biology of bone healing had not been explored, and information was scant. In 1949, a Belgian surgeon, Robert Danis, published a book entitled Théorie et Pratique de l ’Ostéosynthèse, which was his second book on fracture fixation and presented his concepts of early functional rehabilitation following rigid fracture fixation.

The goal of the Arbeitsgemeinschaft der Osteosynthese (AO) pioneers was to evaluate scientifically the application of surgical fixation to patient care and to improve surgical procedures so that the outcome for the injured patient could be maximised. In 1984, the necessity for improvements in fracture fixation and patient care led to restructuring of the AO as an organisation that currently oversees scientific research, development, education and international endeavours. The mission of AO, which has 6,900 AO-Trauma members and 4,500 AO-Trauma faculties worldwide, is based on achieving highly effective patient care. The dominance of American practice in orthopaedic surgery has been well established. The American Academy of Orthopedic Surgeons (AAOS) has the largest number of members and the largest number of foreign members (http://www3.aaos.org/About/diversity/index.cfm). Of the top 20 orthopedic journals with the highest IFs, 17 are American. There is a much higher volume of operative procedures in the US compared to other countries, and the application of expansive implants is redundant [24].

The International Society of Orthopaedic Surgery and Traumatology (SICOT), the European Federation of National Associations of Orthopedics and Traumatology (EFFORT), the Asia Pacific Orthopedic Association (APOA), the AAOS, the AO, participation in the World Campaign of the Bone and Joint Decade, the exchange of international students and fellows and assistance for young orthopaedic surgeons from outside the United States and Europe participating in our academic conferences and environments will present a positive prospective strategy for the globalisation of each orthopaedic culture worldwide.

All of these multinational tasks will only be accomplished if we take the next step in globalisation [11]. This step entails the abolition of nationality-based bias. Readiness to critically assess innovations and projects from other countries and to acknowledge applications that have been verified to be effective would be advantageous to musculoskeletal care.

A further crucial aspect that should be taken into account is that there is marked variation in awareness about plagiarism among scientists from different cultural backgrounds. Most of the Eastern cultures consider knowledge to be “common property”, which is free to be used by anyone; hence, the routine acknowledgment of other individuals for supplying information (i.e., intellectual property) for further application is not deemed essential. Plagiarism and other forms of academic misconduct are ubiquitous to academic fields in various countries, but to different degrees.

Any debate regarding plagiarism necessitates determination of whether the context under which “plagiarism” is conceived in American academia is the invention of a particular cultural history rather than one that is universally common. It may be inadequate to designate this problem as a difference between “western” and “non-western” cultural values. Nevertheless, academicians who have grown up in a non-U.S. academic environment may have different ideas of intellectual property rights. With the tendency to more globalisation, these issues are of great concern with regard to publishing papers.

Summary

Research has been defined as a coin with tribute on one side and responsibility on the other. Choices regarding the treatment of orthopaedic disorders are usually based on accessible information contained within the published literature. Hence, orthopaedic journals are obligated to select manuscripts based on research quality and scientific originality. Patient care may be compromised if the integrity of the literature is disregarded. Scientific tenure is the paramount indicator of success, and reputation is the essential platform of exchange. Scientific quality should be the main factor in publishing a paper, not author identity, country-based associations, or financial entities. Improving the quality of orthopedic research will lead to advancements in patient care. Our international group aims to implement a professional publishing culture based on standards that will identify the violation of gold-standard publishing and identify options to improve the quality of research to meet future challenges.

Despite our great advances in research, validity is still clearly lacking in the accessible studies. Consequently (and unfortunately), the majority of cases do not present a clear, documented treatment choice; rather, we still depend upon the opinions of surgeons with special expertise that extend beyond the boundaries of research standards and guidelines. To encourage an optimal culture of research performance, orthopaedic organisations in leadership positions need to explicitly support a standardisation of the peer review process while inspiring and rewarding those researchers who can harvest new information rather than recycled evidence. Considering the globalisation of the field of orthopaedics, reforming the peer review process on an international scale will be the ultimate challenge to improve the quality of research and realise a future that includes better patient care worldwide. In conclusion, we strongly believe that international exchange, cooperation and standardised reformation of the peer review process will one day allow all orthopaedic clinicians to deliver superior and equivalent medical treatments to their patients, regardless of their geographic location.