Evidence-based surgery – evidence from survey and citation analysis in orthopaedic surgery

Abstract

INTRODUCTION

The results of a survey on evidence-based surgery (EBS) among members of the American Academy of Orthopedic Surgeons (AAOS) and the British Orthopaedic Association (BOA) are presented. The study also analyzes the citations earned by articles with different levels of evidence (LOE) to see if LOE have any bearing on the importance attached to the articles by authors and contributors to the journals.

SUBJECTS AND METHODS

The questionnaire was e-mailed to 1000 randomly chosen consultant orthopaedic surgeons who were members of either the AAOS or the BOA. Participants were provided with the option of responding through web-based entry. For citation analysis, citation data were gathered from the Journal of Bone and Joint Surgery (American volume) between the years 2003 and 2007 (5-year period).

RESULTS

The survey showed that awareness and access to EBS have improved greatly over the years. At the present time, these factors are not important barriers to the implementation of EBS in clinical practice in developed countries. There was a statistically significant difference in those with and without additional qualifications with regard to the approach to EBS. However, an equal percentage of surgeons with and without additional qualifications felt that it was difficult to adhere to EBS guidelines in daily clinical practice. Citation analysis showed that readers of professional journals attach importance to LOE category of the article and tend to cite level-I evidence articles more than other articles.

Advocates of evidence-based surgery (EBS) seem to be certain that it is a pragmatic concept that ought to be universally embraced for the benefit of both patients and orthopaedic science in general.¹ Many beliefs and customs exist in orthopaedic surgery for which there is not much evidence.² Sackett et al.³ defined evidence-based medicine as ‘the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients'. To make such use of available evidence, it is important to be able to assess the available information critically. Surgeons do not seem to implement new evidence immediately (especially those involving new procedures), even if it is available. They prefer to wait for trusted and influential leaders in the community to pronounce their verdict about the new knowledge.⁴ Levels of evidence were thought to help busy orthopaedic surgeons acquire the most relevant valid information in the shortest possible time and with the least possible effort.⁵

Subjects and Methods

This was a ‘cross-sectional’ survey among 1000 randomly selected members of American Association of Orthopedic Surgeons (AAOS) and the British Orthopaedic Association (BOA). A questionnaire was designed, partly based on the patterns of the questionnaires used for earlier surveys on evidence-based practice in the specialty of orthopaedics and in other branches of medicine.^6-10 The survey focused on access, practical applicability, future concerns and the attitude towards EBS. The questionnaire was in a single web page containing 31 items. Details of the survey design are given in Table 1. AAOS and BOA were selected as they include a large number of orthopaedic surgeons who are familiar with EBS. The e-mail addresses were obtained from the AAOS and BOA directory of members.

Table 1.

Parameters included in survey design

Parameters	Yes	No
Log file analysis	*
Unique site visitor identification (using Traffic backend stat counter)	*
Incentive for participation		*
Screening by institutional review board	*
Pre-testing of questionnaire	*
Adaptive questioning		*
Completeness check (using My SQL database)	*
Randomisation of items in questionnaire		*
Use of online survey generator		*
Randomisation of participants (computer generated random numbers)	*
Prior consent from participants		*
Advertisement regarding survey		*
Data protection	*
Neutral response option	*
Review of response (prior to submission)	*

This was a double-blind survey done by a software professional without any involvement in the study. The questionnaire was e-mailed directly to the participants and provision was made for web-based entry (<http://www.ebossurvey.com>). Data were collected over 6 months from the date of first mailing. Subsequent to first mailing, four reminders were sent at intervals of 4 weeks. Participation was voluntary. The purpose of the survey was explained to the participants. The web-site was designed such that multiple entries from a single responder (single IP address) was not possible. The survey has been reported according to the ‘CHERRIES statement’ on web-based surveys.¹¹

Citations are regarded as indicators of the significance and relevance of articles following publication.^12,13 Evidence-based classification of an article can be considered as a tool to assess the quality of the article prior to publication in the journal. The electronic version of the Journal of Bone and Joint Surgery (American volume) publishes citation data along with the articles and citations were counted using that data base. To exclude the effect of journal self-citations, citation rates were also calculated after excluding self-citations. Levels of evidence was introduced in the J Bone Joint Surg Am in 2003 and we have gathered the citation data between 2003 and 2007 (5-year period), allowing a minimum of 2.5 years for accumulation of citations for articles published in 2007.

Results

Results of the survey

A total of 658 orthopaedic surgeons responded (response rate of 65.8%) and the responses were complete in 94% of questionnaires. Of respondents, 82.3% were in the age group 36-65 years and 65.8% were employed in non-teaching hospitals or in private practice with the remaining 34.2% in teaching hospitals. Of respondents, 18.3% had additional qualifications such as PhD or MPH, while the rest did not have additional qualifications that indicate higher research experience. The results of the significant aspects of the survey are summarised in Table 2.

Table 2.

Summary of the overall results of the survey (values rounded to nearest whole number)

Ignore the recommendations of new level-I evidence	2%
Discuss with colleagues before adopting new level-I evidence	12%
Reject article recommendation with low level of evidence	14%
Never changed practice based on EBS recommendation	17%
Always changed practice based on EBS recommendation	18%
Had time for detailed reading of journals	23%
Easy to adopt EBS guidelines in practice	31%
EBS highly useful	36%
Concerns about restriction of freedom as a clinician	45%
Acceptance of article recommendation in spite of low level of evidence	48%
EBS occasionally useful	52%
Fully familiar with EBS	53%
Comfortable doing online EBS search in the out-patient clinic	62%
Comfortable with online EBS search during ward rounds	63%
Concerned about patient insistence on treatment based on EBS	64%
Difficult to adhere to EBS guidelines in practice	69%
Concerned about potential for abuse of EBS by lawyers	76%
Had time only for selective reading of journals	77%
Impression about article influenced by level of evidence classification	78%
Concerned that new treatment procedure may be denied funding citing low LOE	79%
Concerned that research funding may be denied citing low LOE	81%
Circumspect about financial conflict of interest even in level-I evidence	82%
Feel that EBS culture should be promoted	85%
Willing to alter practice immediately in accordance with new level-I evidence	86%
Familiar with EBS concept	92%
Had access to EBS literature	99%

For a few specific questions, we compared the responses of the participants as a whole with the responses of the participants with and without additional academic qualifications (PhD or MPH). The results are summarised in Table 3. In this study, it was assumed that possession of higher qualification was evidence of a higher degree of familiarity with research methodology in general. Of those with additional qualifications, 89.8% said that they were comfortable with the bio statistical vocabulary used in the journals as opposed to 72% of those without additional qualifications. The difference was statistically significant (P = 0.00007, chi-square test). Of those with additional qualifications, 46% had received formal training in EBS literature search compared to 28% in the group without additional qualifications. The difference was statistically significant (P = 0.000098). In the additional qualification group, 68.8% said that it was difficult to adhere to EBS guidelines in daily practice compared with 69.4% in the other group. The difference was statistically not significant (P = 0.7356). Interestingly, only 21.2% of those in the additional qualification group said that they would alter their practice immediately if the conclusion of a level-I study was different from their own experience. This is in contrast to the response of the overall group where 85.6% were ready to alter their practice immediately upon the recommendations of a level-I study. The difference was statistically significant. The vast majority in the additional qualification group preferred to discuss with their colleagues before altering their practice.

Table 3.

Comparison between additional qualification group and general group

	General group	Additional qualification group
Comfort with bio-statistical vocabulary	72%	90%
Formal training in EBS literature search	28%	46%
Difficulty in adhering to EBS guidelines	69%	69%
Speed of response to level-I evidence	86%	21%

Results of citation analysis

The data regarding the number of articles, total number of citations and citations after excluding journal self-citations are given in Figures 1 and 2. The average citations per article was 4.9 for level-I articles, 4.0 for level-II articles, 3.7 for level-III articles and 3.4 for level-IV articles. The average citation per article after excluding journal self-citations were 2.5 for level-I articles, 1.7 for level-II articles, 1.3 for level-III articles and 1.4 for level-IV articles. Journal self-citations were significantly higher in level II, III and IV categories (P < 0.001, chi-square test). In level I category, external citations were significantly higher (P < 0.001).

Figure 1.

Number of studies in each level of evidence per year between 2003 and 2007.

Figure 2.

Total citations and journal self-citations for each level of evidence.

Discussion

Criticisms of evidence-based medicine are largely based on epistemology and ethics. Epistemologically, it argued that the medical findings are often subject to subsequent dis-proval and the ‘evidence’ does not withstand the test of time.^14-16 The quality of evidence is affected by the quality of research on which the evidence is based.^17-19 There is insufficient evidence that implementation of EBM has led to better clinical outcome.^18-20-23 Concerns have been expressed in the past regarding familiarity and ease of access to EBS literature and practical applicability of EBS. Ethical concerns include the use of evidence-based medicine to deny life-saving emergency services to areas which truly required them,²⁴ loss of therapeutic freedom of the clinicians,²⁵ denial of funding to clinical researchers on the grounds of lack of evidence,²⁶ and exploitation of EBM data by pharmaceutical companies to market products aggressively.²⁷ Our survey was designed to assess the prevalence of these concerns among surgeons. It was aimed at orthopaedic surgeons, but the results have relevance to other branches of surgery as well.

Results of the present survey indicate that the majority of orthopaedic surgeons (around 92%) had basic level of awareness of EBS. Also, 99% of surgeons had access to EBM literature; this is higher than earlier surveys.^6,7,10 This indicates that access to EBM literature and web-based resources has improved over the years and cannot be considered as major impediments in the implementation of EBS. Supporters of EBS have suggested that surgeons should check the evidence on the web in their out-patient clinics and by the patient's bed side when deciding the best course of action. Hand-held devices were useful to provide surgeons with ‘real-time access’ to the internet when confronted with difficult clinical situations.²⁸ In our survey, more than 60% of surgeons were willing to access the web for EBS literature in out-patient clinics or ward rounds. In the past, it was thought that doctors in peripheral hospitals may have had difficulty in accessing information regarding EBS.²⁹ However, access to EBS information has improved. There was a statistically significant difference in the level of comfort with biostatistical vocabulary between those who had additional qualifications and research experience and those who did not. This could be partly due to the fact that significantly higher proportion of those with additional qualifications had received formal training in EBM. Of the participants in our survey, 85% felt that EBS should be promoted. Nearly 86% of surgeons said that they would change their practice based on level-I evidence whereas only around 47% said that they would do so on the basis of literature lesser than level-I evidence. This shows that level of evidence is considered important by a significant number of surgeons. The results are similar to those of previous surveys on evidence-based orthopaedic surgery.^8-10

In spite of the enthusiasm for EBS, most surgeons (69%) felt that there are limitations to the application of EBS in practice. This is similar to the survey by Poolman et al.,⁹ wherein only 20% of the participants used EBM information in clinical decision making. Since familiarity and access to EBS are no longer major issues, efforts to improve the practical application of EBS should focus on formal training of surgeons in EBM methods. In the survey by Hanson et al.,⁸ 90% of participants agreed that there was a need for training surgeons in research methodology. In our survey, 90% of participants with additional qualifications were comfortable with the biostatistical vocabulary when compared with 72%, who did not have additional qualification. Poolman et al.⁹ reported that younger age, possession of PhD degree and working in academic institutions were associated with better comprehension of EBS. In this survey, those in the additional qualification group were actually more reluctant than the overall group in incorporating the recommendations of a level-I study when their own experience was otherwise. Familiarity with the EBS methods may increase the awareness of its limitations too. Even when there is familiarity, access and formal training in EBM methods, it may be difficult to apply EBS to an individual patient due to socio-economic reasons, facilities available or due to the choice of treatment by the patient. EBM ignores the benefits of clinical experience.⁷ EBM recommendations apply to patient cohorts and may not resolve the dilemma of a clinician faced with an individual patient.^32,33 In such situations, clinical judgement may be more appropriate than adhering to evidence-based guidelines.³⁴

Attitudes of the authors towards levels of evidence can be gauged by analysing the citations. In the present study, average citation rate decreased from 4.9 per article for level-I articles to 3.4 per article for level-IV articles. Even when journal self-citations were discounted, the average citation rate for level-I articles (2.5/article) was still higher than the rate for level-IV articles (1.4/article). While there was no statistical significance between overall citation rates of articles belonging to different LOE categories, the difference between categories became highly statistically significant once journal self-citation rates were excluded from calculation. This suggests that level-I studies were considered important by contributors of other journals in the same or related fields. This is a positive trend in the EBS culture. We have not found similar data in the literature to compare our results. The drawback of using citation as a marker of the quality is that citation based metrics such as ‘impact factor’ have numerous limitations.^30,31 More citations for articles with higher levels of evidence do not necessarily imply that the quality of the studies were very high. It is, perhaps, just a reflection of the perception of other authors about LOE.

Apprehensions regarding the possible abuse of EBS (by lawyers, patients and funding authorities) are present among majority of orthopaedic surgeons. Nearly 76% of the surgeons felt that EBS concept could be exploited by lawyers who may argue that patient was harmed by treatment based on lower levels of evidence. Around 64% felt that patients could become more demand treatment based on LOE. It remains to be seen whether these apprehensions translate into realities. Readers seem to be cautious in accepting the recommendations of even level-I studies in the presence of financial conflicts of interest. Nearly 82% of the respondents in our study stated that they would be circumspect regarding the recommendations of such studies even if the level of evidence is high. Commercial sponsors may exploit the clinicians’ implicit trust in evidence-based medicine to publish reports favourable to their products.²⁷ Previous surveys have not addressed these issues but the issues are important enough to attract further studies. Over 80% felt that authorities could deny them research funding citing levels of evidence or restrict their choice of research topics. Some 45% felt that clinician's freedom in implementing treatment options may also be restricted. The responses received in our survey seem to add to the growing list of concerns regarding EBM in the recent times.

Conclusions

Is EBS going to be the hallowed ground supporting the edifice of surgical science in future? Or is it a pedantic trend that will make way for some other trend in the future? Time and further research hold the answers. However, studies including ours seem to indicate that most orthopaedic surgeons welcome the EBS culture and the level of awareness and access to EBS have improved significantly. Practical application seems to the key issue and this can probably improve with formal training. Our survey has brought to the attention of the community certain hitherto unaddressed concerns such as possible abuse of the EBS by lawyers and patient community. It has also shown that articles with high levels of evidence seem to enjoy a certain citation advantage. It remains to be seen whether these higher citations would eventually turn out to be sign-posts of high quality research.