Role of Evidence-Based Reviews in Surgery in teaching critical appraisal skills and in journal clubs

Abstract

Background

Evidence-Based Reviews in Surgery (EBRS) is a program developed to teach critical appraisal skills to general surgeons and residents. The purpose of this study was to assess the use of EBRS by general surgery residents across Canada and to assess residents’ opinions regarding EBRS and journal clubs.

Methods

We surveyed postgraduate year 2–5 residents from 15 general surgery programs. Data are presented as percentages and means.

Results

A total of 231 residents (58%, mean 56% per program, range 0%–100%) responded: 172 (75%) residents indicated that they know about EBRS and that it is used in their programs. More than 75% of residents who use EBRS agreed or strongly agreed that the EBRS clinical and methodological articles and reviews are relevant. Only 55 residents (24%) indicated that they used EBRS online. Most residents (198 [86%]) attend journal clubs. The most common format is a mandatory meeting held at a special time every month with faculty members with epidemiological and clinical expertise. Residents stated that EBRS articles were used exclusively (13%) or in conjunction with other articles (57%) in their journal clubs. Most respondents (176 of 193 [91%]) stated that journal clubs are very or somewhat valuable to their education.

Conclusion

The EBRS program is widely used among general surgery residents across Canada. Although most residents who use EBRS rate it highly, a large proportion are unaware of EBRS online features. Thus, future efforts to increase awareness of EBRS online features and increase its accessibility are required.

Evidence-Based Reviews in Surgery (EBRS) is a joint program of the Canadian Association of General Surgeons (CAGS) and the American College of Surgeons (ACS) and was developed to teach critical appraisal skills to practising surgeons and residents.¹ Monthly packages, which consist of a clinical and a methodological article, are available during the academic year between October and May. Participants are asked to critically appraise the clinical article using the methodological article as a guide. The EBRS program is accessible online, with participants discussing both the clinical and methodological aspects of the article with their peers on a listserv. The EBRS articles are also distributed to all general surgery residency programs; they can be discussed in a journal club format or residents can access and complete the packages electronically on their own time.

Teaching critical appraisal skills has become a mandatory component of surgical residency curricula in many countries, including the United States and Canada. Studies have shown substantial gain in knowledge after teaching critical appraisal skills to medical school graduates.^2,3 Workshops and journal clubs are often used to teach critical appraisal skills to residents and have been shown to be effective.^4–7 Critical appraisal skills can be particularly valuable to residents training in surgical residencies.^7–9 The EBRS program enables residents to learn critical appraisal skills while evaluating articles relevant to general surgery. Previously, the program has been shown to be effective in improving critical appraisal skills among general surgeons.¹⁰ Many general surgery training programs across Canada have adopted EBRS to teach critical appraisal skills to residents, and program directors have rated it positively. However, it is not clear how EBRS is currently being used by general surgery residents and how it can be improved. We surveyed all general surgery residents across Canada to assess their awareness of EBRS and evaluate their opinions regarding different aspects of the program. We also aimed to assess residents’ participation in journal club activities and their attainment of critical appraisal skills. Our results will enable future modifications of the program to better target the needs of the general surgery trainees.

Methods

The program directors of the 17 general surgery programs across Canada were contacted by letter in August 2010 to explain the survey and obtain a list of postgraduate year 2–5 residents in their programs. Fifteen of the programs’ directors agreed to participate in the study. We developed packages containing a cover letter explaining the study and inviting each resident to participate along with a 28-item survey and individually addressed them to each resident in the 15 programs. We then sent the packages to the program director or assistant, who in turn distributed and collected them from the residents after the survey was completed. Generally, surveys were completed during residents’ academic half-day. The programs couriered the survey packages back to us.

The survey assessed residents’ critical appraisal training, their participation in journal clubs, the use of EBRS in their journal clubs, their use of EBRS outside of journal clubs and their opinions about the value of journal clubs and EBRS. The survey was anonymous and no identifying information was collected. Statistical analysis was performed using SPSS version 16.0 (SPSS).

The Mount Sinai Research Ethics Board approved our study protocol.

Results

The survey was distributed to 400 general surgery residents, of whom 231 (58%) completed the survey. On average, 56% (range 0%–100%) of the residents from each program responded. Forty-three (19%) respondents indicated they had a graduate degree (MSc, PhD).

A total of 172 (75%) residents indicated that they know about EBRS or that EBRS is included in their program’s curriculum. Of those, 93 (54%) residents recalled completing at least 1 of the monthly packages during the 2009–2010 year. Of these 172 residents, 101 (59%) rated the value of the EBRS packages. Overall the residents rated the EBRS articles and reviews highly, as shown in Table 1.

Table 1.

Residents’ rating of EBRS articles and reviews

	Response; %
Statements about EBRS	Strongly agree	Agree	Neutral	Disagree	Strongly disagree
The clinical articles are relevant	21	58	20	1	0
The clinical articles are interesting	18	62	19	1	0
The clinical articles have scientific validity	17	67	15	1	0
The methodological articles relevant	12	64	21	3	0
The methodological articles are easy to understand	15	53	28	4	0
The clinical reviews are relevant	24	57	18	1	0
The clinical reviews address current status of practice	27	58	14	1	0
The methodological reviews are relevant	17	58	24	1	0
The methodological reviews clarify methodological issues	20	50	27	3	0

EBRS = Evidence-Based Reviews in Surgery.

Of the 172 residents, only 55 (32%) residents stated that they access EBRS online. Of those who use EBRS online, 25 (46%) reported accessing EBRS online once every 3 months, 16 (29%) access it once every month, 2 (4%) access it twice per month, 1 (2%) reported accessing it once a week and 11 (20%) reported other frequencies. Nineteen (35%) residents who access EBRS online reported the listserv discussions to be worthwhile; however, only 7 (13%) reported participating in the listserv discussions. Twenty-three (42%) residents indicated that a listserv discussion exclusively for residents would be of interest to them. When asked about what alternative tools they would use to access and/or discuss EBRS, 28 chose a smartphone app, 17 chose a podcast, 6 chose a Facebook group, and 5 chose an online discussion forum.

When asked about other features of EBRS, only 16 of 123 (13%) respondents stated they use the archives section of EBRS, with frequencies ranging from a few times a month to a few times a year. However, of those who do access the archives section, 14 found it easy to access, and all 16 found the archives section helpful. Only 18 of 112 (16%) residents reported using the free access offered by EBRS to various journals to download journal articles.

Role of EBRS in journal clubs and critical appraisal training

Overall, 198 (86%) respondents indicated that they attend journal clubs. Of those, 151 (76%) indicated that journal clubs are held at a special time (e.g., evenings), 34 (17%) indicated that they are held as part of scheduled rounds or conferences and 11 (6%) indicated that they are part of the educational half-day. When asked about the frequency of journal club meetings, 143 (73%) residents stated they attend journal clubs once a month, 5 (3%) twice a month, 6 (3%) every week and 42 (21%) indicated other frequencies (every 2 mo, every 4 mo, every 6 mo or at random times). One hundred twenty-three of 178 (69%) respondents indicated that attendance at journal club is mandatory. As shown in Table 2, about two-thirds of residents stated that faculty members with clinical and methodological expertise attend journal club. A smaller proportion of journal clubs are attended by other faculty, fellows, off-service residents and medical students. With regards to the value of journal clubs to residents’ education, 176 of 193 (91%) reported journal clubs as either very valuable or somewhat valuable to their education.

Table 2.

Participants in journal clubs, as indicated by residents

Participants	Responses; no. (%)
At least 1 faculty member with clinical expertise	136 (69)
At least 1 faculty member with methodological expertise	127 (64)
Medical students	90 (45)
Fellows	59 (30)
Off-service residents	43 (22)
All general surgery faculty	40 (20)
Other	4 (2)

Twenty-six of 198 residents (13%) indicated that they use EBRS articles exclusively in their journal club meetings, 112 (57%) indicated that they use EBRS articles in conjunction with other articles and 44 (22%) indicated that EBRS articles are not used in their journal club meetings. Sixteen residents (8%) did not answer this question.

When asked about the use of EBRS in critical appraisal training, 135 of 221 (61%) residents indicated that they only use EBRS to learn critical appraisal skills, while 85 of 218 (39%) reported they use or have access to other programs to learn critical appraisal skills. Examples of the reported programs included critical thinking courses, academic half-days, principles of surgery sessions, rounds or other teaching sessions.

Discussion

Recognizing that skills, which assist in lifelong learning, are necessary for all clinicians; critical appraisal skills are now a mandatory component of residency curricula. Critical appraisal skills equip residents with the skills necessary to evaluate and assess the value of advances reported in the medical literature. A group of surgeons interested in evidence-based medicine initiated EBRS under the auspices of the CAGS in 2000. The program was developed to provide critical appraisal training for residents.¹ Subsequently, at the urging of the CAGS leadership, EBRS has been made available to all CAGS members and has been shown to be effective in teaching critical appraisal skills.¹⁰ Also, EBRS continues to be provided to all general surgery programs in Canada and the United States free of charge.

Previous surveys have shown that most residency program directors in Canada find EBRS to be a useful addition to their teaching programs. However, its use and perceived value among general surgery residents have never been assessed. We sought to determine what features are valued and whether modifications could improve its value to general surgery trainees. The EBRS articles are usually discussed in a journal club format; therefore, in this survey we chose to assess opinions about journal clubs as well as the use of EBRS.

The response rate to our survey was 58%. Generally the results of the survey were positive. About 85% of participating residents stated they attend journal clubs and 75% indicated that they know about EBRS and/or use it in their programs. Of those who responded, about 80% stated they use the EBRS articles in their journal clubs although most do not use them exclusively. Most respondents (61%) also stated that they receive critical appraisal training primarily through EBRS. Furthermore, of those who were aware of EBRS, 70%–85% agreed or strongly agreed with statements about the value of the clinical and methodological articles and reviews (Table 1). On the other hand, some responses suggested that residents were unaware of EBRS and some of its features.

Although 75% of residents stated that they were aware of EBRS, only 54% recalled completing a package in the last year. As well, from comments provided by the residents in the survey, it appeared that some were not aware that the articles used in their journal clubs were EBRS articles and that perhaps they thought that the articles were prepared by their own program. Thus, from the program standpoint, it suggests that we need to brand EBRS better. This is important not only to advertise EBRS but also to make residents aware that they can access articles and participate in the discussions even if their residency program does not use the EBRS in their teaching curriculum. Moreover, it is important that residents know about the other features on the website and how to access them even after completing their residencies. Early exposure to evidence-based medicine has been shown to have significant effects on future clinical decision-making.¹¹ The EBRS program aims to familiarize surgery residents with evidence-based medicine concepts and to be a resource for lifelong learning, maintenance of certification and recertification; thus, it is important that residents are aware of it before they go into practice.

Another concern is that only one-third of the respondents access EBRS online, and those who do report that they access the EBRS website infrequently. Only 7% stated that they participate in the listserv discussion group and only 13% stated that they use the EBRS archives section. Again, this may be owing to poor promotion of the EBRS program and its features. Over the last decade and with the advance of technology and availability of online resources, numerous online tools have been created and used in resident training across various specialties. Examples of such tools include custom-built online curricula, resident curriculum portals, online portfolio systems, physician-only social network websites and the use of existing social media (e.g., YouTube, Facebook).^12,13 Providing the EBRS packages online and setting up a listserv discussion group for residents might be of value to residents and might be a way to increase participation; however, only 23 residents felt this would be of value to them. As well, in a previous randomized controlled trial in which our group compared the effectiveness of using EBRS online versus in a journal club format, we found that participation in an online version was low.¹⁴

On the other hand, with the emerging technology, newer and more creative methods of delivering information, such as smartphone apps or podcasts, may be more effective in reaching more users. Our results showed that a smartphone app or podcast were the most popular alternative means of accessing EBRS among residents. The importance of such alternative methods of accessing information for medical trainees has been established by several other authors in various specialties.^15–17 Therefore, future efforts can be targeted at better educating the residents about the various online features of EBRS and providing residents with alternative methods of accessing EBRS to enhance utilization.

While EBRS teaches critical appraisal skills in the context of relevant clinical articles, the primary goal is to teach critical appraisal skills that can be used to evaluate other articles. While generally there was agreement that the methodological articles and reviews are relevant and worthwhile, the item that received the lowest rating was residents’ agreement with the statement that “The methodological articles are easy to understand,” where 68% of residents agreed or strongly agreed with the statement. Thus, further efforts are required to determine how methodological concepts can be explained and taught more effectively.

Finally, we assessed residents’ opinions about their participation in journal clubs. Most respondents (86%) reported participating in journal clubs as part of their programs. The most common journal club format was a mandatory meeting held at a special time (e.g., evenings) every month with faculty members with epidemiological and clinical expertise in attendance. These findings are similar to the results of surveys of program directors of various surgical programs across Canada and the United States.^18,19 Similarly, a recent systematic review of journal clubs done by our group found that monthly scheduled meetings, mandatory attendance, use of a critical appraisal training checklist and inclusion of methodological/epidemiological articles were common features of journal clubs in surgical specialties.²⁰ The results of the present study show that journal clubs are valued highly by residents and that most (70%) use EBRS as part of their journal clubs either exclusively or in addition to other articles.

Limitations

There are some limitations to our survey. The response rate to the survey was 58%, and only 75% of the respondents were aware of EBRS and answered questions related to EBRS. Although a higher response rate would have been preferred, Asch and colleagues²¹ have reviewed the medical literature and reported that the mean response rate to physician surveys is 54%, which is similar to the response rate for our survey. Second, the scope of the survey was limited to assessing whether critical appraisal skills are taught in general surgery residency programs in Canada and to assessing the acceptance of EBRS by residents. This study did not address whether this leads to adherence to the practice of evidence-based medicine and improved patient outcomes.

Conclusion

Our results show that EBRS is currently being used by general surgery residency programs and their residents across Canada. However, future efforts can increase the use of EBRS among programs not currently using it. The EBRS program is rated highly by residents who use it and is the only method for teaching critical appraisal skills in most general surgery programs. It provides users with various online features; however, a large proportion of residents are unaware of all these features. Further efforts are needed to better educate the residents about the various online features of EBRS or to provide residents with alternative means of accessing EBRS, such as smartphone apps. Finally, our results show that journal clubs are a component of most general surgery residency programs across Canada and are valued highly by residents. The EBRS program has the potential to be implemented in existing journal club activities and hence equip the residents with the necessary critical appraisal skills to practise evidence-based medicine.