Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2011 Nov 18.
Published in final edited form as: Acad Emerg Med. 2010 Oct;17(Suppl 2):S54–S61. doi: 10.1111/j.1553-2712.2010.00889.x

Incorporating Evidence-based Medicine into Resident Education: A CORD Survey of Faculty and Resident Expectations

Christopher R Carpenter 1, Bryan G Kane 1, Merle Carter 1, Raymond Lucas 1, Lee G Wilbur 1, Charles S Graffeo 1
PMCID: PMC3219923  NIHMSID: NIHMS334436  PMID: 21199085

Abstract

Background

The Accreditation Council for Graduate Medical Education (ACGME) invokes evidence-based medicine (EBM) principles through the practice-based learning core competency. The authors hypothesized that among a representative sample of emergency medicine (EM) residency programs, a wide variability in EBM resident training priorities, faculty expertise expectations, and curricula exists.

Objectives

The primary objective was to obtain descriptive data regarding EBM practices and expectations from EM physician educators. Our secondary objective was to assess differences in EBM educational priorities among journal club directors compared with non–journal club directors.

Methods

A 19-question survey was developed by a group of recognized EBM curriculum innovators and then disseminated to Council of Emergency Medicine Residency Directors (CORD) conference participants, assessing their opinions regarding essential EBM skill sets and EBM curricular expectations for residents and faculty at their home institutions. The survey instrument also identified the degree of interest respondents had in receiving a free monthly EBM journal club curriculum.

Results

A total of 157 individuals registered for the conference, and 98 completed the survey. Seventy-seven (77% of respondents) were either residency program directors or assistant / associate program directors. The majority of participants were from university-based programs and in practice at least 5 years. Respondents reported the ability to identify flawed research (45%), apply research findings to patient care (43%), and comprehend research methodology (33%) as the most important resident skill sets. The majority of respondents reported no formal journal club or EBM curricula (75%) and do not utilize structured critical appraisal instruments (71%) when reviewing the literature. While journal club directors believed that resident learners’ most important EBM skill is to identify secondary peer-reviewed resources, non–journal club directors identified residents’ ability to distinguish significantly flawed research as the key skill to develop. Interest in receiving a free monthly EBM journal club curriculum was widely accepted (89%).

Conclusions

Attaining EBM proficiency is an expected outcome of graduate medical education (GME) training, although the specific domains of anticipated expertise differ between faculty and residents. Few respondents currently use a formalized curriculum to guide the development of EBM skill sets. There appears to be a high level of interest in obtaining EBM journal club educational content in a structured format. Measuring the effects of providing journal club curriculum content in conjunction with other EBM interventions may warrant further investigation.

Keywords: evidence-based medicine, knowledge translation, faculty development

Evidence-based medicine (EBM), perhaps more aptly named “research-enhanced practice,” is optimally the overlap of clinical expertise, best available evidence, and a patient’s unique values, all of which guide bedside decision-making in a dynamic fashion.1,2 The use of EBM tools can provide a structured, efficient approach for clinicians based on two guiding principles. First, not all evidence is created equal. In other words, a hierarchy of evidence exists, with weaker evidence manifest by increasing potential for bias. The second principle is that evidence alone is never enough, and the absence of evidence is not synonymous with evidence of absence.1 Evidence does not make decisions; people do. Authoritarian management decisions should not exclude individual patient preferences.3

Evidence-based medicine principles are increasingly important for emergency medicine (EM) residents to learn. Seasoned clinical expertise is invaluable, but is sometimes insufficient to ensure high-quality health care.4 Half of patients already fail to receive guideline-directed management.5 Traditionally, continuing medical education has been the primary mode of postresidency knowledge acquisition, but didactic conferences inconsistently improve knowledge acquisition or yield practice change.6,7 Additionally, industry can and has manipulated clinician behavior through the use of opinion leaders and marketing strategies that commonly misrepresent research findings.8–12 The Accreditation Council for Graduate Medical Education (ACGME) and the Council of Emergency Medicine Residency Directors (CORD) define the following elements for the core competency of practice-based learning:

  1. Locate, appraise, and use scientific evidence related to the patient’s health problems and the larger population from which they are drawn;

  2. Apply knowledge of study design and statistical methods to critically appraise the medical literature;

  3. Use information technology to enhance personal education and improve patient care.13

At a minimum, EM residency graduates need to have acquired a healthy skepticism for innovations as the science of knowledge translation advances the frontiers of health care.14,15 EBM is one framework often referenced for developing these competencies in residents, but trainees are being provided inadequate or inconsistent learning experiences. A recent survey of 65 EM program directors found that while 80% report EBM within their curriculum, only 22% provide more than 5 hours of exposure annually.16 Barriers to EBM dissemination in residency programs include inadequate time and trained personnel, incomplete awareness of EBM resources, insufficient funding, and faculty apathy.16 Although anecdotal reports of successful postgraduate EBM models have been described,17 skeptics question the data to support these models and EBM.18,19

Our primary objective was to obtain descriptive data on current EBM practices and anticipated outcomes from a representative group of EM educators attending the CORD annual academic assembly. Our secondary objective was to conduct a stratified analysis of journal club directors compared with non–journal club directors, with the hypothesis that the majority of EBM knowledge acquisition in EM residencies is self-contained within the journal club educational model. To address our study hypothesis, we sought to analyze sustainable, locally successful models of EM graduate medical education (GME) EBM curricula from diverse EM residency settings for educational elements that could be applied in other residency settings to overcome common barriers.

METHODS

Study Design and Population

In March 2010, CORD offered seven nonoverlapping workshops in best practices in EM residency training over 3 days as part of the annual academic assembly. More than 100 program directors and academic EM faculty attended the workshop on “Incorporating Evidence Based Medicine Into Your Residency.” In preparation for the didactic workshop, two authors of this article (CRC, CSG) solicited the CORD membership for examples of sustained, well-established components of GME EBM curricula. Among respondents, a panel of interested EBM leaders was formed in September 2009 to delineate the objectives for the didactic session and this article. This survey study was approved by the Eastern Virginia Medical School Institutional Review Board.

Survey Content and Administration

A survey instrument to explore current attitudes towards and actual teaching of EBM was developed via iterative feedback with a qualitative research expert. The final survey instrument was distributed three weeks before the CORD meeting via e-mail to all best practices track registrants (n = 157) with a follow-up e-mail 5 days before the meeting for all nonrespondents. The survey was also handed out in paper format at the CORD meeting, and attendees were asked to hand in their responses at the end of the didactic session. Those who had completed the e-mail survey were asked not to turn in a paper copy as well.

The investigators planned a priori to conduct a subset analysis of journal club directors compared with other respondents to identify significant differences representing opportunities or barriers to promoting EBM curricular initiatives in GME. This analysis was planned based on the hypothesis that journal club represents the vehicle by which EBM is taught in EM residencies.

All survey responses were entered into an Excel database (Microsoft Corp., Redmond, WA) on a double password–protected computer maintained behind two locked doors. Prior to releasing the data to investigators, the responses were stripped of all identifiers by administrative personnel not involved in the study design or manuscript preparation. All duplicate responses were eliminated prior to data entry.

Data Analysis

Univariate data analysis was performed using SPSS (version 16.0, SPSS, Inc., Chicago IL) by one author (CRC) employing chi-square analysis for dichotomous proportional data and factorial logistic regression analysis when more than two categorical responses were possible.

RESULTS

The survey was completed by 98 participants, including 34 via e-mail and 64 at the CORD annual academic assembly. There were no significant differences between e-mail and written respondents in experience or job title. Two respondents (a resident and non-EM physician) were eliminated from this analysis to provide a more homogeneous sampling of CORD faculty attendees. The results for the remaining 96 respondents are described in Table 1. The majority of respondents (91%) were beyond 5 years postresidency and practicing in a university-based EM training program (57%). Also, the majority of respondents serve as either the program director or the associate / assistant program director (77%). While 25% personally administer journal club, almost half (48%) attend journal club on a relatively infrequent (<60%) basis. The majority of respondents do not use either a formal critical appraisal instrument (71%) or an established EBM curriculum for journal club (75%). Most (78%) do not use any extramural sources for journal club, such as ACP Journal Club, Best Evidence in Emergency Medicine, EM Abstracts, or Annals of Emergency Medicine Journal Club.20–24 Although most respondents are not currently using available EBM resources, the large majority (89%) expressed interest in a free monthly EBM curriculum to use as their journal club if one were available.

Table 1.

Overall Survey Responses (N = 96)

Respondents
(%*)
Years postresidency
   1–4 9 (9)
   5–9 34 (35)
   10–19 38 (40)
   >20 15 (16)
Practice setting
   University-based EM residency 55 (57)
   Community-based EM residency 35 (36)
   Affiliate hospital with EM resident rotator 1 (1)
   Hospital with no EM residents 2 (2)
   Other 3 (3)
Academic position
   Chair or vice chair 9 (9)
   Program director 29 (30)
   Associate or assistant program director 45 (47)
   Research director 3 (3)
   Site director 1 (1)
   Academic faculty, clinical researcher 1 (1)
   Academic faculty, educational researcher 3 (3)
   Academic faculty, other 4 (4)
   Clinically affiliated faculty 1 (1)
Personally teach EBM principles 60 (63)
Personally administer journal club 24 (25)
Personal annual journal club attendance
   Once or twice (1%–20%) 11 (11)
   A few times (21%–40%) 14 (15)
   Frequently (41%–60%) 21 (22)
   Almost always (61%–80%) 28 (29)
   Always (81%–100%) 22 (23)
Use structured critical appraisal instrument (Users’ Guide) 28 (29)
Use extramural source for journal club 21 (22)
Have predetermined journal club curriculum 24 (25)
Single most important EBM skill—all EM physicians
   Formulate question 7 (7)
   Search strategy 7 (7)
   Understand research methodology 32 (33)
   Determine ideal study statistical approach 1 (1)
   Differentiate statistical and clinical significance 6 (6)
   Apply research results to individual patients 41 (43)
Most important outcome of GME EBM curricula—residents
   Become expert critical appraisers 26 (27)
   Differentiate minimally and significantly flawed studies 43 (45)
   Ability to find reliable secondary peer reviewed material 27 (28)
Most important outcome of GME EBM curricula—faculty
   Become expert critical appraisers 53 (55)
   Differentiate minimally and significantly flawed studies 32 (33)
   Ability to find reliable secondary peer reviewed material 9 (9)
EM residents must be able to critically appraise research
   Strongly disagree 7 (7)
   Disagree 12 (13)
   Neutral 9 (9)
   Agree 43 (45)
   Strongly agree 24 (25)
EM faculty must be able to critically appraise research
   Strongly disagree 7 (7)
   Disagree 6 (6)
   Neutral 11 (12)
   Agree 36 (37)
   Strongly agree 33 (34)
EM resident critical appraisal proficiency
   None 0
   Minimal 18 (19)
   Moderate 52 (54)
   High 22 (23)
   Very high 4 (4)
EM faculty critical appraisal proficiency
   None 0
   Minimal 4 (4)
   Moderate 30 (31)
   High 43 (45)
   Very high 17 (18)
EM residents should know how to use secondary sources†
   Strongly disagree 11 (12)
   Disagree 2 (2)
   Neutral 3 (3)
   Agree 30 (31)
   Strongly agree 50 (52)
EM faculty should know how to use secondary sourcesi†
   Strongly disagree 7 (7)
   Disagree 0
   Neutral 3 (3)
   Agree 29 (30)
   Strongly agree 56 (58)
Interested in free monthly EBM curriculum 85 (89)
Open in a new tab

EBM = evidence-based medicine; GME = graduate medical education.

*

Proportions may not add up to 100% due to rounding and exclusion of nonrespondents.

†

Examples of Cochrane library, EM Abstracts, and Best Evidence in Emergency Medicine provided to survey participants.

Survey respondents identified the application of research findings to individual patients (43%) and understanding research methodology (33%) as the quintessential EBM skills for resident and faculty emergency physicians. Expected GME EBM curricula outcomes for resident physicians and faculty physicians differed. Resident physicians were expected to finish their training with the ability to differentiate minimally biased research from significantly flawed studies (45%), while faculty were to develop sufficient capabilities to become expert critical appraisers (55%). Most respondents felt that residents (71%) and faculty (74%) need to possess sufficient critical appraisal expertise to make appropriate real-time patient care decisions during clinical shifts. Most felt that residents (83%) and faculty (88%) should learn how to identify and access secondary peer-reviewed sources relevant to their patient’s care following an effective GME EBM program.

The analysis comparing journal club directors to non–journal club directors is summarized in Table 2. Journal club directors were not more likely to teach EBM principles, use structured critical appraisal forms, or have an established journal club curriculum. More non–journal club directors favored the ability to distinguish minimally and significantly flawed research (25% of journal club directors vs. 52% of non–journal club directors; p = 0.05). Journal club directors instead favored the ability to find and use secondary peer-reviewed sources (46% vs. 21%; p = 0.05). Most physicians felt that finding and using secondary sources (e.g., ACP Journal Club, Best Evidence in Emergency Medicine) should be part of the skill set for residents (88% in both groups) and faculty (80% among journal club directors, 92% among non–journal club directors). Both journal club directors and nondirectors identified the ability to apply research findings to individual patients as the single most important EBM skill for all EM physicians (42% journal club directors and 40% of non–journal club directors).

Table 2.

Survey Responses Stratified by Journal Club Director (N = 91)*

Journal Club
Directors (%)		 Non–Journal
Club Directors (%)
Academic position
   Program director, associate or assistant PD 20 (83) 52 (78)
   Non-PD 4 (17) 15 (22)
Personally teach EBM principles 18 (75) 40 (60)
Personal annual journal club attendance
   > 40% 21 (88) 45 (67)
Use structured critical appraisal instrument (Users’ Guide) 8 (33) 19 (29)
Use extramural source for journal club 7 (30) 13 (20)
Have predetermined journal club curriculum 6 (26) 17 (27)
Single most important EBM skill—all EM physicians
   Formulate question 2 (9) 5 (8)
   Search strategy 2 (9) 5 (8)
   Understand research methodology 8 (35) 23 (35)
   Determine ideal study statistical approach 0 1 (1)
   Differentiate statistical & clinical significance 1 (4) 5 (8)
   Apply research results to individual patients 10 (42) 27 (40)
Most important outcome of GME EBM curricula—residents
   Become expert critical appraisers 7 (29) 18 (27)
   Differentiate minimally and significantly flawed studies 6 (25) 35 (52)
   Ability to find reliable secondary peer-reviewed material 11 (46) 14 (21)
Most important outcome of GME EBM curricula—faculty
   Become expert critical appraisers 11 (46) 38 (58)
   Differentiate minimally and significantly flawed studies 9 (38) 23 (35)
   Ability to find reliable secondary peer-reviewed material 4 (17) 4 (6)
EM residents must be able to critically appraise research
   Strongly disagree or disagree 4 (17) 15 (22)
EM faculty must be able to critically appraise research
   Strongly disagree or disagree 4 (17) 9 (13)
EM resident critical appraisal proficiency
   None 0 0
   Minimal 3 (13) 13 (19)
   Moderate 13 (54) 38 (57)
   High 7 (29) 13 (19)
   Very high 1 (4) 3 (4)
EM faculty critical appraisal proficiency
   None 0 0
   Minimal 2 (8) 2 (3)
   Moderate 6 (25) 23 (34)
   High 12 (50) 28 (42)
   Very high 4 (17) 12 (18)
EM residents should know how to use secondary sources†
   Strongly disagree or disagree 3 (12) 8 (12)
EM faculty should know how to use secondary sources†
   Strongly disagree or disagree 5 (20) 5 (8)
Interested in free monthly EBM curriculum 23 (96) 59 (88)
Open in a new tab
*

Five respondents did not answer this question.

†

Examples of Cochrane library, EM Abstracts, and Best Evidence in Emergency Medicine provided to survey participants.

EBM = evidence based medicine; GME = graduate medical education; PD = program director.

DISCUSSION

This is the largest survey to ascertain EM GME EBM curricular practice and expectations to date. Our sampling of EM educators suggests that understanding research methodology, finding reliable sources of secondary peer-reviewed content, and applying research findings within the context of patient care are important skills for all emergency physicians. These findings are consistent with the ACGME requirements for developing “systems-based practices” that encourage appropriate and efficient use of medical, social, and technological systems, with the ultimate goal of improving patient care.25,26 Despite this recognition among our respondents, a minority have an established EBM curriculum, routinely use user guides to the medical literature, or take advantage of extramural peer-reviewed resources. These findings appear to support our hypothesis that wide variability in EBM resident training, faculty expertise, and curricula exists.

For more than a century, journal club has been the de facto vehicle to develop critical appraisal expertise during residency.27 Traditionally, journal club has focused on research data appraisal without expanding to include the EBM competencies of defining a clinical question before finding the evidence, communicating complicated scientific interpretations to patients to inform health care decision-making, or knowledge translation.28 Another limitation of the traditional journal club approach is that negative studies are frequently not published.29–31 Furthermore, the effect of journal club on resident EBM skill uptake is inconclusive.32–35 Finally, literature linking new evidence acquisition to practice change and the ultimate goal of improved patient outcomes as the result of journal club or EBM curricula is lacking.35–37

One means of moving the traditional journal club toward one embedded with EBM is to use PICO (PICO = patient, intervention, comparison, outcome) questions. Focusing on resident generated or “real world” clinical problems as a source for choosing articles helps add relevance.38 Additionally, journal club websites and databases allow clinicians timely access to previously appraised topics during patient care. Multiple programs archive their journal club–derived critically appraised topics for public access on the Internet.39,40 One of these programs has published the results of its journal club infrastructure, instructional instruments, and final product reviews in the EM literature.17,41–46 These archives serve the function of internally developed secondary sources, and the websites include links to free online statistical calculators, external secondary peer-reviewed synopses, and other institutions’ EBM curricula.47 This practice is entirely consistent with the viewpoints of EM program directors in our survey who felt that the ability to both analyze research articles and access secondary peer-reviewed sources are important skills. An additional benefit of adopting use of these websites and databases may be to cultivate an EBM culture and warrants further investigation.

Another method of creating a culture of research-enhanced practice within a residency program may include providing incoming residents with introductory EBM lectures during orientation, including topics such as the formulation of relevant clinical questions in the form of PICOs, efficient Web-based search strategies for primary and secondary peer-reviewed sources, and an introduction to key EBM resources such as the Users’ Guide to the Medical Literature.38,48 An additional means to embed EBM into a didactic program is to replace core content lectures with a resident-formulated, faculty-mentored literature review and critical appraisal prior to presenting to their peers. This approach has already been described in the literature.49,50

Previously identified barriers to EBM dissemination in residency include inadequate instructional time and trained personnel, incomplete awareness of EBM resources, insufficient funding, and faculty apathy.16 In our opinion, fundamental elements for success include departmental chair support, shared faculty expertise, and a local EBM champion.51–53 Integrating EBM into other areas of the didactic curriculum and into clinical practice may offer a more holistic EBM teaching approach that extends beyond journal club as the sole pedagogical approach. It also increases the available instructional time previously noted to be lacking. Evidence-based workshops that teach question formation or effective search strategies can be efficient ways to teach fundamental skills, such as forming PICO questions and learning literature search strategies. These workshops can be augmented by enlisting expertise from reference librarians or guest faculty.54

It has been suggested that residents should generate clinically relevant PICO questions based on their patients. The residents should then proceed to document their search strategy, the time required for their search, and their conclusions for each question based on the search.55 This process has been successfully employed in at least one EM residency program. The anticipated benefit of such an experience should be the resident, in real time, connecting EBM to patient care. One immediate objective of an EBM curriculum could be teaching residents and faculty how to find and utilize high-quality secondary peer-reviewed information sources that few are currently using.54,56 One means to document such utilization is to track online electronic reference resource use by end-users. Proprietary software packages (e.g., Horizon Business Insight, McKesson Corp., San Francisco, CA) make it possible to determine utilization practices for resident and faculty regarding digital library holdings.57 Institutional library services departments are potential partners in this effort, as the documentation of database use by clinicians may assist in justifying their budgetary process. This documentation may have the potential added benefit of serving as ACGME “Phase 3” outcome data. One author’s institution has instituted this online resource tracking (BGK).

Emergency medicine residency programs can quantify learners’ EBM knowledge acquisition using the Fresno test, a validated 12-question measure of EBM skills.58 The results could be used to inform curricular development. When any program is both designing its residency EBM curriculum and delineating its faculty development plan, our data suggest that it is important to recognize that expectations for EBM expertise differ for faculty and residents. Whereas residents are expected to attain the ability to identify significantly flawed studies, faculty should master critical appraisal of the medical literature. These differences should direct local administrative and educational leaders to provide sufficient opportunities for faculty to acquire proficiency in critical appraisal.59,60

When asked if they would be interested in receiving a monthly EBM journal club curriculum that included relevant articles, expert critical appraisals of the articles, and worksheets with answer keys, most responded affirmatively.61 We believe that providing such content could address some of the barriers to EBM dissemination that we have described.16,62,63

LIMITATIONS

The survey was not validated, although the instrument has content and face validity. We surveyed a small proportion of CORD members and cannot be certain that our sampling is representative of the organizational whole. The CORD membership is 737 individuals from 221 programs, including allopathic and osteopathic programs, along with fellowships.64 Our survey consisted of 98 respondents. The small sample size means that a Type II error is possible. Furthermore, survey respondents likely represent a select subset of CORD members likely to attend the annual meeting and participate in educational projects. Additionally, we did not track the number of residency programs that were represented by our individual respondents, so multiple surveys may have represented the same program. Therefore, our results may have a spectrum bias and lack external validity for all of the EM GME community.

CONCLUSIONS

Among emergency medicine educators, the attainment of evidence-based medicine proficiency by faculty and resident learners is an expected outcome of graduate medical education training, although the specific domains of expertise differ between faculty and residents. Despite this, only a minority of respondents currently use a formalized curriculum for EBM skills’ development. Journal club directors’ primary evidence-based medicine educational objective for resident learners is to identify secondary peer-reviewed sources, whereas non–journal club directors favor learners’ ability to distinguish significantly flawed research designs. There appears to be a high level of interest in obtaining EBM journal club educational content in a structured format. Measuring the effects of providing journal club curriculum content in conjunction with other evidence-based medicine interventions may warrant further investigation.

Acknowledgments

The authors thank John A. Ullian, PhD, for his guidance in developing the survey instrument. Additionally, we thank Cora Sias for collecting all survey responses to strip them of all identifiers prior to releasing them to the investigators. We acknowledge Teresa Vallera and Andrew Worster, MD, for their time and assistance in putting the Best Evidence in Emergency Medicine manual together for conference attendees. We also offer our appreciation to Francis Counselman, MD, and Lawrence Lewis, MD, for their insightful review of earlier versions of the manuscript.

Disclosures: This publication was made possible by Grant Number UL1 RR024992 from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research. Its contents are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. Dr. Carpenter is a faculty member for Best Evidence in Emergency Medicine and was supported by the Washington University Goldfarb Patient Safety award.

Footnotes

Presented at the Council of Emergency Medicine Residency Directors Academic Assembly, Orlando, FL, March 4, 2010.

References

  • 1.Guyatt G, Haynes B, Jaeschke R. The philosophy of evidence-based medicine. In: Guyatt G, Rennie D, Meade MO, et al., editors. Users’ Guides to the Medical Literature. 2nd ed. New York: McGraw-Hill; 2008. pp. 9–16. [Google Scholar]
  • 2.Strauss SE, Richardson WS, Glasziou P, et al. Evidence-based Medicine: How to Practice and Teach EBM. 3rd ed. Edinburgh, Scotland: Elsevier; 2005. [Google Scholar]
  • 3.Haynes RB, Devereaux PJ, Guyatt GH. Physicians’ and patients’ choices in evidenced based practice. BMJ. 2002;324:1350. doi: 10.1136/bmj.324.7350.1350. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260–273. doi: 10.7326/0003-4819-142-4-200502150-00008. [DOI] [PubMed] [Google Scholar]
  • 5.McGlynn EA, Asch SM, Adams J, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–2645. doi: 10.1056/NEJMsa022615. [DOI] [PubMed] [Google Scholar]
  • 6.Forsetlund L, Bjorndal A, Rashidan A, et al. Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2009;2 doi: 10.1002/14651858.CD003030.pub2. CD003030. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Gene-Hern H, Wills C, Alter H, et al. Conference attendance does not correlate with emergency medicine residency in-training examination scores. Acad Emerg Med. 2009;16 Suppl 2:S63–S66. doi: 10.1111/j.1553-2712.2009.00596.x. [DOI] [PubMed] [Google Scholar]
  • 8.Fugh-Berman A, Ahari S. Following the script: how drug reps make friends and influence doctors. PLoS Med. 2007;4 doi: 10.1371/journal.pmed.0040150. e150. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Moynihan R. Key opinion leaders: independent experts or drug representatives in disguise? BMJ. 2008;336:1402–1403. doi: 10.1136/bmj.39575.675787.651. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Carpenter CR, Sherbino J. How does an ‘ opinion leader’ influence my practice? CJEM. 2010;12(5):431–434. doi: 10.1017/s1481803500012586. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Garattini S, Chalmers I. Patients and the public deserve big changes in evaluation of drugs. BMJ. 2009;338:b1025. doi: 10.1136/bmj.b1025. [DOI] [PubMed] [Google Scholar]
  • 12.Othman N, Vitry A, Roughead EE. Quality of pharmaceutical advertisements in medical journals: a systematic review. PLoS One. 2009;4:e6350. doi: 10.1371/journal.pone.0006350. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Hayden SR, Dufel S, Shih R. Definitions and competencies for practice-based learning and improvement. Acad Emerg Med. 2002;9:1242–1248. doi: 10.1111/j.1553-2712.2002.tb01584.x. [DOI] [PubMed] [Google Scholar]
  • 14.Berwick DM. Disseminating innovations in health care. JAMA. 2003;289:1969–1975. doi: 10.1001/jama.289.15.1969. [DOI] [PubMed] [Google Scholar]
  • 15.Diner BM, Carpenter CR, O’Connell T, et al. Graduate medical education and knowledge translation: role models, information pipelines, and practice change thresholds. Acad Emerg Med. 2007;14:1008–1014. doi: 10.1197/j.aem.2007.07.003. [DOI] [PubMed] [Google Scholar]
  • 16.Kuhn GJ, Wyer PC, Cordell WH, Rowe BH. A survey to determine the prevalence and characteristics of training in evidence-based medicine in emergency medicine residency programs. J Emerg Med. 2005;28:353–359. doi: 10.1016/j.jemermed.2004.09.015. [DOI] [PubMed] [Google Scholar]
  • 17.Carpenter CR, Katz E, Char D. Re: Journal club and teaching evidence-based medicine. J Emerg Med. 2006;31:306–308. doi: 10.1016/j.jemermed.2006.06.005. [DOI] [PubMed] [Google Scholar]
  • 18.Kuhn G, Cordell W, Rowe B, et al. In reply: Journal Club and EBM in emergency medicine. J Emerg Med. 2006;31:307–308. [Google Scholar]
  • 19.Tobin MJ. Counterpoint: evidence-based medicine lacks a sound scientific base. Chest. 2008;133:1071–1074. doi: 10.1378/chest.08-0077. [DOI] [PubMed] [Google Scholar]
  • 20.Haynes RB, Mulrow CD, Huth EJ, Altman DG, Gardner MJ. More informative abstracts revisited. Ann Intern Med. 1990;113:69–76. doi: 10.7326/0003-4819-113-1-69. [DOI] [PubMed] [Google Scholar]
  • 21.Haynes RB, Cotoi C, Holland J, et al. Second-order peer review of the medical literature for clinical practitioners. JAMA. 2006;295:1801–1808. doi: 10.1001/jama.295.15.1801. [DOI] [PubMed] [Google Scholar]
  • 22.McMaster University, Hamilton, Ontario and Washington University, St. Louis, Missouri. [Accessed Jul 16, 2010];Best Evidence in Emergency Medicine. Available at: http://www.beemsite.com/ [Google Scholar]
  • 23.Bukata R, Hoffman J. The Center for Medical Education. [Accessed Jul 18, 2010];Emergency Medical Abstracts. Available at: http://prod2.ccme.org/ema/ [Google Scholar]
  • 24.Barrett TW, Schriger DL. Annals of Emergency Medicine Journal Club. Emergency department crowding is associated with poor care for patients with severe pain. Ann Emerg Med. 2008;51:6–7. doi: 10.1016/j.annemergmed.2007.11.022. [DOI] [PubMed] [Google Scholar]
  • 25.Chapman DM, Hayden S, Sanders AB, et al. Integrating the Accreditation Council for Graduate Medical Education Core competencies into the model of the clinical practice of emergency medicine. Ann Emerg Med. 2004;43:756–769. doi: 10.1016/j.annemergmed.2003.12.022. [DOI] [PubMed] [Google Scholar]
  • 26.Hobgood C, Promes S, Wang E, et al. Outcome assessment in emergency medicine–a beginning: results of the Council of Emergency Medicine Residency Directors (CORD) emergency medicine consensus workgroup on outcome assessment. Acad Emerg Med. 2008;15:267–277. doi: 10.1111/j.1553-2712.2008.00046.x. [DOI] [PubMed] [Google Scholar]
  • 27.Linzer M. The journal club and medical education: over one-hundred years of unrecorded history. Postgrad Med J. 1987;63:475–578. doi: 10.1136/pgmj.63.740.475. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Hatala R. Is evidence-based medicine a teachable skill? Ann Emerg Med. 1999;34:226–228. doi: 10.1016/s0196-0644(99)70236-2. [DOI] [PubMed] [Google Scholar]
  • 29.Lee K, Bacchetti P, Sim I. Publication of clinical trials supporting successful new drug applications: a literature analysis. PLoS Med. 2008;5:e191. doi: 10.1371/journal.pmed.0050191. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Ross JS, Mulvey GK, Hines EM, Nissen SE, Krumholz HM. Trial publication after registration in ClinicalTrials.gov: a cross-sectional analysis. PLoS Med. 2009;6 doi: 10.1371/journal.pmed.1000144. E1000144. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Song F, Parekh-Bhurke S, Hooper L, et al. Extent of publication bias in different categories of research cohorts: a meta-analysis of empirical studies. BMC Med Res Methodol. 2009;9:79. doi: 10.1186/1471-2288-9-79. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Woods JR, Winkel CE. Journal club format emphasizing techniques of critical reading. J Med Educ. 1982;57(10 Pt 1):799–801. doi: 10.1097/00001888-198210000-00011. [DOI] [PubMed] [Google Scholar]
  • 33.Alguire PC. A review of journal clubs in postgraduate medical education. J Gen Intern Med. 1998;13:347–353. doi: 10.1046/j.1525-1497.1998.00102.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Bazarian JJ, Davis CO, Spillane LL, Blumstein H, Schneider SM. Teaching emergency medicine residents evidence-based critical appraisal skills: a controlled trial. Ann Emerg Med. 1999;34:148–154. doi: 10.1016/s0196-0644(99)70222-2. [DOI] [PubMed] [Google Scholar]
  • 35.Ebbert JO, Montori VM, Schultz HJ. The journal club in postgraduate medical education: a systematic review. Med Teacher. 2001;23:455–461. doi: 10.1080/01421590120075670. [DOI] [PubMed] [Google Scholar]
  • 36.Jenicek M. Evidence-based medicine: fifteen years later. Golem the good, the, bad, the ugly in need of a review? Med Sci Monit. 2006;12:R241–R251. [PubMed] [Google Scholar]
  • 37.Coomarasamy A, Khan KS. What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ. 2004;329:1017. doi: 10.1136/bmj.329.7473.1017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Guyatt G, Rennie D, Meade MO, et al. Users’ Guides to the Medical Literature. 2nd ed. New York, NY: McGraw-Hill; 2008. [PubMed] [Google Scholar]
  • 39.Graffeo CS. Eastern Virginia Medical School. [Accessed Mar 29, 2010]; emjournalclub.com. Available at: http://www.emjournalclub.com/ [Google Scholar]
  • 40.Carpenter CR. Washington University in St. Louis School of Medicine Emergency Medicine Journal Club. [Accessed Mar 22, 2010];Washington University School of Medicine. Available at: http://emed.wustl.edu/em_journal_club.html. [Google Scholar]
  • 41.Carpenter CR. Kappa statistic. CMAJ. 2005;173:15–16. doi: 10.1503/cmaj.1041742. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Carpenter CR. Review: adding dexamethasone to standard therapy reduces short-term relapse for acute migraine in the emergency department. Evid Based Med. 2009;14:121. doi: 10.1136/ebm.14.4.121. [DOI] [PubMed] [Google Scholar]
  • 43.Carpenter CR. The San Francisco Syncope Rule did not accurately predict serious short-term outcome in patients with syncope. Evid Based Med. 2009;14:25. doi: 10.1136/ebm.14.1.25. [DOI] [PubMed] [Google Scholar]
  • 44.Carpenter CR, Keim SM, Crossley J, Perry JJ. Post-transient ischemic attack early stroke stratification: the ABCD(2) prognostic aid. J Emerg Med. 2009;36:194–200. doi: 10.1016/j.jemermed.2008.04.034. [DOI] [PubMed] [Google Scholar]
  • 45.Carpenter CR, Keim SM, Seupaul RA, Pines JM. Differentiating low-risk and no-risk PE patients: the PERC score. J Emerg Med. 2009;36:317–322. doi: 10.1016/j.jemermed.2008.06.017. [DOI] [PubMed] [Google Scholar]
  • 46.Carpenter CR, Keim SM, Upadhye S, et al. Risk stratification of the potentially septic patient in the emergency department: the Mortality in the Emergency Department Sepsis (MEDS) score. J Emerg Med. 2009;37:319–327. doi: 10.1016/j.jemermed.2009.03.016. [DOI] [PubMed] [Google Scholar]
  • 47.Carpenter CR. Washington University in St. Louis School of Medicine Emergency Medicine Journal Club Toolbox. [Accessed Mar 22, 2010];March 22; Available at: http://emed.wustl.edu/em_links.html. [Google Scholar]
  • 48.McKibbon A, Wyer P, Jaeschke R, et al. Finding the evidence. In: Guyatt G, Rennie D, Meade MO, et al., editors. Users’ Guides to the Medical Literature. New York, NY: McGraw-Hill; 2008. pp. 29–58. [Google Scholar]
  • 49.Wyer PC. The critically appraised topic: closing the evidence-transfer gap. Ann Emerg Med. 1997;30:639–640. doi: 10.1016/s0196-0644(97)70083-0. [DOI] [PubMed] [Google Scholar]
  • 50.Sadosty AT, Goyal DG, Gene-Hern H, et al. Alternatives to the conference status quo: summary recommendations from the 2008 CORD Academic Assembly Conference Alternatives Work Group. Acad Emerg Med. 2009;16 Suppl 2:S25–S31. doi: 10.1111/j.1553-2712.2009.00588.x. [DOI] [PubMed] [Google Scholar]
  • 51.Brandt TL, Romme CR, Lar usso NF, LaRusso NF, Lindor KD. A novel incentive system for faculty in an academic medical center. Ann Intern Med. 2002;137:738–743. doi: 10.7326/0003-4819-137-9-200211050-00009. [DOI] [PubMed] [Google Scholar]
  • 52.Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. 2006;28:497–526. doi: 10.1080/01421590600902976. [DOI] [PubMed] [Google Scholar]
  • 53.Reece EA, Nugent O, Wheeler RP, Smith SW, Hough AJ, Winter C. Adapting industry-style business model to academia in a system of performance-based incentive compensation. Acad Med. 2008;83:76–84. doi: 10.1097/ACM.0b013e31815c6508. [DOI] [PubMed] [Google Scholar]
  • 54.Oliver KB, Dalrymple P, Lehmann HP, McClellan DA, Robinson KA, Twose C. Bringing evidence to practice: a team approach to teaching skills required for an informationist role in evidence-based clinical and public health practice. J Med Libr Assoc. 2008;96:50–57. doi: 10.3163/1536-5050.96.1.50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55.Corrall CJ, Wyer PC, Zick LS, Bockrath CR. Evidence-based emergency medicine. How to find evidence when you need, it, part 1: databases, search programs, and strategies. Ann Emerg Med. 2002;39:302–306. doi: 10.1067/mem.2002.122149. [DOI] [PubMed] [Google Scholar]
  • 56.Garg A, Turtle KM. Effectiveness of training health professionals in literature search skills using electronic health databases–a critical appraisal. Health Info Libr J. 2003;20:33–41. doi: 10.1046/j.1471-1842.2003.00416.x. [DOI] [PubMed] [Google Scholar]
  • 57.HBI Horizon Business Insight. Yardley Management Solutions, Inc.; [Accessed Apr 1, 2010]. Available at: http://www.ymsolutions.com/hbi-horizon-business-insight. [Google Scholar]
  • 58.Ramos KD, Schafer S, Tracz SM. Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003;326:319–321. doi: 10.1136/bmj.326.7384.319. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Green ML. Evidence-based medicine training in graduate medical education: past, present and future. J Eval Clin Pract. 2000;6:121–138. doi: 10.1046/j.1365-2753.2000.00239.x. [DOI] [PubMed] [Google Scholar]
  • 60.Taylor RS, Reeves BC, Ewings PE, Taylor RJ. Critical appraisal skills training for health care professionals: a randomized controlled trial. BMC Med Educ. 2004;4:30. doi: 10.1186/1472-6920-4-30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Worster A, Carpenter CR, Upadhye S. Best Evidence in Emergency Medicine Journal Club. [Accessed Jul 16, 2010]; Available at: http://www.beemsite.com/journal.html. [Google Scholar]
  • 62.Green ML, Ruff TR. Why do residents fail to answer their clinical questions? A qualitative study of barriers to practicing evidence-based medicine. Acad Med. 2005;80:176–182. doi: 10.1097/00001888-200502000-00016. [DOI] [PubMed] [Google Scholar]
  • 63.van Dijk N, Hooft L, Wieringa-de Waard M. What are the barriers to residents’ practicing evidence-based medicine? A systematic review. Acad Med. 2010;85:1163–1170. doi: 10.1097/ACM.0b013e3181d4152f. [DOI] [PubMed] [Google Scholar]
  • 64.Mulder B. Personal communication with Carpenter Cr regarding CORD membership numbers. 2010 June 21; [Google Scholar]

RESOURCES