A Model Research Curriculum for Emergency Medicine Residency: A Modified Delphi Consensus

Abstract

Background

Research training is important for all resident physicians to be able to interpret and critically appraise scientific literature. It is particularly important for those desiring future careers in academics or research. However, there is limited literature on research training in residency. The ideal scope and content of a model curriculum for research in emergency medicine (EM) residency programs has not been recently defined.

Methods

We used a modified Delphi method to achieve expert consensus as to what content should be included in a model research curriculum for EM residents as well as for those who choose to undertake an elective in EM research.

Results

Eight experts in EM clinical and education research participated in the modified Delphi process with two rounds of responses. Cronbach's alpha was 0.82 for round 1 and 0.95 for round 2. A curricular outline of 44 items was produced by this consensus process.

Conclusions

A model curriculum for EM research is proposed here, along with references to assist residencies in building curricular components.

Establishing the scope of curricula for emergency medicine (EM) residents can be a difficult task. In such a dynamic and broad field, it can be challenging to define the aspects of practice that are most important and that are less essential. This also holds true for teaching EM residents about the practice and interpretation of research. A strong understanding of research is essential for anyone who hopes to understand scientific evidence as it emerges and implement best practices for patient care. Further, residents hoping to embark on an academic career will almost certainly need to be familiar with research methods and how to disseminate investigatory findings, even if they are not destined to become primary “researchers.”

The question of a “model curriculum” for research in EM has been addressed previously. ¹ That document, however, is now more than 25 years old and covers a broad scope, including topics not likely to be covered until advanced training such as a research fellowship or advanced degree. The Model of the Clinical Practice of Emergency Medicine, another essential standard‐setting document, has been updated more recently. ² It includes items relevant to EM research under the heading “practice‐based learning and improvement,” but these items are not detailed, and include “research,” “evidence‐based medicine,” and “interpretation of the literature.” One study found that nearly one‐third of all residencies have a resident research rotation. ³ Additionally, recent investigations of the “scholarly activity requirement” by EM residencies found wide variation in application, and many residencies identified significant limitations to this kind of learning‐by‐doing approach to teaching scholarship and research. ⁴ Residency programs would benefit from further guidance as to what EM research topics are most essential in the current day and age. In particular, EM research today is broader in scope, more sophisticated, and far more voluminous than was the case 25 years ago, making the skills related to discernment and creation of high‐quality research all the more important. Resident and faculty engagement in scholarship, to include peer‐reviewed and other forms of research, is included in the Emergency Medicine Program Requirements, IV.D.2.b).(1) and IV.D.2.b).(2). ⁵

We sought to develop a consensus among a group of EM research and education experts for a model research curriculum. This curriculum is intended to include topics that should ideally be part of the overall curriculum for EM residency and also extends to topics that may be covered in a research elective. The final product should be applicable to faculty planning the research curriculum for the entire residency and for those planning an elective experience. We further attempted to identify resources and citations that may be of benefit to residency programs as they build and administer education programs related to this content.

METHODS

Study Design

We used a modified Delphi technique to achieve consensus on elements of a model curriculum for research in EM. The Delphi technique builds content validity and is an accepted method to develop consensus by eliciting expert opinions through multiple rounds of questionnaires. The institutional review board at the Wake Forest School of Medicine approved this study.

Study Protocol

We assembled a research team with experience in EM resident education and research to evaluate the literature and develop an initial research curriculum for EM residents, with the intention of covering the scope of items that might be taught to a resident who participates in the normal residency curriculum as well as an elective in research. We chose to frame the curricular scope in this way (“normal” residency content plus research elective) because the way in which content is partitioned between those two activities may vary considerably by program, and we sought a guide that would cover topics important to both settings.

Previous work in this area ¹ as well as the results of unpublished surveys of EM program directors conducted by SAEM and CORD were consulted. In addition, multiple existing EM research curricula were evaluated. Based on these resources, the research team assembled an initial curriculum outline with 57 items [Table 1]. We then established consensus on the checklist using a modified Delphi technique consisting of two rounds of polling the expert panel.

Table 1.

Curriculum Items

Item Number	Delphi Round 1		Delphi Round 2		Final Version of Item
	Mean ± SD	Modification	Mean ± SD	Modification
1	8.375 ± 1.41	None	8.25 ± 1.39	None	I. Introduction to Research
2	8 ± 1.69	Wording change	6.75 ± 2.12	None	a. Importance of research to practice and implementation of professional best practice
3	8.25 ± 1.39	None	8.875 ± 0.35	None	b. Posing a research question
4	8 ± 1.31	None	8.38 ± 0.74	None	c. Introduction to study design
5	7.25 ± 2.05	Wording change	8 ± 1.31	None	II. Appraisal of the Literature
6	7.88 ± 0.99	None	8.38 ± 0.5	None	a. Searching the literature to answer a question
7	7 ± 1.60	None	7.13 ± 1.25	None	b. Types of studies
8	5.88 ± 1.46	None	6.88 ± 0.99	None	i. Classification by time: retrospective,  prospective
9	6.13 ± 1.13	None	6.88 ± 0.99	None	ii. Classification by control: case control, cohort, randomized controlled trial
10	6 ± 0.93	None	6.75 ± 1.04	None	c. Criterion standard comparisons
11	7.13 ± 1.55	None	7.63 ± 0.92	None	d. Internal and external validity
12	6.38 ± 1.30	Wording change	7.63 ± 0.74	None	e. Determining significance
	5.38 ± 2.07	Removed			f. Understanding the literature in the era of social media*
	3.63 ± 2.72	Removed			i. Deciphering blog posts*
	4.88 ± 2.90	Removed			ii. Appraising the quality of various electronic resources*
13		Added	7.88 ± 0.99	None	g. Matching measurements with objectives and conclusions
14	7.5 ± 1.41	None	7.63 ± 1.41	None	III. Basic Statistical Concepts
15	7.13 ± 1.25	None	7.13 ± 1.25	None	a. Samples and populations
16	7.75 ± 1.16	None	7.25 ± 1.04	None	b. Hypothesis testing
17	7.13 ± 1.31	None	7.25 ± 1.04	None	c. p‐values
18	7.5 ± 1.31	None	7.38 ± 0.92	None	d. 95% CIs
19	6.88 ± 1.46	None	6.75 ± 1.28	None	e. Alpha (type I) errors
20	6.88 ± 1.46	None	6.75 ± 1.28	None	f. Beta (type 2) errors: power analysis, sample size
21	6.63 ± 1.41	None	7 ± 1.41	None	g. Specific statistical procedures (intro)
	6.38 ± 1.60	None	6.88	Removed	i. Student's t‐test*
	6.63 ± 1.60	None	6.5 ± 1.07	Removed	ii. Linear regression*
	6.5 ± 1.51	None	5.88 ± 1.25	Removed	iii. Logistic regression*
22	7.13 ± 1.36	None	6.88 ± 0.99	None	h. Measurement error and types of biases
23	7.5 ± 1.07	None	7.63 ± 0.92	None	i. Outcomes
24	7.13 ± 1.46	None	7.38 ± 1.41	None	ii. Sensitivity and specificity
25	6.5 ± 1.31	None	6.75 ± 1.49	None	iii. ORs
26	6.25 ± 1.28	None	6.63 ± 1.51	None	iv. Relative risk
27		Added	6.88 ± 1.25	None	v. Reliability/inter‐rater agreement
28		Added	6.38 ± 1.51	None	vi. Domains of validity for assessment instruments
29	7.38 ± 1.06	None	7.25 ± 1.98	None	IV. Specific types of research
30	6.88 ± 1.36	None	7.63 ± 1.77	None	a. Quantitative research
31	6.88 ± 1.55	None	7.25 ± 1.91	None	b. Qualitative research
32	6.88 ± 1.55	None	7.63 ± 1.51	None	c. Survey research
33	6.25 ± 0.89	None	6.75 ± 1.91	None	d. Meta‐analysis and systematic reviews
	5.5 ± 1.69	None	5.38 ± 1.77	Moved to special interest	e. Research on particular topics*
	6.88 ± 1.36	None	6.25 ± 1.75	Moved to special interest	i. Clinical research*
	5.25 ± 1.83	None	4.13 ± 2.2	Moved to special interest	ii. Basic science research*
	5.75 ± 1.49	None	5.5 ± 1.77	Moved to special interest	iii. Translational research*
	6.13 ± 1.89	None	6 ± 1.07	Moved to special interest	iv. Education research*
		Added	6.13 ± 1.73	Moved to special interest	1. Conceptual frameworks*
		Added	6 ± 1.41	Moved to special interest	2. Assessment methods*
	5.75 ± 1.49	None	5.25 ± 1.58	Moved to special interest	v. Population research*
34	7.63 ± 1.69	None	7.88 ± 1.81	None	V. Research ethics and the IRB
35	8 ± 0.93	None	7.63 ± 1.92	None	a. Informed consent
36	6.25 ± 1.28	None	6.13 ± 1.73	None	b. Equipoise and randomized trials
37	7.125 ± 1.64	Wording change	7.63 ± 1.69	None	c. Conflicts of interest
	3.13 ± 1.55	Removed			d. Animal research*
38	6.63 ± 1.51	None	7.25 ± 1.67	None	VI. Conducting research
	8.25 ± 1.39	Removed (redundancy)			a. Formulating an answerable question*
	8.125 ± 0.83	Removed (redundancy)			b. Searching the literature*
39	8.125 ± 0.99	None	7.5 ± 1.41	None	c. Selecting appropriate study design
40	7 ± 1.51	None	6.75 ± 1.49	None	d. Data management
41	8.13 ± 1.46	None	8.25 ± 1.16	None	e. Writing an abstract and a manuscript
42	8 ± 1.4	Wording change	8.13 ± 1.13	None	f. Publishing and presenting your work ‐ dissemination avenues
	5.75 ± 2.66	Removed			g. Securing funding*
43	6.63 ± 1.69	Wording change	7.25 ± 1.28	None	h. Peer review for acceptance to journals and meetings
44		Added	6.88 ± 1.64	None	i. Collaboration in research endeavors, including mentor/mentee relationships

^*Items eliminated or moved.

The expert panel consisted of eight emergency physicians from eight different institutions with diverse backgrounds including EM research directors, residency leaders, and national leaders in EM research and education (see acknowledgements). All experts had experience both in teaching EM residents and in the practical realities of research and, in many cases, in creating and implementing EM research curricula. Previous work has suggested that six to 10 experts is an appropriate number for obtaining stable results in the modified Delphi method. ⁶ , ⁷ , ⁸ Study data and Delphi polling were collected and managed using REDCap electronic data capturing tools hosted at the Wake Forest School of Medicine. ⁹ The participants were unaware of the identity of the other participants.

In the initial round, participants rated each item in the curriculum based on a Likert scale of 1 to 9, with 1 to 3 labeled “not important,” 4 to 6 labeled “somewhat important,” and 7 to 9 labeled “very important.” The experts were invited to make suggestions for editing the language of the curriculum items. Participants were also asked to suggest additional items for the curriculum if they felt that an important item was missing.

Between rounds the research team reviewed the results and revised the curriculum based on comments from the expert panel. It was decided a priori that any item receiving a mean rating of less than 3 would be dropped and any item with a mean rating greater than 7 should be included, with those falling in between subject to the judgment of the research team, with the aim of both adhering to the opinion of the experts and creating a coherent curriculum.

In the second round, participants were provided with aggregate response data as well as edited language as suggested by responses in the previous round. They were then asked to rate the items again and make further suggestions. Outcome measures include the mean rating of importance by our experts (on a scale of 1–9) for each checklist item, and the degree of agreement among experts, as calculated by Cronbach's alpha.

RESULTS

The response rate for each round was 100%. At the end of the first round, mean ratings for each item were evaluated and comments were assessed. Descriptive results for each item in round 1 can be seen in Table 1. Cronbach's alpha for round 1 was 0.82, indicating a good level of agreement. Based on the expert scores and the narrative comments, seven items were removed after round 1, including two for redundancy with other curriculum items. Six items had wording changes. The wording changes were largely efforts to broaden the scope of items to include all types of research relevant to EM. For example, an item containing the phrase “determining clinical significance” was amended to “determining significance.” Six items were added, including items related to collaboration in research work, as well as specific topics related to education research and measurement. Items were added if there seemed to be consensus among multiple experts or if those items seemed to add materially to the curriculum in the judgment of the research team. A second round was then undertaken with the amended item list.

After the second round, no further wording changes were made and final decisions on items to keep or discard were reached. Three items were removed, all of which were specific statistical tests that had a low percentage of expert responses labeling them very important. Similarly, relatively low levels of consensus were reached on items under the heading “research on particular topics.” Since those topics were individually felt to be very important to some, the category was split off into a “special interest” designation whereby the curriculum could be adapted to the needs and interests of individual resident learners. At this point, given a high degree of agreement and few suggested changes, it was felt that consensus had been achieved and further polling was halted. Cronbach's alpha for round 2 was 0.95, indicating an excellent level of agreement. After the consensus model curriculum was finalized, an annotated version with references for each item was created by the study team, which can be found in Figures 1 and S1.

Figure 1.

Model research curriculum for EM residents.

DISCUSSION

The curriculum produced here represents a consensus that fits broadly into the themes of previous and similar curricula, but also focuses more specifically on the EM resident learner and tools needed to navigate the world of EM research in the present day. The expert panel favored items that dealt with fundamental concepts of research as well as those that addressed practicalities of conducting sound, ethical investigations. They did not approve items that were more targeted toward electronic resources or that addressed literature discussion via social media, likely in the belief that fundamental understanding of EM research would translate to any arena. It is still possible that a robust methodology for preparing residents for interpreting literature in a social media context is needed, but falls outside the scope of this curriculum. Items that pertained to more advanced research skills, such as “securing funding,” were also not scored highly or consistently. There was some divergence of scores for items that addressed particular areas of research. Given this, and the likelihood that individual resident interest may drive what they choose to explore, it was felt best to place these particular areas in a separate category from the main curriculum.

Context for This Curriculum

Twenty years after the first model research curriculum for EM was published, ¹ an EM research primer was published by the American College of Emergency Physicians. ¹⁰ Both documents were written by research faculty for two different target audiences: respectively, research faculty initiating a research training program and research trainees considering an academic career. The current research curriculum is the first in EM to be conceived using the modified Delphi technique to build a consensus document for resident research curriculum. The original 1992 curriculum was broader in scope than what would be used by the majority of EM residency programs, while the ACEP research primer was not intended to be comprehensive, but rather “a short manual to kindle and motivate trainees and novice investigators.”

The current document is aimed at what can be accomplished over 3 or 4 years in a residency training program. It aims to produce clinicians who will be able to interpret and apply scientific literature to their patient care as well as academicians who may pursue a wide variety of EM subspecialty disciplines, including research.

One strength of the current curriculum compared to the first is the fact that it has been updated to reflect current curricular needs, balanced with feasibility. Second, the selection and priority of topics included in this curriculum were the result of a rigorous modified Delphi technique. Without such research techniques to build a balanced consensus, documents authored by traditional committees are subject to the potential bias of a minority opinion strongly voiced or repeatedly offered, dominating, or fatiguing the opinions of other committee authors.

Use of This Curriculum in a Research Elective

For residency programs that sanction electives during the residency training experience, many offer a research elective with mentoring by a research faculty member. Frequently centered around an individual research project, the work may be dedicated to data collection, data validation, data analysis, and manuscript preparation under the guidance of a research mentor over 4 weeks or a month‐long time frame. The current curriculum and its references, which list more general topics, can be used to round out the resident's educational experience, for what otherwise may be a narrowly focused course of study most pertinent to the research project. Some programs will want to include specific lectures on particular elements of this curriculum, while others may choose from the reference list of this curriculum as suggested reading or self‐study exercises during such an elective.

LIMITATIONS

This study and the resulting model curriculum have important limitations. First, it is essential to recognize that this is not meant to be an exhaustive accounting of all topics of interest related to EM research. Rather, this is intended as a consensus distillation of essential topics that should be covered during EM residency for those interested enough to pursue a research/scholarship elective. There may be important areas of research and scholarship, including quality improvement as an example, which are not addressed here but are essential to resident education. The modified Delphi technique also carries limitations. By starting with a previously identified topic list, it is possible that the research team influenced the outcome by creating inertia on the list for the expert panel. Interpretation of the results, and thus creation of the final topic list, involved some judgment decisions, but some degree of editorial discretion is inevitable using this technique. It is possible that a larger or differently composed panel may have yielded different results, but as previously discussed, ⁶ eight experts have been shown in the past to yield stable results and our panelists were selected for their expertise in both EM research and EM resident education. Some studies using the modified Delphi technique use additional rounds, but given the high degree of expert agreement and lack of substantive new suggestions in round 2, it was felt that two rounds were sufficient in this case.

CONCLUSIONS

Using the modified Delphi technique for consensus building, we propose a model curriculum for research for emergency medicine residency, with topics that can extend to form the basis of a research elective. We further provide a bibliography of sources and articles that can be used to build educational material for each topic.

The authors thank our expert panel—Holly Caretta‐Weyer, MD, Clinical Assistant Professor, Assistant Program Director, Stanford Department of Emergency Medicine; Wendy Coates, MD, Clinical Professor of Emergency Medicine, Senior Education Specialist, Harbor–UCLA Department of Emergency Medicine; D. Mark Courtney, MD, Professor, Executive Vice Chair for Academic Affairs, UT Southwestern Medical Center Department of Emergency Medicine; Michael Gottlieb, MD, RDMS, Assistant Professor, Director of Emergency Ultrasound, Rush Medical College Department of Emergency Medicine; Jaime Jordan, MD, Assistant Professor of Clinical Emergency Medicine, Associate Program Director, UCLA Ronald Reagan/Olive View; Simon Mahler, MD, MS, Professor, Research Director, Wake Forest Department of Emergency Medicine; Mark Mycyk, MD, Associate Professor, Research Director, Toxikon Consortium, Cook County Health and Hospitals System; and Colleen Smith, MD, Assistant Professor, Associate Program Director, Icahn School of Medicine at Mount Sinai Department of Emergency Medicine.

Supporting information

Figure S1. References model research curriculum for EM residents

AEM Education and Training 2021;5:1–8