Abstract
Background
Workflow efficiency (WFE) is essential to the practice of emergency medicine (EM), but a standardized approach to measuring and teaching it during residency is lacking. In this study we sought to describe how EM residency programs in the United States currently measure and teach WFE and to assess the relative importance of WFE teaching to EM residency program leaders.
Methods
We conducted a cross‐sectional survey of all accredited EM residency training programs in the United States in Fall 2019. We invited all allopathic EM residency programs to participate in the study by directly emailing program directors and assistant/associate program directors. We conducted the study and performed descriptive statistics using SurveyMonkey software.
Results
We received a total of 133 responses out of 190 total programs (70%) with proportionate representation from 3‐ and 4‐year programs and all regions of the United States. When asked to what extent teaching efficiency should be a priority compared to other educational goals, 65% of program leaders responded with “significant” or “moderate” priority. Most EM programs collect WFE data on their residents, either by tracking patients per hour (78%) or by written evaluations (59%). Common methods for providing WFE data to residents were: “individual data provided along with deidentified rank” (35%), “data provided only during private feedback meetings” (26%), and “no data or rank provided to residents” (16%). Regarding targeted WFE teaching to residents, 88% reported utilizing general on‐shift teaching, 48% reported teaching WFE during formal didactics, and 45% during dedicated private feedback sessions.
Conclusion
This national study of allopathic U.S. EM programs suggests that most EM program leaders do value WFE teaching. However, we found no consistent approach among programs for tracking or distributing resident WFE data, and many programs lack a formalized way to teach efficiency to their residents.
INTRODUCTION
Emergency medicine (EM) physicians must simultaneously care for large numbers of patients, while also often juggling supervising residents and advanced practice providers, placing high demands for workflow efficiency (WFE) on providers. 1 Venugopal et al. 2 define WFE as the “ability to manage multiple Emergency Department (ED) patients through multi‐tasking and strategic interventions, expedite treatment and disposition decisions without compromising safety, quality of care or documentation.” These skills include managing a high average patient load, leading multidisciplinary teams, performing direct patient care and documentation, and “running the board” while avoiding extraneous tasks. 3
If EM physicians are expected to safely care for high numbers of patients per shift, it follows that WFE is a desirable outcome of residency training. While the Accreditation Council for Graduate Medical Education (ACGME) indirectly mentions aspects of WFE in their core competencies, they do not provide a way to meet these requirements and currently there is no standard approach to measuring and teaching this skill set during residency. 4 Furthermore, the Model for Clinical Practice of Emergency Medicine (EM Model) put forth by the American Board of Emergency Medicine (ABEM) mentions physician tasks such as multiple patient care, further supporting the need for creating a standardized approach to teaching WFE. 5 While previous studies have demonstrated that EM residents see more patients as they advance through training, the specific ways in which WFE skills develop and are tracked over time have not been described. 6 , 7
To address this gap in understanding we developed a survey targeting the following questions: (1) How do EM residency programs in the United States currently teach and measure WFE and (2) What importance do EM residency program leaders assign to WFE education in comparison to other educational goals? It is our hope that a clearer understanding of current educational practice with regard to WFE might inform the needs assessments of programs seeking to strengthen their approach to this key area of clinical training.
METHODS
Study design and population
We performed a cross‐sectional survey of ACGME‐accredited allopathic EM residency programs in the United States. We utilized an electronic survey to collect information from program leaders regarding WFE measurement and education. We conducted the study and performed descriptive statistics using SurveyMonkey software (SVMK Inc.).
We were unable to find a validated survey examining WFE practices that we could use in our study and therefore created our own de novo. Our survey consisted of multiple‐choice and short‐answer items. For multiple‐choice items where an “other” choice was available, participants were permitted to enter a free‐text response. To ensure sound best practices in our survey design, we utilized an evidence‐based approach when creating our questionnaire. 8 , 9 This involved careful crafting of our goals and objectives congruent with our survey method, conducting a thorough literature review on the topic, soliciting expert input on our survey items and conducting an online pilot of our survey before its launch.
Each member of the research group reviewed the survey for content validity and response process with at least one subject matter expert, including EM program directors (PDs). The authors then piloted the survey with a small group of EM faculty for usability and design of the electronic survey and gathered feedback using a think‐aloud technique. 9 After recording suggestions from the pilot participants, the research group reconvened to edit the final survey for distribution. Items that were flagged as ambiguous, difficult to answer, or minimally related to the constructs of interest were revisited and revised by the study group.
The target population for the study was program leaders (i.e., PDs, assistant and associate program directors [APDs]) at allopathic EM residency programs in the United States. In Fall 2019, we identified ACGME‐accredited EM training programs through their accreditation data system, yielding a total of 190 programs. To prevent duplication, we invited only one member of the program leadership from each program based on available contact information with preference for seniority (i.e., PD over assistant PDAPD).
We identified contact information for potential participants through the ACGME accreditation data system, Society for Academic Emergency Medicine Residency Directory, searches of program websites, and personal knowledge by study team members. We collected data between Fall 2019 and Winter 2020. During the last 2 months of data collection, if the PD had not responded, we then invited the APD(s) listed on the program website. No compensation was provided for study participation.
Survey content and administration
This brief electronic survey consisted of eight multiple‐choice items and one short‐answer item. We geared the survey items toward the following: the collection and distribution of WFE metrics, methods for teaching WFE to residents, and the relative importance of WFE compared to other educational goals. Item selection choices were developed collaboratively among the study authors and piloted with a small group of PDs to include as many expected answer choices as possible. We included the following demographic variables for participating programs: geographic region (Western, Southern, Midwest, and Northeast), residency program format (3‐ or 4‐year program and/or combined program), and the academic role and years of experience of survey respondents. To ensure confidentiality we did not collect data that would identify individual programs.
The final survey instrument is available for viewing (Appendix S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10598/full). To incorporate all available data and maximize response rate, completion of all survey questions was not required. The study was certified as exempt by the institutional review board of ChristianaCare. We contacted EM program leaders via emails containing individual links to the survey. Program leaders were made aware of the purpose of the study and that participation would be voluntary and anonymous.
Data analysis
Survey responses were tracked on the SurveyMonkey platform and raw data were downloaded for analysis. We performed descriptive statistics for multiple‐choice items utilizing the SurveyMonkey platform software, which automatically summarized individual survey responses into usable graphs. Free‐text responses were reviewed and analyzed by the author group to generate response themes, followed by tabulating those themes to identify patterns for discussion.
RESULTS
Survey response rate and characteristics of survey respondents
A total of 142 of 190 allopathic EM programs (70%) responded to the survey. We found nine instances of duplicate responses from the same program (PD and APD or multiple APDs). When this occurred, only the data provided by the most senior program leader were utilized for analysis. Roughly 95% of survey respondents answered every item on the survey.
The demographic characteristics of survey respondents are summarized in Table 1. Seventy‐four percent of respondents were from 3‐year categorical programs, 25.76% were from 4‐year categorical programs, and of these programs 4.54% had a combined program. The vast majority of respondents (92%) were PDs with the remaining being assistant PDs or APDs. Seventy‐four percent of respondents reported being in their current position for 2 or more years. The geographic distribution of programs was roughly representative of the landscape of allopathic EM residencies in the United States, with 15.15% of responses from the Western United States, 26.52% from the Midwest, 25.76% from the Southern United States, and 32.58% from the Northeastern United States.
TABLE 1.
Demographics of participating residency programs. Type of residency program was listed as “choose all that apply,” while the other questions only allowed for one selection
| Survey components | No. of respondents (%) |
|---|---|
| Survey setting | |
| Northeastern | 43 (32.58%) |
| Midwest | 35 (26.52%) |
| Southern | 34 (25.76%) |
| Western | 20 (15.15%) |
| Type of residency program (choose all that apply) | |
| Categorical (3‐year EM) | 98 (74.24%) |
| Categorical (4‐year EM) | 34 (25.76%) |
| EM/IM | 3 (2.27%) |
| EM/FM | 3 (2.27%) |
| Respondents | |
| PD | 122 (92.42%) |
| APD | 8 (6.06%) |
| Assistant PD | 2 (1.52%) |
| Time at current position (y) | |
| <2 | 35 (26.52%) |
| 2–5 | 54 (40.91%) |
| >5 | 43 (32.58%) |
Abbreviations: APD, associate program director; FM, family medicine; IM, internal medicine; PD, program director.
WFE metrics
The various types of WFE collected by EM programs are summarized in Figure 1. The majority of programs (91.73%) reported gathering at least one type of efficiency metric, while 11 programs (8.27%) reported gathering no efficiency data. The top two WFE metrics gathered by programs were patients per hour (PPH; 78.2%) and written faculty evaluations on efficiency (58.65%). Standardized direct observation assessment tool (SDOT) or other structured observation tools by faculty to assess WFE among residents were reported by 30 programs (22.56%). Relative value units (RVUs) per hour, a ubiquitous efficiency metric for practicing EM physicians, were tracked among residents by 16 programs (12.03%). Additional WFE data reported by EM programs included critical care charting, underbilling, coding level, site seen (resuscitation area vs. main ED vs. pediatrics), “room to doctor” time, admission rates, and CT utilization rates. The majority of programs reported providing WFE data in some form to their residents (Figure 2). The most common methods for sharing WFE data with residents were via deidentified class rank (46 programs, 34.85%), private feedback meetings (34 programs, 25.76%), and individual WFE data without class rank (15 programs, 11.36%). Twenty‐one programs (15.91%) reported that they do not provide WFE data to their residents.
FIGURE 1.
Methods for WFE data collection among allopathic EM training programs in the U.S. survey respondents were allowed to report all methods of data collection they currently employ. RVUs, relative value units; WFE, workflow efficiency
FIGURE 2.
Methods used among allopathic EM training programs to share WFE data with residents. Respondents were allowed to report all methods for data sharing currently employed by their program. For the option labeled “other,” short‐answer free text was allowed and a summary of comments can be found in the body of the paper. WFE, workflow efficiency
Teaching WFE
The ways in which EM programs teach WFE to residents are depicted in Figure 3. By far the most commonly reported approach to WFE education was through on‐shift teaching and feedback (116 programs, 87.88%). The most commonly reported methods for teaching WFE outside of the clinical environment were through conference didactics (64 programs, 48.48%) and during one‐on‐one feedback sessions (59 programs, 44.70%).
FIGURE 3.
Methods used among allopathic EM training programs to teach WFE to residents. Respondents were allowed to report all teaching methods currently employed by their program. For the option labeled “other,” short‐answer free text was allowed and a summary of comments can be found in the body of the paper. WFE, workflow efficiency
Relative importance of WFE among programs’ educational goals
The majority (65.16%) of program leaders rated WFE as being of “moderate” or “significant” importance in the training of residents, relative to other educational goals (Figure 4). Only one program leader responded that WFE had no priority in residency education.
FIGURE 4.
Relative importance of teaching WFE in comparison to other educational goals, as reported by program leadership of allopathic EM training programs in the U.S. WFE, workflow efficiency
Program leaders’ suggestions for teaching WFE
We included a short‐answer question in our survey to garner ideas of how respondents imagine WFE should be taught in an ideal setting. There were 79 responses from program leaders imagining how this could be done. Twenty‐two responses expressed the desire to provide more direct observation of residents by experienced faculty or senior residents. Several program leaders (20 comments) expressed the desire to be able to better track WFE metrics among individual residents to provide more targeted feedback. Thirteen respondents also expressed the desire for simulation‐ or workshop‐based approaches to teaching efficiency, with one respondent wishing for a “choose your own adventure” style of game as an educational method.
Multiple respondents acknowledged the difficulty of teaching efficiency in an ED environment filled with efficiency barriers and expressed the desire for a better method for teaching WFE to residents. Representative quotations included the following:
"If something were proven effective, easily accessible, and could be practically applied, we would implement it."
"Any methods would be an improvement; we are definitely open to the idea."
"I think having access/creating an evidence‐based curriculum around efficiency would be so helpful. Each department is different but understanding general principles would be useful."
DISCUSSION
Prior research has evaluated multitasking and task‐switching in the ED, skills that are recognized by the ACGME as essential to the practice of EM. 10 , 11 , 12 A number of studies have shown that resident efficiency measured with common metrics such as PPH, ED length of stay, and resident RVUs improves with each PGY level. 6 , 7 , 13 , 14 A previous study also showed that efficiency improved with increased ED crowding, although decreased for ED overcrowding. 15 However, the ways in which EM training programs specifically teach and measure WFE to residents have not been studied previously. We believe that this cross‐sectional survey provides an accurate snapshot of the educational practices of allopathic EM programs with regard to measuring and teaching WFE.
We found that the large majority of training programs collect resident WFE data, most commonly in the form of PPH or written shift evaluations. While the majority of EM programs reported collecting WFE data in some form, we found variability in how these data are provided to residents. Given that WFE metrics such as PPH are both ubiquitous and heavily emphasized in community practice, it is concerning that a more cohesive approach to reporting WFE to residents has not been embraced among EM training programs. 16 With the multifaceted nature of patient care and competing demands placed on residents’ time and attention in the clinical environment, direct observation is perhaps the criterion standard for assessing WFE. One observational time‐and‐motion study found that resident physicians spend 32% of their time on direct patient care, 47% on indirect patient care (such as charting, talking on the phone, and gathering supplies), and 21% on non–patient care tasks. 17 Ideally, educators could use information such as this to give targeted feedback to residents on areas to develop greater efficiency. While appealing, this approach has understandable practical limitations in terms of time and cost. Perhaps unsurprisingly, a minority of programs (22%) reported using direct observation shifts to assess WFE among residents.
While a number of publications have provided global recommendations for how residents can improve WFE, 3 , 18 , 19 , 20 , 21 the specific approaches utilized by residency programs to teach these skills have not been described. Our study identified on‐shift feedback as the most popular approach to teaching WFE, reported by 88% of respondents. Less than 50% of programs reported having dedicated didactic time for teaching WFE. Prior studies have shown that WFE‐directed teaching leads to measurable improvements in practice and have recommended that residency training programs “may benefit from dedicated curricular planning …. [utilizing] problem‐based learning seminars, simulations, or workshops.” 22 , 23 Despite the potential of such interventions to improve residents’ WFE, our survey showed that only 15% and 7% of programs utilized simulations and workshops, respectively, to teach efficiency.
Our final question examined the relative importance program leaders give to WFE in relation to other educational goals, with the majority of respondents stating that WFE training is of moderate or significant importance. In spite of the apparent value placed on WFE teaching, pedagogical approaches for teaching WFE have emphasized informal workplace teaching over structured didactics. Chan et al. 24 discussed five strategies for teaching flow management and efficiency through coaching and “think‐aloud” modeling techniques. Another paper by Chan et al. 25 discussed methods for coaching during chaotic environments, identifying two main types of strategies to teach ED management: (1) workplace‐based methods, including both observation and in situ instruction, and (2) principle‐based advice. Our literature review revealed only two curricula that offered workshops to teach efficiency 2 , 26 and one board game–themed tabletop simulation. 27
While there is an overall paucity of resources on teaching efficiency, there are numerous possible ways this could be accomplished. In addition to the activity‐based methods of simulation, workshops, and board games, other small‐group sessions could have success using targeted teaching methods and goal‐oriented outcomes. In the absence of focused one‐on‐one observation, ethnographic evaluation using time‐motion and foot‐traffic data could be a useful tool to gather data on inefficiencies for targeted teaching. Affective methods for teaching efficiency could be tapped into using self‐reported tools that gather feedback from the resident both during and after shifts, followed by targeted responses by faculty. Finally, in a shortage of faculty and limited finances, didactics could be built into resident conference time in order to teach high‐yield efficiency practices as a way to engaging a larger audience.
LIMITATIONS
This was a survey‐based study, and the results are subject to the limitations inherent to this type of data collection. Because this was a cross‐sectional study, only one period of time was evaluated, and it is possible that results would vary if multiple years were incorporated longitudinally. Additionally, data were collected from only one member of each program's residency leadership. This approach might have contributed to a framing bias, and it is possible that we would have found different results had residents or a broader swath of faculty been surveyed. Survey answers were anonymous and confirmation of the accuracy of individual program data was not possible. We attempted to mitigate this bias by targeting senior program leadership, with 92% of respondents being PDs. Furthermore, the anonymity and lack of ability to confirm may have also contributed toward a social desirability bias, wherein an overestimation of current measuring and teaching of efficiency could have been selected. In addition, many of the variables utilized in our survey and selected by responders involve subjective rather than objective measurements of efficiency, data distribution, and teaching.
While the response rate to our survey was 70% with a representative distribution both geographically and of 3‐ and 4‐year programs, our data may not entirely reflect the educational practices of EM residency programs with respect to tracking and teaching WFE and may also not reflect the practices of osteopathic programs, as well.
CONCLUSION
The majority of emergency medicine residency programs track and distribute residents’ workflow efficiency data, but their sources of data and methods of dissemination vary widely. While emergency medicine program leaders identify workflow efficiency as being of moderate to high importance educationally, few programs offer a cohesive strategy for helping learners develop this essential skill set. It is still unclear how workflow efficiency can best be taught, leaving room for future projects to further elucidate.
CONFLICT OF INTEREST
The authors have no potential conflicts to disclose.
AUTHOR CONTRIBUTIONS
Guy Carmelli, Erin E. Watson, Nadia A. Villarroel, William W. Dixon, and Samuel O. Clarke contributed to the study concept and design, acquisition of the data, analysis and interpretation of the data, drafting of the manuscript, and critical revision of the manuscript.
Supporting information
Appendix S1. The MERC at CORD Resident Workflow Efficiency in Emergency Medicine.
Presented virtually at the American College of Emergency Physicians Scientific Assembly, November 2020.
Supervising Editor: Daniel Runde, MD.
REFERENCES
- 1. Hooker EA, Mallow PJ, Oglesby MM. Characteristics and trends of emergency department visits in the United States (2010–2014). J Emerg Med. 2019;56:344‐351. [DOI] [PubMed] [Google Scholar]
- 2. Venugopal R, Lang E, Doyle K, Sinclair D, Unger B, Afilalo M. A workshop to improve workflow efficiency in emergency medicine. CJEM. 2008;10:525‐531. [PubMed] [Google Scholar]
- 3. Bobb MR, Ahmed A, Van Heukelom P, et al. Key high‐efficiency practices of emergency department providers: a mixed‐methods study. Acad Emerg Med. 2018;25:795‐803. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. ACGME Common Program Requirements. Accreditation Council for Graduate Medical Education website. July 1, 2020. Accessed December 4, 2020. https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2020.pdf.
- 5. Beeson MS, Ankel F, Bhat R, et al. The 2019 Model of the Clinical Practice of Emergency Medicine. J Emerg Med. 2020;59:96‐120. [DOI] [PubMed] [Google Scholar]
- 6. Henning DJ, McGillicuddy DC, Sanchez LD. Evaluating the effect of emergency residency training on productivity in the emergency department. J Emerg Med. 2013;45:414‐418. [DOI] [PubMed] [Google Scholar]
- 7. Brennan DF, Silvestri S, Sun JY, Papa L. Progression of emergency medicine resident productivity. Acad Emerg Med. 2007;14:790‐794. [DOI] [PubMed] [Google Scholar]
- 8. Rickards G, Magee C, Artino AR Jr. You can't fix by analysis what you've spoiled by design: developing survey instruments and collecting validity evidence. J Grad Med Educ. 2012;4:407‐410. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Artino AR Jr, La Rochelle JS, Dezee KJ, Gehlbach H. Developing questionnaires for educational research: AMEE Guide No. 87. Med Teach. 2014;36(6):463‐474. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Smith D, Miller DG, Cukor J. Can simulation measure differences in task‐switching ability between junior and senior emergency medicine residents? West J Emerg Med. 2016;17:149‐152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Ratwani RM, Fong A, Puthumana JS, Hettinger AZ. Emergency physician use of cognitive strategies to manage interruptions. Ann Emerg Med. 2017;70:683‐687. [DOI] [PubMed] [Google Scholar]
- 12. Skaugset LM, Farrell S, Carney M, et al. Can you multitask? Evidence and limitations of task switching and multitasking in emergency medicine. Ann Emerg Med. 2016;68:189‐195. [DOI] [PubMed] [Google Scholar]
- 13. Dowd MD, Tarantino C, Barnett TM, Fitzmaurice L, Knapp JF. Resident efficiency in a pediatric emergency department. Acad Emerg Med. 2005;12:1240‐1244. [DOI] [PubMed] [Google Scholar]
- 14. DeBehnke D, O'Brien S, Leschke R. Emergency medicine resident work productivity in an academic emergency department. Acad Emerg Med. 2000;7:90‐92. [DOI] [PubMed] [Google Scholar]
- 15. Kirby R, Robinson RD, Dib S, et al. Emergency medicine resident efficiency and emergency department crowding. AEM Educ Train. 2019;3:209‐217. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Vukmir RB, Howell RN. Emergency medicine provider efficiency: the learning curve, equilibration and point of diminishing returns. Emerg Med J. 2010;27:916‐920. [DOI] [PubMed] [Google Scholar]
- 17. Hollingsworth JC, Chisholm CD, Giles BK, Cordell WH, Nelson DR. How do physicians and nurses spend their time in the emergency department? Ann Emerg Med. 1998;31:87‐91. [DOI] [PubMed] [Google Scholar]
- 18. Egan HM, Swanson MB, Ilko SA, Pomeranz KA, Mohr NM, Ahmed A. High‐efficiency practices of residents in an academic emergency department: a mixed‐methods study. AEM Educ Train. 2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Sadosty A, Kruse B, Vadeboncoeur T. Five simple steps to improve an emergency physician's efficiency. Am J Emerg Med. 2008;26:1056‐1057. [DOI] [PubMed] [Google Scholar]
- 20. Denny CJ, Steinhart BD, Yu R. Improving physician flow and efficiency in the emergency department. CJEM. 2003;5:271‐274. [DOI] [PubMed] [Google Scholar]
- 21. Campbell SG, Sinclair DE. Strategies for managing a busy emergency department. CJEM. 2004;6:271‐276. [PubMed] [Google Scholar]
- 22. Jain S, Frank G, McCormick K, Wu B, Johnson BA. Impact of physician scorecards on emergency department resource use, quality, and efficiency. Pediatrics. 2015;136:ve670‐9. [DOI] [PubMed] [Google Scholar]
- 23. Chan TM, Van Dewark K, Sherbino J, Schwartz A, Norman G, Lineberry M. Failure to flow: an exploration of learning and teaching in busy, multi‐patient environments using an interpretive description method. Perspect Med Educ. 2017;6:380‐387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Chan TM, Sherbino J, Welsher A, Chorley A, Pardhan A. Just the facts: how to teach emergency department flow management. CJEM. 2020;22:459‐462. [DOI] [PubMed] [Google Scholar]
- 25. Chan TM, Van Dewark K, Sherbino J, Lineberry M. Coaching for chaos: a qualitative study of instructional methods for multipatient management in the emergency department. AEM Educ Train. 2019;3:145‐155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Pitre C, Pettit K, Ladd L, Chisholm C, Welch JL. Physician time management. MedEdPORTAL. 2018;14:10681. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Huang SY, Sneath P, Tsoy D, et al. P073: The GridlockED board game: using serious games for medical education. CJEM. 2018;20:S82‐S83.28693650 [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Appendix S1. The MERC at CORD Resident Workflow Efficiency in Emergency Medicine.