Medical Education Scholarship During a Pandemic: Time to Hit the Pause Button, or Full Speed Ahead?

Many journal editors are reporting a surge in manuscript submissions that coincides with the COVID-19 pandemic. In April and May 2020, the Journal of Graduate Medical Education (JGME) received more than twice the number of submissions in comparison to the prior year, and the excess papers did not all concern the pandemic. While some clinicians working in emergency medicine, intensive care, and hospital specialties were flooded with increased clinical care in 2020, we suspect that other educators discovered new time for writing as they sheltered in place. Or perhaps there are other confounders for this rise in submissions, such as the impetus provided by thoughts of our own mortality when surrounded by illness and death. Nothing focuses the mind so well as thoughts that our time may be limited.

At this time, we are struck with a simultaneous need for research on what works in medical education during a crisis and on the difficulty of conducting research under indefinite pandemic circumstances. Fortunately, medical education is one the few types of research not paused by institutions and governments. Let's consider what questions we most need to answer now, and what strategies for developing credible evidence to support these answers may be used.¹ Given our perspectives as “editor-educators,” these ideas are inevitably limited by our own contexts. We anticipate creative grassroots research endeavors, not listed below, which will produce new contributions to graduate medical education (GME).

What Are the Questions?

The JGME submission system is overflowing with descriptions of new resident deployment mechanisms, virtual educational activities, institutional and program communication systems, and ethical decision-making frameworks to balance needs for service, education, safety, and public health. These descriptions are highly useful in the first few weeks of a pandemic, to provide guidance to programs that are entering the early phases of a disaster. However, these innovations do not determine effectiveness or evaluate for success. Nor do these descriptions advance more fundamental understandings of why particular observations occur (or do not occur) in GME contexts. An essential question often remains unspecified in these early phase manuscripts: What would define “success”? We suggest these 6 questions, also listed in Box 1.

Box 1 Potential Scholarship Questions for a Pandemic

• What do prior studies show us about graduate medical education (GME) during pandemics and other disasters?

• What are the most important GME-related outcomes and how do we measure them during a pandemic?

• Can residents and fellows achieve the minimum competencies they need for graduation, during reduced clinical activities and a focus on one infectious disease?

• Can virtual teaching methods achieve “equivalent” learning outcomes to live teaching? Or, what is the minimum amount of clinical experience for learning?

• What are evidence-based strategies to strengthen professionalism, resilience, and well-being for trainees, faculty, and key staff during a pandemic?

• With the current funding of GME by a mixture of federal, state, and other sources, what are the economic effects of the pandemic on GME programs?

1. What do prior studies reveal about GME during pandemics and other disasters?

Reviews of the literature within and outside medicine may be useful to identify the quality of prior work, best practices, and remaining questions for future work. A literature search is the starting place for determining what we know vs. what we need to know. Prior work may not be exactly applicable, yet a deeper understanding and synthesis of past insights and methods, whether successful or unsuccessful, may provide useful information for educators and researchers.^2–4 Consider also developing a curated collection of relevant articles, which may provide a foundation for further study. “The science of curation is a niche in itself and the future will see increasing rigor in critical appraisal methods.”⁵

2. What are the most important GME-related outcomes and how do we measure them during a pandemic?

Based on current submissions to JGME, educators appear most concerned with these outcomes: continuity of formal education; adequate supervision of trainees; achievement of competencies—particularly those beyond knowledge; resident safety; and resident well-being and burnout. For quantitative studies, researchers will need to precisely define these outcomes and measurement methods in a rapidly changing environment and over time—longitudinally. Measuring acceptability of new interventions is important but not enough to determine benefit. The “low-hanging fruit” of in-training and board certification examinations will undoubtedly be used to measure knowledge outcomes for pandemic vs. non-pandemic cohorts; knowledge assessed by tests will not be sufficient to inform actions for the next disaster.

Pre- and immediate posttests of knowledge for new virtual teaching modes are “proof of efficacy” studies and usually not helpful: bright, motivated GME trainees will learn regardless of how you teach.⁶ As the pandemic waxes and wanes, there may be more opportunities to compare virtual vs. live teaching strategies. Similarly, cross-sectional measurements in GME trainees of well-being, resilience, stress, and burnout add little to our understanding of the best approaches to strengthen resident and fellow—and faculty—abilities to learn during times of great changes and stress. Qualitative research or realist methods may be more conducive to determining what works, when, and with whom.⁷

Descriptions of virtual assessments, including virtual observed structured clinical examinations (vOSCEs) for resident graduation requirements, are appearing. Will virtual assessments adequately replace live assessments, such as direct observation, 360-degree reviews, and actual OSCEs? Will simulated procedures adequately substitute for participation in patient procedures? In what assessment measures will we put our faith and future patients' welfare?

3. Can residents and fellows achieve minimum competencies for graduation, after months of closed clinics, canceled elective procedures, and a shift of training toward one infectious disease?

It has also been suggested that the heightened focus on patient welfare and teamwork during the pandemic will enhance learning, with more trainees achieving “aspirational” milestones. As clinical activities re-open, many sites are reducing the number of patients, to allow physical distancing and time for additional cleaning, which will continue to affect educational opportunities.

This question of competency achievement (and sustaining this achievement) is really the question for GME, particularly if residency and fellowship experiences continue to be skewed or diluted over the next year or longer. Much has been written about time-based vs. competency-based length of training; these articles often focus on trainees accomplishing competencies earlier. Will the pandemic require longer training times to achieve minimal competence, for more trainees? Will post-graduation learning need to be enhanced? Will medical students, moved to “virtual clerkships,” require greater supervision by faculty when they enter residencies in 2020, 2021, and 2022? Or is there enough discretionary time in medical school to shift from electives and scholarship to essential clinical experiences, as the pandemic ebbs and flows?

4. Can virtual teaching methods achieve “equivalent” learning outcomes to live teaching?

Or, how little clinical time is needed for learning, with a robust virtual learning program? For many programs, some clinical experiences are already limited due to emerging technologies, reduced work hours, competition from other health professions trainees, and other factors. Some trainees may be able to move swiftly from virtual or simulation activities to actual patients and require little additional practice. Others will need more time with attending role models, live patients, and actual procedures.

Professionalism and interpersonal communication skills seem least likely to fare well with virtual methods, whereas practice-based learning and improvement and systems-based knowledge competencies may translate easily. We hope that each specialty not only develops virtual teaching materials, but also expends additional efforts to compare these to other models. There has been a rush to create virtual materials, without taking the time to determine how well they work and how to best integrate into other experiences. Faculty are exhausted by the efforts to create and use the new materials. Given their potential benefits in time and cost, which experiences should remain virtual after the pandemic recedes?

5. What are evidence-based strategies to strengthen professionalism, resilience, and well-being, for trainees, faculty, and key staff during a pandemic?

Nearly every paper we have reviewed has stressed the importance of interventions to promote trainee well-being: Which of these strategies worked best and should be implemented in future disasters, or in 2021, if COVID-19 rises again? Daily group Zoom check-ins vs. chief resident daily emails vs. mental health specialist outreach vs. trainee family education vs. free food and housing vs. thank-you parades—programs need to compare and determine those most useful, especially in resource-constrained times. It seems as if any reasonable idea has been tried, can we determine which are helpful?

In contrast, we have seen much less written about faculty burnout: faculty have transformed curricula overnight from precepting and conferences to online materials and methods, and usually must manage substantial ongoing patient responsibilities. The onslaught of urgent curriculum conferences and emails is overwhelming: What works best for maintaining faculty well-being over time?

6. With the current funding of GME by a mix of federal (Medicare), state (Medicaid), and other sources, what are the economic effects of the pandemic on GME programs?

Many programs already have a tenuous hold on solvency, particularly those in smaller, underserved, or rural communities. Hospital consolidations—followed sometimes by GME closures—are a concern. Hospitals have increased residency positions beyond pre-established caps, although usually with clinical revenues and in specialties that are highly remunerative, rather than in shortage fields: primary care, general surgery, and psychiatry. Will new positions be jettisoned in the future? In the United States and around the world, those in the health professions have been heralded as “heroes” and the value of trainees in pandemic “hot spots” has been absolutely clear. However, this won't pay the bills and the dire straits of some state budgets may not permit GME bailouts. Research demonstrating the economic benefits of GME may be an even more critical area of study, at this time.

Potential Strategies

As always, the nature of the questions asked will determine the choice of methods: quantitative, for measuring and comparing outcomes, evaluating the impact of exposures on outcome, and testing hypotheses; qualitative, for phenomena that are poorly understood and to explore the lived experiences of educators and learners, and generate hypotheses; and mixed methods, for questions that benefit from both approaches. This article does not review how to conduct high-quality medical education research (see Box 2 for JGME resources, which also link to other key resources). We are suggesting strategies for working during these unpredictable times.

Box 2 JGME Resources for Medical Education Scholarship.

General

Podcast: JGME Instructions for Authors: Tips for Stress-free, Successful Submissions to JGME. http://journalofgme.libsyn.com/

Sullivan GM. Resources for clinicians becoming clinician educators. J Grad Med Educ. 2015;7(2):153–155. doi:10.4300/JGME-D-15-00098.1.

Maggio LA, Sewell JL, Artino AR Jr. The literature review: a foundation for high-quality medical education research. J Grad Med Educ. 2016;8(3):297–303. doi:10.4300/JGME-D-16-00175.1.

Picho K, Artino AR Jr. 7 deadly sins in educational research. J Grad Med Educ. 2016;8(4):483–487. doi:10.4300/JGME-D-16-00332.1.

O'Brien BC, Balmer DF, Maggio LM. Finding our way through shades of gray: 6 virtues to guide researchers in planning, conducting, and writing up research. J Grad Med Educ. 2017;9(5):555–559. doi:10.4300/JGME-D-17-00546.1.

Analysis

Sullivan GM. A primer on the validity of assessment instruments. J Grad Med Educ. 2011;3(2):119–120. doi:10.4300/JGME-D-11-00075.1.

Sullivan GM, Feinn R. Using effect size—or why the P value is not enough. J Grad Med Educ. 2012;4(3):279–282. doi:10.4300/JGME-D-12-00156.1.

Sullivan GM. FAQs about effect size. J Grad Med Educ. 2012;4(3):283–284. doi:10.4300/JGME-D-12-00162.1.

Sullivan GM, Artino AR Jr. Analyzing and interpreting data from Likert-type scales. J Grad Med Educ. 2013;5(4):541–542. doi:10.4300/JGME-5-4-18.

Assessing Quality

Podcast: Publish or Perish, Is There a Paper in Your Poster? http://journalofgme.libsyn.com/

Blanchard BD, Nagler A, Artino AR Jr. Harvest the low-hanging fruit: strategies for submitting educational innovations for publication. J Grad Med Educ. 2015;7(3):318–322. doi:10.4300/JGME-D-15-00228.1.

Institutional Review Board (IRB)

Sullivan GM. Education research and human subject protection: crossing the IRB quagmire. J Grad Med Educ. 2011;3(1):1–4. doi:10.4300/JGME-D-11-00004.1.

Sullivan GM. IRB 101. J Grad Med Educ. 2011;3(1):5–6. doi:10.4300/JGME-D-11-00005.1.

Qualitative Methods

Sullivan GM, Sargeant J. Qualities of qualitative research: part I. J Grad Med Educ. 2011;3(4):449–452. doi:10.4300/JGME-D-11-00221.1.

Sargeant J. Qualitative research part II: participants, analysis, and quality assurance. J Grad Med Educ. 2012;4(1):1–3. doi:10.4300/JGME-D-11-00307.1.

Varpio L, Artino AR Jr. Answering the mail: replying to common questions about qualitative inquiry. J Grad Med Educ. 2015;7(4):667–668. doi:10.4300/JGME-D-15-00416.1.

Teherani A, Martimianakis T, Stenfors-Hayes T, Wadhwa A, Varpio L. Choosing a qualitative research approach. J Grad Med Educ. 2015;7(4):669–670. doi:10.4300/JGME-D-15-00414.1.

Nimmon L, Paradis P, Schrewe B, Mylopoulos M. Integrating theory into qualitative medical education research. J Grad Med Educ. 2016;8(3):437–438. doi:10.4300/JGME-D-16-00206.1.

Baker L, Phelan S, Snelgrove R, Varpio L, Maggi J, Ng S. Recognizing and responding to ethically important moments in qualitative research. J Grad Med Educ. 2016;8(4):607–608. doi:10.4300/JGME-D-16-00384.1.

Balmer DF, Rama JA, Martimianakis MA, Stenfors-Hayes T. Using data from program evaluations for qualitative research. J Grad Med Educ. 2016;8(5):773–774. doi:10.4300/JGME-D-16-00540.1.

Watling C, Cristancho S, Wright S, Varpio L. Necessary groundwork: planning a strong grounded theory study. J Grad Med Educ. 2017;9(1):129–130. doi:10.4300/JGME-D-16-00693.1.

Varpio L, Meyer H. A lesson from the qualitative Rip Out series: let go of expectations for universally applicable “gold standards” for qualitative research. J Grad Med Educ. 2017;9(2):154–156. doi:10.4300/JGME-D-17-00014.1.

Surveys

Rickards G, Magee C, Artino AR Jr. You can't fix by analysis what you've spoiled by design: developing survey instruments and collecting validity evidence. J Grad Med Educ. 2012;4(4):407–410. doi:10.4300/JGME-D-12-00239.1.

Magee C, Byars L, Rickards G, Artino AR Jr. Tracing the steps of survey design: a graduate medical education research example. J Grad Med Educ. 2013;5(1):1–5. doi:10.4300/JGME-D-12-00364.1.

Willis GB, Artino AR Jr. What do our respondents think we're asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ. 2013;5(3):353–356. doi:10.4300/JGME-D-13-00154.1.

Sullivan GM, Artino AR Jr. How to create a bad survey instrument. J Grad Med Educ. 2017;9(4):411–415. doi:10.4300/JGME-D-17-00375.1.

Phillips AW, Artino AR Jr. Lies, damned lies, and surveys. J Grad Med Educ. 2017;9(6):677–679. doi:10.4300/JGME-D-17-00698.1.

Reviews

Sullivan GM. Why are medical education literature reviews so hard to do? J Grad Med Educ. 2018;10(5):481–485. doi:10.4300/JGME-D-18-00676.1.

Writing

Sullivan GM. Writing education studies for publication. J Grad Med Educ. 2012;4(2):133–137. doi:10.4300/JGME-D-12-00044.1.

Sullivan GM. So you want to write? Practices that work. J Grad Med Educ. 2013;5(3):357–359. doi:10.4300/JGME-D-13-00204.1.

Sullivan GM. What to do when your paper is rejected. J Grad Med Educ. 2015;7(1):1–3. doi:10.4300/JGME-D-14-00686.1.

Wong BM, Sullivan GM. How to write up your quality improvement initiatives for publication. J Grad Med Educ. 2016;8(2):128–133. doi:10.4300/JGME-D-16-00086.1.

Sullivan GM, Simpson D, Yarris LM, Artino AR Jr. Writing author response letters that get editors to “yes.” J Grad Med Educ. 2019;11(2):119–123. doi:10.4300/JGME-D-19-00161.1.

Collaborate with others and work in a group:

Cross-specialty teamwork has been a hallmark of hospital-based work at this time: anesthesiology attendings working with internal medicine and pediatric trainees, cardiology attendings supervising surgical trainees, and trainees teaming with nurses and respiratory therapists. Thus, consider horizontal collaboration across specialties in your institution or region. Consider vertical collaboration with C-suite colleagues, attendings, residents, and medical students. Currently, medical students are doing more electives, particularly virtual electives, and asking for scholarship opportunities that are outside a lab.

Think outside—way outside—the box:

If ever there were a time for innovation, it is now. With ongoing changes to GME in structure, supervision, clinical experiences, or assessments, consider approaches not previously popular or feasible. Over time curricula get set in stone, as change takes effort and time, which are always in short supply. In 2020 GME has been altered, markedly for some specialties, which may open the door to new rotation configurations, teaching modes, performance assessments, and communications.

Go beyond surveys and consider virtual and other sources of data:

The use of social, visual, and audio-based media by residents, fellows, and faculty are new data sources for analysis. From public diaries, blogs, and Twitter to specialty-related websites, the proliferation of information during the pandemic serves as a source of virtual data to potentially address GME-related questions. Analyses of the content of work rounds, now often virtual or otherwise not at the patient bedside, trainee and faculty journal reflections, and virtual focus groups are additional potential data sources. While virtual learning poses challenges to assessment, it may also present new opportunities: harness virtual learning modes to capture new assessment data to inform competency-based decisions.

Seize opportunities now:

Embrace the warmth and kindness that is being showered on health care workers. This will not last. Use the current enhanced attention of course directors, program directors, administrators, other faculty, patients, and families. Enhanced teamwork is a theme we have seen in many manuscript submissions; this collaboration is both a phenomenon to study and a means to better study and understand phenomena.

Consider every new project a potential research endeavor and seek Institutional Review Board (IRB) approval or exemption:

Many IRBs consider education work exempt from review and others will perform an expedited review. Some initiatives will not come to full fruition or have inadequate numbers or iterations for wider dissemination. Yet at this unstable time, efforts to determine whether an initiative requires some form of external review will be appreciated later, and this work may also serve to crystalize your plans and team.

Conclusions

You probably have surmised our answer to the title of this editorial—full speed ahead for medical education scholarship. This is not only a time for binging on old movies or obsessively checking your news feed for the most recent count of COVID-19 cases. This is a time to gather your team, push forward with an IRB application (if needed), and be creative in identifying and answering questions that relate to this unusual time in medical history. This is the time to work together to determine how we best train the physicians who will care for us now and during the next disaster.