Reconceptualizing the emergency medicine resident scholarly requirement: Proposed framework and rubric

Abstract

Background

The completion of a scholarly project is a common program requirement by the Accreditation Council for Graduate Medical Education (ACGME) for all residency training programs. However, the implementation can vary significantly between programs. Lack of generalizable standards for scholarly projects required of all trainees within ACGME‐accredited residencies has led to a large range of quality and effort put forth to complete these projects. Our goal is to introduce a framework and propose a corresponding rubric for application to resident scholarship to quantify and qualify the components of scholarship to better measure resident scholarly output across the graduate medical education (GME) continuum.

Methods

Eight experienced educators and members of the Society for Academic Emergency Medicine Education Committee were selected to explore the current scholarly project guidelines and propose a definition that can be universally applied to diverse training programs. Following a review of the current literature, the authors engaged in iterative, divergent, and convergent discussions via meetings and asynchronous dialogue to develop a framework and associated rubric.

Results

The group proposes that emergency medicine (EM) resident scholarship should (1) involve a structured process, (2) generate outcomes, (3) be disseminated, and (4) be peer reviewed. These components of resident scholarly activity are achieved whether this is a single project encompassing all four domains, or multiple smaller projects that sum to the whole. To assist residency programs in assessing a given individual resident's achievement of the standards set forth, a rubric is proposed.

Conclusion

Based on current literature and consensus, we propose a framework and rubric for tracking of resident scholarly project achievement in an effort to elevate and advance EM scholarship. Future work should explore the optimal application of this framework and define minimal scholarship goals for EM resident scholarship.

INTRODUCTION

Scholarship is a defining component of academic medicine and a core requirement for resident training. ¹ Perhaps the most widely accepted definition of scholarship is Boyer's, which states scholarship “must have clear goals, be adequately prepared, use appropriate methods, achieve outstanding results, communicate effectively, and [scholars must] reflectively critique their work.” ² In addition, scholarship comprises four domains: discovery, integration, application, and teaching. ² The Accreditation Council for Graduate Medical Education (ACGME) lists multiple ways to satisfy the scholarly requirement including peer‐reviewed publications and didactic presentations. ¹ Within the context of scholarship, the final product by a resident to satisfy this requirement can vary widely, creating challenges for programs as they attempt to guide residents. We believe there is a clear need for a reconceptualized framework for scholarly activity guidelines given that (1) scholarly activity is an ACGME requirement and (2) examples of scholarly activity acceptable to ACGME standards vary substantially in effort and quality. Our framework does not seek to redefine scholarship but simply to assist programs in both quantifying and qualifying the components of resident scholarship to ensure all ACGME requirements are met. We propose a new framework and rubric for emergency medicine (EM) residents and residency leadership or faculty to use as a reference when working on scholarly activities with the goal of promoting medicine and academic achievement within our specialty. By utilizing aspects of Boyer's definition, we believe we have produced a conceptual framework that is generalizable and applicable to a wide variety of graduate medical education (GME) training programs.

METHODS

We undertook this paper to explore the current concepts of resident scholarship and describe an approach for advancing this definition and application. Eight members from the Society for Academic Emergency Medicine Education Committee, with both experience in education research and residency program administration, comprised our team. Members represented a variety of EM residency programs, reflecting diversity in program length (3‐ vs. 4‐year), program type (e.g., academic, county based, community affiliated), and geographic location.

In order to understand the current construct of resident scholarship, we began with a structured literature search through PubMed and Google Scholar assessing for relevant articles. We utilized search terms including “scholarship,” “research,” and “residency.” This was supplemented with gray literature searches and review of residency guidelines, including those available on the ACGME website. ³ Resources were identified and discussed as a group with selection for inclusion based upon group consensus. The authors then summarized the key concepts and engaged in a series of meetings to discuss and refine the concept using a series of iterative divergent and convergent approaches to explore the phenomena of interest. ⁴ Discussions continued until the group had reached full consensus. The outlines and frameworks were then converted to text and figures. A draft of the manuscript and rubric was sent to four EM faculty in residency leadership (Program Directors or Assistant/Associate Program Directors), three EM research faculty, one EM general faculty, and two EM residents for comments. These comments were then discussed by the group and incorporated based on consensus. There were nine total meetings lasting approximately 1 h each and supplemented by extensive asynchronous dialogue.

RESULTS

We reviewed approximately 40 articles meeting the search criteria outlined above. After multiple discussions focused on the relevance and impact of each article in regard to EM resident scholarship, 24 articles were included. The key concepts identified were scholarship in higher education, resident scholarship definition and purpose, and satisfying the scholarly requirement.

Scholarship in higher education

The most widely accepted definition of scholarship is Boyer's, which includes “four separate, yet overlapping, functions”: the scholarship of discovery (new knowledge), the scholarship of integration (put knowledge in perspective), the scholarship of application (apply knowledge to address problems), and the scholarship of teaching (transmit, transform, and extend knowledge). ²

In 1997, using information collected by Carnegie scholars, Glassick et al. outlined six standards by which scholarship can be measured: clear goals, adequate preparation, appropriate methods, significant results, effective presentation, and reflective critique. ⁵ , ⁶

In 1999, Hutchings and Shulman delved deeper into expectations associated with the scholarship of teaching concluding that “scholarship of teaching is not synonymous with excellent teaching.” ⁷ Specifically, they outlined three essential and one desired feature of teaching: it must be public, open to critique and evaluation, in a form others can build on, and involve inquiry into the methods and practice of teaching itself as the desired feature. ⁷

Exploring the definition of resident scholarship

The rationale for requiring scholarly work as part of residency training is due to not only the importance of the experience alone, but also to ensure that all residents are capable of critically assessing literature so they may continue to incorporate new practices and stay current on the ever‐changing medical field. ⁸ The ACGME common program requirements regarding research curriculum (training) and scholarship have changed significantly over time. Resident scholarly requirements in early versions of the ACGME program requirements were vague. ⁹ Currently, the only mandate set forth by the ACGME for scholarly activity is that “the curriculum must advance residents' knowledge of the basic principles of research, including how research is conducted, evaluated, explained to patients, and applied to patient care.” ¹ However, the ACGME requirements do not limit scholarly work to research and state only that “residents should participate in scholarly activity,” leaving a clear opportunity for further clarification of what constitutes scholarly activity.

Each program is required to align the scholarly requirement to the program's mission(s) and aims. Types of scholarship can include an array of activities such as quality improvement, population health, educational innovations, and teaching. Currently, the EM‐specific ACGME program requirements for scholarship are almost identical to the ACGME common program ACGME requirements. ¹⁰ Since 2020, the EM specialty‐specific program requirements have defined several possible examples of scholarship acceptable to the ACGME (Table 1).

TABLE 1.

Potential methods to satisfy the resident scholarly activity requirement. ¹⁰

Classic research‐oriented activities

Basic science of clinical outcomes research

Grant leadership

Journal editorial board member or editor

Research project

Serving as a journal reviewer

Publications

Articles or publications

Book chapters

Non‐peer reviewed publications

Non‐peer reviewed resources (print or electronic)

Peer‐reviewed publications

Textbooks

Presentations

Podium presentations

Poster presentations

Presentation of grand rounds

Quality improvement presentations

Webinars

Workshops

Other activities

Quality improvement project

Service on professional committees

We also looked beyond ACGME requirements and conducted a brief review of accreditation requirements from other countries with established systems of EM training and recognition as a specialty. This was limited by a dearth of EM‐specific GME governing bodies internationally, but we reviewed requirements by the ACGME International, ¹¹ as well as the Royal College of Emergency Medicine (United Kingdom), ¹² Royal College of Physicians and Surgeons (Canada), ¹³ and the Australasian College of Emergency Medicine (Australia/New Zealand). ¹⁴ Notably, similar to the ACGME in the United States, these international accrediting bodies are vague on the definitions and requirements related to resident scholarly activity. International requirements range from recognizing the need to train residents in interpretation and integration of research into practice, to the need for residents to participate in research with required time and mentorship without explicitly defining what the participation criteria are.

A 1999 consensus of research directors proposed standards declaring that the “primary role of the scholarly project is to instruct residents in the process of scientific inquiry, to teach problem‐solving skills, and to expose the resident to the mechanics of research” (Table 2). ¹⁵ To meet these goals, the group proposed that scholarly activity should have the general elements of hypothesis generation, data collection, analytical thinking, and interpretation of results. These standards were proposed partially in response to the argument that EM did not have a robust, specialty‐specific body of literature to justify its existence as a distinct specialty. ¹⁵ At that time, 80% of EM programs had a formal research requirement, 35% mandated a research manuscript, and 31% required major involvement in a project. ¹⁶ Of note, although the 1999 consensus recommended that the various elements of scholarly activity be documented in some written form, the overall focus was on “process not product.” ¹⁵

TABLE 2.

1999 Consensus roles of the resident scholarly activity. ¹⁵

1. Instruct residents in the process of scientific inquiry

2. Expose residents to the mechanics of research

3. Teach residents lifelong skills including search strategies and critical appraisal

4. Teach residents how to formulate a question, search for the answer, and evaluate its strength

In 2019, Kane et al. utilized a consensus‐building methodology with a range of participants including academic faculty, department chairs, program directors, and residents, to prioritize the desired goals, definitions, and outcomes of EM resident scholarly activities. ¹⁷ While the 1999 goals of a scholarly activity were no longer prioritized, Kane's work reaffirmed all elements of scholarly activity from the 1999 consensus and developed “best‐practice, measurable outcomes” related to resident scholarly activity. ¹⁷ These include: written documentation of the project, research/quality improvement protocol (developed and implemented), research paper (including hypothesis, data collection/analysis and conclusion), research abstract, or an oral research presentation. ¹⁷ The scope of these outcomes is narrower than that proposed by the ACGME (Table 1) ¹³ and does not include digital scholarship, a platform with significant recent growth into medical academia. In 2020, Husain et al. developed consensus guidelines for digital scholarship, with metrics (impact, role, and quality) to aid in using this medium for academic promotion, which could very easily be applied to resident scholarship. ¹⁸

Refining the resident scholarship expectation

We used the foundation of scholarship for higher education, as applied to EM in the 1999 consensus, and then honed the concept with subsequent literature. ¹⁵ , ¹⁶ , ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ , ²⁴ Using the broader ACGME requirements as guardrails, we developed a more detailed rubric that residency programs could use to define and assess resident scholarly activity.

We began by framing resident scholarly activity using Boyer's four elements of scholarship. ² These four elements, and the associated standards developed by Glassick, ⁶ can be applied to the ACGME examples of scholarly activities referred to above. Boyer concluded that “scholarship is really not a single‐part function but is a four‐part function with all parts inextricably interlocked.” ²⁵ When we consider expectations for scholarship within GME, it is necessary to recognize that an individual may not participate in all four of Boyer's elements but instead may focus on only one component as they are exposed to and learn the scholarly process. As is explicitly stated in the ACGME requirements, training programs and their trainees have a wide range and diversity of interests, ² so it is of the utmost importance that any measure of scholarly activity takes this into account.

In order to reframe and refine the resident scholarly requirement with the above considerations, we propose aligning Boyer's definition of scholarship (with influences from the works outlined above) into the following four scholarship domains such that EM resident scholarship should: (1) involve a structured process, (2) generate outcomes, (3) be disseminated, and (4) be peer reviewed (Table 3). These four domains effectively encompass not only Boyer's definition, but they also build upon the 1999 consensus definition of resident scholarly activity and align with the standards proposed by Glassick and Hutchings and Shulman. It is also important to note that the goals of resident scholarly activity can be achieved whether this is a single project encompassing all four domains or multiple smaller projects that sum to the whole.

TABLE 3.

Correlation of the standards to assess scholarship to EM resident scholarly project domains.

Proposed framework for Scholarship Domains	Glassick (2000)	Hutchings & Shulman (1999)	Summer (1999)	Kane (2019)
Process	Clear goals Adequate preparation Appropriate methods Significant results	Not included	Problem identification Hypothesis formulation Information gathering Data collection & analysis Analytic thinking	Develop and implement protocol Reaffirmed Summer (1999) elements
Outcomes	Significant results	Able to be reproduced and built upon by others	Interpretation of results	Interpretation of results
Dissemination	Effective presentation	Public	Documented in some written form	Documented in some written form Research abstract submission Oral research presentation
Peer‐review & Feedback	Critique	Available for peer review & critique according to the standards of a field	Not included	Innovative metrics including peer‐review and internal evaluation

Exploring the scholarly domains

Process

Scholarship must follow a process that adheres to the scientific method. Process generally involves a combination of clear goals, adequate preparation, appropriate methods, and verifiable outcomes, as well as other administrative components (Table 4). This domain can apply to a variety of scholarly pursuits, which is important for the adaptation of the framework to various EM programs and/or individual resident interests. Based on the type of scholarly project (e.g., quality improvement, educational innovation), the resident should be expected to adhere to best practices related to process. A full discussion of best practices for each type of academic project is beyond the scope of this paper but can be considered and individualized at the program level. In order to define work as scholarship, a resident should define their process formally, and we propose this framework as a guide.

TABLE 4.

Exploring the scholarly domains.

Process components for resident scholarship Identify scholarly question/need with clear goals Perform adequate literature search Identify appropriate approach (e.g., methods and analysis, teaching design) Submit IRB or equivalent (if applicable) Acquire data (e.g., conduct research study, needs assessment) Interpret results/data Draft abstract/manuscript/presentation/curriculum 2 Resident scholarly activity outcomes classified by types of scholarship Scholarship of Discovery (advancing knowledge) Case report Quality assurance/improvement project Systematic review/Scoping review Original article–randomized‐controlled trial (RCT) Original article–non‐RCT Abstract related to above examples Scholarship of Integration (synthesizing knowledge) Narrative review, Conceptual review Systematic review/Scoping review Textbook chapters Evidence‐based medicine guidelines Scholarship of Application (applying knowledge) Scientific committees, guideline panels (participation & leadership) Departmental Institutional Local/Regional National Teaching (disseminating knowledge) Didactic, Grand Rounds, Group teaching (small/large), Workshops Departmental Institutional Local/Regional National 3 Levels of dissemination Department only Institution Regional National International Online* (Assessment to vary based on reach/platform) 4 Peer review timeline and expertise Pre‐submission review Local committee (e.g., program leadership, faculty with expertise in the area of the scholarship) Mentor review Submission review Journal peer review Web based/digital Scientific conference Presentation feedback/evaluation Post submission review Peer network rating/reviews Social network commentary Review of evaluation data Letter to the editor

Outcomes

Scholarship must produce verifiable results and create tangible outcomes, which may take a variety of formats depending on the type of scholarship and the process used, and can reinforce or add to existing scientific knowledge. This requirement for scholarship fulfills Hutchings and Shulman's definition that the work must be made public, available for critique, and able to be reproduced and built on by others. ⁷ The impact of the work may vary according to the type of outcome and can be variously measured such as by incorporation into guidelines, citation in a manuscript, or number of downloads or uses of instructional materials. Specifically, we build on the framework developed by Grady et al. and teleologically use the outcomes of potential resident scholarly activities to guide the form of scholarship being assessed by our framework (Table 4). ²⁶

Dissemination

Scholarship must be shared in an open forum. Hutchings and Shulman stated that to be considered scholarship, the work must be made public as one of their three criteria for scholarship. ⁷ Whether that forum is a prestigious journal or a departmental poster presentation, sharing the work product allows it to become a building block for other scholars, subjects it to scientific scrutiny, and assures the work's validity and safety for application.

Social media and digital scholarship have become a critical space for the dissemination of knowledge, outreach to community and policy makers, and creation of communities of practice. ²⁷ , ²⁸ , ²⁹ , ³⁰ This new realm brings multiple novel challenges such as appraising the quality and appropriateness of the content, evaluating the impact on the academic and general populations, and creating a system to reward scholars who use these new media. ³¹ Digital scholarship experts have begun publishing consensus guidelines for digital scholarship in academic promotion. ¹⁸ Many of these guidelines are applicable to resident scholarship and may help programs integrate this platform into more classic levels of dissemination (Table 4). A full discussion of best practices for international dissemination is beyond the scope of this paper but can be individualized by program leadership.

Peer review

Peer review has been defined as “the assessment by experts of material submitted for publication.” ³² Peer review is an essential component of scholarship and separates true scholarly work from expert opinion and self‐published material. The definition of “experts” may draw scrutiny; however, it need not include only the most widely recognized or previously published scholars in a given field. Ideally, it should include those with advanced knowledge of the content and relevant processes in the selected area in addition to having independence from the project. Whether or not resident scholarly work eventually undergoes the formal process of peer review during journal submission, it should at least be subjected to independent review by local experts (e.g., program faculty) to ensure the fidelity of both content and process. Therefore, peer review should include review of scholarly work prior to dissemination or implementation. Table 4 summarizes the submission process for peer review as well as various levels of peer review expertise. It is important to consider peer review as a critical feedback component that also involves self‐critique and reflection in the overall scholarly activity process.

Operationalizing the resident scholarly activity rubric

In reframing and refining resident scholarly activity in this paper, our goal is to provide a framework and a rubric for individual residency programs to quantify and qualify the scholarly activity engagement demonstrated by each resident. Herein, we provide a rubric to operationalize the process with the intended use for residency programs at the individual resident level (Table 5). The rubric can be used qualitatively (e.g., meets/does not meet) or, should programs choose, quantitatively by assigning points based on specific activities that can be weighted based on program aims and preference.

TABLE 5.

Blank rubric template.

		Types of scholarship (Boyer)
Domains	Standards (Glassick)	Discovery	Integration	Application	Teaching
Process	Clear goals
	Adequate preparation
	Appropriate methods
Outcomes	Verifiable Results
Dissemination	Effective presentation
Peer review & feedback	Reflective critique

We recognize that the goals and capabilities of individual training programs vary substantially. Therefore, we afford individual programs the flexibility to determine the level of assessment (including whether it will be measured qualitatively or quantitatively based on a point system), specific point assignments (if applicable) with adjustments based on percent effort, as well as the minimum requirements based on the circumstances unique to their context, program aims, and resources.

We considered feedback from external reviewers with expertise in education and research that the rubric should contain uniform “cut‐offs”; in other words, defined minimum standards that residents must meet. However, we were compelled to recognize the substantial diversity of aims and resources across programs. Therefore, it is our opinion that it would be unwise for us to include predetermined “cut‐offs” in this first iteration. Future iterations of the rubric could include minimum criteria determined either by the ACGME Review Committee for EM (EM‐RC) or via a consensus methodology.

As programs consider implementing this rubric, we recommend the following practices:

1. Program and departmental research leadership should first determine whether they prefer to use a qualitative system or a quantitative weighted point system.

   1. Each group will need to determine how and to what extent a resident's work meets requirements for each domain. This must be done in a standardized and predetermined manner, preferably by a group rather than an individual to ensure transparency and fairness.
   2. If selecting a weighted point system, the group should determine weighting in relation to the context, mission, and aims unique to that program.

2. Residents should complete activities across all domains to satisfy the scholarly activity requirement.

3. Residents may complete the scholarly activity requirement with a single activity within a single domain or several activities across various domains and/or types of scholarship.

4. Residents are encouraged to explore both traditional and nontraditional scholarly activities.

Overall, this rubric provides trainees with an objective and transparent mechanism to understand program expectations and to guide their scholarly endeavors. It may provide an opportunity for program, institutional, and even ACGME leadership to evaluate resident scholarly activity within programs more objectively. Most of all, the rubric supports the EM‐RC's objectives and flexibility for training programs related to the scholarly activity requirement. ¹ Two example scenarios of resident scholarship are included in the supplemental material to guide programs in implementing the rubric.

Next steps

We have created a conceptual framework that reframes and refines the resident scholarly activity requirement. We then used our framework to develop a rubric for the transparent quantification of scholarly work. We hope that this framework and rubric can be applied to all GME programs to: (1) advance and expand the allowable types of scholarship; (2) establish program‐level expectations for resident scholarship; (3) objectively and transparently compare the quality of resident scholarship; and (4) potentially serve as a building block for the development of similar framework applicable to faculty scholarship in academic medicine.

Several areas generated significant discussion during this process, and these seem likely to be areas of further debate and work. Beyond the question of the “minimum hard cut‐off” discussed above, uncertainty also remains about any possible hierarchy in scholarly products. For example, is a randomized‐controlled trial more “scholarly” than a narrative review of a clinical topic presented in a departmental conference? Further complicating this question is the value proposition for residents involved in these scholarly endeavors. What does the resident learn about the process of scientific discovery by participating in a grant‐funded clinical research study as opposed to generating a quality improvement project and following it through to its completion? Should programs that are hosted at an academic medical center have different standards for scholarly projects as opposed to programs in community‐based hospitals who presumably have less access to research mentorship? We hope that the development and refinement of our proposed domains and rubric will provide necessary structure to programs who seek to answer these questions, at least in regard to EM residency requirements.

The next steps in this process will be to implement and refine the rubric in “real use” cases within EM programs. We anticipate that this will best be achieved with a dedicated residency group to lead and apply the rubric in a standardized fashion. However, more experience and research on implementation will be needed to define best practices.

CONCLUSIONS

We propose a framework and rubric that meet current and anticipated future minimum ACGME requirements for resident scholarly activity. These tools add necessary specificity to the existing definition of scholarship by creating a more robust scholarly product while remaining adaptable to any program's unique mission and aims. Our overall goal is to elevate and advance EM scholarship, and our hope is that this adaptable rubric can serve as a starting point to meet the individual needs of each distinctive residency. EM residency programs that choose to adopt our rubric to determine whether a resident has met their scholarly activity requirement can be assured that residents who comply with the standards of the rubric have not just met but exceeded the ACGME's scholarly activity requirement.

AUTHOR CONTRIBUTIONS

All authors contributed to all aspects of the manuscript. Study concept & design: M. Tyson Pillow, Michael Gottlieb, Anne Messman, Nicholas D. Hartman, Alina Tsyrulnik, David Barnes, Michelle D. Lall, Harsh Sule. Analysis of previously published work: M. Tyson Pillow, Michael Gottlieb, Anne Messman, Nicholas D. Hartman, Alina Tsyrulnik, David Barnes, Michelle D. Lall, Harsh Sule. Development of framework and rubric: M. Tyson Pillow, Michael Gottlieb, Anne Messman, Nicholas D. Hartman, Alina Tsyrulnik, David Barnes, Michelle D. Lall, Harsh Sule. Drafting of manuscript, editing and revisions: M. Tyson Pillow, Michael Gottlieb, Anne Messman, Nicholas D. Hartman, Alina Tsyrulnik, David Barnes, Michelle D. Lall, Harsh Sule.

CONFLICT OF INTEREST STATEMENT

The authors have no conflict of interests to report. MTP: no conflict of interests to report. MG: no conflict of interests to report. AM: no conflict of interests to report. NDH: no conflict of interests to report. AT: no conflict of interests to report. DB: no conflict of interests to report. MDL: no conflict of interests to report. HS: no conflict of interests to report.

Pillow MT, Gottlieb M, Messman A, et al. Reconceptualizing the emergency medicine resident scholarly requirement: Proposed framework and rubric. AEM Educ Train. 2023;7(Suppl. 1):S33–S40. doi: 10.1002/aet2.10878