Scholarly Activity in the Next Accreditation System: Moving From Structure and Process to Outcomes

Editor's Note: The ACGME News and Views section of JGME includes data reports, updates, and perspectives from the ACGME and its review committees. The decision to publish the article is made by the ACGME.

In their editorial on scholarship in this issue of Journal of Graduate Medical Education, Simpson et al¹ discuss scholarly activities, drawing on 3 articles on this topic^2–4 and exploring the foundation for scholarship laid by Glassick et al,⁵ as well as the expectations for resident scholarly activity and scholarship, as defined in the Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements. In this companion article, we report on the documentation of scholarly activities for residents and faculty, as the accreditation system is transitioning to the more outcomes-based approach of the next accreditation system (NAS).

Scholarly Activity Reporting in the NAS

In July 2013, 7 specialties transitioned into the NAS, and 1 year later, in July 2014, the remaining 19 specialties and the transitional year will enter NAS. A key attribute of the new system is a more outcomes-based approach, with annual reporting of data, including information on resident and faculty scholarly activities. It is important to note that programs have traditionally reported the scholarly activities of residents, fellows, and faculty in program information forms (PIFs). PIFs include a section on resident scholarship and curricula vitae (CV) documentation of their scholarly activities. Additionally, programs reported faculty participation through detailed CVs that collected data on professional activities; selected bibliographies; selected review articles, chapters, and textbooks; participation in local, regional, and national activities; presentations; abstracts; and grants. Data were collected in a free text format for the interval since the last accreditation site visit.

In the NAS, PIFs are no longer used. Programs provide annual data updates on process and outcomes that permit the Residency Review Committee (RRC) to track the performance of the program and its residents, intervene quickly in programs that demonstrate deficiencies, and facilitate innovation in programs that demonstrate desired outcomes. The new element in the NAS entails a shift from recording and reporting these activities only in preparation for the periodic accreditation site visit to a succinct annual reporting of key dimensions of scholarly activities for faculty and trainees via the new scholarly activity section in the accreditation data system. This is in keeping with annual data collection in the NAS to ensure an ongoing timely review of programs' performance and to allow for timely follow-up for programs exhibiting potential performance problems while allowing a significant lengthening of the interval between accreditation visits for high-performing programs.⁶

The data collection form, which was created by a task force composed of RRC chairs working with ACGME administration, consists of a table with columns describing the different types of scholarly activity expected by the RRCs, as spelled out in the Common Program Requirements (box 1, shown in boldface).

Box 1 ACGME Common Program Requirements Pertaining to Scholarship and Scholarly Activity

II.B. Faculty

II.B.5. The faculty must establish and maintain an environment of inquiry and scholarship with an active research component. (Core)

II.B.5.a) The faculty must regularly participate in organized clinical discussions, rounds, journal clubs, and conferences. (Detail)

II.B.5.b) Some members of the faculty should also demonstrate scholarship by one or more of the following:

II.B.5.b).(1) peer-reviewed funding; (Detail)

II.B.5.b).(2) publication of original research or review articles in peer-reviewed journals, or chapters in textbooks; (Detail)

II.B.5.b).(3) publication or presentation of case reports or clinical series at local, regional, or national professional and scientific society meetings; (Detail)

II.B.5.b).(4) participation in national committees or educational organizations. (Detail)

II.B.5.c. Faculty should encourage and support residents in scholarly activities. (Core)

IV.B. Residents

IV.B.1. The curriculum must advance residents' knowledge of the basic principles of research, including how research is conducted, evaluated, explained to patients, and applied to patient care. (Core)

IV.B.2. Residents should participate in scholarly activity. (Core)

IV.B.3. The sponsoring institution and program should allocate adequate educational resources to facilitate resident involvement in scholarly activities. (Detail)

V.B. Faculty Evaluation

V.B.2. These evaluations should include a review of the faculty's clinical teaching abilities, commitment to the educational program, clinical knowledge, professionalism, and scholarly activities (emphasis added). (Detail)

Shifting from Structure and Process to Outcomes

While the bolded sections of box 1 suggest an orientation toward outcomes, in the form of grants, publications, and presentations, the majority of the standards for scholarly activity in the Common Program Requirements still reflect the focus on structure and process inherent in the prior accreditation system. This includes process requirements such as “Faculty should encourage and support residents in scholarly activities [II.B.5.c], and the requirement for organized clinical discussions, rounds, journal clubs, and conferences [II.B.5.a], to ensure a structure for scholarly activity.” Structure and process standards can create the conditions for scholarly activity but, by themselves, are not measures of the success of these efforts. Some requirements focus on a process but suggest an early focus on outcomes, such as the standards for neurological surgery, which require that “Residents must participate in the development of new knowledge . . . ” (IV.B.2.a).

Prior to entry into the NAS, several specialties, including emergency medicine, orthopedic surgery, diagnostic radiology, and urology as well as several Phase II specialties, had already moved to more outcomes-focused requirements for scholarly activities. All accredited core specialties with outcomes-focused requirements for scholarly activities for the program director, faculty, and/or residents are shown in box 2. In addition, the RRCs for family medicine, internal medicine, and diagnostic radiology have defined expectations for scholarly activities in other RRC documents (box 2).^7–9

Box 2 Outcomes-Focused Specialty-Specific Requirements for Scholarly Activities^a

Program Requirements for Allergy and Immunology

Educational Program/Residents: IV.A.6.a).(2) 25 percent of the program must be devoted to scholarly activities and research. (Detail)

IV.B.2.a) Under faculty member supervision, each resident must design and conduct allergy and/or immunology research that is either laboratory-based, epidemiologic, continuous quality improvement, or clinical investigation-based. (Outcome)

IV.B.2.a).(1) Residents must present their research findings orally and in writing. (Outcome)

Program Requirements for Colon and Rectal Surgery

Faculty: II.B.5.d) At least one faculty member must be actively involved in regional or national specialty societies. (Core)

II.B.5.e) At least one faculty member must be regularly active in scholarly inquiry. Research performed by a resident must not substitute for active faculty involvement. (Core)

Program Requirements for Emergency Medicine (Phase I)

Faulty: II.B.6.d).(1) At minimum, each individual core physician faculty member must produce at least one piece of scholarly activity per year (averaged over the past five years). (Detail)

II.B.6.d).(1).(a) At minimum, this must include one scientific peer-reviewed publication for every five core physician faculty members per year (averaged over the previous five-year period). (Detail)

Program Requirements for Orthopaedic Surgery (Phase I)

Educational Program/Residents: IV.B.2.a) Each resident must demonstrate scholarship through at least one of the following activities:

IV.B.2.a).(1) participation in sponsored research; (Outcome)

IV.B.2.a).(2) preparation of an article for a peer-reviewed publication; (Outcome)

IV.B.2.a).(3) presentation of research at a regional or national meeting; (Outcome)

IV.B.2.a).(4) participation in a structured literature review of an important topic. (Outcome)

Program Requirements for Nuclear Medicine

Faculty: II.B.5.d) When averaged over the preceding five years, each core faculty member must demonstrate participation in at least one scholarly activity annually. (Detail)

Educational Program/Residents: IV.A.6.a).(4).(g).(i) documentation of scholarly activity, such as publications, announcement of presentations; (Detail)

IV.A.6.a).(4).(g).(ii) any additional materials requested by the program director; (Detail)

IV.A.6.a).(4).(g).(iii) submission of a scholarly activity to the program director for evaluation by the completion of the NM3 year. (Detail)

Program Requirements for Obstetrics and Gynecology

Faculty: II.B.5.d) Documentation of scholarly activity on the part of the program and the faculty must be submitted at the time of program review. (Detail)

Program Requirements for Otolaryngology

>Educational Program/Residents: IV.B.2.a).(2) The research experience should result in a completed manuscript suitable for publication in a peer-reviewed journal. (Outcome)

Program Requirements for Diagnostic Radiology (Phase I)

Educational Program/Residents: IV.B.2.b) During their training, all residents must engage in a scholarly project under faculty supervision. (Core)

IV.B.2.b).(1) This may take the form of laboratory research, or clinical research, or the analysis of disease processes, imaging techniques, or practice management issues. (Detail)

IV.B.2.b).(2) The results of such projects must be published or presented at institutional, local, regional, or national meetings, and included in the resident's learning portfolio. (Outcome)

Program Requirements for Radiation Oncology

Faculty: II.B.5.d) The majority of both physician and PhD faculty should demonstrate scholarship as defined above. (Detail) (Note: This refers to the 4 categories of scholarly activity in the Common Program Requirements).

Program Requirements for Surgery

Faculty: II.A.3.e) scholarly activity in at least one of the areas of scholarly activity delineated in Section II.B.5 of this document. (Detail)

II.B.5.e) While not all members of the faculty can be investigators, clinical and/or basic science research must be:

II.B.5.e).(1) ongoing in the residency program; (Detail)

II.B.5.e).(2) based at the institution where residents spend the majority of their clinical time; (Detail)

II.B.5.e).(3) performed by faculty with frequent, direct resident involvement. (Detail)

Program Requirements for Thoracic Surgery

Program Director: II.A.4.p) provide evidence that faculty members are actively engaged in the education and scholarly productivity of residents. (Core)

Program Requirements for the Transitional Year

Educational Program/Residents: IV.B.2.b) Participation should include residents' presentation of a case report or a presentation to colleagues on a subject of interest, and/or development of a research or quality improvement project. (Detail)

Program Requirements for Urology (Phase I)

Program Director: II.A.3.d) documented clinical and scholarly expertise in urology that must be apparent from the program director's curriculum vitae. (Core)

Educational Program/Residents: IV.B.3.a) Residents must demonstrate scholarly activity by manuscript preparation, lectures, teaching activities, abstracts, and the active performance of research or participation in clinical studies and reviews. (Outcome)

^aProgram requirements are categorized into Core, Detail, Resource, and Outcome requirements.

Benefits and Practical Implications

A key benefit of the new approach to reporting scholarly activity is that it standardizes data acquisition and reduces the burden on programs by replacing the labor-intensive updates of faculty CVs with a succinct form for reporting scholarly activities. It also creates clear definitions for each category of scholarly activity for faculty and residents (box 3). The scholarly activity form is designed to simplify data entry by requiring only the PubMed identification for faculty and resident publications in journals listed with the National Library of Medicine/PubMed. For presentations, data entry is reduced to the number of presentations made during the previous year, and for other publications such as book chapters, books, and articles not listed in PubMed,^* only the number of these publications is required. A benefit for the RRCs is that the new format allows timely, quantitative, and objective reviews of scholarly activity. Over time, the new model will also allow for the exploration of viable added dimensions on scholarly activity that fit the various roles that faculty, researchers, and medical educators occupy (box 3).¹⁰

Box 3 ACGME Definitions of Scholarly Activity

Faculty Scholarly Activity Definitions

• a.Publications—Publications recognized by the National Library of Medicine (NLM)—PubMed IDs—up to 4
• b.Abstracts/Presentations/Posters—Number of abstracts, posters, and presentations given at international, national, or regional meetings
• c.Other Presentations—Number of other presentations given (grand rounds, invited professorships), materials developed or other work presented in non–peer review publications—including peer-reviewed, but not recognized by NLM
• d.Chapters—Number of chapters or textbooks published
• e.Grants—Number of grants for which faculty member had leadership role (principal investigator, co-principal investigator, or site director)
• f.Leadership Role—Had an active leadership role (such as serving on committees or governing boards) in national medical organizations or served as reviewer or editorial board member for a peer-reviewed journal
• g.Education Leadership/Materials—Held teaching responsibility for seminar, conference series, or course coordination (such as arrangement of materials, assessment of participants' performance) for any didactic training within the sponsoring institution or program (note, this is not a single presentation or lecture)

Resident/Fellow Scholarly Activity Definitions

• a.Publications—Publications recognized by the NLM—PubMed IDs—up to 3.
• b.Abstracts/Presentations/Posters—Number of abstracts, posters, and presentations given at international, national, or regional meetings
• c.Chapters—Number of chapters or textbooks published
• d.Research Project—Participated in funded or nonfunded basic science or clinical outcome research project
• e.Teaching/Education—Lecture or teaching presentation (such as grand rounds or case presentations) of at least 30 minutes duration at the sponsoring institution or program

In the coming months, RRCs in Phases I and II will gain experience evaluating the scholarly activities that are appropriate and expected for their specialties, the specific requirements, and, over time, the Common Program Requirements likely will reflect an increasing focus on outcomes in the form of scholarly output. Practical implications for programs are 3-fold: the first implication entails having a good understanding of the current requirements for scholarly activity in the specialty and conducting an internal assessment of the program's current performance in this area. The second implication involves complete and accurate reporting of faculty and resident scholarly activities in the ACGME template. The ACGME has developed 2 sets of frequently asked questions (FAQs) pertinent to this topic. One set deals mostly with technical questions about data entry and the accreditation data system¹¹; and a new set of FAQs to be placed on the ACGME website in the near future will offer general guidance about the content of the information entered and also will provide advice and answers for several common situations, such as the time frame for reporting scholarly activities and how to report research published in a journal that is not listed in PubMed. The third practical implication is that program leaders will be able to perform ongoing reviews of the accreditation standards and of RRC communications for the specialty to stay informed about changes, including when RRCs specify new or more defined outcome expectations for resident and faculty scholarly activity.

Conclusions

The ACGME's new approach to reviewing faculty and resident scholarly activities is in keeping with the recommendations of Glassick et al⁵ that “documentation should provide evidence that enables the scholar and his or her colleagues … to apply a set of agreed-upon standards to a body of scholarly work.”5(p39) The aim is to contribute to an environment in which expectations for scholarly work, and for appropriate ways to document this work,¹² are transparent for faculty and residents, where residents learn and can apply the principles of high-quality scholarship and where faculty and residents have opportunities to participate in the creation of new knowledge relevant to clinical care, quality improvement, or education.