A dedicated one-month Anesthesiology Professional Practice rotation for first-year residents is associated with an increase in scholarly activity during residency

Abstract

Study Objective:

We designed and implemented an academic non-clinical rotation and its association with subsequent resident scholarly productivity was determined. We hypothesized that participation in the rotation would be associated with increased scholarly activity during residency.

Design:

Retrospective educational comparative study, of two cohorts of anesthesiology residents in the graduating classes of 2015–2020.

Setting:

Large anesthesiology residency program at a U.S. academic medical center

Intervention:

A one-month academic rotation for first-year anesthesiology residents has been implemented since 2014. The rotation curriculum broadly covers important topics for scholarly projects and provides introductions to academic faculty and institutional resources.

Measurements:

The scholarly products (abstracts, publications, book chapters, research protocols, and grant applications) were quantified using Scholarly Activity Points, a previously described metric that accounts for significance and the resident’s contribution. Total Scholarly Activity Points for each resident and number of publications prior to residency were determined for both cohorts. Segmented regression was employed with Scholarly Activity Points as the outcome; participation in APP, and prior publications were used as input variables.

Main Results:

Resident participation in the novel rotation was significantly associated with higher Scholarly Activity Points. The confounding variable of pre-residency publication count was not significantly correlated to this increase.

Conclusions:

Immersion in a one-month academic program during the anesthesiology internship may contribute to increased scholarly productivity during residency.

1.0. Introduction:

Anesthesiology has previously lagged behind other disciplines in some metrics of academic productivity [1, 2], but considerable strides have been made in the past decade [3]. Important mechanisms exist to foster scholarly productivity among medical students interested in anesthesiology [4] and to provide post-graduate dedicated research opportunities via extramural funding [5]. Another opportunity to promote anesthesiology scholarship is to integrate research education as part of residency [6].

Since 2014, the anesthesiology residency program at the University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania, U.S.A., has implemented a dedicated one-month, non-clinical didactic program for all postgraduate year 1 (PGY1) residents. We have branded this unique rotation the Anesthesiology Professional Practice (APP) rotation. The APP rotation’s overall goal is to comprehensively educate residents on the non-clinical subjects pertinent to professional practice as consultant anesthesiologists, including key topics that have been implemented in shorter-duration dedicated curricula focused on business, leadership, and professional development [7, 8]. Our APP program was also thoughtfully designed to provide enhanced education on academic scholarship in residency, filling a long-known need in anesthesiology graduate medical education [9]. During this time, our residency program did not change its expectations or requirements regarding resident research. However, we expected the APP rotation would help residents develop interest and expertise in scholarly products early in residency. A marker of success would be increased resident academic productivity following the implementation of the APP curriculum.

A core goal of the APP rotation is to provide an early opportunity for PGY1s to pursue scholarly projects. In a preliminary analysis of the first class to participate in the APP rotation, we found a high proportion of PGY1 residents were already engaged in scholarly activity just two months after completing the rotation [10]. To better characterize this effect, we retrospectively compared the scholarly products of two cohorts of residents from a 6-year period: those PGY1s that participated in the APP rotation and residents that did not participate because they entered the program either prior to implementation of the rotation or as advanced PGY2 residents. Using a previously-developed summary score, called Scholarly Activity Points (SAP) [11], we were able to quantify the number and impact of scholarly products during residency for each resident in both groups. Our primary hypothesis was that participation in the APP rotation would be associated with increased SAP scores. We accounted for time and publications prior to residency as potential confounders.

2.0. Materials and Methods:

This retrospective educational research project was deemed exempt by the local institutional review board (University of Pittsburgh). We adhered to all relevant standards for the responsible conduct of research in the performance of this study.

2.1. Rotation Curriculum Description

To summarize the APP curriculum for this publication, we tabulated the scheduled lectures and online training materials for the years under study. We also totaled the hours that residents spent on each learning activity by year, and each activity was classified according to the professional practice domain in which it best fit. The proportion of the APP curriculum devoted to each domain was then calculated as the average hours in the domain divided by the total didactic time for the APP residents.

A broad range of content is covered in the APP rotation, drawing from the expertise available from many faculty within our department, as well as involving high-quality presenters from the health system leadership, faculty from other departments in the University of Pittsburgh, and some members of the wider local professional community. The curriculum covers 12 key areas: research, professional development, leadership, wellness, patient safety, education, quality improvement, malpractice, clinical operations, business, global health, and innovation. Additionally, we schedule the APP rotation each year to overlap the Anesthesiology® annual meeting (sponsored by the American Society of Anesthesiologists) in October and provide support for all PGY1 residents to attend. Their guided experience in attending this premier anesthesiology conference is an integral part of the APP rotation that provides additional insight from renowned experts on topics in these same professional practice learning domains as the APP curriculum.

2.2. Study Design

The first three classes of categorical (entering PGY1) residents who participated in the APP rotation (graduating classes of 2018–2020) were analyzed as the APP Group. Though scholarly topics were covered in the rotation curriculum, residents in the APP group were not informed that their scholarly activity would be tracked and analyzed for this study. There were also no changes in expectations for participation in scholarly projects as part of the residency program requirements.

Residents that entered our advanced (non-categorical) program into a PGY2 spot from the graduating classes of 2018–2020 were included in the unexposed, No APP group for analysis. Though PGY1 year program obviously differs, clinical anesthesiology training in PGY2–4 years was otherwise the same between categorical and advanced residents in both cohorts. Additionally, all residents from the three classes who matriculated prior to the start of the APP rotation (graduating classes of 2015–2017) were included in the No APP group.

2.3. Data Collection

To explore whether any unknown factors affecting resident scholarly productivity over time may have existed, we queried PubMed for each resident and identified publications on which they were an author prior to residency. Any ambiguity in identifying authorship based on name was clarified by contacting the former resident by email. There were three such cases, and none of these failed to resolve the question.

Other data was obtained from residency office records. All scholarly products (abstract presentations, publications, book chapters, Institutional Review Board protocols, and grant applications) created during residency training are collected from each resident. The completeness of this list is verified with each resident during an in-person exit interview with the program director.

2.4. SAP Calculation

Individual resident scholarly productivity during residency was quantified using SAP, according to the previously described calculation [11]. The SAP calculation takes into account the scope and impact of each scholarly product. The total SAP score for a scholarly product was calculated using a relative weight value, determined by the type of product. Several modifiers are then applied, to reflect the degree of the resident’s effort and the academic rigor of the product. Five scholarly product types are included, and a standard base weight value was assigned to each, based on the typical relative complexity of the product: abstract presentations are 50 points; journal publications are 150 points; book chapters are 50 points; submitted research regulatory board protocols are 75 points; grant application submissions are 100 points.

The preliminary SAP score for each scholarly product was then multiplied by a set of modifiers relevant to the work, with higher factors indicating greater contributory effort or academic value. Abstract presentations were scored according to authorship, meeting venue, presentation repetition, and original research. The authorship modifier determines the individual resident’s contribution to the product (first authorship, x 1.0; other authorship, x 0.5). The meeting venue modifier values the product according to the meeting setting (international or national, x 1.0; regional or local, x 0.5). Repeated presentations (such as the same abstract at multiple meetings) and presentations of case reports are both valued x 0.5. As an example, a resident as the second author at an international meeting for the first presentation of a case report would be scored as 50 (standard weight value for an abstract) x 0.5 (second author modifier) x 1.0 (international meeting modifier) x 1.0 (first presentation modifier) x 0.5 (case report modifier), for a total SAP of 12.5. Journal publication modifiers include authorship, impact factor, and manuscript category. The publishing journal’s impact factor for the year preceding the article’s publication was used as the modifying multiplication factor. Impact factors were obtained from the InCites Journal Citation Report (Clarivate Analytics) [12]. Possible manuscript categories include original research (x 1.0), review articles (x 0.75) and case reports (x 0.5.) Research regulatory board protocols are modified by authorship and the reviewing agency (full board, x 1.0; expedited review, x 0.5; and exempt, x 0.25). Book chapters are modified only by authorship. Grant applications are factored by authorship, award, and the submission agency (federal, x 2.0; foundation, x 1.5; department seed grants, x 1.0). Grant applications resulting in awards are modified by x 1.5. Other scholarly product types, including a book review or Letter to the Editor, are given a base SAP weight value of 25 points, modified by authorship and peer-reviewed publication (yes, x 1.0; no, x 0.5).

The total SAP score for each resident was calculated by summing the final SAP score for all their scholarly products during residency. SAP data was grouped by each resident’s class year and their group membership (APP versus No APP).

2.5. Statistical Analysis

The distribution of SAP scores within the groups did not display normality, so medians and quartile ranges were calculated to describe data within each cohort. All SAP data was plotted by APP exposure group, and a best-fit linear trendline added to the scatter plot. We analyzed our data using segmented regression, to account for the interrupted time series design [13]. The SAP data was transformed by the natural logarithm (ln) for statistical analysis, with per-resident ln(SAP total) as the dependent variable. Exposure to the APP curriculum was the key input variable, and we additionally included the number of peer-reviewed indexed publications authored or co-authored prior to residency. A threshold of p < 0.05 was considered statistically significant, and analysis was performed using SAS 9.4 (Carey, NC, USA).

3.0. Results:

3.1. APP Curriculum

Figure 1 quantitatively displays the proportion of the APP didactic time devoted to the 12 main professional practice topic areas covered. Research education comprised the largest domain in the curriculum, and a complete categorized list of all APP curriculum elements is included in a Supplemental Table (Supplementary Material 1). On average, residents completed approximately 90 total hours of didactic material in the APP rotation each year.

Fig. 1.

The pie chart shows the average percentage of didactic lecture time spent in 12 key areas (excluding Introduction) during the early immersion rotation. The partial total hours spent for each area can be calculated using 90 h as the total lecture duration (ex., Research 30%: 27 h).

3.2. Residency Class and Group Characteristics

The numbers of residents comprising each graduating class are listed in Table 1, delineating the total number of residents that entered as categorical versus advanced residents, and the numbers that participated in the APP rotation (available only to categorical PGY1 residents). In total, 71 residents were included in the No APP group and 37 in the APP group. Table 1 also lists, by graduating class year, the numbers of indexed peer-reviewed publications and number of PhD holders within the APP and No APP groups. With small and equal number (2) of PhD degrees in both cohorts, this variable was not further considered in the analysis. One or more months of dedicated research rotation was elected by 46% of residents in the No APP Group and 49% of residents in the APP Group. Full utilization numbers are in a Table in Supplementary Material 2, showing that average research months taken per resident were 0.97 in the No APP Group and 0.89 in the APP Group.

Table 1.

Resident data, by graduating class year, for APP and No APP cohorts.

	Number of Residents				Prior Publications				PhD Degrees
Class Year	Program Track		No APP Group	APP Group	No APP Group		APP Group		No APP Group	APP Group
	Cat.	Adv.			Total	Avg.	Total	Avg.
2015	10	9	19	-	7	0.4	-	-	1	-
2016	11	9	20	-	7	0.4	-	-	0	-
2017	12	8	20	-	13	0.7	-	-	1	-
2018	11	6	6	11	4	0.7	15	1.4	0	1
2019	10	5	5	10	1	0.2	20	2.0	0	0
2020	16	1	1	16	0	0.0	11	0.7	0	1

Legend: Cat.= categorical; Adv.= advanced; APP= Anesthesiology Professional Practice (novel rotation); Avg.= average (arithmetic mean within group). ‘-’ indicates no data, as the APP rotation had not yet been implemented.

3.3. SAP Analysis

Figure 2 shows the SAP distributions by residency class, showing individual data points and the median and quartile ranges for data within each cohort (class by APP group). For precision in visualizing important values, the Y-axis range has been truncated resulting in two outliers excluded from the top of the chart: one No APP group resident with 2749 SAPs from the class of 2016, and an APP Group resident with 3141 SAPs from the class of 2018. All other resident SAP totals were less than 1600. For full reporting, all data points are shown in a complementary Figure, Supplementary Material 3. All data points were included in the segmented regression analysis.

Fig. 2.

Box and whisker plots of SAP data organized by residency class year and APP exposure cohort. The No APP group is shown in blue and the APP group is shown in red. Note that two outliers with SAP totals above 1600 are not shown, for better visualization of the other data (see manuscript text and Figure, Supplementary Digital Content 2 for full data display). Means are shown by ‘x’; medians are shown by the straight line; box edges represent the 25th and 75th quartiles; whiskers are shown at 1.5 times the interquartile range. (Note: APP stands for “Anesthesia Professional Practice,” which is the local term for the early immersion rotation.) (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Figure 3 is a scatterplot showing all SAP data by graduating class year. The figure is divided into two panels, with the No APP Group in blue on the left and the APP Group in Red on the right. Best-fit linear trendlines are overlaid on the data, and the similar-colored curved lines represent 95% confidence bounds (calculated from the data).

Fig. 3.

A scatterplot (with the best-fit linear trendlines and 95% confidence bounds) of the scholarly activity points generated by the residents who did not undergo the early immersion rotation during their PGY-1 internship (left) and those who had the opportunity (right). (Note: APP stands for “Anesthesia Professional Practice,” which is the local term for the early immersion rotation.)

Results of the segmented regression analysis are shown in Table 2. For clarity, variables have been divided into Intercept and Slope terms before (Pre-APP) and after (Post-APP) implementation of the APP rotation. Note that data in Table 2 are SAP totals transformed by the natural logarithm. Prior to the APP implementation, SAP totals were significantly greater than zero (Intercept Pre-APP), and there was no significant trend towards increasing or decreasing over time (Slope Pre-APP). After the implementation, SAP significantly increased (Intercept Post-APP), and there was no significant change in the yearly trend (Slope Post-APP). The potential confounding variable of pre-residency publication count (PrePubs, in Table 2) was not significantly associated with increasing SAP totals. Thus, consistent with our primary hypothesis, participation in the APP rotation was significantly associated with increasing SAP totals.

Table 2.

Coefficients and statistics for parameters in segmented regression model.

	Parameter Estimate	Standard Error	t Value	P	95% Confidence Limits
Model Variables
					Lower	Upper
Intercept Pre-APP	4.40	0.54	8.11	<0.0001	3.35	5.46
Slope Pre-APP	−0.18	0.26	−0.71	0.4787	−0.69	0.33
Intercept Post-APP	2.05	0.76	2.71	0.009	0.56	3.54
Slope Post-APP	−0.35	0.35	−1.01	0.3178	−1.04	0.34
PrePubs	0.16	0.08	1.95	0.0554	0.00	0.24

Legend: Values listed are the natural logarithm of Scholarly Activity Point totals. Variables are divided into pre- & post- APP implementation. Intercept & Slope reflect, respectively, the value and temporal trend, in the time periods before and after implementation of the APP rotation. PrePubs is based on the number of publications prior to residency.

4.0. Discussion:

One possible benefit of the APP rotation would be an increase in resident scholarly activity, including initiating research projects and disseminating scholarly work through presentations and publications. Notably, the rotation curriculum is diverse, with research-focused lectures comprising only 30% (27/90 hours) of the curriculum. However, our department commonly produces scholarly projects across non-traditional domains such as wellness [14], clinical operations [15], and education [16], such that all APP curriculum domains may contribute to increased scholarly products.

Several features of the rotation could help facilitate increased scholarly activity in the APP Group. The rotation schedule, which includes no evening or weekend clinical duty, allows ample time for PGY1 residents to explore academic and research interests. Further, the rotation provides an early introduction of PGY1 residents to academic faculty, many of whom participate as lecturers, panelists, or facilitators in APP sessions. This was expected to facilitate pairing resident interests to potential project mentors, and feedback from APP alumni has repeatedly validated that this matching has occurred during the rotation. Finally, the time spent on research education during the APP rotation likely facilitates residents’ scholarly interest and research preparedness. Several sessions focus on best practices for an academic project’s progressive phases, from design to dissemination. Additionally, the financial support for and specific guidance on attending the Anesthesiology® annual meeting is an integral part of the APP rotation. Attending this conference and observing scholarly presentations may serve as an additional motivator for the future generation of scholarly products.

Like approximately one-third of residency programs [17], we have had a structured resident research education program that predated the APP curriculum innovation. This includes the requirement of developing a systems-based practice improvement project and presenting or publishing at least one scholarly work outside our institution. Both groups under comparison also had the option of doing up to six months of closely mentored resident research rotations. Though our previous work has suggested that this research curriculum alone may increase scholarly products [6], these educational offerings were available to both resident cohorts in this study. Further, similar utilization of our elective research rotation between cohorts suggest that this did not contribute to the differences in SAPs. This draws an important distinction between the APP rotation, which is intended as a broad overview of many professional practice topics for all residents, compared to more immersive research track experiences, which may contribute to increased scholarly output for a subset of residents [18].

We have modelled available data, with the aim of determining if there was a change in scholarly activity, measured by SAP totals during residency, following introduction of our novel rotation during the PGY1 year of our graduating class of 2018. Our results suggest that SAP totals were higher among those residents that participated in the APP rotation. Further, scholarly productivity prior to residency, as measured by the number of peer-reviewed publications indexed in PubMed was not a significant confounder. With only three years of post-APP data included in this analysis, long-term trends in scholarly activity are difficult to discern, and a regression to the mean is certainly possible. Though a future study of additional classes could be undertaken to increase the sample size and provide more longitudinal data, such a study design would require significant additional time.

We had previously demonstrated that a non-clinical research rotation represented a significant institutional investment to resident education [19]. Similarly, the cost for all PGY1 residents to attend APP sessions, with coverage by nurse practitioners, can be estimated at $450 per day [20] for a total rotation cost of $135,000 annually. Coverage by nurse anesthetists providing hands-on anesthesia care would be considerably higher [19], highlighting another advantage of placing this curriculum in the PGY1 year. Faculty participation in APP lectures was not a part of the faculty-lecturer financial incentive system previously described [16], and their non-clinical time additionally represents an annual indirect expense exceeding $40,000. These costs underscore the importance of demonstrating a quantifiable return on investment, particularly as a program directory survey identified the lack of resident non-clinical time and leadership support as significant barriers to resident research productivity [17].

4.1. Limitations

Additional limitations of this study are important to discuss. This is a non-randomized retrospective study of a curriculum intervention that occurred at a fixed time point. Though we have taken a thoughtful approach to identify confounders (most notably publications prior to residency), all factors that might influence an individual resident’s propensity for scholarly activity cannot be fully modelled in our analysis. This is exemplified by the presence of one high-performance outlier in both groups, which illustrates the potential for a small number of residents to potentially influence cohorts of this size. An important weakness is the 3-year time frame post-intervention that is captured, which limits the ability to characterize future trends in scholarly activity for residents that do (or do not) participate in the APP rotation. Finally, guided attendance at the Anesthesiology® annual meeting is an integral feature of the rotation and the impact of that experience cannot be assessed separately from the other rotation curriculum.

4.2. Conclusions

In 2014, we implemented a dedicated one-month non-clinical rotation, called Anesthesiology Professional Practice, for our PGY1 anesthesiology residents. We have since had the first three resident classes who participated in that rotation complete residency. Compared to a larger group that was not exposed to the APP rotation, participation in the APP rotation was correlated with greater scholarly productivity during residency, as measured by the Scholarly Activity Point system. Further, this relationship was not affected by the number of publications prior to residency. This association suggests a possible tangible benefit of an early immersion program for scholarly and academic education.