The Effectiveness of Direct Supervision by an Attending Compared To a Senior Resident on Quality of Supervision of Novice Anesthesiology Residents: A Randomized Study

Abstract

Background:

New trainees are directly supervised by either an attending physician or a senior resident under indirect supervision from an attending physician. The main objective was to evaluate which type of direct supervision (attending vs. senior resident) would result in better quality of supervision to novice residents during their first month of training.

Methods:

Novice anesthesiology residents were randomized to receive direct supervision by an attending anesthesiologist or a senior resident during their introduction month of intraoperative anesthesia. The primary outcome was a validated instrument to evaluate supervision performance of the instructor. The secondary outcome was a validated anxiety scale.

Results:

The overall mean supervision score across the study days was greater in the residents who were directly supervised by attendings, mean (standard error [SE]) of 3.88 ± 0.03 compared with direct supervision by a senior resident, mean (SE) of 3.77 ± 0.03 a mean difference of 0.11 (95% confidence interval [CI], 0.05-0.16), P = .0012. Five of 9 individual items on the supervision survey were significantly greater in the group directly supervised by attendings compared with residents. There was no difference between groups regarding anxiety scores. In contrast, there was a mild association between supervision scores and Spielberger State-Trait Anxiety Inventory-6 anxiety scores, correlation coefficient = 0.23 (95% CI, 0.08-0.39), P < .0035.

Conclusions:

We detected better supervision scores when novice anesthesiology residents were directly supervised by attendings when compared with senior residents. Nevertheless, direct supervision by senior residents still provided supervision scores consistent with a safe supervision practice.

Introduction

Appropriate supervision of trainees is a core program requirement of the American Council of Graduate Medical Education (ACGME) accreditation.¹ Unlike its use for billing purposes, the term “supervision” is used here to describe the oversight provided to ensure the quality of patient care and the progression of trainee clinical skill development.¹ Poor supervision of anesthesiology residents can not only decrease the educational experience of trainees, but it is associated with medical errors and potential patient harm.²

Incoming anesthesiology residents begin their first month of intraoperative training with an initial period that involves direct, constant supervision that requires either an attending overseeing 2 residents, or an attending working independently. To achieve this level of supervision, new trainees are either paired with an attending physician or a senior resident. The paired group design and the duration of the initial weeks of clinical anesthesiology may vary across residency programs. Nonetheless, it is currently unknown what type of direct supervision (eg, attending vs senior resident) is more beneficial to novice residents during their first month of training. This knowledge would help academic anesthesiology departments select the most appropriate supervision model for novice anesthesiology residents.

The main objective of the current investigation was to compare 2 models of direct supervision of anesthesiology residents during their introduction to intraoperative anesthesia. We hypothesized that novice residents paired with a senior resident as a direct supervisor would have a better quality of supervision experience than residents paired with an attending anesthesiologist as a direct supervisor. In addition, we sought to evaluate if anxiety levels of novice trainees would be different between the study groups.

Materials and Methods

This prospective, randomized educational study was conducted at the Department of Anesthesiology at Brown University and approved by the Lifespan Institutional Review Board, Providence, Rhode Island (#1734757). Written informed consent was obtained from all residents and the trial period extended from July 8, 2022, to July 29, 2022. The study is reported following the CONSORT guidelines for reporting randomized studies.³

The incoming class of anesthesiology residents was introduced to the study during orientation by the lead author. Inclusion criteria consisted of first-year clinical anesthesia residents (CA-1) (postgraduate year [PGY]-2) assigned to American Society of Anesthesiologists physical status (ASA PS) 1 or 2 patients. Exclusion criteria included residents who were assigned to ASA PS 3 patients, assigned to an operating room with other anesthesia providers (certified registered nurse anesthetists), prior formal training in anesthesia (accredited or nonaccredited residency training in all American or international programs), or those residents who declined to participate.

Enrolled residents were randomly assigned daily into 2 groups (in a 1:1 ratio) using a computer-generated randomization sequence with a block size of 4. Nonidentifying codes were used to collect the anonymous responses given by the participants. Each day the incoming residents were assigned a unique 3-digit code that only the statistician would know to facilitate matching the resident's data across multiple days. Residents randomized to group A were assigned an attending physician for constant and direct 1:1 supervision in the operating room and group B were assigned to a senior resident for direct 1:1 supervision with an attending anesthesiologist providing indirect supervision. Senior residents were defined as current residents who are in their postgraduate clinical anesthesia year 2 or 3 (PGY-3 or -4). The direct supervisor in either group, the attending or senior resident, supervised only 1 room at a time and did not leave the room after induction of anesthesia.

The residents were asked to complete a survey at the end of the day during didactics session before departing the hospital. The surveys were deposited into a locked box located in the conference room. Surveys were collected by an independent research associate who was not directly involved in the study.

The survey included a validated instrument developed by de Oliveira Filho et al.⁴ to assess the evaluation of the individual supervision performance. The instrument consists of 9 questions evaluating the following domains: (1) planning perianesthesia care, (2) providing feedback, (3) being available, (4) giving opportunities/fostering resident autonomy, (5) stimulating patient-based learning, (6) demonstrating professionalism, (7) being present during critical events, (8) demonstrating interpersonal skills, and (9) being concerned about safety. Each question was scored on a 4-point Likert scale (never = 1, rarely = 2, frequently = 3, and always = 4). For each CA-1 resident, a supervision score was calculated for each study day by averaging responses to the 9 components. In addition to assessing individual quality of instructor supervision, the aggregate mean scores across a group have been used to assess an array of quality metrics.

A separate questionnaire was administered to measure the state and trait components of anxiety. This validated and widely used short form of the Spielberger State-Trait Anxiety Inventory contains 6 questions (STAI-6) on feelings of ease or upset.⁵ The abbreviated short form was derived from the traditional full STAI assessment tool in which anxiety scores range from 20 to 80, with higher scores indicating higher levels of general anxiety. The STAI-6 tool scores range from 6 to 24 and can be converted to scores that are compatible with the STAI instrument.⁵ The conversion formula used was the STAI-6 anxiety score divided by 6 then multiplied by 20. It has been demonstrated that the conversion does not change the results as the mean and SD are shifted by a factor.^6,7

The primary outcome was the difference in mean scores of the 9 items of the supervision instrument between groups. The secondary outcome included resident self-reported anxiety as measured by the STAI-6 instrument.

We used a convenience sample of all available novice residents (20) during their first 16 days of clinical training. We selected the first 16 days as the cut off because in prior years, few residents were deemed ready by our clinical competence committee to be indirectly supervised after 16 days of clinical training. To examine the relationship between score (supervision score and STAI-6 anxiety score, separately) and study group (attending or senior resident), a generalized linear mixed model accounting for repeated measures was used. This procedure models the correlations of the observations within the same CA-1 resident. Furthermore, it models for the correlations among the repeated measures for each CA-1 resident, for each supervisor type (attending or senior resident). To determine if the association between study group and score varied over the study period, the model included an interaction term for study group and day of the study. For each study group, the overall least square mean and corresponding standard error of the mean were calculated, along with the 95% confidence interval (95% CI). Results were also calculated for each study group, on each of the 16 study days. This method was used for both supervision score and STAI-6 anxiety score. In addition, individual components of the supervision score were compared between the study groups. Each of the 9 components of the supervision score was measured on a Likert scale with a possible value of 1 through 4. These ordinal categories were compared between study groups using a repeated measures cumulative logit proportional odds model. A correlation coefficient for supervision score and STAI-6 anxiety score was calculated using the method of Hamlett et al.⁸ for using mixed models to assess correlation in the presence of replication.

A P value of less than .05 was considered statistically significant. All statistical analyses were conducted with the use of SAS software version 9.4 (SAS Institute Inc).

Results

Twenty CA-1 residents were randomized daily to one of the study groups over the first 16 days of anesthesia training. Eight of 20 (40%) residents were women and 12 (60%) were men. The follow-up surveys were fully completed by all but 2 residents (days 11 and 14) throughout the study duration. The average clinical experience of the 42 supervising attendings is 16.6 years. Senior supervising residents comprised 12 CA-2 and 10 CA-3. There were no significant differences between senior residents’ supervision and faculty supervision regardless of the type of anesthesia administered (P = .743). Senior residents supervised 191 general anesthesia cases, 49 monitored anesthesia care cases, and 21 regional anesthesia cases compared with faculty supervision in 231 general anesthesia cases, 55 monitored anesthesia care cases, and 20 regional anesthesia cases. A study flow diagram is presented in Figure 1.

Figure 1. Study flow diagram.

There was no difference in supervision scores between CA-1 men and women for each study group. The overall mean supervision score across the study days was greater in the residents who were directly supervised by attendings, mean (SD) of 3.88 ± 0.034 compared with direct supervision by a senior resident, mean (SD) of 3.77 ± 0.034, mean difference of 0.11 (95% CI, 0.05-0.16), P = .0012. Five of 9 individual items on the supervision survey were significantly greater in the group directly supervised by attendings compared with residents (Table 1). The interaction between the study groups and day of the study was not significant (Figure 2). The day of study was also not significant.

Table 1.

Individual Items of the Supervision Scale Among Attendings and Senior Residents

Items	Mean ± SD	Odds Ratio	95% CI		P Value
			Lower	Upper
The instructor provides me timely, informal, nonthreatening comments on my performance and shows me ways to improve.	3.49 ± 0.58	0.50	0.29	0.87	.013
The instructor is promptly available to help me solve problems with patients and procedures.	3.72 ± 0.47	0.77	0.41	1.45	.420
The instructor uses real clinical scenarios to stimulate my clinical reasoning, critical thinking, and theoretical learning.	3.50 ± 0.59	0.48	0.29	0.79	.004
The instructor demonstrates theoretical knowledge, proficiency at procedures, ethical behavior, and interest/compassion/respect for patients.	3.70 ± 0.48	0.32	0.16	0.63	.001
The instructor is present during the critical moments of the anesthetic procedure (eg, anesthesia induction, critical events, complications).	3.84 ± 0.37	0.54	0.23	1.27	.155
The instructor discusses with me the peri-anesthesia management of patients prior to starting an anesthetic procedure and accepts my suggestions when appropriate.	3.61 ± 0.51	0.75	0.42	1.33	.319
The instructor teaches and demands the implementation of safety measures during the perioperative period (eg, anesthesia machine checkout, universal precautions, prevention of mediation errors, etc.).	3.72 ± 0.47	0.42	0.22	0.77	.006
The instructor treats me respectfully and strives to create and maintain a pleasant environment during my clinical activities.	3.77 ± 0.42	0.44	0.20	0.98	.044
The instructor gives me opportunities to perform procedures and encourages my professional autonomy.	3.59 ± 0.61	0.54	0.29	1.02	.057

Abbreviation: CI, confidence interval.

Presented as mean and SD and odds ratio with 95% CI. Each question was scored on a 4-point Likert scale (never = 1, rarely = 2, frequently = 3, and always = 4). Odds ratio represents the odds of the attending Likert score being in the lower Likert category compared with the senior resident score.

Figure 2. The interaction between the difference of supervision scores and day of the study among first-year clinical anesthesia residents (CA-1) supervised by attendings and senior residents. Least square means and 95% confidence interval of supervision scores reported during the study period.

There was no difference in anxiety level as measured by STAI-6 between the attending and the senior resident direct supervision groups, mean difference of −0.90 (95% CI, −2.6 to 0.81), P = .28 (Figure 3). When investigating the relationship between anxiety scores and supervision scores, there was a mild association with a correlation coefficient of 0.23 (95% CI, 0.08-0.39), P = .0035. The correlation between anxiety score and supervision score for attendings was 0.24 (95% CI, 0.05-0.42), P = .011, and for senior residents it was 0.26 (95% CI, 0.07-0.44), P = .008 (Figure 4).

Figure 3. The interaction between the difference of the 6-item State-Trait Anxiety Inventory (STAI-6) anxiety scores and day of the study among first-year clinical anesthesia residents (CA-1) supervised by attendings and senior residents. Least square means and 95% confidence interval of STAI-6 anxiety scores reported during the study period. Anxiety scores range from 20 to 80 with higher scores indicating higher levels of general anxiety.

Figure 4. Scatter plot of the interaction between supervision score and 6-item State-Trait Anxiety Inventory (STAI-6) anxiety scores among first-year clinical anesthesia residents (CA-1) supervised by attendings and senior residents. Triangles represent supervising attendings and circles represent supervising senior residents.

Discussion

The most important finding of the current investigation was the significant difference in mean aggregate supervision scores between novice anesthesiology residents directly supervised by attendings when compared with novice residents directly supervised by senior residents. The difference in supervision scores was significantly greater in novice residents directly supervised by attendings for 5 of 9 individual items of the supervision scale. Taken together, our results suggest that direct supervision by attendings may be preferable to optimize supervision of new anesthesiology residents.

Another important finding of the current investigation was that both supervision strategies achieved a mean score greater than 3.5. This is particularly important because scores lower than 3 have been previously associated with medication errors in anesthesiology trainees.² Although direct supervision by attending anesthesiologists resulted in greater supervision scores by novice anesthesiologists when compared with direct supervision by a senior resident, both can be implemented from a patient safety perspective.

Our results are clinically important, as the introduction of new residents to academic anesthesiology departments who are orienting residents during the month of July often leads to staffing challenges. Although our results favor direct supervision of novice residents by an attending, it does support the safety of direct supervision of novice residents by senior residents when few attendings are available.

It was interesting to note a small positive association between perceived supervision scores and anxiety scores. In contrast, we did not detect a difference in anxiety scores between novice residents directly supervised by attendings when compared with senior residents. Nonetheless, we detected a direct correlation, although a small one, between daily anxiety scores and perceived supervision.

Prior investigators have evaluated the effect of supervision on the clinical training of anesthesiology residents. Dexter and colleagues⁹ demonstrated that faculty supervision of anesthesia residents serves as an independent measure of an anesthesiologist's contribution to the quality of patient care. De Oliveira et al.² performed a national survey and detected that lower supervision scores were associated with greater self-reported medical errors by anesthesiology trainees. Furthermore, a mean supervision score across an entire department can be used for the overall assessment of resident training program supervision.¹⁰ To the best of our knowledge, this is the first study to evaluate supervision of novice anesthesiology residents and the first randomized study using supervision of residents as the primary outcome.

The current study should be interpreted only in the context of its limitations. First, the study period was limited to 16 working days to reflect the first month of training, but some programs may extend intraoperative orientation for new residents beyond 1 month. Although there was a statistical difference between attending and resident supervision, the actual scores may not correlate to an educational difference. Further, there are patient-related limitations that are inherent to cases assigned to new residents (case complexity, lack of subspecialty anesthesia). Evaluation tools that rate performance may be subjected to the halo effect; however, the number of supervisors in both study groups minimized the potential bias. In addition, indirect attending management of the supervising senior residents may pose a bias in the final assessment, as it is not strictly independent supervision by the senior resident. Last, despite being randomized, the study was performed in 1 single mid-sized residency class and this may limit generalizability of our results. Therefore, future studies to confirm or refute our findings are warranted.

In summary, we detected better supervision scores when novice anesthesiology residents were directly supervised by attendings when compared with senior residents. Nevertheless, direct supervision by senior residents still provided supervision scores consistent with a safe supervision practice. Academic anesthesiology programs seeking to optimize supervision of novice trainees may assign them to direct supervision by an attending, but programs that may have staff limitations can provide adequate supervision by assigning novice residents to direct supervision by a senior resident.

Funding Statement

Disclosures: None