The Amount of Supervision Trainees Receive during Neonatal Resuscitation is Variable and Often Dependent on Subjective Criteria

Abstract

Objective

Measure variation in delivery room supervision provided by neonatologists using hypothetical scenarios and determine the factors used to guide entrustment decisions.

Study Design

A survey was distributed to members of the American Academy of Pediatrics Section on Perinatal Pediatrics. Neonatologists were presented with various newborn resuscitation scenarios and asked to choose the level of supervision they thought appropriate and grade factors on their importance in making entrustment decisions.

Results

There was significant variation in supervision neonatologists deemed necessary for most scenarios (deviation from the mode 0.36–0.69). Post-graduate year of training and environmental circumstances influence the amount of autonomy neonatologists grant trainees. Few neonatologists have objective assessment of a trainees’ competence in neonatal resuscitation available to them and most never document how the trainee performed.

Conclusion

Delivery room supervision is often determined by subjective evaluation of trainees’ competence and may not provide a level of supervision congruent with their capability.

Introduction

Traditionally, after successful completion of the Neonatal Resuscitation Program (NRP) and one year of post-medical school training, residents are entrusted to lead uncomplicated neonatal resuscitations without supervision or formal assessment of their skills. (1) Their presumed competence in neonatal resuscitation is based on rotations in the newborn intensive care unit (NICU) with the associated delivery room experience trainees gain during their first year of residency.

In 1996, the Accreditation Council for Graduate Medical Education (ACGME) limited the amount of time residents could spend taking care of pediatric and neonatal intensive care patients to 6 months of their 3-year residency, including daytime rotations and night call. This was followed by duty hour restrictions in 2003, which became more restrictive in 2011. The 2011 guidelines limited postgraduate year (PGY)-1 residents to a maximum of 16 hours consecutive work. (2) As a result, contemporary first year residents spend significantly less time practicing neonatal resuscitation than they did 20 years ago and are less prepared for a more senior role.(2, 3) Subsequently, trainees entering a neonatology fellowship have less delivery room and NICU experience than they had before the restrictions were actualized. Despite the limited time residents have to become proficient in neonatal resuscitation, the American Board of Pediatrics expects that pediatricians are able to resuscitate and initiate stabilization of a neonate. (4)

We surveyed members of the American Academy of Pediatrics (AAP) Section on Perinatal Pediatrics to determine the level of supervision neonatologists thought necessary for a trainee in various hypothetical scenarios involving newborn resuscitation. Neonatologists also graded the importance of system, task, and trainee factors when deciding the level of supervision they give trainees. We hypothesized that neonatologists would rely heavily on the PGY of training to make entrustment decisions and that there is at least a moderate amount of variation in the level of supervision neonatologists grant trainees.

Methods

The survey was piloted at Indiana University prior to dissemination through the AAP Section on Perinatal Pediatrics. Baseline variability in supervision was assessed with a cross-sectional study that surveyed the 41-neonatology faculty associated with Indiana University. Neonatologists were given 4 hypothetical newborn resuscitation scenarios. The scenarios were written such that the convenience of attending the resuscitation for the neonatologist varied with regard to time of day or night and their proximity to the resuscitation. The scenarios also varied with regard to additional team members present for the delivery and the PGY of the trainee leading the resuscitation. The team members included a neonatal nurse practitioner (NNP) in only one of the scenarios because NNPs may have significant neonatal resuscitation experience (5) and their participation could influence the amount of supervision a trainee receives from neonatologists.

After each scenario, neonatologists were asked how much supervision they thought appropriate for the physician trainee in the scenario. They were able to choose from one of five levels of supervision in accordance with the five levels of supervision proposed by the International Competency-Based Medical Education Collaborative (6). Neonatologists were instructed that there are no right or wrong answers and they should select the level of supervision that most closely resembles how they currently practice. Prior to distribution, representative neonatologists reviewed the survey content to eliminate ambiguity and ensure clarity. The number of scenarios were limited to 4 to minimize the amount of time to complete the survey in an effort to maximize the percent response. Responses were collected in REDCap, a secure web-based application for managing online surveys.

After the survey was piloted at Indiana University, it was sent electronically to neonatologists through the e-mail directory of the AAP Section on Perinatal Pediatrics. Using REDCap’s branching logic feature, only neonatologists who work with physician trainees were able to complete the survey. The response rate for the pilot survey performed at Indiana University was 71%. The AAP Section on Perinatal Pediatrics represents about 2/3 of practicing neonatologists in the United States and Canada. REDCap branching logic collected responses from only those neonatologists who practice within a training program. Since the percent of neonatologists in the Section on Perinatal Pediatrics that supervise trainees is not available, we were not able to accurately calculate the national response rate for neonatologists. Approximately 16% of neonatology fellows surveyed through the AAP responded to the survey. Demographic information was collected from each neonatologist that completed the survey. Qualitative variation in responses was assessed using deviation from the mode (ModVR). Multinominal logistic regression was used to assess an association between neonatologists’ demographic characteristics and the amount of autonomy they grant trainees.

In the survey sent to the Section on Perinatal Pediatrics, neonatologists were given a list of factors that have been reported to influence entrustment decisions in other specialties. They were asked to rank each factor independently on a Likert-scale from 1–5, with 5 being the most influential in their decision about the level of supervision a trainee would need for newborn resuscitation. We also asked neonatologists how often they give feedback to trainees performing neonatal resuscitation and how often they document trainee competence in leading a resuscitation. Potential responses were always, often, sometimes, or never.

Members of the Section on Perinatal Pediatrics who identified themselves as neonatology fellows completed a separate survey regarding the level of supervision they have received in the delivery room and what factors they thought faculty took into consideration when making entrustment decisions. Fellows could respond to each question as often, sometimes, rarely or never.

There were no incentives offered to neonatologists or neonatology fellows to complete the survey.

Results

Of the 41 neonatologists associated with Indiana University, 29 responded to the survey for a 71% response rate. For the survey sent to the Section on Perinatal Pediatrics, 308 physicians that work in a hospital with physician trainees responded to the survey, of which 274 were neonatologists and 31 neonatology fellows. In the national survey, there was variability as to whether neonatologists take their night call from home (36.7%), in the hospital (39.3%), or a combination of both (24%). The majority of neonatologists who responded to the national survey (78.6%) work at a hospital with a neonatology fellowship-training program. Neonatologists were asked if NNPs routinely attend deliveries with trainees; 19.4 % of neonatologists reported that NNPs only attend high-risk deliveries with trainees, 53.6% reported that NNPs attend all deliveries with trainees, and 27% report that NNPs do not attend deliveries.

There was considerable variation in the level of supervision neonatologists deemed necessary in 3 of the 4 scenarios for both the pilot survey and the national survey (Table 1). The scenario with least variation (scenario 3) took place in the middle of the day with the neonatologist in close proximity to the resuscitation and a NNP on the resuscitation team. The scenario with the most variation (scenario 1) took place at 3 AM with the neonatologist taking call from home and a first year fellow leading the resuscitation team of residents without a NNP. In the national survey, there was no overall correlation between the amount of time since completing fellowship and the level of autonomy neonatologists thought appropriate for trainees in the scenarios (p=0.07).

Table 1.

Variation in level of supervision neonatologists thought necessary in 4 different resuscitation scenarios. IU represents the pilot survey performed at Indiana University (n=29) and AAP represents the survey sent to the Section on Perinatal Pediatrics (n=274). ModVR values are between 0 and 1 with low values corresponding to a small amount of variation. A value above 0.5 indicates moderately large variability.

Scenario	Medical condition	Time of day	Proximity of neonatologist	Level of trainee	IU ModVR	AAP ModVR
1	Term gastroschisis	3 AM	At home	PGY4	0.69	0.69
2	33 weeks congenital anomalies	Early PM	In- hospital call	PGY6	0.78	0.65
3	Unplanned extubation	AM rounds	In the NICU	PGY5	0.14	0.36
4	Term meconium	afternoon	In office near NICU	PGY3	0.78	0.52

Neonatologists were asked about factors they take into consideration when deciding the level of supervision a trainee needs in a situation that may require neonatal resuscitation (summarized in Table 2). The anticipated medical condition of the infant, prior resuscitation experience with the trainee, and trainees’ PGY were the most influential factors for most neonatologists. Approximately half of neonatologists depend heavily on word of mouth about a trainee to determine the level of supervision. Opportunity to give feedback, trainee confidence in their own resuscitation skills and trainee oral presentations on rounds were important determinates of entrustment for only about 1/3 of neonatologists. Few neonatologists said they took environmental circumstances, such as time of day or their proximity to the resuscitation, into strong consideration. A concern about legal liability was also not a major concern for most neonatologists.

Table 2.

Factors neonatologists take into consideration when making entrustment decisions for newborn resuscitation. Responses were on a Likert-scale from 1–5 with 5 being the most influential. The percentile represents the percent of neonatologists that responded with a level of ≥ 4.

Trainee, task, and system factors that neonatologists use to determine the level of supervision needed for neonatal resuscitation	Percentile
Anticipated medical condition of the infant	95.9
Prior resuscitation experience with trainee	88.2
Post graduate year of training	84.6
Word of mouth about trainee’s competence	56.4
Opportunity to give feedback	39.2
Trainee confidence in their own resuscitation skills	39.0
Trainee’s oral presentations on rounds	24.6
Concern regarding legal liability	17.0
Objective assessment of trainees’ previous resuscitations	14.8
Environmental circumstances (time of day)	7.7

Neonatologists indicated that their exposure to an individual trainee is usually more than 5 weeks of daytime rounding and/or more than 5 night calls per year. However, the survey did not distinguish between their time spent with residents versus fellows. Only 14.8% of neonatologists reported that they have an objective assessment of a trainees’ competence in neonatal resuscitation available to them when making entrustment decisions about trainees they have not observed longitudinally. Although there are neonatologists who attend deliveries for the opportunity to give feedback on trainee performance (Table 2), 33.3% only sometimes and 23.6 % never document trainee competence in newborn resuscitation when they observe a trainee lead the resuscitation.

Of the neonatology fellows who responded to the survey, 70.9% reported that at some point during their fellowship they have resuscitated a newborn with what they felt was inadequate supervision from an attending neonatologist. Conversely, 58.1 % reported that at times they do not feel as though they have sufficient autonomy in the delivery room for their level of training. When asked what factors they think neonatologists take into consideration when determining the appropriate level of supervision, they concurred with the responses from neonatologists that the PGY of training (83.9% -often) and anticipated medical condition of the infant (67.7% -often) are major determinants. However, contrary to the responses from neonatologists, 41.9% of fellows believe that often the time of day and convenience for the neonatologist influence the amount of supervision they receive for newborn resuscitations. Fellows also reported that the level of supervision they received is inconsistent amongst faculty at their own institution (54.9% reported wide to moderate variation).

Discussion

The ACGME is moving towards entrustable professional activities (EPAs) as a framework for the assessment of residents. An EPA is a “task or responsibility that can be entrusted to a trainee once sufficient competence is reached to allow for unsupervised practice.” (7) The pediatric EPA “resuscitate, initiate stabilization and triage to align care with severity of illness”, requires multiple complex competencies. (4) Although few neonates require full resuscitation after delivery, 10% need some assistance to establish ventilation. (8) Successful neonatal resuscitation requires an identified team leader who can delegate tasks, effectively communicate, and coordinate the team. Effective ventilation of the lungs remains the most important step in neonatal resuscitation (9). A physician trainee who serves as the team leader should be proficient in the use of the resuscitation equipment, the initial steps in stabilization including bag-mask ventilation, and know when to call for additional help.

We confirmed our hypothesis that there is moderate variability in the level of supervision neonatologists deem necessary for neonatal resuscitation. This variation is not secondary to institutional differences. When our survey was piloted at a single institution, we found the exact same pattern of variation as in the national survey (Table 1). Neonatology fellows who completed the survey confirmed that they feel supervision is inconsistent among faculty. The only scenario in which there was no significant variation in response was scenario 3, with a NNP as part of the team and the neonatologist in close proximity to the resuscitation. It may be that the circumstances in scenario 3 were less ambivalent than the other scenarios. Certainly, the presence of a NNP on the resuscitation team could influence the amount of supervision neonatologists think is necessary. Interestingly, the scenario that takes place at 3 AM with the neonatologist taking call from home has the most variation in both the local and national surveys. The impression of nearly half of neonatology fellows answering the survey was that neonatologists do take time of day and proximity into consideration when deciding whether to attend a delivery. However, neonatologists who completed our survey reported that they do not consider environmental circumstances such as proximity and time when deciding on the appropriate level of supervision. Extenuating circumstances, such as time of day, have been previously reported to influence entrustment of clinical trainees. (6)

The limited number of fellow responses may not give an accurate representation of fellow experience in training programs. Despite this limitation, it is concerning that of the fellows who responded, 70.9% indicated that at times they felt inadequately supervised at a delivery. There is a sense that trainees, including fellows, may desire more instruction during neonatal resuscitation. At the other end of the spectrum, fellows reported often feeling over supervised. An interview with anesthesia residents showed the same dichotomy. PGY1 residents reported working with inadequate supervision whereas residents further along in training felt over supervised. (10) Once trainees have successfully demonstrated competence, progressive independence is an important aspect of clinical training. Not giving trainees suitable trust deprives them of the opportunity to practice unsupervised and provide supervision to junior trainees (entrustment level 5).

There is limited data about the driver behind entrustment decisions for neonatal resuscitation in teaching hospitals. Reports from other specialty training programs show that faculty use a variety of resident qualities to determine the appropriate level of supervision. In a qualitative analysis of interviews with supervising physicians of an inpatient general medicine service, PGY of training, resident confidence, leadership, communication, anticipatory specialty and medical knowledge are taken into strong consideration when deciding when to entrust residents with unsupervised tasks. (11) Focus groups in obstetrics and gynecology reported that self-assessment by the trainee played an important role. (12) Interviews with internal and emergency medicine faculty revealed they use language assessments as an indication of trainees’ clinical competence. (13)

Medical education literature demonstrates that skills are acquired at different rates among individuals and the level of training should not be used as the sole surrogate for competence (14, 15). Instructors grading the performance of more senior trainees performing newborn resuscitation on a mannequin demonstrated that about half failed their assessment (16). Despite the evidence to the contrary, neonatologists responding to our survey indicated that they rely on the PGY of training to make entrustment decisions. Overall, many determinants of supervision appear to be relatively subjective and independent of objective and specific measures of trainee performance in neonatal resuscitation.

Many centers use simulation to teach and practice neonatal resuscitation. (17) We did not query neonatologists about using trainee performance in simulation to make entrustment decisions because simulation is generally intended for education rather than evaluation. Experts in education and simulation believe that competency during simulation should not be considered adequate evidence of clinical competence.(18, 19)

Ideally, the amount of supervision should depend on longitudinal observation of trainees. The ACGME requires “faculty interaction with trainees be of sufficient duration to assess their knowledge and skills in order to delegate the appropriate level of patient care authority and responsibility.” (20) Neonatologists who responded to our survey reported that they rely on their previous experience with trainees to make entrustment decisions (Table 2). Most neonatologist completing the survey work with both residents and fellows but the question about previous experience with a trainee did not distinguish between the two. With the limited time residents spend in the NICU, neonatologists are unlikely to have sufficient exposure to an individual resident to make informed entrustment decisions. Neonatologists practicing in a large fellowship program that includes satellite hospitals may also not have sufficient previous experience with an individual fellow to make entrustment decisions based on longitudinal observation. Neonatologists responding to our survey indicated that in lieu of sufficient previous experience with a trainee, only 14.8% have access to an objective assessment of trainees’ competence in neonatal resuscitation. This is not surprising given that neonatologists who responded to our survey also indicated that when they supervise a trainee lead a resuscitation, 33.3% only sometimes and 23.6 % never document competence. An assessment tool used in the delivery room to document trainee performance would give neonatologist more objective data on which to base their entrustment decisions. Cumulative sum analysis has been used to determine procedural competency in bag-mask ventilation for anesthesia residents (21) and perhaps should be used to establish competency in neonatal resuscitation.

A fundamental limitation of this study is that it is a survey and may not represent what neonatologists actually do in clinical practice. There are inherit problems with surveys; such as, differences in the response option format may alter results.(22) Piloting our survey locally mitigated some of the common problems with survey design. For instance we avoided leading or vague questions, double negatives, overlapping responses, and nonspecific or unfamiliar terms (23). For the hypothetical scenarios, we measured variability in the responses rather than the absolute response which diminishes some of the drawbacks of a cross sectional survey. However, without information on the proportion of neonatologists in the AAP section on Perinatal Pediatrics who supervise physician trainees, we cannot be confident that neonatologists who responded to our survey are representative of the majority of academic neonatologists.

Conclusion

The decision whether to entrust a trainee to lead a neonatal resuscitation unsupervised often is not based on an objective assessment of their competence. There appears to be a lack of standardization with regard to how much supervision trainees should have when leading a neonatal resuscitation. This study highlights the inconsistency in the supervision of trainees in neonatal resuscitation. With regard to trainee characteristics and level of entrustment, our survey demonstrates that at least some neonatologists still rely heavily on postgraduate year of training. Additionally, the inconvenience of attending a resuscitation yielded inconsistent results in our survey and thus this factor may influence some neonatologists’ decision making.

The challenge is to objectively measure trainees’ competence in newborn resuscitation and provide a level of supervision congruent with their capability. Allowing trainees to lead a neonatal resuscitation unsupervised without an objective assessment of their competence poses a potential patient safety threat in the first minutes of life, an especially vulnerable period. This study underscores the need for a validated real-time assessment tool of competence in neonatal resuscitation and standardization of delivery room supervision.