Psychometric Evaluation of the Modified Neonatal Resuscitation Program^® Adherence Assessment Tool When Utilized for In Situ Simulation and Telesimulation Scenarios

Abstract

Objective:

Neonatal resuscitation is a high acuity, low occurrence event that requires precision, teamwork, and rapid decision-making. Simulation using NRP^® guidelines allows teams to build competency and preparedness, improving neonatal outcomes. Accurate, reliable performance evaluation during simulation is essential to the provision of meaningful feedback. Evidence supporting the reliability and validity of existing tools when used in telesimulation (TS) is limited. We sought to evaluate the psychometric properties of the modified NRP^® (mNRP^®) tool when assessing interprofessional team performance in onsite and TS environments.

Study Design:

We employed a methodological design to conduct secondary analyses of data from a larger study using simulation to enhance resuscitation skills as assessed by the mNRP^®. Item-level data from 96 simulations was subjected to Classical Test Theory-based item analyses including evaluation of item difficulty, discrimination, and item-to-total correlation; reliability; and exploratory factor analysis (EFA). Interrater reliability (IRR) between novice and expert raters was assessed for a subset of cases.

Results:

Item difficulties reflected a desirable mix of difficulty in endorsement, suggesting that items capture a range of guideline adherence. Most item discrimination (range: 0.05–0.81) and corrected item-total correlation (range: 0.005–0.68) values had moderate-to-strong, positive correlations with total scores, indicating discriminative ability. EFA yielded three-and-four-components for the onsite and TS groups, respectively. Cronbach’s alpha was 0.76 (onsite) or 0.78 (TS). Lowest vs. highest mNRP^® quartiles differed significantly in both formats (p<0.001 for each), supporting construct validity. IRR ranged from 0.5 to 0.9, supporting moderate to good agreement between novice and expert raters.

Conclusions:

Findings provide evidence supporting the reliability and validity of the mNRP^® tool when applied in both the in situ and TS settings. IRR was acceptable for expert and novice evaluators. This analysis provides additional validity evidence for the mNRP^® when used to evaluate interprofessional team performance in both onsite and TS formats.

Introduction

Neonatal resuscitation is a high acuity, low occurrence (HALO) event that requires precision, teamwork, and rapid decision-making.¹ Simulation has been shown to be an effective training modality for improving neonatal resuscitation.^2–4 Maintaining these skills is particularly challenging for healthcare teams in rural settings with low birth volumes.^1–7 Neonatal Resuscitation Program^® (NRP^®) knowledge and comfort levels with complex resuscitation have been shown to be lower for both nurses and physicians in rural settings which can lead to disparities in neonatal outcomes compared to higher volume centers.^7,8 In rural settings, access to longitudinal simulation-based training becomes essential in maintaining team neonatal resuscitation skills.^7,9 To address the challenge of maintaining neonatal resuscitation proficiency in rural settings, we implemented a longitudinal telesimulation (TS) training program across 5 hospitals in our rural healthcare system.⁹ To evaluate program efficacy and team performance over time, we assessed team adherence to NRP^® in each simulation using a previously published NRP^® adherence assessment tool,^10,11 adapted to reflect updates in the NRP^® 8th edition.¹² The NRP^® adherence assessment instrument was first published by a group from the Neonatology Department of the University of California.¹⁰ Van der Heide et al adapted the tool for the 3rd edition of NRP^® and collected construct validity and reliability evidence for the tool in a training setting using a group of pediatric residents performing simulated resuscitation.^11,13 However, no validity evidence exists to understand tool performance with practicing interprofessional teams training in situ or via TS to support use in this context. Use of a current, robust, and psychometrically sound assessment tool is critical to being able to provide teams with meaningful and actionable performance feedback, as well as to assess changes in performance over time.

Therefore, the primary objective of this study was to evaluate the psychometric properties of the NRP^® adherence assessment tool, modified for the 8th edition of NRP^® (mNRP^®), when used for assessment of interprofessional delivery room team neonatal resuscitation performance in both TS and in situ simulation training. A secondary objective was to determine whether non-expert raters can reliably use the mNRP^® through evaluation of interrater agreement between novice and expert raters. By strengthening the evaluative framework alongside simulation delivery, we aim to ensure that neonatal resuscitation training in rural settings is not only accessible and ongoing, but also measurable and meaningful.

Materials and Methods

Study design

This was a secondary data analysis from a larger prospective, observational study on the use of simulation to enhance neonatal resuscitation skills as assessed by the mNRP^® (Figure 1).^9,14 The analysis was a part of a larger study that was reviewed by the Maine Health Institutional Review Board as exempt human subjects research.

Figure 1.

Modified NRP^® Scoring Tool domains and items evaluated.

Notes: ECG = electrocardiogram; O₂ = oxygen; FiO₂ = fraction of inspired oxygen; PPV = positive pressure ventilation; CO₂ = carbon dioxide; HR = heart rate; BPM = beats per minute; mL = milliliters; kg = kilograms; IV = intravenous. Adapted from van der Heide et al. (2006), with updates from the NRP 8^th edition (Weiner, 2021) and Gelbart et al. (2010).^11,12,23

Dataset

The dataset used for this secondary analysis was from the Maine Ongoing Outreach Simulation Education (MOOSE) telesimulation program which delivered interprofessional neonatal resuscitation scenarios to rural community hospitals (Supplementary Material 1).^9,15 The MOOSE program generated a dataset that included team adherence to NRP guidelines using the mNRP for each of the hospitals the program was implemented at. The subset of data utilized for this analysis included the rater, date of TS session, and the subcategory and cumulative scores for NRP adherence. Subcategory scores included Situational Awareness and Team Function, Preparation and Initial Steps, Communication of Heart Rate to Lead Resuscitator, Oxygen/Bag Mask Ventilation, Intubation, Cardiac Compressions, Access, and Drugs/Volume; each subcategory received a subcategory scores and subcategory scores were totaled to calculate the cumulative score, which was then converted to a percentage to represent percent adherence to NRP guidelines.

Measurement

Initially two neonatologists (MM and AZ) scored every recorded simulation independently using the mNRP and then met to reconcile rating assessments, rewatching videos to come to agreement as necessary. The mNRP^® was adapted from van der Heide and colleagues (2006) to accommodate the 8^th edition of NRP^®.¹¹ These adaptations included addition of ECG monitoring, lack of immediate suctioning for infants with meconium, and removal of naloxone use. These item-level mNRP^® scores for each simulation event served as the dataset for the current study.

Midway through the program, an interclass coefficient (ICC) was established between the two neonatologist raters using independent scores from 12 simulations before rater reconciliation. The ICC was 0.99 for overall scores and no less than 0.93 for each section of the scoring tool; from that point on, one neonatologist scored each recording. In the current study, a separate ICC analysis using 13 different recorded TS videos to determine the reliability of including NRP novice raters (two medical students not trained in NRP) with minimal (one hour) of rater training and one of the two neonatologists was conducted.

Data analysis

Categorical variables were summarized using frequencies and proportions while continuous variables have been summarized using measures of central tendency and dispersion. Confidence intervals (CIs) were calculated using the exact method and significance was set at an α of 0.05. Data were analyzed using SPSS for Windows v. 30.0 (IBM Corp., Armonk, NY) and R (R Core Team, 2024) v. 4.2.1 statistical software.

Reliability analysis.

Internal consistency reliability and interrater reliability were evaluated for the mNRP^® tool. Internal consistency reliability was assessed through computation of Cronbach’s coefficient α for the scale, with a minimum value for acceptable internal consistency being 0.70.¹⁶ Interrater reliability was examined for a subset of simulation cases (N=13) by calculating the intraclass correlation coefficient to examine the relationship between scores assigned by the four blinded, independent raters (two neonatologists, and between one neonatologist and two medical students). A two-way random effects model based on the mean rating (k = 3) and examining absolute agreement was selected.¹⁷

Item analyses.

Item analyses included assessments of item difficulty (arithmetic mean) and item discrimination (point biserial index). Items with low corrected item-total correlations (below 0.30) were identified.¹⁸

Factor analysis.

Modified NRP^® tool items were subjected to exploratory factor analytic techniques. Appropriateness for factoring was determined using the Kaiser-Meyer-Olkin statistic, assessment of the determinant |R|, and Bartlett’s test of sphericity. Following principal component analysis with iterations, oblique (direct oblimin) rotation and Kaiser normalization were completed to examine the internal structure of the data. Oblique rotation was used recognizing the correlated nature of the identified components.¹⁹ Kaiser-Guttman’s criterion of using all unrotated components with eigenvalues greater than 1.0 was applied to identify components accounting for the maximum amount of variance.^16,20 An additional criterion for determining the number of components to retain was that each component needed to account for the variance of at least two items.²¹

Validity assessments.

Construct validity for the mNRP^® tool was evaluated using the known-groups method.²² Here, for each simulation type cohort, we compared mean cumulative mNRP^® scores for those known to be the poorest (lowest quartile of scores) and best (highest quartile of scores) performing teams using the t-test for independent samples.

Results

Item-level modified NRP^® assessment tool data from 96 simulation events comprised the sample; 23 (24%) were onsite simulations and 73 (76%) were telesimulations. Characteristics of the participating sites and simulation events are summarized in Table 1.

Table 1.

Characteristics of study sites and simulation events.

Characteristic	n (%)
Critical access hospital designation
Yes	4 (50.0)
FORHP rural designation
Yes	7 (87.5)
Licensed beds
≤ 25 beds	4 (50.0)
26–100 beds	1 (12.5)
101–200 beds	3 (37.5)
Average Number Live Births Annually
100–200	2 (25.0)
201–300	3 (37.5)
301–400	0 (0.0)
401–500	1 (12.5)
501–600	1 (12.5)
>600	1 (12.5)
Simulation type
In-person	23 (24.0)
Telesimulation	73 (76.0)

Notes: FORHP = Federal Office of Rural Health Policy

Item analysis

Onsite simulation cohort.

For the onsite simulation group, inter-item correlations ranged from 0.017 for the drug score and heart rate assessment item pair to 0.983 for the overall equipment check and equipment check item pair (Table 2). Similarly, uncorrected item-to-total correlations (item discrimination values) ranged from 0.055 for cumulative score and oxygen total score to 0.775 for cumulative score and ventilation-BVM. As the item of interest itself is included in the computation of the uncorrected correlation (r), an inflated r value results. To combat this, the correlation is re-calculated without the item of interest, resulting in corrected item-to-total correlations. For the onsite simulation cohort, corrected item-to-total correlations ranged from 0.005 for cumulative score and oxygen total score to 0.559 for cumulative score and ventilation-BVM. As the modified NRP^® tool data are continuous in nature, item difficulties are the arithmetic mean for each item, rather than a ratio of correct to all responses as is the case for dichotomous data. Item difficulties for the modified NRP^® tool applied to the onsite simulation cohort ranged from 4.70 to 26.40. Table 3 presents item difficulties and item discrimination (uncorrected and corrected item-to-total correlations) values for the modified NRP^® tool.

Table 2.

Item-to-total correlations for the onsite simulation cohort.

	Group Function	Equipment Check	Overall Equipment Check	Heart Rate	Oxygen Total	Ventilation BVM	Intubation Technique	Total Intubation	Cardiac Compression Technique	Drug Score
Group Function	1.00
Equipment Check	0.509 *	1.00
Overall Equipment Check	0.553 **	0.983 **	1.00
Heart Rate	0.354	0.112	0.107	1.00
Oxygen Total	0.015	−0.384	−0.306	0.019	1.00
Ventilation BVM	0.572 **	0.511 *	0.521 *	0.566 **	0.074	1.00
Intubation Technique	0.304	0.054	0.041	0.297	0.156	0.146	1.00
Total Intubation	0.340	−0.014	0.070	0.207	0.266	0.105	0.949 **	1.00
Cardiac Compression Technique	0.313	0.268	0.232	0.225	0.168	0.568 **	0.069	−0.011	1.00
Drug Score	0.188	0.370	0.330	0.017	−0.275	0.350	0.389	0.302	0.231	1.00

Notes:

^*= correlation is significant at the 0.05 level (Pearson’s correlation, 2-tailed);

^**= correlation is significant at the 0.01 level (Pearson’s correlation, 2-tailed); BVM = bag valve mask

Table 3.

Item difficulty, uncorrected and corrected item-total correlations, and alpha if item deleted for the onsite simulation cohort.

Item	Item difficulty*	Uncorrected item-total correlation**	Corrected item-total correlation†	Cronbach’s Alpha if item deleted
Group Function	4.90	0.651	0.543	0.687
Equipment Check	7.90	0.451	0.506	0.685
Overall Equipment Check	7.85	0.464	0.475	0.687
Heart Rate	4.70	0.468	0.374	0.703
Oxygen Total	4.90	0.055	0.005	0.717
Ventilation BVM	21.45	0.775	0.559	0.643
Intubation Technique	26.40	0.552	0.525	0.655
Total Intubation	24.70	0.551	0.412	0.679
Cardiac Compression Technique	15.90	0.647	0.341	0.702
Drug Score	23.40	0.653	0.465	0.671

Notes:

^*= item difficulty is the arithmetic mean score for each item;

^**= uncorrected item-total correlation: Pearson’s correlation coefficient (r) for each item and the cumulative total score;

^†= corrected item-total correlation: Inflation-corrected point biserial correlation value (σ_pbis) for each item and the cumulative total score; BVM = bag valve mask.

Telesimulation cohort.

Data for the telesimulation group were examined in a similar fashion. Inter-item correlations ranged from 0.047 for the total intubation score and equipment check item pair to 0.972 for the overall equipment check and equipment check item pair for the telesimulation group (Table 4). Uncorrected item-to-total correlations (item discrimination values) ranged from 0.333 for cumulative score and heart rate assessment to 0.814 for cumulative score and ventilation-BMV. For the telesimulation cohort, corrected item-to-total correlations ranged from 0.124 for cumulative score and heart rate assessment to 0.680 for cumulative score and group function score. Item difficulties for the modified NRP^® tool applied to the telesimulation cohort ranged from 5.29 to 29.66. Table 5 summarizes item difficulties and item discrimination (uncorrected and corrected item-to-total correlations) values for the modified NRP^® tool.

Table 4.

Item-to-total correlations for the telesimulation cohort.

	Group Function	Equipment Check	Overall Equipment Check	Heart Rate	Oxygen Total	Ventilation BVM	Intubation Technique	Intubation Total	Cardiac Compression Technique	Drug Score
Group Function	1.00
Equipment Check	0.264 *	1.00
Overall Equipment Check	0.260 *	0.972 **	1.00
Heart Rate	0.170	0.181	0.175	1.00
Oxygen Total	0.530 **	0.338 **	0.323 **	0.147	1.00
Ventilation BVM	0.602 **	0.328 **	0.330 **	0.190	0.570 **	1.00
Intubation Technique	0.305 *	0.153	0.178	−0.056	0.095	0.255	1.00
Total Intubation	0.263 *	0.047	0.081	−0.051	0.055	0.157	0.880 **	1.00
Cardiac Compression Technique	0.565 **	0.079	0.100	0.222	0.421 **	0.704 **	0.098	0.078	1.00
Drug Score	0.343 **	0.253 *	0.291 *	0.177	0.176	0.230	0.316 **	0.204	0.191	1.00

Notes:

^*= correlation is significant at the 0.05 level (Pearson’s correlation, 2-tailed);

^**= correlation is significant at the 0.01 level (Pearson’s correlation, 2-tailed); BVM = bag valve mask.

Table 5.

Item difficulty, uncorrected and corrected item-total correlations, and alpha if item deleted for the telesimulation cohort.

Item	Item Difficulty	Uncorrected Item-total correlation	Corrected item-total correlation	Cronbach’s Alpha if Item Deleted
Group Function	6.09	0.707	0.680	0.648
Equipment Check	10.27	0.345	0.322	0.684
Overall Equipment Check	10.03	0.370	0.355	0.682
Heart Rate	5.29	0.333	0.124	0.698
Oxygen Total	5.51	0.528	0.479	0.678
Ventilation BVM	27.94	0.814	0.637	0.602
Intubation Technique	29.66	0.489	0.487	0.648
Total Intubation	28.23	0.445	0.339	0.673
Cardiac Compression Technique	24.51	0.784	0.450	0.679
Drug Score	18.04	0.567	0.340	0.677

Notes:

^*= item difficulty is the arithmetic mean score for each item;

^**= uncorrected item-total correlation: Pearson’s correlation coefficient (r) for each item and the cumulative total score;

^†= corrected item-total correlation: Inflation-corrected point biserial correlation value (σ_pbis) for each item and the cumulative total score; BVM = bag valve mask.

Reliability

Internal consistency reliability.

Internal consistency reliability, as measured by Cronbach’s alpha coefficient, was 0.76 for the onsite simulation cohort and was 0.78 for the telesimulation cohort using the modified NRP^® tool. We did not evaluate test-retest reliability, or the ability of the instrument to provide consistent results over time, as performance on the modified NRP^® measure is expected to vary, rather than remain stable, over time.

Interrater reliability.

For a subset of 13 simulation events, intraclass correlation coefficients (ICCs) and their associated 95% CIs were computed to examine agreement between raters using the mNRP^® tool. The ICC for the cumulative score for all raters was 0.74 (95% CI: 0.36–0.91). Subcategory scores ranged from 0.45 (95% CI: −0.35–0.82) for ventilation to 0.88 (95% CI: 0.70–0.96) for oxygen, indicating very good agreement between raters. Supplemental Table 1 summarizes these findings across all mNRP^® domains.

Exploratory factor analysis.

The mNRP^® tool, as applied to the onsite simulation cohort, was subjected to principal components analysis, with application of the aforementioned retention criteria resulting in a three-component solution accounting for 74.0% of common variance (Supplemental Tables 2 and 3). Examination of the scree plot of eigenvalues (Supplemental Figure 1, Panel A) provided further evidence supporting a parsimonious three-component solution. This approach was repeated for the telesimulation cohort, resulting in a four-component solution accounting for 79.3% of common variance (Supplemental Tables 2 and 3; Supplemental Figure 1, Panel B).

Validity

Construct validity.

The mean cumulative score for onsite simulations was 104.1 (SD 27.160, 95% CI: 92.1–116.2; range 60–152) while the mean for telesimulations was 124.3 (SD 27.063, 95% CI: 118.0–130.7; range 46–172). In comparing the top (high score = excellent performance) and bottom (low score = poor performance) quartile cumulative scores for each simulation format (onsite and telesimulations), statistically significant differences were noted. For the onsite group, the mean difference in cumulative score was −72.2 points (95% CI: −87.0 - −57.4 points), t=−12.027, df=5.863, p<0.001, equal variances not assumed. Similarly, for the telesimulation group, the mean difference in cumulative score was −67.0 points (95% CI: −76.4—57.7 points), t=−14.785, df=23.701, p<0.001, equal variances not assumed.

Discussion

Accurately assessing neonatal resuscitation simulation performance is essential in ensuring that care teams receive meaningful, actionable feedback that can support skill development. In this study we demonstrated strong validity evidence for a modified Neonatal Resuscitation Program (mNRP^®) adherence assessment tool, adapted for the 8^th edition of NRP^®. Our analysis demonstrated very good interrater reliability (IRR) between expert and novice raters as well as internal consistency reliability, suggesting the tool performs consistently across different evaluators and simulation formats. Construct validity was supported by the tool’s ability to differentiate between high and low performing teams, while item-level analysis showed that the items represent a reasonable distribution of difficulty, revealing that the tool appropriately challenges and assesses teams across skill levels. Notably, performance outcomes did not differ significantly between in-person and TS environments, supporting the tool’s effectiveness across different settings.

The findings of this study add to the work of van der Heide et al., who demonstrated the reliability and construct validity of a previous version of the NRP^® adherence tool used with pediatric residents.¹¹ Our study extends this work by adapting the tool to the 8^th edition guidelines and applying it in rural interprofessional team settings using both in-situ and TS formats. To our knowledge, this is the first study to evaluate the tool’s validity evidence under these specific conditions.

An unexpected finding emerged with item-level analysis related to the oxygen item. The in-person simulation group showed significantly lower item-to-total correlation values for oxygen compared to its performance in the TS group. While this item performed within acceptable ranges in the TS cohort, it fell below 0.4 for the in-situ simulations. This discrepancy may be due to the smaller sample size in the in-situ group. Additionally, the oxygen component includes four separate scoring elements, which may introduce more variability and subjectivity in rater interpretation.

Despite the discrepancy with oxygenation, cumulative score and other subscore performance patterns were consistent between TS and in-situ groups. This suggests that the mNRP^® tool functions similarly across both settings, an important finding in addressing the need for improved access to NRP^® TS for rural healthcare systems.

While these findings are significant, certain limitations must be considered. There were fewer in-person simulations compared to TS, which could impact the ability to detect differences between the two settings. Additionally, the IRR analysis came from a smaller subset of simulations (n=13), introducing some uncertainty and limiting the precision of the results. Despite this, IRR was strong enough that a smaller sample size was adequate for statistical significance and appropriately narrow confidence intervals, though increasing sample size in the future for this study overall would improve the precision of our point estimates. Additionally, our rural training sites also request simulations for more common neonatal emergencies than a “full code” such as neonatal respiratory depression that responds to effective ventilation or pre-term deliveries. Our analysis of mNRP^® tool validity was only for full resuscitation scenarios (including chest compressions), a very rare occurrence at low birth volume hospitals. Additionally, release of the ninth update to NRP^® is expected in the upcoming year. As these cyclic updates are made to the tool to align with emergent evidence and clinical best practices, modification and further validity assessment of mNRP^® properties will be needed.

These findings have meaningful implications for neonatal resuscitation training, particularly in rural or resource-limited settings. As simulation continues to grow as an important aspect to medical education, having access to a valid assessment tool that functions well across different delivery formats allows for more accessible training programs. The tool’s ability to deliver consistent, meaningful feedback across simulation environments means rural hospitals can maintain high standards in neonatal resuscitation education with the use of TS. This could help close gaps in provider readiness and reduce disparities in neonatal outcomes across rural and urban settings.⁷

Looking ahead, future studies could build on this work by applying an item response theory (IRT) framework to better understand how specific areas of the tool perform compared to others. IRT could help assess whether certain items function differently in different environments or team compositions and could be used to aid in the development of a shorter assessment tool. As we attempt to establish more sustainable local training programs, there is also a need for a more feasible assessment tool that can be completed real time by educators and does not require video recording and later review. Traditionally, updates to NRP^® occur regularly, approximately every five years unless emergent evidence necessitates an earlier update. As these guidelines are updated, new simulation technologies emerge, and new settings are encountered, continued reassessment of the psychometric properties of NRP^® adherence tools is indicated. Additionally, while this study focused on simulated performance, future studies could explore whether scores from the mNRP tool correlate with clinical outcomes, a key step in demonstrating predictive validity.

Conclusions

Classical test-theory based assessment of the psychometric properties of the mNRP^® tool support the instrument as sufficiently reliable and valid for use in assessing neonatal resuscitation team performance in both in-person and telesimulation formats. A valid assessment tool is critical to improving clinician performance during high acuity low occurrence events by enabling accurate feedback during training and facilitating assessment of change over time. Future health professions educators, clinicians, and researchers can apply findings from this study as they assess team adherence to NRP^®. Further, as innovations in simulation technology continue to evolve, additional analysis will be necessary to evaluate tool performance when applied in those settings and novel circumstances.

Supplementary Material

Supplementary Figure 1

Supplemental Figure 1. Scree plots for the onsite (Panel A) and telesimulation (Panel B) cohorts.

Panel A. Scree plot for the onsite simulation cohort.

Panel B. Scree plot for the telesimulation cohort.

Notes: Scree plot of eigenvalues for the mNRP^® scoring tool when use in onsite (Panel A) and telesimulation (Panel B) cohorts. The scree plot in Panel A depicts a sharp descent in the curve, or point of inflection, at the fourth component while the inflection point for Panel B is at the fifth component. Cattell’s (1966) recommendation of retaining only components to the left of the inflection point supports three- and four-component solutions, respectively.²⁴

Key Points.

• Neonatal resuscitation is a high-acuity, low-occurrence event.

• Many rural clinicians feel underprepared for HALO events.

• Simulation-based training can improve preparedness.

• Performance is assessed using the mNRP^® tool.

• Psychometric features support the tool for use in onsite and telesimulation formats.