Abstract
Objective
Develop a standardized simulation method to assess clinical skills of ICU providers
Design
Simulation assessment
Setting
Simulation laboratory
Subjects
Residents, CCM Fellows, ACNP students
Interventions
Performance scoring in scenarios from multiple CCM competency domains.
Measurements and Main Results
384 performances by 48 participants were scored using checklists (% correct) and holistic ‘global’ ratings [1(unprepared) to 9(expert)]. 180 were scored by two raters. Mean checklist and global scores(±SD) ranged from 65.0%(±16.3%) to 84.5%(±17.3%), and 4.7(±1.4) to 7.2(±1.2). Checklist and global scores for CCM Fellows and senior ACNP students (Experienced Group, n=26) were significantly higher than those for the Novice ACNP students (Novice Group, n=14) (75.6±15.6% vs.68.8±21.0%, and 6.1±1.6 vs. 5.4±1.5, respectively; p<0.05). Residents (Intermediate Group, n=8) scored between the two (75.4±18.3% and 5.7±1.7). 38.5% of the Experienced Group scored in the top quartile for mean global score, compared to 12.5% of the Intermediate and 7.1% of the Novice Groups. Conversely, 50% of the Novice Group scored in the lower quartile (<5.3), compared to 37.5% of the Intermediate and 11.5% of the Experienced Groups. Psychometric analyses yielded discrimination values >0.3 for most scenarios, and reliability for the 8-scenario assessments of 0.51 and 0.60, with inter-rater reliability of 0.71 and 0.75, for checklist and global scoring, respectively.
Conclusions
The simulation assessments yielded reasonably reliable measures of CCM decision-making skills. Despite a wide range of performance, those with more ICU training and experience performed better, providing evidence to support the validity of the scores. Simulation-based assessments may ultimately prove useful to determine readiness to assume decision-making roles in the ICU.
Keywords: Patient Simulation, Critical Care, Intensive Care Units, Educational Measurement, Clinical competence, Nurse Practitioners
Introduction
Several million people in the United States admitted annually to the intensive care unit (ICU) rely on the skill of ICU clinicians for life-saving interventions during their ICU stay (1–4). As ICU admissions continue to rise (2), there has been a parallel increase in the demand for qualified ICU providers (3–6). For physician specialists (‘intensivists’), this demand has been amplified by consumer advocacy for intensivist-led ICU care, which has been shown to reduce ICU mortality and lengths of stay (1–3, 7–9). The result has been a growing shortage of intensivists which has been exacerbated by safety-related work hour restrictions for physician trainees, who served as ‘front line’ ICU providers to manage the intensivist supply-demand gap (10–12). A rapid expansion of the critical care work force has ensued to now include a large number of non-physician ‘advanced practice’ ICU providers (APPs) (13–15), largely comprised of acute care nurse practitioners (ACNPs) and physician assistants (PAs) with ICU training and experience. In collaboration with intensivists, these new ‘front line’ ICU providers are responsible for moment-to-moment coverage and management decisions in the ICU (14–16), and are thus essential to produce the improved ICU outcomes with intensivist-led ICU care (7, 8, 17, 18).
The educational preparation of physician trainees, ACNPs, and PAs who fill front line ICU provider roles differ considerably, but all must be able to promptly recognize and respond appropriately to changes that portend a serious deterioration in a patient’s condition (13–15). The often limited ICU exposure during their professional education (19–20), however, can lead to early learning-based errors, which can have devastating for patients, and often for the provider(s), whose negative experience leads them to believe they are unsuited for ICU work, and drop out of the ICU workforce (21).
Simulation provides an experiential learning environment that improves health care professional preparation while eliminating the potential for patient harm (22–25). Simulation may also offer a potential means to assess providers clinical decision-making skills using a standardized approach and environment (26).
The aims of the study were 1) to develop an inventory of simulated ICU conditions (scenarios) that could be used to assess whether participants have acquired the cognitive skills necessary to assume ‘front line’ ICU provider roles, and 2) to gather evidence to support the psychometric adequacy of the performance metrics for the scenarios. From a validity perspective, we hypothesized that participants with more ICU experienced would perform better. The longer term goal was to develop a standardized simulation methodology that can be used to reliably assess provider readiness to manage patients in the ICU.
Methods
Scenarios
A list of common ICU conditions was developed through review of administrative databases and published critical care provider competencies (19, 20, 27). Academic intensivists (n=14) were surveyed using a Delphi technique to narrow the list to 16 conditions used to construct scenarios (28). The research team developed and refined the scenarios including the standardized patient behaviors and responses, and the scripted responses from the ‘standardized nurse’. Patient findings were simulated using Meti HPS® and iStan® electromechanical mannequins. Additional laboratory and/or radiologic data were available upon request of the participant. Two 8-scenario forms (“A” or “B”) were designed to be equivalent and completed in a session lasting 90-120 minutes (see Table 1).
Table 1. Critical Care Scenarios.
Simulated ICU conditions and information provided to participants at the beginning of each scenario.
| Form A Scenarios | Form B Scenarios |
|---|---|
|
1. 3rd Degree Heart Block. 75 year old patient admitted after treatment of ventricular tachycardia with cardioversion. Amiodarone is infusing. HR 40, BP 80/40, RR 12, O2 sat 92%. Patient is responsive, but disoriented. |
1. Atrial Fibrillation. 75 year old female patient 12 hours following posterior cervical spinal fusion for atlantoaxial subluxation. HR 140, BP 80/40, RR 12, 02 Sat 92%. Patient is responsive, but disoriented. |
|
2. ARDS. 60 year old male intubated and ventilated for 24 hours following colectomy, now with worsening blood gases. FiO2 is 100%, O2 Sat is 89, increasing dyspnea and hypoxia. HR 130, BP 110/80, RR 28, O2 sat 88%. Patient appears dyspneic. |
2. Pulmonary Embolus. 60 year old male admitted to ICU 48 hours following colectomy with increasing dyspnea and hypoxia. HR 130, BP 110/80, RR 28, O2 sat 88%. |
|
3. Asthma. 40 year-old patient in the ICU 12 hours after total colectomy and end ileostomy. HR 120, BP 120/85, RR 24, O2 sat 89% on 2 liters Nasal Cannula. Patient is alert, dyspneic and complains of incisional pain. |
3. Atelectasis. 40 year-old patient in the ICU 36 hours after total colectomy and end ileostomy. HR 120, BP 120/85, RR 24, O2 sat 89% on 2 liters Nasal Cannula. Patient is alert, dyspneic and complains of incisional pain. |
|
4. Low Urine Output (Obstructed Foley) 45 year-old patient 24 hours after non-anatomic liver resection. HR 110, BP 130/90, RR 16, O2 sat 95%. Patient is complaining of abdominal pain. Urine output is absent for 2 hours. |
4. Low Urine Output (Hypovolemia) 45 year-old patient 24 hours after non-anatomic liver resection. HR 110, BP 80/40, RR 12, O2 sat 92%. Urine output has been 10 ml for the last three hours. A fluid bolus was administered an hour earlier. |
|
5. Cardiogenic Shock. 50-year-old male. Transferred from floor to ICU. One week after perforated diverticulitis and colectomy with primary anastomosis. HR 40, BP 80/40, RR 12, O2 sat 95%. Patient is responsive, but disoriented. |
5. Septic Shock. 50-year-old male. Transferred from floor to ICU. One week after perforated diverticulitis and colectomy with primary anastomosis. HR 40, BP 80/40, RR 12, O2 sat 95%. Patient is responsive, but disoriented. |
|
6. Endotracheal Tube Cuff Leak. 62-year-old ventilated patient in the ICU on POD 1 after resection of abdominal wall tumor. Ventilator is alarming (low pressure). HR 115, BP 100/70, RR 12, O2 sat 86%. Patient is responsive, but disoriented. |
6. Mucus Plug. 62-year-old ventilated patient in the ICU on POD 1 after resection of abdominal wall tumor. Ventilator is alarming (high pressure). HR 115, BP 100/70, RR 12, O2 sat 86%. |
|
7. Hyperkalemia due to Pulseless Leg. 25 year-old ICU POD 1 after ORIF right leg, soft tissue and extensive vascular repair secondary to MVA. HR 110, BP 110/70, RR 12, O2 sat 95%. Patient is responsive, but disoriented. Monitor tracing indicates peaked T waves. Patient is responsive, but disoriented. Potassium returns 7.4 meq/dL |
7. Hyperkalemia due to Crush Injury. 25 year-old ICU POD 1 after ORIF right leg, soft tissue and extensive vascular repair secondary to MVA. HR 110, BP 110/70, RR 12, O2 sat 95%. Monitor tracing indicates peaked T waves. Patient is responsive, but disoriented. Potassium returns 7.4 meq/dL |
|
8. Ischemic Stroke. 65 year old patient in the ICU on POD 1 after duodenal resection. HR 90, BP 130/70, RR 12, O2 sat 95%. Patient is responsive, but disoriented. On neurologic examination, patient has right-sided weakness. |
8. Opioid Overdose. 65 year old patient in the ICU on POD 1 after colonic resection. HR 70, BP 100/60, RR 6, O2 sat 88%. Patient is responsive, but disoriented. Pupils are 2mm. |
Participants
The study was approved by the Institutional Review Board and informed consent was obtained from 48 participants. These included 14 novice and 4 senior ACNP students, 8 residents, and 22 CCM fellows. 21 were male and 27 were female. Average age was 32.7 years (range 26-52). The ACNPs completed undergraduate nursing programs between 1990 and 2012. The novice ACNPs were evaluated during the first 2 months, and the senior ACNPs in the last 2 months, of their year-long ACNP clinical training. The ACNP students had an average of 7.5 years (SD=6.2) of prior critical care nursing experience. The residents and CCM fellows graduated from medical school between 2003 and 2015. The residents included 7 PGY-1s and 1 PGY-2 from Anesthesiology or Surgery training programs. The CCM fellows completed residency programs in Anesthesiology (n=8), Emergency Medicine (n=8), or General Surgery (n=6). The residents had less than 3 months of critical care experience. Most CCM fellows (82%) had at least 8 months of critical care experience at the time of the simulation assessments. All participants had previous simulation experience.
Simulation Assessment
During each participant’s assessment, an instructor, a ‘standardized’ nurse, and a simulation operator managed the standardized simulation experience. The instructors were certified critical care specialists and members of the critical care faculty and the research team. In a few instances, instructors had prior exposure to a participant. However, in no case did an instructor feel there was an issue with objectivity as a rater. The ‘standardized’ nurse, who acted as the ‘bedside’ ICU nurse, had 20 years of experience as a critical care nurse and 5 years of experience in simulation education. For each scenario, the participants were ‘called’ by the bedside ICU nurse who provided the scripted reason for the call (Table 1) and answered participant’s questions with the scripted responses. In the control room adjacent to the simulation suite, the operator managed the mannequin interface in a standardized fashion, and the instructor scored the performance.
Checklist Scoring
The participants were scored analytically by the instructor during the live viewing of the performance using scenario-specific checklists that included items considered important for diagnosis and initial management (see Supplemental Data).
Global Performance Scoring
To assess overall management, participant performances were also ‘holistically’ scored by the instructor using a global rating scale from 1 (‘unprepared’) to 9 (‘expert’).
The 48 participants each completed an 8 scenario assessment (A or B) for a total of 384 individual performances. 180 were scored by a second rater from a video recording. Prior to viewing performances, raters reviewed the checklist scoring rubric and global rating scale.
Post-assessment survey
Participants completed post-assessment surveys regarding the simulation experience using a 5-point scale (1=strongly agree to 5=strongly disagree). Participants were asked about whether (1) the patient presentations and environment were realistic, (2) the scenarios were representative of practice, (3) the scenarios were relevant to practice, (4) the scenarios would be stress provoking in practice, and (5) the scenarios would improve clinical performance.
Participant Performance
Descriptive statistics were used to summarize performance by scenario. Scenario discrimination (D) values (the correlation between individual scenario scores and total scores across scenarios) were calculated to provide a measure of how well performance on a single scenario predicted the participant’s overall performance.
Reliability
Variance components were calculated to estimate the reliability of checklist and global rating scores for an 8 scenario assessment with a single rater. Inter-rater reliability was calculated for the 180 scenarios scored by two raters.
Group and Scenario Comparisons
Comparisons were made between ‘Experienced’ (CCM fellows and senior ACNP students, n=26), ‘Intermediate’ (residents, n=8), and ‘Novice’ (novice ACNP students, n=14) Groups. Group performances were described with descriptive statistics. Quartiles were also determined for average global performance and used to compare performance by Group. Significant differences in performance related to scenario and Group were evaluated using repeated measures analysis of variance (RM-ANOVA) in which scenario and Group were the independent variables, and either checklist or global scores were the dependent variables. Post-hoc one-way ANOVA and Scheffe tests were employed to explore significant scenario and Group effects.
Quality of the Simulation Experience
Participants’ responses to the post assessment survey were summarized (mean, SD, min, max).
Results
Scenario Performance
Average checklist and global scores, and discrimination (D) values, for each scenario are presented in Table 2. For most scenarios, discrimination (D) values were >0.30 and were higher for global scores compared to those for checklist scores.
Table 2. Performance on Form A and B Scenarios.
Number of assessments for each scenario (n), Mean Checklist and Global scores (±SD), and Discrimination (D) values.
| Form A Scenarios | Checklist (%) Global (1-9) (Mean±SD) | D* | Form B Scenarios | Checklist Global (Mean±SD) | D* |
|---|---|---|---|---|---|
|
3rd degree HB (n=23) |
69.6±16.8 5.2±1.5 |
0.47 0.55 |
A Fib (n=25) |
70.2±28.0 5.0±2.2 |
0.66 0.53 |
|
ARDS (n=23) |
71.5±18.6 5.41±1.9 |
0.60 0.71 |
PE (n=23) |
65.0±16.3 5.5±1.5 |
0.68 0.24 |
|
Asthma (n=25) |
70.5±15.7 6.0±1.1 |
0.57 0.61 |
Atelectasis (n=25) |
78.70±13.2 6.0±1.3 |
0.47 0.70 |
|
Obstructed Foley (n=23) |
76.3±15.8 7.2±1.2 |
0.02 0.35 |
Hypovolemia (n=25) |
76.2±18.8 6.4±1.0 |
0.32 0.61 |
|
Cardiogenic shock (n=23) |
72±10.9 4.7±1.4 |
0.23 0.65 |
Septic Shock (n=25) |
81.4±10.3 6.0±1.1 |
0.12 0.42 |
|
Cuff Leak (n=25) |
70.2±22.6 6.1±1.9 |
0.40 0.48 |
Mucus Plug (n=23) |
69.1±15.7 5.5±1.6 |
0.60 0.45 |
|
Pulseless Leg (n=25) |
74.7±19.9 5.8±1.9 |
0.63 0.74 |
Crush Injury (n=23) |
84.5±17.3 6.6±2.0 |
0.60 0.55 |
|
Stroke (n=25) |
71.0±19 6.2±1.2 |
0.54 0.34 |
Opioid OD (n=23) |
76.2±14.1 6.0±1.3 |
0.37 0.40 |
D Discrimination-Correlation of Mean Score on Scenario with Overall Mean Performance
Reliability
The reliabilities of the 8-scenario assessments with a single rater were 0.51 and 0.60 for checklist and global scores, respectively, with an estimate of 0.67 and 0.75, respectively, for a full 16-scenario assessment. Based on the double scored scenarios, inter-rater reliabilities were 0.71 and 0.75 for checklist and global scoring, respectively.
Individual Participant Global Scores by Group
The mean global scores for participants in the Experienced, Intermediate and Novice Groups are shown in Figure 1. The Experienced Group had the highest mean global score (6.1±1.6, Mean±SD), and accounted for 83% (10/12) of scores in the top quartile (>6.5) (38.5% of the Group). Notably, all of the senior ACNPs (n=4) scored in the top quartile. Mean global scores were lower in the Intermediate (5.7±1.7) and Novice Groups (5.4±1.5), with only 1 top quartile score from the Intermediate Group (12.5% of the group), and 1 from the Novice Group (7.1% of the group). Conversely, 58% (7/12) of scores in the lower quartile (<5.3) were from the Novice Group (50.0% of that group), compared to 3 from the Intermediate Group (37.5% of that group), and 3 from the Experienced Group (11.5% of that group).
Figure 1. Global Score by Group and Quartile.
Individual participant (n=48) average global scores in the Novice (◆ novice ACNP students, n=14), Intermediate (• resident physicians, n=8) and Experienced Groups (▲ critical care fellows, n=22; ◊ senior ACNP students, n=4). ()
Mean for Groups; (------) upper/lower quartile demarcation lines.
Group and Scenario Comparisons
Mean checklist and global scores for the Experienced, Intermediate, and Novice Groups are shown in Table 3. Based on RM-ANOVA for checklist scores, there was a significant Group by scenario interaction effect (F28,44=1.63, p<0.05), as well as significant Group (F3,44=4.23, p<0.05) and scenario effects (F14,44=1.91, p<0.05). For global scores, the RM-ANOVA did not demonstrate a significant Group x scenario interaction effect (F28,44=1.1, p>0.05) but there were again significant Group (F2,44=4.13, p<0.05) and scenario (F14,44=3.72, p<0.05) effects identified. Post hoc analysis (ANOVAs and Scheffe’s test) indicated significant differences for both checklist and global scores between the Experienced (75.8±15.6, and 6.1±1.6, respectively) and Novice Groups (68.8±21.0 and 5.4±1.5, p<0.05). Based on performance of the Experienced and Novice Groups, the effect sizes (ES) for the checklist and global score differences were 0.42 and 0.46, respectively. Differences in checklist and global scores between the Intermediate Group (75.4±18.3 and 5.7±1.7) and the other two Groups were not significant.
Table 3. Performance by Group.
Mean of average checklist score (% correct) and average global scores by Group (±SD), and range of individual scenario scores.
| Group | Scores | N | Mean | Range |
|---|---|---|---|---|
|
Experienced Group (Fellows/senior ACNP students) |
Checklist (%) Global (1-9) |
26 | 75.6±15.6* 6.1±1.6* |
30.0-100 1.0-9.0 |
|
Intermediate Group (Residents) |
Checklist (%) Global (1-9) |
8 | 75.4±18.3 5.7±1.7 |
33.3-100 2.0-9.0 |
|
Novice Group (Novice ACNP students) |
Checklist (%) Global (1-9) |
14 | 68.8±21.0 5.4±1.5 |
11.1-100 1.0-8.0 |
significantly different from Novice Group by 2-way repeated measures ANOVA and post-hoc Scheffe’s test (see Text).
Quality of the Simulation Experience
The responses to the post-assessment survey questions are shown in Table 4. Average responses indicated that participants ‘strongly agreed’ that the scenarios were (1) representative and (2) relevant to practice, (3) stress provoking in the clinical setting, and (4) would improve their clinical performance (mean 4.7/5 to 4.8/5 ± 0.4-0.6, Table 4). Participants ‘agreed’ on average that scenarios were realistic (mean 4.0/5 ± 0.7), but responses of ‘disagree’ were recorded by some participants.
Table 4. Quality of the Simulation Experience.
Mean response, SD, min and max for participant responses to post-simulation assessment survey (1=strongly disagree; 2=disagree; 3=neutral; 4 agree; 5 strongly agree).
| Variable | Mean | SD | Min | Max |
|---|---|---|---|---|
| The simulated patient and environment were realistic | 4.0 | 0.7 | 2 | 5 |
| The selected simulation scenarios represented events that might occur in practice | 4.8 | 0.4 | 4 | 5 |
| The chosen scenarios were important to practice | 4.8 | 0.4 | 4 | 5 |
| The events included are stress provoking in clinical settings | 4.7 | 0.6 | 3 | 5 |
| Practice scenarios will improve my performance in clinical settings | 4.7 | 0.5 | 3 | 5 |
1=strongly disagree, 2=disagree, 3=neutral, 4=agree, 5=strongly agree
Discussion
This study is the first to use multi-scenario simulation to assess the performance of a diverse group of critical care providers. The results support the psychometric adequacy of the scores derived from the simulation assessment designed to measure cognitive decision-making skills of potential ICU ‘front line’ providers. Based on variance components analysis, ‘holistic’ global scores were more reliable than checklist scores. Using a holistic framework, the raters’ ability to consider how the participant communicated and prioritized their management, something not captured by checklist scoring, likely contributed to the more precise measures of decision making ability. The high discrimination (D) values (≥0.6) for many scenarios indicate that the participant performance on these scenarios more effectively predicted overall performance. The lower D values (<0.3) for a few scenarios suggests that these scenarios may not differentiate along the ability continuum, perhaps because the scenario was too easy (e.g., ’Obstructed Foley’). For the future development of simulation-based assessments, there is a need to re-assess the constructs measured by the individual scenarios and whether the content is truly appropriate for all those being assessed. The individual scenario discriminations, combined with the moderate reliability estimates, highlight the need for multiple, content valid, scenarios to precisely estimate overall critical care ability. Similar to previous simulation-based assessments, we found that the reliability of the assessment scores was most dependent on the number of scenarios, not the number of raters per given scenario (23, 24, 26). Overall, the reliability of the eight scenario assessments was moderate (0.51-0.60), with sufficient precision for determining progress in training and providing formative feedback to front-line provider candidates. The simulation assessments in their present form may also be useful to identify training program gaps, and to potentially identify individual providers who may benefit from additional initial oversight when assuming ICU front line provider roles. A higher reliability estimate (r>=0.80) would be required to make licensing, credentialing or certification decisions (29).
The structured assessment development process for the scenarios, based on actual clinical management problems, provides evidence to support the scoring and generalization inferences in Kane’s validity framework (29). The relationship between participants’ prior ICU experience and performance on the simulation assessment provides additional ‘criterion’ validity evidence, and supports the extrapolation inference in the Kane framework; namely, that the simulation performance scores reflect ability in actual ‘real life’ ICU provider roles (29). In this regard, it was notable that 100% of the senior ACNPs scored in the top quartile for overall global scores, in comparison to 27.3% of critical care fellows, 12.5% of the residents, and 7.1% of the novice ACNP students. This finding is consistent with prior clinical trials which have demonstrated that fully trained ACNPs compare favorably to residents and critical care fellow using objective clinical outcome measures in actual ICU settings (14, 16, 30, 31). While derived from a relatively small sample, this finding strengthens the extrapolation validity argument, and further supports the utilization of ACNPs in these roles (10, 13, 15).
The long term goal of this research is to provide an assessment methodology that transcends the various training pathways of ICU providers. In this regard, there are six separate training and certification pathways for physician intensivists, and equally diverse pathways for ACNPs and PAs, with programs that differ considerably with respect to training and experience. As noted by Kaplan et al., the separate credentialing and certification processes “creates confusion among practitioners and patients, and establishes a barrier to practice and credentialing standardization” (20). The simulation approach presented here provides a standardized method to assess performance. Refinements to the simulation assessment tool (e.g., additional and/or more discriminating scenarios) to decrease measurement error should allow for more precise estimates of ability that could ultimately meet the high psychometric standards required for summative assessments such as those used for certification (29, 32). The addition of some form of criterion-referenced standard setting would be an additional next step to develop a minimum performance standard that could be used to make valid competency decisions about an ICU front-line provider’s decision-making skills (33).
There are limitations of our study. The simulation assessment employed a set of scenarios directed at cognitive skills involved in early recognition and management and did not assess procedural skills or abilities for developing longer term care plans. The simulation setting may also not perfectly reflect the provision of care in ICU settings. The scenarios were purposely shortened to allow for greater sampling of provider performance, and the simulation scenarios were not designed to capture the complex prioritization realities often required to effectively manage patients in the ICU. Finally, it is recognized that individual participant’s performance in an actual clinical environment may be different than that in the simulation environment.
This study adds to a growing body of knowledge that advanced simulation methods may provide a means to assess clinical competence (24, 26). Further refinements should ultimately produce a standardized competency-based assessment tool that could be used to determine whether individuals with a range of training and experience are adequately prepared for ICU provider roles. Ultimately, the goal is to improve patient safety and optimize outcomes of ICU patients by assuring that providers assigned to roles as front line providers in the ICU are adequately prepared.
Supplementary Material
Acknowledgments
This study was conducted in the Washington University Clinical Simulation Center, Washington University School of Medicine, 660 S. Euclid Avenue, St. Louis, MO, 63110
Support: This work was funded by the AHRQ Grant: R18 HS022265-01 Critical Care Management: A Simulation-Based Assessment of Decision-Making Skills.
Copyright form disclosure: Drs. Boyle, Murray, and Woodhouse’s institutions received funding from the Agency for Healthcare Research and Quality (AHRQ). Drs. Boyle and Murray received support for article research from the National Institutes of Health. Dr. Beyatte received funding from the AHRQ.
Footnotes
The remaining authors have disclosed that they do not have any potential conflicts of interest.
References
- 1.Angus DC, Shorr AF, White A, et al. COMPACCS Critical care delivery in the United States: distribution of services and compliance with Leapfrog recommendations. Crit Care Med. 2006;34:1016–1024. doi: 10.1097/01.CCM.0000206105.05626.15. [DOI] [PubMed] [Google Scholar]
- 2.Utilization of Intensive Care Services, 2011. HCUP Statistical Brief #185. available at: http://www.hcup-us.ahrq.gov/reports/statbriefs/sb185-Hospital-Intensive-Care-Units-2011.pdf.
- 3.Shorr AF, Angus DC. Do intensive care unit patients have intensive care unit physicians? Unfortunately not. Crit Care Med. 2006;34:1834–1835. doi: 10.1097/01.CCM.0000219377.55633.50. [DOI] [PubMed] [Google Scholar]
- 4.Halpern NA, Pastores SM, Oropello JM, et al. Critical care medicine in the United States: addressing the intensivist shortage and image of the specialty. Crit Care Med. 2013;41:2754–2761. doi: 10.1097/CCM.0b013e318298a6fb. [DOI] [PubMed] [Google Scholar]
- 5.The Critical Care Workforce: A Study of the Supply and Demand for Critical Care Physicians. Vol. 2017. US Department of Health & Human Services; Washington, DC: 2004. [Google Scholar]
- 6.Angus DC, Kelley MA, Schmitz RJ, et al. Caring for the critically ill patient. Current and projected workforce requirements for care of the critically ill and patients with pulmonary disease: can we meet the requirements of an aging population? JAMA. 2000;284:2762–2770. doi: 10.1001/jama.284.21.2762. [DOI] [PubMed] [Google Scholar]
- 7.Estimating Lives and Dollars Saved from Universal Adoption of the Leapfrog Safety and Quality Standards: 2008 Update. Available at: https://www.researchgate.net/publication/228472000_Estimating_Lives_and_Dollars_Saved_from_Universal_Adoption_of_the_Leapfrog_Safety_and_Quality_Standards_2008_Update]
- 8.Pronovost PJ, Angus DC, Dorman T, et al. Physician staffing patterns and clinical outcomes in critically ill patients: a systematic review. JAMA. 2002;288:2151–2162. doi: 10.1001/jama.288.17.2151. [DOI] [PubMed] [Google Scholar]
- 9.Parikh A, Huang SA, Murthy P, et al. Quality improvement and cost savings after implementation of the Leapfrog intensive care unit physician staffing standard at a community teaching hospital. Crit Care Med. 2012;40:2754–2759. doi: 10.1097/CCM.0b013e31825b26ef. [DOI] [PubMed] [Google Scholar]
- 10.Pastores SM, O’Connor MF, Kleinpell RM, et al. The Accreditation Council for Graduate Medical Education resident duty hour new standards: history, changes, and impact on staffing of intensive care units. Crit Care Med. 2011;39:2540–2549. doi: 10.1097/CCM.0b013e318225776f. [DOI] [PubMed] [Google Scholar]
- 11.Barger LK, Ayas NT, Cade BE, et al. Impact of extended-duration shifts on medical errors, adverse events, and attentional failures. PLoS Med. 2006;3:e487. doi: 10.1371/journal.pmed.0030487. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Landrigan CP, Rothschild JM, Cronin JW, et al. Effect of reducing interns’ work hours on serious medical errors in intensive care units. N Engl J Med. 2004;351:1838–1848. doi: 10.1056/NEJMoa041406. [DOI] [PubMed] [Google Scholar]
- 13.Gershengorn HB, Johnson MP, Factor P. The use of nonphysician providers in adult intensive care units. Am J Respir Crit Care Med. 2012;185:600–605. doi: 10.1164/rccm.201107-1261CP. [DOI] [PubMed] [Google Scholar]
- 14.Kleinpell RM, Ely EW, Grabenkort R. Nurse practitioners and physician assistants in the intensive care unit: an evidence-based review. Crit Care Med. 2008;36:2888–2897. doi: 10.1097/CCM.0b013e318186ba8c. [DOI] [PubMed] [Google Scholar]
- 15.Moote M, Krsek C, Kleinpell R, et al. Physician assistant and nurse practitioner utilization in academic medical centers. Am J Med Qual. 2011;26:452–460. doi: 10.1177/1062860611402984. [DOI] [PubMed] [Google Scholar]
- 16.Scherzer R, Dennis MP, Swan BA, et al. A Comparison of Usage and Outcomes Between Nurse Practitioner and Resident-Staffed Medical ICUs. Crit Care Med. 2017;45:e132–e137. doi: 10.1097/CCM.0000000000002055. [DOI] [PubMed] [Google Scholar]
- 17.McCambridge M, Jones K, Paxton H, et al. Association of health information technology and teleintensivist coverage with decreased mortality and ventilator use in critically ill patients. Archives of Internal Medicine. 2010;170:648–653. doi: 10.1001/archinternmed.2010.74. [DOI] [PubMed] [Google Scholar]
- 18.Lilly CM, Cody S, Zhao H, et al. Hospital mortality, length of stay, and preventable complications among critically ill patients before and after tele-ICU reengineering of critical care processes. JAMA. 2011;305:2175–2183. doi: 10.1001/jama.2011.697. [DOI] [PubMed] [Google Scholar]
- 19.Buckley JD, Addrizzo-Harris DJ, Clay AS, et al. Multisociety task force recommendations of competencies in Pulmonary and Critical Care Medicine. Am J Respir Crit Care Med. 2009;180:290–295. doi: 10.1164/rccm.200904-0521ST. [DOI] [PubMed] [Google Scholar]
- 20.Kaplan LJ, Shaw AD. Standards for education and credentialing in critical care medicine. JAMA. 2011;305:296–297. doi: 10.1001/jama.2010.1997. [DOI] [PubMed] [Google Scholar]
- 21.Laschinger HK. Job and career satisfaction and turnover intentions of newly graduated nurses. J Nurs Manag. 2012;20:472–484. doi: 10.1111/j.1365-2834.2011.01293.x. [DOI] [PubMed] [Google Scholar]
- 22.Wayne DB, Didwania A, Feinglass J, et al. Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest. 2008;133:56–61. doi: 10.1378/chest.07-0131. [DOI] [PubMed] [Google Scholar]
- 23.Lighthall GK, Barr J, Howard SK, et al. Use of a fully simulated intensive care unit environment for critical event management training for internal medicine residents. Crit Care Med. 2003;31:2437–2443. doi: 10.1097/01.CCM.0000089645.94121.42. [DOI] [PubMed] [Google Scholar]
- 24.Murray DJ, Boulet JR, Avidan M, et al. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology. 2007;107:705–713. doi: 10.1097/01.anes.0000286926.01083.9d. [DOI] [PubMed] [Google Scholar]
- 25.Rodriguez-Paz JM, Kennedy M, Salas E, et al. Beyond “see one, do one, teach one”: toward a different training paradigm. Postgrad Med J. 2009;85:244–249. doi: 10.1136/qshc.2007.023903. [DOI] [PubMed] [Google Scholar]
- 26.Boulet JR, Murray DJ. Simulation-based assessment in anesthesiology: requirements for practical implementation. Anesthesiology. 2010;112:1041–1052. doi: 10.1097/ALN.0b013e3181cea265. [DOI] [PubMed] [Google Scholar]
- 27.American Nurses Credentialing Center Information. Available at: http://www.nursecredentialing.org/AcuteCareNP.
- 28.Delphi process: a methodology used for the elicitation of opinions of experts. Available at: https://www.rand.org/content/dam/rand/pubs/papers/2006/P3925.pdf.
- 29.Cook DA, Brydges R, Ginsburg S, et al. A contemporary approach to validity arguments: a practical guide to Kane’s framework. Medical Education. 2015;49:560–575. doi: 10.1111/medu.12678. [DOI] [PubMed] [Google Scholar]
- 30.Hoffman LA, Miller TH, Zullo TG, et al. Comparison of 2 models for managing tracheotomized patients in a subacute medical intensive care unit. Respiratory Care. 2006;51:1230–1236. [PubMed] [Google Scholar]
- 31.Hoffman LA, Tasota FJ, Zullo TG, et al. Outcomes of care managed by an acute care nurse practitioner/attending physician team in a subacute medical intensive care unit. American Journal of Critical Care. 2005;14:121–130. [PubMed] [Google Scholar]
- 32.Peeters MJ, Beltyukova SA, Martin BA. Educational testing and validity of conclusions in the scholarship of teaching and learning. American Journal of Pharmaceutical Education. 2013;77:186. doi: 10.5688/ajpe779186. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Boulet JR, Murray D, Kras J, et al. Setting performance standards for mannequin-based acute-care scenarios: an examinee-centered approach. Journal of the Society for Simulation in Healthcare. 2008;3:72–81. doi: 10.1097/SIH.0b013e31816e39e2. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.