Abstract
Background:
The objective of this study was to investigate whether previous experiences within an anesthesiology-based Objective Structured Clinical Examination (OSCE) assessing communication and professionalism skills was associated with improved performance in a subsequent anesthesiology-based OSCE scenario.
Methods:
This retrospective multi-center study used the performance data of 44 Post Graduate Year 4 clinical anesthesia residents from 3 US anesthesiology residency programs on an OSCE scenario that assessed the residents' effectiveness of discussing anesthesiology-specific treatment options with a high-risk patient. Residents from 2 of the programs had no prior anesthesiology-based OSCE experience. Residents from the third program had previously participated in 4 separate multi-scenario anesthesiology-based OSCE sessions in the 2 years prior to this study. Participating residents completed the same scenario at their respective institutions' simulation center. Ten performances were randomly selected for double rating to assess the interrater reliability of the assessments. Interrater reliability was good for the scenario (intraclass correlation coefficient = 0.66, 95% confidence interval = 0.12–0.90). Performance difference between groups with different OSCE experience status were examined using an independent sample t test, with a Wilcoxon-Mann-Whitney test as a sensitivity analysis.
Results:
Independent sample t test found prior OSCE experience was significantly associated with higher performance scores (t = 2.53, P = .02). The Wilcoxon-Mann-Whitney test result confirmed this finding (z = 3.28, P = .001).
Conclusions:
Findings from this study provide preliminary evidence that anesthesiology-based OSCE experience is associated with improved performance in an OSCE scenario, specifically regarding discussions of treatment options with high-risk patients.
Keywords: Simulation, clinical education, anesthesiology, patient communication, milestone, OSCE
Introduction
Since the mid-1990s, Objective Structured Clinical Examinations (OSCE) have been used to assess medical students for licen-sure.1,2 Recently, the American Board of Anesthesiology (ABA) added an OSCE component to the primary board certification process, the first of its kind for any of the American Board of Medical Specialties' member boards. Designing and implementing an OSCE training program in an anesthesiology residency program to prepare residents for the new exam is time consuming, resource intensive, costly, and logistically challenging. Nonetheless, many anesthesiology residency programs have organized their own OSCE programs to familiarize and prepare their residents for this exam format.3
However, it is not known if an anesthesiology-based OSCE program that gives learners formative and summative assessments leads to improved performance in future anesthesiology-based OSCEs. There is minimal data available regarding the utility of providing OSCE experiences as a method of preparation for a high-stakes OSCE assessment. Thus, demonstrating an OSCE's added value to an anesthesiology residency curriculum is critical for justifying the effort and expenses of implementing such a program.
The objective of this study was to investigate whether practice and experience within an anesthesiology-based OSCE is associated with improved performance in a future anesthesiology-based OSCE experience. Specifically, we investigated resident performances of communication of anesthesiology-based treatment options to a high-risk patient in an OSCE scenario. The results of this investigation will help provide guidance to residency programs in determining the value of designing and implementing an OSCE program intended to prepare residents for a high-stakes OSCE assessment.
Materials and Methods
Design and Sample
This retrospective multi-center study used the performance data of 44 Post-Graduate Year 4 (PGY4) clinical anesthesiology residents from 3 US anesthesiology residency programs on an OSCE scenario. We obtained institutional review board approval with a waiver of documentation of informed consent at all institutions participating in this study. Prior to this study, 2 programs (ie, Programs 1 and 2) had not implemented any anesthesiology-based OSCE experiences for their residents. One program (ie, Program 3) started delivering an OSCE program consisting of 5 communication and professionalism scenarios requiring interaction with a standardized patient or standardized clinician on a biannual basis for all clinical anesthesia residents during the 3 years prior to this study. As a result, Program 3's PGY4s participated in 4 separate anesthesiology-based OSCE sessions involving a total of 20 communication and professionalism scenarios in the 2 years prior to this study. The class size of the 3 programs were 15, 21, and 14 respectively. Three PGY4s from Program 1 and 3 from Program 3 were not able to perform the scenario because of clinical work or other duties. Thus, the final sample consisted of performance data from 44 PGY4s.
Resident performance during simulation-based OSCE scenarios for Accreditation Council for Graduate Medical Education (ACGME) milestone assessments are strongly associated with residents' experience level, time in training, and clinical performance evaluations.4 The OSCE scenario analyzed in this study focused on educating a patient regarding her labor analgesia treatment options. The patient in the scenario was high-risk for developing complications (ie thrombocytopenia and pregnancy-induced hypertension), which was a clinical circumstance that most residents were likely exposed to during their residency training. The residents were expected to assess the patient's labor analgesia needs and subsequently discuss the risks and benefits regarding the treatment options for labor analgesia. See Appendix A for details of the scenario stem, script, and grading sheet. We chose to assess this particular scenario because it allowed for assessment of communication skills, professionalism, and development of a treatment option strategy, based on the foundation of senior resident-level anesthesiology medical knowledge. The scenario required residents to interact with a standardized patient actor, but it did not involve any partial task trainers, technical skills, or interpretation of clinical data. The scenario was created at the experienced institution. The modified Delphi method for developing the scenario, as well as the process of implementing and scoring the scenarios, was previously described.4,5 The modified Delphi process was used to develop the case stem, standardized patient actor script, and the grading sheet that assessed the communication skills, professionalism, and ability to develop treatment options expected of a senior resident. Global rating scores of performances were used in this study because previous studies have shown them to be more reliable than checklist scores.6,7
All of the participating PGY4 residents from the 3 institutions completed the same scenario at their respective institutions' simulation center. Each institution used the same case stem, standardized actor script, and grading sheets. All performances were video recorded for evaluation in a delayed manner.8 Ten videos were randomly selected and double-rated by one faculty from the resident's affiliative program and an independent faculty evaluator from a different institution. Three faculty evaluators served in the independent evaluator role with random assignment of videos. The independent faculty evaluators were blinded to the sampled residents' clinical capabilities, such as daily faculty evaluations from real clinical patient care or medical knowledge, as demonstrated in standardized test scores. All faculty raters received training on scoring for both the standardized binary behavioral checklists (Yes/No demonstration of behaviors/actions) and global rating scale (0–5).4 The faculty raters from the control institutions traveled to the experienced institution to watch the training of the standardized patient actor, observe the actual implementation of the scenario with resident participants, and receive faculty rater training from the study's corresponding investigator. Interrater reliability was assessed with intraclass correlation coefficient. A one-way random-effect model was used given that each video was rated by a different pair of raters from affiliative programs and external independent raters.9 Intraclass correlation coefficient found a single measure of 0.66, 95% CI = 0.12–0.90, which suggested an interrater reliability of good or substantial significance for using a single rater approach in scoring the performance.10,11 Hence, there is a good amount of consistency between the evaluations from the raters who were aware of the residents' prior clinical performances and the independent raters who did not know the residents' prior clinical performances. Therefore, for the data included in the analysis, each resident received a single evaluation from a faculty member from their institution.
Statistical Analysis
Descriptive statistics (ie, means, medians, standard deviations) of background variables and OSCE performance were calculated. The differences between groups with different OSCE experience status in terms of background variables and scenario performance were examined using an independent sample t test, with a Wilcoxon-Mann-Whitney test as a sensitivity analysis. The analyses were performed using SAS 9.4 (SAS Institute Inc., Cary, North Carolina).
Results
Descriptive statistics of the residents' prior exam achievement are summarized in Table 1. When combining the participants into 2 groups based on experience, independent sample t test results found a significant difference between groups on OSCE performance (t = 2.53, P = .02), demonstrating higher performance scores in the experienced group. The Wilcoxon-Mann-Whitney test confirmed the result (z = 3.28, P = .001). See Table 2 for the summary of performance by group. There was no statistically significant difference between the experienced group and the combined comparison group on the In-Training Examination (ITE) 2016 Scaled Scores (t = 1.49, P = .14; z = 1.18, P = .24) or Step 2 United States Medical Licensing Examination (USMLE) scores (t = 1.24, P = .22; z = 0.92, P = .36). Additionally, all participants passed the ABA Basic Exam on their first attempt.
Table 1.
Descriptive Statistics of Background Variables of Resident by Program
| Program | No. | Experiencea | Basic Failb | Step 2c | ITE2016SSd | ||
|---|---|---|---|---|---|---|---|
| Mean (SD) | Median | Mean (SD) | Median | ||||
| 1 | 12 | No | 0 | 238.90 (10.56) | 237.00 | 33.25 (4.69) | 33.50 |
| 2 | 21 | No | 0 | 241.14 (13.24) | 244.00 | 35.48 (4.50) | 37.00 |
| 3 | 11 | Yes | 0 | 246.36 (17.12) | 242.00 | 37.18 (5.49) | 39.00 |
a Experience = Previous experience with anesthesiology specific OSCEs.
b Basic Fail = Number of residents who failed the Basic Exam.
c Step 2 = USMLE Step 2 score.
d ITE2016SS=ITE 2016 scaled score.
Table 2.
Summary of Performance on the Labor Analgesia Treatment Options OSCE Scenario
| Experience | No. | Mean (SD) | Median |
|---|---|---|---|
| No | 33 | 3.15 (0.80) | 3.00 |
| Yes | 11 | 3.91 (1.04) | 4.00 |
| Group Difference (P) | .02a | .001b |
a P value based on independent sample t test.
b Wilcoxon-Mann-Whitney test.
Discussion
Results from this study provide preliminary evidence that prior exposure to anesthesiology-based OSCEs is associated with improved future anesthesiology OSCE performance. The addition of an OSCE component to the ABA's primary certification examination will require anesthesiology residency programs to consider whether or not to dedicate resources towards preparing residents for this exam, which is heavily focused on assessing communication and professionalism skills.
All of the residents who participated in this study also previously participated in the USMLE Step 2 Clinical Skills exam. Although the USMLE Step 2 Clinical Skills exam is an OSCE where students are required to demonstrate a skill, the format and skills assessed in that exam are quite different from the ABA OSCE and of low relevance to the skills required for successful performance in an anesthesiology-based OSCE on communication and professionalism. For instance, the USMLE Step 2 Clinical Skills exam requires students to perform histories and physical exams, develop differential diagnoses, and physically write notes across a spectrum of clinical specialties ranging from psychiatry to pediatrics. Conversely, the ABA OSCE exam focuses on the communication, professionalism, and technical skills specific to the practice of anesthesiology and doesn't require candidates to physically assess patients since all assessment information is presented in case stems.12 Additionally, much of the evaluation of students on the USMLE Step 2 Clinical Skills exam is based upon a written note, whereas the ABA OSCE exam doesn't require the candidate to complete any notes, records, or other forms of documentation. Consequently, the results of this study suggest that the investment of time, funding, and space by anesthesiology residency programs to run an anesthesiology-based OSCE program may indeed be necessary to provide residents with experiential learning opportunities that translate into improved performance during future anesthesiology-based OSCE examinations, such as the ABA OSCE examination.
Beyond preparation for a board certification exam, residency programs are also using OSCEs to provide a unique opportunity to both teach and evaluate residents on the ACGME milestone subcompetencies, especially those that are either rare or hard to observe in daily clinical settings (eg management of ethical dilemmas or disclosing adverse outcomes to patients).3,13 OSCEs have proven to be a feasible and reliable approach to overcome many of the barriers to milestone assessments.4,14 Findings from this study lend further justification for residency programs to continue to dedicate time, funding, and resources towards implementing OSCEs at their residency program.
Despite the study's encouraging findings, there are some limitations. First, this is an observational study based on residents from 3 US residency programs, which may not fully represent the national population. Second, we examined some background variables that could have confounded the relationship of interest. Despite finding no significant between-group difference for any of these variables, other variables may still exist that are associated with a residency program either having or not having a formal OSCE program in the first place, which could subsequently relate to the performance differences. For future studies using a similar design, other clinically relevant covariates, such as clinical evaluations, could be collected. The OSCE scenario in the study assessed communication and professionalism skills, and foundational knowledge of obstetric anesthesiology was necessary for the residents to demonstrate appropriate communication skills. For instance, if a resident was unaware of the risk of providing neuraxial analgesia in a patient with thrombocytopenia, their ability to competently communicate the available options of labor analgesia would have been impaired. There may have also been some differences in the hidden curriculum15,16 between institutions that may have contributed to the differences in performance. For instance, although none of the programs had a formal lecture series on communication and professionalism skills specific to the content areas evaluated in the scenario, certain faculty members in their respective departments may have given informal instruction (eg written or verbal feedback after a clinical assignment) on the communication skills assessed in the scenario. If possible, randomization of residents to the group assignments (eg providing exposure to an OSCE versus no exposure) could be implemented into future study design. Additionally, the interrater reliability evidence of the OSCE scenario was good, especially when considering that the evidence for interrater agreement for humanistic elements is often low.17–19 Nevertheless, if personnel and financial resources permitted, it would be ideal if a consistent group of independent evaluators who were blinded to the residents' identities rated all participants' performances. In this study, when examining the interrater reliability of the sampled performances, we ensured that each selected video was double-rated by 1 external evaluator who was blinded to the resident's identity. Still, work is warranted to improve the assessment tools and increase the number and heterogeneity of samples for the purpose of obtaining more valid inferences of the relationship between prior OSCE experience and performance. Lastly, since the scenario investigated only communication and professionalism skills, the results are not necessarily applicable to OSCE scenarios dealing with technical skills or data interpretation.
In conclusion, findings from this study provide preliminary evidence that prior anesthesiology-based OSCE experience is associated with improved future OSCE performance. These results suggest that providing OSCE experiences to residents in preparation for high-stakes OSCE examinations, such as the ABA OSCE, may lead to higher levels of performance as a result of the experiences.
Appendix
Appendix A. Scenario Stem, Script and Grading Sheet
Scenario Stem
Background/setting: You are one of two daytime residents working on the OB floor and you have been called to labor room 4 by one of the L&D nurses. She tells you that her patient, who was admitted 1 hour ago, needs an epidural because she is not coping well with her contraction pain. Your co-resident obtained a history from the patient, but was called away for a STAT cesarean section prior to completing the consent
OSCE scenario objective: Your task is to evaluate the patient and propose a treatment plan.
HPI: 28yo F, G1P0 at 39 weeks EGA who presented to L&D in spontaneous labor. Spontaneous rupture of membranes occurred 2 hours ago. Her pregnancy has been complicated by gestational hypertension, thrombocytopenia, and incomplete prenatal care.
PMHx: Gestational Hypertension
PSHx: none
Allergies: NKDA
Meds: prenatal vitamins
Social: (-) smoker, (-) drinker, (-) other drugs.
NPO: had breakfast 3 hours ago
Vitals
BP: 141/82
HR: 88
SPO2: 99%
PE
General: AAOx3, gravid
Airway: Mallampati 1, 4 finger-breadths thyromental distance, no loose teeth or hardware
Lungs: clear bilaterally
Heart: normal rate, normal rhythm, no murmurs
Neuro: moving all extremities, normal deep tendon reflexes
Extremities: trace edema b/l lower
Labs
@ 20 wks EGA
WBC 9.3
Hgb 11.2 g/dL
Hct 36.3 %
Plt 81
@ 38 wks EGA
Strep B Screen Negative
Scenario Script
Setting: L&D
Laboring woman on a stretcher having painful contractions every 3 minutes.
1st contraction 30 seconds after the resident walks in and lasts 10–15 seconds.
2nd contraction halfway through the conversation
3rd contraction at the end of the conversation.
Patient in gown with pillow to give effect of pregnant patient
Chair beside bed for clinician use.
| State | Resident | Actor Role = Laboring Patient |
|---|---|---|
| Initial Interaction | Introduction: “Hello, I'm Dr. X, an anesthesiology resident physician. Your nurse called me and said you want an epidural. How are you doing?” | “Yes, I need an epidural right now! What do I need to do to get this done as soon as possible?” |
| Response 1 | May explain the need to ask more about the past medical history | Answer any past medical history questions based on preop sheet If the resident doesn't ask about bleeding issues or platelet counts state, “I'm healthy, but I know I had one low blood count early in my pregnancy.” Then restate extreme interest in immediate pain control. |
| Response 2 | Should explain the need to obtain additional blood tests prior to epidural placement. (Specifically platelet count and coags). | “Why do I need these tests?” |
| Response 3 | Should explain the concern for bleeding and risk of injury with epidural/spinal hematoma. | If resident mentions risk of bleeding or hematoma formation but doesn't give a thorough explanation ask, “What happens if I bleed into my spine?” Accept that labs need to be done prior to epidural. “What are my other options right now though? I can't wait that long!” |
| Response 4 | Resident may offer any of the following pain relief options while the labs are being run: IV narcotics (ie, fentanyl, morphine, Stadol) Inhaled Nitrous Oxide Doula Services Reassurance (ie, “you're going to be ok”, or “it won't take that long”) |
If offered narcotics state, “Will the narcotics hurt my baby?” If offered nitrous oxide state, “What affects will the nitrous have on my baby?” If offered a Doula state, “I am not interested in that. I just want pain relief. If only reassurance is given state, “This baby is killing me. I need something for the pain.” |
| Response 5 | Resident should explain effects of:
|
|
| Resolution | “As soon as we know it is safe to do your epidural I'll come back and get started.” | If narcotics were offered, “Ok, I'll try the narcotics (fentanyl or morphine) while I wait for my blood work.” If narcotics not offered and nitrous oxide is offered then state “Ok, I'll try the nitrous while I wait for my blood work” If offered both narcotics and nitrous state, ““Ok, I'll try the nitrous while I wait for my blood work” |
Scenario Grading Sheet
| Scenario Objectives | Milestone Competency | Milestone Level | Milestone Rubric | Specific Action or Behavior | Achieved (Y/N) | Evaluation Comments |
|---|---|---|---|---|---|---|
| 1. Assess the need for labor analgesia 2. Notes the increased risk for bleeding and the need for laboratory analysis prior to epidural placement 3. Offers alternatives to an epidural for labor analgesia and discusses the risks of these alternatives. |
Milestone: PBLI4: Education of patient, families, students, residents, and other health professionals ABA Content Outline Skill Area: Treatment Options |
Entry | Discusses medical plans and responds to questions from patients and their families. | 1. Resident introduces him/herself as an anesthesiology resident physician. 2. Resident confirms with patient if she is interested in pain management or acknowledges that patient is interested. 3. Resident offers at least one option for labor analgesia. |
1. 2. 3. |
|
| Junior | Explains anesthetic care to patients and their families | 1. States that an epidural cannot be safely performed prior to rechecking the platelet count ONLY. 2. Offers IV narcotics OR inhaled nitrous oxide as an alternate labor analgesic |
1. 2. |
|||
| Mid | Effectively explains subspecialty anesthetic care to patients and their families | 1. States that epidural cannot be done prior to checking platelet count AND coags (PT, aPTT, INR, +/− Fibrinogen) 2. Offers IV narcotics AND inhaled nitrous oxide (although not concurrently) as alternate labor analgesics while patient waits for labs to be completed |
1. 2. |
|||
| Senior | Explains anesthesia care and risk to patients and their families with conditional independence | 1. Explains increased risk of neuraxial hematoma with low platelet count. 2. Has a discussion with regarding the concern for ITP, HELLP, or Preeclampsia and the need for coags. 3. Reassures patient that narcotics and nitrous oxide are safe labor analgesics |
1. 2. 3. |
|||
| Adv. | Serves as an expert on anesthesiology to patients, their families, and other health care professionals | 1. Resident has an in-depth discussion of risk of narcotics and/or nitrous oxide. (Example: Narcotics - decreased FHR variability, possible slowed respiratory rate, somnolence if given immediately before delivery. Nitrous has > 100 years of safe labor analgesic use. No known adverse neonatal outcomes. Unknown risk of cognitive deficits, but very short low dose exposure so probably no impact. | 1. | |||
| Global Rating Score | out of 5 |
Footnotes
Previous presentations: Part of this paper was presented in the 2018 Spring meeting of Society for Education in Anesthesia, Louisville, Kentucky.
Funding/Support: None.
Declarations of interest: None.
References
- 1.Whelan GP. Educational Commission for Foreign Medical Graduates: clinical skills assessment prototype. Med Teach. 1999;21(2):156–60. doi: 10.1080/01421599979789. [DOI] [PubMed] [Google Scholar]
- 2.Papadakis MA. The Step 2 clinical-skills examination. N Engl J Med. 2004;350(17):1703–5. doi: 10.1056/NEJMp038246. [DOI] [PubMed] [Google Scholar]
- 3.Isaak RS, Chen F, Arora H et al. A Descriptive Survey of Anesthesiology Residency Simulation Programs: How Are Programs Preparing Residents for the New American Board of Anesthesiology APPLIED Certification Examination? Anesth Analg. 2017;125(3):991–8. doi: 10.1213/ANE.0000000000002189. [DOI] [PubMed] [Google Scholar]
- 4.Isaak RS, Chen F, Martinelli SM et al. Validity of Simulation-Based Assessment for Accreditation Council for Graduate Medical Education Milestone Achievement. Simul Healthc. 2018;13(3):201–210. doi: 10.1097/SIH.0000000000000285. [DOI] [PubMed] [Google Scholar]
- 5.Isaak R, Chen F, Hobbs G et al. Standardized mixed-fidelity simulation for ACGME milestones competency assessment and objective structured clinical exam preparation. Med Sci Educ. 2016;26(3):437–41. [Google Scholar]
- 6.Cohen DS, Colliver JA, Robbs RS, Swartz MH. A large-scale study of the reliabilities of checklist scores and ratings of interpersonal and communication skills evaluated on a standardized-patient examination. Adv Health Sci Educ Theory Pract. 1996;1(3):209–13. doi: 10.1007/BF00162917. [DOI] [PubMed] [Google Scholar]
- 7.Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73(9):993–7. doi: 10.1097/00001888-199809000-00020. [DOI] [PubMed] [Google Scholar]
- 8.Isaak R, Stiegler M, Hobbs G et al. Comparing Real-time Versus Delayed Video Assessments for Evaluating ACGME Sub-competency Milestones in Simulated Patient Care Environments. Cureus. 2018;10(3):e2267. doi: 10.7759/cureus.2267. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155–63. doi: 10.1016/j.jcm.2016.02.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Cicchetti DV. Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology. Psychol Assess. 1994;6(4):284–90. [Google Scholar]
- 11.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. [PubMed] [Google Scholar]
- 12.The American Board of Anesthesiology OSCE Example Scenarios. http://www.theaba.org/PDFs/APPLIED-Exam/OSCE-Exemplar-Scenarios Accessed July 31, 2019.
- 13.Rebel A, Hester DL, DiLorenzo A, McEvoy MD, Schell RM. Beyond the “E” in OSCE. Anesth Analg. 2018;127(4):1092–6. doi: 10.1213/ANE.0000000000003317. [DOI] [PubMed] [Google Scholar]
- 14.Comert M, Zill JM, Christalle E et al. Assessing Communication Skills of Medical Students in Objective Structured Clinical Examinations (OSCE)--A Systematic Review of Rating Scales. PLoS One. 2016;11(3) doi: 10.1371/journal.pone.0152717. e0152717. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Stergiopoulos E, Fernando O, Martimianakis MA. “Being on Both Sides”: Canadian Medical Students' Experiences With Disability, the Hidden Curriculum, and Professional Identity Construction. Acad Med. 2018;93(10):1550–9. doi: 10.1097/ACM.0000000000002300. [DOI] [PubMed] [Google Scholar]
- 16.Russ-Sellers R, Blackwell TH. Emergency Medical Technician Training During Medical School: Benefits for the Hidden Curriculum. Acad Med. 2017;92(7):958–60. doi: 10.1097/ACM.0000000000001579. [DOI] [PubMed] [Google Scholar]
- 17.Arnold L. Assessing professional behavior: yesterday, today, and tomorrow. Acad Med. 2002;77(6):502–15. doi: 10.1097/00001888-200206000-00006. [DOI] [PubMed] [Google Scholar]
- 18.Wilkinson TJ, Wade WB, Knock LD. A blueprint to assess professionalism: results of a systematic review. Acad Med. 2009;84(5):551–8. doi: 10.1097/ACM.0b013e31819fbaa2. [DOI] [PubMed] [Google Scholar]
- 19.Johnson D, Cujec B. Comparison of self, nurse, and physician assessment of residents rotating through an intensive care unit. Crit Care Med. 1998;26(11):1811–6. doi: 10.1097/00003246-199811000-00020. [DOI] [PubMed] [Google Scholar]