Abstract
Background
Simulation offers a means to assess resident competence in communication, but pediatric standardized patient simulation has limitations. A novel educational technology, avatar patients (APs), holds promise, but its acceptability to residents, educational relevance, and perception of realism have not been determined.
Objective
To determine if APs are acceptable, provide a relevant educational experience, and are realistic for teaching and assessment of a complex communication topic.
Methods
Pediatric residents at one academic institution participated in an AP experience from 2019 to 2021 consisting of 2 scenarios representing issues of medical ambiguity. After the experience, residents completed a survey on the emotional relevance, realism, and acceptability of the technology for assessment of their communication competence.
Results
AP actor training required approximately 3 hours. Software and training was provided free of charge. Actors were paid $30/hour; the total estimated curricular cost is $50,000. Sixty-five of 89 (73%) pediatric residents participated in the AP experience; 61 (93.8%) completed the survey. Forty-eight (78.7%) were emotionally invested in the scenarios. The most cited emotions evoked were anxiety, uncertainty, concern, and empathy. The conversations were rated by 49 (80.3%) as realistic. APs were rated as beneficial for learning to communicate about medical ambiguity by 40 (65.5%), and 41 (66.7%) felt comfortable having APs used to assess their competence in this area.
Conclusions
Pediatric residents were emotionally invested in the AP experience and found it to be realistic. The experience was rated as beneficial for learning and acceptable to be used for assessment of how to communicate medical ambiguity.
Introduction
All residency programs are required to report resident competence to the Accreditation Council for Graduate Medical Education (ACGME) through the Milestone Project.1,2 Certain Pediatric Milestones, such as those addressing challenging communication skills like ambiguity and shared decision-making in patient care, are difficult to assess.3 Medical ambiguity has been defined as a lack of clarity in the clinical decision-making process.4 Pediatric residency program directors and simulation experts identified communication of medical ambiguity as appropriate for simulation-based assessment.3 Assessing these discussions is challenging in the clinical setting as they may not occur during faculty observation.5 First-year internal medicine residents indicated that only 5% of their first patient experiences delivering bad news occurred with faculty observation.6 When faculty are present, they are more likely to take the lead in challenging conversations rather than mentoring the trainee to do so.5
Although standardized patients (SPs) are an effective educational and assessment strategy for improving communication skills,7-11 recruiting pediatric SPs may be problematic for difficult medical discussions because of ethical and developmental considerations and the limited availability of child actors.12 When children have served as SPs, the focus has been on clinical reasoning rather than communication, and feedback from these SPs on their emotional experience was mixed.13 Realistic communication encounters in pediatrics are challenging to coordinate when multiple SPs are needed to represent both caregivers and their children.
To circumvent these challenges, alternative simulation modalities may be preferable. Clinicians can interact with computerized “virtual patients” via software branching algorithms. Virtual patients have been used in health care teaching14-20 and assessment18,19,21,22 of clinical reasoning and shared decision-making,23 but have more limited use in assessing communication since computer algorithms do not yet allow for the nuanced conversations between live people.
Avatar patients (APs) are realistic animated representations of people who are voiced and controlled by live, trained actors. APs have taught communication skills to nursing students24,25 and pediatric residents to recognize respiratory distress in babies.26 There are little data on the use of APs for teaching or assessing resident communication skills. One study used prerecorded AP responses that were perceived as realistic and beneficial for learning.27 For complex discussions, such as communicating medical ambiguity and initiating shared decision-making, APs with prerecorded answers are unlikely to reproduce realistic encounters.
We developed a curriculum using APs to assess resident competency in communicating medical ambiguity. We sought to determine if APs are acceptable, provide a meaningful educational experience, and are realistic for teaching and assessment of a complex communication topic.
Methods
Setting
We conducted the study from 2019 to 2021 at a medium-sized pediatric residency program in a large Northeastern US city. We recruited participants from the categorical pediatric, preliminary child neurology, child psychiatry, and combined medicine-pediatric residencies over 2 academic years.
Objectives
Are avatar patients an acceptable, meaningful, and realistic educational technology for teaching and assessing resident competency in communicating medical ambiguity?
Findings
Most residents found the experience to be realistic and a worthwhile educational experience as well as acceptable to be used in teaching and assessing their competence in communicating medical ambiguity.
Limitations
The study was conducted at one center in one clinical field so generalizability is unclear. Further, the intervention is expensive and therefore potentially not accessible to every training program.
Bottom Line
Avatar patients are an innovative educational technology for teaching and assessing resident competence in communication, and further studies are needed to determine the effects of using avatar patients to improve and measure resident competence.
Educational Technology
We partnered with Mursion, an educational technology company that developed screen-based APs voiced by live actors in the roles of children and adults. These APs interact with trainees through verbal and nonverbal communication during a simulated scenario. Encounters required a computer with Zoom, camera, microphone, and speaker for the participants and AP actors to communicate. Mursion provided free access to their software and actors in exchange for the ability to use the study scenarios with other clients. The actor was paid $30/hour through grant funding. Mursion was not involved in study development, analysis, or manuscript writing. The standard Mursion start-up cost for the first 500 hours of simulation is $50,000 with additional sessions at $140/hour.
Scenario Design
The authors designed 2 commonly encountered pediatric scenarios (Figure) featuring ambiguous medical situations without clear-cut answers. In scenario 1, the objective was to discuss the decision to perform a lumbar puncture on a febrile 30-day-old infant in the emergency department (online supplementary data). In scenario 2, the objective was to discuss next steps for a teenager expressing suicidal ideation in the outpatient setting (online supplementary data). Shared decision-making was crucial to both scenarios. The scenarios were piloted with trained APs by the study leader (A.F.V.) playing the participant role and either achieving or missing the predefined desired behaviors for each case; see the cases (online supplementary data) for behavior definitions. One actor responded to the participant in real time using voice modulating technology to play the roles of 2 people in each scenario.
Figure.
View of What Participants See While Interacting With the Avatar Patients in the Clinical Environment
Note: On the top is the scenario 30-day-old who has a fever with the parents pictured, and on the bottom is the scenario of the teen who is expressing suicidal ideation, shown with her mother.
Intervention
All residents (n=89) in the categorical pediatric, preliminary child neurology, child psychiatry, and combined medicine-pediatric residency programs were scheduled for both AP scenarios, followed by an acceptability survey as part of their required curriculum. No residents opted out of study inclusion. In both scenarios, all relevant medical guidelines were provided immediately beforehand so that the focus would not be participant medical knowledge. Participants met their APs on Zoom, where the actor introduced the scenarios. Total intervention time was 45 minutes. After each scenario, participants participated in a brief debriefing session with the actor and completed an acceptability survey.
Measurement
We developed a 13-question survey on demographics, perceived realism of the APs, emotions evoked by APs, and the educational value of APs (provided as online supplementary data). To develop this, we conducted a literature review to identify and incorporate previously developed survey questions that were applicable to our study. A survey developed by Mursion28 for teachers using their technology to simulate teaching experiences was adapted for question 4. A survey comparing different educational technologies was used for question 3, and a survey designed for medical students to compare SPs to manikin-based simulation was used for question 7.29 The latter 2 have validity evidence supporting its use in this capacity.30 Twelve questions were answered on scales. One free text question was included in which respondents were asked to give 1- to 2-word answers. Introductory information, question order, survey layout, and wording were designed to maximize response rate while decreasing measurement error using internet survey design best practices.31 The final acceptability survey was reviewed by the entire team and distributed using Qualtrics software.
Data Analysis
Descriptive analysis was conducted for the quantitative survey questions using SPSS Statistics, version 24 (IBM Corp). One free-text survey question asked for 1- to 2-word responses describing emotions raised by the scenarios. Two study authors (A.F.V., L.M.) independently coded the responses by putting like responses into categories and counting how many times each category was mentioned. Minor coding differences were reconciled.
This study was approved by the Institutional Review Boards at Mass General Brigham, Weill Cornell Medicine, and Maine Medical Center.
Results
Study Participants
Sixty-five out of 89 (73%) residents from the categorical pediatric, preliminary child neurology, child psychiatry, and combined medicine-pediatric residencies completed the experience. Sixty-five residents were eligible in 2019-2020 and an additional 24 residents were eligible in 2020-2021. Reasons for non-participation included session cancellation during the COVID-19 pandemic, being called into work unexpectedly, Zoom challenges, personal emergency, or graduation. Sixty-one of 65 (93.8%) residents who completed the AP experience completed the survey. Table 1 describes resident respondent characteristics including postgraduate year and program type.
Table 1.
Respondent Postgraduate Year (PGY) and Training Program
| Current PGY (N=59) | n (%) |
| PGY-1 | 33 (55.9) |
| PGY-2 | 11 (18.6) |
| PGY-3 | 15 (25.4) |
| PGY-4 | 0 (0) |
| Current Training Program (N=60) | n (%) |
| Preliminary | 6 (10) |
| Categorial | 37 (61.7) |
| Medicine-pediatrics | 9 (15) |
| Pediatrics-neurology | 5 (8.3) |
| Pediatrics-psychiatry | 3 (5) |
Realism and Emotional Investment
A majority of respondents (78.7%, 48 of 61) reported being emotionally “invested” or “extremely invested” on a 5-point scale of extremely uninvested to extremely invested (Table 2). When asked what emotions were evoked by the scenario, anxiety, uncertainty, concern, empathy, sadness, stress, and fear were the most often cited emotions (Table 3). Overall, the majority of respondents rated the AP environment and the conversations in the scenarios as realistic (Table 2).
Table 2.
Realism of Avatar Patient Experience (N=61)
| Question | Extremely Invested, n (%) | Invested, n (%) | Neutral, n (%) | Uninvested, n (%) | Extremely Uninvested, n (%) |
| To what extent did you feel emotionally invested in the scenario? | 8 (13.1) | 40 (65.6) | 10 (16.4) | 2 (3.3) | 1 (1.6) |
| Question | Very Realistic, n (%) | Realistic, n (%) | Neutral, n (%) | Unrealistic, n (%) | Very Unrealistic, n (%) |
| How real did the avatar patient clinical environment feel (in other words, how easily were you able to suspend disbelief)? | 7 (11.5) | 30 (49.2) | 17 (27.9) | 5 (8.2) | 2 (3.3) |
| How real did the conversation with the avatar patient and family members feel (in other words, how easily were you able to suspend disbelief)? | 18 (29.5) | 31 (50.8) | 10 (16.4) | 1 (1.6) | 1 (1.6) |
| How real did the scenario of a fever in a baby feel? | 12 (19.7) | 41 (67.2) | 7 (11.5) | 1 (1.6) | — |
| How real did the scenario of suicidal ideation in a teenager feel? | 17 (27.9) | 30 (49.2) | 9 (14.8) | 3 (4.9) | 2 (3.3) |
Table 3.
Free Text Responses Regarding Emotions the Avatar Patient Scenarios Raised (N=44)
| Emotion | Count |
| Anxiety | 14 |
| Uncertainty | 9 |
| Concern | 8 |
| Empathy | 8 |
| Sadness | 6 |
| Stress | 6 |
| Fear | 5 |
| Discomfort | 4 |
| Fear | 3 |
| Felt realistic | 3 |
| Felt unrealistic | 1 |
| Hesitance | 1 |
| Less anxiety than with standardized patients | 1 |
| Impatience | 1 |
| Angst | 1 |
| Dread | 1 |
| Confidence | 1 |
| Appreciation for being trusted | 1 |
| None | 1 |
Educational Value of the AP Experience
APs were rated as “beneficial” or “very beneficial” by the majority of respondents for learning to communicate with patients about medically ambiguous situations and increasing their confidence in these discussions. Most would recommend the use of APs and would be comfortable with faculty using AP experiences to assess them for a complex communication milestone (Table 4).
Table 4.
Benefits of Avatar Patients
| Question | Very Beneficial, n (%) | Beneficial, n (%) | Somewhat Beneficial, n (%) | Not at All Beneficial, n (%) | Never Used This Modality, n (%) |
| How beneficial was participation in the avatar patient scenarios to your learning about communicating with patients about ambiguous medical situations? (N=61) | 16 (26.2) | 24 (39.3) | 16 (26.2) | 5 (8.2) | 0 (0) |
| Question | Definitely, n (%) | Probably, n (%) | Neutral, n (%) | Probably Not, n (%) | Definitely Not, n (%) |
| Did this avatar patient experience increase your confidence in talking to families and patients about ambiguous situations? (N=61) | 6 (9.8) | 26 (42.6) | 17 (27.9) | 11 (18) | 1 (1.6) |
| Would you recommend the use of avatar patients to other physicians to help with communication skill development? (N=61) | 16 (26.2) | 25 (41) | 14 (23) | 4 (6.6) | 2 (3.3) |
| Would you use avatar patients again to practice your communication skills? (N=61) | 12 (19.7) | 25 (41) | 13 (21.3) | 9 (14.8) | 2 (3.3) |
| Question | Extremely Comfortable, n (%) | Comfortable, n (%) | Neutral, n (%) | Uncomfortable, n (%) | Very Uncomfortable, n (%) |
| How comfortable would you, as a resident, feel having your residency program use avatar patient scenarios like the ones you just completed to help place you on the following milestone: “Recognize that ambiguity is part of clinical medicine and recognize the need for and utilize appropriate resources in dealing with uncertainty” (Professionalism 6)2 (N=60) | 9 (15) | 31 (51.7) | 12 (19.7) | 4 (6.7) | 4 (6.7) |
Discussion
Most pediatric residents felt emotionally invested in conversations using AP technology. The majority of residents indicated their acceptance of the AP format, suggesting that APs may be a viable technology to improve complex communication skills in formative training and summative assessment.
Our findings underscore the impact of APs in creating emotional investment and conversational realism in simulated clinical scenarios. Nursing students24,25 and pediatric residents27,32 find APs to be realistic and helpful for teaching communication skills. Few studies have used VPs or APs voiced by a live person rather than branching narrative algorithms or natural language processing.27,32,33 Without a live person, such technology deprives learners of the authenticity that comes with reciprocal conversation and nonverbal communication skills, particularly for nuanced topics. Unlike other studies where medical personnel voice VP patients,34-36 we used lay person actors to increase the realism of these conversations. Importantly, using live actors on Zoom permitted the APs to see and respond to resident nonverbal facial cues. We believe these helped increase the realism and emotional investment residents felt. Most residents found APs helpful for improving confidence in communicating ambiguity and engaging in shared decision-making. Moreover, most would repeat the AP experience for their learning, be comfortable (or at least neutral) with its use for assessment, and would recommend it for difficult conversations.
In terms of feasibility, the financial and resource costs to using a software platform for AP encounters such as Mursion include purchasing a minimum of 500 hours of simulation for $35,000, plus approximately $15,000 for actor training in the first year. Each 1-hour simulation session beyond the first 500 hours costs $140. The total cost of our project would have been $50,000 even though we only used 65 of the 500 purchased hours. On the other hand, when SPs are employed in communication simulations, the costs may be considerable when multiple SPs are needed for a cast of characters. Training one SP and then conducting 2 cases for 65 residents is estimated to cost from $2,000 to $10,000.37-39 That cost would approximately double to portray a parent and a child together. Neither includes travel costs for the SPs if remote Zoom technology is not used. In contrast, a single adult actor using voice modification can portray multiple APs. In cases involving children, communicating with both child and parent is often imperative to demonstrating a communication skill. Recruiting child SPs can be challenging, and the use of minors raises important ethical considerations.12,40 Training APs and SPs is resource intensive. In our study, we trained the actor on both cases over 1 hour, and then piloted both cases with performance feedback over 2 hours.
Our study has limitations. The study was conducted at one site in one clinical field and findings may not be generalizable. The actor training was brief and may have been insufficient for the broad range of resident behaviors displayed. This may have led to the better actor responses over time and the experience being different for residents who participated later in the study. Residents in this study only had a single exposure to APs. Comfort and perceived value with APs could increase with further exposure. While our survey used many of the best practices in survey design,31 we did not perform cognitive interviewing prior to implementation. Some survey questions may be perceived as leading residents to answer affirmatively, such as asking “how” rather than “whether” the technology was beneficial. We asked residents to respond in a few words to describe their emotions during the AP experience, making a more nuanced qualitative analysis impossible. We asked for short responses to increase response rate on the resident survey. Further, it is not clear if the emotions raised related to the communication content or the AP technology.
Our next step is to assess resident performance on the AP experience. We hope to develop a tool that allows for assessment of competency in coping with ambiguity.
Conclusions
We evaluated realism, acceptability, and educational value of APs to pediatric residents for communication involving medical ambiguity. Residents were emotionally invested and found the AP modality to be realistic and beneficial for their learning and would repeat it and recommend it to others. Importantly, they agreed with its use to assess competency.
Supplementary Material
Acknowledgments
The authors wish to thank Katherine Sparger, MD, for her help in writing the suicidal ideation scenario, and the MGfC residents for their participation in this study.
Funding Statement
Funding: This study was funded by a Massachusetts General Hospital Department of Medicine Center for Education and Innovation Scholarship Grant. Mursion Inc and Massachusetts General Hospital entered into a research donation agreement; Mursion supplied the avatar patient technology and platform free of charge in exchange for the rights to use the simulation scenarios created for this study with their other clients. No money was exchanged between Mursion and the study team or institution. Mursion supplied the avatar patient actors who were paid by the study team using the grant funding received from Massachusetts General Hospital. The PI for this grant was Ariel Frey-Vogel, MD, MAT. The funder and Mursion had no role in study design, data analysis, collection, or interpretation, in the writing of the manuscript, or in the decision to submit the manuscript for publication.
Footnotes
Conflict of interest: The authors declare they have no competing interests.
This work was previously presented at the virtual Mass General for Children's Medical Education Summit, October 2021.
References
- 1.Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system—rationale and benefits. N Engl J Med . 2012;366(11):1051–1056. doi: 10.1056/NEJMsr1200117. [DOI] [PubMed] [Google Scholar]
- 2.American Board of Pediatrics, Accreditation Council for Graduate Medical Education. The Pediatrics Milstone Project. Accessed September 2, 2021. https://www.abp.org/sites/abp/files/pdf/milestones.pdf.
- 3.Mallory LA, Calaman S. Lee White M, et al., editors. Targeting simulation-based assessment for the pediatric milestones: a survey of simulation experts and program directors. Acad Pediatr . 2016;16(3):290–297. doi: 10.1016/j.acap.2015.09.007. [DOI] [PubMed] [Google Scholar]
- 4.Dogra N, Giordano J, France N. Cultural diversity teaching and issues of uncertainty: the findings of a qualitative study. BMC Med Educ . 2007;7(1):8. doi: 10.1186/1472-6920-7-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Collins K, Hopkins A, Shilkofski NA, Levine RB, Hernandez RG. Difficult patient encounters: assessing pediatric residents' communication skills training needs. Cureus . 2018;10(9):e3340. doi: 10.7759/cureus.3340. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Orlander JD, Fincke BG, Hermanns D, Johnson GA. Medical residents' first clearly remembered experiences of giving bad news. J Gen Intern Med . 2002;17(11):825–840. doi: 10.1046/j.1525-1497.2002.10915.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Falcone JL, Claxton RN, Marshall GT. Communication skills training in surgical residency: a needs assessment and metacognition analysis of a difficult conversation Objective Structured Clinical Examination. J Surg Educ . 2014;71(3):309–315. doi: 10.1016/j.jsurg.2013.09.020. [DOI] [PubMed] [Google Scholar]
- 8.Porcerelli JH, Brennan S, Carty J, Ziadni M, Markova T. Resident ratings of communication skills using the Kalamazoo adapted checklist. J Grad Med Educ . 2015;7(3):458–461. doi: 10.4300/JGME-D-14-00422.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Gee DW, Phitayakorn R, Khatri A, Butler K, Mullen JT, Petrusa ER. A pilot study to gauge effectiveness of standardized patient scenarios in assessing general surgery milestones. J Surg Educ . 2016;73(6):e1–e8. doi: 10.1016/j.jsurg.2016.08.012. [DOI] [PubMed] [Google Scholar]
- 10.Short MW, Jorgensen JE, Edwards JA, Blankenship RB, Roth BJ. Assessing intern core competencies with an Objective Structured Clinical Examination. J Grad Med Educ . 2009;1(1):30–36. doi: 10.4300/01.01.0006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Solomon AB, Reed R, Benkov K, et al. Using the Objective Structured Clinical Examination to assess ACGME competencies in pediatric gastroenterology fellows. J Pediatr Gastroenterol Nutr . 2017;64(4):e92–e95. doi: 10.1097/MPG.0000000000001450. [DOI] [PubMed] [Google Scholar]
- 12.Khoo EJ, Schremmer RD, Diekema DS, Lantos JD. Ethical concerns when minors act as standardized patients. Pediatrics . 2017;139(3):e20162795. doi: 10.1542/peds.2016-2795. [DOI] [PubMed] [Google Scholar]
- 13.Lane JL, Ziv A, Boulet JR. A pediatric clinical skills assessment using children as standardized patients. Arch Pediatr Adolesc Med . 1999;153(6):637–644. doi: 10.1001/archpedi.153.6.637. [DOI] [PubMed] [Google Scholar]
- 14.Forsberg E, Ziegert K, Hult H, Fors U. Assessing progression of clinical reasoning through virtual patients: an exploratory study. Nurse Educ Pract . 2016;16(1):97–103. doi: 10.1016/j.nepr.2015.09.006. [DOI] [PubMed] [Google Scholar]
- 15.Botezatu M, Hult H, Tessma MK, Fors U. Virtual patient simulation: knowledge gain or knowledge loss. Med Teach . 2010;32(7):562–568. doi: 10.3109/01421590903514630. [DOI] [PubMed] [Google Scholar]
- 16.Marei HF, Donkers J, Van Merrienboer JJG. The effectiveness of integration of virtual patients in a collaborative learning activity. Med Teach . 2018;40(suppl 1):96–103. doi: 10.1080/0142159X.2018.1465534. [DOI] [PubMed] [Google Scholar]
- 17.Wilkening GL, Gannon JM, Ross C, et al. Evaluation of branched-narrative virtual patients for interprofessional education of psychiatry residents. Acad Psychiatry . 2017;41(1):71–75. doi: 10.1007/s40596-016-0531-1. [DOI] [PubMed] [Google Scholar]
- 18.Close A, Goldberg A, Helenowski I, Schuller M, DaRosa D, Fryer J. Beta test of web-based virtual patient decision-making exercises for residents demonstrates discriminant validity and learning. J Surg Educ . 2015;72(6):e130–e136. doi: 10.1016/j.jsurg.2015.10.011. [DOI] [PubMed] [Google Scholar]
- 19.Hege I, Kononowicz AA, Kiesewetter J, Foster-Johnson L. Uncovering the relation between clinical reasoning and diagnostic accuracy—an analysis of learner's clinical reasoning processes in virtual patients. PLoS One . 2018;13(10):e0204900. doi: 10.1371/journal.pone.0204900. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Watari T, Tokuda Y, Owada M, Onigata K. The utility of virtual patient simulations for clinical reasoning education. Int J Environ Res Public Health . 2020;17(15):5325. doi: 10.3390/ijerph17155325. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Kenny PG, Parsons TD, Rizzo A. A comparative analysis between experts and novices interacting with a virtual patient with PTSD. Stud Health Technol Inform . 2009;144:122–124. doi: 10.3389/conf.neuro.14.2009.06.049. [DOI] [PubMed] [Google Scholar]
- 22.Oliven A, Nave R, Baruch A. Long experience with a web-based, interactive, conversational virtual patient case simulation for medical students' evaluation: comparison with oral examination. Med Educ Online . 2021;26(1):1946896. doi: 10.1080/10872981.2021.1946896. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Jacklin S, Maskrey N, Chapman S. Shared decision-making with a virtual patient in medical education: mixed methods evaluation study. JMIR Med Educ . 2021;7(2):e22745. doi: 10.2196/22745. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Sweigart L, Hodson-Carlton K. Improving student interview skills: the virtual avatar as client. Nurse Educ . 2013;38(1):11–15. doi: 10.1097/NNE.0b013e318276df2d. [DOI] [PubMed] [Google Scholar]
- 25.Anderson JK, Page AM, Wendorf DM. Avatar-assisted case studies. Nurse Educ . 2013;38(3):106–109. doi: 10.1097/NNE.0b013e31828dc260. [DOI] [PubMed] [Google Scholar]
- 26.Zackoff MW, Real FJ, Sahay RD, et al. Impact of an immersive virtual reality curriculum on medical students' clinical assessment of infants with respiratory distress. Pediatr Crit Care Med . 2020;21(5):477–485. doi: 10.1097/PCC.0000000000002249. [DOI] [PubMed] [Google Scholar]
- 27.Herbst R, Rybak T, Meisman A, et al. A virtual reality resident training curriculum on behavioral health anticipatory guidance: development and usability study. JMIR Pediatr Parent . 2021;4(2):e29518. doi: 10.2196/29518. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Classroom Simulator Survey. Accessed August 6, 2020. https://docs.google.com/forms/d/e/1FAIpQLSePyhh-aQMRGzgoCnLncklDp8E_HHlowjsPGZyOitOtqsYOwQ/viewform?edit_requested=true&usp=embed_facebook.
- 29.Alsaad A, Davuluty S, Bhide V, Lannen A, Maniaci M. Assessing the performance and satisfaction of medical residents utilizing standardized patient versus mannequin-simulated training. Adv Med Educ Pract . 2017;8:481–486. doi: 10.2147/AMEP.S134235. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Kapralos B, Cristancho S, Porte M, Backstein D, Monclou A, Dubrowski A. Westwood JD, Westwood SW, FelländerTsai L, et al., editors. Serious games in the classroom: gauging student perceptions. Medicine Meets Virtual Reality 18 Vol 163 IOS Press. 2011. pp. 254–259. [PubMed]
- 31.Dillman D. Mail and Internet Surveys The Tailored Design Method 2nd ed. John Wiley and Sons; 2007. [Google Scholar]
- 32.Real FJ, DeBlasio D, Ollberding NJ, et al. Resident perspectives on communication training that utilizes immersive virtual reality. Educ Health . 2017;30(3):228. doi: 10.4103/efh.EfH_9_17. [DOI] [PubMed] [Google Scholar]
- 33.Lee J, Kim H, Kim KH, Jung D, Jowsey T, Webster CS. Effective virtual patient simulators for medical communication training: a systematic review. Med Educ . 2020;54(9):786–795. doi: 10.1111/medu.14152. [DOI] [PubMed] [Google Scholar]
- 34.Kava BR, Andrade AD, Marcovich R, Idress T, Ruiz JG. Communication skills assessment using human avatars: piloting a virtual world Objective Structured Clinical Examination. Urol Pract . 2017;4(1):76–84. doi: 10.1016/j.urpr.2016.01.006. [DOI] [PubMed] [Google Scholar]
- 35.Andrade AD, Bagri A, Zaw K, Roos BA, Ruiz JG. Avatar-mediated training in the delivery of bad news in a virtual world. J Palliat Med . 2010;13(12):1415–1419. doi: 10.1089/jpm.2010.0108. [DOI] [PubMed] [Google Scholar]
- 36.Real FJ, DeBlasio D, Beck AF, et al. A virtual reality curriculum for pediatric residents decreases rates of influenza vaccine refusal. Acad Pediatr . 2017;17(4):431–435. doi: 10.1016/j.acap.2017.01.010. [DOI] [PubMed] [Google Scholar]
- 37.Carpenter JL. Cost analysis of Objective Structured Clinical Examinations. Acad Med . 1995;70(9):828–833. [PubMed] [Google Scholar]
- 38.Bosse HM, Nickel M, Huwendiek S, Schultz JH, Nikendei C. Cost-effectiveness of peer role play and standardized patients in undergraduate communication training. BMC Med Educ . 2015;15(1):183. doi: 10.1186/s12909-015-0468-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Gillette C, Stanton RB, Rockich-Winston N, Rudolph M, Anderson HG. Cost-effectiveness of using standardized patients to assess student-pharmacist communication skills. Am J Pharm Educ . 2017;81(10):6120. doi: 10.5688/ajpe6120. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Lewis K, Bohnert C, Gammon W, et al. The Association of Standardized Patient Educators (ASPE) Standards of Best Practice (SOBP) Adv Simul. 2017. 2(10) [DOI] [PMC free article] [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.