Abstract
Introduction
Some surgical site infections (SSI) could be prevented by following adequate infection prevention and control (IPC) measures. Poor compliance with IPC measures often occurs due to knowledge gaps and insufficient education of healthcare professionals. The education and training of SSI preventive measures does not usually take place in the operating room (OR), due to safety, and organisational and logistic issues. The proposed study aims to compare virtual reality (VR) as a tool for medical students to learn the SSI prevention measures and adequate behaviours (eg, limit movements…) in the OR, to conventional teaching.
Methods and analysis
This protocol describes a randomised controlled multicentre trial comparing an educational intervention based on VR simulation to routine education. This multicentre study will be performed in three universities: Grenoble Alpes University (France), Imperial College London (UK) and University of Heidelberg (Germany). Third-year medical students of each university will be randomised in two groups. The students randomised in the intervention group will follow VR teaching. The students randomised in the control group will follow a conventional education programme. Primary outcome will be the difference between scores obtained at the IPC exam at the end of the year between the two groups. The written exam will be the same in the three countries. Secondary outcomes will be satisfaction and students’ progression for the VR group. The data will be analysed with intention-to-treat and per protocol.
Ethics and dissemination
This study has been approved by the Medical Education Ethics Committee of the London Imperial College (MEEC1920-172), by the Ethical Committee for the Research of Grenoble Alpes University (CER Grenoble Alpes-Avis-2019-099-24-2) and by the Ethics Committee of the Medical Faculty of Heidelberg University (S-765/2019). Results will be published in peer-reviewed medical journals, communicated to participants, general public and all relevant stakeholders.
Keywords: education & training (see medical education & training), infection control, surgery
Strengths and limitations of this study.
This study is the first randomised controlled multicentre trial to assess virtual reality (VR) to teach infection prevention and control measures.
This study will be performed simultaneously in three European universities
Enrolment of medical students on a voluntary basis could introduce a bias because more keen and motivated students or with previous experience in an operating room (OR) could want to enrol.
This study will not allow assessing the long-term effect of VR to change medical students behaviours in the OR.
Introduction
Surgical site infections (SSI) contribute to healthcare-associated infections. The European Centre for Disease Prevention and Control estimated that 799 185 new cases of SSI and 16 049 attributable deaths occur every year in the European Union.1 Many SSIs could be prevented by following adequate infection prevention and control (IPC) measures such as optimal hand hygiene, timely preoperative antimicrobial prophylaxis or minimised circulation of personnel in the operating room (OR).2 3
Non-compliance with IPC measures often occurs due to knowledge gaps and insufficient education and training of healthcare professionals or students.4 For medical students, the highly technical OR environment may appear complex. Education and training in IPC measures and adequate behaviours with patients undergoing surgery cannot usually take place in the OR or may be difficult to organise, due to restricted areas and tight or unexpected surgical schedules. This lack of knowledge may lead to limited awareness of basic SSI prevention rules in students and trainees when accessing the OR, possibly compromising patients’ safety.5
Simulation methods are increasingly used in clinical teaching to address some of the challenges mentioned. Simulation centres are set up in medical schools to teach clinical procedures using a variety of approaches, including models, scenarios, video recording and direct feedback/discussion with trainees.6 7
Among the methods included in simulation pedagogy, virtual reality (VR) is an emerging and innovative technology which may offer multiple advantages over more traditional approaches for learners and instructors.8–10 VR presents a large list of advantages which are as follows: autonomous training available 24/7, total traceability, appealing support for learners who are disinterested in traditional training, and adaptable to the profile and level of the learner. Furthermore, the trainees are fully involved in the scenario, providing a ‘close to real life’ feeling for a better learning process.11 Finally, VR allows the immersion in a reality-like environment that recreate genuine conditions of the clinical environment, including the OR, with people who can talk, move and react to the player’s actions. Despite a vast number of applications, this technique is currently mainly employed for teaching complex surgical techniques or rare/urgent situations (massive admission of gunshot victims or control of Ebola outbreak).12–14
The aim of this study is to compare VR with conventional teaching as a tool to teach medical students SSI prevention measures and adequate behaviours in the OR.
Methods and analysis
Study setting
The study will take place in three universities: Grenoble Alpes University, France (3960 medical students from 2018/2019, with 211 in the third year), Imperial College London, UK (2188 medical students from 2018/2019, with 359 in the third year) and University of Heidelberg, Germany (2645 medical students from 2018/2019, with 358 in the third year).
Trial design
This is a randomised controlled multicentre trial entitled VIP Room (VIrtual reality for the oPerating Room) comparing an educational intervention based on VR simulation with a conventional education strategy. We used the SPIRIT (Standard Protocol Items Recommendations for Interventional Trials) checklist to guide the reporting of our protocol and the TIDieR (Template for Intervention Description and Replication) to guide the reporting of components of our intervention (table 1).15 16
Table 1.
Overview of the VIP Room study, as per TIDieR criteria
| TIDieR criteria | Description of intervention and quality control procedures |
| Brief name | VIP Room |
| Why? | Medical students lack training and education opportunities about IPC in OR due to organisational constraints. VR offers opportunities to resolve such constraints, as seen in other experiences of VR application to clinical education and training |
| What materials? | Ad hoc VR-based simulation of OR environments presenting a formative assessment of student learning via multiple methods |
| What procedures? | Participants provided with VR headset where they are immersed in OR simulation, are presented with knowledge and education about IPC and face in-simulation assessments |
| Who provided? | VR session led by postdoctoral nurses, pharmacists, physicians and practitioner researchers, experts in education about IPC, and with experience in developing and evaluating electronic tools such as apps, games and VR |
| How? | Delivered via individual VR commercial headset. Planned engagement of 30 to 45 min per session |
| Where? | University classrooms |
| When and how much? | One to five sessions (depending on the student’s progression) |
| Tailoring | Standardised scenarios for all participants |
IPC, infection prevention and control; OR, operating room; TIDieR, Template for Intervention Description and Replication; VIP Room, VIrtual reality for the oPerating Room; VR, virtual reality.
For each of the three sites, the trial design will include two parallel groups of students (1:1):
The first group will be randomised to receive the VR education intervention (30 to 45 min per session, one to five sessions depending on the student’s progression). This group will not receive any theory prior to the simulation.
The second group will receive the conventional method (a 1-hour standardised slide-based lecture and two real-life videos about hand hygiene and hand gloving in the OR).
The participants will be informed about the study purposes and procedures, and have time for reflection before inclusion. After getting their signed consent, participants from each university will be randomised to both groups during the school year 2019/2020. If students decline to participate, no disadvantages will arise for them.
Student eligibility criteria
All third-year medical students during the 2019/2020 school year without any contraindication to use VR (epilepsy, vertigo, psychiatric pathology, pregnancy, cardiac pacemaker or defibrillator, coronary cardiac pathology, hearing aid, strabismus and amblyopia) will be eligible to take part in the study. The students will commit to not be under the influence of alcohol, recreational or prescription drugs during the intervention.
Third-year medical students were selected because this population is usually free of the OR environment (no visit to OR as part of their training) and without SSI prevention measures knowledge at this stage.
Intervention
The intervention is a VR-based scenario where the student is immersed (figure 1A, B). The VR contains two real-life videos about hand hygiene and hand gloving in the OR. They need to use googles to visualise the scenario (figure 1C). The scenario was built beforehand by selecting the 10 most important IPC measures to teach to students the prevention of SSI. The selection of these 10 measures included a review of current recommendations published by the WHO and public health authorities or scientific societies in France, England and Germany, respectively. Infection control practitioners of the three universities performed the review and compared the measures between the three countries. Identical measures between the three countries and those most relevant for medical students were selected.3 17–20 The scenarios were developed in local languages according to countries (French, English and German).
Figure 1.
Pictures of the scenario in VR and tests of the scenario with headsets in Grenoble Alpes University. (A and B) Pictures of the scenario. (C) Tests of the scenario with headsets in Grenoble Alpes University. VR, virtual reality.
The VR scenario includes three different environments: the first environment (E1) takes place in the anaesthesiologist consultation room before the surgery, the second (E2) takes place in the preoperative room, which includes the locker room and the washing room, and the third (E3) takes place in the OR. Each environment presents the participants with different scenes, each of them a mixture of information and then a task for the participant to complete. For instance, participants may be shown a video clip about hand gloving procedures in the OR, and then asked to select the right or wrong answer. During the proceedings, the performance of the students is monitored using different indicators (duration of the session and number of errors). The duration of a session in each environment is 10 to 15 min. The total duration of a session is 30 to 45 min. Different versions of the environment (five maximum) are available. The students are able to choose and/or repeat the session.
The learning outcomes of E1, E2 and E3 as defined by Bloom and Krathwohl21 22 are described in table 2. The VR group receives a feedback by email after the session
Table 2.
The 10 most important surgery site infection prevention measures chosen in VIP Room study
| Environment | Description |
| E1: Anaesthesiologist consultation room | Preoperative bathing: explain the basic concept of preoperative bathing, illustrate the areas with the highest microbiological load of the human body |
| Decolonisation with mupirocin ointment with or without chlorhexidine bodywash for the prevention of Staphylococcus aureus infection in nasal carriers: describe the indications for preoperative S. aureus decolonisation, describe the decolonisation procedure | |
| Optimal timing for preoperative SAP: understand the impact of SAP on SSI development and resistance rates, define the usage of SAP for patients based on the previously learned risk assessment (ie, contamination classes), describe the correct timing of SAP application | |
| Hair removal: understand the impact of hair removal before surgery, identify appropriate methods of hair removal | |
| E2: Preoperative room | Surgical hand preparation: describe the correct procedure of surgical hand hygiene including steps and time, identify the correct substances for hand hygiene, and demonstrate the correct movement from the washing room to the OR while holding hands up and without touching anything |
| E3: OR | Outfit: chose the correct clothing before entrance in OR, identify the pitfalls of clothing (ie, no wearing of rings, jewellery, artificial nails, watches, coverage of mouthpieces around nose and mouth and coverage of hoods of all hair), describe the correct procedure of gloving |
| Guarantee sterility: identify errors of sterility in the OR | |
| Air treatment: understand the importance of airflow in the OR and demonstrate the correct regulation of airflow in the OR | |
| Surgical site preparation: identify the correct substances for surgical site disinfection | |
| Drapes and gowns: identify and describe the correct placing of drapes and gowns on the patient |
OR, operating room; SAP, surgical antibiotic prophylaxis; SSI, surgical site infection; VIP Room, VIrtual reality for the oPerating Room.
Enhancing the fidelity of delivery and adherence to the intervention
Implementation of the intervention is done entirely by local teams. The local site coordinator is the direct link between the centre and the study team, and is in charge of daily study activities. The coordinator leads the implementation of the intervention (awareness and communication of the study, delivery of education intervention, handling of VR goggles, leading VR session, etc.). The fidelity of the intervention is ensured by regular communication between the three study sites to discuss the project in detail, expectations, responsibilities, key elements of the intervention and progress. Any implementation difficulties are collectively discussed, with the aim of finding solutions.
The local coordinator engages with relevant education managers to ensure institutional buy-in and facilitate participant recruitment.
Control conditions
As for the intervention group, the control group will receive a conventional teaching. This 1-hour didactic, classroom-based lecture includes the 10 selected SSI measures and two real-life videos about hand hygiene and hand gloving in the OR.
Outcomes
The main outcome of the study will be the difference between scores obtained at the IPC exam between the first group and the second group. This assessment corresponds to the Kirkpatrick scale level 2.23 24 The written exam will be the same in the three countries in local language. The exam will include multiple choice questions. The questions have been designed to assess the knowledge of the students taught using the two methods. Questions were created before randomising students.
The secondary outcomes of the study include: the satisfaction and the student’s progression for the VR group. The satisfaction will be evaluated with a questionnaire (Kirkpatrick scale level 1). The student’s progression will be evaluated by the number of sessions, the length of time that participants spend on each VR environment and session, and the number of errors per sessions by time.
Study timeline
The trial schedule of enrolment, interventions and assessments is presented in the figure 2. The study will take place during the 2019/2020 school year.
Figure 2.
Trial schedule of enrolment, interventions, and assessments (as recommended by SPIRIT). IPC, infection prevention and control; SPIRIT, Standard Protocol Items Recommendations for Interventional Trials.
Sample size
Based on previous students’ results of 2018/2019 exam in Grenoble Alpes University (student in fourth year of medical school in 2019/2020), we expect a mean of 14.5 points in the control group and a SD of 2.5 points. Our hypothesis is an expected mean difference of 1.5 points between participants exposed to the conventional education method and those exposed to the VR intervention, which gives an expected effect size of 0.6. Therefore, the minimal number of evaluable students required in each study group for a statistical power of 80% at an alpha risk of 5% is 45. To mitigate participant consent withdrawal, recruitment will include 55 students per groups. We want to estimate the mean difference in each university; thus, we will include 55 students per group and per university.
Recruitment
We will recruit voluntary students in the three participating universities on the academic year 2019/2020. At each site, the principal investigator of each country will provide an information sheet; special session of information for students will be planned according to their university schedule. The written agreement of participants will be collected. As per sample size, once 110 participants are enrolled in the study per site, we will cease recruitment. At Grenoble Alpes University, students will receive the information sheets during/after the welcome day for third-year students. At Heidelberg University, students will receive the information sheets during/after an infection control lecture. At Imperial College London the study will be advertised on the three campuses via flyers. It will also be disseminated to the CHERSNet (Centre for Higher Education Research and Scholarship) Network due to the focus of the group on developing and using novel educational platforms, and the MERU (Medical Education Research Unit). The local study coordinator will also disseminate information about the study at the end of scheduled classes. In each university, the help of the third-year Student Representative Office to disseminate and market the study will be arranged, as well as the responsible lecturers for year three in medicine.
Assignment of interventions
Sequence generation (ie, randomisation)
An identification number will be assigned to students in each university. Students will be randomised in each university to one of the two study groups (ie, VR or lecture) using computer-generated random numbers. As all participants will be included before the start of the study, they will be randomised simultaneously, so there is no need for restriction of the simple randomisation. The only stratification variable is the university.
Blinding
Participants will be aware of their allocation to one or other study group. The examiner for the IPC exam and the statistician performing the analysis will be blinded to the identification of the group.
Data collection and management
Sociodemographic characteristics (age and gender) of the participants and previous experience in OR will be collected at the beginning of the study using paper questionnaire. For the VR group, the number of session, the number of errors incurred by the participants per session as well as the time spent on each environment will be collected. At the end of the study, participants allocated to receive the VR intervention will also complete a questionnaire to assess their level of satisfaction to the Kirkpatrick model level 1. The Geneva Emotion Wheel and the System Usability Scale will be used.25–27
We will add open-ended questions about the improvement of the scenario and hygiene education in the VR group.
At the end of the year, the score in IPC exam will be collected with the detail of the questions.
Data monitoring, harms and auditing
We do not have a Data Safety and Monitoring Board because we consider that our trial poses minimal risk to participants. We do not plan to conduct any interim analyses for our primary outcome. We do not plan any audits of trial conduct.
We will monitor, however, the proportion of students allocated to the VR intervention group reporting any adverse effects during, or following the use of the VR goggles, including severe fatigue, severe stress/anxiety or headache. We anticipate these effects, if any, to be infrequent and transient, as per the existing literature.
Statistical methods
The data will be analysed first with intention-to-treat analysis and then per protocol for each university. The variables will be described by group. Qualitative variables will be described in numbers and percentages. Quantitative variables will be described by the median and IQR. Student scores in the assessment session will be compared between groups by a Student’s t-test. Then, we will do a meta-analysis on original data from these three studies and we will study the interaction between group and centre. The potential benefit of VR may depend on other variables collected; this will be studied by linear regression to analyse the interaction between groups and other covariates (age and gender), using data from all centres. For all tests, we will use two-sided p values, with alpha <0.05 level of significance. We do not plan any correction for multiple testing. If less than 5% of the scores (the primary outcome) are missing, we will not impute the missing scores. If between 5% and 20% of the scores are missing, we will impute the missing scores by multiple imputation. If more than 20% of the scores are missing, we will not impute the missing scores and will interpret the results with caution. We will not impute any other missing data. Quantitative analyses will be performed with Stata 15 or later.28
Patient and public involvement
Representative students were consulted about the design of the study to define the conventional method for the control group. They gave their opinion before the scenario building about suitable VR environment (duration, avatar…). They have motivated their classmates to participate in the study.
Ethics and dissemination
Consent, confidentiality and access to data
The three studies conform to the principles for all medical research (Declaration of Helsinki), respecting the applicable legal precepts regarding the protection of personal data, as well as European regulation and the regulation in each country. This study has been approved by the Medical Education Ethics Committee of the Imperial College London (MEEC1920-172), the Ethical Committee for the Research of Grenoble Alpes University (CER Grenoble Alpes-Avis-2019-099-24-2) and the Ethics Committee of the Medical Faculty of Heidelberg University (S-765/2019).
Participating students will be informed by the co-investigators before the education, using an information letter validated by the relevant research ethics committee in each country.
A code will be attributed to students and data will be anonymised. The correspondence between the code and student will be kept by the investigators. These documents will be stored in a closed cabinet. All electronic data will be maintained in password-protected computers and stored in encrypted form.
Dissemination policy, authorship eligibility and data sharing plans
We aim to disseminate our findings regardless of negative or null results. We plan to present the results of this research project at national and international scientific meetings. We aim to publish our results in open-access journals, so they are widely available to interested international audiences. All named authors in the protocol will be offered participation in the final and any subsequent papers. We will not use any professional writing services.
Due to institutional policies, we will not be able to grant public access to the participant-level data set. We may be able to provide a statistical code. Relevant persons responsible for education at the participating universities will be informed of the results.
The development and use of new cognitive education tools may improve the compliance of future surgeons and surgical teams with SSI preventive measures, and promote best practices during surgical procedures.
Supplementary Material
Acknowledgments
We thank Sylvie Levet and all representative students in each university for advice and help.
Footnotes
Twitter: @castrocloud
CM and GB contributed equally.
NTM and CL contributed equally.
Contributors: CM and GB contributed equally. NTM and CL contributed equally. CM, CL, NTM, GB, JLB, HT and AC designed the study. CM, CL, NTM and GB drafted the protocol and EC-S, VME, AC, HT, PG, PA, JP, BR-D and JLB revised the manuscript. HT built the analysis plan and calculated the sample size and will do the analysis. VE, EC-S, CM, GB, NTM and CL contributed to the bibliography. PA, JP, PG, BR-D and JLB contributed to the methodology to build the scenario. CL, NTM, GB, ECS, CM and VME built the scenario and the conventional lecture. AC, EC-S, CL, VE, NTM and CM prepared the documents for the Ethics Committees.
Funding: This work was supported by EIT Health grant number (19205). ECS was supported by the National Institute for Health Research Health Protection Research Unit (NIHR HPRU) (HPRU-2012–10047) in Healthcare Associated Infection and Antimicrobial Resistance at Imperial College London in partnership with Public Health England, and acknowledges the support of the NIHR Imperial Patient Safety Translational Research Centre. He is a National Institute for Health Research (NIHR) Senior Nurse and Midwife Research Leader. GB was supported by ESRC as part of the Antimicrobial Cross Council initiative supported by the seven UK research councils, and also the support of the Global Challenges Research Fund.
Competing interests: None declared.
Patient consent for publication: Obtained.
Provenance and peer review: Not commissioned; externally peer reviewed.
References
- 1.Cassini A, Plachouras D, Eckmanns T, et al. Burden of six healthcare-associated infections on European population health: estimating incidence-based Disability-Adjusted life years through a population Prevalence-Based modelling study. PLoS Med 2016;13:e1002150. 10.1371/journal.pmed.1002150 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Berríos-Torres SI, Umscheid CA, Bratzler DW, et al. Centers for disease control and prevention guideline for the prevention of surgical site infection, 2017. JAMA Surg 2017;152:784–91. 10.1001/jamasurg.2017.0904 [DOI] [PubMed] [Google Scholar]
- 3.World Health Organization Global guidelines for the prevention of surgical site infection, 2016. Available: https://www.who.int/gpsc/global-guidelines-web.pdf?ua=1 [Accessed 21 Aug 2019]. [PubMed]
- 4.Tacconelli E, Buhl M, Humphreys H, et al. Analysis of the challenges in implementing guidelines to prevent the spread of multidrug-resistant gram-negatives in Europe. BMJ Open 2019;9:e027683. 10.1136/bmjopen-2018-027683 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.O'Neill R, Shapiro M, Merchant A. The role of the operating room in medical student education: differing perspectives of learners and educators. J Surg Educ 2018;75:14–28. 10.1016/j.jsurg.2017.06.013 [DOI] [PubMed] [Google Scholar]
- 6.Hexter AT, O'Dowd-Booth C, Hunter A. Factors that influence medical student learning in the operating room. Med Teach 2019;41:555–60. 10.1080/0142159X.2018.1504163 [DOI] [PubMed] [Google Scholar]
- 7.Sutherland LM, Middleton PF, Anthony A, et al. Surgical simulation: a systematic review. Ann Surg 2006;243:291–300. 10.1097/01.sla.0000200839.93965.26 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kononowicz AA, Woodham L, Georg C E, et al. Virtual patient simulations for health professional education. Cochrane Database Syst Rev 2018;6:CD012194. [Google Scholar]
- 9.Kyaw BM, Saxena N, Posadzki P, et al. Virtual reality for health professions education: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res 2019;21:e12959. 10.2196/12959 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Car J, Carlstedt-Duke J, Tudor Car L, et al. Digital health education collaboration digital education in health professions: the need for Overarching evidence synthesis. J Med Internet Res 2019;21:e12913. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Izard SG, Juanes JA, García Peñalvo FJ, et al. Virtual reality as an educational and training tool for medicine. J Med Syst 2018;42:50. 10.1007/s10916-018-0900-2 [DOI] [PubMed] [Google Scholar]
- 12.Gentry SV, Gauthier A. L’Estrade Ehrstromet B al. J Med Internet Res 2019;21:e12994. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Akl EA, Sackett KM, Erdley WS, et al. Educational games for health professionals. Cochrane Database Syst Rev 2013;1:CD006411. [DOI] [PubMed] [Google Scholar]
- 14.Ragazzoni L, Ingrassia PL, Echeverri L, et al. Virtual reality simulation training for Ebola deployment. Disaster Med Public Health Prep 2015;9:543–6. 10.1017/dmp.2015.36 [DOI] [PubMed] [Google Scholar]
- 15.Chan A-W, Tetzlaff JM, Gøtzsche PC, et al. Spirit 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ 2013;346:e7586. 10.1136/bmj.e7586 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hoffmann TC, Glasziou PP, Boutron I, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014;348:g1687. 10.1136/bmj.g1687 [DOI] [PubMed] [Google Scholar]
- 17.KRINKO Prävention postoperativer Wundinfektionen. Bundesgesundheitsbl 2018;61:448–73. [DOI] [PubMed] [Google Scholar]
- 18.National Institute for Health and Care Excellence Surgical site infections: prevention and treatment, 2019. Available: https://www.nice.org.uk/guidance/ng125/resources/surgical-site-infections-prevention-and-treatment-pdf-66141660564421 [Accessed 22 Aug 2019]. [PubMed]
- 19.Société Française d’Hygiène Hospitalière Gestion préopératoire Du Risque infectieux. Hygiènes 2013;21:1–116. [Google Scholar]
- 20.Société Française d’Anesthésie et de Réanimation Actualisation de recommandations : antibioprophylaxie en chirurgie et médecine interventionnelle (patients adultes), 2018. Available: http://www.cpias.fr/nosobase/recommandations/Antibioprophylaxie-RFE-mise-a-jour-2018.pdf [Accessed 22 Jan 2020]. [DOI] [PubMed]
- 21.Bloom BS, Englehart MB, Furst EJ, et al. Taxonomy of educational objectives, the classification of educational goals. Handbook I: the cognitive domain. New York: McKay Co Inc, 1956. [Google Scholar]
- 22.Krathwohl DR. A revision of Bloom's taxonomy: an overview. Theory Pract 2002;41:212–8. 10.1207/s15430421tip4104_2 [DOI] [Google Scholar]
- 23.Kirkpatrick D. Great ideas revisited. techniques for evaluating training programs. revisiting Kirkpatrick' S four-level model. Training Dev 1996;50:54–9. [Google Scholar]
- 24.Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels. San Francisco Berrett-Koehler Publishers, 1998: 127. [Google Scholar]
- 25.Brooke J. SUS-a quick and dirty usability scale. Usability Eval Ind 1996;189:4–7. [Google Scholar]
- 26.Scherer KR. What are emotions? and how can they be measured? Social Science Information 2005;44:695–729. 10.1177/0539018405058216 [DOI] [Google Scholar]
- 27.Scherer KR, Shuman V, Fontaine JRJ, et al. The GRID meets the Wheel: Assessing emotional feeling via self-report : Fontaine JRJ, Scherer KR, Soriano C, et al., Components of emotional meaning: a sourcebook. Oxford: Oxford University Press, 2013: 281–98. [Google Scholar]
- 28.StataCorp Stata statistical software: release 15. College Station, TX: StataCorp LLC, 2017. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.