Abstract
Background
Graduates of simulation fellowship programmes are expected to have the ability to perform a variety of simulation specific skills at the time of graduation. Currently, simulation fellowship directors have access to tools to assess the ability of a fellow to debrief learners. However, there is no tool to assess a simulation fellow’s competency in technical skills. The purpose of our manuscript was to develop and obtain content validation of a novel instrument designed to assess a simulation fellow’s ability to perform the five core simulation technical skills.
Methods
The study protocol was based on a methodology for content validation of curriculum consensus guidelines. This approach involves a three-step process, which includes the initial delineation of the curricular content. This was then followed by the validation of the curricular content using survey methodology and lastly obtaining consensus on modifications using Delphi methodology.
Results
Two rounds of modified Delphi methodology were performed. Seventy-four respondents provided feedback on the round 1 survey and 45 respondents provided feedback on round 2. The final assessment tool has five elements and 16 subitems with four optional subitems.
Conclusion
The Evaluation of Technical Competency in Healthcare Simulation tool provides an instrument developed from a national consensus of content experts. This tool provides simulation fellowship directors a method to evaluate fellows’ competency in technical skills.
Keywords: simulation, simulation fellowship, simulation technician, simulation fellowship assessment
Introduction
Medical simulation is now a mainstream teaching methodology that demonstrates improved clinical performance and associated clinical outcomes.1–9 One manifestation of the increased utilisation of this teaching methodology is the demand for medical simulation fellowship-trained faculty. Scholars estimate the 55 medical simulation fellowship programmes in North America alone, represent a 10-fold growth in the last decade.10 11 Currently, neither the Accreditation Council on Graduate Medical Education (ACGME) or Royal College of Physicians and Surgeons of Canada accredits medical simulation fellowships. The lack of accreditation leads to varied curricula and experiences for simulation fellowship graduates. However, graduates of simulation fellowship programmes are expected to have the ability to perform a variety of simulation-specific skills at the time of graduation prior to becoming simulation centre directors and faculty.12 These skills include, but are not limited to, being able to effectively debrief learners, design simulation scenarios and cases, and perform basic simulation technical skills.12
The expectations of a foundational skill set for simulation fellowship-trained faculty have created a need for fellowship directors to assess fellows’ ability to perform simulation-specific skills. Currently, simulation fellowship directors have access to a limited number of tools to assess the ability of a fellow to debrief learners including the Debriefing Assessment for Simulation in Healthcare and the Objective Structured Assessment of Debriefing.13 14 However, there is no tool to assess a simulation fellow’s competency in technical skills.
Simulation technicians are expected to be able to perform five core tasks.15 These include equipment set-up and breakdown, programming scenarios into software, operation of software during simulation, audiovisual support and on-site simulation maintenance.15 The purpose of this study is to develop and obtain content validation of a novel instrument designed to assess a simulation fellow’s ability to perform the five core simulation technical skills.
Methods
Study protocol
The study protocol was based on the methodology employed by Cumyn and Harris for content validation of curriculum consensus guidelines.16 This approach involves a three-step process, which includes the initial delineation of the curricular content. This was then followed by the validation of the curricular content using survey methodology and lastly obtaining consensus on modifications using Delphi methodology.16
Initial delineation of curricular content
A literature search was performed to identify the key elements of a simulation technician’s skill set using search terms including ‘simulation technician’, ‘simulation technician assessment’, ‘simulation fellowship’, ‘simulation fellowship curriculum’ and ‘medical simulation fellowship’. This search did not identify any current assessment tools to evaluate simulation fellows’ technical skill set. The initial blueprint for the assessment tool was developed by two current medical simulation directors, who completed 1 year medical simulation fellowships. This initial blueprint was then evaluated by four subject matter experts (SME), which included three simulation fellowship graduates and a simulation technician with experience at an institution with a simulation fellowship programme.
Validation of curricular content using survey methodology
Simulation fellowship directors, simulation technicians and simulation educators in the USA and Canada participated in the analysis of the assessment tool blueprint. An expert was defined as a simulation fellowship director, a simulation technician or a simulation educator who has a background in simulation technology or operations. The survey was distributed to a list of simulation fellowship directors published by previous scholars, simulation technicians on The Gathering of Healthcare Simulation Technology Specialists listserv and the Society for Simulation in Healthcare Simulation Operations and Technology section listserv.10 In addition, a web-based search was performed to identify any additional simulation fellowship directors who were not included on the previously published list using key search terms: ‘simulation fellowship’, ‘simulation fellowship director’, ‘medical simulation fellowship’, ‘anesthesia simulation fellowship’, ‘emergency medicine simulation fellowship’ and ‘surgical simulation fellowship’.10
A web-based survey (www.surveymonkey.com) was disseminated that asked experts to evaluate each element of the assessment tool and whether it should be kept, modified or deleted. The experts were asked to provide an explanation for any modification or deletion they suggested. In addition, experts could suggest extra content they felt was missing from the initial assessment tool. Once the survey results were compiled, a free marginal kappa was calculated for each subitem to assess inter-rater agreement. Based on the work of previous scholars, a value greater than 0.7 was used to denote satisfactory inter-rater agreement.17 Additional content that was suggested by experts was analysed through an inductive qualitative analysis approach. Two coauthors independently reviewed the qualitative comments and identified thematic groups. Any discrepancy was resolved through discussion, review of relevant quotations and a third reviewer if necessary.
Obtaining consensus on modification using modified Delphi methodology
Assessment tool subitems that did not achieve satisfactory inter-rater agreement were collected and disseminated in a second survey. During the second survey, experts rated assessment tool subitems on a 5-point scale: strongly disagree, disagree, neutral, agree and strongly agree. Subitems that were rated by 75% of experts greater than or equal to 4 were included in the final assessment tool.
Statistical analysis
Statistical analyses of both rounds 1 and 2 were performed by using SAS (V.9.4; SAS Institute). In round 1, categorical data are presented as counts and percentages. The distribution of survey responses to each subitem within each element is summarised using frequencies and percentages for each of the three response categories: keep, modify or delete. Since no restrictions were placed on the marginal distributions, that is, raters were free to place as many subitems in each category as they felt appropriate without quota restriction, the free marginal kappa is used to determine if consensus is reached.
In round 2, categorical data are also presented as counts and percentages. The distribution of survey responses to each subitem within each element is summarised using frequencies and percentages for each of the five response categories: strongly disagree, disagree, neutral, agree and strongly agree. Additionally, mean and SD are provided for each subitem.
The Institutional Review Boards at Johns Hopkins University and Florida Atlantic University determined this study did not qualify as human subjects research. During the study, all experts were blinded from each other’s identities.
Results
Round 1 data
Seventy-four respondents provided feedback on the round 1 survey. Most of them were simulator directors (46%) and simulation technicians (27%), with the remaining 27% listed as ‘other’ commonly listing leadership positions or faculty titles that covered the scope of either a simulation specialist or simulation faculty member and were included (eg, operations manager, clinical nurse educator). Of the respondents included in the study, 20% had 3–5 years of experience, 30% had 5–10 years of experience and 35% had more than 10 years of experience. Sixty-two per cent were involved in teaching or evaluating simulation fellows, and 59% evaluated the quality of simulation fellows’ technical skills as informal/formative feedback during and after simulations.
In this round, respondents were asked to indicate for each element, whether the subitem should be deleted, kept or modified. Three subitems reached free marginal kappa above 70%: element 4/subitem 1 (Kf=0.76), element 4/subitem 2 (Kf=0.76) and element 5/subitem 2 (Kf=0.83). Few participants offered two responses for some subitems and their answers were excluded from kappa calculations for those particular subitems (table 1). Subitems that did not meet consensus were analysed using an inductive qualitative method. Qualitative responses were reviewed independently by two coauthors and thematic groups were agreed on (table 2).
Table 1.
Free marginal kappa
| Delete | Keep | Modify | Raters (n) | Kfree | |
| n (%) | n (%) | n (%) | |||
| Element 1: Demonstrates ability to operate software during simulations. | |||||
| Operates a variety of full-body simulators with ‘on the fly’, preprogrammed and automated scenarios. | 2 (2.7) | 61 (82.4) | 11 (14.9) | 74 | 0.547 |
| Appropriately manages simulator physiologic responses to learner. | 4 (5.5) | 63 (86.3) | 6 (8.2) | 73 | 0.627 |
| Manages hybrid simulation scenarios that include full-body simulators, standardised patients, partial task trainers and other simulation technology. | 4 (5.5) | 63 (86.3) | 6 (8.2) | 73 | 0.627 |
| Operates wireless simulators for special population scenarios (emergency medical services, disaster, and so on). | 11 (14.9) | 55 (74.3) | 8 (10.8) | 74 | 0.371 |
| Element 2: Demonstrates ability to programme simulators. | |||||
| Ability to programme user interfaces for different models of full-body simulators. | 6 (8.3) | 53 (73.6) | 13 (18.8) | 72 | 0.363 |
| Ability to programme a variety of clinical scenarios for various learner populations. | 3 (4.2) | 57 (80.2) | 11 (15.5) | 71 | 0.498 |
| Ability to customise preprogrammed scenarios to match objectives. | 2 (2.8) | 64 (88.9) | 6 (8.3) | 72 | 0.692 |
| Programmes appropriate physiologic responses to learner actions at scenario branch points. | 5 (7.1) | 55 (78.6) | 10 (14.3) | 70 | 0.456 |
| Ability to programme multiple simulators for multipatient or hybrid simulation. | 11 (15.7) | 47 (67.1) | 12 (17.1) | 70 | 0.247 |
| Element 3: Demonstrates ability to perform basic maintenance on a variety of simulators. | |||||
| Ability to perform basic maintenance and repairs on a variety of full-body simulators (eg, hardware issues related to pneumatics, basic electrical connections, switches and hoses; software connections including Wi-Fi and simulator direct connections resulting from Internet Protocol (IP) connections and media access control (MAC) addresses). | 12 (17.4) | 42 (60.9) | 15 (21.8) | 69 | 0.16 |
| Ability to perform basic maintenance and repairs on partial task trainers (eg, replacement of tubes, skins and simulated blood). | 7 (10.0) | 52 (74.3) | 11 (15.7) | 70 | 0.371 |
| Ability to perform basic maintenance and repairs on surgical skills trainers (eg, software updates, internet connectivity, learner registration and login). | 10 (14.1) | 43 (60.6) | 18 (23.4) | 71 | 0.165 |
| Element 4: Demonstrates ability to appropriately set up and break down simulation equipment. | |||||
| Selects appropriate simulation equipment to pair with scenario objectives. | 4 (5.7) | 64 (91.4) | 2 (2.9) | 70 | 0.757 |
| Demonstrates ability to set up for in situ simulation scenarios. | 3 (4.3) | 64 (91.4) | 3 (4.3) | 70 | 0.756 |
| Demonstrates ability to set up for standardised patient and multipatient scenarios. | 4 (5.7) | 60 (85.7) | 6 (8.6) | 70 | 0.612 |
| Demonstrates ability to set up hybrid simulation scenarios/special populations/patient safety scenario. | 4 (5.7) | 62 (88.6) | 4 (5.7) | 70 | 0.682 |
| Demonstrates ability to perform basic moulage on both simulators and standardised patients. | 4 (5.8) | 57 (82.6) | 8 (11.6) | 69 | 0.542 |
| Element 5: Demonstrates ability to provide audiovisual support for simulation scenarios. | |||||
| Demonstrates ability to set up and operate audiovisual equipment and software including learning management systems. | 5 (7.1) | 55 (78.6) | 10 (14.3) | 70 | 0.457 |
| Demonstrates ability to record and playback simulation scenarios. | 2 (2.9) | 66 (94.3) | 2 (2.9) | 70 | 0.833 |
| Demonstrates ability to create assessments and analyse basic data in learning management systems. | 4 (5.7) | 58 (82.9) | 8 (11.5) | 70 | 0.548 |
| Demonstrates ability to operate audiovisual equipment for multipatient scenario/special populations. | 7 (10.0) | 53 (75.7) | 10 (14.3) | 70 | 0.397 |
| Demonstrates ability to troubleshoot simulator hardware, software and interconnectivity issues. | 7 (10.0) | 51 (72.9) | 12 (14.1) | 70 | 0.346 |
| Demonstrates ability to set up mobile audiovisual support for simulation scenarios outside the simulation lab (in situ simulations, emergency medical services field training, and so on). | 8 (11.4) | 54 (77.1) | 8 (11.4) | 70 | 0.424 |
| Rating scale | |||||
| E-TeCHS tool rating scale | 1 (1.5) | 50 (74.6) | 16 (23.9) | 67 | 0.412 |
E-TeCHS, Evaluation of Technical Competency in Healthcare Simulation.
Table 2.
Round 1 qualitative analysis
| Theme | Description |
| Element 1 | |
| Clarify and include specifics about the meaning of full-body simulators and ‘on the fly’ operation. | ‘For number one, mostly keep but may want to use language other than ‘on the fly’ because people may not know what that means’; ‘I am not sure what you mean by a variety of full body simulators. Does that mean you can operate different vendors? Baby child adult?’ |
| Change the verb from manages to executes since the learner is being evaluated on operating the software. | ‘Modify the ‘hybrid simulations’ item by specifying what you mean by ‘manages’ vis-à-vis this instrument (ie, evaluate technical competence) and this Element (ie, operate software).’ |
| Delete the term ‘wireless’. | ‘Most of the time our simulators are not hardwired.’ |
| Element 2 | |
| Programming different clinical scenarios and not different models of simulators is more valuable as some simulation labs have only one brand of simulator. | ‘Ability to program a variety of clinical scenarios adequate for the learners educational goals’; ‘Only concern here is some labs may not have ’different models' of simulators. May only have one brand.’ |
| Programming appropriate physiologic responses and customising preprogrammed scenarios were redundant. | ‘The item with scenario branch points is the same as a previous question, just worded differently.’ |
| Programming one simulator requires same skill as programming multiple simulators. | ‘Programming three scenarios takes the same skill set as programming one scenario. You just do it three times.’ |
| Element 3 | |
| Separate into two subitems, one focused on hardware and one focused on software. | ‘Break apart hardware issues from IT/network connection issues.’ |
| High-fidelity maintenance is too technical and simulation fellows do not need this skill set. | ‘Them knowing the basic repairs are essential, but knowing how to repair internal parts of simulators take more time to understand and sometimes it is not something you do at all b/c it will void warranties that you may have on your equipment’; ‘Full body simulator maintenance and repair should be left to sim techs and industry reps.’ |
| Surgical skills trainers are not universal. | ‘Surgical skill trainers is very specific and not available in all places.’ |
| Element 4 | |
| Separate phrases ‘standardized patient’ and ‘multi-patient’ scenarios into distinct subitems. | ‘SP and multi-patient scenarios should be broken out into two separate competencies, as they require different skills.’ |
| Remove the term ‘basic’ from the phrase ‘basic moulage’ as this term could have vastly different interpretations. | ‘Modify by defining what you mean by ‘basic’. (My first moulage was a glass shard embedded in an upper arm. Is that basic?)’; ‘This term is used in a wide variety of circumstances and can mean different things in different situations.’ |
| Element 5 | |
| Separate ‘audiovisual equipment’ and ‘learning management systems’ into distinct subitems. | ‘LMS use and AV use are very different skills. break apart.’ |
| Creating assessments and analysing data are not technical skills, while inputting assessments and using an LMS is a technical skill. | ‘Creating assessments requires reliability and validity considerations that I believe are beyond the expectation of the technician. BUT, data analysis in the LMS would be appropriate.’ |
| Delete the phrase ‘demonstrates ability to operate audiovisual equipment for multi-patient scenario/special populations’ as it was redundant. | “Modify or Delete? 1st item is ’setup and operate av’ so this would umbrella under that, yes? Else specify unique qualities of ‘multi-patient scenario/special populations’.” |
| Modify the phrase ‘troubleshoot simulator hardware, software and interconnectivity issues’ as these are felt to be outside the scope of this element. | ‘Modify by possibly moving to a different Element bc ‘troubleshoot simulator hw, sw and interconnectivity’ is outside the scope of this Element, that is, provide av support.’ |
| Rating scale | |
| 7-point scale is too many rating options. | ‘Too many ratings, making it difficult for individuals to choose between 2 adjacent levels’; ‘7 points is too much!’ |
AV, audiovisual; b/c, because; hw, hardware; IT, information technology; LMS, learning management system; reps, representatives; sim, simulation; SP, standardised patient; sw, software; techs, technicians.
Round 2 data
Forty-five respondents provided feedback on round 2. They were asked to rate on a Likert scale of 1–5 (strongly disagree to strongly agree) the subitems proposed. Most of subitems received at least 75% of agree or strongly agree responses, with exception of element 3/subitem 1 (65.1%), element 5/subitem 2 (73.8%), element 5/subitem 4 (61.9%) and element 5/subitem 6 (66.7%). The lower mean and higher SDs also confirmed the lower consensus on these subitems compared with all other subitems (table 3).
Table 3.
Round 2 data
| Element/subitem | SD | D | N | A | SA | A+SA | Responses (n) | Mean | SDev |
| n (%) | n (%) | n (%) | n (%) | n (%) | % | ||||
| Element 1: Demonstrates ability to operate software during simulations. | |||||||||
| Operates a variety of full-body simulators (adult, paediatric, birthing, and so on) with on-demand changes dependent on learner actions (‘on the fly’), preprogrammed and automated scenarios. | 3 (4.5) | 15 (34.1) | 27 (61.4) | 95.5 | 45 | 4.56 | 0.58 | ||
| Manages simulator physiologic responses based on learner actions or interventions. | 1 (2.3) | 3 (6.8) | 13 (29.6) | 27 (61.4) | 90.9 | 44 | 4.5 | 0.73 | |
| Executes hybrid simulation scenarios that include full-body simulators, standardised patients, task trainers and other simulation technology. | 1 (2.3) | 4 (9.1) | 13 (29.6) | 26 (59.1) | 88.7 | 44 | 4.45 | 0.76 | |
| Operates simulators for special population scenarios outside the simulation lab (emergency medical services, disaster, mass casualty, and so on). | 1 (2.3) | 5 (11.4) | 20 (45.5) | 18 (40.9) | 86.4 | 44 | 4.25 | 0.75 | |
| Element 2: Demonstrates ability to programme simulators. | |||||||||
| Ability to programme user interfaces for full-body simulators for a variety of clinical scenarios. | 2 (4.5) | 5 (11.4) | 15 (34.1) | 22 (50.0) | 84.1 | 44 | 4.29 | 0.85 | |
| Ability to customise preprogrammed scenarios to include appropriate branch points to match learning objectives. | 2 (4.5) | 1 (2.3) | 20 (45.5) | 21 (47.7) | 93.2 | 44 | 4.36 | 0.75 | |
| Element 3: Demonstrates ability to perform basic simulator maintenance. | |||||||||
| Ability to perform basic maintenance and repairs on a variety of full-body simulators (eg, hardware issues related to pneumatics, basic electrical connections, switches and hoses). | 1 (2.3) | 8 (18.6) | 6 (14.0) | 13 (30.2) | 15 (34.9) | 65.1 | 43 | 3.77 | 1.19 |
| Ability to troubleshoot software connections including Wi-Fi and simulator direct connections resulting from Internet Protocol (IP) connections and media access control (MAC) addresses. | 1 (2.3) | 2 (4.7) | 7 (16.3) | 16 (37.2) | 17 (39.6) | 76.8 | 43 | 4.07 | 0.99 |
| Ability to perform basic maintenance and repairs on task trainers (eg, replacement of tubes, skins and simulated blood). | 1 (2.3) | 3 (7.0) | 5 (11.6) | 14 (32.6) | 20 (46.5) | 79.1 | 43 | 4.14 | 1.04 |
| Element 4: Demonstrates ability to appropriately set up and break down simulation equipment. | |||||||||
| Demonstrates ability to set up standardised patient environment and scenarios. | 4 (9.5) | 24 (57.1) | 14 (33.3) | 90.5 | 42 | 4.24 | 0.62 | ||
| Demonstrates ability to set up multipatient scenarios, hybrid simulation scenarios and special population scenarios. | 6 (14.3) | 22 (52.4) | 14 (33.3) | 85.7 | 42 | 4.19 | 0.67 | ||
| Demonstrates ability to perform moulage on simulators and standardised patients. | 3 (7.1) | 7 (16.7) | 15 (35.7) | 17 (40.5) | 76.2 | 42 | 4.1 | 0.93 | |
| Element 5: Demonstrates ability to provide audiovisual support for simulation scenarios. | |||||||||
| Demonstrates ability to set up and operate audiovisual equipment and software. | 1 (2.4) | 1 (2.4) | 8 (19.1) | 15 (35.7) | 17 (40.5) | 76.2 | 42 | 4.1 | 0.96 |
| Demonstrates ability to set up and operate learning management systems. | 2 (4.8) | 3 (7.1) | 6 (14.3) | 18 (42.9) | 13 (31.0) | 73.8 | 42 | 3.88 | 1.09 |
| Demonstrates ability to input assessment tools and use data analytics functions within learning management systems. | 5 (11.9) | 11 (26.2) | 16 (38.1) | 10 (23.8) | 61.9 | 42 | 3.74 | 0.96 | |
| Demonstrates ability to troubleshoot problems with audiovisual equipment and software. | 1 (4.76) | 1 (2.4) | 6 (14.3) | 17 (40.5) | 16 (38.1) | 78.6 | 41 | 4.05 | 1.03 |
| Demonstrates ability to set up mobile audiovisual support for simulation scenarios performed outside the simulation lab (in situ simulations, emergency medical services field training, and so on). | 1 (2.4) | 3 (7.1) | 10 (23.8) | 13 (31.0) | 15 (35.7) | 66.7 | 42 | 3.9 | 1.05 |
| Rating scale | 2 (4.8) | 3 (7.1) | 4 (9.5) | 16 (38.1) | 17 (40.5) | 78.6 | 42 | 4.02 | 1.12 |
A+SA, agree+strongly agree; D, disagree; N, neutral; SA, strongly agree; SD, strongly disagree.
Discussion
The initial assessment tool was revised and refined resulting in a final tool reaching consensus among respondents (figure 1). The tool began with five elements and 23 subitems and was revised to five elements and 16 subitems with four optional subitems. Previous scholars have identified the core simulation fellowship curricular content that was used to develop the initial technical evaluation tool.10–12 15 In order to gain expert consensus on the assessment tool, the approach outlined by Cumyn and Harris for content validation of curriculum consensus guidelines using Delphi methodology was employed successfully.16
Figure 1.
E-TeCHS tool—final.
Feedback from respondents after the first survey was used to modify the initial assessment tool. For example, within the element, demonstrates ability to operate software during simulations, the subitems were modified based on qualitative feedback to clarify terms and include specifics about the meaning of full-body simulators and ‘on the fly’ operation. The terms were modified based on definitions provided in the Simulation Healthcare Dictionary.18 In a later element, demonstrates ability to appropriately set up and break down simulation equipment, two subitems met consensus after the first survey. These subitems were selects appropriate simulation equipment to pair with scenario objectives and demonstrates ability to set up for in situ simulation scenarios. Both of these subitems are foundational subitems for simulation fellows. Frallicciardi et al list pairing appropriate simulation equipment with scenario objectives and demonstrating proficiency in the application of in situ simulation as core simulation fellowship curricular elements in the domain of technical operations and techniques.12 Lastly, the overarching rating scale for all elements, originally a 7-point Likert scale, was revised to a 5-point Likert scale. Simulation SMEs felt the 7-point scale had too many rating descriptors and was harder to use than a 5-point scale.
The revised assessment tool, which had five elements and 20 subitems, was reassessed in a second survey. Consensus was reached on 13 out of the 17 subitems that had not previously reached consensus, with consensus on the revised 5-point Likert rating scale, as well. The four subitems that did not meet consensus after two rounds of modified Delphi methodology included: performing maintenance and repairs on simulators; operating learning management systems; inputting assessment tools; and using data analytics functions within learning management systems and set-up for mobile audiovisual support for simulation scenarios performed outside the simulation lab. These subitems were kept as optional subitems within the survey as the qualitative results show these subitems to be more specific to a simulation technician skill set (figure 1). By having these optional subitems in the final assessment tool, this tool can potentially be used to evaluate both simulation fellows and simulation technicians, providing wider utilisation within the simulation community.
Even though simulation fellowships remain greatly varied in duration and curriculum, scholars have begun to define a common core curriculum.11 12 Simulation fellowship directors have only a few validated tools to assess the competency of simulation fellows in a very limited number of domains.13 14 One of the seven domains of core curricular content is competency in technical skills.12 By using the Evaluation of Technical Competency in Healthcare Simulation (E-TeCHS) assessment tool to give feedback, fellowship directors have the ability to identify deficits and focus training to improve performance. As the tool is used for summative evaluations, we plan to use the tool in upcoming studies to gain additional validation data.
As graduate medical education moves towards a competency-based medical education (CBME) model, the importance of validated assessment tools to provide feedback is paramount. CBME requires more frequent feedback with validated assessment tools.19 Frallicciardi et al reflect this when they identify program administration domains for simulation fellowships.12 One of the two domains is the assessment of the fellow. This includes the fellowship director providing assessments at least twice per year and tracking progress through fellowship.12 The E-TeCHS tool has the ability to track progress of competency in technical skills and be used for biannual summative feedback. This tool can be used to further develop a consensus curriculum for competency in technical skills for simulation fellows and potentially be used by simulation directors to guide job descriptions for simulation faculty and simulation technician positions.
The limited initial consensus may have been due to the lack of a standardised simulation fellowship curriculum and varied backgrounds of respondents. After the first round of the modified Delphi methodology, consensus was only reached on three subitems out of a total of 23 subitems. Simulation fellowships are non-ACGME accredited, allowing fellowship directors to individually tailor the fellowship experience.11 Depending on the duration of a simulation fellowship, a fellow may not have time to master all domains of the simulation curriculum.12 For example, fellowship directors may want to focus on competency in debriefing and curriculum development, while not placing an emphasis on technical skills. In addition, survey participants were either simulation fellowship faculty or simulation technicians. This divide may have led respondents to view competency in technical skills differently.
Limitations
This study has several limitations. Although we were able to reach over 70 different simulation fellowship directors and technicians, not all fellowship programmes were represented in this study. The exact number of simulation fellowship programmes is unknown, as there is a continuously expanding number of fellowship programmes and no comprehensive list of programmes exists. The survey generalisability is limited as we had only 74 respondents on the first survey, which likely represents a small percentage of the total number of potential participants in North America. Furthermore, the second survey had only 45 respondents, as it could only be distributed to individuals who completed the first survey. Lastly, the qualitative inductive process is subjective and may introduce potential bias.
Conclusion
The E-TeCHS tool provides an instrument developed from a national consensus of content experts. This tool provides simulation fellowship directors a method to evaluate fellows’ technical competency. Additional research using the E-TeCHS tool is needed to further validate the ability to measure technical competence of medical simulation fellows.
Footnotes
Contributors: PGH, SSA, MFB, MRJ and RAA all made substantial contributions to the design of the work, acquisition and interpretation of data for the work, drafting the work and revising it critically for important intellectual content, final approval of the version to be published and agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy of any part of the work are appropriately investigated.
Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests: None declared.
Provenance and peer review: Not commissioned; externally peer reviewed.
Patient consent for publication: None declared.
References
- 1. Levine AI, DeMaria S, Schwartz AD, et al. The comprehensive textbook of healthcare simulation. New York, NY: Springer, 2013. [Google Scholar]
- 2. Okuda Y, Bryson EO, DeMaria S, et al. The utility of simulation in medical education: what is the evidence? Mt Sinai J Med 2009;76:330–43. 10.1002/msj.20127 [DOI] [PubMed] [Google Scholar]
- 3. Okuda Y, Bond W, Bonfante G, et al. National growth in simulation training within emergency medicine residency programs, 2003-2008. Acad Emerg Med 2008;15:1113–6. 10.1111/j.1553-2712.2008.00195.x [DOI] [PubMed] [Google Scholar]
- 4. Passiment M, Sacks H, Huang G. Medical Simulation in Medical Education: Results of an AAMC survey. 2011https://www.aamc.org/download/259760/data (Cited 02 Jul 2018).
- 5. Barsuk JH, Cohen ER, McGaghie WC, et al. Long-term retention of central venous catheter insertion skills after simulation-based mastery learning. Acad Med 2010;85(10 Suppl):S9–S12. 10.1097/ACM.0b013e3181ed436c [DOI] [PubMed] [Google Scholar]
- 6. Wayne DB, Didwania A, Feinglass J, et al. Simulation-based education improves quality of care during cardiac arrest team responses at an academic teaching hospital: a case-control study. Chest 2008;133:56–61. 10.1378/chest.07-0131 [DOI] [PubMed] [Google Scholar]
- 7. Wayne DB, Butter J, Siddall VJ, et al. Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. J Gen Intern Med 2006;21:251–6. 10.1111/j.1525-1497.2006.00341.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Kessler DO, Auerbach M, Pusic M, et al. A randomized trial of simulation-based deliberate practice for infant lumbar puncture skills. Simul Healthc 2011;6:197–203. 10.1097/SIH.0b013e318216bfc1 [DOI] [PubMed] [Google Scholar]
- 9. Zendejas B, Cook DA, Hernández-Irizarry R, et al. Mastery learning simulation-based curriculum for laparoscopic TEP inguinal hernia repair. J Surg Educ 2012;69:208–14. 10.1016/j.jsurg.2011.08.008 [DOI] [PubMed] [Google Scholar]
- 10. Ahmed RA, Frey JA, Hughes PG, et al. Simulation Fellowship Programs in Graduate Medical Education. Acad Med 2017;92:1214. 10.1097/ACM.0000000000001780 [DOI] [PubMed] [Google Scholar]
- 11. Ahmed RA, Frey J, Gardner AK, et al. Characteristics and core curricular elements of medical simulation fellowships in North America. J Grad Med Educ 2016;8:252–5. 10.4300/JGME-D-15-00276.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Frallicciardi A, Vora S, Bentley S, et al. Development of an emergency medicine simulation fellowship consensus curriculum: Initiative of the society for academic emergency medicine simulation academy. Acad Emerg Med 2016;23:1054–60. 10.1111/acem.13019 [DOI] [PubMed] [Google Scholar]
- 13. Brett-Fleegler M, Rudolph J, Eppich W, et al. Debriefing assessment for simulation in healthcare: development and psychometric properties. Simul Healthc 2012;7:288–94. 10.1097/SIH.0b013e3182620228 [DOI] [PubMed] [Google Scholar]
- 14. Arora S, Ahmed M, Paige J, et al. Objective structured assessment of debriefing: bringing science to the art of debriefing in surgery. Ann Surg 2012;256:982–8. 10.1097/SLA.0b013e3182610c91 [DOI] [PubMed] [Google Scholar]
- 15. Bailey R, Taylor RG, FitzGerald MR, et al. Defining the simulation technician role: Results of a survey-based study. Simul Healthc 2015;10:283–7. 10.1097/SIH.0000000000000103 [DOI] [PubMed] [Google Scholar]
- 16. Cumyn A, Harris IB. A comprehensive process of content validation of curriculum consensus guidelines for a medical specialty. Med Teach 2012;34:e566–e572. 10.3109/0142159X.2012.668623 [DOI] [PubMed] [Google Scholar]
- 17. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74. 10.2307/2529310 [DOI] [PubMed] [Google Scholar]
- 18. Lopreiato JO. Healthcare Simulation Dictionary. Rockville, MD: Agency for Healthcare Research and Quality, 2016:16;0043. AHRQ Publication No. [Google Scholar]
- 19. Holmboe ES, Sherbino J, Long DM, et al. The role of assessment in competency-based medical education. Med Teach 2010;32:676–82. 10.3109/0142159X.2010.500704 [DOI] [PubMed] [Google Scholar]