More Is Not Always Better in Simulation. Learners’ Evaluation of a “Chest Model”

Abstract

Background: Fidelity in simulation is an important design feature. Although it is typically seen as bipolar (i.e., “high” or “low”), fidelity is actually multidimensional. There are concerns that “low fidelity” might impede the immersion of learners during simulation training. “Locally built models” are characterized by decreased cost and reduced “structural” fidelity (how the simulator looks) while satisfying “functional” fidelity (what the simulator does).

Objective: To 1) describe the use of a locally built chest tube model in building a mastery-based simulation curriculum and 2) describe evaluation of the model from learners in different stages and contexts.

Methods: The model was built on the basis of key functional features of the assigned training task. A curriculum that combined progressive difficulty and opportunities for deliberate practice and mastery was developed. An analysis of the learner’s survey responses was performed using SAS studio (SAS Software).

Results: We describe the process of creating the chest tube model and a curriculum in which the model is used for increasing levels of difficulty to reach skill mastery. Learners at different stages and in different contexts, such as practicing physicians and trainees from developed and developing countries, evaluated the model similarly. We provide validity evidence for the content, response process, and relationship with other variables when using the model in the assessment of chest tube insertion skills.

Conclusion: As demonstrated in our chest tube critical care medicine curriculum, the locally built models are simple to build and feasible to use. Contrary to current thinking that low-fidelity models might impede immersion in simulation training for experienced learners, the survey results show that different learners provide very similar evaluations after practicing with the model.

Data for the effectiveness of simulation training are abundant, to the point that some have deemed the absence of simulation training unethical (1–4). Acute lifesaving procedures, like thoracotomy, are characterized by their low frequency (i.e., they are less commonly indicated) and critical need for competence (i.e., they are acute and time sensitive, and the initial attempts must be successful). Therefore, simulation is an attractive training tool for such procedures.

Fidelity is an important design feature, and although it is thought of as bipolar (high or low), it is multidimensional and varies with the learning context, objectives, trainee, educator, learner, and task (5). Often, fidelity is simplified across two dimensions: structural (how the simulator appears) and functional (what the simulator does). The structural dimension is often accorded much more importance, but functional fidelity is more important for actually achieving learning goals (5).

Every teacher has had the experience of improvising a simple model to teach learners a variety of tasks, especially in the current context of many learners and shorter duty hours (6). Some teachers have even embarked on building a model locally. A “locally built model,” in which local experts and educators identify key functional parameters of the task and then build simple models to fit the structure and resources of their institution, offers the following benefits: decreasing cost, producing a number of models, and allowing easy access for training and making iterative changes on feedback while satisfying functional fidelity. If needed, these simple models can be integrated in the curricula with other existing mannequins or simulated patients to yield increasing levels of task difficulty to match the learner’s increasing expertise (7).

Several groups have already created their own models to practice the skill of chest tube insertion by targeting the two most challenging steps (intercostal space dissection and advancing through different tissues) or including the two most essential elements, the ribs and overlying layer (8–10), with very good feedback from participants. However, these groups did not describe the process by which the design was created or how they integrated their task trainer in the curriculum, and the feedback was mainly from trainees.

Although there is abundant evidence for the effectiveness of training with simple models, especially for novice learners, there are also concerns that locally built models are too simple and may impact suspension of disbelief and the learner’s immersion in the simulation training, especially for advanced learners (11–14).

The objectives of this study were to:

• •Describe the process of creating a locally built model for chest tube insertion and identifying how the task trainer is integrated in a mastery-based simulation curriculum that enables progressive levels of difficulty for programs with other existing resources (complex mannequins and simulated participants).
• •Describe and compare evaluation of the model from different levels of learners in different contexts.

Methods

Locally Built Model

An expert group of 12 faculty defined key functional features of the task, including palpation of the ribs and intercostal spaces by mimicking chest wall thickness and parietal pleura, the presence of fluid or air under pressure with aspiration, and avoidance of the neurovascular bundle as well as important structures, such as the heart, liver, and spleen. A low-cost model to accommodate the above was built by technicians in our institution.

Integrating the Model into the Curricula

To integrate the model into a curriculum and accommodate different levels of learners, we created a curriculum that combines progressive levels of difficulty and opportunities for deliberate practice and feedback. The curriculum was based on educational learning theories such as mastery-based learning (trainees must achieve a certain proficiency in a task before moving on to the next and must meet the same objectives with varied learning time) (15), deliberate practice (individualized training designed by a teacher to improve performance through repetition) (16), and the challenge-point framework (for maximal learning to occur, the challenge of the task should be appropriate to the level of the learner) (7).

In accordance with the important element of Mastery-Based Curricula, at the end of the simulation, training participants would be assessed with direct observation of procedural skills (DOPS), a global rating scale that has published validity evidence (17, 18) and encompasses technical skill assessment as well as pre- or postprocedural management. Twelve faculty agreed that a minimum passing score (MPS) would be a minimum score of 5 in all 11 elements of the DOPS. If a trainee did not meet the MPS, they would be provided with more time in the simulation setting as facilitated by an instructor.

Evaluation of the Model

To explore how different type of learners would evaluate the model, we created a survey to target: functional fidelity (questions 1–3), structural fidelity (questions 4 and 5), and overall experience (questions 6–8). The survey was then distributed to 110 learners who used the model. Their responses to the survey questions were collated into three categories: “unsatisfactory” and “needs improvement” were considered one category, “good” remained “good,” and “very good” and “outstanding” were treated as one category. The categorical responses of the three groups of physicians to each question were compared using the chi-square test and P value reported. All analyses were done in SAS studio software version 9.4 (SAS Software). We also looked at whether all participants would continue to train in the program until they had reached the MPS.

Results

Locally Built Model

The “chest model” was composed of several layers simulating part of the chest wall and hard plastic representing the ribs. The inside was filled with a foam structure (lung) into which a glove containing fluid or air could be placed to represent a pleural effusion or pneumothorax. The model was covered with a deep elastic layer representing the parietal pleura, on top of which was placed a spongy layer representing subcutaneous fat, and the topmost layer was a thin cover representing skin. A slim tube filled with red liquid mimicking the neurovascular bundle ran underneath each upper rib (Figure 1).

Figure 1.

Locally built chest tube model. (A) Part of the chest wall with plastic ribs and two straps to connect the model to a simulated patient or mannequin. (B) Addition of the fluid filled tubing to mimic the neurovascular bundle. (C) Addition of the most superficial (closest to the rib cage) glove filled with fluid or air under tension to mimic pleural effusion or pneumothorax. (D) Addition of a dark fluid–filled glove in the inferior aspect to mimic the spleen or liver. (E) Addition of large glove filled with air to mimic the lung. (F) Addition of the cover to mimic chest wall. (G) Layer mimicking chest-wall components from the most superficial to the deepest: thin layer for skin, foamy layer for subcutaneous tissue, and elastic green tough layer mimicking parietal pleura. (H and I) The model is combined with a high-fidelity mannequin to create a scenario of an intubated and ventilated child with pneumothorax.

Thoracotomy Curricula

The simulation program included a didactic element dealing with important anatomy, indications/contraindications, and an overview of the procedure. Trainees would undertake simulation practice and then practice with an instructor who would provide confidential, individualized feedback, followed by independent practice on the simulator.

When trainees had achieved competence performing the procedure independently with the part task trainer, they would practice the skills at a simulated bedside using two scenarios featuring “hybrid simulation.” In the first scenario—tension pneumothorax, an emergency—the part task trainer would be connected to a high-fidelity mannequin (Medical Education Technologies Inc.), representing a sedated, intubated, and mechanically ventilated patient. In the second scenario—drainage of empyema, a semielective procedure—the part task trainer would be attached to a standardized patient. Taken together, the trainee completing these two tasks would demonstrate competence in many dimensions separate from the procedure itself (Figure 2).

Figure 2.

Critical care medicine “Chest Tube Skills” curriculum. After the in-class/didactic portion of curriculum and having successfully completed their MCQ test, the participants progress to the simulation training. The simulation initially focuses on procedural skills only (locally built model) and then progresses to participants having to manage two scenarios (model is combined with a simulated patient and a high-fidelity mannequin). After successful completion of simulation training, participants perform the procedures at bedside while supervised. DOPS = direct observation of procedural skills; ICU = intensive care unit; MCQ = multiple-choice question.

The chest model was used to train critical care medicine and pediatric trainees at our institution as well as practicing physicians at two conferences (training was provided as a daylong course during the preconference events). All training sessions were instructor-led and initially included small groups, with time left for self-practice in between. Critical care medicine and pediatric trainees in our institution had no prior experience with chest tube insertion. We did not collect data on the previous experience of practicing physicians. A satisfactory completion was determined when the participant reached the MPS. Practicing physicians and pediatric residents all practiced in simulation and achieved the MPS goal. Practicing physicians reached the MPS at their first attempt. Trainees reached the MPS with variable time, but the time taken or the number of assessment to reach the MPS was not recorded. Pediatric intensive care unit (PICU) trainees (30) completed all stages of the curriculum, including supervised practice, and all of them reached the MPS. Their performance in real life was not recorded as part of this study. All instructors reported that the simulation curriculum represented all aspects of the procedure for assessment with DOPS.

The most common technique practiced was the Seldinger technique. Only some of the PICU trainees (10 of 30) practiced the surgical technique.

Evaluation of the Chest Model

The learners were asked to fill out an anonymous survey about the task trainer to determine whether the model was acceptable to learners and how they evaluated it, despite being a locally built model with limited structural fidelity. The survey was completed at the end of the simulation training, after all participants had had an opportunity to train through all the stages of the curriculum. None of the participants refused to complete the training because of the simple model or quit before reaching the MPS.

A total of 110 learners were asked to fill out the anonymous survey; all completed the survey. They were divided in three groups: practicing physicians in Canada (16), trainees in Canada (66), and practicing physicians in developing countries (28). All groups engaged with the model and evaluated it similarly (Table 1). There were no main differences between groups in terms of structural or functional fidelity or overall experience with the model. Practicing physicians in developing countries evaluated the structural fidelity as worse than the functional fidelity. We did not separate the data for the 10 trainees who practiced the surgical technique but were reassured that as a group, none of the trainees evaluated the model as “unsatisfactory/needs improvement” in all its aspects.

Table 1.

Survey results

Question	Physicians Practicing in Developing Countries (n = 28)	Physicians Practicing in Canada (n = 16)	Residents Practicing in Canada (n = 66)	P Value
Permits initial development of skills for placement of a chest tube				0.06
Unsatisfactory/needs improvement	0	6.3	0
Good	32.1	25	15.2
Very good/outstanding	67.9	68.8	84.8
Has necessary elements for learning the skills				0.02
Unsatisfactory/needs improvement	0	12.5	0
Good	21.4	12.5	9.1
Very good/outstanding	78.6	75	89.4
Useful for maintaining the skills				0.08
Unsatisfactory/needs improvement	10.7	0	0
Good	21.4	12.5	16.7
Very good/outstanding	67.9	81.3	83.3
Anatomic representation of rib cage				0.04
Unsatisfactory/needs improvement	3.6	25	4.5
Good	39.3	12.5	25.8
Very good/outstanding	57.1	62.5	74.2
Sufficient realism for training				0.35
Unsatisfactory/needs improvement	3.6	12.5	3
Good	35.7	18.8	24.2
Very good/outstanding	60.7	68.8	71.2
I would continue to use it in the future				0.11
Unsatisfactory/needs improvement	7.1	6.3	0
Good	21.4	25	15.2
Very good/outstanding	71.4	68.8	80.3
Confidence doing the procedure in a patient				0.30
Unsatisfactory/needs improvement	0	12.5	3
Good	32.1	18.8	22.7
Very good/outstanding	67.9	68.8	72.7
Overall experience with the model				0.11
Unsatisfactory/needs improvement	0	6.3	0
Good	25	25	13.6
Very good/outstanding	75	68.8	87.9

Questions 1–3 target functional fidelity, questions 4 and 5 target structural fidelity, and questions 6–8 target the overall experience with the model.

Discussion

As concerns about patient safety increase, so does the use of simulation for medical skills training. Different simulation models and various educational strategies are being employed to integrate simulation training with the real-life clinical environment_. Although the evidence to support the use of simulation in health care is overwhelming (1–4), important issues remain, such as acquisition and maintenance costs of high technology models as well as the provision of trained personnel to use such high technology. These issues are important, as cost will limit the number of, availability of, and access to simulators. For most centers, these are important considerations in the setting of budgetary constraints.

Norman and colleagues compared the learning benefits of high-fidelity and low-fidelity simulations (19). Although the gains of simulation were clear in all studies reviewed, no significant difference in learning, as measured by clinical performance, are apparent between high-fidelity and low-fidelity models.

The concept of fidelity in medical simulation is imprecise and confusing, and its classification (high vs. low) may be overly simplistic (5). Structural fidelity (or physical resemblance) does not always guarantee educational effectiveness, as the latter depends on the objectives of the training, the learner’s level, and the functional aspects of the simulator.

The purpose behind the locally built model was to fulfill the functional fidelity required for the chest tube insertion while minimizing time and cost and maximizing availability and access. Although several models have been described, the core contribution here is the process of building a model by asking experts to identify key functional features, acquiring feedback from different levels of learners in different contexts, and providing a curriculum map that increases the complexity of the training to match the trainee’s growing level of knowledge and expertise.

We also provide some validity evidence for using this chest model and our curricula to assess the skills of the chest tube placement. Using Mesick’s validity framework (20), we provide content validity (a group of experts decided the key functional features of the model and approved its use, and none of the users evaluated the chest model unsatisfactory in the survey completed at the end of the training), response process validity (the assessors could easily complete the assessment tools, as the model, especially when combined with other simulation technology, afforded all elements necessary to perform and assess the skill), and relationship with other variables validity (the faculty reached the MPS in only one attempt and trainees improved with practice). Evidence for internal structure (reliability studies were not conducted) and consequences (or predictive validity evidence) were not collected, but the literature shows that there is no linear relationship between simulation and real-life performance, as our performance is context-dependent, especially in the PICU, where multiple real-life stressors and acutely deteriorating patients coexist (1).

There are concerns that learners might not practice with simple simulation models. The participants of the training program found the simulator sufficiently realistic and useful to improve the skills of chest tube placement. When analyzing the trainees’ assessment of the model, the concordance of opinion among groups might be explained by the fact that once the functional fidelity is satisfied, learners treat structural issues in the same way. In findings similar to those of other studies, novice trainees needed variable time to reach the MPS, but that time was not recorded (21).

There are several limitations to this study. The survey is a subjective measure and does not guarantee the performance of participants in a real-life context, after being trained with our chest model. The fact that none of the participants refused to train with the simple chest model or discontinue training before reaching the MPS could be attributed to the fact that once enrolled in a curriculum, participants might feel “judged” if they discontinue the training. We did not record the variable time taken to reach the MPS from novice trainees. Lastly, the validity evidence presented here (content, response process, and relationship with other variables) is, like most published studies, limited (22).

Conclusions

Many programs struggle with financial constraints. We provide a summary of the process of building a simple simulation model and integrating the model into a simulation curriculum as well as limited validity evidence for using the model for assessment of these skills. We also demonstrate a detailed evaluation of both structural and functional fidelity.