Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2023 Oct 27.
Published in final edited form as: Proc Hum Factors Ergon Soc Annu Meet. 2022 Oct 27;66(1):1867–1871. doi: 10.1177/1071181322661243

Slow and Steady: Examining the impact of hands-on instructions on learnability of a training simulator to enhance development of core skills in Central Venous Catheterization

Haroula M Tzamaras 1, Dailen C Brown 2, Jessica M Gonzalez-Vargas 1, Jason Z Moore 2, Scarlett R Miller 1
PMCID: PMC10202001  NIHMSID: NIHMS1899672  PMID: 37223777

Abstract

Instructional design is the theory surrounding how learners perceive information and is prevalent in simulation-based medical education. Simulation is used for a variety of medical procedures including central venous catheterization (CVC). The dynamic haptic robotic trainer (DHRT) is a CVC teaching simulator developed to specifically focus on training the needle insertion portion of CVC. While the DHRT has been validated to teach CVC as well as other training methods, an opportunity was seen to redesign the instructions of the DHRT to increase the learnability of the system. A hands-on instructional walkthrough was designed. A group trained with the hands-on instructions was compared to a previous group to assess initial insertion performance. Results indicate that changing the instructional method to be hands-on may have an impact on system learnability and help reinforce development of core components of CVC.

INTRODUCTION

Hands-on learning and simulation has been proven to add value to curriculum for skill-training when learning aims are clear (Khaled et al., 2014). Hands-on training is highly relevant in medical education because it gives trainees the opportunity to learn as they do. Simulation-based learning has become more common over the last decade because it provides hands-on, experiential learning without putting patients at risk, eliminates the time constraints of trying to match a clinical schedule, and removes the burden of teaching from experienced clinicians (Datta et al., 2012; Maus, 2011).

Simulator learning is highly regarded in medical education. Many newer methods of non-instructor-based simulation have been proven to train users as well as traditional instructor-based methods. Two studies, one on dental education and one on surgical education, compared the learning gains of training groups that were trained on a traditional instructor-based trainer verses a newer non-instructor based simulator and both studies showed no significant differences in learning between the two groups (Chen et al., 2020; Plessas, 2017).

One example of a medical procedure where simulated learning is common and beneficial is central venous catheterization (CVC). CVC is a complex medical procedure where a doctor inserts a catheter into one of three central veins to facilitate medication delivery via direct access to the heart; the most common insertion vein is the internal jugular vein (IJV) (Taylor & Palagiri, 2007). CVC has a high associated complication rate which can be partially correlated to how well the person doing the procedure was trained (McGee & Gould, 2003). One complication of CVC is accidental arterial puncture, which means inserting the needle into the carotid artery instead of the vein when trying for venous access (McGee & Gould, 2003). One of the reasons for this is that the anatomical structure of the internal jugular vein and the carotid artery change from patient to patient, so the operator needs a strong understanding of how to differentiate between the two vessels. Traditional CVC training methods consist of a static manikin trainer with one anatomy and need an instructor to give feedback (Soffler et al., 2018). This type of training is useful, but lacks in patient anatomical structure variability, self-guided learning, and material durability.

To combat some of the issues associated with traditional CVC training, a group of researchers developed the dynamic haptic robotic trainer (DHRT) (D. Pepley, Yovanoff, Miller, et al., 2016). The DHRT is a haptic-based procedural simulator consisting of a haptic robotic arm and syringe, simulated ultrasound screen, and usable ultrasound probe to simulate needle insertion into the IJV. The most recent version of the DHRT can be seen in figure 1. To provide automated feedback, the DHRT system collects performance data on the angle of the needle during insertion (degrees), the distance from the needle to the center of the vein (cm), number of attempts (whole number), percent of time spent aspirating and if the trainee punctures through the backwall of the vein. In addition, the difficulty of the case is presented. Case difficulty is based on variables that impact how hard it would be for a doctor to insert a central line in a real patient case such as vein depth, skin thickness, vein diameter, and closeness between the IJV and the carotid artery (D. F. Pepley et al., 2017). This data is then aggregated into a final score as seen in Figure 2. The DHRT was developed to train the needle insertion portion of CVC for initial venous access and has been validated to train this as well as traditional manikin methods. (Yovanoff et al., 2016). The DHRT is currently being used in CVC training curriculum at multiple medical centers.

Figure 1:

Figure 1:

Open in a new tab

The most recent version of the DHRT system

Figure 2:

Figure 2:

Open in a new tab

The summary screen for scoring on the DHRT system

Medical procedures often have learning curves for novices which also apply to the simulators for these procedures (Madsen et al., 2014; D. Pepley, Yovanoff, Mirkin, et al., 2016). The DHRT is not an exception. One way to lessen the simulator learning curve is by improving the initial learnability of the system. As defined in a previous study on usability in medical training, learnability is how well a first-time user can understand and use the simulator (Saleem et al., 2007). For CVC, it is pertinent that through the instructional process trainees learn both the system and the procedure. As such, we decided to redesign the instructions of the DHRT system to better help the user understand the key steps of CVC while also optimizing their performance. Performance on the DHRT is measured each time a participant engages in a trial; their score changes based on how well they do with each variable improving individually. Over time, the hypothetical ideal score should show higher aspiration rates, lower distance to the center of the vein, lower number of attempts, no backwall punctures, and an angle between 30 and 45 degrees.

Instructional design is a body of knowledge consisting of theories surrounding how teaching procedures should be developed to maximize student learning (Molenda et al., 2003). One idea of how instructional design applies to medicine is through multimedia learning, or the cognitive theory surrounding how people perceive and learn information through words and images (Mayer, 2010). This type of learning is especially prevalent in simulation which heavily consists of simulated images and situations. Effective medical simulators have several instructional design features in common including repetitive practice, cognitive interactivity, and clinical variation (Cook et al., 2013). A paper on simulated learning for medical education argues that medical simulators must be catered to adult learners through ideas like controlling the sequence of tasks and offering guidance throughout the simulation (So et al., 2019).

The original instructions for the DHRT are one video that goes through the process of how to use the system without explaining each step. The main problem is that the instructions go over the entire procedure at once and do not provide room for the user to break down the steps to better their understanding. The purpose of this study was to increase the learnability of the DHRT system by developing a hands-on instructional walkthrough giving an interactive overview of the simulator and procedure, and then to validate the utility of this new method by comparing the first trial performance of a training group with the new method to one without.

RESEARCH OBJECTIVES

The purpose of this study was to determine if changing the instructional method for the DHRT system from one video to a hands-on instructional walkthrough impacts the learnability of the DHRT. Specifically, the study was designed to answer the following research questions (RQs):

RQ1: Can past experience, self-efficacy, and training type predict successful performance on the first insertion attempt?

Successful performance in this case refers to whether or not the participant was able to obtain venous access by the end of the trial without puncturing the carotid artery. As it is possible in the DHRT system to end a trial with venous access even after an arterial puncture, these participants will also be filtered out for this RQ because an arterial puncture is a severe complication of CVC that trainees should be trained to avoid (McGee & Gould, 2003). The primary objective of this question is to determine if changing the instructional type has an impact on the learnability of the system. Based on the intentions and the instructional design theories implemented into the hands-on training, the hypothesis is that training type will significantly contribute to successful performance.

RQ2: Can training group be predicted based on first insertion performance metrics?

The primary objective of this question is to determine if one group performed better than the other based on needle angle, distance to vein center, aspiration rate, and number of insertion attempts. Each of the variables of interest were either implicitly or explicitly trained in the hands-on, walkthrough-based training. As such, the hypothesis is that it should be possible to predict training group based on performance metrics.

METHODS

Two datasets were used for this research. The first dataset was taken from a study conducted at a medical center with the old instructional method in July 2021. The second dataset was taken from a study conducted at the same medical center with the new hands-on instructional method in January 2022.

Hands-On Instructions Development

In the literature a hands-on approach is effective in medical education for simulation-based learning (Cook et al., 2013; So et al., 2019). As such, this instructional method was applied to the DHRT system including proven methods such as specific task order and guidance throughout. The simulation was broken down into eight steps focusing on the mechanical and cognitive skills that residents most often struggle with on the DHRT. These skills include using the probe for ultrasound guidance, recognizing anatomical structures on the ultrasound, aspirating a syringe, tracking the needle on the ultrasound screen, and recognizing venous access.

The hands-on training starts with individual steps for needle tracking and centering, aspirating, and ultrasound usage and builds up to the steps being done together to help segment learning and mastery of each skill. For each step in the walkthrough training, a video explanation was provided and followed up with an activity assessment that the user had to go through based on the information in the video and the skill being targeted. The idea was that each step would be re-emphasized through the assessment and allow the user to learn and build as they go. While proper needle angle is not explicitly taught in this new system, it is implicitly taught through other exercises as certain angles are required to pass the activity correctly. An example of the perspective of the training video, activity, and type of feedback given in the walkthrough can be seen in Figure 3.

Figure 3:

Figure 3:

Open in a new tab

(left) Perspective of what the training videos look like during the walkthrough. (middle) An example of an activity in the hands-on walkthrough. (right) How feedback is provided to make the learning hands-on

Participants

The original instructions cohort, referred to from now on as the control group, from July 2021, consisted of 17 participants (NC=17); 5 had never had any sort of training in CVC. The hands-on instructions cohort, referred to from now on as the intervention group, consisted of 14 participants (NI=14); 7 had never had any sort of training in CVC. A summary of demographic data for both groups can be seen in Table 1.

Table 1:

Demographic Summary

Control Group Intervention Group
Gender
Male 11 3
Female 6 11
Prior Training
Yes 12 7
No 5 7
Race/Ethnicity
White 7 13
Asian 4 0
Hispanic 2 0
Other 4 1
Open in a new tab

Procedure

Participants in the control group underwent an online training for CVC prior to attending the study. Upon arrival, participants consented to being part of the study and filled out a pre-training self-efficacy form for how confident they felt in their abilities to conduct CVC. Then they watched a short video explanation of the DHRT and did six trials on the simulator. Finally, they filled out a post-training self-efficacy form. Participants in the intervention group underwent the same procedure except instead of a single video explanation, they went through the new, hands-on walkthrough of the DHRT. Figure 4 shows the overall procedural flow.

Figure 4:

Figure 4:

Open in a new tab

Diagram summarizing the procedural flow

RESULTS AND DISCUSSION

Simulator data was saved for each participant. The data was used to answer the research questions in the remainder of this section. All statistics were analyzed with SPSS (v. 27.0) assuming a significance level of 0.05.

RQ1: Can past experience, self-efficacy, and training type predict successful performance?

The objective of RQ1 was to determine if we could predict if a person’s first trial insertion score would be a pass based on exposure to previous CVC training, pre-training self-efficacy, and training group. Training group in this case means whether they were in the control or intervention group. To have a successful insertion, a participant must both have ended the trial in the vein and not have punctured the carotid artery. An arterial puncture, ending the procedure when the needle is outside of the vein, or both, were counted as failures. This is depicted in the waffle chart in Figure 5.

Figure 5:

Figure 5:

Open in a new tab

Visual depiction of first insertion failures

Based on the first insertion failure data, 57% of the intervention group had a successful insertion on their first trial whereas only 47% of the control group did. This is promising because it indicates that the hands-on instructional method may train participants better for obtaining venous access. Interestingly, from Figure 5 it can be observed that more people in the intervention group had a failure due to both arterial puncture and ending out of the vein. What this could potentially indicate is that people in this group were more likely to recognize an arterial puncture and end the simulation without continuing to try for venous access, which is the safer thing to do during an actual procedure if this complication arises. Considering the uneven sample sizes, more research would need to be done to validate this claim; however, it is a promising observation that the hands-on training method may better prepare participants to handle an arterial complication.

Additionally, a binomial logistic regression was performed to further answer this question. In this regression model, there was one dichotomous dependent variable being predicted based on one continuous independent variable and two nominal independent variables. Linearity of the continuous variable with respect to the logit of the dependent variable was assessed via the Box-Tidwell procedure which found that the continuous variable was linearly related to the logit of the dependent variable. All other assumptions were met and the model was constructed accordingly. The area under the ROC curve was .675 (95% CI, .481 to .869) which is not quite an acceptable level of discrimination, however, because the acceptable range starts at .7 and the data was close, the analysis proceeded with the model regardless. The logistic regression model and variables in the regression equation were not statistically significant, (χ2(3) = 2.242, p = .524), see Table 2.

Table 2:

Logistic regression predicting successful first insertion based on pre-procedural self-efficacy, past CVC training, and training type

B SE Wald df p Odds Ratio 95% CI for Odds Ratio
Lower Upper
Pre-Efficacy .738 .588 1.58 1 .210 2.092 .660 6.628
Past Training .758 .842 .811 1 .368 2.134 .410 11.11
Training Type .028 .800 .001 1 .972 1.028 .214 4.928
Constant −2.11 1.61 1.72 1 .189 .122
Open in a new tab

While it cannot be concluded from this analysis that those in the intervention group were more likely to be successful in their first insertion attempt, the odds ratio of one training group doing better than the other is close to 1, meaning there is no statistical difference in likelihood of a successful first insertion based on training group according to this data. Further research will be conducted to determine if with a larger sample size, training type will be able to predict successful first insertion.

RQ2: Can training group be predicted based on first insertion performance metrics?

The objective of RQ2 was to determine if out of those who were successful on their first trial, performance metrics differ based on instructional group. The performance metrics of interest for this question were distance to vein center, needle angle, aspiration rate, and number of insertion attempts. Any participants who did not achieve successful venous access and/or punctured the carotid artery were removed for this analysis. The control group was the positive predicted value and the intervention group was the other value. In this regression model, there was one dichotomous dependent variable being predicted based on one nominal independent variable and three continuous independent variables. Linearity of the continuous variables with respect to the logit of the dependent variable was assessed via the Box-Tidwell procedure which found that all continuous variables were linearly related. All other assumptions were met and the regression was performed. It should be noted that in this analysis, angle was reported as a categorical variable, either in range or out of range. This distinction resulted in the in-range group only having 5 cases, a sample size that is not ideal for logistic regression.

The logistic regression model was statistically significant (χ2(4) = 10.604, p = .031). The model explained 64.6% of the variance in the model and correctly classified 75% of cases. Sensitivity was 87.5% and specificity is 62.5%. The positive predictive value was 70% and the negative predicted value was 83%. Of the predicted variables, none were statistically significant, as seen in Table 3. However, from the odds ratios an increased aspiration rate showed that the odds of being in the control group decreased, with an odds ratio of .983. This indicates that a better aspiration rate was more related to the intervention. The odds ratios also indicated that the chances of being in the control group decrease with an increase in number of insertions attempts and distance to vein center. This finding indicates that these performance metrics may not be greatly improved by the new, hands-on walkthrough training. Further investigation is needed with larger sample sizes to validate these results.

Table 3:

Logistic regression predicting training group based on first insertion performance metrics

B SE Wald df p Odds Ratio 95% CI for Odds Ratio
Lower Upper
Angle −22.744 17135.452 .000 1 .999 0 0 -
Aspiration Rate −0.017 .025 .479 1 .489 .983 .936 1.032
Number Attempts −1.401 1.571 .796 1 .372 .246 .011 5.35
Distance to Vein Center −1.428 6.313 .051 1 .821 .240 0 56640.471
Constant 4.936 4.649 1.127 1 .288 139.163
Open in a new tab

CONCLUSION

The main findings from this research are that the hands-on simulator likely contributes to an increase in system learnability. While further validation is needed, this finding is important because it indicates that breaking down a medical procedure into steps for the instructions is better for learning the simulation and the procedure verses having just one instructional video. This knowledge is beneficial to the engineering and human factors communities because it is aiming to determine the best way for designers to create systems for medical training while optimizing learning gains.

This study had several limitations that are important to note and may have impact on results. One limitation is the differences in medical background between each group. Prior knowledge of CVC varies between people, and it is possible that this impacts simulator usage and skill acquisition in a way that we were unable to control for in the analysis. Another limitation is sample size. The control group had a sample size of 17 and the intervention group had a sample size of 14, with only 9 people completing all 6 trials. It is difficult to compare these groups for meaningful results. Future research will test these differences in instructional design methods with a larger sample size to further validate these findings.

ACKNOWLEDGEMENTS

This work was supported by the national Heart, Lung, and Blood Institute of the National Institutes of Health (NIH) under Award Number RO1HL127316. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Coauthors Dr. Moore and Miller owns equity in Medulate, which may have a future interest in this project. Company ownership has been reviewed by the University’s Individual Conflict of Interest Committee.

REFERENCES

  1. Chen HE, Sonntag CC, Mirkin KA, Pepley DF, Han DC, Moore JZ, & Miller SR (2020). From the simulation center to the bedside: Validating the efficacy of a dynamic haptic robotic trainer in internal jugular central venous catheter placement. American Journal of Surgery, 219(2), 379–384. 10.1016/j.amjsurg.2019.10.026 [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Cook DA, Hamstra SJ, Brydges R, Zendejas B, Szostek JH, Wang AT, Erwin PJ, & Hatala R (2013). Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis. In Medical Teacher (Vol. 35, Issue 1). 10.3109/0142159X.2012.714886 [DOI] [PubMed] [Google Scholar]
  3. Datta R, Upadhyay KK, & Jaideep CN (2012). Simulation and its role in medical education. Medical Journal Armed Forces India, 68(2), 167–172. 10.1016/S0377-1237(12)60040-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Khaled A, Gulikers J, Biemans H, van der Wel M, & Mulder M (2014). Characteristics of hands-on simulations with added value for innovative secondary and higher vocational education. Journal of Vocational Education and Training, 66(4), 462–490. 10.1080/13636820.2014.917696 [DOI] [Google Scholar]
  5. Madsen ME, Konge L, Nørgaard LN, Tabor A, Ringsted C, Klemmensen AK, Ottesen B, & Tolsgaard MG (2014). Assessment of performance measures and learning curves for use of a virtual-reality ultrasound simulator in transvaginal ultrasound examination. Ultrasound in Obstetrics and Gynecology, 44(6), 693–699. 10.1002/uog.13400 [DOI] [PubMed] [Google Scholar]
  6. Maus TM (2011). Simulation: The importance of “hands-on” learning. Journal of Cardiothoracic and Vascular Anesthesia, 25(2), 209–211. 10.1053/j.jvca.2011.01.007 [DOI] [PubMed] [Google Scholar]
  7. Mayer RE (2010). Applying the science of learning to medical education. Medical Education, 44(6), 543–549. 10.1111/j.1365-2923.2010.03624.x [DOI] [PubMed] [Google Scholar]
  8. McGee DC, & Gould MK (2003). Complications of Central Venous Catheterization. The New England Journal of Medicine, 348(12), 1123–1133. 10.1016/j.jamcollsurg.2007.01.039 [DOI] [PubMed] [Google Scholar]
  9. Molenda M, Reigeluth C, & Nelson LM (2003). Instructional Design. Encyclopedia of Cognitive Science, 574–578. [Google Scholar]
  10. Pepley DF, Gordon AB, Yovanoff MA, Mirkin KA, Miller SR, Han DC, & Moore JZ (2017). Training Surgical Residents With a Haptic Robotic Central Venous Catheterization Simulator. Journal of Surgical Education, 74(6), 1066–1073. 10.1016/j.jsurg.2017.06.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Pepley D, Yovanoff M, Miller S, & Han D (2016). A Virtual Reality Haptic Robotic Simulator for Central Venous Catheterization Training. Journal of Medical Devices, 10(September), 1–2. 10.1115/1.4033867 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Pepley D, Yovanoff M, Mirkin K, Han D, Miller S, & Moore J (2016). Design of a virtual reality haptic robotic central venous catheterization training simulator. Proceedings of the ASME Design Engineering Technical Conference, 5A-2016, 1–8. 10.1115/DETC2016-59560 [DOI] [Google Scholar]
  13. Plessas A (2017). Computerized Virtual Reality Simulation in Preclinical Dentistry: Can a Computerized Simulator Replace the Conventional Phantom Heads and Human Instruction? Simulation in Healthcare, 12(5), 332–338. 10.1097/SIH.0000000000000250 [DOI] [PubMed] [Google Scholar]
  14. Saleem JJ, Patterson ES, Militello L, Anders S, Falciglia M, Wissman JA, Roth EM, & Asch SM (2007). Impact of Clinical Reminder Redesign on Learnability, Efficiency, Usability, and Workload for Ambulatory Clinic Nurses. Journal of the American Medical Informatics Association, 14(5), 632–640. 10.1197/jamia.M2163 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. So HY, Chen PP, Wong GKC, & Chan TTN (2019). Simulation in medical education. Journal of the Royal College of Physicians of Edinburgh, 49(1), 52–57. 10.4997/JRCPE.2019.112 [DOI] [PubMed] [Google Scholar]
  16. Soffler MI, Hayes MM, & Smith C (2018). Advances in Medical Education and Practice Dovepress Central venous catheterization training: current perspectives on the role of simulation. Advances in Medical Education and Practice, 9–395. 10.2147/AMEP.S142605 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Taylor RW, & Palagiri AV (2007). Central venous catheterization. Critical Care Medicine, 35(5), 1390–1396. 10.1097/01.CCM.0000260241.80346.1B [DOI] [PubMed] [Google Scholar]
  18. Yovanoff M, Pepley D, Mirkin K, Moore J, Han D, & Miller S (2016). Improving medical education: Simulating changes in patient anatomy using dynamic haptic feedback. Proceedings of the Human Factors and Ergonomics Society, 603–607. 10.1177/1541931213601138 [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES