A new approach for the acquisition of trauma surgical skills: an OSCE type of simulation training program

Abstract

Background

Worldwide, trauma-related deaths are one of the main causes of mortality. Appropriate surgical treatment is crucial to prevent mortality, however, in the past decade, general surgery residents’ exposure to trauma cases has decreased, particularly since the COVID-19 pandemic. In this context, accessible simulation-based training scenarios are essential.

Methods

A low-cost, previously tested OSCE scenario for the evaluation of surgical skills in trauma was implemented as part of a short training boot camp for residents and recently graduated surgeons. The following stations were included bowel anastomosis, vascular anastomosis, penetrating lung injury, penetrating cardiac injury, and gastric perforation (laparoscopic suturing). A total of 75 participants from 15 different programs were recruited. Each station was videotaped in high definition and assessed in a remote and asynchronous manner. The level of competency was assessed through global and specific rating scales alongside procedural times. Self-confidence to perform the procedure as the leading surgeon was evaluated before and after training.

Results

Statistically significant differences were found in pre-training scores between groups for all stations. The lowest scores were obtained in the cardiac and lung injury stations. After training, participants significantly increased their level of competence in both grading systems. Procedural times for the pulmonary tractotomy, bowel anastomosis, and vascular anastomosis stations increased after training. A significant improvement in self-confidence was shown in all stations.

Conclusion

An OSCE scenario for training surgical skills in trauma was effective in improving proficiency level and self-confidence. Low pre-training scores and level of confidence in the cardiac and lung injury stations represent a deficit in residency programs that should be addressed. The incorporation of simulation-based teaching tools at early stages in residency would be beneficial when future surgeons face extremely severe trauma scenarios.

Globally, over 1 billion people suffer traumatic injuries, being responsible for 6 million deaths annually [1]. Mortality is concentrated in the first hours and is often caused by massive bleeding from big vessels or highly irrigated organs; creating fatal outcomes that are preventable with prompt and effective surgical care [2–5].

In rural areas and low-income countries, young general surgeons with little to no experience in independent practice are often on call at emergency services during non-business hours, which is when the highest number of accidents occur. However, nationally and internationally, it has been described that general surgeons do not feel qualified to take on this task [6]. In North America, up to 40% of surgical residents do not feel confident enough for their independent practice even though they spend 5 years training as residents [6, 7]. Regulations limiting residents’ training hours and the advent of non-operative management have been among the main reasons to explain this situation [8, 9]. In the past year, the COVID-19 pandemic has aggravated this to an unprecedented extent [10]. A significant reduction in surgical volume alongside severe shortages of personnel requiring residents to be redistributed to roles that involve taking care of non-surgical patients has drastically diminished residents’ caseload exposure [11, 12]. Even though programs have implemented different strategies to cope with these difficulties, the level at which it has hindered the quality of surgical training has yet to be seen. In this context, and considering that simulation-based teaching tools have proven to be effective in a variety of settings, the incorporation of simulation scenarios is even more crucial than before [13]. However, elevated costs can affect simulation tools’ availability. Therefore, easy-to-access and easy-to-implement alternatives are especially relevant. In previous work, our team developed a low-cost Objective Structured Clinical Examination (OSCE) scenario for the evaluation of surgical skills in trauma surgery [14]. In this article we implemented the OSCE as a training tool for residents, measuring the improvement of skills, and evaluating the impact it had on self-perception of competence.

Methods

An observational study was designed. Participants were invited to join in a three-day trauma surgical skills boot camp. Pre- and post-training performance were assessed alongside self-confidence to independently perform the procedures.

Recruiting

Surgical residents at different levels of training from 15 different programs across the country and recently graduated general surgeons from several regions were invited to participate. A minimum sample size of 11 participants per group was established using a power of 0.8 and alpha of 0.05.

Therefore, 75 participants were recruited and distributed into five groups of 15 trainees each to be enrolled in a 3-day training program. Traveling and hosting expenses were covered.

Assessment and training

Our team previously designed and tested under the Messick validity framework an Objective Structured Clinical Examination scenario (OSCE) for evaluation of surgical skills in trauma [14, 15]. The OSCE was now used to assess participants' performance and guide their training. It consisted of 5 simulation-based stations: bowel anastomosis (hand-sewn side- to-side entero-enterostomy on a bovine bowel), vascular anastomosis (end-to-end hand-sewn anastomosis in a perfused 5 mm porcine carotid or iliac artery), laparoscopic suturing (laparoscopic suturing of gastric perforation in an ex vivo porcine stomach model), penetrating lung injury (hand-sewn tractotomy in ex vivo ventilated porcine lungs), and penetrating cardiac injury (bleeding control and suturing of three standardized injuries: right atrium, right ventricle away from the coronary artery, and left ventricle in the vicinity of the coronary artery in a perfused ex vivo porcine model). The cardiac model was perfused using a pulsatile roller pump connected to the aorta, delivering up to 5 lt/min of fake blood, with a pumping heart effect of 120–140 beats per minute [14, 16–18].

These procedures were selected due to their reported frequency in Chilean Hospitals [19, 20] and guidelines developed by the Chilean Society of Surgeons regarding the minimal competencies needed to be a general surgeon in Chile. This is due to the reality observed in most hospitals in Chile, which do not have specialized surgical trauma departments, or on-call teams for surgical specialists (vascular, cardiothoracic, or gastrointestinal surgeons). Therefore, most trauma-related surgeries in our country are performed by general surgeons.

Participants had 20 min to complete each station.

For pre- and post-training assessments, each procedure was recorded in high definition and blindly evaluated by experts in an asynchronous manner through a novel technology-enhanced learning platform [21, 22]. Participants were evaluated using a modified OSATS (Objective Structured Assessment of Technical Skills), a widely used global rating scale designed to assess proficiency in practical surgical skills [23]. The total score is based on subscores in five domains: respect for tissue, time and motion, instrument handling, the flow of operation and forward planning, and knowledge of the specific procedure. Each domain is assessed on a scale from 1 to 5; the maximum possible score for each station was 25 points. In addition, a task-specific rating scale (SRS) was developed according to experts’ opinions on essential aspects that had to be included. This “checklist” type of assessment was specific to each station, with varying maximum scores.

A trauma surgery boot camp was designed, with training sessions guided by expert trauma surgeons from our institution. Participants were divided into groups to deliberately practice on the models while receiving direct feedback from experts. The training sessions lasted 2 h for each station.

Self-perception of skills

A self-assessment survey was designed and applied before and after training. The survey contained Likert-type [24] questions regarding participants’ self-perception of competence performing different trauma procedures as the leading surgeon [20].

Statistical analysis

Data were analyzed with RStudio® (2019 v1.2.5001 on base R, Boston, US) using nonparametric statistics. Pre- and post-training performance and self-confidence were compared using a paired, two-tailed Wilcoxon signed-rank test with an alpha set at 0.05 and a power of 80%. To compare scores among groups, the Kruskal–Wallis test for multiple comparisons was used to assess whether there were significant differences among the groups in terms of scores and procedural times. The Dunn test for pairwise comparisons was used to determine in which groups the differences were statistically significant.

Results

Pre-training OSATS results showed statistically significant differences among the previous levels of expertise for the cardiac injury [postgraduate year 1 (PGY1) vs recently graduated surgeons (RGS), p = 0.0282], bowel anastomosis [postgraduate year 2 (PGY2) vs postgraduate year 3 (PGY3) p = 0.0189], and vascular anastomosis [PGY1 vs PGY3, p = 0.0423] stations (Table 1). In the cardiac injury station, first-year residents got the lowest score of all stations, with a median of 15 points.

Table 1.

OSATS pre-training

Station	OSATS pre-training (range-median)				Pairwise comparisons (p-value)
	PGY1	PGY2	PGY3	RGS
Penetrating cardiac injury	10–22 (15)	12–23 (17.5)	13–24 (20)	10–24 (20.5)	PGY1 vs RGS p = 0.0282
Penetrating lung injury	14–22 (17)	15–23 (18)	15–24 (18)	14–25 (18)	Non-significant
Bowel anastomosis	17–24 (22)	16–24 (20)	17–25 (23)	17–25 (23)	PGY2 vs PGY3 p = 0.0189
Laparoscopic suturing	11–25 (20)	11–25 (21)	14–24 (22)	16–25 (21–5)	Non-significant
Vascular anastomosis	11–19 (17)	13–22 (18.5)	14–23 (20)	11–24 (19)	PGY1 vs PGY3 p = 0.0423

Comparison among groups according to the previous level of expertise

PGY1 postgraduate year 1, PGY2 postgraduate year 2, PGY3 postgraduate year 3, RGS recently graduated surgeons

In terms of the SRS performance before training, bowel and vascular anastomosis stations showed significant differences in performance. For the bowel anastomosis station, Median pre-training scores were very similar for all groups, however, there was 1 participant in the PGY2 group that got only 3 points. A statistically significant difference was found between that group when compared to PGY3 and RGS, which correlates with what was obtained with OSATS (Table 2).

Table 2.

SRS pre-training

Station	SRS pre-training range (median)				Pairwise comparisons (p-value)
	PGY1	PGY2	PGY3	RGS
Penetrating cardiac injury	2–6 (4)	2–7 (4)	3–7 (5.5)	2–7 (5)	Non-significant
Penetrating lung injury	0–7 (2)	0–8 (3)	0–8 (4)	1–8 (4.5)	Non-significant
Bowel anastomosis	7–14 (13)	3–14 (11)	8–15 (13)	8–15 (13)	PGY2-RGS p = 0.0283 PGY2-PGY3 p = 0.0034
Laparoscopic suturing	13–21 (17)	18–21 (18)	14–21 (19.5)	14–21 (20)	Non-significant
Vascular anastomosis	16–23 (21)	18–28 (21.5)	18–28 (23)	18–27 (23)	PGY1 vs PGY3 p = 0.0116

Comparison among groups according to the previous level of expertise

PGY1 postgraduate year 1, PGY2 postgraduate year 2, PGY3 postgraduate year 3, RGS recently graduated surgeons

In the vascular anastomosis station, statistically significant differences were also found, even though all groups got very similar pre-training scores.

As it is shown in Table 3, in terms of individual results, participants’ achieved a statistically significant improvement after the boot camp in the majority of the stations, with p values < 0.0001 in both grading scales for the cardiac, lung, bowel, and vascular stations. For the laparoscopic suturing station, p values of 0.0002 and 0.0006 were obtained for OSATS and SRS, respectively.

Table 3.

SRS post-training

Station	SRS post-training range (median)				Pairwise comparisons (p-value)
	PGY1	PGY2	PGY3	RGS
Penetrating cardiac injury	4–8 (8)	5–8 (7)	4–8 (8)	5–8 (8)	Non-significant
Penetrating lung injury	6–8 (7)	2–8 (7.5)	6–8 (8)	5–8 (8)	Non-significant
Bowel anastomosis	9–15 (13)	8–15 (14)	8–15 (14)	9–15 (14)	Non-significant
Laparoscopic suturing	14–21 (19)	13–21 (19)	15–21 (19)	17–21 (20)	Non-significant
Vascular anastomosis	19–24 (22)	20–27 (24)	20–28 (24.5)	20–28 (23.5)	Non-significant

Comparison among groups according to the previous level of expertise

PGY1 postgraduate year 1, PGY2 postgraduate year 2, PGY3 postgraduate year 3, RGS recently graduated surgeons

Regarding the time to complete the procedure, statistically, significant differences were found in all stations. In the pulmonary tractotomy, bowel anastomosis and vascular anastomosis stations’ time to complete the procedure increased after training. The biggest increase was found in the lung injury station with a median increase of 6.11 min after training.

In the cardiac injury station, to better assess participants’ performance timing was subdivided into bleeding control, ventricular repair, and atrial repair. Participants showed a statistically significant improvement in bleeding control and ventricular repair, diminishing the median time 35 and 27 s, respectively (Table 4).

Table 4.

Cardiac injury station time to complete the procedure

Penetrating cardiac injury	Pre time (s)	Post time (s)	p-value
Bleeding control	1–657 (57)	2–274 (22)	< 0.0001
Ventricular repair	16–1076 (114)	37–641 (87)	0.0238
Atrial repair	30–878 (147)	29- 479 (138)	0.2020

Regarding the self-perception of competence survey applied before and after training, all participants reported an improvement in self-confidence after training (Fig. 3, Table 5). In the bowel anastomosis station, the PGY1, PGY2, and PGY3 groups showed statistically significant improvements. The PGY1 group was more heterogeneous after training but showed the highest median improvement among all groups, while the PGY2 and PGY3 groups improved median scores and reduced interquartile range. Even though non-statistically significant, the RGS started with the highest median level of confidence, and after training the group dispersion was also reduced.

Fig. 3.

A–E Confidence level results per station. PGY1 postgraduate year 1, PGY2 postgraduate year 2, PGY3 postgraduate year 3, RGS recently graduated surgeons

Table 5.

Assessment of self-confidence to perform the procedure

From 1 to 5, How confident would you feel as the leading surgeon in the following cases?
Bowel injury: performing a latero-lateral manual anastomosis		Transfixing lung injury: performing a tractotomy		1 cm penetrating ventricular injury: bleeding control and myocardial repair		Complete section of the proximal femoral artery: vascular anastomosis		Traumatic gastric perforation: laparoscopic suturing
Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post
3.6	4.5	2.7	4.2	2.7	4.2	2.7	4.1	3.1	4.1

Results expressed as mean

For the Vascular injury and lung Injury station, all groups improved median scores with similar levels of dispersion.

In the cardiac injury station, all groups improved their median level of confidence with the PGY-1 showing almost no dispersion before training, since most participants rated their confidence level as 1, and only 3 as 2 (the lowest confidence level among all stations). The highest heterogeneity is shown in the PGY-3 group, with results ranging from 1 to 5. Finally, in the laparoscopic suturing station, all groups improved their median level of confidence, with the highest dispersion shown in the first 2 years of residency.

Discussion

As it was previously stated, the importance of developing a low-cost, easy-to-access trauma surgery skills training system is crucial to impact the global burden of trauma injuries, which is particularly important in middle to low-income countries. OSCE scenarios have previously shown to be effective assessment tools for practical skills in medicine, which aligns with previous work developed by our team in which we designed and tested an OSCE scenario for the evaluation of trauma surgical skills [14]. This OSCE type of training system has considerably lower costs than other training courses available in our country, such as the American College of Surgeons Advanced Trauma Operative Management (ATOM) that cost up to 1.000 USD in our country [25]. In comparison, the system that we propose has an estimated cost of 330 USD per participant (Table 6).

Table 6.

Costs associated with the development of the boot camp training

	Costs per 6 participants OSCE	Cost per participant
Simulation Center Rent (includes high definition video recordings)	420	70
Station assistants’ (5)	324	54
OSCE monitor	65	1083
Stations instrumentalsa	680	1133
Ex vivo tissue	188	3133
Suture materials	278	4633
General supplies	18	3
Total	1973	32,879

All costs are expressed in USD

^aThese instrumentals cost 6.800 USD and can be used for up to 10 OSCE

In this article we aimed to evaluate the OSCE stations as a training system, testing it in participants with different backgrounds and levels of training. The assessment tools used for this OSCE contemplated the widely validated global rating scale OSATS, and task-specific rating scales (SRS) for each procedure. The SRS were developed based on the opinion of experts at our institution.

The results showed participants significantly improved their performance after a training session with the models (Figs. 1 and 2), improvements that were particularly noticeable in the cardiac, pulmonary, bowel, and vascular stations. When analyzing procedural times (Table 7), it is interesting to notice that in three stations (pulmonary, bowel, and vascular anastomosis) time significantly increased after training. This could be a reflection of a deficiency in participants’ previous knowledge of the technique, meaning they had to “learn” how to perform those procedures in the boot camp instead of practicing a previously acquired skill. These findings align with previous work done by Vela et al. that showed a very low level of preparedness in general surgeons to deal with complex trauma cases [20]. Particularly noticeable were the results obtained in the pulmonary tractotomy station, which is a procedure not often performed by general surgery residents, and a median increase of 6.11 min was observed. The cardiac injury and laparoscopic suturing stations showed significant reductions in procedural times, which may indicate participants had a previous notion on how to perform those procedures and used their training time to become more proficient. It is particularly significant from a clinical perspective a 35-s reduction in time to control myocardial bleeding.

Fig. 1.

OSATS individual scores before and after training. *p-value < 0.01, **p-value < 0.001

Fig. 2.

SRS individual scores before and after training. *p-value < 0.01, **p-value < 0.001

Table 7.

Comparison of scores and time to complete the procedure before and after training

Station	Pre OSATS	Post OSATS	p-value	Pre SRS	Post SRS	p-value	Pre time (s)	Post time (s)	p-value
Penetrating cardiac injury	10–24 (19)	17–25 (24)	< 0.0001	2–7 (5)	4–8 (8)	< 0.0001	Described in detail in Table 2
Penetrating lung injury	14–25 (18)	16–25 (23)	< 0.0001	0–8 (3)	2–8 (8)	< 0.0001	144–1083 (565)	571–1153 (932)	< 0.0001
Bowel anastomosis	16–25 (22)	18–25 (23)	< 0.0001	3–15 (13)	8–15 (14)	< 0.0001	481–1150 (777)	523–1082 (880)	0.0002
Laparoscopic suturing	11–25 (22)	12–25 (22)	0.0002	8–21 (19)	13–21 (19)	0.0006	240–927 (593)	209–921 (507)	0.0004
Vascular anastomosis	11–24 (18)	13–25 (21)	< 0.0001	16–28 (22)	19–28 (23)	< 0.0001	511–1042 (801)	476–1115 (856)	0.0006

In terms of participants’ scores according to the previous level of expertise (Table 8), significant differences in performance were found among the 4 groups in the cardiac injury, bowel anastomosis, and vascular anastomosis stations. Those differences disappeared after training, except for the cardiac injury station, where statistically significant differences among groups persisted.

Table 8.

OSATS post-training

Station	OSATS post-training range (median)				Pairwise comparisons (p-value)
	PGY1	PGY2	PGY3	RGS
Penetrating cardiac injury	17–25 (23)	17–25 (23)	17–25 (25)	20–25 (24)	PGY1 vs PGY3 p = 0.0052 PGY2 vs PGY3 p = 0.0194
Penetrating lung injury	16–24 (22)	19–25 (22.5)	19–25 (23.5)	20–25 (23)	Non-significant
Bowel anastomosis	19–25 (23)	18–25 (23)	21–25 (24)	21–25 (24)	Non-significant
Laparoscopic suturing	14–25 (21)	12–25 (22)	17–25 (22.5)	16–25 (23)	Non-significant
Vascular anastomosis	16–22 (20)	13–25 (21)	16–25 (22)	15–25 (20.5)	Non-significant

Comparison among groups according to the previous level of expertise

PGY1 postgraduate year 1, PGY2 postgraduate year 2, PGY3 postgraduate year 3, RGS recently graduated surgeons

Regarding self-perception of competence, overall participants reported an improvement in self-confidence (Table 6, Fig. 3). In the bowel anastomosis station, residents improved their confidence level alongside reducing the group’s dispersion, except for first-year residents, which was also the group with the biggest median improvement and higher dispersion. A possible explanation for this could be that as residents move forward on their training, they tend to level up in this technique, as they possess more tools to reach a successful learning curve. Something similar happened in the laparoscopic suturing station, where higher heterogeneity was observed during the first two years of residency. On the other hand, in the cardiac injury station, the highest level of dispersion was shown in the PGY3 group which alongside extremely low pre-training results obtained by PGY1 might reflect that abilities required for a successful learning curve in this procedure are acquired later during the course of training.

Limitations

The number of participants was limited because of geographical constraints. In some cases, trainees had to travel over a thousand miles to attend the boot camp. In addition, our country has a limited number of general surgery residents. Since the participants of this study came from 15 certified general surgery residency programs available in our country, we believe this sample is heterogeneous and representative of our reality.

On secondhand, we acknowledge a weakness in one of the scales used for assessment, the task-specific rating scales did not undergo a specific validation process.

Finally, this study contemplates assessment in simulated procedures, showing acquisition of technical skills in this scenario. Further studies are required to evaluate predictive validity with a transfer of skills from this boot camp training to the real scenario, the operating room.

Conclusion

An improvement in the objective scores, time to perform the procedure, and self-perception of surgical skills were shown, proving that this training system is effective and can be used as an effective training strategy for the acquisition of surgical skills in trauma surgery. Extremely low objective scores and level of confidence at the cardiopulmonary and lung stations, show a deficit in the training program and should be addressed. If simulation-based programs like this one are included in the first years of residency, general surgeons could have better skills and confidence when facing these extremely severe trauma scenarios.

Funding

This study was funded and supported by Chilean Research Grant FONDECYT REG.1171908 from CONICYT.

Declarations

Disclosures

Drs. Catalina Ortiz, Javier Vela, Caterina Contreras, Francisca Belmar, Ivan Paul, Analia Zinco, Juan Pablo Ramos, Pablo Ottolino, Pablo Achurra, Nicolas Jarufe, and Julian Varas were financed and supported by a Chilean Research Grant FONDECYT REG 1171908 from CONICYT and by the Department of Digestive Surgery, Faculty of Medicine, Pontificia Universidad Católica de Chile. Dr. Julian Varas is the Founder of Training Competence, an official spinoff startup from the Pontificia Universidad Católica de Chile. The startup and the Pontificia Universidad Católica de Chile are the owners of the rights and distribution of the C1DO1 platform used for the assessment in this study. Dr. Adnan Alseidi has no conflict of interest or financial ties to disclose.