Evaluation of a simulation-based workshop on clinical performance for emergency physicians and nurses

Abstract

BACKGROUND:

Simulation-based medical education has been growing rapidly and becomes one of the most popular teaching methods for improving patient safety and patient care. The Simulation Subcommittee of the Hong Kong College of Emergency Medicine organized an educational program emphasizing the team training, clinical decision-making and communication skills. This study aimed to evaluate the attitude of the participants toward a new training program and the change in the knowledge on clinical performance in emergency physicians and nurses after attending the educational program.

METHODS:

A course evaluation form was filled in by the participants at the end of the workshop. An assessment of 20 multiple-choice questions with 5 options was administered to the participants before and after the 2-day simulation-based training workshop.

RESULTS:

A total of 72 doctors and nurses working in the Accident and Emergency Department were enrolled. The average pretest and posttest scores were 12 and 14.3 respectively. The percentage improvement in the mean score of the pretest and posttest was 11.5%. The Chi-square test showed significant improvement in the pretest and posttest score grading (P=0.00). Paired t-test revealed significant difference between the mean scores of the pretest and posttest (P=0.00).

CONCLUSIONS:

Participants had positive attitude toward this new training program. Significant improvement of the knowledge on clinical performance in healthcare professionals in the Accident and Emergency Department was observed after the participation in this simulation-based educational program.

INTRODUCTION

In recent years, simulation-based medical education has been growing rapidly and becomes one of the most popular teaching methods for the improvement of patient safety and patient care. There is growing evidence in the literature on the benefits of clinical simulation.[1] Studies[2–4] have demonstrated the effectiveness of simulation in the teaching of basic science and clinical knowledge, procedural skills, teamwork, and communication as well as assessment at the undergraduate and graduate medical education levels. This method is particularly suitable for training medical personnel working in the Emergency Department (ED) due to the dynamic, complex nature of the working environment and its interdisciplinary nature. The Simulation Subcommittee of the Hong Kong College of Emergency Medicine (HKCEM) organized an educational program called “2-day simulation-based training workshop on clinical performance at emergency medicine” in the period of 2013–2014. The workshop put emphasis on the team training, clinical decision-making and communication skills. Acquisition of these non-technical skills could avoid lots of medical blunders and errors.[5] This study was conducted to determine if this educational program was an effective means to enhance clinical performance for emergency personnel.

Binstadt et al[6] proposed the “clinical performance pyramid” which assumed that clinical performance hinged on 4 highly intertwined domains namely knowledge, decision-making, skill performance, and teamwork. Each domain was built on one another with medical knowledge at the base, which informed clinician’s decision-making at the next level. The third level was technical skill and the highest state was achieved when a highly competent individual functioned effectively as a member of a team to accomplish a specific outcome.

To achieve our goal of enhancing clinical performance of frontline ED staff, we designed the workshop content based on these 4 domains. After completion of the workshop, the participants were expected to 1) apply medical knowledge relevant to the specific emergency scenarios in emergency medicine (EM); 2) demonstrate procedural skills relevant to the specific emergency scenarios in EM; 3) compare different strategies of medical decision-making in EM; 4) apply different elements of teamwork in emergency scenarios. And the objective of this study was to evaluate the attitude of the participants and their change in the knowledge on clinical performance after attending this new training workshop.

METHODS

This was a cross-sectional prospective pretest-posttest study. The subjects were enrolled from doctors and nurses working in the ED. They were nominated by the head of the department throughout the 16 Accident and Emergency Departments in Hong Kong. Five identical 2-day workshops were held in 2013–2014. The first 2 workshops were held in the Simulation Center of the Pamela Youde Nethersole Eastern Hospital. The last 3 workshops were held in the Hong Kong Jockey Club Innovation Learning Center for Medicine which was inaugurated in December 2013. This was a commissioned training program sponsored by the Hospital Authority. Each workshop enrolled 12–16 doctors and nurses currently working in the ED of Hong Kong. They were led by 5–6 facilitators who were doctors and nurses with formal training by professionally recognized training centers on using simulation as the teaching method.

At the beginning of the workshop, the participants were required to complete the pretest assessment which consisted of 20 multiple choice questions with 5 options (Table 1). The questions focused not only on the clinical knowledge about common acute medical problems in EM, but also teamwork, communication skills in difficult situations and models of decision-making relevant to the unique working environment of the ED. Each question carried one mark. One mark was awarded for the correct answer. No mark was given to any wrong answer or question answered with more than 2 choices. The content validity of the pre- and post-test was accomplished by the tool that was devised and reviewed by a group of EM specialists and senior nurses who were experienced trainers in their own right.

Table 1.

Multiple choice questions (The options of each question were omitted for simplicity)

After the pre-test assessment, the participants were divided into groups of 4–6 and rotated through eight different scenarios which comprised of acute, high-stake clinical situations encountered in daily practice of emergency medicine. One example of the eight scenarios is illustrated in Table 2. Each scenario was followed by a debriefing session led by the facilitators. Each scenario session lasted 1 hour. Interactive lectures were conducted in between the scenarios. The topics included teamwork, medical decision-making and communication skills for difficult situations which were all essential components for forming an effective and efficient team in the ED. After the workshop, the participants were asked to complete the posttest assessment which was identical to the pretest assessment. The marking system was same as that of the pretest. The participants were also invited to fill in a course evaluation form which consisted of 13 items with the lowest mark at 1 and the highest mark at 5. The course evaluation form was specially designed for this workshop. It would be used as a template for other similar simulation-based education programs organized by this center in the future.

Table 2.

Example of case scenario

Statistical analysis

The Statistical Package for Social Science (SPSS) version 22.0 for Windows was used for statistical analysis. Descriptive statistics, namely range of scores, mean scores and standard deviation, was used for the description of data distribution. Data on the scoring grades were tabulated using frequency and percentage. The Chi-square test was applied to test for association between various grading of the pretest and posttest. Paired t-test was used to assess any significant difference between the mean scores of the pretest and posttest.

RESULTS

In the 72 participants enrolled, 26 were Emergency physicians and 46 were nurses working in the ED. Their ranking and years of experience in the ED was listed in Tables 3 and 4. All the participants finished the workshop and completed the pretest and posttest assessment. The average pretest and posttest scores were 12 and 14.3 respectively. The range of the pretest score was 4–19, and that of the posttest score was 7–20. The improvement in the mean of pretest and posttest scores was 2.3 (11.5%). Paired t-test revealed statistically significant difference between the mean scores of the pretest and posttest (P=0.00).

Table 3.

Ranking of the participants

Table 4.

Year (s) of experience in the ED

The grading of the participants was summarized in Table 5. The passing grade, as in many tests, was defined as 50% of the total marks. Grade B was categorized as the third quartile of the total marks while grade A as 90% or above. Among the 15 participants who scored grade D in the pretest, 12 of them (80%) achieved the passing grade C or above after the workshop. It was observed that participants who were in lower grades in the pretest achieved higher grades in the posttest (Table 6). Those who were in higher grades in the pretest continued to be in higher grades in the posttest. The Chi-square test revealed that there was significant difference in the number of participants securing higher grades between the pretest and posttest (P=0.00).

Table 5.

Distribution of score grading between the pretest and posttest

Table 6.

Cross tabulation of the pretest and posttest scores

In the course evaluation form (Table 7), the scores varied from 2 to 5. The mean scores of all items were above 4 which was ranked as high by the participants. Most of the participants would recommend this workshop to their colleagues as reflected by the highest socre among all the items. Most of them agreed that the debriefing experience allowed them to see their own mistakes (as shown as the second higheset score) and simulation-based training was more appropriate for them than non-simulation based training. Overall, they reacted postively toward the workshop.

Table 7.

Course evaluation form

DISCUSSION

The objective of this study was to evaluate the attitude of the participants to the new training program and the change in the knowledge on clinical performance. This study found the workshop could improve knowledge and the participants were positive about the course. The participants performed significantly better on the posttest assessment after attending the 2-day simulation-based training workshop on clinical performance as shown by the statistically significant result in the paired t-test analysis. Over 56% of the participants achieved a higher grade in the posttest.

Most of the participants in this group found that simulation-based education was more appropriate for them and the debriefing session was a good way of reflective learning. Although not specified how this was more appropriate for them, it was believed that the characteristics of adult learning proposed by Knowles et al[7] explained why such simulation-based training method was well received by our participants. Adults needed to see the value and relevance of what they learnt. They liked to be actively involved in the learning process. Also, adults needed opportunity for reflection and feedback for improvement. This finding coincided with the survey of educators conducted by Paige et al.[8] They found that debriefing, a specific form of feedback, was the most important part of training using simulation, and a respondent called it ’heart and soul’ of simulation-based training. The simulation-based teaching method has been used successfully in many other educational activities. Coupled with deliberate practice, it has been proven an effective means of educational method in terms of acquisition and retention of knowledge and skills.[9,10] Overall, they were satisfied with this format of teaching and would recommend others to join the workshop.

As pointed out by Croskerry et al,[11] the unique operating characteristics, combined with the complex and myriad activities of EM, predicted vulnerability to a multitude of errors. No doubt that procedural skill sets were important for EM staff’s clinical performance. Most of our time, especially for emergency physicians, was involved in cognitive process which included thinking and reasoning. The majority of diagnostic failures, probably over 75%, could be attributed to physician thinking failure.[12] Therefore we introduced the Dual Process Model for decision-making[13,14] and the concept of cognitive forcing strategies[15] in the interactive lectures. These topics were well received and provoked great interest to the participants as attested by their course evaluation scores and the feedback at the free text section. In our workshop, we were only able to introduce such concepts and highlighted their pertinence to our daily practice. More comprehensive coverage could only be achieved through other focused courses or by the participants’ own pursuance.

We incorporated the framework of TeamSTEPPS[16] to train teamwork skills of our participants. It was a teamwork system designed for healthcare professionals to improve patient safety, communication and teamwork skills among healthcare professionals. Those non-technical skills required in demonstrating effective teamwork simply could not come into operation in a vacuum. Clinical situations and certain challenging events would be needed to bring about the context for demonstrating such teamwork skills. Therefore, high fidelity medical simulation appeared to be a promising method for enhancing teamwork training.[17] Thanks to the newly-established and well-equipped Hong Kong Jockey Club Innovation Learning Centre for Medicine, the workshop could be run smoothly with the high-fidelity simulation facilities and support from the well-trained staff.

Though the workshop was well-received by the participants on the whole, the pre-course reading materials were the course component which scored the lowest mark (average 3.7, range 2–5). The pre-course materials were not as welcomed as they were expected to be. The study was not designed to find out the reason. It would be related to intrinsic nature of emergency medicine which covered a broad spectrum of diseases. The scenarios were made up of a wide variety of critical clinical situations from different specialties. The pile of clinical knowledge together with non-technical skills contributed to an overwhelming amount of pre-course materials. The participants might not have enough time to go through the material before the workshop. Effort would be put on the redesign of the pre-course material in the coming workshop.

Simulation-based trainings are most successful when they are incorporated into the standard curriculum instead of an additional components.[18] It will be increasingly used as an assessment tool for accreditation for licensure and maintenance of certification.[19,20] It might be integrated into the formal curriculum of the specialist training for emergency physicians in Hong Kong in the future as it has already been incorporated in some specialties for high-stakes examinations in many other countries.[21]

There were several limitations in this study. First, the pretest and posttest instruments were the same, potentially introducing testing bias. Second, the format of the assessment was not comprehensive. The use of multiple choice questions as the instrument of assessment only tested the knowledge of the participants, but not the skills and behavior. The reliability and validity of the instrument was not addressed as well. In the future workshop, a variety of validated assessment tools could be incorporated in the pretest and posttest assessment to more truly reflect the change in the knowledge, skills and behavior of the participants. Although no standardized and widely accepted methods for assessing non-technical skills were currently available, there were a number of commonly used behavioral rating scores designed by different institutions for measuring the outcomes of teamwork training in medicine. For examples, the Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS),[22] the Anesthetists’ Nontechnical Skills (ANTS) evaluation,[23] the Mayo High Performance Teamwork Scale (MHPTS).[24] The Ottawa GRS consists of six 7-point scales with descriptive anchors corresponding to five core CRM domains, namely leadership, problem solving, situational awareness, resource utilization, communication, and one overall performance domain. It has strong validity and inter-rater reliability for assessing crisis resource management skills and team leader performance in a simulated environment across a variety of clinical scenarios and training levels. This behavioral assessment tool could be used for rating the pre- and post-scenario test in the upcoming workshop. Third, the posttest assessment was carried out immediately after the workshop. The participants had fresh memory of what they had learnt in the workshop. However, the review conducted by Custers[25] suggested that in medical education approximately two-third to three-fourth of knowledge would be retained after one year, with a further decrease to slightly below fifty percent in the next year. Further assessment was necessary to test the retention of the knowledge and skills after the workshop in the future. Lastly, the sample size was small in this study. As the workshop continues to be run in the future, more participants can be accumulated to increase the data pool and validate the study.

In conclusion, this 2-day-simulation-based training workshop improved the knowledge of clinical performance of the participants and was well received. Future studies could focus on how the improvement of knowledge on clinical performance could lead to enhancement of the patient care and safety and how this type of training might be integrated into the formal curriculum of the specialist training for emergency physicians.