Abstract
Introduction:
Evaluation of students’ learning in clinical education system is one of the most important and challenging issues that facilities in this field have been facing. The present study aimed to evaluate the role of feedback during evaluation in increasing emergency medicine residents’ clinical skills.
Method:
The present experimental study was performed on all second year emergency medicine residents of two educational hospitals, Tehran, Iran, with switching replications design and before-after method. They were randomly allocated to two groups (with or without feedback) and evaluated three times regarding chest ultrasonography for trauma patients, using direct observation of procedural skills (DOPS) and valid and reliable checklist. Data were analyzed using SPSS 20.
Results:
30 emergency medicine residents with the mean age of 36.63 ± 30.30 years were allocated to two equal groups (56.7% male). Studied groups were similar regarding the baseline characteristics. In both groups, obtained scores showed a significant increase from the first to the third evaluation (p < 0.001). Mean scores of first and second evaluations were 10.24 ± 0.77, 17.73 ± 0.46 in feedback receivers and 9.73 ± 0.77 and 12.13 ± 0.47 in others (p < 0.001). Mean third score after switching groups were 18.53 ± 0.22 in feedback receivers and 18.99 ± 0.22 in others (p = 0.213).
Conclusion:
Based on the findings of the present study, giving feedback after evaluating the second year emergency medicine residents regarding chest ultrasonography for trauma patients, led to a significant improvement in their scores in future evaluations and consequently their skill.
Key Words: Education, medical, graduate, emergency medicine, learning, formative feedback
Introduction
Training of clinical skills is one of the essential parts of medical education. Medical education facilities have been developed for creating measurable changes in clinical practice (1). Evaluation of students’ learning in clinical education system is one of the most important and challenging issues that facilities in this field have been facing (2). In most clinical courses, evaluation methods do not have sufficient efficiency for assessment of clinical skills (3-5). Common evaluation methods have failed to assess students’ clinical skills accurately, they are not considered as tools for learning. To solve prominent problems, possible strengths and weaknesses of educational facilities were counted and then positive aspects of the educational facilities were improved and their shortcomings were eradicated (6). Investigators in medical education field assessed some methods of clinical evaluation such as performance observation, 360 degree evaluation, objective structured clinical examination, mini-clinical evaluation exercise (Mini-CEX) and direct observation of procedural skills (DOPS), which have been used for evaluation of medical students and their clinical environments.
Effective assessment of medical students’ clinical skills can improve their motivation and help professors have a more accurate measurement of their actual skills (7). DOPS is known as a new procedure for evaluation of clinical practice skills of medical residents and students, which provides a suitable feedback tool for medical professors (2). Use of formative feedback has positive effects on professional behavior, attitude and communication for medical students (8).
Emergency department is one of the critical hospital wards in which residents must have clinical skills to be prepared for primary health care procedures, diagnosis and treatment of referred patients. Based on the above-mentioned points, the present study aimed to evaluate the role of feedback during evaluation in increasing emergency medicine residents’ clinical skills.
Methods
Study design and setting
The present experimental study was performed on all second year emergency medicine residents of two educational hospitals affiliated with Shahid Beheshti University of Medical Sciences (Imam Hossein and Loghman hakim), Tehran, Iran, from June to November 2015, with switching replications design and before-after method. The study aimed to evaluate the role of feedback during evaluation (FDE) in increasing residents’ skills regarding chest ultrasonography for trauma patients. All researchers adhered to ethical recommendations of Helsinki protocol and confidentiality of participants’ information during the study period.
Ethical consideration
Individuals participated in the study by giving informed consent, in addition, data were analyzed as a pool, and the name of individuals was not mentioned. The researchers promised to share the results with the participants on demand. All the patients that participated in rating the skills of residents were selected based on the indications provided in references and were given sufficient information regarding the method of the study. Any patient that wanted to leave the study or the hospital was excluded from the study. Replacements were selected for the 8 patients that decided to leave the study or the hospital. This study did not interfere with the routine diagnostic and treatment process of the patients or impose additional costs on them.
Participants
Study participants were selected from second year emergency medicine residents, who had passed their first year promotion exam, using census sampling. All participants were trained regarding chest ultrasonography for trauma patients during their first year of the residency program. They were randomly divided into 2 groups of Loghman Hakim (group 1) or Imam Hossein Hospital (group 2), using simple randomization method. In each studied hospital, one emergency medicine professor, who was informed regarding the study design and DOPS method, was responsible for evaluation of residents. Both attends, who were responsible for evaluation of the participants, were informed regarding the method of the study and its details in a 4-hour coordinator session.
Procedure
After preparing essential tools for doing the procedure, participants were evaluated regarding chest ultrasonography for blunt trauma patients using DOPS method.
Initially, all the participants (separately in both study hospitals) were evaluated using a checklist designed for this purpose and DOPS method in off-duty hours and their scores were recorded. At this stage, after the evaluation was finished, the residents in Loghman Hakim Hospital were given feedback on their weak and strong points for 5 minutes but those in Imam Hossein Hospital were evaluated traditionally and without feedback.
After 2 months, the evaluation was repeated for residents of both hospitals. However, this time evaluation method was switched between the 2 groups and this time residents of Imam Hossein Hospital were given feedback and those in Loghman Hakim Hospital took were evaluated traditionally and without feedback. Finally, about 2 months later, all the second year residents of both hospitals were evaluated again, for the third time.
Duration of evaluation was 15 minutes in all 3 stages, and when the residents were supposed to receive feedbacks it would be organized in 5 minutes. Effort was made to provide similar environmental conditions such as time, place, ultrasonography device, evaluated patients, etc. for both groups.
Data gathering
Data gathering tool was a standard (20-item) checklist prepared according to evidence-based texts regarding the requirements that should be met in this procedure (appendix 1). Items of the checklist were designed to evaluate 3 general parts of the procedure including preparation (4 items), carrying out the procedure (13 items), and post-procedure measures (3 items). Each correct answer was given 1 point and for wrong answers, no point was given. Consequently, the maximum and minimum obtainable scores for each participant were 20 and 0, respectively. A score between 0 and 10 was considered as fail and a score between 10 and 20 was considered as pass. Score above 17 was classified as an excellent score.
Appendix 1.
Evaluation checklist of chest ultrasonography for trauma patients using DOPS method
| Items | Proper performance | Points * | |
|---|---|---|---|
| Preparation | Yes | No | |
| 1. Correctly identifying the patient | |||
| 2. Introducing self and being professional | |||
| 3. Knowing indications of the procedure | |||
| 4. Proper position of patient and device | |||
| Ultrasonography performance | |||
| 1. Preparing proper tools | |||
| 2. Correct selection of probe | |||
| 3. Evaluation regarding pneumothorax | |||
| 4. Identifying and evaluating the 3 zones | |||
| 5. Evaluating all intercostal spaces | |||
| 6. Identification of pleural cavity | |||
| 7. Lung point sign evaluation | |||
| 8. Pleural sliding evaluation | |||
| 9. Evaluation of comet tails/B-line | |||
| 10. Evaluation of seashore sign in M-mode | |||
| 11. Evaluation regarding hemothorax | |||
| 12. Right side evaluation of hemothorax | |||
| 13. Left side evaluation of hemothorax | |||
| Post-procedure | |||
| 1. Saving and printing the image | |||
| 2. Interpretation of findings | |||
| 3. Proper decision based on findings | |||
: Each right answer gets 1 point and wrong answers receive no point (minimum 0 and maximum 20).
Validity of the prepared checklist was approved by 10 emergency medicine professors with more than 5 years of experience from various universities. Reliability of the tools was calculated in a pilot study on 15 people and its correlation coefficient was estimated to be 85% based on Cronbach’s alpha.
Statistical analysis
Study variables were analyzed by SPSS software version 20. Mean ± standard deviation or frequency and percentage were used for quantitative and qualitative variables, respectively. Paired sample t-test and chi-square were used for comparing quantitative and qualitative variables between study groups. The Greenhouse-Geisser Correction test was used for assessing the impact of repeatedly performing DOPS. All P-values less than 0.05 were considered as significant.
Results
Baseline characteristics
30 emergency medicine residents with the mean age of 36.63 ± 30.30 (29 - 47) years participated (56.7% male). They were randomly allocated to either Loghman Hakim (group 1) or Imam Hossein hospitals (group 2). Table 1 shows the baseline characteristics of the participants. The two studied groups were similar regarding the baseline characteristics. 25 (83.4%) residents performed chest ultrasonography more than five times (56.7% more than 10 times) and 25 (83.3%) residents believed that chest ultrasonography for trauma patients had medium to high difficulty level.
Table 1.
Baseline characteristics of participants
| Variable | Loghman Hakim | Imam Hossein | P |
|---|---|---|---|
| Mean age (year) | 34.93 ± 5.40 | 38.33 ± 5.23 | 0.091 |
| Sex | |||
| Male | 9 (60) | 8 (53.3) | 0.71 |
| Female | 6 (40) | 7 (46.7) | |
| Background experience1 (time) | |||
| 1 -4 | 3 (20) | 2 (13.3) | 0.879 |
| 5- 9 | 4 (26.7) | 4 (26.7) | |
| ≥ 10 | 8 (53.3) | 9 (60) | |
| Background attitude 2 (difficulty) | |||
| Low | 4 (26.7) | 1 (6.7) | 0.241 |
| Medium | 10 (66.7) | 11 (73.3) | |
| High | 1 (6.7) | 3 (20) | |
| Mean promotion exam score 3 | 81.06 ± 6.29 | 82.42 ± 6.27 | 0.565 |
Data were presented as mean ± standard deviation or frequency (percentage);
1: Previous experience of chest ultrasonography;
2: Attitude of the resident regarding the difficulty of the procedure;
3: Score in exam taken for promotion from year 1 to year 2 of the residency program (out of 100).
Outcomes
Mean scores of the residents in the first, second, and third evaluations were 10.00 ± 0.54, 14.93 ± 0.32, and 18.72 ± 0.15, respectively. Figure 1 shows the scores obtained by the residents for the 3 evaluations done based on the hospital. In both groups, obtained scores showed a significant increase from the first to the third evaluation (p < 0.001) according to Greenhouse-Geisser Correction test.
Figure 1.
Trend of scores (minimum 0 and maximum 20) in three evaluations of studied groups (p < 0.001, based on Greenhouse-Geisser Correction
Mean score obtained for preparation, scan and post-scan items among all residents were 3.34 ± 0.91, 9.48 ± 3.16, and 1.72 ± 0.97, respectively. Tables 2 and 3 compare the scores obtained by each group in the 3 tests qualitatively and quantitatively. Mean scores of first and second evaluations were 10.24 ± 0.77, 17.73 ± 0.46 in feedback receivers and 9.73 ± 0.77 and 12.13 ± 0.47 in others (p < 0.001). Mean third score after switching groups were 18.53 ± 0.22 in feedback receivers and 18.99 ± 0.22 in others (p = 0.213). As can be seen, residents in Loghman Hakim Hospital were significantly better in the second exam both quantitatively and qualitatively.
Table 2.
Comparison of mean evaluation scores between groups
| Evaluation | Loghman Hakim | Imam Hossein | P-value |
|---|---|---|---|
| Preparation (0 – 4) | |||
| First | 3.00 ± 0.75 | 2.1 ± 0.92 | 0.009 |
| Second | 4.00 ± 0.00 | 3.00 ± 0.85 | < 0.0001 |
| Third | 3.93 ± 0.26 | 4.00 ± 0.00 | 0.326 |
| Scan (0 – 13) | |||
| First | 6.87 ± 2.39 | 6.27 ± 2.43 | 0.501 |
| Second | 11.60 ± 0.99 | 7.33 ± 1.95 | < 0.001 |
| Third | 12.53 ± 0.74 | 12.26 ± 0.59 | 0.287 |
| Post scan (0 – 3) | |||
| First | 0.40 ± 0.63 | 1.20 ± 0.67 | 0.002 |
| Second | 2.13 ± 0.35 | 1.80 ± 0.86 | 0.176 |
| Third | 2.53 ± 0.64 | 2.27 ± 0.71 | 0.287 |
| Total (0 - 20) | |||
| First | 10.26 ± 2.99 | 9.73 ± 2.94 | 0.626 |
| Second | 17.73 ± 1.09 | 12.13 ± 2.29 | <0.001 |
| Third | 18.93 ± 0.96 | 18.53 ± 0.74 | 0.213 |
Data were presented as mean ± standard deviation.
Discussion
Based on the findings of the present study, giving feedback after evaluating the second year emergency medicine residents regarding chest ultrasonography for trauma patients, led to a significant improvement in their scores in future evaluations and consequently their skill.
Table 3.
Comparison of score quality between groups
| Evaluation | Loghman Hakim | Imam Hossein | P-value |
|---|---|---|---|
| First | |||
| Failed | 8 (53.3) | 5 (33.3) | 0.462 |
| Passed | 7 (46.7) | 10 (66.7) | |
| Excellent | 0 (0) | 0 (0) | |
| Second | |||
| Failed | 0 (0) | 2 (13.3) | < 0.001 |
| Passed | 3 (20) | 13 (86.7) | |
| Excellent | 12 (80) | 0 (0) | |
| Third | |||
| Failed | 0 (0) | 0 (0) | NA |
| Passed | 15 (100) | 15 (100) | |
| Excellent | 0 (0) | 0 (0) |
Data were presented as frequency and percentage; NA: not applicable.
Previous studies have shown that assessment of trainees via DOPS can significantly improve skill learning among medical students and residents in some countries (9-11). In most medical education curriculums, DOPS was used for evaluation of main clinical skills such as intravenous cannulation, lumbar puncture, and endotracheal intubation. These procedures were observed in different clinical settings including outpatient clinics and emergency departments.
Barton et al. and Thomas-Gibson et al. studies indicated that DOPS method causes an increase in residents’ competence in the fields of colonoscopy and endoscopy (12, 13). According to the findings of Hauck S et al., use of DOPS for medical students in ultrasound training field caused a significant increase in their knowledge, confidence, self-motivation and understanding of anatomical structures.
It is recommended that medical trainers observe medical students while doing the procedure on the patients, and ask them about indications, complications and post procedure cares and immediately give them feedback for future improvement of their skills.
Charleen Liu, who is active in the emergency medicine training field in the United Kingdom, believes that the recent trend in medical education is rapidly moving from traditional routine examinations to collecting evidence of clinical competence and professional behavior observed in clinical environments (work-based learning) through methods such as DOPS, mini-CEX, and case-based discussion (CbD). This is consistent with the highest level of the Miller education pyramid (14).
It is believed that some factors in DOPS such as more practice in implementation of procedure, evaluations in several stages, and giving feedback in each stage can explain improvement of clinical skill among study participants.
Based on these results of Holmboe et al., DOPS leads to significant changes in students' behavior and competence improvement, as well as increase of teachers’ confidence and satisfaction compared to traditional methods (15, 16).
The results of the current study revealed the significant effect of giving feedback during educational evaluation. Scores of the second evaluation was significantly higher both qualitatively and quantitatively in the group that had received feedback regarding their strong and weak points compared to those who had not. However, after switching the method of evaluation in the second phase, the scores of both groups were at the same level in the third evaluation. Among the secondary outcomes of the study was that it seems that regular evaluation using DOPS itself can improve the skill score of the residents.
Limitations:
Among the limitations of the present study are the small sample size and evaluation of the residents by 2 separate attends. In addition, considering the previous familiarity of attends with residents we cannot be sure if a conflict of interest existed in their scoring or not. However, in this regard all efforts were made to minimize these limitations by randomly dividing the residents and giving explanations to the evaluators.
Conclusion:
Based on the findings of the present study, giving feedback after evaluating the second year emergency medicine residents regarding chest ultrasonography for trauma patients, led to a significant improvement in their scores in future evaluations and consequently their skill.
Acknowledgment
The authors would like to thank all the staff of Medical Education Faculty of Shahid Beheshti University of Medical Sciences and Clinical Research Development Center of Loghman Hakim Hospital for their cooperation.
Author contribution:
All authors passed four criteria for authorship contribution based on recommendations of the International Committee of Medical Journal Editors.
Conflict of interest:
None.
Funding:
None.
References
- 1.Abedini SAS, Aghamolaee T, Jome Zadeh A, Kamjo A. Clinical education problems from the perspective of nursing and midwifery students Hormozgan University of Medical Sciences. Journal of Hormozgan University ofMedical Sciences. 2009;12:249–53. [Google Scholar]
- 2.Crossley J, Humphris G, Jolly B. Assessing health professionals. Medical education. 2002;36(9):800–4. doi: 10.1046/j.1365-2923.2002.01294.x. [DOI] [PubMed] [Google Scholar]
- 3.Nouhi SFA, Foroud A, editors Investigation of the problems in clinical evaluation according to clinical teachers of medicine and dentists in Kerman University of Medical Sciences 6th Medical Education Congress. 6th Medical Education Congress S; 2003; Tehran. hahid Beheshti University of Medical Sciences; [Google Scholar]
- 4.Rushforth HE. Objective structured clinical examination (OSCE): review of literature and implications for nursing education. Nurse education today. 2007;27(5):481–90. doi: 10.1016/j.nedt.2006.08.009. [DOI] [PubMed] [Google Scholar]
- 5.Schoonheim-Klein M, Walmsley AD, Habets L, van der Velden U, Manogue M. An implementation strategy for introducing an OSCE into a dental school. European journal of dental education : official journal of the Association for Dental Education in Europe. 2005;9(4):143–9. doi: 10.1111/j.1600-0579.2005.00379.x. [DOI] [PubMed] [Google Scholar]
- 6.Smith-Strom H, Nortvedt MW. Evaluation of evidence-based methods used to teach nursing students to critically appraise evidence. The Journal of nursing education. 2008;47(8):372–5. doi: 10.3928/01484834-20080801-08. [DOI] [PubMed] [Google Scholar]
- 7.Franko DL, Cousineau TM, Trant M, Green TC, Rancourt D, Thompson D, et al. Motivation, self-efficacy, physical activity and nutrition in college students: randomized controlled trial of an internet-based education program. Preventive medicine. 2008;47(4):369–77. doi: 10.1016/j.ypmed.2008.06.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Heinzow HS, Friederichs H, Lenz P, Schmedt A, Becker JC, Hengst K, et al. Teaching ultrasound in a curricular course according to certified EFSUMB standards during undergraduate medical education: a prospective study. BMC medical education. 2013;13(1):1. doi: 10.1186/1472-6920-13-84. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians' clinical performance: BEME Guide No. 7. Medical teacher. 2006;28(2):117–28. doi: 10.1080/01421590600622665. [DOI] [PubMed] [Google Scholar]
- 10.Beard J, Strachan A, Davies H, Patterson F, Stark P, Ball S, et al. Developing an education and assessment framework for the Foundation Programme. Medical education. 2005;39(8):841–51. doi: 10.1111/j.1365-2929.2005.02236.x. [DOI] [PubMed] [Google Scholar]
- 11.Beard JD, Jolly BC, Newble DI, Thomas WE, Donnelly J, Southgate LJ. Assessing the technical skills of surgical trainees. The British journal of surgery. 2005;92(6):778–82. doi: 10.1002/bjs.4951. [DOI] [PubMed] [Google Scholar]
- 12.Barton JR, Corbett S, van der Vleuten CP, Programme EBCS. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointestinal endoscopy. 2012;75(3):591–7. doi: 10.1016/j.gie.2011.09.053. [DOI] [PubMed] [Google Scholar]
- 13.Thomas-Gibson S, Bassett P, Suzuki N, Brown G, Williams C, Saunders B. Intensive training over 5 days improves colonoscopy skills long-term. Endoscopy. 2007;39(09):818–24. doi: 10.1055/s-2007-966763. [DOI] [PubMed] [Google Scholar]
- 14.Liu C. An introduction to workplace-based assessments. Gastroenterology and Hepatology from bed to bench. 2012;5(1):24. [PMC free article] [PubMed] [Google Scholar]
- 15.Holmboe ES, Hawkins RE, Huot SJ. Effects of training in direct observation of medical residents' clinical competence: a randomized trial. Annals of Internal Medicine. 2004;140(11):874–81. doi: 10.7326/0003-4819-140-11-200406010-00008. [DOI] [PubMed] [Google Scholar]
- 16.Chen W, Liao S-c, Tsai C-h, Huang C-c, Lin C-c, Tsai C-h. Clinical skills in final-year medical students: the relationship between self-reported confidence and direct observation by faculty or residents. ANNALS-ACADEMY OF MEDICINE SINGAPORE. 2008;37(1):3. [PubMed] [Google Scholar]