Abstract
Background:
Despite the benefits of the objective structured assessment of technical skills (OSATS) and it appropriateness for evaluating clinical abilities of nursing students , few studies are available on the application of this method in nursing education.
Objectives:
The purpose of this study was to compare the effect of using OSATS and traditional methods on the students’ learning. We also aimed to signify students’ views about these two methods and their views about the scores they received in these methods in a medical emergency course.
Patients and Methods:
A quasi-experimental study was performed on 45 first semester students in nursing and medical emergencies passing a course on fundamentals of practice. The students were selected by a census method and evaluated by both the OSATS and traditional methods. Data collection was performed using checklists prepared based on the ‘text book of nursing procedures checklists’ published by Iranian nursing organization and a questionnaire containing learning rate and students’ estimation of their received scores. Descriptive statistics as well as paired t-test and independent samples t-test were used in data analysis.
Results:
The mean of students’ score in OSATS was significantly higher than their mean score in traditional method (P = 0.01). Moreover, the mean of self-evaluation score after the traditional method was relatively the same as the score the students received in the exam. However, the mean of self-evaluation score after the OSATS was relatively lower than the scores the students received in the OSATS exam. Most students believed that OSATS can evaluate a wide range of students’ knowledge and skills compared to traditional method.
Conclusions:
Results of this study indicated the better effect of OSATS on learning and its relative superiority in precise assessment of clinical skills compared with the traditional evaluation method. Therefore, we recommend using this method in evaluation of students in practical courses.
Keywords: Students, Evaluation, Learning
1. Background
Nowadays, the importance of nurses’ tasks and responsibilities in performing nursing cares attracted a lot of attention. It is necessary for nurses to make an evolution in their traditional roles to accommodate with the needs of the modern society (1). The nurses’ knowledge is visible during their work in clinical environment. Nursing education should prepare its students to enable them perform their professional responsibilities in a changing clinical setting (2-4).
Because of the importance of clinical training, appropriate clinical evaluation strategies should be adopted to determine the success rate of clinical training (5, 6). Such strategies are important in evaluating the quality of teaching and learning processes (7), screening students weaknesses, increasing their motivation and helping them increase their efforts in attainment of their educational goals and in assisting teachers to assess their activities (8-13).
Despite the importance of clinical evaluation, this task is still time-consuming and a baffling problem which relies mostly on teachers subjective judgments. Moreover, most of the trainees and students are not satisfied with the ways and the results of evaluation (5, 14, 15).
Evidence shows that most of the newly graduated nurses have sufficient theoretical knowledge but they are not clinically proficient (16). It has been shown that the objective evaluation methods are appropriate alternatives to traditional manners. These methods are especially effective in giving immediate feedbacks to students on their flaws (17). Although the objective methods of evaluation, such as objective structured practical examination (OSPE) and objective structured assessment of technical skills (OSATS) are time consuming and need more human and financial resources, but bring more satisfaction to both students and their teachers (12, 18).
OSATS is an examination for evaluating qualification in practical skills, which is organized objectively in different stations and students are asked to do special clinical tasks in each station. This method was firstly introduced in Toronto University by the department of surgery in 1990 (19). Despite the benefits of OSATS and it appropriateness for evaluating clinical abilities of nursing students (20), few studies are available on the application of this method in nursing education.
2. Objectives
The purpose of this study was to compare the effect of using OSATS and traditional methods on the students’ learning. We also aimed to assess students’ views about these two methods and their views about the scores they received in these methods in a medical emergency course.
3. Patients and Methods
This quasi-experimental study was performed in nursing and midwifery school of Gonabad University of Medical Sciences (GUMS) in 2013. The study sample included 45 first semester students in nursing and medical emergencies passing a course on fundamentals of practice. All students were selected by a census method. Inclusion criteria were willingness to participate in the study and having no excessive absences from the class sessions. An absence of more than three sessions was selected as the exclusion criteria.
For gathering data and evaluating students, checklists were prepared using the ‘text book of nursing procedures checklists’ (21). Ten faculty members in the nursing school of GUMS confirmed the appropriateness and content validity of checklists and its reliability was also confirmed using the Cronbach’s alpha coefficient (0.80). Checklists were a collection of questions, words or sentences related to each skill and assessors marked questions, expressions, or sentences that were more relevant to respondents performance. Moreover, we designed a questionnaire containing items about learning rate and students’ estimation of their received scores with respect to their evaluation. These items were responded on a four point Likert scale ranging from strongly agree (= 4) to completely disagree (= 1).
In this study, traditional exam was implemented at midterm and the OSATS at the end of the term. At the beginning of semester, the content of the course was divided into two parts, one to be taught in a half of the semester. Skills composed in each part of the content were matched regarding their levels of difficulty or easiness. Therefore, a half of the skills were taught at the first half of the semester and were then evaluated using traditional methods in midterm and the other half of skills taught at the second half of the semester were evaluated using the OSATS method at the end of the semester. In each exam (i.e. traditional of OSATS), every student was asked to select a skill randomly; then he or she was asked to prepare the needed tools and perform it on a mannequin. Scoring in traditional tests depends on the students’ ability to answer the oral questions, exact performance, having self-confidence and the speed of performance. Assessment was performed using prepared checklists and at the end, they provided a questionnaire to rate their learning, estimate their received scores with respect to their evaluation and express their opinions about this method.
After completion of the second half of the semester, the OSATA exam was performed. At this time, seven stations with trained examiners were prepared for testing. The examiners remained fixed to the stations for all the students. All examiners were faculty members with a Master of Science in nursing and were previously trained in the objective structural test of clinical skills, how to implement it, their responsibilities as examiners, the structure of the checklists, the structure and the time allocated to each station. At first, all students were gathered in a class to prevent information exchanging during the test. In each station, students read its guideline posted outside the station. The students were asked to perform what they were asked to do, leave the current station after hearing the bell voice and enter the next one and similarly turn in other stations too. Each station timed about 15 minutes. The examiner presented in each station rated the students’ performance using a specific checklists. After the last station, the students responded to the questionnaire about the evaluation method again and leaved the testing environment. Finally, an expert weighted the scores of the stations and summing up them to yield a total score for each student.
3.1. Ethical Considerations
The Ethics Committee of GUMS approved the study protocol. All ethical issues such as obtaining informed consent and avoiding plagiarism were followed. The respondents were assured of the confidentiality of their personal information in dissemination of the findings. All participants signed a written informed consent before participation in the study and were briefed about the purpose and methodology, including the benefits and the structure of OSATS.
3.2. Data Analysis
Data was analyzed using SPSS 13 (SPSS Inc., Chicago, IL, USA). Descriptive and inferential statistics were used. Moreover, paired t-test was used to compare the students’ mean scores on the two exams and independent samples t-test was used to compare the students’ mean scores for gender, field of study and marital status.
4. Results
The mean age of students was 20.70 ± 3.80 years with a range of 18 to 33 years. Sixty percent of students were nursing students and the remaining were studying medical emergencies. Most of the students were males (66.7%) and single (84.4%).
The mean of the students’ score in OSATS was significantly higher than their mean score in traditional method (P = 0.01). Moreover, the mean of self-evaluation score after the traditional method was relatively the same as the score the students received in the exam. However, the mean of self-evaluation score after the OSATS was relatively lower than the scores the students received in the OSATS exam (Table 1).
Table 1. Comparing the Mean and Standard Deviation of Students’ Scores According to Objective Structural Assessments and Traditional Assessments a.
| Type of Exam | Mean ± SD | Range of Score | P Value b |
|---|---|---|---|
| Self-evaluating score in traditional methods | 15.75 ± 2.38 | 8 - 20 | 0.07 |
| Self-evaluating score in OSATS | 16.81 ± 2.41 | 10 - 20 | |
| Final score of traditional exam | 15.20 ± 1.01 | 13 - 17 | 0.01 |
| Final score of OSATS exam | 17.93 ± 0.96 | 16 - 20 |
a Abbreviation: OSATS, objective structured assessment of technical skills.
b Paired t-test.
As Table 2 shows, no significant differences were found between the students’ mean scores in OSATS or in traditional method for their gender, field of study and marital status.
Table 2. Frequency Distribution and Mean Standard Deviation of Students’ Score for Their Gender, Field of Study and Marital Status a.
| Variable | Type of Exam | ||||
|---|---|---|---|---|---|
| No. (%) | OSATS | Traditional | |||
| Mean ± SD | P Value | Mean ± SD | P Value | ||
| Gender | 0.7 | 0.3 | |||
| Female | 15 (33.4) | 17.93 ± 0.98 | 5.27 ± 1.10 | ||
| Male | 30 (66.6) | 17.93 ± 0.96 | 15.17 ± 0.98 | ||
| Field of study | 0.3 | 0.8 | |||
| Nursing | 27 (60) | 18.19 ± 0.87 | 15.15 ± 0.98 | 15.15 ± 0.98 | |
| Medical emergency | 18 (40) | 17.56 ± 0.98 | 15.28 ± 1.07 | 15.28 ± 1.07 | |
| Marital status | 0.6 | 0.1 | |||
| Single | 38 (84.5) | 18.03 ± 0.94 | 15.18 ± 1.06 | ||
| Married | 7 (15.5) | 17.43 ± 0.97 | 15.29 ± 0.75 | ||
a Abbreviation: OSATS, objective structured assessment of technical skills.
Most students believed that OSATS can evaluate a wide range of students’ knowledge and skills compared to traditional method. Besides, they believed that this method provides better learning opportunities, but needs more practice to gain success. Moreover, 95.6% of students believed that the quality of OSATS was higher than traditional one and this test is more accurate in evaluating students’ clinical skills. In addition, 60% of them declared that the evaluation had an important effect on their learning. According to the students, if the students were aware of the evaluation method, they would try more to learn. The items “better evaluation of the student’s weakness” and “better evaluation of the students’ strengths” were selected more for the OSATS than the traditional method (Table 3). Moreover, 93.3% of the students were in agreement using OSATS in the next semesters.
Table 3. Frequency Distribution of Students’ Opinions About the Evaluation Methods a,b.
| Items | Type of Answer | |||||||
|---|---|---|---|---|---|---|---|---|
| Totally Agree | Agree | Disagree | Totally Disagree | |||||
| Traditional | OSATs | Traditional | OSATS | Traditional | OSATS | Traditional | OSATS | |
| Evaluation of wide range of skills and practical methods | 1 (2.2) | 15 (33.3) | 19 (42.2) | 28 (62.2) | 18 (40) | 2 (4.4) | 7 (15.6) | 0 |
| Increase in performance speed | 2 (4.4) | 12 (26.7) | 15 (33.3) | 30 (66.7) | 19 (42.2) | 3 (6.7) | 9 (20) | 0 |
| Needed time to learning | 2 (5) | 11 (24.4) | 7 (15) | 29 (64.4) | 27 (60) | 5 (11.1) | 9 (20) | 0 |
| Deeper learning compared to other methods | 7 (15) | 13 (28.9) | 9 (20) | 29 (64.4) | 18 (40) | 3 (6.7) | 11 (25) | 0 |
| Considering more detailed points | 10 (22.2) | 20 (44.4) | 23 (51.1) | 24 (53.3) | 9 (20) | 1 (2.2) | 3 (6.7) | 0 |
| Better evaluation of the student’s weaknesses | 10 (22.2) | 16 (35.6) | 19 (42.2) | 24 (53.3) | 12 (26.7) | 3 (6.7) | 4 (8.9) | 2 (4.4) |
| Better evaluation of the student’s strength | 11 (25) | 11 (24.4) | 16 (35) | 30 (66.7) | 13 (30) | 4 (8.9) | 5 (10) | 0 |
| Need to more practice to earn success | 5 (10) | 22 (48.9) | 7 (15) | 19 (42.2) | 22 (50) | 4 (8.9) | 11 (25) | 0 |
| More emphasis on practical skills | 9 (20) | 12 (26.7) | 22 (48.9) | 28 (62.2) | 9 (20) | 4 (8.9) | 5 (11.1) | 1 (2.2) |
| More precision in evaluating clinical skills | 4 (8.9) | 18 (40) | 7 (15.6) | 24 (53.3) | 20 (44.4) | 3 (6.7) | 14 (31.1) | 0 |
| Requiring oral questions in traditional method and written questions in OSATS | 6 (13.3) | 16 (35.6) | 24 (53.3) | 20 (44.4) | 9 (20) | 8 (17.8) | 6 (13.3) | 1 (2.2) |
a Abbreviation: OSATS, objective structured assessment of technical skills.
b All data presented as No. (%).
5. Discussion
The present study showed that the mean score of students in OSATS was significantly higher than the traditional method. Moreover, in this study, we could not find any significant differences between students’ mean scores regarding variables such as gender, field of study and marital status. Our finding on the difference between the two methods was consistent with the results of some previous studies (19, 22-25). For instance, Rahman et al. in a study on 400 physiology students compared the OSPE and traditional method and reported that the students’ score was significantly higher in OSPE than traditional evaluations (26). In contrast, Pishkar et al. compared the OSPE and traditional method and reported that students’ score in traditional evaluation method was higher than OSPE (22).
In the current study, 62.2% of students believed that OSATS could evaluate a wide range of their skills. Similar findings were reported in studies conducted by Imani et al. (27), Pierre et al. (28), and Menezes et al. (29) who investigated other objective evaluation methods such as Objective Structured clinical examination (OSCE) and OSPE.
Most students participated in the present study believed that the OSATS was superior to the traditional method for being more precise, better evaluation of students strengths and weaknesses and inducing deeper learning. Although at the time of this study, no similar studies were available on OSATS, our findings are consistent with results of Pierre et al. (28) and Schoonheim-Klein et al. (30) who studied the OSCE method, which is similar to the OSATS in its nature and structure. On the other hand, most students in a previous study reported that the objective assessment method used (i.e. OSCE) was not useful to them (31). As reported by Alinier et al. (32), the OSATS has some limitations, but these limitations should not be an obstacle in using this useful method.
In conclusion, the results of this study indicated the better effect of OSATS on learning and its relative superiority in precise assessment of clinical skills compared with the traditional evaluation method. Therefore, we recommend using this method in evaluation of students in practical courses.
This study had some limitations such as limited number of students who participated in the study, not having a control group, and not using a randomized design. Then the results may not fully be generalized to all students. Given the results of the present study, we suggest to conduct similar research with a control group and a larger sample size.
Acknowledgments
This article was retrieved from corresponding author’s thesis to receive M.Sc. degree in nursing education in Gonabad University of Medical Sciences. Special thanks to all teachers, professors, officials, students and Gonabad University of Medical Sciences’ staff and training center of clinical skill for their helping and support. The project of this study was approved at GUMS with the code of p/4/20.
Footnotes
Authors’ Contributions:Mohammad Reza Mansoorian contributed in the idea of research, designed the study, interpreted the data and prepared initial draft of manuscript and its revision. Shahla Khosravan and Mehri Alaviani contributed in the idea of research and designed the study. Ali Alami performed the design study, analyzed and interpreted data and prepared initial draft of manuscript and its revision. Marzeih Sadat Hosseiny contributed in the idea of research, collected data, interpreted the data and prepared the initial draft of manuscript and its revision. The three authors approved the final version of manuscript for submission.
Funding/Support:This research was approved and granted by the Deputy of Nursing and Midwifery of Gonabad University of Medical Sciences and the authors are grateful for their support.
References
- 1.Farzianpour F, Monzavi A, Yassini E. Evaluating the quality of education at dentistry school of tehran university of medical sciences. Dent Res J . 2011;8(2):71–9. [PMC free article] [PubMed] [Google Scholar]
- 2.Butler MP, Cassidy I, Quillinan B, Fahy A, Bradshaw C, Tuohy D, et al. Competency assessment methods - tool and processes: a survey of nurse preceptors in Ireland. Nurse Educ Pract. 2011;11(5):298–303. doi: 10.1016/j.nepr.2011.01.006. [DOI] [PubMed] [Google Scholar]
- 3.Liang L. The gap between evidence and practice. Health Aff. 2007;26(2):119–21. doi: 10.1377/hlthaff.26.2.w119. [DOI] [PubMed] [Google Scholar]
- 4.Chiang VC, Chan SS. An evaluation of advanced simulation in nursing: a mixed-method study. Collegian. 2014;21(4):257–65. doi: 10.1016/j.colegn.2013.05.003. [DOI] [PubMed] [Google Scholar]
- 5.Noohi E, Motesadi M, Haghdoost AA. [Clinical Teachers' Viewpoints towards Objective Structured Clinical Examination in Kerman University of Medical Science]. Iran J Med Educ. 2008;8(1):113–20. [Google Scholar]
- 6.Sayf AA. Measurement, evaluation and educational Evaluation. Tehran: Doran Publication; 2007. [Google Scholar]
- 7.Iqbal M, Khizar B, Zaidi Z. Revising an objective structured clinical examination in a resource-limited Pakistani Medical School. Educ Health. 2009;22(1):209. [PubMed] [Google Scholar]
- 8.Amin Z, Chong YS, Khoo HE. Practical guide to medical student assessment. World Scientific; 2006. Avilable from: http://www.mums.ac.ir/shares/meddept/meddept/E-Books/ZubairAminPracticalGuidetoMedicalStudentAssessment.pdf. [DOI] [Google Scholar]
- 9.Cox M, Irby DM, Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387–96. doi: 10.1056/NEJMra054784. [DOI] [PubMed] [Google Scholar]
- 10.Joyce B. Developing an assessment system: facilitator's guide. Accreditation Council Grad Med Educ. 2006;1:15–7. [Google Scholar]
- 11.Aronson L, Niehaus B, Hill-Sakurai L, Lai C, O'Sullivan PS. A comparison of two methods of teaching reflective ability in Year 3 medical students. Med Educ. 2012;46(8):807–14. doi: 10.1111/j.1365-2923.2012.04299.x. [DOI] [PubMed] [Google Scholar]
- 12.AbdiShahshahani M, Ehsanpour S, Yamani N, Kohan S. The evaluation of reproductive health PhD program in Iran: The input indicators analysis. Iran J Nurs Midwifery Res. 2014;19(6):620–8. [PMC free article] [PubMed] [Google Scholar]
- 13.Fernandez-Sola C, Granero-Molina J, Marquez-Membrive J, Aguilera-Manrique G, Castro-Sanchez AM. [Implementation of the new clinical practice training model in Andalusia: a qualitative evaluation of the Nursing and Physiotherapy degrees]. Enferm Clin. 2014;24(2):136–41. doi: 10.1016/j.enfcli.2013.09.004. [DOI] [PubMed] [Google Scholar]
- 14.Wilkinson TJ, Frampton CM. Comprehensive undergraduate medical assessments improve prediction of clinical performance. Med Educ. 2004;38(10):1111–6. doi: 10.1111/j.1365-2929.2004.01962.x. [DOI] [PubMed] [Google Scholar]
- 15.Cheater FM, Baker R, Reddish S, Spiers N, Wailoo A, Gillies C, et al. Cluster randomized controlled trial of the effectiveness of audit and feedback and educational outreach on improving nursing practice and patient outcomes. Med Care. 2006;44(6):542–51. doi: 10.1097/01.mlr.0000215919.89893.8a. [DOI] [PubMed] [Google Scholar]
- 16.Akhtar-Danesh N, Valaitis R, O'Mara L, Austin P, Munroe V. Viewpoints about collaboration between primary care and public health in Canada. BMC Health Serv Res. 2013;13:311. doi: 10.1186/1472-6963-13-311. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Rushforth HE. Objective structured clinical examination (OSCE): review of literature and implications for nursing education. Nurse Educ Today. 2007;27(5):481–90. doi: 10.1016/j.nedt.2006.08.009. [DOI] [PubMed] [Google Scholar]
- 18.Berkenstadt H, Ziv A, Gafni N, Sidi A. Incorporating simulation-based objective structured clinical examination into the Israeli National Board Examination in Anesthesiology. Anesth Analg. 2006;102(3):853–8. doi: 10.1213/01.ane.0000194934.34552.ab. [DOI] [PubMed] [Google Scholar]
- 19.Niitsu H, Hirabayashi N, Yoshimitsu M, Mimura T, Taomoto J, Sugiyama Y, et al. Using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale to evaluate the skills of surgical trainees in the operating room. Surg Today. 2013;43(3):271–5. doi: 10.1007/s00595-012-0313-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Hasan S, Malik S, Hamad A, Khan H, Bilal M. Conventional/traditional practical examination (CPE/TDPE) versus objective structured practical evaluation (OSPE)/semi objective structured practical evaluation (SOSPE). Pak J physiol. 2009;5(1):58–64. [Google Scholar]
- 21.Mirzabeigi GH, Shams SH, Jafari Y, Shirazi F, Heidari SH. [Text book of nursing procedures checklist]. Tehran: Nursing organization of Islamic republic of Iran; 2009. [Google Scholar]
- 22.Pishkar Mofrad Z, Navidian A, Robabi H. [An assessment of traditional and objective structured practical evaluation methods on satisfaction of nursing students in Zahedan Faculty of Nursing and Midwifery: A comparing]. J Med Educ Dev. 2013;7(4):2–14. [Google Scholar]
- 23.Zare A, Masooleh SR, Chehrzad MM, Roshan ZA. Chehrzad MM, Roshan ZA. [Comparing two methods of evaluation, Objective structured practical examination and traditional examination, on the satisfaction of Students in Shahid Beheshti faculty of nursing and midwifery]. Two Q Nurs Midwifery Sch Gilan. 2008;18(59):22–30. [Google Scholar]
- 24.Chipman JG, Schmitz CC. Using objective structured assessment of technical skills to evaluate a basic skills simulation curriculum for first-year surgical residents. J Am Coll Surg. 2009;209(3):364–70. doi: 10.1016/j.jamcollsurg.2009.05.005. [DOI] [PubMed] [Google Scholar]
- 25.Buerkle B, Pueth J, Hefler LA, Tempfer-Bentz EK, Tempfer CB. Objective structured assessment of technical skills evaluation of theoretical compared with hands-on training of shoulder dystocia management: a randomized controlled trial. Obstet Gynecol. 2012;120(4):809–14. doi: 10.1097/AOG.0b013e31826af9a9. [DOI] [PubMed] [Google Scholar]
- 26.Rahman N, Ferdousi S, Hoq N, Amin R, Kabir J. Evaluation of objective structured practical examination and traditional practical examination. Mymensingh Med J. 2007;16(1):7–11. doi: 10.3329/mmj.v16i1.239. [DOI] [PubMed] [Google Scholar]
- 27.Imani M, Hosseini Tabatabaie MT. Is osce Successful in pediatrics. Med Edu J. 2005;6(2):153. [Google Scholar]
- 28.Pierre RB, Wierenga A, Barton M, Branday JM, Christie CD. Student evaluation of an OSCE in paediatrics at the University of the West Indies, Jamaica. BMC Med Educ. 2004;4:22. doi: 10.1186/1472-6920-4-22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Menezes RG, Nayak VC, Binu VS, Kanchan T, Rao PP, Baral P, et al. Objective structured practical examination (OSPE) in Forensic Medicine: students' point of view. J Forensic Leg Med. 2011;18(8):347–9. doi: 10.1016/j.jflm.2011.06.011. [DOI] [PubMed] [Google Scholar]
- 30.Schoonheim-Klein ME, Habets LL, Aartman IH, van der Vleuten CP, Hoogstraten J, van der Velden U. Implementing an Objective Structured Clinical Examination (OSCE) in dental education: effects on students' learning strategies. Eur J Dent Educ. 2006;10(4):226–35. doi: 10.1111/j.1600-0579.2006.00421.x. [DOI] [PubMed] [Google Scholar]
- 31.Faryabi J, Farzad M, Sinaei N. [Kerman Dental School Students' comments about the evaluation method of objective structured clinical exam (OSCE)]. Strides Dev Med Educ. 2009;6(1):34–9. [Google Scholar]
- 32.Alinier G, Hunt WB, Gordon R. Determining the value of simulation in nurse education: study design and initial results. Nurse Educ Pract. 2004;4(3):200–7. doi: 10.1016/S1471-5953(03)00066-0. [DOI] [PubMed] [Google Scholar]