The effect of direct observation of procedural skills/mini-clinical evaluation exercise on the satisfaction and clinical skills of nursing students in dialysis

Abstract

BACKGROUND:

Since the purpose of medicine science is health promotion, education of this group is very important. The use of new evaluation methods is one of the first educational needs. Given that many conventional clinical evaluation methods are not able to fully evaluation students in the clinical settings and only evaluate limited information, This study is designed to evaluate the impact of direct observation of procedural skills (DOPS) and mini-clinical evaluation exercise (mini-CEX) on nursing students and their clinical satisfaction skills.

MATERIALS AND METHODS:

This quasi-experimental study was conducted in 2018 to evaluate the dialysis unit of the 6^th semester nursing students of Babol University of Medical Sciences in Iran. Samples were selected by the census method and then random allocation. Data collection was performed using two questionnaires and two checklists to evaluate the clinical skills and satisfaction of the two groups of testing and control. Moreover, the data were analysed with using the SPSS software (version 18, IBM SPSS Inc., Chicago, IL, USA) through the descriptive and analytical statistics (Chi-square test and t-test).

RESULTS:

The results showed that the intervention group compared with the control evaluation score of 13/73 ± 2/44, 11/74 ± 2/43, P < 0.002, respectively. Furthermore, the mean score of satisfaction of the intervention group with the DOPS method compared to the traditional method of the control group was 72/50 ± 7/31, 63/48 ± 9/31, P < 0.001, respectively. As well as, the mean score of satisfaction with the mini-CEX method of the intervention group compared to the traditional method of the control group was 73/28 ± 7/78, 63/48 ± 9/31, P < 0.002, respectively.

CONCLUSIONS:

The results of this study showed that students who were evaluated by DOPS and mini-CEX methods had a higher score of clinical performance evaluation and higher level of satisfaction. Nursing professors can improve the learning process and the satisfaction of nursing students by using direct feedback on such evaluation methods.

Introduction

Today, providing education to students is under debate in numerous countries. Moreover, since the main goal of medical science is to improve health status, education is of utmost importance. Hence, one of the basic educational needs is adopting new policies and methods to workforce students and improve their performance.[1] An important part of student education is dedicated to clinical education, so identifying the factors that affect the quantity and quality of clinical education can help solve relevant problems.[2]

Evaluation is an integral and fundamental part of medical education. It allows us to decide how much students have learned and whether they have reached the required standards. Today, competency-based curricula in the medical education are also expected to be considered, so it is recommended that methods be used to evaluation students who assess actual competence and performance.[3] Effective evaluation not only enhances students' motivation, but also helps the teacher evaluate their activities. Many conventional clinical evaluation methods are not able to fully assess students in the clinical settings and only assess limited information. While in clinical education, acquiring the necessary ability in the face of the patient, illness and how to manage it is one of the basic necessities.[1]

There is a wide range of evaluations that can be used to influence feedback on the effectiveness of training and education programs.[4] Providing feedback during the evaluation is especially important for student learning progress.[5] At present, numerous methods have been designed to evaluate students, including portfolio test, objective structured clinical examination, mini-clinical evaluation exercise (Mini-CEX), and direct observation of procedural skills (DOPS).[6]

DOPS is known as one of the recent practical evaluation methods suitable for providing an opportunity to give constructive feedback and direct students' concentration and attention towards necessary points to fulfill the required skills.[7] DOPS is a student-centered evaluation method that reinforces self-reliant learning because the student must identify his or her own learning needs and also choose the evaluation method, time, and location. In other words, DOPS provides the opportunity to learn, monitor, and respond.[3] Accordingly, this issue may be to a large extent related to the feedback provided to students.[8] Other studies have also highlighted that DOPS is a useful tool for the evaluation practical skills which is endowed with desirable reliability and acceptability; however, further research is required to establish its educational effectiveness and efficiency as a tool based on the clinical needs.[9]

Mini-CEX is one of the new evaluation tools that are effective because it has immediate feedback.[10] In the Mini-CEX evaluation method a clinical faculty member normally evaluates learners' performance as they encounter patients for 15–20 min. The main objective of this method is to provide feedback based on observed performance. Each encounter with a patient lasts 15 min and 5–10 min is also allocated to giving feedback to learners. Therefore, if feedback on student evaluation is not provided, it may not work well.[11]

The clinical nursing education planners further believe that clinical education is the foundation of nursing education.[12] Hence, the more enriched the clinical education, the more efficient the current students as tomorrow nurses.[9] Observation and evaluation of learners during performing procedures on patients and giving appropriate feedback to them can thus lead to obtaining and improving skills among them.[13] Feedback after the direct observation is very meaningful, especially when it addresses a learner's immediate concerns and provides information to the student that leads to improved learning.[14] The implementation of any of the evaluation tools depends on students' acceptance under its effect. Students' attitudes and levels of satisfaction with activities such as evaluation can also influence quality of education, learning, and skill acquisition. Having knowledge about students' satisfaction with educational issues is also of utmost importance which can lead to fulfillment of educational goals.[15] Students generally welcome the opportunity to be observed by experienced individuals and to receive immediate feedback.[16]

The results of research studies on evaluation in Iran have revealed that not only the principles of students' performance are ignored in majority of evaluation methods, but also the use of traditional methods in evaluates has led to waste of time and money with no effects on improvement of quality of care and correction of performance.[1] Considering the limitations of traditional evaluation methods as well as lack of texts about the impacts of novel evaluation methods on levels of satisfaction and clinical skills among nursing students, in the present study the effects of DOPS/mini-CEX and traditional methods on the levels of satisfaction and clinical skills among nursing students was compared.

Materials and Methods

This quasi-experimental study was conducted in 2018 to evaluate the dialysis unit of the 6^th semester nursing students of Babol University of Medical Sciences in Iran. Samples were selected by the census method and then random allocation. The inclusion criteria were selection of dialysis internship course (semester 6) and also willingness to participate in the study. According to the census sampling method, all the students enrolled in the sixth semester of nursing taking dialysis internship course for 3 weeks were selected as study samples then with random allocation divided into two groups: intervention group (n = 32) and control group (n = 31) from the beginning to the end of the semester using a coin toss to determine odd or even number of the groups. Due to the small number of the students in the dialysis internship course, an entire semester was considered in this study. In order to comply with ethical principles, the students in the intervention group were informed that participation in the study was optional and if they did not wish to be included in the intervention group for any reasons, they could be moved to the control group and this did not have any effects on their scoring. It should be noted that no displacement occurred. The purpose and the methodology of the study and how to use the evaluation tools and DOPS/mini-CEX were also presented to the students orally and in writing. This research project was approved with the code IR.SBMU.PHARMACY.REC.1398.146 at the Shahid Beheshti University of Medical Sciences.

Among the skills required based on educational and behavioral goals in dialysis nursing curriculum-based internship course, three skills were selected for evaluation using DOPS and three skills were chosen for mini-CEX. Such skills for DOPS were setting and priming a dialysis machine, connecting a patient to a dialysis machine, and disconnecting a patient from a dialysis machine. The given skills evaluated through mini-CEX were preparation of equipment, treatment of cramps during dialysis, and treatment of hypertension during hemodialysis. These skills were selected since the chance to do so by the students in other units was very low and they were in fact specific to dialysis. In the implementation method, after teaching the required skills to students and their fulfillment in the intervention group, they were evaluated using DOPS and mini-CEX. This means that, in the first step, the skills were observed within 15 min and feedback was given for 5 min. In the second step, the same skill was observed again after 1 week and the strengths and weaknesses of the students were presented to them through feedback, and finally, in the third step, 1 week later, the same skills were repeated and evaluated and the final scores were assigned. The skills were also taught to individuals in the control group through the traditional method, and the evaluation was completed in just one step in a routine form. The DOPS was fulfilled through 15 steps to connect a patient to a dialysis machine, 14 steps to remove a machine, and 11 steps to set and prime a machine. The mini-CEX was fulfilled using 9 steps to prepare the equipment, 6 steps to provide treatment for cramps during dialysis, and 6 steps to provide treatment for hypertension during dialysis through a 4-point Likert-type scale from very favorable to unfavorable which was scored through cooperation with students and teachers. Each domain had a maximum of 3 scores, so that the score obtained by each student at the end of was 18 by maximum.

Besides, the students in the intervention group were asked to complete the satisfaction questionnaire through DOPS and mini-CEX which contained 18 items using a 5-point Likert-type scale. The control group students were also asked to answer the satisfaction questionnaire items using the traditional method. The content validity index (CVI) and the content validity ratio (CVR) of the initial checklists and the research instruments were also confirmed by 10 professors, head nurses, and experienced nurses working in the dialysis unit. Based on the Lawshe's table, the minimum value of CVR, if the number of people determining the content was 10 individuals, was by 0.62.[17] Accordingly, the statements that obtained the minimum score were selected. To examine CVI using the given formula, the score higher than 0.79 was considered good, and the scores 0.70–0.79 and those below 0.70 needed revisions and were removed, respectively.[18] The CVI-CVR of DOPS/mini-CEX checklists and those of the satisfaction questionnaire using DOPS/mini-CEX were more than 0.85 as reported by the experts. To determine the reliability of DOPS/mini-CEX checklists, the observer agreement was determined using the Cohen's kappa coefficient by 0.89. To determine the reliability of the questionnaire, Cronbach's alpha was used which was α = 0.89, α = 0.92, and α = 0.87 for DOPS/mini-CEX and satisfaction questionnaires and the traditional method; respectively. The data were analyzed using the SPSS Statistics software (version 18, IBM SPSS Inc., Chicago, IL, USA). To describe the data, frequency, percentage, mean, and standard deviation were employed. To test the hypotheses for qualitative and quantitative variables, the Chi-square test and Shapiro–Wilk normality test were, respectively, fulfilled. If the variables were normal, parametric independent t-test was used and if the variables were not normal, the nonparametric Mann–Whitney U-test was employed. In these tests, 0.05 was considered as the significance level.

Results

Of the total number of 63 students, 52% were women and 71% were single. There was no significant statistical difference between the sex groups and marital status using the Chi-square test. The mean age of the intervention group compared with the control group was 23.5 and 22 years, respectively, which was not a statistically significant difference according to the results of Mann–Whitney U-test [Table 1].

Table 1.

Distribution of demographic variables in the intervention and control groups

Demographic variables	Intervention group	Control group	P
Age (year), mean±SD	23.50±4.52	22±0.856	0.84
Gender, frequency (%)
Male	13 (41.9)	17 (53.1)	0.38
Female	18 (58.1)	15 (46.9)
Marital status
Single	23 (74.2)	22 (68.8)	0.63
Married	8 (25.8)	10 (31.2)

SD=Standard deviation

The mean final scores of DOPS/mini-CEX steps of the intervention and control groups were 13.73 and 11.74, respectively. There were also statistically significant differences between the intervention group and the control group, according to the results of the independent t-test (P < 0.002). Moreover, the levels of satisfaction among students in the intervention group using DOPS/mini-CEX compared with the traditional method by the control group showed a statistically significant difference based on the results of the independent t-test (P < 0.001), as shown in Table 2.

Table 2.

Comparison of assessment through direct observation of procedural skills and mini-clinical evaluation exercise steps and levels of satisfaction in intervention and control groups

Variables	Groups	Mean±SD	P
DOPS/mini-CEX evaluation	Intervention	13.73±2.438	<0.002
	Control	11.74±2.435
DOPS levels of satisfaction traditional method, levels of satisfaction	Intervention	72.500±7.3132	<0.001
	Control	63.483±9.3197
Mini-CEX levels of satisfaction traditional method, levels of satisfaction	Intervention	73.281±7.7884	<0.001
	Control	63.483±9.3197

DOPS=Direct Observation of Procedural Skills, CEX=Clinical Evaluation Exercise, SD=Standard deviation

Discussion

The purpose of this study was to examine the effects of DOPS/mini-CEX compared with the traditional method on improving the clinical skills and levels of satisfaction in nursing students through evaluation methods. The findings revealed that both groups were homogeneous in terms of demographic information, so that no statistically significant differences were observed between the two groups in terms of gender, age, and marital status which were in line with the study by Habibi et al. (2015).[19]

The results of this study showed that evaluation using DOPS and mini-CEX was more effective in terms of improving skills among nursing students compared with the traditional method. Furthermore, the results of the present study were consistent with the findings by Habibi et al. in which the use of modern evaluation methods, i.e., DOPS/mini-CEX and the traditional method on clinical skills of nursing students were compared.[20] The findings of another study by Lörwald et al. revealed that mini-CEX and DOPS lead to positive effects on student performance.[21] The results of the study by Kuhpayehzadeh et al. showed that the use of mini-CEX could generally lead to a significant progress among interns in the emergency department.[22] Moreover, this study found that feedback content could make mini-CEX as a rich evaluation instrument, and it was of great value in terms of critical and supportive feedback. Contrary to the findings of the present study, Bindal et al. reached to this conclusion that the given evaluation method was not a valuable educational tool and highlighted the use of a clinically-based evaluation instrument that needed sufficient planning and time allocation.[23] What was of importance in the study by Bindal et al. was that only 76% of the students and 65% of the consultants responded the questionnaires. As well, 43% of the consultants and 33% of the students had not received any education in the domain of DOPS, the evaluations were normally unplanned, and the time for DOPS evaluation was also short.

According to the authors, gaining more scores in the intervention group can be considered as the effect of feedback factors and learning environment, which includes the relationship between the student and the teacher or learning culture.[24] Because students have a better understanding of their strengths and weaknesses after oral feedback, this can strengthen students' learning. Kumar et al. also noted that perhaps the most important advantage of DOPS is immediate feedback in the form of constructive suggestions and opinions in the field of health care.[25] In addition, Vafaei et al. 2017 noted that giving feedback to students will significantly improve their grades in subsequent evaluation and thus their skills.[26]

Of the other findings of the present study was the significant difference in the levels of satisfaction among the students in the intervention group than the control group. This meant that students using DOPS/mini-CEX had higher levels of satisfaction than those employing the traditional method. In agreement with the recent study, the results of the investigation by Hoseini et al. on the clinical experiences of undergraduate students with mini-CEX in Iran showed that the midwifery undergraduate students were relatively satisfied with mini-CEX.[15] Of the strengths pointed out by the students were its motivational effects due to being objective as well as the relatively positive opinions on its feasibility. Although some students had stated that mini-CEX was time-consuming, it was argued that sequential evaluation of all capabilities in this test could bring up this idea.[15] Moreover, Erfani Khanghahi and Ebadi Fard Azar in the study concluded that most participants were satisfied with DOPS and also pointed out that DOPS might be used as a valuable and effective evaluation method in medical education. They also put emphasis on greater attention to these tests.[7] Moreover, in the study by Jalili et al., 79% of the students were satisfied with DOPS, and the rest showed moderate-to-high levels of satisfaction.[16]

May be what leads to more satisfied students in the intervention group than the control group evaluation was found to provide feedback and motivate students. In support of this statement Erfani Khanghahi and Ebadi Fard Azar (2018) in their study noted that one possible reason for the relatively good scores can motivate participants by the evaluation method.[7]

Limitations

One of the strengths of DOPS and mini-CEX evaluation methods is the provision of feedback to participants and the promotion of independence and practical skills during evaluation. Because it improves teamwork, increases clinical skills and deep learning of students. However, stressful and time-consuming to use such methods teachers and lack of interest can be considered as the main drawback to their use. One limitation of this study is that only students of Babol University of Medical Sciences were studied. Due to the nature of the study, no pretests were taken from the students. Therefore, generalizations must be made with caution.

Ethical consideration

This article is the result of a research project approved by Shahid Beheshti University of Medical Sciences (Code: IR. SBMU. PHARMACY. REC.1398.146). After being introduced to the hospital, the researcher explained the purpose of the study to the students. Ensuring that ethical principles were kept confidential, students' satisfaction with publishing the results of the study's findings was satisfactory.

Conclusions

The results of this study showed that students who were evaluated by DOPS and mini-CEX more scores obtained from clinical skills. Furthermore, the satisfaction rate of students who were evaluated by DOPS and mini-CEX methods was higher than students with traditional evaluation methods. It is recommended that DOPS and mini-CEX methods replace traditional methods for evaluating all nursing procedures in different periods. Nurse instructor to provide feedback during the evaluation can achieve the objectives of each course. With the help of these methods, participatory evaluation methods between the student and the instructor can replace the one-sided evaluation by the instructor. Nursing educators using DOPS and mini-CEX evaluation can strengthen the motivation of students in their clinical skills.

Financial support and sponsorship

The study was financially supported by the Shahid Beheshti University of Medical Sciences Tehran, Iran.

Conflicts of interest

There are no conflicts of interest.