Comparison of A Mini-Clinical evaluation exercise with the routine logbook in assessing clinical competence of operating room trainees: A parallel randomized controlled trial

Abstract

Background

Enhancing the skills of future operating room students, who will have a crucial role in surgical procedures, necessitates the adoption of innovative training techniques. This study aimed to comparison of the mini-CEX with the routine logbook in assessing clinical competence of operating room trainees.

Methods

This research was a parallel randomized controlled trial conducted on undergraduate students in the operating room department of Khomein Faculty of Medical Sciences in Iran from June 2022 to January 2024. The participating students (60 individuals) were divided into two groups: Mini-CEX (30 individuals) and Logbook (30 individuals). Students were assessed by implementing the Logbook method in the control group and the Mini-CEX method in the intervention group. Both methods were implemented in three steps with a one-week interval. Data were analyzed by SPSS-16 using descriptive statistics, chi-square test, t-test and repeated measures ANOVA with post-hoc tests.

Results

There were no statistically significant differences in terms of gender, age, and grade point average between students in the two groups (p > 0.05). The mean scores of final assessments in the experimental group (by Mini-CEX methods) were significantly (P < 0.001) higher than the control one (Logbook method).

Conclusion

It seems that applying the Mini-CEX method resulted in the improvement of clinical skills among students. Accordingly, operating room professors can improve the learning process by using direct feedback on such evaluation methods.

Clinical trial number

not applicable.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12909-025-07560-6.

Introduction

Clinical education is pivotal in honing the professional skills of operating room students, who need to undergo training in various surgical interventions [1–3]. Enhancing the skills of future operating room students, who will have a crucial role in surgical procedures, necessitates the adoption of innovative training techniques [4, 5]. Evaluation Assessment is a fundamental aspect of medical education [6]. Traditional assessments tend to prioritize basic or clinical knowledge over competence in real clinical settings, making it difficult to accurately assess clinical competence [7]. It is crucial to monitor students’ progress during practicum to ensure they are gaining exposure to the necessary medical skills and procedures required [8]. Logbooks document students’ progress through the curriculum, detailing their observation and practice of a wide range of clinical skills and procedures [9]. Several methods have been implemented to assess clinical performance, with the Mini-Clinical Evaluation Exercise (Mini-CEX) being among them. Widely recognized for its validity and reliability, the Mini-CEX serves as an effective tool for evaluating clinical skills, enhancing assessor-student relationships, enabling meaningful feedback, pinpointing performance issues, and promoting learning [10]. This evaluation technique is widely embraced in Western nations, requires less time than traditional case presentations, and provides structured feedback that aids both students and professors in making informed decisions [11]. Hejri’s study aimed to investigate the use of mini-CEX for assessing undergraduate and graduate medical trainees and showed that this method has acceptable validity, reliability, and educational impact [12]. The heavy responsibilities and increased expectations of competence placed on operating room staff include managing workload, maintaining speed and precision in operations, following a variety of procedures, and ensuring rapid patient discharge [13]. Additionally, operating rooms can be dangerous environments for patients, with some postoperative complications resulting from clinical incompetence, potentially leading to patient mortality and injury [14, 15]. Regular assessment of operating room staff' competencies helps them identify their needs for additional knowledge, skills, personal development, and integrated learning experiences, enabling them to recognize both their strengths and weaknesses [16, 17]. Competency assessment is also important in terms of ethical values ​​and professional accountability [18]. As per Ahmadi’s study, which aimed to evaluate the clinical competence of operating room staff, the lowest score was associated with the general practical skills dimension [15]. Currently, operating room technology students are evaluated by their supervisors at the conclusion of their internship. However, there is no comprehensive assessment available to evaluate them in terms of clinical skills or competencies [19, 20]. An assessment of this nature necessitates the utilization of methods and tools capable of overall skill level generalization. However, according to the evidence, no singular method has been suggested in this domain [6]. This study aimed to comparison of the mini-CEX with the routine logbook in assessing clinical competence of operating room trainees.

Method

Study design

This research was a parallel randomized controlled trial conducted on undergraduate students in the operating room department of Khomein Faculty of Medical Sciences, Markazi Province in Iran from June 2022 to January 2024.

Participants

Our study sample comprised undergraduate operating room nursing students. Sampling was conducted using a census method involving all students enrolled in the undergraduate operating room curriculum, with no participants declining to take part. The authors utilized the G*power 3.1.9.4 tool from Heinrich-Heine-Universität Düsseldorf, Germany, to calculate the sample size. Specifically, the F-test of repeated measures ANOVA was analyzed with the following specified parameters: an alpha probability of 0.05, power of 0.8, effect size of 0.25, and correlation of 0.5. Based on these parameters, the software determined that a sample size of 28 cases was required.

In total, 60 students participating in the Orthopedic Surgery clerkship course were included in the study. Inclusion criteria consisted of students who had passed the theoretical components of scrubbing and the orthopedic surgery course, students in their final year of bachelor’s studies (7th and 8th semesters), and those willing to participate. Students with a certificate in “evaluation methods”, working in surgical centers, those on transfers, or visiting students were excluded from the study.

Randomization and blindness

After completing a written consent form, students were divided into intervention and control groups, each with 30 participants. For random allocation, alternate block randomization [21] was used. Students were categorized by gender and randomly divided into two intervention and control groups using an internet and computer-based random sequence with an allocation ratio of 1:1. Given the nature of the intervention in the present study, blinding was not possible. The design and implementation of this study was based on the CONSORT guidelines (Fig. 1).

Fig. 1.

The flow diagram of the study

Instrument and data collection

The study comprised two main phases: (1) Preparation of tools and assessor training, and (2) Implementation of the evaluation program. During the initial phase, extensive research was conducted through a comprehensive library and electronic search to validate the checklists. The research focused on how scrub nurses implement the Mini-CEX and Logbook evaluation methods. Subsequently, an initial draft of the checklist was prepared for each instrument.

The Mini-CEX method was conducted using a checklist consisting of 31 items which evaluated six different domains. The purpose of completing this checklist was to assess how the Mini-CEX test impacts the clinical evaluation of trainees in the operating room during orthopedic surgeries. This checklist is specifically created to facilitate the administration of the Mini-CEX test and evaluate scrub nurses. The Mini-CEX test is a workplace-based evaluation tool that allows assessors to directly observe learners’ job skills without interference. Evaluation scoring is conducted using a 5-point Likert scale in alignment with the scoring guide provided to the assessor. Lastly, immediate written and oral feedback is given to the students by the assessor.

These domains included the following:

• - Sterilization (5 items).

• - Preparing the patient and surgical field for surgery (3 items).

• - Organizing instruments and Table (7 items).

• - Team communication (4 items).

• - Performance management during the procedure (7 items).

• - Completion of the surgical procedure in the final third of the surgery time (5 items) (refer to Supplementary Appendix 1 for details).

The Mini-CEX checklist items evaluated scrub nurses based on a 5-point scale: 0–1 below standard, 2–3 standard, and 4–5 above standard. The performance was interpreted as follows based on scores: 0) not appropriate, (1) far below standard, (2) below standard, (3) meet the standard, (4) above standard, and (5) far above standard. The minimum and maximum scores were 0 and 155, respectively. Mini-CEX was performed using a researcher-made checklist whose reliability was approved through Cronbach’s alpha method, which was calculated to be 0.869.

Logbook method was performed using logbook of Bachelor of Science in surgical technology approved by Isfahan University of Medical Sciences [22]. The logbook checklist consisted of 13 expected skills. As a result, the minimum and maximum scores were 0 and 65, respectively. The logbook has four sections:

(1) Student demographic information [age, average in the last semester, gender], (2) Evaluation of student competency based on a 5-point scale including 13 items [preparing the room (at the beginning of the shift), secondary prep, furniture and tables cover, drape patient, prepare the sterile field, start the procedure, exposure, hemostasis with an emphasis on tourniquet, surgical instruments, suturing, counting (gas, needles, and tools), wound dressing, end of surgery], (3) Three separate tables that show the skills and competencies required to complete the three stages of the test [each table has a score from 1 to 13, with the student’s score determined by the professor], (4) Incomplete skills were marked, and feedback was provided to the student.

The student’s competency level in each skill is specified as a color pattern as a criterion for the assessor. Ten experts, who were the faculty members of clinical affairs from the universities of medical sciences of Facility A, Facility B, and Facility C. Facility A, Facility B and Facility C approved the validity of Mini-CEX and Logbook checklists through Delphi and face-to-face techniques. The validity of Mini-CEX and Logbook checklists through Delphi and face-to-face techniques. The Delphi method resulted in an excellent degree of agreement between 10 experts. Two-way random effect model and average measurement were used to determine intraclass correlation coefficient (ICC) and its significance for determining consistency using SPPS-25. The average measure ICC was 0.990 with a 95% confidence interval from 0.984 to 0.994 (F(32,288) = 98.842, p < 0.001) [23].

Intervention program

In the initial phase, prior to the evaluation, the necessary training protocol for the two new methods was presented. The assessor from the Operating Room Faculty underwent a 4-hour workshop conducted by a PhD in medical education with a background in operating rooms. This session included instructions on conducting the tests, offering feedback, and assigning scores for each checklist. The assessor was given the checklists and their respective scoring guidelines. Only one assessor was chosen due to the lack of cooperation from other instructors, variations in strictness or leniency among different assessors and their job positions, and other factors that could impact students’ scores [24].

In the second phase, students were assessed following a 15-minute briefing session to introduce the tools, secure consent, and utilize the Logbook method in the control group and the Mini-CEX method in the intervention group. Subsequently, an evaluator employed the Mini-CEX evaluation method to directly observe and assess participants’ abilities in three steps with a one-week interval. During the initial step, skills were observed throughout the surgical procedure and feedback was provided for 15 min. In the second stage, the same surgery was observed again after one week, with students receiving feedback on their strengths and weaknesses. Lastly, in the third step occurring one week later, the same skills were repeated and evaluated, and final scores were assigned.

The evaluator offered feedback utilizing the guide and scores, ensuring objectivity in the evaluation process. Following the completion of the procedure, trainees promptly received verbal and written feedback, enabling them to gain insights into their clinical performance.

The Logbook evaluation method consisted of three steps, each conducted one week apart. Additionally, a separate table was used to display skills. Students were required to provide necessary information concerning the skills, such as the date and method of completion. Following this, the assessor had to review and approve the table. If any interventions were necessary during the evaluation, the assessor adjusted the scores accordingly. The assessor directly observed the student’s performance throughout the surgical procedure [25].

At this stage, only the students’ competencies were identified, and the assessor marked the skills in which the student did not meet the competency level in separate tables. The student’s competency level in each skill is indicated by a color pattern as a criterion for the assessor. In the first step, the skills were observed during the surgical procedure, and students received 5 min of feedback. In the second step, the same surgery was observed again after 1 week, and students received feedback to assess their competency level.

In the third step, conducted a week later, the same skills were reassessed, and final scores were determined. In this process, students received verbal feedback solely on their competence in carrying out the designated functions in the Logbook. The time taken to conduct the assessment, complete the Logbook, and fulfill the Mini-CEX checklists aligned with the surgery duration and the student’s role as a scrub nurse.

Statistical analysis

There were 2 main variables in this study: [1] Students’ score of assessment with a Logbook method and [2] with a Mini- CEX method in surgical clinical skills as a scrub nurse.

Data were analyzed by SPSS-16 using descriptive statistics (frequency and relative frequency for qualitative and mean and standard deviation for quantitative variables) and inferential statistics (chi-square test, t-test and repeated measures ANOVA with post-hoc tests). All assumptions for the repeated measures ANOVA were met.

Results

The participating students (60 individuals) were divided into two groups: Mini-CEX (30 individuals) and Logbook (30 individuals).

There were no statistically significant differences in terms of gender, age, and grade point average between students in the two groups. Approximately 73% and 57% of the students in the Mini-CEX group and Logbook group were female, respectively, with the remainder being male (P-value = 0.176). The mean age of students in the Mini-CEX and Logbook groups were 21.9 and 21.6, respectively (P-value = 0.440). Additionally, the mean GPA of students in the Mini-CEX and Logbook groups were 16.8 and 17.2, respectively (P-value = 0.170). The results are presented in Table 1.

Table 1.

Descriptive statistics of demographic variables (N = 60)

			Group		P-value*
			Mini-CEX(n = 30)	Logbook (n = 30)
Gender	Male		8 (26.7)	13 (43.3)	0.176
	Female		22 (73.3)	17 (56.7)
		Age	21.90 ± 1.69	21.63 ± 0.81	0.440
		GPA	16.83 ± 1.03	17.22 ± 1.06	0.170

Frequency (percent) or mean ± standard deviation

^* Based on chi-square and t-test results

To assess and compare the average scores of the Logbook and mini-CEX checklists, the scores of the two tools were initially tabulated on a 100-point scale. The average scores of students in the Mini-CEX group rose from 81.57 in the initial test to 91.87 and 97.42 in the subsequent two tests. Similarly, in the Logbook group, the average scores increased from 44.36 in the first test to 54.87 and 65.64 in the second and third tests, respectively. This data supports the hypothesis that repeated Mini-CEX and Logbook tests impact student performance. More specifically, the results indicate a significant improvement in student performance after the third test compared to the initial one. The results are presented in Table 2; Fig. 2.

Table 2.

Descriptive statistics of student’s score in different levels of Time*Group and Post-Hoc results of repeated measures ANOVA

			Fist exam	Second exam	Third exam
			Mean (Std-Deviation)	Mean (Std-Deviation)	Mean (Std-Deviation)	P-value*
Scrub	Group	Mini-CEX	81.57(7.23)	91.87(5.43)	97.42(4.24)	< 0.001
		Log-book	44.36(18.81)	54.87(13.96)	65.64(13.51)	< 0.001
	Test	Mean Difference of groups	37.21	36.99	31.78
		P-value**	< 0.001	< 0.001	< 0.001

^*Multivariate test of mean difference of scores in each group

**Post-Hoc t- tests with Bonferroni correction for mean difference of groups

Fig. 2.

Student’s Mean Score in Different Levels of Time*Group

The results of repeated measures ANOVA show that not only the mean scores of scrub nursing students increase significantly over time (P-value < 0.001), but difference in mean scores of two groups (Mini CEX and Logbook) is also significant (P-value < 0.001). Based on the results of this analysis, the effect of time on the group was also significant (P-value = 0.001). The results are presented in Table 3.

Table 3.

Results of repeated measures ANOVA

		source	Sum of Squares	df	Mean Square	F	P-value	Effect size
	Within–groups*	Time	10391.161	1.450	7167.626	257.172	< 0.001	0.816
		Time*group	284.075	1.450	195.950	7.031	0.004	0.108
Scrub		Error	2343.522	84.085	27.871
	Between- groups	Group	56167.726	1	56167.726	149.781	< 0.001	0.978
		Error	21749.885	58	374.998

*Greenhouse-Geisser

Discussion

Our research findings demonstrated a notable enhancement in students’ scores across the three assessment stages. Additionally, the mean score in the Mini-CEX group was significantly higher than in the Logbook group. This study revealed that consistent use of Mini-CEX resulted in substantial improvements in students’ skills and performance. This approach enabled students to track their progress, concentrate on enhancing their strengths, and address areas requiring improvement. In comparison to trainees using logbooks, those utilizing Mini-CEX achieved higher overall performance scores with mean scores of 81.57 (SD = 7.23), 91.87 (SD = 5.43), and 97.42 (SD = 4.24). The third assessment indicated significant enhancements in trainee performance across most domains.

The Mini-CEX improves and complements teaching and learning with prompt feedback. Moreover, formative assessments have been proven to be more effective than summative assessments in providing feedback. Our findings align with previous research, indicating that implementing new assessment methods repeatedly can enhance student performance. In their study, Batra et al. demonstrated that utilizing Mini-CEX enhances the learning environment during residency, leading to improvements in medical interview skills, physical examination skills, interpersonal and professional qualities, as well as counseling skills [26]. Vafaei et al. also noted that feedback will significantly improve students’ grades and skills in subsequent evaluation [27]. Gandomkar et al. reported that although Mini-CEX was originally designed to evaluate the clinical skills of internal residency programs, its easy application in real environments has made it a widely used instrument in medical disciplines such as nursing and midwifery [24]. The mini-CEX can be utilized across various clinical scenarios and workplace environments, enabling trainees to obtain feedback from a diverse range of supervisors. Additionally, it evaluates a range of competencies that are crucial in delivering high-quality care [28].

The effectiveness of Mini-CEX implementation remains uncertain. Unlike our study, the findings from Soemantri et al.‘s study revealed that the majority of feedback given on the Mini-CEX evaluation forms was either general comments on students’ strengths or general recommendations. Less than 30% of the feedback comments were geared towards enhancing students’ performance in a particular aspect, with an even smaller proportion of specific feedback aimed at bolstering good performance [29]. Castanelli et al. had some difficulties in finding time to schedule assessments and deliver timely feedback in busy clinical workplaces [30]. We meticulously planned and organized the meetings to ensure that no busy work schedules were disrupted. Surprisingly, Joshi et al. [11], highlighted nervousness, although our study did not focus on this aspect. Moreover, there have been no efforts to assess the cost-effectiveness of this tool. A study conducted in Switzerland concluded that the Mini-CEX often does not align with learning objectives, if at all [31].

The present study had several limitations. Firstly, there was no published Mini-CEX instrument available to evaluate surgical clinical skills for scrub nurses in orthopedic surgical residency. As a result, we developed an assessment tool specific to operating room students working as scrub nurses in orthopedic surgeries (see Supplementary Appendix 1). Secondly, the study was carried out with a small number of participants, and certain instructors opted not to adopt new assessment methods. Additionally, due to the nature of the research, conducting a pretest was not feasible.

Thirdly, it is crucial to acknowledge potential limitations, such as biases that may arise from the relationship between assessors and students, which could have influenced the study outcomes. To mitigate these concerns, we ensured that the same experienced assessor evaluated both the control and intervention groups. We also provided 4-hour training sessions for assessors and adhered to assessment guidelines. However, maintaining confidentiality of students’ demographic information to prevent the assessor’s prior knowledge (halo effect) from impacting assessment scores was challenging. The lack of blinding could have introduced bias in the scoring process due to the halo effect, potentially obscuring the true effectiveness of the Mini-CEX instrument. For future studies, employing multiple blinded assessors or anonymous assessments should be considered to minimize this risk.

This study compared students’ competency, as assessed by traditional clinical evaluation (Logbook), in the intervention and control groups based on a pilot design. A statistically significant difference in traditional clinical assessment scores was noted between the Mini-CEX and control groups (97.42 vs. 65.64).

Conclusion

The present study indicates that Mini-CEX if properly implemented and monitored at the undergraduate level can change trainees’ performance. Based on our experience, the Mini-CEX is an acceptable and practical tool for assessing operating room students. Any evaluation can be called successful if it encourages students to look for more information. It is therefore suggested that the focus of assessment should be shifted from “assessment of learning” to “assessment for learning”. For “assessment for learning”, tools like multidomain feedback are employed to identify the strengths and weaknesses of an individual student, thereafter using this information to enhance learning and maximize competency.

Additionally, it is recommended that the Mini-CEX tool be implemented as an assessment and teaching method for clinical skills among operating room students due to its effectiveness and cost efficiency. This approach is believed to be engaging and beneficial in enhancing student competency. This study focused on the clinical learning of scrub nursing students during orthopedic surgeries. Future studies should explore other specialized surgery clerkships in operating room education. Moreover, further study is needed to assess the performance circulating nursing students as they are also a part of operation.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Author contributions

Conceptualization: S B; Data curation: SB, ZZ; Formal analysis: ZZ, MK; Methodology: S B, A S; Project administration: S B, A S, N SH; Writing–original draft: N SH, S B, A S; Writing–review & editing: all authors.

Funding

This research was supported by Khomain University of Medical Sciences (No: 99000004).

Data availability

The datasets generated and analyzed during the current study are not publicly available because they contain raw data from study participants, and sharing these data requires participants’ permission. But are available from the corresponding author on reasonable request.

Declarations

Ethical approval and consent to participate

Ethical approval was obtained from the Human Research Ethics Committee at the Khomein University of Medical Sciences (Code IR.KHOMEIN.REC.1399.009). All study participants provided written informed consent. Confidentiality and anonymity were ensured. All procedures performed in studies involving human participants were by the ethical standards of the institutional and national research committee and with the 1964 Helsinki Declaration.

Informed consent

All participants provided written informed consent.

Consent for publication

Not Applicable.

Competing interests

The authors declare no competing interests.