Assessment of the Performance of Orthopedic Residents in Clinical Practice

Abstract

Objective  To evaluate the performance of orthopedic residents while conducting clinical examinations.

Methods  The Mini Clinical Evaluation Exercise (Mini-CEX) was applied by three teaching doctors at four different moments. The instrument was adapted by the authors for use in orthopedics, with the development of descriptors for each evaluated skill. Supervisors were trained to use the Mini-CEX by the principal investigator through teaching materials and discussions, with standardization of the instrument descriptors.

Results  The mean scores obtained in the 4 evaluations for each of the 21 residents reveal improvement in the performances of residents in all skills assessed from the 1 ^st to the 4 ^th meeting.

Conclusions  We have found that the performance of orthopedic residents presented a satisfactory evolution, with progressive improvement in all skills.

Introduction

Reliable methods for assessing clinical resident's performance are a problem faced by teaching specialists worldwide. ¹ Direct observation is a good tool for analyzing them in the workplace, allowing for the opportunity to observe what they know and can do. However, although teaching physicians and residents work in the same place, observation is traditionally informal and performed inconsistently in almost all medical specialties. ²

To encourage teaching doctors to perform observations, the American Board of Internal Medicine proposed the Mini Clinical Evaluation Exercise (Mini-CEX), a 10- to 20-minute direct observation assessment of trainees' clinical skills. The evaluation takes place during trainee-patient meetings; interaction between the resident and a patient is observed, after which the evaluator provides the resident with feedback in the form of one-page written assessment rating such skills as “overall clinical competence” and “humanistic qualities/professionalism,” which must be signed by both the observer and the trainee. ³

This method has been used in such specialties as cardiology, ⁴ anesthesia, ⁵ pediatrics, ⁶ internal medicine, ⁷ and chiropractic. ⁸ However, no studies on its use in orthopedics have been found. Thus, our objective was to evaluate the performance of orthopedic residents while conducting clinical examinations.

Methods

Study Design and Location

The study was a longitudinal prospective study, performed from January to July 2016, in two residency programs in orthopedics, accredited by the Ministry of Education and recognized by the Brazilian Society of Orthopedics and Traumatology, in a capital of the Central-West region of Brazil.

Participants

Nonprobability sampling was employed. We included the postgraduate first-year (R1), second-year (R2), and third-year (R3) orthopedic residents in the two residency programs that agreed to participate in the study, and the participants signed informed consent forms. Excluded from the study were those belonging to (i) nonaccredited services of the Brazilian Society of Orthopedics or (ii) trainees of subspecialties of orthopedics (R4), such as spine, knee, shoulder, and foot and ankle surgery.

Data Collection

The Mini-CEX was applied by three teaching physicians of the respective hospitals in orthopedics. They were trained to use the Mini-CEX by the principal investigator through teaching materials and discussions, with standardization of the instrument descriptors. The Mini-CEX was used during individual resident-patient interactions in the emergency room, infirmary, and ambulatory departments. Its application occurred on dates formally scheduled by the appraisers to ensure that all residents experienced the evaluation on the same day, in their respective places of operation. Each resident was evaluated at four different times, with an interval of at least 30 days between evaluations.

Instrument Used

The Mini-CEX, created by the American Board of Internal Medicine, presents six clinical skills, as follows: (i) medical interviewing (clinical history); (ii) physical examination, (iii) humanistic quality/professionalism; (iv) clinical judgment; (v) counselling skills (communication and advice); and (vi) organization/efficiency. The instrument was adapted by the authors for use in orthopedics, with the development of descriptors for each evaluated skill. The process of adaptation of the instrument took place in specific meetings, in which 4 teaching physicians with more than 10 years of participation in the residency program in orthopedics, and 2 heads of residency training of this specialty were present. After 5 meetings with an average duration of 120 minutes, the instrument was adapted and defined such that each skill had 9 descriptors. If the objective of the respective descriptor was achieved, the resident received a point, respecting the original's maximum of nine points. The scoring scale was divided into 1 to 3 (unsatisfactory); 4 to 6 (satisfactory), and 7 to 9 (outstanding). The methodology used by Abadie et al. was used to develop the descriptors.

Outcome

The dependent variable was the performance of the orthopedic residents in the physical examination, evaluated by the Mini-CEX instrument.

Independent Variables

The following variables were used: age (in years lived); gender (male/female), and year of residency (R1/R2/R3).

Availability of Data and Materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Data Analysis

The data were recorded in a Microsoft Excel (Microsoft Corp., Remdond, WA, USA) spreadsheet, and the statistical analyses were performed using the IBM SPSS Statistics for Windows, Version 23.0 (IBM Corp., Armonk, NY, USA). The quantitative variables were presented as mean and standard deviation. The total Cronbach α coefficient was calculated at each scale to evaluate the internal validity of the Mini-CEX instrument. Normality was verified using the Shapiro-Wilk test. Nonparametric statistics were applied if data were not normal. The evolution of the residents (R1, R2, and R3) throughout the duration of the study was performed using the Friedman test, followed by analysis of multiple posthoc comparisons. The probability of rejecting the null hypothesis was 5%.

Results

The total number of residents was 33, 12 of whom were excluded because 3 were R4, and 9 were nonaccredited. Thus, the final sample consisted of 21 residents, 7 in each year of residence, and all male. The mean age was 25.86 years ( ± 2.27) for the R1s, 27.50 years ( ± 1.74) for the R2s, and 27.14 ( ± 1.97) for the R3s.

The total Cronbach α coefficient was 0.88 when all the skills of the Mini-CEX instrument were considered, with a 95% confidence interval ranging from 0.79 to 0.94. The “clinical reasoning” and “clinical history” skills were the ones with the highest values, 0.83 (0.67–0.92 CI) and 0.76 (0.52–0.89 CI), respectively.

The general evaluations using the Mini-CEX of the performance of residents during the clinical examinations on the 4 test days are shown in Table 1 . A comparison of the scores from the first evaluations with the last ones showed significant improvement of all the clinical skills.

Table 1. Assessment of clinical skills using the mini-CEX in four meetings.

Competence	Meetings (mean ± standard deviation)				p *
	First	Second	Third	Fourth
Physical examination	2.67 ± 1.02 d	3.71 ± 0.96 c	5.86 ± 0.96 b	6.57 ± 1.25 a	< 0.001
Medical interview skill	2.86 ± 1.39 d	3.95 ± 1.02 c	6.00 ± 0.95 b	7.00 ± 0.89 a	< 0.001
Clinical judgment	4.19 ± 1.33 c	4.90 ± 0.83 c	6.57 ± 1.16 b	7.38 ± 0.86 a	< 0.001
Humanistic quality	4.14 ± 1.24 c	5.19 ± 1.03 b	6.14 ± 1.11 a	6.76 ± 1.04 a	< 0.001

Friedman test followed by Post-Hoc analysis.

^*Different letters indicate significant differences.

The mean scores obtained in the 4 evaluations for each of the 21 residents are presented in Table 2 , broken out individually by residency year (R1, R2, R3) and skills. The results reveal improvement in the performances of the R1, R2, and R3 residents in all skills assessed from the first to the fourth evaluation.

Table 2. Mean scores obtained in four meetings according to resident's postgraduate year.

Competence	Y *	First	Second	Third	Fourth	p **
Physical examination	R1	1.86 ± 1.07 d	3.00 ± 0.58 c	4.86 ± 0.90 b	6.29 ± 0.49 a	<0.001
	R2	3.17 ± 0.75 b	3.67 ± 1.21 b	6.33 ± 0.52 a	6.83 ± 0.75 a	0.001
	R3	3.00 ± 0.76 b	4.38 ± 0.52 b	6.38 ± 0.52 a	6.63 ± 1.92 a	0.001
Medical interview skill	R1	1.43 ± 1.13 c	3.00 ± 0.58 c	5.14 ± 1.07 b	6.29 ± 0.49 a	0.001
	R2	3.50 ± 0.84 c.b	4.50 ± 1.22 b	6.17 ± 0.41 a	7.00 ± 0.89 a	< 0.001
	R3	3.63 ± 0.92 c	4.38 ± 0.52 c	6.63 ± 0.52 b	7.63 ± 0.74 a	0.001
Clinical judgment	R1	2.86 ± 1.07 c	4.29 ± 0.49 c	5.57 ± 1.13 a	6.57 ± 0.53 a	0.001
	R2	4.67 ± 1.03 c	5.33 ± 0.82 c	6.67 ± 0.82 b	7.83 ± 0.75 a	0.001
	R3	5.00 ± 0.76 b	5.13 ± 0.83 b	7.38 ± 0.74 a	7.75 ± 0.71 a	< 0.001
Humanistic quality	R1	3.71 ± 1.11 b	4.86 ± 1.21 b	5.71 ± 1.11 a	6.71 ± 1.11 a	0.001
	R2	4.17 ± 1.47 b	5.00 ± 1.26 b	6.00 ± 1.10 a	6.33 ± 0.52 a	0.002
	R3	4.50 ± 1.20 d	5.63 ± 0.52 c	6.63 ± 1.06 b	7.13 ± 1.25 a	< 0.001

^**Friedman test followed by Posthoc analysis.

^*Y – postgraduate year. Different letters indicate significant differences. R1–first year resident, R2–second year resident, R3–third year resident.

Discussion

A comparison of the mean scores of the Mini-CEX scores of the four evaluations showed that there was a significant improvement in the performance of the residents. It is interesting to note that this evidence contrasts with what existing data in the literature. A systematic review of 119 papers on the tools for evaluation and direct observation of clinical skills, using the same instrument, showed that there was no improvement in clinical skills and patient care. ⁹

Another study, published in Pediatrics , used the Mini-CEX in 4 to 6 evaluations of 23 residents over a year. In that study, the clinical skill that showed the most improvement was humanistic qualities/professionalism, and the skill with the least improvement was physical examination. ¹⁰ This partially corroborates our findings as it supports a general progression over the course of the evaluation; however, it differs from our findings in that we found significant improvement in the residents' physical examination skill: the scores increased from 2.67 for the first test day to 6.57 ( p  < 0.001) for the last test day. Physical examination and medical interviewing were the skills that our study found to improve most over the course of the evaluations. This can be explained by differences in feedback styles and focus (in our study, more feedback was given on physical examination) between the different Mini-CEX studies.

Other medical specialties also presented satisfactory results in relation to physical examination skills. Another study's evaluations of pediatric residents found an average of 6.1 for this skill. ⁶ Similar results were found in 108 cardiology residents, with a mean of 7.1, 7.5, 7.5, and 8.0 found in R1, R2, R3, and R4 residents, respectively. ⁴

The present study shows that the average score for physical examination skills for the R1s on the first test day was the lowest one found. This can be attributed to the R1s' relative newness to the specialty, less time in theoretical/practical classes, and less observation by supervisors in their consultations. In addition, the first evaluation happened shortly after the R1s' entry in the of “orthopedics program, when they were only equipped with the knowledge acquired in the undergraduate level; this suggests a lack of specialty teaching during medical school and the need to correct this gap during medical residency.

It can be observed that the residents showed improvement in their clinical skills during the serial evaluations; however, the R2s and R3s did not evolve from the third to the fourth assessment in the area of physical examination skills, probably due to having been in the specialty for a longer time and having already achieved greater skills and satisfactory scores at the time of the study.

It is important that orthopedic residents perform optimally, especially in the physical examination. To achieve this goal, it is necessary for teaching physicians to use the best teaching and learning resources possible. Teaching is most effective when residents are involved both physically and mentally—namely, through direct involvement in patient care—by encouraging residents to take notes while studying, especially about relevant questions to ask supervisors and/or patients. ¹¹ Another way to improve residents' performances would be to focus on their “development zone”—that is, to start from the exact point of their gaps or misconceptions rather than teaching what is already known. Thus, it is crucial to identify the limits of residents' knowledge. ¹¹

Teaching orthopedic physical examination is challenging. There are several variables involved in the process, such as the clinical skill of the supervisor, the willingness of the patient to be evaluated in a group as an illustrative example as physical examination signs are taught, and the interests of the residents. The best strategy for teaching this skill is to start with a supervisor with top-level skills and a strong aptitude for teaching. Attributing this function to someone without either ability would only perpetuate residents' bad habits.

Another effective strategy to achieve a quantitative increase in skill is teaching orthopedic physical examination at the bedside. This is perhaps one of the best ways to promote its improvement. This learning scenario occurred routinely in the 1960s, but it became infrequent in the 1990s. ¹² Physical examination skills cannot be taught effectively in theory lectures without the presence of a real or simulated patient. ⁹

The present study becomes relevant in the context of a lack of research in this area, showing the positive evolution of the performance of residence with the Mini-CEX, as no similar studies were found related to orthopedics. Our findings contribute to the knowledge of this subject, with special significance for our target population. We performed a longitudinal and prospective study, without memory bias, which gave the data obtained greater credibility. Another strength of the study would be the chosen evaluation instrument. The Mini-CEX was designed, based on (and is used in) real situations, to be a comprehensive yet streamlined tool for evaluating clinical skills and is, therefore, a distinguishing instrument. The Mini-CEX also enables evaluations in different scenarios and with different levels of complexity. ³

Because there was no previous sample calculation of the subjects to be evaluated, the final nonprobability sample was considered small, coming from only two hospitals. Statistically, nonrandom selection procedures may not guarantee representativeness; consequently, our findings should not be generalized to a broader population. Despite the need for new studies on the validity and feasibility of the Mini-CEX in the Portuguese language, this study did demonstrate that the instrument tested has a high reliability and internal consistency.

Conclusion

This study found that the performance of residents of orthopedics and traumatology during clinical examinations presented a progressive improvement in all skills, regardless of the year of residency. The research favors the development of other studies to broaden the understanding of the phenomenon studied or to empirically confirm the results obtained.