How Are We Doing? The Problem of In-Training Evaluation

“How am I doing?” asks the 3rd-year clerk, 2 weeks into her inpatient medicine rotation. We find time in the busy day to reflect on the month to date, to discuss her self-assessment and my observations of her performance on rounds, and to discuss goals and resources for future learning. We meet 2 weeks later to review the department's evaluation form I have completed for her, and for her to give me feedback on the month. The evaluation form goes off to the department in a blue envelope, and the student goes on to her next rotation.

“How am I doing?” I wonder. How has my memory of her observed encounters and skills prevailed in these evaluations and feedback sessions? What factors have influenced my global ratings of this student? Have I helped this student learn? Have I helped the school verify some meaningful aspect of her clinical competence at this point in her training? The literature suggests that my preceptor colleagues and I rarely observe our students sufficiently to collect valid data on their clinical skills. And sadly the evaluation forms we complete so dutifully often fail to help us structure our observations with sufficient specificity and discrimination to yield a meaningful assessment.¹

In 1991 the Liaison Committee on Medical Education mandated that schools develop performance-based assessment methods. The OSCE (Objective Structured Clinical Examination) introduced in the 1980s has been championed as a promising system of assessment that verifies competency in the observed clinical skills. Use of standardized patients and structured examinations may most closely approximate true assessment of students' competence in skills needed for clinical practice.¹ It is acknowledged, however, that the OSCE is a limited form of assessment and that it is both resource intensive and cumbersome as a teaching tool.² Is there opportunity then to review our approach to clinical teaching and observation and to improve the reliability and validity of preceptor assessments of students as they are working?

In their article “Clinical Work Sampling: A New Approach to the Problem of In-Training Evaluation,” Turnbull and colleagues challenge us to recognize the inadequacies of our current approaches to in-training evaluation and to take a fresh look at our clinical evaluation system. Recognizing that “retrospective recall” may be the most significant bias we face, the authors adapted an established industry assessment system of work sampling to create a model that captures performance data during regular encounters in the course of the workday. They call their model, appropriately, “Clinical Work Sampling” (CWS), which distinguishes it from either continuous or timed or random work sampling models employed in workforce assessment settings. With CWS the authors provide structured instruments for assessing the skills and performance of ward clerks during the course of the workday each day, and they provide these instruments to multiple observers. Their study addresses the feasibility, reliability, and validity of obtaining these evaluations of competencies considered necessary for the practice of medicine.³

The evaluation instruments were designed for preceptors (for skills related to admission and patient management on the wards), for nurses (for skills related to interactions with nurses and other members of the health care team, resource management, and discharge planning), and patients (for skills related to communication, advocacy, and professionalism). The feasibility of the CWS approach was measured by the return rate of completed evaluation forms for each student. Reliability was computed for the average score based on the number of forms returned, and evidence for systematic differences among examiners was sought. Content and concurrent validity were assessed.

It is disappointing but not surprising that capturing patient assessments of student performance was less feasible in this study, and further work to facilitate patient participation in this process will be needed. Similarly it is disappointing that the Multidisciplinary Team ratings were found to be unreliable in this study, since evaluations from allied health professionals have previously correlated well with other valid measures of student performance. The authors question appropriately whether daily nursing notes were sufficient in this case to overcome possible recall bias in completing the end-of-month assessment form.

The good news, however, is that the Admission Rating forms and Ward Rating forms, completed by supervising preceptors, were identified as feasible, reliable, and valid assessments of student performance. The reliability of as few as four to eight assessments was found by the authors to be comparable to large-scale objective assessments used for licensure and certification, and distinguished from traditional, unreliable end-of-rotation clinical ratings. This suggests that preceptors can make accurate observations during focused assessments during the course of the daily work of a ward team, and that these are preferable to the end-of-month global ratings that are based on imperfect memory of these events and others alone. That raters in this study failed to distinguish skill levels across behaviors merits further study as our charge as educators remains to verify both global competence and the achievement of skill competencies that comprise our view of global competence.

Structured clinical observations have been described as a method for preceptors to teach clinical and interpersonal skills in the setting of a clinical encounter, using criterion-reference skill checklists, observation sheets, and skill guidelines. In a recent study in Pediatrics, Lane and Gottlieb reported that brief structured observations of student encounters with patients permitted identification of skill competency as well as skill deficiencies that could be addressed immediately with feedback.⁴ Lane's structured observations made for teaching purposes and Turnbull's work sampling strategy for summative assessment may represent new ways for clinical preceptors to think of their skills as educators and evaluators. Using regular, frequent, and structured assessments, preceptors may greatly increase the usefulness of their feedback to learners and their assessments submitted to departments.

It is likely that in-training evaluation will continue to benefit from the information gained by OSCEs and other standardized patient teaching and evaluation methods. It is also imperative that we continue to help our learners develop self-assessment skills that they will develop throughout their professional careers. Along that continuum we have relied on clinical faculty to continue to provide teaching and assessment information, as imperfect and biased as that information remains. We now have evidence that CWS and other structured assessments are feasible, reliable, and valid in the clinical setting, and may ultimately help physicians, allied health professionals, patients, and learners all know “how we're doing” in our clinical classroom, and how our learners practice the clinical skills they've mastered.—Nancy Ryan Lowitt, MD, EdM,University of Maryland School of Medicine, Baltimore, Md.