An Objective Structured Clinical Examination to Improve Formative Assessment for Senior Pediatrics Residents

Abstract

Background

Residency programs are developing new methods to assess resident competence and to improve the quality of formative assessment and feedback to trainees. Simulation is a valuable tool for giving formative feedback to residents.

Objective

To develop an objective structured clinical examination (OSCE) to improve formative assessment of senior pediatrics residents.

Methods

We developed a multistation examination using various simulation formats to assess the skills of senior pediatrics residents in communication and acute resuscitation. We measured several logistical factors (staffing and program costs) to determine the feasibility of such a program.

Results

Thirty-one residents participated in the assessment program over a 3-month period. Residents received formative feedback comparing their performance to both a standard task checklist and to peers' performance. The program required 16 faculty members per session, and had a cost of $624 per resident.

Conclusions

A concentrated assessment program using simulation can be a valuable tool to assess residents' skills in communication and acute resuscitation and provide directed formative feedback. However, such a program requires considerable financial and staffing resources.

Editor's Note: The online version of this article contains the assessment tools used in the study.

Introduction

Residency programs must improve the quality and breadth of trainee assessment and feedback to ensure that graduates are prepared for independent practice. Faculty assessments and in-training examination scores are inadequate performance measures when used as a main assessment methodology.^1,2 In addition, the Accreditation Council for Graduate Medical Education requires a breadth of trainee assessment with improved measures.³ Pediatrics residency leadership at the Ann & Robert H. Lurie Children's Hospital of Chicago partnered with the kidSTAR Medical Education Program, a team of physicians and nurses with backgrounds in simulation and medical education, to develop a performance-based objective structured clinical examination (OSCE) using various forms of simulation. Our overall aims were to (1) assess clinical skills and provide formative feedback to the individual residents, and (2) determine resident preparation for senior responsibilities. Our objective in this article is to describe the design, implementation, and costs of this formative assessment program.

Methods

We used Kern et al's 6-step approach to curriculum development⁴ to design the assessment program, and we decided that an OSCE format fit our need for performance-based assessment.

Needs Assessment

We performed a literature search to examine the use of OSCEs for formative purposes with residents. We identified several studies that evaluated how to improve OSCE validity, including combining it with other assessment methods.^5–8 Several prior studies have examined the validity of OSCEs as assessment tools in graduate medical education,^9–12 but none have used OSCEs to assess resuscitation skills in pediatrics residents.

Program leadership expressed an interest in assessing residents as they transitioned to the final year of training with its increased patient care responsibility. Also, this time frame allowed for an opportunity to address any deficiencies identified before graduation. Our goal was to observe residents performing activities that are traditionally not directly supervised, including management of an acutely decompensating patient requiring critical intervention and communication with clinicians and families.

We developed a 5-station OSCE. The number was determined by time constraints and the amount of faculty required. Three stations were immersive simulations involving patients with acute decompensation events, including respiratory (bronchiolitis/apnea), cardiac (pulseless ventricular tachycardia), and neurologic (status epilepticus) etiologies. Clinical performance and handoffs were assessed in these stations. A fourth case used a standardized patient to assess communication with a parent. The final case involved answering multiple pager calls about different patients to assess residents' ability to manage and prioritize acute patient care, and to communicate with clinical team members.

A faculty member who is a content expert and experienced in running simulated case scenarios wrote each case. The SEGUE framework tool¹³ was used to assess communication skills during the standardized parent encounter the first year. All the assessment tools we developed are available as online supplemental material.

Five third-year residents participated in a pilot test of the OSCE in February 2012, and we solicited feedback about cases, logistics, and assessment forms. Adjustments were made based on this feedback.

Assessments were scheduled from March to May 2012 for all 32 second-year residents. All assessments were conducted at our medical school's simulation center, which contains multiple patient rooms and adjacent meeting rooms with connected observation facilities. Residents had access to pediatric and hospital-specific equipment and reference materials.

Five to 6 residents participated at each of the six 3½-hour sessions. Residents were usually scheduled during rotations without mandatory afternoon activities. One resident was unable to participate due to coverage issues, leaving 31 participants.

This study was reviewed and approved by our Institutional Review Board.

For this study, we gathered data on the number of staff required to run the assessment program as well as the budget required for development and implementation, including tracking the number of faculty and cost for each session.

Results

Staffing

table 1 outlines the staffing requirement for each session. The kidSTAR Medical Education Program is primarily composed of physicians, so they were used as confederates for multiple roles. It was not possible to have the same faculty present for every assessment session, but we attempted to keep consistency in staffing when possible.

TABLE 1.

Staffing Requirements for Sessions

Implementation

table 2 illustrates the assessment process. Residents began with an orientation, during which we clarified expectations, discussed issues of realism and confidentiality, and emphasized that the primary aim was to provide feedback. After orientation, a chief resident gave participating residents a sign-out on all patients they would encounter during the assessment. The sign-out was conducted using the program's handoff format, replicating the computer-generated sign-out, and allowed them to ask questions about patients.

TABLE 2.

Assessment Process Timeline

TABLE 2.

Extended

Feedback

Residents initially received a group debriefing at the conclusion of all stations. After the first 2 sessions, residents expressed interest in immediate feedback after each case, so we adjusted the stations to allow faculty to review performance for 5 to 10 minutes after each case. When all assessments were complete, we compiled reports that compared individual resident scores to peers' scores for each case, and included written comments from the standardized parent case. Residency leadership discussed individual reports during residents' semiannual reviews, and encouraged residents to incorporate results into their individual learning plans.

Budget

The total cost of running the program was $19,348 (not including faculty time) or $624 per resident. Table 3 shows the breakdown of costs.

TABLE 3.

Budget

Satisfaction Survey

Residents filled out an evaluation after their session. Residents reported that cases were representative of their usual case mix, although a few residents commented that managing ventricular tachycardia was not appropriate for their training level. Both on this survey, and in the group debriefing, several residents remarked that they were anxious before the assessment, but found the actual assessment fun and educational. Residents who participated in the 4 later sessions in which immediate feedback was provided expressed gratification regarding this immediate feedback during group debriefings and in the satisfaction survey.

Discussion

We learned several lessons during implementation of this assessment program. First, the pilot session was valuable to learn about logistics, cases, and assessment tools. Second, the assessment necessitated significant staff and facility time. One way to reduce physician requirements would be to use nurses and paramedics as case “confederates” and a research administrator to direct participants to the next station, instead of using physicians for these roles.

This is the first description of an OSCE-style assessment of pediatrics residents' care of acutely ill patients. Previously published OSCEs for this learner group described evaluation of nonacute patient skills.^9,10 Some studies have looked at OSCEs to assess the care of acutely ill patients for emergency medicine residents,^11,12 but they did not provide information about staff requirements and cost.

Finally, the residents' desire for immediate feedback was easy to accommodate. However, immediate feedback may influence resident performance on subsequent cases in the assessment. For example, residents who were taught 2-person bagging techniques during the bronchiolitis case were quick to adopt this for a seizing patient in a later case. We recognize the immediate feedback affected their future performance, but feel it is important to capitalize on teachable moments during formative sessions.

Conclusion

The acute care OSCE received an overwhelmingly positive response from both residents and residency program leadership. The residents' survey comments highlighted that they found assessments to be intimidating, but that they thought the educational value made up for this issue. The session had significant time, faculty, and facilities costs, which may make it less feasible for some programs.