Use of Cognitive Simulation During Anesthesiology Resident Applicant Interviews to Assess Higher-Order Thinking

Abstract

Background

It is difficult to assess applicants' higher-order cognitive thinking skills during conventional resident interviews. Application metrics currently employed are useful indicators of academic and personal success in targeted areas, yet value of this information in predicting future clinical performance is limited.

Objective

We developed an assessment tool to evaluate higher-order cognitive function in real time during anesthesiology resident applicant interviews.

Methods

During the 2014–2015 residency interview season, we integrated simulation training into applicant interviews to evaluate higher-order cognitive skills. Our 5-minute simulation emphasized the Team STEPPS 2-Challenge Rule and explored candidates' critical thinking, analytical decision making, and response to stress. Participating applicants were evaluated using an outcomes-based checklist targeting desired responses. We also sent applicants a post-National Resident Matching Program survey to assess their perceptions of the simulation's value and educational utility.

Results

A total of 90 applicants (75% of all applicants) participated in the simulation, which taught residents about important patient safety concepts and provided the program with real time information about their critical thinking ability. All applicants were confident or very confident that they would both speak up and know what to say if they encountered a patient safety breach as a result of participating in this exercise. Simulation performance affected desirability status for 35% of participating applicants, where 23% of applicants ranked higher, and 12% ranked lower compared to baseline application status.

Conclusions

Cognitive simulation training was useful in assessing resident applicant higher-order thinking skills and in helping stratify candidates in conjunction with standard application metrics.

Editor's Note: The online version of this article contains the 2-Challenge Rule briefing and debriefing, the simulation evaluation checklist, and the Post-National Resident Matching Program Match Resident Applicant Survey.

What was known and gap

Traditional residency application interviews generally are not able to assess applicants' higher-order critical thinking skills.

What is new

Use of a brief simulation with the Team STEPPS 2-Challenge Rule to explore candidates' critical thinking skills and response to stress.

Limitations

Single site, single specialty study reduces generalizability; simulation tools lacks predictive validity evidence.

Bottom line

The simulation assessment added information that stratified candidates and was acceptable to applicants.

Introduction

Anesthesiology residents must be able to make quick, accurate, and sound medical decisions during critical events. The ability to do so effectively requires higher-order cognitive skills that include knowledge base application, good judgment, situational awareness, critical thinking, analytical decision making, diplomacy, advocacy, and psychological resilience. Identifying these desirable characteristics in residency applicants during the formal interview can be daunting, as traditional interviews are not designed to evaluate higher-order thinking ability.¹

In recent years, the multiple mini interview (MMI) and behavioral event interviewing (BEI) have gained popularity as a way to better assess applicants' performance and behavioral skills. The MMI evaluates higher-order cognitive skills, with predictive validity supported at both undergraduate² and postgraduate³ levels. Typically, MMIs engage candidates in 8 to 9 scenario-based encounters, each facilitated by trained staff. Candidates remain at each station for a few minutes, and assessors rate them using a structured rubric.^4,5 Although the MMI is valuable, it requires considerable personnel resources, and the regimented interaction between interviewers and candidates poses its own limitations.⁵

BEI has increased in popularity as a tool to better understand residency applicants' personal and work-related experiences.⁶ Because of its success rate in predicting future on-the-job behavior compared to traditional interviewing,⁷ BEI has been endorsed by residency program directors.⁸ Unfortunately, many applicants are now familiar with the concept of BEI⁹ and come prepared with scripted answers, which vary little among candidates, rendering BEI less effective as a useful differentiating evaluative tool.

To capitalize on the strengths of MMI and BEI, avoid their limitations, and evaluate desirable higher-order cognitive skills in real time, we integrated a single, 5-minute, high-stakes, scenario-based simulation encounter into our anesthesiology resident applicant interview process. Our primary objective was to examine whether the addition of a brief simulation would (1) provide added useful information for the resident selection committee; (2) be feasible within the constraints of the interview schedule; and (3) be acceptable to the applicants.

Methods

Setting and Participants

Baystate Health, a Tufts University affiliate, is an 800-bed teaching hospital and tertiary care referral center for western Massachusetts. Each year, we interview approximately 120 applicants for 4 postgraduate year 1 positions and 5 clinical anesthesia (CA) 1 positions. The brief simulation exercise replaced a standard faculty interview, and only candidates who were randomly assigned to interview with the simulation instructor participated in the exercise. As a result, 90 of the 120 applicants during the study period participated in the simulation. Nonparticipating applicants were not penalized in the rankings.

Intervention

We adapted a simulation that we use during CA-1 boot camp that emphasizes teaching and application of the Team STEPPS 2-Challenge Rule.¹⁰ The 2-Challenge Rule states that all health care providers are obligated to question medical decision making repeatedly, using advocacy and inquiry, if they witness a potential patient safety breach. This simulation is simple to operationalize yet intense, allowing exploration of a candidate's critical thinking skills and response to stress.

In our scenario (provided as online supplemental material), the simulation instructor played the surgeon, the applicant played the anesthesiologist, and all other perioperative team members were virtual; the instructor imparted comments if and when indicated. Applicants were expected to use advocacy, inquiry, diplomatic conversation, and resourceful thinking to prevent an impending patient safety breach using this 1:1 role play.

Applicants were informed about the simulation exercise at the beginning of the day when the interview agenda was announced and were told that their simulation performance would be factored into their overall evaluation, although it would not be weighted heavily, since this was a pilot project.

A single interviewer, who is a certified simulation instructor, conducted the 5-minute simulation and the 10-minute debriefing. After an initial 30-second briefing, the simulation lasted 4 minutes, 30 seconds, and the same scenario was used for each applicant to standardize performance assessment. An instructive 10-minute debriefing followed each session during which time the 2-Challenge Rule was defined and recommendations were made on how to implement it clinically and overcome common obstacles (provided as online supplemental material).

Resident applicants were evaluated using a 20-item outcomes-based checklist targeting commonly employed Team STEPPS intervention strategies to prevent a patient safety breach (provided as online supplemental material). Eight of the 20 checklist items evaluated advocacy and inquiry skills, which are critical for effective communication. Checklists were completed by the instructor and scored by adding the number of observed behaviors, with a total possible score of 20. The behavior-based checklist was designed to minimize rater bias, and scores were interpreted with qualitative comments regarding applicant performance. Scores and comments were integrated with other standard application materials to rank candidates.

Based on standard application metrics (table 1), applicants were stratified into the following quadrants prior to numerical ranking: “Highly Desirable,” “Desirable,” “Probably Undesirable,” and “Undesirable.” Applicants' checklist scores were interpreted in the context of this stratification (table 2).

Table 1.

Standard Application Metrics and Simulation Assessments

Table 2.

Credentials and Simulation Performance Resident Stratification

Post-NRMP Match Resident Applicant Survey

To evaluate the utility and educational value of this simulation exercise, the authors developed a voluntary, anonymous, 14-item survey (provided as online supplemental material), which was e-mailed after the 2015 National Resident Matching Program (NRMP) Match to all resident applicants who participated in the simulation. The survey was reviewed by core faculty for clarity, but was not tested prior to distribution. The survey was determined to be exempt from human subjects research by the Baystate Institutional Review Board. Data were captured and managed via REDCap.¹¹

Results

Simulation Checklists

Resident applicants were evaluated using a 20-item outcomes checklist targeting commonly employed Team STEPPS intervention strategies to prevent a patient safety breach (provided as online supplemental material). Scores between 15 and 20 were categorized as excellent, between 10 and 14 were satisfactory, and 9 or below were deemed poor. For the 90 simulation participants, the mean checklist score was 14.2, the median was 15, and the modal score was 16. Forty-five of 90 applicants (50%) received excellent simulation scores, 32 (36%) received satisfactory scores, and 13 (14%) received scores in the poor range.

Integration of Checklist Scores Into Applicant Stratification

Some applicants' stratification changed as a result of this aggregated data (table 3). Simulation performance affected desirability status in 35% of applicants, where 23% ranked higher and 12% ranked lower, and 97% of these applicants moved to an adjacent desirability quadrant. Most notable was the percentage increase in the “Desirable” quadrant as a result of down-ranking applicants from “Highly Desirable,” as well as up-ranking of applicants from “Probably Undesirable.” Fewer than 3% of applicants moved more than 1 quadrant based on their simulation performance.

Table 3.

Presimulation and Postsimulation Resident Stratification

The decision to down-rank resident applicants as a result of the simulation was not made lightly, as this was a pilot project and candidates had little time to prepare psychologically. Resident applicants were ranked higher if they did well on the simulation exercise, independent of baseline credentials.

Post-Match Applicant Survey Results

Of the 90 applicants, 63 (70%) completed the follow-up survey. Of survey respondents, 61% (30 of 49) reported that the simulation exercise did not affect their ranking of the residency training program, while 31% (15 of 49) ranked the program higher, and 8% (4 of 49) ranked it lower.

The educational value of the simulation resonated with applicants, as 100% of respondents were confident or very confident that they would both speak up and know what to say if they encountered a patient safety breach as a result of participating in the 2-Challenge Rule simulation.

Nearly half of respondents (47%, 23 of 49) noted that they subsequently taught others important principles of the 2-Challenge Rule that they learned during the simulation, and 10% (5 of 49) subsequently employed the 2-Challenge Rule clinically. In addition, 94% (45 of 48) of respondents stated that they would apply principles learned about the 2-Challenge Rule to patient care experiences in the future.

The survey also invited open-response comments, allowing applicants to reflect on their simulation experience. Comments were consistent with feedback received during our postinterview debriefing, most notably that the majority of respondents enjoyed the simulation exercise and found it innovative and unique.

A few respondents did note that the simulation added an additional layer of stress to an already stressful day, and some wondered how their performance would affect their rank status after composite data were collected.

Discussion

Resident selection committee members thought that resident applicant simulation training performance was useful in helping stratify candidates when used in conjunction with standard application metrics to determine rank order status.

Because this was a pilot innovation, there are no comparison studies referenced in the literature to assess the true added value of the simulation exercise. With additional research, we hope that predictive validity can be established with respect to our simulation design in assessing higher-order cognitive skills, in a manner similar to the MMI.

There was no additional cost associated with implementing the simulation, as it was conducted in lieu of a standard faculty interview. This did result in some applicants having 1 less interview session.

This study has limitations. It was conducted at a single site, reducing the ability to generalize. Only applicants randomly assigned to interview with the faculty simulation instructor participated in the simulation exercise. In addition, we lack data on the predictive validity of our simulation as an indicator of clinical performance. Longitudinal studies are needed to assess whether a resident applicant's simulation training performance correlates with resident competency.

Based on the utility of this simulation exercise both as an applicant stratification tool and an educational opportunity, we plan to continue using cognitive simulation training during future resident applicant interviews. The case scenario will be changed, however, to avoid “stock answer” pitfalls that have been associated with BEI. We acknowledge that this will admittedly be increasingly difficult to avoid, if this interview method gains popularity.

Conclusion

When used in conjunction with standard application metrics to determine rank status, cognitive simulation during anesthesiology residency applicant interviews was useful in assessing resident applicant higher-order thinking skills and in helping stratify candidates.