Bias Breakers: Continuous Practice for Admissions and Selection Committees

Abstract

Introduction

Admissions and selection committees face challenges in identifying and mitigating biases in policies, processes, and discussions. Past bias training has focused on defining bias and presenting the negative impact of bias for committees.

Methods

This interactive training used committee comments, reflection, and audience response to enhance the contextual recognition of bias in admissions and selection processes and practices. For each bias type, we presented specific mitigation strategies and examples. The workshop was offered at four medical schools between December 2020 and April 2021. Participants were committee members (n = 126), largely medical school faculty, involved in MD, MD/PhD, and residency program selection at participating schools. A paired pre- and postworkshop assessment was conducted for each session to determine effectiveness of the workshop.

Results

Mean scores for each of the postassessment items ranged from 4.0 to 4.2 and were statistically significantly different from the preassessment scores per respective item. The results of a paired two-way t test found that these pre- to postworkshop assessment score increases were statistically significant across all assessment questions (ps < .001). Participants reported in their comments that the workshop was effective in establishing a safe and judgment-free learning environment to explore and identify biases and build skills and confidence for mitigating them.

Discussion

Interactive and applied bias training can be an effective strategy to advance committee culture and practice in recognizing and mitigating bias. This workshop provides committees with ongoing tools for equity practice in selection and decision-making.

Educational Objectives

By the end of this activity, learners will be able to:

• 1.Identify the types of bias present in cognition and decision-making.
• 2.Identify manifestations of bias in committee processes, policies, discussions, and decisions.
• 3.Apply strategies and skills to mitigate bias in committee processes, policies, discussions, and decisions.
• 4.Build skills to establish a culture of bias recognition for practitioners and committee members.
• 5.Achieve more diverse, equitable, and inclusive outcomes by recognizing and mitigating bias during selection processes.

Introduction

Admissions and selection committees face challenges in mitigating structural and interpersonal bias during their processes and deliberations. Over the last 10 years, a great deal of attention has focused on unconscious bias awareness and training for committees in undergraduate admissions, residency selection, and faculty search processes.¹ Many medical schools have implemented stand-alone or annual training experiences that generally take place during the onboarding of committee members.^2,3 These trainings are often delivered as a didactic workshop focused on general aspects of interpersonal unconscious bias and do not address the structural biases in the admissions process. Onetime training does not typically employ active engagement in bias recognition as applied to the admissions context. A continuous learning pathway to support a culture of bias recognition also tends to be lacking in typical bias training. To achieve more equitable admissions outcomes, committees must continuously engage in the recognition of biases and practice a culture of bias mitigation.^4,5

While holistic review practices in admissions have shown some success in diversifying UME, bias remains a significant barrier in committee processes. Existing literature has shown how structural and interpersonal bias perpetuates inequitable outcomes in assessments used for admission to undergraduate and graduate programs, thus not always fulfilling the aims of holistic review.^6,7 Holistic review in admissions, a process that recognizes varying levels of social capital, academic readiness, identity privilege, personal/economic resources, and different lived experiences, can facilitate a well-rounded and diverse class of students. Even stronger outcomes may be achieved through intentional, continuous bias recognition and mitigation by committees that build these skills into their culture and practice.

The Bias Breakers workshop improves upon existing efforts to address bias in admissions and selection committee practices by leveraging reflection and self-awareness as vehicles to identify and name both structural and interpersonal biases deeply embedded in individual mindsets and in the practices of admissions and selection committee environments. As leaders in diversity, equity, and inclusion (DEI), we have grown weary of trainings that raise awareness and ultimately conclude by merely admiring the problem and phenomenon of bias. Onetime delivery for committees is insufficient in changing selection practices, and this workshop is a response that adds to the literature by answering the question “We know about bias, now what?” Our Bias Breakers workshop specifically draws on the work of John Dewey,^8,9 who is credited with being a key originator of the concept of reflection and considered it to be a special form of problem-solving. Dewey viewed reflection as an active and deliberative cognitive process, one involving sequences of interconnected ideas, taking into consideration underlying beliefs and knowledge and allowing for doubt and perplexity before possible solutions are reached.¹⁰

Bias Breakers adapts Dewey's theoretical notion of reflective action by (1) identifying over 23 structural and interpersonal biases that often go unseen and unchecked in existing practices of admissions and selection committees; (2) illustrating the interplay and manifestation of these biases in the behaviors of individual members and in the committee as a whole; (3) providing specific individual and committee mitigation strategies that support changes to the mindsets, practices, and policies in the admissions and selection committee environment; and lastly, (4) providing continuous learning tools to support intentional development for admissions professionals and committees that can be utilized as series of learning opportunities over the course of the admissions season. For example, before meetings, one or two slides from the learning tool can be presented for discussion to keep the concept of bias in the forefront of consciousness, thus following Dewey's view that a complete cycle of professional doing needs to be coupled with reflection, ultimately leading to action.⁸ The four theoretical components of Bias Breakers aim to instill habits of thought and practice cultivating self-discipline and committee norms that contribute to a culture of bias recognition and mitigation. Since bias is inherent in human cognition, continuous tools are needed to maintain awareness and mitigation skills.

Methods

We utilized the six-step Kern model for the design, implementation, and evaluation of our workshop.¹¹ The applied framework began with problem identification and general needs assessment. We, the authors, were experienced admissions and DEI practitioners with a combined 30+ years of experience who had witnessed many of these challenges through firsthand discussions, document reviews, and committee proceedings. Through literature review, peer conversations, and requests at our own institutions, we assessed a need for practical bias training that was both interactive and ongoing.

Next, we established goals and objectives. The overall goal of the workshop was to provide committees with a concrete set of tools to recognize and mitigate bias in their tools, processes, and discussions. The learning objectives included the following:

• 1.Identify the types of bias present in cognition and decision-making.
• 2.Identify manifestations of bias in committee processes, policies, discussions, and decisions.
• 3.Apply strategies and skills to mitigate bias in committee processes, policies, discussions, and decisions.
• 4.Build skills to establish a culture of bias recognition for practitioners and committee members.
• 5.Achieve more diverse, equitable, and inclusive outcomes by recognizing and mitigating bias during selection processes.

These objectives were met through the use of various examples of committee comments representing the types of biases present in admissions and selection. The committee comments were generated from our collective experience as admissions committee members. Using actual committee comments was critical to the learning design, as it brought a level of specificity that learners could relate to. Participants used an anonymous audience response interface to reflect on how biases manifested in their specific institution or committee after being introduced to a bias type with examples of comments. Our visual aids were custom graphics designed to facilitate the learning objectives by reinforcing the bias concepts through pictures. Seeing the pictures in the initial slide deck, review slide deck, and summary handout helped participants recall concepts with visual anchors when revisiting them later in the season or during a meeting.

The workshop employed innovative and interactive educational strategies, including audience response and reflection. The material was designed for active learning with audience response polling and free-text responses. Audience response, an effective tool allowing CME learners to enhance their engagement,^12,13 was particularly essential since these workshops were conducted online using group meeting platforms during the COVID-19 pandemic. The technology could be used in a classroom or in virtual meetings. We collaborated with medical illustrators who created original graphics for the learning tools to incorporate visual learning. Pictures enhanced engagement in learning and improved comprehension and recall.¹⁴

Although we delivered the slide deck in a single workshop, we also created a review slide deck for spaced repetition, an effective learning strategy for meaningful retention of information.¹⁵ Committees might already be in the habit of reviewing a mission statement or conflict of interest agreement at each meeting. The Bias Breakers review deck could be used similarly for continuous review and enhancement of committee culture and practice. Our final learning strategy was reflection, an effective tool for providing meaning, improving decision-making, fostering lifelong learning, and recognizing gaps in knowledge or practice.¹⁶

For the implementation phase of the Kern model, we reached out to institutions in need of bias training for their admission and selection committees. The 90-minute workshop was presented at four different medical schools. Three schools hosted the workshop for their UME admissions committees; one school was a residency selection committee within a department and an MD/PhD committee done through recorded delivery. The medical schools volunteered to host the workshop virtually, and it was voluntary and provided free of charge. Institutional review board approval was obtained in advance of the project through the California University of Science and Medicine.

Designed for admissions and selection committees, the workshop was also applicable for residents, fellows, and other health care professionals. The ideal facilitator was a professional in a leadership position with experience in bias training or DEI. Facilitators needed to be familiar with admissions and selection processes. They could be a staff or faculty member in the Office of Admissions with experience in bias training or an individual serving in an institutional role in DEI.

The workshop had several components, including an interactive PowerPoint presentation (Appendix A), a facilitator guide (Appendix B), a review slide deck (Appendix C), and a bias overview handout (Appendix D). The workshop could be delivered online or in person. The interactive polling, if administered in person, served to provide a safe, anonymous space for participants to discuss bias and how it manifested in their committees. We designed surveys (Appendix E) for participants using a paired pre-post design to measure change in accomplishing the learning objectives. Although the assessment for the sessions included here utilizes online delivery, we have also delivered the workshop in person with equally successful results. The required elements for the workshop included the following:

• •PowerPoint presentation (Appendix A): the main content for the workshop. It began with an overview of bias and then followed a uniform format through 23 cognitive biases. Each bias started with comments, followed by definition of its type, an outline of mitigation strategies, and audience reflection/application. The visuals on each slide enhanced the concepts presented.
• •Facilitator guide (Appendix B): a slide-by-slide guide for facilitators featuring session timing and key points within the PowerPoint presentation.
• •Bias Breakers review deck (Appendix C:): a consolidated version of the 23 biases covered in the workshop. Each slide in the deck presented the type of bias, definition, mitigation, and a corresponding visual. This supplemental tool could be provided to committees for later use. As committees meet, one slide could be presented as a refresher. Practitioners could also include one slide in communication disseminated throughout the season to committee members.
• •Bias Breakers quick reference (Appendix D): a reference sheet with all 23 biases listed on a single, double-sided page for practitioners to print for their committees to have on hand and review. The visuals would remind committee members of the concepts. We recommend providing the reference handout to all committee members for each meeting.
• •Pre- and postworkshop surveys (Appendix E): evaluation tools asking attendees to assess their knowledge and skills around bias recognition and mitigation before and after the workshop on 5-point Likert-type scales (1 = very poor or none, 5 = excellent or very high). Colleagues may wish to do a pre-post assessment, as we did, to report on training outcomes to internal or external stakeholders.

One or two cofacilitators administered the workshop, having allotted 2–3 hours to review the content prior to the training. To increase the effectiveness of the workshop, facilitators utilized an audience response system (e.g., the polling feature in Zoom or Teams, Slido, Polleverywhere, Kahoot, TurningPoint, etc.). Note that the polling feature was used whether the workshop was delivered online or in person, as anonymous responses allowed participants to share examples of bias without fear of judgment. Setup of questions in the audience response system took about an hour for experienced users. Whether the workshop was presented in person or online, the ability to project/share the PowerPoint presentation and audience responses was needed. Printed copies of the handout (Appendix D) were optional.

The minimum length of the workshop for content delivery was 90 minutes. A 120-minute version also allowed ample time for discussion. In our experience, participants were very engaged, and the interactive element enhanced learning and investment in newly acquired skills. Discussion time helped participants apply the learning to their specific committee context. Timings for both versions are outlined below:

• •
  90-minute workshop:

  • ○
    Preworkshop evaluation (optional).
  • ○
    Introductory slides on bias review: 10 minutes.
  • ○
    Bias Breakers: 75 minutes (about 3 minutes per bias).
  • ○
    Questions: 5 minutes.
  • ○
    Postworkshop evaluation (optional).
• •
  120-minute workshop:

  • ○
    Preworkshop evaluation (optional).
  • ○
    Introductory slides on bias review: 10 minutes.
  • ○
    Bias Breakers: 75 minutes (about 3 minutes per bias).
  • ○
    Discussion: 30 minutes.
  • ○
    Questions: 5 minutes.
  • ○
    Postworkshop evaluation (optional).

The workshop learning objectives and content were evaluated by participants using a pre-post design. Each participant was asked to create a unique identification code so that their evaluations could be paired for analysis. The evaluations included comments so we could get a sense of the learner experience and areas of the workshop that were effective or needed improvement. We received overwhelmingly positive evaluations and feedback.

Results

This workshop was implemented with four committees at the following institutions: Loyola University Chicago Stritch School of Medicine, Icahn School of Medicine at Mount Sinai (one UME and one GME group), and University of Illinois College of Medicine (N = 176). Individuals invited to the workshop were current members of their admissions committees for MD, MD/PhD, or residency programs at their respective institutions. The majority of workshop participants in our sample were MD committee members. The workshop was facilitated by two presenters, both senior associate deans in DEI roles in medical education with experience serving on an admissions committee and/or as a UME admissions dean. Across all institutions, a total of 126 unique participants submitted both pre- and postworkshop assessments (up to 74 for MD admissions committee members and up to 52 for MD/PhD or residency program committee members). Across each of the five pre- and postworkshop assessment items, the number of completed matched responses ranged from 124 to 126. Only responses that had both pre- and postworkshop assessment answers associated with identical anonymized IDs were considered in statistical analysis.

The mean scores for each of the preworkshop assessment items ranged from 3.0 to 3.5. On average, respondents felt least confident in their individual ability to apply strategies to mitigate bias in admissions/selection processes (M = 3.0, SD = 0.8), followed by their committee's capability to establish a culture of bias recognition for admissions practitioners and committee members (M = 3.3, SD = 0.7), their individual ability to identify the manifestations of bias admissions and selection processes (M = 3.3, SD = 0.7), their individual knowledge of bias in cognition and decision-making (M = 3.4, SD = 0.7), and, finally, most confident in their committee's capability to achieve more equitable outcomes by recognizing and mitigating bias (M = 3.8, SD = 0.8). Mean scores for each of the postworkshop assessment items ranged from 4.0 to 4.2 and were statistically significantly different from the preworkshop assessment scores per respective item. After attending the workshop, respondents reported feeling confident in their committee's capability to establish a culture of bias recognition for admissions practitioners and committee members (M = 4.0, SD = 0.8), followed by their committee's capability to achieve more equitable outcomes by recognizing and mitigating bias (M = 4.0, SD = 0.7), their individual ability to apply strategies to mitigate bias in admissions/selection processes (M = 4.1, SD = 0.7), and, finally, most confident in their individual ability to identify the manifestations of bias in admissions and selection processes (M = 4.2, SD = 0.7).

Mean scores across all assessment items increased from the pre- to postworkshop assessments. Items demonstrating the largest score increases corresponded to those that asked respondents about beliefs in their individual ability to effect change. Specifically, responses regarding individual ability to apply strategies to mitigate bias in admissions/selection processes increased by 1.0 rating units, responses regarding individual ability to identify the manifestations of bias admissions and selection processes increased by 0.9 rating units, and responses describing individual knowledge of bias in cognition and decision-making increased by 0.8 rating units. Mean score increases for assessment items evaluating respondents’ belief in the ability of their committees to effect change were similar, though slightly smaller in magnitude. Responses regarding the capability of the respondents’ committee to establish a culture of bias recognition for admissions practitioners and committee members increased by 0.7 rating units, while respondents’ level of confidence in their committee's capability to achieve more equitable outcomes by recognizing and mitigating bias increased by 0.4 rating units. The results of a paired two-way t test found that these pre- to postworkshop assessment score increases were statistically significant across all assessment questions (ps < .001). The absolute value of each associated t statistic was greater than 6.2.

A total of 126 and 124 respondents answered the fifth and sixth postworkshop assessment items, respectively. On average, the quality of the Bias Breakers presentation was rated very highly (M = 4.5, SD = 0.6), and participants strongly agreed that the learning objectives of the workshop were met (M = 4.5, SD = 0.6).

Residency program admissions participants could respond to one free-text question in the preworkshop assessment, and all participants could respond to up to two free-text questions in the postworkshop assessment. Nearly all participants (73 out of 74, 99%) answered the preworkshop question, most commonly stating that they hoped to learn strategies and techniques for addressing/mitigating bias in their institutions; to help reduce, mitigate, or eliminate bias; or to improve their knowledge/understanding of bias. Many respondents were interested in increasing their personal awareness of existing biases and recognizing their own personal biases. Several also hoped to learn tools for addressing bias that could be shared with their colleagues in medical admissions. In response to the first postworkshop free-text question, nearly half of respondents (60 out of 124, 49%) wrote that using real-life examples of bias was a strength of the workshop. Others indicated that the presenters effectively communicated the different types of bias and strategies for addressing/mitigating them, utilized interactive elements, and implemented anonymous, judgment-free polling via software. In response to the second question, nearly half of respondents (21 out of 50, 42%) suggested that future workshops could include more practical strategies and exercises (e.g., case studies) and more time for group discussion.

Discussion

The data support the idea that this was an effective workshop for bias training and mitigation. All sessions had strong engagement from participants that demonstrated their learning and application of the concepts. During the residency program workshop, polling software revealed strong participation in the form of real-time audience responses to a sample of four questions. Of the 36 individuals who responded to the first polling question (“What comes to mind when you hear the word bias?”), the majority associated the term bias with concepts of unfairness, racism, inequity/disparity, and judgment. This initial question helped learners recognize that they generally associated bias with something negative, which made talking about it or even learning about it difficult. After this response, facilitators pointed out that bias was part of how human brains work and that people have various biases that may have positive or negative influences on the selection process.

Forty-eight individuals responded to the second question (“What qualities do you not favor or dislike in applicants?”), with participants stating that arrogance, overconfidence, self-centeredness, apathy/disinterest, and inauthenticity were among qualities they disliked in applicants. Conversely, 61 respondents to the third question (“What qualities do you tend to favor in applicants?”) indicated that enthusiasm/high energy, passion, self-awareness, compassion and empathy, articulateness, authenticity, flexibility, and humility were among qualities they liked in applicants. After these exercises, participants were able to reflect on how subjective these qualities were and how they overwhelmingly concerned interpersonal interpretation of the applicant, not necessarily direct observation or objective data. Facilitators engaged in discussion with participants about how favoring humility as a characteristic was a type of bias and how these tendencies had the potential to become harmful when they went unrecognized and unacknowledged as part of the factors that drive decision-making.

After presenting the actual committee comments for each bias type, the workshop integrated bias-mitigation practice by providing an opportunity for participants to reflect on how that specific bias type could be observed within themselves and others. The reflexive and reflective elements were particularly effective in helping to reduce the stigma around bias and move committees forward in effective mitigation practice. Finally, 37 individuals responding to the final question (“What are some mindsets, practices, or behaviors that impede the committee from fostering a culture of accountability for our biases?”) stated that tradition/unwillingness to change, groupthink and/or denial and acceptance of bias, time constraints, fear of speaking out, lack of training regarding bias, and fear of embarrassment were the among the most important barriers impeding admissions committees from fostering a culture of accountability regarding their bias. Giving participants an opportunity to reflect on barriers helped reduce stigma and normalize the challenges associated with managing bias in a committee process.

Although the examples we chose for this workshop came from UME, the content could be easily adapted and applied using comments for faculty search committees and resident selection committees. The tools presented are universally applicable to decision-making processes and can help committees advance their culture and practice of bias recognition and mitigation. As we emphasize in the workshop, the goal of training is not to eliminate bias but to illuminate it.

Limitations

The interactive aspect of this workshop could be considered a limitation for facilitators not familiar with audience response systems and software. We recognize that we have a bias that these tools are easy to use, but some individuals may be intimidated by the prospect of setting up and using audience response polling. For a novice to this type of learning modality, practice sessions may be helpful to ensure that the technology works as planned and the facilitator is comfortable with the interface. However, this may increase the preparation time required for giving the workshop. Other limitations include limited institutional scope of just four schools, which were selected by convenience from our professional networks. We chose schools that were ready and willing to engage on this topic, and therefore, we cannot assess the workshop's effectiveness for committees whose leaders may be less supportive or whose culture may not yet be ready to engage in this way.

COVID-19 impacted the workshop delivery, and two groups in our sample participated in a recorded version of the workshop. The effectiveness of the workshop, according to postworkshop surveys from the recorded sessions, was not impacted by the delivery method of live versus recorded. We believe that the concepts framing and illuminating bias have merit alone, as those viewing the recorded version showed improvements in awareness of bias and confidence in addressing it. Participants using the recorded version did not experience live delivery with facilitated reflection and therefore were not able to compare the experience in the evaluations for themselves. We conclude that reflection and audience response in real time enhanced learning for participants because those who experienced the live version also demonstrated benefits in our survey questions and expressed gains in their comments.

We further acknowledge that we, the authors, are experienced leaders in DEI with long-standing experience with admissions committees. This may have impacted the results of the workshop, and we are unable to assess whether practitioners and leaders with less experience would experience similar results.

Finally, we emphasize that eliminating biases is not the goal of the workshop, as that is impossible. Illuminating bias and increasing awareness of how it operates in selection and admissions discussions are the goals. The manifestations and impacts of bias are vast, and our workshop and tools are a resource to assist committees in advancing their recognition, acknowledgment, and mitigation of bias.

Conclusion

Bias Breakers is an effective tool for assisting admissions and selection committees to increase knowledge and confidence regarding bias-mitigation practice. The interactive nature, safe learning environment facilitated via polling software integration, and reflective approaches help participants develop actionable strategies for change at their own institutions. We hope that the additional review deck and reference guide make integration of bias-mitigation practice into all phases of the process more feasible. More time and attention should be dedicated to bias mitigation in medical education and possibly be extended to other practice areas, such as patient care and pedagogy.