Implementing a Holistic Review Toolkit for Faculty Recruitment and Retention

Abstract

Introduction

A diverse workforce improves health care, educational outcomes, and research agendas. Currently, faculty from historically excluded groups remain underrepresented in academic medicine. Resources are needed in academic medical centers for those charged with recruiting and retaining a diverse and talented workforce.

Methods

Informed by the successful use of holistic review in medical student admissions, the Association of American Medical Colleges (AAMC) developed and piloted a toolkit for use in faculty recruitment and retention at five academic medical centers. Institutions led toolkit-guided holistic review workshops ranging from 2.5 to 3 hours in duration for leaders and faculty to identify and prioritize selection criteria used to modify faculty recruitment and retention materials and processes. AAMC surveys at pilot initiation and at two time points during implementation assessed satisfaction with and perceptions of the toolkit.

Results

At the first survey time point, the average toolkit rating across five institutions ranged from 3.2 to 4.4 (out of a maximum of 5.0) for helpfulness and from 3.5 to 4.9 for ease of use. At the second survey time point, the helpfulness of the tools received average ratings of 3.0–4.8. In respondents’ qualitative comments, we identified varying strategies for implementation and few barriers to implementation other than reluctance of some faculty to use scripted tools and concerns about anonymity in the exit interviews.

Discussion

The toolkit was well received and easy to implement. It will be important to study the use of these materials across other institutions, with attention to their impact on faculty diversity.

Educational Objectives

By the end of this session, learners will be able to:

• 1.Assess their institutional readiness for incorporating holistic review strategies for faculty recruitment, selection, and advancement.
• 2.Describe how holistic recruitment practices can expand their institution's search criteria to help the institution achieve its mission and goals.
• 3.Apply the holistic review framework to their institution's faculty recruitment, selection, and advancement processes.

Introduction

A diverse workforce improves health care, educational outcomes, and research agendas.¹ In addition, workforce diversity has frequently been cited as a strategy to improve health across populations, including historically marginalized groups. Elevating health equity as the fifth component of the quintuple aim highlights the disproportionate impacts of health inequities and underscores the relevance of diversity, equity-mindedness, and conscious inclusion to leverage the full benefits of a diverse workforce.^2,3

Despite efforts to improve and sustain faculty diversity, a 4-decade analysis of US medical faculty showed that while female representation has improved, other groups that have been historically underrepresented in medicine (URiM) remain underrepresented in clinical faculty; even greater disparities persist at the leadership level.⁴ Our definition of URiM, meaning underrepresented relative to the numbers in the general population and by setting, has been based on the National Institutes of Health's definition of underrepresented populations in research and includes individuals from racial and ethnic groups that have been shown to be underrepresented, individuals with disabilities, individuals from disadvantaged backgrounds, and other aspects of diversity.^5,6 Compounding efforts to retain diverse faculty, promotion rates for racially and ethnically URiM faculty are lower than their White and Asian colleagues.⁷ The literature about faculty diversity primarily focuses on the impact of implicit bias training for faculty search and recruitment committees; however, these resources do not include toolkits to increase diversity at every stage of the recruitment process, and they do not extend beyond the recruitment phase to include advancement and retention.^8–10 Because holistic review has been shown to be effective in increasing student diversity, we have adapted its principles for use in recruiting and retaining faculty.^11,12

In 2007, the Association of American Medical Colleges (AAMC) launched an advisory committee to formalize the concepts of holistic review for application in medical school admissions. The resulting approach is designed to align with an organization's mission and aims to promote diversity in the medical workforce to address societal needs.¹¹ Since its introduction in 2010, medical school admissions committees have used the holistic review framework to evaluate applicants comprehensively, considering their experiences, attributes, and academic metrics in relation to the institution's mission. In 2016, Baylor College of Medicine (BCM) partnered with the AAMC to pilot application of the holistic review framework in faculty recruitment and selection.¹³ A key adaptation of the faculty recruitment framework was the addition of competencies. Expanding on this collaboration and recognizing the significance of addressing faculty retention, a national working group was created to investigate the framework's broader application across the faculty life cycle.¹⁴ In July 2019, an AAMC-sponsored webinar was held to socialize the need for and premise of the pilot and to invite webinar participants to apply. The webinar participants represented 10 medical schools and one teaching hospital, seven of which applied. The pilot required leadership and human resources endorsement; all applicants had institutional support and were accepted into the pilot. The multisite pilot launched in December 2019. This pilot aimed to adapt and implement the framework for faculty recruitment and retention, addressing a critical gap: the need for processes that increase diversity within the faculty. Due to pandemic challenges, two institutions did not complete the pilot. This left five sites, two from the south, one from the northeast, and two from the west, to fully implement the faculty recruitment and retention pilot.

The final pilot, involving five institutions, has demonstrated promise as a strategy for advancing faculty diversity goals.¹⁴ This project aimed to assess the feasibility and perceptions of applying the tools provided in implementing a holistic approach to faculty recruitment and retention. We assessed readiness for holistic review and satisfaction with applying the model and tools at each institution.

The comprehensive toolkit is designed to provide educational resources and tools for leaders and teams in academic medical centers who focus on faculty recruitment, advancement, and retention. This publication provides information about our pilot's unique application of the toolkit, perceptions of the toolkit, and lessons learned during its implementation.

Methods

Phase 1: Education and Investigation (Steps 1–3)

Five academic medical centers were selected by the AAMC to participate in this pilot based on their organizational commitment to make changes in their recruitment and retention practices.¹⁴ These institutions were diverse in size, location, and type (i.e., public or private) and included BCM, University of Colorado School of Medicine, University of Mississippi Medical Center School of Medicine, University of Nevada, Reno, School of Medicine, and University of Rochester School of Medicine and Dentistry. As the leads of this pilot project, we each assembled institutional implementation teams in late 2019 to build on the work initially piloted by BCM and the AAMC's holistic review team.¹⁵ Before proceeding, it was determined that institutional review board review was not needed for participation.

We worked closely with AAMC staff to create a comprehensive toolkit of resources informed by best practices across the faculty life cycle.¹⁵ The resources were organized to lead institutions stepwise through the change process and included evidence-based tools originally designed by the second author for the AAMC's Leading: Top Skills, Attributes, and Behaviors Critical for Success¹⁶ and Recruiting: Proven Search and Hiring Practices for the Best Talent¹⁷ publications. The AAMC's legal counsel was also engaged to advise on the structure and process of the pilot. Of note, although employment law has not changed based on the June 2023 Supreme Court decision on race-conscious admissions (the law governing employment practice and decisions for hiring is covered by Title VII of the Civil Rights Act, which has long prohibited the consideration of an applicant's race),¹⁸ the AAMC encourages institutions to consult their legal counsel on federal and state laws when restructuring existing strategies to improve the recruitment, retention, promotion, and evaluation of leaders, faculty, staff, and learners representing groups that have been historically marginalized. Additionally, students, faculty, staff, and the local community should all be engaged, and their input considered when adopting and applying these strategies.

Creating implementation teams was a critical preliminary step because this group of people was responsible for shepherding change at each of our institutions. Implementation team members were stakeholders with a deep understanding of institutional context and culture; team members with experience in facilitation, data analysis, project management, and diversity strategic planning; and representatives who could drive change. An overview of and implementation guidance for the entire AAMC Holistic Faculty Recruitment and Retention Initiative can be found in Appendix A. In step 1, implementation teams, drawing on the framework outlined in Appendix B, sought buy-in from key institutional leaders, such as deans, leaders in faculty affairs, department chairs, and human resource professionals. Once the key stakeholders were oriented to equity-minded hiring practices, we reviewed our institutions’ readiness to engage in institutional change using a standardized assessment developed by the AAMC (Appendix C).

During the next step, we collected baseline faculty demographic data (Appendix D), reviewed our current search practices against the AAMC's description of Holistic Search Committee Phases and Steps (Appendix E), identified gaps in our current processes, and updated our search plans based on holistic principles.

Step 3 required each pilot institution to deliver the holistic principles workshop. Workshops ranged in length from 2.5 to 3 hours and were delivered in early 2020. The workshop had several objectives: (1) to learn about the holistic review framework's application for faculty hiring, (2) to select expanded and holistic search criteria aligned with institutional mission and goals, and (3) to identify opportunities to change recruitment and hiring practices. The criteria selected and actions taken in these workshops laid the foundation for the remainder of the pilot process. Tools available for the completion of step 3 included the Holistic Principles Facilitator Guide (Appendix F), which provided an outline of in-person and virtual setup, materials needed, delivery instructions, and a content outline. Each school identified a skilled facilitator in advance to deliver the workshops. A complete slide deck with editable speaker notes was also provided (Appendix G). Each site evaluated its own workshops, but we here provide a standard evaluation based on the workshop goals (Appendix H).

Three activities guided discussion and decision-making among recruiting teams (Appendix I). In activity 1, participants selected Experience, Attribute, Competency, and Metric (EACM) criteria that are best aligned with their institutional mission to prioritize during the faculty and staff selection process. By the end of activity 1, each group had identified their top two to three selection criteria within each domain of the EACM framework. Once the EACMs had been identified, activity 2 helped each group define and operationalize each prioritized criterion. They also developed, using activity 3, an action plan to move the work forward within their implementation unit. Some institutions completed these activities in one sitting, while other institutions divided content delivery and assigned the activities as intersession work.

Phase 2: Recruitment, Selection, and Retention Materials and Process Updates (Steps 4–6)

After the workshop, members of each institution's implementation team used the toolkit materials to align their existing recruitment, selection, onboarding, and retention materials and processes with holistic review principles and the priorities identified. Each pilot site worked with the appropriate stakeholders at their own institution to update their hiring processes and practices. The development and update of recruitment and selection materials constituted a highly integrated process, and although materials were presented as separate areas of focus, changes in one process often affected another. For example, as job descriptions changed, those changes affected interview questions and how candidates were being evaluated. We outline the tools in each area and describe how the holistic principles were incorporated into existing institutional resources below.

In step 4, we began by updating job descriptions, interview questions, submission requirements, and applicant evaluation materials based on the EACMs identified by each implementation team. Tools included Job Description Tools and Resources (Appendix J), which provided sample job description guidelines and an example of a job description showing how BCM incorporated their priority EACMs into a search as well as a curated list of recommended job-posting sites. Interview Questions Tools and Resources (Appendix K) included sample holistic interview questions to identify priority EACMs and competence in diversity, equity, and inclusion. These tools and resources were adapted by each team. Additionally, guidance on interview questions that could pose legal risk to organizations were summarized (Appendix K). Lastly, Interview Rating Tools and Resources sample submissions requirements and a sample candidate rating tool used by BCM were provided as models that could be adopted by each team (Appendix L). A 360-degree reference-checking best-practice guide was also provided (Appendix M).

Simultaneously, utilizing the EACM framework in step 5, each team reviewed their search process, updated their search plans, and conducted hiring package analyses for equity purposes. Search Process Tools (Appendix N) included an overview of the characteristics of a good search, a good-practice guide for the application process, and a template to get feedback on how the candidate experienced the overall search process. Resources on how to form a search committee (Appendix P) and help mitigate bias throughout the recruitment and selection process (Appendix P) were provided.

In step 6, retention materials at each pilot site were reviewed, including onboarding materials, career development tools, and exit interview processes. A modifiable department chair onboarding guide was provided for reference (Appendix Q). Career development tools included a series of tools for holding career development discussions (Appendix R). Additionally, a comprehensive mentoring resource guide was developed by the University of Colorado Anschutz Medical Campus Department of Family Medicine because of pilot participation (Appendix S). Lastly, a sample exit survey and exit interview questions were developed by BCM (Appendix T). These materials were adapted for use at each institution. The phase of updating materials and processes ranged from 6 to 18 months depending on the institution.

Phase 3: Implementation and Continuous Improvement (Steps 7–8)

After the materials had been updated, we presented and socialized the updated forms and interview questions to hiring managers and search committees (step 7). To operationalize new hiring processes, the University of Colorado Anschutz Medical Campus Department of Family Medicine developed an Equitable Hiring Tool to help guide and track their new hiring processes (Appendix U). Finally, in step 8, we continually monitored and improved our processes. We reviewed how the process was going with our pilot team and tracked changes in hiring and retention over time. Tools to help with this tracking can be found in Appendix V. As with all new policies and interventions, it was important to communicate this information broadly. We developed communication strategies recommending that all pilot institutions share their work and outcomes broadly at their institution.

Evaluation of the Toolkit

We assessed the impact of our pilot by participating in a multisite evaluation led by the AAMC, including surveys throughout the implementation process designed to measure each institution's satisfaction with and perceptions of the toolkit. Each institution provided its feedback on the faculty holistic review process and rated the helpfulness of each domain of tools at two points during the implementation. In June of 2021, pilot institutions completed an initial evaluation survey about the utility of the resources, ease of adoption, and estimated amount of time to adapt the tools. It is important to note that at the time of this survey administration, some institutions had not yet had the opportunity to fully adopt the toolkit, as some of our recruitment efforts were halted by the pandemic, budget cuts, and hiring freezes. As the implementation of the pilot progressed, we completed a second survey in the summer of 2022 to assess the helpfulness of the tools used in postworkshop implementation and asked open-ended questions to assess strategies for adaptation, how we used the tools, and barriers to adoption. At this point in the implementation, most institutions had implemented the tools for recruitment, and two out of five had implemented the tools for retention. Every institution completed both surveys (see both surveys in Appendix W).

Results

In June of 2021, participants evaluated the holistic faculty recruitment tools created for this pilot. The tools in the toolkit were assessed on how helpful each was, how easy it was to adapt each one for respondents’ particular use, and how long it took to adapt them. Each institution identified one individual to complete the survey on behalf of the pilot team (n = 5). Table 1 shows the average ratings of the five pilot institutions. Overall, the toolkit was found to be helpful, with ratings of helpfulness ranging from 3.2 to 4.4 on a 5-point scale (1 = not at all helpful, 5 = very helpful). Institutions also evaluated the tools as easy to adapt for their particular use, with ratings ranging from 3.5 to 4.9 on a 5-point scale (1 = very difficult, 5 = very easy). Participants were also asked how many hours it took for them to adapt the tools, and responses ranged from less than 4 hours to 4–8 hours.

Table 1. Pilot Participant Evaluation Feedback: Utility, Ease of Use, and Time to Adapt Tools (June 2021 Administration).

The Sample Interview Questions and the Holistic Review Applicant Evaluation Tool were viewed as the most useful recruitment tools (average of 4.4). The Sample Interview Questions had the easiest reported adaptation of use (4.9) and were adapted on average in less than 4 hours. Similarly, the Holistic Review Applicant Evaluation Tool was considered easy (3.7) to quickly adapt (4–8 hours). Onboarding and Retention Tools received the lowest rating (3.2) but were considered easy to adapt (4.1), with adaptation taking 4–8 hours.

In the summer of 2022, pilot institutions were again asked to provide feedback on the helpfulness of the tools as well as answer open-ended questions assessing barriers to implementation, strategies for adaptation, and how the tools were used in their search process. Table 2 summarizes the evaluation data and provides representative quotes of themes found in the qualitative responses about strategies for implementation, application of the tools, and barriers to implementation. During this administration, participants were not asked to evaluate the Holistic Review Applicant Evaluation Tool.

Table 2. Pilot Participant Feedback: Tool Implementation, Application, and Implementation Barriers (Summer 2022 Administration).

At this administration, participants continued to rate the tools as helpful, with scores ranging from 3.0 to 4.8 on a 5-point scale (1 = not at all helpful, 5 = very helpful). The Sample Interview Questions were again rated as the most useful (4.8). Illustrative comments on strategies for implementation showed that questions based on the EACMs that teams had selected and agreed upon were used to screen candidates during interviews and to replace less structured processes for screening applicants. When asked about barriers to implementation, most participants identified none, and one institution noted that faculty can be resistant to scripted interviews. The Exit Interview Tools were the lowest rated (3.0), and institutions used them as a template to create their own interview or script. One school mentioned the barrier of threats to anonymity when using the Exit Interview Tools, as demographic data were included in the questionnaire and this could compromise anonymity in a small school.

Discussion

Across the two survey points, our results show overall favorable impressions of the educational activity with respect to helpfulness of the tools as well as their ease of adaption and adoption. This pilot project monitored the application of a holistic faculty recruitment systems model in five dissimilar institutions and revealed favorable reception to the process and toolkit resources. We found that the toolkit was well received and easy to implement and that barriers to adoption were documented. Survey results focused on the helpfulness of tools, time to adapt the recruitment tools, and barriers to implementation. At the 2021 administration, helpfulness of faculty recruitment tools was high (range: 3.2–4.4), as was recruitment tools’ ease of use (3.5–4.9), suggesting that the tools were easy to use and adapt. The perceived time to adapt recruitment tools for use was less than 8 hours across all nine tools. The 2022 administration also revealed high satisfaction with the tools (3.0–4.8), with varying strategies for implementation and specific barriers to adoption noted. Thus, satisfaction with the toolkit was high, and ease of use and time to adapt the tools for use across the five institutions were reasonable and feasible.

We developed a process including a workshop activity and learned that application and implementation of a faculty-focused holistic recruitment systems model required flexibility to accommodate variations across the five institutions. This project began in 2019, and all institutions experienced challenges related to the global pandemic. In addition, the racial awakening incited by the murder of George Floyd and others contributed to the climate of our nation. Institutionally, there were direct effects on the workforce including lifestyle changes, various forms of discrimination and prejudice, environmental implications, health and mental health concerns, and delivery of educational programming.

This multisite pilot of the faculty holistic review toolkit included numerous strengths: (1) The framework drew upon the evidence-based holistic review process for medical student selection; (2) it filled a gap in the literature regarding resources to recruit a diverse faculty workforce by providing a toolkit that all institutions rated highly; (3) the institutions in the pilot were highly diverse, suggesting that the toolkit could be successfully implemented across a range of academic health centers with faculty and staff; (4) the model ranges beyond recruitment to address advancement and retention; and (5) the model provides a promising practice to increase equity in the hiring process.

The limitations of our work varied across a small number of sites, and each institution's operational team independently created implementation priorities, which limits the generalizability of our findings. Additionally, since each institution completed its own workshop evaluation, we could not report on the workshop results as a whole. To better standardize the workshop evaluation for this publication, we have created an evaluation (Appendix H) that aligns with the learning objectives for the workshop (slide 2 in Appendix G). The timing of the pilot with the COVID pandemic limited our ability to evaluate the effectiveness of the toolkit with respect to faculty retention, as only two of the five pilot institutions were able to implement retention tools. The small sample size and the short timeline of the pilot project compared with the long-term nature of this assessment were also limitations. Although the pilot primarily focused on faculty, the framework has applicability to staff. Additional limitations include a lack of assessment of the outcomes of the toolkit, especially the ability of the processes and tools to increase the diversity of faculty who are recruited and retained in academic medicine. Data collection on these outcomes is underway at our pilot institutions, and next steps include analysis and dissemination of these data.

This holistic faculty recruitment pilot demonstrated a found pilot approach, where best practices from holistic admissions were applied to faculty recruitment and retention across five institutions, receiving favorable feedback. Creating a diverse workforce is an ongoing process with significant long-term implications for institutions. Using this model, as detailed in Appendix A, is the first step. The next steps involve expanding the toolkit's implementation beyond the pilot institutions to assess its short- and long-term impacts on a broader scale while continuing to monitor outcomes within the pilot group. Additionally, the incorporation of the latest legislative rulings into our processes may need to occur, depending on location.