Preliminary Evaluation of Applicants to Master's Programs in Speech-Language Pathology Using Vignettes and Criteria From a Holistic Review Process

Abstract

Purpose:

Little is known about how others evaluate applicants to master's programs in speech-language pathology along criteria used during holistic review despite more programs adopting holistic review. This knowledge gap limits our understanding of whether holistic admissions may offer a more equitable pathway to entering speech-language pathology. This study investigated how faculty and PhD students evaluated applicants to master's speech-language pathology programs along criteria used during holistic review.

Method:

We administered a survey online through a Qualtrics platform. Respondents (N = 66) were faculty and PhD candidates in U.S. speech-language-hearing departments. Survey blocks included demographics, professional background, and vignettes. Vignettes featured profiles of applicants to master's programs in speech-language pathology. Vignettes systematically varied in the indicators of applicant criteria, which were specified at low, moderate, or high levels or not specified. After reading each vignette, respondents rated the applicant and indicated their admissions decision. Analysis included descriptives.

Results:

Relative to an applicant who was at a high level for all indicators except cultural and linguistic diversity, respondents ranked applicants who varied in their indicators of criteria levels lower. Respondents were also less likely to make an explicit “accept” decision (vs. “waitlist” or “reject”) for this latter group of applicants.

Conclusions:

Even when implementing criteria used during holistic review, applicants who vary from a “high-achieving” stereotype may still face barriers to entry. Future work is needed to understand the precise nature of how holistic admissions review may play out in actual practice and help increase diversity in the profession.

In the United States, some graduate speech-language pathology graduate programs are adopting holistic review and moving beyond traditional indicators to evaluate applicants (Guiberson & Vigil, 2021). From an intersectional perspective, holistic review removes barriers to entry that disproportionately impact racial/ethnic minorities (Crenshaw, 1989). However, it is unknown how criteria used during holistic review function in speech-language pathology. Of particular interest is how indicators of applicant quality (e.g., recommendation letters) relate to evaluation of personal characteristics (i.e., criteria used during holistic review) and admissions decisions. These criteria are often social constructs, such that applicant ratings rely on individual interpretation of what a strong profile looks like. For example, a faculty member who uses a personal statement to evaluate academic ability might actually be judging the applicant's adherence to stylistic conventions in academic writing, which is something that applicants from more privileged backgrounds are more able to emulate (McGlynn, 2016).

More broadly, there is a need to understand how holistic review in speech-language pathology may help diversify the profession. The American Speech-Language-Hearing Association (ASHA, 2020a, 2020b) reports that less than 10% of its members are members of color, and even fewer are bilingual service providers. These shortages may be tied to underrepresentation of culturally and linguistically diverse (CLD) master's students in speech-language pathology (Horton-Ikard et al., 2010). In 2018–2019, 34.6% of applicants to master's programs in speech-language pathology received an offer of admission, among a total of 60,784 applications to master's degree programs (Council of Academic Programs in Communication Sciences and Disorders [CAPCSD] & ASHA, 2020). Of 19,185 master's students overall in speech-language pathology, 20.3% of first-year students were racial/ethnic minorities (CAPCSD & ASHA, 2020). However, it is unknown how many applications came from unique, minority applicants (CAPCSD & ASHA, 2020). Considering that minorities are persistently underrepresented in ASHA, it may be that few minorities ultimately become speech-language pathologists (SLPs) because admission is the point of access for entry to the profession (Association of American Medical Colleges [AAMC], 2013; Boske et al., 2018). In all, understanding how applicants are evaluated through holistic review is essential for understanding pathways forward for intersectional excellence—and excellence overall—in the profession. This report describes how faculty and doctoral students evaluate applicants to master's programs in speech-language pathology along criteria used during holistic review.

Holistic Review

Holistic review is a selection process that considers the whole applicant, including what they would bring to the program (AAMC, 2010, 2021; Posselt, 2016). Its purpose is to create a flexible, individualized evaluation of applicant abilities relevant to success (AAMC, 2010; Kent & McCarthy, 2016). The tenets of holistic review are as follows: (a) criteria are broad, mission aligned, and consider diversity as integral to excellence; (b) using applicant experiences, personal characteristics, and academics to inform applicant review in an equitable and evidence-based manner; (c) consideration of applicants' contributions to the class, institution, and profession; and (d) consideration of applicant race/ethnicity to achieve institutional goals related to institutional mission (AAMC, 2010, 2013, 2021; Glazer et al., 2014). As such, diversity is not the goal but a means to achieving educational goals and institutional missions (AAMC, 2010, 2014).

Practices and Procedures

To ensure fidelity of holistic review, programs must have practices and procedures in place for each stage of holistic admissions: screening, interviewing, and selection of applicants for admission (Glazer et al., 2014). The holistic review scoring model provides guidance on specific practices and procedures (AAMC, 2013; Glazer et al., 2014). Under this model, schools may adopt the following: (a) evaluation of applicant criteria related to specific missions or goals of the school (e.g., research mission), (b) using an admissions mission statement that includes diversity, (c) consideration of nonacademic criteria in addition to academic metrics in screening, (d) evaluation of nonacademic criteria related to applicant background or experiences in screening (e.g., socioeconomic status [SES]), (e) selection of students from the waitlist using the school's missions or goals as guidance, and (f) providing training for the admissions committee related to school mission and diversity (Glazer et al., 2014).

Holistic Review in Speech-Language Pathology and the Health Professions

Speech-language pathology. A survey explored holistic review in accredited graduate speech-language pathology programs nationwide (Guiberson & Vigil, 2021). We report the key findings here. First, although a majority of programs reported using holistic review, their exact usage varied: 14% reported using holistic review, 46% reported using some holistic review practices, and 40% reported not using holistic review. Of programs using at least some holistic review practices, 72% reported a positive impact, with more diversity in the program, more well-rounded students, and better identification of clinically competent applicants. Furthermore, although diversity increased, applicant grade point average (GPA) and graduate record examination (GRE) scores and graduate outcomes (i.e., graduation rate, employment rate, and Praxis outcomes) did not change. Of the programs not using holistic review, 51% indicated concerns that admitted students would not be as academically prepared. Furthermore, only 29% of respondents believed CLD students faced barriers to entry to master's programs, including academic ability and preparation and proficiency in mainstream American English.

Second, programs varied in the aspects they implemented at each stage. In screening, 85% of programs reported using a GPA cutoff (range: 2.75 to ≥ 3.3), and 73% of respondents ranked GPA as the most important criterion. Few (26%) programs reported using a GRE cutoff score. In applicant review and selection, 61% of respondents ranked GPA and 30% ranked GRE scores as an important or the most important criterion, respectively. Nearly half the respondents reported considering bilingualism/multilingualism during these stages. Other common practices across programs included consideration of nonacademic criteria (e.g., interpersonal skills, oral communication skills, professionalism, and critical thinking) and diversity essay responses.

Overall, these findings suggest holistic review is on the rise in speech-language pathology, although many programs do not fully follow the AAMC (2013) model. If only some practices are in place, holistic review may not increase diversity (Boske et al., 2018; Cahn, 2015). Furthermore, it is unknown how admissions committees evaluate criteria across applicants.

Health professions. A nationwide survey of health professions schools in nursing, medicine, dentistry, public health, and pharmacy found that nearly 50% of nursing schools and over 75% of the remaining schools had implemented holistic review (Glazer et al., 2014). Like speech-language pathology, individual schools varied in what holistic review practices they adopted (Glazer et al., 2014). Schools—particularly those which had adopted many holistic review practices—reported that implementing holistic review increased diversity and had the same or improved student outcomes (Glazer et al., 2014). However, individual practices alone, such as eliminating the GREs as an admissions requirement, may be insufficient for increasing diversity (Cahn, 2015). Importantly, schools implementing holistic review reported 2–3 times more student engagement in the community, teamwork and cooperation, and openness to different perspectives than schools that did not (Glazer et al., 2014).

Conceptualizations of Holistic Review

Although AAMC (2013) has provided an established definition of holistic review, other conceptualizations of holistic review have emerged that are also relevant to understanding holistic review in practice. For example, as shown in Figure 1, different interpretations are whole file, whole person, and whole context (Bastedo et al., 2018). Although Bastedo et al. (2018) developed this framework to study holistic review practices of undergraduate admissions officers, it is well suited to exploring graduate admissions in speech-language pathology.

Figure 1.

Potential holistic review process. GPA = grade point average; GREs = graduate record examinations.

Whole file. Under this approach, committee members consider the application file. Although committee members may consider both academic variables (e.g., GPA and GRE scores) and nonacademic variables (e.g., extracurricular activities), how these factors inform admissions decisions depend on individual committee members. Because evaluation of the application file at face value does not necessarily include consideration of personal characteristics or academic or family background, committee members may miss relevant information to contextualize the application file materials of CLD applicants.

Whole person. In addition to the application file, committee members consider the applicant as a unique individual with achievements in terms of involvement, leadership, background, and what they will contribute to their cohort and program (i.e., applicant fit). This approach may pose barriers to CLD applicants because it does not take into account the context, environment, or lived experiences of applicants. Academic and family background afford individuals with different opportunities, such that some experiences may be a function of privilege and not ability. Furthermore, it is unclear how committee members perceive applicant fit for those who are not of the dominant majority in communication sciences and disorders (CSD).

Whole context. In addition to whole-person characteristics, this approach takes into account the context, environment, and lived experiences of applicants. Committee members consider the applicant as a unique individual from educational and family circumstances that shaped who they are. This approach is the most inclusive yet requires the most interpretation. If committee members are skilled at interpreting application materials beyond what is explicitly stated or present, such that they understand the potential barriers that an applicant faced in their pursuit of a master's program in speech-language pathology, they may credit them for their accomplishments. If committee members are less skilled, they may underestimate their abilities.

Admissions Application Materials

As shown in Figure 1, common application materials to master's programs are GPA, GRE scores, letters of recommendation, a personal statement, and a resume or curriculum vitae (CV; Baggs et al., 2015; Kent & McCarthy, 2016; Michel et al., 2019; Okahana et al., 2018). These components vary in their predictive value of graduate school outcomes, and interpretation of them may be subject to bias.

GPA

Predictive value. The predictive value of GPA in identifying qualified candidates is uncertain, despite an emphasis on these metrics in the reviewal of applicants to master's programs in speech-language pathology (Guiberson & Vigil, 2021). Some studies have found undergraduate GPA in speech-language pathology (Baggs et al., 2015; Boles, 2018; Halberstam & Redstone, 2005; Ryan et al., 1998) and undergraduate GPA (Forrest & Naremore, 1998; Halberstam & Redstone, 2005; Troche & Towson, 2018) to be predictive of graduate school outcomes, as defined by Praxis outcomes, graduate GPA, and comprehensive exam performance. Elsewhere, undergraduate GPA and GPA in speech-language pathology have not predicted graduate school outcomes (Anderson et al., 2017; Richardson et al., 2020).

Potential for bias. Utilizing GPA as an indicator of applicant quality may give rise to bias. Despite emphasis on a near-perfect GPA, the GPA of admitted students to speech-language pathology master's programs is variable (Polovoy, 2014; Sylvan et al., 2020), in terms of average GPA of accepted students to U.S. master's programs (range: 3.14–3.97; CAPCSD & ASHA, 2020; Koay et al., 2016) and international grading systems (Michel et al., 2019).

Findings beyond speech-language pathology also suggest there are issues with using GPA as an indicator, including heavy reliance on GPA in screening and final admissions decisions (Kent & McCarthy, 2016). In psychology, Black/African American and Hispanic/Latinx students are underrepresented in doctoral programs relative to their White peers, despite having the same GPA (Callahan et al., 2018). Such underrepresentation is not limited to graduate programs. Black/African American and Hispanic/Latinx students who had guaranteed admission to top undergraduate institutions by earning a top GPA were less likely than their White peers to apply and more likely to apply to lower ranked schools (Black et al., 2015). When there are racial/ethnic differences in GPA, they may be due to educational resource availability (Michel et al., 2019) and instructor mindset (Canning et al., 2019; Gershenson & Papageorge, 2018). Across all students at a large public university, Black/African American, Hispanic/Latinx, and Native American students had lower grades in science, technology, engineering and mathematics courses than White or Asian students; this gap was twice as large in classes where faculty believed academic ability was a fixed ability (Canning et al., 2019).

GRE Scores

Predictive value. GRE scores have limited predictive value of graduate outcomes. In speech-language pathology, some studies have found GRE scores to be predictive of Praxis outcomes, graduate GPA, and comprehensive exam performance (Anderson et al., 2017; Baggs et al., 2015; Boles, 2018; Kjelgaard & Guarino, 2012; Ryan et al., 1998; Troche & Towson, 2018). Elsewhere, GRE scores have not predicted graduate outcomes (Anderson et al., 2017; Richardson et al., 2020). Despite this mixed evidence, over a quarter of 110 speech-language pathology faculty reported their programs use a GRE cutoff and rated GRE scores as an important or the most important in both screening decisions and application selection (Guiberson & Vigil, 2021). In the adjacent field of psychology, analysis of doctoral student enrollment revealed that Black and Hispanic/Latinx students were underrepresented relative to White students, despite having equally high GRE scores (Callahan et al., 2018). More broadly, as per C. Miller and Stassun (2014), the GRE is a more accurate indicator of skin color and sex than of ability and long-term success.

Potential for bias. Across all GRE takers, American Indian, Black, and Hispanic/Latinx examinees have performed lower than White and Asian students (Bleske-Rechek & Browne, 2014; Educational Testing Service [ETS], 2019). Of all GRE takers between July 2018 and June 2019, those who were Asian scored higher on GRE Quantitative than all other racial/ethnic groups, those who were White and non-Hispanic scored higher on GRE Verbal than all other groups, and those who were White and non-Hispanic or Asian scored higher on analytical writing than all other groups (ETS, 2019).

This finding has several implications. The first involves how admissions committees interpret test scores (Messick, 1989). Although GRE scores are clearly not lower for every racial/ethnic minority applicant, structural racism systematically denies underrepresented minority groups of low SES access to resources (e.g., educational opportunity; Kendi, 2020). Thus, if minorities have lower GRE scores, it may be due to structural racism in terms of economic success, educational opportunity, and bias in the educational environment (Lucey & Saguil, 2020). The second implication involves how admissions committees use GRE scores (Messick, 1989). If programs use GRE scores as a singular benchmark instead of integrating multiple sources of information, they risk misusing the test (ETS, 2019) and effecting racialized outcomes, in that admissions may not mention race but systematically exclude racial/ethnic groups (Powell, 2012). Indeed, when GRE scores were used as a cutoff in the biomedical sciences, nearly two thirds of Black/African American, Native, and Hispanic/Latinx applicants were triaged, but only 26% of White male applicants were triaged (Wilson et al., 2019). In all, interpretation and use of the GRE must account for inequity.

Personal Statements

Predictive value. Personal statements may not reliably reflect the abilities of students. When measured using idea density, the quality of personal statements did not predict graduate GPA or comprehensive exam outcomes (Anderson et al., 2017). However, when evaluating personal statements using grammar, content, and apparent knowledge of and commitment to the field of speech-language pathology, statement quality has correlated with graduate GPA (Halberstam & Redstone, 2005). These last two criteria are subjective (Halberstam & Redstone, 2005). Furthermore, assessing grammar as an indicator of personal statement quality may give rise to linguistic bias (Politzer-Ahles et al., 2020).

Potential for bias. Some of the criteria that admissions committee members evaluate through personal statements, such as apparent knowledge of and commitment to speech-language pathology, depend on personal judgment (Halberstam & Redstone, 2005). CLD applicants to master's programs in speech-language pathology may be at a disadvantage relative to their White peers in developing personal statements, especially with respect to writing skills (Fuse, 2018). Findings from medicine support this possibility. Nearly half the students across three cohorts reported receiving help from others in developing their personal statement for medical school (Albanese et al., 2003). CLD applicants may not have the same access to help developing a personal statement as do their White peers, who are likely to know previous applicants willing to share materials or to have the financial wherewithal to access paid services (Albanese et al., 2003). Hence, the personal statements of CLD applicants could vary in their quality because of differences in resource availability and not ability (Kendi, 2020).

Letters of Recommendation

Predictive value. Letters of recommendation may have limited predictive utility in admissions. On one hand, letter of recommendation quality—as measured by recommender prestige, apparent depth of knowledge of the applicant, reasons for recommending the applicant, and level of enthusiasm of recommendation—has predicted graduate GPA in speech-language pathology (Halberstam & Redstone, 2005). However, a large-scale meta-analysis found that letters only weakly predicted graduate GPA, performance rating from faculty, and degree attainment (Kuncel et al., 2014). Furthermore, letters of recommendation only accounted for a negligible proportion of unique variance in graduate GPA and faculty ratings of graduate school performance (Kuncel et al., 2014). Although not specific to CSD, these findings suggest letters of recommendation have limited power in identifying qualified applicants to graduate programs.

Potential for bias. Using letters of recommendation to select applicants may introduce bias. First, some applicants may be at a disadvantage in obtaining the experiences requisite for strong letters of recommendation. Amid general student concerns about having to work outside school and obtaining letters of recommendation, Black/African American and Hispanic/Latinx master's students in speech-language pathology have reported being of lower SES than their White peers (Fuse, 2018; Fuse & Bergen, 2018). Thus, if CLD applicants of lower SES must work outside school, they have less time for coursework, research, and extracurricular activities, all of which may help them develop relationships with letter writers and demonstrate the characteristics for a strong letter (Fuse, 2018).

Second, some applicants may be at a disadvantage in receiving strong letters of recommendation, even when they are as equally qualified as peers of dominant backgrounds. Of all undergraduate students applying to a research experience program, minority applicants versus White applicants and applicants from institutions that were not research intensive versus research-intensive institutions received different letters of recommendation, despite having the same GPA (Houser & Lemmons, 2018). Although letters for White students tended to describe them in terms of cognitive ability, productivity, and insight, letters for Black/African American and Hispanic/Latinx students tended to describe them in terms of affect and emotion (Houser & Lemmons, 2018). Furthermore, graduate programs nationwide have reported knowing the author of letters of recommendation as an influence on the admissions process (Okahana et al., 2018). In all, these findings warrant caution in using letters of recommendation to assess applicant quality.

Resumes or CVs

Predictive value. The resume or CV includes many of the other components: GPA, GRE scores, accomplishments, and experiences that are referred to in a personal statement and letters of recommendation. Because the resume or CV is essentially an organized listing of a subset of what is in other application materials, then the criticisms about the predictive utility of other application materials apply here. For example, research experience on the resume or CV is also probably mentioned in the personal statement, and undergraduate institution and performance are probably also mentioned in recommendation letters.

Potential for bias. Interpreting the resume or CV without considering the full array of factors that shaped the applicant may lead to lower ratings for CLD applicants, particularly those who are from less privileged backgrounds (Bastedo et al., 2018). Master's students in speech-language pathology have reported feeling overwhelmed by the application process, such that application materials may not fully align to program expectations (Sylvan et al., 2020). At the same time, many applicants to the health professions report receiving external help in preparing their applications (Albanese et al., 2003). Hence, interpreting resumes or CVs at face value may affect bias against applicants without access to outside help (Albanese et al., 2003).

Summary

Altogether, previous findings on application materials highlight the importance of evidence-based holistic review. There is no singular set of reliable predictors of applicant quality. An additional concern is that previous studies did not include rejected applicants, which limits the ability to predict later outcomes; thus, the predictive value of application materials may be even lower than what it appears (Michel et al., 2019; Ryan et al., 1998). Given underrepresentation in speech-language pathology, confounds relevant to cultural and linguistic diversity may exist in the prediction of graduate outcomes, as such evidence informs admissions committees on what to consider. This problem is circular in nature; if programs fail to diversify, it is impossible to know what predicts graduate success across diverse backgrounds.

Holistic Review Criteria

As shown in Figure 1, admissions committees evaluate application materials for personal characteristics or criteria. However, the evaluation of criteria may be subject to bias, such that applicants who do not fit the stereotype of a speech-language pathology student may face additional obstacles in entering the profession (Rogus-Pulia et al., 2018; Shapiro et al., 2002).

A Framework for Noncognitive Variables in Holistic Review

A framework for noncognitive variables in holistic review for all students comes from Sedlacek (1993), who argued for the importance of noncognitive variables in holistic review. Noncognitive variables, which are qualitative metrics indicative of personal characteristics, entail experiential and contextual factors “relating to adjustment, motivation, and student perceptions” (Sedlacek, 2011, p. 180). Importantly, these variables may best predict success in nontraditional students: (a) positive self-concept, (b) realistic self-appraisal, (c) ability to successfully handle a system that was not designed for them (i.e., graduate admissions), (d) preference for long-term goals over short-term ones, (e) availability of a strong support person, (f) successful leadership experience, (g) demonstrated community service, and (h) knowledge acquired in or about a field (Sedlacek, 2004). These variables are an indicator for success in higher education for all students and must be considered in order to truly generate diverse and socially just admissions decisions (Sedlacek, 1993, 2004, 2005, 2011). For reference, programs in the Council for Graduate Studies most commonly identified past academic performance, critical thinking, program fit, and writing ability as qualities relevant to master's admissions (Kent & McCarthy, 2016).

An Instantiated Example of Holistic Review Criteria in Speech-Language Pathology

An example of holistic review criteria comes from the University of Kansas Intercampus Program in Communicative Disorders (University of Kansas, Department of Hearing and Speech, 2018). The admissions committee evaluates applicants for criteria, which are social constructs whose evaluation depends on the indicators used and the interpretation of admissions committee members (Boske et al., 2018). Although a detailed analysis is beyond the present scope, it is not always clear how the criteria below align to the noncognitive variables as proposed by Sedlacek (1993). For example, the criteria do not mention or allude to an applicant's ability to handle a system that may not be designed for them.

Academic ability and preparation. Academic ability and preparation refer to the need to have a firm foundation in speech-language-hearing and broader knowledge of related areas, with the goal of being able to apply this knowledge in clinical practice. Indicators of this criterion include overall and GPA in speech-language-hearing, letters of reference, and resume.

Communication skills. Communication skills refer to the need of SLPs to communicate with clients, families, and other professionals using oral and written language. Indicators of this criterion include personal statement, letters of recommendation, and resume.

Interpersonal skills. Interpersonal skills refer to the need of SLPs to work collaboratively and effectively with clients, families, and other professionals. Indicators of this criterion include teamwork experience and clinical experience on the resume.

Analytical skills. Analytical skills refer to the need of SLPs to critically read, analyze, interpret, and apply research to evidence-based clinical practice, thus requiring a foundation in research, critical thinking, and clinical application. Indicators of this criterion include an essay, research, and clinical experience on the resume and letters of reference.

Potential for professionalism. Potential for professionalism refers to the need for SLPs to be organized, reliable, respectful, and able to grow from constructive feedback. Indicators of this criterion include letters of reference and personal statement.

Potential for leadership. Potential for leadership refers to the need of SLPs to advocate for their clients and for the profession. Indicators of this criterion include leadership experience on the resume and letters of reference.

Cultural and linguistic diversity. Cultural and linguistic diversity refers to the need for SLPs to work effectively with diverse and multilingual clients from a variety of backgrounds that differ from their own. Indicators of this criterion include personal or academic cultural experiences on the resume and letters of reference.

Indicators of Criteria

In addition to the application materials, indicators of criteria from the example are teamwork experience, clinical experience, and research experience. As with the application materials, these indicators may have limited predictive ability and potential for bias.

Teamwork experience. Teamwork may not reliably indicate applicant quality because effective teamwork may be something that CLD applicants do not highlight in their application materials as an individual accomplishment. For example, Native American academics from tribal communities have reported a gap between their cultural norms and those of predominantly White academia (Dvorakova, 2019). Although relationality and communal cooperation were central to their respective cultures, academia emphasized individualism (Dvorakova, 2019). Similarly, Korean undergraduates reported a greater sense of “oneness” with members of a whole (e.g., family and friend networks), whereas their White peers reported a greater sense of individualism (Lim et al., 2011). Thus, sense of self—and of one's strengths, including teamwork as a type of accomplishment or skill—are culturally situated; CLD applicants may not consider positive teamwork experiences as an individualistic skill to explicitly mention.

Students may also face inequity in gaining teamwork experience. Undergraduate students in CSD have reported relying on cohort mates for social support (Roos & Schreck, 2019). Furthermore, younger SLPs who recently graduated from master's programs have demonstrated significant bias against speakers with they perceive to have a “nonnative” accent (Chakraborty et al., 2019). Together with the potential for homophily, one possibility is that CLD students are less able to access social support from their peers in a predominantly White profession (Rogus-Pulia et al., 2018). Those with intersecting identities in multiple marginalized groups (i.e., racial/ethnic minority plus being perceived as a nonnative speaker of English) may face more barriers (Crenshaw, 1989).

Research experience. Prior research experience may not predict academic performance, degree attainment, and clinical performance in the health sciences and professions (A. Miller et al., 2020). Research experience is oftentimes unpaid, such that it may be accessible only to those who can afford to provide unpaid labor (A. Miller et al., 2020). Consequently, using research experience as an indicator of analytical skills may reflect access to opportunity and disadvantage CLD applicants (Houser & Lemmons, 2018; A. Miller et al., 2020). In speech-language pathology, family financial support is predictive of admissions outcomes to master's programs. Therefore, students who work outside school (who are disproportionately minority students) may be less likely to have research experience and appear to have less strong analytical skills due to inequity (Fuse, 2018; Fuse & Bergen, 2018).

In addition, undergraduate research experience may only be available at some schools (Houser & Lemmons, 2018; A. Miller et al., 2020). Even when paid research opportunities are available, students from institutions that are not research intensive and community colleges have been underrepresented in the applicant pool compared to their peers from research-intensive institutions, with 40% of 389 students applying versus 70% expected (Houser & Lemmons, 2018). Thus, using prior research experience as an indicator may favor applicants at institutions with research opportunities (A. Miller et al., 2020). By the same token, committees may perceive applicants to have weaker analytical skills, simply because research opportunities were unavailable at their institutions.

Clinical experience. Little is known about prior clinical experience as a reliable predictor of graduate outcomes in speech-language pathology. Findings from the allied health professions suggest clinical experience may not reliably predict graduate success. In nursing, prior clinical experience did not predict graduate GPA (El-Banna et al., 2015; Patzer et al., 2017) or program completion (Niemczyk et al., 2018). Similarly, in medicine, prior clinical experience did not predict medical school GPA, medical licensing exam outcomes, or later assessment of expertise and professionalism (Artino et al., 2012; in contrast, see Shah et al., 2018). In all, these findings highlight the importance of caution in using clinical experience as an indicator.

Using clinical experience as an indicator also gives rise to potential bias. As with research experience, undergraduate clinical experience, such as internships, is often unpaid. Therefore, the same concerns with accessibility of research experience also apply to clinical experience. In addition, CLD students may face more hurdles than their White peers in clinical settings. For example, minority supervisors in psychology have reported spending the most time discussing multicultural issues if their supervisee was a minority; in contrast, White supervisors spent the least amount of time discussing multicultural issues if their supervisee was White and more time if their supervisee was a minority (Hird et al., 2004). One conclusion is that minorities must navigate multicultural issues as an everyday reality, thus adding to the burden of gaining clinical experience (Hird et al., 2004). Explicitly, minorities may be perceived as having more difficulties in clinical settings, even though the underlying issue is underrepresentation, such that clinical supervision methods and perceptions of clinical competency are based on the dominant majority alone (Buchanan & Wiklund, 2020). Altogether, these findings highlight one way in which clinical experience may create bias.

Summary

In all, teamwork experience, clinical experience, and research experience may have limited utility in admissions if they are used as gatekeeping mechanisms and considered in the absence of applicant background. As Figure 1 shows, there are many steps in the application process, such that application materials may or may not truly reflect applicant characteristics and criteria ratings may or may not correspond to explicit admissions decisions. How do admissions committees interpret applicants of diverse backgrounds using holistic review criteria? In the section that follows, we propose a methodology for exploring this question.

Vignettes as a Research Methodology

Vignettes or information in a narrative paragraph format is an emerging method for probing real-world decision making. Medicine, speech-language pathology, academic reviewing, and admissions have used vignettes to examine the decision-making practices of gatekeepers as related to quality of care (i.e., the series of decisions that lead to improved outcomes) and evaluation in higher education contexts. We suggest that the admissions decisions brought about by holistic review are akin to quality of care, in that they may to lead to improved program outcomes (AAMC, 2010, 2014).

Quality of Care

In Peabody et al.'s (2000) study, physicians read eight vignettes and made decisions, with the outcome being quality of care. The study manipulated vignettes for clinical symptomatology and presented them in a simple or complex clinical scenario. In measuring quality of care or the “goodness” of decision making versus patient outcomes, this study removed the potential confounds of characteristics beyond the control of individual practitioners (e.g., underlying conditions) to isolate the role of practitioner knowledge. Importantly, findings showed that quality of care as measured by the vignettes was closer to the quality of care as measured by standardized patients (i.e., the gold standard in medicine) than chart abstraction (i.e., a report of diagnostic information).

Selin et al. (2019) expanded upon this methodology to explore quality of care in the context of SLP clinical decision-making practices for children with specific language impairment (SLI). As in Peabody et al. (2000), the study manipulated clinical symptomatology across vignettes and removed confounds of characteristics beyond the control of individual SLPs (e.g., workplace policies for eligibility) by instructing respondents to use only best professional judgment and to assume neutral workplace conditions. Although all children in the vignettes had SLI, their characteristics were specified at impaired, borderline, or typical levels or not specified. This structure allowed for the examination of both child and SLP characteristics. Findings revealed SLPs identified children with SLI for services at higher rates than reported in the literature, thus indicating a higher quality of care than in actuality. In all, vignettes may be an effective method for understanding the role of individuals in decision-making practices.

Evaluation in Higher Education Contexts

Politzer-Ahles et al. (2020) used vignettes to explore how faculty and PhD students in CSD evaluate academic writing. The study manipulated conference abstracts to vary along one parameter: whether they conformed or not to international academic English. Respondents rated the vignettes using criteria, such as scientific quality and clarity of writing. Results showed that the abstract written in language that conformed less to international academic English received lower ratings of scientific quality than the abstract written in language that conformed to international academic English, despite having identical substantive content. Hence, vignettes may be useful for evaluating how those in CSD interpret and evaluate criteria across diverse contexts.

Turning to admissions, Bastedo et al. (2018) used vignettes to explore the decision-making practices of undergraduate admissions officers. Respondents made admissions decisions using full hypothetical admissions files for applicants who were of the same race, ethnicity, gender, college, and major but varied in their coursework, educational background, and academic metrics (i.e., grades and test scores). Two came from an upper middle-class high school with a strong or less strong academic background. The third came from a lower SES high school with the least strong academic background and had fewer opportunities in their academic environment. Respondents made decisions under one of two conditions: limited information or detailed information to contextualize their performance. Findings revealed that providing context on applicant background resulted in a higher admissions rate and that respondents who considered not only academic performance but also personal characteristics and applicant background were more likely to admit the applicant from the low-SES background. Thus, utilizing hypothetical profiles of applicants as vignettes may be informative for understanding evaluation of applicants using holistic review criteria.

This Study

Taking together what is known about holistic review and vignettes as a methodology, this study explored the evaluation of applicants along criteria used during holistic review. To isolate the role of individual interpretation in evaluation, respondents completed vignette items under neutral conditions (i.e., using only best professional judgment, assuming the application was complete, and evaluating the applicant as is). Thus, the research questions were as follows:

1. Considering criteria used during holistic review, are applicants from CLD backgrounds less likely to be accepted into master's speech-language pathology programs than their peers from dominant backgrounds?

2. Do applicant ratings predict admissions decisions?

Method

Ethics

The institutional review board at The Hong Kong Polytechnic University approved this study. Methods for the experiment were preregistered at https://osf.io/5ygzw. We report any analyses that deviate from the primary analyses of the preregistration as exploratory.

Sampling Procedure

To recruit a broadly representative respondent base, recruitment included posting information inviting study participation online in national professional groups: ASHA Students to Empowered Professionals Board; ASHA Special Interest Groups 1 (Language Learning & Education), 10 (Academic Affairs), and 14 (Multiculturalism) discussion boards; and social media groups, such as Clinical Research for SLPs on Facebook. Data collection took place online from mid-July 2020 to mid-September 2020 on Qualtrics (http://www.qualtrics.com). Respondents elected whether to participate by reading an information statement, indicating consent, and completing the survey with the ability to stop and return to it over a 2-week period. There was no compensation for participation.

The target sample size was 100–200 participants. This sample size was based on previous research in speech-language-hearing (Selin et al., 2019) using similar methodologies. The stopping rule was to collect data until the survey had 100 completed observations. If data collection yielded over 10 responses per week, the survey would stay open until 200 responses were collected. However, if data collection yielded less than 10 responses per week across a 2-week period, the survey would close. Here, responses decreased to six responses in the penultimate week of data collection and one response in the ultimate week of data collection. Given that data collection took place during the COVID-19 pandemic and participants did not receive compensation, the authors determined reaching the target sample size was unlikely and ceased data collection when the survey had 66 responses. Of those 66 participants, 53 completed the first block (demographics), and 35 completed the survey. Inspection of the data did not suggest that any particular variables influenced attrition.

Participant Characteristics

To participate in this study, respondents had to be a faculty member, PhD student, or PhD candidate at an accredited program for speech-language pathology or equivalent (e.g., CSD) in the United States. The study included PhD students and candidates because they are likely to become faculty and serve on admissions committees. There were no restrictions based on demographic characteristics. As shown in Table 1, participant demographics were consistent with ASHA demographics. Participants were mostly Caucasian, non-Hispanic, and women. About half held a research doctoral degree, and about half held a master's degree. There was diversity in current positions, with the most common being a PhD student or candidate, an associate professor or equivalent, and an assistant professor or equivalent. Over two thirds served on a master's admissions board.

Table 1.

Respondent characteristics.

Characteristic	n	%
Race
American Indian, Eskimo, or Aleut	0	0
Asian or Pacific Islander	3	9
Black or African American	3	9
Caucasian or White	30	86
Ethnicity
Hispanic	2	6
Gender
Men	4	11
Women	31	89
Education level
AuD or equivalent	1	3
Bachelor's or equivalent	0	0
Master's or equivalent	16	46
PhD or equivalent	17	49
SLPD or equivalent	1	3
Current position
Assistant professor or equivalent	6	17
Associate professor or equivalent	7	20
Clinical professor or equivalent	4	11
Full professor or equivalent	5	14
Lecturer or equivalent	2	6
PhD student or candidate	11	31
Serving on admissions committees
Master's admissions	24	69
Other	8	23

Note. Current positions add up to more than 100%, because one person was both an assistant professor and PhD student. AuD = Doctor of Audiology; SLPD = Doctor of Speech-Language Pathology.

Instrument

The authors developed and piloted the survey with PhD candidates and faculty in speech-language pathology. Pilot testers provided feedback that informed survey revision, with key considerations being survey length and providing definitions for holistic admissions criteria. Participants completed an online survey (see Appendix) implemented in Qualtrics. To respect privacy, respondents did not provide institution-specific information.

In the main portion of the survey, participants read six vignettes describing hypothetical applicants. Prior to the vignettes, the survey instructed respondents to use their best professional judgment to evaluate each applicant as is under the assumption that each applicant had a complete application and that there was no applicant interview or other available information. The survey included definitions of each criterion for use in applicant ratings from the University of Kansas Intercampus Program in Communicative Disorders (University of Kansas, Department of Hearing and Speech, 2018). After reading this information, the survey presented six vignettes.

To avoid bias, the vignettes used initials and did not specify gender, race, or ethnicity (Bertrand & Mullainathan, 2004; Milkman et al., 2015; Simonsohn, 2015). Furthermore, to control for order effects, respondents read and rated vignettes in a randomized order. As shown in Table 2, indicators of seven criteria from a holistic review framework were conceptualized at a low, moderate, or high level. GPA scores for each level were based on findings about GPA for master's programs in CSD from the research literature (Koay et al., 2016; Polovoy, 2014; Sylvan et al., 2020). As shown in Table 3, the authors systematically manipulated indicators of these criteria across vignettes, such that applicants varied by level and specification (i.e., specified or not specified, meaning that information was not provided). Not specifying information allowed for the opportunity to examine default judgments.

Table 2.

Conceptualization of indicator levels of criteria.

Criterion	Low	Moderate	High
Academic ability and preparation	Evidence from one area (responsibilities or work during undergraduate, double major, research experience, or clinical experience) or GPA = 0.0–3.4	Evidence from some areas or GPA = 3.5–3.7	Evidence from all areas and high GPA = 3.8–4.0
Communication	Positive communication in one area (class, clinic, or workplace), with one communication partner (faculty, supervisor, or peers), in one modality (oral or written)	Positive communication in some areas, with some communication partners, in some modalities	Positive communication in all areas, with all communication partners, in all modalities
Interpersonal skills	Worked well with one person from one group (families, individuals with disabilities, peers, or other professionals) in one context (work, clinic, or classroom)	Worked well with some people, from some groups, in some contexts	Worked well with all people from all groups in all contexts
Analytical skills	Evidence of minimal analytical skills from one area (research, critical thinking, or clinical application)	Evidence of some analytical skills from some areas or mixed evidence across areas (e.g., high in some, low in others)	Evidence of high analytical skills in all areas
Professionalism	Minimal evidence from one area (organization, reliability, respectfulness, or response to constructive feedback)	Some evidence from some areas or mixed evidence across areas (e.g., high in some, low in others)	Evidence of professionalism in all areas
Leadership	Evidence of leadership experience or leadership qualities in one context (research, clinic, class, or organizational experience)	Evidence of leadership experience in some contexts or mixed evidence of leadership (e.g., high in some, low in others)	Evidence of leadership in all contexts
Cultural and linguistic diversity	Evidence of minimal previous work with people from backgrounds different from their own in one area (personal or academic)	Evidence of semiconsistent previous work with people from backgrounds different from their own in some areas	Evidence of extensive previous work with people from backgrounds different from their own in all areas

Note. Although the process of evaluating criteria and determination of ratings is subjective, this framework offers one way of conceptualizing the levels of indicators that reviewers use to inform their evaluation of applicants.

Table 3.

Vignette design of applicants to master's programs in speech-language pathology.

Criteria	A.B.	B.C.	C.D.	D.E.	E.F.	F.G.
Academic ability and preparation	High	Moderate	Not specified	Moderate	Low	High
Communication	Low	Low	Not specified	Moderate	Low	High
Interpersonal skills	Low	Moderate	Moderate	High	Not specified	High
Analytical skills	High	Low	Not specified	Moderate	Moderate	High
Professionalism	Moderate	Moderate	Moderate	High	Not specified	High
Leadership	Low	Moderate	Moderate	Not specified	Low	High
CLD	Low	Moderate	High	High	High	Low

Note. High = criterion specified and indicators of this criterion were highly positive; Moderate = criterion specified and indicators of this criterion were moderately positive or ambiguous; Not specified = information not included in the vignette; Low = criterion specified and indicators of this criterion were minimally positive; CLD = cultural and linguistic diversity.

For example, applicant A.B. graduated from a private college with a 3.8 GPA. Their resume showed that they worked as a teaching assistant for one semester and had an internship with the general counsel of General Electric Corporate. Thus, indicators of their academic ability and preparation (i.e., GPA and previous professional experience) were high. Furthermore, in their essay, A.B. wrote about growing up in an ethnic enclave. Because they did not specify whether this experience entailed working with people from backgrounds different from their own, indicators of their cultural and linguistic diversity were low. Their letters of recommendation came from three professors, one of whom supervised the student in their work as a teaching assistant. The professors reported that the student wrote well for assignments and produced comprehensive reports and client plans in clinic. Therefore, indicators of their analytical skills (i.e., academic writing) were high. However, the professor also reported they rarely participated in group work in class. Thus, indicators of their interpersonal skills (i.e., teamwork) were low. Furthermore, the supervising professor wrote that the student could be hard to reach and received below-average student ratings, indicating their communication skills and potential for leadership were low. Given the mixed findings (i.e., produced comprehensive plans, worked as a teaching assistant, and difficult to reach), indicators of their potential for professionalism were moderate.

After reading each vignette, respondents rated the applicant on criteria using a 5-point Likert scale from weak to very strong: (a) academic ability and preparation, (b) communication skills, (c) interpersonal skills, (d) analytical skills, (e) potential for professionalism, (f) potential for leadership, and (g) cultural and linguistic diversity. Respondents also selected an explicit admissions decision as admit, waitlist, or reject. To maximize the likelihood of capturing first impressions, respondents could not return to previous vignettes and post hoc change answers.

In addition to the vignettes, participants also answered questions about their own demographic background and professional background. Demographic items included race and ethnicity using categories from the National Institutes of Health, as well as gender. Professional background items included education level, current position, current experience on a master's admissions board for speech-language pathology or related programs, current experience on other admissions boards for speech-language pathology or related programs, and factors in applicant reviewal. Respondents could select multiple options from a list of factors in applicant reviewal. To prevent bias in responses, as the survey never explicitly stated the questions under review, professional background items (c)–(e) came after the vignettes. The demographic items and professional background items (a) and (b) came before the vignettes.

Measures

The key dependent variable was whether or not the applicant is accepted (i.e., whether they are in the top 50% of applicants) by a given respondent. This was calculated on a per-respondent basis. In other words, for each respondent, the 5-point Likert scale ratings across seven criteria were averaged into one number for each applicant, and then within that respondent, the six applicants were ranked. The top three applicants were considered “accepted” by that respondent, and the bottom three as “not accepted.” Thus, each applicant–respondent pair has an “accept” or “not accept” decision. This cutoff was determined by information from an actual accredited program in speech-language pathology, which accepts the top 40%–50% of applicants. Because it is a highly ranked program, this study adopted a 50% cutoff.

Analytic Strategy

Incomplete surveys were excluded from analysis. To compare the likelihood of acceptance for the applicant from the dominant background to that of the applicants from other backgrounds, the analytic plan was to dummy code applicants (with “0” for students from nondominant backgrounds and “1” for the student from a dominant background) and regress acceptance on applicant background using the following generalized (logistic) mixed-effects model: glmer(Acceptance ~ 0 + Background|Rater), data, family = “binomial.” The random effects in this model fit a different effect of Background (i.e., difference between the dominant background applicant and the others) for each rater but do not fit different intercepts for each rater. Secondary analyses included comparison of the likelihood of acceptance for the applicant from the dominant background to that for each other applicant. Exploratory analyses included descriptive analysis of likelihood of acceptance between respondent groups, which were determined by self-reported consideration of factors in admissions. Following Bastedo et al. (2018), respondents who selected “application file,” “unique characteristics,” “family background,” and “educational background” were coded as “whole context,” and those who did not were coded as “not whole context.” In addition, exploratory analyses also included descriptive analysis of likelihood of acceptance by criteria.

Results

We report preliminary findings of how respondents, or faculty and doctoral students in CSD, ranked and made explicit admissions decisions (i.e., admit, waitlist, and reject) for hypothetical applicants presented in vignettes. Again, respondents rated hypothetical applicants along seven criteria used during holistic review from an actual program.

Applicants of Varying Indicator Levels for Criteria Were Lower Ranked

Figure 2 shows, for each applicant, the proportion of respondents who ranked this applicant among their top three, alongside the proportion of respondents who gave the applicant an explicit “accept” decision. Applicant F.G. was far more likely to be accepted than the others. In fact, every rater ranked this applicant among their top three. This situation rendered our planned statistical analysis moot, as logistic regression is not possible when one condition has 100% of one kind of response, since the logit function is undefined for proportions of 0% or 100%. Nevertheless, the results support the conclusion that the applicant from the stereotypically “successful” background, who had a high indicator level for all criteria except for cultural and linguistic diversity, was more likely to be accepted than the applicants of varying backgrounds—who also had moderate or high indicator levels for cultural and linguistic diversity. Although our results do not prove cultural and linguistic diversity influenced the likelihood of acceptance across vignettes, they do suggest that likelihood of acceptance varied by it. This is because CLD background is confounded with other factors in this data set. Although applicant F.G. had the highest GPA, which is an important criterion in admissions decisions for master's programs in speech-language pathology (Guiberson & Vigil, 2021), and high indicator levels for other criteria, all other applicants varied significantly more in their indicator levels (see Table 2 for details).

Figure 2.

Likelihood of acceptance when acceptance is based on scores from holistic review criteria ratings (dark bars) versus when acceptance is based on explicit decisions (light bars).

Holistic Review Approaches and Criteria

As an additional exploration, we examined whether respondents who reported using a whole-context approach (i.e., considered applicant file, personal characteristics, and educational and family background) yielded different admission outcomes (as a function of ratings that translated into individual rankings) than respondents who reported using a whole-file (i.e., considered applicant file) or whole-person (i.e., considered applicant file plus personal characteristics) approach. As shown in Figure 3, the three candidates who generally received lower rankings (A.B., B.C., and C.D.) were slightly more likely to be accepted by whole-context raters than by raters who did not take a whole-context approach. This effect was more pronounced in candidate E.F. Recall from Figure 2 that candidate E.F. received a fairly good ranking (and thus high likelihood of acceptance when acceptance was determined by ranking) but did not receive many explicit “accept” decisions. It appears that high rankings for E.F. were especially driven by raters who embodied a whole-context approach. The only candidate who received a worse ranking from whole-context raters than other raters was D.E.

Figure 3.

Each candidate's likelihood of acceptance by raters who did not take a whole-context approach (dark bars) versus by raters who did take a whole-context approach (light bars).

Finally, we examined the relationship between the other properties of the candidate described in each vignette (see Table 2) and the candidate's likelihood of acceptance. It was not possible to analyze these data with regression, given the abovementioned problem (i.e., cells with 100% or 0% acceptance), the small amount of data overall, and the repeated-measures nature of the data (which preclude using a simple logistic regression and necessitate a mixed-effects logistic regression, which is difficult to get to converge without a large amount of data in each cell). Without regression, it is impossible to attribute increases or decreases in acceptance likelihood to any particular factor, since many of these factors are confounded. Nevertheless, some tentative trends can be noted from Figure 4. This figure shows, for each factor, how likely applicants were to be accepted as a function of how much of that factor they had. For example, the solid red line for “academic ability” shows that applicants whose academic ability was not specified (i.e., not described) in their vignette had a very low probability of being accepted. In contrast, applicants whose indicators of academic ability were “low” or “high” had about a 60% chance of being accepted, and applicants whose indicators of academic ability were “medium” had about a 40% chance of being accepted. It is shown from the figure that indicator levels of communication skills were fairly strongly associated with the outcomes, applicants whose communication skills were not described had a very low chance of acceptance, and applicants whose indicators of communication skills were “high” had a very high chance of acceptance. The biggest predictors of acceptance appear to be having indicators of communication and potential for professionalism at a high level. As shown in Table 2, these are precisely the indicator levels that the applicant F.G. had for these two criteria and the other applicants did not.

Figure 4.

Likelihood of acceptance as a function of indicators of seven different applicant criteria. CLD = cultural and linguistic diversity.

Discussion

This study explored how faculty and doctoral candidates rated vignettes of applicants to master's programs in speech-language pathology. Respondents were likely to rank applicants varying in levels of indicators of criteria lower than the applicant who was “high” across all indicators except for cultural and linguistic diversity. Respondents were also more likely to make an explicit “accept” decision for the latter applicant.

Equity in Admissions

Overall, the applicants in the vignettes reflected the real-world complexity of applicants. We manipulated vignettes to vary in the indicators of holistic review criteria that admissions committees use to evaluate applicants: academic ability and preparation, communication skills, cultural and linguistic diversity, interpersonal skills, analytical skills, potential for professionalism, and potential for leadership. Cultural and linguistic diversity was not manipulated independently from other factors; we did not compare applicants who were maximally similar other than their cultural and linguistic background.

A potential criticism of this study could be that if A.B. through E.F. received lower ratings than F.G., that could have occurred because of other factors (e.g., they had “weaker” applications) rather than because of their CLD background. That argument, however, presupposes that the goal of admissions should be for committees to ensure applicants with an equal demonstration of indicators of academic ability get equal admissions outcomes. We are approaching the problem, however, from an antideficit and systemic perspective (AAMC, 2013; Urban Universities for Health, 2016). In holistic review, committees should take on the responsibility of creating an environment that honors and respects applicants' backgrounds, including potential for growth, in their evaluation systems. This necessarily includes admissions and extends to ongoing evaluation of students in the program (AAMC, 2013).

In practice, being of a CLD background is often confounded with many of the indicators that committees evaluate—not because students from CLD backgrounds are weaker, but because of structural inequities which often set up students from dominant backgrounds with more chances to show their academic ability, leadership potential, and other characteristics that graduate admissions committees tend to recognize (Kendi, 2020; McGlynn, 2017). Thus, if one makes the assumption that students from CLD backgrounds often face structural barriers that students from non-CLD backgrounds do not and thus that a CLD student who is just as qualified as a comparable non-CLD student may nevertheless appear weaker along certain indicators (Bleske-Recheck & Browne, 2014; Fuse, 2018; Michel et al., 2019), then a goal of admissions committees should not be to achieve admissions outcomes that are blind to an applicants' background. Rather, the goal should be to create equitable admissions policies that work against inequitable outcomes (Powell, 2012). This may mean ranking an applicant of a CLD background higher than a non-CLD applicant with comparable or slightly higher ratings on personal characteristics, such as academic achievement or potential for leadership, which structurally favor applicants from dominant backgrounds. As per Bastedo et al. (2018), such a ranking would reflect an appreciation of applicant academic and family background. Under such a view, demonstrating that holistic admissions is effective would not require showing that a CLD applicant gets the same (i.e., equal) outcome as a maximally similar non-CLD applicant, that is, why we did not manipulate CLD status independently of other personal characteristics?

To be clear, this study does not assume that all CLD applicants are lower on criteria than their peers from dominant backgrounds, nor does it argue that all reviewers are not culturally responsive. Our argument is that, in light of empirical evidence documenting the systemic barriers that CLD students are likely to face, admissions committees risk evaluating them as lower on criteria if they do not proactively plan for just interpretation and use of application materials and indicators of applicant quality (Messick, 1989). At the same time, admissions committee members are diverse themselves, with respect to their evaluation of applicants (Bastedo et al., 2018). Here, the fact that the highest likelihood of acceptance occurred when indicators of all applicant characteristics, except for cultural and linguistic diversity, were high, which only F.G. had, may or may not be coincidental. As individuals who have succeeded in the field of speech-language-hearing, respondents may have been predisposed to favor those who were similar (Rogus-Pulia et al., 2018; Wilson et al., 2019). Respondents may have more positively ranked applicants where they felt they could identify “success,” which may be tied to how well indicators of applicant characteristics conformed to their own backgrounds.

Recommendations for Graduate Admissions Processes Using Holistic Review

In full holistic review, the time commitment required for evaluation of all application components is significant. The preliminary results of this study suggest that at least one step of a holistic review process (i.e., ratings of criteria in applicants and subsequent ranking) may face challenges, to recognizing excellence across diverse applicant profiles. Nevertheless, with careful development and implementation, holistic review processes may increase diversity—and ultimately, educational excellence—without a substantial workload increase for admissions committees (Wilson et al., 2019). General recommendations from holistic admissions in the health professions include creating an admissions mission statement that includes diversity and balancing academic and nonacademic criteria in initial screening of applicants (Artinian et al., 2017). Here, we offer CSD-specific recommendations for programs seeking to develop effective holistic review processes in graduate admissions.

First, considering that applicant ratings in this study favored the applicant many committees would consider to be the most traditionally successful, admissions committees may have to pursue training to learn about diversity, how to assess characteristics and barriers across diverse cultures, and their own biases (AAMC, 2020; Michel et al., 2019; Zerwic et al., 2018). However, learning about bias alone is insufficient. To actually counter bias, effective steps include having faculty panels that include faculty of diverse backgrounds (or faculty who recognize excellence across diverse backgrounds) review materials, proactively planning an order in which application materials will be reviewed and implementing candidate interviews (Okahana et al., 2018).

Second, given the broader issue of underrepresentation among CSD faculty, programs might consider bringing in CLD alumni to serve as interviewers or advisory board members coaching admissions committees on how to mitigate bias in their decision-making structures and processes (Okahana et al., 2018). Just as CLD faculty may be effective in mentoring CLD students in speech-language pathology (Saenz, 2000), CLD alumni may be effective in interviewing applicants while also helping to not overburden minority faculty who are oftentimes very few and asked to represent all minority groups (Addams et al., 2010). Furthermore, CLD interviewers may be more likely to recognize the barriers that CLD students often face through their own experiences. For example, a CLD mentee shared with one of the authors that they received a low grade in a clinical course. Knowing the student was insightful with cross-cultural perspectives, the mentor probed for more information. It became evident that their training did not include information on the cultural norms or expectations for clinical interactions. Thus, the CLD student had to figure out (a) that there were cultural norms that differed from their own background, (b) what those norms were, and (c) how to acquire this additional set of norms before even approaching clinical training itself. In contrast, their peers of dominant backgrounds were able to bypass (a) through (c) and focus on clinical training. This instantiated example runs counter to narratives, such as that of Ebert and Kohnert (2010), which proposes personality traits drive clinical competence. Such a narrative suggests clinical competence is fixed and may yield racialized outcomes, especially considering the severe underrepresentation of diverse faculty in the discipline of CSD who are arguably better equipped than their White counterparts to appreciate excellence across diverse backgrounds (Canning et al., 2019; Powell, 2012).

Third, programs aiming to implement holistic review must consider not only their admissions processes but also their overall structure to ensure students graduate and advance in the field of speech-language pathology. If the aim is to truly diversify the profession in an intersectional way (Buchanan & Wiklund, 2020; Crenshaw, 1989), programs must also reevaluate and revise their ideas of excellence, outcomes, and supports from preadmissions to postgraduation. For example, if a program outcome, such as passing the Praxis, stands alone without supports in place to ensure students of all backgrounds have a fair opportunity at passing the Praxis, then CLD students and students of marginalized backgrounds may face additional challenges in entering the profession even if they receive admissions offers. Similarly, programs must proactively plan for how they will meaningfully support students throughout their time in the program (Girolamo & Ghali, 2021). One example of a support relevant to re-envisioning excellence is inclusive teaching, such that students of all backgrounds have opportunities to acquire and demonstrate academic and clinical excellence.

Limitations

This study had several limitations. First, the sample size was well below the target sample size. Since data collection took place during the COVID-19 pandemic and a period of civil unrest in the United States, potential participants may have had less availability for study participation or received an influx of invitations to participate in such studies. Although the small sample size limits the precision of the estimates of the effect sizes we examined, we believe the findings are still useful as a preliminary demonstration of how evaluators use holistic review criteria, and we hope future studies will contribute more data to further clarify these patterns. Second, presenting applicant characteristics in vignettes may differ from what admissions committee members view in reality. Clearly, admissions committees complete many more steps of holistic review and, thus, evaluate applicants at each step of the process: screening, interviews, and selection of applicants for offers of admission. However, we pursued this methodology given the aim of conducting an exploratory investigation of holistic review criteria in master's speech-language pathology programs, evidence of vignettes as a valid measure of real-world decision-making behavior, and constraints on survey length to encourage completion.

Future Directions

This study highlights the need for future research on holistic review in speech-language pathology. One direction entails evaluating breakdown in holistic review in terms of diversity. A future study might implement the approach of Bastedo et al. (2018), which employed a survey questionnaire with full hypothetical application files plus interviews. If holistic review fails to diversify accepted applicants, it would be useful to examine which steps of the process do or do not work. In a situation such as the one our study mimicked, in which respondents reviewed brief profiles, problems may occur not in the way the committee evaluates the vignettes but in how committee members construct profiles based on the application materials in the first place. One aim of holistic review is to consider the whole context, such that strengths in some areas may offset weaknesses in other areas (Wilson et al., 2019). Further work is needed to understand how interpretation of applicant criteria plays out at each step, and evaluating holistic review models, perhaps using the model from AAMC (2010), may inform this area (Okahana et al., 2018). Future research could also explore what practices support admissions committees in considering an applicant's personal characteristics plus background. Evidence from biosciences and nursing suggests that training members for admissions committees on holistic review, as well as on the interpretation and use of master's application materials, is effective at increasing diversity (Addams et al., 2010; Okahana et al., 2018; Urban Universities for Health, 2016; Wilson et al., 2019; Zerwic et al., 2018). However, training efficacy has yet to be tested in speech-language pathology.

A Final Note: Measurement and Merit

This study underlined the need to critically question admissions processes, even if they sound promising. Following AAMC (2013), we must consider the fundamental principles of holistic review in the context of speech-language pathology. What constructs, or personal characteristics, should admissions committees measure? How should committees measure these characteristics? If committees use an evaluation system where characteristics are treated as fixed (e.g., Canning et al., 2019; Ebert & Kohnert, 2010), rather than ones which are socially situated, may or may not indicate excellence, or which could be cultivated through academic and clinical training, does this create or hinder excellence in the profession? Finally, who are our admissions systems built for and not built for? As Mandulak (2021) noted, “the resistance and difficulty with change, with respect to…assumptions about merit and achievement may be so well-entrenched in our processes within our programs” (p. 4). To achieve excellence for our profession, we must not only reshape our notions of merit but also restructure our systems to be for all students.

Appendix

An online study on academic peer reviewing

You are invited to participate in a study conducted by Dr. Stephen Politzer-Ahles, who is a staff member of the Department of Chinese and Bilingual Studies in The Hong Kong Polytechnic University. The project has been approved by the Human Subjects Ethics Subcommittee (HSESC) of The Hong Kong Polytechnic University (HSESC Reference No. HSEARS20200703001).

The aim of this study is to better understand entry to speech-language pathology master's programs in the United States. You will be asked to read profiles of students while pretending that these are applicants to a master's program in speech-language pathology. For each profile, you will be asked to rate applicant quality. It is hoped that the results of this experiment will help us understand more about admissions for master's speech-language pathology programs.

The experiment has no risks or direct benefits to you. All information related to you will remain confidential and will be identifiable by codes only known to the researcher. You have every right to withdraw from the study before completing the survey, without penalty of any kind. The survey is expected to take between 10 and 15 minutes to complete.

If you would like to obtain more information about this study, please contact Dr. Stephen Politzer-Ahles (Tel. no.: +852 27662891/e-mail: sjpolit@polyu.edu.hk).

If you have any complaints about the conduct of this research study, please do not hesitate to contact Miss Cherrie Mok (cherrie.mok@polyu.edu.hk), Secretary of the HSESC of The Hong Kong Polytechnic University, in writing (c/o Research Office of the University), stating clearly the responsible person and department of this study as well as the HSESC Reference Number.

Thank you for your interest in participating in this study.

Stephen Politzer-Ahles, PhD

Principal Investigator

◯ I consent to participate in this study.

◯ I do not consent to participate in this study.

Screener

I am affiliated with a CAA-accredited program in speech-language pathology or the equivalent (e.g., communication science and disorders, communicative disorders) within the United States.

◯ No

◯ Yes

I am a PhD student/candidate or faculty member (nontenure track, tenure track, or tenured).

◯ No

◯ Yes

Survey instructions

Thank you for taking time to complete this survey. The purpose of this survey is to examine how faculty and PhD students/candidates evaluate applicants to master's programs in speech-language pathology. Completion of this survey is anonymous.

You can complete the survey over multiple sessions. Please keep in mind that you will not be able to return to previous questions once you move to the next page. The survey should take 10–15 minutes.

Demographic information

In order to track how representative this survey's respondent base is of the demographics of the American Speech-Language-Hearing Association, please answer the following questions.

(a) Select the item(s) that best describe yourself.

□ American Indian, Eskimo, Aleut

□ Asian or Pacific islander

□ Black or African American

□ Caucasian or White

□             Other:

□ Don't know

□ Prefer not to say

(b) Of Spanish-Hispanic/Latinx origin (select one)

◯ No

◯ Yes

◯ Don't know

◯ Prefer not to say

Which of the following describes your gender identity? Check all that apply.

□ Nonbinary

□ Multigender

□ Gender fluid

□ Agender/no gender

□ Genderqueer

□ Male

□ Female

□ Prefer not to respond

□             An option not listed here:

Professional background

What is the highest level of education that you have completed?

◯ Bachelor's degree or equivalent

◯ Master's degree or equivalent

◯ AuD or equivalent

◯ SLPD or equivalent

◯ PhD or equivalent

◯             Other:

What is your current position?

◯ PhD student or candidate

◯ Assistant professor or equivalent

◯ Associate professor or equivalent

◯ Clinical professor (nontenure track) or equivalent

◯ Full professor or equivalent

◯ Lecturer or equivalent

◯ Research associate/scientist or equivalent

◯             Other:

Applicant profiles

The next section of the survey will present six profiles of prospective students who have applied to a master's program in speech-language pathology and ask you to evaluate each applicant.

Note: For this section, please use only your best professional judgment to evaluate each applicant as is. Assume there is no applicant interview or other available information and that each applicant has the required materials in their application. You may use the following information in your evaluation:

• Academic ability and preparation: Students need foundational knowledge in core speech-language-hearing concepts in addition to knowledge from related disciplines so they are able to learn how to apply this knowledge to clinical situations.

• Communication skills: SLPs need to communicate clearly and effectively with clients, families, and other professionals in spoken and written formats.

• Interpersonal skills: Given the importance of working with clients, families, and other professionals, SLPs need to be able to work effectively in collaboration with a wide range of people.

• Analytical skills: To implement evidence-based practice, SLPs need to critically read, analyze, interpret, and apply research to clinical practice. Accordingly, SLPs need a firm foundation in research, critical thinking, and clinical application.

• Potential for professionalism: Because SLPs work in busy and sometimes stressful environments, they need to be organized, reliable, and respectful. SLPs are also lifelong learners and need to be able to grow from constructive feedback.

• Potential for leadership: SLPs advocate for their clients to ensure they receive appropriate services and may also advocate for their profession at the local, state, or national level.

• Cultural and linguistic diversity: SLPs have diverse and multilingual caseloads, such that they need to value and work well with people from a variety of backgrounds that differ from their own.

Vignette 1

A.B. graduated from a private college with a 3.8 GPA. Their resume shows that they worked as a Teaching Assistant (TA) for one semester and had an internship with the general counsel of General Electric Corporate. In their essay, A.B. wrote about growing up in an ethnic enclave. Their letter of recommendation writers were three professors, one of whom supervised the student in their work as a TA. One professor reported that the student wrote strong essays in class and produced comprehensive written reports and client plans in clinic. The other professor wrote that A.B. rarely participated in group work in class. The supervising professor wrote that the student could be hard to reach and received below average student ratings as a course TA.

Rate A.B. on each of the following:

	Weak	Somewhat weak	Neither weak nor strong	Strong	Very strong
Academic ability and preparations	◯	◯	◯	◯	◯
Communication skills	◯	◯	◯	◯	◯
Interpersonal skills	◯	◯	◯	◯	◯
Analytical skills	◯	◯	◯	◯	◯
Potential for professionalism	◯	◯	◯	◯	◯
Potential for leadership	◯	◯	◯	◯	◯
Cultural and linguistic diversity	◯	◯	◯	◯	◯

Please indicate your admissions decision for A.B. based on the information available.

◯ Admit

◯ Waitlist

◯ Reject

Please leave any optional comments or reasoning for your ratings and admissions decision for A.B.

           

           

           

           

Vintage 2

B.C. is a first-generation college graduate. Their resume shows they had a 3.3 overall GPA, had a 3.1 GPA in speech-language-hearing, and worked full time during college as a server. In their essay, B.C. wrote about finding free classes in their community to learn another language. Their letter of recommendation writers were their former employer and two professors. The employer reported that B.C. showed initiative at work and streamlined the process of taking and delivering orders to customers. One professor noted that the student's speaking style was not appropriate for clinic (i.e., spoke too casually with grammatical errors). The other professor wrote that despite showing interest for the profession, classmates reported difficulty working with the student on group projects due to their lack of availability.

Rate B.C. on each of the following:

	Weak	Somewhat weak	Neither weak nor strong	Strong	Very strong
Academic ability and preparations	◯	◯	◯	◯	◯
Communication skills	◯	◯	◯	◯	◯
Interpersonal skills	◯	◯	◯	◯	◯
Analytical skills	◯	◯	◯	◯	◯
Potential for professionalism	◯	◯	◯	◯	◯
Potential for leadership	◯	◯	◯	◯	◯
Cultural and linguistic diversity	◯	◯	◯	◯	◯

Please indicate your admissions decision for B.C. based on the information available.

◯ Admit

◯ Waitlist

◯ Reject

Please leave any optional comments or reasoning for your ratings and admissions decision for B.C.

           

           

           

           

Vignette 3

C.D. graduated from a state university. Although sparse, their resume shows that they led some type of community cultural programming with a local nonprofit organization. The organization has an ethnic name, but the cultural connection is unclear. In their essay, C.D. wrote about how they needed to increase their flexibility to improve their leadership skills. Their letter of recommendation writers included two professors. One professor wrote that C.D. served in an affinity organization and worked well with fellow officers. However, they seemed to have difficulties getting along with a significant proportion of their classmates in clinic. The other professor wrote that the student was enthusiastic about their interests in the field of speech-language-hearing and sought out clinical shadowing opportunities.

Rate C.D. on each of the following:

	Weak	Somewhat weak	Neither weak nor strong	Strong	Very strong
Academic ability and preparations	◯	◯	◯	◯	◯
Communication skills	◯	◯	◯	◯	◯
Interpersonal skills	◯	◯	◯	◯	◯
Analytical skills	◯	◯	◯	◯	◯
Potential for professionalism	◯	◯	◯	◯	◯
Potential for leadership	◯	◯	◯	◯	◯
Cultural and linguistic diversity	◯	◯	◯	◯	◯

Please indicate your admissions decision for C.D. based on the information available.

◯ Admit

◯ Waitlist

◯ Reject

Please leave any optional comments or reasoning for your ratings and admissions decision for C.D.

           

           

           

           

Vignette 4

D.E. is an international university graduate with borderline TOEFL scores and an overall GPA of 8.68 out of 10. Their resume shows they had several years of experience in the health professions as some type of student hourly or research assistant. Although their essay was short and choppy, D.E. wrote about working with families with children from a variety of backgrounds and with a wide range of speech-language service needs and their families in their country. Their letter of recommendation writers were two professors and one lab principal investigator who was their former boss. The professors' letters indicated that the student was compliant in following directions for coursework and clinic but required significant clinical supervision. The PI wrote that they interacted with families well.

Rate D.E. on each of the following:

	Weak	Somewhat weak	Neither weak nor strong	Strong	Very strong
Academic ability and preparations	◯	◯	◯	◯	◯
Communication skills	◯	◯	◯	◯	◯
Interpersonal skills	◯	◯	◯	◯	◯
Analytical skills	◯	◯	◯	◯	◯
Potential for professionalism	◯	◯	◯	◯	◯
Potential for leadership	◯	◯	◯	◯	◯
Cultural and linguistic diversity	◯	◯	◯	◯	◯

Please indicate your admissions decision for D.E. based on the information available.

◯ Admit

◯ Waitlist

◯ Reject

Please leave any optional comments or reasoning for your ratings and admissions decision for D.E.

           

           

           

           

Vignette 5

Student E.F. graduated from a minority-serving institution with an overall GPA of 3.0 and a 3.4 GPA in speech-language-hearing. Their resume shows they are fluent in two languages. In their essay, E.F. wrote about being raised by their extended family and being inspired to pursue a career in the profession to help care for others as they did for their younger relatives. Their letter of recommendation writers included two professors. One was a tenured professor who was an officer in ASHA. They wrote that the student showed limited initiative in learning to use course materials independently. The other, a clinical faculty member, wrote that the student demonstrated strong critical thinking skills in their term papers but lagged behind their classmates in contributing to class discussion.

Rate E.F. on each of the following:

	Weak	Somewhat weak	Neither weak nor strong	Strong	Very strong
Academic ability and preparations	◯	◯	◯	◯	◯
Communication skills	◯	◯	◯	◯	◯
Interpersonal skills	◯	◯	◯	◯	◯
Analytical skills	◯	◯	◯	◯	◯
Potential for professionalism	◯	◯	◯	◯	◯
Potential for leadership	◯	◯	◯	◯	◯
Cultural and linguistic diversity	◯	◯	◯	◯	◯

Please indicate your admissions decision for E.F. based on the information available.

◯ Admit

◯ Waitlist

◯ Reject

Please leave any optional comments or reasoning for your ratings and admissions decision for E.F.

           

           

           

           

Vignette 6

Student F.G. graduated with a 4.0 GPA. Their resume shows that they worked as an undergraduate research assistant and served on the boards of the National Student Speech-Language-Hearing Association and of a Greek organization. In their essay, F.G. wrote about what they learned in a month-long study-abroad volunteer program with children and in organizing an annual fundraiser for children with special education needs. Their letter of recommendation writers were two professors and the principal investigator of the lab where they worked. One professor wrote that they were frequently the leader during group work in class. The lab PI wrote that they were a good fit for the lab team with sharp thinking, clear communication skills, and responded promptly to electronic communications.

Rate F.G. on each of the following:

	Weak	Somewhat weak	Neither weak nor strong	Strong	Very strong
Academic ability and preparations	◯	◯	◯	◯	◯
Communication skills	◯	◯	◯	◯	◯
Interpersonal skills	◯	◯	◯	◯	◯
Analytical skills	◯	◯	◯	◯	◯
Potential for professionalism	◯	◯	◯	◯	◯
Potential for leadership	◯	◯	◯	◯	◯
Cultural and linguistic diversity	◯	◯	◯	◯	◯

Please indicate your admissions decision for F.G. based on the information available.

◯ Admit

◯ Waitlist

◯ Reject

Please leave any optional comments or reasoning for your ratings and admissions decision for F.G.

           

           

           

           

Professional background continued

Do you serve on an admissions board for a master's program in speech-language pathology, communication science and disorders, communicative disorders, or the equivalent?

◯ Yes

◯ No

Do you serve on some other admissions board (e.g., bachelor's or doctoral) in speech-language pathology or communication science and disorders?

◯ Yes

◯ No

Please indicate what you consider in applicant reviewal. Select all that apply.

□ Application file (i.e., application materials)

□ Family background

□ Previous educational environment

□ Undergraduate alumni status (i.e., whether or not student attended school for undergraduate degree)

□ Unique contributions the applicant would bring to the program

□             Other:

Do you have any comments regarding this survey? Thank you for your time and effort.

           

           

           

           

Funding Statement

This work was supported by T32 DC000052 (Director: Mabel L. Rice) and R01 DC001803 (PI: Mabel L. Rice).