Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 May 6.
Published in final edited form as: Train Educ Prof Psychol. 2021 Apr 5;15(3):211–218. doi: 10.1037/tep0000336

Who Are We Missing? Examining the Graduate Record Examination Quantitative Score as a Barrier to Admission into Psychology Doctoral Programs for Capable Ethnic Minorities

Jennifer M Gómez 2,3, Annmarie Caño 4, Boris B Baltes 1
PMCID: PMC9075693  NIHMSID: NIHMS1751794  PMID: 35528021

Abstract

The field of psychology must racially/ethnically diversify to create a workforce that can meet the needs of education, training, and interventions in an increasingly pluralistic society. Systemic bias in psychology doctoral programs’ admissions process may partially account for relatively few psychologists being underrepresented minorities (URMs). The use of the Graduate Record Examination Quantitative score (GRE-Q) is one important modifiable barrier. The purpose of the current study is to go beyond replicating the association between the GRE-Q and desired doctoral outcomes by examining if a cut-off score for the GRE-Q as a proxy for potential to succeed in psychology doctoral programs disproportionately impacts URMs. Participants (N = 226) were psychology doctoral students at a Carnegie-classified Highest Research Activity (R1) large Midwestern university, who were admitted to graduate school from 2001 to 2011. Our findings show that, while controlling for undergraduate grade point average (GPA) and prior master’s degree attainment, the GRE-Q predicted grades in two required graduate statistics courses and overall graduate GPA. Importantly, all students, regardless of their GRE-Q score, demonstrated competence in their statistics coursework, as assessed by their course grades. Moreover, we found that guidelines that bar admission into the psychology doctoral program for students with low GRE-Q scores would have disproportionately impacted URMs, resulting in 44% being barred admission versus only 17% of their White/Asian/Pacific Islander counterparts. Practical implications include introducing holistic review protocols into the admissions process, while educating faculty on how heavy emphasis on the GRE-Q contributes to inequitable exclusion of capable URMs.

Keywords: GRE, holistic review, psychology doctoral programs admissions, inclusive excellence, grit

The field of psychology needs to racially and ethnically diversify to create a workforce that can meet the needs of education, training, and intervention in an increasingly pluralistic society (American Psychological Association [APA], 2003; American Psychological Association [APA], 2017; Callahan et al., 2018; Gómez, 2015; Hall, 2006; Hall et al., 2016; Kohout et al., 2014). Yet, there is evidence that diversity with respect to racial and ethnic group representation is lacking in psychology across the training life course. For instance, a recent report summarizing national data and offering strategies to address marginalization indicated that 84% of psychology faculty and instructional staff and nearly 80% of tenured/tenure-track faculty identified as White (Boyd et al., 2017). Similar to their findings regarding the lack of diversity among faculty ranks, Boyd et al. (2017) reviewed data showing that nearly three-quarters of psychology doctorate degree earners identified as White (Boyd et al., 2017). Because psychology doctoral programs serve as a gateway into the profession, it is imperative to examine the extent to which the graduate admission process may interfere with creating a workforce that can meet contemporary and future demands.

Systemic bias in psychology doctoral programs’ admissions process may partially account for relatively few psychologists being underrepresented minorities (URMs; Vasquez & Jones, 2006). The use of the Graduate Record Examination Quantitative score (GRE-Q) is one important modifiable barrier. Though past research has documented the GRE-Q’s link with graduate student success (e.g., Kuncel et al., 2001), there remains some controversy around the heavy emphasis placed on the GRE-Q (e.g., Petersen et al., 2018; Sampson & Boyer, 2001), with the Educational Testing Service (ETS) (2019) and other work suggesting it should be assessed alongside a range of other applicant factors, such as background, experience, and education, due to its limitations and potential for bias against URMs and others (Miller & Stassun, 2014; Pacheco et al., 2015; Posselt, 2014; Sternberg & Williams, 1997). Beyond the predictive value of the GRE-Q lies other important questions: When focusing on the predictive nature of the GRE-Q, what—and more importantly, whom—are we missing? Is it possible that we are missing capable, qualified URMs in our admissions process when the research focuses on the extent to which the GRE-Q predicts psychology doctoral student success? To examine these questions, the purpose of the current study is to go beyond replicating the association between the GRE-Q and desired doctoral outcomes by examining if using a cut-off score on the GRE-Q as a proxy for potential to succeed in psychology doctoral programs disproportionately impacts URMs.

GRE and Doctoral Admissions

Test scores are often used as indicators of merit, with the argument being that they assess the cognitive capacity required for advanced study (Kuncel et al., 2001, 2010). Even though the “test score = merit” conceptualization uses this myth of meritocracy to hide structural racial inequality (Au, 2016; Stewart & Haynes, 2016), standardized test scores, such as the GRE, have been a central part of the graduate admissions process at many elite institutions (Posselt, 2014). One reason for this is the empirical evidence supporting the predictive validity of the GRE. Several studies have shown that the GRE predicts first-year Graduate grade point average (GGPA), overall grade point average (GPA), and publication counts (e.g., Kalat & Matlin, 2000; Kuncel et al., 2001). Yet, other studies have shown weak correlations between GRE scores and graduate student outcomes. For instance, in a sample of psychology doctoral students, Sternberg and Williams (1997) found a weak correlation between GRE and first year GGPA and that the GRE did not predict faculty ratings of student success, second year GGPA, and dissertation reader ratings. They concluded that a heavy reliance on the GRE is not appropriate for doctoral admissions in psychology.

A number of researchers argued that Sternberg and Williams (1997) did not account for sampling and statistical concerns that could have contributed to an underestimation of GRE effects. For instance, the average scores in the sample were high and the scoring distribution was not normal (Andre & Hegland, 1998). Kuncel et al. (2001) reported that after they corrected for range restriction and criterion unreliability, the correlations of the GRE with the first-year grades and research ratings were stronger than reported in Sternberg and Williams (1997). Kuncel and colleagues have since conducted other studies applying similar corrections to show that the GRE is significantly correlated with a variety of graduate outcomes (Kuncel et al., 2001, 2010). In their analyses, the GRE appears to be more strongly correlated with graduate GPA than with degree completion or faculty ratings of student performance.

Given the aforementioned findings, is the mere correlation of the GRE with outcomes sufficient to warrant the use of such standardized tests as indicators of whether a student merits admission to a graduate program? Many researchers and educators urge caution. For instance, some maintain that standardized test performance is vulnerable to expectations, framing, socioeconomic background, and personal traits (Croizet, 2008; Kalsbeek et al., 2013; Vasquez & Jones, 2006), increasing the difficulty with which to interpret scores. Others argue that merit should not be assessed with a standardized test because “deservingness” is based on a host of factors that include program goals and needs, which rarely focus on a score. Instead “deservingness” is based on desired traits such as perseverance, creativity of thought, community engagement, and entrepreneurship, for example (Morrison & Morrison, 1995). Similarly, ETS states that GRE scores are but one piece of information to be evaluated in the admissions process (ETS, 2019). In addition, ETS (2019) also urges caution about interpretations of GRE scores among groups that are underrepresented in their testing samples (e.g., URM) and obtain lower tests scores than White men and continuing-generation college students. Consequently, GRE-Q’s should not be used as an initial screening measure (ETS, 2019), as doing so results in women and URMs being immediately eliminated from STEM applicant pools (Miller & Stassun, 2014).

The aforementioned results showing that the GRE is a systemic barrier to access to doctoral programs are concerning, perhaps particularly for graduate programs that purport to use Inclusive Excellence principles (Association of American Colleges and Universities, 2002). In this respect, giving inordinate weight to GRE-Q scores run counter to achieving Inclusive Excellence. Therefore, another way to examine the utility of the GRE-Q in psychology doctoral program admissions is to examine to what extent it hampers or interferes with Inclusive Excellence goals. In order for psychology to be an equitable profession, systemic, predictable barriers that disproportionately impact URMs negatively should be eliminated. Additionally, the field of psychology needs to recruit the best future researchers, instructors, supervisors, and clinicians in order for psychology to remain relevant and effective in an increasingly pluralistic society (e.g., Hall, 2006). In fact, the principle of Inclusive Excellence (Association of American Colleges and Universities, 2002) states that excellence requires diversity. Therefore, the common question of “Is the GRE-Q a good predictor of success?” may be missing the point. A different question that is aligned with Inclusive Excellence is: If we are relying on the GRE as a proxy for potential to succeed in psychology doctoral programs, what stellar students—who, given the opportunity, could go on to transform the field of psychology for the better—are being locked out of the psychology profession? Furthermore, whose talent, skill, stamina, and grit (Miller & Stassun, 2014) is not being predicted by the GRE-Q? Simply put, regardless of whom the GRE-Q is working for, a question that directly relates to equity and diversifying psychology is whom is the GRE-Q systematically working against?

Purpose of the Study

Diversifying psychology is needed to address complex phenomena facing our society (e.g., Hall, 2006). Though the GRE-Q is often conceptualized as a proxy for intelligence, preparation for graduate school, and potential, heavy emphasis on the GRE-Q may result in a systematic barrier to entry into the psychology profession for qualified URMs (e.g., Miller & Stassun, 2014). Therefore, one of the goals is to eliminate barriers in the admissions process that bar capable URMs from entry into psychology doctoral programs, and thus the field. The purpose of the current study is threefold: (a) Test the extent to which the GRE-Q, controlling for undergraduate GPA and prior master’s degree attainment, predicts relevant doctoral student outcomes; (b) Determine, by GRE-Q scoring groupings (from very low scorers to very high scorers), percentage of students who failed the three graduate statistics course sequence; and (c) Compare the proportion of URM versus more highly represented (MHR; White and Asian American/Pacific Islander) students who would have been barred access to the psychology doctoral program if a cut-off score was used.

Specifically, we hypothesized that:

  • Hypothesis 1: Controlling for undergraduate GPA and prior master’s degree attainment, the GRE-Q will be associated with higher grades in the three graduate statistics courses sequence, overall graduate GPA, number of publications in the first 5 years of graduate school, time to advance to candidacy, and time to degree completion.

  • Hypothesis 2: Compared to MHR students, a greater proportion of capable URM students would have been barred access to the psychology doctoral program if a strict cut-off score had been implemented (GRE-Q: 600).

Method

Participants

Participants were psychology doctoral students at a Carnegie-classified Highest Research Activity (R1) large Midwestern university, who were admitted to and attended graduate school from 2001 to 2011 and reported GRE scores using the older scoring method (max score of 800 on the GRE-Q). In alignment with the National Science Foundation (NSF)/National Center for Science & Engineering Statistics’ (2017) determination of underrepresentation in science and engineering including Black/African American, Hispanic/Latina/o, and American Indian/Alaska Native, the total sample (N = 226) was coded as either URM or MHR, with URM being Black/African American, Hispanic/Latina/o/x, Arab, or American Indian/Alaska Native, and MHR being White, Asian American, or Pacific Islander. Upon graduate admission, participants were in their mid-20’s on average (M = 24.83 years, SD = 4.98), with no significant age differences (t(28.63) = −.53, ns) between URM (N = 23; 25.30 years, SD = 4.51) and MHR (N = 203; M = 24.78, SD = 5.03) students. As measured with check boxes for male and female, gender distribution in the sample is reflective of the field of psychology, with 72.6% women for the total sample, which was not statistically different (χ2(1) = .70, ns) between URM (65.2%) and MHR (73.4%) students. The majority of incoming students had not already earned a master’s degree (86.7%, no master’s degree), which did not significantly vary by URM (87.0%, no master’s degree) versus MHR (86.7%, no master’s degree) students (χ2(1) = .00, ns). The racial/ethnic breakdown for the whole sample of graduate students was majority White (82.3%), followed by Asian American/Pacific Islander (7.5%), Black/African American (6.2%), Hispanic/Latina/o/x (1.77%), Arab (1.3%), and American Indian/Alaska Native (.88%). The URM group were 60.9% Black/African, 17.4% Hispanic/Latina/o/x, 13.0% Arab, and 8.7% American Indian/Alaska Native. The MHR group was mostly White (91.6%), with the remainder being Asian/Asian American/Pacific Islander. Students represented the four interest areas in the department, including the American Psychological Association (APA)-accredited clinical psychology program (total: 46.0%; URM = 8.7%), the cognitive/developmental/social psychology program (total: 20.8%; URM = 17.0%) the industrial-organizational (I/O) psychology program (total: 19.9%; URM = 11.1%), and the behavioral and cognitive neuroscience program (total: 13.7%; URM = 3.3%).

Procedure and Measures

This study received a Concurrence of Exemption from the university’s institutional review board (IRB). Data were collected from the university’s database of doctoral students admitted from 2001 to 2011. The following information was collected: student-provided undergraduate GPA on a 4-point scale, whether the student had a master’s degree prior to entry into the doctoral program, and GRE-Q submitted through ETS.

In addition, letter grades for the following quantitative courses were collected: Quantitative Methods in Psychology I (t-tests, analysis of variance, correlation; typically taken in the first semester of the first year of doctoral study), Quantitative Methods in Psychology II (reliability, validity, and scale development; typically taken in the second semester of the first year), and Multivariate Analysis in Psychology (multiple regression, MANOVA, factor analysis; typically taken in the first semester of the second year of study). All statistics courses include learning and conducting relevant statistical analyses.

Several indicators of doctoral degree progress and achievement were also recorded, including final doctoral GPA on a 4-point scale (upon degree completion or departure from the program), time to doctoral candidacy in months, and time to doctoral degree completion in months. Finally, a research assistant conducted a search in Google Scholar to assess the number of publications per student during the first 5 years of graduate school.

Results

The purpose of the current study was to assess the role of the GRE-Q as a potential barrier to access to psychology doctoral programs for URMs. The total sample performed moderately well on undergraduate and graduate criteria related to and including the GRE-Q, with MHR students on average entering with higher undergraduate GPA and GRE-Q, and in graduate school performing better in the last required statistics course, overall graduate GPA, and number of publications (Table 1). Of note, the mean grades in graduate school were above a 3.0 average for URM and MHR students. Correlational analyses among all variables show that the predictor variables are not highly correlated with one another (not shown).

Table 1.

Scoring Demographics of the Total Sample, URMs, and More Highly Represented Students

Total sample, N = 226 URMs, N = 20 More highly represented students, N = 206 t-Testa
Metrics M (SD) M (SD) M (SD) T-statistic (df)
Undergraduate GPA 3.61 (.34) 3.39 (.44) 3.63 (.32) 2.28* (18.71)
GRE-Q 657.42 (72.58) 596.09 (90.79) 664.41 (67.00) 3.50** (24.80)
Graduate stats course 1 3.89 (1.28) 3.79 (.28) 3.91 (.21) 1.93 (25.36)
Graduate stats course 2 3.60 (.38) 3.45 (.46) 3.62 (.37) 1.72 (24.37)
Graduate stats course 3 3.73 (.40) 3.42 (.47) 3.82 (.18) 3.09** (20.53)
Graduate GPA 3.81 (.20) 3.68 (.30) 3.82 (.18) 2.23* (24.00)
Publications in graduate schoolb 1.83 [2.00] (3.18) .71 [.00] (1.27) 1.95 [2.00] (3.30) 3.40** (56.85)
Time to candidacy in months 46.36 (12.03) 47.59 (13.11) 46.23 (11.95) −.41 (18.96)
Time to degree completion in months 76.24 (16.01) 80.76 (16.11) 75.71 (15.97) −.23 (19.82)
Open in a new tab

Note. All grades are calculated on a 4.0 scale, with 3.0 being a “B” average, and 4.0 being “A+.”

URMs = underrepresented minorities; GPA = grade point average; GRE-Q = Graduate Record Examination Quantitative score.

a

All t tests, equal variances not assumed.

b

Due to positive skewness, [median] additionally calculated for Publications in Graduate School; t-test uses mean scores.

*

p < .05.

**

p < .01.

Hypothesis 1 was partially supported. When controlling for undergraduate GPA and prior master’s degree attainment, GRE-Q was associated with higher grades in the first two required graduate statistics courses and overall graduate GPA; GRE-Q was not related to higher grades in the third required graduate statistics course, number of publications in the first 5 years of graduate school, time to candidacy, or time to completion of degree (Table 2). We next ran semi-partial correlations to isolate the variance explained by each outcome while controlling for the other predictors (Table 3). We found that all effect sizes were small. Undergraduate GPA had a larger effect on grades in the third graduate statistics course and Graduate GPA compared to the GRE-Q. Conversely, compared to the GRE, prior master’s degree attainment had a larger effect on grades in the first graduate statistics course. Finally, the GRE-Q had a larger effect than on grades in the second graduate statistics course and time to degree completion (Table 4).

Table 2.

Multivariate Analysis of Covariance (MANCOVA): Controlling for Undergraduate

Outcomes Undergraduate GPA
F(1, 119)
(η2)		 Master’s degree
F(1, 119)
(η2)		 GRE quantitative score
F(4, 119)
(η2)
Graduate stats course 1 4.27*
(.04)		 13.33***
(.10)		 4.00**
(.12)
Graduate stats course 2 13.63***
(.10)		 3.60
(.03)		 4.55**
(.13)
Graduate stats course 3 10.01**
(.08)		 .23
(.00)		 1.59
(.05)
Graduate GPA 23.71***
(.17)		 1.34
(.01)		 5.03**
(.15)
Number of publications 1.15
(.01)		 .57
(.01)		 .62
(.02)
Time to candidacy .14
(.00)		 .36
(.00)		 1.24
(.04)
Time to degree completion 3.73
(.03)		 .60
(.01)		 1.68
(.05)
Open in a new tab

Note. GPA, and Prior Master’s Degree Attainment—GRE Quantitative score as a predictor of graduate.

Statistics courses 1, 2, 3, graduate GPA, number of publications in first 5 years of graduate school, time to candidacy, and time to degree completion. GPA = grade point average; GRE-Q = Graduate Record Examination Quantitative score.

*

p < .05.

**

p < .01.

***

p < .001.

Table 3.

Semi-Partial Correlations Between Each Predictor and Outcome Variable While Controlling for Other Predictors

Outcomes Undergraduate GPA (r) Master’s degree (r) GRE-Q (r)
Graduate stats course 1 .16* .27** .20**
Graduate stats course 2 .23** .14* .27**
Graduate stats course 3 .23** .07 .15*
Graduate GPA .21** .10 .12*
Number of publications −.07 .07 .08
Time to candidacy −.09 .01 −.09
Time to degree completion .02 −.15* −.22**
Open in a new tab

Note. GPA = grade point average; GRE-Q = Graduate Record Examination Quantitative score.

*

p < .05.

***

p < .01;

one-tailed.

Table 4.

Impact of Predictors: Effect Size Comparisonsa of Undergraduate GPA, Prior Master’s Degree Attainment, and GRE Quantitative Score on Graduate Statistics Courses 1, 2, 3, Graduate GPA, Number of Publications in First 5 Years of Graduate School, Time to Candidacy, and Time to Degree Completion

Outcomes Undergraduate GPA Master’s degree GRE quantitative score
Graduate stats course 1 Small Smallb Small
Graduate stats course 2 Small Small Smallb
Graduate stats course 3 Smallb — Small
Graduate GPA Smallb — Small
Number of publications — — —
Time to candidacy — — —
Time to degree completion Small Smallb
Open in a new tab

Note. GPA = grade point average; GRE-Q = Graduate Record Examination Quantitative score.

a

Effect sizes from semi-partial correlations (this table), with cut-off for size of effects from Richardson (2011).

b

Comparatively biggest effects on each outcome.

Finally, more URM students scored below the 600-score cut-off on the GRE-Q (Table 5). Therefore, in support of Hypothesis 2, a greater proportion of URM students (43.5%) compared to MHR students (16.8%) would have been barred admission to the psychology doctoral program had a strict cut-off score been implemented (χ2(1) = 9.41, p = .002). Of note, all students who fully matriculated from all GRE-Q score groupings passed the statistics courses with a 3.0 (B) or better. In other words, using a cut-off on the GRE-Q would have resulted in barring admission to students who would have passed the required graduate statistics courses.

Table 5.

GRE-Q Score Groupings by URM Versus More Highly Represented Students

GRE-Q Score Groupings Total, N = 226 N (%) Underrepresented minorities (URM), N = 23 N (%) Whites/Asian Americans/Pacific Islanders, N = 203 N (%)
Lowest to 550 25 (11.06) 9 (39.13) 16 (7.88)
551–600 19 (8.41) 1 (4.3) 18 (8.87)
601–650 63 (27.88) 8 (34.8) 55 (27.09)
651–700 56 (24.78) 2 (8.7) 54 (26.60)
700+ 62 (27.43) 3 (13.0) 59 (29.06)
Open in a new tab

Note. Percentages denote the total number of ethnic grouping (URM vs. More Highly Represented) in each GRE-Q Score Group.

URM = underrepresented minorities; GRE-Q = Graduate Record Examination Quantitative score.

***

Significant differences by URM versus More Highly Represented group in GRE-Q scoring grouping, Chi-square (4) = 23.69, p = .000; though two cells expected count less than N = 5 participants.

Discussion

With a specific focus on assessing the potential disproportionate impact that the use of the GRE-Q in graduate admissions may have on URMs, the purpose of the current study was twofold: (a) attempt to replicate the finding that higher scores on the GRE-Q is associated with desired outcomes in psychology doctoral programs and (b) extend this work by examining the outcomes of students with lower GRE scores compared with their higher scoring counterparts, with a focus on any disproportionate impact to URMs. The current study builds upon existing GRE literature by focusing on a psychology program in a university that is classified as a Highest Research Activity institution (R1) by the Carnegie Classification of Institutions of Higher Education, with a wider range of test scores and a lower average GRE-Q score than other work (e.g., Sternberg & Williams, 1997).

In partial support of Hypothesis 1, while controlling for undergraduate GPA and prior master’s degree attainment, the GRE-Q was associated with grades in the first two graduate statistics courses and overall graduate GPA, but not the third graduate statistics course, publications in the first 5 years of graduate school, time to advance to candidacy, or time to degree completion. As assessed by semi-partial correlations, each of the effect sizes were small, with the comparatively biggest effects being undergraduate GPA on grades in the third statistics course and overall graduate GPA, prior master’s degree attainment on grades in the first graduate statistics course, and GRE-Q on grades in the second graduate course and time to degree completion. This mixed finding corresponds with work suggesting that while GRE-Q does predict some success in graduate school (Kuncel et al., 2001; Kuncel & Hezlett, 2007), it does not predict all outcomes (Sternberg & Williams, 1997). Of importance in interpreting these correlations is that there is a restricted range within the outcome variables. Specifically, all students earned good grades.

Hypothesis 2 was fully supported. Almost half of URM students would have been barred access to the psychology doctoral program if a strict cut-off score had been implemented, compared to less than 20% of White/Asian American/Pacific Islander students. This is in line with the literature suggesting that overreliance on standardized tests has a discriminatory impact on URMs (Croizet, 2008; Croizet & Dutrévis, 2004). This finding is further in line with ETS (2019) itself, which cautions that care should be taken when interpreting scores for people with nonmajority educational and cultural backgrounds. Of note, our study only included students who were enrolled in our program. Therefore, our data is not able to assess the percentage of URMs who actually were barred admission. Given that the percentage of URM students over the course of 10 years was 10.2% of the total psychology doctoral student population, it is likely that the use of the GRE-Q did result in lower admittance for URMs. Finally, our choice of a proposed cut-off score of 600 on the GRE-Q was based on the larger culture in the site department in which applicants with scores below a 600 received more attention, often requiring additional justification for their admission. Importantly, this idiosyncratic arbitrariness can be found in faculty members’ perceptions of acceptable versus unacceptable GRE scores, with some universities refusing to admit students with scores below an arbitrary cut-off (Posselt, 2014).

Implications

Admissions provides a unique opportunity to increase URM representation in the training to workforce pipeline in psychology (Callahan et al., 2018). Because of the limitations, implications, and disproportionate impact of using the GRE-Q, some science doctoral programs are no longer requiring the GRE for admittance (Langin, 2019; Lau & Miller, 2020). Moreover, even programs that continue to require the GRE are embracing holistic review to consistently and intentionally evaluate application materials to select the students who are most likely to live out the values of the program (Roberts & Ostreko, 2018). Holistic review considers both quantitative and qualitative evidence, including noncognitive and personal attributes (Kent & McCarthy, 2016), for all students not just URMs. Indeed, the College Board has noted that the GRE “does not and cannot measure all the qualities that are important in predicting success” (ETS, 2019). With a de-emphasis in the admissions process, clinical psychology programs can also de-emphasize the GRE-Q on their web pages (Roberts & Ostreko, 2018).

Although holistic review is an important step toward implementing equitable admissions processes, much more needs to be done to recruit qualified URMs to improve recruitment, achievement, and degree receipt in URMs (Maton et al., 2006). For instance, Posselt (2014) has cautioned that simply removing the GRE is not enough because it is still possible to estimate GRE scores from applicants’ training experiences and qualities (e.g., whether they attended an elite undergraduate institution). Therefore, it is advised that programs create standardized metrics, weighting criteria, and rubrics for assessment, with committee members being trained to minimize bias and ensure that selection is based on the values articulated by the committee (Mathur et al., 2019). Additionally, URM representation in existing graduate students and faculty are important predictors of successful recruitment (Muñoz-Dunbar & Stanton, 1999; Rogers & Molina, 2006), including engaging current URM faculty and students in recruitment efforts, representation in coursework (Roberts & Ostreko, 2018)—such as offering a full diversity course, and engaging in diversity issues research (Rogers & Molina, 2006).

Limitations

The present study is not without its limitations. First, Kuncel et al. (2001) and others have argued that individual samples are too restrictive. They purport that meta-analyses and large-scale studies are most appropriate in understanding the correlates of test scores with performance. However, an aim of the current study is to address local concerns that might not be addressed in such studies (e.g., correlates of performance in key courses), while contributing quality data for larger investigations, including meta-analyses, of the correlates of standardized test scores. Second, because our sample included only matriculated students, we have no information about how applicants with even lower GRE-Q scores would have fared in our doctoral program outcomes. Therefore, to the extent that the GRE-Q was a barrier to admission for applicants, our study is unable to examine the full impact for unadmitted students. Future studies can track unadmitted students to examine their access to and success in other doctoral programs. Though rare in the site department, we do not have information on if students retook any courses. Additionally, our dataset was limited to self-reported undergraduate GPA, without the verification of official transcripts. Finally, future studies should use an intersectional approach (Crenshaw, 1989, 1991) to examine how race and gender together impact trajectories, as we were underpowered to do so in the current study given the low numbers of URM groups.

Conclusion

The COVID-19 worldwide pandemic has prompted academic psychologists to reappraise the limitations of the GRE, with serious discussions questioning the legitimacy of its use in admissions at all (Lau & Miller, 2020). In the current study, we demonstrated how the GRE-Q may be used in psychology doctoral admissions in a way that bars access to qualified candidates, disproportionately impacting capable URMs. This inequity in access to psychology doctoral programs is antithetical to efforts for greater diversity and inclusion. The goal of such efforts is to create a workforce of doctoral-level psychology professors, researchers, advisors, supervisors, mentors, and clinicians that can serve an increasingly pluralistic American society. Even though psychology doctoral programs are beginning to abandon the use of the GRE entirely (e.g., Stanford University) or at least minimize its weight in the admissions process (e.g., holistic review), many faculty, departments, and administrators remain resistant to such change. It is true that the multifaceted process of deliberately and consistently implementing changes that result in more diversity and inclusion, including but not limited to the admissions process, requires courage in at least three domains. First, stakeholders, including faculty, must engage in critical self-examination regarding the true state of diversity, equity, and inclusion (DEI) in their respective departments and universities. Second, in the midst of limited resources, leadership, including faculty, must implement and assess necessary changes that systemically promote DEI. Third, faculty and other stakeholders must tolerate the discomfort and guilt that accompanies such self-reflection and honest environmental appraisal, while continuing to implement these changes. By engaging in such institutional courage (Freyd, 2018; Freyd & Smidt, 2019), we can create equal access to psychology doctoral education, while bettering our respective psychology departments and universities, our field, and all the communities within and outside of academia that psychologists serve.

Public Significance Statement.

The use of the Graduate Record Examination Quantitative Score (GRE-Q) in psychology Ph.D. program admissions may interfere with the field’s efforts to create a racially and ethnically diverse workforce that can meet the educational, training, and intervention needs of diverse populations. In a study of psychology Ph.D. students in a Carnegie-classified Highest Research Activity (R1) large Midwestern university, we found that strict guidelines that bar admission into the psychology doctoral program for students with low GRE-Q scores would have disproportionately impacted underrepresented minorities (URMs), resulting in 44% being barred admission versus only 17% of their White/Asian/Pacific Islander American counterparts. With the COVID-19 worldwide pandemic sparking reappraisal of the GRE, practical implications include introducing holistic review protocols into the admissions process, while educating faculty on how heavy emphasis on the GRE-Q contributes to inequitable exclusion of capable URMs.

Acknowledgments

We would like to thank Ambika Mathur for her support of holistic review and Alia Allen and Victoria Guster-Brown for their help with data collection.

Biography

Jennifer M. Gómez is an assistant professor in the Department of Psychology and Merrill Palmer Skillman Institute for Child & Family Development (MPSI) at Wayne State University. She received her doctorate from University of Oregon in 2017 and completed a clinical internship at the Charleston Consortium Psychology Internship Training Program. She is a guest editor of Journal of Trauma & Dissociation and serves on multiple editorial boards. Her research, including over 60 peer-reviewed articles, book chapters, and pieces for the general public, uses cultural betrayal trauma theory to examine violence victimization in marginalized youth, young adults, and elders. Her professional interests are related to cultural competency/humility, multiculturalism, and relational cultural therapy. She is a National Academy of Sciences Kavli Fellow (2019) Ford Fellow (2015–2016; 2018–2019), MCUAAAR Scientist (2020–21), Board Member and Cultural Liaison of the Metro Detroit Association of Black Psychologists, Board Member and Chair of the Research Advisory Committee of the Center for Institutional Courage, and Member of the External Advisory Committee for the Campus, Climate, & Culture Initative (C3I) at Dartmouth College.

Annmarie Caño is Dean of the College of Arts and Sciences and Professor of Psychology at Gonzaga University in Spokane, WA (USA). She earned her master’s and doctoral degrees in psychology from Stony Brook University and her bachelor’s degree in psychology from Princeton University. A Fellow of the American Psychological Association in two divisions (Society for Health Psychology and Society for Couple and Family Psychology), Caño has researched emotion regulation and intimacy in couples facing chronic illness as well as diversity and inclusion in higher education. She has over 70 publications and has served as principal investigator on four National Institutes of Health grants and coprincipal investigator on a National Science Foundation grant aimed at diversifying the professoriate. Caño also curates and edits public scholarship as an editorial board member of the National Center for Institutional Diversity’s Spark Magazine.

Boris B. Baltes is Associate Provost for Faculty Affairs and Professor of Psychology at Wayne State University (WSU). Before joining the Provost’s office, he served as Chair of the Department of Psychology at WSU. His major research interests include examining biases in performance appraisal, age and workplace issues, as well as work–family conflict/balance. His work, including over 70 publications and chapters, has appeared in many journals including the Journal of Applied Psychology, Organizational Behavior and Human Decision Processes, Journal of Management, and the Journal of Organizational Behavior. He is a guest editor for the Journal of Organizational Behavior and is on the editorial board of various journals. He is also a fellow of the Society for Industrial/Organizational Psychology (a division of the American Psychological Association).

References

  1. American Psychological Association (APA). (2003). Guidelines on multicultural education, training, research, practice, and organizational change for psychologists. American Psychologist, 58, 377–402. [DOI] [PubMed] [Google Scholar]
  2. American Psychological Association (APA). (2017). Multicultural guidelines: An ecological approach to context, identity, and intersectionality. http://www.apa.org/about/policy/multicultural-guidelines.pdf [DOI] [PubMed]
  3. Andre T, & Hegland S (1998). Range restriction, outliers, and the use of the Graduate Record Examination to predict graduate school performance. American Psychologist, 53(5), 574–575. 10.1037/0003-066X.53.5.574 [DOI] [Google Scholar]
  4. Association of American Colleges and Universities. (2002). Greater expectations: A new vision of learning as a nation goes to college.
  5. Au W (2016). Meritocracy 2.0: High-stakes, standardized testing as a racial project of neoliberal multiculturalism. Educational Policy, 30, 39–62. [Google Scholar]
  6. Boyd B, Caraway J, & Flores Niemann N (2017). Surviving and thriving in academia: A guide for members of marginalized groups. American Psychological Association. [Google Scholar]
  7. Callahan JL, Smotherman JM, Dziurzynski KE, Love PK, Kilmer ED, Niemann YF, & Ruggero CJ (2018). Diversity in the professional psychology training-to-workforce pipeline: Results from doctoral psychology student population data. Training and Education in Professional Psychology, 12, 273–285. 10.1037/tep0000203 [DOI] [Google Scholar]
  8. Crenshaw K (1989). Demarginalizing the intersection of race and sex: A Black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. University of Chicago Legal Forum, 140, 139–167. [Google Scholar]
  9. Crenshaw K (1991). Mapping the margins: Intersectionality, identity politics, and violence against women of color. Stanford Law Review, 43, 1241–1299. [Google Scholar]
  10. Croizet JC (2008). The pernicious relationship between merit assessment and discrimination in education. In Adams G, Biernat M, Branscombe NR, Crandall CS, & Wrightsman LS (Eds.), Decade of behavior. Commemorating brown: The social psychology of racism and discrimination (pp. 153–172). American Psychological Association; 10.1037/11681-009 [DOI] [Google Scholar]
  11. Croizet JC, & Dutrévis M (2004). Socioeconomic status and intelligence: Why test scores do not equal merit. Journal of Poverty, 8, 91–107. [Google Scholar]
  12. Educational Testing Service (ETS). (2019). Graduate Record Examination (GRE): Guide to the use of scores. https://www.ets.org/s/gre/pdf/gre_guide.pdf
  13. Freyd JJ (2018, January 11). When sexual assault victims speak out, their institutions often betray them. The Conversation. https://theconversation.com/when-sexual-assault-victims-speak-out-their-institutions-often-betray-them-87050 [Google Scholar]
  14. Freyd JJ, & Smidt AM (2019). So you want to address sexual harassment and assault in your organization? Training is not enough; Education is necessary. Journal of Trauma & Dissociation, 20, 489–494. [DOI] [PubMed] [Google Scholar]
  15. Gómez JM (2015). Microaggressions and the enduring mental health disparity: Black Americans at risk for institutional betrayal. Journal of Black Psychology, 41, 121–143. 10.1177/0095798413514608 [DOI] [Google Scholar]
  16. Hall GCN (2006). Diversity in clinical psychology. Clinical Psychology: Science and Practice, 13, 258–262. [Google Scholar]
  17. Hall GCN, Yip T, & Zárate MA (2016). On becoming multicultural in a monocultural research world: A conceptual approach to studying ethnocultural diversity. American Psychologist, 71, 40–51. 10.1037/a0039734 [DOI] [PubMed] [Google Scholar]
  18. Kalat JW, & Matlin MW (2000). The GRE Psychology Test: A useful but poorly understood test. Teaching of Psychology, 27, 24–27. 10.1207/S15328023TOP2701_4 [DOI] [Google Scholar]
  19. Kalsbeek D, Sandlin M, & Sedlacek W (2013). Employing noncognitive variables to improve admissions, and increase student diversity and retention. Strategic Enrollment Management Quarterly, 1, 132–150. [Google Scholar]
  20. Kent JD, & McCarthy MT (2016). Holistic review in graduate admissions. Council of Graduate Schools. Council of Graduate Schools. [Google Scholar]
  21. Kohout JL, Pate WE, & Maton KI (2014). An updated profile of ethnic minority psychology: A pipeline perspective. In Leong FTL, Comas-Díaz L, Nagayama Hall GC, McLoyd VC, & Trimble JE (Eds.), APA handbook of multicultural psychology: Volume 1. Theory and research. American Psychological Association. [Google Scholar]
  22. Kuncel N, Hezlett S, & Ones DS (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127, 162–181. [DOI] [PubMed] [Google Scholar]
  23. Kuncel NR, & Hezlett SA (2007). Standardized tests predict graduate students’ success. Science, 315, 1080–1081. [DOI] [PubMed] [Google Scholar]
  24. Kuncel NR, Wee S, Serafin L, & Hezlett SA (2010). The validity of the Graduate Record Examination for master’s and doctoral programs: A meta-analytic investigation. Educational and Psychological Measurement, 70, 340–352. [Google Scholar]
  25. Langin K (2019, May 29). A wave of graduate programs drops the GRE application requirement. Science Magazine. https://www.sciencemag.org/careers/2019/05/wave-graduate-programs-drop-gre-application-requirement [Google Scholar]
  26. Lau A, & Miller GA (2020). Do clinical science programs need GRE scores in admissions this year, or ever? Psychological Clinical Science Accreditation System Newsletter. https://www.pcsasnews.org/doweneedthegres [Google Scholar]
  27. Mathur A, Cano A, Dickson MW, Matherly LH, Maun C, & Neale AV (2019). Portfolio review in graduate admissions: Outcomes of a pilot program. Strategic Enrollment Management Quarterly, 7(1), 7–24. [Google Scholar]
  28. Maton KI, Kohout JL, Wicherski M, Leary GE, & Vinokurov A (2006). Minority students of color and the psychology graduate pipeline: Disquieting and encouraging trends, 1989–2003. American Psychologist, 61, 117–131. 10.1037/0003-066X.61.2.117 [DOI] [PubMed] [Google Scholar]
  29. Miller C, & Stassun KA (2014). A test that fails. Nature, 510, 303–304. [Google Scholar]
  30. Morrison T, & Morrison M (1995). A meta-analytic assessment of the predictive validity of the quantitative and verbal components of the Graduate Record Examination with graduate grade point averages representing the criterion of the graduate success. Educational and Psychological Measurement, 55, 309–316. [Google Scholar]
  31. Muñoz-Dunbar R, & Stanton AL (1999). Ethnic diversity in clinical psychology: Recruitment and admission practices among doctoral programs. Teaching of Psychology, 26, 259–263. [Google Scholar]
  32. National Science Foundation, National Center for Science and Engineering Statistics. (2017). Women, minorities, and persons with disabilities in science and engineering: 2017 [Special Report NSF 17–310, Arlington, VA]. nsf.gov/statistics/wmpd/.
  33. Pacheco W, Noel R, Porter J, & Appleyard C (2015). Beyond the GRE: Using a composite score to predict the success of Puerto Rican students in a biomedical PhD program. CBE Life Sciences Education, 14, 1–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Petersen SL, Erenrich ES, Levine DL, Vigoreaux J, & Gile K (2018). Multi- institutional study of GRE scores as predictors of STEM PhD degree completion: GRE gets a low mark. PLOS ONE, 13, Article 1. 10.1371/journal.pone.0206570 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Posselt JR (2014). Toward inclusive excellence in graduate education: Constructing merit and diversity in PhD admissions. American Journal of Education, 120, 481–514. [Google Scholar]
  36. Richardson JT (2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review, 6, 135–147. [Google Scholar]
  37. Roberts MC, & Ostreko A (2018). GREs, public posting, and holistic admissions for diversity in professional psychology: Commentary on Callahan et al. (2018). Training and Education in Professional Psychology, 12, 286–290. 10.1037/tep0000219 [DOI] [Google Scholar]
  38. Rogers MR, & Molina LE (2006). Exemplary efforts in psychology to recruit and retain graduate students of color. American Psychologist, 61, 143–156. 10.1037/0003-066X.61.2.143 [DOI] [PubMed] [Google Scholar]
  39. Sampson C, & Boyer PT (2001). GRE scores as predictors of minority students’ success in graduate study: An argument for change. College Student Journal, 35, 271–279. [Google Scholar]
  40. Sternberg R, & Williams W (1997). Does the Graduate Record Examination predict meaningful success in the graduate training of psychologists? American Psychologist, 52, 630–641. [DOI] [PubMed] [Google Scholar]
  41. Stewart S, & Haynes C (2016). An alternative approach to standardized testing: A model that promotes racial equity and college access. Journal of Critical Scholarship on Higher Education and Student Affairs, 2, 122–136. [Google Scholar]
  42. Vasquez MJT, & Jones JM (2006). Increasing the number of psychologists of color: Public policy issues for affirmative diversity. American Psychologist, 61, 132–142. [DOI] [PubMed] [Google Scholar]

RESOURCES