Who benefits from elite colleges’ decreased reliance on high-stakes standardized tests? Evidence from a quasi-field experiment

Abstract

Despite the increasingly popular perception that elite colleges’ reliance on high-stakes standardized tests to select students exacerbates social inequality, researchers have not demonstrated that a reduced emphasis on such tests actually alters individuals’ admission outcomes. Using all applications submitted in 2022 to National Taiwan University, the most prestigious university in Taiwan, we show who among the applicants would have been admitted under the country’s prior college admission system, which was entirely based on high-stakes standardized test scores, and contrast them with those actually admitted in the current system, which emphasizes holistic evaluations and considers many other criteria. The results indicate that for the majority of students the admission outcomes would be unchanged. Even among those whose outcomes would differ, class backgrounds are not clearly related to the direction of change in the outcome. Rather than students of upper- or lower-class backgrounds, women are the unequivocal beneficiary from the decreased reliance on high-stakes standardized tests. The analysis suggests that the better high school grades and more comprehensive application materials submitted by women applicants contribute to a considerable part of the female advantage in a more holistic admission system. Evidence from this quasi-field experiment has implications for how changes in the structural elements of the educational system affect elite college access.

Researchers on social stratification and mobility have long been interested in how structural elements of the educational system contributes to inequality (Turner, 1960; Werfhorst & Mijs, 2010). Although individuals from upper class or with other favorable ascriptive characteristics (e.g., of the dominant racial group) tend to have higher educational attainment and enhanced access to elite colleges across the world (Breen et al., 2009; Jerrim et al., 2015; Lee, 2013), the ways in which the educational system screens and sorts individuals can potentially moderate the effects of ascribed characteristics on educational outcomes (Pfeffer, 2008). A system featuring early tracking, for example, is shown to strengthen the link between parental socioeconomic backgrounds and educational achievements and widen overall disparities in student performances (Hanushek & Wößmann, 2006; Van de Werfhorst, 2018, 2019). Conversely, a high level of standardization in curriculum, resources, and student evaluation criteria is thought to reduce inequality in the quality of education received by students across different geographic areas and family conditions, thereby decreasing the influence of socioeconomic backgrounds (Werfhorst & Mijs, 2010).

Among elements of the schooling system that can potentially amplify social inequality, the use of high-stakes standardized tests—that is, tests with substantial impacts on the transition to the next educational level, such as Scholastic Assessment Test (SAT) and American College Testing (ACT)—in college admissions has received especially great attention in recent decades (Furuta et al., 2020; Grodsky et al., 2008). Because students from affluent families are more able to pay for tutoring and other services to boost their performance at high-stakes standardized tests (Buchmann et al., 2010), critics maintain that relying on these tests to assess student quality facilitates unequal access to highly competitive colleges by socioeconomic background (Alon & Tienda, 2007; Au, 2010; Dixon-Román et al., 2013; Soares, 2007). High-stakes standardized tests are therefore argued to help elites reproduce their advantages.

Not all researchers agree that the reliance on high-stakes standardized tests by itself causes or exacerbates social inequality. Because children from privileged backgrounds receive greater parental investment and more opportunities to learn from an early age (Lareau, 2011), they may very well develop higher cognitive skills by the time they take high-stakes standardized tests for college (Grodsky et al., 2008). In this sense, standardized test scores merely reflect existing patterns of social stratification; they have little independent effect on inequality. Even more, some contend that using standardized tests in college admissions can potentially reduce the advantages of students from upper-class families (Kariya, 2020). To differentiate college applicants, evaluators inevitably need to weigh other qualifications more when they focus less on standardized test scores. Class-based inequality could manifest more in extracurricular activities or applicants’ essay writing than in test performance (Alvero et al., 2021; Snellman et al., 2015; Tompsett & Knoester, 2023).

To shed light on the debate on high-stakes standardized tests’ role in educational inequality, this study uses a unique data set and a quasi-field experiment to examine who, or whether anyone, truly benefits from the reliance on such tests in elite college admissions. We focus on elite college attendance because it is tied to the attainment of top job positions and high income (Jia & Li, 2021; Zimmerman, 2019). Our data contain detailed information of all applications received by National Taiwan University (NTU), the most prestigious and desirable college in Taiwan (Luoh, 2002), in 2022. Similar to other East Asian countries, Taiwan has undergone a series of educational reforms since the early 2000s, moving away from the college admission system in which students were selected entirely based on their performance at the once-a-year national entrance examination (Lee & Park, 2014; Lin et al., 2014; Luoh, 2018). Currently, most students seek college admission through direct applications, which still require them to take a once-a-year standardized exam—namely, the General Scholastic Ability Test (GSAT)—but the exam performance is only part of the evaluation. The eventual admission decisions are also based on evaluations of high school records, written materials, and additional interviews and/or major-specific tests required by the department applicants aspire to enter.¹

Because the seats available for each NTU department have always been predetermined by the Ministry of Education in Taiwan and independent of the number or quality of applications received, we can use the number of seats and applicants’ GSAT scores to determine who would have been accepted into NTU under the previous admission system, which is based on students’ standardized test scores only. We can then contrast those would have been admitted with those admitted in the current system, which also considers applicants’ high school records, essays, extracurricular merits, previous academic works, and results of the interview and/or additional tests in the second round. Because the change in admission criteria resembles a shift in the treatment in an experiment, our data enable us to create a quasi-experiment and uncover who among the aspiring students would experience alterations in the admission outcome with the change in treatment.

Although our analysis is based on one university in Taiwan, the national context, which we discuss in a later section, makes the NTU applicants likely representative of the highest-performing students in a given cohort, which are similar to applicants to the most prestigious universities in the world. Given this, and given that our quasiexperiment accounts for all but the treatment, we argue that the NTU’s experience could broadly inform the impact of selection criteria on elite college admission results, at least for East Asian countries, which share many features of the school system (Marginson, 2011; Yu, 2009). This study thus contributes to our knowledge of how structural designs of the educational system affect inequality in elite college access. Unlike prior research using college reports of student compositions to assess the effect of admission criteria (Belasco et al., 2015; Bennett, 2022; Luoh, 2018), our analysis shows how the admission outcomes would vary for a fixed group of applicants, thus avoiding the results being confounded by differences in applicant qualities and socioeconomic composition over time. As our data contain more detailed applicant information than previous studies, we also expand the literature by more comprehensively investigating how applicant characteristics, including their family wealth, neighborhood resources, and high school ranking, are linked to a change in the admission outcome with shifts in selection criteria. Because Taiwan’s college admission reforms that have weakened the importance of high-stakes standardized tests are similar to those occurring across East Asia (Kariya, 2020; Lee & Park, 2014), our analysis of who benefits from these reforms have especially important implications for other countries in that region.

1. High-stakes standardized tests and elite college admissions

Scholars of social stratification have long been interested in the access to elite colleges (Lee, 2013). Despite the worldwide expansion of higher education, students of highly selective universities continue to disproportionately come from privileged families and well-off neighborhoods (Alon & Tienda, 2007; Chetty et al., 2023; Ding et al., 2021; Luoh, 2018), leading to the argument that elite colleges ultimately serve to reproduce upper-class advantages (Zimmerman, 2019). Until recently, nearly all U.S. elite universities required applicants to take standardized tests such as SAT and ACT, making the performance in these tests a decisive factor in elite college enrollment. In East Asia, which traditionally relied heavily on tests to evaluate students, stellar performance in high-stakes standardized tests (i.e., entrance exams) is even more critical to the admission into elite colleges (Ono, 2007; Yu, 2009).

Because students from families with higher socioeconomic status tend to perform better in high-stakes standardized tests, many argue that such tests serve as a mechanism of social reproduction (Alon & Tienda, 2007; Au, 2010; Dixon-Román et al., 2013; Lee, 2013; Soares, 2007). Proponents of this perspective follow Bowles and Gintis’ (1976) view that academic sorting ensures parental advantages and disadvantages to be passed to children, while considering high-stakes standardized tests as uniquely unmeritocratic (Alon & Tienda, 2007). This is because upper-class children can use costly tutoring or test-preparation services, which underprivileged children cannot afford, to gain advantage in test performance, thereby enhancing their chances of entering elite colleges (Buchmann et al., 2010). In this sense, high-stakes standardized testing ultimately helps exclude disadvantaged children from elite positions in society and maintain the status quo (Furuta et al., 2020).

Although much research demonstrates test performance gaps along various dimensions of social inequality (Dixon-Román et al., 2013; Everson & Millsap, 2004), these gaps could equally support an alternative proposition: Results of high-stakes standardized tests are merely reflection of social inequality; the reliance on these tests by itself does not add advantages or disadvantages to children of differing ascribed characteristics. With greater parental resources devoted to learning opportunities from birth (Lareau, 2011), children of well-off parents may develop better cognitive and noncognitive skills, which could boost their test performance regardless of whether they use test-preparation services (Grodsky et al., 2008). In this scenario, making test scores optional or weigh less would not affect the admission outcomes of students from varying backgrounds.

Existing research also suggests a third theoretical possibility: High-stakes standardized tests constitute an equalizer (Kariya, 2020). This proposition is derived from the fact that reduced importance of high-stakes standardized tests in college admissions must coincide with increased weights on other criteria, such as applicants’ written statements and extracurricular achievements. Lower-class children may be even more disadvantaged in non-test performances, as they tend to engage in fewer extracurricular activities and have less worldly experience to write about than their more priviledged peers (Lareau, 2011; Snellman et al., 2015; Tompsett & Knoester, 2023). Indeed, a recent study finds that college applicants’ household income is more closely correlated with their essay content and style than SAT scores (Alvero et al., 2021). Another shows that upper-class children’s advantage in nonacademic credentials gives them an edge over middle-class children in elite college admissions (Chetty et al., 2023). Prior evidence from East Asia also suggests that standardized testing could be an equalizer. Before the recent reforms that shifted to incorporate other evaluation criteria (Kariya, 2020; Lee & Park, 2014; Lin et al., 2014; Luoh, 2018), the combination of highly standardized curriculums and frequent use of standardized tests to assess students’ learning in East Asia has been praised to reduce class differences in cognitive development, resulting in relatively lower inequality in college access by socioeconomic status (Broaded, 1997; Ojima & von Below, 2010).

Because assessments of essays and nonacademic credentials are inevitably subjective, it is also possible that other signals that do not necessarily reflect individuals’ merits become more important as the reliance on high-stakes standardized test scores decreases, making elite college admissions less equitable. For example, in contexts with early educational tracking like Taiwan (Broaded, 1997), the vast majority of elite college students come from a small number of high schools (Luoh, 2002, 2018). The label of coming from a prestigious high school may give students an unfair advantage, which they would not have in high-stakes standardized tests.

Prior researchers have attempted to adjudicate the three competing perspectives discussed above, but their data and methods are rarely conducive to making a causal claim. Alon and Tienda (2007), for instance, use survey data to calculate the associations of SAT scores and high school class rank with college acceptance. With the estimated coefficients, they then simulate racial minorities’ probabilities of being admitted if highly selected colleges weigh the class rank more and the SAT score less than they actually did. Although their simulation indicates that racial minority students would gain seats, the real-world admission results may differ. Realistically, applicants of higher SAT scores are also likely to have better-written essays, transcripts reflecting more advanced learning, and greater extracurricular engagement, none of which are observed in Alone and Tienda’s (2007) analysis. Because the link between SAT scores and college acceptance may largely reflect the close correlation between the scores and other merits leading to elite college admissions, eliminating SAT scores alone may not alter the admission results.

Other studies take advantage of the test-optional movement among U.S. colleges before and since the COVID-19 pandemic to show changes in the percentages of students from racial minority groups and low socioeconomic backgrounds with colleges’ adoption of the policy that allows applicants not to submit standardized test scores (Bennett, 2022; Rosinger et al., 2022; Sweitzer et al., 2018). This research design nevertheless suffers from the possibility that the changes in student composition merely result from increases in minority or underprivileged applicants at test-optional colleges, as such applicants may see the new policy as a signal that the college embraces diversity (Belasco et al., 2015). In addition, shifts in student composition over time could reflect other social forces that enhance disadvantaged students’ educational opportunities. Without knowing who applied and failed each year, it is ultimately impossible to tell how elite college acceptance rates among various social groups have changed over time.

In contrast to existing research, our study resembles a more rigorous field experiment in which we can observe how the reduced emphasis on standardized tests affects the actual admission outcomes for the same group of applicants. Because we have more information about the applicants than most studies, we also extend the literature by addressing whether a weakened emphasis of standardized tests leads other status signals, such as the high school attended, to be more influential in elite college admissions. Unlike research using college reports of student composition to assess the effect of admission criteria (Belasco et al., 2015; Bennett, 2022; Luoh, 2002), we can measure the full spectrum of applicants’ socioeconomic status—not just whether they are especially financially disadvantaged (e.g., receiving Pell grants). Thus, we can test not just whether the reliance on standardized tests hurts the very disadvantaged but also whether the decrease in this reliance benefits students from wealthy families and well-off neighborhoods.

2. College admission system in Taiwan

Similar to other East Asian societies, Taiwan historically relied heavily on standardized tests to evaluate students (Broaded, 1997; Yu, 2009). After nine years of mandatory education, students must pass entrance examinations to enter public high schools. For advancing to college, passing the national college entrance examination, which was held once a year and covered all materials taught in high school, used to be the only way. Partly to eliminate the “examination hell” East Asian adolescents are known for undergoing (Lee & Larson, 2000; Ono, 2007) and partly to cultivate more balanced intellectual development (Lin et al., 2014), Taiwan began to move away from using only the entrance examination to decide about college admissions since the 1990s, with a drastic departure from that system in 2002 (Luoh, 2018).

The current admission system allows high school students to have multiple chances of gaining college admissions in their senior year, through three distinct channels. Like other East Asian countries, Taiwanese students decide about their majors before entering college (Brinton, 1993); hence all different channels is about obtaining a seat in a given department within a certain university, not within a university. First, upon taking the GSAT, which is offered in January of the senior year, students with excellent high school grades and GSAT scores could be recommended by their high schools to university departments. Because these students are not guaranteed to be recommended to their favorite department or university, they may decline the offered spot and opt for the second channel: direct applications.

For students aiming for elite colleges, direct applications give them much greater control. It is important to note that much of Taiwan’s educational competition today is about entering elite colleges, as higher education enrollment is nearly universal for the current generation (Marginson, 2011). What makes this competition especially intense is that the ranking of colleges, with NTU on top (Luoh, 2002), is virtually unequivocal, owing to the history of using only national entrance exam scores to determine admissions.² With the direct application channel, students aspiring to attend elite universities simply send in their GSAT scores and dossiers, which contain their high school transcripts, essays and statements, reports of extracurricular merits, and academic work they have done before (e.g., a term paper), to the specific university departments they wish to enter. The departments select students into the second round entirely based on the GSAT scores. Most departments require applicants in the second round to undergo additional interviews or department-specific tests. All departments complete the second round and make admission decisions by April. Students can then decide to accept an admission offer, or, if they do not receive a satisfactory offer or any offer, they can still take the national college entrance exam in July and obtain a spot based on their scores from that exam. This final route is essentially identical to the way students used to be admitted to university departments in Taiwan. Students who fail to secure a seat in the university or department they desire or fail to be admitted anywhere may spend an extra year preparing for the GSAT or application materials and go through the process again the following year.

Among all students entering Taiwan’s colleges in 2021, 17% of them were admitted through high school recommendations, 57% through direct applications, and 26% through the national entrance exam held in July (Department of Education of Taiwan, 2023). Because direct applications constitute the main channel via which colleges recruit students, and because it incorporates the most holistic evaluations, our study focuses on admissions through this channel.

3. Methods

3.1. Data

Data for this study are from the application records of NTU in 2022.³ Altogether, NTU’s departments received 12,526 applications from 6769 students, with some applicants applying to multiple departments. Although not all college-bound students applied to NTU, the selection to do so was unlikely related to socioeconomic backgrounds. This is because NTU is a public university with rather low tuition and fees (around $1000 per semester).⁴ Perhaps more important, NTU is widely considered as the single most prestigious university in Taiwan, a society where students from all class backgrounds greatly value the acceptance to high-ranking universities (Luoh, 2002; Yee, 1989). More likely, the selection to apply to NTU is based on the competitiveness of students’ GSAT scores, as the departments make a first-round cut, typically selecting twice or three times as many applicants as their available seats, based on these scores.⁵ Only in the second round, each department’s admission committee reviews and scores the submitted application dossiers, which contain applicants’ high school transcripts, essays, extracurricular records, and self-selected academic works from high school (e.g., term papers, written creative works, science experiment reports). Nearly all departments require the second-round candidates to also be interviewed and/or take additional, department-specific tests (e.g., writing a computer program or a mathematical proof). The final admission is based on the ranking of a composite score made up by the scores of the dossier, GSAT, interview, and/or department-specific test (s). The formula for this composite score is predetermined and varies by department, with the average weight of the GSAT score around 30%.

Because only those who have passed the first round have a chance to be admitted, we selected the applications in the second round for the analysis. In 2022, 4312 applications entered and completed the second round (with 904 withdrawn upon passing the first round)⁶. We excluded 27 applications for a multidisciplinary program that did not exist before the educational reforms and 7 applications from students attending high schools overseas (for whom we could not measure their high school’s ranking), resulting in 4278 observations in the analytic sample.

3.2. Analytic strategy and measurement

To show for whom the outcome changes with the shift in admission criteria, we used each department’s published first-round GSAT-based selection criteria and available seats to identify the applications that would have been accepted if the GSAT score is the only selection criterion. The GSAT-only selection method resembles the one prior to Taiwan’s reforms of college admission. All departments make the first cut of the applications based on GSAT scores, but they can decide which subject tests of the GSAT they would consider and how they would weigh the scores of various subjects, given that the GSAT includes multiple subject tests (Chinese, English, math, advanced math, natural science, and social science)⁷ For example, a department can make the first-round selection based on the sum of all test scores or the sum of just a few subjects’ scores. It can also weigh, for example, the math or English test score 50% more than others if the faculty members believe the ability in math or English is more crucial to the future success in the department. Typically, the departments in science, technology, engineering, and math fields compare students based on their scores of advanced math, natural science, and English, with some also considering Chinese, and the other departments do based on Chinese, English, social science, and math scores. The department’s selection criteria are predetermined and made available to the public, so students can assess their chances of making it to the second round based on their GSAT percentile scores and the department’s selection criteria. They are likely to decide to which department to apply based on this assessment.

In the cases two or more applicants with the same scores were competing for the very last seat available in the department, we broke the tie using the department’s published tie-breaking methods (e.g., comparing the test score of a subject that was not included in the original ranking criteria). In the extremely rare cases where we could not break ties even using all the department’s tie-breaking criteria, we selected those with the highest total GSAT scores across all subjects—not all applicants take the tests on all available subjects—based on the rationale that Taiwan’s previous education system generally favored the “well-rounded” students who did well on all subjects. If this criterion still failed, we broke the ties randomly.

We then compared the accepted applications based on GSAT only with those accepted under the current admission system, which we refer to as the “holistic evaluation” system. To show how moving even further away from the standardized-test-only system may affect admission outcomes, we further simulated the admission results with a hypothetical system in which the standardized test performance simply serves as a threshold criterion (i.e., applications must be above each department’s minimum GSAT score to pass the first round), and the admission decisions are solely based on the evaluations of the application dossiers. We then compared the results of this hypothetical system, which is somewhat similar to graduate school application system in the United States, with the actual results under the current system.

The comparisons just described enable us to uncover the “winners,” who experience a positive admission result change, “losers,” who undergo a negative change, and unaffected, who would be rejected or accepted either way, from the shift in admission criteria. Because the unaffected do not contribute to any change in the student composition, we focus on how the winners and losers differ in sociodemographic traits. We first descriptively compared the winners’ characteristics with the losers’ to examine the effect of admission criteria on NTU’s student composition. We then fitted linear probability models to show the characteristics associated with winning instead of losing from the decreased reliance on standardized tests.

The specific characteristics examined in the analysis included applicants’ gender (female vs. male), disadvantaged status, high school ranking, high school type (private vs. public), family wealth, and neighborhood resources. We included gender because research indicates that girls tend to score lower in math tests but have higher grades than boys (Buchmann et al., 2008; Marks, 2008); the changing emphasis on test performance may affect male and female applicants differently. Regarding disadvantaged status, the Taiwanese government considered students who were aboriginal, residing in remote areas (e.g., Kinmen and Matsu Islands), or from households below poverty levels as disadvantaged. We created a dummy using the government definitions.

Because the admission committee may evaluate the dossiers of students from high-ranking schools more favorably, an indicator for applicants’ high school’s performance ranking was created for our analysis. We calculated the proportion of students from the applicant’s high school who passed NTU’s first-round screening in the observed year. We then ranked the high schools in our data from the highest to lowest proportions. The performance ranking of high schools we created is likely consistent with the general public’s perception of high school ranking in Taiwan. Because Taiwanese students must pass regional entrance exams to enter public high schools, and because each year’s entry score for every public high school is publicly available, there is a high consensus about the ranking among public high schools. Students also tend to choose schools according to the previous year’s ranking. This sorting process ensures that highly ranked Taiwanese high schools, on the whole, do better in placing students into elite colleges (Luoh, 2002, 2018). We further distinguished public from private high schools because the latter traditionally were considered as less prestigious, despite their students’ enhanced GSAT performance in recent years.

One key question we address is how change in admission criteria alters the elite college access of students of different socioeconomic backgrounds. NTU’s application records, unfortunately, do not contain parental information. We therefore linked the applicants’ permanent home address to external data sources to create two indicators of socioeconomic backgrounds. First, to measure applicants’ family wealth, we used the address of their family’s permanent home to locate the last 20 residential real estate sales on the same street or town (in the case of rural areas) from public records. We then calculated the median sale price per ping (about 3.31 square meters), the unit used to measure housing size in Taiwan, and used it to approximate applicants’ family home worth. Because Taiwan’s urban residents largely live in condominiums or apartments (and the rural population is small), the variation in housing quality and price within the same street is relatively small. Given Taiwan’s high homeownership rate (>85%), and given that most people’s main investment is their home (Chen, 2020), the home worth is a reasonable proxy for family wealth. To reduce the skewedness of the distribution, we took the natural log of the median housing price per ping and used the transformed measure in the models.

Second, because students from differing socioeconomic backgrounds may face unequal neighborhood resources, which could affect their academic success (Lei & Yu, 2021), we constructed an indicator of applicants’ local educational resources using the rate of college attainment in their local communities. Specifically, we linked the applicant’s permanent address to census data to calculate the proportion of people between 25–29 years old with bachelor’s degrees in the applicant’s zip code area. We calculated the proportion within a specific age group to avoid confounding differences in residents’ age with disparities in educational resources across neighborhoods, as Taiwan’s rate of university attainment varies drastically by birth cohort.

In addition, we differentiated applicants based on whether they graduated from high school before 2022; those who did likely had undergone the college application process at least once before. In East Asia, students who fail to be admitted to the departments or universities they wish to enter often decide to spend another year preparing and apply again the next year (Ono, 2007; Yu, 2009). Such applicants could face the stigma of having failed before.

Applicants’ high school grade average (on the scale of 0–100) and dossier length were further introduced to show how the information in the dossier explains applicants’ advantages or disadvantages with the shift in admission criteria. Because the admission requirements set no page limit for the written materials submitted, the dossier’s number of pages varies widely in our data. Reviewers of the application materials may see the pages included in the dossier as an indicator of applicants’ effort and rate them accordingly.

Because some applicants sent applications to multiple departments, we clustered by applicant and estimated robust standard errors in the models. We also included dummies for the number of applications the applicant sent to NTU across departments and for the applicant’s number of applications in NTU’s second round, respectively, in additional models. These dummies help control for unmeasured characteristics that may affect admission outcomes, such as the applicant being an overachiever or having less tailored, hence subpar, application dossiers.

We provide descriptive statistics of the aforementioned variables in Table 1. Because 9.5% of the applications in NTU’s second round do not have values for high school grade average, multiple imputation with 20 sets of imputations were used to fill in the grades. We should note that those applications still have the high school transcripts in their dossiers; it is just that NTU’s database only has the grade information when the transcripts were directly sent by applicants’ high schools and electronically imported into the data file. Nearly all applications missing high school grades are from applicants who graduated before 2022 and could not have their former high schools send the transcripts directly.

Table 1.

Characteristics of analytic sample.

	Percentage/mean
Applicant female	42.3
Applicant from disadvantaged background	5.4
Applicant earlier-year graduate	9.3
Applicant family home worth (in 10,000 NTD) per ping (0.7-187.4)	42.3 (28.3)
Applicant local educational resource index (0-1)a	0.7 (0.5)
Applicant high school ranking (1-136)b	15.9 (19.7)
Applicant attending private high school	18.9
Applicant high school grade average (0-100)c	83.4 (5.5)
Dossier number of pages (0-563)d	117.9 (67.9)
Applicant’s number of applications submitted to NTU (1-5)	2.9 (1.4)
Applicant’s number of applications in NTU’s 2nd round (1-5)	2.2 (12)

Note: The unit of analysis is applications. The values followed by parentheses are mean values, while the others are in percentage. The numbers in parentheses after the mean values are standard deviations. The numbers in parentheses following each variable name indicate the range of the values. N = 4278

^a.The index is created based on the proportion of people 25-29 with college degrees in the local area.

^b.The ranks were coded continuously even when there are ties at a given rank.

^c.The value is calculated with only those with high school grade average.

^d.One application did not complete the submission of the dossier, thus with the page number being 0. This application would have been rejected either way and whether to include it in the sample has no effect on our results. The lowest page number for the dossiers submitted is 2.

4. Results

Table 2 shows the numbers of applications accepted and rejected based on the GSAT scores only and in the current (holistic) system. A quarter of the applications would be accepted either way (1079/4278 = 25%), and nearly a half of the applications would be rejected either way (1899/4278 = 47%). The remaining applications would have had different outcomes if the admission system had not changed.

Table 2.

Admission results based on current and former criteria.

Standardized test only	Holistic evaluation
		Admitted	Rejected	Total
	Admitted	1079	650	1729
	Rejected	650	1899	2549
	Total	1729	2549	4278

Table 3 compares the characteristics of applicants who would have been accepted but are now rejected (losers) with those of applicants who would have been rejected but are now accepted (winners). Interestingly, the two groups are not significantly different in family home worth, a proxy for parental wealth, local educational resources, or the likelihood to have the government-designated disadvantaged status, suggesting that the decreased reliance on standardized tests hardly has any effect on the socioeconomic status of NTU’s student body. The change in admission criteria also does not affect the share of students from high-ranking high schools, as the averages of high school ranking between the winner and loser groups are very similar. The winners and losers nevertheless differ in some ways. Most notably, the share of female applicants is significantly larger among the winners; thus, the shift to a holistic evaluation system has significantly raised the number of female applicants admitted into NTU. In addition, the winners are less likely to have graduated from high school in earlier years than the losers, and they are less likely to be from private high schools.

Table 3.

Descriptive statistics of winners and losers with change in admission methods.

	Losers		Winners		Difference test
	Mean	S.D.	Mean	S.D.
Female	0.35	0.48	0.52	0.50	**
Disadvantaged background	0.01	0.10	0.00	0.07
Earlier-year graduate	0.13	0.34	0.06	0.24	**
Family home worth (in NT $10,000)	44.50	28.91	43.86	28.59
Local educational resources	0.73	0.05	0.72	0.05
High school ranking	14.22	17.17	14.30	17.40
Private school attendance	0.24	0.43	0.14	0.35	**
N	650		650

Note: The unit of analysis is applications.

^**p < 0.01;

^*p < 0.05;

^†p < 0.1 (two-tailed t-test)

Table 4 presents a series of linear probability models with increasing complexity to show factors associated with winning instead of losing from the shift to the holistic evaluation system. Models 1–2 confirm again that women are the main beneficiary of the reduced reliance on standardized test scores, and that accounting for potential gender differences in the high school attended and socioeconomic backgrounds does not alter women’s advantage. Including the high school grade average and length of dossier, as in Model 3 and 4, reduces the coefficient for being female by 40% ([0.167–0.100]/0.167 =0.40). This finding suggests that a substantial part of women’s gained advantage with the change to a more holistic evaluation system is attributable to their better high school grades and more comprehensive application dossiers. Adjusting for men’s and women’s different college application behavior—in terms of the number of NTU departments they applied to or the number of second-round challenges they were willing to undertake—further decreases women’s advantage somewhat, but being female is still significantly associated with winning instead of losing from the change in admission criteria.

Table 4.

Linear probability models on factors associated with winning instead of losing from the shift to a holistic admission system.

	Model 1	Model 2	Model 3	Model 4	Model 5	Model 6
Female	0.167 ** (0.031)	0.166 ** (0.031)	0.122 ** (0.030)	0.100 ** (0.030)	0.093 ** (0.030)	0.079 ** (0.030)
Disadvantaged background	−0.231 (0.146)	−0.267† (0.148)	−0.195 (0.155)	−0.198 (0.158)	−0.250† (0.151)	−0.202 (0.137)
Earlier-year graduate	−0.189 ** (0.047)	−0.204 ** (0.048)	−0.170 ** (0.047)	−0.090† (0.051)	−0.107 * (0.050)	−0.117 * (0.049)
Log family home worth		0.026 (0.030)	0.049† (0.029)	0.054† (0.029)	0.075 ** (0.029)	0.078 ** (0.029)
Local educational resources		−0.747 * (0.376)	−0.913 * (0.364)	−0.948 ** (0.360)	−1.104 ** (0.356)	−1.050 ** (0.351)
High school ranking		0.001 (0.001)	0.001 (0.001)	0.001 (0.001)	0.000 (0.001)	0.000 (0.001)
Private high school attendance		−0.167 ** (0.041)	−0.208 ** (0.039)	−0.210 ** (0.039)	−0.218 ** (0.039)	−0.196 ** (0.038)
High school grade average			0.026 ** (0.003)	0.023 ** (0.003)	0.023 ** (0.003)	0.024 ** (0.003)
Dossier number of pages				0.001 ** (0.000)	0.001 ** (0.000)	0.001 ** (0.000)
Applicant no. of applications (ref. One): Two					−0.093 * (0.046)
Three					−0.058 (0.046)
Four					−0.211 ** (0.053)
Five					−0.178 ** (0.047)
Applicant no. of applications in 2nd round (ref. One): Two						−0.141 ** (0.035)
Three						−0.177 ** (0.040)
Four						−0.300 ** (0.051)
Five						−0.322 ** (0.063)
Constant	0.448 ** (0.022)	0.863 ** (0.229)	−1.277 ** (0.329)	−1.218 ** (0.323)	−1.084 ** (0.322)	−1.182 ** (0.320)

Note: A total of 1300 applications, of which the outcome would vary with change in admission criteria, are included in each model. Values in parentheses are clustered standard errors.

^**p < 0.01;

^*p < 0.05;

^†p < 0.1 (two-tailed t-test)

The multivariate models in Table 4 also confirm that those who graduated from high school earlier and likely had applied to college before fare significantly worse with the decreased emphasis on standardized test performance. Similarly, those attending private high schools were less likely among the winners than losers, and this tendency remains after accounting for the applicants’ high school grades and dossier lengths. Given that those reapplying to college typically have taken private cramming lessons to improve their GSAT scores or essays during the extra year, the fact that both such students and applicants from private high schools tend to lose out in the new system suggests that the reduced reliance on standardized tests weakens the return to private educational services as far as elite college admission is concerned. At the same time, it is also possible that private high school students and earlier-year graduates are worse off in the holistic evaluation system because the evaluation process allows more subjective judgment. Because private high schools used to be the last resort for middle school graduates who fail to pass the entrance exam to a public high school in Taiwan, and because those reapply to college bear the label of having failed before, they may suffer from evaluators’ bias in the holistic admission system.

Results from the models provide mixed evidence on whether the reduced emphasis on standardized test performance, on the whole, has benefited students from lower socioeconomic backgrounds. Once controlling for many other characteristics, those from neighborhoods with better educational resources were less likely to be among the winners with the shift to the holistic evaluation system. Within the same local area, however, students from families with higher housing prices are actually more likely to be among the winners than losers.

Next, Table 5 presents linear probability models predicting a positive change in admission outcome, as opposed to a negative one, if the admission system moves from the current one to a hypothetical one that uses GSAT scores merely as a threshold criterion and entirely relies on evaluations of application dossiers to make admission decisions. Just 928 applications would receive different outcomes from the current ones; the outcomes for the vast majority (78%) would not change. In fact, even when comparing the exam-only admission system to this hypothetical system—that is, changing the weight of GSAT performance from 100% to 0% in the final ranking of applications—the outcomes for the majority of the applications (61%) are still unchanged. The models show that women will benefit even more if college admission neglects standardized test scores and relies on submitted dossiers completely. Women’s better high school grades explain a large part of this advantage. Moreover, when using only the evaluation scores of written materials, as opposed to the current system, to decide about admissions, students from lower-ranking high schools, surprisingly, are more likely to gain seats at NTU, although the effect size is small. This finding suggests that instead of favoring more prestigious high schools, dossier evaluators may have tried to achieve a more balanced representation across high schools.

Table 5.

Linear probability models predicting positive instead of negative change in admission outcomes if based on application dossiers only.

	Model 1	Model 2	Model 3
Female	0.100 ** (0.035)	0.056 (0.034)	0.043 (0.034)
Disadvantaged background	0.361 ** (0.053)	0.372 ** (0.056)	0.402 ** (0.057)
Earlier-year graduate	−0.293 ** (0.053)	−0.314 ** (0.068)	−0.315 ** (0.067)
Log family home worth	0.008 (0.035)	0.043 (0.035)	0.043 (0.034)
Local educational resources	0.519 (0.410)	0.341 (0.402)	0.365 (0.401)
High school ranking	0.005 ** (0.001)	0.004 ** (0.001)	0.004 ** (0.001)
Private high school attendance	−0.202 ** (0.047)	−0.258 ** (0.046)	−0.243 ** (0.047)
High school grade average		0.021 ** (0.004)	0.023 ** (0.004)
Dossier number of pages		0.000 (0.000)	0.000 (0.000)
Applicant no. of applications (ref. One): Two		−0.095 * (0.047)
Three		−0.147 ** (0.050)
Four		−0.125 * (0.058)
Five		−0.180 ** (0.059)
Applicant no. of applications in 2nd round (ref. One): Two			−0.098 * (0.039)
Three			−0.167 ** (0.049)
Four			−0.182 ** (0.060)
Five			−0.323 ** (0.105)
Constant	0.012 (0.262)	−1.778 ** (0.407)	−1.963 ** (0.410)

Note: A total of 928 applications, of which the outcome would vary with change in admission criteria, are included in each model. Values in parentheses are clustered standard errors.

reversed coded; a larger value indicating a higher ranking.

^**p < 0.01;

^*p < 0.05;

^†p < 0.1 (two-tailed t-test)

5. Conclusions

Scholars and policy makers have long debated on the role of high-stakes standardized tests in elite college admissions, but a rigorous experiment that tests for whom a changed emphasis on standardized test performance would alter the admission outcome has been lacking. With a quasi-field experiment, we show that the shift away from relying completely on such tests has surprisingly little effect on the elite college access of disadvantaged social groups in Taiwan. The vast majority of NTU applicants would have the same outcomes in the exam-only admission system as in the holistic one; and this would still be the case if the scores of standardized tests are not considered at all in the final ranking of applications. Even for those whose outcomes would differ, it is not clear that their socioeconomic backgrounds matter. The average family wealth and neighborhood resources among the students are virtually unchanged under the different admission systems. The multivariate analysis shows that, net of other factors, those from neighborhoods with lower educational resources benefit from a reduced emphasis on standardized test performance. However, within the same zip code area those from wealthier families fare better in the holistic admission system. Because resourceful neighborhoods tend to be affluent, the added disadvantage to applicants from less wealthy families is likely offset by the added advantage to applicants from worse-off neighborhoods. Taken together, our findings provide weak support for the social reproduction or equalizer hypothesis. The emphasis on high-stakes standardized tests hardly alters the advantages or disadvantages of students from varying socioeconomic backgrounds in elite college admissions; the tests thus have little independent effect on intergenerational mobility.

Rather than students from upper- or lower-class backgrounds, we find that women are the main beneficiary from the lowered reliance on high-stakes standardized tests in elite college admissions. The less weight assigned to standardized tests and the more to written materials submitted by applicants in the decision process, the more likely women would experience a positive change in the admission outcome. Our multivariate models suggest that female students’ better high school grades and possibly putting more effort into preparing the application dossiers explain a considerable part of this female advantage. Although the Taiwanese context may differ from Western contexts in some ways, the pattern we uncover is consistent with worldwide research showing that girls tend to put more effort in schoolwork and earn higher grades than boys (Buchmann et al., 2008; Marks, 2008; Trautwein et al., 2006). A U.S. study examining changes in student compositions with the introduction of test-optional policies also finds increases in female students with the adoption of such policies (Bennett, 2022). All these prior findings suggest that the female advantage in elite school access under a more holistic, instead of test-driven, evaluation system likely exists elsewhere.

Although our experimental results are based on NTU applications only, our study sample is likely comparable to other elite university applicants in similar contexts, such as other East Asian countries, making our results generalizable. Given this, we can derive several broader implications for educational policies and the literature on educational inequality. First, because our quasi-field experiment indicates a low connection between high-stakes standardized tests and class-based inequality, the decision regarding the use of such tests in school admissions should be primarily based on whether the tests provide useful information to predict students’ success in college and beyond. Similarly, the evaluation of recent educational reforms across East Asia to incorporate a wider range of admission criteria in school transitions should focus on whether the reforms have achieved goals unrelated to inequality (e.g., reducing student stress).

Second, our results suggest that when the reliance of standardized tests decreases, signals of prior success or failure, such as having applied for college before and graduating from a private high school, could sway admission results more. Thus, the consideration about abandoning high-stakes standardized tests should also include the potential consequence of disproportionately rewarding those achieving academic success earlier on, even if class-based inequality is not a concern. This consideration may be especially relevant to East Asian societies, where students’ earlier failure can be easily assumed based on their high schools’ reputations or years of graduation.

Finally, even though we do not find major shifts in class-based inequality with the change in elite college admission criteria, our result that women are the clear winners with the lowered emphasis on standardized tests is important to our understanding of gender disparities in the schooling system. Whether the system rewards good grades, diligence, or test performance more could lead to differing educational outcomes for girls and boys. Because our finding about women’s gained advantage is likely generalizable beyond East Asia, future research should also explore how the widespread adoption test-optional policy since the COVID-19 epidemic may further contribute to the gender gap favoring women in college attainment in the United States and other Western countries (Buchmann et al., 2008).