Difficulties with anonymous shortlisting of medical school applications and its effects on candidates with non-European names: prospective cohort study

Abstract

Objective

To assess the feasibility of anonymous shortlisting of applications for medical school and its effect on those with non-European names.

Design

Prospective cohort study.

Setting

Leeds school of medicine, United Kingdom.

Subjects

2047 applications for 1998 entry from the United Kingdom and the European Union.

Intervention

Deletion of all references to name and nationality from the application form.

Main outcome measures

Scoring by two admissions tutors at shortlisting.

Results

Deleting names was cumbersome as some were repeated up to 15 times. Anonymising application forms was ineffective as one admissions tutor was able to identify nearly 50% of candidates classed as being from an ethnic minority group. Although scores were lower for applicants with non-European names, anonymity did not improve scores. Applicants with non-European names who were identified as such by tutors were significantly less likely to drop marks in one particular non-academic area (the career insight component) than their European counterparts.

Conclusions

There was no evidence of benefit to candidates with non-European names of attempting to blind assessment. Anonymising application forms cannot be recommended.

Key messages

• It is cumbersome to anonymise the current Universities and Colleges Admissions Service form as a candidate's name may appear up to 15 times

• Anonymised application forms may still be identified as being from candidates from ethnic minority groups

• More thorough anonymising of application forms, such as deletion of cultural activities, would edit out some personal attributes and may disadvantage these candidates

• Anonymous assessment of applications cannot be recommended

Introduction

In the United Kingdom there is huge competition to study medicine, with in excess of 13 000 applications through the Universities and Colleges Admissions Service for just over 4000 places. A series of recent studies has found that the likelihood of success is less among applicants from ethnic minority groups than among white applicants.^1–6 A study based on 1991 entry indicated that the situation was improving.⁵ Even so, when seven other mainly academic aspects of the application were taken into account not being from an ethnic minority group remained a significant predictor of success. A recent study looking at all home applicants for entry in 1996 and 1997 found a greater disadvantage for applicants from ethnic minority groups than previously.⁶ This study was, however, limited for technical reasons by not being able to include data on GCSE grades, which form a large part of the selection process and which were important predictors of success in previous studies.⁵

Most unsuccessful applicants are rejected solely on assessment of their application form—that is, at the ‘shortlisting’ stage before being invited for interview. It is during shortlisting that students from ethnic minority groups are believed to be disadvantaged.⁵ The application form contains no explicit reference to the applicant's ethnic background, so it seems likely that any discrimination must be based on the applicant's name. For this reason it has been suggested that the whole of the shortlisting process be performed anonymously.⁵

We decided to assess the feasibility of assessing forms anonymously within the current admissions system of the Universities and Colleges Admissions Service. In addition, we assessed the impact of doing so on the shortlisting system we have used at Leeds school of medicine for the past four years.

Methods

Shortlisting process

Our shortlisting process involves each application form being assessed separately by two of three admissions tutors (including AL). A score from zero to 20 points is awarded made up of four components including career insight (4 points), non-academic activities (6 points), academic profile (4 points), and suitability for a medical career as described by the confidential reference (6 points). When assessing applications, admissions tutors are unaware of the other selector's score. The sum of the two scores then forms the sole basis of the decision to reject, accept, or interview the applicant, although the threshold values may vary throughout the year according to the numbers and quality of applications received.

Anonymising forms

We studied all home (including EU) applicants to the medical course for entry in 1998 except for graduate applications, which are assessed separately. An admissions clerk, who played no part in the assessment of applicants, anonymised forms. All text to be deleted was first overwritten with a red marker pen, which allowed the text underneath to be easily read, and this comprised the ‘open’ application. The form was then photocopied, which rendered the highlighted section indecipherable, and this comprised the ‘blind’ form. The following text was deleted: full name, email address, country of birth, applicant's signature, and all references to the applicant's name found in the personal statement or confidential reference sections. Batches of about 100 forms with alternate blind and open applications were sent to each selector, with care being taken to avoid any selector assessing the same applications twice. In this way selectors always read alternate open and blind forms, of different applicants, and each form was assessed both open and blind by two different selectors. The short time available for processing of applications and other commitments by selectors prohibits equity in the number of forms assessed and equal randomisation of pairs of selectors.

Assignment of applicants to ethnic group

Ethnic background was determined by two administrative clerks not concerned with the selection process. From the full name, applicants with non-European sounding names were classified as from an ethnic minority group and only coded as such when both clerks regarded the name as non-European. Although not a strict definition of ethnic background, this is the definition most relevant to the potential discrimination under investigation. For each applicant the following data were recorded: order of application, non-European name, sex, scores for individual components of the assessment by both selectors, and final outcome of the application (offer or reject). When assessing forms blind, selectors were asked to indicate whether they had identified the applicant as being from an ethnic minority group from information provided on the form.

Analysis of data

Data were analysed in three stages with spss. Firstly, to confirm that applicants from ethnic groups receive lower marks, total score was modelled by linear regression. Secondly, differences between blind and open scores for each individual were compared to assess the effect of blinding. Finally, component scores were dichotomised at about the overall median so that numbers of candidates dropping more marks than average could be analysed for each component. Our interest was only in discordant scores, and we had no prior view as to whether blinded scores would be better or worse than open scores. Poisson regression of discordant scores was used to assess whether blinded scores were more worse (or less better) in the applicants from ethnic minority groups, which would indicate the putative discrimination.

Results

In total 2047 applications were included in the analysis, of which 1485 (72.5%) were adjudged to have a European name by the administrative staff. Overall, 166 (29%) forms (including two adjudged European) were identified as being from an applicant from an ethnic minority group by the selector assessing the anonymous form, whereas the remaining 398 were classified as unidentified ethnic minority group (table 1).

Table 1.

Identification of blinding by scorer, and percentage of candidates from ethnic minority groups identified by each blinded scorer

Scorer	European group	Unidentified ethnic minority group	Identified ethnic minority group
A	512	103	94 (48)
B	403	96	63 (40)
C	568	199	9 (4)
Total	1483	398	166 (29)*

^*Includes two applicants with European names. 

Association of ethnic group with total score

The mean (SD) of total score was 28.7 (4.6) points (fig 1). The corresponding figures for each group were: European 29.2 (4.3), unidentified ethnic minority 27.2 (5.1), and ethnic minority identified 27.7 (4.3). This group difference was highly statistically significant (P<0.0001) by linear regression with or without adjustment for potentially confounding factors (sex, time of application, combination of scorers). Early application and being female (difference 1.3, 95% confidence interval 0.9 to 1.7 points) were also both significantly associated with higher scores.

Figure 1.

Distribution of total scores at shortlising for all applicants for entry in 1998 according to study groups

Comparison of blind and open scoring

The differences between blind and open total scores did not significantly differ from zero in any of the three groups (fig 2). Using linear regression to control for confounding factors as above, the unidentified ethnic minority group had similar differences between blind and open scores to the European group (0.00, −0.20 to 0.20). The identified ethnic minority group had larger differences of blind minus open scores than the European group (0.25, −0.04 to 0.53), points compatible with minor positive discrimination, but this was not statistically significant (P=0.09).

Figure 2.

Comparison of scores awarded blind and open for each of subgroups defined by surname

Analysis of component scores

Analysis of the components of the blind score showed the identified ethnic minority group to be less likely than the other groups to drop marks for the career insight component (table 2). In each of the other components the pattern was that the European candidates were least likely to drop marks, with the identified and unidentified ethnic minority groups performing similarly.

Table 2.

Number (%) of candidates dropping marks on each component of blinded score

Component	European group (n=1483)	Unidentified ethnic minority group (n=398)	Identified ethnic minority group (n=166)
Career insight*	579 (39)	180 (45)	42 (25)
Non-academic activities†	564 (38)	200 (50)	85 (51)
Academic profile‡	630 (42)	222 (56)	92 (55)
Career suitability§	573 (39)	175 (44)	75 (45)

Dropped: ^*>1 of 4 points; ^†>2 of 6 points; ^‡any of 4 points; ^§>1 of 6 points. 

The differences observed between component scores for the ethnic groups were assessed for differential effects between blind and open scoring—that is, for discrimination affected by anonymising the application form (table 3). For the career insight component blinding was found to have a differential effect across the groups. Whereas European applicants were approximately equally likely to drop marks on either assessment, fewer applicants from unidentified ethnic minority groups dropped marks on open assessment and fewer applicants from identified ethnic minority groups scored worse on blind assessment. There was no evidence of other such differential effects for either non-academic activities, academic profile, or career suitability.

Table 3.

Comparison between discordant blind and open component scores. Values are numbers of applicants

Component	European group	Unidentified ethnic minority group	Identified ethnic minority group
Career insight:
Good open, poor blind	263	99	16
Poor open, good blind	240	51	27
Non-academic activities:
Good open, poor blind	210	53	24
Poor open, good blind	210	68	26
Academic profile:
Good open, poor blind	158	40	18
Poor open, good blind	157	40	18
Career suitability:
Good open, poor blind	240	50	27
Poor open, good blind	253	72	28

Discussion

Defining ethnicity

Classification by non-European name is not identical to ethnicity (table 4). However, if the lesser success of applicants from ethnic minority groups at shortlisting is due to discrimination then the name is the clearest marker of ethnicity available to the selector and has been shown previously to predict shortlisting outcome.⁵

Table 4.

Validation of ethnic grouping by admissions clerks. Values are numbers (percentages)

UCAS code	European group	Ethnic minority group	Total
White	1284 (98)	31 (2)	1315
Asian	38 (8)	436 (92)	474
Black	22 (55)	18 (45)	40
Other	17 (31)	37 (69)	54
Unrecorded	30 (70)	13 (30)	43
Missing	94 (78)	27 (22)	121
Total	1485	562	2047

UCAS: Universities and Colleges Administration Service. 

Outcome for ethnic minority applicants

In keeping with previous data both nationally and at Leeds school of medicine, applicants from ethnic minority groups scored less well than European applicants. Scores peaked just below 31-32 points, which for entry in 1998 was where the division between reject and interview and offer occurred. Thus the observed mean difference of just 1.5 to 2 points influenced the outcome for a large number of applicants.

Anonymising forms

Making application forms anonymous proved difficult and required a photocopy of already copied applications, with a further reduction in legibility of some forms. The applicant's name appears at least three times on each form and often up to 15 times, so complete removal required close scrutiny that was time consuming.

Anonymising application forms failed to achieve our aim, with one assessor able to identify nearly 50% of those deemed to be from ethnic minority groups. This was mostly found from the personal statement on the application form in which many applicants write about cultural activities and beliefs, and from GCSE examination passes in Asian language subjects. Retrospective questioning showed that scorer C did not consider the latter sufficient evidence of ethnic background to warrant classification as ethnic minority group identified, whereas scorers A and B did, giving rise to the variations (table 1).

Effects of anonymous shortlisting

Even when application forms were successfully anonymised the lack of difference between open and blinded overall scores suggested that disadvantage did not result from direct discrimination by selectors. The only statistically significant difference between blind and open assessment on any component of the total score pointed to positive discrimination, with more than anticipated of the ethnic minority group identified having better blind than open scores. This observation, on one quite subjective component, led to only a 0.25 (−0.04 to 0.53) difference in total score: not sufficient to make a practical difference. Nevertheless, any form of discrimination, whether positive or negative, is highly undesirable and offers another reason why anonymising applications is detrimental to the fairness of the selection process. Admissions tutors were aware of this study, and this finding may indicate a heightened awareness by the selectors that their performance with respect to racial discrimination was being assessed. It is possible that the selectors may also, for the same reason, have avoided negative discrimination during the study period, although these behaviours are clearly difficult to test.

For applications to be anonymised on a national scale the Universities and Colleges Admissions Service would have to delete names, which will be easy when electronic applications become universal. In addition, applicants and schools would have to avoid referring to names and other markers of ethnicity throughout the form. If this resulted in exclusion of outside activities and some GCSE subjects for applicants from ethnic minority groups this would clearly increase disadvantage.

To the authors' knowledge the only selection criterion currently used in the United Kingdom that has been shown to be justifiable is A level grades, which are reasonable predictors of success in the first three years at medical school.⁷ A selection system based solely on A level grades might therefore be considered ideal but would continue to disadvantage candidates from some ethnic minority groups⁸ and other groups in society and would also require the introduction of a lottery system to reduce the large number of applicants with high A level grades. Although UK medical schools continue to use selection systems including non-academic criteria, anonymous application forms cannot be advocated. Further research is urgently needed to determine whether or not the use of these criteria is justified.