In Reply to: Applicant Attitudes Toward the Association of American Medical Colleges’ Standardized Video Interview

In “Applicant Attitudes Toward the Association of American Medical Colleges’ Standardized Video Interview,”1 the authors describe their own program's applicant experiences with the EMSVI. The viewpoints of the applicants toward the SVI were largely negative. While the authors do identify the limitations of their study, we as members of the AAMC Standardized Video Interview Work Group believe that additional context is important to consider.

The data they published add to the body of literature around applicants’ experience with the SVI. In summer and fall 2017, the AAMC surveyed everyone who took the SVI.2 The first survey was administered immediately after applicants completed the SVI, and the second survey was administered after SVI scores were released. Both surveys addressed applicants’ experiences taking the SVI and their attitudes toward it. Results from the surveys had a much larger sample size, with each exceeding 2,000 respondents; users were found to have more mixed, but more positive, perceptions of the SVI. Consistent with the authors’ results, it found that applicants had generally negative reactions toward the SVI. They were skeptical of its ability to assess the target competencies and add value to the selection process. However, applicants also had more positive attitudes about the procedural aspects of the SVI (e.g., clarity of instructions, time allotted to respond) and preparation materials.

The authors’ main conclusion drawn in the paper is that applicants believe the SVI did not add value to their applications; the vast majority do not support its use. These findings are not particularly surprising, given that applicants already feel overwhelmed with the residency application process. They are also consistent with the employment literature, which shows that applicants have generally negative attitudes toward structured interviews and video‐based interviews.3, 4, 5 However, we must question what role these data should play in the national discussion of how we should be using novel selection tools in residency selection. Applicant reactions about issues of policy, procedure, usability, and preparation are very helpful because they can inform us of ways to improve the tool itself or educational needs. The best people to judge whether the tool is an accurate reflection of applicants’ competencies and whether it adds value to the selection process, however, are the program directors. Recently published evidence shows that the SVI does offer perceived value to program directors, which also offers a stronger argument for content validity.6 What validity evidence does student satisfaction and perception bring to the discussion? Is student perception of the SVI more important to consider than their opinion on the role of other components of their application, such as USMLE scores or the personal statement? The authors could make a stronger argument as to why student subjective opinion of the SVI is uniquely valuable and why it should inform its inclusion in the selection process.

Regarding applicant concern about bias that the authors uncovered in this paper, data do show that there are no differences in SVI scores by race or ethnicity.7 However, it is valid to question how program directors viewing the videos may introduce bias themselves. The SVI, along with existing application components such as the candidate photo, letters of recommendation, and even clerkship grades are all potential sources of unconscious bias, and the medical education community should be examining these threats as well. AAMC provides training, including unconscious bias training, to program directors who signed up for the SVI pilot. In addition, it offers programs that wish to conduct a blind review of applications the ability to “turn off” SVI videos. This paper raises questions that should be asked about the value and limitations of all aspects of the residency application we are using right now.

Finally, the authors suggest that use of the SVI is an inevitability, although the AAMC has clearly indicated that the SVI is a pilot process. We will need to closely follow the results of the multiyear study further testing the validity of the SVI, currently under way as a partnership between the AAMC and 17 emergency medicine residency programs.

Applicants are certainly important stakeholders in the residency selection process, and their perspectives are valuable. However, the main conclusion we draw from the work of Winfield et al, is that we need to continue efforts toward demonstrating the validity evidence we do have for the SVI particularly by following matched residents over time in their residency training. Most importantly, we in the medical education community need to be applying the same scrutiny the SVI is undergoing to our historical application tools as well.