Applicant Attitudes Toward the Association of American Medical Colleges’ Standardized Video Interview

Abstract

Background

The Standardized Video Interview (SVI) was developed by the American Association of Medical Colleges to allow applicants to include objective data about professional behaviors and interpersonal and communication skills. Although the SVI pilot was administered to individuals applying to emergency medicine (EM) residency programs during the 2018 Electronic Residency Application Service (ERAS) cycle, little data have been published evaluating the applicant's perceptions. This survey aims to assess EM residency applicant attitudes toward the SVI.

Methods

During the 2018 ERAS application season an anonymous survey was administered to interviewees at one urban Accreditation Council for Graduate Medical Education–approved EM residency. Respondents were asked questions regarding the production of their video interviews, thoughts regarding the additive value of the SVI, and individual demographic data such as ethnicity and sex. Participation was optional.

Results

A total of 219 of 238 candidates completed the survey representing a 92% response rate. While the majority of applicants did not feel that their ethnicity impacted their application, 58.1% of those who did self‐identified as African American or Asian. A total of 8.7% of respondents felt the SVI added information about their professional behaviors and 11% felt that it added information about interpersonal and communication skills. Only 2.8% of survey respondents felt the SVI should remain a portion of the ERAS application.

Conclusions

Most respondents felt that the SVI was not an accurate representation of their interpersonal and communication skills or their professionalism and that it did not add value to their applications. While most cohorts were not concerned about bias regarding sex, ethnicity, sex, or age, a small subset felt that there was a potential for the SVI to bias the party reviewing their applications. Very few applicants felt the SVI should remain a part of the ERAS application. Applicant attitudes toward the SVI are largely negative and require further investigation prior to becoming a standard part of applicants’ ERAS files.

Applicants for emergency medicine (EM) residency programs’ 2018 application cycle participated in a Standardized Video Interview (SVI) pilot developed by the American Association of Medical Colleges (AAMC). This project required applicants to film their responses to a series questions administered in real time via an online portal and submit their videos to the Electronic Residency Application Service (ERAS) for inclusion into their residency applications.1 The SVI operational pilot was developed in 2017 to provide objective and standardized information about the applicant in two specific Accreditation Council for Graduate Medical Education (ACGME) competencies that may not be readily discernable in other areas of an applicant's file, namely, their knowledge of professional behaviors along with their interpersonal and communication skills.1 The SVI and a composite score were viewable by EM residency programs with each candidate's ERAS application.

The SVI consists of six questions presented online via text prompts. The questions are not related to clinical knowledge and focused on subcompetencies such as emotional intelligence, ethics, empathy, and cultural competence. For example, “describe a time you worked with a challenging patient. What was the situation? What actions did you take?”2 The applicants have no prior knowledge of the questions and are given 30 seconds to read and reflect followed by a 3‐minute period to record a video response. The video responses are produced at the applicant's discretion at no increased cost to them. Modalities used included personal computers, electronic tablets, or smartphones.

Applicants have one opportunity to answer each question. Their responses are subsequently rated on a standardized scale developed specifically for the SVI and assigned a score between 1 and 5.1 With a total of six questions, cumulative scores range from a low of 6 to a high of 30, with a mean (±SD) of 18.7 (±2.8) per the 2017 AAMC pilot data. Six raters are assigned per interview and a different rater is assigned to each question with the goal of reducing any one raters influence on the overall score.1 All participating raters were selected from a pool of individuals “experienced in human resources and assessment.”3 Selected individuals then completed the AAMC's unconscious bias training, are educated on EM resident job duties, and utilize a standardized method to evaluate responses. Following 12 hours of in‐person training, raters practice rating applicants and receive direct feedback on their performance. Raters complete a total of 20 hours of training before being allowed to formally score applicants.3

It is likely the SVI will become a standard part of applicants’ ERAS files for all specialties. While the SVI was developed by the AAMC to provide information to residency programs regarding the aforementioned competencies, there is limited published literature regarding the viewpoint of those most affected by its implementation, i.e., the applicants themselves. Some have called for a slowdown of SVI implementation until applicant representation in the SVI process is established.4 Others have questioned whether the SVI can provide an adequate measure of professionalism. We sought to assess applicant attitudes toward the SVI pilot at one EM residency program as well as information regarding the applicants’ production of their individual SVIs.

Methods

This was a survey study of applicants at one large urban public hospital's ACGME‐approved EM residency program. The survey was administered to all applicants who underwent in‐person interviews for the residency program's 2018 application cycle. A survey instrument was used to ensure anonymity and minimize response modification by the applicants in the desire to garner a more favorable outcome during their interview day.

The survey was developed by the authors after a literature review determined that a relevant external survey had yet to be published. The survey was edited by faculty for clarity and relevancy. The survey was subsequently piloted to current EM resident physicians to determine if the survey was clear, minimized bias, and void of leading questions to increase construct validity. It was then presented to current medical students who were not participating in the upcoming ERAS application season to determine relevancy and ease of interpretation. The survey was separated into the following sections: SVI production; preparation; content; scoring; and demographic information such as race, sex, and ethnicity. The above domains were evaluated to gather general data on preparation and production modalities, to assess applicants’ perceived utility of the SVI, applicants’ knowledge of the scoring process, and perceived bias (Data Supplement S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10355/full).

The survey was administered during the 11 interview days of the 2018 application cycle spanning from October 17, 2017, to January 30, 2018. It was completed at the end of the interview day to minimize the potential confounder of an applicant attempting to answer survey items in a way that could be perceived as beneficial to his or her interview process. Subjects were allowed to answer however many of the questions as they wished and were allotted as much time as needed to respond, and all answered questions were included in the data. Survey data were subsequently abstracted into a computerized spreadsheet by trained research assistants. The study was approved by the local institutional review board. Additionally, the AAMC was contacted and permission was given to complete the study.

Results

A total of 219 of 238 applicants completed some portion of the survey representing a 92% response rate. Demographic details of respondents are presented in Table 1. Self‐reported SVI scores ranged from 10 to 29 with a median score of 20. SVI production modalities were fairly consistent among responders with most using a laptop computer to produce their SVI (193/219, 88.1%) which was filmed in their personal or private residence (151/219, 68.9%). A minority of respondents produced their SVI in a medical school facility (56/219, 25.6%). Technical difficulties were noted by 16 of 219 respondents (7.3%), and only three of 219 respondents reported that their medical school used a professional video production company to complete their SVI (1.4%).

Table 1.

Demographic Information of Survey Respondents

Characteristic	Respondents (n = 219)	AAMC 2017 Pilot Data (n = 1,760)
Sex (n = 205)
Female	86 (42.0)	734 (39.5)
Male	119 (58.0)	516 (60.5)
Age, years (n = 202)
Median (IQR)	27 (26–28)	NR
Range	24–42	NR
Race/ethnicity (n = 205)
American Indian or Alaska Native	2 (1.0)	NR
Asian	31 (15.1)	455 (25.6)
Black or African American	30 (14.6)	116 (10.6)
Hispanic, Latino, of Spanish origin	23 (11.2)	126 (7.4)
White	109 (53.2)	921 (52.6)
Multiple	4 (2.0)	NR
Other	6 (2.9)	NR

Data are reported as n (%) unless otherwise reported.

AAMC = American Association of Medical Colleges; IQR = interquartile range; NR = not reported.

Table 2 represents applicant attitudes toward the SVI. Overall 32% (70/2019) of applicants were satisfied with preparation for the SVI by the AAMC, 42% (96/219) felt that their medical schools offered adequate preparation. (96/219, 42%). A total of 22.8% (50/219) applicants felt they were understood the scoring process. While some respondents reported concerns that characteristics such as age, sex, sex, ethnicity, and appearance had an effect on their SVI scores, 75.5% to 92% felt that there was no effect for the listed categories. Although AAMC data from the 2017 to 2018 SVI administration showed that there were no scoring differences along race or ethnicity, 58.1% of applicants expressing concern over a negative effect of their ethnicity on their SVI score self‐identified as either African American or Asian (Table 3).

Table 2.

Applicant Attitudes Toward the SVI

Characteristic	Yes	No	Unsure
Production and preparation
Were you satisfied with the technical quality of your SVI?	149 (68.0)	25 (11.4)	45 (20.5)
Were you provided with adequate preparation by the AAMC to answer the questions of the SVI?	70 (32.0)	100 (45.7)	49 (22.4)
Did your medical school provide you with any preparation and/or coaching about the SVI?	96 (44.0)	115 (52.8)	7 (3.2)
Content
Did your SVI reflect additional information on your knowledge of professional behaviors not available elsewhere in your ERAS application?	19 (8.7)	174 (79.5)	26 (11.9)
Did your SVI reflect additional information on your interpersonal and communication skills not available elsewhere in your ERAS application?	24 (11.0)	171 (78.4)	23 (10.6)
If you had the ability to redo any of your answers, would you change any?	107 (48.9)	46 (21.0)	66 (30.1)
Scoring
Did you feel adequately informed of the scoring process prior to filming your SVI?	50 (22.8)	156 (71.2)	13 (5.9)
Should the SVI remain a part of the ERAS application?	6 (2.8)	187 (86.2)	24 (11.1)

Data are reported as n (%).

AAMC = American Association of Medical Colleges; ERAS = Electronic Residency Application Service; SVI = Standardized Video Interview.

Table 3.

Applicant Concerns Regarding Bias on the SVI

	What Effect, if Any, Do You Think the Following Personal or SVI Characteristics Had on the Score of Your SVI?
	Negative	No Effect	Positive
Age (n = 196)	9 (4.6)	180 (91.9)	7 (3.6)
Sex (n = 193)	15 (7.8)	167 (86.5)	11 (5.7)
Gender (n = 194)	18 (9.3)	164 (84.5)	10 (5.2)
Ethnicity (n = 194)	31 (16.0)	147 (75.7)	16 (8.2)
Physical appearance (n = 194)	28 (14.4)	146 (75.3)	20 (10.3)
Video production value (n = 196)	50 (25.5)	133 (67.9)	13 (6.6)

Data are reported as n (%).

SVI = Standardized Video Interview.

Discussion

The SVI was developed to provide objective data in respect to applicants’ professional behaviors along with their interpersonal and communication skills with the overall goal to add depth to their applications and potentially increase the number of interviews granted to applicants that may not have otherwise been considered.1 If the SVI can serve to identify additional proficiencies in these competencies, it is likely that its use will expand to residency applications across all medical specialties. Unfortunately, there have not been many studies evaluating the perspective of those most affected by the implementation of the Standardized Video Interview, the residency applicants. This study is one of the first to query applicants who have completed the SVI on their production modalities and their attitude toward its use in ERAS.

The overarching goal of the SVI was to provide objective data not available elsewhere in an ERAS application. To this end, only 11% of applicants felt that the SVI accomplished this goal.

The majority of respondents felt that they were insufficiently prepared to answer the SVI questions by either the AAMC or their medical schools and that they lacked an adequate knowledge of the scoring process. It is not clear what preparatory materials were given to applicants prior to implementation of the SVI. Furthermore, most of them would have revised their answers had they been given the opportunity to do so. These concerns should be addressed by both the AAMC and the applicants’ medical schools. Additionally, while the majority of respondents were unconcerned about scoring biases related to their age, sex, and gender, a minority expressed concern regarding the impact of video production value, ethnicity, and personal physical appearance on their SVI score. Only 2.8% (6/219) of respondents reported that the SVI should remain a part of the ERAS application. This could be an understandable response to any and all of the following factors: yet another requirement being placed on the shoulder of applicants, frustration with the perceived lack of transparency in scoring, the underlying potential for bias, and a concern that the Standardized Video Interview does not add value to their application.

While the data presented in this study do provide a snapshot of applicant attitudes toward the SVI, they are inherently limited due to selection bias. This study was conducted at only one ACGME‐approved EM residency program. Additionally, only applicants that were selected to interview were included in the study that potentially excludes applicants with lower SVI scores. Furthermore, although steps were taken to minimize potential confounders, including making the survey anonymous and administering it after the interview day was complete, the potential for self‐reporting bias still exists. Additionally, the authors were unable to follow‐up with respondents due to the anonymity of the process, but this was felt best as it allowed more honest reporting by applicants.

As residency programs prepare their rank lists, there is always a leap of faith in interpreting how an applicant's past performance and in‐person interview will translate to residency performance. Utilizing additional information to bridge that gap is a worthy endeavor; however, adding additional work on the applicants’ end, when they are already performing clinical rotations, studying clinical content, paying for medical school, and paying for their interview process, may be intrinsically unfair to them if they do not identify a clear benefit or, even worse, perceive the potential introduction of bias.

In March 2019 Academic Medicine published an article5 evaluating the validity of the Standardized Video Interview scores. There were two studies. Study 1 (2016 cohort) included 855 applicants applying to EM, pediatrics, and internal medicine. Study 2 (2017 cohort) included 3,532 applicants applying to EM during the 2018 ERAS cycle. It was noted in their article that there were none‐to‐small correlations between SVI scores and unrelated academic variables such as Step scores, implying that the SVI is “measuring something different than academic performance.”5 While this may be true, further research is needed to ensure that the SVI score is an accurate representation of professional behaviors and interpersonal and communication skills. This study does not expel applicant concern that the Standardized Video Interview may not accurately reflect these domains.

Conclusion

Despite the stated goal of providing an additional measurement of competencies for residency applicants that may be lacking in the current ERAS format, applicant attitudes toward the SVI are largely negative. These attitudes need to be respected and criticisms fully explored before the expansion of the pilot to a larger scale. Further work is needed to determine whether the SVI helps programs via allowing objective measures of constructs not typically available, adds an unnecessary burden to the application process, or potentially introduces biases that may hurt a select subset of applicants.

The authors acknowledge the Academic Associates Program of the Department of Emergency Medicine of Cook County Hospital for their assistance in enrolling study subjects and abstracting data. Specifically, we acknowledge the roles of the Research Associates Program Coordinator, Lum Rizvanolli, and Clinical Research Coordinator, Errick Christian.

Supporting information

Data Supplement S1. Standardized Video Interview (SVI) Questionnaire.

AEM Education and Training 2020;4:5–9.