Abstract
Background:
This prospective study investigated whether in-person interviews affected interviewer assessments of anesthesiology residency applicants at an academic medical center, and which applicant characteristics influenced interview performance.
Methods:
Eighteen faculty members involved in residency recruitment between November 2019 and January 2020 documented preinterview (after full application review) and postinterview scores of the applicants on a scale of 1 to 5. Faculty also reported the relative contributions of specific interview characteristics (personality, physical appearance, professional demeanor, discussion regarding academic/scholarly activity, and level of interest in the specialty) to their postinterview assessments. Mixed-effects models were used to assess whether interviews changed faculty assessment of applicants, and what the relative contributions of applicant characteristics were to faculty assessments.
Results:
A total of 696 interviews were conducted with 232 applicants. The postinterview scores differed significantly from the preinterview scores (estimated mean difference, 0.09 ± 0.02; P < 0.0001). The characteristics most affecting postinterview scores were positive impressions of applicants’ personalities (marginal mean change in postinterview score, 0.259; 95% confidence interval, 0.221–0.297) and negative impressions of applicants’ professional demeanor (marginal mean change, −0.257; 95% confidence interval, −0.350 to −0.164).
Conclusions:
In-person interviews significantly affected residency applicants’ scores. Personality and professional demeanor influenced scores more than did other characteristics examined. Further studies are needed to clarify the relevance of in-person interviews to the assessment of residency applicants.
Keywords: Residency, anesthesiology, recruitment, interview, graduate medical education
Introduction
In-person interviews for anesthesiology residency applicants allow an applicant to familiarize themselves with the hospital, program, and faculty, and allow the program to engage and interact with the applicant. However, they can also be extremely time-consuming and costly for both applicants and residency programs.1–3 Although anesthesiology resident recruitment has historically included in-person interviews, it remains unclear how these interviews affect the likelihood of matching those residents deemed to be the best fit for the program.
Numerous reports suggest that program directors consider the interview among the most important factors in resident selection.4–8 Many appreciate the interview as a means to find those applicants who would fit in well in their program, as well as an opportunity to gauge applicant characteristics that are not as easily apparent in the written application.6,7 Although the association between interview performance and final rankings of applicants has been investigated in other medical specialties, in anesthesiology it remains unknown. A study evaluating dermatology residency applicants demonstrated a stronger correlation between interview scores and rankings than between rankings and scores derived solely from application review.9
In light of the cost and time required for in-person interviews, the relevance of such interviews relative to other application materials is an important question. Whereas information such as grades and exam scores may be more objective, written essays are not spontaneous and may not convey personal characteristics associated with success such as grit.10 The interview may be a way to better assess these, in conjunction with other application items such as letters of recommendation. Existing data suggest that different types of information on residency applications correlate with different performance dimensions. One 2013 meta-analysis found that exam scores strongly correlated with in-training examination performance but had a weaker association with more subjective performance measures, such as faculty evaluations.11
We sought to evaluate the impact of the in-person interview on the assessment of anesthesiology residency applicants in a tertiary care academic department. We hypothesized that the in-person interview affected the final scoring of candidates, which ultimately influences candidate ranking at this institution. To test our hypothesis, we surveyed faculty members immediately after their live interviews regarding their assessment of applicants before and after the interview. Our primary outcome was the difference between preinterview and postinterview scores. In addition, we assessed which factors most affected changes in assessment after the in-person interview.
Materials and Methods
Participants
This prospective study was deemed exempt from formal review by the institutional review board at the University of Chicago (IRB19-1663). Residency applicants were selected to interview for 18 residency spots based on an initial review of their written applications, including review of board exam scores, medical-school grades, letters of recommendation, activities, and experiences. Each application was reviewed individually by 1 of 3 designated faculty members; no rigid cutoffs or algorithms were used to grant interviews. All applicant interviews were included in this study. Participants included all 18 faculty interviewers who participate in the resident recruitment committee, which comprises the program director team and other faculty members who were invited to participate in resident recruitment based on their interests or roles in the department.
Survey and Data Collection
All data collection took place at the University of Chicago. Normally, an online survey (SurveyMonkey Inc, Portland, Oregon) is used to record faculty interviewers’ assessments of residency applicants at the University of Chicago, which they complete immediately after the interview (postinterview score). For purposes of this study, 2 questions were added to the survey (Supplemental Online Material): a preinterview score and an inquiry regarding applicant characteristics (discussed in further detail later). A statement was also included at the beginning of the survey noting that participation in the survey implies consent by the participating faculty.
Each applicant had 3 separate interviews, each with 1 member of the faculty recruitment committee. The time allotted per interview varied from 15 to 20 minutes, and a faculty member interviewed 8 to 13 applicants a day. Therefore, not every faculty member interviewed every applicant. Survey responses were deidentified by an administrative assistant before being distributed for analysis in this study—each faculty member was assigned a letter and each applicant was assigned a number. Only 1 response was allowed per survey invitation, and faculty members were informed that their participation in the study was completely voluntary.
Separate survey links for each applicant were provided to faculty interviewers by the program coordinator in advance of the interview day. The 2 survey study questions which were added for research purposes were marked to delineate that they were for study purposes only and would not have any influence on the ranking of applicants (Supplemental Online Material). The study questions included a preinterview score as well as a multipart question inquiring which characteristics (personality/communication/interpersonal skills, physical appearance, professional demeanor, discussion regarding academic/scholarly activity, level of interest in the specialty based on interview discussion, and other) influenced a change in postinterview scoring. These characteristics were chosen partially based on prior studies in this field.7,12,13 The postinterview score, a regular measure in the recruitment survey, was also used for this study.
Our protocol dictated that faculty interviewers were to document the preinterview score on the survey before meeting the applicant. They were then able to reaccess the survey links immediately after the interview to complete the remaining survey questions. The preinterview score (on a scale from 1 to 5) was based on a review of the applicant’s application before the interview and included information such as age, gender, Alpha Omega Alpha Honor Medical Society status, United States Medical Licensing Examination Step 1 score, letters of recommendation, personal statement, ethnicity, medical-school transcript, Medical Student Performance Evaluation, a photograph of the applicant, and documentation regarding research, volunteer and work experiences. Questions regarding which applicant interview characteristics influenced postinterview scores were graded on a 5-point subjective scale. The postinterview score (using the same 1–5 scale as the preinterview score) was generated after the in-person interview. In determining the postinterview score, the faculty member was asked to consider all aspects of the applicant, via assessment of the written application materials as well as the interview. To anchor the scoring of applicants and provide some standardization between faculty interviewers, references were provided to the faculty for numerical preinterview and postinterview scores (1: “Would not want here,” 2: “A potential challenge to get through,” 3: “A passing resident,” 4: “Upper ½ of residency class,” 5: “Offer signing bonus now!”). For context, each applicant’s 3 postinterview scores are averaged to create a preliminary ranking order of candidates at this institution. The complete list of study questions is available in the Supplemental Online Material.
Additional Data
Additional information gathered from the applicant’s file included age, gender, Alpha Omega Alpha status, United States Medical Licensing Examination Step 1 score, and self-reported race/ethnicity. This last piece of information was collected to determine whether the applicant self-identified as a member of racial or ethnic group that is underrepresented in medicine,14 which is an evolving category, but at the time of data collection was considered to include Black/African American, Latinx, and Native American people.
Statistical Analysis
The primary outcome, change in applicant scores after the in-person interview, was defined and established a priori at initiation of the study design. We used a mixed-effects model that includes 2 random effects to account for multiple interviews per applicant and multiple interviews per interviewer. Specifically, we used a multilevel cross-classified model in which applicants and interviewers were cross-classified, because each applicant interviewed with 3 different faculty members and each faculty member interviewed multiple applicants.15
The secondary outcome, relative contribution of specific characteristics to faculty postinterview scoring of applicants, was also defined and established a priori at initiation of the study design. The same mixed-effects model was used. Results are reported as percentages or means ± SD. The results of the mixed-effects model are reported both as estimates ± standard error and marginal means with 95% confidence interval (CI). All P values are 2-sided. All the analyses were conducted using SAS version 9.4 (SAS Institute, Cary, NC). This article adheres to the applicable Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines16 (Supplemental Online Material).
Results
Data were collected from November 2019 through January 2020. Two hundred and thirty-two applicants were each interviewed by 3 faculty members in a 1-on-1 setting for 18 available residency spots. A total of 696 interviews were completed by 18 faculty members. All faculty participating in residency interviews (18/18) agreed to participate in the study. Data from 13 interviews were removed from analysis due to incompletion of study questions. Thus, the overall survey response rate was 98%. The average age of applicants was 27.3 (± 2.6) years, and 99/232 (43%) of the applicants were female. Additional demographic characteristics are listed in Table 1.
Table 1.
Applicant Demographics (N = 232)
| Characteristic | n (%) |
|---|---|
| Age, mean ± SD, y | 27.3 ± 2.6 |
| Gender | |
| Male | 133 (57) |
| Female | 99 (43) |
| Alpha Omega Alpha member | |
| Yes | 29 (12.5) |
| No | 203 (87.5) |
| USMLE Step 1 score, mean ± SD | 240.1 ± 11.0 |
| Underrepresented in medicine | |
| Yes | 21 (9) |
| No | 211 (91) |
Abbreviation: USMLE, United States Medical Licensing Examination.
Applicants were scored on a scale from 1 (weakest) to 5 (strongest). The marginal mean (model-based predicted means) preinterview score was 3.93 (95% CI, 3.85–4.01) and the postinterview score was 4.02 (95% CI, 3.94–4.10; Table 2). The postinterview score was higher than the preinterview score (estimated mean difference, 0.09 ± 0.02; P < 0.0001). For calibration, a change in score by 0.09 away from the mean applicant score led to a ranking change of 10 to 15 positions (average, 12.5) at this institution in 2020. As crude percentages, the postinterview score went up in 52.3% (364/696) of interviews, went down in 22.3% (155/696), and did not change in 23.6% (164/696; missing data in 1.9%, 13/696).
Table 2.
Preinterview and Postinterview Scores a
| Marginal Mean | SE | 95% CI | |
|---|---|---|---|
| Pre | 3.93 | 0.042 | 3.85–4.01 |
| Post | 4.02 | 0.042 | 3.94–4.10 |
Abbreviations: CI, confidence interval; SE, standard error.
aScore range was from 1 (weakest) to 5 (strongest).
All characteristics (personality, physical appearance, professional demeanor, discussion regarding scholarly activity, interest in the specialty) identified by the faculty were shown to affect the postinterview scores in this group of applicants (all Ps < 0.0001; Table 3). Faculty identified “other” as a meaningful characteristic 18 times. When factors were analyzed to determine their relative effects on interview scoring, a positive impression of the applicant’s personality (marginal mean change in postinterview score, 0.259; 95% CI, 0.221–0.297) and a negative impression of the applicant’s professional demeanor (marginal mean change in post-interview score, −0.257; 95% CI, −0.350 to −0.164) had the largest effects (Figure 1A–E).
Table 3.
Associations Between Characteristics and Score Changes
| Assessment of Characteristics | Estimatea | SE | P | Marginal Mean Differencea | 95% CI |
|---|---|---|---|---|---|
| Personality | |||||
| None | Reference | 0.032 | −0.016 to 0.079 | ||
| Negative | −0.276 | 0.029 | <.0001 | −0.244 | −0.295 to −0.193 |
| Positive | 0.227 | 0.023 | <.0001 | 0.259 | 0.221 to 0.297 |
| Physical Appearance | |||||
| None | Reference | 0.080 | 0.031 to 0.130 | ||
| Negative | −0.309 | 0.055 | <.0001 | −0.229 | −0.342 to −0.116 |
| Positive | 0.160 | 0.032 | <.0001 | 0.241 | 0.173 to 0.309 |
| Professional Demeanor | |||||
| None | Reference | 0.019 | −0.024 to 0.061 | ||
| Negative | −0.276 | 0.048 | <.0001 | −0.257 | −0.350 to −0.164 |
| Positive | 0.221 | 0.024 | <.0001 | 0.239 | 0.196 to 0.283 |
| Discussion Regarding Scholarly Activity | |||||
| None | Reference | 0.059 | 0.011 to 0.111 | ||
| Negative | −0.295 | 0.044 | <.0001 | −0.237 | −0.324 to −0.149 |
| Positive | 0.143 | 0.025 | <.0001 | 0.202 | 0.153 to 0.251 |
| Interest in the Specialty | |||||
| None | Reference | 0.050 | 0.005 to 0.096 | ||
| Negative | −0.264 | 0.052 | <.0001 | −0.214 | −0.316 to −0.111 |
| Positive | 0.157 | 0.025 | <.0001 | 0.207 | 0.158 to 0.257 |
Abbreviations: CI, confidence interval; SE, standard error.
aEstimate is the mean score change after the interview compared with the reference group. Marginal mean difference is the mean change between preinterview and postinterview scores.
Figure 1.
Factors leading to changes in applicant scoring. Marginal postinterview score changes are displayed by interviewer impression of each characteristic: A, personality, communication or interpersonal skills; B, physical appearance; C, professional demeanor; D, discussion regarding academic or scholarly activity; E, level of interest in the specialty. N = number of interviews.
Discussion
In this prospectively conducted survey of anesthesiology faculty participating in residency recruitment at a tertiary academic medical department, we found that the in-person interviews conducted as part of residency recruitment in 2019–2020 significantly improved the interviewers’ assessment of the applicants. A positive rating on personality and a negative rating on professional demeanor had the greatest impact on the postinterview score.
Our findings are consistent with prior studies in other specialties that have examined factors that may affect scoring on in-person interviews. Studies of other medical specialties have also shown a significant influence of interviews on applicant ranking.17–19 Additionally, interview performance has been found to be predictive of good clinical performance and high examination scores in residency.20,21 In a 1983 study of medical residency applicants, Boor et al observed that physical appearance may have influenced scoring of female applicants, and professional demeanor influenced the scoring of both male and female applicants.12 Our findings also suggest that professional demeanor is an important element of the interview and that a poor demeanor, as interpreted by the interviewer, had the greatest negative effect on interviewers’ assessment of interview performance out of the characteristics we evaluated. On the other hand, applicants perceived as having positive personality traits benefited greatly from in-person interviews.
Our data suggest that interviews meaningfully affect the anesthesiology residency application process. Although the mechanism underlying the relevance of factors such as demeanor to anesthesia residency assessment are unclear, aspects of anesthesia practice that require a favorable demeanor such as situational awareness and the need to rapidly establish a rapport with patients and interact with multiple members of the perioperative care team may play a role.22
Although our study argues for the continued use of live interviews in residency recruitment, it does not address whether other strategies such as video interviews might be equally effective. Such an approach would not only offer cost savings23,24 for applicants and programs but might also enable a wider pool of applicants who may not have the time, resources, or transportation options to “visit” distant residency programs. However, an inability to visit the physical residency location may worsen the ability of applicants themselves to assess the residency program. Existing evidence suggests that video interviews may be an equivalent option with respect to admissions. A 2016 study of a single US anesthesiology program found that admission rates for residency applicants completing in-person versus virtual interviews did not differ.25 Further studies are needed to clarify how a virtual interview experience compares to a live format, and whether the impact of a virtual interview relative to other sources of information for assessing applicants will change. As recruitment has become entirely virtual during the COVID-19 pandemic, it is possible that the impact of the interview on both the program and the applicant will change.
Our study has limitations. It was performed within a single academic anesthesiology department and thus may not be reflective of all anesthesiology residency programs. Data collected from faculty interviewers were based on their self-reported subjective survey responses obtained before (preinterview score) and immediately after the interview (all other interview questions), and opinions may have evolved over time. While our statistical analysis did take into account the fact that the same candidate interviewed with 3 different faculty members, and that the same faculty member interviewed many different candidates, we did not take into account the variability in the number of interviews per day, the length of the interviews, or the differences in score changes between faculty members. In addition, the applicant characteristics queried in the faculty survey were identified partially based on prior studies7,12,13 but were not validated for the purposes of this survey. It is possible that other characteristics which have greater influence on the inperson interview were not included in this study. Faculty members did identify “other” characteristics as meaningful 18 times in this study—with individual responses varying considerably from 1 word to complete sentences. Thus, we chose not to include these data in our analysis. Additionally, the lack of consensus on desirable personality traits and professional demeanor may have contributed to observed differences. Because our data were completely deidentified as to applicant and faculty members, we were unable to further analyze the effect of faculty characteristics on the scoring of applicants. Finally, we assessed the effect of the interview on postinterview scores immediately after the interview; therefore, this study does not take into account the comparative assessment between applicants that may contribute to adjustments in the final candidate ranking. While we did not study the effect of interviewer scores on the final ranking of applicants, given that our scores are very closely related to our rank list, we suspect the two to be similarly related.
In summary, we found that interviewer impressions of residency candidates in an academic anesthesia program improved after a live interview. Our study builds upon prior investigations demonstrating the relevance of in-person interviews on residency candidate scoring, and offers an objective assessment of the importance of interviews for anesthesiology residency programs. Additionally, positive ratings on personality and professional demeanor had the greatest positive impact on the postinterview score. Further studies are needed to explore the relevance of in-person interviews to anesthesia resident performance, as well as how in-person interviews differ from virtual interviews in applicant scoring.
Acknowledgments
We thank Theresa Cumming, Director of Operations, Anesthesia and Critical Care, University of Chicago, Chicago, IL, USA, for aggregating and deidentifying the data for analysis.
Footnotes
Funding disclosures: None.
Conflicts of interest: The authors declare no competing interests, except the following: Sajid Shahul holds NIH funding R21HL148488.
Supplemental Online Material
Consent Script and Study Questions Included in the Study Survey
The preinterview score was completed before the interview. The remaining two questions were completed immediately after the interview.
Consent Script
We are performing a study to evaluate the impact of in-person interviews on scoring of residency applicants. The questions which are denoted with an asterisk (*) are for purposes of a study related to resident recruitment and will not be used in any way for scoring purposes for the applicant. This study was approved by University of Chicago’s IRB (IRB19-1663). Participation in this study is completely voluntary, and responses will be blinded for interviewer and applicant names prior to being analyzed. Completion of study questions implies consent for participation in this study. For any questions regarding this study, please contact Sarah Nizamuddin at snizamuddin@dacc.uchicago.edu.
Questions
*Preinterview score: _____ (Please score to 1 decimal point, i.e. 3.2)
Scale for reference:
Post-interview score: _____ (Please score to 1 decimal point)
Scale for reference:
*For each factor below, please rate the level of influence they had on your post-interview score. If a category did not influence you to change your score after the interview, then select “0.” (For example, personality may have had a high level of influence on you changing your score because they had a a) wonderful personality and were very friendly (++) or because they had a b) unfriendly personality and seemed rude at times (--). Alternatively, you might select “0” for personality because they had a “normal” personality and it did not therefore influence you to change your score):
-- - 0 + ++
STROBE Statement
Following is the checklist of items that should be included in reports of cohort studies.
1 – “would not want here”
2 – “a potential challenge to get through”
3 – “passing resident”
4 – “Upper ½ of residency class”
5 – “Offer signing bonus now!”
1 – “would not want here”
2 – “a potential challenge to get through”
3 – “passing resident”
4 – “Upper ½ of residency class”
5 – “Offer signing bonus now!”
Personality/communication/interpersonal skills
Physical appearance (i.e., neatness, grooming, or other)
Professional demeanor
Discussion regarding academic/scholarly activity
Level of interest in the specialty based off interview discussion
Other (optional) _______
| Item No | Recommendation | Page No | |
|---|---|---|---|
| Title and abstract | 1 | (a) Indicate the study’s design with a commonly used term in the title or the abstract | 3 |
| (b) Provide in the abstract an informative and balanced summary of what was done and what was found | 3 | ||
| Introduction | |||
| Background/rationale | 2 | Explain the scientific background and rationale for the investigation being reported | 5–6 |
| Objectives | 3 | State specific objectives, including any prespecified hypotheses | 5–6 |
| Methods | |||
| Study design | 4 | Present key elements of study design early in the paper | 6–8 |
| Setting | 5 | Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection | 6 |
| Participants | 6 | (a) Give the eligibility criteria, and the sources and methods of selection of participants. Describe methods of follow-up | 6 |
| (b) For matched studies, give matching criteria and number of exposed and unexposed | N/A | ||
| Variables | 7 | Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give diagnostic criteria, if applicable | 8–9 |
| Data sources/measurement | 8* | For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than one group | 8–9 |
| Bias | 9 | Describe any efforts to address potential sources of bias | 9 |
| Study size | 10 | Explain how the study size was arrived at | N/A |
| Quantitative variables | 11 | Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen and why | 9 |
| Statistical methods | 12 | (a) Describe all statistical methods, including those used to control for confounding | 8–9 |
| (b) Describe any methods used to examine subgroups and interactions | 8–9 | ||
| (c) Explain how missing data were addressed | 8–9 | ||
| (d) If applicable, explain how loss to follow-up was addressed | N/A | ||
| (e) Describe any sensitivity analyses | N/A | ||
| Results | |||
| Participants | 13* | (a) Report numbers of individuals at each stage of study—eg numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed | 9 |
| (b) Give reasons for non-participation at each stage | 9 | ||
| (c) Consider use of a flow diagram | N/A | ||
| Descriptive data | 14* | (a) Give characteristics of study participants (eg demographic, clinical, social) and information on exposures and potential confounders | 9 |
| (b) Indicate number of participants with missing data for each variable of interest | 9 | ||
| (c) Summarise follow-up time (eg, average and total amount) | N/A | ||
| Outcome data | 15* | Report numbers of outcome events or summary measures over time | 9–10 |
| Main results | 16 | (a) Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (eg, 95% confidence interval). Make clear which confounders were adjusted for and why they were included | 9–10 |
| (b) Report category boundaries when continuous variables were categorized | 9-10 | ||
| (c) If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period | N/A | ||
| Other analyses | 17 | Report other analyses done—eg analyses of subgroups and interactions, and sensitivity analyses | N/A |
| Discussion | |||
| Key results | 18 | Summarise key results with reference to study objectives | 10 |
| Limitations | 19 | Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias | 11–12 |
| Interpretation | 20 | Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence | 12 |
| Generalisability | 21 | Discuss the generalisability (external validity) of the study results | 12 |
| Other information | |||
| Funding | 22 | Give the source of funding and the role of the funders for the present study and, if applicable, for the original study on which the present article is based | N/A |
*Give information separately for exposed and unexposed groups.
Note: An Explanation and Elaboration article discusses each checklist item and gives methodological background and published examples of transparent reporting. The STROBE checklist is best used in conjunction with this article. It is freely available on the Web sites of PLoS Medicine at http://www.plosmedicine.org/, Annals of Internal Medicine at http://www.annals.org/, and Epidemiology at http://www.epidem.com/. Information on the STROBE Initiative is available at http://www.strobe-statement.org.
References
- 1.Benson NM, Stickle TR, Raszka WV., Jr. Going “fourth” from medical school: fourth-year medical students’ perspectives on the fourth year of medical school. Acad Med . 2015;90(10):1386–93. doi: 10.1097/ACM.0000000000000802. [DOI] [PubMed] [Google Scholar]
- 2.Wood JS, David LR. Outcome analysis of factors impacting the plastic surgery match. Ann Plast Surg . 2010;64(6):770–4. doi: 10.1097/SAP.0b013e3181b4bcf5. [DOI] [PubMed] [Google Scholar]
- 3.Brummond A, Sefcik S, Halvorsen AJ, et al. Resident recruitment costs: a national survey of internal medicine program directors. Am J Med . 2013;126(7):646–53. doi: 10.1016/j.amjmed.2013.03.018. [DOI] [PubMed] [Google Scholar]
- 4.Wagoner NE, Suriano JR, Stoner JA. Factors used by program directors to select residents. J Med Educ . 1986;61(1):10–21. doi: 10.1097/00001888-198601000-00002. [DOI] [PubMed] [Google Scholar]
- 5.Makdisi G, Takeuchi T, Rodriguez J, Rucinski J, Wise L. How we select our residents—a survey of selection criteria in general surgery residents. J Surg Educ . 2011;68(1):67–72. doi: 10.1016/j.jsurg.2010.10.003. [DOI] [PubMed] [Google Scholar]
- 6.Nallasamy S, Uhler T, Nallasamy N, Tapino PJ, Volpe NJ. Ophthalmology resident selection: current trends in selection criteria and improving the process. Ophthalmology . 2010;117(5):1041–7. doi: 10.1016/j.ophtha.2009.07.034. [DOI] [PubMed] [Google Scholar]
- 7.Al Khalili K, Chalouhi N, Tjoumakaris S, et al. Programs selection criteria for neurological surgery applicants in the United States: a national survey for neurological surgery program directors. World Neurosurg . 2014;81(3–4):473–7.e2. doi: 10.1016/j.wneu.2013.08.019. [DOI] [PubMed] [Google Scholar]
- 8.National Resident Matching Program, Data Release and Research Committee Results of the 2018 NRMP Program Director Survey . Washington, DC: National Resident Matching Program; 2018. [Google Scholar]
- 9.Kamangar F, Davari P, Azari R, et al. The residency interview is still paramount: results of a retrospective cohort study on concordance of dermatology residency applicant evaluators and influence of the applicant interview. Dermatol Online J . 2017;23(5):22. [PubMed] [Google Scholar]
- 10.Duckworth AL, Quinn PD. Development and validation of the Short Grit Scale (Grit-S) J Pers Assess . 2009;91(2):166–74. doi: 10.1080/00223890802634290. [DOI] [PubMed] [Google Scholar]
- 11.Kenny S, McInnes M, Singh V. Associations between residency selection strategies and doctor performance: a meta-analysis. Med Educ . 2013;47(8):790–800. doi: 10.1111/medu.12234. [DOI] [PubMed] [Google Scholar]
- 12.Boor M, Wartman SA, Reuben DB. Relationship of physical appearance and professional demeanor to interview evaluations and rankings of medical residency applicants. J Psychol . 1983;113(1):61–5. doi: 10.1080/00223980.1983.9923557. [DOI] [PubMed] [Google Scholar]
- 13.Stephenson-Famy A, Houmard BS, Oberoi S, et al. Use of the interview in resident candidate selection: a review of the literature. J Grad Med Educ . 2015;7(4):539–48. doi: 10.4300/JGME-D-14-00236.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Association of American Medical Colleges Diversity and inclusion: underrepresented in medicine definition. aamc.org/what-we-do/diversity-inclusion/underrepresented-in-medicine Accessed December 4, 2020.
- 15.Goldstein H. Multilevel cross-classified models. Sociol Methods Res . 1994;22(3):364–75. [Google Scholar]
- 16.von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, for the STROBE Initiative The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147:573. doi: 10.7326/0003-4819-147-8-200710160-00010. [DOI] [PubMed] [Google Scholar]
- 17.Christakis PG, Christakis TJ, Dziura J, Christakis JT. Role of the interview in admissions at the University of Toronto ophthalmology program. Can J Ophthalmol . 2010;45(5):527–30. doi: 10.3129/i10-041. [DOI] [PubMed] [Google Scholar]
- 18.Gong H, Jr, Parker NH, Apgar FA, Shank C. Influence of the interview on ranking in the residency selection process. Med Educ . 1984;18(5):366–9. doi: 10.1111/j.1365-2923.1984.tb01284.x. [DOI] [PubMed] [Google Scholar]
- 19.Swanson WS, Harris MC, Master C, et al. The impact of the interview in pediatric residency selection. Ambul Pediatr . 2005;5(4):216–20. doi: 10.1367/A04-149R1.1. [DOI] [PubMed] [Google Scholar]
- 20.Chen F, Arora H, Martinelli SM, et al. The predictive value of pre-recruitment achievement on resident performance in anesthesiology. J Clin Anesth . 2017;39:139–44. doi: 10.1016/j.jclinane.2017.03.052. [DOI] [PubMed] [Google Scholar]
- 21.Brothers TE, Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ . 2007;64(6):378–85. doi: 10.1016/j.jsurg.2007.05.003. [DOI] [PubMed] [Google Scholar]
- 22.Patterson F, Knight A, Dowell J, et al. How effective are selection methods in medical education? a systematic review. Med Educ . 2016;50(1):36–60. doi: 10.1111/medu.12817. [DOI] [PubMed] [Google Scholar]
- 23.Shah SK, Arora S, Skipper B, et al. Randomized evaluation of a web based interview process for urology resident selection. J Urol . 2012;187(4):1380–4. doi: 10.1016/j.juro.2011.11.108. [DOI] [PubMed] [Google Scholar]
- 24.Pasadhika S, Altenbernd T, Ober RR, Harvey EM, Miller JM. Residency interview video conferencing. Ophthalmology . 2012;119(2):426.e5. doi: 10.1016/j.ophtha.2011.09.032. [DOI] [PubMed] [Google Scholar]
- 25.Vadi MG, Malkin MR, Lenart J, et al. Comparison of web-based and face-to-face interviews for application to an anesthesiology training program: a pilot study. Int J Med Educ . 2016;7:102–8. doi: 10.5116/ijme.56e5.491a. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Consent Script and Study Questions Included in the Study Survey
The preinterview score was completed before the interview. The remaining two questions were completed immediately after the interview.
Consent Script
We are performing a study to evaluate the impact of in-person interviews on scoring of residency applicants. The questions which are denoted with an asterisk (*) are for purposes of a study related to resident recruitment and will not be used in any way for scoring purposes for the applicant. This study was approved by University of Chicago’s IRB (IRB19-1663). Participation in this study is completely voluntary, and responses will be blinded for interviewer and applicant names prior to being analyzed. Completion of study questions implies consent for participation in this study. For any questions regarding this study, please contact Sarah Nizamuddin at snizamuddin@dacc.uchicago.edu.
Questions
*Preinterview score: _____ (Please score to 1 decimal point, i.e. 3.2)
Scale for reference:
Post-interview score: _____ (Please score to 1 decimal point)
Scale for reference:
*For each factor below, please rate the level of influence they had on your post-interview score. If a category did not influence you to change your score after the interview, then select “0.” (For example, personality may have had a high level of influence on you changing your score because they had a a) wonderful personality and were very friendly (++) or because they had a b) unfriendly personality and seemed rude at times (--). Alternatively, you might select “0” for personality because they had a “normal” personality and it did not therefore influence you to change your score):
-- - 0 + ++
STROBE Statement
Following is the checklist of items that should be included in reports of cohort studies.
1 – “would not want here”
2 – “a potential challenge to get through”
3 – “passing resident”
4 – “Upper ½ of residency class”
5 – “Offer signing bonus now!”
1 – “would not want here”
2 – “a potential challenge to get through”
3 – “passing resident”
4 – “Upper ½ of residency class”
5 – “Offer signing bonus now!”
Personality/communication/interpersonal skills
Physical appearance (i.e., neatness, grooming, or other)
Professional demeanor
Discussion regarding academic/scholarly activity
Level of interest in the specialty based off interview discussion
Other (optional) _______
| Item No | Recommendation | Page No | |
|---|---|---|---|
| Title and abstract | 1 | (a) Indicate the study’s design with a commonly used term in the title or the abstract | 3 |
| (b) Provide in the abstract an informative and balanced summary of what was done and what was found | 3 | ||
| Introduction | |||
| Background/rationale | 2 | Explain the scientific background and rationale for the investigation being reported | 5–6 |
| Objectives | 3 | State specific objectives, including any prespecified hypotheses | 5–6 |
| Methods | |||
| Study design | 4 | Present key elements of study design early in the paper | 6–8 |
| Setting | 5 | Describe the setting, locations, and relevant dates, including periods of recruitment, exposure, follow-up, and data collection | 6 |
| Participants | 6 | (a) Give the eligibility criteria, and the sources and methods of selection of participants. Describe methods of follow-up | 6 |
| (b) For matched studies, give matching criteria and number of exposed and unexposed | N/A | ||
| Variables | 7 | Clearly define all outcomes, exposures, predictors, potential confounders, and effect modifiers. Give diagnostic criteria, if applicable | 8–9 |
| Data sources/measurement | 8* | For each variable of interest, give sources of data and details of methods of assessment (measurement). Describe comparability of assessment methods if there is more than one group | 8–9 |
| Bias | 9 | Describe any efforts to address potential sources of bias | 9 |
| Study size | 10 | Explain how the study size was arrived at | N/A |
| Quantitative variables | 11 | Explain how quantitative variables were handled in the analyses. If applicable, describe which groupings were chosen and why | 9 |
| Statistical methods | 12 | (a) Describe all statistical methods, including those used to control for confounding | 8–9 |
| (b) Describe any methods used to examine subgroups and interactions | 8–9 | ||
| (c) Explain how missing data were addressed | 8–9 | ||
| (d) If applicable, explain how loss to follow-up was addressed | N/A | ||
| (e) Describe any sensitivity analyses | N/A | ||
| Results | |||
| Participants | 13* | (a) Report numbers of individuals at each stage of study—eg numbers potentially eligible, examined for eligibility, confirmed eligible, included in the study, completing follow-up, and analysed | 9 |
| (b) Give reasons for non-participation at each stage | 9 | ||
| (c) Consider use of a flow diagram | N/A | ||
| Descriptive data | 14* | (a) Give characteristics of study participants (eg demographic, clinical, social) and information on exposures and potential confounders | 9 |
| (b) Indicate number of participants with missing data for each variable of interest | 9 | ||
| (c) Summarise follow-up time (eg, average and total amount) | N/A | ||
| Outcome data | 15* | Report numbers of outcome events or summary measures over time | 9–10 |
| Main results | 16 | (a) Give unadjusted estimates and, if applicable, confounder-adjusted estimates and their precision (eg, 95% confidence interval). Make clear which confounders were adjusted for and why they were included | 9–10 |
| (b) Report category boundaries when continuous variables were categorized | 9-10 | ||
| (c) If relevant, consider translating estimates of relative risk into absolute risk for a meaningful time period | N/A | ||
| Other analyses | 17 | Report other analyses done—eg analyses of subgroups and interactions, and sensitivity analyses | N/A |
| Discussion | |||
| Key results | 18 | Summarise key results with reference to study objectives | 10 |
| Limitations | 19 | Discuss limitations of the study, taking into account sources of potential bias or imprecision. Discuss both direction and magnitude of any potential bias | 11–12 |
| Interpretation | 20 | Give a cautious overall interpretation of results considering objectives, limitations, multiplicity of analyses, results from similar studies, and other relevant evidence | 12 |
| Generalisability | 21 | Discuss the generalisability (external validity) of the study results | 12 |
| Other information | |||
| Funding | 22 | Give the source of funding and the role of the funders for the present study and, if applicable, for the original study on which the present article is based | N/A |
*Give information separately for exposed and unexposed groups.
Note: An Explanation and Elaboration article discusses each checklist item and gives methodological background and published examples of transparent reporting. The STROBE checklist is best used in conjunction with this article. It is freely available on the Web sites of PLoS Medicine at http://www.plosmedicine.org/, Annals of Internal Medicine at http://www.annals.org/, and Epidemiology at http://www.epidem.com/. Information on the STROBE Initiative is available at http://www.strobe-statement.org.