Overcoming Content Specificity in Admission Interviews: The Next Generation?

We read with interest the recent articles related to admissions interviews and appreciate that this has been an area of study and publication.^1,2 We would, however, like to discuss the differences noted among the 2 most recent publications on this subject. To our understanding, the multiple mini-interview (MMI), as described by Cameron and colleagues, seems to describe the next generation of interviewing and an evolution from more traditional interview formats like that described by Kelsch and colleagues. While not stated directly in the Cameron article, content specificity is an important concern for interviewers, for which the MMI format was designed to address. This concern is not addressed with a single-occurrence traditional interview, including those with multiple interviewers. (Admittedly, our college’s current interview process is similar to that described by Kelsch.)

Content specificity has been found within assessment types throughout education and is known to limit reliability.^3,4 Literature discussing content specificity has suggested that little can be done to avoid it confounding results. It has, however, been demonstrated by and is a key concept behind the improved reliability of the objective structured clinical examination (OSCE) format over yesteryear’s oral clinical examinations. In fact, the MMI is simply an admissions OSCE.⁵ Therefore, with larger numbers of MMI stations generally yielding less unreliability due to content specificity, incorporating an MMI seems a current best practice approach to control and minimize this score variability.

As expected, in a recent analysis of our recent PGY1 program interview candidates, we also found this. Using generalizability theory, much like others in medical education have, our interview process had a variability that was explained by various facets. We established 4 separate panels/stations, each consisting of 2 interviewers, and interviewed 24 residency candidates. Analyzing the resulting data, candidates accounted for 74% of the variation (ie, true variance that we want), interview stations for 3.4%, interviewers for 2.5% (ie, inter-rater reliability), and candidate-station interaction (ie, content specificity) for 13.5%, while residual error was 6.6%. Notably, our reliability (g coefficient) was 0.787 and could improve to 0.847 if we had only 1 interviewer and 8 separate interview stations. To compare with Kelsch, our intraclass correlation was 0.832 and Cronbach alpha was 0.868. That is, we had slightly less inter-rater divergence, though this caused only minimal variance compared with other variance sources.

Sinking substantial resources into attempts to alleviate concerns with inter-rater reliability (ie, training) should have perspective; we did not train our interviewers to use our interview rubric for this event. Others have been more condemning of training.⁶ While inter-rater congruency and reliability are important and often highly focused-upon area, literature has shown content specificity to have a larger role in decreasing reliability of candidate or participant performance assessment.

Along with our own college, we greatly encourage our colleagues across the country to progress towards the MMI; content specificity and true reliability may depend on it.