Table 3.

Themes and Representative Quotes

Theme	Quotes
Screening applications and offering interviews: A standardized process flawed by subjectivity
• Subjectivity in interview invitation decisions	A hundred applications for two spots is pretty competitive. And so the first cuts, so to speak, is really to get down to 30–40 applicants. So with a hundred, it’s about cutting 60%, essentially. Sixty to seventy percent essentially. And so that’s the hard thing. And that’s where it comes in with bias and stuff, because there’s not a great way to do that other than the application. And as objective as we try to make it, it’s really hard because there’s not really a set objective data that says that if an applicant has X, Y, or Z, then they’re going to be successful.
• Differential weights to sections of the application	I think that having the way that the composite score has been developed for us, that relegates the weight of the test score and the med school grades to a very small proportion of the overall score, was the approach that we chose to take to try and mitigate that implicit bias.
• Coded application language	We all just have this language we use. It’s like if someone’s great, you give them the highest level of recommendation, outstanding, just all of these accolades you try and fill in there. And then if you have someone who’s not as great, you’re like, “Really improved over the course of residency, was eager to learn, did well,” so just like less enthusiastic qualifiers is how we read between the lines of like, “Oh, they don’t actually think this person’s amazing because if they thought they were amazing, they would say they were amazing. They think they’re okay.”
• Variable definitions of applicant red flags	So what are some red flags? They’re almost anything. Somebody who emails and communicates too often. Like, that’s a red flag. Somebody who’s too desperate and pushing too hard to try to get into the program. That’s an issue. People who, I’m unhappy to say it, but people had to repeat a year of medical school. I mean, that’s not a red flag. That’s just a fact, right? People fail their steps. People who had to take a leave for medical reasons. Even though that could be something as simple as, “I had pneumonia.” More often than not, it’s a mental health issue, and those usually get worse during residency instead of better. And I know I’m not supposed to pay attention to all those things, but you can’t help but notice there’s something, even though I don’t know the details, if somebody’s gone for a month for whatever reason, I’m like, “Something funny.”
• Perception of training programs and their clinical experiences	Residency quality...it talks about is this a university based program with a lot of exposure or is this a hospital based program that maybe doesn’t have all the resources kind of thing and that metric in particular I think we kind of couch a little bit on just because we wanna, it’s both an exposure thing, but it can also bias us, right? Because like, some applicants aren’t necessarily gonna end up in university based things, so they can’t really be held accountable for the fact that maybe they don’t have the best exposure.
Interviews: Personality matters
• Conversational interviews focused on shared interests	I like that idea of structured questions ’cause I think it takes bias out of it because if you just talk about your shared experiences, then you’re going to really, you might miss out on people who you know you might, downgrade folks who don’t have a shared experience as don’t have the share similar shared experience with you and stuff.
• Differences in perception of applicant personality	There’s some interactions that people have that are very different from other people’s interactions. And so sort of weighing those interactions...we get currently Zoom meetings for three and a half hours to sort of try to figure out somebody’s personality and we have a big stack of paper. And so I do sort of personally take any sort of flags that anybody has in that time period as pretty concerning. If somebody can’t cover up something over a three-hour interaction, then how often is that going to come up in real life?
• Interpretation of communication and interpersonal skills	Academically everybody’s going to have strong applications, but it’s kind of their interpersonal skills that we’re looking for. So any feedback that the fellows can give is really helpful and they are part of our selection meeting. And that also goes for... our fellowship coordinator, who does a lot of the, she is the person probably who interacts the most with the candidates and for 90% of them, she has nothing to say. And then she’ll definitely highlight a couple that she thought worked with her incredibly well or incredibly poorly. And it really does mean a lot to us when we hear that. I pay clear attention to...their social skills, because a pediatrician has to be able to talk to 2-year olds, to 8-year olds, to teenagers, to parents, to grandparents. You got to be smooth in conversation. And there are people who aren’t. And that, to me, is another red flag. So if there’s a 265 on their Steps, but they cannot make eye contact, they can’t pay attention, they seem easily flustered, then the parents aren’t going to trust them, the kids aren’t going to trust them.
• Burden of travel for in-person interviews	There’s some real socioeconomic barriers to the interview process. I mean, there’s socioeconomic barriers all throughout medicine, but the interview process is really one of them... it costs me a lot of money and if I get the interview and I fly out there and I get a hotel room for the night and all of these things, it’s prohibitive. It’s completely prohibitive. I was very fortunate that I was able to go on every interview I was offered, and that cost a ton of money. But lot of people don’t have that privilege. They don’t have that, that good fortune.
Determination of the rank list: The rise and fall of applicants
• Applicant movement based on subjective feelings	We have them write in feedback, but that’s really the first time that they’re like, “Oh yeah, that person was a total weirdo or something, or oh, we really loved her.” So then sometimes we move people around based on that. I see her point about not having people move too much because we’re trying to use this fairer system. But there has been some movement with people, absolutely. Not the way it used to be, where somebody would be number three and then they’d be number 40. We were wondering if that was fair. Because it was a lot of, “I liked that.” It was just a little bit too much, we didn’t think it was fair.
• Variation in methods to create initial rank list	We’ll have the morning of interviews. Then, everyone who interviewed meets after and we develop this running rank list. So people weigh in and then we plug people in as we go, and we always make changes at the end, but we plug people in as we go.
• Group selection committee decisions versus Program Director only decisions	I cut out everybody else who could possibly be biased, and I just did it myself. So, that’s one process. That’s probably the big one, honestly.
• Role of faculty sponsorship of applicants	If someone I trust is like, “This person is fabulous,” that means the world to me.
The recruitment process: Values, challenges, and diversity
• Diverse workforce reflective of patients served	One important piece of that is starting with the people who are providing the healthcare and the experiences that those people bring into their perspective. So it’s very important to me that we actively recruit people from all backgrounds, and in particular underrepresented in medicine backgrounds, because our patient population comes from a racial pool that is part of the underrepresented in medicine group.
• Geographic location	We are very racially divided city. That has been baked into the history of Saint Louis since it was founded, and there are major problems with the way that the health care system integrates within the community.
• Data driven reflection	I think the data really show us how we’re doing. And so by reviewing that, it’s actually really informative for me to go through and with the process and identify any red flags...I also look at other metrics. So how did we do in terms of underrepresented medicine and BIPOC etcetera, in the different categories? Did we match better in this category or not? And then try to think about how we can do better to recruit.