Not the Last Word: A Joint Registry for Resident Selection

Faculty invest countless hours in residency selection—reading applications, interviewing candidates, and debating rank lists. Yet there’s an irony to all this effort: Despite their commitment to evidence-based medicine in clinical practice, these surgeons do not routinely apply the same data-driven rigor to residency selection.

Contemporary studies on residency selection bear an uncanny resemblance to surgical investigations in the time before Ernest Avery Codman revolutionized clinical research by establishing “end result” evaluation as the standard [3]. In the BC (Before Codman) era, surgeons evaluated their work by weak proxy metrics. The Halsted radical mastectomy was deemed well-performed if the surgeon removed every last fiber of the pectoralis minor. The necessity of surgery, to say nothing of this radical approach, wasn’t questioned. Today, we celebrate recruiting students with high USMLE scores and research productivity much as Halsted celebrated complete muscle removal—with the same omission of clear evidence that ephemeral success in hitting those marks predicts enduring success in the long run.

We must ask and answer better questions. How many residents did we select who will not meet the needs of our future patients? How many promising candidates were turned away? And beyond these issues of false positives and negatives, we must also ask, at what cost—in time, resources, and opportunity—does all of this come?

Examining the issue of false negatives is essentially impossible. At the least, there is the intractable problem of confounding: Those rejected are not preserved in amber for future study, but rather go on to live their lives elsewhere. But we must start somewhere. While identifying long-term false positives may be difficult, short-term failures are easier to pinpoint. Programs must perform root cause analyses whenever a resident fails out, drops out, or simply peters out, producing progressively weaker work over time. From these analyses, unifying themes might emerge, such as “many publications, but second-tier clinical rotation grades,” or “really jumped on the rank list after interviewing.” Research productivity is not necessarily correlated to clinical skills and may even be negatively correlated. Similarly, charisma, good looks, or a shared hobby may endear candidates to their interviewers, but don’t speak to their potential as surgeons.

The question of efficiency can also be addressed. At present, applicants are expected to engage in an arms race of research projects, auditions, USMLE extremism, and other “pick me!” antics. For many applicants, that entire adventure can be a glorious waste of time and money. These daunting demands may also discourage otherwise excellent candidates from applying.

We need data to drive improvement. We could start with an investigation about how expensive the process is. What if good residents could be selected without any of the current hoopla of hoop jumping? I’d love to see an intrepid program “rank to match” one or two applicants on a (mostly) random basis, track their outcomes, and let us know.

A simpler step would be to examine the effectiveness of the interview, following a method suggested by Schenker et al. [4]: rank all applicants before the interviews, conduct the interviews, and then rerank them. A comparison between the two lists will reveal the value of the interview. My guess is that most candidates won’t move much—and the ones who move a lot in the upward direction are particularly perilous (see “charisma” above).

Still, we need to think bigger than that.

To address the systemic shortcomings of residency selection, I propose a “Joint Registry” for resident selection, modeled after orthopaedic implant registries. This anonymized database would track progress metrics—OITE scores, internal evaluations, ABOS certification—as a function of the information gathered during selection. Over time, it would reveal which criteria predict success and identify patterns of underperformance.

To be sure, high OITE scores, excellent evaluations, and earning an ABOS certificate are not the ultimate outcomes of interest. The purpose of residency is to train individuals who contribute meaningfully over the course of a career, and studying that is doubly challenging. First, there’s the issue of time. If Solon’s wisdom applies—“count no man happy until he has died”—then we might say, “count no resident successful until he or she has retired from practice.” That would require one very long-term study!

Second, there will always be reasonable disagreement about what constitutes success. Motivated reasoning might lead me to define success in terms that look suspiciously like my own career trajectory, but there are at least 39 other valid perspectives—and that's in my 40-person department alone. That said, history shows that progress in defining meaningful outcomes often begins with other, imperfect measures. For example, the study of end results in breast cancer treatment began with survival alone, but evolved to include cosmesis, function, and now even patient satisfaction. Similarly, the process of studying medium-term outcomes in residency could open the door to examining other, better endpoints as consensus on their importance emerges.

Such a registry would require collaboration among programs—a “joint effort,” so to speak—to achieve sufficient sample sizes, but its potential to replace outdated proxies like USMLE scores with evidence-based metrics will make it worth it.

While such a project faces logistical, legal, and cultural challenges, it has the potential to bring evidence-based rigor to what is now a subjective process; at least it will be possible to perform a SWOT analysis on it (Fig. 1). On the other hand, continuing the status quo with our selection process seems like performing one radical mastectomy after another—possibly futile, harmful, and costly. It’s time to apply the evidence-based standards we demand in clinical medicine to the critical task of residency selection.

Fig. 1.

A SWOT (strengths, weaknesses, opportunities, threats) analysis of the proposed joint registry for resident selection.

Raymond W. Liu MD

 

Victor M. Goldberg Professor, Pediatric Orthopaedics, Rainbow Babies and Children’s Hospital, Case Western Reserve University

The residency selection process remains challenging and resource-consuming, both for applicants and residency programs. Dr. Bernstein’s proposition is insightful and bold: The development of a national registry could provide essential data to better understand and improve this process for both parties. As with any data registry, there are two key questions regarding which outcome parameters will be assessed and which predictor variables will be used to correlate with those outcomes. For outcomes, one option would be to have multiple key educators score the chief residents of each graduating class. Although this is still an imperfect assessment tool, differentiating graduating residents can help us identify factors associated with performance during residency [5], and doing this on a prospective basis may provide strong outcomes data.

Identification of predictor variables is equally, if not more, challenging. Most assessment tools—such as USMLE test scores, clerkship grades, letters of recommendation, AOA status, and even research accomplishments—largely reflect individual academic performance. Upon starting internship, these students transition to a work environment, which involves a different set of skills for working in teams. Key among these are the “soft skills,” such as professionalism and interpersonal skills, which are often the key issues with a resident who is struggling.

Theoretically, letters of recommendation could help us assess a student’s skills that are relevant to residency. Unfortunately, the attempted standardization of such letters has been met with the challenge of substantial score inflation [1]. And in the narrative portions, the intention of the letter-writer often is not understood by the review committees [2].

One solution already in practice is the away rotation. After 4 weeks, programs can feel more confident about a medical student’s ability to function in clinical training, and the student has a much clearer understanding of the culture and expectations of the residency program. Unfortunately, students can only do a few away rotations, which creates a situation in which programs know a lot about a few students and very little about most of them, with similar patterns in terms of what students know about programs. A potential solution would be to have shorter, 1 week away rotations. However, these would present substantial logistical challenges for both parties, particularly in terms of organizing dates, housing, and onboarding. In addition, programs may not have a clear assessment of an applicant after only a week of interaction.

A different approach would be a circuit away rotation. Three residency programs that are relatively close in proximity and have similar cultures could partner up and have medical students spend approximately 9 days at each institution. This would allow enough time for students to take two or three call nights. Partnering between institutions could simplify scheduling, housing, and even onboarding. At the end of each circuit, chief residents from each institution could meet virtually to pool feedback about the performance of each student. In the end, programs would have key information on a much larger number of applicants, and applicants would have the opportunity to learn about a much large number of programs. If programs shared their information with a national registry, it would be possible to conduct statistical analysis, resulting in a national ranking system on a relatively level playing field. Although such an approach would present a large administrative burden for each residency program, I hope this would be offset to some degree by decreasing the amount of effort necessary for the selection process, as well as potentially decreasing the number of residents who are not a good fit with the program, saving everyone substantial time and effort.

Mara L. Schenker MD, MBA

 

Professor of Orthopaedics, Emory University School of Medicine, and Associate CMIO, Chief of Orthopaedics, Grady Memorial Hospital

Dr. Bernstein’s idea for a registry to improve how we select residents couldn’t have come at a better time. After 15 years of reviewing residency applications (and going through the process myself), my observation is that not much has changed. We still rely on subjective or flawed metrics, like gut feelings, USMLE scores, or research productivity. It’s time to find and focus on data that actually predict success. I fully support the joint registry as proposed by Dr. Bernstein, and I’d even go one step further: We need to start collecting quantitative data on surgical competency. This wouldn’t just help us pick the right people, it could also transform how we train them.

Imagine if we had reliable, objective, validated tools to assess an applicant’s ability to succeed as an orthopaedic surgeon. Away rotations, while valuable, are far from perfect. They provide a chance to evaluate fit and grit—qualities that are harder to fake over a month than a 1-day interview—but even these in-person “audition” rotations have limits. And we don’t know if the traits these away rotations often reveal in students, like teamwork and adaptability, truly predict long-term success.

Dr. Bernstein is right: Defining failure is easy. We know what it looks like when someone doesn’t make it through residency or struggles to provide safe care. But success? That’s harder to pin down. Is it technical skill, leadership, or simply the provision of high-quality, compassionate care?

A registry could help us answer these questions. Collecting objective data on metrics like board pass rates, patient outcomes, career satisfaction, and academic and leadership contributions could help us uncover what predicts success, not just in residency, but over a career. It could also highlight areas we’re falling short in: Are we overlooking great candidates who don’t check traditional boxes? Are we overvaluing things like research productivity, which may not correlate as well as other things?

Building this registry won’t be easy. Programs will need to collaborate, and there will be legal, logistical, and cultural hurdles. But the payoff—a fairer, more effective selection process and better, happier surgeons—is worth it.

Our profession prides itself on innovation and evidence-based practice. This is our chance to lead by example. It’s time to stop relying on guesswork and gut instincts, which amplify implicit biases that can unfairly disadvantage certain candidates and perpetuate inequality. Whether through a registry or other approaches, we owe it to future residents—and their patients—to make residency selection and training more evidence-based and personalized.