Not the Last Word: Roll Them Bones—Selecting Orthopaedic Surgery Residents by Lottery

Robert H. Frank PhD, an economist at Cornell University, likes to tell the story about how he died suddenly in November 2007 [7].

Dr. Frank was playing tennis on campus when his heart stopped. Because the tennis courts are five miles from the ambulance dispatch center, the story of Dr. Frank’s sudden death should have been told by the newspapers, not by the man himself. But Dr. Frank was lucky. Moments before he went down, an ambulance crew reported to the scene of a crash nearby, only to discover that its services would not be needed. This ambulance, returning to the dispatch center empty-handed, happened to pass the tennis courts as the call for help was aired. The crew arrived straightaway and shocked the professor back to life.

Dr. Frank’s story prompted him to study the role of random events. He found that almost invariably, success requires luck—maybe not to the extent he experienced, but at least somewhat. Frank thus asks his readers to recognize that “chance plays a much larger role in important life outcomes than most people imagine” [7].

For Dr. Frank, a greater awareness of chance suggests changes in tax policy—that the “lucky” should be more willing to support those less fortunate. For me, a greater recognition of the role of luck points to a better way of selecting residents—namely, using a lottery.

At the least, I advocate using an explicitly random process to choose the applicants invited for an interview.

To be sure, applications must be screened, and only those deemed qualified would participate. But if the appropriate thresholds are crossed [5], the applicant gets entered into the lottery pool.

Grades and scores are given far too much weight in the current residency selection process. All students smart enough to get into and out of medical school are smart enough to make it in orthopaedics. They may be missing something, but intelligence is not it. Recall that only two or three generations ago, orthopaedics was not a particularly popular career choice. Medical students from the middle of the class were welcome and the field blossomed nonetheless.

Beyond that, although a stellar record implies a high IQ and conscientiousness, brains and grit are hardly the whole story. So many other important attributes—empathy creativity, and integrity, to name but three—are not captured by test scores.

Exceptionally high scores can also divert attention from other deficiencies. As a baseball fan years ago, I admired Reggie Jackson and the home runs he hit far into the upper deck [10]. Of course, clearing the fence by one foot or 100 puts the same number of runs on the scoreboard, and Mr. Jackson, as I tend to forget, but his detractors are quick to remind me, struck out more than any player in history.

Moreover, many of the distinctions on which admissions decisions are based are not statistically meaningful.

At bottom, programs that use a lottery will be better off. Trying to discern the absolute best from a large pool of more-than-adequate alternatives is a good, general purpose recipe for misery [12]. Yet there are specific, practical benefits as well.

A lottery markedly decreases the effort needed to find qualified candidates. Yes, the applicant who writes a paper in Nature describing her discovery of a new bone or ligament can be selected for an interview without much additional thought. But cases like that are rare. Most candidates are superficially indistinguishable, and any program aiming for fairness must invest in a time-consuming (and possibly contentious) analysis of applications. Programs will be ever more [9] compelled to find fast and fair ways to select candidates for interviews. A lottery will do it.

(I’d be willing to settle for a system in which half the interview slots are distributed by lottery. Currently employed fast and fair rules, such as “interview our own students and all visiting students who rotate here”, can gobble up many of the available spaces. And I will save my proposals for improving the interview process for a future column.)

Beyond that, a lottery can also increase racial diversity, as today’s excessive screening thresholds [11] excludes capable students from underrepresented groups [13].

A lottery will also be liberating. Raj Rao’s advice [3] to applicants, to avoid undue attachment to a particular residency program (and the suffering it brings), applies to programs as well. A program director who can blame the lottery for selecting a bad resident will more readily dismiss that resident when that’s needed; a program director who personally selected a failing resident may discover that ego defenses impedes timely recognition of the resident’s deficiencies.

But so much for the logical arguments. If you are a decision-maker—a program director or chairman who has the power to determine how residents are selected—I implore you to think about your own experience. Can you identify an instance where luck helped you get where you are? Can you think of a classmate, just as talented as you but a little less fortunate, who missed the boat accordingly? I am sure you can. Luck plays a large part in all of our lives and the world is a better place when we acknowledge that fact. So, ditch the status quo and lead. Offer at least some of your interview invitations by lottery.

Cato T. Laurencin MD, PhD

Albert and Wilda Van Dusen Distinguished Endowed Professor of Orthopaedic Surgery and Chief Executive Officer, The Connecticut Convergence Institute for Translation in Regenerative Engineering

The University of Connecticut

Orthopaedic surgery exists in a world with an embarrassment of riches. We are privileged to be in an environment where we can develop and work with new technologies like advanced arthroplasty, next generation biologics, and techniques in clinical care and rehabilitation that achieve tremendous outcomes for our patients. It is no wonder that orthopaedic surgery attracts the best and brightest young people to our field.

But the system is broken.

Large numbers of medical students compete for far few places. At great expense, some students apply to more than 100 programs, many obscure to them, in the hopes of securing a coveted spot. Many medical students spend multiple months on away rotations to help them in the orthopaedic surgery admissions process. Often, this takes away valuable time from achieving a comprehensive medical school education.

The system must change. Some—including Dr. Bernstein—have suggested a lottery system for selecting residents, which may be a choice of last resort. Before that, some constructive changes to the system should be considered. I present below a five-point plan.

1. Medical students can apply to a maximum of 40 residency programs. The current system has encouraged students to apply to upwards of 150 residency programs [4], and many if not most, apply to more than 100. Residency directors, faced with a burgeoning number of applicants, choose who they know—their own students or those that rotate through their hospitals. There is also pressure to take students from home programs knowing the intensity of competition and that other programs are being parochial in their decisions. By limiting the number of applications, students choose programs they are truly interested in. Residency directors know that with a limitation on numbers of programs, the students that apply to their programs have a true interest and desire for their program. The number of applications per program will drop, and a more careful and holistic review of applications will be possible.

2. Increase the number of residency positions. The number of residency positions in the United States has not kept pace with the numbers of medical students completing school and applying for spots [8]. This is unsustainable. There must be across-the-board policy changes to increase residency positions, especially in view of the historic expansion in numbers of allopathic and osteopathic schools, and the expansion in medical school class size taking place in established schools. This will necessitate a concerted effort by organized medicine, but should also be the responsibility of all in the academic community to recognize the issue and act accordingly.

3. Create 2-week away rotations. The current medical student away rotation is 1 month. In my experience, a two-week period is sufficient for providing a meaningful focused educational experience for the student, allowing the student to assess her/his appeal for the program, and giving faculty the opportunity to offer a meaningful assessment of the student. There are important by-products of creating 2-week away rotations. Under current scenarios, students would have the ability to visit twice the number of locations that they currently visit, giving a wider range of programs from which to gain an educational experience. At the same time, residency directors will be under less pressure to interview all rotators since the choices for students have been doubled through the shortened rotations. Finally, students may be encouraged to gain a more-comprehensive medical education with less pressure to undertake multiple months of orthopaedic rotations.

4. Rethink scores. Board scores were originally meant to assess whether students have met competency levels. They are now used by many programs as a filtering tool. The use of board scores as a major factor in decision making is unfair for several reasons including the fact that students passing the exam do not have a second opportunity to improve their scores. Scores should be grouped into top 5%, bottom 5%, and those in the middle. No student should be filtered out of a program selection process if they lie in the middle. Orthopaedic surgery applicants already have average scores at the top of the profession (along with dermatology, otolaryngology, and plastic surgery). Therefore, we have an opportunity to create a more-holistic admissions programs recognizing that applicants to our programs on average have very strong scores.

5. Promote diversity. Orthopaedic surgery programs are late to the game, but the value of diversity in medicine is undeniable. Our ability to develop new solutions for our patients, even new science will hinge on our having diverse teams. Without action by residency directors and faculty in turning the tide and promoting diversity, the future of our profession may be in question.

Andrew R. Spector MD, FAASM

Associate Program Director, Neurology Residency

Duke University School of Medicine

In a casino, the house wins by giving gamblers the illusion of control. I can hit or stand trying to reach 21, but the outcome is eventually determined by the randomness of shuffled cards. I use the data that are available at the time, but I really don’t know what card I’m going to get when I hit. Try as we might to analyze a residency application, maybe we’d actually do better by leaving it to chance.

Reviewing a residency application is like playing cards with all the cards face up on the table. We get to see everything. We have grades to review, publications to count, and letters of recommendation to pore over. The question in our case, though, is how reliable are the data upon which we make our interview decisions?

Some medical schools grade their rotations pass/fail. Other medical schools have highly inflated grades. Supervising physicians want to see their students match successfully, and thus are disincentivized to say anything negative on an evaluation lest it appear on a Medical Student Performance Evaluation (MSPE) and impede a top match. Furthermore, evaluations of rotation performance could be based more on likeability than skill or knowledge. Application reviewers have little hope to understand what any given grade might mean.

Letters of recommendation are also fraught with problems. I’ve never seen one that says anything bad about a candidate. We are left to decipher which letters are just good and which ones are really good. We also know that letters can be gender-biased [14].

After recognizing the problems inherent in an applicant’s medical school transcript and letters of recommendation, the reaction is naturally to revert to standardized testing. Unfortunately, United States Medical Licensing Examination® (USMLE®) Step 1 scores stratify by race with students from racial backgrounds that are underrepresented in medicine (black/African-American, Hispanic, Native American, Hawaiian/Pacific Islander) performing worse, on average [6, 13]. Meanwhile, there is no evidence that a slightly lower scoring black applicant will perform any worse as a surgeon than a slightly higher scoring applicant of another race. The USMLE® Score Interpretation Guidelines are explicit: “Small differences in Step examination scores alone should not be used as the basis for selection decisions …” [16]. Yet, when we implement arbitrary USMLE® cutoffs as a method of screening applicants, this is exactly what we are doing.

There is no algorithm that can take bad data and make a good decision, and applications are full of bad data. But is a lottery really the best alternative?

Dr. Bernstein identifies the primary flaw in the lottery system: “To be sure, applicants must be screened, and only those deemed qualified would participate.”

The trouble is that we don’t actually know what makes someone “qualified.” As any survey of program directors will tell you, everyone defines “qualified” differently. And if being qualified means high Board scores, good grades, or strong letters, we are right back where we started trying make sense of inconsistent and potentially biased data.

The other concern with a lottery system is that it would maintain the status quo at a time we are trying to make residency more diverse. In orthopedic surgery, only 16% of applicants [1] and 19% of faculty are women [2]. Dr. Bernstein suggests that since so many candidates are indistinguishable, we should use a lottery to pick the interviewees. In that scenario, women would be expected to receive 16% of the interviews. To move the needle on diversity, though, we need to be interviewing a higher percentage of women who meet Dr. Bernstein’s minimal qualification threshold, thereby increasing the pool of women who are eligible to match. A pure lottery system would not do that.

A lottery certainly has merit. It would save a good deal of time and stress on the part of reviewers, and it would reduce dependence on flawed data that we use to make highly subjective decisions. However, if we really want to improve the residency selection process, we should start by improving the data. For example, MSPE should be required to list students’ weaknesses along with their strengths. Additionally, we could implement structured letters of recommendation rather than narrative. Finally, USMLE® score reports are already facing heavy scrutiny and seem likely to be revised soon [15]. Whether we use a lottery or not, we will always be faced with the dilemma of declaring some applicants “qualified” and others “not.” Once we sort this out, we can leave chance to the gamblers.