CORR Insights®: What Are the Trends in Research Publication Misrepresentation Among Orthopaedic Residency and Fellowship Applicants From 1996 to 2019? A Systematic Review

Where Are We Now?

The Collaborative Institutional Training Initiative Program (https://about.citiprogram.org/) has required courses for researchers at many institutions. I have taken its human subjects research program for five different institutions, and must retake it every few years. Among its many modules, research misconduct is defined as including fabrication, falsification, and plagiarism. Research misconduct can threaten the public’s trust in science, harms our profession, and causes funders who might support our work to be skeptical. Prior work suggests that misconduct is neither common nor exceptionally rare, with a meta-analysis of surveys estimating that about 2% of scientists admit to having fabricated or falsified data themselves—while also reporting a higher rate in colleagues at 14% [8]. Self-reports of plagiarism are likewise estimated at about 2%, with an even higher estimated frequency of 30% in colleagues [17].

In this month’s Clinical Orthopaedics and Related Research®, Burkhart et al. [5] reported on trends in research misrepresentation among applicants to orthopaedic residency and fellowship. They identified ongoing research misrepresentation among the very people who are seeking to be our field’s future. As the authors note, the high stakes of residency application particularly may generate an environment in which applicants are tempted to fabricate or otherwise inflate their research accomplishments. Similar work in other fields has suggested that specific institutional or country factors, particularly cash-reward policies, were more associated with research misconduct than widespread “publish or perish” concerns [10]. I agree with Burkhart et al. [5] that the structure of orthopaedic surgery residency applications may drive research misrepresentation.

The math of the match for orthopaedic surgery is particularly stark. Last year, roughly half of all applicants to orthopaedic surgery did not obtain a residency position in the field (1739 applicants, 875 positions) [3, 15]. Although some of this may have been related to the pandemic, that ratio has held roughly steady for several cycles. What is in the process of changing, however, is that the United States Medical Licensing Examination Step 1 score is now pass/fail, and medical schools are increasingly moving away from course grades or class ranking. In the very competitive orthopaedic surgery match, the loss of these objective (even if flawed) ways for applicants to distinguish themselves may influence behavior in negative ways and may push some toward misconduct. This situation relates to, and is compounded by, the high number of programs to which applicants have applied in recent years. Reviewing an average of 775 applications per program to decide who to interview is a challenge to the concept of holistic assessment [3]. Research, specifically deliverables in the form of abstract posters or podium presentations and peer-reviewed publications, remains a quantifiable means for comparing applicants. And applicants clearly understand this, as demonstrated by the average of 16.5 research deliverables in 2022 for United States MD seniors who successfully matched [14], and an increasing number of applicants willing to lengthen their pipeline to practice by taking a dedicated research year [4, 7].

Imagine all the years of work that go into putting together an application for orthopaedic surgery residency; I’m sure that many readers won’t have to imagine it—they’ve done it. You know that research deliverables are an important aspect of the application, but they’re not directly within the applicant’s control. You also know that getting more interviews increases your odds of a successful match.

In this month’s CORR® [5], we see the temptation to embellish research accomplishments and then hope (not without reason) that program directors and review committees are too overwhelmed with applications to check reported accomplishments line-by-line is too great for some to resist. This is the reality. The question now is what to do about it.

Where Do We Need To Go?

Research misconduct among applicants is at least partially a symptom of the deficiencies of our current orthopaedic surgery residency application process. We are unlikely to turn back the clock toward scored Step 1 examinations; even if possible, that may not be desirable [17]. The ability of program directors and recruitment committees to do a holistic review—perhaps with spot-checking of research activities—is inherently limited by time. A reduction in the average number of programs applied to would improve the ability to do a holistic review, and more efficiently allow applicants and programs of mutual interest to identify each other. The preference signaling instituted for this cycle of the orthopaedic surgery match is very promising, though data on the impact of preference signaling are limited [1]. There has been a modest reduction in programs applied to per applicant, from 87 to 77 [2]; if applicants get few interviews to programs they did not signal, and those data are made widely available, then the number of programs applied to may drop further in subsequent cycles.

Anecdote is no substitute for evidence. That said, my experience this year as someone who advises medical students and as a residency program director was substantially improved over the past two cycles. We received about 150 fewer applications this year, which hopefully reflects less of a shotgun-style approach. About half of our applicants sent a preference signal. Although all applications were reviewed, there were far more holistically qualified applicants who signaled us than interview positions available. We ultimately only interviewed applicants who signaled their interest in our program. Importantly, medical students I spoke with also felt their interest in signaled programs, even in the absence of geographic connection, was reflected in their interview invites. If preference signaling leads to further reductions in program applications per student, then programs will have more time to review each application. More time for holistic review will help.

As we continue to ask ourselves how best to recruit the next generation of orthopaedic surgeons to join our field, it is worth considering what is measured and what we emphasize. Do we really want people to routinely add years to their training path to do research, with the primary purpose being to pad their application? I don’t think so. Like a well-functioning joint replacement or principled fracture stabilization construct, we ought to design our residency application process to achieve the results we seek. As seen in survey-based research, program directors value away rotations highly [6]. These visiting rotations are useful educationally and professionally, and provide great on-the-ground assessment of potential future residents. How many of these rotations to allow or encourage remains a debated question. Two such rotations, three if a student lacks a home orthopaedic program, certainly seem reasonable. Gaining away-rotation consistency in our recommendations would be a step forward.

A now-arcane piece of conventional wisdom was that a poor Step 1 score would likely keep you out of orthopaedic surgery residency, but a good score would not get you in. Perhaps the same could be true of research, and oft-repeated by medical student advisors: A total lack of research may keep you out of our field, but a ton of research by no means guarantees entry. A corollary could be: Learn and be part of productive meaningful research. More research is not necessarily a good thing, and papers (often low-quality, low-impact work) are being published at an ever-increasing rate [12]. Building verifiable research (or other) skills, such as microcredentials in data science or critical scientific reading, is likely more useful than publishing another low-impact paper. A variety of microcredentials [13] or entrustable professional activity badges [16] may be alternative means of meaningful differentiation.

How Do We Get There?

If you accept the idea that research misrepresentation among orthopaedic applicants is partially a symptom of the competitive and relatively data-poor application process, then efforts to improve the match are worthwhile. Applicants and programs seek reliable useful information. The Orthopaedic Residency Information Network (http://orin.aoassn.org/) is a valuable recent addition, and all residency programs should put as much information as possible onto it for the benefit of applicants. The Universal Interview Offer Date program has reduced the months-long constant stress and email checking among applicants, though clearly some room exists for ongoing improvement [11]. Will preference signaling reduce the application fever over time, allowing for improved holistic review and less research misconduct? Accurate reporting of the numbers of preference signals at each residency program, and the proportion of interviews given to signaled and nonsignaled applicants, would be a step forward.

There are other short-term steps we can take. Research mentorship is thought to be protective against research misconduct [9]. Talk to your medical students and residents about research ethics and the potentially dire ramifications of questionable research activities. For orthopaedic surgeons lucky enough to benefit from the intelligence, hard work, and productivity of medical students, I have another recommendation: Please offer to review their Electronic Residency Application Service applications. In addition to reading a personal statement, look at their research section and provide guidance if needed on how work is reported. It is a shame when, rarely, a well-liked applicant is interviewed but then found to have a clear fabrication on the Electronic Residency Application Service application; this can quickly move someone from likely-to-match to entirely off the rank list.

Research misconduct is a real if uncommon problem that requires mitigative management. I recently retook my Collaborative Institutional Training Initiative Program courses, which I grumble about and delay, yet always learn something from. Individually, as teams, and in decisions we make as a specialty, we can root out questionable research practices while promoting the joys of high-quality scholarship.