Editorial: Should Orthopaedic Residents Be Required to Do Research, or Would Critical Reading Programs Be a Better Use of Their Time?

Sometimes in this space, editors tackle deeply nuanced issues that may or may not have obvious solutions.

Not this month. Instead, here, I'll offer up a practical suggestion that will return dividends to generations of orthopaedic surgeons and the patients they treat: The Accreditation Council for Graduate Medical Education (ACGME) should no longer require that orthopaedic residents participate in research. Instead, residency programs should ensure their trainees graduate with the ability to read orthopaedic journal content with discernment. Additionally, the ACGME should set and monitor standards for training programs on the topic of teaching the critical appraisal of orthopaedic evidence.

The need to be able to draw good information out of a noisy environment has never been more pressing. The number of orthopaedic journals continues to grow: Clarivate, the company that sells the Impact Factor, listed 116 orthopaedic journals in 2020 (the last year for which statistics are available), up more than 50% from only 5 years earlier. Scimago, another indexing portal, lists nearly 300 [17]. The number of predatory journals in the environment now may exceed 10,000 [5]. The proliferation of medical content on preprint servers has only diminished the signal-to-noise ratio [10], to the serious detriment of our patients [9]. There is no way to keep up with this torrent of information.

Fortunately, we don’t have to. Discerning readers don’t read it all; they read what counts, and they read it with a thoughtful, critical eye. To do that requires the pulling of an ever-decreasing amount of signal from an ever-widening bandwidth of noise, the grasping of a few needles from a landscape of multiplying haystacks.

Being choosy is the first step of the critical appraisal process. But many residents never really learn that step, nor the important ones that follow. And because we graduate class after class of residents who have never been exposed even to rudimentary critical appraisal toolkits, we create generation after generation of orthopaedic surgeons who perform ineffective, discredited, and even harmful treatments in large numbers on the basis of anecdotal experience, eminence-based medicine (as opposed to the evidence-based kind), and even advertisements [16].

The examples are too numerous to count. There were important, unanswered safety questions both about metal-on-metal hip replacements and recombinant human BMP, to name just two recent disasters, at the time both of those interventions began to see wide use. That being so, much of the harm those treatments caused should have been preventable. If those examples feel like ancient history already, consider this: About 700,000 arthroscopic meniscectomies are performed annually in the United States [8], knee arthroscopy accounts for the septic arthritis in about 1 in 20 of the patients who develop that potentially devastating condition [4], and the best-available evidence suggests that many of those procedures are not indicated [12].

It doesn’t have to be this way. Numerous toolkits are available that can help learners become discerning readers of original medical and surgical research. Some 30 years ago, JAMA began publishing a series of “User’s Guides to the Medical Literature”, which provide straightforward approaches for the critical appraisal of clinical research of every sort [15]. Full lists of the articles in that series are available widely on the internet [7], and versions of this content even have been bundled into a paperback book that fits conveniently into the white-coat pocket of any resident [6]. Here at CORR®, we’ve summarized the key critical appraisal principles in ways that are easy to digest [11] and created a freely available online critical appraisal tool (click on “Peer Reviewer Tool” on the right-hand side of www.clinorthop.org to access it). Although we designed it for peer reviewers, it works well for any readers who want to practice finding the soft spots in the articles they read.

But my purpose here is not to sell books that I didn’t write, nor even to promote tools offered by the journal I edit. My purpose is to suggest that our lack of success at teaching residents how to read thoughtfully is both important and easily remedied.

Before going too deep, though, it’s important to acknowledge that the typical 5-year orthopaedic training program already is overstuffed. If something big and important—like formal training in the critical appraisal of published research—needs to enter the curriculum, some other ravenous consumer of time and treasure needs to go away. That resource sink is resident research.

Fewer than one in five orthopaedic surgeons work in academic practice [3]; more than 80%, therefore, may never write a paper after graduation. Despite that, the ACGME requires every orthopaedic resident in the United States to participate in sponsored research, and every program must provide each of those residents at least “60 days of protected time” to do it [1]. The ACGME has articulated many standards for resident research experiences, which I won’t bore you with here; suffice it to say there are no similarly stringent guidelines for journal clubs or critical appraisal programs.

In fact, the phrase “critical appraisal” appears only once in the 61-page single-spaced program-requirements document [1], at the end of a long list of other skills, and there are no requirements for resident journal clubs. Although the ACGME’s “Orthopaedic Surgery Milestones” document [14] lists the critical appraisal of evidence as an upper-level resident skill, without infrastructure, resources, or standards for teaching that skill, it’s not obvious to me how residents might develop it. It’s my observation that most don’t.

To remedy this, I believe the ACGME should begin with specific, detailed guidelines for programs that set standards for how to teach critical appraisal. The ACGME might work with the American Board of Orthopaedic Surgery (ABOS) to make sure that these skills are tested in place of some of the musculoskeletal minutiae that turn up on written and oral board examinations. Developing such guidelines would not be difficult, and probably should be done at the core ACGME level, rather than at the subspecialty level, since such standards would generalize well across surgical and medical specialties.

As an important aside, the ABOS seems to be more forward-thinking on this. The ABOS allows its diplomates to maintain certification using an annual journal-based assessment process that replaces the high-stakes once-a-decade trivia contest that once was the normative way to recertify. In the new pathway, test takers read 15 articles for content retention each year. If I could tweak that process but a bit, I would suggest only that the ABOS add a critical appraisal element to it—for example, ask test takers what the likely impact on a study’s results would be if a large proportion of its patients were lost to follow-up, or how to tell the difference between a finding that is “statistically significant” and one that is clinically important [13]. I also believe the ABOS could easily step into a helpful role here in terms of incenting residents to become better readers; a critical appraisal toolkit would fit beautifully into the ABOS’s existing “Residency Skills Modules” [2].

Critics will say that doing research teaches a resident how to read research. While to some degree this may be true, it’s inefficient. If the resident’s project is to evaluate a surgical treatment, that resident will, for the most part, read other studies about therapies and perhaps some lab studies, but probably few or no studies about diagnostic tests, studies about exposure, risk and harm, or studies of myriad other designs that are important to get comfortable with. To learn critical appraisal, you have to do critical appraisal, and—like any other complex but important skill—do it under the guidance of those who do it well, using teaching tools that work, such as those I’ve mentioned.

Concerned academicians may worry that my suggestion will eliminate research from training programs, but I think it will have the opposite effect: It will allow those residents who are most interested in research and those faculty members with the most robust research programs to connect. This approach gives those who want to go deep a better experience and lets the others—and their sponsors—focus on things that will help them far more in the long run. It also frees up time and resources to set all learners up with the critical appraisal skills that will support a lifetime of learning and safe practice.

Research for all is a low-value activity that takes tremendous amounts of time and money. Critical appraisal for all is less expensive, more educationally valuable, and ultimately better for the patients whom our graduating residents will treat.

The ACGME should drop its research requirement and increase its focus—and its program requirements—on teaching residents how to be more effective consumers of clinical studies published in orthopaedic journals.