Editor’s Spotlight/Take 5: Nationwide Inpatient Sample and National Surgical Quality Improvement Program Give Different Results in Hip Fracture Studies

Almost 100 years ago, an orthopaedic surgeon born in the 19th century wrote of our professional responsibility to study and record what he called patients’ “end results” [4]. Despite his high level of accomplishment and stature as a full-time surgeon at the Massachusetts General Hospital, Ernest Amory Codman MD was ostracized for his beliefs (and perhaps for his candor) [2, 5].

But we now know that his candor, shocking for its time, was prescient. Among other things, Codman wrote:

“Hospitals, if they wish to be sure of improvement,

1. Must find out what their results are.

2. Must analyze their results, to find their strong and weak points.

3. Must compare their results with those of other hospitals…” [3]

Dr. Codman’s dreams — outcomes research and registries — now are familiar tools to us. More recently, orthopaedic clinician-scientists have mined large public, nonprofit, and private-payer databases such as the National Surgical Quality Improvement Program (NSQIP) and the National Inpatient Sample (NIS).

According to the American College of Surgeons, which runs the program, the NSQIP is “the leading nationally validated, risk-adjusted, outcomes-based program to measure and improve the quality of surgical care in the private sector”; they assert that the program is “so effective that EACH YEAR [emphasis theirs] a hospital uses it, on average, it has the opportunity to: Prevent 250-500 complications, save 12-36 lives, [and] reduce costs by millions of dollars” [1]. The prospectively maintained NSQIP database, which makes up a part of this program, is fed by a sample of patients from participating institutions, and tracks results out to a month after surgery.

The NIS, by contrast, is part of a “federal-state-industry partnership” sponsored by the Agency for Healthcare Research and Quality. Its website states that the NIS “is part of a family of databases and software tools developed for the Healthcare Cost and Utilization Project. The NIS is the largest all-payer inpatient health care database in the United States, yielding national estimates of hospital inpatient stays. Unweighted, it contains data from approximately 8 million hospital stays each year. Weighted, it estimates roughly 40 million hospitalizations” [6].

What could be wrong with any of this? A database is a database is a database, right? Seemingly not. In this month’s CORR^®, Jonathan N. Grauer MD’s group at Yale University drilled into both sources to ask some basic questions about hip fractures in “Nationwide Inpatient Sample and National Surgical Quality Improvement Program Give Different Results in Hip Fracture Studies.” They found that while hip fracture patients were demographically similar in both databases, and lengths of stay looked nearly identical, the presence of key comorbidities were off by more than a factor of two, with some being higher in NIS, and others in NSQIP. More troublingly for the casual reader, inpatient complications were twice as high in the NIS, but because the database does not capture data after discharge, NIS underestimated postoperative deaths by more than half relative to the NSQIP.

This matters to you whether you practice orthopaedic surgery or if you study it. Read the eye-opening report from Dr. Grauer’s group in this issue. These databases are readily accessible to curious clinician scientists, and we are seeing manuscript submissions from these sources with increasing regularity; presumably most other journals are, as well. You will see more and more papers drawn from them, because it is possible to ask — and answer — big questions using these tools. It also is possible to be badly misled by results incautiously drawn from these sources. Caveat lector.

Two authors of this report, the lead author Daniel D. Bohl MPH, and the senior author Dr. Grauer (Fig. 1), highlight what we should watch out for — and point to what we can learn — when using these new tools, in the “Take 5” interview that follows.

Fig. 1.

Mr. Bohl (L), Dr. Grauer (R), and the rest of the Yale University team found that while hip fracture patients were demographically similar in both databases, the presence of key comorbidities were off by more than a factor of two, with some being higher in NIS, and others in NSQIP

Take Five Interview with Daniel D. Bohl MPH and Jonathan N. Grauer MD, authors of “Nationwide Inpatient Sample and National Surgical Quality Improvement Program Give Different Results in Hip Fracture Studies”

Seth S. Leopold MD:Congratulations on some excellent work, and thank you for sending this fascinating paper to CORR^®. How did you get interested in using these large databases as a research tool?

Jonathan N. Grauer MD: A few years ago, my research team and I noticed a small number of studies being presented at national conferences and appearing in the literature that drew on large national databases. I was impressed by their large sample sizes and useful results. Concurrent with this, a member of my lab noted that he had been involved in such studies in a previously held general surgery position. I did some investigation and found out that we could obtain access to several such databases. Since then, my team and I have utilized these databases to address questions we believe are clinically important.

Dr. Leopold:Can you provide a short list of the major available databases from which we will be seeing more research in years to come, and tell us just a little bit about the populations each one includes?

Daniel D. Bohl MPH: The most commonly used inpatient database in orthopaedics is the NIS. This database has been around since the 1980s and has been used in orthopaedics now for about 10 years. Its use has increased substantially in just the last couple of years. Patients are selected into NIS through a national sampling of hospitals across the country.

There are other inpatient databases. For example, United Health Care has made a subset of their inpatient orthopaedic data available through a private data-analysis group named Pearl Diver. Similarly, the Centers for Medicare & Medicaid Services Standard Analytic Files are available from the federal government. The disadvantage of each of these databases compared to NIS is that they are restricted to patients with a particular type of payer (Medicaid, Medicare, or a specific private insurance company).

These inpatient databases all derive their information from retrospectively identified inpatient reimbursement claims. However, there is an exciting new breed of databases, pioneered by the American College of Surgeons, which identifies patients prospectively. Of these, the most commonly used in orthopaedics is the NSQIP. Recently, the American College of Surgeons has added a pediatric version of the NSQIP, as well as a National Trauma Data Bank. For each of these databases, the patients included are a representative sample of patients from across the United States — and, like NIS, in these databases there is no restriction based on the type of payer.

Dr. Leopold:What kinds of questions do you believe can be answered using these databases that have been inaccessible to us using other research tools like single-center trials, multicenter collaborations, and national registries?

Dr. Grauer: There are three types of questions for which database studies are particularly well suited. The first type of question involves rare postoperative outcomes. By achieving sample sizes not possible through single-center trials or multi-center collaborations, database studies can identify risk factors that smaller studies cannot.

The second type of question involves generalization of results to the United States population. Databases like NIS and NSQIP use high-quality sampling techniques intended to identify populations of patients believed to be representative of the US population of surgical patients. By doing so, national databases enable investigators to generate accurate national estimates and perform valid investigations of trends over time.

The last type of question involves the difference between performing a treatment in the controlled setting of a randomized trial and performing it in real community practice, where most orthopaedic treatments are actually performed. Because these databases sample community hospitals, in addition to academic ones, the procedures can be analyzed as they take place in the real world, without the biases that may arise from the controlled setting of a research trial.

Dr. Leopold:What kinds of questions are not well addressed using these tools?

Dr. Grauer: Many of the available national databases (including NIS) examine postoperative outcomes only during the inpatient stay. This limits the types of outcomes that can be analyzed to only those that take place very early in the postoperative course. The American College of Surgeons has improved on this with databases like NSQIP, which gather patient data after discharge to Postoperative Day 30; however, even this is still quite limiting, particularly in orthopaedics, as many outcomes we are interested in take place years down the road.

In addition, the results of studies based on national databases are dependent on the quality of the data on which they are built. As many of the databases are heavily reliant on ICD-9 coding, which has limitations, there can be inherent biases to study results (as we have shown in the current study). The implementation of ICD-10 may reverse some of these limitations and improve the power of databases reliant on administratively coded data.

Dr. Leopold:Finally, your study points to the ways that a casual reader can be mislead by carelessly or incompletely presented research drawn from these sources. What should readers watch out for when reading studies that seek to answer the “big questions” (like hip fracture research) using these databases?

Mr. Bohl: The reader should be skeptical of any study that claims to identify the “incidence” or “rate” of adverse events from a national database. What these studies are really doing is reporting the rate at which those adverse events are documented in the database. This difference may seem subtle, but it is important. For example, in an administrative database like NIS, data is based on ICD-9 coding, which is not designed for outcome analysis, but rather billing purposes. If it is not in the hospital’s financial interest to report an adverse event, it may be less inclined to do so; similarly, if there is a financial incentive to report an adverse event, there may be a bias towards over-reporting. As we show, this type of bias can lead to significantly different estimates.

The study we conducted compared two populations of patients undergoing the same procedure for the same indication, each selected from one of two commonly used national databases (NIS and NSQIP). We focused on inpatient adverse events, because while NSQIP reports both inpatient and outpatient adverse events up to the 30^th postoperative day, NIS reports inpatient adverse events only. Interestingly, the majority of inpatient adverse events we observed were reported at similar rates between the databases. Unfortunately, other important inpatient adverse events we observed were reported at rates that differed widely between the two databases (by more than two-fold). Much of this may be related to definitional differences. As an example, NSQIP has very strict criteria for documentation of adverse events like acute kidney injury and urinary tract infection based on criteria involving combinations of signs, symptoms, and laboratory results. In contrast, there are no such discrete guidelines for the ICD-coding used in NIS. Our data cannot be used to determine whether each database is over-reporting or under-reporting, as we do not have access to the individual patient’s medical records, but the differences between the databases highlights the need for caution in interpreting study results.

Finally, the reader should take into consideration the postoperative time frame during which adverse events are being captured. An inpatient-only database, like NIS, is much less likely to capture adverse events that occur later in the postoperative time course (for example, surgical site infection) compared with a database like NSQIP that captures both pre- and post-discharge adverse events. Using inpatient-only data for these types of adverse events can generate misleading results.