CORR Insights®: Is the Number of National Database Research Studies in Musculoskeletal Sarcoma Increasing, and Are These Studies Reliable?

Where Are We Now?

All physicians want to make the right decisions for our patients, and evidence-based medicine is the standard by which we make these decisions. This calls upon us to consider the best-available scientific evidence, our clinical experience, and the patient’s values. Although randomized controlled trials (or meta-analyses of randomized trials) are at the top of the clinical evidence pyramid, these have inherent limitations, and sometimes—when the conditions are rare and heterogeneous in clinical presentation—these trials are impossible or nearly impossible to perform. This is often the case for sarcoma care. In those settings, there is necessarily greater emphasis on clinical experience. To support this kind of decision-making, it can be advantageous to use large databases to guide care, because these databases provide information drawn from the experiences of many clinicians.

Because every part of healthcare—patient registration, the health record, billing, and clinical registries—is digital, there is great potential for observational research and quality-improvement initiatives. Much as in social media and business, big-data analytics offer the possibility for precision medicine using these digital data, and this is alluring [7]. But there is minimal empirical evidence to support that it can be done in helpful ways.

With this as background, I commend Lawrenz et al. [8] for their study, “Is the Number of National Database Research Studies in Musculoskeletal Sarcoma Increasing, and Are These Studies Reliable?” in this issue of Clinical Orthopaedics and Related Research®. In it, they systematically reviewed where we are regarding sarcoma registry research. Unsurprisingly, they demonstrated that the number of published studies using large cancer databases are increasing dramatically. More surprising (perhaps except for those already involved in database research) is their finding that there can be substantial inaccuracies in data, especially in fields that require subjective clinical interpretation. Based on these findings, surgeons should be careful when interpreting any results of these studies, especially if they are negative, because inaccuracies in data would tend to add noise to any statistical analysis, leading to underpowering and the risk of a Type II error. To this end, in the hierarchy of medical evidence, I believe research from large databases should be considered as comparable, in terms of level of evidence, as expert opinion is. Arguably each surgeon can be considered an expert in his or her field, so consistent evidence from multiple database studies with adequate accounting of inherent biases would be required to change the practice of an expert surgeon.

Where Do We Need To Go?

Large databases are inherently attractive to researchers because of their relative ease of access, large sample size allowing for advanced statistical analyses, and potential generalizability, given their real-world representation of a large group of patients across multiple centers [6]. However, given the inconsistencies of generated evidence, the current state of database research leaves much to be desired. Lawrenz et al. [8] suggested that misunderstanding the purpose and limitations of the databases, variability in statistical approach and handling of missing data, and differences in periods evaluated account for some of these drawbacks. Because most large databases were developed for a purpose other than the research questions being asked in any given study, there will usually be concerns about the validity of the generated results. For example, most administrative datasets were designed for billing, which makes assessing clinical outcomes more difficult. Based on this, future studies should start with a verification audit of important clinical data, and any future database designs should include validated outcome assessments in their data elements.

van Walraven and Austin [9] listed five unique issues that can bias the results of secondary analyses of databases. These include understanding why the databases were created, determining the accuracy of diagnostic and procedural codes, distinguishing between clinical importance and statistical significance, accounting for any time-dependent variables, and clustering data. It is incumbent on authors to address these issues in research design and reporting. Poor data quality and inaccuracy are major factors that affect the evidence generated by database studies. Thus, for this type of research to have a meaningful impact, investigators must clarify which data elements (such as intervention, case mix, or outcome) are critical to their analyses, and they must determine the database’s accuracy for representing these variables.

Lawrenz et al. [8] noted a 19% to 77% discrepancy between cancer registries and the actual clinical gold standard of physician review for several variables including clinical stage, margins, and adjunct treatment received. Other work has noted similar discrepancies in important clinical variables such as adverse events when recorded by clinicians, dedicated research clinicians, and administrative coders [4]. Moreover, many of these issues are not addressed in the Reporting of Studies Conducted Using Observational Routinely Collected Health Data checklist [2], and further guidelines should be developed so researchers can design high-quality database research whose results can be interpreted by readers [6, 8]. But we’ve known all that for some time; future studies need to tell us the accuracy of clinical information and provide a clear description of what the authors considered the minimum clinical important difference, so that readers can interpret the findings in light of effect sizes that would matter to patients.

Some societies have recognized this and have built registries that include valid clinical outcome tools. For example, the American Academy of Orthopaedic Surgeons Musculoskeletal Tumor Registry includes patient-centered outcome tools including the Patient-Reported Outcome Measurement Information System, Musculoskeletal Tumor Society score, and Toronto Extremity Salvage Score, and encourage clinicians to enter procedural, comorbidity, and tumor details [1]. However, clinicians still need to incorporate these forms into their clinical workflows, and it remains unclear whether this improves the completeness or accuracy of the collected data.

The emergence of electronic health records comes with the promise of digitizing data for quality initiatives and research. The Musculoskeletal Tumor Registry has even developed specific forms for tumor details that can be incorporated into the Epic electronic health record platform. However, recent reviews demonstrated inconsistent evidence that electronic records improve the accuracy and completeness of clinical information [3, 5]. Gianfrancesco and Goldstein [5] listed several factors that may affect the accuracy of electronic health record data. They include representativeness; data availability, especially if collected for billing, clinical, or epidemiologic needs; consistency of data; unstructured data; and missing measurements and visits. Without proper clinician engagement, entering data into the electronic health record will likely be fraught with errors.

Natural language processing is a promising field that can address many of these challenges by using artificial intelligence algorithms to accurately identify data events in the clinical record. But much work and research need to be done. This process relies on expert clinician reviews of thousands of procedures across multiple different settings to create training and validating datasets for each specific variable of concern. To date, only a handful of such algorithms have been developed. Regardless, this appears to be a promising field to leverage the digitalized data available in electronic health records to accurately and efficiently abstract data.

How Do We Get There?

Although reporting data accuracy may become an important metric, it is not helpful for the reader if the data are inaccurate, which seems to be the case in some large-database research. Thus, it is becoming increasingly important that databases should be designed to obtain accurate clinical data. Studies suggest that this can be achieved through trained clinicians entering data into databases, but this will require engagement and buy-in by front-line clinicians because it can be time-consuming. Further research to audit and report the accuracy of data is required to determine where investment of clinician time for this effort should be placed. Accurate natural-language processing algorithms for key clinical data elements are being developed. Further research should be directed to determine whether this results in more efficient and accurate abstraction of data from electronic health records.

The ultimate vision of precision medicine, in which big-data analytics can use the vast resources of global digitized information to present the estimated effect of a proposed treatment path based on previous patients’ similar experiences, is still in its infancy. Until then, researchers and clinicians should continue to conduct database research to collect evidence to not only better inform patient care, but also learn from and improve on the limitations of this type of evidence.