National Databases for Assessment of Quality

Abstract

With the rise in the availability of large health care datasets, database research has become an important tool for colorectal surgeon to assess health care quality and implement practice changes. In this chapter, we will discuss the benefits and drawbacks of database research for quality improvement, review common markers of quality for colorectal surgery, provide an overview of frequently used datasets (including Veterans Affairs Surgical Quality Improvement Program, National Surgical Quality Improvement Project, National Cancer Database, National Inpatient Sample, Medicare Data, and Surveillance, Epidemiology, and End Results), and look ahead to the future of database research for the improvement of quality.

The Rise of Database Research in Colorectal Surgery

With the growing availability of large-scale databases and increasing scrutiny on the quality of health care in the United States, database research has become a valuable resource for colorectal surgeons to assess surgical outcomes and implement practice changes. The large sample sizes and breadth of data in these databases facilitate the analysis of current practices and quality measures with the goal of improving patient care, outcomes, and cost. ¹ The development and formalization of surgical databases have contributed to a growing body of research over the last several decades. Large dataset analyses have provided evidence to support shifts in colorectal surgical practices, including changes in colorectal cancer screening to address disparities in colorectal cancer outcomes for underserved minority populations, ² the success of minimally invasive techniques in improving postoperative outcomes such as surgical site infection (SSI), ³ and the evolution of enhanced recovery protocols to reduce the length of stay and complication rates after colectomy. ⁴ In this chapter, we will discuss the benefits and drawbacks of database research for quality improvement, review common markers of quality for colorectal surgery, provide an overview of frequently used datasets (including Veterans Affairs Surgical Quality Improvement Program [VASQIP], National Surgical Quality Improvement Project [NSQIP], National Cancer Database [NCDB], National Inpatient Sample [NIS], Medicare Data, and Surveillance, Epidemiology, and End Results [SEER]), and look ahead to the future of database research for the improvement of quality.

Benefits of Database Research for Surgical Quality Improvement

When compared with single-institution case series historically used to evaluate surgical outcomes, large datasets offer many advantages in the quality and content of data, as well as the execution of statistical analyses. Each dataset utilizes a specified data abstraction method across institutions, improving transparency, consistency, and generalizability of the data. While the rigor and specificity of the definitions vary across datasets, many clinically abstracted data utilize standardized definitions and data are collected by formally trained abstractors, helping to ensure high reliability across institutions. Often datasets provide quality assurance through data audits, though these methods differ from database to database. Once a dataset is secured for analysis, the large numbers provide increased statistical power leading to a broader representation of patient factors (sex, age, race, and ethnicity), hospital features (size and accreditations), or regional characteristics (geographic regions). Databases are an efficient way to sample larger and more generalizable surgical patient populations with the opportunity to compare across institutions. Multi-institutional datasets allow researchers to study differences in outcome by region, socioeconomic status, temporality, and following practice changes. Additionally, databases can facilitate studies on rare diseases by pooling cases across institutions for a larger sample size and power. National databases have allowed surgeons to move beyond using a small provider or institution-based series to well-powered, standardized, and generalizable data to enact quality improvement measures.

Drawbacks of Databases

Databases are only as valuable as the data therein. The quality of data in any given database may be undermined by multiple factors. Users must consider the potential for selection bias, as each dataset is subject to a different sampling strategy. Certain datasets sample by hospital population (for example, the Veterans Affairs [VA] system) or insurance provider (those with Medicare insurance coverage). Datasets are limited by the presence and absence of certain data points, and varying methods for collecting these data (i.e., reimbursement codes vs. trained clinical abstractors) can introduce unrecognized confounding. ⁵ Researchers must understand the data collection methods and definitions, carefully set inclusion and exclusion criteria, and consider potential confounders to produce valid and generalizable results. Missing or incomplete data can lead to mistaken conclusions. Furthermore, many of these databases require a fee for access or membership to an organization for use, which can be cost prohibitive for those attempting to access data, limiting research and even practice changes to those with the means to afford it.

Quality Measures Relevant in Colorectal Surgery

National database research is integral to the quality improvement movement, allowing for data-driven practice changes across all phases and levels of patient care. Quality improvement efforts can be implemented at the level of the individual health care provider, group, hospital, system, region, and even nation. Measuring outcomes is necessary for patients and health care systems alike but is particularly critical to surgeons. Major abdominal surgery affects patients' health and quality of life, often with high costs to the individual and the health care system. It is important to track surgical outcomes to improve efficiency, decrease cost, and eliminate error. Multiple organizations work to define, track, and improve surgical quality in the United States including the Centers for Medicare and Medicaid Services (CMS), the Centers for Disease Control and Prevention (CDC), the Agency for Healthcare Research and Quality (AHRQ), and the National Quality Forum (NQF). National quality metrics provide the surgical community with common endpoints and allow for comparison across providers, hospitals, communities, and the country. Table 1 summarizes some of these important markers of quality for the colorectal surgeon.

Table 1. Common quality measures relevant to the colorectal surgeon.

Hospital encounter	Perioperative	Colorectal specific
30-d readmission (CMS)	Pre-operative prophylactic antibiotics (NQF)	12 regional lymph nodes in colon cancer (NQF, AJCC, NCI)
Mortality (CMS)	Prolonged intubation (AHRQ patient safety indicator, NQF)	Colon surgery outcome measure (risk-adjusted morbidity/mortality composite, NQF, ACS)
Length of stay (CMS)	Postoperative renal failure (AHRQ patient safety indicator, NQF)	Colorectal cancer resection pathology reporting includes T and N staging (NQF, ACS)
Opioid utilization (CMS)	Postoperative wound dehiscence (AHRQ patient safety indicator)	Colonoscopy screening for colorectal cancer (CMS)
Pressure ulcer rate (AHRQ patient safety indicator)	Postoperative hemorrhage or hematoma rate (AHRQ patient safety indicator)	Anastomotic leak rate (CMS, ACS)
In hospital fall with hip fracture (AHRQ patient safety indicator)	30-d reoperation rate (CMS, ACS)	Surgical site infection after colectomy (CMS)

Abbreviations: ACS, American College of Surgeons; AHRQ, Agency for Healthcare Research and Quality; AJCC, American Joint Committee on Cancer; CMS, Centers for Medicare and Medicaid Services; NCI, National Cancer Institute; NQF, National Quality Forum.

Note: provides examples of quality measures reported across the hospital encounter, perioperative period and those more specifically applicable to colorectal surgeons, followed in parentheses by a relevant agency using the measure. See abbreviation list on page 2.

The CMS, part of the U.S. Department of Health and Human Services, developed the Meaningful Measures Initiative to streamline quality measures, empower healthcare consumers, increase equity in health care, and improve outcomes for patients and hospitals alike. ⁶ These measures are subsequently utilized by CMS to drive Medicare and Medicaid reimbursement with merit-based payment. In addition to their Meaningful Measures initiatives, CMS has also worked with the CDC to collect SSI data following the CMS mandate to reduce SSI related to hysterectomy and colectomy. ^{7 8 9} The CDC manages the National Healthcare Safety Network which is the most widely utilized system in the United States for tracking infection, including SSI related to colectomy, a critical metric for the colorectal surgeon. ¹⁰

The AHRQ was developed to improve the quality and safety of health care in the United States. The AHRQ invests in research and educational endeavors, sets quality measures, and generates data to be used by providers and policymakers to track performance at the hospital, local, regional, state, and/or national level across time. Most relevant to surgeons, the AHRQ has developed multiple quality indicators to measure health care quality and safety during inpatient admissions and related to interventions or procedures. The AHRQ Patient Safety Indicators measure potentially avoidable patient safety events and are publicly reported. Many of these measure postoperative or postprocedural complications (e.g., postoperative sepsis). Similarly, several AHRQ Inpatient Quality Indicators are directly related to surgery (e.g., mortality rate after esophageal or pancreatic resection).

The NQF, created in 1999, is an organization that promotes and improves health care quality. It is a not-for-profit, evidence-driven, and consensus-based organization utilized by federal and state governments as well as private groups to evaluate health care delivery and foster quality improvement. ¹¹ The NQF has approximately 300 endorsed measures, with many surgical measures overseen by the American College of Surgeons (ACS) as the measure steward. Those relevant to the colorectal surgeon range from general measures such as length of stay and readmission rates to measures more specific to colorectal practice such as follow-up intervals for colonoscopy screening, colectomy morbidity and mortality outcomes, and removal of at least 12 regional lymph nodes for colon cancer resections. ¹²

Common National Databases for the Colorectal Surgeon

There are many national datasets available to study surgical practices, measure quality, and drive quality improvement in colorectal surgery. Table 2 summarizes some of the most commonly utilized large-volume national datasets applicable for colorectal surgery research.

Table 2. Common national databases for the colorectal surgeon.

Database	Type of data	Strengths	Weaknesses	Relevance to colorectal surgery
VASQIP	Clinical, trained abstractors	-Audited data -Highly reliable -Clinical detail on comorbidities, laboratory values, operative complexity	-VA population limits generalizability -No longitudinal data	Clinically relevant surgery-specific data
NSQIP	Clinical, trained abstractors	-Audited data -Highly reliable -Clinical detail on comorbidities, laboratory values, operative complexity	-Hospital participation required -No longitudinal data	Detailed data for colectomy and proctectomy (procedure targeted)
NCDB	Clinical, trained abstractors (registrars)	-Audited data -Generalizable to cancer care across United States (70% of all cancer cases) -Longitudinal data	-CoC participation required -Substantial missing data in some categories (<50% available for analysis)	Large sample of colorectal cancer cases
AHRQ/NIS	Administrative/claims	-All-payer data -Large size (7 million hospitalizations)	-Claims-based -Subject to under and over-coding -No physiologic data/laboratory values -No longitudinal data -Represents 20% of inpatient encounters limiting generalizability	No colorectal-specific data points
Medicare	Administrative/claims	-Generalizable to older adults age 65+ (70% of this population) -Longitudinal data	-Claims-based -Subject to under and over-coding -No physiologic data/laboratory values -Does not include services uncovered by Medicare	No colorectal-specific data points
SEER	Clinical, trained abstractors (registrars)	-Population based -Diverse population -Tumor and treatment-specific characteristics	-Missing key data (e.g., comorbidities, surgical technique – laparoscopy or open, radiation dosing) -Limited geographic areas -No longitudinal data	Differentiates cancer location (proximal, distal colon, and rectum)

Abbreviations: AHRQ, Agency for Healthcare Research and Quality; CoC, Commission on Cancer; NCDB, National Cancer Database; NIS, National Inpatient Sample; NSQIP, National Surgical Quality Improvement Project; SEER, Surveillance, Epidemiology, and End Results; VA, Veterans Affairs; VASQIP, Veterans Affairs Surgical Quality Improvement Program, Medicare Data.

Note: Some of the most commonly utilized large-volume national datasets applicable for colorectal surgery research including their strengths, weaknesses, and relevance to the colorectal surgeon.

The VASQIP database was the first large-volume clinical database of its kind. ¹³ It was created to study surgical outcomes at the hospital level throughout the VA system. The VASQIP utilizes trained abstractors to gather clinical data leading to the high reliability of the dataset. Abstractors review a sampling of surgical cases from VA hospitals to follow 30-day outcomes (the exception to this is patient mortality which can be abstracted when linked to the VA system). Specific operative variables are collected including case time, wound class, and whether the case was emergent or elective. ^{14 15} One drawback of this dataset is the limited generalizability of the VA population to the overall population: the data skews largely male and majority Caucasian. ¹⁵ Additionally, the VASQIP database only tracks 30-day outcomes with the exception of mortality, limiting its ability to capture long-term postoperative outcomes.

The NSQIP is a database supported by the ACS with the intent of providing opportunities for hospital-level quality improvement. The database, based on the successful VASQIP program, began collecting data in the early 2000s. ¹³ Clinical data are collected from hospitals by trained reviewers including 30-day surgical outcome measures, length of stay, readmissions, morbidity, and mortality. NSQIP collects preoperative patient characteristics (e.g., comorbidities and smoking status) and intraoperative data (e.g., length of procedure and presence of residents/trainees) that are helpful in more thoroughly evaluating risk for postoperative morbidity and mortality. Using hierarchical modeling, the data are risk adjusted for both patient and procedural factors and provided back to participating hospitals through performance reports. ¹⁶ The NSQIP is highly regarded and widely utilized in the world of surgical database research. Its weaknesses, similar to VASQIP, are the lack of long-term outcomes (NSQIP tracks 30-day outcomes), that data access is limited by hospital participation and a lack of hospital identifiers for public use. Most relevant for the colorectal surgeon, NSQIP offers a “procedure-targeted” option in which hospitals collect additional data for specific procedures (colectomy and proctectomy) to provide more detail relevant to surgical subspecialties. The specialty-specific data for colorectal surgery include the use of preoperative steroids or immunosuppressants (such as those for inflammatory bowel disease), details of the preoperative bowel preparation (mechanical prep, oral antibiotic, or both), preoperative chemotherapy, intraoperative approach (laparoscopic, robotic, open, or hybrid), and postoperative outcomes such as ileus, anastomotic leak, and pathology data for cancer cases. ¹⁷ Such data have contributed to culture shifts in colorectal surgery. For example, NSQIP analyses consistently demonstrate equivalent if not superior surgical outcomes for minimally invasive techniques compared with open surgery for colectomy, supporting increased uptake of these surgical approaches. ¹⁸ Additionally, the NSQIP platform has been used to facilitate collaborative quality improvement efforts, such as a multi-institution NSQIP pilot to assist hospitals in the implementation of enhanced recovery protocols; analysis after the enhanced recovery pilot demonstrated improved length of stay and reduced complication rates following implementation of enhanced recovery. ⁴ A similar NSQIP initiative is underway for operations performed for inflammatory bowel disease. ^{19 20}

NCDB, a joint venture between the American Cancer Society and the American College of Surgeons Commission on Cancer (CoC), was established in 1989 and is a hospital-based system that captures greater than 70% of all new cancer diagnoses in the United States. ²¹ The NCDB collects information on hospital facility type and volume as well as specifics about each cancer diagnosis including staging information. Two key outcome measures tracked by the NCDB include readmission and survival. Benefits of this dataset include that it is highly representative of cancer care across the United States and allows for review and analysis of treatment patterns over time. The NCDB tracks specific variables about diagnosis, staging, treatments, and survival. Furthermore, it is the largest sample of colorectal cancer cases in any cancer registry. ²² Unfortunately, the NCDB is restricted in access to CoC hospitals and therefore is missing data from hospitals and/or treatment facilities without CoC designation.

NIS Database is a publicly available subset within AHRQ's Healthcare Cost and Utilization Project that contains data for more than 7 million inpatient hospital stay. NIS is a claims-based dataset, that is, it tracks billing data submitted by hospitals that reflect the International Classification of Diseases (ICD)-9/10 codes, Current Procedural Terminology codes, and costs. There are several limitations to using claims-based data. Most importantly, it is the lack of clinical data (vital signs, laboratory data, and imaging data). Furthermore, these data can be subject to “overcoding” (simple items are more commonly coded) and “undercoding” (procedures/diagnoses that do not reimburse well or are difficult to capture will be missed). Specific to NIS, the major limitation is that the sample is only representative of 20% of hospitalizations in the United States.

The CMS is the primary health insurance program in the United States for individuals 65 years of age and older and those who qualify for disability benefits through the Social Security Administration. ²³ Several forms of Medicare data files are available through the CMS website. Medicare data are gathered by coders reviewing reimbursement claims for health care services. Because this dataset captures the majority of Americans greater than 65 years of age, the sample is large and closely representative of the older population. Many researchers gain further utility by linking Medicare data to other databases like cancer registries. ⁵ Because Medicare data are not limited to a single health care episode, these are better suited to longitudinal study of patients through different phases of health care (inpatient, outpatient, surgical, etc.). A key weakness of this dataset is that it is derived from billing data and is based on ICD codes, similar to the NIS. Because Medicare captures patients 65 years and older, the generalizability of the data to younger populations is limited.

The SEER database was created as a collaboration between National Cancer Institute (NCI), the CDC, and regional/state cancer registries. ²⁴ SEER utilizes state-reported, population-based data from a diverse population across the United States. A key strength of this dataset is that it captures specific characteristics of cancer diagnosis and treatment. An important resource for the colorectal surgeon, SEER differentiates cancer location in more detail, allowing for analysis of outcome variation between proximal colon, distal colon, and rectal cancers. SEER can be utilized to evaluate colorectal cancer screening protocols and identify areas for improvement in screening. In fact, research utilizing SEER data has increased our understanding of racial disparities, sex differences, and the impact of age on colorectal cancer outcomes. ²

Future Directions and Additional Considerations

Over the past decade, new databases have emerged that are directly linked to an electronic health record (EHR) and encompass multi-institutional research networks. These datasets are unique in several ways: (1) incorporation of standardized data from the electronic medical record eliminating the need for abstraction (Cosmos and The National Patient-Centered Clinical Research Network [PCORnet]), (2) widespread availability of biologic samples or genetic dataset (The Veteran's Affairs Million Veteran Program [MVP] and All of Us), and (3) access to patient-reported data ranging from social determinants of health to mental health to FitBit data (All of Us).

The EHR company, Epic, established Cosmos in 2019 with the goal of creating “a universe of data that drives evidence-based research and individualized patient care.” ²⁵ Cosmos has over 140 million patients, 6.1 million of whom have cancer, representing 1,000 hospitals across the country with data from over 4.7 billion associated encounters. Aside from having a large sample size, the data are integrated across inpatient and outpatient records and incorporate nontraditional items such as survey data, birth records, and social determinants of health. The key advantage of Cosmos for quality research is the ability to perform quick analyses to address questions in real time. For instance, Cosmos data found that cervical, colon, and breast cancer screening rates were 86 to 94% lower during 2020 than prior years, highlighting a need for future efforts to catch individuals who delayed care due to the coronavirus disease 2019 pandemic. ²⁶

The PCORnet is a nationwide network divided into eight Clinical Research Networks that contain EHR data for over 66 million individuals. ²⁷ PCORnet has a common data model which makes analysis of data across different institutions and EHRs easier. Data from PCORnet have been applied to surgical outcomes research. For example, Arterburn et al utilized the PCORnet database to compare outcomes of over 65,000 bariatric surgery patients by the bariatric procedure type. ²⁸

The Veteran's Affairs MVP has enrolled nearly 1 million Veterans, many of whom have associated genetic, lifestyle, and EHR data. The MVP cohort is mostly male with individuals who have been followed for a longitudinal period (having received most of their care through the VA system). The MVP has become established as a research database with over 100 peer-reviewed publications, though none to date are colorectal specific. A key strength of MVP is the availability of genomics data that can be linked to diagnosis codes which, in the age of personalized medicine, could prove meaningful in the diagnosis and treatment of colon cancer and outcomes research. ²⁹

The National Institutes of Health All of Us Research Program's (All of Us) goal is to recruit individuals that have traditionally been underrepresented in medical research due to race and ethnicity, socioeconomic status, gender identity, etc. It has enrolled nearly 500,000 individuals and continues to recruit new participants. All of Us provides access to alternative forms of data that are not available in the EHR, while also allowing participants to link their EHR-record to the All of Us research platform. Participants complete surveys on topics including overall health, past medical and surgical history, and health care access as part of the intake process. Participants also attend an in-person appointment for blood collection (of a wide range of laboratory values and genetic testing), vital signs, and weight measurements. Over 357,000 samples have been collected thus far. The All of Us data are designed to facilitate the linkage of EHR data to biological samples and patient-reported data within one research portal. ³⁰

Although they provide access to a wider range of multi-institutional and standardized data along with individuals who are underrepresented in medical research, these databases have limitations for utilization in quality research. For instance, the PCORnet data are limited by the granularity of data available within the EHR and may require natural language processing to extract information such as tumor grade and resection margin status. Furthermore, institutions that are not members of PCORnet or Cosmos must pay for data access and service use. ^{25 27} Finally, All of Us and MVP do not require EHR linkage so many patients will not have the granularity of information needed to investigate postoperative outcomes.

Despite the utility of large datasets and the promise of new directions in database research, it is important to note that the proliferation of data research has led to a wealth of studies that must be interpreted with a critical lens. In their 2021 article, “Same Data, Opposite Results?” Childers and Maggard-Gibbons highlight two manuscripts with conflicting conclusions despite using the same data. ³¹ This article underscores the importance of thoughtful study design to elevate the quality of research. To provide guidance for future database studies, JAMA Surgery published a 12-part series, “The Guide to Statistics and Methods Reporting Guidelines,” a useful resource to the colorectal surgeon when performing database research. Finally, quality improvement efforts are best served by examining one's local data with a similarly critical lens.

Conclusion

Large datasets remain a valuable resource for colorectal surgeons. When used thoughtfully, database research can lead to practice changes that improve patient outcomes and health care delivery. Such work is critical to evaluate evidence-based practices and promote continuous improvement in the field of colorectal surgery for the wellbeing of patients, providers, and health care systems. Colorectal surgeons should continue to use these databases thoughtfully in the quest to improve quality for all patients.

Abbreviations

ACS

American College of Surgeons

AHRQ

The Agency for Healthcare Research and Quality

CDC

The Centers for Disease Control and Prevention

CMS

The Centers for Medicare and Medicaid Services

CoC

American College of Surgeons Commission on Cancer

CPT

Current Procedural Terminology

CRNs

Clinical Research Networks

EHR

electronic health record

HCUP

AHRQ's Healthcare Cost and Utilization Project

HHS

United States Department of Health and Human Services

ICD

International Classification of Diseases

MVP

The Veteran's Affairs Million Veteran Program

NCDB

The National Cancer Database

NCI

National Cancer Institute

NIS

The National Inpatient Sample

NQF

The National Quality Forum

NSQIP

The National Surgical Quality Improvement Project

PCORnet

The National Patient-Centered Clinical Research Network

SEER

Surveillance, Epidemiology, and End Results

SSI

surgical site infection

VA

Veterans Affairs

VASQIP

The Veterans Affairs Surgical Quality Improvement Program

Funding Statement

Funding JM: NIH Metabolism and Nutrition Training Program T32 (grant DK 007665). HJ: Surgical Oncology T32 Training Grant (T32CA090217 – NIH).