The National Anesthesia Clinical Outcomes Registry: A Sustainable Model for the Information Age?

Abstract

Anesthesiologists care for patients of all ages, with all conceivable comorbidities, in every kind of health care facility. This leads to a significant challenge in the collection of data to describe the specialty, and in the development of evidence-based performance measures for anesthesiologists. Whereas narrowly defined medical specialties have developed registries based on manual abstraction of clinical data from the medical record (e.g., cardiac surgery), this approach would be prohibitively expensive for anesthesiology, and is unlikely to generate statistically useful data when major adverse outcomes occur a handful of times in tens of thousands of cases. The American Society of Anesthesiologists (ASA) addressed this challenge in 2008 by funding a related organization, the Anesthesia Quality Institute (AQI), to develop the National Anesthesia Clinical Outcomes Registry (NACOR). The technical development of this registry and the approach taken to define the specialty of anesthesiology and the performance of anesthesiologists may serve as a model for other specialty society efforts.

Background

Anesthesiology has a long history of promoting patient safety. Anesthesiologists are the great facilitators of modern health care, ensuring the safe and efficient conduct of uncomfortable and invasive procedures for dozens of medical and surgical specialties. Anesthesiologists care for patients of all ages, with all conceivable comorbidities, in every kind of health care facility, but have never had a national perspective on the scope of this work.

While other medical specialties have been able to develop clinical data registries concentrated on a single patient population, relying on abstraction from a relatively small number of medical records, this option has not been economically feasible in anesthesia. This leads to a significant challenge in collection of data to describe the specialty, and in development of evidence-based performance measures for anesthesiologists.1 With serious intraoperative adverse events occurring in fewer than 1 case in 200, the number of records needed to demonstrate meaningful trends is enormous; success in delivering quality care makes it paradoxically harder to build an effective registry. The cost of manual data abstraction for thousands of cases in each practice challenges the sustainability of any registry attempting to develop a picture of national anesthesia practice.

The American Society of Anesthesiologists (ASA) addressed this challenge in 2008 by funding a related organization, the Anesthesia Quality Institute (AQI), to develop the National Anesthesia Clinical Outcomes Registry (NACOR). In 2009, NACOR began operations and has grown rapidly. The technical development of this registry, its sustainability in the future, and the approach taken to define the specialty of anesthesiology and the performance of anesthesiologists, may serve as a model for other specialty society efforts.

The Anesthesia Quality Institute (AQI)

The AQI is a 501(c)(3) nonprofit corporation. AQI is housed at ASA headquarters in suburban Chicago, and currently includes a dozen employees.

At present 75 percent of AQI funding comes from ASA as an annual grant. The remaining 25 percent of funding comes from endowment income, research grants and contracts, and sales of reports and services. ASA’s investment to date in AQI is approximately $4.5 million over five years.

In exchange for access to ASA resources, AQI provides members with discounted participation in NACOR; groups with 100 percent ASA membership pay nothing to participate in the registry. Non-ASA anesthesia providers pay $500 per year. This model has allowed ASA to invest in a registry on a large scale, as a common benefit for all members, and as a sustainable resource. In discussions with other successful specialty society registries, this kind of commitment to scope and scale appears to be a predictor of long-term success.

AQI manages multiple registry projects to benefit the profession of anesthesiology. In addition to NACOR these include the Anesthesia Incident Reporting System (AIRS), a web-based program to capture adverse events and near misses arising during anesthetic care; and the Closed Claims Project registry, which houses structured and narrative data abstracted from malpractice files made available by the companies that insure anesthesiologists. The AQI is designated by the Agency for Healthcare Research and Quality as a Patient Safety Organization (PSO), and accredited by the Centers for Medicare and Medicaid Services (CMS) as a reporting registry for the Physician Quality Reporting System (PQRS). PSO designation increases the confidence of would-be contributors that their data will be safe and secure, thus enhancing sustainability.

Designation as a PQRS reporting registry at first offered no value, because most practitioners were already achieving incentives on a claims-made basis through their billing software. As the PQRS program evolves into the Qualified Clinical Data Registry (QCDR) mechanism, however, NACOR participation will become an indispensable resource for anesthesiologists wishing to qualify for performance incentives or avoid penalties. NACOR was designated as a QCDR in spring, 2014, and we anticipate a sharp increase in registry participation in 2015. Providing this “one-stop shop” regulatory compliance service to ASA members is part of our strategy to ensure sustainability; clinical registries must adopt to the regulatory climate that will drive physician participation.

The National Anesthesia Clinical Outcomes Registry (NACOR)

NACOR was created in late 2009, and the first cases were entered in January, 2010.2 Unlike other specialty society registries, NACOR was designed from the outset to conform to the evolution of health care into the Information Age. All data are collected by periodic electronic reporting directly from anesthesia practice and health care facility information technology (IT) to the registry. NACOR does not rely on professional abstraction of patient medical records with cross-loading into the registry, as is the case for most traditional or “eyeball” registries. Figure 1 illustrates the flow of data into NACOR.

Figure 1.

Flow of Data into the National Anesthesia Clinical Outcomes Registry (NACOR)

Notes: SSRS: SQL Server Reporting Service; OLAP: Online Analytical Processing; SSIS: SQL Server Integration Services; SFTP: Secure File Transfer Protocol; HIT: Health Care Information Technology.

Participation

Participation in NACOR does not require any additional documentation by anesthesia professionals; the NACOR Minimum Data Set is contained in the digital information required for payment of any anesthesia claim and is, therefore, attainable by any anesthesia practice in the United States. Lowering this barrier to participation has helped AQI start quickly and grow rapidly. Participation has now reached the point where the data “snowball” is visibly useful to the membership, thus contributing to sustainability.

Required Minimum Data Set

While the Minimum Data Set defines the floor for contribution of case information, we impose no limits on the ceiling. The simplest case records in NACOR include about 20 structured fields, but the most comprehensive files can include thousands of data points per case, including minute-by-minute vital signs, medication doses, granular time stamps, patient history, procedure notes, and fluids in and out. In the Information Age the storage of data and bandwidth for data transfer are both effectively infinite (80 percent of the AQI budget is devoted to personnel costs; less than 5 percent is required for maintenance of cutting edge hardware and software).

NACOR was consciously constructed to absorb any and all digital data available from a given practice or facility. This includes billing system information—the starting point—as well as data from Anesthesia Information Management Systems (AIMS), broader facility electronic health care records, internal quality capture software, and even postdischarge patient satisfaction measurement. AQI works with vendors of health care IT to make data transfer possible and with participating practices to incrementally harvest more comprehensive electronic data as they become available.

NACOR’s Design Is Driven by the Need to Analyze All Cases from Diverse Sources

This design of the NACOR registry makes case specific data somewhat unique. First, NACOR is a census registry rather than a sample. NACOR includes every case, every day, from participating practices. Because the data transmission is passive and automated, it would require conscious activity and significant energy for a provider or group to remove a record from the registry. At the level of national aggregation this enables a holistic and representative view of the practice of anesthesia in the United States. A second corollary of the mechanics of NACOR is the heterogeneity of data in the registry. Many case records include only minimal administrative data; a portion (25 percent) include outcome information from quality capture programs, and an even smaller fraction (10 percent) include complete information from AIMS. Similar information may be coded differently in different software. NACOR currently accepts data files from about three dozen billing, quality, AIMS and electronic health record (EHR) vendors. While all of their efforts are mapped into a common schema in NACOR (http://www.aqihq.org/files/AQI_XML_Schema2_0_Documentation(1).pdf), subtle differences may exist in how similar information is reported.

While structured common definitions exist for many key variables (e.g., surgical CPT codes, ASA Physical Status, RxNorm medications), for others there is a wide range of definitions based on when the measure is captured, who is reporting it in the records and what the local criteria for reporting may be.3 The occurrence of postoperative nausea and vomiting, for example, exhibits substantial definitional heterogeneity. This finding motivates us to retain the context of each piece of data as well as the data themselves. For example, we would capture both that patient XYZ had nausea and vomiting and that the definition used was the one for practice ABC. We also use great care when reporting benchmarks based on outcomes with divergent definitions, choosing to report only those measures with common definitions, and only to those practices that are using the definition in question.

Regular Convening for the User Community

In addition to taking care with analysis and reporting of elements with divergent definitions, AQI sponsors an annual Definitions Conference to bring together end-user anesthesiologists and vendor representatives to agree on common formats for recording and reporting anesthesia data. The need for convening is clearly recognized by all stakeholders, including the vendors themselves, and as a nonprofit AQI is well positioned to further ongoing discussion without obvious commercial bias. The gradual coalescence of outcome definitions that AQI is encouraging will be one of our most important activities over the long run, and something that contributes to the sustainability of our efforts.

Progress to Date

As of March, 2014, there were 317 anesthesia groups contracted with NACOR. Of these groups, 268 had completed the NACOR survey, providing descriptive information about their practices, their providers, and the facilities covered; and 243 practices have sent at least one file of case-level data, while 195 are “in production.” These practices send one or more monthly files, with data on every anesthesia case performed by the group each day. Practices contributing to NACOR include 13,905 physician anesthesiologists; 9,324 nurse anesthetists; and more than 3,500 residents and nurse trainees. This represents 20–25 percent of the national workforce participating, and provides a representative sample of national data.

The Minimum Data Set for submission of a case to NACOR includes the facility where the procedure was performed; the surgical and anesthesia codes; the date and duration of the procedure; the patient’s age, sex, ZIP code, and ASA Physical Status; codes for all anesthesia providers; and the type of anesthesia performed. With addition of metadata about the facility and the providers (obtained in the practice survey, which each group completes annually), the Minimum Data Set is about 30 elements per case.

While all groups transmit this much information, many do much more. Fifty percent of NACOR practices include codes for compliance with the PQRS. About 25 percent of practices include more detailed data on short-term outcomes of the case, generated through software that captures self-reported complications noted at the time of patient discharge from the Post Anesthesia Care Unit (PACU). At present about 10 percent of all practices include AIMS data in their case submissions in a usable format. These files, which can include thousands of data points, contain q1 minute vital signs (from up to 20 different monitors), every dose of any medication administered in the operating room (OR), granular time stamps for OR events, and complete fluid input and output information. At present about 30 percent of NACOR participants are using electronic records in the OR, meaning there is substantial opportunity for AQI to build further interfaces to expand NACOR data collection to AIMS records.

Table 1 shows the distribution of facilities at which NACOR groups provide anesthesia. The large number of “undesignated” facilities are those indicated in the newest case files for which the practice has not yet provided demographic information. (By experience, these will eventually be distributed across the known facility types in approximately the existing proportions.) The large number of these facilities indicates the dynamic nature of anesthesia and procedural medicine today, with rapid expansion of services into multiple environments. The median practice in NACOR provides services in nine facilities, with some groups now incorporating hundreds of anesthesiologists working at dozens of hospitals, surgery centers and offices.

Table 1.

Facilities Represented in the National Anesthesia Clinical Outcomes Registry Database

Facility Type	Count
Medium Community Hospital (100–500 beds)	615
Undesignated (survey not complete)	585
Freestanding Surgery Center	494
Attached Surgery Center	180
University Hospital	166
Surgeon Office	145
Large Community Hospital (over 500 beds)	134
Small Community Hospital (fewer than 100 beds)	101
Specialty Hospital	70
Pain Clinic	44
Total	2534

Table 2 shows the number of cases in NACOR by facility type (including the “undesignated” category). The 16 million cases accumulated over four years represent about 4 million per year. One of the features of passive data collection from existing records is that when a new practice is enrolled and its first test file is successfully mapped, it is usually possible to pull data from a time earlier than the enrollment point as well as prospectively. The AQI will therefore seek records back to its “birthday” on January 1, 2010, if they exist in the same software. This allows the first benchmarking reports to new practices to begin with up to four years of trend data already in place, another “rapid start” satisfier that is encouraging participation. Figure 2 shows the overall growth of NACOR through 2013.

Table 2.

Cases in the National Anesthesia Clinical Outcomes Registry, by Facility Type

Cases by Facility Type	Count
University Hospital	1,408,366
Large Community Hospital (over 500 beds)	2,757,572
Medium Community Hospital (100–500 beds)	5,720,880
Small Community Hospital (fewer than 100 beds)	459,230
Specialty Hospital	196,789
Attached Surgery Center	753,163
Freestanding Surgery Center	1,690,231
Pain Clinic	27,612
Surgeon Office	78,414
Undesignated	2,877,201
Total	15,969,458

Figure 2.

Growth of the National Anesthesia Clinical Outcomes Registry (NACOR) by Numbers of Practices, Facilities, Providers, and Cases per Quarter and Year from 2010–2013

Of the 16 million records now available in NACOR, approximately 2 million are anesthesia stand-alone procedures: chronic pain injections, intubations and line placements in the intensive care unit, and epidural analgesia for uncomplicated vaginal deliveries. The remaining 14 million cases are procedural anesthesia, in which the anesthesia providers are caring for a patient undergoing a surgical or medical procedure performed by another physician. Figures 3 and 4 show the most common of these procedures by count and by the amount of anesthesia time consumed.

Figure 3.

Most Frequent Anesthetics in the National Anesthesia Clinical Outcomes Registry by Case Number and Patient Age from 0 to 90 Years

Figure 4.

Top 20 Anesthetics in the National Anesthesia Clinical Outcomes Registry, as Determined by the Percentage of All Anesthesia Minutes Devoted to That Case Type

Participating practices have continual, online access to their own data and to national benchmarks. Each day, 20,000–100,000 cases are added to NACOR, and new “data cubes” are generated each night within the registry. By logging in to the NACOR Reports Server, practices have access to 40 reports from their own data, many of which can be subreported by facility, provider, or common case types. Where appropriate, national benchmarks are also shown (Figure 5 shows the distribution of all cases by age and gender, for example). Anesthesia practices use these reports every day to facilitate local quality improvement by tracking trends in performance, identifying where they are out of step with national norms, and understanding their community of patients and providers. Data at the level of individual providers is available to meet the Ongoing Professional Practice Evaluations (OPPE) required by hospitals surveyed by The Joint Commission, and to contribute to data needed for maintenance of certification.

Figure 5.

Number of Cases in the National Anesthesia Clinical Outcomes Registry, by Patient Age and Sex

In 2014 the AQI will be adding more specific peer-group benchmarks organized by facility type, teaching status, and geographic region. The AQI Champion physician in each group is encouraged to review the reports on a regular basis and look for those in which the practice or individual providers are obvious outliers. While outlier status may indicate artifacts in data collection or transmission, it may also indicate a quality issue deserving of further attention. Some issues will prove to be structural in the practice, based on the kinds of patients and procedures performed (e.g., increased postoperative nausea and vomiting when caring for a younger, more female population). Others will demonstrate real shortcomings in the practice, and these can be actively addressed.

At the national level, the data available from NACOR has been invaluable to the ASA. Understanding the most common cases has, for example, helped to focus development of the perioperative surgical home model for coordinated surgical care of those patients. Observed variation in clinical outcomes has led to targeted creation of new educational products, such as the Advanced Cardiac Life Support for Anesthesiologists course and maintenance of certification modules for obstructive sleep apnea and perioperative glycemic control. Unusual cases and complications from AIRS are used to develop a monthly teaching article in the ASA Newsletter. And the national benchmarking data have been used to develop a new practice consultation service for the ASA, focused on quality outcomes and business efficiency.

Lessons Learned

Development of NACOR has moved very quickly, in part because of our “Ready, Fire, Aim” philosophy. Direct collection of existing electronic data has proven to be a powerful model, enabling rapid growth and some quick wins for the AQI in terms of national-level insight into the specialty. While rapid collection and reporting of data has contributed to sustainability, this model has created some challenges as well.

Engaging Health Information Technology (IT) Vendors Can Facilitate Implementation

Substantial amounts of clinically relevant data are available in any health care environment, but may be held in numerous IT systems. A registry such as NACOR must continually invest in “bridges” to each of these systems, something that can take a substantial amount of technician time. Fortunately, we have found that the vendors themselves are willing to assist.

First, they benefit from the subject-matter experts of the registry sharing with them desired formats and definitions on behalf of the specialty, something that reduces their own need for research; and second, as the registry project gains momentum they can advertise the ability to share data on behalf of practices and facilities as a business advantage of using their software. The AQI has engaged these motivations from the beginning by freely sharing any translation and mapping routines we have developed, and by recognizing willing collaborators as “Preferred Vendors” and licensing them to advertise this fact. This approach to vendor engagement has contributed to the sustainability of the model.

Mapping Is Necessary Due to Local Customization of Anesthesia Information Management Systems (AIMS)

AIMS are highly customized products, which tend to be heavily modified to fit the needs of a particular practice. First-generation AIMS were written as stand-alone software by IT savvy anesthesiologists; these products are intuitive and popular with the users, but have interoperability issues with other components of a facility’s EHR. Second-generation AIMS, which are now taking over the market, are built as components of enterprise software systems and have greatly improved connections to the global electronic record, but suffer from a lack of specificity for providers. They tend to be clunkier, with the need for more customization at the level of individual facilities and practices.

For NACOR, this local customization means the expenditure of substantial effort to map hundreds of data elements into a common schema. We have been able to work closely in this regard with the Multicenter Perioperative Outcomes Group (MPOG), a consortium of university anesthesia practices committed to development of an industry-wide standard electronic data set.4 MPOG’s goal is clinical research, while AQI’s goal is local quality management. This has allowed cooperation rather than competition between our efforts. At present we have working solutions for 6 of the 12 most common AIMS, and partial solutions for the remainder. This continues to be an area requiring heavy investment at both the local and national levels.

Missing Data Is an Ongoing Challenge that Requires Investment

When using NACOR to answer clinical or administrative questions, we must first deal with the issue of missing data. Because of the heterogeneous nature of the information collected, and the vagaries of digital translation and transmission, it is not unusual for data elements to be missing from case records. This can happen with data for individual cases on a random basis, but is more often a systematic occurrence at the level of the practice or facility software, when a specific data element is not captured, not mapped (or mapped incorrectly), or not transmitted to NACOR.

To address these challenges, we routinely audit the submissions of every practice, systematically identify missing elements, and work with the local AQI Champion to resolve issues, but this is a time-consuming process that may be overwhelmed by the sheer quantity of information flowing in. The practical consequence is that the answer to any research query must begin with an analysis of the number of cases that include the necessary data elements, and an assessment of the representativeness of the resulting data set. Fortunately, when beginning with 17 million records, even a query requiring very specific data from AIMS (e.g., the timing of muscle relaxant use in abdominal surgery) can be addressed with tens of thousands of records from dozens of practices, with good balance by geographic region, hospital size and academic versus private practice.

The Need to Fill Gaps in Knowledge: Postoperative Follow-up

A final challenge in NACOR is posed by data that simply don’t exist but would be highly desirable. Approximately half of anesthesia practices do not contact their patients to inquire about postoperative outcomes following PACU discharge, and thus cannot transmit any such information to NACOR. Having those groups participate in the administrative data gathering portion of the registry, however, gives the AQI access to the practice and practitioners and creates an opportunity for coaching and exhortation.

We have created and published how-to instructions for these groups, broadcast consensus standards for postoperative outcome assessment, and promoted the products of vendors who can help create these data. In the long run every practice and every provider will be required to document their performance in order to comply with regulatory requirements. One mission of the AQI is to provide the easiest path forward for these groups. Doing so will help make registry participation indispensable, and thus contribute to the sustainability of our efforts.

Summary

Although the ASA’s investment in the AQI and NACOR has been substantial, progress has been rapid enough to demonstrate its value. Any registry is an exercise in altruistic common good—the value of the final product is determined by the number of practices and providers who choose to participate—which suggests that creation through a specialty society, using a portion of every member’s dues, may be the most effective way to get it launched. Once a sufficient mass of data has accumulated, the resulting information will be of use to a broad range of stakeholders and will provide the society and its members with utility in many different venues.

In NACOR’s case there have been a number of uses of the accumulated data including local quality improvement, hospital regulatory requirements, individual maintenance of certification, clinical research, investigation of patient-centered outcomes, federal research grants and contracts, industry market research, administrative benchmarking for hospitals and practices, non-CME education, regulatory representation of the specialty in Washington, and up-to-date national information on the practice of anesthesiology for ASA’s leadership.

As a next step, the AQI is leveraging its experience in creating NACOR to develop the next generation of clinical registries, built from the ground up to work with electronic data. A new model registry will begin with collaboration by professional societies around a specific service line or area of care (e.g., obstetrics), then will follow with expert definition of desired outcome metrics. Registry participants will build standard structured data elements into the clinical record systems of their EHR, in such a way that subsequent transmission to a registry is facilitated. We have embarked on several new collaborations to test this model, and hope in the long run to achieve both universality of reporting and commonality of data.