Institution of Monthly Anesthesia Quality Reports Does Not Reduce Postoperative Complications Despite Improved Metric Compliance

Abstract

Background

While quality programs have been shown to improve provider compliance, few have demonstrated conclusive improvements in patient outcomes. We hypothesized that there would be increased metric compliance and decreased postoperative complications after initiation of an anesthesiology quality improvement program at our institution.

Methods

We performed a retrospective study of all adult inpatients having anesthesia for a twelve-month period that spanned six months before and after program implementation. The primary outcome was the rate of complications in the post-implementation period. Secondary outcomes included the change in proportion of complications and compliance with quality metrics.

Results

We studied a total of 9,620 adult inpatient cases, subdivided into pre- and post-implementation groups (4,832 vs 4,788.) After multivariate model adjustment, the rate of any complication (our primary outcome) was not significantly changed (32% to 31%; adjusted P=0.410.) Of the individual complications, only wound infection (2.0% to 1.5%; adjusted P=0.020) showed a statistically significant decrease. Statistically and clinically significant increases in compliance were seen for the BP-02 Avoiding Monitoring Gaps metric (81% to 93%, P<0.001), both neuromuscular blockade metrics (NMB-01 76% to 91%, P<0.001; NMB-02 95% to 97%, P=0.006), both tidal volume metrics (PUL-01 84% to 93%, P<0.001; PUL-02 30% to 45%, P<0.001), and the TEMP-02 Core Temperature Measurement metric (88% to 94%, P<0.001).

Conclusions

Implementation of a comprehensive quality feedback program improved metric compliance but was not associated with a change in postoperative complications.

Introduction

The performance of anesthesia providers can be measured by drawing from the wealth of structured data available in modern electronic medical record systems. Automated vital sign recording in the anesthesia record coupled with structured notes for preoperative, handoff, and postoperative care allows for precise and objective measurement of perioperative quality. While anesthesiology quality initiatives have been shown to improve process compliance, the real benefit to patients is in improved patient outcomes.[1–3]

The Multicenter Perioperative Outcomes Group (MPOG) and Anesthesiology Performance Improvement and Reporting Exchange (ASPIRE) were established with the goal of improving perioperative patient care and safety through the use of electronic healthcare data and quality improvement initiatives supported with evidence-based medicine.[4,5] ASPIRE uses MPOG data to generate provider-specific performance feedback across various domains of care.

Our department implemented ASPIRE to provide monthly “report cards” measuring perioperative quality metrics for individual providers as well as the department. This program enabled our anesthesia providers to benchmark performance against peers in the department, and to benchmark our department against other participating institutions.

Effective intervention strategies ultimately rely on both motivation to change and the process of behavioral change itself.[6] The transtheoretical model of “readiness for change” describes stages of precontemplation, contemplation, preparation, action, and maintenance. Practice feedback reports can push clinicians to the next stage.

There are few research studies linking measures of anesthesiologist quality with outcomes. A 2002 study by Silber et al found that lack of board certification was associated with worse outcomes.[7]

In a previous article our group demonstrated that quality metric performance improved with the institution of this quality program.[8] In this study we hypothesize that improved compliance with process-oriented quality metrics translates into improved patient care by a reduction in the rate of any complication in the post-operative period.

Methods

This retrospective study was approved by the Institutional Review Board at the Memorial Sloan Kettering Cancer Center. The study followed the STROBE (strengthening the reporting of observational studies in epidemiology) and RECORD (reporting of studies conducted using observational routinely collected health data) reporting guidelines.[9,10] We included all patients who had anesthesia at our institution between April 1, 2017 and April 1, 2018. Patients were excluded if they were under 18 years of age, had an ASA physical classification score of 5, or were outpatients (defined as zero day length of stay). When a patient had more than one case in the study period, only the first case was retained to meet the assumption of independent observations. As the first ASPIRE email was sent to providers at the end of September, we used a cutoff date of October 1, 2017 to separate the pre- and post-ASPIRE groups. Existing monthly quality meetings were continued, so there was no “wash-out” or “wash-in” period used. In advance of the first email distribution, all staff were educated about the ASPIRE program via educational lectures at two dedicated staff meetings. Additionally, one on one sessions were held by the ASPIRE leadership for staff with additional questions or concerns. Finally, the initial “report card” email was preceded by an introductory and explanatory email at two weeks and one week prior to implementation.

ASPIRE quality measure compliance data was acquired from the MPOG Coordinating Center and linked to our local MPOG repository and institutional data warehouse using MPOG case identifiers. Supplemental Table 1 shows a list of all measures that were used throughout the study period. Each case-measure outcome was one of excluded, passed, or failed.

Postoperative complications were chosen based on their relationship to one or more of the process measures studied. The BP-01 and BP-02 measures monitor the anesthesiologist’s ability to avoid hypotension, whether measured or not. Hypotension during non-cardiac surgery has been linked to myocardial injury, acute kidney injury, and compartment syndrome. [11,12] The MED-01 measure monitors the frequency of use of naloxone and flumazenil for narcotic or benzodiazepine overdose. Excess benzodiazepine is linked to postoperative delirium.[13] The three temperature measures (TEMP-01, -02, -03) focus on avoiding hypothermia. Hypothermia has been linked to adverse myocardial outcomes, surgical wound infection, and is thought to contribute to postoperative delirium.[14,13] The NMB-01 and -02 measures assess how well anesthesiologists properly reverse neuromuscular blockade. Inadequate reversal is associated with postoperative pulmonary complications.[15] ASPIRE measures to ensure low tidal volume (PUL-01 and -02) seek to limit the damage caused by hyperventilation, which can lead to postoperative pulmonary complications.[16] The postoperative nausea and vomiting (PONV) measures assess compliance with current best practices to prevent PONV.[17] The TOC-02 measure is met if the anesthesia provider documents a formal handoff to the recovery unit team, a practice which is thought to reduce adverse events.[18] Postoperative cardiac or pulmonary complications can lead to unexpected intensive care unit admissions.[19] The inpatient 30-day mortality rate for patients who had a postoperative complication was 9.84% in 2006.[20]

Postoperative outcomes were determined from our institutional data warehouse using the best clinical or administrative data available for each outcome. Mortality at 30 days was determined from our institutional vital status registry, which includes both in-hospital and out-of-hospital deaths. Out of hospital death reports come from family notifications, the Social Security Death Index, obituary searches, and the New York City Department of Health. Deaths occurring within two days of surgery were validated with manual chart review. Vital status for all patients was captured on May 22, 2019, slightly more than one year after the end of the date range for this study. Cardiovascular, delirium, and pulmonary complications were determined using the International Statistical Classification of Diseases 10th Revision (ICD-10) discharge codes for the visit that included the patient’s procedure using mappings that have been described elsewhere.[21–23] Compartment syndrome was assessed using ICD-10 diagnosis codes starting with T79.A, M79.A, or M62.2. Postoperative wound infection was assessed using ICD-10 diagnosis codes matching T81.4*XA where “*” is any value. Discharge code outcomes were considered true if the patient’s visit had at least one matching discharge code not marked as “Present on Admission”. Immediate postoperative transfer from the OR to the ICU was determined based on the structured handoff in the anesthesia record. An outcome of postoperative nausea and vomiting was determined to be present for a case if any antiemetic medication was administered during the first 6 hours of the postoperative acute care unit stay. Antiemetic medications include ondansetron, palonosetron, granisetron, prochlorperazine, aprepitant, metoclopramide, and dronabinol. The postoperative nausea vomiting outcome was limited to general anesthesia cases. Renal failure was determined to be present if any provider on the case failed the ASPIRE AKI-01 metric.[24] This metric defines renal failure as a 0.3 mg/dL or higher increase in serum creatinine from baseline within 48 hours, or an increase in serum creatinine to 1.5 times baseline or higher.

Statistical Analysis

Compliance with ASPIRE metrics pre and post implementation were compared using the chi-square test.

To examine whether ASPIRE implementation decreased the rate of complications within the inpatient population, we tested whether the proportion of complications were different in the pre-ASPIRE versus post-ASPIRE periods. Statistical tests performed were the chi-square test of independence or Fisher’s exact test if the number of complications was small. For complications with at least 50 events, an adjusted P value for the ASPIRE coefficient was calculated using multivariable logistic regression predicting outcomes of interest after adjusting for age, sex, ASA physical status (3 and 4 vs 1 and 2), and anesthesia duration. A value of P < 0.05 was considered significant. R version 3.5.3 (R Foundation for Statistical Computing, Vienna, Austria) was used for all statistical analyses.

Results

There were 45,876 anesthetic cases at our institution between April 1, 2017 and April 1, 2018. After excluding outpatients and visits after the index visit, 9,995 cases were in the sample. Excluding pediatric (age ≤ 18) and ASA 5 cases resulted in a final total of 9,620 first case unique adult inpatient cases.

The demographics of the study cohort are described in Table 1. The cohorts were similar in age, BMI, female gender, ASA Physical Status over 2, and rate of emergency procedures. Anesthesia duration was slightly longer in the post-ASPIRE cohort, with a median 222 minutes (IQR 127, 324) versus 198 minutes (IQR 101, 298) in the pre-ASPIRE cohort. Distribution of cases among surgical services was similar except for gastroenterology, interventional radiology, and urology.

Table 1.

Distribution of baseline characteristics stratified by pre and post MPOG/ASPIRE period for inpatient first cases

Characteristic	Overall N = 9,620	Pre-ASPIRE N = 4,832	Post-ASPIRE N = 4,788	P value
Age at time of procedure	62 (51, 71)	62 (51, 71)	62 (51, 71)	0.6
BMI (kg/m2)	26.6 (23.3, 30.7)	26.5 (23.2, 30.5)	26.8 (23.4, 31.0)	0.012
Unknown	159	85	74
Duration of anesthesia (mins)	211 (114, 310)	198 (101, 298)	222 (127, 324)	<0.001
Female	4,988 (52%)	2,464 (51%)	2,524 (53%)	0.10
ASA Physical Status	7,618 (79%)	3,911 (81%)	3,707 (77%)	<0.001
1	54 (0.6%)	25 (0.5%)	29 (0.6%)
2	1948 (20%)	896 (19%)	1052 (22%)
3	6886 (72%)	3516 (73%)	3370 (70%)
4	732 (7.6%)	395 (8.2%)	337 (7.0%)
Emergency Procedure	223 (2.3%)	125 (2.6%)	98 (2.0%)	0.091
Service				<0.001
Breast	123 (1.3%)	65 (1.3%)	58 (1.2%)
Colorectal	915 (9.5%)	461 (9.5%)	454 (9.5%)
Gastric/Mixed Tumor	654 (6.8%)	321 (6.6%)	333 (7.0%)
Gastroenterology	760 (7.9%)	439 (9.1%)	321 (6.7%)
Gynecology	691 (7.2%)	330 (6.8%)	361 (7.5%)
Head & Neck	674 (7.0%)	303 (6.3%)	371 (7.7%)
Hepatopancreatobiliary	705 (7.3%)	346 (7.2%)	359 (7.5%)
Interventional Radiology	1281 (13%)	703 (15%)	578 (12%)
Neurosurgery	679 (7.1%)	335 (6.9%)	344 (7.2%)
Orthopedics	412 (4.3%)	210 (4.3%)	202 (4.2%)
Other	201 (2.1%)	104 (2.2%)	97 (2.0%)
Plastic	228 (2.4%)	104 (2.2%)	124 (2.6%)
Pulmonary	247 (2.6%)	123 (2.5%)	124 (2.6%)
Thoracic	1174 (12%)	565 (12%)	609 (13%)
Urology	876 (9.1%)	423 (8.8%)	453 (9.5%)

Statistics presented as median (IQR) for continuous variables, n (%) for count variables. P value computed using Mann-Whitney U test for continuous variables and chi-square test for count variables. Service “Other” includes Anesthesia, Bone Marrow Medicine, Cardiology, Dental, Ophthalmology, Pediatric Surgery, and Radiation.

Abbreviations: BMI=body mass index, ASA=American Society of Anesthesiologists

The postoperative complication rates for patients in the study cohort are shown in Table 2. The incidence of any complication was not significantly changed (32% to 31%; adjusted P=0.410.) Of the individual complications, only wound infection (2.0% to 1.5%; adjusted P=0.020) showed a statistically significant decrease. While the decrease in 30-day mortality was significant prior to adjustment (3.8% to 2.8%, P=0.005) it was not significant after multivariable adjustment (adjusted P=0.410.)

Table 2.

Proportion of patients who experienced clinical complications pre- vs post-ASPIRE implementation within inpatient first case cohort.

Complication	N	Pre-ASPIRE	Post-ASPIRE	P	Adj P
Any Complication	9,620	1,523 (32%)	1,490 (31%)	0.689	0.410
30-Day Mortality	9,620	183 (3.8%)	132 (2.8%)	0.005	0.230
Cardiovascular	9,620	24 (0.5%)	28 (0.6%)	0.653	NA
Compartment Syndrome	9,620	0 (0%)	1 (<0.1%)	0.498	NA
Delirium	9,620	102 (2.1%)	106 (2.2%)	0.782	0.729
ICU Transfer	9,620	5 (0.1%)	17 (0.4%)	0.018	NA
PONV (within GA)	8,154	662 (16%)	669 (16%)	0.957	0.860
Pulmonary	9,620	429 (8.9%)	418 (8.7%)	0.826	0.826
Renal Failure	9,620	323 (6.7%)	310 (6.5%)	0.708	0.203
Wound Infection	9,620	95 (2.0%)	72 (1.5%)	0.097	0.020

Statistics presented as n (%). P value calculated using the chi-square test of independence or Fisher’s exact test was performed for each complication, depending on sample size. Adjusted P value for ASPIRE coefficient calculated using multivariable logistic regression predicting outcomes of interest after adjusting for age, sex, ASA physical status (3/4 vs 1/2), and anesthesia duration. This value was only calculated for complications with at least 50 events.

Abbreviations: ASA=American Society of Anesthesiologists, ICU=Intensive Care Unit, PONV=Postoperative nausea and vomiting, GA=general anesthesia.

The compliance rates for the ASPIRE metrics are shown in Table 3. Each metric has a different denominator since not every case qualified for every metric. Statistically and clinically significant increases in compliance were seen for the BP-02 Avoiding Monitoring Gaps (81% to 93%, P<0.001) metric, both neuromuscular blockade metrics (NMB-01 76% to 91%, P<0.001; NMB-02 95% to 97%, P=0.006), both tidal volume metrics (PUL-01 84% to 93%, P<0.001; PUL-02 30% to 45%, P<0.001), and the TEMP-02 Core Temperature Measurement metric (88% to 94%, P<0.001).

Table 3.

Pass rate for quality measures pre- vs post-ASPIRE implementation within inpatient first case cohort.

Quality Measure	N	Pre-ASPIRE, N = 4,832	Post-ASPIRE, N = 4,788	P value
BP-01 Low MAP Prevention	9,412	4,666 (98%)	4,598 (99%)	0.14
BP-02 Avoiding Monitoring Gaps	9,328	3,843 (81%)	4,290 (93%)	<0.001
MED-01 Avoiding Medication Overdose	8,391	4,191 (100%)	4,186 (100%)	0.016
NMB-01 Train of Four Taken	6,822	2,497 (76%)	3,206 (91%)	<0.001
NMB-02 Reversal Administered	6,785	3,111 (95%)	3,399 (97%)	0.006
PONV-01 Postoperative Nausea and Vomiting	5,388	2,457 (89%)	2,360 (90%)	0.6
PUL-01 Low Tidal Volume <10 mL/kg IBW	5,891	2,408 (84%)	2,818 (93%)	<0.001
PUL-02 Low Tidal Volume < 8 mL/kg IBW	5,891	861 (30%)	1,361 (45%)	<0.001
TEMP-01 Active Warming	7,280	3,506 (98%)	3,667 (99%)	0.4
TEMP-02 Core Temperature Measurement	7,663	3,336 (88%)	3,634 (94%)	<0.001
TEMP-03 Perioperative Temperature Management	7,651	3,784 (100%)	3,857 (100%)	n/a
TOC-02 Post Anesthetic Transfer of Care	9,520	4,742 (99%)	4,688 (99%)	0.053

Statistics are presented as n (%). N is the subset of cases that qualified for that metric. When a case metric was evaluated for multiple providers the result for the earliest qualifying attending was used. The chi-square test of independence was performed for each metric.

Abbreviations: MAP=mean arterial pressure, MI=myocardial infarction, IBW=ideal body weight; n/a=not applicable.

Discussion

Hospital quality improvement initiatives are widespread because of a recognition that better patient care requires sustained high quality. The hope is that high quality will improve objective outcomes. In our prior work we showed that the implementation of a quality improvement plan centered around a monthly quality report card was associated with improvement in several process metrics, but no improvement in outcome metrics.[8] In this study we looked at the effects of this program on postoperative complications in an inpatient cohort. We found no change in overall complication rate but did find a decrease in wound infection rate.

Six out of 14 quality metrics showed significant improvement in the post implementation period. The observed improvement in wound infection rate could be partially explained by collective improvement in the core temperature measurement metric (TEMP-2: 88% to 94%, P<0.001,) as hypothermia has been associated with surgical site infection.[25] The other improved metrics for blood pressure measurement, neuromuscular blockade reversal, and low tidal volume cannot be linked directly to improved wound infection. The substantial improvement in compliance with the pulmonary and neuromuscular blockade measures would suggest a reduction in postoperative pulmonary complications, but no such change was observed. Also, the improvement in BP-02 was not accompanied by a reduction in postoperative cardiovascular complications, compartment syndrome, or acute kidney injury. However, the improvement in metrics spanning several areas suggests that the quality program implementation led to an improved quality focus by the entire group.

The lack of improvement in other quality metrics was likely because providers were already over 95% compliant pre-implementation.

There are few published studies associating a quality improvement program with a reduction in postoperative complications. In one study a perioperative redesign of glucose measurement and treatment was instituted along with a computerized decision support system to remind providers to check glucose and deliver appropriate therapies. All observed metrics associated with glucose management improved, including a reduction in surgical site infections.[26] A study of cardiac surgery patients demonstrated a reduction in pulmonary complications after the implementation of a quality improvement program.[27] A study of trauma patients confirmed that participation in a regional collaborative quality initiative program improved outcomes and reduced costs.[28] Monthly report cards for anesthesia departments have been described in the literature, but have not been associated with a reduction in postoperative complications.[29] Much of the published work in anesthesia informatics interventions use outcomes strongly associated with provider behavior such as blood pressure measurement gaps.[30] Postoperative patient outcomes involve more factors than just the anesthesiology team so associations between intervention and outcome are more difficult to demonstrate.

The results in this study confirm earlier results showing that improvement in process measures does not reliably translate to better postoperative outcomes. For example, improvement in the timing of antibiotic prophylaxis did not reduce surgical site infection.[31] The reason for the lack of association between better practice and better outcomes is dependent on the intervention and the institution. One common reason is that anesthesia teams at the institution are already regularly performing the intervention. In this situation, an incremental improvement cannot be expected to yield measurable improvements. For example, in our study the pass rate for the PONV-01 measure was already 89% before ASPIRE and afterward only increased to 90%. An increase to 95% or even 99% may not have been enough to measurably improve the postoperative outcome of PACU rescue with supplemental antiemetic.

The lack of association between quality measure performance and result is not a reason to stop the behaviors encouraged by the measures. The measures are evidence-based and supported by many anesthesiologists at a variety of institutions. Implementing these quality measures as part of a coordinated, structured program promotes a culture of self-reflection which is a necessary component of professionalism in anesthesiology.[32] However, the lack of a strong association between a well implemented quality program and improved patient outcomes makes quality programs difficult to justify in hospital environments that are focused on return on investment.

This study has some limitations. The before-after study design is susceptible to confounding by unmeasured concurrent changes in practice. Several complications are collected from administrative data which may not capture all postoperative complications. Mortality in this cohort may be underreported since some deaths may not have been communicated to the hospital vital status registry by the time it was queried for this study. Quality programs are by necessity individualized to an institution’s practice and culture, so implementation of this program at a different center may result in a different pattern of improvement. While all staff were obligated to participate in the program, we were unable to assess the amount of time individuals spent reviewing the report cards.

After implementing a comprehensive anesthesia quality improvement program, we found no improvement in postoperative complications despite significant improvements in quality metric compliance. Future work is needed to determine if these results can be replicated at institutions beyond our own.