Perception Is Reality: quality metrics in pancreas surgery – a Central Pancreas Consortium (CPC) analysis of 1399 patients

Abstract

Introduction

Several groups have defined pancreatic surgery quality metrics that identify centers delivering quality care. Although these metrics are perceived to be associated with good outcomes, their relationship with actual outcomes has not been established.

Methods

A national cadre of pancreatic surgeons was surveyed regarding perceived quality metrics, which were evaluated against the Central Pancreas Consortium (CPC) database to determine actual performance and relationships with long-term outcomes.

Results

The most important metrics were perceived to be participation in clinical trials, appropriate clinical staging, perioperative mortality, and documentation of receipt of adjuvant therapy. Subsequent analysis of 1399 patients in the CPC dataset demonstrated that a R0 retroperitoneal and neck margin was obtained in 79% (n = 1109) and 91.4% (n = 1278) of cases, respectively. 74% of patients (n = 1041) had >10 lymph nodes harvested, and LN positivity was 65% (n = 903). 76% (n = 960) of eligible patients (surgery first approach) received adjuvant therapy within 60 days of surgery. Multivariate analysis demonstrated margin status, identification of >10 lymph nodes, nodal status, tumor grade and delivery of adjuvant therapy within 60 days to be associated with improved overall survival.

Conclusions

These analyses demonstrate that systematic monitoring of surgeons' perceived quality metrics provides critical prognostic information, which is associated with patient survival.

Introduction

The issue of quality in health care in the United States continues to be intensely debated, for a variety of reasons. Firstly, it is unclear whether documentation of and adherence to the multitude of proposed quality metrics actually affect clinical outcomes.^{1, 2, 3} Secondly, the financial ramifications of measuring, implementing and demonstrating success or failure with regards to quality metrics are multi-billion dollar outcomes, with heavily invested parties.^{4, 5, 6, 7} Going forward, it will be critical to decipher what clinical outcomes are most important to not only providers but also patients, and determine what role payers will have with regards to reimbursement.

There is no shortage in the literature about quality improvement projects in surgery. Perhaps most visible is the American College of Surgeons National Surgical Quality Improvement Program (NSQIP), developed to identify and address variability in surgical care at a national level. While the initial investment in human and financial capital appeared to be paying off both fiscally and in higher quality surgical care, recent reports indicate that NSQIP may not be the most useful quality improvement initiative.^{8, 9, 10, 11, 12, 13} Others, including the progressive Leapfrog Initiative and RAND corporation, have also put forth initiatives designed to improve value in health care, primarily by improving patient outcomes in a cost-effective manner.^{14, 15} These have been met with moderate resistance and mixed results and the lack of a collective mandate from these organizations is undoubtedly a factor.

Whereas the fore-mentioned programs most frequently address process and systems improvement at an institutional level, there is a paucity of disease site-specific initiatives in place to address complex, though perhaps, rarer, disease processes. For example, pancreatectomy represents an ideal operation to refine disease-specific quality measurement with both malignant and benign processes contributing to its requirement. The conduct of the operation along with its pre- and post-operative care is intricate, and the adverse effects of individual or institutional poor provider quality, or simply the natural history of the disease can be magnified due to significant morbidity inherent to the operation.

This knowledge gap has begun to be addressed by leaders in outcomes research and quality improvement. In 2009, Bilimoria et al. published a consensus statement regarding quality metrics in pancreas surgery, based on expert opinion and an assessment of the power of national cancer registry data to capture the agreed upon components of care surrounding pancreatectomy.¹⁶ Subsequently, Callery et al. queried experts in pancreatic surgery regarding the adequacy, and need for refinement, of previously identified quality metrics.¹⁷

These sentinel studies serve as a basis for the next step in evaluating the utility of these quality metrics. To continue this line of investigation, we sought to validate the usefulness of proposed quality metrics by surveying a national cohort of specialist surgeons regarding proposed quality metrics in pancreas surgery, identifying whether survey responses were actually adhered to, and correlating these survey results with clinical outcomes utilizing a comprehensive database from a multi-center consortium.

Methods

Survey development and administration

Following University of Cincinnati Institutional Review Board approval, a 49-item survey was constructed and sent to the membership of the Americas Hepato-Pancreato-Biliary Association (AHPBA) after approval of AHPBA leadership. An email, containing a link to the survey, was sent to 1223 members. This survey was distributed to both United States (71%) and International (29%) HPB surgeons. Survey recipients were, in order of frequency, active (68.9%), candidate (15%) and senior members (3.4%); only 3.6% were non-surgeons (allied health members). The survey was largely comprised of questions regarding pancreatic surgery quality metrics identified in an original report by Bilimoria et al.¹⁶, and was augmented with additional metrics believed to be clinically relevant to the comprehensive care of a patient undergoing pancreatic surgery. The survey questions were generally categorized into: (i) infrastructure, such as availability of multimodality care with one institution, presence of on-site interventional radiology, institutional participation in clinical trials, etc.), (ii) provider-specific details, including the monitoring of surgeon case volume, 10 or more lymph nodes harvested during pancreatectomy, operative time less than 10 h, and (iii) documentation efforts, including clear documentation of resection margin status, tumor grade, reason for patient not undergoing potentially curative resection when tumors were deemed resectable. Survey respondents were asked to answer, in the affirmative, if the proposed quality metrics were either ‘important’ and/or ‘routinely performed’ at their institution.

Correlation of survey results with clinical outcomes

The Central Pancreas Consortium (CPC) database was queried for all pancreaticoduodenectomy procedures for non-metastatic cancer from 2001 to 2013 (n = 1399). The CPC is a collaboration of tertiary care, academic medical centers that perform a high volume of pancreatic resections annually, and that share de-identified data regarding interventions and outcomes. This dataset has been used extensively for clinical research.^{18, 19} Pertinent quality metrics identified by survey responses were then correlated with CPC data. Specifically, provider and pathology-specific data were evaluated with respect to overall survival. Included in the analyses were objective, measurable outcomes including receipt of adjuvant or neoadjuvant therapy, tumor grade, lymph nodes harvested (number) and pathologic status (negative/positive), pancreatic neck and retroperitoneal margin status (R0/R1/R20) and perioperative (30 day) mortality. All survey questions were either inherent characteristics of CPC members or were captured in their institutional review/research missions.

Statistical analyses

Survey results were organized by both simple frequency tables and stratified by degree of concordance and discordance between what respondent's viewed as ‘important’ versus what are ‘routinely performed’. CPC data, using overall survival as the primary outcome, were subjected to both univariate and multivariate analyses. Univariate analyses were performed either by Chi-square (categorical variables) or Wilcoxon Rank-sum (continuous variables) analyses. Median overall survival was calculated via the method of Kaplan–Meier, and Cox proportional hazards analysis were performed to investigate clinicopathologic factors associated with overall survival (SAS, Cary, NC). A p-value of <0.05 was considered significant.

Results

One hundred and three surveys were finalized, for a response rate of 8.4%. Survey questions generally pertained to three categories of measurement: (i) infrastructure (e.g. availability of computed tomography/magnetic resonance imaging, interventional radiology, endoscopic retrograde cholangiopancreatography on site or participation in clinical trials), (ii) provider-specific (e.g. surgeon is certified by the American Board of Surgery or operative time), and (iii) documentation efforts (e.g. use of the College of American Pathology checklist or nodal status) (Table 1).

Table 1.

Infrastructure, provider and documentation quality metrics queried and analyzed

Infrastructure	Provider	Documentation
Cases discussed by multidisciplinary team	Surgeon ABS certified	Use of College of American Pathology (or equivalent) checklist
High institutional volume (>12 cases/year)	No cancer-directed surgical therapy for stage IV disease	Nodal status (how many harvested, positivity)
Availability of CT/MRI on site	Complication rate/severity	R0 resection rate
Participation in clinical trials	Documentation of reason for no surgery for stage I/II patients	No R2 resections
Chemotherapy and radiation therapy available on-site	Monitoring of surgeon volume
Time to operation	Initiation of adjuvant therapy within 2 months
Institutional monitoring of outcomes (M&M, quality metrics)	Post-pancreatectomy hemorrhage
Availability of ancillary/ICU services 24 h/day	Operative time
Monitoring and documentation of 2 and 5 year survival	Pancreatic leak rate
Readmission rate	Operative blood loss
Risk-adjusted mortality (30/90 day) < 5%	Documentation of receipt (or omission) of adjuvant therapy
Availability of advanced endoscopy/IR on site	Documentation of pre-operative risk/benefit analysis
	Documentation of pre-operative clinical staging
	Documentation of metastases, relationship of tumor to vasculature
	Re-operation rates
	Vascular resection rates

Initially, responses were stratified by what respondents believed to be the most important (Fig. 1) and least important (Fig. 2) quality metrics, and compared to actual institutional practices in each case The most common affirmative responses to the question of ‘is …. an important quality metric in pancreas surgery’ were participation in clinical trials (89.4%), clinical TNM staging for unresected patients (84.6%), and institutional monitoring of estimated blood loss and monitoring of 30/90 day mortality (84.6% and 84.2%, respectively). Conversely, respondents believed a number of proposed quality metrics are unimportant. The most common answers, in terms of belief of unimportance, were: presence of chemotherapy and radiation therapy on-site (70.5%), documentation of both clinical and pathologic stage (73.5%), no cancer-directed surgery for patients with clinical stage IV disease (74.3%) and performing 12 or more institutional cases per year (74.5%).

Figure 1.

Survey respondent beliefs relating MOST important quality metrics to actual performance (most frequent 10 positive responses)

Figure 2.

Survey respondent beliefs relating LEAST important quality metrics to actual performance (most frequent 10 negative responses)

Survey responses were then stratified by respondent discordance (large differences between actual institutional practice and belief of importance, Fig. 3) and concordance (most alignment between perceived importance and actual practice, Fig. 4). Though participation in clinical trials continued to be perceived as the most important, just over half of respondents' institutions (55.5%) actually participated in clinical trials. Large discordance between perceived importance and actual practice was also found in accurate monitoring of blood loss, institutional monitoring of margin negative resection rates, and institutional monitoring of 2 and 5 year overall survival. In contrast, respondents reported the most concordance between perceived importance and actual practice in: (i) documentation of stage-specific treatment plans, (ii) monitoring of institutional case volume, (iii) performance of pharmaceutical VTE prophylaxis in surgical patients, and (iv) no cancer-directed surgical therapy for clinically stage IV patients.

Figure 3.

Top ten quality metrics with the MOST discordance between perceived importance and performance

Figure 4.

Top ten quality metrics with the LEAST discordance between perceived importance and performance

The CPC dataset was queried for objective, measurable patient and pathologic-specific variables that could be considered both (i) quality metrics and (ii) impact long-term outcomes (i.e. overall survival) (Table 2). In the cohort of 1399 patients undergoing PD, 30 day mortality was 2.9%, with a median overall survival of 19.7 months. Approximately 10% of patients underwent neoadjuvant therapy, while 76% of the remaining eligible patients (surgery first approach, 960/1263) underwent adjuvant therapy. Almost 90% of tumors were either moderate or high grade. 74.4% of patients had greater then 10 lymph nodes harvested, and approximately 2/3 of the cohort had carcinoma found in at least one regional lymph node. An R0 pancreatic neck and retroperitoneal margin was found in 91.4% and 79.3% of patients, respectively.

Table 2.

Descriptive analyses of patient and pathologic-specific variables in multi-institutional dataset

Variable	N (%)
Median survival	19.7 months
Receipt of neoadjuvant therapy
No	1258 (89.9)
Yes	136 (9.7)
Unknown	5 (0.4)
Receipt of adjuvant therapy
No	344 (24.6)
Yes	960 (68.6)
Unknown	95 (6.8)
Grade (differentiation)
Well	113 (8.1)
Moderate	815 (58.2)
Poor	442 (31.6)
Unknown	29 (2.1)
Lymph nodes harvested
≤10	358 (25.6)
>10	1041 (74.4)
Lymph node status
Negative	375 (26.8)
Positive	903 (64.5)
Unknown	121 (8.7)
Estimated blood loss (median/25%/75% IQR)	500 (250–800)
Perioperative (30 day) mortality
No	1359 (97.1)
Yes	40 (2.9)
Neck margin
R0	1278 (91.4)
R1	121 (8.6)
Retroperitoneal margin
R0	1109 (79.3)
R1	264 (18.9)
R2	26 (1.9)

Multivariate Cox proportional hazards analysis demonstrated that a number of perceived operative and pathologic variables were indeed associated with long-term survival (Table 3). R1 resections margins—both pancreatic neck and retroperitoneal—as well as lymph node positivity and poorly differentiated tumors were all associated with poorer overall survival (HR 1.37, 1.26, 1.59 and 2.01, respectively). Receipt of adjuvant therapy and number of lymph nodes harvested were also associated with survival, though in a more complex fashion. Because the proportional hazards assumption was not true for both variables (i.e. hazard for the outcome in question was not consistent across the continuum of survival), hazard ratios were calculated at the 25th, 50th, and 75th percentile of overall survival. For both receipt of adjuvant therapy and a lymph node harvest of >10 lymph nodes, as overall survival increased, the hazard ratio for death decreased. Specifically, the HR (95% CI) for death with respect to receipt of adjuvant therapy for sequential 25th/50th/75th survival was 1.78 (1.38–2.31), 1.28 (1.03–1.58), and 0.7 (0.58–0.85) respectively. For lymph node harvest of>10, the HR (95% CI) was 2.99 (2.25–3.98), 1.89 (1.5–2.38) and 0.83 (0.67–1.01) for sequential 25th/50th/75th survival, respectively. As such, the data indicate that each of these clinical variables were associated with improved survival as patients lived longer.

Table 3.

Multivariate analyses of pathologic variables and association with survival

Variable		Hazard ratio	95% CI	p-Value
Retroperitoneal margin	R0	Referent
	R1	1.37	1.13–1.64	<0.01
	R2	1.55	0.79–2.70	0.16
Pancreatic neck margin	R0	Referent
	R1	1.26	0.97–1.62	0.07
Adjuvant Therapy	Yes vs. No at 25th percentile of OS	1.78	1.38–2.31	<0.01
	Yes vs. No at 50th percentile of OS	1.28	1.03–1.58	0.03
	Yes vs. No at 75th percentile of OS	0.70	0.58–0.85	<0.01
Lymph node harvest	>10 vs. <= 10 at 25th percentile of OS	2.99	2.25–3.98	<0.01
	>10 vs. <= 10 at 50th percentile of OS	1.89	1.50–2.38	<0.01
	>10 vs. <= 10 at 75th percentile of OS	0.83	0.67–1.01	0.07
Lymph node status	Negative	Referent
	Positive	1.59	1.32–1.93	<0.01
Grade	Well differentiated	Referent
	Moderately differentiated	1.48	1.09–2.05	0.01
	Poorly differentiated	2.02	1.47–2.83	<0.01

OS--Overall survival.

Discussion

The definition of ‘quality’ continues to evolve, but most globally it must refer to the ‘real world’ implementation, documentation and outcomes in patients undergoing treatment for their diagnosis. Here, we have attempted to provide the first disease-specific study comprehensively incorporating these three facets of care in patients requiring pancreaticoduodenectomy. We have done this by first assessing national opinions, from a variety of providers in a spectrum of practice settings, then using these results to understand and validate the impact of these metrics on overall survival of patients undergoing resection for pancreatic cancer—arguably the most important clinical outcome.

We have shown that pancreas surgeons most often value institutional and multi-disciplinary care of the patient above many other objective measures of quality, and practitioner-level quality metrics such as case number/year, practitioner decision-making and documentation are less important to providers nationally. Furthermore, there is significant discordance between what practitioners believe to be important and what is actually being practiced. Participation in clinical trials, a unique outlier, is almost universally believed to be important, but only approximately half of respondents to our survey actually offered clinical trials to patients. This discordance may not be as significant as others, as not every patient with pancreas cancer requires participation in clinical trials—there are level 1 data that help guide management for this disease. However, there is significant discordance beyond clinical trial. The remaining parameters with the most discordance between perceived importance and actual practice are: institutional monitoring of EBL, margin status, 2 and 5 year survival rates, and administration of appropriate peri-operative antibiotics. This level of discordance warrants investigation into how such variability, or failure to meet a perceived demand, can be addressed within the current systematic framework of caring for the patient with pancreas cancer.

However, we have also found significant agreement between perceived quality metrics and actual practices in our study. The most concordance, near identical percentages of affirmative responses to both belief of importance and institutional behavior, were found in documentation of stage-specific treatment plans and monitoring of institutional case volume. Following these two parameters with the highest concordance, the 3rd, 4th, and 5th highest levels of concordance were found in application of appropriate VTE prophylaxis, appropriate withholding of cancer-directed surgery for patients with stage IV disease, and availability of multidisciplinary care. Finally, we have shown that measurable, objective quality metrics are, in fact, associated with long-term outcome. Our data confirm that previously identified, and perceived, variables such as margin status, grade and lymph node status, and receipt of multi-disciplinary care are associated with improved survival. Logically, then, if these metrics matter, we should keep track of them. Resources and efforts to maintain treatment and outcome-specific databases, in an effort to inform quality improvement initiatives, would appear to be worthy of institutional or society-level support.

In 2009, the American College of Surgeons' Pancreatic Cancer Quality Indicator Development Expert Panel was assembled to parlay years of pancreas surgery experience into 43 valid quality indicators.¹⁶ Subsequently, Bilimoria and colleagues used the American College of Surgeons National Cancer Data Base to evaluate national adherence to these measures, finding that significant variability exists in the United States with respect to these proposed metrics. In 2013, Dr. Callery and colleagues surveyed pancreas surgeons in the United States, asking them to stratify the importance of proposed quality metrics in pancreas surgery and align them with Institute of Medicine health care quality domains.¹⁷ Respondents to this survey ranked perioperative (mortality, complications, etc.) measures most importantly, while patient-centered measures such as patient satisfaction and cost were least important. These data provided insight into the national pulse of pancreas surgeons with regards to quality metrics, and helped us to understand where patient-centeredness may be improved. However, despite the strengths of these studies, a common knowledge gap persists, specifically, as to whether these beliefs, and/or practice patterns, translate into improved patient outcomes.

Most generally, the current study addresses the specific question that many skeptics of quality improvement projects pose. ‘Does any of this really matter’? Clinical trials are our most reliable source of data to suggest that an intervention, in clearly defined populations, improves survival. However, clinical trial participants represent a homogeneous, largely unrepresentative patient population without broader application to the ‘real world’ of pancreas cancer heterogeneity. To date, we have had no ‘real world’ evidence that adherence to quality metrics in pancreatic cancer actually correlate with long-term outcomes. It is our hope that these data contribute to the movement for measurement, documentation and application of key quality metrics that have been proposed by experts and substantiated by practitioners, now through at least two national surveys.

There are limitations to our study that should be noted. Most importantly, our study cohort is small and lacks a comparator group; the CPC is comprised of high volume centers that largely adhere to the quality metrics posed. Without a cohort of centers that either don't document or don't adhere to many of the metrics described herein, the conclusions of our study are somewhat weakened. Secondly, not every metric outlined is easily associated with overall survival (e.g. presence of ERCP on site). Though we believe many of these institutional and provider-specific features to be important to the overall care of the patient, quantifying them is difficult—the qualitative nature of many of these metrics (e.g. clinical trial availability, possessed by all CPC members) don't lend themselves to quantitative analyses. Thirdly, some metrics highlighted in this study are not explicitly documented in the CPC dataset. This issue is a microcosm of a broader resource utilization problem—who will pay for the ever-increasing documentation, cataloguing and continual updating required to make these datasets useful? Perhaps adherence to, and documentation of these quality metrics is prohibitively expensive in the current health care climate.

Our data reflect national attitudes towards quality metrics in pancreatic surgery, how often these metrics are actually adhered to, and how selected, measurable metrics are associated with clinical outcomes. Importantly, the addition of data from the CPC demonstrates that ‘it can be done’. Measurement, adherence, and analysis with regards to perceived quality metrics do matter. A national goal should be to strive towards adoption and systematic use of these metrics as we aim to improve outcomes for the pancreas cancer patient.

Conflicts of interest

None to declare.