Outcomes Improvement is Not Continuous Along the Learning Curve for Pancreaticoduodenectomy at the Hospital Level

Abstract

Background

Most studies on learning curves for pancreaticoduodenectomy have been based on single-surgeon series at tertiary academic centers or are inferred indirectly from volume-outcome relationships. Our aim is to describe mortality rates associated with cumulative surgical experience among non-teaching hospitals.

Study Design

Observational study of a statewide in-patient database. Analysis included hospitals that began performing pancreaticoduodenectomy between 1996–2010, as captured by the California Office of Statewide Health Planning and Development database. Cases were numbered sequentially within each hospital. The same sequential series (e.g. first 10 cases, 11th through 20th cases) were identified across hospitals. The outcome measure was in-hospital mortality.

Results

A total of 1,210 cases from 143 non-teaching hospitals were analyzed. The average age was 63 years and the majority of patients were non-Hispanic white. The median overall mortality rate was 9.75%. The mortality rate for the first ten aggregated cases was 11.3%. This improved for subsequent cases, reaching 7.1% for the 21^st-30^th cases. However, the mortality rate then increased, reaching 16.7% by the 41^st-50^th cases before falling to 0.0% by the 61^st-70^th cases.

Conclusions

Initial improvement in surgical outcomes relative to cumulative surgical experience is not sustained. It is likely that factors other than surgical experience affect outcomes, such as less rigorous assessment of comorbidities or changes in support services. Vigilance regarding outcomes should be maintained even after initial improvements.

INTRODUCTION

Learning and mastering new techniques is a common process that occurs throughout a surgeon’s career. The idea of a learning curve has been used to describe the adoption of new surgical techniques and technology and its associated outcomes. The curve is typically considered to have three parts: the starting point which is a combination of a surgeon’s individual experiences and background, the slope during which the measured parameter defining success is changing with increasing experience, and the plateau at which point there are no further significant changes in success parameters for the surgeon. At this point, the physician is considered experienced. ¹

Detailing the learning curve for a given procedure is a difficult task. In the book “Outliers”, Malcolm Gladwell popularized the notion that 10,000 hours of guided practice is required to achieve mastery in success in any field regardless of personal aptitude.² The data behind that assertion, however, is limited. For surgeons, the mastery of their trade can be broken down into individual skills, such as suturing and gaining exposure, which are practiced in every case. However, for a complex surgical procedure such as open pancreaticoduodenectomy (PD), many factors contribute to the resulting outcome, including the ancillary support system and not just the individual surgeons’ capabilities.

The current understanding of the learning curve for PD is derived from less than ten studies based on single surgeons at tertiary academic centers. ^3–6 Cameron et al. suggest that a surgeon should perform at least 15 PD per year to be considered a high volume surgeon and have improved mortality rates; similarly, Fisher et al. suggest greater than 11 PD per year is sufficient. ^{4, 6} Nevertheless, the finding that a surgeon that does 15 PD per year has improved outcomes does not, strictly speaking, imply that a surgeon’s outcome will improve after he or she reaches the 15^th case. It is unknown whether these findings, based on a single surgeon’s experience at tertiary academic centers with access to advanced endoscopy services and skilled interventional radiologists, can be generalized to all surgeons or hospitals in a community.

Learning curves are sometimes inferred indirectly from volume-outcome relationships based on multi-institutional datasets. For example, Birkmeyer and colleagues found that in-hospital mortality rates at low-volume hospitals were 3- to 4-fold higher than high volume hospitals, demonstrating a strong association between institutional volume and mortality.⁷ This cross-sectional analysis has limited utility in truly depicting the learning curve, as it does not follow progression over time.

Assessment of the true learning curve for open PD across multiple hospitals is essential in helping guide surgical training and evaluation. The aim of this study is to describe the learning curve for open PD at the hospital level by analyzing mortality rates associated with cumulative surgical experience among a large group of hospitals, utilizing a uniquely complete population database from the State of California

METHODS

Retrospective analysis of the California Office of Statewide Health Planning & Development (OSHPD) inpatient-discharge administrative database was performed from 1996 to 2010. This administrative, longitudinal database includes all inpatient discharges from California licensed hospitals.

Inclusion criteria were hospital admissions coded by International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9) procedure codes 52.51 for proximal pancreatectomy, 52.6 for total pancreatectomy with synchronous duodenectomy and 52.7 for radical pancreaticoduodenectomy. In order to isolate hospitals that began performing PD during the study period, hospitals that performed the procedure in 1994 or 1995 were excluded. Cases across all years were numbered sequentially within each hospital. The same sequential series (e.g. first 10 cases, 11th through 20th cases) were identified and aggregated across hospitals. The primary outcome measure was in-hospital mortality

Patient demographics, including age, sex and race, expected primary payer, Charlson comorbidity index, hospital types (i.e., teaching versus non-teaching), sequential series aggregates and in-hospital mortality were recorded. The Charlson comorbidity score is an index of comorbidities based on the presence or absence of certain diagnoses in the patient. These are then combined together in a weighted formula.⁸ Hospital teaching status was defined by the affiliation of the institution with a general surgical residency program.

Statistical analyses were performed using STATA 11.1 software (StataCorp, College Station, TX, USA). Bivariate analysis of in-hospital mortality and sequential series aggregates was performed using the Pearson chi-square test. Statistical significance was accepted at a p-value ≤ 0.05.

RESULTS

A total of 1,210 patients were analyzed (Table 1). The average age was 63 years old, with almost an even distribution of males and females and 61% of patients being non-Hispanic white. As many as 83% of patients had a Charlson comorbidity score of greater than 3. All hospitals analyzed were non-teaching. The average overall in-hospital mortality rate across all institutions was 9.75%.

Table 1.

Univariate analysis of patients undergoing pancreaticoduodenectomy based on the California Office of Statewide Health Planning and Development database from 1996 to 2010.

Demographics	Count (Percent)
Total Admissions	1,210
Age, mean (sd), years	63.2 (15.1)
Sex
Female	597 (49.34%)
Male	613 (50.66%)
Race
Non-Hispanic White	735 (61.05%)
Black	79 (6.56%)
Hispanic	238 (19.77%)
Asian	129 (10.71%)
Indian/Other	23 (1.91%)
Insurance
Medicare or Private Coverage	689 (81.73%)
Medi-Cal, Worker's Compensation, County Indigent Programs, Other Government, Other Indigent, Self Pay	154 (18.27%)
Charlson
0	48 (3.97%)
1–2	150 (12.40%)
3+	1,012 (83.64%)
Hospital Type
Teaching Hospital	0
Non-Teaching Hospital	1,210 (100%)
In-hospital Mortality
Died	118 (9.75%)
Cumulative Experience
1st–10th	673 (56.13%)
11th–20th	233 (19.43%)
21st–30th	112 (9.34%)
31st–40th	69 (5.75%)
41st–50th	54 (4.50%)
51st–60th	32 (2.67%)
61st–70th	16 (1.33%)
71st–80th	10 (0.83%)
Number of Hospitals
1st–10th	143
11th–20th	31
21st–30th	15
31st–40th	8
41st–50th	6
51st–60th	5
61st–70th	2
71st–80th	1

Mortality rates of PD over time were analyzed (Figure 1) and varied from a trough of 2.53% in 2005 to a peak of 16.67% in 1997. This wide variation is likely explained by the refinement of technique and improvement in surgical critical care over the past decade, although there was no statistical difference between mortality rates in 1996 and 2010.

Figure 1.

Mortality rates versus year of admission of patients undergoing pancreaticoduodenectomy based on the California Office of Statewide Health Planning and Development database from 1996 to 2010.

Of all admissions assessed, 56.1% of cases were included in the 1^st-10^th series aggregate. This tapered to 9.3% for the 21^st-30^th cases and 4.5% in the 41^st-50^th series aggregate. A total of 143 institutions performed their first PD within the study period. Of those, 31 hospitals performed greater than 11 cases and this decreased to 6 hospitals that performed more than 40 cases. From 1996 to 2010, the annual caseload varied from 62 (5.12% of total cases) in 1996 to 86 (7.11%) in 2010, with a peak of 97 cases (8.02%) in 2000 (Table 2).

Table 2.

Pancreatiodoudenectomy cases per year based on the California Office of Statewide Health Planning and Development database from 1996 to 2010.

Year	Count (Percent)
1996	62 (5.12%)
1997	66 (5.45%)
1998	74 (6.12%)
1999	67 (5.54%)
2000	97 (8.02%)
2001	96 (7.93%)
2002	90 (7.44%)
2003	76 (6.28%)
2004	80 (6.61%)
2005	79 (6.53%)
2006	80 (6.61%)
2007	80 (6.61%)
2008	82 (6.78%)
2009	95 (7.85%)
2010	86 (7.11%)

Mortality rates of sequential series aggregates of cumulative experience were analyzed (Figure 2). The average mortality rate for the first ten aggregated cases was 11.3%. This mortality rate improved for subsequent cases, reaching 7.1% for the 21^st-30^th cases. However, the mortality rates then increased after the 20^th case, reaching 16.7% by the 41^st-50^th cases before falling to 0.0% by the 61^st-70^th cases.

Figure 2.

Mortality rates of sequential series aggregates of cumulative experience of patients undergoing pancreaticoduodenectomy, based on the California Office of Statewide Health Planning and Development database from 1996 to 2010.

DISCUSSION

As new procedures are incorporated into practice, learning curves are defined in order to understand trends in patient morbidity and mortality over the adoption time frame. PD is an infrequent and complex procedure, thus there has been limited assessment of the learning curve. It has been shown that both individual surgeon experience and hospital experience over time are important factors in determining patient outcomes. ^{7, 9, 10} The exact point along the learning curve when a surgeon is considered proficient has not been defined, but it has been demonstrated that usually more than twelve per year are required for a surgeon to achieve improved patient outcomes; however, these studies are based on single surgeons at tertiary and high-volume academic centers. The availability of skilled endoscopists and interventional radiology services, as well as established postoperative recovery pathways inherently contribute to superior outcomes in patients undergoing PD.^11–16 Such reports are therefore difficult to generalize to the broad surgical community.

In our study, we analyzed 143 non-teaching hospitals’ initial experience with PD in order to describe the learning curve across institutions. In contrast to many single-institution studies to date, our study found that initial improvement in surgical outcomes relative to cumulative surgical experience is not sustained. In other words, a hospital’s PD mortality rate does not decrease in a step-wise fashion; instead, after an initial improvement in mortality rate, there is a rise in mortality before it again improves. Possible explanations of this non-continuous learning curve include expanded patient selection criteria or less rigorous assessment of risk factors. For example, Prakash and colleagues found that for laparoscopic colorectal resection, as surgeons become more experienced there was a trend towards including more technically difficult patients who were normally not considered for laparoscopic colorectal surgeries. This demonstrates the expanded patient selection criteria that could lead to a rise in mortality after an initial improvement.¹⁷ Based on our study findings, it is important to continue to exercise caution throughout the learning curve, even with early improvements in patient outcomes.

There are additional interesting findings from our study. We found an average in-hospital mortality rate of 9.75% across the study time frame. This represents solely in-patient mortality and does not take into account outcomes of patients discharged to skilled nursing facilities, which is a common occurrence after PD. We also examined the morality rate over time and found it to be variable, but with a trend towards improving outcomes from 1996 to 2010. This mortality rate is higher relative to what is routinely reported in contemporary literature. ^{6, 18–21} The higher mortality rate is likely attributed to the fact that we are exclusively analyzing hospitals that performed their 1^st PD within the study time frame and in the process, eliminate most academic and referral centers that have been performing PD prior to 1995. This relegates the analysis to primarily low-volume and non-teaching centers. However, these inclusion criteria precluding high volume referral centers eliminate the publication bias in which the literature tends to report better outcomes than the true real-world outcomes, making our results more applicable to the surgical community.^{22, 23} This high mortality rate further emphasizes the existence of a learning curve as well as the importance of centralization of care at high-volume centers, as this is potentially the most effective intervention to improve pancreatic cancer outcomes.

Another important observation of our analysis is that mortality rates after PD varied from a peak of 17% to 0% late in the learning curve, suggesting that the center’s proficiency with the procedure drastically impacts postoperative outcomes. In a landmark study by the Michigan Bariatric Surgery Collaborative, Birkmeyer and colleagues conducted a study where the technical skills of 20 bariatric surgeons were assessed by peer surgeons and correlated with their clinical outcomes. The authors found that surgeons with skill ratings in the bottom quartile were associated with higher complication, mortality, reoperation and readmission rates.²⁴ While millions of research dollars are invested in the study of biomarkers, chemotherapeutic agents and the biology of pancreatic adenocarcinoma, with modest improvements over decades,^{25, 26} little research has gone into looking at the surgeon’s skill.²⁷ Given the overall poor survival rates for pancreatic cancer, the differences in outcomes between surgeons and within a surgeon’s cases exceed that of any difference brought by advances in chemotherapeutic agents. Perhaps more efforts should be centered on what specifically affects change along the learning curve, for both the surgeon and the center, and identify interventions that would decrease poor outcomes for centers early in the learning curve.

One of the main strengths of this study is its longitudinal and multi-institutional study design. This is the first study to compare multiple different institutions’ initial experiences with PD, thereby making the learning curve data more generalizable. This is possible because the California OSHPD database is a multi-institutional dataset that consistently captures all the cases from the same institutions for multiple years. Other datasets, such as Nationwide Inpatient Sample, NSQIP or Medicare, have either a different sample of hospitals every year or an incomplete sample of cases at the hospital and surgeon levels. Additionally, the California OSHPD database includes patients of all ages undergoing the procedure with the ICD-9 codes of choice, as opposed to the Medicare and Medicaid database, which limits data to only patients above 65 years of age.

Nevertheless, there are important limitations to this study. First, we only assessed patient mortality as our primary endpoint. Utilizing the endpoint of mortality as the sole outcome metric without considering operative variables (estimated blood loss, operative duration) or morbidity rates may not be an accurate depiction of the learning curve of a procedure. ^28–30 We also were unable to analyze 5-year survival and disease-free survival data, which would have provided another interesting facet to the learning curve. Another limitation was that the number of hospitals with greater than 50 cases decreased from 143 to 5 hospitals. The OSHPD database includes data from 1995 to present, thus the number of hospitals that performed their first case within the study period and had a large volume of cases was small. This limits our ability to draw broad conclusions about the learning curve since we have a small sample size. Additionally, we focused on non-teaching and low volume hospitals so our results represent the learning curve in this specific subset of hospitals. Learning in low volume hospitals may be very different than high volume hospitals. As with any study attempting to describe learning curves, this study does not account for personal aptitude. Hambrick and colleagues found that only 30% of the variance in musicians’ and chess players’ performance ranking could be accounted for by how much time they spent practicing. ³¹ Additionally, the learning curve is not adjusted for time. Many aspects of surgery are likely to change over 15 years, including surgical education and technology. While improvements in PD outcomes have been noted over the last 15 years, there was no statistical difference between the mortality rate in 1996 and 2010 within the study cohort. Finally, the common limitation of all learning curve studies to date is that the study does not account for the ancillary support systems, such as gastroenterology or interventional radiology, surrounding the surgeon or the coalescence of a unit or team as volume increases. Schmidt and colleagues analyzed the role of individual surgeon experience compared with overall institutional experience and found that annual PD volume was the only measure that improved mortality. ³² However, they also found that experienced surgeons had improved morbidity, demonstrating the complex interplay between surgeon and institutional experience, which is difficult to quantify.

In conclusion, the learning curve for PD at the hospital level is not continuous. Factors other than surgical experience may affect outcomes, such as expanded patient selection criteria or less rigorous assessment of risk factors. Initial improvements in mortality are not sustained and vigilance regarding outcomes should be maintained throughout the learning curve. Future studies and efforts need to be centered on identifying variables that affect change along the learning curve, and potential interventions (i.e. coaching, observation of surgeons with superior skills) to minimize poor outcomes for surgeons early in the learning curve.