CUSUM analysis of J-pouch surgery reflects no learning curve after board certification

Abstract

Objective

To investigate changes in morbidity and mortality associated with ileal J-pouch surgery performed during the first 3 years of a single surgeon's practice to determine the presence or absence of a learning curve after fellowship training.

Methods

From July 2002 to July 2005, an observational study of postoperative outcomes was undertaken, in which 30-day and inhospital morbidity and mortality were assessed. A total of 37 patients (17 women and 20 men) underwent the surgery; their average age was 32 (range 16–51) years. The operation was performed for ulcerative colitis n = 31), familial adenomatous polyposis n = 4) and indeterminate colitis n = 2); 32 were diverted and 5 were not. Predicted morbidity and mortality were 31.66% and 1.47%, respectively. Observed morbidity and mortality were 29.7% and 0%, respectively. I used a risk-adjusted cumulative sum (CUSUM) model to compare observed outcomes with predicted outcomes according to a validated scoring system and to analyze outcomes with adjusting for risk on a case-by-case basis.

Results

CUSUM analysis revealed a flat curve trending down over the duration.

Conclusion

CUSUM methodology permits documentation of quality control during the first 3 years of practice. The experience of a single board-certified colorectal surgeon reveals acceptable results in the first 3 years of practice, with no obvious learning curve. The results suggest that fellowship training and board certification conferred reasonable proficiency in J-pouch surgery before the onset of practice.

Quality assurance is always an important aspect of clinical practice.¹ Descriptions of morbidity and mortality for a given procedure offer a crude estimate of quality. Interpretation of such data, however, is limited because these end points cannot take into account specific aspects of the patient population that may increase or decrease the risk of complications or death. Risk-adjusted analysis therefore produces more meaningful data.

The Portsmouth Physiologic and Operative Severity Score for the Enumeration of Mortality and Morbidity (P-POSSUM)^2,3 is a validated instrument developed to take such information into account in patients undergoing general surgical procedures. Parameters reviewed in this study can be found in Table 1 and Table 2.

Table 1

Table 2

Traditionally, this scoring system has been applied to predict outcomes in a specific group of patients, which have then been compared with the actual outcomes. Prospective analysis produces more meaningful information when the objective is to determine whether variance is occurring in data over time. One instrument that allows for such analysis is the plotting of cumulative sums (CUSUMs).⁴

Improved outcomes for a given procedure may occur over time (the learning curve). There may be many explanations for this, such as improved patient selection, better surgical technique or optimization of postoperative care. Subspecialty training provides focused experience within a given area of expertise in the hope that it will diminish the impact of a learning curve once surgeons begin to practise and thereby encourage optimal surgical outcomes.

This study was designed to prospectively compare risk-adjusted predicted outcomes with actual outcomes on a case-by-case basis for patients undergoing ileal J-pouch surgery. The objective was to determine the presence or absence of a learning curve for a single surgeon after board certification in colorectal surgery.

Methods

After board certification and the inception of the surgeon's practice, consecutive patients undergoing elective ileal J-pouch surgery were entered into a database between July 2002 and July 2005. The data set included patient demographics, procedure performed, indication for surgery, rates of diversions, physiological and operative severity scores, predicted morbidity and mortality and observed 30-day morbidity and mortality. Physiological scores were determined from the patient assessment and blood work carried out on the day of the operation. In regard to operative severity scores, all J-pouch operations were considered complex major procedures. Blood loss and levels of contamination were based on categorization by the operating surgeon. All procedures were elective; the “mode of surgery” for all was therefore “elective.” Malignancy was only observed in a single case of familial adenomatous poliposis.

A CUSUM analysis was then performed. For this study, CUSUM was defined as S_n = Ε (X_i–X_o) where X_i = 1 for success, X_i = 0 for failure and X_o = the predicted risk-adjusted morbidity (determined on the basis of the P-POSSUM score). For example, if the risk of a complication is determined to be 10% (0.1) and it is not observed for a given case, the deviation plotted will be –0.1. In the same example, if a complication is observed, the deviation plotted will be 0.9. Nine successes are therefore cancelled out by one failure. This means that the surgeon is less significantly penalized for patients with significant comorbidity and more meaningfully rewarded when complex patients have optimal results despite their high risk. Straight curves suggest a steady state where predicted outcomes match observed outcomes. A downward slope suggests improving outcomes over time. An upward slope suggests increasing complications exceeding predicted outcomes over time.

Results

Table 3 shows patient demographics, procedure performed, indication for surgery and the rates of diversion. Table 4 shows predicted and observed morbidity. The CUSUM plot is depicted in Figure 1. The end point of small bowel obstruction was defined as any patient who required a nasogastric tube for decompression (nasogastric tubes are not routinely used by the author). None of the patients required laparotomy for small bowel obstruction. One patient manifested high tempatures postdischarge and subsequently decompressed an abscess into his J-pouch before returning for follow-up. Clinical signs of sepsis promptly subsided. A clinical leak was confirmed on pouchogram. The patient remains diverted, and an attempt at revision is planned.

Table 3

Table 4

FIG. 1. CUSUM plot demonstrating flat curve sloping downward, suggesting morbidity is not exceeding predicted outcomes on a case-by-case basis. The shape of the curve does not demonstrate an obvious learning curve, suggesting reasonable proficiency from the beginning of practice. CUSUM = cumulative sum.

Discussion

CUSUM methodology is an effective way to examine quality assurance over time. It offers more information than the simple crude measure of morbidity and mortality. In several surgical disciplines, it has been shown to effectively track improved outcomes with increasing experience and at various levels of practice^5–7; it has also been shown to effectively assess the impact of training.⁴ In addition, the methodology has been used to “signal” trends that identify substantial changes in outcomes correlated with changes in clinical practice.^8–10 This single case series suggests acceptable outcomes in the first 3 years of surgical practice after board certification.

This study is limited by its use of P-POSSUM. Although this instrument is validated for general surgery, it has been criticized as less sensitive for colorectal surgery.¹¹ It is possible that it overestimates potential morbidity, thereby preventing the detection of acceptable results in a CUSUM plot. Continued study is ongoing to develop instruments specific to the discipline of colorectal surgery. I am not aware of any instrument specific to J-pouch surgery.

The time frame is short and the sample size is small in this study, which may have prevented the production of more meaningful data. I plan to continue to follow surgical volumes over time.

A more meaningful end point for the detection of a learning curve could be debated. It is assumed that morbidity and mortality are surrogate markers for quality of surgery; another marker of quality may be pouch survival. Such an end point may demonstrate not only the surgeon's ability to perform the surgery but also his or her ability to limit and deal with complications unique to J-pouch surgery.

Tekkis and colleagues¹² at the Cleveland Clinic have recently examined this. Defining pouch failure as pouch excision, formation of a permanent ileostomy or pouch-related mortality at any time during the follow-up period, Tekkis demonstrated an overall survival rate of 95.6% at 5 years. His work also demonstrated a learning curve with improvement in pouch failure rates after 23 cases. Because pouch failure is a time-related end point, this was not felt to be a fair assessment to use in our patient population. It will be interesting to see whether the acceptable surgical complication rates observed in this prospective analysis translate into reasonable pouch failure survival rates in the future.

Potential drawbacks to using pouch failure as a surrogate marker of quality should be considered. Not all pouch failures are related to the quality of pouch management. Patient-related factors may affect the circumstances under which the pouch is created (e.g., body habitus, short mesentery). Anal function may be modified postoperatively (e.g., injury after vaginal delivery). Crohn's disease might manifest itself after pouch creation, compromising pouch function. Inherent bias exists in that many surgeons might refer to diversion for a complication as “temporary” (thus not defining it as a pouch failure) without ever managing to restore intestinal continuity. Most important, this end point does not permit the surgeon continuous surveillance of short-term results and thus lacks practicality. Using the methodology applied in this paper might allow for earlier identification of important positive or negative changes in clinical approach that could be addressed much earlier than with an end point of pouch failure.

Last, comparative data drawn from other recently certified surgeons would lead to more meaningful conclusions regarding whether or not the experience reported here is typical of other newly certified colorectal surgeons. More study is required to determine the presence or absence of learning curves for given procedures and the number of cases required to demonstrate proficiency. Such information would have important implications for training and practice after board certification.

Competing interests: None declared.

Accepted for publication June 18, 2007

Correspondence to: Dr. P.H.D. Colquhoun, London Health Sciences Centre, University Hospital, C8.128 — 339 Windermere Rd., London ON N6A 5A5; fax 519 663-3906; patrick.colquhoun@lhsc.on.ca