Can Physician Gestalt Predict Survival in Patients with Resectable Pancreatic Adenocarcinoma?

Abstract

Purpose

Clinician gestalt may hold unexplored information that can be capitalized upon to improve existing nomograms. The study objective was to evaluate physician ability to predict 2-year overall survival (OS) in resected pancreatic ductal adenocarcinoma (PDAC) patients based on preoperative clinical characteristics and routine CT imaging.

Methods

Ten surgeons and two radiologists were provided with a clinical vignette (including age, gender, presenting symptoms, and pre-operative CA19-9 when available) and preoperative CT scan for 20 resected PDAC patients and asked to predict the probability of each patient reaching 2-year OS. Receiver operating characteristic curves were used to assess agreement and to compare performance with an established institutional nomogram.

Results

Ten surgeons and 2 radiologists participated in this study. The area under the curve (AUC) for all physicians was 0.707 (95%CI 0.642–0.772). Attending physicians with >5 years experience performed better than physicians with <5 years of clinical experience since completion of post-graduate training (AUC=0.710, 95% CI [0.536–0.884] compared to AUC=0.662, 95% CI [0.398–0.927]). Radiologists performed better than surgeons (AUC=0.875, 95% CI [0.765–0.985] compared to AUC= 0.656, 95% CI [0.580–0.732]). All but one physician outperformed the clinical nomogram (AUC=0.604).

Conclusions

This pilot study demonstrated significant promise in the quantification of physician gestalt. While PDAC remains a difficult disease to prognosticate, physicians, particularly those with more clinical experience and radiologic expertise, are able to perform with higher accuracy than existing nomograms in predicting 2-year survival.

INTRODUCTION

Pancreatic cancer is one of the most lethal cancers, affecting more than 300,000 people annually, and is the seventh leading cause of cancer mortality worldwide[1, 2]. Even for patients who present early in their disease course and undergo resection, the median overall survival (OS) remains fewer than 2 years due to the rapid local and systemic recurrence of disease[3]. Many clinical, pathologic, and genetic biomarkers have been proposed in the prognostication of survival after resection of pancreatic adenocarcinoma (PDAC). Features such as tumor size, resection margin positivity, and presence of lymph node metastases have all been associated with poor survival[4–6]. Memorial Sloan Kettering Cancer Center (MSK) developed a nomogram to assess survival after PDAC resection, incorporating both pre-operative clinical variables and pathologic features, and demonstrated a concordance index of 0.64 for predicting OS at 1, 2, and 3 years after resection[7–9]. Genetic alterations in known cancer driver genes and serum biomarkers have also been identified as potential predictors of both post-operative recurrence patterns and survival[10–12]. However, no single feature or combination of features that would change therapeutic approach has been identified.

In this study, we sought to examine the utility of clinician gestalt as an alternative prognostic and predictive approach. Clinician gestalt represents the organization and integration of pertinent pieces of clinical information into a coherent whole within the context of an individual physician clinical experience. Previous evaluations of physician diagnostic accuracy for post-operative outcomes prediction have demonstrated the potential wealth of unexplored information within clinician gestalt that may be utilized to strengthen existing predictive treatment algorithms[13–15]. These prior studies have included assessments of plastic surgeons on their ability to predict post-operative outcomes after breast reconstruction, orthopedic oncologists on predicting patient functional status after limb salvage surgery, and emergency department and surgical residents on predicting appendicitis. While these studies have shown variations in individual predictive abilities, as a group, physicians showed considerable strength in their predictive accuracy and these estimates could potentially be used in treatment decision analysis. In this context, we sought to examine the predictive ability of physicians within the oncologic setting and to measure quantitatively clinician gestalt. The objective of this study was to evaluate the ability of physicians to predict 2-year overall survival in resected PDAC patients based only on pre-operative clinical characteristics and radiographic imaging.

METHODS

This was a prospective study conducted with 12 participating physicians at a single institution. The institutional review board approved this study. Both trainees and attending physicians of varying experience levels and specialties were included. The experience level of attending physicians were categorized by the number of years since completion of post-graduate training (<2 years, 2–5 years, 5–10 years, >10 years); participants in training were assigned to the <2 years category. Participants also provided assessment of their annual pancreatic resection case volume relative to colleagues within the same specialty at the institution (range 1–5). For example, participants who ranked their annual case volume as slightly higher than colleagues were assigned a score of 4, and participants who reported significantly higher case volume were assigned a score of 5.

Each study participant was presented with case vignettes and pre-operative cross-sectional computed tomography (CT) scans of 20 patients (Figure 1). Using a consecutive series of 134 resected PDAC patients from our institution from 2004 to 2008, the 20 patients were selected to represent a diversity of ages, clinical presentations, and survival outcomes. We limited this study to 20 cases to enable adequate sample size while ensuring optimal physician participation and completion of all included vignettes. This dataset was used as it had been constructed as part of a prior study and provided the necessary 2-year follow-up for all patients. The institutional practice for the management of pancreatic cancer patients has remained consistent between 2004 and 2008 and the present; there have also been no changes in CT protocols during this time. The following inclusion criteria were utilized for final patient selection: resection of pathologically confirmed PDAC, minimum of 2 years post-operative follow-up, and availability of high-quality pre-operative CT scans. Patients with history of neoadjuvant therapies or 90-day post-operative mortality were excluded. Clinical case vignettes were constructed based on information extracted from the electronic medical record and comprised the age, gender, presenting symptoms, and pre-operative CA19-9, when available; the vignettes did not include any reference to specific CT findings or reports. All study participants were interviewed by the same study operator (L.M.P.) in a non-clinical hospital setting, and were informed at the start of the study that the survival distribution of the presented patients may or may not be reflective of the distribution of survival outcomes within the general resected PDAC patient population and that therefore the participants should consider each patient independently, as they would in the clinic. Study participants were read each case vignette and presented with the corresponding CT scan. They were asked to assign a percentage for the probability that each patient would reach 2-year OS after surgery. Study participants were blinded to patient outcome. Additional clinical and pathological data were extracted from medical records to calculate survival predictions using the MSK post-resection PDAC nomogram[7].

Fig. 1.

Two sample patients presented in the study. The complete pre-operative CT scan was provided. A clinical vignette detailing the patient’s age, gender, presenting symptoms, and pre-operative CA19-9, when available, was also provided

Data analysis was performed using IBM SPSS version 22 and Microsoft Excel 2007. Agreement plots and receiver operating curves (ROC) were constructed and the area under the curve (AUC) was evaluated among the physicians as individuals, stratified by experience and annual self-reported PDAC case volume, and in comparison to the MSK nomogram. Pearson correlation coefficient was used to evaluate for relationships between patient variables presented in the case vignettes and physician survival predictions.

RESULTS

Study Participants

The 12 participating physicians comprised 10 surgeons, of whom 3 were surgical oncology fellows and 7 were board-certified attending physicians, as well as 2 board-certified attending radiologists. Five of the attending surgeons had more than 10 years of experience since completion of post-graduate training; one had between 5–10 years of experience, and another had fewer than 2 years. Of the 2 attending radiologists, one had between 5–10 years of experience and one had between 2–5 years of experience. Seven surgeons and both radiologists characterized their annual PDAC case volume as significantly higher than their colleagues of the same specialty within the institution.

Patients and Case Vignettes

The median age at the time of surgery of the 20 patients included in the study was 74 years old (range 45–82 years). The vast majority of patients were symptomatic at time of disease presentation (19/20), with the most common presenting symptoms weight loss (13/20), jaundice (12/20), and abdominal pain (10/20). The median pre-operative CA 19-9 level, which was available in 17 patients, was 133 units/mL (IQR 58-376 units/mL). The median OS for all patients was 14 months (range 4–78 months). Eight patients reached 2-year OS and the remaining 12 patients had OS of fewer than 2 years. Additional patient demographics and characteristics as presented to study participants are listed in Table 1.

Table 1.

Patient demographics and presenting characteristics as presented to study participants, with corresponding actual overall survival.

Case No.	Age	Gender	Presenting Symptoms	CA 19-9, units/mL	OS, months
1	77	M	Jaundice	113	73
2	78	M	Abdominal and back pain, 34 lb weight loss		9
3	63	M	Jaundice, 25 lb weight loss	255	10
4	76	M	Incidental finding	1400	11
5	76	M	Jaundice, new diabetes	557	23
6	45	F	Jaundice, abdominal pain, 5 lb weight loss	526	10
7	78	M	Abdominal pain, 10 lb weight loss	40	29
8	73	M	Anemia	31	55
9	59	M	Abdominal pain, 20 lb weight loss	376	9
10	63	M	Jaundice, 10 lb weight loss	75	27
11	79	F	Abdominal pain, 40 lb weight loss	270	13
12	63	F	Jaundice, abdominal pain, 20 lb weight loss	Not measurable	14
13	47	F	Abdominal pain	63	73
14	80	M	Jaundice, 6 lb weight loss	Not evaluated	10
15	53	M	Jaundice, 20 lb weight loss	133	10
16	72	F	Abdominal pain	Not evaluated	78
17	74	F	Jaundice, abdominal pain, 5 lb weight loss	58	53
18	70	M	Jaundice, 18 lb weight loss	954	4
19	82	M	Jaundice	Not measurable	62
20	78	M	Abdominal pain, jaundice, 4 lb weight loss	196	11

CA 19-9 indicates cancer antigen 19-9; mL, milliliter; OS, overall survival; M, male; F, female

Physician Predictions

The mean AUC over all physicians for predicting 2-year OS was 0.707 (95%CI 0.642–0.772). The 2 radiologists demonstrated a combined AUC of 0.875 (95%CI 0.765–0.985), compared with the 10 surgeons’ combined AUC of 0.656 (95%CI 0.580–0.732) (p = 0.160). When stratified by experience, the mean AUC for physicians with fewer than 5 years of experience was 0.662 (95%CI 0.398–0.927), compared with 0.710 (95%CI 0.536–0.884) for physicians with more than five years of experience (p = 0.350) (Figure 2).

Fig. 2.

The AUC of the nomogram compared to individual physicians, stratified by experience. The black bar is the AUC of the nomogram. The red bars are physicians with <5 years of experience; the green bars are physicians with >5 years experience

Comparison with Post-resection PDAC Nomogram

The MSK nomogram demonstrated an AUC of 0.604 for predicting 2-year OS among the 20 study patients. All but one physician performed with an AUC greater than the nomogram (Figure 2). An agreement plot between the physician predicted probability and actual survival outcome demonstrated that both physicians and the nomogram had similar high accuracies for patients with shorter survival; however, the predicted accuracy of both declined when assessing patients with longer actual survival (Figure 3).

Fig. 3.

Agreement plot between predicted probabilities of 2-year OS and actual 2-year OS. The black line represents the nomogram prediction. The red line presents cumulative physician predictions

DISCUSSION

PDAC remains one of the most difficult cancers to prognosticate due to its significant behavioral, and, subsequently, survival heterogeneity. The limitations and lack of discrimination of the widely used TNM staging system highlights the pressing need to create a robust patient stratification method, both for pre-treatment selection and for providing patient counseling[16, 17]. Ongoing nomogram development has focused on incorporating a broader scope of clinicopathologic, genetic, and biomarker variables to strengthen predictive accuracy; however, these nomograms can rarely achieve an AUC greater than 0.80, especially on external validation[7–9, 18, 19]. In this study, we propose the addition of clinician gestalt, with emphasis on both clinical experience and radiologic expertise, which may significantly enhance the strength of existing nomograms.

When compared to the established, externally-validated MSK nomogram, all but 2 of the 12 participating physicians achieved an AUC greater than the nomogram for prediction of 2-year OS after surgery. While the MSK nomogram incorporates many clinicopathologic factors that can only be obtained after surgery, participants in this study used only pre-operative clinical variables and CT imaging. Although our sample size was limited, the two participating radiologists both achieved an AUC >0.80 and were among the highest within this study. One potential advantage radiologists may have is from their training during which they develop advanced perception skills and focused visual search patterns, which enables them to identify and interpret imaging abnormalities rapidly and effectively[20–22]. While radiologists have less exposure to patients clinically compared to surgeons, their significant expertise and experience in interpreting cross-sectional imaging is potentially more critical in accurately prognosticating outcomes for PDAC patients. Several recent studies have identified quantitative imaging features of PDAC tumors, which characterize spatial relations between image pixels, from CT scans and positron emission tomography (PET) imaging that are prognostic of patient outcomes[23–27]. These quantitative imaging techniques (called radiomics) may represent visually observable characteristics in the images. Intratumoral heterogeneity on imaging has been linked to poor survival with radiomics [28]. In a recent study by our group, we showed that increased heterogeneity in PDAC tumors is related to more aggressive tumors [29]. It is possible that the high performance of the radiologists in this study may represent their ability to recognize these microscopic imaging characteristics on the macroscopic level.

Increased clinical experience, characterized by years in practice and annual case volume, was also associated with improved survival predictions, especially in identifying potential short-term survivors. Sun et al. found similar findings when evaluating the ability of a group of 7 plastic surgeons to predict patient outcomes after breast reconstruction[14]. When evaluated individually, those 7 surgeons demonstrated variable accuracy; however, the average consensus score constructed from combining the individual predictive scores, effectively pooling from the cumulative experience of the group, yielded the highest accuracy. In another study of 24 physicians asked to predict the probability of 25 prostate cancer patients developing bone disease within one year of biochemical recurrence of disease, the cumulative group predictive performance score improved as more physicians were added, reaching a maximum concordance index of 0.750[30]. While the group concordance index approached but never surpassed that of a comparable nomogram, this again demonstrates the significant accuracy with which physicians can predict patient outcomes, with rising accuracy as the level of experience, whether individually or cumulatively as a group, increases.

The impact of clinical experience on improved outcomes is well established, in particular for pancreaticoduodenectomies, the primary surgical treatment for PDAC. In a study of 7,229 patients, Birkmeyer et al. demonstrated a significant decrease in in-hospital mortality after pancreaticoduodenectomy with increasing hospital volume[31]. This effect persisted even when hospitals initially classified within the high-volume quartile were further stratified by volume. As additional studies emerge emphasizing the importance of clinical volume and experience for improved patient outcomes, we believe that clinician gestalt is a critical component of this relationship[32–35].

Determining patient prognosis in PDAC is important for both physicians and patients. Prognosis represents the unique reflection of an individual’s clinical, pathologic, and genetic disease behavior, and is vital in providing patient counseling and informing decisions for treatment selection. PDAC remains a difficult disease for physicians to provide an accurate prognosis for and the search continues for stronger clinical, pathologic, and genetic biomarkers. We demonstrated the potential to quantify clinician gestalt as another valuable component in predicting patient outcomes after PDAC resection. This is of particular importance in diseases, such as PDAC, where survival outliers are important to identify for potentially alternative treatment considerations. The high performance of radiologists in this study compared to surgeons also highlights the prognostic significance of imaging characteristics and their qualitative interpretation in PDAC. Further research is needed to understand the visual search patterns and diagnostic assessment processes radiologists undergo when reviewing cross-sectional images and to potentially develop targeted training to enhance these skills among surgeons and other physicians.

As the participating physicians were unable to evaluate the patients included in the vignettes in-person, they were limited in their ability to perform a comprehensive assessment of the patient. However, the clinical vignettes were constructed to be as representative as possible of the presenting clinical scenario. Because the participating physicians were hepatopancreatobiliary specialists, our results may vary from those conducted in a more diverse clinical setting.

CONCLUSION

Survival and patient outcomes remain difficult to predict accurately in pancreatic adenocarcinoma. Similar to existing statistical models and nomograms, physicians may also have the ability to provide outcomes predictions using their own clinical gestalt, performing risk stratification within the context of clinical experience. The high performance of radiologists in this study for predicting survival outcomes in PDAC patients requires further research to identify the visual search patterns and cognitive processes that contribute to their high discriminatory abilities.