Patient-Reported vs. Claims-Based Measures of Health for Modeling Life Expectancy in Men with Prostate Cancer

Abstract

Purpose:

Life expectancy has become a core consideration in prostate cancer care. While multiple prediction tools exist to support decision-making, their discriminative ability remains modest, which hamper usage and utility. We examined whether combining patient-reported and claims-based health measures into prediction models improves performance.

Methods:

Using SEER-CAHPS, we identified men ≥65 years old diagnosed with prostate cancer from 2004–2013 and extracted four types of data: demographics, cancer information, claims-based health measures, and patient-reported health measures. Next, we compared the performance of five nested competing risk regression models for other-cause mortality. Additionally, we assessed whether adding new health measures to established prediction models improved discriminative ability.

Results:

Among 3,240 cases, 246 (7.6%) died of prostate cancer while 631 (19.5%) died of other causes. The National Cancer Institute Comorbidity Index score was associated but weakly correlated with patient-reported overall health (p<0.001, r=0.21). For predicting other-cause mortality, the 10-year Area Under the Receiver Operating Characteristic Curve improved from 0.721 (demographics only) to 0.755 with cancer information and to 0.777 and 0.812 when adding claims-based and patient-reported health measures, respectively. The full model generated the highest value of 0.820. Models based on existing tools also improved in their performance with the incorporation of new data types as predictor variables (p<0.001).

Conclusion:

Prediction models for life expectancy that combine patient-reported and claims-based health measures outperform models that incorporate these measures separately. Given the modest degree of improvement, however, the implementation of life expectancy tools should balance model performance with data availability and fidelity.

Introduction

Although prostate cancer is the most common cancer and the second leading cause of cancer death in men, most patients with prostate cancer will die from another cause, particularly older men and those in poorer health.^1–4 Moreover, standard treatments like surgery or radiation regularly yield side effects that reduce quality of life.^4,5 Consequently, clinical guidelines recommend against prostate cancer screening and treatment for men expected to live less than 10 years, making life expectancy an essential consideration in prostate cancer care.⁶

Despite this guidance, contemporary practice patterns indicate that many men with limited life expectancy continue to undergo screening and treatment.^7–9 To support higher quality care, several prediction tools have been developed to estimate life expectancy for men with prostate cancer.^10–12 However, these tools have been developed in relatively limited datasets (i.e., administrative or primary data) and offer only modest discriminative ability, which may contribute to their infrequent use in clinical practice.¹³ With the widespread adoption of electronic health records and health information technology, there is now tremendous enthusiasm for “big data” to improve prediction and support more evidence-based care. Models based on structured data elements from electronic health records (e.g., diagnosis/procedure codes, vital signs, imaging, lab results) can àccurately predict hospital readmissions and mortality.¹⁴ Meanwhile, patient-generated data like patient-reported outcomes (PROs) have been shown to be independently prognostic and likely stand as the next step for data integration.¹⁵ With greater data capture, more accurate and robust predictions may become achievable.

In this context, we hypothesized that combining patient-reported and claims-based health measures can improve model performance for life expectancy. Using a novel dataset, we compared nested competing risk regression models for other-cause mortality (OCM) composed of demographics, cancer data, claims-based health measures, and patient-reported health measures. Additionally, we re-assessed existing life expectancy tools for men with prostate cancer. Understanding the incremental value of different data types can help guide the development and implementation of prediction tools for clinical practice.

Materials and Methods

Data Source and Cohort Identification

We analyzed data from SEER-CAHPS, a new resource from the National Cancer Institute (NCI) that links Surveillance, Epidemiology, and End Results (SEER) cancer registry data with Medicare claims and Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys.¹⁶ SEER is a population-based cancer registry that captures incidence, treatment, and mortality data representative of the US population.¹⁷ The Medicare program provides primary health insurance for 97% of the US population aged ≥65, and successful claims linkage is achieved for over 90% of covered patients in SEER.¹⁸ CAHPS surveys patients on their care experience and includes several patient-reported health measures.

As shown in Figure 1, we identified 3,704 new prostate cancer cases diagnosed from 2004–2013 after excluding those identified at death or autopsy (n=47), who died within 1 month after diagnosis (n=11), or lacked 6-month continuous enrollment before and after diagnosis except in the case of death (n=318). Next, we restricted our sample to men aged ≥65 (n=3,444) and excluded cases with missing patient-reported health measures (n=204), creating an analytic cohort of 3,240.

Figure 1.

Study cohort flow diagram with inclusion/exclusion criteria.

Measures of Health Status

Using Medicare data, we applied the National Cancer Institute (NCI) Comorbidity Index (based on the Klabunde modification of the Charlson Comorbidity Index) to inpatient and outpatient claims in the 12 months prior to cancer diagnosis and categorized the score as 0, 1, 2, and ≥3.¹⁹ Additionally, we calculated the number of frailty indicators—claims-based measures developed to approximate performance status, functional dependence, and frailty (Supplement 1)—and categorized the count as 0, 1, and ≥2.²⁰ These indicators have demonstrated strong relationships with treatment and mortality, independent of comorbidity.^21–23

For patient-reported health measures, we used CAHPS data. CAHPS asks respondents to rate their overall health and mental health, which we separately categorized into 4 groups (i.e., excellent, very good, good, fair/poor). Next, we created a binary measure for functional deficit based on “yes” response to any of the following: 1) needing help with personal care; 2) needing help with routine; 3) having limitations due to a physical condition; 4) having deficits in daily living activities. Finally, we evaluated patient-reported smoking status dichotomized as yes/no and imputed missing responses (n=265, 8.2%) using ICD-9 code 305.1 for tobacco use disorder.

Additional Predictors

From SEER we extracted demographics, including age, race/ethnicity, and marital status. We further identified rural/urban status and US region (i.e., Northeast, South, Midwest, West). As a measure of socioeconomic status, we identified the percentage poverty by census tract (i.e., 0-<5%, 5–10%, 10–20%, 20–100%). Additionally, we collected patient-reported education level (i.e., 8^th grade or less, high school graduate or GED, some college or 2-year degree, 4-year or more college graduate) from CAHPS.

Next, we abstracted cancer information from SEER, which is drawn from both clinical and pathologic sources. We categorized prostate specific antigen (PSA) level (i.e., <10, 10–20, >20 ng/mL), American Joint Committee on Cancer staging (i.e., I/II, III, IV), and Gleason score (i.e., 0–6, 7, ≥8). In response to concerns regarding PSA in SEER, reviewers audited the PSA data for all prostate cancer cases diagnosed in 2012. Error rates were significantly lower than predicted and affected risk group assignment in only 0.8% of cases.²⁴ Furthermore, cases from 2004 to 2013 have been reviewed and corrected for the most recent data release.²⁴ For each of these additional predictors, missing values were maintained as a separate category.

Primary Outcome

To evaluate life expectancy, time to OCM served as the primary outcome. For each patient, we classified the vital status as expired from prostate cancer, expired from other causes, or alive according to SEER. Then, we measured the interval from the date of diagnosis to the date of death or until December 31, 2013.

Statistical Methods

First, we assessed the relationship between NCI comorbidity and overall health status and between frailty indicator count and patient-reported functional deficit using chi-squared testing and Pearson correlation coefficient. Then, we evaluated the association (unadjusted and adjusted) of each predictor with OCM using competing risk regression as described by Fine and Gray with OCM as the failure event and prostate cancer mortality as the competing risk.²⁵ Competing risk regression can produce risk-adjusted cumulative incidences over time and has been used by prior studies to model life expectancy.^11,12 The relative association of each predictor is expressed as a subdistribution hazard ratio (SHR) with corresponding 95% confidence interval (CI).

Next, we compared 5 nested models: 1) demographics; 2) demographics and cancer information; 3) demographics, cancer information, and claims-based health measures; 4) demographics, cancer information, and patient-reported health measures; and 5) full model. For each model, we calculated the time-dependent Area Under the Receiver Operating Characteristic Curve (AUC), Akaike Information Criterion (AIC), and pseudo R². Higher AUC and pseudo R² and lower AIC signify better performing models. The goodness-of-fit of nested models were directly compared using the Likelihood Ratio (LR) test.

We then constructed three models based on existing life expectancy tools and explored the value of adding new data types to each. These tools were: 1) the Cho model developed from Medicare claims, which incorporates age, race, and NCI comorbidity;¹⁰ 2) the Daskivich model developed in the Veterans Affairs population, which considers age, race, treatment, PSA, Gleason score, stage, and the Prostate Cancer Comorbidity Index (PCCI)—a validated, claims-based comorbidity index specific to prostate cancer predictive of OCM;^11,26 and 3) the Hoffman model developed from the Prostate Cancer Outcomes Study, which includes age, race, and patient-reported overall health.¹² For the Daskivich model, we applied the PCCI as previously described and categorized treatment as yes/no based on claims (Supplement 2). For each model, we performed competing risk regression and calculated the time-dependent AUC, AIC, and pseudo R². As the Cho/Daskivich models were developed in administrative data and the Hoffman model in patient-reported data, we added patient-reported overall health and smoking to the Cho/Daskivich models and claims-based NCI comorbidity and frailty indicator count to the Hoffman model and repeated the above assessments along with LR testing.

To confirm the robustness of our findings, we performed several sensitivity analyses. First, we examined interactions between age, NCI comorbidity, and overall health. Second, as prostate cancer mortality varies by stage, we repeated our analyses excluding patients with stage IV cancer. Third, we reassessed the Daskivich model using treatment as defined by SEER. Fourth, we repeated our analyses excluding cases with missing data.

This study received exemption from the Institutional Review Board. All statistical testing was 2-sided, completed using computerized software (SAS version 9.4, Cary, NC), and carried out at the 5% significance level.

Results

Among 3,240 cases with median follow-up of 4.3 years, 246 (7.6%) died of prostate cancer and 631 (19.5%) died of other causes. Table 1 summarizes patient characteristics. Most had an NCI comorbidity score (61.9%) or a frailty indicator count (68.6%) of 0. In contrast, 10.9% rated their health as excellent, 32.2% rated it as very good, and 75.6% reported no functional deficit. As illustrated in Figure 2, patient-reported and claims-based health measures were significantly associated (p<0.001) but weakly correlated (r=0.21 for NCI comorbidity and overall health; r=0.13 for frailty indicator count and functional deficit). A number of patients with NCI comorbidity or frailty indicator count >0 reported excellent/very good health (396/1,233, 32.1%) or no functional deficit (758/1,019, 74.4%), respectively.

Table 1.

Patient Characteristics and their Relationship with Other-Cause Mortality

Variable	Levels	%^	Unadjusted SHR (95% CI)	P-value	Adjusted SHR (95% CI)	P-value
Demographics
Age at Diagnosis	>80 yr	23.7	Reference		Reference
	65–70 yr	14.9	0.22 (0.16–0.29)	<.001	0.24 (0.18–0.35)	<.001
	71–75 yr	33.6	0.31 (0.26–0.38)	<.001	0.38 (0.30–0.46)	<.001
	76–80 yr	27.8	0.42 (0.35–0.51)	<.001	0.47 (0.39–0.59)	<.001
Married	No	36.1	Reference		Reference
	Yes	63.9	0.67 (0.59–0.80)	<.001	0.87 (0.74–1.03)	0.108
Race/Ethnicity	Non-Hispanic White	83.7	Reference		Reference
	Black	9.5	1.36 (1.07–1.74)	0.013	0.94 (0.71–1.25)	0.665
	Hispanic	2.7	0.51 (0.26–0.98)	0.043	0.43 (0.22–0.86)	0.017
	Other	4.1	0.89 (0.60–1.35)	0.603	0.55 (0.36–0.86)	0.008
Rural/Urban	Rural	13.1	Reference		Reference
	Urban	86.9	0.79 (0.64–0.98)	0.035	1.03 (0.81–1.32)	0.787
U.S. Region, 2010 Census	South	28.0	Reference		Reference
	Northeast	17.8	0.74 (0.58–0.94)	0.013	0.80 (0.61–1.04)	0.098
	Midwest	11.0	1.14 (0.58–1.56)	0.295	1.05 (0.81–1.35)	0.722
	West	43.2	0.71 (0.59–0.86)	<.001	0.82 (0.66–1.01)	0.059
Poverty Indicator	0%-<5%	25.2	Reference		Reference
	5% to <10%	25.6	1.18 (0.93–1.49)	0.166	1.04 (0.81–1.32)	0.765
	10% to <20%	29.1	1.45 (1.16–1.79)	0.001	1.17 (0.92–1.50)	0.198
	20% to 100%	19.0	1.67 (1.32–2.12)	<.001	1.14 (0.86–1.50)	0.370
	Unknown/Missing	1.1	2.17 (1.10–4.26)	0.025	1.82 (0.90–3.68)	0.095
Education	4-year or more college graduate	29.8	Reference		Reference
	High school graduate or GED	26.3	1.27 (1.01–1.59)	0.041	1.03 (0.81–1.29)	0.840
	Some college or 2-year degree	21.7	1.16 (0.91–1.48)	0.216	1.05 (0.82–1.34)	0.719
	8th grade or less, some high school	19.5	1.98 (1.59–2.46)	<.001	1.24 (0.97–1.59)	0.094
	Unknown/Missing	2.8	2.05 (1.32–3.18)	0.001	1.26 (0.80–2.00)	0.325
Cancer Information
AJCC Stage	Stage I/ II	82.8	Reference		Reference
	Stage III	4.9	0.68 (0.44–1.07)	0.094	0.92 (0.58–1.45)	0.714
	Stage IV	7.2	2.21 (1.60–3.04)	<.001	1.50 (1.05–2.15)	0.027
	Unknown/Missing	5.1	2.35 (1.73–3.18)	<.001	1.50 (1.06–2.14)	0.024
PSA	<10.0 ng/ml	54.2	Reference		Reference
	10.0–20.0 ng/ml	14.5	1.95 (1.57–2.41)	<.001	1.50 (1.20–1.88)	<.001
	>20.0 ng/ml	12.4	2.39 (1.89–3.03)	<.001	1.55 (1.19–2.02)	0.001
	Unknown/Missing	18.9	1.99 (1.63–2.44)	<.001	1.31 (1.05–1.63)	0.018
Gleason Score	0–6	35.3	Reference		Reference
	7	36.2	0.86 (0.71–1.04)	0.115	0.80 (0.66–0.97)	0.026
	8–10	21.4	1.45 (1.17–1.76)	<.001	1.09 (0.87–1.36)	0.468
	Unknown/Missing	7.1	2.38 (1.77–3.21)	<.001	1.34 (0.94–1.90)	0.106
Claims-based Health Measures
NCI Comorbidity Index Score	0	61.9	Reference		Reference
	1	22.4	1.59 (1.32–1.91)	<.001	1.37 (1.13–1.66)	0.001
	2	9.5	1.88 (1.47–2.41)	<.001	1.46 (1.13–1.90)	0.005
	3+	6.2	2.38 (1.98–3.16)	<.001	1.51 (1.12–2.04)	0.007
Frailty Indicator Count	0	68.6	Reference		Reference
	1	27.5	2.01 (1.70–2.37)	<.001	1.50 (1.26–1.79)	<.001
	2+	4.0	3.89 (2.91–5.22)	<.001	2.74 (2.01–3.75)	<.001
Patient-reported Health Measures
Smoking	No	90.6	Reference		Reference
	Yes	9.4	1.46 (1.15–1.87)	0.002	1.40 (1.08–1.81)	0.010
Mental Health	Excellent	34.3	Reference		Reference
	Very Good	33.0	1.53 (1.27–1.89)	<.001	1.16 (0.93–1.46)	0.211
	Good	24.1	2.05 (1.66–2.54)	<.001	1.33 (1.04–1.70)	0.023
	Fair/Poor	8.7	2.42 (1.84–3.18)	<.001	1.13 (0.81–1.57)	0.482
Overall Health	Excellent	10.9	Reference		Reference
	Very Good	32.2	1.44 (1.00–2.09)	0.050	1.25 (0.85–1.84)	0.250
	Good	36.4	2.11 (1.48–3.00)	<.001	1.44 (0.98–2.12)	0.066
	Fair/Poor	20.6	3.77 (2.63–5.39)	<.001	2.24 (1.48–3.40)	<.001
Functional Deficit	No	75.6	Reference		Reference
	Yes	24.4	1.91 (1.62–2.25)	<.001	1.16 (0.96–1.41)	0.112

^{^}Made not sum to 100% due to rounding.

Figure 2.

Relationship between A) patient-reported overall health and NCI comorbidity and B) patient-reported functional deficit and frailty indicator count. Chi-squared testing demonstrated significant relationships (p<0.001). The Pearson correlation coefficient was 0.21 for patient-reported overall health and NCI comorbidity and 0.13 for patient-reported functional deficit and frailty indicator count.

When accounting for all predictors, age, overall health (fair/poor vs. excellent: SHR 2.24, 95% CI 1.48–3.40), and frailty indicator count (≥2 vs. 0: SHR 2.74, 95% CI 2.01–3.75) significantly predicted OCM (Table 1). NCI comorbidity and smoking status were also independently associated with OCM (p<0.05) while functional deficit was not (p=0.112).

The addition of new types of data increased model performance (Figure 3). Model 1 (demographics) had a 10-year AUC of 0.721, which improved to 0.755 when adding cancer information (model 2). Adding claims-based health measures (model 3, AUC 0.777) and patient-reported health measures (model 4, AUC 0.812) improved the discriminative performance over model 2. Finally, the full model (model 5) had the highest AUC of 0.820. It also demonstrated the lowest AIC, highest pseudo R², and a significant LR test, indicating that it was the best performing model.

Figure 3.

Model performance for predicting other-cause mortality. Time-dependent AUC for 5 nested competing risk regression models: 1) demographics; 2) demographics and cancer information; 3) demographics, cancer information, and claims-based health measures; 4) demographics, cancer information, and patient-reported health measures; and 5) full model. Table reports the time-dependent AUC at 10 years, AIC, pseudo R², and LR test results. LR test demonstrates significant difference between models 3 vs. 2, 4 vs. 2, and 5 vs. 4 with p<0.001.

Table 2 summarizes the performance of models based on three existing life expectancy tools. The addition of patient-reported health measures to the Cho/Daskivich models improved the 10-year AUCs from 0.749 to 0.753 and 0.783 to 0.796, respectively, while the addition of claims-based health measures to the Hoffman model improved the AUC from 0.738 to 0.744. These findings did not differ substantively across the four specified sensitivity analyses.

Table 2.

Performance of models based on existing prediction tools with and without additional patient-reported or claims-based health measures

Models	Covariates	AIC&	Pseudo R2^	LR Test	DF	AUC#
Cho model	Base model: Age, Race/Ethnicity, NCI Comorbidity	9188.72	0.09	304.68	8	0.749
	Base model + Overall Health + Smoking	9115.30	0.11	386.09**	12	0.753
Daskivich model	Base Model: Age, Race/Ethnicity, Primary Treatment, PSA, Gleason Score, Cancer Stage, PCCI	9115.09	0.12	400.31	19	0.783
	Base Model + Overall Health + Smoking	9043.33	0.14	480.07**	23	0.796
Hoffman model	Base Model: Age, Race/Ethnicity, Overall Health	9184.58	0.10	310.81	9	0.738
	Base Model + NCI Comorbidity + Frailty Indicator Count	9117.34	0.11	388.06**	14	0.744

^{^}Pseudo R² = 1 - e^-(LRT/n); higher value is better.

^#AUC, time-dependent AUC at year 10 after diagnosis.

^**LR test with P-value <0.001 compared to original model.

^&AIC, smaller value is better.

Discussion

Prostate cancer remains the most common cancer in men in the United States, projected to affect 191,930 new patients and cause 33,330 deaths in 2020.¹ Though substantial, most men with prostate cancer will die from a competing cause. Thus, current guidelines recommend that screening and treatment for prostate cancer be reserved for men expected to live 10 years or longer.⁶ However, prior studies indicate that many men with limited life expectancy continue to undergo screening and treatment.^7–9 One likely cause arises from the limited ability for physicians to predict life expectancy. In a systematic review, only 25% of physician estimates were correct, with most predictions overestimating survival.²⁷

Given the need for accurate prognostication, a variety of prediction tools have been developed. The underlying models have relied upon claims-based or patient-reported health measures but not typically both. Within Medicare data, the NCI modification of the Charlson Comorbidity Index predicts OCM, enables better estimation of life expectancy than US life tables, and has been adapted into clinical tools such as the decision aid employed by the Michigan Urologic Surgery Improvement Collaborative.^10,28 More recently, Daskivich et al. developed the PCCI using chart-abstracted data from the Veterans Affairs population and validated it for administrative data.^11,26 Taking a different approach, Hoffman et. al found that patient-reported overall health predicted mortality in their prospective cohort and constructed a nomogram based on age, race, and patient-reported overall health.¹² In our dataset that contains both claims-based and patient-reported health measures, models based on these existing tools performed similarly to their original description, providing further validation of the predictors in an external, more contemporary cohort.

Additionally, our findings show that combining both claims-based and patient-reported health measures improves model performance for predicting OCM compared to either alone, albeit modestly. These findings may reflect, in part, inherent differences in these measures. For instance, while NCI comorbidity and patient-reported overall health were significantly associated, they were only modestly correlated in this study. Of note, a majority of patients with high NCI comorbidity still reported good overall health while 15.7% with no comorbidities reported fair/poor health. This phenomenon is also consistent with studies that have found PROs to be prognostic above and beyond established clinical measures in patients with cancer.¹⁵ These findings suggest that models incorporating varying measures of health may provide a more robust estimate of life expectancy, especially as more granular patient-generated data are captured and become incorporated through health information technology.

These results should be considered in the context of several limitations. First, as an observational study, the findings remain subject to potential confounding. However, the observed relationships had the expected directionality, indicating face validity. Second, the use of claims data relies largely on the accuracy of coding practices. Nevertheless, diagnosis and procedure codes remain a convention of health services research, have shown high congruence with medical records, and have demonstrated significant association with prostate cancer outcomes.²⁹ Third, the analysis considers a limited panel of PROs, and findings could vary with different or more granular patient-generated data. Fourth, the analysis focuses on discriminative ability rather than calibration, and it does not directly validate the point estimates for life expectancy produced by other models. Fifth, we were unable to compare the utility of different models, such as through decision curve analysis, due to the limited number of patients with 10-year follow-up. Sixth, findings from Medicare data may not be generalizable to younger patients. However, life expectancy is most relevant for older populations.

These limitations notwithstanding, our findings have important implications for the design and implementation of life expectancy tools in clinical practice. The majority of urologists do not use life expectancy tools.¹³ Although advances in health information technology offer new opportunities to build and integrate models into the digital workflow, the question of how to balance model performance, feasibility of data collection, and meaningfulness remains. Incorporating both patient-reported and claims-based health measures can improve model performance. However, electronic PROs collection remains in its infancy and claim-based measures may not be universally accessible in real-time. Though machine and deep learning techniques could be deployed to address these issues (e.g., derive estimates from multiple concurrent models),³⁰ practices and health systems will first need to consider the contextual and operational factors for harnessing data and generating predictions in order to best support their physicians.

Conclusion

Using both patient-reported and claims-based health measures improves model performance for predicting life expectancy for men with prostate cancer. However, the increase in discriminative ability over models using either data types alone remains modest. Design and implementation of life expectancy tools for clinical practice will need to balance model performance with feasibility and fidelity of data collection.

Abbreviations

PROs

patient-reported outcomes

OCM

other-cause mortality

SEER

Surveillance, Epidemiology, and End Results

CAHPS

Consumer Assessment of Healthcare Providers and Systems

NCI

National Cancer Institute

PSA

prostate specific antigen

SHR

subdistribution hazard ratio

CI

confidence interval

AUC

Area Under the Receiver Operating Characteristic Curve

AIC

Akaike Information Criterion (AIC)

LR

Likelihood Ratio

PCCI

Prostate Cancer Comorbidity Index