Validation of Disability Status, a Claims-Based Measure of Functional Status for Cancer Treatment and Outcomes Studies

Abstract

Background

In prior research, we developed a claims-based prediction model for poor patient disability status (DS), a proxy measure for performance status, commonly used by oncologists to summarize patient functional status and assess ability of a patient to tolerate aggressive treatment. In this study, we implemented and validated the DS measure in 4 cohorts of cancer patients: early and advanced non-small cell lung cancers (NSCLC), stage IV estrogen-receptor (ER-) negative breast cancer, and myelodysplastic syndromes (MDS).

Data and methods

1999–2007 SEER-Medicare data for the four cohorts of cancer patients. Bivariate and multivariate logistic regression tested the association of the DS measure with designated cancer-directed treatments: early NSCLC (surgery), advanced NSCLC (chemotherapy), stage IV ER- breast cancer (chemotherapy), and MDS (erythropoiesis-stimulating agents). Treatment model fit was compared across model iterations.

Results

In both unadjusted and adjusted results, predicted poor DS was strongly associated with a lower likelihood of cancer treatment receipt in all four cohorts [early NSCLC (N=20,280), advanced NSCLC (N=31,341), stage IV ER- breast cancer (N=1,519), and MDS (N=6,058)] independent of other patient, contextual, and disease characteristics, as well as the Charlson Comorbidity Index (CCI). Inclusion of the DS measure into models already controlling for other variables did not significantly improve model fit across the cohorts.

Conclusions

The DS measure is a significant independent predictor of cancer-directed treatment. Small changes in model fit associated with both DS and the CCI suggest that unobserved factors continue to play a role in determining cancer treatments.

Introduction

Measurement and control for relevant dimensions of health status represent a key challenge in observational studies of treatment and outcomes using administrative claims data. There are a variety of comorbidity measures based on the presence of selected International Classification of Diseases, revision 9, clinical modification (ICD-9-CM) diagnosis codes.^1,2 Other dimensions of health status may be equally important, yet are difficult to measure from claims.^3,4 Performance status (PS) is commonly used in oncology practice to measure patients’ functional capacity, with an emphasis on physical dimensions.⁵ Due to its prognostic value for survival, PS is often a key factor in determining whether cancer patients are healthy enough to tolerate surgery or aggressive chemotherapy that may be recommended for a given disease and stage.^6,7

Given the clinical importance of PS in making treatment decisions for cancer patients, we previously developed a multivariate prediction model based on administrative claims to capture this dimension of health status.⁸ Data were obtained from the Medicare Current Beneficiary Survey (MCBS), a nationally representative survey of community-based and institutionalized Medicare beneficiaries linked to Medicare claims.⁹ Our dependent variable, a proxy measure for PS referred to as disability status (DS), was based on self-reported functional status information collected during the MCBS. The explanatory variables in our model were indicators for healthcare services used more or less commonly in persons with poor DS (See the online appendix for additional detail on the PS scale, the DS model and development.) Applying the results of our prediction model in the MCBS, we found that predicted poor DS was strongly associated with worse survival.

Given the initial results in development of the DS model and validation of DS within a general Medicare population, we sought to evaluate the predicted DS measure in a cancer-specific sample, and examine whether the addition of information on DS enhances the ability to explain receipt of cancer treatment. We used Surveillance, Epidemiology, and End Results (SEER) registry linked to Medicare enrollment and claims data,¹⁰ and selected four previously studied cohorts of older Medicare beneficiaries with cancer. The four study cohorts and the treatments examined were: 1) surgical resection for early stage (stage I or II) non-small cell lung cancer (NSCLC), 2) chemotherapy for advanced stage (stage IIIB with pleural effusion or stage IV) NSCLC (AdvNSCLC),¹¹ 3) chemotherapy for stage IV (metastatic) estrogen receptor negative (ER-) breast cancer,¹² and 4) erythropoiesis stimulating agent (ESA) use for myelodysplastic syndromes (MDS).¹³ These four cohorts were selected because they illustrate different disease processes and prognosis, types of treatment, and the potential role of PS in determining treatment. Early stage NSCLC is a condition for which surgical intervention has curative potential, whereas AdvNSCLC and breast cancer are not curable, though chemotherapy offers survival benefit and symptom palliation.^14,15 For each of these, the treatment may be contraindicated in a patient with poor PS. Finally, MDS are a group of hematopoietic stem cell neoplasms commonly associated with symptomatic anemia. Despite the availability of newer disease-modifying treatments, many MDS patients receive chronic ESAs as a component of clinical management. While there is no evidence that ESA use modifies survival, it is associated with improved quality of life.^16–18 While ESAs are not contraindicated for individuals with poor PS, poor PS may interfere with the ability to travel to a physician office to receive therapy.

For each disease cohort, we sought to address 3 aims. First, we examined whether patient age, socioeconomic status, and selected other characteristics were correlated with poor DS. Prior research suggests that various patient demographic and socioeconomic characteristics are associated with poor functional status.^19,20 A strong correlation between these patient characteristics and predicted DS would provide convergent validation. Second, we evaluated the ability of DS to explain whether patients received the designated cancer treatment, which would provide predictive validation. In the third aim, we sought to evaluate the predicted DS measure in a cancer-specific sample, and examine whether the addition of information on DS enhances the ability to explain receipt of cancer treatment compared to models with the Charlson Comorbidity Index (CCI) alone or in addition to DS.²¹ The CCI, commonly used in cancer-specific studies, identifies the presence of diagnoses for any of 19 conditions and creates a weighted index reflecting the contribution of those conditions to one-year non-cancer related mortality.

Methods

Study Population

Patients in each cohort were selected from the National Cancer Institute’s SEER-Medicare database. SEER comprises data from 16 regional cancer registries, and includes information on selected patient demographics and clinical characteristics, including cancer site, histology, and date of diagnosis. For Medicare beneficiaries, SEER data are linked to Medicare enrollment and claims (Parts A and B) files, which contain additional information on use of specific health-related services and therapies, as well as patients’ date of death.

Selection criteria for three of the cohorts have been described previously (see references 10–12 for more details) and are summarized for all four cohorts in Table 1. Common inclusion and exclusion criteria were designed to ensure completeness of key data elements and Medicare claims. In addition, each cohort had disease-specific exclusion criteria that had been implemented as part of the prior studies.

Table 1.

Disease site and stage specific data, cohort selection, and covariates

Study Characteristics		Early Stage NSCLC	Advanced NSCLC	Stage IV ER- Breast Cancer	MDS
Number (n)		20,280	31,341	1,519	6,058
Years of Diagnosis Included (yrs)		2001–2005	2001–2005	1999–2005	2001–2005
Years of claims data available		2000–2007	2000–2007	1998–2007	2000–2007
Study Cohort:
Inclusion Criteria	1)	Stage I or II non-small cell lung cancer (NSCLC)	Stage IIIb with pleural effusion or Stage IV	Stage IV (metastatic) estrogen receptor negative (ER-) breast cancer	Myelodysplastic syndromes (MDS)
	2)	Medicare beneficiaries ≥66 years.
	3)	At least 12 months of claims prior to diagnosis.
	4)	Continuous Medicare Part A and B enrollment during the 12 months prior to and the month of diagnosis.			Continuous Medicare Part A and B enrollment during the 12 months prior to diagnosis through death or end of study
	5)	Known date of diagnosis and date of death.
	6)	One unique lung cancer module/record on date of diagnosis, per person.	One unique lung cancer module/record on date of diagnosis, per person.	No prior history of other cancers.	ICD-O-3 histology codes: RA, RARS, RAEB, RCMD, MDS with 5q deletion, therapy-related MDS, and MDS, not otherwise specified.*
Exclusion Criteria	1)	Enrollment in health maintenance organization (HMO) during 12 months prior to and the month of diagnosis.			Enrollment in HMO during 12 months prior to or any time after diagnosis.
	2)	Cancer diagnosed on autopsy.
	3)		Any cancer-directed lung surgery 3 months before and 6 months after diagnosis.	Any claims for other cancer diagnoses.	ICD-O-3 histology codes: RAEB-t, treatment-related AML, atypical CML BCR/ABL negative, CMMoL, and/or chronic neutrophillic leukemia.
	4)				History of chronic renal failure based on diagnoses during 12 months prior to MDS diagnosis.
	5)				Receipt of dialysis during 12 months prior to diagnosis until death or censoring.
Definition of Primary Treatment		Cancer-directed lung resection received.	Chemotherapy received within 90 days of diagnosis.	Any chemotherapy received within 6 months of diagnosis.	Use of any erythropoietic-stimulating agents (ESAs).
Common Covariates		Person-specific age at diagnosis, race/ethnicity, sex, marital status, prior year state buy-in of Part B premium (indicative of receipt of Medicaid or Medicare Savings Program (MSP) participation), residence, region, and diagnosis year. Census tract level median household income, percentage of persons older than age 25 with four or more years of college, and percentage of households with difficulty speaking English.
Disease-Specific Covariates		Stage at diagnosis (stage I or stage II)	Substage at diagnosis (stage IIIB with effusion, stage IV, or advanced, substage unknown)	Tumor grade (well to moderately differentiated, poorly or undifferentiated, or unknown)	Modified French-American-British (FAB) group at diagnosis (lower risk, higher risk, or risk not specified)
		Histology (adenocarcinoma, squamous cell, or other)	Histology (adenocarcinoma, squamous cell, or other)		Receipt of blood transfusion 12 months prior to diagnosis
		Tumor grade (well to moderately differentiated, poorly or undifferentiated, or unknown)	Tumor grade (well to moderately differentiated, poorly or undifferentiated, or unknown)		Had other primary cancer within 5 years prior to MDS diagnosis

NSCLC= non-small cell lung cancer; ER- = estrogen receptor negative; RA = refractory anemia; RARS = RA with ringed sideroblasts; RCMD = refractory cytopenia with multilineage dysplasia; RAEB = RA with excess blasts; RAEB-t = RA with excess blasts in transformation; AML = acute myeloid leukemia; CML = chronic myeloid leukemia; CMMoL = chronic myelomonocytic leukemia.

Measures

The key outcome measure for each cohort was a dichotomous indicator for receipt of the designated cancer-directed treatment. Treatments were measured based on either initial cancer-related therapy reported through the SEER registry data or from ICD-9-CM procedures codes, Healthcare Common Procedure Codes (HCPCS), and/or National Drug Codes (NDC) culled from Medicare Part B claims. The treatments considered were: 1) surgical resection for early stage NSCLC, 2) chemotherapy received within 90 days of diagnosis for AdvNSCLC, 3) chemotherapy received within six months of diagnosis for breast cancer, and 4) receipt of ESAs any time post-MDS diagnosis.

Common socio-demographic characteristics from the SEER-Medicare database are listed in Table 1, as are cohort-specific covariates, such as disease stage or risk group.

We incorporated two measures to capture baseline health status. First, we applied the Deyo adaptation²² of the CCI, modified to exclude cancer diagnoses, to Medicare inpatient, outpatient, and physician claims during the 12-month period before cancer diagnosis. We categorized CCI into 4 groups (0, 1, 2, or ≥3) based on the initial distribution. Second, we generated patient-specific predictions for poor DS. The process involved constructing the healthcare service predictors used in our DS model from Medicare claims from the 12 months prior to each cancer diagnosis date linked to members of each disease cohort. We applied the estimated regression coefficients from the model to the set of constructed measures for each observation to generate a predicted probability of poor DS. The predicted DS values ranged from 0–1, with high values representing a high probability of poor DS. We converted the continuous predicted probability of poor DS into quartile ranges to create a categorical measure of poor DS. For this measure, quartile 1 was the lowest probability of poor DS (highest probability of good DS), while quartile 4 was the highest probability of poor DS. In the DS model development phase, we considered models that permitted interactions between selected variables (i.e., region, year, and Medicaid) and the specific service indicators. In this paper, we focus only on the model without interactions. In sensitivity analyses, we also examined results for predicted DS measures based on the models with interactions, but did not find meaningful differences in the results compared to those presented for the model without interactions.

Statistical Analysis

Bivariate analyses using χ² tests examined the association between the categorical measure of predicted poor DS and selected patient characteristics and receipt of primary treatment for each disease cohort. Multivariate logistic regression models were used to explain receipt of recommended cancer treatment, as well as examine how estimated associations and model fit varied based on the inclusion or exclusion of DS and/or CCI. Results for the C statistic, and treatment prediction (using a 0.5 probability as the cut-point to distinguish treated from untreated) were calculated to assess model fit. We also calculated the integrated discrimination improvement (IDI) measure, which combines information on changes in sensitivity and specificity of the predictions associated with a new explanatory variable, and is more sensitive than the C-statistic.^23,24 For each disease cohort, we estimated multiple models. Specifically, we estimated adjusted models that included either DS only (Model 1), CCI only (Model 2), or both (Model 3), with controls for a variety of patient demographic and disease-specific measures. We also estimated a model (Model 4) that included these covariates only, but neither DS nor CCI. We used Model 4 as the reference case for calculating the IDI.

Results

Patient characteristics for each study cohort are presented in Table 2. The distributions of CCI between early stage NSCLC (n = 20,280), AdvNSCLC (n = 31,341), and MDS (n = 6,058) were similar, with more than half of patients presenting with a CCI of 1 or greater in the 12 months prior to diagnosis. For breast cancer (n = 1,519), almost 64% of patients had a CCI = 0. Poor DS was predicted for 6.7%, 10.1%, 17.1%, and 11.1% of the early stage and AdvNSCLC, breast cancer, and MDS cohorts, respectively.

Table 2.

Characteristics of Medicare beneficiaries with selected cancers, by disease site and stage^*

Patient Characteristic	Early Stage NSCLC, % (n = 20,280)	Advanced NSCLC, % (n = 31,341)	Stage IV ER- Breast Cancer, % (n = 1,519)	MDS, % (n = 6,058)
Received Recommended Cancer Treatment†	62.3	34.3	32.5	64.1
Predicted probability poor DS:
25th percentile	0.70	0.97	2.46	0.70
50th percentile (median)	1.76	2.46	6.00	1.78
75th percentile	4.30	5.26	7.57	4.77
Age at Diagnosis
66–69	18.1	17.4	16.2	8.6
70–74	28.0	25.3	22.7	17.5
75–79	28.4	26.6	23.6	24.9
80–84	17.6	19.1	18.4	26.1
≥85	7.9	11.6	19.1	23.0
Race/Ethnicity
White (Non-Hispanic)	88.2	84.9	84.1	89.1
Black (Non-Hispanic)	7.1	9.2	13.5	5.8
Other (includes Other, Asian, Hispanic, and North American Native)	4.8	6.0	2.5	5.2
Sex
Male	51.8	55.2	0.0	53.5
Marital Status
Currently Married	54.6	50.5	27.3	49.3
Not Currently Married (includes Single, Separated, Divorced, and Widowed)	43.1	46.8	68.4	42.3
Unknown	2.3	2.7	4.3	8.4
Residence
Metro (includes Big Metro and Metro)	84.0	84.0	61.6	84.2
Non-Metro (includes Urban, Less Urban, and Rural)	16.0	16.1	38.5	15.8
>5% Households w/ Difficulty Speaking English, Per Census Tract	25.3	27.4	30.9	25.7
Median Household Income, Per Census Tract ($)	45,171	43,802	44,729	46,555
>25% Persons >25 Years with 4+ Years of College, Per Census Tract	41.6	38.4	38.3	45.7
Prior Year Medicaid/Medicare Savings Program (MSP)	14.0	16.2	17.8	12.3
Substage – Advanced NSCLC
Stage 3B with Effusion	–	15.1	–	–
Stage 4	–	80.6	–	–
Advanced, Substage Unknown	–	4.3	–	–
Stage – Early Stage NSCLC			–	–
I	81.1	–	–	–
II	18.9	–	–	–
Histology – Early and Advanced Stage NSCLC
Adenocarcinoma	46.4	38.1	–	–
Squamous Cell	32.6	18.0	–	–
Large Cell	4.2	4.9	–	–
Poorly Differentiated	0.1	0.0	–	–
Not Otherwise Specified (NOS)	4.1	13.9	–	–
Other	12.7	25.1	–	–
Tumor Behavior/Grade – Early and Advanced Stage NSCLC and Breast Cancer
Well Differentiated	9.5	2.0	2.1	–
Moderately Differentiated	30.1	7.8	14.4	–
Poorly Differentiated	32.4	24.0	36.1	–
Undifferentiated	2.9	2.7	2.6	–
Grade Unknown	25.2	63.6	44.8	–
Modified French-American-British (FAB) Group at Diagnosis – MDS
Lower Risk	–	–	–	34.5
9980 – Refractory Anemia (RA)	–	–	–	16.7
9982 – RA with Ringed Sideroblasts (RARS)	–	–	–	11.9
9985 – Refractory Cytopenia with Multilineage Dysplasia (RCMD)	–	–	–	4.2
9986 – MDS with 5q Deletion (5q Del)	–	–	–	1.8
Higher Risk	–	–	–	13.9
9983 – RA with Excess Blasts (RAEB)	–	–	–	13.9
Risk Not Specified	–	–	–	51.7
9987 – Therapy-Related MDS, NOS	–	–	–	1.2
9989 – MDS, NOS	–	–	–	50.5
Charlson Comorbidity Index (CCI)
0	39.5	44.9	63.9	45.7
1	33.0	29.2	21.3	27.7
2	15.4	13.8	8.3	15.0
≥3	12.2	12.1	6.5	11.6
Healthcare Use (12 Months Prior to Diagnosis)
Hospital Use	28.5	28.1	18.9	41.0
Oxygen and Related Supplies	9.2	9.4	3.4	6.2
Skilled Nursing Facility (SNF) Use	3.5	4.4	3.9	8.5
Walking Aids	4.2	4.9	5.4	6.9
Wheelchair Claims	3.5	4.8	6.4	6.0
Nursing Home Stay	4.1	5.6	7.6	9.1
Blood Transfusion (12 Months Prior to Diagnosis) – MDS	–	–	–	23.6
Had Other Primary Cancer within 5 Years Prior to Diagnosis – MDS	–	–	–	10.6
Year of Diagnosis	–	–
1999	–	–	8.0	–
2000	–	–	19.1	–
2001	19.4	18.3	15.8	17.2
2002	19.3	19.3	17.5	18.2
2003	20.4	20.9	15.6	21.0
2004	20.0	20.4	12.5	22.9
2005	20.9	21.1	11.5	20.7
Region
Midwest	14.6	16.7	17.1	19.2
Northeast	23.1	21.0	29.6	21.5
South	23.9	20.7	17.1	18.0
West	38.4	41.6	36.3	41.4

^*Cohort inclusion and exclusion criteria for each disease are detailed in Table 1, “Comparison of Studies in Medicare Patients, by Disease Type.”

^†Recommended cancer treatments were surgical resection for early stage NSCLC, chemotherapy within 90 days of diagnosis for AdvNSCLC, chemotherapy within six months of diagnosis for breast cancer, and receipt of ESAs any time post-MDS diagnosis.

For both early stage and AdvNSCLC, and use of ESAs for MDS, poor DS was associated with all patient demographic and socioeconomic characteristics examined. Results were similar for breast except that residence and region of the country were not significant. Table 3 reports the associations between each characteristic and the highest probability quartile of poor DS. Women, non-whites, patients not currently married, and patients with prior year Medicaid were more likely, while individuals with higher area income and educational attainment were less likely to be in the top quartile. Of particular interest is the finding of a non-monotonic relationship between age and DS. In all four disease groups, the proportion in the highest DS quartile was higher for age group 66–69 compared to ages 70–74, but then increased again in a monotonic fashion from ages 75–79 upward for early stage NSCLC and from ages 70–74 upward for the other three disease groups. As CCI increased, the proportion in the top quartile of poor DS also increased monotonically for advNSCLC, breast and MDS, while there was a slight deviation from the pattern for early state NSCLC.

Table 3.

Association between highest quartile of predicted DS^† and characteristics of Medicare beneficiaries with selected cancers, by disease site and stage

	Early Stage NSCLC, % (n = 20,280)		Advanced NSCLC, % (n = 31,341)		Stage IV ER- Breast Cancer, % (n = 1,519)		MDS, % (n = 6,058)
Patient Characteristic	Highest quartile predicted DS, %	p value (χ2)‡	Highest quartile predicted DS, %	p value (χ2)‡	Highest quartile predicted DS, %	p value (χ2)‡	Highest quartile predicted DS, %	p value (χ2)‡
Age at Diagnosis		<0.01		<0.01		<0.01		<0.01
66–69	26.7		24.0		24.4		23.8
70–74	24.5		22.5		16.8		19.5
75–79	22.9		23.1		20.1		20.7
80–84	25.7		26.1		27.9		24.7
≥85	28.8		34.8		38.3		37.6
Race/Ethnicity		<0.01		<0.01		<0.01		<0.01
White (Non-Hispanic)	22.5		22.4		21.8		23.5
Black (Non-Hispanic)	47.6		40.4		44.1		42.7
Other (includes Other, Asian, Hispanic, and North American Native)	38.6		37.4		29.7		43.5
Sex		<0.01		<0.01		–		<0.01
Male	24.5		18.2		0.0		19.2
Female	25.6		33.3		25.0		33.1
Marital Status		<0.01		<0.01		<0.01		<0.01
Currently Married	18.8		16.5		13.8		17.4
Not Currently Married (includes Single, Separated, Divorced, and Widowed)	32.5		34.0		29.6		36.0
Unknown	31.3		28.7		21.2		22.4
Residence		<0.01		<0.01		0.91		0.02
Metro (includes Big Metro and Metro)	24.2		24.9		24.8		25.0
Non-Metro (includes Urban, Less Urban, and Rural)	29.3		25.6		25.2		29.0
Median Household Income by Quartiles, Per Census Tract		<0.01		<0.01		<0.01		<0.01
Lowest (0 to 25%)	36.2		34.0		36.7		33.7
Second (26 to 50%)	24.9		24.9		23.7		26.6
Third (51 to 75%)	21.5		22.1		21.6		23.8
Highest (76 to 100%)	17.3		19.1		17.9		18.6
>25% Persons >25 Years with 4+ Years of College, Per Census Tract (Race-Specific)		<0.01		<0.01		<0.01		<0.01
No	29.3		27.7		28.9		29.0
Yes	18.9		20.7		18.6		21.8
>5% Households w/ Difficulty Speaking English, Per Census Tract		<0.01		<0.01		0.11		<0.01
No	23.4		22.7		24.3		23.7
Yes	29.7		31.1		26.4		31.2
Prior Year Medicaid/Medicare Savings Program (MSP)		<0.01		<0.01		<0.01		<0.01
No	18.1		16.5		13.9		19.6
Yes	67.6		69.2		76.0		68.8
Charlson Comorbidity Index (CCI)		<0.01		<0.01		<0.01		<0.01
0	26.0		21.1		19.0		19.9
1	19.7		21.5		27.2		22.2
2	25.4		29.2		38.1		32.7
≥3	35.4		43.2		60.2		47.7
Region		<0.01		<0.01		0.54		0.04
Midwest	21.4		21.8		24.3		24.0
Northeast	19.4		20.7		23.6		23.9
South	29.3		27.1		30.0		26.5
West	27.1		27.4		24.0		27.0

^†Percentages reported (row %) are the proportion of highest quartile of predicted DS per category.

^‡p values calculated by χ² test.

Table 4 reports receipt of recommended cancer treatment for each cohort overall and stratified by predicted DS, by disease type. Treatment rates were highest for early stage NSCLC (62.3%) and MDS (64.1%), while substantially lower for the two groups with metastatic disease (34.3% for AdvNSCLC and 32.5% for stage IV breast cancer). Across all 4 diseases, there was a strong association between DS quartile and treatment, with those in the highest poor DS quartile between 24 and 36 percentage points less likely to receive treatment compared with beneficiaries in the lowest quartile. For example, in early stage NSCLC, 77.3% of those in the lowest quartile received surgery compared to only 41.4% in the highest quartile.

Table 4.

Association between receipt of cancer-directed treatment and quartiles of predicted DS, by disease site and stage ^†, ^‡

	Early Stage NSCLC, % (n = 20,280)*	Advanced NSCLC, % (n = 31,341)*	Stage IV ER- Breast Cancer, % (n = 1,519)*	MDS, % (n = 6,058)*
Disability Status (DS)	Received Surgical Resection	Received Chemotherapy within 90 Days of Diagnosis	Received Chemotherapy within 6 Months of Diagnosis	Received ESAs Any Time Post Diagnosis
Overall, %	62.3	34.3	32.5	64.1
Predicted probability poor DS:
Q1 (lowest probability poor DS)	77.3	49.1	43.1	73.7
Q2	70.7	40.0	34.7	71.0
Q3	60.2	30.2	33.0	66.7
Q4 (highest probability poor DS)	41.4	18.4	19.3	45.5

^†Percentages reported (row %) are the proportion who received treatment by DS.

^‡All comparisons were statistically significant at an alpha level of 0.05 (calculated by χ² test).

^*Model without interactions, elderly (non-disabled) using 0.11 cut-off for early stage and advanced NSCLC, breast cancer, and MDS.

Table 5 presents the results of multivariate logistic regression models, by disease, for the association between predicted poor DS and receipt of recommended cancer-directed treatment. We report only the results associated with DS and CCI, the model fit statistics, including the IDI, and the treatment model predictive ability. The full model results are presented for each disease cohort in Appendix Tables 2–5.

Table 5.

Association between predicted poor DS (quartiles) and primary treatment receipt, by disease type

	Covariate Adjusted Models*
Model #	1			2			3			4
Early Stage NSCLC (Stage I and II), % (n = 20,280)
Dependent Variable = Surgical Resection
	Odds Ratio	95% CI		Odds Ratio	95% CI		Odds Ratio	95% CI		Odds Ratio	95% CI
Independent Variables
Predicted Probability Poor DS:
Q1 (lowest probability poor DS)	REF						REF
Q2	0.74‡	0.66	0.82				0.74‡	0.66	0.82
Q3	0.49‡	0.44	0.54				0.49‡	0.44	0.54
Q4 (highest probability poor DS)	0.23‡	0.20	0.25				0.23‡	0.20	0.25
Charlson Comorbidity Index (CCI)
0				REF			REF
1				1.12‡	1.03	1.22	0.98	0.90	1.07
2				0.96	0.87	1.07	0.87‡	0.78	0.97
≥3				0.73‡	0.65	0.81	0.72‡	0.64	0.81
C Statistic			0.83			0.81			0.83			0.81
IDI statistic (base model=covariates only model)			0.034			0.007			0.04			base
z-statistic			32.29			15.43			35.34			base
p- value			<0.01			<0.01			<0.01			base
Treatment Prediction (Prob = 0.50)
Sensitivity, %			88.1			88.9			88.2			89.0
Specificity, %			60.1			57.4			60.2			57.1
Positive Predictive Value, %			78.5			77.5			78.5			42.9
Negative Predictive Value, %			75.4			75.8			75.5			57.1
Advanced NSCLC, % (n = 31,341)
Dependent Variable = Chemotherapy within 90 Days Post-Diagnosis
	Odds Ratio	95% CI		Odds Ratio	95% CI		Odds Ratio	95% CI		Odds Ratio	95% CI
Independent Variables
Predicted Probability Poor DS:
Q1 (lowest probability poor DS)	REF						REF
Q2	0.74‡	0.70	0.80				0.75‡	0.70	0.80
Q3	0.45‡	0.42	0.49				0.44‡	0.41	0.47
Q4 (highest probability poor DS)	0.27‡	0.25	0.29				0.28‡	0.25	0.30
Charlson Comorbidity Index (CCI)
0				REF			REF
1				0.99	0.93	1.05	0.89‡	0.84	0.94
2				0.80‡	0.74	0.87	0.75‡	0.69	0.81
≥3				0.53‡	0.48	0.58	0.54‡	0.50	0.60
C Statistic			0.73			0.70			0.73			0.70
IDI statistic (base model=covariates only model)			0.032			0.002			0.033			base
z-statistic			24.58			6.2			25.15			base
p- value			<0.01			<0.01			<0.01			base
Treatment Prediction (Prob = 0.50)
Sensitivity, %			37.4			30.5			38.0			28.7
Specificity, %			86.3			87.6			86.3			88.2
Positive Predictive Value, %			58.7			56.1			59.0			55.8
Negative Predictive Value, %			72.6			70.7			72.7			70.3
Stage IV ER- Breast Cancer, % (n = 1,519)
Dependent Variable = Chemotherapy within 6 Months Post-Diagnosis
	Odds Ratio	95% CI		Odds Ratio	95% CI		Odds Ratio	95% CI		Odds Ratio	95% CI
Independent Variables
Predicted Probability Poor DS:
Q1 (lowest probability poor DS)	REF						REF
Q2	0.76	0.55	1.07				0.74	0.53	1.04
Q3	0.58‡	0.41	0.81				0.51‡	0.36	0.72
Q4 (highest probability poor DS)	0.39‡	0.26	0.60				0.40‡	0.26	0.63
Charlson Comorbidity Index (CCI)
0				REF			REF
1				0.79	0.58	1.07	0.68‡	0.49	0.94
2				0.63	0.38	1.02	0.60‡	0.36	0.98
≥3				0.49‡	0.28	0.86	0.52‡	0.29	0.92
C Statistic			0.78			0.78			0.79			0.77
IDI statistic (base model=covariates only model)			0.013			0.007			0.021			base
z-statistic			4.25			3.23			5.54			base
p- value			<0.01			<0.01			<0.01			base
Treatment Prediction (Prob = 0.50)
Sensitivity, %			44.8			45.2			45.0			43.4
Specificity, %			85.1			86.3			85.8			85.8
Positive Predictive Value, %			59.3			61.4			60.5			59.6
Negative Predictive Value, %			76.2			76.5			76.4			75.8
MDS, % (n = 6,058)
Dependent Variable = Receipt of Any ESAs Post-MDS Diagnosis
	Odds Ratio	95% CI		Odds Ratio	95% CI		Odds Ratio	95% CI		Odds Ratio	95% CI
Independent Variables
Predicted Probability Poor DS:
Q1 (lowest probability poor DS)	REF						REF
Q2	0.86	0.73	1.02				0.87	0.74	1.02
Q3	0.73‡	0.62	0.86				0.74‡	0.63	0.87
Q4 (highest probability poor DS)	0.32‡	0.27	0.38				0.33‡	0.28	0.39
Charlson Comorbidity Index (CCI)
0				REF			REF
1				1.01	0.89	1.15	1.01	0.89	1.16
2				0.82‡	0.70	0.97	0.89	0.76	1.05
≥3				0.61‡	0.51	0.73	0.75‡	0.63	0.90
C Statistic			0.66			0.62			0.66			0.61
IDI statistic (base model=covariates only model)			0.037			0.006			0.039			base
z-statistic			14.48			5.67			14.83			base
p- value			<0.01			<0.01			<0.01			base
Treatment Prediction (Prob = 0.50)
Sensitivity, %			87.3			93.6			87.6			94.4
Specificity, %			29.7			14.4			30.1			12.6
Positive Predictive Value, %			68.9			66.1			69.1			65.8
Negative Predictive Value, %			56.7			56.0			57.7			55.9

^‡Indicates a significant odds ratios (OR) at the p < 0.05 level.

^*Covariate adjusted models included: person-specific age at diagnosis, race/ethnicity, sex, marital status, prior year Medicaid/MSP, residence, region, and diagnosis year; and census tract level median household income, percentage of persons older than age 25 with four or more years of college, and percentage of households with difficulty speaking English. Cohort specific covariates also included in the models are listed in Table 1.

Higher probability of poor DS was associated with lower odds of recommended treatment for early stage and AdvNSCLC, respectively, both with and without CCI in the model. In Model 3, the odds of receiving treatment in DS quartile 4 (i.e., highest probability of poor DS) were 77% and 72% less than the odds of treatment in DS quartile 1 for early stage and AdvNSCLC, respectively (p < 0.05). Results were similar for prediction of treatment in breast cancer and MDS, although in most of the adjusted models, the difference between the two lowest DS quartiles was not significant. In Model 3, the odds of receiving treatment in DS quartile 4 were 60% and 67% less than the odds of treatment in DS quartile 1 for breast cancer and MDS, respectively (p < 0.05). In all these models, the odds of receiving treatment decreased with increasing probability of poor DS.

In the adjusted models, the C statistic and sensitivity, specificity, and positive and negative predictive values did not vary substantially by whether DS, CCI, or both were included in the models. The IDI indicates that relative to a model with just the covariates, the addition of each health status measure contributes significantly to model improvements, but the magnitudes were small. For example, in early stage NSCLC, treatment model sensitivity estimates ranged from 88.1 to 89.0, with the highest value for model 4 – the model with covariates only, while specificity was highest in the models with DS. The C statistics were 0.83, 0.81 and 0.83 for adjusted models with DS, CCI, and both, respectively. The specific patterns differed across disease cohorts, with C-statistics ranging from 0.83 in early stage NSCLC to 0.66 for MDS. The differences across diseases likely relate to differences in the underlying treatment rates and the degree to which observable characteristics can explain treatment patterns.

Conclusions

The study results provide strong support for the validity of DS as a measure that can be used in claims-based studies to add information on health status. DS was strongly correlated with several patient characteristics associated with functional status, such as age and socioeconomic status, providing convergent validation for the measure. In addition, DS was a significant predictor of cancer treatment in all four cohorts; in each cohort, a higher probability of poor DS was associated with lower probability of receiving recommended cancer therapy, providing predictive validation. DS remained significant in each model even with the addition of CCI, suggesting that it captures a different dimension of health status,²⁵ and that both can be incorporated as health status controls. This validation research builds on an initial validation, which demonstrated that predicted poor DS was associated with mortality risk within a general Medicare population, and that the association remained even with controls for CCI.⁸

There is an extensive literature that examines the prevalence of comorbidity and the association between comorbidity and receipt of cancer treatment.^26,27 The measures of health conditions in claims-based studies derive from the presence of ICD-9 diagnosis codes identified from claims, either individually or as an index, such as the CCI.^1,2,4 Increasing comorbidity burden is almost always associated with reduced probability of treatment, as we found in our study. The unique contribution of the DS measure is that it allows for the role of functional status to be examined as a predictor of treatment or in stratification of study samples.

The ultimate goal of developing the DS measure was to improve covariate control for health status, with the expectation that inclusion of DS would reduce potential confounding between cancer treatment and outcomes in claims-based comparative effectiveness research studies.⁸ As a necessary first step, we sought to demonstrate that the addition of DS would improve the ability to predict or explain cancer treatment. Hence, the positive results indicating that DS is a significant and independent predictor of cancer treatment (our second aim) are tempered by our finding that the addition of DS to the treatment models did not substantially improve model fit. Although the IDI results suggested significant improvement, the magnitude of the effect was small, and the other fit statistics also demonstrated only small movement across models. In fact, relative to the model with only demographic and socioeconomic covariates, neither the addition of DS nor CCI resulted in large changes in model fit. While we did not expect that the addition of DS would completely eliminate unexplained variation in treatment, a larger effect would have offered greater expectation of reduced confounding for downstream research examining outcomes associated with cancer treatment. Additional refinements to the DS model and measure might further improve on the predictive ability, but it is unlikely that improvements would be dramatic.

The failure of DS to improve treatment model fit suggests that other factors, such as patient-physician communication, social supports, or financial resources, are important, yet unobserved determinants of treatment.²⁶ Unless claims datasets used in outcomes research are enhanced further with survey information on these dimensions, there will be an ongoing need to develop and refine analytic strategies that address confounding by unobserved confounders. One approach is to use propensity score analysis to address confounding by observable factors that affect both the probability of specific treatments and survival^29,30 While propensity score methods have been widely used to study cancer treatment^31–34 they only control for observed factors that affect treatment choice, and cannot address bias associated with unobserved characteristics. An alternative approach is to use instrumental variable analysis^35–37 an econometric technique that is designed to remove the effects of treatment selection based on factors that cannot be observed in the data. Under the right conditions, instrumental variable estimation produces consistent estimates of the effects of treatment on outcomes, and has been used in a number of comparative effectiveness studies of cancer treatments.^38–40 However, finding an appropriate instrumental variable is often difficult, the estimates may vary depending on the instrument chosen,⁴¹ and the estimates often lack precision.^42,43 Hence, further development of the method is warranted.

The limitations of the study are principally those associated with use of claims data. Concerns have been raised about under-reporting of services provided, but validation studies have established the high reporting rates for chemotherapy treatment.⁴⁴ We anticipate that this would be the case for surgical procedures and ESA use, as well. The study examines the role of DS in four diverse cohorts of cancer patients. While it is possible that magnitude and significance of the effects of DS might differ for other cancer sites and stages, we are confident that the overall patterns would persist. Finally, this study examines the role of DS in predicting cancer treatment, but does not explicitly examine the potential role of DS in reducing confounding between cancer treatment and outcomes. This is an area of ongoing research.

DS appears to be a strong predictor of cancer treatment, independent of CCI. From that perspective, it is an important covariate to include in cancer treatment models. However, inclusion of DS did not substantially improve the fit of cancer treatment models as measured by their ability to predict who received treatment. This suggests that, even after controlling for DS, unobserved factors remain that may be important predictors of treatment and potential confounders in claims-based cancer outcomes research.