Evaluating Comparative Effectiveness with Observational Data: Endoscopic Ultrasound and Survival in Pancreatic Cancer

Abstract

Background

A previous observational study reported that endoscopic ultrasound (EUS) is associated with improved survival in older patients with pancreatic cancer. Our objective was to reevaluate this association using different statistical methods to control for confounding and selection bias.

Methods

We used Surveillance, Epidemiology, and End Results (SEER)-Medicare linked data (1992-2007) to identify patients with locoregional pancreatic cancer. We compared two-year survival in patients who did and did not receive EUS using standard Cox proportional hazards models, propensity score methodology, and instrumental variable analysis.

Results

EUS was associated with improved survival in both unadjusted (HR 0.67, 95% CI 0.63-0.72) and standard regression analyses (HR 0.78, 95% CI 0.73-0.84) controlling for age, sex, race, marital status, tumor stage, SEER region, Charlson comorbidity, year of diagnosis, education, preoperative biliary stenting, chemotherapy, radiation, and pancreatic resection. Propensity score adjustment, matching, and stratification did not attenuate this survival benefit. In an instrumental variable analysis, the survival benefit was no longer observed (HR 1.00, 95% CI 0.73-1.36).

Conclusions

Our results demonstrate the need to exercise caution in using administrative data to infer causal mortality benefits with diagnostic and/or treatment interventions in cancer research.

Introduction

Increasingly, investigators are using the Surveillance, Epidemiology, and End Results (SEER) tumor registry and linked Medicare claims data to evaluate the comparative effectiveness of treatment options for cancer patients.¹ With proliferation of these analyses, investigators and policy makers must understand the limitations of using these data to assess outcomes in cancer research. With observational studies, cancer patients are not randomly allocated to treatment groups, leading to potential bias. For example, healthier patients might be more likely to receive aggressive therapy, especially where extensive operations or toxic chemotherapy are involved. Different factors can bias treatment effect estimates in either direction, and has led investigators to recommend caution in using administrative data to assess outcomes of cancer treatment.²

For patients with pancreatic cancer, accurate staging predicts survival and guides management. However, the role of EUS in the evaluation of patients with suspected pancreatic cancer remains controversial. Conventional tests for the diagnosis and staging include computed tomography (CT) or magnetic resonance imaging (MRI). The use of EUS as a diagnostic tool has increased over time from 3% of patients with resected locoregional pancreatic cancer in 1992-1995 to 17% in 2004-2007.³ EUS is invasive, expensive, requires procedural sedation, and can be operator-dependent. Potential complications include hemorrhage, infection, pancreatitis, and bile peritonitis, and can range from 0-10%.⁴ However, EUS allows for tissue diagnosis whereas cross-sectional imaging does not⁵, and may have increased sensitivity as a diagnostic^6-8 and staging^9-11 modality. Ngamruengphong et al.¹² used SEER-Medicare data (1992-2004) to assess the impact of EUS on survival in patients with pancreatic cancer. After controlling for measurable confounding factors, EUS was associated with a 30% relative decrease in mortality.¹²

We hypothesized that these findings were due to selection bias and after controlling for potential unmeasured confounding, any observed survival benefit would be attenuated. We reanalyzed SEER-Medicare data (1992-2007), comparing overall survival in older patients with locoregional pancreatic cancer who received EUS to those who did not. We used standard multivariate regression models, propensity score methods, and instrumental variable analysis.

Methods

Our Institutional Review Board determined the study to be exempt from review.

Data Source

The National Cancer Institute's SEER database is a population-based registry of incident cancers in the U.S., including data on patient/tumor characteristics, treatment, and survival.¹³ We used this database linked with inpatient and outpatient Medicare claims collected by the Centers for Medicare and Medicaid Services.

Cohort Selection

We included patients aged ≥ 66 years with histologically confirmed, locoregional pancreatic adenocarcinoma from 1992-2007. Only those with pancreatic adenocarcinoma as their first primary cancer diagnosis were included. International Classification of Diseases for Oncology, 3^rd Edition (ICD-9-CM) codes for pancreatic adenocarcinoma are shown in Table 1. We included patients enrolled in Medicare Parts A and B for 6 months before and after diagnosis, or until death. We excluded patients diagnosed at autopsy or death. Patients were followed for two years after the date of diagnosis.

Table 1. ICD-9-CM and CPT Codes.

	ICD-9-CM Codes	CPT codes
Diagnosis
Pancreatic adenocarcinoma	8000/3, 8010/3, 8020/3, 8021/3, 8022/3, 8140/3, 8141/3, 8211/3, 8230/3, 8500/3, 8521/3, 8050/3, 8260/3, 8441/3, 8450/3, 8453/3, 8470/3, 8471/3, 8472/3, 8473/3, 8480/3, 8481/3, 8503*	NA
Procedures/Treatment
EUS	NA	76975, 43231, 43232, 43259
Pancreatic head resection	52.6, 52.7, 52.51	48150, 48152, 48153, 48154, 48155
Biliary stent	51.86, 51.87, 51.99	43267, 43268, 43269
Chemotherapy	99.25	Q0083, Q0084, Q0085, J7150, J2353, J2354, J9000-J9999
Radiation	99.21-99.29	77520, 77523, G0256, G0261, 77401-77499
Surgeon Visits
Outpatient	NA	99201-99205, 99211-99215, 99241-99245
Inpatient	NA	99221-99223, 99231-99236, 99238, 99251-99255

^*International Classification of Diseases for Oncology, 3^rd Edition codes

ICD-9-CM: International Classification of Diseases, Ninth Revision Clinical Modification, CPT: Current Procedure Terminology

EUS and Survival

EUS was identified using Current Procedure Terminology (CPT) codes in Medicare carrier files and outpatient standard analytic files (Table 1). Our outcome of interest was overall survival at two years from the date of diagnosis.

Patient Covariates

Covariates included age, sex, race, marital status, education quartile, SEER historic stage, SEER region, Charlson comorbidity index, diagnosis year, tumor stage, pancreatic resection, chemotherapy, radiation, and endostent placement. Education quartiles were determined by the percentage of patients in the patient's zip code area with at least a 12^th grade education. ICD-9-CM procedure codes and CPT codes were used to identify treatment variables (Table 1).

Descriptive Analysis

We calculated summary characteristics for the overall cohort and for patients who did or did not undergo EUS. Chi square and t-tests were used to compare characteristics of categorical variables and continuous variables, respectively, between EUS groups.

From the date of diagnosis, Kaplan-Meier analyses were used to determine overall survival at two years in patients who received EUS compared to those who did not, for the overall cohort and stratified by resection status. Log-rank test was used to test the statistical significance of differences in survival. We also performed Kaplan-Meier analysis of cancer-specific survival (censoring patients who died of other causes) and non-cancer survival (censored if died of cancer).

Standard Cox Proportional Hazards Regression Models

Cox proportional hazards models were used to compare two-year cancer-specific and non-cancer survival between patients who did and did not receive EUS for the overall cohort and stratified by resection status. Covariates in this model were listed previously (Patient Covariates).

Propensity Score Methods

A propensity score is the conditional probability that an individual would receive a certain intervention based on determined covariates.¹⁴ Propensity scores were generated for each patient from a logistic regression model with receipt of EUS as the dependent variable. Independent variables in the propensity score model included the same variables as in the conventional Cox model, except resection. For resected patients, only chemotherapy and radiation administered prior to surgery (neoadjuvant) were included in the propensity score calculation; for unresected patients, receipt of any chemotherapy or radiation was included. We used propensity-based matching, propensity score risk-adjustment, and propensity score stratification.

Propensity-based Matching

Patients who received an EUS were matched based on propensity score to patients who did not receive an EUS by 1:1, 1:2, and 1:3 matching using the Greedy matching technique.¹⁵ Figure 1 shows the overlap in propensity scores for patients who did and did not receive EUS; only overlapping patients were included for analysis. Table 2 shows the balance of measured patient characteristics for the 1:1 matched-pair sample. Cox proportional hazards regression models compared adjusted mortality rates between patients who did or did not receive EUS for the matched-pair samples. Models adjusted for variables that were not balanced by the propensity matching process (resection, chemotherapy, radiation).

Figure 1.

Overlap of propensity scores between EUS groups. The median propensity scores for EUS and non-EUS matched groups were 0.16 (min, max 0.01-0.56) and 0.09 (min, max 0.0001-0.56).

Table 2. Selected Characteristics and Treatments, Endoscopic Ultrasound (EUS)^**.

	Overall Cohort			Propensity Score 1:1 Matched Cohort			Adjusted by PS quintile
Factor	EUS (N=1,185) %	No EUS (N=9,320) %	p-value	EUS (N=1,138)	No EUS (N=1,138)	p-value	p-value*
Age (Mean, SD), year	75.4 ± 6.1	77.4 ± 7.2	<0.0001	75.3 (6.2)	75.4 (6.2)	0.6968	0.5699
Sex			0.0574				0.9741
Female	55.8	58.7		55.6	56.6	0.6422
Race			<.0001			0.9325	0.9692
White	87.2	81.5		87.2	88.6
Black	7.1	9.6		6.9	7.2
Other	5.7	8.9		5.9	6.2
Marital Status			<.0001			0.9392	0.9081
Single	13.9	14.3		13.8	13.4
Married	60.8	51.6		60.7	61.4
Widowed	25.4	34.1		25.5	25.1
Stage			<.0001			0.1289	0.5511
Localized	17.3	27.7		17.0	19.5
Regional	82.7	72.4		83.0	80.5
SEER Region			<.0001			0.9464	0.9952
Atlanta	1.4	4.1		1.3	1.5
Connecticut	10.0	8.6		9.8	8.9
Detroit	16.7	10.9		16.5	15.6
Greater California	14.1	12.1		14.5	14.7
Hawaii	0.5	2.5		0.5	0.6
Iowa	6.1	8.2		6.3	6.3
Kentucky	7.0	5.2		6.5	6.8
Los Angeles	10.3	9.1		10.5	11.8
Louisiana	3.2	5.7		3.3	2.7
New Jersey	15.9	10.2		15.8	17.8
New Mexico	1.4	3.2		1.4	1.8
San Francisco	2.3	5.7		2.3	1.9
San Jose	0.8	3.6		0.9	0.8
Seattle	8.0	7.6		7.9	7.2
Utah	2.3	3.2		2.4	1.7
Charlson Comorbidity			0.1034			0.6889	0.9840
0	47.9	48.3		47.8	47.0
1	31.0	28.6		31.0	30.0
2	12.6	12.6		12.7	14.4
3	8.5	10.5		8.4	8.6
Education			<.0001			0.8788	0.9786
Quartile 1	20.0	25.6		20.1	19.8
Quartile 2	23.9	25.2		23.9	22.9
Quartile 3	28.1	24.6		28.2	29.7
Quartile 4	28.0	24.6		27.9	27.6
Diagnosis Year			<.0001			0.6511	0.8227
1992-1995	2.1	16.9		2.2	1.6
1996-1999	10.3	15.6		10.2	9.5
2000-2003	46.4	31.3		46.1	47.5
2004+	41.2	36.2		41.5	41.5
Chemotherapy (PS model)			<.0001			0.4916	0.3424
Yes	39.83	28.57		39.6	38.2
Radiation (PS model)			<.0001			1.000	0.3338
Yes	32.82	25.02		32.4	32.4
Biliary Stent			<.0001			0.4745	0.1507
Yes	54.8	36.3		54.9	53.4
Surgery			<.0001			0.0006	<.0001
Yes	32.2	20.6		32.6	26.0
Chemotherapy			<.0001			0.0053	<.0001
Yes	59.24	39.89		59.3	53.5
Radiation			<.0001			0.0647	0.0001
Yes	49.37	34.43		49.3	45.4
HSA EUS rate (Mean, SD), %	15.8 (8.25)	10.7 (6.78)	<.0001	15.8 (8.30)	12.7 (6.38)	<.0001	<.0001

PS=propensity score, HSA= Health Services Area

^*Cochran-Mantel_Haensel statistics for stratified analysis. Age and HSA EUS rate were categorized for the analysis

^**Age, Sex, Race, Marital status, stage, SEER region, Charlson comorbidity, Education, Diagnosis year, chemotherapy, radiation, Endostent were used in the logistic regression model to create the propensity score on EUS

Propensity Score Risk-Adjustment

Propensity score adjustment balanced pre-treatment differences between EUS and no-EUS groups (Table 2). A Cox proportional hazards model including the patient propensity score in addition to variables in the standard regression models was used to compare mortality rates in patients who did and did not receive EUS.

Propensity Score Stratification

Patients were stratified into quintiles on the basis of their propensity scores. Cox proportional hazards models comparing mortality rates between EUS groups were performed within each quintile. These models included the same variables in the standard regression models.

Instrumental Variable Analysis

Instrumental variable (IV) analysis was used to adjust for unmeasured confounding. IV analyses require the identification of a variable that is predictive of treatment choice but does not affect outcomes. Thoughtful consideration should be given to selection of an appropriate IV, or significant errors in estimation may occur. An IV must be strongly associated with the exposure of interest, and this association must not be confounded by other variables. Finally, the IV should have no direct effect on the outcome.¹⁶

Percent EUS use in the patient's Health Services Area (HSA), defined as the percent of patients with locoregional pancreatic cancer in the HSA receiving an EUS, was selected as our IV. An HSA is a county or collection of counties that are clustered around a hospital.¹³ HSA-level utilization has been used previously as an effective IV in cases where the use of an intervention of interest varies by hospital.¹⁷ In our study, the percent EUS use per HSA was chosen because this should have no effect on the outcome, overall survival at two years. We identified 153 HSAs; we included HSAs with more than 5 patients to obtain stable estimates of EUS use, leaving 134 HSAs.

Percent EUS use was a continuous measure. We performed a partial F-test on the null hypothesis that there was a zero coefficient for the effect of HSA rate in the first-stage regression model. An F statistic >10 suggests that the instrument is not weak. The null hypothesis was rejected at p<0.0001, with an F statistic of 270. To determine the degree to which the instrument was successful in balancing prognostic factors, we compared patient and treatment characteristics across quintiles of the instrument (Table 3).

Table 3. Instrumental Variable Analysis. Selected Characteristics and Treatments across Quintiles of Health Service Area EUS Rate.

	Quintile of Health Service Area EUS Rate, %
Factor	Q1 (0-4.0) N=1893	Q2 (4.0-10.6) N=2129	Q3 (10.6-13.2) N=2019	Q4 (13.2-16.3) N=2107	Q5 (16.3-60.0) N=2301
Age (Mean, SD)	77.4 ± 7.3	77.1 ± 7.2	77.2 ± 7.0	77.4 ± 7.2	76.7 ± 7.0
Sex
Female	57.26	59.18	56.81	59.90	58.63
Race
White	72.48	83.47	90.00	80.11	83.66
Black	9.93	8.92	5.99	8.12	13.27
Other	17.59	7.61	4.01	11.77	3.07
Marital Status
Single	13.60	14.25	13.65	15.61	14.24
Married	52.09	50.56	54.85	52.45	53.01
Widowed	34.31	35.19	31.51	31.94	32.75
Stage
Localized	28.00	29.59	23.48	24.63	26.76
Regional	72.00	70.41	76.52	75.37	73.24
SEER Region
Atlanta	16.16	3.80	0	0.33	0
Connecticut	0	3.19	26.45	15.19	0
Detroit	0	0	0	0	52.66
Greater California	10.72	18.93	8.52	11.11	11.80
Hawaii	12.84	0	0	0	0
Iowa	9.61	18.18	0	8.26	3.46
Kentucky	5.44	7.00	1.14	1.71	10.29
Los Angeles	0	0	0	46.13	0
Louisiana	11.67	13.10	1.24	1.57	0.35
New Jersey	0	0	23.63	14.33	15.69
New Mexico	8.03	6.53	0	1.04	0
San Francisco	2.22	24.19	0	0	0
San Jose	16.32	1.6	0	0	0
Seattle	2.17	2.07	29.27	0	5.49
Utah	4.81	1.41	9.76	0	0.26
Charlson Comorbidity
0	49.23	49.55	50.72	47.41	44.92
1	29.42	27.20	28.43	28.33	30.57
2	12.15	12.68	11.49	13.00	13.62
>=3	9.19	10.57	9.36	11.20	10.89
Education
Quartile 1	30.22	24.51	15.40	30.02	24.53
Quartile 2	25.56	24.22	21.26	22.51	30.40
Quartile 3	22.69	26.25	29.39	22.61	24.45
Quartile 4	21.53	25.02	33.94	24.86	20.63
Diagnosis Year
1992-1995	16.80	14.84	12.83	16.71	14.96
1996-1999	17.75	13.10	14.02	14.57	15.82
2000-2003	31.11	32.41	34.62	32.65	34.11
2004+	34.34	39.64	38.53	36.07	35.11
Neoadjuvant Therapy
Yes	1.06	1.03	0.94	1.28	1.30
Biliary Stent
Yes	34.13	37.86	41.70	39.39	38.61
Surgery
Yes	20.71	21.00	21.59	23.07	23.30
Chemotherapy
Yes	40.25	36.64	45.27	45.23	43.23
Radiation
Yes	35.55	31.00	38.58	34.69	40.64
Treatment
Resection only	8.03	9.11	8.57	8.92	8.56
Resection+ Adj-Chemotherapy/Radiation	12.68	11.88	13.03	14.14	14.74
Unresected + Ch emotherapy/Radiation	33.60	29.92	37.54	35.45	35.71
Unresected, no Chemotherapy/Radiation	45.69	49.08	40.86	41.48	40.99

Two different estimation methods were employed for the IV analyses. First, we estimated the parameters using a two-stage residual inclusion model, estimating the effect of EUS on mortality at one time point—2 years after diagnosis. The first-stage equation was a logistic regression model predicting EUS as a function of the IV and age, sex, race, marital status, tumor stage, SEER region, year of diagnosis, Charlson comorbidity index, education, and preoperative biliary stenting. From this model, we estimated each patient's probability of receiving EUS and then calculated the residual of EUS prediction. The second-stage equation was a Cox proportional hazards model that included EUS, the residual of EUS prediction, surgery, chemotherapy, radiation, and the same variables in the standard regression models. The second method was an exogenous probit analysis with mortality as the outcome. Patient record data were converted to person-month data, in which each patient was coded as alive or dead for each month of observation. The total number of records for each patient was equal to the number of months of follow-up. For the IV analysis, the probit model was used to analyze person-month data using a previously developed estimation method.¹⁸

Results

Patient Demographics and Unadjusted Analysis (Table 2)

We identified 10,505 patients with locoregional pancreatic adenocarcinoma. EUS was obtained in 11.3% of patients. Patients who underwent EUS were more likely to be younger, white, married, have regional disease, and receive neoadjuvant therapy, endoscopic biliary stent placement, resection, and adjuvant chemotherapy/radiation compared to patients who did not.

Unadjusted Survival and Standard Cox Proportional Hazards Models

Of the 10,505 patients, 9,842 died in the 2-year follow-up period; 8,644 died of pancreatic cancer and 1,198 died of other causes. Receipt of EUS was associated with improved survival at two years compared to patients who did not receive EUS (21.7% vs. 12.8%, p<0.0001; Figure 2A). For both resected (Figure 2B) and unresected patients (Figure 2C), survival was improved with receipt of EUS. Both cancer-specific and non-cancer survival were improved with receipt of EUS (Figures 3A and 3B).

Figure 2.

A. Kaplan-Meier analysis of two-year survival for patients with locoregional pancreatic adenocarcinoma, stratified by receipt of EUS. EUS group had improved survival (21.7% vs. 12.8%, p<0.0001).

B. Kaplan-Meier analysis of two year survival for patients with locoregional pancreatic adenocarcinoma and EUS, in patients who underwent surgery. EUS group had improved survival (42.5% vs. 33.4%, p<0.0001).

C. Kaplan-Meier analysis of two year survival for patients with locoregional pancreatic adenocarcinoma and EUS, in patients who did not undergo surgery. EUS group had improved survival (8.6% vs. 5.6%, p<0.0001)

Figure 3.

A. Kaplan-Meier analysis of cancer-specific survival at two years for patients with locoregional pancreatic adenocarcinoma and EUS. EUS group had improved survival (25.4% vs. 16.4%, p<.0001).

B. Kaplan-Meier analysis of non-cancer survival at two years for patients with locoregional pancreatic adenocarcinoma and EUS. EUS group had improved survival (85.3% vs. 78.4%, p<.0001)

In the Cox proportional hazards model, EUS was associated with improved two-year survival (HR 0.78, 95% CI 0.73-0.84, Table 4). Findings were similar when models were stratified by resection status. EUS was associated with improved survival in the resected (HR 0.74, 95% CI 0.64-0.86) and unresected groups (HR 0.76, 95% CI 0.70-0.83).

Table 4. Cox Proportional Hazard Models, Endoscopic Ultrasound and Two-Year Survival.

Model	Mortality Difference (SE)	Hazard Ratio (95% CI)
Unadjusted Cox Model	-0.401 (0.035)	0.67 (0.63-0.72)
Multivariate Cox Model*	-0.247 (0.037)	0.78 (0.73-0.84)
1:1 PS Matching	-0.263 (0.047)	0.77 (0.70-0.84)
1:2 PS Matching	-0.244 (0.041)	0.78 (0.72-0.85)
1:3 PS Matching	-0.249 (0.040)	0.78 (0.72-0.84)
PS Quintile
Q1	0.013 (0.193)	1.01 (0.69-1.48)
Q2	-0.351 (0.108)	0.70 (0.57-0.87)
Q3	-0.184 (0.083)	0.83 (0.71-0.98)
Q4	-0.292 (0.071)	0.75 (0.65-0.86)
Q5	-0.295 (0.061)	0.74 (0.66-0.84)
PS Adjustment**	-0.237 (0.037)	0.79 (0.74-0.85)
IV Two-stage Residual Inclusion Estimation Analysis†	-0.003 (0.158)	1.00 (0.73-1.36)
IV Probit Analysis‡	-0.051 (0.112)	0.91 (0.62-1.35)

PS=propensity score, IV=instrumental variable, Q=quintile

^*Multivariate Model: Controlled for age, sex, race, marital status, stage, Charlson comorbidity, education, diagnosis year, SEER region, endostent, surgery, chemotherapy, radiation, EUS.

^**Propensity Score Adjustment: Controlled for propensity score, age, sex, race, marital status, stage, Charlson comorbidity, education, diagnosis year, SEER region, endostent, surgery, chemotherapy, radiation, EUS.

^†Instrumental Variable Two-stage Residual Inclusion Estimation: Controlled for age, sex, race, marital status, stage, Charlson comorbidity, education, diagnosis year, SEER region, neo-adjuvant therapy, endostent, surgery, chemotherapy, radiation, EUS.

^‡Instrumental Variable Probit: Controlled for age, sex, race, marital status, stage, Charlson comorbidity, education, diagnosis year, SEER region, neo-adjuvant therapy, endostent, EUS.

Propensity Score Analyses

Table 2 shows cohort characteristics after 1:1 propensity score matching. Propensity-based matching to non-EUS patients was possible for 1,138 of EUS patients (96.0%). The EUS and no EUS groups were well-matched with regard to patient and tumor characteristics, but EUS patients were more likely to receive surgery, chemotherapy, and radiation. Of note, the IV was not balanced in the propensity-matched cohort, suggesting residual unmeasured confounding.

Propensity score methods produced similar survival benefits when compared to standard Cox regression models (Table 4). EUS was associated with a 20-25% relative decrease in mortality, using 1:1, 1:2, and 1:3 propensity-based matching or propensity score risk-adjustment.

Instrumental Variable Analysis

EUS rates ranged from 0% to 60.0% across the 134 HSAs (median 12.0%); 29 HSAs had an EUS rate of 0%.

In the IV analysis using a two-stage residual inclusion estimation (Table 4), the survival benefit previously observed with EUS use was eliminated (HR 1.00, 95% CI 0.73-1.36). Similar findings were observed using the IV probit model (HR 0.91, 95% CI 0.62-1.35).

Discussion

Our study demonstrates the difficulty in determining causal inference in observational studies of cancer outcomes. In evaluating the association between EUS and survival, risk-adjustment using standard regression and propensity score methods appears to be subject to unmeasured confounding or bias. While a previous study documented improved survival in pancreatic cancer patients receiving EUS,¹² there is little biologic plausibility for an effect of EUS on survival, especially in patients who did not receive pancreatic resection.

We suspect that patients who underwent EUS were healthier, had better functional status, and less extensive disease than patients who did not undergo EUS. Non-cancer mortality was also lower in patients who received EUS, supporting our hypothesis. These factors are not easily captured in SEER-Medicare data, resulting in unmeasured bias. EUS may also enable more accurate staging and better allocation of patients to appropriate treatment groups, which could contribute to an apparent survival improvement (stage migration). Finally, EUS may be a surrogate for increased access to resources, overall quality of care, and adequacy of resection. EUS is more readily available at academic institutions in urban areas¹⁹ which are also usually high-volume centers for pancreatic surgery. While EUS may be an important staging modality in the management of some patients with suspected pancreatic cancer, we believe that its association with improved survival is confounded by these factors.

An association between EUS and survival was observed in both standard (Cox regression models) and propensity score (matching and risk-adjustment) risk-adjustment models. Only when we used IV methods was the survival benefit eliminated. Previous studies using observational data to evaluate cancer treatments have shown that propensity score methods were unable to control for confounding related to health status and tumor prognosis. In a review of 69 studies of propensity score methods, only 13% of analyses resulted in significantly different outcomes when compared to conventional adjustment models.²⁰ Giordano et al. observed a non-cancer mortality benefit in patients with prostate cancer who were treated with active treatment compared to those who underwent observation,² and Bosco et al. illustrated that receipt of adjuvant chemotherapy was associated with a higher risk for breast cancer recurrence.²¹

An IV analysis pseudo-“randomizes” patients based on the IV, thereby addressing unmeasured confounders. Hadley et al. used both propensity scores and IV analysis to determine the effect of radical prostatectomy on prostate cancer-specific survival.²² Multivariate regression and propensity score analyses demonstrated a prostate cancer-specific survival benefit with the aggressive treatment. However, in the IV-adjusted analysis there was no association with survival, consistent with prior RCTs.

An IV analysis is meant to inform decision-making on a population level, and not necessarily at the patient level, where there may be significant heterogeneity.²³ IV analyses estimate the treatment effect on the marginal population rather than the average treatment effect. The marginal population is the population that would receive EUS in a high-use HSA but not in a low-use HSA. Our study should be interpreted as answering the population-level question, “Will increasing EUS use at the population level improve survival?” rather than an individual-level question, “Will performing EUS improve the survival of this specific patient?” The comparative effectiveness of EUS to conventional staging modalities in a specific patient is not the aim of IV analyses, and there are circumstances where EUS is necessary at an individual level, benefitting the individual patient.

Our study has several limitations. Our propensity score model demonstrates variability between the survival outcomes of patients in quintile one compared to patients in the other quintiles. This quintile had an average EUS use of 1.7%; in this group there were likely clear contraindications for EUS use. For example, such patients may have been clearly resectable on CT or clearly unresectable with major vascular involvement. In this case, the few patients who underwent EUS were likely not subject to the same unmeasured confounding. In addition, the IV quintiles were not perfectly balanced for racial characteristics, education, and SEER region, indicating that these covariates could have potentially confounded our IV. However, our models controlled for these factors. In addition, effects of any geographic SEER region variability will be controlled by our IV, rate of EUS by HSA. However, there may have been systematic variation in patient health or access to resources across geographic areas that might have influenced the effectiveness of our IV.

The use of SEER-Medicare data to evaluate treatment effects on cancer outcomes has increased dramatically, in part due to a recent emphasis by policymakers on clinical outcomes research.²⁴ As more of these studies are disseminated, it is imperative to understand and address the limitations of this type of research, particularly the considerable likelihood of confounding.²⁵ Our study, by combining multivariate logistic regression, propensity score modeling, and IV analysis, illustrated a variety of techniques to address selection bias. Moreover, implausible results, such as an association between cancer therapy and non-cancer mortality, should be considered in this context, and it may be that some questions cannot be answered with observational data.