Abstract
Background
The National Oncologic PET Registry (NOPR) ascertained changes in the intended management of cancer patients using questionnaire data obtained before and after PET under Medicare’s coverage with evidence development (CED) policy.
Objective
To assess the concordance between intended care plans and care received as ascertained through administrative claims data.
Research Design
Analysis of linked data of NOPR participants from 2006–2008 and their corresponding Medicare claims.
Subjects
Consenting patients age >65 years having their first PET for restaging of bladder, kidney, ovarian, pancreas, prostate, small cell lung or stomach cancer.
Measures
Agreement [positive predictive values and kappas] between NOPR post-PET intended management plans for treatment (systemic-therapy, radiotherapy, surgery or combinations), biopsy, or watching as compared to claims-inferred care 30 days after PET.
Results
8,460 patients with linked data were assessed. 43.5% had metastatic disease. 45.3% had treatment planned (predominantly systemic therapy only), 11.1% biopsy and 43.5% watching. Claims confirmed intended plans (PPV) for single-mode systemic therapy in 62.0%, radiation in 66.0%, surgery in 45.6%, and biopsy in 55.7%. 25.7% of patients with a plan of watching had treatment claims. By cancer type, kappa ranged for systemic-therapy only from 0.17–0.40 and for watching from 0.21–0.41. Agreement rates varied by cancer types but were minimally associated with patient age, performance status, comorbidity or stage.
Conclusion
Among elderly cancer patients undergoing PET for restaging, there was moderate concordance between their physicians’ planned management and claims-inferred actions within a narrow time window. When higher accuracy levels are required in future CED studies, alternative designs will be needed.
Keywords: positron emission tomography, cohort studies, Medicare, medical record linkage, health services research, neoplasm staging
INTRODUCTION
Positron emission tomography (PET) using 18F-fluorodeoxyglucose represents an imaging paradigm based on characterizing metabolic processes. Since 2001, the Centers for Medicare & Medicaid Services (CMS) has covered PET for many common cancers.(1) However, about one-fourth of beneficiaries developing cancer have a non-covered cancer type.(2) For these cancers, it was recognized that developing sufficient evidence to inform coverage decisions would be unlikely. Therefore, CMS initiated a novel mechanism under its coverage with evidence development (CED) policy.(2–5) In response to the CED requirements, the National Oncologic PET Registry (NOPR) opened with a primary objective of measuring PET’s impact on intended patient management by collecting prospective questionnaire data before and after PET.
Two criticisms of NOPR have been that changes in planned management are only a surrogate for actual health outcomes and that the dataset does not document the care actually delivered.(11, 12) Concordance studies, between recommended actions and actual care, are rare and are important for informing the design and implementation of future CED programs using health outcome surrogates. We measured confirmation of intended plans with inferred management using a linked dataset of NOPR participants’ CMS claims following PET for restaging.
PATIENTS and METHODS
NOPR is a prospective data registry (ClinicalTrials.gov #NCT00868582); its operational details, human subject protection procedures, and findings of PET’s impact on intended management were previously reported.(7–10) In brief, the PET facility collects referring physician responses on pre-PET and post-PET forms. The pre-PET form collects the testing indication, the cancer type, working stage, performance status, and the referring physician’s plan if PET were not available. After PET, the referring physician records an estimate of the patient’s stage and management plan in light of the PET findings.
Claims Linkage
We linked NOPR data from December 2006 thorough 2008 for consenting participants to their CMS claim files by matching individual identifiers for the seven most frequent cancer types (Appendix 1, supplemental digital content 1). We assessed the first PET for restaging or suspected recurrence (hereafter labeled as restaging) and excluded patients who were age <65 years; were HMO participants; or for whom we were unsuccessful in linking identifiers, the registry and claim dates for PET differed by >7 days, or the post-PET plan was “other treatment(s) or additional imaging”.
Management Categories
The post-PET categories assessed were watching, biopsy, and treatment. Treatment categories were systemic-therapy (chemotherapy, hormonal, or immunotherapy), radiotherapy, or surgery alone or in combination.
Claims Definitions
Appendix 2, supplemental digital content 2 lists the Current Procedural Terminology (CPT©) codes, the Healthcare Common Procedure Coding System (HCPCS) codes and International Classification of Diseases (ICD-9) codes for outpatient care used to classify claims into inferred-care categories.
Given the numerous cancer types assessed, a list of possible biopsy or surgery CPT codes would likely be incomplete. Instead, the claims-inferred definition of “biopsy” was primarily based on surgical-pathology or cytology CPT coding.
Our preliminary analysis found surgical procedures for complications (e.g., chest tube insertion) rather than directed at the patient’s cancer. Therefore, we used anticipated surgical CPT codes and anesthesia claims after excluding eye, central vascular access, gastrointestinal endoscopy and conscious sedation. Chemotherapy, hormonal therapy and immunotherapy were inferred from professional claims, HCPCS codes for hospital-based chemotherapy and drug J-codes. Radiotherapy codes included all common techniques. These systemic-therapy and radiotherapy definitions were similar to those used in SEER-Medicare analyses.(13–15)
The NOPR forms did not specify an action timeframe. Given the indication of restaging, we used 30 days as the default and explored extending it to 60 days.
Comorbidity
We determined the Klabunde comorbidity index(16), derived from inpatient and physician claims in the preceding 12 months, using a publically available SAS algorithm(17).
Specialties
The CMS provider part-B taxonomy codes for physician specialty designation were used to categorize referring providers. If no specialty or non-physician coding was found, then specialty was coded as “other”.
Statistical Analysis
The initial analyses treated the claims-inferred care as the reference standard to calculate measures of agreement between treatment plans and claims-inferred actual management, including positive predictive value (PPV), raw agreement, and kappa (chance-adjusted agreement).18 For the treatments (systemic-therapy, radiotherapy or surgery), agreement was defined as claims for that type action within the interval without considering other treatments; for biopsy, agreement included any procedures with surgical pathology or cytology claims; and for watching, it was the absence of treatment claims. The measures of agreement were computed separately by cancer type and compared with chi-square tests.
To assess the effect of patient, cancer, and provider factors in predicting agreement, separate logistic regression models were fit for patients with plans for systemic-therapy only, radiation only, surgery only, and watching. The outcome was the indicator of agreement with the plan. Calculations were done with PROC LOGIST in Linux SAS version 9.2.
RESULTS
Clinical characteristics
Table 1 summarizes the cancer type, age, performance status, comorbidity scores, referring specialty and pre-PET plan for our 8,460 patient cohort.
Table 1.
NOPR cohort clinical characteristics
| Cancer Type | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | Bladder | Kidney | Ovary | Pancreas | Prostate | SCL* | Stomach | |
| Patients, n | 8,460 | 1,127 | 1,070 | 2,075 | 862 | 1,974 | 720 | 632 |
| Age, mean in years | 74.1 | 75.2 | 73.7 | 73.1 | 74.1 | 75.3 | 72.8 | 74.6 |
| (25–75% range) | 69–79 | 70–80 | 39–78 | 68–77 | 69–79 | 70–80 | 68–77 | 69–79 |
| ECOG performance status | ||||||||
| Asymptomatic (0), % | 42.6 | 34.3 | 43.2 | 50.5 | 34.7 | 51.5 | 23.0 | 36.6 |
| Fully ambulatory (1), % | 46.1 | 51.5 | 43.7 | 43.0 | 50.8 | 38.8 | 59.5 | 49.6 |
| P.S. 2, 3, or 4, % | 11.3 | 14.2 | 13.0 | 6.5 | 14.5 | 9.7 | 17.5 | 13.8 |
| Post-PET summary stage (%) | ||||||||
| No evidence of disease or low probability of local recurrence | 43.0 | 42.1 | 45.8 | 36.9 | 40.3 | 48.1 | 40.1 | 51.4 |
| Local or nodal disease | 13.5 | 14.6 | 8.3 | 11.1 | 17.6 | 15.1 | 17.4 | 13.0 |
| Metastatic disease | 43.5 | 43.3 | 45.9 | 52.0 | 42.1 | 36.8 | 42.5 | 35.6 |
| Comorbidity score, % | ||||||||
| 0 | 47.9 | 37.1 | 38.5 | 66.3 | 39.2 | 52.4 | 26.7 | 44.8 |
| 1 or 2 | 38.4 | 40.1 | 37.1 | 29.0 | 49.0 | 35.6 | 56.7 | 42.2 |
| ≥3 | 13.7 | 22.8 | 24.4 | 4.7 | 11.8 | 12.0 | 16.7 | 13.0 |
| Referring M.D. specialty, % | ||||||||
| Medical oncology* | 50.6 | 60.0 | 51.0 | 42.8 | 61.9 | 37.7 | 68.8 | 63.3 |
| Radiation oncology | 8.8 | 7.4 | 5.6 | 1.9 | 6.8 | 19.2 | 11.5 | 6.2 |
| Internal medicine† | 8.0 | 7.5 | 12.7 | 5.1 | 7.9 | 9.1 | 8.5 | 7.3 |
| Urology | 6.2 | 8.9 | 10.7 | -- | -- | 15.5 | 0.0 | -- |
| Gynecology/Gynecologic Oncology | 9.0 | 0.4 | 0.5 | 35.7 | -- | -- | 0.6 | -- |
| Surgery | 3.5 | 1.3 | 4.4 | 2.3 | 8.8 | 1.8 | 1.8 | 9.3 |
| Other | 13.9 | 14.5 | 15.1 | 12.1 | 14.5 | 16.7 | 8.9 | 13.9 |
| 30 days post-PET, % | ||||||||
| Hospitalized | 11.5 | 16.0 | 12.7 | 11.5 | 13.1 | 7.5 | 11.2 | 12.5 |
| Death | 1.6 | 3.6 | 1.5 | 0.4 | 2.1 | 1.0 | 3.6 | 1.6 |
| Pre-PET Plan, % | ||||||||
| Watching | 11.7 | 10.3 | 9.8 | 10.4 | 12.8 | 12.5 | 11.7 | 13.9 |
| Additional imaging | 49.8 | 48.2 | 47.8 | 52.9 | 50.5 | 45.2 | 50.9 | 52.5 |
| Biopsy | 12.4 | 15.6 | 21.2 | 12.5 | 9.5 | 11.8 | 9.6 | 11.1 |
| Treatment | 26.1 | 25.9 | 21.2 | 24.2 | 27.2 | 30.4 | 27.9 | 22.5 |
SCL: small cell lung
Internal medicine: Sum of all internal medicine specialties excluding medical oncology and hematology
Overall, the stage distribution showed 43.0% with no evidence or low probability of recurrence (range 36.9%–51.4%), 43.5% with metastatic disease (range 35.9%–52.0%) and 13.5% with local or nodal recurrences (range 8.3%–17%).
The referrers were medical oncologists in one-half of all patients, gynecologists/gynecologic oncologists in one-third of ovarian cancer patients, and urologists in 9%–16% of bladder, kidney and prostate cancer patients. Radiation oncologists and surgeons were infrequent referrers except in prostate and small cell lung (SLC) cancer (radiation oncologists) and stomach cancer (surgeons).
In the 30 days following PET, 1.6% had died (range 0.4%–3.6%).
NOPR Plan
Table 2 summarizes the post-PET intended plans. Modest distribution variations were found by cancer type. Overall, treatment was planned in 45.3% (range 35.9%–52.8%), biopsy in 11.1%, and watching in 43.5% (range 38.7%–50.6%). For all types, the most common treatment plan was systemic-therapy only (range 20.7%–39.1%). Combination therapy, radiotherapy or surgery only were planned in <11% in all cancer types. Combination therapy components were 34.8% systemic-therapy, 12.4% radiotherapy, and 6.4% surgery.
Table 2.
NOPR post-PET intended management plan
| NOPR Intended Management | Cancer Type | |||||||
|---|---|---|---|---|---|---|---|---|
| All | Bladder | Kidney | Ovary | Pancreas | Prostate | SCL* | Stomach | |
| Patients | 8,460 | 1,127 | 1,070 | 2,075 | 862 | 1,974 | 720 | 632 |
| Treatment. N, (%) | 3,835 (45.3) | 504 (44.7) | 395 (36.9) | 1,095 (52.8) | 420 (48.7) | 842 (42.6) | 352 (48.9) | 227 (35.9) |
| Systemic therapy only | 2347 (27.7) | 261 (23.1) | 222 (20.7) | 811 (39.1) | 265 (30.7) | 428 (21.7) | 226 (31.4) | 134 (21.2) |
| Radiation only | 564 (6.7) | 71 (6.3) | 48 (4.5) | 84 (4.0) | 48 (5.6) | 273 (13.8) | 65 (9.0) | 13 (2.1) |
| Surgery only | 294 (3.5) | 50 (4.4) | 51 (4.8) | 52 (2.5) | 33 (3.8) | 33 (1.7) | 6 (0.8) | 34 (5.4) |
| Combination therapies‡ | 631 (7.5) | 123 (10.9) | 74 (6.9) | 151 (7.3) | 74 (8.6) | 108 (5.5) | 55 (7.6) | 46 (7.3) |
| Plans including therapy | ||||||||
| With systemic therapy | 2,944 (34.8) | 380 (33.7) | 291 (27.2) | 955 (46.0) | 338 (39.2) | 527 (26.7) | 278 (38.6) | 175 (27.7) |
| With radiation | 1,046 (12.4) | 173 (15.4) | 112 (10.5) | 113 (5.4) | 114 (13.2) | 374 (18.9) | 116 (16.1) | 44 (7.0) |
| With surgery | 540 (6.4) | 91 (8.1) | 76 (7.1) | 186 (9.0) | 53 (6.1) | 61 (3.1) | 15 (2.1) | 58 (9.2) |
| Biopsy | 939 (11.1) | 150 (13.3) | 146 (13.6) | 176 (8.5) | 74 (8.6) | 248 (12.6) | 60 (8.3) | 85 (13.4) |
| Watching | 3,686 (43.5) | 473 (42.0) | 529 (49.4) | 804 (38.7) | 368 (42.7) | 884 (44.7) | 308 (42.8) | 320 (50.6) |
SCL: Small cell lung
Combination therapies: systemic therapy ± radiotherapy ± surge
30-day Agreement
Table 3 shows by cancer type, the agreement between NOPR intended plan and claims-inferred actions.
Table 3.
Agreement of post-PET plan and claims-inferred actions at 30 days by cancer type
| Bladder | Kidney | Ovary | Pancreas | Prostate | SCL* | Stomach | P-value† | |
|---|---|---|---|---|---|---|---|---|
| Patients (total), n | 1,127 | 1,070 | 2,075 | 862 | 1,974 | 720 | 632 | |
| Systemic therapy only planned, n | 261 | 222 | 811 | 265 | 428 | 226 | 134 | |
| PPV, % | 66.7 | 40.1§ | 65.0 | 65.3 | 64.7 | 64.6 | 51.5 | <0.0001 |
| Raw Agreement,% | 71.2 | 73.7 | 71.1 | 69.8 | 71.6 | 69.3 | 65.7 | 0.025 |
| Kappa | 0.33 | 0.22 | 0.40 | 0.34 | 0.31 | 0.34 | 0.17 | <0.0001 |
| Radiation only planned, n | 71 | 48 | 46 | 48 | 273 | 65 | 34 | |
| PPV, % | 74.6 | 66.7 | 67.4 | 64.6 | 56.1 | 78.4 | 35.3 | 0.002 |
| Raw Agreement, % | 89.4 | 92.4 | 96.9 | 91.8 | 87.3 | 85.3 | 95.3 | <0.0001 |
| Kappa | 0.42 | 0.41 | 0.48 | 0.42 | 0.48 | 0.42 | 0.23 | 0.027 |
| Surgery only planned, n | 50 | 51 | 87 | 33 | 33 | 6 | 37 | |
| PPV, % | 44.0 | 54.9 | 57.5 | 33.3 | 27.3 | 33.3 | 35.3 | 0.020 |
| Raw Agreement, % | 84.5 | 88.9 | 89.5 | 89.2 | 91.5 | 93.9 | 86.7 | <0.0001 |
| Kappa | 0.14 | 0.27 | 0.27 | 0.14 | 0.07 | 0.07 | 0.16 | <.0001 |
| Biopsy planned, n | 150 | 146 | 176 | 74 | 248 | 60 | 85 | |
| PPV, % | 64.7 | 62.3 | 55.1 | 58.1 | 48.0 | 58.3 | 48.2 | 0.015 |
| Raw Agreement, % | 74.4 | 78.6 | 80.5 | 82.3 | 81.2 | 88.5 | 75.2 | <0.0001 |
| Kappa | 0.27 | 0.32 | 0.23 | 0.27 | 0.28 | 0.40 | 0.21 | <.0001 |
| Watching, n | 473 | 529 | 804 | 368 | 884 | 318 | 320 | |
| PPV, % | 70.8 | 80.9 | 74.1 | 73.4 | 73.3 | 68.2 | 66.3 | <0.0001 |
| Raw Agreement, % | 71.0 | 67.0 | 69.1 | 69.6 | 68.2 | 69.4 | 60.8 | 0.001 |
| Kappa | 0.41 | 0.34 | 0.38 | 0.39 | 0.37 | 0.38 | 0.21 | <0.0001 |
PPV: positive predictive value, i. e. proportion of NOPR patient with the plan showing claims confirming actual management.
Raw agreement: percentage of patients whose plans agree with claim for or without a specified action.
Kappa: raw agreement, adjusted for agreement by chance assuming independence of plan and actual management.
SCL: Small cell lung
A Medicare Part D drugs claims sample of a 40% sample of Medicare beneficiaries was available at Dartmouth. 45% of NOPR patients for all cancer types and indications were successfully linked to their Part D claims. We found for kidney cancer patients with chemotherapy only 6 of 222 patients had claims not previously identified.
p-values were calculated for between cancer type differences.
Appendix 3 includes results for agreements at 30 days with 95% confidence intervals plus at 60 days along with 95% confidence intervals.
For systemic-therapy alone, NOPR plans had a PPV of 65–67% for bladder, ovarian, pancreas, prostate, and SLC cancers and 51.5% for stomach cancer; the PPV for kidney cancer was lowest (40.1%), as anticipated. A Part D analysis of 40% of the cohort found 3% additional kidney cancer chemotherapies (see footer Table 3). Raw agreements ranged from 65.7% to 73.7% and kappas ranged from 0.17–0.40, in the slight to fair range. Among the radiotherapy alone cohorts, the PPVs were slighter higher (56.1%–78.4%), as were their kappas of 0.23 to 0.48, in the fair to moderate range.
Surgery only was planned in over 50 patients in kidney, bladder and ovarian cancer. For this group, the PPV ranged from 44.0%–57.5%, with only modest kappas (0.14–0.27).
Among planned combination therapy patients, claims for at least one of the planned therapies were found in 69.0% (data not shown).
A biopsy plan was confirmed by claims in 55.7% (range 48.0%–64.7%). Non-treatment (watching) plans were confirmed in only 76.3% of patients and were lowest in SCL and stomach cancer patients.
Timeframe
Extending the post-PET window to 60 days increased the PPV by 6%–13% for systemic-therapy, 0–6% for radiotherapy, and 12–18% for surgery. However, there was also an overall decline in kappas (Appendix 3, supplemental digital content 3).
Predictors
Table 4 shows the impact of patient age, performance status, comorbidity, cancer type, stage, referring specialty, and pre-PET plan in predicting post-PET plan and claims agreement.
Table 4.
Factors associated with agreement of treatment plan and claims for treatment
| Systemic therapy only | Radiation only | Surgery only | Watching | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ATTRIBUTE | CATEGORY | N | Adjusted PPV* % | p-Value | N | Adjusted PPV* % | p-Value | N | Adjusted PPV* % | p-Value | N | Adjusted PPV* % | p-Value |
| Overall | 2,347 | 62.0 | 564 | 66.0 | 294 | 45.6 | 3,686 | 74.3 | |||||
| Patient Factors | |||||||||||||
| Age | <=75 | 1,452 | 63.1 | 0.49 | 327 | 65.7 | 0.94 | 190 | 51.0 | 0.005 | 2,231 | 74.0 | 0.31 |
| >75 | 895 | 61.2 | 237 | 65.3 | 104 | 33.6 | 1,455 | 74.7 | |||||
| Performance Status | 0 | 862 | 58.8 | 0.03 | 244 | 66.0 | 0.11 | 153 | 47.8 | 0.36 | 1,690 | 78.0 | <0.0001 |
| 1 | 1,191 | 64.0 | 243 | 62.0 | 123 | 43.3 | 1,607 | 70.5 | |||||
| >=2 | 294 | 65.9 | 77 | 74.3 | 18 | 29.2 | 389 | 72.4 | |||||
| Comorbidity | 0 | 1,189 | 63.5 | 0.048 | 254 | 65.6 | 0.97 | 151 | 47.6 | 0.58 | 1,720 | 76.9 | <0.0001 |
| 1 or 2 | 874 | 59.4 | 227 | 65.3 | 103 | 38.5 | 1,429 | 73.8 | |||||
| >= 3 | 284 | 66.7 | 83 | 66.0 | 40 | 49.7 | 537 | 66.5 | |||||
| Cancer Factors | |||||||||||||
| Type | Bladder | 261 | 66.7 | <0.0001 | 71 | 74.6 | 0.004 | 50 | 44.0 | 0.035 | 473 | 74.0 | 0.0003 |
| Kidney | 222 | 40.1 | 48 | 66.7 | 51 | 54.9 | 529 | 82.1 | |||||
| Ovarian | 811 | 65.0 | 46 | 67.4 | 87 | 57.5 | 804 | 72.6 | |||||
| Pancreas | 265 | 65.3 | 48 | 64.6 | 33 | 33.3 | 368 | 77.0 | |||||
| Prostate | 428 | 64.7 | 273 | 56.0 | 33 | 27.3 | 884 | 71.9 | |||||
| SCL | 226 | 64.6 | 65 | 78.5 | 15 | 33.3 | 308 | 73.2 | |||||
| Stomach | 134 | 51.5 | 13 | 38.5 | 34 | 35.3 | 320 | 68.2 | |||||
| Stage | NED | 181 | 54.9 | 0.036 | 95 | 55.9 | 0.28 | 54 | 47.3 | 0.52 | 3,101 | 75.6 | <0.0001 |
| Local | 350 | 59.4 | 202 | 67.7 | 101 | 49.0 | 143 | 72.3 | |||||
| Metastatic | 1,816 | 63.7 | 267 | 67.8 | 139 | 40.5 | 442 | 64.9 | |||||
| Provider Factors | |||||||||||||
| Specialty | Medical Onc | 1,489 | 64.9 | 0.029 | 149 | 54.2 | <0.0001 | 127 | 39.4 | 0.20 | 1,796 | 70.1 | <0.0001 |
| Gyn or Gyn Onc | 271 | 62.3 | 11 | 57.1 | 28 | 33.5 | 308 | 77.2 | |||||
| Internal Med | 115 | 55.2 | 17 | 34.9 | 20 | 43.9 | 368 | 76.9 | |||||
| Surgery | 40 | 61.2 | 13 | 53.2 | 38 | 66.0 | 116 | 76.5 | |||||
| Radiation Onc | 75 | 50.8 | 241 | 80.9 | 11 | 30.2 | 270 | 72.9 | |||||
| Urology | 76 | 44.2 | 41 | 22.1 | 27 | 34.2 | 280 | 83.5 | |||||
| Other | 357 | 56.5 | 133 | 50.1 | 65 | 55.1 | 548 | 78.7 | |||||
| Pre-PET Plan | Other Imaging | 1,104 | 61.5 | 0.044 | 233 | 58.7 | <0.0001 | 100 | 46.5 | 0.011 | 2,101 | 74.1 | 0.0058 |
| Treat | 811 | 65.9 | 255 | 74.8 | 122 | 52.3 | 584 | 69.3 | |||||
| Biopsy | 237 | 58.3 | 42 | 58.9 | 52 | 32.3 | 384 | 77.2 | |||||
| Watch | 195 | 56.7 | 34 | 41.2 | 20 | 25.1 | 617 | 77.5 | |||||
NED: No evidence of disease or low probability of local recurrence. Gyn: Gynecology. Med: Medicine. Onc: Oncology. SCL: small cell lung
PPV: Adjusted estimates of the PPV were provided on the fitted model assuming that the other factors were fixed at their mean or prevalence in the data set.
With the exception of cancer type, the other factors had little impact in predicting PPV (62.0%) for systemic-therapies. If the referrer was a medical oncologist or if the pre-PET plan was also treatment, then the PPV increased minimally (3%–4%’s).
Radiotherapy’s PPV was slightly greater at 66%. Patient factors or cancer stage were non-predictive. Radiotherapy was most commonly planned in prostate cancer, yet it had the second lowest PPV (56%). Not surprisingly, when the referrer was a radiation oncologist, the PPV was greatest (81%).
Surgery alone was an infrequent plan and had a PPV 46%. Age over 75 years or a non-surgeon referrer was associated with even lower PPVs.
The absence of treatment claims in the intended watching cohort was unrelated to age and was more common if patients had a good performance status, low co-morbidity, non-metastatic stage, had kidney cancer, were not referred by a medical oncologist, and had a pre-PET plan other than treatment.
Discussion
We assessed how often referring physicians’ intended management after PET in the NOPR database, is concordant with actual management inferred from Medicare claims for the indication of restaging of previously treated cancers. We found only moderate agreements across all NOPR plan categories—treatment, biopsy or watching (non-action). The PPV of systemic-therapy plans clustered between 64–67% in five different cancer types. Claims confirming plans for invasive procedures (surgery or biopsy) were particularly infrequent. Moreover, over 25% of patients with an observation plan had treatment claims within 30 days. When longer timeframes were used, the PPV rates increased but the overall kappas declined.
We sought correlations that might explain these differences by assessing patient, cancer, and physician factors. This was largely unsuccessful. Patient age, performance status, and comorbidity had little association with PPV within a treatment category. For the two most frequent actions, medical oncologists versus all other specialties had similar PPVs for claims confirming chemotherapy as well as similar crossover rates from watching plans to treatment.
Information gained from PET is only one factor influencing patient management. Unavailable or unmeasured factors include other diagnostic test results, the extent and type of prior treatment, whether the referrer is the patient’s primary cancer care provider, and the physician’s experience with PET. The NOPR management plan reflects the physician’s intent or recommendation shortly after the PET scan is reported, and not necessarily the patient’s agreement to a plan. Also, the accuracy of NOPR questionnaire responses was not independently confirmed by review of the physician’s office records. Accordingly, one source of discordance may be inaccuracy of the NOPR data.
Our results show agreement rates comparable to those of smaller, less detailed analysis by Henderson that assessed 489 NOPR scans for all indications in kidney and pancreatic cancer patients(18). Their analysis found only fair agreement rates for observation and treatment (kappa=0.39 and 0.36).
Another limitation is the expectation that claims will accurately reflect actual clinical actions performed. Since 2000, numerous reports using SEER-Medicare linked data have assessed the completeness of claims in identifying initial chemotherapy(19–24), surgery(25, 26), and radiation(15, 27). Most of these studies used broad time windows (up to one year post-diagnosis). Since our analysis used a narrow 30-day window following PET, inaccurate claims service dates could be a major source of non-matching (false-negatives). Two recent reports partly address this issue. First, Lamont assessed CMS claims against trial data as the reference standard in six CALGB trial cohorts of first-line metastatic chemotherapy.(23) They found that claims correctly identified 78% of the drugs given and the treatment schedule. In a second study, Lund validated SEER initial treatment plans against hospital and outpatient records that were re-abstracted for chemotherapy in four cancer types.(24) They found that the claim sensitivities were very time dependent ranging from 36%–50% at two months to 84%–96% at six months.
Australian investigators have used a design similar to NOPR in that they collected prospective data at three to six centers for a range of cancer types in which PET was used predominantly for initial staging and determined the change in management plans associated with PET.(28–32) They subsequently assessed medical records for actual care over the next 3 to 12 months. They found agreement rates ranging from 53% to 75% (average 65%), similar to our claims-inferred actions.
We previously noted that intended management changes might not always be appropriate.(9) Moreover, changes in intended actions have a presumed, but uncertain, relationship to more tangible health outcomes—progression-free or overall survival(33)—that are so heavily dependent on the effectiveness of the clinical actions chosen.
Our analysis highlights that intended management in patients with previously treated cancer does not consistently reflect implemented action. When higher accuracy levels are required, future evaluations of other diagnostic tests under the CED policy likely will require alternative designs.
Supplementary Material
Acknowledgments
Financial Support: National Cancer Institute Grand Opportunity Award RC2CA148259 and the Academy for Molecular Imaging (BEH). Funding for development of the NOPR was provided by the Academy for Molecular Imaging, but the registry is otherwise self-supported by the fees paid by participating PET facilities.
Contributor Information
Bruce E. Hillner, Department of Internal Medicine and the Massey Cancer Center, Virginia Commonwealth University, Richmond, VA
Tor D. Tosteson, The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth, Lebanon, NH
Anna N. A. Tosteson, The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth, Lebanon, NH
Qianfei Wang, The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth, Lebanon, NH
Yunjie Song, The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth, Lebanon, NH
Tracy Onega, The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth, Lebanon, NH
Lucy G. Hanna, Center for Statistical Sciences, Brown University, Providence, RI.
Barry A. Siegel, Division of Nuclear Medicine, Mallinckrodt Institute of Radiology andSiteman Cancer Center, Washington University School of Medicine, St. Louis, MO
References
- 1.Centers for Medicare and Medicaid Services. AB-01-54: Expanded Coverage of Positron Emission Tomography (PET) Scans and Related Claims Processing Changes. 2000 Dec 15; http://www.cms.gov/transmittals/downloads/R136CIM.pdf.
- 2.Lindsay MJ, Siegel BA, Tunis SR, Hillner BE, Shields AF, Carey BP, et al. The National Oncologic PET Registry: Expanded Medicare Coverage for PET Under Coverage with Evidence Development. AJR Am J Roentgenol. 2007;188(4):1109–13. doi: 10.2214/AJR.06.1175. [DOI] [PubMed] [Google Scholar]
- 3.Tunis SR, Pearson SD. Coverage Options For Promising Technologies: Medicare’s ‘Coverage With Evidence Development’. Health Aff. 2006;25(5):1218–30. doi: 10.1377/hlthaff.25.5.1218. [DOI] [PubMed] [Google Scholar]
- 4.Centers for Medicare & Medicaid Services. National Coverage Determinations with Data Collection as a Condition of Coverage: Coverage with Evidence Development. Document Issued July 12, 2006. http://www.cms.hhs.gov/Transmittals/downloads/R956CP.pdf.
- 5.Tunis SR, Carino TV, Williams RD, II, Bach PB. Federal Initiatives To Support Rapid Learning About New Technologies. Health Aff. 2007;26(2):w140–9. doi: 10.1377/hlthaff.26.2.w140. [DOI] [PubMed] [Google Scholar]
- 6.Hillman BJ, Goldsmith JC. The Sorcerer’s Apprentice: How medical imaging is changing health care. New York: Oxford University Press; 2011. [Google Scholar]
- 7.Hillner BE, Siegel BA, Shields AF, Liu D, Gareen IF, Hanna L, et al. The impact of positron emission tomography (PET) on expected management during cancer treatment: findings of the National Oncologic PET Registry. Cancer. 2009;115:410–8. doi: 10.1002/cncr.24000. [DOI] [PubMed] [Google Scholar]
- 8.Hillner BE, Siegel BA, Shields AF, Liu D, Gareen IF, Hunt E, et al. Relationship Between Cancer Type and Impact of PET and PET/CT on Intended Management: Findings of the National Oncologic PET Registry. J Nucl Med. 2008;49:1928–35. doi: 10.2967/jnumed.108.056713. [DOI] [PubMed] [Google Scholar]
- 9.Hillner BE, Siegel BA, Liu D, Shields AF, Gareen IF, Hanna L, et al. Impact of positron emission tomography/computed tomography and positron emission tomography (PET) alone on expected management of patients with cancer: initial results from the National Oncologic PET Registry. J Clin Oncol. 2008;26(13):2155–61. doi: 10.1200/JCO.2007.14.5631. [DOI] [PubMed] [Google Scholar]
- 10.Hillner BE, Siegel BA, Shields AF, Duan F, Gareen IF, Hanna L, et al. Impact of dedicated brain PET on intended patient management in participants of the national oncologic PET Registry. Mol Imaging Biol. 2011;13(1):161–5. doi: 10.1007/s11307-010-0427-5. Epub 2010/11/17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Levine MN, Julian JA. Registries That Show Efficacy: Good, but Not Good Enough. J Clin Oncol. 2008;26(33):5316–9. doi: 10.1200/JCO.2008.18.3996. [DOI] [PubMed] [Google Scholar]
- 12.Tunis S, Whicher D. The National Oncologic PET Registry: Lessons Learned for Coverage With Evidence Development. J Am Coll Radiol. 2009;6(5):360–5. doi: 10.1016/j.jacr.2009.01.016. [DOI] [PubMed] [Google Scholar]
- 13.Keating NL, Landrum MB, Lamont EB, Bozeman SR, Krasnow SH, Shulman LN, et al. Quality of Care for Older Patients With Cancer in the Veterans Health Administration Versus the Private Sector. Ann Intern Med. 2011;154(11):727–36. doi: 10.7326/0003-4819-154-11-201106070-00004. [DOI] [PubMed] [Google Scholar]
- 14.Lamont EB, Lauderdale DS, Schilsky RL, Christakis NA. Construct validity of medicare chemotherapy claims: the case of 5FU. Med Care. 2002;40(3):p201–11. doi: 10.1097/00005650-200203000-00004. [DOI] [PubMed] [Google Scholar]
- 15.Smith BD, Pan I-W, Shih Y-CT, Smith GL, Harris JR, Punglia R, et al. Adoption of Intensity-Modulated Radiation Therapy for Breast Cancer in the United States. J Natl Cancer Inst. 2011;103(10):798–809. doi: 10.1093/jnci/djr100. [DOI] [PubMed] [Google Scholar]
- 16.Klabunde CN, Legler JM, Warren JL, Baldwin L-M, Schrag D. A Refined Comorbidity Measurement Algorithm for Claims-Based Studies of Breast, Prostate, Colorectal, and Lung Cancer Patients. Ann Epidemiol. 2007;17(8):584–90. doi: 10.1016/j.annepidem.2007.03.011. [DOI] [PubMed] [Google Scholar]
- 17.Klabunde CN. ( http://healthservices.cancer.gov/seermedicare/program/comorbidity.html)
- 18.Henderson LM, Reeder-Hayes K, Hinton SP, Carpenter WR, Chen RC. Comparing physician-reported cancer management plans with Medicare services received. Arch Intern Med. 2012;172(8):664–6. doi: 10.1001/archinternmed.2012.271. Epub 2012/04/25. [DOI] [PubMed] [Google Scholar]
- 19.Lamont EB, Herndon JE, 2nd, Weeks JC, Henderson IC, Lilenbaum R, Schilsky RL, et al. Criterion validity of Medicare chemotherapy claims in Cancer and Leukemia Group B breast and lung cancer trial participants. J Natl Cancer Inst. 2005;97(14):1080–3. doi: 10.1093/jnci/dji189. [DOI] [PubMed] [Google Scholar]
- 20.Du XL, Key CR, Dickie L, Darling R, Geraci JM, Zhang D. External validation of medicare claims for breast cancer chemotherapy compared with medical chart reviews. Med Care. 2006;44(2):124–31. doi: 10.1097/01.mlr.0000196978.34283.a6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Liang SY, Phillips KA, Wang G, Keohane C, Armstrong J, Morris WM, et al. Tradeoffs of using administrative claims and medical records to identify the use of personalized medicine for patients with breast cancer. Med Care. 2011;49(6):e1–8. doi: 10.1097/MLR.0b013e318207e87e. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Warren JL, Harlan LC, Fahey A, Virnig BA, Freeman JL, Klabunde CN, et al. Utility of the SEER-Medicare data to identify chemotherapy use. Med Care. 2002;40(8 Suppl):IV-55–61. doi: 10.1097/01.MLR.0000020944.17670.D7. [DOI] [PubMed] [Google Scholar]
- 23.Lamont EB, Lan L. Sensitivity of Medicare Claims Data for Measuring Use of Standard Multiagent Chemotherapy Regimens. Med Care. 2012 doi: 10.1097/MLR.0b013e31824e342f. Epub 2012/03/14. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Lund JL, Sturmer T, Harlan LC, Sanoff HK, Sandler RS, Brookhart MA, et al. Identifying Specific Chemotherapeutic Agents in Medicare Data: A Validation Study. Med Care. 2011 doi: 10.1097/MLR.0b013e31823ab60f. Epub 2011/11/15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Miller DC, Saigal CS, Warren JL, Leventhal M, Deapen D, Banerjee M, et al. External validation of a claims-based algorithm for classifying kidney-cancer surgeries. BMC Health Serv Res. 2009;9:92. doi: 10.1186/1472-6963-9-92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Cooper GS, Virnig B, Klabunde CN, Schussler N, Freeman J, Warren JL. Use of SEER-Medicare data for measuring cancer surgery. Med Care. 2002;40(8 Suppl):IV-43–8. doi: 10.1097/00005650-200208001-00006. [DOI] [PubMed] [Google Scholar]
- 27.Virnig BA, Warren JL, Cooper GS, Klabunde CN, Schussler N, Freeman J. Studying radiation therapy using SEER-Medicare-linked data. Med Care. 2002;40(8 Suppl):IV-49–54. doi: 10.1097/00005650-200208001-00007. [DOI] [PubMed] [Google Scholar]
- 28.Scott AM, Gunawardana DH, Kelley B, Stuckey JG, Byrne AJ, Ramshaw JE, et al. PET changes management and improves prognostic stratification in patients with recurrent colorectal cancer: results of a multicenter prospective study. J Nuclear Med. 2008;49(9):1451–7. doi: 10.2967/jnumed.108.051615. [DOI] [PubMed] [Google Scholar]
- 29.Scott AM, Gunawardana DH, Bartholomeusz D, Ramshaw JE, Lin P. PET changes management and improves prognostic stratification in patients with head and neck cancer: results of a multicenter prospective study. J Nuclear Med. 2008;49(10):1593–600. doi: 10.2967/jnumed.108.053660. Epub 2008/09/17. [DOI] [PubMed] [Google Scholar]
- 30.Scott AM, Gunawardana DH, Wong J, Kirkwood I, Hicks RJ, Ho Shon I, et al. Positron emission tomography changes management, improves prognostic stratification and is superior to gallium scintigraphy in patients with low-grade lymphoma: results of a multicentre prospective study. Eur J Nucl Med Mol Imag. 2009;36(3):347–53. doi: 10.1007/s00259-008-0958-z. [DOI] [PubMed] [Google Scholar]
- 31.Fulham MJ, Carter J, Baldey A, Hicks RJ, Ramshaw JE, Gibson M. The impact of PET-CT in suspected recurrent ovarian cancer: A prospective multi-centre study as part of the Australian PET Data Collection Project. Gyne Oncol. 2009;112(3):462–8. doi: 10.1016/j.ygyno.2008.08.027. [DOI] [PubMed] [Google Scholar]
- 32.Chatterton BE, Ho Shon I, Baldey A, Lenzo N, Patrikeos A, Kelley B, et al. Positron emission tomography changes management and prognostic stratification in patients with oesophageal cancer: results of a multicentre prospective study. Eur J Nucl Med Mol Imag. 2009;36(3):354–61. doi: 10.1007/s00259-008-0959-y. [DOI] [PubMed] [Google Scholar]
- 33.Staub LP, Lord SJ, Simes RJ, Dyer S, Houssami N, Chen RY, et al. Using patient management as a surrogate for patient health outcomes in diagnostic test evaluation. BMC Med Res Method. 2012;12:12. doi: 10.1186/1471-2288-12-12. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.