Measuring Clinically Significant Chemotherapy-Related Toxicities Using Medicare Claims From Cancer and Leukemia Group B (CALGB) Trial Participants

Abstract

Background

Because the elderly are underrepresented on clinical trials, physicians have few sources of information to estimate the risks (ie, toxicities) and benefits of chemotherapy administration to the elderly.

Objective

Our goal was to determine whether the standard measures of toxicity used in clinical trials could be captured from observational Medicare claims data.

Methods

We identified 175 elderly clinical trial patients treated on 2 Cancer and Leukemia Group B (CALGB) trials (9344, adjuvant breast study and 9730, advanced lung cancer study) and merged participants’ CALGB data with their Medicare data. From CALGB data, we identified the most frequent Extended Clinical Toxicity Critieria grade III/IV toxicities. We reviewed diagnostic and procedure codes from Medicare manuals, developed algorithms to measure the toxicities, and then finalized the algorithms after empiric review of patients’ codes. We compared results of Medicare algorithms to gold standard CALGB toxicity information to calculate test characteristics.

Results

CALGB data documented that 15 grade III/IV chemotherapy-related toxicities occurred in ≥3% of the 175 patients: white blood cell, hemoglobin, platelets, anorexia, nausea, vomiting, diarrhea, stomatitis, sensory neuropathy, motor neuropathy, motor or sensory neuropathy, dyspnea, hyperglycemia, infection, and malaise. Vomiting was the only toxicity identified by the Medicare-based algorithm with a sensitivity, specificity, and area under the receiver operator curve of ≥80%.

Conclusions

The results of this preliminary study suggest that Medicare diagnostic and procedure codes may be of only limited value in measuring clinically significant chemotherapy-related toxicities in elderly Medicare beneficiaries. Future research includes confirming these findings in a larger and more diverse sample.

The elderly are numerically underrepresented^1–3 and possibly physiologically misrepresented⁴ in clinical trials of cancer chemotherapy so that those enrolled in trials are potentially younger and healthier than the average elderly cancer patient. As a result, the benefits and toxicities of chemotherapy in the general population of elderly Americans may not be the same as those described in trial participants. Nevertheless, clinicians need information about the expected benefits and toxicities of chemotherapy in the elderly. A solution to this problem is to supplement clinical trial results with observational results. Prior research has shown Centers for Medicare and Medicaid Services (CMS) data accurately measure chemotherapy use^5–8 and the important end point of disease-free survival,⁹ but the extent to which CMS data accurately measure the toxicity outcomes traditionally reported in clinical trials is not yet known.

Nonetheless, there are increasing numbers of reports in the clinical oncology literature by investigators who seek to make inferences about chemotherapy-related toxicity in the elderly through study of diagnostic codes within Medicare claims data.^10–15 Before further empirical work that has the potential of influencing the cancer care of community elderly is undertaken, basic methodologic work to determine if Medicare claims are in fact accurate measures of chemotherapy-related toxicity is needed. In this pilot study, we seek to estimate the validity of Medicare claims-based toxicity codes through comparison to an external, gold standard measure of toxicity from Cancer and Leukemia Group B (CALGB) clinical trial data. Our goal is to determine whether the standard measures of toxicity used in clinical trials could be captured from observational Medicare claims data.

METHODS

We formed a retrospective cohort consisting of all patients age 65 or older enrolled on 1 of 2 CALGB trials: 9344, “doxorubicin dose escalation, with or without taxol, as part of the cancer adjuvant regimen for node positive breast cancer” (hereafter the “CALGB breast cancer protocol”)¹⁶ (N = 52); and 9730, “single-agent versus combination chemotherapy in advanced non-small cell lung cancer: a CALGB randomized trial of efficacy, quality of life, and cost-effectiveness” (hereafter the “CALGB lung cancer protocol”)¹⁷ (N = 186). We linked the cohorts’ CALGB clinical trial data (eg, demographic information, information pertaining to chemotherapy administration) to their CMS Medicare claims files (ie, denominator file, Carrier file, OUTPT file, MEDPAR file) from the corresponding calendar period to create the “CALGB-Medicare data set.” We were able to match 96% (228 of 238) of the participants to Medicare files, a rate consistent with previous literature.¹⁸ Among these 228 eligible patients, 40 were removed from the analytic sample because of enrollment in HMOs whose claims were not processed through CMS and an additional 13 were removed because of a lack of enrollment in Medicare part B. The final analytic sample was 175 patients. We used this data to perform the criterion validation study described here.

This study was approved by the University of Chicago and Massachusetts General Hospital institutional review boards and conducted in compliance with their regulations. Data quality was ensured by careful review of data by CALGB Statistical Center staff and by the study chairperson. Statistical analyses were approved and confirmed by CALGB statisticians. All analyses were performed using STATA version 8 SE.

In addition to being observed closely for potential survival benefits associated with clinical trial therapy, patients enrolled on clinical trials are also observed closely for evidence of toxicity or adverse events (AEs) related to the same therapies. An AE is any unfavorable and unintended sign (including an abnormal laboratory finding), symptom, or disease temporally associated with the use of a medical treatment or procedure that may or may not be considered related to the medical treatment or procedure. The AEs experienced by patients enrolled on clinical trials are categorized through widely available toxicity metrics.¹⁹ The CALGB documents AEs through the Extended Clinical Toxicity Critieria (ECTC) a descriptive taxonomy that incorporates class, term, and severity of toxicity. Class refers to the organ system involved and term is a unique representation of a specific event used for medical documentation and scientific analyses. Severity is captured through a grading scale that is provided for each AE term. The CALGB’s ECTC shares many features of the National Cancer Institute’s Common Toxicity Criteria for Adverse Events (CTCAE) including the approach to grading. In both taxonomies grade I is a mild AE, grade II is a moderate AE, grade III is a severe AE, grade IV is a life-threatening AE or disabling AE, and grade V a fatal AE (ie, death related to AE). Clinical trialists typically include AE rates in their reports of trial results, often describing frequencies of those AEs graded III and higher. Indeed, the AE measures are accepted by the Food and Drug Administration (FDA) as a regulatory end point that demonstrates the safety of new therapies.

Because CMS claims rely on International Classification of Disease-9-Clinical Modification (ICD-9-CM) diagnostic and procedure codes, Common Procedure Terminology codes, and Health Care Financing Administration Procedure codes to describe patients’ health care encounters for billing purposes, it is at least theoretically possible that CMS claims capture patients’ experiences with clinically important toxicity.

Evaluating CALGB ECTC toxicity information on all 175 patients, we identified 15 possibly chemotherapy-related toxicities with a grade ≥III¹ in at least 3% of the study patients. We reviewed diagnostic and procedure codes from CMS coding manuals to develop initial algorithms to measure the toxicities. We term this “a priori determination of algorithms.” We then finalized the algorithms after empiric review of individual patients’ actual CMS codes incurred during the observation period (ie, date of first trial treatment through 90 days after last trial treatment) (see Appendix 1, which can be found on the Medical Care website, www.lww-medicalcare.com). We term this effort, “post hoc refinement of algorithms.” We then applied the algorithms we established to each patient’s ambulatory and inpatient Medicare claims files during the observation period for evidence of the toxicities of interest. We compared results of each of our CMS toxicity algorithms with a gold standard of CALGB grade III/IV toxicity information to calculate the test characteristics of the CMS algorithms (ie, sensitivity, specificity, and area under the receiver operator curve, ROC). By sensitivity we mean the proportion of the patients known (according to CALGB data) to have experienced the grade III or IV toxicity of interest who are correctly identified through CMS claims as having experienced it during the observation period. By specificity, we mean the proportion of the patients known (according to CALGB data) not to have experienced the grade III/IV toxicity who are correctly identified through CMS claims as not having experienced the toxicity during that same period. The global performance of the measures is summarized by the area under the ROC (graph of sensitivity vs. 1 – specificity); the greater the ROC curve (maximum 1.00) the better the discriminatory accuracy of the measure.^20,21

In a second, but related approach, we evaluated the extent to which the occurrence of CMS “E codes” reflected clinically significant toxicities as documented in CALGB data. Within the CMS ICD-9-CM coding taxonomy, E codes are intended to indicate that a given diagnosis code is the result of “an external cause.” Distinct E codes may accompany any diagnosis code that occurs as a result of “environmental events, circumstances, and conditions as the cause of injury, poisoning, and other adverse effects.”²² We evaluate the extent to which the distinct E code indicating toxicity from “antineoplastic and immunosuppressive drugs,” E933.1, reflected chemotherapy-related toxicities. We compared rates of E codes among patients with and without grade III/IV toxicity using a χ² test.

RESULTS

The sample was 51% female, the mean age was 71.0 years (SD ±4.5 years), and 86% were white, 10% black, 3% Hispanic, and 1% Asian. Table 1 describes demographic and treatment characteristics of the patients according to trial. We categorized the 15 toxicities into the following organ-based groupings: “bone marrow,” “gastrointestinal,” “neurologic,” and “other” and Table 2 describes the rates of each of the toxicities among the sample of 175 patients.

TABLE 1.

Demographic and Disease Characteristics for the Analytic Sample Containing 175 Patients

Variable	Value	No.
Mean age, yr (±SD)	71.0 (±4.5)	175
Sex, proportion		175
Female	0.51	89
Male	0.49	86
Race, proportion		175
White	0.86	150
Black	0.10	18
Hispanic	0.03	6
Asian	0.01	1
Education, proportion		150
<HS	0.26	39
HS graduate	0.36	54
College (at least some)	0.38	57
Marital status, proportion		154
Single (never married)	0.06	9
Married	0.62	95
Divorced	0.11	18
Widowed	0.21	32
Performance status (WHO), proportion		130
0 (fully active, able to carry on all predisease performance without restriction)	0.39	51
1 (restricted in physically strenuous activity but ambulatory and able to carry out work of a light or sedentary nature, eg light house work, office work)	0.49	63
2 (ambulatory and capable of all self care but unable to carry out any work activities. Up and about more than 50% of waking hours)	0.12	16
CALGB protocol, proportion		175
Lung cancer protocol (9730)	0.74	130
Paclitaxel/carboplatin arm		68
Paclitaxel only arm		62
Breast cancer protocol (9344)	0.26	45
Doxorubicin/cyclophosphamide arm		17
Doxorubicin/cyclophosphamide/paclitaxel arm		28

SD indicates standard deviation; HS, high school; WHO, World Health Organization; CALGB, Cancer and Leukemia Group B.

TABLE 2.

Grade III/IV Toxicity Experience Among the Analytic Sample Containing 175 Patients

CALGB Grade III/IV Toxicity	n	Proportion (%)
Bone marrow
White blood cell	68	39
Hemoglobin	14	8
Platelet	22	13
Gastrointestinal
Anorexia	8	5
Nausea	12	7
Vomiting	7	4
Diarrhea	6	3
Stomatitis	5	3
Neurologic
Sensory neuropathy	20	11
Motor neuropathy	12	7
Motor or sensory neuropathy	25	14
Other
Dyspnea	8	5
Hyperglycemia	32	18
Infection	16	9
Malaise	19	11

The test characteristics of each of our toxicity-specific CMS-based algorithms are shown in Table 3. The only toxicity for which the sensitivity, specificity, and area under the ROC are all ≥80% was vomiting.

TABLE 3.

Test Characteristics of the CMS Grade III/IV Toxicity Algorithms

CMS-Based Grade III/IV Toxicity Algorithm	Sensitivity (%)	95% CI	Specificity (%)	95% CI	Area Under ROC Curve (%)	95% CI
Bone marrow
White blood cell	62	49–73	71	66–87	66	59–74
Hemoglobin 1	86	57–98	61	56–71	74	63–84
Hemoglobin 2	64	35–87	81	74–87	73	59–86
Platelet	45	24–68	93	88–96	69	58–80
Gastrointestinal
Anorexia	25	3–65	96	92–99	61	45–77
Nausea	42	5–57	91	86–95	67	52–81
Vomiting	86	42–99	81	74–87	83	69–98
Diarrhea	50	12–88	96	92–98	73	40–82
Stomatitis	20	1–72	98	94–99	59	39–78
Neurologic
Sensory neuropathy	45	23–68	83	76–89	64	52–76
Motor neuropathy	50	21–79	82	75–88	66	51–81
Motor or sensory neuropathy	44	5–36	84	77–89	64	54–74
Other
Dyspnea 1	38	9–76	71	51–67	54	36–73
Dyspnea 2	75	35–97	59	51–67	67	51–84
Hyperglycemia	31	16–50	86	79–91	59	50–67
Infection	75	11–59	82	76–88	79	67–90
Malaise	32	13–57	80	73–86	56	44–67

Overall, only 6 of 175 (3%) of patients had an E933.1 code at any time during the observation period. Among the 111 patients with at least one of the grade III/IV toxicities of interest, only 4.5% had any occurrence of the code. Among the 64 patients without any grade III/IV toxicities of interest, 1.6%, had at least one occurrence of the code (P = 0.303 for this comparison). Results were similar when occurrence of E933.1 and each of the 15 grade III/IV toxicities were compared (results not shown).

DISCUSSION

Through this study, we found that the standard measures of toxicity used in clinical trials are not accurately captured from observational Medicare claims data. Specifically, this validation study shows that for elderly Medicare beneficiaries treated for breast cancer or lung cancer on 1 of 2 randomized phase III CALGB chemotherapy trials, CMS-based algorithms utilizing diagnostic and procedure codes generally did not accurately reflect the experience of clinically significant toxicity (ie, grade III/IV toxicity). Although it was possible to develop an algorithm with sufficient granularity to distinguish clinically significant vomiting (ie, ECTC grade III/IV), no other toxicity could be identified with both the sensitivity and specificity of ≥80%. For example, the sensitivity of a broad algorithm for measuring ECTC grade III/IV hemoglobin was high at 86%, but the specificity was poor (ie, 61%). The other more narrow algorithm that relied only on codes indicating transfusions or use of red blood cell growth factors had a higher specificity (ie, 81%), but lost on sensitivity (ie, 64%).

Our focus on “80%” as a minimum value for test characteristics relates to the fact that for even common events (eg, those with a rate of 40%), a test with a sensitivity of 80% has a positive predictive value (PPV) of only 73% and a test with a specificity of 80% has a negative predictive value (NPV) of 86%. With less frequent events, the PPV is even less favorable. Although the optimal test characteristics may be a matter of taste and debate, PPVs and NPVs lower than this seem to us to be of questionable utility for empirical research that is intended to improve the clinical care of elderly cancer patients and to shape health care policy. Of course, these methods can be used by any analyst who feels the reported sensitivity and specificity are adequate to their needs (eg, in identifying cases for further review by pulling actual charts, someone might accept a high sensitivity and low specificity).

Because certain clinically significant toxicities were rare among the trial subjects, whether our sample size was adequate to evaluate the validity of the claims is an important question. Although we were in fact underpowered to detect a difference from the idealized minimum sensitivity value of 80% for 2 toxicities (ie, ECTC grade III/IV dyspnea and diarrhea), we were adequately powered to reject the null hypothesis that their sensitivities were 80% or higher for all of the remaining 12 toxicities (ie, ECTC grade III/IV white blood cell, hemoglobin, platelets, anorexia, nausea, stomatitis, sensory neuropathy, motor neuropathy, motor or sensory neuropathy, hyperglycemia, infection, and malaise). Although it may be that the poor test characteristics are solely the result of inadequacy of our algorithms and that truly CMS claims do accurately reflect clinically significant chemotherapy toxicities, our strategy that relied on both a priori and post hoc code selection after empiric review of all codes weighs against this explanation.

Instead, it may be that unlike other elements of the cancer treatment paradigm,^5,7,23,24 CMS diagnostic and procedure codes may provide an inadequate lens for viewing clinically significant chemotherapy-related toxicities. Our goal was to determine whether the standard measures of toxicity used in clinical trials could be captured from claims. It is possible that other important outcomes potentially associated with chemotherapy (eg, number of hospitalizations after treatment, number of hospital days after treatment, rates of transitioning from home to hospital to skilled nursing facility) could be identified accurately from claims. However, research would be needed to determine whether a claims-based algorithm could adequately distinguish treatment-related from disease-related events such that it would provide a useful measure of chemotherapy toxicity per se.

The reasons why measurement of “toxicity” is apparently less accurate than measurement of “chemotherapy administration” and “relapsed disease” are not obvious. However, the difficulty may relate to the fact that the taxonomy of the comparatively nuanced ECTC does not have a clear parallel in diagnostic and procedures claims the way that the more explicit “chemotherapy administration” and “relapsed disease” do. That is, chemotherapy administration algorithms relied exclusively on a limited number of procedure codes (ie, intravenous chemotherapy administration and drug codes) and measurement of relapsed disease relied on exclusively a single type diagnostic code (ie, secondary malignancy codes). Our toxicity algorithms used here generally included combinations of both procedure and diagnostic codes to discern what are often transient patient symptoms and findings. It may be that in general transient health states are poorly captured in administrative claims data, either through inadequacy of codes or incomplete coding by providers. Given that we have shown that Medicare claims very accurately captured chemotherapy administration in this cohort previously,⁷ we are not concerned that our findings are the result of confounding related to a systematic failure by the providers to submit or CMS to accept Medicare claims on trial enrollees. However, given the highly select nature of this sample, the applicability of these results to the general population of elderly cancer patients receiving chemotherapy outside of clinical trials is not known.

In summary, despite the ability of CMS claims to accurately measure use of certain anticancer therapies and critical efficacy end points, results of this pilot study suggest that algorithms based on diagnostic and procedure codes from CMS data may not reliably capture critical toxicity end points among patients enrolled in clinical trials. Thus, it is not yet clear that commonly studied CMS-based data sources (eg, Surveillance, Epidemiology, and Endpoints Registry [SEER]-Medicare data, CMS claims data) allow researchers to make accurate inferences regarding the experience of chemotherapy-related toxicity in the general population of elderly patients with solid tumors. Future research will focus on confirming these findings in a larger and more diverse patient sample and exploring other potential CMS-based metrics of chemotherapy-related toxicity.