Hospitalization Risk During Chemotherapy for Advanced Cancer: Development and Validation of Risk Stratification Models Using Real-World Data

Abstract

PURPOSE

Hospitalizations are a common occurrence during chemotherapy for advanced cancer. Validated risk stratification tools could facilitate proactive approaches for reducing hospitalizations by identifying at-risk patients.

PATIENTS AND METHODS

We assembled two retrospective cohorts of patients receiving chemotherapy for advanced nonhematologic cancer; cohorts were drawn from three integrated health plans of the Cancer Research Network. We used these cohorts to develop and validate logistic regression models estimating 30-day hospitalization risk after chemotherapy initiation. The development cohort included patients in two health plans from 2005 to 2013. The validation cohort included patients in a third health plan from 2007 to 2016. Candidate predictor variables were derived from clinical data in institutional data warehouses. Models were validated based on the C-statistic, positive predictive value, and negative predictive value. Positive predictive value and negative predictive value were calculated in reference to a prespecified risk threshold (hospitalization risk ≥ 18.0%).

RESULTS

There were 3,606 patients in the development cohort (median age, 63 years) and 634 evaluable patients in the validation cohort (median age, 64 years). Lung cancer was the most common diagnosis in both cohorts (26% and 31%, respectively). The selected risk stratification model included two variables: albumin and sodium. The model C-statistic in the validation cohort was 0.69 (95% CI, 0.62 to 0.75); 39% of patients were classified as high risk according to the prespecified threshold; 30-day hospitalization risk was 24.2% (95% CI, 19.9% to 32.0%) in the high-risk group and 8.7% (95% CI, 6.1% to 12.0%) in the low-risk group.

CONCLUSION

A model based on data elements routinely collected during cancer treatment can reliably identify patients at high risk for hospitalization after chemotherapy initiation. Additional research is necessary to determine whether this model can be deployed to prevent chemotherapy-related hospitalizations.

INTRODUCTION

Hospitalizations are a common, burdensome, and potentially avoidable complication of cancer treatment.^1-3 Hospitalizations are particularly common among patients with advanced cancer,⁴ with both cancer-related symptoms and treatment-related toxicities as contributing factors.³ These hospitalizations are also costly, representing the largest single component of Medicare spending for patients with advanced cancer.⁵ Furthermore, there is emerging evidence that preempting or avoiding complications that lead to hospitalization may be associated with improvements in survival among patients receiving cancer treatment.^6,7

CONTEXT

• Key Objective To develop and validate an approach for identifying patients at increased risk for hospitalization after initiation of chemotherapy for advanced cancer.

• Knowledge Generated We describe the development and external validation of three risk stratification models designed to identify patients at increased risk for any-cause 30-day hospitalization after initiation of chemotherapy for advanced solid tumor malignancy. A two-variable model using pretreatment sodium and albumin levels stratified patients into a high-risk group (39% of patients, with a 30-day hospitalization risk of 24.2%) and a standard-risk group (61% of patients, with a 30-day hospitalization risk of 8.7%).

• Relevance The two validated risk stratification models described here can be used to identify patients at increased risk for hospitalization during chemotherapy treatment. We encourage further study to confirm and update these risk stratification models in an evolving ambulatory practice environment. Studies testing approaches for preventing hospitalizations in high-risk patients are of particular importance.

Cancer care providers and policymakers are increasingly recognizing the importance of developing new strategies to prevent hospitalizations. Alternative payment models, such as the Centers for Medicare and Medicaid Services Oncology Care Model,^8,9 contain financial incentives to enhance clinic-based, ambulatory care systems and prevent avoidable hospitalizations. A number of cancer-focused delivery system innovations have shown promise for reducing hospitalizations during cancer treatment, including patient navigation programs,¹⁰ proactive telephonic nursing,¹¹ and electronic systems for between-visit symptom reporting.^6,12 Many of these interventions are resource intensive, and targeting outreach interventions to patients at high risk for hospitalization has been identified as a key strategy for preventing unplanned hospitalizations.¹³

Beyond the routine assessment of performance status, risk stratification tools for identifying patients at risk for hospitalization during cancer treatment are not commonly used. Hurria et al^14,15 created a risk stratification tool for identifying patients at increased risk for grade-3-or-higher chemotherapy toxicity, an outcome that can often lead to hospitalization. That risk stratification tool was developed in an elderly patient population and requires geriatric assessment input variables that are not routinely collected in the course of clinical care. Separately, a proof-of-concept study showed that a model using observable patient characteristics provided good discrimination in predicting risk for chemotherapy-related hospitalizations.¹⁶ Here we describe the development and validation of a suite of novel risk stratification models—relying exclusively on routinely collected clinical and administrative data—to identify patients with advanced cancer at increased risk for hospitalization after chemotherapy initiation. An accurate, validated model using routinely collect data has the potential for wide dissemination and application within electronic medical records of US health care systems.¹⁷

PATIENTS AND METHODS

Overview

We sought to develop a suite of clinical prediction models to evaluate the risk of hospitalization after chemotherapy initiation among patients with advanced solid tumor malignancies. The primary study outcome was all-cause hospitalization within 30 days of chemotherapy treatment, and we developed models for two overlapping prediction periods. The first prediction period was the 30 days after the initial chemotherapy treatment day (day 1). The second prediction period was a 30-day interval starting on the day of the first follow-up treatment visit occurring 15 to 22 days after the first treatment day (day N); in most cases, this was the second chemotherapy visit. Candidate predictors were informed by a prior study¹⁶ and were restricted to data elements initially recorded as part of routine health care delivery (in electronic health records [EHRs] and/or health plan administrative data). Study conduct and reporting adhered to the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) statement.¹⁸

Patients and Data Sources

We assembled two patient cohorts for this study; both cohorts comprised patients receiving their first chemotherapy treatment for stage IV or recurrent solid tumor malignancy. Patients were drawn from three Kaiser Permanente (KP) regional health systems that are founding members of the Cancer Research Network (CRN)¹⁹: Colorado (KPCO), Northwest (KPNW), and Washington (KPWA, formerly known as Group Health Cooperative). The CRN is a consortium of integrated health care systems affiliated with the National Cancer Institute.²⁰ Each of these autonomous, community-based health systems uses an Epic EHR. Although the regions share clinical guidelines, patient care is coordinated at the community level.²¹

Data used in this study were derived primarily from the CRN-supported Virtual Data Warehouse (VDW). The VDW contains administrative, EHR, and other clinical data that have been extracted at each site.^19,22 Cancer diagnoses were obtained from the Virtual Tumor Registry component of the VDW, which adheres to standards of the National Cancer Institute SEER program and the North American Association of Central Cancer Registries.²³ The KPCO and KPNW tumor registries (but not the KPWA registry) also collect information about cancer recurrence among patients initially diagnosed with American Joint Committee on Cancer stage I to III cancer. Oral and infused chemotherapy treatments were captured in the VDW infusion, pharmacy, and procedure files.²⁴ Institutional review board approval was obtained from all three sites.

The first of the two cohorts was used for model development and included patients from the KPCO and KPNW regions with stage IV or recurrent²⁵ cancer diagnosed between January 1, 2005, and December 31, 2013. We used a second, independent cohort for model validation; this cohort included patients from the KPWA region with stage IV cancer diagnosed between January 1, 2007, and October 31, 2016. Chemotherapy use was captured during the 180 days after cancer diagnosis.^26,27 Patients receiving chemotherapy for testicular cancer or chemoradiotherapy for head and neck cancer were excluded, because these treatments can be administered with curative intent, and our analyses were focused on patients receiving palliative-intent therapies. Patients receiving chemotherapy for all other solid tumor malignancies were included in cohort selection.

Candidate Predictors of Hospitalization

Candidate predictors included in model development were selected based on clinical expertise and prior evidence of their association with chemotherapy-related hospitalizations.¹⁶ The 15 candidate predictors were age, systolic blood pressure, diastolic blood pressure, pulse rate, creatinine clearance, sodium, calcium, albumin, total bilirubin, absolute neutrophil count, presence of leukopenia and/or thrombocytopenia, multiagent (v single-agent) chemotherapy, hospitalization in the 180 days before chemotherapy initiation (categorized as 0, 1, or ≥ 2), receipt of radiation therapy in the 30 days before chemotherapy initiation, and Charlson comorbidity score (calculated from VDW utilization data, excluding points for advanced cancer).^28-30 Values for laboratory and vital sign data were the most recent recorded results from on or before the reference day of chemotherapy treatment. For the development cohort, all laboratory and vital sign data were from within a maximum of 30 days before the treatment date. For the validation cohort, laboratory data were from within a maximum of 13 days and vital sign data were from within a maximum of 90 days before the reference treatment date. Creatinine clearance was calculated using the Cockcroft-Gault equation, using serum creatinine and measured patient weight.³¹ Presence of leukopenia and/or thrombocytopenia was defined as a platelet count of fewer than 150,000 cells per microliter and/or a WBC count of fewer than 3,800 cells per microliter; inclusion of this candidate predictor was based on the findings of a prior study.¹⁶

Model Development

After identifying eligible patients for the development cohort, we used multiple imputation (five imputed data sets) to develop complete data sets with no missing predictor values. The entire development cohort was used to model the risk of hospitalization in the first 30 days after chemotherapy initiation. Modeling for the second prediction period included the subset of patients from the development cohort who had a return visit for a second chemotherapy treatment between 15 and 22 days after the initial treatment date ("visit two"), without an intervening hospitalization. We examined several multivariable logistic regression models to calculate the predicted probability of hospitalization within 30 days of the reference chemotherapy treatment date for each of the two prediction periods (models of two to 15 variables). With the multivariable logistic regression models, we used the least absolute shrinkage and selection operator (LASSO) for variable selection,³² using one of the imputed data sets and employing a tuning parameter that maximized the 10-fold cross-validation estimate of the C-statistic. We then evaluated several unique models with five complete imputed data sets. After ranking models by their C-statistic, we selected three high-performing models for validation (two models from the first prediction period, and one model from the second prediction period). Because many models had similar discrimination characteristics, we used clinical judgment and the parsimony principle to select the final models for validation.

Model Validation

We used a complete case analysis for model validation, excluding patients from the external validation cohort if they had incomplete data for any of the predictor variables for the selected model. We applied the models fitted in the development cohort to the patients in the validation cohort, calculating the predicted 30-day hospitalization risk from each model. We evaluated three main measures of predictive performance: the C-statistic (ie, area under the curve), the observed hospitalization rate of patients above a high-risk cut point (positive predictive value), and the observed hospitalization rate of patients below the high-risk cut point (1-negative predictive value). We prospectively assigned the high-risk cut point as the risk threshold with 80% specificity in the development cohort. We additionally report the sensitivities and specificities observed in the validation cohort around this same cut point. We evaluated model calibration visually, using calibration plots.

RESULTS

The development cohort included 3,606 patients with a median age of 63 years; all patients were retained for model development, with imputation of missing data elements. The validation cohort included 634 evaluable patients with a median age of 64 years; an additional 387 eligible patients were excluded from the validation analysis because of missing data (complete case analysis). Lung cancer and colorectal cancer were the most common malignancies in both cohorts; platinums, taxanes, and fluoropyrimidines were the most common classes of chemotherapy agents. Additional demographic and clinical details of the study cohorts are listed in Table 1, and descriptive statistics for values of candidate predictor variables are listed in Table 2. The risk of hospitalization in the 30 days after chemotherapy day 1 was 14.6% in the development cohort and 14.7% in the validation cohort. The 30-day hospitalization risk after chemotherapy day N (where N is the day of the first treatment visit between day 15 and day 22) was 10.5% in the development cohort and 10.0% in the validation cohort.

TABLE 1.

Demographic and Clinical Characteristics of Patients Receiving Chemotherapy for Advanced Cancer

TABLE 2.

Descriptive Statistics for Candidate Predictor Variables

Model development yielded three candidate models for predicting the 30-day hospitalization risk. We selected two candidate models for the day 1 period (the 30 days after day 1) and one model from the day N period (the 30 days after day N). Variables for the selected models are listed in Table 3; the parsimonious day 1 model and the day N model both contained the same two predictor variables: blood albumin level and blood sodium level (both entered as continuous variables).

TABLE 3.

Logistic Regression Models to Predict Hospitalization Within 30 Days of Chemotherapy Delivery

Model discrimination for the two-variable (parsimonious) day 1 model yielded a C-statistic of 0.69 (95% CI, 0.62 to 0.75) when tested in the validation cohort. After dichotomization by the predefined risk cut point, 38.5% of patients were assigned to the high-risk group and 61.5% of patients were assigned to the low-risk group. The 30-day hospitalization risk was 24.2% (95% CI, 18.9% to 30.1%) in the high-risk group and 8.7% (95% CI, 6.1% to 12.0%) in the low-risk group (Table 4). Model calibration was satisfactory, as shown in the calibration plot in Figure 1; however, the model seemed to overestimate hospitalization risk at lower levels of predicted risk. The nine-variable day 1 model exhibited modestly improved discrimination and calibration compared with the two-variable model (Table 4; Fig 1), although the larger variable set for this model rendered 83 patients inevaluable because of missing data. The day N prediction model showed moderate to poor discrimination and calibration, with a C-statistic of 0.59 (Table 4; Fig 1).

TABLE 4.

Discrimination and Risk Stratification Performance of Models for Classification of Hospitalization Risk Within the Validation Cohort

FIG 1.

Calibration of models for classification of hospitalization risk within the validation cohort. (A) Day 1 parsimonious model; (B) day 1 expanded model; (C) day N model. Each point represents a nonoverlapping stratum of patients ordered by predicted risk for hospitalization. The diagonal line represents perfect calibration, where observed risk and predicted risk are identical. The day 1 parsimonious and expanded models predict risk of hospitalization in the 30 days after initial chemotherapy treatment (day 1). The day N model predicts risk of hospitalization in the 30 days following a subsequent treatment visit occurring 15 to 22 days after day 1.

DISCUSSION

We describe the development and validation of three prediction models for estimating the risk of hospitalization in patients starting chemotherapy for advanced cancer. Two models for predicting hospitalization risk in the 30 days after chemotherapy initiation showed good discrimination and calibration. A third model for predicting hospitalization risk after a subsequent chemotherapy treatment visit was less successful. These models were developed and validated in independent cohorts, and the two validated models can be implemented with use of data that are already routinely collected in the course of clinical care. As examples of the output of the two-variable day 1 parsimonious model, consider two hypothetical patients with metastatic colon cancer, both women in their 60s. For a patient with serum albumin and sodium in the normal range (albumin, 4.0 g/dL; sodium, 140 mmol/L), the model predicts a 30-day hospitalization risk of 12%. For a second patient with a serum albumin level of 3.0 g/dL and a sodium level of 136 mmol/L, the predicted hospitalization risk is 23%. An interactive Web application and a downloadable spreadsheet implementing these models are available online.³³

Because the objective of our analysis was risk stratification rather than biologic inference, we did not explore the mechanisms by which our predictor variables were related to hospitalization risk. However, the predictors identified in our models have good face validity as indicators of physiologic status and prognosis in cancer. Low albumin levels are a recognized marker of both poor nutritional status and systemic inflammation, and albumin levels have been associated with advanced cancer survival in a number of cancer types.^34,35 Mild to moderate hyponatremia (serum sodium < 136 or 131 mEq/L) has also been associated with adverse survival among patients with cancer.³⁶ Because albumin and sodium are associated with prognosis in cancer, and because they are routinely assessed and documented in EHRs during cancer treatment, these parameters are well suited for use in risk stratification models and clinical decision support tools.

Although the concept of identifying patients at risk for toxicity during cancer treatment is not new, our model is one of few tools to focus on hospitalization risk, rather than chemotherapy-related adverse events more generally. Examples of models for predicting chemotherapy toxicity include the Chemotherapy and Aging Research Group model, which uses 11 input variables to predict the risk of grade-3-or-higher toxicity among adults age 65 years or older (validation C-statistic, 0.65),¹⁵ and the CRASH (Chemotherapy Risk Assessment Scale for High-Age Patients) score, which uses two distinct four-variable models to predict the risk for hematologic and nonhematologic toxicities (independently) among adults ≥ 70 years of age (validation C-statistic, 0.65 for hematologic toxicity and 0.62 for nonhematologic toxicity).³⁷ Both of these toxicity models have been developed exclusively in elderly populations, and both require input data that are not otherwise collected as part of routine care. The PROACCT (PRediction Of Acute Care use during Cancer Treatment) model is a hospitalization model, using the same outcome of 30-day hospitalization that we report here. That model has been described in abstract form, using 11 variables and yielding a validation C-statistic of 0.65.³⁸ A model to evaluate for risk of hospitalization or emergency department visits during a course of radiation treatment (with or without concurrent chemotherapy) was recently reported, using a machine learning approach.³⁹ That model was developed at a single institution using a split-sample development-validation approach and reported a C-statistic of 0.80. Intriguingly, this model also identifies serum albumin as an important predictor of acute care during cancer treatment.

The models reported here compare favorably with the CRASH and PROACCT models; however, each of these models has different strengths and target populations. Although the use of routinely collected data is an advantage of our approach, collection of patient-reported data is becoming easier and more common in the age of the smartphone. As electronic symptom reporting enters routine practice, it seems likely that risk stratification models will be improved by including patient-reported data. The machine learning model for predicting acute care during radiation therapy is of considerable interest as well.³⁹ However, caution is warranted in confirming the generalizability of machine learning models in new clinical settings and across time.⁴⁰ Also, a black box problem can occur in machine learning models, where the contribution of individual predictors to a model prediction is opaque. This opacity can pose substantial implementation and interpretation challenges in real-world clinical settings. Regression models are relatively simple, transparent, and understandable and have a long history of use in medical research and practice.

What purpose can be served by models such as these? The most likely use for a model to predict hospitalization risk is to identify patients who may benefit from especially close follow-up after chemotherapy initiation. The reported success of both patient navigator programs¹⁰ and patient-reported outcome interventions^6,12 has shown that proactive checkups between oncology clinic visits can prevent (or reduce) hospitalizations, presumably through early identification and management of clinical deterioration. Patients with elevated hospitalization risk could also be referred for concurrent palliative care, another intervention with evidence for preventing hospitalizations.⁴¹ By pairing a risk stratification tool with these and other supportive care interventions, the resources of those interventions can be targeted where they are most likely to serve a benefit, enhancing the efficiency of care delivery.

A potential limitation to the generalizability of our findings is that we conducted our study among patients insured through integrated health plans in the United States, as distinct from patients with fee-for-service health insurance. Indeed, the 30-day hospitalization risk of approximately 15% that was seen in these cohorts is somewhat lower than the hospitalization risk observed in other studies, which has often exceeded 20% over the 30 days after chemotherapy initiation.^7,38 Therefore, it is likely that the risk threshold for separating high- and low-risk subgroups will need to be reevaluated in distinct clinical settings. Another limitation of our study is that our findings predate the age of immunotherapy. Checkpoint inhibitor immunotherapies are transforming advanced-cancer treatment, and these therapies carry adverse effect profiles distinct from those of traditional chemotherapy agents. Whether our models will retain their risk stratification properties in patients treated with checkpoint inhibitor therapies is unknown. However, cytotoxic chemotherapies remain a key component of treatment of many advanced cancers, including sometimes in combination with checkpoint inhibitors.

Given the limitations noted, we support efforts to validate these models in a range of care delivery settings and patient populations. Validation studies will be facilitated by the relative ease of use of our models, particularly the parsimonious two-variable model. Beyond additional validation efforts, cancer care delivery research will benefit greatly from studies that test the utility of validated risk stratification models for guiding the use of supportive care interventions. Until the findings of risk stratification tools can be shown to translate into tangible benefits for patients, such as fewer hospitalizations or reduced symptom burden, the potential of such tools will remain unfulfilled.

AUTHOR CONTRIBUTIONS

Conception and design: Gabriel A. Brooks, Alexander R. Menter, Debra P. Ritzwoller, Deborah Schrag

Financial support: Deborah Schrag

Administrative support: Deborah Schrag

Provision of study material or patients: Erin J. Aiello Bowles, Debra P. Ritzwoller, Deborah Schrag

Collection and assembly of data: Gabriel A. Brooks, Erin J. Aiello Bowles, Maureen O'Keefe-Rosetti, Debra P. Ritzwoller, Deborah Schrag

Data analysis and interpretation: Gabriel A. Brooks, Hajime Uno, Erin J. Aiello Bowles, Alexander R. Menter, Anna N.A. Tosteson, Deborah Schrag

Manuscript writing: All authors

Final approval of manuscript: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Gabriel A. Brooks

Consulting or Advisory Role: Abt Associates, CareCentrix

Hajime Uno

Consulting or Advisory Role: Shionogi, ION Pharma

Deborah Schrag

Consulting or Advisory Role: Journal of the American Medical Association

Research Funding: Pfizer

Other Relationship: Journal of the American Medical Association

No other potential conflicts of interest were reported.