Predictive Modeling for Adverse Events and Risk Stratification Programs for People Receiving Cancer Treatment

Abstract

Emergency department visits and hospitalizations are common among people receiving cancer treatment, accounting for a large proportion of spending in oncology care and negatively affecting quality of life. As oncology care shifts toward value- and quality-based payment models, there is a need to develop interventions that can prevent these costly and low-value events among people receiving cancer treatment. Risk stratification programs have the potential to address this need and optimally would consist of three components: (1) a risk stratification algorithm that accurately identifies patients with modifiable risk(s), (2) intervention(s) that successfully reduce this risk, and (3) the ability to implement the risk algorithm and intervention(s) in an adaptable and sustainable way. Predictive modeling is a common method of risk stratification, and although a number of predictive models have been developed for use in oncology care, they have rarely been tested alongside corresponding interventions or developed with implementation in clinical practice as an explicit consideration. In this article, we review the available published predictive models for treatment-related toxicity or acute care events among people receiving cancer treatment and highlight challenges faced when attempting to use these models in practice. To move the field of risk-stratified oncology care forward, we argue that it is critical to evaluate predictive models alongside targeted interventions that address modifiable risks and to demonstrate that these two key components can be implemented within clinical practice to avoid unplanned acute care events among people receiving cancer treatment.

INTRODUCTION

People undergoing cancer treatment face risk of treatment-related toxicities that can lead to unplanned emergency department (ED) visits and inpatient hospital stays.¹ These acute care events account for a large proportion of spending in oncology care,² can lead to incomplete or delayed treatment, and negatively affect patients' quality of life. The ongoing shift toward value- or quality-based payment models within oncology has created a need to focus on delivering effective and efficient care, in part by reducing acute care utilization.^3-5 Risk stratification programs that identify patients at high risk for adverse treatment–related events and provide targeted interventions are an important strategy to reduce acute care utilization in oncology;⁶ however, the integration of such programs within care remains limited.

In this review, we argue that risk stratification programs should consist of three components: (1) a stratification algorithm that accurately identifies patients with modifiable risk(s), (2) intervention(s) that successfully reduce this risk, and (3) the ability to implement the risk algorithm and intervention(s) in an adaptable and sustainable way. Although multiple methods of risk stratification exist, the use of predictive modeling has rapidly gained popularity with a number of models proposed for use among people receiving cancer treatment.^7-16 These models, however, have often not been validated in practice, tested alongside corresponding interventions, or considered implementation challenges, hindering their incorporation into risk stratification programs. The aims of this paper are to (1) review published predictive models for adverse events among people receiving cancer treatment, (2) illustrate challenges faced when using these models in practice, and (3) identify the critical foundation needed to move the field of risk-stratified oncology care forward.

PREDICTIVE MODELING FOR ADVERSE EVENTS IN ONCOLOGY CARE

We identified published papers describing the development of 13 predictive models for adverse events during cancer treatment (Table 1). We included models predicting treatment toxicity or acute care use, including ED visits and hospitalizations. Models predicting other outcomes, such as mortality, or those including people with a cancer diagnosis but not currently receiving treatment were excluded. Included models share many similarities; however, they vary in target population and the nature and scope of included predictors (Table 2).

TABLE 1.

Published Predictive Models for Adverse Events Among People Receiving Active Cancer Treatment

TABLE 2.

Conceptual Domains of Variables Included in Final Predictive Models

Early models tended to focus on predicting treatment toxicity.^7-10 For example, Lyman et al⁷ developed a model to predict severe or febrile neutropenia during the first cycle of chemotherapy among patients with solid tumors or lymphoma. Their model included patient age, laboratory values, cancer type, and chemotherapy regimen features as predictors. Expanding on this model, the Chemo Toxicity Calculator developed by the Cancer and Aging Research Group (CARG) and the Chemotherapy Risk Assessment Scale for High-Age Patients (CRASH) were developed to predict risk of toxicity among patients with solid tumors receiving any line of chemotherapy and patients with any cancer diagnosis receiving first-line through fourth-line chemotherapy, respectively.^8,9,17 Unlike the model developed by Lyman et al, the CARG and CRASH models focused on older adults (age ≥ 65 and ≥ 70 years, respectively), predicted risk of any serious treatment toxicity over the full treatment course, and incorporated predictors from a geriatric assessment (eg, a brief nutritional assessment, the Mini-Mental Status Examination, and measures of instrumental activities of daily living). Each model performed well (ie, c-statistics ranging from 0.65⁹ and 0.83⁷), and each performed similarly well when externally validated in a second patient cohort. More recently, Feliu et al¹⁸ unsuccessfully attempted to predict risk of toxicity among patients with solid tumors age ≥ 70 years.

In general, predictive modeling among people undergoing cancer treatment has shifted from a focus on treatment toxicity to acute care events.^11-16 This shift corresponded with increased data availability via electronic health records (EHRs), incorporation of acute care events as a quality marker in oncology care,⁵ and recognition that not all high-grade toxicity is associated with equal clinical consequence.¹ As with toxicity-focused models, later models performed well in terms of predictive ability^11-16 and similarly varied in terms of target population, treatment type, predicted outcome, and number and type of predictors considered.

For example, models developed by Brooks et al and Feliu et al focused on patients with solid tumors receiving first-line palliative chemotherapy and any-line chemotherapy, respectively.^11,12,16 Hong et al¹³ included only people receiving outpatient radiation therapy. Conversely, the Prediction of Acute Care Use During Cancer Treatment (PROACCT) model¹⁴ and the model developed by Daly et al¹⁵ included patients with any cancer diagnosis receiving any line of antineoplastic therapy. The scope of therapies included also differed. The 2015 Brooks et al¹¹ model and Feliu et al¹⁶ model considered traditional chemotherapy only, whereas the PROACCT model also considered targeted therapy¹⁴ and the Daly et al¹⁵ model further considered immune therapy.

Although each model predicted acute care events, the inclusiveness of the outcome events considered has varied. The initial Brooks et al model¹¹ predicted only chemotherapy-related hospitalizations. The subsequent models from Brooks et al¹² and the Feliu et al¹⁶ model predicted all-cause hospitalization, whereas the PROACCT¹⁴ and Hong et al¹³ models expanded this further to include all-cause ED visits or hospitalization. More recently, the Daly et al¹⁵ model focused on potentially preventable ED visits or hospitalizations. The optimal outcome depends, in part, on the overall goal of the risk stratification program. Yet focusing on potentially preventable acute care visits,⁵ despite being aligned with current oncology quality measures, may not be as relevant to patients as all-cause acute care events.

Many models were developed using EHR data obtained from a single organization, thereby not accounting for events that occurred elsewhere. To minimize this limitation, the Daly et al¹⁵ model excluded patients who lived more than 30 minutes away from the institution. However, increased distance to care has been associated with negative patient outcomes,¹⁹ and thus, excluding more remote patients may limit the ability of predictive models to identify those at highest risk. The PROACCT model overcomes this limitation as it was created using linked population-based administrative and clinical databases in Canada that capture more than 95% of health care services provided,¹⁴ albeit raising questions of applicability to US-based populations who receive health care within a different context.

Regardless of the included patient population or predicted outcome, the authors describe model development strategies that varied along two dimensions: the number of predictors incorporated and the use of data from only structured EHR fields versus structured plus unstructured EHR fields and/or clinical assessments. For example, the Brooks et al¹² model and the PROACCT model¹⁴ included only two (ie, serum albumin and sodium levels) and three (ie, patient age, previous ED visits, and treatment regimen-tumor type) predictors, respectively, each of which was available from structured EHR fields. On the opposite end, the model developed by Daly et al¹⁵ used machine learning with natural language processing (NLP) to include numerous predictors obtained from both structured and unstructured EHR fields. Other predictive models^8,9 required data from clinical assessments that are not frequently performed nor documented within EHRs.

PRACTICE-BASED EXPERIENCE IN USING PREDICTIVE MODELS IN ONCOLOGY CARE

To our knowledge, among published predictive models for adverse events during cancer treatment, only the Daly et al¹⁵ model has been incorporated into clinical practice with a targeted intervention—and then only within the organization in which the model was developed.²⁰ Using this model, the top 25% of patients were categorized as high-risk and accounted for 35% of patients with potentially preventable acute care visits and 51% of potentially preventable inpatient bed days in the first 6 months of treatment.¹⁵ During the pilot program (InSight Care), patients identified as high-risk via either the predictive model or additional pre-established clinical criteria (eg, high-risk comorbidities, high psychosocial distress, or provider-identified barriers to care), were eligible for enrollment in a remote management system including daily symptom reporting.²⁰ The program provided a dedicated team of nurses and nurse practitioners for monitoring and managing patient-reported symptoms. A feasibility pilot study suggested that the program was acceptable to patients and clinicians.²⁰ Notably, despite the relative comprehensiveness of the predictors used, the majority (74%) of patients enrolled in the program were identified for inclusion not via the predictive model, but via clinician referral.

CHALLENGES IN USING PREDICTIVE MODELS FOR RISK STRATIFICATION

The InSight Care feasibility study²⁰ highlighted multiple challenges in using available predictive models in routine care. These challenges primarily center on practice integration and intervention tailoring.

Practice Integration

As evidenced by the diversity of predictors included within models (Table 3), the risk of toxicity and acute care events among oncology patients is driven by a complex interplay between clinical, psychosocial, and other factors.^21-24 Many of these factors, such as performance status and measures of frailty, are not routinely captured in a structured format within EHRs and are therefore difficult to extract automatically. Even when factors are captured in a structured format, there can be important data limitations. For example, medication and comorbidity information may be inaccurate or incomplete among patients receiving care across multiple institutions. Similarly, even when structured fields exist within EHRs, like in the case of cancer stage, if those fields are not routinely populated, then the ability to expeditiously use data in predictive modeling is limited.

TABLE 3.

Variables Included in Final Predictive Models Within Each Conceptual Domain

The results from the InSight Care pilot program illustrated that even with the use of NLP to extract unstructured data, including patient-reported outcomes (PROs) and information from social work and nursing notes, most patients enrolled in the program were not identified as high-risk via the predictive model, but via clinical judgment.²⁰ Further complicating practice integration is the likely trade-off between predictive model comprehensiveness and the ability to operationalize its use. Incorporation of new technology, like NLP, within the EHR is challenging, requiring technical expertise and multiple stakeholder buy-in. Arguably, simpler models may be more readily integrated but also met by more stakeholder skepticism, despite a lack of evidence that complex machine learning algorithms afford superior predictive ability.

Additional challenges include not only provider but also patient acceptability of such models, as well as the ability to keep models up to date regarding therapeutic advances. For example, the practice-based use of models that include detailed treatment regimen information, such as the PROACCT model¹⁴ and CRASH score,⁹ is hampered by the speed with which new agents become available and must be incorporated. The PROACCT model, which was published in 2019, did not include chemotherapies or targeted therapies introduced after 2015, or any immunotherapy. Updating models to include new treatments is a time-consuming task that requires multiple steps and considerably limits dissemination into practice.

Intervention Tailoring

As demonstrated by validation testing, predictive models can be highly accurate in identifying patients at risk of adverse outcomes. However, model-generated scores do not necessarily identify modifiable factors contributing to the patient's risk or the types of intervention(s) needed to modify those risks, nor do models guide the intensity or timing of intervention(s) or how that intensity may need to change over time as modifiable risk factors intensify or abate.

The InSight Care pilot program using the comprehensive Daly et al²⁰ model for risk stratification was used in conjunction with one reportedly similar intervention for all patients: intensive remote symptom monitoring. With so many differing sources of risk and the interaction(s) among these sources not fully understood, a more targeted approach focusing on individual patient modifiable risk factors might be needed. Risk stratification methods other than predictive models, such as the clinical criteria used for the InSight Care pilot program,²⁰ may be more transparent and thus more useful in identifying the appropriate intervention(s) for each patient. As the overall effectiveness of a risk stratification program is inherently linked not only to the ability of the predictive model (or other processes) to accurately identify those at high risk but also to the ability to intervene in ways that can effectively reduce those risks, it will be important to learn from this pilot and other ongoing work whether a one-size-fits-all or tailored intervention approach is more effective at reducing acute care events.

Just as individual patients may require different types of interventions, they may also require differing intensities of the same intervention. A common challenge for programs using PROs, including the InSight Care pilot program, is to determine the optimal frequency of patient assessments and communication of the results to oncology providers.²⁵ More frequent assessments may become burdensome to both patients and staff, resulting in decreased completion rates and alert fatigue. However, infrequent assessments may not detect symptoms early enough to allow time for mitigating intervention. Tailoring the intensity of interventions to individual patients and their changing risks over time not only may present additional implementation challenges but may also allocate intervention resources more efficiently across a larger patient population.

DISCUSSION

We found 13 published models predicting treatment-related toxicity or acute care events among people undergoing cancer treatment. These models incorporated a wide range of predictor variables, most commonly patient vitals or laboratory test results, treatment characteristics, and sociodemographic characteristics. Each model demonstrated comparable predictive accuracy and was advocated for use in risk stratification programs by its developers. Available models shared a general lack of consideration for implementation practicalities during development and, to our knowledge, apart from the Daly et al model, are yet to be evaluated when partnered with specific interventions. As a result, available models afford only one of the three components likely necessary for a successful oncology risk stratification program.

Despite calls for risk-stratified programming as a means of improving oncology care quality, to date, there has been a disconnect between predictive model development and intervention development and testing. A predictive model alone cannot alter a patient's risk or prevent adverse outcomes, rather these models must be combined with and evaluated alongside targeted interventions to address modifiable risk factors. Many different types of interventions exist that could address sources of risk⁶ including symptom monitoring,^26-29 clinical navigation and care coordination,^30-32 patient-centered medical homes,^33,34 health coaching,^35,36 integrated palliative care,³⁷ and multidisciplinary geriatric care.³⁸ Predictive models have been successfully combined with interventions in other areas of oncology care, including a recent randomized trial by Manz et al demonstrating that linking an algorithm estimating patient mortality with clinician-directed behavior nudges led to an increase in serious illness conversations.^39,40 This trial could serve as a model for future studies evaluating the use of predictive models with interventions designed to reduce acute care events.

Given the diversity of factors identified by existing predictive models as contributing to a patient's overall risk of adverse events, multiple types of interventions may be needed to prevent such events among patients with cancer. Importantly, the selected intervention(s) must complement the nature of the risk identified, the outcome predicted by the model, and reflect the overall goal of the risk stratification program. It may be that predictive models could be used as screening tools to identify high-risk patients with additional information collected from those patients to inform individual intervention selection. Predictive models could also be used alongside alternative methods of risk stratification, such as clinical criteria, which may more readily identify modifiable risk factors and needed intervention(s).

Beyond helping guide which patients are most in need of intervention, predictive models may guide the intensity of intervention(s). For example, high-risk patients could receive daily PRO assessments, whereas moderate-risk patients receive these weekly or at the start of each treatment cycle. Although symptom monitoring with PROs has gained popularity within oncology, to our knowledge, this is yet to be paired with risk stratification to tailor assessment frequency or inform which specific PRO measures are used. Practical implementation of PROs across oncology patients has been challenging thus far,^25,41 and pairing PROs with risk stratification may alleviate some of these challenges by reducing the frequency of data collection and reporting for lower-risk patients while maintaining high frequency for those most likely to benefit from close monitoring. Regardless of the specific process used, the goal of risk-stratified programming is not only to predict an event that is preventable and costly but also to intervene in a manner that alters the modifiable factors contributing to that risk, thereby simultaneously enhancing the quality of life of patients.

Although linkage of predictive models with evidence-based interventions is a pressing need to advance risk stratification programs, model improvement is also warranted. Existing models largely focus on patient-level clinical risk factors, rarely considering the broader context within which patients live and receive care. Yet multilevel factors, such as a patient's physical environment, social community, and access to care, drive health outcomes and need to be considered when both evaluating a patient's overall risk and designing accompanying interventions.^42-44 Future efforts could also be directed at developing adaptive risk models, such as that developed by Csik et al,⁴⁵ that can be applied throughout the course of treatment and that incorporate new information to provide a dynamic risk assessment. Doing so will require more comprehensive data systems that link patients' care over time and across institutions with important contextual information.

Finally, of equal importance to the predictive model and corresponding intervention(s) is the ability to implement both within clinical practice. This practical aspect of risk stratification programs has received arguably the least attention, yet it presents a significant hurdle to program success. There are numerous logistical considerations involved, including availability and accessibility of data used as model predictors, financial and personnel constraints, model or intervention acceptability, and incorporation of the predictive model and intervention(s) into existing workflows. Currently, some predictive models are available as online tools.^8,9,11,12 However, this requires a provider to manually find and enter patient data outside of the EHR to calculate estimated risk, a time-consuming and error-prone task. Integrating predictive models within EHRs is likely critical to increasing their functionality and use as part of risk stratification programs. Going forward, partnering with experts in implementation science and engaging patients and the diverse stakeholders knowledgeable of and responsible for intervention development and implementation are important strategies to overcome these barriers.

In conclusion, predictive models, when combined with appropriate interventions, offer potential benefits for oncology risk stratification programs, including improved patient experiences and outcomes as well as decreased costs. The use of predictive models and risk stratification programs in oncology care remains limited, primarily hampered by a lack of linkage with evidence-based interventions targeting modifiable risk factors and practical implementation challenges. To move the field forward, we advocate that predictive models are developed and tested alongside targeted interventions addressing modifiable risks within contexts that simultaneously consider practical implementation strategies.

AUTHOR CONTRIBUTIONS

Conception and design: All authors

Administrative support: Jennifer Elston Lafata

Collection and assembly of data: Chelsea K. Osterman, Megan Fasold, Jennifer Elston Lafata

Data analysis and interpretation: Chelsea K. Osterman, Hanna K. Sanoff, Megan Fasold, Jennifer Elston Lafata

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Predictive Modeling for Adverse Events and Risk Stratification Programs for People Receiving Cancer Treatment

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/op/authors/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).