Derivation and Validation of a Clinical Risk Assessment Model for Cancer-Associated Thrombosis in Two Unique US Health Care Systems

Abstract

PURPOSE

Venous thromboembolism (VTE), especially pulmonary embolism (PE) and lower extremity deep vein thrombosis (LE-DVT), is a serious and potentially preventable complication for patients with cancer undergoing systemic therapy.

METHODS

Using retrospective data from patients diagnosed with incident cancer from 2011-2020, we derived a parsimonious risk assessment model (RAM) using least absolute shrinkage and selection operator regression from the Harris Health System (HHS, n = 9,769) and externally validated it using the Veterans Affairs (VA) health care system (n = 79,517). Bootstrapped c statistics and calibration curves were used to assess external model discrimination and fit. Dichotomized risk strata using integer scores were created and compared against the Khorana score (KS).

RESULTS

Incident VTE and PE/LE-DVT at 6 months occurred in 590 (6.2%) and 437 (4.6%) patients in HHS and 4,027 (5.1%) and 3,331 (4.2%) patients in the VA health care system. Assessed at the time of systemic therapy initiation, the new RAM included components of the KS with the modified cancer subtype, cancer staging, systemic therapy class, history of VTE, history of paralysis/immobility, recent hospitalization, and Asian/Pacific Islander race. The c statistic was 0.71 in HHS and 0.68 in the VA health care system (compared with 0.65 and 0.60, respectively, for KS). Furthermore, the new RAM appropriately reclassified 28% of patients and increased the proportion of VTEs in the high-risk group from 37% to 68% in the validation data set.

CONCLUSION

The novel RAM stratified patients with cancer into a high-risk group with 8%-10% cumulative incidence of VTE and 7% PE/LE-DVT at 6 months (v 3% and 2%, respectively, in the low-risk group). The model had improved performance over the original KS and doubled the number of VTE events in the high-risk stratum. We encourage additional external validation from prospective studies.

INTRODUCTION

Venous thromboembolism (VTE), especially pulmonary embolism (PE) and lower extremity deep vein thrombosis (LE-DVT), is a serious and potentially preventable complication for patients with cancer undergoing systemic therapy.^1,2 It is associated with increased mortality and morbidity, including post-thrombotic syndrome with chronic pain, decreased performance status, prolonged hospitalization, and increased cost of care.^3-5 Randomized controlled trials have demonstrated that prophylactic administration of low-molecular-weight heparin or direct oral anticoagulants may significantly reduce the relative risk of VTE incidence in patients with cancer; nonetheless, the absolute risk reduction was appreciably higher in trials that selected patients with higher risk of VTE^6,7 than those with unselected patients.^8,9 Furthermore, both the efficacy-safety trade-off and cost-effectiveness are favored in patients with the highest VTE risk.^10,11 Therefore, adoption and implementation of pharmacologic thromboprophylaxis strategies in the real-world setting depends on a personalized yet automated and simple approach to accurately risk stratify patients for VTE across diverse populations.¹²

CONTEXT

• Key Objective

• Can we improve the existing clinical risk assessment model (RAM) for venous thromboembolism in patients with cancer receiving systemic therapy?

• Knowledge Generated

• Using retrospective data collected from approximately 90,000 patients with newly diagnosed cancer from two large health care systems in the United States, we performed the initial derivation and external validation study of a new RAM for venous thromboembolism that outperformed the Khorana score at the time of initial systemic therapy initiation. The novel RAM uses common data derivable directly from the electronic health record and works well in populations of differing age, sex, race, and ethnicity.

• Relevance (J.W. Friedberg)

• With further validation, the novel RAM may replace the Khorana score as the preferred thrombosis risk stratification strategy in patients with malignancy.*

• *Relevance section written by JCO Editor-in-Chief Jonathan W. Friedberg, MD.

The Khorana score (KS) is the most widely used risk assessment tool to predict incident VTE in patients with ambulatory cancer initiating chemotherapy.¹³ KS 2 or higher (2+) has been shown to be predictive of improved outcome with pharmacologic thromboprophylaxis in the aforementioned trials with a pooled 6-month VTE incidence of 9.23% in the placebo arms.^6,7,10 Nonetheless, a recent systematic review and meta-analysis of 34,555 patients showed that the overall proportion of VTE captured in the high-risk KS group was only 23%-55% depending on the cutoff (3+ v 2+).¹⁴ Furthermore, the KS is exclusively used in patients with solid tumors and lymphomas (excluding most hematologic malignancy) receiving chemotherapy (excluding modern noncytotoxic treatments). Various other models have been derived over the past decade, such as the Vienna CATS,¹⁵ PROTECHT,¹⁶ ONKOTEV,¹⁷ and COMPASS-CAT,¹⁸ although most have performed poorly in external cohort validations.¹⁹ Pabinger et al²⁰ recently published a simplified model with external validation; however, the use of D-dimer biomarker has precluded its wide adoption in the clinical setting. Finally, most of the available models have been derived in a predominantly White or European homogeneous population.

Recent advancement in clinical informatics, especially in enterprise data warehousing (EDW), machine learning, and natural language processing (NLP), has allowed the conversion of patient-level data into validated computable phenotypes for updated clinical risk assessment.²¹ In this study, we used cancer registry and EDW-linked data sets to derive and externally validate a simplified VTE risk assessment model (RAM) in patients with newly diagnosed cancer receiving systemic therapy.

METHODS

Study Design, Data Source, and Participants

We used retrospective data sets from two independent US health care systems to derive and externally validate the VTE RAM. The derivation data set consisted of patients with cancer from the Harris Health System (HHS), which is an integrated safety-net health care system with two medical centers and 18 outpatient clinics for patients from diverse racial and ethnic backgrounds in the Houston metropolitan area. The validation data set consisted of patients with cancer from the national Veterans Affairs (VA) health care system, which is the largest integrated health care system with 171 medical centers and 1,113 outpatient clinics for veterans. Patients have a high likelihood to receive uninterrupted oncologic care with longitudinal follow-up in these integrated systems.

Patient linkage, data harmonization, cohort exclusion, and variable extraction are described in detail in the Data Supplement (online only). Briefly, we created an integrated database that linked patients with first-ever invasive cancer from the HHS cancer registry with EPIC Caboodle EDW.²² Our external collaborators from the Massachusetts Veterans Epidemiology Research and Information Center used a similar approach to link eligible patients from the national VA Cancer Registry with the VA Corporate Data Warehouse.²³ Patients were excluded if they were not primary users of the health care system, did not receive systemic therapy within the first year of cancer diagnosis, received therapeutic anticoagulation, or had missing key baseline data at the time of initial therapy initiation (Fig 1). A list of 34 candidate predictors of VTE were defined to include existing KS components and cancer-specific and patient-specific factors (Data Supplement). The study was approved by the Institutional Review Board from Baylor College of Medicine and VA Boston Healthcare System. A waiver of informed consent from participants was granted by each Institutional Review Board.

FIG 1.

Patient selection and exclusion in derivation and validation cohorts. CR-DVT, catheter-related deep vein thrombosis; Hb, hemoglobin; HHS, Harris Health System; ICD, International Classification of Diseases; LE-DVT, lower extremity deep vein thrombosis; NLP, natural language processing; PE, pulmonary embolism; Plt, platelet; UE-DVT, upper extremity deep vein thrombosis; VA, Veterans Affairs; VTE, venous thromboembolism.

VTE Definition and Validation

The primary outcome was overall VTE defined as radiologically confirmed symptomatic or incidental PE, proximal or distal LE-DVT, or upper extremity DVT.²⁴ Secondary outcomes included PE/LE-DVT. Superficial venous thrombosis, cerebral venous thrombosis, and splanchnic venous thrombosis were excluded as events. Outcome was evaluated from the index date of systemic therapy initiation until the first outcome event, death, loss of follow-up defined as a 90-day gap without any clinical encounters, or administrative censoring on December 31, 2021. For the RAM derivation and validation, VTE outcomes were primarily evaluated at 6 months after therapy initiation.

The VTE ascertainment algorithm used a combination of structured data (International Classification of Diseases codes, medication) and unstructured data (NLP-extracted radiology impressions).²⁵ Study and site-specific validation for VTE is further detailed in the Data Supplement. The final computable phenotype algorithm had a sensitivity of 96% and a positive predictive value (PPV) of 98% at HHS and a sensitivity of 96% and a PPV of 91% at the VA health care system.

Statistical Analysis

Statistical analyses were performed independently by A.L. and D.G. using Stata/SE 16.1 in the derivation cohort and S.B.M. and J.L. using R 4.0.3/RStudio 1.4.1717 in the validation cohort. The investigators did not have access to the validation data set until the derivation model was finalized. Baseline covariates between the cohorts were compared using the standardized mean difference (SMD) where an SMD > 0.1 was considered significantly different.²⁶ Details on the model derivation and validation are given in the Data Supplement. Briefly, we first reduced the number of clinically plausible candidate covariates using the least absolute shrinkage and selection operator penalized regression.²⁷ We then fitted multivariable logistic regression models from the above covariates and kept those with odds ratios (OR) > 1.2 or < 0.8. Finally, we created simplified linear risk scores from the beta coefficient weights of the remaining covariates and the dichotomized risk group using 7%-8% overall VTE threshold as high-risk on the basis of a previous meta-analysis.²⁸

For external model validation, the final covariates were extracted from the VA database and assigned the same risk score (0- to 5+) and dichotomized risk groups. To account for censoring, bootstrapped time-dependent c statistic at 6 months²⁹ and calibration curves derived from predicted versus observed incidence at 6 months (with death as competing risk)³⁰ were used to assess model discrimination and fit, respectively. The new RAM was compared against the KS through c statistic increment and the concordance/reclassification table without the net reclassification index.^31,32 Subgroup analysis by age, sex, and race/ethnicity was performed.

The sample size of the study was based on availability of data; however, assuming 5% outcome prevalence, 80% power, and a sample size of 9,769, the detectable OR for binary predictors would be 1.28, 1.62, and 1.99 for common (50%, eg, sex), uncommon (5%, eg, Asian race), and rare (2%, eg, VTE history) predictors, respectively. Complete-case analysis without imputation was performed.

RESULTS

Study Population in Derivation and Validation Cohorts

After applying stringent inclusion and exclusion criteria, 9,769 and 79,517 patients with cancer diagnosed from 2011-2020 were included in the HHS derivation and VA health care system validation cohorts, respectively (Fig 1). The two cohorts had very different baseline characteristics with SMD > 0.1 for most baseline covariates (Table 1). Patients from HHS were younger (median age, 55 years) and diverse in sex (56.7% female) and race/ethnicity (15.1% White, 52.0% Hispanic, 26.4% Black, 5.6% Asian/Pacific Islander [API], American Indian, or Alaskan Natives). By contrast, patients from the VA health care system were older (median age, 67 years), predominantly male (4.9% female), and White (71.1% White, 5.3% Hispanic, 20.8% Black, 1.9% API). The cancer type distribution was significantly different. The HHS cohort had more patients with colorectal (11.8% v 7.6%), breast (22.3% v 2.8%), gynecologic (10.0% v 0.1%), and testicular (1.4% v 0.3%) cancers; sarcoma (1.7% v 0.4%); and aggressive lymphoma (5.5% v 3.0%), whereas the VA health care system cohort had more patients with lung (21.4% v 8.3%), prostate (16.6% v 5.8%), liver (5.1% v 2.7%), and bladder cancers (4.6% v 0.8%). For the first-line systemic therapy regimen, patients from HHS were more likely to receive chemotherapy (78.3% v 69.4%), whereas those from the VA health care system were more likely to receive targeted monotherapy (11.5% v 6.5%). Cancer staging (> 50% stage III-IV in both) and time to treatment initiation were not significantly different. Although more patients were recently hospitalized in HHS than the VA health care system (43.9% v 30.0%), the VA health care system patients had more concurrent comorbidities. Finally, although body mass index (BMI) and WBC were similarly distributed, significantly more patients from HHS than the VA health care system had hemoglobin (Hb) < 10 g/dL (20.9% v 12.2%) and platelet (Plt) ≥ 350 × 10⁹/L (27.6% v 14.2%). Consequently, more patients in HHS than the VA health care system had KS 3+ (14.3% v 7.1%).

TABLE 1.

Baseline Covariates in the Derivation and Validation Cohorts

The median follow-ups for continuous VTE assessment (censored if no clinical encounter for > 90 days) were 11.7 months (interquartile range [IQR], 5.8-21.2) in HHS and 14.7 months (IQR, 5.9-31.4) in the VA health care system. At 6-month post-treatment, there were 590 (incidence of 6.2%) VTE and 437 (4.6%) PE/LE-DVT in HHS and 4,027 (5.1%) VTE and 3,331 (4.2%) PE/LE-DVT in the VA health care system. The median follow-up for mortality assessment was 27.8 months (IQR, 11.3-58.3) in HHS and 26.8 (IQR, 9.4-60.1) in the VA health care system. At 6 months post-treatment, there were 1,062 (8.9%) deaths in HHS and 13,628 (17.1%) deaths in the VA health care system. A total of 1,069 (10.9%) and 3,781 (4.8%) patients had no evaluable data at 6 months because of loss to follow-up in the HHS and VA health care system, respectively.

Model Development and Performance

From the initial list of 34 candidate predictors of VTE, 11 covariates were selected in the final model (Table 2). The four-tier cancer subtype classification included a new intermediate-risk group for colorectal cancer; a modified high-risk group with aggressive non-Hodgkin lymphoma (instead of all lymphoma), ovarian/uterine cancer (instead of all gynecologic cancer), myeloma, brain cancer, and soft tissue sarcoma in addition to traditional KS cancer subtypes (lung, bladder, kidney, and testicular); and a modified very high-risk group with cholangiocarcinoma/gallbladder cancer in addition to traditional KS cancer subtypes (esophageal/gastric and pancreatic). The remaining solid and hematologic cancers were grouped as low-risk. When compared with low-risk, intermediate-, high-, and very high-risk cancer types were associated with ORs of 1.36 (95% CI, 1.01 to 1.82), 2.23 (95% CI, 1.81 to 2.74), and 2.26 (95% CI, 1.69 to 3.03) for 6-month VTE. Pretreatment BMI ≥ 35, WBC > 11, Hb < 10, and Plt ≥ 350 were associated with ORs of 1.45 (95% CI, 1.14 to 1.83), 1.34 (95% CI, 1.09 to 1.65), 1.49 (95% CI, 1.23 to 1.80), and 1.24 (95% CI, 1.03 to 1.49), respectively. The other four risk factors included cancer stage III-IV (OR, 1.33 [95% CI, 1.10 to 1.61] v stage I-II or unstaged), recent hospitalization > 3 days (OR, 1.52 [95% CI, 1.26 to 1.83]), history of VTE (OR, 1.67 [95% CI, 0.71 to 3.94]), and history of paralysis/immobility (2.22 [95% CI, 1.31 to 3.75]). Two protective factors included targeted/endocrine monotherapy (OR, 0.48 [95% CI, 0.34 to 0.67] v chemotherapy/immune checkpoint inhibitor [ICI]) and API (OR, 0.33 [95% CI, 0.18 to 0.61] v other races). The above covariates had similar magnitude and significance when we assessed the PE/LE-DVT outcome instead of overall VTE (Table 2).

TABLE 2.

Derivation of the New Risk Assessment Model for VTE at 6 Months After Systemic Therapy Initiation in the Harris Health System Cohort

From the final list of 11 predictors in the HHS derivation cohort, we created a new VTE risk score by adding weighted integer scores (median, 3 [IQR, 2-5]). In this final model, we further simplified the risk score into six groups (0– to 5+) with relatively even distribution of overall patients and incremental VTE events (Table 3 and Fig 2). The derivation c statistic of this simplified RAM was 0.71 (95% CI, 0.69 to 0.72) for overall VTE and 0.72 (95% CI, 0.70 to 0.74) for PE/LE-DVT. In the external validation, the patient group assignment and VTE increment were similar in the VA health care system cohort (Table 3). The external validation c statistic was 0.68 (95% CI, 0.67 to 0.69) for overall VTE and 0.68 (95% CI, 0.67 to 0.69) for PE/LE-DVT. As shown in the calibration curves, the external model calibration for both outcomes was adequate with no significant systemic deviations, except those with a risk score of 5 or higher had a slightly lower observation than predicted (Data Supplement).

TABLE 3.

Performance of the Simplified VTE Risk Assessment Model in Derivation and Validation Data Sets

FIG 2.

Incidence of VTE stratified by the new risk assessment model in derivation and validation cohorts. (A) Overall VTE and (B) PE/LE-DVT in Harris Health System derivation cohort; (C) overall VTE and (D) PE/LE-DVT in the Veterans Affairs health care system validation cohort. LE-DVT, lower extremity deep vein thrombosis; PE, pulmonary embolism; VTE, venous thromboembolism.

Model Risk Group and Comparison With the Existing Model

To mimic the KS classification implemented in previous clinical trials, we dichotomized the risk groups on the basis of a predetermined clinical threshold of 7%-8% overall VTE at 6 months, which corresponded to a score of 3+ for the high-risk group. In the derivation cohort, 49.2% of patients were classified as high-risk with 9.8% VTE incidence and 50.8% were classified as low-risk with 2.8% VTE incidence observed at 6 months. In the validation cohort, 44.8% of patients were classified as high-risk with 7.8% VTE incidence and 54.2% were classified as low-risk with 3.0% VTE incidence observed at 6 months (Data Supplement).

The new RAM grouping had improved discrimination and classification when compared with the KS with the 2+ cutoff. The c statistic for overall VTE prediction improved from 0.65 to 0.71 (increment of 0.06 at HHS) and from 0.60 to 0.68 (increment of 0.08 at the VA health care system; Data Supplement). Furthermore, the new risk score reclassified 22.2% (HHS) and 27.6% (VA health care system) of patients from the original KS risk groups into revised strata with better concordance with the observed VTE risk (Table 4). This reclassification increased the total proportion of potentially preventable VTEs in the high-risk group from 57.6% to 77.8% (20.2% increase at HHS) and 37.1% to 68.4% (31.3% increase at the VA health care system). The new RAM also stratified patients into distinct survival groups (Data Supplement). Finally, we tested the model's performance in demographic subgroups to ensure generalizability. The new RAM had similar discrimination in age, sex, and race/ethnicity subgroups in both HHS and VA health care system cohorts (Data Supplement).

TABLE 4.

Comparison of the New Risk Assessment Model With Original KS

DISCUSSION

Using independent data sets from two racially and ethnically diverse populations with cancer in the United States, we derived and externally validated an intuitive yet simple RAM for VTE. The model included the original KS components with revised cancer subtypes, two cancer-specific predictors (advanced stage and targeted/endocrine therapy), and four patient-specific predictors (history of PE/DVT, history of paralysis/immobility, recent hospitalization, and API). The new RAM stratified approximately 50% of patients with cancer types receiving modern systemic therapy into a high-risk group with a 6-month VTE risk of 8%-10% (7% PE/LE-DVT), and the other half into a low-risk group with a corresponding risk of 3% (2% PE/LE-DVT). Although this was the initial derivation/validation effort, the novel RAM appeared generalizable in different age, sex, and race/ethnicity subgroups.

When compared with the existing risk scores, there are several advantages in our improved RAM (Data Supplement). First, as a modified version of the original KS, the new model further clarified the cancer subtype definition and expanded the eligible population to include all cancer and therapy types. From the increased covariate granularity, we reclassified a quarter of patients in the KS prediction into more appropriate risk groups, thereby increasing the discrimination (c statistic increment of 0.08) and coverage (31% increase of potentially preventable VTE in the high-risk group) in the validation cohort. Second, the new RAM relied only on clinical predictors without specialized biomarkers. The simple additive score could be easily implemented and calculated in real time without requiring an external website or nomogram.

The risk predictors in the current VTE RAM are intuitive and consistent with the previously published literature. The binary classification of WBC, Hb, Plt, and BMI was the foundation of the original KS.¹³ The modification of cancer subtype risk with colorectal cancer,²⁰ sarcoma,²⁰ myeloma, brain cancers, and gallbladder/cholangiocarcinoma⁶ has been used in previous studies. History of VTE, advanced staging, and recent hospitalization have also been shown to be prognostic in similar previous studies,^17,18,33 while history of paralysis/immobility is a well-known predictor.^34,35 We chose to include API as a predictor because the reported cancer-associated VTE incidence was considerably lower in cohort studies in Asia^36,37 than comparable studies in Europe.^38,39 API as a predictor has also been incorporated into other cancer-specific VTE risk models.^40,41 Although we also observed variable VTE risks among Black patients and Hispanic patients compared with White patients,²² we chose not to include them in the final model as their definitions are imprecise and more likely to represent a social construct rather than continental origin. Nonetheless, the new model had similar performance in each race/ethnicity subgroup. Finally, the differential risk associated with chemotherapy and/or ICI versus targeted or endocrine monotherapy has not been systematically incorporated into RAMs although there have been several studies reporting the increased VTE risk associated with ICI.^42,43 In summary, we validated an explanatory RAM supported by both clinical knowledge and literature evidence.

It is important to consider limitations of the study. First, we realize that it is unusual to derive a RAM in a smaller cohort and validate it in a larger one; however, HHS had the gender and racial/ethnic diversity better suited for model derivation. Second, this was a retrospective cohort study with outcomes captured on the basis of computable phenotype algorithms. Although we used the same International Classification of Diseases–based algorithm for outcome ascertainment in both cohorts, the NLP text recognition algorithm was tailored to each health care system because of the differences in missing data and radiology dictation pattern. The final computable phenotype algorithm for VTE was independently validated using a subset of manually reviewed patients in both the derivation cohort (PPV 98%, sensitivity 96%)²⁵ and the validation cohort (PPV 91%, sensitivity 96%). Nonetheless, confirmation in a large prospective study that includes patients with all cancer types receiving all modern systemic therapies with prospectively captured VTE outcome data is recommended. Third, although the novel RAM included all cancer subtypes, including acute/chronic leukemia or myelodysplasia, there were relatively few patients in this category. Since many patients with hematologic malignancy have prolonged therapy-associated thrombocytopenia, the utility of any VTE RAM in this population remains unclear. Finally, our new RAM, like all others except the KS, has not been tested as a predictive biomarker for pharmacologic thromboprophylaxis.

In conclusion, we derived and externally validated a new clinical RAM for VTE in patients with cancer receiving modern systemic therapy. We found 11 clinically relevant risk predictors that could be extracted from retrospective analysis of integrated data warehouses to calculate a simple risk score that outperforms existing clinical models. We believe that this improved cancer-associated thrombosis model has the potential to improve patient selection for personalized thromboprophylaxis across diverse cancer patient populations.

AUTHOR CONTRIBUTIONS

Conception and design: Ang Li, Nathanael R. Fillmore

Collection and assembly of data: Jennifer La, Sarah B. May, Danielle Guffey

Data analysis and interpretation: All authors

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Derivation and Validation of a Clinical Risk Assessment Model for Cancer-Associated Thrombosis in Two Unique US Health Care Systems

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/jco/authors/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).