Novel Application of Predictive Modeling: A Tailored Approach to Promoting HCC Surveillance in Patients With Cirrhosis

Abstract

OBJECTIVE:

There has been increased interest in interventions to promote hepatocellular carcinoma (HCC) surveillance given low utilization and high proportions of late stage detection. Accurate prediction of patients likely versus unlikely to respond to interventions could allow a cost-effective approach to outreach and facilitate targeting more intensive interventions to likely non-responders.

DESIGN:

We conducted a secondary analysis of a randomized clinical trial evaluating a mailed outreach strategy to promote HCC surveillance among 1200 cirrhosis patients at a safety-net health system between December 2014 and March 2017. We developed regularized logistic regression (RLR) and gradient boosting machine (GBM) algorithm models to predict surveillance completion during each of the 3 screening rounds in a training set (n = 960). Model performance was assessed using multiple performance metrics in an independent test set (n = 240).

RESULTS:

Among 1200 patients, surveillance was completed in 41–47% of patients over the three rounds. The RLR and GBM models demonstrated good discriminatory accuracy, with area under receiver operating characteristic (AUROC) curves of 0.67 and 0.66 respectively in the first surveillance round and improved to 0.77 by the third surveillance round after incorporating prior screening behavior as a feature. Additional performance characteristics including the Brier score, Hosmer-Lemeshow test and reliability diagrams were also evaluated. The most important variables for the predictive model were prior screening completion status and past primary care contact.

CONCLUSIONS:

Predictive models can help stratify patients’ likelihood to respond to surveillance outreach invitations, facilitating tailored strategies to maximize effectiveness and cost-effectiveness of HCC surveillance population health programs.

Hepatocellular carcinoma (HCC) is a leading cause of death in patients with compensated cirrhosis and fourth leading cause of cancer-related death worldwide.^1,2 In the United States, HCC is the fastest increasing cause of cancer-related death, with a disproportionate burden in racial/ethnic minorities and those of low socioeconomic status.^3,4 Despite improvements over time, most patients with HCC continue to be diagnosed beyond an early stage when median survival is 1–2 years.

Because of the association between early detection and improved survival, professional societies including the American Association for the Study of Liver Diseases recommend HCC surveillance in at-risk patients, including those with cirrhosis.^5–7 However, HCC surveillance is underused in clinical practice, with less than 1 in 4 patients with cirrhosis receiving any surveillance and lower proportions receiving semiannual surveillance.⁸ Survey studies among primary care providers have identified barriers including lack of knowledge about professional society recommendations and limited time in clinic with competing clinical concerns.^9,10 Similarly, patients with cirrhosis have reported barriers to surveillance including testing costs, transportation, and difficulty with scheduling processes.¹¹

These data highlight the need for interventions to increase HCC surveillance in patients with cirrhosis. Studies evaluating inreach interventions such as provider education or electronic medical record (EMR) reminders have demonstrated success in increasing HCC surveillance; however, these interventions still require a clinic visit to promote surveillance.^8,12 Therefore, there has been increasing interest in population-based outreach strategies to promote preventive care including hepatitis C, colorectal cancer, and HCC screening.^13–16 In fact, many large health systems such as Kaiser Permanente have implemented outreach strategies for colorectal cancer and other preventive care as part of routine practice.^16,17 Although these interventions can be effective for promoting screening completion in large populations of at-risk patients, they can be time-consuming and costly and may only be effective for certain subgroups, raising concerns about cost-effectiveness.¹⁸ Furthermore, continued mailings may not be the optimal strategy for patients who did not complete screening with prior outreach attempts; thus, it may be a more efficient use of resources and staff time to switch to more intense intervention strategies for hard-to-reach populations.¹⁹ If health systems were able to predict patients who were likely or unlikely to respond to outreach efforts, this could allow a more cost-effective approach to outreach as well as facilitate selection of more intensive interventions in likely non-responders.

Herein we leveraged a randomized clinical trial (RCT) of mailed outreach for HCC surveillance to determine whether predictive models could accurately identify responders and non-responders to surveillance invitations.

Methods

Study Population

As previously reported, we performed a pragmatic RCT from December 2014 to March 2017 comparing a mailed outreach strategy, with or without patient navigation, and usual care to increase HCC surveillance in patients with cirrhosis (NCT02312817).^13,20 The RCT was conducted at Parkland Health and Hospital System, the safety-net health system for Dallas County, which offers medical care at low cost for Dallas County residents. The RCT included patients with documented cirrhosis, defined by using International Classification of Diseases (ICD), 9th Revision codes, or suspected cirrhosis, defined as aspartate aminotransferase to platelet ratio index >1.0 in the presence of chronic liver disease.²¹ Patients were required to have ≥1 outpatient clinic visits in the year before enrollment. We excluded patients with Child C cirrhosis, HCC, or significant comorbid conditions; patients without known address or phone number; and language other than Spanish or English. The study was approved by the Institutional Review Board at University of Texas Southwestern Medical Center, with waiver of informed consent to avoid volunteer bias.

Surveillance Intervention

In brief, 1800 patients were randomized in a 1:1:1 ratio to receive usual care with visit-based screening (arm 1), mailed outreach invitations (arm 2), or mailed outreach plus patient navigation (arm 3). Outreach interventions included basic information regarding HCC risk and a recommendation to undergo surveillance, followed by reminder calls 2 weeks later. Patients in arms 2 and 3 received the outreach invitations every 6 months as well as opportunistic, visit-based screening via usual care. Patient navigation in arm 3 included motivational education for patients who declined surveillance. Because this study aimed to identify patients who were more or less likely to respond to outreach efforts, we limited analyses to patients in arms 2 and 3.

Data Collection

To ensure feasibility of applying our prediction model to other health systems, we abstracted candidate variables available in patients’ EMRs and outreach process outcomes. We focused on 6 domains: demographics (age, sex, race/ethnicity, and language), health care access (insurance status, primary care, and hepatology care), clinical liver disease history (etiology, cirrhosis severity, and comorbidity),^22,23 prior screening behavior, zip code–derived neighborhood-level factors, and outreach process outcomes. All patients were linked to zip code level features from the American Community Survey, including educational attainment, commute time, income, health insurance coverage, employment status, and population (Supplementary Material). Outreach process outcomes included outreach study arm assignment (arm 2 vs 3), returned outreach letters (eg, incorrect address), number of phone calls, call direction (incoming vs outgoing), and patient-reported interest during reminder call (eg, not interested, maybe later).

Model Development and Validation

Screening completion was defined as completion of abdominal imaging (ultrasound, contrast-enhanced computed tomography, or contrast-enhanced magnetic resonance imaging) in each of the 6-month surveillance rounds (rounds 1, 2, and 3). In addition to study arm (ie, arm 2 vs 3), we considered 5 sets of variables as defined above: (1) demographics, (2) clinical history, (3) health care access, (4) screening completion status in the prior period(s), and (5) zip code level neighborhood features.

Traditional logistic regression models have 2 limitations in this context: (1) a large set of predictors and interaction effects creates a large number of regression coefficients and overfits the data and (2) correlated predictors can hinder model performance. Hence, we first used a regularized logistic regression model (RLR), which reduces model complexity and number of regression coefficients to be estimated by maximizing the likelihood function using 2 regularization terms. The regularization terms constrain the RLR model so the likelihood function is maximized with the optimal level of complexity.

Although RLR is built on a single strong predictive model, another approach is an ensemble learning approach using multiple base models (ie, decision trees).^24,25 The gradient boosting machine (GBM) algorithm forms a boosting ensemble sequentially: first, the algorithm fits a decision tree (tree 1) to the data; second, it trains another decision tree (tree 2) to the residuals of the first tree; third, it fits a boosting ensemble that consists of trees 1 and 2 to the data and then repeats the process iteratively. Repetitive refinement of models based on the errors of prior modeling representations iteratively converges to an accurate final prediction.

We split the data into a training set that includes a random sample of 80% of patients and test set that includes the remaining 20%. To avoid overfitting, we used a 10-fold cross-validation procedure for hyperparameter tuning. First, we randomly partitioned the training set into 10 subsamples with equal size. We then successively held out 1 subsample for validation while fitting the model on the rest of 9 subsamples. Last, we selected the model with the best average performance.

For RLR, we trained the model for the best performance in the training set by searching for the optimal values of the regularization terms α (which controls the model type: eg, α = 1 for LASSO) and λ (which controls the amount of regularization applied to the model). For GBM, we trained the GBM models for the best performance in the training set by tuning all the hyper-parameters such as the maximum tree depth and the learning rate.

Model Performance Metrics

We evaluated model performance in the test set by using multiple performance metrics, including area under receiver operating characteristic curve (AUROC), Brier score, and logarithmic loss (Logloss). AUROC evaluates the model’s ability to distinguish between true positives and false positives. Unlike AUROC that evaluates the model’s ability to make a binary classification, the Brier score and Logloss evaluate an ability to predict class probabilities by measuring how close the predicted probabilities are to the true status. We used 1000 bootstrapped samples to obtain 95% confidence intervals (CIs) around all metrics. Calibration was assessed by examining (1) reliability diagrams, (2) changes in Logloss before and after calibration, and (3) Hosmer-Lemeshow test. We used Platt scaling to calibrate predicted probabilities of the classifiers. Specifically, we split the test set equally into a calibration set that is used to get the link function of correcting probabilities and a calibration test set that is used to get the final set of corrected probabilities. Finally, we obtained variable importance to identify variables that contributed most to model predictive accuracy. All analyses were done by using R version 3.6.1.

Results

Patient Characteristics

Characteristics of the 1200 patients (600 mailed outreach and 600 mailed outreach plus patient navigation) are detailed in Table 1. Mean age of patients was 55.4 years, and 481 (40.1%) were women. The sample was racially/ethnically diverse with 464 Hispanic (38.7%), 392 black (32.7%), and 328 white (27.3%). More than three fourths of patients (n = 961) had documented cirrhosis. Patients had a median of 4 primary care visits in the year preceding randomization; however, only 310 (25.8%) had hepatology care during that time. Patient characteristics appeared similar between training and test sets (Table 1).

Table 1.

Characteristics of Training and Test Sets

	Training set (N = 960)	Test set (N = 240)	Total (1200)
Patient-level characteristics
Median age (y, IQR)	56.2 (50.4–61.8)	55.1 (47.6–61.8)	56.1 (50.1–61.8)
Sex (% female)	381 (39.7)	100 (41.7)	481 (40.1)
Race/ethnicity
White	263 (27.4)	65 (27.1)	328 (27.3)
Hispanic	371 (38.6)	93 (38.8)	464 (38.7)
Black	315 (32.8)	77 (32.1)	392 (32.7)
Primary language (% English)	723 (75.3)	189 (78.8)	912 (76.0)
Insurance
Medicare	216 (22.5)	81 (33.8)	297 (24.8)
Medicaid	206 (21.5)	48 (20.0)	254 (21.2)
Commercial	30 (3.1)	7 (2.9)	37 (3.1)
Medical assistance	411 (42.8)	83 (34.6)	494 (41.7)
Self-pay	19 (2.0)	5 (2.1)	24 (2.0)
Unknown	78 (8.1)	16 (6.7)	94 (7.8)
Etiology of liver disease
Hepatitis C	472 (49.2)	126 (52.5)	598 (49.8)
Hepatitis B	26 (2.7)	15 (6.3)	41 (3.4)
Alcohol-related	179 (18.6)	40 (16.7)	219 (18.3)
Nonalcohol steatohepatitis	161 (16.8)	34 (14.2)	195 (16.3)
Documented cirrhosis (%)	757 (78.9)	204 (85.0)	961 (80.1)
Child-Pugh class (% Child A)	691 (72.0)	168 (70.0)	859 (71.6)
Charlson comorbidity index (IQR)	2 (1–4)	2 (1–4)	2 (1–3)
No. of primary care visitsa (IQR)	4.0 (2.0–7.0)	3.5 (2.0–7.0)	4.0 (2.0–7.0)
Receipt of hepatology carea	242 (25.2)	68 (28.3)	310 (25.8)
Receipt of patient navigation during outreach	486 (50.6)	114 (47.5)	600 (50.0)
Zip code-level characteristicsb
Education (% Bachelor or higher)	33.5 ± 6.9	34.1 ± 6.2	33.6 ± 6.8
Unemployment rate (%)	4.0 ± 0.4	4.0 ± 0.4	4.0 ± 0.4
Average commute time (min)	27.2 ± 2.2	27.0 ± 1.9	27.19 ± 2.1
Per capita income ($)	35,055 ± 4315	35,521 ± 3723	35,148 ± 4206
Insurance coverage
Private (%)	58.7 ± 7.2	58.7 ± 7.3	58.7 ± 7.2
Public (%)	28.7 ± 4.3	28.5 ± 4.1	28.6 ± 4.2

IQR, interquartile range.

^aIn year before randomization.

^bReported as mean ± standard deviation.

Surveillance Receipt

Receipt of HCC surveillance has been previously reported²⁰ and is illustrated in Figure 1. In brief, HCC surveillance was completed in 47.1%, 43.6%, and 41.0% for rounds 1, 2 and 3, respectively, for patients in the training set and 44.2%, 46.7%, and 43.3% for rounds 1, 2 and 3, respectively, for patients in the test set.

Figure 1.

Completion of HCC surveillance in each screening round.

Summary of Model Performance

Table 2 summarizes model performance for the RLR and GBM models in the test (ie, validation) set. First, both models had moderate discriminatory performance with AUROC scores ≥0.66 for round 1, with improvements over the next 2 rounds. Specifically, the RLR and GBM models had AUROC scores of 0.67 (95% CI, 0.67–0.67) and 0.66 (95% CI, 0.66–0.66), respectively, for round 1, 0.73 (95% CI, 0.72–0.73) and 0.73 (95% CI, 0.73–0.73), respectively, for round 2, and 0.77 (95% CI, 0.77–0.77) and 0.77 (95% CI, 0.77–0.78), respectively, for round 3. The RLR and GBM models had similar Brier scores of 0.19–0.23, with similar improvements over the 3 rounds. Finally, the 2 models had Logloss of 0.64 (95% CI, 0.64–0.64) and 0.65 (95% CI, 0.65–0.65), respectively, for round 1, 0.62 (95% CI, 0.62–0.62) and 0.62 (95% CI, 0.62–0.62), respectively, for round 2, and 0.58 (95% CI, 0.58–0.58) and 0.57 (95% CI, 0.57–0.57), respectively, for round 3.

Table 2.

Summary of Predictive Model Performance in Test Set

	Regularized logistic regression	Gradient boosting machines
Round 1
AUROC	0.67 (95% CI, 0.67–0.67)	0.66 (95% CI, 0.66–0.66)
Brier score	0.23 (95% CI, 0.23–0.23)	0.23 (95% CI, 0.23–0.23)
Logloss	0.64 (95% CI, 0.64–0.64)	0.65 (95% CI, 0.65–0.65)
Round 2
AUROC	0.73 (95% CI, 0.72–0.73)	0.73 (95% CI, 0.73–0.73)
Brier score	0.21 (95% CI, 0.21–0.21)	0.22 (95% CI, 0.22–0.22)
Logloss	0.62 (95% CI, 0.62–0.62)	0.62 (95% CI, 0.62–0.62)
Round 3
AUROC	0.77 (95% CI, 0.77–0.77)	0.77 (95% CI, 0.77–0.78)
Brier score	0.19 (95% CI, 0.19–0.19)	0.19 (95% CI, 0.19–0.19)
Logloss	0.58 (95% CI, 0.58–0.58)	0.57 (95% CI, 0.57–0.57)

AUROC, area under the receiver operating characteristic curve; CI, confidence interval; Logloss, logarithmic loss.

Calibration results are reported in Table 3. The RLR model was well-calibrated across all rounds, whereas the GBM model was not well-calibrated in round 3. Similarly, visual inspection of the reliability diagrams suggested RLR models were well-calibrated across all rounds, and the GBM model was well-calibrated in rounds 2 and 3, with the slope approaching 45° (Figure 2). Platt scaling minimally improved the goodness of fit for RLR and GBM models (Table 3) but did not reduce the Logloss of either model (Supplementary Table 1).

Table 3.

Summary of Calibration Results of the RLR and GBM Models in Test Set

	RLR	RLR Platt scaling	GBM	GBM Platt scaling
Round 1
H-L C-statistics	7.17	7.95	5.52	6.01
H-L C-statistic P value	.52	.44	.70	.65
Round 2
H-L C-statistics	10.13	9.40	15.39	15.53
H-L C-statistic P value	.26	.31	.05	.05
Round 3
H-L C-statistics	13.12	12.50	17.81	20.06
H-L C-statistic P value	.11	.13	.02	.01

GBM, gradient boosting machine; H-L, Hosmer-Lemeshow; RLR, regularized logistic regression model.

Figure 2.

Reliability diagrams for model calibration. (A) GBM model in round 1; (B) GBM model in round 2; (C) GBM model in round 3; (D) RLR model in round 1; (E) RLR model in round 2; (F) RLR model in round 3. All reliability diagrams were plotted using probabilities performed on the test set. For each diagram, patients were classified into 5 quartiles of predicted probability for screening completion, which was compared with observed screening completion. The 45° line delineates a perfectly calibrated model. GBM, gradient boosting machine; RLR, regularized logistic regression model.

Variable Importance

The most important variables, which are based on contribution to the predictive accuracy of the model, are shown in Figure 3. Prior screening completion status was the most important variable for predicting screening completion in rounds 2 or 3, particularly if initiated by the patient after receiving the mailed invitation. Otherwise, patient age, ethnicity, and primary care contact were important in predicting screening outcomes across all rounds, whereas gender, Charlson comorbidity index, insurance coverage, and receipt of hepatology care before randomization were only important in predicting screening outcomes in some rounds (see Supplementary Material for more details).

Figure 3.

Variable importance figures.

Discussion

In this secondary analysis of a RCT evaluating mailed outreach invitations for HCC surveillance, we evaluated machine learning models to predict surveillance completion. We found the 2 models performed similarly and had moderate discriminatory accuracy in round 1 but continued to improve over subsequent rounds with incorporation of prior screening completion behavior, achieving high discriminatory accuracy in the final round. Although several demographic factors such as age were determined to be important in the predictive models, health-seeking behaviors such as prior screening status and primary care contact were 2 of the most consistently observed.

In light of studies demonstrating HCC surveillance underuse,⁸ there is growing literature evaluating interventions to increase utilization; however, most studies have been single-center with limited sample sizes.^8,12,26,27 Therefore, studies have largely evaluated main intervention effects among patients, with limited power to identify subgroups who are more or less likely to respond. Conceptual models of health-seeking behaviors highlight the importance of multilevel influences including those at the individual patient (eg, attitudes, knowledge), organization (eg, delivery system design), and neighborhood.^28,29 In this study, we identified subgroups of patients who received mailed invites who were likely to complete screening and others who were unlikely to do so. These data can inform tailored intervention efforts, in which intensity is adapted to likelihood of response to maximize cost-effectiveness. For example, screening interventions may be unnecessary in patients who are likely to initiate screening via usual care and interventions can be intensified (eg, increased navigation or adding text message reminders) for those who are less likely to respond to mailed outreach alone. Interventions would be best targeted to those who are accepting of screening but unlikely to initiate without additional prompting or navigation to maximize cost-effectiveness of intervention strategies.³⁰ We have shown that a tailored outreach strategy requires higher initial investment for outreach infrastructure but can yield a net return of $500–$620 per patient compared with no outreach because of higher surveillance completion. This strategy may be of particular interest for resource-limited settings, such as safety-net health systems, or integrated health systems that also serve as insurers, such as the Department of Veterans Affairs.

We found the strongest predictor for future screening behavior was prior screening completion. Similar findings have been shown in other cancer screening programs such as colorectal and breast cancers, in which patients who fail to complete initial screening are unlikely to initiate in subsequent rounds.^19,31 Therefore, changing intervention strategy or intensity may be necessary to prompt a response in patients who do not respond to the first round of outreach interventions. Patient-reported barriers highlight the potential for more intensive patient navigation efforts or other stepped intervention approaches to further increase response, although this would need to be evaluated in future trials.^11,32,33

Beyond screening behavior, we found patients with high likelihood of response to outreach invitations were typically older and had more primary care or hepatology visits.³⁴ These factors may help determine initial likelihood of response to outreach invitations, but these simple factors only have moderate predictive accuracy if used in isolation, suggesting all patients may warrant an initial opportunity to respond to outreach. With improving model performance over subsequent rounds after incorporating other features, health systems could change intervention intensity with increased confidence in their ability to predict future behavior. Notably, model performance ranged from AUCs of 0.66 to 0.77 over the rounds, highlighting a need for well-conducted studies to further improve predictive models. For example, an approach to improve performance in year 1 would be to assess the value of including prior screening behavior, which was important in subsequent years.

We acknowledge our study had limitations. While leveraging a large RCT, the study was conducted at a single health system so may not be generalized to all patient populations.^35,36 Second, most predictors were recorded at baseline, and we did not incorporate time-varying covariates. Third, patients were identified by ICD-9/ICD-10 codes and aspartate aminotransferase to platelet ratio index, which have imperfect positive predictive values. Fourth, we could not externally validate the models because of the lack of other cohorts evaluating population-health outreach strategies for HCC surveillance. Fifth, our predictive model includes readily available variables from the EMRs and outreach databases but does not account for patient-level barriers that have been identified from survey and qualitative analyses.^11,37 Finally, all patients in the current study were receiving mailed outreach invitations, which are used by large health systems for other cancer screening programs (eg, colorectal cancer) but not currently widely used for HCC surveillance. However, we believe the study’s weaknesses are outweighed by notable strengths including its large sample size, racial/ethnic diversity, and novel application of machine learning models to predict response to screening invitation. Our study provides a proof-of-concept that predictive models leveraging available EMRs and outreach process data can facilitate tailored surveillance interventions.

In summary, we found predictive models can help stratify patients’ likelihood to respond to HCC surveillance invitations, facilitating tailored outreach invitation strategies to maximize effectiveness and cost-effectiveness.

What You Need to Know.

Background

Mailed outreach interventions can increase hepatocellular carcinoma (HCC) surveillance but may not be equally effective across all patient populations. Predicting patients who are likely or unlikely to respond to outreach efforts could facilitate selection of more intensive interventions in likely non-responders and a more cost-effective approach to outreach.

Findings

We derived and validated predictive models with high calibration and discriminatory accuracy for HCC surveillance completion. The most important variables for predicting response to surveillance invitations were prior screening completion and primary care contact.

Implications for patient care

Predicting likelihood of response to surveillance invitations can facilitate tailored strategies to maximize effectiveness of HCC surveillance population health programs.

Abbreviations used in this paper:

AUROC

area under the receiver operating curve

CI

confidence interval

EMR

electronic medical record

GBM

gradient boosting machine

HCC

hepatocellular carcinoma

ICD

International Classification of Diseases

Logloss

logarithmic loss

RCT

randomized clinical trial

RLR

regularized logistic regression model