Validation of the Hepatocellular Carcinoma Early detection Screening (HES) algorithm in a Cohort of Veterans with Cirrhosis

Abstract

Background & Aims:

Early detection of hepatocellular carcinoma (HCC) through surveillance reduces mortality associated with this cancer. Guidelines recommend HCC surveillance every 6 months for patients with cirrhosis, via ultrasonography, with or without measurement of serum level of alpha fetoprotein (AFP).

Methods:

We previously developed and internally validated an HCC early detection screening (HES) algorithm that included patient’s current level of AFP, rate of AFP change, age, level of alanine aminotransferase, and platelet count in a department of Veterans affairs (VA) cohort with active hepatitis C virus-related cirrhosis. HES score was associated with 3.84% absolute improvement in sensitivity of detection of HCC compared with AFP alone, at 90% specificity, within 6 months prior to diagnosis of this cancer. We externally validated the HES algorithm in a cohort of 38,431 patients with cirrhosis of any etiology evaluated at a VA medical center from 2010 through 2015.

Results:

A total of 4804 cases of HCC developed during a median follow-up time of 3.12 years. At 90% specificity, the HES algorithm identified patients with HCC with 52.56% sensitivity, compared to 48.13% sensitivity for the AFP assay alone, within 6 months prior to diagnosis; this was an absolute improvement of 4.43% (P<.0005). In HCC screening, a positive result leads to follow-up evaluation by computed tomography or magnetic resonance imaging. We estimated that the number of HCC cases detected per 1000 imaging analyses were 198.57 for the HES algorithm vs 185.52 for the AFP assay alone, or detection of 13 additional cases of HCC (P<.0005).

Conclusion:

We validated the HES algorithm in detection of HCC in patients with cirrhosis of any etiology evaluated at VA medical centers. The algorithm offers a modest but useful advantage over AFP alone in HCC surveillance.

The incidence of hepatocellular carcinoma (HCC) in the U.S. has tripled over the last 20 years; however, the prognosis of patients diagnosed with HCC has remained poor with overall 5-year survival less than 12%¹. Early detection of HCC through surveillance programs is a key component in reducing HCC mortality. Patients with advanced HCC have few treatment options, with 5-year survival between 0-10%, while those with early HCC can have potentially curative treatment including surgical resection and liver transplantation, with 5-year survival for patients receiving these treatments >60%².

Most HCC cases (80-90%) occur in patients with cirrhosis¹. The American Association for the Study of Liver Diseases (AASLD) recommends ultrasonography with or without serum α-Fetoprotein (AFP) every 6 months to screen for HCC in patients with cirrhosis³. However, ultrasound is operator dependent and difficult to perform in obese patients. AFP is widely used despite the wide variation in its reported performance⁴. The sensitivity of AFP for HCC screening varies (41%-65%) and the specificity is above 80% across a range of study designs (20ng/ml threshold)⁵. A systematic review found ultrasound with AFP had higher sensitivity for early HCC detection versus ultrasound alone, 63% vs. 45% respectively⁶. Methods to improve AFP performance, and hence potentially improve the performance in combination with ultrasound, are needed.

We have previously reported on the development and internal validation of an AFP-based algorithm in a retrospective cohort of patients with cirrhosis and active hepatitis C virus (HCV) in the national Department of Veterans Affairs (VA) Healthcare System. Richardson et al.⁷ found that in HCV-infected cirrhosis patients without HCC, elevated AFP was associated with elevated alanine aminotransferase (ALT). This motivated the development of the initial algorithm, which included current AFP, age, platelets, ALT, interaction terms (AFP and ALT, and AFP and platelets)⁸; and was later updated to include the rate of change from previous AFP value within the past year⁹. Our algorithm demonstrated sizeable improvement compared to AFP alone in predicting the 6-month HCC risk in our derivation sample, and also performed well in terms of calibration (i.e., agreement of model-derived HCC probabilities with raw HCC probabilities), discrimination (i.e., ability to separate HCC negative from HCC positive cases), and predictive ability in our split-sample analysis. In particular, the patient-level sensitivity within six months prior to HCC corresponding to 10% screening-level false positive rate (FPR) of the HES algorithm was 61.37%, a 3.84% absolute improvement over the sensitivity of AFP alone (57.53%)¹⁰. It is important to examine the performance of the HES algorithm in cirrhosis of any etiology in order to increase the clinical applicability and generalizability of the algorithm.

The VA is the largest integrated health-care provider in the U.S. and provides care to a large number of patients with cirrhosis with varying etiologies. In this analysis, we externally validate the HES algorithm in a large cohort of VA patients with cirrhosis of any etiology and test its ability to improve the likelihood of earlier HCC detection in clinical care settings.

Methods

Study Population

The study cohort included patients with cirrhosis of any etiology identified in the VA Corporate Data Warehouse (CDW), a national repository of VA clinical and administrative data from a network of 153 VA hospital facilities¹¹. Patients were eligible if they had a diagnosis of cirrhosis, evidenced by the presence of either International Classification of Diseases, 9th Revision (ICD-9) codes 571.2 or 571.5, between 10/1/1996 and 5/30/2015. This definition of cirrhosis has been validated against clinical, radiological, histological and biochemical criteria contained in the electronic medical records (EMR) and found to have an 90% and 87% positive and negative predictive values, respectively¹². The cirrhosis index date was the first appearance of ICD-9 codes for cirrhosis. Only patients with at least one valid AFP test after the cirrhosis index date and between 1/1/2010 and 5/30/2015 were included in the study cohort. Supplementary Figure 1 contains a detailed description of the cohort derivation.

We determined HCC diagnosis in the cirrhosis cohort using a sequential procedure. First, we identified patients with probable HCC via ICD-9 codes; defined as at least one inpatient or two outpatient 155.0 codes (but without 155.1)¹³. Next, we verified these HCC diagnoses by incorporating information from the VA Central Cancer Registry (VACCR) and the VA CDW oncology raw data files. The VACCR contains records of cancer cases treated at VA facilities with cancer registry activity and the data abstraction in the VACCR conforms to standards set by the North American Association of Central Cancer Registries¹¹. The VA CDW oncology raw data files are a component of the VA CDW raw domain that contain data tables extracted directly from the EMR for patients with a cancer diagnosis code¹⁴. A manual structured EMR review was performed in a random sample of patients whose HCC diagnosis was uncertain due to the presence of ICD-9 codes indicating HCC but an absence of records in both the VACCR and VA CDW oncology raw data files. We also performed an EMR review in all patients where the VA CDW oncology raw data files indicated cancers of the biliary tract but HCC diagnosis was unclear. In addition, the analysis cohort was restricted to include (1) HCC cases with at least one pre-diagnosis AFP test, (2) controls with at least one AFP test and a minimum of 12 months of follow-up to confirm no HCC. For both cases and controls, we only included AFP tests with ALT and platelet laboratory tests performed within 6 months prior to the AFP test.

Statistical methods

The study goal is to validate the HES algorithm that was developed and refined in a 1997-2005 HCV-related cirrhosis VA cohort^8–10. The inception version of the AFP-based algorithm combined current AFP, age, platelets, ALT values and interaction terms in a 6-month HCC risk prediction model⁸. Other laboratory tests such as aspartate aminotransferase, alkaline phosphatase, hemoglobin, white blood cells, albumin and bilirubin were also evaluated for inclusion and the selected model had the highest discrimination compared to alternatives⁸. The algorithm was subsequently updated to include the rate of change in AFP in the past year⁹; which was later allowed to be optional¹⁰ since not all patients would have an AFP test in the prior year. A cross-sectional re-sampling approach, repeated 100 times, was used to estimate the 6-month HCC risk. Additionally, a rule where a patient has a positive HCC screen if either AFP≥400ng/ml or the predicted HCC probability exceeded a threshold c, which was varied to generate the associated receiver operating characteristic (ROC) curve, was implemented. The threshold of 400ng/ml was chosen as it corresponded to a very low FPR (<1%) in the VA HCV-related cirrhosis cohort.

The patient-level sensitivity within T months was defined as the proportion of HCC cases with at least one positive screen within T months prior to clinical diagnosis, among those with testing in this period. It is reasonable to focus on positive screens in the pre-clinical diagnosis periods since they would likely have led to the earlier detection of HCC. The screening-level FPR was defined as the proportion of positive screening results in either controls or in cases more than T months prior to HCC diagnosis. The FPR was defined at the screening test-level because each false positive result leads to further costly testing and may increase the likelihood of complications and anxiety (See Additional Supplementary Materials for more information).

Bootstrap procedures were used to estimate confidence intervals and assess statistical significance. Specifically, 2000 bootstrap samples were generated using random sampling with replacement. The 95% bias-corrected bootstrap percentile confidence intervals were estimated for patient-level sensitivity at 10% screening-level FPR¹⁵. The bootstrap P-values were the proportion of bootstraps where the sensitivity at 10% screening-level FPR of AFP alone was greater than or equal to the HES algorithm¹⁶.

Model calibration was assessed graphically by plotting the predicted 6-month risk of HCC from the HES algorithm versus the raw probabilities of developing HCC in the next 6 months within risk deciles; the Hosmer-Lemeshow Χ² statistic was used to test for statistical significance. We employed two recalibration approaches, neither of which affected discrimination since they involve one-to-one transformations that maintain rankings. The first multiplied the predicted probabilities by a calibration factor, defined as a ratio of the observed rate of HCC in the next 6 months to the predicted 6-month risk of HCC, estimated in the cohort¹⁷. The second approach was regression-based and included the log-odds of the predicted 6-month risk of HCC ($\widehat{\delta }$) as the sole covariate in a generalized linear model with a logit link function for the binary outcome: HCC diagnosed within the next six months. We fit the model $\log \frac{\text{Pr}\left(\mathit{HCC}\mathit{in}6\mathit{months}\right)}{1-\text{Pr}\left(\mathit{HCC}\mathit{in}6\mathit{months}\right)}=\alpha +\beta \ast \widehat{\delta }$, where α=0 and β=1 would imply the predicted probabilities estimated in the HCV-related cirrhosis cohort were valid. However, if α and/or β significantly deviated from the ideal case, we used the estimated values to re-calibrate the predicted probabilities for the multi-etiology cohort. This approach maintains the relative effects of coefficients estimated in the HCV-related cirrhosis cohort while re-centering and scaling the log-odds to better fit a multi-etiology cohort.

We also compare the HES algorithm to using AFP alone with respect to (1) the probability of correct decision in the high risk group or equivalently the positive predictive value (PPV), (2) the probability of correct decision in the low risk group or equivalently the negative predictive value (NPV), (3) the probability of incorrect decision in the high risk group, (4) the probability of incorrect decision in the low risk group, (5) the number of HCC cases detected per 1000 computed tomography (CT) or magnetic resonance imaging (MRI), and (6) the number of CT/MRI needed to detect an HCC case. For each measure, we estimate the 95% bias-corrected bootstrap percentile confidence intervals and the bootstrap P-values were the proportion of bootstrap datasets where AFP alone has better or equivalent performance vs. the HES algorithm.

All analyses were performed using SAS Enteprise Guide v7.15 and R v3.3.2.

Results

The study cohort included 60,162 patients with cirrhosis of any etiology. We used ICD-9 codes, the VACCR and VA CDW oncology raw data files to classify 4289 patients as HCC cases, 54,094 as cirrhosis controls and exclude 261 patients with contradictory HCC diagnoses. We performed a manual structured EMR review in 167 patients where the oncology data indicated cancers of the biliary tract but the HCC diagnosis was unclear. We had 1801 patients with probable HCC based on ICD-9 codes but no records in either VACCR or the VA CDW oncology raw data files. We reviewed a random sample of 293 patients and found that 272 of 293 (93%) had confirmed HCC. We retained the remaining 1508 un-reviewed patients and evaluated the sensitivity of our results to their inclusion. Supplementary Figure 2 provides additional details on the sequential HCC classification procedure.

The inclusion criteria related to follow-up and concurrent measurement of laboratory tests were applied and Figure 1 describes the construction of the validation analysis cohort, which included 33,627 patients who did not develop HCC during follow-up (controls) and 4804 patients with incident HCC (cases). The median patient follow-up was 3.12 years (95% CI: 3.10-3.15) and the total patient-years of follow-up was 110,936. The annual HCC incidence rate was estimated to be 4.33%. Controls had an average of 2.63 AFP tests (standard deviation=2.17) and 59.77% of controls had more than one AFP test. HCC cases have an average of 2.85 (standard deviation=2.31) AFP tests prior to diagnosis and 62.51% of all HCC cases had more than one AFP test. The median age of patients at their first AFP test was 61 years (interquartile range: 56-64 years). Most patients were men (97.1%) and non-Hispanic whites (62.8%); a substantial proportion was black (20.5%). The etiological risk factors for cirrhosis included alcoholic liver disease (72.53%), HCV (51.15%), nonalcoholic fatty liver disease (16.18%), and hepatitis B virus (8.84%). Supplementary Table 1 includes descriptive statistics of the cohort, stratified by cirrhosis controls and HCC cases.

Figure 1:

Standards for Reporting of Diagnostic accuracy (STARD) flow diagram.

In Figure 2, we report the patient-level sensitivity at the threshold corresponding to 10% screening-level FPR. We evaluated the discrimination performance within 6, 12 and 24 months prior to HCC diagnosis and at any time prior to HCC diagnosis. We compared the HES algorithm to using AFP alone and observed statistically significant improvements in discrimination for our proposed algorithm. The HES algorithm improved sensitivity between 4.43-4.75 percentage points compared to using AFP alone over the range of T considered (p<0.0005). In particular, within six months prior to clinical diagnosis, the patient-level sensitivity corresponding to 10% screening-level FPR of the HES algorithm was 52.56% (95% CI: 50.36- 53.20); a 4.43 percentage point improvement compared to using AFP alone, 48.13% (95%CI: 46.48- 48.62) (p<0.0005).

Figure 2:

Patient-level sensitivity corresponding to 10% screening-level false positive rate for T=6, 12, and 24 months prior to HCC diagnosis and at any time prior to HCC diagnosis (Ever) and the associated 95% bootstrap percentile intervals.

We also evaluated the discrimination performance within non-overlapping intervals (0-6 months, 6-12 months, 12-24 months and >24 months) and report the results in Supplementary Table 2. The HES algorithm significantly improved sensitivity by 4.43, 2.75, and 2.01 percentage points compared to using AFP alone within 0-6 months, 6-12 months and 12-24 months prior to diagnosis, respectively (p<0.0005).

In a sensitivity analysis, we evaluated the performance of the HES algorithm when 1348 HCC cases with probable but unverified HCC were excluded; patient-level sensitivity was 3.78-4.54 percentage points higher for the HES algorithm vs. AFP alone (Table 1) (p<0.0005). We performed subgroup analyses and evaluated the HES algorithm in those with HCV only, alcoholic liver disease only, and in a group with either HCV or alcoholic liver disease (Supplementary Table 3). Across the subgroups, the HES algorithm significantly improved sensitivity vs. AFP alone.

Table 1:

Sensitivity analysis evaluating the HES algorithm when 1,358 unverified HCC cases were excluded.

Patient-level sensitivity corresponding to 10% screening- level false positive rate	3,456 HCC cases; 33,627 cirrhosis controls
	6 months	12 months	24 months	Ever
HES algorithm	52.30 (95% CI:50.32-53.17)	52.81 (95% CI:50.98-53.49)	53.88 (95% CI:52.51-54.49)	54.83 (95% CI:53.26-55.64)
AFP only	48.52 (95% CI:46.28-49.40)	48.54 (95% CI:46.63-49.04)	49.34 (95% CI:47.17-50.19)	50.46 (95% CI:48.46-51.39)
Bootstrap p-values:	<0.0005	<0.0005	<0.0005	<0.0005

The calibration of the HES algorithm within our multi-etiology cirrhosis cohort is displayed in Figure 3. We observed that the predicted probability of HCC underestimated the observed risk of HCC, at all risk deciles. The Hosmer-Lemeshow test statistic confirmed the poor fit of the predicted probability of HCC in the next 6 months to the observed probability (p-value<0.001). We considered a calibration factor procedure but this did not improve the model fit (p-value<0.001). Instead, we performed regression based recalibration and found that $\widehat{\alpha }=0.48$ (p-value <0.001) and $\widehat{\beta }=0.95$ (p-value=0.001) significantly deviated from the ideal case (α=0 and β=1). The resulting regression recalibrated predicted probability of HCC greatly improved fit, with the Hosmer-Lemeshow test statistic indicating no evidence of poor fit (p-value: 0.25).

Figure 3:

Evaluating calibration and recalibration of HES algorithm in a multi-etiology cohort.

We evaluated the consequences of a decision rule that classifies patients as high risk (positive screen) or low risk (negative screen) based on a threshold for the predicted probability of HCC in the next 6 months corresponding to an 10% screening-level FPR (Figure 4). High-risk patients are recommended to undergo additional screening via CT or MRI, while those classified as low risk are recommended to continue with 6-monthly HCC surveillance. The estimated PPV for HCC increased from 18.55% using AFP alone to 19.86% using the HES algorithm and the NPV from 97.56% to 97.72% (p-value<0.0005). These improvements correspond to increasing the number of HCC cases detected per 1000 CT/MRI from 185.52 to 198.57 (p-value<0.0005), or 13 additional HCC cases (7% increase). The estimated number of CT/MRI needed to detect an HCC case with AFP alone was 5.39, while the HES algorithm required 5.04 (p-value<0.0005). Therefore, the HES algorithm reduced the number of CT/MRI needed to detect 100 HCC cases by 35 compared to AFP alone.

Figure 4:

Evaluating the consequences of using the HES algorithm compared to using AFP alone in HCC screening.

Discussion

This study represents a crucial step in the validation of our HES algorithm for HCC detection. The model combines AFP with concomitant ALT, platelets and age to produce an estimate of the risk of developing HCC in the next six months. The HES algorithm was initially developed in cirrhosis patients with active HCV, where we observed the patient-level sensitivity corresponding to 10% screening-level FPR of the HES algorithm was 61.37%, a 3.84% increase over the usual practice (i.e., AFP only)¹⁰. In this study, we have validated the performance of the algorithm in a large contemporary cohort of patients with cirrhosis due to any etiology. The HES algorithm demonstrated statistically significant improvements against AFP alone, with patient-level sensitivity at any time prior to clinical diagnosis corresponding to 10% screening-level FPR of 52.56%, a 4.43% improvement compared to using AFP alone. In addition, the HES algorithm demonstrated clinically significant improvements against the usual practice with the estimated number of true HCC cases detected per 1000 CT/MRIs performed increasing from 185.52 to 198.57.

We observed that the predicted probability of HCC from the HES algorithm underestimated the observed risk of HCC in the next 6 months. The annual HCC incidence rate was 1.66% and 4.33% in the development HCV cohort and the multi-etiology cohort, respectively. Given the differences HCC incidence between the cohorts, it was not surprising that the HES algorithm required recalibration in the multi-etiology cirrhosis cohort.

The VA is the largest integrated health-care provider in the United States and hence we are able to assemble a large, multi-etiology cirrhosis cohort but this research approach does have its limitations. The HES algorithm may have limited generalizability to women and non-veteran populations. While some HCV patients from the development cohort may have survived, HCC-free and included in the current multi-etiology cohort, there was no overlap in the AFP tests included and hence we expect minimal impact on the external validation. In addition, we have limited information on HCC stage at diagnosis, with no staging information in almost 50% of HCC cases. We are currently further validating the HES algorithm in a non-VA cohort that will address many of the limitations of this current study. The cohort will be a more representative sample of the cirrhosis population in the United States and will include more complete information on HCC stage at diagnosis. We will examine both discrimination and calibration of the HES algorithm in this cohort, and in particular evaluate the performance of the HES algorithm in early stage HCC.

Our goal is to develop the HES algorithm that can replace AFP and be used in combination with ultrasonography to improve early detection. Studies show that among HCC patients in the U.S. with a known prior diagnosis of cirrhosis under regular surveillance, 52% received ultrasonography and AFP, 46% received AFP alone and 2% received ultrasonography alone⁴. While we do not advocate for screening with AFP alone, it is occurring. The current study provides evidence that in these patients, the HES algorithm could replace AFP at no cost with minimal additional effort for the clinician and result in modest improvements in the earlier detection of HCC.

In this study, we are unable to perform our key comparison of ultrasound and AFP versus ultrasound and the HES algorithm since the ultrasonography results in the VA EMR appear in free text format within radiology reports. In the absence of a validated automated data extraction procedure, we will need to use a manual structured review of radiology reports. The results from this study show the HES algorithm has improved performance versus AFP alone and provide evidence that constructing a cohort where this comparison is possible, is a worthwhile next step. This process is ongoing in the future non-VA validation cohort with a more manageable sample size. We will be able to evaluate whether the HES algorithm and ultrasound has improved performance over AFP and ultrasound. The HES algorithm could help prioritize ultrasound negative patients for costlier, less accessible imaging that are better able to detect early HCC. MRI was found to have significantly higher sensitivity compared to ultrasound for early HCC detection (83.7% vs. 25.6%), others have found no significant difference in CT and ultrasound⁶.

The added benefit of our HES algorithm (i.e., a 4.43 percentage point improvement corresponding to a relative increase of 9% in patient-level sensitivity and 13 additional HCC cases detected per 1000 CT/MRI corresponding to a 7% relative increase) is modest, but given that it is associated with virtually no added cost, we believe that it will be a useful tool in clinical practice. The framework of the algorithm, which consists of including demographic and clinical features to improve the performance of biomarkers, is innovative and has staying power. It is currently based on AFP since that is the only HCC biomarker recommended for use in HCC surveillance³. Des-γ carboxy prothrombin and lens culinaris agglutinin-reactive AFP are two serum biomarkers that have been evaluated in Phase-2 biomarker studies¹⁸ and recently approved by the Food and Drug Administration for assessing HCC risk. Our HES algorithm could be updated to include these two biomarkers as well.

Given the importance of early detection of HCC towards reducing high HCC mortality, further validation of the HES algorithm in a non-VA cirrhosis population and exploration of improvements to the algorithm, including updating the algorithm to incorporate the etiology of cirrhosis, are worthwhile next steps.

What You Need to Know.

Background: We externally validated an HCC early detection screening (HES) algorithm in a department of VA cohort with active HCV-related cirrhosis

Findings: The HES algorithm identified patients with HCC with 52.56% sensitivity, compared to 48.13% sensitivity for the AFP assay alone, within 6 months prior to diagnosis, at 90% specificity.

Implications for patient care: The HES algorithm incorporates readily available demographic and clinical information and could improve early detection with virtually no added cost.

List of Abbreviations:

AFP

α-Fetoprotein

ALT

alanine aminotransferase

CDW

corporate data warehouse

CT

computed tomography

EMR

electronic medical record

FPR

false positive rate

HCC

hepatocellular carcinoma

HCV

hepatitis C virus

HES

hepatocellular carcinoma early detection screening

ICD-9

international classification of diseases, 9th revision

MRI

magnetic resonance imaging

NPV

negative predictive value

PPV

positive predictive value

ROC curve

receiver operating characteristic curve

VA

U.S. Department of Veterans Affairs Healthcare System

VACCR

VA central cancer registry