Abstract
Background & Aims:
Early detection of hepatocellular carcinoma (HCC) through surveillance reduces mortality associated with this cancer. Guidelines recommend HCC surveillance every 6 months for patients with cirrhosis, via ultrasonography, with or without measurement of serum level of alpha fetoprotein (AFP).
Methods:
We previously developed and internally validated an HCC early detection screening (HES) algorithm that included patient’s current level of AFP, rate of AFP change, age, level of alanine aminotransferase, and platelet count in a department of Veterans affairs (VA) cohort with active hepatitis C virus-related cirrhosis. HES score was associated with 3.84% absolute improvement in sensitivity of detection of HCC compared with AFP alone, at 90% specificity, within 6 months prior to diagnosis of this cancer. We externally validated the HES algorithm in a cohort of 38,431 patients with cirrhosis of any etiology evaluated at a VA medical center from 2010 through 2015.
Results:
A total of 4804 cases of HCC developed during a median follow-up time of 3.12 years. At 90% specificity, the HES algorithm identified patients with HCC with 52.56% sensitivity, compared to 48.13% sensitivity for the AFP assay alone, within 6 months prior to diagnosis; this was an absolute improvement of 4.43% (P<.0005). In HCC screening, a positive result leads to follow-up evaluation by computed tomography or magnetic resonance imaging. We estimated that the number of HCC cases detected per 1000 imaging analyses were 198.57 for the HES algorithm vs 185.52 for the AFP assay alone, or detection of 13 additional cases of HCC (P<.0005).
Conclusion:
We validated the HES algorithm in detection of HCC in patients with cirrhosis of any etiology evaluated at VA medical centers. The algorithm offers a modest but useful advantage over AFP alone in HCC surveillance.
Keywords: ALT, HCV, Longitudinal biomarkers, Predictive modeling
The incidence of hepatocellular carcinoma (HCC) in the U.S. has tripled over the last 20 years; however, the prognosis of patients diagnosed with HCC has remained poor with overall 5-year survival less than 12%1. Early detection of HCC through surveillance programs is a key component in reducing HCC mortality. Patients with advanced HCC have few treatment options, with 5-year survival between 0-10%, while those with early HCC can have potentially curative treatment including surgical resection and liver transplantation, with 5-year survival for patients receiving these treatments >60%2.
Most HCC cases (80-90%) occur in patients with cirrhosis1. The American Association for the Study of Liver Diseases (AASLD) recommends ultrasonography with or without serum α-Fetoprotein (AFP) every 6 months to screen for HCC in patients with cirrhosis3. However, ultrasound is operator dependent and difficult to perform in obese patients. AFP is widely used despite the wide variation in its reported performance4. The sensitivity of AFP for HCC screening varies (41%-65%) and the specificity is above 80% across a range of study designs (20ng/ml threshold)5. A systematic review found ultrasound with AFP had higher sensitivity for early HCC detection versus ultrasound alone, 63% vs. 45% respectively6. Methods to improve AFP performance, and hence potentially improve the performance in combination with ultrasound, are needed.
We have previously reported on the development and internal validation of an AFP-based algorithm in a retrospective cohort of patients with cirrhosis and active hepatitis C virus (HCV) in the national Department of Veterans Affairs (VA) Healthcare System. Richardson et al.7 found that in HCV-infected cirrhosis patients without HCC, elevated AFP was associated with elevated alanine aminotransferase (ALT). This motivated the development of the initial algorithm, which included current AFP, age, platelets, ALT, interaction terms (AFP and ALT, and AFP and platelets)8; and was later updated to include the rate of change from previous AFP value within the past year9. Our algorithm demonstrated sizeable improvement compared to AFP alone in predicting the 6-month HCC risk in our derivation sample, and also performed well in terms of calibration (i.e., agreement of model-derived HCC probabilities with raw HCC probabilities), discrimination (i.e., ability to separate HCC negative from HCC positive cases), and predictive ability in our split-sample analysis. In particular, the patient-level sensitivity within six months prior to HCC corresponding to 10% screening-level false positive rate (FPR) of the HES algorithm was 61.37%, a 3.84% absolute improvement over the sensitivity of AFP alone (57.53%)10. It is important to examine the performance of the HES algorithm in cirrhosis of any etiology in order to increase the clinical applicability and generalizability of the algorithm.
The VA is the largest integrated health-care provider in the U.S. and provides care to a large number of patients with cirrhosis with varying etiologies. In this analysis, we externally validate the HES algorithm in a large cohort of VA patients with cirrhosis of any etiology and test its ability to improve the likelihood of earlier HCC detection in clinical care settings.
Methods
Study Population
The study cohort included patients with cirrhosis of any etiology identified in the VA Corporate Data Warehouse (CDW), a national repository of VA clinical and administrative data from a network of 153 VA hospital facilities11. Patients were eligible if they had a diagnosis of cirrhosis, evidenced by the presence of either International Classification of Diseases, 9th Revision (ICD-9) codes 571.2 or 571.5, between 10/1/1996 and 5/30/2015. This definition of cirrhosis has been validated against clinical, radiological, histological and biochemical criteria contained in the electronic medical records (EMR) and found to have an 90% and 87% positive and negative predictive values, respectively12. The cirrhosis index date was the first appearance of ICD-9 codes for cirrhosis. Only patients with at least one valid AFP test after the cirrhosis index date and between 1/1/2010 and 5/30/2015 were included in the study cohort. Supplementary Figure 1 contains a detailed description of the cohort derivation.
We determined HCC diagnosis in the cirrhosis cohort using a sequential procedure. First, we identified patients with probable HCC via ICD-9 codes; defined as at least one inpatient or two outpatient 155.0 codes (but without 155.1)13. Next, we verified these HCC diagnoses by incorporating information from the VA Central Cancer Registry (VACCR) and the VA CDW oncology raw data files. The VACCR contains records of cancer cases treated at VA facilities with cancer registry activity and the data abstraction in the VACCR conforms to standards set by the North American Association of Central Cancer Registries11. The VA CDW oncology raw data files are a component of the VA CDW raw domain that contain data tables extracted directly from the EMR for patients with a cancer diagnosis code14. A manual structured EMR review was performed in a random sample of patients whose HCC diagnosis was uncertain due to the presence of ICD-9 codes indicating HCC but an absence of records in both the VACCR and VA CDW oncology raw data files. We also performed an EMR review in all patients where the VA CDW oncology raw data files indicated cancers of the biliary tract but HCC diagnosis was unclear. In addition, the analysis cohort was restricted to include (1) HCC cases with at least one pre-diagnosis AFP test, (2) controls with at least one AFP test and a minimum of 12 months of follow-up to confirm no HCC. For both cases and controls, we only included AFP tests with ALT and platelet laboratory tests performed within 6 months prior to the AFP test.
Statistical methods
The study goal is to validate the HES algorithm that was developed and refined in a 1997-2005 HCV-related cirrhosis VA cohort8–10. The inception version of the AFP-based algorithm combined current AFP, age, platelets, ALT values and interaction terms in a 6-month HCC risk prediction model8. Other laboratory tests such as aspartate aminotransferase, alkaline phosphatase, hemoglobin, white blood cells, albumin and bilirubin were also evaluated for inclusion and the selected model had the highest discrimination compared to alternatives8. The algorithm was subsequently updated to include the rate of change in AFP in the past year9; which was later allowed to be optional10 since not all patients would have an AFP test in the prior year. A cross-sectional re-sampling approach, repeated 100 times, was used to estimate the 6-month HCC risk. Additionally, a rule where a patient has a positive HCC screen if either AFP≥400ng/ml or the predicted HCC probability exceeded a threshold c, which was varied to generate the associated receiver operating characteristic (ROC) curve, was implemented. The threshold of 400ng/ml was chosen as it corresponded to a very low FPR (<1%) in the VA HCV-related cirrhosis cohort.
The patient-level sensitivity within T months was defined as the proportion of HCC cases with at least one positive screen within T months prior to clinical diagnosis, among those with testing in this period. It is reasonable to focus on positive screens in the pre-clinical diagnosis periods since they would likely have led to the earlier detection of HCC. The screening-level FPR was defined as the proportion of positive screening results in either controls or in cases more than T months prior to HCC diagnosis. The FPR was defined at the screening test-level because each false positive result leads to further costly testing and may increase the likelihood of complications and anxiety (See Additional Supplementary Materials for more information).
Bootstrap procedures were used to estimate confidence intervals and assess statistical significance. Specifically, 2000 bootstrap samples were generated using random sampling with replacement. The 95% bias-corrected bootstrap percentile confidence intervals were estimated for patient-level sensitivity at 10% screening-level FPR15. The bootstrap P-values were the proportion of bootstraps where the sensitivity at 10% screening-level FPR of AFP alone was greater than or equal to the HES algorithm16.
Model calibration was assessed graphically by plotting the predicted 6-month risk of HCC from the HES algorithm versus the raw probabilities of developing HCC in the next 6 months within risk deciles; the Hosmer-Lemeshow Χ2 statistic was used to test for statistical significance. We employed two recalibration approaches, neither of which affected discrimination since they involve one-to-one transformations that maintain rankings. The first multiplied the predicted probabilities by a calibration factor, defined as a ratio of the observed rate of HCC in the next 6 months to the predicted 6-month risk of HCC, estimated in the cohort17. The second approach was regression-based and included the log-odds of the predicted 6-month risk of HCC () as the sole covariate in a generalized linear model with a logit link function for the binary outcome: HCC diagnosed within the next six months. We fit the model , where α=0 and β=1 would imply the predicted probabilities estimated in the HCV-related cirrhosis cohort were valid. However, if α and/or β significantly deviated from the ideal case, we used the estimated values to re-calibrate the predicted probabilities for the multi-etiology cohort. This approach maintains the relative effects of coefficients estimated in the HCV-related cirrhosis cohort while re-centering and scaling the log-odds to better fit a multi-etiology cohort.
We also compare the HES algorithm to using AFP alone with respect to (1) the probability of correct decision in the high risk group or equivalently the positive predictive value (PPV), (2) the probability of correct decision in the low risk group or equivalently the negative predictive value (NPV), (3) the probability of incorrect decision in the high risk group, (4) the probability of incorrect decision in the low risk group, (5) the number of HCC cases detected per 1000 computed tomography (CT) or magnetic resonance imaging (MRI), and (6) the number of CT/MRI needed to detect an HCC case. For each measure, we estimate the 95% bias-corrected bootstrap percentile confidence intervals and the bootstrap P-values were the proportion of bootstrap datasets where AFP alone has better or equivalent performance vs. the HES algorithm.
All analyses were performed using SAS Enteprise Guide v7.15 and R v3.3.2.
Results
The study cohort included 60,162 patients with cirrhosis of any etiology. We used ICD-9 codes, the VACCR and VA CDW oncology raw data files to classify 4289 patients as HCC cases, 54,094 as cirrhosis controls and exclude 261 patients with contradictory HCC diagnoses. We performed a manual structured EMR review in 167 patients where the oncology data indicated cancers of the biliary tract but the HCC diagnosis was unclear. We had 1801 patients with probable HCC based on ICD-9 codes but no records in either VACCR or the VA CDW oncology raw data files. We reviewed a random sample of 293 patients and found that 272 of 293 (93%) had confirmed HCC. We retained the remaining 1508 un-reviewed patients and evaluated the sensitivity of our results to their inclusion. Supplementary Figure 2 provides additional details on the sequential HCC classification procedure.
The inclusion criteria related to follow-up and concurrent measurement of laboratory tests were applied and Figure 1 describes the construction of the validation analysis cohort, which included 33,627 patients who did not develop HCC during follow-up (controls) and 4804 patients with incident HCC (cases). The median patient follow-up was 3.12 years (95% CI: 3.10-3.15) and the total patient-years of follow-up was 110,936. The annual HCC incidence rate was estimated to be 4.33%. Controls had an average of 2.63 AFP tests (standard deviation=2.17) and 59.77% of controls had more than one AFP test. HCC cases have an average of 2.85 (standard deviation=2.31) AFP tests prior to diagnosis and 62.51% of all HCC cases had more than one AFP test. The median age of patients at their first AFP test was 61 years (interquartile range: 56-64 years). Most patients were men (97.1%) and non-Hispanic whites (62.8%); a substantial proportion was black (20.5%). The etiological risk factors for cirrhosis included alcoholic liver disease (72.53%), HCV (51.15%), nonalcoholic fatty liver disease (16.18%), and hepatitis B virus (8.84%). Supplementary Table 1 includes descriptive statistics of the cohort, stratified by cirrhosis controls and HCC cases.
In Figure 2, we report the patient-level sensitivity at the threshold corresponding to 10% screening-level FPR. We evaluated the discrimination performance within 6, 12 and 24 months prior to HCC diagnosis and at any time prior to HCC diagnosis. We compared the HES algorithm to using AFP alone and observed statistically significant improvements in discrimination for our proposed algorithm. The HES algorithm improved sensitivity between 4.43-4.75 percentage points compared to using AFP alone over the range of T considered (p<0.0005). In particular, within six months prior to clinical diagnosis, the patient-level sensitivity corresponding to 10% screening-level FPR of the HES algorithm was 52.56% (95% CI: 50.36- 53.20); a 4.43 percentage point improvement compared to using AFP alone, 48.13% (95%CI: 46.48- 48.62) (p<0.0005).
We also evaluated the discrimination performance within non-overlapping intervals (0-6 months, 6-12 months, 12-24 months and >24 months) and report the results in Supplementary Table 2. The HES algorithm significantly improved sensitivity by 4.43, 2.75, and 2.01 percentage points compared to using AFP alone within 0-6 months, 6-12 months and 12-24 months prior to diagnosis, respectively (p<0.0005).
In a sensitivity analysis, we evaluated the performance of the HES algorithm when 1348 HCC cases with probable but unverified HCC were excluded; patient-level sensitivity was 3.78-4.54 percentage points higher for the HES algorithm vs. AFP alone (Table 1) (p<0.0005). We performed subgroup analyses and evaluated the HES algorithm in those with HCV only, alcoholic liver disease only, and in a group with either HCV or alcoholic liver disease (Supplementary Table 3). Across the subgroups, the HES algorithm significantly improved sensitivity vs. AFP alone.
Table 1:
Patient-level sensitivity corresponding to 10% screening- level false positive rate |
3,456 HCC cases; 33,627 cirrhosis controls | |||
---|---|---|---|---|
6 months | 12 months | 24 months | Ever | |
HES algorithm | 52.30 (95% CI:50.32-53.17) | 52.81 (95% CI:50.98-53.49) | 53.88 (95% CI:52.51-54.49) | 54.83 (95% CI:53.26-55.64) |
AFP only | 48.52 (95% CI:46.28-49.40) | 48.54 (95% CI:46.63-49.04) | 49.34 (95% CI:47.17-50.19) | 50.46 (95% CI:48.46-51.39) |
Bootstrap p-values: | <0.0005 | <0.0005 | <0.0005 | <0.0005 |
The calibration of the HES algorithm within our multi-etiology cirrhosis cohort is displayed in Figure 3. We observed that the predicted probability of HCC underestimated the observed risk of HCC, at all risk deciles. The Hosmer-Lemeshow test statistic confirmed the poor fit of the predicted probability of HCC in the next 6 months to the observed probability (p-value<0.001). We considered a calibration factor procedure but this did not improve the model fit (p-value<0.001). Instead, we performed regression based recalibration and found that (p-value <0.001) and (p-value=0.001) significantly deviated from the ideal case (α=0 and β=1). The resulting regression recalibrated predicted probability of HCC greatly improved fit, with the Hosmer-Lemeshow test statistic indicating no evidence of poor fit (p-value: 0.25).
We evaluated the consequences of a decision rule that classifies patients as high risk (positive screen) or low risk (negative screen) based on a threshold for the predicted probability of HCC in the next 6 months corresponding to an 10% screening-level FPR (Figure 4). High-risk patients are recommended to undergo additional screening via CT or MRI, while those classified as low risk are recommended to continue with 6-monthly HCC surveillance. The estimated PPV for HCC increased from 18.55% using AFP alone to 19.86% using the HES algorithm and the NPV from 97.56% to 97.72% (p-value<0.0005). These improvements correspond to increasing the number of HCC cases detected per 1000 CT/MRI from 185.52 to 198.57 (p-value<0.0005), or 13 additional HCC cases (7% increase). The estimated number of CT/MRI needed to detect an HCC case with AFP alone was 5.39, while the HES algorithm required 5.04 (p-value<0.0005). Therefore, the HES algorithm reduced the number of CT/MRI needed to detect 100 HCC cases by 35 compared to AFP alone.
Discussion
This study represents a crucial step in the validation of our HES algorithm for HCC detection. The model combines AFP with concomitant ALT, platelets and age to produce an estimate of the risk of developing HCC in the next six months. The HES algorithm was initially developed in cirrhosis patients with active HCV, where we observed the patient-level sensitivity corresponding to 10% screening-level FPR of the HES algorithm was 61.37%, a 3.84% increase over the usual practice (i.e., AFP only)10. In this study, we have validated the performance of the algorithm in a large contemporary cohort of patients with cirrhosis due to any etiology. The HES algorithm demonstrated statistically significant improvements against AFP alone, with patient-level sensitivity at any time prior to clinical diagnosis corresponding to 10% screening-level FPR of 52.56%, a 4.43% improvement compared to using AFP alone. In addition, the HES algorithm demonstrated clinically significant improvements against the usual practice with the estimated number of true HCC cases detected per 1000 CT/MRIs performed increasing from 185.52 to 198.57.
We observed that the predicted probability of HCC from the HES algorithm underestimated the observed risk of HCC in the next 6 months. The annual HCC incidence rate was 1.66% and 4.33% in the development HCV cohort and the multi-etiology cohort, respectively. Given the differences HCC incidence between the cohorts, it was not surprising that the HES algorithm required recalibration in the multi-etiology cirrhosis cohort.
The VA is the largest integrated health-care provider in the United States and hence we are able to assemble a large, multi-etiology cirrhosis cohort but this research approach does have its limitations. The HES algorithm may have limited generalizability to women and non-veteran populations. While some HCV patients from the development cohort may have survived, HCC-free and included in the current multi-etiology cohort, there was no overlap in the AFP tests included and hence we expect minimal impact on the external validation. In addition, we have limited information on HCC stage at diagnosis, with no staging information in almost 50% of HCC cases. We are currently further validating the HES algorithm in a non-VA cohort that will address many of the limitations of this current study. The cohort will be a more representative sample of the cirrhosis population in the United States and will include more complete information on HCC stage at diagnosis. We will examine both discrimination and calibration of the HES algorithm in this cohort, and in particular evaluate the performance of the HES algorithm in early stage HCC.
Our goal is to develop the HES algorithm that can replace AFP and be used in combination with ultrasonography to improve early detection. Studies show that among HCC patients in the U.S. with a known prior diagnosis of cirrhosis under regular surveillance, 52% received ultrasonography and AFP, 46% received AFP alone and 2% received ultrasonography alone4. While we do not advocate for screening with AFP alone, it is occurring. The current study provides evidence that in these patients, the HES algorithm could replace AFP at no cost with minimal additional effort for the clinician and result in modest improvements in the earlier detection of HCC.
In this study, we are unable to perform our key comparison of ultrasound and AFP versus ultrasound and the HES algorithm since the ultrasonography results in the VA EMR appear in free text format within radiology reports. In the absence of a validated automated data extraction procedure, we will need to use a manual structured review of radiology reports. The results from this study show the HES algorithm has improved performance versus AFP alone and provide evidence that constructing a cohort where this comparison is possible, is a worthwhile next step. This process is ongoing in the future non-VA validation cohort with a more manageable sample size. We will be able to evaluate whether the HES algorithm and ultrasound has improved performance over AFP and ultrasound. The HES algorithm could help prioritize ultrasound negative patients for costlier, less accessible imaging that are better able to detect early HCC. MRI was found to have significantly higher sensitivity compared to ultrasound for early HCC detection (83.7% vs. 25.6%), others have found no significant difference in CT and ultrasound6.
The added benefit of our HES algorithm (i.e., a 4.43 percentage point improvement corresponding to a relative increase of 9% in patient-level sensitivity and 13 additional HCC cases detected per 1000 CT/MRI corresponding to a 7% relative increase) is modest, but given that it is associated with virtually no added cost, we believe that it will be a useful tool in clinical practice. The framework of the algorithm, which consists of including demographic and clinical features to improve the performance of biomarkers, is innovative and has staying power. It is currently based on AFP since that is the only HCC biomarker recommended for use in HCC surveillance3. Des-γ carboxy prothrombin and lens culinaris agglutinin-reactive AFP are two serum biomarkers that have been evaluated in Phase-2 biomarker studies18 and recently approved by the Food and Drug Administration for assessing HCC risk. Our HES algorithm could be updated to include these two biomarkers as well.
Given the importance of early detection of HCC towards reducing high HCC mortality, further validation of the HES algorithm in a non-VA cirrhosis population and exploration of improvements to the algorithm, including updating the algorithm to incorporate the etiology of cirrhosis, are worthwhile next steps.
Supplementary Material
What You Need to Know.
Background: We externally validated an HCC early detection screening (HES) algorithm in a department of VA cohort with active HCV-related cirrhosis
Findings: The HES algorithm identified patients with HCC with 52.56% sensitivity, compared to 48.13% sensitivity for the AFP assay alone, within 6 months prior to diagnosis, at 90% specificity.
Implications for patient care: The HES algorithm incorporates readily available demographic and clinical information and could improve early detection with virtually no added cost.
Acknowledgments
Grant support:
This research was supported by an NIH NCI Grants (R01CA190776) and NIDDK (P30DK056338) to Dr El-Serag.
List of Abbreviations:
- AFP
α-Fetoprotein
- ALT
alanine aminotransferase
- CDW
corporate data warehouse
- CT
computed tomography
- EMR
electronic medical record
- FPR
false positive rate
- HCC
hepatocellular carcinoma
- HCV
hepatitis C virus
- HES
hepatocellular carcinoma early detection screening
- ICD-9
international classification of diseases, 9th revision
- MRI
magnetic resonance imaging
- NPV
negative predictive value
- PPV
positive predictive value
- ROC curve
receiver operating characteristic curve
- VA
U.S. Department of Veterans Affairs Healthcare System
- VACCR
VA central cancer registry
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Disclosures: All authors have no conflicts to disclose
Writing Assistance: None
References
- 1.El-Serag HB. Hepatocellular carcinoma. N Engl J Med. 2011;365(12):1118–1127. doi: 10.1056/NEJMra1001683. [DOI] [PubMed] [Google Scholar]
- 2.Bruix J, Sherman M, Practice Guidelines Committee AA for the S of LD. Management of hepatocellular carcinoma. Hepatology. 2005;42(5):1208–1236. doi: 10.1002/hep.20933. [DOI] [PubMed] [Google Scholar]
- 3.Heimbach JK, Kulik LM, Finn RS, et al. AASLD guidelines for the treatment of hepatocellular carcinoma. Hepatology. 2018;67(1):358–380. doi: 10.1002/hep.29086. [DOI] [PubMed] [Google Scholar]
- 4.Davila JA, Morgan RO, Richardson PA, Du XL, McGlynn KA, El-Serag HB. Use of surveillance for hepatocellular carcinoma among patients with cirrhosis in the United States. Hepatology. 2010;52(1):132–141. doi: 10.1002/hep.23615. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Gupta S, Bent S, Kohlwes J. Test characteristics of alpha-fetoprotein for detecting hepatocellular carcinoma in patients with hepatitis C. A systematic review and critical analysis. Ann Intern Med. 2003;139(1):46–50. http://www.ncbi.nlm.nih.gov/pubmed/12834318. [DOI] [PubMed] [Google Scholar]
- 6.Tzartzeva K, Obi J, Rich NE, et al. Surveillance Imaging and Alpha Fetoprotein for Early Detection of Hepatocellular Carcinoma in Patients With Cirrhosis: A Meta-analysis. Gastroenterology. 2018;154(6):1706–1718.e1. doi: 10.1053/j.gastro.2018.01.064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Richardson P, Duan Z, Kramer J, Davila JA, Tyson GL, El-Serag HB. Determinants of serum alpha-fetoprotein levels in hepatitis C-infected patients. Clin Gastroenterol Hepatol. 2012;10(4):428–433. doi: 10.1016/j.cgh.2011.11.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.El-Serag HB, Kanwal F, Davila JA, Kramer J, Richardson P. A new laboratory-based algorithm to predict development of hepatocellular carcinoma in patients with hepatitis C and cirrhosis. Gastroenterology. 2014;146(5):1249–55.e1. doi: 10.1053/j.gastro.2014.01.045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.White DL, Richardson P, Tayoub N, Davila JA, Kanwal F, El-Serag HB. The Updated Model: An Adjusted Serum Alpha-Fetoprotein-Based Algorithm for Hepatocellular Carcinoma Detection With Hepatitis C Virus-Related Cirrhosis. Gastroenterology. 2015;149(7):1986–1987. doi: 10.1053/j.gastro.2015.10.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Tayob N, Richardson P, White DL, et al. Evaluating screening approaches for hepatocellular carcinoma in a cohort of HCV related cirrhosis patients from the Veteran ’ s Affairs Health Care System. BMC Med Res Methodol. 2017;Under revi. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Zullig LL, Jackson GL, Dorn RA, et al. Cancer incidence among patients of the U.S. Veterans Affairs Health Care System. Mil Med. 2012;177(6):693–701. doi: 10.1016/j.biotechadv.2011.08.021.Secreted. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Kramer JR, Davila JA, Miller ED, Richardson P, Giordano TP, El-Serag HB. The validity of viral hepatitis and chronic liver disease diagnoses in Veterans Affairs administrative databases. Aliment Pharmacol Ther. 2008;27(3):274–282. doi: 10.1111/j.1365-2036.2007.03572.x [DOI] [PubMed] [Google Scholar]
- 13.Mittal S, El-Serag HB, Sada YH, et al. Hepatocellular Carcinoma in the Absence of Cirrhosis in United States Veterans is Associated With Nonalcoholic Fatty Liver Disease. Clin Gastroenterol Hepatol. 2016;14(1):124–31.e1. doi: 10.1016/j.cgh.2015.07.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Gonsoulin M. First Time Research Users’ Guide to CDW: Getting Started with this Relational Database. https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/862-notes.pdf. [Google Scholar]
- 15.Efron B and Tibshirani RJ. An Introduction to the Bootstrap. Taylor & Francis; 1994. [Google Scholar]
- 16.Tayob N, Lok ASF, Do K, Feng Z. Improved Detection of Hepatocellular Carcinoma by Using a Longitudinal Alpha-Fetoprotein Screening Algorithm. Clin Gastroenterol Hepatol. 2016;14(3):469–475.e2. doi: 10.1016/j.cgh.2015.07.049. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Pencina MJ, D’Agostino RB, D’Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157–72; discussion 207-12. doi: 10.1002/sim.2929. [DOI] [PubMed] [Google Scholar]
- 18.Marrero JA, Feng Z, Wang Y, et al. Alpha-fetoprotein, des-gamma carboxyprothrombin, and lectin-bound alpha-fetoprotein in early hepatocellular carcinoma. Gastroenterology. 2009;137(1):110–118. doi: 10.1053/j.gastro.2009.04.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.