Abstract
Several noninvasive indices have been proposed for predicting liver cirrhosis (LC), particularly in chronic hepatitis C (CHC). In this study, noninvasive indices for predicting LC and hepatocellular carcinoma (HCC) were compared. A total of 119 chronic hepatitis B (CHB) patients and 240 CHC patients were evaluated in a hospital‐based setting using various predictors for pathologic LC such as aspartate aminotransferase/alanine aminotransferase (AST/ALT) ratio (AAR), AAR‐to‐platelet ratio index (AARPRI), AST‐to‐platelet ratio index (APRI), age‐platelet (AP) index, and platelet counts. In addition, these indices were used to predict LC [based on ultrasound (US)] in a community‐based population of 201 patients with endemic hepatitis C virus (HCV). These indices were evaluated for their ability to predict HCC in CHB and CHC patients (n = 200). In CHB patients, the diagnostic performance of all indices was inadequate for predicting LC (areas under receiver operating characteristic curves < 0.7). Thrombocytopenia consistently demonstrated comparable accuracy to AARPRI ≥ 0.7 in CHB and AP index ≥ 7.0 in CHC patients. The best cut‐off values for APRI, AARPRI, and AP index in predicting LC in CHC were 1.3, 0.8, and 7.0, respectively. The best cut‐off values for APRI, AARPRI, and AP index in predicting LC (based on US) were 1.0, 1.2, and 8.0, respectively, in a HCV endemic community. An AAR > 1.4 might be a useful tool to identify candidates at high risk for HCC. In conclusion, platelet count was both consistent and accurate in predicting LC. An AAR > 1.4 is proposed as a possible surrogate marker for identifying patients at high risk for developing HCC.
Keywords: AAR‐to‐platelet ratio index (AARPRI), Hepatocellular carcinoma, Liver cirrhosis, Noninvasive indices, Platelet
Introduction
Chronic hepatitis B virus (HBV) or hepatitis C virus (HCV) infections cause liver fibrosis which often progresses to liver cirrhosis (LC) [[1], [2]]. Liver fibrosis is used as a parameter to guide antiviral treatment in chronic HBV and HCV infection. Current guidelines recommended liver biopsy before treatment in chronic hepatitis B patients with persistently elevated or intermittently abnormal aminotransferase levels and >105 copies/mL of HBV‐DNA [3]. In patients infected with HCV genotype 1, selected antiviral therapy should include liver fibrosis to evaluate for possible liver fibrosis [4].
Liver biopsy is the current gold standard for the detection of liver fibrosis with some limitations including inter‐ and intraobserver discrepancies and inadequate sample size leading to underestimation in some cases [[5], [6]]. This costly procedure has significant complications in approximately 0.6–5% of patients [[7], [8]]. Recently, elastography and other sophisticated indices such as FibroTest have been reported but are also expensive and not available in every hospital. Hence, a simple and noninvasive method for predicting liver fibrosis is needed. Among the reported noninvasive indices, aspartate aminotransferase/alanine aminotransferase (AST/ALT) ratio (AAR) [9], AST‐to‐platelet ratio index (APRI) [10], platelet count [[11], [12]], Pohl score [13], and age‐platelet (AP) index [14] are based on routinely available laboratory data. However, discrepant results have been reported depending on the patient populations selected [[15], [16]]. Although these indices have been validated by hospital‐based studies of chronic HCV patients before antiviral therapy, such reports may be flawed by selection bias. Few studies of noninvasive indices for chronic HBV patients are currently available [[12], [17], [18], [19]]. Moreover, the accuracy of noninvasive indices is currently unknown in community residents. A study comparing the accuracy of these indices in chronic HBV and HCV patients and community residents can help clarify the potential application and limitations of these indices in predicting LC.
Most hepatocellular carcinoma (HCC) patients in Taiwan and elsewhere have pre‐existing HBV and/or HCV infection [[20], [21]]. According to previous reports, most HBV‐ and HCV‐related HCC cases are diagnosed in cirrhotic patients [22]. Therefore, surveillance is recommended for chronic HBV and HCV patients [23], and limited HCC screening in cirrhotic patients may be cost effective. In our previous study, we proposed a two‐stage HCC screening program, that is, high‐risk identification by a noninvasive serum marker, such as thrombocytopenia, followed by ultrasound (US) screening [24]. Noninvasive indices for predicting LC and identifying candidates at high risk for HCC are needed as part of HCC surveillance.
This study evaluated several noninvasive indices used for predicting pathological LC in chronic HBV and HCV patients. In addition, we studied the use of these indices in diagnosing LC (based on US) in HCV endemic residents. We also compared the accuracy of these indices in predicting HCC development in patients with different viral etiologies.
Patients and methods
The study was performed in accordance with the current Declaration of Helsinki guidelines. The protocol was approved by the Institutional Review Board of Chang Gung Memorial Hospital, Taiwan. All patients gave their written informed consent prior to participation in the study.
The present study was divided into three parts: Study 1 examined consecutive hospital‐based patients to test for any correlation between noninvasive indices and pathological diagnosis of LC. Study 2 examined the validity of these indices in LC using US in a small‐scale community study. Study 3 was a hospital‐based study which determined the proportion of HCC patients identified by these indices based on their viral etiology.
Our previous study [24] revealed that the optimum cut‐off value for predicting LC using platelet counts was 150 × 109/L. At this cut‐off value, sensitivity was 52.5% and specificity was 78% in HBV patients. In HCV patients, sensitivity was 68.2% and specificity was 76.4% using the same cut‐off value. A total of 48% of HCC patients were thrombocytopenic.
In this report, data from the above studies were used to test the validity and usefulness of other noninvasive indices when compared to platelet counts. AST, ALT, and platelet counts were routinely determined in the clinical laboratory. The upper limit of normal (ULN) was 37 IU/L for AST and 40 IU/L for ALT. Laboratory parameters were available prior to liver biopsy. From these routine laboratory values, AAR, APRI, Pohl score, AP index, and platelet count were calculated as described previously [[9], [10], [13], [14]]. A new index, AAR‐to‐platelet ratio index (AARPRI) derived from FIB‐4 by removing age, was also proposed as a predictor of LC (Table 1).
Table 1.
Routine laboratory parameters used to calculate noninvasive indices of liver cirrhosis.
Noninvasive indices | Calculation | AST | AAR | Platelet | Age |
---|---|---|---|---|---|
AAR | AST/ALT | X | X | ||
Pohl score | Positive; AAR ≥ 1 and platelet ≤ 150 × 109/L | X | X | X | |
AARPRI | AAR/[platelet count (× 109/L)/150] | X | X | X | |
APRI | [(AST/ULN)/platelet (× 109/L)] × 100 | X | X | ||
AP index | Age (y): <30 = 0; 30–39 = 1; 40–49 = 2; 50–59 = 3; 60–69 = 4; >70 = 5 Platelet count (× 109/L): >225 = 0; 200–224 = 1; 175–199 = 2; 150–174 = 3; 125–149 = 4; <125 = 5 AP index is the sum of the above (possible value 0–10). | X | √ |
AAR = AST/ALT ratio; AARPRI = AAR‐to‐platelet ratio index; AP index = age‐platelet index; APRI = AST‐to‐platelet ratio index; ULN = upper limit of normal, X = including the factor.
Study 1. Correlation between noninvasive indices and pathologically diagnosed LC
All chronic HBV and HCV patients from 1998 to 2001, with available pathological diagnosis, were enrolled in this study. This study included 122 chronic HBV patients and 244 chronic HCV patients. Those with chronic HBV were positive for HBsAg (HBsAg V2, Abbott Laboratories Diagnostics Division, IL, USA). Chronic HCV patients were positive for anti‐HCV (HCV version 3.0, Abbott Laboratories Diagnostics Division, IL, USA). Patients with other concomitant causes of liver disease such as autoimmune hepatitis or history of alcohol abuse (exceeding 40 g/alcohol intake daily) were excluded from the study. All patients underwent liver needle biopsies (16‐gauge, modified Menghini needle; Hepafix; B. Braun Melsungen AG, Melsungen, Germany) before beginning antiviral treatment at Kaohsiung Chang Gung Memorial Hospital (KCGMH).
The mean length of the biopsy samples was1.8 ± 0.8 cm in chronic HBV patients and 1.7 ± 0.7 cm in chronic HCV patients. All histological grading [modified hepatic activity index (HAI)] and staging (fibrosis score) were performed by an experienced pathologist without knowledge of clinical details or ultrasonographic findings [25]. Laboratory parameters were available within 2 weeks prior to liver biopsy. Three chronic HBV patients and four chronic HCV patients with incomplete AST or ALT data were excluded from statistical analysis.
Study 2. Validation of noninvasive indices for predicting US LC in a HCV‐endemic area
A total of 201 residents aged 40 years or older were enrolled in this study. All individuals had participated in the Adult Preventive Health Examination in 2001. All HBsAg, anti‐HCV, AST, ALT, α‐fetoprotein (AFP), and complete blood count tests as well as an upper abdominal US were performed at the Min‐Sheng Clinic, which is located in Lieujia Township. The prevalence rates for HBsAg, anti‐HCV, both HBsAg and anti‐HCV, and neither were 9.0%, 37.3%, 3.5% and 50.2%, respectively. The US machines used were Toshiba SSA‐220A or SSA‐240A with 3.75‐MHz convex probes (Toshiba, Tokyo, Japan) [25]. Severity of liver parenchymal disease was scored using a simplified scoring system. Selected indicators were change of angle and edge (0, neither; 1, either; 2, both), coarseness of liver parenchyma (0, normal; 1, mild; 2, definite), and splenomegaly (0, none; 1, slight; 2, definite). The scoring system ranged from 0 to 6, and scores of 5 or 6 were defined as cirrhosis [26]. The US examinations were performed by five experienced gastroenterologists. Prescreening training and postscreening reviews were conducted to minimize intraobserver variation and Kappa coefficient showed good agreement between examiners. AAR, APRI, AARPRI, Pohl score, AP index, and platelet count were used to predict the diagnosis of LC based on US.
Study 3. Noninvasive markers coverage rate among HCC patients
Study 3 compared the accuracy of noninvasive indices including AAR, AARPRI, APRI, AP index, and platelet count in predicting HCC. From 2006 to 2007, all KCGMH consecutive patients with confirmed HCC by liver biopsy or with LC without HCC as confirmed by liver biopsy were recruited for analysis. AST, ALT, platelet counts, HBsAg, and anti‐HCV were determined before biopsy. Patients with unavailable AST, ALT, or platelet count data were excluded from analysis.
Statistical analysis
Continuous variables were expressed as mean ± standard deviation (SD). Independent samples were compared by Student t test or one‐way analysis of variance. The contrast factor was applied in a one‐way analysis of variance to test for linear trends. The sensitivity and specificity of continuous variables for diagnosing LC were expressed by the receiver operating characteristic (ROC) curve. The distance between the sensitivity and specificity of each point and the ideal point was calculated as distance = [(1 – sensitivity)2 + (1 – specificity)2]1/2. The best cut‐off point was the point with the shortest distance or maximum accuracy. The cut‐off value was determined based on the level of validity required. The difference between two ROC curves was expressed as the difference between the areas under the ROC curves (AUC) [[27], [28]]. Diagnostic performance was determined by the AUC of each index. Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy were calculated for each cut‐off level.
The stratum‐specific likelihood ratio (SSLR) is the probability of a test result within a given range or stratum when disease is present, divided by the probability of the same test result when disease is absent. We determined the SSLR by means of the formula SSLR = (x1/n1)/(x0/n0), where x1 is the number of patients in the stratum with HCC, n1 is the total number with HCC, x0 is the number of patients in the stratum without HCC, and n0 is the total number of patients without HCC [29].
Results
Clinical characteristics of all populations
In total, 119 chronic HBV patients and 240 chronic HCV patients who had undergone liver biopsy at KCGMH were enrolled in Study 1. A total of 201 patients who had received community adult health examination were enrolled in Study 2. Table 2 shows baseline clinical characteristics for all three populations. Hospital‐based patients were younger than community residents (p < 0.001). AST and ALT levels were higher in chronic HBV and chronic HCV patients compared with the community group (p < 0.001). The prevalence of cirrhosis was also higher in chronic HBV (32.5%) and chronic HCV (27%) patients than in the community group (10.4%). In the hospital group, the mean age was lower in chronic HBV patients (39 ± 12 years) compared with chronic HCV patients (50 ± 12 years) (p < 0.001).
Table 2.
Clinical characteristics of hospitalized chronic hepatitis B and C patients and HCV‐endemic community residents.
Variable | CHB (n = 119) (Mean ± SD) | CHC (n = 240) (Mean ± SD) | Community (n = 201) (Mean ± SD) |
---|---|---|---|
Age (y) | 39 ± 12 a | 50 ± 12 a | 59 ± 12 b |
Male, n (%) | 98 (81) | 147 (60) | 75 (37.3) |
AST (IU/L) | 86 ± 106 | 154 ± 126 | 33 ± 26 b |
ALT (IU/L) | 194 ± 502 | 97 ± 72 | 34 ± 36 c |
Platelet count (× 109/L) | 178 ± 59 | 173 ± 62 | 202 ± 67 d |
Stage F0–2, n (%) | 69 (56.5) | 143 (58.6) | |
F3–4, n (%) | 53 (43.5) | 101 (41.4) | |
Cirrhosis, n (%) | 40 (32.5) e | 66 (27) e | 21 (10.4) f |
CHB = chronic hepatitis B; CHC = chronic hepatitis C.
CHC > CHB, p < 0.001.
Compared with CHB or CHC, p < 0.001.
Compared with CHC, p = 0.017.
Compared with CHB, p < 0.001; compared with CHC, p = 0.003.
Cirrhosis is confirmed as F4 by Knodell score.
Diagnosed with images including ultrasonography.
Correlation between noninvasive tests and liver fibrosis
No noninvasive indices correlated with fibrosis score in chronic HBV patients (Fig. 1). AAR (p = 0.028), APRI, AARPRI, and AP index (p < 0.001) were increased with fibrosis score, and platelet count (p < 0.001) was decreased with fibrosis score in chronic HCV patients (Fig. 2).
Figure 1.
Correlations between noninvasive indices and fibrosis score in chronic HBV patients (A–E). AAR (A), AARPRI (B), APRI (C), AP index (D), and platelet count (E) (p > 0.05). No noninvasive indices correlated with fibrosis score in chronic HBV patients.
Figure 2.
Correlations between noninvasive indices and fibrosis score in chronic HCV patients (A–E). AAR (A) (p = 0.028), AARPRI (B), APRI (C), and AP index (D) (p < 0.001) were increased with fibrosis score. Platelet count (E) (p < 0.001) was decreased with fibrosis score.
Comparisons of ROC curve in predicting LC
Table 3 compares the reliability of AUC of AAR, APRI, AARPRI, AP index, and platelet count in predicting LC. In chronic HBV patients, AUC of AARPRI (0.693, CI = 0.60–0.77) and platelet count (0.672, CI = 0.58–0.76) were significantly higher than APRI (0.529, CI = 0.44–0.62). Although AARPRI had a higher AUC value, it did not significantly differ from platelet count or AAR in chronic HBV patients. The AUC of all indices for chronic HBV patients was less than 0.7. In chronic HCV patients, the AUC of APRI (0.742, CI = 0.68–0.80), AARPRI (0.744, CI = 0.68–0.80), AP index (0.790, CI = 0.73–0.84), and platelet count (0.801, CI = 0.75–0.85) were significantly higher than AAR (0.615, CI = 0.55–0.68). In the community‐based study, the AUC of AAR (0.552, CI = 0.48–0.62) was significantly lower than in the other four tests in predicting LC based on US. The AUC of APRI, AP index, and platelet count all exceeded 0.90. Platelet count had higher AUC but did not significantly differ from APRI and AP index in both hospitalized chronic HCV patients and HCV‐endemic residents.
Table 3.
Comparisons of AUC using different noninvasive methods for predicting liver cirrhosis.
Noninvasive indices | Area under the receiver operating characteristic curve (AUC) | ||
---|---|---|---|
Chronic hepatitis B | Chronic hepatitis C | Community all | |
AAR | 0.660 (0.57–0.74) | 0.615 (0.55–0.68) b | 0.552 (0.48–0.62) b |
AARPRI | 0.693 (0.60–0.77) | 0.744 (0.68–0.80) | 0.854 (0.80–0.90) |
APRI | 0.529 (0.44–0.62) a | 0.742 (0.68–0.80) | 0.908 (0.86–0.94) |
Platelet count | 0.672 (0.58–0.76) | 0.801 (0.75–0.85) | 0.931 (0.89–0.96) |
AP Index | 0.609 (0.52–0.70) | 0.790 (0.73–0.84) | 0.915 (0.87–0.95) |
AAR = AST/ALT; AARPRI = AAR‐to‐platelet ratio index; AP index = age‐platelet index; APRI = AST‐to‐platelet ratio index.
Compared with AARPRI or platelet count, p < 0.05.
Compared with APRI, AARPRI, AP index, or platelet count, p < 0.05.
Sensitivity, specificity, PPV, and NPV cut‐off levels for different predictive indices in making the pathological diagnosis of LC in chronic HBV and HCV
AAR, AARPRI, and AP index cut‐off values for predicting LC in chronic HBV patients were 0.6, 0.7, and 5.0, respectively (Table 4). In chronic HBV patients, the sensitivity of these indices was between 53% and 63% and specificity was between 58% and 82%. AARPRI ≥ 0.7 and platelet count ≤ 150 × 109/L had higher accuracy (73% and 70%, respectively), and PPV (61% and 54%, respectively) for diagnosing LC compared with other indices in chronic HBV patients. In chronic HCV patients, ARPI ≥ 1.3, AARPRI ≥ 0.8, AP index ≥ 7.0, and platelet count ≤ 150 × 109/L were the best cut‐off levels for predicting LC (Table 5). The cut‐off values were higher in chronic HCV patients than in chronic HBV patients. In chronic HCV patients, the sensitivity of these indices was between 55% and 76% and specificity was between 62% and 80%. AARPRI ≥ 0.8, AP index ≥ 7.0, and platelet count ≤ 150 × 109/L had higher accuracy (73%, 75%, and 74%, respectively), and PPV (51%, 53%, and 52%, respectively) for diagnosing LC than the other indices in chronic HCV patients. An AARPRI ≥ 0.8 had lower sensitivity (55%) than the other two indices in chronic HCV patients. The sensitivity at an APRI ≥ 1.3 (76%) was higher but specificity (62%) was lower than the other indices. Hence, an AP index ≥ 7.0 and platelet count ≤ 150 × 109/L were the preferred tests. Moreover, platelet counts ≤ 150 × 109/L had consistent diagnostic accuracy for LC in both chronic HBV and HCV patients. Although the Pohl score had comparable accuracy (75%), PPV (15%) was the lowest of all indices.
Table 4.
Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of noninvasive tests in detecting liver cirrhosis in chronic HBV patients.
CHB | Sensitivity | Specificity | PPV | NPV | Accuracy | Distance b | |
---|---|---|---|---|---|---|---|
AAR | >0.6 a | 63% | 58% | 43% | 75% | 60% | 0.559 |
≥0.7 | 48% | 73% | 48% | 73% | 65% | 0.586 | |
≥0.8 | 35% | 84% | 52% | 72% | 67% | 0.669 | |
≥1.0 | 5% | 92% | 25% | 66% | 63% | 0.953 | |
AARPRI | ≥0.6 | 60% | 72% | 52% | 78% | 68% | 0.488 |
≥0.7 a | 55% | 82% | 61% | 78% | 73% | 0.484 | |
≥0.8 | 45% | 87% | 64% | 76% | 73% | 0.565 | |
≥1.0 | 23% | 91% | 56% | 70% | 70% | 0.775 | |
AP index | ≥5 a | 60% | 66% | 46% | 77% | 64% | 0.525 |
≥6 | 38% | 82% | 52% | 73% | 68% | 0.645 | |
≥7 | 20% | 87% | 42% | 69% | 65% | 0.810 | |
Platelet | ≤150 a | 53% | 78% | 54% | 77% | 70% | 0.518 |
Pohl score | (+) | 0% | 96% | 0% | 66% | 64% | 1.000 |
AAR = AST/ALT; AARPRI = AAR‐to‐platelet ratio index; AP index = age‐platelet index; APRI = AST‐to‐platelet ratio index; CHB: chronic hepatitis B.
Selected cut‐off value.
Distance = [(1 – sensitivity)2 + (1 – specificity)2]1/2.
Table 5.
Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of noninvasive tests in predicting liver cirrhosis in chronic HCV patients.
CHC | Sensitivity | Specificity | PPV | NPV | Accuracy | Distance b | |
---|---|---|---|---|---|---|---|
APRI | ≥1.0 | 88% | 49% | 39% | 92% | 60% | 0.523 |
≥1.2 | 83% | 58% | 43% | 90% | 65% | 0.453 | |
≥1.3 a | 76% | 62% | 43% | 87% | 66% | 0.449 | |
≥2.0 | 48% | 78% | 45% | 80% | 7%0 | 0.564 | |
AARPRI | ≥0.6 | 76% | 57% | 40% | 82% | 62% | 0.492 |
≥0.7 | 61% | 70% | 44% | 82% | 68% | 0.492 | |
≥0.8 a | 55% | 80% | 51% | 82% | 73% | 0.492 | |
≥1.0 | 42% | 87% | 54% | 80% | 74% | 0.594 | |
AP index | ≥6 | 83% | 62% | 45% | 91% | 68% | 0.416 |
≥7 a | 70% | 77% | 53% | 88% | 75% | 0.378 | |
Platelet count | ≤150 a | 68% | 76% | 52% | 87% | 74% | 0.400 |
Pohl score | (+) | 71% | 75% | 15% | 98% | 75% | 0.382 |
AAR = AST/ALT; AARPRI = AAR‐to‐platelet ratio index; AP index = age‐platelet index; APRI = AST‐to‐platelet ratio index; CHC = chronic hepatitis C.
Selected cut‐off value.
Distance = [(1 – sensitivity)2 + (1 – specificity)2]1/2.
Validation of noninvasive indices in HCV‐endemic communities
The best cut‐off values for these indices for predicting cirrhosis based on US in the HCV‐endemic community were ARPI ≥ 1.0, AARPRI ≥ 1.2, AP index ≥ 8.0, and platelet count ≤ 150 × 109/L. Table 6 shows the sensitivity, specificity, PPV, NPV, and accuracy for predicting LC based on US. A platelet count ≤ 150 × 109/L, an AP index ≥ 8.0 and an APRI ≥ 1.0 had better diagnostic accuracy (86%, 87%, and 91%, respectively) than either AAR or AARPRI for predicting LC based on US. Although APRI ≥ 1.0 had the best accuracy (91%) and the best PPV (54%), it had lower sensitivity (71%) compared with either AP index or platelet count. The sensitivity of an AP index ≥ 8.0 (91%) was higher than a platelet count ≤ 150 × 109/L (76%). However, it had similar PPV and NPV to platelet count. Although the Pohl score also had the comparable accuracy of 84%, this measure exhibited inferior sensitivity (64%) in community residents. Thus, we concluded that AP index and platelet count were acceptable indices for predicting US LC based on US in a HCV‐endemic community.
Table 6.
Sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of noninvasive tests in detecting liver cirrhosis in a HCV‐endemic community.
Community | Sensitivity | Specificity | PPV | NPV | Accuracy | Distance b | |
---|---|---|---|---|---|---|---|
APRI | ≥1.0 a | 71% | 93% | 54% | 96% | 91% | 0.298 |
≥1.2 | 62% | 95% | 59% | 95% | 92% | 0.383 | |
≥1.3 | 57% | 95% | 57% | 95% | 91% | 0.433 | |
≥1.5 | 48% | 96% | 56% | 94% | 91% | 0.522 | |
≥2.0 | 38% | 98% | 67% | 93% | 92% | 0.620 | |
AARPRI | ≥0.7 | 95% | 39% | 15% | 99% | 45% | 0.612 |
≥0.8 | 95% | 49% | 18% | 99% | 54% | 0.512 | |
≥1.0 | 81% | 68% | 23% | 97% | 70% | 0.372 | |
≥1.1 | 76% | 73% | 25% | 96% | 74% | 0.361 | |
≥1.2 a | 76% | 80% | 31% | 96% | 80% | 0.312 | |
AP index | ≥6 | 95% | 63% | 23% | 99% | 66% | 0.373 |
≥7 | 95% | 78% | 34% | 99% | 80% | 0.226 | |
≥8 a | 91% | 87% | 44% | 99% | 87% | 0.158 | |
Platelet | ≤150 a | 76% | 88% | 42% | 97% | 86% | 0.268 |
Pohl score | (+) | 64% | 87% | 47% | 93% | 84% | 0.382 |
AAR = AST/ALT; AARPRI = AAR‐to‐platelet ratio index; AP index = age‐platelet index; APRI = AST‐to‐platelet ratio index.
Selected cut‐off value.
Distance = [(1 – sensitivity)2 + (1 – specificity)2]1/2.
Basic characteristics of HCC patients and LC patients without HCC
A total of 200 patients were enrolled in the study, including 127 newly diagnosed HCC patients and 73 LC patients without HCC. The demographics and clinical characteristics of all patients are summarized in Table 7. The patients with HCC were significantly older than LC patients without HCC. In LC patients, HCV infection was responsible for LC in the LC patients with HCC; however, chronic HBV infection and HCV infection were the major contributors to the development of HCC in HCC patients.
Table 7.
Demographics and clinical characteristics of patients with hepatocellular carcinoma (HCC) or liver cirrhosis patients without HCC.
Patients with LC but without HCC (n = 73) | Patients with HCC (n = 127) | p | |
---|---|---|---|
Age (y, mean ± SD) | 55.2 ± 9.7 | 62.1 ± 11.3 | <0.001 |
Gender (%) | 0.027 | ||
Male | 47 (64.3) | 100 (78.7) | |
Female | 26 (35.7) | 27 (21.3) | |
Etiology (%) | 0.003 | ||
HBV | 22 (30.1) | 50 (39.4) | |
HCV | 39 (53.4) | 41 (32.3) | |
HBV + HCV | 8 (10.9) | 10 (7.8) | |
Non‐B/non‐C | 4 (5.5) | 26 (20.5) | |
Liver cirrhosis (%) | 73 (100) a | 80 (64.1) b | <0.001 |
BMI (kg/m2) | 24.7 ± 2.9 | 24.0 ± 3.6 | 0.167 |
AST (IU/L) | 88.2 ± 49.3 | 86.2 ± 64.8 | 0.824 |
ALT (IU/L) | 106.1 ± 74.3 | 74.1 ± 71.5 | 0.03 |
Bilirubin (mg/dL) | 1.11 ± 0.50 | 1.53 ± 4.03 | 0.387 |
AFP (ng/mL) | 23.7 ± 44.7 | 9924.4 ± 4708.4 | 0.02 |
AFP = α‐fetoprotein; ALT = alanine aminotransferase; AST = aspartate aminotransferase; BMI = body mass index; HBV = hepatitis B virus; HCV = hepatitis C virus; Non‐B/non C = etiology outside of HBV or HCV.
Confirmed by pathology.
Diagnosis with images including ultrasonography and/or computed tomography.
Comparisons of noninvasive indices in predicting high‐risk candidates of HCC
The values as assessed by areas under the ROC curves (AUROC) in predicting HCC were 0.710 (95% CI = 0.640–0.773), 0.597 (0.524–0.667), 0.580 (0.507–0.651), 0.572 (0.499–0.643), and 0.558 (0.485–0.629) for AAR, AARPRI, APRI, AP index, and platelet count, respectively (Fig. 3). The diagnostic accuracy of AAR for HCC presence was superior to AARPRI (p = 0.001), APRI (p = 0.013), AP index (p = 0.008), and platelet count (p = 0.004).
Figure 3.
Receiver operating characteristic (ROC) curves for AAR, AARPRI, APRI, AP index, and platelet count. The areas under the ROC curves (AUROC) for HCC presence are 0.710, 0.597, 0.580, 0.572, and 0.558, respectively. AAR is superior to AARPRI, APRI, AP index, and platelet count (p < 0.05).
The optimal cut‐off level in predicting high‐risk candidates of HCC
The cut‐off value of AAR used in diagnosing the presence of HCC, as determined with short distance of [(1 – sensitivity)2 + (1 – specificity)2]1/2, was 1.2. The sensitivity and specificity was 61.6% and 73.2% for all patients, 53.4% and 83.3% for HBV patients, 39.2% and 78.7% for HCV patients, respectively. The cut‐off value, as determined with specificity more than 90%, was 1.4. The sensitivity and specificity was 36.8% and 92.9% for all patients, 39.6% and 93.3% for HBV patients, 27.4% and 91.4% for HCV patients, respectively.
To estimate the likelihood ratios for the presence of HCC, we ranked the values of AAR into three strata. For all patients, the SSLR for HCC presence was 0.59 for AAR < 1.2, 1.28 for AAR between 1.2 and 5.22 for AAR > 1.4, respectively. Stratified according to virus etiology, SSLR was 0.56, 1.38, and 5.95 for AAR < 1.2, 1.2–1.4 and > 1.4 in HBV patients, respectively. For HCV patients, SSLR was 0.77, 0.92, and 3.23 for AAR < 1.2, 1.2–1.4, and > 1.4, respectively (Table 8). Irrespective of viral etiology, the relative risk for the presence of HCC increased with the progression of AAR values.
Table 8.
The sensitivity, specificity, and likelihood ratio for the presence of hepatocellular carcinoma (HCC) stratified according to all patients, patients with hepatitis B virus (HBV), or C virus infection (HCV) analyzed with stratum‐specific likelihood ratio (SSLR) of AAR.
Cut‐off values | Sensitivity | Specificity | Strata | Patients with LC but without HCC | Patients with HCC | SSLR |
---|---|---|---|---|---|---|
All patients (n = 196) a | ||||||
1.2 | 61.6% | 73.2% | <1.2 | 58 | 61 | 0.59 |
1.3 | 44.8% | 87.3% | 1.2–1.4 | 8 | 18 | 1.28 |
1.4 | 36.8% | 92.9% | >1.4 | 5 | 46 | 5.22 |
HBV patients (n = 88) | ||||||
1.2 | 53.4% | 83.3% | <1.2 | 25 | 27 | 0.56 |
1.3 | 46.5% | 83.3% | 1.2–1.4 | 3 | 8 | 1.38 |
1.4 | 39.6% | 93.3% | >1.4 | 2 | 23 | 5.95 |
HCV patients (n = 98) | ||||||
1.2 | 39.2% | 78.7% | <1.2 | 37 | 31 | 0.77 |
1.3 | 33.3% | 85.1% | 1.2–1.4 | 6 | 6 | 0.92 |
1.4 | 27.4% | 91.4% | >1.4 | 4 | 14 | 3.23 |
HBV = hepatitis B virus; HCC = hepatocellular carcinoma; HCV = hepatitis C virus; SSLR = stratum‐specific likelihood ratio.
Four patients with incomplete data were excluded from analysis.
Discussion
Many direct and indirect serum markers are currently used not only to predict liver fibrosis and LC but also to avoid liver biopsies before antiviral therapy. Thrombocytopenia as a surrogate of LC has also been used as a marker to identify groups at high risk for developing HCC in HCV‐endemic areas. We compared noninvasive indices with platelet count for predicting LC. The sensitivity of these indices was also analyzed in predicting HCC patients.
Directed markers such as matrix protein collagens and sophisticated indices such as FibroTest were not used for comparison purposes in this study because they are expensive and not routinely available. We used six simple indices composed of four routine available factors: AST, ALT, platelet count, and age. AAR and APRI used AST for predicting LC because an elevated AST in developing liver fibrosis may be related to reduced AST clearance and mitochondria injury [[30], [31], [32]]. Thrombocytopenia, the simplest serum surrogate for liver fibrosis is caused by decreased thrombopoietin production [[33], [34]], antibody mediated platelet destruction [35], and myelotoxic effects [36]. Confounding factors are infectious and hematological diseases. Age represents time lapse after HCV infection [37] and is combined with platelet count to predict significant fibrosis and necroinflammatory activity in HCV patients [14].
All our indices were validated in three study populations, that is, hospital chronic HBV patients, HCV patients, and HCV‐endemic community residents. The choice of predictive index depended on the diagnostic accuracy in different populations and the manner in which they were utilized. In chronic HBV patients, all indices were not accurate for predicting LC with AUC < 0.70. The preferred tests, AARPRI ≥ 0.7 and platelet count ≤ 150 × 109/L had better accuracy (73% and 70%, respectively) in chronic HBV patients compared with other tests. Platelet count proved superior to APRI for predicting HBV‐related LC similar to the finding of Wai et al. [12]. AST is useful in predicting liver fibrosis in patients with chronic HCV, but not in patients with chronic HBV and alcoholic liver disease due to a differing pathogenesis. The new index AARPRI, derived from FIB‐4, used AST/ALT instead of AST to adjust AST elevation caused by severe hepatic inflammation, which was different from APRI. The factors used in this study are the same as those used for the Pohl score but AARPRI is calculated as a continuous number. AAR had a partial additional effect on platelet count in predicting LC in chronic HBV patients. Because age is related to progressive liver disease caused by HCV infections rather than HBV infections, the AP index is inferior to platelet count and AARPRI [37]. In hospitalized chronic HCV patients, the AP index and platelet count are the preferred tests.
Previous studies showed that thrombocytopenia had a sensitivity between 77% and 91% and specificity exceeding 85% for predicting chronic HCV‐related LC [[9], [31], [37]]. We found that platelet count ≤ 150 × 109/L had lower sensitivity (68%) and specificity (76%) in the current study than in previous reports. The discrepancy may be explained by variable platelet cut‐off values and differing pathological scoring systems. However, platelet count had a consistent cut‐off value and better diagnostic accuracy in chronic hepatitis B and hepatitis C patients.
An AP index ≥ 6 revealed a sensitivity of 67% and specificity of 87% for predicting LC in a recent investigation [38]. We found that an AP index ≥ 7 had similar sensitivity (70%) but lower specificity (77%) than previous studies. We used a higher AP index cut‐off value due to the older mean age of the population studied compared to previous studies. Age did not affect the predictive accuracy for predicting LC in chronic hepatitis C patients. The cut‐off value of APRI was 1.3, which was lower than the value of 2.0 mentioned by Wai et al. [10]. Compared to Wai et al., the sensitivity (48%) and specificity (78%) of our APRI was less, even when we set the cut‐off value of 2.0.
In the HCV‐endemic community, all indices used to predict US diagnosed LC in the community based population were not validated with liver biopsy as liver biopsy was not feasible in all residents in this community. In our previous study [26], US showed a sensitivity of 82.4% and specificity of 70.7% for predicting LC in chronic HCV patients. Hence, ultrasound was used for LC diagnosis instead of liver biopsy in the community screen. The preferred tests for predicting LC diagnosed by US were also the same as those used in hospital HCV patients, except for APRI. APRI had a similar AUC as the AP index and platelet count but was more accurate. APRI had lower sensitivity (71%), higher specificity (93%), and better PPV (54%) compared with AP index or platelet count. The cut‐off value of APRI was lower in the community than in the hospital group. Lower hepatic necroinflammatory activity of community residents might explain the differences. Age and AST would increase the sensitivity and specificity of platelet count individually in a HCV‐endemic community study. The AAR and AAR‐related model, AARPRI, were satisfactory for predicting LC in chronic HCV patients and HCV‐endemic community residents. Our results were consistent with previous studies [[10], [39], [40]].
The Pohl score was a categorical variable and was not used for AUC comparisons. Although the Pohl score improved PPV and NPV related to LC, decreased sensitivity may have limited the numbers of patients who met the score. Although this HCV‐endemic community was not homogeneous enough to test the accuracy of noninvasive indices, our purpose was to validate noninvasive indices in a community setting, especially a HCV‐endemic community.
SSLR was the statistical method used to evaluate the risk of disease by a fixed optimal cut‐off point. Because too many strata cause the likelihood ratios to become unstable and degenerate, we used only three strata. In addition, we used an AAR of 1.4 as the cut‐off value between medium and higher strata as this point showed a significant difference between the two strata irrespective of viral etiology.
Surveillance of HCC requires determining a high enough degree of risk of contracting HCC to trigger such surveillance. Therefore, screening HCC in cirrhotic patients was proposed. In clinical practice, both US and liver biopsy are used to evaluate LC. However, these expensive methods require substantial manpower and are not suitable for large‐scale community screening. The design of a two‐stage HCC screening program would rely upon budget, manpower, and the evaluation of HCC patients using a noninvasive index in a limited high‐risk group. If a serum marker was available for predicting LC in the first stage of community screening, the high‐risk HCC group would be readily identifiable. In the first stage, we suggested using the platelet count to predict LC patients, and in the second stage, AAR could be a suitable surrogate marker for HCC.
In conclusion, all noninvasive indices had poor diagnostic accuracy for predicting LC in chronic HBV patients. A platelet count of ≤ 150 × 109/L demonstrated consistently acceptable predictive value for LC in all populations. In the first stage of HCC screening, values of AARPRI ≥ 0.7 and platelet count ≤ 150 × 109/L could be used for predicting LC in HBV patients. In addition, values of AP index ≥ 7 and platelet count ≤ 150 × 109/L could be used to predict LC in HCV patients. In an HCV‐endemic community, an APRI ≥ 1.0, AP index ≥ 8, and platelet count ≤ 150 × 109/L could be used for predicting high‐risk LC groups. In the second stage of HCC screening, an AAR > 1.4 could provide sufficiency specificity to identify high‐risk HCC patients.
Acknowledgments
This study was supported by research grants to S.‐N.L. from Chang Gung Memorial Hospital (CMRP 8032) and the National Science Council, Republic of China (NSC 91‐2314‐B‐182A‐177). The ultrasonographic machines were sponsored by Mr K.C. Chu. We would like to thank the local clinic in each study area for their cooperation. We would also like to thank the Medical Association and Health Bureau of Tainan County for their administrative support.
References
- [1]. Merican I., Guan R., Amarapuka D., Alexander M.J., Chutaputti A., Chien R.N., et al. Chronic hepatitis B virus infection in Asian countries. J Gastroenterol Hepatol. 2000; 15: 1356–1361. [DOI] [PubMed] [Google Scholar]
- [2]. Marcellin P., Asselah T., Boyer N.. Fibrosis and disease progression in hepatitis C. Hepatology. 2002; 36: S47–S56. [DOI] [PubMed] [Google Scholar]
- [3]. Lok A.S., McMahon B.J.. Chronic hepatitis B: update of recommendations. Hepatology. 2004; 39: 857–861. [DOI] [PubMed] [Google Scholar]
- [4]. Strader D.B., Wright T., Thomas D.L., Seeff L.B.. Diagnosis, management, and treatment of hepatitis C. Hepatology. 2004; 39: 1147–1171. [DOI] [PubMed] [Google Scholar]
- [5]. Bedossa P., Dargere D., Paradis V.. Sampling variability of liver fibrosis in chronic hepatitis C. Hepatology. 2003; 38: 1449–1457. [DOI] [PubMed] [Google Scholar]
- [6]. Regev A., Berho M., Jeffers L.J., Milikowski C., Molina E.G., Pyrsopoulos N.T., et al. Sampling error and intraobserver variation in liver biopsy in patients with chronic HCV infection. Am J Gastroenterol. 2002; 97: 2614–2618. [DOI] [PubMed] [Google Scholar]
- [7]. Perrault J., McGill D.B., Ott B.J., Taylor W.F.. Liver biopsy: complications in 1000 inpatients and outpatients. Gastroenterology. 1978; 74: 103–106. [PubMed] [Google Scholar]
- [8]. Wong J.B., Koff R.S.. Watchful waiting with periodic liver biopsy versus immediate empirical therapy for histologically mild chronic hepatitis C. A cost‐effectiveness analysis. Ann Intern Med. 2000; 133: 665–675. [DOI] [PubMed] [Google Scholar]
- [9]. Giannini E., Risso D., Botta F., Chiarbonello B., Fasoli A., Malfatti F., et al. Validity and clinical utility of the aspartate aminotransferase‐alanine aminotransferase ratio in assessing disease severity and prognosis in patients with hepatitis C virus‐related chronic liver disease. Arch Intern Med. 2003; 163: 218–224. [DOI] [PubMed] [Google Scholar]
- [10]. Wai C.T., Greenson J.K., Fontana R.J., Kalbfleisch J.D., Marrero J.A., Conjeevaram H.S., et al. A simple noninvasive index can predict both significant fibrosis and cirrhosis in patients with chronic hepatitis C. Hepatology. 2003; 38: 518–526. [DOI] [PubMed] [Google Scholar]
- [11]. Murawaki Y., Koda M., Okamoto K., Mimura K., Kawasaki H.. Diagnostic value of serum type IV collagen test in comparison with platelet count for predicting the fibrotic stage in patients with chronic hepatitis C. J Gastroenterol Hepatol. 2001; 16: 777–781. [DOI] [PubMed] [Google Scholar]
- [12]. Wai C.T., Cheng C.L., Wee A., Dan Y.Y., Chan E., Chua W., et al. Non‐invasive models for predicting histology in patients with chronic hepatitis B. Liver Int. 2006; 26: 666–672. [DOI] [PubMed] [Google Scholar]
- [13]. Pohl A., Behling C., Oliver D., Kilani M., Monson P., Hassanein T.. Serum aminotransferase levels and platelet counts as predictors of degree of fibrosis in chronic hepatitis C virus infection. Am J Gastroenterol. 2001; 96: 3142–3146. [DOI] [PubMed] [Google Scholar]
- [14]. Poynard T., Bedossa P.. Age and platelet count: a simple index for predicting the presence of histological lesions in patients with antibodies to hepatitis C virus. METAVIR and CLINIVIR Cooperative Study Groups. J Viral Hepat. 1997; 4: 199–208. [DOI] [PubMed] [Google Scholar]
- [15]. Giannini E., Testa R.. Noninvasive diagnosis of fibrosis: the truth is rarely pure and never simple. Hepatology. 2003; 38: 1312–1313. [DOI] [PubMed] [Google Scholar]
- [16]. Berg T., Sarrazin C., Hinrichsen H., Buggisch P., Gerlach T., Zachoval R., et al. Does noninvasive staging of fibrosis challenge liver biopsy as a gold standard in chronic hepatitis C?. Hepatology. 2004; 39: 1456–1458. [DOI] [PubMed] [Google Scholar]
- [17]. Myers R.P., Tainturier M.H., Ratziu V., Piton A., Thibault V., Imbert‐Bismut F., et al. Prediction of liver histological lesions with biochemical markers in patients with chronic hepatitis B. J Hepatol. 2003; 39: 222–230. [DOI] [PubMed] [Google Scholar]
- [18]. Hui A.Y., Chan H.L., Wong V.W., Liew C.T., Chim A.M., Chan F.K., et al. Identification of chronic hepatitis B patients without significant liver fibrosis by a simple noninvasive predictive model. Am J Gastroenterol. 2005; 100: 616–623. [DOI] [PubMed] [Google Scholar]
- [19]. Zeng M.D., Lu L.G., Mao Y.M., Qiu D.K., Li J.Q., Wan M.B., et al. Prediction of significant fibrosis in HBeAg‐positive patients with chronic hepatitis B by a noninvasive model. Hepatology. 2005; 42: 1437–1445. [DOI] [PubMed] [Google Scholar]
- [20]. Raza S.A., Clifford G.M., Franceschi S.. Worldwide variation in the relative importance of hepatitis B and hepatitis C viruses in hepatocellular carcinoma: a systematic review. Br J Cancer. 2007; 96: 1127–1134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [21]. Yu M.W., Chen C.J.. Hepatitis B and C viruses in the development of hepatocellular carcinoma. Crit Rev Oncol Hematol. 1994; 17: 71–91. [DOI] [PubMed] [Google Scholar]
- [22]. Fattovich G., Stroffolini T., Zagni I., Donato F.. Hepatocellular carcinoma in cirrhosis: incidence and risk factors. Gastroenterology. 2004; 127: S35–S50. [DOI] [PubMed] [Google Scholar]
- [23]. Bruix J., Sherman M.. Management of hepatocellular carcinoma. Hepatology. 2005; 42: 1208–1236. [DOI] [PubMed] [Google Scholar]
- [24]. Lu S.N., Wang J.H., Liu S.L., Hung C.H., Chen C.H., Tung H.D., et al. Thrombocytopenia as a surrogate for cirrhosis and a marker for the identification of patients at high‐risk for hepatocellular carcinoma. Cancer. 2006; 107: 2212–2222. [DOI] [PubMed] [Google Scholar]
- [25]. Knodell R.G., Ishak K.G., Black W.C., Chen T.S., Craig R., Kaplowitz N., et al. Formulation and application of a numerical scoring system for assessing histological activity in asymptomatic chronic active hepatitis. Hepatology. 1981; 1: 431–435. [DOI] [PubMed] [Google Scholar]
- [26]. Hung C.H., Lu S.N., Wang J.H., Lee C.M., Chen T.M., Tung H.D., et al. Correlation between ultrasonographic and pathologic diagnoses of hepatitis B and C virus‐related cirrhosis. J Gastroenterol. 2003; 38: 153–157. [DOI] [PubMed] [Google Scholar]
- [27]. Hanley J.A., McNeil B.J.. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982; 143: 29–36. [DOI] [PubMed] [Google Scholar]
- [28]. Hanley J.A., McNeil B.J.. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983; 148: 839–843. [DOI] [PubMed] [Google Scholar]
- [29]. Peirce J.C., Cornell R.G.. Integrating stratum‐specific likelihood ratios with the analysis of ROC curves. Med Decis Making. 1993; 13: 141–151. [DOI] [PubMed] [Google Scholar]
- [30]. Kamimoto Y., Horiuchi S., Tanase S., Morino Y.. Plasma clearance of intravenously injected aspartate aminotransferase isozymes: evidence for preferential uptake by sinusoidal liver cells. Hepatology. 1985; 5: 367–375. [DOI] [PubMed] [Google Scholar]
- [31]. Okuda M., Li K., Beard M.R., Showalter L.A., Scholle F., Lemon S.M., et al. Mitochondrial injury, oxidative stress, and antioxidant gene expression are induced by hepatitis C virus core protein. Gastroenterology. 2002; 122: 366–375. [DOI] [PubMed] [Google Scholar]
- [32]. Nalpas B., Vassault A., Le Guillou A., Lesgourgues B., Ferry N., Lacour B., et al. Serum activity of mitochondrial aspartate aminotransferase: a sensitive marker of alcoholism with or without alcoholic hepatitis. Hepatology. 1984; 4: 893–896. [DOI] [PubMed] [Google Scholar]
- [33]. Peck‐Radosavljevic M., Zacherl J., Meng Y.G., Pidlich J., Lipinski E., Langle F., et al. Is inadequate thrombopoietin production a major cause of thrombocytopenia in cirrhosis of the liver?. J Hepatol. 1997; 27: 127–131. [DOI] [PubMed] [Google Scholar]
- [34]. Giannini E., Borro P., Botta F., Fumagalli A., Malfatti F., Podesta E., et al. Serum thrombopoietin levels are linked to liver function in untreated patients with hepatitis C virus‐related chronic hepatitis. J Hepatol. 2002; 37: 572–577. [DOI] [PubMed] [Google Scholar]
- [35]. Nagamine T., Ohtuka T., Takehara K., Arai T., Takagi H., Mori M.. Thrombocytopenia associated with hepatitis C viral infection. J Hepatol. 1996; 24: 135–140. [DOI] [PubMed] [Google Scholar]
- [36]. Bordin G., Ballare M., Zigrossi P., Bertoncelli M.C., Paccagnino L., Baroli A., et al. A laboratory and thrombokinetic study of HCV‐associated thrombocytopenia: a direct role of HCV in bone marrow exhaustion?. Clin Exp Rheumatol. 1995; 13 (Suppl. 13): S39–S43. [PubMed] [Google Scholar]
- [37]. Afdhal N.H.. The natural history of hepatitis C. Semin Liver Dis. 2004; 24 (Suppl. 2): 3–8. [DOI] [PubMed] [Google Scholar]
- [38]. Borroni G., Ceriani R., Cazzaniga M., Tommasini M., Roncalli M., Maltempo C., et al. Comparison of simple tests for the non‐invasive diagnosis of clinically silent cirrhosis in chronic hepatitis C. Aliment Pharmacol Ther. 2006; 24: 797–804. [DOI] [PubMed] [Google Scholar]
- [39]. Iacobellis A., Mangia A., Leandro G., Clemente R., Festa V., Attino V., et al. External validation of biochemical indices for noninvasive evaluation of liver fibrosis in HCV chronic hepatitis. Am J Gastroenterol. 2005; 100: 868–873. [DOI] [PubMed] [Google Scholar]
- [40]. Snyder N., Gajula L., Xiao S.Y., Grady J., Luxon B., Lau D.T., et al. APRI: an easy and validated predictor of hepatic fibrosis in chronic hepatitis C. J Clin Gastroenterol. 2006; 40: 535–542. [DOI] [PubMed] [Google Scholar]