Diagnostic Performance of Artificial Intelligence in Detection of Hepatocellular Carcinoma: A Meta-analysis

Abstract

Due to the increasing interest in the use of artificial intelligence (AI) algorithms in hepatocellular carcinoma detection, we performed a systematic review and meta-analysis to pool the data on diagnostic performance metrics of AI and to compare them with clinicians’ performance. A search in PubMed and Scopus was performed in January 2024 to find studies that evaluated and/or validated an AI algorithm for the detection of HCC. We performed a meta-analysis to pool the data on the metrics of diagnostic performance. Subgroup analysis based on the modality of imaging and meta-regression based on multiple parameters were performed to find potential sources of heterogeneity. The risk of bias was assessed using Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) and Prediction Model Study Risk of Bias Assessment Tool (PROBAST) reporting guidelines. Out of 3177 studies screened, 44 eligible studies were included. The pooled sensitivity and specificity for internally validated AI algorithms were 84% (95% CI: 81,87) and 92% (95% CI: 90,94), respectively. Externally validated AI algorithms had a pooled sensitivity of 85% (95% CI: 78,89) and specificity of 84% (95% CI: 72,91). When clinicians were internally validated, their pooled sensitivity was 70% (95% CI: 60,78), while their pooled specificity was 85% (95% CI: 77,90). This study implies that AI can perform as a diagnostic supplement for clinicians and radiologists by screening images and highlighting regions of interest, thus improving workflow.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10278-024-01058-1.

Introduction

Globally, primary liver cancers rank sixth in terms of incidence and third in terms of cancer-related mortality [1]. According to the GLOBOCAN 2018 database, liver cancer is detected in around 800,000 new cases year, leading to more than 700,000 fatalities [2]. Hepatocellular carcinoma (HCC), also known as hepatoma, is the most prevalent primary malignant tumor of the liver, accounting for approximately 70 to 80% of all such malignancies [3]. Fortunately, individuals diagnosed with HCC today have a range of therapeutic modalities available to them, including chemotherapy, radiation therapy, surgical resection, percutaneous ablation, and liver transplantation [4].

As with most diseases of a similar nature, early diagnosis can assist in determining the most appropriate course of treatment and, as a consequence, improve the patient's prognosis with regard to mortality [5–7] and morbidity [7, 8]. The early detection of HCC is greatly aided by diagnostic radiology, which plays a crucial role in the diagnostic process. In a meta-analysis of eighteen studies, Liang et al. reported the pooled sensitivity and specificity of LI-RADS ≥ 3 for HCC diagnosis, 0.86 (95% CI: 0.78–0.91) and 0.85 (95% CI: 0.78–0.90), respectively [9].

The concept of artificial intelligence (AI) refers to machines performing tasks that typically require the intelligence of a human to accomplish. Furthermore, machine learning is an essential component of AI that enables devices to acquire knowledge by analyzing data, identifying patterns, and making decisions with the minimum amount of human intervention [10]. Deep learning (DL) is also a subset of machine learning consisting of three or more layers of neural networks, algorithms based on the concept of a human neuron [11], that process and validate datasets, using training data and its outcome, producing accurate predictions for testing data [12]. Numerous radiological functions, including the detection of abnormal lymph nodes, the assessment of skeletal maturity, and the identification of mammographic lesions, have exhibited encouraging outcomes due to the implementation of AI in radiology [13]. The methods entail utilizing machine learning algorithms such as support vector machines, random forest, and decision tree on radiomics data, which are quantitative characteristics derived from several radiologic modalities. Furthermore, artificial neural networks (ANN) are employed for the analysis of numerical data, while convolutional neural networks (CNN) are utilized for the processing of image data [14].

The incorporation of AI as a supplementary tool for radiologists has undergone a significant transformation over the past few years, potentially representing a paradigm shift in the field [15–17]. Its influence is presently being realized, extending to the domain of interventional radiology [18]. The detection and classification of liver lesions are no exceptions to this rule [19, 20]. At the time we are conducting this review, The US Food and Drug Administration (FDA) has recently approved using several artificial intelligence-enhanced software for medical purposes [21]. Nonetheless, the practical applicability of this concept is still subject to some debate [9, 22–25]. Several recent studies have addressed the constraints and challenges associated with the implementation of AI in the field of medicine [26–28].

Accordingly, we conducted a review to systematically assess the published studies that applied artificial intelligence in the imaging field for diagnosis of hepatocellular carcinoma. Furthermore, we performed a meta-analysis to determine this system’s accuracy in identifying HCC and distinguishing it from other forms of liver masses.

Materials and Methods

Protocol and Registration

This systematic review and meta-analysis was prepared in line with the Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) statement [29]. The authors have submitted this review to the International Prospective Register of Systematic Reviews (PROSPERO) website (CRD42023398040). M.A.S and H.H conducted primary and full-text screenings. M.S.F and H.H. extracted data from the articles. These data were also rechecked by S.M. Study quality assessment was performed by S.M and M.A.S.

Search Strategy and Study Selection

In order to find studies that evaluated an AI algorithm for the detection of HCC, a search in PubMed and Scopus was conducted in September 2022 and subsequently updated in January 2024. We utilized the Medical Subject Headings (MeSH) terms “Hepatocellular Carcinoma” (HCC) and “Artificial Intelligence,” as well as their synonyms and terms found below them in the MeSH hierarchy, to search for relevant articles in online databases (Table E1). We included all articles that met the following criteria: original research studies that developed or validated an AI algorithm for detecting HCC among patients with a hepatic mass (e.g., HCC, metastasis, hemangioma, and cysts) or those without any masses. The following criteria were applied as the exclusion criteria: non-original articles, and studies that true negatives (TN), false positives (FP), true positives (TP), and false negatives (FN) were not mentioned or could not be calculated to prepare a contingency table. In terms of study language, time, or settings, there were no restrictions. Disagreements were resolved by consensus.

Data Extraction

In each study, the following variables were extracted: first author and publication year; country of the study; the number of participants for each trait; imaging modality; utilized algorithm and architecture; imaging dimension; evaluation metrics; and training size; type of validation and validation size; imaging view; the number of true negatives, false positives, true positives, false negatives; sensitivity and specificity; and positive predictive value, area under the curve (AUC), and accuracy. A contingency table was constructed for each study using the nominal data reported therein. To ensure the accuracy of the data for analysis, two reviewers independently reviewed each table.

Statistical Analysis

To evaluate the diagnostic performance of both AI algorithms and radiologists, we performed a meta-analysis of relevant studies. We employed a random-effects model to calculate the pooled sensitivity and specificity, along with their corresponding 95% confidence intervals (CI), if at least three eligible studies were identified. We also constructed summary receiver operating characteristic (ROC) curves to estimate the pooled sensitivities and specificities.

In addition, we conducted a meta-regression analysis to explore potential sources of inter-study heterogeneity, examining covariates of the usage of data augmentation and the modality of each study. All statistical tests were two sided and a significance level of P < 0.05 was used to indicate statistical significance. All quantitative data analysis was performed using Stata version 17 software (Stata Corp, College Station, TX) [30, 31].

Quality Assessment

We assessed the extent to which studies adhered to reporting guidelines using the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) checklist. TRIPOD statement comprises 22 recommendations to promote transparent reporting in studies involving the development or validation of prediction models [32]. We used a modified version of TRIPOD (Table E2) since not every item on this checklist was useful for AI studies; for instance, it is irrelevant to report follow-up time, in diagnostic accuracy studies. In addition, the Prediction Model Study Risk of Bias Assessment Tool (PROBAST) checklist was employed to evaluate the presence of bias and assess the applicability of the included studies [33] (Table E3). This tool utilizes a series of questions pertaining to participants, predictors, outcomes, and analysis to provide both a comprehensive and detailed evaluation.

Publication Bias

To mitigate the effects of publication bias, we conducted a search of the reference lists of the included studies. Additionally, we performed a formal assessment of publication bias using diagnostic log odds ratios and asymmetry testing via regression analysis [34].

Results

Study Selection and Characteristics

The primary database search found 3177 records. Duplication removal excluded 951 studies. All other studies underwent title and abstract screening, resulting in the exclusion of 1936 articles. Finally, out of the remaining studies, 246 were excluded from the full-text screening based on: no correlation to the title (n = 175); non-original studies (n = 9); non-English (n = 1); no controls (n = 4), not using imaging as its diagnosis method (n = 44); insufficient data (n = 9); and full text not available (n = 4). The next phase included backward reference screening on the 44 included studies to identify potentially neglected eligible studies (Fig. 1). Any conflicts were resolved through discussion.

Fig. 1.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram describing the process of study selection

Tables 1, 2, and 3 show each study’s radiologic view and reference standard, which is used to develop an image labeling reference. Sixteen studies used ultrasound (US) [35–50], fifteen used computed tomography (CT) [51–65], and eleven studies used magnetic resonance imaging (MRI) [66–76]. One study utilized both CT and MRI [77], while another study used four different modalities [78]. Each study’s control group and the radiologic view is indicated in Table 1, 2, and 3 and E4. Moreover, as revealed in Tables E5 and E6, 43 studies validated their outcome internally [35–75, 77, 78], while only four used external validations [43, 44, 58, 76].

Table 1.

Study characteristics, internal validation (with a comparison group)

First author	Year	Country	Imaging modality	View	Comparison group	No. of images per set			Reference standard	Model output
						Training	Tuning	Testing
Zhang	2009	Japan	MRI	NR	Expert clinicians	50	-	30	Expert consensus	Classification
Jiang	2019	China	MRI	Horizontal	Version 2018 EASL, LI-RADS criteria	137	-	92	Histopathology	Binary classification
Raluca Brehar	2020	Romania	US	NR	Comparison of multiple algorithms	Image set 1: 10,834 Image set 2: 6672	-	Image set 1: 3224 Image set 2: 2068	Past study	Binary classification
Wang	2021	China	MRI	NR	Expert clinicians	444	-	113	Pathology or follow-up images	Seven-way classification
Paula M. Oestmann	2021	Germany	MRI	NR	Expert clinicians	140	-	10	Histopathology	Classification
Hirotsugu Nakai	2021	Japan	CT	Axial	Expert clinicians	493	-	62	Expert consensus, histopathology	Binary classification
Meiyun Wang	2021	China	CT	Axial	Expert clinicians	7512	-	95	Expert consensus	Classification
Naoshi Nishida	2022	Japan	US	Axial	Expert clinicians	-	-	55	Expert consensus	Classification
Haiping Zhang	2022	China	MRI	NR	Expert clinicians	72	-	30	Biopsy or surgical resection, expert consensus	Binary classification
Chi-Tung Cheng	2022	China	CT	Axial	Expert clinicians	771	-	370	Expert consensus, pathology	Classification
Anisha	2023	India	CT	NR	Comparison of multiple algorithms	-	-	-	NR	classification
Maryam Fotouhi	2023	Iran	MRI	NR	Expert clinicians	274.4	-	78	Pathology or follow-up results	Binary classification
Sutthirak Tangruangkiat	2023	Thailand	Ultrasound	NR	Previously published studies	600	-	150	MRI/CT reports	Classification
Marinela-Cristiana Urhut	2023	Romania	Ultrasound	NR	Expert clinicians-previously published studies	-	-	-	CT or MRI or US	Binary classification
Abhishek Midya	2023	USA	CT	Axial	Expert clinicians-previously published studies	60	-	-	Expert consensus, histopathology	Binary classification
Gou	2023	China	Contrast enhanced CT	NR	Expert clinicians	239	-	103	Histopathology	Binary classification
Xuepeng Zhang	2023	China	Contrast enhanced CT	Axial	Comparison of multiple algorithms-previously published studies	222	-	95	Histopathology	Binary classification

CT computed tomography, CECT contrast-enhanced computed tomography, CEUS contrast-enhanced ultrasonography using Sonzoid, HCC hepatocellular carcinoma, HEM cavernous hemangioma, HEP hepatic abscess, FNH focal nodular hyperplasia, ICC Intrahepatic cholangiocarcinoma, MET hepatic metastasis, GdEOB-DTPA gadoliniumethoxybenzyl-diethylenetriamine penta-acetic acid, MRI magnetic resonance imaging, NR not reported, US ultrasound

Table 2.

Study characteristics, internal validation (without a comparison group)

First author	Year	Country	Imaging modality	View	Comparison group	No. of images per set			Reference standard	Model output
						Training	Tuning	Testing
E-Liang Chen	1998	Taiwan	CT	NR	NR	NR	-	30	Past study	Binary classification
Junji Shiraishi	2008	Japan	US	NR	NR	-	-	103	Biopsy or surgical specimens, expert consensus	Classifying focal liver lesions
Katsutoshi Sugimoto	2009	Japan	US	NR	NR	NR	-	137	Expert consensus	Three-way classification based on physicians’ subjective pattern
S.S. Kumar	2011	India	CT	NR	NR	45	-	45	Past study	Binary classification
Mittal	2011	India	US	NR	NR	Image set 1: 250 Image set 2: 250 Image set 3: 250	Image set 1: 50 Image set 2: 50 Image set 3: 50	Image set 1: 500 Image set 2: 340 Image set 3: 160	Expert radiologist	Binary classification
Costin Teodor Streba	2012	Romania	US	NR	NR	-	-	-	Other imagistic methods (CT and CE-MRI), liver biopsy	Classification
Virmani	2014	India	US	NR	NR	57	-	51	Expert clinicians, patient history	Five-way classification
Ilias Gatos	2017	greece	MRI	NR	NR	NR	-	71	Expert consensus, biopsy	Multiclass PNN classification
Amita Das	2018	India	CT	NR	NR	163	-	62	Past study	Classification
Marie¨lle J. A. Jansen	2019	Netherlands	MRI	Axial	NR	NR	-	213	Expert consensus	Classification
Makoto Yamakawa	2019	Japan	US	NR	NR	NR	-	324	Past study	Four-way classification
Akash Nayak	2019	India	CT	Axial	NR	NR	-	Image set 1: 726 Image set 2: 15	Past study	Binary classification
Oyama	2019	Japan	MRI	3D	NR	N − 1 = 99	-	100	Different for each group	Binary classification
Wenqi Shi	2020	China	CT	Axial, coronal, and sagittal	NR	359	-	90	Expert consensus, histopathology	Binary classification
S.Murugesan	2020	India	X-ray, CT, MRI, and PET	NR	NR	Image set 1: 137 Image set 2: 73	-	Image set 1: 91 Image set 2: 18	Past study	Binary classification
Cao	2020	China	CT	Axial	NR	410	-	107	Histopathology, patient history, and expert clinicians	Classification
Naeem	2020	Pakistan	CT, MRI	Horizontal	NR	NR	-	21,600	Expert clinicians, medical tests, biopsy report	Classification
Qinghua Huang	2020	China	US	NR	NR	Image set 1: 336 Image set 2: 387	-	Image set 1: 30 Image set 2: 30	Biopsy or surgical resection, expert consensus	Binary classification
Ryu	2021	North korea	US	NR	NR	3909	-	400	Different for each group	Segmentation and classification
Takenaga	2021	Japan	MRI	Horizontal	NR	670	474	402	Expert clinicians and archived reports	Detection and classification
Shanshan Ren	2021	China	US	NR	NR	149	-	38	Expert consensus	Classification
Thodsawit Tiyarattanachai	2021	Thailand	US	NR	NR	40,397	-	6191	Pathology and/or MRI/CT reports	Classification
Mubasher Hussain	2022	Pakistan	CT	NR	NR	NR	-	1000	Expert consensus	Multiclass liver tumor classification
Wei-bin Zhang	2022	China	US	NR	NR	305	-	102	Surgery	Binary classification models
Fatemeh Azimi Nanvaee	2023	Iran	US	NR	NR	-	-	-	Expert consensus	Binary classification
Yating Ling	2023	China	Contrast enhanced CT	NR	NR	481	-	120	Histopathology	Binary classification

CT computed tomography, HCC hepatocellular carcinoma, ICC intrahepatic cholangiocarcinoma, FNH focal nodular hyperplasia, FLL focal liver lesions, LT liver transplantation, MWA microwave ablation, RFA radiofrequency ablation, TACE transcatheter arterial chemoembolization, DCE-CT dynamic contrast-enhanced computed tomography, MRI magnetic resonance imaging, NR not reported, N/A not applicable, PET positron emission tomography, US ultrasound

Table 3.

Study characteristics, external validation

First author	Year	Country	Imaging modality	View	Comparison group	No. of images per set			Reference standard	Model output
						Training	Tuning	Testing
Shanshan Ren	2021	China	US	NR	NR	149	-	38	Expert consensus	Classification
Thodsawit Tiyarattanachai	2021	Thailand	US	NR	NR	40,397	-	18,922	pathology and/or MRI/CT reports	Classification
Meiyun Wang	2021	China	CT	axial	Expert clinicians	Model 1: 7512 Model 2: 7512	-	Model 1: 556 Model 2: 82	Expert consensus	Classification
Seongkeun Park	2023	Korea	MRI	NR	NR	-	245	30	Pathology and/or MRI/CT reports	Binary classification

ICC intrahepatic cholangiocarcinoma, HCC hepatocellular carcinomas, LT liver transplantation, MWA microwave ablation, RFA radiofrequency ablation, TACE transcatheter arterial chemoembolization, CT computed tomography, MRI magnetic resonance imaging, US ultrasound, NR not reported

Study Participants

The number of participants in each study differed substantially (median, 223.5; range, 30–23756; interquartile range, 106–382; Table E4). In addition, the proportion of HCC subjects in each study varied widely (median, 43.2%; range, 8.61–80.22; interquartile range, 21.95–60.54; Table E4). Ten studies failed to report the total number of subjects [40, 44, 48–50, 52, 62, 65, 76, 78]. Also, 12 studies did not report the percentage of HCC subjects in the article [39–41, 44, 45, 47, 49, 52, 53, 58, 59, 78].

Algorithm Advancement and Model Output

Tables 1, 2, and 3 shows the widely ranged number of images per set: training (median, 289.7; interquartile range, 140–670), tuning (median, 50; interquartile range, 50–359.5), and test (median, 100; interquartile range, 48–355). Ten studies did not report these data sets separately [35, 36, 40, 46, 51, 54, 59, 67, 68, 77]. It is notable that only two studies used tuning sets [37, 72]. In addition, Table E4 shows data augmentation use, transfer learning utilization, and the validation method in each study.

Thirty-two studies evaluated their model output with sensitivity and specificity, 32 with accuracy, 20 used area under the receiver operating characteristic curve, 9 utilized F1, and 18 articles used positive and negative predictive values.

Quality Assessment

Study adherence to TRIPOD guidelines was variable (Fig. 2). Six items were insufficiently reported in the 44 included studies (< 50% adherence): eligibility criteria for images (43%), sample size estimation (8%), model interpretability (41%), confidence interval reporting (43%), supplementary information availability (39%), and funding source report (49%).

Fig. 2.

Adherence to Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD), and Prediction Model Study Risk of Bias Assessment Tool (PROBAST) reporting guidelines

As illustrated in Fig. 2, the PROBAST assessment revealed 27 (61%) studies with a high risk of bias and only 5 (11%) articles with high applicability concerns. The key factor to this evaluation was studies with small sample-sized internal validation models or with no external validation. As a result of the inclusion and exclusion criteria of the evaluated studies, patient selection bias was high in 19 (43%) studies. However, every study had a low concern for applicability in this domain. 24 (55%) studies had a high risk of bias for analysis. Studies generally indicated their outcomes with a low concern for bias (37; 84%) and applicability (39; 88%).

Meta-analysis

Forty-four studies provided sufficient data to extract 119 contingency tables used for binary HCC diagnosis (Table 4) [35–78]. We calculated 94 contingency tables from 43 studies evaluating algorithm performance on internal validation, seven tables from 4 articles on external validation, and 18 from 6 studies assessing internally validated clinician performance. The pooled sensitivity and specificity for each class of these studies are shown in Table 4. This table also reveals the pooled metrics in HCC detection for each imaging modality. Analysis was not carried out on studies evaluating externally validated clinician performance, in addition to human performance with AI support, as a result of there being less than three of these studies.

Table 4.

pooled sensitivities, specificities, and area under the curve, positive likelihood ratio, negative likelihood ratio, diagnostic odds ratio

Parameter	Sensitivity	Specificity	AUC	Positive likelihood ratio	Negative likelihood ratio	Diagnostic odds ratio	No. of contingency tables
Algorithms, internal validation, all modalities	0.84 (0.81–0.87)	0.92 (0.9–0.94)	0.94 (0.92–0.96)	10.4 (8.2–13.3)	0.17 (0.14–0.21)	60 (41–87)	94
CT studies	0.86 (0.81–0.90)	0.94 (0.91–0.96)	0.96 (0.94–0.98)	14 (9.9–19.9)	0.15 (0.10—0.20)	97 (54–172)	25
MRI studies	0.77 (0.71–0.82)	0.85 (0.81–0.88)	0.88 (0.85–0.91)	5.1 (4–6.5)	0.27 (0.21–0.35)	19 (13–28)	35
US studies	0.88 (0.83–0.91)	0.95 (0.92–0.97)	0.97 (0.95–0.98)	16.5 (10.7–25.5)	0.13 (0.09–0.18)	127 (69–234)	29
Algorithms, external validation	0.85 (0.78–0.89)	0.84 (0.72–0.91)	0.91 (0.88–0.93)	5.3 (3–9.3)	0.18 (0.13–0.26)	29 (15–57)	7
Clinicians, internal validation	0.70 (0.60–0.78)	0.85 (0.77–0.90)	0.84 (0.81–0.87)	4.5 (2.8–7.3)	0.36 (0.26–0.50)	13 (6–26)	18

AUC area under the curve, CT computed tomography, MRI magnetic resonance imaging, US ultrasound

The main results for each group of studies are shown in Table 4. The pooled sensitivity and specificity for internally validated AI algorithms are 84% (95% CI: 81, 87) and 92% (95% CI: 90, 94), respectively. The subgroup analysis of AI algorithm performance at internal validation showed a pooled sensitivity and specificity 86% (95% CI: 81, 90) and 94% (95% CI: 91, 96) for studies using CT scan (Fig. 3), 77% (95% CI: 71, 82) and 85% (95% CI: 81, 88) for studies using MRI (Fig. 4), and 88% (95% CI: 83, 91) and 95% (95% CI: 92, 97) for studies using US (Fig. 5). In addition, AI algorithms that were validated externally had a pooled sensitivity of 85% (95% CI: 78, 89) and specificity of 84% (95% CI: 72, 91). When clinicians were internally validated, their pooled sensitivity was 70% (95% CI: 60, 78), while their pooled specificity was 85% (95% CI: 77, 90).

Fig. 3.

Forest plots of algorithms performance at internal validation for studies using computed tomography. The plot shows the effect size estimate (diamond symbol), confidence interval (horizontal line), and the individual study effect sizes (square symbols) with their corresponding weights

Fig. 4.

Forest plots of algorithms performance at internal validation for studies using magnetic resonance imaging. The plot shows the effect size estimate (diamond symbol), confidence interval (horizontal line), and the individual study effect sizes (square symbols) with their corresponding weights

Fig. 5.

Forest plots of algorithms performance at internal validation for studies using ultrasonography. The plot shows the effect size estimate (diamond symbol), confidence interval (horizontal line), and the individual study effect sizes (square symbols) with their corresponding weights

Meta-regression was only assessed in one parameter, data augmentation use, the only variable with a sufficient number of studies (Table E7). Our analysis revealed that studies using data augmentation had a sensitivity of (82%; 95% CI: 76,88; P, 0.00) and specificity of (94%; 95% CI: 91,97; P, 0.00). Surprisingly, articles that did not augment data had higher sensitivity (85%; 95% CI: 82, 88; P, 0.00) and lower specificity (91%; 95% CI: 89, 93; P, 0.00).

Publication Bias

We performed publication bias analysis on studies assessing algorithm development at internal validation, external validation, and studies evaluating clinician performance on internal validation test sets. The slope coefficient was − 19.74 (95% CI: − 35.59, − 3.90; P = 0.015), − 3.51 (95% CI: − 19.61, 12.586; P = 0.59), and − 33.98 (95% CI: − 62.77, − 5.199; P = 0.024), respectively. As shown, there is a high risk of publication bias in studies evaluating internally validated algorithms (P = 0.015) and internally evaluating clinician performance (P = 0.024). The publication bias for each imaging modality in these studies can be seen in Table E8.

Discussion

AI has recently become a rapidly expanding field and a growing number of studies are evaluating the potential of AI in the detection, classification, prediction, and prognosis of HCC. Hence, we performed the first systematic review and meta-analysis which pools the diagnostic performance measures of AI and clinicians in HCC extensively. To summarize, our analysis pooled the internally and externally validated AI algorithm results of studies and compared them with clinician performance at internal validation. Our study revealed that AI algorithms exceeded clinicians at internal validation with a pooled sensitivity of 84% (95% CI: 81, 87) compared to 70% (95% CI: 60, 78) and a pooled specificity of 92% (95% CI: 90, 94) against 85% (95% CI: 77, 90). Furthermore, out of the 16 respectively. However, these results should be interpreted with caution as it is likely that most studies underestimate clinician diagnostic performances, because the included studies mainly used clinicians with a low level of experience [46, 57, 58, 73, 74, 79]. Therefore, this finding needs clarification from future studies by internally validating algorithms in clinical environments and fairly comparing them with a large number of relevant experienced clinicians to decrease the risk of interrater variation. Also, the subgroup analysis for AI algorithms at internal validation based on the imaging modality revealed a sensitivity and specificity of 86% (95% CI: 81, 90) and 94% (95% CI: 91, 96) for CT scan studies, respectively. MRI studies had a sensitivity of 77% (95% CI: 71, 82) and a specificity of 85% (95% CI: 81, 88). We also performed a subgroup analysis on US studies, revealing a surprisingly higher sensitivity and specificity of 88% (95% CI: 83, 91) and 95% (95% CI: 92, 97), respectively. The high publication bias of internally validated studies shown in Table E8 could have resulted in this counterintuitive finding studies that utilized US as their imaging modality, only the studies by Ren et al. and Tiyarattanachai et al. externally validated their algorithms [80, 81]. It is notable that none of the included studies applied more than one modality to compare the diagnostic performance of AI algorithms between different imaging techniques. This implies the need for further studies utilizing different modalities with externally validated algorithms and lower publication bias to shed more light on this finding. Analysis of AI test sets at external validation showed a pooled sensitivity of 85% (95% CI: 78, 89) and a pooled specificity of 84% (95% CI: 72, 91).

Two sets of studies did not reach the required number (i.e., less than three) to perform a meta-analysis. First, studies evaluating externally validated clinician performance [58, 79]. Kim et al. [79] reported a sensitivity of 98%, specificity of 93%, and accuracy of 94%. Wang et al. reported different diagnostic metrics for each radiologist separately [58]. Due to this insufficient number of studies, comparing these results with AI at external validation is not possible. External validation is an important step in validating diagnostic studies. It can aid in recognizing the need for model updating or recalibration. On the other hand, internal validation can overstate algorithm performance, leading to an insufficiency in model output generalizability [82]. Therefore, further externally validated clinicians in randomized clinical trials are needed for this comparison, to better compare AI with clinicians in the clinical environment. Second, Wang et al. also revealed HCC diagnostic assessments for three clinicians using AI assistance [58]. While both studies by Kim et al. and Wang et al. were associated with a low level of risk of bias in PROBAST and TRIPOD checklists, further studies are needed to shed more light on the diagnostic performance of clinicians with AI assistance and compare them with clinicians and AI algorithms, separately [58, 79].

The limitations of this study should be reviewed when interpreting its results. First, not all studies evaluating AI algorithm’s HCC diagnostic performance reported sufficient data to prepare a contingency table and perform a meta-analysis and, thus, were excluded from this study. We would recommend the future studies to provide diagnostic performance metrics (i.e., the number of TP, TN, FP, and FN cases), to be able to be included in large meta-analysis studies. Second, while adherence to TRIPOD guidelines was mostly acceptable, multiple articles did not report essential study details such as training and testing set information. Third, more than half the studies had a high concern for bias, meaning they overestimated algorithm performance, which may limit the conclusions of the meta-analysis. Last but not least, we faced a high level of heterogeneity between studies. As a result, we tried to perform meta-regression and subgroup analyses to control for the factors responsible for the high level of heterogeneity, and we were able to do so for the imaging modalities and data augmentation use. However, the trait of control group among the included studies were highly heterogeneous, and hepatic lesion classification studies mainly had different comparators, for example, hemangiomas [35–40, 44–46, 49, 51–53, 56, 58, 59, 65, 66, 68, 69, 71–73, 77], hepatic cysts [37, 39, 40, 44–46, 48, 49, 58, 59, 65, 66, 68, 71, 72, 77], metastatic carcinomas [35–40, 45, 46, 49, 53, 55, 56, 58, 61, 66–69, 72, 73], focal nodular hyperplasia [42, 47, 49, 56, 58, 71, 73, 76], and intrahepatic cholangiocarcinoma [43, 49, 57, 58, 64, 65, 71, 73, 74]. Moreover, most of these studies had a lack of necessary data to build contingency tables for each comparator separately. On the other hand, the number of HCC detection studies with a healthy control group that reported sufficient data for contingency table construction needed to be more adequate for meta-analysis. Consequently, we were not able to differ between HCC detection and hepatic lesion classification studies and perform a subgroup analysis based on different comparators. However, this diversity of study comparators may show an appropriate estimate of real-world applicability for HCC diagnosis in the clinical setting, where a variety of patients with different hepatic anomalies refer. Thus, we suggest that future studies use a mixed variety of control groups, to implicate real-life utilization of AI, and also report, separately for each comparator, the required information for contingency table construction.

In conclusion, this study cautiously implies that AI outperforms clinicians in HCC diagnosis at internal validation. The pooled sensitivity and specificity for internally validated AI algorithms were 84% (95% CI: 81, 87) and 92% (95% CI: 90, 94), respectively. Externally validated AI algorithms had a pooled sensitivity of 85% (95% CI: 78, 89) and specificity of 84% (95% CI: 72, 91). When clinicians were internally validated, their pooled sensitivity was 70% (95% CI: 60, 78), while their pooled specificity was 85% (95% CI: 77, 90). However, as discussed before, future studies must utilize clinicians with real-life clinical backgrounds at external validation with separate results for each comparator. Most of the included studies were done on small-scale datasets in high-income developed countries with very little data from lower middle- and low-income countries, which obscures their credibility. Addressing these issues in future studies are necessary steps forward toward clinical application. Nevertheless, at the time of this review, AI can perform as a diagnostic supplement for clinicians and radiologists by screening images and highlighting regions of interest, thus improving workflow. It may also act as a second reader after radiologists, improving diagnostic certainty. AI-assisted diagnosis of HCC offers the potential for early, accurate and consistent detection, leading to improved patient outcomes, reduced healthcare costs, and increased accessibility to quality healthcare services. In addition, the advancements in AI radiomics and the findings of this study support the potential of AI to enhance the diagnostic ability of imaging tools, improve patient management, and contribute to the concept of precision medicine in HCC. However, this requires a thorough clinician understanding of AI performance through transparent reports of future study methods and results to precisely interpret algorithm outputs.

Supplementary Information

Below is the link to the electronic supplementary material.

Funding

None.

Data Availability

The data that support the findings of this study are available from the authors upon reasonable request.

Declarations

Ethics Approval

Not applicable.

Competing Interest

The authors declare no competing interests.