A Deep Learning Algorithm for Liver Metastasis Detection at Contrast-enhanced Abdominal CT in Patients with Colorectal Cancer: A Comparative Study with Radiologists

Abstract

Purpose

To evaluate the performance of a deep learning algorithm (DLA) for detecting liver metastases (LM) in patients with colorectal cancer (CRC) across diverse clinical contexts and compare its accuracy with that of radiologists.

Materials and Methods

This retrospective, bicentric study included patients with CRC who underwent contrast-enhanced abdominal CT between January 2019 and December 2021. The DLA accuracy was assessed at the per-nodule and per-patient levels and compared with that of a senior (R1) and an in-training (R2) radiologist blinded to each other’s results. The LM detection and false detection rates and interreader agreement were determined.

Results

Among 181 patients with CRC (mean age, 64 years ± 13 [SD]; 102 male), 95 had LM and 86 had no LM. In the per-nodule analysis, the DLA LM detection rate was 81% (227 of 280; 95% CI: 76.1, 85.2), with no difference compared with R1 (79%; 222 of 280; 95% CI: 74.2, 83.6; P = .49) or R2 (76%; 214 of 280; 95% CI: 71.1, 81.0; P = .19). Detection rates of DLA increased with lesion size: less than 10 mm, 55% (51 of 93; 95% CI: 44.7, 64.6); 10–19 mm, 91% (96 of 106; 95% CI: 83.5, 94.8); and 20 mm or more, 99% (80 of 81; 95% CI: 93.3, 99.8). Detection of subcapsular LM was comparable across readers (DLA, 90% [113 of 125; 95% CI: 84.0, 94.4]; R1, 91% [114 of 125; 95% CI: 84.9, 95.0]; R2, 89% [111 of 125; 95% CI: 82.1, 93.2]). False detection rates were low (DLA, 22% [39 of 181; 95% CI: 16.2, 28.1]; R1, 20% [37 of 181; 95% CI: 15.2, 26.9]; R2, 26% [47 of 181; 95% CI: 20.1, 32.8]; DLA vs R1, P = .004; DLA vs R2, P = .01). DLA false positives were mainly biliary dilatations (n = 14) and diaphragmatic indentations (n = 12). Interreader agreement was moderate (κ = 0.63–0.75).

Conclusion

DLA performance did not differ from radiologists in detecting LM, with consistent results across lesion sizes and locations.

Keywords: Imaging Modality, Abdomen, Gastrointestinal, Liver, Oncology, Comparative Studies, Segmentation, Diagnosis, Deep Learning

Supplemental material is available for this article.

© RSNA, 2026

See also commentary by Mukherjeein this issue.

Summary

A deep learning algorithm achieved liver metastasis detection rates on contrast-enhanced abdominal CT scans comparable to those of radiologists, with consistent performance across different lesion sizes, locations, and clinical conditions.

Key Points

• ■ A deep learning algorithm (DLA) identified 81% of liver metastases on abdominal contrast-enhanced CT, with no evidence of a difference in detection rates between the DLA and a senior radiologist (R1, 79%; P = .49) or an in-training radiologist (R2, 76%; P = .19).

• ■ Detection rates of DLA increased with lesion size: 55% (51 of 93), 91% (96 of 106), and 99% (80 of 81) for lesions of less than 10 mm, 10–19 mm, and 20 mm or more, respectively.

• ■ False detection rates were low (DLA, 22% [39 of 181]; R1, 20% [37 of 181]; R2, 26% [47 of 181]), with DLA mainly misidentifying biliary dilatations and diaphragmatic indentations (14 and 12 cases, respectively), whereas radiologists more often misidentified heterogeneous parenchymal changes (23 and 30 cases for R1 and R2, respectively).

Introduction

Colorectal cancer (CRC) is the third most common solid cancer worldwide (1). The liver is the most common metastatic site, with 25% of patients showing synchronous and up to 50% developing metachronous liver metastases (LM) (2–5). CT is the primary tool for detecting LM, but sensitivity remains moderate (72%–82%) (6,7). Overlooked LM has been related to low liver-to-lesion contrast, subcapsular location, hepatic steatosis (8), and human factors (eg, fatigue, anxiety, and lapses in concentration) (9).

To overcome these limits, artificial intelligence has been proposed to improve LM detection, particularly through deep learning–based approaches. Deep learning algorithms (DLAs) have emerged as a promising tool to address certain limitations of human interpretation in medical imaging. In thoracic imaging, DLAs have demonstrated considerable improvements in both lung nodule detection and identification rates (10).

DLA applications are under active investigation in liver CT imaging, yet few studies are focused on automatic lesion detection (11,12). These studies include a wide range of focal liver lesions (both benign and malignant), encompassing hyper- and hypoenhancing patterns. Reported diagnostic performances range from 82% to 86% (11,13). Since most CRC LM are hypoenhancing on CT scans, DLAs specifically designed to detect such imaging patterns could achieve superior diagnostic performance. This possibility is supported by the work focused on LM automatic detection at CT by Nakai et al (14), who reported a detection rate of 92.7% among patients with confirmed metastatic disease. However, the authors did not further analyze the relationship between detection rate and metastasis characteristics, such as location, contrast, size, and false positives. Furthermore, their comparison with human reader performance was made retrospectively on the available reports, adding an important interpretation bias, as pointed out by the authors. Further studies have shown a strong relationship between the detection rate and intrinsic characteristics of LM at imaging (15).

In our study, we aimed to assess and validate the performance of a DLA for the automatic detection of LMs in patients with CRC across diverse clinical contexts and compare its accuracy with that of radiologists.

Materials and Methods

Study Patients

Our bicentric observational retrospective study, including chart reviews, was approved by the institutional review board of each center (center 1: CHU d’Angers, Angers, France; center 2: Hôpital Beaujon-APHP, Clichy, France), and the requirement to obtain written informed consent was waived. We included consecutive patients with a histologically confirmed diagnosis of CRC who underwent contrast-enhanced abdominal CT between January 2019 and December 2021 in various clinical contexts (ie, at diagnosis, before chemotherapy, and before liver surgery).

CT examinations were performed within the standard indications of the patient’s pathway: staging at diagnosis or during follow-up or evaluation before potential surgical treatment. Clinical data, laboratory results, and treatment history were collected from medical records. Detailed CT indications and time points are provided in Table S1.

Exclusion criteria were the absence of a portal venous phase CT, prior liver surgery or local-regional treatment, a known concomitant malignancy, and a follow-up shorter than 12 months for patients with no LM. We arbitrarily excluded patients with more than 20 liver nodules or more than 10 LMs to avoid annotation bias and ensure statistical consistency, as such cases often involve diffuse disease with limited clinical impact of per-nodule detection and pose challenges for reliable lesion-by-lesion analysis.

We grouped patients into those without or with LM, the latter being further subdivided into patients with synchronous and metachronous metastases. For patients who underwent liver surgery, we collected pathology data for additional analyses. The patient selection flowchart is presented in Figure 1.

Figure 1:

Study flowchart. LM = liver metastasis.

Reference Standard

In line with European Organization for Research and Treatment of Cancer, European Society of Oncologic Imaging, and European Society of Gastrointestinal and Abdominal Radiology joint recommendations (16), we adopted a rigorous multiparametric approach to classify liver lesions as metastases in the absence of pathology confirmation. The reference standard was established in consensus by two senior radiologists (R.S. and A.P., with 8 and 10 years of experience in liver imaging, respectively), who had access to the entire clinical and imaging dataset, including baseline and follow-up CT, liver MRI, contrast-enhanced US, PET/CT, surgical reports, clinical data, surgical records, histopathologic results, and follow-up information. Any disagreements were handled by involving a third senior radiologist (C.A.), whose assessment was used to reach a majority decision. For nonbiopsied lesions, LM were defined on the basis of typical imaging features across modalities, drawing on evidence from van der Reijd et al (17) and expert consensus guidelines (16). At MRI, typical features included hypovascularity, unenhanced T1-weighted hypointensity, arterial rim enhancement, mild T2-weighted hyperintensity, diffusion restriction, and absence of enhancement in the hepatobiliary phase. These patterns have been shown to retain high diagnostic value after chemotherapy (17). In line with established oncologic PET practice, we considered a standardized uptake value of more than 2.5 indicative of suspicious fluorodeoxyglucose uptake, both as a visual and semiquantitative threshold, and we applied this threshold to support LM diagnosis when fluorodeoxyglucose-avid foci corresponded to focal lesions at CT or MRI (16). We also considered the evolution of indeterminate nodules during follow-up: We classified lesions that showed an increase in size, morphologic changes, or response to systemic therapy as metastases. For metachronous metastases in particular, the diagnosis was supported by the appearance of new lesions not visible on baseline imaging, in the context of disease progression or an increase in tumor marker levels. In patients without histologic confirmation, we considered lesions LM if their imaging characteristics mirrored those of biopsy-proven lesions and their behavior over time (eg, growth, response to treatment, fluorodeoxyglucose avidity) was consistent with metastatic disease.

We considered lesions with size stability for at least 24 months without chemotherapy nonmetastatic lesions (ie, benign nodules). Details concerning reference standard evaluation are provided in Table S2.

Analysis Derived from CT Images

Details about the CT acquisition protocols of both centers are provided in Table S3. Quantitative evaluation of liver and liver nodule attenuation was performed in consensus for all patients during reference standard analysis. Liver attenuation was measured with three circular regions of interest (2-cm diameter) placed in both liver lobes, avoiding vessels and lesions; the mean value was used. Marked hepatic steatosis at the portal venous phase was defined as a liver attenuation of less than 80 HU (18). The LM/liver ratio was calculated by dividing the mean attenuation of each lesion (drawing a manual freehand region of interest at the largest diameter) by the mean liver attenuation. The distance between each lesion and the liver capsule was manually measured in consensus by the two senior radiologists on axial, coronal, or sagittal planes, recording the shortest distance between the lesion edge and the capsule. The largest axial diameter was also measured manually on the axial plane with electronic calipers, according to the Response Evaluation Criteria in Solid Tumors 1.1. Two radiologists, different from those who established the reference standard (S.M. and F.M., one senior and one in training with 7 and 3 years of experience in liver oncology imaging, referred to as R1 and R2, respectively), independently reviewed only the portal venous phase of all CT examinations presented in random order. These radiologists were blinded to clinical information and to each other’s assessments but were aware of the CRC diagnosis. They reported and located all detected liver nodules on a standardized diagram to facilitate comparison with the reference standard. Analysis was performed at the per-nodule and per-patient levels.

DUOnco for Liver

DUOnco Liver (formerly “Care Advisor for Liver”), the DLA we evaluated in our study, is software developed by Guerbet. This software uses a convolutional neural network to detect liver lesions on Digital Imaging and Communications in Medicine images and generate regions of interest to highlight identified areas for radiologists. The software automatically segments the liver volume and detects focal liver lesions on the portal venous phase of CT scans. Nodules are presented as annotated Digital Imaging and Communications in Medicine series integrated into the picture archiving and communication system. The tool performs detection only; it does not classify lesions as benign or malignant. Final characterization is left to the radiologist. The main algorithmic component of the software consists of an ensemble of two U-Nets trained on a database of 3357 patients with various liver conditions. Additional details and characteristics of the training dataset are available in the patent documentation and in previously published work (14,19).

Statistical Analysis

Statistical analysis was performed by two authors (R.S. and A.B.). Categorical variables are presented as absolute frequencies and percentages, and continuous variables are presented as means ± SDs or medians (IQRs), depending on distribution (distribution normality was assessed with the Shapiro-Wilk test). We framed the main comparisons between the DLA and the radiologists (R1, R2) for sensitivity and false detection rate as noninferiority analyses. For sensitivity, our null hypothesis was that DLA sensitivity would be inferior by more than 5%, with the alternative being noninferiority. For the false detection rate, our null hypothesis was that DLA would produce at least 0.1 more false positives per case than the comparator, with the alternative being noninferiority.

We defined the test statistic as the observed difference in proportions shifted by the corresponding noninferiority margin. We assessed significance using permutation tests with 100 000 random label reallocations to approximate the null distribution, and we computed empirical one-sided P values. For sensitivity, we also performed large-sample z tests as a complementary approximation. By definition, P < .05 led to rejection of the null of inferiority and supported a conclusion of noninferiority. In all other comparisons, the null hypothesis tested equality between groups against the alternative of any difference.

Depending on variable type, we used χ² or Fisher exact tests for categorical variable comparison and the Student t test or Mann-Whitney U test for continuous variable comparison. We quantified agreement between DLA and R1 and R2 for liver nodule detection using the Cohen κ, where values of 0.00–0.20, 0.21–0.40, 0.41–0.60, 0.61–0.80, and 0.81–1.00 indicated slight, fair, moderate, substantial, and almost perfect agreement, respectively.

We assessed interreader agreement by calculating the Cohen κ statistic, based on binary detection (1 or 0) of each reference standard nodule. We computed the agreement pairwise across readers and stratified by lesion characteristics. We defined false positives as areas the software or radiologists identified that were neither benign nor malignant lesions but represented normal liver parenchyma or anatomic structures (eg, areas affected by perfusion disorder, focal steatosis) and heterogeneous liver parenchyma.

Reported P values correspond to the procedure applied (noninferiority or equality) and are explicitly indicated in tables and figure legends.

Per-patient and per-nodule detection metrics were computed for each reader and for the DLA, including precision, recall, and F1 score. Precision was defined as true positive/(true positive + false positive), recall as true positive/(true positive + false negative), and F1 score as the harmonic mean of precision and recall.

The 95% CIs were calculated for all proportions and performance estimates, including sensitivity, false detection rates, and interreader agreement metrics.

We performed all statistical analyses with Python, version 3.8 (Python Software Foundation), and SPSS, version 27 (IBM). Venn diagrams were redrawn from DeepVenn.com.

Results

Patient Characteristics

A total of 380 consecutive patients with histologically confirmed CRC who underwent contrast-enhanced abdominal CT between January 2019 and December 2021 were initially screened. Ninety-nine patients were excluded for history of prior liver surgery or liver local-regional treatment, 35 patients without LM had an imaging follow-up shorter than 12 months, and 65 patients presented with 20 or more liver nodules or 10 or more LM, according to protocol-defined exclusion criteria.

The final study cohort therefore included 181 patients (mean age, 64 years ± 13 [SD]; 102 male, 79 female). Among them, 86 patients (mean age, 64 years ± 13; 46 male, 40 female) had no LM, and 95 patients (mean age, 65 years ± 12; 56 male, 39 female) had at least one LM (LM per patient: mean, 2.9 ± 2.3 and median, 2 (IQR, 1–4); LM size: median, 14 mm [IQR, 8–23 mm]). Chemotherapy was administered to 113 of 181 (62.4%) patients before CT examination. Liver resection was performed for 38 patients, and a pathology diagnosis was available for 120 LM. A summary of patient characteristics is presented in Table 1.

Table 1:

Patient and Lesion Characteristics

Characteristic	Total Patients (n = 181)	Patients with Liver Metastasis (n = 95)	Patients without Liver Metastasis (n = 86)	P Value
Patients
Sex				.89
Female	79 (44)	39 (41)	40 (47)
Male	102 (56)	56 (59)	46 (53)
Age (y)	64.3 ± 12.6 (29–88)	64.5 ± 12.2 (29–88)	64.2 ± 12.6 (33–87)	.87
Female	63.5 ± 13.7 (29–87)	62.2 ± 13.4 (29–87)	64.9 ± 15.1 (33–87)	.40
Male	65.0 ± 11.2 (35–88)	66.1 ± 11.3 (38–88)	63.7 ± 10.9 (35–85)	.28
BMI	25.4 ± 4.7 (19.7–36.8)	24.3 ± 6.5 (19.7–33.9)	25.8 ± 4.4 (20.2–36.8)	.07
History of chemotherapy	113 (62)	58 (61)	55 (64)	.68
Liver appearance
Normal	155 (86)	77 (81)	78 (91)	.09
Steatosis	18 (10)	14 (15)	4 (5)	.03
Heterogeneous	8 (4)	4 (4)	4 (5)	.75
Center 1	55 (30)	30 (32)	25 (29)	.14
Center 2	71 (39)	42 (44)	29 (34)
Other centers*	55 (30)	23 (24)	32 (37)
Lesions				.11
Liver metastasis	280/634	280/445	0
Benign lesions	354/634	165/445	189/189
Hepatic cysts	294/354	135/165	159/189
Hepatic angioma	30/354	20/165	10/159
Calcification	21/354	4/165	17/159
FNH	7/354	5/165	2/159
Hepatic adenoma	2/354	1/165	1/159

Note.—Data are numbers with percentages in parentheses or means ± SDs with ranges in parentheses. P values were calculated with the χ² or Fisher exact test for categorical variables and the Student t test or Mann-Whitney U test for continuous variables, as appropriate. BMI = body mass index (calculated as weight in kilograms divided by height in meters squared), FNH = focal nodular hyperplasia.

^*Other centers include CT examinations performed outside centers 1 and 2.

Patients with LM were further divided into synchronous or metachronous metastases groups, including 58 and 37 patients, respectively. In patients without LM, the median follow-up time to confirm LM absence was 38 months (95% CI: 35, 41).

Reference Standard Analysis

A total of 280 LM and 354 benign nodules (BNs) were identified by the reference standard analysis. Detailed characteristics are provided in Table 2 and Table S4 for LM and BNs, respectively. Overall, there were 193 LM in the synchronous LM group and 87 in the metachronous LM group (mean size, 19.1 mm ± 17.5). We found no evidence of a difference in mean LM size between synchronous and metachronous LM (19.1 mm ± 16.3 vs 19.2 mm ± 13.8; P = .96). Ninety-three LM were less than 10 mm, 106 ranged in size between 10 and 19 mm, and 81 were 20 mm or more. The mean CT attenuation of LM was 85.1 HU ± 16.1, with a mean LM/liver ratio of 0.75 HU ± 0.14 (median, 0.73 HU; IQR, 0.65–0.83 HU).

Table 2:

Liver Metastasis Characteristics

Characteristic	Total (n = 280)	Synchronous LM (n = 193)	Metachronous LM (n = 87)	P Value
LM per patient
Mean	2.9 ± 2.3 (1–10)	3.5 ± 2.5 (1–10)	2.6 ± 2.3 (1–9)	.08
Median	2 (1–4)	3 (1–5)	2 (1–3)	.07
LM size (mm)
Mean	19.1 ± 17.5 (2–125)	19.1 ± 16.3 (3–125)	19.2 ± 13.8 (2–72)	.96
Median	14 (8–23)	18 (9–27)	14 (9–24)
Mean LM attenuation (HU)	85.1 ± 16.1 (36.3–133.9)	86.0 ± 30.9 (51.1–133.9)	83.1 ± 28.8 (36.3–120.5)	.46
LM/liver ratio
Mean (HU)	0.75 ± 0.14 (0.34–1.13)	0.74 ± 0.14 (0.46–1.13)	0.76 ± 0.14 (0.34–1.09)	.27
Median (HU)	0.73 (0.65–0.83)	0.72 (0.65–0.83)	0.75 (0.67–0.85)	.29
LM size				.26
≤4 mm	11	10	1
5–9 mm	82	61	21
10–14 mm	63	40	23
15–19 mm	43	27	16
≥20 mm	81	55	26
History of chemotherapy	161 (58)	104 (54)	57 (66)	.006
Segment				.16
1	5	1	4
2	26	16	10
3	24	19	5
4	44	29	15
5	33	25	8
6	40	27	13
7	56	36	20
8	52	40	12
Mean distance from capsule (mm)	4.4 ± 6.2 (0–36)	4.6 ± 6.4 (0–36)	3.9 ± 6.7 (0–35)	.40
Contact with capsule	125 (45)	84 (44)	41 (47)	.64

Note.—Data are numbers, numbers with percentages in parentheses, means ± SDs with ranges in parentheses, or medians with IQRs in parentheses. P values were calculated with the χ² or Fisher exact test for categorical variables and the Student t test or Mann-Whitney U test for continuous variables, depending on data distribution. LM = liver metastasis.

A total of 125 (45%) LM were in contact with the liver capsule, 50 (18%) LM were in the left lateral segment (segments 2 and 3), and 230 (82%) were in the right lobe. A total of 104 of 193 (53.8%) LM and 57 of 86 (66%) LM in the synchronous and metachronous LM groups were present in 58 and 55 patients with a history of chemotherapy, respectively (P = .006).

BNs consisted of 294 of 354 (83.1%) simple hepatic cysts, 30 of 354 (8.5%) hepatic hemangiomas, 21 of 354 (5.9%) isolated hepatic calcifications, seven of 354 (2.0%) focal nodular hyperplasias, and two of 354 (0.6%) hepatocellular adenomas.

We found no evidence of a difference in BN distribution according to segment localization (P = .62), size (P = .08), or history of chemotherapy between groups with and without LM (P = .42), whereas we observed a difference according to lesion type (P = .01).

Per-Nodule Analysis

Overall, the DLA, R1, and R2 detected 427 of 634 (67%), 469 of 634 (74%), and 433 of 634 (68%) liver nodules, respectively, with no evidence of a difference in the detection rate between the DLA and R1 or R2 (DLA vs R1, P = .01; DLA vs R2, P = .76).

The liver nodule detection rates according to lesion size, contact with the liver capsule, and history of chemotherapy are presented in Table 3. The LM and BN detection rates for the DLA, R1, and R2 according to lesion size and different situations are presented in Figure 2. The DLA, R1, and R2 detected 227 of 280 (81%; 95% CI: 76.1, 85.2), 222 of 280 (79%; 95% CI: 74.2, 83.6), and 214 of 280 (76%; 95% CI: 71.1, 81.0) LM, respectively, with no evidence of differences in detection rates between the DLA and R1 or R2 (DLA vs R1, P = .49; DLA vs R2, P = .19). The detection rate increased with tumor size for the DLA, R1, and R2, ranging from 55% (95% CI: 44.7, 64.6), 51% (95% CI: 40.6, 60.5), and 44% (95% CI: 34.4, 54.2), respectively, for LM of less than 10 mm to 99% (95% CI: 93.3, 99.8), 99% (95% CI: 93.3, 99.8), and 98% (95% CI: 91.4, 99.3), respectively, for LM of 20 mm or more.

Table 3:

Liver Metastasis Detection Rates for the Deep Learning Algorithm, Senior Radiologist, and In-Training Radiologist, according to Lesion Size, Contact with Liver Capsule, and Previous Chemotherapy

Characteristic	All LM			Synchronous LM			Metachronous LM
	DLA	R1	R2	DLA	R1	R2	DLA	R1	R2
All LM	81.1 (227/280) [76.1, 85.2]	79.3 (222/280) [74.2, 83.6]	76.4 (214/280) [71.1, 81.0]	82.4 (159/193) [76.4, 87.3]	77.2 (149/193) [70.7, 82.7]	75.6 (146/193) [69.0, 81.4]	78.2 (68/87) [68.4, 85.8]	83.9 (73/87) [74.8, 90.2]	78.2 (68/87) [68.4, 85.8]
DLA vs (%)	…	+1.8 (P5% = .002)	+4.6 (P5% < .001)	…	+5.2 (P5% = .006)	+6.7 (P5% = .002)	…	−5.7 (P5% = .55)	+0.0 (P5% = .21)
0–9 mm	54.8 (51/93) [44.7, 64.6]	50.5 (47/93) [40.6, 60.5]	44.1 (41/93) [34.4, 54.2]	54.9 (39/71) [43.0, 66.3)]	46.5 (33/71) [34.9, 58.4]	42.3 (30/71) [31.0, 54.5]	54.5 (12/22) [33.7, 73.7]	63.6 (14/22) [42.2, 81.2]	50.0 (11/22) [29.9, 70.1]
DLA vs (%)	…	+4.3 (P5% = .10)	+10.7 (P5% = .02)	…	+8.5 (P5% = .05)	+12.7 (P5% = .02)	…	−9.1 (P5% = .61)	+4.5 (P5% = .26)
10–19 mm	90.6 (96/106) [83.5, 94.8]	89.6 (95/106) [82.4, 94.1]	88.7 (94/106) [81.2, 93.4]	97.0 (65/67) [89.9, 99.2]	91.0 (61/67) [82.0, 95.9]	91.0 (61/67) [82.0, 95.9]	79.5 (31/39) [64.5, 89.4]	87.2 (34/39) [73.6, 94.5]	84.6 (33/39) [70.2, 93.0]
DLA vs (%)	…	+0.9 (P5% = .07)	+1.9 (P5% = .05)	…	+6.0 (P5% = .003)	+6.0 (P5% = .003)	…	−7.7 (P5% = .63)	−5.1 (P5% = .50)
20+ mm	98.8 (80/81) [93.3, 99.8]	98.8 (80/81) [93.3, 99.8]	97.5 (79/81) [91.4, 99.3]	100.0 (55/55) [93.5, 100]	100.0 (55/55) [93.5, 100]	100.0 (55/55) [93.5, 100]	96.2 (25/26) [81.7, 99.3]	96.2 (25/26) [81.7, 99.3]	92.3 (24/26) [76.5, 98.0]
DLA vs (%)	…	+0.0 (P5% < .001)	+1.2 (P5% < .001)	…	0 (P5% < .001)	0 (P5% < .001)	…	+0.0 (P5% = .17)	+3.8 (P5% = .09)
Contact with capsule	90.4 (113/125) [84.0, 94.4]	91.2 (114/125) [84.9, 95.0]	88.8 (111/125) [82.1, 93.2]	92.9 (78/84) [84.9, 96.9]	91.7 (77/84) [83.3, 96.3]	89.3 (75/84) [80.2, 94.5]	85.4 (35/41) [71.2, 93.4]	90.2 (37/41) [76.8, 96.6]	87.8 (36/41) [73.8, 95.0]
DLA vs (%)	…	−7.5 (P5% = .17)	−5.1 (P5% = .08)	…	+1.2 (P5% = .07)	+3.6 (P5% = .03)	…	−4.9 (P5% = .49)	−2.4 (P5% = .37)
LM in patients with a history of chemotherapy	86.3 (139/161) [80.2, 90.8]	83.2 (134/161) [76.7, 88.2]	81.4 (131/161) [74.6, 86.6]	86.5 (90/104) [78.1, 92.1]	78.8 (82/104) [69.7, 85.9]	79.8 (83/104) [70.8, 86.8]	86.0 (49/57) [74.5, 92.9]	91.2 (52/57) [80.7, 96.5]	84.2 (48/57) [72.1, 91.9]
DLA vs (%)	…	+3.1 (P5% = .02)	+5.0 (P5% = .007)	…	+7.7 (P5% = .007)	+6.7 (P5% = .01)	…	−5.3 (P5% = .51)	−1.8 (P5% = .16)

Note.—Data are percentages with numerators/denominators in parentheses and 95% CIs in brackets and ± numbers that correspond to differences in detection rates with P_5% in parentheses. P values correspond to noninferiority tests of sensitivity between the deep learning algorithm (DLA) and radiologists, with a 5% margin. LM = liver metastasis, P_5% = noninferiority P value at the 5% margin, R1 = senior radiologist, R2 = in-training radiologist.

Figure 2:

Per-nodule detection rate shown in bar graphs of (A) liver metastasis (LM), (B) benign nodules (BNs), and (C) total liver nodules (LNs) according to size and in spider graphs of (D) LM and (E) BN detection rates according to different situations. DLA = deep learning algorithm, FNH = focal nodular hyperplasia, R1 = senior radiologist, R2 = in-training radiologist.

Per-patient and per-nodule detection metrics (precision, recall, F1 score) are presented in Table S5 and Figure S1; detailed characteristics of detected and undetected LM are presented in Table S6.

For lesions in contact with the liver capsule (n = 125), we found no evidence of a difference in detection rates between the DLA and R1 or R2, with detection rates of 90% (113 of 125; 95% CI: 84.0, 94.4) for the DLA, 91% (114 of 125; 95% CI: 84.9, 95.0) for R1, and 89% (111 of 125; 95% CI: 82.1, 93.2) for R2 (DLA vs R1, P = .17; DLA vs R2, P = .08). Similarly, we found no evidence of a difference in LM detection rates between the DLA and R1 or R2 in patients with or without prior chemotherapy (with chemotherapy [n = 161]: DLA, 86% [95% CI: 80.2, 90.8]; R1, 83% [95% CI: 76.7, 88.2]; R2, 81% [95% CI: 74.6, 86.6]; and without chemotherapy [n = 119]: DLA and R1, 74% [95% CI: 65.4, 81.0; DLA vs R1, P > .99]; R2, 70% [95% CI: 61.0, 77.3; DLA vs R2, P = .74]).

Among 354 BNs, the DLA detected 200 (57%; 95% CI: 51.3, 61.6), R1 detected 247 (70%; 95% CI: 64.8, 74.3), and R2 detected 237 (67%; 95% CI: 61.9, 71.6) BNs (DLA vs R1, P = .003; DLA vs R2, P = .18). For the 294 liver cysts, detection rates were 61% (95% CI: 55.3, 66.2) for the DLA, 73% (95% CI: 67.4, 77.8) for R1, and 65% (95% CI: 59.0, 70.5) for R2 (DLA vs R1, P = .02; DLA vs R2, P = .32). The DLA failed to detect any of the 21 hepatic calcifications, whereas R1 and R2 identified 12 of 21 (57%; 95% CI: 36.5, 75.5) and six of 21 (29%; 95% CI: 13.3, 52.8), respectively. Among 39 hyperenhancing nodules (hemangiomas, focal nodular hyperplasias, and adenomas: mean diameter, 14.5 mm ± 13.7), the DLA detected 22 of 39 (56%; 95% CI: 40.7, 70.1), with no evidence of a difference between the DLA and R1 or R2 (both 21 of 39; 54%; 95% CI: 39.0, 68.1; DLA vs R1 and DLA vs R2, both P = .82). Additional details are provided in Table S7.

The DLA, R1, and R2 detected 39, 33, and 47 false-positive nodules, respectively. Specifically, the DLA, R1, and R2 identified segmental biliary dilatations 14, four, and six times; diaphragmatic indentations 12, six, and eight times; focal fatty pseudolesions nine, eight, and 19 times; and focal nonfatty pseudolesions four, 15, and 11 times, respectively (DLA vs R1, P = .004; DLA vs R2, P = .01). Detailed results are presented in Table 4, and examples of LM detection and annotation by the DLA as well as false-positive detection are presented in Figure 3.

Table 4:

Liver Metastasis False Detection Rates for the Deep Learning Algorithm, Senior Radiologist, and In-Training Radiologist

Characteristic	DLA FP per Case	R1 FP per Case	R2 FP per Case	DLA vs R1	DLA vs R2
All patients	0.22 (39/181) [0.16, 0.28]	0.20 (37/181) [0.15, 0.27]	0.26 (47/181) [0.20, 0.33]	+0.08 (P0.1 = .08)	−0.04 (P0.1 = .045)
Patients with LM	0.33 (31/95) [0.24, 0.43]	0.22 (21/95) [0.15, 0.31]	0.33 (31/95) [0.24, 0.43]	+0.1 (P0.1 = .008)	+0.0 (P0.1 = .23)
Patients with synchronous LM	0.21 (12/58) [0.12, 0.33]	0.26 (15/58) [0.16, 0.38]	0.40 (23/58) [0.28, 0.53]	−0.05 (P0.1 = .051)	−0.19 (P0.1 = .95)
Patients with metachronous LM	0.51 (19/37) [0.36, 0.67]	0.16 (6/37) [0.08, 0.31]	0.22 (8/37) [0.11, 0.37]	+0.35 (P0.1 < .001)	+0.30 (P0.1 < .001)
Patients without LM	0.09 (8/86) [0.05, 0.17]	0.19 (16/86) [0.12, 0.28]	0.19 (16/86) [0.12, 0.28]	+0.02 (P0.1 = .17)	+0.28 (P0.1 < .001)
Patients with a history of chemotherapy	0.24 (27/113) [0.17, 0.33]	0.17 (19/113) [0.11, 0.25]	0.23 (26/113) [0.16, 0.32]	+0.07 (P0.1 = .01)	+0.01 (P0.1 = .15)

Note.—Data are rates with numerators/denominators in parentheses and 95% CIs in brackets and ± numbers that correspond to differences in detection rates with P_0.1 in parentheses. P values correspond to noninferiority tests of false detection rates between the deep learning algorithm (DLA) and radiologists, with a 0.1 false-positive (FP) result per case margin. LM = liver metastasis, P_0.1 = noninferiority P value at the 0.1 false-positive result per case margin, R1 = senior radiologist, R2 = in-training radiologist.

Figure 3:

Example of liver metastases (LM) detection by the deep learning algorithm (DLA) on abdominal, axial contrast-enhanced CT images at portal venous phase in different patients. (A) Two LM (arrows) detected in a 58-year-old male patient with synchronous LM. (B) False-positive detection and annotation in a 47-year-old male patient with synchronous LM and segmental biliary dilatation. The DLA correctly detected the LM (arrow) but identified the dilated biliary segment as an additional liver nodule (arrowhead). (C) Two LM (white arrows) detected by DLA in a 49-year-old male patient with synchronous LM after neoadjuvant chemotherapy, with marked hepatic sinusoidal obstruction syndrome (black arrows). (D) LM (arrow) in fatty liver detected despite low nodule and parenchyma contrast ratio in a 56-year-old female patient with metachronous LM.

Pathology diagnosis was available for 120 LM, of which the DLA, R1, and R2 detected 106 (88%; 95% CI: 81.4, 92.9), 98 (82%; 95% CI: 73.8, 87.6), and 97 (81%; 95% CI: 72.9, 86.9) LM, respectively (DLA vs R1, P = .11; DLA vs R2, P = .03). Among the 10 mucinous LM, the DLA, R1, and R2 detected 90% (nine of 10; 95% CI: 59.6, 98.2), 80% (eight of 10; 95% CI: 49.0, 94.3), and 70% (seven of 10; 95% CI: 39.7, 89.2) of lesions, respectively, with no evidence of a difference in mucinous LM detection rates between the DLA and R1 or R2 (DLA vs R1, P = .53; DLA vs R2, P = .26). For the 57 LM in patients with fatty liver, the DLA detected 88% (50 of 57; 95% CI: 76.8, 93.9), while both R1 and R2 detected 86% (49 of 57; 95% CI: 74.7, 92.7) of lesions, with no evidence of a difference in detection rates between the DLA and R1 or R2 (DLA vs R1 and DLA vs R2, both P = .78). Additional details are provided in Tables S8 and S9.

Per-Patient Analysis

Among the 95 patients with LM, the DLA, R1, and R2 detected at least one LM in 91 (96%; 95% CI: 89.5, 98.4), 91 (96%; 95% CI: 89.5, 98.4), and 90 (95%; 95% CI: 88.2, 97.9) patients, respectively, with no evidence of differences in detection rates between the DLA and R1 or R2 (DLA vs R1, P > .99; DLA vs R2, P = .74). The DLA and R1 successfully identified all LM in 64 of 95 (67%; 95% CI: 57.0, 75.7) patients, while R2 identified all LM in 57 of 95 (60%; 95% CI: 50.0, 69.5) patients (DLA vs R1, P > .99; DLA vs R2, P = .16).

In patients with synchronous LM, the DLA detected at least one LM in 56 of 58 (97%; 95% CI: 88.9, 99.4) patients, compared with 55 of 58 (95%; 95% CI: 86.0, 98.4) patients for R1 and 54 of 58 (93%; 95% CI: 83.4, 97.3) patients for R2, with no evidence of differences between the DLA and R1 or R2 (DLA vs R1, P = .58; DLA vs R2, P = .33). In the metachronous LM group, the DLA detected at least one LM in 35 of 37 (95%; 95% CI: 83.5, 98.6) patients, while both R1 and R2 detected at least one LM in 36 of 37 (97%; 95% CI: 86.7, 99.4) patients, with no evidence of a difference between the DLA and R1 or R2 (both P = .66).

Detection of all LM in patients with synchronous LM was achieved in 39 of 58 (67%; 95% CI: 53.9, 78.1) patients by the DLA, 37 of 58 (64%; 95% CI: 50.8, 75.5) patients by R1, and 32 of 58 (55%; 95% CI: 42.1, 67.2) patients by R2, with no evidence of differences between the DLA and R1 or R2 (DLA vs R1, P = .76; DLA vs R2, P = .20). In the metachronous LM group, the detection of all LM was achieved in 25 of 37 (68%; 95% CI: 51.4, 81.4) patients by the DLA, 27 of 37 (73%; 95% CI: 56.6, 85.3) patients by R1, and 25 of 37 (68%; 95% CI: 51.4, 81.4) patients by R2, with no evidence of differences between the DLA and R1 or R2 (DLA vs R1, P = .79; DLA vs R2, P > .99).

In the entire study population, at least one false-positive nodule was identified by the DLA in 23 of 181 (13%; 95% CI: 8.9, 19.0) patients, compared with 28 of 181 (15%; 95% CI: 11.2, 22.6) patients for R1 and 33 of 181 (18%; 95% CI: 13.1, 25.0) patients for R2 (DLA vs R1, P = .58; DLA vs R2, P = .18). We observed overlap in false positives between the DLA and R1 in five patients and between the DLA and R2 in four patients. Among the 35 patients with only one LM, the detection rates were 97% (95% CI: 85.5, 99.5) for the DLA and 91% (95% CI: 77.6, 97.0) for both R1 and R2, with no evidence of differences between the DLA and R1 or R2 (both P = .62). We found no evidence of differences in detecting all LM in patients with unilobar versus bilobar metastases (unilobar LM: P = .77 for DLA vs R1 and P = .15 for DLA vs R2; bilobar LM: P = .73 for DLA vs R1 and P = .69 for DLA vs R2). A summary of the per-patient detection rates for all LM is presented in Figure 4; details are provided in Table S10.

Figure 4:

Graph shows per-patient detection rate for the totality of liver metastasis (LM) for the deep learning algorithm (DLA), senior radiologist (R1), and in-training radiologist (R2), according to synchronous or metachronous status, number of LM, and unilobar or bilobar localization.

Because of the limited clinical relevance, we do not report the per-patient performance of the DLA, R1, and R2 in detecting BNs.

Interreader Agreement

A summary of interreader agreement for LM detection is presented in Table S11. The interreader agreement was fair to substantial, with κ values ranging from 0.63 ± 0.06 to 0.75 ± 0.05, increasing with lesion size. Among the four patients in whom R1 did not detect any LM and the five patients in whom R2 failed to detect any LM, the DLA successfully identified at least one LM in three patients.

We observed similar fair to substantial interreader agreement for BN detection, with mean κ values ranging between 0.60 ± 0.10 and 0.77 ± 0.08. A Venn diagram summary of LM, BN, and liver nodule detection overlaps among the DLA, R1, and R2 is presented in Figure 5.

Figure 5:

Venn diagram of the numbers of detected liver metastases (A), benign nodules (B), and total liver nodules (C) by the deep learning algorithm (DLA), senior radiologist (R1), and in-training radiologist (R2).

Discussion

In our study, we aimed to evaluate the accuracy of a DLA for detecting LM in various clinical scenarios in patients with CRC and to compare this accuracy with that of radiologists. We found that the accuracy of this DLA in detecting LM and BNs was similar to that of radiologists at the per-patient and per-nodule levels.

The overall per-patient detection rate for liver nodules was 96% with the DLA, similar to that of the senior and in-training radiologists (R1, 96%; R2, 95%). These results are in line with those reported by Nakai et al (14). We found that the performance of the DLA was superior to that reported in previous studies that included a broader range of focal liver lesions (10). The effectiveness of the DLA may be attributable to the hypoenhancing pattern of CRC LM (7) and suggests that the algorithm is particularly well suited for detecting hypoenhancing lesions. This assumption is further supported by the limited ability of the DLA to identify hyperattenuating or hyperenhancing lesions, such as calcifications and benign hepatocellular or vascular lesions. Importantly, the performance of the DLA remained consistent when compared with radiologists across different patient groups, including those with synchronous or metachronous metastases, and patients with and without prior chemotherapy.

On the per-nodule level, smaller lesions (<10 mm) were more challenging to detect than larger lesions, with detection rates of 55% (51 of 93) for the DLA, 51% (47 of 93) for R1, and 44% (41 of 93) for R2, compared with larger lesions (ie, 10–19 mm, 91% [96 of 106] for DLA, 90% [95 of 106] for R1, 89% [94 of 106] for R2). In the same way, detection rates were similar in fatty livers: DLA detected 88% (50 of 57), and both R1 and R2 detected 86% (49 of 57) of LM. These results confirm the well-known technical limitations of CT examinations. Notably, our results were similar to those of Jhaveri et al (20), who reported a sensitivity for the detection of small LM (≤10 mm) at CT of 45% and an interobserver concordance rate of 51%.

In the study of Nakai et al (14), 55% of LM overlooked by radiologists were correctly detected by a DLA. Our findings did not confirm this observation. This discrepancy can be attributed to differences in study design. In our study, the radiologists were aware of the diagnosis of CRC, and they reported findings on a standardized diagram. This design allowed readers to focus solely on one task: detecting liver nodules. In contrast, the study by Nakai et al (14) was based on retrospective reading of CT scan reports, where distractions and incomplete documentation of lesions may have been present, as acknowledged by the authors. The radiologists’ knowledge of the CRC diagnosis in our study may have influenced their ability to detect LM, as similarly observed by Nakai et al (14).

This DLA has the potential to reduce radiologists’ errors in practice. The DLA is unaffected by fatigue, a key advantage over humans in maintaining consistent diagnostic performance. Additionally, the false-positive rate was low, and the nature of the false-positive nodules detected differed between the DLA and radiologists, suggesting that use of the DLA could complement radiologists’ work. The DLA often misidentified dilated biliary structures and diaphragmatic indentations, while pseudolesional appearances in slightly heterogeneous parenchyma misled the radiologists. This discrepancy highlights the differing interpretative mechanisms between artificial intelligence and human intelligence. Human readers may experience pareidolia, perceiving patterns or structures where none exist, while the software lacks anatomic context to assign realistic shapes to structures. Moreover, our study design may have contributed to radiologist overdiagnosis, as they were aware of the CRC diagnosis and were explicitly tasked with identifying secondary liver lesions. The recent Colorectal Cancer Liver Metastases Assessment resection study by Zeeuw et al (21) revealed that the integration of artificial intelligence–based segmentation models directly into the radiologic workflow is feasible and user friendly. We can speculate that a dramatic increase in usable DLA software is associated with the possibility of real-time interaction, allowing radiologists to accept or reject artificial intelligence–generated segmentations on the fly. Such functionality could increase confidence and prevent overdiagnosis, especially when segmentation errors are obvious and easily dismissed by the human reader (21). Future refinements of the algorithm could include incorporating anatomic landmarks and vessel relationships, which may help reduce the false-positive rate related to biliary dilatations and other vascular structures, thereby improving overall specificity.

The use of an integrated DLA with interactive radiologist oversight may reduce the rates of missed or misinterpreted metastases while maintaining control over clinical decision-making. Our study results will serve as a basis for further studies exploring a combined DL-based algorithm with radiologist interpretation to assess whether such an integrated approach can improve overall diagnostic accuracy and reduce errors, as suggested by the recent European Organization for Research and Treatment of Cancer, European Society of Oncologic Imaging, and European Society of Gastrointestinal and Abdominal Radiology joint recommendations (16).

Our demonstration that the performance of this DLA was comparable to that of radiologists in detecting LM opens new possibilities for clinical applications. Although we focused on assessing diagnostic performance in our study, several clinical scenarios may benefit from the integration of this DLA into routine practice. In the preoperative setting, accurate identification of all LM—including small or subcapsular lesions—may impact surgical planning. During follow-up, reproducible detection may improve early recurrence identification, especially in patients previously treated with surgery or systemic therapy. Additionally, integration into multidisciplinary tumor boards could allow the DLA to serve as an automated second reader, reducing oversight risk and standardizing lesion counting in alignment with Response Evaluation Criteria in Solid Tumors 1.1. These potential applications merit prospective investigation to assess their impact on clinical decision-making and patient outcomes (22–24).

The inability of the DLA to accurately identify hyperattenuating lesions relative to the hepatic parenchyma during the portal venous phase may indicate suboptimal performance in detecting certain types of CRC LM, such as calcification of metastases in response to antiepidermal growth factor receptor chemotherapy (25).

Our study had limitations. First, our study design may have introduced selection bias, as we included only patients with sufficient follow-up, which may not reflect the full CRC population. Second, we excluded patients with prior liver surgery and local-regional treatment, such as high-intensity focused ultrasound ablation, irreversible electroporation, or thermal ablation, potentially limiting generalizability and enriching the cohort with more favorable imaging conditions (26). These exclusions were necessary because the software was not trained and tested on the liver with a postsurgical modification. However, these patients often require close imaging surveillance because of high recurrence risks. Third, most examinations came from tertiary centers with experienced radiologists and standardized imaging protocols, which may not represent community practice. Fourth, we did not perform a sample size calculation a priori, as the size was based on a retrospective cohort defined by strict eligibility criteria; therefore, the statistical comparisons presented should be interpreted as hypothesis-generating, providing a foundation for future prospective and adequately powered validation studies. Fifth, the distinction between BNs and LM was made a priori based on the reference standard evaluation; differentiating BNs from LM was beyond the scope of the study and the current capabilities of the algorithm. Metastases located outside the liver, such as peritoneal nodules in contact with the liver capsule, were also absent in our patients (27). Finally, our study was based on a timely assessment with respect to the clinical course of disease and surveillance of these patients, as detailed in the Materials and Methods section. In clinical practice, comparison with the previous examination, if available, is constantly made, which can influence human reader performance. A prospective comparison of clinical performance between the DLA and radiologists is needed but was beyond the scope of our study. In addition, the impact of chemotherapy regimens on liver imaging was only superficially assessed, and a dedicated study addressing treatment-related imaging changes and their relationship with detection performance appears necessary.

Further work should include larger prospective multicenter studies, assessment of longitudinal performance in clinical workflows, and evaluation of the DLA’s impact on patient management and outcomes.

In conclusion, the DLA we evaluated for automatic liver nodule detection demonstrated diagnostic accuracy in identifying LM in patients with CRC that was comparable to that of radiologists. This finding highlights the potential clinical benefits of this DLA and underscores its promise for integration into routine clinical practice.

Disclosures of conflicts of interest

Please see ICMJE form(s) for author conflicts of interest. These have been provided as supplemental materials