Multicenter AI- versus expert-assisted RECIST target lesion measurements in follow-up body CT of patients with cancer

Abstract

Background

Manual lesion measurements remain the standard for assessing oncologic treatment response, despite being time-consuming and prone to substantial interreader variability, which may lead to inconsistent response classification and subsequent variability in treatment decisions.

Purpose

To evaluate the impact of an artificial intelligence (AI) system for assisted lesion measurement on reading time and measurement consistency in follow-up CT examinations using the Response Evaluation Criteria in Solid Tumors (RECIST 1.1).

Methods and Materials

In this retrospective reader study, follow-up chest-abdomen-pelvis CT examinations from 212 oncology patients collected at 2 Dutch hospitals were assessed by 23 readers (15 radiologists, 8 residents) recruited from 11 international institutions under 3 conditions: unassisted, AI-assisted, and expert-assisted (using a prior radiologist’s unassisted measurement). To prevent bias related to the source of the measurements, readers were informed that all support was AI-derived. Primary outcomes were reading time to completion and interobserver measurement variability. For each outcome, a Bayesian generalized linear mixed model was used to analyze the results.

Results

AI assistance significantly reduced per-patient reading time versus unassisted reading (-36.0 s; 95% CI, -53.0 to -22.1). At the lesion level, it was associated with a small increase in variability relative to the expert-derived reference standard (1.32 mm; 95% CI, 0.83-1.91). At the patient level, AI assistance did not meaningfully affect change in the sum of longest diameters (-0.38 mm; 95% CI, -1.57 to 0.72), but increased RECIST outcome agreement by 7.7% (95% CI, 2.8-12.7) compared with unassisted reading. Expert-assisted reading yielded even higher interreader agreement (13.3%; 95% CI, 8.6-18.1).

Conclusion

AI assistance reduced reading time and improved patient-level RECIST agreement, despite a small increase in lesion-level measurement variability. These findings suggest that AI-assisted RECIST assessment may improve workflow and response classification consistency, while also providing a benchmark from expert-assisted reading for future AI development.

Summary 

AI-generated automatic RECIST 1.1 measurement proposals for target lesions on follow-up CT shorten reading time and improve interreader agreement but require more adjustments than expert-generated proposals.

Key results 

• AI measurement proposals reduced reading time by 33.6% (95% CI, 22.5-43.0) compared with unassisted reading, a mean absolute reduction of 36 seconds per patient (95% CI, 22-53 seconds).

• AI assistance was associated with reduced RECIST 1.1 outcome variability, increasing interreader agreement by 7.7% (95% CI, 2.8-12.7) compared with unassisted reading.

• Expert-assisted reading yielded even higher interreader agreement than AI-assisted reading, improving agreement by 13.3% (95% CI, 8.6-18.1) versus unassisted reading.

Introduction

Quantifying oncologic disease progression and evaluating treatment response in longitudinal CT scans currently depends largely on manual lesion measurements, a process that is both time-consuming and labor-intensive. Given the growing demands on radiology services,¹ it is essential to develop and validate methods that can help reduce radiologists’ workloads. However, a review of 2024 medical imaging literature suggests that relatively few artificial intelligence (AI) studies are expected to reduce workload,² highlighting the need for systematic evaluations of AI’s potential to reduce reading time.

Longitudinal lesion measurements are frequently evaluated using standardized criteria such as the Response Evaluation Criteria in Solid Tumors (RECIST 1.1)^3,4 and its immunotherapy-specific extension, iRECIST.⁵ Response evaluation and classification are crucial for clinical decision-making and clinical trial assessment.⁶ However, a meta-analysis has shown substantial interobserver variability in manual RECIST measurements, with relative differences exceeding the 20% threshold used to classify progressive disease.⁷ Selecting appropriate target lesions in baseline examinations is challenging and prone to significant variability.⁸ In contrast, reidentification and measurement of these lesions on follow-up scans is a clearly delineated task, making it particularly suitable for automation.

Automating aspects of RECIST evaluation has been shown to reduce both reading times and inter-observer variability compared with manual evaluation.⁹ Ristow et al.¹⁰ demonstrated that software-assisted iRECIST evaluations significantly decreased reading time and reduced response misclassifications. Their method calculated the sum of longest diameters (SLD) and determined the response classification but did not include measurement automation. In contrast, Hering et al.¹¹ used a registration algorithm to propagate a lesion measured on a baseline examination and automatically resegmented it on follow-up examinations. Hering et al. subsequently evaluated this model in a reader study,¹² showing significant reductions in reading times and improved interobserver agreement. However, that study was limited to whole-body assessment of lymph nodes and soft-tissue lesions and relied on a limited pool of 3 readers.

In the current study, we evaluated AI assistance for follow-up RECIST 1.1 assessment in a large patient population with diverse, RECIST-compliant lesions, and directly compared it with unassisted assessment and assistance from a second expert reader. To our knowledge, this is the first multicenter, multireader comparison of AI-assisted, unassisted, and expert-assisted RECIST 1.1 target lesion measurements. This design allowed us to assess whether AI assistance improves efficiency and reproducibility relative to these reference conditions, while providing an expert-assisted benchmark to guide future AI development. The primary aim was to compare reading time and interobserver measurement variability among the 3 reading conditions. Secondary outcomes included proposal acceptance and modification rates, patient-level change in the sum of longest diameters, and RECIST 1.1 response agreement.

Materials and methods

Data collection

This retrospective study used data collected at the Radboud University Medical Center (Radboudumc) and Jeroen Bosch Hospital (JBZ) in the Netherlands. There is partial subject overlap (209 of 212) with de Grauw et al.,¹³ in which baseline 3-dimensional lesion segmentations of subjects in the current study were used for evaluation of automatic segmentation models. In contrast, the current study uses follow-up lesion measurements for RECIST evaluation, a distinct and nonoverlapping analytic endpoint. None of the 212 study patients were used for AI model training. Given the retrospective design and the anonymization of image metadata and reports, the need for institutional review board approval was waived by the CMO Radboudumc, and the requirement for written informed consent was also waived.

Our clinical dataset consisted of chest-abdomen-pelvis CT examinations performed for oncologic treatment assessment. Dataset details are provided in Table 1. The data selection process is illustrated in Figure 1. To be eligible for inclusion, the original study reports had to describe measurable target lesions with series identifiers and axial slice locations. Only series including axial images with a slice thickness ≤ 5 mm were included.

Table 1.

Clinical dataset patients, lesions, and scanners.

Parameter	Radboudumc	JBZ	Total
No. of patients	107	105	212
Male (%)	56 (52.3%)	66 (62.9%)	122 (57.5%)
Female (%)	51 (47.7%)	39 (37.1%)	90 (42.5%)
Mean age at T1 (years)	62.23 ± 12.34	63.82 ± 10.23	63.02 ± 11.34
Male (years)	59.76 ± 13.46	63.59 ± 9.74	61.42 ± 12.08
Female (years)	64.48 ± 10.85	64.95 ± 10.57	64.20 ± 10.66
No. series pairs	139	138	277
No. target lesions	262	277	539
Mean target lesions per patient	2.49 ± 1.08	2.64 ± 1.03	2.56 ± 1.06
Mean lesion size at T1 (mm)	32.33 ± 24.43	21.88 ± 17.71	27.12 ± 21.94
Time between follow-up (days)	36	24	24
Min
Max	1187	653	1187
Median (IQR)	180 days (IQR 234)	84 days (IQR 53)	99 days (IQR 141)
Voxel spacing (XYZ, mm)	0.30, 0.30, 0.45	0.60, 0.60, 3.00	0.30, 0.30, 0.45
Min
Max	1.07, 1.07, 5.0	0.98, 0.98, 3.00	1.07, 1.07, 5.0
Mean	0.73, 0.73, 2.27	0.77, 0.77, 3.00	0.75, 0.75, 2.63
Acquisition period	04/2016 - 11/2020	04/2013 - 01/2020	04/2013 - 11/2020
Lesion types
Lung	96	85	181
Lymph node	90	78	168
Liver	27	73	100
Soft tissue	16	27	43
Other	33	14	47
Scanner manufacturers (studies):
Siemens (%)	38 (17.8%)	210 (100%)	248 (58.4%)
Toshiba (%)	131 (61.2%)	0 (0%)	131 (30.8%)
Philips (%)	27 (12.6%)	0 (0%)	27 (6.3%)
Canon(%)	17 (7.9%)	0 (0%)	17 (4.0%)
GE (%)	1 (0.5%)	0 (0%)	1 (0.2%)
Contrast (studies):
Yes (%)	165 (77.1%)	185 (88.1%)	350 (82.5%)
No (%)	49 (22.9%)	25 (11.9%)	74 (17.5%)

The lesion type classification scheme and details of the lesions in the 'other’ category are provided in Appendix A. Percentages are in parentheses, means are given with SD (±).

JBZ = Jeroen Bosch Hospital; Radboudumc = Radboud University Medical Center; T1 = first timepoint study; T2 = follow-up study.

Figure 1.

Flowchart of the patient selection and data collection process for the clinical dataset. Examinations were randomly selected for initial annotation. For reports with more than 2 lesions per organ identified, the excess lesions were randomly excluded. JBZ, Jeroen Bosch Hospital; Radboudumc, Radboud University Medical Center.

After randomly selecting 120 qualifying patients per institution, 2 radiologists from the Radboudumc with more than 10 years of experience reading body oncologic CT scans (E.Th.S. and E.J.S.) independently remeasured their target lesions as documented in the clinical reports; discrepancies were resolved by consensus review. Patients were excluded if the annotators could not locate all lesions referenced in the radiology report. To ensure compliance with RECIST 1.1 guidelines, modifications were made where necessary, including the removal of noncompliant excess lesions. The final dataset consisted of 212 patients and 539 lesions. Sample size was determined pragmatically by the number of eligible cases and annotation workload rather than by formal power calculation, which may have limited the ability to detect smaller effects.

AI system

The evaluated system was an automated segmentation workflow consisting of a patch extractor and a three-dimensional segmentation model based on the U-Net architecture.^14,15 The model, developed at Fraunhofer MEVIS, was trained on a heterogeneous dataset of 2408 manually annotated lesions of various types.¹¹ Training, validation, and internal test splits of that dataset are described in the original publication¹¹; none of the CT examinations used here overlapped with the training data. In follow-up examinations, lesions were segmented after propagating the coordinates of the lesion center point from the baseline scan using image registration. Lesion measurements were extracted by first determining the longest axial diameter and then the longest perpendicular short-axis measurement. If the model was unable to generate a measurement, it indicated the expected location of the lesion using the propagated center point.

Participants

Twenty-three readers (8 radiology residents and 15 radiologists) participated in the study. Readers were recruited from Fondazione Policlinico Universitario Agostino Gemelli IRCCS (S.B., A.D., V.L., and L.R.), Radboud University Medical Center (J.R., M.d.R., S.S., and M.S.), University of Naples “Federico II” (A.L., L.M., A.P., and A.S.), Rijnstate Ziekenhuis (H.C.v.H., C.I.V., and B.V.), University Medical Center of the Johannes Gutenberg University Mainz (A.E.O. and S.S.), Sungroup International Hospital (N.T.N.M.), SCB Medical College and Hospital (A.B.G.), University of Salerno (R.C.), Charité - Universitätsmedizin Berlin (L.X.), Jeroen Bosch Hospital (M.J.C.M.R.), and University Medical Center Groningen (D.Y.). Among the 15 radiologists, years in practice after completion of residency ranged from 1 to 28 years (median, 6 years; IQR, 5.5-9.0 years). Among the 8 residents, training years ranged from 1 to 4 years (median, 2.5 years; IQR, 2.0-3.3 years).

Reader study

The reader study was conducted from October to December 2024. It used a custom-built CT analysis software called CIRRUS¹⁶ for the reading environment. Readers were presented with scans from 2 time points (T1 and T2), along with a list of premeasured target lesions identified at T1. Their task was to remeasure the corresponding lesions at T2 in accordance with RECIST 1.1 guidelines. They were not required to calculate the SLD or assess nontarget or new lesions. The reading environment is shown in Appendix B, Figure S4.

For the assisted setting, the software provided measurement proposals for T2. Readers could choose to accept, modify, or replace them with new measurements. Proposals were drawn in equal proportion from 2 sources: (1) our AI system and (2) expert proposals, defined as prior unassisted measurements made by radiologist readers from preceding reading groups. All radiologists (n = 15) from the unassisted groups contributed approximately equally to the expert proposal pool, with 1 radiologist sampled per patient. Readers were informed that all measurements originated from AI, blinding them to the human origin of half of the proposals.

For both assisted and unassisted settings, readers could navigate to measurement locations using the target lesion list. Scans were aligned along the z-axis, allowing linked navigation to measurements. Reading times were automatically recorded but could be paused for interruptions. Total reading time was calculated as the sum of unpaused intervals. Sessions with interruptions were not excluded, as interruptions are part of regular clinical practice and were expected to occur across both assisted and unassisted sessions.

Figure 2 provides a schematic overview of the study design. Study data were split at the patient level and divided into 8 near-equal subsets using stratified random sampling to preserve proportional representation by hospital, lesion counts, and patient sex. Readers were assigned to four groups, based on availability and experience level, and completed the study in a crossover design with randomized case order and a minimum 2-week washout period between reading sessions to mitigate potential learning and carryover effects. Readers received instructions (see Appendix B) and completed 6 training cases to familiarize themselves with the software. Participants were advised to use a high-contrast display, preferably a diagnostic monitor, in a dimly lit room.

Figure 2.

Schematic overview of the reading strategy. Unassisted reading, AI-assisted, Expert-assisted. Readers were assigned to 4 groups (A-D) of unequal size (A = 5, B = 5, C = 6, D = 7). The dataset comprised 212 patients and was partitioned into 8 near-equal subsets. Accordingly, total patient-level readings differed slightly across conditions: Unassisted = 1209, AI-assisted = 1214, Expert-assisted = 1216. Each reader reviewed approximately 75% of the dataset, per-reader workload ranged from 156 to 160 readings overall. Expert proposals were randomly sampled from a radiologist of the preceding reading group, ensuring each radiologist within a group contributed equally to the proposal pool. Numbered boxes represent data subdivisions assigned to the reader groups.

Statistical analysis

For this multireader, multicase crossover study, primary outcomes were analyzed using Bayesian generalized linear mixed models¹⁷ with the brms package (version 2.21.0) in R (version 4.4.2). A shifted lognormal model was applied for the reading time outcome, whereas measurement variability, assessed via the absolute error relative to the expert-assisted mean, was analyzed with a hurdle lognormal model. Patient-level SLD change was analyzed with a Gaussian model and RECIST outcome agreement with a binomial model. Weakly informative priors were used for all model parameters. Analyses were performed at the patient level for reading time (computed by summing series-level times), SLD change and RECIST outcome agreement, whereas measurement variability was analyzed at the lesion level. Additional modeling details, including prior specifications and convergence diagnostics, are provided in Appendix C.

Defining a reference standard for lesion size is inherently challenging because of the natural interobserver variability among experts. As a reference standard, we used the mean measurement of the 2 radiologists in the expert-assisted group, ensuring each measurement was seen by 2 radiologists. This approach introduces circularity, biasing the measurement variability analysis in favor of the expert-assisted condition, which should be taken into account when interpreting results.

Code availability

The AI model used in this study was based on the open-source nnU-Net framework,¹⁵ with implementation details described previously.¹¹ The training dataset and model configuration are not publicly available because of data use restrictions. However, several related open-source lesion segmentation models are available^13,18 and may serve as reference implementations. The statistical analysis code is available from the corresponding author upon reasonable request.

Results

The clinical dataset used for this study included 212 patients (mean age, 63.02 years ± 11.34; 42.5% female) with 539 RECIST target lesions in 277 pairs of CT series. We evaluated the effects of assistance type on reading time, interobserver measurement variability, proposal acceptance and modification rates, and SLD variability. Unless otherwise stated, model-based values are reported as posterior expected means with 95% credible intervals.

Effect of assistance on reading time

We queried our fitted model to assess the overall effects of assistance, as well as the conditional effect of reader experience, while accounting for the number of lesions per patient (the latter is in Appendix A). These results are summarized in Tables 2 and 3. Overall, mean expected reading times per patient were highest for the unassisted condition (105 s; 95% CI, 88-128). Both AI-assisted reading (71 s; 95% CI, 60-86) and expert-assisted reading (56 s; 95% CI, 48-68) showed clinically meaningful time savings. However, because expert-assisted reading requires an initial unassisted read, the mean expected cumulative reading time for the double-reading workflow was 151 s (95% CI, 145-157). Figure 3 shows individual and group-level mean reading times per lesion across the three conditions, split by experience level.

Table 2.

Expected values for the assistance types with the conditional effect of experience level on: reading time (in seconds), absolute measurement deviation compared with the mean of the expert-assisted group (in millimeters), patient-level change in SLD (in millimeters), RECIST outcome agreement based on SLD (in percentage points). 95% credible intervals in brackets.

Assistance	Experience	Mean RT (s)	Mean AE (mm)	Mean SLD Change (mm)	Mean Outcome Agreement (%)
Unassisted	All	105 [88, 128]	3.08 [2.70, 3.53]	−6.30 [-9.54, -2.98]	72.5 [68.3, 76.3]
Unassisted	Residents	107 [80, 146]	3.74 [3.09, 4.50]	−6.62 [-9.97, -3.05]	81.9 [76.8, 86.2]
Unassisted	Radiologists	104 [84, 130]	2.73 [2.33, 3.17]	−6.13 [-9.64, -2.68]	63.4 [56.8, 69.4]
AI	All	71 [60, 86]	4.35 [3.75, 5.07]	−6.73 [-10.18, -3.16]	80.4 [76.3, 84.1]
AI	Residents	67 [51, 88]	4.70 [3.90, 5.77]	−7.23 [-10.94, -3.49]	86.0 [81.6, 89.8]
AI	Radiologists	74 [60, 92]	4.16 [3.55, 4.89]	−6.47 [-10.24, -2.77]	74.7 [68.9, 79.9]
Expert	All	56 [48, 68]	1.51 [1.28, 1.82]	−5.36 [-8.74, -1.90]	85.9 [82.5, 89.1]
Expert	Residents	51 [40, 67]	1.79 [1.43, 2.29]	−5.70 [-9.28, -2.08]	91.2 [87.6, 94.1]
Expert	Radiologists	59 [48, 73]	1.37 [1.12, 1.68]	−5.18 [-8.69, -1.73]	80.8 [75.1, 85.5]

Abbreviations: AE = absolute error per lesion; RT = reading time per patient; SLD = sum of longest diameters.

Table 3.

Effect of the assistance types with the conditional effect of experience level on: reading time (in seconds), absolute measurement deviation compared with the mean of the expert-assisted group (in millimeters), patient-level change in SLD (in millimeters), RECIST outcome agreement based on SLD (in percentage points); positive differences indicate higher values for the first condition; 95% credible intervals in brackets.

Assistance Comparison	Experience	RT Difference (s)	AE Difference (mm)	SLD Difference (mm)	Outcome Agreement Difference (%)
AI—Expert	All	15 [8, 24]	2.84 [2.26, 3.50]	−1.38 [-2.96, 0.24]	−5.60 [-10.16, -1.10]
AI—Expert	Residents	16 [5, 29]	2.90 [2.15, 3.84]	−1.53 [-3.20, 0.18]	−5.17 [-10.08, -0.12]
AI—Expert	Radiologists	15 [7, 26]	2.81 [2.20, 3.53]	−1.30 [-3.26, 0.65]	−6.02 [-13.10, 0.83]
Unassisted—AI	All	36 [22, 53]	−1.32 [-1.91, -0.83]	−0.38 [-1.57, 0.72]	7.73 [2.79, 12.67]
Unassisted—AI	Residents	39 [17, 68]	−0.96 [-1.83, -0.24]	−0.61 [-1.84, 0.57]	4.10 [-0.99, 9.29]
Unassisted—AI	Radiologists	34 [18, 54]	−1.52 [-2.10, -1.00]	−0.26 [-1.58, 1.08]	11.36 [3.95, 18.52]
Unassisted—Expert	All	51 [35, 71]	1.52 [1.21, 1.87]	1.00 [-0.09, 2.14]	13.33 [8.59, 18.07]
Unassisted—Expert	Residents	55 [30, 90]	1.94 [1.42, 2.57]	0.92 [-0.45, 2.11]	9.27 [4.83, 13.99]
Unassisted—Expert	Radiologists	49 [31, 73]	1.30 [0.94, 1.67]	1.04 [-0.31, 2.45]	17.38 [10.06, 24.70]

Abbreviations: AE = absolute error per lesion; RT = reading time per patient; SLD = sum of longest diameters.

Figure 3.

Mean reading time per lesion under each condition for individual readers, with boxplots of reading-time distributions stratified by reader experience level. Unassisted, AI-assisted, Expert-assisted. In the left panel, each point represents the mean time per lesion for a single reader, and vertical dashed lines indicate the overall mean time per assistance condition. In the right panel, the center line indicates the median, the box the interquartile range, and the whiskers extend to 1.5 × IQR. Numeric annotations show the overall means (left) and medians (right).

Effect of assistance on measurement variability

Tables 2 and 3 summarize the impact of assistance type and reader experience on measurement variability, quantified as the absolute error with respect to the expert-assisted radiologists’ mean measurement. Because deviations were calculated relative to the expert-assisted group, this condition is expected to show the lowest error. Overall, the expected mean absolute error was highest in the AI-assisted condition (4.35 mm; 95% CI, 3.75-5.07), followed by unassisted reading (3.08 mm; 95% CI, 2.70-3.53), with expert-assisted reading having the lowest expected error (1.51 mm; 95% CI, 1.28-1.82). Lesion-type-stratified analyses, reported in Appendix A, showed the same overall pattern across categories, with the lowest absolute error in the expert-assisted condition and higher absolute error in the AI-assisted condition than in the unassisted condition. The distribution of measurement differences per reader under unassisted and AI-assisted conditions is shown in Figure S1.

Effect of assistance on SLD and RECIST outcome variability

To assess variability in RECIST outcomes, we computed the SLD post hoc from the individual measurements and identified patients with at least 1 discordant outcome prediction between readers. Progression from nontarget or new lesions was excluded because these were not evaluated. As shown in Table 3, differences in patient-level SLD change were small and uncertain across assistance conditions, indicating no meaningful effect of assistance on estimated tumor burden change. However, assistance improved consistency of RECIST outcome classification: compared with unassisted reading, AI assistance increased interreader agreement by 7.7% (95% CI, 2.8-12.7), and expert assistance increased agreement by 13.3% (95% CI, 8.6-18.1). Figure S3 illustrates the maximum differences in relative SLD change from T1 to T2 across assistance conditions and experience levels.

Proposal acceptance and modification rates

Figure 4 shows lesion-level proposal acceptance and modification rates across assistance conditions, stratified by reader experience. Expert-generated proposals were accepted more frequently than AI-generated proposals by radiologists (76.7% [95% CI, 74.8-78.5] vs 61.3% [95% CI, 59.2-63.4]) and residents (81.0% [95% CI, 78.6-83.2] vs 62.2% [95% CI, 59.2-65.0]). Substantial modification was defined as a greater than 20% relative size change from the proposal or a change in lesion measurability status. Substantial modification was less frequent for expert-generated than AI-generated proposals among radiologists (10.6% [95% CI, 9.4-12.0] vs 22.6% [95% CI, 20.8-24.5]) and residents (8.8% [95% CI, 7.3-10.7] vs 20.6% [95% CI, 18.3-23.2]). Additional descriptive results are provided in Appendix A.

Figure 4.

Proposal acceptance rates under different assistance conditions, grouped by experience level. / AI assistance, / Expert assistance. Right-side plots depict the frequency of change magnitudes (relative to proposal size) for modified measurements. ‘m → nm’ (measurable → nonmeasurable) indicates lesions initially measured by the proposer but deemed nonmeasurable by the reader; ‘nm → m (nonmeasurable → measurable’) indicates lesions initially considered nonmeasurable but subsequently measured by the reader.

Discussion

In this retrospective study, we evaluated the impact of AI and expert assistance on reading times and interobserver variability in RECIST target lesion measurement on body CT of patients with cancer, using a diverse group of international readers with varying experience levels. Our results provide strong evidence that AI assistance reduces reading times compared with unassisted reading. AI assistance also reduced the number of patients for whom there was outcome disagreement based on the SLD. However, we observed a slight increase in deviation from the expert-derived mean lesion size when using AI assistance compared with unassisted reading. Notably, expert assistance outperformed AI assistance in both time efficiency (48% vs. 34% relative reduction) and measurement consistency. These findings suggest that AI assistance can improve RECIST workflow efficiency and patient-level response consistency without requiring a second human reader, but that current AI proposals remain less reliable than expert-generated measurements and still require reader oversight.

There have been increasing calls to automate RECIST assessment,¹⁹ motivated by the need to reduce measurement variability and manual workload. Prior studies have explored various forms of computer-aided support, including automating RECIST outcome calculations,^10,20–22 tools to suggest likely lesion locations on follow-up imaging,²³ and segmentation-based measurement assistance.²⁴ Earlier work proposed automated measurement and segmentation of lesions across time points.^9,12 Similar to these segmentation models, the system evaluated in this study can streamline longitudinal assessment, requiring only manual identification of baseline target lesions. However, compared with earlier work on computer-assisted RECIST evaluation, our study includes a larger and more heterogeneous reader pool, spanning multiple institutions and levels of experience, and evaluates performance on a large, diverse, multicenter dataset of lesions identified during clinical practice.

Strong measurement proposals, whether generated by AI or experts, can substantially reduce interreader variability, with high acceptance rates observed for both assisted conditions. Notably, AI occasionally outperformed human experts in identifying the longest axial diameter, a task that is particularly challenging and time-consuming for large or irregular lesions, as illustrated in Figure 5. AI proposals required adjustment more frequently than their expert-generated counterparts, especially in cases with significant anatomical changes between time points. Nevertheless, even imperfect AI proposals frequently served as a useful localization aid by directing readers to the lesion vicinity, which may reduce search burden relative to unassisted reading, as illustrated in Figure 6. These findings underscore the potential of AI-assisted tools in accelerating RECIST assessments, while also identifying opportunities for further model refinement, particularly in complex cases such as disappearing lesions.

Figure 5.

Partial crop of an axial slice showing a lung lesion annotated under various conditions. Baseline measurement, Expert-assisted measurement, AI-assisted measurement, Unassisted measurements. Note how both the AI and Expert proposal lead to universal agreement on measurement size, whereas in the unassisted condition measurements by radiologists range from 62.6 to 69.5 mm. Shown using lung window (W: 1500 L:-600).

Figure 6.

Partial crop of an axial slice showing an axillary lymph node annotated under various conditions. Baseline measurement, Expert-assisted measurement, AI-assisted measurements, Unassisted measurements, Modified measurements. In this case, the AI system fails to accurately predict the lesion size after significant shrinkage. Shown using soft-tissue window (W: 400 L: 50).

Consistent with our findings, prior meta-analyses have reported substantial interobserver variability in manual RECIST measurements.⁷ Factors such as lesion size, margin sharpness, and lesion type or location are known contributors to inter-observer variability.²⁵ However, both AI and expert assistance reduced variability in longitudinal SLD, as shown in Figure S3. SLD differences ≤ 20% across radiologists were observed in 82.6% of expert-assisted cases, 71.7% with AI assistance, and 56.6% in the unassisted condition. These findings indicate that AI assistance enhances measurement consistency and can reduce variability in RECIST response assessments, although it falls short of the consistency achieved with a second human reader. Further narrowing the gap between AI- and expert-assisted reading will likely require more robust lesion reidentification, better handling of lesions that become nonmeasurable or disappear, and more accurate estimation of the longest axial diameter in large or irregular lesions.

Unlike workflows requiring multiple readers,²⁶ AI-assisted measurements do not add human workload, offering a more scalable and practical solution. AI-assisted RECIST measurement could be implemented as a reader-in-the-loop tool that prelocalizes and premeasures known target lesions, with radiologists retaining responsibility for accepting or modifying each proposal. Such deployment would require monitoring of proposal acceptance and modification rates after implementation and clear documentation of human oversight. Because RECIST measurements can influence treatment-response classification, regulatory evaluation should address not only lesion-level measurement accuracy but also patient-level response agreement and failure modes such as nonmeasurable or disappearing lesions.

Future work could evaluate volumetric tumor assessment as an alternative to conventional RECIST-based measurement. Because our AI system derives measurements from 3-dimensional segmentations, it can identify the longest lesion diameter across all planes rather than being restricted to axial images, as human readers typically are. This capability may be particularly advantageous for irregular or nonspherical lesions, for which volumetric approaches may provide a more faithful representation of tumor burden than conventional unidimensional RECIST metrics.²⁷ Prospective validation in a different healthcare setting and integration into routine PACS workflows would further strengthen the generalizability of these findings.

This study has several limitations. Readers did not use their usual PACS environments, which may have increased reading times relative to routine practice. Additionally, the data were sourced from 2 Dutch institutions, potentially limiting their generalizability to broader, international patient populations. Baseline target lesions were predefined by 2 radiologists, who did not participate in the reader study, based on information from clinical reports. Although this design isolates variability in follow-up lesion measurement, it prevents us from assessing the impact of interobserver differences in baseline lesion selection, an important contributor to variability in RECIST assessments.

This study also did not include nontarget or new lesion assessment. The presence, absence, or change of new and nontarget lesions affects RECIST outcomes, preventing a full guideline-compliant evaluation of outcome. Because these additional assessments also require reading time and were not automated in the present workflow, the observed reading-time savings would likely be attenuated in a complete RECIST 1.1 assessment. Incorporating a detection model would be a logical next step toward a more complete AI-assisted assessment.

The crossover design meant that unassisted, AI-assisted, and expert-assisted measurements for a given patient came from different readers, introducing between-reader variability as a potential confounder. Moreover, because we did not have an independent reference panel to establish consensus lesion measurements, we used the mean measurement of the expert-assisted radiologists as the reference standard. This inherently biases results towards the expert-assisted group for the measurement variability result.

Finally, to prevent bias related to the measurement source, readers were informed that all proposals were AI-generated. This blinding approach was intended to mitigate automation bias and differential trust in human versus automated decision support.²⁸ In a nonblinded setting, readers might place greater trust in expert-derived measurements, potentially widening the observed difference in reading time between AI-assisted and expert-assisted conditions.

In conclusion, this study demonstrates that AI assistance can substantially accelerate RECIST evaluations and improve SLD consistency. Although expert assistance currently achieves superior interobserver agreement and greater time efficiency, AI systems offer a scalable alternative requiring no additional human input. By benchmarking AI support against experienced human readers and using a diverse, international reader cohort, this study provides critical insight into the current capabilities and limitations of AI in RECIST assessment. Integrating AI assistance into clinical workflows could help mitigate workload pressures while maintaining diagnostic rigor and measurement consistency.

Author contributions

Max J.J. de Grauw (Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Validation, Visualization, Writing—original draft, Writing—review & editing), Max Westphal (Formal analysis, Methodology, Software, Validation, Writing—review & editing), Ewoud J. Smit (Conceptualization, Data curation, Writing—review & editing), Ernst Th Scholten (Data curation, Writing—review & editing), Tanja Loßau and Jan Moltz (Software, Writing—review & editing), Silvia Bottazzi, Renato Cuocolo, Anna D’Angelo, Ajo B. George, Hugo C. van Heusden, Adriano Liguori, Valentina Longo, Luigi Mannacio, Nguyen T.N. Minh, Ahmed E. Othman, Andrea Ponsiglione, Joey Roosen, Maarten de Rooij, Luca Russo, Steven Schalekamp, Miranda Snoeren, Arnaldo Stanzione, Sebastian Steinmetz, Carlijn I. Verkroost, Bastiaan Vernhout, Lina Xu, and Derya Yakar (Data curation, Writing—review & editing), Matthieu Rutten (Data curation, Resources, Writing—review & editing), Bram van Ginneken (Conceptualization, Funding acquisition, Supervision, Writing—review & editing), Mathias Prokop (Conceptualization, Supervision, Writing—review & editing), and Alessa Hering (Conceptualization, Methodology, Project administration, Supervision, Writing—review & editing)

Supplementary material

Supplementary material is available at Radiology Advances online.

Funding

This research was supported by the ALTRIS project (Eurostars-3, Call 5, project 4787).

Conflicts of interest

The authors declare that they have no competing financial interests or personal relationships that could have influenced the research in this study.

Data availability

Access to deidentified data can be provided upon reasonable request to the corresponding author, subject to approval by the relevant institutional review boards and completion of an appropriate data-use agreement.