External Validation of a Previously Developed Deep Learning–based Prostate Lesion Detection Algorithm on Paired External and In-House Biparametric MRI Scans

Abstract

Purpose

To evaluate the performance of an artificial intelligence (AI) model in detecting overall and clinically significant prostate cancer (csPCa)–positive lesions on paired external and in-house biparametric MRI (bpMRI) scans and assess performance differences between each dataset.

Materials and Methods

This single-center retrospective study included patients who underwent prostate MRI at an external institution and were rescanned at the authors’ institution between May 2015 and May 2022. A genitourinary radiologist performed prospective readouts on in-house MRI scans following the Prostate Imaging Reporting and Data System (PI-RADS) version 2.0 or 2.1 and retrospective image quality assessments for all scans. A subgroup of patients underwent an MRI/US fusion–guided biopsy. A bpMRI-based lesion detection AI model previously developed using a completely separate dataset was tested on both MRI datasets. Detection rates were compared between external and in-house datasets with use of the paired comparison permutation tests. Factors associated with AI detection performance were assessed using multivariable generalized mixed-effects models, incorporating features selected through forward stepwise regression based on the Akaike information criterion.

Results

The study included 201 male patients (median age, 66 years [IQR, 62–70 years]; prostate-specific antigen density, 0.14 ng/mL² [IQR, 0.10–0.22 ng/mL²]) with a median interval between external and in-house MRI scans of 182 days (IQR, 97–383 days). For intraprostatic lesions, AI detected 39.7% (149 of 375) on external and 56.0% (210 of 375) on in-house MRI scans (P < .001). For csPCa-positive lesions, AI detected 61% (54 of 89) on external and 79% (70 of 89) on in-house MRI scans (P < .001). On external MRI scans, better overall lesion detection was associated with a higher PI-RADS score (odds ratio [OR] = 1.57; P = .005), larger lesion diameter (OR = 3.96; P < .001), better diffusion-weighted MRI quality (OR = 1.53; P = .02), and fewer lesions at MRI (OR = 0.78; P = .045). Better csPCa detection was associated with a shorter MRI interval between external and in-house scans (OR = 0.58; P = .03) and larger lesion size (OR = 10.19; P < .001).

Conclusion

The AI model exhibited modest performance in identifying both overall and csPCa-positive lesions on external bpMRI scans.

Keywords: MR Imaging, Urinary, Prostate

Supplemental material is available for this article.

© RSNA, 2024

Summary

An artificial intelligence model achieved modest performance in identifying both overall and clinically significant prostate cancer lesions on external biparametric MRI scans, potentially streamlining disease management and reducing patient referrals to academic centers.

Key Points

• ■ In a retrospective study of 201 patients, the artificial intelligence model detected 39.7% (149 of 375) and 56.0% (210 of 375) of all intraprostatic lesions on external and in-house MRI scans, respectively (P < .001).

• ■ On external MRI scans, a higher Prostate Imaging Reporting and Data System score (odds ratio [OR] = 1.57; P = .005), larger size (OR = 3.96; P < .001), and higher diffusion-weighted MR image quality (OR = 1.53; P = .02) were associated with better overall lesion detection.

Introduction

In the United States, prostate cancer (PCa) represents the most prevalent noncutaneous malignancy among males and is a major contributor to cancer-associated mortality, accounting for more than 35 000 deaths annually (1). Multiparametric MRI has emerged as a critical tool for ruling out PCa (2) and guiding biopsy needles (3). An MRI/US fusion–guided biopsy demonstrated superior reliability in detecting clinically significant PCa (csPCa) compared with the traditional systematic transrectal US–guided 12-core biopsy (4).

Incorporating MRI into the diagnostic pathway presents challenges due to the variability in MRI scan quality and interpretive expertise (5). The Prostate Imaging Reporting and Data System (PI-RADS) was introduced to standardize MRI diagnosis, offering detailed guidelines for sequence-specific scoring relevant to distinct prostatic zones (6). Single-center prospective studies validated the efficacy of this guideline (7); however, substantial heterogeneity was noted in PI-RADS performance across 26 centers (8). Beyond academic environments, adherence to PI-RADS technical recommendations remains inconsistent, and even when followed, it does not always ensure high-quality images (9). Quality assurance initiatives such as Prostate Imaging Quality are poised to enhance image quality (10), but the effects of these measures on global clinical practices could take considerable time to fully emerge (11).

In recent years, machine learning models and deep learning algorithms have shown promise in various medical imaging applications, including prostate lesion segmentation and detection (12). External testing remains a pivotal facet of artificial intelligence (AI) studies, serving to enhance model performance on previously unencountered data (13). The outcomes of external testing studies are paramount in determining the generalizability and reproducibility of these models (14). Due to inherent resource and time limitations, however, these external testing datasets predominantly stem from single-center sources (15). Additionally, the actual performance of many academically developed AI models has not been tested using data obtained at nonacademic institutions, where many patients initially seek care.

In the current study, we aim to evaluate the performance of a previously developed biparametric MRI (bpMRI)–based deep learning algorithm (16) in detecting intraprostatic lesions on paired external and in-house MRI scans and assess performance differences between each dataset.

Materials and Methods

Study Sample

This retrospective study adhered to Health Insurance Portability and Accountability Act guidelines and received approval from the National Cancer Institute Institutional Review Board. Each patient was registered under at least one of the following clinical protocols: NCT00102544, NCT02594202, or NCT03354416. All patients provided informed written consent before enrollment. Our investigation included consecutive patients who had previously undergone prostate MRI outside our institution (hereafter referred to as external MRI scans) and were subsequently rescanned at our facility (hereafter referred to as in-house MRI scans) between May 2015 and May 2022. Our center is a research institution where all patients referred from other institutions are given the option of repeat imaging for a recent evaluation of their prostate gland to aid in guiding clinical management. Exclusion criteria included prior PCa treatment, inclusion in the AI model training, absence of a bpMRI sequence (T2-weighted MRI or diffusion-weighted MRI [DW MRI]) in external scans, and external scans not dedicated to the prostate (Fig 1). Seventy-five of the 201 patients had been included in prior studies investigating the cancer yields of different PI-RADS categories per PI-RADS versions 2.0 (17) and 2.1 (7). The current study focused on the lesion detection performance of a previously developed AI model on paired external and in-house MRI scans, both of which were used as external testing datasets, as there was no overlap between the current study sample and the original AI development training dataset (16).

Figure 1:

Flowchart delineates the selection process of eligible patients, outlining both the inclusion and exclusion criteria. AI = artificial intelligence, bpMRI = biparametric MRI, TBx = MRI/US fusion–guided targeted biopsy.

Four authors (D.Y., Z.X., J.T., D.X.) are employed by NVIDIA. However, two authors (E.C.Y. and B.T.) analyzed and controlled the entire data in this work, and they are not employed by or a consultant for a company in the medical industry.

Image Acquisition and Interpretation

The external MRI datasets were sourced from nearly 100 institutions, encompassing academic centers, private practices, and community hospitals, to ensure a representative sample of real-world clinical settings. External MRI scans were acquired on a range of scanner models, including those from GE Medical Systems, Hitachi Medical, Philips, Siemens, and Toshiba, using magnetic field strengths of 3 T, 1.5 T, or 1.15 T. The in-house scans were obtained using a 3-T (Achieva 3.0T TX [Philips], Ingenia Elition 3.0T X [Philips], or Verio 3 T [Siemens]) or 1.5-T (Aera 1.5 T [Siemens]) machine. Images were acquired using a surface coil (SENSE; Philips Healthcare). Breakdowns regarding the use of endorectal coil (BPX-30; MEDRAD), scanner vendors, and magnetic field strengths are provided in Tables S1 and S2.

A genitourinary radiologist (B.T., with more than 15 years of experience in PCa imaging) prospectively assessed all in-house images as part of clinical management. Retrospective evaluations of the external MRI scans were not considered in the study. Interpretations from May 2015 to March 2019 adhered to PI-RADS version 2.0 guidelines (18), while those from April 2019 to May 2022 followed PI-RADS version 2.1 (6). Each lesion received an extraprostatic extension grade ranging from 0 to 3 (0, no suspicion of extraprostatic extension; 1, presence of capsular bulge/irregularity or capsular contact length ≥1.5 cm; 2, presence of capsular bulge/irregularity and capsular contact length ≥1.5 cm; 3, overt extraprostatic extension) based on the National Cancer Institute criteria (19). Prostate volumes were planimetrically calculated, and lesion diameters were measured on a single axial T2-weighted MRI section.

MRI/US Fusion–guided Biopsy

A subgroup of patients underwent MRI/US fusion–guided and/or systematic biopsy performed by one of two urologists (P.A.P. or S.G., with more than 15 and 5 years of experience in MRI/US fusion–guided biopsy, respectively) or an interventional radiologist (B.J.W., with 20 years of experience in MRI/US fusion–guided biopsy). Each lesion was sampled twice with MRI guidance with use of a commercially available biopsy platform (UroNav; Invivo).

Histopathologic evaluations were conducted by one of two experienced pathologists (A.T. or M.J.M., both with more than 15 years of experience in genitourinary pathology). International Society of Urological Pathology (ISUP) grade groups (range of categories, 1–5) were used for scoring, with an ISUP grade group of 2 and higher considered csPCa.

Evaluation of Scans with the bpMRI-based AI Model

For external images in which high-b-value DW MRI scans (range, 1400–2000 sec/mm²) were not acquired and apparent diffusion coefficients maps not calculated, diffusion stacks with b values at different levels (range, 0–1000 sec/mm²) were used to generate these pulse sequences (Appendix S1) (6,20,21). External and in-house bpMRI scans were then evaluated for lesion detection using the previously developed cascaded deep learning–based AI model accessible at https://github.com/Project-MONAI/research-contributions/tree/main/prostate-mri-lesion-seg (16). The AI model produced two outputs: a probability map, for which each voxel had a number indicating the likelihood of lesion presence ranging from 0% to 100%, and a binary lesion prediction map, which highlighted only the areas that exceeded a 63% threshold on the probability map. The AI model was originally trained on both an institutional dataset and the publicly available PROSTATEx dataset comprising 1390 patients. Expert lesion segmentations were used as the reference standard in algorithm training, as it was intended to detect all suspicious lesions ranging from PI-RADS 2 to 5, simulating the readout of a radiologist. A three-dimensional U-Net–based deep neural network was used for lesion detection and segmentation, which was cascaded by a three-dimensional residual neural network to filter out benign prostatic hyperplasia nodules (16).

Quality Classification and AI Model Output Evaluation

After a washout period exceeding 6 months, the same genitourinary radiologist who conducted the initial clinical evaluations undertook a retrospective assessment of both external and in-house images. The quality of T2-weighted MRI scans and DW MRI scans was evaluated using a three-tiered scoring system: 2 = diagnostic with minor or no quality issues, 1 = acceptable image quality despite existing quality issues, and 0 = nondiagnostic image.

After the image quality evaluation, AI-generated lesion prediction maps in the simple single-layer output format overlaid on T2-weighted MRI scans were reviewed by the same genitourinary radiologist. AI lesion segmentations that showed any overlap with annotations in the picture archiving and communication system of the lesions documented in the clinical radiology report were classified as true-positive findings (Figs 2, S1), and other AI predictions were categorized as false-positive findings (Fig S2). Anatomic landmarks were used to spatially correlate external scans with in-house MRI scans.

Figure 2:

The diagram is segmented into the clinical workflow, which outlines standard patient care procedures, and research components, which focus on retrospective image quality evaluation and the application of artificial intelligence (AI) on both external and in-house scans. The AI model’s performance is evaluated using radiologist clinical readouts and histopathologic outcomes as reference standards. csPCa = clinically significant PCa, mpMRI = multiparametric MRI, PCa = prostate cancer, TBx = MRI/US fusion–guided targeted biopsy.

PI-RADS and Quality Score Reproducibility Evaluation through a Reader Study

A second genitourinary radiologist (Y.M.L., with more than 9 years of experience in PCa imaging) evaluated a subset of scans, which were selected using a stratified sampling approach to ensure they were representative of the main dataset in terms of PI-RADS and quality scores. This evaluation included identifying an index lesion, assigning a PI-RADS category, and rating the quality of both T2-weighted MRI scans and DW MRI scans.

Statistical Analysis

For overall lesion detection rates, radiologist annotations were used as the reference standard, and all patients were included in the analysis. For PCa and csPCa detection rates, only the targeted biopsy subgroup was considered for the analysis, and targeted biopsy outcomes were the reference standard, using cutoffs of an ISUP grade group of 1 or more and ISUP grade group of 2 or more, respectively. As true-negative findings are typically undefined with radiologist annotations or biopsy results in the lesion-based analysis, specificity and negative predictive value could not be evaluated. Sensitivity (detection rate), positive predictive value, and false discovery rate were calculated for different levels of reference standard (overall, PCa-positive, and csPCa-positive lesion detection). AI lesion detection rates, positive predictive values, and false discovery rates based on external scans were compared with those based on in-house scans. Bootstrap 95% CIs (22) were constructed, and paired comparison permutation tests (23) were performed to test for a difference in these metrics based on external versus in-house scans; the bootstrap involved resampling patients with replacement, and the permutation test involved randomly switching the source of the scan for half the patients during each iteration (ie, for an individual patient, the external label was switched to in-house and vice versa).

Univariable and multivariable associations of lesion detection (detected or not detected) with lesion, patient, and image characteristics were assessed with use of generalized linear mixed-effects models, including a random effects term associated with each patient and using a logit link function. Lesion size, prostate-specific antigen measurements, prostate volume, and number of days between external and in-house MRI scans were analyzed on a log₂ scale. Lesions present in the transition, peripheral, or central zone were coded as indicator variables, and PI-RADS scores, extraprostatic extension grades, and quality metrics were coded numerically (0, 1, and 2 indicated low, moderate, and high, respectively). Odds ratios (ORs) for each univariable association were computed, and Wald tests were used to test whether each OR equaled 1. For the multivariable models, variable selection was performed using forward stepwise regression in which at each step, each variable not included in the model was added and the resulting decrease in the Akaike information criterion was assessed; the variable leading to the largest decrease in Akaike information criterion was then added to the model, and the process stopped when the addition of any further variables resulted in an increase in Akaike information criterion. For external MRI scans, AI lesion detection performance was compared between 3-T and 1.5-T MRI scans with use of the Wald test.

For patient-based analyses, a true-positive finding was defined as the presence of an AI-detected lesion on a scan that was deemed positive by reference standard assessments (radiologist annotation or biopsy-confirmed PCa or csPCa). A true-negative finding was defined as the lack of an AI-detected lesion on scans without positive findings by reference standard assessments (ie, no radiologist-defined, PCa-positive, or csPCa-positive lesions), depending on the level of analysis. Sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated separately for overall, PCa-positive, and csPCa-positive lesion detection. Receiver operating characteristic curves were generated at the patient level using the maximum value obtained from the probability maps, based on the binary outcome per patient for overall lesion, PCa, and csPCa detection. Further details regarding patient and index lesion-based analysis are described in Appendix S1.

Cohen κ analysis was conducted to test interreader agreement for PI-RADS and quality scores. κ coefficients were interpreted as follows: less than 0.20 indicated no agreement; 0.20–0.39, fair agreement; 0.40–0.59, moderate agreement; 0.60–0.79, substantial agreement; and 0.80 or greater, near-perfect agreement. The statistical computations were carried out by a statistician (E.P.H., with more than 15 years of experience in biostatistics) using R software (version 4.2.3; R Project for Statistical Computing). P < .05 was considered indicative of statistically significant difference, and Benjamini-Hochberg procedures were used to control for multiple testing (24).

Results

Patient Characteristics

A total of 369 patients with prior MRI scans acquired at external institutions were rescanned at our institution between May 2015 and May 2022. Patients with a history of PCa treatment (n = 97), patients whose data were included in the initial AI model training (n = 46), those presenting with an incomplete bpMRI examination (n = 23), and those with MRI scans not dedicated to prostate imaging (n = 2) were excluded. The final study sample consisted of 201 male patients with a median age of 66 years (IQR, 62–70 years) and prostate-specific antigen density of 0.14 ng/mL² (IQR, 0.10–0.22 ng/mL²) (Table 1). The median interval between the external and in-house MRI scans was 182 days (IQR, 97–383 days). For those who subsequently underwent biopsy, the median duration from the in-house MRI to biopsy was 28 days (IQR, 7–67 days).

Table 1:

Patient Characteristics

Comparison of Intraprostatic Lesion Detection Performance

A total of 375 intraprostatic lesions were reported in clinical readouts across 188 patients, and 13 patients had no lesions. The lesions were scored as follows: 25.1% (94 of 375) as PI-RADS 5, 44.8% (168 of 375) as PI-RADS 4, 17.6% (66 of 375) as PI-RADS 3, and 12.5% (47 of 375) as PI-RADS 2 or less.

AI detected 39.7% (149 of 375; 95% CI: 34.6, 44.8) and 56.0% (210 of 375; 95% CI: 50.8, 61.0) of all intraprostatic lesions on external and in-house MRI scans, respectively (P < .001) (Fig 3). Detection rates for PI-RADS 4 lesions were 28.0% (47 of 168; 95% CI: 21.0, 35.3) for external scans and 50.6% (85 of 168; 95% CI: 42.7, 58.4) for in-house scans (P < .001), while detection rates for PI-RADS 5 lesions were 79% (74 of 94; 95% CI: 69.7, 87.4) for external scans and 94% (88 of 94; 95% CI: 88.5, 97.9) for in-house scans (P < .001) (Table 2).

Figure 3:

Radar chart compares the external and in-house scans in terms of diagnostic quality proportion of T2-weighted MRI (T2W-MRI), diffusion-weighted MRI (DW-MRI), artificial intelligence model performance for overall lesion detection, as well as prostate cancer (PCa)–positive, clinically significant PCa (csPCa)–positive, and grade group (GG) 3 or greater–positive lesion detection. Dotted lines represent a 10% difference.

Table 2:

Detection Rates of Reported Lesions by PI-RADS Score and TBx Outcome on External and In-House Scans

Correlation of AI-predicted MRI Findings with Histopathologic Outcomes from Biopsy

Only the biopsies performed right after the earliest in-house scan (closest to the MRI scan obtained externally) were included in the analysis when available. Of the total study patients, 64.2% (129 of 201) underwent targeted biopsy, and outcomes of histopathologic analysis were available for 70.1% of the reported lesions (263 of 375). Of these validated lesions, 52.1% (137 of 263) were confirmed to have PCa, with 33.8% (89 of 263) being positive for csPCa and 12.5% (33 of 263) with an ISUP grade group of 3 or above.

Of the lesions confirmed as PCa positive, 52.6% (72 of 137; 95% CI: 44.4, 61.2) were identified by AI on external scans (Fig 4) compared with 73.0% (100 of 137; 95% CI: 65.9, 80.2) on in-house scans (P < .001). For csPCa-positive lesions, AI detection rates were 61% (54 of 89; 95% CI: 50.5, 71.1) for external MRI scans and 79% (70 of 89; 95% CI: 70.1, 87.3) for in-house MRI scans (P < .001). Seventy percent (23 of 33; 95% CI: 53.3, 86.5) of lesions with an ISUP grade group of 3 or above were detected on external scans, and 88% (29 of 33; 95% CI: 76.9, 97.1) were detected on in-house scans (P = .03).

Figure 4:

MRI scans in a 76-year-old patient who presented with a prostate-specific antigen level of 12.5 ng/mL. (A) Initial axial biparametric MRI was conducted at an external institution, with (B) a follow-up examination performed approximately 13 months later at our center. A left mid anterior peripheral zone lesion (Prostate Imaging Reporting and Data System 5) was identified on the in-house MRI scan, which was histologically confirmed as a International Society of Urological Pathology grade group 1 prostate adenocarcinoma. The quality of both T2-weighted MRI (T2W-MRI) and diffusion-weighted MRI was assessed as diagnostic (class 2) for both external and in-house scans. The artificial intelligence (AI) model successfully detected the lesion on both images (arrows). ADC = apparent diffusion coefficient, DWI = diffusion-weighted imaging.

For external scans, there was no evidence of a difference in AI detection rates between 3-T and 1.5-T scans for overall (40.4% [127 of 314] vs 37% [22 of 59], respectively; P = .67), PCa-positive (51.9% [56 of 108] vs 59% [16 of 27], respectively; P = .49), and csPCa-positive (60% [42 of 70] vs 71% [12 of 17], respectively; P = .42) lesion detection (Table S3).

False Discovery Rate and Positive Predictive Value

When considering overall lesion detection, the AI predictions resulted in a total of 82 and 76 false-positive findings on external (range, 0–3 false-positive findings per scan) and in-house (range, 0–4 false-positive findings per scan) MRI scans, respectively. The mean number of false-positive findings across the external MRI dataset (0.41 [95% CI: 0.32, 0.50]) was comparable to that of the in-house MRI dataset (0.38 [95% CI: 0.29, 0.47]) (P = .62). For overall lesion detection, the false discovery rates were 35.5% (82 of 231) on external scans and 26.6% (76 of 286) on in-house scans. The positive predictive value was 64.5% (149 of 231 lesions) for external scans and 73.4% (210 of 286 lesions) for in-house scans.

For patients with available biopsy results (n = 129), the number of false-positive findings were as follows: 66 and 90 for PCa detection on external and in-house scans, respectively, and 84 and 120 for csPCa detection on external and in-house scans, respectively (Table S4). Corresponding false discovery rates were 47.8% (66 of 138 lesions) on external scans and 47.4% (90 of 190 lesions) on in-house scans for PCa detection and 60.9% (84 of 138) and 63.2% (120 of 190), respectively, for csPCa detection. The positive predictive values for PCa detection were 52.2% (72 of 138) on external scans and 52.6% (100 of 190) on in-house scans. For csPCa detection, positive predictive values were 39.1% (54 of 138) on external scans and 36.8% (70 of 190) on in-house images.

Patient-based Results

A total of 188 of 201 patients had an intraprostatic lesion; 77.7% (146 of 188) and 93.1% (175 of 188) were detected by the AI on external and in-house scans, respectively. In patients whose MRI reports indicated no lesions (n = 13), the AI had at least one false-positive prediction in nine and eight patients on external and in-house MRI scans, respectively, resulting in specificity of 31% (four of 13) for external and 38% (five of 13) for in-house scans. The accuracy for overall lesion detection was 74.6% (150 of 201) for external and 89.6% (180 of 201) for in-house scans. Further details regarding the patient-based analysis for overall lesion, PCa, and csPCa detection are provided in Figure S3.

Areas under the receiver operating characteristic curve were 0.60 (95% CI: 0.44, 0.75) for external and 0.77 (95% CI: 0.59, 0.92) for in-house scans for overall lesion detection, 0.62 (95% CI: 0.52, 0.72) for external and 0.80 (95% CI: 0.71, 0.87) for in-house scans for PCa detection, and 0.64 (95% CI: 0.54, 0.74) for external and 0.72 (95% CI: 0.64, 0.81) for in-house scans for csPCa detection (Fig 5).

Figure 5:

Horizontal bar charts and receiver operating characteristic curves for (A) overall lesion, (B) prostate cancer (PCa), and (C) clinically significant PCa (csPCa) detection. Bar charts indicate the patient count in negative (green) and positive (purple) groups. Receiver operating characteristic curves demonstrate the artificial intelligence (AI) performance on external and in-house MRI datasets. AUC = area under the receiver operating characteristic curve.

Image Quality Evaluation

In the assessment of T2-weighted MRI scan quality, external images had a distribution of 40.8% (82 of 201) as diagnostic (class 2), 43.3% (87 of 201) as acceptable (class 1), and 15.9% (32 of 201) as nondiagnostic (class 0) (Fig 6). In contrast, in-house T2-weighted MR images were predominantly graded as class 2 at 79.1% (159 of 201), with 17.9% (36 of 201) rated as class 1 and 3.0% (six of 201) as class 0.

Figure 6:

MRI scans in a 65-year-old patient who presented with a prostate-specific antigen level of 6.3 ng/mL. (A) Initial axial biparametric MRI was conducted at an external institution, followed by (B) a repeat examination approximately 8 months later at our facility. The in-house MRI scan revealed a right apical mid anterior peripheral zone lesion (Prostate Imaging Reporting and Data System [PI-RADS] 5) and a left mid peripheral zone lesion (PI-RADS 2). These lesions were pathologically confirmed as International Society of Urological Pathology grade group 1 and 2 prostate adenocarcinomas, respectively. At quality assessment of the external T2-weighted MRI (T2W-MRI) and diffusion-weighted MRI scans, the images were classified as nondiagnostic (class 0), whereas the in-house T2-weighted MRI scan was deemed diagnostic (class 2) and the DW MRI scan as acceptable (class 1). The artificial intelligence (AI) model successfully detected the right-sided lesion in both external and in-house images (arrows). However, it identified the left-sided lesion only on the in-house MRI scan (arrowheads). ADC = apparent diffusion coefficient, DWI = diffusion-weighted imaging.

For the DW MRI scan quality evaluation, external images were categorized as 35.8% (72 of 201) class 2, 39.3% (79 of 201) class 1, and 24.9% (50 of 201) class 0. The in-house DW MRI scans demonstrated a quality class distribution of 77.1% (155 of 201) as class 2, 13.4% (27 of 201) as class 1, and 9.5% (19 of 201) as class 0.

Reproducibility of PI-RADS Assignment and Quality Classification

A second radiologist reviewed 20.4% of the scans (41 of 201) for reproducibility assessment. The interreader agreement on the index lesion PI-RADS score was moderate (quadratic-weighted κ = 0.49 [95% CI: 0.17, 0.79]; unweighted κ = 0.57 [95% CI: 0.37, 0.75]). For the subset of scans (78% [32 of 41]) for which the same lesion was identified as the index lesion by both readers, the interreader agreement for PI-RADS classifications was substantial (quadratic-weighted κ = 0.76 [95% CI: 0.34, 0.93]; unweighted κ = 0.66 [95% CI: 0.42, 0.88]). The interreader agreement for T2-weighted MRI scan quality scores was fair to moderate (quadratic-weighted κ = 0.43 [95% CI: −0.05, 0.73]; unweighted κ = 0.28 [95% CI: −0.05, 0.58]). For DW MRI scan quality scores, the agreement also ranged from fair to moderate (quadratic-weighted κ = 0.50 [95% CI: 0.16, 0.74]; unweighted κ = 0.31 [95% CI: 0.05, 0.55]).

Associations between AI Performance on External and In-House MRI Scans and Lesion, Patient, and Image Characteristics

The multivariable analysis showed that AI-based detection of intraprostatic lesions on external MRI scans was associated with larger lesion size (OR = 3.96 [95% CI: 2.40, 6.53]; P < .001), higher PI-RADS score (OR = 1.57 [95% CI: 1.15, 2.15]; P = .005), higher quality of DW MRI scans (OR = 1.53 [95% CI: 1.06, 2.20]; P = .02), and lower number of lesions on MRI scans (OR = 0.78 [95% CI: 0.61, 0.995]; P = .045). The detection of PCa-positive lesions on external MRI scans was associated with larger lesion size (OR = 9.40 [95% CI: 4.31, 20.52]; P < .001), higher quality of T2-weighted MRI scans (OR = 1.89 [95% CI: 1.05, 3.40]; P = .03), and higher prostate volume (OR = 2.35 [95% CI: 1.05, 5.26]; P = .04), while the detection of csPCa-harboring lesions was associated with larger lesion size (OR = 10.19 [95% CI: 3.57, 29.08]; P < .001) and lower number of days between MRI scans (OR = 0.58 [95% CI: 0.35, 0.95]; P = .03) (Table 3).

Table 3:

Lesion-level Univariable and Multivariable Analyses for Detection of All, PCa-positive, and csPCa-positive Lesions by Artificial Intelligence on External MRI Scans

For the in-house MRI scans, the multivariable analysis showed that AI-based detection of intraprostatic lesions was associated with larger lesion size (OR = 2.32 [95% CI: 1.56, 3.46]; P < .001), higher PI-RADS score (OR = 1.97 [95% CI: 1.44, 2.70]; P < .001), higher extraprostatic extension grade (OR = 1.96 [95% CI: 1.02, 3.76]; P = .04), and higher quality of the T2-weighted MRI scans (OR = 1.97 [95% CI: 1.06, 3.69]; P = .03). Larger lesion size was associated with PCa-positive (OR = 6.29 [95% CI: 3.11, 12.72]; P < .001) and csPCa-positive (OR = 4.39 [95% CI: 2.02, 9.53]; P < .001) lesion detection on in-house MRI scans (Table 4).

Table 4:

Lesion-level Univariable and Multivariable Analyses for Detection of All, PCa-positive, and csPCa-positive Lesions by Artificial Intelligence on In-House MRI Scans

Results regarding index lesion detection for external and in-house MRI scans are provided in Tables S5 and S6, respectively.

Discussion

The need for automated prostate lesion detection has resulted in the release of numerous AI algorithms, most of which have not yet been validated on diverse multi-institutional datasets. In our study, we aimed to bridge this gap by comparing the performance of a bpMRI-based AI model in detecting intraprostatic lesions on paired external and in-house images. We observed differences in overall lesion detection between external and in-house scans (39.7% [149 of 375 lesions] vs 56.0% [210 of 375 lesions]; P < .001) as well as in csPCa-positive lesion detection (61% [54 of 89 lesions] vs 79% [70 of 89 lesions]; P < .001). On external images, overall lesion detection was associated with higher PI-RADS score (OR = 1.57; P = .005), larger lesion size (OR = 3.96; P < .001), better quality of DW MRI scans (OR = 1.53; P = .02), and lower number of lesions on MRI scans (OR = 0.78; P = .045), whereas csPCa lesion detection was associated with a smaller interval between two scans (OR = 0.58; P = .03) and a larger lesion diameter (OR = 10.19; P < .001).

As PCa is known to be multifocal, we based our main analysis on lesion-level results in contrast to some of the earlier investigations (12,25). Meta-analyses of PI-RADS versions 2.0 and 2.1 reported cancer yields of 17%–20%, 46%–52%, and 75%–89% for categories 3, 4, and 5, respectively (26,27). Because we aimed to evaluate the real-world applicability of the AI model, we opted to include treatment-naive patients with any MRI findings regardless of their PI-RADS assignment and follow-up biopsy status. We observed a stepwise increase in detection rates on external scans with higher PI-RADS scores: 25% for PI-RADS of 3 or less, 28% for PI-RADS 4, and 79% for PI-RADS 5 lesions. The interreader agreement rates for PI-RADS 3 are known to be low (28), and the relatively low detection rate of PI-RADS 4 lesions on external scans may be partially attributed to the upgrading of peripheral zone lesions to category 4 with contrast enhancement, which would otherwise be in category 3 if assessed at bpMRI alone (6).

In our study, we noted a 16%–20% difference between external and in-house scans in overall and cancerous lesion detection rates. Multivariable analysis on external scans revealed that lesions with higher PI-RADS scores and larger sizes were more likely to be detected by the AI, a finding that was also reported by a prior study (29). Notably, a shorter interval between acquisition of image pairs was indicative of better AI performance in detecting csPCa-harboring lesions on external images (OR = 0.58; P = .03). This result could be attributed to the csPCa not exhibiting marked positive signal characteristics at the time of the earlier MRI examination, rendering it not easily perceivable in some cases. Furthermore, as MRI increasingly becomes integral to active surveillance protocols (30), such automated systems might offer valuable assistance in monitoring temporal changes in lesion volume and signal characteristics.

Although external testing of AI models across completely separate or multiple institutions is rare, a few studies have challenged existing AI models in such settings (31,32). For instance, Hamm et al (31) used the publicly available PROSTATEx dataset for external testing, achieving an area under the receiver operating characteristic curve of 0.87 and a lesion-based sensitivity of 90% (297 of 330) for csPCa detection. However, the images for both their training and external testing datasets were obtained using scanners from a single vendor, which limits the generalizability of their findings. In contrast, our study included images from over 20 different scanner types across multiple vendors. Another study by Sun et al (32) evaluated a bpMRI-based AI model using an external testing dataset composed of images from 14 scanners across four vendors and three hospitals. Their model reached a lesion-level sensitivity of 65.4% (284 of 434) for csPCa-positive lesions among 245 patients. Unlike Sun et al (32), who excluded images of poor quality, we included all images regardless of their quality, thereby enhancing the relevance of our findings to real-world clinical settings beyond academic centers.

Image quality is a critical factor in prostate MRI assessment and AI performance (33). In our study, the quality of DW MRI and T2-weighted MRI scans influenced AI performance on external and in-house scans, respectively. Notably, 59% of external T2-weighted MRI scans and 64% of external DW MRI scans were classified as not of optimal quality, echoing findings from previous studies indicating low-quality prostate MRI scans outside academic practices (34,35). Initiatives like automated quality classification (36) and deep learning–based image reconstructions (37) could partially address T2-weighted MRI–associated quality concerns; however, more work is needed for DW MRI. In their recent publication, Tavakoli et al (38) join a growing list of investigators who believe that dynamic contrast-enhanced MRI adds minimal or no value and bpMRI is adequate for diagnostic work-up. However, they did not address the quality variations in T2-weighted MRI, and DW MRI and the value of dynamic contrast-enhanced imaging is best observed when either sequence has unacceptable quality. Our findings suggest that although bpMRI can be used to detect a considerable proportion of lesions, low-quality T2-weighted MRI or DW MRI can degrade lesion detection performance.

Our study had limitations. First, all readouts were performed by one expert genitourinary radiologist. Second, only targeted biopsy histopathologic analysis was used. Although systematic transrectal US–guided 12-core biopsy results could provide additional data, we opted to use targeted biopsy outcomes, as they offer a better correlation with whole-mount pathology. Third, sequence-specific quality evaluation was conducted using a three-point scale, as the initial version of Prostate Imaging Quality scoring system applies only to multiparametric MRI and is assigned on a scan level (10), and we aimed to differentiate the impact of T2-weighted MRI and DW MRI scans. Fourth, due to our study design and reference standard selection, the positive AI predictions that lacked corroboration with in-house MRI evaluations were classified as false-positive findings, possibly causing an excessive penalization. Ideally, conducting prospective studies at each participating institution, where AI-detected lesions undergo verification, would serve as the most effective validation technique. Fifth, there was potential self-selection bias introduced by patients opting in for repeat imaging. These patients may differ from those who decline repeat imaging, which could affect the generalizability of our results. Finally, the AI model tested in our study did not have a detailed explainable component (31); however, having a probability map may help readers combine AI predictions that appear separately on the detection map.

In conclusion, our study underscores the potential of a deep learning AI model to detect intraprostatic lesions from bpMRI scans obtained at various external institutions. The demonstrated level of generalizability of the AI makes it a valuable support tool for local radiologists in nontertiary health care settings. Its application may reduce the need for additional imaging in specialized centers, thereby expediting the diagnosis and treatment of patients with PCa. Future efforts should be directed toward prospective validation of this tool across a diverse range of institutions.