Factors Influencing Variability in the Performance of Multiparametric Magnetic Resonance Imaging in Detecting Clinically Significant Prostate Cancer: A Systematic Literature Review

Abstract

Context:

There is a lack of comprehensive data regarding the factors that influence the diagnostic accuracy of multiparametric magnetic resonance imaging (mpMRI) to detect and localize clinically significant prostate cancer (csPCa).

Objective:

To systematically review the current literature assessing the factors influencing the variability of mpMRI performance in csPCa diagnosis.

Evidence acquisition:

A computerized bibliographic search of Medline/PubMed database was performed for all studies assessing magnetic field strength, use of an endorectal coil, assessment system used by radiologists and inter-reader variability, experience of radiologists and urologists, use of a contrast agent, and use of computer-aided diagnosis (CAD) tools in relation to mpMRI diagnostic accuracy.

Evidence synthesis:

A total of 77 articles were included. Both radiologists’ reading experience and urologists’/radiologists’ biopsy experience were the main factors that influenced diagnostic accuracy. Therefore, it is mandatory to indicate the experience of the interpreting radiologists and biopsy-performing urologists to support the reliability of the findings. The most recent Prostate Imaging Reporting and Data System (PI-RADS) guidelines are recommended for use as the main assessment system for csPCa, given the simplified and standardized approach as well as its particular added value for less experienced radiologists. Biparametric MRI had similar accuracy to mpMRI; however, biparametric MRI performed better with experienced readers. The limited data available suggest that the combination of CAD and radiologist readings may influence diagnostic accuracy positively.

Conclusions:

Multiple factors affect the accuracy of mpMRI and MRI-targeted biopsy to detect and localize csPCa. The high heterogeneity across the studies underlines the need to define the experience of radiologists and urologists, implement quality control, and adhere to the most recent PI-RADS assessment guidelines. Further research is needed to clarify which factors impact the accuracy of the MRI pathway and how.

Patient summary:

We systematically reported the factors influencing the accuracy of multiparametric magnetic resonance imaging (mpMRI) in detecting clinically significant prostate cancer (csPCa). These factors are significantly related to each other, with the experience of the radiologists being the dominating factor. In order to deliver the benefits of mpMRI to diagnose csPCa, it is necessary to develop expertise for both radiologists and urologists, implement quality control, and adhere to the most recent Prostate Imaging Reporting and Data System assessment guidelines.

1. Introduction

Over the last 10 yr, the diagnostic pathway of prostate cancer (PCa) has changed significantly by the advent of multiparametric magnetic resonance imaging (mpMRI) [1,2]. As shown by recently published randomized controlled trials, head-to-head comparisons, and a Cochrane meta-analysis [3-8], mpMRI is the best technique to detect and localize suspicious areas for clinically significant prostate cancer (csPCa), and it allows performing MRI-targeted biopsy (MRI-TBx) [9]. The value of mpMRI and MRI-TBx over systematic transrectal ultrasound (TRUS) biopsy is in reducing diagnoses of insignificant PCa and potentially avoiding unnecessary prostate biopsies in men with negative mpMRI scans. Detection and localization of csPCa are slightly but not significantly better with mpMRI than with TRUS biopsy.

Acquisition and interpretation of prostate mpMRI are evolving with ongoing improvements, which influence its accuracy. These include magnetic field strength, gradient strength, use of an endorectal coil (ERC) [10], different versions of assessment systems, reader experience and inter-reader variability, potential to avoid a contrast agent (ie, biparametric MRI [bpMRI]), and use of computer-aided diagnosis (CAD) and deep-learning tools. Another factor that influences the accuracy of the MRI pathway is the experience of operators performing MRI-TBx. These show a significant variation across reported series [4], affecting the risk of bias of the available data and preventing robust systematic analyses. Moreover, aiming at assessing the variation of mpMRI diagnostic accuracy, variability of biopsy protocols and histopathological reference standards, and heterogeneity of PCa prevalence among different cohorts with the consequent variation of the negative predictive value (NPV) of mpMRI make comparison of studies even more challenging [11].

Despite the large number of studies reporting the accuracy of mpMRI, there is a lack of comprehensive data that specifically address the difference of mpMRI execution and performance. Given the promising role of MRI in csPCa diagnosis, there is a need to systematically review the current literature regarding the factors that influence the variability of mpMRI in the diagnosis of csPCa.

2. Evidence acquisition

2.1. Objective

We aimed to systematically review the current literature assessing the factors influencing the variability of mpMRI performance in detecting csPCa. Magnetic field strength (1.5 vs 3.0 T, including importance of the gradient strength), use of an ERC, assessment system used by the radiologist, inter-reader variability, experience of the radiologists and urologists, use of bpMRI, and use of CAD or deep learning or machine learning for mpMRI assessment were considered potential influencing factors.

2.2. Search strategy

Data collection was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement [12]. A computerized bibliographic search of Medline/PubMed database was searched from inception to June 23, 2019. The search terms used were (prostate cancer OR prostate adenocarcinoma) AND (MRI OR magnetic resonance) AND (coil OR endorectal coil OR surface coil OR magnetic field OR reporting system OR PI-RADS OR Likert OR inter-reader variability OR inter-reader agreement OR biparametric OR radiologist experience OR urologist experience OR learning curve OR CAD OR machine learning OR computer-aided OR artificial intelligence OR neural network).

2.3. Inclusion criteria

As recommended by the PRISMA guidelines, we used the population, intervention, comparator, and outcome (PICO) approach to define study eligibility [12]. Reports were considered relevant if they provided comparative data on the relationship between the aforementioned factors. The performance of mpMRI was defined as PCa detection at either prostate biopsy or after radical prostatectomy. Thus, studies assessing one of the factors without providing any comparison (eg, providing data regarding the accuracy of 1.5 T mpMRI or bpMRI alone) were not included in this review. Noncomparative studies, case reports, editorials, letters, review articles, and meeting abstracts were also not included.

2.4. Systematic review process

Two authors (A.S. and F.G.) independently reviewed a total of 2013 abstracts and selected 77 studies that were finally included in the systematic review for full-text evaluation. Fig. 1 shows the PRISMA flowchart describing the selection process.

Fig. 1 –

Preferred Reporting Items for Systematic Reviews and Meta-analysis flow diagram showing the outcome of the initial and additional searches resulting in the inclusion of full studies in the review. MRI = magnetic resonance imaging.

2.5. Data extraction

Data were independently extracted from all included studies by the same two authors. A standardized data extraction form was created a priori and used to collect data on the study design, number of participants, mpMRI protocol, radiologist experience, and outcome.

2.6. Data analysis

A comprehensive and narrative synthesis of included studies was performed, since a quantitative meta-analytic synthesis was not possible due to the heterogeneity of the studies.

2.7. Risk of bias assessment

The risk of bias and applicability concern in individual studies was assessed independently by the same two authors using the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) criteria [13]. The presence of baseline confounding factors or selection bias, as well as the presence of any bias within mpMRI protocols, mpMRI interpretation, biopsy protocol, and histopathological reference standard, was assessed (Fig. 2 and Supplementary Fig. 1).

Fig. 2 –

Overall summary of risk of bias and applicability concerns across studies based on QUADAS-2 criteria.. QUADAS-2 = Quality Assessment of Diagnostic Accuracy Studies-2.

3. Evidence synthesis

Overall, 2013 publications were found. If it was not clear from the abstract whether the paper might contain relevant data, the full paper was assessed. Seventy-seven articles were included in the final analysis (Fig. 1). Single studies are described in detail in Tables 1-4 and Supplementary Tables 1-3.

Table 1 –

List of studies comparing different mpMRI assessment systems.

Author [ref]	Year	Study design	Number of patients	Scoring systems used	MRI protocol	Number of radiologists	Definition of csPCa	Reference standard	Key findings
Auer et al [34]	2017	Prospective (v1) and retrospective (v2.0)	50	PI-RADS v1 and v2	T2-WI, DWI, and DCE	2	Low grade (Gleason score ≤3 + 4) vs high-grade (Gleason score ≥4 + 3)	Radical prostatectomy	• PI-RADS v1 detects tumor better than v2 (AUC: 0.96 vs 0.90)
									• PZ lesions: PI-RADS v1 (AUC: 0.97) vs v2 (AUC: 0.92)
									• TZ lesions, PI-RADS v1 (AUC: 0.96) vs v2 (AUC: 0.90)
									• PI-RADS v2 resulted in significantly more false-negative results (3% vs 14%) and a similar true positive result (82% vs 80%)
De Visschere et al [35]	2016	Retrospective	245	PI-RADS v1 and v2	T2-WI, DWI, DCE, and MRSI	1	Gleason score ≥7 (including 3 + 4 with prominent but not predominant Gleason 4 component), and/or tumor volume of ≥0.5 cc, and/or tumor stage ≥ T3a	MRI-TBx	• PI-RADS v1 and v2 overall assessment scores were significantly higher (p < 0.001) in patients with csPCa
									• ROC curve: 0.82 for PI-RADS v1 and 0.79 for PI-RADS v2.0 (p > 0.05)
									• Using a threshold of 3, sensitivity was 88.2% and 79.2% (p = 0.001) and specificity was 64.4% and 67.3% (p > 0.05) with PI-RADS v1 and v2, respectively
Feng et al [37]	2016	Retrospective	401	PI-RADS v1 and v2	T2-WI, DWI, and DCE	1	–	MRI-TBx	• AUC: 0.889 for PI-RADS v1 and 0.942 for PI-RADS v2 (p = 0.0001) pooling TZ and PZ together
									• Higher sensitivity in the TZ (96% vs 76%, p = 0.003), similar specificity (90% vs 84%, p = 0.227), and higher accuracy (93% vs 81%, p = 0.002) for PI-RADS v2
Hoffmann et al [38]	2018	Prospective	58	PI-RADS v1 and v2	T2-WI, DWI, and DCE	2	Epstein criteria (PSA density ≥0.15 ng/ml/g, Gleason score >3 + 3, presence of PCa in >3 cores with >50% involvement in any of the cores	Biopsy (n = 58) and radical prostatectomy (n = 29)	• Substantial agreement between radiologists (PI-RADS v.1: kappa 0.71; PI-RADS v.2: kappa 0.69)
Krishna et al [40]	2017	Retrospective	47	PI-RADS v1 and v2	T2-WI, DWI, and DCE	3	Gleason score 3 + 4 with tumor foci ≥0.5 cm3	Radical prostatectomy	• Higher sensitivity for PI-RADS v1 (p = 0.01 and 0.03, radiologists 1 and 2)
									• Moderate interobserver agreement for PI-RADS v2 (k = 0.41) and slight to substantial agreement for PI-RADS v1 (T2-WI, k = 0.32; DWI, k = 0.52; DCE MRI, k = 0.13)
Polanec et al [39]	2016	Retrospective	65	PI-RADS v1 and v2	T2-WI, DWI, and DCE	2	–	MRI-TBx	• Almost perfect inter-reader agreement for PI-RADS v2 and v1 (k = 0.71 and k = 0.81, respectively).
									• No difference in sensitivity between radiologists (p >0.05)
									• Higher specificity using PI-RADS v1 compared with PI-RADS v2 (radiologist 1: p = 0.0078, radiologist 2: p = 0.0313)
Renard-Penna et al [41]	2015	Prospective	118 (but only 50 for inter-reader agreement)	PI-RADS v1 and Likert	T2-WI, DWI, and DCE	2	Cancer core length ≥3 mm and/or Gleason score ≥4	MRI-TBx	• Good levels of agreement for the Likert scale (k = 0.80) and summed PI-RADS (k = 0.73) scoring systems
									• Good levels of agreement for PI-RADS T2-WI (k = 0.61) and DCE (k = 0.71), while only fair consistency (k = 0.53) for DWI
Rosenkrantz et al [42,66]	2013	Retrospective	70	PI-RADS v1 and Likert	T2-WI, DWI, and DCE	3	Any tumor >3 mm in maximal diameter	Radical prostatectomy	• For tumors with Gleason score ≥7:
									- Sensitivity was higher with PI-RADS than with Likert for radiologist 1 (88.6% vs 82.6%, p = 0.032)
									- Sensitivity was similar for radiologist 2 (78% vs 76%, p = 0.467) and radiologist 3 (77% vs 81%, p = 0.125).
									• In the TZ, accuracy was lower with PI-RADS than with Likert for radiologist 1 (70% vs 87%, p < 0.001), radiologist 2 (88% vs 93%, p = 0.002), and radiologist 3 (83% vs 91%, p < 0.001)
Schaudinn et al [43]	2019	Retrospective	40	PI-RADS v1 and v2	T2-WI, DWI, and DCE	2	–	Radical prostatectomy	• PI-RADS v2 showed a trend toward lower sensitivities for:
									- Radiologist 1: 72% (v1) vs 64% (v2; p = 0.426)
									- Radiologist 2: 78% (v1) vs 69% (v2; p = 0.402)
									• Trends were more pronounced in the TZ (p = 0.313) and for low-grade PCa (p = 0.691)
Tewes et al [44]	2016	Retrospective	54	PI-RADS v1 and v2	T2-WI, DWI, and DCE	2	–	MRI-TBx	• Agreement between PI-RADS scores was good (reader 1: k = 0.62; reader 2: k = 0.64)
									• Interobserver agreement was moderate with PI-RADS v2 (k = 0.56) and fair with v1 (k = 0.39)
Wang et al [36]	2018	Prospective	77	PI-RADS v1 and v2.0	T2-WI, DWI, and DCE	2	Gleason score ≥7	MRI-TBx	• PI-RADS v2 exhibited a higher AUC (0.888) than PI-RADS v1 (0.869)
									• PI-RADS v2 had higher sensitivity (75% vs 69%) but lower specificity (90% vs 96%) when than PI-RADS v1 for the assessment of PCa in the TZ

AUC = area under the curve; csPCa = clinically significant prostate cancer; DCE = dynamic contrast-enhanced imaging; DWI = diffusion-weighted imaging; mpMRI = multiparametric MRI; MRI = magnetic resonance imaging; MRI-TBx = MRI targeted biopsy; MRSI = magnetic resonance spectroscopic imaging; PCa = prostate cancer; PI-RADS = Prostate Imaging Reporting and Data System; PZ = peripheral zone; ROC = receiver operating characteristics; T2-WI = T2-weighted imaging; TZ = transition zone.

Table 4 –

List of studies assessing the comparison between bpMRI and mpMRI.

Author	Year	Study design	No. of patients	MRI reporting	No. of readers (experience)	Reference standard	Outcome	Key findings
Stanzione et al [82]	2016	Retrospective	82	bpMRI then mpMRI with 20–30 d interval	2 blinded (10 and 14 yr)	Mixed	Any PCa	• bpMRI and mpMRI showed similar performance for PCa detection with AUC 0.91 and 0.93, respectively (p > 0.05)
Thestrup et al [83]	2016	Retrospective	204	mpMRI then bpMRI with 2 mo interval	2 blinded (experienced)	Mixed	csPCa defined as Gleason score ≥3 + 4	• mpMRI: sensitivity 0.93–1.0, specificity 0.04–0.16, PPV 0.34–0.36, NPV 0.81–1.00
								• bpMRI: sensitivity 0.94–0.96, specificity 0.15, PPV 0.36, NPV 0.83–0.87
Lee et al [84]	2017	Retrospective	123	55 and 68 men received mpMRI and bpMRI, respectively	2 blinded (experienced)	Mixed	csPCa defined as Gleason score ≥3 + 4	• No differences in PCa and csPCa detection rate (41.8% vs 30.9%, p = 0.208 and 82.6% vs 76.2%, p = 0.598)
								• Similar detection of PCa among men who had suspicious lesions in the bpMRI and bpMRI groups (63.3% and 62.5%, respectively, p = 0.7)
Kuhl et al [85]	2017	Retrospective	542	bpMRI then mpMRI in the same session	4 blinded (2–9 yr)	Mixed	csPCa defined as Gleason score ≥3 + 4	• bpMRI and mpMRI csPCa detection was 25.6% (mpMRI detected 1 additional case of csPCa)
								• AUC was 89.1% and 87.2% for bpMRI and mpMRI, respectively
								• PPV was 73.8% vs 69.8% for bpMRI vs mpMRI
Nieuwenhove et al [86]	2019	Retrospective	90	1.5 T bpMRI then 3 T mpMRI after 1 mo	2 blinded (2 and 10 yr)	TRUS-Bx plus MRI-TBx	csPCa defined as Gleason score ≥3 + 4	• Compared with mpMRI, on the lesion-based analysis, bpMRI AUC 0.961 (p < 0.001), sensitivity 95%, specificity 97%, PPV 99%, NPV 89%
								• On the patient-based analysis, bpMRI AUC 0.975, sensitivity 98%, specificity 97%, PPV 98%, NPV 97%
Junker et al [87]	2019	Retrospective	236	bpMRI then mpMRI in the same session	1 blinded (experienced)	Mixed	PCa defined as Gleason score ≥4 + 3	• bpMRI did not show significant differences in diagnostic accuracy or tumor detection rates
								• 94% of PCa were scored identically
								• Omitting DCE changed PI-RADS scores in 9.75% of patients, increasing the number of PI-RADS 3 scores by 8.89% when compared with mpMRI
Sherrer et al [88]	2019	Retrospective	344	bpMRI then mpMRI in the same session	1 blinded (NR)	TRUS-Bx plus MRI-TBx	Any PCa	• The majority of the lesions (552/648, 85%) were visible at bpMRI
								• 15% of bpMRI-negative lesions were positive at DCE; of these 21% (3/14) harbored PCa
De Visschere et al [89]	2017	Retrospective	245	mpMRI scored according to PI-RADS v2 and with PI-RADS v2Alt (only T2-WI and DWI)	NR	Mixed	csPCa defined as Gleason score ≥3 + 4	• DCE was not needed for the determination of the overall assessment category in 80.8% (198/245) of patients
								• AUC was 0.79 for both reporting methods
Choi et al [90]	2019	Retrospective	113	bpMRI then mpMRI with 2 wk interval	2 aware of presence of PCa (7 and 13 yr)	Radical prostatectomy	csPCa defined as Gleason score ≥3 + 4 or volume >0.5 cc	• No significant differences in csPCa detection between bpMRI and mpMRI
								• Higher presence of csPCa in PI-RADS 3 lesions for bpMRI compared with mpMRI
Scialpi et al [91]	2017	Retrospective	41	bpMRI then mpMRI in the same session	2 aware of presence of PCa (experienced)	Radical prostatectomy	Any PCa	• For both bpMRI and mpMRI, sensitivity was similar, and was 100% in PZ and 97.6% and 94.7% in the entire prostate and TZ, respectively
								• bpMRI detected 181 lesions out of 131 detected at final pathology, resulting in 27.6% false-positive and 3.3% false-negative rates
								• Agreement of bpMRI and mpMRI was identical
Gatti et al [92]	2019	Retrospective	68	bpMRI then mpMRI with 1 mo interval	3 groups of 2 readers (1000, 300, and 100 cases read)	Mixed	Any PCa	• Two expert readers performed as well in bpMRI as in mpMRI (sensitivity = 0.91–0.96, AUC = 0.86–0.93; p ≥ 0.10)
								• Readers with 300 cases performed well in mpMRI, but significantly worse in bpMRI: sensitivity = 0.58 vs 0.91 (p < 0.0001) and AUC = 0.73 vs 0.86 (p = 0.01)
Di Campli et al [93]	2018	Retrospective	85	bpMRI then mpMRI in separate sessions	3 blinded (7, 3, and 1 yr)	Mixed	csPCa defined as Gleason score ≥3 + 4	• There was no significant difference regarding the detection of csPCa among the three readers between bpMRI and mpMRI
								• The AUC for bpMRI and mpMRI was 0.68–0.72 (high experience), 0.72–0.70 (medium experience), and 0.60–0.54 (low experience)

AUC = area under the curve; bpMRI = biparametric MRI; csPCa = clinically significant prostate cancer; DCE = dynamic contrast-enhanced imaging; DWI = diffusion-weighted imaging; mpMRI = multiparametric MRI; MRI = magnetic resonance imaging; MRI-TBx = MRI targeted biopsy; NPV = negative predictive value; NR = not reported; PCa = prostate cancer; PI-RADS = Prostate Imaging Reporting and Data System; PPV = positive predictive value; PZ = peripheral zone; TRUS-Bx = transrectal ultrasound biopsy; T2-WI = T2-weighted imaging; TZ = transition zone.

3.1. Risk of bias within studies

The overall risk of bias and applicability concern is given in Fig. 2. The overall methodological quality of the studies was moderate, with 17 studies having a low risk of bias and applicability concern across all domains assessed. Supplementary Fig. 1 shows the risk of bias and applicability concerns for each study.

3.2. Magnetic field

Multiparametric MRI at 3.0 T has an increased signal-to-noise ratio compared with 1.5 T scanners, resulting in higher spatial resolution of T2-weighted (T2-WI) and diffusion-weighted (DWI) imaging.

In total, seven studies assessed the value of the magnetic field relating to mpMRI staging accuracy (Supplementary Table 1) [14-20]. Two studies from 2004 [15,18] were among the first reporting a comparison of 1.5 and 3.0 T scanners even though with slightly different results. Bloch et al [15] reported higher image quality at 3.0 T than at 1.5 T (both with an ERC), while Sosna et al [18] reported a comparable quality for nonendorectal 3.0 T and endorectal 1.5 T mpMRI in 40 men receiving mpMRI and subsequent biopsy. However, these studies are not comparable given that the ERC was not used at 3.0 T [18]. Most importantly, only two studies addressed DWI [19,20] and thus only these studies could evaluate the detection difference between insignificant PCa and csPCa, as for this DWI is the most important sequence in the peripheral zone (PZ). These studies showed a similar Prostate Imaging Reporting and Data System (PI-RADS) assessment for 1.5 and 3 T.

Overall, the majority of the studies did not investigate the detection of csPCa but instead focused on the recognition of any PCa. In this respect, both magnetic field strengths performed equally [14,17,19], but the small sample size and high heterogeneity of these studies make an objective comparison difficult to conduct. In summary, no reliable information could be obtained regarding the detection of csPCa according to field strength regardless of the usage of reception coils.

3.3. Reception coil

A total of 11 studies comparing the use of mpMRI with an external pelvic phased-array coil with or without an ERC were included (Supplementary Table 2) [21-31]. Four studies did not use DWI.

The use of an ERC significantly improves the signal-to-noise ratio irrespective of the magnetic field strength, providing T2-WI with higher spatial resolution and potentially more accurate delineation of the structures `in the transition zone (TZ), which is the key factor in the assessment of csPCa in this zone. However, the addition of an ERC is associated with increased costs, increased artifacts [32], organ deformation, and discomfort for patients. Mirak et al [28] investigated the performance of 3.0 T mpMRI with and without an ERC to detect PCa using PI-RADS v2 guidelines, with whole-mount histopathology as the reference standard. Two subcohorts, with (n = 260) and without (n = 169) an ERC, were analyzed. They concluded that detection rates for any PCa, for the index lesion, and for csPCa were similar in both cohorts, but there was a significantly lower detection rate of anterior and TZ csPCa in the ERC subcohort due to a signal drop in the anterior gland when the ERC was used without an accompanying abdominal phased array coil.

Another study with a direct comparison of ERC versus non-ERC at 3.0 T [22] showed no differences in detecting PCa using PI-RADS v2 guidelines.

Costa et al [26] reported that using an ERC at 3.0 T provides superior sensitivity (78%) for PCa detection when compared with standard and augmented protocols (ie, those with twice as many signal averages; 43% and 60%, respectively) without an ERC.

Turkbey et al [23] compared the utility of T2-WI and DWI at 3.0 T with and without an ERC in detecting PCa in 20 men receiving mpMRI before radical prostatectomy. The authors demonstrated higher sensitivity when using an ERC. The sensitivity and positive predictive value (PPV) of an ERC versus non-ERC MRI were 76% versus 45% and 80% versus 64%, respectively.

Torricelli et al [30] reported that the image quality at 1.5 T with an ERC is superior to that at 3.0 T without an ERC in evaluating tumor conspicuity, capsular infiltration, and seminal vesicle involvement. No significant differences between the two techniques for the involvement of apex and neurovascular bundles and comparable performance for the diagnosis of capsular involvement were observed. The authors concluded that during preoperative PCa staging, 3.0 T mpMRI can provide diagnostic information comparable with that of 1.5 T mpMRI with an ERC.

In summary, the use of an ERC improves signal reception, which slightly improved sensitivity to visualize any PCa. Only one study addressed the value of the ERC with respect to the PI-RADS assessment of csPCa [28]. This study showed a minor disadvantage of the ERC in detecting anterior TZ PCa. Owing to the lack of significant evidence that the ERC improves csPCa assessment, associated increased costs, duration of examination, and patient discomfort, the PI-RADS v2.1 guidelines [33] recommend the ERC to be used only for older 1.5 T scanners with lower gradient strength [30]. Still, the lack of standardized protocols in the available studies makes robust comparisons hard to assess.

3.4. Assessment system

The development of a simple, structured, standardized assessment system was one of the most considerable challenges in prostate mpMRI. Assessment systems have been refined during the years in order to increase the inter-reader agreement, decrease the gap between differently skilled radiologists, and improve communication between radiologists and urologists. Ten studies comparing the use of different assessment systems were included (Table 1) [34-44]. The majority compared PI-RADS v1 [45] and v2 [33]. Differently from PI-RADS v1, PI-RADS v2 defined dominant sequences (T2-WI for the TZ and DWI for the PZ) and decreased the role of dynamic contrast-enhanced (DCE) MRI [46].

Most studies reported a similar diagnostic accuracy for both PI-RADS v1 and PI-RADS v2. However, three studies [36,37,39] showed higher sensitivity of PI-RADS v2 for TZ lesions, and another study by Krishna et al [40] showed that PI-RADS v1 detected approximately 10% more tumors than PI-RADS v2.

De Visschere et al [35] compared the performance of PI-RADS v2 and v1 in 245 biopsy-naïve men with an elevated level of prostate-specific antigen (PSA). They found that PI-RADS v1 and v2 yielded similar accuracy for the detection of csPCa. However, PI-RADS v2 had lower sensitivity than PI-RADS v1 when a score of 3 was used as a threshold for positive mpMRI. The authors suggested that the majority of discrepancies were caused by a suspicious lesion in the PZ on T2-WI but with normal DWI, scored positive with PI-RADS v1 but negative with PI-RADS v2. Indeed, sensitivity of PI-RADS v2 might be less when suspicious lesions on T2-WI but negative on DWI are present in the PZ. Nonetheless, the authors concluded that PI-RADS v2 is preferable because of the more structured and standardized, and simpler approach. Similar results were found by Krishna et al [40] who compared the two assessment systems for the detection of csPCa lesions in 47 men before radical prostatectomy. They found no difference in the overall detection of csPCa but observed higher sensitivity of PI-RADS v1 on T2-WI and DCE.

Feng et al [37] compared PI-RADS v1 and v2 in 401 consecutive biopsy-naïve men with a clinical suspicion of PCa at 3.0 T mpMRI. Both assessment systems had a good diagnostic performance for the detection of csPCa, but the diagnostic accuracy increased from 0.82 to 0.88 with the use of PI-RADS v2 compared with PI-RADS v1 when non–biopsy-naïve men were also included. Interestingly, PI-RADS v2 had a better performance in the TZ (0.92 vs 0.81).

Hoffmann et al [38] evaluated in 58 men whether PI-RADS v2 was more accurate in assessing anterior prostate csPCa than PI-RADS v1. PI-RADS v2 did not improve the accuracy for diagnosing anterior csPCa when compared with PI-RADS v1; however, PI-RADS v2 was more reproducible between radiologists.

Schaudinn et al [43] reported moderate interobserver reliability (k = 0.48) for PCa detection of two radiologists in 40 men before radical prostatectomy, and similar results have been reported by Tewes et al [44], with moderate interobserver agreement (k = 0.56) for PI-RADS v2 and fair agreement (k = 0.39) for PI-RADS v1. The authors concluded that PI-RADS v2 had equivalent diagnostic accuracy to PI-RADS v1 for PCa detection, but with shorter interpretation time for PI-RADS v2.

Two studies compared PI-RADS v1 and two different Likert assessment systems [41,42]. Both showed good inter-reader agreement, although one study reported higher accuracy of the Likert scale for TZ lesions than PI-RADS v1 [42].

In summary, it is recommended to use the most recent PI-RADS guidelines as the main assessment system given the simplified, objective, and standardized approach as well as its particular added value for less experienced radiologists.

3.5. Radiologist and urologist experience

The use of mpMRI to detect csPCa represents a multidisciplinary approach that includes skills both in acquisition and interpretation of mpMRI and in performing MRI-TBx.

Five studies assessed the impact of reader experience on mpMRI diagnostic performance (Table 2). Akin et al [47] and Garcia-Reyes et al [48] were the first to test the learning curve in prostate mpMRI reporting using whole-mount pathology as a reference standard. They showed that a dedicated training curriculum is useful to improve mpMRI interpretation. Rosenkrantz et al [49] assessed the variation of mpMRI diagnostic accuracy in detecting and localizing csPCa among six 2nd-year radiology residents reporting 124 prostate mpMRI scans (both negative and positive). Three out of six readers (50%) received feedback after each examination showing the solution of the preceding case. For readers both with and without feedback, there was an initial rapid improvement, which slowed down after 40 examinations. In the group receiving feedback, accuracy, sensitivity, and specificity improved from 58%, 59%, and 56% to 72–77%, 72–77%, and 74–82%, respectively. Interestingly, the presence of feedback did not significantly affect the accuracy as compared with the group without feedback, showing the effects of self-directed learning, even though readers with feedback felt more confident. Moreover, the feedback was more useful for TZ lesions, suggesting a higher challenge in detecting these tumors, in line with previous studies [47,48]. Pickersgill et al [50] conducted a retrospective review of 459 men receiving mpMRI according to PI-RADS (v1 implemented with v2 during the study period) and a subsequent MRI-TBx if necessary. They showed that the radiologist’s experience did not improve the accuracy in csPCa detection. The authors speculated that the use of PI-RADS might have attenuated the impact of the reader’s experience. However, this study had serious limitations, such as the implementation of PI-RADS v1 to v2 during the study period and an arbitrary definition of radiologist experience (ie, >500 mpMRI examinations). Following the widespread use of mpMRI and the need for dedicated training for radiologists, an online interactive case-based website for prostate mpMRI interpretation using PI-RADS v2 has been proposed [51]. This training course increased the sensitivity (from 58% to 73%, p = 0.003) and the NPV (from 69% to 78%, p = 0.049) of three 2nd-year radiology residents who evaluated separate sets of 60 MR scans before and after the course. Interestingly, there were no significant improvements in the accuracy of the PI-RADS assessment scores (from 46% to 53%, p = 0.149) [51].

Table 2 –

List of studies assessing the relationship between mpMRI performance and radiologist and urologist experience.

Author	Year	Study design	Trainee	Training	No. of patients	Learning curve definition	Definition of csPCa	Key findings
Akin et al [47]	2010	Prospective	11 radiology fellows	• Baseline: 15 MRI scans • 5 interactive lectures • 200 MRI scans over 10 wk	–	Accuracy in identifying peripheral and transitional tumors and ECE	–	• Peripheral PCa: AUC increased from 0.52 to 0.66 (p < 0.001) after the interactive course and then remained stable
								• Transitional PCa: AUC increased from 0.49 to 0.64 (p = 0.01) after the interactive courses and up to 0.68 (p = 0.001) at the end of training
								• ECE: AUC increased from 0.50 to 0.81 (p < 0.0001)
Garcia-Reyes et al [48]	2015	Retrospective	5 radiology fellows	31 MRI scans reinterpreted after 5 yr of dedicated education program	–	Accuracy and confidence in identifying overall and anterior PCa	Gleason score ≥3 + 4	• Index PCa detection: from 74.2% to 87.7% (p = 0.003)
								• Confidence: from 3.75 to 4.22 (scale 1–5)
								• Anterior PCa detection: from 54.3% to 94.3% (p = 0.001)
Rosenkrantz et al [49]	2017	Prospective	6 2nd-year radiology residents	124 MRI scans using PI-RADS v1 score. Readers divided between with and without feedback	–	Progressive accuracy, sensitivity, specificity, PPV, and NPV	Gleason score ≥3 + 4	• Initial rapid improvement in AUC that slowed after 40 examinations
								• AUC and sensitivity improved from 58% and 56% up to 77% and 82%, respectively, in the group with feedback.
								• Feedback did not affect the accuracy improvement significantly
								• Feedback resulted to be more useful for transitional zone PCa
Pickersgill et al [50]	2019	Retrospective	9 radiologists with 2–11 yr of experience	–	459 men receiving MRI for suspicion of PCa	Sensitivity, specificity, PPV, and NPV	Gleason score ≥3 + 4	• Radiologist experience (>500 scans) was associated with decreased sensitivity and NPV
Rosenkrantz et al [51]	2019	Prospective	3 2nd-year radiology residents	Two separate sets of 60 MRI scans reported before and after an online course using PI-RADS v2	–	Accuracy, sensitivity, specificity, PPV, and NPV	Gleason score ≥3 + 4	• Online course significantly improved sensitivity (from 57.8% to 73.3%, p = 0.003) and NPV (from 69.2% to 78.2%, p = 0.049)
								• No increase for specificity and PPV
								• Accuracy of single PI-RADS score assignment did not improve
Gaziev et al [55]	2016	Retrospective	3 urologists experienced in standard biopsy, naïve for MRI-TBx	–	340 men undergoing MRI-TBx	MRI-TBx PCa detection across the entire cohort divided into five subcohorts timely consecutive	Gleason score ≥3 + 4	• Increase in MRI-TBx PCa detection (from 27%to 63%) between the first and last 70 men
								• Improvement in MRI NPV (up to 89% in the most recent cohort)
Calio et al [56]	2017	Retrospective	–	–	1528 biopsy-naïve men undergoing MRI-TBx	MRI-TBx csPCa detection across the entire cohort divided into 3 subcohorts timely consecutive over 9 yr	Gleason score ≥3 + 4	• 13% increase in csPCa detection rate by MRI-TBx over the study period
								• csPCa detection rate at MRI-TBX increased even after multivariate adjustment
Meng et al [57]	2018	Retrospective	4 urologists experienced in standard biopsy	–	1595 men with previous negative biopsy undergoing MRI-TBx	MRI-TBx csPCa detection over the study period and at repeat MRI-TBx	Gleason score ≥3 + 4	• csPCa detection increased 26% with time in men with a PI-RADS 4/5 region of interest
								• On repeat MRI-TBx, 53% of those with PI-RADS 4–5 demonstrated csPCa discordance compared with previous biopsy
Mager et al [58]	2017	Retrospective	1 1st-year resident naïve for MRI-TBx	–	84 consecutive MRI-TBx	MRI-TBx quotient and biopsy time	–	• Significant improvement in both detection quotient and biopsy time after 42 procedures, flattening after 63 biopsies
Kasabwala et al [59]	2018	Retrospective	–	–	173 consecutive MRI-TBx	MRI-TBx accuracy defined as distance between planned and actual core trajectories stored on MRI-TBx fusion software	–	• Significant improvement in targeted biopsy accuracy occurred in up to 98 cases (p < 0.01)
Halstuch et al [60]	2019	Retrospective	Urologists naïve for MRI-TBx	–	779 men undergoing either transrectal (523) or transperineal (256) MRI-TBx	Urologist experience was coded at the total number of MRI-TBx before each procedure	Gleason score ≥3 + 4	• 104 transrectal MRI-TBx and 119 transperineal MRI-TBx are at least necessary to reach the best PCa detection
								• 109 transrectal MRI-TBx and 124 transperineal MRI-TBx are at least necessary to reach the minimum biopsy time
Stabile et al [61]	2018	Retrospective	3 urologists experienced in standard biopsy, naïve for MRI-TBx	–	244 men undergoing MRI-TBx	Urologist experience was coded at the total number of MRI-TBx before each procedure	Gleason score ≥3 + 4	• Urologist experience was associated with better csPCa detection after multivariate adjustment
								• Significant increase in csPCa detection during the first 60 procedures and a flattening after 80 procedures
								• Transperineal MRI-TBx was less affected by urologist experience achieving good csPCa detection since the first procedures
Westhoff et al [62]	2019	Retrospective	22 urologists (9 senior; 13 resident)	–	210 consecutive MRI-TBx	Urologist experience was coded at the total number of MRI-TBx before each procedure	Gleason score ≥3 + 4	• 8 MRI-TBx as the necessary threshold for experience
								• PCa detection rates for low and high experienced (according to the threshold) were 23% and 49%, respectively (p < 0.001)

AUC = area under the curve; csPCa = clinically significant prostate cancer; ECE = extracapsular extension; MRI = magnetic resonance imaging; MRI-TBx = MRI-targeted biopsy; NPV=negative predictive value; PCa = prostate cancer; PI-RADS = Prostate Imaging Reporting and Data System; PPV = positive predictive value.

The quality of MRI-TBx performance plays an equally important role in defining the final diagnostic accuracy of this technique. Similar to systematic ultrasound-guided biopsy [52,53], the experience of the biopsy operator influences significantly the outcome of an MRI-TBx, which can be visual/cognitive, MR-ultrasound fusion, or direct MRI-TBx [54], using either a transrectal or a transperineal approach.

A total of eight studies assessing the learning curve of MRI-TBx were included (Table 2). Gaziev et al [55] demonstrated a significant gradual increase in csPCa detection (from 27% to 63%) in 70 men receiving fusion MRI-TBx. Similarly, Calio et al [56] reported data from three consecutive cohorts of biopsy-naïve men receiving MRI-TBx over a study period of 9 yr. There was a 13% increase in csPCa detection by MRI-TBx from the early to the most recent cohort. Meng et al [57] reported a 26% increase in csPCa detection in 1500 men receiving repeat biopsy. Mager et al [58] attempted in demonstrating the presence of the learning curve effect proposing the MRI-TBx quotient, defined as the ratio between the number of positive targeted cores and the total number of targeted cores. The authors showed a significant learning process, in both detection-quotient and biopsy times; for a novice performer, sufficient learning occurred after 42 procedures, reaching a flattening after 63 biopsies. Kasabwala et al [59] calculated the distance between the planned and the actual core route in the prostatic tissue during a fusion MRI-TBx, and demonstrated a significant improvement in MRI-TBx accuracy after 98 cases. Halstuch et al [60] attempted to identify a minimum number of procedures to reach the best PCa detection using a mathematical algorithm. The authors demonstrated that at least 104 transrectal fusion MRI-TBx and 119 transperineal fusion MRI-TBx are necessary for men with visible PI-RADS 3 lesions, before reaching the plateau phase of csPCa detection. In this context, Stabile et al [61] demonstrated the presence of a learning curve affecting csPCa detection rate even when accounting for several confounders (such as PSA, prostate volume, and PI-RADS score) for both visual and fusion MRI-TBx. The authors showed a steep increase in csPCa detection during the first 60 procedures and a flattening after 80 procedures. Interestingly, it was suggested that the transperineal approach might be less affected by the learning curve effect; hence, it might be easier compared with the transrectal approach when considering MRI-TBx [61]. Finally, Westhoff et al [62] proposed at least eight procedures as experience threshold. However, this result should be interpreted cautiously, considering the limited number of MRI-TBx performed by each of the 22 urologists included in this study.

In summary, when assessing the performance of mpMRI in detecting csPCa, it is nowadays mandatory to indicate the experience of the interpreting radiologists and biopsy-performing urologists to support the reliability of the findings. Less experienced readers and biopsy operators must always be supervised by experienced readers and operators. Moreover, mpMRI performance should be validated internally before widespread adoption. According to Rosenkrantz et al [49], radiologists should have reported at least 100 expert-supervised prostate mpMRI examinations after a dedicated training course, and urologists should have performed between 60 and 100 MRI-TBx before they potentially reach an acceptable level of csPCa detection. Most importantly, additional quality assurance tests are needed. Indeed, mpMRI should be performed only in large-volume centers with validated reading assessment [11]. Further development of quality criteria, quality assessment, and training platforms/courses is needed.

3.6. Inter-reader variability

Although reader experience plays a substantial role in determining mpMRI accuracy, the reporting process is affected by almost inevitable variability among different radiologists, which varies across different studies and centers. Fifteen studies were included for this topic (Table 3).

Table 3 –

List of studies assessing the inter-reader variability of mpMRI.

Author	Year	Study design	Reporting system	No. of MRI/lesions reported	No. of readers	Reader’s experience	Definition of csPCa	Key findings
Quentin et al [63]	2012	Retrospective, single institute	Likert	108 predefined lesions	3 blinded	≥3 yr	–	• Agreement of T2-weighted images, DWI, and DCE k was 0.49, 0.97, and 0.77, respectively
								• PPV range 71–88% (k = 0.48)
								• AUC range 88–96% (k = 0.90)
Schimmöller et al [65]	2013	Retrospective, single institute	PI-RADS v1	164 premarked lesions in 67 MRI	3 blinded	4, 3, and 2 yr	–	• Agreement for all lesions was good to moderate (T2-W1, k = 0.55; DWI, k = 0.64; DCE MRI, k = 0.65)
								• For malignant lesions agreement was better than for benign lesions
Rosenkrantz et al [42,66]	2013	Retrospective, single institute	Likert and PI-RADS v1	55 MRI	3 blinded	2 experienced readers and one inexperienced	Gleason score ≥3 + 4	• Overall agreement between experienced readers was strong and for both the PI-RADS and the Likert scale
								• Overall agreement between experienced and inexperienced readers was moderate to poor
								• Agreement in the PZ was better for Likert than for PI-RADS
Rosenkrantz et al [71]	2016	Retrospective, multi-institute	PI-RADS v2	Two sessions of 40 and 80 MRI with intersession training in between	6 blinded	6 experienced	Gleason score ≥3 + 4	• No substantial difference was observed in the inter-reader agreement between sessions
								• Agreement for PI-RADS ≥4 was 0.593 in PZ and 0.509 in TZ
								• Agreement of PZ and TZ for PI-RADS ≥3 was 81.9% and 76.4%, respectively
								• Overall agreement for PI-RADS ≥3 and ≥4 was 79.2% and 77.8%, respectively
Mussi et al [67]	2019	Retrospective, single institute	PI-RADS v2	160 premarked lesions for 160 MRI single slides	8 blinded	Between 100 and >2000 MRI reported	Gleason score ≥3 + 4	• Coefficient of concordance according to categories was 0.71 considering both zones, 0.72 for PZ and 0.44 for TZ
								• Agreement for PI-RADS ≥3 was 0.48 in PZ and 0.57 in TZ
Glazer et al [68]	2017	Retrospective, single institute	PI-RADS v2	59 patients with a single lesion each	3 blinded	1, 4, and 11 yr	Gleason score ≥3 + 4	• Overall suspicion score agreement was moderate (k = 0.45)
								• There was moderate agreement among overall PI-RADS scores in the PZ (k = 0.46) and fair agreement in the TZ (k = 0.36)
Girometti et al [69]	2019	Retrospective	PI-RADS v2	48 preoperative MRI	3 unblinded; readers aware of the presence of PCa	2, 6, and 8 yr	Gleason score ≥3 + 4	• Moderate agreement in assigning PI-RADS categories to all PCa (k = 0.53) and csPCa (k = 0.47)
								• Assessing csPCa with PI-RADS ≥4 had higher agreement than PI-RADS ≥3 (k = 0.63 vs k = 0.57)
								• Agreement was higher between more experienced readers
Müller et al [70]	2018	Retrospective, two institutions	PI-RADS v1 and v2	126 men with positive MRI received second MRI	NR	NR	Gleason score ≥3 + 4	• Poor level of agreement between the two MRI scans and a statistically significant difference in PI-RADS scores
Smith et al [72]	2019	Retrospective, multi-institute	PI-RADS v2	102 MRI read twice with wash-out period in between	4 blinded	Moderate and high experience	Gleason score ≥3 + 4	• Overall intrareader reproducibility was moderate to substantial (k = 0.43–0.67)
								• Overall inter-reader reproducibility was poor to moderate (k = 0.24)
								• Readers with more experience showed greater inter-reader reproducibility
Hansen et al [73]	2017	Retrospective, multi-institute	Likert	158 MRI	28 blinded	NR for referring readers; second readers >1000 MRI reported	Gleason score ≥3 + 4	• Overall disagreement was 54% (86/158 MRI scans)
								• MRI scans were more often called negative by expert readers (41% vs 20%)
								• Second readings of MRI by expert readers significantly improved NPV and PPV
Sonn et al [78]	2019	Retrospective, single institute	PI-RADS v1 and v2	409 MRI	9 blinded	Median 6 yr (range 1–25)	Gleason score ≥3 + 4	• csPCa detection rate was 3–27% for PI-RADS 3 lesions, 23–65% for PI-RADS 4, and 40–80% for PI-RADS 5 across radiologists
								• 13–60% of men with a PI-RADS <3 lesion harbored csPCa
								• AUC varied from 0.69 to 0.81 across readers
Greer et al [74]	2019	Retrospective, multi-institute	PI-RADS v2	163 MRI	9 blinded	3 high level (<2000 last 2 yr) 3 moderate (500–2000 last 2 yr)	Gleason score ≥3 + 4	• Sensitivity for index lesions was 80.9%, comparable across reader experience • Highly experienced readers had 84.0% specificity versus 55.2% for all others (p < 0.001)
						3 low (<500 last 2 yr)		• Inter-reader agreement was excellent for detecting index lesions (k = 0.87)
								• Agreement on PI-RADS v2 category assignment of index lesions was moderate (k = 0.419)
Ke et al [75]	2018	Retrospective, single institute	PI-RADS v2	183 MRI	6 blinded	6 mo and 2, 3, 4, 5, or 17 yr	–	• Inter-reader agreement was weak to moderate (k = 0.506)
								• AUC varied between 0.88 and 0.95
Purysko et al [76]	2017	Retrospective, single institute	PI-RADS v2	170 MRI with premarked lesions	2 blinded	7 yr	Gleason score ≥3 + 4	• AUC for readers 1 and 2 were 0.871 and 0.882, respectively
								• AUCs were greater for PZ
								• Agreement was good overall (k = 0.63) and fair for TZ lesions (k = 0.53) for PI-RADS ≥3
								• Agreement was excellent for PI-RADS ≥4
Pickersgill et al [77]	2018	Retrospective, single institute	PI-RADS v2	32 MRI	4 blinded	0–548 MRI reported	Gleason score ≥3 + 4	• For PI-RADS ≥3, AUC for csPCa ranged between 47% and 75% (p < 0.001)
								• Team readings did not improve AUC

AUC = area under the curve; csPCa = clinically significant prostate cancer; DCE = dynamic contrast-enhanced imaging; DWI = diffusion-weighted imaging; MRI = magnetic resonance imaging; NPV = negative predictive value; NR = not reported; PCa = prostate cancer; PI-RADS = Prostate Imaging Reporting and Data System; PPV = positive predictive value; PZ = peripheral zone; T2-WI = T2-weighted imaging; TZ = transition zone.

Quentin et al [63] assessed the inter-reader agreement of mpMRI using a five-point (Likert) scale [64]. The authors showed high inter-reader reliability (PPV: 88–96%; k = 0.90) between three blinded radiologists scoring 108 lesions. After the introduction of PI-RADS v1 guidelines, Schimmöller et al [65] reported the agreement of three experienced radiologists scoring 164 premarked lesions. The overall agreement was good to moderate and higher for malignant than for benign lesions. Nonetheless, the clinical application of PI-RADS was still premature, and a diagnostic cutoff had not yet been proposed. Rosenkrantz et al [66] published a series of 55 patients undergoing prostate mpMRI in a single institution who were retrospectively reviewed by three radiologists (two moderately experienced and one inexperienced) using both PI-RADS v1 and Likert scores. For both assessment methods, the agreement was strong between the two experienced readers but poor when compared with the inexperienced reader. Interestingly, the Likert assessment scale had better inter-reader reproducibility than PI-RADS score in the TZ. This was probably due to the greater experience of the readers with their “own” Likert assessment. Since the widespread use of PI-RADS assessment score and the introduction of PI-RADS v2, few studies have assessed its reproducibility, with conflicting results. Mussi et al [67] reported moderate to good agreement between eight radiologists with different levels of experience in using PI-RADS v2. However, this study is hardly applicable to clinical practice since each reader evaluated only one MR slice containing a single specified lesion. Similarly, Glazer et al [68] conducted a retrospective study with three radiologists (with different levels of experience, ranging from 1 to 11 yr) who scored preselected lesions, with moderate agreement for PZ (k = 0.46) and fair agreement for TZ (k = 0.36). Moreover, the authors disclosed that PI-RADS v2 had been introduced recently in their clinical practice, potentially influencing the level of reproducibility. Girometti et al [69] supported the higher level of agreement among experienced radiologists in a monocentric study including three radiologists analyzing 48 MRI scans, with moderate agreement for PI-RADS cutoffs of both ≥3 (k = 0.57) and ≥4 (k = 0.63). Nonetheless, the readers were aware of the preoperative reason of mpMRI and hence of the presence of csPCa. Müller et al [70] reported a poor level of agreement in a cohort of 126 men receiving two consecutive MRI scans at two different institutions. Nonetheless, the design of this study was not devoid of many limitations and a significant bias. In fact, among 292 patients referred to the first institution, 126 men had mpMRI lesions considered challenging to be reliably accessed by systematic or cognitive biopsy. Since equipment for MRI-TBx with fusion technique was not available in the first institution, these men were referred to a second institution where they received a second mpMRI scan before undergoing an MRI-targeted fusion biopsy. In addition, all readers and authors had limited experience and training in reading prostate MRI according to PI-RADS. Moreover, no information regarding the used PI-RADS version was provided. For these reasons, these results should be considered very cautiously. Rosenkrantz et al [71] carried out a multicenter study with six experienced radiologists reporting at two different time points (40 and 80 MRI scans per session) and receiving a training session in between. The authors reported moderate reproducibility of PI-RADS v2, suggesting no benefit from the training session [71]. However, this study was limited by suboptimal image quality in a number of the included centers. Smith et al [72] provided results regarding intra- and inter-reader agreement with a multicenter study on four differently experienced readers. Overall, intrareader agreement was moderate to substantial (60–77% of agreement across different radiologists). Inter-reader agreement was poor to moderate and higher for more experienced radiologists. Hansen et al [73] reported the value of a second opinion by a subspecialized tertiary care center in reviewing mpMRI from seven different regional hospitals. Overall disagreement was 54% (86/158 MRI scans). Specifically, the second reading had significantly improved NPV (0.89 vs 0.72) and PPV (0.61 vs 0.28). Greer et al [74] reported excellent agreement on index lesion identification (k = 0.87) and moderate agreement on individual PI-RADS v2 category assignment (k = 0.419). Other two well-designed studies [75,76] reported similar results, with an area under the curve (AUC) for PCa ranging between 0.88 and 0.95 among six blinded readers [75]. Conversely, two recent studies showed high variability in PI-RADS v2 reporting [77,78]. In particular, Sonn et al [78], in a retrospective study of real-life mpMRI reporting by taking into account nine radiologists and 409 patients, while reporting a low variation in the number of lesions identified, demonstrated high variability in PI-RADS distribution and csPCa detection. The AUC for csPCa ranged between 0.61 and 0.81 [78]. Finally, van der Leest et al [79], in their prospective multicenter head-to-head comparison study, showed high inter-reader agreement of the participating expert radiologists. This was obtained after similar training to that described in the study of Rosenkrantz et al [49].

In summary, most of the well-designed dedicated studies reported moderate agreement when PI-RADS v2 is taken into consideration. Furthermore, the radiologist’s experience is crucial to increase mpMRI reproducibility, with the major concerns related to the variability in csPCa yield and high false-positive rates. Heterogeneity across the studies is still high. Most of the studies on this topic did not provide results about MRI acquisition, reader experience, or training. There is thus still a need for standardized mpMRI-assessment training protocols that should be available widely, in order to improve the general performance of mpMRI and provide more reliable data in this context. Only Rosenkrantz et al [49] and van der Leest et al [79] describe that radiologists should have reported at least 100 expert-supervised prostate mpMRI examinations after a dedicated training course. Further standardization of assessment systems, education, and certification will likely help in reducing the subjectivity and improving the reproducibility among less experienced readers as well.

3.7. Biparametric MRI versus mpMRI

Since the introduction of a standardized reporting system for mpMRI [45], the role of DCE MRI has been controversial. Indeed, PI-RADS v2 downgraded the role of DCE MRI to an additional sequence only for upgrading a PI-RADS 3 to 4 PZ lesion on DWI [33]. In the light of the increasing use of mpMRI in the assessment of csPCa and the need for more optimized and efficient protocols, the use of bpMRI based only on T2-WI and DWI has been proposed by multiple authors [80,81]. The benefits of omitting DCE MRI are related to reduced examination times, reduced costs, and avoiding the risk of adverse events related to the use of contrast agent. Results coming from prospective trials assessing the diagnostic accuracy of bpMRI are promising [80,81] regarding biopsy avoidance and for reducing the detection of insignificant cancers. Comparative studies of mpMRI and bpMRI are mostly retrospective, with significant differences in the methods and not negligible risk of bias (Table 4). To the best of our knowledge, Stanzione et al [82] showed the diagnostic accuracy of bpMRI compared with mpMRI. The authors reported a series of 82 men undergoing mpMRI for the suspicion of csPCa and then receiving systematic biopsy plus eventual MRI-TBx, with 35% of patients receiving radical prostatectomy. Two experienced radiologists blindly reported bpMRI first, followed by mpMRI (ie, with DCE), after an interval of 20–30 d to avoid any recall bias. The overall AUC values of bpMRI and mpMRI for csPCa detection were 0.91 and 0.93, respectively (p > 0.05). Thestrup et al [83] reported similar accuracies of bpMRI and mpMRI in detecting csPCa, although without providing any formal statistical comparisons. Lee et al [84] compared two cohorts undergoing mpMRI and bpMRI for a suspicion of PCa and then receiving visual MRI-TBx in addition to standard systematic biopsy. The authors reported a similar detection of csPCa among men who had suspicious lesions in the bpMRI and mpMRI groups (63% and 62%, respectively). Nonetheless, these results should be interpreted carefully since the two cohorts were not matched randomly.

Further studies reported promising results, although being affected by significant bias mainly concerning the MRI interpretation process (mpMRI and bpMRI read by the same radiologist during the same session) [85] and the reference standard (no biopsy result in men with negative MRI) [86]. The similar diagnostic performance of these two techniques was confirmed in other series [87-89] that attempted to better identify the eventual differences. Specifically, omitting DCE MRI was related to an increasing rate of PI-RADS 3 lesions, slightly better specificity, and worse sensitivity (although never significant) [87,89]. Furthermore, DCE MRI was not needed for the determination of the overall assessment category in 81% of patients [89].

Choi et al [90] compared the ability of bpMRI with that of mpMRI in detecting the index lesion using radical prostatectomy as the reference standard. Two independent radiologists (7 and 13 yr of experience) retrospectively reviewed prebiopsy MRI of 113 men using PI-RADS v2. No significant differences were found in csPCa diagnostic accuracy for bpMRI versus mpMRI for both readers using PI-RADS ≥3 as cutoff. Interestingly, both readers reported significantly higher sensitivity for bpMRI than for mpMRI [90]. Furthermore, inter-reader agreement on PI-RADS assessment score was moderate for both bpMRI (k = 0.540) and mpMRI (k = 0.478). However, both readers of this study were aware that all men underwent radical prostatectomy for csPCa, and this might have affected the overall results. In a similar study, Scialpi et al [91] evaluated the ability of bpMRI and mpMRI to detect single lesions in a cohort of 41 men receiving radical prostatectomy. For both bpMRI and mpMRI, the sensitivity was similar, which was 100% in PZ, and 98% and 95% in the entire prostate and TZ, respectively. Biparametric MRI detected 181/131 lesions at final pathology, resulting in 28% false positives and 3% false negatives rates [91]. Nonetheless, no information regarding readers’ background was provided and specific information regarding the experience is often scarce.

In this context, Gatti et al [92] compared bpMRI and mpMRI according to readers’ experience. The authors conducted a retrospective study on six blinded radiologists, divided into three groups according to the level of experience, reviewing bpMRI and mpMRI protocols of 68 men receiving a prostate biopsy and eventual radical prostatectomy. The authors used a modified version of PI-RADS v2 [89] for bpMRI reading and a cutoff of ≥4 for both protocols. Interestingly, the specificity was quite stable regardless of the protocol and readers’ experience. Significant differences were found for sensitivity and AUC in detecting PCa index lesions, mainly related to the rate of true positives. The effect of experience was more evident when considering bpMRI than mpMRI. Consequently, in the highly experienced group, the performance of bpMRI versus mpMRI was similar (AUC: 0.86 vs 0.93, p = 0.10; true positive: 82% vs 86%, p = 0.13). The accuracy of bpMRI became progressively less if compared with mpMRI with the decrease of experience (0.68 vs 0.77 in the less experienced group). Further, the authors attempted to provide a rough indication on the number of necessary cases to reach an AUC and sensitivity of ≥0.80: 150–200 for mpMRI and 700–800 for bpMRI [92]. Differently, Di Campli et al [93] showed no diagnostic differences between bpMRI and mpMRI, and no significant influence by readers’ experience.

In summary, available evidence from comparative studies suggests that bpMRI might be a potentially valid alternative to mpMRI, particularly for experienced readers, on the condition that DWI is of excellent quality. These findings have also been confirmed in a recent meta-analysis, demonstrating the noninferiority of bpMRI and showing overall nonsignificant higher sensitivity and lower specificity of mpMRI [94]. Moreover, a recent prospective, multi-reader, blinded direct comparison between bpMRI and mpMRI showed similar diagnostic performance in ruling out the presence of high-grade PCa [79].

That being said, the high methodological heterogeneity might have represented a great confounder, and it remains unclear how the performance of bpMRI will translate to less experienced centers and lower-quality images. Indeed, the assessment system used (ie, PI-RADS, dedicated bpMRI PI-RADS), choice of the cutoff, magnetic field, choice of the outcome (ie, PCa, csPCa), and reference standard are the factors varying the most across the studies. Ultimately, randomized prospective studies using noninferiority designs, in men with variable prevalence with clinically meaningful endpoints (biopsy avoidance, detection of csPCa, and clinically insignificant PCa), will be needed to decide on which patient groups can avoid contrast enhancement.

3.8. CAD and deep learning

The first study on PCa detection [95] was carried out by Chan et al [96] in 2003 (Supplementary Table 3). The CAD mainly consisted of quantitative analysis of medical imaging data (ie, mpMRI) with the aim to provide results potentially related to clinical conditions (ie, PCa). Different CAD systems have different strategies for imaging feature analysis, with different diagnostic accuracies [97]. The imaging analysis process includes multiple steps that can change across different CAD systems [97,98].

The CAD has recently been assessed in aiding radiologists during mpMRI interpretation and reporting. Interestingly, the evaluation of CAD as an assisting tool began at a similar time to the proposal of the first standardized assessment system (ie, PI-RADS v1) [45,99]. The inclusion of CAD systems in mpMRI interpretation process would potentially overcome some of the issues affecting diagnostic accuracy, such as reader experience, reproducibility, as well as enhancing the accuracy of mpMRI itself.

Hambrock et al [99] published the first study evaluating the effect of CAD for both less and more experienced radiologists on the differentiation of benign from malignant lesions at mpMRI. The authors demonstrated that, as the addition of CAD significantly improved the performance of less experienced radiologists in detecting PCa, when less experienced radiologists used CAD, they achieved similar performance to that of experienced radiologists. Furthermore, stand-alone CAD had similar diagnostic accuracy to experienced readers. However, this study did not replicate a real-life mpMRI diagnostic pathway, since the CAD system used was able to differentiate only preidentified regions between benign and malignant, but not PCa detection in a whole gland. In a similar study, Niaf et al [100] demonstrated that CAD increased the performance of both experienced and less experienced readers (AUC increase of 2% and 4%, respectively; p = 0.08) [100]. Litjens et al [101], in a standard clinical scenario, showed that the combination of CAD and PI-RADS assessment categorization achieved higher diagnostic accuracy than PI-RADS categorization alone to discriminate between both benign lesions versus PCa (0.88 vs 0.81, p = 0.013) and PCa versus csPCa (0.88 vs 0.78, p < 0.01) [101]. In a similar study based on prostatectomy specimens, Wang et al [102] demonstrated that the combination of CAD information with PI-RADS v2 increased the clinical net benefit for PCa identification as compared with PI-RADS categorization alone. Further studies demonstrated the clinical utility of quantitative analyses of ADC in improving the diagnostic performance when compared with the scoring system alone [103-105]. Giannini et al [106] were the first to replicate a real-life diagnostic pathway including the use of CAD. On a per-patient analysis, the use of CAD achieved higher sensitivity for csPCa (91% vs 81%, p = 0.046), while specificity was not affected. The per-lesion analysis showed a higher number of single lesions detected with CAD assistance. Interestingly, the average reading time with CAD was significantly lower (60s vs 220 s; p < 0.001) [106]. Greer et al [107] tested the effect of CAD in a multi-institutional group of differently experienced and blinded readers interpreting mpMRI in a cohort of men receiving radical prostatectomy for PCa. Sensitivity increased for all experience levels; however, specificity was dependent on reader experience. Improved sensitivity came from lesions scored as PI-RADS <3, as sensitivity for lesions scored as PI-RADS ≥3 was equal. The authors observed that CAD likely helped readers to overcome the “satisfaction of search” limitation, which stems from reduced detection of subsequent lesions after one lesion. However, the improved sensitivity of CAD was balanced by decreased specificity. It is noteworthy that the use of CAD improved agreement between all readers [107]. In an effort to assess the clinical value of a CAD system in a “real-world” scenario, Gaur et al [108] tested a multi-institutional population of 216 men receiving mpMRI. The cases used in this study were diverse in terms of institution-specific acquisition, MR manufacturer, and patient population, and nine readers of different experience levels were included. Overall, sensitivity for index lesions of mpMRI without and with CAD assistance was comparable. The highest benefit of CAD, as compared with mpMRI alone, was observed for moderately experienced readers detecting TZ csPCa. The authors concluded that CAD might help in reducing the rate of false positivity and increasing the sensitivity of moderately experienced readers [108]. Furthermore, recent studies confirmed the clinical utility of CAD use in combination with a Likert score and its generalizability to different field strengths [109,110].

Taken together, these findings suggest that CAD represents a promising tool that generally improves mpMRI efficiency in terms of both diagnostic accuracy and reporting time. The greatest benefit related to the use of CAD potentially lies in improving mpMRI sensitivity for multifocal csPCa, improving diagnostic accuracy in less experienced readers, improving inter-reader agreement, and reducing reporting time. Nonetheless, the majority of the available CAD systems are limited to site-specific predefined sequences, with most of them not considering DCE sequences and therefore site-specific CAD algorithm raising similar caveats to those raised when comparing bpMRI and mpMRI. To date, CAD is limited to research use only, as several aspects still need to be investigated and standardized.

3.9. Discussion

When assessing the role of mpMRI for PCa diagnosis, there are multiple additional aspects that need to be taken into account before drawing conclusions regarding clinical utility. In this study, the factors that might influence the diagnostic accuracy of MRI have been explored systematically. On the basis of our findings, some points need to be discussed.

First, it is clear that all the factors described in this study affect mpMRI accuracy, and are extremely related to each other in determining the ability of mpMRI in detecting csPCa. This is of crucial importance when comparing different studies. For instance, the use of an ERC helps increase mpMRI accuracy for anatomic image detail in the posterior part of the prostate, but without DWI, this does not help in better detecting csPCa. The benefit of the ERC is related to the magnetic field strength, which varies across the studies. In fact, according to our findings, a 3.0 T scan without an ERC and a 1.5 T mpMRI scan with an ERC reach similar staging accuracies, and thus anatomic detail. However, the ERC causes compression of the PZ, which may even result in missing small central posterior clinically significant cancers [25]. Considering that the addition of an ERC is also associated with increased costs, duration of examination, and artifacts, and is uncomfortable for patients, the use of an ERC is suggested only for older 1.5 T MR scanners. The recommended magnetic field strength for prostate mpMRI is 3.0 T [33]. However, an important issue that is discussed in literature is the gradient strengths of scanners: steep gradients are crucial for good DWI. Therefore, the steepness of gradient strength may be more important than the field strength.

Second, the experience of radiologists and urologists is pivotal and affects most of the factors related to mpMRI accuracy, such as inter-reader agreement and accuracy of different assessment systems. In this regard, different assessment systems seem to perform differently according to the experience of the reader. Indeed, standardization of the use of one assessment system (ie, PI-RADS v2) might actually help radiologists with lower experience in prostate mpMRI to reach an acceptable accuracy level easily. In addition, training and certification are important issues. For example, following an expert hands-on course followed by 100 supervised (double) reads may contribute to better interpretation quality and lower inter-reader variability [49,79]. Similarly, the use of bpMRI appears to be most effective for experienced readers, when good image quality is available, whereas those with low experience might still need DCE MRI as a backup in order to achieve acceptable accuracy. The noninferiority of bpMRI would represent a great step toward the widespread use of MRI, allowing a reduction in the costs (up to $300 spared per test [84,87]) and an increase in the availability of the test (15 min saved per patient [79,87]). With this aim in mind, well-designed prospective trials are necessary to provide reliable evidence and draw solid conclusions on bpMRI for PCa diagnosis [79]. Even when assessing the use of CAD, the highest benefit in terms of diagnostic accuracy is observed in less experienced radiologists.

To the best of our knowledge, this is the first systematic review exploring the different factors influencing mpMRI accuracy in detecting csPCa. However, this study is limited by the unsatisfactory evidence of most of the studies included, especially related to different MRI protocols, different outcomes, different mpMRI indications, different csPCa prevalence, variable readers’ experience, and pathological reference standards, with only 22% of studies having both low risk of bias and applicability concerns. Moreover, the lack of reliable pathological reference standards prevented the exploration of the variability of mpMRI performance in the presence of a negative report in the majority of studies. This heterogeneity prevents the possibility to conduct a proper meta-analysis on the effect of these factors. Lastly, further factors might be included as potential influencers of mpMRI accuracy (eg, apparent coefficient diffusion and b value, gradient strength of the MRI scanner, type of scanner, communication methods between radiologists and urologists, and type of study population). Nonetheless, studies specifically addressing these further factors eligible for inclusion in this review are scarce. These limitations are the cause that the findings of this review should be interpreted with caution and within the appropriate context.

4. Conclusions

Even though the role of mpMRI in predicting PCa has been demonstrated widely, several factors influence its diagnostic accuracy and are affecting each other, with the experience levels of the radiologists/biopsy operators being the key confounders. The high heterogeneity across the studies underlines the need for further studies that clarify how they impact the clinical utility of mpMRI for PCa diagnosis. Indeed, the factors assessed in this study concur with the high variability of mpMRI performance and the related clinical utility, consequently limiting the widespread use of mpMRI. In order to deliver the benefits of mpMRI in the diagnosis of csPCa, it is necessary to develop expertise for both radiologists and urologists, implement quality control, and adhere to the most recent PI-RADS assessment guidelines. Without the standardization of mpMRI execution and interpretation, and MRI-TBx technique, the mpMRI pathway may lead to suboptimal care outside large-volume and expert centers mainly due to the increased number of unnecessary biopsies, and false positive and false negative rates [11].