ReIGNITE RT Boost: A prospective, international study of radiation oncologists’ accuracy in contouring prostate tumors for focal radiotherapy boost on conventional MRI alone or with assistance of restriction spectrum imaging (RSI)

Abstract

Purpose:

In a phase III randomized trial, adding a radiation boost to tumor(s) visible on MRI improved prostate cancer (PCa) disease-free and metastasis-free survival without additional toxicity. Radiation oncologists’ ability to identify prostate tumors is critical to widely adopting intraprostatic tumor radiotherapy boost for patients. A diffusion MRI biomarker, called the Restriction Spectrum Imaging restriction score (RSIrs), has been shown to improve radiologists’ identification of clinically significant PCa. We hypothesized that (1) radiation oncologists would find accurately delineating PCa tumors on conventional MRI challenging and (2) using RSIrs maps would improve radiation oncologists’ accuracy for PCa tumor delineation.

Materials & Methods:

In this multi-institutional, international, prospective study, 44 radiation oncologists (participants) and 2 expert radiologists (experts) contoured prostate tumors on 39 total patient cases using conventional MRI with or without RSIrs maps. Participant volumes were compared to the consensus expert volumes. Contouring accuracy metrics included percent overlap with expert volume, Dice coefficient, conformal number, and maximum distance beyond expert volume.

Results:

1604 participant volumes were produced. 40 of 44 participants (91%) completely missed ≥1 expert-defined target lesion without RSIrs, compared to 13 of 44 (30%) with RSIrs maps. On conventional MRI alone, 134 of 762 contour attempts (18%) completely missed the target, compared to 18 of 842 (2%) with RSIrs maps. Use of RSIrs maps improved all contour accuracy metrics by approximately 50% or more. Mixed effects modeling confirmed that RSIrs maps were the main variable driving improvement in all metrics. System Usability Scores indicated RSIrs maps significantly improved the contouring experience (72 vs. 58, p<0.001).

Conclusion:

Radiation oncologists struggle with accurately delineating visible PCa tumors on conventional MRI. RSIrs maps improve radiation oncologists’ ability to target MRI-visible tumors for prostate tumor boost.

Background

Standard radiation therapy for aggressive prostate cancer (PCa) treats the entire prostate gland with an equally distributed radiation dose. In a recent phase III randomized controlled trial, addition of a simultaneous focal radiotherapy boost to PCa lesions visible on MRI (hereafter called tumors) increased disease-free survival from 86% to 93% at 7 years when compared to standard dose delivery.¹ Both local control and regional/distant metastasis-free survival were also improved.² Remarkably, these patient benefits did not come at the cost of additional short- or long-term toxicity.^2,3

Radiation oncologists’ ability to identify prostate tumors on MRI is critical to harnessing the benefits of radiotherapy boost for patients. In the FLAME trial, expert radiologists assisted with target identification, but even subspecialty-trained, experienced radiologists show substantial variability in lesion identification.^4,5 Tumor identification has not been part of most radiation oncology training and presents a major barrier to widely adopting intraprostatic tumor boost for prostate radiotherapy.

A quantitative diffusion MRI biomarker for PCa, called the Restriction Spectrum Imaging restriction score (RSIrs), has been shown to significantly improve diagnostic utility over conventional MRI^6,7 and to reliably correlate with voxel-level presence of PCa when compared to whole-mount histopathology.⁸ RSI involves computational post-processing of images acquired on standard clinical scanners to highlight the restricted intracellular diffusion characteristic of higher-grade PCa.⁹ RSIrs maps are generated from brief acquisitions on standard MRI scanners in as little as approximately two minutes of additional scan time.⁷ An implementation of RSI software is cleared by the FDA for prostate MRI and already commercially available on multiple scanner/vendor platforms. We hypothesized that (1) radiation oncologists would find it challenging to accurately delineate PCa tumors on conventional MRI, even when given a description of the lesion location, and (2) that using RSIrs maps would significantly improve radiation oncologists’ accuracy when contouring the MRI-visible lesion.

Methods

44 radiation oncologist participants with varied levels of experience were enrolled as participants in our study (Table 1). Participants still in training were required to have previously completed a PCa radiation oncology clinical rotation. All study recruitment materials, communications, and procedures were approved by the Institutional Review Board (IRB).

Table 1:

Participant Characteristics

Characteristic	Description	N (% Total)
Country	United States	28 (67%)
	India	5 (12%)
	United Kingdom	2 (5%)
	Canada	2 (5%)
	Slovenia	1 (2%)
	Romania	1 (2%)
	Italy	1 (2%)
	Belgium	1 (2%)
	Singapore	1 (2%)
Training Level	Still in Residency	24 (57%)
	5–10 years in practice	11 (26%)
	>10 years in practice	7 (17%)
Number of intact prostate cases treated in last 12 months	≦12	13 (31%)
	13–24	14 (33%)
	25–50	7 (17%)
	≧50	8 (19%)
Frequency of using MRI for planning intact prostate cases	Not routinely	4 (10%)
	<25% of the time	7 (17%)
	25–50% of the time	11 (26%)
	>50% of the time	20 (48%)
Number focal RT boost cases contoured in last 12 months	None	16 (38%)
	1 to 5	21 (50%)
	6 to 12	5 (12%)
	>12	0 (0%)
Had you heard about the FLAME trial results?	No	4 (10%)
	Yes	38 (90%)

Study participants were asked to contour tumors on 20 patient cases—half with conventional MRI alone and half with conventional MRI plus RSIrs—in each of two sessions at least 1 month apart. Patient cases with clinically localized intermediate-risk or high-risk PCa were selected from a prospectively maintained institutional database under IRB approval. Without informing the participants, 10 of the cases from the first session were interspersed within the second session but with RSIrs either removed or added from the case. Sequences provided from conventional MRI included T₂-weighted, ADC, and DWI (b=0 and b=2000 s/mm²). Multiparametric MRI was acquired per PI-RADS v2.1 (Supplementary Table A). RSIrs maps were displayed as overlays on anatomic T₂-weighted images (Figure 1). Participants contoured tumor volumes using the MIM Zero Footprint™ (ZFP) platform (MIM Software, Cleveland, OH). They were provided with clinical information for each case: patient age, PSA at time of MRI, Gleason score, and number and location of positive cores. Critically, participants were also given the radiologist’s textual description of lesion location and size.

Figure 1: RSIrs maps improve the accuracy of participant target volumes.

Example target volumes from two patient cases (Case 1: A/B and Case 2: C/D) where participants were provided Conventional MRI alone (T2-weighted, DWI, ADC) (left) images or conventional MRI with RSIrs map (orange heat map) (right). Expert volumes are shown in red in all panes. Select participant volumes are highlighted in shades of blue in (A and C). All participant volumes for each case are displayed in (B and D) as a rainbow heatmap, where the color represents the percentage of participants who included that voxel in their target volume.

In the randomized controlled trial (FLAME) that established a benefit to focal radiotherapy boost, the boost target was defined by a radiation oncologist and expert radiologist and consisted of “the macroscopic tumor visible on [multiparametric] MRI” without additional margin.¹⁰ As MRI is known to underestimate the full tumor extent on whole-mount histopathology specimens,¹¹ it is important to note that the randomized trial evidence supports targeting tumor visible on multiparametric MRI. Therefore, in our study, expert volumes were created by consensus interpretation by a radiation oncologist (with 3 years of experience) and two board-certified, sub-specialist GU radiologists (with 5 years and 7 years of experience, respectively) using conventional multiparametric MRI (including dynamic contrast enhanced images) and the clinical/pathologic information for each case. The patients and prostate lesions included in this study had characteristics very similar to those in the FLAME trial (Table 2). Because the clinical standard, per the FLAME trial, is to use tumor visible on multiparametric MRI, the expert-defined targets were delineated on conventional MRI, blinded to RSIrs maps. Volumes were exported as binary masks and analyzed in MATLAB R2021b (Mathworks, Natick, MA). We then verified that each expert-defined lesion was confirmed on histopathology (Table 2). As in clinical treatment of intact PCa, this was defined primarily on biopsy. An expert-defined lesion was considered confirmed on histopathology if PCa was found on (a) targeted biopsy of the lesion and/or (b) on systematic biopsy of the same sector (e.g., right apex) or adjacent ipsilateral sector (e.g., right mid-gland; to allow for known variation in definition of sectors on MRI vs. biopsy). For the 13 cases who underwent radical prostatectomy, we also reviewed the final pathology report. In all, the locations of 33 of 34 lesions were confirmed on histopathology. One PI-RADS 3 lesion lacked histopathologic confirmation and was excluded from further analysis.

Table 2:

Characteristics of patients (n=29) and histopathology-confirmed prostate cancer lesions (n=33) included in this study.

Median age, in years (IQR)		69 (63 – 74)
Median PSA at time of MRI, in ng/mL (IQR)		9.0 (6.4 – 22.7)
Clinical stage	T1c	15
	T2a	2
	T2b	7
	T2c	4
	Unknown	1
Biopsy status per patient	Systematic	12
	Targeted	1
	Systematic & targeted	16
Mean lesions per patient (IQR)		1.14 (1 – 3)
Mean expert contour size, mL (IQR)		1.13 (0.55 – 3.25)
Lesions by zone	PZ only	25
	PZ, extending into TZ	4
	TZ	4
PI-RADS v2.1 score per lesion	3	6
	4	12
	5	15
Gleason score per lesion	3+3	2
	3+4	9
	3+5	1
	4+3	9
	4+4	6
	4+5	5
	5+4	1
Lesions confirmed on	Targeted biopsy	16
	Systematic biopsy	17
	Prostatectomy	13*
Pathological stage	T2a	3
	T3a	8
	T3b	2

PZ: peripheral zone; TZ: transition zone

^*13 patients (15 lesions in all) underwent prostatectomy, with final histopathology confirming the highest Gleason score for all 13 patients. In 6 cases (8 lesions), the pathology report also confirmed the location of the cancer (e.g., right base of prostate); for 2 of these cases, the pathologist also created a manual map of lesion locations (Supplementary material).

Results

Participants made 1604 attempts to delineate targets, creating one participant volume per attempt; these participant volumes were compared to the consensus expert volumes. A complete miss was defined as zero overlap with the expert-defined target. Participants completely missed the expert-defined target in 13.6% of attempts (median; IQR: 9.1% – 23.6%) on conventional MRI alone, compared to 0.0% (0.0% – 4.3%) with RSIrs. 40 of 44 participants (91%) completely missed ≥1 target without RSIrs, compared to 13 of 44 participants (30%) with RSIrs maps (Table 3). On conventional MRI alone, 134 of 762 total contour attempts (18%) completely missed the target, compared to 18 of 842 (2%) with RSIrs maps. There were no clear patterns in the patient cases that had complete misses with regards to lesion grade, PIRADs, size or zone (Supplementary Table B). The two most inaccurate participants completely missed 45% of their targets without RSIrs vs. 5–10% with RSIrs (Figure 2). We measured four metrics of contouring accuracy: percent overlap with expert volume, Dice coefficient, conformal number, and maximum distance beyond expert volume. RSIrs maps improved each accuracy metric by approximately 50% or more (Table 3). The size of participants’ contoured volumes did not significantly differ from that of expert volumes (regardless of whether using RSIrs; p>0.50), but when using conventional MRI alone, the participants’ contoured volumes only covered median 41% (IQR: 13% – 64%) of the expert-defined target. When using RSIrs, participants’ contoured volumes covered median 78% (IQR: 59% – 90%) of the expert-defined target. Improvement in accuracy and reduced variability are illustrated for representative cases in Figure 1.

Table 3:

Summary of accuracy statistics comparing participant volumes to expert volume

	Graphical Equation	Without RSIrs	With RSIrs
# of Complete Misses across all participants		134/762 (17.6%)	18/842 (2.1%)
# of participants with at least 1 Complete Miss		40 (91%)	13 (30%)
% of each participant’s attempts resulting in a Complete Miss, median (IQR)		13.6% (9.1 – 23.6%)	0.0% (0.0 – 4.3%)
Conformal Number, median (IQR)		0.26 (0.06 – 0.43)	0.45 (0.35 – 0.55)
Dice Coefficient, median (IQR)		0.48 (0.18 – 0.64)	0.66 (0.55 – 0.73)
Maximum Distance to Expert, median (IQR)		13.7mm (8.1 – 23.4)	9.17mm (6.3 – 14.9)
% Overlap with Expert, median (IQR)		40.8% (12.6 – 64.0%)	77.6% (58.9 – 89.5%)

Summary of metrics evaluating participant volumes relative to the expert volume. 44 participants generated 762 volumes on conventional MRI without RSIrs and 842 with RSIrs available for analysis (there were two more cases with RSIrs than without RSIrs).

Figure 2: RSIrs maps reduce the frequency of complete misses by participants.

Complete misses as a percentage of the total attempts per participant on Conventional MRI alone (“Without RSIrs”, red) and with RSIrs (grey). No participant had more complete misses when using RSIrs. The red lines show the magnitude of improvement (reduction in complete misses) for each participant.

Mixed-effects models demonstrated that use of RSIrs maps was the main driver of improvement in accuracy metrics, not participants’ experience with prostate tumor boost (Supplementary Tables 1 C-F). Use of RSIrs maps was independently associated with improvement in all accuracy metrics (p<10⁻⁹ for each metric) and was the only independent predictor of accuracy for two metrics (Dice coefficient, and Conformal Number). Routine use of prostate MRI in clinical practice (participants reporting use of MRI in 25%−50% or >50% of their prostate cases) was also associated with improved accuracy for the other two metrics (Percent Overlap with Expert Volume and with Maximum Distance Beyond Expert Volume, p=0.04 and p=0.005, respectively), independent of the improvements in these metrics with use of RSIrs.

After contouring, 42 of 44 participants completed a System Usability Scale (SUS) questionnaire adapted from the US Department of Health and Human Services, with responses on a scale of 0 to 100, with >70 considered “acceptable” and $\le$50 considered “poor” usability.¹² Mean (standard deviation) SUS score improved from 57 (17) to 72 (13) with use of RSIrs maps (p<0.001). On conventional MRI, 36% of participants reported poor usability, compared to 5% with addition of RSIrs maps. On conventional MRI, 26% reported acceptable usability, compared to 48% with RSIrs maps (Figure 3). These usability results and the high frequency of complete misses underscore the inherently challenging nature of tumor contouring and highlight the need for new methods and training before tumor radiotherapy boost can be confidently and widely implemented.

Figure 3: Summary of System Usability Survey results.

A) Distribution of normalized SUS scores for all participants where figure legend and pink bars highlight Mean (SD). p-value is the result of a two-sided paired t-test. (B and C) 2×2 analysis of number of participants rating the contouring experience B) Acceptable (SUS > 70) and C) Poor (SUS ≤ 50). Percentages are percent of responses rating the contouring experience with RSIrs or without RSIrs and is based on 42 questionnaire responses.

Discussion

Our results demonstrate that targeting prostate tumors with focal radiotherapy boost can be very difficult for radiation oncologists. RSIrs, an advanced imaging biomarker, makes it easier for oncologists to accurately identify the visible tumor and represents one way to potentially accelerate wide adoption of focal radiotherapy boost for prostate cancer treatment.

This study used expert radiologist consensus volumes as the reference, with lesion locations verified by histopathology. This approach nicely reflects the procedures used in the FLAME randomized trial to establish a benefit to focal radiotherapy boost. Thus, RSIrs improved radiation oncologists’ ability to accurately delineate tumors visible on multiparametric MRI. This is consistent with previous studies that demonstrated improved voxel-wise PCa detection with RSIrs.⁶ A remaining question for the field is whether there is a meaningful benefit to boosting cancer not visible on multiparametric MRI, whether near the visible lesion^11,13 or elsewhere in the gland.¹⁴ PCa not visible on MRI is genetically distinct from visible tumor and tends to be less aggressive.^15,16 Efforts to improve voxel-level accuracy of RSIrs or other advanced imaging approaches, including PSMA-PET, might yield improved ability to delineate the full tumor extent, as defined on whole-mount histopathology. Such methods would facilitate clinical investigation of the utility of boosting disease not readily visible on conventional multiparametric MRI.

Limitations of this study include the use of images from a single institution. However, the radiation oncologist participants were from various institutions in nine different nations. Ongoing studies are evaluating the quantitative reproducibility of RSIrs maps across scanner platforms and institutions. Also, RSIrs is not yet available at many institutions and is most accurately calculated when dedicated images are acquired when the patient is scanned. Alternative or complementary approaches to improve radiation oncologists’ accuracy should be explored, including additional training, other artificial intelligence techniques for lesion detection, or calculation an approximation of RSIrs from conventional MRI.

Conclusions

We report that identifying intraprostatic tumors on MRI for focal radiotherapy boost is challenging for radiation oncologists, with 18% of attempts in our study (using conventional MRI) resulting in a complete miss of the expert-defined target. Radiation oncologists’ attempts to delineate visible PCa tumors were significantly more accurate and less variable when using RSIrs maps. Use of RSIrs maps has the potential to increase the feasibility of delivering the oncologic benefits of focal tumor boost to all eligible patients.

Data Availability Statement:

De-identified data are available to bona fide researchers for non-commercial use upon request.