Prospective Evaluation of Prostate and Organs at Risk Segmentation Software for MRI-based Prostate Radiation Therapy

Abstract

The segmentation of the prostate and surrounding organs at risk (OARs) is a necessary workflow step for performing dose-volume histogram analyses of prostate radiation therapy procedures. Low-dose-rate prostate brachytherapy (LDRPBT) is a curative prostate radiation therapy treatment that delivers a single fraction of radiation over a period of days. Prior studies have demonstrated the feasibility of fully convolutional networks to segment the prostate and surrounding OARs for LDRPBT dose-volume histogram analyses. However, performance evaluations have been limited to measures of global similarity between algorithm predictions and a reference. To date, the clinical use of automatic segmentation algorithms for LDRPBT has not been evaluated, to the authors’ knowledge. The purpose of this work was to assess the performance of fully convolutional networks for prostate and OAR delineation on a prospectively identified cohort of patients who underwent LDRPBT by using clinically relevant metrics. Thirty patients underwent LDRPBT and were imaged with fully balanced steady-state free precession MRI after implantation. Custom automatic segmentation software was used to segment the prostate and four OARs. Dose-volume histogram analyses were performed by using both the original automatically generated contours and the physician-refined contours. Dosimetry parameters of the prostate, external urinary sphincter, and rectum were compared without and with the physician refinements. This study observed that physician refinements to the automatic contours did not significantly affect dosimetry parameters.

Keywords: MRI, Neural Networks, Radiation Therapy, Radiation Therapy/Oncology, Genital/Reproductive, Prostate, Segmentation, Dosimetry

Supplemental material is available for this article.

© RSNA, 2022

Summary

Custom automatic segmentation software produced clinical quality segmentation masks of the prostate and surrounding organs at risk for MRI-based prostate radiation therapy; physician refinements did not have a statistically significant impact on radiation dosimetry.

Key Points

• ■ The automatic segmentation software produced clinical quality segmentation masks of the prostate and four surrounding organs at risk for MRI-based prostate radiation therapy.

• ■ There was not a statistically significant difference in prostate, rectum, or external urinary sphincter dosimetry parameters quantified from the automatic segmentation masks and from the physician-refined segmentation masks.

• ■ The algorithm produced quality segmentation masks in the presence of implanted biodegradable hydrogel rectal spacers, which were not well represented (<1%) in the development MRI scans.

Introduction

Low-dose-rate prostate brachytherapy (LDRPBT) is a curative radiation therapy option for prostate cancer (1,2). Historically, CT has been a primary imaging modality for LDRPBT because of its ease of use, its accessibility, and the information it provides about photon attenuation throughout the pelvis. However, there is a growing interest in the use of MRI for LDRPBT because of its exquisite soft-tissue contrast and its ability to provide diagnostic information in addition to anatomic imaging (3–5).

Essential workflow steps in MRI-based treatment planning and postimplant quality assessment of LDRPBT are the delineation of the prostate and surrounding organs at risk (OARs) on prostate MR images (6,7). Custom automatic segmentation software has been developed for contouring the prostate and OARs on prostate MR images, has been fully integrated into a commercial treatment planning system, and has demonstrated excellent agreement with physician-delineated contours in retrospective analyses (8).

Although comparisons of predicted segmentations using global similarity metrics have been promising, evaluations that incorporate dosimetric quantifiers of radiation delivery are more clinically relevant. This study evaluates the dosimetry parameters derived from custom automatic prostate MRI segmentation software and compares how the dosimetry changes after physician refinements have been made to the contours.

Materials and Methods

This study was conducted under a Health Insurance Portability and Accountability Act–compliant and institutional review board–approved protocol at the University of Texas MD Anderson Cancer Center. Patients were identified prospectively, and their MR images were analyzed retrospectively.

MRI Acquisitions

Thirty patients with confirmed prostate cancer underwent MRI-based LDRPBT consecutively between February and June 2021. They underwent implantation by using a stranded technique with positive-contrast MRI seed markers (9). The patients were anesthetized and treated under image guidance by using transrectal US and intraoperative cognitive fusion with preimplant T2-weighted MRI. They were imaged with either a turbo spin-echo pulse sequence (Cube, GE; SPACE [sampling perfection with application optimized contrasts using different flip angle evolution], Siemens) or a fully balanced steady-state free precession pulse sequence for postimplant quality assessment (FIESTA-C [constructive fast imaging employing steady-state acquisition], GE; CISS [constructive interference in steady-state], Siemens) (10,11). Postimplant MRI examinations were performed either the day of or the day after treatment. Typical imaging parameters for the postimplant MRI are shown in the Table.

Acquisition Parameters for 30 Patients Evaluated in This Study

Automatic Segmentation Software

A deep learning (DL) algorithm was previously developed to segment the prostate, external urinary sphincter (EUS), seminal vesicles, rectum, and bladder across MRI scans acquired with different image contrasts (8). The algorithm is based on fully convolutional networks (12). The algorithm was implemented into a commercial treatment planning system (MIM Symphony) via a software interface. The predictions of the DL model were displayed to the user in a manner that allowed them to select the operating point of the algorithm (Fig 1A). This is different than most other Dl-based segmentation applications, in which developers typically apply an argmax function to the fully convolutional network predictions. An imaging physics resident (J.W.S.) selected the operating point of the algorithm for each organ of each patient. A board-certified radiation oncologist (S.J.F., H.M., or C.T.) reviewed the automatic contours and made refinements to them wherever necessary. Predictions from a fully automatic implementation of the software (by using a fixed operating point for each organ) were also compared with the final physician-approved contours.

Figure 1:

Automatically generated predictions of the anatomic contours of the five organs for nine different operating points (10%–90% confidence in 10% increments). The patient shown has an implanted biodegradable hydrogel rectal spacer (white arrows). (A) Raw predictions of the anatomic contours of the five organs. The yellow arrows show the predicted boundaries of lower confidence for the rectum (≤50% confidence). (B) The automatically generated contours of the five organs at the operating points selected for analysis. The contours were recolored to match those of the corresponding organs in Figure 2. (C) The final postimplant quality assessment radiation therapy plan. Crosshairs and green cylinders indicate seed locations in adjacent sections. EUS = external urinary sphincter, SV = seminal vesicles.

Radiation Dosimetry

The implanted radioactive seeds were identified in the treatment planning system by a certified medical dosimetrist for each patient. The treatment planning system performed dose-volume histogram analyses of the patients’ prostate, rectum, and EUS by using the radioactive seed locations, specifications of the physical properties of the seeds, and the organ contours. Dose-volume histogram analyses were performed for both the original automatic contours and the physician-refined contours for each patient, and dosimetry parameters were recorded. The dosimetry parameters included prostate D90 (radiation dose delivered to 90% of the prostate), prostate V100 (volume of the prostate receiving 100% of the prescribed dose), prostate V150 (volume of the prostate receiving 150% of the prescribed dose), EUS V200 (volume of the EUS receiving 200% of the prescribed dose), and rectum V100 (volume of the rectum receiving 100% of the prescribed dose).These five dosimetry parameters are the primary metrics of interest in this study because of their clinical relevance and use.

Contour Similarity

The original and physician-refined contours were compared for similarity. Similarity was computed from the confusion matrices by computing the Matthews correlation coefficient (MCC) (13) for all organs. Organ volumes were also computed and compared between original and physician-refined contours, and volume differences were computed.

Statistical Analyses

Distributions of dosimetry parameters and MCCs were compared by using two-tailed nonparametric match-paired Wilcoxon signed rank tests.

Results

The mean ± standard deviation age, weight, and height of the 30 patients were 65.70 years ± 7.76, 99.71 kg ± 23.37, and 1.78 m ± 0.06, respectively.

Median MCC (Fig 2A) between the automatic contours (defined at the patient-specific operating points) and physician-refined contours of the prostate, EUS, seminal vesicles, rectum, and bladder were 0.972 (95% CI: 0.957, 0.981), 0.840 (95% CI: 0.707, 0.965), 0.981 (95% CI: 0.961, 0.997), 0.999 (95% CI: 0.996, 0.999), and 0.998 (95% CI: 0.997, 1.000), respectively. Median volume differences (Fig 2C) were 0.072 mL (95% CI: −0.219, 0.960 mL), −0.089 mL (95% CI: −0.326, 0.000 mL), 0.226 mL (95% CI: 0.001, 0.576 mL), 0.000 mL (95% CI: −0.036, 0.003 mL), and 0.012 mL (95% CI: −0.005, 0.351 mL), respectively.

Figure 2:

(A, B) Matthews correlation coefficient (MCC) and (C, D) volume difference computed between the automatic contours and physician-refined contours. Plots A and C correspond to the patient-specific operating point, and plots B and D correspond to the fixed operating point. EUS = external urinary sphincter, SV = seminal vesicles.

None of the dosimetry parameters (Fig 3A, 3B) were significantly affected by the physician refinements. Median dosimetry parameters computed using the original contours versus median dosimetry parameters using the physician-refined contours were as follows: prostate D90, 109.600% versus 110.200%, P = .815; prostate V100, 94.100% versus 94.020%, P = .766; prostate V150, 66.080% versus 64.930%, P = .786; rectum V100, 0.000 versus 0.000 mL, P = .261; and EUS V200, 0.028 versus 0.018 mL, P = .079.

Figure 3:

Comparison of dosimetry parameters computed from the automatic contours and physician-refined contours. (A) Dosimetry parameters of the prostate, computed as a percentage of the prescribed dose (D90) or a percentage of the contoured volume receiving a percentage of the prescribed dose (V100 and V150). (B) Dosimetry parameters of the rectum and EUS, reported as a volume of the contoured organ volume. D90 = dose delivered to 90% of the prostate, EUS = external urinary sphincter, V100 = volume receiving 100% of the prescribed dose, V150 = volume receiving 150% of the prescribed dose, V200 = volume receiving 200% of the prescribed dose.

Results for the fully automatic implementation were similar to those in which the operating point was selected on a per-patient basis (Figs 2–3 and Table E1 [supplement]). The median dosimetry parameters at patient-specific operating point versus median dosimetry parameters at fixed operating point were as follows: prostate D90, 109.600% versus 110.567%, P = .984; prostate V100, 94.100% versus 93.795%, P = .919; prostate V150, 66.080% versus 65.838%, P = .318; rectum V100, 0.000 versus 0.000 mL, P = .700; and EUS V200, 0.028 versus 0.024 mL, P = .615.

Discussion

Custom Dl solutions for automatic segmentation applications in medical imaging are routinely proposed. However, only a small fraction has made it into clinical practice. This study presents our initial experiences using custom and fully integrated Dl-based automatic prostate MRI segmentation software for prostate radiation therapy. We observed high levels of similarity between the automatic contours and the physician refinements to the automatic contours. Moreover, we did not observe a statistically significant difference in dose-volume histogram parameters computed from the automatic contours and physician-refined contours. These results suggest that the automatic segmentation software produced clinical quality segmentation masks of the prostate and surrounding OARs for prostate radiation therapy.

Interobserver variability and intraobserver variability are two sources of error introduced in contours performed by humans. Interobserver variability has been shown to influence the quality assessment of LDRPBT and can affect the assessment of implant adequacy (14,15). Intraobserver variability has also been shown to introduce uncertainties in the postimplant quality assessment of LDRPBT (16,17). Although dl-based automatic contouring algorithms do have interalgorithm variability, the predictions for a given algorithm are deterministic, eliminating the intracomponent of variability. The removal of the intracomponent of variability is an additional benefit of using automatic contouring algorithms and may help improve the consistency of contouring for MRI-based LDRPBT.

In addition to circumventing the intracomponent variability, DL offers two additional primary benefits to MRI-based prostate radiation therapy. The first is that DL can help to improve workflow efficiency. Executing the automatic segmentation algorithm typically requires approximately 1 minute or less for a given MRI; this is in comparison with the approximately 20 minutes or more required by a radiation oncologist. The second is that DL can help to improve the quality of prostate and OAR boundaries. In a separate retrospective study, we have found that the DL algorithm used in this study produces higher similarity metrics against a reference standard compared with the interobserver variability of radiation oncologists (data not shown).

The largest refinements to the prostate occurred primarily at the base and apex of the prostate. Recent studies evaluating automatic segmentation software for prostate-only radiation reported similar findings (18). Additionally, a recent observer study also showed that these regions had the largest interobserver variability among human observers delineating the prostate at MRI (19). Interobserver variability studies for CT-based LDRPBT also reported large discrepancies in delineating the prostate base and apex (20). The delineation of the prostate base and apex remains the most challenging areas of prostate delineation.

The EUS is a relatively new structure that has been investigated for MRI-based LDRPBT. Dose constraints for the EUS have been established for LDRPBT (21), and EUS dosimetry is assessed after implantation to estimate potential radiation-induced adverse effects. The MCC measurements indicated that the largest variation between the original automatic contours and physician-refined contours occurred for the EUS. This result is consistent with a recent observer study, which showed that, among five organs, human observers had the highest interobserver variability for delineating the EUS (19). The annotations used to train the dl-based automatic segmentation software were based on human annotations, which contain large variability in EUS contours. This is likely the reason the EUS required the most refinements among all the organs. Nevertheless, the physician refinements did not produce a statistically significant difference in EUS V200.

Aside from the bladder contours, the rectum contours required the least refinements by physicians. A noteworthy observation was that the automatic segmentation software produced clinical quality contours of the rectum in the presence of biodegradable hydrogel spacers (Fig 1B) (22). Less than 1% of patients in the 295 development MRI scans for the DL algorithm were implanted with hydrogel spacers, yet 23% of the patients in this study were implanted with hydrogel spacers. The algorithm had a tendency to include the hydrogel spacer in the rectum contours at lower values of the decision threshold but not at higher values (Fig 1A).

The MCC was suggested to be a superior metric to the F1 score (or Dice similarity coefficient) (13). We found the MCC to provide similar values as the F1 score in this segmentation application (Table E1 [supplement]); however, as a general-purpose metric, we prefer the MCC over F1 score. One advantage of the MCC is that it accounts for all four contingencies of the confusion matrix; the F1 score only considers true-positive (TP) findings, false-positive (FP) findings, and false-negative (FN) findings. Another advantage of MCC is that it can be positive or negative and therefore indicate either agreement or disagreement, respectively; F1 score is only able to quantify agreement. The F1 score can be useful in scenarios in which only contour agreement is of interest; however, it quantifies the harmonic mean of precision and recall. This property of the F1 score can lead to ambiguity when precision and recall are not of equal importance.

The calculation of the F1 score is similar to that of intersection over union (IoU) (equal to Jaccard index), which is another metric that quantifies contour similarity. Both IoU and the F1 score account for true-positive findings, false-positive findings, and false-negative findings, but F1 score averages false predictions (F1 score = TP/[TP + 0.5 × [FP + FN]]), whereas IoU does not (IoU = TP/[TP + FP + FN]). Therefore, IoU could be considered a more conservative measure of contour similarity compared with the F1 score and may also be more appropriate than F1 score in some scenarios. It should be noted that the F1 score and IoU are related by the equation F1 = 2 × IoU/(1 + IoU); IoU will always be less than or equal to the F1 score.

One limitation of our study was the relatively small number of patients who were evaluated. We treat approximately four patients per week with LDRPBT, and we plan to expand this study in the future to include its prospective use on a larger patient cohort. A second limitation was that we did not evaluate preimplant MR images in this study. Contours on preimplant MR images determine planned treatment volumes. The clinical assessment of the dl-based segmentation software on preimplant MR images is the topic of a commissioning study that is currently being conducted. A third limitation was that the patients investigated in this study were treated at the same institution where the automatic segmentation software was developed. Internal experiments have shown that the software produces accurate contours on externally acquired MRI scans and that it can translate across different MRI techniques. However, additional studies are needed to evaluate its implementation at institutions outside of our own and for other MRI-based radiation therapy applications (eg, external beam radiation therapy planning).

The initial evaluation of custom and fully integrated DL-based automatic segmentation software demonstrated the ability to produce clinical quality contours on the prostate and surrounding OARs. Its use in the clinic is expected to improve workflow efficiency and potentially improve the consistency and quality of MRI-based prostate radiation therapy procedures.