A geometry and dose-volume based performance monitoring of artificial intelligence models in radiotherapy treatment planning for prostate cancer

Highlights

• •Clinical data can be used to monitor output of artificial intelligence models.
• •For deep learning contouring, clinically relevant changes can be detected using dose-volume analysis, in combination with a new Local Surface Dice score.
• •Dose-volume metrics have to report dose changes at a minimal critical volume.
• •Out-Of-Distribution detection using both dose-volume and geometrical metrics, can be used for data selection for transfer learning.

Abstract

Background and Purpose

Clinical Artificial Intelligence (AI) implementations lack ground-truth when applied on real-world data. This study investigated how combined geometrical and dose-volume metrics can be used as performance monitoring tools to detect clinically relevant candidates for model retraining.

Materials and Methods

Fifty patients were analyzed for both AI-segmentation and planning. For AI-segmentation, geometrical (Standard Surface Dice 3 mm and Local Surface Dice 3 mm) and dose-volume based parameters were calculated for two organs (bladder and anorectum) to compare AI output against the clinically corrected structure. A Local Surface Dice was introduced to detect geometrical changes in the vicinity of the target volumes, while an Absolute Dose Difference (ADD) evaluation increased focus on dose-volume related changes. AI-planning performance was evaluated using clinical goal analysis in combination with volume and target overlap metrics.

Results

The Local Surface Dice reported equal or lower values compared to the Standard Surface Dice (anorectum: (0.93 ± 0.11) vs (0.98 ± 0.04); bladder: (0.97 ± 0.06) vs (0.98 ± 0.04)). The ADD metric showed a difference of (0.9 ± 0.8)Gy for the anorectum ${\text{D}}_{{1\text{cm}}^3}$. The bladder ${\text{D}}_{{\text{5cm}}^3}$ reported a difference of (0.7 ± 1.5)Gy. Mandatory clinical goals were fulfilled in 90 % of the DLP plans.

Conclusions

Combining dose-volume and geometrical metrics allowed detection of clinically relevant changes, applied to both auto-segmentation and auto-planning output and the Local Surface Dice was more sensitive to local changes compared to the Standard Surface Dice. This monitoring is able to evaluate AI behavior in clinical practice and allows candidate selection for active learning.

1. Introduction

Every step in the radiation therapy (RT) workflow has been exposed to Artificial Intelligence (AI) solutions and the benefits have been demonstrated [1], [2], [3], [4]. However, most studies report on retrospective or simulated evaluations, which does not represent a clinical setting [5].

To date, auto-segmentation tools contour a variety of structures on both CT and MR with high accuracy [6], [7], [8], [9], [10]. Some papers even report feasibility of treatment plan creation based on AI generated structures [11]. However, AI-segmented regions of interests (ROIs) close to the target volume still need verification [12], [13]. Small user-adjustments also remain necessary to ensure quality and guideline compliance, mainly in case of changing guidelines or image acquisition protocols [14].

Automated treatment plan generation in general improves planning efficiency and reduces plan quality variability [15]. Different approaches are available and able to create treatment plans for a variety of pathologies and dose prescriptions [16], [17], [18], [19]. Despite these benefits, the main challenges remain safe implementation, ongoing maintenance of an AI model and adaptation to changing workflows and procedures [20]. Stereotactic body radiation therapy (SBRT) treatments, delivering high doses in a limited number of fractions, require even more stringent quality assurance (QA) measures [21].

Guidelines for safe AI implementation and QA were published [22] and the success rate of AI in RT will largely depend on the interpretability and data-model dependency [23]. Also, data standardization and integration into existing clinical workflows remains a challenge [24]. Validation and commissioning reports help documenting intended use and limitations of a model. Reviewing automated plans using checklists will improve patient safety as well [25]. However, automated and independent QA of the AI models is preferable as more efficiency in the radiotherapy workflow is needed [26]. Uncertainty is embedded in AI models and continuously monitoring the behavior on unseen data can make AI even more efficient by guiding users towards cases that need attention [27].

Continuously monitoring AI output in clinical routine benefits from the large number of patients, but also requires new, combined metrics, both geometrical and dose-volume related, to detect the clinical relevance of anomalous output and to allow the selection of appropriate datasets for further optimization of AI models within the framework of expert-augmented machine learning [28].

This study investigated how problem-specific clinical knowledge from experts can be automatically extracted from an AI supported workflow using a combined analysis of both geometrical and dose-volume metrics. Performance of both segmentation and planning models, clinically implemented for SBRT prostate treatments, was evaluated. In addition, metrics with specific thresholds were proposed to detect outliers for model retraining.

2. Material and Methods

2.1. Patient data

Fifty SBRT prostate cancer patients were simulated on a Brilliance Big Bore (Philips, The Netherlands) or a SOMATOM Go-Sim CT scanner (Siemens, Germany). All auto-segmentations and automated treatment plans were created in our treatment planning system (TPS) (RayStation 11B, RaySearch Laboratories AB, Sweden) and all AI models were commercially available and trained, validated and tested by the company. The AI-segmentation model was initially trained on local data of the Iridium Netwerk (Belgium), and data of the University Health Network (UHN) (Canada) was used for AI-planning model training. All patients were treated at the Iridium Netwerk following the PACE Trial guidelines (ClinicalTrials.gov Identifier: NCT01584258) in five fractions, received a dose of 7.25 Gy per fraction to the PTV, which was a uniform expansion of the CTV of 5 mm, except in the posterior direction where a margin of 3 mm was used. The training dataset of UHN was based on a uniform PTV expansion of 7 mm. Due to de-identified nature of the data, the need for ethics review was waived for this retrospective study.

2.2. Clinical AI models

The Deep Learning Segmentation (DLS) model (RSL Male Pelvic CT (v1.0.0), RaySearch Laboratories AB, Sweden) segmented five different organs (prostate (as CTV), anorectum, bladder and left and right femoral head) and used the simulation CT as input and outputted deep learning ROIs (R_DL) based on the ACROP and RTOG guidelines [29], [30]. In clinical routine, contours were reviewed by a radiation oncologist (RO) and corrected when needed, providing planners with clinically approved ROIs (R_clin) (Fig. 1). The CTV generated by the model differed from the clinically used CTV on two accounts: inclusion of seminal vesicles and the use of MRI. As described in the PACE Trial guidelines, part of the seminal vesicles will be included in the CTV, depending on the patient’s risk group (low, intermediate or high). Additionally, MRI imaging was used in clinical routine to determine the anatomical borders of the prostate, which impacts the CTV contour. Therefore, the CTV contour was not further evaluated.

Fig. 1.

Schematic overview of a deep learning guided contouring and dose planning workflow. The input is the CT image on which the DLS model generates AI contours R_DL, followed by a radiation oncologist’s review resulting in contours R_clin. Based on R_clin, the DLP model creates plan P_DL which can be further optimized by a planner leading to plan P_clin within the fine-tune optimization step. All datasets (R_DL, R_clin, P_DL, P_clin) are stored in the database and were used for DLS and DLP monitoring.

The Deep Learning Planning (DLP) model configuration (RSL-Prostate-3625-SBRT (v3.0.0), RaySearch Laboratories AB, Sweden) consisted of a two-step approach: prediction/mimicking of a 3D-dose distribution based on input ROIs, followed by a post-processing optimization. The latter facilitated model sharing between clinics as prediction/mimicking highly relies on the training dataset, the post-processing allowed additional output tuning based on local, clinical needs. A deep learning generated plan (P_DL) was the result of running the model (Fig. 1). An optional ‘Fine-tune optimization’ step allowed manual tweaking of the treatment plan to fulfill most of the patient specific requirements, if appropriate. This final plan was the clinical plan (P_clin) which was used for treatment delivery.

Prior to clinical use, DLS and DLP model commissioning was performed on five patients. For DLS, all R_DL‘s were evaluated by an experienced radiation oncologist (RO) and scored based on quality and timesaving. For DLP, the model’s output P_DL was retrospectively compared against manually created, clinically used plans and quality was scored using the Plan Quality Index (PQI) [31]. To assess treatment machine deliverability, patient specific plan QA was performed using the SunCHECK platform (Sun Nuclear Corporation, Melbourne, USA).

2.3. Auto-segmentation performance monitoring

Continuous monitoring compared R_DL with R_clin using both geometrical and dose-volume parameters on bladder (#50) and anorectum (#50) ROIs. The geometrical performance of the model was assessed via the Standard Surface Dice 3 mm and a new Local Surface Dice 3 mm. Both metrics were calculated based on following definition of a Surface Dice with a tolerance of 3 mm:

$Surface\_Dice\left(S_{\text{DLS}},S_{\text{clin}},3\text{mm}\right)=\frac{\left|S_{\text{DLS}}\cap B_{\text{clin}}^{\text{3mm}}\right|+\left|S_{\text{clin}}\cap B_{\text{DLS}}^{\text{3mm}}\right|}{\left|S_{\text{DLS}}\right|+\left|S_{\text{clin}}\right|}$where $S_{\text{DLS}}$ and $S_{\text{clin}}$ were the surfaces of R_DL and R_clin and $B_{\text{DLS}}^{\text{3mm}}$ and $B_{\text{clin}}^{\text{3mm}}$ were the tolerance regions, 3 mm in- and outside the corresponding surface $S$ [32]. The Standard Surface Dice used the entire ROI volumes as input for $S_{\text{DLS}}$ and $S_{\text{clin}}$. On the other hand, the Local Surface Dice only evaluated differences between $S_{\text{DLS}}^{\prime }$ and $S_{\text{clin}}^{\prime }$ where $S^{\prime }$ is the surface $S$ cropped at the transversal slices 2 cm away from the target, as visualized in the sagittal plane of Fig. 2 and this distance was chosen based on used thresholds in the online adaptive setting [33], [34]. The Standard Surface Dice already provided clinically relevant and quantitative measures [35], [36], the local variation further increased the focus on the clinically relevant changes in the vicinity of the target. Dose-volume impact of the applied changes to R_DL was investigated by evaluating different dose volume constraints for both R_DL and R_clin on the clinically approved dose distribution P_clin. The absolute dose difference (ADD) at different clinically relevant dose levels, as listed in Table 1, were reported next to the average dose (D_average) and the near maximum dose (${\text{D}}_{{0.03\text{cm}}^3}$). For every dose volume constraint, the absolute dose differences ADD = |D_clin – D_DLS| were calculated as a change in a contour might have an impact on the reported dose value. Segmented organs were categorized into four groups, based on an ADD of 100 cGy and a mean Local Surface Dice minus one Standard Deviation (SD). Zone 1 resembled patients with a high Local Dice score and an ADD less than 100 cGy, while in Zone 2 the reported ADD became larger than 100 cGy. Zone 3 contains patients with a Local Dice score that was one SD less than the mean Local Dice. In Zone 4 both reported values showed larger deviations. Patient with hip prostheses were marked with a red border.

Fig. 2.

Typical output of DLS and DLP models. White, DLS generated ROIs are plotted next to the clinical correct yellow bladder and brown anorectum. On the sagittal plane, the dashed red lines mark the boundaries for the Local Surface Dice calculation. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Table 1.

Mandatory and optimal ROI dose constraints for SBRT.

	Dose volume constraints
	Dose (Gy)	Minimum volume (%)
Target		Mandatory	Optimal
PTV	36.25	90	95
	Dose (Gy)	Maximum volume (% or cm3)
OAR		Mandatory	Optimal
Rectum	18.1	50 %
	29.0	20 %
	36.0	2 cm3	1 cm3
Bladder	18.1	40 %
	37.0	10 cm3	5 cm3
Femoral heads	14.5	–	5 %
Bowel	18.1	5 cm3
	30.0	1 cm3
Penile bulb	29.5	–	50 %
Urethra	42.0	–	50 %

2.4. Auto-planning performance monitoring

In clinical routine, P_DL was autonomously created via the DLP process and a consistent shape of the dose distribution (Fig. 2) was obtained for all fifty patients. P_clin underwent minor adaptations from scaling to the dose prescription to further optimize the plan for better fulfilling the clinical goals or finding a different balance between OAR sparing and target coverage. To quantify the differences between P_DL and P_clin, clinical goals of Table 1 were evaluated on both dose distributions against R_clin volumes. Additionally, spatial information, such as ROI volumes and overlap ratios between ROI and PTV, were reported as the dose prediction method uses ROIs as input to the model. Geometrical metrics related to the PTV margin are standardized using:

$$x\prime =\frac{x-\text{average}(X)}{\text{stdev}(X)}$$

with $x$ the PTV volumes, or the overlap volume between the PTV and an organ, and $X$ the corresponding values of all fifty plans. The applied scaling allowed comparing volume-based metrics between the training set and the local data in case different PTV margins were used. All these parameters were calculated for both the autonomously created P_DL’s and clinically approved P_clin’s.

Target volume reported standardized PTV volumes because of the difference in PTV margin and the standardized overlap volume considered the intersection with both bladder and anorectum. Model’s performance was classified based on the optimal clinical goal analysis for both P_DL and P_clin and four different groups could be distinguished: green dots resemble patient ROIs for which P_DL did achieve the optimal dose constraint and not further optimization of P_clin was needed (P_DL was selected for treatment). In case P_DL did not pass the clinical goal, but P_clin did, the plan was marked orange. Patient ROIs for which both P_DL and P_clin were not able to fulfill the most stringent clinical goal were marked as red. Finally, if a different compromise between organ sparing and target coverage was preferred a dot can turn black in case P_DL did pass the clinical goal, but P_clin did not.

To compare reported means, a T-test for two independent samples was performed in Python 3.9 using the SciPy package.

3. Results

3.1. Model commissioning

The commissioning phase on five patients revealed necessity of no (cf. femoral heads) to minor (cf. bladder, anorectum) corrections with high timesaving for DLS. For DLP, a significantly (p < 0.001) better sparing of the anorectum (V18.1 Gy = (23.9 ± 7)% for the manual plan and (15.3 ± 6)% for P_DL) is reflected in a better PQI for P_DL (65.4 %) compared to manual planning (74.4 %). No other significant clinical goals differences were detected. In general, the number of Monitor Units increases when comparing manual plans against P_DL: (1963 ± 192) and (2680 ± 200) (p < 0.001) respectively. The gamma pass rate (3 %2mm) remained identical: (100 ± 0.1)% for manual plans and (99.9 ± 0.2)% for P_DL.

3.2. Auto-segmentation – Performance monitoring

The Standard Surface Dice 3 mm and the Local Surface Dice 3 mm reported values of (0.98 ± 0.04) and (0.93 ± 0.11) (p = 0.006) respectively for the anorectum, and (0.98 ± 0.04) and (0.97 ± 0.06) (p = 0.3) for bladder. The anorectum was evaluated against four different ADD’s: ${\text{D}}_{{0.03\text{cm}}^3}$ (0.5 ± 0.5)Gy, ${\text{D}}_{{1\text{cm}}^3}$ (0.9 ± 0.8)Gy, ${\text{D}}_{{\text{2cm}}^3}$ (1.3 ± 1.2)Gy and D_average (0.6 ± 0.6)Gy (p < 0.001). Bladder reported ADD’s of (0.2 ± 0.3)Gy,(0.7 ± 1.5)Gy, (1.5 ± 2.3)Gy and (0.6 ± 1.2)Gy (p < 0.001) for respectively ${\text{D}}_{{0.03\text{cm}}^3}$, ${\text{D}}_{{\text{5cm}}^3}$, ${\text{D}}_{{10\text{cm}}^3}$ and D_average. All information is visualized in Fig. 3.

Fig. 3.

On the x-axis, Absolute Dose Difference (ADD) for anorectum and bladder were reported for different critical volumes. On the y-axis, the Local Surface Dice is plotted and the color of each dot resembles the Standard Surface Dice as shown by the color bar. The dashed lines resemble the ADD of 1 Gy and the first standard deviation of the Local Surface Dice. Each graph is divided into four zones, reflecting perfect DLS segmentations in zone 1, minor corrected DLS ROIs with large dose-volume impact (zone 2), major corrected DLS ROIs outside the high dose region and less clinical impact in zone 3 and zone 4 shows major corrections with large difference in ADD.

3.3. Auto-planning – Performance monitoring

Clinical goal evaluation revealed that P_DL did fulfill all mandatory clinical goals (Table 1) in 90 % of the cases, while all where fulfilled in P_clin. The success rate dropped to 32 % and 74 % respectively for P_DL and P_clin when the optimal dose constraints were evaluated. The geometrical analysis showed significant volume differences between the training and the local dataset for respectively PTV ((154 ± 45)cm³; (120 ± 50)cm³ with p < 0.001), bladder ((141 ± 51)cm³; (226 ± 136)cm³ with p < 0.001) and rectum ((58 ± 18)cm³; (76 ± 29)cm³ with p < 0.001).

Fig. 4 also reveals clinical data points that were outside the first SD of the training data.

Fig. 4.

Clinical goal analysis comparing DLP output against the clinical plan for both bladder (D5cm³ < 37 Gy), anorectum (D1cm³ < 36 Gy) and PTV (V36.25 Gy > 95 %) and every colored dot resembles a patient ROI belonging to one of the four clinical goal evaluation groups. The blue distribution plots above and to the right of each scatter plot show the corresponding data of the training data. The dashed blue lines show the first standard deviation of this training data. The lower left graph visualizes the difference in PTV volume between local and training data. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

4. Discussion

Model commissioning proved its advantage with respect to both time and quality of models’ output. Nevertheless, the results largely depended on the quality and variety of the commissioning test set and even large retrospective studies are not representative and not able to provide a full estimate of the models’ clinical behavior [5]. Also, retrospective analysis often compared against ground truth datasets for both segmentation and planning models [37], [38], [39]. So, proper performance monitoring tools are appropriate and must deal with variations inherent to a clinical workflow such as, due to timesaving, mainly relevant segmentation errors will be corrected in the vicinity of the target volumes as this will impact the dose distribution and the lack of ground truth data [12]. For DLP, different PTV margins compromise direct comparison between training and local datasets. Despite the feasibility to automatically create deliverable treatment plans, detecting sub-optimal created plans remains a challenge [40].

DLS performance monitoring must be able to detect local and certainly small changes with a large potential impact on the dose distribution. Fig. 3 shows that the Local Surface Dice is more sensitive compared to the Standard Surface Dice in cases where the Standard Dice is high (>0.9), but the Local Dice score is low (<0.7) and even significantly different in case of rectum analysis. Additionally, the ADD improves revealing cases with a high Local Surface Dice (>1SD) but with large (>1Gy) difference in reported dose. Literature already showed the dose-volume impact, but either a new plan was optimized, introducing additional uncertainty [13], or only ROIs were reported further away from high doses [41]. By comparing both R_DL and R_clin to the approved dose distribution of P_clin, no plan optimization bias was introduced. The combination of both geometrical and dose-volume metrics proves its advantage as the Local Surface Dice is sensitive to detect large deviations close to the target volumes (Fig. 3, zone 3 and 4) and the ADD can detect small changes in the vicinity of the high dose region (Fig. 3, zone 2). The bladder in Fig. 2 is a zone 2 example. Deviations in zone 3 can be categorized as clinically less relevant with no major impact on the dose distribution, while the opposite is true in zone 4. Fig. 3 also shows that ${\text{D}}_{{0.03\text{cm}}^3}$ and D_average are more robust as a dose-volume metric compared to the ${\text{D}}_{{\text{1cm}}^3}$, ${\text{D}}_{{\text{2cm}}^3}$, ${\text{D}}_{{\text{5cm}}^3}$ or ${\text{D}}_{{10\text{cm}}^3}$ as the mean ADD is higher for the latter and differ significantly. Too small volumes (${\text{D}}_{{0.03\text{cm}}^3}$) are not sensitive enough in case a ROI shifts in a homogenous dose distribution. Further, average doses report on too large volumes, which decreases sensitivity as well. In addition to previous papers, a minimal critical volume (>1cm³) is recommended in case of dose-volume analysis of AI segmented contours. A different dose-volume analysis can be obtained by creating a treatment plan directly on R_DL to investigate how DLS does impact the dose distribution. As this approach inherently increases the user interaction, we currently only evaluated the ADD parameter.

Femoral head and prostate segmentations were not reported. The former was never adapted manually, leading to a Standard Surface Dice score of 1 for all femoral heads, both left and right. The latter was impacted by the local delineation guideline of the prostate contour and the superposition of Gaussian distributions when plotting the clinical PTV volume data also reveals the different, risk group dependent, PTV definitions in Fig. 4. The different PTV margin necessitated data standardization, the risk-group dependent delineations made it impossible to monitor the quality of the prostate segmentation.

DLP monitoring showed a lower mean PTV volume in the clinical data compared to the training data, which originates from the smaller PTV margin and it impacts the target-ROI overlap volume. As P_DL did fulfill almost all mandatory clinical goals, comparison of the plan quality between P_clin and P_dlp did focus on the fulfillment of the optimal dose constraints. To date, dose normalization can differ between P_clin and P_dlp which may impact the dose comparison, but has the advantage that P_dlp is created fully autonomously. Fig. 4 shows the impact of both organ and overlap volume on the clinical goal fulfillment. Larger overlap volumes between OAR and target lead to a more complex plan optimization problem as reflected by the red dots for which even P_clin could not fulfill the clinical goal. This increased complexity dominates the incapability of automatically creating a clinically acceptable P_dlp and user interaction is needed to find the optimal balance between organ sparing and target coverage. It is unclear if adding these plans to the training dataset will improve model’s output. In case of increasing ROI volume, the dose prediction might be impacted as large ROI volumes are not represented in the training dataset and most of the orange dots have a volume that exceeds the first SD of the training dataset. Consequently, these plans are interesting candidates to add to the training dataset to improve performance and robustness of the model. Volume differences between local data and training data might be introduced by different bladder and anorectum preparations between the different centers. This observation strengthens the need for OOD detection to better understand the behavior of AI models and at the same time the robustness of the dose prediction is proven as DLP hardly generates plans causing major violations.

In conclusion, combining geometrical and dose-volume metrics is of added value when monitoring the performance of both AI based segmentation and planning models. For segmentation, the Local Surface Dice is more sensitive compared to the Standard Surface Dice and detection of changes in the vicinity of the target volume can further improve in combination with ADD analysis if a critical volume of at least 1 cm³ is used. For planning, reporting volumes in combination with clinical goal analysis enables OOD detection. Future work might explore analyzing dose-volume metrics of plans created directly on R_DL in an automated way as well as improving the comparison between P_dlp and P_clin by taking into account the dose normalization.

CRediT authorship contribution statement

Geert De Kerf: Conceptualization, Methodology, Writing – original draft, Writing – review & editing. Michaël Claessens: Conceptualization, Writing – original draft, Writing – review & editing. Fadoua Raouassi: Conceptualization. Carole Mercier: Validation, Writing – review & editing. Daan Stas: Validation, Writing – review & editing. Piet Ost: Validation, Writing – review & editing. Piet Dirix: Validation, Writing – review & editing. Dirk Verellen: Writing – review & editing, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.