Topological data analysis visualization for interpretable assessment of AI contouring quality

Abstract

Advances in artificial intelligence have increased the availability of auto-segmentation tools. However, conventional accuracy metrics cannot capture regional segmentation differences between AI models or with respect to reference segmentations, necessary to interpret contouring variations. To address this, we developed a novel distance metric based on topological data analysis (TDA) to evaluate 3D point cloud representations of segmentations applied to six organs-at-risk (OARs) and lung gross tumor volume (GTV).

A total of 34 CTs and 54 CBCTs were analyzed to compare a bespoke AI segmentation method with reference clinical contours. TDA involved: (1) converting segmentations into 3D point clouds, (2) clustering them into regions via K-means with fixed seeds and cluster numbers determined by the Elbow method, (3) constructing directed graphs for AI and reference clusters using centroids as nodes, and (4) computing distances using unbalanced optimal mass transport. Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95) were also calculated.

TDA successfully identified local regions of high deviation in both OARs and GTVs of varying shapes. It correlated positively with HD95 and negatively with DSC based on Pearson’s correlation coefficient. Computation was efficient, averaging 1.72 s, and TDA effectively highlighted regions of greatest mismatch, providing quantitative visualization of poor concordance.

In conclusion, we developed a new TDA metric for comparing auto-segmentation of GTV and OARs. Importantly, it allows visualization of mismatching regions thus potentially allowing faster contour editing and evaluation.

1. INTRODUCTION

Advances in Artificial Intelligence has increased availability of auto-segmentation tools. However, currently used accuracy metrics cannot distinguish subtle details about the spatial location or the shape mismatch. Furthermore, distance metrics like Hausdorff distance (HD) and mean surface distance (MSD) can be too sensitive to outliers and lack interpretability regarding locations with poor concordance with respect to a reference segmentation. As a result, it is difficult to differentiate performance variations in methods at regional level. Understanding discrepancies in segmentation in regional level is important in order to assess systematic deviations in specific regions of interest, such as portions of organs occurring close to target where high dose is delivered. Careful evaluation of limitations in model performance is essential to ensure accurate contouring and precise dose delivery during radiotherapy planning.

Hence, we developed a new distance metric that measures similarity of segmentations represented as 3D point clouds using topological data analysis (TDA). Specifically, we focused on developing an interpretable approach to assess segmentation variations by visualizing regional differences in segmentations. In this study, we selected to visualize differences between a bespoke AI model and reference contours, albeit the same approach can be used to visualize differences between two different automated segmentation methods. The aim of this preliminary study is to test the feasibility of TDA in radiotherapy (RT) planning settings.

2. METHODOLOGY

2.1. Datasets used for analysis

Two different datasets consisting of 34 simulation CT scans and 54 cone beam CT (CBCT) scans acquired from patients with lung cancer and who underwent conventionally fractionated radiation treatment (RT) were analyzed. The first dataset was used to assess segmentation variations of six organs-at-risk (OAR) with respect to clinical delineations used for radiation treatment planning, and the second CBCT dataset was used to evaluate differences in the segmentation of the primary gross tumor volume (GTV) with respect to expert delineations. CBCT dataset consisting of multiple scans acquired longitudinally during the course of RT involving 6 patients was used to evaluate GTV contours because it allowed to assess performance differences longitudinally and CBCT presents a more challenging case for evaluating segmentation differences compared to CT scans.

AI contours for OAR were generated using a published method called multiple resolution residual network (MRRN) [1]. AI-based GTV contours were produced using the same architecture but trained using an approach to extract features signaling contrast between tumor and the normal tissue called cross modality educed distillation learning (CMEDL) [2].

2.2. Overview of proposed interpretable segmentation quality assessment using TDA

To assess segmentation quality beyond global similarity indices, we designed a clustering-based framework that quantifies and localizes regional mismatches between the ground truth (GT) and auto-segmentation (SEG) masks (Figure 1). The workflow consists of four main steps: (1) k-means clustering, (2) bipartite cluster matching, (3) distance computation using unbalanced optimal mass transport (UOMT), and (4) visualization of cluster with largest deviations.

Figure 1.

Overall workflow of the proposed approach

2.3. TDA workflow

Each structure including organ-at-risk (OAR) and tumor GTV was extracted using bespoke AI models [1], [2]. The largest connected component of the individual segmentations were retained to remove noise and artifacts. The binary masks were then converted into 3D point clouds by extracting the 3D coordinates of the masks, representing the spatial distribution of the structure. To capture sub-structural heterogeneity, the voxel point clouds were partitioned into clusters using k-means (Equation 1) [3], where J is the objective function, k is the number of clusters, n is the total number of voxels, x_i is the i-th voxel, and c_j is the centroid of cluster j.

$$J={\sum }_{j=1}^k{\sum }_{i=1}^n{‖x_i^{\left(j\right)}{c}_j‖}^2$$ (1)

The optimal number of clusters, k, for each mask was determined automatically via the elbow method [4], where the reduction in clustering inertia was assessed across candidate cluster numbers. Clusters from GT and SEG masks were subsequently matched based on their spatial proximity, ensuring one-to-one correspondence even in cases where the number of clusters differed using the following bipartite graph matching based on Sinkhorn distance [5] (Equation 2).

$$d_{M,\alpha }\left(r,c\right)≔\underset{P\in U_{\alpha }〈r,c〉}{\text{min}}〈P,M〉$$ (2)

For each matched cluster pair, the intra-cluster dispersion was assessed by measuring the deviation of individual voxel coordinates within the cluster with respect to the cluster centroid along all three directions and summarized using Euclidean metric. Next, the maximum deviation along each spatial dimension (x, y, z) was extracted, representing the furthest voxel displacement per axis. Individual cluster spreads were then averaged to obtain a single scalar measure of cluster spread. This process was done for GT and SEG, subsequently resulting in two arrays, one for GT and one for SEG, each containing n average distances, f1 and f2, respectively. Finally, the UOMT was computed using f1 and f2 graphs [6]. UOMT was used to compute differences between the two graphs as it does not require mass conservation and is not sensitive to outlier clusters. The final TDA metric is defined as the UOMT between two distributions of average distances of cluster centroid to all other points in the clusters.

2.3. Assessment of TDA metric

The TDA metric provides a quantitative assessment of segmentation differences between the evaluated models as a whole. In addition, for completeness and to measure correlations between the TDA metric and conventional metrics, the Dice Similarity Coefficient (DSC) and the 95th percentile Hausdorff distance (HD95) were computed using Pearson’s correlation coefficient. Slice-wise measures of the same metrics were also computed to capture the spatial variation in performance across the cranio-caudal axis.

We further evaluated our framework using a toy experiment with simple 2D cases designed to test the TDA metric under variations in cluster size, outliers, and under- or over-segmentation. Cases where conventional metrics such as DSC and HD failed were specifically included to assess TDA’s response, demonstrating its robustness and sensitivity to subtle segmentation discrepancies.

2.4. TDA-based visualization

Visualization of local segmentation differences was performed by identifying the cluster with the largest deviation between the AI-generated and reference contours. For each OAR, the tool first presents three-dimensional renderings of both segmentations after clustering, followed by the cluster with maximal deviation as determined by the UOMT computation. The corresponding central CT slice is then displayed with overlays of the individual segmentations, and additional slices within the cluster can be examined when necessary. To further assess the clinical impact of these deviations, supplementary metrics, including dosimetric evaluations, can also be computed for the segmented regions within the cluster.

3. RESULTS

3.1. TDA-based visualization

Figure 2 presents the TDA visualization results, highlighting the cluster with maximal deviation for each structure. In elongated structures such as the esophagus and spinal cord, deviations were primarily due to differences in contour length at the proximal and distal boundaries. For GTV, discrepancies were concentrated at the contour periphery between ground truth and auto-segmentation. The method further identifies the corresponding CT slices for each cluster, enabling precise localization and quantitative assessment of segmentation mismatches.

Figure 2.

TDA-clustered ground truth, segmentation as well as the identified worst cluster and its CT slice: examples in esophagus, PBT and spinal cords

3.2. Correlation plot with conventional metrics

Through our correlation analysis, we found that the UOMT distance from our proposed method was significantly correlated with DSC across all OARs except the left lung. In contrast, correlations with HD95 were generally weaker and did not reach statistical significance for the GTV and left lung. Overall, UOMT showed negative correlations with DSC and positive correlations with HD95, indicating that larger UOMT values reflect poorer agreement between the two contours being compared.

3.3. Validating the TDA metric on straightforward examples

Figure 3 shows representative examples comparing ground truth (solid line) and segmentation (green) in straight forward examples. In the size analysis (Figure 3a), although the discrepancy in DSC is merely a single pixel when compared to the respective references, there are substantial differences observed between case 1 and case 2. On the other hand, HD/HD95 metrics exhibit no or minimal differences. TDA also indicates a slight variation. Next, the outlier detection capabilities of the metrics were examined (Figure 3b). While DSC showed no difference, there were observable variations in the HD/HD95 metrics, indicating a higher sensitivity in detecting outliers. TDA shows somewhat gradual increase as outlier becomes worse, with a smaller range compared to HD/HD95. Lastly, the effect of over- and under-segmentation was observed (Figure 3c). Case 1 and 2 are under- and over-segmented, respectively, compared the reference by one layer and two layers of pixels respectively. Among the metrics, DSC showed the most noticeable differences between under and over segmentations. There is no difference in HD/HD95 and TDA.

Figure 3.

(a) Effect of volume size on metrics, (b) impact of outliers and their positions, (c) effect of under- and over-segmentation. Abbreviations: DSC – Dice similarity coefficient; HD – Hausdorff distance; HD95 – 95th percentile Hausdorff distance; TDA – topological data analysis.

4. CONCLUSIONS

We introduced a clustering-based TDA workflow to evaluate segmentation quality and localize regional mismatches. TDA effectively identifies discrepancies in OARs and GTV, guiding attention to relevant CT slices. Correlations with DSC were generally significant, while correlations with HD95 were weaker, highlighting TDA’s complementary sensitivity to spatial deviations. Overall, TDA provides a more comprehensive assessment than conventional metrics, particularly for detecting localized differences that may require clinical review.