Evaluation of inter- and intra-observer variations in prostate gland delineation using CT-alone versus CT/TPUS

Abstract

Background

This study aims to explore the role of four-dimensional (4D) transperineal ultrasound (TPUS) in the contouring of prostate gland with planning computed tomography (CT) images, in the absence of magnetic resonance imaging (MRI).

Materials and methods

Five radiation oncologists (ROs) performed two rounds of prostate gland contouring (single-blinded) on CT-alone and CT/TPUS datasets obtained from 10 patients who underwent TPUS-guided external beam radiotherapy. Parameters include prostate volume, DICE similarity coefficient (DSC) and centroid position. Wilcoxon signed-rank test assessed the significance of inter-modality differences, and the intraclass correlation coefficient (ICC ) reflected inter- and intra-observer reliability of parameters.

Results

Inter-modality analysis revealed high agreement (based on DSC and centroid position) of prostate gland contours between CT-alone and CT/TPUS. Statistical significant difference was observed in the superior-inferior direction of the prostate centroid position (p = 0.011). All modalities yielded excellent inter-observer reliability of delineated prostate volume with ICC > 0.9, mean DSC > 0.8 and centroid position: CT-alone (ICC = 1.000) and CT/TPUS (ICC = 0.999) left-right (L/R); CT-alone (ICC = 0.999) and CT/TPUS (ICC = 0.998) anterior-posterior (A/P); CT-alone (ICC = 0.999) and CT/TPUS (ICC = 1.000) superior-inferior (S/I). Similarly, all modalities yielded excellent intra-observer reliability of delineated prostate volume, ICC > 0.9 and mean DSC > 0.8. Lastly, intra-observer reliability was excellent on both imaging modalities for the prostate centroid position, ICC > 0.9.

Conclusion

TPUS does not add significantly to the amount of anatomical information provided by CT images. However, TPUS can supplement planning CT to achieve a higher positional accuracy in the S/I direction if access to CT/MRI fusion is limited.

Introduction

Computed tomography (CT) and magnetic resonance imaging (MRI) are two commonly used imaging modalities for localisation and delineation of the prostate gland. MRI has proven to be superior in terms of contrast resolution allowing detailed visualisation of both the prostate and its peri-prostatic structures [1, 2]. CT/MRI is therefore the modality of choice for prostate gland delineation, leveraging on the advantages of MRI whilst accounting for its lack of tissue electron density values required for dose calculation. Moreover, Debois et al. [3] reported reduced inter-observer delineation variability of up to 3.5 times on CT/MRI compared to CT. However, studies have reported significant fusion uncertainties of the prostate gland on CT/MRI of up to 8mm, attributable to different-day imaging and differing pre-scan protocols [4, 5].

On the other hand, the four-dimensional (4D) transperineal ultrasound (TPUS) Clarity® system (Elekta AB, Stockholm, Sweden) is an emerging, non-invasive imaging modality in radiotherapy. Good image quality from the high spatial resolution and better soft tissue visualisation attributed to the short scan path length between the prostate gland and the perineum may allow more accurate delineation of the prostate gland on TPUS images [6, 7]. Similar to MRI, TPUS lacks the electron density information for heterogeneous dose calculation. Yet, TPUS is easily integrated into the simulation room via infrared fiducial markers tracking with the position of its probe and the corresponding TPUS images, with respect to the isocentre of the simulation room, known with sub-millimetre accuracy [7]. Compared to CT/MRI, a more accurate CT/TPUS registration is possible with minimum variations in patient’s position, bladder and rectum volume as the TPUS images are acquired during the CT simulation session. Consequently, this could substantially reduce fusion uncertainties, making TPUS a promising multimodality imaging tool to improve target delineation.

In our clinical practice, due to cost and logistical issues, only pre-androgen deprivation therapy (pre-ADT) staging MRI scans (2 months prior to planning CT) are available and routinely used for prostate cancer volumetric modulated arc therapy (VMAT) treatment planning, which potentially affects accuracy of target delineation. Hence, this feasibility study aimed to explore the role of 4D TPUS in assisting the delineation of prostate gland with planning CT images, in the absence of post-ADT MRI. The objectives of this study include: (a) to analyse the inter-modality comparison of prostate gland delineation and (b) to quantify the degree of inter- and intra-observer variation of prostate gland delineation on CT-alone and CT/TPUS scans. This study will provide initial confirmatory analysis of CT/TPUS for treatment planning, ultimately leading to its potential clinical applications in the future.

Materials and methods

Patient demographics

Ethics approval was obtained through the local ethics committee in March 2019 (CIRB ref no. 2019/2071) and informed consent was obtained from all patients. In this retrospective pilot study, CT/TPUS images from 10 patients were used to assess the inter-modality and inter- and intra-observer variation of prostate gland contours. All patients [median (SD) age: 78 ± 6.29 years] received ADT and TPUS-guided radiotherapy between October 2018 to July 2019 (Supplementary File — Tab. S1). Prior to each procedure, patients were instructed to void their bladder followed by drinking 400 mL of water and waiting for 30 minutes to obtain a comfortably full bladder. Patients were also advised to empty their bowels prior to CT simulation and daily treatment.

CT/TPUS images acquisition and registration

Planning CT and 4D TPUS scans acquired during the same session for each patient were retrieved. The patients were scanned supine, arms on chest and legs slightly bent on the Clarity® system autoscan probe kit (ASPK) knee rest. Planning CT images were acquired with slice thickness of 2.5 mm and 60 cm FOV (16-slice GE lightspeed) (GE Healthcare, Chicago, IL, USA). 4D TPUS images were acquired immediately before and after the CT scan (time lapse < 1 min). The 4D TPUS imaging using the Clarity® system involved the use of an autoscan probe mechanically positioned at the perineum, reducing operator dependency and displacement of the prostate gland and organs at risk due to probe pressure effects [7]. The radiation therapists (RTTs) completed a five-day theoretical and technical handling workshop on the operation of the Clarity® TPUS system from an application specialist. As described by Lachaine and Falco [6], the Clarity® TPUS system uses infrared tracking technology that enables spatial registration between TPUS and CT images, performed on automatic-fusion and contouring (AFC) workstation. Overall, 10 sets of CT-alone and 10 CT/TPUS images were analysed.

Contouring workflow

Five radiation oncologists (ROs) (associate consultants and above), Dr A to E (Supplementary File — Tab. S2) contoured the prostate gland independently on the Varian Eclipse TPS. Two rounds of blinded contouring were performed with scheduled contouring tasks (Supplementary File — Tab. S3) for each RO to delineate the prostate gland first on CT-alone followed by CT/TPUS, with an interval of one week between each modality to mitigate risks of recall bias. The first round of contours provided data for inter-modality and inter-observer reliability analysis while data for intra-observer reliability analysis was gathered from the second round two months later to reduce the effects of recall bias on intra-observer analysis.

Statistical analysis

Median (IQR) were analysed for the following parameters: (i) prostate volume (ii) Dice similarity coefficient (DSC) which quantifies inter-modality prostate volume overlap, with DSC > 0.8 indicating good volume overlap and, (iii) centroid position in the left-right (L/R), anterior-posterior (A/P) and superior-inferior (S/I) directions [8]. Mean (SD) DSC and the intraclass correlation coefficient (ICC) with 95% confidence intervals (CI) were also derived to reflect inter- and intra-observer variability and reliability of parameters, respectively, with ICC > 0.90 indicating excellent reliability [9].The statistical significance for inter-modality differences in prostate volume and centroid position were assessed with the Wilcoxon signed-rank test where significant, pairwise comparison through post-hoc analysis with Bonferroni adjustment was performed. Two-way mixed effects models assessed the statistical significance of ICC estimates with 95% CI based on mean-rating (k = 5) and consistency for inter-observer variation, and absolute agreement for intra-observer variation. All analysis was performed using SPSS Statistics Version 25.0 (IBM Corp., Armonk, NY). A two-sided p-value < 0.05 was considered significant in this work.

Results

Inter-modality contour variation

Prostate volume/DSC

Contours of prostate volumes obtained from CT-alone and CT/TPUS resulted in median (IQR) of 48.78 mL (33.22 mL to 62.6 mL) and 47.24 mL (34.43 mL to 64.31 mL), respectively. Of the 10 sets of corresponding images, seven sets of CT/TPUS images showed decreased prostate volumes compared to its corresponding CT-alone images. However, the median prostate volume difference was not statistically significant (p = 0.65). The corresponding median (IQR) DSC of the prostate volume between CT-alone and CT/TPUS was 0.905 (0.87 to 0.91).

Prostate centroid position

The median (IQR) centroid position difference of the prostate gland in the L/R (x), A/P (y) and S/I (z) directions for each patient, derived from the delineation of the five ROs in CT-alone and CT/TPUS, can be found in the Supplementary File — Table S4A and S4B. Overall, the observed difference in prostate centroid position between the two modalities were sub-millimetres (Supplementary File — Tab. S4C).

The centroid position differences in the three cardinal planes assessed by the Wilcoxon signed-rank test showed a statistically significant centroid difference in the S/I direction (p = 0.011) (Tab. 1). However, no statistical differences were observed in the L/R and A/P directions (p > 0.05).

Table 1.

Statistical significance of centroid position difference in left-right (L/R), anterior-posterior (A/P) and superior-inferior (S/I) directions as derived by Wilcoxon signed-rank tests

	Centroid position
	L/R	A/P	S/I
Z	−1.087a	−0.204b	−2.552b
Asymp. Sig (2-tailed)	0.277	0.838	0.011

^aBased on negative ranks;

^bBased on positive ranks.

Inter-observer variation

Excellent inter-observer reliability as demonstrated by the achieved ICC (95% CI) at 0.993 (0.983–0.998) was observed for the delineated prostate volume on CT-alone and CT/TPUS (p < 0.001). Significant extent of agreement between the ROs in the delineated prostate volume on both imaging modalities was achieved (p < 0.001).

On CT-alone, Dr A/E have the largest spatial overlap with a mean (SD) DSC of 0.90 (0.02), while Dr A/D have the lowest mean (SD) DSC of 0.84 (0.06). On CT/TPUS, Dr B/E have the greatest spatial overlap with a mean (SD) DSC of 0.90 (0.02), while Dr A/D, Dr C/D, and Dr D/E have the lowest mean (SD) DSC of 0.84 (0.05).

Delineated prostate volumes on CT-alone and CT/TPUS were comparable with percentage difference ranging from 16.4–40.4% to 16.2–41.6%, respectively. Figure 1 illustrates the variations in prostate volumes contoured by the five ROs on CT/TPUS. It was found that Dr D’s delineated mean prostate volume was consistently the lowest amongst all ROs on both imaging modalities: CT-alone (44.6 cm³), CT/TPUS (40.9 cm³). There was excellent inter-observer reliability for the prostate centroid position on CT-alone and CT/TPUS for all directions (p < 0.001) (Tab. 2). Overall, high agreement was observed in the prostate centroid position between the ROs for all directions in both imaging modalities (p < 0.001).

Figure 1.

Delineated prostate volume of each patient by Doctors A–E on computed tomography (CT)/ transperineal ultrasound (TPUS)

Table 2.

Inter-observer intraclass correlation coefficient (ICC ) with 95% confidence interval (CI) for centroid position in left/right (L/R), anterior/posterior (A/P) and superior/inferior (S/I) directions

Imaging modality	Centroid Position
	L/R		A/P		S/I
	ICC (p < 0.001)	95% CI	ICC (p < 0.001)	95% CI	ICC (p < 0.001)	95% CI
CT-alone	1.000	0.999–1.000	0.999	0.997–1.000	0.999	0.998–1.000
CT/TPUS	0.999	0.998–1.000	0.998	0.996–1.000	1.000	0.999–1.000

CT — computed tomography; TPUS — transperineal ultrasound

Intra-observer variation

Excellent intra-observer reliability was observed for all ROs in both imaging modalities with ICC > 0.9 (p < 0.001). Likewise, mean DSC > 0.8 was achieved for all ROs in both modalities. Intra-observer reliability was excellent in both imaging modalities for the prostate centroid position as ICC > 0.9 (p < 0.001). Excellent intra-observer agreement in the prostate centroid position was achieved for both CT and CT/TPUS in all directions (p < 0.001).

Average prostate centroid position delineated by each of the ROs in the second round of contouring was observed to be in the same directions as in the first round of contouring — right, posterior and inferior, on both imaging modalities. Figures S2A–C (Supplementary File) illustrate the corresponding centroid position of the delineated prostate gland on CT/TPUS in the L/R, A/P, S/I directions.

Discussion

Inter-modality contour variation

High consistency and agreement in prostate gland contours based on DSC and centroid position were observed between CT-alone and CT/TPUS, with smaller prostate volume on CT/TPUS suggesting a better visualisation of the prostate gland compared to CT-alone. In another study, exceptional correlation in prostate gland localisation was also observed when TPUS was compared to CT comparing 3D positional data [10]. Despite statistically significant (p < 0.001) extent of inter- and intra-observer agreement in both imaging modalities, residual observer variability could be attributed to clinical experience of ROs.

Inter- and intra-observer variation

High intra-observer ICC for CT/TPUS achieved in majority of the ROs (Dr A, Dr B and Dr E) could be attributed to consistency in image quality and observer interpretation of TPUS images (as supplementary to CT). Besides, mean (SD) DSC on CT-alone and CT/TPUS were comparable for all ROs with excellent intra-observer reliability. Additionally, the high inter- and intra-observer reliability (ICC > 0.9) among ROs in the S/I direction on CT/TPUS could suggest the value-added role of registering TPUS with planning CT images to enhance the precision of prostate gland delineation, particularly in the absence of an updated (post-ADT) MRI. Although Dr D’s ICC was the lowest across both imaging modalities when compared to the other ROs (p < 0.001), high reliability was still achieved with ICC > 0.8 (p < 0.001) and mean DSC > 0.8 in both modalities. This may suggest that the ROs’ clinical experience have an impact in the delineation of the prostate volume.

Values of CT/TPUS images

Evidence in literature reported an overestimation of prostate volumes contoured on CT by over 30% compared to CT/MRI [11, 12]. However, the use of outdated MRI scans in the current clinical process invalidates the benefits MRI brings into treatment planning. For instance, the use of pre-ADT MRI scans (considering the impact of ADT on prostate volume) and inconsistency in patient preparation (i.e. hydration and bowel protocol) and MRI scan duration may consequently vary scanned bladder and rectum volumes and relative geometric position of the prostate to other soft tissues in CT and MRI images. Such uncertainties in the use of CT /pre-ADT MRI fusion might confound the accuracy of target delineation and impact inter-observer reliability. Although MRI is a gold standard, an alternative imaging modality for prostate gland delineation may prove useful when access to CT/post-ADT MRT fusion is limited. High median (IQR) DSC value of 0.905 (0.87 to 0.91) between CT-alone and CT/TPUS generally indicates a high degree of prostate volume contour agreement between the two modalities [8]. Additionally, pairwise comparison of the prostate gland centroid position did not elicit statistically significant differences in both L/R and A/P directions. Yet, decreased median prostate volumes on seven sets of CT/TPUS images compared to their corresponding CT-alone images as contoured by the five ROs could be attributed to the better visualisation of the prostate gland on CT/TPUS dataset compared to CT-alone. Accuracy of prostate volume measurements using TPUS has previously been validated against trans-rectal ultrasound and found to be an accurate and less invasive alternative imaging method [13]. Recently, emerging evidence reported by Camps et al. [14] has also demonstrated how TPUS images can value-add visualisation of anatomical structures (e.g. prostate, bladder, rectum) within the pelvic region of prostate cases. In the post-prostatectomy setting, TPUS was also found to be a reliable resource for assessment of pelvic anatomical landmarks pre- and post-surgery prostatectomy [15, 16]. Above studies have highlighted the role of TPUS in the delineation of prostate cancer during treatment planning by supplementing CT images in the absence of an updated MRI.

On a similar note, inter- and intra-observer reliability of contours on CT/TPUS was observed to be the highest in the S/I (z-plane) direction. This was consistent with previous studies that reported better visualisation of the prostate gland in the S/I direction on ultrasound [17, 18]. Inter-modality difference between CT-alone and CT/TPUS was also the greatest and statistically significant in the S/I direction (p < 0.05). Thus, it could be inferred that the sagittal image plane of TPUS produced excellent image quality for visualisation, interpretation and co-registration where the cross-sectional soft tissue information of the prostate, penile bulb, bladder and anterior rectal boundaries were sufficiently distinguished [18].

Clinical experience of ROs

The methodology used, training and experience of the doctors involved and subjective interpretation of the prostate gland were, generally, sources of inter-observer variability [19]. This variability was more prevalent in observers with limited ultrasound imaging experience [7], with one study reporting statistically significant inter-observer variation when localising the prostate gland between two groups of ROs with varying experience (> 1 or < 1 year) [17]. In spite of the presence of variability, inter-observer variability in interpretation of the image decreases with growing ultrasound imaging experience [17]. The same holds for the intra-observer variability during TPUS imaging [18] and ultrasound image interpretation. Thus, increased level of user confidence and the potential of TPUS with requisite good image quality could possibly reduce inter- and intra-observer variability.

Similarly, findings in our study revealed inter-observer delineation variability amongst the doctors. Dr A and Dr E showed the closest and highest mean (SD) DSC values indicating excellent spatial overlap. In contrast, Dr A and Dr D had the lowest spatial overlap. This demonstrated the varying degree of inter-observer variability which could be attributed to clinical experience and variance in clinical practice. The contouring fidelity could well be a function of the physician’s age and experience with the use of TPUS imaging of the prostate. Hence, specific training such as recognising and differentiating normal TPUS appearance of the prostate and surrounding structures (i.e. bladder, penile bulb, urethral, symphysis pubis and rectum) on TPUS images may help enhance contouring consistency. Further study would be needed to increase the sample size of the physicians and evaluate the clinical outcomes caused by variability in delineation of the prostate gland.

Future developments

Moving forward, the use of post-ADT MRI scans and strict adherence to contouring guidelines could further reduce inter- and intra-observer variability. The use of updated MRI, as gold standard imaging, should be included during the planning process combined with CT and compared with TPUS used for contouring the prostate gland. Separately, deep learning-assisted contour (DLAC) could be explored to improve inter-observer reliability and consistency [20, 21]. With an adequate training dataset, DLAC can deliver exceptional segmentation results even if input images show large divergence in body size or shape [22, 23]. It was found that compared to manual delineation, the DLAC introduced a much lower coefficient of variation indicating its promise as an up-and-coming approach to achieve greater accuracy, consistency and efficiency [21, 23, 24].

Conclusion

A high degree of prostate volume/DSC and centroid position agreement were observed between CT-alone and CT/TPUS. Although TPUS does not add significantly to the amount of anatomical information provided by CT images, supplementing TPUS to CT images could achieve higher positional accuracy in the S/I direction if access to CT/MRI fusion is limited or bridge any inconsistency between CT and MRI due to different imaging day or rectal filling. Future exploration of auto-segmentation with the help of DLAC may further reduce inter- and intra-observer variability.