Comparative analysis of artificial intelligence-based contouring of cardiac substructures on computed tomography scans for radiation therapy

Graphical abstract

Highlights

• •Cardiac substructures from 11 auto-contouring solutions were evaluated.
• •Dice Similarity Coefficient across solutions ranged from 0.03 to 0.93.
• •Hausdorff distance (95th percentile) ranged from 5 mm to 70 mm.
• •Change in contrast enhancement impacted volumes by up to −41% and +122%.
• •Harmonization of contouring rules and training data transparency are needed.

Abstract

Background and purpose

Artificial intelligence-based contouring tools enable assessment of radiation doses to cardiac substructures beyond mean heart dose. This study examined inter-solution variations in raw contours and the impact of non-contrast enhancement on contours for each solution.

Materials and methods

Contrast-enhanced (CE) and non-contrast-enhanced (NCE) breath-hold thoracic computed tomography (CT) scans, sequentially acquired during the same imaging session for twenty lung cancer patients, were used. Seven commercial, three open-source, and one in-house AI solutions were evaluated. On CE-CTs, solutions were compared using Dice Similarity Coefficient (DSC) and 95th percentile of Hausdorff distance (HD95) across each pair of solutions. Then, the effect of non-contrast enhancement on contours was assessed using volume ratios between NCE-CT and CE-CT for each solution.

Results

Typically, ten cardiac substructures were contoured by most of the solutions. For the whole heart, cardiac chambers and great vessels, the average median DSC was above 0.8 for 55 of the 123 structure-solution pairs (45%), and the average median HD95 was below 10 mm for 47 of the 123 structure-solution pairs (38%). For the coronary arteries, the average median DSC ranged between 0.03 and 0.50 and the average median HD95 ranged between 19 mm and 70 mm. Non-contrast enhancement influenced results variably; volume differences were below 10% for 84 of the 123 structure-solution pairs (68%).

Conclusions

Automatic contouring solutions exhibited inter-solution variability for cardiac substructures that may have clinical impact. Greater transparency and standardisation of models, ideally through international consensus and shared datasets, are essential.

1. Introduction

The growing interest in cardiac substructures and associated dose–effect relationships, beyond the mean heart dose, represents a significant advancement in the field of radiation therapy [1]. The anatomical and functional complexities of the heart are essential to consider when assessing the existence of dose–effect relationships [2].

With the advent of automatic delineation, the possibility of sparing the dose delivered to specific cardiac substructures becomes a tangible reality. However, automation should not be accompanied by excessive confidence, especially to address gaps in knowledge, despite undeniable advantages in terms of speed and reproducibility.

Recently, several studies have compared artificial intelligence contouring solutions for both targets and organs of interest [3], [4], [5]. However, none of these studies provided comparisons of the automatic delineation of the cardiac substructures. In contrast, several studies have evaluated the performance of automatic delineation in comparison to manual delineation [6], [7], [8], [9], [10].

The purpose of this study was to compare Artificial Intelligence (AI)-based contouring solutions for cardiac substructures and evaluate the influence of non-contrast enhancement on contouring performance for each solution. No medical supervision or corrections were involved in this process. It is an illustrative comparison of inter-solution variability in cardiac substructures contours rather than a solution ranking by clinicians.

2. Materials and methods

2.1. Datasets

Twenty patients undergoing radiation therapy for a primary lung cancer were selected based on the availability of both contrast-enhanced (CE) and non-contrast-enhanced (NCE) computed tomography (CT) scans sequentially acquired during the same imaging session, ensuring identical patient positioning and comparable anatomy. These CT scans were acquired in inspiratory apnea on a SOMATOM go.Open Pro machine (Version P83B, Siemens Healthineers AG, Erlangen, Germany). Additional information is provided in Table S1.

Patients at the Institut de Cancérologie de l’Ouest (ICO, Saint Herblain, France) received written information about the research use of their medical data. Absence of objection was verified, and data transfers complied with GDPR. Under French regulations, this research did not require a human subjects review board or special authorisation from the health authority.

2.2. Automatic contouring solutions

Only automatic solutions based on AI and available in 2024 were tested. Colleagues from various centres and developer teams were solicited to provide contours from their solution based on the anonymized dataset. Various solutions were tested: 6 commercial solutions (Limbus contour, Version 1.8, Limbus AI Inc, Regina, Saskatchewan, Canada; GBS/Contour+, Version 1.2, MVision AI Inc, San Antonio, Texas, USA; AutoContour, Version 2.4, RADformation Inc, New York City, New York, USA; Syngo.Via/AI-Rad Companion Organs RT, Version VB80, Siemens Healthineers AG, Erlangen, Germany; Mediq, Version 1.1, Synaptiq Technologies SRL, Cluj-Napoca, Romania; ART-Plan/Annotate, Version 2.2, TheraPanacea SAS, Paris, France), 1 commercial solution with cardiac substructures partially distributed (RayStation, Version 12A Research, RaySearch Laboratories AG [11]), 1 in-house solution nicknamed MDANDERSON [12], 3 open-source solutions (PlatiPy, Version 0.6.1 [13], being hybrid – AI for the whole heart only; TotalSegmentator, Version 2.0.4 [14]; STOPSTORM, Version 3, Model C (605) and Model B (603) [15]).

Manual delineation was not considered in this study as the focus was on illustrating inter-solution variability. Hereafter, the abbreviated name of the developer team is used for the commercial solutions, and the software name or nickname is used for the open-source and in-house solutions. Available information regarding the training datasets is provided in Table S2. Although some solutions were not trained on both CE-CT and NCE-CT scans, investigations were done for research purposes.

Table 1 summarizes the cardiac substructures available per solution according to the version number. The cardiac substructures’ names were formatted to conform to the AAPM TG-263 guidelines (A = Artery, V = Vessel, S = Superior, I = Inferior) with the exception of the Left Main (LM) and Left Anterior Descending (LAD) coronary arteries. These coronary arteries were grouped under the A_LM_LAD name as some solutions do not discriminate them.

Table 1.

Cardiac substructures available per artificial intelligence contouring solution. The version number must be considered since modifications of models regularly occur.

Developer team	LIMBUS AI Inc	MVISION AI Inc	RADFOR-MATION Inc	RAYSEARCH LABORATORIES AG	SIEMENS HEALTHINEERS AG	SYNAPTIQ TECHNOLOGIES SRL	THERAPA-NACEA SAS	Wasserthal et al, 2023	Finnegan et al, 2023	van der Pol et al, 2025	Chen et al, 2024
Software	Limbus Contour	GBS / Contour+	AutoContour	RayStation	Syngo.Via / AI-Rad Companion Organs RT	Mediq	ART-Plan / Annotate	Total Segmentator	PlatiPy	STOPSTORMa	MDANDERSONa
Version	1.8.0 B3	1.2.5	2.4.4	12A Research	VB80	1.1.0	2.2	2.0.4	0.6.1	3 Model C + Model Bb
Heart	X	X	X	X	X	X	X	Xc	X		X
Heart_APulm	X	X		X				c
Atrium_L	X	X		X	X	X	X	X	X	X	X
Atrium_R	X	X		X	X	X	X	X	X	X	X
Ventricle_L	X	X		X	X	X	X	X	X	X	X
Ventricle_R	X	X		X	X	X	X	X	X	X	X
A_Aorta	X	X	X	X	X	X	X	X	X	X	X
A_Pulmonary	X	X	X	X	X	X	X	X	X	X	X
V_Pulmonary				X				X			X
V_Venacava_I		X	X	X	X	X	X	X		X	X
V_Venacava_S	X	X	X	X	X		X	X	X	X	X
A_Coronary_L	Xd	Xd	Xd	X	Xd		X	Xd	X	X	X
A_LAD	Xd	Xd	Xd	X	Xd		X	Xd	X	X	X
A_Cflx				X				Xd	X	X	X
A_Coronary_R				X				Xd	X	X	X
Valve_Mitral				X					X	X	X
Valve_Tricuspid									X	X	X
Valve_Aortic									X	X	X
Valve_Pulmonic									X	X	X
CN_Sinoatrial									X
CN_Atrioventricular									X
Ventricle_L_Myo				X	X	X	X
Pericardium	Xe		X				X

^aAttributed name for the study.

^bValves, coronary arteries.

^cHeart corresponding to the union of the heart chambers and the pulmonary artery, but not contoured as a whole: could have been named Heart_APulm.

^dInclude in the same structure.

^ePericardium being the union of the Heart and the coronary arteries / Pericardium_APulm being the union of the Heart_APulm and the coronary arteries.

For the A_Aorta, the V_Venacava_S and the V_Venacava_I, it was not intended to capture the differences in terms of external extent outside the cardiac area. For example, the A_Aorta can be delineated from the heart to the aortic arch or also include the descending aorta depending on the solution; the V_Venacava_I can be delineated with varying extents in the abdominal region depending on the CT extent of the training data. Thus, the contours were limited to the same external extent before analysis (same superior limit for the A_Aorta and V_Venacava_S, and same inferior limit for the V_Venacava_I). For the A_Aorta, only the ascending part was considered. Highly fragmented contours were excluded from the analysis as they hindered the metric computation: this concerned only the coronary artery network from TOTALSEGMENTATOR.

2.3. Metrics

The first comparison was performed on apnea CE-CT scans, where cardiac substructures are most easily distinguished by humans. The metrics of interest were the Dice Similarity Coefficient (DSC) and the 95th percentile of Hausdorff distance (HD95). They are complementary, especially for small structures, as the DSC provides information on the structure overlap and the HD95 provides a “near-maximum” distance. Additionally, the surface Dice Similarity Coefficient at 3 mm (sDSC_3mm) was computed. A Python script within RayStation Treatment Planning System (Version 12A, RaySearch Laboratories), using built-in functions and under property of RaySearch Laboratories, was used to compute them. The packages and version used with Python 3.8.7 were scipy 1.9.3, numpy 1.19.5, panda 0.3.1 and openpyxl 3.1.3. For each metric, the median was computed based on the 20 CE-CT scans and the inter-solution variability of contours was assessed by averaging these medians across the various pairs of solutions (excluding the self-comparison).

The second comparison was conducted between the apnea NCE-CT scans and the apnea CE-CT scans for each solution independently. As successive acquisitions under apnea were acquired, no accurate structure positioning can be guaranteed. Thus, only volume ratios (VR) were computed using the formula (1) prior to assessing the medians from the 20 paired CT scans. For this analysis, raw contours without any cut of the great vessels were considered, as it is a comparison solution-per-solution.

$${\mathit{VR}}_{\mathit{structure},solution}\left(\%\right)=\frac{\mathit{Volume}\mathit{on}\mathit{NCE}{\mathit{CT}}_{\mathit{structure},solution}-\mathit{Volume}\mathit{on}{\mathit{CECT}}_{\mathit{structure},solution}}{\mathit{Volume}on{\mathit{CECT}}_{\mathit{structure},solution}}\times 100$$ (1)

3. Results

Significant qualitative differences in contouring of the whole heart and cardiac substructures between solutions were observed (Fig. 1, Fig. 2; Figs. S1-S14). The range for the average median DSC and HD95 across pairs of solutions varied widely depending on the solution and the cardiac substructure, spanning from 0.03 to 0.93 for DSC and from 5 mm to 70 mm for HD95 (Fig. 3; Figs. S15 and S16; surface Dice Similarity Coefficient at 3 mm (sDSC_3mm) on Fig. S17). For the whole heart, the cardiac chambers and the great vessels, the average median DSC was above 0.8 for 55 of the 123 structure-solution pairs (45%), and the average median HD95 was below 10 mm for 47 of the 123 structure-solution pairs (38%). For the coronary arteries, the average median DSC ranged between 0.03 and 0.50 and the average median HD95 ranged between 19 mm and 70 mm.

Fig. 1.

Illustration of differences on adopted contouring rules depending on the artificial intelligence contouring solution (A= Heart, B = Heart_APulm, C = Atrium_L, D = Ventricle_L).

Fig. 2.

Illustration of contouring variability depending on the artificial intelligence contouring solution (A = A_Aorta, B = Atrium_R, C = A_LM_LAD).

Fig. 3.

Heatmap of the average (i.e. mean) of median DSC and HD95 across all pairs of artificial intelligence contouring solutions for each cardiac substructure (twenty contrast-enhanced CT scans with lung cancer).

For the whole heart (Heart and Heart_APulm), considerable variability was observed at the upper limit of the heart, the heart apex, and the separation with great vessels (in particular, inferior vena cava and pulmonary veins). Regarding the upper limit of the heart, one solution (RAYSEARCH) contoured a few additional slices compared to the other solutions, as exhibited by the average median HD95 of 21 mm. Another solution (TOTALSEGMENTATOR) stood out from the other solutions, with an average median DSC of 0.75 and HD95 of 29 mm, due to its delineation limited to the cardiac chambers and the pulmonary artery.

For the cardiac chambers, differences in definitions were focused on the left ventricle and the left atrium. For the left ventricle, two solutions (SYNAPTIQ and TOTALSEGMENTATOR) excluded the myocardium, resulting in an average median DSC of 0.69 and 0.66 respectively. For the left atrium, one solution (RAYSEARCH) applied an axial limitation between the left atrium and the pulmonary veins, resulting in an average median DSC of 0.65 and HD95 of 20 mm. Two solutions (MDANDERSON and SIEMENS) had larger average median HD95 than the other solutions for the left ventricle and the right ventricle respectively (17 mm and 15 mm respectively), with globally smaller axial contour size.

For the great vessels, the main contour variabilities corresponded to the transition between the vena cavae and the right atrium, the pulmonary artery and the right ventricle, and the pulmonary veins and the left atrium. For the aorta, three solutions (PLATIPY, RADFORMATION, THERAPANACEA) had average median DSC and HD95 very different from the other solutions, with values between 0.45–0.63 and 27–39 mm respectively. The differences were observed in the vicinity of the aortic valve. An exclusion of the aortic root from the aorta was observed for one solution (RADFORMATION), without being separately contoured. For the pulmonary artery, one solution (PLATIPY) differed markedly from the others (average median DSC and HD95 of 0.18 and 45 mm respectively), with a significantly shorter cranio-caudal extent. All three solutions contouring the pulmonary veins were highly different (average median DSC and HD95 below 0.4 and close to 30 mm respectively), especially one (RAYSEARCH) joining the left and right branches through the left atrium. For the superior vena cava, two solutions (PLATIPY and THERAPANACEA) were highly different from the others (average median DSC and HD95 of 0.17–0.34 and 28–37 mm), with a significantly shorter cranio-caudal extent. For the inferior vena cava, all the solutions exhibited a good agreement, except one (RADFORMATION), which had significantly shorter cranio-caudal and axial extents and also missed the structure in multiple cases.

For the coronary arteries, differences in the axial size of the contours were observed, with larger contours adopted by two solutions (RAYSEARCH and MDANDERSON). In some cases, discrepancies arose due to both the anatomical location of the coronary artery and their axial size, as reflected by some very low average median DSC (0.06 and 0.24) for the A_LM_LAD of two solutions (PLATIPY and STOPSTORM). However, differences also stemmed from variations in the cranio-caudal extent of the contoured coronary arteries, as reflected by some average median HD95 (32 mm and 53 mm), especially for two solutions (MDANDERSON and STOPSTORM). For the A_Coronary_R and the A_Cflx, both the DSC and the HD95 (between 0.03 and 0.16 and between 29 mm and 70 mm, respectively) highlighted the differences between the solutions contouring them (MDANDERSON, PLATIPY, RAYSEARCH and STOPSTORM), with differences in axial size and length for the coronary contours. For the A_Coronary_R, one solution (RAYSEARCH) had a larger drawing of the proximal part and extended it to the heart apex between the two ventricles. For the A_Cflx, one solution (RAYSEARCH) considered the multiple marginal branches into the contouring rule by drawing a coronary artery wall over the left ventricle wall, which differed significantly from the other solutions.

For the mitral valve, the differences essentially arose from the axial size of the structure, resulting in an average median DSC (around 0.45) and HD95 (around 10 mm).

The heatmap of median volume ratios (Fig. 4 and Fig. S18) highlighted the contour variability due to the presence/absence of contrast enhancement between solutions and across structures, with values ranging from −41% to 122%. One solution (MDANDERSON) appeared particularly sensitive to the contrast changes for the Ventricle_L/R, the V_Pulmonary, the coronary arteries, and the mitral valve. The contour review revealed that the Ventricle_L/R sometimes lacked portions of their cavities on the CE-CT scans, and the extent of the coronaries varied between the CE- and NCE-CT scans. For the other solutions, only a few substructures were sensitive to this contrast change: one (RADFORMATION) with the A_Pulmonary and the A_LM_LAD, a second one (THERAPANACEA) with the A_Aorta, a third one (LIMBUS) with the Atrium_R. For the A_Aorta (THERAPANACEA), the contour review revealed that portions of the aortic arch were missing, as well as variable portions of the ascending and/or descending aorta. Although other differences were less pronounced for most of the structure-solution couples, several still fell within the 10–20% range.

Fig. 4.

Heatmap of the median volume ratio (%) from the twenty pairs of apnea non-contrast-enhanced / contrast-enhanced CT scans for each cardiac substructure depending on the artificial intelligence contouring solution.

4. Discussion

Different automated solutions were applied to the same CT scans to evaluate their ability to delineate multiple cardiac substructures. To delineate multiple cardiac substructures. First, our analysis revealed significant inter-solution variabilities, especially for the coronary arteries. Second, the assessment of the contrast-enhancement impact on the contours showed that except for one solution and a few structures in three others, most solutions exhibited low sensitivity to the contrast variation.

Contour variabilities between solutions might have stemmed from differences in the training contouring rules and the intrinsic model properties (e.g., clinical data limited to certain indications, imaging characteristics, contour variability due to limited contrast at transitions between cardiac substructures, algorithmic differences). For most solutions, the inter-solution variability was comparable in magnitude to the inter-observer variability quantified in previous studies [15], [16]. When substantial differences were observed between NCE- and CE-CT scans, it was not always possible to determine superiority without medical review. As expected, the solution trained exclusively on NCE-CT scans performed better on NCE-CT scans, highlighting the importance of consistency between training and clinical data.

This study did not aim to rank solutions for cardiac substructure contouring, but rather to provide feedback on the observed variations, which could guide future model improvements depending on the model purpose. Based on a limited dataset for lung treatments from a single centre and CT scanner, the findings are not generalisable but remain informative. Some solutions were occasionally used outside their original training scope (e.g. at least one solution exclusively trained on NCE-CT scans, at least two solutions trained on CT scans with a different slice thickness), which may have contributed to the observed variability.

The impact of the observed contour variations on the dose reporting was not assessed in this study. A more comprehensive evaluation would require access to consensus contours, such as STAPLE contours [17], generated by a consortium of experts. This would enable a meaningful comparison between dose distributions derived from raw contours and those based on expert-reviewed contours. Also, such investigation would require a comprehensive analysis across a wide range of configurations (e.g., target location relative to the heart, treatment technique, and dose gradient), warranting a dedicated study. Moreover, evaluating impact on dose reporting for cardiac substructures remains challenging due to the lack of consensus on relevant dose metrics, despite emerging evidence [18], [19]. Existing literature on this topic is often limited to one specific cancer type and one treatment technique, such as left breast cancer treated with tangential fields [20] or motion uncertainty impact in lung cancer [10]. For the moment, these findings cannot be easily generalised to other clinical scenarios. However, they remain of interest, as automatic contouring could potentially be used without medical review to enable the study of dose–effect relationships in large cohorts.

Several guidelines and atlases have been developed to standardize the delineation of cardiac substructures, for both clinical and research applications [16], [21], [22], [23], [24], [25]. Dedicated contouring recommendations for AI models of such substructures have also been proposed [11]. For the whole heart, the Global Harmonization Group [26] reviewed six existing definitions and retained two – Heart and Heart_APulm – as reference standards. The extension of their work to the cardiac substructures would be highly valuable to the radiotherapy community. For coronary arteries, cardiac valves and conduction nodes, surrogate volumes are recommended due to uncertainties (limited visibility, motion artifacts, poor interobserver reproducibility) [1], [11], [27], [28]. Such volumes, also referenced as a Planning organ at Risk Volume-like (PRV) definition, can potentially complicate modelling, as contours are not based on a grey-level transition.

Further consensus is required regarding clinically relevant substructures and their subdivision [16], [20], [23], depending on the treatment indication (e.g. long-term follow-up of breast cancers, mid-term follow-up of lung cancers treated by stereotactic body radiation therapy, short-term follow-up of stereotactic arrhythmia radioablation). The coronary arteries could be divided into up to 10 segments [23]. The left ventricular myocardium could be divided into 4–5 segments [16], [23] or 17 segments [13]).

Additionally, a consensus is needed regarding the imaging used for delineating cardiac substructures. While NCE-CT remains standard for radiotherapy planning, with CE-CT occasionally used for better target visualisation, the role of CE-CT dedicated to cardiac substructures, registered to the planning NCE-CT, should be evaluated, despite the inherent uncertainties of registrations. This also raises questions about AI models development – whether to use a single model for all situations or multiple specialised models. Based on the training information collected, some solutions aimed to provide models suitable for both NCE- and CE-CT scans, whereas others focused exclusively on a specific type of CT scan. In the future, dedicated CE-CT for more reliable and evaluable cardiac substructure contouring could be recommended, especially if certain dose–effect relationships become widely accepted.

Greater transparency from developer teams – especially commercial ones – regarding training datasets and validation processes is essential for clinicians and researchers, as outlined by an ESTRO Working Group on AI in Radiation Therapy and now supported by the ESTRO AI Focus Group. They proposed a “Model Card” [29] to outline a comprehensive set of information that should be disclosed.

To conclude, this study illustrated and quantified inter-solution variability in automatic cardiac substructure contouring. Sensitivity to contrast change was generally low, suggesting robustness for most structure-solution pairs. While AI tools reduce intra-centre variability and save time in delineating all cardiac substructures [30], [31] thereby facilitating the investigation of dose–effect relationships for large cohorts, consensus guidelines for standardising and qualifying the results from the automatic models are urgently needed. It is to be hoped that this study will contribute to the harmonisation and increased transparency of AI models for cardiac substructures. Based on the study findings, an ESTRO Working Group on cardiac substructures in radiation therapy has been initiated to review current practices and work toward expert consensus. In an optimal scenario, joint efforts to provide expert-labelled datasets – validated by a multicentre panel and tailored to each specific population, with continuous adaptation as imaging evolves – would enable all developer teams to build upon a common foundation.

CRediT authorship contribution statement

Alexandra Moignier: Writing – original draft, Supervision, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Tanguy Perennec: Writing – review & editing, Investigation. Elise Prangères: Formal analysis, Data curation. Bastien Bernard: Resources. Angela Botticella: Resources. Xinru Chen: Resources. Robert Finnegan: Resources. Sandrine Huger: Writing – review & editing, Resources. Anna Karlhede: Resources. Thomas Lacornerie: Writing – review & editing, Resources. Fredrik Löfman: Resources. Jérémy Palisson: Writing – review & editing, Resources. Charlotte Robert: Resources. Killian Sambourg: Resources. Jonas Söderberg: Resources. Remus Stoica: Resources. Grégory Delpon: Writing – review & editing, Investigation. Elvire Martin-Mervoyer: Writing – review & editing, Investigation. François Thillays: Writing – review & editing, Investigation. Loïg Vaugier: Writing – original draft, Supervision, Methodology, Investigation.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Loïg Vaugier and Alexandra Moignier have a partnership with RaySearch Laboratories for the detailed heart model development. Angela Boticella has partnership with Therapanacea for the detailed heart model development. Some co-authors are employees of commercial companies as stated in the affiliations.

Appendix A. Supplementary data

The following are the Supplementary data to this article: