Impact of reconstruction kernel variability on segmentation consistency in low-dose thoracic CT

Abstract

Structural thoracic segmentation with computed tomography (CT) has become a ubiquitous step in medical image processing. Several tools—including TotalSegmentator—now provide efficient, open methods for labeling regions of interest (ROI) in CT. However, variability in CT reconstruction kernels, especially between soft and hard kernels, can affect image characteristics and potentially impact automatic segmentation, downstream analysis, and clinical decisions. Yet, this variability and its impact on downstream analysis have not been fully investigated in a systematic way. To address this, we examine the consistency of thoracic region segmentations produced by TotalSegmentator when applied to 9,529 CT images reconstructed with paired soft and hard kernels from the National Lung Screening Trial (NLST) dataset. It includes 9,529 CT scan pairs from 9,519 subjects (5,733 men, 3,796 women; ages 43–74), with each subject contributing one scan session, including data from multiple vendors (Siemens, Philips and GE Medical Systems) and reconstruction kernels, drawn from the Lung Screening Study (LSS) (n=6,794), American College of Radiology Imaging Network (ACRIN) with biomarkers (n=1,488), and ACRIN without biomarkers (n=1,247), with 10 duplicated subjects treated as separate individuals. We quantified kernel-induced variability using Dice similarity coefficient, volumetric and intensity differences across 84 thoracic regions. We evaluated the relationship between the average and difference of volumes, Hounsfield Units (HU) and standard deviation of HU of ROI. Additionally, we assessed manufacturer-specific differences in segmentation consistency between Philips, Siemens and GE Medical Systems. Graphical representations with Bland-Altman plots show that some thoracic regions (e.g. heart) are more robust to kernel variation than others (e.g. thyroid gland), and Siemens scanners show the lowest segmentation variability between kernel types. These findings provide early insights into when kernel-induced variability affects automatic segmentation performance differently across the body in clinical and research workflows, therefore emphasizing the need for consistent CT reconstruction or kernel harmonization.

1. INTRODUCTION

Over 300 million CT scans are performed each year ¹, generating substantial amount of medical imaging data. These images originate from scanners from various manufacturers and are reconstructed using different types of kernels: soft kernels, enhancing low-contrast resolution by enhancing low-frequency components, are typically used for soft tissues and hard kernels, enhancing spatial resolution and edges by emphasizing high-frequency components, for bones and lung tissues. These different reconstruction kernels yield noticeably different images, particularly impacting intensity values and automated segmentation outputs, such as those produced by TotalSegmentator (Figure 1). Recent advances in machine learning and deep learning have enhanced the usage of CT scans beyond individual-scale usage for medical diagnostics, allowing automated processing at a population level scale. As large volumes of CT data are being processed, studies highlight the variability of CT scans. Variability in segmentation, introduced by different reconstruction kernels, can create differences in measurements.

Figure 1.

Reconstruction kernels (Standard at left and Bone at right) of the same subject show differences in intensity and volume in segmentations masks created by TotalSegmentator. The overlapping of the spleen’s segmentations of the Standard kernel (green) and Bone kernel (red) in the yellow box shows more inconsistencies than the liver (blue box) and the T1 vertebrae (pink box). Overlapping the masks from both kernels shows inconsistency in volume, underlining the influence of reconstruction kernels on segmentation.

The original TotalSegmentator paper reports a Dice similarity score of 0.943 between the segmentation prediction and the manually verified truth across 104 anatomical structures, based on 1,204 CT scans using different kernels from different manufacturers (Siemens, GE Medical Systems and Philips) ². However, the robustness of radiomic features within ROI varies with kernel choice. For example, intensity features have been found more robust compared to texture and wavelet features, and the lungs have been found more robust compared to lymph nodes. These findings were demonstrated while evaluating the robustness of 1,386 features from four relevant tissue types (non-small cell lung cancer (NSCLC), malignant pleural mesothelioma and interstitial lung disease related to systemic sclerosis) in three different lung diseases (primary tumor, largest involved lymph node ipsilateral and contralateral lung) between CT scans reconstructed from a soft and hard kernel (Standard-Lung and Standard-Detail from GE Medical Systems and B30f-I70f from Siemens) ³. Moreover, technical variability between scanners can be mistaken for biologically meaningful differences between tumors. This phenomenon was tested by analyzing radiomics features (intensity, standard deviation of intensity, busyness, coarseness, contrast, entropy, texture strength, and uniformity) extracted from the same phantom scanned on 17 different CT scans acquired using Toshiba, GE, Philips and Siemens scanners. The variability between scanners in the values of radiomics features observed in their phantom was at least as large than the variability between actual lung tumors from 20 NSCLC patients, implying that technical variability can mask or confound meaningful biological features ⁴. Furthermore, variations in tumor diameter measurements exceeding 1–2 mm can occur on immediate reimaging within a period of minutes, as shown while comparing side-by-side CT scans of the same patient with the same reconstruction kernel a few minutes apart. The study was conducted on 30 patients, with NSCLC and a lung lesion above 1 cm, totaling 90 paired measurements as the process for each patient was done individually by three professionals ⁵. Similarly, another study on temporal scanner variability, involving 32 NSCLC patients, highlighted size measurement differences in tumors of 8% or more, exceeding the computer-based measurement variability, while being clinically significant in evaluating therapy response ⁶. Given these inconsistencies, there have been recent developments to mitigate kernel differences. For example, harmonization ^7,8 can reduce inconsistencies between paired inspiratory (hard kernels) / expiratory (soft kernels) scans in emphysema quantification across different scanners and protocols, or normalization ⁹ can reduce the emphysema score between different reconstruction kernels. Although numerous studies acknowledge inconsistencies between reconstruction kernels, specific investigation of the consistency of segmentation across thoracic regions remains limited.

In this paper, we address this gap by investigating the consistency of thoracic segmentation in a large dataset of 9,529 subjects from the National Lung Screening Trial (NLST) dataset across three manufacturers and six pairs of kernels (Siemens: B30f-B50f and B30f-B80f, Philips: B-D and C-D and GE Medical Systems: Standard-Bone and Standard-Lung) (Figure 2). We focus on volume, intensity, and standard deviation of intensity as radiomic features, employing the Dice similarity score as a segmentation measure, and present our results through graphical representations and Bland-Altman plots ¹⁰.

Figure 2:

Cumulative age distribution of subjects by reconstruction kernel pair and manufacturer. The dataset includes 9,519 subjects, with the majority scanned using GE Medical Systems kernels (blue), and the fewest using Philips kernels (green). Only 10 subjects were scanned with both B_D and C_D kernel pairs from Philips. The youngest subject, scanned with the Siemens B30f_B50f kernel pair, is 43 years old (red arrow). Another subject is 54 years old (purple arrow), and all others fall within the 55–74 age range.

2. METHODS

We obtained data from the NLST Dataset where participants were randomized to obtain a chest X-Ray or a low-dose chest CT scan ¹¹. The inclusion criteria involved subjects in the age range of 55–74 years with a smoking history of at least 30 pack years ¹¹. Though in our dataset, there are two individuals outside of the specific age range (Figure 2), indicating an anomaly in the way the subjects were selected. For every participant within a given manufacturer, we consider a scan reconstructed with a hard kernel and a corresponding soft kernel by matching their acquisition parameters that include peak kilovoltage, reconstruction diameter, exposure time, tube current, exposure and spiral pitch factor. This results in “paired” reconstruction kernels across all available manufacturers. A total of 10,229 pairs were acquired out of which 309 pairs were excluded due to mismatch in acquisition parameters. We conducted quality assurance on the remaining pairs by computing the difference image between the hard and soft kernel to ensure that the images had a one-to-one correspondence. Additionally, we quantitatively verified the nature of the paired data by computing the root mean squared error (RMSE) between the hard and soft kernel images. Using the RMSE and the difference images, we excluded 391 pairs, resulting in 9,529 usable pairs (2,785: B30f_B50f, 458: B30f_B80f, 3,969: STANDARD_BONE, 1,021: STANDARD_LUNG, 1,128: C_D, 168: B_D).

Dice similarity scores between soft and hard segmentation masks and the volume of each mask for each ROI of each subject had already been computed using masks generated from TotalSegmentator. We used this list of paired soft/hard CT scans as a baseline for our dataset and further extracted mean intensity and standard deviation of intensity from both the masks and the original CT images (Figure 3).

Figure 3.

Each pair of kernels is processed using TotalSegmentator, generating 84 thoracic segmentations. For each ROI’s kernel, mean intensity, standard deviation of mean intensity, volume and shape of the segmentation are computed. Then they are analyzed using graphical representations, and Bland-Altman plots depending on ROIs and manufacturers.

If both the soft and hard kernel segmentation masks were empty for a given ROI, the corresponding computations were left undefined (NaN).

2.1. Computed Measurements

The Dice Similarity score quantifies the spatial overlap between segmentation masks obtained from soft and hard reconstruction kernels. It is defined as:

$$\text{Dice}\left(\mathit{\text{soft}},\mathit{\text{hard}}\right)=\frac{2\left|\mathit{\text{soft}}\cap \mathit{\text{hard}}\right|}{\left|\mathit{\text{soft}}\right|+\left|\mathit{\text{hard}}\right|}$$ #(1)

where $\left|\mathit{\text{soft}}\right|$ and $\left|\mathit{\text{hard}}\right|$ represent the number of voxels in each mask, and $\left|\mathit{\text{soft}}\cap \mathit{\text{hard}}\right|$ the number of voxels in their intersection. All Dice scores were rounded to three decimals.

Volumes, intensities and standard deviations of intensities were computed for each mask of each kernel, for all ROI of all subjects. Volumes for each type of kernel (soft and hard) were computed as:

$$\mathit{\text{Volume}}=N\times s_x\times s_y\times s_z$$ #(2)

where $N$ is the number of voxels in the mask, and $s_x,s_y,s_z,$ the size of a voxel in the x, y and z dimensions. Volumes are expressed in mm³ (or mL). Mean intensity was calculated as the average voxel intensity, expressed in Hounsfield Units (HU). Standard deviation of intensity was computed as:

$$\sigma =\sqrt{\frac{1}{N}\sum _{i=1}^N{\left({\text{HU}}_i-\mu \right)}^2}$$ #(3)

where $Nis$ the number of voxels in the mask, ${HU}_i$ the intensity value of the voxel $i$, and $\mu$ the mean value of intensity of the mask.

2.2. Analysis

To analyze the characteristics of the masks between kernels, we represented the measurements graphically (Figure 3). The average Dice similarity score was computed across all subjects and visualized on the segmentation of a randomly selected representative subject. The choice of the subject was arbitrary, as the focus was on the mean Dice similarity score rather than the ROI shape. Visualization employed ITK-SNAP software and a colormap to highlight spatial gradients. A joint distribution of the log-transformed volume and Dice similarity score was plotted with each ROI of all subjects. Color intensity was used to emphasize data density. Bland-Altman plots ¹⁰ were generated for three metrics (volume, intensity and standard deviation of intensity) to visualize consistency (difference of metric between soft and hard kernel segmentations) depending on the metric’s average. Analyses were performed for all ROIs and subjects, and additionally for the heart and thyroid gland of all subjects, as these regions had not been previously examined.

3. EXPERIMENTAL RESULTS

With the measurements described in the previous section, we evaluated segmentation performance across ROI, spatial location, characteristics, reconstruction kernels, and manufacturers. Our analysis consisted of three approaches: spatial consistency, consistency based on ROI characteristics, and consistency across manufacturers.

3.1. Consistency by spatial location

To assess the consistency of ROIs based on anatomical location, we visualized the average Dice similarity score for each ROIs, computed across all subjects, and mapped these values onto the segmentation of a randomly selected CT scan (Figure 4). The least consistent ROIs (red) were the portal and splenic veins (0.58337), the sacrum (0.63466), the common right artery (0.70721), the left adrenal gland (0.73392), and the gallbladder (0.736117). These regions are all predominantly located in the lower abdomen, except for the common right artery. The most consistent ROI (white and light yellow) were the lungs (all five parts above 0.99), the trachea (0.98977), the liver (0.98386), the sternum (0.98188), the left and right clavicles (0.97759 and 0.97669), all located in the lungs area. These results suggest that ROI’s consistency is correlated with anatomical location: higher consistent areas are in and around the lungs, and less consistent areas are in the lower thoracic region.

Figure 4:

The axial, sagittal and coronal views of a segmentation of ROI colored by the value of the average Dice coefficient of all subjects between hard and soft kernels show that higher-performing regions are the lungs (white). Lower-performing regions (red) are mostly found around smaller or more complex structures. This heatmap underlines variability in segmentation quality across anatomy.

3.2. Consistency by ROI’s characteristics

To analyze the consistency of characteristics (volume, mean intensity, standard deviation of intensity) between soft and hard kernels, we used Bland-Altman plots, complemented by a joint distribution analysis for the volume. The first characteristic studied was the volume. First, we examined the relationship between log-transformed volumes of ROIs and their Dice similarity score across all subjects using a joint distribution plot (Figure 5). Most ROIs exhibited a log volume between 8 and 12, corresponding to e⁸ =2.980 cm³ and e¹² =162.75 cm³, and Dice similarity scores above 0.8. Barely any subjects with log volumes greater than 12, corresponding to e¹² =162.75 cm³ showed a Dice similarity score below 0.8. This suggests a relationship between the volume of an ROI and its consistency between soft and hard kernels: ROIs with a high volume above 162.75cm³ almost always have a Dice above 0.8. Second, we examined ROIs volumes’ consistency between soft and hard kernels using a Bland-Altman plot across all ROI and subjects (Figure 6). Most of the points were negative, indicating that segmentation volumes of hard kernels are generally larger than soft kernels. Consistency remained stable below approximately 1,500 cm³ but declined with increasing volumes. This reveals that volumes of segmentations are larger when produced by hard kernels and tend to be less consistent for ROIs with larger mean volumes.

Figure 5.

The Dice score between soft and hard kernels for all ROI per their log volume for all subjects indicates that the highest concentration of subjects has a Dice close to 1. Above a log volume of 12, corresponding to e¹² = 162 cm³, all the ROI have a Dice score above 0.8.

Figure 6.

Bland-Altman plot of the mean of volume by difference in volume of ROI between soft and hard kernels for all subjects across all ROI and manufacturers shows variation in the consistency of volume across ROI volumes and manufacturers. Below a certain volume near 2,000 cm³, the differences in the volumes between soft and hard kernels are close to 0. Above that point, the differences in volume increase with the mean volume. The manufacturer Siemens has the most subjects with a small difference in volume and GE Medical Systems has the largest.

The second characteristic analyzed was the mean intensity of ROIs, studied via a Bland-Altman plot across all ROI and subjects (Figure 7). Most differences were positive, indicating segmentation of soft kernels produce greater intensities than hard kernels. The plot revealed varying consistency levels depending on mean intensity ranges:

• From −1,000 to −500 HU: decreasing consistency.

• From −500 to 50 HU: increasing consistency. All values seem to be above a decreasing straight line.

• Around 50 HU: consistency is stable within 50 HU of absolute difference between kernels.

• Above 50 HU: consistency is stable within 200 HU of absolute difference between kernels.

Figure 7.

Bland-Altman plot of the mean of intensity by difference in mean intensity of ROI between soft and hard kernels for all subjects across all ROI and manufacturers shows variation in the consistency of mean intensity across ROI mean intensity and manufacturers. The repartition is asymmetrical. Near the mean intensity of zero, the differences in the mean intensity between soft and hard kernels are the lowest. The Siemens manufacturer has the most subjects with a small difference of mean intensity and GE Medical Systems has the largest.

Therefore, the consistency of mean intensities of ROIs depends on the mean intensity of the segmentation mask. They are given higher values in soft kernels, and their consistency is decreasing between −1,000 and −500 HU, increasing between −500 and 50 HU and rather stable above 50 HU.

The third characteristic considered was the standard deviation of intensity, analyzed via a Bland-Altman plot across all ROI and subjects (Figure 8). Most differences were negative, indicating greater intensity variability within hard kernel segmentations. Consistency decreased as mean standard deviation in intensity increased to around 700 HU. The lowest differences appear to be following a straight decreasing line along the red arrow. Above 700 HU, the differences stabilized around zero, suggesting better consistency. This downward trend suggests a loss in consistency with increasing variability in intensity, until the standard deviation of intensity reaches 700 HU.

Figure 8.

This Bland-Altman plot of the mean of standard deviation of intensity by the difference in mean intensity of ROI between soft and hard kernels for all subjects across all ROI and manufacturers shows variation in the consistency of standard deviation in intensity across ROI’s mean standard deviation and manufacturers. The differences are mainly negative, indicating that hard kernels have a higher standard deviation in intensity than soft kernels. The red arrow highlights a negative trend, suggesting that the difference in intensity variability decreases as the overall variability increases. GE has a broader spectrum than Philips and Siemens.

Overall, the consistency of reconstruction kernels on segmentation strongly depends on the characteristics of the ROIs: volume, intensity, and standard deviation of intensity.

3.3. Consistency by manufacturer

To analyze segmentation consistency between manufacturer (Siemens, GE Medical Systems and Philips), we re-examined our Bland-Altman plots for ROI characteristics, using manufacturer-specific colors (GE Medical Systems: dark blue, Philips: magenta, Siemens: light blue). We assessed the consistency between manufacturers across characteristics of ROIs: volume, mean intensity, and standard deviation of intensity.

To begin with, we investigated the manufacturers’ volume consistency on a Bland-Altman plot across all ROIs and subjects (Figure 6). All manufacturers overlapped and followed the same trend of increasing inconsistencies with the mean volume increasing. Siemens had the narrowest spread of points, followed by Philips then GE. This suggests that Siemens demonstrates the highest consistency in mean intensity measurements, followed by Philips and then GE. Next, we analyzed the manufacturers’ intensity consistency on a Bland-Altman plot across all ROIs and subjects (Figure 7). All three manufacturers followed the same trends with different intensity. Overall, Siemens had the greatest number of differences close to zero for all mean intensities, followed by Philips then GE. Once again, this suggests that the consistency in the intensity of Siemens is better than Philips, and Philips more than GE. Finally, we examined the manufacturers’ standard deviation of intensity consistency on a Bland-Altman plot across all ROIs and subjects (Figure 8). All three manufacturers follow the same decreasing trend. The values of Philips and Siemens appeared to be separated into several distinct and decreasing scatters. The wideness of the spread of GE was as large as Siemens and Philips combined. Siemens and GE differences seemed to be strictly above a decreasing line indicated by the red arrow. Above 700 HU, all three manufacturers stay around 0. In this case, no manufacturer clearly outperformed the others.

Overall, all three manufacturers overlapped and followed the same global trends. Siemens outperformed Philips and GE concerning volume and intensity, whereas no manufacturer showed best performance regarding standard deviation of intensity.

4. SENSITIVITY

To validate our findings conducted over 84 ROIs, we performed a focused sensitivity analysis on two selected ROIs: the heart and the thyroid gland. For those two ROIs, we generated specific Bland-Altman plots of volume, mean intensity and standard deviation of intensity across all subjects (Figure 9: (a) heart, (b) thyroid gland) and analyzed the subplot focusing on ROIs characteristics then manufacturers.

Figure 9.

Comparison of Bland-Altman plots of mean intensity, standard deviation of mean intensity and volume between the heart (a) and the thyroid gland (b) for all subjects by manufacturer. The heart seems more consistent than the thyroid gland in terms of mean volume and mean intensity, and less in terms of standard deviation of intensity.

4.1. Sensitivity of consistency by ROI’s characteristics

The heart’s mean volume ranged between 250 – 900 cm³, with volume differences between soft and hard kernel mostly within ± 25 cm³. In contrast, the thyroid gland’s mean volume was roughly around 0–25 cm³, with volume differences ranging from −10 to 2 cm³. The heart, showing more values within its limits of agreement, was more consistent than the thyroid. This supports earlier findings that ROIs with volumes above 162 cm³ exhibit more consistency. Also, the heart had a larger volume average spread and showed bigger differences in volume than the thyroid gland, supporting the findings that differences in volume increases with mean volume. The thyroid gland seems less consistent than the heart as more of its values are below its lower limit of agreement. The heart’s mean intensity ranged roughly from 200 to 550 HU, with a difference in mean intensity from −75 to 25 HU. The thyroid gland’s mean intensity was mainly around 50 – 150 HU, with a difference in intensity ranging from −50 to 50 HU. Both ROIs’ differences in intensity are about the same, aligning with previous results, as both mean intensities are above 50 HU, where consistency is rather stable. The thyroid gland seems less consistent in intensity than the heart as more of its values are below its lower limit of agreement. The standard deviation of the heart ranged around 20 – 450 HU and its difference between −10 and −300 HU. The standard deviation of the thyroid gland ranges roughly around 20 – 250 HU and its difference between −10 and −250 HU. Both are negative and decreasing, with the heart spanning a broader range in both mean and difference, supporting previous findings. The heart exhibits more inconsistencies in standard deviation of intensity than the thyroid as a large portion of its values is below the lower limit of agreement. In summary, all findings with individual analysis of the heart and thyroid gland align with the previous findings with all ROIs. The heart shows more consistency than the thyroid gland for intensity and volume characteristics.

4.2. Sensitivity of consistency by manufacturers

Across all plots, the manufacturers generally follow the same trends except for intensity and standard deviation in intensity of the heart. For both ROIs, Siemens (light red and light green) demonstrated the highest consistency in volume between kernels. Philips (blue and yellow) and GE (dark red and dark green) are more spread out. For the heart’s intensity, Siemens (light red) is separated into two parts: one above the bias, overlapping with GE (blue) and another under the bias overlapping with Philips (dark red). For the thyroid’s intensity, Siemens (light green) showed the most consistency; GE (dark green) was above the bias and Philips (yellow) under the bias. For both ROIs, Siemens exhibits the best consistency in intensity; Philips is below the bias, and GE is above the bias. Each manufacturer showed two distinct regions for the heart’s standard deviation of intensity. One part of each manufacturer overlapped near the bias around 150 – 400 HU, some of Siemens and Philips overlapped near the bias around 50 HU, and Siemens and GE follow a similar decreasing trend around 200 – 450 HU. Philips seems to have the most consistency across all means of standard deviation in intensity. For the thyroid gland, all manufacturers follow the same decreasing trend, and none seems to have higher consistency. Siemens showed more consistency in volume and intensity, according to the previous findings. The variability within manufacturers is higher for the heart’s intensity and standard deviation of intensity.

5. CONCLUSIONS

This study systematically examined the variability in thoracic CT segmentation induced by different reconstruction kernels across three manufacturers. We demonstrated that segmentation consistency depends on anatomical location, ROI characteristics, and scanner manufacturer. Consistency was higher around the lungs, and lower in the lower abdominal regions. Larger in volume ROIs achieved high Dice similarity score above 0.8, also exhibiting greater absolute differences in segmentation volume between soft and hard kernels. Segmentation consistency was stable above 50 HU, and consistency in intensity variation decreased with intensity variation. Hard kernels tended to generate larger volumes, lower intensity values, and higher intra-region intensity variability. Manufacturer-wise, Siemens scanners produced more consistent segmentations in terms of volume and intensity while no manufacturer clearly outperformed the others for variability in intensity. Our sensitivity analysis on the heart and thyroid gland confirmed that these results are verified at individual ROI level, with the heart demonstrating greater segmentation stability for consistency in volume and intensity. We reaffirmed prior findings that segmentations of lung tissues are robust to kernel variation. Our results highlighted coherent effects: differences in standard deviation of intensity decrease with increasing standard deviation, implying more variability when the variability increases. However, some results concerning volume seemed contradictory: ROIs with a large volume have a high Dice similarity score but higher inconsistencies. This phenomenon likely arises from the fact that larger volumes have larger variations that are proportionally less important. Interestingly, the scatters of manufacturers sometime appeared to cluster into two distinct groups. A further analysis by specific kernel pair could maybe help understand those clusters. Overall, our study extended previous works by analyzing 84 ROI in the thoracic region and then the heart and thyroid gland of 9,529 subjects with paired soft/hard reconstructed CT scans, showed consistency tendencies across characteristics of ROI and showed some ROI segmentations are more consistent between different kernel reconstruction than others. These findings underline the necessity of accounting for reconstruction kernel effects.