Accuracy of patient-specific organ dose estimates obtained using an automated image segmentation algorithm

Abstract.

The overall goal of this work is to develop a rapid, accurate, and automated software tool to estimate patient-specific organ doses from computed tomography (CT) scans using simulations to generate dose maps combined with automated segmentation algorithms. This work quantified the accuracy of organ dose estimates obtained by an automated segmentation algorithm. We hypothesized that the autosegmentation algorithm is sufficiently accurate to provide organ dose estimates, since small errors delineating organ boundaries will have minimal effect when computing mean organ dose. A leave-one-out validation study of the automated algorithm was performed with 20 head-neck CT scans expertly segmented into nine regions. Mean organ doses of the automatically and expertly segmented regions were computed from Monte Carlo-generated dose maps and compared. The automated segmentation algorithm estimated the mean organ dose to be within 10% of the expert segmentation for regions other than the spinal canal, with the median error for each organ region below 2%. In the spinal canal region, the median error was $-7\%$, with a maximum absolute error of 28% for the single-atlas approach and 11% for the multiatlas approach. The results demonstrate that the automated segmentation algorithm can provide accurate organ dose estimates despite some segmentation errors.

1. Introduction

Radiation dose due to computed tomography (CT) scans is a growing public health concern.^1,2 Several states have mandated the reporting of CT radiation dose, and these dose-reporting efforts have had positive effects on protocol standardization.³ However, the dose metrics used in these reports, CT dose index (CTDI) and dose-length product, represent the dose to a plastic cylinder and were originally designed for scanner quality assurance, not for quantifying patient dose.⁴ For example, a recent study found more than 100% variation between different methods of estimating the effective dose of pediatric CT scans and currently reported dose metrics.⁵

Numerous approaches have been studied for patient-specific organ dose estimation.^6–21 Although approximate methods based on CTDI conversion factors or phantom library models are available, they may not be able to model all the patient and scan-specific factors that affect organ dose such as tube voltage (kV), tube current settings and modulation, beam collimation, beam shaping filters, patient centering, gantry start angle for helical scans, and patient-specific anatomy.

The current gold standard for accurate organ dose estimation is to create a three-dimensional (3-D) dose map for an individual scan by performing Monte Carlo simulations of x-ray transport through the patient.^6–8,19 Inputs to the program are the (previously) reconstructed images, where Hounsfield units have been converted to physiologic materials such as tissue and bone, and a CT scanner model describing the incident x-ray fluence profile. Manual/semiautomated CT image segmentation is then performed to delineate organ boundaries thus allowing for computation of individual organ doses from the dose maps. The manual input required for segmentation is a potential issue for routine organ dose estimation and is the subject of this study.

The overall goal of this work is to develop a rapid, accurate, and fully automated software tool to estimate patient-specific organ doses from CT scans. The tool will (1) rapidly compute spatially dependent dose distributions (dose maps) and (2) automatically segment the CT images to determine specific organ boundaries thus allowing for computation of individual organ doses from the dose maps. In a separate study, we investigated the use of a deterministic Boltzmann transport equation (BTE) solver (Acuros CT-Dose, Varian Medical Systems. Acuros CT-Dose is currently a research tool and not available for commercial use) to rapidly compute the dose maps.²² Alternatively, many groups have investigated graphics processing unit-based Monte Carlo simulation for accelerated computation of dose maps.^23,24 This study investigates the feasibility of the second step of the process—automated segmentation—as required to compute individual organ doses.

Automated CT segmentation algorithms are commercially available and are often used to assist radiation therapy treatment planning. Accurate delineation of organ boundaries is important for the task of treatment planning; therefore, manual editing is generally required to adjust the output organ contours after automatic segmentation. For the task of CT organ dose estimation, the relevant outputs are organ dose metrics rather than the organ contours. We hypothesize that accurate CT organ dose estimation may be possible despite some segmentation errors, as small errors at the organ boundary may have a minimal effect since dose (mGy) is the deposited energy normalized by the mass. The purpose of this study is to quantify the performance of a commercial segmentation algorithm applied to automated organ dose estimation. Methods for robustly estimating organ dose from the segmentation outputs are also investigated.

2. Methods and Materials

The organ dose estimation method investigated in this work is illustrated in Fig. 1. This study quantified the effects of the automatic segmentation algorithm on organ dose estimates, with dose maps generated through gold standard Monte Carlo approaches. Parallel efforts are under way to validate the BTE solver dose estimates.²²

Fig. 1.

Flowchart of organ dose estimation method investigated in this work. The CT image set is automatically segmented using a commercial algorithm. The contours are then applied to the dose map, which in this work was generated through Monte Carlo simulations after assigning materials to the voxels based on CT number. The organ dose for each region was estimated as the mean dose within the segmented organ contours.

2.1. Autosegmentation Algorithm

The autosegmentation algorithm investigated in this work (Smart Segmentation Knowledge Based Contouring^® v15.0, Varian Medical Systems, Palo Alto, California, referred to as “Smart Segmentation” in this work) uses a combination of feature-based and atlas-based methods.²⁵ Some regions, such as the bones, lungs, and eyes, are automatically segmented based on image features. Remaining organs are segmented using an atlas-based approach.²⁶ The atlas contains CT image sets classified by body region (e.g., head/neck, chest, and pelvis) where the organs-of-interest (e.g., brain, parotid glands, breasts, lungs, heart, bladder, and rectum) have been manually segmented. For each new dataset to be segmented, the commercial implementation suggests an expert case based on patient age, gender, weight, exam type, and a similarity score. The similarity score for head and neck cases is calculated based on three features and is intended to be a quickly computed, approximate metric. First, a point match is performed using automatically detected landmarks, and the residual error of this point match is evaluated. The second feature uses the difference of the body outlines of the patient and expert cases after a rigid registration based on the point match is applied. Last, the ratio of the total low-density volume to the total body volume is compared. These features take into account the registration algorithm, and the similarity score attempts to predict the similarity after the deformable registration is performed.

The expert atlas case is deformably registered to the current case. The registration algorithm uses a variation of the Demons algorithm with a multiresolution pyramid to increase convergence speed and improve robustness to image distortions.²⁶ The resulting deformation field is used to propagate the expert case organ boundaries onto the current case, thereby defining the organ boundaries for the current case. The algorithm is clinically used for radiation therapy treatment planning, which requires a high degree of accuracy in identifying the organ boundaries. Therefore, for treatment planning, the contours are typically revised based on the judgment of a human expert. In a previous head and neck study, the registration algorithm was demonstrated to deform the expert case to be within 2 mm of the reference (manually segmented) locations for 99% of the voxels.²⁶ In a breast study, the algorithm produced 94% volume overlap with the reference segmentations.²⁷ The commercial segmentation algorithm was used without modification in this study and without manual adjustment of the contours, as we hypothesize that organ dose can be estimated robustly despite small errors in the segmented contours.

2.2. Validation CT Datasets

Twenty previously acquired head-neck CT datasets included in the Smart Segmentation software were used for this study, with a scan range from the top of the head down to part of the chest, including the top of the lungs. The CT scans were acquired at 120 kV with varying acquisition parameters and on different scanners. The slice thicknesses were 5-mm (7 scans), 2.5-mm (2 scans), and 1.25-mm (11 scans). The datasets represent a range of ages from 49 to 82 and classified as “normal” BMI. All datasets contained pathology with right (10 datasets), left (3 datasets), and bilateral (7 datasets) tumors.

2.3. Monte Carlo Dose Map Generation

The voxels in the CT datasets were first automatically classified as being air, lung, adipose, water, soft tissue, muscle, or bone and assigned densities based on CT number. The thresholds and density ranges were based on previously published values.²⁸ The tissue maps were inputted to a Monte Carlo simulation tool (GEANT4²⁹) for dose estimation as Monte Carlo methods are considered the gold standard for dose estimation. The simulations modeled a clinical CT scanner with 8-cm detector thickness, 120-kV tube voltage, a helical pitch of 1, and 400 projection views per rotation. The scan range was adjusted for each phantom to ensure complete coverage of the phantom. The simulations modeled $5\times {10}^6$ photons per projection view, which was empirically determined to provide voxel dose map estimates with less than 1% standard deviation. The simulations output a dose map at the same voxel sampling as the original CT datasets, representing the radiation dose deposited in each voxel.

2.4. Expert and Automated Segmentation

Expertly segmented contours for each of the 20 CT cases were available from the Smart Segmentation atlas library. The expert contours were created and reviewed by a team of staff radiation oncologists at an academic hospital. The following nine regions were common to all cases used in this study: bone, brain, eyes (left and right), lungs (left and right), parotid glands (left and right), and spinal canal. While these regions were originally segmented for treatment planning, they also provide a useful test case for CT organ dose estimation. The atlas similarity score calculated by the commercial software was noted for all pairs of cases.

A leave-one-out validation study was performed, where every case was automatically segmented with each of the remaining cases used as the expert atlas, resulting in 19 automated segmentations for each of the 20 datasets. This provides a conservative estimate of algorithm performance, as all cases were used as an atlas, regardless of similarity to the current case. A second study restricted the atlases to cases with 1.25-mm slice thickness. The third study used only the best matched atlas (highest similarity score) for each of the 20 cases.

The 3-D Dice coefficients³⁰ between the automated and expert segmentations were calculated to evaluate the similarity between the expert and algorithm segmented regions.

2.5. Organ Dose Estimation

Each contoured region output by the segmentation algorithm was further constrained by the CT number tissue segmentation, so that a voxel was only included in the organ region if its CT number was within the range of the expected tissue type. The mean dose was calculated in each segmented region by averaging the voxelized dose map estimates within the segmented regions. The mean organ dose calculated in the expert segmented contours served as the ground truth for each case, with the expert segmentations also constrained by the CT number tissue segmentation. The percent error in organ dose was calculated relative to the ground truth values as

$$\%\text{Error}=100·\frac{({\text{Dose}}_{\text{algorithm}}-{\text{Dose}}_{\text{expert}})}{{\text{Dose}}_{\text{expert}}}.$$

The potential of a simple multi-atlas segmentation approach to reduce the error and variability of organ dose estimates was also investigated. The multiatlas mean organ dose estimates were calculated as the average of the dose values obtained from each atlas, with the number of atlases varied between two and nine. This multiatlas approach is mathematically equivalent to averaging the binary masks output by the segmentation algorithm for each expert case. The mean organ dose is then calculated as a weighted average, with the pixels in the averaged organ mask providing the weights, as illustrated in Fig. 2.

Fig. 2.

(a) The binary mask representing the spinal canal region for one patient as segmented using one atlas case. The overlaid contour represents the expertly segmented spinal canal for this case. (b) The binary mask obtained by segmenting the spinal canal using a different atlas. (c) The average of masks obtained from 10 different atlas cases. The values in the averaged mask provide weights for estimating the mean organ dose.

3. Results

Figure 3 plots the distribution of the error in mean organ dose aggregated across all patient cases segmented using every other case as the atlas. The bone, eye, and lung regions were segmented by the algorithm based on features, while the remaining organs were segmented using the atlas registration. Figure 4 plots the distribution of mean organ dose error when the expert atlas cases were restricted to those with slice thickness of 1.25 mm. When comparing these results of Figs. 3 and 4, it can be seen that using a thin slice atlas reduced the range of dose errors across cases, while the median values remained relatively constant when using thinner slices. For example, the left parotid gland had a 0.2% median percent organ dose error when using atlases at all slice thicknesses, compared with a median error of −0.6% when using only the thin slice atlases.

Fig. 3.

Percent error in mean dose for regions segmented using the automated single-atlas algorithm. The results are aggregated across 20 cases, with each case segmented using every other case as an atlas.

Fig. 4.

Percent error in mean dose for regions segmented using the automated single-atlas algorithm, using only atlas cases with 1.25-mm slice thickness. The results are aggregated across 20 patient cases.

Figure 5 plots the percent error in mean organ dose versus the Dice coefficient for each case in the leave-one-out validation study. A Dice coefficient of one represents complete agreement between the expert and algorithm segmented regions, while a Dice coefficient of zero signifies no overlap between the segmented regions. The results demonstrate high-segmentation accuracy (high Dice coefficients) for the bone, brain, and lung regions. The parotid gland regions demonstrated a wide range of segmentation accuracy. The spinal canal demonstrated the highest range of mean organ dose error, with relatively high-segmentation accuracy ($\text{Dice coefficient}>0.6$) for most cases. The Pearson correlation coefficient between absolute organ dose error and Dice coefficient was $-0.73$ (bone), $-0.43$ (brain), $-0.48$ (right eye), $-0.52$ (left eye), $-0.99$ (left lung), $-0.65$ (right lung), $-0.34$ (parotid left), $-0.46$ (parotid right), and $-0.23$ (spinal canal). The relationship of decreasing organ dose error with increasing Dice coefficient was statistically significant for all organs ($p<0.05$), although the parotid gland and spinal canal showed the weakest correlation, as illustrated in the example cases in Figs. 7 and 8. Figure 6 plots the percent error versus Dice coefficient for cases where the atlas expert case was restricted to 1.25-mm slice thickness. Comparing Figs. 5 and 6 demonstrates that using a thinner slice atlas reduced the segmentation error, most noticeably in the parotid gland.

Fig. 5.

Organ dose error plotted against the Dice coefficient for each organ region. Each point represents the results from one patient case segmented using one atlas case. Labeled points A and B are discussed in more detail in Figs. 7 and 8, respectively.

Fig. 7.

Example of parotid gland result with $<1\%$ dose error despite segmentation errors (labeled as data point A in Fig. 5). (a) The green contour is the expert segmentation of the parotid gland in the patient case. (b) The red contour is the expert segmentation of the parotid gland in the atlas case. Notice that the atlas case is reconstructed at a different voxel size than the expert case, which is accounted for by the automated segmentation algorithm. (c) The red contour is the parotid gland segmentation output by the automated algorithm after mapping the atlas contour to the current case. (d) The expert and algorithm contours overlaid on the dose map.

Fig. 8.

Example of spinal canal result with 18% dose error despite high-segmentation accuracy (labeled as data point B in Fig. 5). (a) The green contour is the expert segmentation of the spinal canal in the patient case. (b) The red contour is the expert segmentation of the spinal canal in the atlas case. Notice that the atlas case is reconstructed at a different voxel size than the expert case, which is accounted for by the automated segmentation algorithm. (c) The red contour is the spinal canal segmentation output by the automated algorithm after mapping the atlas contour to the current case. (d) The expert and algorithm contours overlaid on the spinal canal region of the dose map.

Fig. 6.

Organ dose error plotted against the Dice coefficient for each organ region, where the atlas cases were restricted to those with 1.25-mm slice thickness. Each point represents the results from one patient case segmented using one atlas case.

Two interesting scenarios are evident in Fig. 5 plot of organ dose error versus segmentation accuracy. In the data point labeled A, both the Dice coefficient and parotid gland dose error were low, demonstrating low organ dose error ($<1\%$) despite high-segmentation error ($\text{Dice coefficient}=0.04$). Figure 7 further investigates this scenario, presenting the expert segmentation of the patient and atlas cases, as well as the segmented contour output by the algorithm. The expert and algorithm contours are also displayed on the dose map. Figure 7 demonstrates no overlap between the expert and algorithm contours for this slice. However, because the dose map is similar within both segmented regions, the parotid gland dose is estimated with high accuracy.

The data point labeled B in Fig.  5 presents the opposite scenario, where the error in the spinal canal dose was high (18%), despite the high-segmentation accuracy ($\text{Dice coefficient}=0.83$). Figure 8 presents the expert and algorithm segmented contours for this case as well as the dose map. The depicted slice is in the neck region, where there is little soft tissue to shield the spinal canal, causing variation in the dose map within the spinal canal region. Also, this lack of shielding in the neck region causes these slices to have the highest overall spinal canal dose; therefore, dose errors in the neck region dominate the overall spinal canal dose error. These results demonstrate substantial error in the spinal canal dose despite the good agreement between expert and algorithm segmentations.

In the leave-one-out validation study, each case was segmented with every other case as the atlas, regardless of the similarity score between the two cases. The study also investigated the error in mean organ dose obtained when using the atlas case with the highest similarity score. The results are presented for the parotid gland and spinal canal, as these two regions demonstrated the highest organ dose error in Figs. 3 and 4. Figure 9 plots the maximum, minimum, and average absolute mean parotid gland and spinal canal dose errors for each of the 20 patient cases, with the atlas restricted to those with 1.25-mm slice thickness. The error obtained when using the atlas case with the highest similarity score is also plotted. These plots demonstrate, for each patient, the best and worst case organ dose accuracy that would be obtained from the atlas set, the error obtained, on average, by selecting a random atlas, and also the error obtained when using the best matched atlas. The results demonstrate that using the atlas with the highest similarity score generally performed similarly to the average error and prevented the maximum error in all but one case.

Fig. 9.

The minimum, maximum, and average absolute percent errors in mean organ dose for each patient across all atlases with 1.25-mm slice thickness for the (a) parotid gland and (b) spinal canal. The error obtained using the atlas with the highest similarity score is also plotted.

Figures 10 and 11 present the results of the investigated multiatlas method, where the organ dose was estimated as the average of dose values obtained from multiple atlases. The atlases were restricted to those with 1.25-mm slice thickness. The results are presented for the spinal canal and parotid gland, as these organ regions demonstrated the highest organ dose error. The maximum absolute error in organ dose was found to decrease as the number of atlas-based segmentations increased (Fig. 10). As shown in Fig. 10, the mean absolute error remained relatively constant as the number of atlases increased. Figure 11 plots the distribution of organ dose error obtained by averaging the results from nine atlas cases. When compared with the single-atlas results in Fig. 4, the results demonstrate reduction in the maximum organ dose errors across patients and atlases for the multiatlas estimates compared with the single-atlas estimates, with similar median organ dose errors for both approaches.

Fig. 10.

The mean and maximum absolute errors in parotid gland and spinal canal organ dose estimated using the multiatlas approach with different numbers of atlases.

Fig. 11.

Percent error in mean dose for regions segmented using multiatlas dose estimation method, using nine atlases with 1.25-mm slice thickness. The results are aggregated across 20 patient cases.

4. Discussion

This pilot study presents a preliminary evaluation of a commercial automated segmentation algorithm for CT organ dose estimation. The purpose of this study was to establish the baseline organ dose estimation performance of an existing commercial algorithm while identifying potential areas for future improvement. The results demonstrated that the mean organ dose was estimated with less than 5% error for the brain, lung, and bone regions when compared with gold standard estimates using expert segmentations. The eye and parotid gland regions demonstrated median errors below 5%, with maximum errors of 7% for the eye and 17% for the parotid gland. The maximum error was reduced to 4% for the eye and 8% for the parotid gland using the multiatlas averaging approach. The spinal canal was the only investigated region with a bias in the organ dose estimates, with a median error of approximately $-7\%$. In most cases, the mean spinal canal dose was underestimated when using the automated segmentation algorithm (Fig. 5). As demonstrated in Fig. 7, the dose within the spinal canal region of the neck is spatially variable and thus more susceptible to segmentation errors. For all organs, there was a statistically significant relationship of decreasing organ dose error with increasing segmentation accuracy, although this relationship was weakest for the spinal canal, with a Pearson correlation coefficient of $-0.23$.

This study found that using atlas cases with thin slices (1.25 mm) reduced the maximum dose errors. This study did not investigate the effects of the in-plane voxel dimensions, which varied across the atlas cases. The slice thickness results suggest that high-resolution atlas cases would likely improve organ dose estimation.

Figure 9 plots the range of errors for each patient case across the atlas set for the challenging parotid gland and spinal canal regions. This plot demonstrates how the dose accuracy depends on selected atlas case. For example, for each patient case, there was at least one atlas case that resulted in spinal canal dose error of less than 5%. More robust dose estimation may be possible by improved identification of the “best match” atlas, where the best match may vary by organ region, suggesting the need for a multiatlas approach. This study demonstrated reduction in maximum error when using a simple multiatlas averaging technique. However, even with the multiatlas averaging technique, the spinal canal region demonstrated 11% maximum absolute error and $-7\%$ mean error. More robust multiatlas approaches have been proposed and may be beneficial to further improve dose estimation accuracy.³¹ Additional studies are needed to compare the accuracy of the proposed patient-specific organ dose estimation method with previously proposed methods that are not patient specific^{9,10,17,20,21} including commercial software packages that estimate organ dose to phantom models.²⁰

This pilot study modeled a CT acquisition with 8-cm detector thickness and a helical pitch of one, as CT scanners are trending toward larger volume coverage. Different acquisition geometries and pitch settings may increase the variation of the dose maps within small organs, causing higher sensitivity to segmentation errors. This study used expertly segmented contours available from the Smart Segmentation library. Although the expert contours were created and reviewed by a team of radiation oncologists, information about inter- and intra observer variability was unavailable for use in the current study. Future work can compare the segmentation accuracy and organ dose error of automatic segmentation with that of the variability seen between different observers. Another limitation of this study is that the automated segmentation algorithm was evaluated only using head/neck cases. Additional evaluation with a larger number of testing datasets is required for other CT scan regions that may be more challenging such as the abdomen. The results of the current study demonstrate that the sensitivity of organ dose accuracy to segmentation accuracy varies by organ region. Accurate parotid gland dose estimation was possible despite high-segmentation error, because the dose map was relatively uniform in the region around the parotid gland. The variation of the dose map within the spinal canal in the neck region led to higher dose error despite high-segmentation accuracy. Therefore, future work is needed to identify regions that are sensitive to segmentation accuracy, such as the spinal canal, and to develop more accurate segmentation approaches or robust organ dose estimation strategies for these regions.

5. Conclusion

The automated segmentation algorithm estimated the mean organ dose to be within 10% of the expert segmentation for regions other than the spinal canal, with median error for each organ region below 2%. In the spinal canal region, the median error was $-7\%$, with a maximum absolute error of 28% for the single-atlas approach and 11% for the multiatlas approach. The results demonstrate that the automated segmentation algorithm can provide accurate organ dose estimates despite some segmentation errors.

Biographies

Taly Gilat Schmidt received her PhD in electrical engineering from Stanford University. She is an associate professor of biomedical engineering at Marquette University, where she directs the Medical Imaging Systems Laboratory. Her research interests include medical imaging system design, optimization, and reconstruction.

Adam S. Wang received his PhD in electrical engineering from Stanford University in 2012. Currently, he is a senior research scientist in the Ginzton Technology Center at Varian Medical Systems. His research interests include x-ray and CT imaging physics and image reconstruction.

Thomas Coradi received his master’s degree in electrical engineering from ETH Zurich, Switzerland. Currently, he is a senior software developer and research scientist at Varian Medical Systems, Switzerland. His research interests include image processing and automatic segmentation.

Benjamin Haas: Biography is not available.

Josh Star-Lack received his PhD in electrical engineering and computer science from the University of California at Berkeley. Currently, he is a senior research manager and scientist at Varian Medical Systems.

Disclosures

Authors Adam Wang, Thomas Coradi, Benjamin Haas, and Josh Star-Lack are employees of Varian Medical Systems. Author Taly Gilat Schmidt has no financial interests relevant to the presented work. Schmidt receives research funding from GE Healthcare for projects outside of the submitted work.