WHAT IS THE BEST WAY TO CONTOUR LUNG TUMORS ON PET SCANS:MULTI-OBSERVER VALIDATION OF A GRADIENT-BASED METHOD USING A NSCLC DIGITAL PET PHANTOM

Abstract

Purpose

To evaluate the accuracy and consistency of a gradient-based PET segmentation method, GRADIENT, as compared to manual (MANUAL) and constant threshold (THRESHOLD) methods.

Methods and Materials

Contouring accuracy was evaluated with sphere phantoms and clinically realistic Monte Carlo PET phantoms of the thorax. The sphere phantoms were 10–37 mm in diameter and were acquired at 5 institutions emulating clinical conditions. One institution also acquired a sphere phantom with multiple source-to-background ratios (SBR) of 2:1, 5:1, 10:1, 20:1, and 70:1. One observer segmented (contoured) each sphere with GRADIENT and THRESHOLD from 25–50% at 5% increments. Subsequently, seven physicians segmented lessions (7–264ml) from 25 digital thorax phantoms using GRADIENT, THRESHOLD, and MANUAL.

Results

For spheres < 20 mm in diameter, GRADIENT was the most accurate with a mean absolute %error in diameter of 8.15% (10.2%SD) compared to 49.2% (51.1%SD) for 45% THRESHOLD (p < 0.005). For larger spheres the methods were statistically equivalent. For varying SBR, GRADIENT was the most accurate for spheres > 20 mm, (p < 0.065) and < 20 mm (p < 0.015).

For digital thorax phantoms, GRADIENT was the most accurate, (p-value < 0.01), with a mean absolute %error in volume of 10.99% (11.9%SD) followed by 25% THRESHOLD at 17.5% (29.4%SD), and MANUAL, at 19.5% (17.2%SD). GRADIENT had the least systematic bias, 4 with a mean %error in volume of −0.05% (16.2%SD) compared with 25% THRESHOLD at - 2.1% (34.2%SD) and MANUAL at −16.3% (20.2%SD) (p-value < 0.01). Inter-observer variability was reduced using GRADIENT compared to both 25% THRESHOLD and MANUAL (p-value < 0.01, Levene's Test).

Conclusion

GRADIENT was the most accurate and consistent technique for target volume contouring. GRADIENT was also the most robust for varying imaging conditions. GRADIENT has the potential to play an important role for tumor delineation in radiation therapy planning and response assessment.

INTRODUCTION

Fluorodeoxyglucose Positron Emission Tomography (FDG PET) scans are used in lung cancer management for the initial staging [1], radiation therapy (RT) planning [2] and the evaluation of tumor response to therapy [3–4]. Advances in radiation therapy technology have improved the ability to deliver highly conformal therapy for smaller tumors and have increased the need for accurate and consistent definition of tumor boundaries. It has been demonstrated that applying PET to RT planning changes gross target volume (GTV) in more than 50% of NSCLC cases [2] and is particularly valuable in patients whose tumors blend with atelectasis in computed tomography (CT) image volumes. Great inter-observer variability has been reported in CT definition of GTV in lung cancer [5] indicating the limitations of CT for tumor definition. Therefore, there is great interest in lowering this variability, possibly with application of PET images for tumor delineation. Before PET can be widely applied for that purpose, standards must be established for the contouring technique.

There is currently no consensus as to the optimal technique for delineating (segmenting) PET target volumes. Various approaches are used which include:

1. Manual contouring (MANUAL), in which the physician determines the tumor outline based on visual perception of the tumor border.

2. Threshold methods, which define the tumor border within a region-of-interest placed over the tumor by including all tissue with activity greater then a defined level. Absolute thresholds define the tumor border based on a minimum SUV level. Suggested SUV levels have included 2.0 [5], 2.5 [6], or 3.0 +/− 1.6 from a recent study looking for the absolute threshold level that produced volumes most similar to pathology measurements for 9 NSCLC patients [7]. Percent constant threshold methods (THRESHOLD) define the tumor border based on a percentage of the maximum activity within the tumor. All tissue with activity greater than that percentage is included within the tumor volume. The impact of lesion size and source-to-background ratio on volumes obtained with constant threshold methods has been previously reported [1] [8] [9] [10]. A recent study demonstrated that in order to obtain image-derived volumes equal to pathology volumes in nine NSCLC patients, constant thresholds levels between 20–42% of maximum were required [7]. Adaptive threshold methods use parameters such as tumor size and the ratio of tumor to background levels to define the threshold level. [8].Currently there is no consensus as to the appropriate threshold method or best threshold levels for tumor segmentation. This variability is one factor limiting use of PET for tumor definition in radiation oncology. Most clinicians continue to rely on the CT-derived volume as the gold standard for GTV contouring and use PET as an ancillary tool, mostly to prevent omitting hyper-metabolic areas from patient’s GTV or to identify the interface between tumor and atelectasis.

3. Gradient edge detection identifies tumor based on a change in count levels at the tumor border. One proposed method requires in the following order: a denoising tool, a deblurring tool, a gradient estimator and a watershed transform [11]. This method is sensitive to voxel size, varying image resolution and noise, which requires adjusting one or more of these tools making it less realistic for routine clinical use. The gradient method evaluated in this paper, GRADIENT, (MIM Software Inc., Cleveland, OH), calculates spatial derivatives along tumor radii then defines the tumor edge based on derivative levels and continuity of the tumor edge.

Our goal was to evaluate the accuracy, bias, and consistency of GRADIENT compared to traditional manual and percent threshold contouring methods. In this paper we used first the experimental sphere phantoms to evaluate the impact of different PET cameras, sphere size, reconstruction methods and source to background ratios on border detection with both THRESHOLD and GRADIENT. Subsequently, in order to more closely emulate clinical reality we evaluated and compared three methods of PET tumor contouring - MANUAL, THRESHOLD, and GRADIENT. We used Monte Carlo PET thorax phantoms [12] which have been designed to simulate both lung tumors and mediastinal lymph node metastases. Since the true volumes of these simulated tumors and lymph nodes are known, they serve as the gold standard for the volumes contoured by the physicians.

METHODS AND MATERIALS

Contouring Methods

MANUAL CONTOURING (MANUAL)

Each observer used a manual contouring tool of their choice (pen, 2D or 3D paintbrush) provided in MIM (MIM Software Inc., Cleveland, OH) to delineate the structure of interest by visually outlining the boundaries. Five observers used both 3D and 2D brushes, one used 3D only and one used pen only. The structure could be contoured in any cross section and viewed in either a single slice or a splash page of contiguous slices. Each observer was able to adjust image contrast levels according to their own preference to allow for optimal visualization of the structure.

CONSTANT THRESHOLD CONTOURING (THRESHOLD)

The THRESHOLD contouring method relies on including all voxels that are greater than a defined percent of the maximum voxel within an operator-defined sphere. Cross-sectional circles are displayed in all three projections (axial, sagittal, and coronal) as the operator defines the sphere size and location to ensure three-dimensional coverage of the structure of interest. The structure could be contoured in any cross section and viewed in either a single slice or a splash page of contiguous slices. Each observer adjusted image contrast levels according to their own preference to allow for optimal visualization of the structure.

GRADIENT EDGE DETECTION (GRADIENT)

The gradient method relies on an operator defined starting point near the center of the lesion. As the operator drags out from the center of the lesion 6 axes extend out giving the operator visual feedback for the starting point of gradient segmentation. Spatial gradients are calculated along each axis interactively and the length of an axis is restricted when a large spatial gradient is detected along that axis. The 6 axes define an ellipsoid that is then used as an initial bounding region for gradient detection. The observers in the study were instructed to begin by selecting the image slice in which they identified the tumor to appear largest. The observer was then instructed to localize at a point near the center of the lesion in this slice and drag from that point until the six axes approximated the boundaries of the lesion (Figure 1). After releasing the mouse button the edges of the structure were automatically calculated and outlined. For very irregularly shaped structures, which are not well defined by the 6 axes, observers were instructed to use the gradient method one or more times to add to the initial contour by dragging out from a point near the center of the omitted region. The operator added regions until they were visually satisfied that the entire structure is included in the contour.

FIGURE 1.

Monte Carlo thorax phantom with simulated NSCLC lesion and mediastinal lymph node. Radii are created semi-automatically during GRADIENT segmentation and provide limits for boundary definition.

Characteristics of Sphere Phantoms

PET scans were acquired for commercially available sphere phantoms with five different PET scanners in five different institutions, Table 1. The institutions were instructed to emulate clinical acquisition and reconstruction methods used at that institution. All institutions were instructed to acquire the phantom with source to background levels ranging from 5:1 to 10:1. The NEMA/IEC 2001 phantom with spine insert was acquired with the GE and Siemens scanners and the Data Spectrum Elliptical Lung-Spine Body phantom without inserts was acquired with the Philips scanner. Six spheres with diameter sizes of 10, 13, 17, 22, 28, and mm were obtained for the NEMA/IEC 2001 phantom and six spheres with diameter sizes of 12.4, 15.6, 22.2, 25.2, 28, and 32.8 mm were obtained for the Data Spectrum phantom. The GE DLS camera also acquired the NEMA/IEC 2001 phantom on four additional days with source to background ratios of 2:1, 10:1, 20:1 and 70:1.

Table 1.

PET cameras, acquisition and reconstruction methods used to create sphere phantom image volumes

Manufacturer	Model	Reconstruction Method	Voxel Size	Background Bath Activity uCi/cc	Source to Background Ratio	Frame Duration Minutes
GE	DST	OSEM 2D	2.0 × 2.0 × 3.3	0.08	10	3
GE	DLS	OSEM 3D	3.9 × 3.9 × 4.3	0.41	2,5,10,20,70	2.8
GE	DSTE	FBP 2D	3.1 × 3.1 × 3.3	0.15	8	1.5
Siemens	Biograph	OSEM 2D	4.1 × 4.1 × 5.0	0.14	5.2	1.5
Phillips	Allegro	RAMLA_3D	4.0 × 4.0 × 4.0	0.13	6	3

Delineation Protocol for Sphere Phantoms

Commercially available software, MIM, (MIM Software Inc., Cleveland, OH) was used for THRESHOLD and GRADIENT contouring methods in this study. One observer contoured all sphere phantoms using the GRADIENT and 25–50% THRESHOLD method in 5% increments. Only the three largest spheres were contoured for the NEMA/IEC 2001 phantom acquired with 2:1 SBR since smaller spheres were difficult to visualize. In all other phantoms all spheres were contoured. To improve the consistency of threshold analysis, the same size region was used for each sphere of the same size in each of the four NEMA/IEC 2001 phantoms. The operator defined region was visually centered on the phantom sphere and the size was chosen to include the entire phantom sphere with minimal extra volume.

Characteristics of Digital Thorax Phantoms

The twenty-five clinically realistic digital PET Monte Carlo phantoms of NSCLC [12] used in this study had been created by adding lesions to the Zubal thorax phantom [13]. The lesions were added using the Monte Carlo simulation system for emission tomography (SimSET) software (http://depts.washington.edu/simset/html/simset_main.html). The Monte Carlo phantoms emulated real-life PET acquisition and reconstruction by modeling photon interaction within the lungs and PET camera followed by typical reconstruction. The clinical PET system simulated was a Reveal HD (CTI) scanner. The phantom authors [12] developed a method that resulted in image intensity distributions that were statistically confirmed to be similar to those in actual patient 18F-FDG PET image volumes of NSCLC tumors.

The phantom simulated 31 intra thoracic lesions of varying size, shape, location, and ¹⁸F-FDG activity distribution. Lesions ranged in size from 7 ml to 264 ml. Lesion locations were within the lung, adjacent to the mediastinum or chest wall, and within the mediastinum simulating tumors and lymph nodes. The volume of each lesion was determined prior to simulation which included forward projection to create PET sinograms and reconstruction to create PET image volumes. Typical cross sectional images are shown in Figure 1 including initial axes for GRADIENT contour definition.

Delineation Protocol for Digital Thorax Phantoms

Seven observers, two senior radiation oncologists (WC, MWW), one senior radiation oncologist with radiology training (YA), one radiation oncology resident (NO), and three PET trained nuclear radiologists (PFF, PK, FDA) segmented each tumor using all three segmentation tools used in the study: MANUAL, THRESHOLD, and GRADIENT.

Each observer received an online training session for each of the segmentation tools. For each of the digital PET scans a screen capture was created identifying the location of the tumor and the corresponding label to be used for that tumor (“lesion1” or “lesion2”). MANUAL delineation was performed first to avoid bias from the automatic contouring results and was separated by at least one day from the automatic delineation methods (THRESHOLD and GRADIENT).

Contouring Method Comparisons

Accuracy (how close the segmentation result is to the true value) is evaluated by calculating mean absolute percent error in diameter for the sphere phantoms and mean absolute percent error in volume for the Monte Carlo phantoms. Bias (the systematic difference of the segmentation result from the true value) is evaluated by calculating mean percent error in diameter for the sphere phantoms and mean percent error in volume for the Monte Carlo phantoms. It is important to evaluate both accuracy and bias for a segmentation technique since one technique could have better accuracy but a larger bias or vice versa. Consistent bias may have additional means for correction following automatic contour definition such as a consistent increase/decrease in boundary size. Statistical significance for paired data is calculated using student’s t-test. Levene’s Test is used to measure inter-observer variability.

RESULTS

Sphere Phantom

The spherical diameter which would result in the volume obtained by each segmentation method, THRESHOLD and GRADIENT, was calculated. This calculated diameter was compared to the known diameter of the sphere in order to quantify the accuracy of each segmentation algorithm. The mean and absolute percent error in diameter was combined for all cameras with source to background ratios between 5:1 to 10:1 using thresholds at 20, 25, 30, 35, 40, 45, and 50% (Figure 2). Results were separated into spheres greater than and less than 20mm. The 20 mm cutoff was chosen since accuracy significantly deteriorated for spheres less than 20 mm in diameter for all thresholds. As seen in Figure 2, the 45% THRESHOLD demonstrated the best accuracy and had the least bias with spheres > 20 mm and was thus chosen for further comparison with GRADIENT.

FIGURE 2.

Sphere phantom cumulative volume segmentation errors for five PET cameras at seven different threshold levels. Errors are displayed for spheres with diameters > 20 mm and < 20 mm

GRADIENT accuracy was consistent for different scanners, varying SBR and sphere sizes, Figure 3. In contrast, accuracy of 45% THRESHOLD deteriorated with decreases in both SBR and sphere size. The 45% THRESHOLD accuracy was also less for different PET cameras than GRADIENT’s accuracy. For sphere diameters greater than 20 mm errors from 45% THRESHOLD are very similar for all 5 cameras. For sphere diameters less than 20mm the errors increase for 3 of the 5 cameras.

FIGURE 3.

Sphere phantom results comparing the optimum 45% THRESHOLD method to GRADIENT. The y-axis of all four graphs is the percent error between the actual sphere diameter and the measured sphere diameter determined from the volume of the segmented sphere. The 0% y-axis represents perfect accuracy. The top row compares similar source to background results in five cameras and the bottom row evaluates the effect of varying source to background ratio in one camera.

The accuracy and bias for GRADIENT and THRESHOLD are reported in Table 2. For sphere phantoms of varying SBR, GRADIENT was significantly more accurate for spheres < mm (p < 0.015) and also was more accurate for spheres > 20 mm with a trend towards significance (p < 0.065). GRADIENT was also significantly more accurate (p < 0.005) and had less bias than THRESHOLD for spheres less than 20 mm diameter for the situation of multiple cameras with similar SBR. For the spheres larger than 20 mm with the same SBR, the methods were statistically equivalent.

Table 2.

Sphere phantom volume segmentation comparisons for 45% THRESHOLD and GRADIENT methods. Mean % error demonstrates bias and mean absolute % error demonstrates accuracy.

	Mean % Error		Mean Absolute % Error
	Diameter > 20mm	Diameter < 20mm	Diameter > 20mm	Diameter < 20mm
45% THRESHOLD (Multiple Cameras)	−1.64%	39.45%	3.94%	49.20%
GRADIENT (Multiple Cameras)	−0.69%	4.41%	4.19%	8.15%
p- value	0.660	0.004	0.846	0.005
45% THRESHOLD (Varied S/B* Ratios)	16.7%	42.6%	18.2%	44.7%
GRADIENT (Varied S/B* Ratios)	0.09%	7.9%	3.49%	13.4
p- value	.063	0.02	0.065	0.015

^*Source to Background Ratio

Digital Thorax Phantom

For the digital NSCLC phantoms, GRADIENT was the most accurate technique, (p-value < 0.01), with a mean absolute % error of 11.00% (11.9% SD) followed by 25% THRESHOLD at 17.5% (29.4% SD), and MANUAL, at 19.5% (17.2% SD). GRADIENT also had the least systematic bias, (p-value < 0.01), with a mean % error of −0.05% (16.2% SD) compared to the next smallest, 25% THRESHOLD, at −2.1% (34.2%SD). The mean % error for MANUAL was 16.3% (20.2% SD).

Recognizing after the first five observers that 25% THRESHOLD had the least error, the last two observers, Observer 2 and Observer 3, also contoured the phantom using 20% THRESHOLD to see if the error continued to decrease with even smaller thresholds. For both of these observers the 20% error was greater than either the 25% error or the 30% error and therefore was not studied further (data not shown).

Inter-observer variability was significantly reduced when using GRADIENT compared to either 25% THRESHOLD or MANUAL (p-value < 0.01, Levene's Test). The only significant difference between nuclear radiologists and radiation oncologists for both bias and accuracy for all three contouring methods was the bias for 25% THRESHOLD, Table 3. Figure 4 shows linear regression lines relating real and measured lesion volume for each observer for 25% THRESHOLD, GRADIENT and MANUAL.

Table 3.

Comparison of Radiologist and Radiation Oncologist Contouring Mean % error demonstrates bias and mean absolute % error demonstrates accuracy.

	Mean % Error			Mean Absolute % Error
Technique	Radiation Oncologists	Radiologists	P Value	Radiation Oncologists	Radiologists	P Value
GRADIENT	−0.80%	0.94%	0.28	11.5%	10.4%	0.50
25% THRESHOLD	−8.90%	6.90%	0.0005	14.7%	21.3%	0.11
Manual	−17.6%	−14.3%	0.28	20.8%	17.6%	0.16

FIGURE 4.

Thorax Monte Carlo phantom volume segmentation results for seven Radiologists/Radiation Oncologists using MANUAL, 25% THRESHOLD and GRADIENT contouring methods. Linear regression line for 31 segmentations with each method is displayed for each observer.

DISCUSSION

We have compared a gradient-based segmentation technique to percent constant threshold and manual contouring for delineating lesions on PET scans. We have obtained results in sphere phantoms (in order to assess the influence of technical factors related to PET scanning) and then in a realistic thorax phantom, imitating human lung tumors. In particular, GRADIENT was compared to the commonly used THRESHOLD method.

The GRADIENT method was more accurate in measuring sphere diameter than the 45% THRESHOLD and specifically was more accurate for all sphere sizes with varying SBR and for small spheres, measuring less than 2 cm in diameter, for a single SBR. There is no best single threshold for all volumes as the optimal threshold is a function of the volume. Smaller volumes require a larger threshold because of the partial volume effect. Camera resolution, reconstruction method and filtering can alter the maximum voxel value in the sphere which will then impact the threshold level, expressed in percent of maximum, used to generate contours [14]. The GRADIENT method however is not affected by a change in maximum counts but is a function of the local relative change in image count levels at the tumor/sphere boundary. The GRADIENT method is also not sensitive to varying background since the GRADIENT method does not depend on the magnitude of the count change but rather to the location of the maximum count change. Our phantom experiments have indicated that ideal imaging conditions for THRESHOLD are a large uniform intensity source in a constant background. For these conditions, phantoms with spheres greater than 2 cm in diameter GRADIENT and THRESHOLD were not statistically different.

The results from the clinically realistic thorax phantoms demonstrated that for typical lung tumor distributions, size, and locations in the thorax, GRADIENT was the most accurate segmentation method with more consistent results among different observers and less bias when compared to the MANUAL or the best constant threshold, 25% THRESHOLD. Interestingly the only significant difference between radiologists and radiation oncologists contouring results were for mean percent error based on 25% THRESHOLD. The radiologists’ contours had significantly bigger volume, Table 3.

For the simulated lung and mediastinal lesions the most accurate threshold level was 25% while for the sphere phantoms greater than 20 mm 45% was the most accurate threshold level. This difference is probably best explained by the varied activity distributions in the simulated thorax lesions with areas of both high and low counts as compared to the uniform activity in the spheres. Higher thresholds, resulting in higher activity cutoff levels, tended to underestimate the edge of the thorax lesion in the lower activity portions of the tumor. The optimum threshold for all lesions is related to the variability of activity distributions in the group of lesions being studied. An additional problem with the constant threshold method is for varied background in the region of the tumor, as illustrated in Figure 5. In this example, the 25% THRESHOLD contours are too large for the mediastinal lesion resulting from the background activity in the mediastinum being higher than the 25% of maximum activity level cutoff.

FIGURE 5.

Comparison of GRADIENT and 25% THRESHOLD segmentations of a NSCLC and a mediastinal lesion. The GRADIENT contours are magenta and the 25% THRESHOLD contours are blue.

A significant problem with Monte Carlo simulations of cancer is validating that the simulation accurately emulates patient tumor activity distributions. We were fortunate to have available a NSCLC Monte Carlo phantom that was validated by several statistical properties to accurately emulate FDG distributions found in NSCLC patients [12]. In addition to evaluating these statistical properties, medical imaging professionals visually validated the simulated tumors as being similar to patient tumors [12]. Additional Monte Carlo simulations for different cancer sites, various PET camera simulations and reconstruction methods would be beneficial to identify tumor contour accuracy for various imaging conditions.

In addition to further phantom studies, the GRADIENT method needs to be validated in actual patient PET images. An ultimate validation of any tumor-contouring method would require comparison to pathologic resected tumor volumes. However, accurate pathologic measurements of tumor dimensions and volume are difficult to obtain [15]. Prospective clinical trials will be necessary to confirm advantages of 18F-FDG PET for initial tumor contouring in radiation therapy planning.

CONCLUSION

GRADIENT was the most accurate and consistent method for contouring tumor volumes on PET when compared to manual and constant threshold methods for both sphere phantoms and clinically realistic Monte Carlo PET phantoms with simulated lung and nodal lesions in the thorax. Additionally, GRADIENT was the most robust technique for different PET cameras and varying imaging conditions. These encouraging results will have to be validated in PET scans from actual patients with lung cancer. Improved consistency of tumor contour definition will enable clinical trials results to be more comparable between institutions which will enhance evaluation of the impact of PET-defined tumor volumes in radiation therapy and treatment planning.