Validation of phantom‐based harmonization for patient harmonization

Abstract

Purpose

To improve the precision of multicenter clinical trials, several efforts are underway to determine scanner‐specific parameters for harmonization using standardized phantom measurements. The goal of this study was to test the correspondence between quantification in phantom and patient images and validate the use of phantoms for harmonization of patient images.

Methods

The National Electrical Manufacturers' Association image quality phantom with hot spheres was scanned on two time‐of‐flight PET scanners. Whole‐body [¹⁸F]‐fluorodeoxyglucose (FDG)‐PET scans were acquired of subjects on the same systems. List‐mode events from spheres (diam.: 10–28 mm) measured in air on each scanner were embedded into the phantom and subject list‐mode data from each scanner to create lesions with known uptake with respect to the local background in the phantom and each subject's liver and lung regions, as a proxy to characterize true lesion quantification. Images were analyzed using the contrast recovery coefficient (CRC) typically used in phantom studies and serving as a surrogate for the standardized uptake value used clinically. Postreconstruction filtering (resolution recovery and Gaussian smoothing) was applied to determine if the effect on the phantom images translates equivalently to subject images. Three postfiltering strategies were selected to harmonize the CRC _mean or CRC _max values between the two scanners based on the phantom measurements and then applied to the subject images.

Results

Both the average CRC _mean and CRC _max values for lesions embedded in the lung and liver in four subjects (BMI range 25–38) agreed to within 5% with the CRC values for lesions embedded in the phantom for all lesion sizes. In addition, the relative changes in CRC _mean and CRC _max resulting from the application of the postfilters on the subject and phantom images were consistent within measurement uncertainty. Further, the root mean squared percent difference (RMS _pd ) between CRC values on the two scanners calculated over the three sphere sizes was significantly reduced in the subjects using postfiltering strategies chosen to harmonize CRC _mean or CRC _max based on phantom measurements: RMS _pd of the CRC _mean values in subjects was reduced from 36% to < 8% after harmonizing CRC _mean , while RMS _pd for CRC _max was reduced from ~33% to < 6% after harmonizing CRC _max with a different strategy. However, with this strategy designed to harmonize CRC _max , the RMS _pd for CRC _mean only improved to ~14% in subjects.

Conclusions

The consistency of the CRC measurements between the phantom and subject data demonstrates that harmonization strategies defined with phantom studies track well to patient images. However, quantitative agreement between different scanners as represented by the RMS _pd depends on the metric chosen for harmonization.

1. Introduction

PET/CT (e.g., using [¹⁸F]‐fluorodeoxyglucose (FDG)) has become an important tool for clinical diagnosis, staging, and prognosis in oncology in part because it allows for quantification of various clinical metrics, the most common of which is the standardized uptake value (SUV). While SUV is representative of the tracer uptake, it depends on both physiological (e.g., blood glucose level, patient weight, time after injection) and technical sources of variation. There are a number of technical factors related to scanner performance characteristics, acquisition protocols, and reconstruction algorithms and parameters that can affect these metrics^{1, 2, 3} and, therefore, the accuracy of quantitative PET studies. The value of a multicenter study is especially dependent upon reducing the variability of quantification across the various scanners and sites involved, as better precision allows for a reduction in the number of subjects needed to obtain statistically significant findings. Differences in reconstruction protocols⁴ and scanner hardware, however, lead to variations in performance, not only among the many models of scanners available, but also within the same scanner operated at different institutions. These findings have prompted a number of organizations (e.g., the North American Quantitative Imaging Biomarker Alliance, European Association Research Ltd accreditation program, and Clinical Trials Network of the Society of Nuclear Medicine and Molecular Imaging) to undertake efforts to harmonize scanners through standardization of scan acquisition, image reconstruction, and image analysis methods. The University of Pennsylvania and the University of Washington are part of a multi‐institutional National Cancer Institute (NCI)‐funded Harmonization Initiative led by the University of Iowa. The NCI Harmonization Initiative aims to determine the reconstruction parameters and postfiltering necessary for harmonization of oncologic studies and to assess the impact of harmonization on multicenter studies.⁵ These various organizational efforts all use standardized phantom measurements to determine scanner‐specific acquisition and reconstruction protocols. In these efforts, harmonization is defined as minimizing in some sense (e.g., root mean square difference) the distance between the curves of contrast recovery coefficient as a function of sphere size, where these curves have been sampled at a finite number of sizes ranging between 10 and 37 mm. It has been found⁵ that an overall harmonization between different scanners is possible through a choice of reconstruction parameters that may include a postfiltering procedure, but that it may not provide the same agreement for all sphere sizes. The NCI harmonization project uses a broader range of sphere sizes (8.5–44 mm) to impose stricter requirements on the harmonization than other efforts.

While harmonization using phantom scans has been shown, its consistency when applied to patient studies for the various harmonization strategies has yet to be demonstrated. Several studies have undertaken this task and made important progress.^{6, 7, 8} Our study is distinguished from previous work in its aim to directly measure the impact of phantom harmonization on patient harmonization by comparing quantification metrics for phantom and human subject studies where lesions of known uptake have been embedded synthetically into the subjects' list‐mode data.

2. Materials and methods

2.A. Phantom studies

The phantom used is the National Electrical Manufacturers' Association (NEMA) image quality (IQ) phantom.⁹ We followed the filling and acquisition protocol prescribed by the NCI Harmonization Initiative.⁵ The phantom was filled with ~20 MBq of ¹⁸F in the background. The standard set of glass‐walled spheres from the NEMA IQ study (diameters: 10, 13, 17, 22, 28, and 37 mm) were all filled with activity at a sphere:background ratio of 9.7:1, as prescribed by the NCI harmonization protocol. While the NCI Harmonization Initiative calls for plastic‐walled spheres, we have measured no appreciable difference in contrast recovery results with the different wall materials. For this project, we focus only on the 10‐, 17‐, and 28‐mm diameter spheres for later comparison with patient data. The phantom data were acquired in list‐mode for ten 3‐min acquisitions. For this study, one of the 3‐min scans was used in the lesion‐embedding studies for comparison with subject data, while the ten 3‐min scans on the Ingenuity TF were merged to form a 30‐min dataset for validation of the lesion‐embedding technique, described below.

2.B. Scanners and reconstruction algorithm

Two scanners were used for the comparison of the phantom and human subject studies in this investigation: the Philips Ingenuity TF¹⁰ and a prototype time‐of‐flight (TOF) scanner developed at the University of Pennsylvania, LaPET.^{11, 12} These scanners were chosen because they use a common framework for generating list‐mode data, thereby enabling the same methodology of lesion embedding to be applied, as described in the following section. Their intrinsic performance characteristics also differ sufficiently to result in different contrast recovery coefficient (CRC) performance. The Ingenuity TF uses 4 × 4 × 22 mm³ lutetium oxyorthosilicate (LYSO) pixels and has a spatial resolution of 4.9 mm at a radial offset of 1 cm, an energy resolution of 11.6%, and an intrinsic timing resolution of 495 ps. LaPET uses 4 × 4 × 30 mm³ LaBr₃ crystals with a modified electronics design and has a measured spatial resolution of 5.8 mm, an energy resolution of 7%, and an intrinsic timing resolution of 375 ps. To control the reconstruction of the datasets, data from both scanners were reconstructed using an off‐line list‐mode TOF ordered subsets expectation maximization (OSEM) algorithm¹³ without resolution modeling, using five iterations with 25 chronological subsets, which approximates the clinical reconstruction protocol on the Ingenuity TF. The same reconstruction parameters (“default reconstruction”) were deliberately chosen rather than separately optimizing the reconstruction for the two scanners, so differences in performance between the systems and the need for harmonization would be evident. All corrections for physical effects (e.g., scatter, attenuation, randoms) were applied during reconstruction. All images were reconstructed using blob basis functions¹⁴ and interpolated to 2‐mm voxels in the final image.

2.C. Lesion‐embedding technique

Embedding lesions of known uptake into subject data allows for quantification of lesion uptake under clinical conditions, since the ground truth is known. Because the true lesion uptake is known, the method of lesion embedding^{15, 16, 17} was chosen in this study for the subject measurements. The lesion‐embedding technique, in brief, involves scanning spheres filled with¹⁸F in air at known locations in the field‐of‐view of the scanner. List‐mode events from a particular sphere are randomly interspersed with the subject's list‐mode data to create a fused dataset, with the number of added events chosen to achieve a desired activity ratio, where this calculation takes into account events already present in the subject's background in the region of the sphere. In order to include the effects of attenuation by the subject that were not present during the sphere‐in‐air acquisition, events from the sphere's list‐mode data are randomly removed according to the probability of attenuation along the line of response of each event prior to merging the datasets, using the subject's attenuation sinogram. The resulting number of sphere events approximates that which would have been detected from a lesion inside the subject. The lesion‐embedding technique implicitly includes partial volume blurring of the sphere as well as spill‐in of activity from the neighboring background of the subject into the region of the sphere. The lesion‐embedding algorithm was updated from that described previously¹⁷ to compensate for attenuation of the sphere filled with water, which impacts the calculation of the number of sphere events to embed. While this effect is small for the 10‐mm lesions used in previous studies, self‐attenuation within larger spheres is non‐negligible. Reconstructing spheres with attenuation correction leads to an average change in the reconstructed sphere image counts and corresponding reduction in number of sphere events to be embedded of 3.7%, 9.1%, and 17%, compared to reconstructing without attenuation, for the 10‐, 17‐, and 28‐mm spheres, respectively. The embedded sphere events accounted for < 1% of the total events in the merged datasets, so no attempt was made to model the very small additional scatter that would have been introduced if the lesion had been present in the subject. Data with embedded lesions were corrected using the same scatter and randoms corrections as the original data without lesions.

For this study, data were collected with three spheres that were identical to those used in the NEMA IQ measurement, with diameters of 10, 17, and 28 mm, spanning a range of sphere sizes where the curves of CRC vs. sphere diameter change most rapidly. The spheres were filled with ¹⁸F and scanned on each scanner, with 10 million events per position collected (acquisitions were ~30 s) to ensure that a sufficient number of events were collected to allow for lesions to be embedded at the desired activity ratio with respect to background. Approximately 50 positions per sphere were collected on each scanner, with each position separated from the others by at least 3 cm center‐to‐center to avoid overlap.

The lesion insertion methodology was tested on the Ingenuity TF using the summed 30‐min scan of the NEMA IQ phantom. The longer scan duration was chosen to minimize variability in the results caused by statistical noise, so any discrepancies between embedded spheres and those measured in the phantom could be discerned. This measurement was repeated on three separate days to determine the uncertainty of the measurement including all sources of measurement error, including those due to human error. Eight spheres of each of the three sizes were embedded into the background region of one of the 30‐min phantom scans, in locations that did not overlap with the NEMA spheres. To avoid overlap between the embedded lesions, separate fused phantom datasets were created for each sphere size. Because the spheres measured in the phantom (“measured spheres”) have a cold wall (thickness = 1 mm) that reduces the spill‐in of activity from the background into the volumes of interest,¹⁸ while embedded lesions do not displace background activity and are thus wall‐less, a numerical correction was performed on the measured sphere results to account for the cold wall. The correction factors for volumes of interest (VOIs) with diameters equal to the inner diameter of the spheres ranged from 1.01 for the 28‐mm sphere to 1.06 for the 10‐mm sphere, which has a larger ratio of surface area to volume inside the sphere and, thus shows a greater effect of the cold wall on VOI results. The cold wall correction was applied to all measured sphere CRC _mean results; a cold wall correction was not applied to CRC _max results, since the location of the voxel with maximum uptake relative to the center of the sphere is not known (i.e., the maximum voxel is not necessarily at the exact center of the sphere). The CRC values for the embedded lesions were then compared to the average corrected CRC values of the measured spheres in the phantom. A similar comparison was performed for the LaPET scanner, although only a single 3‐min phantom study was available.

2.D. Clinical metrics

The CRC was used in this study as a proxy for the SUV used clinically, as other groups have done.^{5, 8} Both CRC and SUV are scaled estimates of lesion uptake, so CRC is an appropriate surrogate for SUV. CRC is the metric used in the NCI harmonization project, defined by NEMA⁹ as:

$$CRC=\frac{\left(\frac{S}{B_{local}}-1\right)}{A-1}$$ (1)

where S is the lesion uptake (activity concentration), B _local is the average value of the local background, and A is the true activity ratio (9.7). In this study, S was chosen to be equal to the average voxel value within the lesion, calculated using a VOI of the same diameter as the lesion (the measured inner diameter of the sphere) and centered over each lesion (for CRC _mean ), or to the maximum value within the VOI (for CRC _max ). B _local was calculated for each sphere by averaging the voxel value of the same (single) VOI as that used to calculate S for CRC _mean but drawn on the phantom or subject image without lesions embedded. Unlike clinical studies, in this work we know the true size and uptake of the lesions.

Both mean and maximum values of CRC were determined. Although mean values are commonly used for standardized scanner performance measurements, such as those defined by NEMA, it is important also to consider the maximum value, as SUV _max is routinely used in the clinic. While the maximum value has the potential for increased statistical uncertainty, it is less dependent on the lesion size and partial volume effects or on the definition of the VOI. The peak value of CRC (CRC _peak ), determined from a 1‐cm³ spherical VOI centered on the voxel with maximum uptake in the sphere, was also calculated for the 17‐ and 28‐mm spheres; because the 1‐cm³ VOI is larger than a 10‐mm sphere, CRC _peak was not calculated for the 10‐mm spheres.

2.E. Subject studies

Whole‐body data were acquired for four subjects on each scanner using ~550 MBq of [¹⁸F]‐FDG (3 min/bed position); scanning was performed 60 min postinjection for the Ingenuity TF subjects and ~100 min postinjection for the LaPET subjects, which were acquired sequentially following a clinical scan on a different system. Since this study used lesion‐free patient data and looked only at lesions embedded at a known uptake with respect to the local background, the variation in postinjection time has little impact on the results. The Ingenuity TF subject data were obtained retrospectively and were anonymized before being included in this study. The LaPET subject studies were performed as part of a research study¹⁷; the institutional review board (IRB) of the University of Pennsylvania approved this study, and all subjects signed a written informed consent before the study. The four subjects had BMIs ranging from 25 to 38 (Ingenuity TF) and 25 to 36 (LaPET) and were free of any lung or liver lesions; different subjects were scanned on the two systems. For the Ingenuity TF scanner, the transmission image was derived from a low‐dose CT scan. For the LaPET scanner, a ¹³⁷Cs scan was used for attenuation correction.¹⁹

For each subject, 3–8 lesions of each size were embedded into both the right lung and liver background regions. Lesions were added such that no two lesions overlapped on the same image. The range in number of lesions used reflects the fact that the larger lesions could not always fit in the same, nonoverlapping places as the smaller lesions, and organ sizes varied among the subjects. Lesions were embedded at a constant ratio of lesion uptake to local background (B _local ) of 9.7:1 to achieve constant CRC. To reduce the statistical variability of the LaPET results, which had fewer spheres measured at locations that overlapped with the subjects' lung and liver regions, five different sets of events for each sphere location were embedded in the lung and liver, and the results averaged. The CRC values were then averaged over all lesion locations in all four subjects for each organ studied (lung and liver).

2.F. Effect of postfiltering on phantom and subject measurements

Harmonization of scanners may be performed using postfilters on one or both of the scanners to achieve agreement between the curves of CRC vs. sphere size for the scanners. The postfilters were applied to phantom and subject data from the Ingenuity TF to assess how well subject and phantom results track. Two postfiltering methods were investigated for this study. (a) To decrease the CRC of the lesions, a Gaussian postfilter, as has been used in previous investigations,^{7, 20} was applied; for this study, the postfilter was applied to the images using MATLAB (The MathWorks Inc.). Two Gaussian filters with a full‐width at half‐maximum (FWHM) of 4 mm and 6 mm were used. These filter widths were found to provide the degree of change needed to harmonize CRC _mean results from the Ingenuity TF with those from the LaPET scanner, as described below. (b) The CRC of the lesions was also increased using a Lucy‐Richardson (LR) deconvolution postfilter^{21, 22, 23}; this filter is analogous to methodology implemented by Philips for point spread function (PSF) modeling and was chosen because it allows the user to improve the image resolution without additional reconstruction. It has been previously investigated for PET applications, using the method of sieves to control noise.^{24, 25, 26} For this study, one iteration was used with a 6‐mm sieve blur. While the LR filter was not applied to the Ingenuity TF data during harmonization to the LaPET (next section), LR postfiltering is the method of resolution modeling available on the Ingenuity TF and was used here to study how well the subject and patient data tracked for both image blurring and sharpening. The changes in measured CRC with these postfilters relative to no postfiltering were compared between phantom and subjects to determine how well the phantom and subject results tracked. A similar analysis was performed for the LaPET data for the LR filter.

2.G. Harmonization of scanners

Harmonization of the CRC values of the Ingenuity TF and LaPET scanners was accomplished with phantom scans by applying various postfilters to the images from both scanners, using a Gaussian filter to systematically reduce CRC and the LR postfilter to increase CRC. No standard metric yet exists to characterize how different (or harmonized) scanners are; the root mean squared percent difference (RMS _pd ) over the 3 lesion sizes between the CRC values was used in this study as a metric to characterize the overall difference between the scanners. The RMS _pd conceptually acts in a similar way to a least squares harmonization performed on the entire CRC vs. diameter curves (e.g., as in Ref. 8), but this metric measures the relative differences in CRC, unlike the least squares difference, which measures absolute differences in CRC. The percent difference of CRC values between the scanners for a given sphere size i (%diff(i)) was defined as

$$\%diff(i)=\frac{CR{C}_2-CR{C}_1}{(CR{C}_2+CR{C}_1)/2}\ast 100,$$ (2)

where the CRC values were averages over all spheres of a given diameter for each scanner. For this study, subscript 1 refers to the Ingenuity TF, and subscript 2 refers to the LaPET system. The RMS _pd of the CRC values was calculated over the three sphere sizes as

$$RM{S}_{pd}=\sqrt{\frac{1}{3}\sum _{i=1}^3\%diff{(i)}^2}.$$ (3)

Because the optimal harmonization strategy will depend on the imaging situation (e.g., several different scanners at one institution, the same scanner model at multiple sites, or multiple different scanners at different sites), three harmonization strategies were studied, two based on minimizing the differences between the CRC _mean values and one chosen to minimize the differences between CRC _max values of the three spheres. One of these strategies used only Gaussian postfiltering on one scanner, while the other two strategies used a hybrid approach of LR postfiltering on one system with Gaussian postfiltering on the other. Several strategies were chosen, since it may not be possible to harmonize both metrics simultaneously for certain combinations of scanners using a single strategy given that the curves of CRC _mean and CRC _max as a function of diameter can have different shapes. The strategies were as follows: (a) The LaPET images had no postfiltering applied, while the CRC values for the Ingenuity TF were decreased using a 6‐mm FWHM Gaussian postfilter applied to the reconstructed images; the 6‐mm width was empirically determined to be optimal by filtering with a range of Gaussian filters to minimize the RMS _pd between the CRC _mean values measured in the phantom on the two scanners. (b) The CRC values for the LaPET scanner were increased using the LR postfilter with one iteration and a 6‐mm sieve blur, while the CRC values of the Ingenuity TF were simultaneously decreased using a 4‐mm FWHM Gaussian postfilter that was determined empirically to minimize the RMS _pd of the CRC _mean values between the Ingenuity TF scanner and LaPET system with LR filtering, since the LR postfilter alone provided suboptimal harmonization. (c) The CRC values for the LaPET scanner were increased using the LR postfilter, as in strategy 2, while the CRC values of the Ingenuity TF were simultaneously decreased using a 2.5‐mm FWHM Gaussian postfilter that was determined empirically to minimize the RMS _pd of the CRC _max values between the Ingenuity TF scanner and LaPET system with LR filtering. The same filters were then applied to the subject scans.

3. Results

3.A. Measured vs. embedded spheres in phantom

Table 1 shows a comparison of CRC _mean and CRC _max values for the measured spheres and embedded lesions in the NEMA IQ phantom for the 30‐min scans on the Ingenuity TF scanner. Table 2 shows a similar comparison for a single 3‐min scan on the LaPET scanner. The uncertainties of the embedded lesion CRC values were calculated as the standard error of the mean (SEM) of the 8 embedded lesion values. The individual results for the three replicates of measured spheres on the Ingenuity TF are shown in parentheses in Table 1 rather than calculating the SEM because of the limited number of measurements. Based on the agreement between the measured and embedded CRC values, particularly for the 17‐ and 28‐mm spheres, and the reduced variability seen for the embedded lesions compared with the variability of the measured spheres, lesions embedded in the phantom were used for the remainder of this work in the comparisons with the subject data.

Table 1.

Comparison of CRC _mean (top) and CRC _max (bottom) on the Ingenuity TF scanner for measured spheres and lesions embedded into the NEMA IQ phantom

	10 mm	17 mm	28 mm
CRC mean
Measured	0.35 (0.41, 0.33, 0.31)a	0.58 (0.58, 0.59, 0.58)	0.69 (0.68, 0.68, 0.70)
Embedded	0.390 ± 0.005b	0.59 ± 0.01	0.684 ± 0.003
CRC max
Measured	0.64 (0.78, 0.58, 0.56)	0.95 (0.96, 0.94, 0.95)	1.00 (0.96, 1.00, 1.04)
Embedded	0.69 ± 0.01	1.00 ± 0.02	1.02 ± 0.01

^aResults are shown in parentheses for the three separate phantom measurements.

^bUncertainties shown are the standard error of the measurement (SEMs) of the 8 embedded lesion values.

CRC _mean , mean contrast recovery coefficient; CRC _max , maximum contrast recovery coefficient.

Table 2.

Comparison of CRC _mean (top) and CRC _max (bottom) on the LaPET scanner for measured spheres and lesions embedded into the NEMA IQ phantom

	10 mm	17 mm	28 mm
CRC mean
Measured	0.25	0.45	0.58
Embedded	0.253 ± 0.003a	0.446 ± 0.002	0.58 ± 0.01
CRC max
Measured	0.46	0.84	1.01
Embedded	0.43 ± 0.02	0.81 ± 0.02	0.99 ± 0.03

^aUncertainties shown are the SEMs of the 8 embedded lesion values.

CRC _mean , mean contrast recovery coefficient; CRC _max , maximum contrast recovery coefficient.

3.B. Embedded spheres in phantom vs. subjects

Figure 1 shows the average CRC _mean and CRC _max values for the 10‐, 17‐, and 28‐mm lung and liver lesions in each of the patients on the Ingenuity TF. The results demonstrate that the CRC values do not vary significantly among the patients. The average CRC (both mean and maximum values) for each subject was within 10% of the overall average values over all four subjects for all sphere sizes. For that reason, the subject results were averaged over all subjects and all lesions in each organ in the rest of the paper. Table 3 shows the CRC _mean and CRC _max values for the embedded lesions in the 3‐min phantom scan and the subjects' lung and liver regions for the Ingenuity TF and LaPET scanners without any postfiltering. Results shown for the lung and liver regions are averages over all of the lesions of a given size in the four subjects. The uncertainties listed in Table 3 are the SEMs of the lesion values (8 lesions embedded in the phantom and a total of 24–34 lesions in the lung and 49–57 lesions in the liver for the Ingenuity TF and 12–14 lesions in the lung and 15–20 lesions in the liver for the LaPET). The average CRC _mean values of lesions embedded in the lung and liver were well correlated with those for the lesions of the same size in the phantom; all values for lesions in the lung and liver were within 5% for the Ingenuity TF (range: −2.5% to 5.0%) of the respective phantom values and 5% for the LaPET (range: −4.4% to 0.0%). The percent differences of the average CRC _max values for lesions embedded in the lung and liver were also within 5% of those in the phantom for the Ingenuity TF (range: −1.4% to 4.2%) and ~6% for the LaPET (range: 2.3% to 6.1%). There was no systematic correlation in the difference between phantom and subject results with sphere size.

Figure 1.

Average CRC _mean and CRC _max values for (a) 10‐mm, (b) 17‐mm, and (c) 28‐mm lung and liver lesions for each of the patients on the Ingenuity TF. 8–16 lesions were embedded in each subject's lung and liver for a total of 24–34 and 49–57 lesions in the lung and liver, respectively, in all subjects. The error bars are the standard deviations over the lesions in each patient and organ. The BMI values for subjects 1–4 were 25, 28, 28, and 38 kg/m², respectively.

Table 3.

Comparison of CRC _mean and CRC _max on the Ingenuity TF (top) and LaPET (bottom) scanners without postfiltering for lesions embedded in the phantom and lung and liver regions in the subjects

	CRC mean			CRC max
	10 mm	17 mm	28 mm	10 mm	17 mm	28 mm
Ingenuity TF
Phantom	0.40 ± 0.007a	0.59 ± 0.003	0.68 ± 0.004	0.71 ± 0.02	1.07 ± 0.01	1.18 ± 0.02
Lung	0.39 ± 0.005	0.59 ± 0.009	0.69 ± 0.009	0.70 ± 0.01	1.12 ± 0.02	1.23 ± 0.02
Liver	0.42 ± 0.005	0.61 ± 0.004	0.70 ± 0.006	0.74 ± 0.01	1.10 ± 0.01	1.20 ± 0.01
LaPET
Phantom	0.25 ± 0.003	0.45 ± 0.002	0.58 ± 0.01	0.43 ± 0.02	0.81 ± 0.02	0.99 ± 0.03
Lung	0.24 ± 0.003	0.43 ± 0.005	0.56 ± 0.01	0.44 ± 0.01	0.85 ± 0.02	1.05 ± 0.01
Liver	0.25 ± 0.007	0.45 ± 0.007	0.58 ± 0.01	0.45 ± 0.01	0.83 ± 0.02	1.05 ± 0.02

^aUncertainties shown are the SEMs of the lesion values.

CRC _mean , mean contrast recovery coefficient; CRC _max , maximum contrast recovery coefficient.

3.C. Effect of postfiltering on phantom and subject measurements

Figure 2 (left) shows the average CRC _mean and CRC _max values on the Ingenuity TF for the phantom and lung and liver regions without any postfilter and after application of 4‐mm Gaussian, 6‐mm Gaussian, and LR postfilters; the percent changes in these values with respect to the results without postfiltering (Table 3) are listed in Table 4. Results are also shown in Fig. 2 (right) for the LaPET scanner without postfiltering and with the LR postfilter, with the corresponding percent changes in Table 4. The uncertainties shown in the table were determined by error propagation of the SEMs of the CRC values with and without postfiltering. Differences between the changes with postfiltering for lesions embedded in the phantom and those in the subjects are within measurement uncertainty. However, the relative impact of the filters on the CRC metrics is dependent on the size of the lesion, as expected.

Figure 2.

Comparison of average (top) CRC _mean and (bottom) CRC _max values for 10‐, 17‐, and 28‐mm lesions before and after application of various postfilters on the (left) Ingenuity TF and (right) LaPET scanners for the lesions embedded in the phantom and lung and liver regions in the subjects. The postfilters on the Ingenuity TF included 4‐mm FWHM Gaussian, 6‐mm Gaussian, and Lucy‐Richardson (LR) deconvolution postfilters; only the LR postfilter was applied to LaPET images. The results shown are averages over multiple lesions embedded in the phantom and lung and liver of each subject: 8 lesions in the phantom and 3–8 lesions in each subject's lung and liver; the results for the four subjects were averaged together for these comparisons. Error bars shown are the standard deviations of the measurements.

Table 4.

Percent change^a in CRC _mean and CRC _max on the Ingenuity TF scanner after application of various postfilters for lesions embedded in the phantom and lung and liver regions in the subjects. Results are also shown for LR postfiltering on the LaPET scanner

	CRC mean			CRC max
	10 mm	17 mm	28 mm	10 mm	17 mm	28 mm
4‐mm Gaussian
Phantom	−21 ± 2b	−10 ± 1	−5 ± 1	−28 ± 3	−12 ± 1	−13 ± 2
Lung	−20 ± 2	−10 ± 2	−6 ± 2	−29 ± 2	−13 ± 2	−12 ± 2
Liver	−20 ± 2	−10 ± 1	−5 ± 1	−27 ± 1	−11 ± 1	−11 ± 2
6‐mm Gaussian
Phantom	−38 ± 2	−20 ± 1	−11 ± 1	−48 ± 3	−22 ± 1	−19 ± 2
Lung	−38 ± 1	−20 ± 2	−11 ± 2	−49 ± 2	−24 ± 2	−19 ± 2
Liver	−37 ± 1	−20 ± 1	−11 ± 1	−47 ± 1	−21 ± 1	−17 ± 1
Lucy‐Richardson
Phantom	29 ± 3	19 ± 1	10 ± 1	39 ± 5	33 ± 2	18 ± 3
Lung	32 ± 3	20 ± 4	11 ± 2	41 ± 4	35 ± 3	20 ± 3
Liver	30 ± 3	19 ± 1	10 ± 1	40 ± 2	33 ± 2	18 ± 2
Lucy‐Richardson (LaPET)
Phantom	27 ± 3	19 ± 1	10 ± 3	35 ± 10	31 ± 6	12 ± 4
Lung	25 ± 3	18 ± 2	10 ± 3	33 ± 6	31 ± 4	17 ± 2
Liver	28 ± 6	21 ± 2	10 ± 3	37 ± 6	33 ± 4	18 ± 3

^aThe percent changes were calculated for the values shown in Fig. 3 with respect to the corresponding values without filtering (Table 3).

^bUncertainties shown were determined by error propagation of the SEMs of the CRC values with and without postfiltering.

CRC _mean , mean contrast recovery coefficient; CRC _max , maximum contrast recovery coefficient.

3.D. Harmonization of scanners

Figure 3 shows CRC _mean , CRC _peak , and CRC _max values for the different sphere sizes in the phantom for the Ingenuity TF and LaPET scanners with the default reconstruction for each scanner and for the harmonization strategies described earlier. Tables 5 and 6 show the percent differences between CRC _mean and CRC _max values, respectively, for lesions in the phantom and subjects for the two scanners before and after harmonization. The uncertainties reported in the tables were determined by error propagation of the SEMs of the CRC measurements for the multiple lesions in the phantom, lung, and liver of the four subjects on the two scanners. The RMS _pd of the CRC _mean values for lesions in the phantom and in the lung and liver exceeded 30% before harmonization. Using strategy 1, the RMS _pd values for CRC _mean were reduced to < 8% for both the phantom and subjects; with strategy 2, the RMS _pd values for CRC _mean were reduced to < 5%. The corresponding harmonization of the CRC _max values was not as good, with RMS _pd reduced from > 30% to only < 12% with strategy 1 and < 15% with strategy 2, although the results are consistent between phantom and subject measurements. Using strategy 3, which was based on harmonization of CRC _max values from the phantom measurements, the RMS _pd of CRC _max was reduced to < 6% in the subject data, with the corresponding RMS _pd of CRC _mean reduced to < 14%, demonstrating slightly more variation between phantom and patient measurements, especially for the 10‐mm sphere.

Figure 3.

Comparison of CRC _mean , CRC _peak , and CRC _max values as a function of sphere diameter for phantoms on the Ingenuity TF and LaPET scanners (a) without harmonization and using (b) harmonization strategy 1 (6‐mm Gaussian postfilter applied to the Ingenuity TF), (c) harmonization strategy 2 (4‐mm Gaussian postfilter applied to the Ingenuity TF and LR deconvolution postfilter applied to LaPET), and (d) harmonization strategy 3 (2.5‐mm Gaussian postfilter applied to the Ingenuity TF and LR deconvolution postfilter applied to LaPET). Error bars shown are the standard deviations of the measurements.

Table 5.

Percent differences^a between CRC _mean on the Ingenuity TF and LaPET scanners for lesions embedded in the phantom and lung and liver regions in the subjects before and after harmonization

	10 mm	17 mm	28 mm	RMS pd (%)
Unharmonized
Phantom	−45 ± 2b	−28 ± 1	−16 ± 2	32
Lung	−49 ± 2	−31 ± 2	−21 ± 2	36
Liver	−51 ± 3	−30 ± 2	−19 ± 2	36
Strategy #1
Phantom	1 ± 3	−6 ± 1	−4 ± 2	5
Lung	−3 ± 2	−9 ± 2	−9 ± 2	8
Liver	−6 ± 3	−9 ± 2	−8 ± 2	8
Strategy #2
Phantom	1 ± 3	−1 ± 1	−1 ± 2	1
Lung	−5 ± 2	−4 ± 2	−6 ± 2	5
Liver	−5 ± 3	−2 ± 1	−4 ± 2	4
Strategy #3
Phantom	−13 ± 2	−8 ± 1	−2 ± 3	9
Lung	−19 ± 2	−11 ± 2	−10 ± 2	14
Liver	−19 ± 3	−8 ± 1	−8 ± 2	13

^a% Difference was calculated using Eq. (2).

^bUncertainties shown were determined by error propagation of the SEMs of the CRC values for the two scanners.

CRC _mean , mean contrast recovery coefficient; RMS _pd , root mean squared percent difference calculated over the three sizes.

Table 6.

Percent differences^a between CRC _max on the Ingenuity TF and LaPET scanners for lesions embedded in the phantom and lung and liver regions in the subjects before and after harmonization

	10 mm	17 mm	28 mm	RMS pd (%)
Unharmonized
Phantom	−48 ± 5b	−27 ± 3	−18 ± 3	34
Lung	−46 ± 4	−28 ± 3	−16 ± 2	32
Liver	−49 ± 3	−28 ± 2	−14 ± 2	33
Strategy #1
Phantom	16 ± 6	−2 ± 3	3 ± 3	9
Lung	20 ± 4	0 ± 3	6 ± 2	12
Liver	13 ± 3	−4 ± 2	5 ± 2	9
Strategy #2
Phantom	14 ± 6	12 ± 3	7 ± 3	11
Lung	17 ± 4	14 ± 3	13 ± 2	15
Liver	13 ± 4	12 ± 2	15 ± 2	13
Strategy #3
Phantom	−6 ± 6	5 ± 3	2 ± 4	5
Lung	−4 ± 4	5 ± 3	6 ± 2	5
Liver	−6 ± 3	5 ± 2	8 ± 2	6

^a% Difference was calculated using Eq. (2).

^bUncertainties shown were determined by error propagation of the SEMs of the CRC values for the two scanners.

CRC _max , maximum contrast recovery coefficient; RMS _pd , root mean squared percent difference calculated over the three sizes.

4. Discussion

The average CRC _mean values for the lesions embedded in the phantom on the Ingenuity TF (Table 1) were within 1.7% (range: −0.9% to 1.7%) of those for the measured spheres in the phantom for the 17‐ and 28‐mm spheres; a larger difference (11.4%) was observed between the embedded and measured 10‐mm spheres. Similar results were seen for the LaPET scanner (Table 2), although there was only one phantom measurement. The measured results for CRC _max on the Ingenuity TF also show more variation for the smallest 10‐mm sphere. We have measured little statistical variation in shorter replicate phantom scans (standard error of the mean over 10 3‐min scans: < 3% for CRC _mean and < 4% for CRC _max ); therefore, the larger variability observed in the 30‐min scans for the measured 10‐mm sphere in Table 1 is likely a result of nonstatistical or human errors (e.g., small air bubbles during filling), despite our best efforts to carefully fill the spheres for each acquisition. The results illustrate the trade‐offs associated with harmonization based on measured uptake in small spheres: small spheres are important to measure differences in CRC related to spatial resolution but are more susceptible to nonstatistical errors. The good overall agreement in CRC values between the embedded lesions and the physical spheres and the small variability of CRC values for the embedded lesions in the phantom demonstrate the utility of the lesion‐embedding technique as a proxy for actual lesions in patient studies.

Figure 1 indicates that CRC is consistent across subjects and also shows that CRC is relatively insensitive to differences in background uptake (i.e., lung vs. liver). The lack of intersubject variation supports our treatment of the lesions in all subjects as a single ensemble for each organ. As seen in Table 3, CRC _mean values for the lung and liver for all lesion sizes were within 5% of the values for lesions embedded in the phantom for both scanners, indicating that phantom CRC values with cold wall correction represent patient CRC values well. In addition, the variability of CRC values for lesions embedded in all subjects is of similar magnitude as that observed in the phantom for all sphere sizes.

Figure 2 and Table 4 show that all postfilters studied led to changes in CRC _mean and CRC _max that were consistent across the phantom and subject organs and indicate that phantom measurements can predict the effect of postfilters (e.g., as may be utilized for harmonization) on patient CRC measurements. In addition, application of the LR postfilter on the two scanners led to consistent changes in uptake measurement, especially for CRC _mean . This result is further supported by the results of harmonization between the Ingenuity TF and LaPET scanners in Fig. 3 and Tables 5 and 6. The Ingenuity TF has systematically higher CRC values than the LaPET scanner using the unharmonized reconstruction. This is due primarily to the better spatial resolution of the Ingenuity (4.9 mm vs. 5.8 mm), whereas the better TOF resolution of the LaPET scanner (375 ps vs. 495 ps) has little effect on the CRC and more impact on the precision of the measurement, not explicitly measured in this study. The approach demonstrated in this work for two systems with different spatial and TOF resolutions is general, however, and the results will apply to other scanners, including future generation systems with better spatial and/or TOF resolution.

Both strategies 1 and 3 resulted in good harmonization of both CRC _mean and CRC _max between the two scanners; strategy 2 resulted in the closest harmonization of CRC _mean but poorer harmonization of CRC _max . For CRC _peak , the results after harmonization using the three strategies were in between those for CRC _mean and CRC _max , with strategy 2 showing the poorest harmonization. Although the particular strategies chosen to harmonize CRC _mean for the scanners did not lead to perfect harmonization of all CRC values between the two systems, especially CRC _max , they nonetheless tracked well between phantom and subjects over the range of sphere sizes studied. The strategy selected to harmonize CRC _max (strategy 3) similarly resulted in suboptimal harmonization of CRC _mean values while the phantom and subjects tracked well. The results for CRC _peak also tracked well between the phantom and subjects (not shown).

The ability to harmonize two scanners depends on a number of factors, including the dependence of measured CRC on sphere diameter.⁵ Application of the postfilters to images from the two scanners significantly reduced differences between the two CRC curves, as measured by the RMS _pd , although the differences for the 10‐mm CRC _mean values were not as close for the phantom, lung, and liver lesions. These results indicate that optimal harmonization over a range of lesion sizes (10–28 mm) can be difficult to achieve. In addition, none of the strategies chosen completely harmonized CRC _mean , CRC _peak , and CRC _max metrics together. Although the RMS _pd values for CRC _mean and CRC _max were reduced with all strategies studied, different strategies led to better agreement (lower RMS _pd ), depending on which metric was used for harmonization. This result suggests that harmonization should be done using the clinical metric of interest (i.e., maximum vs. peak vs. mean uptake value).

Figure 3 also shows the impact of harmonization on the accuracy of the uptake measured in the phantom. While harmonization strategy 1 led to better agreement of CRC _mean between the two scanners, the CRC _mean values of the Ingenuity TF dropped by 38, 20, and 11% for the 10‐, 17‐, and 28‐mm spheres, respectively. Strategies 2 and 3 used Lucy‐Richardson deconvolution postfiltering of the LaPET data, so those values increased (+35% for CRC _mean of the 10‐mm sphere) with a decrease (28% for the 10‐mm sphere) in the Ingenuity TF values, due to the use of a Gaussian postfilter. These results show that caution is needed in developing harmonization strategies, since scanner harmonization can be achieved in many ways but with different effects on bias. In addition, different harmonization strategies will also influence the precision of uptake measurements (not studied in this work), and both accuracy and precision will play a role in determining the optimal harmonization strategies.

While we used only a single activity ratio in this study, phantom studies with different activity ratios have shown only small differences in CRC values, and using a different activity ratio would not have affected the choices of strategies for harmonization or the resultant comparisons made in this study between phantom and subject data. We placed embedded lesions in both the lung and liver organs in multiple patients, since we could reliably place multiple lesions in these organs and these two organs represent different background environments. We did not observe systematic differences between lung and liver results and expect the results are representative of lesions in other organs. The variability of the measured sphere results in the phantom, particularly for the 10‐mm sphere, indicates that multiple measurements are critical for phantom‐based harmonization efforts although many phantom measurements can become impractical.

In this work, we used lesions embedded in subject data to demonstrate how well phantom results in harmonization studies will translate to patient studies. The embedded lesions used as surrogates for clinical lesions differ from actual tumors that may not be spherical and do not necessarily have uniform uptake. However, this study represents an important first step toward validating the use of phantoms in the development of harmonization strategies. Based on the good agreement between phantom and subject data in this study, we do not see a need to perform similar patient studies for future harmonization efforts and can rely, instead, on phantom measurements.

5. Conclusions

This study used a lesion‐embedding technique to insert lesions into a phantom and lung and liver regions of subjects scanned on two TOF‐PET scanners to determine the validity of basing scanner harmonization on phantom measurements. The validity of the lesion‐embedding technique for this study was demonstrated by comparing embedded lesions to measured spheres in a phantom and showed good correlation. The viability of using a phantom to track the expected changes with postfiltering, as may be used in harmonization approaches, in subjects was also shown. Our results demonstrate that the phantom CRC _mean performance predicts patient CRC _mean performance with and without postfiltering strategies; phantom CRC _max performance also predicts patient CRC _max performance. These results support the further development of harmonization protocols using phantom studies as their basis. This work also showed the difficulty in simultaneously achieving good harmonization (i.e., RMS _pd under 5% over a range of lesion sizes) of both the CRC _mean and CRC _max metrics using the same postfilters.

Conflict of interest

None.