Accurate and Efficient Measurement of Channelized Hotelling Observer-based Low-contrast Detectability on the ACR CT Accreditation Phantom

Abstract

Background:

Current CT quality control (QC) for low-contrast detectability relies on visual inspection and measurement of contrast-to-noise ratio (CNR). However, CNR numbers become unreliable when it comes to non-linear methods, such as iterative reconstruction and deep-learning-based techniques. Image quality metrics using channelized Hotelling observer (CHO) have been validated to be well correlated with human observer performance on phantom-based and patient-based tasks, but it has not been widely used in routine CT quality control mainly because the CHO calculation typically requires a large number of repeated scans in order to provide accurate and precise estimate of index of detectability (d’).

Purpose:

The main goal of this work is to optimize channel filters and other CHO parameters and accurately estimate the low-contrast detectability with minimum number of repeated scans for the widely used American College of Radiology (ACR) CT accreditation phantom so that it can become practically feasible for routine CT quality control tests.

Methods:

To provide a converged d’ value, an ACR phantom was repeatedly scanned 100 times at three dose levels (24, 12 and 6 mGy). Images were reconstructed with two kernels (FBP Br44 and IR Br44–3). d’ as a function of number of repeated scans was determined for different number of background regions of interest (ROIs), different number of low-contrast objects, different number of slices per each object, and different channel filter options. A reference d’ was established using the optimized CHO setting, and the bias of d’ was quantified using the d’ calculated from all 100 repeated scans. The variation of d’ at each condition was estimated using a resampling method combining random subsampling among 100 repeated scans and bootstrapping of the ensembles of signal and background ROIs.

Results:

Optimized parameters in CHO calculation were determined: 2 background ROIs per object, 4 objects per low-contrast object size, 9 non-overlapping slices per object, and a 4-channel Gabor filter. The bias and uncertainty were estimated at different numbers of repeated scans using these parameters. When only one single scan was used in the CHO calculation, the bias of d’ was below 6.2% and the uncertainty 15.6–19.6% for the 6, 5, and 4-mm objects, while with three repeated scans the bias was below 2.0% and uncertainty 8.7%−10.9% for the three object sizes.

Conclusion:

With optimized parameter settings in CHO, efficient and accurate measurement of low-contrast detectability on the commonly used ACR phantom becomes feasible, which could potentially lead to CHO-based low contrast evaluation in routine quality control tests.

Introduction

Accurate image quality evaluation is critical in determining the minimum radiation dose that is required in a CT protocol without sacrificing diagnostic performance¹. The clinical evaluation by physicians is the standard approach² but it is toilsome, and the results cannot be universally applied to other scanner models and reconstruction algorithms³. Therefore, there is a pursuit of objective and quantitative evaluation methods for diagnostic tasks that are highly correlated with physicians’ performance.

Task-based image quality evaluation using model observers have been proposed to assess the medical image quality^4–6. These metrics have been shown capable of providing objective image quality assessment in CT^7–15. One of the successful model observers is channelized Hotelling observer (CHO)⁴. Tremendous progress towards task-based image quality assessment using CHO has been made during the past decade^{13, 14, 16–18}. Its correlation with human observer performance in low-contrast detection, classification and localization tasks in clinical CT has been demonstrated^{8, 14, 15, 19–22}. Studies have been performed to improve its computational efficiency and inter-laboratory variation^{23, 24}. Despite these progresses, CHO has not been adopted in routine CT quality control, mainly because of the lack of efficient, accurate, and precise implementation. A large number of repeated scans is typically required in CHO calculation to provide accurate and precise estimate of index of detectability (d’)^{24, 25}, which is impractical to implement in routine CT quality control.

In this work, we optimized the measurement and computation of CHO on the widely used American College of Radiology (ACR) accreditation phantom. With the optimized use of channel filter, number of slices per object, number of signal and background regions of interest (ROIs), we determined the minimal number of repeated scans that are required to achieve a reliable estimate of low-contrast detectability in the ACR phantom. This study may enable efficient and accurate measurement of CHO-based low-contrast detectability in routine CT quality control.

Materials and Methods

2.1. Data Acquisition and Preparation

The low contrast module of the ACR phantom was scanned on a Siemens SOMATOM Force scanner. The module has a length of 4 cm in the z direction and contains five object sizes (6, 5, 4, 3, 2 mm) at the same contrast level of 6 HU. Each size has four identical rods. The scan was conducted with 120 kV, rotation time of 0.5 s and helical pitch of 0.6 with the automatic exposure control disabled. Three tube current settings were used: 360, 180 and 90 effective mAs, which translates to volume CT dose index (CTDI_vol) values of 24, 12, and 6 mGy, respectively. 100 scans were repeated at each dose level. Images were reconstructed with 1) 3 mm slice thickness and 0.5 mm slice increment and 2) 0.6 mm thickness and 0.5 mm increment, combined with two reconstruction methods, 1) filtered backprojection (FBP) with a convolution kernel of B44 and 2) iterative reconstruction (IR, SAFIRE, Sinogram Affirmed Iterative Reconstruction, Siemens Healthcare) with a convolution kernel of Br44 and a strength setting of 3. The reconstruction field of view (FOV) was 210 mm with a pixel size of 0.41 mm. Example images reconstructed by FBP and IR methods at each of the three dose levels are shown in Figure 1(a). Three object sizes were studied: 6, 5 and 4 mm. One set of ROIs (red) was extracted for the signal while up to three sets (blue, yellow, and magenta) could be extracted for the background, as seen in Figure 1(b).

Figure 1.

Images of the ACR low contrast module and ROI extraction: a) Sample CT images of low-contrast objects at three different dose levels with FBP (Br44) and IR (Br44–3) reconstructions and 3 mm slice thickness and b) Ensemble averaged image of the low-contrast module with ROIs labelled for objects (red) and background (blue, yellow, and magenta) used in the CHO analysis.

2.2. Figure of Merit

In this work, we adopted the 2D CHO model that has been validated to have a good correlation with the human observer²². The test statistics of the CHO are generally $\lambda ={\omega }_{CHO}\cdot g_c$, where $g_c$ is the channelized image and ${\omega }_{CHO}$ is the template. The template ${\omega }_{CHO}$ can be calculated by a scatter matrix ($s_C$) and the difference between the channelized ensemble average of the signal (${\overline{g}}_{sc}$) and the background (${\overline{g}}_{bc}$),

$${\omega }_{CHO}=S_c^{-1}\left[{\overline{g}}_{sc}-{\overline{g}}_{bc}\right]$$ (1)

Where ${\overline{g}}_{sc}=U^T{\overline{g}}_s$ and ${\overline{g}}_{bc}=U^T{\overline{g}}_b$ with ${\overline{g}}_s$ and ${\overline{g}}_b$ being the ensemble average of the original signal and background images, and $U$ being the matrix representation of the channel filters. The scatter matrix in the Hotelling observer can be expressed as $S_c=\frac{1}{2}\left[K_{sc}+K_{bc}\right]$, where $K_{sc}$ and $K_{bc}$ are the covariance matrices of the channelized images. $K_{sc}$ and $K_{bc}$ can be calculated by the covariance of the signal ($K_s$) and background ($K_b$): $K_{sc}=U^T{K}_{s}U$ and $K_{bc}=U^T{K}_{b}U$.

A variety of channel filter have been developed in the past, including square channels⁵, difference of Gaussians²⁶, Laguerre-Gauss polynomials^27–31, and Gabor channels³⁰. In this study, Gabor filters were applied in the CHO analysis as it has been demonstrated to model human visual system responses³². The single-slice Gabor filter can be expressed as¹³

$$Ga\left(x,y\right)=\text{exp}\left[-\frac{4\left(ln2\right)\left[{\left(x-x_0\right)}^2+{\left(y-y_0\right)}^2\right]}{{\omega }_s^2}\right]\times \text{cos}\left[2\pi f_c\left(x-x_0\right)cos\theta +\left(y-y_0\right)sin\theta +\beta \right]$$ (2)

where ${\omega }_s$ is the channel width in spatial domain, $f_c$ the central frequency, $\theta$ the orientation, and $\beta$ the phase factor. The 2D Gabor filter has the same size as the ROI box and is sampled in a x-y coordinate system with the center being x0 and y0. The passband in the spatial frequency domain is expressed as $\left[f_c-\frac{{\omega }_f}{2}, f_c+\frac{{\omega }_f}{2}\right]$ with ${\omega }_f$ being the bandwidth, and the channel width ${\omega }_s$ can be calculated by ${\omega }_s=\frac{4ln2}{\pi {\omega }_f}$.

After the template ${\omega }_{CHO}$ and the test statistic $\lambda$ were calculated, an internal noise was added as the calibration reference to match the model observer and human observer performance: ${\lambda }^{\prime }=\lambda +\alpha \cdot x$, where $\alpha$ is a calibration factor and $x$ is a zero-mean normally distributed random variable whose standard deviation is square root of the variance of the test statistics for the signal-absent images: ${\sigma }^2=var\left({\omega }_{CHO}^t{g}_{bc}\right)$ ^{22, 24}. The internal noise calibration factor was set to 2.25 based on a previous experiment²².

The signal-to-noise ratio (SNR) of the test statistic, which is also often referred to as index of detectability ($d^{\prime }$), is a commonly employed figure of merit (FOM) to measure the class separability between signal-present and signal absent⁴, and it is defined as

$$d^{\prime }= \frac{{〈\lambda 〉}_1-{〈\lambda 〉}_0}{\sqrt{\frac{1}{2}{\sigma }_1^2+\frac{1}{2}{\sigma }_0^2}}$$ (3)

where ${〈\lambda 〉}_k$ and ${\sigma }_k^2$ (k = 0, 1) are the conditional mean and variance of $\lambda$ under the hypothesis of signal present or signal absent, respectively.

2.3. Determining Factors of Minimum Number of Images

2.3.1. Number of repeated scans

It is well-known that the d’ estimation by fewer samples has a large error bar and bias²⁵. Depending on the sampling method, either hold-out or resubstitution, the bias could be negative or positive. A large number of samples can be obtained in CT by acquiring a large number of repeated scans, assuming each scan is identical except for statistical variations²⁴. The samples of fewer scans were selected by random subsampling from the 100 scans without replacement. The standard deviation of d’ was estimated by using a bootstrapping method, which is performed by randomly sampling (with replacement) all the signal and background ROIs from a given number of scans 500 times. A resubstitution method was used to estimate the d’, where the training and testing were on the same set of images³³.

2.3.2. Selection of background ROIs

While the ROI locations for the signal are fixed, there is some freedom to choose the number of background ROIs as well as their locations. With a pixel size of 0.41 mm, the 6-, 5-, and 4-mm objects would occupy 14.6, 12.2, and 9.8 pixels, respectively. In order to ensure the objects were properly covered by the ROIs, the ROI size was 24 × 24 for the 6 mm rods and 20 × 20 for the 5 mm, and 18 × 18 for the 4 mm rods. In addition, the d’ calculated by one set of background ROIs (blue ROIs in Figure 1) were compared to those by two (blue and yellow ROIs) and three sets (blue, yellow, and magenta ROIs). We picked background ROIs both closer to the center of the phantom and closer to the perimeter because the noise level and texture may differ at different locations of the phantom. The background ROI locations (blue, yellow, or magenta) were chosen by the following rules. The blue and yellow background ROIs were symmetrically placed on two sides of the signal ROIs at a distance of ROI size in the horizontal direction for the 6- and 5-mm signal ROIs and in the vertical direction for the 4-mm objects. For the 5 mm objects, the blue background ROIs were shifted upward by 4 pixels (and downward by 4 pixels for the yellow ROIs) to avoid overlapping with neighboring signal ROIs. The magenta background ROIs were placed on the extended lines of the signal ROIs and the blue background ROIs, and its distance to the nearest signal ROI or blue background ROI is equal to the average distance between the 4 low-contrast objects.

2.3.3. Number of low contrast objects

The number of low contrast objects can also affect the CHO performance. Since each object size consists of four identical rods in the ACR low contrast module, they could be grouped into a larger ensemble, which can potentially improve the accuracy and precision of the d’ estimation. Therefore, the d’ estimated by a larger ensemble with all 4 rods was compared with that by smaller ensembles that only use one of the 4 rods.

2.3.4. Number of slices per object

Each low-contrast rod extends along the longitudinal direction by 4 cm. Multiple slices from each rod can be used to further improve the d’ calculation. The center 2.7 cm of each rod were used to generate multiple slices to avoid potential signal distortion near the edge of the low-contrast module. With a slice thickness of 3 mm, up to 9 non-overlapping slices were extracted.

2.3.5. Gabor filter

Given the limited number of repeated scans, it is desirable to reduce the number of channel filters to improve the bias and precision of CHO computation²⁵. A previously validated 12-channel Gabor filter was selected as the starting point for optimization²², which is composed of four passbands, three orientations, and one phase. The selected passbands were [1/64, 1/32], [1/32, 1/16], [1/16, 1/8], and [1/8, 1/4] cycles/pixel, which correspond to center frequencies of 3/128, 3/64, 3/32, and 3/16 cycles/pixel with bandwidths of 1/64, 1/32, 1/16, and 1/8 cycles/pixel, respectively. The three orientations were 0, π/3, and 2π/3 and the phase was 0. Since the low contrast objects in the ACR phantom have circular symmetry and the object sizes are limited below 6 mm, the high-pass bands in Gabor filter and some orientations might not be necessary for accurate d’ estimation. To minimize the number of passbands, a few different configurations were investigated: (1) two mid passbands ([1/32, 1/16] and [1/16, 1/8]), (2) two wide passbands ([1/64, 1/16] and [1/16, 1/4]), (3) two low passbands ([1/64, 1/32] and 1/32, 1/16]), and (4) one low passband ([1/64, 1/16]) were investigated while a two-orientation option (0 and π/2) and a single orientation (0) were examined for the reduction of orientations.

Results

3.1. Dependency on Number of Repeated Scans

Figure 2 shows the d’ estimated for the three object sizes as a function of number of repeated scans used in the calculation. The images were taken at the 24 mGy dose level and reconstructed with IR Br44–3 and 3 mm slice thickness. These images will be used for all CHO calculations and analysis unless otherwise specified. The image ensemble was formed by the inner set of background ROIs (blue ROIs in Figure 1), one rod per object size, one slice per object (central slice of the rod), and multiple repeated scans. The total number of signal ROIs and background ROIs in this image ensemble are both equal to the number of repeated scans. The CHO was calculated by the 12-channel Gabor filter. Theoretically, a 12-channel filter requires a minimum of 12 pairs of signal-present and signal-absent images to make the scatter matrix S_c invertible. One can see that the d’ estimated by a small number of scans was highly inaccurate accompanied by high uncertainties. As the number of repeated scans increases, the d’ values gradually approach their convergence while the precision of the estimations also steadily improves. The uncertainty in the d’ estimations was caused by the signal and background variations among different scans as well as the internal noise.

Figure 2.

Index of detectability (d’) as a function of number of repeated scans for the 6, 5, and 4 mm low-contrast objects. The d’ was estimated using 12-channel filter on one low-contrast object, one slice per object, and one set of background ROIs.

3.2. Dependency on Background ROIs

Figure 3 compares the d’ as a function of number of repeated scans using one and multiple sets of background ROIs for the 6 mm objects. One can see that use of multiple background ROIs presented significant improvement in CHO computation by producing a more stable estimate of scatter matrix when the number of signal-present images were limited. The first meaningful d’ estimation where the scatter matrix S_c becomes invertible occurred at 12 repeated scans by two backgrounds and 10 by three backgrounds, in contrast to 16 by one set of backgrounds. Furthermore, the d’ converged faster as the number of background ROIs increased, which suggested that fewer scans would be required with multiple background ROIs. However, due to limited space, the third set background ROIs were not in the immediate vicinity of the objects and may introduce extra bias to the d’ measurement. Therefore, two sets of background ROIs (blue and yellow ROIs in Figure 1) were chosen for the CHO analysis in all following sections. The total number of signal ROIs in this image ensemble is equal to the number of repeated scans, and the total number of background ROIs is equal to the number of repeated scans x the number of background ROIs.

Figure 3.

d’ estimated by one (blue ROIs in Figure 1), two (blue and yellow ROIs), and three (blue, yellow, and magenta ROIs) sets of background ROIs with one low-contrast object, one slice per object, and the 12-channel filter for the 6-mm objects.

3.3. Dependency on Number of Low Contrast Objects

The impact of combining the data from 4 objects on d’ estimations is shown in Figure 4. With one slice per object and all 4 objects in the ensemble, the 12-channel CHO could start producing meaningful d’ estimations using as few as 3 scans, and the accuracy of the d’ estimation was significantly improved at lower number of scans. It is worth pointing out that the d’ measured by four objects are similar to those measured by one object at four times the number of scans, as illustrated in Table 1. It is of importance because it suggests that using the large ensemble of all four objects would only require 1/4 the number of scans to achieve the same d’ estimation. In addition, it is also reasonable to think that the d’ curve by one object which yet to reach its convergence would converge to the same value produced by four objects given sufficient scans. The total number of signal ROIs in this image ensemble is equal to the number of repeated scans x 4 objects, and the total number of background ROIs is doubled as each object ROI uses two background ROIs.

Figure 4.

Index of detectability (d’) estimated by a single object and by all four objects for the 6-mm low-contrast rods with one slice per object and the 12-channel filter.

Table 1.

Index of detectability estimated for the 6-mm low-contrast rods with one slice per object and the 12-channel filter using (1) a single object at 12, 20, 50, and 100 repeated scans, and (2) four objects at 3, 5, 12, and 25 repeated scans. Both the d’ and the standard deviation of d’ (denoted by std(d’)) were calculated.

Number of Scans		N = 12	N = 20	N = 50	N = 100
1 object	d’	5.26	4.06	3.50	3.34
	std(d’)	1.82	0.87	0.40	0.29
Number of Scans		N = 3	N = 5	N = 12	N = 25
4 objects	d’	5.20	3.85	3.38	3.20
	std(d’)	1.84	0.77	0.39	0.25

3.4. Dependency on Number of Slices per Object

After incorporation of all four rods for each size of the low-contrast objects, Figure 5 demonstrates the improvement of d’ estimations by using multiple non-overlapping slices per object. The total number of signal ROIs in this image ensemble is equal to the number of repeated scans x 4 objects x number of slices, and the total number of background ROIs is doubled as each object uses two background ROIs. Quantitatively, when the difference of d’ between N = 50 and N = 100 is within 1.0% (flatten out), we consider that the d’ at N = 100 is a good representation of the converged value. For example, when all 4 objects, 2 background ROIs per object, and 9 non-overlapping slices per scan were used, the relative difference in d’ between N = 50 and N = 100 were below 0.2% for all the 6-, 5-, and 4-mm objects, which clearly met the criterion, and the d’ at N = 100 were considered converged. The d’ estimations and standard deviation of d’ are given for the 6-mm objects at different number of slices in Table 2. After including 9 non-overlapping slices, the d’ estimates approximated the Gaussian distribution, the average d’ measured by three repeated scans only over-estimates over its converged value by 3.0% for the 6 mm object.

Figure 5.

Index of detectability (d’) estimated for the 6-mm object by 1, 3, and 9 non-overlapping slices.

Table 2.

Index of detectability estimated for the 6-mm low-contrast rods with 4 objects and the 12-channel filter as well as 1, 3, and 9 slices per object. Both the d’ and the standard deviation of d’ (denoted by std(d’)) were calculated.

Number of scans			N = 3	N = 6	N = 9	N = 15	N = 25	N = 50	N = 100
1 slice	d’	5.20		3.71	3.45	3.34	3.20	3.15	3.11
	std(d’)	1.84		0.67	0.52	0.42	0.25	0.17	0.12
Number of scans		N = 1		N = 2	N = 3	N = 5	N = 10	N = 25	N = 100
3 slices	d’	5.08		3.72	3.48	3.32	3.22	3.15	3.12
	std(d’)	1.67		0.65	0.55	0.37	0.25	0.14	0.07
9 slices	d’	3.37		3.16	3.12	3.08	3.05	3.03	3.03
	Std(d’)	0.52		0.32	0.26	0.19	0.14	0.09	0.04

3.5. Dependency on Channel Filter

The use of 4 low contrast objects and 9 slices per object have increased the accuracy of d’ estimations at low number of scans, but there is still room for improvement by optimizing the channel filter. Six other channel-filter options, including 4 passbands and 2 orientations, 4 passbands and 1 orientation, 2 mid passbands and 2 orientations, 2 wide passbands and 2 orientations, 2 low passbands and 2 orientations, and 1 low passband and 2 orientations were compared with the previously validated filter choice of 4 bands and 3 orientations, all of which can be visualized in Figure 6. Figure 7(a) shows the d’ estimated by these filters with 9 slices per object and 100 repeated scans. The total number of signal ROIs in this image ensemble is equal to 4 objects x 9 slices x 100 repeated scans, and the total number of background ROIs is doubled as each object uses two sets of background ROIs. Compared to the initial 4 passbands and 3 orientations (d’ = 3.03), there was a slight loss of detectability by 4 passbands and 2 orientations (d’ = 2.91) and a substantial one by 1 orientation (d’ = 2.18). Therefore, the number of orientations could be reduced to 2. Among the 2-orientation choices, the 2 mid passbands presented a large drop in d’ at the 6-mm objects (d’ = 2.34) while the 2 low passbands (d’ = 2.88), 2 wide passbands (d’ = 2.96) and 1 low passband (d’ = 2.94) maintained the d’ levels of those by 4 passbands. However, only the 2 low passband filter was eventually selected because both 2 wide passbands and 1 low passband do not work well for relatively bigger object sizes (e.g., 12 mm), which are typically used in some of the other low-contrast testing phantoms. We optimized the CHO for the ACR phantom, but meanwhile we expect this CHO model would also work for some similar low-contrast phantoms that may have low-contrast objects bigger than 6 mm.

Figure 6.

Channel filter settings investigated in this work: i) 4 passbands and 3 orientations, ii) 4 passbands and 2 orientations, iii) 4 passbands and 1 orientation, iv) 2 wide passbands and 2 orientations, v) 2 mid passbands and 2 orientations, vi) 2 low passbands and 2 orientations, and vii) 1 low passband and 2 orientations.

Figure 7.

(a) Index of detectability (d’) estimated by various channel filter choices: i) 4 passbands and 3 orientations, ii) 4 passbands and 2 orientations, iii) 4 passbands and 1 orientation, iv) 2 wide passbands and 2 orientations, v) 2 mid passbands and 2 orientations, vi) 2 low passbands and 2 orientations, and vii) 1 low passband and 2 orientations. The three dotted lines from top to bottom represent the d’ values of the first point (4 passbands and 3 orientations) for the 6-, 5-, and 4-mm objects, respectively. (b) d’ estimated by 4 passbands and 2 orientations and 2 low passbands and 2 orientations channel filters with 9 non-overlapping slices per object.

Although the selected channel filter only contributed to limited improvement in the precision of a d’ measurement, it could largely increase the d’ accuracy. Figure 7(b) shows the improvement in d’ accuracy with 9 slices per object. Compared to the 4 passbands and 2 orientation channel filter setting, the d’ calculated by the optimized 2 passbands and 2 orientations setting became closer to its converged value by 30.5% with a single scan and 27.4% with three repeated scans.

3.6. Optimized CHO Setting and Minimum Number of Scans

The optimized CHO setting for the ACR phantom was established by a large ensemble of 4 low contrast objects and 9 slices per object combined with the 4-channel Gabor filter of 2 low passbands and 2 orientations. Under the optimized CHO setting, d’ references for the 4-, 5-, 6-mm objects were calculated using 100 repeated scans, and the overall improvement was quantified in Table 3. With the optimized CHO setting, the estimation bias could be reduced to below 5.6% of the reference d’ value for the 6-mm objects using a single scan and to below 2% if three repeated scans were used. Figure 8 shows the histogram of d’ estimations by the initial CHO setting of 1 low contrast object, 1 slice per object, and the 12-channel filter with 12 repeated scans, and by the optimized CHO setting with 1 and 3 repeated scans. The d’ distribution was simulated by bootstrapping the signal and background ROIs from the selected scans as described in Section 2.3.1. One can clearly see that not only did the initial CHO setting overestimated the detectability, but there was also great uncertainty of the measurement for all object sizes. By contrast, the optimized CHO setting provided highly accurate and precise d’ estimations. For the 6 mm objects scanned at 24 mGy, d’ estimated by 3 repeated scans only has a bias of 1.83% and there is a 95.4% probability that it would lie within two standard deviations (± 2 × 8.7%) of the mean. If an even higher accuracy and precision were desired, one could increase the number of repeated scans.

Table 3.

Bias and RSD of d’ estimated for the low contrast objects at 24 mGy by the optimized CHO setting with 4 low-contrast objects, 9 slices per object, and the 4-channel Gabor filter.

Number of scans			N = 1	N = 2	N = 3	N = 5	N = 10
Optimized CHO	6 mm	Bias	5.53%	3.07%	1.83%	1.18%	0.62%
		RSD	15.6%	10.9%	8.70%	6.63%	4.70%
	5 mm	Bias	5.84%	2.76%	1.62%	0.81%	0.39%
		RSD	16.9%	11.9%	9.36%	7.03%	5.32%
	4 mm	Bias	6.21%	3.00%	2.02%	1.14%	0.36%
		RSD	19.6%	14.0%	10.9%	8.79%	5.84%

Figure 8.

Histogram of d’ measurements for the low-contrast objects at 24 mGy under three different settings: 1) the initial 1 low contrast object, 1 slice per object, and the 12-channel filter with 12 repeated scans; 2) the initial 1 low contrast object, 1 slice per object, and the 12-channel filter with 12 repeated scans; 3) optimized CHO setting with 1 single scan; and 4) optimized CHO setting with 3 repeated scans.

3.7. Optimized CHO Setting Applied to Various Dose Levels and Reconstructions

The optimized CHO setting was applied to various dose levels and reconstructions with three repeated scans. Figure 9 shows that the d’ values increase as dose level for all object sizes, the 3 mm slices present higher detectability than the 0.6 mm slices, and the iterative reconstructions produce slightly higher d’ than the FBP method. The IR method could increase the detectability level by 11.5% and 27.6% while increasing the RSD by 2.1% and 4.1% in the 3-mm and 0.6-mm slices, respectively.

Figure 9.

The optimized CHO setting applied to various dose levels and reconstructions. (a) d’ at 3 different dose levels for the 3 low-contrast objects (6, 5, and 4 mm). Images were reconstructed by the IR method (Br44–3 kernel) and 3 mm slice thickness. (b) d’ at 3 different dose levels for different reconstruction settings: 3 mm slice thickness with both the FBP method (Br44 kernel) and IR method (Br44–3 kernel); 0.6 mm slice thickness with both the FBP method (Br44 kernel) and IR method (Br44–3 kernel). Only the results of the 6 mm low-contrast object were shown.

Discussion

As one of the most popular choices of model observers, CHO has been validated to correlate well with human observer performance in many low-contrast detection and localization tasks^{8, 14, 15, 22}. It has a great potential to provide an objective and quantitative measure of low–contrast detectability. However, its adoption in routine CT quality control and performance evaluation has been quite limited, partially because of the lack of an efficient, accurate, and precise implementation. A large number of samples is required in CHO calculation to provide an accurate and precise estimate of index of detectability (d’)^{24, 25}. In clinical CT, a large number of samples are typically obtained using many repeated scans, which is impractical to implement in routine quality control. In this study, we attempt to optimize the CHO calculation for the most widely used ACR CT accreditation phantom, including how to select background ROIs, number of low contrast objects, number of slices per object, and CHO channel filter. After selecting an optimized set of CHO parameters, we expect to provide an efficient, accurate, and precise method for measuring low-contrast detectability for routine CT quality control.

As expected, our results showed that increasing the number of scans, number of slices and number of objects in the ensemble could significantly reduce the d’ estimation bias and uncertainty. The finding about the number of scans is consistent with our previous results²⁴, and the other two findings about the number of slices and number of objects in the ensemble could be easily understood following the same principle.

The fundamental reason why a large number of samples are needed is to reduce the statistical variation of the signal and noise from one single image sample. The low-contrast detectability from any two measurements (at exactly the same scan and reconstruction condition) could differ tremendously. Figure 10 demonstrates how visibility of low-contrast objects could vary between two different scans (at identical slice location). Quantitative measurement or visual inspection based on one single slice from one single scan is only one of the noise and signal realizations, which may not reflect the true imaging performance of the system at a given data acquisition and reconstruction condition. To reduce this variation, ideally, one would need a large number of repeated scans to generate a large number of sample images at an identical location surrounding a low-contrast object, which is impractical to implement in practice. Fortunately, the ACR CT phantom has 4 low-contrast cylindrical objects at each size and these 4 objects of each size were designed to be identical, placing at very close locations. Therefore, we were able to utilize these objects (at different z-locations or x-y locations) to mimic a large number of samples for the same low-contrast object using only very few scans.

Figure 10.

The signal and background of the 6-mm rods at 24 mGy with 3-mm slice thickness from two different slices. Display window width/level 60/100 HU. The arrows point to the objects that became visibly missing.

The 4 rods of each object size are identical in terms material, density, and manufacturing process. The manufacturers of the ACR CT accreditation phantom, such as Gammex, are required to go through strict quality assurance to make sure those low-contrast rods are within a predefined specification. In principle, those 4 rods can be considered as 4 realizations of the same signal. By grouping the 4 objects together, a larger ensemble of the signal can be created so that the d’ can converge much faster. In reality, the introduction of 4 objects cannot be completely independent realizations and the misalignment of ROIs for 4 objects may affect the d’ values. However, its effect on d’ is believed to be quite small. Compared to d’ obtained by using 1 object (100 scans, 9 slices per scan, and the initial 12-channel filter), the d’ by 4 objects under the same condition were lowered by only 3.4%, 1.3%, and 1.6% for the 6-, 5-, and 4- mm objects, respectively. Therefore, the misalignment of 4 signal ROIs is not expected to be a major limiting factor for the practical use of the proposed method for calculating the CHO-based low-contrast detectability in the context of quality control testing. Another concern about misalignment of 4 objects is that it could result in a smoother version of the template, which may reduce its response to high passbands of the channel filter. However, we believe the effect is quite small as well. Compared to the d’ obtained by 1 object (100 scans, 9 slices per scan, and the 4-channel filter: 2 low passbands and 2 orientations), the d’ by 4 objects were lowered by 3.2%, 0.8%, and 0.9% for the 6-, 5-, and 4- mm objects, respectively. There is also a possibility of manufacturing imperfection, along with other scanning and reconstruction related factors that can cause small variations among the 4 rods. This variation is included in the error bar of the d’ value for each object size as a single index for low-contrast detectability. Beyond that, there are some variations among different ACR phantoms due to manufacturing process. If the proposed method is going to be used for quality control purpose, then additional work is required to determine what the acceptable range of d’ is at each object size for a large number of ACR phantoms, taking into account both the requirement of the diagnostic task (abdomen or head routine protocols) and the variations among phantoms.

Similarly, the misalignment across multiple slices could also affect the converged d’. However, its effect is also limited as can be seen from Table 2 and Figure 11, the latter of which compares the d’ values using 1 single slice vs. multiple non-overlapping slices at 100 scans. The converged values with 9 slices only differ from 1 single slice by 2.8%, 2.9%, and 4.1% for the 6-, 5-, and 4-mm objects, respectively. Note that the relative standard deviation with 1 single slice at 100 scans were 3.9%, 4.1%, and 4.6% for the three object sizes, respectively. Therefore, misalignment should not be a major limiting factor for the practical use of the proposed method for calculating CHO-based low-contrast detectability.

Figure 11.

Index of detectability (d’) estimated by 1–9 non-overlapping slices at 100 scans with the initial 12-channel Gabor filter and 4 objects per object.

One of the assumptions of CHO is independent realizations of signal and noise, and thus non-overlapping slices were used in the study. However, increasing the number of slices beyond 9 non-overlapping slices may bring benefits in reducing the number of repeated scans. As shown in Figure 12, by using 20 and 50 overlapping slices evenly distributed across the selected 2.7 cm range of the ACR low-contrast module, the converged d’ value remained the same while the accuracy was improved at very limited number of scans. The drawback of using overlapping slices is that it breaks CHO’s assumption of independent realizations, and the seemingly improved precision would no longer be reliable, and estimation of the real precision would require truly independent measurements in a repeatability test.

Figure 12.

Index of detectability (d’) estimated for the 6-mm object by 9 non-overlapping slices and 20 and 50 overlapping slices evenly distributed across the 2.7 cm range of the ACR low-contrast module.

Using the detectability levels established by the 12-channel Gabor filter which was validated previously using human observer²², it was found that the number of channel filters for the ACR low contrast module could be reduced to 4 without sacrificing the accuracy. This finding is not surprising as the passband selection is dependent on the low-contrast object size. For the ACR low-contrast objects, the two passbands selected are sufficient to cover the frequency range of those objects. Adding more passbands has negligible contribution to the estimated detectability value, but will impose more need for training the template, which requires more image samples. It is also important to reiterate that the optimized channel filters could greatly improve the d’ accuracy but have only limited contribution to the precision.

The d’ estimation also depends on the ROI size³⁴. A reliable d’ could only be obtained with a sufficiently large ROI. As the number of channel filter increases, bigger ROIs are required. In a previous study by Ferrero et al.³⁴, a 40-channel would require an ROI size of 44 while a 12-channel filter required 14 for a reliable d’ prediction. In this study, the channel filter considered were from 2 to 12, the ROI sizes (28 × 28 for the 6-mm objects, 20 × 20 for the 5- and 4-mm objects) were well above the threshold and did not expect to affect the d’ estimations.

Mathematical modeling for CHO-based index of detectability have been presented under various types of noise and they demonstrated that bias and uncertainty of the estimator can be reduced if either the mean of the two classes or the difference of the two classes is known^{35, 36}. However, those models are not readily applicable to our work mainly because the class means or their differences are difficult to obtain in the experimental setting of measuring d’ of the ACR low-contrast objects. With the use of 2 sets of background ROIs, 4-low-contrast objects, 9 slices per object, and the 4-channel Gabor filter, d’ converged very rapidly and the reference d’ can be achieved in an experimental setting.

The minimum number of scans required should be determined by the accuracy and precision of the d’ measurement. With the selected CHO setting, a bias of below 5.6% was achieved for the 6 mm object size using a single scan and below 2% using three repeated scans, as shown in Table 3. In routine quality control, typically only one scan is acquired. With one single scan, the d’ estimated by the selected CHO setting has a 95.4% probability to be within ± 15.6% of the reference d’ for the 6 mm object size. If 3 repeated scans are acquired, then the estimated d’ is within ±8.7% of the reference d’ (95% confidence interval). These results showed that, with the proposed method, it is practically feasible to measure the low-contrast detectability in the ACR CT phantom in an efficient, accurate and precise manner.

Iterative reconstruction has shown slightly higher detectability over FBP in the overall d’ level at all dose levels. The fact that the average detectability level were improved by 11.5% and 27.6% in the 3-mm and 0.6-mm slices are consistent with the state-of-the-art studies on the low-contrast evaluation of IR methods, where IR could allow approximately 25% dose reduction without degradation of low-contrast detectability³.

Conclusion

We optimized four key factors that could significantly improve the CHO performance for the ACR phantom, including the number of background ROIs, the number of low contrast objects, the number of slices per object, and the channel filter selection. The optimized CHO setting consists of 2 sets of background ROIs, 4 objects of each object size, 9 slices per object, and a 4-channel Gabor filter. With the optimized CHO setting, highly accurate and precise d’ estimates were achieved using only a few scans. If only one single scan was used, the bias of d’ was below 6.2% and uncertainty 15.6%−19.6% for the 6-, 5-, and 4-mm objects, while with three repeated scans the bias was below 2.0% and there is a 95.4% probability that a d’ estimate would lie within (± 2 × 8.7%−10.9%) of the respective reference d’ values for the three object sizes. This study could potentially enable CHO-based low contrast evaluation in routine quality control tests.

Data availability statement

Authors will share data upon reasonable request to the corresponding author.