Design of a practical model-observer-based image quality assessment method for x-ray computed tomography imaging systems

Abstract.

The use of a channelization mechanism on model observers not only makes mimicking human visual behavior possible, but also reduces the amount of image data needed to estimate the model observer parameters. The channelized Hotelling observer (CHO) and channelized scanning linear observer (CSLO) have recently been used to assess CT image quality for detection tasks and combined detection/estimation tasks, respectively. Although the use of channels substantially reduces the amount of data required to compute image quality, the number of scans required for CT imaging is still not practical for routine use. It is our desire to further reduce the number of scans required to make CHO or CSLO an image quality tool for routine and frequent system validations and evaluations. This work explores different data-reduction schemes and designs an approach that requires only a few CT scans. Three different kinds of approaches are included in this study: a conventional CHO/CSLO technique with a large sample size, a conventional CHO/CSLO technique with fewer samples, and an approach that we will show requires fewer samples to mimic conventional performance with a large sample size. The mean value and standard deviation of areas under ROC/EROC curve were estimated using the well-validated shuffle approach. The results indicate that an 80% data reduction can be achieved without loss of accuracy. This substantial data reduction is a step toward a practical tool for routine-task-based QA/QC CT system assessment.

1. Introduction

For detection tasks and combined detection/estimation tasks, in which the observer must detect a signal and estimate parameters of the signal (e.g., contrast and size), the channelized Hotelling observer (CHO)^1–11 and the channelized scanning-linear observer (CSLO)¹² have successfully been utilized to measure image quality. To implement these two observers, first-order and second-order statistics are needed, which are typically estimated using sample images from a training dataset. For an $M$-element regions of interest (ROI), without channelization, the inverse of the sample covariance matrix exists only when the number of images in the training dataset is larger than $M+1$. In a CT imaging system, however, it is difficult to have a training dataset larger than $M+1$ because $M$ is usually very large—even for a small ROI. The process of channelization in which the image data are processed by a small number of channels to produce channel outputs provides a solution to this problem. The number of channels is typically much less than $M$. For instance, the number of dense-difference of Gauss (DDOG)¹³ channels selected in this study is 10, so the dimension of each channelized sample becomes 10, whereas the ROI size might contain $M{=100\times 100=10}^4\mathrm{pixels}$. Thus, the minimum required training size reduces to 11 instead of ${10}^4+1$ in this specific example. The channelization makes the CHO and CSLO possible for limited datasets in CT systems. Although the channelized model observers become possible tools in CT imaging systems, the number of required images to estimate both the mean vector and the covariance matrix accurately is still quite large. Thus, conventional channelized model observers cannot be used as a quality assurance tool for routine system validation. In this study, an approach is proposed that reduces the required number of CT images and enables routine system evaluations using model observers for task-based assessments of image quality.

The goal of this study is to explore different methods to reduce data demand for computing CHO/CSLO performance and propose a practical model-observer-based approach for potential routine QA/QC of CT imaging systems. We will use high-dose images to estimate the first-order statistics, i.e., the mean signal image, and use the leave-one-out covariance (LOOC)¹⁴ method to estimate the second-order statistics, i.e., the covariance matrices.

2. Materials and Methods

2.1. Real Data Preparation

In this work, low contrast (LC) objects embedded in four different phantoms were imaged on a GE Discovery CT750 HD CT system. Two of the phantoms were QRM-LC-FD1 and QRM-LC-FD4 phantoms (QRM Quality Assurance in Radiology and Medicine GmbH, Moehrendorf, Germany) and the other two were MITA CCT 183 and MITA CCT 189 phantoms (The Phantom Laboratory, Salem, New York). The LC objects embedded in each phantom varied in size, contrast, or both. In this study, we used them differently for different study tasks. The MITA CCT 183 phantom was used for pure detection tasks and the validation of our proposed ideas, the QRM-LC-FD1 was used for combined detection and size estimation tasks, the QRM-LC-FD4 was used for combined detection and contrast estimation tasks, and the MTIA CCT 189 phantom was used for a combination of detection, size and contrast estimation tasks. The properties of LC objects in these four phantoms are listed in Tables 1–4. Only axial scans were used in this work. The slice thickness was 0.625 mm and the collimator aperture used was 20 mm. The x-ray voltage was kept at 120 kVp and current range was tuned to ensure that the task being performed was neither too difficult nor too easy.

Table 1.

MITA CCT183 phantom.

Object	Diameter (mm)	Contrast relative to background (HU)
1	3	14
2	5	7
3	7	5
4	10	3

Table 2.

QRM-LC-FD1 phantom.

Object	Diameter (mm)	Contrast relative to background (HU)
1	3	$-2$
2	5	$-2$
3	7	$-2$
4	10	$-2$

Table 3.

QRM-LC-FD4 phantom.

Object	Diameter (mm)	Contrast relative to background (HU)
1	5	$-5$
2	5	$-10$
3	5	$-25$
4	5	$-50$

Table 4.

MITA CCT189 phantom.

Object	Diameter (mm)	Contrast relative to background (HU)
1	3	14
2	5	7
3	7	5
4	10	3
5	15	14
6	15	7
7	15	5
8	15	3

Various radiation dose protocols were used and for each dose level, 50 repeat scans were acquired. The image reconstruction algorithm used in this study was filtered backprojection (FBP). From each of these scans, 10 individual realizations of signal and noise ROI were extracted from each scan for each different low-contrast signal. Thus, there were 500 individual image pairs for each LC object at every dose level. The images were reconstructed at a field of view of 180 mm with a matrix size of $512\times 512$ image pixels. ROIs with a size of $100\times 100\text{pixels}$ containing the LC object under study in the center (signal-present case) and the noise (signal-absent case) were extracted from the image slices, which is about $35\mathrm{mm}\times 35\mathrm{mm}$ in size.

2.2. Simulation Data Preparation, Analytical Solution, and Baseline

Since the number of acquired images is always limited in experiments, simulation data were generated to find the baseline performance and help understand the variability in our methods. Because the two key components, first- and second-order statistics, needed in our observer models are the same for pure detection tasks and combined detection and estimation tasks, without loss of generality, our simulation was focused on the case of pure detection task. Simulated image pairs for signal-absent and signal-present classes are given by

$$\text{signal}-\text{absent image}{\mathbf{g}}_1=\mathbf{n}+\mathbf{b}=\mathbf{HW}+\mathbf{b},$$ (1)

$$\text{signal}-\text{present image}{\mathbf{g}}_2=\mathbf{n}+\mathbf{b}+\mathbf{s}=\mathbf{HW}+\mathbf{b}+\mathbf{s},$$ (2)

where subscript index 1 and 2 denote signal-absent and signal-present, respectively. The signal $\mathbf{s}$ is circular LC objects with blurred edges. The term $\mathbf{b}$ is the phantom background and was estimated using real signal-absent images. The noise structure $\mathbf{n}$ was simulated using correlated Gaussian noise. The convolution of a blurring filter with white Gaussian noise can be represented with matrix operator $\mathbf{H}$ and a vector $\mathbf{W}$, respectively. The matrix $\mathbf{H}$ can be generated from the autocorrelation function ${\mathbf{R}}_{\mathbf{G}}$ that was estimated using real images. ${\mathbf{R}}_{\mathbf{G}}$ is given by

$${\mathbf{R}}_{\mathbf{G}}=\frac{1}{N}\sum _{n=1}^N{\mathbf{g}}_{1,n}^{\mathrm{real}}{\mathbf{g}}_{1,n}^{\mathrm{real}}=⟨{\mathbf{g}}_1{\mathbf{g}}_1^t⟩=⟨(\mathbf{H}\mathbf{W}+\mathbf{b}){(\mathbf{H}\mathbf{W}+\mathbf{b})}^t⟩={\sigma }^2\mathbf{H}{\mathbf{H}}^t+\mathbf{b}{\mathbf{b}}^t,$$ (3)

where superscript real signifies real image data, $t$ denotes the transpose operation, and $\sigma$ is the standard deviation estimated from real image data. ${\mathbf{R}}_{\mathbf{G}}$, $\sigma$, and $\mathbf{b}$ are all known and dose level dependent. The blurring kernel $\mathbf{H}$ is calculated by taking the square root of $\frac{\mathbf{1}}{{\sigma }^2}({\mathbf{R}}_{\mathbf{G}}-\mathbf{b}{\mathbf{b}}^{\mathbf{t}})$.

The figure-of-merit (FOM) of this study is the area under the ROC curve (AUC). The analytical solution of AUC can be calculated through the noise covariance matrix.¹⁵ The noise covariance ${\mathbf{K}}_{\mathbf{n}}$ can be determined via

$${\mathbf{K}}_{\mathbf{n}}=⟨\mathbf{n}{\mathbf{n}}^t⟩=⟨(\mathbf{H}\mathbf{W}){(\mathbf{H}\mathbf{W})}^t⟩=⟨\mathbf{H}\mathbf{W}{\mathbf{W}}^t{\mathbf{H}}^t⟩={\sigma }^2\mathbf{H}\mathbf{I}{\mathbf{H}}^t={\sigma }^2\mathbf{H}{\mathbf{H}}^t={\mathbf{R}}_{\mathbf{G}}-\mathbf{b}{\mathbf{b}}^{\mathbf{t}}.$$ (4)

The AUC can be approximated through the SNR of the test statistics using

$$\mathrm{SNR}=\sqrt{{\mathbf{s}}^t{\mathbf{K}}_{\mathbf{n}}^{-1}\mathbf{s}}=\sqrt{{\mathbf{s}}^t{({\mathbf{R}}_{\mathbf{G}}-\mathbf{b}{\mathbf{b}}^{\mathbf{t}})}^{-1}\mathbf{s}},$$ (5)

$$\mathrm{AUC}=\frac{1}{2}[1+\mathrm{Erf}(\frac{\mathrm{SNR}}{2}\left)\right],$$ (6)

where SNR is the signal-to-noise ratio, and Erf is the error function. Since ${\mathbf{R}}_{\mathbf{G}}$ is a function of dose level so the SNR and AUC are also functions of dose level.

To validate the simulated images, the radial average noise power spectrum (NPS)¹⁶ profile of the simulation and the real images were compared. The comparison (Fig. 1) indicated that the simulated noise background is very similar to the real noise background. An example of a simulated image and a real image is shown in Fig. 2. The analytically derived AUC values (solid line) were compared with the AUC values calculated using the CHO method from 20,000 pairs of simulation images (dashed line in Fig. 3), and the comparisons demonstrated that the above derivation of the analytical solution is reasonable.

Fig. 1.

Radial average NPS profile at 120 kVp 80 mAs: simulation data (black-circle solid line) and real data (black-dashed cross line).

Fig. 2.

Signal-present image averaged by 500 images at 120 kVp 80 mAs: (a) simulation image and (b) real image.

Fig. 3.

Signal-absent image at 120 kVp 80 mAs: (a) simulation image and (b) real image.

To better understand any biases present while using a limited number of real data, we compared the AUC values generated using finite numbers of simulated images with the AUC values generated by the analytical solution and 20,000 pairs of simulated images. We found that the AUC values of CHO using 240 training images and 260 testing images (solid line with triangle) were very close to both the analytical solution (solid line) and the 20,000 image pairs CHO (dashed line) for all four objects with accuracy ranging from 98.25% to 99.87% (Fig. 4). This high accuracy in the simulation study indicates that 500 real image pairs will result in negligible bias and can be used as the baseline for the rest of our experimental study.

Fig. 4.

Comparisons of AUC values of analytical solutions versus CHO using 20,000 images versus CHO using 500 images. (a) Object 1: 3 mm 14 HU, (b) object 2: 5 mm 7 HU, (c) object 3: 7 mm 5 HU, and (d) object 4: 10 mm 3 HU.

2.3. Traditional Approach

2.3.1. Channelized Hotelling observer

The CHO approach requires a relatively large amount of training data to estimate the means and covariance matrices to generate accurate estimates of task performance. The image template calculated by the CHO is given by

$${\mathbf{W}}_{\mathrm{CHO}}^t={\overline{\mathbf{s}}}^t{\mathbf{S}}^{-1},$$ (7)

where $\overline{\mathbf{s}}$ is the sample estimated mean signal and $\mathbf{S}$ is the common covariance matrix defined by averaging the sample covariance matrices of the two classes. They are computed by the following equations:

$${\mathbf{x}}_i={\mathbf{T}}^{\mathbf{t}}{\mathbf{g}}_i,$$ (8)

$${\mathbf{m}}_i=\frac{1}{N_i}\sum _{k=1}^{N_i}{\mathbf{x}}_{i,k},$$ (9)

$$\overline{\mathbf{s}}=({\mathbf{m}}_2-{\mathbf{m}}_1),$$ (10)

$${\mathbf{\Sigma }}_i=\frac{1}{N_i-1}\sum _{k=1}^{N_i}({\mathbf{x}}_{i,k}-{\mathbf{m}}_i){({\mathbf{x}}_{i,k}-{\mathbf{m}}_i)}^t,$$ (11)

$$\mathbf{S}=\frac{1}{L}\sum _{i=1}^L{\mathbf{\Sigma }}_i,$$ (12)

where subscript index $i=\mathrm{2,1}$ refers to signal-present and signal-absent classes, respectively, and ${\mathbf{m}}_i$ is the mean of channelized data set ${\mathbf{x}}_i$ for $i^{\prime }$th class. In this paper, $\mathbf{T}$ is the 10 DDOG channels proposed by Abbey and Barrett,¹⁷ ${\mathbf{g}}_i$ is the image vector before channelization, ${\mathbf{\Sigma }}_i$ is the sample covariance matrix for the $i$th class, and $L$ is the number of classes. Since there are only two classes in the pure detection task, $L=2$ for the detection task.

2.3.2. Channelized scanning-linear observer

Similar to the CHO, the first- and second-order statistics need to be calculated before generating the FOM for the CSLO. The concept of scanning-linear estimation¹³ involves the calculation of the mode of the posterior density or to approximate maximum a posteriori (MAP) estimation. This calculation requires a scan over parameter space to compare solutions and find the maximum. The general formula of MAP estimation is

$$\begin{gathered} {\hat{\mathbf{\theta }}}_{\mathrm{MAP}}={\mathrm{argmax}}_{\theta }[\mathrm{pr}(\mathbf{\theta }|\mathbf{g})] \\ ={\mathrm{argmax}}_{\theta }[\frac{\mathrm{pr}(\mathbf{g}|\mathbf{\theta })\mathrm{pr}(\mathbf{\theta })}{\mathrm{pr}(\mathbf{g})}], \end{gathered}$$ (13)

where the $\mathbf{\theta }$ and $\hat{\mathbf{\theta }}$ are the parameters we are interested in and their estimated parameters, respectively, $\mathrm{pr}(\mathbf{g}|\mathbf{\theta })$ is the likelihood of data conditioned on parameters, $\mathrm{pr}(\mathbf{\theta })$ and $\mathrm{pr}(\mathbf{g})$ are the likelihood of parameter $\mathbf{\theta }$ and dataset $\mathbf{g}$. To easily optimize this function, we will consider the Gaussian likelihood as was done in Ref. 13. Note this approximation does not imply that joint pdf $\mathrm{pr}(\mathbf{g},\mathbf{\theta })$ is also Gaussian. Based on this approximation, the conditional likelihood $\mathrm{pr}(\mathbf{g}|\mathbf{\theta })$ can be described by

$$\mathrm{pr}(\mathbf{g}|\mathbf{\theta })\cong \frac{1}{\sqrt{2{\pi }^M\det ({\mathbf{K}}_{\mathbf{g}|\mathbf{\theta }})}}\exp \{\frac{-1}{2}{[\mathbf{g}-\overline{\mathbf{g}}(\mathbf{\theta })]}^t{\mathbf{K}}_{\mathbf{g}|\mathbf{\theta }}^{-1}[\mathbf{g}-\overline{\mathbf{g}}(\mathbf{\theta })]\},$$ (14)

where $\overline{\mathbf{g}}(\mathbf{\theta })$ is the mean image averaged over the parameters $\mathbf{\theta }$, ${\mathbf{K}}_{\mathbf{g}|\mathbf{\theta }}$ is the sample covariance matrix conditioned on the parameters, and $\det (·)$ is the determinant of the matrix. Instead of evaluating the covariance matrix and its inverse for every parameter, our second approximation is to use the mean of ${\mathbf{K}}_{\mathbf{g}|\mathbf{\theta }}$ averaged over all $\mathbf{\theta }$. To avoid the exponential term of Eq. (14), the common strategy is to operate the natural logarithm on both sides and ignore the term independent of the parameter $\mathbf{\theta }$, which leads the second line of Eq. (13) to become

$$\ln [\mathrm{pr}(\mathbf{g}|\mathbf{\theta })]\cong \frac{-1}{2}{[\mathbf{g}-\overline{\mathbf{g}}(\mathbf{\theta })]}^t{\overline{\mathbf{K}}}_{\mathbf{g}}^{-1}[\mathbf{g}-\overline{\mathbf{g}}(\mathbf{\theta })]+\ln [\mathrm{pr}(\mathbf{\theta })].$$ (15)

Thus, the scanning-linear estimator that maximizes the posterior density under these approximations is equivalent to

$${\hat{\mathbf{\theta }}}_{\mathrm{SL}}(\mathbf{g})={\mathrm{argmax}}_{\theta }\{\overline{\mathbf{g}}{(\mathbf{\theta })}^t{\overline{\mathbf{K}}}_{\mathbf{g}}^{-1}\mathbf{g}-\frac{1}{2}\overline{\mathbf{g}}{(\mathbf{\theta })}^t{\overline{\mathbf{K}}}_{\mathbf{g}}^{-1}\overline{\mathbf{g}}(\mathbf{\theta })+\ln [\mathrm{pr}(\mathbf{\theta })]\}.$$ (16)

The first term is a linear operation applied on the testing image data by $\overline{\mathbf{g}}{(\mathbf{\theta })}^t{\overline{\mathbf{K}}}_{\mathbf{g}}^{-1}$ from the training data set. The second term is a shifted term due to the different parameters $\mathbf{\theta }$. The third term can be ignored in our study because $\mathrm{pr}(\mathbf{\theta })$ is a constant. So the final equation becomes

$${\hat{\mathbf{\theta }}}_{\mathrm{SL}}(\mathbf{g})={\mathrm{argmax}}_{\theta }[\overline{\mathbf{g}}{(\mathbf{\theta })}^t{\overline{\mathbf{K}}}_{\mathbf{g}}^{-1}\mathbf{g}-\frac{1}{2}\overline{\mathbf{g}}{(\mathbf{\theta })}^t{\overline{\mathbf{K}}}_{\mathbf{g}}^{-1}\overline{\mathbf{g}}(\mathbf{\theta })].$$ (17)

This observer operates on the data linearly, even though, in general, the linear template is a nonlinear function of $\mathbf{\theta }$. In the estimation process, the observer seeks the value of $\mathbf{\theta }$ that will maximize this linear operation.

For the channelized scanning-linear observer (CSLO), all image data needs to go through channelization before using Eq. (17). The input to the CSLO is the channelized testing images of different unknown physical properties, ${\mathbf{x}}^{\text{testing}}(\mathbf{\theta })$. The output of the CSLO in this study is the estimated parameters denoted by the vector ${\hat{\mathbf{\theta }}}_{\mathrm{CSL}}(\mathbf{x})$. The relation between the input and output, according to Eq. (17), then can be described as

$${\hat{\mathbf{\theta }}}_{\mathrm{CSL}}(\mathbf{x})={\mathrm{argmax}}_{\mathbf{\theta }}[\mathbf{s}{(\mathbf{\theta })}^t{\mathbf{S}}^{-\mathbf{1}}{\mathbf{x}}^{\text{testing}}(\mathbf{\theta })-\frac{1}{2}\mathbf{s}{(\mathbf{\theta })}^t{\mathbf{S}}^{-\mathbf{1}}\mathbf{s}(\mathbf{\theta })],$$ (18)

where $\mathbf{s}(\mathbf{\theta })$ is $\mathbf{T}*\overline{\mathbf{g}}(\mathbf{\theta })$, $\mathbf{S}$ is $\mathbf{T}{\overline{\mathbf{K}}}_{\mathbf{g}}^{-\mathbf{1}}{\mathbf{T}}^{\mathbf{t}}$ and ${\mathbf{x}}^{\text{testing}}(\mathbf{\theta })$ is $\mathbf{T}*{\mathbf{g}}^{\text{testing}}(\mathbf{\theta })$.

2.4. Proposed Approach

In this paper, we will develop methods for computing observer performance that rely on less real data and can thus be computed more readily. Specifically, we will study whether we can use 40 images from one high-dose acquisition to estimate the signal and 20 images at the appropriate dose to estimate the covariance matrix $\mathbf{S}$. In the following sections, the methods we used to estimate signal and covariance matrix will be illustrated.

2.4.1. Estimated signal

Unlike the traditional method, the proposed estimated signal in detection is given by

$${\overline{\mathbf{s}}}_{\text{High-dose}}=({\mathbf{m}}_{2\text{High-dose}}-{\mathbf{m}}_{1\text{High-dose}}),$$ (19)

where ${\mathbf{m}}_{2\text{High-dose}}$ and ${\mathbf{m}}_{1\text{High-dose}}$ are the mean of the channelized image data sets of signal-present and signal-absent cases from high-dose images, respectively. The calculation of ${\mathbf{m}}_{i\text{High-dose}}$ is the same as ${\mathbf{m}}_i$ in Eq. (9) but requires less data. For combined tasks, the estimated signal with fewer samples in CSLO is given by

$${\mathbf{s}}_{\text{High-dose}}(\mathbf{\theta })=\mathbf{m}(\mathbf{\theta })=\frac{1}{N}\sum _{k=1}^N{\mathbf{x}}_{k\text{High-dose}}(\mathbf{\theta }),$$ (20)

where ${\mathbf{x}}_{k\text{High-dose}}$ is the channelized image data of signal-present image from high-dose images of class $i$.

2.4.2. Estimated covariance matrix

The covariance matrix was estimated by the LOOC method proposed by Hoffbeck and Landgrebe.¹⁴ In pattern recognition, an important problem is the effect of limited training samples on classification performance because training data is not always abundant and is expensive. When the ratio of the number of training samples to the dimensionality of data becomes small, deterioration of classification happens because the parameter estimation becomes variable. The LOOC method developed by Hoffbeck and Landgrebe estimates the covariance matrix robustly when the training data are limited. The covariance matrix was estimated by the combination of a sample covariance matrix, a common covariance matrix, and the diagonal matrices associated with them. The equation of $i^{\prime }$th covariance matrix proposed by Hoffbeck and Landgrebe is given by

$${\mathbf{C}}_i({\alpha }_i)={\alpha }_{i1}\mathrm{diag}({\mathbf{\Sigma }}_i)+{\alpha }_{i2}{\mathbf{\Sigma }}_i+{\alpha }_{i3}\mathbf{S}+{\alpha }_{i4}\mathrm{diag}(\mathbf{S}),$$ (21)

where elements of mixing parameters ${\alpha }_i=[\begin{array}{cccc}{\alpha }_{i1}& {\alpha }_{i2}& {\alpha }_{i3}& {\alpha }_{i4}\end{array}]$ are required to sum to unity

$$\sum _{j=1}^4{\alpha }_{ij}=1.$$ (22)

The values of mixing parameters ${\alpha }_i$ is selected to maximize the average log Gaussian likelihood of leave-one-out samples, which is called leave-one-out likelihood (LOOL). It should be noted here that the values of mixing parameters ${\alpha }_i$ are different at different radiation dose levels. In general, the more samples of ${\alpha }_i$, the better the improvement, i.e., less bias compared to baseline. Both 50 and 100 samples for ${\alpha }_i$ were investigated, and similar performance could be found in both cases when using our proposed approach. Due to the sufficiently high accuracy provided by 100 samples, 100 random samples of ${\alpha }_i$ were used independently for each dose level in this study. The LOOL is computed as follows:

$${\mathrm{LOOL}}_i({\alpha }_i)=\frac{1}{N_i}\sum _{k=1}^{N_i}\ln \{f[{\mathbf{x}}_{i,k}|{\mathbf{m}}_{i/k},{\mathbf{C}}_{i/k}({\alpha }_i)]\},$$ (23)

where subscript index $i/k$ means that the quantity is computed without using $k$’th samples from class $i$. The function $f$ is the Gaussian likelihood

$$f[{\mathbf{x}}_{i,k}|{\mathbf{m}}_{i/k},{\mathbf{C}}_{i/k}({\alpha }_i)]=\frac{1}{\sqrt{{(2\pi )}^P|{\mathbf{C}}_{i/k}|}}\exp [\frac{-1}{2}{({\mathbf{x}}_{i,k}-{\mathbf{m}}_{i/k})}^t{\mathbf{C}}_{i/k}^{-1}({\mathbf{x}}_{i,k}-{\mathbf{m}}_{i/k})],$$ (24)

where $P$ is the number of pixels. In this study, $P$ is equal to the number of channels, 10, because all the image vectors were channelized by 10 DDOG channels.¹⁷ The associated leave-one-out quantities are then computed as follows:

$${\mathbf{m}}_{i/k}=\frac{1}{N_i-1}\sum _{\begin{array}{c}j=1\\ j\ne k\end{array}}^{N_i}{\mathbf{x}}_{i,j},$$ (25)

$${\mathbf{\Sigma }}_{i/k}=\frac{1}{N_i-2}\underset{\begin{array}{c}j\ne k\end{array}}{\sum _{j=1}^{N_i}}({\mathbf{x}}_{i,j}-{\mathbf{m}}_{i/k}){({\mathbf{x}}_{i,j}-{\mathbf{m}}_{i/k})}^t,$$ (26)

$${\mathbf{S}}_{i/k}=\left(\frac{1}{L}\underset{\begin{array}{c}j\ne i\end{array}}{\sum _{\begin{array}{c}j=1\end{array}}^L}{\mathbf{\Sigma }}_j\right)+\frac{1}{L}{\mathbf{\Sigma }}_{i/k},$$ (27)

$${\mathbf{C}}_{i/k}({\alpha }_i)={\alpha }_{i1}\mathrm{diag}({\mathbf{\Sigma }}_{i/k})+{\alpha }_{i2}{\mathbf{\Sigma }}_{i/k}+{\alpha }_{i3}{\mathbf{S}}_{i/k}+{\alpha }_{i4}\mathrm{diag}({\mathbf{S}}_{i/k}).$$ (28)

Once the LOOL ${\mathbf{C}}_i$ of each class is determined, the final covariance matrix used in our approach can be calculated by the average of all covariance matrices from all classes. The final covariance matrix can be written as

$$\mathbf{C}=\frac{1}{L}\sum _{i=1}^L{\mathbf{C}}_i.$$ (29)

We will use $\mathbf{C}$ as the estimated covariance matrix using the LOOL method for the rest of this paper.

2.5. Figure-of-Merit and the Relevant Mean and Variance

It is now widely accepted that image quality should be evaluated using task-based criteria such as the performance of an observer (model or human) performing a medically relevant task. However, because human observer studies are typically very costly and time consuming, the use of a mathematical model observer becomes a very attractive alternative. In this study, we focus on the application of data reduction on CHO and CSLO in CT imaging systems. The input to our observer models is channelized image ${\mathbf{x}}^{\text{testing}}$. Note here that ${\mathbf{x}}^{\text{testing}}$ could be either signal-present or signal-absent and if there is a signal present, the size and contrast of the signal may vary depending upon which phantom is used. The output of our observer models is the test statistics. For the detection task, the test statistics can be described as

$${\mathbf{t}}_l=\mathbf{W}{\mathbf{x}}_l^{\text{testing}},$$ (30)

where $l=1$ for signal-absent and $l=2$ for signal-present case, respectively, and $\mathbf{W}={\mathcal{S}}^{\mathbf{T}}{\overline{\mathbf{K}}}^{-1}$. For the combined task of detection and estimation, the test statistics can be expressed as

$${\mathbf{t}}_l[{\mathbf{\theta }}_{\mathbf{l}}={\hat{\mathbf{\theta }}}_{\mathrm{CSLO}(\mathrm{HL}-\mathrm{CSLO}),\mathbf{l}}],$$ (31)

where $\mathbf{\theta }$ is the physical property of signal. ${\hat{\mathbf{\theta }}}_{\mathrm{CSLO}(\mathrm{HL}-\mathrm{CSLO}),\mathbf{l}}$ is given by

$${\hat{\mathbf{\theta }}}_{\mathrm{CSLO}(\mathrm{HL}-\mathrm{CSLO}),\mathbf{l}}({\mathbf{x}}_l^{\text{testing}})={\mathrm{argmax}}_{\mathbf{\theta }}[\mathcal{S}{(\mathbf{\theta })}^T{\overline{\mathbf{K}}}^{-1}{\mathbf{x}}_l^{\text{testing}}-\frac{\mathbf{1}}{\mathbf{2}}\mathcal{S}{(\mathbf{\theta })}^{\mathbf{T}}{\overline{\mathbf{K}}}^{-1}\mathcal{S}(\mathbf{\theta })],$$ (32)

where the channelized signal $\mathcal{S}$ and the channelized covariance $\mathbf{K}$ can be estimated by the traditional method and proposed method. They can be described as

$$\mathcal{S}=\{\begin{array}{l}{\mathbf{s}}_{\text{full}}\text{using}240\text{training samples}\\ {\mathbf{s}}_{\text{less}}\text{using}40\text{training samples}\\ {\mathbf{s}}_{\text{High-dose},\text{less}}\text{using}40\text{high-dose training samples}\end{array},$$ (33)

$$\mathbf{K}=\{\begin{array}{l}{\mathbf{S}}_{\text{full}}\text{using}240\text{training samples}\\ {\mathbf{S}}_{\text{less}}\text{using}20\text{training samples}\\ {\mathbf{C}}_{\text{less}}\text{using}20\text{training samples}\end{array},$$ (34)

$({\mathbf{s}}_{\text{full}},{\mathbf{S}}_{\text{full}})$ case is the baseline as mentioned before. The $({\mathbf{s}}_{\mathrm{less}},{\mathbf{S}}_{\mathrm{less}})$ case is the traditional approach with less data and $({\mathbf{s}}_{\text{High-dose},\mathrm{less}},{\mathbf{C}}_{\mathrm{less}})$ case is the proposed approach using less data. It should be noted that there are two other obvious methods for reducing the data requirements: using only ${\mathbf{s}}_{\mathrm{High}\text{-}\mathrm{dose},\mathrm{less}}$ and conventional covariance estimates or using ${\mathbf{C}}_{\mathrm{less}}$ with a conventional estimate of the signal. However, we found that using either of these methods alone is suboptimal in terms of the performance compared with our proposed method and will not be presented here for brevity. To investigate the improvement of our proposed method with less data, we focused on the comparisons among these three cases, i.e., baseline, traditional approach, and the proposed approach using the high-dose signal and the LOOL-estimate of covariance.

ROC curve and EROC curve¹⁸ can be generated after generating the test distributions ${\mathbf{t}}_l$. The FOM of CHO and CSLO is the area under ROC curve (AUC) and the area under EROC (EAUC) curve, respectively. Since the selection of mixing parameters ${\alpha }_i$ could depend upon the training data, the mean values and variances of the FOM for all tasks were estimated by the shuffle method.¹⁹

2.6. Validation Using Simulation Data

To verify our proposed method, three approaches were studied using both simulation and real-data study: (1) baseline-traditional approach using 240 training samples and 260 testing samples; (2) traditional approach using 20 training samples and 80 testing samples; and (3) proposed approach using 20 training samples and 80 testing samples. The baseline results represent, to our best approximation, the results we would obtain with nearly infinite data. Thus, the closer to the baseline, the higher the accuracy is. The accuracy is defined as the ratio of the mean of the AUC value estimated by the testing method to the mean value of the AUC estimated by baseline. Again, first- and second-order statistics, needed to be estimated, are the same for the pure detection task and the combined detection and estimation task. Without loss of generality, the validation will be focused on the case of the pure detection task using simulation data. The MITA CCT183 phantom was used for this verification study. The accuracy is in the range of 88% to 98% for the traditional method using fewer data in simulation. For our proposed method using fewer data, the accuracy is in the range of 97% to 100%. The comparisons have been shown in Fig. 5. The simulation suggested that our proposed method provides more accurate results than the traditional approach while the amount of data used in training and testing was reduced to 20% of the original amount of data.

Fig. 5.

Simulation study. Comparisons of AUC values for 500 image pairs CHO (baseline), 100 image pairs CHO (traditional approach), and 100 image pairs HL-CHO (proposed approach) from simulation image data sets. (a) Object 1: 3 mm 14 HU, (b) object 2: 5 mm 7 HU, (c) object 3: 7 mm 5 HU, and (d) object 4: 10 mm 3 HU.

3. Results

In our study, four tasks were investigated: (1) detection tasks, (2) detection and size estimation tasks, (3) detection and contrast estimation tasks, and (4) detection, size, and contrast estimation tasks. For all tasks in this study, the location of signal was known exactly and was always in the center of ROIs. For each task, three methods were considered: (i) the baseline, traditional method with full data, (ii) traditional method with less data, and (iii) proposed method (HL-CHO/HL-CSLO) with less data. Full data refers to using 240 images for training and 260 images for testing. Less data means that there are only 20 images for training and 80 images for testing. All the images were real CT images. The FOM for the pure detection task and the combination of detection and estimation tasks is AUC and EAUC value, respectively. The mean and variance of AUC or EAUC at each dose level are estimated by the shuffle method using 20 different shuffles of the data. The accuracy is defined by the ratio of mean value of the less data to the mean value of baseline. The x-ray current range was chosen such that the FOM value was neither close to 1 nor close to the guessing value. FOM value equal to 1 means that the observer can easily detect if the signal/tumor exists and distinguish the physical properties of signal/tumor. The guessing value will be described later in this paper.

For the pure detection task, the MITA CCT 183 phantom was used. With ROC analysis, the guessing observer has an AUC of 0.5. The accuracy is 85% to 92% and 97% to 100% for CHO and HL-CHO with less data, respectively. The plots of comparisons are shown in Fig. 6. The results of the pure detection task indicated that our proposed approach can efficiently reduce the required data without losing accuracy.

Fig. 6.

Detection task. Comparisons of AUC values for 500 image pairs CHO (baseline), 100 image pairs CHO (traditional approach), and 100 image pairs HL-CHO (proposed approach) from real image data sets. (a) Object 1: 3 mm 14 HU, (b) object 2: 5 mm 7 HU, (c) object 3: 7 mm 5 HU, and (d) object 4: 10 mm 3 HU.

For the detection and size estimation task, the QRM-LC-FD1 phantom was used for this task. The blind guessing value is $\frac{1}{2}\frac{1}{\text{the number of estimated physical properties}}$. Again the $\frac{1}{2}$ comes from detection by guessing. The number of estimate physical properties, in this case, is 4. So the blind guessing value is 0.125. In this task, the observer not only had to distinguish if the signal exists, but also estimate the size of signal. The results are shown in Fig. 7. In this task, accuracy is 84% to 88% and 97% to 99% for CSLO and HL-CSLO method, respectively. Based on the results of this study, the proposed approach needs only 20% of the full dataset to reach about the same performance of the traditional approach with the full dataset.

Fig. 7.

Detection and size estimation tasks. Comparisons of EAUC values for 500 image pairs CSLO, 100 image pairs CSLO, and 100 image pairs HL-CSLO from real image data sets.

For the detection and contrast estimation tasks, the phantom QRM-LC-FD4 was used for this task. Similar to the task of combined detection and size estimation, there were also four objects of the same size with four different contrasts so the blind guessing value is also 0.125. The observer, in this task, had to tell if there is a signal in the testing image or not and determined the contrast of the signal. Results show that the accuracy is 92% to 96% for CSLO and 97% to 100% for HL-CSLO. The plot of EAUC versus radiation dose level is shown in Fig. 8. For the CSLO approach, the accuracy in the combined detection and contrast estimation tasks is better than the accuracy in the combined detection and size estimation tasks due to the contrasts of objects in QRM-LC-FD4 phantom being higher than the contrasts of objects in QRM-LC-FD1 phantom. This increases the accuracy in the lower dose range. Similar to the previous task, the proposed approach still maintains higher accuracy compared with the traditional approach.

Fig. 8.

Detection and contrast estimation tasks. Comparisons of EAUC values for 500 image pairs CSLO (baseline), 100 image pairs CSLO (traditional approach), and 100 image pairs HL-CSLO (proposed approach) from real image data sets.

In the study of combination of detection, size, and contrast estimation, the observer had not only to decide whether the signal is in the image or not but also make decisions on the size and the contrast of the detected signal. The MITA CCT 189 phantom was used. The blind guessing value in this complicated case is 0.0625 according to the guessing value equation discussed above and the number of estimate physical properties listed in Table 4. The accuracy of CSLO method is 88% to 91% and the accuracy of HL-CSLO is 94% to 99%. The results are shown in Fig. 9. For the CLSO approach with less data, the accuracy in this task is slightly higher than the combination task of the detection and size estimation. This does not imply that the CSLO approach with less data in the combination tasks of detection, size, and contrast estimation has better performance because the background of phantom (MITA CCT 189) used in this task was different. The proposed approach, HL-CSLO, again keeps high accuracy even in the low-radiation dose level range.

Fig. 9.

Detection, size, and contrast estimation tasks. Comparisons of EAUC values for 500 image pairs CSLO (baseline), 100 image pairs CSLO (traditional approach), and 100 image pairs HL-CSLO (proposed approach) from real image data sets.

The accuracies for all the tasks performed by the traditional method and proposed method are listed in Table 5. All the studies were based on 80% data reduction. Overall, the traditional approaches, CHO and CSLO, both suffer while the data reduced especially in the case of lower radiation dose level. The proposed approaches, HL-CHO and HL-CSLO, have stable performance with high accuracy for all the radiation dose levels in all tasks.

Table 5.

Accuracy summary of 80% data reduction.

	Traditional method	Proposed method
Detection task	85.88% to 91.16%	97.38% to 99.86%
Detection and size estimation tasks	84.43% to 87.51%	97.13% to 98.85%
Detection and contrast estimation tasks	92.84% to 95.28%	97.46% to 99.69%
Detection, size and contrast estimation tasks	88.17% to 90.65%	94.80% to 98.33%

4. Conclusions and Discussions

In our studies in which both simulated and real data were tried separately, we found that the proposed methods can match the baseline with high accuracy while using much less data. This is viewed as a positive result since we would ideally like our proposed approach to have the same performance as the baseline. Thus, instead of using 500 image pairs (240 training and 260 testing), the approach requires only 100 image pairs (20 training and 80 testing) for every dose level for all the tasks. This implies that instead of 50 scans for every radiation dose level, we need only 10 scans—a substantial reduction. The 40 images generated at high dose levels can be repeatedly used at different dose levels. The amount of image data required can be reduced up to 80% with our proposed method. Thus, the usage of model observers to assess image quality is a step closer to becoming a practical tool for routine QA/QC of CT imaging systems.

The high-dose images efficiently estimate the signal information with much less data because of the higher signal-to-noise ratio in the image data. Note this property is valid only for images from a linear reconstruction algorithm such as FBP. For iterative reconstruction images, the signal estimated by high dose will be an approximation because the processing is nonlinear and dependent upon the dose level. The sample covariance matrix is an unbiased estimator of the true covariance matrix when given large samples. In the limit of large samples, the sample covariance matrix approaches the true covariance. But if the number of samples is small, then a poor estimate of the covariance matrix occurs. In classification tasks, this results in over-fitting when the same data used to estimate the covariance are used to test the classifier. Over-fitting generally means that biases are introduced into the classification problem. Under-fitting implies that a classifier is too simple to match the real complexities in the data. Shrinkage, which is also called regularization, is a statistical scheme that combines a nominally overfit model with one that is manifestly underfit to produce a model with little bias.²⁰ In this study, we focus on the approach of LOOL. The channelized covariance matrices estimated by the process of maximizing the LOOL makes the channelized covariance matrices more accurate. The covariance matrix estimated by LOOL mixes of the sample channelized covariance matrices and common channelized covariance matrices. This hybrid of matrices makes the inverse of matrix more stable with a sample size closer to the dimension of the channelized vector. The combination of channelized estimated signals from high-dose image and channelized-LOOL covariance matrices makes the AUC or EAUC values very close to the baseline.

Our results consistently indicated that the data can be reduced up to 80% if our proposed method is applied on the CHO and CSLO. Thus, for both pure detection and combination of detection and estimation tasks, the proposed methods, HL-CHO and HL-CSLO, can be used to quantify the image quality performance such as the dose saving capability and the low contrast detectability (LCD) performance of a CT system. It should be noted that the LOOL method was demonstrated on FBP images but does not rely on any of the assumptions that FBP uses. The simulation showed that our baseline was close to the true baseline and did rely on FBP-like assumption. This step was performed only for demonstration purposes and is not an integral part of the LOOL method. The limitation in this study is that the signal shape has to be circular symmetric due to the selection of the D-DOG channels. Thus, under controlled imaging conditions, HL-CHO and HL-CSLO can also be used to compare the image quality performance of different algorithms, components on one system. They can also be potentially used as a tool for the acceptance test for a newly developed iterative reconstruction method.

Biographies

Hsin-Wu Tseng received his BS degree in physics from National Tsing Hua University, Taiwan, his MS degree in physics from University of Florida, Gainesville, Florida, and his PhD in optical sciences and engineering from the University of Arizona, Tucson, Arizona. His research focuses on task-based image quality accessments, CT physics, and algorithm developments.

Jiahua Fan received his BS and MS degrees in electrical engineering from Tianjin University, China, and his PhD in electrical and computer engineering from University of Arizona, Tucson, Arizona. After working on medical-display and medical image processing as an assistant research scientist at University of Arizona, he joined GE Healthcare, Waukesha, Wisconsin. Currently, he is a principal scientist and works on image quality, physics and reconstruction topics of CT systems. He has over 70 scientific publications and 12 patents issued.

Matthew A. Kupinski is a professor at the University of Arizona in the College of Optical Sciences. He received his PhD in 2000 from the University of Chicago and joined the faculty at the University of Arizona in 2002. He is the recipient of the 2007 Mark Tetalman Award and has worked in diverse areas of imaging including x-ray, gamma-ray, diffuse optical, magnetic resonance, and neutron imaging.