A deep learning‐ and partial least square regression‐based model observer for a low‐contrast lesion detection task in CT

Abstract

Purpose

This work aims to develop a new framework of image quality assessment using deep learning‐based model observer (DL‐MO) and to validate it in a low‐contrast lesion detection task that involves CT images with patient anatomical background.

Methods

The DL‐MO was developed using the transfer learning strategy to incorporate a pretrained deep convolutional neural network (CNN), a partial least square regression discriminant analysis (PLS‐DA) model and an internal noise component. The CNN was previously trained to achieve the state‐of‐the‐art classification accuracy over a natural image database. The earlier layers of the CNN were used as a deep feature extractor, with the assumption that similarity exists between the CNN and the human visual system. The PLSR model was used to further engineer the deep feature for the lesion detection task in CT images. The internal noise component was incorporated to model the inefficiency and variability of human observer (HO) performance, and to generate the ultimate DL‐MO test statistics. Seven abdominal CT exams were retrospectively collected from the same type of CT scanners. To compare DL‐MO with HO, 12 experimental conditions with varying lesion size, lesion contrast, radiation dose, and reconstruction types were generated, each condition with 154 trials. CT images of a real liver metastatic lesion were numerically modified to generate lesion models with four lesion sizes (5, 7, 9, and 11 mm) and three contrast levels (15, 20, and 25 HU). The lesions were inserted into patient liver images using a projection‐based method. A validated noise insertion tool was used to synthesize CT exams with 50% and 25% of routine radiation dose level. CT images were reconstructed using the weighted filtered back projection algorithm and an iterative reconstruction algorithm. Four medical physicists performed a two‐alternative forced choice (2AFC) detection task (with multislice scrolling viewing mode) on patient images across the 12 experimental conditions. DL‐MO was operated on the same datasets. Statistical analyses were performed to evaluate the correlation and agreement between DL‐MO and HO.

Results

A statistically significant positive correlation was observed between DL‐MO and HO for the 2AFC low‐contrast detection task that involves patient liver background. The corresponding Pearson product moment correlation coefficient was 0.986 [95% confidence interval (0.950, 0.996)]. Bland–Altman agreement analysis did not indicate statistically significant differences.

Conclusions

The proposed DL‐MO is highly correlated with HO in a low‐contrast detection task that involves realistic patient liver background. This study demonstrated the potential of the proposed DL‐MO to assess image quality directly based on patient images in realistic, clinically relevant CT tasks.

1. Introduction

Quantitative and objective image quality assessment is critical to optimize radiation dose and scanning protocols in clinical CT. Human observer (HO) studies to determine diagnostic performance and clinical image quality are typically limited by cost, efficiency, and intra‐ and interobserver variabilities. Traditional image quality metrics are inappropriate for the newer CT systems when iterative reconstruction (IR) or other nonlinear algorithms are used,^{1, 2, 3, 4} and thus have limited use in quantifying overall image quality for given diagnostic tasks. The use of mathematical model observers (MOs), as objective and efficient alternatives to HOs,^{5, 6, 7, 8, 9, 10, 11, 12, 13} have become popular in task‐based CT image quality assessment, due to the limitations of traditional methods and the growing needs for CT protocol optimization. Once MOs are determined to be highly correlated with HOs in clinically relevant tasks, MOs can be used to estimate the HO performance, which is the ultimate measure of diagnostic image quality.

Strong correlation between traditional anthropomorphic MOs and HOs has been demonstrated for low‐contrast object detection, localization, and differentiation tasks in uniform phantom backgrounds with varying experimental conditions.^{8, 9, 14, 15, 16, 17, 18} Some of these studies performed a simple two‐dimensional (2D) object detection task in 2D static images (i.e., static viewing mode) with real^{8, 9, 14} or simulated uniform phantom background.^{17, 18} Other studies^{15, 16} performed a three‐dimensional object detection task in a uniform phantom background, where readers scrolled through multiple images (i.e., cine viewing mode) to identify lesion‐present cases. Nevertheless, these studies did not include patient anatomical variability and nonuniformity, which may affect HO performance more than stochastic image noise alone.^{19, 20, 21} Thus, it is not clear if the performance of traditional MOs would be highly correlated with the performance of HOs for realistic tasks in real patient images. This is partially due to the difficulty of obtaining sufficient statistics from the patient images with largely varying anatomical background.

A few recent studies aimed to address the above challenges by developing new MOs based on deep learning techniques that use convolutional neural networks (CNN),^{22, 23, 24, 25} since CNN is biologically inspired multilayer perceptron that could simulate the human visual cortex. In Alnowami et al.'s work ²² and Massanes et al.'s work,²³ CNN‐based MOs achieved object detectability comparable to HOs for detection tasks that involved simulated or clinical mammograms. In Kopp et al.'s study,²⁴ another CNN‐based MO also achieved similar performance to HOs for an object detection task that involved a uniform background in CT phantom images. In Zhou and Anastasio's study,²⁵ a CNN was trained to approximate the ideal observer, using computer‐simulated uniform background with correlated noise. These prior studies constructed MOs as a single CNN model with relatively shallow architecture and these MOs were typically tested using simplistic detection tasks that involved neither complicated anatomical background (e.g., CT chest/liver images) nor the realistic human visual searching process. They did not include calibration steps that are typically used to degrade MO performance to the same level of HO performance. Therefore, the validity of these previous CNN‐based MOs in realistic tasks that involve significant background variability and real visual searching process of radiologists remains unclear.

In this work, we propose to develop a new framework of task‐based image quality assessment using deep learning‐based MO (DL‐MO) that would directly operate on patient CT images. Different from previous CNN‐based MOs, the proposed MO is constructed as an ensemble model via transfer learning strategy. We validate the new DL‐MO in a low‐contrast lesion detection task that involves CT images with patient anatomical background.

2. Materials and methods

2.A. DL‐MO framework

The proposed DL‐MO was constructed as a pipeline of a pretrained CNN, a PLS‐DA model, and an internal noise component (Fig. 1). First, the input images were preprocessed and fed into the pretrained CNN to generate CNN codes (i.e., feature vectors). The preprocessing steps are detailed in Sections 2.B.2 and 2.B.3. Then, CNN codes were further engineered by a PLS‐DA model to generate initial test statistics λ ₀. Finally, the internal noise was added to λ ₀ to generate the final test statistics λ used for performance analyses. Each component of the proposed DL‐MO is further explained as follows.

Figure 1.

The framework of the proposed deep learning‐based model observer (DL‐MO), including a pretrained deep convolutional neural network (CNN), partial least square regression discriminant analysis (PLS‐DA) model, and internal‐noise component.

2.A.1. Pretrained CNN

In this study, we hypothesized that the state‐of‐the‐art CNN architectures can mimic the feed‐forward information flow and response of human visual system for lesion detection tasks. Given ten thousands of free parameters in these CNNs, direct training of such neural networks from scratch requires a careful preparation of a huge amount of labeled training data to avoid overfitting, which is typically not practical in the field of medical imaging. Thus, we constructed the DL‐MO using the transfer learning strategy, that is, applying the knowledge gained while solving one task to address a different but relevant task.²⁶ Briefly, transfer learning adapts any prior machine learning model (e.g., CNN) pretrained for a pair of source domain D _s and source task T _s to another pair of target domain D _T and target task T _T . The prior model defines a functional mapping from D _s to T _s :

$$D_s=\left\{\mathcal{X},Pr\{X\}\right\},$$ (1)

$$T_s=\left\{\mathcal{Y},Pr\{Y|X\}\right\},$$ (2)

where $X=x_1,x_2,\dots ,x_m\in \mathcal{X}$ is the m‐dimensional feature vectors that belong to the source feature space $\mathcal{X}$, Pr {X} is the corresponding marginal probability distribution of X, $Y=y_1,y_2,\dots ,y_n\in \mathcal{Y}$ is the n‐dimensional encoded label in the source label space $\mathcal{Y}$, and Pr {Y|X} is the corresponding conditional probability distribution of Y given X. Then, the prior model is adapted to generate a new mapping from D _T to T _T :

$$D_T=\left\{{\mathcal{X}}^{\prime },Pr\{X^{\prime }\}\right\},$$ (3)

$$T_T=\left\{{\mathcal{Y}}^{\prime },Pr\{Y^{\prime }|X^{\prime }\}\right\},$$ (4)

where ${\mathcal{X}}^{\prime }$, Pr {X ^′}, ${\mathcal{Y}}^{\prime }$, and $Pr\{Y^{\prime }|X^{\prime }\}$ are defined similarly as in Eqs. (1) and (2). Of note, $X^{\prime }$ represents the intrinsic features that are explored and exploited by the prior model from the input images. Assuming the similarity between D _s and D _T , the prior model could be used as a fixed feature extractor to generate new features $X^{\prime }$ on D _T , and a secondary model (e.g., statistical model) can be trained over $X^{\prime }$ for T _T . The degree of similarity between D _s and D _T could be evaluated in terms of the image content (e.g., the object types) and the image features that may be relevant to the source/target tasks.

Specifically, when the prior model is a CNN, the feature (also termed as CNN codes) could be directly extracted from any intermediate layer in the network architecture. The selection of CNN layer often follows a generic guideline, that is, the earlier layers are likely to provide domain‐invariant feature, while the later layers tend to generate domain‐specific feature. In this work, we used a 50‐layer Residual net (termed as ResNet‐50)²⁷ as the prior model, which was pretrained over ImageNet ²⁸ (a database of natural images, that is, D _s is the natural image domain). As pointed out in Ref. 29, residual net could be interpreted as the encoder part of deep convolutional framelets, and thus, the corresponding CNN codes could be represented in a matrix form:

$$X^{\prime }=\Phi ·\rho \left(H-\rho (H·\Psi )·\tilde{\Psi }\right)$$ (5)

where H is the Hankel matrix of the input of the layer used for feature extraction, Ψ and $\tilde{\mathrm{\Psi }}$ denote subsequent local bases (i.e., convolutional filters) in the same residual block (i.e., the fundamental building block in Residual net), ρ(·) is the ReLU (i.e., Rectified linear unit) activation function, and Φ denotes the nonlocal bases (e.g., pooling operator). The Eq. (5) provides the mathematical representation of $X^{\prime }$ for the special case that the prior model is Residual net, and thus, the specific form of $X^{\prime }$ would vary if a different prior model is used. Furthermore, the local and nonlocal bases are redundant and non‐orthonormal, which results in inner correlation and high dimensionality of $X^{\prime }$. Since inner correlation and high dimensionality could degrade the performance of the secondary model for T _T , $X^{\prime }$ needs to be further processed by proper feature engineering methods.

2.A.2. PLS‐DA

To construct the secondary model for T _T , we selected the PLS‐DA model^{30, 31} which incorporated the model generation with an embedded feature engineering procedure that addressed the inner correlation and high dimensionality of ${\mathit{X}}^{\prime }$. Specifically, we used the PLS‐DA that was implemented with the SIMPLS algorithm.³² The basic principle of PLS‐DA is briefly summarized as follows. The input CNN codes ${\mathit{X}}^{\prime }$ were standardized and then represented as a linear form:

$$X^{\prime }=V·P+E$$ (6)

where V is the X‐score matrix (i.e., each column is a PLS component), P is the predictor loading matrix, and E is the predictor residual. In this work, T _T was considered as a binary classification task, and the ideal model response was also represented as a linear form:

$$C=U·Q+F$$ (7)

where $C=\left\{C_k,k=1,2,\dots \right\}$ is the ideal model response for each sample of X’, that is, C is a discrete label vector (C _i  = 1 for the ith lesion‐present case, and C _j  = −1 for the jth lesion‐absent case), U is the C‐score matrix (i.e., the linear combination of the responses that has the maximal covariance with the PLS components), Q is the response loading matrix, and F is the response residual. Based on the Eqs. (6) and (7), the loading matrices P and Q are calculated by regressing $X^{\prime }$ and C across V and U,^{32, 33} respectively, that is, $p_j^T={\left(X^{\prime }\right)}^T·\frac{v_j}{\Vert v_j^T{v}_j\Vert }$ and $q_j^T=C^T·\frac{u_j}{\Vert u_j^T{u}_j\Vert }$ where p _j and q _j are the jth row of P and Q, respectively, and v _j and u _j are the jth column of V and U, respectively. In the SIMPLS algorithm, it is assumed that there exists a linear inner relation U = V · D +${\mathit{F}}^{″}$, where D is a diagonal matrix and ${\mathit{F}}^{″}$ is the residual. Furthermore, the SIMPLS algorithm calculates the X‐score matrix V as V = X′ · W, where the weight matrix W is calculated in an iterative process that involves solving the eigenvectors for the successively deflated $X^{\prime }$ · C. More details about this process can be found in Ref. 32. Then, a prediction model was created by calculating a regression coefficient vector B that mapped $X^{\prime }$ to C as follows:

$$C=U·Q+F=X^{\prime }·B+F^{\prime }$$ (8)

$$B=W·D·Q$$ (9)

When the residual terms F, ${\mathit{F}}^{\prime }$, and ${\mathit{F}}^{″}$ are negligible, the model response ${\mathit{C}}^{\prime }$ (also denoted as PLS‐DA model test statistics λ ₀), for any new CNN codes $X_{\mathit{new}}^{\prime }$ from unknown origin can be simply formulated as $C^{\prime }=X_{\mathit{new}}^{\prime }·B$. Furthermore, ${\mathit{C}}^{\prime }$ is typically a set of continuous values, although the PLS‐DA model was trained with the discrete label vector C. Thus, a decision threshold could be determined to classify the lesion‐present/absent cases.

2.A.3. Internal noise component

The use of internal noise was to model the inefficiency and variability of HO performance in lesion detection task. Specifically, we used the decision variable internal noise as used in the prior works^{8, 9, 15} that employed the conventional channelized Hotelling observer:

$$\lambda ={\lambda }_0+\alpha ·x$$ (10)

where λ is the final DL‐MO test statistics, α is a constant weighting factor, and $x\in N\left(0,std{({\lambda }_{0,bkg})}^2\right)$ denotes a normal random variable with zero expectation and a variance equal to that of PLS‐DA test statistics for lesion‐absent cases (denoted as λ _0,bkg ). The value of α was determined by calibrating DL‐MO performance and HO performance at one preselected experimental condition. After calibration, the same value of α was used in the other conditions.

2.B. Data preparation

Image data were prepared for comparing DL‐MO and HO at different experimental conditions with varying lesion attributes (i.e., size and contrast), radiation dose, and image reconstruction types. The procedure of data preparation is summarized as follows.

2.B.1. CT exam data collection

Routine abdominal CT exams of seven adult patients were selected from our clinical data registry. A staff abdominal radiologist supervised the procedure of case selection. The case selection based on the following inclusion criteria: patients agreed with retrospective usage of their medical records for research purposes; these exams were acquired from the same single‐source 128 slice CT scanner (SOMATOM AS+, Siemens Healthineer, Germany), following a routine abdominal CT protocol used at our institution; these patients were proven to be lesion free via clinical follow‐up. The supervising radiologist excluded the cases that did not meet the inclusion criteria, had fatty liver, and had obese patients. Of note, multiple images of each exam were used to generate the experimental trials for both DL‐MO and HO studies. The major parameters of the scanning and reconstruction protocol are listed in Table 1. A validated noise insertion tool was used to simulate additional CT exams acquired with 50% and 25% of the routine radiation dose.³⁴

Table 1.

Scanning and reconstruction protocol

Tube voltage	100–120 kV
CTDIvol	11.6 ± 3.38 mGy
Effective mAs	245.9 ± 40.5
Reconstruction algorithms	WFBP/IR (SAFIRE)
Reconstruction kernels	B30/I30‐2
Slice thickness/increment	3/3 mm

WFBP: weighted filtered back projection; IR (SAFIRE): iterative reconstruction (sinogram affirmed iterative reconstruction); B30: WFBP reconstruction kernel with medium sharpness; I30‐2: SAFIRE reconstruction kernel with comparable sharpness to B30 and a strength setting of 2.

2.B.2. Data generation for MO and HO studies

The volumetric CT images of a real liver metastatic lesion were numerically modified to create lesion models with four different sizes (5, 7, 9, and 11 mm) and three different contrast levels (15, 20, and 25 HU). These lesion models were inserted into multiple locations in normal patient liver images, using a previously validated projection‐based method.³⁵ The lesion locations were validated by the same supervising radiologist who selected the original cases. The projection data, with inserted noise and lesions, were reconstructed using a weighted filtered back projection algorithm (WFBP)³⁶ and an iterative reconstruction algorithm—sinogram affirmed iterative reconstruction (IR: SAFIRE).³⁷ The major parameters of image reconstruction are also listed in Table 1. We generated image data for 12 experimental conditions with varying radiation dose, lesions size, lesion contrast, and reconstruction methods (Table 2), each condition with 77 pairs of lesion‐present and lesion‐absent trials. To generate lesion‐present trials, volumes‐of‐interests (VOIs, 60 × 60 mm² in axial plane) centered at each lesion were extracted (Fig. 2). Each VOI included five consecutive images. Lesion‐absent trials were generated in the same way, but from patient images without the inserted lesions. Both DL‐MO and HO studies were performed on all of these datasets (12 conditions × 154 trials).

Table 2.

All 12 experimental conditions in low‐contrast lesion detection tasks

Condition	#1	#2	#3	#4	#5	#6	#7	#8	#9	#10	#11	#12
Size (mm)	5	7	9	11	5	7	9	11	5	5	5	5
Contrast (HU)	15	15	15	15	15	15	15	15	20	20	20	25
Dose levela	FD	FD	FD	FD	FD	FD	FD	FD	QD	HD	FD	FD
Reconstructiona	WFBP	WFBP	WFBP	WFBP	IR	IR	IR	IR	IR	IR	IR	IR

FD:full routine dose; HD: half routine dose; QD: quarter routine dose.

^aWFBP: weighted filtered back projection; IR: iterative reconstruction (SAFIRE).

Figure 2.

Example images of lesion‐present and lesion‐absent cases (only the middle image out of the five consecutive images for each trial is shown) across different experimental conditions. The inset image illustrates the middle slice (zoom reconstruction) of the base lesion model used in this study. Upper left: lesion size was varied across 5, 7, 9, and 11 mm, lesion contrast was 15 HU, computed tomography (CT) images were acquired at full routine radiation dose (FD) with the weighted filtered back projection. Upper right: the experimental condition was similar to that in upper left, except that CT images were reconstructed using iterative reconstruction (IR). Bottom left: lesion size was 5 mm, lesion contrast was 20 HU, CT images were acquired with IR, but radiation dose was varied across full, half and quarter of routine dose level (FD, HD, & QD, respectively). Bottom right: lesion size was 5 mm, CT images were acquired with IR at full radiation dose, but lesion contrast was varied across 15, 20, and 25 HU. The arrows indicate the lesion locations.

2.B.3. Data augmentation

To improve the performance of DL‐MO, we used several data augmentation strategies to augment the experimental trial images, including image conversion, cropping, z‐direction interpolation, and small angle rotation. Image conversion was used to convert the dynamic range of CT images to that of the natural images in ImageNet, while the other strategies were mainly used to increase the amount of training samples. These strategies were briefly introduced as follows. In image conversion, the original images were transformed to the grayscale of [0, 255] after applying an abdominal display window (WW/WL: 400/40 HU), that is, to make CT images more similar to the natural images in ImageNet. Specifically, the CT numbers were restricted to the dynamic range defined by the abdominal display window (i.e., [−160, 240] HU), and then were normalized to the range [0, 255]. The central three images of each VOI were retrieved to form a pseudo color image as the input of DL‐MO, by stacking the three images as RGB channels. This is because ResNet‐50 was pretrained to classify the natural images with RGB channels. Thus, the first and the last images of each VOI were excluded from the training of DL‐MO. Nevertheless, we expect such trivial adjustment will not downgrade DL‐MO performance, since the central three images already contained the most significant signal information. As for cropping, additional multisized VOIs were cropped out of each VOI. The size of these VOIs uniformly ranged from 7.4 × 7.4 mm² to 52.0 × 52.0 mm² in axial plane. These augmented VOIs were not resized to 224 × 224 pixels (i.e., the typical image size used in ImageNet), and thus, the dimension of the extracted CNN codes depended on the size of the input VOI. For instance, a VOI with 70 × 70 × 3 voxels (i.e., 52.0 × 52.0 × 9 mm³) would yield a CNN code with 5 × 5 × 256 features, at the 26th convolutional layer. So, zero padding was applied to each feature channel of the extracted CNN codes from smaller VOIs to ensure that all CNN codes had consistent dimension (i.e., 5 × 5 × 256 features per sample). In z‐direction interpolation, a voxel‐wise interpolation along z‐direction was used to generate more VOIs (still with 3 mm slice increment). For small angle rotation, each VOI was rotated by a random angle from a uniform distribution within the range of [−5.0°, 5.0°], and a “nearest neighbor” interpolation method was used to generate the rotated images, to avoid significantly altering image texture. Together, these data augmentation strategies generated 9, 424 additional lesion‐present and lesion‐absent trials for each experimental condition.

2.C. DL‐MO studies

We constructed 12 DL‐MOs separately to perform a two‐alternative forced choice detection (2AFC) task, that is, one DL‐MO for each experimental condition. The 26th convolutional layer of ResNet‐50 and the most significant 20 PLS components were used to build each DL‐MO. The configuration of the CNN layer and PLS components was empirically determined using a coarse grid search strategy in each experimental condition, that is, experimentally evaluating DL‐MO performance across different configurations. Note that the internal noise was excluded during grid search process. The impacts of different CNN layers and PLS components are demonstrated in Section 3.A. A resubstitution strategy³⁸ was used to calculate DL‐MO test statistics, that is, the training dataset and the testing dataset were both generated out of the same set of CT images. A bootstrapping method was used to estimate the mean and the standard deviation of the test statistics. The bootstrapping was repeated for 200 times at each experimental condition. In each bootstrapping iteration, the training samples were randomly resampled (with replacement) from the augmented dataset, and the number of training samples was equal to that of the augmented dataset (i.e., 9424 samples). DL‐MO performance was also calibrated with the averaged HO performance at experimental condition #10, by adjusting the weighting factor α of the internal noise component (See Section 3.C.).

2.D. HO studies

Four medical physicists were recruited to perform the HO studies. Each reader performed the 2AFC detection task for the same 12 experimental conditions used in the DL‐MO study in three reading sessions, with four conditions at each session. For each trial, lesion‐present case and lesion‐absent case were displayed side by side with random placement, using the same graphical user interface as used in prior works^{8, 15} (Fig. 3). Furthermore, HOs were required to scroll through five images per VOI to identify the lesion‐present cases (i.e., multislice viewing mode). In each reading session, HOs were encouraged to take breaks to avoid fatigue, and they were also allowed to revisit any previous trials and modify their interpretation. The viewing parameters were fixed: all images were displayed with the soft tissue window (WW/WL: 400/40 HU); the viewing distance approximately 50 cm.

Figure 3.

Graphical user interface that was used in human reader studies for the two‐alternative forced choice lesion detection task with multislice scrolling viewing mode.

2.E. Figure of merit and statistical analyses

For the DL‐MO study, the area under receiver operating characteristic curve (AUC_ROC) was used as the figure of merit (FOM) to represent the performance at each experimental condition. The ROC curve was generated by varying the decision threshold and compared with the test statistics of signal present and signal absent images. The mean and the standard deviation (STD) of AUC_ROC were calculated based on the 200 bootstrapping iterations. For the HO study, we used percent correct (PC) as the FOM to represent the HO performance in each 2AFC detection task. In each experimental condition, PC was calculated by dividing the number of correctly identified lesion‐present cases over the total number of trials. Note that AUC_ROC is considered to be equivalent to PC in 2AFC detection task^{38, 39} if the model observer response follows a normal distribution. Thus, the normality of DL‐MO test statistics was evaluated to verify the use of AUC_ROC as the FOM of DL‐MO in the 2AFC task. Specifically, we qualitatively tested the normality of λ ₀ (excluding internal noise) separately for lesion‐present/absent images in each experimental condition, since the internal noise component had been formulated as a normal random variable. Finally, the strength of correlation and agreement between DL‐MO and HOs was evaluated across all experimental conditions, using Pearson's product moment correlation analysis and Bland–Altman plots.

3. Results

3.A. Grid search for optimal CNN layer and PLS components

An example of coarse grid search is shown in Fig. 4. The value of AUC_ROC was moved toward saturation, as the number of PLS components increased. This is because the feature variance was better represented by more PLS components. However, the use of a larger set of PLS components may result in a higher chance of overfitting the DL‐MO, so we intended to select the fewest PLS components that still yield a high performance. Furthermore, the 26th convolutional layer was able to achieve higher AUC_ROC with fewer PLS components, which suggested a higher feature transferability toward the target task T _T . Therefore, we selected the 26th convolutional layer and the first 20 PLS components as the optimal configuration for building DL‐MO.

Figure 4.

An example heatmap of the area under receiver operating characteristic curve (AUC_ROC; without internal noise, that is, α = 0) acquired from the coarse grid search at experimental condition #10, with varying configurations of convolutional layer and partial least square (PLS) components. These configurations involved the 14th to 38th convolutional layers in ResNet‐50, and the 1st to 100th PLS components. The value of AUC_ROC is represented by grayscale intensity.

Several examples of the CNN codes from ResNet‐50 are illustrated in Fig. 5. It is typically challenging for human readers to directly interpret these CNN codes from the deep layers. However, the difference between lesion‐present/absent CNN codes revealed the presence of the encoded structural information around the lesion location, that is, the central region of each lesion‐present image. This phenomenon has indicated the capability of ResNet‐50 in preserving the lesion‐relevant information.

Figure 5.

Examples of the convolutional neural network (CNN) codes and the corresponding absolute difference from a pair of lesion‐present/absent images, at the 26th convolutional layer of ResNet‐50. The inset computed tomography (CT) images (70 × 70 pixels) illustrate a pair of lesion‐present/absent examples from condition #1, and the arrow indicates the lesion location. These CNN codes corresponded to six randomly selected feature channels at the corresponding network layer. The images at each column correspond to the same convolutional filter channel. The display window of the difference images was narrowed down ([0, 0.4]) for the convenience of illustration.

Furthermore, some examples of the PLS components and regression coefficients (i.e., the template B) are illustrated in Fig. 6. As presented in Section 2.A.2, the PLS components were used to generate the template B that provided a set of weights to combine all features to form the test statistics λ ₀. The features at the central region of the CNN codes (i.e., the lesion location) generally received stronger weights than those at the peripheral region. This finding indicated that the proposed model observer used the lesion‐relevant information and the surrounding patient anatomy to form the final test statistics.

Figure 6.

Examples of partial least square (PLS) components and regression coefficients (i.e., the template B), from the experimental condition #1. The corresponding convolutional neural network (CNN) codes were extracted from the 26th convolutional layer of ResNet‐50. (a) The most significant 20 PLS components from a pair of lesion‐present/absent images. (b) The PLS regression coefficients with respect to the randomly selected feature channels from the 26th convolutional layer of ResNet‐50 and were reshaped to match the spatial location of image features, for the convenience of illustration. The display window was [0, 0.06].

3.B. Normality of test statistics λ ₀

The normality of the test statistics λ ₀ was qualitatively evaluated using the normal probability plots.⁴⁰ Briefly, the sorted samples of λ ₀ (i.e., the y‐axis) were plotted against a theoretical normal distribution (i.e., the x‐axis) that was generated as the normal order statistics medians, and a straight line was fitted to the sample points to indicate the similarity to normality, that is, there existed a strong similarity if the majority of the sample points appeared along this line. As illustrated in Fig. 7, the statistical distribution of λ ₀ well approximated normal distribution. Thus, it is reasonable to consider the AUC_ROC of DL‐MO as an equivalent FOM of the PC of HOs in the 2AFC task.

Figure 7.

Examples of normal probability plots of test statistics λ ₀ (without internal noise) for lesion‐present/absent cases in experimental conditions #10 (upper left), #9 (upper right), #7 (bottom left), and #8 (bottom right). The distribution of λ ₀ approximated the normal distribution.

3.C. Internal noise calibration

The calibration was carried out on the experimental condition #10. Specifically, the value of the weighting factor α was increased from 0.1 to 5.5 with an increment of 0.01 per step. The resultant AUC_ROC of DL‐MO was compared with the averaged PC of HOs at the same condition to determine a proper value of α (Fig. 8). The value of α was determined to be 1.1, that is, the internal noise was 1.1 times of the noise of λ _0,bkg .

Figure 8.

The weighting factor α was adjusted to calibrate deep learning‐based model observer (DL‐MO) performance to the averaged human observers (HOs) performance at the experimental condition #10, that is, lesion size was 5 mm, lesion contrast was 20 HU, radiation dose was half of routine dose level, and computed tomography images were reconstructed by iterative algorithm. The value of α was varied from 0.1 to 5.5 with a uniform interval of 0.1. Every second sample was plotted out for the convenience of illustration. The error bars indicate the standard deviation of DL‐MO performance. AUC_ROC denotes the area under the receiver operating characteristic curve.

3.D. Statistical analyses

The 12 experimental conditions were regrouped into four scenarios for convenience of comparing DL‐MO with HO (Fig. 9). The AUC_ROC of DL‐MO was comparable to the PC of HO in each condition. Pearson's correlation coefficient was 0.986 (with 95% CI [0.950, 0.996]). Bland–Altman plots indicated that there was no statistically significant discrepancy between DL‐MO and HO (Fig. 10). The bias between DL‐MO and HO was 0.25%. The upper and the lower limits of agreement were 4.00% (with 95% CI [1.90%, 6.11%]) and −3.51% (with 95% CI [−5.61%, −1.40%]), respectively.

Figure 9.

Comparison of deep learning‐based model observer (DL‐MO) and human observer (HO) in four scenarios. (a) Lesion size was varied across 5, 7, 9, and 11 mm, lesion contrast was 15 HU, computed tomography (CT) images were acquired with iterative reconstruction (IR) at full routine dose (FD). (b) Similar to the scenario in (a), but CT images were reconstructed with weighted filtered back projection (WFBP). (c) Lesion size was 5 mm, lesion contrast was 20 HU, CT images were reconstructed with IR across full, half, and quarter of routine radiation dose level (FD, HD & QD). (d) Lesion size was 5 mm, lesion contrast was varied across 15, 20, and 25 HU; CT images were acquired with IR at FD. The error bars indicate the standard deviation of the area under receiver operating characteristic curve (AUC_ROC).

Figure 10.

Bland–Altman plot of the difference of area under the receiver operating characteristic curve across all 12 experimental conditions. The two dashed lines denote the upper and the lower limits of agreement (LOA). The solid line denotes the bias. The error bars indicate the 95% confidence interval of LOA.

4. Discussion

In this study, we proposed a framework of DL‐MO that incorporated a state‐of‐the‐art deep CNN, PLS‐DA model, and an internal noise component. The comparison between the proposed DL‐MO and HOs was carried out over multiple experimental conditions with varying lesion attributes, radiation dose, and reconstruction types. We observed strong correlation and agreement between DL‐MO and the averaged HO performance.

The proposed DL‐MO framework was based on a fundamental hypothesis that there exists similarity between the state‐of‐the‐art CNN architectures and human visual system in object detection. This hypothesis was at least partially supported by several prior studies, which systematically investigated the correlation between CNN architectures and human neural response using carefully designed psychophysical experiments.^{41, 42, 43} Nonetheless, the corresponding degree of similarity varied a lot across different CNN architectures, which could be attributed to the different feature representation power of those CNNs. If using a different pretrained CNN architecture in the DL‐MO framework, the strength of the correlation and agreement with HOs may be altered. In this work, it is conjectured that the state‐of‐the‐art CNN architecture is more likely to be human‐like, due to the large feature representation power of these CNNs. The pretrained ResNet‐50 was used in this study, since it has demonstrated outstanding learning capacity in comparison with other networks in multiple challenging image classification and object detection tasks.²⁷ It is possible that other state‐of‐the‐art CNN architecture may also work, but thus far, there is still a lack of theoretical guideline on what types of CNN architectures could be optimally correlated with HOs. The experimental validation similar to the current study is an effective way to select a proper architecture from the existing CNNs.

Besides the internal noise weighting factor α, DL‐MO has two additional free parameters, that is, the CNN layer used for feature extraction, and the number of PLS components used for building PLS‐DA model. These two parameters need to be properly determined to achieve reasonable DL‐MO performance in the target task T _T . It is difficult to provide a theoretical prediction of the optimal configuration of CNN layer and PLS components, which is partially attributed to the extreme complexity of estimating the feature transferability of different layers. The feature transferability is negatively affected by two issues: the specificity of the CNN layers and the coadaption among neighboring layers.⁴⁴ That being said, it is more likely for the features to transfer more poorly, when the following scenarios occur: source task T _s is less similar to target task T _T , the CNN layer has higher specialization toward T _s , or the CNN was split at the coadapted layers. On the contrary, it is straightforward to experimentally validating the DL‐MO performance with respect to varying configurations of CNN layer and PLS components, that is, the coarse grid search strategy, to determine the appropriate CNN layer and PLS components that drive the DL‐MO to achieve high efficiency. Therefore, we used the coarse grid search strategy to empirically determine a proper selection of CNN layer and PLS components.

Conventional anthropomorphic MOs can estimate the effect of varying technical parameters on human perception for simple target structures and tasks using computer simulation or phantom images, while the proposed framework will provide a new MO, using deep learning techniques, which will enable the direct estimation of diagnostic performance using patient CT images for optimizing the CT protocols. Nevertheless, there exists some similarity between the DL‐MO framework and conventional anthropomorphic MOs. Take the channelized Hotelling observer (CHO) as an example: the pretrained CNN served as the fixed feature extractor, which may be viewed as a generic extension of the explicitly crafted channel filters (e.g., Gabor filters or dense difference of gaussian filters); the regression coefficient vector B (from the PLS‐DA model) was analogous to the template vector ω used in CHO; the internal noise component was directly extended from its counterpart in CHO. Furthermore, the AUC_ROC of DL‐MO and the PC of HOs were considered as equivalent FOMs as was done in the prior works^{8, 15} that involved CHO, since the DL‐MO response followed normal distribution. If DL‐MO response failed the normality test, the “template application”³⁹ may be used as an alternative method to calculate PC as the FOM of DL‐MO. Briefly, the template would be directly applied to each pair of lesion‐present/absent cases, and the cases with larger test statistics would be typically considered as the lesion‐present ones. Thus, the PC of DL‐MO can be calculated across all experimental conditions. A systematic comparison between DL‐MO and the conventional MOs could be an interesting topic. However, there has been no agreement on how to apply conventional MOs to acquire sufficient statistics from the largely varying anatomical background. Thus, the comparison between DL‐MO and the conventional MOs is beyond the scope of the current study.

The application of data augmentation is not limited to the training of deep CNN models. In this work, the augmented data were used to improve the performance of PLS‐DA model, instead of fine‐tuning the pretrained ResNet‐50. The data augmentation was expected to induce more image feature variability and capture the underlying image statistics, which would be useful for improving the PLS‐DA model robustness and reducing overfitting. Similarly, previous studies have already used different types of data augmentation strategies with PLS and other classical machine learning techniques. For instance, noise augmentation was used to suppress the overfitting and improve PLS robustness against various types of random noise.^{45, 46} Elastic deformation and affine transformation were also effective in reducing overfitting and improving the accuracy of support vector machine and extreme learning machine.⁴⁷

A large independent testing dataset was not prepared to evaluate the DL‐MO performance, as the data acquisition procedure was very resource‐intensive, especially for the collection of human reader performance. To complete this study with a comprehensive evaluation of DL‐MO performance, we present here a preliminary validation of the estimated DL‐MO generalization accuracy (i.e., the AUC on unseen data) with respect to the configuration of the 26th convolutional layer and the most significant 20 PLS components. Briefly, the estimated AUC from the resubstitution method was compared with the counterparts from two additional cross‐validation methods, including the tenfold cross‐validation (10f‐CV) and the bootstrap cross‐validation (BS‐CV) methods⁴⁸ (Fig. 11). Note that the internal noise was excluded from the comparison. In the 10f‐CV method, the original experimental trials were randomly split into ten independent sub‐groups. Then, a single subgroup was selected as the validation dataset, while the other subgroups were used for training the DL‐MO. This process was repeated for ten times till each subgroup had been used once as the validation dataset. The averaged AUC across all ten validation datasets was used as the estimated DL‐MO generalization accuracy. In the BS‐CV method, the 10f‐CV was nested within the standard bootstrapping. The bootstrapping was repeated for 200 times with replacement (i.e., 200 resampled datasets), and the 10f‐CV was carried out on each resampled dataset to calculate the mean AUC from the validation datasets. Then, the mean AUC from the 200 resampled datasets was further averaged to estimate the DL‐MO generalization accuracy. In both methods, data augmentation was only applied to the training subsets to generate additional training samples, while each validation dataset had a small number of trials (n = 14 per condition). Of note, the 10f‐CV method tends to underestimate the generalization accuracy with a large variance, especially when using a small dataset.⁴⁹ Compared to 10f‐CV, the BS‐CV method could achieve a more accurate estimation of the generalization accuracy with small samples, while suppressing the variance.^{48, 50} Considering the characteristics of these methods and the small size of the validation datasets, the three methods yielded comparable estimation of the generalization accuracy across all experimental conditions. Thus, we consider that there was no significant bias (i.e., overfitting) in the estimated DL‐MO performance. Note that the difference among the three methods shall further diminish as the sample size increases.

Figure 11.

Comparison of the deep learning‐based model observer generalization accuracy that was estimated for the configuration of the 26th network layer and the most significant 20 PLS components, using the resubstitution (denoted as “Resub”), tenfold cross‐validation (denoted as “10f‐CV”), and bootstrap cross‐validation [denoted as bootstrap cross‐validation (“BS‐CV”)] methods, respectively, across all 12 experimental conditions. The generalization accuracy was gauged with the area under receiver operating characteristic curve (AUC_ROC). The condition indices were rearranged in the ascending order of the AUC_ROC value, for the convenience of illustration.

We acknowledge several limitations in the present work. First, we recruited nonradiologist readers (i.e., medical physicists) for HO study, and the number of readers may be limited. Second, although multislice scrolling was used as the viewing mode to mimic realistic tasks, the 2AFC lesion detection task is the simplest form of localization tasks, in which the lesion characteristics and locations were known exactly. Thus, the effects of anatomical background variability and tissue nonuniformity on HOs performance might be limited, compared with clinically realistic tasks. Third, we used only one base lesion model and one type of CT exam in data preparation. In clinical practice, radiologist readers are typically challenged to identify lesions with various imaging features. Meanwhile, it is desirable if the correlation between the proposed DL‐MO and HOs could be determined in multiple types of CT exams. Finally, the use of resubstitution method may result in positive bias in the FOM of DL‐MO, although such bias could decrease as the number of samples increases.³⁸ We aim to address these limitations in the follow‐up study that employs radiologist readers and more realistic lesion localization tasks with patient abdominal/chest CT exams. Our preliminary results have suggested that a strong correlation between the proposed DL‐MO and HOs may be established in those challenging scenarios.⁵¹

The proposed DL‐MO framework could be readily applied for similar lesion detection tasks that involve different types of scanning protocols, lesion models, CT systems, and reconstruction algorithms, by retraining the DL‐MO using the images acquired from the new experimental conditions. In our ongoing work,⁵¹ we have observed the possibility of reusing the same configuration of CNN layer and PLS components for abdominal and chest CT exams with different base lesion models, whereas the weighting factor of the internal noise component needs readjustment. The further study would be necessary to validate these preliminary findings with additional types of CT exams and lesion models. If the retuning of the DL‐MO parameters (e.g., CNN layer and PLS components) is needed, one could still use the straightforward grid search strategy to find a proper configuration of DL‐MO parameters. Although the PLS has been successfully used in many data analytics tasks that involved small sample size with much larger feature dimensionality,^{52, 53, 54} the data augmentation strategies tend to be helpful to further enhance the model performance.

5. Conclusion

A framework of deep learning‐based model observer for task‐based image quality assessment was introduced. The performance predicted by DL‐MO was highly correlated with the HO performance in a 2AFC lesion detection task that involved realistic patient liver background and multislice viewing mode. This validation provided preliminary evidence for using DL‐MO to evaluate diagnostic image quality in realistic CT tasks.

Conflicts of Interest

Dr. Cynthia H McCollough is the recipient of a research grant from Siemens Healthcare. The other authors have no relevant conflicts of interest to disclose.