Robust Explanation Supervision for False Positive Reduction in Pulmonary Nodule Detection

Abstract

Background:

Lung cancer is the deadliest and second most common cancer in the United States due to the lack of symptoms for early diagnosis. Pulmonary nodules are small abnormal regions that can be potentially correlated to the occurrence of lung cancer. Early detection of these nodules is critical because it can significantly improve the patient’s survival rates. Thoracic thin-sliced Computed tomography (CT) scanning has emerged as a widely used method for diagnosing and prognosis lung abnormalities.

Purpose:

The standard clinical workflow of detecting pulmonary nodules relies on radiologists to analyze CT images to assess the risk factors of cancerous nodules. However, this approach can be error-prone due to the various nodule formation causes, such as pollutants and infections. Deep learning algorithms have recently demonstrated remarkable success in medical image classification and segmentation. As an ever more important assistant to radiologists in nodule detection, it is imperative ensure the deep learning algorithm and radiologist to better understand the decisions from each other. This study aims to develop a framework integrating explainable AI methods to achieve accurate pulmonary nodule detection.

Methods:

A robust and explainable detection framework is proposed, focusing on reducing false positives in pulmonary nodule detection. Its implementation is based on an explanation supervision method, which uses nodule contours of radiologists as supervision signals to force the model to learn nodule morphologies, enabling improved learning ability on small dataset, and enable small dataset learning ability. In addition, two imputation methods are applied to the nodule region annotations to reduce the noise within human annotations and allow the model to have robust attributions that meet human expectations. The 480, 265, and 265 CT image sets from the public LIDC-IDRI dataset are used for training, validation, and testing.

Results:

Using only 10, 30, 50 and 100 training samples sequentially, our method constantly improves the classification performance and explanation quality of baseline in terms of AUC and IoU. In particular, our framework with a learnable imputation kernel improves IoU from baseline by 24.0% to 80.0%. A pre-defined Gaussian imputation kernel achieves an even greater improvement, from 38.4% to 118.8% from baseline. Compared to the baseline trained on 100 samples, our method shows less drop in AUC when trained on fewer samples. A comprehensive comparison of interpretability shows that our method aligns better with expert opinions.

Conclusions:

A pulmonary nodule detection framework was demonstrated using public thoracic CT image datasets. The framework integrates the robust explanation supervision technique to ensure the performance of nodule classification and morphology. The method can reduce the workload of radiologists and enable them to focus on the diagnosis and prognosis of the potential cancerous pulmonary nodules at the early stage to improve the outcomes for lung cancer patients.

1. Introduction

According to the American Cancer Society, lung cancer is the deadliest and second most common cancer in the United States. Pulmonary nodules are small abnormal regions that can potentially be correlated to a sign of lung cancer. Early detection of these nodules can increase the patient’s survival rate from lung cancer¹ early treatment. Computed tomography (CT) has been widely used for nodule diagnosis and prognosis via thoracic scanning protocol that provides sufficient spatial resolutions to detect the abnormalities of the lung². Radiologists can analyze the thoracic CT images and forecast the risk factors of cancerous nodules. However, pulmonary nodules can be caused by abnormal growth of cells, pollutants, autoimmune diseases, fungal infections, respiratory system infections, and scar tissues. The complexity of nodule formation increases the workload of radiologists and the uncertainty of human factors to miss the diagnosis of lung cancerous nodules at the early stage³. Recently, deep learning (DL) algorithms have shown great success in medical applications regarding classification^4–6 and segmentation^7–9. This work aims to propose a framework that leverages the state-of-the-art explainable AI methods^10,11 to increase the accuracy of pulmonary nodule detection.

DL has been implemented into clinical radiation treatment planning for organ segmentation; for instance, the U-Net¹² is one of the most popular models recognized through international conferences^13–15. Conventional model architectures, such as residual networks, can also achieve good segmentation results in clinical applications by integrating multi-modality and multi-level contextual information¹⁶. The success of these DL models raises the motivation to use computer-aided detection (CAD) systems to assist radiologists in nodule delineation. To effectively detect nodules from thoracic CT scans, two fundamental challenges need to be overcome: (a) the variety of nodule morphology leads to difficulties in identification; (b) the lack of interpretability leads to unconvincing or agnostic reasoning processes, limiting clinical translation¹⁷. The first issue has been widely explored^18–21 for lung nodules with various geometrical features. While most current CAD-based nodule detection systems merely focus on exploiting the information from CT images, it has yet to be widely investigated how to leverage the rich information from physician nodule contours. As for the second issue, the development of model interpretability is still in its infancy.

Inspired by Explainable AI^22–25 techniques, human-understandable explanations in DL models have become a promising solution to interpretable CAD systems. One mainstream explanation tool is to visualize where the model “sees”.^26,27 This approach generates an attention map indicating which part of the input is responsible for the model prediction. Based on an explanation tool, explanation supervision technique integrates expert opinions in the DL model to make it more focused on the lung nodule regions. Expert opinions can be expressed through explanation annotations, which can inform the model where to look for potential cancer nodules. Such a technique synthesizes both images and explanation annotations to supervise the model training²⁸. Linsley et al²⁹. Demonstrated how to use the explanation annotations to give accurate predictions that are in line with human expectations. Mitsuhara et al³⁰ implemented explanation supervision technique to customize an attention-branching network³¹ to enhance image prediction. Several recent studies^32,33 have shown that explanation supervision techniques can be flexibly implemented into various DL models. However, medical experts’ opinions are noisy compared to mathematical formulas and may bias the DL model, returning physically unreasonable outcomes. For example, different radiologists may have different contours of the nodule’s margins, and the contoured region of the nodule may be inaccurate.

RES (Robust Explanation Supervision)³⁴ is a generic framework for guiding visual explanation that allows DL models to be trained using even noisy annotations. For example, inaccurate nodule contours will be imputed based on a prior about the inaccuracy. In this work, we leverage the RES method in our proposed nodule detection framework to enhance DL-based pulmonary nodule detection by reducing false positives. Our framework, called RXD (Robust and eXplainable Detection), can enable robust and explainable pulmonary nodule identification by leveraging expert knowledge of contours while reducing the inaccuracy of human annotations. Another factual issue to consider is that the amount of data available is usually scarce. Previous studies^35,36 had exhibited that small dataset learning was achievable by integrating expert opinions into DL training. This work also explores the feasibility of using the proposed nodule detection framework for small dataset learning.

2. Materials and methods

2.1. Dataset

We used a nonproprietary dataset^37–39, the Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) dataset, to evaluate the proposed pulmonary nodule detection framework. The LIDC dataset are one of the most common multi-institute datasets for lung nodule detection used by many previous investigations^40–42. The dataset includes thoracic CT scans from 1,010 patients with slice thickness ranging from 0.6 to 5.0 mm, acquired from seven academic centers and eight medical imaging companies. The CT scanners included GE Medical Systems LightSpeed Plus, Philips Brilliance, Siemens Definition, Emotion, and Sensation scanner models, and Toshiba Aquilion. The CT images were acquired by using single energy scans including spectrum energies of 120 kV, 130 kV, 135 kV, and 140 kV with the mean tube current of 222 mA. The CT slice thicknesses were ranging from 0.6 mm to 5.0 mm, and the mean pixel width was 0.688 mm. Due to the involvement of multiple CT scanners, the reconstruction kernels varied among vendors, but all images were acquired based on the thoracic protocols. Four experienced radiologists adequately evaluated the thoracic CT images via a two-stage process to ensure precise and accurate nodule annotations. At the first stage, each radiologist independently reviewed the images and classified the lesions into three categories (nodules ≥3 mm, nodules <3 mm, and non-nodules). Based on previous work⁴³, the concept of a nodule might not encompass a singular entity that can be easily defined, and LIDC Research Group⁴⁴ defined these three categories to impose different requirements for radiologists to characterize nodules regarding shape, lobulation, margin, and likelihood of malignancy⁴⁵. Then the most likely nodule contours were concluded from the best knowledge from all radiologists. In this work, we aim to use the data for the nodule detection purpose, and we classified positive and negative data for images with and without nodules.

2.2. Data preparation

This work consisted of 1,010 CT image sets from 1,010 patients with total image slices of 68,130. Patients’ CT image sets were randomly sampled (without resampling) into training, validation, and testing that included 480, 265, and 265 CT image sets, which contains 55,755, 6,507, and 5,868, CT image slices, respectively. We used patch-based training with the bounding boxes for nodule contour annotations. For the nodule annotations, a typical 50% consensus consolidation of radiologist annotations was performed to reflect that the ground truth nodule regions were agreed by at least two radiologists. Then, the bounding boxes of nodule contour annotations were spatially extended by 20 pixels to obtain the corresponding image blocks. The CT images that contain the image blocks are positive samples, while the remaining images are negative samples. Imagine a Rubik’s cube, the center of which is an image block containing a nodule, and the axial slices of the other 26 blocks were extracted as negative samples. Blocks that exceed the image boundaries are discarded. As a result, 2,625 positive samples and 65,505 negative samples were obtained, which is an extremely imbalanced dataset. A balancing strategy was involved in training to mitigate the impact of imbalance, while the original proportions were maintained in validation and test sets to simulate the real-world situation. The balancing strategy uses the patient IDs of the positive samples to draw the same number of negative samples as the positive ones. Table 1 shows the data distribution of the training set, validation set and test set. To build a truly robust CAD system, we considered all samples to be available with no restrictions on the thickness of the slices. All samples were normalized between 0 and 1, and then converted to PNG format for ease of use.

Table 1.

Data distribution of the training set, validation set and test set. The training set was balanced, while validation and test sets maintained the original proportions.

	Training set	Validation set	Test set
Positive samples	50%	4%	4%
Negative samples	50%	96%	96%

The axial slice of each nodule contour annotation was used as a human explanation annotation. Typically, explanation supervision requires human explanation annotations for both positive and negative samples to guide the model where to attend. However, there is no such annotation for negative samples in the LIDC-IDRI dataset. As a further incentive for the model to learn to distinguish between positive and negative samples, we generated an explanation annotation for each negative sample with the same shape as the negative sample, all with a value of 0 (given that the human explanation annotation for the nodule region has a value of 1). Figure 1 shows examples of positive and negative samples, along with the corresponding explanation annotations. All image patches and nodule region annotations of positive samples were obtained with the help of pylidc⁴⁶ package.

Figure 1.

Examples of positive and negative samples, along with the corresponding explanation annotations. The explanation annotation of the positive sample is a black and white mask, where the white region represents the nodule region, which has a value of 1 at each pixel. The black region represents the non-nodule region, which has a value of 0 at each pixel. As a comparison, the explanation annotation of the negative sample is a black mask with a value of 0 for each pixel.

2.3. Methods

2.3.1. Robust explanation supervision for pulmonary nodule detection

We considered the pulmonary nodule detection task as a two-dimensional (2D) image classification task since physicians’ contours were based on transversal CT image slices. This nodule detection problem can be formulated as: suppose $x\in {ℝ}^{C\times H\times W}$ is a 2D slice with $C$ channels, height $H$ and width $W$, and $y\in \left\{\mathrm{0,1}\right\}$ is the class label of $x$, where 1 denotes positive and 0 denotes negative. A deep neural network (DNN) should learn the mapping function $f$ from each $x$ to its label $y$.

Each positive sample in the LIDC-IDRI dataset initially contains nodule contour annotations from four radiologists, which can provide abundant information about nodule contour and location. A natural idea is to exploit these nodule contour annotations to improve the training effect. This can be achieved by using explanation supervision technique, which incorporates explanation annotations (i.e., these nodule contour annotations) into supervision signal, improving both classification performance and inference attribution³⁴. However, to our knowledge, no one has applied explanation supervision to clinical diagnosis, leaving the wealth of information in explanation annotations untapped. RES is a framework that utilizes explanation annotations to improve both the performance and the explanation quality of backbone DNN. To achieve this, a robust explanation loss term is introduced into its overall loss function as follows:

$$\underset{⬚}{\min }\sum _i^N{ℒ}_{Pred}\left(f\left(x^{\left(i\right)}\right),y^{\left(i\right)}\right)+\lambda {ℒ}_{Exp}\left(〈M^{\left(i\right)},H^{\left(i\right)}〉\right),$$ (1)

where ${ℒ}_{Pred}\left(f\left(x^{\left(i\right)}\right),y^{\left(i\right)}\right)$ denotes the prediction loss, which is the cross-entropy loss in this work. In the second loss term, $M^{\left(i\right)}\in {ℝ}^{H\times W}$ denotes the model explanation generated for the $i\mathrm{th}$ sample using the given explanation method; $H^{\left(i\right)}\in {\left\{-\mathrm{1,0},1\right\}}^{H\times W}$ denotes the explanation annotations for the $i\mathrm{th}$ sample. Since the explanation annotations are different for positive and negative samples, $H^{\left(i\right)}$ is further represented by a combination of two types of binary explanation annotations as follows:

$$H^{\left(i\right)}=\left\{\begin{array}{c}{P}_{\mu }^{\left(i\right)}{y}^{\left(i\right)}=1\\ {-N}^{\left(i\right)}{y}^{\left(i\right)}=0\end{array}\right.,$$ (2)

where $N^{\left(i\right)}\in {\left\{\mathrm{0,1}\right\}}^{H\times W}$ denotes the non-nodule lesion annotations of negative samples, $P_{\mu }^{\left(i\right)}\in {\left\{\mathrm{0,1}\right\}}^{H\times W}$ denotes nodule region annotations of positive samples and $\mu$ denotes their consensus level. Typically, $\mu$ is set to 50%, meaning that a pixel is considered within the nodule region if it is included in annotations of at least two radiologists. For the positive $i\mathrm{th}$ sample, if the pixel at coordinate $(\mathrm{j},\mathrm{k})$ is within the nodule region, it is assigned $P_{\mu ,j,k}^{\left(i\right)}=1$ (i.e., $H_{j,k}^{\left(i\right)}=1$); otherwise, it is assigned $P_{\mu ,j,k}^{\left(i\right)}=0$ (i.e., $H_{j,k}^{\left(i\right)}=0$). For the negative $i\mathrm{th}$ sample, the pixel at coordinate $(\mathrm{j},\mathrm{k})$ is assigned $N_{j,k}^{\left(i\right)}=0$ (i.e., $H_{j,k}^{\left(i\right)}=0$) because non-nodule lesions are not annotated in the LIDC-IDRI dataset. The hyper-parameter $\lambda$ is used to adjust the impact of the explanation loss, thus the baseline can be seen as a special case of RES when $\lambda$ is equal to 0. In this work, $\lambda$ is set to 0.05 based on grid search via classification performance on the validation set. Now, we further specify the robust explanation loss term ${ℒ}_{Exp}$. First, assume that the ideal model explanation generated for the $i\mathrm{th}$ sample be ${\tilde{M}}^{(i)}={\tilde{H}}^{(i)}$ in the range $[-1,1]$, given that:

$${\tilde{H}}^{(i)}=\left\{\begin{array}{c}{\tilde{P}}_{\mu }^{(i)}{y}^{\left(i\right)}=1\\ -{\tilde{N}}^{(i)}{y}^{\left(i\right)}=0\end{array}\right.,$$ (3)

where ${\tilde{P}}_{\mu }^{(i)}\in [0,1]$ and ${\tilde{N}}^{(i)}\in [0,1]$ denote the ideal nodule region annotations of positive samples and non-nodule lesion annotations of negative samples, respectively. However, due to the potential inaccuracy of human nodule region annotations, ${\tilde{P}}_{\mu }^{(i)}$ remains unknown. Moreover, non-nodule lesions in negative samples are not annotated in the LIDC-IDRI dataset, resulting in ${\tilde{N}}^{(i)}$ being unknown either. Therefore, a hyper-parameter $\alpha$ is introduced to represent the assumption about the discrepancy between $H^{\left(i\right)}$ and ${\tilde{H}}^{(i)}$. The difference between $M^{\left(i\right)}$ and $H^{\left(i\right)}$ is measured by the L1 loss function followed by a normalization function $norm(\cdot )$, which divides the L1 loss by the non-zero area of the explanation annotations. The L1 loss was used to ensure the robustness of training attention-based deep learning models³³. Then, the modified ${ℒ}_{Exp}$ is as follows:

$$\max \left\{0,norm\left(\sum _i^N\left|M^{\left(i\right)}-H^{\left(i\right)}\right|\right)-\alpha \right\},$$ (4)

based on the sensitivity analysis of hyper-parameter $\alpha$ on LIDC-IDRI dataset in the following section, $\alpha$ is set to 0.003. With the assumption about ideal nodule and non-nodule distributions made, ${ℒ}_{Exp}$ is then further extended to learn from noisy human explanation annotations. To measure the absolute difference between the continuous $M^{\left(i\right)}$ and the binary $P_{\mu }^{\left(i\right)}$, $N^{\left(i\right)}$, hyperbolic tangent function together with a slope controller $\gamma$ is applied to binarize $M^{\left(i\right)}$:

$$tanh\left(\gamma \left(M^{\left(i\right)}-a\right)\right),$$ (5)

where $\gamma$ is set to 5 and $a$ is the threshold value, obtained by an efficient adaptive threshold searching algorithm. A distance function $dist(\cdot )$ is also applied to measure their relative distance:

$$dist\left(M^{\left(i\right)},h_{\varphi }\left(H^{\left(i\right)}\right)\right),$$ (6)

where $h_{\varphi }(\cdot )$ is a mapping function with kernel $\varphi$ that maps $P_{\mu }^{\left(i\right)}$ and $N^{\left(i\right)}$ to continuous values in the range [0, 1]. By switching kernels, different imputations are performed for the boundaries of human explanation annotations. The $dist(\cdot )$ is the L1 loss function because it is more robust.³³ In addition, an indicator function $1(\cdot )$ is added to force the model to learn only from the annotated regions. To summarize, the final ${ℒ}_{Exp}$ is as follows:

$$\begin{gathered} \underset{\theta ,a,\varphi }{\mathrm{m}\mathrm{a}\mathrm{x}}\left\{0,norm\left(\sum _i^N\left|tanh\left(\gamma \left(M^{\left(i\right)}-a\right)\right)-H^{\left(i\right)}\right|\cdot 1\left(H^{\left(i\right)}\ne 0\right)\right)-\alpha \right\} \\ +\sum _i^N\left|M^{\left(i\right)}-h_{\varphi }\left(H^{\left(i\right)}\right)\right|\cdot 1\left(H^{\left(i\right)}\ne 0\right), \end{gathered}$$ (7)

where $\theta$ is the given model parameter.

2.3.2. Framework formulation

Figure 2 depicts the structure of the proposed pulmonary nodule detection framework. The DNN model within the proposed framework can be trained using the explanation loss function defined in Eq. (7) with different kernel functions. The proposed framework using DNN models trained with a pre-defined Gaussian kernel and a learnable kernel are denoted RXD-G and RXD-L, respectively. RXD-G and RXD-L allow the underlying deep learning models and kernel functions to learn from the estimated version of the node annotations, thus reducing the chance of being misled by noise in the annotations. The proposed framework implemented RXD-G and RXD-L to explore which kernel functions can work adequately with DNN to achieve the nodule detection task. Ultimately, the model not only has high predictive power, but also generates a prediction basis that is more consistent with that of the radiologist. The commonly used ResNet18⁴⁷ is used as the backbone DNN model. The overall model structure is shown in Figure 2(b). No structural changes are required for RXD-G, while RXD-L adds an imputation layer at the bottom of the backbone DNN. The imputation layer is a convolutional layer for estimating the ideal nodule region by binary nodule region annotations, converting it to a continuous value in the range [0, 1].

Figure 2.

Overview of the RXD framework. Subfigure (a) illustrates the input data, which includes 2D image slices and nodule region annotations. For RXD-L, the input annotations are of the original shape (i.e., 224 × 224); while for RXD-G, the input annotations need to be down sampled first. Eventually, RXD-L and RXD-G produce attention maps with the size (i.e., 7 × 7). Subfigure (b) shows the overall model architecture when the backbone model is ResNet18. Subfigure (c) shows the output including the intermediate results (imputations and attention maps) of the model. In the case of RXD-L, the imputation is an annotation processed by the imputation layer. In the case of RXD-G, the imputation is an annotation processed by the Gaussian kernel. The attention map is produced by Grad-CAM (described in Section 2.3.3) and can be used to generate visualizations. Subfigure (d) contains symbol descriptions.

2.3.3. Deep neural network training

Existing methods for reducing false positives use only the information in the image, while both the image and the explanation annotation are used in explanation supervision. To generate model explanations that are comparable to human annotations, the widely used Grad-CAM²⁷ was used as the explanation technique. The model explanations generated via Grad-CAM were normalized to [0, 1], by dividing the maximum attention value for each sample. As shown in Figure 2, all images were resized to 224 × 224 to fit the backbone model (i.e., ResNet18 in this work), resulting in a 7 × 7 attention map. Then, the attention map was used to calculate the explanation loss with the corresponding explanation annotation, which was down sampled to the same size as the attention map. In this way, we can avoid distorting the original model explanation when scaling up. In the same sense, for visualization and explanation quality evaluation, we took the opposite approach: scaling up the attention map to the same size as the explanation annotation. In the process of calculating the explanation loss in RES, explanation annotations need to be imputed and converted to continuous values. Due to the differences in implementation, RXD-G imputes the explanation annotations after down sampling, while RXD-L imputes the original annotations. Figure 3 illustrates their different ways of imputation.

Figure 3.

Illustration of different imputation methods of RXD-G and RXD-L, where L-imputation and G-imputation represent the imputation methods of RXD-L and RXD-G, respectively. For RXD-G, original explanation annotations are first down sampled to 7 × 7. Then, the down sampled annotations are imputed by a 3 × 3 Gaussian kernel with Gaussian standard deviation of 0. For RXD-L, original explanation annotations are imputed by a convolution kernel of size 64, stride 32, and padding 16.

Based on that explanation supervision also exploits the rich information in explanation annotations and is thus particularly advantageous when only a few samples are available, we trained our method with sample sizes of 10, 30, 50 and 100. These samples were obtained by a balancing approach: first, half of the samples were drawn from all positive samples in the training set; second, the other half of the samples were drawn from all negative samples in the training set with the same patient IDs as the positive samples. Random horizontal flipping was applied to the samples in training. To accelerate the validation, each time we drew 10% of the positive and negative samples from the validation set. All models were trained using the Adam⁴⁸ optimizer with a learning rate set to 0.0001 and a batch size of 20. After up to 50 epochs of training, the model with the best AUC in validation was used for testing. Our method was implemented using the deep learning library Pytorch⁴⁹, based on a graphics processing unit (GPU) NVIDIA T4.

2.4. Performance evaluation

We conducted a comparative study to evaluate the performance of RXD-G and RXD-L. The benchmarks we used included baseline ResNet18, a false positive reduction method called Models Genesis (MG⁵⁰) and an explanation supervision method called HAICS³². MG is a powerful self-supervised learning framework that uses anatomical structures commonly found in medical images as supervised signals, significantly outperforming scratch learning models⁵⁰. Its 2D version was used in our comparison. HAICS is a recently proposed framework, which utilizes scribbled annotations as supervision signals³². Its implementation for image classification tasks was used in our comparison. For a fair comparison, all methods used the same backbone DNN (i.e., ResNet18) and were trained with the same settings as in Section 2.3.3.

The classification performance and explanation quality of our RXD-G and RXD-L, baseline (ResNet18), MG and HAICS in terms of reducing false positives are analyzed. The AUC (area under the curve) is used as a measure of classification performance. IoU (Intersection over Union) is used to measure the quality of explanation. This work aims to explore the RXD framework regarding small datasets learning, and we designed numerical experiment using training data with different sample sizes including 10, 30, 50, and 100 samples to evaluate the proposed framework. The framework integrates the deep neural networks and Gaussian and learnable kernels to achieve lung nodule detection with heat maps. These kernel functions with different random seeds results in stochastic outputs from the proposed framework. Based on the previous investigations^51,52, five trials with different random seeds can achieve consistent model response while training a deep learning-based model. This work focuses on the feasibility study of using explainable AI to learn from small datasets for accurate pulmonary nodule detection. Therefore, we used the suggested five trials from the previous work^51,52 for learning from each dataset with different sample sizes. In each trial, we sampled from the training and validation sets with different random seeds. A case study is also conducted to provide a human assessment of interpretability, where attention maps for all methods trained with sample sizes of 10, 30, 50 and 100 are shown. To further compare the explanation quality of all methods with sample sizes of 50 and 100, the IoUs calculated using different thresholds for the model explanations (since they are continuous) are shown.

3. Results

3.1. Sensitivity analysis of hyper-parameter $\alpha$

The hyper-parameter $\alpha$ in Eq. (4) represents the human expected priors of the ideal nodule and non-nodule distributions. Since the actual distributions are usually unknown, the optimal value for $\alpha$ varies depending on the dataset and the quality of explanation annotations. Figure 4 depicts the hyper-parameter sensitivity analysis for searching the optimal $\alpha$ values based on LIDC-IDRI dataset with different sample sizes. By definition, $\alpha$ ranges from 0 to 1. According to the curves in Figure 4, the model AUC and IoU vary with $\mathrm{\alpha }$, reaching a peak when $\mathrm{\alpha }$ equals 0.003 for all sample sizes. The optimal hyper-parameter setting was applied to both RXD-L and RXD-G, since they share the same assumption about ideal nodule and non-nodule distributions.

Figure 4.

Mean AUC and IoU for different hyper-parameter $\mathrm{\alpha }$ values of RXD-G with a sample size of (a) 10, (b) 30, (c) 50 and (d) 100 through five trials.

3.2. Quantitative analysis of model performance

The quantitative measurement of classification performance and explanation quality are presented in table 2. In general, our RXD-G and RXD-L outperformed other methods in both classification performance and explanation quality. Specifically, RXD-G beat all other methods in terms of AUC with sample sizes of 10, 30, 50 and 100, and had a higher IoU, showing a robust capacity. For RXD-L, although it did not differ significantly from the baseline in terms of AUC, it still offered a huge improvement in terms of IoU. Specifically, RXD-L improved IoU by 24.0% to 80.0% compared to the baseline. RXD-G achieved even greater improvements, improving by 38.4% to 118.8%.

Table 2.

Performance is compared to Baseline (ResNet18), Models Genesis (MG) and HAICS with sample sizes of 10, 30, 50 and 100. At each sample size, the best results for each metric are in bold, with the second-best results being double underlined. The values in parentheses indicate the percentage improvement compared to the baseline trained with the same sample size. The means and standard deviations are calculated based on five trials with different seeds.

Sample size	Method	AUC↑	Enhancement from Baseline	IoU↑	Enhancement from Baseline
10	Baseline	0.661±0.023	-	0.033±0.009	-
	MG	0.600±0.043	(−9.3%)	0.032±0.022	(−4.5%)
	HAICS	0.669±0.052	(1.1%)	0.027±0.015	(−18.3%)
	RXD-L (proposed)	0.665±0.058	(0.3%)	0.042±0.036	(24.0%)
	RXD-G (proposed)	0.670±0.021	(1.2%)	0.046±0.033	(38.4%)
30	Baseline	0.709±0.038	-	0.029±0.011	-
	MG	0.668±0.046	(−5.7%)	0.047±0.008	(63.2%)
	HAICS	0.710±0.056	(0.2%)	0.043±0.004	(47.6%)
	RXD-L (proposed)	0.698±0.061	(−1.6%)	0.050±0.006	(73.6%)
	RXD-G (proposed)	0.722±0.046	(1.9%)	0.063±0.015	(118.8%)
50	Baseline	0.753±0.025	-	0.039±0.015	-
	MG	0.669±0.060	(−11.2%)	0.042±0.009	(7.8%)
	HAICS	0.751±0.029	(−0.3%)	0.056±0.014	(44.4%)
	RXD-L (proposed)	0.752±0.013	(−0.2%)	0.064±0.007	(66.8%)
	RXD-G (proposed)	0.764±0.038	(1.4%)	0.069±0.010	(80.0%)
100	Baseline	0.834±0.018	-	0.050±0.006	-
	MG	0.735±0.040	(−11.9%)	0.059±0.006	(19.6%)
	HAICS	0.820±0.014	(−1.7%)	0.092±0.019	(86.7%)
	RXD-L (proposed)	0.831±0.016	(−0.4%)	0.089±0.013	(79.8%)
	RXD-G (proposed)	0.836±0.014	(0.2%)	0.086±0.007	(73.9%)

This suggests that for pulmonary nodules, the Gaussian imputation method works better on small sample sizes. This makes sense because a single-layer imputation block may not be capable of learning such a variety of nodule morphologies, while a pre-defined Gaussian kernel can be less affected.

To show the improvement in training sample utilization efficiency, we report the decrease in AUC of RXD-G compared to baseline. Figure 5 shows the percentage decrease in AUC of baseline and RXD-G trained with smaller sample sizes compared to the AUC of baseline trained on 100 samples. At each sample size, the longer the bar, the more AUC decreases from baseline trained on 100 samples. In all cases, the decrease in AUC of RXD-G is smaller than baseline, indicating that RXD improves the utilization efficiency of the training samples.

Figure 5.

Percentage decrease in AUC of baseline and RXD-G trained on smaller sample sizes compared to the AUC of baseline trained on 100 samples through five trials. The gray and blue bars represent the decrease in AUC for baseline and RXD-G, respectively. At each sample size, the longer the bar, the more the AUC decreases compared to the AUC of the baseline trained on 100 samples.

3.3. Qualitative analysis of explanation quality

A case study is conducted to compare the explanation quality of all models. Model-generated explanations are presented by overlaying a heatmap on the original image, where the warmer colored regions are given more importance. Figures 6–9 show some explanation visualization results of all models with sample sizes of 10, 30, 50 and 100, respectively. The first column is the original image, where nodule region is circled in red. The following columns are model explanations generated by baseline, MG (i.e., Models Genesis), HAICS, RXD-L and RXD-G, respectively. Ideally, the warm regions in model explanation should overlap precisely with the nodule regions in each row, indicating that the prediction is completely dependent on the nodule regions. In general, all models yielded more consistent explanations with the nodule regions when trained on a larger sample size. Baseline was more concentrated in non-nodule areas, showing a serious bias. MG can focus on the right area, but with severe imprecision. HAICS produced precise attributions but can be severely biased if trained on insufficient samples. RXD-L and RXD-G can correctly focus on the nodule region even when trained on a small number of samples, with RXD-L performing less biased and RXD-G more precise.

Figure 6.

Selected explanation visualization results of all methods with sample size of 10. In the first column, the ground truth nodule region is circled in red. The model-generated explanations are represented by heatmaps overlaid on the original images, where the warmer colored regions are given more importance. Baseline refers to ResNet18, MG and HAICS are comparative methods, while RXD-G and RXD-L are our methods.

Figure 9.

Selected explanation visualization results of all methods with sample size of 100. In the first column, the ground truth nodule region is circled in red. The model-generated explanations are represented by heatmaps overlaid on the original images, where the warmer colored regions are given more importance. Baseline refers to ResNet18, MG and HAICS are comparative methods, while RXD-G and RXD-L are our methods.

3.4. Quantitative analysis of explanation quality

The previous experiment shows the model explanations for only a few samples. For a more comprehensive study, we changed the threshold of attention values in model explanations and recomputed IoU. When the threshold is small (e.g., 0.1), regions with very low influence on the model prediction results are also used to compute the degree of overlap with the nodule region; when the threshold is high (e.g., 0.9), only regions with high influence on the model prediction results are used to compute the degree of overlap with the nodule region. Using Figure 9 to better illustrate, the yellow-green areas in the model explanation may have an attention value of 0.1, while the red areas in the model explanation may have an attention value of 0.9. Figures 10 and 11 show the IoU obtained by each model using different thresholds of attention values when the sample size is 50 and 100, respectively, with the horizontal axis showing the different attention value thresholds and the vertical axis showing the average IoU, and all data points are the average of the model test results obtained through five trials. In general, the highest IoU is obtained for all models when the threshold is around 0.8. When the sample size is 50, RXD-G and RXD-L obtained the highest and second highest results at each threshold, respectively, with HAICS in third place, while Models Genesis and Baseline were significantly lagged. This contrast is more pronounced when the sample size is 100 and the gap between HAICS and RXD-L and RXD-G became smaller. Regardless of the sample size, the IoU of Models Genesis and baseline is low at every threshold, suggesting that neither relies primarily on the nodule region for classification. In contrast, explanation supervision methods (HAICS, RXD-L and RXD-G) are highly dependent on nodule regions, suggesting that explanation supervision methods are able to learn attributions that are more consistent with human cognition.

Figure 10.

The IoU obtained by each method using different attention value thresholds when the sample size is 50. The horizontal axis shows the different attention value thresholds, and the vertical axis shows the average IoU. All data points are the average of the model test results obtained through five trials.

Figure 11.

The IoU obtained by each method using different attention value thresholds when the sample size is 100. The horizontal axis shows the different attention value thresholds, and the vertical axis shows the average IoU. All data points are the average of the model test results obtained through five trials.

4. Discussion

Pulmonary nodule detection is essential for early diagnosis of potential lung cancer, which can significantly improve the survival rate. Since not all nodules are cancerous, it is critical for a computer algorithm to track the geometrical feature of nodules such that radiologists can evaluate if a nodule is growing over time. Training on images without any guidance will cause the model to focus on meaningless regions. Whereas humans, especially experts, focus their attention on important parts with key information (e.g., module regions), which would be more effective. Aligning the model's focus with that of humans through an explanation-guided training process prevents sub-optimal models from wasting attention on trivial regions. With a robust learning framework, the model is able to filter annotation noise, resulting in better performance. The proposed pulmonary nodule detection framework delivers both classification results to indicate the presence of nodules and heat maps to present the likelihood of how DL models learn nodule distribution based on patient anatomy. Such a framework can help us understand if the DL model predicts the results by coincidence or if the model learns the spatial distribution of pulmonary nodules, which is helpful for lung lesion prognosis.

Table 2 indicates that the baseline consistently achieves stable classification performance with an average AUC of 0.74±0.07 regarding different training sample sizes. However, its average IoU is 0.04±0.01, which indicates that the baseline does not learn much of the nodule morphology. This model has limited values as it cannot provide spatial details of nodules to allow radiologists to further investigate the risk of potentially cancerous lesions. For small dataset learning (sample size=30), the proposed RXD-G can enhance the AUC and IoU values by 1.9% and 118.8%, compared to the baseline. Even in the extreme case (sample size=10), RXD-G can still increase the AUC and IoU values by 1.2% and 38.4% from the baseline model. Figure 5 shows that the AUC of RXD-G drops by 13.4% when the training sample size decreases from 100 to 50. In contrast, the AUC of the baseline reduces by 15%. These observations suggest that the proposed framework achieves the optimal nodule classification performance and explanation quality for small dataset learning, including sample sizes from 10 to 50.

Interpretability is another important aspect to assess an explainable AI. In this sense, beyond the calculation of IoU in a single case, a more comprehensive comparison is needed. Figures 6–9 exhibit model explanations in real scenarios where the model explanations are overlaid on the original images in the form of heatmaps. Nodule regions are relatively small and provide a difficult attribution test for all methods. The baseline and the two comparative methods expose their poor attribution, with either high bias or high variance. While our RXD-L and RXD-G constantly focus on the nodule region, with RXD-L less biased and RXD-G more precise. The color indicates the importance for the prediction. A model can assign high importance to incorrect regions, or low importance to correct regions. To evaluate model attributions in all cases, we change the threshold of attention values and recompute IoU. Figure 10 and Figure 11 show the IoU curves with different thresholds, where there is a clear distinction between explanation supervision methods and others. In particular, when sample size is small (sample size=50), our RXD-L and RXD-G have better consistency between model attributions and expert attributions.

DNN with multilayer perceptrons has been proven to be a universal approximator⁵³, and its hierarchical model structures can determine the underlying non-linear correlation behind the data^54,55. However, the selection of hyper-parameters can impact the learning performance of DNN. Figure 4 shows the sensitivity analysis for the hyper-parameter $\mathrm{\alpha }$, which regularizes the explanation loss given in Eq. (4). Since the goal is to achieve optimal nodule classification and morphology results, the $\mathrm{\alpha }$ should be 0.003 in this work to maximize the AUC and IoU values. The proposed framework can achieve optimal AUC and IoU values when the training samples are less than 50. The generated heatmaps indicate the importance of weighting for nodule regions on the CT images. Future work will likely focus on providing the probability distribution maps of lung nodules to support the diagnosis and prognosis of cancerous lung nodules and risk-informed medical decision-making.

5. Conclusions

A pulmonary nodule detection framework was demonstrated with expandability using the nonproprietary LIDC-IDRI dataset. The proposed framework enables the DL models to learn nodule classification, and most importantly, the proposed method ensures that the models consistently forecast nodule morphology as expected from human perspectives. The method can enhance the accuracy of computer-aided nodule detection systems and reduce the uncertainty of misclassification due to human factors. Such a method can reduce the workload of radiologists and enable them to focus on investigating the prognosis of cancerous nodules for early treatment to improve patient outcomes and survival rates.

Figure 7.

Selected explanation visualization results of all methods with sample size of 30. In the first column, the ground truth nodule region is circled in red. The model-generated explanations are represented by heatmaps overlaid on the original images, where the warmer colored regions are given more importance. Baseline refers to ResNet18, MG and HAICS are comparative methods, while RXD-G and RXD-L are our methods.

Figure 8.

Selected explanation visualization results of all methods with sample size of 50. In the first column, the ground truth nodule region is circled in red. The model-generated explanations are represented by heatmaps overlaid on the original images, where the warmer colored regions are given more importance. Baseline refers to ResNet18, MG and HAICS are comparative methods, while RXD-G and RXD-L are our methods.