Efficiency and reproducibility in pulmonary nodule detection in simulated dose reduction lung CT images

Highlights

• •Simulated reduced dose images were interpreted to detect the impact on nodule detection efficiency and reproducibility.
• •The repeated reading of the original images was performed to remove the effect of intraobserver variability.
• •The result with the simulation image at a standard deviation of 21 was similar to that of original images.
• •The result with the simulation image at a standard deviation of 29 may be inferior to that of original images.
• •The reading time was not extended in the simulated images despite the marked subjective image quality deterioration.

Abstract

Purpose

To determine the reproducibility and productivity of reduced dose chest computed tomography (CT) using a nodule detection task.

Materials and methods

Eighty-eight consecutive non-contrast CT examinations were performed using an automatic exposure system with a reference standard deviation of 8.5. Simulated raw data of a reduced dose scan (standard deviation at 21 and 29) were generated with a dose simulator. Original and simulated raw data were reconstructed to series of 7-mm-thick images (Original, Simulation A, Simulation B). In the first part of the reading experiment, three readers independently interpreted these images (88 cases × 3 series) and recorded the size, type, and location of the pulmonary nodules. The reading time for every case was recorded. In the second part of the experiment, the repeated interpretation of standard dose images was performed by two readers. Concordance or discordance of nodule detection between the first and the repeated reading result was assessed.

Results

A statistically significant difference in the detected nodule counts for lesions less than 5 mm by one reader was observed in simulation B images. Discordance of the interpretation result was found only in ground-glass nodules larger than 5 mm detected by one reader in simulation B images. There was no statistically significant difference in the reading time among the three image types.

Conclusion

Simulated standard deviation 21 images can reproduce the image interpretation result of original images, whereas simulated standard deviation 29 images may compromise the accuracy of nodule assessment. The effect on the reading time was not observed with dose reduction simulation.

1. Introduction

There is a concern about the adverse effect to patients caused by medical imaging, primarily the induction of malignant tumors as a late effect [1]. Since computed tomography (CT) examinations are responsible for the majority of radiation exposure by medical images, dose reduction in CT is attracting attention [2].

Among the radiation dose reduction measures, automatic exposure control (AEC) has contributed significantly to the optimization of radiation exposure in the CT examination [3], and it is recommended to use the automatic exposure control (AEC) function of the CT scanners to minimize radiation damage and capture images at the lowest set dose possible [4]. Previous studies have demonstrated the efficacy of AEC systems in radiation dose reduction [5,6].

Nonetheless, the selection of the optimal image quality setting is crucial. Dose reduction might lead to lower accuracy of the interpretation results or require a higher level of attention by the readers. Images with a high level of noise and artifacts may make it challenging for the radiologists to deliver accurate interpretation results. Images of lower quality also may decrease the productivity of the radiologists. Therefore, it is necessary to elucidate the influence of dose reduction on the image interpretation results and the reading time.

Comparison of multiple images obtained with different radiation doses is necessary to assess the effect of dose reduction. Acquiring the standard dose image and the simulated dose image is a viable method [7]. However, there is a potential problem of the increase in radiation exposure to the study subjects. Because the images obtained with multiple scans are not identical, the difference between the two images might lead to an overestimation of the discrepancy in the interpretation results [8].

By using the dose reduction simulation, it becomes possible to compare the image by the standard dose and the interpretation of the dose reduced image [9]. Moreover, acquiring multiple simulation images for comparison is achievable [9].

The purpose of this study is to assess the efficiency and reproducibility of nodule detection and to evaluate the effect in the reading time by using simulated reduced dose images with two simulation images.

2. Materials and methods

The institutional review board approved this retrospective study, and the requirement to obtain informed consent was waived. Eighty-eight consecutive patients who underwent non-contrast chest CT and were identified based on the radiology reports of the examination were included in this study.

2.1. CT examinations

All CT examinations were performed on a 64 detector-row CT scanner (Aquilion 64; Canon Medical Systems Corporation, Otawara, Tochigi, Japan). The scans were conducted by the routine non-contrast enhanced chest CT protocol. Tube current was adjusted using the automatic exposure control system equipped with the scanner, with the target image quality was set to a standard deviation (SD) of 8.5 for soft tissue reconstruction. The other scan parameters were as follows: peak tube voltage of 120 kV, gantry speed of 0.5 s per rotation, slice collimation 0.5 mm × 64, table feed 52 mm/s, and pitch factor 0.813.

2.2. Preparation of simulation images

The projection data of the 88 patients were retrieved from an archive. The projection data were anonymized by removing all the patient-specific data and transferred to the noise simulator [10]. Simulated projection data were generated in the simulator to simulate the data obtained with the target at the SD values of 21 (simulated raw data A) and 29 (simulated raw data B). These simulated data were transferred back to a CT scanner.

A series of contiguous 7-mm-thick images were reconstructed from each of three projection data sets (original, simulation A, and simulation B) with a standard lung reconstruction algorithm (FC 51). The scan parameters were deleted from DICOM files for blind interpretation.

The 264 image series, consisting of 88 original image series and the same number of simulation images (simulation A and simulation B images) were organized in random order into three image series sets with image series of the 88 unique cases. Therefore, three image series (original, simulation A, and simulation B) of one patient appears only once in one image series set.

2.3. Reading experiment

The image interpretation experiment has two parts. In the first part, two board-certified radiologists with 20 and 13 years’ experience and a radiologist in training with 4 years’ experience (reader A, B, and C, respectively) participated in the reading experiment. The three readers had three interpretation sessions to complete the reading of three image series sets, with washout periods of at least 1 month between the sessions.

The readers interpreted the 88 image series in each reading session. Images were viewed in a DICOM viewer software distributed for a research purpose (DICOM Viewer, Yakami DICOM tools) [11,12]. The readers reviewed the images with a lung window setting (window level 600 HU; window width 1500 HU). The readers were requested to detect all nodules or masses that appeared in the images and record the location, size, type, and number of lesions. The lungs were divided into six areas (right upper lobe, right middle lobe, right lower lobe, left apicoposterior and anterior segments, left lingular segments, and left lower lobe). Nodules of less than 5 mm in diameter, nodules of 5–9 mm in diameter, nodules of 10–29 mm in diameter, and nodules of 30 mm or more in diameter were recorded separately. Nodules of 5 mm or more in diameter were classified into pure ground-glass, mixed, or solid types. At the same time, the overall image quality of the image series was recorded by the readers using a five-point scale (1, very good; 2, good; 3, fair; 4, poor; 5, very poor).

In the second part of the reading session, two of the three readers had the second reading sessions of the original images. The repeated interpretation was conducted to determine whether there is a statistically significant difference between (1) the interpretation discrepancies between interpretations of the original and those of the simulated images and (2) the interpretation discrepancies between the first and the second interpretations of the original images.

In this part, the 88 original image series were presented in random order. The result recoding methods were identical to the previous sessions, but the reading time and subjective assessment of image quality were not recorded at this time. The repeated interpretation was made after a washout period of at least 2 months after the previous interpretation of three randomized image sets.

2.4. Statistical analysis

All statistical analyses were performed with R x64 3.3.2 (The R Foundation for Statistical Computing, Vienna, Austria) [13]. Objective image quality was assessed by calculating SDs of the Hounsfield Unit values in the regions of interest set on the area outside the body of the subject. Regions of interest were placed on the image of the four anatomical levels (at pulmonary apex, aortic arch, interventricular septum, and right diaphragm level).

Differences in the number of nodules detected in 528 (6 areas × 88 cases) anatomical areas in the first part of the experiment were evaluated with the Wilcoxon signed-rank test for three readers. Based on the results of the repeated interpretation session by two readers in the second part of the experiment, the differences in the observed number of nodules between the first part of the reading session (original and simulated images) and the second part of the reading session (original images) were determined for 528 regions. Concordance or discordance of the observed number of nodules between the two parts was determined in the 528 regions for four groups of lesions, as follows: (a) nodules of less than 5 mm in diameter, (b) ground-glass nodules of 5 mm or more in diameter, (c) mixed or solid nodules of 5–10 mm in diameter, and (d) mixed or solid nodules of 10 mm or more in diameter. Thus, the 528 observations in the first part of the experiment by two readers for the original and two simulation images are judged to be concordant or discordant. Then, the observations of pairs of two image types (original and simulation A, original and simulation B) were organized in contingency tables. The difference in concordant or discordant observations between the two types of images was evaluated with the McNemar test [9].

Differences in the SDs of the CT numbers in the 352 (4 levels × 88 series) regions of interest calculated were evaluated with a paired t-test The differences in the subjective image quality among original and simulated reduced dose images were assessed with the Wilcoxon signed rank test. The difference in the reading time was evaluated with a paired t between the original and simulated images. The difference was considered to be significant when the p-value was less than 0.05.

3. Results

The subjects consisted of 44 female and 44 male patients with a mean age of 64.4 years. The patient body weight ranged from 39.0 to 88.8 kg (mean ± SD, 58.6 ± 11.9). The patients underwent CT examinations for the follow-up of known lung nodules (33 patients), assessment of suspected lung nodules (12 patients), screening for lung lesion (11 patients), known lung cancer (10 patients), known lung metastasis (10 patients), postoperative surveillance (8 patients), and suspected pulmonary metastases (4 patients).

The SDs of the CT numbers in the regions of interest are presented in Table 1. The average SD in the regions of interest in original, simulation A, and simulation B images ranged from 36.6 to 69.5, from 83.39 to 103.8, and from 118.9 to 160.7, respectively. Examples of original and simulation images are presented in Fig. 1, Fig. 2, Fig. 3. There were statistically significant differences in the SD between original and simulation A or B images (p < 0.001). The combined image quality scores reported by three readers are shown in Table 2. There were statistically significant differences between original and simulation A or simulation B images (p < 0.001).

Table 1.

The standard deviations of the CT numbers in the regions of interest.

		Image types
		Original	Simulation A	Simulation B
Image level	Pulmonary apex	37±13.1	101.2±48.9	160.7±82.2
	Aortic arch	36.6±14.6	92.7±36.7	147.7±64.3
	Interventricular septum	45.9±41	83.4±36.1	119±38.2
	Right diaphragm	69.5±85.8	103.8±79.3	138.4±80.1

Statistically significant difference between original and simulation A images and simulation B images (p < 0.00001).

Fig. 1.

Area of ground-glass opacity in the right lower lobe measuring 7 mm was detected by a reader on the original image (left) but missed on the simulation A image (middle) and simulation B image (right). Another 5-mm solid nodule (not shown) is found in the right middle lobe and identified on both original and simulation images.

Fig. 2.

Area of ground-glass opacity measuring 21 mm in the left upper lobe was detected by a reader on the original image (left) and simulation A image (middle) and missed on simulation B image (right). The increased noise and streak artifacts decrease disease conspicuity.

Fig. 3.

A solid 5-mm nodule in the left upper lobe is depicted in the original image (left) with a well-defined border. The margin of the nodule become blurred by the overlapping streak artifacts in simulation A image (middle) and almost effaced in simulation B image (right).

Table 2.

Image quality scores recorded by three readers for original images and simulation A and B images.

	Original	Simulation
		A	B
1, Very good	162	1	0
2, Good	96	24	0
3, Fair	6	198	46
4, Poor	0	40	205
5, Very poor	0	1	13

Table 3 shows the number of nodules recorded by the three readers. There is no statistically significant difference in the number of detected nodules between original and simulation A images. Nodules less than 5 mm in diameter were identified more frequently on original images than on simulation B images (p < 0.05) by two readers.

Table 3.

The number of detected nodules.

		Reader 1				Reader 2				Reader 3
		<5 mm	5–9 mm	10–29 mm	≥30 mm	<5 mm	5–9 mm	10–29 mm	≥30 mm	<5 mm	5–9 mm	10–29 mm	≥30 mm
Original	GGN	57	34	11	0	3	36	16	1	23	19	6	0
	Solid	44	49	24	4	165	79	39	5	265	73	23	4
	Total	101	83	35	4	168	115	55	6	288	92	29	4
Simulation A	GGN	39	36	10	0	7	39	20	0	29	28	6	0
	Solid	31	37	26	4	164	91	39	5	211	56	24	5
	Total	70	73	36	4	171	130	59	5	240	84	30	5
Simulation B	GGN	35	38	8	0	9	26	17	0	35	29	4	2
	Solid	32	36	28	5	152	84	38	5	187*	53	27	4
	Total	67*	74	36	5	161	110	55	5	222	82	31	6

Simulation A images, simulated images for standard deviation of 21; simulation B images, simulated images for standard deviation of 29; GGN, ground-glass nodules;*p-value <0.05 (Wilcoxon signed-rank test).

The results of the interpretation result correlation between the two parts of the reading experiment were represented in 2 × 2 contingency tables (Table 4). There were no significant differences in concordance between the original and simulated images, except for the ground-glass nodules larger than 5 mm detected by Reader 1, where more incidences of discordance were observed in simulated B images. This finding suggests that the simulation B images for the SD of 29 could not be equivalent to the original image regarding diagnostic accuracy.

Table 4.

Concordance and discordance between the first reading and the second reading.

Simulation A images
Reader 1		Simulated		Reader 2		Simulated
		Concordant	Discordant			Concordant	Discordant
(a) All nodules < 5 mm
Original	Concordant	451	30	Original	Concordant	383	38
	Discordant	18	29		Discordant	38	69
			p = 0.083				p = 1
(b) Pure ground-glass nodules > 5 mm
Original	Concordant	428	40	Original	Concordant	472	21
	Discordant	30	30		Discordant	22	13
			p = 0.232				p = 0.879
(c) Solid or mixed nodules >5 and ≤ 10 mm
Original	Concordant	483	16	Original	Concordant	443	27
	Discordant	13	16		Discordant	22	36
			p = 0.577				p = 0.475
(d) Solid or mixed nodules >10 mm
Original	Concordant	481	21	Original	Concordant	435	29
	Discordant	12	14		Discordant	29	35
			p = 0.117				p = 1

Simulation B images
Reader 1		Simulated		Reader 2		Simulated
		Concordant	Discordant			Concordant	Discordant
(a) All nodules < 5 mm
Original	Concordant	452	29	Original	Concordant	376	45
	Discordant	22	25		Discordant	46	61
			p = 0.327				p = 0.917
(b) Pure ground-glass nodules > 5 mm
Original	Concordant	422	46	Original	Concordant	472	21
	Discordant	28	32		Discordant	19	16
			p = 0.036				p = 0.752
(c) Solid or mixed nodules >5 and ≤10 mm
Original	Concordant	483	16	Original	Concordant	437	33
	Discordant	15	14		Discordant	30	28
			p = 0.857				p = 0.705
(d) Solid or mixed nodules >10 mm
Original	Concordant	477	25	Original	Concordant	428	36
	Discordant	13	13		Discordant	36	28
			p = 0.052				p = 1

Simulation A images, simulated images for standard deviation of 21; simulation B images, simulated images for standard deviation of 29.

The reading times of the three readers for original images, simulation A images, and simulation B images are presented in Table 5. There were no statistically significant differences in the reading time spent by the reader between the original images and simulation A or B images (p > 0.05). The differences in the reading time between readers 1, 2, and 3 were statistically significant for original images and simulation A and B images.

Table 5.

Reading time spent on one case by three readers.

	Original (mean ± SD)	Simulated A (mean ± SD)	p-value	Simulated B (mean ± SD)	p-value
Reader 1	409.9±245.7	409.6±317.2	0.995	371.1±199.2	0.2
Reader 2	320.3±96.1	323.8±99.6	0.77	344.4±100.2	0.057
Reader 3	125.4±63.1	115.4±50	0.059	122.2±59.5	0.597

Means and standard deviations of the reading time are expressed in seconds. P-values for the differences between original images and simulated images are shown for simulation A images A and simulation B images.

4. Discussion

The noise level of the original images and the simulation A and B images differed significantly. The measured SDs of lung reconstruction images are approximately five times higher than that of the reference SD. The measured value supported that the simulation of reduced dose images is carried out as expected. The reference SD in simulated A image (SD 21) and B image (SD 29) is 2.41 and 3.41 times higher than the original image (SD 8.5). Therefore, if the actual reduced dose scan replaced the original scan, it would lead to approximately 6.10 and 11.6 times less radiation dose, although the amount of reduced dose depends on the image quality setting of the original scan.

The image quality scores clearly showed the significant difference in subjective image quality between the three types of images. The median score for the simulation B turned out to be 2 (poor), suggesting that the radiologists would not accept the use of images with this image noise level.

The number of detected nodules was similar for all the images. However, the difference is observed in two subgroups of nodules less than 5 mm in diameter, suggesting the possible decrease in the sensitivity of small nodules in the simulated reduced dose image for the SD of 29. This result is congruent with the result of the concordance rate. The simulation B images could negatively affect the detection of the ground-glass nodules.

In comparing the reading time of the three readers in the first part of the experiment, no difference between the image types were detected. The reading efficiency was not affected using the simulated reduced dose images, even with the simulation B images, the image quality of which is judged to be poor.

Simulated reduced dose images were used in this study. In general, to demonstrate the validity of reduced dose images, the results of the reduced dose images need to be compared with the reading results using the “standard” images that are deemed to have unequivocally sufficient image quality. Therefore, the review of multiple images of the same subject is necessary. In this study, instead of repeating the scan on the same patient, the reduced dose images were obtained through the reduced dose simulation. Simulation of reduced dose images was used in previous studies [14,9,15,10]. The simulation method applied in this study is based principally on the same method as the earlier simulation study [10].

The comparison of the diagnostic result is ideally made with the gold standard. However, building a gold standard is not always feasible, and therefore the diagnostic equivalence needs to be confirmed by other methods. Thus, the repeated reading of the original images was conducted in this study.

Direct comparison of the two diagnostic results (original and reduced dose images) may significantly underestimate the agreement rate of the observations between the two image series because of the intra-observer disagreement. It is necessary to eliminate the untoward effect of the internal disagreement on the appraisal of the diagnostic utility of the reduced dose images. Therefore, in addition to the reading result of the original and simulated images in the first part of the experiment, separate reading results of the original images were collected in the second part of the experiment. Based on the concordant and discordant observations, the reproducibility of the interpretation result was assessed with the McNemar test [9]. If the concordant and discordant observations were significantly different between the two types of images, the pair of images were deemed diagnostically inequivalent.

The current result implies that simulation B images might compromise the diagnostic accuracy. The number of nodules and the detection reproducibility with simulation B images are inferior to the original images in some subclasses. An increased noise level decreases the lesion conspicuity and possibly lowers the sensitivity for lesions. The lower limit of HU in normal lung was conventionally set at -950 HU in the emphysema quantification study [16]. The median HU for adenocarcinoma in situ was reported to be -667 ± 112 [17]. The difference of median HU between normal lung and ground-glass lesion is estimated to be less than 300. The average SD in the regions of interest in simulation B images ranged from 118.9 to 160.7. The high level of noise could hinder the recognition of low-contrast lesions as demonstrated in Fig. 1, Fig. 2. For solid lesions, the tolerable noise level should be markedly higher than for ground-glass lesions. However, clustered bands of streak artifacts may blur the margin of the lesion, as in the case in Fig. 3. The loss of lesion margination can preclude the recognition of the small lesions.

In this study, the effect on the nodule detection task was evaluated. The result suggests that the accuracy of nodule detection does not change with simulation A image but might be compromised with simulation B image, for the nodules less than 5 mm in diameter and for ground-glass nodules.

A recently published guideline stated that nodules less than 6 mm in diameter would not need a routine follow-up examination in low-risk individuals [18]. A follow-up study of persistent ground-glass nodules without a solid component is recommended every 2 years. Some nodules less than 5 mm might remain unnoticed with the simulation B image, but it does not change the management plan for low-risk individuals. Pure ground-glass nodules might not be accurately identified in simulation B images, which may influence the management. Therefore, routine application of the images comparable to the simulation B image might not be recommended, although the risk of missing the lung cancers that require prompt intervention is expected to be insignificant.

The result of this study derives from the experiment using images produced with filtered back projection. Images reconstructed with iterative reconstruction (IR) have different image characteristics which may alter the detectability of lesions. Although lowered noise of IR image improves the visibility the faint nodules, IR images are known to appear "plastic-like" or blurred [19], which might obscure small ground-glass nodules. As the balance of two opposing effects depends on the type and the implementation method of IR, it is difficult to make a generalized conclusion about the impact of IR on the nodule detection task. More researches need to be carried out to elucidate the effects of reconstruction methods on the nodule diagnosis. One study suggested that the detectability of the subsolid nodules using IR methods are acceptable [20], implying that the use of IR does not impede the detection of nodular lung lesions.

Recently, the technology of artificial intelligence is progressing rapidly, and computer-assisted detection (CADe) may be an integral piece in the working environment of the radiologists, especially in mass-screening settings [21]. Because CADe functions involve various pre-processing steps, the optimal image quality as CADe input data may differ from that as the optimal image quality for the human readers. Consequently, the result of the current study may not be directly translatable into the computer-assisted pulmonary nodule detection. However, even if the CADe becomes integrated into the expertise of the diagnostic radiology, human eyes need to examine the result from CADe system, and the result of the current study still holds valid.

There are limitations in the current study. First, 7-mm-thick images were used for review. Currently, thinner images are more commonly used. The archived raw data are originally acquired to produce 7-mm images. The dose simulation system works to produce the simulated images of the same thickness. Generation of images with different thickness would make the study result harder to interpret. Therefore, 7-mm images were used for the reading experiment. Second, the automatic exposure control system used in the current study is based on the selection of SD. The implementation of the function varies among the vendors [[22], [23], [24]]. In other scanner models, the image quality is specified with noise index, reference mAs, baseline mAs, or others. Therefore, the direct application of the current result to scanner models from other manufacturers is not possible.

In conclusion, the simulated images at reference SD of 21 can be used with the same level of nodule detection sensitivity, reproducibility, and reading efficiency. The images with the reference SD of 29 might result in inferior diagnostic accuracy.