A virtual clinical trial using projection-based nodule insertion to determine radiologist reader performance in lung cancer screening CT

Abstract

Task-based image quality assessment using model observers is promising to provide an efficient, quantitative, and objective approach to CT dose optimization. Before this approach can be reliably used in practice, its correlation with radiologist performance for the same clinical task needs to be established. Determining human observer performance for a well-defined clinical task, however, has always been a challenge due to the tremendous amount of efforts needed to collect a large number of positive cases. To overcome this challenge, we developed an accurate projection-based insertion technique. In this study, we present a virtual clinical trial using this tool and a low-dose simulation tool to determine radiologist performance on lung-nodule detection as a function of radiation dose, nodule type, nodule size, and reconstruction methods. The lesion insertion and low-dose simulation tools together were demonstrated to provide flexibility to generate realistically-appearing clinical cases under well-defined conditions. The reader performance data obtained in this virtual clinical trial can be used as the basis to develop model observers for lung nodule detection, as well as for dose and protocol optimization in lung cancer screening CT.

1. Introduction

Tremendous efforts have been devoted to answering an important question in clinical CT: How low can the radiation dose be reduced without sacrificing diagnostic performance? The key to this question relies on accurate image quality assessment at various conditions.

Human observer studies by radiologists for a specific clinical task are the reference standard for task-based image quality assessment. However, two major challenges exist. First, human reader study is extremely resource intensive and results are sometimes difficult to interpret. Second, a large number of patient cases with positive findings are typically required. Collection of sufficient number of cases with desired distribution of lesion characteristics, as well as establishing the ground truth through follow-ups, is extremely time-consuming [1].

To overcome the first challenge, task-based image quality assessment using model observers are currently under active investigation, which may provide an efficient and quantitative approach to dose optimization [2–4]. However, before task-based metrics can be used in practice, their correlation with radiologist performance needs to be established. As the basis for model observer development, it is important to determine radiologist performance at various dose and reconstruction configurations for specific clinical tasks such as lung nodule detection.

To overcome the second challenge, a large number of positive cases can be efficiently generated by using a projection-based lesion insertion tool, which was recently developed and validated in our lab [5, 6]. In addition, a low-dose simulation tool can be used to simulate multiple dose levels from existing patient cases at routine dose level [7].

In this study, we present a virtual clinical trial using the projection-based lesion insertion and low-dose simulation tools to determine radiologist performance on lung-nodule detection as a function of radiation dose, nodule type, nodule size, and reconstruction methods. To the best of our knowledge, this is the first virtual clinical trial utilizing an accurate projection-based nodule insertion method. A previous study exist to determine radiologist performance on nodule detection in pediatric CT [8], but an image-based nodule insertion tool was used, which may not be able to maintain the true appearance of nodule due to the lack of incorporation of image reconstruction in the nodule insertion process.

2. Methods

2.1 Projection-based nodule insertion and low-dose simulation

We recently developed and validated a projection-based lesion insertion technique [5, 6], which inserts segmented lesions to projection data of patients prior to image reconstruction. This technique can mimic the realistic appearance of the lesion since the impact of image reconstruction on lesion appearance and noise texture is inherently incorporated. In addition, the diagnostic task can be well controlled by specifying the lesion characteristics (location, size, attenuation). The technique has been validated in a patient study involving 32 inserted lung nodules [6], which demonstrated that it is difficult for radiologists to differentiate those inserted nodules from real ones.

After nodule insertion on the projection data, noise was inserted to simulate multiple dose levels. Noise model incorporated the effect of bowtie filter, automatic exposure control, and electronic noise and was validated on the scanner [7]. Projection data were finally reconstructed using either the filtered-backprojection (FBP) or iterative reconstruction (IR) methods. A flowchart of the process is shown in Figure 1.

Figure 1.

A flowchart illustrating the process of lesion insertion and low-dose simulation from patient CT raw data.

2.2 Patient case preparation for virtual clinical trial

Fifty patient cases were retrospectively selected from an existing case library on low-dose lung cancer screening CT acquired from a 192-slice scanner. Inclusion criteria is that each lung at mid-level (middle lobe/lingula, within an approximate 3 cm range) is completely negative for nodules. In the original case library, each patient was scanned twice: routine low-dose chest CT (120 kV, 50 quality reference mAs (QRM), nominal CTDIvol 3.6 mGy) and ultra-low-dose CT (100 kV with an added tin filter, 50 QRM, nominal CTDIvol 0.17 mGy). Only the 120 kV scans were used since that is a relatively higher dose and multiple lower-dose levels needed to be simulated. To save reading time, only a 3 cm length from each lung was used (30 images).

Two nodules were used, one ground glass nodule (GGN, 5.4 mm, −660 HU), one part-solid nodule (PSN, 3.4 mm, −442 HU). The GGN was also digitally modified to create 2 additional sizes (3.4 and 7.4 mm). The nodule was inserted into a predetermined location, following the guidance of a radiologist using the following rules: (1) Have the nodule abut a small blood vessel (solid, but fainter, white lines); (2) Don’t cross an airway or fissure; (3) Don’t place a nodule against the margins of the lung; (4) If possible, orient oval nodules so the long axis is oriented radially from the center of the chest. Each lung had at most one nodule.

After nodule insertion, noise was inserted to the full dose data to simulate 3 additional dose levels: 50%, 25%, 10%. The QRM of the four dose levels are: 50, 25, 12.5, and 5, corresponding to nominal CTDIvol of 3.6, 1.8, 0.9, 0.36 mGy, respectively. The 25 QRM corresponds to the dose level used in the national lung-cancer screening trial (NLST). To evaluate the impact of nodule attenuation, nodule size, radiation dose, and reconstruction method, images at 12 different conditions were generated: (1–4): GGN, 100%, 50%, 25%, 10% dose levels, reconstruction kernel Bv49 with a IR strength setting of 2 (Bv49-2, ADMIRE, Siemens Healthcare); (5–8): PSN, 100%, 50%, 25%, 10% dose levels, Bv49-2; (9–10): GGN, 2 additional sizes, 50% dose level, Bv49-2; (11–12): GGN, 50% dose level, 2 additional reconstructions Bv49 and Bv49-4. Each condition consists of 100 lungs, 70 of them containing nodules, 30 without nodules.

2.3 Observer studies

This is a signal-known-exactly (SKE) detection and localization task in the lung. A Matlab-based graphical user interface was developed for this human observer study. The nodule to be detected is displayed at the side of the test images. Four human observers (2 radiologists, 2 fellows) were enrolled to perform the reading. Each reader read the cases in at least 4 sessions, each session having 3 conditions. For the 3 conditions at the same session, the condition with higher noise level and/or smaller nodule size was read earlier. There were at least 3 days between sessions. The order of 100 lungs at each condition was randomized.

For each case, the reader marked the most suspicious location for the nodule, if there is any, and provided a score in a six-point scale (0: Definitely no nodule; 1: Very likely absent, but if there is a nodule, this is the most suspicious location; 2: Likely absent, but if there is a nodule, this is the most suspicious location; 3: Likely present, this is the most suspicious location; 4: Highly likely present, this is the most suspicious location; 5: Definitely present).

2.4 Data analysis

Both receiver operating characteristic (ROC) and localized ROC (LROC) analysis were performed using the non-parametric approach to quantify the observer performance for each of the 12 conditions at different dose levels, reconstruction methods, nodule types, and nodule sizes. Area under the ROC (AUC_ROC) and LROC curves (AUC_LROC) were calculated using a non-parametric approach, which was used as the figure of merit for the reader performance.

3. Results

Figure 2 shows examples with a GGN nodule inserted, but with varying nodule sizes and dose levels. Figure 3 displays the ROC and LROC curves for the 5.4 mm GGN nodule at 4 different dose levels (10%, 25%, 50%, and 100%) from one of the readers. Figure 4 shows the AUC_LROC as a function of dose levels for each of the 3 readers and all readers together for the 5.4 mm GGN nodule and the 3.4 mm SN nodule, respectively. The readings for one of the readers, varying nodule sizes and reconstructions are still ongoing.

Figure 2.

Example images with a GGN nodule inserted at the same location, but with 3 nodule sizes and 4 different dose levels.

Figure 3.

ROC and LROC curves for the 5.4 mm GGN nodule at 4 different dose levels (10%, 25%, 50%, and 100%). The results from one of the readers are shown here.

Figure 4.

AUC_LROC as a function of dose levels for each of the 3 readers and all readers together for the 5.4 mm GGN nodule (left) and the 3.4 mm SN nodule (right), respectively.

4. Conclusions

We presented a virtual clinical trial using an accurate projection-based lesion insertion tool in lung cancer screening CT. Radiologist performance on lung-nodule detection was efficiently determined in a well-controlled manner, as a function of radiation dose, nodule type, nodule size, and reconstruction methods.

To the best of our knowledge, this is the first virtual clinical trial utilizing an accurate projection-based nodule insertion method. Different from previous studies that used image-based nodule insertion tool, our simulation maintains the true appearance of lung nodules and enable flexible control of virtual clinical tasks. The reader performance data obtained in this study can be used as the basis to develop model observers for lung nodule detection, as well as a reference for dose and protocol optimization in lung cancer screening CT.