Computer-Aided Diagnosis of Lung Cancer and Pulmonary Embolism in Computed Tomography — A Review

Abstract

Computer-aided detection (CADe) and computer-aided diagnosis (CADx) have been important areas of research in the last two decades. Significant progresses have been made in the area of breast cancer detection, and CAD techniques are being developed in many other areas. Recent advances in multidetector row CT have made it an increasingly common modality for imaging of lung diseases. A thoracic examination using thin-section CT contains hundreds of images. Detection of lung cancer and pulmonary embolism on CT examinations are demanding tasks for radiologists because they have to search for abnormalities in a large number of images and the lesions can be subtle. If successfully developed, CAD can be a useful second opinion to radiologists in thoracic CT interpretation. In this review, we summarize the studies that have been reported in these areas, discuss some challenges in the development of CAD, and identify areas that deserve particular attention in future research.

Introduction

Computer-aided detection (CADe) and computer-aided diagnosis (CADx) have been important areas of research in the last two decades. Because of the high prevalence of breast cancer and challenges in interpretation of mammograms, most of the early work on CAD (CADe or CADx) was devoted to detection and characterization of masses and microcalcifications on mammograms. However, in the last decade, numerous studies on the development of CAD techniques have been reported for other diseases and imaging modalities. Lung cancer is the leading cause of cancer death in both men and women. The interpretation of thoracic computed tomography (CT) scans for lung nodules is a demanding task for radiologists, and the risks of false negative detection and benign nodules being recommended for biopsy are high. There is a potential for improvement if CADe and CADx are available for lung nodules in CT scans. A different, but related area is the detection of pulmonary embolism (PE) in CT pulmonary angiography (CTPA). PE is a common, potentially fatal condition in all age groups associated with significant morbidity and mortality in untreated patients, and radiologists may benefit from CADe because of the complexity of the pulmonary vascular structures and the large number of vessels to be inspected for PE in each case. In this review paper, we will focus our discussion on CADe and CADx of lung nodules and CADe of PE on CT examinations.

Challenges in lung cancer detection on CT examinations

In the United States, it is estimated that there will be 160,390 deaths from lung cancer and that 213,380 new cases will be diagnosed in 2007 (1). Lung cancer remains the leading cause of cancer deaths for men since the 1950's and for women since 1987. The overall prognosis of lung cancer is very poor. The 5-year survival rate is only about 16% for all stages combined (1). However, if detected and resected at its earliest stage (stage I), the 5-year survival rate can reach 70% (2–4).

Unfortunately, previous studies failed to show a significant reduction in mortality by screening with chest radiography (CXR) (5–8) despite the improvement in stage distribution, resectability, survival, and fatality in lung cancer. Lung cancer screening was therefore not recommended in clinical practice. Interest in screening was revived when CT was shown to have higher sensitivity in detecting small, early stage lung cancer than CXR. The Early Lung Cancer Action Project (ELCAP) investigated the usefulness of annual low dose CT screening for lung cancer in a high risk population and found that low dose CT can detect 4 times more malignant lung nodules than CXR, and 6 times more stage I malignant nodules which potentially are more curable (9). More recently, the International ELCAP (I-ELCAP) study showed that the 10-year survival of patients with stage I lung cancer detected on CT screening and who underwent surgical resection within 1 month reached 92%, and concluded that CT screening can detect lung cancer that is curable (10). However, another multi-center study found that, in comparison with the predictions from a model, there were a 3-fold increase in cancer detection and 10-fold increase in lung resection but no decline in diagnoses of advanced lung cancer or mortality rate (11). A 30-site randomized controlled study (National Lung Screening Trial (NLST)), sponsored by the National Cancer Institute (NCI), has enrolled about 50,000 participants to compare the effect of screening using helical CT or chest x-rays on the mortality rate of lung cancer patients. The results of the study will not be available until about 2010.

Although there is controversy over whether CT screening may reduce lung cancer mortality, there is consensus that CT allows the detection of more and smaller lung nodules than CXR. In the National Emphysema Treatment Trial (NETT), 25.6% of the 446 emphysema patients were found to have non-calcified nodules (12). In ELCAP, 23.3% of the patients were found to have non-calcified nodules by CT, which represented a 3-fold increase in sensitivity than CXRs (9, 12). This increase in sensitivity comes at the price of an increased workload for radiologists. A major potential difficulty in using helical CT for screening is the dramatic increase in the number of images that need to be interpreted for each case. Another potential difficulty is the additional resources that will be needed for clinical management of the expected screening detected nodules. Different criteria are being used by physicians to manage lung nodules in current clinical practice (9, 13). Many nodules are recommended to be followed up. However, the rate of the screening detected nodules being resected is still high at about 20 to 40% (14). A study of 426 patients who underwent video-assisted thoracoscopic surgery (15) indicated that 42.5% of these cases were benign. Keagy et al. (16) found that 40% of their patients with benign nodules were subjected to thoracotomy for presumed malignant disease. It is therefore important to establish, if possible, more reliable criteria to estimate the likelihood of malignancy of the lung nodules based on image information without resorting to invasive procedures, thereby reducing the potential patient morbidity and additional health care costs associated with lung cancer screening. FDG-enhanced PET scans (17) have been found to provide high sensitivity and good specificity for differentiating nodules as malignant and benign but the procedure will involve radioactivity, relatively high costs, and may not be available in many medical facilities. The I-ELCAP study (10) demonstrated that, with workup protocols that mainly used repeated CT scans to estimate nodule growth, the negative biopsy rate could be as low as 8% in the patient cohort. However, short-term follow up with repeated CT will further increase radiologists' workload.

Although CT has a much higher sensitivity than CXR, missed cancers are not uncommon in CT interpretation (18–21). The main causes for missed cancers include detection errors and characterization errors. Detection errors can be attributed to factors such as oversight or failure to detect the lesion among other structures. Characterization errors may be attributed to the difficulty in differentiating malignant lesions from benign nodules. The latter can also cause a radiologist to overestimate the likelihood of malignancy and recommend biopsy for benign lesions. Double reading may reduce missed diagnosis but it doubles the demand on radiologists' time. Some criteria have been suggested to estimate the likelihood of malignancy of solitary pulmonary nodules (13, 22–27). Computer-assisted classification of malignant and benign lung nodules has been attempted and promising results were reported (28–32). Gurney et al. (25, 33) used Bayesian analysis and an artificial neural network (34) to classify radiographic and clinical features and achieved a higher accuracy than subjective classification by radiologists. However, these computer classifiers used radiologist-identified image features, the extraction of which are both time consuming and subject to inter- and intra-observer variations. Subtle change in nodule volume, especially when the nodule is small, is difficult to discern visually on CT images.

CAD, in which a computer is trained to automatically detect or characterize the lesions of interest on the images, can be a viable approach to improving the accuracy of lung nodule detection and characterization on CT studies. A CADe system may provide a second opinion by alerting the radiologist to areas of concern, reducing the chance of overlook. CADx techniques have the potential to improve the specificity of cancer detection by estimating the likelihood of malignancy of a detected lesion, and predicting which cases are most suitable for a specific management option. In addition, CAD may reduce inter- and intra-observer variability in image interpretation. A number of studies have demonstrated the usefulness of applying CAD to interpretation of thoracic CT scans. Some of these studies will be reviewed below.

Computerized detection of lung nodules

Development of CADe systems for lung nodule detection on CXRs has a long history and is still on-going. Factors that limit radiologists' detection accuracy, namely, the overlapping ribs and the low contrast sensitivity of CXRs for subtle abnormalities, also affect the performance of computerized detection. However, CXR is the most commonly performed procedure in medical imaging and has the advantages of being low cost and low dose. A number of investigators have reported the performance of CADe systems demonstrating various levels of success (35–48). Although the false positive (FP) rates of CADe systems are generally higher than desired, retrospective observer performance studies indicated that CADe systems could significantly improve radiologists' accuracy in detection of lung nodules in CXRs (49–54). A commercial lung nodule CADe system for CXR was approved by FDA in 2001 but no large scale prospective clinical trial have been reported to date.

Development of CADe methods for the detection of lung nodules has been extended to CT as the potential efficacy of CT for lung cancer screening is being assessed. Although the specific computer vision techniques used in the different CADe systems differ, the overall scheme can generally be described in several major steps. First, the lung regions are isolated from other anatomical structures by segmentation of the CT images. Potential juxta-pleural nodules attached to the pleura are usually excluded during lung region segmentation and need to be recovered with boundary refinement techniques. Nodule detection is performed only within the lung regions and along their boundaries in the subsequent steps. Nodule enhancement preprocessing may be applied to the lung regions to enhance nodules and suppress other background structures. The lung regions are then prescreened for nodule candidates. Feature descriptors that can characterize the detected objects are extracted from each nodule candidate. Rule-based and/or other classifiers are trained to classify nodules and FPs based on the extracted features. Alternatively, neural network may be trained to differentiate nodules from other lung structures by recognizing the disease patterns without extracting individual features. The suspected nodules are then marked on the CT scan and displayed as output of the CAD system.

The performances of various CADe systems for nodule detection on CT scans are summarized in Table 1, which includes representative studies from different groups but is by no means exhaustive. The performance of a CADe system depends on a number of factors including, but not limited to, (1) the image acquisition and reconstruction parameters of the CT scans, (2) the size of the nodule, (3) the composition and location of the nodule, (4) the reference standard, (5) the data set size, and (6) whether the reported results are obtained from resubstitution (same data set used for training and testing), validation (optimizing system design using the validation results as a guide), or independent testing (test set not used until the system design is completed and fixed). Some of the information is included in the summary but the readers should refer to the original literature for more details.

Table 1.

Studies on the development of CADe systems for detection of lung nodules in thoracic CT examinations.

References	No. of cases (no. of exams)	Section thickness/interval	Nodule sizes	Total no. of nodules	Sensitivity	FP rate
Giger et al. 1994 (130)	8	10 mm	3–18 mm	2–21/case	94%	1.25 /case
Kanazawa et al. 1998 (131)	450 (Screening in Japan)	10 mm	> 4 mm?	230	90%	not reported
Armato et al. 2001 (132)	43	7–10 mm	3–24 mm mean=6.5 mm	171	70%	1.5/section
Ko et al. 2001(133)	8 (16 exams)	5 mm (8 scans) 10 mm (8 scans)	> 3mm	295	91%	2.3/case
Brown et al. 2001 (134)	17	5–10 mm	5–30 mm	36	86%	11 /case
Lee et al. 2001 (135)	20	10 mm	66 < 10 mm 23: 10–20 mm 9 > 20 mm	98	72%	1.1/section 30.8/case
Armato et al. 2002 (136)	31 (38 LD exams)	10 mm	5–25 mm mean=11.5 mm	50	80%	1.0/section, 28.3/case
Wormanns et al. 2002 (137)	85 (88 LD exams)	5 mm	5–16 mm	68	38%	5.8/case
Gurcan et al. 2002 (138)	34	5 mm	2–25 mm mean=8.9 mm	63	84%	1.74/slice
Armato et al. 2003 (73)	38	5 mm	3–29 mm mean=14.7 mm	82	Std filter 71% Lung filter 71%	0.5/section (26/case) 0.4/section
Suzuki et al. 2003 (139)	63 (LD)	10 mm	4–27 mm, mean=13.5 mm	71	80.3%	0.18/section, 4.8/case
Brown et al. 2003 (140)	15	0.5–1 mm	57 (≤3mm) 22 (3–10 mm)	79	100% >3 mm 70% ≤ 3 mm	15/case (2 cm of thorax)
Zhao et al. 2003 (141)	8 w/ simulated 1 w/ real	3.75 mm 7 mm	2–7 mm	266 simulated 4 real	84.2% 75%	5/case 9/case
Goo et al. 2003 (142)	50	7–8 mm	≥5 mm	26	65%	8/case
Paik et al. 2004 (143)	8	not reported	≥6 mm	not reported	90%	5.6/case
Lee et al. 2004 (144)	15	1–1.25 mm	All (2–30mm) Solitary >5 mm Juxtapleural	309	81% 93% 79% 71%	28.8/case
Arimura et al. 2004 (145)	106	10 mm	6–26 mm	131	81%	0.28/section
McCulloch et al. 2004 (146)	50 (LD)	2.5 mm	5.0–17.1 mm	35 solid 8 subsolid	70%	8.3/case
Awai et al. 2004 (57)	82	7.5 mm	3–30 mm mean=8.9 mm	78	80%	0.87/section 37.7/case
Ge et al. 2005 (147)	56 (82 exams)	1–2.5 mm	3–30.6 mm median=7.8 mm	116	80%	0.34/section
Bae et al. 2005 (148)	20	1 mm	3–27 mm	164	95.1%	6.9/case
Rubin et al. 2005 (62)	20	1.25mm/0.6 mm	≥3 mm mean=5.1 mm	195	65% 76%	3/case 10/case
Lin et al. 2005 (149)	29	10 mm	10–30 mm	393	89.3%	0.21/section
Kim et al. 2005 (150)	10	1 mm/1 mm 5 mm/1 mm 5 mm/5 mm	≥3 mm	126 121 114	95.2% 94.2% 88.6%	5.4/case 9.7/case 23.6/case
Li et al. 2005 (60)	38 (LD, missed cancers)	10 mm	6–26 mm mean=12 mm	10 GGO 16 mixed 12 solid	87%	3/case
Armato et al. 2005 (151)	393 (LD)	10 mm	3–30 mm	470 69 (malig)	70% 83% (malig) 80% (retrained malig)	1.6 /section 1.6/section 0.85/section (mean=28 sections/case)
Kim et al. 2005 (129)	14	1–5 mm	5–28 mm, mean=15 mm	29 GGO	89.7%	0.89/section
Brown et al. 2005 (61)	8 (LD)	1.25 mm	2.5–12.8 mm mean=5.2 mm	22	86.4%	2.64 FPs/case (40 sections/case, 5 cm of thorax)
Marten et al. 2005 (152) (ICAD)	20	0.75 mm/0.6 mm	1–29.6 mm median=4.4 mm	135	76.3%	0.55/case
Marten et al. 2005 (59) (ICAD)	20	0.75/0.6 2.0/1.2 4.0/2.7	61: < 4 mm 58: 4–9.9 mm 15: > 9.9 mm	135*	73.9% 59.0% 4.4%	0.79/case 1.05/case 4.47/case
Peldschus et al. 2005 (153) (R2 system)	100	1.25–3 mm	not reported	160	Reference radiologists checked CADe marks only	1.25/case
Roy et al. 2006 (154)	38	5 mm	3–30 mm mean=14.7 mm	82	70%	0.28/section or 0.03/section
Boroczky et al. 2006 (155)	25 (38 scans)	1.3 mm	>3.5 mm	52	100% (no loss of TP in FP classification)	−56.4% (FP reduction)
Sahiner et al. 2006 (156)	27	1–2.5 mm	3–12.9 mm mean=5.3 mm	33 (internal) 31 (juxtapleural)	80%	3.9/case 9.7/case Total 13.6/case
Sahiner et al. 2007 (157)	48 (30 pos, 18 neg)	1.5–3 mm	3–36.3 mm median=5.5 mm	70	79%	4.9/case
Wang et al.:2007 (158)	12	3 mm	4–20 mm	47	100%	1.75/case
Sahiner et al. 2007 (65)	85 (52 pos, 33 neg)	1.5–3 mm	3.1–19.6 mm median=5.6 mm	118	78%	5.5/case

LD = low dose CT, GGO=ground-glass opacity

R2= CADe system by R2 Technologies, ICAD=CADe system by Siemens Medical Solutions.

^*Subgroup numbers add up to 134 while a total of 135 was given in the paper.

As can be seen from Table 1, the image acquisition and reconstruction parameters of the data sets used in the different studies vary over a wide range. Many studies, especially the early ones, used thick-slice CT scans that would limit the sensitivity for detection of small nodules. More recent studies tended to use thin-section reconstruction obtained from multidetector row CT scans with some having sub-millimeter slice intervals. Several studies collected the cases from lung cancer screening using low-dose CT, while others used higher dose diagnostic CT scans that may offer higher signal-to-noise ratios and thus better detectability for small nodules. Most of the studies included lesions with diameters ranging from 3 mm to 30 mm, following the size range that radiologists consider to be clinically significant. However, some studies only considered nodules greater than 4 mm, 5 mm, or as large as 10 mm, and others included nodules smaller than 3 mm. Some researchers estimated the size as the longest dimension of a nodule measured on the CT sections and others defined the size as the average of the two dimensions of a bounding box enclosing the nodule on the CT section where it had the largest area. Nodule characteristics have a strong influence on the detection accuracy. Many studies included only noncalcified solid nodules and some might include small fractions of ground-glass, mixed, or calcified nodules. Ground-glass nodules are more difficult to detect than solid nodules for both radiologists and CADe systems because of their low contrast relative to the background. Juxta-pleural nodules are usually more difficult for the computer to detect than internal nodules; the proportion of the two types varied in the different data sets used in the studies. Because the presence of most lung nodules was not biopsy-proven, the reference standards for the majority of the cases were established by one or several radiologists. The “truth” relative to which the CADe performance was assessed thus depended on the number and the experiences of the radiologists who provided the reference standards, the consensus process, and the criteria used for truth. In the majority of the studies the data set was small and the training and evaluation of the CADe system was performed on the same data set. Even if a resampling method such as leave-one-out was used for estimation of the test performance, the reported performance would likely be optimistically biased because the CADe system might have been designed using the test result on the available data set as a guide. The techniques and parameters of the CADe system would be tuned to the characteristics of the small data set used and thus may not be generalizable to those of the patient population at large. The degree of bias would depend on the training sample size, with smaller sample leading to greater bias (55). Because of the many factors that may affect the CADe performance and because the information about the data set and the training and evaluation methods may not have been described thoroughly in the papers, the performances of the different CADe systems generally cannot be directly compared.

One of the most challenging processes in the development of CADe or CADx systems is the collection of a sufficiently large data set with ground truth or reference standard which encompasses case samples representative of the patient population. The NCI recognized the need of CAD for lung CT interpretation and supported the Lung Imaging Database Consortium (LIDC) to collect a standard database of lung CT images for this purpose (56). The availability of this standard database may alleviate some of the problems in comparing different CADe systems although it still depends on how the researchers use the LIDC public data set, namely, whether they use the data set for both training and testing, design the methods and parameters based on the test results on the data set, or reserve the data set for true independent testing.

Effect of CADe on radiologists' detection of lung nodules

The usefulness of a CADe system depends on whether it can improve radiologists' detection of significant lung nodules in clinical practice. Despite the fact that two commercial CADe system for lung nodule detection in CT examinations have been approved by FDA since 2004, no large scale prospective clinical trial of CADe for evaluation of the utility of CADe systems in routine clinical practice has been reported to date. A number of retrospective observer performance studies have been conducted as summarized in Table 2. Radiologists' detection of lung nodules on CT scans is affected by the same factors as those that influence computerized detection, in addition to other clinical considerations in the decision making process that do not exist in laboratory observer studies. Therefore, the absolute performances of the radiologists without and with CADe in these studies with a limited data set may not be generalized to their performance in clinical settings. However, observer studies were designed to evaluate the relative changes in the radiologists' detection performance when reading with CADe. The impact of some of these factors on the relative performances will likely be smaller, although the extent of improvement will certainly depend on the characteristics of the lesions in the data set and whether the CADe system detects lesions that are complementary to those detected by the radiologist.

Table 2.

Observer performance studies for evaluation of the effects of CADe on radiologists' detection of lung nodules in thoracic CT examinations.

References	No. of exams	Section thickness/interval	Nodule sizes	Total no. of nodules	No. of observers	Reading without CAD	Reading with CAD	P value
Awai et al. 2004 (57)	50 (36 pos, 14 neg)	7.5 mm	3–29 mm mean=8.1 45 (3–10 mm)–11 (11–29 mm)	56	5 Rad–5 Resid	AFROC A1: All 0.64 Rad: 0.63 Resid: 0.66	AFROC A1: All 0.67 Rad: 0.66 Resid: 0.68	P<0.01 P<0.01 P=0.02
Marten et al. 2004 (58) ICAD	18	0.75/0.6 mm	49: <4 mm 43: 4–9.9 mm 4: >9.9 mm	96 (89 solid, 2 mixed, 5 calcified)	4 Rad, 2 read with CAD	ROC Az: Rad1: 0.71 Rad3: 0.49	ROC Az Rad1: 0.93 Rad3: 0.79	P<0.05 P<0.05
Marten et al. 2005 (59) (ICAD)	20	0.75/0.6 mm 2.0/1.2 mm 4.0/2.7 mm	61: <4 mm 58: 4–9.9 mm 15: > 9.9 mm	135*	2	Az, Sens, FPs/case CAD: 0.80, 74%, 0.79 Rad1: 0.70, 53%, 0.26 Rad2: 0.70, 53%, 0.21	Az, Sens, FPs/case Rad1: 0.93, 93%, 0.11 Rad2: 0.95,91%, 0.05	P<0.05 P<0.05
Li et al. 2005 (60)	27 (17 pos, 10 neg) (LD)	10 mm	6–17 mm mean=10 mm	18 (6 GGO 10 mixed 1 solid)	All 14 Rad 6 (multiformat) 8 (cine)	Az, Sens 0.763, 52% 0.757, 49% 0.768, 54%	Az, Sens 0.854, 68% 0.862, 71% 0.848, 67%	P=0.002,0.001 P=0.04, 0.02 P=0.01, 0.006
Brown et al. 2005 (61)	8 (6 pos, 2 neg) (LD) (5 cm of thorax)	1.25 mm	2.5–12.8 mm mean=5.2 mm	22	39 chest Rad 95 nonthoracic Rad 68 non-Rad	13 readers: Sens, FPs/case 64%, 0.144 JAFROC FOM 0.78	13 readers: Sens, FPs/case 81.9%, 0.173 JAFROC FOM 0.84	P<0.01, <0.01 P>0.05
Rubin et al. 2005 (62) (Simulation)	20 (19 pos, 1 neg)	1.25/0.6 mm	≥3 mm mean=5.1 mm	195	3	Sens, FPs/case CAD: 65%, 3 CAD: 76%, 10 Sens: 63% (2 Rads) 50% (indiv Rads)	(Simulation at 65% CAD thresh) Sens: 76%	P<0.05
Das et al. 2006 (63) (R2, NEV)	25 (23 pos, 2 neg)	12 scans: 2/1.5 mm 13 scans: 1.0/0.5 mm	mean 3.4 mm 89 < 5 mm 27 ≥ 5 mm	116	3	Sens: Rad1: 68% Rad2: 78% Rad3: 82% Sens, FPs/case R2: 73%, 6 NEV: 75%, 8	Sens: R2, NEV Rad1: 79%, 79% Rad2: 90%, 90% Rad3: 84%, 86% (FP rate not reported)	P=0.005, 0.116 P=0.081, 0.032 P=0.123, 0.161
Yuan et al. 2006 (64) (Simulation)	150 (134 pos, 16 neg)	1.25 mm (CAD) 2.5 mm (Rad)	291: <4 mm 310: 4–10 mm 27: >10 mm	628	1	Sens: Rad: 83% R2:73% @ 3.19 FPs/case	Predicted sens increase 21.2%	--
Sahiner et al. 2007 (157)	48 (30 pos, 18 neg)	1.5–3 mm	3–36.3 mm median=5.5 mm	70	4	Sens, FPs/case CAD: 79%, 4.9 Rad: 78%, 0.42 JAFROC FOM: 0.83	Sens, FPs/case Rad: 86%, 0.46 JAFROC FOM: 0.88	P=0.03
Sahiner et al. 2007 (65)	85 (52 pos,33 neg)	1.5–3 mm	3.1–19.6 mm median=5.6 mm	118	6	Sens, FPs/case CAD: 78%, 5.5 Rads: 77.1%, 0.55 JAFROC FOM: Rads: 0.806	Sens, FPs/case Rads: 84.9%, 0.61 JAFROC FOM: Rads: 0.843	P=0.003, 0.04 P=0.007

LD=low dose CT, pos=positive, neg=negative, Rad=radiologists, Resid=residents,

R2= CADe system by R2 Technologies, NEV=CADe system by Siemens Medical Solutions, ICAD=CADe system by Siemens Medical Solutions.

^*Subgroup numbers add up to 134 while a total of 135 was given in the paper.

Awai et al. (57) conducted an observer study in which 50 CT examinations were read by 5 board-certified radiologists and 5 residents. They found that the detection performance in terms of the area under the alternative free response receiver operating characteristic (AFROC) curves was significantly improved for either group of readers. Marten et al. (58) conducted an observer study to evaluate the performance of an ICAD system (Siemens Corporate Research) using 18 thin-section CT cases. Two of four participating radiologists read without and then with CADe. Both readers demonstrated statistically significant improvement in the area under the ROC curve (A_z). However, it is not clear how the ROC analysis was performed in cases with multiple nodules, and how much uncertainty would be introduced by an ROC analysis using a 3-point confidence rating scale. The same group (59) also compared the detection performances of the CADe system by having two radiologists read without and with CADe for 20 cases reconstructed at 3 different section thicknesses. They found that the section thickness of the CT data had much stronger effect on the performance of their CADe system than on that of the radiologists. CADe improved the radiologists' performance significantly for the 0.75 mm and 2-mm section thickness but had only marginal influence at 4-mm section thickness. Li et al. (60) evaluated the effects of CADe on radiologists' detection of peripheral lung cancers missed in clinical practice. Using two different display formats (multiformat and cine), they found that CADe could improve radiologists' detection sensitivity and A_z regardless of display format. Brown et al. (61) collected observer data from 202 participants at an RSNA annual meeting. The readers read a data set of 8 cases without and then with their CADe system and provided confidence ratings for nodule detection. They found that there were statistically significant increases in nodule detection and FP rates for all types of observers. For the 13 readers who finished all 8 cases, the sensitivity and the false positive (FP) rates increased significantly; their average figure-of-merit (FOM) from jackknife FROC (JAFROC) analysis also increased but did not achieve statistical significance. Rubin et al. (62) reported a simulated observer study with 20 CT scans and 3 radiologists. The radiologists and the CADe system performed the detection independently. The authors simulated a radiologist's reading with CADe by assuming an ideal situation in which the radiologist would accept all true positive marks by the CADe but reject all FP marks. The average sensitivity of the radiologists would then increase significantly from 50% to 76% at a CADe system performance of 65% sensitivity and 3 FPs/case. In a study by Das et al. (63), the nodule detection sensitivities of 3 radiologists reading 25 CT scans without and with CADe were compared. Two commercial CADe systems were used, one with a sensitivity of 73% at 6 FPs/case and the other with a sensitivity of 75% at 8 FPs/case for this data set. The results showed that the sensitivities of all three radiologists increased although the increase was significant for only one of the three radiologists using either CADe system. The effects of CADe on the readers' FP rates were not discussed. Yuan et al. (64) compared the detection accuracy of a commercial CADe system with one radiologist in 150 CT examinations. The radiologist detected 83% of the nodules while the CADe system had a sensitivity of 73% at 3.19 FPs/case. The radiologist had higher sensitivity in detecting peripheral and juxta-pleural nodules but the CADe system was more sensitive to hilar and central nodules. The authors predicted an increase in the radiologist's sensitivity by 21.2% because of the complementary detection of nodules in different regions of the lungs but no prediction on the FP rate was made. Sahiner et al. (65) conducted an observer study with 6 radiologists and 85 CT examinations and analyzed the detection performance with JAFROC methodology. They found that the average FOM of the radiologists improved with CADe for nodules greater than 3, 4, 5, or 6 mm in diameter, although the increase was significant only for the size thresholds of 3 mm and 4 mm. The average sensitivity and FP rate at the size threshold of 3 mm for the radiologists also increased significantly by 10% and 11%, respectively.

The retrospective observer studies therefore demonstrated that even experienced radiologists will overlook some lung nodules in CT scans, and that CADe can significantly reduce the false negatives. However, the data sets used in these studies to date are relatively small. The one study that included over 100 cases only used one radiologist, and the improvement in sensitivity was a prediction rather than true measurement of observer performance change. How CADe will influence radiologists' interpretation in clinical practice is still unknown. The controversy on the significance of early detection of small lung nodules in the management, long term survival and mortality rates of lung cancer patients will also play a role in the consideration of implementing CADe for clinical use.

Computerized characterization of lung nodules

A number of studies have been reported on the development of CADx systems for characterization of malignant and benign lung nodules on CT scans as summarized in Table 3. Henschke et al. (66) trained a neural network to differentiate malignant and benign nodules by recognizing the feature patterns extracted from the images. The neural network correctly classified all 14 malignant nodules and 11 of the 14 benign nodules. Kawata et al. (67) extracted surface curvatures and ridge lines as features to describe 47 malignant and 15 benign nodules from 62 cases. Good separation was demonstrated between malignant and benign classes using pairs of features but no classification result was reported. The same group (68) also investigated the feasibility of developing an image-guided decision support system that would retrieve from a reference database nodule images having morphological and internal features consistent with those of the query nodule. The performance of the system was not known because they only showed image retrieval result of one query nodule. McNitt-Gray et al. (69, 70) designed a linear discriminant classifier in two studies using 31 and 32 nodules, respectively. They found that texture features derived from the spatial gray level dependence matrices could provide high classification accuracy ranging from 90.3% to 100%, sensitivity ranging from 88.2% to 100% and specificity ranging from 92.3% to 100%, depending on the number of texture features used. They cautioned that the results might be overly optimistic because of the small data sets available. Matsuki et al. (71) trained a back-propagation neural network using 99 malignant and 56 benign nodules. The input to the neural network included 7 clinical parameters and 16 radiologic findings rated by experienced radiologists. The neural network achieved an A_z value of 0.951. Lo et al. (72) performed 3D segmentation and extracted features that described the shape, size, texture, and vascularity of the lung nodules. A neural network trained with 24 malignant and 24 benign nodules achieved an A_z of 0.89. Armato et al. (73) evaluated a serial approach in which automated nodule detection was followed by automated nodule classification using a low-dose CT data set from a lung cancer screening program in Japan that contained 401 benign nodules and 69 malignant nodules. The nodule candidates at the output of the automated detection program included 335 of the nodules (59 malignant, 276 benign) among other FPs, which were then input to a classifier to differentiate malignant nodules from the other objects. The classifier achieved an A_z of 0.79 regardless of whether the FPs were manually separated from the nodules before a leave-one-out evaluation. This serial approach of detection followed by classification represents one potential implementation of fully automated analysis of CT scans for lung cancer. Aoyama et al. (74) developed a lung nodule classification scheme using a low dose CT data set containing 76 cancers and 413 benign nodules from the same source as the study of Armato et al. (73). With a 10 mm slice thickness, the nodules were covered by only 1 to 3 slices. They designed a classifier using features extracted from the individual slices and then estimated the likelihood of malignancy of a nodule on the slice with the largest nodule cross section or by merging the information from multiple slices. The best performance with an A_z of 0.846 was obtained from the latter approach, which was higher than the average A_z of 0.7 from 17 radiologists reading a subset of the nodules. Using the same data set, Suzuki et al. (75) trained a multi-massive training artificial neural network (multi-MTANN) to classify the malignant and benign lung nodules. With six MTANN in parallel and an integration ANN to merge the outputs of the six MTANNs, they achieved an A_z of 0.882 and 100% sensitivity for identifying the malignant nodules at a specificity of 48%. The same group (76, 77) further developed their computerized classification scheme that used a linear discriminant classifier (74) to analyze nodules on thin-section CT images. The scheme achieved an overall A_z of 0.937 in a data set of 61 malignant and 183 benign nodules that included GGO, mixed, and solid nodules. The classification accuracy for the three types of nodules was 0.919, 0.852, and 0.957, respectively. Shah et al. (78) investigated computerized classification of malignant and benign lung nodules on thin-section CT images using a data set of 48 malignant and 33 benign nodules. The features of a nodule were extracted from its contour manually outlined on a single representative slice by a thoracic radiologist. They compared linear discriminant analysis, quadratic discriminant analysis, a logistic regression classifier, and a decision tree classifier and reported that the four classifiers achieved A_z of 0.92, 0.87, 0.88, and 0.68, respectively. Using a different data set with 33 malignant and 21 benign nodules, Shah et al. (79) trained a decision tree classifier with image features and obtained a sensitivity of 91% and a specificity of 67%. The same group (80) investigated the utility of a computerized classification system designed by using features extracted from volumetric thin-section CT image data acquired before and after the injection of contrast media. The nodules were segmented from the CT volume using a semi-automatic method. Features were extracted from the image data and derived as the difference in the attenuation features between the post-contrast volume at the maximum enhancement and the pre-contrast volume. They compared three classifiers in different feature spaces and obtained A_z values ranging from 0.69 and 0.92. The best performance (0.92) was achieved by a logistic regression classifier using feature descriptors of the solid component of the nodules. Mori et al. (81) analyzed thin-section CT scans obtained at 3 time points: before, 2 minutes after, and 4 minutes after contrast enhancement. They extracted three features describing the shape and attenuation of the nodules segmented from the volumetric CT data and designed a linear discriminant classifier using the three features at each time point. In a data set of 35 malignant and 27 benign nodules, they achieved an A_z of 0.91, 0.99 and 1.0, respectively, at the three time points. Awai et al. (82) trained an ANN to differentiate malignant and benign nodules using shape and density features from 34 nodules. The ANN achieved an A_z of 0.795 for a test data set of 18 malignant and 15 benign nodules. Way et al. (83) developed a lung nodule classification system using morphological and texture features extracted from nodules segmented by an automated 3D active contour model. The trained linear discriminant classifier achieved an A_z of 0.83 in a data set of 44 malignant and 52 benign nodules. In a later study (84), the performance of the CADx system was increased to 0.86 with an improved feature space and an enlarged data set of 124 malignant and 132 benign nodules. Hadjiiski et al. (85) recently incorporated interval change information obtained from serial CT examinations into the feature space for classification of lung nodules. In a data set of 103 temporal pairs of 39 malignant and 64 benign nodules, a linear discriminant classifier achieved an A_z of 0.85, which is higher than that of 0.78 using the features extracted from the current CT scan alone.

Table 3.

Studies on the development of CADx systems for characterization of malignant and benign lung nodules on thoracic CT examinations.

References	No. of exams	Section thickness/interval	Nodule sizes	Total no. of nodules	Malig	Benign	Az	Accuracy	Sensitivity	Specificity
Henschke et al. 1997 (66)	not reported	5 mm	<30 mm	28	14	14	~0.79 (from graph)	89% (25/28)	100%	79%
Kawata et al. 1998 (67)	62	1 mm	6–25 mm	62	47	15	not reported
McNitt-Gray et al. 1999 (69)	31	≤3 mm	5–30 mm	31	17	14	not reported	90.3% (28/31)	88.2%	92.9%
McNitt-Gray et al. 1999 (70)	32	≤3 mm	5–30 mm mean=16.9	32	19	13	0.992–1.0	90.6%–100%	89.5%–100%	92.3%–100%
Matsuki et al. 2002 (71)	155	2 mm	<30 mm	155	99	56	0.951
Lo et al. 2003 (72)	48	Thinsection CT	not reported	48	24	24	0.89
Armato et al. 2003 (73)	393 (LD)	10 mm	3–30 mm mean=8.9	470 335	69 59	401 276	0.79
Aoyama et al. 2003 (74)	415 (LD)	10 mm	<30 mm	489	76	413	0.846
Kawata et al. 2004 (68)	174	0.5 mm	not reported	174	98	76	not reported
Li et al. 2004 (76, 77)	228	1 mm	3–20 mm	244	61	183	0.937
Suzuki et al. 2005 (75)	415 (LD)	10 mm	<30 mm	489	76	413	0.882		100%	48%
Shah et al. 2005 (78)	81	≤3 mm	6–57 mm mean=22	81	48	33	0.92
Shah et al. 2005 (79)	54	not reported	6–54 mm mean=24	54	33	21		81%	91%	67%
Shah et al.. 2005 (80)	35 pre-, post- contrast enhanced	≤3 mm	6–54 mm mean=25	35	19	16	0.69–0.92
Mori et al. 2005 (81)	62 pre-, post- contrast enhanced	2 mm	5–25 mm mean=14	62	35	27	0 min:0.91 2 min: 0.99 4 min: 1.0	85% 92% 100%	94% 100% 100%	74% 89% 100%
Awai et al. 2006 (82)	33	1–1.25 mm	<30 mm	33	18	15	0.795	73%	72%	75%
Way et al. 2006 (83)	58	1.25–5 mm	3.9–59.8 mm mean=17.3	96	44	52	0.83
Way et al. 2007 (84)	152	1–7.5 mm	3–36 mm	256	124	132	0.86
Hadjiiski et al. 2007 (85)	43	0.625–3 mm	2–30 mm	103	39	64	0.85

LD=low dose CT

Effect of CADx on radiologists' characterization of lung nodules

Several studies have been conducted to evaluate the effects of CADx on radiologists' accuracy for characterization of malignant and benign lung nodules; they are summarized in Table 4. Matsuki et al. (71) evaluated the usefulness of a trained ANN (Table 3) for assisting radiologists in differentiating malignant and benign nodules. Three groups of observers including 4 attending radiologists, 4 radiologist fellows, and 4 radiology residents participated in the study. The performance of each of the three groups improved significantly, and the average A_z of all 12 readers increased significantly from 0.831 to 0.959. Li et al. (76, 77) conducted an observer study using 28 malignant and 28 benign nodules. A trained linear discriminant classifier that could distinguish the malignant from the benign nodules with an A_z of 0.831 was used as an aid. They found that the performance of every reader (7 thoracic and 9 other radiologists) increased with CADx. The average A_z improved significantly from 0.785 to 0.853. The radiologists' recommendations were changed by use of CADx in 18% of the readings, of which 68% would have a beneficial effect. In addition, 69% of the changed recommendations regarding biopsy would have a beneficial effect. Shah et al. (79) conducted a study to evaluate the classification accuracy for 15 malignant and 13 benign nodules by 8 radiologists using image data alone, with additional clinical data, and then with CAD output. The CADx system used image features as input to a decision tree classifier. The system achieved a sensitivity of 91% at a specificity of 67% (Table 3). The A_z value of each of the 8 readers (2 thoracic radiologists, 2 general radiologists, 1 thoracic radiology fellow, and 3 radiology residents) increased with CADx. The average A_z for all readers increased significantly from 0.75 (image data and clinical data) to 0.81 with the use of CADx output. Awai et al. (82) trained an ANN for classification of lung nodules (Table 3) and evaluated its impact on radiologists' performance in an observer study using a data set of 18 malignant and 15 benign nodules. The average A_z of 19 readers (10 body imaging radiologists and 9 residents) and that of the group of residents increased significantly whereas the increase in the average A_z for the group of radiologists did not achieve statistical significance. Way et al. (84) developed a CADx system for automated segmentation and classification of lung nodules (Table 3). The CADx system was used in an observer study to compare radiologists' performance without and with CADx. Six thoracic radiologists read a data set of 124 malignant and 132 benign nodules from 152 patients. The average A_z of the six radiologists was found to improve significantly with the use of CADx.

Table 4.

Observer performance studies for evaluation of the effects of CADx for characterization of malignant and benign lung nodules in thoracic CT examinations.

References	No. of exams	Section thickness/interval	Nodule sizes	Total no. of nodules	Malig	Benign	Az CAD system	No. of observers	Az without CAD	Az with CAD	P-value
Matsuki et al. 2002 (71)	50	2 mm	<30 mm	50	25	25	0.951	12: 4 Rad 4 fellows 4 Resid	0.831 0.933 0.821 0.759	0.959 0.984 0.932 0.961	<0.001 <0.001 <0.001 <0.001
Li et al. 2004 (76) Li et al. 2006 (77)	56	1 mm	6–20 mm	56	28	28	0.831	16 7 thoracic Rad 9 other Rad	0.785	0.853	0.016
Shah et al. 2005 (79)	28	not reported	6–54 mm mean=24	28	15	13	Sens 91% Spec 67%	8 (2 thoracic, 2 general, 1 thoracic fellow, 3 Resid)	0.75	0.81	0.02
Awai et al. 2006 (82)	33	1–1.25 mm	<30 mm	33	18	15	0.795	19: 10 Rad 9 Resid	0.843 0.910 0.768	0.924 0.944 0.901	0.021 0.19 0.009
Way et al. 2007 (84)	152	1–7.5 mm	3–36 mm	256	124	132	0.86	6 thoracic Rad	0.82	0.84	<0.01

Rad=radiologists, Resid=residents

These studies demonstrated the potential of CADx for assisting radiologists in making diagnostic decision for lung nodules in CT examinations. However, the data sets used in these studies were small. The characteristics of the nodules in these data sets would likely be different from case samples randomly drawn from patient population. There are also considerations that may affect radiologists' diagnostic decisions in clinical practice that do not play a role in observer studies. How radiologists may respond to the CADx system output in clinical settings cannot be easily predicted from the results of retrospective studies. CADx will have to be evaluated in prospective clinical trials in order to assess the impact of the computerized classification on biopsy recommendations.

Challenges in pulmonary embolism detection on CTPA examinations

PE is a common and potentially fatal condition associated with significant morbidity and mortality in untreated patients. Prompt and accurate diagnosis of PE has been shown to greatly influence patient outcome (86, 87). CTPA has been reported to be an effective means for clinical diagnosis of PE (88–95). CT has advantages over conventional pulmonary angiography and ventilation/perfusion (V/Q) scan because of its direct imaging of the blood clot, better interobserver agreement, greater accuracy, and possibility to explain patient's sign and symptoms (88–90, 96). The main limitations of single-detector spiral CT has been the detection of small peripheral emboli (97–101) and the isolated subsegmental emboli (91). The main reason for inadequate detection of pulmonary emboli (PEi) in these small vessels is partial volume effects and cardiac and respiratory motions (92). Although the clinical significance of small PEi has not been established, small PEi may produce significant morbidity in patients with underlying cardiorespiratory disease (95), and may indicate a risk for recurrence of more significant emboli among stable patients. Studies (102–104) also indicated that the presence of peripheral PEi may be an indicator for current deep vein thrombosis thus potentially heralding more severe embolic events. In addition, it is important to estimate the total burden of pulmonary vascular clots in patients with acute PE to determine proper therapy and to improve patient outcome (105–110).

The advent of MDCT offers the possibility of detecting subtle PEi in subsegmental arteries (92, 94, 100, 101, 111, 112). The improved visibility results in substantially higher detection rates for subsegmental PEi, especially for obliquely oriented vessels, and better agreement among readers (92–94). However, a thin-section MDCT study of PE routinely produces 500–600 transverse images to cover the chest (101). Radiologists have to visually track the vessels down to the 6^th-order branches of subsegmental pulmonary arteries to search for PEi. False negatives (FN, missed diagnosis) are not uncommon because of the complexity of the images and the large number of vessels to be tracked in each case. As shown in the latest results of the PIOPED II study (96, 113), even with the use of MDCT, the sensitivity was moderate (83% at a specificity of 96%), suggesting that CTPA may not be sufficient as a stand-alone procedure for PE screening. With CT venography (CTV) added to CTPA, the sensitivity increased to 90% with a specificity of 95%. A combination of CTPA and CTV may be more promising for PE screening (96) but it increases costs and radiation risk.

CADe may be a viable approach for assisting radiologists in this demanding task and reducing the chance of missing PEi (101, 114). With advanced computer vision techniques, the computer may be trained to automatically track the pulmonary vessels, distinguish the arteries from the veins, detect suspicious PE locations by searching along the arteries, and finally alert the radiologists to the regions of interest for suspicious PEi. If CADe can improve the sensitivity and specificity for the detection of small peripheral emboli with CTPA, it may reduce unnecessary workup with other diagnostic procedures or provide more accurate information for selecting treatment options.

Computerized detection of pulmonary embolism

Automated detection of PE on CT images is a challenging area of computer vision application. PE detection is more difficult than lung nodule detection because of the vast network of pulmonary arteries in the lungs and their variable sizes. This area has not attracted the interest of the CAD community until recently. The few studies that have been performed so far are very preliminary. Masutani et al. (115) developed a computerized method for PE detection based on volumetric image analysis. They selected 19 (11 positive and 8 normal) cases from 30 clinical cases, excluding the cases for which the definition of truth or “gold standard of detection” was difficult. One radiologist marked 21 thrombi with volume greater than 10 mm³ in the 11 positive cases. The system could detect 100% and 85% of the 21 thrombi with 7.7 and 2.6 FPs/case, respectively, when the PE volume was between 16 mm and 64 mm³. Of the 143 FPs for all cases, 92% were related to soft tissues such as lymphoid tissue surrounding vessels. They did not describe the characteristics of the thrombi, such as the percentage of occlusion by the PEi, and how the PEi distributed in segmental and subsegmental arteries, which would reveal the degree of subtlety of the PEi in the study. With only 21 PE samples, the data set would be too small to represent the large varieties of PEi that may be encountered in clinical images.

Das et al. (116) evaluated the performance of a commercial system for PE detection on CTPA scans using a data set that contained 33 cases with 186 segmental and 120 subsegmental PEi. The system achieved a sensitivity of 88% for segmental and 78% for subsegmental PEi with 4 FPs/case. In a study by Digumarthy et al. (117) using the same commercial system, 39 consecutive patients with high clinical suspicion were included with criteria of good contrast opacification, absence of significant motion artifacts and pulmonary disease. The reference standard included 270 PEi in arteries greater than 4 mm in diameter. The CADe system detected 92% of the PEi at an average FP rate of 2.8 per case. Jeudy et al. (118) also evaluated the same commercial system using a data set of 22 cases. A total of 251 PEi were identified as reference standard, including 188 in the segmental and 63 in the subsegmental arteries. They reported a sensitivity of 80% for the segmental PEi and 76% for the subsegmental PEi at an FP rate of 1.8 per case. Schoepf et al. (119) conducted a similar study to evaluate the same commercial system using CTPA exams of 36 patients. Consensus reading by two radiologists, with a third for adjudication, identified 130 segmental PEi and 107 subsegmental PEi in 23 patients while the other 13 patients were found to be negative for PE. The system detected 92% of the segmental PEi and 90% of the subsegmental PEi at an FP rate of 4.8 per case. Maizlin et al. (120) evaluated the same commercial system for detection of PE in 104 CTPA cases. Clinical reading and two radiologists identified 45 PEi in 15 of the patients. The CADe system detected 18 central and segmental PEi and 8 subsegmental PEi in 8 patients but missed 14 proximal and 5 subsegmental PEi in 7 patients, corresponding to an overall sensitivity of 57.8% at an FP rate of 0.93 per case.

Das et al. (121) evaluated a different CADe system developed by another commercial company using a data set of 45 cases. Twenty nine cases were found to have a total of 213 PEi in all vessel levels. The CADe system detected 82% of the PEi at a median FP rate of 3 per case. Buhmann et al. (122) also reported the performance of this commercial CADe system for PE detection on CTPA scans of 40 patients. An expert panel of two radiologists identified 212 PEi in 18 patients, of which 65 were centrally located (i.e., in the pulmonary truncus, main, lobar, and first-order segmental arteries), and 147 were peripherally located (i.e., in the higher-order segmental and subsegmental arteries). The CTPA scans were considered to have good image quality and only 5 of the scans had respiratory motion artifacts. The CADe system detected 74% of the central and 82% of the peripheral PEi at an average FP rate of 3.85 per case. Engelke et al. (123) assessed the performance of the same commercial system and its effects as a second reader on the detection of PE by two experienced and two inexperienced chest radiologists. A total of 1116 PEi (72 mediastinal, 133 lobar, 465 segmental, and 455 subsegmental. Note that the sum of subgroups equals 1125, which is different from the total of 1116 given in the paper) were identified in the CTPA scans of 56 patients by the consensus of two independent experienced radiologists. The CADe system had an overall sensitivity of 30.7% at an FP rate of 4.1 per patient whereas the sensitivities of the radiologists without CADe ranged from 77% to 93%. Despite the low sensitivity of the CADe system, the sensitivities of radiologists with CADe increased to a range of 83% to 96%. The overall performance in terms of A_z of individual radiologists also increased with CADe although only the improvement for the inexperienced radiologists achieved statistical significance (p=0.041).

Zhou et al. (124) developed an automated vessel segmentation and PE detection system for CTPA images (124–128) and conducted several studies to evaluate the performance of the system. The reference standards were provided by thoracic radiologists who identified PE locations, estimated the percent diameter occlusion, and rated the conspicuity of each embolus. If a contiguous PE volume occluded more than one branch of arteries, the radiologist virtually split the PE volume according to the branching of the artery by marking the PE segment in each branch as a separate PE. In a preliminary study (124), they used a data set of 14 cases, 8 of which had extensive lung parenchymal or pleural disease. A total of 163 PEi were identified in the data set including 94 PEi that were located in arteries proximal to the subsegmental level and 69 subsegmental PEi. The results showed that, for the PEi that had a conspicuity>2 and occluded the vessel by 20 to 80%, the CADe system detected an average of 64% of the subsegmental PEi and 84% of the other PEi, with an average of 14.4 FPs/case. In a recent study, Zhou et al. (128) used an independent test set of 43 CTPA scans to evaluate the performance of the CADe system. A total of 435 PEi were identified in the artery branches by experienced radiologists, of which 263 were in arteries proximal to the subsegmental and 172 were in the subsegmental, respectively. At an average of 33 and 24 FPs/case, the system achieved sensitivities of 81% and 78%, respectively, for PEi that were proximal to the subsegmental, and 79% and 73%, respectively, for subsegmental PEi. Few research groups have participated in the development of CADe systems for PE detection to date. Most of the studies only evaluated commercial systems and reported preliminary results using relatively small sets of case samples. It is expected that the performance of a CADe system will depend strongly on the characteristics of PEi such as their size distributions, percentage of occlusion by PE to an artery, and the diameter of the artery being occluded, or patient conditions such as whether there are other significant pulmonary diseases, and the quality of the CT scans such as the degree of contrast filling and motion artifacts. However, the criteria for determination of PEi in the reference standards were not clearly defined in most studies. These factors would have to be taken into consideration or a common data set has to be used if a meaningful comparison among the performance of different systems is desired. Furthermore, there have been no observer studies to assess the effects of CADe on radiologists' detection of PE on CTPA, which may be another indication of the early stages of the research. Further efforts for development of the CAD technologies will be needed in this area.

Discussion

From this brief review of the CADe systems for lung nodule and PE detection and of the CADx systems for lung nodule characterization, it is apparent that the developments in these areas are still at an early stage. Although commercial CADe systems seem to be more mature, the studies reported to date used very limited data sets and their performances in the general patient population have yet to be evaluated. For lung nodule detection, the commercial CADe systems and the majority of CADe research have focused on solid lung nodules. Since non-solid nodules have a high likelihood for malignancy and are more likely to be missed by radiologists, development of CADe techniques for these nodules will be important. Only two studies included a substantial fraction of non-solid nodules (60, 129) but the sample sizes in both studies were very small. For lung nodule diagnosis, one of the most important piece of information that radiologists use for assessment of the likelihood of malignancy of a nodule is its growth rate measured in repeated CT examinations. Only one study (85) to date incorporated interval change information into the design of the CADx system. No commercial systems are available for CADx so far, probably because of liability concerns for delaying a biopsy recommendation if a lesion turns out to be malignant. The performance and robustness of CADx systems are expected to be much higher than those for CADe systems before they can be considered to be clinically practical. For PE detection, only one of the reported studies had a PE-positive case sample size greater than 50. The reference standards for PE cases are even more difficult to establish than those for other diseases because of the numerous artery segments where PE can occur and the time and effort required for radiologists to inspect every segment down to the subsegmental levels in each case. This may be one of the reasons that the reported performances of the CADe systems for PE varied over a wide range. Furthermore, suboptimal imaging quality caused by the presence of other lung diseases, motion artifacts, and poor contrast opacification is routinely encountered in clinical CTPA cases and should be taken into consideration during classifier training to ensure the robustness of the trained system.

As discussed above, the most challenging step in the development of a CADe or CADx system is often the collection of a sufficiently large database for training and testing the computer algorithms. Ideally, the characteristics of the lesions and cases in the database should be representative of the patient population and the ground truths regarding the lesion locations and other relevant disease information should be included in the database. However, many lesions are not individually biopsy-proven, a reference standard is therefore determined instead based on expert radiologists' consensus and other diagnostic information. Even if a lesion is biopsy-proven, its location has to be manually labeled on the images by radiologists before the ground truth can be useful for algorithm development. Although a large number of cases may exist in the patient archives of hospitals, the extensive efforts required to produce ground truths or reference standards make it difficult to collect a large and comprehensive database. The methods for determination of ground truth or the criteria for establishing reference standard as well as the methods for scoring the true lesions and FPs also affect the apparent performance of a CADe system. For lung nodules, the publicly available LIDC database will be an invaluable resource that may accelerate new developments for computer-aided nodule detection and diagnosis. However, the database is still not very large and does not include serial CT examinations. For PE detection in CTPA, the lack of a large database will be a major roadblock for CADe development. Researchers will need to devote extensive efforts and resources to collect a representative database in order to further advance CADe technologies in this area. Finally, whether a CAD system can improve radiologists' performance in clinical practice will depend not only on the accuracy of the CAD system, but also on a number of other factors, such as radiologists' experience with and confidence in the CAD system and whether they use the system properly as a second opinion and maintain vigilance in their first reading. These factors cannot be simulated in laboratory observer performance studies. It is important to study the impact of CAD with properly designed prospective clinical trials. Understanding these issues may help radiologists take best advantage of CAD and improve patient care.

Table 5.

Studies on the development of CADe systems for detection of pulmonary embolism in CT examinations.

References	No. of cases	No. of positive cases	Section thickness	Presence of artifacts and lung disease	PE location	Total no. of PEi	Sensitivity	FP rate
Masutani et al. 2002 (115)	19	11	1.5 mm	No	not reported	21 (>10 mm3)	85%	2.6/case
Das et al. 2003 (116) (R2)	33	33	0.75–1.25 mm	not reported	Segmental and subseg.	306 (186 seg, 120 subseg)	88% (seg) 78% (subseg)	4/case
Zhou et al. 2005 (124)	14	14	1.25 mm	8 cases with extensive lung disease	Proximal to subseg. and subseg	163 (94 prox-subseg, 69 subseg)	84% (prox-subseg) 64% (subseg)	14.4/case
Digumarthy et al. 2006 (117) (R2)	39	33	not reported	No	Arteries ≥4mm	270	92%	2.8/case
Jeudy et al. 2006 (118) (R2)	22	22	not reported	not reported	Segmental and subseg.	251 (188 seg, 63 subseg)	80% (seg.) 76% (subseg.)	1.8/case
Das et al. 2006 (121) (Siemens)	45	29	1 mm	not reported	Lobar Segmental Subseg.	213	82%	median 3/case
Zhou et al. 2007 (128)	43	43	1.25 mm	yes	Proximal to subseg, and subseg	435 (263 prox-subseg, 172 subseg)	78% (prox-subseg) 73% (subseg.)	24/case
Schoepf et al. 2007 (119) (R2)	36	23	1.25 mm	21 cases with lung disease		130 seg 107 subseg	92% (seg) 90% (subseg)	4.8/case
Maizlin et al. 2007 (120) (R2)	104	15	1.25 mm	Lung disease not reported, severe motion cases excluded	Central, segmental, subsegmental	45 (32 central, seg, 13 subseg)	56.3% (central, seg) 61.5% (subseg)	0.93/case
Buhmann et al. 2007 (122) (Siemens)	40	18	1 mm	5 with motion artifacts, lung disease not reported	Central and peripheral	212 (65 central, 147 peripheral)	74% (central) 82% (peripheral)	3.85/case
Engelke et al. 2007 (123) (Siemens)	56	56	0.6 mm	Non-analyzable arteries excluded	Mediastinal, lobar, segmental, subsegmental	1116* (72 mediastinal, 133 lobar, 465 seg, 455 subseg)	30.7% 21% (mediastinal) 23% (lobar) 41% (seg) 28% (subseg)	4.1/case

R2= CADe system by R2 Technologies, Siemens=Siemens CADe system

^*Subgroup numbers add up to 1125 while a total of 1116 was given in the paper.