Computer-assisted detection of pulmonary embolism: evaluation of pulmonary CT angiograms performed in an on-call setting

Abstract

Purpose

The purpose of the study was to assess the stand-alone performance of computer-assisted detection (CAD) for evaluation of pulmonary CT angiograms (CTPA) performed in an on-call setting.

Methods

In this institutional review board-approved study, we retrospectively included 292 consecutive CTPA performed during night shifts and weekends over a period of 16 months. Original reports were compared with a dedicated CAD system for pulmonary emboli (PE). A reference standard for the presence of PE was established using independent evaluation by two readers and consultation of a third experienced radiologist in discordant cases.

Results

Original reports had described 225 negative studies and 67 positive studies for PE. CAD found PE in seven patients originally reported as negative but identified by independent evaluation: emboli were located in segmental (n = 2) and subsegmental arteries (n = 5). The negative predictive value (NPV) of the CAD algorithm was 92% (44/48). On average there were 4.7 false positives (FP) per examination (median 2, range 0–42). In 72% of studies ≤5 FP were found, 13% of studies had ≥10 FP.

Conclusion

CAD identified small emboli originally missed under clinical conditions and found 93% of the isolated subsegmental emboli. On average there were 4.7 FP per examination.

Introduction

Acute pulmonary embolism (PE) is one of the most frequent potentially fatal diseases. Unfortunately, because of its unspecific clinical symptoms, pulmonary embolism is difficult to diagnose clinically. In two-thirds of the cases the diagnosis is missed if no additional diagnostic procedures are carried out [1–3].

The diagnostic imaging method of choice for the detection of PE in most institutions is contrast-enhanced pulmonary multidetector CT angiography (CTPA) [4–8]. However, each CTPA examination produces on average 300–500 axial images. Thin sections need to be reviewed because it is known that small emboli can be missed on thick sections [9]. Meticulous review of all CT slices is therefore time-consuming and requires a high level of attentiveness. The prevalence of PE in patients sent to CTPA is moderate to low, in the range of 10–35% according to current literature [10, 11]. The chance of missing small emboli increases with time pressure and anatomical and technical complexity and decreases as readers become more experienced [12, 13].

Computer-aided detection (CAD) algorithms have been developed to help exclude pulmonary embolism and to improve the detection performance of observers. The purpose of this study was to assess the stand-alone performance of a computer-assisted detection (CAD) prototype on evaluation of pulmonary CT angiograms (CTPA) performed in an on-call setting. Performance of the CAD algorithm as a stand-alone system was compared with a reference standard and the original reports.

Materials and methods

Patient selection

In this institutional review board-approved study, we retrospectively included all 292 consecutive CTPA studies performed in a university hospital during night shifts and weekends over a period of 16 months between January 2007 and April 2008. All patients had been referred to the radiology department for CTPA because of suspected acute PE. Fourteen patients were excluded from further evaluation for the following two reasons: six CTPA examinations because of streak artefacts based on non-elevated arm positions and metallic material that made a diagnostic evaluation of the imaging impossible and eight examinations because the CAD algorithm did not work for these data sets. In two of these cases, the CAD algorithm failed because the patient had a pneumothorax on the left side or the left lung had been surgically removed. The other six patients had a trachea tube for respiratory ventilation resulting in a connection between extrathoracic air and intrapulmonary air and subsequent failure of the segmentation of the trachea and lungs.

The final study group therefore consisted of 278 patients: 138 male, 140 female, mean age 57 years (range 18–88). There were 133 inpatients (13 patients came from the intensive care unit), 5 patients from the outpatient clinic and 140 patients submitted to the emergency unit. In total, 23% studies were originally reported as positive for PE and 77% as negative.

CT technique

All CTPA examinations were acquired using 16- or 64-detector-row CT; 178 patients were examined by 16-detector-row CT (MX 8000 IXDT or Brilliance-16, Philips Medical Systems, Cleveland, OH) and 100 patients underwent 64-detector-row CT (Brilliance-64, Philips Medical Systems, Cleveland, OH). All CT images were acquired in a caudo-cranial direction from the level of the diaphragm to the lung apices within a single breath hold. A standard PE protocol was applied. The Stellant Dual CT Injector (Medrad Europe BV, Beek, The Netherlands) was used for intravenous bolus injection.

The 64-slice CTPA were obtained with 120 kV, 100 mAs, 64 × 0.625 mm collimation, rotation time 0.4 s and pitch 1.172. All images were reconstructed with 0.9-mm slice thickness and 0.45-mm overlap. Patients received 90 ml of i.v. contrast medium (Ultravist 300, Schering, Berlin, Germany) injected at a flow rate of 5.0 ml/s and followed by a 30-ml NaCl chaser bolus. The 16-slice CTPA were obtained with 90 kV, 180 mAs, 16 × 0.75 mm collimation, rotation time 0.5 s, and pitch 1.188. All images were reconstructed with 1.0-mm slice thickness and 0.5-mm overlap. Patients received 90 ml of i.v. contrast medium (Ultravist 300), injected at a flow rate of 4.0 ml/s and followed by a 40-ml NaCl chaser bolus.

For both techniques, a bolus tracking method was applied with the ROI in the pulmonary trunk. The threshold to start data acquisition was set at 150 HU and a start delay of 8 s after reaching the trigger threshold was used.

Original reports had been based on the evaluation of thin axial images. Multiplanar reconstructions and maximum intensity projections (MIP) were used at the discretion of the interpreting resident or radiologist.

Data collection and analysis

All examinations were analysed by CAD prototype software (Philips Healthcare, Best, The Netherlands). For the analysis of the CAD lesions, a reference standard was established by independent evaluation by two readers. A researcher specially trained in reading PE studies (R.W.) and an experienced chest radiologist (>15 year experience, C.S.P.) evaluated all datasets separately. In the case of discordant findings between these two readers, a third experienced chest radiologist (>15 years experience, M.P.) was consulted. The readers were instructed to digitally mark each intravascular thrombus with regard to its anatomical location (e.g. pulmonary lobe) and the anatomical level of its proximal end (central, lobar, segmental and subsegmental). Standard nomenclature, derived from Boyden [14] and from Jackson and Huber [15], was used to identify the segmental and subsegmental structures. Care was taken that thrombi with continuous extension in various branching vessels were counted as a single lesion and that small subsegmental emboli were included.

In a second step, the CAD findings were compared with our reference standard and considered as true positive, false negative or false positive (Figs. 1, 2, 3). To define a CAD finding as true positive that had not primarily been seen by one of the readers, the presence of an intravascular contrast defect had to be securely confirmed by the other readers.

Fig. 1.

Classis embolus found by CAD

Fig. 2.

Subsegmental embolus missed by primary readers

Fig. 3.

False-positive candidate lesion found by CAD due to motion artifacts

Principles of the CAD algorithm

CAD is a fully automated pulmonary emboli detection prototype system (Philips Healthcare, Best, The Netherlands). It starts from automatic segmentation of the trachea and lungs. The resulting lung volume after morphological processing contains both tissue and air. Subsequently, pulmonary vessels are segmented within this volume, starting with segmentation of all structures above -100 HU. Second, a vessel-tracking algorithm is applied to extract the pulmonary vessels [16]. After pruning the vessels, vessel cross-sectional images perpendicular to the centrelines are computed. Grey value analysis of these cross-sectional images is applied to find candidate locations for pulmonary embolism. No pulmonary emboli are found outside the lung segmentation. Features for identifying pulmonary embolism include stretches of vessels that are completely occluded as well as areas that contain contrast-to-tissue transitions. Candidate lesions that the CAD identifies as potential PE are clustered and presented to the reader, indicated by an ROI around the affected vessel. The processing requires about 30 s per examination and is carried out automatically in the background during initial evaluation by the radiologists. Candidate lesions are then shown only on demand.

Statistical analysis

Statistical analysis was performed with SPSS (version 15.0 statistics UK). Sensitivity for the detection of PE by CAD was assessed on a per-patient as well as on a per-lesion basis, the latter with respect to the main, lobar, segmental and subsegmental levels. Specificity and the negative predictive value (NPV) of an examination without any candidate lesions found by the CAD algorithm were calculated.

Results

Distribution of emboli and sensitivity per embolus

Our reference standard determined that 68 out of 278 patients were positive for PE. A total of 377 emboli were detected in the 68 patients. Twenty-three emboli were localised in the main pulmonary arteries, 41 in lobar, 80 in segmental and 233 in the subsegmental arteries. In 37 patients, there were emboli extending into the main and/or lobar arteries. Thirty-one patients had only segmental and/or subsegmental emboli. Fourteen patients had isolated subsegmental emboli. The per-embolus sensitivity of the CAD as a stand-alone system was 87, 78, 79 and 61% respectively for emboli in the main, lobar, segmental and subsegmental arteries.

Patient-based evaluation

CAD correctly identified PE in seven patients originally reported as negative. In five patients, CAD found a solitary embolus in a segmental (n = 2) or subsegmental (n = 3) artery. In two patients, CAD found two subsegmental emboli.

On a per-patient basis, the sensitivity of the CAD system was 94% (64/68) and the specificity 21% (44/210). CAD correctly identified 93% (13/14) of the patients with isolated subsegmental emboli.

In four patients, we found no embolism although the original reports had been positive. In 21% of the negative cases, there were no FP findings by CAD. The negative predictive value (NPV) of an examination without any candidate lesion found by the CAD algorithm was 92% (Tables 1 and 2).

Table 1.

Diagnosis according to the original reports versus the reference standard

	Standard positive	Standard negative	Total
Original report positive	61	4	65
Original report negative	7	206	213
Total	68	210	278

Table 2.

Diagnosis based on the CAD findings compared with the reference standard

	Standard positive	Standard negative	Total
CAD positive	64	166	230
CAD negative	4	44	48
Total	68	210	278

Analysis of false positives

For the remaining 278 examinations, the CAD algorithm showed on average 4.7 FP lesions per examination (median 2, range 0–42) with most lesions located in non-arterial structures such as veins (30%) or intrapulmonary opacifications (22%). There were on average 3.5 FP lesions (median 2.5, range 0–15) in the group of patients with PE and 5.0 (median 2, range 0–42) in the group without PE. In 72% of the examinations CAD indicated ≤5 FP, in 13% of the examinations CAD indicated ≥10 FP. Nine examinations had more than 20 FP lesions, and all of them were negative for the presence of PE according to the standard.

Discussion

The arterial pulmonary vasculature is a complex anatomical structure whose detailed analysis requires a structured reading approach for the detection of emboli. Thorough analysis of the pulmonary vasculature requires scrolling through the 300–500 axial CT sections multiple times, a time-consuming technique which requires continuous concentration.

The purpose of this study was to assess the performance of this CAD prototype and get an estimation of its potential diagnostic impact. We tested CAD as a stand-alone system. We chose to analyse CTPA examinations that had been performed during night shifts and weekends because they are often read by less experienced colleagues or under time pressure. For these reasons, these examinations may especially benefit from the availability of a CAD system. We compared the CAD results with our reference standard and the original reports.

Our results demonstrate that CAD picked up segmental and subsegmental emboli that were confirmed by the reference standard in seven patients who had been described not to have a PE in the original report. In two patients a solitary embolus had been missed in a segmental artery, in three patients a solitary subsegmental embolus was missed and in two patients two subsegmental emboli were missed.

Our results are compatible with previously published results from smaller studies that both included reader performance. In a study by Engelke et al., 56 CTPA were evaluated by two experienced and two inexperienced readers with CAD as a second reader [17]. They showed the benefit of CAD for the detection of emboli at the segmental and subsegmental levels for the less experienced readers. Das et al. came to the same conclusions [18]: at the subsegmental level the sensitivity of three readers (with 1 and 6 years of experience) increased with CAD from 80 to 92%, from 82 to 90% and from 63 to 81%. However they only selected good quality images (vessel enhancement >200 HU) and excluded patients with underlying lung diseases. In both studies statistics were carried out only on a per-lesion basis.

It is important to note that in our study, we consecutively included all off-hour studies obtained within a certain time period. Thus we also explicitly included images of limited quality, e.g. on the basis of suboptimal enhancement or breathing artefacts. The study group also included images showing underlying lung disease, such as infiltrates or atelectases, another factor that might negatively influence the performance of a CAD system. In total we had to exclude 14 examinations: 8 because the CAD algorithm did not work for these data sets and 6 scans that were considered as non-diagnostic on the basis of massive streak artefacts.

Detection of emboli by CAD has been shown to depend on the level of the embolus (segmental or subsegmental) and the degree of obstruction. The study by Zhou et al. used 14 positive patients to test their CAD algorithm. Their sensitivity for partially (20–80%) occluded arteries was 84% for emboli in central, lobar and segmental arteries and dropped to 64% for emboli in subsegmental arteries [19]. If the vessel was minimally obstructed (≤20%), sensitivity dropped to 65 and 33% respectively. The sensitivity decreased further to 59 and 27% respectively with an obstruction of ≥80%.

We included all obstruction levels and found the sensitivity of CAD as a stand-alone system to be 87% for the central, 78% for the lobar, 79% for the segmental and 61% for the subsegmental emboli.

On a per-patient basis, sensitivity of the CAD was 94%. Despite the variable image quality within the study group and the presence of underlying lung disease (in n = 200), CAD was able to identify 13 out of 14 patients with isolated subsegmental emboli. This emphasises the potential added value of the system for the detection of small emboli, a task for which human observers are also known to have reduced sensitivity. The clinical importance of isolated subsegmental emboli is still uncertain: it is suggested that isolated small emboli may be an indicator of deep venous thrombosis and therefore predictor of more severe embolic events in the future. Furthermore it was stated that the clinical relevance of small peripheral emboli is larger in individuals with cardiopulmonary restriction [8].

A large proportion (n = 62/90, 69%) of subsegmental emboli in our study that were missed by CAD were present in patients with central or lobar emboli. For these cases of central emboli, the detection of additional subsegmental emboli by CAD would have played a negligible clinical role unless the total embolus burden were used to direct patient management. Some subsegmental emboli (n = 12/90, 13%) were missed because they were smaller than 2 mm, which is below the threshold of the CAD algorithm, or because they had led to complete obstruction with a sudden cessation of the subsegmental arteries, which was not detected by CAD.

CAD missed 4 patients out of 68 positive patients with in total one lobar embolus, four segmental emboli and one subsegmental embolus. These four false-negative examinations were not characterised by especially low vascular enhancement or overlying motion artefacts. From a technical point of view, we were not able to identify obvious reasons why CAD was not able to identify the emboli in these four patients.

Most CTPA examinations done in clinical routine are eventually negative for the presence of PE. We found a negative predictive value (NPV) of 92% for examinations in which CAD did not find any candidate lesion. While an NPV of 92% is not yet sufficient for complete reliance on a negative CAD reading, it underlines the potential role of CAD as a second reading to reassure a reader when excluding PE. On the other hand, we have to state that only 44 out of the 210 negative cases did indeed have a completely negative CAD reading. Most negative PE examinations show differing numbers of false-positive CAD lesions. The specificity of the algorithm on a per-patient basis is very low at 21% indicating that there is still a need for considerable improvement and that readers have to learn to efficiently rule out false-positive candidate lesions if the application of CAD is to be beneficial.

Our results are comparable to those of the study by Walsham et al. [20]. They found an NPV of 84% and a specificity of 20% in a smaller (n = 100), but also non-selected patient group. While Zhou et al. showed an average of 14.4 false positives per study, our system produced 4.7 false positives per study. Under the assumption that a maximum of 5 false positive lesions per examination marked by CAD represents an acceptable level in clinical routine, this criterion was met in 72% of our patients. Thus most examinations produce a relatively low number of false-positive lesions. The median number of false-positive lesions was 2 with most located in non-arterial structures such as veins (30%) or intrapulmonary opacifications (22%). Thus it is likely that in most scans the false-positive lesions can easily be sorted out. The exact impact of the false-positive lesions on diagnostic accuracy and reading time, however, remains to be determined in a reader study. More than 10 false-positive lesions were seen in 13% of the CTPA examinations; not surprisingly, higher numbers of false-positive lesions correlated with lower image quality. Further reduction of false-positive lesions per image is needed as too many may have a negative influence on diagnostic performance, lead to an unnecessarily prolonged reading time and might even cause unnecessary treatment.

Our study suffers from some limitations. There is no absolute reference standard for CTPA studies. Diagnostic pulmonary catheter angiography has been completely substituted by multidetector CTPA and had additionally been questioned as reference standard anyway. We therefore established a reference standard by independent evaluation of two readers with a third reader in case of discordant findings.

We evaluated the stand-alone performance of the CAD software to provide an estimate of the performance of the algorithm. In our study group, 10% of PE patients were originally missed by the on-call radiologist, all of whom had PE identified by CAD. However, comparison between the performance of CAD as a stand-alone system and that of the original reports is not sufficient to define the role of CAD in a clinical setting. The potential benefit of a CAD system strongly depends on the interaction with the reader, his or her experience and ability to discriminate between true- and false-positive candidate lesions. More studies are necessary to evaluate the use of CAD as a second reading and to assess its impact on the reader’s capability not only to detect emboli but also to exclude pulmonary embolism.

In summary, CAD can identify segmental and subsegmental pulmonary emboli that are missed by on-call radiologists, with an average of 4.7 false-positive lesions per examination. A CTPA without any CAD candidates has a high negative predictive value, which may serve as reassurance for less experienced readers. Furthermore CAD found most of the isolated subsegmental emboli, which are clinically the most difficult emboli to find.