The Interobserver Agreement between Residents and Experienced Radiologists for Detecting Pulmonary Embolism and DVT with Using CT Pulmonary Angiography and Indirect CT Venography

Abstract

Objective

We wanted to prospectively evaluate the interobserver agreement between radiology residents and expert radiologists for interpreting CT images for making the diagnosis of pulmonary embolism (PE).

Materials and Methods

We assessed 112 consecutive patients, from April 2007 to August 2007, who were referred for combined CT pulmonary angiography and indirect CT venography for clinically suspected acute PE. CT scanning was performed with a 64×0.5 collimation multi-detector CT scanner. The CT studies were initially interpreted by the radiology residents alone and then the CT images were subsequently interpreted by a consensus of the resident plus an experienced general radiologist and an experienced chest radiologist.

Results

Two of the 112 CTs were unable to be interpreted (1.7%). Pulmonary artery clots were seen on 36 of the thoracic CT angiographies (32%). The interobserver agreement between the radiology residents and the consensus interpretation was good (a kappa index of 0.73). All of the disagreements (15 cases) were instances of overcall by the resident on the initial interpretation. Deep venous thrombosis was detected in 72% (26 of 36) of the patients who had PE seen on thoracic CT. The initial and consensus interpretations of the CT venography images disagreed for two cases (kappa statistic: 0.96).

Conclusion

It does not seem adequate to base the final long-term treatment of PE on only the resident's reading, as false positives occurred in 13% of such cases. Timely interpretation of the CT pulmonary angiography and CT venography images should be performed by experienced radiologists for the patients with suspected PE.

Pulmonary embolism (PE) is one of the most serious complications of deep venous thrombosis (DVT). More than 90% of PEs arise from clots in the deep veins of the legs and pelvis, and the primary risk factor for recurrent pulmonary embolism is the presence of residual proximal venous thrombosis (1). Making the diagnosis of PE remains difficult in clinical practice because the clinical findings are nonspecific and all the available objective tests have practical or clinical limitations (2).

CT angiography has recently been accepted by many facilities as the noninvasive test of choice for the detection or exclusion of pulmonary embolus. Acute pulmonary embolus is considered an emergency condition, and the CT angiograms for suspected pulmonary embolus are frequently obtained after routine work hours. At many academic institutions, these CT angiograms are initially reviewed by residents, with the subsequent final review being done by an attending radiologist early the next morning.

We performed this prospective study on the consecutive patients with suspected pulmonary embolism to evaluate the interobserver agreement between the radiology residents and the expert radiologists for the radiology residents' interpretation of multi-detector CT (MDCT) images to diagnose pulmonary embolism. For this purpose, the residents first interpreted the MDCT studies alone in an emergency setting, and then they evaluated the studies again within 12 hours as a member of a panel of radiologists. Thus, we determined the interobserver agreement of the first reading by the radiology residents.

MATERIALS AND METHODS

The institutional ethics committee approved this study, and informed consent was obtained from the patients or from the patients' relatives if the patients were unable to grant informed consent.

Patients

During a five-month period (April 2007 to August 2007), all the consecutive patients who were referred to undergo combined 64-detector row CT pulmonary angography (CTPA) and indirect CT venography for the suspicion of PE were included in this study. Consent for participation was granted by all 112 patients (53% males, mean age : 65 years, age range : 20-87 years). There was no statistically significant difference of age between the genders (p > 0.05).

Study Design

This observational study was designed to measure the interobserver agreement between radiology residents and expert radiologists for the radiology residents' independent interpretation of MDCT images to diagnose PE. The consensus interpretations of the same images were done several hours later by a panel of experienced radiologists, as detailed below.

Procedures

CT scanning was performed with a 64-detector (multislice) CT scanner (Aquillion 64, Toshiba, Tokyo, Japan). The lungs were scanned from the base to the apex in the caudocephalic direction with using the following parameters: collimation of 64×0.5 mm, 3.5 cm/sec table movement per gantry rotation, a rotation time of 0.75 seconds, 225 mAs and 120 kVp. The injection of contrast material was performed using an automatic power injector (CT injector; Ulrich Medical, Ulm-Jungingen, Germany) at a flow rate of 3.5 mL/sec. All the patients received 150 mL of Omnipaque 350 mg I/mL (Amersham Health, Cork, Ireland). CT venography was performed 180 seconds after the start of injection, from the diaphragm to the patella, by using a sequential acquisition of 4×4 mm thick sections at 40-mm intervals, 125 mAs and 120 kVp. The total scanning time was 7.4 ± 1.45 seconds for the entire chest, depending on the covered volume.

Image Interpretation

The CT studies were viewed on a Vitrea 2® workstation (Vital Images, Plymouth, MA) by one of 10 radiology residents with 2-5 years experience (mean experience: 3 years). Within 12 hours, all the MDCT studies were viewed again on the same workstation by a panel consisting of a general radiologist (nine years of experience) and a chest radiologist (20 years of experience). Both the residents and the expert radiologists were able to review the lung window images. In case of disagreement with the initial interpretation, the second interpretation was communicated to the lead attending physician to guide the patient's care. Agreement between the initial and consensus panel readings was assessed by using kappa statistics.

Detecting PE on the CT angiograms included analysis of the main, lobar, segmental and subsegmental pulmonary arteries. The vascular signs of pulmonary embolism were central partial intravascular filling defects surrounded by contrast medium, eccentric partial filling defects surrounded by contrast medium, or complete filling defects that were not surrounded by contrast medium and they occupied the total vessel section (3, 4). Thoracic CT angiography was considered nondiagnostic if the enhancement of the pulmonary arteries was insufficient to visualize a clot; if breathing and motion artifacts or the underlying lung disease hindered the examination of at least one segmental artery, or if an image was not conclusive (regardless of the location) (2). The CT findings were considered normal if no signs of PE were present on the technically adequate images.

The CT findings that were diagnostic for DVT were complete or partial filling defects with enlargement of the vein (5). The CT venography was considered nondiagnostic if the venous enhancement was insufficient to visualize a clot, or if artifacts from orthopedic materials prevented proper visualization (6). The CT venography was considered normal if no venous abnormalities were found on the technically adequate images.

Statistical Analysis

The interobserver agreement among the initial and consensus readings was assessed by using kappa statistics (7). Comparisons with a p value less than 0.05 were considered statistically, significantly different. The statistical analyses were performed using SPSS for Windows®, version 15.0 (SPSS Inc., Chicago, IL).

RESULTS

The study protocol was completed for all 112 patients who entered the study. Clots within pulmonary arteries were seen on 36 of 112 (32%) thoracic CT angiography examinations. Of these, 33 had clots within a segmental artery and three had a clot in only a subsegmental artery. There was disagreement between the initial and consensus interpretations of the 110 technically adequate pulmonary MDCT images for 15 cases (kappa statistic: 0.73 [95% confidence interval, CI: 0.68-0.78]). All of these were instances of overcall by the resident on the initial interpretation (Table 1). The consensus panel pointed out hypoattenuation within a few segmental arteries as being a consequence of motion artifact in each of the false-positive case. None of these 15 patients had DVT noted on the CT venography. All of the false positive clots were located at the subsegmental level. 53% of them (8 of 15) were located in multiple subsegments, including the subsegments that were in the vicinity of the heart. The other seven false positive clots were located in only one subsegment, and especially in both lower lobes or in the left lingular division. The thoracic CT images were nondiagnostic for two of 112 patients, and this was due to lung motion in both cases. The CT venography was positive for both of these patients.

Table 1.

Interpretation of 64-slice Pulmonary CT Angiography Images by Radiology Residents and by Consensus Panel of Radiologists for Determining Presence of Pulmonary Embolism

Note.-kappa = 0.73, 95% CI: 0.68-0.78

Deep vein thrombosis was found in 28 of 112 (23%) patients by the CT venography. Two of these patients had nondiagnostic thoracic CT exams and 26 of these patients also had positive thoracic CT exams for PE, resulting in 72% (26 of 36) of those patients with positive thoracic CT scans having DVT. None of the three patients with PE that was limited to the subsegmental arteries had a positive CT venography exam. No patient with negative thoracic CT angiography had DVT diagnosed by the CT venography. The CT venography images were nondiagnostic in six of 112 patients (5%). Two of these were due to poor enhancement, three were due to artifacts from orthopedic materials, and one was due to both of these problems. The initial and consensus interpretations of the CT venography images disagreed for two cases (kappa statistic: 0.96 [95% CI: 0.93-0.98]) (Table 2).

Table 2.

Interpretation of CT Venography Images by Radiology Residents and by Consensus Panel of Radiologists for Determining Presence of Deep vein Thrombosis

Note.-kappa = 0.96, 95% CI: 0.93-0.98

DISCUSSION

Multi-detector row CT has become the first-line modality for imaging those patients who are suspected of having PE. Because of the technical and logistical issues that go with conventional pulmonary angiography, many institutions now use only CT venography and CTPA for diagnosing PE. At many academic institutions, radiology residents provide the initial interpretation of the CT examinations obtained after working hours and on weekends, including the CTPA examinations to diagnose or exclude PE. This initial CTPA report is typically relied upon when making the initial clinical management decisions. In the present study, we wished to examine the concordance between the initial interpretations given by the radiology residents and the final consensus interpretation made by the two experienced radiologists.

Shaham et al. (8) found that the residents' preliminary interpretations of pulmonary CT angiography during the off hours were reasonably accurate when compared with those of radiology specialists (kappa statistics of 0.7 and 0.8, respectively). The study by Ginsberg et al. (9) compared interpretations of 663 CTPA exams by the on-call radiology fellows versus those by the radiology faculty, and the overall agreement was 93% (kappa: 0.8). Our findings also indicated good correlation (kappa: 0.7) between the readings of our radiology residents and those of the experienced radiologists. All the discrepancies between the two readings were concerned with initial false-positive interpretations, and all were due to motion artifacts, which were misinterpreted as pulmonary emboli. In an article by Garg et al. (10), the authors found moderately good interobserver agreement for DVT with using CT venography, with disagreement for 17 (12%) of 146 venography studies and a kappa of 0.59. We found very good interobserver agreement for DVT with using CT venography (kappa: 0.96). The only two discrepancies between the two readings were concerned with the initial false-positive interpretations and these were due to flow artifact on the CT venography.

Wittram et al. (11) found that respiratory motion artifacts can cause misdiagnosis of pulmonary embolism, and respiratory motion artifacts are the most common cause of an indeterminate CTPA. They concluded that the quality of the CTPA needs to be first considered by the radiologist when a patient presents with suspected PE. If the quality of the study is poor, then the radiologist should identify which pulmonary arteries have been rendered indeterminate and the radiologist should decide whether additional imaging is necessary. Jones and Wittram (12) found motion artifacts to be responsible for 74% of the indeterminant CTPAs. In the present study, both the residents and expert radiologists were able to review the lung window images. The residents reported false positive findings in 15 patients even with reviewing the lung window images. None of these 15 patients had DVT or PE at the segmental level. We think that if the segmental arteries are normal and DVT is not present on CT venography, then one should be reluctant to diagnose a subsegmental PE so as to avoid unnecessary anticoagulation therapy.

Musset et al. (2) reported that isolated subsegmental PE was found in 3% of the PE patients with interpreting the single-detector row helical CT. Remy-Jardin et al. (13) found no case of isolated subsegmental embolism in their patients who were examined with multi-detector row CT. Angiography showed PE limited to the subsegmental pulmonary arteries in 6% of the patients with PE in the PIOPED (prospective investigation of pulmonary embolism diagnosis) study (14). In a retrospective examination of 76 patients with positive findings for PE, Oser et al. (15) identified isolated subsegmental PE in 23 of the patients (30%). The rate of isolated subsegmental PE has varied among the patient populations reported on in the medical literature, and these patient populations varied widely from each other. The rate of diagnosing isolated subsegmental PE in this study was 8% (3/36), and this rate is within the reported range in the previous literature.

Our patients with PE, according to the CTPA, had a higher proportion of DVT (72%) than is usually reported. Loud et al. (1) detected DVT in 50% (58 of 116) of the patients with PE. Rhee et al. (16) concluded that the addition of CT venography to CTPA significantly increases (from 13% to 17%) the rate of diagnosis of thromboembolic disease over that with using CTPA alone. In our current study, none of the three patients with PE limited to the subsegmental level had DVT, according to the CT venography, but CT venography made the diagnosis of DVT in two patients who were not diagnosed via CTPA. These findings are in agreement with those of Rhee et al. (16), that is, important information is gained when CT venography is added to CTPA in the workup of suspected PE patients.

Despite the improved accuracy of multi-detector row CT over single-detector row CTPA, adding indirect CT venography to the scanning protocol results in an additional diagnosis of thromboembolism in 20% to 33% of patients (16-18). A 40 mm interval was used between the venous images because small isolated thrombi are unusual (19), and undergoing contiguous CT venous phase imaging results in a substantially higher radiation dose without increasing the sensitivity for detecting DVT (1). Yankelevitz et al. (20) reported that near peak enhancement of the lower extremity veins could be achieved in most patients at 2 minutes after CTPA. In our study, the CT venous phase images were acquired 3 minutes after the initiation of contrast medium infusion into an arm vein, and this allowed uniform enhancement in the lower extremity veins in the majority of patients, and even in those patients with slower circulation times. Using this protocol, CT venography was non-diagnostic due to poor enhancement in 5% (6 of 112) of the patients.

The limitations of our study include the small number of patients. In addition, no follow-up, other tests or clinical follow-up were used to confirm the MDCT interpretations.

The radiology residents were fairly good at interpreting the CTPA images, and there were no instances of false-negative interpretation. If the segmental arteries are normal and DVT is not present on CT venography, then one should be reluctant to diagnose a subsegmental PE. Motion artifact is likely in such cases. It does not seem adequate to base the final long-term treatment only on the resident's reading, as false positives occurred for 13% of the patients. We suggest that timely interpretation of the CTPA and CT venography images of the patients with suspected PE should be performed by experienced radiologists.