Variabilities in Reference Standard by Radiologists and Performance Assessment in Detection of Pulmonary Embolism in CT Pulmonary Angiography

Abstract

Annotating lesion locations by radiologists’ manual marking is a key step to provide reference standard for the training and testing of a computer-aided detection system by supervised machine learning. Inter-reader variability is not uncommon in readings even by expert radiologists. This study evaluated the variability of the radiologist-identified pulmonary emboli (PEs) to demonstrate the importance of improving the reliability of the reference standard by a multi-step process for performance evaluation. In an initial reading of 40 CTPA PE cases, two experienced thoracic radiologists independently marked the PE locations. For markings from the two radiologists that did not agree, each radiologist re-read the cases independently to assess the discordant markings. Finally, for markings that still disagreed after the second reading, the two radiologists read together to reach a consensus. The variability of radiologists was evaluated by analyzing the agreement between two radiologists. For the 40 cases, 475 and 514 PEs were identified by radiologists R1 and R2 in the initial independent readings, respectively. For a total of 545 marks by the two radiologists, 81.5% (444/545) of the marks agreed but 101 marks in 36 cases differed. After consensus, 65 (64.4%) and 36 (35.6%) of the 101 marks were determined to be true PEs and false positives (FPs), respectively. Of these, 48 and 17 were false negatives (FNs) and 14 and 22 were FPs by R1 and R2, respectively. Our study demonstrated that there is substantial variability in reference standards provided by radiologists, which impacts the performance assessment of a lesion detection system. Combination of multiple radiologists’ readings and consensus is needed to improve the reliability of a reference standard.

Introduction

The advent of multidetector computed tomography (MDCT) scanner provides thinner section acquisition with greater speed, allowing higher image quality reconstructed with improved spatial, contrast, and temporal resolutions [1]. CT pulmonary angiography (CTPA) has been widely used as an effective noninvasive modality for clinical diagnosis of pulmonary embolism (PE) [2]. With MDCT, various studies have reported that CTPA had high sensitivity (60–100%) and specificity (80–90%) for central PE detection but the sensitivity for segmental and subsegmental PE was still low [3–20]. The prospective investigation of pulmonary embolism diagnosis (PIOPED) II trial reported that CTPA had an overall sensitivity of 83% with a specificity of 96% and positive predictive values of 97%, 68%, and 25% for PE in main or lobar, segmental, and subsegmental arteries, respectively [1]. PIOPED II also reported that the addition of CT venography (CTV) to CTPA improved the ability to detect PE with improved sensitivity of 90%, but the ability of ruling out PE remained the same compared with CTPA alone. The main challenge with the diagnostic interpretation of CTPA is the difficulty of finding PE in small subsegmental arteries, and the inadequate detection of subtle PEs with small percentages of occlusion in larger proximal to segmental arteries. The confidence in detecting the presence or absence of a PE clot relies not only on the visibility of the arteries which could be adversely affected by motion and streak artifacts, partial volume averaging, insufficient vascular opacification with contrast, or lung abnormality [21, 22], but also on the experience of individual radiologists in interpretation of chest CT [23]. The improved visibility with modern MDCT results in substantially higher detection rates for subsegmental emboli and better agreement among readers, but further improvement is still needed, especially for those distal or minimally occlusive clots [12–14, 19, 24].

Small PEs may produce significant morbidity in patients with underlying cardiorespiratory disease [25] and may indicate a risk for recurrence of more significant PEs among stable patients. Early detection of isolated subsegmental defects will allow management decision to be made and anticoagulant treatment be administered if necessary [19, 26] to reduce the risk of further deterioration of the condition. In addition, it is important to estimate the total burden of pulmonary vascular clots in patients with acute PE to determine proper therapy and to improve patient outcome [27–32]. Therefore, it is clinically important to improve the accuracy of CTPA in the detection of small PEs [19, 22, 26].

Computer-aided detection (CAD) may be a viable approach for assisting radiologists in this demanding task as a second opinion and reducing the chance of false negatives [33]. With advanced machine learning and computer vision techniques, the CAD system can be trained to automatically trace the pulmonary vessels, distinguish the arteries from the veins, detect suspicious PE locations by searching along the arteries, and finally alert the radiologists to the region of interest (ROI) for suspicious PEs [34, 35]. Using computer to automatically detect PE in CT is a challenging task because of the vast network of pulmonary arteries in the lungs and their variable sizes. The few studies that have been performed to date are very preliminary using relatively small data sets [36], and very few studies have been conducted to evaluate the value of CAD as a second reader in PE detection. [37–40].

We are developing a CAD system to assist radiologists in PE detection. A key step in developing CAD systems is to collect a large data set with annotated lesion locations in the case samples for training and testing. For diseases such as PE that have no clinically biopsy-proven locations of individual lesions, a common approach is to establish reference standard by radiologists’ manual marking of the lesions in the images. The reference standards for PE in CTPA are more difficult to establish than those for many other diseases because of the numerous arterial segments where PE can occur and the time and effort required for radiologists to inspect every segment down to the subsegmental levels in each case. Establishing the reference standards is also an equally important and challenging step for conducting human reader studies to compare lesion detectability in images with multiple target lesions and no biopsy ground truths. In this study, we evaluated the variability of the radiologist-identified PE locations, analyzed how consensus reading can improve the reference standard, and revealed the need to further revise the reference standard as the computer or human readers being tested may detect additional true lesions missed in the reference standard. This study demonstrated the importance of using a multi-reader, multi-step process in establishing a reliable reference standard for developing and validating CAD systems, for studying the impact of CAD on radiologists’ reading, or for comparing different imaging techniques and modalities by observer performance studies.

Materials and Methods

Materials

With approval of the Institutional Review Board (IRB), we collected 40 inpatient CTPA PE-positive cases retrospectively from the patient files at the University of Michigan (UM) to assess the variations between radiologists’ marking of PE. The images were acquired with GE MDCT scanners with technique factors of 120–140 kVp and 300–600 mAs and 1.25 mm reconstruction interval.

Methods

Reference Standard Provided by Radiologists

We developed a computer graphical user interface (GUI) for radiologists to identify the PE locations. Figure 1 shows a screen shot of the GUI that allows the users to view the CT slices in a cine-paging mode and adjust the window and zoom settings to improve visualization. Using the GUI, the radiologists interactively marked the PE location on every slice and split the single PE volume that occluded multiple branches of arteries by marking the PE segment in each branch as a separate PE [33, 41]. Figure 2a illustrates an example of a single PE volume being split into three at the branching point of the vessel. For each PE segment, radiologists marked the approximate location of its center and the starting and ending slices with a cursor, identified the anatomical level of the artery (trunk, main, lobar, segmental, and subsegmental), measured the diameter of the artery with an electronic ruler, visually estimated the percentage of PE occlusion in the artery, and provided a confidence rating of the location having a true PE and a rating of conspicuity, both on a scale of 1 to 5 (5 = most confident or most conspicuous) [41]. The radiologist also manually tracked the PE along the occluded arterial branch by placing points (dot markers) on the PE [1]. Figure 2b&c shows an example of the manually marked PE points on a slice intersecting the PE. We developed a semiautomatic computer tool to generate a volume of the PE using a supervised region growing method [33, 42–44] and the dot markers as seeds. Each volume along the track of dot markers was then identified as an individual PE.

Fig. 1.

A screenshot of our computer graphic user interface for identifying PE on CTPA images

Fig. 2.

a Illustration of the splitting of a single PE volume by marking the PE segment in each arterial branch as a separate PE. b, c The points by radiologists tracking the PE location with the vessel are shown as dot markers

Each case was first read independently by two experienced thoracic radiologists (R1 and R2), referred to as the initial readings. A semi-automatic method was developed to determine the agreement in the marked PE locations between the two radiologists. First, for a PE marked by one radiologist (e.g., R1), all PE locations marked by the other radiologist (i.e., R2) were automatically searched for the maximum overlap region to find an initial match. The number of overlapped voxels was used as the overlap metric, and the two PE locations was counted as overlap if their overlap metric > 0. The initial match was also performed for matching the markers of R2 to those of R1. Each PE with or without a matched PE location was then visually examined to confirm or correct the agreement or disagreement. As shown in Fig. 2a, the way of splitting a contiguous PE into separate segments might vary between readers, e.g., R1 might mark two PEs (PE1 and PE2) but R2 might mark all three PEs. We counted the number of PEs split from a single PE as the maximum number of separate PEs among all radiologists so that the final number of separate PEs was counted as three in this example.

For any PE location that was found to disagree in the two independent readings, the location would be re-read by the two radiologists independently. In the re-reading, each radiologist assessed each discordant marking, without knowing which radiologist marked the location or the likelihood and conspicuity ratings in the initial reading. In this setting without the tedious search of new PE locations in the arteries branch by branch, the reader could concentrate on examining the regions of interest and identifying them as true or false PE. For the markings that still could not reach agreement between the two radiologists in the re-reading, the two radiologists assessed these markings together to reach a consensus. The set of PE locations after the consensus is considered to be the reference standard.

Computer-Aided Detection System for PE

In our previous study [41], we developed a CAD system for PE detection. Our CAD system includes the following processes: vessel enhancement and segmentation, vessel tree construction, prescreening of suspicious PE locations, feature extraction and classification for false positive (FP) reduction, and output of the detected PE locations. The details of the CAD system have been described elsewhere [33, 41]. Briefly, the lung region is first extracted using thresholding and morphological operations. The pulmonary vascular structures including the vessel bifurcations are enhanced using a 3D multiscale filters based on the analysis of the eigenvalues of Hessian matrices. The pulmonary vessel tree is then extracted using a 3D hierarchical expectation-maximization (EM) segmentation method [45]. A parallel multi-prescreening method identifies volumes of interest (VOIs) that contain suspicious PEs along the extracted vessels. This method uses two independent prescreening methods to detect suspicious PEs in parallel: (A) global search using an adaptive EM thresholding method applied to the entire vessel tree at multiple scales and (B) local search for the transition regions of CT values between normal and PE occluded vessel segments in a local cubic VOI centered at a given point along the centerline of the tracked vessels. Nine features including contrast, intensity difference between the detected object and its background, and object size were extracted to differentiate the true and false PEs [41]. A linear discriminant analysis (LDA) classifier with stepwise feature selection was then trained to reduce FPs. The CAD system was trained with 69 CTPA PE-positive cases collected in the PIOPED clinical trial. The performance of the CAD system was evaluated with the 40 independent UM test set by the free response receiver operating characteristic (FROC) analysis [41].

To study the impact of reference standard on the evaluation of a lesion detection system, we compared the CAD marks and the radiologists’ reference standard of PE locations in the 40 test cases, including the initial independent readings of R1 and R2, and the consensus of R1 and R2, respectively. For the CAD marks that were scored as false positives based on the reference standard from the consensus of R1 and R2, radiologist R1 and a third experienced thoracic radiologist (R3) independently examined the marked locations to judge whether there were true PEs. For the CAD marks that the two radiologists’ (R1 and R3) judgments disagreed, the two radiologists again assessed the marks together to reach a consensus.

Results

Figure 3 shows the reading process of the two radiologists for PE marking and the reading results. For the 40 PE cases, 447 and 510 PEs were initially marked by radiologists R1 and R2, respectively. After the semi-automatic matching of the markings and applying the rule of splitting contiguous PEs such that the maximum splitting was used if the splitting by the two radiologists was different (see Section III), the initial markings by R1 and R2 were recounted as 475 and 514, respectively. A total of 444 PE marks were found to be in agreement and 101 in disagreement between the two radiologists for a total of 545 (= 444 + 101) individual PEs marked by R1 and R2.

Fig. 3.

Process for determining reference standard by two experienced thoracic radiologists. Note that the number of PE marks was counted after applying the rule of using the maximum splitting if the splitting of contiguous PEs by the two radiologists was different

The two radiologists then independently re-read the 101 PE marks in 36 cases that were not in agreement. After this second reading, the two radiologists agreed on 96 of the 101 marks in 36 cases and did not agree on 5 marks in 5 cases. The disagreed marks were re-read by the two radiologists together and determined to be 3 true PEs and 2 FPs. After the final consensus, 65 (64.4%) of the 101 marks were determined to be true PEs, and the remaining 36 (35.6%) were FPs (Fig. 3). Of the 65 true PEs, 17 and 48 were marked by R1 and R2, respectively, in their initial reading. Of the 36 FPs, 14 and 22 were marked by R1 and R2, respectively. With this process, a total of 509 (= 444 + 65) PEs were identified and used as reference standard. R1 identified 461 (= 444 + 17) and R2 identified 492 (= 444 + 48) of the true positives (TPs). Figure 4 shows examples of the false negatives (FNs) and FPs by radiologists.

Fig. 4.

Examples of the false-negative (FN) and false-positive (FP) PE locations by radiologists (R1 or R2) in their initial reading before consensus, a FN by R1, with 90% occlusion in a subsegmental artery (likelihood = 5, conspicuity = 2), b FN by R2 with 20% occlusion in a segmental artery (likelihood = 3, conspicuity = 1), c FP mistakenly marked by R1 in a vein, d FP marked by R2 due to partial volume effect

Figure 5 shows the impact of reference standard on scoring the FROC curves of our CAD system, using the initial readings of R1 and R2, and the consensus of R1 and R2 as reference standards, respectively. The FROC curve changed substantially between using the markings by R1 and R2, or their consensus reading as the reference standard for scoring.

Fig. 5.

Effect of reference standard on scoring FROC curves of our CAD system for PE detection. FROC curves of CAD performance scored by using initial readings of R1, R2 and their consensus reading as the reference standards were compared

After the radiologists (R1 and R3) examined the CAD marks that were scored as FPs, 23 of the CAD marks in 14 cases that were not included in the reference standard were determined to be true PEs that were not marked by radiologists (R1, R2). With the 23 FP CAD marks re-classified as true PEs by the radiologists and included in the reference standard, a total of 532 PEs (= 509 + 23) were identified as true PEs in 40 cases. Figure 6 shows the revised FROC curve after the initial FP marks by the lesion detector are scrutinized by the expert panel, further changing the reference standard. For comparison, the FROC curve using R1’s initial reading as reference standard, which had the largest differences from the other curves (see Fig. 5), was also plotted. At a test sensitivity of 80%, the FPs marked by the CAD system were changed from an average of 31.3 FPs/scan to 20.6 FPs/scan (a total of 824 FPs for 40 cases). The result demonstrates the large discrepancy in performance assessment that could occur with different processes of annotating data sets.

Fig. 6.

Effect of reference standard on scoring FROC curves. Comparison of the lowest and the highest FROC curves scored by the reference standard obtained from different stages to illustrate the range of variation in scoring the performance of the detection system

Discussion

There is large variability between two radiologists’ manual markings of PE locations in their initial reading. R2 had much higher sensitivity than R1. R1 missed 48 and R2 missed 17 of the 65 PEs that were detected by the other radiologist, while CAD can detect 33 of the 48 PEs missed by R1 and 11 of the 17 PEs missed by R2. As a result, if one radiologist, e.g., R1, is available to provide reference standard, the FROC curve for a lesion detection system can be substantially lower (Fig. 5).

Even after the CAD marks were scored with the reference standard obtained from consensus of two radiologists, the additional step of checking the CAD marks scored as FPs by the expert panel still could find more true lesions. Of the 23 true PEs recovered, 17 were missed by both radiologists, and 6 were missed by one of the radiologists (2 of the 6 TPs were marked by R1 and 4 by R2 in the initial reading but discarded as FPs after re-reading and consensus); thus, a total of 21 and 19 of the 23 PEs were missed by R1 and R2, respectively. These corrections in the reference standard further changed the FROC curve.

For the PE detection task in CTPA, a number of factors, such as the vast network of pulmonary vessels, the small size of subsegmental arteries, and the blurring caused by motion and partial volume effects, affect the visibility of arteries and PEs inside the arteries. The image quality of the CTPA strongly impacts the detectability of PE by radiologists or CAD system, which may be part of the reasons that cause the disagreement between radiologists. In this study, the majority of the initial 101 disagreed PE locations in 36 cases were marked in the small arteries, specifically, 69.3% (70/101) in the subsegmental and 24.8% (25/101) in the segmental level of the arteries. However, even PEs in the large major arteries can be missed, i.e., 1.0% (1/101) in the main and 5.0% (5/101) in the lobar arteries. In radiologists’ manual marking, the FPs were mainly caused by partial volume effects or regions in the distal pulmonary veins that have lower CT values similar to those of PE occluded arteries. After consensus, it was found that a total of 65 PEs were not marked in the initial reading by the two experienced radiologists. Of these, 48 PEs that were missed by R1 included 1, 4, 7, and 36 PEs located in the main, lobar, segmental, and subsegmental arteries, respectively, and 17 PEs that were missed by R2 included 5 and 12 PEs in the segmental and subsegmental arteries, respectively. Although CTPA allows assessment of the distal pulmonary vessels, the vast amount of vessels and the hundreds of CT slices in the CTPA exam make it challenging for radiologists to evaluate each distal subsegmental artery. Our study demonstrated that consensus reading of the CTPA cases improved the sensitivity and reference standard.

The evaluation of lesion detection performance either by a machine learning system or by a radiologist depends on the reference standard used for the evaluation. We used PE detection in CTPA as an example in this study because detecting PE is a particularly challenging task. The inter-radiologist variability [46, 47] is known to be high so that we can demonstrate the effects using only a small number of radiologists to provide reference standard and a small number of cases, as the multi-stage reading is time consuming for radiologists. However, this problem is not unique to PE detection. The observation in this study is applicable to other multi-target detection tasks such as lung nodules or liver lesions in CT. To produce a reliable reference standard, ideally one should have multiple experts read the cases independently, compare the results, identify the differences, re-read the discordant cases, and try to reach a consensus. Since it is difficult to assemble multiple experts at the same sitting to read a large number of cases, the process often requires several iterations of independent readings to reach consensus or to reduce to a small number of discordant cases for final consensus reading. For multi-reader multi-case observer study [48] comparing different reading conditions or modalities, the additional step of checking all the discordant marks between each observer and the reference standard is especially important because a change in the reference standards could reverse the relative ranking of the conditions or modalities being compared, and the change could be large after combining multiple radiologists’ readings of the same set of cases.

It may be noted that the “final” reference standard used in our demonstration study would likely change if we have the same radiologists to go through the process again, or we have different group or different number of radiologists to read the same set of cases. A reference standard based on subjective reading, even with experts, is never the ground truth, due to inter- and intra-reader variabilities. However, repeated readings and eventual consensus can improve the reliability of a reference standard. Such reference standard is often the “best” one can have because patients with disease such as PE will not undergo biopsy, or patients with disease such as lung cancer may have biopsy of the main lesion of concern but no biopsy of other possible lesions. For retrospective collection of cases from patient files, one cannot request other more definitive procedures or tests for confirmation of the lesions. Even for prospective collection, requesting additional procedures for a large number of patients is often impractical because of the costs and other considerations. To develop CAD systems for medical imaging, especially those used deep learning methods, collecting a large enough set of training and test cases with good reference standard is the most important but costly step [49]. In practice, the data set size and the number of expert radiologists are often limited by the resources available to generate the reference standard.

There are limitations in this study. First, because the interpretation of a CTPA study demands extensive reading time from radiologist who has to visually track each vessel, each of our radiologist readers only read the 40 cases once to mark PEs and re-read only the disagreed PEs. We therefore were not able to evaluate the intra-observer variability for each radiologist. Second, our study only included positive PE cases to evaluate the variability of radiologists in the annotation of lesion locations as reference standard. It is worth noting that the PIOPED II study reported a high specificity of about 96% for PE detection on MDCT. Another study investigated inter-observer agreement [23] suggested that a second radiologist’s reading of a negative examination could be enough to rule out PE. Including negative PE cases, for example, 50 negative cases, may only add about 2 FN cases to our study, which would not significantly change our results. To best utilize the limited time available from our radiologists, we therefore did not include normal (PE negative) cases in the study. Third, we had only two experienced radiologists’ reading to build the reference standard. The variability between the two radiologists and the changes in the reference standard at the various stages of the reading process indicated that the reference standard could be further improved. Ideally, a reference standard in complicated cases such as PE in CTPA should be built with multiple readings by a large number of experienced radiologists and an iterative consensus process. Nevertheless, the purpose of the current study is not to build a perfect reference standard as our resources are limited, but to demonstrate the changes in the performance assessments as the reference standard changes and the process that can improve a reference standard. The eventual comparison of the performance of different lesion detectors (radiologists and/or CAD system) should also be viewed with caution if the reference standard is not adequately scrutinized.

Conclusion

Our study demonstrated that there is substantial variability in reference standards provided by radiologists, which impacts the performance assessment of a lesion detection system. Combination of multiple radiologists’ readings and consensus is needed to improve the reliability of a reference standard.