Highlights
-
•
Direct findings of pulmonary embolism on MRA chest exms had the highest interobserver agreement for vessel cutoff (k-.52, p value- ,.0001).
-
•
Indirect findings for pulmonary embolism on MRA chest had the highest interobserver agreement for pleural effusions (k-.56, p value = .0001).
-
•
There was high interobserver agreement for meaurement of the pulmonary artery and the right ventricle/left ventricle ratio.
Abbreviations: CE-MRA, contrast enhanced magnetic resonance angiography; PE, pulmonary embolism; CTPA, computed tomography pulmonary angiography; SGRE, spoiled gradient recalled echo; ICC, intra class correlation; RV/LV, ratio of the right ventricular to left ventricular minor axis measurements; PA, pulmonary artery
Keywords: Magnetic resonance angiography, Contrast enhanced, Pulmonary embolism, Reader agreement
Abstract
Background
Accurate diagnosis of pulmonary embolism (PE) using contrast enhanced MRA (CE-MRA) requires awareness of both the direct and indirect findings of PE.
Purpose
To evaluate reader agreement of the direct and indirect findings of PE on CE-MRA.
Methods
We evaluated pulmonary artery diameter, right ventricle/left ventricle ratio, and clot/vessel lumen signal intensity ratio. Also, eight direct and eight indirect findings of PE were interpreted twice by two radiologists with different experience levels. The prevalence, and intra- and inter-reader agreement for the direct and indirect findings of PE were recorded. Statistical analysis of the measurements was assessed using intraclass correlation while Cohen’s kappa test determined inter- and intra-reader agreement.
Results
We reviewed 66 positive CE-MRA exams, 10 of which cases were used for training. The largest PE for each of the remaining 56 cases (40 woman) were included in this analysis (38.9 ± 19.7 (mean age (years) ± S.D.)). The highest interobserver agreement for the direct findings were vessel cutoff (κ = 0.52, 95 % CI = (0.30, 0.74), p < .0001) and bright clot (κ = 0.51, 95 % CI = (0.26, 0.78), p = .0001). The highest interobserver agreement for the indirect findings were for atelectasis (κ = 0.67, 95 % CI = (0.49, 0.87), p < .0001), pleural effusions (κ = 0.56, 95 % CI = (0.32, 0.79), p = 0001) and blank slate sing (κ = 0.56, 95 % CI = (0.18, 0.94), p < .0001).
Conclusion
The indirect findings of atelectasis and pleural effusion had better interobserver reproducibility than the direct findings of vessel cutoff and bright clot. The intraobserver reproducibility of the direct and indirect findings is dependent on experience level.
Summary statement
Using contrast enhanced magnetic resonance angiography for the diagnosis of pulmonary embolism, the indirect findings of atelectasis and pleural effusion had better interobserver reproducibility than the direct findings of vessel cutoff and bright clot.
1. Introduction
Pulmonary embolism (PE) is a leading cause of acute cardiovascular death in the world, following only myocardial infarction and stroke in incidence [1,2]. PE is fatal in up to 30 % of affected patients [3,4]. The mortality rate can be decreased to 8% with proper treatment [5,6]. For these reasons, prompt diagnosis and treatment is critical for optimal patient outcomes.
Computed tomography pulmonary angiography (CTPA) is currently the standard of care for the diagnosis of PE [7]. CTPA can be performed and interpreted rapidly and has resulted in its widespread use in the emergency setting. However, some authors have shown that the use of CTPA has resulted in “over diagnosis” of this disorder [8]. Other limitations of CTPA include increasing concerns about the effects of ionizing radiation, particularly for younger women [[9], [10], [11], [12]]. Further, many patients have contraindications to iodinated contrast, including those with renal failure (eGFR < 30) or a history of allergic/anaphylactoid reaction to iodinated contrast material.
Contrast enhanced magnetic resonance angiography (CE-MRA) is increasingly used for the diagnosis of PE and requires neither iodinated contrast nor ionizing radiation. Over the last ten years, we have performed over 10,000 of these examinations for the diagnosis of PE. Accurate interpretation requires that the radiologist be aware of the many direct and indirect findings of PE as well as artifacts associated with these acquisitions. Many of these findings have a correlate on CTPA exams, while others are unique to CE-MRA (See online supplement).
Therefore, the purpose of this study was to determine (a) the reproducibility of routine measurements on CE-MRA exams in patients with PE, (b) differences between readers of varying experience and (c) which direct and indirect signs of PE had the highest intra- and interobserver agreement.
2. Materials and methods
2.1. Study design
This was a HIPAA compliant and IRB-approved retrospective study. We studied consecutive of patients who underwent CE-MRA for the diagnosis of PE from May 2008–May 2014. All patients had CE-MRA performed for the diagnosis of PE were included in the study. These participants were also evaluated as part of a separate publication on outcomes [13], although the current analysis focused on reader agreement. An initial listing of the patient exams for this study was obtained through a search of the radiology picture archiving and communication system (PACS) and radiology information system (RIS). The search included a query for all thoracic CE-MRA scans and their corresponding final reports within the study period. All CE-MRA exams were reported by sub-specialty trained cardiovascular radiologists all with more than six years of experience with CE-MRA interpretation for the diagnosis of PE (SKN, TMG, CJF, SBR, and MLS). All of the cases used in this study were re-reviewed by consensus (MLS, Reader 1, and Reader 2) to confirm the initial clinical report of a positive PE. All exams that were determined to be positive for PE were included in this analysis. (Fig. 1- Flow chart)
Fig. 1.
Participant flow chart for this study. The readers had a joint training session for all of the direct and indirect findings of pulmonary embolism that were enumerated along with practicing the methodology for the measurements of the right ventricle/ left ventricle ratio, main pulmonary artery and the modified method for separating a non-occlusive filling defect from Gibbs’ artifact. (Abbreviations: CE-MRA- contrast enhanced magnetic resonance angiography, PE- pulmonary embolism).
2.2. MRA protocol
The MRA imaging protocol used in this study has been previously published [14]. Briefly this included the following pulse acquisitions: (A) localizer 3-plane single-shot fast spin-echo; (B) pre-contrast, pulmonary arterial phase, immediate delayed-phase, and a low flip angle delayed-phase 3D contrast-enhanced T1 weighted MRA with near isotropic spatial resolution and full chest coverage with an interpolated voxel size of 0.7 × 0.7 × 1.0 mm3; and (C) 2D post-contrast fat-saturated T1-weighted spoiled gradient recalled (SGRE) echo images. For intravenous contrast, a weight-based dose of 0.1 mmol/kg of gadobenate dimeglumine (Multihance™, Bracco Diagnostics, Princeton, NJ) diluted to a total volume of 30 mL with saline and power-injected at 1.5 mL/s followed in 20 mL of normal saline injected at the same rate, was used. This protocol is very simple and requires six separate breath holds of about 17 s with 10–20 second rest periods for a total table time of less than 10 min.
2.3. Analysis
All CE-MRA exams known to have PE, were evaluated independently by two readers. The readers first had a joint training session consisting of ten cases that was supervised by a proctor (MLS) for each of the measurement methods and the various direct and indirect signs of PE on CE-MRA. A pre-evaluation training session for Reader 2 included 20 training cases: 10 CE-MRA positive for PE and 10 negative for PE. Each reader independently assessed each study twice for the direct and indirect findings in order to assess inter and intra reader agreement using intra class correlation (ICC) (Fig. 1). The reading sessions were separated by two weeks apart to minimize any recall bias. Reader 1 had two years of experience interpreting CE-MRA for the presence of PE. Reader 2 had one month of experience interpreting these exams.
First, each reader assessed measurements that are typically made in the setting of PE stratification for severity of disease burden an indirectly the presence of pulmonary hypertension. These included the following: the clot/vessel lumen signal intensity ratio separating PE from the Gibbs artifact [15], the pulmonary artery (PA) diameter and right ventricle/ left ventricle (RV/LV) short axis ratio (See online supplement). ICC was employed to determine intra and inter reader agreement for these measurable variables.
Secondly, each study was assessed for the presence of PE with false positive exams excluded from further analysis. The exam was re-evaluated by the two readers along with an additional more experienced reader (MLS) to reach a consensus about the presence or absence of the PE. The largest PE for each exam determined by consensus was included in the study and assessed for the presence of each of the direct and indirect findings [16] (See online supplement).
The following direct findings of PE were tallied by each reader on two separate occasions (See online supplement): (a) non occlusive filling defect within the enhanced pulmonary artery lumen, (b) occlusive filling defect completely obstructing the pulmonary artery, (c) vessel cutoff on thin-slab maximum intensity projection images and/or on direct coronal CE-MRA exams, (d) double bronchus sign, (e) central dot sign, (f) ghost vessel sign, (g) bright clot sign and (h) a filling defect within the pulmonary artery found on the post gadolinium fat saturated SGRE images (this sequence is employed to survey the chest for any ancillary findings) [16] (Table 1, Fig. 2). The following indirect findings were included (see online supplement): (a) pulmonary venous stasis, (b) atelectasis, (c) perfusion defect, (d) pleural effusion, (e) visceral pleural enhancement, (f) white-black-white sign, (g) pulmonary infarction and (h) blank slate sign [16] (Table 1, Fig. 3). The direct findings of PE only found on MRI/MRA include: double bronchus sign, central dot sign, ghost vessel sign, and bright clot sign [16]. There are also unique indirect findings of PE at CE-MRA include: visceral pleural enhancement, white-black-white sign, and blank slate sign [16]. Thus, each reader reviewed 56 patients (mean age (yrs.) ± (SD) = 37.2 ± 17.8 (40 woman), 43.3 ± 12.8 (16 man)) with 56 pulmonary emboli for the presence or absence of eight indirect and eight direct findings of PE on two separate occasions. In total, for the two readers reading each case twice, there were 3584 data entry fields (56 PE * (8 indirect +8 direct findings) * 2 readers* two reading sessions = 3584 data fields) that were manually entered onto spread sheets.
Table 1.
Definition of direct and indirect findings of pulmonary embolism on MRA.
| Direct findings of PE on MRA | Definition |
|---|---|
| Filling defect in the pulmonary arteries | Low signal intensity clot in the pulmonary arteries after contrast enhancement |
| Vessel cutoff sign | Vessel completely obstructed and amputated on Maximum Intensity Projection images |
| Double bronchus signa | Two low signal intensity structures in cross-section- one representing a bronchus and the other representing the occlusive thrombus |
| Central dot signa | Central high intensity in the clot |
| Ghost vessel signa | Enhancement of a vessel wall surrounding an obstructing embolus on delayed contrast enhances MRA images |
| Bright clot signa | High T1 signal intensity from intraluminal methemoglobin on a pre-contrast image |
| Indirect findings of PE on MRA | |
| Pulmonary venous stasis | Higher signal intensity vein related to slow flow and delayed timing of the peak enhancement time course of contrast through this vein |
| Atelectasis | Strong enhanced lung reflects collapsed lung |
| Perfusion defects | Wedge shaped areas of low signal intensity due to loss of lung parenchymal enhancement |
| Visceral pleural enhancementa | liner enhancement along lung surface |
| White-black-white signa | A perfusion defect surrounded by enhancing lung |
| Pulmonary infarction | High T2 weighted wedge shaped area of lung parenchyma without enhancement |
| Blank slate signa | A large area of completely black lung that has no signal intensity with in it |
Abbreviations: PE-pulmonary embolism, MRA- magnetic resonance angiography.
Findings unique for contrast enhanced MRA.
Fig. 2.
Direct findings of pulmonary embolism (PE) at contrast enhanced pulmonary magnetic resonance angiography (CE-MRA): (A) Axial CE-MRA showing an occlusive filling defect in right pulmonary artery (arrow) and non-occlusive filling defect in left pulmonary artery(dotted arrow); (B) Coronal MRA at 1.25 mm showing vessel cutoff of the truncus anterior (upper arrow) and interlobar (lower arrow) pulmonary arteries by PE; (C) Axial post contrast fat saturated T1-wieghted spoiled gradient recalled echo images showing the “double bronchus sign” -neighboring two low signal intensity oval structures with the medial one (straight arrow) being an occlusive PE in the right pulmonary artery and the lateral one (dotted arrow) the right main bronchus; (D) Axial CE-MRA showing the “central dot” sign high intensity in clot which helps to distinguish clot from a Gibbs’ artifact; (E) Axial delayed phase CE-MRA showing the “ghost vessel” sign wherein there is delayed enhancement of the wall surrounding the occlusive clot, which is likely related to inflammation within the vessel wall secondary to the presence of the thrombus; (F) Coronal pre contrast T1 weighted MRA showing the “bright clot sign”-high intensity PEs (arrow) from methemoglobin of the clot in the left and right lower lobe pulmonary arteries.
Fig. 3.
Indirect signs of pulmonary embolism (PE) found on contrast enhanced pulmonary magnetic resonance angiography (CE-MRA): (A) Coronal CE-MRA showing a large perfusion defect in the right lower lobe as areas of very low signal intensity with a lack of vasculature (arrow, blank slate sign) and a smaller one in the left lower lobe with vessel enhancement (dashed arrow, simple perfusion defect); (B) Coronal CE-MRA showing the “white-black-white” sign of a perfusion defect (within the circle “B”) surrounded on both sides by normally perfused lung (within the circle “W”); (C) Axial post contrast fat saturated spoiled gradient echo image showing a pleural effusion (dashed arrow), compressive atelectasis (straight arrow) and a non-occlusive PE (arrow-PE); (D) Axial CE-MRA showing the enhancement of the right parietal pleural surface (dashed arrow) and PE in the right lower love pulmonary artery; (E) Axial T2-weighted image showing pulmonary infarction detected as a high signal intensity area (arrow); (F) Axial CE MRA of the respective pulmonary vein draining the right lower lobe posterior segment with high signal within the draining vein (dashed arrow) due to a slower transit time when compared to the contralateral left lower lobe pulmonary vein (arrow) without a PE.
2.4. Statistical analysis
Statistical analysis was conducted by a statistician (CL). For the continuous measurements, agreement was assessed via the ICC using the ‘psych’ package in R (V 3.6.1) [17,18]. For the discrete measurements, agreement was assessed via an unweighted kappa statistic using the ‘irr’ package [19]. All point estimates were accompanied by parametrically estimated 95 % confidence intervals (95 % CI) and p-values. Due to the exploratory nature of this study, no multiple testing corrections were applied to calculated p-values. A p-value less than 0.05 was considered statistically significant.
3. Results
Between May 2008 and May 2014 there were 902 CE-MRA studies performed to assess for the presence of PE. Of these studies, 69 were initially interpreted as positive for PE. The image quality was limited for 3 of these exams (4%). Ten studies were used to create a training set and the remaining 56 studies were included in the analysis. Upon consensus review, 29 patients had a single PE, 17 had two PEs and the remaining 10 patients had three or more PEs. There was no false positive PE exam in the 56 studies. Only the largest PE, as determined by consensus for each patient, was analyzed for direct and indirect findings. There were three studies without pre-contrast T1 weighted images, one study without a delayed-phase contrast-enhanced T1 weighted MRA and 19 studies without a post-contrast fat-saturated T1-weighted SGRE acquisition.
3.1. Continuous variables
The ICCs for the three continuous measurable variables described above (clot/vessel lumen signal intensity ratio, RV/LV short axis ratio and the main PA diameter) are shown in Table 2. Bland-Altman plots for reader variability of these variables are shown in Fig. 4. The inter-observer agreement for the RV/LV ratio (ICC = 0.65, 95 % CI = (0.48, 0.78), p < .0001) and main PA diameters (ICC = 0.66, 95 % CI = (0.49, 0.79), p < .0001) were very good with high intra-observer agreement as well.
Table 2.
Intra and Interobserver agreement between two separate readings made by the experienced and novice observer for the presence of three continuous variables in contrast enhanced pulmonary angiography exams know to be positive for pulmonary embolism.
| CE-MRA Imaging Measurement | Number of emboli measured |
ICC | 95 % C.I. | p value |
|---|---|---|---|---|
| Clot/vessel ratio (R1) | 54 | 0.38 | 0.12−0.58 | 0.0022 |
| Clot/vessel ratio (R2) | 54 | 0.31 | 0.06−0.54 | 0.0093 |
| Clot/vessel ratio (Interobserver) | 54 | 0.40 | 0.15−0.60 | 0.0011 |
| Main PA (R1) | 56 | 0.88 | 0.81−0.93 | <.00001 |
| Main PA (R2) | 56 | 0.81 | 0.70−0.89 | <.00001 |
| Main PA (Interobserver) |
56 | 0.66 | 0.49−0.79 | <.00001 |
| RV/LV Ratio (R1) | 56 | 0.88 | 0.80−0.93 | <.00001 |
| RV/LV Ratio (R2) | 56 | 0.78 | 0.65−0.86 | <.00001 |
| RV/LV Ratio (Interobserver) |
56 | 0.65 | 0.48−0.78 | <.00001 |
Abbreviations: R1- Reader one, R2- Reader 2, Intraobserver- Interobserver variability, Clot/vessel ratio- clot/vessel lumen signal intensity ratio, Main PA- transverse measurement of the main pulmonary artery maximum diameter, RV/LV ratio- the modified 4 chamber (straight axial) measurement of right ventricle/left ventricle minor axis ratio, ICC- intraclass correlation coefficient, 95 % C.I.−95% confidence interval about estimated ICC, p value – p-value from subsequent hypothesis test.
Fig. 4.
Bland-Altman plot of reader variability for (a) clot/vessel lumen signal intensity ratio, (b) main pulmonary artery, (c) Right ventricle /Left ventricle ratio. The bias is the average pairwise difference between readers, the p-value comes from a paired t-test, and the range is the difference between the upper and lower confidence limits.
3.2. Direct findings
The Cohen kappa values for the inter- and intra-observer agreement for the eight direct findings of PE at CE-MRA (see Table 1, Fig. 2 and online supplement) for the two readers are shown in Table 3 along with the prevalence of each finding. The two direct findings of PE with the highest inter-observer agreement in the direct findings of PE were vessel cutoff on MRA (κ = 0.52, 95 % CI = (0.30, 0.74), p = .0001) and bright clot sign (κ = 0.51, 95 % CI = (0.26, 0.78), p = .0001). These findings also had a very high intra-reader agreement (vessel cutoff: κ = 0.44, 95 % CI = (0.20, 0.68), p < .001, κ = 0.48, 95 % CI = (0.25, 0.70), p < .0002, respectively) (bright clot sign: κ = 0.67, 95 % CI = (0.45, 0.89), p < .0001, κ = 0.64, 95 % CI = (0.41, 0.86), p < .0001, respectively). The two direct findings with the highest prevalence were the occlusive filling defect (N = 43) and vessel cutoff (N = 42).
Table 3.
Experienced and novice observer agreement for the direct findings of pulmonary embolism found on contrast enhanced pulmonary magnetic resonance angiography.
| Direct Finding of PE at CE-MRA | Consensus Number of findings |
Reader Prevalence time 1, time 2 |
N | kappa | 95 % C.I. | P value |
|---|---|---|---|---|---|---|
| Non-occlusive, R1 | 12 | 19,20 | 56 | 0.88 | 0.75−1.00 | <0.0001 |
| Non-occlusive, R2 | 12 | 14,16 | 56 | 0.46 | 0.19−0.72 | 0.0006 |
| Non-occlusive, Interobserver | 12 | 56 | 0.36 | 0.10−0.62 | 0.0056 | |
| Occlusive, R1 | 43 | 36,35 | 56 | 0.88 | 0.76−1.00 | <0.0001 |
| Occlusive, R2 | 43 | 38,39 | 56 | 0.54 | 0.31−0.78 | <0.0001 |
| Occlusive, Interobserver | 43 | 56 | 0.44 | 0.20−0.69 | 0.0009 | |
| Vessel Cutoff, R1 | 42 | 36,33 | 56 | 0.44 | 0.20−0.68 | 0.001 |
| Vessel Cutoff, R2 | 42 | 31,39 | 56 | 0.48 | 0.25−0.70 | 0.0002 |
| Vessel Cutoff, Interobserver | 42 | 56 | 0.52 | 0.30−0.74 | 0.0001 | |
| Double Bronchus, R1 | 26 | 24,27 | 56 | 0.53 | 0.31−0.75 | 0.0001 |
| Double Bronchus, R2 | 26 | 10,13 | 56 | 0.77 | 0.51−0.95 | <0.0001 |
| Double Bronchus, Interobserver | 26 | 56 | 0.45 | 0.24−0.65 | 0.0001 | |
| Central Dot, R1 | 34 | 21,16 | 56 | 0.32 | 0.07−0.58 | 0.01 |
| Central Dot, R2 | 34 | 32,25 | 56 | 0.54 | 0.33−0.75 | <0.005 |
| Central Dot, Interobserver | 34 | 56 | 0.34 | 0.12−0.57 | <0.0001 | |
| Ghost Vessel, R1 | 31 | 23,17 | 55 | 0.54 | 0.31−0.76 | <0.0001 |
| Ghost Vessel, R2 | 31 | 34,36 | 55 | 0.53 | 0.39−0.83 | <0.0001 |
| Ghost Vessel, Interobserver | 31 | 55 | 0.27 | 0.27−0.69 | 0.0001 | |
| Bright clot, R1 | 18 | 16,13 | 53 | 0.67 | 0.45−0.89 | <0.0001 |
| Bright clot, R2 | 18 | 12,18 | 53 | 0.64 | 0.41−0.86 | <0.0001 |
| Bright clot, Interobserver | 18 | 53 | 0.51 | 0.26−0.78 | 0.0001 | |
| Filling Defect on FS-SGRE, R1 | 28 | 28,31 | 34 | 0.62 | 0.24−1.00 | 0.0001 |
| Filling Defect on FS-SGRE, R2 | 28 | 28,28 | 34 | 0.60 | 0.24−0.95 | 0.0005 |
| Filling Defect, Interobserver | 28 | 34 | 0.19 | −0.20−0.58 | 0.3 |
Abbreviations: PE- pulmonary embolism, CE-MRA- contrast enhanced magnetic resonance angiography, C.I.- confidence interval,R1- reader 1 with 2 years of experience reading CE-MRA, R2- reader 2 with one month of experience reading CE-MRA. FS-SGRE – Fat saturated spoiled gradient echo sequence.
3.3. Indirect findings
The inter- and intra-observer agreement results for the eight indirect findings of PE at CE-MRA (see Table 1, Fig. 3 and online supplement) are summarized in Table 4. The indirect finding with the highest inter-observer agreement was the presence of atelectasis (κ = 0.67, 95 % CI = (0.49, 0.87), p < .0001). The two findings with the next highest inter-observer agreement were the presence of pleural effusion (κ = 0.56, 95 % CI = (0.33, 0.79), p < .0001) and the blank slate sign (κ = 0.56, 95 % CI = (0.18, 0.94), p < .0001). Intra-observer agreement for these variables was also high for these variables (atelectasis: κ = 0.54, 95 % CI = (0.32, 0.76), p < .0001, κ = 0.82, 95% CI = (0.66, 0.97), p < .0001, respectively) (pleural effusion: κ = 0.79, 95% CI = (0.62, 0.97), p < .0001, κ = 0.88, 95% CI = (0.75, 1.0), p < .0001, respectively) (blank slate sign: κ = 0.64, 95% CI = (0.26, 1.0), p < .0001, κ = 1.0, 95% CI = (1.0, 1.0), p < .0001, respectively). The two indirect findings with the highest prevalence were perfusion defects (N = 42) and the white-black-white sign (N = 34).
Table 4.
Experienced and novice observer agreement for the indirect findings of PE found on contrast enhanced pulmonary magnetic resonance angiography.
| Indirect Finding of PE at MRA | Consensus number of findings | Prevalence | N | kappa | 95 % C.I. | P value |
|---|---|---|---|---|---|---|
| Pulmonary venous stasis, R1 | 10 | 11,12 | 56 | 0.40 | 0.11−0.69 | 0.003 |
| Pulmonary venous stasis, R2 | 10 | 3,2 | 56 | 0.37 | −0.18−0.93 | 0.004 |
| Pulmonary venous stasis Inter-reader |
10 | 56 | 0.06 | −0.18−0.31 | 0.5 | |
| Atelectasis, R1 | 21 | 27,32 | 56 | 0.54 | 0.32−0.76 | <0.0001 |
| Atelectasis, R1 | 21 | 24,23 | 56 | 0.81 | 0.66−0.97 | <0.0001 |
| Atelectasis Inter-reader |
21 | 56 | 0.67 | 0.49−0.87 | <0.0001 | |
| Perfusion Defect, R1 | 42 | 39,37 | 56 | 0.59 | 0.37−0.82 | <0.0001 |
| Perfusion Defect, R2 | 42 | 37,41 | 56 | 0.66 | 0.45−0.87 | <0.0001 |
| Perfusion Defect, Inter-reader |
42 | 56 | 0.51 | 0.27−0.75 | 0.0001 | |
| Effusion, R1 | 21 | 17,18 | 56 | 0.79 | 0.62−0.97 | <0.0001 |
| Effusion, R2 | 21 | 20,19 | 56 | 0.88 | 0.75−1.0 | <0.0001 |
| Effusion, Inter-reader |
21 | 56 | 0.56 | 0.32−0.79 | <0.0001 | |
| Pleural Enhancement, R1 | 22 | 10,14 | 56 | 0.47 | 0.20−0.75 | 0.0003 |
| Pleural Enhancement, R2 | 22 | 23,19 | 56 | 0.70 | 0.51−0.88 | <0.0001 |
| Pleural Enhancement, Inter-reader |
22 | 56 | 0.31 | 0.09−0.54 | 0.0058 | |
| White-Black-White, R1 | 34 | 22,14 | 56 | 0.60 | 0.39−0.81 | <0.0001 |
| White-Black-White, R2 | 34 | 20,25 | 56 | 0.67 | 0.47−0.86 | <0.0001 |
| White-Black-White, Inter-reader |
34 | 56 | 0.47 | 0.23−0.71 | 0.0005 | |
| Pulmonary Infarction, R1 | 12 | 11,12 | 54 | 0.94 | 0.84−1.0 | <0.0001 |
| Pulmonary Infarction, R2 | 12 | 6,7 | 54 | 0.74 | 0.46−1.0 | <0.0001 |
| Pulmonary Infarction, Inter-reader |
12 | 54 | 0.52 | 0.22−0.82 | <0.0001 | |
| Blank Slate, R1 | 5 | 4,5 | 56 | 0.64 | 0.26−1.0 | <0.0001 |
| Blank Slate, R2 | 5 | 6,6 | 56 | 1.0 | 1.0−1.0 | <0.0001 |
| Blank Slate, Inter-reader |
5 | 56 | 0.56 | 0.18−0.94 | <0.0001 |
Abbreviations: PE- pulmonary embolism, C.I.- confidence interval, R1- reader 1 with 2 years of experience reading CE-MRA, R2- reader 2 with one month of experience reading CE-MRA.
4. Discussion
In this work we investigated that the high reproducible observations of pulmonary embolism on contrast enhanced magnetic resonance angiography were the presence of a vessel cutoff and bright clot sign for the direct findings, and the presence of atelectasis, pleural effusion and blank slate sign for the indirect findings. The interobserver reproducibility of the indirect findings were better than the direct findings.
For the continuous variables that are well defined CE-MRA measurements, we found that there was good inter- and intra-observer agreement for the main PA diameter and the RV/LV ratio [20]. The clot/vessel lumen signal intensity ratio for separating PE from the Gibbs artifact did not perform quite as well but was still a reproducible metric. This method was originally described by Bannas et al [15], and there is no reproducibility study for this method. For the nominative observations that were studied, high reproducible inter observations associated with PE on CE MRA includes both common findings with CTPA (a vessel cutoff and atelectasis and pleural effusion) and unique findings on CE MRA, with no direct correlate on CTPA (bright clot sing and blank slate sign). Recognition of these new signs may be helpful for the interpreting radiologist, as pulmonary emboli may be very difficult to directly observe at CE-MRA. The intra observer agreement for each direct and indirect finding was different by the experience level of readers. It is not surprising that there is variability in the interpretation of CE-MRA exams for the direct and indirect signs for PE. With respect to prior work in determining reader agreement, Bannas et al showed that in an animal model of PE, there was not the same efficacy [21]. In that study, Reader 1 had a sensitivity of 100% (11/11) and a specificity of 100% (13/13) for PE detection on a per-animal basis for both MR angiography and ultrashort TE [21]. While Reader 2 had a sensitivity of 91% and specificity of and 92% [21]. There is variation in the diagnosis of PE at CTPA as well [22,23]. The correct interpretation of pulmonary MRA examinations for the diagnosis of PE is dependent on experience level and knowledge of the direct and indirect findings unique to this modality. The combined use of direct and indirect findings is key to observing the presence of emboli on this exam. These data confirm the need for appropriate training for accurate interpretation of PE using the CE-MRA.
There are limitations to this study. First, the 10 training cases were chosen randomly and may not have provided clear cut examples of each finding making it more challenging for the novice reader to become comfortable with less familiar findings. Second, additional training of the novice reader prior to beginning this observational study may have been helpful. Third, the ground truth for this study was the final radiology report and a consensus read, however, there was not a second confirmatory imaging exam showing the presence of PE. Related to this, it is also likely that some patient with a true PE, but a negative CE-MRA exam were not included in this study. Fourth, the readers were instructed to choose the largest embolus when there was more than one present on an exam. It is possible that in some cases the two readers did not pick the same embolus for their observations and clot/vessel lumen signal intensity ratio measurement. Fifth, additional readers would have been helpful to more completely determine the intra- and inter-reader variabilities for these findings.
In summary, (a) The reproducibility of the measurable quantities (main PA diameter, RV/LV ratio and clot/vessel lumen signal intensity ratio) on contrast enhanced magnetic resonance angiography exams in patients with pulmonary embolism was good, (b) The intraobserver agreement of the direct and indirect findings was dependent on experience level. and (c) The high interobserver agreements were shown in vessel cutoff and bright clot for the direct findings, atelectasis, pleural effusion and blank slate sign for the indirect findings.
Footnotes
Supplementary material related to this article can be found, in the online version, at doi:https://doi.org/10.1016/j.ejro.2020.100256.
Appendix A. Supplementary data
The following are Supplementary data to this article:
References
- 1.Sadigh G., Kelly A.M., Cronin P. Challenges, controversies, and hot topics in pulmonary embolism imaging. AJR Am. J. Roentgenol. 2011;196:497–515. doi: 10.2214/AJR.10.5830. [DOI] [PubMed] [Google Scholar]
- 2.CADTH Optimal Use Reports . Canadian Agency for Drugs and Technologies in Health. Copyright (c) 2016 CADTH; Ottawa (ON): 2016. Optimal Strategies for the Diagnosis of Acute Pulmonary Embolism: A Health Technology Assessment - Project Protocol. [PubMed] [Google Scholar]
- 3.Nikolaou K., Thieme S., Sommer W., Johnson T., Reiser M.F. Diagnosing pulmonary embolism: new computed tomography applications. J. Thorac. Imaging. 2010;25:151–160. doi: 10.1097/RTI.0b013e3181d9ca1d. [DOI] [PubMed] [Google Scholar]
- 4.Carson J.L., Kelley M.A., Duff A., Weg J.G., Fulkerson W.J., Palevsky H.I., Schwartz J.S., Thompson B.T., Popovich J., Jr., Hobbins T.E., Spera M.A., Alavi A., Terrin M.L. The clinical course of pulmonary embolism. N. Engl. J. Med. 1992;326(19):1240–1245. doi: 10.1056/NEJM199205073261902. [DOI] [PubMed] [Google Scholar]
- 5.Tapson V.F. Acute pulmonary embolism. N. Engl. J. Med. 2008;358(10):1037–1052. doi: 10.1056/NEJMra072753. [DOI] [PubMed] [Google Scholar]
- 6.Kearon C., Akl E.A., Comerota A.J., Prandoni P., Bounameaux H., Goldhaber S.Z., Nelson M.E., Wells P.S., Gould M.K., Dentali F., Crowther M., Kahn S.R. Antithrombotic therapy for VTE disease: antithrombotic therapy and prevention of thrombosis, 9th ed: American College Of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest. 2012;141(2 Suppl):e419S–494S. doi: 10.1378/chest.11-2301. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Stein P.D., Fowler S.E., Goodman L.R., Gottschalk A., Hales C.A., Hull R.D., Leeper K.V., Jr., Popovich J., Jr., Quinn D.A., Sos T.A., Sostman H.D., Tapson V.F., Wakefield T.W., Weg J.G., Woodard P.K., PIOPED II Investigators Multidetector computed tomography for acute pulmonary embolism. N. Engl. J. Med. 2006;354(22):2317–2327. doi: 10.1056/NEJMoa052367. [DOI] [PubMed] [Google Scholar]
- 8.Burge A.J., Freeman K.D., Klapper P.J., Haramati L.B. Increased diagnosis of pulmonary embolism without a corresponding decline in mortality during the CT era. Clin. Radiol. 2008;63:381–386. doi: 10.1016/j.crad.2007.10.004. [DOI] [PubMed] [Google Scholar]
- 9.Brenner D.J., Hall E.J. Computed tomography—an increasing source of radiation exposure. N. Engl. J. Med. 2007;357(22):2277–2284. doi: 10.1056/NEJMra072149. [DOI] [PubMed] [Google Scholar]
- 10.Hall E.J., Brenner D.J. Cancer risks from diagnostic radiology: the impact of new epidemiological data. Br. J. Radiol. 2012;85(1020):e1316–1317. doi: 10.1259/bjr/13739950. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Pearce M.S., Salotti J.A., Little M.P., McHugh K., Lee C., Kim K.P., Howe N.L., Ronckers C.M., Rajaraman P., Sir Craft A.W., Parker L., Berrington de González A. Radiation exposure from CT scans in childhood and subsequent risk of leukaemia and brain tumours: a retrospective cohort study. Lancet. 2012;380(9840):499–505. doi: 10.1016/S0140-6736(12)60815-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Mathews J.D., Forsythe A.V., Brady Z., Butler M.W., Goergen S.K., Byrnes G.B., Giles G.G., Wallace A.B., Anderson P.R., Guiver T.A., McGale P., Cain T.M., Dowty J.G., Bickerstaffe A.C., Darby S.C. Cancer risk in 680,000 people exposed to computed tomography scans in childhood or adolescence: data linkage study of 11 million Australians. BMJ. 2013;346:f2360. doi: 10.1136/bmj.f2360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Repplinger M.D., Nagle S.K., Harringa J.B., Broman A.T., Lindholm C.R., Francois C.J., Grist T.M., Reeder S.B., Schiebler M.L. Clinical outcomes after magnetic resonance angiography (MRA) versus computed tomographic angiography (CTA) for pulmonary embolism evaluation. Emerg. Radiol. 2018;25:469–477. doi: 10.1007/s10140-018-1609-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Schiebler M.L., Nagle S.K., François C.J., Repplinger M.D., Hamedani A.G., Vigen K.K., Yarlagadda R., Grist T.M., Reeder S.B. Effectiveness of MR angiography for the primary diagnosis of acute pulmonary embolism: clinical outcomes at 3 months and 1 year. J. Magn. Reson. Imaging. 2013;38:914–925. doi: 10.1002/jmri.24057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Bannas P., Schiebler M.L., Motosugi U., Francois C.J., Reeder S.B., Nagle S.K. Pulmonary MRA: differentiation of pulmonary embolism from truncation artefact. Eur. Radiol. 2014;24:1942–1949. doi: 10.1007/s00330-014-3219-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Schiebler M.L., Benson D., Schubert T., Francois C.J. Noncontrast and contrast-enhanced pulmonary magnetic resonance angiography. In: Kauczor H.U., Wielpütz M.O., editors. MRI of the Lung. Medical Radiology. Springer; 2017. pp. 21–52. doi: 10.1007/174_2017_57. [Google Scholar]
- 17.W. Revelle . psych: Procedures for Personality and Psychological Research, Northwestern University Evanston, Illinois, USA https://CRAN.R-project.org/package=psych.
- 18.R Core Team . R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing Vienna, Austria https://www.R-project.org/.
- 19.Matthias Gamer, Jim Lemon, Ian Fellows Puspendra Singh. irr: Various Coefficients of Interrater Reliability and Agreement. https://CRAN.R-project.org/package=irr. Accessed October 28, 2019.
- 20.Kang D.K., Ramos-Duran L., Schoepf U.J., Armstrong A.M., Abro J.A., Ravenel J.G., Thilo C. Reproducibility of CT signs of right ventricular dysfunction in acute pulmonary embolism. AJR Am. J. Roentgenol. 2010;194:1500–1506. doi: 10.2214/AJR.09.3717. [DOI] [PubMed] [Google Scholar]
- 21.Bannas P., Bell L.C., Johnson K.M., Schiebler M.L., François C.J., Motosugi U., Consigny D., Reeder S.B., Nagle S.K. Pulmonary embolism detection with three-dimensional ultrashort echo time MR imaging: experimental study in Canines. Radiology. 2016;278:413–421. doi: 10.1148/radiol.2015150606. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Liang J., Gotway M.B., Terzopoulos D., Sostman H.D. Interobserver agreement in the diagnosis of acute pulmonary embolism from computed tomography pulmonary angiography and on the effectiveness of computer-aided diagnosis. Am. J. Emerg. Med. 2011;29:465–467. doi: 10.1016/j.ajem.2010.12.030. [DOI] [PubMed] [Google Scholar]
- 23.Costantino G., Norsa A.H., Amadori R., Ippolito S., Resta F., Bianco R., Casazza G., Biagiotti S., Rusconi A.M., Montano N. Interobserver agreement in the interpretation of computed tomography in acute pulmonary embolism. Am. J. Emerg. Med. 2009;27:1109–1111. doi: 10.1016/j.ajem.2008.08.019. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.