Inter-observer variability of visual analysis of “stress”-only adenosine first-pass myocardial perfusion imaging in relation to clinical experience and reading criteria

Abstract

To assess the inter-observer agreement of adenosine “stress”-only visual analysis of perfusion MR images in relation to experience and reading criteria. 106 adenosine perfusion MR examinations out of 350, 46 consecutive positive examinations and 60 randomly selected negative examinations were visually analysed by three individual readers (two residents and a technician) with different levels of experience. Readings (blinded for any information) were compared with the reading of an expert radiologist. After a month the examinations were presented again (randomly) without knowledge regarding the first readings. This time readings were performed with the systematical use of reading criteria. Agreement with the expert reading was good for the most experienced resident (k = 0.88). Kappa was 0.48 for the least experienced, and 0.57 for the technician. After the second systematical reading inter-observer agreement increased to 0.9, 0.68 and 0.77 respectively. Overall kappa increased from 0.59 to 0.71. The use of reading criteria significantly improved the performance of the least experienced reader (P = 0.01). Visual analysis of adenosine “stress”-only first-pass perfusion MR images has moderate to very good agreement. Performance is experience related, but the systematic use of reading criteria significantly increased performance for the least experienced observer.

Introduction

Adenosine stress first-pass perfusion imaging is increasingly used for the detection of myocardial ischemia. Diagnostic accuracies in recently published papers are good [1–7]. There is however still considerable heterogeneity in used clinical protocols. Among others the use of (semi-) quantitative analysis or a more straightforward visual reading, which is more often used in clinical routine. Previous studies have shown that quantitative analysis of perfusion imaging and a visual reading both have a similar good correlation to coronary angiography (CAG) [8–11]. A visual reading in this respect is more straightforward and with currently possible resolutions it might also be easily performed by less experienced observers. There is however wide variability in used and proposed criteria for visual assessment of perfusion abnormalities [1, 2, 4–7, 10, 12–24]. Ranging from no pre-defined criteria to in part contradictory criteria.

The assessment of adenosine “stress” perfusion imaging only, for a visual reading, has a number of advantages. Imaging time is reduced as well as analysis time. There is no waiting time between stress and rest perfusion, and a second bolus of contrast media can be avoided. Therefore no influence of the first contrast administration on the signal intensity of the second perfusion images exists. Using only stress perfusion imaging also eliminates the necessity to spatially match stress and rest perfusion imaging. Unfortunately perfusion imaging may suffer from artefacts, which may sometimes resemble ischemic perfusion defects [25].

The aim of this study is to assess the inter-observer variability of a visual reading of adenosine stress-only perfusion imaging. Additionally the impact of experience and the use of systematic reading criteria was assessed.

Materials and methods

Patient population

Forty-six consecutive patients with a positive adenosine first-pass myocardial perfusion MR examination, together with 60 randomly selected patients with a negative adenosine first-pass myocardial perfusion MR examination were included. These patients were selected out of 350 patients that were referred to our institution between January 2005 and May 2007.

The studied population consisted of 59 men and 47 women; mean age 61.2 ± 9.9 years. All patients had a clinical necessity to exclude myocardial ischemia and patients did not have a prior myocardial infarction. Exclusion criteria were: patients with an acute coronary syndrome, atrial fibrillation, severe arterial hypertension (>220/120), CMR-incompatible metallic implants, known claustrophobia, asthma, chronic obstructive pulmonary disease and patients using dipyridamol.

MR imaging protocol

All anti-anginal medication was stopped 4 days before the adenosine perfusion MR examination. After the patient was positioned on the scanning table, intravenous access was established via an anticubital vein. Scanning was performed at 1.5 T using a magnetom Avanto MRI system (Siemens Medical Solutions, Erlangen, Germany). ECG monitoring leads, a 12 channel phased-array surface coil covering the heart and a brachial blood pressure cuff were applied. A single lead ECG was continuously monitored on the MRI-console. Systolic and Diastolic blood pressures were recorded using an automatic device (Welch-Allyn, Emro-medical) at baseline and during adenosine infusion. Blood pressure and heart rate were recorded.

After 3 min of adenosine infusion (0.140 mg/kg/min) during the first pass of 0.1 mmol/kg gadopentetate dimeglumine Omniscan^® with a flow rate of 5 ml/s flushed with 15 ml 0.9% NACL (flow rate 5 ml/s) a perfusion sequence, with typical parameters from the beginning of the study, was started: TrueFisp: TR, 157.25; TE 0.98 ms; TI 100 ms; α 50°; FOV 300 × 300; slice-thickness 8 mm; matrix 76 × 128; iPAT 2. or TSENSE: TR, 170.34 ms; TE 0.99 ms; TI 110 ms; α 50°; FOV 300 × 300; slice-thickness 8 mm; matrix 76 × 128; iPAT 2. Leading to an acquisition of three short-axis slices with single heartbeat temporal resolution. During the examination a radiologist and a cardiologist were present in the MR suite, to monitor the condition of the patient and to evaluate the images directly.

Image analysis

Perfusion series were visually analysed by an experienced radiologist, using a 16 segment model. Defining a relevant perfusion defect as a perfusion abnormality in at least two segments at consecutive planes of the left ventricle or one segment of the most apical slice (when present in the LCX distribution area).

The selected examinations were archived and viewed on a dedicated workstation (ViewPro versie 3.2.0.12, Rogan Delft, Veenendaal, the Netherlands). The examinations were anonimized and randomized. Three observers with different levels of experience performed a visual reading of the examinations (two residents, one with two years, one with two months of experience did a visual reading, and a reading was performed by a technician). The observers were fully blinded to clinical information, CAG results and adenosine stress MR related information. The observers had to state whether there was a perfusion abnormality indicative of myocardial ischemia. The observers were blinded to the results obtained by the other observers. Furthermore studies were presented to the observers in a different, random order. No prior joint training session was organized. All individual readings were compared to the expert reading. After the first reading all examinations were presented again, this time using all the systematic reading criteria as proposed in the literature in integrated fashion [1, 2, 4–7, 10, 12–24], see Table 1, Readers were kept uninformed regarding there results and there was at least one month between both readings.

Table 1.

Important reading criteria as proposed in the literature

• Perfusiondefect (PD) more than 1/3 of wall thickness (more than subendocardium)

• At least two neighbouring segments involved

• >5 heartbeats after maximum signal intensity in LV cavity

• PD definetely darker than surrounding myocardium

• >3 heartbeats after peak enhancement of most normal appearing region

• PD is region of interest with lowered peak signal intensity

• Focal region of myocardium with lowered contrast enhancement

• PD in at least two segments

• PD more than 50% of wall thickness

• 4 point scale (0 normal; 1 probably normal; 2 probably abnormal; 3 abnormal)

• Hypo-enhancement in coronary flow areas

• Ischemic PD does not fluctuate in signal intensity

• Lowered signal intensity in at least one segments

• Perfusion defects persits beyond the point of peak enhancement

Statistical analysis

Summary values are expressed as mean with standard deviation.

Agreement was measured between readers and consensus reading using Cohen’s Kappa. Fleiss kappa [26] was used to evaluate overall agreement. Grading of Kappa values was set at poor for 0–0.2; fair for 0.21–0.41; moderate for 0.41–0.6; good for 0.61–0.8 and very good for 0.81–1.0. Comparison for statistical significance between both readings for all observers using the McNemar test (P < 0.05). Data analysis was performed using SPSS 14.0 for windows and R (version 2.5.0) for Fleiss kappa calculations.

Results

Clinical and haemodynamic data of all patients is presented in Table 2. Mean heart rate at the time of adenosine stress perfusion imaging was 88.3, compared to 75.3 in rest. Systolic blood pressure at the time of adenosine perfusion imaging is 146.1, compared to 152.4 in rest. This being a normal and expected reaction to the adenosine, making it diagnostic studies.

Table 2.

Clinical and haemodynamic data

Variable	Mean or %
Age, years	61.2 ± 9.94
Male, %	56
Body weight, kg	77.4 ± 14.3
Resting diastolic blood pressure	87.0 ± 11.1
Adenosine diastolic blood pressure, mmHg	83.4 ± 10.3
Resting systolic blood pressure, mmHg	152.4 ± 25.7
Adenosine systolic blood pressure, mmHg	146.1 ± 22.4
Resting heart rate, bpm	75.3 ± 16.8
Adenosine heart rate, bpm	88.3 ± 16.5

Values are expressed as mean ± SD or percentage

Overall Average Fleisch kappa coefficient of reading adenosine stress-only perfusion images, regarding the question if there is a perfusion abnormality suggestive of myocardial ischemia was 0.59 for all readers and average Cohen’s kappa was 0.64. Individual kappa values were 0.88 for the most experienced resident, 0.48 for the less experienced resident and 0.57 for the technician.

Then a second reading was performed, this time with the systematical use of all the reading criteria as proposed in the literature, Table 1. Overall Fleiss kappa value after this reading was 0.71 Individual readings: 0.90 for the most experienced resident, 0.68 for the less experienced and 0.77 for the technician. In total 30 mismatches were present out of 318 readings.

Figures 1, 2, 3 illustrates different scenarios of agreement on visual analysis of adenosine stress first-pass perfusion imaging.

Fig. 1.

Three images from a basal short-axis perfusion run. With contrast arrival in the LV cavity on the left. Myocardial enhancement in the middle image and washout in the last image. Clear perfusion defect in the lateral wall identified correctly by all observers

Fig. 2.

Normal Myocardial perfusion identified correctly by all observers despite small artefacts

Fig. 3.

Adenosine-stress first pass perfusion images with discrepancy between readers. Image with motion artefact and some what larger susceptibility artefact, occuring early

Comparison between the first and second reading of all observers showed no significant difference for the most experienced observer (P = 1.00) and the technician (P = 0.186). The least experienced observer however significantly improved (P = 0.01).

Discussion

The main finding of this study is that visual assessment of adenosine stress-only first-pass myocardial perfusion imaging has moderate to very good agreement. This agreement is experience related and increases with the systematical use of reading criteria for less experienced observers. More experienced readers probably already use more of the criteria published in the literature that help differentiate between real perfusion defects and artefacts. This does imply that visual analysis of adenosine stress-only images is easily learned and may help increase clinical implementation. To the best of our knowledge this is the first study to specifically look at the inter-observer variability of adenosine stress-only first pass myocardial perfusion imaging.

Standardized visual reading criteria need to be set, due to the only moderate overall agreement. This is supported by the fact that agreement increases considerably after the second reading using the proposed criteria to an overall good agreement. This, for instance, compares quite favorably with other screening methods, like for instance mammography [27–29].

Some proposed reading criteria presented in the literature may not always be applicable, for instance defining that a perfusion defects indicative of myocardial ischemia has to be present in more than one segment fails to detect a significant distal stenosis of the left circumflex (LCX). Similar results have been found in nuclear stress perfusion imaging [30].

This study was set-up for relatively less experienced readers to little experience. It can be assumed that this lowered overall agreement. However for an examination to be easily implemented into routine clinical practice and more widespread acceptance an examination also needs to be interpretable for less experienced readers. With the use of specific reading criteria and training by more experienced readers it can be assumed that overall agreement will increase.

Delayed contrast enhancement imaging (DCE) was not routinely performed. We believe stress-only analysis should be reserved for patients with unknown CAD, and no previous myocardial infarctions. As proposed by Klem et al. [5], image analysis should when, DCE clinically indicated, begin with DCE images. In other patient categories, like the one studied in this paper, image analysis should start with the adenosine stress first-pass perfusion imaging, since this will save unnecessarily prolongation of the examination for patients. Stress perfusion analysis can in this regard be used as an arbiter for additional series. If normal, with noticeable effect of the adenosine, a rest perfusion is unnecessary.

Results from the second reading may be influenced by a general learning effect, to minimize this effect we used a relatively large data set and there was at least one month between both readings. Images were presented in a different order on both occasions. Furthermore observers were kept uninformed regarding their results of the first reading.

In conclusion, with relative little experience in the visual interpretation of stress perfusion MR images there is only moderate overall agreement, the use of systematic reading criteria considerably increases agreement with an expert reading. This will make the examination available for less experienced readers.