Inter- and Intra-Observer Repeatability of Aortic Annulus Measurements on Screening CT for Transcatheter Aortic Valve Replacement (TAVR): Implications for Appropriate Device Sizing

Abstract

Objectives:

To investigate intra- and inter-observer repeatability of aortic annulus CT measurements for transcatheter aortic valve replacement (TAVR) by readers with different levels of experience and evaluate the impact of different multi-reader paradigms to improve prosthesis sizing.

Methods:

82 TAVR screening CTAs were evaluated twice by three raters with six (R1=radiologist), three (R2=3D-laboratory technician) or zero (R3=medical student) years of experience. Results were translated into hypothetical TAVR size recommendations. Intra- and inter-observer repeatability between single readers and three different multi-reader paradigms ([A]: two readers, [B]: three readers, or [C]: two readers + an optional third reader) were evaluated.

Results:

Intra-observer variability did not differ significantly (range: 50.1-67.8mm²). However, we found significant differences in mean inter-observer variance (p=0.001). Multi-reader paradigms led to significantly increased precision (lower variability) for scenarios [B] and [C] (p=0.03, p<0.05). Compared to single readers, all multi-reader strategies clearly lowered the rate of discrepant device size categorization between repeated measurements (22-26% to 5-10%).

Conclusions:

Aortic annulus CT measurements for TAVR are highly reproducible. Multi-reader strategies provide higher precision than evaluations from single readers with different levels of experience and could effectively be implemented with two readers and an optional third reader (Paradigm C) in a clinical setting.

Introduction

Transcatheter aortic valve replacement (TAVR) is an established method for treatment of intermediate and high surgical risk patients suffering from symptomatic critical aortic stenosis [1-3] and has been shown to be safe, efficient and non-inferior to conventional open surgery in recent prospective multicenter trials [4, 5]. Accurate assessment of the aortic root and heart valve sizing are crucial to minimize post-procedural complications, above all paravalvular aortic regurgitation, which is associated with increased morbidity and mortality [6-8]. Three-dimensional (3D) imaging techniques, such as transthoracic echocardiography (TEE) and computed tomography (CT), in particular, provide more comprehensive information on the complex aortic root and annulus anatomy and geometry, thus allowing for more accurate prosthesis sizing. Recently, the use of CT has been shown to reduce the incidence of paravalvular regurgitation after TAVR compared to echocardiography [9, 10]. Therefore, CT has been adopted as an essential part of TAVR intervention planning as it enables simultaneous and objective assessment of the aortic root and the most suitable access route, guiding prosthesis selection [11].

Although (semi-) automatic approaches for measuring the aortic annulus have been introduced in recent years [12-14], manual evaluations remain the widely used assessment standard. Yet, limited data exist on the intra- and inter-observer variability of TAVR CT measurements made by observers with different levels of experience or the impact of different multi-reader strategies on the variability of the measurements.

The aims of our study were to evaluate inter- and intra-reader variability of aortic annulus measurements on screening CT for TAVR made by readers with different levels of experience, assess the consequences for device sizing, and evaluate the impact of different multi-reader strategies to propose an improved, practical evaluation algorithm that could increase the precision of annulus sizing for TAVR treatment planning.

Materials and Methods

Study cohort

This retrospective Health Insurance Portability and Accountability Act (HIPAA)-compliant cohort study was approved by the local institutional review board. A total of 86 consecutive patients (28 females, 58 males; mean age: 79±13.4 years; mean BMI: 29±6.7) who successfully underwent TAVR screening CT-angiography (CTA) between 12/2011-05/2013 were evaluated.

CT data acquisition

All 86 CT datasets were acquired on a 64-slice multi-detector CT (MDCT) scanner (LightSpeed VCT64, GE Healthcare, Waukesha, WI). None of the patients received additional beta blockade before the CT scan. The TAVR screening CT protocol consisted of two scans with separate contrast injections: 1) A non-gated CTA of the abdomen and pelvis after the injection of 150 ml iodixanol 320 (Visipaque, GE Healthcare Inc. Princeton, NJ) and a 50 ml saline chaser at a flow rate of 5 ml/s into the right antecubital vein and 2) a retrospectively electrocardiogram (ECG)-gated CTA of the thorax. For the thorax CTA, 60 ml of iodixanol 320 were injected at a flow rate of 5 ml/s, followed by 50 ml of saline at the same rate. Thorax CTA scans were started manually when pulmonary artery and ascending aorta visually appeared equally bright in the fluoro-scan at the level of the carina.

Further scan parameters for the thorax CTA were: 0.35 s rotation time, 40 mm collimation, 120 kVp tube voltage, 600 mAs tube current, no tube current modulation. Reconstructions of the heart from the thorax CTAs were performed with a slice thickness and an increment of 0.625 mm, a medium smooth reconstruction kernel (GE Standard), field of view (FOV): 32 cm, adaptive statistical iterative reconstruction (ASIR): SS30, and time increments of 10% steps through the cardiac cycle. For further evaluation, 3D multiplanar reconstructions (MPRs) of the cardiac CTA datasets were created at an offline post-processing workstation without semi-automated TAVR package (Vitrea 3D software, Vital images, Minnetonka, MN).

Image quality ratings

Image quality of all subjects with regards to motion artifacts and image contrast was assessed subjectively by one reader (S.N.) to identify non-diagnostic datasets.

Assessment of aortic annulus plane and area measurements

Cardiac CTA datasets were evaluated in a randomized blinded fashion by three independent readers, a fellowship-trained cardiothoracic radiologist (S.N.) with six years of experience and a 3D-laboratory technician (C.B.) with three years of experience performing quantitative pre-TAVR measurements, as well as a medical student (S.S.) without experience. Before performing measurements on the TAVR datasets, the student received initial training on how to measure the aortic annulus area on 15 separate cardiac CTAs that were not included in the subsequent analysis.

In accordance with state-of-the-art recommendations at that time [15], the aortic annulus plane was identified manually by creating double-oblique multiplanar reconstructions through the most caudal hinge points of all three aortic cusps (Figure 1).

Figure 1.

Aortic annulus measurements. The aortic annulus plane was identified manually by creating double-oblique multiplanar reconstructions (A and B) through the most caudal hinge points in of all three aortic cusps. Free-form areas were measured during visually determined systole by manually free-hand tracing the aortic valve annulus while excluding all calcifications (C).

Annulus areas were measured during systole by manually free-hand tracing the aortic valve annulus while excluding all calcifications (Figure 1).

The systolic phase was chosen individually by each reader through visual inspection and identification of the cardiac phase that resulted in the smallest left ventricle volume.

Since dose modulation was turned off for these acquisitions, the radiation dose (and therefore noise) was the same for each cardiac phase. If gating artifact compromised the annular measurement on the best systolic phase, the reader was allowed to make annular measurements on the adjacent phase with the least amount of artifact. This approach was chosen because it best reflects actual clinical practice.

To allow comparison with previously published studies, we additionally calculated the area derived effective diameter during diastole using the following formula:

$$D_{\mathit{eff}}=2\sqrt{\frac{\text{annulus area}}{\pi }}$$

Where the diastolic annulus area was determined in the same way as the systolic annulus area, by manually free-hand tracing the aortic valve annulus in the visually determined diastolic phase, while excluding all calcifications.

Repeated measurements were performed by all three readers at least two weeks after the first measurements in a separately randomized order to ensure blinding to the initial measurements.

Statistical analysis

Ordinal variables of image quality results are described by frequencies and percentages. Subjects with non-diagnostic image quality due to either motion artifacts or contrast bolus timing were excluded from all further analysis.

Depending on the form of distribution, which was assessed by the Shapiro-Wilk test, overall inter-observer variance of the mean across readers was tested using either one-way repeated measures ANOVA or the Friedman test, respectively, as a non-parametric alternative. Pairwise intra- and inter-reader comparisons were performed using the paired Student t-test for continuous parametric variables or the Wilcoxon-Signed-Ranks test (with Bonferroni correction, α=0.05) as a non-parametric alternative.

To assess the inter- and intra-reader variability of annular measurements, we performed Bland-Altman analyses, including calculations of bias and 95% limits of agreement (LOA) along with their respective 95% confidence intervals (CI). Narrow limits of agreement indicate a high precision (or low variability). For evaluation of inter-reader variability, values from the 1^st and 2^nd measurements were treated as independent variables.

Analysis of the impact on device size decision-making from systolic aortic annulus measurements between single readers or several alternative multiple reader paradigms was done in Microsoft Excel 2016. A total of three different multi-reader paradigms were assessed: (A) Averaging the measurements of the two best single readers with the highest precision, (B) averaging the measurements of all three single readers, and (C) like paradigm-A but including the third reader (like paradigm-B) if the two best readers differed in their device decision.

The balloon-expandable Edward Sapien 3 device (Edwards Lifesciences, Irvine, CA, USA) served as an example to demonstrate the potential impact of the different possible scenarios. The overlap in area between the different device sizes, as specified by the vendor, was disregarded for the purposes of this study by using the average in overlap as a hypothetical cut-off between two consecutive device sizes (Table 1). All further statistical analyses were performed using SPSS (IPSS Inc., version 23.0, IBM, Chicago, IL). Two-sided p-values <0.05 were regarded as statistically significant.

Table 1:

Hypothetical device sizing, based loosely on Edward Sapien 3.

Device size	Range from-to [mm2]
too small	1.0 - 273
20 mm	273 - 341.5
23 mm	341.5 - 430
26 mm	430 - 543
29 mm	543 - 683
too big	683 - 1000

Results

Image quality

None of the exams were excluded due to the presence of severe motion artifacts. However, non-diagnostic image contrast resulted in the exclusion of a total of four subjects. Therefore, only 82 out of 86 subjects were included in the subsequent analysis.

Intra-reader variability

Due to technical factors that resulted in the loss of two CT datasets, the technician was only able to evaluate 80 subjects. Both, radiologist and student evaluated 82 subjects.

Table 2 summarizes the results for intra-, inter-, and multi-reader Bland-Altman statistics, including bias and 95% LOA. Figure 2 shows the associated intra-reader Bland-Altman plots. The Bland-Altman analysis revealed no significant bias between measurements made by a single reader (radiologist: −1.5 mm², p=0.61; technician: 3.8 mm², p=0.32; student: −5.7 mm², p=0.13, Figure 2. The radiologist was the best single reader with a precision of ±50.1 mm² [95% CI: 40.4, 59.7 mm²], followed by the student (±64.9 mm² [95% CI: 52.4, 77.4 mm²]), and the technician (±67.8 mm² [95% CI: 51.5, 81 mm²]), Figure 2. However, these differences were not statistically significant (radiologist vs. technician p=0.14; radiologist vs. student p=0.20; overlap of 95% CI in Figure 3).

Table 2:

Intra-, inter-, and multi-reader reliability. The radiologist and medical student had the lowest measurement variability (highest precision) and were therefore the two best single readers. Confidence interval (CI). Limits of agreement (LOA).

	Intra-Reader Agreement			Inter-Reader Agreement			Multi-Reader Agreement
	Radiologist	3D Tech	Med. Student	Radiologist vs. 3D Tech	Radiologist vs. Med. Student	3D Tech vs. Med. Student	Average of 2 best readers	Average of 3 readers	Only if R1≠R2 in device decision, take average of all 3 readers
Bias (mm2) [95% CI] p-value	−1.5 [−7.1, 4.1] p=0.61	3.8 [−3.9, 11.5] p=0.32	−5.7 [−12.9, 1.6] p=0.13	−15.0 [−20.3, −9.7] p=0.001	5.7 [0.6, 10.7] p=0.13	−9.3 [−15.3, −3.4] p=0.001	−3.6 [−7.9, 0.8] p=0.11	−1.1 [−4.7, 2,5] p=0.54	−3.7 [−7.4, 0.1] p=0.06
95% LOA (±1.96 s) [95% CI] (mm2)	±50.1 [40.4, 59.7]	±67.8 [51.5, 81.0]	±64.9 [52.4, 77.4]	±66.6 [57.5, 75.7]	±64.4 [55.7, 73.1]	±74.4 [64.3, 84.5]	±38.9 [31.4, 46.4]	±31.8 [25.6, 38.0]	±33.5 [27.0, 40.0]

Figure 2.

Bland-Altman plots, illustrating intra-observer differences for first and second measurements of single readers (upper row), inter-observer differences (middle row), and differences between first and second measurements for the different multi-reader paradigms.

Figure 3.

Single and multiple readers’ precision with mean (dot), upper and lower 95% limits of agreement (whiskers) and their 95% confidence intervals (CI, grey shadowed areas). There is no significant difference in precision (no overlap of 95% CI) between the best single reader (radiologist) and the averaged results of the two best single readers (Average (2)=radiologist and student). However, precision improves significantly (no overlap of 95% CI) when averaging the results of all three readers (Average (3)) or when a third reader is only added in case of discrepant device size decisions of the first two readers (If R1≠R2, then Average (3)). The plot also illustrates the possible impact of single or multiple readers’ precision (or variability) on choosing a certain device size (here loosely based on the Edward Sapien 3 aortic prosthesis).

Inter-reader variability

Results of the Friedman test showed an overall significant difference in mean inter-observer variance across readers (p=0.001). Further pairwise comparison between the average measurements of the readers revealed a significant bias between the radiologist and the technician (−15.0 mm², p=0.001; LOA: ±66.6 mm² [95% CI: 57.5, 75.7 mm²]), as well as the student and the technician (−9.3 mm², p=0.001; LOA: ±74.4 mm² [95% CI: 64.3, 84.5 mm²]), whereas there was no significant bias between the radiologist and the student (5.7 mm², p=0.13; LOA: ±64.4 mm² [95% CI: 55.7, 73.1 mm²]), (Table 2, Figure 2).

Inter-observer variability based on area derived diameter

The evaluation of the area-derived effective diameter obtained from diastolic measurements yielded mean values of 24.5 ± 2.8 mm (radiologist), 24.7 ± 2.7 mm (technician), and 24.4 ± 2.6 mm (student). As in the free-form area analysis, there was a statistically significant inter-observer bias between the radiologist and the technician (−0.23 mm, p=0.001; LOA: ±2.2 mm [95% CI: 1.9, 2.5 mm]) and the student and the technician (−0.29 mm, p=0.0001; LOA: ±2 mm [95% CI: 1.7, 2.3 mm]). The mean difference between the radiologist and the student was again not significant (−0.05 mm, p=0.68; LOA: ±1.8 mm [95% CI: 1.6, 2.0 mm]).

Multiple reader paradigms

For the evaluation of the multi-reader paradigm C (also see “Statistical analysis” section), the radiologist and student were taken as the best single readers and the technician as the optional third reader in case of discrepancy. Table 2 and Figures 2 and 3 show the relative performance of the different multiple reader paradigms. The Bland-Altman analysis revealed no significant bias between the first and the second measurements for any of the three multiple-reader scenarios (average of two readers: −3.6 mm², p=0.11; average of three readers: −1.1 mm², p=0.54; average of two readers in case of agreement in device decision or average of 3 readers in case of disagreement: −3.7 mm², p=0.06, Table 2). All three multi-reader paradigms resulted in an improvement in precision (lower variability) compared to either single reader approach. While the change in precision between the best single reader (radiologist) and the averaged results of the two best readers (radiologist and student: ±38.9 mm² [95% CI: 31.4, 46.4 mm²]) was not statistically significant (p=0.20, overlap of the 95% CI in Figure 3), there was a statistically significant improvement in precision when averaging the results of all three single readers (±31.8 mm² [95% CI: 25.6, 38 mm²]) compared to the best single reader (p=0.03, no overlap of the 95% CI in Figure 3). Furthermore, the scenario in which a third reader was only added in case the first two readers disagreed in their device size decision also revealed a significant improvement in precision (±33.5 mm² [95% CI: 27.0, 40.0 mm²], p<0.05, no overlap of the 95% CI in Figure 3). (Note: We also investigated the “2+1” scenario in which the measurements of two least precise single readers (technician and student) were averaged and an optional third reader (radiologist) could be added if the first two readers differed in their device decision. Even for this less favorable scenario we found an improvement in precision compared to either single reader approach.)

Impact on device size

Figure 3 illustrates the possible impact of single or multiple readers’ precision on choosing a certain device size, illustrated using hypothetical device size categories loosely based on the Edward Sapien 3 aortic prosthesis (removing the overlap between categories for illustrative purposes).

Intra-reader variability would have resulted in a change in device size (either one category smaller or bigger) in 22% (18/82) with the radiologist, 25% (20/80) with the technician, and 26% (21/82) with the student (Table 3). Averaging multiple readers lowered the number of changes in device size to 10% (8/82) and 5% (4/80) with two or three or readers, respectively. The scenario in which a third reader was only included when the first (best) two readers disagreed in the chosen device size, resulted in a change in device size in 10% of cases (8/81).

Table 3:

Rate of changes in device size between repeated measures with different single- and multi-reader approaches.

	Radiologist	3D Tech	Med. Student	Average of 2 readers	Average of 3 readers	Only if device recommendation R1≠R2, average 3 readers
Change in device size	22% (18/82)	25% (20/80)	26% (21/82)	10% (8/82)	5% (4/80)	10% (8/81)

The scenario in which a third reader was only included when the first (worst) two readers disagreed in the chosen device size, resulted in a change in device size in 9% of cases (7/80).

Discussion

In this study we retrospectively evaluated TAVR CTA aortic annular measurements from single readers with different levels of experience and several multi-reader scenarios to determine the intra- and inter-observer variability and assess the impact on device sizing.

We found significant differences between certain single readers which suggests that the level of experience of each reader does not necessarily agree with the reliability of their measurements. The implementation of consensus readings, as simulated in our multiple reader scenarios, led to an increase in precision (decrease in variability) compared to any of the single readers. The increase in precision was, however, only statistically significant when using three readers or when using two readers who only included a third rater in case of disagreement between their device sizing recommendations.

Previous studies on the inter-observer reliability of TAVR CT measurements have reported good or excellent intraclass correlation coefficients (ICC) [16, 17]. However, we note that the ICC may be misleading when the variable of clinical importance is a discrete variable such as recommended device size. For example, two different sets of measurements by two readers (reader A and reader B) may produce the exact same ICC while having very different discordance rates in device size recommendations. The use of 95% LOA provides a numerical measure of the variability that allows direct comparison with the published guidelines for TAVR device size selection. Bland-Altman plots are therefore better suited to assess differences in variability and bias between repeated measurements [18]. Our results for mean intra-observer differences (1.5-5.7 mm²) and mean inter-observer differences (5.7-15 mm²) for annulus area measurements, as well as mean inter-observer differences for area derived diameters (0.05-0.29 mm) are in line with previously published studies that reported mean intra-observer differences of 1 mm² [19] for annulus area measurements and mean inter-observer differences of 8 mm² for annulus area [19] or 0.28-0.6 mm for area derived diameter measurements, respectively [20, 21].

Results for intra-observer precisions of area measurements from our single readers did not differ significantly and ranged between about ±50 to ±65 mm², while inter-observer precisions between the single readers were slightly lower (about ±64 to ±74 mm²). Blanke et al. reported higher intra- and inter-observer precisions from annulus area measurements made by two expert readers (±31 mm² and ±37.1 mm², respectively) [19].

To the best of our knowledge, no work has so far evaluated the impact of different multi-reader scenarios on the precision of TAVR planning CT measurements and the impact of the different approaches on device sizing. In our study, precision improved with all multi-reader paradigms (about ±32 to ±39 mm²) compared with the single reader evaluations and led to a clear decrease in the rate of discrepant device size categorizations (from 22-26% with single readers to 5-10% with multiple readers). Although precision was the highest when using three readers, consensus readings between two readers who only added a third rater in case of disagreement in their sizing recommendations might be the most reasonable alternative approach for TAVR CT evaluations with respect to practicability and cost efficiency. In clinical practice, the first evaluation and size recommendation could be performed by either the 3D laboratory or radiologist. Additionally, TAVR device vendors often provide their own measurements of TAVR CT datasets and provide a separate prosthesis size recommendation that can be utilized as a second rating. In case of deviating size recommendations between the first and second rating, a third and final rater can then be consulted to re-evaluate the images, with a final decision on the most appropriate aortic annulus prosthesis size for the patient being made in the setting of a multidisciplinary conference including cardiology, cardiothoracic surgery, and radiology.

Finally, our findings suggest that including an estimate of the inherent variability in TAVR CT screening measurements may provide additional useful context for clinicians to include in their decision-making.

This study has several limitations. In previous studies, the area or perimeter derived effective diameter and basal ring average diameters were shown to be the most reproducible values reflecting aortic annulus size [17, 20]. In our study, we primarily analyzed the aortic annulus area in accordance with the vendor’s recommendations for the Edward Sapiens 3 device. In contrast to linear measurements, area measurements are bound to magnify errors due to the square of the radius in the formula for an area. A recently published study by Schwarz et al. could show an equally high predictive value for mean diameter, annulus area and circumference for prosthesis size [22], while in their review article Blanke et al. recommended to report both the mean diameter of the short and long axis and the area measurement of the basal ring [11].

Moreover, the choice of prosthesis size as well as the rate of changes in device size category are being influenced by several other factors that could not be incorporated in the scope of this study: e.g. the device type (different sizing thresholds), and different anatomical criteria, such as annular diameter, distance to the coronaries, degree and extent of calcification, diameter of the sinutubular junction, and curvature of the sinuses of Valsalva. Furthermore, the reproducibility of the method itself, although crucial for appropriate device sizing, and our hypothetical multi-reader paradigms may suffer from systematical errors. Due to a lack of a reference standard, we could not determine the accuracy of our measurements. Also, the retrospective design did not allow us to evaluate the possible impact of the different single- or multiple-observer paradigms on clinical outcomes. Further larger studies are required to assess the differences between manual single and multi-reader strategies in comparison with (semi-) automatic aortic annulus measurements.

Conclusions

Accurate assessment of annular dimensions is crucial for appropriate device sizing. Therefore, awareness and knowledge of your readers intra- and inter-reader precision is necessary and incorporating these differences into TAVR CT reports could help clinicians to interpret results in a clinical context. Furthermore, multi-reader strategies for TAVR CT measurements provide higher precision than evaluations from single readers with different levels of experience. A reasonable approach could effectively be implemented with two readers and an optional third reader in a clinical setting.

Highlights.

• Multi-reader TAVR-CT evaluations have a lower measurement variability than single-reader measurements

• Multi-reader approaches reduce the rate of discrepant device size categorization

• Aortic annulus CT measurements for TAVR are highly reproducible