Inter- and Intraobserver Variability in the Assessment of Brain Arteriovenous Malformation Angioarchitecture and Endovascular Treatment Results

Abstract

BACKGROUND AND PURPOSE: Several angiographic features of brain arteriovenous malformations (BAVMs) have been associated with an increased risk of hemorrhage. However, interpretation of these features may not be consistent between observers. We conducted a study to determine inter- and intraobserver agreement of various angioarchitectural characteristics of BAVM.

MATERIALS AND METHODS: Two experienced interventional neuroradiologists independently reviewed pre- and post-endovascular treatment angiograms from 50 consecutive patients. Axial CT and/or MR images before treatment were included. We collected the following data: Spetzler-Martin grades, number of involved arterial territories, associated aneurysms by location (circle of Willis, feeding artery, intranidal, and venous), and nidus reduction after endovascular treatment (<33%, 33%–66%, and >66%). The reviewers were compared with each other, and 1 was compared with himself after a 3-month interval. Measures of agreement were performed by using the kappa statistic (κ) for nominal data and the weighted κ for ordinal data.

RESULTS: Inter- and intraobserver agreement were higher for assessment of the Spetzler-Martin grade (weighted κ = 0.70/0.75) and nidus size reduction after endovascular treatment (κ = 0.74/0.77). Inter- and intraobserver agreement were inferior for findings concerning feeding artery aneurysms (κ = 0.19/0.36), intranidal aneurysms (κ = 0.34/0.35), and venous aneurysms (κ = 0.50/0.67).

CONCLUSION: Angiographic characteristics of BAVMs considered as risk factors for hemorrhage, such as aneurysms, are not reliably detected on global angiograms between different observers. In contrast, the Spetzler-Martin grading system and angiographic results of endovascular treatment can be used with high observer agreement.

The management of brain arteriovenous malformations (BAVMs) remains a formidable challenge, despite recent development of combined technique treatment protocols.^1,2 Evaluation of results published in the literature is hampered by a lack of standardization of both treatment protocols and a definition of lesion characteristics or angiographic features. The role of a conservative approach, surgery, endovascular treatment, or radiosurgery varies tremendously according to local preference and availability of various persons with expertise. This variation may be explained by 3 factors: First, the natural history of the disease remains poorly defined. Second, referral patterns differ greatly and most of the literature comes from large centers that see highly selected cases; and third, there is a lack of standardization in reporting results due to difficulty with nomenclature.

In an effort to identify BAVMs at risk of hemorrhage, several angioarchitectural features thought to portend either an increased or decreased hemorrhagic risk, such as associated aneurysms (circle of Willis, feeder, nidal, or venous) and collateral circulation, were studied.^3–9 However, the reliability in detecting these features among different observers remains poorly documented and has been questionable in some studies.^10,11 In addition, to our knowledge, the reliability in reporting nidus reduction after embolization has never been studied. Guidelines for definition of BAVM angioarchitectural features were published in 2001,¹² but the reproducibility in reporting the angioarchitectural features according to the definitions given in this article was not assessed.

We initiated a retrospective study of all patients with BAVM referred to our department. As a preliminary study, we wanted to evaluate the local agreement between observers in reporting well-known BAVM angiographic features as well as in reporting endovascular treatment results.

Materials and Methods

From a cohort of 370 patients managed between January 1994 and December 2005, the pre- and post-endovascular treatment angiograms of 50 consecutive patients with BAVM were independently reviewed by 2 experienced interventional neuroradiologists (D.R., A.W.). All angiograms were obtained in a single institution and stored on an image archive system. The angiograms included all anteroposterior, lateral, and oblique views of global internal carotid and vertebral injections performed before and after embolization. Superselective injections were not available. Axial CT and/or MR images before treatment were included for each observer in the study. The observers were not presented with precise definitions of the angioarchitectural features that were assessed. One observer (D.R.) reviewed all the angiograms a second time, after a 3-month interval, for the intraobserver component of the study.

Several data were collected; differences were noted and recorded for each criterion. The Spetzler-Martin surgical grade, including eloquence of the adjacent brain, nidus size, and pattern of venous drainage (deep or superficial), was assessed.^13,14 Endovascular treatment result was evaluated by visual estimation of the decrease in size of the nidus after endovascular treatment and was classified in 3 categories (<33%, 33%–66%, and >66%). Collateral circulation was evaluated by the number of arterial territories involved in the BAVM supply (1 or more). These territories were anterior cerebral, middle cerebral, posterior cerebral, superior cerebellar, anteroinferior cerebellar, and posteroinferior cerebellar artery. Aneurysms associated with the malformation were divided according to their location, either on the circle of Willis, on a feeding artery of the BAVM, inside the nidus, or on a draining vein.¹²

All aneurysms except those located on the circle of Willis were afterward grouped as perinidal aneurysms.

Inter- and intraobserver agreement was measured by using the kappa statistic (κ) for nominal data and weighted κ for ordinal data.^15,16 Observer agreement was considered good (substantial to almost perfect) when the κ value was above 0.60.^15,16 All analyses were performed with SPSS version 11.0 (Statistical Package for the Social Sciences, SPSS, Chicago, Ill), except for confidence intervals (CIs) for κ and weighted κ tests, which were calculated by using Statistical Analysis Software version 8.2 (SAS Institute, Cary, NC).

Results

Inter- and intraobserver agreements for each characteristic are summarized in Fig 1. Intraobserver was generally superior to interobserver agreement, but both showed similar trends for each characteristic. Prominent findings were strong observer agreement for Spetzler-Martin grading [weighted κ inter = 0.70 (95% CI, 0.54–0.85), weighted κ intra = 0.75 (95% CI, 0.49–0.84)], including its different components: nidus size [κ inter = 0.75 (95% CI, 0.20–0.67), κ intra = 0.81 (95% CI, 0.20–0.67)], eloquence of adjacent brain [κ inter = 0.71 (95% CI, 0.20–0.67), κ intra = 0.86 (95% CI, 0.20–0.67)], or venous drainage pattern [κ inter = 0.80 (95% CI, 0.20–0.67), κ intra = 0.76 (95% CI, 0.20–0.67)]. Estimation of nidus size reduction after endovascular treatment also showed substantial observer agreement [κ inter = 0.74 (95% CI, 0.58–0.90), κ intra = 0.76 (95% CI, 0.61–0.92)]. A much weaker agreement was observed in identification of generally accepted hemorrhagic risk factors such as aneurysms on arterial feeders [κ inter = 0.19 (95% CI, −0.12–0.51), κ intra = 0.36 (95% CI, 0.03–0.69)], into the nidus [κ inter = 0.34 (95% CI, 0.05–0.62), κ intra = 0.35 (95% CI, 0.09–0.59)], or on the draining vein [κ inter = 0.50 (95% CI, 0.20–0.67), κ intra = 0.67 (95% CI, 0.20–0.67)]. Inter- and intraobserver agreement for perinidal aneurysms, without precise location, rises and almost become substantial [κ inter = 0.62 (95% CI, 0.20–0.67), κ intra = 0.51(95% CI, 0.45–0.87)].

Fig 1.

Intra- and interobserver agreement calculated for each architectural feature, measured by unweighted or weighted κ with 95% CI. Observer agreement was considered good (substantial to almost perfect) when the κ value was above 0.60.

Discussion

This preliminary work addressed the observer agreement in reporting commonly used angioarchitectural features of BAVMs. Substantial inter- and intraobserver agreement was shown in the evaluation of the Spetzler-Martin grades and their components. Our agreement was better than that observed by Al-Shahi et al (κ = 0.43).¹¹ This discrepancy may be attributed to a single-center effect. The fact that observers in the present study had access to multiple imaging modalities (angiography, CT, and MR imaging), instead of angiography alone, may also explain our higher agreement rate. Du et al¹⁰ recently observed an interesting discrepancy between neuroradiologic and neurosurgical assessment of Spetzler-Martin grading, the former having a tendency to undergrade whereas the latter would overgrade lesions. These authors called for caution when comparing patient selection and treatment results on the basis of imaging interpretation between different specialty groups. According to our results, we believe that the Spetzler-Martin grading system is a simple and reproducible method to classify lesions.

What is more interesting, despite the level of subjectivity involved, the assessment of nidus reduction after embolization showed substantial agreement in our cohort (κ = 0.74). To our knowledge, it is the first interobserver report of such an assessment.

The detection of aneurysms remote from the BAVM (circle of Willis) showed excellent inter- and intraobserver agreement. However, reporting of aneurysms within the vicinity of the nidus showed only fair agreement (Fig 1). Lumping together all aneurysms in the perinidal area (arterial feeder, nidal, and venous) may increase the agreement, though it is less than ideal. It is unclear whether detection of aneurysm presence was reported differently from aneurysm location because we do not know if the aneurysms interpreted were the same for both observers or, for the intraobserver study, if the aneurysms interpreted were the same at both times of observation. The subjectivity in reporting these lesions is very high because image superimpositions are numerous on global angiograms. (Figs 2 & 3) Superselective injection could increase sensitivity and specificity, and some investigators may propose that it should be done in every case for a comprehensive evaluation of a BAVM. Practically, however, a decision on treatment selection is commonly necessary before such selective studies are available. The role of 3D imaging (CT, MR imaging, or 3D angiography) could also enhance sensitivity, but this remains to be proved.¹⁷

Fig 2.

Anteroposterior (A) and lateral (B and C) views of the right carotid angiogram that shows a right frontal BAVM. There was agreement for the Spetzler-Martin grade II collateral circulation and the presence of the carotid aneurysm. However, we noted a disagreement about the presence or absence of a nidus aneurysm (arrow).

Fig 3.

Axial CT scan (A) showing an occipital hematoma in a 23-year-old man. Anteroposterior (B) and lateral (C) views of the right carotid angiogram showing a right occipital BAVM. There was agreement for the Spetzler-Martin grade, including the size, eloquence, venous drainage, as well as for the presence of a paranidal aneurysm responsible for hemorrhage. However, interobserver disagreement was about the precise location of the aneurysm, either on the feeding artery or into the nidus.

In a study comparing the characteristics of BAVMs from 2 large referral centers, Halim et al¹⁸ found a contradictory relationship between the presence of aneurysms and the risk of hemorrhage. In 1 center, there was a positive odd ratio of hemorrhagic risk in the presence of aneurysms, whereas it was negative in another center. The authors explained this by differences in referral patterns, but they also discussed the problem of aneurysm conspicuity on angiograms. Our study seems to favor this second hypothesis.

The debate on the impact of the presence of intranidal aneurysms in terms of hemorrhagic risks is ongoing. In an acutely ruptured BAVM, a focal vascular ectasia, especially if it relates anatomically with the hematoma, may represent the point of rupture lacking a true vascular wall (pseudoaneurysm), perhaps indicating a higher risk of a second hemorrhage than a focal ectasia in an unruptured lesion (true aneurysm). Whatever the risks involved, the results of our study do not undermine the importance of intranidal aneurysms per se, but they are a plea for caution in using descriptions of aneurysms in clinical decision making or in comparing results between centers.

The strengths of the present study are the following: First, a specialty tertiary referral center bias is somewhat reduced because our institution is a comprehensive referral center offering all treatment modalities for most BAVMs in a large metropolitan area. Second, cases evaluated were on consecutive patients. Third, an intraobserver agreement assessment was added to evaluate consistency of readings. Strict adherence to published definitions of angioarchitectural features¹³ was not performed to replicate everyday practice. We acknowledge that interobserver agreement could be increased if universally accepted criteria were used. Weaknesses include the single-center effect that precludes general conclusions on interinstitutional comparison of results and the limited number of cases studied.

Conclusion

In this series, Spetzler-Martin grading, assessment of nidal reduction after embolization of BAVMs, and detection of remote arterial aneurysms showed high inter- and intraobserver agreement. On the other hand, more subtle angioarchitectural features, such as the presence of perinidal aneurysms, were not reported with reliability.