Inter- and Intraobserver Agreement in CT Characterization of Nonaneurysmal Perimesencephalic Subarachnoid Hemorrhage

Abstract

BACKGROUND AND PURPOSE:

The perimesencephalic pattern of SAH as seen on unenhanced CT is associated with significantly better outcomes when compared to an aneurysmal pattern of SAH. The aim of this study was to determine the degree of inter- and intraoberver agreement for characterization of the NAPH as seen on unenhanced CT.

MATERIALS AND METHODS:

We retrospectively reviewed the CT scans of 37 patients with spontaneous SAH, all of whom had undergone CT within 24 hours of onset of headache symptoms. All patients had undergone conventional cerebral angiography to confirm or exclude aneurysms or other vascular pathology. All 37 cases were angiographically confirmed nonaneurysmal SAHs. Four readers with neuroradiology subspecialty training independently evaluated CT images to characterize the hemorrhage pattern as compatible with the well-described NAPH. Each reader performed a second reading session blinded to the initial readings. The first and second sets of readings were performed approximately 4 months apart. Inter- and intraobserver agreement for characterization of the NAPH was determined by using the κ statistic.

RESULTS:

Of the 37 angiographically confirmed nonaneurysmal SAHs, there was unanimous agreement as to the hemorrhage pattern in 29 (78%) cases and disagreement in 8 (22%) cases. Overall, intraobserver agreement was good (κ = 0.80). Interobserver agreement was also good (κ = 0.79).

CONCLUSIONS:

Overall, inter- and intraobserver agreement for the NAPH was good. There was, however, a level of disagreement among observers, thus suggesting that clinicians should be cautious when deciding whether to pursue follow-up imaging.

Known causes of SAH include intracranial aneurysms, arteriovenous malformations, bleeding intracranial tumors, and trauma. There is a subset of patients in whom the cause of an SAH is unknown and cannot be determined on follow-up imaging.¹ These patients often exhibit a pattern of hemorrhage on CT scans that is known as NAPH. This pattern was first described by van Gijn et al in 1985² as an SAH in which blood was limited primarily to the perimesencephalic cisterns with no evidence of aneurysm on angiography.

The determination of the NAPH pattern is important because this pattern is associated with better clinical outcome than that seen with aneurysmal SAH. Studies have shown that >90% of patients who present with the NAPH pattern have limited morbidity, and mortality is extremely rare.^2–4 These observed outcomes are in contrast to those of untreated aneurysmal SAH, in which morbidity and mortality rates are >50%.

The relationship between the NAPH pattern and aneurysms has been studied extensively. It has been demonstrated that between 7% and 17% of posterior circulation aneurysmal ruptures can present with the NAPH pattern on CT.^5,6 In addition, studies have shown that the likelihood of finding an aneurysm on angiography when a patient presents with this pattern is between 3% and 9%.^5,7 Recent advances in CTA have led to substantial increases in its use as a primary technique for work-up of patients with suspected aneurysmal SAH. The persistent imperfect sensitivity of CTA for aneurysm detection precludes its use in many centers as a stand-alone imaging technique in SAH. However, given the diminished pretest probability of aneurysm in the setting of the NAPH pattern, some practitioners have suggested that CTA may be sufficient for diagnosis.⁸

Because the characterization of a given hemorrhage as matching the NAPH pattern leads to changes in patient triage, it would be useful to know how reliable such CT characterizations are. Labeling a given patient's pattern on CT as the NAPH pattern is not necessarily black and white; it remains possible that observers may disagree as to whether a given SAH meets the described pattern.

To better define the variation among independent readers regarding whether a given CT scan satisfies the criteria for the NAPH, we performed a formal inter- and intraobserver variability study on a consecutive case series. These findings will allow improved understanding of the reliability of denoting a given CT pattern as compatible with the NAPH pattern, to improve triage decisions and the quality of patient care.

Materials and Methods

Four observers retrospectively analyzed a series of 37 CT scans in patients who presented with SAH in an effort to determine whether each CT scan represented the classic pattern of NAPH. The criteria for this pattern were the following: the center of the hemorrhage located in front of the mesencephalon without blood in the interhemispheric and lateral Sylvian fissures (except for minute amounts) and no significant intraventrical hemorrhage.²

The observers represented a varied level of experience: Two were senior faculty and interventional-trained neuroradiologists (D.F.K., H.J.C., both with 15 years of experience), 1 was junior faculty in neuroradiology (J.M., with 3 years of experience), and the fourth was an interventional neuroradiology fellow (J.B.W., with 3 years of experience). Each observer was charged with analyzing each patient's CT scan twice and was blinded to any readings by other observers or previous readings by the same observer. The first and second set of readings were performed approximately 4 months apart. Before the first set of readings, each participant was given the NAPH criteria described above.

Selection of Patients and Images

The patient data used in this study were gathered by performing an institutional data base search for those patients who presented to the Mayo Clinic (Rochester, Minnesota) with SAH from 2002 to 2006. All CT scans were noncontrast images and were obtained from a 4- or 16-section LightSpeed scanner (GE Healthcare, Milwaukee, Wisconsin). Every patient had CT performed within 24 hours of presentation, and each patient included for analysis also had a conventional cerebral angiogram obtained as part of their evaluation. Ten patients had one 4-vessel cerebral angiogram following CT, 24 patients had 2 follow-up 4-vessel cerebral angiograms, and 2 patients had 3 follow-up angiograms. Those patients with findings on CT positive for SAH but negative on cerebral angiography for aneurysm or other vascular pathology represented the subset of patients in this study. The observers were allowed to review every image section of the initial CT scan to make their determination.

Angiography Techniques

Typically, 5F or 6F catheters were placed into the internal carotid or vertebral arteries. All DSA examinations were performed by using a biplane digital angiography suite (Integris; Philips Medical Systems, Best, the Netherlands). A volume of 16 mL of nonionic contrast medium was injected through a 5- to 6F catheter by use of an injector with a velocity of 4 mL/s. Biplane DSA images of the entire circulation were obtained.

Statistics

To evaluate inter- and intraobserver agreement, we calculated κ statistics. κ values have traditionally been defined as <0 = no agreement, 0.0–0.20 = slight agreement, 0.21–0.40 = fair agreement, 0.41–0.60 = moderate agreement, 0.61–0.80 = good agreement, and 0.81–1.00 = very good agreement. We also determined the proportion of cases in which there was unanimous agreement on the presence of the NAPH pattern, and we defined cases with unanimous agreement as those in which all 8 reads (2 per reader for 4 readers) were exactly the same.

Results

Perimesencephalic versus Nonperimesencephalic Nonaneurysmal SAH

All 37 cases in this study were angiographically confirmed nonaneurysmal SAH. Of the 37 cases, there was unanimous inter- and intraobserver agreement on 29 cases (78.4%). Of these 29 cases, 10 were determined to have the NAPH pattern and 19 were determined not to have the perimesencephalic pattern.

Of the 8 cases in which there was disagreement, in 3 cases 1 observer on 1 read recorded the NAPH pattern as present; and in 4 cases, 2 observers on 1 read reported a NAPH pattern. In 1 case, 2 observers reported on both reads that the NAPH pattern was present and 2 readers reported on both reads that the NAPH pattern was not present. Thus, of the 8 cases in which there was disagreement, there was a general consensus that the NAPH pattern was not present in 7 cases and that 1 case was inconclusive.

Intraobserver Agreement

The average κ value for intraobserver agreement was 0.80. This signifies good agreement. Two observers had κ values of 0.87, 1 observer had a κ value of 0.76, and 1 had a κ value of 0.69. Thus, the range of agreement was good to very good.

Interobserver Agreement

The average κ value for interobserver agreement was 0.79, signifying good agreement. The range of κ values among observers was 0.73–0.88, indicating that interobserver agreement was good to very good. Overall, all observers agreed on the nature of the SAH in 29 of the 37 images (78%). Major results from this study are summarized in the Table. An example of a case in which there was disagreement is found in Fig 1.

Major results

Measure	Result
Average intraobserver κ	0.80
Range intraobserver κ	0.69–0.87
Average interobserver κ	0.79
Range interobserver κ	0.73–0.88
% cases with unanimous agreement	78.4

Fig 1.

Head CT scan of an SAH with angiographically negative findings in a 53-year-old man. Disagreement in the reading of this CT scan centered around the amount of blood that extended into the right lateral Sylvian fissure (arrow).

Discussion

Inter- and intraobserver agreement on the presence of the NAPH pattern on CT is of great importance because patients who present with this pattern generally have a significantly better prognosis and require less aggressive management than patients who do not. If there is a lack of consistency in determining the presence of the NAPH pattern on CT, then this would suggest that aggressive follow-up studies such as angiography are necessary to determine the source of the bleed. Our study demonstrated that on average, there is good interobserver (κ = 0.79) and intraobserver (κ = 0.80) agreement in determining whether a nonaneurysmal SAH demonstrates the NAPH pattern as described by van Gijn et al in 1985.² On the other hand, there was disagreement among observers in 22% of cases as to whether the bleed matched the NAPH pattern. Thus, careful judgment should be exercised before making triage decisions when patients present with CT images that are suggestive of a NAPH bleed.

There are very limited data available on inter- and intraobserver agreement for recognizing the NAPH pattern on CT. Velthuis et al⁹ conducted a study in which 2 neuroradiologists read unenhanced CT scans of 40 patients with posterior fossa hemorrhages and determined that the interobserver agreement was κ = 0.87 for indentifying the NAPH pattern. Rinkel et al¹⁰ conducted a similar study; however, only patients with 1 angiogram with negative findings were included, and interobserver agreement between 2 neuroradiologists was found to be κ = 0.89. In both of these studies, ground truth was usually established by 1 angiogram with negative findings, whereas in our study, 28 of the 37 patients had at least 2 angiograms with negative findings. Our study demonstrated that there was good interobserver agreement (κ = 0.79). However, there was a large range in agreement between different pairs of observers (κ = 0.73 to κ = 0.88). In addition, these previous studies did not examine intraobserver agreement, which is of equal significance to interobserver agreement.

Limitations

The primary limitation of the study is its retrospective nature. All cases in this study were selected as angiographically confirmed nonaneurysmal SAH. Previous studies have demonstrated ≤10% of SAHs that match the NAPH pattern are due to aneurysm rupture. Such cases would have been excluded from our study; thus, it is possible that inter- and intraobserver agreement for the presence of the NAPH pattern of nonaneurysmal SAH may have been different if this experiment was performed on a larger series including posterior fossa hemorrhages. In addition, in this study, we did not seek to determine the sensitivity and specificity for detection of the NAPH pattern on CT; the purpose of this study was only to assess inter- and intraobserver agreement for the detection of the NAPH pattern.

Future Directions

The results of this study suggest that inter- and intraobserver agreement on the presence of the NAPH pattern on CT scans is good. We also found that in approximately 22% of cases, there was disagreement as to whether the SAH matched the NAPH pattern. This implies that agreement may not be strong enough to suggest that the presence or absence of this pattern is reliable enough for a single observer's opinion to be taken into account when deciding on the future management of the patient. If there is a low index of suspicion for the presence of an intracranial aneurysm, follow-up CT angiography, rather than DSA, can be used to accurately exclude aneurysms.^8,9

In this study, we also found that in a most of cases, nonaneurysmal SAH does not match the NAPH pattern. Given that such cases have not been as well characterized as NAPHs, it seems necessary to study these cases extensively so as to determine whether one can develop an algorithm that could be used to reliably determine the best way to manage these cases.

Conclusions

We demonstrated that interobserver and intraobserver agreement for the detection of the NAPH pattern on unenhanced CT scans is good to very good. There is, however, a level of disagreement among observers, thus suggesting that clinicians should be cautious when deciding whether to pursue follow-up imaging.

Abbreviations

CTA

CT angiography

DSA

digital subtraction angiography

NAPH

nonaneurysmal perimesencephalic subarachnoid hemorrhage

SAH

subarachnoid hemorrhage