Perfusion imaging for delayed cerebral ischemia detection in patients following ruptured aneurysmal subarachnoid hemorrhage: Interrater reliability assessment

Abstract

Background

Delayed cerebral ischemia (DCI) following aneurysmal subarachnoid hemorrhage (aSAH) is associated with adverse neurological outcomes. Early and accurate diagnosis of DCI is crucial to prevent cerebral infarction. This study aimed to assess the diagnostic accuracy and interrater agreement of the visual assessment of neuroimaging perfusion maps to detect DCI in patients suspected of vasospasm after aSAH.

Methods

In this case-control study, cases were adult aSAH patients with DCI who underwent magnetic resonance perfusion or computed tomography perfusion (CTP) imaging in the 24 h prior to digital subtraction angiography for vasospasm diagnosis. Controls were patients with dizziness and no aSAH on CTP imaging. Three independent raters, blinded to patients’ clinical information, other neuroimaging studies, and angiographic results, visually assessed anonymized perfusion color maps to classify patients as either having DCI or not. Tmax delay was classified by symmetry into no delay, unilateral, or bilateral.

Results

Perfusion imaging of 54 patients with aSAH and 119 control patients without aSAH was assessed. Sensitivities for DCI diagnosis ranged from 0.65 to 0.78, and specificities ranged from 0.70 to 0.87, with interrater agreement ranging from 0.60 (moderate) to 0.68 (substantial).

Conclusion

Visual assessment of perfusion color maps demonstrated moderate to substantial accuracy in diagnosing DCI in aSAH patients.

Background

Delayed cerebral ischemia (DCI) manifests as neurological deficits that arise after aneurysmal subarachnoid hemorrhage (aSAH). These deficits cannot be primarily attributed to surgical complications, rebleeding, hydrocephalus, seizures, or systemic factors such as sedation, infection, metabolic derangements, hypotension, or hypoxia. ¹ DCI is distinguished by the appearance of new ischemic lesions on neuroimaging that were not present at the initial onset of aSAH.^1,2 DCI negatively impact the prognosis of patients following aSAH, yet the pathophysiologic basis of DCI remains incompletely understood.^3,4 DCI is believed to stem from a combination of factors, including large-vessel cerebral vasospasm, and various brain injury processes triggered by aneurysm rupture and early brain injury. ¹ While macrovascular vasospasm plays a crucial role in DCI pathophysiology, microvascular perfusion deficits, thrombosis, blood–brain barrier disruption, impaired cerebral autoregulation, autonomic nervous system dysfunction and cortical spreading depolarization are also implicated in DCI.^5,6

The diagnosis of DCI in critically ill patients presents challenges; while serial neurological exams are essential, they offer limited insights in severe aSAH cases. ⁷ Transcranial Doppler ultrasonography is commonly used for large vessel vasospasm detection, but it primarily measures velocities in the major arteries of the circle of Willis. ⁸ Computed tomography angiography (CTA) can identify arterial narrowing indicative of vasospasm, but CTA lacks sensitivity for DCI detection.^9,10 In the progression of DCI, reduced cerebral perfusion precedes ischemia. Therefore, evaluating cerebral perfusion is crucial to assess early microvascular perfusion deficits. Perfusion imaging is an ideal tool in this context, as it can detect perfusion abnormalities and identify microcirculatory alterations beyond the macroscopic detection capacity of CTA. ¹¹

However, the diagnostic capability of perfusion imaging for DCI remains understudied. Previous studies have explored the reliability of visually assessing computed tomography perfusion (CTP) color maps, alone or in conjunction with CTA, to detect cerebral vasospasm, yielding varying levels of agreement and diagnostic accuracy.^12–14 In this case-control study, we aimed to use qualitative assessment of perfusion color maps exclusively to identify the presence of DCI.

Methods

The research protocol received approval from the Institutional Review Board (IRB #36199) at Stanford University School of Medicine. Given the observational and retrospective nature of the analysis involving anonymized data, the requirement for informed consent for this study was waived.

Cases were identified from our institutional database that includes patients with aSAH and a clinical diagnosis of DCI who underwent endovascular treatment of vasospasm. At our institution, DCI is diagnosed by the neurocritical care attending physician based on the 2010 definition, which includes new focal neurological impairment (e.g., hemiparesis, aphasia, apraxia, hemianopia, or neglect) or a decrease of at least 2 points on the Glasgow Coma Scale lasting for at least 1 h. This deterioration must not appear immediately after aneurysm occlusion and cannot be attributed to other causes based on clinical assessment, computed tomography (CT) or MRI brain scans, and laboratory studies. ¹ Inclusion criteria were patients older than 18 years with confirmed ruptured aSAH and completion of CT perfusion (CTP) or MR perfusion (MRP) within 24 hours prior to angiography conducted during vasospasm treatment. Exclusion criteria were uninterpretable cerebral perfusion imaging due to excessive patient motion, failed contrast bolus, or other artifacts.

Controls were drawn from a separate institutional dataset comprised of patients who arrived at our emergency department with symptoms of dizziness. Inclusion criteria were patients >18 years of age with an emergency room diagnosis of dizziness, who underwent CTP upon admission, and lacked a diagnosis of aSAH or any other acute stroke. Patients with uninterpretable cerebral perfusion imaging due to patient motion, contrast bolus failure, or other artifacts were excluded.

Patient demographic and clinical data were retrieved from electronic health records.

Imaging protocols

All patients with aSAH included in the study were diagnosed DCI. At our institution, a clinical diagnosis of DCI based on neurological deterioration triggers a CT to exclude causes such as hydrocephalus and re-bleeding, and a CTA to look for signs of vasospasm. When CTA shows evidence of vasospasm, patients are taken to the angiography suite for endovascular vasospasm treatment. All patients included in this study had DCI with CTA showing evidence of vasospasm as the cause of DCI and subsequently underwent digital subtraction angiography (DSA) for suspected vasospasm as the cause of DCI. Vasospasm diagnosis was evaluated via DSA following the injection of the internal carotid or vertebral arteries. The assessment of vasospasm by experienced neuro-radiologists encompassed the evaluation of bilateral internal carotid arteries (ICA), anterior cerebral arteries (A1 and A2-A3 segments), middle cerebral arteries (M1 and M2-M3 segments), vertebral arteries (V4 segment), and the basilar artery. Vasospasm at DSA was considered as narrowing of the vessel caliber compared to baseline DSA/CTA of at least 25%.

CT imaging was conducted using General Electric or Siemens CT scanners equipped with 64–256 detector rows. Following the protocol, a non-contrast head CTA were initially obtained in the axial plane with a slice thickness of 0.625 mm and a ratio of 0.984:1/39.37 cm, employing acquisition parameters of 80 kV and 550 mA. During CTA, iodinated contrast was infused into an antecubital vein at a flow rate of 3–5 ml/s, with image acquisition occurring either 15 second post-injection or through bolus triggering in the ascending aorta, achieving a density of 80 Hounsfield units. The acquisition of CTA image ran from the aortic arch to the vertex of the brain.

CTP imaging was performed on General Electric CT scanners with a single or dual-slab acquisition after the CTA, with 1–2 minutes interval between sequences to allow the contrast from the CTA to achieve a stable concentration before CTP acquisition. CTP scans were executed with a field of view (FOV) of 22, employing acquisition parameters of 80 kV, 200–250 mA, 5 mm slice reconstruction, a scan duration of 45 seconds (maximum 60 seconds), and administration of 35 ml of iodinated contrast medium alongside 20 ml of NaCl at a rate of 5 ml/s (minimum 4 ml/s). Dynamic cine image acquisition started 5–7 seconds after the injection of iodinated contrast medium.

Brain magnetic resonance imaging (MRI) was performed using a 3.0-T GE MR750 MRI scanner equipped with an 8-channel GE HR brain coil (GE Healthcare, Milwaukee, WI). Each MRI study included diffusion-weighted imaging (DWI), intravoxel incoherent motion (IVIM), time-of-flight magnetic resonance angiography (TOF-MRA), gradient recalled echo (GRE), and MRP employing dynamic susceptibility contrast technique (DSC-PWI). Perfusion-weighted imaging (PWI) was conducted post-intravenous administration of MultiHance (Bracco, Milan, Italy) into an antecubital vein at a rate of 4.0 ml/s via a power injector. PWI parameters included repetition time (TR) of 1800 ms, echo time (TE) of 35 ms, flip angle of 80°, and a slice thickness of 5 mm.

Qualitative analysis of perfusion maps

CTP and MRP maps were automatically derived using automated RAPID software (RAPID, version 4.9, iSchemaView, Menlo Park).

The qualitative DCI detection algorithm on perfusion maps is shown in Figure 1. Anonymized perfusion color maps, which included Tmax multi-threshold maps delay, regional cerebral blood flow (rCBF) and regional cerebral blood volume (rCBV) maps, and mean transit time (MTT), underwent visual assessment by three independent raters. These raters were distinct from those who evaluated vasospasm on DSA. The perfusion maps raters consisted of one neuro interventional radiologist and two stroke neurologists, each with over 10 years of neuroimaging experience. To mitigate potential bias, raters were blinded to patients’ clinical information, other neuroimaging studies, and angiographic results.

Figure 1.

Delayed cerebral ischemia detection algorithm on perfusion maps. rCBV: regional cerebral blood volume; rCBF: regional cerebral blood flow; MTT: mean transit time; Tmax: time to maximum.

As depicted in Figure 1, two separate reading assessments were carried out on the same patients. For both assessments, DCI was categorized as either absent or present. In the first assessment, raters were asked to adjudicate DCI based on the pattern or Tmax delay. Tmax delay was categorized by symmetry into no delay, unilateral delay, or bilateral delay. “No delay” referred to the absence of Tmax delay, “unilateral delay” as the presence of Tmax delay on one brain hemisphere, and “bilateral delay” as Tmax delay in both hemispheres. In evaluating Tmax symmetry, DCI was deemed absent if there was no Tmax delay or if Tmax delay was bilateral (symmetric). Conversely, DCI was considered present if the Tmax delay was unilateral (asymmetric). In the second assessment, readers reviewed rCBV, rCBF, MTT, and Tmax perfusion maps to determine DCI absence or presence.

Statistical analysis

Categorical parameters were compared between the aSAH cases and the controls without aSAH using the Chi-square test, while continuous variables were compared using a t-test.

To evaluate the diagnostic value of the qualitative review of CTP and MRP and its maps to detect DCI, sensitivity, specificity, and overall accuracy were calculated for the 3 raters’ assessments. We also compared correlated frequencies using McNemar's test. We assessed the agreement among raters for DCI diagnosis and their agreement with the diagnosis of vasospasm at angiography. The agreement calculation was based on the Gwet's AC1 statistic, which is a robust agreement coefficient, similar to Cohen's Kappa, but less affected by prevalence and imbalanced marginal probability. ¹⁵ The level of agreement was interpreted according to Landis and Koch as follows: slight agreement (0–0.2); fair agreement (0.21–0.4); moderate agreement (0.41–0.6); substantial agreement (0.61–0.8); and almost perfect agreement (0.81–1.0). ¹⁶

All statistical tests were double-sided and assumed to be significant at p < 0.05. Data management and analyses were performed in Statistical Analysis System (SAS) software v9.4.

Results

This pilot study included the initial 100 consecutive cases of aSAH from an institutional database of approximately 270 adult patients. These patients had undergone either CTP or MRP imaging prior to DSA for suspected vasospasm following aSAH. Subsequently, they were diagnosed with vasospasm during DSA. Of the first 100 patients, 54 patients with angiographic vasospasm met the inclusion criteria. The mean ± standard deviation age of patients was 52 ± 13 years, and 33 (61%) were female. Controls were drawn from a cohort of 124 patients presenting to our emergency department with a diagnosis of dizziness and who had undergone CTP. Following the exclusion of 5 patients due to substantial movement artifacts, a total of 119 controls were included. 67 (56%) of the control patients were female and their mean ± standard deviation age was 65 ± 17 years.

The Enrollment Flow Diagram is shown in Figure 2.

Figure 2.

Enrollment flow diagram.

The demographic details, the aSAH and vasospasm characteristics of cases are shown in Table 1.

Table 1.

Characteristics of patients with delayed cerebral ischemia (n = 54).

Age	52 ± 13
Female sex, n (%)	33 (61%)
Comorbidities, n (%)
Hypertension	23 (43%)
Hyperlipidemia	7 (13%)
Smoking	18 (33%)
Alcohol use	12 (22%)
Drug abuse	8 (15%)
Location of aneurysm rupture, n (%)
Acom	21 (39%)
MCA	11 (20%)
ICA	9 (17%)
Pcom	3 (6%)
Vertebral	4 (7%)
Other	Superior Hypophyseal artery (n = 1), Ophthalmic artery (n = 1), PCA (n = 1), Anterior choroidal artery (n = 1), PICA (n = 1), Basilar (n = 1)
Hunt and Hess grade, n (%)
II	13 (24%)
III	28 (52%)
IV	10 (19%)
V	3 (6%)
Fisher grade, n (%)
II	2 (4%)
III	26 (48%)
IV	24 (44%)
Aneurysm repair, coiling/clipping, n (%)	30 (56%)/24 (44%)
Vasospasm treatment
IA nicardipine, n (dose range in mg)	53 (1.67–60)
Angioplasty, n (%)	20 (37%)
mRS at 6 months follow-up, n (%)
0	2 (4%)
1	5 (9%)
2	8 (15%)
3	6(11%)
4	4 (7%)
5	4 (7%)
6	9 (17%)
Lost to follow-up	16 (30%)

Acom: anterior communicating artery; MCA, middle cerebral artery; ICA, internal carotid artery; Pcom, posterior communicating artery; PICA, posterior inferior communicating artery; PCA, posterior cerebral artery; IA: Intra-arterial; mRS: modified Rankin Scale.

Raters visually assessed CTP and MRP color maps to identify the presence of DCI based on the pattern of perfusion deficit (Figure 1). When raters based their determination of DCI on the symmetry of Tmax delay only (Table 2), the interrater agreement coefficient (Gwet's AC1 statistic) among the three raters was 0.68 (0.60–0.76), which demonstrates substantial agreement among raters and the overall agreement coefficient of the three raters with the angiographic vasospasm diagnosis was 0.54 (0.46–0.61) which indicates moderate agreement. Variability was observed among the raters. Rater 1, the neuro interventional radiologist, achieved the highest specificity at 0.82 when diagnosing DCI by visually inspecting the symmetry of Tmax color maps delay, with an accuracy of 0.76 (0.70–0.83), which reflects substantial agreement with the diagnosis of angiographic vasospasm. Rater 2 achieved the highest sensitivity at 0.66 when diagnosing DCI using the same method, with an accuracy of 0.72 (0.65–0.78), which indicates substantial agreement with the diagnosis of vasospasm at angiography. Unilateral Tmax delay occurred in 32 out of 54 DCI cases (59%) and in 48 out of 119 controls (40%).

Table 2.

Diagnostic performance of the visual rating of the symmetry of Tmax maps on CTP perfusion for the diagnosis of angiographic vasospasm.

Symmetry of Tmax
	R1	R2	R3
TN	95	86	95
FP	21	30	21
FN	19	18	23
TP	34	35	30
Sensitivity %	64.2	66	56.6
Specificity %	81.9	74.1	81.9
NPV %	83.3	82.7	80.5
PPV %	61.8	53.6	58.8
Accuracy with diagnosis (95%CI)	0.76 (0.70–0.83)	0.72 (0.65–0.78)	0.74 (0.67–0.81)
Interrater agreement (95%CI)	0.68 (0.60–0.76)
Agreement of the 3 raters with diagnosis (95%CI)	0.54 (0.46–0.61)

Tmax, time-to-maximum; CTP, computed tomography perfusion; R, rater; TP, true positives; FP, false positives;

FN, false negatives; TP, true positives; CI: confidence interval.

To potentially increase the sensitivity and specificity of the raters’ visual assessment of perfusion maps to diagnose DCI, the same raters evaluated the perfusion maps from the same patients in a second round of rating when we incorporated in the evaluation additional visual grading of all perfusion maps (rCBF, rCBV, MTT, Tmax) (Table 3). The interrater agreement coefficient among the three raters for all perfusion maps lowered from 0.68 to 0.60 (0.51–0.69) which indicates moderate agreement, and the agreement coefficient for all perfusion maps with an angiographic vasospasm diagnosis was 0.57 (0.50–0.65) which indicates moderate agreement. Rater 1 achieved the highest specificity at 0.87 when diagnosing DCI, which represents an improvement from the prior values of 0.82 specificity. The accuracy was similar at 0.80 (0.74–0.86), which signifies substantial concordance with the angiographic vasospasm diagnosis. On the other hand, Rater 2 achieved the highest sensitivity at 0.78 and an accuracy with the angiographic vasospasm diagnosis at 0.72 (0.66–0.79) which indicates substantial agreement. However, there was still a noteworthy difference between this rater's rating and the angiographic vasospasm diagnosis, p < 0.03. In fact, this rater classified 36 cases as false positives, which constitutes a 46% false positive rate.

Table 3.

Diagnostic performance of the visual rating of perfusion maps (CBF, CBV, MTT, Tmax) for angiographic vasospasm diagnosis in patients with delayed cerebral ischemia.

	CBF, CBV, MTT, Tmax
	R1	R2	R3
TN	104	83	91
FP	15	36	28
FN	19	12	14
TP	35	42	40
Sensitivity (95%CI)	0.65 (0.52–0.78)	0.78 (0.67–0.89)	0.74 (0.62–0.86)
Specificity (95%CI)	0.87 (0.81–0.93)	0.70 (0.62–0.78)	0.77 (0.69–0.84)
Accuracy with diagnosis (95%CI)	0.80 (0.74–0.86)	0.72 (0.66–0.79) *	0.76 (0.69–0.82) **
Interrater agreement (95%CI)	0.60 (0.51–0.69)
Agreement of the 3 raters with diagnosis (95%CI)	0.57 (0.50–0.65)

CBF, cerebral blood flow; CBV, cerebral blood volume; MTT, mean transit time; Tmax, time-to-maximum; R, rater; TP, true positives; FP, false positives; FN, false negatives; TP, true positives; CI: confidence interval.

^*p = 0.0005, compared to angiographic vasospasm diagnosis.

^**p = 0.03, compared to angiographic vasospasm diagnosis.

Discussion

In our study focusing on the visual assessment of global patterns of perfusion delay, including rCBF, rCBV, Tmax, and MTT, to determine their efficacy in indicating the presence of vasospasm in patients with DCI after aSAH, we observed moderate to substantial interrater agreement with the diagnosis of vasospasm as confirmed by angiography.

Neurological deterioration following aSAH prompts diagnostic examinations mainly focused to detect macrovascular vasospasm as the cause of DCI. While transcranial Doppler evaluation of large cerebral artery velocities and CTA are commonly employed for this purpose, both techniques primarily assess changes in large artery velocities and calibers, thus lacking sensitivity to microvascular perfusion deficits implicated in the DCI symptomology.^3,9,10 Perfusion imaging, on the other hand, offers the capability to detect early perfusion abnormalities and identify microcirculatory changes.

The diagnostic accuracy of our raters’ visual assessment of perfusion maps ranged from 0.72 to 0.8, with sensitivity ranging from 0.65 to 0.78 and specificity from 0.70 to 0.87. Cremers et al. ¹² visually assessed CTP color maps to diagnose DCI in 71 aSAH patients and reported a sensitivity of 0.72 for diagnosing DCI, which increased to 0.82 for patients with severe vasospasm defined as >50% decrease in vessel diameter on CTA.

In our study, there was only moderate to substantial interrater agreement, ranging from 0.6 to 0.68, and moderate agreement among the three raters with the angiographic diagnosis of vasospasm, which ranged from 0.54 to 0.57. Wintermark et al. ¹³ examined interobserver agreement between 2 raters for qualitative analysis of CTP maps in 33 patients, reporting values of 0.79 for MTT, 0.66 for CBF, and 0.83 for CBV. Their observed agreement was higher than in our study. However, they assessed five anatomical regions (frontal, temporal, parietal, occipital/thalami, and basal ganglia/insula) and evaluated each map (MTT, CBF, and CBV) separately. Given the current understanding that DCI represents a microperfusion deficit affecting the entire brain rather than specific regions or vascular territories, we chose to assess perfusion parameters globally. Unlike Wintermark's study, ours included both MRP and CTP data, whereas theirs focused solely on CTP. Additionally, our study incorporated a control group of patients without aSAH to evaluate model performance. Therefore, it's plausible that the differences in our perfusion map evaluation protocol contributed to the lower agreement observed in our study.

Recently, Heitkamp et al. ¹⁴ evaluated the interrater reliability of three neuroradiologists in detecting vasospasm using a combination of CTA and CTP imaging in 46 patients. The interrater reliability for identifying severe vasospasm was fair when CTP was added to CTA. However, they did not use CTP alone but only in combination with CTA, and they assessed each vessel separately. Furthermore, they found moderate agreement regarding the decision to undergo endovascular treatment, which increased to substantial when considering only senior raters. These findings are similar to our study, in which the accuracy of DCI diagnosis was higher for Rater 1, the neuro interventional radiologist, who may have more experience in reading perfusion imaging.

From ours and previous studies, it emerges that the variability in interrater agreement for detecting DCI on perfusion imaging, ranges from fair to substantial, and the agreement with the diagnosis of angiographic vasospasm is higher among senior raters suggesting that the visual assessment of perfusion maps alone might not be ideal for diagnosing DCI and/or guiding treatment selection. As acknowledged in recent years, although angiographic vasospasm may cause neurological deterioration and DCI in patients with aSAH, it is not the only cause of DCI and other mechanism are implicated. Previous studies ¹⁷ have shown territories of hypoperfusion, as measured by positron emission tomography, in regions not affected by angiographic vasospasm, which indicates that other factors may contribute to perfusion reduction and neurological deterioration following aSAH. Microvascular perfusion deficits are among the causes of neurological deteriorations, and these can be detected by perfusion imaging, nevertheless, the visual assessment alone might not be sufficient, as demonstrated by ours and previous studies showing variable agreement even among expert raters.

Our study has two strengths compared to previous research. First, our raters were blinded to earlier imaging studies and clinical information, a departure from common clinical practice. Despite this, our raters showed moderate agreement in diagnosing vasospasm, with sensitivity similar to previous studies. Second, our larger sample size compared to previous studies enhances the robustness of our findings.

However, our study also has several limitations. Firstly, our control group included patients without aSAH, which introduces incorporation bias that may affect the reported diagnostic accuracy of the visual assessment of perfusion maps to diagnose DCI. Secondly, our study is retrospective, which limits our ability to draw definitive conclusions about whether our assessment algorithm can predict the need for endovascular therapy, as all patients had vasospasm confirmed by angiography and underwent endovascular treatment. Thirdly, despite having a larger sample size compared to previous studies, our overall sample size remains relatively small, which reduces the strength of our findings. Larger studies with a control group of aSAH patients without vasospasm are necessary to confirm and generalize our results.

Conclusions

In summary, our study focused on assessing global perfusion delay patterns to detect DCI in aSAH patients, showed moderate to substantial interrater agreement with angiography-confirmed vasospasm diagnoses. While our findings underscore the limitations of visual assessment alone, particularly in comparison to the higher accuracy demonstrated by experienced neuroimaging specialists, they also highlight the potential of perfusion imaging to diagnose DCI. However, further research is needed to explore quantitative automatic assessment and larger patient cohorts, as these may yield different results and enhance the utility of perfusion imaging in patients with aSAH.