Pooled Assessment of Computed Tomography Interpretation by Vascular Neurologists in the STRokE DOC Telestroke Network

Abstract

Background and Purpose

The objective of this pooled analysis was to determine level of agreement between central read and each of two groups (spoke radiologists and hub vascular neurologists) in interpreting head computed tomography (CT) scans of stroke patients presenting to telestroke network hospitals.

Methods

The Stroke Team Remote Evaluation Using a Digital Observation Camera (STRokE DOC and STRokE DOC-AZ TIME) trials were prospective, randomized, outcome-blinded trials comparing telemedicine and teleradiology to telephone-only consultations. In each trial, the subjects’ CT scans were interpreted by the hub vascular neurologist in the telemedicine arm, and by the spoke radiologist in the telephone arm. We obtained a central read for each CT using adjudicating committees blinded to treatment arm and outcome. The data were pooled and results reported for the entire population. Kappa statistics and exact agreement rates were used to assess interobserver agreement for radiographic contraindication to recombinant tissue plasminogen activator (rt-PA), presence of hemorrhage, tumor, hyperdense artery, acute stroke, prior stroke, and early ischemic changes.

Results

Among 261 analyzed cases, the agreement with central read for presence of radiological rt-PA contraindication was excellent for hub vascular neurologist (96.2%, κ=0.81, 95%CI 0.64–0.97), spoke radiologist report (94.7%, κ=0.64, 95%CI 0.39–0.88), and overall (95.4%, κ=0.74, 95%CI 0.59–0.88). For rt-PA treated patients (N=65), overall agreement was 98.5%, and vascular neurologist agreement with central read was 100%.

Conclusions

Both vascular neurologists and reports from spoke radiologists had excellent reliability in identifying radiologic rt-PA contraindications. These pooled findings demonstrate that telestroke evaluation of head CT scans for acute rt-PA assessments is reliable.

Introduction

Approximately 795,000 people experience a new or recurrent stroke every year, making stroke the 4^th leading cause of death in America.[1] Only 3–4% of stroke patients, however, receive thrombolysis.[2] Many rural and suburban hospitals may not be fully equipped to administer thrombolysis, and many do not have stroke units and/or specialty trained stroke practitioners, posing barriers to treatment.[3] Telestroke networks have been established to provide access to specialized vascular neurologists in order to improve patient outcomes.[4, 5]

As the window for thrombolytic treatment is limited, and the CT scan is instrumental in determining acute treatment eligibility, [6, 7] it is important to have immediate radiologic interpretation to determine patient eligibility for thrombolysis. There are multiple methods by which spoke hospitals may obtain immediate radiographic head CT scan evaluations when assessing patients for rt-PA administration. Hospitals may have local radiologists at their spoke center available to interpret and report preliminary findings to the emergency department in real time, or may have alternative methods such as Nighthawk or other 3^rd party vendors available to perform these reads in real time. In acute stroke management, stroke specialists often independently review and interpret head CT scans of stroke patients for purposes of making acute rt-PA treatment decisions. As such, the reliability of non-radiologist acute stroke CT scan interpretations has been questioned for these real time rt-PA cases, in standard practice and for telestroke practices. The American Heart Association/American Stroke Association guidelines emphasize the need for high-quality studies assessing agreement of radiologists and nonradiologists engaged in emergency telestroke decision-making.[6] To our knowledge, there are few studies that adequately analyze the reliability of real-time CT scan interpretation by stroke neurologists via telemedicine.[8–11] In two prior clinical telestroke trials, [12, 13] CT scans were interpreted both by stroke neurologists and by local spoke radiologists, allowing an opportunity to assess the much needed question of reliability. The objective of this pooled analysis was to determine the level of agreement between a central read and each of two groups - spoke hospital radiologists and hub hospital vascular neurologists - in interpreting real time head CT scans of patients with acute stroke.

Methods

The Stroke Team Remote Evaluation Using a Digital Observation Camera trials (STRokE DOC and STRokE DOC-AZ TIME) were prospective, randomized, outcome-blinded trials comparing audiovisual telemedicine and teleradiology stroke evaluations to telephone-only consultations. The methodology and results of these studies have been previously published.[12–17] These two trials assessed eligibility for IV rt-PA and correctness of decision-making at two hubs (San Diego and Phoenix) and six spokes. In both studies, the subjects’ CT scans were interpreted by the hub vascular neurologist in the telemedicine arm, and by the spoke radiologist in the telephone arm. In the telephone arm, the spoke radiologist report was viewed by the emergency room physician, and read verbatim to the consulting vascular neurologist and documented in the case report form (CRF). There was no communication between the vascular neurologist and the interpreting neuroradiologist.

In each of the two original STRokE DOC trials, a central read was then performed for each CT scan by adjudicating committees blinded to treatment arm and outcome. The original manual of procedures for the STRokE DOC trial describes the convening of a central adjudicating committee to act as a consensus gold standard review body. These committees were to be made up of both fellowship trained stroke specialists, who often read CT scans for acute rt-PA decisions, and board certified neuroradiologists. Slight methodological differences were present between the two trials regarding makeup of the central adjudicating committee due to resource availability. In the Mayo Clinic, Phoenix, Arizona trial, the consensus interpretation was rendered by two neuroradiologists, while the consensus opinion for the University of California, San Diego, California trial was rendered by three fellowship trained stroke specialists and one board certified neuroradiologist (with this vote weighted double). Final decision in each trial was made by consensus vote. Similar to standard acute stroke radiologic interpretations, the central adjudicating committees had only brief one-line statements of CT-indication, such as “right hemiparesis and aphasia,” in a purposeful design to accurately reproduce actual practice.

The data were pooled and results reported for the entire population. Baseline characteristics were compared between the San Diego, California and Phoenix, Arizona hub sites to assess for significant heterogeneity and ensure the data could be appropriately pooled and reported as a large sample size. Kappa statistics with 95% confidence intervals and exact percent agreement rates were used to assess interobserver agreement. Primary outcome was radiologic contraindication to rt-PA, which was determined by presence of any intracranial hemorrhage, brain neoplasm, or prominent early ischemic changes (EIC) exceeding one third of the middle cerebral artery territory. Secondary outcomes included presence of intracerebral hemorrhage (ICH), subarachnoid hemorrhage (SAH), subdural hematoma (SDH), tumor, hyperdense artery, prior stroke, edema, and EIC. Statistical software R version 2.14.0was used for all statistical analyses.

Results

Of 277 patients who were randomized originally, 261 (130 telestroke and 131 spoke radiology) DICOM images and corresponding CRFs were available for analysis. Patients were excluded if the baseline CT scan was not available (n=13) or in a few cases if linked CRFs were not available to provide the one-line clinical history to the central adjudication body (n=3). The baseline characteristics and risk factors have been previously reported.[13] The baseline characteristics of both the Arizona and San Diego trial groups were first compared to assess for heterogeneity. As expected, significant difference between the two groups was found for ethnicity (more Hispanic in San Diego). There was also a higher proportion of patients with pre-stroke mRS of 3–6 in the much larger San Diego trial. Agreement rates and κ statistics for each arm, as compared to central read, are presented in the table. Overall agreement with central read and proportions of CT scans with particular findings are also noted.

Table 1.

Presence of Radiographic Characteristics on CT Scans and Agreement Between Central Read, Vascular Neurologist, and Spoke Radiologist

Radiologic Feature	Telestroke Arm (N=130)			Spoke Radiology Arm (N=131)			Overall (N=261)
	Feature Present, No. (%)*	% Agree	Kappa (95% CI)	Feature Present, No. (%)*	% Agree	Kappa (95% CI)	Feature Present, No. (%)*	% Agree	Kappa (95% CI)
Normal Scan	38 (29)	75	0.35(0.17,0.53)	30 (23)	77	0.5(0.36,0.65)	68 (26)	76	0.43(0.31,0.54)
Acute Stroke	33 (25)	75	0.29(0.11,0.48)	30 (23)	78	0.22(0.02,0.41)	63 (24)	76	0.26(0.13,0.4)
Chronic Stroke	53 (41)	86	0.71(0.58,0.83)	52 (40)	79	0.53(0.39,0.68)	105 (40)	83	0.62(0.53,0.72)
Hyperdense Artery	13 (10)	94	0.66(0.44,0.88)	15 (11)	89	0.17(−0.08,0.42)	28 (11)	91	0.44(0.25,0.63)
Intraparenchymal Hemorrhage	10 (8)	99	0.95(0.85,1.00)	6 (5)	99	†	16 (6)	99	0.91(0.81,1.01)
Subarachnoic Hemorrhage	3 (2)	99	†	4 (3)	97	†	7 (3)	98	†
Tumor	1 (1)	100	†	1 (1)	98	†	2 (1)	99	†
Subdural Hematoma	0 (0)	100	†	1 (1)	100	†	1 (0)	100	†
Contraindication to tPA	12 (9)	96	0.81(0.64,0.97)	8 (6)	95	0.64(0.39,0.88)	20 (8)	95	0.74(0.59,0.88)

^*Reflects the presence of the feature observed by the central read.

^†When disease prevalence is very high or very low, the Kappa values are decreased relative to the percentage of agreement.

The overall agreement for presence of radiological contraindications to thrombolysis was found to be excellent (95.4%, κ=0.74, 95%CI 0.59–0.88). Agreement was slightly higher for hub vascular neurologist’s direct read (96·2%, κ=0.81, 95%CI 0.64–0.97) than for spoke radiologist’s initial readout to the ED (94.7%, κ=0.64, 95%CI 0.39–0.88) when each was compared to the central adjudication read.

In acute stroke management, there are particular areas of practice variability that make assessing for rt-PA inclusion/exclusion more complex, and warrant well-trained specialists with significant experience making these treatment decisions. One such clinical example is the assessment of the degree of “mild deficit” which would be a contraindication to rt-PA treatment. Similarly, there are areas of radiologic interpretation that have this same variability. There may still be disagreement between equally qualified stroke specialists regarding the distinction between severe EIC and frank hypodensity (FHD), which would potentially constitute a radiographic contraindication to rt-PA treatment. As an exploratory analysis, we excluded 20 patients in whom the degree of EIC was the sole reason for potential radiographic contraindication to rt-PA. In this subgroup analysis, the agreement was even stronger (98.3%), with 100% agreement between vascular neurologist and central read regarding contraindication to rt-PA. For the rt-PA treated subset (N=65), overall agreement was 98.5%, and vascular neurologist agreement with central read again reached 100%.

Secondary outcomes included vascular neurologist and spoke radiologist exact percent agreement with central read on the presence of: normal scan (74.6%, 77.1%), acute stroke (74.6%, 77.9%), intracerebral hemorrhage (99.2%, 98.5%), subarachnoid hemorrhage (98.5%, 96.9%), subdural hematoma (100%, 100%), tumor (100%, 97.7%), and hyperdense artery (93.8%, 88.5%).

Discussion

In the context of a combined telestroke network, designed to assess patients with acute stroke syndromes presenting to community hospitals, this pooled analysis found that both vascular neurologists and reports from spoke radiologists had excellent reliability in identifying radiologic rt-PA contraindications. This effect was also seen in the rt-PA treated subset. This analysis shows that both vascular neurologist and local spoke radiologist techniques are able to assess acute stroke code head CT scans accurately in real time.

The agreement was slightly higher for vascular neurologists, which was perhaps due to methodology. In the local CT read arm, the CRF documentation was of the “wet read” given by the local radiologist to the emergency department, and may not have included all subtle findings unrelated to whether the scan had rt-PA contraindications. Several reports have demonstrated reliability of teleradiology CT scan interpretation by stroke neurologists, though these studies have been limited by small sample size, [9] failure to record the vascular neurologist read in real time, [8] or combining stroke neurologists’ interpretation with second opinions from local radiologists.[11] This study adds to the recently published data from the Arizona STRokE DOC-AZ TIME study, [10] by analyzing a significantly greater number of patients, and reporting the entire pooled analysis (total N=261).

Early ischemic changes seen on initial head CT may present higher risk of post-thrombolysis hemorrhage[18], however, the threshold for extent of early changes necessary for contradiction of treatment is still controversial. While studies have shown that stroke neurologists can reliably determine the extent of EIC using Alberta Stroke Program Early CT Score (ASPECTS) [19], data is mixed regarding reliable interpretation via teleradiology.[8, 11] One recent study reported that “8 of 9 clinically relevant discrepant CT findings were due to underestimation of the extent of EIC” by vascular neurologists.[11] At least 20 patients in our study had EIC that were significant enough to warrant possible contraindication to rt-PA; when such patients were excluded in our exploratory analysis, the agreement between vascular neurologists and central read regarding contraindication to rt-PA was perfect. This highlights the need for a method to accurately assess EIC, as well as a framework for interpreting the assessment in the context of treatment decisions. Because early ischemic changes (EIC) should not be a contraindication to rt-PA, the limited reliability in determining the presence of EIC may not be relevant to treatment decisions. However, there is still controversy regarding the distinction between severe EIC and frank hypodensity (FHD), which would potentially constitute a radiographic contraindication to rt-PA treatment. Further research regarding these distinctions may still be warranted.

One limitation of this study is the low incidence of secondary endpoints such as tumor, subdural hematoma, and SAH that were observed in these two trials, which limited the applicability of the Kappa statistic and provided fewer opportunities to statistically assess differences among the groups. Another limitation is that the vascular neurologists may have received more information than the spoke radiologists or central read, which may have given vascular neurologists a diagnostic advantage. However, the approach is most representative of actual practice, where the treating physician has the full clinical picture, while the radiologists have only the one-line indication entered in the CT scan request. These pooled analysis findings of the STRokE DOC-AZ TIME and STRokE DOC clinical trials demonstrate, in a much larger cohort, that vascular neurologists are able to effectively and immediately interpret head CT scans for the purpose of determining appropriateness of rt-PA administration in the setting of acute stroke.