Validating the TeleStroke Mimic (TM) Score: A Prediction Rule for Identifying Stroke Mimics Evaluated over Telestroke Networks

Abstract

Background and Purpose

Up to 30% of acute stroke evaluations are deemed stroke mimics, and these are common in telestroke as well. We recently published a risk prediction score for use during telestroke encounters to differentiate stroke mimics (SM) from ischemic cerebrovascular disease (iCVD) derived and validated in the Partners TeleStroke Network. Using data from 3 distinct US and European telestroke networks, we sought to externally validate the TM Score in a broader population.

Methods

We evaluated the TM Score in 1,930 telestroke consults from the University of Utah, Georgia Regents University, and the German TEMPiS Network. We report the AUC in ROC curve analysis with 95% CI for our previously derived TM score in which lower TM scores correspond with a higher likelihood of being a stroke mimic.

Results

Based on final diagnosis at the end of the telestroke consultation, there were 630/1930 (32.6%) SM in the external validation cohort. All six variables included in the score were significantly different between patients with iCVD vs. SM. The TM Score performed well [AUC 0.72 (95% CI 0.70, 0.73; p<0.001)], similar to our prior external validation in the Partners National Telestroke Network.

Conclusions

The TM Score’s ability to predict the presence of a stroke mimic during telestroke consultation in these diverse cohorts was similar to its performance in our original cohort. Predictive decision-support tools like the TM score may help highlight key clinical differences between mimics and stroke patients during complex, time-critical telestroke evaluations.

Introduction

Telestroke has grown significantly in the past decade and has entered mainstream care for patients with acute stroke.¹ Defined as the use of telecommunication technologies to provide medical information and services to stroke patients,² telestroke has been adopted and implemented by multiple different types of healthcare organizations across the United States and abroad.³ Telestroke enables stroke patients to be remotely evaluated, thereby allowing optimal treatment and management in medically underserved areas, removing geographical disparities in access to expert care.^{1, 4} Decision-analytic models demonstrate that telestroke is cost-effective from both a societal and a hospital perspective.^{5, 6}

While a significant percentage of stroke patients are now provided initial care through telestroke consultations,⁴ its ease of use, greater availability and cost effectiveness have led to a high number of consults by emergency department (ED) physicians. However, it is estimated that 5% to 30% of ED patients suspected of having an acute stroke end up with a diagnosis of a stroke mimic. Seizures, migraine, psychogenic disorders, and toxic/metabolic causes are the most common nonvascular conditions that mimic stroke.^7–14 The use of telestroke consultations to evaluate stroke mimics, and the use of IV tPA in these cases while safe,^{12, 14–16} may lessen the cost effectiveness of this method of stroke evaluation.

We previously performed a retrospective analysis of all the patients managed in our large Partners National Telestroke Alliance program and developed a score to identify stroke mimics based on presenting characteristics over the Telestroke network¹⁷. In our current study, we assessed the validity of our Telestroke Mimics Score (TM Score) across 3 distinct US and European telestroke networks.

Methods

Patient Population

We report data from 3 distinct US and European telestroke networks, namely University of Utah Telestroke Program (UTS) in Salt Lake City, Utah; Augusta University, formerly Georgia Regents University, Telestroke Network (GRU) in Augusta, Georgia and the TeleMedical Project for integrative Stroke Care (TEMPiS) in Bavaria, Germany. Data was collected from the clinical documentation captured in the telestroke registry or case log utilized at each center. Each center uses a different method for data collection, so personnel at each center abstracted the pertinent de-identified variables from their database and shared them with the study team where they were combined for statistical analysis. We requested the data regarding the variables which we previously found to be significantly associated with stroke mimics and used to develop the TM Score. These variables were age in years at time of stroke; medical history at time of consult of atrial fibrillation, hypertension, or seizure disorder; and the binary variables of an initial, telestroke-performed NIH stroke scale score of >14 or ≤ 14 and the presence or absence of facial weakness at presentation. Patients with missing variables from GRU were excluded from the analysis, while at UTS and TEMPiS if history of atrial fibrillation, hypertension, or seizure was not recorded, it was considered not present. The data that support the findings of this study are available from the corresponding author upon reasonable request.

The 3 centers included in the analysis each provide 24 hour coverage of telestroke for community hospitals in their geographic region using real-time videoconferencing and teleradiology. During the study period, UTS had 23 regional spoke hospitals in its network providing care in the state of Utah and the neighboring states of Nevada, Idaho, Wyoming, and Colorado. (Website: https://healthcare.utah.edu/neurosciences/neurology/telestroke-program.php) GRU extends stroke care across the large state of Georgia and has 24 regional hospitals in its network. (http://www.augusta.edu/mcg/neurology/specialties/stroke /telestroke.php) TEMPiS consists of a network of two specialized stroke centers (Munich-Harlaching and the University of Regensburg) and 15 regional hospitals in eastern Bavaria/Germany, detailed descriptions have been published previously. ¹⁸

Ethics approval was obtained by each center before sharing the data and IRB waived the need for patient consent.

Definitions

Cerebrovascular Disease (CVD)

Patients diagnosed with acute ischemic stroke, sub-acute ischemic stroke and transient ischemic attack were grouped together as cerebrovascular disease. There were no cases of telestroke consultation for intracerebral hemorrhage due to exclusion by study criteria.

Stroke Mimics (SM)

This group consisted of patients who were not diagnosed with cerebrovascular disease but had a telestroke consult. Diagnoses of patients in this group included migraine/headache, seizure, encephalopathy, conversion disorder, etc.

TeleStroke Evaluation, Workflow and Diagnostic Classification

The typical workflow associated with telestroke consultation has been described in our prior paper. Briefly, during telestroke consultation the stroke expert has the opportunity to review imaging, examine the patient with the help of a bedside assistant, perform an NIH Stroke Scale (NIHSS), and discuss the details of the presentation and symptoms with the patient and any family or witnesses present at the bedside. At the conclusion of the teleconsultation, the stroke expert assigns a diagnosis based on review of all the available clinical and imaging data. For these analyses, diagnoses were classified as either ischemic cerebrovascular disease (iCVD) if they were acute or subacute ischemic stroke or TIA; otherwise they were assigned a diagnosis of stroke mimic (SM). Hemorrhagic strokes (i.e., subarachnoid and intracerebral hemorrhages) were excluded from the study.

Statistical Analysis

The TM Score which we previously derived and validated in our Partners National Telestroke Network is listed below.

• TM Score = (Age in years multiplied by 0.2) points + 6 points (if history of atrial fibrillation + 3 points (if history of hypertension) + 9 points (if facial weakness present) + 5 points (if NIHSS > 14) – 6 points (if history of seizure disorder)

Based on an inflection point in the NIHSS distribution, a categorical cutpoint of NIHSS > 14 was chosen to construct a binary variable in our derivative analysis.¹⁷ Figure 1 depicts the simple nomogram which was developed for ease of use by physicians during telestroke consult. In our current study, categorical variables were analyzed by Chi-square test and continuous variables by independent sample t-test or Wilcoxon rank sum test to compare mean or median differences. The TM Score for any individual patient was determined by summing the points assigned for each factor present. Model discrimination was assessed by the receiver-operating characteristic (ROC) area under curve (AUC) which is equivalent to the c-statistic.¹⁹ All statistical analyses were performed using SPSS v. 20.

Figure 1.

Nomogram for predicting Telestroke stroke mimics using TM Score.

TM Score = (Age multiplied by 0.2) + 6 (if history of atrial fibrillation) + 3 (if history of Hypertension) + 9 (if facial weakness) + 5 (if NIHSS > 14) – 6 (if history of seizure)

Results

The patient characteristics between stroke mimics and iCVD patients were compared at each site separately and then as a pooled cohort. While a few variables were not significantly different in univariate testing at the site level, likely due to small sample size, they were all significant in the pooled validation cohort. The results for each site are reported individually as follows:

UTS provided data on 187 telestroke consultations, of which 44% were SMs. SM patients had significantly lower NIHSS, less often had NIHSS of > 14, less often had facial weakness on presentation, and more often had a history of seizure; SM were more often, but not significantly, younger and with less atrial fibrillation (Table 1).

Table 1.

Demographics and clinical characteristics of patients with ischemic cerebrovascular disease as compared to those with stroke mimics at University of Utah Telestroke Program.

	iCVD (n=104)	SM (n=83)	P-value
Age (years)	65.6 ± 16.0	61.6 ± 16.1	0.10
NIHSS [median (IQR)]	4 (2, 8)	3 (1, 5)	0.02
NIHSS > 14	12.5%	4.8%	0.05
Facial Weakness	50.0%	21.7%	<0.001
Atrial Fibrillation	16.3%	13.3%	0.55
Hypertension	49.0%	49.4%	0.96
Seizure Disorder	1.9%	9.6%	0.02

TEMPiS provided data on 1,024 telestroke consultations, of which 24% were SMs. SM patients were significantly younger, had lower NIHSS at presentation, less often had NIHSS > 14, less often had facial weakness, and less often had a history of atrial fibrillation or hypertension with more often a history of seizure (Table 2).

Table 2.

Demographics and clinical characteristics of patients with ischemic cerebrovascular disease as compared to those with stroke mimics at Telemedical Project for integrative Stroke Care (TEMPiS) in Germany.

	iCVD (n=775)	SM (n=249)	P-value
Age (years)	73.0 ± 13.9	63.7 ± 18.1	<0.001
NIHSS [median (IQR)]	2 (0, 5)	0 (0, 2)	<0.001
NIHSS > 14	7.6%	0.8%	<0.001
Facial Weakness	30.1%	10.0%	<0.001
Atrial Fibrillation	20.0%	9.6%	<0.001
Hypertension	68.1%	60.6%	0.03
Seizure Disorder	4.1%	9.6%	<0.001

GRU provided data on 719 telestroke consultations, of which 41% were SMs. SM patients were significantly younger, had lower NIHSS at presentation, less often had NIHSS > 14, less often had facial weakness, and less often had a history of atrial fibrillation or hypertension with more often a history of seizure (Table 3).

Table 3.

Demographics and clinical characteristics of patients with ischemic cerebrovascular disease as compared to those with stroke mimics at Georgia Regents University Telestroke Network.

	iCVD (n=421)	SM (n=298)	P-value
Age (years)	67.6 ± 13.6	59.5 ± 16.9	<0.001
NIHSS [median (IQR)]	4 (1, 9)	2 (0, 5)	<0.001
NIHSS > 14	15.7%	9.7%	0.02
Facial Weakness	50.1%	22.1%	<0.001
Atrial Fibrillation	11.6%	6.7%	0.02
Hypertension	72.0%	56.4%	<0.001
Seizure Disorder	1.9%	10.7%	<0.001

When the 3 sites were combined into a pooled validation cohort, all 6 variables included in the TM score were significantly different between SM and iCVD patients (Table 4). Our external validation cohort performed well on ROC curve analysis with an AUC of 0.72 (95% CI 0.70, 0.73; p<0.001). When analyzed separately for each network, AUC were similar across the networks (suppl. Figure I–III). These data confirm that the prediction rule generates an acceptable degree of classification in this population (Figure 2). Of note, the rate of SMs was lower in the TEMPiS cohort than the US sites (24.3% vs. 42%, p=0.005).

Table 4.

Demographics and clinical characteristics of patients with ischemic cerebrovascular disease as compared to those with stroke mimics at all validation centers combined.

	iCVD (n=1300)	SM (n=630)	P-value	Odds Ratio (95% CI)
Age (years)	70.6 ± 14.2	61.5 ± 17.4	<0.001	1.04 (1.03. 1.05)
NIHSS [median (IQR)]	3 (1, 7)	1 (0, 4)	<0.001	1.05 (1.04, 1.07)
NIHSS > 14	10.6%	5.6%	<0.001	2.02 (1.38, 2.96)
Facial Weakness	38.2%	17.3%	<0.001	2.95 (2.33, 3.73)
Atrial Fibrillation	17.0%	8.7%	<0.001	2.14 (1.57, 2.93)
Hypertension	67.8%	57.1%	<0.001	1.58 (1.30, 1.93)
Seizure Disorder	3.2%	10.2%	<0.001	0.30 (0.20, 0.44)

Figure 2.

Response operator curve for the TM Score for external validation data from the 3 telestroke centers.

Discussion

Using 3 distinct US and European telestroke networks, we have shown that a model based on the 6 factors that were associated with SMs allows us to differentiate between SMs and patients with true iCVD on telestroke consults with reasonable model performance. The described predictive model represents a real-time clinical decision-making aid to prompt consideration of the possibility of a SM and is not intended as a diagnostic classification tool.

To our knowledge, this is the largest cohort study to date evaluating SMs in both a national and international telestroke network sample. Rates of SMs presenting at the spokes of our 2 US-based telestroke networks (UTS and GRU) were higher than that reported for patients presenting directly to stroke centers in the US, possibly because community referral hospitals are less experienced or are encouraged to call for consultation more frequently. The rate in TEMPiS was lower, and at the upper end of the reported range of 5–25% SM. In one of the earliest studies on SM patients by Harbison et al., the SM rate was 27% in a series of 487 consecutive patients who were directly admitted to a stroke service over a 6 month period.²⁰ Hand et al reported that among 350 consecutive patients with focal brain dysfunction of sudden onset presenting to an urban teaching hospital, 31% were SMs.⁹ A recent series from a single stroke center found that 27% of patients referred from the ED as a stroke code did not have a cerebrovascular disorder, and the proportion of inpatient SMs among patients admitted for suspected cerebrovascular disease?? was even higher.²¹ The lower rate of SMs in TEMPiS may be because their practice model is different from that in most US telestroke programs, in that patients consulted upon in TEMPiS generally all remain at the referring hospital. Additionally, despite a smaller network of only 15 hospitals, TEMPiS has higher consult volumes and dedicates one quarter of its budget to site education. This combination of factors may result in better recognition and exclusion of likely SMs upfront by the referring providers. While the true cause of this difference is uncertain, it is reassuring that the score still performs well even in a scenario where the incidence of the condition is dramatically lower.

Several clinical factors that predict the presence of SM have been previously identified including that patients with SMs are younger and have fewer vascular risk factors.^{9, 10, 12, 15, 22} The current confirmed this as well while being one of the first to explore these factors in patients managed during telestroke consultations nationally and internationally. They were also observed to have less facial droop and a lower initial median NIHSS. Chang et al. also reported a lower frequency of focal weakness and a lower median NIHSS in SM patients.²³ Commensurate with the available literature, the SM group had a significantly higher percentage of patients with a medical history of seizure. ^{9, 12, 14}

Recognizing patient characteristics that differentiate iCVD patients from SM patients can be extremely useful when evaluating patients during telestroke consults. One interesting finding from our data is the association between absence of facial weakness and a SM diagnosis. This suggests that patients with conditions that mimic stroke often do not have facial weakness, and this may be an important feature to focus on during the evaluation of potential SMs during telestroke evaluation. Still, across the cohort, 17% of patients with SM had facial weakness and thus its presence should not be construed as diagnostic of iCVD. Similarly, the presence of a seizure disorder raises the likelihood that the current symptoms may be due to seizures with a Todd’s paralysis, however, a careful evaluation is still required because some of these patients will have a new iCVD event. Advanced imaging may be useful in these patients to exclude ischemia, though in the acute tPA window, this may be impractical, particularly for a community regional hospital.

As telestroke is rapidly being adopted,⁴ neurologists can anticipate an increased incidence of SM in the population of telestroke consultations. It may be beneficial to have a simple prediction rule which can be used to heighten awareness. However, a major issue with prediction rules is that physicians have found prediction rules difficult to implement in real-time use.²⁴ The prediction rule presented herein is based on the information easily available at the time of the initial ED evaluation, utilizes variables that are intuitive and biologically plausible, and is calculated in a straightforward manner. We have also produced a nomogram, which could be printed and carried on a pocket-sized card and used to estimate the likelihood of a patient being a SM.

There are inherent limitations in the interpretations of the current study design. First, it is a retrospective analysis of prospectively collected datasets. There may be incomplete data capture and inaccuracies in data abstraction, and risk factors may have been abstracted with some variability across sites. Because we only analyzed clinical factors previously identified as predictive of SM, we may have missed some important patient characteristics in this dataset which could be associated with SMs and add discriminating power to our prediction rule. Random measurement error and misclassification can lead to dilution bias and underestimation of the effects of the tested risk factors. However, the factors in our prediction rule were routinely recorded and are relatively unlikely to be systematically influenced by the consulting neurologist judgment; however they might have been omitted if a careful history was not obtained or was not available at the time of consultation. The stroke neurologist’s clinical diagnosis (iCVD vs. SM) is the reference standard used in these analyses to classify patients. We had no alternative method available to further validate this diagnosis since only a fraction of patients are transferred to the hub hospital post-consultation for further evaluation or definitive imaging. Last, it is possible that SMs evaluated in person might have different characteristics then those seen over telestroke where a referring physician has already applied some judgment as to the likelihood of iCVD by initiating the consultation. Therefore, our findings should also be replicated in traditional, in-person environments before being applied in these scenarios. An ongoing analysis is exploring the predictive capacity of our TM score in in-person evaluation.

In conclusion, we believe that as telestroke consultation expands, increasing numbers of patients with SM will be evaluated. These SM patients differ substantially from their counterpart iCVD patients in their vascular risk profiles and other characteristics. Decision-making support tools based on predictive models, like the TM Score we developed and have now externally validated at 3 distinct US and European telestroke networks, may help clinicians consider alternative diagnosis and potentially help identify SMs in a setting with telemedicine support.