JURaSSiC

Abstract

Objective:

We compared the accuracy of clinicians and a risk score (iScore) to predict observed outcomes following an acute ischemic stroke.

Methods:

The JURaSSiC (Clinician JUdgment vs Risk Score to predict Stroke outComes) study assigned 111 clinicians with expertise in acute stroke care to predict the probability of outcomes of 5 ischemic stroke case scenarios. Cases (n = 1,415) were selected as being representative of the 10 most common clinical presentations from a pool of more than 12,000 stroke patients admitted to 12 stroke centers. The primary outcome was prediction of death or disability (modified Rankin Scale [mRS] ≥3) at discharge within the 95% confidence interval (CI) of observed outcomes. Secondary outcomes included 30-day mortality and death or institutionalization at discharge.

Results:

Clinicians made 1,661 predictions with overall accuracy of 16.9% for death or disability at discharge, 46.9% for 30-day mortality, and 33.1% for death or institutionalization at discharge. In contrast, 90% of the iScore-based estimates were within the 95% CI of observed outcomes. Nearly half (n = 53 of 111; 48%) of participants were unable to accurately predict the probability of the primary outcome in any of the 5 rated cases. Less than 1% (n = 1) provided accurate predictions in 4 of the 5 cases and none accurately predicted all 5 case outcomes. In multivariable analyses, the presence of patient characteristics associated with poor outcomes (mRS ≥3 or death) in previous studies (older age, high NIH Stroke Scale score, and nonlacunar subtype) were associated with more accurate clinician predictions of death at 30 days (odds ratio [OR] 2.40, 95% CI 1.57–3.67) and with a trend for more accurate predictions of death or disability at discharge (OR 1.85, 95% CI 0.99–3.46).

Conclusions:

Clinicians with expertise in stroke performed poorly compared to a validated tool in predicting the outcomes of patients with an acute ischemic stroke. Use of the risk stroke outcome tool may be superior for decision-making following an acute ischemic stroke.

Stroke is a leading cause of adult disability and can be devastating for patients and their families. Approximately two-thirds of stroke survivors have long-term functional deficits that can diminish their quality of life.^1,2 Patients and families naturally have questions about prognosis after an acute ischemic stroke. Advanced age, stroke severity, and a cardioembolic mechanism are the most common factors associated with death and disability after ischemic stroke.^3–6 The process underlying clinicians' prognostication is complex and difficult, especially when considering the diversity of stroke syndromes and the multiple factors influencing outcomes.⁷ Clinicians involved in the acute management of patients with ischemic stroke provide outcome predictions based on their assessment of the clinical context and the knowledge they have acquired from previous experience. Although a variety of risk prediction tools are available to estimate the likelihood of death or disability after stroke, they are not commonly used in clinical practice. A better understanding of the accuracy of clinicians' prognosis estimations vs a validated risk score tool compared to observed stroke outcomes could help identify unknown biases and sources of error.

Our objectives were to 1) determine clinician accuracy and consistency in estimating the probability of stroke outcomes, 2) evaluate factors influencing clinician estimations, and 3) compare clinician and validated risk score prognostic estimations.

METHODS

The Clinician JUdgment vs Risk Score to predict Stroke outComes (JURaSSiC) study randomized 111 clinicians with expertise in acute stroke care to predict clinical outcomes of 5 assigned case-based ischemic stroke scenarios.

Participants were enrolled in the study if they were practicing clinicians who provided stroke care in Ontario and were directly involved in medical decision-making during the initial presentation or hospitalization. There was no restriction on age, expertise, years of experience, or level of training.

To reflect actual clinical practice, ischemic stroke cases for scenario development were selected to be representative of the 10 most common clinical presentations (n = 1,415). This was possible by creating a patient profile matrix matched by age, sex, stroke severity, stroke subtype, presence of vascular risk factors, glucose on admission, preadmission status, and risk stratum. These selection conditions were applied to a pool of more than 12,000 ischemic stroke patients participating in the Registry of the Canadian Stroke Network (RCSN).^8,9 The RCSN is a clinical database of patients seen in the emergency department or admitted to hospital with an acute stroke in one of the 12 regional stroke centers in Ontario, Canada. Acute ischemic stroke was confirmed in all registry patients by neuroimaging. Further registry details may be obtained from the RCSN Report at www.rcsn.org and in previous publications.^8,10

The 10 representative clinical scenarios were then divided into 2 matched sets (A and B) to allow a unique but equal allocation of case mix (e.g., low, moderate, and high probability of a favorable or poor outcome) (see case-based allocation strategy/matrix, figure e-1 on the Neurology® Web site at www.neurology.org) with each clinician evaluating 5 cases. Thrombolysis was not included in the case scenarios. Advanced age, high NIH Stroke Scale score (NIHSS), and nonlacunar stroke subtype are well-known factors associated with poorer stroke outcomes.^6,10,11 Cases more likely to have a poor outcome were defined a priori if 2 of the following conditions were present: age >75 years, NIHSS >15, or a nonlacunar stroke subtype. The remaining cases were classified as being less likely to have a poor outcome.

Administration of case-based scenarios.

Clinician predictions were obtained independently in a single session conducted during standard working hours (8 am to 5 pm) in a quiet room. Information on initial stroke severity (NIHSS score) and degree of disability was available and explained at the beginning of the session. The use of electronic devices or Web tools (with the exception of the iScore) was permitted as per each clinician's routine clinical practice. All participants received a group of 5 cases with similar content structure, word count (mean words per case: 45 [range 41–51]), and case details (which were variables represented in the iScore), ranging from low (27%) to high (94.7%) risk of death or disability at discharge (see appendix e-1). There was no imposed time limit; on average, sessions lasted 15 minutes. No incentives were offered. In addition to the 5 cases for evaluation, clinicians were asked to provide minimal demographic information, to provide a few details about their practice, and to report their overall level of confidence with their predictions using a scale ranging from 0 (lack of confidence) to 100 (fully confident).

The iScore was used as the validated risk score tool to estimate the risk of short- and long-term mortality, functional outcomes, and response to thrombolysis after an acute ischemic stroke.^10,12–14 The iScore was calculated based on the provided information for each case scenario. Details of the selection of variables for the iScore, data sources, and the creation and conceptualization of the iScore have been published elsewhere.^10,12 An online Web-based tool (www.sorcan.ca/iscore) and iPhone version are available for clinical use.

Main outcome measures.

Clinicians were asked to estimate the probability of 3 alternatives of outcomes for each case: 1) death or disability at discharge defined as a modified Rankin Scale score (mRS) 3–6 (primary outcome); 2) 30-day mortality (secondary outcome); and 3) death or institutionalization at discharge (secondary outcome). Actual observed poststroke all-cause mortality was obtained through linkages to the Ontario Registered Persons Database (RPDB) at the Institute for Clinical Evaluative Sciences. The RPDB is a population-based administrative database that includes basic demographic data and date of death for all residents in the province.

Statistical analysis.

The probability of death or disability at discharge (as well as other outcomes) for each case was derived from a determined number of ischemic stroke patients matched by age, sex, stroke severity, stroke subtype, risk factors, glucose on admission, preadmission status, and risk stratum. Predictions within the 95% confidence interval (CI) for the actual outcomes were considered accurate. This was calculated using exact binomial CI for proportions from the 1,415 stroke patients. A sensitivity analysis was performed using a relative difference of 30% from the actual outcome criteria. If clinicians provided a range of estimates, the average was used for the primary analysis.

A generalized linear mixed model was used to determine factors associated with clinician accuracy. The variables, which were specified a priori, included clinician specialty, position, number of stroke patients per year, confidence in estimates, clinician age, years in practice, and case risk level. For the binary outcome, success was considered if the clinician estimate for the particular outcome of interest was within the 95% CI based the actual outcome. Because each clinician contributed more than one observation (i.e., case), a mixed-effect model was needed to control for within-clinician correlation. Medians were used for continuous variables for the Forest-style plots. To evaluate the association between case mix and accurate estimations, we categorized cases into high and low likelihood of poor outcome as explained above.

Statistical analysis was performed using R software (version 2.14.0), Vienna, Austria. The R package lme4.0 was used for the mixed-effect models. MegaXL application (Epigear, Wilston, Australia) was used to create the forest plots. All tests were 2-tailed, and p values <0.05 were considered statistically significant.

Standard protocol approvals, registrations, and patient consents.

This study was registered in ClinicalTrials.gov NCT01657279. Approval was obtained from the St. Michael's Hospital ethics review board. As the RCSN was created for improving quality of care, patient consent was waived. Verbal consent was obtained for participating clinicians.

RESULTS

Clinician characteristics.

The 111 clinicians made 1,665 predictions for groups of 5 case scenarios from the pool of 1,415 patients. Four estimations (0.2%) were unreadable or uninterpretable, leaving 1,661 for the analysis. Patient outcomes were based on a mean of 141 (range 37–349) observations of actual patients with an ischemic stroke. Table 1 gives patient and table 2 clinician characteristics. Clinician mean age was 40 ± 12 years; two-thirds were male. Fifty percent were active staff clinicians, including 47 (42%) neurologists, 45 (41%) internists, 11 (10%) emergency clinicians, and 8 (7%) board-certified vascular neurologists. The mean number of stroke patients assessed per clinician/year was 98 (±151); 92 (84%) provided acute stroke care (initial 48 hours).

Table 1.

Characteristics of patients with an ischemic stroke^a

Table 2.

Participants' characteristics

Clinician accuracy.

Of 1,661 clinician estimates, 536 (32.3%) fell within the 95% CI of observed outcomes. When analyzing each outcome separately, clinicians provided estimations within the 95% CI range of actual outcomes in 16.9% of cases for death or disability at discharge, 46.9% for 30-day mortality, and 33.1% for death or institutionalization at discharge (table 3). More than 40% of clinicians (71 of 174) predicted a low chance of death or disability at discharge (range 2%–60%) for cases with high-risk characteristics (observed rate 90%). Similarly, 33% of clinicians (36 of 110) predicted a low chance of death at 30 days (range 0%–10%) for cases with high-risk characteristics (observed rate 25%). There were no significant differences in the estimations provided by clinicians with neurology vs internal medicine training background (accuracy for death or disability: 17.6% [internal medicine] vs 20.8% [neurology]) (table 3).

Table 3.

Outcome measures^a

Among those participants who were not able to provide accurate estimations, most clinicians underestimated the risk of death or disability at discharge as 386 of 461 (83.7%) estimations were below the lower CI of the actual outcome. Clinicians overestimated the risk of death at 30 days (210/554 [37.9%] estimations were above and 84/554 [15.2%] were below the lower CI of the actual outcome). Similar trends were observed in the stratified analysis by case mix (table 3).

There was considerable variability in predictions at the individual clinician level (figure 1). Depending on the case, 70%–100% of clinician estimates were outside the 95% CI of observed outcomes. Similar findings were observed for the secondary outcomes. In contrast, 90% of the iScore-based estimates were within the 95% CI of observed outcomes.

Figure 1. Scatterplot of clinicians' estimations, iScore predictions, and actual outcomes.

Scatterplot of observed outcomes (95% confidence interval [CI]) (red triangle), the iScore (blue X), and clinicians' (black dot) estimations by each stroke case for the outcomes of interest. (A) Death or disability. (B) The 30-day mortality. (C) Institutionalization or death. Each black dot represents the individual clinician's estimation. Multiple black dots horizontally aligned represent more clinicians providing similar estimations. Note the wide range of clinicians' estimations by case, with the great majority outside of the 95% CI of the observed outcome.

Overall, nearly half (n = 53; 47.7%) of clinicians provided primary outcome predictions outside of the 95% CI of actual outcomes in all 5 cases. Less than 1% (n = 1) provided accurate predictions in 4 of the 5 cases and no clinician was accurate for all 5 cases. Only 5.6% (n = 6) of clinicians provided accurate estimates in 4 of 5 cases (none for the 5 cases) for death at 30 days or institutionalization at discharge. Clinicians were slightly better at predicting death at 30 days, with 20.9% (n = 23) able to provide an accurate estimation in 4 or 5 cases.

We found no significant differences in the results by changing the accuracy criteria (within 30% of the actual outcome considered accurate). For the primary outcome, 105 of 555 (18.9%) estimations were accurate. Similar findings with minor differences were observed for the secondary outcomes (data not shown). At the case level, 59.3%–100% of clinicians' estimates were outside the 30% criteria.

Effect of clinician confidence.

The mean confidence level in estimating stroke outcomes was 38% (range 0%–100%). There was no difference in the accuracy of outcome predictions by clinician's confidence level (i.e., higher clinician confidence was not associated with more accurate predictions; p = 0.85; table 3 and figure 2).

Figure 2. Forest plot representing variables associated with clinicians' accuracy.

Forest plot representing variables associated with clinicians' accuracy for outcomes of interest. (A) Death or disability (modified Rankin Scale [mRS] 3–5) at discharge. (B) Death at 30 days. (C) Institutionalization or death. Note that high risk cases were associated with clinicians' outcome predictions. Further details are explained in the text. CI = confidence interval; OR = odds ratio.

Factors independently associated with accurate estimations.

In multivariable analyses, the presence of high-risk patient characteristics (older age, high NIHSS score, and nonlacunar subtype) was the only variable associated with more accurate clinician estimates of 30-day mortality (odds ratio [OR] 2.40, 95% CI 1.57–3.67). Similar associations were observed for death or disability at discharge (OR 1.85; 95% CI 0.99–3.46) (figure 2). The lack of these characteristics was associated with more accurate estimations for noninstitutionalization or survival (OR 1.43, 95% CI 1.13–1.63). Clinician characteristics and higher levels of confidence were not associated with more accurate predictions.

DISCUSSION

The prediction of either a clinical response to a specific intervention or prognosis for individual patients is difficult. The cognitive processes involved in clinical reasoning for prognostication are complex.^15,16 There is limited information on how clinicians make decisions or estimate outcomes. Decisions based on erroneous predictions may result in incorrect patient and family expectations, and potentially inappropriate treatment, counseling, or discharge planning (e.g., longer than expected length of hospitalization, placement).

The findings of our study suggest that clinicians with expertise in stroke care performed poorly in estimating the probability of key clinical outcomes associated with ischemic stroke. For example, only about 1 in 6 (16.9%) clinician estimations for the primary outcome were within the 95% CI of the actual observed outcomes. Further, clinicians more accurately predicted more severe, singular outcomes such as death at 30 days relative to other composite outcomes such as death or disability at discharge. In contrast, a validated risk prediction tool was considerably more accurate (90%) than clinician estimation of the probability of clinical outcomes. Specifically, fewer than 1% of clinicians were accurate in 4 out of 5 cases and none was correct in all cases for prediction of the primary outcome. In the multivariable analysis, the presence of certain high-risk patient characteristics was associated with more accurate clinician predictions. No associations were found for clinician age, type of expertise (neurology vs medicine training), level of training, years of practice, or number of stroke patients seen per year and prediction accuracy.

There are a few studies evaluating the accuracy of clinicians' prognostic estimates, and none compare those predictions with a risk score and actual clinical outcomes. A survey of emergency clinicians and neurologists showed that only 11% (95% CI 0%–22%) could correctly predict the benefit of thrombolysis and only 39% were able to correctly estimate the risk of symptomatic and fatal intracerebral hemorrhage.¹⁷ An observational study including neurovascular fellows assessed outcomes in 66 patients with subarachnoid hemorrhage.¹⁸ Final outcomes were determined by phone interview and dichotomized into good (mRS 0–2) and poor (mRS 3–6). Estimates were only provided by a single fellow during intensive care unit rotation. The same fellow provided estimates for the same patient each day. Overall, clinician accuracy ranged from 78% to 88% with no significant day-to-day variation.¹⁸ Of note, fellows were able to use variables known to predict outcomes including scales such as the Hunt-Hess scale, Fisher score, Glasgow Coma Scale on admission, and need for mechanical ventilation. Similar results in clinician predictions were observed in the SUPPORT study using the APACHE score in critically ill patients.¹⁹

Inaccuracies in clinicians' estimations may occur due to 1) overemphasis on positive findings or minimization of pertinent negative information; 2) disregarding facts inconsistent with a favored hypothesis; 3) misinterpretation of the evidence; or 4) the diverse potential effect of multiple competing factors affecting outcome in different directions. For example, clinicians may overestimate the outcome for patients with a moderate to severe stroke at a young age. Some researchers believe that extreme cases (those with very low or very high probability of achieving a good outcome) can be identified by pattern recognition, whereas intermediary cases need systematic generation and testing of hypotheses.²⁰ Categorizing a patient's presentation as easy or difficult is a function of the knowledge and experience of the clinician.¹⁶ As shown in our study, many diagnostic and prognostic variables are so complex in their interaction that an accurate answer may not be reached solely on personal experience or greater expertise.^21–23

Judgment errors are often ascribed to limitations in our cognitive capacities to deal with imperfect information and the human tendency to adopt shortcuts in reasoning.¹⁶ Our study showed better predictions for death at 30 days among high-risk (vs low-risk) cases. This may be explained by clinicians giving more importance to age, stroke severity, and stroke subtype, the 3 most common factors influencing stroke outcomes. Alternatively, clinicians may overestimate the probability of a more serious condition, as patients and their families may experience a “relief” reaction if the outcome is better than expected.

Our study has limitations and strengths. First, it is possible that the absence of some variables (e.g., imaging data) from the case scenarios may have contributed to clinician inaccuracies. Predictions, however, were more accurate with the validated risk tool, even though it does not include imaging criteria. Second, case scenarios reflected the clinical situation early after hospital admission. Clinician accuracy may improve in a different setting or period (e.g., prior to discharge). Moreover, subspecialization in medicine may have contributed to decrease clinicians' prognostic ability for specific conditions as some clinicians may not follow patients after discharge. Third, the statistical stability of regression models may be limited by the scarcity of data.

Strengths of our study include a random allocation of clinicians to cases, a large number of predictions based on the most common “real-world” stroke case scenarios reflecting actual clinical practice, selection of clinically relevant outcomes, diversity of clinician expertise, and the concurrent comparison of clinicians' estimations with a validated risk score and actual stroke outcomes.

Clinicians may wonder whether our results offer a practical means of more accurately predicting patient prognosis compared to their clinical judgment. A recent systematic review concluded that computerized clinical decision support systems improve clinician performance.²⁴ Our results suggest that prognosis in ischemic stroke patients may not be accurately estimated on clinical judgment alone. Previous studies^6–9 in conjunction with our results suggest that the use of a validated clinical risk score may improve clinician performance and assist in providing more accurate prognostic information when counseling stroke patients and their families.

GLOSSARY

CI

confidence interval

JURaSSiC

Clinician JUdgment vs Risk Score to predict Stroke outComes study

mRS

modified Rankin Scale score

NIHSS

NIH Stroke Scale

OR

odds ratio

RCSN

Registry of the Canadian Stroke Network

RPDB

Ontario Registered Persons Database

AUTHOR CONTRIBUTIONS

Gustavo Saposnik: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data, statistical analysis, study supervision. Robert Cote: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval. Muhammad Mamdani: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, study supervision, obtaining funding. Stavroula Raptis: drafting/revising the manuscript, study concept or design, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, acquisition of data, study supervision. Kevin Thorpe: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, statistical analysis. Jiming Fang: analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval, statistical analysis. Donald Redelmeier: drafting/revising the manuscript, study concept or design, accepts responsibility for conduct of research and will give final approval. Larry B. Goldstein: drafting/revising the manuscript, analysis or interpretation of data, accepts responsibility for conduct of research and will give final approval.

STUDY FUNDING

Supported in part by the Heart and Stroke Foundation of Canada (HSFC) and the Ontario Ministry of Health. Role of the funding source: Dr. Saposnik is supported by the Distinguished Clinician-Scientist Award from the HSFC.

DISCLOSURE

G. Saposnik is supported by the Distinguished Clinician Scientist Award from the Heart & Stroke Foundation of Canada. R. Cote, M. Mamdani, S. Raptis, K. Thorpe, J. Fang, D. Redelmeier, and L. Goldstein report no disclosures. Go to Neurology.org for full disclosures.