Abstract
Introduction
We sought to determine the feasibility and validity of estimating post-stroke outcomes using information available in the electronic medical record (EMR) through comparison with outcomes obtained from telephone interviews.
Methods
The Greater Cincinnati Northern Kentucky Stroke Study is a retrospective population-based epidemiology study that ascertains hospitalized strokes in the study region. As a sub-study, we identified all ischemic stroke patients who presented to a system of 4 hospitals during the study period 1/1/2015–12/31/2015 and were discharged alive. Enrolled subjects (or proxies for cognitively-disabled patients) were contacted by telephone at 3 and 6 months post-stroke to determine current place of residence and two functional outcomes—the modified Rankin Score (mRS) and the EuroQol (EQ-5D). Concurrently, the lead study coordinator, blinded to the telephone assessment outcomes, reviewed all available EMRs to estimate outcome status. Agreement between outcomes estimated from the EMR with “gold-standard” data obtained from telephone interviews was analyzed using the kappa statistic or interclass correlation (ICC), as appropriate. For each outcome, EMR-determined results were evaluated for added value beyond the information readily available from the stroke hospital stay.
Results
Of 381 ischemic strokes identified, 294 (median [IQR] age 70 [60–79] years, 4% black, 52% female) were interviewed post-stroke. Agreement between EMR and telephone for 3-month residence was very good (kappa=0.84, 95% CI 0.74–0.94), good for mRS (weighted kappa=0.75, 95% CI 0.70–0.80), and good for EQ-5D (ICC=0.74, 95% CI 0.68–0.79). Similar results were observed at 6 months post stroke. At both 3 and 6 months post stroke, EMR-determined outcomes added value in predicting the gold standard telephone results beyond the information available from the stroke hospitalization; the added fraction of new information ranged from 0.25 to 0.59.
Conclusions
Determining place of residence, mRS, and EQ-5D outcomes derived from information recorded in the EMR post-stroke, without patient contact, is feasible and has good agreement with data obtained from direct contact. However, we note that the level of agreement for mRS and EQ-5D was higher for proxy interviews and that the EMR often reflects health care providers’ judgments that tend to overestimate disability and underestimate quality of life.
Keywords: Ischemic stroke, outcome, modified Rankin, EuroQol, residence, electronic medical record, telephone assessment
Introduction
Stroke survivors commonly face enduring disabilities from the initial stroke event.[1] Long-term outcomes, such as living arrangement, functional status, and quality of life are important and meaningful for patients and their families as they transition from the acute care setting. However, in the United States, long-term outcomes are rarely measured consistently after discharge.[2] There has been a recent push for more patient-centered outcomes in the improvement process of healthcare policies and systems of care,[3] but person-to-person contact with stroke patients post discharge can be difficult due to the complexity of transitions of care and changes in living arrangements associated with stroke disabilities. With the advancement of the electronic medical record (EMR) and health information exchange systems, enormous amounts of patient data are being captured and increasingly used beyond clinical applications.[4,5] The National Institutes of Health Stroke Scale (NIHSS) and several other stroke assessment scales have been shown to have acceptable validity and reliability when derived using data from the EMR.[6,7,8]
We sought to determine the level of agreement of outcomes about place of residence, functional impairments, and quality of life derived from EMR data with “gold standard” responses obtained through telephone interview, and to quantify the added value of the information obtained from the EMR over variables readily available during the stroke hospital stay.
Methods
The Greater Cincinnati Northern Kentucky Stroke Study (GCNKSS) is a retrospective population-based epidemiology project that ascertains hospitalized strokes via ICD-discharge codes. Methods have been described in detail elsewhere.[9,10] As a sub-study, we identified all acute ischemic stroke patients who presented to one healthcare system of 4 community hospitals during the period 1/1/2015–12/31/2015, were discharged alive, and were contacted via telephone calls at 3 and 6 months post-stroke to determine current place of residence and measures required for completion of the modified Rankin Score (mRS) and the EuroQol (EQ-5D). Place of residence was defined as the location where the patient was listed to live which can be the patient’s own home, apartment, relative’s home, a facility or type of institution such as nursing home, skilled nursing facility, or rehabilitation facility. To be eligible, the patient had to pass a cognitive screen. If the patient failed, a proxy could complete the interview (see Supplemental Information for additional details). Concurrently, the lead study coordinator, blinded to telephone results, reviewed all available EMR records of healthcare encounters through one week beyond the corresponding post-stroke time point of interest (3 and 6 months post-stroke) to estimate outcome measures. At the time of the 2015 data collection study period, this lead study coordinator had over 17 years of experience with research and 17 years with medical record review by serving as the study coordinator for the GCNKSS. Training for deriving outcomes from the EMR consisted of piloting the EMR data collection form on 27 patients along with consensus agreement through discussions with the study Principle Investigators on all records. On average, the EMR review took 30 minutes per patient per time point. Institutional review boards of all involved research institutions and hospitals approved the study protocol.
Data Analyses
Because the focus of this project was on stroke survivors, patients who died prior to the interview period of interest were not included in the analysis. Place of residence was dichotomized as home (including living with relatives or friends) versus not home. The five dimensions of the EQ-5D (mobility, self-care, usual activities, pain/discomfort, and anxiety/depression) were converted into a utility index that ranges between −0.11 and 1 by applying United States weights.[11] Place of residence, mRS, and EQ-5D estimated using EMR data were compared with the gold-standard responses from telephone interviews at both 3 and 6 months post-stroke. Agreement between EMR and telephone data was assessed using Cohen’s kappa statistic for place of residence, weighted kappa statistic for ordinal mRS, intraclass correlation (ICC) for the EQ-5D utility index, and weighted kappa statistic for the individual EQ-5D domains. The strength of agreement based on the kappa/ICC statistics was classified as poor (<0.20), fair (0.21–0.40), moderate (0.41–0.60), good (0.61–0.80), and very good (0.81–1.00).[12] For outcome measures that did not reach “very good” agreement overall, agreement measures were computed stratified by telephone interviewee (patient or proxy).
Demographic and clinical characteristics were compared between patients with consistent versus inconsistent telephone and EMR outcomes. An inconsistent result was defined, in three ways according to our predefined outcomes of interest: 1) as telephone not equal to EMR for place of residence, 2) a >1 point difference for mRS, and 3) a >0.24 point difference, twice the minimal important difference, for EQ-5D.[13]
To determine if there was added value in reviewing the EMR at 3 and 6 months post-stroke beyond just using information readily available from the stroke hospital stay, measures of added value were computed and compared for each outcome at each time point. For binary outcome (place of residence) and ordinal outcome (mRS), the C-index, Somers’ D, and the likelihood ratio chi-square value (LR χ2) were computed. For continuous outcome (EQ-5D index), the mean squared prediction error, mean absolute percentage error, and adjusted R2 were computed. Added value was assessed by comparing indexes from the base model, which included only information available from the stroke hospital stay, to the larger model, which also contained the EMR-determined outcome at the same time point as the telephone-determined outcome, 3 or 6 months post-stroke. The place-of-residence base model included age, sex, retrospectively-determined NIHSS at presentation (rNIHSS) [14], and place of residence at discharge. The mRS base model included age, history of diabetes, severe white matter disease, rNIHSS, mRS prior to stroke, and mRS at discharge. [15] The EQ-5D base model included age, diabetes, history of depression, mRS prior to stroke, and mRS at discharge. The fraction of new information from the larger model (the proportion of variation explained by the EMR-determined outcome) was calculated, along with a test comparing the models (LR tests for place of residence and mRS outcomes, and the t-test for EQ-5D). In addition to the added value analysis, we compared the test characteristic measures (C-index, accuracy, sensitivity, specificity, false positive, false negative) for the outcome measures with and without EMR information by using cut points for classification. We dichotomized mRS at 0-2 versus 3-5 and EQ-5D at ≥0.83 versus <0.83 corresponding to utility weight values. [16] We also dichotomized the predicted probabilities at ≥0.50 versus <0.50 that were obtained from the discharge variables only model and the model adding EMR at each specific time point (3 and 6 months post-stroke).
Results
A total of 381 adult (age≥18 years) acute ischemic stroke patients discharged alive from this healthcare system were identified for the year 2015, of which 294 were interviewed post-stroke (293 at 3 months post-stroke, and 217 at 6 months post-stroke). See Supplemental Figure 1 for documentation of reasons for non-participation and other missing data. The median age of the 294 individuals was 70 years (IQR 60–79), 4% were black, and 52% were female (Table 1).
Table 1.
Demographics and stroke risk factors of the 2015 ischemic stroke outcome cohort by outcome data availability
| Characteristic | Ischemic Cohort (n=381) |
Unable to Interview (n=87) |
Outcome cohort (n=294) |
3-mo Cohort (n=293) |
6-mo Cohort (n=217) |
|---|---|---|---|---|---|
| Age, median (IQR) | 69 (60-79) | 67 (57-81) | 70 (60-79) | 70 (60-79) | 71 (61-79) |
| Race (black) | 18 (5%) | 5 (6%) | 13 (4%) | 13 (4%) | 8 (4%) |
| Sex (female) | 195 (51%) | 42 (52%) | 153 (52%) | 153 (52%) | 112 (52%) |
| Pre-stroke mRS, median (IQR) | 1 (0 to 2) | 1 (0 to 3) | 1 (0 to 3) | 1 (0 to 3) | 1 (0 to 2) |
| Hypertension | 309 (81%) | 71 (82%) | 238 (81%) | 238 (81%) | 174 (80%) |
| Diabetes | 159 (42%) | 44 (51%) | 115 (39%) | 115 (39%) | 81 (37%) |
| Current smoking | 114 (30%) | 28 (32%) | 86 (29%) | 85 (29%) | 59 (27%) |
| Atrial fibrillation | 92 (24%) | 24 (28%) | 68 (23%) | 68 (23%) | 52 (24%) |
| CAD | 144 (38%) | 33 (38%) | 111 (38%) | 111 (38%) | 83 (38%) |
| CHF | 75 (20%) | 23 (26%) | 52 (18%) | 52 (18%) | 34 (16%) |
| Prior Stroke | 68 (18%) | 16 (18%) | 52 (17%) | 52 (18%) | 35 (16%) |
| rNIHSS total score, median (IQR) | 2 (1-5) | 2 (1-7) | 2 (1-5) | 2 (1-5) | 2 (1-4) |
IQR indicates interquartile range; CAD, coronary artery disease; CHF, congestive heart failure; rNIHSS, retrospective National Institutes of Health Stroke Scale.
Place of Residence
Agreement between EMR and telephone place of residence was very good at 3 months (kappa=0.84, 95% CI 0.74–0.94) and 6 months (kappa=0.87, 95% CI 0.75–1.00) (Supplemental Table 1).
Modified Rankin Score (mRS)
The distribution of mRS at discharge, 3 months, and 6 months post stroke is shown in Figure 1. The EMR reviewer tended to score disability higher than what was reported by telephone interviews. The 3-month EMR median mRS (IQR) was 3 (2–4) versus 2 (1–4) by telephone, and the 6-month median (IQR) was 3 (1.5–3) versus 2 (1–3). There was good agreement between EMR and telephone for mRS at both 3 months (weighted kappa=0.75, 95% CI 0.70–0.80) and 6 months (weighted kappa=0.71, 95% CI 0.65–0.78). Agreement at specific levels of the mRS showed improved agreement at the tails of the distribution (0 and 5) compared with scores in the middle (1, 2, 3, 4). In particular, the mRS score of 2 had poor agreement (kappa=0.05 at 3 months, and 0.12 at 6 months) (Supplemental Table 2). For cases with telephone interview done by proxy, agreement was good (weighted kappa=0.77 at 3 months, and 0.66 at 6 months); it was only moderate for patient interviews (weighted kappa=0.54 at 3 months, and 0.51 at 6 months) (Table 2).
Fig 1.
Marginal distribution for A) modified Rankin scores and B) EQ-5D
Table 2.
Agreement between EMR and telephone interview for modified Rankin score and EQ-5D at 3 and 6 months post-stroke
| mRS | EQ-5D | |||||||
|---|---|---|---|---|---|---|---|---|
| 3 months | 6 months | 3 months | 6 months | |||||
| n | wKappa (95% CI) | n | wKappa (95% CI) | n | ICC (95% CI) | n | ICC (95% CI) | |
| Overall | 293 | 0.75 (0.70-0.80) | 216 | 0.71 (0.65-0.78) | 293 | 0.74 (0.68-0.79) | 216 | 0.66 (0.58-0.73) |
| Subgroup | ||||||||
| Interviewee – Proxy | 100 | 0.77 (0.67-0.86) | 60 | 0.66 (0.51-0.80) | 100 | 0.70 (0.59-0.79) | 60 | 0.67 (0.51-0.79) |
| Interviewee – Self | 193 | 0.54 (0.43-0.64) | 156 | 0.51 (0.39-0.63) | 193 | 0.57 (0.47-0.66) | 156 | 0.42 (0.28-0.54) |
wKappa = weighted Kappa, ICC=intraclass correlation coefficient.
EuroQol (EQ-5D)
The distribution of EQ-5D at 3 months and 6 months post stroke is shown with boxplots in Figure 1. The EMR reviewer tended to score quality of life lower than what was reported by telephone interviews. The 3-month EMR median EQ-5D (IQR) was 0.78 (0.60–0.83) versus 0.81 (0.60–0.85) by telephone, and the 6-month median (IQR) was 0.78 (0.69–0.84) versus 0.83 (0.71–1.00). There was good agreement between EMR and telephone for EQ-5D at both 3 months (ICC=0.74, 95% CI 0.68–0.79) and 6 months (ICC=0.66, 95% CI 0.58–0.73).
Agreement for specific domains of the EQ-5D was only fair for pain and anxiety domains, with better agreement for mobility, self-care, and usual activities domains (Supplemental Table 4). For cases with telephone interview done by proxy, agreement was good (ICC=0.70 at 3 months and 0.67 at 6 months); it was only moderate for patient interviews (ICC=0.57 at 3 months and 0.42 at 6 months) (Table 2).
Inconsistent Results
Few patients (10 at 3 months and 4 at 6 months) had inconsistent results between telephone and EMR for place of residence and no substantial differences in demographic and clinical characteristics were noted between those with consistent versus inconsistent findings for this outcome (Supplemental Table 5). For mRS, patients with inconsistent results were less likely to have discharge mRS at the extremes (mRS 0, 4, and 5) compared with those with consistent findings. For EQ-5D, patients with inconsistent results had higher pre-stroke functional disability compared with those with consistent results.
Added value
At both 3 and 6 months post stroke there is very strong evidence that EMR-determined outcomes at each time point added value in predicting the gold standard telephone interview outcomes responses beyond the variables obtained during the stroke hospital stay (Table 3). All statistical indexes of added value indicated improvement when EMR outcome was added to the model. The fraction of new information from EMR was 0.54 and 0.59 for place of residence, 0.38 and 0.31 for mRS, and 0.31 and 0.25 for EQ-5D, at 3 and 6 months, respectively. In comparing the test characteristic measures, results were similar in that EMR added value for all outcome measures at both time points, 3 and 6 months post-stroke, although for EQ-5D the increase in the C-index with EMR added was not statistically significant at 6 months (Supplemental Table 6).
Table 3.
Summary of the added value provided by EMR record review in estimating post-stroke Outcomes
| Statistical Indexes of Added Value from EMR | |||
|---|---|---|---|
| Place of residence | |||
| Place of residence – 3 months | C-index | Somers’ D | LR χ2 |
| Discharge variables onlya | 0.84 | 0.68 | 63.71 |
| Discharge variablesa + EMR at 3 months | 0.94 | 0.89 | 139.84 |
| Fraction of new information from EMR | - | - | 0.54 (p-value <0.01) |
| Place of residence – 6 months | C-index | Somers’ D | LR χ2 |
| Discharge variables onlya | 0.83 | 0.66 | 38.58 |
| Discharge variablesa + EMR at 6 months | 0.98 | 0.95 | 87.12 |
| Fraction of new information from EMR | - | - | 0.59 (p-value <0.01) |
| mRS | |||
| mRS – 3 months | C-index | Somers’ D | LR χ2 |
| Discharge variables onlyb | 0.77 | 0.53 | 174.81 |
| Discharge variablesb + EMR at 3 months | 0.85 | 0.70 | 281.67 |
| Fraction of new information from EMR | - | - | 0.38 (p-value <0.01) |
| mRS – 6 months | C-index | Somers’ D | LR χ2 |
| Discharge variables onlyb | 0.75 | 0.51 | 136.12 |
| Discharge variablesb + EMR at 6 months | 0.83 | 0.65 | 196.13 |
| Fraction of new information from EMR | - | - | 0.31 (p-value <0.01) |
| EQ-5D | |||
| EQ-5D – 3 months | MSPE | MAPE | Adjusted R2 |
| Discharge variables only c | 0.038 | 50% | 0.41 |
| Discharge variables c + EMR at 3 months | 0.026 | 38% | 0.59 |
| Fraction of new information from EMR | - | - | 0.31 (p-value <0.01) |
| EQ-5D – 6 months | MSPE | MAPE | Adjusted R2 |
| Discharge variables only c | 0.028 | 31% | 0.39 |
| Discharge variables c + EMR at 6 months | 0.021 | 27% | 0.52 |
| Fraction of new information from EMR | - | - | 0.25 (p-value <0.01) |
LR χ2 = likelihood ratio chi-square value, MSPE=mean squared prediction error, MAPE=mean absolute percentage error. For C-index, Somers’D, LR χ2, and adjusted R2 higher values indicate better model performance. For MSPE and MAPE lower values indicate better model performance.
Logistic regression was used to predict place of residence, cumulative logistic regression was used to predict ordinal mRS, and linear regression was used to predict EQ-5D.
Place of residence base model includes age, sex, rNIHSS, and place of residence at discharge.
mRS base model includes age, diabetes, severe white matter disease, rNIHSS, mRS prior to stroke, and mRS at discharge.
EQ-5D base model includes age, sex, rNIHSS, depression, mRS prior to stroke, and mRS at discharge.
Discussion
The EMR encompasses many sources of medical information longitudinally captured during routine delivery of health care. This study shows that it is feasible to use information available in the EMR without any patient contact to determine important post-stroke outcomes. It also suggests that reviewing the EMR post-stroke adds important predictive information in determining outcomes beyond variables available during the stroke hospitalization. In addition, we were able to capture EMR outcome information on all stroke patients in this cohort, whereas we were not able to obtain telephone interview outcome data for 87 (23%) of the 318 patients, primarily due to patient/proxy refusal to participate or the inability to contact the stroke patient by telephone. Thus, in a stroke epidemiology study such as the GCNKSS, the EMR provides more complete outcome data while eliminating the time-consuming need for personal contact.
This study showed good to very good agreement between EMR and the gold-standard telephone interview. For mRS, agreement varied by the specific level of mRS, with less than ideal performance when the telephone interview mRS was 2. Because a mRS of 2 is defined by slight disabilities, these conditions may not be noted consistently in the medical record compared with more extreme disabilities (such as needing a walker) or no disability at all. In addition, patients with a telephone interview mRS of 2 were less likely to have an interview done by proxy compared to patients with a telephone mRS score not 2 (12% versus 38%, p<0.01 at 3 months and 14% versus 35%, p<0.01 at 6 months) and we showed that agreement was better when the telephone interview was completed by a proxy. Our findings are inconsistent with those of Quinn et al., who showed poor agreement (kappa=0.34) between mRS derived from patient case records and gold-standard video-based mRS assessment. [17] This difference in findings may be due to the Quinn study’s smaller sample size of 50 patients and the variable range of follow-up from the index stroke event (median of 16 weeks, range 2–56 weeks). In addition, our study had access to information captured from the index stroke hospitalization (including discharge mRS), and we allowed a proxy to provide the gold-standard data if the patient could not do so, whereas all data in the Quinn study were self-reported. Nevertheless, the Quinn study did report, consistent with our findings, better agreement at the extremes of mRS.
Because disabilities caused by a stroke event can impede stroke survivors from reporting on their own outcomes, outcome assessment by proxy, caregiver or family member, is commonly substituted. [18,19] Previous studies have shown that proxies tend to rate patient outcomes worse than patients rate their own outcomes [20,21] and in particular tend to over-report disabilities for patients 65 years and older. [22] However, the agreement measures stratified by interviewee (patient or proxy) in our study showed that agreement between EMR-determined outcome and the gold standard telephone interview was better when the telephone interview was completed by a proxy. We also noted that the EMR reviewer, reflecting the judgment of patients’ health care providers, tended to score disability higher and quality of life lower than what was reported by telephone interview. This is consistent with previous studies that have shown that health care providers tend to rate patients as having worse functioning and health than do the patients themselves. [23, 24]
This study is not without limitations. First, it was conducted in a single health care system. Future work is needed to determine if the findings from this study are generalizable to other care systems and other geographical regions, as the quality of the EMR-derived outcomes depends on the quality of the medical information recorded in the EMR. If our study results can be replicated in another study setting, another care system in another geographic region or country, this would provide evidence of external validity of EMR derived outcomes. Future studies will be able to utilize our described methods and analytic process to compare their findings with our results, and replication of these findings will provide support for broader generalizability. For example, a researcher might replicate reviewing the EMR records along with telephone interview derived outcomes and compare results in a similar way to those presented in this study. Researchers in other countries may select country specific utility weights, such as the EQ-5D weights for the United Kingdom, or other important outcomes and other disease settings to expand on our findings. Conceptual replication that produces similar results will greatly increase the confidence in the generalizability of EMR derived outcomes. In addition, all EMR outcome measures reported here were reviewed and scored by a single experienced lead study coordinator. Future work is needed to determine rater variability and agreement in collecting and defining the EMR outcomes. Our cohort had relatively mild strokes, consistent with what has been reported in our population in prior study periods. [25] Despite this, significant disability still occurred, reflected by mRS 3-6 in roughly 50% of the cohort; therefore, the typically considered initially “mild” stroke severity should not have impacted our evaluation on outcome assessment methods. This work shows promise in using EMR information to estimate post-stroke outcomes without patient contact. Reviewing the EMR post-stroke adds important predictive information in determining outcomes beyond variables available during the stroke hospitalization.
Supplementary Material
Acknowledgments
Funding Sources
This work was supported by National Institutes of Health, National Institute of Neurological Disorders and Stroke Division, grant R01NS30678.
Conflict of Interest Statement
Heidi Sucharew, Dawn Kleindorfer, Jane Khoury, Kathleen Alwell, Mary Haverbusch, Robert Stanton, Felipe De Los Rios La Rosa, Simona Ferioli, Adam Jasne, Eva Mistry, Charles J. Moomaw, Jason Mackey, Sabreena Slavin, Michael Star, Kyle Walsh, Daniel Woo, and Brett M. Kissela are supported by a research grant (NINDS R01NS30678). Dawn Kleindorfer is a consultant for Genentech. Felipe De Los Rios La Rosa is a member of Boehringer Ingelheim speaker’s bureau. Jason Mackey also reports grants from Indiana University (IU) Health/IU School of Medicine, IU Clinical and Translational Sciences Institute, and Patient Centered Outcomes Research Institute outside the submitted work, medicolegal work, and is an NIH Loan Repayment recipient. Stacie Demel reports grant support from the University of Cincinnati and personal fees from Genentech. Kyle Walsh is on the speaker’s bureau for Portola Pharmaceuticals. Eva Mistry reports funding from NINDS (K23NS113858).
Footnotes
Statement of Ethics
Institutional review boards of all involved research institutions and hospitals approved the study protocol (University of Cincinnati IRB 2013-3959).
At the time of the telephone call, the registered nurse informed the potential participant of the option not to participate. By agreeing to answer the nurse’s interview questions a verbal assent was obtained. Institutional review boards of all involved research institutions and hospitals approved the study protocol.
Data Availability
Qualified investigators may request access from the University of Cincinnati to obtain deidentified data (https://www.gcnkss.com).
References
- 1.Young J, Forster A. Review of stroke rehabilitation. BMJ. 2007; 334:86–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Katzan IL, Spertus J, Bettger JP, Bravata DM, Reeves MJ, Smith EE, et al. Risk adjustment of ischemic stroke outcomes for comparing hospital performance: a statement for healthcare professionals from the American Heart Association/American Stroke Association. Stroke. 2014; 45:918–944. [DOI] [PubMed] [Google Scholar]
- 3.Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012; 307:1583–1584. [DOI] [PubMed] [Google Scholar]
- 4.Weiskopf N, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 2013;20:144–151. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Cowie MR, Blomster JI, Curtis LH, Duclaux S, Ford I, Fritz F, et al. Electronic health records to facilitate clinical research. Clin Res Cardiol. 2017;106:1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Kasner SE, Chalela JA, Luciano JM, Cucchiara BL, Raps EC, McGarvey ML, Conroy MB, Localio AR. Reliability and validity of estimating the NIH Stroke Scale score from medical records. Stroke. 1999; 30: 1534–1537. [DOI] [PubMed] [Google Scholar]
- 7.Stavem K, Lossius M, Ronning OM. Reliability and validity of the Canadian Neurological Scale in retrospective assessment of initial stroke severity. Cerebrovasc Dis. 2003; 16: 286–291. [DOI] [PubMed] [Google Scholar]
- 8.Barber M, Fail M, Shield M, Stott DJ, Langhorne P. Validity and reliability of estimating the Scandinavian Stroke Scale score from medical records. Cerebrovasc Dis. 2004; 17: 224–227. [DOI] [PubMed] [Google Scholar]
- 9.Broderick J, Brott T, Kothari R, Miller R, Khoury J, Pancioli A, et al. The Greater Cincinnati/Northern Kentucky Stroke Study: preliminary first-ever and total incidence rates of stroke among blacks. Stroke. 1998;29:415–421. [DOI] [PubMed] [Google Scholar]
- 10.Kleindorfer D, Broderick J, Khoury J, Flaherty M, Woo D, Alwell K, et al. The unchanging incidence and case-fatality of stroke in the 1990s: A population-based study. Stroke. 2006;37:2473–2478. [DOI] [PubMed] [Google Scholar]
- 11.Calculating the U.S. Population-based EQ-5DTM Index Score [http://www.ahrq.gov/rice/EQ5Dscore.htm]
- 12.Altman DG. Practical statistics for medical research. London: Chapman and Hall; 1991. [Google Scholar]
- 13.Kim SK, Kim SH, Jo MW, Lee SI. Estimation of minimally important differences in the EQ-5D and SF-6D indices and their utility in stroke. Health Qual Life Outcomes. 2015; 13:32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Williams LS, Yilmaz EY, Lopez-Yunez AM. Retrospective assessment of initial stroke severity with the NIH Stroke Scale. Stroke. 2000; 31:858–862. [DOI] [PubMed] [Google Scholar]
- 15.Kissela B, Lindsell CJ, Kleindorfer D, Alwell K, Moomaw CJ, Woo D, et al. Clinical prediction of functional outcome after ischemic stroke: The surprising importance of periventricular white matter disease and race. Stroke. 2009; 40:530–536. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Dijkland SA, Voormolen DC, Venema E, Roozenbeek B, Polinder S, Haagsma JA, Nieboer D, Chalos V, Yoo AJ, Schreuders J, van der Lugt A, Majoie CBLM, Roos YBWEM, van Zwam WH, van Oostenbrugge RJ, Steyerberg EW, Dippel DWJ, Lingsma HF; MR CLEAN Investigators. Utility-Weighted Modified Rankin Scale as Primary Outcome in Stroke Trials: A Simulation Study. Stroke. 2018; 49:965–971. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Quinn TJ, Ray G, Atula S, Walters MR, Dawson J, Lees KR. Deriving modified Rankin scores from medical case-records. Stroke. 2008;39:3421–3423. [DOI] [PubMed] [Google Scholar]
- 18.Williams LS, Bakas T, Brizendine E, Plue L, Tu W, Hendrie H, et al. How valid are family proxy assessments of stroke patients’ health-related quality of life? Stroke. 2006; 37: 2081–2085. [DOI] [PubMed] [Google Scholar]
- 19.Pedersen PM, Jorgensen HS, Nakayama H, Raaschou HO, Olsen TS. Aphasia in acute stroke: Incidence, determinants, and recovery. Annals of Neurology. 1995; 38: 659–666. [DOI] [PubMed] [Google Scholar]
- 20.Duncan PW, Lai SM, Tyler D, Perera S, Reker DM, Studenski S. Evaluation of proxy responses to the Stroke Impact Scale. Stroke. 2002; 33: 2593–2599. [DOI] [PubMed] [Google Scholar]
- 21.Epstein AM, Hall JA, Tognetti J, Son LH, Conant L Jr. Using proxies to evaluate quality of life. Can they provide valid information about patients’ health status and satisfaction with medical care? Medical Care. 1989; 27(3 Suppl), S91–S98. [PubMed] [Google Scholar]
- 22.Todorov A, Kirchner C. Bias in proxies’ reports of disability: Data from the National Health Interview Survey on disability. American Journal of Public Health. 2000; 90(8), 1248–1253. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Sneeuw KC, Sprangers MA, Aaronson NK. The role of health care providers and significant others in evaluating the quality of life of patients with chronic disease. J Clin Epidemiol. 2002; 55(11):1130–1143. [DOI] [PubMed] [Google Scholar]
- 24.Sprangers MA, Aaronson NK. The role of health care providers and significant others in evaluating the quality of life of patients with chronic disease: a review. J Clin Epidemiol. 1992; 45:743–60. [DOI] [PubMed] [Google Scholar]
- 25.Reeves M, Khoury J, Alwell K, Moomaw C, Flaherty M, Woo D, Khatri P, Adeoye O, Ferioli S, Kissela B, Kleindorfer D. Distribution of National Institutes of Health stroke scale in the Cincinnati/Northern Kentucky Stroke Study. Stroke. 2013; 44:3211–3213. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Qualified investigators may request access from the University of Cincinnati to obtain deidentified data (https://www.gcnkss.com).