Stroke risk calculators in the era of electronic health records linked to administrative databases

“Medicine is a science of uncertainty and an art of probability”

--Sir William Osler, 1904

Electronic health records (EHRs) linked to administrative databases can unlock the potential of clinical data by overcoming limitations of current cohort-derived risk calculators. EHRs can facilitate implementation of risk-based approaches, whereas current manual or web-based risk calculators pose prohibitive practical barriers to widespread use. We discuss how EHRs can be used 1) to derive increasingly precise risk equations calibrated to the populations in which they will be applied, 2) to facilitate communication of individual risk to patients, and 3) to provide population-based risk information for researchers, administrators, and policymakers.

Application and derivation of risk calculators: A Primer

Several risk calculators directly predict the outcome of stroke.^1–3 Stroke is also included as part of a composite outcome in other calculators⁴, an approach that may grow in popularity as the concept of stroke as a cardiovascular disease (CVD) risk equivalent becomes more widely accepted.

Guidelines state that the level of stroke and CVD risk should inform decision-making about initiating treatments such as aspirin or lipid-lowering agents.⁵ These strategies are based on the observation that the respective relative risk reduction of aspirin and statin therapies is similar for most sub-populations, and therefore the absolute benefit of treatment is proportional to the absolute risk of stroke or CHD. Because clinicians do not accurately estimate cardiovascular risk,^6–7 adhering to these guidelines requires the use of explicit risk calculators.

At least 110 stroke and CVD risk scoring methods exist.⁸ Early systems typically relied on a points-based system that required clinicians to manually calculate the sum of points associated with various risk factors. More recent calculators allow clinicians to input parameters directly into a website that uses a multivariate equation to predict risk. A recent study comparing two formats of the Framingham calculator suggests that the equation is more accurate than the points based approach.⁹

Derivation

Most risk calculators are derived from data in prospective cohort studies (see Table).

Table 1. Stroke and cardiovascular disease risk calculators.

	STROKE-SPECIFIC SCORES			CARDIOVASCULAR DISEASE SCORES
	Observational Cohort Studies		Observational Cohort Study		Randomized Controlled Trials		Administrative Databases
	FRAMINGHAM STROKE1	ARIC STROKE2	CHS STROKE3	FRAMINGHAM GENERAL CVD4	REYNOLD’S RISK SCORE10	REYNOLD’S RISK SCORE11	QRISK212
DERIVATION COHORT	Framingham Heart Study	ARIC	CHS	Framingham Heart Study	Women’s Health Study	Physicians’ Health Study	QRESEARCH
Age at baseline	55–84	45–65	65+	30–74	45+	50+	35–74
Sample size	5,734	14,685	5,711	8,491	16,400	10,824	1,535,583
Number of strokes	472	434	399	177
Number of events				1,174	766	1,294	140,115
Primary outcome	any stroke	ischemic stroke	any stroke	CHD, angina, TIA/stroke, PAD, CHF	CHD, coronary revascularization ischemic stroke	CHD, TIA/stroke	any stroke
TRADITIONAL RISK FACTORS
Age	X	X	X	X	X	X	X
Sex	X	X	X	X			X
Systolic BP	X	X	X	X	X	X	X
BP medication	X	X	X	X			X
Cholesterol			X	X	X	X	X
Cigarette smoking (Y/N)	X	X	X	X	X	X	X
Total # of cigarettes		X	X
Diabetes	X	X	X	X			X
Left ventricular	X	X	X
Hypertrophy
Family history of CHD	X	X	X
Atrial fibrillation					X	X	X
LABORATORY BIOMARKERS & CLINICAL MEASUREMENTS & DIAGNOSES
Hs-CRP					X	X
Creatinine			X		X
Chronic renal disease							X
Impaired fasting glucose			X
BMI		X					X
15 foot walk time			X
Rheumatoid arthritis							X
Peripheral vascular disease			X
DEMOGRAPHIC FACTORS	Framingham Heart Study	ARIC	CHS	Framingham Heart Study	Women’s Health Study	Physicians’ Health Study	QRESEARCH
Ethnicity		X	X		X		X
Neighborhood SES							X

The quintessential cohort study initiated in Framingham, Massachusetts in 1948 revolutionized the preventive approaches to CVD and pioneered the logistical and statistical methods of risk prediction. Most calculators apply to primary prevention populations without a history of cerebrovascular disease, although some calculators have also been developed to predict stroke in the setting of atrial fibrillation¹³, and following a recent transient ischemic attack (TIA)¹⁴ or stroke.¹⁵ Randomized controlled trials (RCTs) are also used to derive CVD calculators such as the Reynold’s Risk Score.^10–11

EHRs linked to administrative databases are increasingly used to derive and validate risk prediction models. For example, the Anticoagulation and Risk Factors in Atrial Fibrillation (ATRIA) study used 13,559 patients included in a clinical database of Kaiser Permanente of Northern California to predict the risk of warfarin-associated hemorrhage. ¹⁶

Large sample sizes facilitate study of uncommon risk factors

The primary limitation of relying on prospective cohort studies to derive risk calculators is the relatively small number of outcomes. The cohorts used to derive the Framingham and Atherosclerosis Risk in Communities (ARIC) stroke scores were based on 472 and 434 stroke events, respectively -- numbers insufficient to evaluate multiple risk factors and their interactions. For example, age modifies the relative risk of stroke associated with numerous risk factors, including blood pressure, smoking and atrial fibrillation^17–18 but few cohort-based CVD risk scores – and none of the stroke-specific risk scores -- include parameters for these age-related interactions. Atrial fibrillation is excluded from the ARIC stroke score (Table) because men with atrial fibrillation at baseline experienced only 2 stroke events; and no women with atrial fibrillation had a stroke. In another example, the prevalence of rheumatoid arthritis is approximately 1% in the general population and studies much larger than the Framingham cohort have established that rheumatoid arthritis is associated with a 30% increased risk of stroke. ¹⁹ However, we would expect fewer than 60 Framingham participants to have rheumatoid arthritis based on the original study size of 5,734, and the expected number of strokes in this subgroup is less than 15. Even if the risk calculator perfectly predicted risk in this group it would have negligible impact on overall metrics of discrimination, calibration, or reclassification.

When a country uses a national EHR platform linked to an administrative database then the entire population effectively participates in a registry. For example, all 73,538 patients with atrial fibrillation not treated with vitamin K antagonists in Denmark in the period 1997–2006 contributed to an analysis to conclusively show that CHA₂DS₂-VASc is more valid for stroke prediction in patients categorized as being at low and intermediate risk by CHADS₂.¹³ The QRISK2 score to predict a composite outcome of CVD and stroke was derived using 1.5 million patients in the QRESEARCH database of 551 clinical practices in the United Kingdom.¹² QRISK2 was internally validated using 0.75 million patients from the same database, and externally validated in 1,583,106 patients in the THIN database of 382 practices in the UK.²⁰ Compared to the cohort studies and RCTs used to derive Framingham and Reynold’s risk scores (Table) the number of patients and events contributing to the derivation and validation of QRISK2 is vast: nearly 3.9 million patients, or 14% of UK population aged 34–74 years, experienced 211,580 CVD events.²⁰ The large size facilitates inclusion of parameters for uncommon risk factors, minority ethnic populations, low socioeconomic (SES) populations, and interaction terms to account for important effect modification by age for eight risk factors.

Administrative datasets address threats to external validity

Systematic over- or under-estimation of risk occurs whenever the Framingham Risk Score (FRS) is applied in populations with different risk factor prevalence or different 10-year risk of stroke/CVD from the original Framingham cohort. The FRS discriminates reasonably well in other countries and ethnic groups but re-calibration is required.²¹ Recalibration, in turn, requires knowledge of absolute 10-year event rates, as well as the mean values for continuous model parameters (such as age, blood pressure, and cholesterol) measured in the population in which the score will be applied. Practicing clinicians rarely have access to event rates and risk factor values specific to the populations they serve,²² so analysis of local EHR or administrative databases would be needed for recalibration.

The large secular decline in CVD risk in the United States has lowered the absolute CVD risk from the era when cohort studies were initiated. In order to predict the shifting target of absolute risk in different populations and over time, risk equations ideally would be derived using contemporary data from the actual populations in which they will be applied. For example, QRISK2 in the UK is updated at least annually.²⁰ Just as there is an ongoing need to recalibrate equations, there is also an ongoing need to validate them using local EHR or administrative databases.

Addressing missing data in administrative databases

The primary strength of cohort studies and RCTs is their ability to minimize misclassification of exposures and outcomes by achieving high ascertainment of baseline risk factors and by adjudicating cardiovascular events and deaths. Conversely, the perceived Achilles heel of administrative databases is incomplete or inaccurate data. For example, in a study that assessed the discrimination of FRS using Veterans Administration (VA) databases, 16% of patients were missing data on blood pressure and 27% did not have a recorded cholesterol levels in an administrative database from 5 VA Medical Centers. In addition, the dataset underestimates outcome events because patients can be admitted with a stroke to non-VA hospitals. Despite this extent of missing data, we found discrimination of the FRS based on VA administrative data was comparable to other risk prediction tools.²³

In the future, the Meaningful Use (MU) program run by the Centers for Medicare and Medicaid Services (CMS) should minimize missing data in EHR-linked administrative databases. Some of the MU objectives require entry of information such as race, ethnicity, vital signs, and laboratory in discrete data fields instead of existing as free text in a progress note. Until then, there are several approaches in limiting validity threats due to missing data. One approach leverages existing data from the medical literature. For example, the proprietary Archimedes® model combines information from a clinical database of a large group-based health maintenance organization with data from cohort studies as well as randomized controlled trials.²⁴ Another approach is to impute missing data if the database is sufficiently large. The QRESEARCH database is missing cholesterol values for over two-thirds of participants. However, the number of participants with complete data far exceeded the number included in traditional cohort studies, and this permits the use of robust multiple imputation methods. Multiple imputation formally assumes that data are missing at random (MAR), and this assumption may not hold in administrative datasets that rely on clinically available information. Methods exist to address the possible bias introduced when data are not missing at random (NMAR). For example, variables can be included in the imputation model that do not appear in the risk equation, imputations can be weighted to reflect their plausibility under specified mechanisms of NMAR, and weighted imputations can be combined with inverse-probability methods. Additional research may be necessary to determine the ideal methods for handling missing data. Risk equations should be externally validated in independent datasets and their performance should be compared to existing tools. QRISK2 has been validated in a large independent dataset and appears to improve calibration and reclassification compared to the FRS in the United Kingdom.²⁰

In addition to the problem of missing data, some risk score components are unlikely to be stored as discrete data in administrative datasets. For example, the CHS stroke score includes 15-foot walk time and the lifetime number of cigarettes smoked,² and Reynold’s Risk Score includes family history of premature CHD (see Table). However, it is possible to improve ascertainment of important risk factors. For example, a dedicated assessment of family history using a patient questionnaire identified five times as many patients with a family history of premature CHD as a review of the clinic EHR in one study, and this active assessment resulted in reclassification of 5% of the clinic population into a higher risk category eligible for treatment.²⁵

Most administrative databases also do not include emerging risk factors that are not routinely measured in clinical practice such as coronary artery calcium. However, such components are no more likely to be available for practicing clinicians who do not use EHRs. Researchers interested in prospective evaluation of new risk factors may find it simpler and cheaper to perform novel assessments on patients already integrated into an existing EHR than to conduct a new dedicated cohort study.

Facilitating patient communication about risk

The Institute of Medicine Comparative Effective Research Prioritization project included among its list of 100 priority topics to “compare the effectiveness of adding information about new [risk factors]… with standard care in motivating behavior change and improving clinical outcomes.” EHRs can instantly display predicted risk derived from the original equation models without manual data entry, saving precious time in clinical encounters. EHRs can incorporate patient preferences to facilitate informed choice. For example, patients can view risk scenarios by selecting combinations of care options ²⁶ and choose from multiple state-of-the-art risk communication formats to complement simple absolute and relative risks; such as number-needed-to-treat, ratios and percentages that may improve comprehension.²⁷

Because EHR-derived risk scores can be quickly calculated at the point of care, there is flexibility in choosing what to present. Multiple risk scores from different calculators can be shown. Even when calculators produce divergent risk estimates, ²⁸ clinicians can still take into account this lack of consensus in their decision-making. EHRs can also accommodate alternative time horizons such as “lifetime risk”²⁹ which may be particularly relevant to younger patients.

EHRs do not definitively establish whether to treat patients whose predicted risk is near treatment thresholds. Guidelines currently recommend lipid-lowering therapy for a patient with a 10-year predicted risk of CVD of 10.1%, but would not recommend treatment for a similar patient with a predicted risk of 9.9%, even though the prediction intervals around these estimates would sufficiently overlap to render them indistinguishable (a prediction interval for an individual observation is related to, but wider than, a confidence interval for a population parameter). Currently available risk calculators do not provide information related to the uncertainty of their predictions. Large linked administrative databases can improve precision of risk prediction and reduce the number of people whose prediction intervals approximate clinical decision thresholds, but additional research is necessary to explore whether and how to integrate information on uncertainty of risk prediction into clinical guidelines and facilitate patient-centered decision-making.

EHRs may also facilitate use of risk calculators by the public. Just as clinicians do not accurately estimate cardiovascular risk,^6–7 patients’ perceived stroke risk does not correlated well with either their Framingham-calculated or observed stroke risk.³⁰ In particular, patients at higher risk for stroke (and most other diseases) are more likely to underestimate their risk, a phenomenon described as “unrealistic optimism.”³⁰ Patients state that they prefer seeing their individualized stroke risk, but evidence is lacking to demonstrate that this additional information improves risk factor control or medication adherence.^31–32 The AHA Go Red Heart CheckUp and Life’s Simple 7^™ websites allows individuals to enter specific risk factor values, view their CVD risk, and explore how modification of risk factors changes their risk. EHRs potentially can automate this process and permit patients to directly view their updated stroke risk. EHRs can link risk assessments to clinical decision aids and mobile patient reminders to improve comprehension and adherence. ³³

Despite the power and flexibility of EHRs, substantial challenges hinder the translation of improved predictive accuracy into clinical utility. Numerical literacy in the general population remains low. The science of clinical risk communication is in its infancy and the ideal format and clinical setting to display and discuss vascular risk remains poorly understood. The increased uptake of EHRs will not automatically improved risk communication, but it should facilitate the evaluation of alternative approaches and accelerate the uptake of proven methods.

Facilitating documentation of disparities and implementation of interventions to improve population health

Systematic under-treatment of entire sub-populations holds ethical implications for clinical practice. For example, the current FRS underestimates CVD risk among low SES groups and this systematic bias may lead clinicians to under-treat low SES groups (and to over-treat high SES groups), thus inadvertently exacerbating disparities. In the United States, 15% of persons of low SES status are classified into a higher risk category when education and income are added to the FRS,³⁴ and people whose predicted 10-year CVD risk is 6% or 13% according to FRS will experience a true risk of 10% and 20% if they live in a low-income neighborhood.³⁵ Efforts to incorporate patient-reported measures such as education into EHRs may soon improve the quality of SES data available in administrative databases. ³⁶ QRISK2 accounts for excess CVD risk among the poor by including a measure of neighborhood deprivation linked to each patient using their home address. EHRs facilitate the automated linkage of patients to similar indices of neighborhood deprivation developed in the United States.

By contributing clinical parameters to administrative databases, EHRs facilitate precise risk estimation for millions of patients at a time. Identifying the highest risk individuals permits targeting of intervention to those most likely to benefit. The Medicare Shared Savings Plan incentivizes Accountable Care Organization to reduce health care utilization such as hospitalizations by identifying and treating a list of high-risk patients.³⁷

If a risk tool is used to prioritize enrollment of patients in a care intervention to improve population health, an alternative to using an identical risk threshold for every individual patient is to base patient selection criteria on the resource constraints of a care intervention. For example, if resources exist to enroll 1000 patients in an intervention then an organization may elect to replace criteria based on an absolute risk threshold (eg: 10% risk of CVD in 10 years) with an alternative ranking criteria that selects individuals with the highest 1000 risk scores. EHRs by themselves do not address the normative and political questions required to define “high risk” thresholds or other treatment criteria but they provide a flexible platform to derive, evaluate and implement these myriad approaches.

Facilitating the use of risk calculators

In order to improve clinical decisions, risk calculators must first be used. Data are lacking, but utilization of risk prediction tools is likely to be low in most settings. The Joint Commission’s new Tobacco Cessation Performance Measure-Set ³⁸ provides an example of how to incentivize and track the implementation of risk calculators. The Tobacco Measure-Set requires hospitals to identify and document the smoking status of all patients, provide eligible patients with cessation counseling and medication, prescribe cessation medication at discharge and document tobacco-use status one month later. Similarly, clinicians could be incentivized to use risk calculators through quality measures. Such a measure could state that clinicians should provide patients with their own CVD and stroke risk and provide counseling on how to reduce the risk. An EHR could facilitate performance of such a measure by automatically calculating level of risk.

Current outcome measures in control of atherosclerotic risk factors are based on categorical thresholds of continuous risk factors. There is an ongoing vigorous debate about the relative merits of simpler so-called “treat-to-target” strategies that target a specific LDL or blood pressure level versus more accurate “risk-based”³⁹ or “individualized guideline” ⁴⁰ strategies informed by risk calculators. Future measures should strongly consider augmenting (or replacing) specific treatment goals with rewards proportional to achieved reductions in overall risk.

Conclusion

The accurate prediction of risk for atherosclerotic events such as stroke presents clinicians and patients with an opportunity to practice patient-centered, personalized medicine, and provides administrators and policymakers with a powerful tool to efficiently target care interventions to individuals and populations most likely to benefit. EHRs may facilitate a transition to a future in which patient- or clinician-selected risk scores are automatically calculated for clinicians to inform treatment decisions, and used by hospitals, accountable care organizations and insurers for risk adjustment and the prioritization of high-cost interventions. Health systems such as the Veterans Administration or others located in the “Stroke Belt” may choose to develop their own stroke-specific risk scores based on their own unique populations. Eventually EHR-linked databases that currently reside in clinical information silos could be linked so that every person contributes in real-time to the derivation and re-calibration of risk calculators. Once privacy concerns are addressed, an EHR-linked database that includes the US population of 311 million would contribute approximately 850,000 years of person-time and over 1,600 first stroke events every single day.