Key Points
Question
Can electronic medical records be used to reliably categorize risk of future hypoglycemia-related emergency department or hospital use in patients with type 2 diabetes?
Findings
We developed and validated a risk stratification tool that categorized patients’ 12-month risk of hypoglycemia-related utilization using only 6 electronic medical record–based inputs (patient history of hypoglycemia-related utilization, insulin use, sulfonylurea use, emergency department use, chronic kidney disease, and age). Tool performance was validated in 2 fully independent populations.
Meaning
This hypoglycemia risk stratification tool could facilitate efficient targeting of population management interventions to reduce hypoglycemia risk and improve patient safety.
This study develops and validates a risk stratification tool to categorize risk of future hypoglycemia-related emergency department or hospital use in patients with type 2 diabetes.
Abstract
Importance
Hypoglycemia-related emergency department (ED) or hospital use among patients with type 2 diabetes (T2D) is clinically significant and possibly preventable.
Objective
To develop and validate a tool to categorize risk of hypoglycemic-related utilization in patients with T2D.
Design, Setting, and Participants
Using recursive partitioning with a split-sample design, we created a classification tree based on potential predictors of hypoglycemia-related ED or hospital use. The resulting model was transcribed into a tool for practical application and tested in 1 internal and 2 fully independent, external samples. Development and internal testing was conducted in a split sample of 206 435 patients with T2D from Kaiser Permanente Northern California (KPNC), an integrated health care system. The tool was externally tested in 1 335 966 Veterans Health Administration and 14 972 Group Health Cooperative patients with T2D.
Exposures
Based on a literature review, we identified 156 candidate predictor variables (prebaseline exposures) using data collected from electronic medical records.
Main Outcomes and Measures
Hypoglycemia-related ED or hospital use during 12 months of follow-up.
Results
The derivation sample (n = 165 148) had a mean (SD) age of 63.9 (13.0) years and included 78 576 (47.6%) women. The crude annual rate of at least 1 hypoglycemia-related ED or hospital encounter in the KPNC derivation sample was 0.49%. The resulting hypoglycemia risk stratification tool required 6 patient-specific inputs: number of prior episodes of hypoglycemia-related utilization, insulin use, sulfonylurea use, prior year ED use, chronic kidney disease stage, and age. We categorized the predicted 12-month risk of any hypoglycemia-related utilization as high (>5%), intermediate (1%-5%), or low (<1%). In the internal validation sample, 2.0%, 10.7%, and 87.3% were categorized as high, intermediate, and low risk, respectively, with observed 12-month hypoglycemia-related utilization rates of 6.7%, 1.4%, and 0.2%, respectively. There was good discrimination in the internal validation KPNC sample (C statistic = 0.83) and both external validation samples (Veterans Health Administration: C statistic = 0.81; Group Health Cooperative: C statistic = 0.79).
Conclusions and Relevance
This hypoglycemia risk stratification tool categorizes the 12-month risk of hypoglycemia-related utilization in patients with T2D using only 6 inputs. This tool could facilitate targeted population management interventions, potentially reducing hypoglycemia risk and improving patient safety and quality of life.
Introduction
Advances in diabetes clinical care and medical treatment have reduced the risk of long-term complications and mortality for the more than 25 million Americans who have diabetes. However, iatrogenic hypoglycemia associated with glucose-lowering medication use has become a critical public health and drug safety concern. Severe hypoglycemia is defined as an event necessitating assistance from another person to actively administer carbohydrates, glucagon, or other resuscitative actions. Such assistance is often rendered professionally in emergency department (ED) or hospital encounters and is captured as hypoglycemia-related utilization.
Whereas the risk of severe hypoglycemia is known to be elevated in patients with type 1 diabetes, the risk has been historically underappreciated among patients with type 2 diabetes (T2D), which make up most of the diabetes population. Hypoglycemia is now one of the most frequent adverse events in patients with T2D and is more common than acute hyperglycemic emergencies (eg, hyperosmolar hyperglycemic state), particularly among older patients and those with a longer history of diabetes. One in 4 emergency hospitalizations for adverse drug events is related to hypoglycemia, and these rates are higher in older patients. Severe hypoglycemia has been associated with falls and automobile accidents, cardiovascular autonomic dysfunction and ventricular arrhythmia, dementia, and death. Patients report that fear of hypoglycemia can dissuade them from initiating newly prescribed insulin. Hypoglycemia is also strongly predictive of poorer health-related quality of life and more diabetes distress. Hypoglycemia-related utilization is costly; total annual direct medical costs were estimated at approximately $1.8 billion in 2009 in the United States.
The risk of hypoglycemia varies widely in patients with T2D. Whereas interventions to prevent hypoglycemia exist, there are no validated methods to target these interventions efficiently. Accordingly, we developed and validated a hypoglycemia risk stratification tool to categorize 12-month risk of hypoglycemia-related emergency department (ED) or hospital use among patients with T2D. This study was approved by the institutional review boards of Kaiser Permanente, the Bedford Veterans Health Administration, and Group Health Cooperative; the requirement that informed consent be obtained from study participants was waived.
Methods
Study Design
We used a prospective cohort study design to develop a risk tool to categorize the 12-month risk of hypoglycemia-related ED or hospital use. Selection of prebaseline candidate predictors was based on a literature review of clinical risk factors associated with hypoglycemia and limited to data typically available in electronic medical records (EMRs). We derived and internally validated this tool in a split sample (4:1) of 206 435 adult patients with T2D in an integrated health care delivery system (Kaiser Permanente Northern California [KPNC]) using clinical and demographic data from EMRs. We used recursive partitioning in the derivation sample to create a risk classification tree. The classification tree leaf nodes were further categorized into high-, intermediate-, or low-risk groups on the basis of predicted risk and then transcribed into the hypoglycemia risk stratification tool. After testing the tool in the internal sample, we conducted external validation in 2 completely independent samples of patients with T2D from the Veterans Administration Diabetes Epidemiology Cohort (DEpiC) (VA sample, n = 1 335 966) and from Group Health Cooperative (GH sample, n = 14 972).
Study Population
Using EMR data from KPNC, we identified 233 330 adults (≥21 years as of the baseline date of January 1, 2014) with diabetes with continuous health plan membership for 24 months prebaseline and pharmacy benefits for 12 months prebaseline. We excluded 24 719 patients with unknown diabetes type and 3615 with probable type 1 diabetes according to an algorithm (based on age of onset <30 years and use of insulin alone). The remaining 206 435 eligible patients with T2D were randomly split into an 80% derivation sample (n = 165 148) for tool development and a 20% internal validation sample (n = 41 287). Similar eligibility criteria were applied in the creation of the 2 external validation samples.
Outcome
Our outcome was the occurrence of any hypoglycemia-related ED or hospital use during 12 months postbaseline. This was defined by having any ED visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained according to a validated definition (any of the following International Classification of Diseases, Ninth Revision (ICD-9), codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes). Secondary discharge diagnoses for hypoglycemia were not used because they are often attributable to events that occurred during the ED or hospital encounter (eg, inpatient insulin management, sepsis, acute renal failure).
Exposures
On the basis of a literature review of clinical risk factors associated with hypoglycemia in T2D, we selected 156 (122 categorical and 34 continuous) candidate clinical, demographic, and behavioral predictor variables for model development (eTable 1 in the Supplement). To increase the utility of our prediction model in other health care settings (transportability, usability), we excluded variables that were expensive or impractical to collect, had ambiguous meanings, or were not typically available in an EMR. Medication exposures were based on pharmacy dispensings during 6 months prebaseline; laboratory values were based on the last test results within 2 years prebaseline; and prior medical events (eg, history of hypoglycemia-related utilization using the same outcome definition) were based on all available prebaseline records (maximum, 18 years of medical history; mean [SD], 16.5 [3.4] years).
Statistical Analysis
We used standard methodology for model development including a split sample and internal and external validation. We first regressed the outcome (any hypoglycemia-related utilization) on each of the 156 candidate predictors using univariate logistic regression models to generate odds ratios. We selected candidate variables that had a resulting P < .10. We then used recursive partitioning (using SAS JMP, version 12) on the selected candidates to construct a binary classification tree to predict the occurrence of at least 1 hypoglycemia-related utilization episode 12 months postbaseline. Recursive partitioning is widely used to generate clinical decision support tools. This method uses a machine-learning, nonlinear, and nonparametric approach to split (“partition”) events into pairs of subgroups based on continuous or categorical predictors, and has the unique advantage that it identifies complex nested interactions, unlike linear modeling methods. Recursive partitioning also optimizes cut points rather than relying on prespecification. Thus, the resultant classification tree identifies predictors that may be important for 1 segment of the population but not others, as well as identifying critical thresholds in continuous or ordinal predictors.
We pruned branches from the classification tree in an attempt to optimize predictive accuracy (performance), model simplicity, practicality of implementation, and intuitive clinical interpretation. Overly complex models (overfitting), while potentially offering somewhat greater precision, may be less practical, increase the decision and classification costs (expense and time of compiling the predictors), and introduce propagated error associated with predictors measured with uncertainty.
Validation Studies
Model accuracy was assessed in the internal validation sample using standard metrics.(pp255-310) Discrimination, the ability of a model to accurately distinguish between subjects who do vs do not develop the outcome, is based on the area under the receiver-operator curve (C statistic), with greater than 0.7 classified as good discrimination. We also visually assessed calibration (the extent to which the predicted risks over- or underestimate the observed risks) using calibration plots. Given that our goal was not to quantify the numeric probability of a hypoglycemia episode for a given patient but rather to stratify our population into categories of risk, we focused on model discrimination over calibration.
We further evaluated components of generalizability (ie, reproducibility in patients not used for the derivation of the model) and transportability (practical application in different settings) in the 2 external primary care T2D populations (VA and GH samples). Because these samples included a different disease severity and case mix of patients with T2D from distinct geographical locations with different methods for identifying patients with diabetes, the validation exercises also tested the spectrum, geographic, and methodological transportability of the model.
Sensitivity Analyses
We developed this risk stratification tool using all of the patients’ available medical histories at KPNC (up to 18 years); however, long enrollment may be uncommon in other health care settings. Thus, we conducted further analyses to evaluate whether our tool was sensitive to restrictions in available length of enrollment (prebaseline period transportability). We also evaluated temporal sensitivity (historical transportability) of the tool by applying it to KPNC data in the subsequent year (using a baseline date of January 1, 2015). This tool was optimized for patients with T2D, and because not all health care settings can reliably determine diabetes type from their EMR, we also evaluated tool performance when patients with type 1 diabetes were included (ie, sensitivity to misclassification). Finally, as a measure of ecological validity of this tool, we evaluated the association between the predicted level of risk of hypoglycemia-related utilization and actual, self-reported severe hypoglycemia events based on a 2005 survey of 15 231 patients with T2D.
Results
Model Selection
The final classification tree was based on 6 patient-specific variables: total number of prior episodes of hypoglycemia-related ED or hospital utilization (0, 1-2, ≥3 times), number of ED encounters for any reason in the prior 12 months (<2, ≥2 times), insulin use (yes/no), sulfonylurea use (yes/no), presence of severe or end-stage kidney disease (dialysis or chronic kidney disease stage 4 or 5 determined by estimated glomerular filtration rate of ≤29 mL/min/1.73 m2 calculated by the Chronic Kidney Disease Epidemiology Collaboration creatinine equation) (yes/no), and age younger than 77 years (yes/no) (Figure 1). This classification tree resulted in 10 mutually exclusive leaf nodes, each yielding an estimated annual risk of hypoglycemia-related utilization, which were categorized as high (>5%), intermediate (1%-5%), or low (<1%). In the KPNC internal validation sample, 2.0% were categorized as high risk, 10.7% as intermediate risk, and 87.3% as low risk.
Figure 1. Classification Tree for Hypoglycemia-Related Emergency Department (ED) or Hospital Use.
Hypoglycemic-related utilization was defined by having any ED visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained with any of the following International Classification of Diseases, Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes. The classification tree was developed using the 808 out of 165 148 T2D adults (derivation sample) from Kaiser Permanente who had such utilization (4.9 events per 1000 person-years) in 2014. The classification is based on 6 predictor variables from the electronic medical record and resulted in 10 mutually exclusive leaf nodes. The criterion for each node is displayed with the corresponding number of individuals (n) who met that criterion. The 12-month observed rate of any hypoglycemia-related ED or hospital use is displayed in each leaf node and categorized as high (>5% risk), intermediate (1%-5% risk), or low risk (<1% risk). CKD indicates chronic kidney disease.
We then transcribed the classification model into a simple, checklist style, hypoglycemia risk stratification tool by mapping the combinations of risk factors to high, intermediate, or low risk of having any hypoglycemia-related utilization in the following 12 months (Figure 2). This tool instructs the user to identify only 1 of 6 mutually exclusive options, where the first 5 are each defined by a unique combination of predictor variables, and the sixth option is indicated only after ruling out all other options (eTable 2 in the Supplement provides the source code).
Figure 2. Hypoglycemia Risk Stratification Tool.
aHypoglycemia cases were ascertained with any of the following International Classification of Diseases, Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes.
bHypoglycemic-related utilization was defined by having any emergency department (ED) visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia.
Patient Characteristics
We compared the distribution of the 6 predictor variables in our derivation and validation samples (Table 1). There were no significant differences in the distribution of the 6 predictors between the KPNC derivation vs validation samples, but there were significant differences across external validation samples. The proportion of men and women was similar in the KPNC and GH samples, while the VA sample was predominantly men. The mean age was similar across sites, although the proportion older than 77 years was greater in the VA (24.6%), followed by KPNC (17.8%), and GH samples (13.2%). The VA sample had the highest proportion of patients with severe or end-stage kidney disease (3.7%), and 3 or more prior hypoglycemic events (0.4%) vs 2.0% and 0.1%, respectively, in GH. Insulin use was lower in the KPNC samples (20.3%) compared with the GH (30.9%) and VA samples (30.2%). Sulfonylurea use was higher in the KPNC samples (34.9%) compared with the VA (25.0%) and GH (22.5%) samples. The observed annual rate of hypoglycemia-related utilization was lower in GH patients (0.30%) compared with the KPNC derivation sample (0.49%) and the VA (0.51%).
Table 1. Demographic Characteristics, Baseline Predictor Variables, and Outcome Rates in the Internal Derivation and Validation Samples (Kaiser Permanente Northern California [KPNC]) and the 2 External Validation Samples (Group Health Cooperative [GH] and Veterans Administration [VA]).
| Parameter | Derivation Sample | Validation Samples | Contrasts, P Value | ||||
|---|---|---|---|---|---|---|---|
| KPNC (n = 165 148) |
KPNC (n = 41 287) |
GH (n = 14 972) |
VA (n = 1 335 966) |
KPNC Derivation vs KPNC Validation Sample |
KPNC vs GH Validation Samples |
KPNC vs VA Validation Samples |
|
| Demographic Characteristics | |||||||
| Sex, No. (%) | |||||||
| Male | 86 572 (52.4) | 21 705 (52.6) | 7671 (51.2) | 1 288 576 (96.5) | .59 | .005 | <.001 |
| Female | 78 576 (47.6) | 19 582 (47.4) | 7301 (48.8) | 47 390 (3.5) | |||
| Age, mean (SD), y | 63.9 (13.0) | 64.0 (13.0) | 63.4 (12.2) | 68.8 (11.1) | .38 | <.001 | <.001 |
| Race/ethnicity, No. (%) | |||||||
| White | 73 796 (44.7) | 18 531 (44.9) | 10 276 (68.6) | 927 577 (69.4) | .94 | <.001 | <.001 |
| Black | 17 039 (10.3) | 4253 (10.3) | 986 (6.6) | 231 210 (17.3) | |||
| Hispanic | 26 924 (16.3) | 6676 (16.2) | 818 (5.5) | 70 429 (5.3) | |||
| Other | 45 066 (27.3) | 11 238 (27.2) | 2344 (15.7) | 42 170 (3.2) | |||
| Unknown | 2323 (1.4) | 589 (1.4) | 548 (3.7) | 64 580 (4.8) | |||
| Model Input Variables | |||||||
| Prior hypoglycemic-related utilization,a No. (%) | |||||||
| None | 159 704 (96.7) | 39 897 (96.6) | 14 672 (98.0) | 1 282 378 (96.0) | .72 | <.001 | <.001 |
| 1-2 times | 4948 (3.0) | 1268 (3.1) | 284 (1.9) | 48 194 (3.6) | |||
| ≥3 times | 496 (0.3) | 122 (0.3) | 16 (0.1) | 5394 (0.4) | |||
| Current diabetes medications, No. (%) | |||||||
| Sulfonylurea | 57 701 (34.9) | 14 405 (34.9) | 3362 (22.5) | 333 756 (25.0) | .85 | <.001 | <.001 |
| Insulin | 33 578 (20.3) | 8391 (20.3) | 4629 (30.9) | 403 538 (30.2) | .97 | <.001 | <.001 |
| Emergency department visits in prior year, No. (%) | |||||||
| 0-1 | 148 662 (90.0) | 37 104 (89.9) | 13 532 (90.4) | 1 165 838 (87.3) | .37 | .07 | <.001 |
| ≥2 | 16 486 (10.0) | 4183 (10.1) | 1440 (9.6) | 170 128 (12.7) | |||
| Chronic kidney disease stage,b No. (%) | |||||||
| 1-3 | 160 122 (97.0) | 40 021 (96.9) | 14 670 (98.0) | 1 286 719 (96.3) | .81 | <.001 | <.001 |
| 4-5 | 5026 (3.0) | 1266 (3.1) | 302 (2.0) | 49 247 (3.7) | |||
| Age ≥77 y, No. (%) | 29 410 (17.8) | 7344 (17.8) | 1980 (13.2) | 329 055 (24.6) | .92 | <.001 | <.001 |
| Outcome Rates | |||||||
| Hypoglycemia-related utilization during follow-up (2014),a No. (%) | 808 (0.49) | 192 (0.47) | 45 (0.3) | 6731 (0.51) | .53 | .008 | .27 |
Hypoglycemic-related utilization was defined by having any emergency department visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained with any of the following International Classification of Diseases, Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes.
Chronic kidney disease stage 1 to 3 determined by estimated glomerular filtration rate greater than 29 mL/min/1.73 m2 calculated by the Chronic Kidney Disease Epidemiology Collaboration creatinine equation; stage 4 or 5 determined by estimated glomerular filtration rate less than or equal to 29 mL/min/1.73 m2 or requirement of dialysis.
Model Validation
Internal validation of the classification tree model indicated high discrimination (C statistic = 0.83) and good calibration (no significant differences between predicted and observed risk: Pearson χ2 goodness-of-fit P = .31) (Table 2). The odds ratios of hypoglycemia-related utilization among those categorized as high relative to low risk were large in each sample: KPNC internal validation sample (34.6; 95% CI, 24.2-49.3), VA (23.3; 95% CI, 21.9-24.7), and GH (20.7; 95% CI, 8.6-45.0; P < .001).
Table 2. Calibration of Expected and Observed Rates of Hypoglycemia-Related Utilizationa Across the 3 Risk Strata in the Validation Samples.
| Risk Strata | No. (% of Sample) | No. (% of Strata) | P Valueb | |
|---|---|---|---|---|
| Observed | Expected | |||
| Kaiser Permanente (Internal) Validation Sample | ||||
| Low | 36 041 (87.3) | 75 (0.21) | 76 (0.21) | .75 |
| Intermediate | 4429 (10.7) | 62 (1.40) | 71 (1.60) | |
| High | 817 (2.0) | 55 (6.73) | 53 (6.49) | |
| Total | 41 287 | 192 (0.47) | 201 (0.48) | |
| Group Health (External) Validation Sample | ||||
| Low | 13 261 (88.6) | 23 (0.17) | 28 (0.25) | .02 |
| Intermediate | 1481 (9.9) | 14 (0.95) | 24 (1.69) | |
| High | 230 (1.5) | 8 (3.48) | 15 (5.65) | |
| Total | 14 972 | 45 (0.30) | 67 (0.47) | |
| Veterans Administration (External) Validation Sample | ||||
| Low | 1 096 945 (82.1) | 2691 (0.25) | 2558 (0.23) | <.001 |
| Intermediate | 204 913 (15.3) | 2192 (1.07) | 3527 (1.72) | |
| High | 34 108 (2.6) | 1848 (5.42) | 2255 (6.61) | |
| Total | 1 335 966 | 6731 (0.50) | 7857 (0.62) | |
Hypoglycemic-related utilization was defined by having any emergency department visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained with any of the following International Classification of Diseases, Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes; rates were calculated as the number of patients with any hypoglycemia-related utilization during the year per 1000 patients.
P value from Pearson χ2 goodness-of-fit test. For each leaf node in the classification tree, we multiplied the number of patients by the predicted rate of hypoglycemia events to generate the number of expected events in that node. We then summed the expected events from all nodes within each of the 3 risk levels to obtain expected event counts by risk level. The expected rates for each 3-level stratum will therefore vary between samples due to case-mix differences.
The tool also performed well in terms of discrimination in the external validation samples (VA C statistic = 0.81; GH C statistic = 0.79). Visual inspection of the calibration plots showed a reasonable match between the predicted and observed risk of hypoglycemia-related utilization within the 10 leaf nodes (Figure 3). However, the tool somewhat overestimated risk among the leaf nodes in the intermediate- and higher-risk categories in the external validation samples.
Figure 3. Calibration Plots Comparing the Expected vs Observed 12-Month Rate of Having Any Hypoglycemia-Related Utilizationa for the Interval Derivation Sample From Kaiser Permanente Northern California (KPNC) (n = 165 148), the KPNC Internal Validation Sample (n = 41 287), the External Validation Sample From Group Health (GH) (n = 14 972), and the External Validation Sample From the Veterans Administration (VA) (n = 1 335 966).
aHypoglycemic-related utilization was defined by having any emergency department visit with a primary diagnosis of hypoglycemia or a hospitalization with a principal diagnosis of hypoglycemia. Hypoglycemia cases were ascertained with any of the following International Classification of Diseases, Ninth Revision, codes: 251.0, 251.1, 251.2, 962.3, or 250.8, without concurrent 259.8, 272.7, 681.XX, 682.XX, 686.9X, 707.1-707.9, 709.3, 730.0-730.2, or 731.8 codes.
Sensitivity Analyses
In the sensitivity analyses for length of available medical history, good discrimination was confirmed despite shorter medical history (C statistic = 0.82, 0.83, 0.84 for ≤2, ≤5, and ≤10 years, respectively). We then evaluated temporal sensitivity (historical transportability). There was good discrimination (C statistic = 0.83) even after KPNC experienced a significant 21% increase in the rate of hypoglycemia-related utilization (0.48% vs 0.59% for 2014 and 2015, respectively). We also evaluated sensitivity to misclassification of diabetes type by including all diabetes patients in our sample and found good discrimination (C statistic = 0.84).
In the assessment of ecological validity, there was a strong association between predicted risk of hypoglycemia-related utilization and self-reported severe hypoglycemia (ie, hypoglycemia necessitating assistance in the past 12 months). Patients categorized as high risk by the tool were 5 times more likely (49.7% vs 9.2%; P < .001) to self-report a severe hypoglycemic episode relative to those categorized as low risk.
Discussion
Health care systems currently lack an evidence-based method for efficiently and systematically identifying patients with T2D at risk of hypoglycemia-related ED or hospital use. We developed and validated a pragmatic hypoglycemia risk stratification tool that uses 6 factors to categorize the 12-month risk of hypoglycemia-related utilization. This tool uses EMR data only and requires no patient contact; it offers an efficient, low-cost approach for identifying patients for targeted interventions to reduce their risk of hypoglycemia. Because of the harms and costs associated with hypoglycemia, high-risk patients are candidates for an elevated level of scrutiny. Identifying medication overtreatment and including hypoglycemia rates as a health plan–level quality measure have been recommended to drive accountability and quality improvement.
This tool is intended to offer a practical method to risk stratify patients for population management. For example, intensive interventions aimed at reducing hypoglycemia risk could be targeted at the minority of patients with T2D in the high-risk category (2% of patients with diabetes at KPNC). These interventions could include deintensifying or simplifying medication regimens, addressing impaired hypoglycemic awareness, prescribing glucagon kits or continuous glucose monitors, making referrals to clinical pharmacists or nurse care managers, providing additional diabetes education, and regularly asking about hypoglycemia events occurring outside the medical setting. Clinician discussions could address potential contributors to hypoglycemia, including behavioral (eg, meal skipping), psychosocial (eg, food insufficiency), or socioeconomic (eg, deprivation) factors. Similarly, a lower-cost, less intensive intervention could be designed for patients in the intermediate-risk category (11% at KPNC). The intervention could include system-level structural modifications such as risk-based glycemic targets, automated clinical alert flags in the EMR, and automated messaging to patients with elevated risk. Moreover, the tool could be modified to identify specific subsets of risk groups such those with 3 or more hypoglycemia-related ED or hospital encounters whom our model identifies as having the highest risk.
While it is unknown to what extent clinicians are aware of a patient’s hypoglycemia risk, there is evidence that clinicians and patients with diabetes do not communicate about hypoglycemia events that occur outside clinical settings. Almost all (roughly 95%) severe hypoglycemia events may go clinically unrecognized because they did not result in ED or hospital use. In an internal review of EMRs of KPNC patients with T2D, hypoglycemia was absent from the problem lists in 85% of patients categorized by our tool as being at high risk, underscoring the potential for this tool to increase clinicians’ awareness of the risk of hypoglycemia in their patients with T2D.
Limitations
Some limitations should be noted. The final classification tree was one of many possible options, chosen on the basis of performance (eg, C statistic), parsimony, and pragmatism (eg, we excluded predictors that are typically unavailable, impractical, or costly to assess in usual care settings). In developing the model, we excluded secondary discharge diagnoses for hypoglycemia because these events may occur during the ED or hospital encounter (eg, inpatient insulin management, sepsis, acute renal failure), rather than being a cause of the encounter. On the other hand, hypoglycemia could be a secondary diagnosis if the primary or principal diagnosis is trauma due to an automobile accident or a serious fall caused by hypoglycemia. Although we did not include those events in our model development, we estimate that this would include less than 2% of ED encounters (data not shown). When validating this model, we emphasized discrimination rather than calibration given that our goal was to risk stratify patients into broad categories rather than predicting the continuous level of risk. Thus, while the tool successfully stratifies the population into 3 levels of risk, it should not be used to estimate the probability of hypoglycemic-related utilization for an individual patient. Discrimination performance of the tool was validated internally and externally in 2 large, independent populations, suggesting generalizability. Geographic, methodologic, and spectrum transportability were also demonstrated because these external validation populations were from different geographical locations, used differing methods of caring for and identifying patients with T2D, and had different distributions of disease severity and case mix. While the observed rates of hypoglycemia-related utilization were lower in GH (0.3%) and somewhat higher in VA (0.51%) compared with KPNC sample (0.49%), the tool had good discrimination in all 3 samples. Some of the inconsistent findings are attributable to sparse data in the GH validation sample; only 45 events were observed during follow-up. It is also possible that the 2 external sample populations may experience more prevention efforts than at KPNC, explaining the somewhat lower than expected risk in higher-risk patients in the validation samples.
The tool was designed to predict hypoglycemia-related utilization and thus did not take into account severe hypoglycemia occurring outside the health care system (roughly 95% of hypoglycemic events necessitating third-party assistance are treated by persons other than medical professionals, eg, friends or family). However, we demonstrated a strong and significant association between the tool’s stratification of the risk of hypoglycemia-related utilization and a patient’s self-reported severe hypoglycemia events. Therefore, we believe that preventive interventions targeting patients identified as high risk by this tool may reduce the rate of severe hypoglycemic events that do not result in utilization or clinical recognition. Finally, the tool also proved robust in internal validations when patients with type 1 diabetes were included and when the availability of longitudinal EMR data was restricted.
This risk stratification tool was developed and validated in 3 vertically integrated health care delivery systems (KPNC, GH, and the VA). The tool logic may be programmed into a system’s EMR to allow for automated risk stratification based on the EMR data. Practical barriers may complicate implementing this tool for population management in horizontally integrated or nonintegrated health care delivery systems, for example, where pharmacy claims may not be readily available.
Conclusions
This tool offers a practical, EMR-based method to stratify patients with T2D by their 12-month risk of hypoglycemia-related ED or hospital utilization. This tool could be integrated with targeted preventive interventions to facilitate population management, which ultimately could reduce future hypoglycemia risk and improve patient safety. The 2 criteria indicating high risk are easily memorized (ie, ≥3 previous episodes of hypoglycemia-related utilization, or 1 or 2 episodes if treated with insulin). The criteria for intermediate risk are more nuanced and therefore may be less likely to provoke clinical action in primary care without prompting. Health care systems could adopt a 2-level intervention, with intensive (more expensive) interventions reserved for high-risk patients and less intensive (lower cost) interventions for the intermediate-risk patients. Implementation of this tool could conceivably increase clinician awareness of patients’ hypoglycemia risk. Clinical researchers may also find this tool helpful in identifying patients at high risk for hypoglycemic episodes for either purposeful inclusion or exclusion in clinical trials of novel therapies and diagnostic tests. Quality improvement and impact studies are needed to evaluate whether and how implementation of this hypoglycemia risk stratification tool may influence clinician behavior, patient decision making, drug safety, and hypoglycemia incidence. Future research is needed to develop patient-centered and cost-effective interventions to reduce hypoglycemia risk in those identified as being at high risk for hypoglycemia.
eTable 1. Candidate variables evaluated in our classification tree modeling
eTable 2. Source code for hypoglycemia risk stratification tool
References
- 1.Gregg EW, Li Y, Wang J, et al. Changes in diabetes-related complications in the United States, 1990-2010. N Engl J Med. 2014;370(16):1514-1523. [DOI] [PubMed] [Google Scholar]
- 2.Pogach L, Aron D. Balancing hypoglycemia and glycemic control: a public health approach for insulin safety. JAMA. 2010;303(20):2076-2077. [DOI] [PubMed] [Google Scholar]
- 3.Lee SJ. So much insulin, so much hypoglycemia. JAMA Intern Med. 2014;174(5):686-688. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Lipska KJ. Improving safety of diabetes mellitus management. JAMA Intern Med. 2014;174(10):1612-1613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Workgroup on Hypoglycemia, American Diabetes Association Defining and reporting hypoglycemia in diabetes: a report from the American Diabetes Association Workgroup on Hypoglycemia. Diabetes Care. 2005;28(5):1245-1249. [DOI] [PubMed] [Google Scholar]
- 6.Majumdar SR, Hemmelgarn BR, Lin M, McBrien K, Manns BJ, Tonelli M. Hypoglycemia associated with hospitalization and adverse events in older people: population-based cohort study. Diabetes Care. 2013;36(11):3585-3590. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Lipska KJ, Ross JS, Wang Y, et al. National trends in US hospital admissions for hyperglycemia and hypoglycemia among Medicare beneficiaries, 1999 to 2011. JAMA Intern Med. 2014;174(7):1116-1124. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Huang ES, Laiteerapong N, Liu JY, John PM, Moffet HH, Karter AJ. Rates of complications and mortality in older patients with diabetes mellitus: the diabetes and aging study. JAMA Intern Med. 2014;174(2):251-258. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Budnitz DS, Lovegrove MC, Shehab N, Richards CL. Emergency hospitalizations for adverse drug events in older Americans. N Engl J Med. 2011;365(21):2002-2012. [DOI] [PubMed] [Google Scholar]
- 10.Signorovitch JE, Macaulay D, Diener M, et al. Hypoglycaemia and accident risk in people with type 2 diabetes mellitus treated with non-insulin antidiabetes drugs. Diabetes Obes Metab. 2013;15(4):335-341. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Stahn A, Pistrosch F, Ganz X, et al. Relationship between hypoglycemic episodes and ventricular arrhythmias in patients with type 2 diabetes and cardiovascular diseases: silent hypoglycemias and silent arrhythmias. Diabetes Care. 2014;37(2):516-520. [DOI] [PubMed] [Google Scholar]
- 12.Whitmer RA, Karter AJ, Yaffe K, Quesenberry CP Jr, Selby JV. Hypoglycemic episodes and risk of dementia in older patients with type 2 diabetes mellitus. JAMA. 2009;301(15):1565-1572. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.McCoy RG, Van Houten HK, Ziegenfuss JY, Shah ND, Wermers RA, Smith SA. Increased mortality of patients with diabetes reporting severe hypoglycemia. Diabetes Care. 2012;35(9):1897-1901. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Zoungas S, Patel A, Chalmers J, et al. ; ADVANCE Collaborative Group . Severe hypoglycemia and risks of vascular events and death. N Engl J Med. 2010;363(15):1410-1418. [DOI] [PubMed] [Google Scholar]
- 15.Karter AJ, Subramanian U, Saha C, et al. Barriers to insulin initiation: the translating research into action for diabetes insulin starts project. Diabetes Care. 2010;33(4):733-735. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Laiteerapong N, Karter AJ, Liu JY, et al. Correlates of quality of life in older adults with diabetes: the diabetes & aging study. Diabetes Care. 2011;34(8):1749-1753. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Nicolucci A, Pintaudi B, Rossi MC, et al. The social burden of hypoglycemia in the elderly. Acta Diabetol. 2015;52(4):677-685. [DOI] [PubMed] [Google Scholar]
- 18.Peusens G, De Jonghe K, De Rood I, et al. Phytoplasmas in pome fruit trees: update of their presence and their vectors in Belgium. Commun Agric Appl Biol Sci. 2015;80(2):143-148. [PubMed] [Google Scholar]
- 19.Amiel SA, Dixon T, Mann R, Jameson K. Hypoglycaemia in type 2 diabetes. Diabet Med. 2008;25(3):245-254. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Meulstee M, Whittemore R, Watts SA. Development of an educational program on prevention of hypoglycemic events among elderly veterans with type 2 diabetes. Diabetes Educ. 2015;41(6):690-697. [DOI] [PubMed] [Google Scholar]
- 21.Rondags SM, de Wit M, Snoek FJ. HypoAware: development and pilot study of a brief and partly web-based psychoeducational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia. Diabet Med. 2016;33(2):184-191. [DOI] [PubMed] [Google Scholar]
- 22.Elliott J, Rankin D, Jacques RM, et al. ; NIHR DAFNE Research Study Group . A cluster randomized controlled non-inferiority trial of 5-day dose adjustment for normal eating (DAFNE) training delivered over 1 week versus 5-day DAFNE training delivered over 5 weeks: the DAFNE 5 × 1-day trial. Diabet Med. 2015;32(3):391-398. [DOI] [PubMed] [Google Scholar]
- 23.Seaquist ER, Anderson J, Childs B, et al. ; American Diabetes Association; Endocrine Society . Hypoglycemia and diabetes: a report of a workgroup of the American Diabetes Association and the Endocrine Society. J Clin Endocrinol Metab. 2013;98(5):1845-1859. [DOI] [PubMed] [Google Scholar]
- 24.Heinemann L, Devries JH. Evidence for continuous glucose monitoring: sufficient for reimbursement? Diabet Med. 2014;31(2):122-125. [DOI] [PubMed] [Google Scholar]
- 25.Miller DR, Pogach L. Longitudinal approaches to evaluate health care quality and outcomes: the Veterans Health Administration diabetes epidemiology cohorts. J Diabetes Sci Technol. 2008;2(1):24-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Huang ES, Liu JY, Moffet HH, John PM, Karter AJ. Glycemic control, complications, and death in older diabetic patients: the diabetes and aging study. Diabetes Care. 2011;34(6):1329-1336. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Ginde AA, Blanc PG, Lieberman RM, Camargo CA Jr. Validation of ICD-9-CM coding algorithm for improved identification of hypoglycemia visits. BMC Endocr Disord. 2008;8:4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Miller SI, Wallace RJ Jr, Musher DM, Septimus EJ, Kohl S, Baughn RE. Hypoglycemia as a manifestation of sepsis. Am J Med. 1980;68(5):649-654. [DOI] [PubMed] [Google Scholar]
- 29.Moons KG, Altman DG, Vergouwe Y, Royston P. Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ. 2009;338:b606. [DOI] [PubMed] [Google Scholar]
- 30.Royston P, Moons KG, Altman DG, Vergouwe Y. Prognosis and prognostic research: developing a prognostic model. BMJ. 2009;338:b604. [DOI] [PubMed] [Google Scholar]
- 31.Altman DG, Vergouwe Y, Royston P, Moons KG. Prognosis and prognostic research: validating a prognostic model. BMJ. 2009;338:b605. [DOI] [PubMed] [Google Scholar]
- 32.Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. New York, NY: Springer; 2009. [Google Scholar]
- 33.Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information. Ann Intern Med. 1999;130(6):515-524. [DOI] [PubMed] [Google Scholar]
- 34.Breiman L, Friedman JH, Olshen RA, Stone CJ. Classification and Regression Trees. Monterey, CA: Wadsworth & Brooks/Cole Advanced Books & Software; 1984. [Google Scholar]
- 35.Fonarow GC, Adams KF Jr, Abraham WT, Yancy CW, Boscardin WJ; ADHERE Scientific Advisory Committee, Study Group, and Investigators . Risk stratification for in-hospital mortality in acutely decompensated heart failure: classification and regression tree analysis. JAMA. 2005;293(5):572-580. [DOI] [PubMed] [Google Scholar]
- 36.Lewis RJ. An Introduction to Classification and Regression Tree (CART) Analysis. 2000. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.4103&rep=rep1&type=pdf. Accessed July 6, 2017.
- 37.Hosmer DW, Lemeshow S. Applied Logistic Regression. New York, NY: John Wiley & Sons; 1989. [Google Scholar]
- 38.Chambless LE, Cummiskey CP, Cui G. Several methods to assess improvement in risk prediction models: extension to survival analysis. Stat Med. 2011;30(1):22-38. [DOI] [PubMed] [Google Scholar]
- 39.Moons KG, de Groot JA, Bouwmeester W, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014;11(10):e1001744. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Meurer WJ, Tolles J. Logistic regression diagnostics: understanding how well a model predicts outcomes. JAMA. 2017;317(10):1068-1069. [DOI] [PubMed] [Google Scholar]
- 41.Walter LC, Brand RJ, Counsell SR, et al. Development and validation of a prognostic index for 1-year mortality in older adults after hospitalization. JAMA. 2001;285(23):2987-2994. [DOI] [PubMed] [Google Scholar]
- 42.Moffet HH, Adler N, Schillinger D, et al. Cohort profile: the Diabetes Study of Northern California (DISTANCE)—objectives and design of a survey follow-up study of social health disparities in a managed care population. Int J Epidemiol. 2009;38(1):38-47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Levey AS, Bosch JP, Lewis JB, Greene T, Rogers N, Roth D; Modification of Diet in Renal Disease Study Group . A more accurate method to estimate glomerular filtration rate from serum creatinine: a new prediction equation. Ann Intern Med. 1999;130(6):461-470. [DOI] [PubMed] [Google Scholar]
- 44.Senior PA, Bellin MD, Alejandro R, et al. ; Clinical Islet Transplantation Consortium . Consistency of quantitative scores of hypoglycemia severity and glycemic lability and comparison with continuous glucose monitoring system measures in long-standing type 1 diabetes. Diabetes Technol Ther. 2015;17(4):235-242. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Olsen SE, Bjørgaas MR, Åsvold BO, et al. Impaired awareness of hypoglycemia in adults with type 1 diabetes is not associated with autonomic dysfunction or peripheral neuropathy. Diabetes Care. 2016;39(3):426-433. [DOI] [PubMed] [Google Scholar]
- 46.Rondags SM, de Wit M, van Tulder MW, Diamant M, Snoek FJ. HypoAware—a brief and partly web-based psycho-educational group intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycaemia: design of a cost-effectiveness randomised controlled trial. BMC Endocr Disord. 2015;15:43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Shepard JA, Vajda K, Nyer M, Clarke W, Gonder-Frederick L. Understanding the construct of fear of hypoglycemia in pediatric type 1 diabetes. J Pediatr Psychol. 2014;39(10):1115-1125. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Rondags SM, de Wit M, Twisk JW, Snoek FJ. Effectiveness of HypoAware, a brief partly web-based psychoeducational intervention for adults with type 1 and insulin-treated type 2 diabetes and problematic hypoglycemia: a cluster randomized controlled trial. Diabetes Care. 2016;39(12):2190-2196. [DOI] [PubMed] [Google Scholar]
- 49.Pogach L, Aron D. The other side of quality improvement in diabetes for seniors: a proposal for an overtreatment glycemic measure. Arch Intern Med. 2012;172(19):1510-1512. [DOI] [PubMed] [Google Scholar]
- 50.Tseng CL, Soroka O, Maney M, Aron DC, Pogach LM. Assessing potential glycemic overtreatment in persons at hypoglycemic risk. JAMA Intern Med. 2014;174(2):259-268. [DOI] [PubMed] [Google Scholar]
- 51.McCoy RG, Lipska KJ, Yao X, Ross JS, Montori VM, Shah ND. Intensive treatment and severe hypoglycemia among adults with type 2 diabetes. JAMA Intern Med. 2016;176(7):969-978. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52.Lipska KJ, Kosiborod M. Hypoglycemia and adverse outcomes: marker or mediator? Rev Cardiovasc Med. 2011;12(3):132-135. [PubMed] [Google Scholar]
- 53.Rodriguez-Gutierrez R, Lipska KJ, McCoy RG, Ospina NS, Ting HH, Montori VM; Hypoglycemia as a Quality Measure in Diabetes Study Group . Hypoglycemia as an indicator of good diabetes care. BMJ. 2016;352:i1084. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Rodriguez-Gutierrez R, Ospina NS, McCoy RG, Lipska KJ, Shah ND, Montori VM; Hypoglycemia as a Quality Measure in Diabetes Study Group . Inclusion of hypoglycemia in clinical practice guidelines and performance measures in the care of patients with diabetes. JAMA Intern Med. 2016;176(11):1714-1716. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Bonds DE, Miller ME, Dudl J, et al. Severe hypoglycemia symptoms, antecedent behaviors, immediate consequences and association with glycemia medication usage: secondary analysis of the ACCORD clinical trial data. BMC Endocr Disord. 2012;12:5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Seligman HK, Bolger AF, Guzman D, López A, Bibbins-Domingo K. Exhaustion of food budgets at month’s end and hospital admissions for hypoglycemia. Health Aff (Millwood). 2014;33(1):116-123. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57.Basu S, Berkowitz SA, Seligman H. The monthly cycle of hypoglycemia: an observational claims-based study of emergency room visits, hospital admissions, and costs in a commercially insured population. Med Care. 2017;55(7):639-645. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58.Berkowitz SA, Karter AJ, Lyles CR, et al. Low socioeconomic status is associated with increased risk for hypoglycemia in diabetes patients: the Diabetes Study of Northern California (DISTANCE). J Health Care Poor Underserved. 2014;25(2):478-490. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59.Frier BM, Jensen MM, Chubb BD. Hypoglycaemia in adults with insulin-treated diabetes in the UK: self-reported frequency and effects. Diabet Med. 2016;33(8):1125-1132. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60.Östenson CG, Geelhoed-Duijvestijn P, Lahtela J, Weitgasser R, Markert Jensen M, Pedersen-Bjergaard U. Self-reported non-severe hypoglycaemic events in Europe. Diabet Med. 2014;31(1):92-101. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 61.Sarkar U, Karter AJ, Liu JY, Moffet HH, Adler NE, Schillinger D. Hypoglycemia is more common among type 2 diabetes patients with limited health literacy: the Diabetes Study of Northern California (DISTANCE). J Gen Intern Med. 2010;25(9):962-968. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 62.Lipska KJ, Warton EM, Huang ES, et al. HbA1c and risk of severe hypoglycemia in type 2 diabetes: the Diabetes and Aging Study. Diabetes Care. 2013;36(11):3535-3542. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eTable 1. Candidate variables evaluated in our classification tree modeling
eTable 2. Source code for hypoglycemia risk stratification tool
