Predicting the six-month risk of severe hypoglycemia among adults with diabetes: development and external validation of a prediction model

Abstract

Aims

To develop and externally validate a prediction model for the six-month risk of a severe hypoglycemic event among individuals with pharmacologically treated diabetes.

Methods

The development cohort consisted of 31,674 Kaiser Permanente Colorado members with pharmacologically treated diabetes (2007–2015). The validation cohorts consisted of 38,764 Kaiser Permanente Northwest members and 12,035 HealthPartners members. Variables were chosen that would be available in electronic health records. We developed 16-variable and 6-variable models, using a Cox counting model process that allows for the inclusion of multiple six-month observation periods per person.

Results

Across the three cohorts, there were 850,992 six-month observation periods, and 10,448 periods with at least one severe hypoglycemic event. The 6-variable model contained age, diabetes type, HgbA1c, eGFR, history of a hypoglycemic event in the prior year, and insulin use. Both prediction models performed well, with good calibration and c-statistics of 0.84 and 0.81 for the 16-variable and 6-variable models, respectively. In the external validation cohorts, the c-statistics were 0.80–0.84.

Conclusions

We developed and validated two prediction models for predicting the six-month risk of hypoglycemia. The 16-variable model had slightly better performance than the 6-variable model, but in some practice settings, use of the simpler model may be preferred.

1. Introduction

Hypoglycemia is a potentially life-threatening complication of diabetes treatment, particularly in individuals treated with insulin or sulfonylurea drugs that stimulate insulin secretion (1–4). A clinically useful prediction tool for hypoglycemia would potentially enable providers and healthcare systems to identify patients at high risk and initiate anticipatory interventions to reduce the risk of severe hypoglycemia through goal modification, medication changes, or focused patient education.

A recent review by the VA QUERI program examined predictors of severe hypoglycemia in adults with type 2 diabetes (1). Important risk factors for severe hypoglycemia included intensive glycemic control, history of hypoglycemia, renal insufficiency, history of microvascular complications, longer diabetes duration, lower education level, African American race, and history of dementia. Gender, age, and lower body mass index (BMI) were not consistently associated with risk of hypoglycemia, although higher age and lower BMI were associated with increased risk of hypoglycemia in the two largest individual studies. However, this review did not describe a clinically useful and externally validated prediction rule. Using data from DCCT/EDIC on 1,441 individuals with type 1 diabetes, Lagani et al developed a prediction rule for hypoglycemia (2). They used a separate cohort of 393 individuals with type 2 diabetes as a validation data set. Their model included five variables, several of which are not easily collected from electronic medical records (marital status, strict vs standard insulin regimen, total insulin daily dose, family history of type 2 diabetes, and past history of severe hypoglycemia).

For individuals using frequent home blood glucose monitoring or continuous glucose monitors, either with or without an insulin pump, computer algorithms have been developed to predict the very short term risk of hypoglycemia (over the next minutes to hours) (5, 6). However, models based on clinical risk factors commonly available in electronic health records (EHRs) to predict the longer term risk of severe hypoglycemia (over days to months) are lacking.

The aims of this study were: 1) to develop a multivariable model to predict the six-month risk of severe hypoglycemia requiring medical intervention among individuals receiving pharmacologic treatment for diabetes, within one integrated health care delivery system (Kaiser Permanente Colorado) using information available in the electronic health record and other available clinical data sources; and 2) to externally validate the prediction model at two other sites (Kaiser Permanente Northwest and HealthPartners).

2. Subjects, materials, and methods

2.1 Study population

This study included three members of the SUPREME-DM (SUrveillance, PREvention, and ManagEment of Diabetes Mellitus) consortium, a group of 11 member organizations of the Health Care Services Research Network (HCSRN) (7, 8). The development cohort was based in Kaiser Permanente Colorado (KPCO) which serves the Denver-Boulder metropolitan areas. We used two validation cohorts, based in Kaiser Permanente Northwest (KPNW; serving the Portland, OR and Vancouver, WA metropolitan areas) and HealthPartners (HP; Minneapolis, Minnesota). Research institutes embedded in these health systems have developed a distributed virtual data warehouse that contains information on demographics, outpatient pharmacy dispensing, laboratory tests and laboratory results, and diagnosis and procedure codes from outpatient and inpatient health care encounters from their electronic health record and administrative data systems (8). The distributed virtual data warehouse allows for common variable definitions to be applied across study sites.

This study was approved by the Kaiser Permanente Colorado Institutional Review Board (IRB), and each participating site ceded oversight to the Kaiser Permanente Colorado IRB.

2.2 Cohort identification

We first identified a population of adults with diabetes using previously described methods (7, 9). Specifically, we defined the diabetes recognition date as the earlier of one inpatient diagnosis (ICD-9-CM 250.x, 357.2, 366.41, 362.01–362.07, either primary or secondary) or any combination of two of the following events, using the date of the first event in the pair as the identification date: 1) Hemoglobin A1c ≥ 6.5% (48 mmol/mol); 2) fasting plasma glucose ≥ 126mg/dl; 3) random plasma glucose ≥ 200mg/dl; 4) outpatient diagnosis code (same codes as for inpatient); 5) any single anti-hyperglycemic medication dispensing. When the two events were from the same source (e.g., two outpatient diagnoses or two elevated laboratory values), we required them to occur on separate days no more than two years apart. Dispensings of metformin or thiazolidinediones with no other indication of diabetes were not included because these agents may be used for diabetes prevention or to treat polycystic ovarian syndrome. Periods of pregnancy were excluded. Information on diabetes status was collected starting in 2000.

For each individual, the index date was the earliest date on or after 1/1/2007 when the following criteria were met: 1) diabetes case definition was satisfied, 2) enrolled continuously in the health plan for at least 6 months after the index date, 3) at least 12 months of continuous enrollment prior to the index date, 4) received any glucose lowering drug class medication on or within 100 days preceding the index date, and 5) at least 20 years of age on the index date. Individuals were censored at the earliest of: disenrollment for greater than 90 days, death, pregnancy, or 9/30/2015. For HealthPartners, which provides health insurance for patients receiving care outside HealthPartners Medical Group (HPMG) clinics, we additionally required that individuals have an available body mass index at the initial index date, indicating those who were receiving medical care within HPMG (and therefore would have complete EHR data).

2.3 Outcome assessment

We defined severe hypoglycemic events using a modified version of the algorithm initially developed by Ginde (10). Our definition is based on ICD-9 codes (primary and secondary) collected in emergency departments and inpatient encounters. Events were included that met at least one of the following criteria: 1) a code for 251.0, 251.1, 251.2, or 962.3, or 2) a code for 250.8x without a code for 259.8, 272.7, 681.xx, 682.xx, 686.9x, 707.1–707.9, 709.3, 730.0–730.2, or 731.8. We also excluded events that were the result of intentional overdoses (ICD-9 codes: E950.x-E958.x, E980.x-E988.x, or V62.84 accompanied by 960.xx-989.xx or 870.xx-897.xx), as the predictors for those events likely differ from the predictors for unintentional hypoglycemic events. Finally, we grouped all events that occurred within 7 days and considered them to be a single event.

2.4 Candidate predictor variables

We started with the list of potential predictors identified in the systematic review by Bloomfield et al (1). Because our population included individuals with both type 1 and type 2 diabetes, we also included diabetes type as a potential predictor. Using subject matter expertise, we narrowed the potential predictor variables to the 16 variables we felt were most likely to have good predictive abilities (Table 1) and would be readily available in most EHR systems. These included demographic variables (age, race/ethnicity), type of diabetes, body mass index, hemoglobin A1c, estimated glomerular filtration rate (eGFR), utilization (recent hospitalization, emergency department, and severe hypoglycemic event), important comorbidities (retinopathy, atherosclerotic cardiovascular disease, depression, heart failure), and medication use (insulin, metformin, and number of classes of glucose lowering medications). Diabetes type was defined using a modification of an algorithm developed by Klompas et al (11). For BMI, hemoglobin A1c, and serum creatinine, we used the most recent value in the 2 years preceding the initial index date. We estimated GFR using the CKD-EPI estimating equation, assuming non-black race if race was unknown (12). We assessed whether there had been any hospitalization or utilization of the emergency department in the previous 365 days. Comorbidities were defined using relevant ICD-9 codes, as defined in Table 1. Medication use was defined as prescription dispensing for that medication type in the previous 100 days. In addition to the 16-variable model, we considered a simplified model with 6 variables: age, diabetes type (type 1 or 2), hemoglobin A1c, eGFR, history of a hypoglycemic event in the prior 365 days, and insulin use. These variables were selected a priori using expert knowledge, and were considered to be the minimum set of variables that clinicians would accept in a prediction model.

Table 1.

Distribution of baseline characteristics of the development and validation cohorts at baseline.

	Development cohort, Kaiser Permanente Colorado	Validation cohort, Kaiser Permanente Northwest	Validation cohort, HealthPartners
	N (%)	N (%)	N(%)
Total N	31,674	38,765	12,035
Number of six-month observation periods	325,529	406,001	119,462
Number of six-month observation periods with severe hypoglycemic events	4,366	5,108	974
Age (years) mean (std)	60.3 (13.5)	59.3 (13.4)	59.0 (14.5)
Race/ethnicity
Non-Hispanic white	19,247 (60.8%)	30,097 (77.7%)	8,893 (73.9%)
Non-Hispanic black	2,028 (6.4%)	1,332 (3.4%)	1,337 (11.1%)
Hispanic	6,043 (19.1%)	2,303 (5.9%)	462 (3.8%)
Other	2,073 (6.5%)	3,348 (8.6%)	1,268 (10.5%)
Missing	2,283 (7.2%)	1,676 (4.3%)	75 (0.6%)
Type 1 diabetes	1,346 (4.3%)	1,612 (4.2%)	628 (5.2%)
Body mass index, kg/m2, mean (std); % missing	32.7 (7.4); 4.3%	34.6 (8.2); 2.7%	33.4 (7.7); 0%
A1c, %, mean (std); % missing	8.1 (2.0), 6.7%	8.0 (1.8), 5.4%	7.8 (1.9), 13.7%
A1c, mmol/mol, mean (std)	65 (21.9)	64 (19.7)	62 (20.8)
eGFR, mL/min/1.73m2, mean (std); % missing	75.3 (22.2); 4.3%	82.7 (23.7); 4.0%	80.3 (25.3); 10.8%
Hospitalization within the previous 365 days	4,240 (13.4%)	5,965 (15.4%)	2,374 (19.7%)
Emergency department visit within the previous 365 days	6,229 (19.7%)	10,860 (28.0)	2,340 (19.4%)
Severe hypoglycemia within the previous 365 days	444 (1.4%)	549 (1.4%)	166 (1.4%)
Retinopathy	1,587 (5.0%)	2,257 (5.8%)	894 (7.4%)
Cardiovascular disease	4,125 (13.0%)	4,319 (11.1%)	1,994 (16.6)
Depression	2,930 (9.3%)	5,361 (13.8%)	2,072 (17.2%)
Heart failure	1,311 (4.1%)	1,895 (4.9%)	706 (5.9%)
Insulin use	6,745 (21.3%)	8,505 (21.9%)	3,185 (26.5%)
Metformin use	22,666 (71.6%)	27,942 (72.1%)	7,824 (65.0%)
Number of classes of glucose lowering medication, mean (std)	1.4 (0.6)	1.4 (0.7)	1.5 (0.8)

Individuals were considered to have type 1 diabetes if the number of type 1 diabetes ICD-9 codes (250.x1, 250.x3) divided by the number of type 2 diabetes ICD-9 codes (250.x0, 250.x2) was greater than 0.5, and either: 1) there were no dispensings for diabetes medications other than insulin or metformin, or 2) there was a dispensing for glucagon (11). We defined retinopathy as at least two of the following ICD-9 codes in the previous 2 years: 362.0x or 250.5x. We defined cardiovascular disease as the presence of coronary artery disease (at least two codes in the previous 2 years: 410–414, 429.2, 429.7), myocardial infarction (one code from primary discharge diagnosis from hospitalization in the previous 2 years: 410.xx), CABG/stent/angioplasty (one code from primary discharge diagnosis from hospitalization in the previous 2 years: 36.0x, 36.1x, 36.2x; CPT codes: 33533–33536, 33510–33523, 92920–92944, 92980–92982, 92984, 92995–92996), or stroke (one code from primary discharge diagnosis from hospitalization in the previous 2 years: (430, 431, 433.x1, 434 [excluding 434.x0], 436). Medication use was defined as a prescription dispensing for that medication type in the previous 100 days. We considered 11 classes of glucose lowering medications (alpha-glucosidase inhibitors, amylin analogs, biguinides ([metformin], DPP-4 inhibitors, GLP-1 receptor agonists, short acting insulin, long acting insulin, meglitinides, SGLT2 inhibitors, sulfonylureas, and thiazolidinediones).

2.5 Observation periods

For both the 16- and 6-variable sets, we developed a model to predict the risk of a severe hypoglycemic event in the next six months, allowing individuals to have multiple six-month observation periods. For the first observation period, we used the index date as the baseline date (beginning of the observation period), and defined the predictor variables at the time of the baseline date. For the second observation period, we used the index date + 182 days as the baseline date, and again defined the predictor variables at the time of the new baseline date. This process was continued until the individual was censored. For all variables except for diabetes type, we redefined the variables using the most recent information available at the time of the new baseline date. For BMI, hemoglobin A1c, and serum eGFR, we carried forward the most recent non-missing value, in order to minimize missing data. Within a given observation period, we only counted the first hypoglycemic event. However, individuals were not censored if they had a hypoglycemic event, so that an individual could have multiple observation periods with hypoglycemic events.

2.6 Missing data

The following variables had missing data: race/ethnicity, BMI, hemoglobin A1c, and eGFR (Table 1). We conducted multiple imputation to impute each missing value five times (13). For continuous variables (BMI, hemoglobin A1c, and eGFR), we used a linear regression approach with available predictors as covariates in SAS procedure PROC MI. For race/ethnicity, we imputed missing values using site-specific distributions of race/ethnicity.

2.7 Statistical analysis

2.7.1 Model development

We developed the models using statistical methods described by Harrell and Steyerberg and endorsed by the Prognosis Research Strategy (PROGRESS) Group (14–18). As described above, we identified candidate predictors of hypoglycemia a priori. To avoid over-fitting the model, we required 20 hypoglycemic events per degree of freedom. For age and hemoglobin A1c we included linear and quadratic terms.

To allow for recurring severe hypoglycemic events and changes in covariate values over follow-up time, a Cox model for counting process was used for fitting the prediction model (19–21). In the counting process model, multiple observations for each individual were created. Each observation reflects a follow-up of six months, whether an individual experienced an event during the six months, and values of covariates at the beginning of the interval. Survival data in counting processes formulation can be modeled with existing statistical software (e.g., SAS PROC PHREG).

We validated the model internally using Harrell’s bootstrap resampling method to estimate slope shrinkage (degree of optimism, or overfitting) in the development cohort (15). We calculated the c-statistic and the D-index to assess discrimination or separation between high- and low-risk individuals (22). The D-index measures a model’s ability to separate the higher- and lower-risk patients. Each individual has a predicted risk based on the Cox model. The D-index is the hazard ratio comparing individuals above the median predicted risk to individuals below the median according to their observed risk of the event. A D-index of 1 means that the model failed to separate higher- and lower-risk patients (22). In assessing the agreement between observed and predicted risk, we randomly selected one six-month observation period per person, calculated the observed and predicted risk, repeated this process 1000 times, and then pooled the 1000 estimates to provide summary estimates.

2.7.2 External validation

We validated our prediction models externally using the KPNW and HP cohorts according to methods described by Royston and Altman (23). Specifically, we calculated the six-month predicted risk of severe hypoglycemia for each individual in the KPNW and HP cohorts using the Cox regression model’s exponentiation of the linear predictor based on coefficients derived from the KPCO cohort (24). We imputed missing values before calculating the predicted risks, as in the KPCO cohort. We then calculated c-statistic and D-index for the KPNW and HP cohorts.

We used SAS Version 9.4 (SAS Statistical Institute, Inc.) for all analyses.

3. Results

3.1 Model development

There were 31,674 individuals in the KPCO cohort (Appendix Figure 1), and 325,529 observation periods, with a mean of 10.3 observation periods per individual. There were 4,366 observation periods with severe hypoglycemia events, and total of 4,727 severe hypoglycemia events.

The baseline variables at the time of the first observation period are shown in Table 1. Just over 95% of the cohort had type 2 diabetes. The unadjusted hazard ratios and hazard ratios from the full 16-variable model are shown in Table 2. Age and hemoglobin A1c showed a U-shape relationship, with the lowest risk between ages 45 and 55 and for hemoglobin A1c between 6.5% and 7.5% (48 to 58 mmol/mol). There was a lower risk of hypoglycemia in the earlier observation periods, but there was good agreement between the observed and predicted risks throughout (Appendix Figure 2).

Table 2.

Hazard ratios (95% CIs) for severe hypoglycemic events, Kaiser Permanente Colorado.

	Unadjusted hazard ratios (95% CI)	Hazard ratios (95% CI) from the 16-variable model	Hazard ratios (95% CI) from the 6-variable model
Age (per year)	0.92 (0.91 0.94)	0.96 (0.95 0.98)	0.94 (0.93 0.96)
Age-squared	1.00 (1.00 1.00	1.00 (1.00 1.00)	1.00 (1.00 1.00)
Race/ethnicity
Non-Hispanic white	1.00	1.00	--
Non-Hispanic black	1.06 (0.95 1.19)	1.19 (1.05 1.34)	--
Hispanic	0.88 (0.81 0.95)	1.02 (0.94 1.11)	--
Other	0.81 (0.71 0.92)	0.95 (0.83 1.09)	--
Type 1 diabetes	2.89 (2.64 3.17)	1.76 (1.57 1.98)	1.98 (1.78 2.20)
Body mass index (per unit in kg/m2)	0.97 (0.97 0.97)	0.98 (0.98 0.99)	--
A1c (per %)	1.23 (1.10 1.37)	0.93 (0.84 1.03)	0.90 (0.81 0.99)
A1c-squared	0.99 (0.98 1.00)	1.01 (1.00 1.01)	1.01 (1.00 1.02)
eGFR (per unit in mL/min/1.73m2)	0.97 (0.97 0.97)	0.99 (0.99 0.99)	0.98 (0.98 0.98)
Hospitalization within the previous 365 days	3.35 (3.15 3.57)	1.53 (1.42 1.64)	--
Emergency department visit within the previous 365 days	2.93 (2.76 3.11)	1.55 (1.44 1.66)	--
Severe hypoglycemia within the previous 365 days	7.37 (6.78 8.01)	2.35 (2.14 2.59)	4.06 (3.72 4.42)
Retinopathy	2.95 (2.77 3.14)	1.28 (1.19 1.37)	--
Cardiovascular disease	2.50 (2.36 2.66)	1.23 (1.15 1.32)	--
Depression	1.81 (1.70 1.93)	1.36 (1.27 1.45)	--
Heart failure	3.25 (3.03 3.49)	1.26 (1.16 1.36)	--
Insulin	3.33 (3.13 3.54)	1.60 (1.45 1.76)	2.34 (2.19 2.51)
Metformin	0.37 (0.35 0.39)	0.64 (0.59 0.71)	--
Number of classes of glucose lowering medication	1.23 (1.19 1.27)	1.23 (1.17 1.31)	--

The slope-shrinkage statistic, which assesses the extent to which the model may be over-fit to the development data, was 0.996, indicating very little over-fitting (0.4%). Summary statistics from the internal validation of the Cox regression model indicated very good performance, with a c-statistic of 0.84 and a D-statistic of 1.42. There was good agreement between observed and predicted risk per quintile of predicted risk, with underestimation of the risk in the highest risk quintile (Figure, Table 3). The risk scores in Table 3 can be computed using the equation found in the Appendix Table. The observed rate of hypoglycemia was 50 times higher in the highest risk quintile vs the lowest risk quintile (Table 3).

Figure.

A. Observed and predicted rate of hypoglycemia, by quintile of predicted risk, 16-variable model, Kaiser Permanente Colorado.

B. Observed and predicted rate of hypoglycemia, by quintile of predicted risk, 6-variable model, Kaiser Permanente Colorado.

Table 3.

Severe hypoglycemia rate per 1000 person half-years by quintile of predicted risk and range of risk scores, by site.

16-variable model
	Development cohort, Kaiser Permanente Colorado		Validation cohort, Kaiser Permanente Northwest		Validation cohort 2, HeathPartners
Risk quint iles	Observed severe hypoglycemia rate per 1000 person half-years	Range of risk scores	Observed severe hypoglycemia rate per 1000 person half-years	Range of risk scores	Observed severe hypoglycemia rate per 1000 person half-years	Range of risk scores
1	1.0	0.036–0.206	1.3	0.020–0.066	0.3	0.024–0.069
2	2.2	0.206–0.318	2.5	0.066–0.096	0.8	0.069–0.102
3	5.6	0.318–0.527	5.2	0.096–0.157	2.4	0.102–0.168
4	12.3	0.527–1.052	11.5	0.157–0.318	6.6	0.168–0.336
5	52.6	1.052–39.574	42.6	0.318–11.070	31.0	0.336–10.169
6-variable model
1	1.4	0.024–0.106	1.6	0.007–0.034	0.7	0.010–0.033
2	3.1	0.106–0.158	3.3	0.034–0.049	1.4	0.033–0.048
3	6.0	0.158–0.246	5.6	0.049–0.074	2.7	0.048–0.075
4	12.0	0.246–0.442	11.1	0.074–0.134	6.3	0.075–0.140
5	50.9	0.442–14.356	40.1	0.134–3.405	29.1	0.140–2.911

Risk quintiles were based on exponentiation of the linear scores. Results are averaged across the 5 imputed datasets. We randomly selected one six-month observation period per person, calculated the observed and predicted risk, repeated this process 1,000 times, and then pooled the 1,000 estimates to provide the summary estimates.

The simplified model with 6 variables is showed in Table 2, and also performed very well, but with a lower c-statistic (0.81 vs 0.84) and D-statistic (1.29 vs 1.42). The slope shrinkage statistic was similar at 0.998. Agreement between observed and predicted risk per quintile of predicted risk was similar to that of the 16-variable model (Figure).

Table 4 illustrates cut points that could be used to identify different segments of the population. Depending on the model used, one-quarter to one-third of individuals had a less than 5% risk of severe hypoglycemia in the next six months, while nearly one-fifth had a greater than 20% risk of severe hypoglycemia in the next six months.

Table 4.

Range of prediction cut point values derived from the hypoglycemia prediction models, Kaiser Permanente Colorado.

	16-variable model		6-variable model
Predicted severe hypoglycemia rate per	Risk score cut point	Proportion of patients (%)	Risk score cut point	Proportion of patients (%)
1000 person half-years
≥ 100	5.690	1.2	3.253	0.4
≥ 50	2.828	4.8	1.503	4.2
≥ 20	1.099	19.2	0.597	19.1
≥ 10	0.545	39.3	0.297	42.8
≥> 5	0.272	67.8	0.148	75.9

Results are averaged across the 5 imputed datasets. We randomly selected one six-month observation period per person, calculated the observed and predicted risk, repeated this process 1,000 times, and then pooled the 1,000 estimates to provide summary estimates.

3.2 External validation

The baseline characteristics of the KPNW and HP cohorts are shown in Table 1. In the KPNW cohort, there were 38,765 individuals, 406,001 observation periods, and 5,108 observation periods with events. In the HP cohort, there were 12,035 individuals, 119,462 observation periods, and 974 observation periods with events. The 3 cohorts were largely similar, with the exception of different racial/ethnic compositions that reflect differences in the underlying communities. The rates of severe hypoglycemia were similar in KPCO and KPNW, and lower at HP.

For both the 16-variable model and the 6-variable models, the performance statistics were similar in the validation cohorts compared to the development cohort. For the 16-variable model, the c-statistics were 0.83 and 0.84 and the D-statistics were 1.36 and 1.59 for KPNW and HP, respectively. For the 6-variable model, the c-statistics were 0.80 for both validation cohorts and the D-statistics were 1.22 and 1.41 for KPNW and HP, respectively.

4. Discussion

We developed 16-variable and 6-variable models to predict the six-month risk of severe hypoglycemia among individuals receiving pharmacologic treatment for diabetes. Both models showed good calibration and discrimination, with c-statistics of 0.84 and 0.81. The 16-variable model performed somewhat better, but at the cost of greater complexity.

The individual predictors showed similar relationships to prior studies (1, 25–27). Age and hemoglobin A1c both had U-shaped relationships, with the lowest risks among those in the middle of the age and hemoglobin A1c range. A severe hypoglycemic episode in the previous 365 days was the strongest predictor. Individuals with a variety of comorbidities (retinopathy, cardiovascular disease, depression, and heart failure) were at higher risk of severe hypoglycemia, as were individuals with type 1 diabetes, recent hospitalizations/ED visits, worse renal function, lower BMI, use of a higher number or glucose lower medications, and insulin use. Metformin use was associated with a lower risk of hypoglycemia. Our models improve upon those of Lagani by using only variables that are widely available in electronic health records and other electronic data systems (2).

Rates of severe hypoglycemia are difficult to compare across studies, due to different inclusion criteria for cohorts (i.e. all individuals with diabetes, individuals with diabetes receiving pharmacologic treatment, individuals with diabetes on insulin) and different definitions for hypoglycemia. We did note an increasing risk of severe hypoglycemia in later observation periods. By definition, individuals in late observation periods had aged since earlier periods and likely had developed more comorbidities, both of which predicted hypoglycemic events. Thus, we believe that the analytic samples in the later observation periods were “sicker” than in earlier periods, a hypothesis that is supported by the similar performance of the prediction models in the earlier and later observation periods.

These models could be used both on an individual basis and for population management. In the individual setting, the equations in the Appendix Table and the cut points in Tables 3 and 4 can be used to calculate risks of individuals. Health care providers could thus use the 6-variable model to identify individuals that are at high risk of hypoglycemia, and then evaluate whether treatment interventions or patient education would be useful. Such interventions could include adjustment of glycemic control goals, reductions in the intensity of the treatment regimen (either through decreasing medications doses or discontinuing medications), more frequent home monitoring or use of a continuous glucose monitor, ensuring that the patient has glucagon available for reversal of hypoglycemia, and lifestyle modifications such as a more regular meal schedule. Periodic evaluation for de-escalation of diabetes treatment, particularly in the elderly, has been advocated (28, 29).

For population management, either the 16-variable or 6-variable prediction model could be used to identify a segment of the population with high risk of severe hypoglycemia. This approach could support various intervention strategies, including notification of individual health care providers, or clinical pharmacist outreach and counseling for patients at high risk of severe hypoglycemia. It would be easiest to implement these prediction models at the population level in an integrated health care system that contains information from both EHR and claims data. Five of the 6 variables in the 6-variable model are readily available in most outpatient EHR systems that include laboratory data. The sixth variable, history of severe hypoglycemia, requires either an EHR system that also contains inpatient information, or a system that is able to combine EHR and claims information.

Strengths of our study include the use of two external validation cohorts, access to a wide range of clinical predictors, and use of statistical techniques that make full use of the available data. The study also has limitations. We were unable to examine hypoglycemic events that did not result in medical attention. We were unable to determine whether the hypoglycemic events occurred prior to seeking medical attention or after an individual was already in the emergency department or the hospital. However, the majority of events were from the emergency department setting. The algorithm we used to identify hypoglycemic events is a validated, published hypoglycemia coding algorithm with an estimated positive predictive value of 89%, indicating some misclassification of events (10). We were unable to compare the predictive ability of our model to that of clinicians’ judgment, which could be explored in future work. Our findings may not be generalizable to other health care settings, such as fee-for-service practices or individuals who lack health insurance. We included both individuals with type 1 and type 2 in our cohort. While approximately 95% of our cohorts had type 2 diabetes, the hazard ratios for type 1 vs type 2 diabetes were 1.8 and 2.0, indicating that much of the variance in risk between type 1 and type 2 diabetes was captured by other variables (such as previous hypoglycemic events and insulin use).

5. Conclusions

In conclusion, we developed and validated 16-variable and 6-variable models to predict the six-month risk of severe hypoglycemia among a population of individuals with pharmacologically treatment diabetes. These models should be evaluated in other patient populations, but appear to be promising tools to help quantify hypoglycemia risk, and indicate the need for preventive measures.