Accuracy of a Risk Alert Threshold for ICU Hypoglycemia: Retrospective Analysis of Alert Performance and Association With Clinical Deterioration Events

OBJECTIVES:

To quantify the accuracy of and clinical events associated with a risk alert threshold for impending hypoglycemia during ICU admissions.

DESIGN:

Retrospective electronic health record review of clinical events occurring greater than or equal to 1 and less than or equal to 12 hours after the hypoglycemia risk alert threshold was met.

SETTING:

Adult ICU admissions from June 2020 through April 2021 at the University of Virginia Medical Center.

PATIENTS:

Three hundred forty-two critically ill adults that were 63.5% male with median age 60.8 years, median weight 79.1 kg, and median body mass index of 27.5 kg/m².

INTERVENTIONS:

Real-world testing of our validated predictive model as a clinical decision support tool for ICU hypoglycemia.

MEASUREMENTS AND MAIN RESULTS:

We retrospectively reviewed 350 hypothetical alerts that met inclusion criteria for analysis. The alerts correctly predicted 48 cases of level 1 hypoglycemia that occurred greater than or equal to 1 and less than or equal to 12 hours after the alert threshold was met (positive predictive value = 13.7%). Twenty-one of these 48 cases (43.8%) involved level 2 hypoglycemia. Notably, three myocardial infarctions, one medical emergency team call, 19 deaths, and 20 arrhythmias occurred greater than or equal to 1 and less than or equal to 12 hours after an alert threshold was met.

CONCLUSIONS:

Alerts generated by a validated ICU hypoglycemia prediction model had a positive predictive value of 13.7% for real-world hypoglycemia events. This proof-of-concept result suggests that the predictive model offers clinical value, but further prospective testing is needed to confirm this.

KEY POINTS.

• Question: What are the accuracy of and clinical events associated with a risk alert threshold for ICU hypoglycemia?

• Findings: We retrospectively reviewed 350 hypothetical alerts that correctly predicted 48 cases of level 1 hypoglycemia occurring greater than or equal to 1 and less than or equal to 12 hours after the alert threshold was met (positive predictive value= 13.7%). Notably, three myocardial infarctions, one medical emergency team call, 19 deaths, and 20 arrhythmias occurred greater than or equal to 1 and less than or equal to 12 hours after an alert threshold was met.

• Meaning: Alerts generated by a validated ICU hypoglycemia prediction model had positive predictive value of 13.7% for real-world hypoglycemia events.

Randomized controlled trials demonstrate that ICU hypoglycemia is strongly associated with increased morbidity and mortality (1). The well-established biochemical, hemodynamic, and electrophysiological changes that occur during hypoglycemia (2) make it an ideal target for predictive analytics monitoring; however, few studies have focused on model development specifically for ICU hypoglycemia (3–5). We recently described a pathophysiologic signature of impending ICU hypoglycemia that incorporated hemodynamic and electrophysiological bedside monitoring data in a logistic regression model (6). A necessary step in translating this model to clinical practice is understanding how it would perform when operationalized as a real-time alert. Toward this goal, the aim of the current study was to retrospectively quantify the accuracy of and identify the nature of clinical events associated with large, abrupt increases (i.e., spikes) in hypoglycemia risk during ICU admissions at the University of Virginia (UVA) Medical Center.

MATERIALS AND METHODS

We performed a retrospective analysis of adult ICU admissions where Prediction Assistant, CoMET inside (Premier, Charlotte, NC) was in place from June 2020 through April 2021 at the UVA Medical Center. Prediction Assistant collected laboratory results and flowsheet vital signs from the electronic health record (EHR) along with continuous cardiorespiratory monitoring data from the UVA Kafka System in real time (7). Prediction Assistant used these data to estimate the relative risk of impending ICU hypoglycemia based on our validated multivariable logistic regression model containing 41 independent predictors (6). For this, the model was employed in the current cohort to estimate the probability of hypoglycemia in the next 12 hours; then, that probability was divided by 0.00436 (i.e., the average probability of hypoglycemia in the next 12 hr). This study (“Chart Review for Predictive Modeling in the Hospital”; Institutional Review Board [IRB] #22152) was reviewed by the UVA IRB for Health Sciences Research, and the need for IRB approval and informed consent was waived on August 7, 2020. All procedures were followed in accordance with the Helsinki Declaration of 1975.

We analyzed relative risk estimates every 15 minutes. We focused on large spikes in risk defined as an increase of greater than or equal to 10 units compared with the average 2–3 hours prior (Fig. 1) in order to reduce false positives in this early testing phase. For example, the threshold was met if risk increased from 2 to 12 units or from 0.3 to 10.3 units. Based on our initial model development and validation, we expected these spikes to occur about two times per day (6). We examined only the first alert threshold met during an ICU admission (i.e., if multiple thresholds were met in one admission, only the first was examined, and the remainder were excluded).

Figure 1.

Time series demonstrating fingerstick glucose (A) and hypoglycemia relative risk (B) values around the time an ICU hypoglycemia alert threshold was met. Note that the alert threshold (solid vertical line in A and B) was met prior to onset of level 1 hypoglycemia (i.e., <70 mg/dL; represented by the dashed horizontal line in A).

We also examined the EHR for clinical events associated with these risk spikes. For this, we assessed the period of time greater than or equal to 1 but less than or equal to 12 hours after the spike. We considered this enough time to allow clinicians to see patients and intervene (i.e., no less than 1 hr) but not so long as to lose association of the spike with the event (i.e., more than 12 hr). Hypoglycemia categories were consistent with those recommended by the American Diabetes Association’s Standards of Medical Care in Diabetes (8): level 1 hypoglycemia was a blood or fingerstick glucose value less than 70 mg/dL (3.9 mmol/L) and greater than or equal to 54 mg/dL (3.0 mmol/L), and level 2 hypoglycemia was a blood or fingerstick glucose value less than 54 mg/dL (3.0 mmol/L). We recorded the time of first level 1 and/or level 2 hypoglycemia as well as whether intravenous dextrose, oral glucose tablets, and/or liquid sugar (e.g., orange juice) were administered as treatment and whether the subcutaneous insulin dose was adjusted. We also noted other clinical deterioration events: medical emergency team call/visit, arrhythmia on electrocardiogram, myocardial infarction, and/or death. Statistical analyses were performed in GraphPad Prism (GraphPad Software, San Diego, CA).

RESULTS

Identification and Treatment of ICU Hypoglycemia

Table 1 provides demographic and clinical data for the study cohort (n = 342 patients). Notably, the cohort was 63% male, had median age of ~61 years, and was distributed among five different ICUs. The median increase in hypoglycemia relative risk prior to an alert was 10.57.

TABLE 1.

Demographic and Clinical Characteristics of the Study Cohort (n = 342)

Variable	Value
Sex, n (%)
Male	217 (63.45)
Female	125 (36.55)
Race, n (%)
White	223 (65.2)
African-American	90 (26.3)
Other	19 (5.6)
Unspecified	5 (1.5)
Asian	3 (0.9)
Multi-Race	2 (0.6)
ICU, n (%)
Coronary care unit	85 (24.29)
Medical intensive care unit	64 (18.29)
Cardiovascular intensive care unit	66 (18.86)
Surgical-trauma intensive care unit	87 (24.86)
Thoracic cardiovascular post-operative unit	48 (13.71)
Age, median years (IQR)	60.82 (49.94–71.88)
Weight, median kilograms (IQR)	79.1 (66.3–96.8)
Body mass index, median kg/m2 (IQR)	27.5 (23.6–32.8)
Rise in hypoglycemia relative risk that generated an alert, median (IQR)	10.57 (10.24–11.33)

IQR = interquartile range.

We reviewed 589 total threshold alerts, 350 (59.4%) of which were first alerts for the ICU admission. These alerts correctly predicted 48 cases of level 1 hypoglycemia (positive predictive value [PPV] = 13.7%), 21 (44%) of which involved level 2 hypoglycemia. During the study period, there were 199 hypoglycemic episodes for which no alert was dispatched (sensitivity = 19.4%). All 48 hypoglycemia cases were treated with intravenous dextrose. Three (6.3%) were treated with liquid sugar via oral or enteral access, whereas none were treated with oral glucose tablet administration. In seven cases (14.6%), the subcutaneous insulin regimen was adjusted less than 12 hours after the alert.

Association of Clinical Deterioration Events With ICU Hypoglycemia Alerts

We identified three myocardial infarctions, one medical emergency team call, 19 deaths, and 20 arrhythmias that occurred greater than or equal to 1 and less than or equal to 12 hours after an alert threshold was met.

DISCUSSION

Appropriate glycemic control is a necessary but often overlooked component of quality-driven inpatient healthcare. However, ICU hypoglycemia is consistently linked to greater morbidity and mortality and is routinely identified as the limiting factor for improving glycemic control (1, 9). These points emphasize the need for a more proactive approach to ICU hypoglycemia. Herein, we studied a predictive model’s ability to prospectively identify ICU hypoglycemia and found that our risk alert threshold had a PPV of 13.7% and a sensitivity of 19.4% for true hypoglycemia events. We note, however, that the hypothetical alert system was calibrated to dispatch only two alerts per day in order to limit false positives during this initial testing phase and that this calibration inherently reduced sensitivity. Dispatching false positive alerts that send a provider to the bedside when the patient is not heading toward hypoglycemia is a critical issue that should be avoided, as previous studies indicate that it does not take many false positives before a provider stops responding to or overrides alerts (10, 11). The PPV performance of our alert threshold is in-line with similar studies in this field. For example, Mathioudakis et al (12) developed and validated a machine learning model to predict near-term hypoglycemia risk in non-ICU patients that achieved PPV values of 9% during internal validation and 12–13% during external validation. Our alert threshold achieved a similar PPV for hypoglycemia in an ICU population, providing important data as we seek to incorporate predictive analytics monitoring into clinical trials of a prospective alert system.

We also identified numerous adverse clinical events that occurred after an alert, including cardiac arrhythmias, myocardial infarction, and death. Although it is uncertain whether hypoglycemia directly contributed to or caused these adverse clinical events and the current study was not designed to answer that question, there are numerous reports that support their relationship. For example, the pronounced sympathoadrenal activation during hypoglycemia is known to cause abnormal cardiac repolarization that can induce cardiac arrhythmias (2).

Our study has several limitations that should be noted. First, data collection was limited to one tertiary academic medical center with a high proportion of medically complex patients that may limit generalizability. Second, we note that risk spikes of greater than or equal to 10 have different meanings depending on the baseline level. For example, a rise from 0.1 to 10.1 is a larger relative increase in probability than a rise from 2 to 12. Nonetheless, we felt this was a clinically acceptable approach for future trials and implementations. Finally, we developed and validated our predictive model in insulin-treated ICU patients, but the alert threshold was employed in all ICU patients for the current study.

CONCLUSIONS

Alerts generated by our validated ICU hypoglycemia prediction model had a PPV of 13.7% for real-world hypoglycemia events. To complete impact analysis, we are currently planning a cluster-randomized controlled clinical trial where we will incorporate our model into a prospective alert system and test its impact on hypoglycemia and associated endpoints like mortality and length of stay.