A Methodology to Compare Insulin Dosing Recommendations in Real-Life Settings

Abstract

Background:

We propose a methodology to analyze complex real-life glucose data in insulin pump users.

Methods:

Patients with type 1 diabetes (T1D) on insulin pumps were recruited from an academic endocrinology practice. Glucose data, insulin bolus (IB) amounts, and self-reported alcohol consumption and exercise events were collected for 30 days. Rules were developed to retrospectively compare IB recommendations from the insulin pump bolus calculator (IPBC) against recommendations from a proposed decision aid (PDA) and for assessing the PDA’s recommendation for exercise and alcohol.

Results:

Data from 15 participants were analyzed. When considering instances where glucose was below target, the PDA recommended a smaller dose in 14%, but a larger dose in 13% and an equivalent IB in 73%. For glucose levels at target, the PDA suggested an equivalent IB in 58% compared to the subject’s IPBC, but higher doses in 20% and lower in 22%. In events where postprandial glucose was higher than target, the PDA suggested higher doses in 25%, lower doses in 13%, and equivalent doses in 62%. In 64% of all alcohol events the PDA would have provided appropriate advice. In 75% of exercise events, the PDA appropriately advised an IB, a carbohydrate snack, or neither.

Conclusions:

This study provides a methodology to systematically analyze real-life data generated by insulin pumps and allowed a preliminary analysis of the performance of the PDA for insulin dosing. Further testing of the methodological approach in a broader diabetes population and prospective testing of the PDA are needed.

Current standards of care for patients with type 1 diabetes (T1D) advocate for tight control of blood glucose (BG).¹ One treatment challenge for patients with T1D is optimization of postprandial glucose levels.^2-4 To help patients achieve improved glucose regulation, continuous subcutaneous insulin infusion devices (CSII, aka “insulin pumps”) sometimes coupled with continuous glucose monitoring systems (CGMs), have been developed. Although devices can assist patients in making insulin dosing decisions through the use of bolus calculators, it is unknown how accurate the bolus recommendations are in real-life scenarios when complex lifestyle choices, such as exercise and alcohol intake, have to be considered in decision making. Recent data suggests that patients are often confused and inconsistent when trying to factor in these behaviors when deciding insulin doses.^5,6

Models exist to study insulin delivery recommendations in controlled, simulated settings. Before undergoing clinical trials, a common practice to facilitate the design, development and testing of diabetes technology is to use in silico methods.^7-12 Recently, Wong et al proposed a method to retrospectively compare insulin bolus (IB) recommendations using Intensive Care Unit (ICU) data.¹³ They concluded that in silico comparisons appear to be an efficient nonclinical method for allowing rapid and inexpensive identification of computer-based protocols that justify expensive and burdensome clinical trials.

Although models exist to study IB recommendations in controlled environments, there is a lack of methods capable of analyzing glucose data simultaneously with patient behaviors. The aims of this study were to (1) develop an analytic method to retrospectively compare prandial IB recommendations, (2) apply the proposed method in a real-life setting to test the performance of an evidence-based proposed decision aid (PDA) against the bolus calculator of an insulin pump, and (3) share lessons learned from collecting, aggregating and analyzing real-life data generated by insulin pumps and self-reported patient behaviors.

Methods

Description of the iDECIDE Evidence-Based Insulin Bolusing Dosing Decision Aid

iDECIDE, the PDA evaluated here, is an evidence-based decision aid to recommend IB doses, carbohydrate intake, or both, by taking into account carbohydrates and alcohol consumed, and/or exercise plans.¹⁴ The PDA was deployed as a smartphone app to help patients with T1D incorporate varied lifestyle choices simultaneously into decisions about prandial insulin dosing. The PDA is based on the formula proposed by Colin¹⁵ to include alcohol,^16,17 exercise,^18-21 and the absorption rate of rapid-acting insulin.²² The PDA corrects to the nearest target glucose setting when the blood glucose is out of range, but would not account for the CGMS trendline. Exercise is accounted for based on body weight and duration and intensity of exercise, while the alcoholic beverage type and volume consumed are necessary to adjust for alcoholic beverages.

When the user launches the PDA application the first time he is prompted to set up a diabetes profile: weight, insulin-to-carbohydrate ratios, target BG levels, correction factors and active insulin time.²³ Although participants did not set up their user profile for the study, those that did not use paper logs interacted with the self-reporting module to log (1) exercise, describing duration and intensity, (2) food intake, specifying food type, serving size and carbohydrate content, and (3) alcohol intake, indicating number of drinks, size, and type of drink (Figure 1). In addition, when self-reporting plans, the user is expected to enter the BG reading. The PDA subsequently recommends an IB or carbohydrate intake by incorporating current evidence on the way food and alcohol carbohydrates and exercise influence BG, but these recommendations were assessed retrospectively and were not provided to the participants.

Figure 1.

Screenshots of the iDECIDE mobile application. (A) Self-reported exercise plans. (B) Self-reported plans for food and alcohol consumption. (C) Summary of relevant preprandial information. (D) Advice to take 5 grams of snack carbohydrates to avoid exercise-induced hypoglycemia.

Participant Recruitment

Following Institutional Review Board approval, 31 study participants were recruited from an outpatient academic endocrinology practice. Patients with T1D 18 years or older who had been under the care of the endocrinology team while on CSII therapy using a Medtronic (Minneapolis, MN) insulin pump for at least one year were eligible to participate.

Data Collection

Participants were asked to continue their usual fitness and nutrition routine. For 30 days, participants recorded their exercise activity and alcohol consumption via paper logs or the self-reporting module of the PDA, according to subject’s preferences. Exercise was recorded by start time, duration and intensity, and categorized as light, moderate or vigorous. Alcohol was recorded by tracking drink time, type, volume, and number (eg, 6PM, 1 pint of beer, no carbohydrates entered). Carbohydrate content was entered in the insulin pump. After 30 days, logs were manually encoded into tables or downloaded from a secure cloud-based server.

Self-reported data on exercise and alcohol was used as input for the PDA. For exercise, the PDA recommends an IB or carbohydrate intake by considering body weight and intensity and duration of exercise.^18-21 For alcohol, the PDA accounts for the carbohydrates of the alcoholic drinks based on type, volume and count.

CSII data from the corresponding 30-day timeframe was downloaded in tabular format. CSII device data included carbohydrates recorded by the participant, glucose levels either from a CGMS or capillary BG monitor or both, amount of insulin delivered, pump settings, and the IB suggested by the insulin pump bolus calculator (IPBC).

Retrospective Comparison of Two Insulin Bolus Calculators

To evaluate the performance of the PDA against conventional approaches to prandial insulin dosing, the authors adapted methodology from Wong et al.¹³ For this study, the conventional approaches to insulin dosing were defined as either use of the IPBC or participant’s self-determined doses. The PDA’s recommendations were compared against those made by the participant’s IPBC, or against the participant when they either overrode or neglected to get advice from their IPBC (Figure 2).

Figure 2.

Method used to retrospectively compare recommendations from two insulin bolus calculators, BCa and BCp. If the recommendations from BCa and BCp were within 10% of each other they were considered to be equivalent. If the BG at t_i+1 was low, then the smaller of the two recommendations from BCa and BCp was considered appropriate; if they were equivalent then neither was considered appropriate. If the BG at t_i+1 was at target, then the smaller of the two recommendations from BCa and BCp was defined as appropriate, preferring recommendations that could avoid hypoglycemic events; if they were equivalent then both were considered appropriate. If the BG at t_i+1 was above target, then the larger of the two recommendations from BCa and BCp was deemed appropriate; if they were equivalent then neither was considered appropriate. We considered that one calculator outperformed the other if there was a major performance enhancement over the competitor calculator. In the case of on target postprandial BG, we consider that a lower insulin dose recommendation outperformed higher insulin dosing advice, potentially avoiding a hypoglycemic event.

The “appropriateness” of an IB was defined as one that brings the postprandial glucose to the desired target.¹³ The method assumes that a conventional insulin dosing calculator, BCa (ie, IPBC or the participant), has made an IB recommendation. The point in time when BCa made the IB suggestion and when the insulin was delivered is referred to as the initial time, t_i. The method assumes that a proposed insulin dosing calculator, BCp (eg, PDA), is retrospectively executed at the same data point, t_i, to compare at time t_i+1 the effect on BG of the insulin suggestion from BCp against the actual suggestion that was made by BCa. We considered that one calculator “outperformed” another calculator if there was a major performance enhancement over the competitor. For instance, in the case of a low postprandial BG we consider that a lower insulin dose recommendation outperformed higher insulin dose advice, potentially avoiding a hypoglycemic event.

Applying this methodology requires that each preprandial BG at t_i can be paired with a corresponding postprandial BG at t_i+1. For meal events and BG corrections, we defined t_i+1 to be the first BG reading obtained 3 hours ± 15 minutes, after t_i. This time frame was chosen considering that the majority of the carbohydrate load and the rapid-acting insulin analog bolus would have been absorbed and BG levels would have stabilized.¹⁶ The BG readings at t_i+1 were broken into three categories, based on predetermined individual target BG levels obtained from the insulin pump settings of each participant. The analysis determines which calculator provided at time t_i an IB recommendation that would have placed the participant closer to their target BG based on the category of the actual BG reading at t_i+1. In the case of a target postprandial BG reading, we considered that a smaller insulin recommendation outperforms a larger recommendation because it could have avoided a hypoglycemic event.

The method outlined in Figure 2 was used to compare the appropriateness of two calculators, BCa and BCp, and assumes that BCa (IPBC) has made IB recommendations that were delivered to the patient. A variation of that method is needed to assess the appropriateness of recommendations from BCp (PDA) when there is no available data from BCa (ie, no recommendation from the IPBC).

Assessing the Appropriateness of an Insulin Bolus Recommendation for Alcohol and Exercise

Conventional IPBCs do not provide IB recommendations for alcohol. For these cases the method explained in Figure 3 was adopted. The postprandial time frame of interest, t_i+1, was defined as the first BG reading obtained within 3 hours ± 15 minutes. This time frame neglects to consider any delayed effects from alcohol induced hypoglycemia and primarily focuses on the carbohydrates associated with alcoholic beverages.

Figure 3.

Method used for assessing the appropriateness of the recommendations from the proposed decision aid (PDA), when patients choose to consume alcohol, for which the insulin pump bolus calculator does not provide insulin dosing recommendations. If the BG at t_i+1 was low or at target and the PDA did not recommend insulin the recommendation from the PDA was appropriate; if the PDA recommended insulin the recommendation was not considered appropriate. If the BG at t_i+1 was high and the PDA recommended insulin the recommendation from the PDA was appropriate; if the PDA did not recommend insulin the recommendation was not considered appropriate. Given that our PDA is not compared against another calculator, outperformance is not defined.

As with alcohol ingestion, when participants exercised there were no recommendations made by the IPBC. For those cases, we used the method in Figure 4. We modified the window of t_i+1 to be the first BG reading within 15 minutes of finishing exercise as recorded by the participant to detect any immediate effects of exercise-induced hypoglycemia. For example, if the participant finished exercising at 8:30 am, we used the first available BG between 8:30 and 8:45 am. In the case of exercise, the PDA’s recommendations could be a carbohydrate snack in addition to an IB dose. For exercise scenarios, the appropriateness of the IB and/or carbohydrate was defined as in Figure 4.

Figure 4.

Method used for assessing the appropriateness of the recommendations from the proposed decision aid (PDA) when patients choose to exercise, for which the insulin pump bolus calculator does not provide insulin dosing or carbohydrate intake recommendations. If the BG at t_i+1 was low or at target and the PDA suggested nothing or suggested consuming carbohydrates the recommendation from the PDA was considered appropriate; if the PDA recommended insulin, then the recommendation was deemed not appropriate. If the BG at t_i+1 was high and the PDA suggested insulin the recommendation from the PDA was considered appropriate; if the PDA suggested no insulin or recommended consuming carbohydrates, then the recommendation was not considered appropriate. Given that the PDA is not compared against another calculator, outperformance is not defined.

Data Analysis

Computer programs were written to automate the process of collating and analyzing the data generated by the insulin pumps with the self-reported patient behaviors. In addition, assessing the performance of the PDA at t_i+1 against the IPBC, or against participants’ self-dosing choices when the IPBC was not used as anticipated was automated. Comparisons were made according to participant glucose targets (below, at, or above target).

Results

Participant Characteristics and Data

There were 31 participants recruited for the study, with 4 withdrawals. Of the remaining 27 participants, a subset of 15 participants (Table 1) had preprandial glucose readings paired with t_i+1 BG readings, with 13 of them on CGMS (9 on Minimed 530G-551, 3 on Minimed 530G-751, and 1 on Paradigm Revel-723).

Table 1.

Demographics of 15 Subjects With Type 1 Diabetes.

Characteristic	Value
Age (years)	48.7 (13.9)
% women	73.3
% white	93.3
Hemoglobin A1C	7.5 (1.2)
Diabetes duration (years)	26.9 (11.8)
Duration on insulin pump (years)	11.5 (5.3)
Daytime low/high target BG	89.9 (8.6) / 112.3 (10.8)
# Analyzable exercise events/day	1.1 (0.34)
# Analyzable alcohol events/day	0.2 (0.18)

Data are mean (SD) or %.

A total of 2104 events had postprandial glucose readings that allowed for a comparison between the IPBC and the PDA, and there were 419 events where the PDA was compared against cases where the participants did not use their IPBC, they overrode the IPBC recommendations, or they did not provide a prandial BG. There were 235 exercise and 105 alcohol events that had sufficient data for analysis. Most (56%) exercise events were of moderate intensity. There were few (14%) alcohol events where participants accounted for the carbohydrates associated with the beverage.

IPBCs allow different settings (BG target, insulin-to-carbohydrate ratio, and correction factor) throughout the day and the PDA accounted for these different settings for each participant at each time of day. While participants used different Medtronic insulin pumps, all used the same formula for computing IB recommendations, and an adaptation of Mudaliar et al²⁴ for computing active insulin. The Medtronic 530G includes a threshold suspend feature, that is designed to automatically stop insulin delivery when the CGMS value falls below a patient-specific preset threshold. There were 5 insulin suspension events that occurred in close temporal proximity to events of interest; such low frequency did not warrant removing data from the analysis.

Comparison of iDECIDE Against the IPBC or Patient

We used the method described in Figure 2 to compare the appropriateness of the PDA’s recommendations against events when the patient followed the IPBC recommendations for BG correction doses and/or carbohydrate loads that included a prandial and postprandial BG.

First assessed was how the PDA (ie, iDECIDE) compared against the IPBC (Table 2). The IPBC brought the participants to target glucose levels in 13% (278/2104) events, below target in 10% (207/2104) and above target in 77% (1619/2104). When considering very low and very high postprandial BG, the BG was below 70 mg/dl in 3% (55/2104) and over 180 mg/dl in 35% (737/2104). When considering instances where glucose was below target, the PDA would have recommended an appropriately smaller dose in 14% (28/207), but a larger dose in 13% (27/207) and an equivalent IB in 73% (152/207). For glucose levels at target, the PDA would have suggested an equivalent IB in 58% (162/278) compared to the subject’s IPBC, but a higher dose in 20% (56/278) and lower in 22% (60/278). In events where postprandial glucose was higher than target, the PDA would have suggested a higher dose in 25% (406/1619), a lower dose in 13% (212/1619), and an equivalent dose in 62% (1001/1619). Overall, the PDA would have recommended an equivalent dose compared to the IPBC in 63% (1315/2104) of IB decisions.

Table 2.

Results From the Retrospective Comparison of the Appropriateness of the Recommendations From the Proposed Decision Aid (PDA) Against the Insulin Pump Bolus Calculator (IPBC), and From the PDA Against the Patient’s Self-Dosing Choices.

Event type	Postprandial BG (mg/dl)	PDA insulin recommendations			Total
		Larger dose	Smaller dose	Equivalent dose
IPBC	Low (<target)	27a	28b	152c	207
	Target (participant target)	56a	60b	162d	278
	High (>target)	406b	212a	1001c	1619
	Total	489	300	1315	2104
Participant	Low (<target)	10a	23b	21c	54
	Target (participant target)	18a	24b	4d	46
	High (>target)	107b	86a	126c	319
	Total	135	133	151	419

^aIPBC (or participant) was appropriate and the PDA recommendation was not appropriate, IPBC (or participant) outperformed the PDA.

^bPDA recommendation was appropriate and IPBC (or participant) was not appropriate, PDA outperformed the bolus calculator (or patient). When the PDA recommends a lower insulin dose recommendation than the bolus calculator (or participant) and the postprandial BG is on target, the PDA could potentially avoid a hypoglycemic event and therefore outperformed the bolus calculator (or participant).

^cEvents where the PDA and IPBC (or participant) recommendations were not appropriate.

^dEvents where the PDA and IPBC (or participant) recommendations were appropriate.

We used the method described in Figure 2 to compare the appropriateness of the PDA against decisions made by the participant (Table 2). The participants self-dosing led to above target postprandial glucose in 76% (319/419), below target in 13% (54/419) while participants only achieved target glucose levels in 11% (46/419). There were 3% (14/419) of the events with a postprandial BG below 70 mg/dl and 37% (154/419) over 180 mg/dl. When considering instances where glucose was below target, the PDA would have recommended an appropriately smaller dose in 43% (23/54), a larger dose in 19% (10/54), and an equivalent IB dose in 38% (21/54). For glucose levels at target, the PDA would have suggested an equivalent IB amount in 9% (4/46) compared to the subject’s own decision, but a higher dose 39% (18/46) and lower in 52% (24/46). In situations where postprandial glucose was greater than target, the PDA would have suggested a higher dose in 34% (107/319), a lower dose in 27% (86/319), and an equivalent dose in 39% (126/319). Overall, the PDA would have recommended an equivalent IB in only 36% (151/419) of instances compared to when the participants made their own decisions.

Assessment of the Appropriateness of iDECIDE’s Recommendations for Exercise and Alcohol

In cases of exercise and alcohol the pump does not suggest insulin. In these cases, the PDA is only assessed based on the BG outcomes since it could not be compared against the IPBC. We used the method described in Figure 3 to assess the appropriateness of the PDA’s recommendations when alcohol consumption was recorded. As reported earlier, patients self-reported accounting for the carbohydrate content of the beverage in 15 of the 105 events. As indicated in Table 3, in 64% (67/105) of overall alcohol events the PDA would have provided appropriate advice. The PDA performed well when the postprandial BG was high with 78% (64/82) appropriate IB recommendations, but had poor performance when the postprandial BG was at target with only 5% (1/19) recommendations deemed appropriate.

Table 3.

Results From Assessing the Appropriateness of the Recommendations Regarding Insulin Dosing for Alcohol Consumption From the Proposed Decision Aid (PDA).

Postprandial BG	PDA recommendations		Total
	Appropriate	Not appropriate
Low (<target)	2	2	4
Target (participant target)	1	18	19
High (>target)	64	18	82
Total	67	38	105

We used the method described in Figure 4 to assess the appropriateness of the PDA’s recommendation before exercise (Table 4). The PDA appropriately suggested insulin or to ingest carbohydrates in 75% (176/235). Similar to the alcohol results, the PDA performed well when post exercise BG was high 87% (154/178), but only made appropriate suggestions in 37% (10/27) and 40% (12/30) when the post exercise BG was low or target, respectively. There were 26 exercise events that had a duration of 90 minutes or longer and the PDA made appropriate recommendations in only 27%.

Table 4.

Results From Assessing the Appropriateness of the Recommendations Regarding Insulin Dosing and Carbohydrate Ingestion for Exercise From the Proposed Decision Aid (PDA).

Post exercise BG	PDA insulin dose and carbohydrate recommendations		Total
	Appropriate	Not appropriate
Low (<target)	10	17	27
Target (participant target)	12	18	30
High (>target)	154	24	178
Total	176	59	235

Discussion

Although advances in in silico models technology have allowed for incorporation of new features into existing technologies to improve BG control, these often do not account for variables that affect BG (eg, exercise, stress, sleep, and illness). Decision aids that assist patients with T1D to make better prandial insulin dosing decisions are needed, particularly when patients must account for multiple simultaneous lifestyle variables that may impact BG levels.

One of the main differences between this study and others that retrospectively evaluated the performance of prandial insulin dosing recommendations is the source of the clinical data. For instance, previous studies have compared the effectiveness of insulin dosing recommendations in controlled environments such as in the ICU,^13,25 where glucose control is closely monitored and tracked and lifestyle behaviors are not a factor. In contrast, this study focused on free-living outpatients who made their own choices about insulin therapy, and where individual lifestyle choices have the potential to impact treatment decisions and outcomes.

One of the analytic challenges we encountered when developing, testing, and comparing the effectiveness of insulin dosing recommendations is the complex nature of data generated by free-living participants. In our study, many of the self-management and daily living activities recorded by the participants occurred in tight temporal succession and could not be assessed as isolated events. This required development of a new analytic approach to evaluating the data. An unexpected positive outcome of this study was gaining a better understanding of patients’ self-management behaviors as they interact with insulin pumps.^5,6

The methodology outlined here permitted an assessment of how our PDA would perform when used in different scenarios. When compared to the IPBC embedded in the subject’s insulin pump, the PDA in general was noninferior, recommending IB doses equivalent to the IPBC standard in 63% of decisions overall and nearly equivalent number of smaller doses when glucose levels were below or at target. There were some instances where the PDA was superior to the IPBC, such as when it would have recommended larger doses in cases when postprandial glucose levels were above target. Initial analysis of the PDA in cases where the doses were too large or small, provided insights which were used to improve performance with continuing analysis necessary for further refinement of the recommendations.^26,27 For instance, we used an initial setting of 3 hours of active insulin time to calculate IOB. To improve performance, this was later adjusted to 4 hours which reduced the number of inappropriate recommendations that could have led to hypoglycemia. In the future, the PDA will adapt to the insulin action time specified for each patient.

Employing the analytic paradigms developed here, we also assessed the performance of the PDA when there was a lack of recommendations from the IPBC with exercise and alcohol events. In these analyses the postprandial glucose was used as the outcome measure. For cases involving alcohol consumption, the PDA may have offered an advantage when deciding a compensatory insulin bolus. The PDA could have improved postexercise BG when the duration was 90 minutes or less and the PDA should be restricted to such events until further study.

There are limitations to the study. This study incorporated self-reported data for exercise, meal and alcohol behaviors. It is possible participants did not record all these events, or may have recorded them inaccurately. Also, participants’ insulin pump settings were not adjusted for the study. Inappropriate insulin pump settings, such as basal rates, could have influenced the results. Sample sizes for alcohol and exercise events were small with respect to the larger comparisons involving the IPBC. The study also did not consider late-onset hypoglycemia that can arise from engaging in exercise, and possibly when consuming alcohol. To automate the analysis, we opted against determining an appropriate postexercise timeframe on a case-by-case basis and instead focused on the immediate effects of exercise by employing a standard 15-minute postexercise timeframe. Considering BG levels outside of the time-frames used for analysis in this study is another important factor to consider in the future when assessing and calibrating IB calculators.

In addition, the analysis was done retrospectively. A prospective analysis, where the PDA makes suggestions in real time, would help further delineate its capabilities, improve performance and assess user acceptance. A recent analysis suggests that mobile apps can offer advantages in diabetes management, but more rigorous studies are needed.²⁸ Finally, the analytic methods tested here were for a very specialized group of patients (T1D on insulin pumps) and we did not conduct an analysis of the outcomes in relation to A1c scores. Testing these methodologies in a wider selection and more diverse population of patients (eg, T1D patients on multiple daily insulin injections or patients with type 2 diabetes) would be needed to test the generalizability of the approach.

Conclusion

We introduced an analytic method to use prospective real-life data to retrospectively compare insulin dosing recommendations. This method was used to assess the recommendations of an evidence-based decision aid. Additional prospective testing of the proposed decision aid with a bigger patient cohort is being planned to further validate the proposed method.