Outcome Measures for Outpatient Hypoglycemia Prevention Studies

Abstract

Systems are being developed that utilize algorithms to predict impending hypoglycemia using commercially available continuous glucose monitoring (CGM) devices and to discontinue insulin delivery if hypoglycemia is predicted. In outpatient studies designed to test such systems, CGM-measured glycemic indices will not only be important outcome measures of efficacy but, in certain cases, will be the only good outcome. This is especially true in short-term studies designed to reduce hypoglycemia since the event rate for severe hypoglycemic events is too low for it to be a good outcome, and milder hypoglycemia often will be variably detected. Continuous glucose monitoring inaccuracy can be accounted for in the study design by increasing sample size and/or study duration.

Introduction

The prevention of hypoglycemia is a goal of all management strategies for type 1 diabetes mellitus (T1DM). Although the hope for the future is that a well-functioning artificial pancreas can essentially eliminate hypoglycemia, the development of a fully automated closed-loop artificial pancreas will still take many years to develop and validate. An important first step would be to discontinue insulin delivery if hypoglycemia is predicted and the patient does not respond by decreasing pump insulin delivery or eating carbohydrate to raise the glucose level. With this goal in mind, algorithms are being developed to predict impending hypoglycemia using commercially available continuous glucose monitoring (CGM) devices. The initial step in clinical testing of a system that predicts hypoglycemia based on CGM and discontinues pump insulin delivery when impending hypoglycemia is predicted must take place in a well-monitored inpatient setting. After the algorithm has been optimized, the next step is to evaluate the efficacy and safety in an outpatient setting. In designing such a hypoglycemia prevention trial, it is necessary to decide what is the most appropriate outcome measure to assess the efficacy of the algorithm and how the sample size and study duration should be determined.

A system that discontinues insulin delivery when hypoglycemia is predicted would be particularly valuable during the overnight period when hypoglycemia occurs frequently and is not detected. In a study conducted by the Diabetes Research in Children Network (DirecNet), the incidence of nocturnal hypoglycemia (glucose ≥60 mg/dl) in children with T1DM was 48% on nights following afternoon exercise and 28% on nights after no exercise.¹ In another study, 57% of adult subjects who did not have a bedtime snack became hypoglycemic (<70 mg/dl) with an average duration of hypoglycemia of 4.4 h.²

We will be conducting an outpatient clinical trial to assess efficacy and safety of a nocturnal hypoglycemia prediction algorithm. Participants will receive insulin through an insulin pump and utilize a CGM device. These two devices will communicate with a bedside computer that will receive glucose data, run a computer program with the algorithm, and turn off the insulin pump when hypoglycemia is predicted. In the clinical trial, a randomization schedule on the computer defines whether the algorithm is being implemented each night (intervention) or not (control).

In this commentary, we provide background information with respect to selection of outcome measures for such a trial and the impact of outcome selection on sample size and trial duration. Hypoglycemia outcomes could be (1) blood glucose documentation of a glucose value below a specified level with or without symptoms of hypoglycemia, (2) a severe hypoglycemic event (SH), or (3) CGM documentation of a glucose value below a specified level with or without symptoms of hypoglycemia. Herein, we will demonstrate that CGM-measured hypoglycemia is the only practical outcome measure for a study of this type. The principles that are presented are pertinent to any clinical trial in which the objective is prevention of nocturnal hypoglycemia or hypoglycemia in individuals with hypoglycemia unawareness.

Documented Hypoglycemia with Blood Glucose Measurement as an Outcome

Although, as noted earlier, biochemical hypoglycemia over- night is common, the vast majority of the time, nocturnal biochemical hypoglycemia does not awaken the individual.³ It is a reason people with diabetes can have hypoglycemic seizures during the night rather than awakening (55% of severe episodes occurred during the night in the Diabetes Control and Complications Trial⁴ and 75% of seizures in children have been reported to occur during the night⁵). The use of a real-time sensor with an alarm also is no guarantee that a subject will be awakened at the time of hypoglycemia. When subjects were videotaped wearing a real-time glucose sensor while sleeping, they awoke only to 29% of the alarms,³ and during the Juvenile Diabetes Research Foundation (JDRF) randomized clinical trial with subjects wearing sensors with real-time alarms, the mean duration of hypoglycemia when it occurred was 81 ± 75 min, with 47% of such nights having at least 1 h, 23% at least 2 h, and 11% at least 3 h of hypoglycemia.⁶ The prolonged duration of hypoglycemia with real-time alarms indicates that we cannot rely on a study participant's awakening to document hypoglycemia even if a sensor alarms. In addition, awakening the individual in the middle of the night to measure blood glucose would not be feasible, because participants would not do this for a large number of consecutive nights and a single time point during the night could miss the occurrence of hypoglycemia at other times during the night. Thus, an outcome relying on the subject awakening and documenting an overnight hypoglycemic blood glucose value would result in a very high false negative rate, limiting the feasibility of such an outcome.

Severe Hypoglycemia Events

As indicated earlier, many episodes of nocturnal hypoglycemia do not awaken the individual and thus would not be detected. In contrast, a SH (e.g., seizure, coma, or incapacitation sufficient that assistance was needed to treat the event) will be apparent. Thus, the only type of clinical hypoglycemic event that might be considered for an outcome in a nocturnal hypoglycemia study would be a SH. However, use of a SH as an outcome lacks feasibility in a clinical trial of this type, because the occurrence rate is very low. As a result, the required sample size would be extraordinarily and unfeasibly large and/or the trial would need to be extremely long.

Table 1 shows published rates of SH without respect to day or night. As can be seen, studies have been fairly consistent with SH rates in the range of 13 to 18 events per 100 person-years among adults and children with intensively treated T1DM. Even if it is assumed that nocturnal events are more common than daytime events, it is likely that the rate of nocturnal events is no greater than 10 events per 100 person-years.

Table 1.

Published Rates of Severe Hypoglycemia Events in Type 1 Diabetes Mellitus^a

Study/group	N	Follow-up	SH rateb
DCCT Intervention Group7	711	6.5 years	61.2
DCCT Control Group7	730	6.5 years	18.7
DirecNet Glucowatch Biographer RCT8	200	6 months	∼12
JDRF CGM RCT9	436	12 months	17.9
STAR 3 Insulin Pump Group10	244	1 year	13.3
STAR 3 Insulin Injections Group10	241	1 year	13.5

^aIncludes both day and night. DCCT, Diabetes Control and Complications Trial; STAR, Sensor-Augmented Pump Therapy for A1C Reduction.

^bEvents per 100 person-years.

In a trial, the SH rate in the control group might be greater if eligibility was restricted to individuals more likely to experience a SH [e.g., low hemoglobin A1c (HbA1c) and/or self-report of prior SHs], but this may limit recruitment, and the necessary sample size will still be very large. In the JDRF CGM randomized control trial (RCT),there were 36 subjects who had self-reported a SH in the 6 months prior to study entry, and the subsequent rate observed for these subjects was 13.1 nocturnal (midnight–6:00 am) SHs per 100 person-years (unpublished data).

Table 2 provides approximate sample sizes under a wide range of possible rates of nocturnal SH and study durations to achieve 90% power with a two-tailed test and type 1 error rate = 5% based on a Poisson model. Even if the nocturnal SH rate were projected to be 15 events per 100 person-years, the sample size for a 3-month study exceeds 3400 and the sample size for a 6-month study exceeds 1700.

Table 2.

Sample Size and Study Duration Estimates for a Clinical Trial Using Severe Hypoglycemia as the Primary Outcome^a

SH rate in control groupb	Length of follow-up	SH relative reduction by interventionc
		50%	25%	10%
5	1 month	31,300	145,000	968,000
	3 months	10,400	48,200	323,000
	6 months	5220	24,100	161,000
15	1 month	10,400	48,200	323,000
	3 months	3480	16,100	108,000
	6 months	1740	8030	53,800
30	1 month	5220	24,100	161,000
	3 months	1740	8030	53,800
	6 months	870	4010	26,900
45	1 month	3480	16,100	108,000
	3 months	1160	5350	35,900
	6 months	580	2680	17,900

^aTotal number of subjects. Half would be randomized to each group. Assumes 90% power with type 1 error rate of 5% (two-tailed) and 1:1 allocation to intervention and control. This model likely underestimates the true sample size necessary because it assumes that every subject has the same Poisson rate whereas the true distribution is likely a mixture of Poisson outcomes.

^bNocturnal events per 100 person-years.

^cFor example, a 25% reduction when the SH rate is 30 in the control group would result in a rate of 22.5 in the intervention group.

Continuous-Glucose-Monitoring-Measured Hypoglycemia

The problems inherent in using a blood glucose measurement to document overnight hypoglycemia are avoided by using CGM to assess hypoglycemia. Current CGM devices record a glucose measurement every 1 to 5 minutes, depending on the device. For certain outpatient studies designed to reduce hypoglycemia, particularly overnight studies, CGM may be the only good outcome. A drawback to using current generation CGM as an outcome measure is that its accuracy is less than blood glucose measurements. However, with available data to estimate the variability of CGM, this inaccuracy can be accounted for in the study design by increasing sample size as will be discussed later.

In a nocturnal hypoglycemia prevention trial, CGM-measured outcomes could include the occurrence of a single glucose level below a threshold such as 60 or 70 mg/dl, consecutive glucose values below a threshold, or such measures as area under the curve or low blood glucose index. From prior analyses of the JDRF CGM RCT data set (unpublished), all four of these outcome measures are highly correlated. When a single glucose level is the outcome, there are several considerations in selecting the level to use for this purpose. The outcome level (1) should be at or below the level at which a counter-regulatory response to hypoglycemia occurs, (2) should be rare in people without diabetes, (3) should occur with sufficient frequency overnight in people with diabetes with respect to determination of the sample size for the study, and (4) should be at or above the glucose level at which the CGM hypoglycemia alarm is set. Setting the threshold to be 60 mg/dl, 70 mg/dl, or a level in between could satisfy all four criteria.

As summarized by the American Diabetes Association Workgroup on Hypoglycemia,¹¹ the counter-regulatory response to decreasing blood glucose is triggered at a blood glucose level of 65–70 mg/dl, and frequent episodes of lower glucose levels, including asymptomatic nocturnal hypoglycemia, lead to defective counter regulation and hypoglycemia unawareness. We (and others) have found that 1.5% of glucose readings in people who do not have diabetes are between 60 and 70 mg/dl (≈4 readings or 20 min each day) while values ≥60 mg/dl make up only 0.2% of CGM glucose readings in people who do not have diabetes.¹² To achieve sufficient statistical power, sample size and study duration will be highly dependent on the frequency of hypoglycemia in the control arm (i.e., hypoglycemia prediction algorithm not active). Data from the JDRF CGM RCT were used to estimate the frequency of CGM-determined nocturnal hypoglycemia that might be expected in the control arm of a nocturnal hypoglycemia prevention trial (Table 3).¹³ Continuous glucose monitoring data were evaluated from 10:00 pm to 6:00 am during the first 6 months of the study in 231 subjects in the CGM treatment group (age 8–72 years, baseline HbA1c 4.7–10.6%). Analysis was restricted to 25,473 nights with at least 6 h of CGM data, ranging from 3 to 182 per subject. Results for frequency of hypoglycemia at different glucose thresholds are summarized in Table 3. It can be seen that, as expected, the hypoglycemia rate is greater at a threshold of 70 mg/dl than at 60 mg/dl, which is greater than the rate at 50 mg/dl. It also can be seen in the table that rates are similar for an outcome of a single value below the threshold and for an outcome requiring two consecutive values below the threshold. From a sample size perspective, the hypoglycemia rate for a threshold of 50 mg/dl is too low for a trial to be feasible. A level of either 60 or 70 mg/dl could be feasible, although the sample size requirement will be lower for a threshold of 70 mg/dl because of the higher hypoglycemia rate.

Table 3.

Frequency of Nocturnal Hypoglycemia in the Juvenile Diabetes Research Foundation Continuous Glucose Monitoring Randomized Control Trial^a

	Hypoglycemic threshold
Binary outcomes	≥70 mg/dl	≥60 mg/dl	≥50 mg/dl
	% of nights
Single value hypoglycemic	25.4%	14.8%	7.6%
2 consecutive values hypoglycemic	23.3%	13.2%	6.7%
20 consecutive minutesa hypoglycemic	18.4%	9.7%	4.5%
60 consecutive minutes hypoglycemic	9.7%	4.6%	2.0%

^aContinuous glucose monitoring data were evaluated from 10:00 pm to 6:00 am during the first six months of the study in 231 subjects in the CGM treatment group (age 8–72 years, baseline HbA1c 4.7–10.6%). Analysis was restricted to the 25,473 nights with at least 6 h of CGM data, ranging from 3 to 182 per subject.

Table 4 provides sample sizes and number of study nights for a range of treatment effects in order to have statistical power of 90% with a type 1 error rate of 5%. As can be seen, the sample size will be lower with a threshold of 70 mg/dl than of 60 mg/dl for a relative 50% reduction in the frequency of nocturnal hypoglycemia. It can also be seen in the table that another factor favoring 70 mg/dl as the threshold is that CGM accuracy is better at 70 mg/dl than at 60 mg/dl as will be discussed in the next section.

Table 4.

Sample Size^a and Study Duration Estimates for Binary Hypoglycemia Outcomes in a Nocturnal Hypoglycemia Prevention Trial^b

Treatment effect (intervention versus control)	Nights per subject	Unadjustedc
		Total subjects	Total nights
Outcome threshold 70 mg/dl
25% versus 12.5%	14	26	364
	28	13	364
	42	9	378
	56	7	392
25% versus 8.3%	14	13	182
	28	7	196
	42	5	210
	56	4	224
Outcome threshold 60 mg/dl
15% versus 7.5%	14	47	658
	28	24	672
	42	16	672
	56	12	672
15% versus 5%	14	24	336
	28	12	336
	42	8	336
	56	6	336

^aThe sample size estimates apply to a study design using any type of CGM device with false positive and false negative rates comparable to those described in the text.

^bAssuming two-tailed test with alpha = 0.05 and 90% power.

^cBinomial variance decreased by 12% to account for subject serving as their own control in the crossover design. Correlation from repeated nights from the same subject using sensor data were estimated from data obtained in the JDRF CGM RCT.⁶ As described in the text, for an outcome threshold of 70 mg/dl, the number of nights should be increased by a factor of 3 to account for misclassification and increased by a factor of 5 for a threshold of 60 mg/dl.

Effect of Continuous Glucose Monitoring Measurement Error on Sample Size

Measurement error can impact study results when CGM data are used as the outcome variable. When the outcome is a binary event such as whether or not hypoglycemia occurred, using CGM to measure outcome will give some false positives (indicating hypoglycemia when it did not actually occur) and some false negatives (missing true occurrences of hypoglycemia). Statistical analysis of CGM data as an outcome, therefore, must consider the potential for misclassification of the outcome when the outcome is binary and a bias and/or increased variance when the outcome is continuous compared with using blood glucose measurements as the outcome. Nondifferential bias (i.e., noise) such as this can be accounted for in the study design by increasing the sample size.

Bias can occur when the sensor systematically overestimates or underestimates the percentage of nights and the false positive rate fails to cancel out the false negative rate. Data from two DirecNet studies comparing overnight Navigator CGM glucose values with central laboratory blood glucose determinations indicate that the false negative rate is approximately 20% and the false positive rate approximately 20% for hypoglycemia occurring some time during the night for a threshold of 70 mg/dl and 35% and 20%, respectively, for a threshold of 60 mg/dl (unpublished). When the false positive and false negative rates are similar, a CGM-measured rate of biochemical hypoglycemia will be relatively unbiased. However, even if the false positive and false negative rates differ, use of a control group eliminates differential bias due to misclassification in the analysis of randomized trial data. If the null hypothesis is correct and the intervention has no beneficial effect, then the same bias affecting the observed rate of biochemical hypoglycemia would occur in both the intervention and the control groups, and thus any such bias could not produce a treatment group difference.

Although these data indicate that there will not be differential bias when using a sensor to determine a hypoglycemia outcome, sensor inaccuracy nevertheless potentially dilutes the observed treatment effect. This can be estimated by subtracting the estimated false positive and false negative rates as follows to determine the proportion of the true treatment effect that will be observed assuming the null hypothesis of no treatment group difference is false. For instance, if both false positive and false negative rates are 20% (the rates that we have estimated for a threshold of 70 mg/dl), this would equate with dilution of the treatment effect to be 60% of the true treatment effect (100–20–20%). However, if the rates are 20% and 35% (the rates that we have estimated for a threshold of 60 mg/dl), then the dilution of the treatment effect is approximately 45%. Knowledge of the degree of potential misclassification of a hypoglycemia outcome can then be used to adjust the sample size so that statistical power will not be reduced by misclassification of outcome. In this example, this would increase the sample size by a factor of 60%⁻² ≈ 3 for a threshold of 70 mg/dl and by a factor of 45%⁻² ≈ 5 for a threshold of 60 mg/dl.

Conclusion

In outpatient studies conducted to test strategies to reduce nocturnal hypoglycemia, CGM-measured glycemic indices will not only be the most important outcome measures of efficacy, but in many cases be the only good outcomes. This is especially true in short-term studies designed to reduce hypoglycemia since the event rate for SHs is too low for it to be a good outcome, and milder hypoglycemia often will be undetected. Although current generation CGM has greater inaccuracy than blood glucose measurements, with available data to estimate variability of CGM, this inaccuracy can be accounted for in the study design by increasing sample size and/or study duration.

Glossary

Abbreviations

(CGM)

continuous glucose monitoring

(DirecNet)

Diabetes Research in Children Network

(HbA1c)

hemoglobin A1c

(JDRF)

Juvenile Diabetes Research Foundation

(RCT)

randomized control trial

(SH)

severe hypoglycemic event

(T1DM)

type 1 diabetes mellitus

Funding

Study funding was provided by the Juvenile Diabetes Research Foundation, Inc. Grant #22-2009-802.