Alarm Characterization for Continuous Glucose Monitors Used as Adjuncts to Self-Monitoring of Blood Glucose

Abstract

Background

Continuous glucose monitoring (CGM) devices available in the United States are approved for use as adjuncts to self-monitoring of blood glucose (SMBG). Alarm evaluation in the Clinical and Laboratory Standards Institute (CLSI) guideline for CGM does not specifically address devices that employ both CGM and SMBG. In this report, an alarm evaluation method is proposed for these devices.

Method

The proposed method builds on the CLSI method using data from an in-clinic study of subjects with type 1 diabetes. CGM was used to detect glycemic events, and SMBG was used to determine treatment. To optimize detection of a single glucose level, such as 70 mg/dl, a range of alarm threshold settings was evaluated. The alarm characterization provides a choice of alarm settings that trade off detection and false alarms. Detection of a range of high glucose levels was similarly evaluated.

Results

Using low glucose alarms, detection of 70 mg/dl within 30 minutes increased from 64 to 97% as alarm settings increased from 70 to100 mg/dl, and alarms that did not require treatment (SMBG >85 mg/dl) increased from 18 to 52%. Using high glucose alarms, detection of 180 mg/dl within 30 minutes increased from 87 to 96% as alarm settings decreased from 180 to 165 mg/dl, and alarms that did not require treatment (SMBG <180 mg/dl) increased from 24 to 42%.

Conclusion

The proposed alarm evaluation method provides information for choosing appropriate alarm thresholds and reflects the clinical utility of CGM alarms.

Introduction

Alarms indicating the need for therapeutic intervention are one of the most useful features of continuous glucose monitoring (CGM) devices. Characterization of CGM alarm performance, however, is not straightforward.

The earliest characterizations of CGM alarms borrowed from the methods used for evaluating diagnostic tests, namely sensitivity and specificity for the diagnosis of hypo- and hyperglycemia.¹ However, there are three significant drawbacks to using these methods: (1) they were developed to evaluate diagnostic tests, and the technical requirements for diagnosing and monitoring glucose fluctuations are significantly different; (2) the concepts of sensitivity and specificity are not intuitively understood by CGM users (patients with diabetes and diabetes health care professionals); and (3) sensitivity and specificity do not relate to the timeliness of the diagnosis, and timeliness is essential for CGM alarms.

The Clinical and Laboratory Standards Institute (CLSI) produced a guideline for evaluating CGM alarms² based on an alternative analysis method developed by the Diabetes Research in Children Network Study Group.³ In-clinic studies with reference readings at an interval of 15 minutes or less are required for the analysis. Under this method, hypo- and hyperglycemia are defined by frequent, successive reference readings beyond hypo- or hyperglycemic thresholds. A CGM alarm must sound within ±30 minutes of the beginning of a glycemic event; otherwise the event is considered undetected. A detection time of ±15 minutes is considered optimum. Alarms not associated with actual hypo- or hyperglycemia are false alarms. By reporting the rate of true and false alarms, deficiencies in the diagnostic method were addressed.

This report presents an alarm characterization that builds on the CLSI guideline. The analysis considers devices approved as adjuncts to self-monitoring of blood glucose (SMBG), which includes all CGM devices currently approved for use in the United States. The CLSI guideline does not consider adjunctive devices per se, but the adjunctive use of these devices is essential for meaningful alarm characterization. CGM is superior to SMBG for the detection of glycemic events, while the strength of SMBG is providing the instantaneous glucose value required for treatment decisions. Adjunctive CGM necessarily includes both CGM and SMBG. When the strengths of both are optimized, the true clinical utility of CGM alarms can be described.

Another weakness in the CLSI alarm evaluation method is its failure to recognize the acceptable margins of error for glucose monitors.⁴ The CLSI method considers glucose alarms that are not perfectly correct as being completely wrong. This is not necessarily valid in a clinical setting. In the proposed method, the clinical utility of CGM alarms, from the patients' perspective, will be the fundamental measure of alarm performance.

Materials and Methods

Clinical Study

The clinical study has been described previously and conforms to the CLSI guideline for in-clinic studies with frequent reference readings.^4–6 Fifty-eight subjects with type 1 diabetes were enrolled at three sites comparing the FreeStyle Navigator^® continuous glucose monitoring system (FreeStyle Navigator CGM; Abbott Diabetes Care, Alameda, CA) to venous glucose measurements with the Yellow Springs Instrument (YSI) 2300 STAT Plus glucose analyzer (YSI Life Sciences, Yellow Springs, OH).

In the clinical study, two sensors were inserted into each subject, one on the back of the arm and one on the abdomen. Venous measurements were taken at 15-minute intervals for a total of 50 hours for each subject. The subjects' clinic time was scheduled to incorporate the entire 5-day lifetime of the sensors. Insulin and glucose challenges were administered to ensure hypoand hyperglycemic conditions (Registration Number NTC00920881 on Clinicaltrials.gov). Raw CGM data were postprocessed using the FreeStyle Navigator TRU-Start^™ calibration algorithm, which calls for calibration with SMBG 1, 2, 10, 24, and 72 hours after insertion of a 5-day sensor. A limitation of the protocol is the use of venous blood as a reference for a device calibrated with capillary blood. Capillary blood glucose (BG) is higher than venous BG by 2–5 mg/dl at fasting, and the positive bias increases after meals⁷; this induced an average positive CGM bias of 6 mg/dl versus the reference.

Analytical Methodology

Continuous glucose monitoring readings were paired with YSI values taken within the same minute, and YSI whole blood measurements were multiplied by 1.12 to obtain plasma equivalent values.⁷ The start of a hypoglycemic event was defined by the first of multiple successive reference readings below the hypoglycemic threshold, and the end of the event was defined by two successive reference readings above the threshold. An exception was made for brief events involving a single reference point. To eliminate events caused by random error in the reference test, the single point was required to be >2 standard deviations beyond the threshold (e.g., using a standard deviation of 3 mg/dl for YSI measurements <100 mg/dl, a single reference point for a 70-mg/dl hypoglycemic event required an YSI measurement <64 mg/dl). Alarms originated with the first CGM point below the hypoglycemic threshold and ended with two successive CGM values above the threshold. Hyperglycemic events and alarms were defined similarly, except that the YSI error for a single-point hyperglycemic event was a 3% coefficient of variation for glucose >100 mg/dl.

Low Alarm Characterization

Although there is not universal agreement on the glucose level that defines hypoglycemia, 70 mg/dl is generally accepted as mild hypoglycemia that should be treated⁸; however, as glucose control improves and the frequency of severe hypoglycemia is reduced by improved monitoring, a more aggressive level of 60 mg/dl could be considered. Both levels were used as hypoglycemic thresholds in this analysis.

Limitations of the CLSI evaluation method can be understood by examining cases of CGM alarm “failure” in detail. Figure 1 shows YSI and CGM in good agreement, but this would be considered a failure to detect BG of 70 mg/dl by the CLSI method because the CGM was 7 mg/dl above the threshold. In Figure 2, there is also good agreement between YSI and CGM values, but this example would be labeled a false alarm for 70 mg/dl by the CLSI standard because the YSI value was 77 mg/dl when the CGM alarmed. In both cases the CGM information may have had clinical relevance to the patient. The CLSI analysis was based on detection of 70 mg/dl with an alarm threshold setting of 70 mg/dl, which allows no margin for error.

Figure 1.

Failure to detect a hypoglycemic event by CLSI evaluation.

Figure 2.

Hypoglycemic false alarm by CLSI evaluation.

To improve CGM alarm analysis, the purpose of glucose monitoring must be examined from the patients' perspective. Patients' goals are to detect or avoid hypoglycemia using available CGM and SMBG tools. CGM alarms can be optimized to give maximum detection by setting the alarm threshold higher than the hypoglycemia threshold. When the alarm threshold is higher, the number of false alarms will increase, but false alarms should not lead to inappropriate treatment because all alarms should be confirmed by SMBG.

When the CGM alarm threshold is raised to increase detection, additional difficulties with the CLSI method become apparent. For example, if the alarm threshold was increased to 85 mg/dl in Figure 1, there would still be a failure to detect the 70-mg/dl event because the alarm would have sounded before the ±15- or ±30-minute time limits. From the patients' perspective, however, this alarm is not a failure. Because the alarm threshold was >70 mg/dl, early indication would be expected. Confirmatory SMBG would indicate that CGM was reporting glucose accurately, and the patient could make the correct therapeutic decision based on the SMBG result and the available CGM trend information. In general, any alarm indicating low, descending glucose in advance of a true hypoglycemic condition is useful to patients. Treating the descending glucose when it is still in the normal range will avoid hypoglycemia.

Late alarms are another matter. If an alarm occurs after a patient has been exposed to the potential dangers of hypoglycemia, it has not fully performed its function. Although placing a time limit on early detection is not necessary or useful, a time limit for alarm activation after the start of an event is essential.

Allowing early detection of hypoglycemia leads to a quandary: an alarm that detects an event can also be a false alarm, as depicted in Figure 3. In this example, the CGM device was reporting glucose accurately, and an 85-mg/dl alarm setting was used as a warning for impending hypoglycemia. However, when the alarm sounded, the BG was >85 mg/dl, and therefore the alarm was false based on the threshold of 85 mg/dl. However, using the CGM trend information, this alarm was valid because the glucose was decreasing and the patient became hypoglycemic within 30 minutes. This alarm was therefore clinically useful to the patient.

Figure 3.

Hypoglycemic event detected with a false alarm.

Late alarms also fall into a gray area. Late alarms are not credited for detecting an event, but they are also not false alarms because they occur when BG is low. In the formal diagnostic methodology, the categories are black and white: true positive and true negative or false positive and false negative. With CGM, there are gray areas, and an alarm can be useful without being perfectly correct.

False alarms also need to be examined from the patients' perspective. By formal definition, a false alarm for detection of 70 mg/dl occurs at BG levels >70 mg/dl. When the alarm threshold is set >70 mg/dl to improve detection, the formal definition of false alarms loses its meaning. If CGM is used as directed, CGM alarms are not actually false because SMBG, not CGM, is used for treatment, but CGM alarms that occur when treatment is not necessary are inconvenient. An excessive number of alarms of this type will require an excessive number of SMBG tests to confirm them and can lead to alarm fatigue, where alarms are ignored or turned off. In the recommended characterization, the term “alarm when treatment not necessary” was substituted for “false alarm.” The negative connotation associated with this description will be readily understood by users.

Hypoglycemia treatment is required when glucose is <70 mg/dl; however, when glucose is descending, as indicated by CGM activating a low alarm, it might be prudent to treat glucose in the normal range to avoid hypoglycemia. Because there is also a margin for error in the SMBG reading, in the interest of patient safety, administration of carbohydrates to raise BG to a safer level should be considered when BG is ≤85 mg/dl and descending after a low glucose alarm. For patients in good control using the more aggressive hypoglycemic borderline of 60 mg/dl, administering carbohydrates at 85 mg/dl is untenable, and the treatment level is reduced to ≤70 mg/dl with glucose descending.

In the alarm characterization, a range of alarm settings was evaluated to provide users an informed choice. The range was specifically chosen to highlight the clinical utility of the FreeStyle Navigator CGM and to provide users with information to make appropriate choices for alarm settings based on product performance.

Ideally, alarm characterization by CGM manufacturers should provide sufficient information for users to set an appropriate alarm level and to judge the performance of the device at that setting. At each alarm setting, the following information should be reported:

• Time limit for detection: 15 or 30 minutes

• Detection rate of 70-mg/dl events

• CGM alarm <85 mg/dl

• Relative frequency of alarms

• Alarm rate when treatment is not necessary (BG >85 mg/dl or BG 70–85 mg/dl and rising)

• The sequence is repeated for detection of 60 mg/dl and treatment at 70 mg/dl.

Table 1 presents CGM low alarm characterization by the proposed method.

Table 1.

CGM Low Alarms Characterized by Proposed Method

70-mg/dl hypoglycemic border
CGM alarm setting (mg/dl)	70	80	90	100
Detection rate of 70 mg/dl within 15 minutes	54%	76%	90%	95%
Detection rate of 70 mg/dl within 30 minutes	64%	86%	93%	97%
Number of alarms	176	256	375	449
Alarm rate when treatment not necessary (blood glucose >85 mg/dl)a	18%	21%	38%	52%
60-mg/dl hypoglycemic border
CGM alarm setting (mg/dl)	60	70	80	90
Detection rate of 60 mg/dl within 15 minutes	47%	66%	80%	90%
Detection rate of 60 mg/dl within 30 minutes	54%	75%	85%	94%
Number of alarms	122	176	256	375
Alarm rate when treatment not necessary (blood glucose >70 mg/dl)b	33%	36%	49%	69%

^aWhen SMBG after a low alarm is ≤85 mg/dl and descending, administration of carbohydrates should be considered to raise glucose to a safer level. When SMBG after a low alarm is >85 mg/dl, it is not necessary to raise the glucose level immediately.

^bA patient in good glycemic control using the more aggressive 60-mg/dl hypoglycemic border would administer carbohydrates when SMBG is ≤70 mg/dl and descending.

If the performance of the CGM is different under certain circumstances, e.g., first day of insertion or nighttime, the alarm performance during these times should be reported separately.

High Alarm Characterization

Many of the same considerations for characterizing low alarms also apply to high alarms, but there are some important differences. Because there is no universal threshold for the treatment of hyperglycemia, a range of treatment levels must be evaluated. Detection with high alarms is entirely analogous to detection with low alarms. The definition of “treatment not necessary,” however, is fundamentally different. Because insulin treatment is the motivation for setting high alarms, treatment and detection levels are the same. Table 2 presents CGM high alarm characterization by the proposed method.

Table 2.

CGM High Alarms Characterized by Proposed Method

Treatment level 140 mg/dl
CGM alarm setting (mg/dl)	140	135	130	125
Detection rate of 140 mg/dl within 15 minutes	79%	87%	91%	95%
Detection rate of 140 mg/dl within 30 minutes	90%	95%	97%	98%
Number of alarms	534	545	544	528
Alarm rate when treatment not necessary (blood glucose <140 mg/dl or descending)	17%	24%	34%	42%
Treatment level 180 mg/dl
CGM alarm setting (mg/dl)	180	175	170	165
Detection rate of 180 mg/dl within 15 minutes	75%	79%	85%	90%
Detection rate of 180 mg/dl within 30 minutes	87%	90%	94%	96%
Number of alarms	561	559	582	574
Alarm rate when treatment not necessary (blood glucose <180 mg/dl or descending)	24%	28%	35%	42%
Treatment level 240 mg/dl
CGM alarm setting (mg/dl)	240	230	220	210
Detection rate of 240 mg/dl within 15 minutes	70%	84%	92%	96%
Detection rate of 240 mg/dl within 30 minutes	86%	92%	97%	98%
Number of alarms	410	436	473	499
Alarm rate when treatment not necessary (blood glucose <240 mg/dl or descending)	30%	39%	52%	64%
Treatment level 300 mg/dl
CGM alarm setting (mg/dl)	300	290	280	270
Detection rate of 300 mg/dl within 15 minutes	73%	84%	93%	94%
Detection rate of 300 mg/dl within 30 minutes	88%	93%	95%	97%
Number of alarms	215	230	266	342
Alarm rate when treatment not necessary (blood glucose <300 mg/dl or descending)	35%	46%	61%	72%

Results

Low Alarms

The proposed method (Table 1) provides comprehensive information for detecting hypoglycemia with low alarms. Although the setting of 70 mg/dl did not detect 70 mg/dl BG reliably, the detection rate improved at higher alarm settings, from 54 to 95% for 15-minute and from 64 to 97% for 30-minute detection. The downside of the higher settings is readily apparent; as the setting was increased from 70 to 100 mg/dl, the number of alarms increased from 176 to 449, and the percentage of “alarms when treatment not necessary” increased from 18 to 52%. A setting of 90 mg/dl may be optimal for many users; the detection of 70 mg/dl was high (90 and 93% for 15- and 30-minute detection, respectively), and a minority of alarms (38%) occurred when treatment was not necessary. At the 60-mg/dl hypoglycemic border, detection rates were lower and unnecessary alarms were higher.

The information provided is useful for special situations. There are times when detection could be sacrificed to minimize unnecessary alarms. The highest setting might be best for patients with severe hypoglycemia unawareness. In this case, detection as high as 95% might be a worthwhile trade-off for the additional SMBG tests required to confirm the high number of unnecessary alarms.

High Alarms

Similar to low alarms, the proposed method (Table 2) provides information for making the trade-off between detection and unnecessary alarms. For example, for a BG level of 140 mg/dl, as the alarm setting decreased from 140 to 125 mg/dl, the detection rate increased from 79 to 95% for 15-minute and from 90 to 98% for 30-minute detection. The percentage of alarms that occurred when treatment was not necessary increased from 17 to 42%. A setting of 135 mg/dl might be considered optimal because the detection was high (87 and 94% for 15- and 30-minute detection, respectively), and the number of alarms that did not require treatment was relatively low (24%). Similar judgments can be made at each treatment threshold.

Discussion

The CLSI low alarm analysis of the data set provided the following information for the detection of hypoglycemia (glucose <70 mg/dl) with an alarm setting of 70 mg/dl:

• Detection rate in ±15 minutes: 52%

• Detection rate in ±30 minutes: 63%

• False alarm rate: 35%

This analysis provides no other useful information for hypoglycemia detection with alarms. The CLSI guideline does not consider the implication of CGM devices as adjunctive tools. The method proposed in this article is appropriate for currently available devices and provides information for selecting an alarm level and meaningful clinical performance parameters associated with the selection.

The essence of the proposed method is a redefinition of detection and false alarms that recognizes the claims and intended use of the CGM device. With regard to detection, the CGM is not a “hypoglycemia detector”; it is a monitoring device that either detects hypoglycemia or provides information to avoid hypoglycemia. For the purpose of monitoring, the ability to avoid is no less valuable than the ability to detect. False alarms have been redefined to relate strictly to treatment. An “alarm when treatment not necessary” replaces “false alarm,” and the treatment level is not necessarily the alarm setting.

One drawback to the analysis relates to the reporting of alarm frequency. The values for “number of alarms” are somewhat arbitrary; they depend on the size of the study and the glucose control of the subjects. In product labeling, it would be more useful to replace these numbers with a verbal description. For example, “At lower alarm settings, the frequency of high alarms will increase.”

Continuous glucose monitoring values tend to lag SMBG glucose due to device lag and the physiological lag of interstitial glucose to BG. On average, CGM alarms will appear to occur “late,” but this is counteracted by a reduction in the alarms that appear unnecessarily “early.” An informed user can choose alarm thresholds that account for lag. The user should also be taught that apparent alarm errors due to lag, i.e., when CGM alarms do not match the SMBG confirmation, will tend to be greatest when the rate of change is highest.

Although the suggested alarm characterization appears to provide sufficient information to optimize alarm settings, there are limitations to the methodology. The data set is from a selection of patients at different levels of glucose control, but the experience of an individual patient can vary significantly with glucose control. For example, a high alarm setting of 140 mg/dl could be useful for a patient in excellent glycemic control, but if glycemic control is not good, the alarms would sound for hours when glucose is high, making them useless and annoying. Device imprecision will also affect users as the alarm performance will vary somewhat from one sensor to the next. The alarm characterization provides a starting point for selecting alarm settings, but true optimization must be performed empirically by individual patients.

The proposed alarm characterization was designed to provide the most useful information to users. More details of the analysis, including actual numbers of events and alarms, as well as confidence intervals, should be reported for regulatory review. Additional analyses, such as diagnostic sensitivity and specificity, could also be appropriate for reviewers with the expertise to interpret the results. The key is to provide a relevant analysis for the intended audience.

Conclusion

The method for evaluating CGM alarm performance in the CLSI guideline uses formal definitions of true alarm and false alarm that do not incorporate the acceptable error limits for a glucose monitoring device. The method also does not optimally describe current CGM devices, which are approved as adjuncts to SMBG. As a result, the analysis does not reflect the current intended use of CGM and is not particularly useful to users. The proposed method reflects the claims and intended use of current CGM devices. It provides useful information for choosing appropriate alarm levels and reflects the true clinical utility of the CGM alarms.

Abbreviations

CGM

continuous glucose monitoring

CLSI

Clinical and Laboratory Standards Institute

SMBG

self-monitoring of blood glucose

YSI

Yellow Springs Instrument