Evaluation of Factors Affecting CGMS Calibration

Abstract

Background

The optimal number/timing of calibrations entered into the Continuous Glucose Monitoring System (“CGMS”; Medtronic MiniMed, Northridge, CA) have not been previously described.

Methods

Fifty subjects with T1DM (10–18y) were hospitalized in a clinical research center for ~24h on two separate days. CGMS and OneTouch^® Ultra^® Meter (“Ultra”; LifeScan, Milpitas, CA) data were obtained. The CGMS was retrospectively recalibrated using the Ultra data varying the number and timing of calibrations. Resulting CGMS values were compared against laboratory reference values.

Results

There was a modest improvement in accuracy with increasing number of calibrations. The median relative absolute deviation (RAD) was 14%, 15%, 13% and 13% when using 3, 4, 5 and 7 calibration values, respectively (p<0.001). Corresponding percentages of CGMS-reference pairs meeting the ISO criteria were 66%, 67%, 71% and 72% (p<0.001). Nighttime accuracy improved when daytime calibrations (pre-lunch and pre-dinner) were removed leaving only two calibrations at 9p.m. and 6a.m. (median difference: −2 vs. −9mg/dL, p<0.001; median RAD: 12% vs. 15%, p=0.001). Accuracy was better on visits where the average absolute rate of glucose change at the times of calibration was lower. On visits with average absolute rates <0.5, 0.5-<1.0, 1.0-<1.5 and ≥1.5mg/dL/min, median RAD values were 13% vs. 14% vs. 17% vs. 19%, respectively (p=0.05).

Conclusions

Although accuracy is slightly improved with more calibrations, the timing of the calibrations appears more important. Modifying the algorithm to put less weight on daytime calibrations for nighttime values and calibrating during times of relative glucose stability may have greater impact on accuracy.

Introduction

The Medtronic Minimed continuous glucose monitoring system (“CGMS”; Medtronic MiniMed, Northridge, CA) uses a retrospective calibration based upon 3–4 glucose meter test results entered by the patient each day, but the current recommendations for calibrating the CGMS sensor provide no guidelines for determining when calibration values should be obtained. Moreover, there has been little data published on factors which affect the accuracy of glucose sensor calibration, and many questions remain. Are 3 or 4 calibration values adequate, or will additional calibration values significantly improve sensor accuracy? Is it better to obtain calibrations when glucose values are more stable, i.e. preprandial, or should they be obtained pre and postprandial to provide calibration points over a broader glucose range? There have been concerns over the accuracy of continuous glucose sensors especially overnight¹, but there are no published data on how the use of daytime calibrations affect nighttime readings.

In order to address these issues, we systematically reviewed the effect of using 3, 4, 5, or 7 calibration values on the accuracy of eighty-eight 24-hour CGMS runs. We were particularly interested in whether postprandial glucose values improved or decreased sensor accuracy, and whether greater accuracy was observed overnight when calibration values were based on nighttime, instead of daytime readings.

Methods

The full protocol is described elsewhere² and is briefly summarized here. Fifty subjects with type 1 diabetes (average age was 14.8 ± 1.7 years; 44% female; 90% Caucasian) were hospitalized in a clinical research center (CRC) for approximately 24 hours on two different days. During one of these visits, the subject exercised on a treadmill for 75 minutes in the late afternoon.

The subject was admitted to the CRC in the late morning where a CGMS sensor was inserted in either the buttocks (N=57), abdomen (N=34), hip (N=7) or thigh (N=2). Glucose measurements obtained from the One Touch^® Ultra^® Meter (“Ultra”; Lifescan, Milpitas, CA) were used to calibrate the CGMS. The subject wore the CGMS until discharge from the CRC early the following morning. Both CGMS and Ultra data were downloaded for analysis.

Blood samples for central laboratory determination of serum glucose concentrations were obtained from an intravenous catheter during the exercise session and hourly overnight from 10 p.m. to 6 a.m. Glucose determinations were made at the DirecNet Central Laboratory at the University of Minnesota using a hexokinase enzymatic method, which has been proposed as the reference method for measuring glucose.^{3, 4} Glucose measurements were also made with the Ultra meter at the same times listed above and additionally before lunch and dinner and on the hour from 2–4 p.m and 7–9 p.m.

Statistical Analysis

The CGMS system uses a retrospective calibration algorithm that incorporates both future and previous data points in the calculation of each glucose value (i.e., does not give “real-time” values). The algorithm employs a moving average smoothed regression of the calibration values. Background sensor current is handled by choosing one from a range of allowable offsets which best fit the data. Although four calibration values from a home glucose meter are recommended per day, the calibration will be performed with fewer meter readings as long as there is at least one calibration point within 12 hours of the sensor reading.

Medtronic MiniMed provided a computer program to implement this algorithm based on the raw signal measured by the sensor for any given set of calibration values. This program was used to recalibrate each CGMS 4 different ways standardizing the number and timing of the calibrations taken from the Ultra meter (Table 1).

Table 1.

Timing of Calibrations

Time of Day	3	4	5	7
	Calibrations	Calibrations	Calibrations	Calibrations
Pre-lunch	X	X	X	X
3 p.m.				X
Pre-dinner		X	X	X
9 p.m.	X	X	X	X
Midnight			X	X
3 a.m.				X
6 a.m.	X	X	X	X

The resulting CGMS values were paired with laboratory reference glucose measurements made within ±2.5 minutes after accounting for a 2.5-minute processing lag. The following difference measurements were calculated for each CGMS-reference pair:

• Difference: CGMS minus reference glucose value.

• Relative absolute difference (RAD): absolute value of the difference divided by the reference value (expressed as a percentage).

• ISO criteria (binary assessment): CGMS within ±15mg/dL when reference ≤75mg/dL or within ±20% when reference >75mg/dL.

The bootstrap resampling technique⁵ was used to account for correlated data from the same subject to test whether accuracy was associated with the number of calibrations entered (treated as a continuous variable).

Analyses of sensitivity and false positives for hypoglycemia were based on events rather than discrete points. A CGMS episode of hypoglycemia below 70 mg/dL was defined as at least 2 readings ≤70mg/dL with no intermediate values >80 mg/dL. Distinct episodes were required to be separated by at least 30 minutes. An episode was considered confirmed if there was at least one reference value ≤70 mg/dL during the episode. Otherwise, the episode was considered a false positive if there was a reference value >10 mg/dL higher than the concurrent CGMS value. If neither of these conditions were met, the episode was considered unevaluable. Sensitivity was defined as the percentage of all laboratory and Ultra values ≤70 mg/dL for which there was a CGMS episode ≤70 mg/dL (or ≤ reference value + 10 mg/dL if the reference was between 61–70 mg/dL) within ±30 minutes. Multiple laboratory/Ultra values ≤70 mg/dL within 30 minutes were considered a single event and only counted once in the evaluation of sensitivity.

Accuracy was also evaluated by the average absolute rate of glucose change at the time of calibration. Rate of change was calculated using CGMS values 10 minutes apart (5 minutes prior to and following the time of each calibration). The absolute value of this rate of change was averaged over 4 calibration times (pre-lunch, pre-dinner, 9 p.m., 6 a.m.) for each visit. Accuracy measures were adjusted for the rate of change at the time of the reference value by stratifying into 4 groups (<0.5, 0.5-<1.0, 1.0-<1.5 and ≤1.5 mg/dL per minute) and weighting each group proportional to its total number of CGMS-reference pairs in the overall dataset.

Analyses were limited to sensors functioning for at least 15 hours and visits where both Ultra and CGMS measurements were available at all 7 calibration time points listed in Table 1. This resulted in 12 of the 100 visits being excluded from analysis (5 switched the sensor in the middle of the visit, 4 inserted the sensor in the afternoon missing the pre-lunch calibration, 2 had <15 hours of CGMS data due to frequent skipping and 1 had the CGMS reject the 6 a.m. calibration). The remaining 88 visits used for analysis included all 50 subjects (i.e., no subject had both of their visits excluded).

Results

CGMS Use in the CRC

The CGMS was used for a median (25^th, 75^th percentiles) 20 hours per visit (20, 21) ranging from 16 to 24. There were 153 CGMS-reference pairs during exercise and 944 pairs at other times. The median (25^th, 75^th percentiles) number of pairs per visit was 13 (9, 16) ranging from 7 to 19. The median number of calibrations originally entered into the CGMS during the CRC visit was 5 (4, 5) ranging from 2 to 10 (three or fewer for 12 visits, four for 27 visits, five for 33 visits and six or more for 16 visits). Based on these original calibrations, the median RAD excluding exercise was 15% overall and 18% when reference values were ≤70 mg/dL, 64% of pairs met the ISO criteria. Sensor accuracy was unaffected by insertion site (data not shown).

Number of Calibration Values

There was a modest, but statistically significant improvement in accuracy according to the number of calibrations retrospectively re-entered into the CGMS. The median RAD excluding the exercise session was 14%, 15%, 13% and 13% when using 3, 4, 5 and 7 calibration values, respectively (p<0.001). Corresponding percentages of pairs meeting the ISO criteria were 66%, 67%, 71% and 72%, respectively (p<0.001). This trend appeared more pronounced during hypoglycemia (Table 2). Increasing the number of calibrations did not appear to correct the tendency for the CGMS to read low overnight (10 p.m. – 6 a.m.) with median differences of −5, −9, −8 and −8 mg/dL, respectively, but there was a suggestive trend towards lower overnight false positive rates (48% vs. 37% vs. 37% vs. 35%, respectively; p=0.08).

Table 2.

CGMS Accuracy Statistics Assessed from Laboratory Reference Values by Number of Calibrations Retrospectively Entered

		Differenceamedian (mg/dL)				Relative Absolute Differenceb(RAD)median				ISO Criteriacpercentage
		Number of Calibrations				Number of Calibrations				Number of Calibrations
	N	3d	4e	5f	7g	3d	4e	5f	7g	3d	4e	5f	7g
During Exercise	153	+6	+5	+7	+11	19%	15%	16%	15%	56%	63%	63%	65%
Excluding Exercise	944	−4	−7	−5	−5	14%	15%	13%	13%	66%	67%	71%	72%
Reference Glucoseh
≤70	139	+11	+8	+9	+9	24%	19%	19%	17%	56%	65%	63%	71%
71–120	346	−2	−5	−3	−4	14%	15%	12%	13%	64%	65%	71%	72%
121–180	217	−11	−17	−16	−16	12%	14%	13%	12%	73%	70%	74%	74%
181–240	140	−14	−19	−17	−19	13%	14%	13%	13%	65%	67%	67%	71%
>240	102	−15	−17	−12	−9	10%	12%	10%	10%	76%	69%	75%	77%
Time of Dayh
Daytime (9 a.m.–10 p.m.)	139	+2	+4	+8	+11	21%	17%	17%	17%	54%	65%	63%	63%
Overnight (10 p.m.–6 a.m.)	805	−5	−9	−8	−8	13%	15%	13%	12%	69%	67%	72%	74%
10 p.m.–1 a.m.	276	−1	−5	−2	−4	15%	15%	13%	12%	64%	66%	74%	72%
2 a.m.– 4 a.m.	266	−3	−8	−7	−8	14%	15%	13%	12%	67%	65%	69%	72%
5 a.m – 6 a.m	263	−10	−14	−14	−11	12%	14%	14%	12%	75%	70%	73%	78%
Rate of Changehi
<0.5 mg/dL/min	190j	−7	−9	−7	−9	13%	13%	11%	13%	66%	69%	73%	74%
0.5 – <1.0 mg/dL/min	433j	−5	−8	−4	−4	12%	14%	13%	11%	71%	71%	69%	74%
1.0 – <1.5 mg/dL/min	198j	−4	−6	−3	−3	15%	17%	14%	13%	64%	60%	72%	68%
≥1.5 mg/dL/min	123j	+6	+13	0	+12	18%	19%	17%	14%	58%	58%	62%	70%

^a– Defined as the CGMS value minus laboratory reference value.

^b– Defined as absolute value of the difference divided by the reference (expressed as a percentage).

^c– ISO criteria defined as CGMS within ±15mg/dL of reference when reference ≤75mg/dL or within ±20% when reference >75mg/dL.

^d– 3 values used for calibrations: Ultra measurements from pre-lunch, 9 p.m. and 6 a.m.

^e– 4 values used for calibrations: Above 3 plus pre-dinner.

^f– 5 values used for calibrations: Above 4 plus midnight.

^g– 7 values used for calibrations: Above 5 plus 3 p.m. and 3 a.m.

^h– Excludes exercise session.

ⁱ– Absolute rate of change at each calibration entry averaged over visit.

^j– Because the rate of change is calculated by the CGMS, the number of pairs in each group varies slightly by the number of calibrations. Numbers given for the 4-calibration scenario.

Similar trends were observed during exercise (median RAD: 19% vs. 15% vs. 16% vs. 15% for 3, 4, 5 and 7 calibrations, respectively, p=0.22; ISO: 56% vs. 63% vs. 63% vs. 65%; p=0.24), but these did not achieve statistical significance. The CGMS correctly detected 17/26 (65%) cases of exercise-induced hypoglycemia (reference glucose ≤70mg/dL) when 3 calibrations were used, 18/26 (69%) when 4 calibrations were used and 15/26 (58%) when 5 or 7 calibrations were used. False positive rates were 13% (3/23), 0% (0/22), 5% (1/19) and 7% (1/15), respectively.

Timing of Calibrations

Additional re-calibration schemes were run to explore how the timing of the calibrations might affect accuracy. The 4-calibration scheme (pre-lunch, pre-dinner, 9 p.m., 6 a.m.) was used as a baseline for comparison. Removing the two daytime (pre-lunch and pre-dinner) calibrations actually improved nighttime accuracy (median difference: −2 vs. −9 mg/dL, p<0.001; median RAD: 12% vs. 15%, p=0.001; ISO: 73% vs. 67%, p=0.004; but the false positive rate for hypoglycemia was not significantly different: 44% vs. 37%, p=0.19).

Changing the 3 daytime calibration values from pre-prandial to post-prandial (i.e., calibrating at 2 p.m., 7 p.m., 10 p.m. and 6 a.m.) did not affect accuracy. The median difference (daytime and nighttime combined excluding the exercise session) was −5 vs. −7 mg/dL, p=0.31; median RAD: 14% vs. 15%, p=0.50; ISO percentage: 68% vs. 67%, p=0.62; false positive rate for hypoglycemia: 38% vs. 38%, p=0.91.

Rate of Change during Calibration

The CGMS was significantly more accurate on visits where the average absolute rate of change at the time of calibration (measured by the CGMS and averaged over the 4 calibration times: pre-lunch, pre-dinner, 9 p.m. and 6 a.m.) was lower. Adjusting for the rate of change at the time of the reference value (i.e., when accuracy was being evaluated), the median difference excluding exercise was −9 vs. −8 vs. −6 vs. +13mg/dL on visits with average absolute rates of change at the time of calibration <0.5, 0.5-<1.0, 1.0-<1.5 and ≥1.5 mg/dL per minute, respectively (p=0.001). Corresponding values for the median RAD were 13% vs. 14% vs. 17% vs. 19% (p=0.05); ISO: 69% vs. 71% vs. 60% vs. 58% (p=0.04); and false positive rate for hypoglycemia: 18% vs. 38% vs. 49% vs. 39% (p=0.19). Average absolute rate of change at the time of calibration was not confounded with mean glucose level (data not shown).

Discussion

A previous inpatient investigation from our group in 90 children with T1DM demonstrated that the median RAD of the original CGMS sensor was 19%⁶. The percent of values meeting ISO criteria for the original sensor was only 53% overall and 41% for reference values≤70 mg/dL. The new CGMS Gold system was introduced in 2002, and our results confirm that the accuracy of this system is significantly improved when compared to the original version, particularly for values at or below 70 mg/dL^{6, 7}.

The primary aim of this study was to determine whether different calibration schemes could be utilized to improve further CGMS accuracy. Our results showed a slight improvement in sensor accuracy when additional calibration values are added with the median RAD improving from 14–15% with 3 or 4 calibrations each day, to 13% with 5 or 7 calibrations each day. Additional calibration values also tended to improve sensor performance during exercise in our subjects. In calibrating an implanted glucose sensor, Choleau, et.al⁸ assessed the benefit of using 1, 2, or 3 discrete one-point calibrations retrospectively each day. In their studies, increasing the number of calibration points from 1 to 2 to 3 also caused a slight improvement in the percent of values observed in Clark Error Grid zones A+B (84.8%, 88.1%, and 89.4%, respectively).

If the output from the subcutaneous sensor was always linearly related to the blood glucose and there was no change in the sensitivity of the sensor to glucose, a single calibration point would be sufficient. This assumes there is a direct relationship of the interstitial glucose to the blood glucose which does not vary with changing glucose concentrations. In actuality the interstitial glucose concentrations lag changes in the serum glucose by 4 to 10 minutes ^{8, 9}, and therefore when glucose levels are changing rapidly this physiologic lag in interstitial glucose levels could result in a less accurate calibration. This relationship (interstitial to blood glucose) becomes more divergent at lower blood glucose levels ¹⁰. In contrast, when glucose levels are stable and there is equilibration between the interstitial and serum glucose levels, this error should be reduced. Our results confirm that sensor accuracy is decreased when blood glucose levels are changing rapidly. The median RAD using 4 calibration points was 13% when the rate of change was <0.5 mg/dL/min, but when the rate of change increased to ≥1.5 mg/dL/min the median RAD increased to 19%, and values meeting ISO criteria decreased from 69% to 58%.

The CGMS uses a retrospective calibration algorithm, but these results also apply to sensors that give glucose values in “real-time”. Our data support the wisdom of not allowing a calibration value to be entered into a real-time sensor if the glucose is changing by more than 2 mg/dL/min; a paradigm currently being employed in the Freestyle Navigator™ Continuous Glucose Monitor (Abbott Diabetes Care, Alameda, CA)¹¹.

Accuracy was not affected by using calibration values obtained either pre-prandially or about 2 hours post-prandially. Therefore blood tests to calibrate a sensor can be done when it is convenient for the patient, either pre-prandially or post-prandially, as long as the glucose values are obtained when there is not a rapid rate of change in glucose levels.

In past studies we have observed a consistent bias for the CGMS to read lower at night, and this was also observed by McGowan et. al.¹. A similar bias towards lower CGMS readings overnight was also observed in the present study. Using 4 calibration values the median bias during the day was +4 mg/dL, and overnight the bias was −9 mg/dL. This was not the result of more calibration values being obtained during the day than at night, because when we increase the number of nighttime calibration values, (including midnight and 3 a.m. calibration values so that 4 values were obtained bracketing and including the night) the daytime bias remained positive at +11 mg/dL and the nighttime bias remained negative at −8 mg/dL (Table 2). When we limited calibration values to those only obtained overnight, this bias was resolved (−2 mg/dL), but when we added back daytime calibration values the negative nighttime bias (−9 mg/dL) is again present. This would suggest the sensor output is higher during the day than overnight, the etiology for this bias was not assessed in this study. There could be a physiologic difference in the subcutaneous blood glucose, oxygen or blood flow overnight resulting in lower subcutaneous glucose measurements, or it is possible that decreased activity at night allows for nocturnal sensor “biofouling”. A separate calibration algorithm for overnight readings might therefore improve accuracy.

In summary, there is little improvement in sensor accuracy by including more than 4 calibrations each day, and only a slight improvement when increasing from 3 to 4 calibrations each day. Sensor accuracy was not significantly affected by obtaining glucose values preprandially or about 2 hours postprandially. Sensor accuracy was most improved by obtaining calibration values when there was not a rapid rate of glucose change. There is a bias for the sensor to read lower at night, and removing daytime calibration values improves overnight sensor accuracy. Modifying the algorithm to put less weight on daytime calibrations for nighttime values and calibrating during times of relative glucose stability may have the greatest impact on optimizing the use of calibration values.

Appendix

Writing Committee

Bruce A. Buckingham, MD; Craig Kollman, PhD; Roy W. Beck, MD, PhD; Andrea Kalajian, MS; Rosanna Fiallo-Scharer, MD; Michael J. Tansey, MD; Larry A. Fox, MD; Darrell M. Wilson, MD; Stuart A. Weinzimer, MD; Katrina J. Ruedy, MSPH; and William V. Tamborlane, MD

The DirecNet Study Group

Clinical Centers

Listed in alphabetical order with clinical center name, city, and state. Personnel are listed as (PI) for Principal Investigator, (I) for co-Investigator and (C) for Coordinators.

1. Barbara Davis Center for Childhood Diabetes, University of Colorado, Denver, CO: H. Peter Chase, MD (PI); Rosanna Fiallo-Scharer, MD (I); Jennifer H. Fisher, ND, RN (C); Barbara Tallant, RN, MA (C)

2. Department of Pediatrics, University of Iowa Carver College of Medicine, Iowa City, IA: Eva Tsalikian, MD (PI); Michael J. Tansey, MD (I); Linda F. Larson, RN (C); Julie Coffey, MSN (C)

3. Nemours Children’s Clinic, Jacksonville, FL: Tim Wysocki, PhD, ABPP (PI); Nelly Mauras, MD (I); Larry A. Fox, MD (I); Keisha Bird, MSN (C); Kelly L. Lofton, RN (C)

4. Division of Pediatric Endocrinology and Diabetes, Stanford University, Stanford, CA: Bruce A. Buckingham, MD (PI); Darrell M. Wilson, MD (I); Jennifer M. Block, RN, CDE (C); Paula Clinton, RD, CDE (C)

5. Department of Pediatrics, Yale University School of Medicine, New Haven, CT: Stuart A. Weinzimer, MD (PI); William V. Tamborlane, MD (I); Elizabeth A. Doyle, MSN (C); Kristin Sikes, MSN (C)

Coordinating Center

Jaeb Center for Health Research, Tampa, FL Roy W. Beck, MD, PhD; Katrina J. Ruedy, MSPH; Craig Kollman, PhD; Dongyuan Xing, MPH; Andrea Kalajian, MS; Cynthia R. Stockdale

University of Minnesota Central Laboratory

Michael W. Steffes, MD, PhD; Jean M. Bucksa, CLS; Maren L. Nowicki, CLS; Carol A. Van Hale, CLS; Vicky Makky, CLS

National Institutes of Health

Gilman D. Grave, MD; Barbara Linder MD, PhD; Karen K. Winer, MD;

Data and Safety Monitoring Board

Dorothy M. Becker, MBBCh; Christopher Cox, PhD; Christopher M. Ryan, PhD; Neil H. White, MD, CDE; Perrin C. White, MD