Lack of accuracy of continuous glucose sensors in healthy, nondiabetic children: Results of the Diabetes Research in Children Network (DirecNet) Accuracy Study

Abstract

Objective

The workup of hypoglycemia requires frequent glucose sampling. We designed these studies to determine if the Continuous Glucose Monitoring System (CGMS) and the GlucoWatch G2^® Biographer (GW2B) are sufficiently accurate to use in nondiabetic children.

Study Design

15 healthy children (9–17 years, 11 boys) wore a GW2B and a CGMS during 24-h, and reference serum glucose was measured hourly during the day and half-hourly overnight.

Results

Compared with the reference glucose, the median absolute difference in concentrations measured by the GW2B (487 pairs) was 13 mg/dL and by the CGMS was 17 mg/dL (668 pairs), with 30% and 42% of values using the GW2B and CGMS, respectively, deviating >20 mg/dL from the reference value. The GW2B reported values <60 mg/dL in 73% of subjects, the CGMS in 60%. In none of these episodes was serum glucose truly low. Spurious high glucose concentrations were also observed with the sensors. The mean reference glucose was lowest at 5AM (89 mg/dL) and highest at 11:30PM (106 mg/dL) during the 24-h.

Conclusions

Neither the CGMS nor GW2B is accurate enough to establish population standards of the glycemic profile of healthy children and cannot be recommended in the workup of hypoglycemia in nondiabetic youth.

Introduction

Continuous glucose sensors have been developed to provide frequent glucose determinations throughout the day and night in individuals with diabetes. In the last few years, an explosion of data have been gathered on the use of continuous glucose sensors, and the application of this technology to the management of patients with diabetes has been the subject of intense investigation^1–17. Two continuous glucose monitors, the GlucoWatch^® G2^™ Biographer (“GW2B”; Cygnus, Inc., Redwood City, CA), and the Continuous Glucose Monitoring System, CGMS™ (“CGMS”; Medtronic Minimed, Northridge, CA) are FDA-approved for use in children with diabetes. Both instruments measure interstitial fluid glucose concentrations, which are normalized for serum values using internal algorithms. Our group, the Diabetes Research in Children Network (DirecNet), has conducted studies assessing the accuracy of these sensors in diabetic children and have found relatively large differences between sensor data and reference serum glucose levels^{18, 19}. The detection of false hypoglycemic values has been particularly common in diabetic children⁷. Although developed for use in diabetes, continuous glucose sensors have also been used in the management of children with hypoglycemia due to glycogen storage diseases¹⁶, and the sensors, if accurate, have a strong theoretical application in the diagnosis of hypoglycemic conditions in childhood or in the glucose monitoring of acutely ill neonates or children in the intensive care unit. To date, however, the accuracy of these devices has not been thoroughly studied and compared against frequently sampled serum reference blood glucose concentrations in nondiabetic youth.

The characterization of glucose values in healthy children during day and nighttime is critical to the interpretation of glucose profiles in pathological states, such as diabetes and hypoglycemic conditions; yet the glucose profile of normal healthy children has not been well described. In nondiabetic children, serum glucose concentrations have been characterized in a fasting state and during oral glucose tolerance tests²⁰, but data are limited on the normal range of serum glucose concentrations over a 24-hour period. Studies in normal children of different ages report average blood glucose concentrations in the 80–110mg/dL range^21–23.

In view of the important potential uses of the continuous glucose sensors in non-diabetics also, determination of their accuracy and limitations is of paramount importance. We hence designed a study in healthy, nondiabetic children to determine if either of the two commercially available continuous glucose sensors (CGMS and/or GW2B) is sufficiently accurate to use as a tool to establish norms for the glycemic profile of nondiabetic children. The study also provided data on the degree of physiological fluctuations in serum glucose concentrations during the day and night in these children.

Methods

This study was conducted by the Diabetes Research in Children Network (DirecNet) at five academic centers in the United States. The protocol and informed consent forms were approved by each center’s institutional review board. Informed consent was obtained from the parents/guardians and, when age appropriate, assent was obtained from the subjects.

Eligibility Criteria and Assessment

Subjects were recruited by each center’s diabetes clinic and the majority of the subjects had a friend or knew someone with diabetes. Eligibility criteria included (1) age 7.0 to less than 18.0 years; (2) no history of diabetes, no history of positive islet cell antibody testing, a normal HbA1c (<6.0% measured with the DCA2000+, Bayer Diagnostics, Tarrytown, NY), and no family history of type 1 or type 2 diabetes in a sibling or parent; (3) body mass index (BMI) between the 10^th to 90^th percentile for age and sex²⁴, (4) no medication use of any type in the prior 7 days, (5) a normal hematocrit (according to normal range at the clinical center’s laboratory) and weight ≥16.0 kg (because of blood volume considerations), and (6) no skin abnormalities contraindicating sensor use.

Study Procedures

Prior to the start of the study, the study staff involved in the placement and management of the sensors underwent a formal training and certification procedure. The subjects were admitted to each center’s clinical research center (CRC) for approximately 26 hours. Upon CRC admission, one GW2B was placed and, after two hours, calibrated by trained study staff. A second GW2B was placed either immediately following the calibration of the first GW2B or if deferred, no more than nine hours later. Additional sensors were placed when indicated so that at least one GW2B would be functioning for the 24 hours of the study. A CGMS sensor was inserted in the abdomen or upper buttocks by study staff. Simultaneous use of a second CGMS sensor was optional. The instruments were calibrated using glucose values from either capillary or venous blood measured with One Touch^® Ultra^® meters (Life Scan, Milpitas, CA). Approximately two-thirds of the Ultra glucose measurements used to calibrate the sensors were from venous blood, which we found to be more accurate than Ultra measurements from capillary blood.²⁵ Since in home sensor use, capillary blood is used for calibration, our results might be slightly overestimating accuracy of the sensors.

Blood samples were obtained through an intravenous catheter hourly during the day (7:00 AM to 9:00 PM) and every half hour overnight (9:30 PM to 6:30 AM) for serum glucose determinations. The first hourly measurement was made at least one hour following insertion of the CGMS. Blood was withdrawn through the catheter, allowed to clot, then separated and frozen until shipped. Analysis was done at the DirecNet Central Biochemistry Laboratory at the University of Minnesota. Glucose levels were measured using a hexokinase enzymatic method^{26, 27}. There was no fixed dietary plan provided and subjects ate the meals selected from the CRC menus ad lib.

Statistical Methods

The underlying principle in the sample size estimation was to determine the number of subjects required for a pre-specified width of a two-sided 95% confidence interval for each measure of accuracy comparing the sensor glucose values with the reference glucose values. With a sample size of 15 subjects and 20 sensor-reference glucose pairs per subject (total of 300 pairs), the half-width of a 95% confidence interval for the mean absolute deviation is approximately 5 mg/dL.

The procedure for matching sensor and reference glucose measurements has been reported previously^{18, 19}. For each matched pair, the absolute difference between the reference glucose value and the sensor glucose value was computed (in mg/dL) and the percentages of sensor values within 10 mg/dL and within 20 mg/dL of the paired reference value were determined.

During the course of the study, Medtronic MiniMed modified the sensor fabrication process that had been in place since 1999. Thirteen subjects used “original” sensors and two subjects used “modified” sensors. Since there were only two subjects who used the modified sensor, formal statistical comparisons were not made between the original and modified sensors.

Variability of reference glucose levels in individual subjects was assessed by calculating the standard deviation (SD) of the reference glucose values around each subject’s 24-h mean.

Results

The 15 subjects who participated in the study had a median age of 12.4 (range 9–17 years); 11 were male and 14 Caucasian. Mean (SD) body mass index was 20.9 (3.6) kg/m². Ten subjects completed the full 24-hour protocol, four subjects completed 12 to 23 hours, and one subject completed less than 12 hours. Three of the five early study discontinuations were related to difficulties maintaining the intravenous line; the other two were at the subject’s request. None of the subjects experienced more than minimal skin irritation using either the GW2B or the CGMS sensors. Figure 1 shows representative profiles of the reference and the sensor glucose values for two subjects over the course of the hospitalization.

Figure 1 A &B.

Representative plots of the sensor and reference glucose values over a 24-hour period. In green is the GWB tracing and in blue the CGMS tracing. Solid points represent the reference laboratory glucose. Two tracings of the same color represent simultaneous use of two given monitors.

GW2B Accuracy

The 15 subjects used 39 sensors during the study. There were 487 GW2B-reference paired glucose values, with 286 of the pairs from overnight sampling (11:00 PM to 6:00 AM) and 201 of the pairs from 6:30 AM to 10:30 PM. The median difference between the GW2B and reference glucose values was −3 mg/dL (25^th, 75^th percentiles: −17, 11 mg/dL). The absolute value of the differences between the GW2B and reference glucose values ranged from 0 to 65 mg/dL with a median of 13 mg/dL (25^th and 75^th percentiles: 6, 23). The GW2B values were within 10 mg/dL of the reference values 40% of the time and within 20 mg/dL 70% of the time. Results were similar for daytime and nighttime sampling (Table 1).

Table 1.

GW2B and CGMS Accuracy Summary Statistics Overall and According to Time of Day

	Total	11:00 PM – 6:00 AM	6:30 AM – 10:30 PM
GW2B
# of paired data points	487	286	201
Absolute difference* mg/dL median (25th, 75th percentiles)	13 (6, 23)	13 (6, 24)	13 (7, 21)
Values within 10 mg/dL percentage	40%	41%	40%
Values within 20 mg/dL percentage	70%	66%	74%
CGMS
# of paired data points	668	307	361
Absolute difference* mg/dL median (25th, 75th percentiles)	17 (8, 29)	18 (8, 29)	16 (7, 29)
Values within 10 mg/dL percentage	34%	32%	36%
Values within 20 mg/dL percentage	58%	56%	60%

^*Absolute difference defined as the absolute value of the sensor glucose value minus the reference glucose value (always positive).

During the 24 hours of sampling, there were 108 GW2B values <60 mg/dL, with 11 of the 15 subjects having at least one such value. Seven of the 108 values were single isolated low readings (i.e., prior and subsequent values ≥70 mg/dL) and 101 had at least one contiguous value <70 mg/dL. Of these 108 sensor values <60 mg/dL, 33 coincided with a reference glucose measurement. For all 33, the paired reference value was >70 mg/dL, ranging from 74 to 107 mg/dL.

We analyzed not only the frequency of lows but also the occurrence of high glucose values during sampling. There were 22 GW2B values >150 mg/dL, with 6 of the 15 subjects having at least one such value. Twenty of 22 values had at least one contiguous value >140 mg/dL. Of these 22 sensor values >150 mg/dL, 7 coincided with a reference glucose measurement. For all 7, the paired reference value was ≤140 mg/dL, ranging from 101 to 140 mg/dL. Minimum and maximum reference and sensor glucose values are given for each subject in Table 2.

Table 2.

Maximum and Minimum Glucose Values by Subject.

	Maximum Glucose (mg/dL)			Minimum Glucose (mg/dL)
		Sensor			Sensor
Subject*	Reference	GWB	CGMS	Reference	GWB	CGMS
1	143	176	194	77	64	71
2	140	192	159	67	53	71
3	137	145	240	84	52	53
4	136	163	189	89	55	72
5	133	137	154	80	65	68
6	132	171	140	78	61	58
7	128	130	176	78	40	50
8	125	147	153	74	50	55
9	124	157	177	74	43	56
10	119	179	172	60	57	47
11	114	150	156	63	51	61
12	110	116	122	71	58	55
13	109	147	114	36	69	71
14	107	113	181	76	43	62
15	99	142	150	77	40	58

^*Subjects sorted by maximum reference glucose value.

^†Restricted to sensor readings between the first and last reference blood glucose draws.

During periods of simultaneous use of two GW2B sensors, there were 445 paired sensor glucose values. The two paired values were within 10 mg/dL of each other 36% of the time and within 20 mg/dL 62% of the time.

CGMS Accuracy

Six of the 15 subjects used two CGMSs simultaneously. There were 668 CGMS-reference paired glucose values, with 307 of the pairs being from overnight sampling (11:00 PM to 6:00 AM) and 361 of the pairs being from 6:30 AM to 10:30 PM. The median difference between the CGMS and reference glucose values was 6 mg/dL (25^th and 75^th percentiles: −9, 23.5 mg/dL). The absolute value of the differences between the CGMS and reference glucose values ranged from 0 to 127 mg/dL with a median of 17 mg/dL (25^th and 75^th percentiles: 8, 29). The median remained 17 mg/dL when the analysis was limited to the 16 sensors that met Medtronic Minimed’s criteria for an optimal day¹⁸. The CGMS values were within 10 mg/dL of the reference values 34% of the time and within 20 mg/dL of target 58% of the time. Results were similar for daytime and nighttime sampling (Table 1).

Accuracy appeared to be better for the 71 paired values from the 3 modified sensors compared with the 597 values from the 18 original sensors. For the original sensors the median absolute difference was 17 mg/dL (25^th and 75^th percentiles: 8, 30) compared with 11 mg/dL (25^th and 75^th percentiles: 4, 23) for the modified sensors. The glucose values of the original sensors were within 10 mg/dL of the reference values 32% of the time and within 20 mg/dL of the reference value 57% of the time compared with 48% and 72% respectively with the modified sensors.

During the 24 hours of sampling, there were 134 CGMS values <60 mg/dL, with 8 of the 15 subjects having at least one such value. None of the 134 values was a single isolated low reading. Of these 134 sensor values <60 mg/dL, 14 coincided with a reference glucose draw. For all 14, the paired reference value was >70 mg/dL, ranging from 82 to 105 mg/dL.

There were 363 CGMS values >150 mg/dL, with 11 of the 15 subjects having at least one such value. None of the 363 values was a single isolated high reading. Of these 363 sensor values >150 mg/dL, 52 coincided with a reference glucose draw. For all 52, the paired reference value was <140 mg/dL, ranging from 74 to 135 mg/dL.

During periods of simultaneous use of two CGMS sensors, there were 1,476 paired sensor glucose values. The two paired values were within 10 mg/dL 24% of the time and within 20 mg/dL 44% of the time.

Reference Serum Glucose Concentrations

When examining only the 523 reference glucose values, the mean serum glucose concentrations were narrow during the 24 hours of sampling with a mean ± SD of 98 ± 13 mg/dL from 9:30 PM to 6:30 AM and 97 ± 14 mg/dL from 7:00 AM to 9:00 PM. The highest and lowest hourly or half-hourly mean values were 107 mg/dL and 89 mg/dL, respectively (Figure 2). On an individual subject level, the median high-low range was 51 mg/dL, with the widest range being 73 mg/dL and the narrowest range being 22 mg/dL. The median SD value was 12 mg/dL ranging from 4 to 19 mg/dL.

Figure 2.

Mean and range for reference serum glucose values over a 24-hour period.

The lowest reference glucose was from a subject (a 12 year old white male) who had two reference values below 60 mg/dL (36 mg/dL at 10:30 PM and 41 mg/dL at 5:00 AM). The subject’s 10:00 PM and 11:00 PM values were 96 mg/dL and 91 mg/dL respectively, and the 4:30 AM value was 69 mg/dL. The CRC staff had difficulties with the IV line, which was removed shortly after the 5:00 AM blood draw and not replaced. At the time of the 36 mg/dL value, the CGMS glucose was 90 mg/dL and the GW2B value was 97 mg/dL. At the time of the 41 mg/dL reference value, the CGMS glucose was 88 mg/dL and the two GW2B values were 88 mg/dL and 91 mg/dL.

Discussion

This is the first study to evaluate the accuracy of the GW2B and the CGMS in healthy, nondiabetic children compared against reference venous serum samples. The GW2B and CGMS sensor glucose values differed from the reference values by >20 mg/dL 30% and 42% of the time, respectively. The recently modified CGMS sensor was evaluated in two subjects and accuracy appeared better, but the sensor values continued to differ from the reference values by >20 mg/dL 28% of the time.

In order to be used to establish population norms for glucose levels in nondiabetic children, the accuracy of the glucose measurements must approach that of laboratory-measured blood glucose, a level that neither the CGMS nor the GW2B currently approach. Hence these sensors are unable to provide conclusive normative data in nondiabetic individuals. This is particularly important as the use of these sensors has been used by some to evaluate glycemic profiles of children with hypoglycemic conditions such as glycogen storage disease¹⁶. If accurate, they could be ideal for the initial assessment of children with complaints suggestive of hypoglycemia, instead of using frequently sampled blood during a hospitalization. They also could be considered in monitoring glycemic levels in neonates and critically ill children. Unfortunately, this generation of sensors is not accurate enough.

We do recognize that these sensors were not developed to assess glucose levels in nondiabetic individuals and that the accuracy required to establish population norms is greater than that required to monitor the glucose levels in diabetic patients. However, in separate recent publications, we reported our experience on the accuracy of these sensors in 91 diabetic children, similarly studied in an inpatient CRC setting^{18, 19}, and we found that the degree of sensor error was similar to what we found in this study of nondiabetic children. The absolute difference for the GW2B was 16% and for the CGMS was 18%. The recently modified CGMS sensors performed better than the original sensors (median absolute difference 11% versus 19%). Neither the GW2B nor the CGMS was accurate in the detection of hypoglycemia in diabetic children²⁸.

Anecdotal information abounds of healthy subjects wearing the continuous glucose sensor having frank hypoglycemic values, particularly at night. Whether these decrements in glucose concentrations are real or represent an inherent analytical/accuracy issue was not previously clear, as these sensors had not been extensively used in healthy children. One study of CGMS use in 25 normal children in the home environment reported 8.2 ± 7.9% of serum glucose values below 70 mg/dL overall, and 17.9 ± 18.3% during the nocturnal period (12:00 AM to 6:00 AM), but reference laboratory glucose values were not obtained to validate the reliability of these measurements²⁹. In our study, none of the sensor values <60 mg/dL were confirmed with a reference glucose measurement. These data also indicate that the current generation of glucose sensors has limited usefulness in the diagnosis and management of hypoglycemic disorders in children. Our results also indicate that sensor glucose values in the hyperglycemic range are likely to be spurious.

Interestingly, the sensors used also detected multiple hyperglycemic values (>150mg/dL) in these nondiabetic children, and in all of those in which there was a paired reference glucose concentration available the glucose concentration was <140mg/dL. Collectively, these data indicate that both spurious lows and high readings can be obtained with both sensors.

Our study adds to the limited data in the literature on the range of glucose values in healthy, nondiabetic children. The range in serum glucose concentrations measured by the central laboratory was narrow during the 24 hours of sampling. In our healthy children the median standard deviation of the reference glucose values was only 12 mg/dL. In contrast, in the children with type 1 diabetes who were studied under similar conditions, the median SD score was 69 mg/dL. One subject had two reference glucose values in the hypoglycemic range, but there was difficulty maintaining the intravenous line. In view of the fact that the reference glucose levels at the contiguous half-hour time points were not low and the fact that the subject’s sensor values at these times were not low, we believe that it is likely that these low values represented erroneous measurements.

In conclusion, our data support that: 1) neither the GW2B nor CGMS is accurate enough to provide normative glycemia profiles for nondiabetic children, 2) previous reports of low sensor glucose values in nondiabetic children may represent inaccurate glucose measurements, and 3) serum glucose concentrations in nondiabetic children are tightly controlled.

Appendix

Writing Committee

Nelly Mauras, MD (chair); Roy W. Beck, MD, PhD; Katrina J. Ruedy, MSPH; Craig Kollman, PhD; William V. Tamborlane, MD; H. Peter Chase, MD; Bruce A. Buckingham, MD; Eva Tsalikian, MD; Stuart A. Weinzimer, MD; Andrea D. Booth, MS; Dongyuan Xing, MPH

The DirecNet Study Group: Clinical Centers

Listed in alphabetical order with clinical center name, city, and state. Personnel are listed as (PI) for Principal Investigator, (I) for co-Investigator and (C) for Coordinators. 1. Barbara Davis Center for Childhood Diabetes, University of Colorado, Denver, CO: H. Peter Chase, MD (PI); Rosanna Fiallo-Scharer, MD (I); Jennifer H. Fisher, ND, RN (C); 2. Department of Pediatrics, University of Iowa Carver College of Medicine, Iowa City, IA: Eva Tsalikian, MD (PI); Michael J. Tansey, MD (I); Linda F. Larson, RN (C); 3. Nemours Children’s Clinic, Jacksonville, FL: Tim Wysocki, PhD, ABPP (PI); Nelly Mauras, MD (I); Kristen M. Gagnon, MS, RD (C); Pauline Todd, RN (C); 4. Division of Pediatric Endocrinology and Diabetes, Stanford University, Stanford, CA: Bruce A. Buckingham, MD (PI); Darrell M. Wilson, MD (I); Jennifer M. Block, RN, CDE (C); Elizabeth L. Kunselman, RN, CDE (C); 5. Department of Pediatrics, Yale University School of Medicine, New Haven, CT: William V. Tamborlane, MD (PI); Stuart A. Weinzimer, MD (I); Elizabeth A. Doyle, MSN (C); Coordinating Center: Jaeb Center for Health Research, Tampa, FL: Roy W. Beck, MD, PhD; Katrina J. Ruedy, MSPH; Craig Kollman, PhD; Andrea D. Booth, MS; Dongyuan Xing, MPH; Data and Safety Monitoring Board: Dorothy M. Becker, MBBCh; Christopher Cox, PhD; Christopher M. Ryan, PhD; Neil H. White, MD, CDE; Perrin C. White, MD; University of Minnesota Central Laboratory: Michael W. Steffes, MD, PhD; Jean M. Bucksa, CLS; Maren L. Nowicki, CLS; National Institutes of Health: Gilman D. Grave, MD; Barbara Linder MD, PhD; Karen K. Winer, MD