The Accuracy of the CGMS™ in Children with Type 1 Diabetes: Results of the Diabetes Research in Children Network (DirecNet) Accuracy Study

Abstract

Objective

To assess the accuracy of the Continuous Glucose Monitoring System, CGMS™ (“CGMS”) in children and adolescents with type 1 diabetes when compared with reference serum glucose levels during spontaneous fluctuations in glucose levels over 24 hours and during acute hyper- and hypoglycemia.

Research Design and Methods

Ninety-one subjects with type 1 diabetes (3.5 to 17.7 years old) wore 1 or 2 CGMSs while blood samples were obtained for serum glucose determinations (made at a central laboratory) hourly during the day, every 30 minutes overnight, and every 5 minutes during meal-induced hyperglycemia and insulin-induced hypoglycemia tests, resulting in 6,778 CGMS reference glucose pairs. CGMS function was assessed on each of the three days of sensor life.

Results

The median relative absolute difference (RAD) between the CGMS and reference values was 18% (25^th, 75^th percentiles= 8%, 34%). Similar results were obtained on each of the three days of sensor life. Accuracy was worse during hypoglycemia than during hyperglycemia. Modified sensors that first became available in November 2002 were more accurate than were the original sensors (median RAD= 11% vs. 19%) and had better precision (r= 0.92 vs. r= 0.77) during time periods in which two CGMSs were simultaneously used.

Conclusions

The CGMS sensors that have been in clinical use until recently are often inaccurate in quantifying glucose values in children with T1DM. However, recent modifications to the sensor have resulted in substantially better accuracy and reliability. This improved function, if confirmed by additional data, may enhance the clinical utility of the CGMS.

The results of the Diabetes Control and Complications Trial (DCCT) indicate that most patients with type 1 diabetes mellitus (T1DM) should receive intensive treatment aimed at lowering HbA1c levels as close to normal as possible.¹ However, this goal of therapy has been particularly difficult to achieve by clinicians who care for youth with T1DM in clinical practice.² Even in the research setting of the DCCT, intensively-treated adolescents had higher HbA1c levels and a 50% increase in the frequency of severe hypoglycemic events compared with intensively-treated adults in the study.³ A number of new approaches to management have been introduced since the end of the DCCT that offer the possibility of lowering both HbA1c levels and the risk of severe hypoglycemia. Important advances have been made in the development of glucose sensor devices for near-continuous monitoring of interstitial glucose concentrations, which have great potential in the clinical practice of diabetes.

The Continuous Glucose Monitoring System, CGMS™ (“CGMS”; Medtronic MiniMed, Northridge, CA) was the first glucose sensor approved by the Food and Drug Administration. It has been used by a number of investigators to characterize blood glucose profiles in youth with T1DM.^4–7 A common observation has been that the CGMS reveals marked post-prandial hyperglycemia during the day and frequent, prolonged, and asymptomatic hypoglycemia during the night. Interpretation of these findings is limited, however, because the accuracy of the CGMS in children and adolescents has not been established. Most of the CGMS validation studies were performed in adults and were based on comparison with glucose meter values rather than blood glucose levels measured by instruments commonly used in clinical laboratories. Moreover, several recent reports have raised questions regarding the accuracy and reproducibility of the CGMS in diabetic and non-diabetic subjects).^{7, 8}

The Diabetes Research in Children Network (DirecNet) is a NIH-funded collaborative study group that consists of five clinical centers, a central laboratory, a coordinating center, and representatives from NICHD and NIDDK. A major objective of DirecNet is to critically evaluate the clinical usefulness of current and future glucose sensor devices in youth with T1DM. The purpose of this paper is to report our Study Group’s findings with respect to CGMS function in the DirecNet Accuracy Study. A companion study examining the accuracy of the GlucoWatch® G2^™ Biographer (“GW2B”, Cygnus, Inc., Redwood City, CA) is reported separately.⁹

Methods

We conducted a study to test the performance of glucose sensors against frequently sampled serum glucose levels analyzed at a central laboratory. Subjects with type 1 diabetes in the age range of 3.5 to 17.7 years old years were studied in an inpatient, clinical research center setting. Study procedures were conducted to examine sensor accuracy during acute hyper- and hypoglycemia, as well as during spontaneous fluctuations in glucose levels over 24 hours. Blood samples were obtained for reference serum glucose determinations (made at a central laboratory) hourly during the day, every 30 minutes overnight, and every 5 minutes during meal-induced hyperglycemia and insulin-induced hypoglycemia tests. The study methods and informed consent procedures are detailed in a companion manuscript on the GW2B.⁹

In order to assess CGMS function over the entire 72 hours of its lifespan, within each of three age groups (1 to <7, 7 to <12, and 12 to <18), the CGMS was inserted in approximately equal numbers on each of three start days (48 hours prior to admission, 24 hours prior to admission, and on admission to the CRC). Study staff inserted the sensor in the abdomen or upper buttocks. For cases in which the sensor was inserted prior to admission, parents and subjects were instructed in home use of the CGMS, including the requirement to input at least four calibration values a day using a new One Touch^® Ultra^® Meter (“Ultra”; Lifescan, Milpitas, CA) that was provided by the study.

Each subject was admitted to the CRC for 24 to 26 hours. A CGMS sensor was inserted by study staff in those subjects who had not started use of the CGMS as an outpatient. In subjects already using the CGMS, the sensor was checked and it was replaced if it was not functioning. Each subject was offered the option of wearing a second CGMS during the inpatient stay (in some subjects, the second CGMS was started concurrently with the first CGMS one or two days prior to admission). During this admission, study staff entered calibration values obtained from the Ultra meter (primarily using capillary blood). Prior to hospital discharge and at a follow-up visit three to five days following discharge, the skin in the area of each sensor insertion site was inspected for signs of erythema and inflammation.

Statistical Methods

The principles underlying the study’s sample size and the statistical methods are described in the companion manuscript.⁹

For the accuracy analyses, the CGMS glucose measurements were matched to reference measurements from blood samples drawn within 2.5 minutes of each other, after adjusting the CGMS time by 2.5 minutes (to account for the averaging of glucose measurements made over the prior 5 minutes). For evaluating precision between two simultaneous CGMSs, sensor values were matched within 2.5 minutes of each other.

In addition to the overall analyses, we performed a separate set of accuracy analyses to compare CGMS values from “optimal days” and “nonoptimal” days. The CGMS software analyzes the relationship between sensor outputs and blood glucose calibration values and classifies each day of sensor function as optimal or nonoptimal. If the range (minimum to maximum) of meter values is at least 100 mg/dL, then a day is considered optimal when the mean relative absolute difference between at least three paired sensor glucose and meter glucose values is ≤28% and the correlation coefficient between paired sensor glucose and meter glucose values is ≥0.79 whereas if the range is less than 100 mg/dL, then the day is considered optimal when the relative mean absolute difference is ≤18%.

During the course of the study, Medtronic MiniMed changed the manufacturing process of the sensor that had been in place since 1999 by modifying the sensor fabrication process. Accuracy analyses were conducted separately for the “original” and “modified” sensors.

Results

Between May 30 and November 18, 2002, informed consent was obtained for study participation from 97 subjects. One subject withdrew prior to hospital admission without having a CGMS inserted. Five subjects, who had a CGMS inserted, withdrew after hospital admission; in no cases was the subject’s withdrawal related to sensor use. The age of the 91 subjects remaining for analysis ranged from 3.5 to 17.7 (mean 9.9) years; 51% were female and 85% Caucasian (Table 1).

Table 1.

Demographic and Clinical Characteristics of Study Subjects

	N=91
Gender: female N(%)	46 (51)
Age Mean ± SD (years)	9.9 ± 4.1
Race/Ethnicity N(%)
White	77 (85)
Hispanic or Latino	7 (8)
African-American	4 (4)
Other	3 (3)
Duration of Diabetes by Age Group Mean ± SD years
All ages	4.7 ± 3.2
< 7 (N=30)	3.1 ± 1.0
7–<12 (N=29)	3.7 ± 2.4
12–<18 (N=32)	7.0 ± 3.9
Insulin Route N (%)
Pump	39 (43)
Injections	52 (57)
HbA1c Mean ± SD	7.8 ± 1.2
BMI percentile Mean ± SD	67% ± 23%

Seventy-seven of the subjects were studied using original sensors (40 used one CGMS and 37 used two simultaneous CGMSs) and 13 were studied using the modified sensors as described in the methods section (2 used one CGMS and 11 used two simultaneous CGMSs). One subject was studied using one of each type. The meal-induced hyperglycemia test was conducted for 84 subjects and the insulin-induced hypoglycemia test was conducted for 54 subjects.

Sensor Accuracy

There were 6,778 CGMS-reference paired glucose values: 3,906 for the hour/half-hour blood draws, 1,136 for the insulin-induced hypoglycemia test, 1,341 for the meal-induced hyperglycemia test, and 395 at other times. The number of paired CGMS-reference glucose values averaged 74 ± 34 per subject (median= 64, interquartile range= 49 to 105, range= 4 to 142).

For the 6,778 paired CGMS-reference glucose values, the median difference between the glucose sensor and reference glucose level was only 2 mg/dL and the mean difference was −2.9 mg/dL (not significantly different than 0 mg/dL), indicating that the sensor did not systematically under or over read glucose levels. On the other hand, the median RAD was 18% (Table 2). The Pearson correlation between the paired CGMS and reference glucose values was 0.80 and the proposed ISO criteria (for reference glucose value ≤75 mg/dL, CGMS value within ± 15 mg/dL and for reference glucose value >75 mg/dL, CGMS value within ± 20%) were met for 56% of the CGMS values. Sixty-one percent of values were within zone A of the modified error grid and 94% were within zones A + B.

Table 2. CGMS Accuracy Summary Statistics.

(N=6,778 sensor-reference pairs)

	Mean (95% confidence interval)	Median (25th, 75th percentiles)
Difference mg/dL*	−2.9 (−8.0, 2.1)	2 (−27, 28)
Absolute Difference mg/dL†	38.0 (34.7, 41.7)	28 (12, 50)
Relative Difference‡	4% (1%, 8%)	2% (−16%, 20%)
Relative Absolute Difference§	25% (23%, 27%)	18% (8%, 34%)

^*Difference is the sensor glucose value minus the reference glucose value.

^†Absolute difference is the absolute value of the difference.

^‡Relative Difference is the difference divided by the reference glucose value (expressed as percentage).

^§Relative absolute difference is the absolute value of the relative difference (expressed as percentage).

As shown in Table 3, CGMS accuracy showed no meaningful variation according to subject age or body mass index. Accuracy appeared slightly better for girls than for boys and for daytime versus nighttime values. There was no indication of a meaningful degradation of accuracy in the sensors being evaluated during the first 24 hours of their lifespan compared with those evaluated during the last 24 hours. The sensors meeting MiniMed criteria for optimal had slightly better accuracy than did the sensors considered nonoptimal. The median number of calibration values entered into each CGMS during the inpatient stay was 6 (25^th, 75^th percentiles = 5,7). Accuracy did not appear to be appreciably enhanced by the entry of more than the generally recommended minimum of four calibration values per day.

Table 3.

Accuracy Summary Statistics According to Various Factors

	# Paired Data Points	Relative Absolute Difference (median)	ISO criteria met* (percentage)	P-values†
Overall	6,778	18%	56%	-
Subject Age (years)				0.11/0.20
< 7	1,572	15%	63%
7–< 12	2,259	18%	54%
≥ 12	2,947	19%	53%
Gender				0.06/0.04
Female	3,401	16%	60%
Male	3,377	20%	52%
Body Mass Index				0.92/0.87
≤50th percentile	1,555	17%	57%
>50th percentile	5,223	18%	55%
Time of Day‡				0.005/0.009
Daytime (6:30AM–10:30PM)	2,045	16%	60%
Nighttime (11PM–6AM)	1,861	19%	53%
Sensor Age (hours)				0.95/1.00
0–< 24	3,634	18%	55%
24–< 48	1,873	17%	57%
≥ 48	1,271	18%	56%
Plasma Glucose Level (mg/dL)				< 0.001/< 0.001
≤ 70	646	35%	42%
71–120	1,495	22%	47%
121–180	1,548	18%	55%
181–240	1,507	15%	63%
> 240	1,582	14%	63%
Hypoglycemia Test	1,136	22%	50%	0.01/0.08
Hyperglycemia Test	1,341	16%	58%
Optimal Days	4,660	16%	59%	0.01/0.005
Non Optimal Days	2,118	21%	48%

^*ISO criteria: for reference glucose value ≤75 mg/dL, CGMS value ±15 mg/dL and for reference glucose value>75 mg/dL, CGMS value within ±20%.

^†The first p value is for RAD and the second p-value is for ISO criteria met.

^‡Includes only the hourly and half-hour reference glucose blood draws.

Accuracy with Original Versus Modified Sensors

As shown in Table 4, accuracy was substantially better on all measures with the 25 CGMSs that used modified sensors compared with the 115 that used original sensors: median RAD (11% versus 19%, P<0.001), and percentage meeting ISO criteria (72% versus 53%, P<0.001). Among the 21 modified sensors meeting the optimal criteria, the median RAD for the 934 sensor values was 11%, and 70% of the sensor values met ISO criteria.

Table 4.

Comparison of Accuracy of Original and Modified Sensors

	Original Sensors N=5,658 Paired Values	Modified Sensors N=1,120 Paired Values	P-value
Relative Absolute Difference Median (25th, 75th percentiles)	19% (9%, 36%)	11% (5%, 23%)	< 0.001
Modified Clarke Error Grid Zone A percentage	58%	78%	< 0.001
Modified Clarke Error Grid Zone A+B percentage	93%	98%	<0.001
ISO Criteria Met According to Reference Glucose Level*
≤70 mg/dL	41%	48%	0.37
71–120 mg/dL	45%	60%	0.07
121–180 mg/dL	53%	70%	0.02
181–240 mg/dL	60%	77%	0.004
>240 mg/dL	58%	81%	0.007
All Values	53%	72%	< 0.001
Variation of Accuracy Among Sensors†	N=112 Sensors	N=25 Sensors
Sensors with Median RAD <10%	9%	36%	0.01
Sensors with Median RAD <20%	54%	88%	<0.001
Sensors with Median RAD <30%	80%	100%	<0.001
Sensors with ≥80% of Values Meeting ISO Criteria	9%	24%	0.07
Sensors with ≥60% of Values Meeting ISO Criteria	35%	80%	< 0.001

^*ISO criteria: for reference glucose value ≤75 mg/dL, CGMS value ±15 mg/dL and for reference glucose value>75 mg/dL, CCGMS value within ±20%.

^†Excludes 3 original sensors with fewer than ten paired reference glucose values.

The higher accuracy of the modified sensors was present across the range of reference glucose levels (Table 4). However, for the modified as well as the original sensors, accuracy was better at higher reference glucose levels than low glucose levels (RAD: P= 0.003 and <0.001, respectively; ISO criteria met: P= 0.03 and <0.001, respectively). Accuracy was computed separately for each sensor. Nine (36%) of the 25 modified sensors with at least 10 pairs had a median RAD <10% and 6 (24%) had at least 80% of values meeting ISO criteria whereas only 10 (9%) and 10 (9%) respectively of the 112 original sensors met these criteria. On the low end of accuracy, none of the modified sensors had a median RAD =30% and 5 (20%) had fewer than 60% of values meeting ISO criteria whereas 22 (20%) and 73 (65%) respectively of the original sensors were this inaccurate.

Precision

During the time periods when the subjects were using two CGMSs, the 13,669 paired glucose values from original sensors correlated at 0.77 and the 5,021 paired glucose values from modified sensors correlated at 0.92. Compared with the original sensors, the modified sensors had a lower median RAD (P< 0.001) and a higher percentage of paired values that differed by no more than 10% (P< 0.001) (Table 5). The results appeared similar when the sensor pairs were divided into those from the same 24-hour period of sensor life and those from different 24-hour periods of sensor life (data not shown).

Table 5.

Precision During Time Periods of Simultaneous Use of Two CGMSs

	Original Sensors	Modified Sensors
# of Paired Data Points	13,669	5,021
Absolute Difference*Median mg/dL (25th, 75th percentiles)	28 (12, 55)	17 (7, 34)
Relative absolute difference† Median (25th, 75th percentiles)	17% (7%, 33%)	10% (4%, 20%)
Values within 10% percentage	32%	50%
Values within 20% percentage	55%	76%
Pearson Correlation	0.77	0.92

^*Absolute difference is the absolute value of the difference.

^†Relative absolute difference is the absolute value of the relative difference (expressed as percentage).

Sensor Function

Among the 36 sensors placed two days prior to admission, 33 (92%) were still functioning at the time of admission and 23 (64%) were still functioning at the end of study. Among the 42 sensors placed one day prior to admission, all were still functioning at the time of admission and 26 (62%) were functioning at the end of the study. Among the 64 sensors placed on the day of admission (includes 2 sensors with no reference glucose pairs), 57 (89%) were still functioning at the end of the study. Five percent of the possible 67,680 readings during the time that the sensor was functioning were skipped (2%, 9%, and 12% respectively during the 1^st, 2^nd, and 3^rd 24 hours of sensor life, P< 0.001).

Adverse Effects

There were no adverse effects that occurred related to the CGMS use and no serious effects of the insulin-induced hypoglycemia test or other study procedures.

Discussion

In this study, 91 subjects who spanned the age range of children and adolescents with type 1 diabetes used more than 140 CGMS sensors that provided over 6,000 paired sensor values to compare with reference glucose determinations made at a central laboratory. The overall percentage of skipped readings was low and the majority of sensors that were started two days prior to admission remained functional for the full 3 days. There were no insertion site problems or other adverse effects from wearing the device.

When all of the sensor results were analyzed together, the median error (expressed as relative absolute difference) was 18% and the mean error was 25%. Accuracy was considerably better during periods of hyperglycemia than during periods of hypoglycemia and appeared slightly better during daytime than nighttime hours. Potential modifying factors, including subject age, sensor age, and BMI, did not substantially affect the accuracy of the CGMS. Accuracy appeared better for girls than boys, but in view of the multiple statistical comparisons being made, this difference may be a chance finding. When we excluded days that the CGMS considered nonoptimal, accuracy improved but the median error was still 16%. This compares with a median error of 6% with the Ultra meters (from venous samples) used in our study. The proposed ISO criteria for assessing the accuracy of glucose meters (that the meter value should be ±15 mg/dL if ≤75 mg/dL and ±20% if >75mg/dL versus the reference value) is met by most meters at least 85% of the time.¹⁰ These criteria were met 96% of the time by our 1,741 Ultra glucose values but were met only 56% of the time by the CGMS values.

Some^11–14 but not all⁷ studies have reported better correlations between CGMS and reference values than we observed in this study. However, in two of the studies that reported a close correlation between sensor and reference values, the reference glucose values were also used to calibrate the CGMS,^{11, 13} and in the third study almost half of the values used to assess accuracy were also used to calibrate the CGMS.¹⁴ Thus, the correlations in those studies may be at least in part a measure of how well the calibration equation was functioning rather than being a true measure of accuracy.

All previously published CGMS accuracy studies used what we have referred to as the “original” sensors. Sensors manufactured by a modified sensor fabrication process, which first became available near the end of the study, were used by 14 of our 91 subjects. We found that CGMS accuracy was substantially improved with the modified sensors. In comparison with the original sensor, the modified sensor was more accurate on all measures and demonstrated better precision during time periods when two sensors were being simultaneously used. It remains to be determined whether the improved accuracy of modified sensors will continue to be observed in larger numbers of subjects. If so, clinicians will be able to interpret the results of CGMS profiles with much greater confidence than with the original sensors.

In summary, the CGMS sensors that have been in clinical use until recently are frequently inaccurate in quantifying glucose values in children with T1DM. However, newly modified sensors functioned substantially better than the original sensors, offering promise that the CGMS may be of greater value for evaluating glycemic trends than was indicated by our results using original sensors.

Appendix

Writing Committee

Bruce A. Buckingham, MD; Roy W. Beck, MD, PhD; William V. Tamborlane, MD, Elizabeth A. Boland, MSN; H. Peter Chase, MD; Michael J. Tansey, MD; Nelly Mauras, MD; Stuart A. Weinzimer, MD; Katrina J. Ruedy, MSPH; Andrea D. Booth, MS; Craig Kollman, PhD.

The DirecNet Study Group

Clinical Centers

Listed in alphabetical order with clinical center name, city, and state. Personnel are listed as (PI) for Principal Investigator, (I) for co-Investigator and (C) for Coordinators.

1. Barbara Davis Center for Childhood Diabetes, University of Colorado, Denver, CO

   H. Peter Chase, MD (PI); Rosanna Fiallo-Scharer, MD (I); Jennifer H. Fisher, ND, RN (C)

2. Department of Pediatrics, University of Iowa Carver College of Medicine, Iowa City, IA

   Eva Tsalikian, MD (PI); Michael J. Tansey, MD (I); Linda F. Larson, RN (C)

3. Nemours Children’s Clinic, Jacksonville, FL

   Tim Wysocki, PhD, ABPP (PI); Nelly Mauras, MD (I); Kristen M. Gagnon, MS, RD (C); Pauline Todd, RN (C)

4. Division of Pediatric Endocrinology and Diabetes, Stanford University, Stanford, CA

   Bruce A. Buckingham, MD (PI); Darrell M. Wilson, MD (I); Jennifer M. Block, RN, CDE (C); Elizabeth L. Kunselman, RN, CDE (C)

5. Department of Pediatrics, Yale University School of Medicine, New Haven, CT

   William V. Tamborlane, MD (PI); Stuart A. Weinzimer, MD (I); Elizabeth A. Boland, MSN (C)

Coordinating Center

Jaeb Center for Health Research, Tampa, FL

Roy W. Beck, MD, PhD; Katrina J. Ruedy, MSPH; Craig Kollman, PhD; Andrea D. Booth, MS; Gladys N. Bernett, MBA, MHA; Pamela S. Moke, MSPH; Lara M. Labastie

Data and Safety Monitoring Board

Dorothy M. Becker, MBBCh; Christopher Cox, PhD; Christopher M. Ryan, PhD; Neil H. White, MD, CDE; Perrin C. White, MD

University of Minnesota Central Laboratory

Michael W. Steffes, MD, PhD; Jean M. Bucksa, CLS; Maren L. Nowicki, CLS

National Institutes of Health

Gilman D. Grave, MD; Barbara Linder MD, PhD; Karen K. Winer, MD