Accuracy Evaluation of Blood Glucose Monitoring Systems in Children on Overnight Closed-Loop Control

Abstract

This pilot study evaluated the difference in accuracy between the Bayer Contour® Next (CN) and HemoCue® (HC) glucose monitoring systems in children with type 1 diabetes participating in overnight closed-loop studies. Subjects aged 10-18 years old were admitted to a clinical research center and glucose values were obtained every 30 minutes overnight. Glucose values were measured using whole blood samples for CN and HC readings and results were compared to Yellow Springs Instrument (YSI) reference values obtained with plasma from the same sample. System accuracy was compared using mean absolute relative difference (MARD) and International Organization for Standardization (ISO) accuracy standards. A total of 28 subjects were enrolled in the study. Glucose measurements were evaluated at 457 time points. CN performed better than HC with an average MARD of 3.13% compared to 10.73% for HC (P < .001). With a limited sample size, CN met ISO criteria (2003 and 2013) at all glucose ranges while HC did not. CN performed very well, and would make an excellent meter for future closed-loop studies outside of a research center.

Accurate measurement of blood glucose levels is necessary for evaluation of continuous glucose monitor performance and glucose control in automated closed-loop insulin delivery systems.¹ Both the Yellow Springs Instrument (YSI, YSI 2300 STAT Plus™, YSI Life Sciences Inc, Yellow Springs, OH) and HemoCue® (HC, HemoCue AB, Angelholm, Sweden) blood glucose monitoring systems are currently accepted by the US Food and Drug Administration (FDA) as reference instruments and have been utilized in inpatient closed-loop studies.^2-4 However, YSI is a laboratory instrument limited to use in an inpatient clinical setting, posing a challenge as closed-loop studies move into the home environment. Previous studies^5-7 have questioned the accuracy of the HC system, which is further limited by its cumbersome size and relatively large fingerstick blood volume requirements, making the transition to the home setting difficult with HC, which is not approved for home use.

The Bayer Contour® Next meter (CN, Bayer Healthcare, Whippany, NJ) is a home blood glucose monitoring system that has displayed very good accuracy in previous studies.^8-10 To establish a better in-home reference glucose device for calibrating continuous glucose sensors and assessing the performance of closed-loop systems, we compared the performance of CN meter and HC systems using YSI as the reference instrument.

Research Design and Methods

The study protocol was approved by the Stanford University Human Subjects Institutional Review Board. Parents or guardians of each study subject gave written informed consent, and subjects less than age 18 years gave written assent. Eligibility criteria for the subjects included clinical diagnosis of type 1 diabetes, using insulin therapy for at least 1 year, and aged 10-18 years.

The study protocol involved an inpatient admission to evaluate the performance of overnight closed-loop artificial pancreas systems. The closed-loop systems consisted of 3 components: an insulin infusion pump, a continuous glucose sensor, and an algorithm which translates data from the glucose sensor to determine insulin delivery. The closed-loop algorithm was designed to increase the time spent in glucose range of 70-150 mg/dl at the onset of the night, with a target glucose of 120 mg/dl by morning. As part of this study, we analyzed the performance of the CN and HC glucose monitoring systems.

Glucose measurements were obtained with the commercially available CN meter and 2 instruments often used in clinical trials: YSI laboratory reference analyzer, which uses glucose oxidase technology, and HC analyzer, which uses glucose dehydrogenase technology. For CN and HC systems, control solution checks were performed on each device prior to subject sampling on each night of testing according to the manufacturer’s instructions. Each night, the YSI was calibrated against YSI standards of 50 mg/dl, 500 mg/dl and 900 mg/dl, and only utilized if readings were within 2% of the YSI standards.

Upon admission, an indwelling intravenous catheter was placed so that samples could be obtained every 30 minutes or every 15 minutes if <70 mg/dl. At each time point, an intravenous blood sample was obtained by study staff to provide 2 CN measurements, 1 HC measurement, and 2 YSI measurements. CN and HC values were measured using venous whole blood. The remaining blood sample was collected in tubes containing ethylenediamine-tetraacetic acid (EDTA) and centrifuged to obtain plasma, which was analyzed for glucose using the YSI.

Statistical Analysis

The absolute difference (CN or HC value minus YSI reference value) and relative absolute difference (absolute difference divided by reference value) were analyzed for each matched pair. The mean of the relative absolute difference (MARD) was calculated at different glucose ranges for CN and HC systems.

This study was not conducted to satisfy the International Organization for Standardization (ISO) accuracy standards for glucose monitoring devices as we did not study enough subjects or have enough samples in the hypoglycemic and hyperglycemic ranges proposed in these standards. However, we did compare the 2 meters for their ability to achieve results within the stated ranges for accuracy:

1. ISO 15197:2003¹¹ (95% of readings within ±15 mg/dl of the reference for glucose levels <75 mg/dl, and ±20% of the reference for glucose levels ≥75 mg/dl), and

2. ISO 15197:2013¹² (95% of readings within ±15 mg/dl of the reference for glucose values <100 mg/dl, and ±15% of the reference for glucose levels ≥100 mg/dl).

The Mann-Whitney test was used for comparison of accuracy between glucose measuring systems. Bias was assessed with Bland-Altman plots¹³ using the difference between the YSI reference glucose and the CN or HC value. Results are presented as mean ± standard deviation unless otherwise indicated.

Results

A total of 28 subjects were enrolled in the study. The average age was 15.2 ± 2.3 years, with average duration of diabetes 7.6 ± 4.5 years, and average hemoglobin A1c of 8.9 ± 0.9%.

There were 465 blood samples obtained for glucose measurement. Eight samples were not included in the analysis due to clotting or dilutional effect giving rise to inaccurate measurements. Therefore, there were 457 time points with at least 1 reference YSI value available for analysis of corresponding CN or HC values. In total, this provided 855 YSI values (including 398 pairs and 59 unpaired values), 902 CN values (451 pairs), and 391 HC values (single value only) available for analysis. Due to technical difficulties in sampling and blood processing, there were 6 time points with no CN readings and 66 time points with no HC values.

An attempt was made to obtain 2 YSI readings at each time point, but technical issues including sample clotting prevented this from occurring. Reference YSI glucose values ranged from 56 to 379 mg/dl. Of these, 11 (2.4%) were less than 70 mg/dl, 433 (94.7%) were between 70-250 and 13 (2.8%) were above 250 mg/dl. YSI performance is illustrated in Supplemental Figure 1.

The MARD for CN measurements across all values (n = 902) was 3.13%, with a MARD of 2.69% for values <100 mg/dl and 3.25% for values ≥ 100 mg/dl (Table 1). The meter met both 2003 and 2013 ISO criteria in all glucose ranges, although there were insufficient meter readings in the hypo- and hyperglycemic ranges (Table 2). The accuracy and bias of the CN meter is shown in Figure 1a. The precision of the CN meter is shown in Supplemental Figure 2.

Table 1.

Mean Absolute Relative Difference Results.

	Glucose <100 mg/dl	Glucose ≥100 mg/dl	All glucose concentrations (56-379 mg/dl)
Contour Next MARD	2.69%	3.25%	3.13%
	n = 200	n = 702	n = 902
HemoCue MARD	9.44%	11.09%	10.73%
	n = 85	n = 306	n = 391

Table 2.

ISO Accuracy Standards.

	ISO 15197:2003 standards		ISO 15197:2013 standards
	% within ±15 mg/dl for <75 mg/dl	% within ±20% for ≥75 mg/dl	% within ±15 mg/dl for <100 mg/dl	% within ±15% for ≥100 mg/dl
Contour Next	100%	99.4%	100%	98.4%
	(38 of 38)	(859 of 864)	(200 of 200)	(691 of 702)
HemoCue	100%	93.5%*	89.4%*	80.7%*
	(19 of 19)	(348 of 372)	(76 of 85)	(247 of 306)

^*Represents failure to achieve ISO accuracy standards (ie, <95%). Contour Next met ISO criteria for all glucose ranges.

Figure 1a.

Contour Next Bias.

The MARD for the HC analyzer across all values (n = 391) was 10.73%, with a MARD of 9.44% for values <100 mg/dl and 11.09% for values ≥ 100 mg/dl (Table 1). The HC failed to meet ISO 15197:2003 standards with 93.5% of values within ±20% of reference glucose ≥ 75 mg/dl. In addition, with our limited sample size, HC failed to meet ISO 15197:2013 standards with 89.4 % of values within ±15 mg/dl of reference glucose < 100 mg/dl, and 80.7% of values within ±15% of reference glucose ≥ 100 mg/dl (Table 2). The accuracy and bias of the HC system is shown in Figure 1b. The Bland-Altman plot of HC values showed a positive bias, especially at higher glucose ranges.

Figure 1b.

Hemocue (HC) bias.

Discussion

In a clinical research center setting, we analyzed the performance of CN and HC glucose monitoring systems in children with type 1 diabetes on overnight closed-loop control. In this pilot study, CN proved more accurate than the HC system. The average MARD for the CN meter across all values was 3.13% compared to 10.73% for the HC system (P < .001). Though we had a limited sample size of 28 subjects and a limited number of hypoglycemic and hyperglycemic values, our CN results did meet ISO glucose meter accuracy standards (ISO 15197:2003 and ISO 15197:2013), while the HC system did not.

These findings have important implications for future closed-loop research studies. First, since the HC system is currently considered by the FDA an acceptable reference glucose measurement system, it would be our recommendation that CN would be a reasonable reference glucose meter for outpatient closed-loop studies. The sample requirement for HC is 10 µl compared to 0.6 µl for CN meter, making CN meter the obvious choice if frequent fingerstick capillary glucose values are to be obtained.

In addition, the accuracy of the CN meter indicates that it is a suitable instrument for calibrating and ensuring continuous glucose monitor (CGM) accuracy, which is particularly important in the outpatient setting. The range of glucose values in this study is comparable to the expected range of glucose values in someone using a nocturnal closed-loop system in-home.

Our study results are consistent with recent studies on the accuracy of the CN blood glucose monitoring system. The North American Comparator Trial⁸ evaluated the accuracy of 5 home systems, including the Contour Next EZ, ACCU-CHEK® Aviva, FreeStyle Freedom Lite®, OneTouch® Ultra® 2 and TRUEtrack®, and determined CN to be the most accurate system with a MARD of 4.7% overall, including a MARD of 0.65% in the <70 mg/dl range. Freckmann et al⁹ evaluated 12 commercially available systems including the European version of the Contour Next (XT), ACCU-CHEK Aviva, BGStar™, and OneTouch® Verio™ Pro among others at the full distribution of glucose ranges based on DIN EN ISO 15197:2003 criteria. The Contour XT system performed very well with 100% of values in zone A on consensus error grid analysis and 100% within ±15 mg/dL and ±20% of reference glucose values (n = 200). Our study results corroborate these previous findings, identifying the CN as one of the most accurate currently available home glucose meters.

HC accuracy results in our study are similar to what others have reported. Buhling et al⁶ reported an average difference from the reference method of –9.4 ± 5.9% for HC, worse than the 5 portable meters analyzed in that study. HC has displayed a trend to systematically overestimate blood glucose levels,^5,7 which was seen in our study as well.

There are limitations to our study. For example, there were few samples less than 70 mg/dl or greater than 250 mg/dl because the subjects were in a study of overnight closed-loop control where glucose values were targeted to be within a narrow range. As such, the study did not follow the DIN EN ISO15197:2003 (or 2013) bin distribution requirements for different glucose concentrations. Another issue was missed glucose readings. A few readings were missed due to technical issues, including samples clotting. Despite indications for CN use with fingerstick capillary whole blood samples, values were obtained using venous whole blood. In a study of 2 home glucose meters, capillary glucose readings had a –2.4 to –0.9 mg/dl bias in the fasting state, which increased to a +14 to +22 mg/dl bias postprandially.¹⁴ Since these samples were obtained overnight, they were predominantly in a fasting state, so the capillary to venous bias was minimal. In our studies, any capillary to venous bias was applied equally to both CN and HC samples. Neither meter would have been affected by difference in capillary compared to venous oxygen levels, since they are based on a glucose dehydrogenase enzyme assay, which is not oxygen-sensitive.

Conclusions

In this pilot study of 28 subjects, CN was more accurate than the HC glucose measuring system, meeting the new ISO criteria in the sample distribution of these closed-loop studies, whereas HC failed to meet the new ISO criteria. With proper sampling technique, CN would be an excellent meter to be used in calibrating CGM devices in-home. We have demonstrated very good accuracy for the range of glucose values that would be expected in typical overnight closed-loop studies, making CN meter or a meter with similar or better accuracy, an excellent choice for future overnight in-home closed-loop systems.