A Comprehensive Performance Evaluation of Five Blood Glucose Systems in the Hypo-, Eu-, and Hyperglycemic Range

Abstract

Background:

The objective was to evaluate the performance (in terms of accuracy, precision, and trueness) of 5 CE-certified and commercially available blood glucose (BG) systems (meters plus test strips) using an innovative clinical-experimental study design with a 3-step glucose clamp approach and frequent capillary BG measurements.

Methods:

Sixteen subjects with type 1 diabetes participated in this open label, single center trial. BG was clamped at 3 levels for 60 minutes each: 60-100-200 mg/dL. Medical staff performed regular finger pricks (up to 10 per BG level) to obtain capillary blood samples for paired BG measurements with the 5 BG systems and a laboratory method as comparison.

Results:

Three BG systems displayed significantly lower mean absolute relative deviations (MARD) (ACCU-Chek® Aviva Nano [5.4%], BGStar® [5.1%], iBGStar® [5.3%]) than 2 others (FreeStyle InsuLinx® [7.7%], OneTouch Verio®IQ [10.3%]). The measurement precision of all BG systems was comparable, but relative bias was also lower for the 3 systems with lower MARD (ACCU-Chek [1.3%], BGStar [–0.9%], iBGStar [1.0%]) compared with the 2 others (FreeStyle [–7.2%], OneTouch [8.9%]).

Conclusions:

This 3 range glucose clamp approach enables a systematic performance evaluation of BG systems under controlled and reproducible conditions. The random error of the tested BG systems was comparable, but some showed a lower systematic error than others. These BG systems allow an accurate glucose measurement at low, normal and high BG levels.

The performance of blood glucose (BG) measurement systems (BG meters plus test strips) is a much debated and researched topic. Reliable BG measurements are needed to ensure that patients with diabetes make the same clinical decisions (eg, select appropriate insulin dose) as they would do if the glucose measurements were made with a laboratory or even better with a reference method.¹ BG systems are considered accurate if they show high precision and high trueness, that is, both the random error (measurement imprecision or variability) and the systematic error (measurement bias) are low.²

The evaluation of BG systems should be performed in accordance with standard guidelines like the International Organization for Standardization (ISO) (ISO 15197:2013 In vitro diagnostic test systems—Requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus). Such guidelines describe in detail test procedures to be followed. They also specify the allowed measurement deviation of the measurement results from the laboratory measurements and the maximum number of inaccurate results or outliers acceptable. The analytical goals defined by the ISO guidelines are now widely adopted by manufacturers of BG systems and regulatory agencies.

Despite the prerequisite of demonstrating compliance with ISO 15197 before market approval of a new BG system, the performance of currently available BG systems (= after market approval) has been fiercely debated after reports about BG systems on the market failing ISO evaluations.^3-7 Urgent action calls have been published by the European Association for the Study of Diabetes (EASD) and the Diabetes Technology Society (DTS) advocating the need for postmarketing surveillance programs and development of test protocols to assess BG system performance.

In this article an evaluation approach is presented that employs a 3-step glucose clamp protocol to systematically evaluate the performance of BG systems under controlled and reproducible conditions at 3 clinically relevant glucose levels. It is not an aim to develop an approach that could replace the ISO 15197 evaluation, but the presented glucose clamp method may provide a different level of evaluation for the performance of BG systems. The key feature of the glucose clamp technique is keeping the BG concentration of an individual stable at any target level. It is believed that repeated blood sampling for longer periods of time at such a stable BG concentration is a unique and valuable feature that will allow an assessment of the measurement precision of a BG system under the same conditions as during which accuracy and trueness are evaluated.

The aim of this study was to use the glucose clamp approach for a complete performance evaluation of 5 commercially available BG systems.

Methods

This study was an open label, single center trial investigating the accuracy, precision and trueness of 5 commercially available BG systems. Twenty subjects with type 1 diabetes [4 female and 16 male, age 49 ± 12 years (mean ± SD), BMI 23.8 ± 2.1 kg/m², HbA1c 7.4 ± 0.7%) participated in this trial. The study protocol was approved by an independent ethics committee (Ethikkommission der Ärztekammer Nordrhein) and performed in accordance with the Declaration of Helsinki⁸ and Guidelines of Good Clinical Practice.

Blood Glucose Systems

Five BG meters with corresponding test strips were tested: BGStar® (referred to as system 1 in the remaining text of this article), iBGStar® (= system 2, both system 1 and 2 are distributed by Sanofi, Frankfurt, Germany), ACCU-Chek® Aviva Nano (= system 3, Roche Diagnostics, Basel, Switzerland), FreeStyle InsuLinx® (= system 4, Abbott Diabetes Care, Alameda, CA, USA), and OneTouch Verio®IQ (= system 5, LifeScan, Milpitas, CA, USA). All BG systems have a CE mark and all devices and test strips were purchased at a local pharmacy. For each BG system 4 meters were evaluated. At the start of each test day, the proper functioning of the BG systems was verified using a control solution according to the manufacturer’s instructions. For each experiment 2 meters for each of the systems mentioned above were used and test strips were from the same batch.

Laboratory Glucose Measurement

Laboratory glucose measurements were performed with the YSI 2300 STAT Plus Glucose Analyzer (YSI Inc, Yellow Springs, OH, USA) from plasma samples obtained from finger pricks. To ensure correct and reliable functioning of the YSI analyzer, internal quality control measures were in place, including daily checks with glucose standard solutions, timely replacement of the measurement membranes and yearly servicing. For each measurement, a 100 µL capillary blood sample was obtained from a finger prick performed by a trained technician. The capillary blood sample was collected in a standard (containing neither anticoagulant nor other additives) 0.5 mL Eppendorf tube and centrifuged immediately after sampling. The plasma aliquot was analyzed in duplicate (black and white sensor) with the laboratory analyzer.

Glucose Clamp Procedure

After an initial screening visit, eligible subjects were invited to return to the trial center for 1 test day with a 3-step glucose clamp procedure and frequent capillary blood sampling (Figure 1). In the morning a blood sample for hematocrit determination was taken and subjects were connected to a glucose clamp device (Biostator, Miles Laboratories Inc, Elkhart, IN, USA) via 2 indwelling cannulas. The clamp device continuously measures the subject’s BG (through continuous venous blood sampling) and controls at the same time a variable rate IV glucose infusion. The glucose infusion rate is automatically adjusted via an integrated mathematical algorithm to maintain the BG concentration of the subject at a constant level. The glycemia of each subject was clamped at 3 different levels (200-100-60 mg/dL) for approximately 1 hour per clamp step. The clamp device was reprogrammed to a new level at the start of each step. To lower the glycemia at the end of a given step to achieve the next level, a variable rate IV insulin infusion was started manually via an external infusion pump.

Figure 1.

Data from a 3-step clamp experiment, illustrating the plasma glucose clamp profile (gray continuous line), the clamp target levels at 200, 100, and 60 mg/dL (black continuous lines) with ± 10% deviation windows (interrupted black lines) and frequent capillary blood sampling for simultaneous BG system (not shown) and YSI (red dots) measurements. Plasma glucose was kept constant at each clamp step for approximately 1 hour during which 10 capillary samples were taken.

As soon as the subject’s BG (as measured by the laboratory analyzer) was within ± 10% from a given clamp level, a period of repeated capillary blood sampling was started. If during this sampling period the subject’s BG deviated more than 10% from the clamp level, the data point was included in the analysis, but further sampling was halted for a short period of time to bring the subject’s BG back into the target window. Per clamp level and per subject 10 capillary blood samples were obtained by finger pricking with a single-use safety lancet. Before each finger prick the hands of the subject were cleaned. Immediately after obtaining the capillary blood sample the following 4 steps were performed:

1. A 100 µL aliquot was collected for laboratory plasma glucose measurement. The capillary blood drop was wiped from the finger and a new blood drop was obtained by light pressure or by performing another finger prick if needed.

2. Plasma glucose concentration was measured by a BG system in duplicate (2 meters).

3. Plasma glucose concentration was measured by another BG meter system in duplicate (2 meters).

4. A second 100 µL aliquot was collected for a repeated laboratory plasma glucose measurement.

BG system measurement results were obtained instantaneously; laboratory plasma glucose results were obtained within 10 minutes after obtaining the capillary samples.

Statistical Analysis

Data were excluded from statistical analysis if:

• - BG system or laboratory glucose measurement values were missing

• - the drift between the first and second laboratory measurement was >4 mg/dL at BG concentrations ≤100 mg/dL or >4% at BG concentrations >100 mg/dL

• - a technical or handling error occurred

• - the hematocrit value was outside the validated range of the BG meters (20-60%)

The quality of the 3 clamp periods was evaluated by calculating how tight the subject’s BG was kept at the given target levels:⁹

• control deviation (mg/dL), ie, the mean difference between the laboratory BG measurements and the target BG level

• CV (%), ie, the spread of laboratory BG measurements around the target BG level

To assess the performance of the BG systems, the result of each BG meter measurement was compared to the mean result of the 2 duplicate laboratory plasma glucose measurements, performed immediately before and after the BG system measurements.

The following endpoints were calculated for the BG systems:

Measurement accuracy—Primary endpoint: mean absolute relative deviation (MARD); secondary endpoints: absolute and relative number of BG system results within ± 15 mg/dL of laboratory glucose concentrations <100 mg/dL and within ± 15% of laboratory glucose concentrations ≥100 mg/dL, the absolute and relative number of BG system results within error zones A-E of the consensus error grid (CEG).¹⁰

Measurement precision—Primary endpoint: CV as a measure for the variation of BG system results around the clamp levels; secondary endpoints: precision absolute relative deviation (PARD) as a measure of variability between 2 meters of the same system.¹¹

Measurement bias—Primary endpoint: Bland-Altman plots with relative bias and 95% limits of agreement (LA).

The endpoints were calculated for each clamp level and overall (ie, the 3 clamp levels combined). A comparison addressing the potential differences of MARDs and PARDs respectively was done for BG systems tested and assessed using mixed effect linear models with BG systems as fixed effects and subjects as random effects. To account for the multiple comparisons, the significance level was adjusted to .005 for each individual difference. Within the model the least square means, the differences between least square means of MARD of BG systems and corresponding 2-sided 95% parametric confidence interval were calculated. To calculate the primary precision endpoint (CV), the systems’ plasma glucose readings were normalized to the clamp level by correcting any deviation of the laboratory plasma values from the clamp level:

$$\mathrm{PGcorr}=\mathrm{PGmeas}+(\mathrm{clamp\; level}-\mathrm{PGref})$$

Where PGcorr is the normalized BG systems result, PGmeas is the originally measured BG systems result, clamp level is the target plasma glucose concentration and PGref is the laboratory plasma glucose concentration measured.

This normalization reduces glucose variability introduced by the clamp methodology with deviations around the clamp level (aim ≤ ± 10%). The normalization procedure does not correct for the measurement variability introduced by the laboratory measurement (≤2.5 mg/dL or ≤2.5%, whichever is larger).

Results

The data set of 1 subject was excluded from analysis after repeated difficulties to obtain sufficient amounts of blood from the capillary blood sampling. Data of 15 subjects were analyzed, providing up to 342 paired system-laboratory data points per BG system.

Laboratory Glucose Measurements

For each of the 15 subjects, glucose was measured twice and in duplicate with the YSI laboratory glucose analyzer at 30 sample points, yielding a total of 450 comparator data points. A total of 26 laboratory BG measurements (5.8%) were excluded from the analysis because either the drift between the first and second laboratory measurement was too large or the second confirmative laboratory BG measurement could not be performed due to insufficient plasma sample.

Glucose Clamp Characteristics

The total duration of each clamp experiment was between 5 and 6 hours (each of the 1-hour clamp periods was preceded by an approximate 45-minute BG stabilization period). Clamp quality assessed by control deviation and variability (CV) was 2.1 mg/dL (6.8%), 2.9 mg/dL (5.4%), and 4.6 mg/dL (3.7%) for the 60-100 and 200 mg/dL clamp periods respectively.

BG System—Accuracy

The measurement accuracy, expressed by the primary accuracy endpoint (MARD) was similar at each clamp level and overall for the BG systems 1-3 (Table 1). These 3 BG systems showed significantly higher accuracy overall compared to the 2 other systems (P < .005). Of these 2 systems, system 4 was more accurate than system 5 (P < .005). All BG systems showed the lowest accuracy (ie, the highest MARD) at the lowest clamp level. Mean absolute deviation at this 60 mg/dL clamp level was 3.8, 4.1, 4.1, 5.9, and 8.4 mg/dL for systems 1 to 5 respectively. Furthermore, no significant differences were found between 2 meters of the same BG system.

Table 1.

Data for the Primary Accuracy Endpoint MARD for Each Blood Glucose Target Level and Overall.

	BGStar	iBGStar	ACCU-Chek Aviva Nano	Freestyle InsuLinx	OneTouch Verio IQ
Measurement accuracy—MARD in %—for meters 1 and 2 combined (meter 1 / meter 2)
60 (mg/dL)	6.2*†	6.6*†	6.6*†	9.5*	13.6
	(6.2 / 6.2)	(7.3 / 6.0)	(7.7 / 5.6)	(10.3 / 8.7)	(13.0 / 14.2)
100	4.6*†	4.6*†	5.1*†	6.8*	10.6
	(4.3 / 4.9)	(4.2 / 4.9)	(5.6 / 4.6)	(7.1 / 6.5)	(10.8 / 10.5)
200	4.5*†	4.7*†	4.3*†	6.8	6.7
	(4.4 / 4.7)	(4.8 / 4.6)	(4.5 / 4.2)	(7.2 / 6.4)	(7.3 / 6.2)
Overall	5.1*†	5.3*†	5.4*†	7.7*	10.3
	(4.9 / 5.3)	(5.5 / 5.2)	(5.9 / 4.8)	(8.2 / 7.2)	(10.3 / 10.2)

Mean result for meters 1 and 2 combined is shown with the MARD for each system’s BG meter separately in parentheses. The blood glucose (BG) systems are listed in order of increasing overall MARD (meters 1 and 2 combined). MARD, mean absolute relative deviation. *P < .005 vs OneTouch Verio IQ; ^†P < .005 vs Freestyle InsuLinx.

Results for the secondary accuracy endpoints are presented in Table 2 and partly reflect the result of the primary analysis. The 3 systems with highest accuracy continued to show high accuracy with a large percentage (98.2-99.1%) of results within the accuracy limits specified in the ISO 15197:2013 guidelines. System 4 performed equally well with 98.2% of measurements within the specified limits for minimum accuracy, whereas system 5 did not have at least 95% of all readings within those limits. Despite the significant differences between systems in the primary endpoint (MARD), all systems showed high clinical accuracy with 100% results within Zone A of the CEG.

Table 2.

Secondary Endpoints for Accuracy.

	BGStar	iBGStar	ACCU-Chek Aviva Nano	Freestyle InsuLinx	OneTouch Verio IQ
Number and percentage of results within 15/15 limits* for meters 1 and 2 combined
Overall	331/334	330/336	337/342	334/340 (98.2%)	305/330
	(99.1%)	(98.2%)	(98.5%)		(92.4%)
Consensus error grid—percentage of results within clinically accurate zone for meters 1 and 2 combined
Zone A	100	100	100	100	100

^*Absolute and relative number of BG system results within ± 15 mg/dL of laboratory glucose concentrations <100 mg/dL and within ± 15% of laboratory glucose concentrations ≥100 mg/dL according to ISO 15197:2013.

BG System—Precision

With CVs in the range of 5.8-6.8% precision was overall similar for all BG systems (Table 3).

Table 3.

Precision Endpoints.

	BGStar	iBGStar	ACCU-Chek Aviva Nano	Freestyle InsuLinx	OneTouch Verio IQ
Measurement precision—CV in % for meters 1 and 2 combined
60 (mg/dL)	7.9	8.8	7.6	7.8	6.6
100	5.4	5.9	6.3	5.2	5.4
200	5.8	5.9	5.7	5.6	5.6
Overall	6.4	6.8	6.5	6.2	5.8
Within-system measurement precision—PARD in %
Overall	4.7	5.1	4.9	3.4	4.8

PARD, precision absolute relative deviation.

Results for the secondary precision endpoint, PARD, are also presented in Table 3. Across all clamp levels, the tested BG systems had mean PARD values <5.2% (system 4 had the lowest PARD with 3.4%). PARD was slightly lower at the hypoglycemic level than at the eu- or hyperglycemic clamp level. Differences between systems were generally small and not statistically significant.

BG System—Trueness

Results of the Bland-Altman Analysis displaying the measurement bias together with the 95% LA are presented in Figure 2. The relative bias for each of the clamp levels and overall is provided by Table 4. Overall, across clamp levels, systems 1-3 had similar biases with comparable 95% LA. One system (system 5) showed the highest measurement bias, overestimating plasma glucose measurements on average (positive bias), while 1 other system (system 4) underestimated plasma glucose concentrations on average (negative bias). The LA range was overall similar for all systems (between 24.3 and 26.7%), another indication that the systems did not differ much from one another with respect to precision performance.

Figure 2.

Bland-Altman plots including the measurement bias and 95% limits of agreements (LA) for the overall result. Red and blue squares indicate results from the 2 separate BG meters. The BG average is the mean of the laboratory and BG meter measurement value.

Table 4.

Trueness, Calculated as Measurement Bias for Each BG System and Clamp Level.

	BGStar	iBGStar	ACCU-Chek Aviva Nano	Freestyle InsuLinx	OneTouch Verio IQ
Trueness—measurement bias in % for meters 1 and 2 combined
60 (mg/dL)	–1.9	0.4	4.3	–8.9	11.9
100	–1.6	0.3	0.9	–6.5	9.8
200	0.8	2.3	–1.4	–6.1	5.3
Overall	–0.9	1.0	1.3	–7.2	8.9

Discussion

Accurate BG measurements are supportive for a successful diabetes management. Even though the ISO 15197:2013 guideline describes the performance criteria that BG systems need to fulfill for market approval, postsurveillance programs with alternative test protocols have been demanded in the light of unexpected low performance of marketed systems in daily practice.¹² The advantage of the presented glucose clamp approach is that it allows a simultaneous quantitative and reproducible evaluation of measurement accuracy, precision and bias of BG systems over the clinically relevant glucose range with a relatively small number of volunteers. In Table 5 we listed these and further advantages as well as limitations of the presented clamp approach for evaluating the performance of a BG system.

Table 5.

Advantages and Limitations of the Presented Glucose Clamp Methodology to Assess Performance of BG Systems.

Use of the glucose clamp to assess performance of BG systems
Advantages
• Glucose concentrations within the clinical meaningful range (40-400 mg/dL) can be achieved in a controlled manner without the need for diluting or spiking blood samples.
• Stable blood glucose in individual can be established repeatedly at target concentrations for a longer period of time, which allows a clinical evaluation of a BG system’s precision.
• Accuracy, precision, and trueness can be evaluated at the same time in a single clinical experiment.
• A relatively small number of volunteers is only needed with a repeated sampling protocol to obtain enough sample points for a robust analysis. In the current performance evaluation of 5 BG systems, 180 sample points were generated for each BG meter for a single strip assessment, which is considerably greater than the approximate 100 sample points generated with conventional approaches.
Limitations
• Automated glucose clamp devices are available to a few specialized research centers only.
• Glucose clamp studies are generally more costly.
• Multiple finger pricks (10 finger pricks at each of the 3 clamp levels) are performed for each volunteer during a short period of time.

This evaluation showed that glucose measurements made with 3 BG systems (systems 1-3) were overall more accurate (= lower MARD) than those made with 2 others. The random error (precision) of the tested BG systems was comparable, but a lower systematic error (bias) for systems 1 to 3 gives these a more accurate performance at low, normal and high BG levels.

Despite having a higher MARD, 1 BG system (system 4) had nearly all measurements within the in the ISO15197:2013 guideline defined ranges for accuracy and all its measurements were in Zone A of the Error Grid. These results indicate that the higher systematic error observed with this system did not impact its performance within the specified accuracy limits and results can be considered clinically accurate. With 1 other BG system (system 5) measurements were as or even more precise than any other system, but a large systematic error of 8.9% on average resulted in numerically less accurate results compared to the other systems. The clinical accuracy of this system was comparable to the other 4 systems with all measurements within Zone A.

In general, the highest MARD was observed at the 60 mg/dL clamp level. This finding is not surprising if one considers that at low glucose concentrations even a small measurement error may result in a relatively large deviation. For this reason, we (and many studies) also reported measurement accuracy at low glucose concentrations as mean absolute deviation.

The information obtained with this glucose clamp approach about the accuracy of the BG systems tested are comparable with those reported in studies which were performed according to the methodological approach required by the ISO 15197 guidelines.^13-15 A direct comparison of the BGStar system accuracy evaluated by the clamp method and a reduced scale protocol of the ISO 15197:2013 guideline showed good agreement between both test procedures with MARD within 0.8% from another and both methods identified a similar amount of outliers, typically in the range of 1-2% for the standard 15/15 limits of ISO15197:2013 and 9-10% for more restrictive 10/10 limits.¹⁶ These comparisons indicate that the glucose clamp approach can provide robust and valuable data about the accuracy of BG system performance.

The trial also had some limitations: One limitation is that according to the manufacturer of 1 system (system 3), glucose laboratory measurements for this system should be analyzed using a hexokinase method. However, in this trial all laboratory measurements were made with the YSI 2300 StatPlus Analyzer which uses the glucose oxidase method for glucose analysis. The YSI analyzer was chosen in this study as it is used by many BG system manufacturers as laboratory glucose analyzer for the calibration of test strips and widely accepted as laboratory measurement device for BG system performance evaluations. However, a recent comparison of the glucose oxidase and hexokinase method showed an overall systematic bias of approximately 3%, indicating that the choice of comparison method can have a substantial impact on the performance results of a BG system.¹⁷ A further limitation of the study is that only a single lot of test strips was evaluated for each BG meter make. As test strip variability presents a major issue for BG system performance, future evaluations with the clamp approach should include testing multiple test strip lots.

Despite the ability of the clamp method to keep BG in a tight range around the clamp level, the subject’s BG will never be completely stable at the desired target level for 2 reasons: (1) variability is introduced by the clamp control algorithm and (2) variability is introduced by the laboratory measurement error. As explained in the methods section, we reduced the glucose variability introduced by the clamp method by correcting any deviation of the laboratory values from the clamp level. Without this correction, the measurement precision of the BG systems would falsely indicate poorer performance.

As presented in Table 5, a lower sample size with multiple sampling in the same subject can be regarded as an advantage with regard to the costs and speed of clinical trial conduct, but also bears a risk that a systematic error introduced by 1 experiment confounds the overall study results. A separate analysis (not included here) did not show any subject-specific influence on the results.

The hypoglycemic glucose range is generally regarded as the most critical for accurate BG system performance. Low glucose samples for accuracy evaluation are often prepared in a laboratory by dilution, a step which may introduce error in the measurement result. In this study, we controlled the subject’s BG at 60 mg/dL for about 1 hour and capillary samples taken were immediately measured for glucose content. A potential risk of this approach is that hypoglycemia in general is suggested to be related to a higher risk of micro and macrovascular complications.^18,19 When studying subjects under hypoglycemic conditions in a trial, the clamp method with relatively stable glucose concentrations around 60 mg/dL may however be considered one of the safer alternatives compared to those that induce short swings with low blood glucose concentrations in a less controlled manner.

As a final comment to the study design, the aim of this manuscript and study was not to evaluate the performance of many/all BG systems currently available, but to study the glucose clamp approach with a few systems that act as an example. The BG systems evaluated in this study are all considered high-performance systems. In a future study it would be of great interest to also include low-cost BG systems to quantify supposedly greater performance differences between systems with the clamp approach.

Conclusions

In conclusion, the proposed glucose clamp approach allows evaluation of BG system performance under controlled conditions at therapeutically relevant hypo- and hyperglycemic levels. Compared to standard test protocols relatively few volunteers are needed and additional performance parameters can be determined simultaneously. This approach enables determination of the precision with which the BG systems measure glucose; this is not possible with any other approach as it requires glycemia to be kept constant for some period of time. In this trial, the random error of all BG systems tested was comparable; however, the lower systematic error observed with some systems provides these systems with a better accuracy at low, normal and high BG levels.