Fundamental Importance of Reference Glucose Analyzer Accuracy for Evaluating the Performance of Blood Glucose Monitoring Systems (BGMSs)

Abstract

Background:

As blood glucose monitoring system (BGMS) accuracy is based on comparison of BGMS and laboratory reference glucose analyzer results, reference instrument accuracy is important to discriminate small differences between BGMS and reference glucose analyzer results. Here, we demonstrate the important role of reference glucose analyzer accuracy in BGMS accuracy evaluations.

Methods:

Two clinical studies assessed the performance of a new BGMS, using different reference instrument procedures. BGMS and YSI analyzer results were compared for fingertip blood that was obtained by untrained subjects’ self-testing and study staff testing, respectively. YSI analyzer accuracy was monitored using traceable serum controls.

Results:

In study 1 (N = 136), 94.1% of BGMS results were within International Organization for Standardization (ISO) 15197:2013 accuracy criteria; YSI analyzer serum control results showed a negative bias (−0.64% to −2.48%) at the first site and a positive bias (3.36% to 6.91%) at the other site. In study 2 (N = 329), 97.8% of BGMS results were within accuracy criteria; serum controls showed minimal bias (<0.92%) at both sites.

Conclusions:

These findings suggest that the ability to demonstrate that a BGMS meets accuracy guidelines is influenced by reference instrument accuracy.

Blood glucose monitoring system (BGMS) technology advancements have improved the potential accuracy of blood glucose concentration measurements,¹ which is reflected in the increasingly stringent guidelines adopted by regulatory agencies.^2,3 BGMS accuracy is assessed by comparing BGMS results with those from a laboratory glucose analyzer as a reference^2,4; thus, the accuracy of the reference glucose analyzer is important to discriminate small differences between BGMS results and reference glucose analyzer results. Herein, we present data from 2 clinical studies, which assessed the performance of the same BGMS, to illustrate the importance of reference glucose analyzer accuracy in evaluating BGMS performance.

Methods

The performance of the improved CONTOUR^® (Model 7220) BGMS (Ascensia Diabetes Care, Parsippany, NJ) was assessed in 2 clinical studies. The improved CONTOUR BGMS contains an updated algorithm and uses currently available CONTOUR test strips. Each study was performed at 2 study sites; the BGMSs, test strip lots, and study sites were identical in both studies. In each study, untrained subjects performed a fingertip test using the BGMS; in parallel, study staff obtained a fingertip blood sample for measurement using a YSI analyzer (YSI Life Sciences, Inc, Yellow Springs, OH) to obtain the reference value. The accuracy of the YSI analyzer was monitored using National Institute of Standards and Technology (NIST) traceable serum controls.⁵ These controls were tested on the YSI analyzer prior to study initiation and throughout the study.

In study 1, 6 serum controls (blood glucose concentration range, 24-582 mg/dL [1.3-32.3 mmol/L]) were tested in ≥3 runs and monitored to be within ±5 mg/dL (±0.28 mmol/L) or ±5% of the target for controls having blood glucose concentration levels of <100 and ≥100 mg/dL (<5.6 and ≥5.6 mmol/L), respectively. In study 2, 4 serum controls (blood glucose concentration range, 50-395 mg/dL [2.8-21.9 mmol/L]) were tested in ≥6 runs over 3 days and monitored to be within ±3.5 mg/dL (±0.19 mmol/L) or ±3.5% of the target for controls having blood glucose concentration levels of <70 and ≥70 mg/dL (<3.9 and ≥3.9 mmol/L), respectively. Study 1 serum controls with the lowest and highest glucose concentrations were excluded to allow for direct comparisons with YSI analyzer accuracy results from study 2. In both studies, the same 4 controls were tested daily prior to the first plasma sample to assure that the YSI analyzer was performing as expected.

BGMS testing in the 2 studies generally followed the same procedures; differences between the studies were related to YSI analyzer procedures. In summary, the YSI analyzers in study 2 were maintained and monitored to be within tighter agreement than those in study 1, with NIST traceable serum controls throughout the study. The YSI analyzers were calibrated immediately prior to each subject plasma sample in study 2, while the calibration was performed every 4 tests or 15 minutes in study 1. Throughout each test day during study 2, the accuracy of the YSI analyzer was monitored by testing a single control after each subject plasma sample, thus assuring that the performance of the YSI analyzer had not changed. Last, 1 sip of the plasma (or control) was tested in study 2, and 2 sips of each were measured in study 1. Further details of the reference instrument testing procedures in each study are provided below.

In study 1, the YSI analyzers and operators were supplied by the clinical site. Prior to starting the study, 6 traceable serum controls were run multiple times over the course of ≥3 days and each morning of the study before testing the first subject sample. The following established criteria were applied: for control levels <75 mg/dL (levels 1 and 2), the mean result was within ±5 mg/dL of the target levels; for control levels from 75 mg/dL to 500 mg/dL (levels 3, 4, and 5), the mean result was within ±5% of the target levels; for control levels >500 mg/dL (level 6), the mean result was within ±6% of the target levels. For the sample testing sequence, the YSI analyzer was calibrated every 5 sips or every 15 minutes. Each subject plasma sample was run twice (2 sips) to provide 4 values, 2 values from each probe. All 4 values were averaged to provide a single reference glucose value.

In study 2, the YSI analyzers and operators were supplied by Bayer HealthCare, the predecessor-in-interest of Ascensia Diabetes Care. Prior to the study, multiple YSI analyzers were tested with 4 traceable serum controls (levels 2-5) across multiple days and with multiple operators. YSI analyzers that exhibited the least bias relative to the serum target values were selected to be sent to the 2 sites. At each site, prior to starting the study, 4 traceable serum controls were run multiple times over the course of ≥3 days as well as each morning of the study before testing the first subject sample. The serum control range of expected values was the same as used in study 1, that is, within ±5% for levels 3, 4, and 5; the only tightened value was for level 2 to be within ±2.5 mg/dL. The test sample was run once (1 sip) to provide 2 values, 1 value from each of 2 probes. These 2 values were averaged for the final glucose value. For the subject plasma sample testing sequence, the YSI analyzer was calibrated immediately before each subject sample was tested, followed by a serum control sample with a glucose range that aligned with the subject sample blood glucose value. If the serum control value was outside the limits described previously, the YSI analyzer was recalibrated and the entire sequence was repeated for that sample.

Performance of the BGMS was assessed using International Organization for Standardization (ISO) 15197:2013 Section 8 accuracy criteria²: ≥95% of results shall fall within ±15 mg/dL (±0.8 mmol/L) or ±15% of the blood glucose concentration reference results for samples having YSI analyzer blood glucose concentration levels of <100 or ≥100 mg/dL (<5.6 and ≥5.6 mmol/L), respectively, for people with diabetes.

Results

Study 1

A total of 136 subjects enrolled in the study, all with type 1 or type 2 diabetes. Serum control results demonstrated that the YSI analyzers at both study sites met the manufacturer’s standards; however, a positive bias for the YSI analyzer was observed at site 1 (3.36%-6.91% deviation from the NIST serum levels) and a negative bias was observed at site 2 (−0.64% to −2.48% deviation from the NIST serum levels; Figure 1A).

Figure 1.

YSI analyzer serum control results for study 1 (A) and study 2 (B). A single result outside the control range was permissible, as long as the average result was within limits.

Evaluation of subject-obtained capillary fingertip results showed that 94.07% (127/135) of results overall were within the specified accuracy criteria. At site 1, 92.75% (64/69; confidence limits, 84.13%-96.87%) of results met these criteria; at site 2, 95.46% (63/66; confidence limits, 87.47%-98.44%) of results met the same criteria.

Study 2

A total of 372 subjects were enrolled, 329 of whom had type 1 or 2 diabetes. The YSI analyzers at both study sites met the manufacturer’s standards, as demonstrated by YSI analyzer serum control results; however, in contrast to study 1, minimal bias for the YSI analyzer (<0.92% deviation from the NIST serum level) was observed at both study sites in study 2 (Figure 1B).

For subjects with diabetes, 97.84% (317/324) of subject-obtained capillary fingertip results overall met the specified accuracy criteria. These criteria were met by 96.86% (154/159; confidence limits, 92.85%-98.65%) of results at site 1 and 98.79% (163/165; confidence limits, 95.69%-99.67%) of results at site 2.

Discussion

Results from these analyses demonstrate that the ability to obtain a trustworthy assessment of BGMS performance depends on the accuracy of the reference glucose analyzer against which the BGMS is compared. Inasmuch as BGMS technology advancements have improved potential BGMS accuracy, systematic variances observed with some reference glucose analyzers^6,7 suggest that the accuracies for these instruments must be closely monitored when evaluating narrower acceptable margins of error applicable to a new BGMS.

When we evaluated the performance of the same BGMS in 2 different clinical studies that only differed by the reference instrument procedures used (ie, different YSI analyzers and tracking methods), the conclusions varied. While the YSI analyzers in both studies met the manufacturer’s standards, YSI analyzer results were biased relative to serum controls to a greater extent in study 1 than in study 2.

The differences in conclusions between these studies suggest that the accuracy of the reference glucose analyzer may impact the reported accuracy of a BGMS. As BGMS accuracy guidelines are tightened, these situations may become more common. In some cases, reference glucose analyzer error could have a negative impact on the perceived performance of a BGMS, which may have been the case in study 1. In other instances, reference glucose analyzer error could cause a BGMS to surpass accuracy criteria that it would not have met otherwise (eg, if both the BGMS and reference glucose analyzer have a negative bias, the BGMS could appear to be more accurate than it actually is and therefore meet accuracy criteria despite being biased and possibly because of being biased). Reference glucose analyzer errors that previously may have been considered acceptable may now be consequential as (1) BGMS accuracy improves and (2) BGMS guidelines become more stringent because even “small” biases in reference glucose analyzer measurements could influence whether a BGMS is approved for use in people with diabetes.

Ascensia Diabetes Care has consequently revised its reference glucose analyzer procedures, with a new YSI analyzer range of expected blood glucose values. The ranges for the blood glucose values set in Ascensia Research and Development are for levels 1 and 2 to be within ±2.5 mg/dL and for levels 3 to 6 to be within ±5% of the mean value obtained from the hexokinase assay. The ±5% incorporates the observed values from the YSI analyzer (±2 standard deviations) along with a known bias (at higher blood glucose values) between the hexokinase assay and the YSI analyzer.

Conclusions

As a result of these analyses, Ascensia Diabetes Care has revised its reference glucose analyzer procedures. With stricter BGMS accuracy guidelines, it is more important for BGMS manufacturers to monitor the accuracy of reference measurements when assessing the performance of new BGMSs, particularly because the results of these studies determine whether a BGMS obtains regulatory approval.