Tighter Accuracy Standards within Point-of-Care Blood Glucose Monitoring: How Six Commonly Used Systems Compare

Abstract

Introduction

Blood glucose monitoring systems (BGMS) are used in the hospital environment to manage blood glucose levels in patients at the bedside. The International Organization for Standardization (ISO) 15197:2003 standard is currently used by regulatory bodies as a minimum requirement for the performance of BGMS, specific to self-testing. There are calls for the tightening of accuracy requirements and implementation of a standard specifically for point-of-care (POC) BGMS.

Methods

The accuracy of six commonly used BGMS was assessed in a clinical setting, with 108 patients' finger stick capillary samples. Using the accuracy criteria from the existing standard and a range of tightened accuracy criteria, system performance was compared. Other contributors to system performance have been measured, including hematocrit sensitivity and meter error rates encountered in the clinical setting.

Results and Discussion

Five of the six BGMS evaluated met current accuracy criteria within the ISO 15197 standard. Only the Optium Xceed system had >95% of all readings within a tightened criteria of ±12.5% from the reference at glucose levels ≥72 mg/dl (4 mmol/liter) and ±9 mg/dl (0.5 mmol/liter) at glucose levels <72 mg/dl (4 mmol/liter). The Nova StatStrip Xpress had the greatest number of error messages observed; Optium Xceed the least. OneTouch Ultra2, Nova StatStrip Xpress, Accu-Chek Performa, and Contour TS products were all significantly influenced by blood hematocrit levels.

Conclusions

From evidence obtained during this clinical evaluation, the Optium Xceed system is most likely to meet future anticipated accuracy standards for POC BGMS. In this clinical study, the results demonstrated the Optium Xceed product to have the highest level of accuracy, to have the lowest occurrence of error messages, and to be least influenced by blood hematocrit levels.

Introduction

Blood glucose monitoring systems (BGMS) are used in the hospital environment to manage blood glucose levels in patients at the bedside. While laboratory testing of plasma glucose levels is considered most accurate and precise, the nature of the remote testing causes a delay in treatment based on laboratory results. Blood glucose monitoring systems deliver rapid blood glucose measurements on whole blood at the bedside. Therefore, monitoring blood glucose with accurate BGMS is an integral component of effective bedside diabetes management and general glucose homeostasis within a hospital population. International Organization for Standardization (ISO) 15197,¹ the international standard that specifies accuracy requirements of blood glucose monitoring, was published in 2003 and is currently used by regulatory bodies as a minimum requirement for the performance of BGMS used for self-testing.

The minimum acceptable accuracy for results produced by BGMS, according to the ISO 15197 standard, is as follows: 95% of the individual glucose results shall fall within ±15 mg/dl (0.83 mmol/liter) of the results of the manufacturer's measurement procedure at glucose con-centrations <75 mg/dl (<4.2 mmol/liter) and within ±20% at glucose concentrations ≥75 mg/dl (≥4.2 mmol/liter).

The Clinical Laboratory Standards Institute (CLSI) has an existing guideline document that is currently undergoing revision that specifically concerns point-of-care (POC) BGMS and includes accuracy evaluation.² The recommendation in this guideline is currently the same as the requirement in the ISO 15197 standard.

There are proposals from professional and regulatory bodies not only to tighten the 9-year-old ISO 15197 standard, but also to implement a subject-appropriate standard specifically for BGMS being used in the hospital (POC) environment^3,4 that must be adhered to. There has also been discussion of implementation of a secondary criteria for results that fall outside of the specification and that these should be reported to safeguard the patient population against clinically incorrect treatment.⁵ When the new CLSI guideline is approved, it is expected to have tighter accuracy criteria and to be recognized and enforced by some regulatory bodies, specifically for POC testing.

Accuracy of six BGMS have been evaluated and compared using existing and tightened accuracy criteria that are anticipated with the pending updates to ISO 15197 and CLSI C30-A2.

System performance cannot be assessed by adherence to accuracy criteria alone. Other factors that affect the performance of BGMS as a whole must be considered. Alternative methods were used to compare the system's performance. Although infrequently, error messages occur and glucose meters do not provide a glucose result, and clinically, the “occurrence can be devastating,” as reported elsewhere.⁶ Financial implications of these error rates should also be considered.

Method

The purpose of this evaluation was to compare six BGMS in a clinical setting with patient finger stick samples, as is the intended use of the systems, where day-to-day differences would be included. The systems tested can be found in Table 1.

Table 1.

Systems Evaluated in This Study and Operating Ranges Defined by Manufacturer

System	Manufacturer	Assay range	Hematocrit range	Sample types	Methodology
Optium Xceed	Abbott Diabetes Care Ltd., Witney, United Kingdom	20-500 mg/dl	20-70%	Capillary, arterial, venous, neonatal	Glucose dehydrogenase nicotinamide adenine dinucleotide amperometric
Accu-Chek Performa	Roche Diagnostics, Mannheim, Germany	10-600 mg/dl	10-65%	Capillary, arterial, venous, neonatal	Quinoprotien glucose dehydrogenase pyrroloquinoline quinone Amperometric
OneTouch Ultra2	Lifescan Inc., Milpitas, CA	20-600 mg/dl	30-55%	Capillary	Glucose oxidase amperometric
Contour TS	Bayer Healthcare, Mishawaka, IN	10-600 mg/dl	0-70%	Capillary	Glucose dehydrogenase flavin-adenine dinucleotide amperometric
Nova StatStrip Xpress	Nova Biomedical, Waltham, MA	10-600 mg/dl	20-65%	Capillary, arterial, venous, neonatal	Glucose oxidase and glucose dehydrogenase amperometric
Nova Max	Nova Biomedical, Waltham, MA	20-600 mg/dl	25-60%	Capillary	Glucose oxidase amperometric

The Optium Xceed, Accu-Chek Performa, and Nova StatStrip Xpress are products developed for hospital use. Other products have also been indicated for health care professional (HCP) use but have a narrower applicability in a hospital environment (i.e., capillary blood only, narrower hematocrit range; see Table 1).

The study took place in a diabetes outpatient clinic at the Royal South Hants Hospital, Southampton. The study was performed according to a clinical study protocol in compliance with the International Conference on Harmonisation Guidelines for Good Clinical Practise⁷ and with ethics committee approval. Patient consent was obtained, and subjects were eligible for enrollment regardless of gender or type of diabetes, with an exclusion criteria for those under 18 years of age.

A YSI 2300 STAT Plus glucose analyzer (YSI Life Sciences, Yellow Springs, OH) served as the comparative reference method in the study, and its accuracy was validated with NERL glucose standards, traceable to the National Institute of Standards and Technology Standard Reference Material SRM917c. The YSI whole blood glucose result was converted to provide plasma equivalent values (multiplied by 1.12), and this result was used to compare with the test strip results. According to the manufacturers' device labeling, all BGMS in this study were calibrated to a plasma reference.

All systems and supplies were stored, handled, and operated according to the manufacturers' instructions. One randomly selected test strip lot was evaluated with each of the BGMS, and test order was randomized on each day of testing. A trained operator tested the subjects' fingertip blood in duplicate for each system. Immediately after applying blood to the six systems, blood was collected from the same finger stick into a heparin tube for testing in duplicate on the YSI analyzer. The protocol specifies that the YSI test must be completed within 15 min of the first meter test and that the duplicate YSI results must be within ±4 mg/dl (±0.2 mmol/liter) of each other. Duplicate hemoglobin tests were performed on the HemoCue analyzer (Angleholm, Sweden) to determine the hemoglobin concentration for each capillary whole blood sample collected. The hemoglobin concentration was used to calculate the hematocrit level for each sample.

Data analysis was performed using statistical package SAS.⁸ Accuracy was evaluated using the accuracy criteria within the current ISO 15197 standard. (Although this study was not performed in strict compliance with the methodology outlined in ISO 15197:2003, the methods for system accuracy data analysis and assessment have been followed. The results of this analysis are used to compare accuracy of the BGMS used in the study rather than for direct assessment of whether the BGMS meet current accuracy requirements. Deviations from ISO 15197:2003 test methodology were as follows: (1) There was no artificial manipulation of samples to achieve extremely high and low glucose concentrations, hence the distribution of sample glucose concentrations specified in ISO 15197:2003 was not achieved. (2) While at least 200 test strips were used for each of the BGMS, these were not sampled from at least 10 vials or packages.)

Because the exact details of the proposed tightened ISO 15197 and CLSI criteria are unknown at present, a range of tighter criteria has been used to assess system accuracy by calculating the percentage of meter results within ±12.5% and ±15% of the reference value for glucose concentrations ≥100 mg/dl (5.6 mmol/liter) and within ±12 and ±15 mg/dl (0.67 and 0.83 mmol/liter) of the reference value for glucose concentrations <100 mg/dl (5.6 mmol/liter). Accuracy was also evaluated at an alternative accuracy threshold of 72 mg/dl by calculating the percentage of results that fall within ±12.5% from the reference value for glucose concentrations ≥72 mg/dl (4.0 mmol/liter) and within ±9 mg/dl (0.50 mmol/liter) at glucose concentrations <72 mg/dl (4.0 mmol/liter).

Alternative methods were also used to compare the systems [mean absolute percent bias (MAPB) from reference, mean absolute percent residual (MAPR)] as well as observing each system's hematocrit sensitivity and meter error rates encountered in the clinical setting.

Results

A total of 108 diabetes subjects participated in this study over 24 days between December 7, 2010, and February 15, 2011. Samples from 15 subjects were not evaluated on Nova StatStrip Xpress because of limited strip supplies. Parallel analysis has been completed on a data set with these samples removed for all systems to allow a side-by-side comparison of product performance on the same samples (and the conclusions reported here for Nova StatStrip Xpress were not affected). A further sample was removed from the analysis of the OneTouch Ultra2 system because the hematocrit level was below the operating range of that system.

Based on the YSI results, blood glucose concentrations of the subjects ranged from 49.4 mg/dl (2.74 mmol/liter) to 443.9 mg/dl (24.6 mmol/liter), mean of 187.4 mg/dl (10.4 mmol/liter). Hematocrit levels ranged from 26% to 52% (mean of 43%).

System Accuracy Analysis

Each system's accuracy performance at differing accuracy intervals is summarized in Table 2.

Table 2.

Blood Glucose Monitoring System Accuracy Results

Accuracy thresholda	75 mg/dl(4.2 mmol/liter)	100 mg/dl(5.6 mmol/liter)	100 mg/dl(5.6 mmol/liter)	72 mg/dl(4 mmol/liter)	Secondary criteriaoutside of ±15 mg/dl(0.83 mmol/liter)/20%c
Accuracy criteriabelow / abovethe threshold	Within ±15 mg/dl (0.83 mmol/liter)/20%b	Within ±15 mg/dl (0.83 mmol/liter)/15%	Within ±12 mg/dl (0.67 mmol/liter)/12.5%	Within ±9 mg/dl (0.50 mmol/liter)/12.5%
Optium Xceed	216/216 (100%)	216/216 (100%)	211/216 (98%)	210/216 (97%)	0/216 (0.0%)
Accu-Chek Performa	209/214(98%)	196/214 (92%)d	185/214 (86%)f	185/214 (86%)f	5/214 (2.3%)
Contour TS	212/216 (98%)	209/216 (97%)	202/216 (94%)	201/216 (93%)	4/216 (1.9%)
Nova Max	201/216 (93%)d,e	192/216 (89%)e	179/216 (83%)e	176/216 (81%)e	15/216 (6.9%)
Nova StatStrip Xpress	184/185 (99%)	182/185 (98%)	172/185 (93%)	172/185 (93%)	1/185 (0.5%)
OneTouch Ultra2	207/214 (97%)	197/214 (92%)f	185/214 (86%)f	182/214 (85%)f	7/214 (3.3%)

^aConcentration at move from mg/dl difference to % difference from the reference value. Results are combined for above and below the accuracy threshold stated, as required by ISO 15197.

^bCurrent ISO and CLSI criteria.

^cA secondary criteria anticipated to be implemented to ensure all results are reported; there is expectation that a limit will be set (e.g., no more than x%, unknown at present).

Shaded areas indicate significant difference from

^dOptium Xceed and Nova StatStrip Xpress;

^eOptium Xceed, Nova StatStrip Xpress, and Contour TS; and

^fOptium Xceed only (p < .0034).

Five of the six systems evaluated met the current accuracy criteria of ISO 15197. The Nova Max system had <95% within the acceptance criteria (93%). When tightened to ±15% from the reference (±15 mg/dl at glucose levels <100 mg/dl), only the Optium Xceed (100%), Contour TS (97%), and Nova StatStrip Xpress (98%) had >95% within the criteria. On further tightening of the criteria to ±12.5% from the reference (±12 mg/dl at glucose levels <100 mg/dl) only the Optium Xceed system had >95% of results within the criteria.

Table 2 shows how the systems compare with an accuracy criteria of ≥95% of results within 12.5% from the reference (±9 mg/dl at glucose levels <72 mg/dl). Only the Optium Xceed system had ≥95% of all results within this criteria (97%). In addition, the percentage of results that exceeded the potential secondary criteria of ±20% from the reference [±15 mg/dl (0.83 mmol/liter) at glucose levels <72 mg/dl (4 mmol/liter)] for each system can be found in Table 2. Optium Xceed (0.0%), Nova StatStrip Xpress (0.5%), and Contour TS (1.9%) had the least amount of results outside of the criteria. Accu-Chek Performa, Nova Max, and OneTouch Ultra2 all had greater than 2% of results outside this secondary criteria.

Additional Accuracy Measures

Mean absolute percent bias (also known as mean absolute relative difference) provides a measure of the absolute relative difference between the test strip and reference results (i.e., the magnitude of the difference but not the direction). Mean absolute percent residual provides a measure of the absolute relative difference between the test strip results and the regression line of that data. Figure 1 illustrates the MAPB from the reference (YSI) and the MAPR, with 95% confidence intervals (based on between-sample variability).

Figure 1.

Mean absolute percent bias from the reference (YSI) and MAPRs, with 95% confidence intervals. According to each system's labeling, the reference method for the Optium Xceed, Contour TS, One Touch Ultra2, Nova Max, and Nova StatStrip Xpress systems is the YSI. The Accu-Chek Performa system uses a Hexokinase reference; MAPR has been included here for this instance.

It can be seen in Figure 1 that the MAPB is smaller (i.e., strip result closer to reference value) for Optium Xceed and Nova StatStrip Xpress. The Nova Max system had the highest MAPB, followed by the OneTouch Ultra2 system. There is a similar trend in MAPR, with Optium Xceed exhibiting the lowest and OneTouch Ultra2 and Nova Max the highest, indicating poorer performance. The Accu-Chek Performa system had an improved MAPR because it was calibrated to a hexokinase reference and not to the YSI used in this study. The data discussed are also shown in Table 3, together with traditional summary statistics.

Table 4 summarizes the number of times a meter failed to give a result (i.e., when error messages occurred). The Optium Xceed system gave the least amount of errors messages (0%) and Nova StatStrip Xpress the highest (4.6%).

Figure 2 summarizes the hematocrit performance for each of the systems, using the hematocrit level from each of the capillary finger stick samples.

Figure 2.

Percentage bias (between strip response and plasma equivalent reference) against hematocrit with a regression line (slope%/%) and r² value for each system.

Hematocrit sensitivity is estimated in this study by the gradient (slope %bias/%hematocrit) of the regression line. The Optium Xceed system has the smallest (0.01%/%) and the OneTouch Ultra2 system has the largest (−0.38%/%) hematocrit slope, followed by the Nova StatStrip Xpress (0.35%/%). A one-sample t-test was performed (p values < .05 indicated slopes significantly different to zero, i.e., the results are influenced by hematocrit level) to evaluate the impact of hematocrit levels (range of 26% to 52%). Results found the Optium Xceed (p = .9093) and the Nova Max (p = .2118) systems were not significantly influenced by hematocrit levels. Accu-Chek Performa (p = .0032), Contour TS (p = .0002), Nova StatStrip Xpress (p < .0001), and the OneTouch Ultra2 (p < .0001) systems were all significantly influenced by hematocrit. The hematocrit range observed in this study is narrower than might be observed in certain hospital environments such as intensive care or neonatal wards. This is a direct consequence of the unaltered samples being used in this study.

Conclusion

Monitoring of blood glucose with an accurate device is an integral component of effective diabetes management or general glucose homeostasis within a hospital popula-tion. The ISO 15197 standard,¹ the international standard that specifies accuracy requirements of blood glucose monitoring, was created in 2003. Blood glucose monitoring systems technology has advanced greatly in that time, and the standard is now considered not only out of date for today's level of technology, but also not stringent enough for current hospital demands. A revision of this standard and generation of a standard specifically for POC BGMS, which is likely to include a secondary criteria for results that fall outside of the primary accuracy criteria, is in progress.

In this study, five of the six systems met the current accuracy criteria detailed within ISO 15197:2003 and recommended by CLSI C30-A2: Optium Xceed, Accu-Chek Performa, Contour TS, Nova StatStrip Xpress, and OneTouch Ultra2. In this clinical evaluation, with a tightened accuracy criteria of ±15% from the reference (±15 mg/dl at glucose levels <100 mg/dl), for three of the six systems tested, 95% of results fall within the criteria. These systems were Optium Xceed, Contour TS, and Nova StatStrip Xpress, and from results seen in this study, they are more likely to meet standards in the new version of ISO 15197 if its accuracy criteria are tightened to ±15%/15 mg/dl from the reference.

It is expected that future CLSI guidelines that focus on POC use in a hospital environment will have tighter criteria than the updates to ISO 15197, which is intended for systems used in self-testing. Using the evidence gained in this study, the Optium Xceed system is more likely to meet the new CLSI guideline if its accuracy criteria is tightened to ≥95% of results within ±12.5% from the reference [±9 mg/dl (0.5 mmol/liter) <72 mg/dl (4 mmol/liter)]. In this clinical evaluation, the remaining systems evaluated did not meet this criteria. If secondary criteria are set for results that previously would have gone unreported, data from this study suggest that Optium Xceed, Contour TS, and Nova StatStrip Xpress will fare well in comparison with the remaining systems since all exhibited less than 2% of results outside of the criteria.

When observing MAPB or MAPR, depending on the reference method of the system, the accuracy looks favorable for the Optium Xceed and Nova StatStrip Xpress systems, both with low MAPB and MAPR.

Infrequently, blood glucose meters do not provide a result;⁶ however, in a POC setting, the financial implications of these errors should be considered. While the Optium Xceed system had no error messages reported, all other systems encountered error messages during the study, with Nova StatStrip Xpress exhibiting the highest rate (4.6%). Based on a typical 600-bed hospital, consuming a hypothetical 350,000 strips per annum, an error rate of 4% would equate to a loss of 14,000 test strips per annum. This would have financial implications not only in terms of strip replenishment, but also with valuable time lost for HCPs and nurses.

Hematocrit sensitivity of systems is most accurately demonstrated when using fresh whole blood samples, as is the intended use of the systems. Statistical analysis confirmed that two of the six systems were not influenced by hematocrit, the Optium Xceed and Nova Max systems. All other systems (Nova StatStrip Xpress, OneTouch Ultra2, Accu-Chek Performa, and Contour TS) were significantly influenced by blood hematocrit levels. This level of hematocrit sensitivity was not expected from the systems that have been designed for use in a hospital environment, especially since a hospital population will exhibit a greater range in hematocrit levels, as a direct consequence of either illness or treatment.

In conclusion, during this clinical study, five of the six meters evaluated met the current accuracy criteria detailed within ISO 15197 and CLSI. When comparing the performance of all six systems with tighter accuracy requirements, results from this study suggest the Optium Xceed system is most likely to meet the newly anticipated standards. In this clinical setting with patient samples, results demonstrated the Optium Xceed system to have the highest level of accuracy, to have the lowest occurrence of errors, and to be least influenced by hematocrit.

Glossary

Abbreviations

(BGMS)

blood glucose monitoring systems

(CLSI)

Clinical Laboratory Standards Institute

(HCP)

health care professional

(ISO)

International Organization for Standardization

(MAPB)

mean absolute percent bias

(MAPR)

mean absolute percent residual

(POC)

point-of-care

Funding

This clinical study was funded by Abbott Diabetes Care.

Disclosures

Charlotte Robinson is an employee of Abbott Diabetes Care.