System Accuracy Evaluation of 43 Blood Glucose Monitoring Systems for Self-Monitoring of Blood Glucose according to DIN EN ISO 15197

Abstract

Background

The accuracy of systems for self-monitoring of blood glucose is important, as reliable measurement results are a prerequisite for therapeutic decisions.

Methods

This system accuracy evaluation study was performed according to DIN EN ISO 15197:2003 for 43 Conformité Européenne (CE)-labeled blood glucose (BG) monitoring systems. Measurement results of each system were compared with results of the designated comparison method (manufacturer’s measurement procedure): glucose oxidase method (YSI 2300 glucose analyzer) or hexokinase method (Hitachi 917/ cobas 501).

Results

Complete assessment according to the International Organization for Standardization (ISO) standard was performed for 34 out of 43 systems, and 27 (79.4%) meet the requirements of the standard, i.e., ≥95% of their results showed at least the minimum acceptable accuracy. For 9 of the 43 systems, complete accuracy assessment was not performed due to an oxygen sensitivity (manufacturer’s labeling). The bias (according to Bland and Altman) of all 43 evaluated systems ranged from -14.1% to +12.4%.

Conclusions

From the 34 systems completely assessed, 7 systems did not fulfill the minimal accuracy requirements of the ISO standard. The CE mark apparently does not guarantee that all BG systems provide accuracy according to the standard. Because inaccurate systems bear the risk of false therapeutic decisions, regular and standardized evaluation of BG meters and test strips should be requested in order to ensure adherence to quality standards.

Introduction

Self-monitoring of blood glucose (SMBG) with glucose monitoring systems is widely recognized as an integral component of adequate diabetes management that enables patients to control their blood glucose (BG) levels effectively.^1–4 Several studies have demonstrated that tight BG control is essential for diabetes patients to avoid late complications.^1,5 The clinical benefits of SMBG in diabetes patients are widely accepted, and today, SMBG is recommended for all people with diabetes, particularly for adjustment of insulin in patients with multiple daily injections.^2,6–8

A multitude of SMBG systems is available on the market. An increasing number of new systems have been introduced. Physicians and patients are looking for guidance to choose between systems with different price ranges, including well-established systems as well as completely new systems, e.g., systems providing new technologies.^9–13

The accuracy of a SMBG measurement is imperative for the reliability of results and, finally, the medical outcome in diabetes therapy. DIN EN ISO 15197:2003¹⁴ is an internationally accepted standard defining performance requirements for BG systems for SMBG, e.g., concerning accuracy. The standard states that ≥95% of the BG system measurement results shall fall within ±15 mg/dl of the results of the manufacturer’s measurement procedure at glucose concentrations <75 mg/dl and within ±20% at glucose concentrations ≥75 mg/dl. A revised version of the International Organization for Standardization (ISO) standard, expected to be published in 2012, includes tighter criteria for the minimum accuracy of BG systems.¹⁵ The current draft revision of ISO 15197 states that ≥95% of the system measurement results shall fall within ±15 mg/dl of the results of the manufacturer’s measurement procedure at glucose concentrations <100 mg/dl and within ±15% at glucose concentrations ≥100 mg/dl.

In Europe, manufacturers of BG systems have to provide evidence of conformity with the ISO standard in order to get the Conformité Européenne (CE) mark for their products. However, an evaluation study published in 2010 showed that more than 40% of the systems investigated did not fulfill the minimum accuracy criteria of the ISO standard.¹⁶

The aim of this study was to evaluate measurement quality standard of a broad range of current BG systems available on the market, i.e., all of them are CE marked. In total, 43 BG systems from 19 manufacturers were evaluated according to the accuracy requirements requested by DIN EN ISO 15197:2003.

Materials and Methods

The study was conducted from 2009 to 2011 in compliance with the German Medical Devices Act at the Institut für Diabetes-Technologie GmbH in Ulm, Germany. The study protocol was approved by the Ulm University ethics committee, and the competent authority was notified. Informed consent forms were signed by all participants. The procedure to evaluate system accuracy applied in this study is described in detail in DIN EN ISO 15197:2003.¹⁴ Deviations from this standard are described here.

Subjects and Test Procedure

Adult patients (≥18 years old) with diabetes type 1 and type 2 as well as subjects without diabetes were included. Exclusion criteria were as follows: pregnancy or lactation period for female subjects, severe acute disease, and severe chronic disease endangering the subject due to the study. Interruption criteria for individual subjects were retraction of written informed consent and incidences or adverse events interfering with the study continuation. At least 100 subjects were included for each system tested. For each BG system (BG meter with test strips), two individual BG meters were used. The BG meters were replaced in case of failure. For each system, measurements were performed on at least 10 days. Suitable control procedures were performed daily prior to the test procedure. The tests were performed by clinical personnel well trained to the limitations of the BG system, the manufacturer’s device labeling, the safety practices, and the test protocol. The tests were performed in a laboratory setting with controlled room temperature (23 ± 5 °C) and humidity (according to the manufacturers’ specifications).

Self-Monitoring of Blood Glucose Systems

The 43 evaluated BG systems are listed in Table 1. These BG systems have been selected in order to give a comprehensive overview of systems with the CE label. The following two criteria have been defined as prerequisites: the product must have a CE mark and BG meter and test strips must have been available in the required quantities for an ISO assessment. In addition, the investigator tried to ensure a representative overview from an international perspective. Selection also should give insights about accuracy of BG systems from established manufacturers as well as from new providers and BG systems for which new technologies are claimed. This is why products from different countries fulfilling these prerequisites have been considered. None of the 43 BG systems were already assessed in the study published in 2010.¹⁶ The Accu-Chek® Active system was an older version, and for Accu-Chek Aviva and FreeStyle Lite®, test strips with another chemistry have been used. Inclusion criteria for the evaluation of BG systems were as follows: Only systems labeled for SMBG were included. For each system, one test strip lot was evaluated. Test strips were taken from at least seven different packages or vials. The packages or vials were changed after approximately 10 subjects. For the strip-free Accu-Chek Mobile system, which incorporates 50 tests in a cassette, a new test cassette was used for each subject.

Table 1.

Forty-Three Blood Glucose Systems Evaluated (Listed Alphabetically)^a

BG system	Manufacturer	Reference method	Calibration	Test strip enzyme	Study date	Test strip lot	Expiry date (test strip)	O2 dependency in labeling
Accu-Chek Active	Roche Diagnostics GmbH, Germany	HK	Plasma	GDH	04/2011–05/2011	23433231	05/2012	No
Accu-Chek Aviva	Roche Diagnostics GmbH, Germany	HK	Plasma	GDH	11/2010–02/2011	490018	11/2011	No
Accu-Chek Aviva Nano	Roche Diagnostics GmbH, Germany	HK	Plasma	GDH	04/2011–05/2011	490068	03/2012	No
Accu-Chek Compact Plus	Roche Diagnostics GmbH, Germany	HK	Plasma	GDH	01/2009	20684541	10/2009	No
Accu-Chek Go	Roche Diagnostics GmbH, Germany	HK	Whole blood	GDH	01/2009	22472532	11/2009	No
Accu-Chek Mobileb	Roche Diagnostics GmbH, Germany	HK	Plasma	GDH	11/2010–02/2011	27705231	09/2011	No
Accu-Chek Mobileb	Roche Diagnostics GmbH, Germany	HK	Plasma	GDH	05/2011–06/2011	27802741	09/2012	No
Accu-Chek Performab	Roche Diagnostics, GmbH, Germany	HK	Plasma	GDH	01/2009	320137	12/2009	No
Accu-Chek Performab	Roche Diagnostics GmbH, Germany	HK	Plasma	GDH	02/2011–03/2011	470049	12/2011	No
Accu-Chek Performa Nano	Roche Diagnostics GmbH, Germany	HK	Plasma	GDH	05/2011–06/2011	470137	05/2012	No
Bayer Contour® usb	Bayer Consumer Care AG, Switzerland	GOx	Whole blood	GDH	03/2010	9FC3A02	06/2011	No
Beurer GL32	Beurer GmbH, Germany	GOx	Plasma	GOx	04/2011–05/2011	V43/4	10/2012	No
Beurer GL40	Beurer GmbH, Germany	GOx	Plasma	GOx	03/2010	U13/001	12/2010	No
BGStar™	AgaMatrix Inc., USA	GOx	Plasma	GOx	07/2011–08/2011	HD14WB26C	10/2012	Yes
Biocheck TD-4225	TaiDoc Technology Corporation, Taiwan	GOx	Plasma	GOx	07/2009–10/2009	TD08J123-B06	04/2010	No
Element™	Infopia Co. Ltd., Korea	HK	Plasma	GOx	06/2010–07/2010	S4ND09L14	12/2011	Yes
FreeStyle Freedom Lite®	Abbott Diabetes Care Inc., USA	GOx	Plasma	GDH	05/2011–06/2011	1067212	12/2011	No
FreeStyle Lite	Abbott Diabetes Care Inc., USA	GOx	Plasma	GDH	06/2010–07/2010	1055813	08/2011	No
Futura Monometer®	TaiDoc Technology Corporation, Taiwan	GOx	Whole blood	GOx	06/2010–07/2010	TD09C117-B02	12/2010	No
GlucoCheck Classic	TaiDoc Technology Corporation, Taiwan	GOxc	Plasma	GOx	05/2011–06/2011	TD10C129-B04	12/2011	No
GlucoCheck Comfort	aktivmed GmbH, Germany	HK	Plasma	GOx	11/2010–02/2011	S3FC10E27	05/2012	Yes
GlucoCheck XL	aktivmed GmbH, Germany	GOx	Plasma	GDH	04/2011–05/2011	TD10J114-B0E	04/2012	No
GlucoHexal® IId	Med-WatchDoc GmbH & Co. KG, Germany	GOx	Plasma	GDH	03/2010	E09D012577	11/2010	No
GlucoRx (TD-4230)	TaiDoc Technology Corporation, Taiwan	GOxc	Plasmae	GOx	02/2011–04/2011	TD09L109-B85	09/2011	No
GlucoSmart® Swing	MSP bodmann GmbH, Germany	GOx	Whole blood	GDH	03/2010	A09F05222	01/2011	No
GlucoTel	BodyTel Europe GmbH, Germany	GOx	Plasma	GDH	07/2009–10/2009	7121403	12/2009	No
Gluco-test Plus+ TD-4230	TaiDoc Technology Corporation, Taiwan	GOx	Plasma	GOx	07/2009–10/2009	TD08E114-E06	11/2009	No
iBGStar™	AgaMatrix Inc., USA	GOx	Plasma	GOx	07/2011–08/2011	HS01WZ34B	07/2012	Yes
iDia™	IME-DC GmbH, Germany	GOxc	Plasma	GDH	09/2010–10/2010	GS005A	12/2011	No
IME-DC Fidelity	IME-DC GmbH, Germany	GOxc	Whole bloode	GOx	07/2009–10/2009	DS159A2	02/2010	No
iXell	Genexo Sp, zo.o., Poland	GOxf	Plasmaf	GOx	05/2011–06/2011	TD10K112-B0C	08/2012	No
iXell OLED	Genexo Sp, zo.o., Poland	GOxf	Plasmaf	GOx	04/2011–05/2011	TD10K112-B0C	08/2012	No
microdot®+	Cambridge Sensors Limited, UK	GOx	Plasma	GDH	07/2011–08/2011	0060802	06/2012	No
Omnitest® 3	B. Braun Melsungen AG, Germany	GOx	Plasma	GOx	07/2011–08/2011	G5KI14	09/2012	Yes
OneTouch® Verio™	LifeScan Inc., USA	GOx	Plasma	GDH	03/2010	2993603	12/2010	No
OneTouch Verio Pro	LifeScan Europe, Switzerland	GOx	Plasma	GDH	02/2011–03/2011	3078405	01/2012	No
OneTouch VITA™	LifeScan Inc., USA	GOx	Plasma	GOx	01/2009	2841992	12/2009	Yes
Pura™	Bionime Corporation, Taiwan	HK	Plasma	GOx	03/2010	1196232	05/2011	No
SeniorLine GM210	Bionime Corporation, Taiwan	GOx	Whole blood	GOx	07/2009–10/2009	1186235	05/2010	No
smartLAB® genie	HMM Diagnostics GmbH, Germany	GOxc	Plasmae	GOx	07/2009–10/2009	022081101	11/2009	Yes
smartLAB global	HMM Diagnostics GmbH, Germany	GOxc	Plasmae	GOx	06/2010–07/2010	045100303	10/2011	Yes
WaveSense™ Jazz™	AgaMatrix Inc., USA	GOx	Plasma	GOx	11/2010–02/2011	HJ29WT32C	11/2011	Yes
Wellion® CALLA light	MED TRUST Handelsges.m.b.H., Austria	GOx	Plasma	GOx	07/2011–08/2011	TJS002L	11/2011	No

^aReference methods (GOx or HK), calibration (plasma or whole blood), and test strip enzyme (glucose dehydrogenase or GOx) according to the manufacturer’s labeling. GDH, glucose dehydrogenase.

^bAccu-Chek Mobile and Accu-Chek Performa were both tested with different test strip chemistries. The test strip chemistry was either maltose dependent (test strips evaluated first) or maltose independent (test strips evaluated second).

^cClear information about the reference method was not available in manufacturer’s labeling; requests were made to provide information.

^dGlucoHexal test strip lot was recalled from the market in June 2010, after at least 11 months availability on the market.

^eClear information about the calibration was not available in manufacturer’s labeling; requests were made to provide information.

^fInformation was not available; repeated requests were unanswered at the time of manuscript submission.

Reference Measurement

Reference measurements were performed with the following two different methods for all BG systems: glucose oxidase (GOx) (YSI 2300 STAT Plus™ glucose analyzer, YSI Life Sciences, Yellow Springs, OH; measurements were performed at the study site) and hexokinase (HK) [Hitachi 917 (from January 2009 to August 2010)/cobas® 6000 c501 (since August 2010), Roche Diagnostics GmbH, Mannheim, Germany; measurements were performed at a Deutsche Akkreditierungsstelle-accredited calibration laboratory of Roche Diagnostics GmbH].

The accuracy of the GOx method was verified measuring NERL Glucose Standards (Thermo Fisher Scientific, East Providence, RI), verified against National Institute of Standards and Technology (NIST) (Gaithersburg, MD) reference material. The accuracy of the HK method was verified measuring NIST Standard Reference Material 965a (from January 2009 to February 2011) or 965b (since February 2011). In addition, for both systems, internal and external quality control measurements were performed, as required by the German national standard.¹⁷

The accuracy of the measurement results of each BG system was evaluated in comparison with the results of the reference measurement specified by the manufacturer (manufacturer’s measurement procedure).

The BG meters displayed either whole blood BG values or plasma equivalent BG values in mg/dl or mmol/liter (calibration, Table 1). Reference measurements with the GOx method were performed from capillary whole blood samples; reference measurements with the HK method were performed from hemolyzed and deproteinized whole blood samples. Both reference measurement methods provided whole blood BG values in mg/dl. For plasma-calibrated systems, whole blood BG values were converted to plasma equivalent values, and these results were used for comparison with the BG system results. Measurement results from the GOx method were converted from whole blood BG values to plasma equivalent BG values as follows: plasma equivalent BG value (in mg/dl) = whole blood BG value (in mg/dl)/[1 - (0.0024 × hematocrit value [in %])].¹⁸ Results from the HK method were converted from whole blood BG values to plasma equivalent BG values as follows: plasma equivalent BG value (in mg/dl) = 1.11 × whole blood BG value (in mg/dl). For the GlucoTel™ system, a conversion factor was described in the manufacturer’s manual that was used: plasma equivalent value (in mg/dl) = 1.12 × whole blood BG value (in mg/dl).

For 8 systems, complete or clear information about the reference measurement procedure and/or the calibration, required for system accuracy evaluation according to DIN EN ISO 15197:2003, were not documented in the manufacturers’ labeling. Even though repeated requests were made, for 2 systems (iXell® and iXell OLED), information about the reference measurement procedure and information about the calibration have not been provided (Table 1). For evaluation of iXell and iXell OLED, we used the GOx method as reference measurement procedure, and we used plasma equivalent values, because most of the available BG systems are plasma calibrated.

Test Protocol

DIN EN ISO 15197:2003 specifies the distribution of the blood samples into different concentration categories. In this evaluation, we used slightly modified limits of these concentration categories, because the limits are not clearly defined and differ between the English and the German version of the standard (Table 2). In deviation to the current standard (but in accordance with the 2011 draft of the new ISO standard), blood samples are distributed into the different concentration categories based on the mean reference results of the manufacturer’s measurement procedure instead of the determined BG values with the systems.

Table 2.

Distribution of Glucose Concentration according to DIN EN ISO 15197 with Slight Modifications

Percentage of samples	Glucose concentration mmol/liter (mg/dl)
5	<2.8 (≈ <50)
15	≥2.8–<4.35 (≈ ≥50–<80)
20	≥4.35–<6.7(≈ ≥80–<120)
30	≥6.7–<11.15 (≈ ≥120–<200)
15	≥11.15–<16.65 (≈ ≥200–<300)
10	≥16.65–<22.2 (≈ ≥300–<400)
5	≥22.2 (≈ ≥400)

Native capillary blood samples were used at BG concentrations of 50 to 400 mg/dl. If sufficient numbers of native samples with BG concentrations <50 mg/dl were not available, additional samples were prepared as follows: the blood samples were collected in lithium heparin tubes, incubated at room temperature to allow for glycolysis, and gently mixed before testing. If sufficient numbers of native samples with BG concentrations ≥400 mg/dl were not available, additional samples were prepared as follows: the blood samples were collected in lithium heparin tubes, supplemented with concentrated glucose solution (40% glucose in 0.9% NaCl), and gently mixed before testing.

At least 100 fresh capillary blood samples from 100 subjects were collected (distribution of BG concentrations as described earlier). For each subject, the hematocrit value was checked to be within 30% and 55%. For determination of the hematocrit, capillary whole blood was collected in heparinized capillaries (double test). After centrifugation, the hematocrit was read on an alignment chart.

Samples were collected from fingertips by skin puncture. The steps of the sample sequence for BG systems were as follows:

1. Sample collection for the two reference measure-ment procedures: (a) a sample (100 µl) for the GOx method was collected using a lithium heparin tube, and the BG concentration was measured in duplicate; (b) a sample (20 µl) for the HK method was hemolyzed and deproteinized in tubes containing 400 µl of 0.33 mmol/liter perchloric acid—these tubes were centrifuged, and the supernatants were transferred to fresh tubes and stored at -20 °C for later triplicate testing.

2. BG measurements with up to three BG systems (meter 1 and meter 2, respectively).

3. Taking of samples for the two reference measurement procedures (sample collection and measurement as described earlier).

Residual blood was wiped off the finger before the sample collection for each reference measurement procedure and before measurement with each BG system. Measurements with meter 1 and meter 2 of the respective system were normally carried out from the same drop of blood, except for systems with test fields (Accu-Chek Active system and Accu-Chek Mobile system), where blood was wiped off before the measurement with each BG meter.

Statistical Analyses

The entire data evaluation was performed at the study site. Data were excluded from statistical analysis if a handling error occurred, no reference value was available, a technical error was documented, the data set was not complete, the hematocrit value was outside the defined range (30% to 55%), the maximum number of samples in a given BG concentration category was already reached, or the drift between the first and second reference measurement was >4 mg/dl at BG concentrations ≤100 mg/dl or >4% at BG concentrations >100 mg/dl. Data of 100 subjects were included in the system accuracy evaluation for each system according to the ISO 15197 standard. Calculations were performed in mmol/liter, with a conversion factor of 18.02.

The accuracy of each of the 43 SMBG system results was evaluated by comparison with respective mean result of the reference measurement obtained immediately before and after the measurements with the system.

According to the ISO standard, at BG concentrations <75 mg/dl, the relative number of system results within ±15, ±10, and ±5 mg/dl and, at BG concentrations ≥75 mg/dl, the relative number of system results within ±20%, ±15%, ±10%, and ±5% of the reference measurement were calculated. For assessment of the overall accuracy of a system, the number of system results within ±15 mg/dl at BG concentrations <75 mg/dl was added to the number of system results within ±20% at BG concentrations ≥75 mg/dl.

In this study, the preparation procedure of modified blood samples with BG concentrations <50 and ≥400 mg/dl (as described earlier) did not ensure constant oxygen concentrations of the blood samples. This might cause systematic measurement bias on BG systems with an oxygen dependency (as mentioned in the manufacturer’s labeling; Table 1). Therefore, data of modified blood samples (BG concentration <50 and ≥400 mg/dl) were excluded from overall system accuracy calculation of these 9 systems (Table 1). In these cases, a complete system accuracy assessment and determination of acceptability of the system according to the ISO standard was not performed.

To illustrate the accuracy of the 43 systems according to the ISO standard, the agreement between each BG system and the mean reference result was plotted in a difference plot. The difference plot shows the deviation of single measurement results of a BG system from the reference measurement. It shows both random and systematic deviations, which reflect the total measuring error of a system. The average bias (%) of the results of each BG system was calculated according to Bland and Altman¹⁹ using the formula

$$\frac{1}{n}\sum ^{n}2\times \frac{(BG-reference)}{(BG+reference)}\times 100,$$

where BG is a single measurement result, reference is the mean value of the reference measurements before and after the BG system measurement, and n is the number of BG system results. For the calculation of the average bias of each system, only 180 data sets of native blood samples with BG concentrations ≥50 and <400 mg/dl were taken into account. The average bias is shown with 95% limits of agreement (≈1.96 × standard deviation).

Additionally, system accuracy of each BG system was evaluated in accordance to the current draft revision of ISO 15197 with the BG concentration threshold of 100 mg/dl (previously 75 mg/dl). The blood samples were distributed into the different concentration categories as mentioned earlier according to DIN EN ISO 15197:2003 with slight modifications. At BG concentrations <100 mg/dl, the relative number of system results within ±15, ±10, and ±5 mg/dl and, at BG concentrations ≥100 mg/dl, the relative number of system results within ±15%, ±10%, and ±5% of the reference measurement was calculated. For assessment of the overall accuracy of a system, the number of system results within ±15 mg/dl at BG concentrations <100 mg/dl was added to the number of system results within ±15% at BG concentrations ≥100 mg/dl.

Results

The percentage of BG system results within different deviation ranges is shown in Tables 3 and 4. According to the current ISO standard, system results within ±15, ±10, and ±5 mg/dl of the reference results at BG concentrations <75 mg/dl and system results within ±20%, ±15%, ±10%, and ±5% of the reference results at BG concentrations ≥75 mg/dl are calculated (Tables 3 and 4). For the completely assessable 34 of 43 BG systems, the overall accuracy assessment and the conformity of the system according to the ISO standard are shown in Table 3. For these 34 systems, all 200 obtained results per system from 100 subjects could be compared with the reference results (Table 3). For 9 systems with an oxygen dependency (manufacturer’s labeling), only 180 results from 90 subjects were calculated, and complete system accuracy assessment was not performed (Table 4).

Table 3.

Accuracy Results of the Completely Assessable 34 of 43 Blood Glucose Systems^a

		DIN EN ISO 15197:2003									Current draft revision of ISO 15197
				BG concentration <75 mg/dl			BG concentration ≥75 mg/dl						BG concentration <100 mg/dl			BG concentration ≥100 mg/dl
BG system	Reference method	Within accuracy limits (±15 mg/dl and ±20%)		±15 mg/dl	±10 mg/dl	±5 mg/dl	±20%	±15%	±10%	±5%	Within accuracy limits (±15 mg/dl and ±15%)		±15 mg/dl	±10 mg/dl	±5 mg/dl	±15%	±10%	±5%
		n	%	%	%	%	%	%	%	%	n	%	%	%	%	%	%	%
Accu-Chek Active	HK	(200/200)	100.0	100	100	61	100	100	100	81	(200/200)	100.0	100	100	72	100	100	79
Accu-Chek Aviva	HK	(200/200)	100.0	100	100	87	100	99	91	64	(198/200)	99.0	100	97	80	99	91	64
Accu-Chek Aviva Nano	HK	(200/200)	100.0	100	100	84	100	99	94	65	(199/200)	99.5	100	96	80	99	95	65
Accu-Chek Compact Plus	HK	(200/200)	100.0	100	88	23	100	100	91	63	(200/200)	100.0	100	86	27	100	94	69
Accu-Chek Go	HK	(200/200)	100.0	100	100	97	100	100	97	78	(200/200)	100.0	100	100	94	100	96	79
Accu-Chek Mobileb	HK	(199/200)	99.5	98	98	75	100	100	97	67	(199/200)	99.5	98	98	73	100	96	66
Accu-Chek Mobileb	HK	(200/200)	100.0	100	100	78	100	100	93	64	(200/200)	100.0	100	98	71	100	94	66
Accu-Chek Performab	HK	(199/200)	99.5	100	100	79	99	99	93	67	(199/200)	99.5	100	95	72	99	94	68
Accu-Chek Performab	HK	(198/200)	99.0	100	98	75	99	98	93	68	(196/200)	98.0	98	97	78	98	92	66
Accu-Chek Performa Nano	HK	(200/200)	100.0	100	100	93	100	100	96	66	(200/200)	100.0	100	98	87	100	96	64
Bayer Contour usb	GOx	(194/200)	97.0	100	84	61	96	88	63	31	(182/200)	91.0	90	69	45	91	68	34
Beurer GL32	GOx	(192/200)	96.0	80	48	30	100	98	90	56	(189/200)	94.5	85	62	40	99	91	56
Beurer GL40	GOx	(198/200)	99.0	97	95	53	99	94	78	43	(192/200)	96.0	98	90	52	95	78	42
Biocheck TD-4225	GOx	(187/200)	93.5	73	40	13	99	96	81	48	(183/200)	91.5	76	52	22	97	80	49
FreeStyle Freedom Lite	GOx	(200/200)	100.0	100	100	98	100	100	98	91	(200/200)	100.0	100	100	98	100	98	90
FreeStyle Lite	GOx	(200/200)	100.0	100	100	95	100	100	100	86	(200/200)	100.0	100	100	93	100	100	86
Futura Monometer	GOx	(182/200)	91.0	93	75	23	91	78	59	28	(165/200)	82.5	90	68	27	79	59	26
GlucoCheck Classic	GOx	(191/200)	95.5	100	93	60	94	86	66	38	(177/200)	88.5	97	84	55	85	66	38
GlucoCheck XL	GOx	(191/200)	95.5	98	95	63	95	89	67	40	(182/200)	91.0	97	92	58	88	65	40
Glucohexal IIc	GOx	(162/200)	81.0	61	24	8	86	74	57	28	(143/200)	71.5	50	21	7	80	64	32
GlucoRx (TD-4230)	GOx	(170/200)	85.0	98	45	13	82	63	38	14	(141/200)	70.5	83	41	14	65	40	14
GlucoSmart Swing	GOx	(193/200)	96.5	95	63	24	97	88	60	28	(182/200)	91.0	84	53	15	94	67	33
GlucoTel	GOx	(190/200)	95.0	87	71	37	97	93	70	40	(183/200)	91.5	89	70	33	92	72	41
Gluco-test Plus+ TD-4230	GOx	(198/200)	99.0	100	98	75	99	94	80	45	(190/200)	95.0	96	94	76	95	79	42
iDia	GOx	(191/200)	95.5	100	85	50	94	90	70	41	(184/200)	92.0	96	80	47	90	71	40
IME-DC Fidelity	GOx	(183/200)	91.5	80	48	18	94	88	78	44	(175/200)	87.5	72	43	18	94	85	48
iXell	GOxd	(199/200)	99.5	100	100	70	99	91	77	42	(185/200)	92.5	100	98	68	89	75	38
iXell OLED	GOxd	(198/200)	99.0	95	78	45	100	98	81	42	(194/200)	97.0	97	85	52	97	79	39
microdot+	GOx	(198/200)	99.0	97	97	71	99	94	81	43	(190/200)	95.0	98	95	62	94	83	43
OneTouch Verio	GOx	(199/200)	99.5	100	87	34	99	96	90	62	(198/200)	99.0	97	78	40	100	95	64
OneTouch Verio Pro	GOx	(193/200)	96.5	93	63	20	98	90	70	38	(183/200)	91.5	88	53	21	93	75	40
Pura	HK	(200/200)	100.0	100	92	55	100	100	75	30	(200/200)	100.0	100	95	48	100	74	30
SeniorLine GM210	GOx	(144/200)	72.0	10	0	0	88	72	48	26	(120/200)	60.0	15	0	0	79	55	29
Wellion CALLA Light	GOx	(182/200)	91.0	68	33	10	97	85	68	48	(165/200)	82.5	68	33	13	89	73	52

^a200 results from 100 subjects were evaluated.

^bAccu-Chek Mobile and Accu-Chek Performa were both tested with different test strip chemistries. The test strip chemistry was either maltose dependent (test strips evaluated first) or maltose independent (test strips evaluated second).

^cGlucoHexal test strip lot was recalled from the market in June 2010, after at least 11 months availability on the market.

^dInformation was not available.

Table 4.

Accuracy Results of Nine Blood Glucose Systems with an Oxygen Dependency on Measurement Results (as Mentioned in the Manufacturer’s Labeling)^a

		DIN EN ISO 15197:2003								Current draft revision of ISO 15197
			BG concentration <75 mg/dl			BG concentration ≥75 mg/dl					BG concentration <100 mg/dl			BG concentration ≥100 mg/dl
BG system	Reference method	Within accuracy limits (±15 mg/dl and ±20%)	±15 mg/dl	±10 mg/dl	±5 mg/dl	±20%	±15%	±10%	±5%	Within accuracy limits (±15 mg/dl and ±15%)	±15 mg/dl	±10 mg/dl	±5 mg/dl	±15%	±10%	±5%
		n	%	%	%	%	%	%	%	n	%	%	%	%	%	%
BGStar	GOx	(179/180)	100	93	83	99	96	87	62	(175/180)	98	92	80	97	87	62
Element	HK	(172/180)	97	90	53	95	83	63	29	(153/180)	90	79	48	83	64	30
GlucoCheck Comfort	HK	(178/180)	97	87	60	99	95	75	42	(171/180)	96	86	56	95	74	41
iBGStar	GOx	(173/180)	100	93	57	95	90	74	36	(165/180)	96	90	52	90	72	35
Omnitest 3	GOx	(172/180)	97	90	70	95	91	79	48	(165/180)	94	86	64	91	79	48
OneTouch VITA	GOx	(180/180)	100	100	83	100	99	85	51	(178/180)	98	93	72	99	87	50
smartLAB genie	GOx	(172/180)	75	36	11	99	98	86	56	(170/180)	84	52	23	98	87	57
smartLAB global	GOx	(173/180)	86	75	43	98	94	80	51	(167/180)	91	85	57	93	79	48
WaveSense Jazz	GOx	(178/180)	100	75	39	99	94	78	45	(173/180)	100	75	38	95	80	47

^a180 results from 90 subjects were evaluated. Data of prepared blood samples (BG concentration <50 and ≥400 mg/dl) were excluded. GOx, glucose oxidase; HK, hexokinase.

Twenty-seven (79.4%) of the 34 completely assessable systems fulfilled the minimum accuracy requirements of the ISO standard (Table 3). According to the current draft revision of ISO 15197, only 18 (52.9%) of 34 systems fulfilled the minimum accuracy requirements: ≥95% of the BG system results fall within ±15 mg/dl of the reference measurement results at glucose concentrations <100 mg/dl and within ±15% at glucose concentrations ≥100 mg/dl.

For all evaluated 43 BG systems, the agreement between the BG system results and the mean reference results according to DIN EN ISO 15197:2003 is illustrated in Figures 1A–1C. For each system, all 200 obtained results (BG concentration <50 to ≥400 mg/dl) are shown (Figures 1A–1C).

Figure 1A.

Difference plots of 43 BG systems. Black lines, system accuracy in accordance with DIN EN ISO 15197:2003; dashed lines, system overall accuracy determination according to the current draft revision of ISO 15197. For 9 BG systems with oxygen dependency (as mentioned in the manufacturer’s labeling), data of modified blood samples were excluded from overall system accuracy evaluation. For these 9 systems, the boundaries of concentration categories, including only unprepared blood samples (BG concentration ≥50 and <400 mg/dl) and categories that may include prepared blood samples (BG concentration <50 and ≥400 mg/dl), are marked by dashed perpendicular lines.

Figure 1C.

Difference plots of 43 BG systems. Black lines, system accuracy in accordance with DIN EN ISO 15197:2003; dashed lines, system overall accuracy determination according to the current draft revision of ISO 15197. For 9 BG systems with oxygen dependency (as mentioned in the manufacturer’s labeling), data of modified blood samples were excluded from overall system accuracy evaluation. For these 9 systems, the boundaries of concentration categories, including only unprepared blood samples (BG concentration ≥50 and <400 mg/dl) and categories that may include prepared blood samples (BG concentration <50 and ≥400 mg/dl), are marked by dashed perpendicular lines.

Figure 1B.

Difference plots of 43 BG systems. Black lines, system accuracy in accordance with DIN EN ISO 15197:2003; dashed lines, system overall accuracy determination according to the current draft revision of ISO 15197. For 9 BG systems with oxygen dependency (as mentioned in the manufacturer’s labeling), data of modified blood samples were excluded from overall system accuracy evaluation. For these 9 systems, the boundaries of concentration categories, including only unprepared blood samples (BG concentration ≥50 and <400 mg/dl) and categories that may include prepared blood samples (BG concentration <50 and ≥400 mg/dl), are marked by dashed perpendicular lines.

The bias according to Bland and Altman with limits of agreement of all investigated systems is shown in Figure 2 for the 43 evaluated BG systems. For the calculation of the bias, only data of native samples (BG concentration ≥50 and <400 mg/dl) were taken into account. The bias ranged from -14.1% (GlucoRx [TD-4230]) to +12.4% (SeniorLine® GM210; Figure 2). The bias was smallest for the Accu-Chek Aviva Nano system (bias, -0.1% with ±10.3% limits of agreement) and highest for GlucoRx (TD-4230; bias, -14.1% with ±20.4% limits of agreement).

Figure 2.

Bias according to Bland and Altman. Error bars represent 95% limits of agreement (≈ 1.96 × standard deviation). For the calculation of the bias of each system, only data of 180 unprepared blood samples (BG concentrations ≥50 and <400 mg/dl) were included. a: Accu-Chek® Mobileand Accu-Chek® Performa were both tested with different test strip chemistries. The test strip chemistry was either maltose dependent (left) or maltose independent (right). b: GlucoHexal® test strip lot was recalled from the market in June 2010, after at least 11 months’ availability on the market.

Discussion

Assessment of the system’s overall accuracy and deter-mination of conformity to DIN EN ISO 15197:2003 were performed with 43 systems. Of the 34 systems, for which a complete system accuracy assessment could be performed, 27 (79.4%) fulfilled the minimum accuracy requirements of the standard. Considering the tighter criteria of the current draft revision of ISO 15197, only 18 (52.9%) of these 34 systems fulfilled these minimum accuracy requirements. For 9 of the 43 evaluated systems, complete system accuracy assessment was not performed because of an oxygen dependency specified in the package insert.

The present study is focused on analytical accuracy of the BG systems under laboratory conditions and does not represent their total system accuracy^20,21 when used by patients. However, the study provides an overview about the measurement quality of a broad range of CE-marked products available on the market. In a study published in 2010, 59% of 27 investigated BG systems fulfilled the minimum accuracy requirements of the ISO standard.¹⁶ Both studies demonstrate that the CE mark of a BG system does not ensure that the minimum required accuracy criteria are fulfilled in all cases. An important issue in this context is the tendency of health insurance companies and pharmacy associations (e.g., in Germany) to require the automatic supply of low-priced systems for SMBG to patients with diabetes in order to reduce health care spending.²² Our study shows that systems with a CE mark do not necessarily exhibit equal quality and therefore should not be used interchangeably without further considerations such as evaluation of measurement accuracy.²³

The minimum accuracy requirements defined by the ISO standard apply to BG measurements over the full clinical relevant range. However, the accuracy of a BG system is probably not constant over the complete range of BG values and may exhibit different measurement qualities at different BG ranges. Previous discussions have already mentioned the evaluation of a system for the different clinically relevant BG ranges (hypoglycemic range, euglycemic range, and hyperglycemic range) separately.^21,24 This would ensure more detailed information about the analytical quality of a system, which is needed for a better categorization of systems to ensure correct therapeutic decisions. The categorization of BG systems into different quality classes and for different patient groups with specific needs for accuracy is frequently discussed.^21,25,26

Detailed comparison of different BG systems is difficult and has certain limitations. The evaluated systems are calibrated with either the GOx method or the HK method. Measurement errors of both reference methods as well as measurement differences of approximately up to 8% between these methods contribute to inaccuracy of the overall result that is not due to the systems per se.²⁷ Additionally, the results may vary depending on whether whole blood or plasma samples are used for reference measurements. Furthermore, the conversion factor from whole blood BG values to plasma equivalent BG values is specific for the manufacturer. Not all manufacturers stick to the recommendations of the International Federation of Clinical Chemistry²⁸ on reporting of BG results. In order to improve the comparability of system assessments by manufacturers, it would be useful to standardize the manufacturer’s reference measurement method and to further complete the standardization of the calibration mode to plasma calibration.

For nine systems, the test strip chemistry is labeled to be sensitive to blood oxygen content variations. For some of these nine systems, measurement results obtained in blood samples with glucose concentrations <50 or ≥400 mg/dl were remarkably different from the reference method. According to the ISO standard, blood samples with glucose concentrations <50 mg/dl can be obtained by incubation of capillary blood samples to allow glucose to hydrolyze, whereas glucose concentrations >400 mg/dl can be obtained by supplementation with glucose. However, different effects like oxygen consumption by blood cells as well as rapid equilibration with the oxygen in the ambient air, e.g., by air bubbles as well as by diffusion through gas-permeable blood collection tubes, make it quite difficult to maintain constant oxygen content in these modified samples. The preparation procedure employed in this study could also not ensure constant oxygen partial pressure in the modified samples. Several previous studies reported that some test strips, especially those with glucose oxidase enzyme reaction, are sensitive to oxygen and that high oxygen concentrations may lead to system results lower than the true value.^29–33 Most of the published system accuracy evaluation studies either do not evaluate samples with BG concentration <80 and ≥300 mg/dl (or not sufficient numbers) or use venous blood.^34–39 Main reasons for doing so are most likely the difficulty of designing a controlled human study or an adequate procedure to obtain capillary blood samples in hypoglycemic and hyperglycemic ranges.

Conclusions

In summary, 34 out of 43 BG systems were completely assessed, and 27 (79.4%) of these 34 systems fulfill the minimal accuracy requirements of the standard DIN EN ISO 15197:2003. Only 18 (52.9%) of 34 systems fulfilled the minimal accuracy requirements if tighter criteria of the current draft revision of ISO 15197 are considered. Because inaccurate systems bear the risk of false therapeutic decisions, regular and standardized evaluation of BG meters and test strips should be requested in order to ensure adherence to quality and accuracy standards.

Glossary

(BG)

blood glucose

(CE)

Conformité Européenne

(GOx)

glucose oxidase

(HK)

hexokinase

(ISO)

International Organization for Standardization

(NIST)

National Institute of Standards and Technology

(SMBG)

self-monitoring of blood glucose

Funding

This study was funded by a grant from Roche Diagnostics GmbH, Mannheim, Germany.

Disclosures

Guido Freckmann received speaker’s honoraria and refunding of traveling expenses for congress participation from Roche Diagnostics GmbH, Mannheim, Germany, the sponsor of this study, and from Berlin-Chemie AG, Berlin, Germany.