Analytical Performance of Glucose Monitoring Systems at Different Blood Glucose Ranges and Analysis of Outliers in a Clinical Setting

Abstract

We investigated the analytical accuracy of 27 glucose monitoring systems (GMS) in a clinical setting, using the new ISO accuracy limits. In addition to measuring accuracy at blood glucose (BG) levels < 100 mg/dl and > 100 mg/dl, we also analyzed devices performance with respect to these criteria at 5 specific BG level ranges, making it possible to further differentiate between devices with regard to overall performance. Carbohydrate meals and insulin injections were used to induce an increase or decrease in BG levels in 37 insulin-dependent patients. Capillary blood samples were collected at 10-minute intervals, and BG levels determined simultaneously using GMS and a laboratory-based method. Results obtained via both methods were analyzed according to the new ISO criteria. Only 12 of 27 devices tested met overall requirements of the new ISO accuracy limits. When accuracy was assessed at BG levels < 100 mg/dl and > 100 mg/dl, criteria were met by 14 and 13 devices, respectively. A more detailed analysis involving 5 different BG level ranges revealed that 13 (48.1%) devices met the required criteria at BG levels between 50 and 150 mg/dl, whereas 19 (70.3%) met these criteria at BG levels above 250 mg/dl. The overall frequency of outliers was low. The assessment of analytical accuracy of GMS at a number of BG level ranges made it possible to further differentiate between devices with regard to overall performance, a process that is of particular importance given the user-centered nature of the devices’ intended use.

Glucose monitoring systems (GMS) are routinely used by patients with diabetes to monitor glycemic control and modify treatment as needed. In this respect, the analytical performance of GMS is of course of crucial importance. The accuracy of GMS is usually assessed according to the Guidelines of the International Organization for Standardization (ISO). A new standard has recently been published with more stringent criteria than in the previous edition.^1,2 Minimum acceptable system accuracy requirements for GMS now specify that ≥95% of the glucose meter results may not differ more than ±15 mg/dl from the reference method at glucose concentrations < 100 mg/dl (previously < 75 mg/dl) and ±15% (previously ±20%) at glucose concentrations ≥ 100 mg/dl.¹ Furthermore, concerning prevalence and range of outliers, it is now specified that >99% of measured glucose values must fall in zones A and B of the consensus error grid.

During the consultation process of the new criteria it has been suggested that the analytical performance of GMS is highly dependent on its intended use.^3-6 For example, insulin-dependent patients under intensified glycemic control require highly accurate devices for adequate insulin dosing, whereas patients with Type 2 diabetes who are treated with medication with no or little risk of hypoglycemia could use less accurate GMS. It has, therefore, been suggested that the analytical accuracy of GMS should be tested not only in relation to BG level ranges below and above 100 mg/dL but also in relation to different glycemic ranges, as already previously proposed for continuous glucose monitoring systems.⁷

The aim of the present study was to analyze test results from 27 GMS obtained in a clinical setting, with regard to (1) analytical accuracy according to the new ISO accuracy limits as well as after stratification into 5 different BG level ranges and (2) frequency and extent of outliers.

Methods

A total of 37 insulin-dependent patients with diabetes without severe acute or chronic concomitant disease composed the study group; they participated in a study on a new implantable glucose biosensor.⁸ Mean values for age and duration of diabetes were 52.3 ± 8.9 and 16.3 ± 4.0 years, respectively. Metabolic control was stable, and Hba1c values ranged from 6.4% to 8.3% (46.45 to 67.21 mmol/mol). Aside from antidiabetic, antihypertensive, and lipid-lowering medications, patients did not take any other substances that might influence GMS glucose measurements, such as high-dose paracetamol, salicylate, or vitamin C. Hematocrit levels were within the normal range (median 42%, range 36%-47%).

BG measurement procedure are described in detail elsewhere⁸ and can be briefly summarized as follows: glucose excursions at levels ranging between 50 and 300 mg/dl were induced by administration of a carbohydrate-rich meal and/or injection of an appropriate dose of insulin. Capillary blood (20 µl) was taken from the fingertip using lancets at 10-minute intervals and immediately hemolyzed for laboratory glucose determination. Capillary blood glucose was measured simultaneously using 2 commercially available GMS. Laboratory and GMS measurements were always performed in the same order. BG determinations were not performed in duplicate. Each measurement series took 3.5-4.5 hours and consisted of approximately 20-26 paired determinations performed by well-trained technicians at normal room temperature and humidity. Each patient participated in 6-10 measurement series over a period of 9 months.

The following commercially available GMs were tested: Accu-Chek® Compact, Accu-Chek® Mobile, Accu-Chek® Aviva Nano (Roche Diagnostics, Mannheim, Germany); BG Star®, iBG Star® (AgaMatrix Inc, Salem, NC, USA); Breeze®, Contour® Plasma, Contour® USB, Contour® USB next, Contour XT® (Bayer Health Care, Leverkusen, Germany); FineTouch® (Terumo Corp, Tokyo, Japan); FreeStyle Lite® (Abbott Diabetes Care, Witney, UK); GL 40®, GL 50® (Beurer Medical, Ulm, Germany); Glucomen LX®, Glucomen LX plus® (Menarini Diagnostics, Florence, Italy); GlucoSmart® Swing (MSP Bodmann, Bobingen, Germany); my glucohealth® (Entra Health Systems LCC, San Diego, CA, USA); mylife Pura® (Ypsomed AG, Burgdorf, Switzerland); Omnitest 3® (B. Braun, Melsungen, Germany); One Touch Ultra easy®, One Touch Verio Pro®, One Touch Verio IQ®, One Touch Vita® (LifeScan Inc, Milpitas, CA, USA); smartLAB mini®, smartLAB sprint® (HMM Diagnostics, Dossenheim, Germany); Wellion Calla® (MedTrust, Marz, Austria). At least 3 different lots of corresponding test strips were used.

All devices carry the CE (Conformité Européene) label and provide plasma-calibrated results. For each type of GMS 2 individual devices with the corresponding test strips were used in parallel during the study. Prior to each measurement series, devices were tested to ensure proper functioning. Reference measurements were performed at Diabetesinstitut Heidelberg by Hitado Super GL (Hitado GmbH, Möhnesee, Germany) using the glucose oxidase method. The analyzer was validated by internal (CV always < 3%) and external quality control measurement (tests always in required limits; Referenzinstitut für Bioanalytik, Bonn, Germany) as required by the German national standard.⁹ Results from the laboratory method were converted from whole BG values to plasma-equivalent BG values according to the following formula: plasma equivalent BG (mg/dl) = 1.11 × whole BG (mg/dl).

Glucose oxidase was the reference method for all GMS with the exception of the Accu-Chek devices, which are calibrated using the hexokinase method. BG levels measured using the hexokinase method are known to be 3.5-6.7% higher than values measured by the glucose oxidase method.^10,11 For the assessment of the Accu-Chek devices, BG values of the reference method were therefore adjusted by +5%.

The study was approved by the relevant Ethics Committee and was conducted in accordance with the Declaration of Helsinki. All subjects provided written informed consent prior to participation. The study was performed between May 2010 and July 2012.

Biometric Evaluation

BG values measured by GMS were compared with reference results according to the new ISO accuracy limits, that is (1) at reference BG levels < 100 mg/dl by calculating the percentage of GM results within a tolerance range up to 15 mg/dl and (2) at reference BG levels ≥ 100 mg/dl by giving the percentage of GMS results within a tolerance range of up to 15%.The overall accuracy of devices was assessed by combining the results obtained at BG concentrations < and > 100 mg/dl. To evaluate the measurement performance within more closely defined BG ranges, glucose values were classified into 5 groups (50-99 mg/dl, 100-149 mg/dl, 150-199 mg/dl, 200-249 mg/dl, and 250-300 mg/dl, respectively) and accuracy was assessed according to the new ISO criteria mentioned above. Outliers were analyzed regarding prevalence and extent by the consensus error grid as demanded by the new accuracy requirements.¹

Results

The number of paired BG determinations and the percentage of measurements meeting the new ISO accuracy limits in overall as well as at BG concentrations < 100 mg/dl and > 100 mg/dl are given in Table 1. An average of 274 paired measurements were done per device. Only 12 of the 27 GMS tested met the new accuracy requirements. Overall accuracy of GMS varied widely (80.4%-99.6%), with only 4 devices reaching levels of accuracy of ≥ 98% (device numbers 2, 3, 9, 10). When data pairs obtained at BG level ranges < and > 100 mg/dL were analyzed separately, 14 GMS met the new limits at the lower and 13 GMS at the higher BG level ranges.

Table 1.

Accuracy Results of GMS in Alphabetical Order.

				Glucose meter results fulfilling error limits at:
Device number	Glucose meter	n	Overall accuracy (%)	n	BG concentration < 100 mg/dl (%)	n	BG concentration ≥ 100 mg/dl (%)
1	Accu-Chek Aviva Nano	303	97.0	35	91.4	268	97.8
2	Accu-Chek Compact	282	99.6	41	100.0	241	99.6
3	Accu-Chek Mobile	294	99.3	47	97.9	247	99.6
4	BG Star	302	90.1	44	90.9	258	89.9
5	iBG Star	269	97.4	44	97.7	225	97.3
6	Breeze	249	86.7	39	92.3	210	85.7
7	Contour Plasma	284	94.7	59	96.6	225	94.2
8	Contour USB	248	96.4	34	100.0	214	95.8
9	Contour USB next	275	99.6	37	97.3	238	100.0
10	Contour XT	288	99.7	41	100.0	247	99.6
11	FineTouch	292	93.5	41	80.5	251	95.6
12	FreeStyle Lite	301	97.7	42	100.0	259	97.3
13	GL 40	254	94.9	37	97.3	217	94.5
14	GL 50	251	86.5	32	93.8	219	85.4
15	Glucomen LX	251	92.0	34	82.4	217	93.5
16	Glucomen LX plus	246	91.5	43	93.0	203	91.1
17	GlucoSmart Swing	253	91.7	34	79.4	219	93.6
18	my glucohealth	285	91.9	45	95.6	240	91.3
19	mylife Pura	276	97.5	46	100.0	230	97.0
20	Omnitest 3	252	80.2	36	86.1	216	79.2
21	One Touch Ultra easy	284	96.8	44	97.7	240	96.7
22	One Touch Verio Pro	292	92.8	31	83.9	261	93.9
23	One Touch Verio IQ	283	96.8	40	97.5	243	96.7
24	One Touch Vita	296	96.3	49	98.0	247	96.0
25	Smart Lab Mini	270	87.4	42	81.0	228	88.6
26	Smart Lab Sprint	280	87.1	33	84.8	247	87.4
27	Wellion Calla	251	91.6	36	77.8	215	94.0

Results fulfilling accuracy requirements are given in bold.

Table 2 gives the number of observations in the more narrowly defined BG level ranges and the percentages of GMS results that met the new limits. About one-third (35%) of BG values in the 50-99 mg/dl range fall below a BG level of 80 mg/dl. In the case of the lower BG level ranges (50-99 and 100-149 mg/dl) new accuracy requirements were met by 14 and 12 GMS, respectively. The number of devices meeting accuracy requirements increased in line with BG level ranges, resulting in a total of 19 devices meeting accuracy requirements at the highest BG level range (Figure 1).

Table 2.

Accuracy Results of GMS After Differentiation in 5 BG ranges.

		BG 50-99 mg/dl		BG 100-149 mg/dl		BG 150-199 mg/dl		BG 200-249 mg/dl		BG 250-300 mg/dl
Device number	Glucose meter	Data pairs n	Max 15 mg/dl (%)	Data pairs n	Max 15% (%)	Data pairs n	Max 15% (%)	Data pairs n	Max 15% (%)	Data pairs n	Max 15% (%)
1	Accu-Chek Aviva Nano	35	91.4	53	96.2	60	98.3	86	96.5	69	100.0
2	Accu-Chek Compact	41	100.0	36	97.2	68	100.0	68	100.0	69	100.0
3	Accu-Chek Mobile	47	97.9	55	100.0	52	100.0	81	98.8	59	100.0
4	BG Star	44	90.9	65	89.2	66	89.4	60	90.0	67	91.0
5	iBG Star	44	97.7	70	100.0	56	94.6	56	98.2	43	95.3
6	Breeze	39	92.3	36	80.6	58	82.8	69	84.1	47	95.7
7	Contour Plasma	59	96.6	48	91.7	47	93.6	70	94.3	60	96.7
8	Contour USB	34	100.0	46	97.8	49	91.8	57	98.2	62	95.2
9	Contour USB next	37	97.3	70	100.0	71	100.0	53	100.0	44	100.0
10	Contour XT	41	100.0	66	100.0	73	98.6	66	100.0	42	100.0
11	FineTouch	41	80.5	57	100.0	65	96.9	75	93.3	54	92.6
12	FreeStyle Lite	42	100.0	62	95.2	59	93.2	78	100.0	60	100.0
13	GL 40	37	97.3	51	92.2	41	90.2	61	98.4	64	95.3
14	GL 50	32	93.8	64	85.9	56	91.1	49	87.8	50	76.0
15	Glucomen LX	34	82.4	50	90.0	44	95.5	56	91.1	67	97.0
16	Glucomen LX plus	43	93.0	57	94.7	57	93.0	42	92.9	47	83.0
17	GlucoSmart Swing	34	79.4	51	84.3	52	94.2	74	97.3	42	97.6
18	my glucohealth	45	95.6	68	91.2	76	88.2	60	96.7	36	88.9
19	mylife Pura	46	100.0	51	96.1	53	98.1	84	97.6	42	95.2
20	Omnitest 3	36	86.1	34	82.4	62	77.4	70	74.3	50	86.0
21	One Touch Ultra easy	44	97.7	63	95.2	54	96.3	72	98.6	51	96.1
22	One Touch Verio IQ	40	97.5	68	92.6	70	97.1	68	98.5	37	100.0
23	One Touch Verio Pro	31	83.9	60	78.3	64	96.9	67	98.5	70	100.0
24	One Touch Vita	49	98.0	69	95.7	61	96.7	78	96.2	39	94.9
25	Smart Lab Mini	42	81.0	54	83.3	58	94.8	73	83.6	43	95.3
26	Smart Lab Sprint	33	84.8	52	78.8	64	82.8	62	88.7	69	97.1
27	Wellion Calla	36	77.8	43	90.7	54	98.1	72	94.4	46	91.3
Mean (± SD) of data pairs		40 ± 6		56 ± 11		59 ± 9		67 ± 11		53 ± 11
Percentage of all data pairs		14.5		20.4		21.4		24.4		19.3

Results fulfilling accuracy requirements are given in bold.

Figure 1.

Number of GMS meeting the new accuracy limits in 5 different BG ranges.

To rate the overall measurement performance of individual GMS at the specified BG level ranges, we counted the number of BG level ranges at which the devices met the criteria specified. Figure 2 gives the results of this ranking procedure. All 12 GMS that fulfilled the new overall accuracy criteria met the requirements of at least 4 of the 5 BG level ranges (ranking group A). Seven GMS met the criteria at 2 or 3 BG level ranges, all regarding BG levels > 150 mg/dl (ranking group B). Only 3 of these devices reached the requirements at normal or low BG levels. The rest of the devices (n = 6) fulfilled the requirements in only 1 or no BG level (ranking group C).

Figure 2.

Number of BG ranges in which the new accuracy limits were met. Triangles mark GMS that fulfilled overall accuracy requirements shown in Table 1.

Prevalence and extent of outliers have been assessed by CEG (Table 3). All GMS fulfilled the new criteria, that is >99% of measured glucose values fall in zones A and B of the consensus error grid. However, the prevalence of “benign reading errors” was different. Devices of ranking group A showed an average prevalence of 0.34%, those of group B 2.04%, and those of group C 2.83%

Table 3.

Percentage of Measurements in Zones A-C of the Consensus Error Grid.

Device number	Glucose meter	Zone A, clinically accurate (%)	Zone B, benign reading error (%)	Zone C, overcorrection (%)
1	Accu-Chek Aviva Nano	100.0	0.0	0.0
2	Accu-Chek Compact	100.0	0.0	0.0
3	Accu-Chek Mobile	100.0	0.0	0.0
4	BG Star	98.3	1.7	0.0
5	iBG Star	99.6	0.4	0.0
6	Breeze	95.6	4.6	0.0
7	Contour Plasma	98.5	1.5	0.0
8	Contour USB	100.0	0.0	0.0
9	Contour USB next	100.0	0.0	0.0
10	Contour XT	99.3	0.7	0.0
11	FineTouch	100.0	0.0	0.0
12	FreeStyle Lite	99.6	0.4	0.0
13	GL 40	98.8	1.2	0.0
14	GL 50	95.2	4.8	0.0
15	Glucomen LX	98.0	2.0	0.0
16	Glucomen LX plus	97.5	2.5	0.0
17	GlucoSmart Swing	98.8	1.2	0.0
18	my glucohealth	96.5	3.5	0.0
19	mylife Pura	98.9	1.1	0.0
20	Omnitest 3	96.0	4.0	0.0
21	One Touch Ultra easy	98.9	1.1	0.0
22	One Touch Verio IQ	98.2	1.8	0.0
23	One Touch Verio Pro	99.6	0.4	0.0
24	One Touch Vita	100.0	0.0	0.0
25	Smart Lab Mini	99.6	0.4	0.0
26	Smart Lab Sprint	95.4	4.2	0.4
27	Wellion Calla	99.6	0.4	0.0

Discussion

This study produced the following key findings: First, only 12 out of 27 devices assessed met the new ISO accuracy limits; the numbers of GMS meeting the accuracy limits at BG ranges < and > 100 mg/dl were similar. Second, accuracy of GMS testing across a range of BG levels proved to be more accurate at higher BG levels than at normal or lower BG levels; the differentiation of several BG ranges performed in this study allows for a more detailed assessment of device accuracy. Third, the frequency of outliers was very low; >99% of measured glucose values fall in zones A and B of the consensus error grid.

The accuracy performance of GMS is determined by a number of factors, such as the processes involved in the production of devices and test strips, calibration procedures, physical factors (ambient temperature, altitude), presence of interfering substances, and user error. The analytical accuracy of devices, which is the focus of this study, accounts for a major proportion of overall performance of devices. Using laboratory-based tests in accordance with ISO standards and including a total of 43 devices, Freckmann et al were able to show that only one-half of the devices (52.9%) reached the minimum accuracy requirement if the new ISO criteria were applied.¹² In our study using a clinical setting, only 12 devices (44%) met these requirements, which is in line with Freckmann et al’s results and confirms our previous study with a smaller number of devices.¹³

Responses to the consultation process on the new ISO criteria included the suggestion that instead of testing accuracy at BG < 100 mg/dl and > 100 mg/dl, it should be tested at a number of different BG level ranges, making it possible to further differentiate between devices with regard to overall performance.^3-6 The aim of this more detailed approach to accuracy testing is to identify devices that produce highly accurate measurements, making them suitable for use by patients on intensified insulin therapy. Devices that fail to produce laboratory accuracy at levels below 100 mg/dl might be suitable for use by patients whose therapy regimen excludes the risk of hypoglycemia. Results from the current study were stratified into 5 BG level ranges, ranging from 50 to 300 mg/dl. Analysis revealed only few devices meting the accuracy limits at low and normal BG levels (on average n = 13, ie, 48.1%), but a higher number at BG levels > 250 mg/dl (on average n = 19, ie, 70.3%). These differences in performance are easily overlooked when testing includes only 2 BG ranges (<100 mg/dl and >100 mg/dl). In the current study the numbers of GMS meeting accuracy requirements at BG levels < 100 mg/dl and > 100 mg/dl were almost identical (14 and 13, respectively).

It therefore appears reasonable that accuracy testing should be conducted across specific BG level ranges to better assess the overall performance of glucose monitoring systems, as suggested previously.^3-6

Our ranking procedure showed that devices performing highly accurate measurements at low and normal BG levels usually also produce accurate measurements at high BG levels (ranking group A). The number of “benign reading errors” was very low (0.34%). These devices meet the new overall accuracy requirements and are therefore suitable for use by patients on intensified insulin therapy, who take their own therapeutic decisions. The 7 devices of ranking group B that met the accuracy criteria at BG levels > 150 mg/dl showed an overall accuracy just below 95% (92-94.9%). Prevalence of “benign reading errors” was also low (2.04%). Thus, it could be discussed if these GMS are suitable for patients with no or little risk of hypoglycemia. Ranking group C, in which GMS met the accuracy requirements in only 0-1 BG ranges, showed a low overall accuracy (<91%) and higher prevalence of “benign reading errors” (2.83%). Thus, these GMS seem not suitable for patients to monitor their metabolic situation. With regard to the suggestion, to define accuracy requirement of GMS also on a “user-related” basis, these results show that accuracy testing should be conducted across more than 2 specific BG levels. Perhaps the cost of glucose monitoring could be lowered since there is no need to use exclusively high-tech GMS for all diabetic patients.^3-6

Detailed explanations for why more than half of the glucose monitoring systems produced relatively unfavorable results particularly in the lower and normal BG ranges could not be found. This lack of accuracy in the lower BG ranges has been found also in previous studies.^14,15 Given that all testing was conducted by trained staff, patient error can be excluded as a contributory factor. None of the patients involved presented with serious complications, nor did they take any substances other than antidiabetic, antihypertensive, and lipid-lowering medications. All had normal hematocrit levels, and as far as ambient temperature and humidity were concerned, test conditions were within the normal range throughout the study. It would appear, therefore, that any differences in the results produced by the measurement systems were likely to be due to differences in the manufacturing processes involved. For instance, a recent study by Baumstark et al¹⁶ was able to show that a large number of test strips also differed concerning the measurement accuracy. Inaccuracies might furthermore be explained by errors and differences resulting from the type of reference method used for device calibration.^17,18

The current study’s clinical setting might be regarded as a limiting factor since accuracy testing is usually conducted in accordance with the relevant ISO standard, and therefore in a laboratory setting. Key differences in this regard would be the numbers of patients to be investigated (n = 100), requirements regarding the range of glucose concentrations to be included in the samples at levels < 50 mg/dl and > 300 mg/dl, as well as testing in duplicate.

Our study included 37 patients who were undergoing a number of tests as part of a biosensor study.⁸ Due to the protocol of this study, blood samples did not cover glucose concentrations of < 50 mg/dl or > 300 mg/dl, and no artificial BG level ranges were produced. The fact that glucose determination did not include testing in duplicate resulted in a study setting that was closer to real-world conditions than a laboratory setting. In spite of these limitations, results obtained in our clinical setting shows a good congruence with results obtained in a laboratory setting according to ISO standard.¹² Thirteen devices (numbers 1-4, 8, 12, 13, 17, 19, 20, 22, 24, 27) were investigated in both studies. Average levels of overall accuracy were very similar in both types of study, with 95.3 ± 5.3% recorded in the laboratory setting and 94.2 ± 5.2% in the clinical setting.

In conclusion, this study performed in a clinical setting reveals that only fewer than half of the current GMS fulfill accuracy requirements according to the new ISO accuracy limits. However, the assessment of analytical accuracy at a number of several BG ranges made it possible to further differentiate devices with regard to overall performance, a process that is of particular importance given the user-centered nature of the devices’ intended use.