Assessment of System Accuracy, Intermediate Measurement Precision, and Measurement Repeatability of a Blood Glucose Monitoring System Based on ISO 15197

Abstract

Background:

Analytical quality of blood glucose monitoring systems (BGMS) is an important aspect for many diabetes patients. Sufficiently high analytical quality is required for adequate diabetes therapy.

Methods:

In this study, system accuracy and measurement precision of a BGMS were assessed based on ISO 15197:2013. For system accuracy, this standard requires a specific glucose distribution and at least 95% of results obtained with the BGMS in capillary blood to fall within ±15 mg/dl or ±15% (at glucose concentrations <100 mg/dl or ≥100 mg/dl, respectively) of corresponding comparison method results, and at least 99% of results to be found within clinically acceptable consensus error grid (CEG) zones A and B. Based on ISO 15197:2013, intermediate measurement precision, using control solution, and measurement repeatability, using venous blood samples, were analyzed by calculation of standard deviations (SDs) and coefficients of variation (CV) at glucose concentrations <100 mg/dl or ≥100 mg/dl, respectively, although ISO 15197:2013 does not specify acceptance criteria.

Results:

The BGMS fulfilled system accuracy requirements with ≥99% of results within ±15 mg/dl or ±15% of the comparison method results, and 100% of results in CEG zones A and B. Intermediate measurement precision analysis showed SD ≤2.2 mg/dl and CV ≤2.3%. Analysis of measurement repeatability showed SD ≤2.1 mg/dl and CV ≤2.4%.

Conclusion:

System accuracy requirements of ISO 15197:2013 were fulfilled by the BGMS. As ISO 15197:2013 does not specify precision requirements, precision analysis results were compared with those reported for other BGMS in the literature and found to be similar.

Blood glucose monitoring systems (BGMS) are often used by diabetes patients on an insulin regimen as basis for their therapeutic decisions, like administration of insulin or rescue carbohydrates. Accuracy of the results displayed by BGMS is therefore an important issue in the use of BGMS, because only sufficiently high accuracy allows diabetes patients adequate diabetes control.

According to the International Organization for Standardization’s (ISO) standard ISO 15197:2013, which establishes minimum requirements for BGMS for self-monitoring that should result in acceptable performance and which also describes procedures for demonstrating compliance with these minimum requirements, accuracy comprises the two concepts of trueness and precision.¹ Trueness describes how large the deviation between an average of replicate measurement results and a comparative (ie, reference) results is. Trueness is inversely related to the systematic measurement error (bias), so that maximum trueness is achieved in absence of bias. Precision on the other hand is inversely related to random measurement errors that lead to differences within replicate measurement results. Both of these concepts play a role in assessing the reliability of measurement results from a BGMS, because a high level of accuracy can only be achieved if the measurement results are sufficiently true and precise.

ISO 15197:2013 requirements regarding analytical performance of BGMS include assessment of system accuracy in the hands of trained professionals, whose results can also be used for bias estimation, and assessment of precision under varying measurement conditions over an extended period of time (intermediate measurement precision) and under similar measurement conditions over a short period of time (measurement repeatability). ISO 15197:2013 was harmonized as EN ISO 15197:2015 with regulations of the European Union, without changes to the requirements and procedures.

This study focused on assessments of system accuracy, intermediate measurement precision, and measurement repeatability of the blood glucose monitoring functionality of a multifunctional monitoring system.

Methods

This study was performed at the Institut für Diabetes-Technologie, Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm (IDT) in Germany between November 2017 and January 2018. Prior to subject recruitment, the study was approved by the responsible ethics committee, and exempted from approval by the competent authority. All local regulations and requirements of Good Clinical Practice (DIN EN ISO 14155:2012) were followed.

IDT is a testing laboratory accredited according to DIN EN ISO/IEC 17025:2005 and 98/79/EC in terms of test procedures for analytical and user performance evaluation according to DIN EN ISO 15197 by the Deutsche Akkreditierungsstelle GmbH, the national accreditation body for the Federal Republic of Germany. Evaluation of system accuracy, intermediate measurement precision, and measurement repeatability is included in these test procedures.

Blood Glucose Monitoring System

The BGMS used in this study was the BG monitoring functionality of the FORA® 6 Connect (GD82) Multi-functional Monitoring System (ForaCare Suisse AG, St. Gallen, Switzerland). System characteristics are shown in Table 1.

Table 1.

Blood Glucose Monitoring System Characteristics.

System	Lot	Lot number	Test strip enzyme	Measurement range	Hematocrit range	Manufacturer
FORA 6 Connect (GD82) Multi-functional Monitoring system (BG monitoring functionality)	1	WG17H104-CEE	Glucose dehydrogenase	10-600 mg/dl	0-70%	ForaCare Suisse AG, St. Gallen, Switzerland
	2	WG17H604-CEE
	3	WG17H904-CEE

This multifunctional system can be used with different types of test strips that allow for measurement of BG, hematocrit, hemoglobin, β-ketone, total cholesterol, and uric acid concentrations. In this study, the system was used with BG test strips.

Study Procedures

System accuracy

System accuracy testing was performed based on requirements of ISO 15197:2013. To obtain the required number of 100 evaluable data sets, 113 subjects were enrolled. Before BGMS measurements with each of 3 reagent system lots were conducted, subjects were asked to wash and dry their hands. Study personnel subsequently collected a capillary blood sample from the subject’s fingertip by skin puncture and measured the sample’s glucose concentration in duplicate with 2 test meters using test strips from the same vial. This step was repeated for all three reagent system lots with, in total, 6 meters. Room temperature was checked to be within 23°C ± 5°C and humidity was checked to be within the range indicated in the manufacturer’s labelling during each measurement. The hematocrit value was checked to be within the range of 0% and 70% based on the manufacturer’s measurement specifications on an additional sample. Hematocrit values were determined with an alignment chart after sample centrifugation in heparinized capillaries.

Comparison values were obtained using a hexokinase-based method (Cobas Integra® 400 plus; Roche Instrument Center, Rotkreuz, Switzerland), because the manufacturer’s reference method is also hexokinase-based. Based on daily measurements with standard reference material 965b of the National Institute of Standards and Technology (Gaithersburg, MD) with four target concentrations, bias ranged from 0.1% to 1.7%, and CV was ≤2.4%.

Aliquots for comparison measurements were collected before and after a sample’s glucose concentration was measured with all 3 of the test strip lots. After centrifugation of the respective blood samples, measurements were performed in duplicate in capillary plasma.

The stability of the samples’ glucose concentrations was verified by calculating the difference in glucose concentrations between the two consecutively drawn aliquots. Results were checked to be ≤4 mg/dl at glucose concentrations ≤100 mg/dl and ≤4% at glucose concentrations >100 mg/dl, otherwise data were excluded.

BG concentrations of samples were distributed according to ISO 15197:2013 over the clinically relevant concentration range (5% ≤50 mg/dl, 15% >50 to 80 mg/dl, 20% >80 to 120 mg/dl, 30% >120 to 200 mg/dl, 15% >200 to 300 mg/dl, 10% >300 to 400 mg/dl, and 5% >400 mg/dl). The assignment of samples to the applicable concentration range was based on the average of the comparison method measurement results.

Adjustment of glucose concentrations by glycolysis or glucose supplementation was allowed to obtain samples with glucose concentrations <50 and >400 mg/dl, however, unaltered samples were preferred over altered samples for data evaluation. Oxygen partial pressure of altered samples was checked to be within the range of 55 to 100 mmHg by using a blood gas analyzer (OPTI™ CCA-TS, OPTI Medical Systems Inc, Roswell, GA, USA).² Altered samples were applied to the test strip from a syringe; unaltered samples were applied directly from the fingertip.

Blood samples for adjustment of glucose concentrations measured with the BGMS and the comparison method, as well as all samples for comparison measurements were collected in lithium-heparin tubes. Unadjusted glucose samples for BGMS measurement were measured directly from the fingertip.

Intermediate measurement precision

Intermediate measurement precision was evaluated based on ISO 15197:2013. Three control solution samples representing glucose concentrations from 30 to 50 mg/dl, 96 to 144 mg/dl and 280 to 420 mg/dl were measured once with each of 10 test meters per test strip lot and day. Measurements were performed by 2 users (each user performed 5 measurements per control solution interval, test strip lot and day) on 10 subsequent days with 3 test strip lots resulting in a total of 900 separate BGMS measurements which were performed under environmental conditions as specified by the manufacturer.

Additional measurements were performed in duplicate with a YSI 2300 STAT Plus (YSI Inc, Yellow Springs, OH) glucose analyzer in aliquots of the same control solution samples to assess sample stability on each day and over the course of all 10 days using the stability criteria mentioned above. YSI 2300 STAT Plus was used, because Cobas Integra 400 plus was not intended to be used with aqueous samples according to the manufacturer’s labeling.

Measurement repeatability

Based on requirements of ISO 15197:2013, the repeatability test was performed with 10 test meters and 3 reagent system lots on 5 venous blood samples collected in lithium-heparin tubes. Samples from 5 different subjects and with the following glucose concentration ranges stipulated by ISO 15197:2013 were collected: (1) 30 to 50 mg/dl, (2) 51 to 110 mg/dl, (3) 111 to 150 mg/dl, (4) 151 to 250 mg/dl, (5) 251 to 400 mg/dl. Glucose concentrations of the samples were measured 10 times with each of the 10 test meters by a single user on 5 consecutive days. With a specific sample, all measurements with the 3 different test strip lots were performed on the same day. BGMS measurements were performed under environmental conditions as specified by the manufacturer. Sample temperature was checked to be within 23°C ± 5°C before starting the first measurement for a specific sample and it was not allowed to differ more than ±2°C from that value until the last measurement for the same sample.

Additional measurements were performed in duplicate with a Cobas Integra 400 plus glucose analyzer in aliquots of the same samples to assess sample stability over the course of measurements. Stability criteria were the same as for system accuracy evaluation.

Data Analysis

Data management and evaluation was performed based on requirements of ISO 15197:2013 at the study site.

System accuracy

As first criterion, ISO 15197:2013 requires ≥95% of measurement values per reagent system lot either falling within ±15 mg/dl at glucose concentrations <100 mg/dl or within ±15% at glucose concentrations ≥100 mg/dl. For data analysis, the difference between each of the 200 BGMS measurements per test strip lot and the corresponding mean comparison method result was calculated. Accuracy results were visualized in a difference plot (Figure 1). Second, the relative number of data points falling within the clinically acceptable zones A and B of the consensus error grid³ (CEG) was calculated for all of the 3 reagent system lots taken together.

Figure 1.

Difference plot for the investigated blood glucose monitoring system. Bold black lines indicate system accuracy limits of ISO 15197:2013/EN ISO 15197:2015. The three investigated test strip lots are displayed in different colors and icons.

The relative bias of the measurement results was determined according to Bland and Altman⁴ for each reagent system lot and the percentage distribution of all data points into the 8 different zones of the surveillance error grid (SEG) was calculated using the SEG software.^5,6

Data were excluded from evaluation if the change between first and second comparison duplicate measurement indicated unstable glucose concentrations, if the CV within a comparison duplicate measurement was >5%, if the study personnel documented a procedure-related error, in case of hemolytic sample, or if a sufficient number of samples within a glucose concentration category was already reached.

Intermediate measurement precision

For data evaluation, mean value, standard deviation (for BGMS results <100 mg/dl) and coefficient of variation (CV) (for BGMS results >100 mg/dl) were calculated for each test strip lot and analysis of variance components was performed for the following factors: meter, reagent system lot, reagent system vial, user, and measurement day.

On the eighth day, one BG meter had to be replaced because of a device deficiency. Data that were obtained with the deficient meter on the day of replacement were excluded from analysis. Using the replacement meter, all scheduled measurements were repeated.

Measurement repeatability

For each glucose concentration (ie, sample), the mean BGMS result and the standard deviation (SD) or the coefficient of variation (CV) (for samples with mean BGMS result <100 mg/dl or ≥100 mg/dl, respectively) were calculated for each test strip lot separately.

No data were excluded. Measurements were repeated in case of the BGMS showing an error message.

Results

System Accuracy

Glucose concentrations ranged from 40 mg/dl to 456 mg/dl, and samples were distributed by glucose concentration according to requirements of ISO 15197:2013.

Between 99.0% and 99.5% of BGMS results were found within ±15 mg/dl or ±15% of comparison method results, which is consistent with ISO criteria (Table 2). Applying more stringent criteria described in ISO 15197:2013, 86.0% to 90.0% were found within ±10 mg/dl or ±10%, and 57.5% to 61.5% were found within ±5 mg/dl or ±5%.

Table 2.

System Accuracy Results and Measurement Bias for the Investigated Blood Glucose Monitoring System.

	System accuracy	CEG	CEG		SEG	SEG
Lot	Within ±15 mg/dl or ±15%	Zone A	Zone B	Biasa	No risk (risk score ≤0.5)	Lower slight risk (risk score >0.5 to ≤1.0)
1	99.5% (199/200)	99.67% (598/600)	0.33% (2/600)	−3.9%	97.50% (585/600)	2.50% (15/600)
2	99.0% (198/200)			−2.3%
3	99.5% (199/200)			−3.9%

^aBias, ie, the systematic measurement difference between the blood glucose monitoring system and the reference method, was calculated according to Bland and Altman.⁴

Difference plots for the three test strip lots are shown in Figure 1.

CEG analysis showed that all measurement results fell within the clinically acceptable zones A and B which is consistent with ISO criteria (Table 2).

In SEG analysis, 2.50% of data pairs were found in the zone associated with slight risk of either a hypoglycemic or a hyperglycemic event. All other results were associated with no risk (risk score ≤0.5) (Table 2).

The comparison of BGMS results and the hexokinase-based method’s results revealed negative biases for all test strip lots ranging from -2.3% to -3.9% (Table 2). For comparison method results of <75 mg/dl, BGMS results were predominantly found to be lower than comparison method results (ie, negatively biased).

Intermediate Measurement Precision

Intermediate measurement precision yielded an SD of ≤2.2 mg/dl for samples with glucose concentrations of <100 mg/dl and a CV of ≤2.3% for samples with glucose concentrations ≥100 mg/dl (Table 3).

Table 3.

Intermediate Measurement Precision Results for the Investigated Blood Glucose Monitoring System.

	Glucose concentration range (average BGMS result)
Lot	30 to 50 mg/dl (43.3 mg/dl)	96 to 144 mg/dl (133.3 mg/dl)	280 to 420 mg/dl (322.1 mg/dl)
1	1.9 mg/dl	1.8%	1.9%
2	2.2 mg/dl	2.2%	2.3%
3	1.8 mg/dl	2.1%	1.8%

Results given in mg/dl are standard deviations; results given in percent are coefficients of variation.

Analysis of variance was performed for the following components: BGMS meter, test strip vial (dependent from the meter), test strip lot, user, and measurement day.

Considering the low SD and CV found in this study, none of the evaluated components had a systematic, relevant effect on the intermediate measurement precision’s outcome compared to residual effects which were defined as part of the total variance which cannot be explained with the stated variance components.

Measurement Repeatability

The repeatability testing resulted in an SD of ≤2.1 mg/dl for samples with blood glucose concentrations <100 mg/dl and in a CV of ≤2.4% at glucose concentrations ≥100 mg/dl (Table 4).

Table 4.

Measurement Repeatability Results for the Investigated Blood Glucose Monitoring System.

	Glucose concentration range (average BGMS result)
Lot	30 to 50 mg/dl (27.1 mg/dl)	51 to 110 mg/dl (71.1 mg/dl)	111 to 150 mg/dl (131.5 mg/dl)	151 to 250 mg/dl (221.8 mg/dl)	251 to 400 mg/dl (311.3 mg/dl)
1	1.8 mg/dl	1.8 mg/dl	1.9%	1.5%	1.4%
2	2.0 mg/dl	2.0 mg/dl	2.4%	1.6%	1.4%
3	2.1 mg/dl	1.7 mg/dl	2.4%	1.6%	1.6%

Results given in mg/dl are standard deviations; results given in percent are coefficients of variation.

Discussion

System accuracy and measurement precision (intermediate measurement precision and measurement repeatability) of a BGMS were assessed in this study based on ISO 15197:2013 requirements. Both aspects indicate how reliable measurement results from the investigated BGMS are. System accuracy shows how well individual results match the “true” glucose concentration (estimated by the corresponding comparison results). Depending on the magnitude, differences can affect therapeutic decisions to a relevant degree. These results are supplemented by precision analysis, because the level of variability between results from replicate measurements in the same sample becomes apparent.

ISO 15197:2013 requirements regarding system accuracy were fulfilled by the investigated system with at least 99% of results within the specified differences of ±15 mg/dl or ±15% and 100% of results in the clinically acceptable CEG zones A and B. In SEG analysis, almost all results were found in the “no risk” zone, the remaining results were associated with a slight risk of hypo- or hyperglycemia. Both types of error grid analysis are intended to assess the clinical risk associated with BGMS results, and both types of analysis are based on survey results. Whereas the CEG analysis distributes pairs of BGMS and comparison results into five different risk zones,³ SEG analysis provides more detail by associating these pairs to risk scores on a near-continuous scale.⁵

Diabetes patients may not be aware of test strip lot-to-lot differences in bias, thus expecting no systematic differences between their measurement results, so that varying bias may lead to reduced quality of diabetes therapy. Substantial lot-to-lot variability regarding bias was reported in some studies.^7-9 However, bias analysis in this study showed only small differences between the three test strip lots.

Put into context of reports for other BGMS in the literature,^10-13 the investigated BGMS showed similar results in the precision analysis (SD ≤2.2 mg/dl and CV ≤2.3% for intermediate precision analysis with control solution samples, and SD ≤2.1 mg/dl and CV ≤2.4% for measurement repeatability with blood samples). However, verification of compliance with established guidelines was not possible, because neither ISO 15197:2013 nor FDA requirements for over-the-counter BGMS stipulate minimum requirements for precision analysis.^1,14 A goal of CV ≤5.0% was set forth by the Scandinavian Evaluation of Laboratory Equipment for Primary Health Care (SKUP) group,¹⁵ although a different methodological approach is used, so that results may not be directly comparable.

As required by ISO 15197:2013, glucose measurements were performed in a controlled environment by trained personnel, so that results may not be completely representative of the accuracy as perceived by diabetes patients. BGMS often show more accurate results in the hands of trained professionals than in the hands of lay-users,^16-22 and ambient conditions like temperature are also known to possibly affect the measurement results.^23,24 In addition, transport and storage conditions might differ from those experienced by users, because meters, test strips, and control solution were provided by the manufacturer, and thus procured through different channels than those available to users.

In the past, measurement performance of some BGMS was found to be inadequate,^25-28 so that the assumption that BGMS available on the market perform sufficiently well may be unfounded. To provide patients with the necessary tools for adequate diabetes therapy, BGMS performance should be assessed independently both before and after market introduction of BGMS.

Conclusion

In this study, the investigated BGMS fulfilled system accuracy requirements of ISO 15197:2013, and precision analysis showed results similar to what is reported for other BGMS in the literature.