Mean Absolute Relative Difference of Blood Glucose Monitoring Systems and Relationship to ISO 15197

Guido Freckmann; Jochen Mende; Stefan Pleus; Delia Waldenmaier; Annette Baumstark; Nina Jendrike; Cornelia Haug

doi:10.1177/19322968211001402

. 2021 Mar 24;16(5):1089–1095. doi: 10.1177/19322968211001402

Mean Absolute Relative Difference of Blood Glucose Monitoring Systems and Relationship to ISO 15197

Guido Freckmann ¹, Jochen Mende ^1,^✉, Stefan Pleus ¹, Delia Waldenmaier ¹, Annette Baumstark ¹, Nina Jendrike ¹, Cornelia Haug ¹

PMCID: PMC9445334 PMID: 33759584

Abstract

Background:

The analytical quality of a blood glucose monitoring system (BGMS) is often assessed according to the requirements described in the international standard ISO 15197. However, the mean absolute relative difference (MARD) is sometimes used as well. This analysis aims at providing empirical data from BGMS evaluation studies conducted according to ISO 15197 and at providing an estimation of how MARD and percentage of measurement results within ISO accuracy limits are related.

Methods:

Results of 77 system accuracy evaluations conducted according to ISO 15197 were used to calculate MARD between BGMS and a laboratory comparison method’s results (glucose oxidase or hexokinase method). Additionally, bias and 95%-limits of agreement (LoA) using the Bland and Altman method were calculated.

Results:

MARD results ranged from 2.3% to 20.5%. The lowest MARD of a test strip lot that showed <95% of results within ISO limits was 6.1%. The distribution of MARD results shows that only 3.6% of test strip lots with a MARD equal to or below 7% showed <95% of results within ISO limits (2.2% of all test strip lots). Bias of test strip lots that showed ≥95% of results within the limits ranged from −10.3% to +7.4%. The half-width of the 95%-LoA of test strip lots that showed ≥95% of results within the limits ranged from 4.8% to 24.0%.

Conclusion:

There is a threshold MARD that may allow an estimate whether ISO 15197 requirements are fulfilled, but this statement cannot be made with certainty.

Keywords: accuracy, ISO 15197:2013, mean absolute relative difference, self-monitoring of blood glucose

Introduction

Systems for self-monitoring of blood glucose (SMBG) are widely perceived as a key component in diabetes management and self-monitoring of blood glucose is routine practice nowadays.¹ Blood glucose monitoring systems (BGMS) allow tight glycemic control and thus support health care professionals and patients in making therapy decisions in order to prevent short- and long term-complications. Prerequisite to that is that the measurement accuracy² of a BGMS is adequate so that adverse metabolic conditions such as hyperglycemia or hypoglycemia can reliably be prevented.^3-6 Thus, the clinical value of a BGMS is directly linked to its measurement accuracy.

In most regions of the world, BGMS are required to meet specific accuracy guidelines. Current guidelines include the International Organization for Standardization (ISO) standard ISO 15197 or guidance documents by the U.S. Food and Drug Administration (FDA).^7,8

In the European Union, manufacturers of BGMS have to provide evidence of conformity with ISO 15197 in order to get the Conformité Européenne (CE) label for their products. ISO 15197, which was first published in 2003 and revised with more stringent system accuracy criteria in 2013 (harmonized as EN ISO 15197:2015) specifies requirements for SMBG devices, eg, with regard to system performance, accuracy, and precision.⁷

Regarding system accuracy, accuracy criteria of ISO 15197:2013 define a BGMS to be sufficiently accurate if ≥95% of results fall within ±15 mg/dl of a laboratory reference measurement method’s results at BG concentrations <100 mg/dl and within 15% at BG concentrations ≥100 mg/dl (system accuracy criterion A). Additionally, ≥99% of results shall be found within the clinically acceptable zones A and B of the consensus error grid (CEG) (system accuracy criterion B).⁷

Although system accuracy assessment of BGMS according to ISO 15197 is suitable for evaluation of individual BGMS and has been intensively performed within the last years,^9-15 other approaches were performed as well. The mean absolute relative difference (MARD), for example, has traditionally been used to assess the accuracy of continuous glucose monitoring (CGM) systems^16-19 and was also used in studies comparing measurement accuracy of BGMS.^10,11,20 MARD represents measurement accuracy as a single numeric value, but does not distinguish between bias and imprecision. Another limiting factor of MARD is that it may vary depending on glucose concentrations. A lower MARD indicates higher accuracy. However, system accuracy assessed according to ISO 15197 cannot be directly translated into MARD values, which means that a threshold MARD value that is likely to be sufficient enough to satisfy ISO criteria can only be probabilistically estimated but cannot be deterministically defined.^21,22

The relationship between MARD and percentage of results within ISO limits has been modelled earlier.^21,22 This publication aims at providing empirical data from BGMS evaluation studies conducted according to ISO 15197 and at providing an estimation of how MARD and percentage of measurement results within ISO accuracy limits are related.

Materials and Methods

In this retrospective analysis, data from system accuracy evaluations (n = 200 measurements each) of 465 different test strip lots were used to calculate MARD between BGMS results of 169 different systems and a laboratory comparison method’s results.

In total, 809 combinations of BGMS test strip lot and comparison method were available.

All studies were conducted between 2008 and 2019 at the Institut für Diabetes-Technologie Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm (IfDT), Ulm, Germany, in compliance with the German Medical Devices Act and under consideration of the Declaration of Helsinki.

The IfDT is a testing laboratory and is accredited according to DIN EN ISO/IEC 17025:2005 and 98/79/EC in terms of test procedures for analytical and user performance evaluation according to DIN EN ISO 15197 by the Deutsche Akkreditierungsstelle GmbH, the national accreditation body for the Federal Republic of Germany, since March 2015. Evaluation of system accuracy is included in these test procedures.

The respective study protocols were approved by the responsible Ethics Committee, and, if required, by the German Federal Institute for Drugs and Medical Devices. All local regulations and requirements of Good Clinical Practice (DIN EN ISO 14155) were followed and all participants gave written informed consent.

Only adults (≥18 years) with diabetes type 1 or diabetes type 2 as well as subjects without diabetes were included. Prior to study start, a physician checked eligibility for study participation and the subjects’ medical history and medication with regard to interfering substances (eg, acetaminophen or ascorbic acid). Further exclusion criteria included eg, pregnancy, lactation period, compromising mental constitution, severe acute or chronic disease and legal incompetence.

System Accuracy Test Requirements According to ISO 15197

Experimental procedures were performed by trained study personnel based on the requirements described in detail in ISO 15197:2003, clause 7.3, or ISO 15197:2013, clause 6.3.^7,23 Experimental procedures did not differ substantially between the 2 versions of ISO 15197.

Glucose concentrations of the different capillary blood samples were distributed as specified in ISO 15197 based on the values of the respective laboratory comparison method’s results.⁷

Blood glucose measurements were performed in duplicate on an individual sample at an ambient temperature of 23°C ± 5°C and in compliance with the manufacturer’s specifications.⁷

Reference Measurements

Reference measurements were performed in duplicate on each individual sample. In the evaluated studies, reference measurements were performed with either a glucose oxidase (GOD) based procedure (YSI 2300 STAT Plus™ glucose analyzer; YSI Incorporated, Yellow Springs, OH, USA), a hexokinase (HK) based procedure (Cobas Integra^® 400 plus, Roche Instrument Center, Rotkreuz, Switzerland; Cobas^® c111 analyzer, Roche Instrument Center; Hitachi 917, Roche Diagnostics GmbH, Mannheim, Germany; Cobas^® 6000 c501, Roche Diagnostics GmbH) or both.

As verification of sample stability, the drift between mean values of consecutive duplicate reference measurements must not exceed 4 mg/dl (0.22 mmol/l) at BG concentrations <100 mg/dl (5.55 mmol/l) and 4% at BG concentrations ≥100 mg/dl (5.55 mmol/l).⁷

For 406 combinations of test strip lot and comparison method, the HK method was used as reference method. For 403 combinations of test strip lot and comparison method, the GOD method was used as reference method. For 295 combinations of test strip lot and comparison method, comparison measurements were available from at least 1 glucose analyzer using the HK method in addition to a dataset obtained with a glucose analyzer using the GOD method.

Statistical Analysis

For each system, 200 evaluable data points from at least 100 capillary samples from different subjects were analyzed. According to ISO 15197, the acceptability was determined by calculating the percentage of BGMS results within system accuracy limits of ISO 15197:2013.

This post-hoc analysis includes studies in which only 1 test strip lot was tested as well as studies in which 3 test strip lots were tested.

The MARD between the result of a BGMS and a comparison method is sometimes used in the literature to describe the analytical accuracy of a BGMS, and it is also traditionally used for CGM systems. MARD is given by the formula $MARD = \frac{1}{N} \sum_{i = 1}^{N} | \frac{B G_{i} - C o m p_{i}}{C o m p_{i}} |$ in which BG_i is the i-th BG or CGM result and Comp_i is the corresponding comparison method’s result. Lower MARD values indicate a better analytical performance.

The system accuracy data were used to establish which MARD results will likely lead to having at least 95% of results within ISO 15197:2013 accuracy limits. For each combination of test strip lot and comparison method, MARD was calculated on the basis of 200 glucose measurements. The lower and upper MARD with which a BGMS is most likely to fail or most likely to meet ISO 15197 acceptance criterion A was determined. The range in between these values can be considered as a range of uncertainty, where it is not sufficiently clear if a BGMS fails or passes the requirements of acceptance criterion A.

Additionally, bias and 95%-limits of agreement (LoA) using the Bland and Altman method were calculated. According to Bland and Altman²⁴ the 95% limits of agreement for normal distributions are defined as the mean (difference) ± 1.96 × s (with “s” being the standard deviation of the differences).

Second-order polynomial regressions were calculated to model the association between percentage of results within ±15 mg/dL and ±15% (or ±10 mg/dl and ±10%) for glucose concentrations <100 mg/dL and ≥100 mg/dl, respectively, and the MARD value.

Results

The MARD results for 809 combinations of BGMS test strip lot and comparison method ranged from 2.3% to 20.5%. The lowest MARD of a test strip lot that showed <95% of results within ISO limits was 6.1%, whereas the highest MARD of a test strip lot that showed ≥95% of results within ISO limits was 9.7% (Figure 1).

Figure 1. — MARD in relation to the percentage of results within ISO 15197:2013 for all 809 combinations of BGMS test strip lot and comparison method. Green line: lowest (best) MARD coinciding with <95% of results within the limits. Orange line: highest (worst) MARD coinciding with ≥95% of results within the limits. Blue line: second-order polynomial regression (y = −0.4062x² + 0.3690x + 0.0856, R² = 0.835).

The cumulative distribution of MARD results shows that 3.6% of test strip lots with a MARD equal to or below 7% show <95% of results within ISO limits (2.2% of all test strip lots) and that 4.1% of test strip lots with a MARD above 9% show ≥95% of results within these limits (0.6% of all test strip lots) (Figure 2).

Figure 2. — Cumulative distribution of MARD for all 809 combinations of BGMS test strip lot and comparison method having <95% (n = 252) or ≥95% (n = 557) of results within ISO 15197:2013 limits. Black lines: upper and lower limits of the range of uncertainty, where it is not sufficiently clear if a BGMS fails or passes the requirements of acceptance criterion A.

Figure 3 shows the relationship between MARD, bias and limits of agreement. Bias ranged from −17.9% to +16.8%. The lowest bias of a test strip lot that showed <95% of results within ISO limits was 0.0%, whereas the highest bias of a test strip lot that showed ≥95% of results within the limits was −9.9% and +7.4%, respectively.

The half-width of the 95%-LoA ranged from 4.8% to 36.4%. The narrowest half-width of LoA of a test strip lot that showed <95% of results within ISO limits was 7.9%. The widest half-width of a test strip lot that showed ≥95% of results within ISO limits was 24.0%.

As expected, lower MARD values tended to coincide with narrower limits of agreement and smaller bias than higher MARD values.

When applying the more stringent criteria of ±10 mg/dl (0.56 mmol/l) and ±10%, which ISO 15197 recommends to report, the lowest MARD of a test strip lot that showed <95% of results within these limits was 4.1%, whereas the highest MARD of a test strip lot that showed ≥95% of results within these limits was 5.5% (Figure 4).

Figure 4. — MARD in relation to the percentage of results within accuracy limits of ±10 mg/dl or 10% of the comparison method’s result for all 809 combinations of BGMS test strip lot and comparison method. Green line: lowest (best) MARD coinciding with <95% of results within the limits. Orange line: highest (worst) MARD coinciding with ≥95% of results within the limits. Blue line: 2nd-order polynomial regression (y = −0.0409x² − 0.1096x + 0.1874, R² = 0.903).

20 data points are below the scale (less than 45% of results within accuracy limits of ±10 mg/dl or ±10%).

The cumulative distribution of MARD results shows that 24.9% of test strip lots with a MARD equal to or below 5% show <95% of results within ±10 mg/dl or ±10% (6.4% of all test strip lots) and that 2.3% of test strip lots with MARD above 5% show ≥95% of results within these limits (1.7% of all test strip lots) (Figure 5).

Figure 5. — Cumulative distribution of MARD for all 809 combinations of BGMS test strip lot and comparison method having <95% (n = 638) or ≥95% (n = 171) of results within accuracy limits of ±10 mg/dl or 10% of the comparison method’s result. Black lines: upper and lower limits of the range of uncertainty, where it is not sufficiently clear if a BGMS fails or passes the requirements of acceptance criterion A.

Discussion

Adequate measurement accuracy of BGMS is crucial for appropriate diabetes management. In this study, the relationship between MARD and percentage of measurement results within ISO 15197:2013 accuracy limits was investigated.

One limitation of accuracy assessment according to ISO 15197 is that ISO 15197 acceptance criterion A only makes a binary assessment if measurement results are within this accuracy criterion or if they are not, whereas the extent of inaccuracy (ie, the difference between BGMS result and comparison method result) is not considered. For the calculation of MARD, however, the extent is relevant.

Moreover, not only the measurement accuracy itself is clinically relevant, but also the direction of the measurement error. A systematic negative or positive measurement difference can lead to systematic deviations from an optimal diabetes therapy. MARD does not distinguish between positive and negative errors or between systematic and random errors. Information about the direction and the extent of the measurement error are, to some degree, provided by the consensus error grid (CEG) and by the newly proposed surveillance error grid (SEG), which provides asymmetric and nonlinear boundaries between risk categories of clinical outcome.^25,26

In a performance evaluation of 3 BGMS according to ISO 15197:2013 from 2015, Bedini et al.¹⁰ observed a dependency of insulin dosing error and MARD; the lower the MARD, the lower the insulin dosing error.

Pardo and Simmons showed that accuracy results, as defined by ISO 15197:2013 requirements, cannot be directly translated into MARD values but that probabilistic models about the distribution of measurement results and the likelihood of obtaining specific sets of measurement results support estimations of ISO percentages based on MARD values.²² They developed a Bayesian model for the probability of a BGMS to pass ISO 15197:2013 accuracy criteria under the assumption that its MARD was known. Their conclusion was that the MARD of a BGMS should be no greater than 3.25% (most stringent), 4.25% (most likely), or 5.25% (most liberal) to be practically certain of passing ISO criteria.²² When assessing CGM systems, however, a MARD value of <10% is considered to describe a good analytical performance.²⁷ With regards to CGM, though, measurements for MARD calculation are performed in 2 different compartments (the interstitial fluid of the subcutaneous tissue and capillary, venous, or arterial blood), whose glucose levels are similar only during steady state conditions which are not present most of the day.²⁸ Furthermore, MARD is strongly dependent on study design.²⁹ BGMS accuracy assessment according to ISO 15197 follows strictly the requirements stipulated in ISO 15197, which implies that BGMS accuracy studies are performed under controlled laboratory conditions. CGM performance studies are not performed in a standardized manner. Hence, MARD of CGM accuracy assessment and MARD of BGMS accuracy assessment cannot be directly compared.

The evaluation of our empirical data suggests that a BGMS likely has acceptable analytical quality with respect to ISO 15197:2013, if its MARD is 6.1% or less which is markedly above Pardo’s most liberal estimate, but coincides more closely with his computations describing the relationship between MARD and single measurement results falling within ISO accuracy limits. However, the highest MARD of a BGMS passing ISO criteria we observed was 9.7% which does not correlate with the considerable number of cases in the Bayesian model having a MARD >9.7% when characterizing the relationship between single measurement values and MARD.²² It seems that the model parameters could benefit from optimization.

Generally speaking, our results indicate that a BGMS’ chance of failing ISO 15197:2013 acceptance criterion A is very low if its MARD is below 7% and that a BGMS likely does not pass this criterion if its MARD is above 9.7%. However, not only compliance or non-compliance with ISO 15197 accuracy limits is important for adequate diabetes therapy, bias and precision of a BGMS also influence insulin therapy. The 95%-LoA include both systematic bias and imprecision and indicate the expected measurement variation of a BGMS.²⁴ Bias of systems fulfilling ISO 15197 accuracy limits ranged from −10.3% to +7.4%, whereas the widest half-width of LoA was 24.0%. This indicates the level of systematic error and imprecision patients with diabetes or health care professionals can expect while making therapy decisions when using CE-labeled BGMS. However, it should be considered that BGMS which do not show any bias at all still can miss ISO requirements due to imprecision, but that a relatively high imprecision not necessarily leads to a non-fulfillment of ISO accuracy criteria.

As MARD often varies by glucose concentration, the exact distribution of values might affect the results of this analysis. However, values were distributed according to ISO 15197, which stipulates specific glucose concentration ranges and percentages of results within these ranges.

Another limitation of this post-hoc analysis is the limited number of cases. Model calculations allow more substantiated predictions, especially at the lower and upper extremes of a BGMS’ measuring range.

It should be noted that accuracy results depend on the comparison method used and that the use of only 1 comparison method for accuracy assessment could lead to misrepresented accuracy results if the investigated BGMS is calibrated against another method.²⁰ This does, however, not impact the relationship between MARD and fulfillment of ISO 15197 accuracy criteria.

It is not possible to deterministically define a threshold MARD that assures sufficient accuracy of a BGMS to satisfy ISO criteria but it is possible to define a range for MARD values that provides a reasonable certainty that a BGMS will pass ISO 15197 accuracy criteria if its MARD falls within this range.²²

Thus, similar to MARD being insufficient in describing a CGM system’s performance when used as the only metric, it is not advisable that measurement performance of a BGMS is solely judged based on its MARD or solely in view of compliance with ISO 15197 requirements.²² The consideration of multiple measures of accuracy, such as bias and percentages of accurate readings (eg, as designated by ISO 15197) above and below a threshold, allows a more thorough assessment of a BGMS’ performance.

Conclusion

While ISO 15197 specifies ranges in which a BGMS’ measurement results shall fall when compared to a comparison method, there is no consensus for how a sufficient or insufficient MARD is defined. The relationship between the probability of meter performance falling within ISO limits and the meter’s MARD value is complex.

This study showed that BGMS with a MARD as low as 6.1% might still miss ISO 15197 accuracy criteria and BGMS with a MARD of 9.7% can pass ISO 15197 accuracy criteria.

Thus, while knowing that MARD is below a threshold may allow an estimate that ISO 15197 requirements are fulfilled, this statement cannot be made with certainty.

Acknowledgments

We would like to thank our subjects who participated in this study, and all our staff involved in the study procedures.

Footnotes

Abbreviations: BGMS, blood glucose monitoring system; CE, Conformité Européenne; CEG, consensus error grid; CGM, continuous glucose monitoring; FDA, U.S. Food and Drug Administration; GOD, glucose oxidase; HK, hexokinase; ISO, International Organization for Standardization; LoA, limits of agreement; MARD, mean absolute relative difference; SEG, surveillance error grid; SMBG, self-monitoring of blood glucose.

Declaration of Conflicting Interests: The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: G.F. is general manager of the Institut für Diabetes-Technologie Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm (IfDT, Ulm, Germany), which carries out clinical studies on the evaluation of BG meters and medical devices for diabetes therapy on its own initiative and on behalf of various companies. G.F./IfDT have received speakers’ honoraria or consulting fees from Abbott, Ascensia, Dexcom, i-SENS, LifeScan, Menarini Diagnostics, Metronom Health, Novo Nordisk, PharmaSense, Roche, Sanofi, Sensile and Ypsomed. J.M., S.P., D.W., A.B., N.J. and C.H. are employees of IfDT.

Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This data review was self-funded.

ORCID iDs: Guido Freckmann Inline graphic https://orcid.org/0000-0002-0406-9529

Stefan Pleus Inline graphic https://orcid.org/0000-0003-4629-7754

Delia Waldenmaier Inline graphic https://orcid.org/0000-0003-3280-2369

Annette Baumstark Inline graphic https://orcid.org/0000-0002-3439-7400

References

1. American Diabetes Association. Standards of medical care in diabetes—2020. Diabetes Care. 2020;43(suppl 1):S1. [DOI] [PubMed] [Google Scholar]
2. Schnell O, Hinzmann R, Kulzer B, et al. Assessing the analytical performance of systems for self-monitoring of blood glucose: concepts of performance evaluation and definition of metrological key terms. J Diabetes Sci Technol. 2013;7(6):1585-1594. [DOI] [PMC free article] [PubMed] [Google Scholar]
3. Budiman ES, Samant N, Resch A. Clinical implications and economic impact of accuracy differences among commercially available blood glucose monitoring systems. J Diabetes Sci Technol. 2013;7(2):365-380. [DOI] [PMC free article] [PubMed] [Google Scholar]
4. Dave KR, Tamariz J, Desai KM, et al. Recurrent hypoglycemia exacerbates cerebral ischemic damage in streptozotocin-induced diabetic rats. Stroke. 2011;42(5):1404-1411. [DOI] [PubMed] [Google Scholar]
5. Hellman R. Glucose meter inaccuracy and the impact on the care of patients. Diabetes Metab Res Rev. 2012;28(3):207-209. [DOI] [PubMed] [Google Scholar]
6. Koivikko ML, Karsikas M, Salmela PI, et al. Effects of controlled hypoglycaemia on cardiac repolarisation in patients with type 1 diabetes. Diabetologia. 2008;51(3):426-435. [DOI] [PubMed] [Google Scholar]
7. International Organization for Standardization. In Vitro Diagnostic Test Systems: Requirements for Blood-Glucose Monitoring Systems for Self-Testing in Managing Diabetes Mellitus (ISO 15197:2013). International Organization for Standardization; 2015. EN ISO 15197:2015. [Google Scholar]
8. Food and Drug Administration. Self-monitoring blood glucose test systems for over-the-counter use: guidance for industry and Food and Drug Administration staff. Accessed November 21, 2018. http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM380327.pdf
9. Baumstark A, Jendrike N, Pleus S, Haug C, Freckmann G. Evaluation of accuracy of six blood glucose monitoring systems and modeling of possibly related insulin dosing errors. Diabetes Technol Ther. 2017;19(10):580-588. [DOI] [PubMed] [Google Scholar]
10. Bedini JL, Wallace JF, Pardo S, Petruschke T. Performance evaluation of three blood glucose monitoring systems using ISO 15197: 2013 accuracy criteria, consensus and surveillance error grid analyses, and insulin dosing error modeling in a hospital setting. J Diabetes Sci Technol. 2015;10(1):85-92. [DOI] [PMC free article] [PubMed] [Google Scholar]
11. Bedini JL, Wallace JF, Petruschke T, Pardo S. A multicenter performance evaluation of a blood glucose monitoring system in 21 leading hospitals in Spain. J Diabetes Sci Technol. 2015;10(1):93-100. [DOI] [PMC free article] [PubMed] [Google Scholar]
12. Freckmann G, Baumstark A, Jendrike N, et al. System accuracy evaluation of 27 blood glucose monitoring systems according to DIN EN ISO 15197. Diabetes Technol Ther. 2010;12(3):221-231. [DOI] [PubMed] [Google Scholar]
13. Freckmann G, Baumstark A, Schmid C, Pleus S, Link M, Haug C. Evaluation of 12 blood glucose monitoring systems for self-testing: system accuracy and measurement reproducibility. Diabetes Technol Ther. 2014;16(2):113-122. [DOI] [PubMed] [Google Scholar]
14. Freckmann G, Schmid C, Baumstark A, Pleus S, Link M, Haug C. System accuracy evaluation of 43 blood glucose monitoring systems for self-monitoring of blood glucose according to DIN EN ISO 15197. J Diabetes Sci Technol. 2012;6(5):1060-1075. [DOI] [PMC free article] [PubMed] [Google Scholar]
15. Pleus S, Baumstark A, Jendrike N, et al. System accuracy evaluation of 18 CE-marked current-generation blood glucose monitoring systems based on EN ISO 15197:2015. BMJ Open Diabetes Res Care. 2020;8(1):e001067. [DOI] [PMC free article] [PubMed] [Google Scholar]
16. Freckmann G, Pleus S, Grady M, Setford S, Levy B. Measures of accuracy for continuous glucose monitoring and blood glucose monitoring devices. J Diabetes Sci Technol. 2019;13(3):575-583. [DOI] [PMC free article] [PubMed] [Google Scholar]
17. Laffel L. Improved accuracy of continuous glucose monitoring systems in pediatric patients with diabetes mellitus: results from two studies. Diabetes Technol Ther. 2016;18(suppl 2):S223-S233. [DOI] [PMC free article] [PubMed] [Google Scholar]
18. Steineck K, II, Mahmoudi Z, Ranjan A, Schmidt S, Jørgensen JB, Nørgaard K. Comparison of continuous glucose monitoring accuracy between abdominal and upper arm insertion sites. Diabetes Technol Ther. 2019;21(5):295-302. [DOI] [PubMed] [Google Scholar]
19. Wadwa RP, Laffel LM, Shah VN, Garg SK. Accuracy of a factory-calibrated, real-time continuous glucose monitoring system during 10 days of use in youth and adults with diabetes. Diabetes Technol Ther. 2018;20(6):395-402. [DOI] [PMC free article] [PubMed] [Google Scholar]
20. Freckmann G, Pleus S, Link M, et al. Accuracy evaluation of four blood glucose monitoring systems in unaltered blood samples in the low glycemic range and blood samples in the concentration range defined by ISO 15197. Diabetes Technol Ther. 2015;17(9):625-634. [DOI] [PubMed] [Google Scholar]
21. Breton MD, Kovatchev BP. Impact of blood glucose self-monitoring errors on glucose variability, risk for hypoglycemia, and average glucose control in type 1 diabetes: an in silico study. J Diabetes Sci Technol. 2010;4(3):562-570. [DOI] [PMC free article] [PubMed] [Google Scholar]
22. Pardo S, Simmons DA. The quantitative relationship between ISO 15197 accuracy criteria and mean absolute relative difference (MARD) in the evaluation of analytical performance of self-monitoring of blood glucose (SMBG) systems. J Diabetes Sci Technol. 2016;10(5):1182-1187. [DOI] [PMC free article] [PubMed] [Google Scholar]
23. International Organization for Standardization. In Vitro Diagnostic Medical Devices: Measurement of Quantities in Biological Samples—Metrological Traceability of Values Assigned to Calibrators and Control Materials. International Organization for Standardization; 2003. EN ISO 17511:2003. [Google Scholar]
24. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;327(8476):307-310. [PubMed] [Google Scholar]
25. Klonoff DC, Lias C, Vigersky R, et al. , Error Grid Panel. The surveillance error grid. J Diabetes Sci Technol. 2014;8(4):658-672. [DOI] [PMC free article] [PubMed] [Google Scholar]
26. Parkes JL, Slatin SL, Pardo S, Ginsberg BH. A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes Care. 2000;23(8):1143-1148. [DOI] [PubMed] [Google Scholar]
27. Kovatchev BP, Patek SD, Ortiz EA, Breton MD. Assessing sensor accuracy for non-adjunct use of continuous glucose monitoring. Diabetes Technol Ther. 2015;17(3):177-186. [DOI] [PMC free article] [PubMed] [Google Scholar]
28. Heinemann L, Schoemaker M, Schmelzeisen-Redecker G, et al. Benefits and limitations of MARD as a performance parameter for continuous glucose monitoring in the interstitial space. J Diabetes Sci Technol. 2020;14(1):135-150. [DOI] [PMC free article] [PubMed] [Google Scholar]
29. Reiterer F, Polterauer P, Schoemaker M, et al. Significance and reliability of MARD for the accuracy of CGM systems. J Diabetes Sci Technol. 2017;11(1):59-67. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr1-19322968211001402] 1. American Diabetes Association. Standards of medical care in diabetes—2020. Diabetes Care. 2020;43(suppl 1):S1. [DOI] [PubMed] [Google Scholar]

[bibr2-19322968211001402] 2. Schnell O, Hinzmann R, Kulzer B, et al. Assessing the analytical performance of systems for self-monitoring of blood glucose: concepts of performance evaluation and definition of metrological key terms. J Diabetes Sci Technol. 2013;7(6):1585-1594. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr3-19322968211001402] 3. Budiman ES, Samant N, Resch A. Clinical implications and economic impact of accuracy differences among commercially available blood glucose monitoring systems. J Diabetes Sci Technol. 2013;7(2):365-380. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr4-19322968211001402] 4. Dave KR, Tamariz J, Desai KM, et al. Recurrent hypoglycemia exacerbates cerebral ischemic damage in streptozotocin-induced diabetic rats. Stroke. 2011;42(5):1404-1411. [DOI] [PubMed] [Google Scholar]

[bibr5-19322968211001402] 5. Hellman R. Glucose meter inaccuracy and the impact on the care of patients. Diabetes Metab Res Rev. 2012;28(3):207-209. [DOI] [PubMed] [Google Scholar]

[bibr6-19322968211001402] 6. Koivikko ML, Karsikas M, Salmela PI, et al. Effects of controlled hypoglycaemia on cardiac repolarisation in patients with type 1 diabetes. Diabetologia. 2008;51(3):426-435. [DOI] [PubMed] [Google Scholar]

[bibr7-19322968211001402] 7. International Organization for Standardization. In Vitro Diagnostic Test Systems: Requirements for Blood-Glucose Monitoring Systems for Self-Testing in Managing Diabetes Mellitus (ISO 15197:2013). International Organization for Standardization; 2015. EN ISO 15197:2015. [Google Scholar]

[bibr8-19322968211001402] 8. Food and Drug Administration. Self-monitoring blood glucose test systems for over-the-counter use: guidance for industry and Food and Drug Administration staff. Accessed November 21, 2018. http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM380327.pdf

[bibr9-19322968211001402] 9. Baumstark A, Jendrike N, Pleus S, Haug C, Freckmann G. Evaluation of accuracy of six blood glucose monitoring systems and modeling of possibly related insulin dosing errors. Diabetes Technol Ther. 2017;19(10):580-588. [DOI] [PubMed] [Google Scholar]

[bibr10-19322968211001402] 10. Bedini JL, Wallace JF, Pardo S, Petruschke T. Performance evaluation of three blood glucose monitoring systems using ISO 15197: 2013 accuracy criteria, consensus and surveillance error grid analyses, and insulin dosing error modeling in a hospital setting. J Diabetes Sci Technol. 2015;10(1):85-92. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr11-19322968211001402] 11. Bedini JL, Wallace JF, Petruschke T, Pardo S. A multicenter performance evaluation of a blood glucose monitoring system in 21 leading hospitals in Spain. J Diabetes Sci Technol. 2015;10(1):93-100. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr12-19322968211001402] 12. Freckmann G, Baumstark A, Jendrike N, et al. System accuracy evaluation of 27 blood glucose monitoring systems according to DIN EN ISO 15197. Diabetes Technol Ther. 2010;12(3):221-231. [DOI] [PubMed] [Google Scholar]

[bibr13-19322968211001402] 13. Freckmann G, Baumstark A, Schmid C, Pleus S, Link M, Haug C. Evaluation of 12 blood glucose monitoring systems for self-testing: system accuracy and measurement reproducibility. Diabetes Technol Ther. 2014;16(2):113-122. [DOI] [PubMed] [Google Scholar]

[bibr14-19322968211001402] 14. Freckmann G, Schmid C, Baumstark A, Pleus S, Link M, Haug C. System accuracy evaluation of 43 blood glucose monitoring systems for self-monitoring of blood glucose according to DIN EN ISO 15197. J Diabetes Sci Technol. 2012;6(5):1060-1075. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr15-19322968211001402] 15. Pleus S, Baumstark A, Jendrike N, et al. System accuracy evaluation of 18 CE-marked current-generation blood glucose monitoring systems based on EN ISO 15197:2015. BMJ Open Diabetes Res Care. 2020;8(1):e001067. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr16-19322968211001402] 16. Freckmann G, Pleus S, Grady M, Setford S, Levy B. Measures of accuracy for continuous glucose monitoring and blood glucose monitoring devices. J Diabetes Sci Technol. 2019;13(3):575-583. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr17-19322968211001402] 17. Laffel L. Improved accuracy of continuous glucose monitoring systems in pediatric patients with diabetes mellitus: results from two studies. Diabetes Technol Ther. 2016;18(suppl 2):S223-S233. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr18-19322968211001402] 18. Steineck K, II, Mahmoudi Z, Ranjan A, Schmidt S, Jørgensen JB, Nørgaard K. Comparison of continuous glucose monitoring accuracy between abdominal and upper arm insertion sites. Diabetes Technol Ther. 2019;21(5):295-302. [DOI] [PubMed] [Google Scholar]

[bibr19-19322968211001402] 19. Wadwa RP, Laffel LM, Shah VN, Garg SK. Accuracy of a factory-calibrated, real-time continuous glucose monitoring system during 10 days of use in youth and adults with diabetes. Diabetes Technol Ther. 2018;20(6):395-402. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr20-19322968211001402] 20. Freckmann G, Pleus S, Link M, et al. Accuracy evaluation of four blood glucose monitoring systems in unaltered blood samples in the low glycemic range and blood samples in the concentration range defined by ISO 15197. Diabetes Technol Ther. 2015;17(9):625-634. [DOI] [PubMed] [Google Scholar]

[bibr21-19322968211001402] 21. Breton MD, Kovatchev BP. Impact of blood glucose self-monitoring errors on glucose variability, risk for hypoglycemia, and average glucose control in type 1 diabetes: an in silico study. J Diabetes Sci Technol. 2010;4(3):562-570. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr22-19322968211001402] 22. Pardo S, Simmons DA. The quantitative relationship between ISO 15197 accuracy criteria and mean absolute relative difference (MARD) in the evaluation of analytical performance of self-monitoring of blood glucose (SMBG) systems. J Diabetes Sci Technol. 2016;10(5):1182-1187. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr23-19322968211001402] 23. International Organization for Standardization. In Vitro Diagnostic Medical Devices: Measurement of Quantities in Biological Samples—Metrological Traceability of Values Assigned to Calibrators and Control Materials. International Organization for Standardization; 2003. EN ISO 17511:2003. [Google Scholar]

[bibr24-19322968211001402] 24. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;327(8476):307-310. [PubMed] [Google Scholar]

[bibr25-19322968211001402] 25. Klonoff DC, Lias C, Vigersky R, et al. , Error Grid Panel. The surveillance error grid. J Diabetes Sci Technol. 2014;8(4):658-672. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr26-19322968211001402] 26. Parkes JL, Slatin SL, Pardo S, Ginsberg BH. A new consensus error grid to evaluate the clinical significance of inaccuracies in the measurement of blood glucose. Diabetes Care. 2000;23(8):1143-1148. [DOI] [PubMed] [Google Scholar]

[bibr27-19322968211001402] 27. Kovatchev BP, Patek SD, Ortiz EA, Breton MD. Assessing sensor accuracy for non-adjunct use of continuous glucose monitoring. Diabetes Technol Ther. 2015;17(3):177-186. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr28-19322968211001402] 28. Heinemann L, Schoemaker M, Schmelzeisen-Redecker G, et al. Benefits and limitations of MARD as a performance parameter for continuous glucose monitoring in the interstitial space. J Diabetes Sci Technol. 2020;14(1):135-150. [DOI] [PMC free article] [PubMed] [Google Scholar]

[bibr29-19322968211001402] 29. Reiterer F, Polterauer P, Schoemaker M, et al. Significance and reliability of MARD for the accuracy of CGM systems. J Diabetes Sci Technol. 2017;11(1):59-67. [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Mean Absolute Relative Difference of Blood Glucose Monitoring Systems and Relationship to ISO 15197

Guido Freckmann, MD

Jochen Mende, MSc

Stefan Pleus, MSc

Delia Waldenmaier, MSc

Annette Baumstark, PhD

Nina Jendrike, MD

Cornelia Haug, MD