New Criteria for Assessing the Accuracy of Blood Glucose Monitors Meeting, October 28, 2011

Abstract

Glucose meters (GMs) are routinely used for self-monitoring of blood glucose by patients and for point-of-care glucose monitoring by health care providers in outpatient and inpatient settings. Although widely assumed to be accurate, numerous reports of inaccuracies with resulting morbidity and mortality have been noted. Insulin dosing errors based on inaccurate GMs are most critical. On October 28, 2011, the Diabetes Technology Society invited 45 diabetes technology clinicians who were attending the 2011 Diabetes Technology Meeting to participate in a closed-door meeting entitled New Criteria for Assessing the Accuracy of Blood Glucose Monitors. This report reflects the opinions of most of the attendees of that meeting.

The Food and Drug Administration (FDA), the public, and several medical societies are currently in dialogue to establish a new standard for GM accuracy. This update to the FDA standard is driven by improved meter accuracy, technological advances (pumps, bolus calculators, continuous glucose monitors, and insulin pens), reports of hospital and outpatient deaths, consumer complaints about inaccuracy, and research studies showing that several approved GMs failed to meet FDA or International Organization for Standardization standards in post-approval testing. These circumstances mandate a set of new GM standards that appropriately match the GMs’ analytical accuracy to the clinical accuracy required for their intended use, as well as ensuring their ongoing accuracy following approval. The attendees of the New Criteria for Assessing the Accuracy of Blood Glucose Monitors meeting proposed a graduated standard and other methods to improve GM performance, which are discussed in this meeting report.

Introduction

Since the 1980s, portable glucose meters (GMs) have become widely accepted clinical devices for monitoring glucose levels. Glucose meters have transformed diabetes care by enabling the identification of hyperglycemia as the major cause of diabetes complications and have made possible the use of intensive glucose control (IGC) to prevent complications using multiple daily injections, insulin pumps, and diabetes medicatons.^1–3 Glucose meters are now routinely used for self-monitoring of blood glucose (SMBG) as well as point-of-care glucose monitoring by health care providers in outpatient and inpatient settings.

Glucose meters are assumed to be accurate and are widely used to make therapeutic decisions. Yet there are significant differences in accuracy between and within meter brands, which is often unknown to clinicians and their patients. For example, brand-to-brand variations may be discovered only after a health insurer has selected a lower cost meter as the preferred brand and a user performs simultaneous glucose tests on the old and new GMs. Reports of errors and inaccuracies continue to occur as GMs are used in increasingly diverse clinical and home settings that have varied requirements for accuracy and safety.

Courtney Harper Lias, Ph.D., director of the Division of Chemistry and Toxicology Devices at the Food and Drug Administration (FDA), reported that there were 100 deaths associated with potential GM inaccuracies between 1992 and 2009 and 12,672 serious injuries from 2004 to 2008.⁴ Richard Hellman, M.D., past president of the American Association of Clinical Endocrinologists (AACE), in a response to a New York Times editorial regarding GM accuracy, said, “There is considerable concern from many quarters that the lower-than-optimal accuracy of glucose meters in current use, both in the inpatient and out-patient setting, is a cause of errors in insulin dosage, with resultant morbidity, and possibly mortality.”⁵ In fact, the Endocrine Society (TES) has recommended against using continuous glucose monitoring (CGM) devices in hospital critical care units, because they are calibrated with GMs and there are limited data on the effects of interference from the common clinical conditions (hypoxia, anemia, hypotension) and typical drugs (catecholamines) used in these settings on GM accuracy.⁶

Due to these concerns, the FDA and various medical societies, including the American Diabetes Association (ADA), TES, AACE, and the Diabetes Technology Society, have begun a dialogue to establish new industry standards for GMs. At the request of David Klonoff, M.D., editor-in-chief of this journal, we convened a group of 45 diabetes clinicians on October 28, 2011, at the 11th Annual Diabetes Technology Meeting in San Francisco. These clinicians participated in a “by invitation only” meeting entitled New Criteria for Assessing the Accuracy of Blood Glucose Monitors. They provided their opinions on a wide variety of issues surrounding the present state-of-the-art of point-of-care glucose monitoring and discussed various ways to improve the current situation. This article reflects our interpretation of the opinions and recommendations of this clinical panel.

Background and Current Status

A patient’s ability to maintain their glucose in a predefined goal range and obtain beneficial therapeutic outcomes depends on their ability to measure their glucose accurately at frequent intervals. The ADA considers SBMG to be an integral part of diabetes treatment for both type 1 and type 2 diabetes.^7,8 Their recommendation for patients using multiple daily injections is to perform blood glucose (BG) measurements three or more times a day, at a minimum, and to utilize SMBG to maximize euglycemia and minimize hypoglycemia.⁷

The ADA proposed the first standard for GMs in 1987, recommending that accuracy be within ±10% of the reference reading for 100% of values.⁹ Shortly after publication of results of the Diabetes Control and Complications Trial in 1993 that showed risk reductions in several microvascular complications between 54% and 76% in the IGC group,¹ the ADA recommended that GMs have a total error within ±5% for 100% of readings, partly to minimize the frequency and severity of hypoglycemia for those attempting to achieve tight glucose control.¹⁰

Ideally, these clinical accuracy requirements would be equivalent to analytic standards that would permit appropriate treatment decisions to be made directly from GM results. However, the FDA’s regulatory standards must take into account the currently achievable performance of today’s meters. Consequently, the current FDA and International Organization for Standardization (ISO) standards, which were designed to reduce measurement errors and undesirable clinical outcomes while minimizing any cost increases or inconvenience that might decrease their use or benefits, are less stringent than the clinical accuracy goals proposed by the ADA (Table 1).

Table 1.

Current Glucose Meter Performance Recommendations and Standards

Clinical accuracy recommendations			% within rangea
ADA 19879	±10%		100%
ADA 199410	±5%		100%
Meter approval standards
FDA	±15 < 75 mg/dl	±20% > 75 mg/dl	95%b
ISO 15197 200311	±15 mg/dl	±20%	95%b

^aFor GM values compared with laboratory values taken at the same time.

^bBoth FDA and ISO standards allow 5% of meter values to be outside these limits. There is no limitation on the clinical severity of these outliers.

The desire to update the FDA standard is being driven by improvements in meter accuracy, reports of hospital and outpatient deaths as noted earlier, consumer complaints about inaccuracies, and the potential for clinical error with current standards. Research studies also show that several currently approved GMs failed to meet FDA or ISO standards in post-approval testing.^12–14 Indeed, Margaret Hamburg, M.D., commissioner of the FDA, in response to a 2009 letter from the AACE about GM accuracy based on the 2003 ISO standard, said that the “FDA, in fact, argued strongly in support for stronger criteria when the document was in development and even considered voting against the standard.”¹⁵

Since 1987, the Clarke and/or Parke error grids have been used to assess the extent of serious clinical errors in meter accuracy. While they have served some useful purpose, the expert consensus of those at this 2011 meeting was that these error grids, introduced in 1987, were no longer meaningful, partly because very few meters have ever failed this criteria and partly because of the seemingly arbitrary divisions between the zones.¹⁶

In addition, there was general agreement that the accuracy performance of GMs needs to be matched appropriately to the clinical requirements for the setting in which they are used. For example, highly accurate devices are required for any patient using IGC, such as during pregnancy, with small children, in patients who have hypoglycemia unawareness, and in hospitals (emergency room, intensive care unit, critical care unit, recovery room). Less accurate GMs could be used for outpatients using split-dose insulin or oral agents, where the risk of lower accuracy is less critical. Some clinicians have requested GMs capable of diagnosing diabetes in the clinical setting based on ADA criteria. It was generally agreed that GMs should be labeled appropriately so that the documented accuracy of each GM was consistent with its intended clinical use. And finally, the panel expressed concern that any new standard be easily understood and that start-up companies be able to comply with a new approval standard.

What Is Accuracy?

The term “accuracy” used in this paper in relation to GMs is an inclusive term that includes all aspects of accuracy. These aspects include accuracy and precision, shown in Figure 1, as well as a low bias, close linearity, and a narrow limit of agreement as depicted in Figure 2. Optimal GM accuracy can be thought of as a minimum of total error.

Figure 1.

Blood GM accuracy and precision.¹⁷ (A) Accuracy measures how close the average of a hundred or more GM values over a wide range of glucose readings is to an average of the laboratory values taken at the same time. Accuracy ignores the error in individual values. (B) Precision shows how consistent glucose readings are with each other, or how closely a series of meter values agree with each other, regardless of how close they are to the reference lab value. Precision is often measured as coefficient of variation. (C) Having both accuracy and precision in one meter is the best option. An ideal glucose meter will have a low mean absolute relative error.

Figure 2.

Blood GM bias and linearity.¹⁷ Bias is the average of systematic or built-in meter errors, and is usually measured as the percentage difference above or below a reference lab’s values. An ideal bias is 0.0%. Linearity shows upward or downward trends in meter values relative to lab reference values. An ideal linearity is 1.0. The 95% limit of agreement shows the two lines where 95% of the meter results can be found relative to the reference values. An ideal limit of agreement shows two lines very close to the zero line. In the example above, bias and linearity of a sample meter are shown for the low, middle, and high glucose ranges, while the limit of agreement for all values is shown by the upper and lower dashed lines.

Why Is Better Accuracy Needed?

Accurate GMs and test strips are crucial for patients and health care providers to identify and correct hypo-glycemia as well as to identify and correct hyperglycemia safely through insulin dosing. Erroneous insulin doses can arise when a GM is inaccurate, is imprecise, has positive or negative bias, or is nonlinear where the dosing error gets greater or smaller as the glucose changes. Any error introduced into insulin dosing decisions complicates the patient’s path to euglycemia. Each error magnifies the propagation of error and adds to the risk of hypo- and hyperglycemia.

Clinicians would prefer to have some highly accurate GMs available for critical situations, such as diagnosing diabetes (a fasting glucose ≥ 126 mg/dl, a 2 h post-glucose load of ≥200 mg/dl, or a hemoglobin A1c of ≥6.5%) and pre-diabetes (≥100 mg/dl, ≥140–199 mg/dl, or 5.7–6.4%, respectively).¹⁸ Thus quality and accuracy of GMs and test strips are crucial not only to identify hypo- and hyperglycemia and to dose insulin for safe lowering of hyperglycemic values, but also, ideally, to diagnose dysglycemic conditions accurately.

Today’s improved technology for insulin delivery demands greater GM accuracy. Insulin pumps can accurately deliver insulin in doses of 0.025 U or less, with a precision of the order of 0.06% for someone who uses 40 U/day. This dosing precision is significantly compromised when the GM used to determine these insulin doses varies in accuracy by ±20%. With a target glucose of 100 mg/dl, a pump’s bolus correction calculator cannot safely lower readings higher than 200 mg/dl when 95% of BG readings may range from 160 to 240 mg/dl, and 5% of readings may be outside this range. Thus an accurate insulin correction dose for a GM reading of 200 mg/dl may produce a glucose outcome between 60 and 140 mg/dl. As glucose values rise above 200 mg/dl, accurate correction doses based on a relatively inaccurate GM will create ever-increasing risks for hypoglycemia or ongoing hyper-glycemia despite very accurate dosing by the pump.

Even if the ISO standard tightens the accuracy requirement to ±15% above 75 mg/dl, insulin dosing errors with IGC are still likely. With this standard, a GM reading of 300 mg/dl can actually be a plasma value between 255 and 345 mg/dl. After insulin is accurately delivered to correct the “300 mg/dl” reading ±15% or 45 mg/dl, the glucose may end up somewhere between 55 and 145 mg/dl in 19 out of 20 readings.

Calibration of all current commercial CGM devices is based on GM readings. Yet CGM error is greatly reduced when a CGM device is calibrated using laboratory venous plasma glucose levels rather than GM readings.¹⁹ Accurate calibration of CGM devices is becoming ever more critical as an increasing number of patients use their CGM readings to calculate an insulin dose even though this is not FDA approved. This provides an additional demand for graduated GM accuracy levels within an overall approval standard. Finally, CGM devices are being used more frequently to automatically detect glycemic variability as a risk factor for complications and accuracy of the GM used to calibrate the device increases the sensitivity of these machine-learning algorithms to detect glycemic variability.

Determinants of Accuracy

Numerous GM-related factors can distort accuracy, including device error, test strip manufacturing defects, test strip lot variations, underfilling of test strips, GM time-setting errors, environmental factors (temperature extremes, humidity, and high altitude), plus a variety of blood substrates and other factors (anemia, hypoxia, oxygen therapy, icodextrins, and pressor agents). In addition to GM error, other factors, such as carbohydrate counting, insulin dosing, activity levels, and others, can also complicate the patient’s path to euglycemia. Any error introduced into insulin dosing decisions adds to the propagation of total error.

Importantly, these errors are magnified in the “real world” because GM accuracy diminishes in the hands of patients compared with health professionals,²⁰ whenever alternate test sites are used,²¹ and when a patient uses more than one brand of meter, where bias may vary by as much as 40%, as shown in Table 2. Bolus calculators that correct for insulin stacking are common in insulin pumps. They are also becoming more widely available in meters and phone applets for those who use insulin pens and injections, thus increasing the need for accuracy among a greater number of patients using insulin.²²

Table 2.

Variations in Clinical Test Results: Central Laboratories Compared with Glucose Meters

Number of devices tested	GM Precision: % coefficient of variation in tested devices	Maximum bias between two devices
29 Central laboratories23	2.5–4.3%	11%
17 GMs23	Up to 12–14%	41%
9 GMs12	3.2–8.1%	13.6% (−10.4% to +3.2%)
5 GMs24	6–11%	—

One modeling study found that a GM with a total error of 5% would cause insulin dosing errors in 8% to 23% of doses, with dosing errors doubling when the BG meter error was 10%.²⁵ A simulation study of 100 patients treated with intensive insulin therapy using the Yale protocol in an intensive care unit setting using 45 sets with varied GM bias and imprecision demonstrated that there was significantly more variability and hypo-glycemia with greater imprecision.²⁶

Re-Certification for Determination of Accuracy Post-Food and Drug Administration Approval

Several studies have revealed a need for ongoing evaluations of GMs following FDA approval. In a review of the accuracy of 27 meters previously approved for the 2003 ISO 15197 standard, only 16 actually met this standard in post-approval testing.⁷ Another study found that 3 of 9 meters failed the ISO standard when testing was performed by patients,¹² while another study revealed that, during patient use of 21 GMs, 16% of BG measurements were more than 20% above or below the reference value.¹³ The total error found in this GM testing is far greater than the current ISO standard of 5% or less. The authors suggested that, since “inaccurate BG monitoring systems bear the risk of false treatment decisions by the diabetes patient and subsequent possible severe health injury, manufacturers should regularly and effectively check the quality of BG meters and BG test strips.”¹³

A universal set of control solutions is needed to test GM accuracy in low, moderate, and high glucose ranges. For example, three vials with glucose concentrations within 3% of 45, 160, and 320 mg/dl would allow clinical and home monitoring of meter and test strip accuracy. Rather than having a loose range of acceptable performance, the meter’s readings would be compared with the control solution’s stated concentration.

Clinicians also need a simple post-approval process to verify the accuracy of individual patients’ GMs and test strips. One way to do this is to have control solutions available to check a meter in the clinic or home. Current control solutions are rarely used, partly because their acceptable calibration spans a wide range that is useless for evaluation of accuracy. For example, one manufacturer accepts as clinically accurate any of their meters that can read a control glucose concentration of 345 mg/dl as a glucose reading between 261 and 391 mg/dl. This approval range of -25% to +16% differs significantly in direction, though not in breadth, from the current FDA standard. Current control solutions also have a limited shelf life that adds to the already high cost of SMBG.

Proposal for a New Food and Drug Administration Glucose Meter Graduated Standard

The FDA currently uses one standard to approve or disapprove all GMs. Yet GMs are used in a variety of clinical situations that have diverse accuracy require-ments. The need for GM accuracy is highly dependent on its intended use. For example, patients with type 2 diabetes who are on medications with little or no risk of hypoglycemia would not require a meter that is as accurate as insulin-requiring patients whose glycemic goal is within the normal range or critically ill patients who are being treated with tight glycemic control protocols. The ability to use GMs in hospital settings requires strict safety requirements.

In a new paradigm, gradients of accuracy would allow GMs to be matched to specific clinical needs. These include hospital use, IGC with a bolus calculator, pregnancy, use in a child who is unable to communicate hypoglycemia symptoms, those with hypoglycemia unawareness, calibration of a CGM device, or those with type 2 diabetes treated with diet or oral agents. Such intended-use gradients would ensure that the GM being prescribed meets a patient or hospital’s need for accuracy. One example of how meters might be rated for intended use is shown in Figure 3.

Figure 3.

One example of how meters might be rated for intended use.

Alternatively, letter grades with pluses and minuses (similar to bond ratings) could be used. For example, for a meter whose intended use is for inpatient or diagnostic use and has an accuracy of ≤5 mg/dl in the 30–75 mg/dl range, ≤5% for 76–180 mg/dl, and ≤10% above 180 mg/dl would receive an A+, A+, B+, whereas a GM with the same intended use that could not achieve the accuracy in the 30–70 mg/dl range but was able to achieve ≤5% in the other ranges would get a B+, A+, A+. (The percentage of readings within each range might be used to give plus and minus ratings.) Although a few newer GMs would pass a strict 80% of readings within ±10 mg/dl below 100 mg/dl and within ±10% above 100 mg/dl, no current meter would meet the ADA’s recommended goal of ±5%.

Accuracy gradients would hopefully encourage continuing accuracy improvements over time. For values below 75 mg/dl, gradients could be set up as increments of ±5, 8, and 10 mg/dl, while percentage difference would be used for values above 75 mg/dl in increments of 2.5% from 5% to 15% and perhaps temporarily to 17.5% and 20%. The increment of ±10 mg/dl below 75 mg/dl could be used in conjunction with percentages of 12.5% through 20%, ±8 mg/dl could be used with ±7.5% and ±10%, and ±5 mg/dl could be used with ±5%. A clinical oversight committee would be needed to determine what percentages within and near each gradient would qualify a meter for approval for use in that gradient category.

One goal of a new graduated standard is to reduce the frequency and severity of clinical errors caused by outliers. If the proposed ISO standard continues to allow 5% of readings as outliers, the risk of potentially serious clinical error will remain. For example, someone on insulin who tests four times a day might encounter a significant dosing error once every five days. For every million glucose tests, 50,000 insulin doses could create a significant dosing error. For this reason, reducing outlier frequency to 97% or 98% should be considered.

Similar to the need for greater accuracy in GMs, GM clock settings also need greater accuracy and dependability, especially for patients on insulin where dosing adjustments in clinic settings is determined from the timing of hypo-glycemic and hyperglycemic events. For this reason, GM clock settings should be clearly visible, easy to adjust when an incorrect time is displayed, and not be lost when a battery falls out or is changed.

Transition Time to a New Standard

While an improved standard is being implemented, there is a need to minimize any disruption of GM manufacturing and distribution. Pat Bernhardt, M.T.(ASCP), scientific reviewer of diagnostic devices for the FDA’s Center for Devices and Radiological Health, has stated that 85% of current meters meet the ±15% criteria and 49% would meet the ±10% criteria, but only 22% would meet the ±5% criteria. For readings under 75 mg/dl, only 28 of the 40 meters tested (72%) would meet the ±10 mg/dl criteria, while 18 of 40 (45%) would meet the ±5 mg/dl criteria.²⁷

In contrast with the FDA assessment, post-approval testing of 27 meters currently ISO qualified found that, at BG concentrations > 75 mg/dl, only 37.0% of GMs were within ±15%, 3.7% within ±10%, and none were within ±5%.¹² It is not clear why such a large variation between approval and post-approval testing would occur.

To maintain GM availability, the current standard might remain in place for a set number of years, with the possible exception that outliers be reduced to 97% or 98%. A minimum qualification gradient could be granted during this period if a manufacturer did not want to seek a higher gradient ranking for a meter approved prior to January 2012. New meters and any existing meter could apply for a gradient rating above the minimum designation based on data previously submitted to the FDA. Yearly or biyearly retesting of GM accuracy would be required to maintain a gradient rating.

Methods to Determine Glucose Meter Performance

Glucose meter accuracy can be assessed in several ways: computer simulation modeling, direct comparison to a laboratory reference value, and direct evaluations of patient performance within a clinical setting.¹⁶ Simulation models are ideal for selection of clinical accuracy requirements, while direct comparison of GM values with laboratory reference values is the best way to assess meter performance for FDA approval. Evaluating performance in clinical settings after approval ensures patient safety and ongoing quality in the manufacturing process.

For FDA or ISO approval, the results from a GM is compared with simultaneous matched specimens measured in a reference device whose accuracy is equivalent to a clinical reference laboratory. Glucose meter accuracy may vary at different glucose ranges, so both bias and linearity should be reported within different glycemic ranges, such as low (30–70 mg/dl), moderate (71–180 mg/dl), and high (181–400 mg/dl).

In this process, at least 200 GM values, evenly spread over the full range of glucose values, might be measured on at least 10 different meters using a minimum of three lots of test strips. Each finger stick GM reading is compared with a simultaneous spun capillary venous sample measured on a YSI 2300 or equivalent clinical laboratory device. Rather than expose patients to these glucose extremes, consideration should be given to spiking venous blood samples to attain hyperglycemic samples and allowing the venous sample to sit for varied periods of time to allow natural glycolysis to attain hypoglycemic values.

Summary

The diabetes clinicians who participated in the New Criteria for Assessing the Accuracy of Blood Glucose Monitors meeting, October 28, 2011, recommended that the FDA and BG monitoring companies work together to provide accurate products with new technology.

The panel members noted that meter users often assume that FDA approval means that their meter meets strict accuracy requirements and that all readings are trustworthy. However, the panel members felt that this is not always the case. Accordingly, the panel members felt that there is a need to revise the long-standing criteria for GM approval to improve patient safety. For these reasons and others mentioned herein, the panel members believe that a new FDA approval standard for GMs that includes accuracy gradients is needed.

Because there is a wide range of issues surrounding the accuracy and intended use of GMs, the panel recommended that a consensus group be formed to set error gradients, evaluate risk allowances, and recommend intended use guidelines.

Specifically, the panel proposed that

• a minimum standard be determined for FDA approval at ±10 mg/dl below 75 mg/dl and ±15% above 75 mg/dl for 95% of readings with less than 2% of readings being more than ±15 mg/dl below 75 mg/dl or more than ±15% above 75 mg/dl;

• different accuracy gradients for GMs be included within an overall graduated approval standard to incorporate the varied needs of individual and hospital users, as well as countries^28–31 that wish to set their own approval standard using narrower or wider criteria;

• both laboratory performance and patient ease of use should be assessed prior to FDA approval;

• each meter undergo annual or biannual off-the-shelf postmarket retesting to ensure their accuracy;

• there be clear public disclosure of each GM’s performance shortly after testing data have been submitted to the FDA;

• in addition to percentage deviations above 75 mg/dl, any potentially serious errors in mg/dl differences, such as differences larger than 30 or 40 mg/dl that might lead to an insulin dosing error, should be indicated;

• all test strip failures that occur during GM studies, such as the appearance of an error code on the GM or failure to get a test result due to application of an inadequate blood sample, should be included in the FDA submission data;

• each meter’s performance be assigned to the clinical setting(s) for which its use is approved;

• GMs and strips be clearly labeled with their specific accuracy capabilities and intended use, employing an easily understood schema;

• GM clock times should be clearly visible on the meter’s display, be easy to change, and not be lost when a battery is changed;

• a reliable and precise tri-level (low, medium, and high) set of glucose control solutions be developed to verify an individual GM’s accuracy in clinical settings; and

• patient education in how to understand and use their GM data should be emphasized by meter manufacturers through provision of easily readable package inserts.

Glossary

Abbreviations

(AACE)

American Association of Clinical Endocrinologists

(ADA)

American Diabetes Association

(BG)

blood glucose

(CGM)

continuous glucose monitoring

(FDA)

Food and Drug Administration

(GM)

glucose meter

(IGC)

intensive glucose control

(ISO)

International Organization for Standardization

(SMBG)

self-monitoring of blood glucose

(TES)

the Endocrine Society

Disclosures

John Walsh is a consultant for Abbott and Roche Accu-Chek, Frank Schwartz has received a Medtronic investigator-initiated research grant, and Robert Vigersky has received a DexCom Corp. investigator-initiated research grant.