Postmarket Surveillance of Blood Glucose Monitor Systems Is Needed for Safety of Subjects and Accurate Determination of Effectiveness in Clinical Trials of Diabetes Drugs and Devices

In this issue of Journal of Diabetes Science and Technology (JDST) Philis-Tsimikas and colleagues report that it was necessary to revise the protocol of a multicenter drug study because of a safety concern that the FDA-cleared blood glucose monitoring system (BGMS) used in this study was not accurate. In this study, compared to a prior similar clinical trial with a similar study population, mean values of hemoglobin A1c (HbA1c) and fasting plasma glucose concentrations were lower, while mean self monitored blood glucose (SMBG) values were higher. Because of safety concerns, the original glycemic data collection system (MyGlucoHealth blood glucose meter + electronic diary) was discontinued and replaced with an alternate BGMS. The actions of the study sponsors were intended to ensure that the scientific integrity of the pharmaceutical trial was not compromised.¹

The Purpose of Blood Glucose Monitoring

BGMSs are an essential tool for monitoring control of diabetes. Marketing of BGMSs is controlled in the United States by the US Food and Drug Administration (FDA). Clearance by the FDA requires conformance to performance guidelines, which have been gradually requiring greater analytical accuracy (see Table 1). Accuracy of results is critically important first, for the safety of subjects to know whether they require immediate treatment to modify their glucose levels and second, so that medications can be titrated to reach a point of maximal effectiveness as defined by each trial. This tool is particularly important during many clinical trials where the outcome of treatment is defined with a blood glucose value.

Table 1.

History of Performance Standards for Self-Monitoring Blood Glucose Systems for Over-the-Counter Use

Year	Source	Percentage of data specified	Details
2003	ISO	95	95% of results must be < ±15 mg/dL if < 75 mg/dl and < ±20% if ≥ 75 mg/dL
2013	CLSI	98	95% of results must be < ±12 mg/dL if < 100 mg/dL and < ±12.5% if ≥ 100 mg/dL
			No more than 2% > 15 mg/dL if < 75 mg/dL and > 20% if ≥ 75 mg/dL
2013 / 2015	ISO	95	95% of results must be < ±15 mg/dl if < 100 mg/dl and < ±15% if ≥ 100 mg/dl
2014	FDA	100	99% of results must be < ±7 mg/dL if < 70 mg/dL and < ±10% if ≥70 mg/dL
			0% of results must be > 15 mg/dL if < 70 mg/dL and > ±20% if ≥ 70 mg/dL
2016	FDA	98	95% of results must be < ±12 mg/dL if < 75 mg/dL and < ±12% if ≥ 75 mg/dL
			98% of results must be < ±15 mg/dL if < 75 mg/dL and < ±15% if ≥ 75 mg/dL
2018	FDA	99	95% of results must be < ± 15% and 99% of results must be < ± 20%

Table has been modified from Krouwer.²

Accuracy is doubly important in a clinical trial where the endpoint is hypoglycemia, because a falsely low reading may lead to an incorrect conclusion that a treatment may not be safe—when it actually is safe. Also, a falsely elevated glucose level can lead to excessive titration of medication and can lead to an increased number of hypoglycemic episodes. A positively biased BGMS will usually manifest itself by two outcomes: (1) an unusually large number of hypoglycemic readings and (2) a lower than expected HbA1c level, because blood glucose levels will be driven downward by excessive and unnecessary medication titration. These two outcomes both occurred in the clinical trial that was discussed in the Philis-Tsimikas article.¹

The risks of a clinical trial are spelled out in an informed consent form (ICF) for a subject to sign. For a trial of a diabetes drug or device requiring blood glucose monitoring for safety, I have never seen an ICF that presents a risk of the study as an inaccurate blood glucose reading that can result in excessive insulin dosing and an increased risk of hypoglycemia. If this were a known problem with BGMSs used in clinical trials, then such a risk might well be considered unsafe by many institutional review boards (IRBs) and patients, and far fewer trials of diabetes products would be authorized by IRBs.

Accuracy Results in a Two-Part Surveillance Study

In this issue Pfützner and colleagues as well as Demircik and colleagues each report one part of the results of a two-part surveillance study that they were contracted by Novo Nordisk to conduct in order to evaluate the analytical accuracy of the BGMS used in the Philis-Tsimikas study. These two accuracy studies were performed generally in accordance with the ISO15197:2015 (the European harmonized version of ISO15107:2013) guidelines with additional data collection in the hypoglycemic range (below 100 mg/dl) where the BGMS was suspected to be most inaccurate. First, Pfützner and colleagues concluded that the tested BGMs did not meet the minimum accuracy criteria specified for this study.³ Second, Demircik and colleagues concluded that the BGMs met repeatability requirements, but their studies also demonstrated a significant positive measurement bias in the low range (below mg/dL).⁴ In addition, in their studies the product failed the ISO15197:2015 criteria for hematocrit interference. These two analytical accuracy studies were a form of postmarket surveillance testing. This type of testing is not the same as registration testing or determining whether a product is FDA or ISO compliant. In a surveillance study to meet predetermined performance criteria, the number of tests performed might be fewer than what is mandated for registration, not every test procedure is necessarily performed, and in some cases, the product testing method or reference method is not performed exactly as mandated by the manufacturer, in order for the surveillance testers to save time or money or for the testing to be more convenient for test subjects.⁵ An ideal surveillance study for accuracy will be performed as closely as possible to registration methods and reasons for any deviation from the required method as laid out in the standard or guidance will be explained.

Both Pfützner and Demircik were not able to obtain their test materials from an environmentally controlled supply chain where the strips could be certified as having been stored at room temperature and humidity before being sent to these investigators. In the two studies, however, all strip lots and devices received at their common test site in Germany from various geographical locations displayed the same measurement bias, which suggested that the inaccuracy was due to a systematic problem, rather than improper storage.

Linearity studies performed by Demircik and colleagues on glycolyzed specimens did not measure pO2, which must be maintained at a steady concentration when evaluating glucose oxidase-based BGMSs like the one tested in these three articles. The literature contains data suggesting both that measurement and formal stabilization of ambient pO2 is⁶ and is not⁷ necessary for accurate testing of BGMSs containing this enzyme.

Although the results by Pfützner, Demircik, and their colleagues cannot be used to definitely conclude that the MyGlucoHealth would not meet registration criteria for accuracy if the product were to be tested now according to all the criteria mandated by FDA or ISO, their two studies are nevertheless important. The results demonstrated by these investigators in the low range (below 100 mg/dl) were very striking. They reported that 203 of 300 specimens tested with the BGMS were outside the ±15% acceptance limit in this range, whereas per ISO 15197 2013 no more than 15 of their specimens should have been outside this range and per FDA 2018 draft, no more than 15 specimens out of every 300 specimens of any blood glucose level should have been outside of this ±15% range (see Table 1). The data collected by Pfützner, Demircik, and their colleagues are highly suspicious for inaccuracy and suggests a hypothesis that the MyGlucoHealth would not pass 15197:2013 or FDA 2018 draft criteria if it were to be tested now for accuracy according to package insert instructions by way of a comparison method exactly specified by either regulatory agency.

Accuracy of BGMSs in the Medical Literature

In the two most extensive literature reviews of BGMS accuracy, approximately half of the FDA–cleared BGMSs selected by investigators met ISO 15197:2003 and approximately three-fourths of the FDA-cleared BGMSs selected by investigators met ISO 15197:2013.^8,9 In 2015, Klonoff and Prahalad published a literature review of articles published between 2010 and 2014 that presented data about the frequency of inaccurate performance using ISO 15197:2003 and ISO 15197:2013 as target standards. Of the reported systems, they identified 33 as having been cleared by the FDA. Among these systems, 24 out of 32 (75%) met ISO 15197:2003 and 15 out of 31 (48%) met the stricter ISO 15197:2013 standard. They concluded that a significant proportion of FDA-cleared BGMSs do not perform at the level for which they were cleared or according to international standards of accuracy, and that such poor performance leads to adverse clinical and economic consequences.⁸

In 2018 King and colleagues extended the Klonoff/Prahalad dataset by three years by assessing the accuracy of blood glucose monitors from articles published between 2010 and 2017 using ISO 15197:2003 and/or ISO 15197:2013 as target standards. Of the reported systems, they identified 59 as having been cleared by the FDA. Among these systems, 43 out of 57 (75%) met the ISO 15197:2003 standard and 26 out of 56 (46%) met the stricter ISO 15197:2013 standard. They concluded that failure to meet performance levels mandated by standards can result in deleterious clinical and economic effects.⁸

Diabetes Technology Society Surveillance Study of Cleared Blood Glucose Monitor Systems

In 2018, Klonoff et al. reported the largest study of the accuracy of FDA-cleared BGMSs. This Diabetes Technology Society study assessed the accuracy of the 18 leading selling BGMSs as of 2015. These products represented approximately 90% of the US market. A total of 1,035 subjects were recruited to have a capillary blood glucose level (BG) measured on six different systems along with a reference capillary sample prepared for plasma testing at a reference laboratory. Products were obtained from consumer outlets and tested in three triple-blinded studies. A compliant BG result was defined as within 15% of a reference plasma value (for BG >100 mg/dL [5.55 mmol/L]) or within 15 mg/dL (0.83 mmol/L) (for BG <100 mg/dL [5.55 mmol/L]), which was similar to the requirement of ISO 15197:2013 standard. The proportion of compliant readings in each study was compared against a predetermined accuracy standard similar to, but more lenient than, current regulatory standards. Only 6 of the 18 systems met the predetermined accuracy standard in all three studies. The authors concluded that cleared BGMSs do not always meet the level of analytical accuracy currently required for regulatory clearance and that this information could assist patients, professionals, and payers in choosing products and regulators in evaluating post clearance performance.¹⁰

Current FDA Position on Postmarket Testing of Devices

In recent years, FDA has been increasingly supportive of programs for postmarket surveillance of cleared medical devices. In 2012 FDA released a bulletin called Strengthening Our National System for Medical Device Postmarket Surveillance. It contained a four-point plan to achieve this goal, which would: (1) establish a unique device identification system and promote its incorporation into electronic health information; (2) promote the development of national and international device registries for selected products; (3) modernize adverse event reporting and analysis; and (4) develop and use new methods for evidence generation, synthesis, and appraisal.¹¹

In 2016, the FDA awarded the Medical Device Innovation Consortium (MDIC) $3 million in seed funding to establish the National Evaluation System for Health Technology Coordinating Center (NESTcc). The Coordinating Center seeks to support the sustainable generation and use of timely, reliable, and cost-effective real-world evidence (RWE) throughout the medical device lifecycle. This can be achieved by using real-world data¹² that meets robust methodological standards; 2) is generated in the course of clinical care and everyday life by patients, providers, or payers; and 3) is intended for enhancing regulatory and clinical decision making.¹³

On November 20, 2018, FDA commissioner Scott Gottlieb, MD, and Jeff Shuren, MD, director of the Center for Devices and Radiological Health, together announced updates to the FDA Medical Device Safety Action Plan to enhance postmarket safety.¹⁴ They stated that their goal was to ensure that the FDA is consistently first among the world’s regulatory agencies to identify and act upon safety signals related to medical devices. They also stated that access to robust and timely data, including more extensive and informative postmarket data and RWE, is central to empowering the FDA to identify, communicate, and act on new or increased medical device safety concerns. They mentioned that such data serves as the foundation of FDA’s commitment to improving US postmarket medical device surveillance and that this commitment is one of the core pillars of their safety plan. They advocated an evolution beyond their current postmarket surveillance system—which is largely passive and relies on device users to report problems, sometimes resulting in underreporting. They stated that FDA is moving to an active surveillance system that relies on RWE and timely receipt of robust safety information.

BGMSs are among the most widely used devices and these devices generate many complaints to FDA. In the 7-month period from January through July 2018, of the 579 357 adverse events reported across all medical devices in the FDA adverse event database called the Manufacturer and User Facility Device Experience (MAUDE) database there were 10,837 adverse events for BGMSs. FDA is clearly aware of the benefits of postmarket surveillance of medical devices, and BGMSs are a type of product where such surveillance can be useful.

What Is Needed for Surveillance of Blood Glucose Monitor Systems Used in Clinical Trials of Diabetes Drugs and Devices

Inaccurate BGMSs used in clinical trials of diabetes drugs and devices expose subjects to safety risks related to unsafe titration of therapy that are generally not disclosed in an informed consent. The reason for lack of this disclosure might be that the inaccuracy of these products is not widely recognized by industry clinical trialists, in spite of many published articles demonstrating inaccurate performance. Furthermore, FDA bases approvals for diabetes products in part on results that are generated from BG values—these include fasting glucose concentrations, mean glucose concentrations, and the number of documented events with low glucose concentrations (hypoglycemic events). In some cases continuous glucose monitoring (CGM) data is used by FDA for approvals and every CGM comes with instructions that if the clinical condition does not match the glucose level generated by the CGM, then the patient should check a BG level with their monitor, presumably because it is expected that the BG value will be highly accurate.

The articles in this issue of JDST by first authors Philis-Tsimikas, Pfützner, and Demircik^1,3,4 demonstrated a potential safety risk to subjects volunteering for clinical trials, who used a BGMS that the three sets of authors found to be inaccurate according to the protocols that they used. While none of the three protocols was exactly the same as what is specified in the ISO and FDA standards, the three studies showed similar results and suggest that this BGMS would not perform according to these standards, if a formal postmarket test or a surveillance test were to be performed on them with methods specified by FDA or ISO.

Accurate performance by a BGMS in a formal surveillance program using an expert consensus-derived surveillance protocol could provide reassurance to stakeholders in clinical trials. These stakeholders include research subjects, research clinicians, product manufacturers, IRBs, and regulatory agencies. Although FDA has announced plans to take actions that will allow them to retire outdated predicates,¹⁵ especially in cases where safer or more effective technology has emerged, I am aware of no serious talk about retiring BGMSs that use old technology. Successful performance by a BGMS in a surveillance program would indicate that a clinical trial using this product is employing safe and accurate BGMS equipment to monitor outcomes and determine treatment. A model for such a program is the Diabetes Technology Society Surveillance Program for Cleared Blood Glucose Monitor Systems, which reported results of 18 BGMSs in 2018.¹⁰ This program tested the 18 leading selling BGMSs as of 2015. A diabetes product manufacturer conducting a clinical trial today might decide to use a newer product than one of those 18, or a legacy product that was not one of the 18 leading selling products in 2015 when the products to be tested were selected. The products in that surveillance trial, reported in 2018, did not did not contain cellular transmission to the cloud. Many products with this feature are now becoming popular for clinical trials because they upload glucose data automatically.

To my knowledge, the Novo Nordisk study¹ is the first study of a diabetes product to ever be significantly reorganized because of suspicion that the FDA-cleared BGMS used in the study was inaccurate and unsafe. Novo Nordisk is to be congratulated for (1) conducting in-house testing, (2) contracting with Pfützner, Demircik, and their colleagues to independently monitor the performance of their study’s FDA-cleared BGMSs, and (3) switching to another brand of BGMS when their data suggested that their study BGMSs appeared to be inaccurate or possibly unsafe.

Conclusions

Unless a formal postmarket surveillance program for identifying inaccurate BGMSs is established, then the experience reported by Philis-Tsimikas and colleagues might occur again. A formal postmarket review program for BGMSs used in clinical trials of diabetes products is needed now to (1) protect the safety of subjects in clinical trials of diabetes products, to (2) produce the most accurate data for regulatory decisions on these products, and (3) align with the FDA’s drive for postmarket review of devices¹⁶ and drugs.¹⁷