Analysis of “Multicenter Evaluation Study Comparing a New Factory-Calibrated Real-Time Continuous Glucose Monitoring System to Existing Flash Glucose Monitoring System”

Abstract

In an article in the Journal of Diabetes Science and Technology, Ji et al. report on the accuracy of a new factory calibrated continuous glucose monitoring system, the AiDEX CGM (Microtech Medical Company, Ltd., Hangzhou, China). This is the first report from a new manufacturer of a highly accurate factory calibrated CGM. The authors report that the accuracy of the AiDEX CGM is comparable to previous results from Abbott Diabetes Care and Dexcom. However, the study protocol was significantly different from previous studies. This study is the first of numerous studies likely from other manufacturers of CGM technology. It raises the question of how to evaluate sensor performance in the coming era of mass adoption of CGM and increased use of automated insulin delivery systems.

The number of patients using continuous glucose monitoring (CGM) systems has increased dramatically in recent years. Current estimates are that 5 million people with diabetes worldwide may be using CGM systems – approximately 1 million on Dexcom and 4 million on Abbott Diabetes Care CGM devices. There are many factors responsible for this growth, most notably the greater awareness of the clinical benefit of CGM by doctors, nurses, people with diabetes and caregivers. However, the recent growth in CGM adoption has been accelerated as well by the regulatory approval for use of the devices without confirmatory fingersticks and the development of factory calibrated systems that obviate the need for intermittent fingersticks for calibration. As the adoption of CGM increases further worldwide and as more people with diabetes use automated insulin delivery systems, it will be increasingly important to have mature and well-validated methods for evaluating the accuracy and reliability of new and old CGM systems.

In an article in this journal, Ji et al ¹ describe the results of a clinical study of a new factory calibrated CGM system, the AiDEX CGM system developed by Microtech Medical Company, Ltd. (Hangzhou, China). The sensor is pre-assembled into the applicator which is deployed with a simple single step procedure. The system uses a re-usable transmitter and has a dedicated receiver with a built-in blood glucose meter. Study results were reported from a relatively small study (n = 124) and showed MARD of 9.08% compared with venous reference and 10.2% compared with capillary fingerstick reference. The authors compared their results to previously published results from Dexcom and Abbott Diabetes Care for their factory-calibrated systems which reported MARDs of 9.0% and 9.4%, respectively.

However, the design of the AiDEX CGM study may make it difficult to draw definitive comparisons between this study and the cited studies by Dexcom and Abbott Diabetes Care. First, the AiDEX study was done primarily in subjects with type 2 diabetes (88.7%) whereas the Abbott Diabetes Care and Dexcom studies cited in the paper were done primarily in subjects with type 1 diabetes (approximately 80% to 100% depending on the study). Patients with type 2 diabetes typically have less glycemic variability and hence lower rates of change than patients with type 1 diabetes. ² Since high rates of change are associated with larger CGM errors, ³ the subject demographics of different studies may make it difficult to compare simple metrics such as MARD. Studies of new CGM technology should provide documentation of the glucose and rate of change distributions in their data such as simple histogram plots to facilitate comparison with other studies. Second, the AiDEX study excluded data collected on day one immediately after sensor insertion. Since many CGMs have widely reported discrepancies on the first day after insertion compared with subsequent days, this adds to the challenges associated with comparing studies with different protocols.

The mean absolute relative difference (MARD) is a simple and widely used metric for characterizing CGM accuracy. In 2009, Wentholt and DeVries ⁴ wrote a paper on newly published results from the Dexcom Seven in which they asked whether the change in MARD from the Dexcom STS (23%) to the Dexcom Seven (13.2%) represented a“summit for the accuracy of needle-type sensors”. In the article, they speculated whether CGMs would ever achieve MARDs of 10% or less. Six years later, Kovatchev ⁵ used theoretical analysis and in silico methods to conclude that a MARD of 10% reported by Dexcom was sufficient to provide safe use of CGMs for non-adjunctive use.

Reiterer et al ⁶ and Heinemann et al ⁷ have written critiques on MARD as a figure of merit for CGM accuracy, particularly for comparison between different CGM systems studied in different clinical studies. One obvious shortcoming of the use of MARD as a figure of merit for CGM accuracy is that it represents an aggregate value over an ensemble of individual sensors. A low aggregate value of MARD for a large number of sensors may include a broad distribution with numerous outlier sensors with MARDs of 20%, 30%, 40%, or higher. A number of authors have supplemented the aggregate MARD for all sensors with a histogram distribution of individual sensor MARD values. Improvements from one generation of CGM device to another were shown by Christiansen et al ⁸ from changes in the histogram distribution of individual sensor MARDs comparing the Dexcom G4 Platinum with its predecessor, the Dexcom Seven Plus. Garcia et al, ⁹ similarly, demonstrated improvements in accuracy with a prototype of the Dexcom G5, the Dexcom G4AP developed for use in artificial pancreas clinical studies, compared with its predecessor, the G4 Platinum, from the narrower width of the histogram and the fewer number of outlier sensors. Bailey et al ¹⁰ published the results of a study of an early version of the Abbott Freestyle Libre with an aggregate MARD of 11.2% compared with blood glucose meter values and a histogram distribution of the MARD per sensor as a means of illustrating the extent of variation of response between sensors. Finally, Denham ¹¹ used histogram distributions of individual sensor MARD to compare the first day performance of 2 factory-calibrated systems, the Abbott FreeStyle Libre 2 and the Dexcom G6. Although the aggregate MARD given by Ji et al in their paper is impressive, especially for a factory-calibrated device from a new CGM manufacturer, it would be helpful to understand the MARD distribution of individual sensors in their study and in future studies with other new CGM devices.

Individual sensors with anomalously high MARD values may increase the risk for adverse events. Shapiro^12-14 highlighted the importance of the clinical consequences of individual sensor outliers in a series of 3 papers questioning the FDA’s approval of the Dexcom G5 for non-adjunctive use. He examined data from the FDA Manufacturer and User Facility Device Experience (MAUDE) database and found more than 25,000 reports of sensor inaccuracy with the Dexcom G5 CGM from January 2015 to October 2016 and another 15,000 or more reports of sensor inaccuracy from November 2016 to June 2017 ¹⁴ . Although the average accuracy (aggregate MARD) had improved with the Dexcom G5 compared with previous generations of Dexcom CGMs, as Shapiro noted, reports in the MAUDE database strongly suggested problems with sporadic large errors with the sensors. Furthermore, these errors resulted in numerous adverse clinical events including loss of consciousness, seizures, automobile accidents, diabetic ketoacidosis, hospitalizations, ICU admissions and deaths. More recently, Krouwer^15-17 has published a series of papers analyzing reports of injuries associated with CGM inaccuracy in the MAUDE database. Krouwer found an increase in injuries associated with CGM from 1,623 in 2018 to 5,937 in 2019 and 4,710 in 2020 overlapping the period in which the FDA approved CGM devices for non-adjunctive use and 2 manufacturers, Dexcom and Abbott Diabetes Care, introduced factory-calibrated sensors. Krouwer notes that although these numbers are large, the numbers of CGM users are larger still hence these events must be considered as rare occurrences. Unfortunately, as he notes, clinical studies for regulatory approval are conducted for the purpose of evaluating CGM performance. These studies typically have relatively small samples (n ≤ 300) and are often done under well-supervised conditions. The risk of adverse clinical events from large sporadic CGM errors may not be apparent from initial regulatory clinical studies but then manifest itself later when the products are on the market in tragic reports of injury in the MAUDE database and elsewhere.

The paper by Ji et al cited small numbers of points in the erroneous zones of the Parke consensus error grid (6 examples in hypoglycemia, 10 examples in euglycemia and 33 examples in hyperglycemia). No further details were provided. In the Special Controls released by the FDA with the de novo classification of the Dexcom G6 as a Class II product under the new iCGM designation, the FDA noted two conditions for which they required there be no occurrences: i) sensor values less than 70 mg/dL when the corresponding actual blood glucose value is greater than 180 mg/dL and ii) sensor values greater than 180 mg/dL when the corresponding actual blood glucose value is less than 70 mg/dL. Reports of new CGM systems, such as the factory-calibrated system described by Ji et al in their paper, should report on these criteria as well. More importantly, future CGM systems should have improved methods of fault detection in order to identify these large sensor anomalies and alert the user before they result in life-threatening clinical events.

CGM adoption is likely to continue to increase as CGM becomes the standard of care for type 1 diabetes and as clinical studies continue to find benefit in type 2 diabetes. There will likely be numerous new CGM systems from other manufacturers such as the Microtech Medical Company AiDEX device in the paper by Ji et al. These systems will have to achieve comparable accuracy, cost of goods and ease of use relative to existing systems in order to compete in the marketplace. There is an opportunity for new systems to differentiate themselves from existing CGM devices by not only having comparable average accuracy (eg, aggregate MARD) but a tighter distribution of individual sensor MARD values, fewer outlier sensors and fewer large sporadic errors. The increase in adoption of automated insulin delivery systems amongst people with type 1 diabetes further heightens the need for new CGM devices that have high accuracy and reliability. The combination of non-adjunctive use, factory calibration and automated insulin delivery places greater burdens on sensor reliability than previously. Future CGM systems may require multiple redundant sensors as well as internal reference sensors to protect against outlier measurements, large sporadic errors and the risk of rare but catastrophic clinical outcomes.