Accuracy of an Off-Label Transmitter and Data Manager Paired With an Intermittent Scanned Continuous Glucose Monitor in Adults With Type 1 Diabetes

Abstract

Background:

This work evaluates the accuracy and agreement between the FreeStyle Libre sensor (FSL) and an off-label converted real-time continuous glucose monitor (c-rtCGM) device consisting of the MiaoMiao transmitter and the xDrip+ application which can be coupled to the FSL.

Methods:

Four weeks of glucose data were collected from 21 participants with type 1 diabetes using the c-rtCGM and FSL: two weeks with a single initial calibration (uncalibrated) and two weeks with a daily calibration (calibrated). Accuracy and agreement evaluation included mean absolute relative difference (MARD), the %20/20 rule, Bland-Altman plots, and the Consensus Error Grid analysis.

Results:

Values reported by the c-rtCGM system compared with the FSL resulted in an overall MARD of 12.06% and 84.71% of the results falling within Consensus Error Grid Zone A when the device is calibrated. For uncalibrated devices, an overall MARD of 17.49% was obtained. Decreased accuracy was shown in the hypoglycemic range and for rates of change greater than 2 mg/dL/min. The between-device bias also incremented with increasing glucose values.

Conclusion:

Measurements recorded by the c-rtCGM were found to be accurate when compared with FSL data only when performing daily c-rtCGM device calibrations. High drops in accuracy and agreement between devices occurred when the c-rtCGM was not calibrated.

Introduction

Diabetes mellitus management has been revolutionized with advances in glucose monitoring. Continuous glucose monitors (CGMs) and intermittent scanned glucose monitors (isCGM) have been developed to permit frequent glucose monitoring and to provide time-series glucose data revealing temporal trends in glucose control.^1-3 Various glucose monitors have been developed, assessed, and Food and Drug Administration (FDA)-approved.^4-7 In particular, the isCGM FreeStyle Libre Sensor (FSL) has been reported to have an overall mean absolute relative difference (MARD) of 11.4% ⁸ and 13.2%. ⁹ Other sensor accuracy comparison studies can be found in the literature.^10,11

The CGM and isCGM devices have been shown to improve glycemic control, mitigate hypoglycemic episodes, and reduce glycemic variability in diabetes treatment.^5,12-14 However, factors limiting the distribution of CGM sensors and the time required to develop medical-accepted devices have been criticized in the diabetes community.^3,15 This has led to develop various hardware and software solutions under the scope of “do-it-yourself” technologies and the #WeAreNotWaiting movement, facilitating the availability of open-source platforms and some devices that do not yet have regulatory approval but are gaining popularity among users.^16-18

The “low” cost of the FSL has prompted the development of glucose data transmitters that avoid the isCGM’s scanning requirement and ultimately obtain a functional real-time CGM. One of them that has been highlighted is the MiaoMiao (MM) device, which is placed over the FSL to transmit glycemic data to a paired smartphone with a compatible application such as xDrip+.^15,19-21 Advantages including reduction of fear from hypoglycemia, HbA1c levels, and the number of hypoglycemic and hyperglycemic episodes have been evidenced when using the MM and xDrip+ in comparison with using the FSL alone,^15,19 However, the suitability of these readers in a clinical setting has not been evaluated to date. Similar technologies related to smartphone applications for managing glucose data have been reported in the literature.^16,21-23

As glucose sensing technologies, off-label transmitters, and smartphone apps become more popular, it is important to scrutinize each system’s accuracy because its understanding has implications for treatment decisions made using these technologies. Therefore, the contribution of this work relies on the evaluation of the accuracy and agreement of using the c-rtCGM when coupled with the FSL. In addition, an exploratory report on the quality of the connection between devices was presented in terms of the number of disconnection events. This combination was chosen given its availability in Colombia and its low cost in comparison with other brands.

Methods

Devices Used in the Study

FreeStyle Libre sensor

FreeStyle Libre sensor (version 1) is an isGM device, factory-calibrated, requires one hour to warm up, and has a 14-day wear lifetime.^8,13 The sensor detects glucose levels ranging from 40 to 500 mg/dL. After it is scanned, the current glucose value, a glucose trend arrow, and the last eight hours of glucose readings can be observed. ²⁴ Data from the reader can be downloaded via a USB port and a computer software.

MiaoMiao

It is a transmitter paired with the FSL to obtain a c-rtCGM using near-field communication to read raw data every five minutes. ²⁵ It sends the data via Bluetooth to a paired smart device without going through the official FSL algorithm. ¹⁵ The data must be processed in a nonofficial CGM application to exhibit the glucose value. ¹⁹

xDrip+

This is an open-source CGM application for smartphones that allows the visualization of glucose measurements received from a transmitter. The app accounts for alerts, a predictive low-forecasting feature, and instant data synchronization between phones and transmitters. ²⁶ The only non–open-source feature of the app is the algorithm to convert raw data into glycemic values. The algorithm is based on glucose calibrations typically obtained from the FSL reader or fingerstick measurements. One calibration is required before the app can collect data. Based on this and subsequent data, a linear equation is calculated to perform the conversion, and it is updated every time a calibration is entered.

Smartphone

Each subject received a Kalley’s Element Play Smartphone with a SIM card and the data manager app installed. The device’s operating system is Android 10 Go Edition; it has built-in memory of 32 GB, and RAM memory of 1 GB. It also allows Bluetooth connection to receive the data from the MM transmitter.

Independence of the Study

None of the manufacturers of the devices were involved in the present study.

Study Design and Participants

A single-center comparative reading study was conducted using the c-rtCGM and isCGM FSL. The main outcome was quantitative performance composed of accuracy, agreement, and clinical accuracy of the c-rtCGM related to the FSL. Adults with type 1 diabetes with at least one year since diagnosis and treated with Continuous Subcutaneous Insulin Infusion (CSII) or Multiple Daily Insulin (MDI) were included. Pregnant women and patients with liver insufficiency or chronic kidney disease with glomerular filtration of less than 30 mL/min were excluded.

Participants performed all isCGM and transmitter installation and removals. They were trained on how to use the device in a two-hour session before starting data collection. The isCGM was attached to the participant’s left arm. The data manager application was installed on Android smartphones provided by the study. The configuration master-slave was used to monitor the data collection. The participant held the master smartphone with the blind configuration for the xDrip+ app to avoid any treatment decision-making based on it. The slave device was handled by the study group to monitor the connection and data reception. During sensor use, the subjects were at home in free-living conditions and instructed to perform self-monitoring blood glucose (SMBG) at least once per day. The subjects were also required to scan the sensor at least every seven to eight hours.

Data Analysis

Data preprocessing was performed to remove segments where data were missing from any of the devices. Pairs of glucose values with a sampling frequency of 15 minutes were obtained by pairing data reported by the FSL (15-minute frequency) with data from the c-rtCGM (5-minute frequency) to match the closest timestamp among devices. Accuracy metrics included the MARD, the median absolute relative difference (MeARD), the precision absolute relative difference (PARD), and the %20/20 agreement rate.^27,28 The outcome metrics were computed in the overall range of blood glucose (BG) values and across BG concentration ranges. The metrics were also analyzed at various glucose rate of change (RoC): <−2, −2 to <−1, −1 to <0, 0 to 1, >1 to 2, and >2 mg/dL/min.

Agreement between devices was visually analyzed with the Bland-Altman plot allowing for differential and proportional biases to be estimated in the setting of heteroscedastic measurement errors (as the data are not normally distributed). ²⁹ Also, the Consensus Error Grid analysis was performed, where Zones A and B represent clinically accurate measurements and altered clinical action, respectively, and Zones C, D, and E are defined as altered clinical action that could affect the clinical outcome, could have significant clinical risk, and could have dangerous consequences, respectively. ³⁰

Ethics

The study was approved by the Ethics Committee of the Pablo Tobon Uribe Hospital, Medellin, Colombia. All patients provided informed consent, and their privacy was protected by restricting identification to the physician, while the remaining authors accessed a deidentified database.

Results

Twenty-one subjects completed the study. Baseline patient characteristics are reported in Table 1 and glucose outcome metrics in Table 2. Each participant used the c-rtCGM for 14 days without performing calibrations (only the initial) and additional 14 days with the instruction to perform a daily calibration. After cleaning and pairing of data from both devices, 26 979 pairs of glucose values were available for comparison when considering daily calibrations and 27 286 without calibrations.

Table 1.

Characteristics of the Study Participants.

Characteristic	Mean [IQR] / frequency (percentage)
Age (y)	39.0 [36]
Gender (male)	11 (52.4)
Body weight (kg)	64.0 [20.9]
Height (cm)	160 [20]
Body mass index (kg/m2)	24.6 [5.90]
Multiple daily injections	15 (71.4)
Continuous subcutaneous insulin infusion	6 (28.6)
Insulin daily dose (U)	39.0 [27.7]
HbA1c (%)	7.70 [0.600]
Time in range (%)	58.0 [26.0]
Glucose CV (%)	35.8 [11.3]

Overall N = 21. Continuous variables are reported as median [IQR], and categorical variables are reported as absolute and relative frequencies. Abbreviation: IQR, interquartile range; CV, coefficient of variation.

Table 2.

Population Glucose Outcomes.

	Calibrated		Uncalibrated
	FSL	c-rtCGM	FSL	c-rtCGM
Mean BG (mg/dL)	156.53 ± 23.62	153.89 ± 19.11	143.57 ± 28.95	157.34 ± 33.88
SD of BG (mg/dL)	49.02 ± 16.66	50.63 ± 17.26	42.21 ± 17.04	59.75 ± 22.88
Coefficient of variation (%)	30.74 ± 7.56	32.49 ± 9.38	28.41 ± 8.57	37.34 ± 11.74
Percentage of time in range (%)
BG <54 mg/dL	0.49 ± 1.23	1.17 ± 1.60	0.49 ± 1.18	3.78 ± 4.91
BG <70 mg/dL	1.89 ± 2.83	3.83 ± 4.64	2.95 ± 3.16	7.33 ± 7.38
BG 70-180 mg/dL	68.89 ± 18.81	68.56 ± 18.08	73.73 ± 17.98	59.87 ± 19.54
BG >180 mg/dL	29.22 ± 18.32	27.61 ± 16.08	23.32 ± 17.41	32.81 ± 19.05
BG >250 mg/dL	6.56 ± 8.44	5.96 ± 6.86	3.46 ± 4.70	11.47 ± 11.51
BG >300 mg/dL	1.79 ± 3.31	1.78 ± 2.54	0.91 ± 1.71	3.97 ± 5.99

Values are reported as mean ± SD. Abbreviations: c-rtCGM, converted real-time continuous glucose monitor; FSL, FreeStyle Libre sensor; BG, blood glucose; SD, standard deviation.

The MARD, PARD, MeARD, and %20/20 results between the FSL and the c-rtCGM with and without calibrations are reported in Table 3. When calibrating the c-rtCGM, the overall MARD was 12.06%, and 82.55% of the pairs followed the %20/20 rule. In contrast, for uncalibrated devices, the overall MARD was 17.49%, and 65.01% of the pairs met the %20/20 rule. The MARD for individual sensors is shown in Figure 1.

Table 3.

Difference Analysis Between FSL and the c-rtCGM Readings.

	Calibrated				Uncalibrated
	MARD (%)	PARD (%)	MeARD (%)	%20/20	MARD (%)	PARD (%)	MeARD (%)	%20/20
Overall	12.06	12.18	9.68	82.55	17.49	17.08	15.04	65.01
BG <70 mg/dL	24.47	23.25	21.31	64.14	27.38	31.68	30.88	60.70
BG 70-180 mg/dL	11.87	11.98	9.47	82.30	16.89	16.81	14.10	66.66
BG 180-250 mg/dL	11.24	11.47	9.19	84.84	18.39	16.47	16.57	59.47
BG 250-300 mg/dL	12.42	13.10	12.08	83.76	20.08	17.67	18.97	53.02
BG >300 mg/dL	11.84	12.44	11.95	84.94	13.55	12.34	13.51	73.68
ROC <−2 mg/dL/min	12.78	11.91	9.68	80.91	21.20	19.23	19.15	53.90
ROC −2 to <−1 mg/dL/min	11.65	11.25	9.01	83.57	18.47	17.42	17.01	60.18
ROC −1 to <0 mg/dL/min	11.39	11.16	8.97	84.40	17.49	16.90	15.15	65.01
ROC 0 to 1 mg/dL/min	11.05	11.15	8.57	86.08	16.37	15.99	13.91	68.24
ROC >1 to 2 mg/dL/min	10.79	11.38	8.92	87.81	16.07	16.32	13.28	70.67
ROC > 2 mg/dL/min	15.11	16.87	14.07	71.65	15.78	17.23	11.66	71.02

Abbreviations: c-rtCGM, converted real-time continuous glucose monitor; FSL, FreeStyle Libre sensor; MARD, mean absolute relative difference; PARD, precision absolute relative difference; MeARD, median absolute relative difference; BG, blood glucose; ROC, rate of change.

Figure 1.

Histogram of the MARD per sensor for all values of blood glucose. Abbreviation: MARD, mean absolute relative difference.

Accuracy across glucose ranges and RoC is included in Table 3. For both calibrated and uncalibrated c-rtCGMs, the lowest similarity among sensors was obtained for BG values <70 mg/dL. For calibrated devices, the pairs where the FSL reported values between 180 and 250 mg/dL resulted in better accuracy with the c-rtCGM. For uncalibrated devices, the results for the range between 70 and 180 mg/dL are highlighted, with MARD being 16.89%, and 66.66% of the pairs accomplished the %20/20 agreement rule.

Regarding the glucose RoC ranges, the highest accuracy among sensors considering calibrations occurred when the FSL reported values increasing by no more than 2 mg/dL/min. The lowest performance, with MARD of 15.11%, PARD of 16.87%, and 71.65% of pairs following the %20/20 rule, was obtained when RoC was >2 mg/dL/min. In the case of uncalibrated device, better accuracy is reported when the values increase by more than 2mg/dL/min, and declining accuracy is obtained when glucose decreases by more than 2 mg/dL/min.

Figures 2 and 3 show the extended Bland-Altman plot for measurements with and without calibrations and the Consensus Error grid, respectively.

Figure 2.

Left panels: Bland-Altman plot for measurements (a) with calibrations and (b) without calibrations. Right panels: Bias plot. Abbreviations: c-rtCGM, converted real-time continuous glucose monitor; FSL, FreeStyle Libre sensor; LoA, limits of agreement; BLUP, best linear unbiased predictions.

Figure 3.

Consensus error grid analysis of FreeStyle vs MiaoMiao reported measurements. Left panel: results when considering calibrations for the c-rtCGM. Right panel: results with uncalibrated c-rtCGM. Abbreviations: c-rtCGM, converted real-time continuous glucose monitor; FSL, FreeStyle Libre sensor.

Also, the main events associated with data loss are summarized. In total, 20 data loss events were reported. The duration of the events is reported in Figure 4(a) and the method to resolve the issues can be seen in Figure 4(b). For manual solutions, a troubleshooting guide developed by the study group was followed, and personal assistance to the participant was given if necessary. Other events were reported but not related to the transmitter or the application. These were: sensor disconnection from the participant’s body, unexpected discharge of the MM battery, and failure of the FSL.

Figure 4.

(a) Duration of events requiring troubleshooting. Blue events are related to data loss despite having a stable connection with the smarthphone app, and orange events are associated with disconnection from the app. (b) Method to reestablish data acquisition for data loss events despite a stable connection with the app. Abbreviation: c-rtCGM, converted real-time continuous glucose monitor.

Discussion

This study compared the accuracy and agreement of a c-rtCGM with the FSL outcomes. Overall, a similar MARD among both devices to that reported for the FSL related to SMBG values was obtained, ⁹ with a substantial number of measurements following the %20/20 rule. However, these results are limited to the c-rtCGM with daily calibrations, and for glucose ranges 70-180 mg/dL and >180 mg/dL. This is also evidenced in the outcome metrics in Table 2, as there are no significant clinical differences between devices. When BG <70 mg/dL, calibrations would be recommended to decrease potential errors. Note that the reported time in hypoglycemia differs by about 2%, which may be significant. When no calibrations are used, clinically significant differences are obtained across all ranges.

The differences between both devices are also observed in the Bland-Altman plot. From Figure 2, there is a proportional bias for both calibrated and uncalibrated c-rtCGMs. Lower glucose values show a negative bias and higher values show a positive bias. Despite the uncalibrated devices exhibiting the same trait, the proportional and differential biases are more pronounced (almost 10 times greater than when calibrating).

Regarding the effect of glucose RoC, it can be noticed that under stable conditions, with glucose fluctuations less than 2 mg/dL/min, the MARD was under 12% when calibrating the device and under 17.5% when uncalibrated. The MARD increases as the glucose RoC increases. The distinctions obtained at different RoC can be explained by the linear mapping function used by the c-rtCGM to transform the raw value into the resulting outcome. When the glucose rapidly changes, it can deviate quickly from the mapping curve, obtaining a larger error. Therefore, for rapid variations in glucose, a calibration would be recommended before making an insulin-dosing decision.

Despite the uncalibrated c-rtCGM’s lower accuracy, both groups reported low clinical risk, with more than 99% of readings falling within the Consensus Error Grid Zones A and B. However, for uncalibrated devices, a tendency was observed toward zone upper B as the glucose values increased. This trend was also observed in the Bland-Altman plot. Nevertheless, this trend appears to be deterministic, as when there are no calibrations, the mapping function of the raw data is not updated and the difference grows.

The accuracy results for the calibrated device have important clinical implications as one of the main advantages of CGMs is the ability to obtain real-time glucose level readings as well as glucose trends, providing the information to make insulin-dosing decisions.

Limitations of the current study include that a single set of devices were used per subject at once, rather than performing the calibration/no-calibration comparison at the same time using two sets of equipment on the same participant. In addition, no randomization was considered for the study, as for all subjects the first two weeks correspond to the uncalibrated c-rtCGM device and the following two weeks to the calibrated device. The study also had limited controlled conditions due to participants’ free living conditions, leading to a delay in detecting connection problems and in calibration. An event-driven calibration was not performed (eg, every time BG <70 mg/dL), but rather once a day in the morning hours. This calibration method was chosen to avoid overburdening the participants.

Conclusion

A comparative analysis of the accuracy and agreement between the FSL and a c-rtCGM was performed. The results obtained show high accuracy and agreement between the measurements of both devices as long as daily calibrations are provided to the c-rtCGM. The accuracy and agreement between devices drop considerably when daily c-rtCGM calibrations are not performed. This suggests performing daily calibrations, or even event-driven calibrations, to improve agreement among devices. In addition, an exploratory report of disconnections was made with the c-rtCGM, showing that the transmitter and the application generally are stable, which is promising for a device that could be coupled with a closed-loop insulin infusion system.