Post-Market Surveillance Assessment of the Clinical Accuracy of a Blood Glucose Monitoring System with an Improved Algorithm for Enhanced Product Performance

Abstract

Background:

On-going manufacturer-led post-market surveillance (PMS), assessing the clinical accuracy of blood glucose monitoring (BGM) systems, is critical to substantiate the performance of such products for people with diabetes.

Materials and Methods:

Batches of Verio test-strip product were randomly and routinely selected over the period from launch of an improved-algorithm product to reporting date and sent to 3 clinic sites for clinician-led accuracy assessment. Accuracy is reported as per recently adopted FDA guidance for BGM systems, EN ISO 15197:2015 and MARD/MAD (Mean absolute relative difference/Mean absolute difference).

Results:

Thirty-three individual test-strip batches were evaluated corresponding to 506 unique donors. Accuracy performance - FDA: 98.9% of values within ±15% of comparator; ISO: 99.0% within ±15 mg/dL or ±15% at <100 mg/dL (<5.55 mmol/L) or ≥100 mg/dL (≥5.55 mmol/L) glucose, respectively. Overall MARD was 4.19% with a MARD range of 3.54%-5.73% across all test strip batches.

Conclusions:

This post-market surveillance program demonstrates the new BGM system consistently meets measures of clinical accuracy specified by regulators. This program supports a growing demand by regulators for real-world evidence demonstrating consistent in-market product efficacy as opposed to the current largely passive approach that relies on assessment of reports filed by device users.

Introduction

Blood glucose monitoring (BGM) systems remain critical tools for most people globally with diabetes who rely upon them to support diabetes management decisions where access to continuous glucose monitoring (CGM) systems is limited. Approval to market BGM products in the United States is governed by the Food and Drug Administration (FDA) who provide guidance regarding levels of performance for regulatory clearance. ¹ From a user-evaluation perspective, BGMs must demonstrate 95% of all results to be within ±15% of the corresponding comparator result and 99% of results within ±20% of comparator across the claimed range. While not required by FDA guidance, MARD (Mean Absolute Relative Difference) provides another measure of accuracy, most widely quoted for CGM systems, but frequently used as an additional measure of BGM accuracy. To better meet end-user needs, BGM test-strip manufacturers strive to deliver new designs with improved clinical accuracy to market.

Regulatory submission of a BGM system is based on data from testing a relatively small number of batches of representative product, most likely manufactured within a short timeframe prior to that submission. However, the performance of test-strips should be continuously assessed over time (in-clinic) to ensure product quality and to demonstrate continued precision over the manufacturing process. Consequently, post-market surveillance (PMS) of BGM systems has been an ongoing area of concern, ² with the Diabetes Technology Society (DTS) undertaking their own PMS exercise in 2016-2017 to assess the clinical performance of the 18 most sold BGMs in the U.S. market at that time.^3,4

As commercially available products, the responsibility for continued assurance of clinical accuracy claims should lie with the legal manufacturer. LifeScan, as legal manufacturer of BGM systems routinely conducts such PMS.^5-8 Reported here is the accumulated surveillance data, since launch, of the U.S. version of the Verio Reflect BGM system, which incorporates a next-generation algorithm delivering improved clinical accuracy.

Methods

Test-strip

The Verio test-strip design incorporates co-facially opposed thin-film palladium and gold layers, separated by an insulating spacer, described in further detail elsewhere. ⁵ The palladium surface of the sample chamber (0.4 µL volume) is coated in reagent, that includes the enzyme FAD-GDH (flavin adenine dinucleotide-glucose dehydrogenase) that rapidly-solubilizes on blood application. A pulsed voltage-time waveform is applied during the 5 second test. The information-rich current-time transient response is algorithmically processed to give a hematocrit, temperature and interferent corrected response. The US Verio Reflect BGM system evaluated here employs an improved form of algorithm that further reduces the effect of electroactive interferences on system output.

Clinic Measurement

Clinical accuracy was assessed across 3 manufacturer-affiliated UK clinic sites: Highland Diabetes Institute, Inverness; Birmingham Heartlands Hospital, Birmingham; Royal Infirmary, Edinburgh. Each week, a minimum of one test-strip batch meeting the manufacturer’s product release criteria was randomly selected and sampled for clinic evaluation purposes. In clinic, a target of 100 strips per batch were assessed on 100 individual subjects with capillary fingertip blood samples being drawn and tested as per manufacturer’s instructions for use. Study inclusion criteria included a diagnosis of diabetes, a measured hematocrit of 20%-60% v/v and informed consent. No exclusions were made based on medications. Immediately following BGM measurement, 2 additional capillary samples were taken: A minimum 200 µL sample, collected into a 300 µL Microvette blood collection tube (Lithium:Heparin) for glucose determination by the comparator system. The time period between end of Microcuvette collection and start of centrifugation did not exceed 1 minute. An additional ~75 µL capillary blood was collected in a heparinized capillary tube for hematocrit measurement using a Hettich Hematocrit 210 capillary centrifuge.

All testing was facilitated by trained clinic staff. Following study completion, the validity of all datapoints were assessed against standard clinical acceptance criteria (hematocrit values within BGM claimed range, valid paired comparator value, adherence to protocol-time limits). The dataset is not entirely composed of unique donors since subjects may have visited the clinic site on more than one occasion over the evaluation period.

Comparator System

Microvette capillary blood samples were immediately centrifuged, with glucose content of the plasma fraction being determined, in duplicate, on 2 separate YSI 2300 STAT PLUS™ analyzers (Yellow Springs Instrument Co Inc, Yellow Springs, OH). The performance of the YSI 2300 (banks of 4 instruments) has previously been assessed in our laboratories using NIST standard reference material 965b (glucose in frozen human serum; levels 1, 2, 3, and 4). Pooled standard deviations, mean differences, mean errors and accuracy metrics were assessed and met the prescribed acceptance criteria.

Clinical Accuracy Calculations

The difference (bias) of a BGM value (test i) vs the corresponding comparator-derived value (YSI comparator i) was calculated as either absolute or relative difference values where:

$$AbsoluteDifferencei{}_{(BGM)}=testi-YSIcomparatori$$

and

$$\begin{array}{l}RelativeDifferencei{}_{(BGM)}=\\ \frac{\mathbf{t}\mathbf{e}\mathbf{s}\mathbf{t}i-\mathbf{Y}\mathbf{S}\mathbf{I}\mathbf{c}\mathbf{o}\mathbf{m}\mathbf{p}\mathbf{a}\mathbf{r}\mathbf{a}\mathbf{t}\mathbf{o}\mathbf{r}i}{\mathbf{Y}\mathbf{S}\mathbf{I}\mathbf{c}\mathbf{o}\mathbf{m}\mathbf{p}\mathbf{a}\mathbf{r}\mathbf{a}\mathbf{t}\mathbf{o}\mathbf{r}i}\times 100\%\end{array}$$

FDA accuracy analysis requires the percentage of relative difference values within ±15% or ±20% of comparator to be recorded, while EN ISO 15197:2015 accuracy analysis requires the percentage of values within either ±15 mg/dL (glucose <100 mg/dL) or ±15% (glucose ≥100 mg/dL) of comparator, respectively to be recorded.

MARD calculation: Firstly, the absolute (Abs) relative deviation (ARD) for a BGMS value (test i) and corresponding comparator value (YSI comparator i) was calculated:

$$\begin{array}{l}ARD(\%)=\\ \frac{Abs\left[\mathrm{Difference}\left(\mathrm{test}i,\mathrm{YSI}\mathrm{comparator}i\right)\right]}{\mathrm{YSI}\mathrm{comparator}i}\times 100\%\end{array}$$

Mean absolute relative difference (MARD) from N paired test and comparator values was then calculated:

$$MARD(\%)=\frac{1}{N}\sum _{i=1}^{i=N}ARD$$

Absolute difference instead of ARD may be used at low glucose:

$$\begin{array}{l}Absolutedifferenc{e}_{(glucose<100mg/dL)}\\ =Difference(testi,YSIcomparatori)\end{array}$$

From which mean absolute deviation (MAD) may be determined. MAD is often quoted at low glucose as it provides a more readily interpretable quantitative measure of the difference between a test system and comparator.

Legacy-algorithm Performance Evaluation

The production-standard BGM meters used in clinic permitted post-measurement assessment of individual current-time transients gathered via the manufacturer’s Medidata RAVE clinical registry (Medidata Solutions, NY). This allowed post-processing of the transient library using the both new algorithm and legacy algorithm code, facilitating direct comparisons between the next-generation and legacy-algorithm.

Results

Clinical Accuracy by FDA and EN ISO 15197:2015 Criteria

A total of 33 unique test-strip batches were evaluated equating to 3273 valid values from 506 unique donors. The distribution of clinic BG values, determined by comparator (Table 1) have a similar distribution to those seen in the manufacturer’s previous clinic PMS datasets.^5-8

Table 1.

Comparator Method Glucose Value Distribution.

Glucose range (mg/dL)	Number of values	Percent (%)
≤50	10	0.3
50-≤80	40	1.2
80-≤120	320	9.8
120-≤200	1274	38.9
200-≤300	1094	33.4
300-≤400	422	12.9
>400	113	3.5
Total	3273	100.0

Accuracy was assessed as per FDA and ISO 15197 accuracy definitions (Table 2), although it is to be appreciated that the methodology followed differed from current FDA and ISO 15197 accuracy assessment methodologies that require the user to perform the test. ISO 15197 has a further accuracy assessment termed “system accuracy,” which permits a health care professional to perform the test but requires glucose values to be proportionally distributed within prescribed glucose ranges. However, this latter approach permits artificial manipulation of capillary blood samples to facilitate meeting the glucose distribution profile at extremes of glucose. At glucose levels <100 mg/dL, 99.4% (173/174) values were within ±15 mg/dL of comparator, while at glucose levels ≥ 100 mg/dL, 99.0% (3067/3099) values were within ±15%. A total of 99.6% of values (3261/3273) were within zone A of the Consensus Error Grid (CEG), with all other values within zone B. Figure 1 records meter relative difference values vs comparator values and indicates no systematic deviation in relative difference as a function of comparator glucose concentration with maximum recorded relative difference values of +28.3% and −26.3%.

Table 2.

Test-strip Performance vs FDA and ISO 15197: 2015 Clinical Accuracy Criteria.

Criterion	Sample size (n)	Within accuracy criterion (n)	Within accuracy criterion (%)	Lower confidence interval (%)	Pass/fail
FDA (±15%) 1	3273	3238	98.9	98.6	Pass
FDA (±20%) 2	3273	3262	99.7	99.4	Pass
ISO (±15 mg/dL or 15%) 3	3273	3240	99.0	98.7	Pass
Zone A + B of consensus error grid 4	3273	3273	100.0	na	Pass

¹95% of results to be within ±15% of corresponding comparator result.

²99% of results to be within ±20% of comparator result.

³95% of results to be within ±15 mg/dL (0.83 mmol/L) of corresponding comparator result at glucose values <100 mg/dL (5.55 mmol/L) or 15% of comparator result at glucose ≥100 mg/dL (5.55 mmol/L).

⁴At least 99% of values, to be within zones A + B of the Consensus Error Grid (CEG).

Figure 1.

Individual meter results expressed as percent difference values to corresponding comparator value (n = 3273).

The Surveillance Error Grid (SEG) was developed to classify the risk posed by the magnitude of the inaccuracy of a glucose value (from a BGM or CGM) subsequently leading to an inappropriate treatment decision.^9,10 Eight levels of risk have been defined, ranging from “none” (risk level 0-0.5), through to extreme (>3.5). SEG analysis showed 99.3% (3249/3271) of the BGM values to be within the no risk zone, while 0.7% (22/3271) of values scored in the slight risk (lower) zone (Figure 2). No values were recorded in any of the other risk zones.

Figure 2.

Surveillance Error grid (SEG) plot of individual meter glucose value vs corresponding comparator value (n = 3271). Note: Two values from the n = 3273 dataset could not be included in SEG analysis since while the comparator values were below the 600 mg/dL cut-off, the corresponding meter values were >600 mg/dL.

Clinical Accuracy by MARD and MAD

The MARD for the full clinic dataset of 33 strip batches was 4.19% (Table 3). Applying an ISO 15197 equivalent glucose cut-off criterion to the dataset yielded a MARD of 4.17% (glucose ≥100 mg/dL) and a MAD of 3.88 mg/dL (glucose <100 mg/dL). MAD was also calculated for the 33 values within the hypoglycaemic range (<70 mg/dL) where a mean absolute deviation of 2.58 mg/dL was recorded. To quantify batch-to-batch differences, MARD was calculated by individual batch, ranging from 3.54%-5.73% across the 33 batches.

Table 3.

Clinical PMS Dataset Expressed by Mean Absolute Relative Deviation (MARD) and Mean Absolute Deviation (MAD) Across Different Glucose Ranges.

Metric	Glucose range (mg/dL)	MARD or MAD value	Number of paired data values
MARD	39-599	4.19%	3273
MARD	≥100	4.17%	3099
MAD	<100	3.88 mg/dL	174
MAD	<70	2.58 mg/dL	33

Next-Generation Algorithm vs Legacy-algorithm

The next-generation algorithm gave 98.9% of values within ±15% of comparator (lower confidence interval [LCI] 98.6%), a higher figure when compared to the legacy-algorithm (97.6% [97.1%]). Corresponding figures within ±20% of comparator for next-generation and legacy algorithms were: 99.7% [99.4%] and 99.5% [99.2%], respectively. Against the ISO 15197-derived criterion of ±15 mg/dL/±15%, 99.0% [98.7%], and 97.6% [97.2%] of values met the requirement for next-generation and legacy, respectively. Both algorithms gave 100% of values within zones A and B of the CEG, with 12 and 19 values within zone B for the next-generation and legacy algorithm, respectively.

An improvement in MARD performance was similarly observed by application of the improved algorithm, with reductions in overall MARD from 5.17% to 4.19% and MARD (≥ 100 mg/dL glucose) from 5.10% to 4.17%. Similar reductions in MAD were also recorded, falling from 5.28 mg/dL to 3.88 mg/dL (<100 mg/dL glucose) and 3.51 mg/dL to 2.58 mg/dL (<70 mg/dL). The range of MARDs by batch for the next-generation algorithm was 3.54%-5.73%, an improvement over legacy-algorithm performance of 4.37%-6.06%.

Discussion

Accuracy

The BGM subjected to this on-going post-market surveillance (PMS) meets clinical accuracy requirements, as set out by current FDA guidance. This HCP-led approach represented the most pragmatic means for the manufacturer to obtain clinical performance data from finger-tip capillary bloods on an on-going basis for multiple strip-batches and BGM platforms. For the record, the performance of this BGM system in terms of lay-user evaluation, clinical validation and extreme glucose evaluation has been reported elsewhere. ¹¹ This dataset, comprising 3273 paired BGM-comparator values, represents an approximately 10-fold increase in the paired data-points required for an FDA lay-user clinical verification study. As this PMS activity continues, the growing dataset will be used to continually assess product performance.

MARD

The FDA approach of reporting clinical accuracy in terms of percentage of results within a given relative value of the corresponding comparator value is binary: values are either within or not within the acceptance threshold. MARD offers an alternative measure of accuracy, serving to quantify the extent of the mean deviation of a set of test system results from the comparator method. That is not to say that MARD alone should be an indicator of accuracy, since it is non-directional in terms of indicating a positive or negative bias to comparator. Furthermore, MARD has been shown to be influenced by multiple factors^12,13 including glucose concentration and time of day of measurement. ¹⁴

The overall MARD of 4.19% (range 3.54-5.73) recorded in this study compared well to leading BGM products derived from independent evaluations. ¹⁵ While MARD does not differentiate between systematic errors (bias) and random errors (precision), a product with a consistently low batch MARD is indicative of a BGM with a relatively narrow systematic error. Bedini et al. ¹⁶ have modelled the effect of hypothetical bolus insulin dosing errors based on BGM MARD accuracy, concluding that systems with lowest MARD reduce the likelihood of clinically significant events. Furthermore, a product with lower MARD is more beneficial since it would manifest itself, in the hands of the end-user, as a less perceptible change in reading as the user starts a new vial of strips from a new strip batch.

The SEG risk levels are such that, at low glucose (<100 mg/dL), a small absolute (mg/dL) inaccuracy in a BGM value can result in a significant shift in risk level. While points plotted on the SEG represent individual BGM-comparator pairings, the overall MAD values of 3.88 mg/dL (n = 174) and 2.58 mg/dL (n = 33) for glucose values <100 and <70 mg/dL respectively indicate how BGM performance supports diabetes management decisions within this sensitive low euglycaemic/hypoglycaemic range.

Several efforts have been made to relate MARD values to the likelihood of meeting ISO performance. Pardo and Simmons ¹⁷ explored the relationship between MARD and ISO, concluding that batches with MARDs of 4.25% were “most likely” to meet the ISO standard, with values of 3.25% and 5.25% representing the “most stringent” and “most liberal” conditions, respectively). However, this analysis was based on a modelling approach. An empirical evaluation by Freckmann et al. ¹⁸ summarizing the results of 77 separate system accuracy studies (169 BGMs, 809 test-strip lots) performed in their laboratory found that all strip batches with a MARD of ≤6.1% met EN ISO 15197:2015, while batches with MARD of ≤4.1% yielded performance equivalent to ±10 mg/dL/±10% ISO accuracy.

Next Generation Algorithm

Most BGM systems use redox enzymes such as glucose oxidase or glucose dehydrogenase as specific biorecognition agents to oxidize glucose in the sample with transfer of electrons facilitated by reduction of a mediator. For electrochemical BGMs, the reduced mediator is re-oxidized at a suitably positively charged working electrode. However, certain compounds in blood, whether exogenous or endogenous, may also undergo oxidation at the electrode, adding a non-glucose specific contribution to the raw signal. BGM or CGM devices (whether using blood or interstitial fluid), will contain endogenous interferences, such as uric acid, and may contain exogenous interferences, notably ascorbic acid or acetaminophen, that are electroactive and so may also affect the reported glucose value.

The next generation algorithm, first implemented in the BGM system reported here, exploits the pulsed nature of the potential-time waveform applied to the test-strip to further desensitize the system to electroactive interferences. The potentials of the 2-electrode design are switched relative to each other during the 5 s assay, with the raw-signal contribution of any electroactive interfering species present being determined by assessing the relative magnitudes of response between the reagent coated and non-reagent coated electrodes. The outcome of this algorithmic enhancement is evidenced by the FDA clinical accuracy and MARD improvements recorded (98.9% vs 97.6% within ±15% of comparator; 5.17 vs 4.19% vs 5.17% MARD).

Sponsor Bias in BGM Studies

An in-depth comparison of the clinical accuracy obtained in this study against published accuracy for other BGMs is not included. Such an analysis would present multiple datasets from multiple investigators with often conflicting data regarding product performance. In some cases, the assessment method may be responsible for differences observed, such as choice of comparator employed by the manufacturer for product release and that used by a researcher for product assessment. ¹⁹ Furthermore the concept of sponsor-bias cannot be ignored. ²⁰ This phenomenon relates to studies in which the sponsor’s product performs favourably against carefully selected competitor systems, while independently funded studies, undertaken on the same products, head-to-head, can result in different outcomes. This may be evidenced by comparing results from BGMs investigated within the DTS surveillance program vs sponsor-supported evaluations of the same products. Crucially, while it is recognized that independent investigators will undertake a sponsor-funded evaluation with complete integrity, the study sponsor themselves may often reserve the right to choose whether to publish the data (this is not the case for investigator-initiated studies or in situations where the sponsor provides unrestricted funds, giving the researcher freedom to publish their findings). The opportunity to withhold unfavourable data is eliminated if a manufacturer adopts and adheres to a routine PMS program.

Glucose test strips are manufactured on a batch basis, with the goal of producing batches with consistent clinical performance. However, the mean bias between batches varies due to differences in raw materials, strip stability and manufacturing processes and conditions. Batches that more closely agree with the comparator (or reference) performance, that is, have a mean bias closer to 0% will generally exhibit high accuracy. Therefore, a batch with a mean bias of ~0% will more likely out-perform a batch with a bias closer to the outer limits of the release conditions. This generalized pattern may be seen in Freckmann et al.’s analysis, where only a small proportion of strip batches exhibiting centred biases failed to meet the ISO standard (however, it should also be noted that a significantly non-centred batch can also met the standard should the batch exhibit tight precision). ¹⁸ A properly constructed and ongoing PMS program, based on an assumption that production batches are truly sampled randomly and that data from all tested batches is reported, removes the opportunity for a sponsor to pre-select a batch for assessment based on known performance, and subsequently decide whether to publish the findings based on a retrospective evaluation of the results.

Benefit of a BGM Post-market Surveillance (PMS) Program

The demand remains for next generation BGMs that deliver improved clinical accuracy. These systems are needed across diverse global markets where CGM adoption remains low due to cost, availability and support. Many CGM systems still require frequent calibration, in turn requiring reliable, high accuracy BGMs to maximize clinical performance and reduce risk of detrimental insulin dosing decisions.^21,22 CGM-users are also encouraged to compare values to BGMs in situations where CGM values do not match symptoms. ²³ Klonoff ²⁴ has recently commented on a case where a multi-centre drug study required protocol revision due to safety concerns regarding the accuracy of an FDA-cleared BGM and has called for PMS of devices for the safety of subjects and accurate determination of the effectiveness of diabetes drugs and devices. Others also argue that current standards of BGM and indeed CGM performance are inadequate, proposing monitoring the performance of devices in the field. ²⁵

Regulators and leading commentators are actively calling for PMS activities.^24-27 The FDA advocates moving from the current largely passive approach of device users reporting issues to a more active real-world evidence-based approach. ²⁸ Similarly, recent changes to the Medical Device Regulation (MDR) and In Vitro Diagnostic Regulation (IVDR) in Europe have led to a substantial change in legislation requiring more thorough oversight by manufacturers of medical devices once in market. ²⁹

A properly constructed BGM (and future CGM) PMS programme will ensure that production batch selection is random, frequent, on-going and made available for public scrutiny, eliminating the opportunity to supress undesirable performance data. In this study, the manufacturer/sponsor followed a defined standard operating procedure to ensure appropriate random selection of batches, sampling frequency and study execution and was undertaken at 3 independent clinics, per the recommendations of the diabetes technology society (DTS).

Conclusions

While regulators enforce standards that ensure BGMs meet fundamental requirements, there is no absolute requirement for manufacturers to demonstrate that their products consistently maintain those same standards of performance. Traditionally, independent researchers, or interested industry bodies such as the Diabetes Technology Society, do test BGMs against their claims and publish the findings. However, such activities are costly, often with logistical issues in obtaining representative quantities of test materials and are generally limited to a point in time evaluation with limited numbers of test-strip batches. We contend that the responsibility to ensure that BGMs continue to meet their claims lies not with researchers or societies, but with manufacturers. Reported here is just such a longitudinal study based on a novel high-accuracy BGM system deploying an improved algorithm design, demonstrating both consistent batch performance against recent FDA guidance for BGMs and a clear improvement in clinical accuracy based on an improved algorithm design.