Do the New FDA Guidance Documents Help Improving Performance of Blood Glucose Monitoring Systems Compared With ISO 15197?

Abstract

Recently, the Food and Drug Administration (FDA) published guidance documents for point-of-care testing (POCT) and over-the-counter (OTC) blood glucose monitoring systems (BGMS). These are expected to improve weaknesses of the already established ISO 15197:2013 standard accuracy evaluation for OTC BGMS; however, the proposed criteria and procedures rather raise new questions. As an example, the OTC guidance stipulates tighter accuracy criteria in the low glycemic range than the POCT guidance for no obvious reason. Furthermore, a clear definition of requirements for the reference method is missing under consideration of the strict accuracy criteria. External surveillance of BGMS after introduction to the market is an issue that is addressed neither in the FDA documents, nor in ISO 15197, but in the Clinical and Laboratory Standards Institute (CLSI) guideline POCT12-A3.

In 2016, the Food and Drug Administration (FDA) published two guidance documents for blood glucose monitoring systems (BGMS).^1,2 Drafts of these documents had already been published in 2014. Requirements of the FDA guidance for BGMS for point-of-care testing (POCT) were recently discussed by Krouwer in a letter to the editor.³ He criticized that the guidance stipulates limits that are not stringent enough, even though they are stricter than those of other standards for BGMS. Krouwer argues that defining limits which only have to be fulfilled by 95% or 98% of results, therefore leaving results unspecified, is insufficient. However, in technical systems showing random measurement error it is hard to ensure that 100% of measurements are within acceptable limits. It is important to note that the FDA guidance for over-the-counter (OTC) BGMS² and International Organization for Standardization (ISO) 15197:2013⁴ include a criterion specifying 99% of results.

Krouwer’s letter to the editor inspired us to address further shortcomings of the FDA guidance documents and discuss different standards regarding the performance of BGMS.

Standards for Blood Glucose Monitoring Systems

We focused on accuracy requirements in four different standards: the FDA guidance on POCT systems,¹ the FDA guidance on OTC BGMS,² ISO 15197:2013 requirements for BGMS for self-testing,⁴ and the Clinical and Laboratory Standards Institute (CLSI) guideline POCT12-A3.⁵ These standards describe requirements for BGMS with two different intended users: health care professionals and lay users performing self-monitoring of blood glucose (SMBG). These BGMS are not directly comparable; however, in most cases measurement technology is similar.

While the FDA guidance documents are applicable for premarket notifications (510(k)s) in the United States and ISO 15197:2013 is harmonized in Europe as EN ISO 15197:2015⁶ and can be used to demonstrate conformity with parts of EU in-vitro-diagnostics regulations, POCT12-A3 refers to postmarket evaluation of BGMS. Each of the four documents describes procedures for the assessment of BGMS accuracy as well as accuracy criteria, but each of the respective expert committees seemed to have their own, separate opinion regarding the requirements (Table 1).

Table 1.

Relevant Accuracy Requirements of Standards for BGMS.

	ISO 15197:20134		FDA guidance OTC2	FDA guidance POC1	CLSI guideline POCT12-A35
Title	In vitro diagnostic test systems—requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus		Self-monitoring blood glucose test systems for over-the-counter use Guidance for industry and Food and Drug Administration staff	Blood glucose monitoring test systems for prescription point-of-care use Guidance for industry and Food and Drug Administration staff	Point-of-care blood glucose testing in acute and chronic care facilities
Scope	SMBG, premarket	SMBG, premarket	SMBG, premarket	POCT, premarket	POCT, postmarket
Accuracy evaluation	System accuracy	User performance evaluation	Method comparison/user evaluation	Method comparison/user evaluation	Comparing results of the POCT BGMS and a laboratory instrument
Test conditions	Laboratory, controlled temperature	Reflect expected use	Reflect expected use	Reflect expected use	Trained personnel, laboratory conditions
Number of subjects	At least 100	At least 100	At least 350 (at least 10% naïve, for each claimed alternative anatomical site [eg, forearm, palm, etc])	At least 350 (for each claimed sample type (arterial, venous, capillary blood) and at least 50 from every vulnerable sub population), 100-150 neonatal samples if applicable, 50-75 per claimed anticoagulant	At least 100
Measurements performed by	Study personnel familiarized with system and trained in use	Intended users provided device labeling (user manual), no other training or assistance but limited self-practice	Intended users provided device labeling (user manual), no training or assistance	At least 9 intended POC operators	Personnel familiarized with system and trained in use
Sample type	Fresh capillary blood samples	Own fingertip capillary blood	Own fingertip capillary (or alternate anatomical site(s)) blood	Claimed sample type (eg, arterial, venous, capillary, heelstick whole blood, etc)	Fresh capillary blood or other blood specimen following device labeling
Test strips	3 lots, at least 10 vials	1 lot	At least 3 lots, 10 vials	At least 3 lots, 10 vials	1 lot
Test range	Defined distribution in glucose concentration categories, span the measuring interval	Not specified	Should span claimed range	Should span claimed range	Should span claimed range
Low and high glucose ranges	• At least 8 unaltered samples <80 mg/dl, at least 5 unaltered samples >300 mg/dl • Altered samples: spiking or glycolysis	Not required	• At least 10 unaltered samples <80 mg/dl, at least 10 unaltered samples >250 mg/dl • Altered samples (at least 50 each) (spiking, glycolysis) separate study may be performed in laboratory setting	• At least 10 unaltered samples <80 mg/dl, at least 10 unaltered samples >300 mg/dl • Altered samples (at least 50 each) (spiking, glycolysis) separate study in laboratory setting but performed by intended users	• Altered samples: spiking or glycolysis according to procedure recommended by manufacturer
Reference measurement procedure	• “measurement procedure that conforms to the traceability requirements of ISO 17511” • Verify trueness and precision • Two measurements required (glucose concentration stability) • At least duplicate measurements	• “measurement procedure that conforms to the traceability requirements of ISO 17511” • Verify trueness and precision • At least duplicate measurements	• “a laboratory-based glucose measurement method that has been well-validated for precision and accuracy and that is traceable to a higher order” • Two measurements optional	• “a laboratory-based glucose measurement method that has been well-validated for precision and accuracy and that is traceable to a higher order” • Two measurements optional	• “traceable to materials and measurement procedures of a higher order” • Imprecision ≤ 2.9%, bias ≤ 2.2%,total error ≤ 6.9% • Duplicate measurements required
Data analysis	Data exclusion criteria defined, eg: • Error in measurement • BGMS quality control results out of tolerance • Reference method quality control results out of tolerance • Sample out of specifications • Deviation between first and second reference measurement	Data exclusion criteria defined, eg: • Error in measurement observed by user • BGMS quality control results out of tolerance • Reference method quality control results out of tolerance • Sample out of specifications	Inclusion of all results	Inclusion of all results	Data exclusion criteria defined, eg: • Deviation in test procedure • Deviation from certified material ≤ 4%/4 mg/dl • Deviation within duplicate reference measurements ≤4%/4 mg/dl
Accuracy criteria	• ≥95% within  • ±15 mg/dl <100 mg/dl  • ±15% ≥ 100 mg/dl • ≥99% within CEG Zone A+B	• ≥95% within • ±15 mg/dl <100 mg/dl • ±15% ≥100 mg/dl	• ≥95% within ±15% • ≥99% within ±20%	• ≥95% within  • ±12 mg/dl <75 mg/dl  • ±12% ≥ 75 mg/dl • >98% within  • ±15 mg/dl <75 mg/dl  • ±15% ≥ 75 mg/dl	• ≥95% within  • ±12 mg/dl <100 mg/dl  • ±12.5% ≥ 100 mg/dl • >98% within  • ±15 mg/dl <75 mg/dl  • ±20% ≥75 mg/dl

The two FDA guidance documents for OTC and POCT BGMS have many similarities, especially regarding general requirements. The major difference can be found in the accuracy limits and in additional tests that are required for POCT systems, such as performance evaluation in different intended patient populations and for different sample types. Both FDA guidance documents clearly state that the ISO 15197:2013 criteria are insufficient, and seem to aim at improving the ISO standard’s perceived shortcomings. However, we could not find a rationale in the FDA documents.

Performance Requirements

Comparing the two FDA guidance documents, it is striking that the performance requirements for OTC systems are stricter (95% of results within ±15%, 99% within ±20%) at low glucose concentrations (<75 mg/dl) than those for POCT systems (95% within ±12 mg/dl, 98% within ±15 mg/dl) (Figure 1). As it is not obvious why it should be acceptable that POCT systems can show a larger measurement error in the hypoglycemic range than OTC BGMS, an explanation for this difference would be helpful. The authors of the OTC guidance document seem to be aware that the limits will probably not be fulfilled by all BGMS, especially at low glucose concentrations. Therefore they request manufacturers to raise the lower limit of the indicated measurement range to where the limits can be met, but not above 50 mg/dl.

Figure 1.

Accuracy limits of the four standards at glucose concentrations from 0 to 200 mg/dl.

In the FDA POCT guidance, performance requirements show a step at 75 mg/dl: For a comparison method result of 74 mg/dl, 95% (98%) of measurement results shall be within ±12 (15) mg/dl. For a comparison method result of 76 mg/dl, however, 95% (98%) of readings shall be within ±12% (15%) = ±9.12 (11.4) mg/dl. This means that at 76 mg/dl, the allowed tolerance is considerably lower than at 74 mg/dl for no apparent reason, at least no justification is provided.

Only ISO 15197:2013 directly takes into account the clinical impact of measurement errors. A second accuracy criterion relates to the consensus error grid (CEG) that assigns pairs of BGMS results and corresponding reference values to a clinical risk zone. According to ISO 15197:2013, at least 99% of values shall fall within zones A and B of the CEG. However, the CEG was based on survey data from 1994,⁷ which may have become outdated in the meantime, because of improvements in technology as well as changes in therapy guidelines. In addition, the CEG is not very sensitive, because the outer boundaries of zone B allow BGMS values to be 60% to 110% higher than reference values at glucose concentrations above 50 mg/dl, which could easily lead to hypoglycemia. The use of a more modern error grid like the surveillance error grid⁸ should be considered.

Testing Procedures

Differences between the discussed standards can also be found in the testing procedures. Most obvious is the number of subjects participating in a test. While ISO 15197:2013 and the CLSI guideline require at least 100 subjects from which double measurements shall be performed, both FDA guidance documents require at least 350 subjects. This high number might be explained by the data analysis recommendations, which state that no results, not even outliers or obvious handling errors, may be excluded. Every result that is outside the specified limits has to be justified from a clinical point of view instead. Nevertheless, a higher number of samples could also be achieved by performing multiple measurements on the same subjects, as recommended by ISO and CLSI.

According to the FDA, BGMS measurements should be performed by the intended users, that is, health care professionals for POCT BGMS and diabetes patients for OTC BGMS. The FDA OTC guidance also requires at least 10% of subjects to be naïve to SMBG. For an ISO evaluation, system accuracy measurements are performed by trained personnel; however, an additional, separate lay user evaluation including at least 100 subjects is required. Both FDA guidance documents and the ISO 15197:2013 system accuracy protocol require testing of three individual reagent system lots; for an ISO user evaluation one lot is sufficient.

All standards have in common that BGMS should be tested at glucose concentrations that cover the whole claimed measurement range. ISO even specifies glucose concentration categories and the percentage of samples that should be tested in each category to ensure an adequate distribution of values. To reach values in the very low and high glucose concentrations, samples may be altered by glycolysis or spiking. FDA and ISO limit the number of altered samples, and according to the FDA the analysis of altered samples should be a separate study. According to the CLSI guideline procedures of alteration have to be consistent with the respective manufacturer’s labeling, further verification of suitability of the altered samples is not required. ISO requires demonstrating commutability of native and altered samples, such as according to the CLSI guideline C53-A.

For a complete accuracy evaluation the FDA requires 350 subjects to obtain 350 values plus 2 × 50 for adjusted samples in the low and high glucose range (it is not indicated whether the same subjects may take part in both studies), resulting in at least 450 subjects. ISO requires 100 subjects for a system accuracy evaluation to obtain 600 values, and 100 for a user evaluation (when the user evaluation is performed first and subjects are thus still naïve to the BGMS, the same subjects can participate in user evaluation and accuracy evaluation). More concretely this means that in an FDA evaluation, more than 4 times more subjects have to by recruited, but fewer results are gained. A rationale for this count and a justification from an ethical point of view is not obvious for us.

Measurement of Reference Values

With increasingly strict performance requirements, the role of the method used to obtain reference values becomes more and more important. All four BGMS standards demand a measurement procedure that is traceable, for example, according to ISO 17511⁹ as referenced in ISO 15197:2013. However, concrete requirements like allowed deviations, are not made by FDA nor ISO. The ISO standard and the CLSI guideline also describe procedures to verify trueness and precision of the reference method, for example, by using certified reference material. The CLSI guideline even requires the reference method to fulfil specific performance criteria—compared to this the FDA requirements provide a lot more leeway for the quality of the reference measurement procedure stating that it has to be “well-validated for precision and accuracy.”^1,2 According to the FDA documents, duplicate measurements with the reference method are only optional, whereas according to ISO 15197:2013 at least duplicate reference measurements are mandatory, and reference measurements before and after the BGMS measurements are required in system accuracy evaluations to ensure glucose concentration stability.

Systematic measurement differences (biases) between reference measurement procedures have an increasing impact on the glucose monitoring system’s compliance with the requirements as these become stricter. This does not only apply to differences between different methods, for example, the bias between glucose oxidase and hexokinase methods is reportedly up to 8%,¹⁰ but also to differences among different devices of the same model. For two different YSI 2300 STAT Plus analyzers (one each at two different study sites), for example, Bailey and colleagues¹¹ observed a bias in “National Institute of Standards and Technology (NIST) traceable serum control” measurements of 3.36% to 6.91% in one YSI and −0.64% to −2.48% in the other. Even in the best case, this amounts to a bias difference of 4% that is completely independent from the glucose monitoring system under investigation. Thus, when accuracy of a cleared BGMS is evaluated, like, for example, within the Surveillance Program of the Diabetes Technology Society,¹² there are at least two sources of bias that may impact the performance results: the reference measurement procedure used by the manufacturer to calibrate the system and the reference measurement procedure used for comparison measurements in the evaluation. According to the International Vocabulary of Metrology, a reference measurement procedure is a measurement procedure that is “accepted as providing measurement results fit for their intended use in assessing measurement trueness of measured quantity values.”¹³ The fact that a method is accepted does not necessarily imply adequate quality. Therefore, harmonization of reference methods and definition of specific requirements should be established in calibration of BGMS and performance analyses, in particular when different BGMS are compared.

Requirements for laboratory methods irrespective of whether they are used to measure reference values are rather lenient with regard to quality controls. In Germany, for example, limits for the internal quality assessment are ±11% of the used material’s target value. For the external quality assessment ±15% deviation from the target values measured with a reference method (ID-GC/MS) are allowed.¹⁴ In the United States, deviations of ±6 mg/dl or ±10% from target values are allowed in external quality assessments, but the results are compared to consensus values instead of reference values, thus circumventing the idea behind metrological traceability.¹⁵ In consideration of the tight accuracy criteria for BGMS, the standards should pay much more attention to the definition of clear criteria for the reference method that should be stricter than those proposed by national directives for routine laboratory systems.

Implementation of Standards

The FDA guidance documents were published by the end of 2016, and we do not know of any experience reports about how the guidance documents are implemented by manufacturers. Both guidance documents describe a simplified procedure when equivalence to an already marketed BGMS is shown. However, the proposed procedures indicate complex and expensive studies for manufacturers, but also an extensive analysis by the FDA, which has to review, for example, every justification for measurement results outside of the given accuracy limits.

In the context of accuracy, we would like to dedicate one remark to systems for continuous glucose monitoring (CGM). While there are discussions about whether 98% or 100% of BGMS results should be within given limits and limits are more and more tightened, standards for CGM systems are lacking acceptance criteria and requirements as clear as those of the FDA guidance documents or the ISO standard. Only the relatively unspecified CLSI guideline POCT05A¹⁶ is currently available. As the dissemination of CGM systems is increasing and in light of the FDA recently approving the nonadjunctive use of a CGM system,¹⁷ establishing such standards is long overdue.

Only the CLSI guideline considers the evaluation of BGMS that have already been introduced to the market. The approval or conformity assessment evaluation is a one-time procedure, but performance of systems that once gained FDA approval or a CE mark is never systematically verified again. Accordingly, postmarketing studies repeatedly revealed systems that did not fulfill the respective standards.^18-20 At least, the FDA guidance documents require establishing test strip lot release criteria. It should be taken into consideration whether regular postmarketing surveillance might be at least as important as strict evaluations in the course of the approval.

Summary

The FDA guidance documents were expected to address shortcomings of the already established ISO 15197 standard. Efforts were made to improve on these shortcomings, but new issues came up as well: The stricter accuracy limits seem inconsistent, and the recommended procedures require extensive studies involving a lot of subjects. In addition, the importance of the reference measurement quality was not sufficiently addressed.