The New Glucose Standard POCT12-A3 Misses the Mark

Abstract

POCT12-A3 is a Clinical Laboratory Standards Institute standard for hospitals about hospital glucose meter procedures and performance standards. I have reviewed this standard based on the attributes of an ideal performance standard. POCT12-A3 has tighter limits than its predecessor for 95% of results, the limits widen for 98% of results, and there are no limits for 2% of results. It is hard to fathom that 2% of the results are unspecified and could cause life-threatening results, as glucose meters do not perform this poorly. There should be a specification for unreported results since, by definition, point-of-care-testing assays are time sensitive. POCT12-A3 provides useful advice about the glucose testing procedure but provides evaluation guidance only about analytical performance. Moreover, the recommended protocol to assess meter performance is biased and likely to underestimate the observed performance. The guideline would be improved if its specification were based on an error grid and contained evaluation protocols for user errors.

The new glucose meter guideline POCT12-A3 from the Clinical Laboratory Standards Institute¹ covers glucose meters used by health care professionals; it excludes self-monitoring of blood glucose. As a guideline intended for hospitals, it recommends procedures, glucose meter performance standards, and an evaluation protocol. The previous version of this standard used the self-monitoring of blood glucose standard ISO15197 for glucose meter performance criteria.

As a framework for commenting on POCT12-A3, an ideal performance specification should contain three items:

1. The limits of assay result errors considered to be acceptable;

2. A protocol that describes how to collect data to determine if the limits have been met; and

3. A description of how to analyze the data.

Assay errors mean the distribution of differences from all sources between the candidate glucose meter and reference that will be observed by the clinician under routine use. This description of error is equivalent to total error (or total differences). Total error is a conceptual term meaning errors from all possible sources. The possible error sources in glucose meter measurements can be thought of as coming from three sources:

1. User errors unrelated to the meter, such as failing to wash and dry the site when a capillary sample is taken;

2. Instrument errors, such as a bias due to hematocrit interference; or

3. Errors that are an interaction between the user and instrument, such as a short sample (user) and instrument algorithm to detect short samples (instrument).

Manufacturers evaluate error sources 2 and 3; hospitals should evaluate all three.

There are two ways to evaluate the glucose error distribution:

1. One can conduct experiments to evaluate each specific error source. With all error sources studied, a model can be built to provide the glucose error distribution. Manufacturers often use this approach. It is important to sample not just analytical error sources, but user error sources as well, especially for hospitals.²

2. Additionally, one can conduct a method comparison experiment, which randomly samples error sources that are allowed to be included in the experiment.

POCT12-A3 is largely a high-level guideline, providing useful advice about the total process of glucose testing, but in most areas, it does not provide evaluation details about collecting and analyzing data.

The framework for any performance standard is its acceptance limits, and those of POCT12-A3 are

1. 95% of the results must have differences from the laboratory analyzer less than 12 mg/dl below 100 mg/dl and less than 12.5% above 100 mg/dl and

2. The sum of the number of individual results with errors that exceed 15 mg/dl below 75 mg/dl and exceed 20% at glucose concentrations at or above 75 mg/dl should not exceed 2% of all results.

POCT12-A3 mentions its acceptance criteria “as an indication of clinical need in the marketplace.”¹ Yet these limits cannot be based on medical requirements. If glucose meters just met the second requirement, as a worst case, there could be 20,000 severe glucose meter errors for every million results for a POCT12-A3-acceptable glucose meter. Clearly, this would not be tolerated, and meters do not perform this poorly. So why this specification is set at this level is a mystery. Perhaps the requirement exists to allow an outlier or two in a 100 sample evaluation. The previous requirement from ISO15197 allowed up to 5% results to be unspecified, which prompted critique.^3,4 A simple remedy to fix the limits would be to use an error grid for glucose. Error grids as specified in the Clinical and Laboratory Standards Institute’s EP27-A⁵ also contain a requirement to assess the percentage of results when no result is provided. There are two ways that assays can harm patients: (1) too much error or (2) no result for a time-sensitive assay. The whole purpose of point-of-care testing-assays is to provide rapid results. Manufacturers embed algorithms in their glucose meters to prevent reporting bad results, which is a good thing, but at some point, the frequency of unreported results outweighs the benefit of rapid results, so there should be a specification for the percentage of unreported results, and this parameter should be evaluated.

The high-level guidance in POCT12-A3 covers clinical uses of blood glucose meters, user training, sampling techniques, and quality control, but without specifying a protocol or data analysis. Thus, although this useful advice is given about the process of glucose testing, only a component of the process is evaluated—the analytical properties of the glucose meter through a method comparison experiment. Of course, one can ask legitimate, separate questions such as which, among several glucose meter products, to choose for my laboratory.

In a method comparison experiment, the total error estimated reflects all errors sources allowed to occur in the experiment. In POCT12-A3, either several relevant error sources are excluded from occurring or, if they occur and cause errors, the results are deleted from the analysis. Either of these practices causes bias.

For example, this advice¹ is given about the operators to be used for a glucose meter evaluation:

Operators must read and follow the meter system manufacturer’s instructions (ie, reagent storage, setup, sample requirements, calibration, and operation). Operators need to become familiar with the use of the device through a “device-familiarization period.” This includes knowledge and instruction on how to perform the test, as well as knowledge of how to avoid preexamination (preanalytical) (eg, sample collection and handling), examination, (analytical) and postexamination (postanalytical) (eg, glycolysis or evaporation) error.

There is nothing wrong with this advice, but the personnel performing the experiment should be representative of the personnel who will use the meter, and their training (if any) should be the same as they would obtain in routine practice.

For the evaluation of capillary samples, POCT12-A3 suggests a split sample protocol from a single finger stick, whereby one sample is measured by the glucose meter and the other by the central laboratory, but this excludes the possibility from error due to finger sticks that could be ascertained by comparing a finger stick analyzed by the glucose meter to a venous sample analyzed in the central laboratory. Karon and coauthors⁶ provided evidence that this comparison should not show differences.

POCT12-A3 recommends excluding data if either “blood samples contain known interfering substances” or “blood samples are outside the manufacturer’s specifications for a limiting variable such as hematocrit or partial pressure of oxygen.”¹ This would be legitimate only if it were routine practice to assay for each patient all listed endogenous interferences, including hematocrit and partial pressure of oxygen, and to review medications and compare against the package insert for all listed interfering substances, including allowed hematocrit or partial pressure of oxygen ranges.

I have previously commented that user errors should not be excluded from evaluations.^{7, 8} The guideline would be improved if user studies (see Reference 2 for an example) were added. This might also require a model to combine errors from all sources (user and analytical). As mentioned earlier, there is nothing wrong in determining a component of total error, such as the analytical properties of a glucose meter, but one cannot neglect the most important question: What is the distribution of errors (or differences) that will be observed in routine use?

Funding

This work was funded by Krouwer Consulting.

Disclosure

Jan S. Krouwer is an employee of Krouwer Consulting.