Evaluation of the Analytical Performance of the Coulometry-Based Optium Omega Blood Glucose Meter: What Do Such Evaluations Show?

Abstract

The article entitled “Evaluation of the Analytical Performance of the Coulometry-Based Optium Omega Blood Glucose Meter”, by Solnica and colleagues in this issue of Journal of Diabetes Science and Technology demonstrates that the Optium Omega blood glucose meter meets the analytical requirements for glucose meter performance and it is stated that the results are clinically useful. The authors studied precision, bias, and reagent lot-to-lot error sources. The ultimate goal of an evaluation is to estimate the distribution of errors (from any source) that will be experienced in routine use. The data collection and analysis methods to achieve this are discussed, as are the standards used to compare the results. Claiming clinical usefulness is almost a boilerplate statement in evaluations but meeting standards does not prove clinical usefulness.

In this issue, Solnica and colleagues¹ evaluate the Optium Omega blood glucose meter. These authors report a fairly thorough evaluation of the analytical properties of a glucose meter and compare the results to various glucose meter standards.

Such studies can be viewed in two ways: the suitability of the study methods (data collection and analysis) and the standards to which the results are compared. Implied in any of these studies is the goal to evaluate the clinical usefulness of the new glucose meter. Thus, the following analysis discusses whether studies such as the authors’ study adequately address the clinical usefulness question.

A typical glucose evaluation determines how close a series of patient glucose results are to a reference glucose method (which is as close to truth as is practical). This amounts to characterizing the distribution of glucose differences (new method minus reference) sampled during the study. The study is often accomplished by sequentially sampling a series of patients and was done by Solnica and colleagues.¹ For the evaluation to be meaningful, the results obtained have to be a representative sample from the population about which inferences are desired.² The problem is that, in the real world, glucose errors are caused not just by analytical errors but also by pre- and post-analytical errors, and the latter are often excluded from evaluation protocols. In self-monitoring blood glucose, the patient is the clinician and makes the treatment decision as to what treatment if any is indicated, largely based on the glucose meter result. If that result is in error due to contamination at the lancing site and an incorrect treatment decision is made, clinical usefulness has decreased even if there is no analytical error. Thus, focus on the analytical properties of a glucose meter with concomitant exclusion of pre- and post- analytical error sources means that the clinical usefulness question is only partially answered.

The reason for focusing on the analytical properties is simple—one always wants to know, all things being equal (meaning shelving the pre- and post-analytical question), whether some new glucose meter’s performance is acceptable. The problem is that often the pre- and post-analytical questions are never addressed. Some pre- and post-analytical errors are independent of the glucose meter (washing and drying hands) while other such errors (short sample or reading the result from the display) have interactions with the specific glucose meter.

Regarding analysis methods, one needs to know the distribution of total error. For observational studies, this can be achieved using error grids, which are well known for glucose. One must still recognize the limitations of a typical study sample size, and the authors’ was larger than most. In the United States, there are about 7.2 million patients with diabetes who require insulin.³ If a patient tests three times a day, this gives 7.9 billion glucose results every year. Thus, a typical glucose evaluation (N = 100–300) is a tiny sample and there is a very low likelihood of observing rare but deadly events of any type including analytical. For example, 13 deaths were reported by the Food and Drug Administration (FDA)⁴ over a 12-year period due to maltose interference (a rate of 0.00000001374%). Simply testing a series of patient samples is useful but it is unlikely to identify rare, harmful events.

If one has established the magnitude of individual error sources, as was done by the authors, then one must use models that combine the error sources to arrive at total error estimates (not done by the authors). There are different modeling strategies. The Guide to the Expression of Uncertainty in Measurement method⁵ combines individual error sources to arrive at an overall uncertainty of a result. Risk management methods such as quantitative fault trees⁶ estimate the probability of a top-level event (such as severe harm or death) by combining probabilities of individual error sources that cause the top-level event. Modeling is difficult and rarely occurs in glucose evaluations although it is suited to address the likelihood of rare events.

Given the results, one compares them to a standard. Although several standards have been proposed over the years, the one most currently cited and used by the FDA is the International Organization for Standardization (ISO) glucose meter standard 15197. A previously mentioned problem with this standard⁷ is that one limit is used to distinguish between no harm and harm. This might work for infectious disease or drugs of abuse assays but for glucose, the larger the error, the more the potential harm. With ISO 15197, one is required to meet limits for 95% of the results. With 7.9 billion glucose meter results per year in the United States, this could mean that 360 million glucose meter results per year could cause potential harm for an assay acceptable by ISO 15197 criteria. And since the ISO 15197 criteria are based only on analytical results, errors due to pre- and post-analytical errors would add to the 360 million.

Because glucose meters must be inexpensive to be affordable, their performance will not be as good as laboratory analyzers. Because of economic constraints, the basic concept is sound, to state a percentage of results that must be within certain limits, but a more useful standard would use an error grid (which has multiple limits) and specify the percentages allowed within limits that define each error grid zone. Additionally, a standard should specify a protocol to ensure that all error sources (such as pre- and post-analytical error) are not excluded. There are two parameters to set for each error zone: the magnitudes of the error limits and the percentages allowed. The numerical values selected for the parameters are a controversial topic with patients, physicians, regulators, and manufacturers, all stake-holders and values selected also influenced by the health care economic policy of the region.

Among the potential improvements to glucose standards and evaluations, perhaps the most fruitful will be to make evaluations more representative of actual glucose meter use.

Glossary

Abbreviations

(FDA)

Food and Drug Administration

(GUM)

Guide to the Expression of Uncertainty in Measurement

(ISO)

International Organization for Standardization