Assessment of the Performance of A1CNow⁺ and Development of an Error Grid Analysis Graph for Comparative Hemoglobin A1c Measurements

Abstract

Background: This study investigated the performance of the A1CNow^+® test (Bayer Diabetes Care, Sunnyvale, CA) in a large population of Chinese patients with diabetes.

Subjects and Methods: Hemoglobin A1c (HbA1c) levels in 1,618 Chinese patients with diabetes 10–94 years of age were measured with both the A1CNow⁺ test, from a fingerstick blood sample, and the high-performance liquid chromatography (HPLC) test, using a venous blood sample, within 24 h. The reportable ranges of the HbA1c values were 4.0–13.0% (A1CNow⁺) and 4.1–16.8% (HPLC). An error grid analysis (EGA) method was developed to quantify the accuracy of the A1CNow⁺ results against the HPLC reference results.

Results: The A1CNow⁺ results were highly correlated with the HPLC reference results (r=0.945, P<0.01). Passing–Bablok regression analysis showed a good linear agreement between the two variables, and the linear regression equation fitted as y=−0.10+1.00x (P=0.21). The Bland–Altman difference plot presented that the mean bias of the A1CNow⁺ results minus the HPLC reference results was −0.09% (P<0.001); the 95% confidence intervals for the limits of agreement were −1.28% to 1.09%, with 96.5% of the data points lying within this zone. The results of the EGA showed that 80.2% of the A1CNow⁺ results were accurate, 17.7% were acceptable, 1.9% may lead to inappropriate treatment, and 0.3% may lead to severe clinical consequence.

Conclusions: The A1CNow⁺ test values demonstrated a slight negative bias from the HPLC values. The majority of A1CNow⁺ test values were accurate when compared with results from the reference method.

Background

Hemoglobin A1c (HbA1c) is an index for monitoring average plasma glucose levels over periods of 2–3 months and has a good predictive value for the microvascular complications of diabetes.¹ Measuring HbA1c levels is convenient, compared with the measurement of plasma glucose levels, as fasting or timed samples are not required, and there is a lower within-subject variability associated with its measurements.² With the international efforts in standardizing HbA1c assays, several HbA1c levels have been recommended as goals of glycemic control for diabetes care,^3,4 and an HbA1c of 6.5% has been recommended as the diagnosis criterion of diabetes.^5,6

The current recommended assay for HbA1c is the high-performance liquid chromatography (HPLC) test. However, HPLC requires the collection of venous blood, is performed using a large apparatus, and requires strict laboratory accreditation, which limits its application, especially in remote rural areas. The emergence of portable HbA1c detection instruments may allow these disadvantages to be overcome and enable HbA1c measurements to be more widely used.⁷

The portable HbA1c detection instrument A1CNow^+® (Bayer Diabetes Care, Sunnyvale, CA) has been certified by both the National Glycohemoglobin Standardization Program (NGSP) and the International Federation of Clinical Chemistry and Laboratory Medicine.⁸ The A1CNow⁺ test is a 5-min assay and is easily used by patients and healthcare professionals. The accuracy of A1CNow⁺ test results is, therefore, of clinical importance, and the development of methods to evaluate the accuracy of the test will be useful.

In 1987, Clarke et al.⁹ noted that most studies had only evaluated the statistical accuracy of the self-monitoring of blood glucose, but no study had assessed its clinical usefulness. So they developed an error grid analysis (EGA), which aims to describe the clinical implications of blood glucose measured by the self-monitoring system. EGA has a distinct clinical advantage over other systems. It takes into account the absolute value of the blood glucose measurements and the reference values, the relative difference between these two values, and the clinical significance of this difference. Currently, the Clarke EGA tool has been widely used to visually assess the clinical accuracy of blood glucose level measurements against reference values.

In the context that the importance of measuring HbA1c levels has been realized and point-of-care testing is increasingly widely used, we adapted the Clarke EGA for self-monitoring of blood glucose to assess the clinical accuracy of HbA1c test results, as evaluated by point-of-care testing.

This article investigated the performance of A1CNow⁺ and introduces the first EGA graph of A1CNow⁺ HbA1c test results paired with HbA1c reference results, obtained by HPLC, from a large Chinese population with diabetes.

Subjects and Methods

Subjects

In total, 1,657 Chinese patients with diabetes were recruited from the outpatient and inpatient departments of the Endocrinology and Metabolism of Shanghai Sixth People's Hospital (Shanghai, China) between September and December 2012. HbA1c levels were measured in all patients by both the A1CNow⁺ test and the HPLC method; for each patient, the two tests were performed within 24 h. After the exclusion of 39 subjects with A1CNow⁺ values over 13% (i.e., outside the reportable range of A1CNow⁺ measurements [4–13%]), 1,618 patients with diabetes (1,274 outpatients and 344 inpatients) were included in the study.

This study was approved by the institutional review board of Sixth People's Hospital Affiliated to Shanghai Jiao Tong University, in accordance with the principles of the Helsinki Declaration II. Written informed consent was obtained from each subject.

Methods

HbA1c levels were tested using the A1CNow⁺ device from fingerstick blood samples collected in a capillary tube by trained nurses, according to the manufacturer's procedure guide. The test measures HbA1c levels using both the immunoassay and chemistry methods.¹⁰

Venous blood samples were drawn into EDTA anticoagulant-containing tubes and stored at 4°C. HbA1c levels were determined using HPLC (Variant™ II Turbo; Bio-Rad, Hercules, CA) in the central laboratory of the Shanghai Sixth People's Hospital (an NGSP level I-certified laboratory),¹¹ and these measurements are referred to below as the HPLC reference results. The intra- and inter-assay variances were <0.4% and <0.6%, respectively. The reportable ranges of HbA1c values obtained using the A1CNow⁺ test and HPLC are 4.0–13.0% and 4.1–16.8%, respectively.

All laboratory staff performing the A1CNow⁺ and HPLC assays were blinded to the clinical characteristics of the subjects and the HbA1c test values of the other assay.

Kolmogorov–Smirnov test for normal distribution of HbA1c results

The A1CNow⁺ test results were compared with the HPLC reference results using data from 1,618 paired samples. Deviation of these measurements from the normal distribution was tested using the Kolmogorov–Smirnov test. These continuous data were described as medians (with interquartile ranges [IQRs]).

Correlation of A1CNow⁺ results with the HPLC reference values

The accuracy of the A1CNow⁺ results versus the HPLC reference values was tested. The degree of correlation between the two measurements was evaluated by Spearman's ρ rank correlation coefficient. The agreement of the A1CNow⁺ results against the HPLC reference results was also evaluated with Passing–Bablok regression, a linear regression analysis that is carried out independently of the distribution of the samples and the measurement errors. After calculating the linear regression equation, we tested the slope B and the intercept A to determine the probability that any difference between B and 1 and between A and 0 arose by chance alone. The 95% confidence intervals (CIs) are given.

Analysis of bias using a Bland–Altman difference plot

To analyze bias in these test results, a Bland–Altman difference plot was used to depict the differences between the paired A1CNow⁺ results and HPLC reference results: the A1CNow⁺ results minus the HPLC reference results were plotted on the y-axis (vertical coordinate) against the HPLC references plotted on the x-axis (horizontal coordinate). The 95% CIs for the mean differences (the sample mean difference/average of the difference±1.96 SE) reflect the 95% probability range in which a mean difference population parameter lies.

The 95% CIs for the difference ranges (the sample mean difference±1.96 SD) (i.e., the limits of agreement) illustrate the magnitude by which an individual A1CNow⁺ result deviates from the corresponding HPLC reference result (i.e., the systematic difference). If the line of equality (x=0) is not in the interval, there is a significant systematic difference between the two HbA1c testing methods.

When the average of the differences between the two paired measurements does not equal zero, Bias (Bias=A1CNow⁺ results – HPLC reference values) exists, and the absolute relative error (ARE) must be calculated to describe the deviation, as follows:

Error analysis graph

The assumption of EGA construction

The assumptions of EGA applied to HbA1c test results are based on clinical significance as follows: (1) the tight, moderate, and poor glycemic control ranges are HbA1c <6.5%, 6.5–9.0%, and ≥9.0%, respectively; (2) the acceptable limit of ARE is ±7% of the reference values; (3) inappropriate suggestions are offered to patients to manage HbA1c measurements when tight or poor glycemic control is misidentified as being in the moderate glycemic control range, or vice versa; and (4) major errors can upset tight glycemic control and poor glycemic control regimens and pose the risk of mistaken advice being applied to patient management.

The EGA characteristics

An EGA graph for the HbA1c measurements was developed to quantify the clinical accuracy of the A1CNow⁺ results against the HPLC reference results. EGA defines the HPLC references are on the x-axis, and the estimates measured by point-of-care testing are shown on the y-axis. The diagonal represents perfect agreement between these two paired measurements, with data points above and below the diagonal representing overestimates and underestimates, respectively. It is constructed according to three crucial sets of data. They are the current acceptable limit of HPLC HbA1c measurements, which is±7% ARE of the reference measurements (NGSP manufacturer certification criteria),¹² and the two HbA1c cutoffs, which are levels of 6.5% (stringent glycemic control)³ and 9.0% (poor glycemic control).⁴ In the EGA graph, the plot was divided into the five different zones as follows: Zone A represents the cases in which the A1CNow⁺ results deviated from the HPLC reference results by ≤7% (ARE) or both measurements of HbA1c were <6.5%; Zone B (Zone B1 and Zone B2) represents the cases in which, although the A1CNow⁺ results deviated from the HPLC reference results by >7%, the misidentification remained acceptable; Zone C includes the cases in which the A1CNow⁺ results were ≥9.0% when the true HPLC reference results were ≥6.5% (Zone C1) or cases in which the A1CNow⁺ results were <6.5% when the true HPLC reference results ranged from 8.0% to 9.0% (Zone C2); Zone D consists of the cases where the A1CNow⁺ results ranged from 6.5% to 9.0% when the HPLC reference results were <6.5% (Zone D1) or ≥13.0% (Zone D2); and Zone E represents the cases in which the A1CNow⁺ results were <6.5% but the HPLC reference results were ≥9.0% (Zone E1), or vice versa (Zone E2). Then the glycemic data points were assigned into the five zones based on the ARE of the A1CNow⁺ results against the HPLC references.

In addition, the A1CNow⁺ test values and the HPLC measurements were each classified into two categorical variables of HbA1c levels ≥6.5% or <6.5% and HbA1c levels ≥7.0% or <7.0%, respectively. The κ value, an inter-rater agreement statistic, was calculated to quantify the agreement between the two classifications, using both test methods, on ordinal scales. The κ value is 1 when there is perfect agreement between the two classifications identified by the two methods, and the agreement between the two methods was considered ideal when the κ value was >0.75.

MedCalc (Ostend, Belgium) statistical software (version 12.7.1) was used to perform statistical analyses. The significance level used was a P value of <0.05. MATLAB software (version R2007a; The MathWorks Inc., Natick, MA) was used to perform curve fitting and Loess smoothing and to draw the EGA graphs.

Results

Subjects

In total, 858 men (mean age, 55.8±13.7 years) and 799 women (58.4±13.2 years) were included into this study. The age range of all participants was 10–94 years.

From this population, we obtained 1,618 paired data samples of A1CNow⁺ test results and HPLC references to analyze. The smoothed frequency distribution of the HbA1c values as measured by the A1CNow⁺ and HPLC tests is shown in Figure 1. The HbA1c data obtained did not fit a normal distribution using the Kolmogorov–Smirnov test. The medians (IQRs) of the A1CNow⁺ results and the HPLC reference results were 6.9% (6.1–8.5%) and 7.0% (6.2–8.6%), respectively.

FIG. 1.

Smoothed frequency distribution of the hemoglobin A1c (HbA1c) values measured by the A1CNow⁺ and high-performance liquid chromatography (HPLC) tests. In total, 1,618 paired data samples of A1CNow⁺ test results and HPLC reference values were analyzed. The population percentage on the y-axis is plotted against HbA1c values on the x-axis. Curve fitting and loess smoothing were performed using MATLAB version2007a software. Data given represent the median values. The Kolmogorov–Smirnov test was used for normal distribution analysis. IQR, interquartile range. Color images available online at www.liebertonline.com/dia

Analysis of the agreement between HbA1c results obtained from A1CNow⁺ and HPLC tests

The A1CNow⁺ results were highly correlated with the HPLC reference results (Spearman's coefficient r=0.945, P<0.01). Furthermore, Passing–Bablok regression analysis showed a good agreement between the A1CNow⁺ test results and the HPLC reference results (Fig. 2). The linear regression equation was fitted as y=−0.10+1.00x (P=0.21). The intercept (with 95% CIs) and the slope (95% CIs) were calculated as −0.10 (−0.10 to 0.10) and 1.00 (0.98 to 1.00), respectively.

FIG. 2.

Scatter diagram and linear regression analysis of A1CNow⁺ results (x-axis) versus high-performance liquid chromatography (HPLC) reference results (y-axis) calculated using Passing–Bablok regression analysis. The regression line (solid line) and the identity line (x=y, dashed line) are displayed.

Analysis of the bias of A1CNow⁺ results against HPLC references

The Bland–Altman difference plot (Fig. 3) revealed that the mean difference of the A1CNow⁺ results minus the HPLC reference results (95% CIs) was −0.09% (−0.12% to −0.06%; P<0.001). This result showed that the mean bias of the deviation of the A1CNow⁺ results from the HPLC results was −0.09%. The 95% CIs for the range of the differences (i.e., the upper and lower limits of agreement) were −1.27% to 1.09%, which means that an individual A1CNow⁺ result may deviate by as much as −1.27% to 1.09% HbA1c from the HPLC reference result. On this graph, 96.5% of the data points lay within the limits of agreement. The outliers of the agreement limits only accounted for 3.5% of the data points. In addition, there was no difference between the paired values across the range of HPLC references.

FIG. 3.

A Bland–Altman difference plot to show the differences between the paired A1CNow⁺ and high-performance liquid chromatography (HPLC) reference results (A1CNow⁺ result minus HPLC reference result, y-axis) against the HPLC reference result (x-axis). Horizontal lines have been drawn at the mean difference (solid line) and at the 95% confidence intervals (CIs) of the limits of agreement (for both upper and lower limits of agreement, dashed lines).

Bias and ARE in A1CNow⁺ tests

For the A1CNow⁺ results versus the HPLC reference results, the medians (IQRs) of the absolute value of error and the ARE were 0.3% (0.1–0.5%) and 4.4% (1.9–6.9%), respectively. The proportions of absolute value of error ≤0.5% and ≤0.75% were 76.6% and 88.1%, respectively. In this study, 75.2% of the ARE were ≤7%.

EGA plot

An EGA plot was drawn to evaluate the accuracy of the A1CNow⁺ results against the HPLC reference results (Fig. 4). The results of the EGA showed that 80.2% of the data points fell within Zone A, 17.7% fell within the two B Zones, 0.1% was within the C Zones, 1.9% were within the D Zones, and 0.3% were within the E Zones.

FIG. 4.

The error grid analysis graph showing the A1CNow⁺ results (x-axis) plotted against the high-performance liquid chromatography (HPLC) reference results (%) (y-axis). The error grid analysis scatter plot is classified into five zones (A–E) and depicts the clinical accuracy of the A1CNow⁺ test results. The data points in Zone A represent accurate hemoglobin A1c test results. The data points in Zone B reflect that the biases deviate beyond 7% from the reference values but remain acceptable. Zone C and Zone D data points represent results for which tight or poor glycemic control is misidentified as moderate glycemic control, or vice versa. Data points in Zone E represent results that confuse tight glycemic control and poor glycemic control and that may lead to fully wrong clinical judgment.

Additionally, the κ values (95% CIs) were 0.76 (0.73–0.80) and 0.84 (0.82–0.87), when HbA1c cutoff levels of 6.5% and 7.0%, respectively were adopted.

Discussion

The portable HbA1c detection instrument A1CNow⁺ has an improved performance and has now been certified by both the NGSP and the International Federation of Clinical Chemistry and Laboratory Medicine. In this article, 1,618 paired data samples of A1CNow⁺ test values (range, 4.3–12.9%) from a large sample of Chinese patients with diabetes were compared against reference results obtained using HPLC (range, 4.2–14.6%), and the differences between these results were analyzed to investigate the usefulness of the A1CNow⁺ test. We also introduce the use of an EGA graph to assess the accuracy of the HbA1c measurements.

Analysis of the accuracy of A1CNow⁺ test results

There was a high positive correlation of the A1CNow⁺ measurements with the HPLC results.

Although the Spearman correlation coefficient (r=0.945) in this study was slightly lower than the values of 0.985 and 0.989 reported for the A1CNow⁺ system in previous studies,^13,14 this value remained higher than 0.72 that was reported for the A1CNow⁺ device prior to its NGSP certification.¹⁵ Passing–Bablok regression analysis indicated that the A1CNow⁺ test values had a high agreement with the HPLC results, and the differences between the paired values did not change across the range of HPLC references. In addition, when the results from the two measurement methods were classified by the HbA1c cutoff levels of 6.5% and 7.0% and treated as categorical variables, ideal identification agreements between the results were found, as demonstrated by κ values of 0.764 and 0.844.

Bias analysis of the A1CNow⁺ test results

A 0.5% change in HbA1c level is of clinical significance. The current acceptable limit for grading HbA1c levels, as required by the NGSP, is to achieve 37 of 40 (92.5%) results within±7% ARE.¹²

To investigate bias in the A1CNow⁺ results, a Bland–Altman difference plot (Fig. 3) was drawn, and it showed that the negative bias of the A1CNow⁺ results compared with those obtained by the HPLC method was −0.09% (95% CI, −0.12% to −0.06%). The limits of agreement indicated that an individual A1CNow⁺ result may deviate by as much as −1.27% to 1.09% HbA1c from the HPLC reference result, which is outside the current acceptable limits of±0.75% HbA1c.¹² However, a significant systematic difference between the two methods was not found. In our study, 96.5% of the data points fell within the limits of agreement of −1.27% to 1.09% HbA1c. Furthermore, 76.6% of the biases of the A1CNow⁺ values were within±0.5% HbA1c, and 88.1% of the biases lay within±0.75% HbA1c. These results were better than those reported in a study of children with type 1 diabetes, in which 68% of the results were within 0.5% HbA1c¹⁶ or the GOAL A1C study with 68% of the biases within±0.75% HbA1c.¹⁵

EGA method

The Clarke EGA⁹ was developed in 1987 and has been widely used to assess the clinical accuracy of blood glucose level measurements against reference values. In this article, we adapted the Clarke EGA to assess the accuracy of HbA1c test results, according to three crucial numbers. These values are the current acceptable limit of HPLC HbA1c measurements, which is ±7% (ARE) of the reference measurements (NGSP manufacturer certification criteria), and, in addition, two HbA1c levels have been recommended for diabetes management: an HbA1c of <6.5% is recommended for stringent glycemic control, an HbA1c of ≥9.0% denotes poor glycemic control, and an HbA1c range from 6.5% to 9.0% reflects moderate glycemic control.^3–6

Here the EGA scatter plot is classified into five zones (A–E) and depicts the clinical accuracy of the A1CNow⁺ test results. The data points in Zone A (80.2% of the total number of data points) represent accurate HbA1c test results. The data points in Zone B (17.7% of the total number) reflect that the biases deviate beyond 7% from the reference values but remain acceptable. Zone C and Zone D data points (1.9%) represent results that tight or poor glycemic control is misidentified as moderate glycemic control, or vice versa, which may lead to inappropriate treatment. Data points in Zone E (0.3%) represent results that confuse tight glycemic control and poor glycemic control and that may lead to fully wrong clinical judgment and severe clinical consequence. EGA showed that 97.8% of A1CNow⁺ results were accurate or acceptable measurements, and less than 3% of the A1CNow⁺ results were misleading.

In this study, only 75.2% of test results fell within the ±7% ARE required by the NGSP performance criteria. The reasons for the larger bias displayed by our results could be that, first, the blood samples, although collected from each patient within 24 h, were obtained from different sources: fingerstick capillary blood sample, or venous blood samples. Second, a large sample of patients with diabetes and many investigators were involved in our study, which may have contributed to the greater variability within our results. Third, our study permitted a wider reportable range for both measurements than that reported by the NGSP.

It should be noted that the EGA was constructed on the assumption of the clinical but not statistical significance, and the assertions of the five zones in EGA are not completely rigorous. It should be pointed out that the explanation of the EGA results in our study is still, to some extent, arbitrary. Further comparative statistical methods and justification criteria need to be developed, and construction of EGA may be improved. It is important to establish the allowable error for established zones and limits for erroneous results; for example, those percentages of the test results that fall within Zone A should approach 95%, and the percentages within Zone E should be <1% because they reflect erroneous results and pose a risk of errors in patient management.

In addition, the three crucial cutoff values were set arbitrarily and may be modified according to the new requirements for glycemic control and error, which may result in a different assessment of clinical accuracy for the same test measurements.

Conclusions

The A1CNow⁺ test values showed good agreement with HPLC reference values but demonstrated a slight negative bias from these values. The majority of A1CNow⁺ test values were accurate when compared with results from the HPLC method. Further improvements in the performance of the A1CNow⁺ test may aid HbA1c monitoring in contributing to diabetes management and increase the importance of this method as an effective tool for the management of glycemic control in patients with diabetes.

Acknowledgments

This work was supported by grants from Bayer Diabetes Care, China, the Drug Innovation Program of National Science and Technology Project (number 2011ZX09307-001-02) and the Public Health Key Disciplines of Shanghai (number 12GWZX0104).

Author Disclosure Statement

No competing financial interests exist.