Investigation of Two Models to Set and Evaluate Quality Targets for HbA_1c: Biological Variation and Sigma-metrics

Abstract

Background

A major objective of the IFCC Task Force on implementation of HbA_1c standardization is to develop a model to define quality targets for HbA_1c.

Methods

Two generic models, the Biological Variation and Sigma-metrics model, are investigated. Variables in the models were selected for HbA_1c and data of EQA/PT programs were used to evaluate the suitability of the models to set and evaluate quality targets within and between laboratories.

Results

In the biological variation model 48% of individual laboratories and none of the 26 instrument groups met the minimum performance criterion. In the Sigma-metrics model, with a total allowable error (TAE) set at 5 mmol/mol (0.46% NGSP) 77% of the individual laboratories and 12 of 26 instrument groups met the 2 sigma criterion.

Conclusion

The Biological Variation and Sigma-metrics model were demonstrated to be suitable for setting and evaluating quality targets within and between laboratories. The Sigma-metrics model is more flexible as both the TAE and the risk of failure can be adjusted to requirements related to e.g. use for diagnosis/monitoring or requirements of (inter)national authorities. With the aim of reaching international consensus on advice regarding quality targets for HbA_1c, the Task Force suggests the Sigma-metrics model as the model of choice with default values of 5 mmol/mol (0.46%) for TAE, and risk levels of 2 and 4 sigma for routine laboratories and laboratories performing clinical trials, respectively. These goals should serve as a starting point for discussion with international stakeholders in the field of diabetes.

Introduction

A major objective of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) Task Force on Implementation of HbA_1c Standardization (TF- HbA_1c) is to “Develop quality targets for the measurement of HbA_1c, and on the basis of these targets, and in conjunction with professional bodies, advise on the use of HbA_1c for monitoring, diagnosis and screening of diabetes and glucose intolerance” (1). This paper addresses the development of a model to set and evaluate quality targets. A suitable model should be applicable both within and between laboratories. The IFCC is an international organization that advises on quality targets and therefore the model should be generic and give local regulatory bodies the option to fill specific criteria related to e.g. financial and technical resources.

Table 1 summarizes the key elements considered in the development of the models. A quality target consists of a value assigned by an approved reference measurement procedure, with tolerance limits derived from the quality concept of Total Allowable Error (TAE) (2). In this concept Total Error (TE) is derived from bias and imprecision and compared with the TAE to reveal a pass or fail. The suitability of two TAE based analytical-mathematical models is investigated here: the “Biological Variation” model (BV) (3,4) and the “Sigma-metrics” model (SM) (5). To set quality targets both models require alignment with the Stockholm Hierarchy for analytical quality (6) to define TAE. In this hierarchy there are five levels: a) clinical outcomes in specific clinical settings, b) data derived from biological variation, c) published professional recommendations, d) performance goals set by regulatory bodies or organizers of External Quality Assessment (EQA) schemes, and e) current state of the art. For the SM model all levels can be used, whereas only level b applies to the BV model. Both models have a grading system to categorize the risk of not achieving the set TAE. In the SM model the risk is expressed in sigma (σ) units (the lower σ, the higher the risk of not achieving the set TAE) (5). The BV model has three categories, namely the optimum, desirable and minimum level of performance (3,4).

Table 1. Key elements considered when developing the quality targets.

Topic	Specification
1. Anchor	Reference Method
2. A Quality Concept	Total Allowable Error Total Error derived from Bias and Imprecision
3. Analytical-Mathematical Models	Biological Variation (BV) Sigma-metrics (SM)
4. Model components
a. Criteria TAE	Intra- and Inter-individual Variation (BV) Goals derived from the Stockholm Hierarchy (SM)
b. Risk Grading	Optimum-Desirable-Minimum (BV) n-sigma (SM)
c. Evaluation of each model	within laboratories (BV and SM) between laboratories (BV and SM)
5. Factors to be considered when setting criteria	Aim of the test Test Environment State of the Art Economical Affordability Practical Aspects Clinical needs

Several considerations may have an impact on the specifically defined criteria within both generic models: the aim of the test (e.g. diagnosis versus monitoring), test environment (e.g. Central Lab versus POCT), technical limitations (what is the state of the art), economic limitations (what cost is economically affordable), practical considerations (workload and availability of bias and imprecision data), and clinical needs (4).

This study evaluates the suitability of the BV and SM models within and between laboratories by using data from External Quality Assessment/Proficiency Testing (EQA/PT) programs in the US and Europe. In compliance with the Consensus Statement on HbA_1c of the International Diabetes Federation (IDF), European Association for the Study of Diabetes (EASD), American Diabetes Association (ADA) and IFCC, data will be expressed in Système International (SI) units with National Glycohemoglobin Standardization Program (NGSP) units (7) in brackets. It should be noted that the mathematical relationships between the IFCC and NGSP systems are complicated; conversion factors are summarized in Table 2 and details can be found in reference (8).

Table 2. Mathematics for both BV and SM models.

Total Error
1. TE = |B| + 2I
Biological Variationa
2. $\text{TAE}=1.65(\mathrm{f}1\times \text{CVI})+\mathrm{f}2\sqrt{{(CVI)}^2+{(CVG)}^2}$
				CVI = 2.77% (NGSP 1.9%)
				CVG = 8.32% (NGSP 5.7%)
	Classification	f1	f2	TAE at HbA1c = 50 mmol/mol (6.7%)
				In Relative Units (%)		In Absolute Units (Concentration)
				IFCC	NGSP	IFCC mmol/mol	NGSP %
	Optimum	0.25	0.125	2.24	1.53	1.1	0.10
	Desirable	0.50	0.250	4.48	3.07	2.2	0.21
	Minimum	0.75	0.375	6.72	4.60	3.4	0.31
Sigma-metrics
3. $\sigma =\frac{(\text{TAE}-|\mathrm{B}|)}{\mathrm{I}}$
Conversion Factors NGSP from IFCC
4. HbA1c in Concentration Units:				NGSP = 0.0915IFCC + 2.15
5. Absolute Bias in Concentrations Units:				NGSP = 0.0915IFCC
6. Imprecision in CV% at 50 mmol/mol:				NGSP = 0.685IFCC
7. Imprecision in CV% in NGSP units:				IFCC = 1.46 NGSP
Symbols
TE	= Total Error			in concentration units
TAE	= Total Allowable Error			in concentration units
|B|	= Absolute Bias			in concentration units
I	= Imprecision			in concentration units
CVI	= Intra Individual Variation			in CV%
CVG	= Inter Individual Variation			in CV%
σ	= Chance Failure			in sigma units
f1	= Factor to classify the imprecision component of TAE
f2	= Factor to classify the bias component of TAE

^aCVI and CVG taken from the Ricos tables in NGSP units and converted to IFCC with equation 7

Methods

Biological Variation Model

The outcome of the BV model varies with HbA_1c concentration (8) and necessitates the choice of one HbA_1c concentration for evaluation. 50 mmol/mol (6.7%) is selected as most relevant as this concentration is close to and in-between the clinical decision limits for diagnosis of diabetes (48 mmol/mol; 6.5%) and a common target for treatment (53 mmol/mol; 7.0%) (9). TE and TAE are calculated with equations 1 and 2 in Table 2. Intra- and inter-individual biological variation is taken from the table published by Ricos (10). TAE is expressed in relative and absolute units . Fig. 1A is the graphical representation of the limits for minimum, desirable and optimum performance. There are two extreme options to achieve an adequate performance level: a perfect bias with a relatively poor imprecision or a perfect imprecision with a relatively poor bias. As an example, the performance of a laboratory with a bias of 1.0 mmol/mol (0.09%) and an imprecision of 1.5% (1.0%) resulting in a TE of 2.5 mmol/mol (0.23%) is drawn in Fig. 1A. This lab meets the BV criterion for minimum performance but fails the desirable and optimum criteria.

Fig. 1. Quality Target Models.

Imprecision in %CV on the x-axis and bias in mmol/mol (IFCC) and % (NGSP) on the y-axis. Quality Targets are shown in colors for the Biological Variation model (Panel A; optimum, desirable, minimum) and with lines for the Sigma-metrics model (Panel B; 2 – 6 σ). Panel C combines both models.The red heart represents the performance of a laboratory with a bias of 1 mmol/mol (0.9%), an imprecision of 1.5% (1.0%) and a TE of 2.5 mmol/mol (0.23%) and meets the “minimum” performance criterion in the Biological Variation model and the “5σ” criterion in the Sigma-metrics model. Panel D shows the impact of varying the set TAE and acceptable σ value on performance.

Sigma-metrics Model

Sigma-metrics provides a universal benchmark for process performances. The performance of any process can be characterized on a “Sigma scale”. Values range from 2 to 6. A higher (4-6) sigma level implies a very low risk to fail (thus a very reliable process) but also a very narrow target range (thus difficult to achieve). A lower (2-4) level means an easier to achieve performance level but also a higher risk to fail (i.e. more results ouside the set TAE value). The target sigma level set should balance optimum quality and risk to fail (5). In the SM model TAE is set according to one of the levels of the Stockholm Hierarchy Consensus (6) along with the risk of failing this goal. The performance of a laboratory is expressed in a sigma score and is calculated with equation 3 in Table 2. This same equation can be used to calculate allowable bias and imprecision at a defined sigma level. This is graphically represented in Fig.1B. To evaluate the model a TAE of 5 mmol/mol (0.46%) has been chosen. This is based on the difference in HbA_1c result in two consecutive HbA_1c tests that guide clinicians to change therapy (11) and is therefore a clinical decision limit (Stockholm Hierarchy level 1). Five mmol/mol (0.46%) is also approximately the difference between the upper level of persons at low risk for diabetes and the decision limit for diagnosis of diabetes (48 mmol/mol; 6.5%) (9,12). As with the BV model there are extremes of bias and imprecision that allow compliance with the TAE: a high bias with a low imprecision or vice versa. Fig.1B also shows the performance of the same laboratory as in the BV model: the TE of 2.5 mmol/mol (0.17%) results in a score of 5.3 sigma.

Comparison Models

In Fig. 1C both models are combined. It is evident that BV limits are tighter than SM limits. This is not surprising as the arbitrary set TAE is higher in the SM model (5 mmol/mol; 0.46%) than those in the BV model (3.4/2.2/1.1 mmol/mol; 0.31, 0.20, 0.10%). In the SM model there is free choice to define TAE and sigma level. The impact is demonstrated in Fig.1D. When a TAE of 3 mmol/mol (0.27%) is set with an acceptable risk of 4σ, the lab fails. But with a TAE of 5 mmol/mol (0.46%) and an acceptable risk of 2σ, the lab passes. In the BV model both TAE and risk categories are fixed by intra- and inter-individual variation. The values from the Ricos tables may be revised following recent literature (13,14,15) which will of course lead to different (tighter) quality targets.

Results

Evaluation between laboratories

Data from the College of American Pathologists (CAP) Survey 2014 GH2-A were used to evaluate the between laboratory performance using each model. Sample GH2-01 had an HbA_1c concentration of 48 mmol/mol (6.49%) and was measured by 3277 laboratories, about 90% of them in the US. The overall and manufacturer/instrument specific bias and between-laboratory CV are taken from the NGSP website (18) to calculate TE and σ (Supplemental Data Table 1,). In Fig. 2 bias and CV are plotted in the combined BV-SM model. To visualize positive and negative bias the figure can be expanded (Supplemental Data Fig. 1). The overall mean (black star) is well outside the minimum performance criterion (amber colored) of the BV model and also not within the 2-sigma limit of the SM model. The mean performance of users within each of the respective manufacturer/instrument combinations is highly variable. The majority of the HPLC instruments meet the 2 sigma criterion of the SM model, whereas, the majority of immunochemical and POCT instruments do not. Only one method (N) achieves the minimum performance level of the BV model. Fig. 2 also indicates the source of poor performance: high imprecision for methods A and C, high bias for methods V and W, high bias and imprecision for methods S and O. The high bias of HPLC instruments V and W is remarkable in relation to their excellent precision. The model allows historical evaluation of this phenomenon as is shown in the Supplemental Data Fig. 2 and Table 2. CAP data for these instruments, dating back to 2006, are evaluated. Imprecision was relatively constant but in the early years 2006-2008 (Fig. 2, blue circles) bias was low with a resulting performance within the 2σ limits of the SM model.

Fig. 2. Models applied to 26 manufacturer/instrument means in CAP 2014 GH2-A survey.

The mean within manufacturer interlaboratory CV is on the X-axis; the mean manufacturer absolute bias is on the Y-axis. The black star represents the overall mean of all Laboratories. The dots (laboratory instruments) and squares (POCT instruments) represent specific manufacturers with colors for analytical principles: Immunoassays (green), Ion Exchange HPLC (red), Affinity HPLC (yellow), Capillary Electrophoresis (blue) and Dry Chemistry (grey). Abbott Architect c System (A), Abbott Architect I System (B), Axis-Shield Afinion (C), Beckman AU systems (D), Beckman UniCel DxC Synchron (E), Bio-Rad D10 (F), Bio-Rad Variant II (G), Bio-Rad Variant II Turbo (H), Bio-Rad Variant Turbo 2.0 (I), Roche Cobas c311 (J), Roche Cobas c500 series (K), Roche Cobas Integra 400 (L), Roche Cobas Integra 800 (M), Sebia Capillarys 2 Flex Piercing (N), Siemens Advia Chemistry Systems (O), Siemens DCA 2000/2000+ (P), Siemens DCA Vantage (Q), Siemens Dimension ExL (R), Siemens Dimension RxL (S), Siemens Dimension Vista (T), Siemens Dimension Xpand (U), Tosoh G7 Auto HPLC (V), Tosoh G8 Auto HPLC (W), Trinity Biotech HPLC (X), Trinity Biotech Premier Hb9210 (Y), Ortho Clin Diag Vitros 5,1 FS, 4600, 5600 Chem System (Z). For more details see Supplemental Data Table 1,

Evaluation within laboratories

EQA/PT programs can be categorized and only the highest category supplies data on bias and imprecision within laboratories (19). To evaluate the SM and BV models within laboratories, results of such a category 1 EQA/PT program are required. CAP does not evaluate data on imprecision within labs. Therefore, data of the 2013 EQA program shared by EQA organizers in Belgium, the Netherlands, Greece and Finland have been used (20). Fig. 3 shows the performances of the individual laboratories at the HbA_1c concentration of 48 mmol/mol (6.5%). A substantial number of labs do not meet the the 2 sigma criterion of the SM model and many more fail even the minimum performance criterion of the BV model. In Table 3 these data are quantified. In the BV model 2, 18 and 48% of the labs meet the optimum, desirable and minimum criteria, respectively. To demonstrate the impact of chosen TAE and sigma-level, calculations for the SM model are made for TAEs ranging from 1 to 10 mmol/mol (0.09 to 0.9%) and sigmas from 2, 4, and 6. Not surprisingly, at high TAE/low sigma nearly all labs pass and at low TAE/high sigma nearly all laboratories fail. At a reasonable TAE of 5 mmol/mol (0.46%) and 2 sigma, 77% of the laboratories pass.

Fig. 3. Models applied to 342 individual laboratories.

Each dot represents an individual laboratory: HPLC (red), Immunoassay (green), Affinity Chromatography (yellow), Capillary Electrophoresis (blue). For reasons of simplicity only 2, 4 and 6σ limits for TAE = 5 mmol/mol (0.46%) of the Sigma-metrics model are shown. Yellow, grey and amber are the limits of optimum, desirable and minimum performance of the biological variation model. Data from the shared 2013 EQA program in Belgium, the Netherlands, Greece and Finland.

Table 3. Demonstration of the impact of different chosen criteria on the % of 342 laboratories passing for the BV and SM models.

Biological Variation		Sigma-metrics
Criterion	% Labs pass	Criterion Chance Failure	TAE 10 (0.91)	TAE 7 (0.64)	TAE 6 (0.55)	TAE 5 (0.46)	TAE 4 (0.37)	TAE 3 (0.27)	TAE 2 (0.18)	TAE 1 (0.09)
Optimum	2	6 σ	71	46	30	19	10	2	1	0
Desirable	18	4 σ	89	67	57	44	25	12	2	0
Minimum	48	2 σ	100	95	89	77	60	38	16	1

Discussion

Analytical/Mathematical Model

An advantage of the BV model is that once inter- and intra-individual variation are established, quality targets are fixed. But at present these parameters are not irrefutable; as stated before different values have been found by different authors (10, 12, 13, 14). One publication even showed different values with different analytical methods (15). The ultimate importance of reliable data on biological variation and their impact on confidence intervals in general has been described by many authors (16,17). Furthermore it is questionable whether the exact intra-individual variation can be established, because the value is of the same order of magnitude as the analytical CV for even the best methods. It is also unknown whether (narrow) biological variation data of a young, healthy, Caucasian population are applicable to other (elderly, type 1 or type 2 diabetes, well/poor controlled, ethnic) populations. A practical point of consideration is that the state of the art is quite far away from even the minimum criterion of the BV model.

A disadvantage of the SM model is that it can lead to endless discussions, without decisions, on setting an appropriate TAE and the subsequent risk to fail the set TAE (sigma). But this can also be an advantage: criteria are flexible and can be adjusted to the application (monitoring versus diagnosis), test environment (developed versus developing countries), state of the art and quality/cost issues (see considerations Table 1). Another advantage is that the criteria are of the highest level of the Stockholm Hierarchy Consensus (6). This criterion is clinically and not analytically/statistically driven and may therefore be more appealing –and thus acceptable for physicians. In developing the most appropriate quality criteria the combined model (Fig.1C) can be used but in daily practice a double target may be confusing. Balancing pros and cons, the TF- HbA_1c suggests that Sigma-metrics is the model of choice.

Criteria

The goal of this paper is to define a generic model to set and evaluate quality targets. Example criteria have been used to evaluate the suitability of each of the models. The TF- HbA_1c will seek collaboration with professional bodies to advocate the model and discuss/refine the most appropriate TAE and risk criteria for monitoring, diagnosis and screening of diabetes and glucose intolerance. This could be done within the framework of the Consensus Statement of HbA_1c of the IDF, EASD, ADA and IFCC (7). A statement on quality targets can be added, and, like the other statements, can be periodically reviewed, revised and published in diabetes related media.

Exploration of the most suitable criteria is useful as a starting point for discussion. The clinical decision limit for diagnosis (48 mmol/mol or 6.5%) is close to the upper limit of the non-diabetic reference range (42 mmol/mol or 6.0%) (9,12). Thus a small difference in HbA_1c performance will have a large impact on interpretation. For example, an increase of the imprecision from 2.7 to 3.7% (1.8% to 2.5% in NGSP Units) results in a 90% increase in the prevalence of diagnosed diabetes (21). A report of the Danish Health and Medicine Authority states that biases of 0.6, 1.7, and 2.8 mmol/mol (0.06, 0.15, 0.25%) increase the number of persons diagnosed with diabetes by 28, 67 and 224%, respectively (22). When this is extended to the 2014 CAP GH2-A results it can be estimated that, if all HbA_1c assays in US had been performed with instruments of the manufacturer with the highest positive bias (+4.2 mmol/mol; +0.39%), the prevalence of diabetes would have been 600% higher than if all assays had been done with instruments of the manufacturer with the highest negative bias (-2.3 mmol/mol; -0.21%). This is a reason to set very tight criteria, or at least to define workable criteria to facilitate quality improvement. But also it is also important that the diagnosis of diabetes, and changes in therapy should not be based on a single HbA_1c result. A second or third sample dramatically reduces the impact of analytical quality on interpretation, most evident in imprecision and to a lesser degree in bias (23). This provides a reason, perhaps, to set more relaxed quality criteria. Some organizations have already set quality targets. From current Danish quality requirements, a maximum CV of 2.8% (1.8%) and a maximum bias of 2.8% (1.8%) a TAE of 4.2 mmol/mol (0.39%) can be derived (21). The CAP acceptance limit of 6% is 4.3 mmol/mol (0.40%) at an HbA1c concentration of 50 mmol/mol (6.7%) (18): these criteria are close to the chosen TAE of 5 mmol/mol (0.46%) in this paper. The models in this paper can be used not only to set quality targets for individual labs, but also to set (desirable) quality goals between laboratories within methods (performance of users of a manufacturer) or between laboratories between methods (overall performance of all labs).

Taking into account these considerations, as well as the state of the art, the TF- HbA_1c suggests initially setting a TAE of 5 mmol/mol (0.46%) with a sigma of 2 for routine laboratories and a sigma of 4 for laboratories performing clinical trials where optimal accuracy and precision is a requirement. However, as stated above this is the starting point for discussion with stakeholders in the field of diabetes and of a drive towards improved quality rather than the final destination or level of quality.

Application

As demonstrated by the examples in this paper the model is suitable to estimate the quality status of HbA1c of a) a single laboratory, b) a single manufacturer, and c) a country. The model also allows comparison over time of analytical performances of manufacturers, which is helpful to select a method or to evaluate whether poor performance in a single laboratory is traceable to either poor operation in that laboratory or to the method used. For valid and fair conclusions it is important to confirm commutability and value assignment with the IFCC Reference Measurement Procedure of the samples in the EQA/PT program from which bias and imprecision data are taken to calculate TE.

Abbreviations

IFCC

International Federation of Clinical Chemistry and Laboratory Medicine

TF- HbA_1c

Task Force HbA_1c

TAE

total allowable error

TE

total error

SM

Sigma-metrics model

BV

biological variation model

EQA

external quality assessment

PT

proficiency testing

POCT

Point of Care Testing

SI

Système International

IDF

International Diabetes Federation

EASD

European Association for the Study of Diabetes

ADA

American Diabetes Association

NGSP

National Glycohemoglobin Standardization Program

CAP

College of American Pathologists

σ

sigma

CVi

intra-individual biological variation

CVg

inter-individual biological variation

B

bias

I

imprecision