Basics of Estimating Measurement Uncertainty

Summary

• All measurements are imperfect and have many potential sources of variation.

• An estimate of measurement uncertainty (MU) provides an interval of values within which the true value is believed to lie with a stated probability, and is therefore a quantitative indication of the reliability of a measurement.

• MU estimates are essential for assessing whether methods are suitable for clinical use and for comparison of results of a similar type.

• MU estimates can help identify method limitations and opportunities for improvement.

Introduction

Regardless of method, repeated measurements on the same sample will generally produce different results if the system is sufficiently sensitive. The dispersion of results obtained from such repeated measurements (imprecision) can be described approximately by a normal probability (Gaussian) distribution, with some 95% of the results falling within ± 2 standard deviations (SD) of the mean value. Measurement results are thus unreliable and should be regarded as best estimates of the true value of the quantities being measured. Some knowledge of the result variability expected from a given measurement procedure is required if results are to be meaningfully compared with other results of the same kind or with decision and legal limits. For many years the error concept has been widely used, including by clinical biochemistry, to quantify measurement unreliability and to set quality goals for methods.

Error Concept

Briefly, total error comprises systematic (bias) and random error (imprecision), e.g. if a reference material is repeatedly measured by a routine method under replicate conditions, the difference between the mean replicate value and the assigned reference value is the bias of the procedure, and the dispersion of the obtained values, expressed as a standard deviation, is the imprecision. Total error is usually expressed as average bias + 1.65 SD (~95 % confidence). A limitation of the approach is that by adding bias and imprecision total error calculates the worst case error for the procedure, and does not recognise that individual results may have less error.

Concept of Measurement Uncertainty (MU)

In the 1990s it was recognised that measurement comparability between laboratories and methods required an internationally agreed approach to estimating and expressing measurement uncertainty, which is described in the ‘Guide to the Expression of Uncertainty in Measurement’ (GUM). Although GUM is primarily for measurements in physics, the principles are applicable to biological and chemical measurements. The terminology of the science of measurement (Metrology) is defined in the ‘International Vocabulary of Basic and General Terms in Metrology’ (VIM).

The basic parameter of MU is the SD, and the symbol for uncertainty is u. Some additional useful terms and definitions are given in the table.

Table.

Common metrological terms and their definition (VIM).

Term	Definition
Coefficient of variation (CV); also termed relative standard measurement uncertainty	Standard measurement uncertainty (SD) divided by the absolute value of the measured quantity value. CV = SD/x or SD/mean value.
Combined standard measurement uncertainty (uc)	Standard measurement uncertainty that is obtained using the individual standard measurement uncertainties associated with the input quantities in a measurement model.
Coverage factor (k)	Number larger than one by which a combined standard measurement uncertainty is multiplied to obtain an expanded measurement uncertainty.
Coverage interval	Interval containing the set of true values of a measurand with a stated probability, based on the information available.
Expanded measurement uncertainty (U)	Product of a combined standard measurement uncertainty and a coverage factor larger than the number one.
Measurand	Quantity intended to be measured.
Measurement uncertainty	Non-negative parameter characterising the dispersion of the quantity values being attributed to a measurand, based on the information used.
Quantity	Property of a phenomenon, body, or substance, where the property has a magnitude that can be expressed as a number and a reference.
Standard measurement uncertainty (u)	Measurement uncertainty expressed as a standard deviation.
Trueness	Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value.
True value	Quantity value consistent with the definition of a quantity.

These and other relevant definitions can be found at the BIPM website http://www.bipm.org/en/publications/guides/vim.html

In practice, bias correction and replicate measurements can reduce, but not completely eliminate systematic and random errors, and therefore total error cannot be exactly known. It follows that the true value of a measured quantity cannot be exactly known either. This assumption is fundamental to the MU approach. The MU concept also assumes that if the bias of a procedure is known, then steps are taken to minimise it, e.g. by re-calibration. However, because the bias value cannot be known exactly, an uncertainty will be associated with such a correction. Thus, in the MU concept, a measurement result can comprise two uncertainties (i) that associated with a bias correction (u_Bias), and (ii) the uncertainty due to random effects (imprecision, u_Imp). Both these uncertainties are expressed as SDs which, when combined together, provide the combined standard uncertainty for the procedure (u_Proc).

Thus the MU approach considers a single measurement result to be the best estimate of the measured quantity, and centred on this the combined standard uncertainty provides an interval of values within which the true value of the measured quantity is believed to lie, with a stated coverage probability. For example, a plasma glucose concentration of 5.4 ± 0.1 mmol/L (95% coverage probability), means the true value for the glucose concentration is believed to lie within the interval 5.3–5.5 mmol/L with a probability of 95%.

The GUM describes a bottom-up approach to estimating MU, by which an uncertainty budget for a given measurement procedure is assembled by identifying all potential sources of uncertainty (e.g. calibration, weighing, pipettings, temperature and instrument fluctuations) and attributing to each an uncertainty estimate as an SD obtained by experiment or from available information. The contributing uncertainties are combined in a mathematical model that best represents their interactions in the measurement process. The combining calculation yields the estimated combined standard measurement uncertainty for the whole procedure (u_c). A simple example is the estimation of the concentration of glucose (c) by weighing it (w) into a known volume of water (v). The weighing and volume uncertainties, estimated as SDs, are then combined in the model c = w/v to calculate the combined standard uncertainty of c.

The GUM bottom-up approach can quickly become unwieldy and mathematically complex. Fortunately, clinical biochemistry measurement methods employ quality control (QC) materials to estimate and monitor whole procedure imprecision, so QC data can be used to estimate the contribution of random effects (u_Imp) to the measurement uncertainty of the whole procedure (u_Proc), with the assumption that the measurand behaves identically in both patient samples and quality control material.

If a procedure has been adjusted for bias, then the uncertainty associated with the correction (u_Bias) may need to be combined with u_Imp to estimate u_Proc. This decision depends on the magnitude of u_bias relative to u_Imp. The t test can be used to objectively assess the relative significance of u_bias, or sometimes a subjective decision is made, e.g. ignore u_Bias if it is <10% of u_Imp. If u_Bias is ignored, then u_Proc = u_Imp. Similarly, if bias itself is assessed as not being significant or it is not evaluated, then again u_Proc = u_Imp. If u_Imp varies significantly across the reportable range, more than one estimate may be required. This top-down approach is generally recognised as a direct estimate of the combined standard uncertainty of the whole procedure (u_c) using the GUM approach.

Estimating MU in the routine biochemistry laboratory

The effort and cost of estimating MU should be commensurate with the clinical quality of measurement required.

MU Goals

It is important that the MU for a given measurement procedure falls within clinically acceptable limits, so that results are of appropriate quality and reliability for patient management. Depending on analyte physiology, specimen type and clinical use of results, MU goals may be based on biological variation, expert group recommendations, or professional opinion. It should be noted that MU goals cannot always be met due to performance limitations of available routine technology.

Step 1: Defining the Measurand

A measurand is the quantity intended to be measured and should be well defined. This is straightforward for analytes that are chemically well characterised, such as sodium or urea, but may be difficult when the measured quantity is method dependent e.g. enzyme activity at a specified pH or temperature, or peptide hormone immunoassays where antibody and epitope specificity may differ between manufacturers. In such cases the measurand is procedure-dependent and measurand definition should include sufficient detail e.g. X Diagnostics two site sandwich β human chorionic gonadotropin immunoassay.

Measurand definition includes the system containing the component (analyte) of interest, e.g. whole blood, plasma, cerebrospinal fluid, and may require further specification (e.g. urine, early morning first void). The measurand description must also identify the kind-of-quantity being examined (e.g. amount-of-substance concentration, amount-of-substance activity, number concentration).

Uncertainty can be associated with the measurand due to:

• Incomplete definition, e.g. antibody interaction with peptide fragments

• Imperfect realisation of the measurand, e.g.

  • - incomplete hormone release from natural binding proteins

  • - inadequate knowledge of environmental effects on measurements

  • - inexact values for standards, calibrators, constants, calculations

  • - approximations, assumptions of method

  • - unaccounted systematic and random effects.

Step 2: Imprecision of Measurement

Because patient results are compared over time with clinical decision limits or previous results, it is essential to estimate imprecision across as many unavoidable standard operating procedure variables as possible, e.g. calibrator and reagent batch changes, instrument maintenance, different operators, environment (intermediate conditions). The intermediate imprecision (u_Imp), expressed as SD, is an estimate of the uncertainty due to the random effects of the whole procedure over time. u_Imp may be required at more than one analyte level across the reportable range. If the procedure has not been adjusted for bias, then

$$u_{\text{Proc}}=u_{\text{Imp}}$$

Step 3: Bias of Measurement

The GUM approach assumes that if the bias of a procedure has been estimated, usually by replicate measurements of a reference material, then it is minimised by re-calibration or by a correction factor. However, bias cannot be known exactly, and so the value used for re-calibration or result correction will have an associated uncertainty (u_Bias) due to the uncertainty of the:

• analyte value assigned to the reference material used to assess the bias (u_Ref)

• mean value of the analyte in the reference material when measured in replicate by the routine procedure (u_Rep)

u_Ref is obtained from the reference material certificate, and u_Rep is the standard error of the mean (SEM) of the replicate measurements.

u_Bias is calculated by combining the two uncertainties:

$$u_{\text{Bias}}={(u_{\text{Ref}}{}^2+u_{\text{Rep}}{}^2)}^{1/2}$$

Hence, the bias of a procedure = Bias value ± u_Bias

u_Bias should be assessed for significance relative to the procedure imprecision (u_Imp) as described earlier. If an appropriate reference material or reference procedure is unavailable, then alternative approaches may be used, e.g. external quality assessment data or inter-laboratory comparisons.

Step 4: Calculation of MU

N.B. Independent uncertainties are combined as variances:

$$u_{\text{C}}={(u_{\text{A}}{}^2+u_{\text{B}}{}^2+\dots u_{\text{X}}{}^2)}^{1/2}$$

Scenario 1. Bias ignored or not evaluated (Appendix 1)

Appendix 1.

Scenario 1. Estimation of MU for a glucose measurement procedure ignoring bias uncertainty (artificial data for illustrative purposes).

Parameter	Data	Calculations	Comment
Measurand	Amount-of-substance concentration of glucose in plasma or serum
Units	mmol/L
Procedure	Hexokinase/G-6-P dehydrogenase
Traceability	Isotope Dilution-Mass Spectrometry method; high purity preparation, NIST SRM 917b		From reagent manufacturer information sheet.
Imprecision
QC Level 1:		04/02/07–08/05/07	Imprecision for intermediate conditions included one batch change of reagent(s) and calibrator, several operator changes, one instrument routine maintenance.
Mean	4.80 mmol/L	n=317 data points
SD (uImp)	0.11 mmol/L
QC level 2:
Mean	15.70 mmol/L	04/02/07–08/05/07
SD (uImp)	0.38 mmol/L	n=320 data points
Bias *	Not assessed		EQA program did not indicate bias problem
Measurement			uProc considered = uc
Uncertainty uProc
Plasma glucose:
4.8 mmol/L	uProc = uImp = 0.11 mmol/L
15.7 mmol/L	uProc = uImp = 0.38 mmol/L
Expanded uncertainty:		k = 2	Multiplication of uProc by a coverage factor of 2 provides an interval of values that is believed to include the true value with a coverage probability of 95%.
4.8 mmol/L	U = 0.22 mmol/L
15.7 mmol/L	U = 0.76 mmol/L
Coverage intervals:			For ease of general use by lab staff and for lab ‘handbooks’, data is rounded.
5.0–10.0 mmol/L	± 0.2 mmol/L
10.0–20.0 mmol/L	± 0.8 mmol/L		Coverage probability 95%

^*Bias was not evaluated or was evaluated and assessed as not significant.

If the procedure has not been adjusted for bias, then the estimated MU of the procedure is the intermediate imprecision expressed as 1 SD i.e.:

$$u_{\text{Proc}}=u_{\text{Imp}}$$

Scenario 2. Bias evaluated (Appendix 2)

Appendix 2.

Scenario 2. Estimation of MU for a glucose measurement procedure including bias uncertainty (artificial data for illustrative purposes).

Parameter	Data	Calculations	Comment
Imprecision (uImp)	0.11 mmol/L		Data from scenario 1. QC Level 1: plasma glucose 4.80 mmol/L.
Bias	Evaluated		Level 3 CRM (certified reference material) 965a measured under replicate conditions (n=10).
CRM 965a: Glucose.			Data from CRM certificate includes assigned value and expanded uncertainty U (k = 2).
Level 3 value:	6.777 mmol/L
U:	0.073 mmol/L		Calculate uCRM from expanded uncertainty.
uCRM	0.0365 mmol/L	Calculated as U/2
CRM 965a value using routine procedure (n=10)	6.9, 7.0, 6.9, 6.8, 7.2, 7.1, 6.9, 6.8, 7.2, 6.9		10 measurements of SRM glucose using routine procedure under repeatability conditions (N.B. artificial data for illustration only).
Mean value	6.97 mmol/L
SD	0.149 mmol/L
SEM	0.047 mmol/L	SEM = 0.149/√10 = uRep	Standard error of mean.
Bias *	0.193 mmol/L	6.970–6.777 mmol/L	Mean of replicate measurements by routine procedure minus CRM assigned value.
uBias	uCRM: 0.0365 mmol/L		uBias is uncertainty of value used for bias correction (0.193 mmol/L).
	uRep: 0.047 mmol/L
	uBias: 0.0595 mmol/L	uBias = (0.03652 + 0.0472) 1/2	uBias calculated by combining uCRM and SEM of mean CRM value estimated by routine procedure (uRep).
Is Bias significant?		t = Bias/uBias	t test; use distribution table for t (2 tailed) values
	t = 3.24	0.193/0.0595	df = degrees of freedom.
	0.01<P<0.02	df = 10−1 = 9	Bias is significant; hence, procedure re-calibration should include offset of −0.193 mmol/L.
Is uBias significant?	uBias/uImp = 0.54	0.0595/0.11	uBias is of significant magnitude relative to imprecision, therefore include in MU estimate.
Measurement Uncertainty			Combined standard uncertainty for whole procedure by combining uBias and uImp.
uProc		uProc = (uBias2 + uImp2)1/2
	0.125 mmol/L 0.1 mmol/L (rounded)	(0.05952 + 0.112)1/2 = 0.125	uProc is rounded to same number of decimal places as reported results (unless calculating U).
Expanded Uncertainty: U	0.25 mmol/L 0.3 mmol/L (rounded)	k = 2; 2 x 0.125 = 0.250	Coverage factor 2 to calculate U. U is rounded to same number of significant figs as reported results.
Coverage interval: 5.0–10.0 mmol/L	± 0.3 mmol/L		Rounded data for general use by lab staff and ‘handbook’. Coverage probability 95%.

^*Bias evaluated and procedure adjusted to minimise estimated bias.

If u_Bias is assessed as being significant relative to u_Imp (e.g. >10%), then it is combined with u_Imp, so that:

$$u_{\text{Proc}}={(u_{\text{Bias}}{}^2+u_{\text{Imp}}{}^2)}^{1/2}$$

Step 5: MU of a Measurement Result Calculated from Other Measurement Results

If a test result is calculated from the results of other measurements (inputs), then the MU of the final result is obtained by combining the uncertainties of the independent inputs with the same rule used above (square root of the sum of the squares), but the choice of using SD or coefficient of variation (CV) depends on whether the inputs interact by addition or subtraction (use SD), or by multiplication or division (use CV).

Scenario 3. MU of calculated plasma anion gap (Appendix 3)

Appendix 3.

Scenario 3. MU of plasma anion gap (u_AG).

Calculation of anion gap	(Na+k) − (Cl+HCO3)
Patient X	Na: 137; K: 4.0; Cl: 106; HCO3: 10 (mmol/L) AG = 25 mmol/L
MU of AG calculated as square root of sum of squares of MUs of contributing results.	uAG = (uNa2 + uK2 + uCl2+ uHCO32)1/2
Calculation of AG uses addition and subtraction; hence, calculate uAG using SDs.	SDNa = 1.48 mmol/L; SDK = 0.04 mmol/L; SDCl = 0.72 mmol/L; SDHCO3 = 0.84 mmol/L
Combining uNa, uK, uCl, uHCO3.	uAG = (1.482 + 0.042 + 0.722 + 0.842)1/2 = 1.85 mmol/L
Coverage factor k = 2.	UAG = 3.7 mmol/L (95% coverage probability)
Result rounded for clinical use.	uAG = 4 mmol/L
Patient result.	AG = 25 ± 4 mmol/L (95% coverage probability)

$$\text{Anion\hspace{0.17em}gap\hspace{0.17em}}(\text{AG})=(\text{Na}+\text{K})-(\text{Cl}+{\text{HCO}}_3)$$

The MU for plasma anion gap (u_AG) is calculated by combining the MUs of the analyte results used in the anion gap (AG) equation.

$$u_{\text{AG}}={(u_{\text{Na}}{}^2+u_{\text{K}}{}^2+u_{\text{Cl}}{}^2+u_{\text{HCO}3}{}^2)}^{1/2}$$

Because the AG is calculated by adding and subtracting the four contributing results, SDs must be used for combining their uncertainties when calculating u_AG.

Scenario 4. MU of calculated creatinine clearance (Appendix 4)

Appendix 4.

Scenario 4. MU of calculated creatinine clearance (u_CrCl).

Calculation of creatinine clearance (Clcr)	$\frac{\text{Urine\hspace{0.17em}creatinine\hspace{0.17em}}(\mu \text{mol}/\text{L})\times \text{Urine\hspace{0.17em}volume\hspace{0.17em}}(\text{mL})}{\text{Plasma\hspace{0.17em}creatinine\hspace{0.17em}}(\mu \text{mol}/\text{L})\times \text{Collection\hspace{0.17em}time\hspace{0.17em}}(min)}$
Results for 68-year-old woman.	Plasma creatinine: 92 μmol/L, SD: 2.26 (QC), CV: 0.0246 Urine creatinine: 2560 μmol/L, SD: 340 (QC), CV: 0.1328 Urine volume 2683 mL, SD: 25 (estimate), CV: 0.0093 Collection time: 24 h (1440 min), SD: 30 (estimate), CV: 0.0208
Creatinine clearance.	$\frac{2560\times 2683}{92\times 1440}=51.8\hspace{0.17em}\text{mL}/min$
MU of Clcr calculated as square root of sum of squares of MUs of contributing results	uCrCl = (u2UCreat + u2UVol+ u2PCreat+ u2Time)1/2
Creatinine clearance calculated using division and multiplication; therefore must calculate uCrCl using CVs.	CVCrCl = (CV2UCreat + CV2UVol + CV2PCreat + CV2Time)1/2 = (0.02462 + 0.13282+ 0.00932 + 0.02082)1/2 = 0.137
SD = measurement result x CV.	SD = 51.8 mL/min x 0.137 = 7.096 mL/min = uCrCl
Coverage factor k = 2.	UCrCl = 14.192 mL/min
Result rounded for clinical use	UCrCl = 14.2 mL/min
Patient result.	51.8 ± 14.2 mL/min (95% coverage probability)

$$\begin{array}{l}\text{Creatinine\hspace{0.17em}clearance}=\\ \frac{\text{Urine\hspace{0.17em}creatinine\hspace{0.17em}}(\mu \text{mol}/\text{L})\times \text{Urine}\hspace{0.17em}\text{volume\hspace{0.17em}}(\text{mL})}{\text{Plasma\hspace{0.17em}creatinine}\hspace{0.17em}(\mu \text{mol}/\text{L})\times \text{Collection\hspace{0.17em}time\hspace{0.17em}}(min)}\end{array}$$

The MU for creatinine clearance (u_CrCl) is similarly calculated by combining the MUs of the values used in the creatinine clearance equation.

$$u_{\text{CrCl}}={(u_{\text{UCreat}}{}^2+u_{\text{UVol}}{}^2+u_{\text{PCreat}}{}^2+u_{\text{Time}}{}^2)}^{1/2}$$

However, because the creatinine clearance calculation uses multiplication and division, the contributing MUs must be expressed as CVs when calculating u_CrCl.

Step 6: Expression of MU

MU is essentially a normal probability distribution of values within which the true value is believed to lie with a stated probability. The MU parameter is 1 SD. The number of significant digits given for an MU should be the same as that used for reported results. For practical purposes, u is usually expanded (U) by a coverage factor (k), commonly two, to provide an increased coverage probability, e.g.

$$\text{Plasma\hspace{0.17em}glucose\hspace{0.17em}}(\text{random})=5.7\pm 0.3\hspace{0.17em}\text{mmol}/\text{L\hspace{0.17em}}(95\%\hspace{0.17em}\text{coverage}\hspace{0.17em}\text{probability}).$$

Further Reading

Useful for Medical Laboratories

Department of Health and Ageing. Requirements for the estimation of measurement uncertainty. National Pathology Accreditation Advisory Council. Canberra, Australia, 2007.

Fraser CG. Biological Variation: From Principles to Practice. Washington, DC, USA: AACC Press; 2001.

Westgard JO. Desirable specifications for total error, imprecision and bias, derived from biologic variation - databases for goal setting. http://www.westgard.com (Accessed 28 February 2008)

White GH, Farrance I. Uncertainty of measurement in quantitative testing. Clin Biochem Rev 2004;25:S1–24.

MU and Analytical Chemistry

EA Guideline A-4/16: Expression of uncertainty in quantitative testing. EA 2003. http://www.europeanaccreditation.org (Accessed 28 February 2008)

Guide to the evaluation of measurement uncertainty for quantitative test results. Eurolab Technical Report No 1/2006, August 2006. http://www.eurolab.org (Accessed 28 February 2008)

Measurement uncertainty revisited: Alternative approaches to uncertainty evaluation. Eurolab Technical Report No 1/2007. http://www.eurolab.org (Accessed 28 February 2008)

Quantifying uncertainty in analytical measurement. Eurachem/CITAC Guide: 2000.http://www.measurementuncertainty.org (Accessed 28 February 2008)

ISO Normative References, Technical Specification

ISO/IEC Guide 99:2007, International vocabulary of metrology – Basic and general concepts and associated terms (VIM). Geneva Switzerland: ISO; 2007.

ISO/IEC Guide 98:1995. Guide to the expression of uncertainty in measurement. Geneva, Switzerland: ISO; 1995.

ISO/TS 21748:2004. Guide to the use of repeatability, reproducibility and trueness estimates in measurement uncertainty estimation. Geneva, Switzerland: ISO; 2004.