Imprecision Profiling

Summary

• Imprecision profiles express precision characteristics of an assay over a range of concentration values. They can convert large quantities of potentially complex data into an easily interpreted graphical summary.

• Imprecision profile estimation does not require precisely structured data. This implies that structured method evaluation data can be easily compared with, or merged with, less structured internal quality control (QC) data (or with data from any other source).

• Although originally conceived for immunoassays, imprecision profiles could, in principle, be used as a summary method with any measurement system where precision varies with level of measurand.

Introduction

Precision profiles (also known as imprecision profiles) were conceived and developed in an immunoassay context by Professor R.P. Ekins.^1–3 The profiles originally defined by Ekins were calculated by combining assay response error with slope of the standard curve to obtain a prediction of error in terms of concentration. Two equations are involved;

$${\sigma }^2(\text{R})=\text{f}(\text{R})$$ (eq. 1)

$$\text{R}=\text{F}(\text{U})$$ (eq. 2)

Equation (eq.) 1 represents the fitted relationship between response error (σ²(R)) and level of response (R). It is typically a simple function such as a straight line, parabola or power function and is used as a weighting function when fitting the standard curve. Eq. 2 represents the fitted standard curve, relating response and concentration (U), and is most commonly a logistic function. Ekins’ precision profile is given by:

$${\sigma }^2(\text{U})={\sigma }^2(\text{R})/{[{\text{F}}^{\prime }(\text{U})]}^2$$ (eq. 3)

where F′Œ(U) denotes the first differential of the standard curve relationship with respect to U (i.e. curve slope). Once eqs 1, 2 have been estimated, eq. 3 is easily evaluated over a series of concentration values. It relates intra-batch error (now more commonly known as repeatability) to concentration and is usually plotted in terms of coefficient of variation (CV ) versus concentration. Immunoassay developers continue to make extensive use of eq. 3 to assess the “shape” of baseline error under varying design conditions and many immunoassay data reduction programs routinely plot the profile as a performance check on today’s assay.

However, there are limitations to this approach. First, the trend to singleton measurement over the last 20 years complicates estimation of eq. 1 and hence the evaluation of eq. 3. Second, eq. 3 is typically applied to single batches of data, but a single batch is not necessarily representative. Eq. 3 is less well suited to data collected from many batches and particularly when the standard curve has undergone positional and/or shape changes as a consequence of reagent lot changes or reagent ageing. Third, this general approach cannot be extended to predict between-assay error (now more commonly known as total error) or higher forms such as between-laboratory error.

The obvious practical solution is to directly fit a simpler variance model to sets of replicated assay results. Baxter was the first to attempt direct estimation, using a parabolic variance model:

$${\sigma }^2(\text{U})={\beta }_1+{\beta }_2\text{U}+{\beta }_3{\text{U}}^2$$ (eq. 4)

where β₁, β₂ and β₃ are the parameters.⁴ We found eq. 4 to be perfectly adequate with data from immunometric (excess reagent) assays, but it lacks sufficient curvature to reliably describe variance relationships for competitive (limited reagent) immunoassays. A simplification of eq. 3 suggested an alternative three-parameter variance function:

$${\sigma }^2(\text{U})={({\beta }_1+{\beta }_2\text{U})}^1$$ (eq. 5)

where β₁, β₂ and J are the parameters.⁵ Parameter J is free to take any value which implies the potential for greater curvature than is offered by a parabolic model. Over a 20 year period we have found eq. 5 to be a good general variance model for immunoassays.

This article consists of a mild critique of the precision guidelines promulgated by the Clinical and Laboratory Standards Institute (CLSI), followed by suggestions for ongoing assessment of precision using directly estimated imprecision profiles.

CLSI Guidelines

CLSI precision guidelines are the product of an expert committee and have acquired high status as a “gold standard” approach to estimating precision. The value of clear guidelines cannot be disputed but it is also important to remember that CLSI procedures are a consensus of many viewpoints. Consensus agreements are not necessarily optimal in specific situations. A scheme that might be entirely adequate for estimating precision of a serum sodium assay, for example, is not necessarily adequate for a complex protein, of unknown chemical structure, measured by immunoassay. The following potential limitations are worth bearing in mind:

Precision Stability

There is an implicit CLSI assumption that precision of any particular assay is more or less unvarying. All that is required, therefore, is a single, short-term precision experiment. That might be tenable for many medical laboratory tests, but it is certainly not universally true. Figure 1 illustrates repeatability imprecision profiles for a manual total serum thyroxine (T4) radioimmunoassay, which compare the best and least well performing operators during 1998 (a period in which several staff changes occurred). Note the large relative difference. This degree of variation is greater than we usually observed but it illustrates what can happen. Some degree of variable precision is expected for manual assays and it is easy to surmise that it would be more or less eliminated by substituting an automated instrument. Our experience with an automated immunoluminometric assay for serum thyrotropin (TSH), over a 10 year period, contradicted that notion to a quite unexpected degree. An initial precision evaluation in 1997 used seven serum pools which were each assayed on 36 occasions over a three month period, during which time three instrument calibrations were performed, using a different lot of calibration materials on each occasion. Five QC specimens were routinely employed during subsequent clinical use of the instrument (serum pools which were prepared and stored identically to the evaluation specimens and replaced at 5–7 month intervals). Figure 2 summarises more than 13000 evaluation or QC specimen results over a 10 year period. Only a single QC period (February through August 2002) produced precision results that matched the initial evaluation. All other QC periods (i.e. QC lots) demonstrated poorer precision, with wide variation. The variable precision illustrated in the Figures is unlikely to be unique and suggests that a CLSI experiment should be regarded as a starting point rather than an end in itself (at least for immunoassays).

Figure 1.

Data (clinical specimen duplicates) and estimated repeatability variance functions (eq. 5) for a manual total T4 radioimmunoassay, which compare the best and least-well performing operators during 1998 (from among eight staff who rotated through the various sections of the Nuclear Medicine Department, Christchurch Hospital). The inset shows the functions replotted in terms of CV (imprecision profiles), adjusted to reflect duplicate measurement, where shaded areas are approximate 95% confidence intervals and the arrows define the assay reference range (55–140 nmol/L). df = degrees-of-freedom.

Figure 2.

Total error imprecision profiles (eq. 5) for a TSH immunoluminometric assay on the Sano./Beckman/Coulter “Access” instrument (Beckman Coulter, Fullerton CA, USA). The profiles represent; 1: initial instrument evaluation, March through May 1997, seven specimens (solid circles), 245 degrees-of-freedom (df); 2: best QC period, February through August 2002 (QC #19), five specimens (open circles), 680 df; 3: worst QC period (QC #11), April through October 1998, five specimens (open squares – one data point off scale), 554 df; 4: overall QC profile (QC #9 through #29), September 1997 through July 2007, 105 specimens (data points omitted for clarity), 12884 df. All profiles reflect singleton measurement. Shaded areas are approximate 95% confidence intervals (note that confidence intervals for the overall profile are barely visible because they have virtually the same width as the plotted profile). In this case the assay reference range (0.25–2.5 mU/L) is defined by dashed vertical lines.

Calibration Effects

The CLSI design does not require multiple re-calibrations. Lot-to-lot variation in calibration materials (including matrix composition) is a well known source of error in immunoassays. In most cases variation is inadvertent and simply reflects technical difficulties, but there have certainly been instances of abrupt, relatively large “corrections” aimed at realigning calibration drift. Calibration effects might be seen as complicating a structured precision experiment but they are nevertheless fully incorporated into results sent to clinicians.

Specimen Matrices

Ward et al., using control materials from two different manufacturers, reported large, systematic and reproducible differences in precision results from an automated TSH immunoassay.⁶ There is no suggestion that large differences are common, but caution must nevertheless be exercised when comparing precision estimates derived from specimens of different types.

Complementary Schemes and Analyses

Variance function estimation can be readily applied to the duplicates accumulated in a CLSI experiment to obtain a repeatability imprecision profile along the lines of Figure 1. Likewise, and assuming four or more specimens have been used in the experiment, a total error profile can be estimated from replications accumulated across days. Repeatability and total error profiles are easily plotted on the same graph. Whereas CLSI analysis-of-variance is necessarily conducted on a specimen-by-specimen basis, imprecision profiles combine all the available data and give an informative view of the bigger picture. This is particularly relevant for immunoassays where relative change in variance can be many millions-fold. The profiles in Figure 2, for example, represent changes in variance of approximately one million-fold, despite the data covering less than 50% of the assay measurement range.

Routine QC materials are often similar to those used in CLSI evaluations and the simple strategy of assaying QC specimens in duplicate, and rotating their ordering and positioning, is effectively equivalent to a continuous CLSI evaluation. In practice, QC specimens are usually measured with the same replication as clinical specimens. Singleton QC results (implying singleton clinical specimen measurement) are entirely suitable for estimating ongoing total error profiles, but they obviously preclude ongoing repeatability profiles. The small additional cost of switching to duplicate QC measurement allows ongoing scrutiny of both repeatability and total error. In tandem they can provide useful insights into the causes of variable total error, in particular, whether it is due to variation in sources associated with repeatability (e.g. dispensing of samples and reagents, signal measurement) versus variation in sources associated with the day-to-day component (e.g. calibration effects).

Variance function estimation is not affected by occasional missing values. Any replicated data of interest can be submitted. We have found it valuable to routinely plot paired repeatability and total error profiles as part of internal QC, with monthly updates based on roll-on, roll-off of data. Once a database has been established routine summaries are easily augmented by any ad hoc comparisons of interest, e.g. different circumstances (Figure 1) or periods (Figure 2). As previously suggested, best-case and worst-case imprecision profiles are an effective way of summarising variable precision.⁷

It should be noted, however, that the high flexibility of variance functions such as eq. 5 means that nonsense profile “shapes” can occur if minuscule quantities of data are used, and especially when the range of mean values is small.⁵ The data generated by applying CLSI guidelines to four or more (widely spaced) specimens could be used as a convenient minimum. Once an initial body of data has been accumulated, roll-on, roll-off can be used to refresh precision estimates. The key is a well-organised system for collecting and maintaining data.

Calculations

A Win32 computer program for direct variance function estimation and plotting (e.g. Figures 1 and 2) is freely available from the AACB website.⁸ A link to the program can be found under Resources > Tools. References to the numerical methods used can be found in the program online help (Help > Contents > Overview > Introduction).

The term degrees-of-freedom (df) appears several times in the figures and figure legends. This quantity represents the independent precision information contained in a set of replicates and is the number of replicates minus one. Thus, a set of duplicate results has 1 df, a set of 20 replicates has 19 df. The overall df associated with an imprecision profile is the sum of the df associated with each data point.

Finally, consider whether imprecision profiles should be adjusted for replication. A repeatability profile estimated from within-day replicates (e.g. duplicates) intrinsically reflects precision for singleton measurement. Dividing variances throughout by 2 or 3 and so on [or dividing standard deviations or CVs by √2 or √3 etc.], are exact adjustments which reflect precision for duplicate measurement, triplicate measurement and so on. This simple adjustment is only appropriate for repeatability profiles. The computer program offers the adjustment, specifically in plots of CV versus mean, e.g. the main part of Figure 1 shows the raw (unadjusted) data and variance functions, but an adjustment to reflect duplicate measurement is used in the inset. This should be routinely applied when reported clinical results are means of duplicates but it could also be used to simply visualise the precision improvement that accrues from increased replication. Total error profiles require a more complicated correction for replication, but there is a simple alternative. In the case of CLSI data, or duplicate QC results, use the first of the duplicate values to estimate a total error profile which reflects singleton measurement (e.g. Figure 2). Use the means of the duplicates for a profile reflecting duplicate measurement.