Use of Capability Index to Improve Laboratory Analytical Performance

Summary

• For detection of errors where the standard deviation is wide compared to clinical requirements, consider use of the Capability Index (Cps).

• Cps is defined as ‘Allowable Limit of Error divided by the standard deviation of between-batch QC measurement’.

• ‘Capable’ assays have Cps >4 compared with ‘incapable’ ones with Cps <4.

• Capability of an analyte can be used to optimise the amount of quality control (QC) required and still maintain appropriate error detection.

• For EQA material, Cps can be used for continuously monitoring individual analytes, comparing laboratory performance with peers and comparison with industry-wide performance.

• The capability approach can result in reducing the number of QCs run per day, reducing costs as well as significantly improving the performance of a number of assays.

Introduction

Limitations of Quality Control

Historically, QC practices have been based on test performance by using the mean and standard deviation of control materials (e.g. mean ± 3SD) as the QC range. Although this may be statistically valid, such practices fail to allow for clinical need: crucial errors may not be detected for tests where the SD is wide compared with clinical requirements. Tests that easily meet clinical needs require a QC strategy with a low probability of false rejection (even if this results in a lower probability of error detection) whereas tests that barely meet clinical needs require a strategy ensuring high error detection (even though this would result in a high false rejection rate).^1,2 Process capability techniques, as derived from industry, enable us to compare our ability to measure an analyte directly with its clinical requirements.

The Concept of 6-Sigma

In the past few years, laboratories have considered the concept of 6-sigma as the quantitative goal for performance acceptability. The fundamental concept of 6-sigma is that 6 SD on each side of the mean of the process should fit within the designated limits, thus ensuring the number of defects (outliers) is very low indeed. The 6-sigma rationale also ensures that any process is relatively robust to unavoidable sources of variation. This is illustrated in Figure 1 which shows the measurement distribution of a process with a 1.5s, i.e. 1.5 SD, shift from the mean. As can be seen, even after a 1.5s shift, considered a standard in industry to account for variation over many cycles of processing, the process is still well within the designated limits. The defect rate per million per sigma metric is shown in Table 1. Also included in the table is the defect rate if the process was to shift by 1.5s. A process exhibiting 3.4 defects per million long-term is regarded as a “6-sigma” process (acknowledging that a shift of up to 1.5s may occur over time).

Figure 1.

The effect of a shift of 1.5s for a process where the capability index is 6 (“6-sigma”), showing that the process stays within the specifications (USL, upper specification limit).

Table 1.

Errors for centred process and after a 1.5s shift.

Sigma Metric	Defect per Million	Defect per Million
	No shift in mean	1.5s shift in mean
1.0	317,400	697,700
2.0	45,400	308,637
3.0	2,700	66,807
4.0	63	6,210
5.0	0.57	233
6.0	0.002	3.4

Capability

A capable process (or test) is one where almost all the measurements, e.g. the daily QC, fall within specified limits (clinical requirements) as represented in Figure 2A whereas Figure 2B shows a process that does not meet the selected specifications.

Figure 2.

Distribution showing a process that meets specifications (A) and one in which there are significant failures (B), where LSL is lower specification limit and USL is upper specification limit.

There are a number of different ways of measuring capability but the simplest for the QC environment is by using the capability index, Cps, which can be considered as a measure of the width of a QC material’s distribution compared with the allowed error defining the specified limits. For the types of measurements being made in the laboratory, the capability index is defined as:

$$\text{Cps}=\frac{\text{ALE}}{\text{SD}}$$

where ALE = span of the allowable limits of error and SD = standard deviation of between-batch QC measurements.

It is assumed in this case that the mean value of the quality control has been determined accurately and thus the bias is zero. There are more extensive definitions of the capability index in which the bias is included (e.g. [ALE − bias]/SD), and books on engineering statistics provide more information on capability measurements.³

For the laboratory, Cps defines the capability of tests as follows:

• < 4 Incapable

• 4 ≤ Cps ≤ 6 Capable

• ≥ 6 Highly capable 6-Sigma

The key to using the capability index is defining the ALE which should be related to the required clinical performance of the analyte. Sources of ALE that can be selected include total error^4,5 and the Royal College of Pathologists of Australasia Quality Assurance Programs (RCPA QAP).⁶ For an incapable assay, the alternatives to ensure appropriate performance are to change the method, which is often not practical, and to use sensitive QC algorithms to maximise error detection.

Capability as a Guide to Economy of QC

Many laboratories run too many unnecessary QC samples. Often laboratories will use the Westgard multi-rule algorithms irrespective of how the analyte performs. The capability of an analyte can be used to optimise the amount of QC necessary for the analyte so as to maintain appropriate error detection. This is done by using the SD to set the acceptable QC range (e.g. mean ± 3SD) and then the capability index to set appropriate Westgard rules for that assay as described in Table 2.

Table 2.

Quality control rules dictated by the capability index.

Capability Index (Cps)	Westgard Rules	Required Number of Quality Controls
≥6	13s	2
4≤ Cps < 6	Full rules	2
< 4	Full rules	4

For Cps <4, either two levels of QC can be run in duplicate or ideally, four different QC materials are used. Although using these algorithms appears complex, they are designed to optimise a low probability for false rejection and a high probability for error detection as shown by power function curves.⁷ Furthermore, if mathematical limits for error detection probability and rejection rates are set using these power charts, the number of QC runs required to achieve the desired performance can be calculated. Generally, laboratories should be trying to achieve a greater than 90% probability of error detection and less than 5% probability of false rejection. Further explanation of the Westgard rules and of the effect of differing numbers of QCs being run is available.⁸

Practical Examples of Using Capability for QC

The simplest way to measure capability is to select the ALE as outlined above, use the SD from your laboratory’s QC charts to calculate the Cps, and then tailor the Westgard rules and number of QCs to run as in Table 2. However, to use this process routinely in the laboratory, the QC requirements for each analyte need to be programmed individually on the analyser and in the QC recording files.

In our laboratory, we use Cps to review the assay performance and rules on a quarterly basis (Figure 3). For the data in Figure 3, the performance shown by the diamonds and circles and Cps <4 suggests that QC should be run with all rules and at four levels, whereas the performance shown by the triangles and squares and Cps, generally 4–6, suggests that QC should be run with all rules and at two levels.

Figure 3.

Capability index for magnesium assessed from QC data. The performance shown by the diamonds and circles, and Cps < 4, suggests QC would be run with all rules and at 4 levels; the performance shown by the triangles and squares and Cps, generally 4–6, suggests that QC should be run with all rules and at 2 levels.

Practical Examples of Using Capability for External Quality Assessment (EQA) Material

For EQA material, Cps can be used for continuously monitoring individual analytes, comparing laboratory performance with peers, comparison with users of the same instrument and with industry-wide performance.

In our laboratory, we use the RCPA QAP end-of-cycle summary data and the ALE from the RCPA QAP to calculate the assay capability in exactly the same way as for QC. We then compare this with the data for the assay peers and the best 20% of laboratories which generates the example shown in Figure 4. From this graph, it can easily be decided that if the assay is highly capable it does not require any intervention but if it is incapable or performing significantly below peer performance, it needs to be reviewed and troubleshooting to take place so that it reaches at least peer performance.

Figure 4.

Capability index (ALE/SD) data determined from the end of cycle QAP and showing a comparison of the laboratory data with the peer group and the best 20% of laboratories. The capability index for the laboratory (♦), best 20% of laboratories (▪) and the instrument peer group (▴) is shown. Horizontal line indicates an example of a Cps of 3.

The same data is also used to generate an instrument plot in which all the analytes for a particular analyser are plotted in a single graph. Again, comparison is against the peer group and the best 20% of laboratories. This plot is similar to the summary plot recently introduced by the RCPA QAP in their end-of-cycle summaries except; in our case, bias is not included.

A further useful plot is the 3 x 3 grid (Figure 5). In this graph, the laboratory capability index is plotted against either the instrument capability for that assay or the best 20% of laboratories. This plot is then used to identify those assays which should be worked on further. If it found that a laboratory assay is performing poorly but the industry as a whole also has a problem, then it is going to be difficult to improve the assay significantly. A good example of such an assay is calcium which has been shown to perform extremely poorly in surveys when compared with the clinical requirements for the assay. However, if the laboratory performance is poor compared to the peer group (top left hand box), then this is an assay that should be worked on as improvements are readily possible.

Figure 5.

A 3 x 3 grid showing the laboratory capability index plotted against the peer group. The data shown are different to those in Figure 4.

Conclusion

The quality demands on laboratories continue to increase and we need to be mindful of the clinical requirements of the results we produce. However, running more QC or more EQA will not necessarily achieve this end, and processes must be put in place to maximise error detection and improve performance without necessarily increasing laboratory costs.

By using the capability index approach as described above, laboratories are able to identify assays for which performance can be readily improved and introduce specific algorithms to ensure acceptable performance is maintained. In our laboratory, adopting this approach has resulted in reducing the number of QCs run per day and its attendant costs as well as significantly improving the performance of a number of assays.