Introduction
External quality assurance (EQA) has been described as the ‘fifth pillar’ of laboratory standardisation, added to the four pillars of reference materials, reference methods, reference laboratories and traceable reference intervals and decision points.1 One vital component of an EQA program is to provide criteria to allow participating laboratories to assess their performance. These criteria, also described as quality standards, may be based on statistical comparison with peers, expert opinion, clinical need or other criteria. In Australia, the RCPA Quality Assurance Programs (RCPAQAP) quality standards are known as the Allowable Limits of Performance (ALP) and are used to assist with interpretation of all numerical QAP results in the program. The stated goal of these limits is that they are clinically based to provide warning of clinically important changes in assay performance. As these limits were derived some years ago2 it is appropriate to review them. This document describes the assessment and implementation of a hierarchical approach to setting quality standards for EQA programs using the approach used by the RCPAQAP Chemical Pathology as an example.
Quality Standards for EQA: The Need for Definitions
Quality standards vary considerably between different EQA programs for the same analytes.3 This may be due to different meanings behind the standards, assessing different data or making different interpretations of available information. However when a quality standard is applied it is important that the meaning of the standard is known. For example, because failing the regulatory standards in use in the USA4 or Germany5 has important consequences for the laboratory’s right to practice, they are usually able to be achieved by nearly all laboratories. By contrast targets set as goals for improvement may not be reached by all laboratories at the time of implementation. For example when the 10% coefficient of variation (CV) requirement for troponin assays was first proposed it was not generally able to be reached at the important decision point although recent assays are able to meet this criterion.6 Thus when EQA quality standards are being used by laboratories, it is important to clearly understand the intention of the standard in order to use it appropriately. Similarly, when comparing quality standards from different EQA programs it is important to be aware of the rationale behind them. Thus in the process of setting quality standards for an EQA program the initial step is to define the purpose and meaning of the standards as this will greatly influence the decision-making process for the EQA as well as guide the response of client laboratories to the standards.
Quality standards for EQA programs may be used on different types of data from the same laboratory. For example standards may be applied to single results shortly after the time of analysis, or to accumulated data at the end of a distribution cycle. Standards for the former are necessarily total error standards because single results will always include errors caused by both bias and imprecision. When multiple results for an analyte are available it is possible to separately assess bias and precision and apply specific standards. For the purposes of this discussion the main focus will be on quality standards applied to individual results as total error standards. The development of separate bias and imprecision standards for accumulated data is also an important process but is not considered here in detail.
Quality Standards and Between Laboratory Variation
The traditional service provided by EQA programs allows individual laboratories to assess the performance of an assay and respond accordingly. More commonly now laboratories have multiple analysers or are parts of laboratory networks in which a patient’s result may be generated from any one of many analysers in the system. It is also becoming more common that patients may have pathology testing performed in many different laboratories, and that results may be collated, either informally in a collection of printouts or formally in a combined database. In these settings a different paradigm becomes apparent. The EQA results provide information about the analytical performance between as well as within laboratories, and quality standards must be designed with this in mind.
For diagnostic purposes, the question is whether different laboratories can use the same decision points or reference intervals, and for monitoring purposes the issue is whether a patient can be reliably monitored using results from different laboratories. This relates to a key issue with regard to common pathology databases, namely whether we can combine the results from different laboratories in the same database and whether the combined data is suitable for diagnosis and/or monitoring. This provides an additional purpose for EQA quality standards to provide information on the use of data in combined pathology databases.
Quality Standards and Targets
When assessing individual results in an EQA program, the quality standard is the allowable difference from a target. Targets may be set in a number of ways including: statistical values such as the median of all returned data or of the relevant method groups; reference measurements performed on the samples; comparison with reference materials; or weighed in values for substances such as drugs. Assessment of results using quality standards expressed as a difference from a target automatically makes the assessment one of total error, i.e. both bias and imprecision are included in the process. This has the effect of making the QAP target the reference standard for the interpretation and thus the choice of target is vitally important. For that reason target setting will be discussed in further detail.
The different approaches and their strengths and weaknesses are as follows:
Use of Reference Measurements
Where reference measurement procedures are available, they offer the most reliable target-setting, especially when native human serum samples or pools are used as the EQA material. In practice some modification of materials is required to achieve the desired range of results, number of analytes and stability during transportation or storage. These modifications may lead to lack of commutability of the material between different methods, often referred to as a matrix effect. Indeed with the exception of native patient samples, commutability should not be assumed without experimental confirmatory evidence.7 Similarly, between-method differences should not be assumed to be due to matrix effects without confirmation.
Use of Overall Medians
Medians may be used with or without statistical removal of outliers. The RCPAQAP uses all data in its calculations of a median. In common with reference measurements, the use of a single target for results for all methods has the advantage of highlighting the possibility of differences between methods which can then be investigated for matrix effects as needed. A weakness is that the overall medians can be influenced by the method composition of the participating laboratories and favours any numerically dominant method. Indeed with two similarly sized method groups which give different results, the median may shift dramatically from challenge to challenge depending on the exact numbers in the groups, and on some occasions may be representative of no group in particular.
Use of Method-specific Medians
Use of medians based on method groups has the advantage of removing the matrix effect but limits the interpretation to the question “Is my method working similarly to other users of the same method?” without providing any information on between-method differences. Method-specific medians are also problematical with open-systems such as are used in some chemistry analysers where there may be a choice of possible comparators. Is the appropriate peer the analyser, an analyser group (i.e. from the same manufacturer), the analytical principle, the reagent manufacturer, the calibrator or some other variable? For closed systems such as automated immunoassay platforms or haematology analysers, the use of medians, either overall or more commonly method-specific, also has the potential to mask changes over time if the entire group is changing as one. This has been seen with a biased calibrator affecting an entire manufacturer group where the change was not apparent because the target (median value) was necessarily changing to the same extent.
Weighed-in Values
While attractive for programs with exogenous analytes such as drugs, there are a number of pitfalls in making simple additions of this type. If a drug is present in the circulation both in native form and as metabolites, this will not be reflected in the EQA material, potentially leading to matrix effects. Addition of drugs to samples in vitro may not produce the same protein-binding seen in vivo, and the presence of multiple drugs in an EQA sample may produce effects different from those seen in patients. Moreover technical problems in obtaining full dissolution of the drug in the sample without degradation can make the weighed-in values erroneous. Thus this method of target assignment needs to be used with caution.
Quality Specifications: The Stockholm Consensus Conference
In 1999 a conference was held in Stockholm under the auspices of the WHO, IFCC and IUPAC. The outcome was a consensus statement titled: Strategies to Set Global Quality Specifications in Laboratory Medicine.8 The strategies consist of a series of approaches arranged in a hierarchy, with the aim where practicable to apply models higher in the hierarchy in preference to those at lower levels. The key statement is summarised in Table 1 with the different levels in the hierarchy considered in more detail below. Implementing the hierarchy involves assessing the available information about a test and applying the highest available criterion. For example, if there is reliable data on biological variation (level 2), this should be used in preference to expert opinion (level 3) or state of the art (level 5).
Table 1.
Draft Consensus Statement
| The main outcome of the Conference was agreement that the following hierarchy of models should be applied to set analytical quality specifications. |
|
| Where available, and when appropriate for the intended purpose, models higher in the hierarchy are to be preferred to those at lower levels. |
Stockholm Consensus Detailed Evaluation
Level I. Evaluation of the effect of analytical performance on clinical outcomes in specific clinical settings:
The requirement of this criterion is the existence of clinical trial data demonstrating that a certain level of analytical performance is needed for a particular clinical outcome. A major problem with this approach is the lack of suitable studies to set the criteria. Ideally a study would compare outcomes when assays with different analytical characteristics are used. More commonly studies are used where a difference in result between individuals in two groups leads to a different outcome, thus demonstrating the need to be able to separate results from the two groups analytically. The most commonly quoted example of this type of data is the Diabetes Control and Complications Trial (DCCT) related to intensity of glucose regulation in type 1 diabetes mellitus.9 The average HbA1c in the two treatment groups were approximately 7.0% (53 mmol/mol) and 8.0% (64 mmol/mol), thus leading to the requirement for assays with coefficients of variation of less than about 3% for assays measuring in % (∼4% for assays measuring in mmol/mol), although this calculation does not include any allowance for within-subject biological variation. While this type of study interpretation shows a significant difference between the groups in the trial this difference does not necessarily equate to separating individuals from the two treatment groups.
Level II. Evaluation of the effect of analytical performance on clinical decisions in general:
data based on components of biological variation
data based on analysis of clinicians’ opinions
Biological variation is an exciting development in the area of assay quality standards as it provides the possibility of an independent, clinically important basis for setting quality specifications. Perhaps one of the most useful ways of looking at biological variation criteria is in providing limits where further improvement is no longer necessary. For example, if an assay meets the optimal standard for monitoring with a CV <0.25 of the within-subject biological variation (e.g. serum transaminases), then the contribution of the assay imprecision to the total result uncertainty is negligible and efforts spent in further reducing assay imprecision are unlikely to be clinically useful. The converse that improvements in assays not meeting biological variation criteria (e.g. serum sodium) will improve clinical decision-making, however, may not necessarily be true, but at the least the assay variation will be a factor in the variability seen in the results.
Low assay bias or imprecision compared with biological variation criteria can be viewed as the highest level in the hierarchy, as further improvements will not change clinical decision-making. The same may not be true for total error limits based on biological variation. For example if an assay has very low imprecision compared with within-subject biological variation, then a significant bias may not be flagged by total error limits applied to the data. There is also the need to consider the quality of the data on which biological variation estimates are made. Does it apply to the analyte concentrations used in the EQA program, does it relate to the time-frames which are relevant for clinical decisions, are important pre-analytical factors such as time of day, fasting status, period of menstrual cycle, sample collection considered? It also needs to be recognised that there will be some analytes where the concept is not readily applicable. Examples include analytes with pulsatile release and exogenous substances like drugs. In some cases with large within subject biological variation (e.g. CRP) it is necessary to apply non-Gaussian statistics to within- and/or between-individual variation, making the process considerably more difficult.
The criteria based on clinician opinion are generally derived from surveys where the effect of different analytical performance is assessed by whether doctors would treat the patient in a different manner. This process has the advantage of linking assay quality to the outcome of clinical decision-making, but there are some difficulties in this approach. For example, there is likely to be a wide variation in clinician opinion and the responses are likely to be strongly affected by the currently available assay quality as this is what the clinicians have been exposed to. Additionally doctors may not be able to separate biological variation from analytical variation in making recommendations. An example of this type of data collection for glucose and HbA1c is seen in the work of Sverre Sandberg and others.10
Level III. Published professional recommendations:
from national and international expert bodies
from expert local groups or individuals
Published professional recommendations on analytical quality provide an important source of information for setting EQA quality standards. They should reflect a synthesis of available information regarding clinical use and analytical standards and result from an interaction between clinical experts and laboratory experts. There may be limitations if the consensus group does not include laboratory experts as it is important to ensure a full understanding of any recommendations. For example, are they related to within or between laboratory performance, to imprecision, bias or total error, and are they minimal or optimal standards? A well known example of expert guidelines are the lipid standards from the Centers for Disease Control and Prevention (CDC) in the USA.11
Level IV. Performance goals set by:
regulatory bodies
organisers of EQA schemes
Examples of regulatory standards are CLIA and RiliBAK as described above. EQA organisers may choose to reflect relevant regulatory standards in their own quality standards as a way of assisting laboratories to meet their regulatory requirements. Certainly setting EQA limits wider than regulatory limits is unlikely to be of benefit to clients. At the least EQA organisers need to be aware of any regulatory standards when setting EQA quality standards.
Level V. Goals based on the current state of the art:
as demonstrated by data from EQA or Proficiency Testing schemes
as found in current publications on methodology.
EQA programs are well placed to apply this type of criterion as they generate data which is directly relevant for the methods and materials used in their programs. Interpretation of the data can be done in a number of ways. The most objective is a strict statistical approach, with or without an outlier exclusion process. For this a numerical value needs to be applied, e.g. should 80, 95 or 99% of laboratories be used to define the standard. Alternatively the standard could be set such that only laboratories clearly outside the distribution of the majority of laboratories are flagged. The state of the art information can also provide information on the number of laboratories which would be flagged outside any proposed quality standards. A high flagging rate may be considered acceptable for aspirational standards, if improvement is both possible end desirable, but inadvisable for regulatory quality standards.
Further Comments on the Stockholm Hierarchy
While the Stockholm consensus statement appears to provide a clear rationale for the setting of quality standards for EQA providers, a closer inspection reveals a number of obvious limitations.
Firstly there is interlinking between the different levels of the hierarchy. For example, application of level IIb, standards based on clinician opinion, is necessarily limited by level V, state of the art, as clinicians have not had access to results better than those currently available in routine laboratories. Similarly levels III and IV, a range of expert opinions, will be made by individuals aware of the limitations of currently available techniques.
Secondly there is the need to be aware of the rationale for the quality standards as described above. A regulatory standard is likely to be set lower than state of the art in order to avoid major problems with the provision of laboratory services. By contrast, if the purpose of the quality standard is to provide an optimal goal with the aim of improving laboratory performance, any limitation of state of the art is immediately apparent and can be addressed.
Once a level of the hierarchy is selected for application there remain further decisions to make. Firstly, there is the identification and assessment of the relevant data. For example if a biological variation criterion is to be applied, it is necessary to ensure that the data are the best available and most relevant. While much accumulated information is available in the database prepared by Carmen Ricos and colleagues,12 each entry is a synthesis of one or more studies and a more detailed evaluation of the original data may be required.
Following the selection of a level within the hierarchy, there are still options to be considered for implementation. This is clearly demonstrated for biological variation where analytical criteria have been set for standards described as minimal, desirable and optimal.13 Similar decisions are required for application of a state of the art criterion as described above.The hierarchy also contains an apparent internal contradiction. If the scientific organisers of an EQA program select a quality standard based on, for example biological variation (level II), then as this is selected by an “expert panel”, this decision can also be described as level IV.
For the implementation of clinically-based quality standards the use of the assay result should also be considered. In general the two main uses for numerical laboratory data are diagnosis and monitoring. These different applications lead to different quality requirements for the assay results. For the purpose of diagnosis, the assay must have a sufficiently low total error (bias and precision) that only a small number of patients may be wrongly diagnosed when the results are compared with a decision point. By contrast for monitoring the requirement is for a low imprecision within the time-frame of the monitoring process to avoid errors of interpretation, either falsely classifying a patient as changing condition, or missing a significant change. Quality standards for these criteria have been developed based on within- and between-person biological variation.13 As the quality standards for monitoring purposes are generally tighter than for diagnosis, if a test has a total error within the required standards for monitoring, then it would usually be considered suitable for diagnosis.
A Process for Setting EQA Quality Standards
Applying the hierarchy to EQA quality standards is an activity which requires clarification of a number of factors as follows:
Assemble personnel with knowledge of laboratory and clinical issues related to the tests involved.
Clearly define the purpose for the quality standard(s).
Agree on the principles to consider and how to apply the different levels of the hierarchy.
Gather information relevant to the different levels of the hierarchy. This would include clinical studies, biological variation data, expert or consensus opinions and current analytical performances.
Clearly document the information used to set the quality standards.
Record the decision and the rationale to allow later review if needed.
This process should allow the setting of documented quality standards where the data, rationale and participants are recorded providing a solid base for the limits used to assess QAP performance.
Different Criteria for Different Analytes
It is an attractive proposition to apply the same quality standard criteria to all analytes in a program. However previous work has shown that this is not possible.14 Where current analytical performance is at a high standard, for example at a level where patients can be monitored using results from different laboratories, it would seem appropriate to set quality standards that identify failures to meet this use even if this standard (monitoring) cannot be met for all analytes. If a lower standard, e.g. for diagnosis, was applied, then a laboratory may not be alerted to a failure that affects monitoring and patient care if the assay is being used for that purpose. It can also be argued that if an analyte reaches a monitoring standard, further work to improve the assay performance is unnecessary and efforts in laboratory quality should be spent in other areas. By contrast, an assay which meets a diagnostic or state of the art criterion may benefit from analytical improvements. Thus if different criteria for quality standards are applied to different analytes, it is necessary to understand which criterion is being applied so as to be aware of the need for possible improvements.
RCPA QAP Chemical Pathology
Over the last three years, the majority of chemical pathology programs in RCPAQAP have undergone a revision of their allowable limits of performance. Examples of the revised limits are shown in Table 2 taken from the General Serum Chemistry program. These revisions have been done by applying the concepts described above. The supporting information in the RCPAQAP circulars provides both the basis for the limits (i.e. either precision or total error) and the standard (optimal, desirable or minimal). This supporting information allows users to identify whether data from assays meeting the quality standard can be used to monitor patients within or between laboratories, or share reference intervals. The detailed information recorded by the working parties performing the revision allows for reassessment in the light of additional information as it becomes available. Other EQA providers have also applied biological variability to determine acceptability limits although this has tended to be limited to a subset of analytes or a specific project. 3,15,16
Table 2.
Allowable Limits of Performance in the RCPAQAP General Serum Chemistry programs.*
| Analyte | Unit | +/− | To | Then | Basis of Limits |
|---|---|---|---|---|---|
| ALT | U/L | 5 | 40 | 12% | Optimal Imprecision |
| Albumin | g/L | 2 | 33 | 6% | Desirable Total Error |
| Alk Phos | U/L | 15 | 125 | 12% | Desirable Total Error |
| Amylase | U/L | 10 | 100 | 10% | Desirable Imprecision |
| AST | U/L | 5 | 40 | 12% | Desirable Imprecision |
| Bicarbonate | mmol/L | 2 | 20 | 10% | Minimal Total Error |
| Bilirubin | μmol/L | 3 | 25 | 12% | Optimal Imprecision |
| Conj. Bili. | μmol/L | 3 | 15 | 20% | Optimal Imprecision |
| Calcium | mmol/L | 0.10 | 2.50 | 4% | Minimal Imprecision |
| Chloride | mmol/L | 3 | 100 | 3% | Minimal Total Error |
| Cholesterol | mmol/L | 0.3 | 5 | 6% | Desirable Imprecision |
| CKMB | U/L | 3 | 15 | 20% | Desirable Imprecision |
| CK | U/L | 15 | 25 | 12% | Optimal Imprecision |
| Creatinine | μmol/L | 8 | 100 | 8% | Minimal Imprecision |
| Ferritin | μg/L | 4 | 27 | 15% | Desirable Imprecision |
| Fructosamine | μmol/L | 15 | 250 | 6% | Minimal Imprecision |
| GGT | U/L | 5 | 40 | 12% | Desirable Imprecision |
| Glucose | mmol/L | 0.4 | 5 | 8% | Desirable Imprecision |
| HDLC | mmol/L | 0.1 | 0.8 | 12% | Minimal Imprecision |
| Iron | μmol/L | 3 | 25 | 12% | Optimal Imprecision |
| TIBC | μmol/L | 4 | 50 | 8% | Minimal Total Error |
| Lactate | mmol/L | 0.5 | 4 | 12% | Optimal Imprecision |
| LD | U/L | 20 | 250 | 8% | Desirable Imprecision |
| Lipase | U/L | 12 | 60 | 20% | Desirable Imprecision |
| Magnesium | mmol/L | 0.1 | 1.25 | 8% | Minimal Total Error |
| Osmolality | mOsm/L | 8 | 266 | 3% | Minimal Total Error |
| Phosphate | mmol/L | 0.06 | 0.75 | 8% | Desirable Imprecision |
| Potassium | mmol/L | 0.2 | 4.0 | 5% | Desirable Imprecision |
| Protein | g/L | 3 | 60 | 5% | Desirable Total Error |
| Sodium | mmol/L | 3 | 150 | 2% | Minimal Total Error |
| Transferrin | g/L | 0.2 | 2.5 | 8% | Minimal Total Error |
| Triglyceride | mmol/L | 0.2 | 1.6 | 12% | Optimal Imprecision |
| Urate | mmol/L | 0.03 | 0.38 | 8% | Desirable Imprecision |
| Urea | mmol/L | 0.5 | 4 | 12% | Desirable Imprecision |
| Cortisol | nmol/L | 30 | 150 | 15% | Optimal Imprecision |
| Free T4 | pmol/L | 1.5 | 12 | 12% | Desirable Total Error |
| Free T3 | pmol/L | 0.7 | 3.5 | 20% | Desirable Total Error |
| Total T4 | μmol/L | 12 | 120 | 10% | Desirable Total Error |
| Total T3 | μmol/L | 0.2 | 1.3 | 15% | Desirable Total Error |
| TSH | mIU/L | 0.1 | 0.5 | 20% | Desirable Imprecision |
For each analyte, the allowable limit has a fixed deviation (+/−) from the target or consensus value up to a particular value (To) and a proportional deviation (Then) at higher values. The basis for the choice of limits is given (Basis of Limits).
Footnotes
Competing Interests: None declared (Ken Sikaris). Janice Gill receives a salary from RCPA Quality Assurance Programs Pty Ltd. Graham Jones has received research support from Roche, and travel support from Roche, Bio-Rad and Abbott.
References
- 1.Panteghini M. Implementation of standardization in clinical practice: not always an easy task. Clin Chem Lab Med. 2012;50:1237–41. doi: 10.1515/cclm.2011.791. [DOI] [PubMed] [Google Scholar]
- 2.Bowyer RC, Geary TD, Penberthy LA, Thomas DW. An Australian quality control scheme for clinical chemistry. The Clinical Biochemist Newsletter. 1981 Mar; [Google Scholar]
- 3.Ricos C, Baadenhuijsen H, Libeer CJ, Hyltoft Petersen P, Stockl D, Thienpont L, et al. External quality assessment: currently used criteria for evaluating performance in European countries, and criteria for future harmonization. Eur J Clin Chem Clin Biochem. 1996;34:159–65. [PubMed] [Google Scholar]
- 4.Clinical Laboratory Improvement Act 2004 Part 493 Laboratory Requirements. Subpart I--Proficiency Testing Programs for Nonwaived Testing. http://wwwn.cdc.gov/clia/regs/subpart_i.aspx#493.929 (Accessed 17 September 2012).
- 5.RiliBÄK (Richtlinien der Bundesärztekammer) Richtlinie der Bundesärztekammer zur Qualitäts-sicherung laboratoriumsmedizinischer Unter-suchungen. http://www.bundesaerztekammer.de/downloads/RiliBAEKLabor201205.pdf (Accessed 17 September 2012).
- 6.Panteghini M, Pagani F, Yeo KT, Apple FS, Christenson RH, Dati F, et al. Evaluation of imprecision for cardiac troponin assays at low-range concentrations. Clin Chem. 2004;50:327–32. doi: 10.1373/clinchem.2003.026815. [DOI] [PubMed] [Google Scholar]
- 7.Miller WG, Jones GRD, Horowitz GL, Weycamp C. Proficiency Testing/External Quality Assessment: Current Challenges and Future Directions. Clin Chem. 2011;57:1670–80. doi: 10.1373/clinchem.2011.168641. [DOI] [PubMed] [Google Scholar]
- 8.Fraser CG, Kallner A, Kenny D, Hyltoft-Petersen P. Introduction: strategies to set global analytical quality specifications in laboratory medicine. Scand J Clin Lab Invest. 1999;59:477–8. [PubMed] [Google Scholar]
- 9.The Diabetes Control and Complications Trial Research Group The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N Engl J Med. 1993;329:977–86. doi: 10.1056/NEJM199309303291401. [DOI] [PubMed] [Google Scholar]
- 10.Skeie S, Perich C, Ricos C, Araczki A, Horvath AR, Oosterhuis WP, et al. Postanalytical external quality assessment of blood glucose and hemoglobin A1c: an international survey. Clin Chem. 2005;51:1145–53. doi: 10.1373/clinchem.2005.048488. [DOI] [PubMed] [Google Scholar]
- 11.CDC Cholesterol Reference Method Laboratory Network Clinical Laboratory Certification Program. http://www.cdc.gov/labstandards/crmln_clinical.html#Protocol (Accessed 17 September 2012).
- 12.Ricos C. Westgard QC: Desirable Specifications for Total Error, Imprecision, and Bias, derived from intra- and inter-individual biologic variation. http://www.westgard.com/biodatabase1.htm (Accessed 17 September 2012). [Google Scholar]
- 13.Fraser CG. Biological variation: from principles to practice. Washington, DC: AACC Press; 2001. [Google Scholar]
- 14.Sciacovelli L, Zardo L, Secchiero S, Plebani M. Quality specifications in EQA schemes: from theory to practice. Clin Chim Acta. 2004;346:87–97. doi: 10.1016/j.cccn.2004.02.037. [DOI] [PubMed] [Google Scholar]
- 15.Sciacovelli L, Secchiero S, Zardo L, Plebani M. External Quality Assessment Schemes: need for recognised requirements. Clin Chim Acta. 2001;309:183–99. doi: 10.1016/s0009-8981(01)00521-6. [DOI] [PubMed] [Google Scholar]
- 16.Scherrer F, Boisson RC, Cartier R, Cohen R, Eynard JC, Poggi B, et al. About the choice of acceptability limits in external quality assessment schemes. Ann Biol Clin. 2007;65:677–84. [PubMed] [Google Scholar]