External Quality Assessment in Croatia: problems, challenges, and specific circumstances

Abstract

In Croatian medical laboratories (ML), external quality assessment (EQA) has a long tradition of almost half of a century. At national level, EQA is provided by Croatian Centre for Quality Assessment in Laboratory Medicine (CROQALM) which is a part of Croatian Society of Medical Biochemistry and Laboratory Medicine.
This case study aimed to summarize the main challenges, which are set to CROQALM and their possible solutions.
CROQALM has 10 schemes, covering majority of analysis for which medical biochemistry laboratories in Croatia are authorized for, including pre-analytical and post-analytical phase of laboratory work. Assessment scheme has three exercises per year. One sample per scheme and exercise is distributed to participants depending on their application. All data transfer and evaluation of the results are done using web interface and statistical software for evaluation of quality in laboratory medicine.
Since CROQALM has relatively small number of participating laboratories (N = 197) with lot of different manufacturers of instruments used for analysis in all schemes, constant challenges are present in the evaluation of the results (commutability problems, statistical analysis etc.). Further, number of participating medical laboratories is even lower for highly specific parameters, which are in the scope of clinical laboratories only.
Despite the obstacles we are faced to, EQA at national level is useful tool regarding standardization and harmonization aspects in total testing process within the country. Furthermore, it gives participating laboratories recognition and proof for meeting expected quality criteria in the community they serve.

Introduction

External quality assessment (EQA) is a widely accepted tool for monitoring and improving method performance in the medical laboratory (ML).

Croatian Centre for Quality Assessment in Laboratory Medicine (CROQALM) is a part of the Croatian Society of Medical Biochemistry and Laboratory Medicine (CSMBLM) and serves as a quality evaluation scheme in laboratory medicine at national level. Participation in CROQALM is mandatory for every ML in Croatia.

The schemes contained in CROQALM are designed to cover majority of analysis preformed in Croatian laboratories and are presented in Table 1. The control samples are distributed one to three times per year, depending on the participants’ application and the scheme schedule. In each scheduled exercise, one control sample per scheme (with instruction for sample preparation and measurement conditions) is sent to every laboratory requiring single measurement of analyte. Variable concentration levels are distributed that cover both normal and pathological ranges where available. At the end of each exercise, statistical evaluation of test results is done using inlab2*QALM software for quality evaluation in laboratory medicine (IN2 Group Ltd., Zagreb, Croatia), for each peer group consisting of seven or more participants. After outlier removal using Tukey model (1), mean, standard deviation and coefficient of variation is calculated for each peer group. The data are further evaluated according to predefined allowable limits of performance and z-scores with graphical presentation and main histogram for every analyte (Figure 1). Each laboratory receives report showing calculated percentage deviation of the result from corresponding peer group mean and z-score in absolute number showing positioning of the individual result regarding target value and standard deviation after outlier exclusion. Also, scheme coordinator comments performance characteristics for a given EQA exercise as a part of summary report. The annual report includes the Certificate of participation with an indication of the scheme in which they participated. The role of CROQALM is to implement and provide an assessment tool for laboratory performance, while the corrective actions upon ‘flagged’ results are responsibility of each individual laboratory. Evaluation of laboratory performance in Croatia is within the competence of CCMB and its professional supervision.

Table 1. External quality assessment scheme in Croatia (CROQALM): an overview.

Scheme	Name of the scheme	Number of exercises per year	Type of control sample	Number of participants (2015)
I	Biochemistry parameters	3	Human-based, lyophilised control sample	180
II	Specific proteins, CRP	3	Human-based lyophilised or liquid control sample	168
III	Laboratory haematology	3	Stabilized donor blood and human-based whole blood sample	183
IV	Laboratory coagulation, Coagulation factors	3 1	Lyophilised control sample	164
V	Drugs	1	Lyophilised control sample	17
VI	Urinalysis: urine test strip, sediment	3 1	Human-based, liquid control sample	178
VII	Analysis of pH, blood gases and electrolytes	3	Liquid control sample	41
VIIA	Point-of-care: pH, blood gases, electrolytes, glucose and lactate	3	Liquid control sample	19
VIII	Hormones, vitamins, tumor and cardiac markers	2	Human-based, lyophilised control sample	94
IX	Glycosylated haemoglobin A1c (HbA1c)	3	Human-based lyophilised control sample	151
X	Pre-analytical phase of laboratory testing	3	Questionnaire design	168
XI	Post-analytical phase of laboratory testing	3	Questionnaire design	160
XII	Sweat chloride test	3	In-house liquid control sample	8

Figure 1.

Example of statistical evaluation of results for alanine-aminotransferase.
Graphs: histogram of all results (main graph), positioning of results in allowable grey areas according to z-score (top right) and allowable limits of performance (down right).
Statistical evaluation: all method group (number of participants, mean, SD, CV) – first line, method-based peer groups (number of participants, mean, SD, CV and laboratory’s result).
Following lines: overview of deviations according to z-score and according to allowable limits of performance.
CV – coefficient of variation. SD – standard deviation.

Number of participants varies slightly from one exercise to the other depending on participants’ application and scheme distribution schedule. The most of the ML in the CROQALM scheme are part of primary health care and thus most results are received for routine haematology and biochemistry analytes. In 2015, number of registered ML that participated in CROQALM was 197. Number of participants is different between schemes, from 8 in sweat chloride test scheme to 183 in laboratory haematology scheme.

CROQALM scheme is an operational tool for medical laboratories in evaluating assay performance and thus improving analytical phase of laboratory work. It also encourages all attempts in setting quality standards for extra analytical phase because pre- and post-analytical phase of the laboratory testing process are major sources of errors for total laboratory process. For this reason, in 2014, two additional schemes were introduced: pre-analytical and post-analytical phase schemes which are not mandatory and serve only for educational purposes. Pre-analytical phase scheme for CROQALM is implemented in cooperation with the Working Group for pre-analytics of CSMBLM using the circulating questionnaire, one of three method types suggested by Kristensen et al. (2), same as post-analytical phase scheme where questions are categorized into four groups according to quality indicators from ‘Model of quality indicators’ proposed by International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) and Working Group ‘Laboratory Errors and Patient Safety’ (3). Results and final reports are presented like absolute numbers and percentages for each answer of questionnaire with coordinators’ comments for these two schemes.

Depending on sample characteristics, target value assignment by reference method and replicate samples in each survey, EQA schemes can be classified into six categories. Category 1 EQA scheme is the most challenging one, requiring distribution of commutable, replicate samples with values assigned by reference method. Consequently, it offers evaluation of bias and reproducibility for an individual laboratory and verification of degree of standardisation and harmonisation of the measurements procedures used (4). Although Croatian EQA scheme aims towards category 1 scheme where possible, from which participants would get the most information and benefit, several issues have to be resolved. They include commutability of distributed control samples, accuracy evaluation through target value assignment by reference method, peer group formation and harmonisation of laboratory results on national level.

The aim of this case report is summary of CROQALM main challenges and activities undertaken in solving main problems.

Sample characteristics: commutability and target values

Reaching commutable samples that would behave in the same manner as patient sample across a variety of methods and instruments used in the scheme is a major prerequisite for both standardisation and harmonisation (4, 5). The issue was of utmost importance in haematology scheme where different analysers use technologies that yield different results for white blood cell count (WBC) and/or their subpopulations and the results are often flagged. After exploring several different commercial samples designed as haematology control samples, it was decided that use of minimally preserved fresh blood samples from blood donors would give the best baseline sample characteristics and mimic actual patient samples. Aliquots of such fresh sample from blood donor were distributed in two exercises in 2015 and gave us the opportunity to evaluate both laboratories’ performances and harmonisation issues. Possible shortcoming for this kind of sample is analytical range that is restricted to values mostly falling within the reference interval.

Although standardisation process is an on-going process that involves both manufacturers and laboratories, the role of an EQA scheme should be to verify the correct implementation of traceability where reference system is available (5, 6). In order to monitor and support all initiatives to harmonize measurement procedure results, distribution of commutable samples in the EQA scheme is required. Commutable control material shows the same mathematical relationship between the results of different measurement procedures as actual patient sample that contained the same analyte concentration. It enables assessing both method and individual laboratory performance evaluation to reference measurement procedure. General biochemistry scheme offers such possibilities since target values determined by reference method are available for many analytes enabling verification of accuracy. Nevertheless, laboratory evaluation is still restricted to peer group consensus value given that the commutability of such material has not been verified yet. Obvious limitation of such approach is recognised and was set as primary challenge to overcome.

The preliminary studies have been started by CROQALM to verify commutability of commercial control material comparing the results of currently used control sample with results obtained from patient serum. The serum is prepared as ‘off the clot’ pooled serum from two donors and fresh aliquots were shipped to participants. All samples were distributed within two days of shipment and participants were instructed to analyse the samples (both control sample and serum sample) on the same run on the instrument. The analytes and instruments used were all part of general biochemistry scheme. Such sample may be considered as native spy-sample used with other EQA providers to monitor commutability of distributed control material (7). Analysing the results of preliminary studies offers the possibility to address both commutability and harmonisation issues in the scheme.

Target value assessing by reference method is another, rather expensive challenge for the small scheme. Obtaining such values from accredited reference laboratory listed in Joint Committee for Traceability in Laboratory Medicine database (8) can be achieved by sending substantial amount of control material to the reference laboratory after assessing its commutability. More practical solution may be collaboration with other EQA providers and reference laboratories in reducing costs by sharing samples with target values determined in reference laboratories (6).

Peer group formation

Peer group consensus value is the basis for laboratories evaluation throughout whole Croatian EQA scheme. Peer group usually consists of laboratories performing the analysis on same instrument, since it can be expected that each instrument will have the same matrix-related bias for a given sample. Besides these matrix-related differences, instrument-based peer groups seem like the only appropriate evaluation of numerous analytes (hormones, tumour and cardiac markers) in immunoassay schemes where lack of standardisation and/or harmonisation yields different results among participating laboratories that use similar principles, yet slightly different methodologies.

Instrument-based peer groups were not the main choice in general biochemistry scheme although it would seem like a reasonable choice when commutability hasn’t been verified. One of the reasons was an attempt to obtain satisfactory statistical evaluation over many instruments used, with many homogeneous instrument groups having less than 10 participants. Besides statistical reasons, many laboratories (up to 20%) use heterogeneous (“open”) systems regarding instrument and reagent manufacturer as well as calibrators, which makes it difficult to decide on appropriate peer (instrument, reagent or calibrator). Nevertheless, the groups were further split to instrument-based subgroups when substantial difference among instruments used was observed (i.e. alkaline phosphatase – IFCC method group, cholinesterase – butyryltiocholine method group). This approach needs constant monitoring and evaluation for upcoming changes regarding group homogeneity. Method-based peer group evaluation is also the current choice for urinalysis (urine test strips and sediment) scheme. The reasons are numerous combinations and different urine test strips and instruments used (resulting in large number of small and heterogeneous peer groups), as well as many laboratories still using visual evaluation of colour-changes observed. The results received for a given control sample are evaluated by CROQALM team using several criteria: package insert assigned values (available from control sample manufacturer), percentage of laboratories reporting the same result and medical decision criteria to which this result can be applied. All of the above criteria are objects for professional opinion from EQA scheme coordinators which results in a consensus acceptable performance range for every parameter.

Allowable limits of performance

Although different models in defining analytical quality specification criteria and the hierarchy of these principles are defined (9), the acceptable limits vary widely among different EQA schemes. Because of the limited data for specifications based on clinical outcome (10), limits based on components of biological variation and current ‘state-of-the-art’ in analytical measurements (models 2 and 3) are currently applied in CROQALM scheme. Allowable limits of performance for the analytes that represent most commonly requested tests in laboratories are documented and well known to participants (11), but the rationale of their choice must be revised and clear to participants. Since individual result is evaluated as the difference from the target value, both bias and imprecision is assessed in one total error of measurement. However, expert opinion regarding the basis for choice between desirable, optimal or minimum specifications derived from intra- and inter-individual biological variation must be explained for participants. For example, the analyte that is used in both monitoring and diagnostic purpose might require limits that are approaching optimal specifications (as it is the case for glucose, where allowable percentage deviation from target value is set to ± 5%), taking into consideration current performances within the EQA scheme, statistical limitations (number of participants in the corresponding peer group), current state-of-the-art etc. On the other hand, if technical or analytical limitations do not allow specifications based on biological variation data to be applied (as for sodium or chloride in general biochemistry scheme) or statistical reasons (small or heterogeneous peer groups) are the reasons to adopt less stringent, state-of-the-art criteria (fibrinogen, antithrombin in coagulation scheme) such particularities must also be explained. It is also mandatory that criteria currently used is revised (especially for the state-of-the-art criteria) and time period for monitoring any upcoming changes in performance characteristics defined.

Currently, all allowable limits of performance are under revision from scheme’s coordinators and their working groups. The hierarchy of acceptable limits, as described in Sandberg et al. (9), must be followed and basis for every limit must be documented and shared with participants. When choosing analytical specifications deriving from biological variation data, one must take into consideration the clinical ‘purpose’ of the analyte (monitoring, diagnosis, clinical classification of patient, point-of-care analysis) and current performance in EQA scheme in order to set more stringent, yet achievable and encouraging allowable limits of performance. Revised limits must be described and the time-period for their re-evaluation defined. Also, dual set of criteria in statistical evaluation of data depending on the concentration of the analyte is under consideration. It is our opinion that having fixed, absolute allowable deviations from target value at low concentrations and proportional deviation limit at higher values would be more appropriate when dealing with small numbers and still ‘fit-for-purpose’. Regardless of rationale used in defining analytical specifications, the basis for their choice must be explained and available to participants (12, 13).

Conclusion

Considering total number of laboratories participating in CROQALM, the scope of analysis and variety of analytical systems within every scheme, it might be thought that the laboratories would benefit more by participating in an internationally organised EQA scheme. Regardless of these obstacles, nationally organised EQA schemes are important for laboratory comparison in addressing standardisation and harmonisation of total laboratory process and give participating laboratories recognition and a proof of meeting expected performance criteria within clinical community they serve.