Implementation of External Quality Assessment Scheme in Clinical Chemistry for District Laboratories in Bhutan

Abstract

External Quality Assessment Scheme (EQAS) involves evaluation of a number of laboratories by an outside agency on the performance of a number of laboratories based on their analytical performance of tests on samples supplied by the external agency. In developing countries, establishment of national EQAS by preparing homemade quality control material is a useful scheme in terms of resources and time to monitor the laboratory performance. The objective of this study is to implement an EQAS to monitor the analytical performance of the district laboratories in Bhutan. Baseline information was collected through questionnaires. Lyophilized human serum including normal and abnormal levels were prepared and distributed to 19 participating laboratories. Nine routine analytes were included for the study. Their results were evaluated using Variance index scores (VIS) and Coefficient of variations (CV) was compared with Clinical Laboratory Improvement Act (CLIA) Proficiency Testing Criteria (PT) for each analyte. There was significant decrease in CV at the end of the study. The percentages of results in acceptable VIS as ‘A’ were 63, 60, 66, 69, 73 and 74, 75, 76 and 79 % in November 2009–July 2010 respectively. From our results, we concluded that, establishment of EQAS through distribution of home-made quality control material could be the useful scheme to monitor the laboratory performance in clinical chemistry in Bhutan.

Introduction

EQAS involves evaluation of a number of laboratories by an outside agency on the performance of a number of laboratories based on their analytical performance of tests on samples supplied by the external agency [1]. It is part of a quality assurance activity that supports the improvement of health care services. The health status of the population in Bhutan has been markedly improved especially during the last 17 years which covers almost 90 % of the country’s population with basic healthcare service [2]. Quality Assurance and Standardization Division (QASD) under the Ministry of Health which was instituted in 2001 has been striving for the continuous improvement of the quality of health care through various quality assurances activities [3]. However, like many other developing countries, most of the hospitals in Bhutan face various limitations due to which EQAS has not been established for district laboratories in Bhutan for clinical chemistry. Lack of suitable control material is another hurdle in the effort to establish EQAS to monitor the laboratory performance. Laboratory reagents and control materials are commercially imported spending considerable amount of country’s revenue. Such high laboratory expenses could be reduced by preparing homemade serum control material and overall laboratory performance would also be improved through effective performance monitoring by implementing EQAS. Lyophilized human serum control is stable for up to 2 years at 2–8 °C [4] and even longer if stored at −20 °C. Methods of preparing home-made lyophilized serum have been described elsewhere [5, 6]. The data analysis and use of the suitable scoring index is another important phase in the process of EQAS. The scoring systems are used for the judgment of individual laboratory indicators as well as of a number of indicators for the assessment of the overall performance [7]. We used overall coefficient of variation (CV) to indicate the precisions and variance index score (VIS) which shows bias of the performance [7, 8].

Methods

Questionnaire Distribution

Following an approval from ethical committee, a set of questionnaire was distributed to 19 laboratories in Bhutan all performing clinical chemistry tests and asked to return back within a week. All the 19 laboratories returned their questionnaires, but one of the laboratories had to drop after few months of study due to breakdown of the chemistry analyzer. The questionnaire was prepared to ensure adequate and reliable information for the studies from each district laboratories. Information included types of equipments, reagents, methods, QC materials and reconstitution technique used by each laboratory. Other information like qualification of the laboratory staff, EQA activities and laboratory safety management were also included.

Specimen Collection

Following the informed consent, about 350 ml each of fresh whole blood was collected from 10 volunteers using the standard procedure. Serum was separated and about 1 ml of it from each donor was sent to the Centre of Medical laboratory Services, Faculty of Medical Technology, Mahidol University to test for HIV-Ab and HbsAg which both were declared negative.

Preparation of Lyophilized Serum Control

We prepared lyophilized human serum control material following the protocol described in the WHO document/LAB/81.4 [9]. Glucose (Glu), urea, creatinine, aspartate aminotransferase (AST), alanine aminotransferase (ALT), alkaline phosphatase (ALP), total bilirubin (TB), total proteins (TP) and albumin (Alb) in the serum were assayed before level adjustment. Serum was divided into four parts to make 2 parts of normal (N₁ and N₂) and 2 parts of abnormal levels (A₁ and A₂). The 2 ml serum contained in each vial was lyophilized in the freeze drier at −20 °C over 14 h. After lyophilization, vials were immediately capped and sealed. Samples were packed in protective box containing plastic sheets along with report forms and instructions for reconstitution.

Sample Distribution

Total of 620 vials of lyophilized samples were prepared and used for the following purposes: 85 % for analysis by district laboratories in Bhutan; 6 % for three ISO certified laboratories to determine the assigned values; 5 % for homogeneity tests and 6 % for stability tests. Considering the transportation problems in Bhutan, we prepared samples for single mailing sufficient for the entire period of study. They were properly packed, labeled with safety information, storage conditions and address of the hospitals and dispatched by air freight.

Instructions for Participants

Results recording form and package insert with instructions on reconstitution technique was included in the package accompanying the specimens. For identification of the results from different laboratories, each vial was labeled carrying the explicit information of the work. A procedure on sample handling and steps of reconstitution was described in the package insert. Participants were asked to use only volumetric glass pipette for reconstitution to prevent volume errors.

Study Design and Performance Monitoring

The two control samples were tested for 9 analytes on the same day in the first week of the month for 10 consecutive months (November 2009–July 2010). Participants were asked to send the results within a week after analysis. Their results were statistically analyzed and feedbacks were immediately sent through emails with suggestions for improvement for those with extreme outliers.

Statistical Analysis

Homogeneity Testing

Homogeneity test was performed to check for the presence of vial-to-vial variations using the IUPAC Protocol (2006) [10]. 10 % of the control sample for each level were randomly selected and analyzed in duplicates. The results were subjected to Cochran’s t test for outliers and compared with critical values at 95 % confidence interval. No outliers were detected and sampling variances were calculated and compared with critical F value at 95 % confidence interval with the 8 degrees of freedom.

Stability Testing

In order to study the stability of our homemade lyophilized serum, a single level control sample was stored at two different temperatures, 4–8 °C and −20 °C and they were analyzed in 3, 5, 7 and 8 months. Results were statistically analyzed using SPSS program and tested for equality of variance by Levene’s test and t test for equality of the means of the results of samples from these two conditions with 8 degrees of freedom at 95 % confidence interval (Table 2).

Table 2.

Results of the stability tests compared with initial values

Analytes	n	Unit	2–8 °C		−10 to −20 °C		P values
			Mean	SD	Mean	SD
Glu	10	mmol/l	6.05	0.08	6.11	0.06	0.02
Blood urea nitrogen (BUN)	10	mmol/l	6.43	0.21	6.78	0.25	0.19
Creatinine	10	µmol/l	194	4.42	186	8.84	0.43
AST	10	IU/l	20	1.6	19	1.8	0.36
ALT	10	IU/l	54	5.0	60	4.5	0.004
ALP	10	IU/l	287	12.0	290	10	0.58
T. bilirubin	10	µmol/l	25	1.8	27	1.8	0.05
Total protein	10	g/l	78	9.0	77	9.0	0.81
Alb	10	g/l	45	1.0	45	1.0	0.82

Setting Assigned Values

Three samples of each level were analyzed for 3 days in duplicate at the three ISO 15189 certified laboratories in Bangkok in Thailand. The three ISO certified laboratories and the analyzers used by each of them to analyze our samples were as follows: Department of Clinical Chemistry at Siriraj Hospital used Modular P800; Department of Clinical Chemistry at the Faculty of Medical Technology in Mahidol University used Hitachi 917 and Department of Clinical Chemistry at Mongkutwattana General Hospital used Cobas Integra^®400. The results from each of these three laboratories were tested for outliers and the mean was trimmed at ±2 Standard deviation (SD) before used as assigned values.

Performance Evaluation

The overall mean, SD and CV were calculated for both the sample trials after removing the outliers using 2 SD. CV for each analyte is compared with CV from the clinical laboratory improvement act (CLIA) proficiency testing criteria (PT) which is also used in this study as the chosen coefficient of variation (CCV) to calculate VIS. A monthly VIS for of each analyte for 18 laboratories is calculated which is used to evaluate monthly results. VIS is calculated as follows:

where X_lab is the result from the participants, AV the assigned values from ISO 15189 certified laboratories and CCV is the chosen coefficient of variation from CLIA PT criteria. When the (i) VIS is lying between 0 and 50 the performance is graded as excellent, (ii) VIS 51–100 as very good (iii) VIS 101–150 as good performance, (vi) VIS 151–200 as acceptable performance, (v) VIS 200–250 as performance needs improvement (vi) VIS >250 as unacceptable. The maximum VIS is up to 400 and the sign is ignored.

For the assessment of the overall performance in the entire studies, we divided the monthly results into 3 categories: A (VIS ≤200) as acceptable; B (VIS: 201–250) as needs improvement and C (VIS >250) as unacceptable.

Results

There were initially 19 laboratories with chemistry analyzer (Table 1). Each of these laboratory analyzed two samples on the same day for nine analytes in each month for 12 rounds. However, one of the laboratories discontinued its participation due to the breakdown of the semi-automated photometer which left without repairing for the entire period of our study. Out of 19 laboratories, only two of them use fully automated analyzer and rest of the laboratories used two types of semi automated analyzers, Screen Master 3000 and BA88. In the homogeneity tests, the sampling variance showed no significant difference from that of critical f values of inhomogeneity for all the analytes at the 95 % confidence interval with 8 degrees of freedom. All the analytes were stable at 2–8 °C with P values greater than 0.05 except for glucose and ALT which showed significant difference with P value 0.02 and 0.004 respectively (Table 2). The overall CVs were decreased at the end of the nine except total protein and bilirubin as shown in Fig. 1a and b. The gradual increase in number of results with VIS ≤200 was observed. The percentage analyte in acceptable VIS as ‘A’ were 63, 60, 66, 69, 73, 74, 75, 76 and 79 % for November 2009–July 2010 respectively. Unacceptable VIS as ‘C’ were 27, 31, 25, 21, 20, 20, 20 and 14 % in November 2009–July 2010, respectively (shown in Fig. 2).

Table 1.

Number of participant laboratories with clinical chemistry tests at various levels of hospitals in Bhutan

Types of hospitals	Bed size	No of hospitals	With chemistry analyzer
National Referral Hospital	350	1	1
Regional Referral Hospital	150	2	2
District Hospitals	20–40	24	15
Military Hospitals	20–40	2	1
Total number of laboratories with chemistry analyzer			19

Fig. 1.

a Overall CVs in November 2009 compared with CCV from CLIA PT criteria. b Overall CVs in July 2010 compared with CCV from CLIA PT criteria

Fig. 2.

Percentage of tests results graded based on VIS as A (VIS ≤200); B (VIS: 201–250) and C (VIS >250) and interpreted as acceptable, needs improvement and unacceptable performance respectively. The graph shows the months of the study in X axis and the number of test results in percentage in Y axis

Discussion and Conclusion

By keeping in constant touch and providing prompt feedback to our participants with suggestions for necessary actions for the improvement, there has been gradual improvement in the performance. The decrease in overall CV compared with CLIA PT shows that the interlaboratory variation was reduced over the period of study and it is also a reflection of the decrease incidence of random errors. Correspondently, gradual increase in number of results with MVIS ≤200 shows overall improvement of performance on individual analyte. However, the values for TP and TB could not be improved during the entire period of study. Poor performance for TB could be due to photolysis during the storage and following the reconstitution. Performance on TP were initially improved but gradually declined in the last few rounds which could be due to the denaturation of the proteins on prolong storage. Moreover, the performance was inconsistent which could be due to the change in reagents and decrease in number of results being reported. Other causes of inconsistent performance in some analytes could be due to random errors resulted from the volume errors in reconstitution, reagent and sample pipetting. The time and temperature dependent changes in activities of analytes and improper storage of our sample should also be considered. The increase in percentage of tests in VIS ≤200 from 63 % in November 2009 to 79 % in July 2010 respectively showed that the overall performance improvement. The decrease in percentage of tests in VIS >250 from 31 % in December to 14 % July 2010 respectively shows that the incidence of some system and random errors have been reduced through corrective actions. In stability testing, time, temperature and light dependent changes in concentration were observed. There are many examples in the literature describing the instability of the biological compounds in lyophilized and liquid serum stored at various temperatures [11–13]. Our sample did not show significant deterioration when stored at 2–8 °C and −20 °C. There was loss of 5 % glucose when stored at 2–8 °C and 3 % at −20 °C at the end of 9 months. The inconsistent and incomplete results due to erratic supply of reagents, the communication gap with the participants were the major challenge in this study. We concluded that implementation of EQAS through distribution of home-made quality control material could improve the laboratory performance in clinical chemistry in Bhutan which could be the cost effective practice in the future.

The study was sponsored by Royal Government of Bhutan with objective to established laboratory quality assurance system in Bhutan.