Evaluation of Sigma Metrics and Westgard Rule Selection and Implementation of Internal Quality Control in Clinical Chemistry Reference Laboratory, Ethiopian Public Health Institute

Abstract

Clinical laboratories are mandated to deliver accurate, reliable, timely and correctly reported result which, used in decision making for disease screening, diagnosis and monitoring. With aid of six sigma principles and metrics it is possible to assess the quality laboratory process and the quality control that is needed to ensure that the desired quality is achieved. Thus, this study was undertaken to evaluate the performance of biochemical parameters by calculating the sigma metrics of individual parameters using internal quality control (IQC) and Proficiency Testing (PT) results. The sigma metrics of 21 clinical chemistry parameters were calculated from COBAS 6000 analyzer with internal quality control (IQC) materials and external quality assurance scheme (EQAS) performance in national clinical chemistry laboratory for the period of six months. We obtained an excellent performance (≥ 6 sigma) for test parameters amylase pancreatic, amylase total, HDL, magnesium, AST, triglyceride, total bilirubin and ALT in both levels of quality control. Urea, creatinine and chloride were failed to meet the minimal sigma performance for both level 1 and 2. Sigma values of 3–6 were observed for ALP, Direct bilirubin, total protein, albumin, glucose, potassium, and phosphate with both levels of quality control. Though, stringent IQC strategy is not mandatory for analytes that scored sigma value ≥ 6. However, continuous monitoring quality control is required for renal function tests and process improvement will be designed for those with poor sigma values.

Introduction

Clinical Laboratory results have an important role in disease diagnosis, control and prevention program by providing timely data or information for patient management and disease surveillance [1]. Producing reliable, reproducible, accurate, timely and correctly reported results are predominantly important because physicians make their clinical decisions mostly in accordance with laboratory results [2]. In this context, accurate and precise test results are crucial for physicians and their patients to decide in screening, diagnosis and monitoring of diseases [3]. The implementation of quality control (QC) strategies is so important to identify situations (analytical errors) when a measurement procedure may not be providing results that are suitable for use of laboratory results of medical decisions [4, 5] As quality improvement efforts towards quality laboratory services, the International Organization for Standardization (ISO-15189) has also recommended systematic assessment and monitoring of quality management systems (QMS) in laboratory [6].

In QMS implementation practice laboratory technologists are trained to focus on achieving the QC results within defined acceptable limits [7]. Quality control measures employed to assess the analytical phase in clinical chemistry laboratory are internal quality control (IQC) and external quality assurance scheme (EQAS) [8]. IQC is a sample material whose matrix is identical to the patient’s sample and has an established concentration range available in to two or three levels covering the medical decision points [9]. The internal quality control is run daily and results are interpreted by using control charts such as Levy Jennings’s and application of the standard Westgard rules [10]. IQC ensures a continuous watch of the analytical system, so as to check whether the results are reliable enough to be released [11]. While external quality control involves analyzing and reporting of known control samples supplied by an external agency at a predefined time interval which in our country experience every three months. External quality control is interpreted by either Z-score or standard deviation index. A Z-score is a calculated value that tells us, as to how many standard deviations, a control result has shifted from the mean value which is expected for that material [12].

These internal QC and/or external QC couldn’t tell us the exact number of defects or errors done by the laboratory [13]. However, the exact number of errors done by the laboratory in the analytical phase cannot be assessed by running internal and external QCs [10]. The error of measurement procedures in the context of statistical QC has typically considered as made up of two components: constant error (bias) and random error (imprecision) [14]. These measurement tools can be comparable with stable analytical performance to the clinical based goals for the estimation of total analytical errors to the quality requirement [15–17]. The index number which detects total analytical errors done by the laboratory can be quantified by employing sigma metrics in the laboratory. The Sigma scale provides guidelines for assay improvement and monitoring [4]. The sigma metrics is expressed numerically and is inversely proportional related to the risk of failure of measurement procedure [18]. Six Sigma is one of the popular quality management system tools usually applied when the outcome of the process can be measured and, employed for process improvement [19].

Sigma values can also be correlated with Westgard Sigma Rules to set the QC procedures as shown in (Table 5). Five-sigma and 6-sigma methods only require a simple QC rule to monitor the method with fewer controls per run. Three-sigma and 4-sigma methods require multiple QC rules to monitor the method with a higher number of controls per run. Methods with a sigma metric of less than 3 are difficult to monitor even with multiple QC rules and many controls per run; these methods should be avoided [20, 21]

Table 5.

Quality control westgard rules used depending sigma metric value of analytes

Sigma metric	QC rules used	QC frequency	Number of runs
6 Sigma	1–3.5 s	One level	Once daily
5 Sigma	1–3 s	Two levels	Twice daily
4 Sigma	1–3 s, R4s, 2 of 2–2 s, and 2 of 3–2 s	Two levels	Twice daily
3 Sigma	1–3 s, R4s, 2 of 2–2 s, 2 of 3–2 s, 4–1 s and 12x	RCA and method performance is required before releasing the result

RCA, Root cause analysis, QC, Quality control

Thus, it is possible to assess the quality of laboratory testing processes and the number of QC that is needed to ensure that the desired quality is achieved with the help of Six Sigma principles and metrics. This study was done to assess the performance of individual biochemical parameters on a Sigma Scale by calculating the sigma metrics for individual parameters and to follow the Westgard guidelines for appropriate Westgard rules and levels of IQC that needs to be processed to improve target analyte performance based on the sigma metrics.

Material and Methods

This is a retrospective study which aimed to calculate the sigma metrics of Ethiopian Public Health Institute (EPHI) clinical chemistry reference laboratory. The QC data was taken from COBAS 6000 (German, Japan Cobas 6000 series of Roche) instrument. The control materials were obtained from Roche itself in lyophilized form and prepared in the laboratory. Both normal/physiological and abnormal/pathological levels of QC were analyzed daily before patient sample have been analyzed and the instrument was calibrated regularly. The QC results were interpreted by using westgard rules of 3SD. Data required for this study were extracted between January 2019 and June 2019.

The parameters assessed for sigma calculations were Amylase-P, Amylase total, AST, ALT, ALP, Total CHOL, Triglyceride, HDL, Creatinine, Urea, Total Protein, Albumin, Glucose, sodium, potassium, chloride, phosphate, Bilirubin direct. Bilirubin total and magnesium. Sigma matrices were calculated using total allowable error, CV% and percent bias for the above-mentioned parameters.

The sigma values for these parameters are calculated with the formula;

$$\text{Sigma Metrics}\left(\sigma \right)=\frac{\left[\text{TEA}\%-\text{Bias}\%\right]}{\text{CV}\%}$$

where in this formula;

Total Allowable Error

It is the total allowable difference from accepted reference value seen in the deviation of single measurement from the target value. TEa values of various parameters were taken from Clinical Laboratories Improvement Act (CLIA) guidelines [22, 23].

Bias

Bias is the systematic difference between the expected results obtained by the laboratory’s test method and the results that would be obtained from an accepted reference method. Bias percentage was derived as follows;

$$\text{Bias}\%=\frac{\left[\text{Our EQA result}-\text{Peer Group Mean}\right]}{\text{Peer Group Mean}}\ast 100$$

Coefficient of Variation

The CV is standard deviation (SD) expressed as a percentage and is a measure of the variability of an assay and is expressed as a percentage. CV was calculated from Roche internal QC for the parameters.

$$\text{CV}\%=\frac{\text{Standard Deviation}\left(\text{SD}\right)}{\text{Mean}}\ast 100$$

Quality Goal Index Ratio (QGI)

The quality goal index (QGI) ratio characterized by the relative extent to which both bias and precision meet their respective quality goals. This was used to analyze the reason for the lower sigma in analytes, i.e., the problem is due to imprecision or inaccuracy or both. The QGI ratio was calculated using the following formula, [23]

$$\text{OGI}=\text{Bias}\%/1.5\ast \text{CV}\%$$

The criteria for interpreting QGI ratio when the six sigma value of test parameters were low were as follows (Table 1).

Table 1.

Criteria for interpreting quality goal index ratio

QGI	Problem
< 0.8	Imprecision
0.8–1.2	Imprecision and inaccuracy
> 1.2	Inaccuracy

QGI, Quality goal index ratio

Results

Table 2 and 3 summarizes the CV% calculated from level 1 and level 2 internal quality data extracted between January 2019 and June 2019. Mean of laboratory and peer group results were taken from reports of external quality assurance scheme over the study period. Bias was calculated from the percentage difference of mean of the peer group and the laboratory values. The sigma metrics for level indicated that 12 (57%) of 21 individual parameters were failed to meet six sigma quality performance. Of these 4 (19%) (sodium, chloride, creatinine and urea with level 1 and Chloride, Creatinine, Total Cholesterol and urea with level 2) of 21 parameters failed to meet the minimum sigma quality performance with metrics of less than three and, another 8 (38%) just meet minimal acceptance performance with sigma metrics between three and six (Fig. 1). We obtained an excellent performance (≥ 6 sigma) for test parameters amylase pancreatic, amylase total, HDL, magnesium, AST, triglyceride, total bilirubin and ALT in both levels of quality control (Fig. 2 & 3). The QGI ratio (Table 4) for parameters failed to meet the minimal sigma quality performance showed imprecision in case of sodium with level 1 and chloride with level 2, inaccuracy in the case of creatnine and urea with both level 1 & 2 and, cholesterol with level 2.

Table 2.

Table showing the average calculated chemistry test parameters bias %, TEa %, CV% and sigma values of normal (level 1) quality control for a period 6 months in EPHI chemistry laboratory

Analyte	Tea% from CLIA	CV%	Our lab EQA mean result	Peer group mean	Bias%	Sigma (σ)
Amylase p	30	1.28	87.6	81.8	7.090465	17.90
Amylase- total	30	1.68	137.1	135.4	1.255539	17.11
ALT	20	3.26	159.7	161.3	−0.99194	6.44
AST	20	1.98	212.4	211.6	0.378072	9.91
ALP	30	5.21	179.2	180.1	−0.49972	5.85
Direct bilirubin	20	3.654	1.297	1.293	0.309358	5.39
Total bilirubin	20	2.941	2.71	2.765	−1.98915	7.48
Total protein	10	2.93	5.47	5.65	−3.18584	4.50
Albumin	10	3.6	3.61	3.64	−0.82418	3.01
Creatinine	10	1.95	3.893	3.636	7.068207	1.50
Urea	9	5.9	58.2	56.4	3.191489	0.98
Glucose	10	2.13	112.1	112.3	−0.17809	4.78
Cholesterol total	10	2.2	178.2	174.9	1.886792	3.69
Triglyceride	25	3.1	147.1	149.4	−1.53949	8.56
HDL	30	1.5	63.1	59.4	6.228956	15.85
Sodium	5	1.52	158.7	157.5	0.761905	2.79
Potassium	5	1.99	4.09	4.17	−1.91847	3.48
Chloride	5	2.75	104.1	103.6	0.482625	1.64
Magnesium	25	1.65	1.342	1.348	−0.4451	15.42
Phosphate	10	1.71	1.1	1.09	0.917431	5.31

Table 3.

Table showing the average calculated bias%, TEa%, CV% and sigma values of pathologic (level 2) quality control for a period 6 months in EPHI chemistry laboratory

Analytes	Tea% from CLIA	CV%	Our lab mean EQA result	Peer group EQA mean	Bias%	Sigma (σ)
Amylase p	30	0.97	87.6	81.8	7.090465	23.62
Amylase- total	30	1.76	137.1	135.4	1.255539	16.33
ALT	20	2.87	159.7	161.3	−0.99194	7.31
AST	20	1.6	212.4	211.6	0.378072	12.26
ALP	30	5.21	179.2	180.1	−0.49972	5.85
Direct bilirubin	20	3.943	1.297	1.293	0.309358	4.99
Total bilirubin	20	2.342	2.71	2.765	−1.98915	9.39
Total protein	10	2.7	5.47	5.65	−3.18584	4.88
Albumin	10	3.04	3.61	3.64	−0.82418	3.56
Creatinine	10	1.56	3.893	3.636	7.068207	1.88
Urea	9	5.6	58.2	56.4	3.191489	1.04
Glucose	10	2.21	112.1	112.3	−0.17809	4.61
Cholesterol total	10	6.3	178.2	174.9	1.886792	1.29
Triglyceride	25	2.2	147.1	149.4	−1.53949	12.06
HDL	30	1.7	63.1	59.4	6.228956	13.98
Sodium	5	1.35	158.7	157.5	0.761905	3.14
Potassium	5	1.86	4.09	4.17	−1.91847	3.72
Chloride	5	2.18	104.1	103.6	0.482625	2.07
Magnesium	25	1.84	1.342	1.348	−0.4451	13.83
Phosphate	10	1.69	1.1	1.09	0.917431	5.37

Fig. 1.

Graph showing Sigma Metrics of normal (level 1) quality control for a period of six months in a clinical chemistry reference laboratory, EPHI

Fig. 2.

Percentage of sigma metrics for analytes (19 test methods, 38 points) in both level 1 and 2 quality controls in a clinical chemistry reference laboratory, EPHI

Fig. 3.

Graph showing Sigma Metrics of abnormal (level 2) quality control of 19 test methods for a period of six months in a clinical chemistry reference laboratory, EPHI

Table 4.

Quality goal index ratio of analytes performed low for sigma for accuracy and precision problem analytes

Analytes	QC level	Bias%	CV%	Sigma	QGI	Problem
Creatinine	Level 1	7.07	1.95	1.5	9.19	Inaccuracy
	Level 2	7.07	1.56	1.88	7.35	Inaccuracy
Urea	Level 1	3.19	5.9	0.98	12.55	Inaccuracy
	Level 2	3.19	5.6	1.04	11.91	Inaccuracy
Sodium	Level 1	0.76	1.52	2.79	0.77	Imprecision
Chloride	Level 1	0.48	2.75	1.64	0.88	Imprecision and inaccuracy
	Level 2	0.48	2.18	2.07	0.70	Imprecision
Cholesterol total	Level 2	1.89	6.3	1.29	7.92	Inaccuracy

Method Decision Chart

Figure 4 showed the demonstration of method decision chart both for level 1 and level 2 quality controls in which imprecision measured with CV% is along the x-axis and inaccuracy expressed with bias (trueness) is along the y-axis. About 17/38 (44.7%) analytes test performance (both level 1 and level 2) showed world class quality (σ ≥ 6) plotted closest to the graph followed by excellent (5σ), good (4σ) and marginal (3σ) zones. The remaining five test performances in both level 1 and level 2 covered the graph below 2σ are tagged as unacceptable. From the method decision chart high sigma metrics found closest to the origin means very few defects or errors are generated while farthest graph indicates lowest sigma value and generates more defect or error beyond acceptable limit.

Fig. 4.

Sigma normalized method decision̄ chart for level 1 and level 2 EPHI Roche Cobas 6000 clinical chemistry analyzed (19 test methods, 38 points). Inaccuracy (bias, trueness) is on the y-axis. Imprecision (CV) is on the x-axis

Discussion

The primary focus of quality management system implementation in clinical laboratories is to provide quality test result which used for diseases screening, diagnosis, monitoring and treatment [24]. Building quality laboratory process is a continuous verification of pre-analytical, analytical and post-analytical process using internal and external audit. Most laboratories design the QC protocol for the number of times and number of levels the IQC is scheduled per day based on the guidelines of National Accreditation bodies. However, good laboratory practice (GLP) requires every individual laboratory to design a customized Individualized Quality Control Plan (IQCP) protocol based on Sigma values obtained from Sigma metric analysis [6]). Sigma values aimed to designing quality control rules, results in the reduction of laboratory errors like false rejection by maintain six standard deviation between the parameter average and its upper and lower limits [19]. Accomplishment of six sigma value is termed as gold standard for referring world class measure of quality. The Six Sigma represents, on a short-term scale, just 3.4 defects per million opportunities, that is, the analytical method is one that is expected to generate less than four erroneous results per million test reports [25].

In our study, we obtained values of six and greater sigma for amylase pancreatic, amylase total, AST, ALT, triglyceride, total bilirubin, HDL and magnesium for both levels of IQC. The highest values for sigma were found for amylase pancreatic with 17.9 and 23.6 for level 1 and 2 and, amylase total with sigma value of 17.11 and 16.33 for level 1 and 2 respectively. Thus, stringent internal quality control rules are not required for these parameters. According to Westgard such analytes with the sigma values of > 6 are required only single rejection rules of 1₃S [23]. As sigma increases, the consistency, reliability, steadiness and overall performance of the test improves, thereby decreasing the unnecessary test reputations and incorrect plan of treatments [26].

However, satisfactory sigma performance, sigma values between 4 and 6 (good/acceptable performance), was observed with ALP, glucose, total protein, direct bilirubin and phosphate in both levels of quality control. Westgard recommends using two levels of quality control, once daily and, following 13 s, 2–2 s and R4s multi-rules [27]. Two analytes albumin and potassium showed sigma value of 3–4. Similarly, total cholesterol and sodium also showed satisfactory sigma values (3–4) in level 1 and level 2 quality controls respectively. Two levels of controls, twice daily and follow 1–3 s, 2–2 s, R4s, and 4–1 s Westgard’s multi-rules should be practiced for these parameters [28, 29].

For parameters such as urea, creatnine and chloride, the sigma value was found to be < 3 for both levels of IQC in our study. In the same way total cholesterol and sodium was found to have sigma value less than 3 for level 2 and level 1 respectively so needs improvement of quality control methods. A method sigma below 3 calls for the adoption of a newer and better method as quality of the test cannot be assured even after repeated QC runs [23]. Westgard, sigma rules also recommended to review the daily workload division and the frequency of internal quality control strategy for analytes having less than three sigma value [29–33]. The QGI ratio for these parameters with sigma < 3 depicts the problem for urea and creatnine for both level 1 and 2 controls which could be due to inaccuracy (QGI > 1.2), however, sodium for level 1 and chloride for level 2 the problem could be due to imprecision (QGI < 0.8).

Numerous studies were done and different sigma metrics were reported. The study done by Kumar and Mohan reported four parameters with < 3 sigma metrics. Chauhan et al. [9] in their study reported, sigma value for urea and creatinine was found < 3 on Cobas Integra Plus analyzer, Mahmood Bushra et al. [33] in other study reported sigma values for urea and creatinine was < 3 on Siemens Dimension RxL Max analyzer for both levels of QCs. Similar findings were found in our study for urea and creatinine. However, sigma values for creatinine and urea were reported between 3 and 6 by Sunil Nanda et al. [34]  > 6 sigma by Sign et al. [35] for creatinine. Excellent sigma performance (≥ 6) was reported for AST, ALT and total bilirubin in the study done by Mahmood Bushra et al. [33], which is similar with our results. The variations in sigma values for few analytes between our study and others can be attributed to the difference in the methodology, Traceability calibrators used, instrument used, quality control material used, and other pre-analytical and analytical conditions [9, 11]

Evaluating and calculating of sigma metric is important in designing and implementing of quality control strategies. Simplified guidelines stated in (Table 5) can be used for choosing the Westgard rules and level of quality control processed. Furthermore, the sigma scale can be applied as a universal benchmark for the comparative evaluation of performance between tests, methods, equipment, and laboratories [36].

For the biochemical parameters with Sigma Scale 6 or above, the performance can be evaluated with one level of QC per day and follow 1–3 s Westgard rule alone. With Sigma Scale 4–6 performance can be evaluated with two levels of control once daily and follow 1–3 s, 2–2 s, R4 s Westgard multi-rules. With Sigma Scale 3–4 use two levels of controls twice daily and follow 1–3 s, 2–2 s, R4s, and 4–1 s Westgard’s multi-rules. The 3-sigma performance is the considered as the minimum for reporting laboratory results. With sigma Scale of < 3, root cause analysis should be performed based on five vital aspects: personnel, equipment, materials, method and environment related to poor performance before the method can be routinely used for releasing the results [29, 32, 33] Quality improvement plan on personnel proficiency and use alternative methods and change of reagents can be done for poor sigma performance analytes [37]. Prioritize the quality improvement plan through evaluation of low sigma value analytes and monitoring in daily quality indicators may improve the errors encountered [38]. Optimizing quality control procedures by increasing the frequency of quality control run can contribute optimally to patient healthcare quality without incurring loss in reagents, control materials, calibrators, labor and effort [39]. Running patient sample in duplicate is also preferable in addition to using maximum affordable levels of quality control like 3 levels/day [40]. Implementation of more stringent QC rules and take correction actions for these parameters are recommended [37]. In general for analytes with low sigma value, reducing analytical bias and imprecision is a key to improve the quality [41].

Conclusion

Since producing accurate and reliable test results is the main role for the laboratory, implementation of sigma metrics is important in Clinical Chemistry Reference Laboratory. Applying six sigma prevents us from inquiring to use stringent criteria in the laboratory which can reduce false rejections. Process improvement of quality control for analytes scored poor sigma metrics (< 3) was the way forward to the laboratory management.

Author Contributions

All authors have accepted responsibility for the entire content of this manuscript and approved its submission.

Funding

None declared.

Data Availability

The whole data supporting this study are included within the manuscript.

Declarations

Conflict of interests

Authors state no conflict of interest.