Application of Six Sigma for evaluating the analytical quality of tumor marker assays

Abstract

Context

The results of detection assays for the same specimen are usually quite different in different laboratories or when tested with different detection systems.

Objective

This study was designed to investigate the value of applying sigma metrics derived from different standards for allowable total error (TEa) in evaluating the analytical quality of tumor marker assays.

Methods

Assays were evaluated for these six tumor markers: total prostate‐specific antigen (tPSA), carcinoembryonic antigen (CEA), alpha‐fetoprotein (AFP), carbohydrate antigen 199 (CA199), carbohydrate antigen 125 (CA125), and carbohydrate antigen 153 (CA153). Sigma values were calculated for two concentrations of quality control products to assess differences in quality of tumor marker assays. Improvement measures were recommended according to the quality goal index, and appropriate quality control rules were selected according to the sigma value.

Results

The sigma value was highest using the higher biological variation‐derived “appropriate” TEa standard: it was sigma ≥6 or higher in 16.7% of tumor markers. Sigma was below 6 for all tumor markers using the other three TEa. CEA, AFP, CA199, CA125, and CA153 required improved precision. The marker tPSA required improve precision and accuracy. According to sigma values by using China's external quality assessment standards, CEA, AFP, CA125, and CA153 require 1_3s/2_2s/R_4s/4_1s multirules for internal quality control, CA199 requires use of 1_3s/2_2s/R_4s/4_1s/8_x multirules, and tPSA requires maximum quality control rules.

Conclusion

Six Sigma is useful for evaluating performance of tumor markers assays and has important application value in the quality control of these assays.

Abbreviations

AFP

Alpha‐fetoprotein

CA 125

Carbohydrate antigen 125

CA 153

Carbohydrate antigen 153

CA199

Carbohydrate antigen 199

CEA

Carcinoembryonic antigen

CLIA

Clinical Laboratory Improvement Amendments of 1988

CV

Coefficient of variation

dpm

Defects per million

EQA

External quality assessment

NCCL

National Center for Clinical Laboratories

QGI

Quality goal index

RCPA

Royal College of Pathologists of Australasia

Tea

Total allowable error

tPSA

Total prostate‐specific antigen

1. INTRODUCTION

Detection of tumor markers is widely conducted in medical laboratories in China. The results of detection assays for the same specimen are usually quite different in different laboratories or when tested with different detection systems.1 Therefore, it is necessary to improve the quality control of tumor marker detection to ensure accurate and credible results.

Sigma refers to “standard deviation” in mathematical statistics and is often used to express the defects per million (dpm) when measuring the performance of production processes. Six Sigma represents a defect rate of detection of 3.4 dpm.2 Six Sigma theory was first applied in the field of medicine by Nevakainen et al.3 As understanding of this theory spread, Six Sigma became an important quality management tool. Therefore, an increasing number of researchers have used Six Sigma to study quality improvement in laboratories.4, 5, 6 The sigma metric can evaluate the measurement performance of different assay processes as the quality targets and their sigma values differ.

The concept of “precision medical programs,” first proposed in the United States, has received widespread attention in recent years. The emergence of precision medicine has brought opportunities and challenges to the field of laboratory medicine and led to higher requirements for accuracy and reliability of results in clinical laboratories.7, 8 Tumor markers have become important clinically in tumor diagnosis, treatment, and monitoring, yet different detection systems often lead to different results. Therefore, efforts are necessary to strengthen the quality control of tumor marker detection to reduce experimental error and provide accurate and reliable data for clinical applications.9 Sigma metric analysis of current laboratory assay values can allow the selection of quality control rules that will effectively reduce the probability of false results, thereby ensuring the output of correct results. When the performance index of an assay reaches a sigma value of 6, simple rules of 1_3s or 1_3.5s can be used for quality control.10 A recent study by Hens et al used three different TEa standards [Ricos biological variation, the Clinical Laboratory Improvement Amendments of 1988 (CLIA 88), and RiliBÄK] to calculate sigma values of clinical chemistry assays. The authors showed that the main factors influencing sigma values were instrument performance and TEa standards. Accurate assay results would require control rules that are either simple or complex, based on the sigma.11

A few reports have evaluated the performance of tumor marker assays using sigma metrics. Accordingly, the aim of this study is to determine different sigma values based on four different standards for total allowable error (TEa) to analyze performance differences in the detection of tumor markers, and to explore their application in the quality control of tumor marker detection.

2. MATERIALS AND METHODS

2.1. Total allowable error standards

This study evaluated four different standards for TEa: (a) external quality assessment (EQA) standards of China; (b) quality requirements of the Royal College of Pathologists of Australasia (RCPA); (c) “appropriate” quality standards derived from biological variation12; and (d) standards from the 2015 quality guide created by the German medical laboratory quantitative analysis and quality assessment committee (RiliBÄK).

2.2. Tumor markers

The following markers were assessed: total prostate‐specific antigen (tPSA), carcinoembryonic antigen (CEA), alpha‐fetoprotein (AFP), carbohydrate antigen 199 (CA199), carbohydrate antigen 125 (CA125), and carbohydrate antigen 153 (CA153).

2.3. Instruments and reagents

The tumor markers were measured by chemiluminescence using the AIA2000 automatic chemiluminescence analyzer and corresponding reagents (TOSOH Co., Ltd., Tokyo, Japan). Calibrator was the original matched reagent. Internal quality control products were obtained from the Randox (lot 1:1575EC; lot 2:1578EC; Crumline, UK). All EQA samples were provided by the National Center for Clinical Laboratories (lots 201711, 201712, 201713, 201714, 201715).

2.4. Assay performance evaluation

2.4.1. Evaluation of precision

The coefficient of variation (CV) was used to represent precision. We collected laboratory quality control data from the tumor marker program at our laboratory over 6 months from January 2017 to June 2017 and determined CV values for two concentration levels of quality control products (level 1 and level 2; Table 1).

Table 1.

Coefficient of variation, bias, and total allowable error based on four standards for six tumor marker assays

Assay	CV (%)		Bias	TEa (%)
	Level 1	Level 2		EQA standards of China	RCPA standards	Biological variability	RiliBÄK
tPSA	6.49	6.35	8.42	25	20	33.6	25
CEA	5.53	5.36	1.24	25	20	24.7	24
AFP	5.34	5.25	1.31	25	20	21.8	24
CA199	6.12	6.36	3.58	25	15	39	24[Link]
CA125	4.22	4.51	3.55	25	20	35.4	24[Link]
CA153	5.12	5.05	2.73	25	15	20.8	24

AFP, alpha‐fetoprotein; CA125, carbohydrate antigen 125; CA153, carbohydrate antigen 153; CA199, carbohydrate antigen 199; CEA, carcinoembryonic antigen; CV, coefficient of variation; EQA, external quality assessment; RCPA, Royal College of Pathologists of Australasia; TEa, total allowable error; tPSA, total prostate‐specific antigen.

Value for the CA153 standard.

2.4.2. Evaluation of bias

Percentage difference was used to evaluate bias. Based on first‐time data from the first tumor marker assays performed in 2017, the laboratory obtained a 5‐percent difference. The absolute value of the mean percentage difference was used to evaluate the bias of our laboratory (Table 1).

2.4.3. Calculation of sigma value

Sigma values were calculated using this formula: sigma = [TEa (%) − bias (%)]/CV (%). Sigma values were determined for four different TEa standards and two concentration levels of quality control products.

2.4.4. Calculation of quality goal index

The quality goal index (QGI) was calculated according to this formula: QGI = bias (%)/1.5 × CV (%). We calculated the QGI for each tumor marker at both concentrations of quality control products, noted differences in QGI corresponding to different sigma values, and determined priority measures for quality improvement. When QGI was <0.8, the CV was relatively large, suggesting that the priority should be to improve precision. When QGI was higher than 1.2, the bias was relatively large, suggesting that the priority should be to improve accuracy. A QGI between 0.8 and 1.2 suggested that both precision and accuracy should be improved.

3. RESULTS

3.1. Distribution of sigma metrics for four total allowable error standards

Using the EQA standards of China, the distribution of sigma values for the six tumor marker assays was as follows (Table 2): more than 6, none; 4 to 6, 66.7% (4/6); and less than 4, 33.3% (2/6). Using the RCPA standards, all sigma values were <4. Using the “appropriate” standards, the distribution of sigma values was as follows: more than 6, 16.7% (1/6); 4 to 6, 33.3% (2/6); and less than 4, 50% (3/6). Using the German RiliBÄK standards, the distribution of sigma values was the same as for the EQA standards of China (Table 2).

Table 2.

Sigma metrics of six tumor marker assays using four different standards for total allowable error target values

Assay	EQA standards of China		Standards of RCPA		Biological variability		RiliBÄK
	Level 1	Level 2	Level 1	Level 2	Level 1	Level 2	Level 1	Level 2
tPSA	2.55	2.61	1.78	1.82	3.88	3.97	2.55	2.61
CEA	4.3	4.43	3.39	3.5	4.24	4.38	4.12	4.25
AFP	4.44	4.51	3.5	3.56	3.84	3.9	4.25	4.32
CA199	3.5	3.37	1.87	1.8	5.79	5.57	3.34	3.21
CA125	5.08	4.76	3.9	3.65	7.55	7.06	4.85	4.53
CA153	4.35	4.41	2.4	2.43	3.53	3.58	4.15	4.21

AFP, alpha‐fetoprotein; CA125, carbohydrate antigen 125; CA153, carbohydrate antigen 153; CA199, carbohydrate antigen 199; CEA, carcinoembryonic antigen; CV, coefficient of variation; EQA, external quality assessment; RCPA, Royal College of Pathologists of Australasia; tPSA, total prostate specific antigen.

3.2. Quality corrective actions

Using the EQA standards of China, the QGI was calculated for tumor marker assays with sigma values <6 (ie, all tumor marker assays). The results showed that for tPSA, both precision and accuracy required improvement, whereas for the other markers, only measures to improve precision are required (Table 3).

Table 3.

Quality target indices and quality improvement measures of six tumor marker assays according to the Chinese EQA standards

Assays	TEa (%)	Sigma values		QGI		Priority measures
		Level 1	Level 2	Level 1	Level 2
tPSA	25	2.55	2.61	0.86	0.88	Precision and accuracy
CEA	25	4.3	4.43	0.15	0.15	Precision
AFP	25	4.44	4.51	0.16	0.17	Precision
CA199	25	3.5	3.37	0.39	0.38	Precision
CA125	25	5.08	4.76	0.56	0.52	Precision
CA153	25	4.35	4.41	0.36	0.36	Precision

AFP, alpha‐fetoprotein; CA125, carbohydrate antigen 125; CA153, carbohydrate antigen 153; CA199, carbohydrate antigen 199; CEA, carcinoembryonic antigen; CV, coefficient of variation; QGI, quality goal index; TEa, total allowable error; tPSA, total prostate specific antigen.

3.3. Measures of internal quality control

Using the EQA standards of China, Westgard sigma rules were used to develop individualized internal quality control measures. The Westgard sigma rules for two levels of controls are shown in Figure 1. The most important aspect is the sigma scale at the bottom of the diagram, which provides guidance for rules that should be applied based on the sigma value of each assay. For CEA, AFP, CA125, and CA153 (with sigma values of 4‐5), 1_3s/2_2s/R_4s/4_1s multirules are required, with two batches of two quality control measurements (N = 2, R = 2) or 1 batch of 4 quality control measurements (N = 4, R = 1). The CA199 assay (with sigma values around 3) requires implementation of 1_3s/2_2s/R_4s/4_1s/8_x multirules, with 2 batches of 4 quality control measurements (N = 4, R = 2) or 4 batches of 2 quality control measurements (N = 2, R = 4). Because the sigma value of tPSA was <3, maximum quality control rules are necessary to ensure quality control (Figure 1).

Figure 1.

Westgard sigma rules for two levels of controls. Sigma value = (total allowable error [%] − |bias [%] |)/coefficient of variation (%)

4. DISCUSSION

Our study used four different TEa standards to calculate sigma values of six tumor markers. RCPA standards are the strictest, “appropriate” standards based on biological variability are the least strict, and the EQA standards of China and German RiliBÄK standards exhibit intermediate levels of strictness. Especially for CA199 and CA 125, these two values are assumed values, the same as for CA153 (Table 1). Using the current EQA standards of China, the sigma values of all six tumor markers in our laboratory were <6, indicating that assay performance for these markers requires quality improvement. Using TEa derived from biological variation, Westgard et al13 calculated sigma values of 53 assays, including CA125, CA153, and CA199; their results were consistent with the results of our current study. Westgard et al13 suggested that laboratorians can use Six Sigma tools as aids in choosing high‐quality assays, further contributing to the delivery of high‐quality healthcare for patients.

We calculated QGI for all six tumor markers, and the sigma values were less than 6 according to the Chinese EQA standards. Our results showed that all of six assays required quality improvement measures to improve precision, and tPSA also required measures to improve accuracy. Westgard et al (2016) established Westgard multirules, which are compiled in a new quality control tool called “Westgard Sigma Rules.” This tool may provide an individualized program to select quality control rules and determine the appropriate number of quality control measurements.14 Using the EQA standards of China and applying the “Westgard Sigma Rules” based on assay sigma levels, an individualized internal quality control scheme can be developed to reduce the false rejection rate, increase the error detection rate, and provide a continuous, feasible method for improving quality of tumor marker detection.

To assess the quality of laboratory testing in the United States, Westgard et al15 used sigma metrics to assess the quality of proficiency testing survey results from several national programs complying with CLIA regulations. Their research showed that sigma values of different analytes are significantly different, and sigma values are mainly distributed from 1.2 to 3.5. These authors therefore suggested that no less than two concentrations of quality control products should be used every day for internal quality control to ensure acceptable test results.15 Xia et al16 also showed that sigma metric is a useful tool to evaluate assay performance and suggested that an assay with a high sigma value could use a simple internal quality control rule, whereas an assay with a low sigma value should be monitored strictly. The conclusions of our study are in agreement with the results of the above scholars, indicating that sigma metrics can play an important role in the field of quality control.

Different standards for assessing quality often lead to different sigma values, which are dynamically changing. According to current quality specifications, clinical laboratories should regularly measure detection performance for tumor markers and develop individualized quality control plans for all analytes to provide long‐term, feasible measures for continuous improvement of assay quality. Six Sigma has an important guiding value for evaluating the quality of tumor marker assays.

AUTHOR CONTRIBUTIONS

QL and F.Y conceived and designed the experiments. QL, MF, WL, and C.Y performed the experiments. WZ and C.Z analyzed the data. QL, MF, and F.Y wrote the paper.

Liu Q, Fu M, Yang F, et al. Application of Six Sigma for evaluating the analytical quality of tumor marker assays. J Clin Lab Anal. 2019;33:e22682 10.1002/jcla.22682