The clinical performance evaluation of novel protein chips for eleven biomarkers detection and the diagnostic model study

Abstract

We aimed to develop and validate two novel protein chips, which are based on microarray chemiluminescence immunoassay and can simultaneously detected 11 biomarkers, and then to evaluate their clinical diagnostic value by comparing with the traditional methods. Protein chips were evaluated for limit of detection, specificity, common interferences, linearity, precision and accuracy. 11 biomarkers were simultaneously detected by traditional methods and protein chips in 3683 samples, which included 1723 cancer patients, 1798 benign diseases patients and 162 healthy controls. After assay validation, protein chips demonstrated high sensitivity, high specificity, good linearity, low imprecision and were free of common interferences. Compared with the traditional methods, protein chips have good correlation in the detection of all the 13 kinds of biomarkers (r≥0.935, P<0.001). For specific cancer detection, there were no statistically significant differences between the traditional method and novel protein chips, except that male protein chip showed significantly better diagnostic value on NSE detection (P=0.004) but significantly worse value on pro-GRP detection (P=0.012), female chip showed significantly better diagnostic value on pro-GRP detection (P=0.005). Furthermore, both male and female multivariate diagnostic models had significantly better diagnostic value than single detection of PGI, PG II, pro-GRP, NSE and CA125 (P<0.05). In addition, male models had significantly better diagnostic value than single CA199 and free-PSA (P<0.05), while female models observed significantly better diagnostic value than single CA724 and β-HCG (P<0.05). For total disease or cancer detection, the AUC of multivariate logistic regression for the male and female disease detection was 0.981 (95% CI: 0.975-0.987) and 0.836 (95% CI: 0.798-0.874), respectively. While, that for total cancer detection was 0.691 (95% CI: 0.666-0.717) and 0.753 (95% CI: 0.731-0.775), respectively. The new designed protein chips are simple, multiplex and reliable clinical assays and the multi-parameter diagnostic models based on them could significantly improve their clinical performance.

Introduction

Nowadays, remarkable progresses have been made in understanding the molecular and cellular events that transform a normal functioning cell into a malignant one [1,2]. Based on these progresses, cancer detection has entered into molecular diagnostic phase and the types of biomarkers are rapidly expanding, including proteins, peptides, hormones, certain DNA, microRNAs, and so on [3,4]. Although it’s very significant to focus on the development of improved therapies for cancer patients, it’s still important to invent multiplex early detection methods, which could scent precancerous changes with high specificity and sensitivity [5]. Finding cancer at its earliest, most treatable stage gives patients the best chance to survive. Common methods recommended by American Cancer Society (ACS) for cancer early detection include many types, such as microbial detection, fecal occult blood test, digital rectal examination, double-contrast barium enema, computed tomography, tumor markers test, and so on [6]. Among these, the detection of tumor markers, which are directly produced or indirectly induced by the tumor cell and could reflect its growth and/or activity, is one of the most convenient means to monitor the insignificant or potential cancer transformation [7].

However, tumor markers are not specific to cancer as their name looks like and the majority of them could be synthesized and released by normal cells, too. Consequently, the specificity of tumor markers is not their presence at tumor sites but the concentrations detected there, which is why normal values are needed [8]. Moreover, present clinical proven methods, such as radioimmunoassay or chemiluminescence, could only detect one biomarker at a time. In other words, they don’t observe high-throughput abilities, which result in low working efficiency and suppress their use in mass screening. On the contrary, protein array technology possesses the greatest potential for screening thousands of recombinant-, fractions- or purified proteins, as well as synthetic peptides and even unknown proteins in a high throughput [9,10]. Furthermore, since various benign pathologies could also produce tumor markers, which give rise to false positives, single tumor marker has relatively low specificity and sensitivity and might sometimes shows equivocal results. Therefore, how to improve the detection efficiency and increase the specificity and sensitivity is the most important aspect of diagnostic methods [11-12]. To break through this bottleneck, using multiple biomarkers or indicators to establish diagnostic models is a common effective strategy, which could integrate the advantages of each index and simultaneously improve both sensitivity and specificity [13-15].

In our study, we have developed two protein chips, which are based on microarray chemiluminescence immunoassay and could simultaneously detect 11 kinds of biomarkers of man or woman, respectively. Specifically, Pepsinogen I (PGI), Pepsinogen II (PGII), pro-Gastrin-releasing peptide (pro-GRP), alpha-fetoprotein (AFP), Neuron-specific enolase (NSE), Carcino-embryonic antigen (CEA), Carbohydrate antigen 125 (CA125), Carbohydrate antigen 199 (CA199) and Carbohydrate antigen 724 (CA724) are common in both male and female chips. Meanwhile, total-Prostate specific antigen (t-PSA) and free-Prostate specific antigen (f-PSA) are unique for male chip, beta-Human chorionic gonadotropin (β-HCG) and Carbohydrate antigen 153 (CA153) are unique for female chip. Then, protein chips were evaluated for limit of detection, specificity, common interferences, linearity, precision and accuracy. After that, diagnostic models were established to improve the detection specificity and sensitivity at the same time. We aimed to evaluate the clinical performance of the protein chips and the feasibility of using multiple biomarkers to establish diagnostic models for cancer detection.

Patients and methods

Patients and healthy controls

Our study was approved by ethics committee of the Chinese PLA General Hospital, Shanghai Changzheng Hospital, Nanjing general hospital of Nanjing military command and the second affiliated hospital of Zhejiang university school of medicine and has been performed in accordance with the ethical standards laid down in an appropriate version of the 2000 Declaration of Helsinki as well as the Declaration of Istanbul 2009. With written consent, the serum samples were collected and detected in four hospitals mentioned above. All the four hospitals used the same batch protein chip detection reagent and machine to ensure that the results were reliable and comparable. 1723 cancer patients, 1798 benign disease patients and 162 healthy controls were enrolled in the study. Both of the cancer and benign patients’ samples were collected before any treatment, such as surgery, chemotherapy or radiation therapy, and further confirmed by histopathological analysis. Healthy controls were enrolled based on their negative results on the blood biomarker test, x-ray, ultrasound, computed tomography examination and fecal occult blood testing. The clinical characteristics of the participants were shown in Table 1.

Table 1.

Characteristics of total studied subjects

Characteristics	Malignant tumor (n=1723)	Benign diseases (n=1798)	Normal control (n=162)
Age (years)
Mean	57.70	57.79	39.53
Range	9-91	9-97	13-54
Sex, n (%)
Male	837 (48.6)	822 (45.7)	97 (59.9)
Female	886 (51.4)	976 (54.3)	65 (40.1)
Tissue distribution, n (%)
Lung	522 (30.3)	152 (8.5)	-
Gastrointestinal	419 (24.3)	183 (10.2)	-
Liver and gall	192 (11.1)	226 (12.6)	-
Others	590 (34.2)	1237 (68.8)	-

Serum collection

A total of 5 ml of peripheral blood samples was collected in tubes containing a separating gel and clot activator. After blood collection, the tubes should be upside down ten times quickly and then rest for about 1 hour. After centrifuging at 2500 rpm for 5 min, the samples could be directly detected by corresponding techniques. If it’s not convenient to do the test at once, the supernatant could be transferred into new tubes, and the serum should be stored in aliquots at -80°C until detection.

Measurement of serum biomarker levels using novel protein chips

The microarray-based sandwich immunoassay kits (Sunlant Bioengineering, Wuxi, China), which are cooperated with protein chips, were used to detect the serum concentrations of cancer related biomarkers. Briefly, serum sample and microarray that was covalently coupled to specific capture antibodies were incubated together. After washing, a cocktail of horseradish peroxidase-labeled antibodies were added to the array, which was shaken at room temperature. Then, the array was transferred to a cuvette with substrate solution and detected by an automatic biochip reader. The results were recorded by standard curves established by five different concentrations.

Measurement of serum biomarker levels using traditional detection methods

PGI, PGII and pro-GRP were detected by an Abbott i2000 instrument (Abbott Laboratories, USA). AFP, NSE, CEA, CA125, CA199, CA724, PSA, f-PSA, β-HCG and CA153 were detected by Roche Cobas e601 instrument (Roche Diagnostics, Germany). The main principle of these methods is competitive protein binding (CPB), however they have different detection methods: electrochemiluminescence detection in Roche Cobas e601 and chemiluminescence detection in Abbott i2000. Detection was conducted according to the manufacturer’s reagent kit manual for specific biomarkers.

Evaluation of detection sensitivity and specificity of protein chips

For sensitivity evaluation, the experiment to determine the limit of detection (LOD) was performed on 3 separate occasions, as follows: the zero concentration calibrator was evaluated for 15 replicates, respectively. The average and 2SDs (Standard deviation) of the determinations was used to calculate LOD. For specificity evaluation, 50 samples of patients with high expression of various biomarkers were detected by protein chips to validate their detection specificity. In addition, 30 samples from hemolysis, hyperlipidemia or jaundice were used to validate the detection stability.

Evaluation of detection linearity of protein chips

For each biomarker, linearity was evaluated using Abbott or Roche calibrators in triplicate measurements. Linearity was also evaluated using a patient sample with high concentration of specific biomarker. This sample was diluted with blank whole blood at different proportions to achieve 7 dilutions. These diluted specimens were each examined in triplicate.

Evaluation of precision performance of protein chips

According to NCCLS EP5-A program [16], both intra- and inter-batch examination used the same machine and standard curve but three reagent box with different lot number. For the former, repeat detecting two internal quality controls for 20 times in one day and then calculating the intra-batch precision. For the latter, repeat detecting one internal quality control for 2 times in one day and continuously monitoring for 20 days and then calculating the inter-batch precision.

Statistical analyses

Statistical analyses were carried out using SPSS 22.0 (SPSS Inc., USA) and MedCalc 13.0 (MedCalc Software, Belgium). Pearson’s correlation coefficient was used for correlation analysis of the associated quantitative data. Kappa test was used to detect the diagnostic consistency between methods. Receiver operating characteristic (ROC) curves were established to evaluate the diagnostic value, and the Areas under the ROC curves (AUC) were compared by z-scores test. The cutoff value, which determined the sensitivity and specificity, was chosen by the Youden index. Binary logistic regression models were built to evaluate the multivariate diagnostic value of multiple biomarkers detected by protein chips. A two-tailed p-value of less than 0.05 was considered statistically significant.

Results

Evaluation of detection sensitivity and specificity of protein chips

The LOD for protein chips were defined here as the determination at 2SDs above the zero concentration calibrators. Based on 3 separate analyses, the average LOD for PGI, PGII, pro-GRP, AFP, NSE, CEA, CA125, CA199, CA724, t-PSA, f-PSA, β-HCG and CA153 was 4 ng/ml, 1.5 ng/ml, 0.02 ng/ml, 2 ng/ml, 2.5 ng/ml, 0.5 ng/ml, 2.3 U/ml, 5 U/ml, 1 U/ml, 0.1 ng/ml, 0.1 ng/ml, 0.4 ng/ml and 5 U/ml. Meanwhile, protein chips could specifically detect every biomarker on the chips without cross-reactivity. Additionally, interference potential was evaluated by determining biomarker concentration for samples from hemolysis, hyperlipidemia or jaundice. The results showed that severe hemolysis samples are prone to lead false positive of NSE index, while samples from hyperlipidemia or jaundice have no impacts on the chip results (data not shown). It suggested that protein chips had comparable detection sensitivity and specificity with traditional methods and could meet the basic requirements of clinical use.

Evaluation of detection linearity of protein chips

After linearity evaluation with Abbot or Roche calibrators and serial dilutions of patient samples, protein chips were found to be linear to detect PGI, PGII, pro-GRP, AFP, NSE, CEA, CA125, CA199, CA724, t-PSA, f-PSA, β-HCG and CA153 in the range of 6-300 ng/ml, 2-200 ng/ml, 0.03-5 ng/ml, 3-1000 ng/ml, 2.6-300 ng/ml, 1-300 ng/ml, 3.5-3000 U/ml, 7-2000 U/ml, 1.35-400 U/ml, 1-300 ng/ml, 0.2-100 ng/ml, 0.6-200 ng/ml and 7-300 U/ml. For every biomarker, the correlation coefficient of linearity was higher than 0.99 (r≥0.99). It suggested that protein chip had good detection linearity and wide linear range.

Evaluation of precision performance of novel protein chips

In order to evaluate the precision performance, protein chips for male and female were respectively used to do the intra- and inter-batch examination. As shown in Supplementary Tables 1, 2, 3 and 4, coefficient of variation (%CV) of male chip in intra-batch examination ranged from 2.4% to 10.0%, %CV of male chip in inter-batch examination ranged from 5.0% to 9.1%, %CV of female chip in intra-batch examination ranged from 1.89% to 9.95%, %CV of female chip in inter-batch examination ranged from 4.66% to 9.56%. It suggested that both male and female protein chips have good detection precision in intra- and inter-batch. In other words, protein chips are reliable detection methods and had good repeatability and stability.

Correlation analysis between the traditional methods and novel protein chips

All kinds of biomarkers were detected by both the traditional methods and novel protein chips, and then the same biomarkers from both chips were analyzed together. As showed in Figure 1, the correlation coefficient for PGI, PGII, pro-GRP, AFP, NSE, CEA, CA125, CA199, CA724, t-PSA, f-PSA, β-HCG and CA153 was 0.935, 0.947, 0.988, 0.985, 0.969, 0.980, 0.980, 0.980, 0.978, 0.979, 0.982, 0.981 and 0.966. And all the p values for them were 0.000. It suggested that protein chips have good correlation and detection consistency with the traditional method in the detection of all the 13 kinds of biomarkers.

Figure 1.

Detection correlation between conventional method and protein chips. New- represents results detected by protein chips, while old- represents results detected by conventional method.

Diagnostic value of the 11 biomarkers detected by traditional methods and novel protein chips for specific cancers detection

After the assay validation and correlation analysis, we examined the diagnostic ability of the biomarkers depending on their clinical utility. Generally, PGI, PGII, CEA, CA199 and CA724 were used for gastrointestinal tumors detection, while pro-GRP and NSE for lung cancer, AFP for liver cancer, t-PSA and f-PSA for prostate cancer, β-HCG for gynecologic tumor, CA153 for female breast cancer, and CA125 for all cancers. The diagnostic value of the two methods on specific cancer screening was evaluated by ROC curves and their AUCs were compared by z-score test. Benefit from the multiplex detection ability of protein chips, we further established binary logistic regression models to evaluate their multivariate diagnostic value in specific cancer screening. The diagnostic models and their specificity (SPC) and sensitivity (SEN) were listed in Supplementary Table 5.

As shown in Figures 2, 3, there were no statistically significant differences between the traditional method and novel protein chips, except that male protein chip showed significantly better diagnostic value on NSE detection (P=0.004) but significantly worse value on pro-GRP detection (P=0.012), female chip showed significantly better diagnostic value on pro-GRP detection (P=0.005). Furthermore, compared with traditional methods, multivariate diagnostic models established on chip results had significantly better diagnostic value as a whole. Specifically, both male and female models had significantly better diagnostic value than single detection of PGI, PG II, pro-GRP, NSE and CA125 (P<0.05). In addition, male models had significantly better diagnostic value than single CA199 and free-PSA (P<0.05), while female models observed significantly better diagnostic value than single CA724 and β-HCG (P<0.05). It suggested that protein chips are comparable with traditional methods in single biomarker detection manner. Once establishing diagnostic models based on multiplex chip results, it observed significantly higher diagnostic value than single detection of 7 biomarkers in 11. It showed that protein chips could not only increase the detection efficiency, but also improve the diagnostic value a lot.

Figure 2.

Receiver operating characteristic curve analysis of biomarkers detected by conventional methods, male chip or multivariate diagnostic models for evaluating the diagnostic value of specific cancer prediction. *P is the p value of z statistic compared between conventional methods and male chip, while #p is the p value of that between conventional method and multivariate diagnostic model.

Figure 3.

Receiver operating characteristic curve analysis of biomarkers detected by old methods, female chip or multivariate diagnostic models for evaluating the diagnostic value of specific cancer prediction. *P is the p value of z statistic compared between conventional methods and female chip, while #p is the p value of that between conventional method and multivariate diagnostic model.

Multivariate diagnostic value of protein chips for total disease or cancer detection

Similarly, based on the multiplex detection ability of protein chips, we have established binary logistic regression models to evaluate their multivariate diagnostic value in total disease or cancer detection. Specifically, in total disease identification, both the cancer and benign disease patients were considered as “disease positive” group, while the normal control as “disease negative” group. Meanwhile, in the case of total cancer identification, the cancer patients were set as “cancer positive” group, while the benign disease patients as “cancer negative” group. The diagnostic models and their SPC and SEN were listed in Supplementary Table 6. As shown in Figure 4, the AUC of multivariate logistic regression for the male and female disease detection was 0.981 (95% CI: 0.975-0.987) and 0.836 (95% CI: 0.798-0.874), respectively. While, that for cancer detection was 0.691 (95% CI: 0.666-0.717) and 0.753 (95% CI: 0.731-0.775), respectively. The results preliminarily indicated that protein chips might have a certain diagnostic value in the detection of total diseases or tumors.

Figure 4.

Receiver operating characteristic curve analysis of multivariate diagnostic models for evaluating the diagnostic value of disease or cancer differentiation. The sensitivity and specificity were determined by the maximum Youden index. AUC: Areas under the curve.

Discussion

Cancer, which is one of the most common causes of disease-associated death, is a major health issue in the world [17]. Fortunately, a substantial proportion of cancers, which are caused by cigarette smoking, heavy use of alcohol or other bad habits, could be prevented if people could reduce such harmful exposure. Moreover, some cancers could even be cured if people could scent precancerous changes and get diagnosed at an early stage, such as cancers of the breast, cervix, colon, and rectum [18-20]. Therefore, early detection and timely diagnosis is the key point of cancer prevention and control, which also predicts good clinical outcomes. Unexpectedly, although developed countries could provide the best services for detection, diagnosis, and treatment, they have the highest incidence of cancer. The highest prevalence proportions of cancer also occur in these populations. This could be explained by the following three reasons. Firstly, developed countries observe high level of medical technology, which could detect more precancerous patients at early stages, while these patients often miss their survival chances in developing countries. Secondly, most developed countries are aging society, which has a higher incidence of cancer. At last, higher incidence of cancer often means that other diseases, such as infectious ones, are less. So, this unexpected incidence level doesn’t mean that there’s no need to achieve more abilities of cancer detection, diagnosis and treatment.

As reported by Chinese National Cancer Institute, lung cancer, gastric cancer, liver cancer, esophageal cancer, colorectal cancer, female breast cancer and cervical cancer were the most common cancers in China [21]. In our study, various biomarkers most related to the aforementioned cancers were detected by clinical proven methods and new designed protein chips, named male chip and female chip, which are based on microarray chemiluminescent immunoassay and could simultaneously detect 11 kinds of biomarkers. Mukoubayashi C and his colleagues have found that serum PGI/II testing is useful in detecting early-stage gastric cancers arising from atrophic gastric mucosa [22]. As reviewed by Ischia J, pro-GRP is the most sensitive serum biomarker in patients with small cell lung cancer and is a significant predictor of poor survival in patients with advanced prostate cancer [22]. Interestingly, β-HCG is not only an important hormone, but also could be used as tumor marker for some gynecological cancers [24]. Although with various levels of SPC and SEN, the rest of the biomarkers targeted by protein chips are common clinical proven tumor markers [25-27].

After comprehensive assay validation, protein chips demonstrated high sensitivity, high specificity, good linearity, low imprecision and were free of common interferences. Compared with the traditional methods, protein chips have good correlation in the detection of all the 13 kinds of biomarkers (r≥0.935, P<0.001). As detailed in the results, we further compared the diagnostic ability among traditional methods, protein chips and diagnostic models based on the multiplex chip results. The ROC curve results suggest that both traditional methods and protein chips are roughly the same in detecting single biomarker for cancer detection, except NSE and pro-GRP. Specifically, female chip observed better diagnostic value in predicting lung cancer through pro-GRP detection, while male chip had worse value of that. In addition, male chip had better diagnostic value in NSE detection for lung cancer discrimination. Once establishing diagnostic models based on multiplex chip results, it could significantly enhance their diagnostic ability. Thereafter, trying to improve the detection specificity and sensitivity of male or female chip, we have established 4 diagnostic formulas based on multivariate logistic regression analyses for total disease or cancer detection, respectively. Considering the SPC and SEN, it showed that protein chips might be useful in total disease detection but play limited roles in cancer discrimination. However, due to the small ratio of normal control in both male (97 in 1756) and female (65 in 1927) group, the high SPC of total disease differentiation might be not as real as it showed.

In conclusion, we have comprehensively assessed the new designed protein chips, which could simultaneously detect 11 kinds of biomarkers closely related to the most common cancers in China. After assay validation and diagnostic value comparison, we found that the new designed protein chips have generally equal diagnostic ability to the traditional methods and they observe high-throughput detection capabilities. Furthermore, combining with multivariate logistic regression, we found that diagnostic models established from the chip results might be useful in cancer detection. In a word, these chips might play important roles in cancer early detection, especially for mass screening, which has urgent need for high-throughput testing methods. Although this study was conducted in four different hospitals, the sample size was still small, especially that of the healthy control group. Besides, the sample size of specific diseases or cancer was also small. Hence, multicenter studies with larger sample sizes will be needed to further validate the value of the chips in diagnosing cancer.

Disclosure of conflict of interest

None.