Abstract
Research use only (RUO) assays do not undergo a validation process similar to test kits used for clinical purposes. Several studies have suggested that RUO assays need to be validated prior to use in any research studies. We evaluated a research use only Luminex platform based assay for measuring serum procalcitonin levels (Bio-Plex ProTM Human Acute Phase Multiplex Assay, Bio-Rad Laboratories, Hercules, CA) for comparability with an FDA cleared assay for procalcitonin (VIDAS B.R.A.H.M.S. PCT Assay, bioMérieux, Durham, NC). We tested 1,072 serum samples collected from patients with suspected sepsis in an intensive care unit setting for the comparison. There was poor correlation of the luminex based assay (r=0.081) with the VIDAS PCT Assay in the clinically relevant measurement range (<10 ng/mL). Additionally the Bio-Plex assay showed poor precision. Mass-spectrometry analysis of material eluted from PCT beads did not reveal any identifiable procalcitonin. The results show that research use only assays need to be validated to determine their suitability for research studies.
Keywords: Biomarkers, luminex, procalcitonin, research use only (RUO) assays, sepsis
Introduction
The discovery and development of putative protein biomarkers for various disease states has increased dramatically with the rise of high-throughput technologies [1]. However, the quality and accuracy of commercially available immunoassays intended to measure these protein biomarkers is of increasing concern. In particular, commercial immunoassays that are sold for research use only, and do not undergo the scrutiny of device/diagnostic regulatory clearance, are of variable quality in terms of analytical sensitivity and specificity [1,2].
In preparation for conducting several studies to identify the potential role of serum biomarkers in informing antibiotic de-escalation algorithms in the intensive care unit (ICU) setting [3], we evaluated a research use only (RUO) multiplex biomarker assay that included procalcitonin (PCT) as one component of the assay panel.
PCT is one of the most studied biomarkers for the diagnosis of bacterial infection, and has shown promising results in multiple studies and in various clinical settings [4-9]. In preliminary validation work performed as part of our predictive study, results from the Bio-Plex ProTM PCT assay raised some concerns about test accuracy. Specifically, given past reported concerns regarding the accuracy of commercially available RUO immunoassays [1,2] we sought to validate the PCT component of this multiplex assay against a Food and Drug Administration (FDA)-cleared clinical PCT assay.
Materials and methods
Samples
Residual blood samples obtained from adult patients with possible bacterial sepsis in the medical ICU (MICU) and surgical ICU (SICU) of the Hospital of the University of Pennsylvania (HUP) [3] and samples from pediatric patients in the pediatric ICU (PICU) of Children’s Hospital of Philadelphia (CHOP), were utilized for this study. Blood collected in tubes without anticoagulant submitted to the core laboratory at HUP or CHOP for routine chemistry or other analysis were sampled within 12-24 hours of collection time. Serum was obtained and aliquoted into 300 ul microtubes. Four serum samples were targeted to be collected from each patient over a 72 hour period as previously described [3]. Serum was stored at -70°C until testing. This study was reviewed and approved by the Institutional Review Board of the University of Pennsylvania.
Procalcitonin assays
The Bio-Plex ProTM Human Acute Phase Multiplex Assay (Bio-Rad Laboratories, Hercules, CA), a research use only assay, consists of 2 kit panels, one with 5 analytes (includes procalcitonin) and a second panel with 4 analytes, and was used for this study. The Bio-Plex ProTM Human Acute Phase Multiplex Assay is a magnetic bead-based (xMAP technology) multiplex assay that enables the simultaneous measurement of nine protein biomarkers, including PCT, in serum or plasma. The assay was performed as per manufacturer’s instructions utilizing a Bio-Rad Bio-Plex 200TM reader, and a Bio-Rad Bio-Plex ProTM Wash Station. A single lot of the Bio-Plex Pro multiplex assay was used for the study (Lot #5036301).
For comparison, we used the VIDAS B.R.A.H.M.S. PCT Assay (bioMérieux, Durham, NC), a one-step immunoassay sandwich method with fluorescent detection that utilizes the bioMérieux VIDAS instrument platform. The VIDAS PCT assay is an FDA cleared clinical assay for the measurement of PCT in serum or plasma as a marker of severe bacterial infection and sepsis. The VIDAS PCT assay was performed as per manufacturer’s instructions, and standard clinical procedures.
Serum samples were thawed on the same day of analysis, and stored at 4°C until testing. The two assays were run concurrently. The Linearity FD Procalcitonin bioMerieux VIDAS and miniVIDAS validation kit from Audit MicroControls (Eatonton, GA) was used to assist in validating and comparing the performance of both assays. The Audit MicroControls kit contains five distinct levels of bovine based serum albumin and PCT which are intended to demonstrate a linear relationship when analyzed.
To evaluate the relative performance characteristics for a different target protein biomarker included in the Bio-Plex Pro assay, high-sensitivity C-reactive protein (hsCRP), which is included in the panel, was also analyzed. A selection of serum samples were run using the Bio-Plex assay and the Olympus Beckman Coulter AU 680 (Beckman) for analysis of hsCRP.
Mass-spectrometry
Identification of serum-based proteins captured by the bead-bound capture antibodies of the Bio-Plex assay was performed using high-resolution liquid chromatography-tandem mass spectrometry (LC-MS/MS). Briefly, Bio-Plex beads coupled to antigen were centrifuged to remove the storage buffer and resuspended in 100 uL of 50 mM ammonium bicarbonate (ABC). The samples were reduced with 10 uL of 100 mM DTT for 1 hr at 60°C and then alkylated at room temperature in the dark for 1 hr using 15 uL of 100 mM iodoacetic acid (IAA). Samples were then incubated at 37°C overnight in a Thermoshaker with 1 ug of trypsin. Trypsinization was stopped by acidifying the sample to make a final concentration of 0.1% trifluoroacetic acid (TFA). Sample clean-up was performed using StageTip C-18 purification as previously described. LC-MS/MS analysis utilized a 60-minute gradient on a C-18 column coupled by nanoelectrospray to a Thermo LTQ Orbitrap mass spectrometer. Protein identification was performed using PEAKS and MaxQuant software with a target 1% false-discovery rate [10,11].
Statistical analysis
Mean, standard deviation (SD) and percent coefficient of variation (%CV) were calculated for values obtained from each assay. Deming regression was performed for the 2-method comparison. All analysis and graphs were prepared using GraphPad Prism (GraphPad Software, Inc., La Jolla, CA).
Results
Of 1,094 blood samples collected, 1,072 (98%) were analyzed using both the Bio-Plex assay and the VIDAS PCT assay. We tested 760 samples from adults, and 334 samples were from pediatric patients. For 22 samples, there was insufficient volume for analysis with both assays and excluded from the analysis.
Standard curves very highly variable across test runs
The package insert for the Bio-Plex PCT assay indicates that the working assay range for PCT is 0.014 to 10 ng/mL. The standard curve for PCT for the specific lot used in this study ranged from 0.048 to 794.0 ng/mL over 8 points. For each standard curve point, values had to be 70-130% of the expected value to be acceptable. The Bio-Plex Manager software and the Five-Parametric Logistic (5PL) were used for curve fitting and data analysis. The manufacturer recommends that at least 6 of the 8 points fall within the 70-130% range for the 5PL. The Bio-Plex Manager software’s automate curve optimization feature was utilized to select the best curve for each assay run. None of the 34 runs produced a standard curve with acceptable results for all 8 points as determined by the Bio-Plex Manager software. Four of the runs (11.8%) missed the lowest point of the curve. For these runs, the seventh point on the curve was the lower limit, which has an expected PCT value of 0.194 ng/mL. Twenty-five runs (73.5%) missed the lowest and second-lowest points on the curve, with the sixth point on the curve being the lowest limit. The sixth point has an expected PCT value of 0.76 ng/mL. Finally, for five runs (14.7%), the three lowest points on the curve did not fall within the acceptable range, with the fifth point on the curve being the lowest limit. The fifth point on the curve has a target value of 3.103 ng/mL. The mean of the lowest measureable value across the 34 runs, was 0.999 ng/mL, with a standard deviation of 0.757. However, this had limited impact as only 20 samples fell outside of the curve (the analytical range of the Bio-Plex assay). All samples that fell outside of the standard curve fell below the lowest measurable point of the curve for that assay run. Samples that fell below the curve for the Bio-Plex assay were assigned the value of the lowest acceptable point for the curve calculated for PCT in that assay run. This ranged from 0.22 to 2.65 ng/mL.
The VIDAS PCT assay has a working assay range of 0.05 to 200 ng/mL. Sixty-five (n=65) samples fell outside of the analytical range of the VIDAS PCT assay. Fifty-one (n=51) of these samples produced a result of less than 0.05 ng/mL. The remaining 14 produced a result of greater than 200 ng/mL. There was poor correlation between the VIDAS PCT assay and BioPlex PCT assay (Figure 1A). An additional analysis was performed focused on the clinically relevant range for PCT and sepsis. This was done by analyzing samples that produced a result of <10 ng/mL with the VIDAS PCT assay. A total of 855 samples (79.8% of samples analyzed with both assays) met these criteria. When comparing the VIDAS PCT assay versus the Bio-Plex Pro assay in this clinically relevant range, there was also poor correlation between the two assays (Figure 1B).
Figure 1.
Correlation of PCT results between the VIDAS and Bio-Plex PCT assay. A: All samples included (n=1072). B: Samples analyzed just in the clinically relevant range, PCT (<10 ng/mL), as determined by the VIDAS PCT assay. Dark line-Deming regression.
Two serum samples, designated Sample A and Sample B, were used to assess precision with the VIDAS PCT assay and Bio-Plex assay. These two samples were run 22 times using the VIDAS PCT assay and 17 times using the Bio-Plex PCT assay. Sample A with the VIDAS PCT assay, had a mean of 0.30 ng/mL with a coefficient of variation (%CV) of 9.11. Sample A with the Bio-Plex assay had a mean of 6.69 ng/mL, with a %CV of 42.31. Sample B with the VIDAS PCT assay had a mean of 7.93 ng/mL, and a %CV of 5.64. Sample B with the Bio-Plex assay had a mean of 6.07, and a %CV of 36.57 (Figure 2). For both samples A and B, the Bio-Plex PCT assay was significantly less precise.
Figure 2.
Analysis of precision for two sample A & B for the Bio-Plex (n=17) and VIDAS (n=22) PCT assays. For sample A: the Bio-Plex assay was much less precise than the VIDAS assays. The means ± SD were 6.69 ± 2.83 and 0.30 ± 0.028 ng/mL, and %CV of 42.31 and 9.11, respectively. For sample B: the Bio-Plex assay was much less precise than the VIDAS assay. The means were 6.07 ± 2.22 and 7.93 ± 0.45 ng/mL, and %CV of 36.57 and 5.64 respectively.
To determine whether this lack of precision was an inherent feature of the Bio-Plex platform, we compared the precision of Bio-Plex results for CRP. CRP results for samples A and B were examined for the same 17 runs as were previously used to determine PCT precision. Sample A had a mean of 1.946 mg/dL for CRP with a %CV of 16.24. Sample B had a mean of 62.66 mg/dL for CRP, %CV of 14.55 (Figure 3). While clinical assays often have a lower %CV than that found for CRP with the Bio-Plex assay, the precision of CRP is within an acceptable range for a research use only assay, as opposed to the precision data generated for PCT.
Figure 3.
Analysis of precision for two samples, A & B, with the Bio-Plex PCT assay for CRP (n=17). Sample A: had a mean of 1.95 ± 0.32 mg/L, with a standard deviation and %CV of 16.24. Sample B: had a mean of 62.66 ± 9.12 mg/L, with a %CV of 14.55.
In addition, seventy-two (n=72) samples were selected to compare the Bio-Plex assay hsCRP component vs the hsCRP assay on the Olympus AU680 (Beckman) instrument used by the Hospital of University of Pennsylvania Chemistry Laboratory. A strong correlation was identified with an r value of 0.996 (Figure 4). The FDA cleared Olympus AU 680 hsCRP assay yields precision data with a %CV of less than 5%.
Figure 4.
Correlation of the Bio-Plex and the Olympus AU680 (Beckman) instrument assays for hsCRP. A strong correlation (Pearsons’s r=0.966) was found between the two assays (n=72).
In the above experiments, Bio-Plex CRP results were significantly more precise than Bio-Plex PCT results, and correlated strongly with another independent FDA cleared hsCRP assay, suggesting that the PCT performance is not solely a function of the Bio-Plex platform itself.
Both the VIDAS and Bio-Plex PCT assays had a strong correlation with the expected values of the MicroControls Linearity Kit (r=0.996 and 0.991 respectively) (Figure 5A-C).
Figure 5.
Correlation of the Linearity FD Procalcitonin bioMerieux VIDAS and miniVIDAS validation kit from Audit MicroControls (AMC PCT), which includes five distinct levels of PCT with the VIDAS and Bio-Plex assay for PCT. A: The VIDAS PCT assay produced a strong correlation (Pearson’s r=0.996) with the target values included in the Audit MicroControls kit. B: The Bio-Plex PCT assay produced a strong correlation (r=0.991) with the target values included in the Audit MicroControls kit. C: When comparing the results of the VIDAS and Bio-Plex assays for PCT with the Audit MicroControls kit, a strong correlation was found (r=0.977).
Note that for the sample of the highest expected value, 185 ng/mL, and a result above the measurable range for the VIDAS PCT assay was calculated (>200 ng/mL). For the calculated r value, the sample was given a value of 200 ng/mL for the VIDAS PCT assay. If we remove this point from the calculations (so that the analysis compares 4 measurement points instead of 5), an r value of 0.9978 is calculated.
There was also a strong correlation when comparing the VIDAS PCT vs. the Bio-Plex assay results with the Audit MicroControls linearity kit, with assigning a value of 200 ng/mL to the sample with the highest expected value for the VIDAS PCT Assay (r=0.977) (Figure 5C).
A series of mass spectroscopy experiments were conducted to determine what human serum proteins bind to the bead-bound capture antibodies of the Bio-Plex assay. Bio-Plex bead pull down experiments did not recover identifiable PCT in any of these experiments.
Discussion
Clinical/translational research studies focused on emerging biomarkers are often dependent on the availability of assays that are designated for research-use only. Given the scientific importance of these studies, it is critical that assays be appropriately validated to ensure the reliability of published data. Indeed, other investigators have identified research-use only kits that have been subsequently shown to measure analytes other than those that are indicated by the manufacturer [12,13]. In this study, we provide an example of this phenomenon that appears to be dependent on the sample matrix, suggesting that research validation, like clinical validation, should be performed with matrix-appropriate samples.
Our data demonstrate that the Bio-Plex PCT assay is unacceptable for measuring PCT in our study samples. Bio-Plex PCT showed poor correlation with an FDA cleared test kit, as well as poor precision characteristics. In comparison, the Bio-Plex hsCRP in the same test kit showed excellent correlation with the laboratory FDA cleared hsCRP assay. The strong correlation found for hsCRP increases the certainty that the poor result for PCT is an issue with the Bio-Plex PCT assay reagents, and is not experimental or procedural related.
The MicroControls Linearity kit results appear to indicate that the Bio-Plex assay is able to bind and quantify PCT. However, when using human serum, the quantification of PCT by the Bio-Rad assay is very poor, especially in the clinically relevant range for the diagnosis of sepsis. These results suggest that the sample matrix, i.e. serum, may contain undefined components affecting the performance characteristics of the assay. Alternatively, the form of PCT utilized in the calibrator kit may not reflect the endogenous form of the protein found in human samples.
When analyzing human serum, the Bio-Plex assay does produce a stronger correlation at higher concentrations. There is the potential that the specificity for PCT of both the capture and quantitative PCT antibodies of the Bio-Plex assay are low, and that they have affinity for another protein in human sera that is competitively binding. When the human serum sample has high concentrations of PCT, PCT is able to outcompete the proposed protein and is measured more accurately leading to a better correlation.
The PCT component of the Bio-Plex assay may also be affected by poor calibration. Indeed, the clinically relevant range is at the bottom of the standard curve, and there was repeated difficulty in achieving acceptable results of the lower points of the standard curve, exacerbating this issue. None of the 34 assay runs produced a standard curve with the lowest point, with an expected PCT value of 0.48 ng/mL. Across the 34 runs, the mean of the lowest point of the standard curve and subsequent lower limit, was 0.999 ng/mL. This is much higher than the clinically relevant cut-off point for PCT of 0.05 ng/mL, where a result below this value indicates a low likelihood of sepsis. The standard curve of the Bio-Plex assay not encompassing the clinically relevant range for PCT indicates that there is a fundamental design flaw for the assay.
We were not able to recover detectable PCT from the Bio-Plex beads after incubation of beads with a variety of concentrations of PCT in serum samples using mass spectrometry analysis. While these experiments were only preliminary, the results support the concept that interfering components or differences in the form of protein present in human serum account for the poor performance characteristics of the Bio-Plex PCT assay.
Our study is not the first to demonstrate issues with research use only bio-analytic test kits. Prassas et al [11] found that a commercial assay for detecting zona pellucida-like domains protein 1 (CUZD1) detected a different protein, CA125. Additional concerns about a commercial assay to detect soluble hemojuvelin were raised by Gutierrez et al [10]. It should be noted that we have been unable to find any previous reports specifically on procalcitonin research assay accuracy.
It is also worth noting that even FDA-cleared assays suffer from known problems in standardization and/or harmonization [14], and thus careful documentation of assay details will be required to ensure that biomarker values are comparable across studies.
Finally, we found that a commercial, research use only assay for measuring procalcitonin was not found to be acceptable for research use in our hands. Commercial biomarker assays that are not FDA cleared for clinical testing, and designated for research use only should undergo validation studies by the investigator prior to initiating large and resource intensive studies.
Acknowledgements
We thank Beverly Ptaskin and Frank Spadea for their assistance. This study was supported by a grant FOA CK11-001 from the Centers for Disease Control and Prevention Epicenters Program (E.L.), KO1-AI103-28 from the National Institutes of Allergy and Infectious Diseases (J.H.). The funding agencies had no role in the design and conduct of the study, collection, management, analysis, and interpretation of the data, or preparation, review or approval of this manuscript.
Disclosure of conflict of interest
None.
References
- 1.Rodland KD. As if biomarker discovery isn’t hard enough: the consequences of poorly characterized reagents. Clin Chem. 2014;60:290–291. doi: 10.1373/clinchem.2013.216382. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Rifai N, Watson ID, Miller WG. Commercial immunoassays in biomarkers studies: researchers beware! Clin Chem. 2012;58:1387–1388. doi: 10.1373/clinchem.2012.192351. [DOI] [PubMed] [Google Scholar]
- 3.Han JH, Nachamkin I, Coffin SE, Gerber JS, Fuchs B, Garrigan C, Han X, Bilker WB, Wise J, Tolomeo P, Lautenbach E Prevention Epicenters Program of the Centers for Disease Control and Prevention. Use of a Combination Biomarker Algorithm To Identify Medical Intensive Care Unit Patients with Suspected Sepsis at Very Low Likelihood of Bacterial Infection. Antimicrob Agents Chemother. 2015;59:6494–6500. doi: 10.1128/AAC.00958-15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Aouifi A, Piriou V, Bastien O, Blanc P, Bouvier H, Evans R, Célard M, Vandenesch F, Rousson R, Lehot JJ. Usefulness of procalcitonin for diagnosis of infection in cardiac surgical patients. Crit Care Med. 2000;28:3171–3176. doi: 10.1097/00003246-200009000-00008. [DOI] [PubMed] [Google Scholar]
- 5.Briel M, Schuetz P, Mueller B, Young J, Schild U, Nusbaumer C, Periat P, Bucher HC, Christ-Crain M. Procalcitonin-guided antibiotic use vs a standard approach for acute respiratory tract infections in primary care. Arch Intern Med. 2008;168:2000–2007. doi: 10.1001/archinte.168.18.2000. discussion 2007-2008. [DOI] [PubMed] [Google Scholar]
- 6.Christ-Crain M, Stolz D, Bingisser R, Muller C, Miedinger D, Huber PR, Zimmerli W, Harbarth S, Tamm M, Muller B. Procalcitonin guidance of antibiotic therapy in community-acquired pneumonia: a randomized trial. Am J Respir Crit Care Med. 2006;174:84–93. doi: 10.1164/rccm.200512-1922OC. [DOI] [PubMed] [Google Scholar]
- 7.de Kruif MD, Limper M, Gerritsen H, Spek CA, Brandjes DP, ten Cate H, Bossuyt PM, Reitsma PH, van Gorp EC. Additional value of procalcitonin for diagnosis of infection in patients with fever at the emergency department. Crit Care Med. 2010;38:457–463. doi: 10.1097/CCM.0b013e3181b9ec33. [DOI] [PubMed] [Google Scholar]
- 8.Enguix A, Rey C, Concha A, Medina A, Coto D, Dieguez MA. Comparison of procalcitonin with C-reactive protein and serum amyloid for the early diagnosis of bacterial sepsis in critically ill neonates and children. Intensive Care Med. 2001;27:211–215. doi: 10.1007/s001340000709. [DOI] [PubMed] [Google Scholar]
- 9.Nobre V, Harbarth S, Graf JD, Rohner P, Pugin J. Use of procalcitonin to shorten antibiotic treatment duration in septic patients: a randomized trial. Am J Respir Crit Care Med. 2008;177:498–505. doi: 10.1164/rccm.200708-1238OC. [DOI] [PubMed] [Google Scholar]
- 10.Gutierrez OM, Sun CC, Chen W, Babitt JL, Lin HY. Statement of concern about a commercial assay used to measure soluble hemojuvelin in humans. Am J Nephrol. 2012;36:332–333. doi: 10.1159/000342519. [DOI] [PubMed] [Google Scholar]
- 11.Prassas I, Brinc D, Farkona S, Leung F, Dimitromanolakis A, Chrystoja CC, Brand R, Kulasingam V, Blasutig IM, Diamandis EP. False biomarker discovery due to reactivity of a commercial ELISA for CUZD1 with cancer antigen CA125. Clin Chem. 2014;60:381–388. doi: 10.1373/clinchem.2013.215236. [DOI] [PubMed] [Google Scholar]
- 12.Cox J, Mann M. MaxQuant enables high peptide identification rates, individualized p. p.b.-range mass accuracies and proteome-wide protein quantification. Nat Biotechnol. 2008;26:1367–1372. doi: 10.1038/nbt.1511. [DOI] [PubMed] [Google Scholar]
- 13.Ma B, Zhang K, Hendrie C, Liang C, Li M, Doherty-Kirby A, Lajoie G. PEAKS: powerful software for peptide de novo sequencing by tandem mass spectrometry. Rapid Commun Mass Spectrom. 2003;17:2337–2342. doi: 10.1002/rcm.1196. [DOI] [PubMed] [Google Scholar]
- 14.Almond A, Ellis AR, Walker SW Scottish Clinical Biochemistry Managed Diagnostic Network. Current parathyroid hormone immunoassays do not adequately meet the needs of patients with chronic kidney disease. Ann Clin Biochem. 2012;49:63–67. doi: 10.1258/acb.2011.011094. [DOI] [PubMed] [Google Scholar]