Performance of InBios ZIKV Detect™ 2.0 IgM Capture ELISA in two reference laboratories compared to the original ZIKV Detect™ IgM Capture ELISA

Abstract

ZIKV Detect™ 2.0 IgM Capture ELISA (InBios International, Seattle, WA) recently replaced the ZIKV Detect™ IgM Capture ELISA and a number of significant changes have been made to the original version. This study compares data generated from the ZIKV Detect™ 2.0 IgM Capture ELISA, to data generated using the original version of the kit. The same sample sets were used in this comparison, and reference test results for these samples were used to assess sensitivity, specificity, accuracy and concordance of results across two laboratories. Average sensitivity increased from 90.4% to 92.5% with the updated kit where the increase was not statistically different, and specificity increased from 79.5% to 97.4%, a statistically-significant difference. Accuracy of the ZIKV Detect™ 2.0 IgM Capture ELISA was 89% compared to 63.9% for the original version of the kit, and agreement across the laboratories increased from 79.5% to 97.4%. With secondary dengue virus infections, specificity increased from 9.3% to 82.6% with the updated kit, primarily due to the change in interpretation criteria that no longer includes “Possible Zika positive.”

A cornerstone in Zika virus (ZIKV) diagnosis is identification of Zika-specific immunoglobulin M (IgM) antibody (Lanciotti et al., 2008). This appears within days of symptom onset and remains detectable for at least 12 weeks (Griffin et al., 2019). Tests are available commercially for the detection of ZIKV IgM antibodies, including assays based upon the envelope glycoprotein (E) and nonstructural 1 (NS1) protein (Lustig et al., 2017; Safronetz et al., 2017; Sloan et al., 2018). Generally, flavivirus IgM assays based upon E protein are highly sensitive but lack virus specificity compared to NS1-based assays due to the presence of cross-reactive immunodominant epitopes in highly-conserved regions of the fusion loop of domain II (Lai et al., 2008).

An article recently published by our group and collaborators (Basile et al., 2018) detailed the relative performance of several commercially-available kits that detect IgM to ZIKV. In that study, results were generated at three independent laboratories and compared to results generated from current reference standard tests. One kit, ZIKV Detect™ IgM Capture ELISA (InBios International, Seattle, WA) has recently been updated to the ZIKV Detect™ 2.0 IgM Capture ELISA, where a number of significant changes have been made to the original version. The study described here compares data generated with ZIKV Detect™ 2.0 IgM Capture ELISA, to previously-published results from the original version of the kit (ZIKV Detect™ IgM Capture ELISA) (Basile et al., 2018). The study was performed using two of the same sample sets employed in this previous study, and results were compared to reference tests (Beaty et al., 1995; Martin et al., 2000). The results for each version of the kit were compared to document any changes in performance and were analyzed to assess consistency across laboratories. Two laboratories: Arbovirus Diseases Branch – Diagnostic and Reference Laboratory (ADB-DRL), CDC, Fort Collins, CO, and Microbial Pathogenesis and Immune Response (MPIR) Laboratory, CDC, Atlanta, GA, participated in this comparison. In the analysis presented here, sensitivity, specificity, accuracy and concordance for the original version of the kit were recalculated using the results from the two laboratories participating in this comparative study, and thus some of the overall values differ from those reported in Basile et al. (2018).

As with the original version of the kit, the ZIKV Detect™ 2.0 IgM Capture ELISA is intended for use with serum only, requires 4 μl per sample diluted 1:100 before use, allows for a maximum of 28 samples per plate, and requires 4–5 h to perform. The kit includes recombinant ZIKV antigen, a cross-reactivity control antigen (CCA, comprising recombinant dengue virus (DENV) and West Nile virus (WNV) antigens (Chang et al., 2003; Davis et al., 2001)) and normal control antigen (NCA). The ZIKV IgM result is expressed as Immune Status Ratio (ISR) which is generated by dividing the optical density (OD)₄₅₀ of the sample reacted on ZIKV antigen by the OD₄₅₀ of the sample reacted on the CCA.

A number of changes to the kit were made between InBios ZIKV Detect™ IgM Capture ELISA, and InBios ZIKV Detect™ 2.0 IgM Capture ELISA, and these are detailed in Table 1. In addition, version 2.0 includes updates to the quality control criteria.

Table 1.

Comparison of features between InBios ZIKV Detect™ IgM Capture ELISA and InBios ZIKV Detect™ 2.0 IgM Capture ELISA.

	Kit feature
Version	Result outcomes	Threshold value for positivity	Detection system
ZIKV Detect™ IgM Capture ELISA	Presumptive Zika Positive	N/A	Mab SLE 6B6C-1 HRP
	Possible Zika Positive
	Presumptive Other Flavivirus Positive
	Negative
ZIKV Detect™ 2.0 IgM Capture ELISA	Presumptive Zika Positive	0.130 + Average OD450 on ZIKV antigen	MAB SLE 6B6C-1 followed by anti-mouse IgG HRP
	Presumptive Other Flavivirus Positivea
	Negative

^aSpecimens formerly in the Possible Zika Positive category are classified as “Presumptive Other Flavivirus Positive” in ZIKV Detect™ 2.0 IgM Capture ELISA if the CCA/NCA ratio is ≥5.0, and “Negative” if the CCA/NCA ratio is < 5.0.

ZIKV Detect™ 2.0 IgM Capture ELISA kits were provided by InBios International for this study, and kits were used according to manufacturer’s instructions for use. Residual samples from panels 1 and 2 described previously (Basile et al., 2018) were employed in this study, in accordance with CDC Institutional Review Board protocol #6773. All samples were stored at −20 °C following the former work. Panel 1 consisted of 281 serum samples including: 64 serologically-determined probable primary ZIKV infections; 47 serologically-determined dengue virus (DENV) including probable primary, secondary, and undetermined infections), 62 undifferentiated flavivirus-positive infections, 10 each of Chikungunya virus (CHIKV), WNV, and yellow fever virus (YFV) samples and 78 negative samples. Panel 1 samples were collected from patients in the U.S. during the normal course of laboratory diagnosis, and reference methods used in the comparisons were the CDC IgM-antibody capture (MAC)-ELISA using ZIKV, DENV, WNV, YFV and CHIKV antigens, as appropriate, and 90% plaque reduction neutralization test (PRNT90). Panel 2 consisted of 50 DENV and 50 ZIKV-positive samples obtained by CDC’s Dengue Branch in San Juan, Puerto Rico, of which approximately 80% were probable secondary flavivirus infections as determined by PRNT. Panel 2 samples were all convalescent specimens taken from RT-qPCR-confirmed cases, where reference results from CDC MAC-ELISA and PRNT90 were also available. The collection dates of ZIKV positive samples in panels 1 and 2 ranged from 1 to 129 days post-onset of symptoms with a median of 11 days.

Sensitivity, specificity, accuracy, and agreement (concordance) across the laboratories were calculated for the ZIKV Detect™ 2.0 IgM Capture ELISA and compared to those of the original ZIKV Detect™ IgM Capture ELISA. Details for these calculations are given in Table 2. To compare the sensitivities and specificities of the two tests, differences in paired testing results between laboratories were checked for using a chi-squared test. If a statistically significant difference in the paired testing results between laboratories was absent, the paired testing results for the laboratories were averaged and McNemar’s test was used to compare the tests. Conversely, if there was a statistically significant difference in the paired testing results between laboratories, McNemar’s test was used separately for each laboratory to compare the tests. The difference in sensitivities/specificities and 95% confidence intervals (CIs) were calculated with the adjusted Wald interval for difference in proportions of matched pairs.

Table 2.

Performance measures of the ZIKV Detect™ 2.0 IgM Capture ELISA compared to the ZIKV Detect™ IgM Capture ELISA using serum panels 1 and 2 combined.

		ZIKV Detect™ 2.0 IgM Capture ELISA							ZIKV Detect™ IgM Capture ELISA
		ADB-DRLa			MPIRb			ADB-DRL+ MPIR	ADB-DRL + MPIR
Measure	N	POS	NEG	% (95% CI)	POS	NEG	% (95% CI)	Overall % (95% CI)	Overall % (95% CI)
Sensitivity	114	105	9	92.1 (85.7, 95.8)	106	8	93.0 (86.8, 96.4)	92.5 (88.4, 95.3)	90.4 (85.8, 93.5)
Specificityc	78	2	76	97.4 (91.1, 99.3)	2	76	97.4 (91.1, 99.3)	97.4 (93.6, 99.0)	79.5 (72.5, 85.1)
Specificityd	205	11	194	94.6 (90.6, 97.0)	10	195	95.1 (91.3, 97.3)	94.9 (92.3, 96.6)	56.6 (51.7, 61.3)
Accuracye				89.0 (85.0, 92.0))			90.0 (86.1, 92.8)	89.5 (86.8, 91.7)	63.9 (60.1, 67.6)
Controls		Coefficient of variation (95% CI)
Positive		15.0 (10.8, 24.6)			16.5 (12.0, 26.4)			15.7 (12.5, 21.4)	34.3 (26.0, 51.1)
Negative		16.4 (11.8, 27.0)			14.5 (10.6, 23.1)			15.6 (12.4, 21.3)	22.9 (17.5, 33.1)
		% agreement (Fleiss’ Kappa statistic: 95% CI)
Agreement		97.4 (0.96: 0.93, 0.99)							79.5 (0.59: 0.51, 0.67)

^aArbovirus Diseases Branch-Diagnostic and Reference Laboratory.

^bMicrobial Pathogenesis and Immune Response Laboratory.

^cSpecificity denominator consists of true negatives only.

^dSpecificity denominator consists of 10 YFV, 10 WNV, 10 CHIKV, 97 DENV and 78 NEG samples.

^eAccuracy based on proportion correctly identified within the following categories: ZIKV = “Presumptive Zika Positive,” DENV and WNV = “Presumptive Other Flavivirus,” and CHIKV and NEG = “Negative” (except for one CHIKV sample that was classified as “Presumptive Other Flavivirus” due to apparent dual infection).

In addition, sensitivities and specificities for primary and secondary ZIKV and DENV infections were established. Unbiased estimates of the coefficient of variation (CV) and 95% confidence intervals (95% CI) were calculated utilizing the “MBESS” package of R software, and the Fleiss’ kappa statistic, used as a measure of inter-laboratory agreement, was calculated using the “raters” package in R software (R Core Team, 2017).

The raw data for Panels 1 and 2 generated using ZIKV Detect™ 2.0 IgM Capture ELISA and ZIKV Detect™ IgM Capture ELISA by ADB-DRL and MPIR, and the comparative reference data, are provided in Supplemental Table 1.

Positive and negative plate controls are used in the kits to validate the individual plate analyses. The CVs of the positive and negative plate control OD’s for the ZIKV Detect™ 2.0 IgM Capture ELISA were 15.7 (95% CI:12.5, 21.4) and 15.6 (95% ci: 12.4, 21.3), respectively. The corresponding values from the original version of the kit were 34.3 (95% CI:26.0, 51.1) and 22.9 (95% CI: 17.5, 33.1) (Table 2).

The inter-laboratory agreement between the ADB-DRL and MPIR laboratories using ZIKV Detect™ 2.0 IgM Capture ELISA was 97.4%, with a Fleiss’ kappa statistic (k) = 0.96 (95% CI: 0.93, 0.99) compared to 79.5% (Table 2) with k = 0.59 (95% ci: 0.51, 0.67) for the original kit (Basile et al., 2018).

Sensitivity and specificity for the ZIKV Detect™ 2.0 IgM Capture ELISA are given in Table 2. The combined sensitivity using confirmed ZIKV positives was 92.5% (95% CI: 88.4%, 95.3%) and the specificity using negative samples only as the denominator, was 97.4% (95% CI: 93.6%, 99.0%). By comparison, the original ZIKV Detect™ IgM Capture ELISA gave a combined average sensitivity and specificity of 90.4% (95% CI: 85.8%, 93.5%) and 79.5% (95% CI: 72.5%, 85.1%), respectively, for the ADB-DRL and MPIR laboratories. When specificity was calculated including negatives plus all non-ZIKV differentiated arbovirus samples, the specificity for the updated kit was 94.9% (95% CI: 92.3%, 96.6%) compared to 56.6% (95% CI: 51.7%, 61.3%) for the original version. The sensitivities of the paired testing results across laboratories were not statistically significantly different (chi-squared test p-value = 0.18). Averaging the test results across the two laboratories, the sensitivities of the two tests were not statistically significantly different (difference = 2.2%, 95% CI: −4.0%–8.3%, McNemar’s p-value = 0.58). For specificities (both when only using negative samples and with all non-ZIKV samples), the paired testing results across laboratories were not statistically significantly different (chi-squared test p-values = 0.15, 0.77 respectively). However, the specificities of the two tests were statistically significantly different in both cases (17.5% difference with 95% CI: 7.5%–27.5%, McNemar’s p-value = 0.001; 37.9% difference with 95% CI: 30.9%–44.9%, McNemar’s p-value < 0.001, respectively).

Accuracy across all the samples was 89.5% (95% CI: 86.6%, 91.7%) for ZIKV Detect™ 2.0 IgM Capture ELISA and 63.9% (95% CI:60.1%, 67.6%) for the original version (Table 2).

Some confirmed ZIKV and DENV-positives were able to be identified as probable primary or secondary infections by reference test results (Tsai et al., 2015; Vorndam and Beltran, 2002). Table 3a compares the sensitivity results from the two laboratories. Overall, the sensitivity for ZIKV primary infections was 91.5% (95% CI: 85.8%, 95.1%) using the ZIKV Detect™ 2.0 IgM Capture ELISA compared to 83.8% (95% CI: 76.9%, 89.0%) for the original. For secondary infections, the overall sensitivity was 94.2% (95% CI: 87.1%, 97.5%), compared to a 98.8% sensitivity (95% CI: 93.7%, 99.9%) with the original version. To get an estimate of specificity, results from probable primary and secondary samples were analyzed. The overall specificity of the updated kit using these samples was 88.4% (95% CI: 82.1%, 92.6%), compared to 19.9% (95% CI: 14.2%, 27.1%) with the original ZIKV Detect™ IgM Capture ELISA (Table 3b).

Table 3.

Sensitivity and specificity of InBios ZIKV Detect™ 2.0 IgM Capture ELISA using samples from probable primary and secondary infections and comparison to results from InBios ZIKV Detect™ IgM Capture ELISA.

A. Sensitivity using confirmed ZIKV positive samples			ADB-DRL			MPIR			InBios ZIKV Detect™ 2.0 IgM Capture ELISA	InBios ZIKV Detect™ IgM Capture ELISAa
Probable status	Reference tests	N	POSb	NEGc	% Sensitivity (95% CI)	POS	NEG	% Sensitivity (95% CI)	Mean sensitivity (95% CI)	Mean sensitivity (95% CI)
Primary	RT-PCR, PRNT90	7	7	0	100.0 (64.6, 100.0)	7	0	100.0 (64.6, 100.0)	100.0 (78.5, 100.0)	100.0 (78.5, 100.0)
Primary	PRNT90 only	64	57	7	89.1 (79.1, 94.6)	59	5	92.2 (83.0, 96.6)	90.6 (84.3, 94.6)	83.6 (76.2, 89.0)
Overall 1° sensitivity		71	64	7	90.1 (81.0, 95.1)	66	5	93.0 (85.6, 97.0)	91.5 (85.8, 95.1)	85.2 (78.4, 90.1)
Secondary	RT-PCR, PRNT90	43	41	2	95.3 (84.5, 98.7)	40	3	93.0 (81.4, 97.6)	94.2 (87.1, 97.5)	98.8 (93.7, 99.9)
Secondary	PRNT90 only	0	0	0		0	0
Overall 2° sensitivity		43	41	2	95.3 (84.5, 98.7)	40	3	93.0 (81.4, 97.6)	94.2 (87.1, 97.5)	98.8 (93.7, 99.9)
Overall sensitivity		114	105	9	92.1 (85.7, 95.8)	106	8	93.0 (86.8, 96.4)	92.5 (88.4, 95.3)	90.4 (85.8, 93.5)
B. Specificity using confirmed DENV positive samples			ADB-DRLa			MPIRb			InBios ZIKV Detect™ 2.0 IgM Capture ELISA	InBios ZIKV Detect™ IgM Capture ELISAc
Probable status	Reference tests	N	POSd	NEGe	% Specificity (95% CI)	POS	NEG	% Specificity (95% CI)	Mean specificity (95% CI)	Mean specificity (95% CI)
Primary	RT-PCRf, PRNT90g	13	0	13	100.0 (77.2, 100.0)	0	13	100.0 (77.2, 100.0)	100.0 (87.1, 100.0)	23.1 (11.0, 42.1)
Primary	PRNT90 only	17h	1	16	94.1 (73.0, 99.7)	1	16	94.1 (73.0, 99.7)	94.1 (80.9, 9834)	44.1 (28.9, 60.5)
Overall 1° specificity		30	1	29	96.7 (83.3, 99.8)	1	29	96.7 (83.3, 99.8)	96.7 (88.6, 99.1)	35.0 (24.2, 47.6)
Secondary	RT-PCR, PRNT90	37	8	29	78.4 (62.8, 88.6)	7	30	81.1 (65.8, 90.5)	79.7 (69.2, 87.3)	4.1 (1.4, 11.3)
Secondary	PRNT90 only	6	0	6	100.0 (61.0, 100.0)	0	6	100.0 (61.0, 100.0)	100.0 (75.8, 100.0)	41.7 (19.3, 68.0)
Overall 2° specificity		43	8	35	81.4 (67.4, 90.3)	7	36	83.7 (70.0, 91.9)	82.6 (73.2, 89.1)	9.3 (4.8, 17.3)
Overall specificity		73	9	64	87.7 (78.2, 93.4)	8	65	89.0 (79.8, 94.3)	88.4 (82.1, 92.6)	19.9 (14.2, 27.1)

^aArbovirus Diseases Branch-Diagnostic and Reference Laboratory.

^bMicrobial Pathogenesis and Immune Response Laboratory.

^cNumbers reported here include data from both ADB-DRL and MPIR and thus differ somewhat from the data reported in Table 5 of Basile et al. (2018).

^dPresumptive Zika Positive classified as POS. Note that in Table 5 of Basile et al. (2018), both Presumptive Zika Positive and Possible Zika positive were classified as POS.

^ePresumptive Other Flavivirus Positive and Negative classified as NEG.

^fReverse transcriptase polymerase chain reaction.

^g90% Plaque-reduction neutralization test.

^hOne sample was excluded from this analysis compared to the previously-published analysis.

Reactivities of ZIKV-positive samples at a range of dates post symptom onset (1–129) observed in the original version of the kit are illustrated in Figure 1a of Basile et al. (2018); 11 false negative results were seen across days 1–11 post symptom onset. The ZIKV Detect™ 2.0 IgM Capture ELISA had six false negative results on days 1–11 post symptom onset at ADB DRL, and six false negative results across days 1–14 at the MPIR laboratory.

The average sensitivity of the ZIKV Detect™ 2.0 IgM Capture ELISA improved by 2 percentage points compared to the ZIKV Detect™ IgM Capture ELISA, although this was not a statistically significant increase. The altered detection system used in ZIKV Detect™ 2.0 IgM Capture ELISA, which originally consisted of HRP-conjugated monoclonal antibody St. Louis encephalitis virus 6B6C-1 (Tsai et al., 1987) could be responsible for this minor difference. More importantly, this alteration could result in improved lot-to-lot consistency due to the difficulties of manufacturing horseradish peroxidase conjugates. The true ZIKV-positive samples that did not result in “Presumptive Zika positive” test interpretations in the ZIKV Detect™ 2.0 IgM Capture ELISA were mostly obtained in the first few days after onset of symptoms.

Specificity was greatly improved in the new version of the kit, especially with DENV-positives. This was largely due to the elimination of the original “Possible Zika Positive” category, where many of the samples that previously gave this result were categorized as “Presumptive Other Flavivirus Positive” using version 2.0 of the kit. The presence of the CCA has been leveraged more effectively in the ZIKV Detect™ 2.0 IgM Capture ELISA to eliminate these false positives. Another factor influencing the increase in specificity was the introduction of the ZIKV antigen OD threshold value, which must be exceeded in order for a sample to be classified as “Presumptive Zika Positive.” This addresses the misclassification of samples when the ZIKV antigen OD₄₅₀ is below that which is typically seen in a normal population of negatives, but which have ISR’s that cause false positive results. The original version of the kit gave 6 false positives due to misclassification as “Possible Zika Positive” at the ADB-DRL and 4 at the MPIR Laboratory. All false positives in this category were resolved as “Negative” in version 2.0 of the kit.

This study included samples with undifferentiated reference results. Sixty-two samples were reference-test classified as “flavivirus”, when the CDC Zika virus diagnostic guidelines were applied https://www.cdc.gov/zika/pdfs/testing_algorithm.pdf. In the guidelines, any sample with PRNT90-positive results to both ZIKV and DENV are classified as undifferentiated flavivirus, even if there is at least a 4-fold difference between the titers. For these samples, all but 9 resulted as “Presumptive Zika Positive” in the ZIKV Detect™ 2.0 IgM Capture ELISA at both labs. This test is intended as a screening assay and therefore, these samples would reflex to PRNT90 in a diagnostic scenario. The data shown in Table 3 illustrate excellent specificity of the ZIKV Detect™ 2.0 IgM Capture ELISA when applied to primary DENV infections, but a reduced specificity among RT-qPCR-confirmed secondary DENV infections. This underscores the importance of obtaining further diagnostic results beyond the screening assay due to the potential for misleading IgM results in secondary flavivirus infections.

A noticeable improvement in agreement of results across the two laboratories was seen in the ZIKV Detect™ 2.0 IgM Capture ELISA compared to the former version. The lack of consistency across laboratories previously seen as a source of concern appears to have been addressed in the updated version of the kit.

One of the two laboratories in this study had to reject two plate runs due to QC failures, and such failures were seen for a few plates in the previous version of the kit, despite careful attention to timing and technique. These limited data suggest that QC failures may be expected to occur in around 10% of runs.

Limitations to this study include the lack of RT-qPCR confirmation for Panel 1 samples, and the non-inclusion of paired serum samples. The study did not include any poor-quality samples, and time had elapsed between the analysis of the original and updated kits, where an additional freeze-thaw cycle of the samples occurred.

Overall, this comparative analysis of the ZIKV Detect™ 2.0 IgM Capture ELISA and its predecessor indicates that considerable improvement has been made to performance in terms of specificity, accuracy and agreement across laboratories for the updated kit, in addition to a small but statistically non-significant increase in sensitivity. These data suggest that the ZIKV Detect™ 2.0 IgM Capture ELISA may be a reliable Zika IgM diagnostic test when used according to regionally-adopted testing algorithms.