Enzyme-Linked Immunosorbent Assay for the Diagnosis of Melioidosis: Better Than We Thought

Abstract

We used Bayesian latent-class models to generate receiver operating characteristic curves and to revise the cutoff values for an enzyme-linked immosorbent assay that has been developed previously for melioidosis. The new cutoff was unbiased towards misclassification caused by an imperfect gold standard and resulted in an increase in both sensitivity (from 66.4% to 80.2%) and specificity (82.1% and 95.0%).

Serological tests represent the basis for the laboratory diagnosis of many infectious diseases. Tests that are quantitative by design are often reported to the clinician as a positive or negative result. This is achieved by defining a cutoff value, which is determined during assay development by comparing the results of the serological test of interest with that of a reference test. This assumes that the reference test has perfect sensitivity and specificity, but if this is not the case, then estimates of sensitivity and specificity for the test under evaluation will be biased and the selected cutoff value suboptimal [1].

Culture represents the gold standard test for the diagnosis of melioidosis, a serious infection due to gram-negative bacilli caused by the biothreat organism Burkholderia pseudomallei. Culture was used as the reference test during the development of an enzyme-linked immuosorbent assay (ELISA)–based serological test for melioidosis [2, 3], but a recent study reported that the true sensitivity of culture was only 60% [4]. We hypothesized that the current ELISA cutoff is suboptimal, and we undertook this study to improve its performance by the selection of unbiased cutoff values using Bayesian latent-class models (LCMs) and the generation of unbiased receiver operating characteristic (ROC) curves.

PATIENTS AND METHODS

The ELISA data analyzed in this study were generated during 2 previously published prospective clinical evaluations of diagnostic laboratory tests for melioidosis [2, 3]. The same patient cohort was used in both studies. In brief, patients were recruited during the period June through October 2004 at the Sappasithiprasong Hospital (Ubon Ratchathani, Thailand) [3]. Inclusion criteria were the presence of a fever (temperature, >38.5°C) in patients aged ≥14 years who were suspected to have melioidosis in the absence of clinical or laboratory findings suggestive of an alternative diagnosis. Patients underwent sampling for culture (blood was obtained from all patients, and urine, pus, respiratory secretions, throat swab, and swabs from surface lesions were obtained as available or clinically appropriate). Isolation of B. pseudomallei from any clinical specimen was defined as culture-positive melioidosis. Serological testing included the indirect hemagglutination test (IHA), immunoglobulin (Ig) M immunochromogenic cassette test (ICT), IgG ICT, and the ELISA being reevaluated here [2, 3]. The serum used in the serological tests was taken at the time of hospital admission. Five different B. pseudomallei antigen preparations were used in the ELISA: affinity-purified antigen, crude organism, lipopolysaccharide (LPS), exopolysaccharide (EPS), and a combination of LPS and EPS [2]. Of the 322 patients recruited [3], 2 were enrolled twice. In both cases, only data from the first enrollment was used in this study.

The results of the 5 ELISA tests were analyzed in 3 ways. First, culture was assumed to be a perfect reference test (ie, with 100% sensitivity and 100% specificity). ROC curves were generated from the sensitivities and specificities of the ELISA tests at all possible cutoff values. Optimal cutoff values, area under the ROC curve (AUROCC), sensitivities, specificities, and overall accuracies (the proportion of patients correctly classified) for the 5 ELISAs were calculated with exact 95% confidence intervals (CIs) using the Stata statistical software package, version 11 (Stata Corp).

Second, we applied Bayesian LCMs to the original dataset using methodology described elsewhere [4]. A class of random effect models described by Dendukuri and Joseph [5] was used to take account of conditional dependence between serological tests. In brief, the LCM calculated prevalence, as well as the sensitivities and specificities of 5 diagnostic tests (culture, IHA, IgM ICT, IgG ICT, and ELISA using affinity-purified antigen) on the basis of the possibility that either an infected or a noninfected patient could have any possible combination of binary test results. The model assumed that, in a given patient, the results of all serological tests were correlated. The model did not assume a single perfect test but regarded each test as imperfect in diagnosing the true disease status (infected or not infected). The true disease status of the patient population was defined on the basis of overall prevalence. All parameters were estimated with 95% credible intervals using OpenBugs, version 3.1.1 (http://www.openbugs.info/w/) [6]. For Bayesian LCMs, specificity of culture was fixed at 100%, and we assumed that we knew nothing about the unknown parameters (eg, prevalence, sensitivities of all 5 tests, and specificities of all 4 serological tests). This analysis was performed sequentially for each of the 5 ELISA antigens.

Third, unbiased ROC curves were developed for the ELISA, and the analysis, using Bayesian LCM, was performed sequentially for each of the possible cutoff values for the 5 ELISAs. The sensitivity and specificity of the ELISA estimated from each cutoff value was used to plot the ROC curve, and cutoff values with the highest utility (highest sensitivity with specificity >95%) were selected as the optimal cutoff value. This unbiased ROC curve method has been used by Nielsen et al [7] and further evaluated by Zhou et al [1], Choi et al [8], and Wang et al [9]. Bayesian P value, deviance information criteria, and the Akaike information criterion were used to recheck the fitness of the models [10].

RESULTS

A total of 320 patients with suspected melioidosis were included in the study. Median age was 54 years (interquartile range, 43–65 years), and 161 patients (50%) were male. One hundred nineteen of 320 patients were culture positive for B. pseudomallei, a prevalence of 37.2% (95% CI, 31.9–42.7). IHA, IgM ICT, and IgG ICT results were positive for 158, 200 and 206 patients. The results of ELISA using affinity-purified antigen, crude organism, LPS, EPS, and a combination of LPS and EPS as antigens were positive for 152, 154, 115, 128, and 141 patients, respectively.

The sensitivities, specificities, accuracies, and AUROCCs of the 5 ELISA tests based on the assumption that culture was a perfect reference test are shown in Table 1. Cutoff values with the highest accuracies were selected. Using these cutoff values (Table 1), none of the 5 ELISA had a sensitivity, specificity, or accuracy of >85%, and all estimates of AUROCC were lower than 0.80.

Table 1.

Optimal Cutoff Values, Sensitivities, Specificities, Accuracies, and Area Under the Receiver-Operating Characteristic (ROC) Curves of 5 Enzyme-Linked Immunosorbent Assays (ELISAs) for Diagnosis of Melioidosis

ELISA	Conventional method, assuming that the gold standard (culture) was perfect, avalue (95% CI)	Bayesian LCM, using cutoff values determined by the conventional method, bvalue (95% CrI)	Bayesian LCMs, using all possible cutoff values,bvalue (95% CrI)
Using affinity-purified antigen
Optimal cutoff value	0.386	0.386	0.263
Sensitivity, %	82.4 (75.4–89.3)	74.6 (67.1–81.8)	79.9 (72.8–86.3)
Specificity, %	73.1 (67.0–79.3)	97.9 (92.4–99.9)	96.6 (90.1–99.8)
Accuracy, %	76.6 (71.5–81.1)	83.8 (78.4–88.1)	86.6 (81.9–90.3)
Area under the ROC curve	0.78 (0.73–0.82)	…	0.91
Using crude B. pseudomallei
Optimal cutoff value	0.482	0.482	0.381
Sensitivity, %	80.7 (72.4–87.3)	75.3 (68.0–82.1)	80.5 (73.6–86.7)
Specificity, %	71.1 (64.4–77.3)	97.9 (92.5–99.9)	95.8 (89.1–99.7)
Accuracy, %	74.7 (69.6–79.4)	84.1 (79.1–87.8)	86.6 (82.2–90.3)
Area under the ROC curve	0.76 (0.71–0.81)	…	0.90
Using LPS
Optimal cutoff value	0.683	0.683	0.269
Sensitivity, %	66.4 (57.2–74.8)	57.4 (50.3–64.8)	80.2 (73.0–86.6)
Specificity, %	82.1 (76.1–87.1)	98.9 (94.8–100)	95.0 (87.5–99.6)
Accuracy, %	76.3 (71.2–80.8)	72.8 (66.9–77.8)	85.9 (81.3–90.0)
Area under the ROC curve	0.74 (0.69–0.79)	…	0.89
Using EPS
Optimal cutoff value	0.133	0.133	0.157
Sensitivity, %	63.9 (54.6–72.5)	56.9 (50.2–64.7)	51.8 (45.3–58.5)
Specificity, %	74.1 (67.5–80.0)	93.4 (86.4–98.1)	95.8 (89.1–99.4)
Accuracy, %	70.3 (65.0–75.3)	69.4 (64.1–73.8)	66.6 (60.9–71.3)
Area under the ROC curve	0.69 (0.64–0.74)	…	0.81
Using a combination of LPS and EPS
Optimal cutoff value	0.434	0.434	0.267
Sensitivity, %	76.5 (67.8–83.8)	69.4 (62.1–76.5)	77.6 (70.7–83.9)
Specificity, %	75.1 (68.6–80.9)	98.5 (94.0–99.9)	95.2 (88.3–99.4)
Accuracy, %	75.6 (70.5–80.2)	80.6 (75.3–84.7)	84.4 (79.7–88.1)
Area under the ROC curve	0.76 (0.71–0.81)	…	0.90

NOTE. Diagnosis of melioidosis was determined by (1) conventional method based on an assumption that culture was a perfect reference test (ie, that it had 100% sensitivity and 100% specificity), (2) Bayesian latent-class model (LCM) using the cutoff values determined by the conventional method, and (3) Bayesian LCMs using all possible cutoff values. CI, confidence interval; CrI, credible interval; EPS, exopolysaccharide; LPS, lipopolysaccharide.

^aEstimates are biased towards disease classification by the imperfect reference test.

^bEstimates are unbiased because they are based on the true status of patients (infected vs uninfected) rather than the imperfect reference test.

The sensitivities, specificities, and accuracies of all 5 ELISA tests calculated using Bayesian LCMs were markedly different from those derived assuming that culture was perfect (Table 1), the latter overestimating sensitivities and underestimating specificities of the ELISA. This was because the true sensitivity of culture was estimated to be very low (60.2%; 95% credible interval, 51.7%–68.5%) for the diagnosis of melioidosis; thus, a number of “infected” but culture-negative patients had been misclassified as “noninfected.” Although these new estimates for ELISA accuracies were unbiased, the cutoff values used might not be optimal because they were selected using conventional ROC curves (Figure 1), in which culture is assumed to be a perfect reference test. To address this, we used Bayesian LCMs with all possible cut-offs and plotted the resulting unbiased ROC curves (Figure 1). This gave estimates of AUROCC that were all >0.89, with the exception of the ELISA using only EPS as antigen. The optimum cutoff values of the 5 ELISA's (excluding EPS) were markedly lower than the previous estimates based on the assumption that culture is perfect (for example, 0.269 vs 0.683 for ELISA using LPS) (Table 1). Using these new optimal cutoff values, the accuracies of these 4 ELISAs were markedly higher than the previous estimates based on the culture is perfect assumption (for example, 85.9 vs 72.8 for ELISA using LPS) (Table 1).

Figure 1.

Unbiased receiver operating characteristic (ROC) curves using Bayesian latent class models (LCMs) with all possible cutoff values (solid line with dot,each cutoff value), compared with the conventional ROC curve based on the assumption that culture was perfect (solid line without dots) for the 5 enzyme-linked immunosorbent assays (ELISAs) for the diagnosis of melioidosis. The 5 ELISA tests used 5 different Burkholderia. pseudomallei antigens, including affinity-purified B. pseudomallei antigen (A), crude organism (B), lipopolysaccharide (C), exopolysaccharide (D), and a combination of lipopolysaccharide and exopolysaccharide (E) as antigen.

DISCUSSION

We have described an unbiased approach using Bayesian LCMs to define optimal cutoff values for quantitative diagnostic tests. The new cutoff values for an ELISA used for the diagnosis of melioidosis were much lower than reported previously, and Bayesian LCMs estimated that the overall accuracy and utility of the ELISA were much higher than that reported previously [2]. Compared with culture, the ELISA test using LPS as the antigen had a sensitivity and specificity of 66.4% and 82.1%, respectively, and thus had no clinical utility. After recalculation using Bayesian LCM with all possible cutoff values, the sensitivity and specificity of this ELISA were 80.2% and 95.0%, respectively, representing a test that could be further developed for use in the clinical setting with a reasonably high degree of accuracy. These sensitivity and specificity values are unbiased, because they are based on the true status of patients (infected vs uninfected), rather than an imperfect reference test.

The true accuracy of ELISA using only EPS as antigen was very low, and the true accuracies of 4 ELISA tests using LPS as one of the presenting antigens were very high. This suggests that LPS, but not EPS, is a good candidate for the further development of a clinically useful, serology-based diagnostic test for melioidosis.

We consider it likely that the biased selection of cutoff values is not specific for melioidosis, but rather a wider problem affecting many serological tests in clinical use for diagnosing infectious diseases, because the reference tests are rarely if ever perfect. Application of the methodology described here would lead to a broader understanding of the utility of the reference tests in the evaluation of quantitative diagnostic tests. This could lead to change in the diagnostic process and clinical practice in many infectious diseases.