Abstract
The enzyme-linked immunosorbent assay (ELISA) has been proposed as an alternative serologic diagnostic test to the indirect immunofluorescence assay (IFA) for scrub typhus. Here, we systematically determine the optimal sample dilution and cutoff optical density (OD) and estimate the accuracy of IgM ELISA using Bayesian latent class models (LCMs). Data from 135 patients with undifferentiated fever were reevaluated using Bayesian LCMs. Every patient was evaluated for the presence of an eschar and tested with a blood culture for Orientia tsutsugamushi, three different PCR assays, and an IgM IFA. The IgM ELISA was performed for every sample at sample dilutions from 1:100 to 1:102,400 using crude whole-cell antigens of the Karp, Kato, and Gilliam strains of O. tsutsugamushi developed by the Naval Medical Research Center. We used Bayesian LCMs to generate unbiased receiver operating characteristic curves and found that the sample dilution of 1:400 was optimal for the IgM ELISA. With the optimal cutoff OD of 1.474 at a sample dilution of 1:400, the IgM ELISA had a sensitivity of 85.7% (95% credible interval [CrI], 77.4% to 86.7%) and a specificity of 98.1% (95% CrI, 97.2% to 100%) using paired samples. For the ELISA, the OD could be determined objectively and quickly, in contrast to the reading of IFA slides, which was both subjective and labor-intensive. The IgM ELISA for scrub typhus has high diagnostic accuracy and is less subjective than the IgM IFA. We suggest that the IgM ELISA may be used as an alternative reference test to the IgM IFA for the serological diagnosis of scrub typhus.
INTRODUCTION
Scrub typhus and murine typhus are important causes of acute febrile illness in Thailand and Laos (1–6). Diagnosis of scrub typhus still relies on serology due to the convenience of sample collection; the relative simplicity of the assays; and the expense, complexities, and deficiencies in sensitivity of other modalities (5, 7, 8). The indirect immunofluorescence assay (IFA) is the serological reference standard despite being inconsistent in terms of antigen composition, diagnostic cutoff (9), and repeatability (10). Enzyme-linked immunosorbent assays (ELISAs) offer a number of advantages over IFA, particularly in terms of simplicity, reading objectivity, and sample throughput. Use of ELISA for acute scrub typhus diagnosis has been previously evaluated (11, 12); however, its optimal sample dilution and cutoff optical density (OD) have not been determined.
Defining the cutoff value of a new serological test is conventionally done by comparing the results of the serological test of interest with the results of a reference test. However, this conventional approach assumes that the reference test has perfect sensitivity and specificity; if this is not the case, then estimates of sensitivity and specificity for the test under evaluation will be biased and the selected cutoff value suboptimal (13). Bayesian latent class models (LCMs) are increasingly used to determine the optimal cutoff value and to estimate the accuracy of a diagnostic test, since they do not need to assume that the accuracy of reference tests is perfect (14–17). We recently used Bayesian LCMs to reanalyze our existing data set from a previously published prospective study (18) and showed that an IgM IFA with a cutoff titer of ≥1:12,800 in admission samples or a 4-fold rise to ≥1:200 in convalescent/discharge samples had low sensitivity and specificity (70.0% and 83.8%, respectively) (19) and that optimal cutoff titers for IgM IFA of ≥1:3,200 in admission specimens or a ≥4-fold rise to ≥1:3,200 in convalescent/discharge specimens provided higher sensitivity and specificity (81.6% and 100%, respectively) (20).
In this study, we used Bayesian LCMs and blood samples collected from our previously published prospective study (18) to systematically determine the optimal sample dilution and optimal cutoff OD and to examine the accuracy of the Naval Medical Research Center (NMRC) “in-house” scrub typhus IgM ELISA.
MATERIALS AND METHODS
Study patients and blood samples.
The data set and blood samples used in this study were from patients recruited into a prospective study of acute febrile illness from August 2007 to August 2008 in Chiangrai, Thailand (18). In brief, 161 patients over 15 years old presenting with acute fever of less than 2 weeks, with three negative malaria blood smears and no evidence of a primary focus of infection, were recruited into the study (18). Every patient was examined for the presence of an eschar. Admission blood samples were collected and tested using in vitro culture for Orientia tsutsugamushi, a nested-PCR (nPCR) assay targeting the 56-kDa gene, a 47-kDa gene-based quantitative real-time PCR (qPCR) assay, a groEL-based qPCR assay, and an IgM IFA. Convalescent-phase or hospital discharge (convalescent/discharge) blood samples were also collected and tested by IgM IFA. All the blood samples were kept at −80°C and tested using the IgM ELISA in 2014.
Ethical statement.
Ethical approval for the prospective study was obtained from the ethical committee of Chiangrai Hospital; the Ministry of Public Health, Thailand; and the Oxford Tropical Research Ethics Committee, United Kingdom. Signed written informed consent was obtained from every patient before sample collection (18).
Diagnostic tests.
In vitro isolation of O. tsutsugamushi (culture) (21) and the 56-kDa gene nPCR assay (22), 47-kDa-based qPCR assay (23), and groEL-based qPCR assay (24) were performed as described previously. The IgM IFA was performed using pooled Karp, Kato, and Gilliam reference strain O. tsutsugamushi antigens as described previously (18). In short, IgM antibodies were detected using IFA slides produced by the Australian Rickettsial Reference Laboratory (ARRL) (Geelong, Australia). Patient sera were serially 2-fold diluted from 1:100 to 1:25,600, and the endpoint was determined as the highest titer displaying specific fluorescence (18).
The IgM ELISA was essentially the same as that previously described by Suwanabun et al. (11). All ELISAs were performed at the Mahidol-Oxford Tropical Medicine Research Unit (MORU), Bangkok, Thailand. In brief, whole-cell antigen lysates of Karp, Kato, and Gilliam reference strains of O. tsutsugamushi and mock-infected cell lysate produced at the Viral and Rickettsial Diseases Department of the NMRC, Silver Spring, Maryland, USA (11, 12), were used as ELISA antigens. The scrub typhus antigen quality was assessed for each strain of O. tsutsugamushi Karp, Kato, and Gilliam for identity (qPCR and sequencing) and adventitious contamination (blind culture to blood agar plates, thioglycolate broth, and qPCR for mycoplasma). To standardize the antigen preparations, each strain was assessed for reactivity with reference scrub typhus-positive and -negative human serum controls at the NMRC. Following assessment, the reactivity of each antigen strain was compared to those of previous antigen batches, and the concentration was adjusted to permit the use of the component antigens at a standard dilution of 1:1,000 in coating buffer. In this evaluation, a single batch of component antigens was used at the MORU. All ELISAs were performed by two experienced operators (A.T. and S.J.). Serum samples were serially 2-fold diluted from 1:100 to 1:102,400. Negative- and positive-control samples were used as a control for within- and between-day consistency and were included in four wells each on each plate (2 per control sample). Bound anti-O. tsutsugamushi IgM antibodies were detected with anti-human IgM peroxidase conjugate (Invitrogen Corporation, USA) and tetramethylbenzidine substrate (KPL Inc., Maryland, USA). The OD was read at a wavelength of 450 nm (minus a reference OD value read at 650 nm) with a microtiter plate reader (Multiskan FC; Thermo Scientific, Finland). The ODs from the mock-antigen wells were subtracted as background absorbance to give a final average total net absorbance (net OD, or OD at 450 nm).
Statistical analysis.
The objective of the study was to determine an optimal sample dilution and a single optimal cutoff titer for IgM ELISA that provided the highest accuracy, not only when an admission sample was initially available, but also when a convalescent/discharge sample was available. Therefore, only patients with both admission and convalescent/discharge samples were included in the study, and we performed two stages of statistical analysis.
The first stage represented the acute clinical situation where only an admission sample was available. We used Bayesian LCMs to generate unbiased receiver operating characteristic (ROC) curves for the sensitivities and specificities of all possible cutoff titers of IgM ELISA in the admission sample alone, without using the convalescent/discharge sample IgM IFA and IgM ELISA results. We performed this for all possible sample dilutions. In brief, Bayesian LCMs estimated prevalence and the sensitivity and specificity of each diagnostic test with their 95% credible intervals (CrIs) using the Markov Chain Monte Carlo (MCMC) method (17). Bayesian LCMs do not assume that any diagnostic test or combination of diagnostic tests is perfect. The true disease status of each patient was estimated by the model in each MCMC iteration and expressed as the overall disease prevalence. The diagnostic tests included in the model were culture, a combination of PCR assays, IgM IFA for the admission sample alone, ELISA IgM for the admission sample alone, and presence of an eschar. The combination of PCR assays was considered positive when at least two out of three PCR assays, targeting the 56-kDa, 47-kDa, and groEL genes, were positive, as previously described (18). Models that took account of correlation between the IgM IFA and the IgM ELISA were used (19, 20, 25). The IgM IFA in the first model was considered positive when the admission IgM IFA titer was ≥1:3,200 (20). Unbiased ROC curves were generated as previously described (20).
The second stage of analysis represented the situation when a convalescent/discharge sample was available. We used Bayesian LCMs to generate unbiased ROC curves for the sensitivities and specificities of all possible cutoff titers of IgM ELISA using paired samples and for all possible sample dilutions. The higher OD of either the admission sample or the convalescent/discharge sample was used. The IgM IFA in the second model was considered positive when the admission IgM IFA titer was ≥1:3,200 or there was a 4-fold rise to ≥1:3,200 in convalescent/discharge samples compared to admission samples (20).
The optimal sample dilution was selected by considering that it should provide a cutoff OD between 1.00 and 2.00, which was recommended by the microtiter plate reader manufacturer as the central and optimal regions of the linear detection range (0 to 3 OD units) (Thermoscientific). The optimal cutoff OD was selected by considering that it should provide the highest accuracy in both situations (i.e., an admission sample alone and with paired convalescent/discharge samples). We evaluated the sensitivity, specificity, and accuracy of all possible ODs between the recommended cutoff ODs suggested by both models side by side. The OD that had the highest overall accuracy was selected as the optimal cutoff OD.
All Bayesian LCMs assumed that no prior information (noninformative priors) about the unknown parameters (i.e., prevalence, sensitivities, and specificities) was available, except that the specificity of the culture was fixed at 100%. Bayesian LCMs were performed in WinBUGS 1.4 (26). Texts S1 and S2 in the supplemental material provide full data sets and the models used, respectively.
Post hoc model evaluation.
To validate the sensitivity of the IgM ELISA estimated by the Bayesian LCMs, we estimated the naive sensitivity of the IgM ELISA in the patients who had a firm diagnosis of scrub typhus. The firm diagnosis of scrub typhus was made on the basis of either a positive blood culture, a combination of positive PCR assays, or the presence of an eschar in our data set. To validate the specificity of the IgM ELISA estimated by the Bayesian LCMs, we estimated the naive specificity of the IgM ELISA results in patients with a firm single diagnosis of other diseases, including murine typhus and dengue, in the data set described previously (19, 20). The diagnosis of murine typhus was made if there was a ≥4-fold increase in the Rickettsia typhi IgM IFA titer between paired samples (using slides coated with R. typhi strain Wilmington; ARRL, Geelong, Australia) (19, 20). The diagnosis of dengue was defined by the detection of NS1 antigen using Panbio ELISAs and IgM antibodies (Panbio, Brisbane, Australia) in paired samples (19, 20).
Evaluation of ELISA repeatability.
Between- and within-day variation was determined by calculating the mean and the standard deviation (SD) of the positive control on each plate to derive the between-plate, between-day, and within-day percent coefficient of variation (%CV) as follows: %CV = (SD/mean) × 100. Median values and interquartile ranges (IQR) were also calculated for the population.
RESULTS
Of 161 patients with acute undifferentiated fever evaluated, 135 (83.8%) had both acute and convalescent/discharge serum samples available for ELISA IgM and were included in the study. The median age was 42 years (IQR, 29 to 52 years; range, 15 to 84 years), and 83/135 (63%) were male. The median duration of fever before admission was 5 days (IQR, 4 to 7 days). The median duration between the admission and convalescent/discharge sample was 12 days (IQR, 3 to 14 days). Culture, a combination of PCR assays, IgM IFA, and the presence of an eschar were positive in 7, 22, 25, and 14 patients, respectively. There was a dominance of IFA titers at the lower end of the dilution scale for IgM titers in both admission and convalescent/discharge samples (Fig. 1).
FIG 1.
Distribution of IgM IFA titers in admission samples and convalescent/discharge samples.
The distribution of IgM ELISA ODs was skewed to the right for all sample dilutions and for both admission and convalescent/discharge samples (Fig. 2A and B). The IgM ELISA ODs in convalescent/discharge samples were generally higher than those in admission samples for all sample dilutions (all P values < 0.001; Wilcoxon paired signed-rank test). For example, at a sample dilution of 1:100, 86 (63.7%) patients had ODs in the convalescent/discharge sample higher than that in the admission sample. Patients who had a high OD at a 1:100 sample dilution also had comparatively high ODs at every dilution for both admission and convalescent/discharge samples. For example, Fig. 2A shows that the patient who had an IgM ELISA OD in the 10th percentile in the admission sample at a 1:100 sample dilution always had IgM ELISA ODs higher than those of the patients who had IgM ELISA ODs in the 20th, and 30th to 90th percentiles in all sample dilutions. The OD difference between each 10th percentile was narrower at the higher sample dilution (Fig. 2A and B).
FIG 2.
Distribution of IgM ELISA ODs in admission samples (A) and convalescent/discharge samples (B) at sample dilutions from 1:100 to 1:102,400. The colored lines show the positions of the patients who had the OD at the 10th, 20th, and every 10th percentile up to the 90th percentile at a sample dilution of 1:100.
Using the ROC curve to evaluate the performance of IgM ELISA with the admission sample alone, we found that the recommended cutoff ODs ranged from 1.854 to 0.028, decreasing with increasing dilution (see Fig. S1 in the supplemental material). For paired samples, the recommended cutoff ODs ranged from 2.427 to 0.053 for the dilution range 1:100 to 1:102,400 (see Fig. S2 in the supplemental material). A dilution of 1:400 was selected as the recommended sample dilution because it had recommended cutoff OD values for both scenarios between 1.00 and 2.00 (see Fig. S1 and S2 in the supplemental material), i.e., in the middle of the linear range of the microtiter plate reader.
To select a single optimal cutoff OD at a sample dilution of 1:400, the sensitivities and specificities of all possible ODs were evaluated between the recommended OD range when only the admission sample was available (OD = 1.010) and that when the paired convalescent/discharge samples were available (OD = 1.474) (Table 1). An OD of 1.474 was selected as the optimal cutoff because it provided the highest overall accuracy. Using a cutoff OD of 1.474 at a sample dilution of 1:400, Bayesian LCMs estimated that the IgM ELISA had a sensitivity of 69.0% (95% CrI, 60.6% to 76.9%) and a specificity of 100% (95% CrI, 99.1% to 100%) for the admission sample alone and a sensitivity of 85.7% (95% CrI, 77.4 to 86.7%) and a specificity of 98.1% (95% CrI, 97.2 to 100%) with the addition of the paired samples (Table 2). The Bayesian LCM estimated that there were 29 scrub typhus patients (95% CrI, 27 to 33 patients), representing a prevalence of scrub typhus at 21.8% (95% CrI, 15.1% to 29.5%) in the study participants. There was no significant difference between the sensitivities of the IgM ELISA and the IgM IFA for admission samples alone (69.0% versus 69.0%; Bayesian P value = 0.50) or between the sensitivities of the two tests using paired samples (85.7% versus 86.2%; Bayesian P value = 0.31).
TABLE 1.
Sensitivity and specificity of IgM ELISA at a sample dilution of 1:400
| Cutoff OD | Model usinga: |
Overall accuracy (%)b | |||
|---|---|---|---|---|---|
| On-admission sample alone |
Paired samples |
||||
| Sensitivity | Specificity | Sensitivity | Specificity | ||
| 1.010 | 78.6 (68.8–85.2) | 100 (98.2–100) | 85.2 (77.4–87.1) | 93.5 (92.5–96.2) | 93.3 |
| 1.023 | 73.3 (64.7–81.5) | 100 (98.2–100) | 85.2 (77.4–87.1) | 93.5 (92.5–96.2) | 93.0 |
| 1.080 | 73.3 (64.7–81.5) | 100 (98.2–100) | 85.2 (77.4–87.1) | 94.4 (93.5–97.1) | 93.3 |
| 1.093 | 70.0 (61.3–77.8) | 100 (98.2–100) | 85.2 (77.4–87.1) | 94.4 (93.5–97.1) | 93.0 |
| 1.165 | 70.0 (61.3–77.8) | 100 (98.2–100) | 85.2 (77.4–87.1) | 95.3 (94.4–98.1) | 93.3 |
| 1.214 | 70.0 (61.3–77.8) | 100 (98.2–100) | 85.2 (77.4–87.1) | 95.3 (94.4–98.1) | 93.3 |
| 1.273 | 70.0 (61.3–77.8) | 100 (98.2–100) | 85.2 (77.4–87.1) | 96.3 (95.3–99.0) | 93.7 |
| 1.275 | 70.0 (61.3–77.8) | 100 (98.2–100) | 85.2 (77.4–87.1) | 96.3 (95.3–99.0) | 93.7 |
| 1.307 | 69.0 (60.6–76.9) | 100 (99.1–100) | 85.2 (77.4–87.1) | 97.2 (96.3–100) | 94.1 |
| 1.474 | 69.0 (60.6–76.9) | 100 (99.1–100) | 85.7 (77.4–86.7) | 98.1 (97.2–100) | 94.4 |
The values are medians and 95% CrIs.
Overall accuracy is an average of the accuracies estimated by both models.
TABLE 2.
Accuracies of diagnostic tests for acute scrub typhus estimated using Bayesian LCMs
| Parametera | Accuracy (%) for model usingb: |
|
|---|---|---|
| On-admission sample alone | Paired samples | |
| In vitro culture for O. tsutsugamushi | ||
| Sensitivity | 24.1 (21.2–26.9) | 24.1 (21.2–25.9) |
| Specificity | 100 | 100 |
| PPV | 100 | 100 |
| NPV | 82.8 (79.7–85.2) | 82.8 (79.7–84.4) |
| Combination of PCR assaysc | ||
| Sensitivity | 69.2 (63.3–73.3) | 66.7 (61.3–70.0) |
| Specificity | 98.1 (95.5–100) | 97.2 (96.3–99.1) |
| PPV | 90.9 (77.3–100) | 86.4 (81.8–95.5) |
| NPV | 92.0 (89.4–92.9) | 91.2 (89.4–92.0) |
| IgM IFAd | ||
| Sensitivity | 69.0 (60.6–76.9) | 86.2 (75.8–92.6) |
| Specificity | 100 (99.1–100) | 100 (99.1–100) |
| PPV | 100 (95.0–100) | 100 (96.0–100) |
| NPV | 92.2 (88.7–94.8) | 96.4 (92.7–98.2) |
| IgM ELISAe | ||
| Sensitivity | 69.0 (60.6–76.9) | 85.7 (77.4–86.7) |
| Specificity | 100 (99.1–100) | 98.1 (97.2–100) |
| PPV | 100 (95.0–100) | 92.3 (88.5–100) |
| NPV | 92.2 (88.7–94.8) | 96.3 (93.6–96.3) |
| Presence of eschar | ||
| Sensitivity | 42.9 (37.9–46.7) | 41.4 (37.5–44.8) |
| Specificity | 98.1 (97.2–100) | 98.1 (98.1–99.1) |
| PPV | 85.7 (78.6–100) | 85.7 (85.7–92.9) |
| NPV | 86.8 (83.5–87.6) | 86.0 (83.5–87.6) |
PPV positive predictive value; NPV negative predictive value.
The values are medians and 95% CrIs.
A combination of PCR assays was defined as positive when at least two out of the three PCR assays (56-kDa nPCR assay, 47-kDa-based qPCR assay, and groEL-based qPCR assay) were positive.
In the model using on-admission samples alone, IgM IFA was defined as positive in those with an admission IgM IFA titer of ≥1:3,200. In the model using paired samples, IgM IFA was defined as positive in those with an admission IgM IFA titer of ≥1:3,200 or at least a 4-fold rise to ≥1:3,200 in the convalescent/discharge IgM IFA titer compared to the admission IgM IFA titer.
In the model using on-admission samples alone, IgM ELISA was defined as positive when the OD was ≥1.474 at a sample dilution of 1:400. In the model using paired samples, IgM ELISA was defined as positive when the OD was ≥1.474 at a sample dilution of 1:400 in either sample.
Post hoc model evaluation.
We evaluated the robustness of the cutoff OD of 1.474 at a sample dilution of 1:400 and showed that the IgM ELISA had a naive sensitivity of 69.0% (20/29) using paired samples in patients with a firm diagnosis of scrub typhus made using other diagnostic tests (Table 3). Eight of the nine patients with negative IgM ELISA results had very low IgM ELISA ODs (<0.5), even though the overall durations between the onset of fever and the convalescent/discharge sample date were 5 to 10 days (n = 3) and more than 10 days (n = 5) (see Table S1 in the supplemental material). The naive specificity of IgM ELISA in patients with a firm single diagnosis of either murine typhus or dengue virus infection was 96.3% (26/27), using paired samples in this group of patients (Table 3).
TABLE 3.
Naive sensitivity of IgM ELISA estimated in those in the study cohort who had a positive blood culture, a combination of PCR assays positive, or the presence of an eschar and in those who had a final diagnosis of either murine typhus or dengue infection
| Population | No. with IgM ELISAa positive | Naive sensitivity (%) | No. with IgM ELISAa negative | Naive specificity (%) |
|---|---|---|---|---|
| Patients who had blood culture positive (n = 7) | 5 | 71.4 (5/7) | NA | NA |
| Patients who had a combination of PCR assays positiveb (n = 22) | 17 | 77.3 (17/22) | NA | NA |
| Patients with the presence of an eschar (n = 14) | 11 | 78.6 (11/14) | NA | NA |
| Overallc (n = 29) | 20 | 69.0 (20/29) | NA | NA |
| Patients with a final diagnosis of murine typhus (n = 8) | NAd | NA | 8 | 100 (8/8) |
| Patients with a final diagnosis of dengue (n = 19) | NA | NA | 18 | 94.7 (18/19) |
| Overall (n = 27) | NA | NA | 26 | 96.3 (26/27) |
IgM ELISA was defined as positive when the OD was ≥1.474 at a sample dilution of 1:400 for either the admission or the convalescent/discharge sample.
A combination of PCR assays was defined as positive when at least two out of the three PCR assays (56-kDa nPCR assay, 47-kDa-based qPCR assay, and groEL-based qPCR assay) were positive.
Patients who had blood culture positive, a combination of PCR assays positive, or the presence of an eschar.
NA, not applicable.
ELISA repeatability.
One hundred and fifty-five ELISA plates were processed on 23 separate days. The minimum number of ELISA plates processed on a single day was 2 plates (days 1 and 2), and the maximum was 10 plates (days 12 and 21) (median, 8 plates/day). Between-plate positive-control values (n = 310 observations) gave a median OD of 2.99 with a %CV of 9.24% (mean, 2.92%; SD, 0.27%; IQR, 2.79% to 3.09%). Between-day mean positive-control values (n = 23 observations) gave a median OD of 3.02 and a %CV of 7.43% (mean, 2.96%; SD, 0.22%; IQR, 2.61% to 3.09%). Within-day variation for the positive-control values expressed as %CV gave a median of 4.0% (mean, 3.69%; SD, 1.81%; IQR, 3.0% to 4.0%).
DISCUSSION
We have described an unbiased approach using Bayesian LCMs to define optimal sample dilution and optimal cutoff OD values for the use of an IgM ELISA for the diagnosis of acute scrub typhus. We demonstrated that the sensitivity and specificity of the IgM ELISA are comparable to those of the IgM IFA with a cutoff of ≥1:3,200 in admission samples or a 4-fold rise to ≥1:3,200 in convalescent/discharge samples compared to admission samples and suggest that the IgM ELISA could be used as an alternative serological reference test instead of the IgM IFA.
In addition to comparable accuracy, there are a number of other reasons why ELISA technologies could replace IFA as the scrub typhus reference serological test. In general, the IFA is a subjective semiquantitative test that requires highly trained operators to prepare the slides and determine the results (9, 10). Furthermore, the IFA does not lend itself to the testing of large numbers of samples, hindering throughput when performing clinical studies or in clinical settings with large numbers of patients. These problems could easily be overcome by the ELISA, which could be automated and is easier to standardize, as it provides fully quantitative and consistent results (11, 12).
The sensitivity and specificity values estimated by our study are unbiased, because they are based on the true status of patients predicted by the Bayesian LCM. Previous studies reported that the NMRC IgM ELISA had higher sensitivity and lower specificity (11, 12) than the immunoperoxidase (IIP) assay (the IIP assay is analogous to IFA with the exception that different substrates are employed) titer of 1:400, which was assumed to be “perfect.” Suwanabun et al. reported that the NMRC IgM ELISA had a sensitivity of 92.9% and a specificity of 93.6% (11). Coleman et al. reported that the NMRC IgM ELISA at a sample dilution of 1:400 had a sensitivity of 94.2% and a specificity of 91.3% (12). However, the sensitivities reported by Suwannabun and Coleman were predicted to be overestimated, and this could be because low cutoff ODs were used and sensitivity was biased toward a group of patients who had IIP assay titers of ≥1:400. The specificities reported by Suwanabun and Coleman could be underestimated because low cutoff ODs were used and the IgM ELISA could be positive in scrub typhus patients who had an IIP assay titer of <1:400. These problems were due to disease misclassification by the imperfect gold standard (14–17). In contrast, the sensitivity and specificity values estimated by our study were not biased toward an imperfect reference test. This is also supported by the post hoc model evaluation showing that the sensitivity and specificity estimated by the Bayesian LCMs are comparable to the naive sensitivity and specificity values estimated in a group of patients with firm diagnoses of scrub typhus and of other diseases, respectively.
The choice of an appropriate sample dilution and diagnostic cutoff is an important step in diagnostic test validation and one that is often overlooked. Often, a generic sample dilution and a cutoff are defined based on comparing the results with the results of the reference tests, which are rarely if ever perfect for diagnosing tropical infectious diseases. Suwanabun et al. used a healthy control group to define the cutoff for the IgM ELISA (11), while Coleman et al. used an IIP assay titer of 1:400 to define the cutoff for the ELISA (12). The rationale for the sample dilution selection was unclear (11, 12). Bayesian LCMs and the generation of unbiased ROC curves have been increasingly used for many diseases (14, 16, 27, 28). In this study, we also show that the selection of sample dilution for the IgM ELISA could be systematically performed. Although all sample dilutions were capable of providing comparable accuracies using the IgM ELISA if an optimal cutoff OD was chosen (see Fig. S1 and S2 in the supplemental material), an optimal sample dilution can be recommended that fits with the optimal OD reading range recommended by the ELISA reader manufacturers. Application of the methodology described here would lead to a broader understanding of the utility of reference tests in the evaluation of quantitative diagnostic tests. This could lead to changes in the diagnostic process and clinical practice for many infectious diseases.
This study has some limitations. First, the sample size in our study was small. Second, the antigenic variation characteristic of O. tsutsugamushi strains can affect assay sensitivity and specificity. Previous studies have characterized the O. tsutsugamushi strains causing human disease (29–34) and presenting in vectors (35–37) in Thailand, showing a dominance of Karp- and Gilliam-like genotype strains, as well as TA716 and TA763 genotype strains. However, the O. tsutsugamushi strains that cause disease in the Chiangrai locality have not been fully characterized, and the incorporation of contemporary strains into the antigenic mix used in the test may increase the sensitivity. This is supported by our finding that some patients who had a firm diagnosis of scrub typhus were IgM ELISA negative, despite the time between the onset of fever and convalescent/discharge sampling being more than 10 days (see Table S1 in the supplemental material). An IgM ELISA using well-characterized contemporary local strain recombinant O. tsutsugamushi 56-kDa outer membrane antigens (38, 39) could be developed and evaluated in clinical settings using appropriate statistical models. Third, the accuracy of diagnostic tests varies based on prevalence, clinical variability, and availability and timing of convalescent-phase samples (40). Further studies to evaluate the optimal cutoff titers and accuracy of the IgM ELISA in different settings are still required.
In conclusion, we propose that IgM ELISA should be used as an alternative serological reference test for acute scrub typhus in the locality of Chiangrai, Thailand, with the recommendation that geographically specific diagnostic cutoffs be determined for other localities and employed using appropriate statistical models.
Supplementary Material
ACKNOWLEDGMENTS
We thank the Chiangrai Prachanukroh Hospital team for their assistance during the collection of specimens for this study.
This work was supported by the Wellcome Trust of the United Kingdom. A.L.R. is an employee of the U.S. government, and this work was prepared as part of his official duties.
We declare no conflicts of interest.
The views expressed in this article are ours and do not necessarily reflect the official policy or position of the Department of the Navy, the Department of Defense, or the U.S. government.
Footnotes
Supplemental material for this article may be found at http://dx.doi.org/10.1128/JCM.02744-15.
REFERENCES
- 1.Paris DH, Chattopadhyay S, Jiang J, Nawtaisong P, Lee JS, Tan E, Dela Cruz E, Burgos J, Abalos R, Blacksell SD, Lombardini E, Turner GD, Day NP, Richards AL. 2015. A nonhuman primate scrub typhus model: protective immune responses induced by pKarp47 DNA vaccination in cynomolgus macaques. J Immunol 194:1702–1716. doi: 10.4049/jimmunol.1402244. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Rodkvamtook W, Gaywee J, Kanjanavanit S, Ruangareerate T, Richards AL, Sangjun N, Jeamwattanalert P, Sirisopana N. 2013. Scrub typhus outbreak, northern Thailand, 2006-2007. Emerg Infect Dis 19:774–777. doi: 10.3201/eid1905.121445. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Dittrich S, Rattanavong S, Lee SJ, Panyanivong P, Craig SB, Tulsiani SM, Blacksell SD, Dance DA, Dubot-Peres A, Sengduangphachanh A, Phoumin P, Paris DH, Newton PN. 2015. Orientia, rickettsia, and leptospira pathogens as causes of CNS infections in Laos: a prospective study. Lancet Glob Health 3:e104–e112. doi: 10.1016/S2214-109X(14)70289-X. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.McGready R, Prakash JA, Benjamin SJ, Watthanaworawit W, Anantatat T, Tanganuchitcharnchai A, Ling CL, Tan SO, Ashley EA, Pimanpanarak M, Blacksell SD, Day NP, Singhasivanon P, White NJ, Nosten F, Paris DH. 2014. Pregnancy outcome in relation to treatment of murine typhus and scrub typhus infection: a fever cohort and a case series analysis. PLoS Negl Trop Dis 8:e3327. doi: 10.1371/journal.pntd.0003327. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Paris DH, Shelite TR, Day NP, Walker DH. 2013. Unresolved problems related to scrub typhus: a seriously neglected life-threatening disease. Am J Trop Med Hyg 89:301–307. doi: 10.4269/ajtmh.13-0064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Mayxay M, Castonguay-Vanier J, Chansamouth V, Dubot-Peres A, Paris DH, Phetsouvanh R, Tangkhabuanbutra J, Douangdala P, Inthalath S, Souvannasing P, Slesak G, Tongyoo N, Chanthongthip A, Panyanouvong P, Sibounheuang B, Phommasone K, Dohnt M, Phonekeo D, Hongvanthong B, Xayadeth S, Ketmayoon P, Blacksell SD, Moore CE, Craig SB, Burns MA, von Sonnenburg F, Corwin A, de Lamballerie X, Gonzalez IJ, Christophel EM, Cawthorne A, Bell D, Newton PN. 2013. Causes of non-malarial fever in Laos: a prospective study. Lancet Glob Health 1:e46–e54. doi: 10.1016/S2214-109X(13)70008-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Koh GC, Maude RJ, Paris DH, Newton PN, Blacksell SD. 2010. Diagnosis of scrub typhus. Am J Trop Med Hyg 82:368–370. doi: 10.4269/ajtmh.2010.09-0233. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Luce-Fedrow A, Mullins K, Kostik AP, St John HK, Jiang J, Richards AL. 2015. Strategies for detecting rickettsiae and diagnosing rickettsial diseases. Future Microbiol 10:537–564. doi: 10.2217/fmb.14.141. [DOI] [PubMed] [Google Scholar]
- 9.Blacksell SD, Bryant NJ, Paris DH, Doust JA, Sakoda Y, Day NP. 2007. Scrub typhus serologic testing with the indirect immunofluorescence method as a diagnostic gold standard: a lack of consensus leads to a lot of confusion. Clin Infect Dis 44:391–401. doi: 10.1086/510585. [DOI] [PubMed] [Google Scholar]
- 10.Phetsouvanh R, Thojaikong T, Phoumin P, Sibounheuang B, Phommasone K, Chansamouth V, Lee SJ, Newton PN, Blacksell SD. 2013. Inter- and intra-operator variability in the reading of indirect immunofluorescence assays for the serological diagnosis of scrub typhus and murine typhus. Am J Trop Med Hyg 88:932–936. doi: 10.4269/ajtmh.12-0325. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Suwanabun N, Chouriyagune C, Eamsila C, Watcharapichat P, Dasch GA, Howard RS, Kelly DJ. 1997. Evaluation of an enzyme-linked immunosorbent assay in Thai scrub typhus patients. Am J Trop Med Hyg 56:38–43. [DOI] [PubMed] [Google Scholar]
- 12.Coleman RE, Sangkasuwan V, Suwanabun N, Eamsila C, Mungviriya S, Devine P, Richards AL, Rowland D, Ching WM, Sattabongkot J, Lerdthusnee K. 2002. Comparative evaluation of selected diagnostic assays for the detection of IgG and IgM antibody to Orientia tsutsugamushi in Thailand. Am J Trop Med Hyg 67:497–503. [DOI] [PubMed] [Google Scholar]
- 13.Zhou XH, Castelluccio P, Zhou C. 2005. Nonparametric estimation of ROC curves in the absence of a gold standard. Biometrics 61:600–609. doi: 10.1111/j.1541-0420.2005.00324.x. [DOI] [PubMed] [Google Scholar]
- 14.Limmathurotsakul D, Chantratita N, Teerawattanasook N, Piriyagitpaiboon K, Thanwisai A, Wuthiekanun V, Day NP, Cooper B, Peacock SJ. 2011. Enzyme-linked immunosorbent assay for the diagnosis of melioidosis: better than we thought. Clin Infect Dis 52:1024–1028. doi: 10.1093/cid/cir080. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lim C, Wannapinij P, White L, Day NP, Cooper BS, Peacock SJ, Limmathurotsakul D. 2013. Using a web-based application to define the accuracy of diagnostic tests when the gold standard is imperfect. PLoS One 8:e79489. doi: 10.1371/journal.pone.0079489. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Choi YW, Collins M, Gardner I. 2006. Bayesian inferences for receiver operating characteristic curves in the absence of a gold standard. Agric Biol Environ Stat 11:19. [Google Scholar]
- 17.Joseph L, Gyorkos TW, Coupal L. 1995. Bayesian estimation of disease prevalence and the parameters of diagnostic tests in the absence of a gold standard. Am J Epidemiol 141:263–272. [DOI] [PubMed] [Google Scholar]
- 18.Paris DH, Blacksell SD, Nawtaisong P, Jenjaroen K, Teeraratkul A, Chierakul W, Wuthiekanun V, Kantipong P, Day NP. 2011. Diagnostic accuracy of a loop-mediated isothermal PCR assay for detection of Orientia tsutsugamushi during acute Scrub Typhus infection. PLoS Negl Trop Dis 5:e1307. doi: 10.1371/journal.pntd.0001307. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Lim C, Paris DH, Blacksell SD, Laongnualpanich A, Kantipong P, Chierakul W, Wuthiekanun V, Day NP, Cooper BS, Limmathurotsakul D. 2015. How to determine the accuracy of an alternative diagnostic test when it is actually better than the reference tests: a re-evaluation of diagnostic tests for scrub typhus using Bayesian LCMs. PLoS One 10:e0114930. doi: 10.1371/journal.pone.0114930. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Lim C, Blacksell SD, Paris DH, Laongnualpanich A, Kantipong P, Day NP, Limmathurotsakul D. 2015. Optimal cut-off titres for scrub typhus indirect immunofluorescence assay in northern Thailand. J Clin Microbiol 53:3663–3666. doi: 10.1128/JCM.01680-15. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Luksameetanasan R, Blacksell SD, Kalambaheti T, Wuthiekanun V, Chierakul W, Chueasuwanchai S, Apiwattanaporn A, Stenos J, Graves S, Peacock SJ, Day NP. 2007. Patient and sample-related factors that effect the success of in vitro isolation of Orientia tsutsugamushi. Southeast Asian J Trop Med Public Health 38:91–96. [PubMed] [Google Scholar]
- 22.Sonthayanon P, Chierakul W, Wuthiekanun V, Blacksell SD, Pimda K, Suputtamongkol Y, Pukrittayakamee S, White NJ, Day NP, Peacock SJ. 2006. Rapid diagnosis of scrub typhus in rural Thailand using polymerase chain reaction. Am J Trop Med Hyg 75:1099–1102. [PubMed] [Google Scholar]
- 23.Paris DH, Blacksell SD, Stenos J, Graves SR, Unsworth NB, Phetsouvanh R, Newton PN, Day NP. 2008. Real-time multiplex PCR assay for detection and differentiation of rickettsiae and orientiae. Trans R Soc Trop Med Hyg 102:186–193. doi: 10.1016/j.trstmh.2007.11.001. [DOI] [PubMed] [Google Scholar]
- 24.Paris DH, Aukkanit N, Jenjaroen K, Blacksell SD, Day NP. 2009. A highly sensitive quantitative real-time PCR assay based on the groEL gene of contemporary Thai strains of Orientia tsutsugamushi. Clin Microbiol Infect 15:488–495. doi: 10.1111/j.1469-0691.2008.02671.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Dendukuri N, Joseph L. 2001. Bayesian approaches to modeling the conditional dependence between multiple diagnostic tests. Biometrics 57:158–167. doi: 10.1111/j.0006-341X.2001.00158.x. [DOI] [PubMed] [Google Scholar]
- 26.Lunn DJ, Thomas A, Best N, Spiegelhalter D. 2000. WinBUGS—a Bayesian modelling framework: concepts, structure, and extensibility. Stat Comput 10:325–337. doi: 10.1023/A:1008929526011. [DOI] [Google Scholar]
- 27.Nielsen SS, Gronbaek C, Agger JF, Houe H. 2002. Maximum-likelihood estimation of sensitivity and specificity of ELISAs and faecal culture for diagnosis of paratuberculosis. Prev Vet Med 53:191–204. doi: 10.1016/S0167-5877(01)00280-X. [DOI] [PubMed] [Google Scholar]
- 28.Wang C, Bruce WT, Yrjo TG, Soren SN. 2007. Nonparametric estimation of ROC curves based on Bayesian models when the true disease state is unknown. J Agric Biol Environ Stat 12:128–146. doi: 10.1198/108571107X178095. [DOI] [Google Scholar]
- 29.Elisberg BL, Sangkasuvana V, Campbell JM, Bozeman FM, Bodhidatta P. 1967. Physiogeographic distribution of scrub typhus in Thailand. Acta Med Biol (Niigata) 15:61–67. [PubMed] [Google Scholar]
- 30.Kollars TM Jr, Bodhidatta D, Phulsuksombati D, Tippayachai B, Coleman RE. 2003. Short report: variation in the 56-kD type-specific antigen gene of Orientia tsutsugamushi isolated from patients in Thailand. Am J Trop Med Hyg 68:299–300. [PubMed] [Google Scholar]
- 31.Blacksell SD, Luksameetanasan R, Kalambaheti T, Aukkanit N, Paris DH, McGready R, Nosten F, Peacock SJ, Day NP. 2008. Genetic typing of the 56-kDa type-specific antigen gene of contemporary Orientia tsutsugamushi isolates causing human scrub typhus at two sites in north-eastern and western Thailand. FEMS Immunol Med Microbiol 52:335–342. doi: 10.1111/j.1574-695X.2007.00375.x. [DOI] [PubMed] [Google Scholar]
- 32.Fournier PE, Siritantikorn S, Rolain JM, Suputtamongkol Y, Hoontrakul S, Charoenwat S, Losuwanaluk K, Parola P, Raoult D. 2008. Detection of new genotypes of Orientia tsutsugamushi infecting humans in Thailand. Clin Microbiol Infect 14:168–173. doi: 10.1111/j.1469-0691.2007.01889.x. [DOI] [PubMed] [Google Scholar]
- 33.McGready R, Blacksell SD, Luksameetanasan R, Wuthiekanun V, Jedsadapanpong W, Day NP, Nosten F. 2010. First report of an Orientia tsutsugamushi type TA716-related scrub typhus infection in Thailand. Vector Borne Zoonotic Dis 10:191–193. doi: 10.1089/vbz.2008.0199. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Ruang-Areerate T, Jeamwattanalert P, Rodkvamtook W, Richards AL, Sunyakumthorn P, Gaywee J. 2011. Genotype diversity and distribution of Orientia tsutsugamushi causing scrub typhus in Thailand. J Clin Microbiol 49:2584–2589. doi: 10.1128/JCM.00355-11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Coleman RE, Monkanna T, Linthicum KJ, Strickman DA, Frances SP, Tanskul P, Kollars TM Jr, Inlao I, Watcharapichat P, Khlaimanee N, Phulsuksombati D, Sangjun N, Lerdthusnee K. 2003. Occurrence of Orientia tsutsugamushi in small mammals from Thailand. Am J Trop Med Hyg 69:519–524. [PubMed] [Google Scholar]
- 36.Lerdthusnee K, Nigro J, Monkanna T, Leepitakrat W, Leepitakrat S, Insuan S, Charoensongsermkit W, Khlaimanee N, Akkagraisee W, Chayapum K, Jones JW. 2008. Surveys of rodent-borne disease in Thailand with a focus on scrub typhus assessment. Integr Zool 3:267–273. doi: 10.1111/j.1749-4877.2008.00100.x. [DOI] [PubMed] [Google Scholar]
- 37.Rodkvamtook W, Ruang-Areerate T, Gaywee J, Richards AL, Jeamwattanalert P, Bodhidatta D, Sangjun N, Prasartvit A, Jatisatienr A, Jatisatienr C. 2011. Isolation and characterization of Orientia tsutsugamushi from rodents captured following a scrub typhus outbreak at a military training base, Bothong district, Chonburi province, central Thailand. Am J Trop Med Hyg 84:599–607. doi: 10.4269/ajtmh.2011.09-0768. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Rodkvamtook W, Zhang Z, Chao CC, Huber E, Bodhidatta D, Gaywee J, Grieco J, Sirisopana N, Kityapan M, Lewis M, Ching WM. 2015. Dot-ELISA rapid test using recombinant 56-kDa protein antigens for serodiagnosis of scrub typhus. Am J Trop Med Hyg 92:967–971. doi: 10.4269/ajtmh.14-0627. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Chao CC, Huber ES, Porter TB, Zhang Z, Ching WM. 2011. Analysis of the cross-reactivity of various 56 kDa recombinant protein antigens with serum samples collected after Orientia tsutsugamushi infection by ELISA. Am J Trop Med Hyg 84:967–972. doi: 10.4269/ajtmh.2011.10-0545. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Leeflang MM, Bossuyt PM, Irwig L. 2009. Diagnostic test accuracy may vary with prevalence: implications for evidence-based diagnosis. J Clin Epidemiol 62:5–12. doi: 10.1016/j.jclinepi.2008.04.007. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.