Abstract
Background:
Leptospirosis is under-diagnosed by clinicians in many high-incidence countries, as reference diagnostic tests are largely unavailable. Lateral flow assays (LFA) that use antigen derived from heat-treated whole cell Leptospira biflexa serovar Patoc have potential to improve leptospirosis diagnosis in resource-limited settings.
Objectives:
We sought to summarize estimates of sensitivity and specificity of LFA by conducting a systematic review and meta-analysis of evaluations of the accuracy of LFA to diagnose human leptospirosis.
Data sources:
On 4 July 2017 we searched three medical databases.
Study eligibility criteria:
Articles were included if they were a study of LFA sensitivity and specificity
Participants:
Patients with suspected leptospirosis
Interventions:
Nil
Methods:
For included articles, we assessed study quality, characteristics of participants, and diagnostic testing methods. We estimated sensitivity and specificity for each study against the study-defined case definition as the reference standard, and performed a meta-analysis using a random-effects bivariate model.
Results:
Our search identified 225 unique reports, of which we included nine (4%) published reports containing 11 studies. We classified one (9%) study as high quality. Nine (82%) studies used reference tests with considerable risk of misclassification. Our pooled estimates of sensitivity (95% confidence intervals) were 79% (70,86%) and specificity 92% (85,96%).
Conclusions:
As the evidence base for determining the accuracy of LFA is small and at risk of bias, pooled estimates of sensitivity and specificity should be interpreted with caution. Further studies should use either reference tests with high sensitivity and specificity or statistical techniques that account for an imperfect reference standard.
INTRODUCTION
Leptospirosis is a common cause of fever in tropical countries and a re-emerging disease globally (1, 2). Diagnosis is challenging as reference standard diagnostic tests such as Leptospira culture, microscopic agglutination testing, and nucleic acid amplification tests have imperfect sensitivity and specificity, are expensive, technically difficult, and not widely available in endemic areas (3). Inexpensive and simple point-of-care tests have been developed that detect anti-Leptospira IgM. These have the potential to be deployed at both the district hospital laboratory and health centre level in low-resource settings for the diagnosis of leptospirosis among febrile patients. Lateral flow assays (LFA) that use whole cell leptospiral antigen from the saprophytic Leptospira biflexa serovar Patoc strain Patoc I are among the most promising point-of-care tests as they are inexpensive and easy to use (4).
The accuracy of LFA has been evaluated in several studies with varied estimates of both sensitivity and specificity. As such, a summary of existing estimates of test performance and an understanding of sources of variation in the estimates is needed. We conducted a systematic review and meta-analysis to summarize the sensitivity and specificity of LFA for diagnosing acute human leptospirosis in patients with suspected leptospirosis, and to identify potential reasons for variation in published estimates of diagnostic accuracy between studies.
METHODS
We conducted our systematic review in accordance with the Preferred Reporting of Systematic Reviews and Meta-Analyses (PRISMA) guidelines (5). We registered our review with the International Prospective Register of Systematic Reviews (PROSPERO registration number CRD42018088566) and our protocol is available at https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=88566.
After developing and piloting search terms, we ran our search on 4 July 2017 using the databases Ovid Medline, Web of Science, and Scopus. In Ovid Medline we used the search terms and operators: ‘(Leptospirosis/*diagnosis OR Leptospirosis/*immunology OR (Leptospir* AND Immunoglobulin M)) AND Humans AND (Sensitivity and Specificity OR *Reference Standards).’ Search terms used for the Web of Science and Scopus databases are shown in Supplementary Material S1. Articles were included if they were a study of LFA sensitivity and specificity among patients with fever. Evaluations of assays other than LFA and evaluations performed in animals were excluded. Articles published in any language and in any year were eligible for inclusion. A single author (MJM) reviewed all abstracts and titles to determine which articles may have relevant data. For those deemed potentially relevant two authors (MJM and JAC) independently reviewed the full text of each article. We assessed study quality using the revised Quality Assessment of Diagnostic Accuracy Studies (QUADAS) criteria known as QUADAS-2 (Table 1) (6). Articles were graded in each category according to the information included in the manuscript, such that when methodological information was not included in the manuscript the quality assessment was downgraded. We graded study applicability in the domains of patient selection, use of the index test and use of the reference test as shown in Table 2. If insufficient information was included within the manuscript to attach a quality grade or applicability grade, we scored it as ‘unclear’.
Table 1.
Criteria for assessing bias in studies evaluating the accuracy of Leptospira biflexa serovar Patoc lateral flow IgM assays for the diagnosis of leptospirosis, published prior to 4 July 2017
| Category | Grade | Criteria |
|---|---|---|
| Participant selection | Grade 1 | • Prospective selection of patients with fever • Cases and controls selected from the same population |
| Grade 2 | • Eligibility determined by factors other than the presence of possible leptospirosis | |
| Grade 3 | • Selection of cases and controls from different populations | |
| Index test (LFA) | Grade 1 | • Assessors blinded to results of reference test when performing point of care test • Threshold for positivity is defined a priori and in keeping with manufacturers’ recommendations |
| Grade 2 | • Threshold not in keeping with manufacturers’ recommendations • Threshold for positivity is not defined a priori |
|
| Grade 3 | • Assessors not blinded to the results of the reference test when performing the point of care test | |
| Reference test | Grade 1 | • Use of MAT on paired serum samples in at least 75% of participants, with or without PCR or culture • Cases defined as participants with a four-fold rise in antibody titres on MAT, or positive culture of Leptospira, or detection of Leptospira DNA |
| Grade 2 | • Use of a MAT on acute-phase serum only, or an IgM ELISA assay, with or without PCR or culture | |
| Grade 3 | • Use of alternative assay as a reference standard | |
| Flow and timing | Grade 1 | • All patients subject to the same reference tests • Reference tests and index tests performed on samples taken at the same time for the illness |
| Grade 2 | • Data presented to allow calculation of sensitivity and specificity • Use of samples collected on different days for index and reference tests |
|
| Grade 3 | • Variation in reference test between participants, such that not all participants are subjected to the same reference test |
Abbreviations: LFA= Leptospira biflexa serovar Patoc lateral flow IgM assays MAT= Leptospira microscopic agglutination test; PCR= Polymerase chain reaction; IgM ELISA= Immunoglobulin M enzyme-linked immunosorbent assay
Table 2.
Criteria for assessing applicability in studies evaluating the accuracy of Leptospira biflexa serovar Patoc lateral flow IgM assays for the diagnosis of leptospirosis, published prior to 4 July 2017
| Category | Grade | Criteria |
|---|---|---|
| Participant selection | No concerns | • Patients included with febrile illness and with a duration of clinical illness from 1-21 days |
| Concerns | • Patients included without febrile illness • Patients of limited duration of illness |
|
| Index test (LFA) | No concerns | • Test used and interpreted according to manufacturer instructions |
| Concerns | • Test not used or interpreted according to manufacturers instructions | |
| Reference test | No concerns | • Reference test included a microscopic agglutination test (MAT) panel with a panel of antigens representing likely circulating serovars. |
| Concerns | • Reference test did not include MAT, or included only limited serovars |
Two authors (MJM and JAC) extracted data in duplicate using a standardized data extraction sheet (Supplementary Material Table S2) and tabulated data in a Microsoft Excel spread sheet (Microsoft Corporation, Redmond, WA, USA). We conducted meta-analysis using the user written programme ‘midas’ in STATA 13.1 (StataCorp, College Station, TX, USA) (7). We constructed forest plots displaying estimated sensitivity and specificity from contingency tables assuming that the reference test was 100% sensitive and specific. Meta-performance characteristics were established using a mixed-effects bivariate model. Publication bias was assessed using Deeks’ funnel plot asymmetry test (8, 9). Deeks’ funnel plot asymmetry test uses linear regression of log odds ratios on inverse root of effective sample sizes. A non-zero slope coefficient is suggestive of significant publication bias, or small study bias (p value <0.10).
RESULTS
Our search identified 225 unique reports. Of these, 32 (14%) were identified as potentially relevant on the basis of title and abstract, and underwent full-text review. We determined that 9 (4%) articles were relevant and these were selected for final inclusion (Figure 1). The 9 published reports contained data from 11 studies evaluating the accuracy of LFA.
Figure 1. Study flow diagram for systematic review and meta-analysis of studies evaluating the accuracy of Leptospira biflexa serovar Patoc lateral flow IgM assays for the diagnosis of leptospirosis, published prior to 4 July 2017.
Study characteristics
Included studies were published from 2001 through 2015. Ten (91%) of 11 studies were performed among patients with fever and one (9%) was performed among patients with uveitis and was not included in the meta-analysis (10). Evaluations were performed among participants from Brazil, India, Italy, Malaysia, the Netherlands, Poland, Seychelles, Slovenia, Sri Lanka, Thailand, United Kingdom, and the United States of America. The sample matrix was serum in all studies. Eight (73%) studies were of cross-sectional design, and 3 (27%) were of two-gate design (11). All studies reported that they performed the LFA according to manufacturers instructions. One (9%) study reported mean duration of symptoms and one (9%) study reported the prevalence of use of antimicrobials prior to testing. The leptospirosis reference test diagnostic criteria used in each study are shown in Table 3. Two (18%) studies reported that ≥75% of participants had paired serum samples tested for leptospirosis (12, 13); five (45%) reported <75% of participants had paired serum samples tested, including two studies that did not provide figures but stated that reference testing for ‘most’ participants was performed on a single serum sample (14-16); and four (36%) reported that for all participants reference testing occurred on single serum samples (10, 12, 17, 18). In setting diagnostic cut-offs for the reference test, seven (64%) studies used MAT titres lower than those recommended by the World Health Organization (19). Four (46%) studies used IgM ELISA as a reference test. Ten (90%) of studies considered the reference standard to be perfect when conducting their analyses, and three studies used latent class analysis, which does not assume the reference standard to perfectly accurate, to analyse their results (20). Two (18%) studies reported the serogroup of infecting Leptospira as determined by the reference test (14). Further information relating to study characteristics is included in Supplementary Material Table S3.
Table 3.
References tests used in studies evaluating Leptospira biflexa serovar Patoc lateral flow IgM assays for the diagnosis of leptospirosis, published prior to 4 July 2017
| Study first author, number (reference) |
Reference test | Number of Leptospira serovars in reference test |
Participants with reference test performed on paired serum samples |
Reference test leptospirosis case definition |
|---|---|---|---|---|
| Smits 1a (12) | MAT | 1 | 100% | ≥ four-fold rise in MAT titre or single MAT titre ≥160 |
| Smits 2a (12) | ELISA | Not stated | 0% | Single antibody titre ≥80 |
| Eapen (17) | ELISA | 2b | 0% | Single antibody titre ≥80 |
| Sehgal (13) | MAT + culture | 12 | 89% | Positive culture, or seroconversion to an MAT titre ≥100, or a four-fold rise in MAT titre, or a single MAT titre≥400 |
| Kannan (10) | MAT | Not stated | 0% | Single MAT titre≥100 |
| Limmathurotsakul (21) | MAT + culture | 20 | 66% | Positive culture or single MAT titre ≥400 |
| Goris 1a (14) | MAT + IgM ELISA + culture | 14b | Not statedc | (i) Single MAT titre with a pathogenic serovar ≥160, (ii) single IgM-ELISA titre ≥160, (iii) positive culture or (iv) seroconversion/four-fold titre rise MAT or IgM ELISA (titre ≤20 to ≥80) in paired samples taken at least 2 days apart |
| Goris 2a (14) | MAT + IgM ELISA + culture | 14b | Not statedc | (i) Single MAT titre with a pathogenic serovar ≥160, (ii) single IgM-ELISA titre ≥160, (iii) positive culture or (iv) seroconversion/four-fold titre rise MAT or IgM ELISA (titre ≤20 to ≥80) in paired samples taken at least 2 days apart |
| Niloofa (15) | MAT | 13 | 28% | Single Mat titre ≥400 |
| Podgorsek (16) | MAT | 1b | 6% | Seroconversion to an MAT titre ≥100, or a four-fold rise in MAT titre, or a single MAT titre≥400 |
| Eugene (18) | MAT + culture | 1b | 0% | Positive culture or single MAT titre ≥100 |
Footnote:
Numbers refer to different datasets from within the same paper;
Serovars included Leptospira biflexa serovar Patoc;
Figures not stated but paper states that ‘In most cases only one sample was received per participant’;
MAT = Leptospira microscopic agglutination test; IgM ELISA= Immunoglobulin M enzyme-linked immunosorbent assay
Study quality
The results of bias assessment are shown in Table 4. We considered a single study to be of Grade 1 quality in each of the four domains. We rated two (18%) studies as Grade 1 and nine (82%) studies as Grade 2 for the reference test domain. This was mostly (Table 3) because a single acute phase serum sample for serologic testing was considered the reference test. We classified one (9%) study as Grade 3 in the patient selection domain due to use of healthy participants from the population as controls and one (9%) study as Grade 3 in the flow and timing domain due to use of different testing algorithms among cases and controls (10, 16). We had applicability concerns about the reference test chosen in five (45%) studies and in an additional one (9%) study there was insufficient information to assess this domain. We also had concerns about the applicability of two (18%) of studies to our question within the patient selection domain.
Table 4.
Bias assessment of studies evaluating the accuracy of Leptospira biflexa serovar Patoc lateral flow IgM assays for leptospirosis published prior to 4 July 2017
| Quality |
Applicability |
||||||||
|---|---|---|---|---|---|---|---|---|---|
| Study first author, number (reference) |
Year | Patient selection |
Index test |
Reference test |
Flow and timing |
Patient selection |
Index test | Reference test |
|
| Smits 1* (12) | 2001 | Grade 2 | Grade 1 | Grade 1 | Grade 1 | No concerns | No concerns | Concerns | |
| Smits 2* (12) | 2001 | Grade 2 | Grade 1 | Grade 2 | Grade 1 | No concerns | No concerns | Unclear | |
| Eapen (17) | 2002 | Grade 2 | Grade 1 | Grade 2 | Grade 1 | No concerns | No concerns | Concerns | |
| Sehgal (13) | 2003 | Grade 1 | Grade 1 | Grade 1 | Grade 1 | No concerns | No concerns | Concerns | |
| Kannan (10) | 2012 | Grade 3 | Grade 1 | Grade 2 | Grade 1 | Concerns | No concerns | No concerns | |
| Limmathurotsakul (21) | 2012 | Grade 2 | Grade 1 | Grade 2 | Grade 1 | No concerns | No concerns | No concerns | |
| Goris 1* (14) | 2013 | Grade 1 | Grade 1 | Grade 2 | Grade 1 | No concerns | No concerns | No concerns | |
| Goris 2* (14) | 2013 | Grade 1 | Grade 1 | Grade 2 | Grade 1 | No concerns | No concerns | No concerns | |
| Eugene (18) | 2015 | Grade 1 | Grade 1 | Grade 2 | Grade 1 | No concerns | No concerns | Concerns | |
| Niloofa (15) | 2015 | Grade 2 | Grade 1 | Grade 2 | Grade 1 | Concerns | No concerns | Concerns | |
| Podgorsek (16) | 2015 | Grade 2 | Grade 1 | Grade 2 | Grade 3 | No concerns | No concerns | No concerns | |
Abbreviations: LFA= Patoc antigen lateral flow assay; Footnote:
Numbers refer to different datasets from within the same paper.
Sensitivity and specificity estimates
The number of participants with, and without leptospirosis who tested positive by LFA for each study is shown in Table 5. The sensitivity and specificity of LFA, estimated in each study and the pooled estimate are shown in Figure 2. In our meta-analysis we included the 10 (91%) studies that recruited patients with suspected leptospirosis. The pooled estimate of sensitivity (95% CI) was 79 (70-86)% and the pooled estimate of specificity was 92 (85-96)%. The study that we classified as of low risk of bias in every domain (13) estimated the sensitivity of LFA as 53 (41-64)% and the specificity as 94 (82-98)%. In the funnel plot (Figure 3) the regression line had a near vertical slope and Deeks’ test indicated funnel plot symmetry consistent with unbiased publication (p = 0.12). We excluded from our meta-analysis, the study by Kannan and colleagues that estimated the sensitivity and specificity of LFA among patients with uveitis as 70 (54-82)% and 69 (53-82)% respectively (10).
Table 5.
Extracted data from studies evaluating sensitivity and specificity of Leptospira biflexa serovar Patoc lateral flow IgM assays published prior to 4 July 2017
| Study first author (reference) |
Assay manufacturer |
True positive | False positive | True negative | False negative | Sensitivity (95% CI) |
Specificity (95% CI) |
||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| N | (%) | N | (%) | N | (%) | N | (%) | ||||
| Smits 1 (12) | KIT | 116 | (27.8) | 28 | (6.7) | 255 | (61.0) | 19 | (4.5) | 0.86 (0.79-091) | 0.90 (0.86-0.93) |
| Smits 2 (12) | KIT | 39 | (43.3) | 6 | (6.7) | 41 | (45.6) | 4 | (4.4) | 0.91 (0.78-0.97) | 0.87 (0.74-0.94) |
| Eapen (17) | Organon | 54 | (27.6) | 6 | (3.1) | 131 | (66.8) | 5 | (2.6) | 0.90 (0.80-0.95) | 0.96 (0.92-0.99) |
| Sehgal (13) | KIT | 37 | (31.6) | 3 | (2.6) | 44 | (37.6) | 33 | (28.2) | 0.53 (0.54-0.82) | 0.94 (0.82-0.98) |
| Kannan (10) | Zephyr | 28 | (36.8) | 11 | (14.5) | 25 | (32.9) | 12 | (15.8) | 0.70 (0.42-0.69) | 0.69 (0.53-0.82) |
| Limmathurotsakul (21) | KIT | 120 | (32.3) | 69 | (18.6) | 165 | (44.5) | 17 | (4.6) | 0.88 (0.81-0.92) | 0.71 (0.64-0.76) |
| Goris 1 (14) | BioMérieux | 74 | (5.3) | 57 | (4.1) | 1235 | (88.2) | 34 | (2.4) | 0.69 (0.59-0.77) | 0.96 (0.94-0.97) |
| Goris 2 (14) | Zephyr | 100 | (3.7) | 56 | (2.0) | 2495 | (91.3) | 83 | (3.0) | 0.55 (0.47-0.62) | 0.98 (0.97-0.98) |
| Eugene (18) | Zephyr | 34 | (40.5) | 12 | (14.3) | 32 | (38.1) | 6 | (7.1) | 0.85 (0.70-0.93) | 0.73 (0.58-0.84) |
| Niloofa (15) | Zephyr | 286 | (32.2) | 121 | (13.6) | 405 | (45.6) | 76 | (8.6) | 0.79 (0.75-0.83) | 0.77 (0.73-0.80) |
| Podgorsek (16) | Zephyr | 29 | (4.9) | 7 | (1.2) | 547 | (92.7) | 7 | (1.2) | 0.81 (0.65-0.90) | 0.99 (0.97-0.99) |
Footnote: Numbers refer to different datasets from within the same paper
Assay manufacturers: KIT: Royal Dutch Tropical Institute, Amsterdam, Netherlands; Organon, Oss, Netherlands; Zephyr Diagnostics, Goa, India; BioMérieux, Marcy-l'Étoile, France
Abbreviation: CI – confidence intervals
Figure 2. Forest plot of sensitivity and specificity of Patoc antigen lateral flow assays for the diagnosis of leptospirosis, published prior to 4 July 2017.
Key: The squares indicate the point estimate of sensitivity or specificity from each study, and the line indicates the 95% confidence intervals. The vertical dotted line indicates the point meta-estimate of sensitivity or specificity, and the diamond indicates the 95% confidence intervals
Figure 3. Funnel plot for a meta-analysis of Leptospira biflexa serovar Patoc lateral flow IgM assays for detecting leptospirosis published prior to 4 July 2017 and Deek’s weighted regression test of funnel plot asymmetry.
Key: 1/root(ESS) indicates the inverse root of the effective sample size
DISCUSSION
We systematically collated published literature on the sensitivity and specificity of leptospirosis LFA point-of-care tests. We identified that most evaluations were at risk of bias, predominantly due to the use of reference test criteria that were likely to misclassify participants. Of the studies included in our analysis, there was substantial heterogeneity in estimated sensitivity and specificity that appears to relate to study design, particularly the choice of leptospirosis reference test, but may also relate to duration of illness, the predominant infecting Leptospira serovars and variation in the production of LFA antigen. As such we consider our pooled estimates for the sensitivity and specificity of LFA to be unreliable and further robust evaluations are needed.
We found that the estimated of sensitivity of Leptospira biflexa serovar Patoc lateral flow IgM assays varied from 55% to 93% (13, 14) and estimates of specificity varied from 57% to 99% (13, 16). Factors relating to study design, particularly the variation in reference tests may account for the variation in apparent diagnostic accuracy. We found that nine (82%) studies were at risk of bias due to concerns in the reference test domain, with risk of misclassification of cases and controls. There are well-documented accuracy concerns with reference tests that make the choice of reference test challenging (21, 22). Leptospira culture is thought to have close to 100% specificity, but it has been estimated to have a sensitivity of <10% (22, 23). PCR of gene targets specific to pathogenic Leptospira are specific for leptospirosis but typically have been shown to have lower sensitivity than IgM serologic assays (23, 24) with sensitivity values such as 36% when compared to MAT (25). MAT serology is often considered the reference serologic test for diagnosis of leptospirosis (22), but also has imperfect sensitivity and specificity. In a recent evaluation of MAT accuracy against culture confirmed leptospirosis cases, Goris and colleagues identified that paired samples with at least 10 days between acute and convalescent samples had a sensitivity of 90% and 88% for 10-19 and ≥20 days respectively (22). By comparison, the sensitivity of diagnosing leptospirosis by using a high titre from a single serum sample was low, at 6% within the first 10 days of illness. The specificity of defining leptospirosis as ≥4-fold rise in MAT antibody titre between acute and convalescent serum samples is considered to approximate 100% (22, 23). Defining leptospirosis as a single high ≥400 titre is consistent with the WHO case definition but is imperfectly specific, especially where leptospirosis is endemic. This is because Leptospira antibodies can persist in serum for several years after acute infection (26). Although case definitions using single antibody titres of ≥160, ≥400, and ≥800 may be appropriate for clinical diagnosis (19, 22, 27), their use as a reference standard in diagnostic test evaluation may lead to misclassification of cases and controls and biased estimates of sensitivity and specificity of novel diagnostic tests. Only two (18%) studies tested predominantly paired serum samples when conducting reference testing for disease classification and potential misclassification was compounded by the variation in the reference test titre used to classify leptospirosis. ELISA is widely used as a screening test for the diagnosis of leptospirosis, however the pooled estimates of sensitivity and specificity (95% CI) of ELISA from a systematic review and meta-analysis were 78 (77–79) % and 91 (91–92) % respectively (28). In addition there was significant heterogeneity across studies that was not fully explained by disease stage, antigen used and antibody detected. The imperfect accuracy and significant heterogeneity suggest that use of ELISA as a reference test will be likely to misclassify the disease state of some study participants.
In addition to the choice of reference test, the variations in estimated accuracy of LFA may reflect varying performance at different stages of the illness. Of the few studies reporting the duration of illness, Sehgal and colleagues found that among participants in the Andaman Islands that LFA had a sensitivity of 53% during the first week of illness, and 86% during the second through fourth weeks of illness (13). The corresponding specificity was 94% during the first week and 89% from the second through fourth weeks (29). Goris and colleagues demonstrated that for two LFA, sensitivity increased from 42%-62% during days 0-4 after onset, to 65%-75% during days 5-10 after onset, and 72%-81% during days 11-20 after onset (14). The single study investigating LFA accuracy among patients with uveitis found accuracy values among the lowest of the studies. This may be due to the wide differential diagnoses of uveitis, the variable interval between leptospirosis infection and uveitis, and the use of immune-suppressants as treatment for uveitis. The geographic setting of the study population may influence test performance through variation in infecting Leptospira serovars and variation in the type and prevalence of diseases other than leptospirosis that cause fever. It was notable that the one report that reported serogroup of the infecting Leptospira found that LFA accuracy was higher among participants infected with Leptospira serogroup Icterohaemorrhagiae (14). One study included in our review noted variation in assay performance over time, which they thought may be due to variability of the antigen among assay production lots (14).
Our meta-analysis has several limitations that influence interpretation. We may not have identified all relevant articles through an incompletely comprehensive search strategy that used subject headings or free text terms individually to describe each concept, as well as use of limiting terms such as ‘humans’ and ‘sensitivity and specificity’. In addition, we may have missed studies that were published in journals not indexed by Ovid Medline, Scopus, or Web of Science, as well as studies that were not published. Combining all studies into a single estimate of sensitivity and specificity may be misleading as there was substantial variation in both study design and in the populations from which participants were drawn. Our meta-analysis assumed that the reference test to had 100% sensitivity and specificity. The reference tests used in most studies included in our meta-analysis have not had their sensitivity and specificity adequately determined. Under these circumstances conventional sensitivity and specificity estimates are likely to underestimate the accuracy of point-of-care diagnostic tests (30, 31). In the context of imperfect reference standards, other authors have used latent class analyses to estimate sensitivity and specificity (23, 30-32). Latent class analysis requires that there are at least four independent diagnostic tests to be able to identify two latent classes (31). This was not possible in our meta-analysis as most studies did not include a sufficient number of independent diagnostic assays.
On the assumption that the true sensitivity and specificity of LFA is at least as high as our pooled estimate, LFA may have a role as a screening assay. In studies from Southeast Asia and Africa, where leptospirosis is endemic, the prevalence of acute leptospirosis has been as high as 10% among febrile patients presenting for healthcare (33, 34). Assuming 10% prevalence of acute leptospirosis among patients tested with LFA, the negative predictive value (95% CI) would be 98 (96-99)%, with 2 (1-4)% of leptospirosis cases missed. This suggests that in high incidence settings that clinicians could use a negative LFA result to exclude leptospirosis, except during the first few days of illness when all serologic assays may have lower sensitivity and negative predictive values (24). However, only 48 (30-66)% of those who tested positive with LFA would truly have leptospirosis. Unless there are suitable confirmatory assays available there is considerable risk that introduction of LFA would result in over-diagnosis of leptospirosis. Over-diagnosis may have implications for individual patients in whom diseases that are also common in countries with high leptospirosis incidence and require specific treatment, such as rickettsiosis, may be falsely discounted.
A key finding of our study is that the evidence base for estimating the sensitivity and specificity of LFA is small and at risk of bias. Further studies are needed. Future studies should use a reference standard with sensitivity and specificity close to 100% or statistical analyses that manage the absence of a perfect reference. We suggest that future evaluations of point-of-care tests should consider the use of MAT on paired serum samples, PCR and culture as leptospirosis reference tests. In addition, as even a combination of these tests is unlikely to have 100% sensitivity, statistical methods that account for imperfect reference test accuracy, such as latent class analysis should be considered (23). Latent class analyses assume that the observations are independent within each of the two latent classes (31). Estimates of sensitivity and specificity are sensitive to violations of this assumption, and therefore estimates obtained by latent class analyses should also be interpreted cautiously, particularly if checks of the validity of the assumption are not reported (31, 35).
Estimates of the sensitivity and specificity varied from 53%-95% and 57-99% respectively with study design, particularly choice of the leptospirosis reference test, and features of the study population contributing to the variation. Our meta-estimates of sensitivity and specificity should be interpreted with caution, but suggest that LFA may have a limited role as a screening test in endemic settings, if appropriate confirmatory testing is available. Future studies evaluating point-of-care diagnostic tests should optimize the sensitivity and specificity of the leptospirosis reference test and consider statistical methods to manage imperfect reference tests.
Supplementary Material
ACKNOWEDGEMENTS
This work was supported by a University of Otago Research Grant, a joint US National Institutes of Health (NIH:www.nih.gov)-National Science Foundation (NSF:www.nsf.gov) Ecology of Infectious Disease program grant (R01TW009237) and the Research Councils UK, Department for International Development (UK) and UK Biotechnology and Biological Sciences Research Council (BBSRC:www.bbsrc.ac.uk) (grant numbers BB/J010367/1, BB/L018926, BB/L017679, BB/L018845), and in part by an US National Institutes of Health International Studies on AIDS Associated Co-infections (ISAAC) award (grant number U01 AI062563) and in part by the Bill & Melinda Gates Foundation funded Typhoid Fever Surveillance in sub-Saharan Africa Program (TSAP) grant (grant number OPPGH5231). MJM received support from University of Otago scholarships: the Frances G. Cotter Scholarship and the MacGibbon Travel Fellowship. MPR received support from National Institutes of Health Research Training Grants (grant numbers R25 TW009343) funded by the Fogarty International Center and the National Institute of Mental Health. KJA received support from the Wellcome Trust (www.wellcome.ac.uk) (grant number 096400/Z/11/Z). MPR, and JAC received support from a US National Institutes of Health National Institute for Allergy and Infectious grant (grant number R01 AI121378).
Footnotes
Presented in part: 67th American Society of Tropical Medicine and Hygiene annual meeting, New Orleans, LA, 28 October - 1 November 2018, poster 497
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
DISCLOSURE
All authors have nothing to disclose
REFERENCES
- 1.Costa F, Hagan JE, Calcagno J, Kane M, Torgerson P, Martinez-Silveira MS, et al. Global morbidity and mortality of leptospirosis: a systematic review. PLoS Negl Trop Dis. 2015;9(9):e0003898. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Hartskeerl RA, Collares-Pereira M, Ellis WA. Emergence, control and re-emerging leptospirosis: dynamics of infection in the changing world. Clin Microbiol Infect. 2011;17(4):494–501. [DOI] [PubMed] [Google Scholar]
- 3.Haake DA, Levett PN. Leptospirosis in humans. Curr Top Microbiol Immunol. 2015;387:65–97. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Picardeau M, Bertherat E, Jancloes M, Skouloudis AN, Durski K, Hartskeerl RA. Rapid tests for diagnosis of leptospirosis: current tools and emerging technologies. Diagn Microbiol Infect Dis. 2014;78(1):1–8. [DOI] [PubMed] [Google Scholar]
- 5.Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine. 2009;6(7):e1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–36. [DOI] [PubMed] [Google Scholar]
- 7.Dwamena B MIDAS: Stata module for meta-analytical integration of diagnostic test accuracy studies. 2009. [Google Scholar]
- 8.Deeks JJ, Macaskill P, Irwig L. The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. Journal of Clinical Epidemiology. 2005;58(9):882–93. [DOI] [PubMed] [Google Scholar]
- 9.Macaskill P, Gatsonis CA, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10 Analysing and presenting results In: Deeks JJ, Bossuyt PM, Gatsonis CA, editors. Cochrane Handbook for systematic reviews of diagnostic test accuracy Version 10: The Cochrane Collaboration; 2010. [Google Scholar]
- 10.Kannan A, Priya CG, Prajna L, Rathinam SR. Efficiency of two commercial kits in serodiagnosis of leptospiral uveitis. Indian J. 2012;30(4):418–22. [DOI] [PubMed] [Google Scholar]
- 11.Rutjes AW, Reitsma JB, Vandenbroucke JP, Glas AS, Bossuyt PM. Case–control and two-gate designs in diagnostic accuracy studies. Clinical Chemistry; 2005. [DOI] [PubMed] [Google Scholar]
- 12.Smits HL, Eapen CK, Sugathan S, Kuriakose M, Gasem MH, Yersin C, et al. Lateral-flow assay for rapid serodiagnosis of human leptospirosis. Clin Diagn Lab Immunol. 2001;8(1):166–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Sehgal SC, Vijayachari P, Sugunan AP, Umapathi T. Field application of Lepto lateral flow for rapid diagnosis of leptospirosis. J Med Microbiol. 2003;52(Pt 10):897–901. [DOI] [PubMed] [Google Scholar]
- 14.Goris MG, Leeflang MM, Loden M, Wagenaar JF, Klatser PR, Hartskeerl RA, et al. Prospective evaluation of three rapid diagnostic tests for diagnosis of human leptospirosis. PLoS Negl Trop Dis. 2013;7(7):e2290. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Niloofa R, Fernando N, de Silva NL, Karunanayake L, Wickramasinghe H, Dikmadugoda N, et al. Diagnosis of Leptospirosis: Comparison between Microscopic Agglutination Test, IgM-ELISA and IgM Rapid Immunochromatography Test. PLoS ONE. 2015;10(6):e0129236. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Podgorsek D, Cerar T, Logar M, Lesnicar G, Remec T, Baklan Z, et al. Evaluation of the immunochromatographic (Leptocheck) test for detection of specific antibodies against leptospires. Wien Klin Wochenschr. 2015; 127(23-24):948–53. [DOI] [PubMed] [Google Scholar]
- 17.Eapen CK, Sugathan S, Kuriakose M, Abdoel T, Smits HL. Evaluation of the clinical utility of a rapid blood test for human leptospirosis. Diagn Microbiol Infect Dis. 2002;42(4):221–5. [DOI] [PubMed] [Google Scholar]
- 18.Eugene EJ, Handunnetti SM, Wickramasinghe SA, Kalugalage TL, Rodrigo C, Wickremesinghe H, et al. Evaluation of two immunodiagnostic tests for early rapid diagnosis of leptospirosis in Sri Lanka: a preliminary study. BMC Infect Dis. 2015;15:319. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.World Health Organization. Report of the second meeting of the Leptospirosis Burden Epidemiology Reference Group. 2011. [Google Scholar]
- 20.Lim C, Wannapinij P, White L, Day NP, Cooper BS, Peacock SJ, et al. Using a web-based application to define the accuracy of diagnostic tests when the gold standard is imperfect. PLoS One. 2013;8(11):e79489. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Limmathurotsakul D, Turner EL, Wuthiekanun V, Thaipadungpanit J, Suputtamongkol Y, Chierakul W, et al. Fool's gold: Why imperfect reference tests are undermining the evaluation of novel diagnostics: a reevaluation of 5 diagnostic tests for leptospirosis. Clin Infect Dis. 2012;55(3):322–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Goris MG, Leeflang MM, Boer KR, Goeijenbier M, van Gorp EC, Wagenaar JF, et al. Establishment of valid laboratory case definition for human leptospirosis. Journal of Bacteriology & Parasitology. 2012;2012. [Google Scholar]
- 23.Limmathurotsakul D, Turner EL, Wuthiekanun V, Thaipadungpanit J, Suputtamongkol Y, Chierakul W, et al. Fool's gold: why imperfect reference tests are undermining the evaluation of novel diagnostics: a reevaluation of 5 diagnostic tests for leptospirosis. Clin Infect Dis. 2012;55(3):322–31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Schreier S, Doungchawee G, Chadsuthi S, Triampo D, Triampo W. Leptospirosis: current situation and trends of specific laboratory tests. Expert Rev Clin Immunol. 2013;9(3):263–80. [DOI] [PubMed] [Google Scholar]
- 25.Ooteman MC, Vago AR, Koury MC. Evaluation of MAT, IgM ELISA and PCR methods for the diagnosis of human leptospirosis. J Microbiol Methods. 2006;65(2):247–57. [DOI] [PubMed] [Google Scholar]
- 26.Cumberland P, Everard CO, Wheeler JG, Levett PN. Persistence of anti-leptospiral IgM, IgG and agglutinating antibodies in patients presenting with acute febrile illness in Barbados 1979-1989. Eur J Epidemiol. 2001;17(7):601–8. [DOI] [PubMed] [Google Scholar]
- 27.Centers for Disease Control and Prevention. Leptospirosis (Leptospira interrogans) 2013 case definition 2013. [21 October 2015]. Available from: https://wwwn.cdc.gov/nndss/conditions/leptospirosis/case-definition/2013/.
- 28.Signorini ML, Lottersberger J, Tarabla HD, Vanasco NB. Enzyme-linked immunosorbent assay to diagnose human leptospirosis: a meta-analysis of the published literature. Epidemiol Infect. 2013;141(1):22–32. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Sehgal SC, Vijayachari P, Sugunan AP, Umapathi T. Field application of Lepto lateral flow for rapid diagnosis of leptospirosis. J Med Microbiol. 2003;52(Pt 10):897–901. [DOI] [PubMed] [Google Scholar]
- 30.Boelaert M, Verloo D, Van Der Stuyft P, editors. Applications of latent class analysis in diagnostic validation, the user perspective International Conference on Statistics in Health Sciences; 2004. [Google Scholar]
- 31.van Smeden M, Naaktgeboren CA, Reitsma JB, Moons KG, de Groot JA. Latent class models in diagnostic studies when there is no reference standard--a systematic review. Am J Epidemiol. 2014;179(4):423–31. [DOI] [PubMed] [Google Scholar]
- 32.Rindskopf D, Rindskopf W. The value of latent class analysis in medical diagnosis. Statistics in medicine. 1986;5(1):21–7. [DOI] [PubMed] [Google Scholar]
- 33.Southeast Asia Infectious Disease Clinical Research N. Causes and outcomes of sepsis in southeast Asia: a multinational multicentre cross-sectional study. Lancet Glob Health. 2017;5(2):e157–e67. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Crump JA, Morrissey AB, Nicholson WL, Massung RF, Stoddard RA, Galloway RL, et al. Etiology of severe non-malaria febrile illness in Northern Tanzania: a prospective cohort study. PLoS Negl Trop Dis. 2013;7(7):e2324. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Albert PS, Dodd LE. A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard. Biometrics. 2004;60(2):427–35. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.