Skip to main content
Bulletin of the World Health Organization logoLink to Bulletin of the World Health Organization
. 2022 Jun 1;100(7):447–458. doi: 10.2471/BLT.21.287496

Rapid diagnostic tests versus RT–PCR for Ebola virus infections: a systematic review and meta-analysis

Tests de diagnostic rapide contre tests RT-PCR pour les infections à virus Ebola: revue systématique et méta-analyse

Pruebas diagnósticas rápidas frente a la RT-PCR para detectar infecciones por el virus del Ébola: revisión sistemática y metanálisis

اختبارات التشخيص السريع في مقابل RT-PCR لعدوى فيروس الإيبولا: مراجعة منهجية وتحليل تلوي

埃博拉病毒感染的快速诊断检测和 RT–PCR 检测:系统回顾和元分析

Экспресс-тесты в сравнении с ОТ-ПЦР для диагностики инфекций, вызванных вирусом Эбола: систематический обзор и метаанализ

Basilua Andre Muzembo a,, Kei Kitahara b, Ayumu Ohno b, Ngangu Patrick Ntontolo c, Nlandu Roger Ngatu d, Keinosuke Okamoto a, Shin-Ichi Miyoshi a
PMCID: PMC9243686  PMID: 35813519

Abstract

Objective

To evaluate the clinical accuracy of rapid diagnostic tests for the detection of Ebola virus.

Methods

We searched MEDLINE®, Embase® and Web of Science for articles published between 1976 and October 2021 reporting on clinical studies assessing the performance of Ebola virus rapid diagnostic tests compared with reverse transcription polymerase chain reaction (RT–PCR). We assessed study quality using the QUADAS-2 criteria. To estimate the pooled sensitivity and specificity of these rapid diagnostic tests, we used a bivariate random-effects meta-analysis.

Findings

Our search identified 113 unique studies, of which nine met the inclusion criteria. The studies were conducted in the Democratic Republic of the Congo, Guinea, Liberia and Sierra Leone and they evaluated 12 rapid diagnostic tests. We included eight studies in the meta-analysis. The pooled sensitivity and specificity of the rapid tests were 86% (95% confidence interval, CI: 80–91) and 95% (95% CI: 91–97), respectively. However, pooled sensitivity decreased to 83% (95% CI: 77–88) after removing outliers. Pooled sensitivity increased to 90% (95% CI: 82–94) when analysis was restricted to studies using the RT–PCR from altona Diagnostics as gold standard. Pooled sensitivity increased to 99% (95% CI: 67–100) when the analysis was restricted to studies using whole or capillary blood specimens.

Conclusion

The included rapid diagnostic tests did not detect all the Ebola virus disease cases. While the sensitivity and specificity of these tests are moderate, they are still valuable tools, especially useful for triage and detecting Ebola virus in remote areas.

Introduction

Ebola virus disease was first discovered in 1976 in the Democratic Republic of the Congo and South Sudan.1 This highly pathogenic disease is often fatal in humans; in past outbreaks the case fatality rate ranged from 25% (37/149) to 90% (128/143).1 At the initial stages of the disease, symptoms include fever, vomiting, diarrhoea, anorexia and fatigue.2 Diagnosing the disease on these symptoms alone is challenging, because they are similar to common endemic diseases present in Africa, such as typhoid fever, malaria and yellow fever.3,4

To confirm the diagnosis of Ebola virus disease, a positive result from a reverse transcription polymerase chain reaction (RT–PCR) test is required.5 However, lateral flow assays (that is, rapid diagnostic tests) are valuable tools to limit the spread of the disease since their fast turnaround time has the potential to trigger early outbreak alerts. For instance, researchers estimated that if a combination of rapid diagnostic tests and RT–PCR assays had been available during the 2013–2016 Ebola virus disease outbreak, the number of infections would have been up to a third less in Sierra Leone.6

Most Ebola outbreaks begin in remote or rural areas1 with limited hospital availability and trained clinicians. Laboratory equipment needed for diagnosis and trained equipment users are rarely available; it can take hours or days to get the RT–PCR results.7 If rapid diagnostic tests for the disease were readily available in high-risk outbreak areas, lives could be saved since the time between virus introduction into a community and implementation of countermeasures could be decreased.8

According to the World Health Organization (WHO), rapid diagnostic tests for Ebola virus should have a desired clinical sensitivity of > 98% and an acceptable clinical sensitivity of more than 95%.9 Since 1976, many Ebola virus rapid diagnostic tests have been developed, but researchers have not yet thoroughly assessed the evidence of their performance in clinical samples. The few rapid diagnostic tests that have been assessed in field conditions demonstrated uncertainty and variability in performance.6 We therefore conducted a meta-analysis to increase the evidence base of current rapid diagnostic tests detecting Ebola virus in suspected cases.

Methods

We conducted a systematic review and meta-analysis of studies that assessed the performance of rapid diagnostic tests for Ebola virus compared with RT–PCR. We followed the Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies.10 This review is registered with the International Prospective Register of Systematic Reviews (CRD42021278280).

We searched MEDLINE®, Embase® and Web of Science for articles published from 1976 to 7 October 2021. Search terms used are available in Box 1. We applied no language restrictions during the search. We also hand-searched the articles included in the reference lists of relevant studies, related key reviews and a book chapter on Ebola rapid diagnostic tests.8 The studies that we retrieved were exported to EndNote software X9 (Clarivate, Philadelphia, United States of America) and from there, we removed duplicated studies.

Box 1. Search terms and keywords used to identify studies on diagnostic accuracy of rapid tests for Ebola virus disease .

For Ebola virus disease we used the keywords:“Ebola virus disease” [MeSH Terms] OR “Ebola virus” [All fields] OR “Ebola” [All fields] OR “hemorrhagic fever” [All fields].

We combined these keywords with: “rapid diagnostic test” [MeSH Terms] OR “Rapid test” [All fields] OR “rapid assay” [All fields] OR “EBOV lateral flow assay” [All fields].

We further narrowed searches by including the following names for index test: “ReEBOV,” “QuickNaviTM-Ebola,” “eZYSCREEN,” “DSTL EVD lateral flow assay,” “OraQuick Ebola rapid antigen test kit,” “SD Q Line Ebola Zaire Ag” and “NMRC EBOV LFI.”

Two authors screened all the titles and abstracts identified through the search, and reviewed the full text of potentially relevant articles against the inclusion and exclusion criteria (Box 2). Each researcher was blind to the selection of the other researcher. We recorded reasons for excluding articles; disagreements were discussed and arbitrated by consensus.

Box 2. Inclusion and exclusion criteria used to identify studies on diagnostic accuracy of rapid tests for Ebola virus disease.

Inclusion criteria:

  • Patients: individuals with suspected Ebola virus disease;

  • Index test: classic and non-classic lateral flow assays carried out in any specimen to diagnose Ebola virus disease;

  • Reference standard: the study evaluates rapid diagnostic tests for Ebola virus against RT–PCR;

  • Outcomes: the study reports sensitivity and specificity of rapid diagnostic tests for Ebola virus or contains sufficient data to calculate sensitivity and specificity, by recreating the 2 × 2 diagnostic table.

Exclusion criteria:

  • Rapid PCR-based Ebola virus tests;

  • Review articles, editorials and non-clinical studies.

RT–PCR: reverse transcription polymerase chain reaction.

Data extraction

Two investigators independently extracted data from individual studies. We then compared extracted data and any disagreements were resolved through discussion.

Prior to data extraction, we designed a standardized data extraction form. Extracted data included: setting, study period, sample size, type of specimen, index test, reference standard and reported conflicts of interest. We also extracted reported sensitivity and specificity, raw data on true positives, false positives, false negatives and true negatives to recreate two-by-two tables.

Methodological assessment

Two authors independently evaluated the methodological quality of the included studies using the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) tool.11 We resolved any disagreements through consensus and further article reading.

Data synthesis and analysis

All statistical analyses were performed using Stata (version 16, StataCorp LP, College Station, USA). We applied a bivariate-effect model to calculate the pooled sensitivity, specificity, positive and negative likelihood ratios and the diagnostic odds ratio (OR). We generated forest plots of the sensitivity and specificity of each data point.

We conducted meta-analysis using the bivariate-effect models because of the heterogeneity expected between studies assessing diagnostic test accuracy.12 We also applied the bivariate-effect model to generate plot hierarchical summary receiver-operating characteristic curves.

Heterogeneity was assessed by visual inspection of forest plots of the sensitivity and specificity and the shape of the hierarchical summary receiver-operating characteristic curves.13

In our subgroup analysis, we stratified studies into seven subgroups based on: specimen type (serum alone, plasma alone, serum and plasma combined, whole blood and capillary blood combined); reference standard (RealStar Filovirus Screen RT–PCR Kit 1.0 [altona diagnostics GmbH, Hamburg, Germany, hereafter altona] alone and other RT–PCR tests such as Trombley assay);14 and exclusion of outliers (studies reporting that the rapid diagnostic test is 100% sensitive or specific). We also performed a subgroup analysis using only the ReEBOV™ Antigen Rapid Test kit (Corgenix Inc., Broomfield, USA; hereafter ReEBOV™) because it had enough data points to be pooled separately.

Results

The searches yielded 113 studies for screening after we removed duplicates (Fig. 1). Of the 36 full-text studies that we evaluated for eligibility, nine studies met the inclusion criteria.1523 Of these, eight were eligible for the meta-analysis.1519,2123 We excluded 10 studies because they were non-lateral flow assays, we were unable to reconstitute the two-by-two diagnostic table or had scarce evidence on clinical performance (e.g. studies showing only analytical performance of the rapid diagnostic test).2433

Fig. 1.

Fig. 1

Flowchart of the selection of articles included in study on the diagnostic accuracy of rapid tests for Ebola virus disease

Study characteristics

The included studies were cross-sectional studies published in English1518,2023 and French19 between 2015 and 2020. Sample size ranged from 105 to 928 tests performed. The studies were conducted in four countries: the Democratic Republic of the Congo,16 Guinea,15,19 Liberia20 and Sierra Leone.17,18,2123

Specimens were from either Ebola-suspected patients, those hospitalized at an Ebola treatment centre or from deceased people. One study tested postmortem oral swab specimens;20 all the studies tested blood specimens: capillary blood,22,23 whole blood,16,17,19 plasma15,17,18,2022 and serum.15,19 Haemolyzed specimens were also included.17 Four studies clearly stated that rapid tests were performed on stored serum15 or stored plasma samples.17,18,21 Further details on study characteristics are shown in Table 1. Five studies indicated participants’ age (available in the data repository).34

Table 1. Studies included in the systematic review and meta-analysis on the diagnostic accuracy of rapid tests for Ebola virus disease.

Study Country Sample size, no. Study design Reference standarda Specimen Year of specimen collection Industry funded Index test No. of samples
% (95% CI)
Prevalence, %
True positive False positive False negative True negative Reported sensitivity Reported specificity
Moran et al., 202015 Guinea 205 Cross-sectional RT–PCR (altona) Stored serum 2014–2015 No Fever Panel Antigen System 98 9 11 87 90 (83–95) 91 (83–96) 53
Ebola Antigen System 84 8 25 88 77 (68–84) 92 (84–96) 53
Ebola–Malaria Antigen duplex system 85 4 24 92 78 (69–85) 96 (89–99) 53
Makiala et al., 201916 Democratic Republic of the Congo 928 Cross-sectional GeneXpert® Ebola Whole blood 2018 No QuickNaviTM-Ebola 68 2 12 846 85 (75–92) 100 (99–100) 9
Wonderlyet al., 201917 Sierra Leone 428 Cross-sectional RT–PCR (altona) Whole blood 2015 No, but tests donated DEDIATEST Ebola 0 0 0 98 100 (96–100)
RT–PCR (altona) Stored plasma 2015 No, but tests donated SD Ebola Zaire Ag 109 2 20 196 85 (77–90) 99 (96–100)
RT–PCR (Trombley) Stored plasma 2015 No, but tests donated SD Ebola Zaire Ag 110 1 46 170 71 (63–78) 99 (97–100)
RT–PCR (altona) Stored plasma 2015 No, but tests donated ReEBOV™ Antigen Rapid Test kit 123 39 9 159 93 (88–97) 80 (74–86)
RT–PCR (Trombley) Stored plasma 2015 No, but tests donated ReEBOV™ Antigen Rapid Test kit 133 29 23 142 85 (79–90) 83 (77–88)
RT–PCR (altona) Stored plasma 2015 No, but tests donated One step Ebola test 125 39 2 158 98 (94–100) 80 (74–86)
RT–PCR (Trombley) Stored plasma 2015 No, but tests donated One step Ebola test 138 26 16 144 90 (84–94) 85 (78–90)
RT–PCR (altona) Stored plasma 2015 No, but tests donated DEDIATEST EBOLA 101 31 26 167 80 (72–86) 84 (79–89)
RT–PCR (Trombley) Stored plasma 2015 No, but tests donated DEDIATEST EBOLA 108 24 46 147 70 (62–77) 86 (80–91)
Colavita et al., 201818 Sierra Leone 210 Cross-sectional qRT–PCR (altona) Stored residual plasma samples 2014–2015 Yes EBOLA Ag K-SET 93 2 12 103 89 (81–94) 98 (93–100) 50
Gallais et al., 201719 Guinea 137 Cross-sectional qRT–PCR (altona; and Weidmann) Whole blood 2015 No eZYSCREEN® 32 1 17 87 65 (50–78) 99 (94–100) 36
144 Cross-sectional qRT–PCR (altona; and Weidmann) Serum 2015 No eZYSCREEN® 41 0 14 89 75 (61–85) 100 (96–100) 38
Phan et al., 2016b,20 Liberia 290 Cross-sectional rRT–PCR (Trombley) Plasma 2014–2015 No NMRC Ebola virus lateral flow immunoassay 88 (75–94) 98 (95–99)
237 Cross-sectional rRT–PCR (Trombley) Postmortem oral swabs 2014–2015 No NMRC Ebola virus lateral flow immunoassay 89 (57–98) 96 (93–98)
Boisen et al., 201621 Sierra Leone 176 Cross-sectional Ebola virus-specific qRT–PCRc Stored plasma 2014 Yes ReEBOV™ Antigen Rapid Test kit 72 2 44 58 62 (53–71) 97 (89–100) 53
Broadhurst et al., 201522 Sierra Leone 105 Cross-sectional RT–PCR (altona) Plasma (for the reference test) and capillary blood (for the rapid tests) 2015 No, but tests donated ReEBOV™ Antigen Rapid Test kit 28 6 0 71 100 (88–100) 92 (84–97) 27
277 Cross-sectional RT–PCR (altona) Whole blood 2015 No, but tests donated ReEBOV™ Antigen Rapid Test kit 45 18 0 214 100 (92–100) 92 (88–95) 16
Walker et al., 201523 Sierra Leone 131 Cross-sectional RT–PCR (altona) Capillary blood 2015 No Defence Science and Technology Laboratory Ebola virus disease rapid diagnostic test 15 9 107 0 100 (78–100) 92 (86–96) 12

CI: confidence interval; PCR: polymerase chain reaction; RT–PCR: reverse transcription PCR; rRT–PCR: real-time RT–PCR; qRT–PCR: quantitative RT–PCR.

a altona refers to RealStar Filovirus Screen RT–PCR Kit 1.0 (altona Diagnostics GmbH, Hamburg, Germany), Trombley refers to the Trombley assay,14 GeneXpert® Ebola (Cepheid, Sunnyvale, United States of America) is a specific RT-PCR that targets the glycoprotein and nucleoprotein genes, Weidmann refers to the Weidmann technique (that is, a quantitative one-step RT–PCR).19

b We were unable to accurately reconstitute the two-by-two table used to calculate specificity and sensitivity.

c RT–PCR specific for Ebola (while not specifically mentioned in the article, the authors cite the Trombley assay in their methods).

Index tests

The included studies evaluated 12 index tests: Reba™;17,21,22 QuickNavi™-Ebola (Denka Seiken, Tokyo, Japan);16 DEDIATEST EBOLA (Senova, Weimar, Germany);17 One step Ebola test (Intec, Xiamen, China);17 SD Ebola Zaire Ag (SD Biosensor, Suwon-si, Republic of Korea);17 EBOLA Ag K-SET (Coris BioConcept, Gembloux, Belgium);18 eZYSCREEN® (Vedalab, Alençon, France);19 NMRC Ebola virus lateral flow immunoassay (Naval Medical Research Center, Bethesda, USA);20 Defence Science and Technology Laboratory Ebola virus disease rapid diagnostic test (Defence Science and Technology Laboratory, Salisbury, United Kingdom of Great Britain and Northern Ireland);23 and three tests using the dual path platform from Chembio Diagnostics (Medford, USA): Fever Panel Antigen System,15 Ebola Antigen System15 and Ebola–Malaria Antigen duplex system.15 Further details on the tests are available from the data repository.34

The tests using the dual path platform are different from classic lateral flow assays as they contain a cartridge with a battery. ReEBOV™ should be stored at 2–8 °C, hence a cold chain is needed which could be a concern for use in remote field conditions.17,22 In addition, ReEBOV™, DEDIATEST EBOLA and SD Ebola Zaire Ag have been reported to have operational biosafety concerns when using the test.17 The NMRC Ebola virus lateral flow immunoassay could not yield readable results in samples containing red blood cells.20

All studies compared the rapid diagnostic test against RT–PCR, notably altona,15,1720,22,23 Trombley assay,17,20 GeneXpert® Ebola (Cepheid, Sunnyvale, USA)16, Ebola-specific quantitative RT–PCR (while not mentioned specifically in the article, the authors cite the Trombley assay in their methods),21 and Weidmann technique (that is, a quantitative one-step RT–PCR).19

Methodological assessment

The results of the quality assessments are available in the data repository.34 In the patient selection domain, 78% (7/9) of the studies had an unclear or high risk of bias, because these studies did not clearly specify random or consecutive recruitment of participants.15,17,18,2023 Furthermore, some studies had suboptimal study design (two studies),15,22 missing information on patients’ exclusion criteria (seven studies)15,17,18,20,21,22,23 and the use of stored blood samples collected for other purposes (three studies).15,17,18 Regarding the reference standard domain, one study was judged as having a high risk of bias for incorrect use of altona’s RT–PCR kit, by modifying the manufacturer’s instructions.22 We judged applicability concerns to be low in the patient selection, index test and reference standard domains.

Five (56%) studies explicitly stated no conflicts of interest.15,16,19,20,23 In two (22%) studies, authors acknowledged having potential conflicts of interest.18,21 Two (22%) studies reported receiving test kits from manufacturers but did not consider it as a conflict of interest.17,22

Meta-analysis

Nineteen data points covering 5332 tests performed were available to summarize the performance of rapid diagnostic tests for Ebola virus.

Table 2 shows the bivariate-effect model estimates for the pooled sensitivity, specificity, positive and negative likelihood ratios and the diagnostic OR. The pooled sensitivity and specificity were 86% (95% confidence interval, CI: 80–91) and 95% (95% CI: 91–97), respectively (Fig. 2). While sensitivity estimates varied widely from 62% to 100% across studies, the range for specificity estimates were narrower (80–100%; Fig. 3).

Table 2. Meta-analysis of rapid diagnostic tests for Ebola virus disease .

Group and study Data points
(no. of studies)
Sample size % (95% CI)
Pooled sensitivity Pooled specificity Positive likelihood ratio Negative likelihood ratio Diagnostic OR
All studies 15 19 , 21 23 19 (8) 5332 86 (80–91) 95 (91–97) 18.0 (9.9–32.9) 0.14 (0.10–0.21) 126 (66–240)
Sample type
Serum and plasma15,1719,21 14 (5) 3754 84 (77–89) 94 (89–97) 13.7 (7.4–25.5) 0.17 (0.12–0.24) 79 (42–148)
Plasma17,18,21 10 (3) 2995 85 (76–91) 93 (85–97) 11.9 (5.7–24.8) 0.16 (0.10–0.25) 74 (34–160)
Serum15,19 4 (2) 759 80 (73–86) 95 (89–98) 17.6 (7.5–41.0) 0.21 (0.15–0.28) 85 (38–190)
Whole blood and capillary blood16,19,22,23 5 (4) 1578 99 (67–100) 98 (91–99) 45.5 (11.3–182.7) 0.02 (0.00–0.45) 3011 (184–49 177)
Reference standard
altonaa,15,17,18,19,22,23 13 (6) 2925 90 (82–94) 94 (90–97) 15.9 (8.9–28.4) 0.11 (0.06–0.19) 146 (73–293)
Other standardsb,16,17,21 6 (3) 2407 78 (69–86) 96 (86–99) 21.5 (5.3–87.2) 0.22 (0.15–0.32) 96 (23–408)
Other
Removing outlier studiesc,15,16,17,18,19,21 15 (6) 4675 83 (77–88) 95 (90–98) 16.9 (8.4–33.9) 0.17 (0.13–0.24) 97 (48–196)
ReEBOV™ Antigen Rapid Test kit17,21,22 5 (3) 1215 95 (70–99) 89 (83–94) 8.9 (5.3–15.1) 0.06 (0.01–0.42) 157 (18–1403)

CI: confidence interval; OR: odds ratio.

a RealStar Filovirus Screen reverse transcription polymerase chain reaction Kit 1.0 (altona Diagnostics GmbH, Hamburg, Germany).

b Gold standard tests were Trombley assay,14 GeneXpert® Ebola (Cepheid, Sunnyvale, United States of America) and RT–PCR specific for Ebola (while not specifically mentioned in the article, the authors cite the Trombley assay in their methods).21

c We excluded studies reporting that rapid diagnostic tests were 100% sensitive or specific.

Fig. 2.

Forest plots of the sensitivities and specificities of Ebola virus rapid diagnostic tests compared with RT–PCR

CI: confidence interval; RT–PCR: reverse transcription polymerase chain reaction.

a RT–PCR specific for Ebola (while not specifically mentioned in the article, the authors cite the Trombley assay in their methods).21

Note: altona refers to RealStar Filovirus Screen reverse transcription polymerase chain reaction Kit 1.0 (altona Diagnostics GmbH, Hamburg, Germany).

Fig. 2

Fig. 3.

Hierarchical summary receiver-operating characteristic curves of the sensitivity and specificity of Ebola virus rapid diagnostic tests compared with RT–PCR

RT–PCR: reverse transcription polymerase chain reaction.

Notes: Each open circle represents the sensitivity and specificity of each included data point (n = 19). The size of each circle is proportional to the sample size of the study. A curve close to the upper left-hand corner of the hierarchical summary receiver-operating characteristic curve indicates better overall performance.35

Fig. 3

Subgroup analysis

In the subgroup analyses, the pooled specificity estimates were more consistent across subgroups compared with pooled subgroup sensitivity estimates (Table 2).

Sample type

Serum and plasma

We included five studies,15,1719,21 representing 3754 samples from 14 data points. Nine tests were assessed: ReEBOV™, DEDIATEST EBOLA, SD Ebola Zaire Ag, eZYSCREEN®, Ebola Antigen System, Ebola–Malaria Antigen duplex system, EBOLA Ag K-SET, One step Ebola test and Fever Panel Antigen System. The pooled sensitivity and specificity were 84% (95% CI: 77–89) and 94% (95% CI: 89–97), respectively.

Plasma

Studies assessing performance of rapid diagnostic tests on plasma samples used either ReEBOV™, DEDIATEST EBOLA, SD Ebola Zaire Ag, EBOLA Ag K-SET, One step Ebola test or Fever Panel Antigen System. We included three studies in this subgroup with 2995 specimens and 10 data points.17,18,21 The pooled sensitivity and specificity were 85% (95% CI: 76–91) and 93% (95% CI: 85–97), respectively.

Whole and capillary blood

Index tests used in this subgroup were eZYSCREEN®,19 QuickNavi-Ebola,16 ReEBOV™22 and Defence Science and Technology Laboratory Ebola virus disease rapid diagnostic test.23 Among five data points and 1578 specimens, the pooled sensitivity and specificity were 99% (95% CI: 67–100) and 98% (95% CI: 91–99), respectively.

Reference standard

altona

We restricted the analysis to 13 data points (2925 specimens tested) where the altona RT–PCR kit had been used as the gold standard.15,1719,22,23 Index tests used in this subgroup were eZYSCREEN®, Ebola Antigen System, Ebola–Malaria Antigen duplex system, DEDIATEST EBOLA, SD Ebola Zaire Ag, EBOLA Ag K-SET, Fever Panel Antigen System, ReEBOV™, One step Ebola test and Defence Science and Technology Laboratory Ebola virus disease rapid diagnostic test. The pooled sensitivity increased to 90% (95% CI: 82–94) compared with the overall estimates (86%; 95% CI: 80–91), while the pooled specificity (94%; 95% CI: 90–97) was similar (95%; 95% CI: 91–97).

Other RT–PCR tests

We included three studies using the following reference standards: Ebola-specific qRT–PCR,21 Trombley17 and GeneXpert® Ebola assays.16 Index tests were ReEBOV™, DEDIATEST EBOLA, SD Zaire Ag, QuickNavi-Ebola and One step Ebola test.

Among six data points and 2407 specimens tested, the pooled sensitivity decreased to 78% (95% CI: 69–86). However, the pooled specificity (96%; 95% CI: 86–99) was comparable to the overall estimates.

Removing outliers

We excluded four data points that had sensitivity22,23 or specificity19 of 100%. We found that the pooled sensitivity slightly decreased to 83% (95% CI: 77–88) compared with the overall sensitivity estimates of 86% (95% CI: 80–91). However, the pooled specificity was the same as the overall specificity estimates; 95% (95% CI: 90–98) versus 95% (95% CI: 91–97).

ReEBOV™

When we considered only studies that used ReEBOV™ as index test (five data points with 1215 tests performed)17,21,22, the pooled sensitivity and specificity were 95% (95% CI: 70–99) and 89% (95% CI: 83–94), respectively (Fig. 4).

Fig. 4.

Forest plots of the sensitivities and specificities of ReEBOV™ Antigen Rapid Test kit for the detection of Ebola virus

CI: confidence interval; RT–PCR: reverse transcription polymerase chain reaction; qRT–PCR: quantitative RT–PCR.

a RT–PCR specific for Ebola (while not specifically mentioned in the article, the authors cite the Trombley assay in their methods).21

Note: altona refers to RealStar Filovirus Screen reverse transcription polymerase chain reaction Kit 1.0 (altona Diagnostics GmbH, Hamburg, Germany).

Fig. 4

Discussion

In this study, we conducted meta-analyses on clinical accuracy studies to assess the performance of rapid diagnostic tests to detect Ebola virus in individuals suspected of having the disease. Compared with the gold standard RT–PCR, the overall pooled sensitivity for rapid tests was 86%, falling short of a desired sensitivity of > 98% and an acceptable sensitivity of > 95% listed in a WHO target product profile document released during the 2013–2016 Ebola virus disease outbreak.9 Furthermore, we show that the overall pooled specificity was 95%, also lower than WHO’s recommended level of > 99%.9

While the clinical value for Ebola virus screening to contain the outbreak is indisputable, our findings suggest that better performing rapid diagnostic tests in field conditions are needed. Our results indicate that current tests miss 14% of cases, which is a considerable proportion because of the contagiousness and high mortality of Ebola virus disease. False-negative results should be minimized to the lowest level possible, since false-negative individuals might infect other people. In hospitals, false-negative patients might be treated with less precaution than positive patients and hence the likelihood of infecting health-care workers and other patients is greater. Allowing false-negative patients to wait at home for a confirmatory RT–PCR test increases the risk of infecting people in the community. In addition, false-negative patients will not be included in contact tracing, which might lead to the transmission chain being sustained. Of note is that a low viral load in the specimen could also lead to a false-negative result, even with a rapid test with high sensitivity.36

False positivity can have severe implications for false-positive individuals and their families since they would be subject to unnecessary quarantines. These individuals can also be exposed to the potential Ebola patients when waiting for the confirmation of the diagnosis.

When assessing in which type of sample the rapid test performed best, we found that tests made on whole or capillary blood had the highest sensitivity, specificity, likelihood ratios and diagnostic OR. Using whole or capillary blood has the advantage that blood centrifugation is not required, which would reduce the turnaround time and make the test more accessible in remote field settings. However, this subgroup analysis included only five data points, from four studies,16,19,22,23 and two of the data points were from a study22 that has been a subject of debate.3740 Hence, further field studies are required to confirm results.

Using a different gold standard also affected the results. The six data points with higher sensitivity all used altona as the gold standard.15,17,22,23 We also noted that pooled sensitivity was higher in studies using altona (90%) compared with other gold standards (78%). However, only six data points were available for other gold standards versus 13 for altona. It is unclear why using altona yielded a sensitivity superior to that of other gold standards. Reduced sensitivity of altona to specimens with cycle threshold values above 30 (i.e. low viral loads) have been observed.5,41 This reduced sensitivity of altona may affect the interpretation of our pooled estimates. Our pooled sensitivity might have been underestimated, since altona may fail to detect positive samples with low viral loads. However, we cannot rule out the possibility that studies using altona might also overestimate the sensitivity of rapid tests, since one study that used altona reported an unusual sensitivity of 100%.22

The review identified sufficient data points for assessing ReEBOV™ performance. This test received an emergency use authorization from the United States Food and Drug Administration (FDA) and WHO during the 2013–2016 Ebola outbreak.26 But, in 2018, FDA revoked this authorization when the new manufacturer (Zalgen Laboratories) that had acquired the company failed to reproduce the claimed test accuracy of ReEBOV™.42 The claimed sensitivity and specificity of the test were 91.8% and 84.6%, respectively.6 Our results are in line with the suboptimal performance: we showed a pooled sensitivity of 95% with a wide confidence interval (95% CI: 70–99), suggesting that the result is overestimated due to included outlier studies.19,22,23 The pooled specificity was also below WHO criteria of 99% analytical specificity.9

This study has some limitations. First, our meta-analysis included some studies with some methodological limitations and studies where tests’ accuracy could have been overestimated. However, this concern was addressed by conducting subgroup analyses. For instance, we assessed whether the pooled estimates differed by removing outliers in the meta-analysis. This approach did indeed change the pooled sensitivity but not its specificity. Second, synthesizing the included studies in one pooled estimate of sensitivity and specificity could be inaccurate as there was substantial variation in the used cut-off for RT–PCR cycle threshold (data repository),34 which limits comparability of rapid tests. These issues are illustrated by pooling only data from ReEBOV™ studies. The results showed a suboptimum pooled sensitivity with a great uncertainty.

Third, some studies used stored blood samples. Using frozen blood samples that were processed through freeze–thaw cycles may have compromised the tests’ sensitivity. Nevertheless, this explanation is unlikely since a newly published study conducted on patients also demonstrated that the sensitivity of three rapid tests (QuickNavi-Ebola, OraQuick Ebola Rapid Antigen Test (OraSure Technologies, Bethlehem, USA) and EBOLA Ag K-SET rapid test) varied, from 40% to 87%.43 Thus, for Ebola rapid tests with potential acceptable sensitivity, well designed clinical studies with larger sample sizes are necessary for an adequate assessment of their current performance.

Finally, we could not perform subgroup analysis comparing test performance by symptoms duration because of lack of data stratified by timing of the tests. Therefore, future field studies need to evaluate the performance of these tests in relation to symptoms duration. As an example, rapid diagnostic tests for severe acute respiratory syndrome coronavirus 2 are more sensitive (83.8%) when used within 7 days of symptoms onset than when used at later than 7 days (sensitivity of 61.5%).44 These limitations, however, would not modify the usefulness of Ebola rapid diagnostic tests as a tool for separating suspected Ebola patients while they are waiting for their RT–PCR results.

The strength of our study is that our literature was extensive, without any language restrictions, although only studies published in English and French were included. However, some studies were excluded because of low evidence on clinical performance, and studies could have been missed by our search strategy.

In conclusion, the results from this meta-analysis suggest that currently there is no commercial rapid diagnostic test for Ebola virus disease that has sufficient sensitivity and specificity to meet WHO standards. Despite the suboptimal performance, these tests still have clinical value because of their rapid turnaround time. Clinicians, especially those in settings where RT–PCR tests are not immediately available, should be aware of the existence, availability and limitations of the rapid tests. A negative result is unreliable in a subject highly suspected of having Ebola virus disease and the result must be confirmed using RT–PCR. Current commercially available tests include ReEBOV™, SD Zaire Ag, OraQuick, Ebola Antigen System and QuickNavi-Ebola,45 and health-care workers can procure rapid tests, such as OraQuick, from the US Centers for Disease Control and Prevention or WHO.46 Our findings stress a great need for more accurate rapid diagnostic tests for Ebola virus. Tests with improved performance should undergo additional field testing, if blood samples are available or conducted during future outbreaks.

Acknowledgements

We thank Matthew James Mclaughlin, Martin Barnabas and Mansongi Biyela Carine. Ngangu Patrick Ntontolo is also affiliated to the Department of Family Medicine and Primary Health, Protestant University of Congo, Kinshasa, Democratic Republic of the Congo.

Funding:

This work was supported by JSPS KAKENHI (Grants-in-Aid for Scientific Research; Grant Number 19K19467) and the Program of the Japan Initiative for Global Research Network on Infectious Diseases, JP22wm0125004, from the Ministry of Education, Culture, Sports, Science and Technology in Japan, and Japan Agency for Medical Research and Development.

Competing interests:

None declared.

References

  • 1.Ebola virus disease. Geneva: World Health Organization; 2018. Available from: https://www.who.int/news-room/fact-sheets/detail/ebola-virus-disease [cited 2021 Oct 2].
  • 2.Bah EI, Lamah MC, Fletcher T, Jacob ST, Brett-Major DM, Sall AA, et al. Clinical presentation of patients with Ebola virus disease in Conakry, Guinea. N Engl J Med. 2015. Jan 1;372(1):40–7. 10.1056/NEJMoa1411249 [DOI] [PubMed] [Google Scholar]
  • 3.Baize S, Pannetier D, Oestereich L, Rieger T, Koivogui L, Magassouba N, et al. Emergence of Zaire Ebola virus disease in Guinea. N Engl J Med. 2014. Oct 9;371(15):1418–25. 10.1056/NEJMoa1404505 [DOI] [PubMed] [Google Scholar]
  • 4.Hartley MA, Young A, Tran AM, Okoni-Williams HH, Suma M, Mancuso B, et al. Predicting Ebola infection: a malaria-sensitive triage score for Ebola virus disease. PLoS Negl Trop Dis. 2017. Feb 23;11(2):e0005356. 10.1371/journal.pntd.0005356 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Janvier F, Gorbatch S, Queval L, Top J, Vigier C, Cotte J, et al. Difficulties of interpretation of Zaire Ebola Virus PCR results and implication in the field. J Clin Virol. 2015. Jun;67:36–7. 10.1016/j.jcv.2015.04.001 [DOI] [PubMed] [Google Scholar]
  • 6.Nouvellet P, Garske T, Mills HL, Nedjati-Gilani G, Hinsley W, Blake IM, et al. The role of rapid diagnostics in managing Ebola epidemics. Nature. 2015. Dec 3;528(7580):S109–16. 10.1038/nature16041 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Rouquet P, Froment JM, Bermejo M, Kilbourn A, Karesh W, Reed P, et al. Wild animal mortality monitoring and human Ebola outbreaks, Gabon and Republic of Congo, 2001–2003. Emerg Infect Dis. 2005. Feb;11(2):283–90. 10.3201/eid1102.040533 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Catherine H, Colin B. A rapid diagnostic test for Ebola virus disease. In: Mabey KAD, editor. Revolutionizing tropical medicine: point-of-care tests, new imaging technologies and digital health. Hoboken: John Wiley & Sons, Inc; 2019. p. 768. [Google Scholar]
  • 9.Target product profile for Zaïre ebolavirus rapid, simple test to be used in the control of the Ebola outbreak in West Africa. Geneva: World Health Organization; 2014. Available from: https://www.finddx.org/wp-content/uploads/2019/03/WHO-TPP-ebola-2014.pdf [cited 2021 Oct 2].
  • 10.McInnes MDF, Moher D, Thombs BD, McGrath TA, Bossuyt PM, Clifford T, et al. ; and the PRISMA-DTA Group. Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies. JAMA. 2018. Jan 23;319(4):388–96. 10.1001/jama.2017.19163 [DOI] [PubMed] [Google Scholar]
  • 11.Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. ; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011. Oct 18;155(8):529–36. 10.7326/0003-4819-155-8-201110180-00009 [DOI] [PubMed] [Google Scholar]
  • 12.Leeflang MM, Deeks JJ, Takwoingi Y, Macaskill P. Cochrane diagnostic test accuracy reviews. Syst Rev. 2013. Oct 7;2(1):82. 10.1186/2046-4053-2-82 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Leeflang MM. Systematic reviews and meta-analyses of diagnostic test accuracy. Clin Microbiol Infect. 2014. Feb;20(2):105–13. 10.1111/1469-0691.12474 [DOI] [PubMed] [Google Scholar]
  • 14.Trombley AR, Wachter L, Garrison J, Buckley-Beason VA, Jahrling J, Hensley LE, et al. Comprehensive panel of real-time TaqMan polymerase chain reaction assays for detection and absolute quantification of filoviruses, arenaviruses, and New World hantaviruses. Am J Trop Med Hyg. 2010. May;82(5):954–60. 10.4269/ajtmh.2010.09-0636 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Moran Z, Rodriguez W, Ahmadou D, Soropogui B, Magassouba NF, Kelly-Cirino C, et al. Comparative performance study of three Ebola rapid diagnostic tests in Guinea. BMC Infect Dis. 2020. Sep 15;20(1):670. 10.1186/s12879-020-05339-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Makiala S, Mukadi D, De Weggheleire A, Muramatsu S, Kato D, Inano K, et al. Clinical evaluation of QuickNaviTM-Ebola in the 2018 outbreak of Ebola virus disease in the Democratic Republic of the Congo. Viruses. 2019. Jun 28;11(7):589. 10.3390/v11070589 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Wonderly B, Jones S, Gatton ML, Barber J, Killip M, Hudson C, et al. Comparative performance of four rapid Ebola antigen-detection lateral flow immunoassays during the 2014-2016 Ebola epidemic in West Africa. PLoS One. 2019. Mar 7;14(3):e0212113. 10.1371/journal.pone.0212113 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Colavita F, Biava M, Mertens P, Gilleman Q, Borlon C, Delli Guanti M, et al. EBOLA Ag K-SET rapid test: field evaluation in Sierra Leone. Clin Microbiol Infect. 2018. Jun;24(6):653–7. 10.1016/j.cmi.2017.10.019 [DOI] [PubMed] [Google Scholar]
  • 19.Gallais F, Gay-Andrieu F, Picot V, Magassouba N, Mély S, Peyrefitte CN, et al. Validation sur le terrain du nouveau test de diagnostic rapide Ebola eZYSCREEN®. Bull Soc Pathol Exot. 2017. Feb;110(1):38–48. French. 10.1007/s13149-016-0540-z [DOI] [PubMed] [Google Scholar]
  • 20.Phan JC, Pettitt J, George JS, Fakoli LS 3rd, Taweh FM, Bateman SL, et al. Lateral flow immunoassays for Ebola virus disease detection in Liberia. J Infect Dis. 2016. Oct 15;214 suppl 3:S222–8. 10.1093/infdis/jiw251 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Boisen ML, Cross RW, Hartnett JN, Goba A, Momoh M, Fullah M, et al. Field validation of the ReEBOV antigen rapid test for point-of-care diagnosis of Ebola virus infection. J Infect Dis. 2016. Oct 15;214 suppl 3:S203–9. 10.1093/infdis/jiw261 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Broadhurst MJ, Kelly JD, Miller A, Semper A, Bailey D, Groppelli E, et al. ReEBOV antigen rapid test kit for point-of-care and laboratory-based testing for Ebola virus disease: a field validation study. Lancet. 2015. Aug 29;386(9996):867–74. 10.1016/S0140-6736(15)61042-X [DOI] [PubMed] [Google Scholar]
  • 23.Walker NF, Brown CS, Youkee D, Baker P, Williams N, Kalawa A, et al. Evaluation of a point-of-care blood test for identification of Ebola virus disease at Ebola holding units, Western Area, Sierra Leone, January to February 2015. Euro Surveill. 2015. Mar 26;20(12):21073. 10.2807/1560-7917.ES2015.20.12.21073 [DOI] [PubMed] [Google Scholar]
  • 24.Sebba D, Lastovich AG, Kuroda M, Fallows E, Johnson J, Ahouidi A, et al. A point-of-care diagnostic for differentiating Ebola from endemic febrile diseases. Sci Transl Med. 2018. Dec 12;10(471):eaat0944. 10.1126/scitranslmed.aat0944 [DOI] [PubMed] [Google Scholar]
  • 25.Couturier C, Wada A, Louis K, Mistretta M, Beitz B, Povogui M, et al. Characterization and analytical validation of a new antigenic rapid diagnostic test for Ebola virus disease detection. PLoS Negl Trop Dis. 2020. Jan 17;14(1):e0007965. 10.1371/journal.pntd.0007965 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Cross RW, Boisen ML, Millett MM, Nelson DS, Oottamasathien D, Hartnett JN, et al. Analytical validation of the ReEBOV antigen rapid test for point-of-care diagnosis of Ebola virus infection. J Infect Dis. 2016. Oct 15;214 suppl 3:S210–7. 10.1093/infdis/jiw293 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.DeMers HL, He S, Pandit SG, Hannah EE, Zhang Z, Yan F, et al. Development of an antigen detection assay for early point-of-care diagnosis of Zaire ebolavirus. PLoS Negl Trop Dis. 2020. Nov 3;14(11):e0008817. 10.1371/journal.pntd.0008817 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Leski TA, Ansumana R, Taitt CR, Lamin JM, Bangura U, Lahai J, et al. Use of the FilmArray System for Detection of Zaire ebolavirus in a Small Hospital in Bo, Sierra Leone. J Clin Microbiol. 2015. Jul;53(7):2368–70. 10.1128/JCM.00527-15 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Fontes CM, Lipes BD, Liu J, Agans KN, Yan A, Shi P, et al. Ultrasensitive point-of-care immunoassay for secreted glycoprotein detects Ebola infection earlier than PCR. Sci Transl Med. 2021. Apr 7;13(588):eabd9696. 10.1126/scitranslmed.abd9696 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Gay-Andrieu F, Magassouba N, Picot V, Phillips CL, Peyrefitte CN, Dacosta B, et al. Clinical evaluation of the BioFire FilmArray® BioThreat-E test for the diagnosis of Ebola Virus Disease in Guinea. J Clin Virol. 2017. Jul;92:20–4. 10.1016/j.jcv.2017.04.015 [DOI] [PubMed] [Google Scholar]
  • 31.Weller SA, Bailey D, Matthews S, Lumley S, Sweed A, Ready D, et al. Evaluation of the Biofire FilmArray BioThreat-E Test (v2.5) for Rapid Identification of Ebola Virus Disease in Heat-Treated Blood Samples Obtained in Sierra Leone and the United Kingdom. J Clin Microbiol. 2016. Jan;54(1):114–9. 10.1128/JCM.02287-15 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Jean Louis F, Huang JY, Nebie YK, Koivogui L, Jayaraman G, Abiola N, et al. Implementation of broad screening with Ebola rapid diagnostic tests in Forécariah, Guinea. Afr J Lab Med. 2017. Mar 31;6(1):484. 10.4102/ajlm.v6i1.484 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Yang M, Ke Y, Wang X, Ren H, Liu W, Lu H, et al. Development and evaluation of a rapid and sensitive EBOV-RPA test for rapid diagnosis of Ebola virus disease. Sci Rep. 2016. Jun 1;6(1):26943. 10.1038/srep26943 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Muzembo BA, Kitahara K, Ohno A, Ntontolo NP, Ngatu NR, Okamoto K, et al. Diagnostic accuracy of Ebola virus lateral flow assays compared with reverse transcription polymerase chain reaction: a systematic review and meta-analysis [data repository]. Meyrin: Zenodo; 2022. 10.5281/zenodo.6586604 [DOI]
  • 35.Chartrand C, Leeflang MM, Minion J, Brewer T, Pai M. Accuracy of rapid influenza diagnostic tests: a meta-analysis. Ann Intern Med. 2012. Apr 3;156(7):500–11. 10.7326/0003-4819-156-7-201204030-00403 [DOI] [PubMed] [Google Scholar]
  • 36.Laboratory diagnosis of Ebola virus disease. Geneva: World Health Organization; 2014. Available from: http://apps.who.int/iris/bitstream/handle/10665/134009/WHO_?sequence=1 [cited 2021 Sep 15].
  • 37.Broadhurst MJ, Semper A, Bailey D, Pollock NR. ReEBOV antigen rapid test kit for Ebola – authors’ reply. Lancet. 2015. Dec 5;386(10010):2255–6. 10.1016/S0140-6736(15)01109-5 [DOI] [PubMed] [Google Scholar]
  • 38.Ölschläger S, Heß M. ReEBOV antigen rapid test kit for Ebola. Lancet. 2015. Dec 5;386(10010):2254. 10.1016/S0140-6736(15)01106-X [DOI] [PubMed] [Google Scholar]
  • 39.Piriou E, Chua AC, Sprecher AG. ReEBOV antigen rapid test kit for Ebola. Lancet. 2015. Dec 5;386(10010):2255. 10.1016/S0140-6736(15)01108-3 [DOI] [PubMed] [Google Scholar]
  • 40.Urassa W, Meurant R, Wood D. ReEBOV antigen rapid test kit for Ebola. Lancet. 2015. Dec 5;386(10010):2253–4. 10.1016/S0140-6736(15)01105-8 [DOI] [PubMed] [Google Scholar]
  • 41.Janvier F, Sagui E, Foissaud V. ReEBOV Antigen rapid test kit for ebola. Lancet. 2015. Dec 5;386(10010):2254–5. 10.1016/S0140-6736(15)01107-1 [DOI] [PubMed] [Google Scholar]
  • 42.Revocation of authorization of emergency use of an in vitro diagnostic device for detection of Ebola Virus. Docket No. FDA-2015-N-0126. Washington, DC: Federal Register; 2018. Available from: https://www.federalregister.gov/documents/2018/08/02/2018-16537/revocation-of-authorization-of-emergency-use-of-an-in-vitro-diagnostic-device-for-detection-of-ebola [cited 2021 Oct 15].
  • 43.Mukadi-Bamuleka D, Bulabula-Penge J, De Weggheleire A, Jacobs BKM, Edidi-Atani F, Mambu-Mbika F, et al. Field performance of three Ebola rapid diagnostic tests used during the 2018-20 outbreak in the eastern Democratic Republic of the Congo: a retrospective, multicentre observational study. Lancet Infect Dis. 2022. Mar 14;S1473-3099(21)00675-7. 10.1016/S1473-3099(21)00675-7 [DOI] [PubMed] [Google Scholar]
  • 44.Brümmer LE, Katzenschlager S, Gaeddert M, Erdmann C, Schmitz S, Bota M, et al. Accuracy of novel antigen rapid diagnostics for SARS-CoV-2: a living systematic review and meta-analysis. PLoS Med. 2021. Aug 12;18(8):e1003735. 10.1371/journal.pmed.1003735 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Castillo-León J, Trebbien R, Castillo JJ, Svendsen WE. Commercially available rapid diagnostic tests for the detection of high priority pathogens: status and challenges. Analyst (Lond). 2021. Jun 14;146(12):3750–76. 10.1039/D0AN02286A [DOI] [PubMed] [Google Scholar]
  • 46.Cnops L, De Smet B, Mbala-Kingebeni P, van Griensven J, Ahuka-Mundeke S, Ariën KK. Where are the Ebola diagnostics from last time? Nature. 2019. Jan;565(7740):419–21. 10.1038/d41586-019-00212-y [DOI] [PubMed] [Google Scholar]

Articles from Bulletin of the World Health Organization are provided here courtesy of World Health Organization

RESOURCES