Diagnostic accuracy of metagenomic next-generation sequencing in diagnosing infectious diseases: a meta-analysis

Abstract

Many common pathogens are difficult or impossible to detect using conventional microbiological tests. However, the rapid and untargeted nature of metagenomic next-generation sequencing (mNGS) appears to be a promising alternative. To perform a systematic review and meta-analysis of evidence regarding the diagnostic accuracy of mNGS in patients with infectious diseases. An electronic literature search of Embase, PubMed and Scopus databases was performed. Quality was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. Summary receiver operating characteristics (sROC) and the area under the curve (AUC) were calculated; A random-effects model was used in cases of heterogeneity. A total of 20 papers were eligible for inclusion and synthesis. The sensitivity and specificity of diagnostic mNGS were 75% and 68%, respectively. The AUC from the SROC was 85%, corresponding to excellent performance. mNGS demonstrated satisfactory diagnostic performance for infections and yielded an overall detection rate superior to conventional methods.

Introduction

Infectious diseases are a leading cause of morbidity and death worldwide. However, early detection of pathogens may be challenging in many clinical scenarios. Moreover, many common pathogens are difficult or impossible to detect using conventional microbiological tests (e.g. culture, smears, immunological tests and polymerase chain reaction (PCR) assays), which makes precise diagnosis challenging. Culture methods are time-consuming and have strict limitations. Smears, immunological tests and multiplex PCR assays will only test for a specific pathogen that must be identified by the clinicians before the test is performed¹. The administration of broad-spectrum antibiotics in the absence of pathogen identification, despite comprehensive testing methods, frequently confounds specific diagnoses, which could lead to more toxic and less effective antimicrobial therapy².

Metagenomic next-generation sequencing (mNGS) is a high-throughput method that can directly detect pathogens (i.e., bacterial species) in clinical specimens and analyze functional genes without the need to pre-select target sequences³. It is especially suitable for novel, rare, and atypical etiologies of complicated infectious diseases. Due to characteristics of speed, sensitivity, culture-independent, hypothesis-free, and unbiased pathogen detection, mNGS may become a routine diagnostic tool, partly replacing more traditional detection methods⁴. Some investigators have even decided to upgrade their model, known as ‘Microbial Index of Pathogenic Bacteria’, by implementing whole metagenome sequencing data for species and strain- level identification of patho-genic bacteria⁵. To date, mNGS has been applied in the diagnosis of pathogens in bloodstream infections^6,7, respiratory tract infections^8,9, tuberculosis¹⁰, meningitis and encephalitis^11,12. However, these studies were limited by small sample sizes. As such, we aimed to perform a systematic accuracy review of diagnostic tests and a meta-analysis to identify, quality appraise, and synthesize the available evidence to inform the implementation of mNGS in diagnosing infectious diseases.

Methods

Literature search

Results of the present systematic review and meta-analysis are reported in accordance with the Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy studies (PRISMA-DTA)¹³. A comprehensive electronic literature search of Embase, PubMed and Scopus databases was performed for relevant studies published up to December 31, 2021. The medical subject heading (i.e., ‘MeSH’) search terms included 'infection' and 'Metagenomic Next-Generation Sequencing'. The reference lists of retrieved studies were also manually searched for additional, possibly eligible studies. Three reviewers independently screened the titles, and abstracts and obtained the full-text of potentially relevant studies; any disagreements were resolved by consensus discussion.

Inclusion and exclusion criteria

Cross-sectional and cohort studies including patients with clinically suspicious infection (including meningitis, bacteremia, fungemia, osteomyelitis, septic arthritis) for whom diagnostic test accuracy data for mNGS were included. Only English language articles were eligible. No restrictions were imposed on the age of the study population.

Studies reporting insufficient data to construct a 2 × 2 table (true positive, false positive, true negative, and false negative), those based on non-human samples, investigations reporting duplicate information already reported in other publications; those not reporting the reference infection diagnostic criteria; or reporting one specific pathogene and abstracts, conference presentations, case reports and letters were excluded.

Data extraction

Data were independently extracted by two reviewers using a standardized protocol and prespecified data extraction forms for diagnostic test accuracy studies¹⁴. Disagreements were resolved by a third investigator. Information regarding study characteristics (including population, period, design, country, and sample size) was extracted.

Quality assessment

The quality of the included studies was independently assessed by two reviewers, using the revised Quality Assessment of Diagnostic Accuracy Studies-2 tool¹⁵.

Statistical analysis

For each study, pooled specificity, pooled sensitivity, pooled negative predictive value (NPV), and pooled positive predictive value (PPV) were calculated based on a bivariate meta-analysis model¹⁶. They are presented as graphical representations in which the boxes mark the values and the horizontal lines represent the confidence intervals (CIs). A summary receiver operating characteristic curve (sROC) was drawn, and the area under the curve (AUC) was calculated to determine the performance of a diagnostic test¹⁷. The criteria for AUC classification were as follows: 0.50 (failure), 0.60–0.70 (poor), 0.70–0.80 (fair), 0.80–0.90 (good) and 0.90–1 (excellent). The Q* index and corresponding standard error (SE), is an additional measure which is the point on the sROC curve closest to the ideal left top-left corner (where summary sensitivities (SN) and summary specificities (SP) meet).

Heterogeneity was evaluated by calculating the I²¹⁸ statistic. DerSimonian and Laird random effects models¹⁹, which include both between and within study heterogeneity, were used to generate summary SP, SN, negative likelihood ratios (− LR), positive likelihood ratios (+ LR) and diagnostic odds ratio (DOR). Heterogeneity was also assessed using forest plots of sensitivity and specificity across studies for variability of study estimates in the hierarchical sROC model (meta-regression). A Cochrane’s-Q p < 0.10 and I² > 50% indicated significant heterogeneity, of SN and SP and LRs, respectively. Furthermore, the risk of bias in the included studies was assessed by using the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS–2) tool¹⁵. Publication bias was assessed by using a funnel plot and Deeks test²⁰. Statistical analysis was performed using MetaDisc version 1.4, Stata version 12.0 (StataCorp LLC, College Station, TX, USA), and Review Manager 5 (version 5.3) (R Foundation for Statistical Computing, Vienna, Austria)²¹.

Results

Characteristics of the included studies

After removing duplicate publications and references checking for additional, potentially eligible studies, a total of 891 studies were screened. Of these, 77 separate publications underwent full text review, resulting in 20 studies included in this systematic review. The study selection process is illustrated in the PRISMA flow diagram (Fig. 1). Six studies were performed in high-income countries, whereas 14 were conducted in low and middle-income countries. The study included 2716 participants. Twelve of the studies were retrospective, and eight were prospective in design. Among the enrolled studies, participants were predominantly adults. The included studies were published between 2017 and 2021. (Table 1).

Figure 1.

Flow diagram of study selection process.

Table 1.

Characteristics of Included Studies.

	Research type	Reference standard	Participants	Age	Country	Sample type
Zhang32	Retrospective	Conventional test	135	Paediatric	China	Blood/CSF
Wang30	Retrospective	Conventional test	55	Adults	China	Pulmonary biopsy/BALF
Rossoff33	Retrospective	Conventional test	79	Pediatric	USA	Plasma
Miller34	Retrospective	Conventional test	95	Pediatric + adult	USA	CSF
Blauwkamp6	Prospective	Conventional test	348	Adults	USA	Plasma
Miao24	Retrospective	Conventional test	511	Adults	Chian	Clinical specimensa
Madi35	Retrospective	Conventional test	86	Adults	Kuwait	Respiratory samplesb
Parize36	Prospective	Conventional test	101	Adults	France	Plasma/nasopharyngeal swabs/ biological fluid
Xing28	Prospective	Conventional test	213	Adults	Chian	CSF
Wang37	Prospective	Clinical diagnosis	63	Adults	Chian	Joint fluid
Chen38	Retrospective	Conventional test	235	Adults	Chian	BALF
Lian39	Retrospective	Clinical diagnosis	51	Adults	Chian	BALF
Peng40	Retrospective	Clinical diagnosis	49	Adults	Chian	BALF
Sun41	Prospective	Conventional test	44	Adults	Chian	BALF
Zhou42	Prospective	Conventional test	159	Adults	Chian	BALF
Chen43	Prospective	Conventional test	162	Adults	Chian	BALF
Jing44	Retrospective	Clinical diagnosis	209	pediatric + adult	Chian	Plasma
Ogawa45	Retrospective	Conventional test	23	Adults	Japan	Tissue
Lee46	Retrospective	Clinical diagnosis	54	Pediatric	USA	Plasma
Cai47	Prospective	Clinical diagnosis	44	Adults	China	Periprosthetic tissues

a: Specimens included bronchoalveolar lavage fluid (BALF), cerebrospinal fluid (CSF), sputum, pleural fluid, tissue, pus, blood, ascetic fluid, bile, secretion, urine, herpes fluid, bone marrow, throat swab, pericardial fluid and saliva; b: The respiratory samples included nasopharyngeal aspirates/wash, nasopharyngeal swab, BALF, tracheal aspirates, sputum, throat swabs, and nasal swabs.

Risk of bias

The risk of bias and applicability concerns according to the QUADAS-2 tool are shown in Fig. 2. All studies demonstrated unclear or low risks of bias.

Figure 2.

Risk of bias and applicability concerns summary.

Meta‑analysis

Heterogeneity test

No correlation was found between sensitivity logarithm and 1-specificity logarithm (Spearman correlation 0.0345 (p = 0.092)). Analysis revealed no threshold effect among the included studies. As is common with meta-analyses investigating results of diagnostic accuracy research, remarkable heterogeneity was present, with sensitivity and specificity estimates varying widely. The Cochran-Q for the pooled DOR was 207.82, (p = 0.00, I² = 88.5%). This suggests that a non-threshold effect was the cause of heterogeneity and a random effect model was used in further analysis.

Random effect model analysis results

The reported diagnostic sensitivity of the mNGS in infectious diseases ranged between 21% and 100% (Fig. 3a), and the reported specificity ranged from 14% to 100% (Fig. 3b). The pooled summary sensitivity reached 75% (95% CI 72–77%, I² = 93.3%) (Fig. 3a) and pooled summary specificity was computed to 68% (95% CI: 66%–70%, I²= 97.4%) (Fig. 3b), indicating significant heterogeneity. The pooled positive LR was 2.8 (95% CI: 2.1–3.77) and the pooled negative LR was 0.32 (95% CI: 0.23–0.46) (Fig. 4a,b).

Figure 3.

Forest plot of estimates results: (a) Sensitivity; (b) Specificity.

Figure 4.

(a) Positive likelihood ratio; (b) Negative likelihood ratio.

Subgroup analysis

Subgroup analysis was also performed to explore the influence of different reference standards in the final result (Supplementary Fig. 1a–d). Two subgroups were formed based on two reference standards: conventional testing and clinical diagnosis. The results confirmed consistent performance.

Heterogeneity analysis

Four components, including “gold standard”, “experimental design”, “age” and “country income” were considered in the meta-regression analysis to explore potential risk of bias. Unfortunately, none of these components exhibited heterogeneity. Due to failure to extract more comprehensive data from the research, it was not further analyzed.

Evaluation of diagnostic accuracy

SROC curves for the mNGS in infectious diseases are presented in Fig. 5a. This figure illustrates the relationship between sensitivities and 1-specificity for the included studies in the pooled analyses. The AUC was considered excellent (AUC = 0.85 (SE = 0.03)). The point at which sensitivity and specificity were equal (Q*) was 0.78 (SE = 0.03). The pooled DOR was 11.94 (95% CI: 6.11–23.34) (Fig. 5b).

Figure 5.

Summary ROC curves.

Publication bias

Deek’s test yielded no evidence of publication bias (P = 0.795). (Supplementary Fig. 2).

Discussion

To our knowledge, the present meta-analysis was the first to systematically review the use of mNGS in diagnosing infectious diseases. Conventional techniques for the detection of pathogens are largely target-dependent tests, which detect a limited number of micro-organisms. However, NGS-based metagenome approaches are target independent and can detect unknown pathogens²². Using the pooled estimate of 75% (95% CI: 72–77%, I²= 93.3%)) at median specificity 68% (95% CI: 66–70%, I² = 97.4). The AUC 85%, which reflected infection using mNGS, was classified as excellent performance.

The DOR reflects the relationship between the diagnostic test and the relevant disease. The pooled DOR was 11.94, reflecting diagnostic efficacy of mNGS in infectious diseases. The pooled positive LR was 2.81 (95% CI: 2.1–3.77), which reflects that the risk of developing the disease was 2.81 times that of not having the disease when the results of next generation sequencing being positive. The pooled negative LR was 0.32 (95% CI: 0.23–0.46), which reflects that the risk of developing the disease was 0.32 times that of not having the disease when the results of NGS are negative. The sROC curve reflects merge indicators of the sensitivity and specificity. The AUC for sROC was 0.85, which reflected high diagnostic efficiency.

Some studies^23–26 demonstrated that mNGS had diagnostic advantages over conventional methods for patients treated with empirical antibiotics before sample collection. The use of empirical antibiotics would significantly lower the detection rate of conventional methods by approximately 20%, while mNGS is not affected²³. The reason may likely be due to the fact that culture methods require the existence of live pathogens and, therefore, are easily influenced by the administration of antimicrobials. On the other hand, high-throughput sequencing needs only to identify DNA fragments of microorganisms, which may explain its relatively higher detection rate after antimicrobial treatment. Moreover, it can shorten turnaround time and detect pathogens without bias²⁷.

NGS also has shortcomings. First, it is not sensitive for intracellular bacteria and fungi in difficulty obtaining circulatory genome DNA^23,28. RNA viruses require reverse transcription before deep sequencing and the amount of DNA segments may be reduced²³. Different NGS technique may introduce bias (Supplementary Table 1). Second, mNGS is relatively expensive. Third, the criteria for diagnosing single pathogens are unclear, and are mainly based on the relative abundance of pathogens, the coverage rate or unique reads of pathogens^8,29. In addition, given the untargeted nature of mNGS, background interference is a fairly common limitation.

Our study also had limitations. The first of which was considerable heterogeneity, the sources of which were extensively explored. Meta-regression results revealed that “experimental design” and “age” may have been the cause of heterogeneity. Another factor that needs to be considered is the clinical heterogeneity exhibited in the included studies such as the number of patients, antibiotic treatment, sampling methods, different reference standards and other unknown factors such as technical variations (e.g. sequencing strategies and platforms), sequence profiling software, prediction models, and batch effects. Second, the number of patients in two studies^30,31 was relatively small, which may have reduced our statistical power. Third, no fourfold contingency tables were feasible for most of the studies because some of the necessary data were calculated based on reported sensitivity and specificity. Fourth, limiting the search strategy to English language publications could have potentially missed some studies. Finally, the included studies may have potentially been affected by selection bias and the use of different reference standards for infectious diseases.

Conclusions

mNGS combined with conventional microbiological testing can improve diagnostic efficiency. We believe that mNGS may be a potential step forward in diagnosing infectious diseases due to its non-invasive, rapid and untargeted characteristics.

Abbreviations

AUC

Area under curve

mNGS

Metagenomic next-generation sequencing

PRISMA-DTA

Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy

NPV

Negative predictive value

PPV

Positive predictive value

CIs

Confidence intervals

sROC

A summary receiver operating characteristic curve

AUC

Area under the curve

SS

Summarise sensitivities

SP

Summarise specificities

− LR

Negative likelihood ratios

+ LR

Positive likelihood ratios

DOR

Diagnostic odds ratio

HSROC

Hierarchical summary receiver operating characteristic

PCR

Polymerase chain reaction

ID

Infectious diseases

Author contributions

Conceived and designed the experiments: J.L. Y.Q.Q. Performed the experiments: J.L Y.Q.D J.Y. Analyzed the data: J.L J.Y. Contributed reagents/materials/analysis tools: J.L Y.Q.D. Wrote the manuscript: J.L. Revised the manuscript: Q.Z. All authors have read and approved the manuscript.

Funding

This study was supported by the Zhejiang Province Public Welfare Technology Application Research Project (CN), China (2019 LGF19H010009), National Natural Science Foundation of China (81971982) and National Natural Science Foundation of China (No. 81903849). The funders had no role in study design, data collection or analysis, decision to publish.

Data availability

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Competing interests

The authors declare no competing interests.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-022-25314-y.