Plasma Metagenomic Next-Generation Sequencing Assay for Identifying Pathogens: a Retrospective Review of Test Utilization in a Large Children's Hospital

Plasma metagenomic next-generation sequencing (mNGS) is a new diagnostic method used to potentially identify multiple pathogens with a single DNA-based diagnostic test. The test is expensive, and little is understood about where it fits into the diagnostic schema. We describe our experience at Texas Children’s Hospital with the mNGS assay by Karius from Redwood City, CA, to determine whether mNGS offers additional diagnostic value when performed within 1 week before or after conventional testing (CT) (i.

ABSTRACT

Plasma metagenomic next-generation sequencing (mNGS) is a new diagnostic method used to potentially identify multiple pathogens with a single DNA-based diagnostic test. The test is expensive, and little is understood about where it fits into the diagnostic schema. We describe our experience at Texas Children’s Hospital with the mNGS assay by Karius from Redwood City, CA, to determine whether mNGS offers additional diagnostic value when performed within 1 week before or after conventional testing (CT) (i.e., concurrently). We performed a retrospective review of all patients who had mNGS testing from April to June of 2019. Results for mNGS testing, collection time, time of result entry into the electronic medical record, and turnaround time were compared to those for CT performed concurrently. Discordant results were further reviewed for changes in antimicrobials due to the additional organism(s) identified by mNGS. Sixty patients had mNGS testing; the majority were immunosuppressed (62%). There was 61% positive agreement and 58% negative agreement between mNGS and CT. The mean time of result entry into the electronic medical record for CT was 3.5 days earlier than the mean result time for mNGS. When an additional organism(s) was identified by mNGS, antimicrobials were changed 26% of the time. On average, CT provided the same result as mNGS, but sooner than mNGS. When additional organisms were identified by mNGS, there was no change in management in the majority of cases. Overall, mNGS added little diagnostic value when ordered concurrently with CT.

INTRODUCTION

Accurate and timely diagnosis of infectious diseases can significantly impact patient outcomes. For example, rapid and sensitive diagnosis of bloodstream infection is important in reducing morbidity and mortality. Every hour of delay in the start of effective antibiotic therapy can lead to a 7.6% increase in patient mortality (1). Conventional culture methods can take 48 h or longer to identify bacterial infections (2). Thus, most initial antimicrobial therapy needs to be empirical and broad spectrum, with changes to more-targeted drugs occurring with organism identification. Many diagnostic methods that utilize nucleic acid detection technology reduce the time to identification to a few hours but are limited to the detection of a single pathogen or a limited panel of pathogens. Plasma next-generation sequencing (mNGS) technology overcomes the limitation of specific pathogen detection by applying next-generation sequencing technology to cell-free DNA from patient plasma. mNGS has the potential to identify more than a thousand pathogens, independently of culture, with claims of rapid turnaround times (TATs) (3).

Karius (Redwood City, CA) supplies the commercially available mNGS test for infectious disease diagnosis (4). Case series have shown that mNGS can identify the etiology of infection in patients with complicated community-acquired pneumonia or in immunocompromised patients with invasive fungal disease when traditional or other advanced diagnostic methods are negative (5, 6). In a prospective cohort study of pediatric patients with relapsed or refractory cancer, mNGS predicted bloodstream infection 3 days prior to blood culture in 12 of 16 cases (7). mNGS tests bear the promise of being noninvasive, with the ability to detect both uncommon and common pathogens in many different clinical scenarios. The sensitivity of mNGS varies, and while the perception may be that mNGS is more sensitive than conventional culture and molecular assays, this has not been proven in published validation studies (3, 8). In a multicenter review of mNGS testing of cerebrospinal fluid (CSF), mNGS failed to identify pathogens in 26 of 204 cases where pathogens were successfully identified by conventional testing (9).

In addition, the low specificity of mNGS assays may pose diagnostic challenges, since dealing with false-positive results can complicate the clinical picture (10). There are case reports of mNGS detecting rare pathogens (11–13) when conventional tests were negative. However, for the vast majority of cases, organisms identified by mNGS were common pathogens that were also detected by conventional testing methods (7, 9).

It is unclear if mNGS offers significant additional diagnostic value, improves sensitivity, or reduces the time to detection when it is performed 1 week before or 1 week after (i.e., concurrently with) conventional testing. Understanding where in the diagnostic workup these costly tests add value is important for diagnostic stewardship initiatives. The goal of this study was to assess the diagnostic value of mNGS compared to that of conventional microbiological methods in a pediatric patient population.

MATERIALS AND METHODS

Study population and ethical considerations.

This study, approved by the Baylor College of Medicine Institutional Review Board (IRB no. H-45342), was a retrospective review of all patients with mNGS testing by Karius as part of clinical care from 1 April 2019 through 30 June 2019 at Texas Children’s Health System (TCH). No treatment decisions were made or altered as a result of this study. TCH is an 827-bed quaternary referral center in Houston, TX. All patients reviewed were ≤18 years of age and had mNGS testing in either the inpatient or the outpatient setting.

Patient data.

Patient demographics, underlying medical conditions, and immune statuses were recorded. Details about mNGS testing, including results, specimen collection time, time of result entry into the electronic medical record (EMR) (referred to below as result time), and turnaround time (time from collection to result time) (TAT) were evaluated. The same data for all conventional testing (culture, serology, organism-specific PCR, multiplex PCR, broad-range 16S/28S PCR/sequencing, and histopathology) ordered 1 week before and 1 week after mNGS testing were also obtained. There are no institutional guidelines or limitations for mNGS ordering at TCH.

Definitions and statistical analyses.

The results of mNGS were compared to the results of conventional tests to determine positive and negative agreement. Positive agreement was defined as the identification of at least one organism by both mNGS and conventional testing within the 2-week time frame. Negative agreement was defined as a negative result by both mNGS and all conventional testing within the 2-week time frame. Results were considered discordant when additional organisms were identified by mNGS or when there was a mismatch between the organisms identified by mNGS and conventional testing. Electronic records of patients with discordant results were reviewed to determine whether antimicrobials were added or changed based on the discordant mNGS result. Descriptive statistical analyses of result time and TAT were performed. The statistical significance of differences in TAT between mNGS and conventional testing was analyzed using the Mann-Whitney test, assuming that observations were not normally distributed. All statistical analyses were performed using GraphPad Prism software, version 5.0.

Data availability.

The data used to compare the test performance of mNGS to that of conventional testing, and the clinical impact of unique organisms identified by mNGS, are included in Data Sets S1 and S2 in the supplemental material.

RESULTS

Patient characteristics.

Sixty patients, with an average age of 8.9 years, had mNGS testing performed during the 3-month study period (Table 1). The majority of patients were either receiving immunosuppressive therapy (62%) or had a primary immunodeficiency (6.7%). The most common indication for testing was evaluation of lesions (e.g., lung nodules) seen on diagnostic imaging (40%).

TABLE 1.

Demographics and patient characteristics

Characteristic	Valuea
Avg age (yr)	8.9
Male sex (%)	48.3
Race
White	15 (25)
Black	11 (18.3)
Hispanic	26 (43.3)
Asian	7 (11.7)
Pacific Islander	1 (1.7)
Underlying medical conditions
Stem cell transplant	13 (21.7)
Solid-organ transplant	8 (13.3)
Hematological conditions
Leukemia	9 (15.0)
Aplastic anemia	4 (6.7)
Sickle cell	2 (3.3)
Solid tumor	3 (5.0)
Congenital heart disease	4 (6.7)
Primary immunodeficiency	4 (6.7)
Autoimmune disease	3 (5.0)
Genetic abnormality	2 (3.3)
No underlying medical condition	8 (13.3)
Indication for mNGSb
Lesion identified on imaging
Lung	16 (27)
Liver	3 (5)
Spleen	2 (3.3)
Brain	2 (3.3)
Fever and neutropenia	6 (10)
Culture-negative sepsis	6 (10)
Cutaneous lesions	5 (8.3)
Meningitis/encephalitis	3 (5)
Culture-negative endocarditis	2 (3.3)
Unclearc	12 (20)

^aExcept where otherwise indicated, values are the number (percentage) of patients with the characteristic.

^bmNGS, plasma next-generation sequencing. Some patients had multiple indications for mNGS.

^cNo clear indication identified from chart review.

Testing results and agreement.

mNGS testing identified a single organism for 22 patients (14 bacteria, 7 viruses, 1 yeast), two or more organisms for 16 patients, and zero organisms for 22 patients. There was 61% positive agreement between mNGS and conventional testing (i.e., there was at least one organism identified by mNGS that matched at least one organism identified by conventional testing) (Table 2; see also Table S1 in the supplemental material) and 58% negative agreement between mNGS and conventional testing (i.e., no organisms identified by either mNGS or conventional testing).

TABLE 2.

Comparison of results by conventional testing and mNGS^a

Result by mNGS	Result by conventional testing
	Positive	Negative
Positive	22	10
Negative	14	14
Total	36	24

^aPositive agreement was 22/36 (61%), and negative agreement was 14/24 (58%).

Organism identification.

Overall, conventional testing identified more organisms per patient (mean, 2.4) than mNGS (mean, 2.0) (Fig. 1). In 28 patients, conventional testing identified one or more organisms not detected by mNGS. In 9 cases, conventional testing identified an organism(s) when mNGS was negative, and in 19 cases, conventional testing identified additional organisms not identified by mNGS (Table S2). In 4 of these 28 cases, conventional testing identified an RNA virus as the only additional organism; mNGS does not detect RNA viruses. In the remaining 24 cases, organisms identified by conventional testing and not by mNGS were organisms that Karius claims in its product literature to detect. For example, mNGS did not identify Mycobacterium tuberculosis (from a 6-month-old child with culture-positive CSF and PCR-confirmed Mycobacterium tuberculosis meningitis. mNGS was negative for a 3-year-old child with herpes simplex virus 1 (HSV-1) detected by both blood PCR and serology. Likewise, mNGS identified Haemophilus influenzae but failed to identify Staphylococcus aureus and Moraxella catarrhalis, which were identified by blood culture, for a 2-year-old child.

FIG 1.

Number of organisms per patient identified by conventional testing and mNGS. The total numbers of organisms identified per patient by each test were compared. Each bar represents a total number of organisms identified by mNGS (red) or conventional testing (blue). Results are categorized based on test agreement subsets. Overall, conventional testing detected more organisms per patient (2.4) than mNGS testing (2.0).

In 23 patients, mNGS identified at least one organism not identified by conventional testing, ranging from one to nine additional organisms. In 16 cases, mNGS identified two or more organisms (polymicrobial result). On review of the medical record, antimicrobials were altered as a result of mNGS for 6 of the 23 patients (26%) (Table S2). Antimicrobials were altered in only 2 of the 16 cases with polymicrobial results (13%) (Table S2). Examples of cases in which mNGS resulted in an antimicrobial change include an 11-year-old stem cell transplant patient for whom antipseudomonal coverage was added when Pseudomonas aeruginosa was identified by mNGS and a 9-year-old patient for whom anaerobic and macrolide coverage was added after Prevotella melaninogenica and Mycoplasma pneumoniae were identified by mNGS. In a 16-year-old with acute lymphocytic leukemia and cutaneous lesions, mNGS identified Candida tropicalis, and antifungal coverage was initiated; however, therapy was later discontinued when a skin biopsy specimen was negative by culture and broad-range PCR.

We compared the types of organisms detected by mNGS and conventional testing. mNGS primarily identified bacteria, DNA viruses, and some yeasts (Fig. 2); organisms in each of those categories were identified by conventional testing also. Conventional testing identified 13 RNA viruses and 2 molds, which were not identified by mNGS testing. The greatest correlation between mNGS and conventional testing was for DNA viruses: a total of 14 DNA viruses were detected by both mNGS and conventional testing among 14 patients (Table S1).

FIG 2.

Types of organisms identified by mNGS and conventional testing. Organisms detected by each test were categorized, and the numbers of organisms of different types found by the two testing methods are compared. Each bar depicts the total number of organisms of a particular type identified by one of the testing methods.

We also compared organism identification by conventional testing in sterile sites to organism identification by mNGS. A total of 36 organisms were identified by conventional testing from sterile sites, and mNGS identified 18 of these 36 organisms (50%) (Table S2). Among six patients with positive blood cultures, mNGS identified the same organism in four patients. The two patients for whom an organism was isolated from blood culture but not identified by mNGS included a 2-year-old with a patent ductus arteriosus who was treated for endocarditis due to Staphylococcus aureus and Moraxella catarrhalis isolated from a peripheral blood culture and a 3-month-old with a history of hypoxic ischemic brain injury for whom Staphylococcus epidermidis was cultured from a central venous catheter. The only positive CSF specimen in this analysis was from a 6-month-old who was later found to have a primary immunodeficiency. The specimen was culture and PCR positive for Mycobacterium tuberculosis, but mNGS did not identify M. tuberculosis; mNGS did identify cytomegalovirus (CMV), which was also identified by CMV PCR from plasma. Overall, conventional testing identified 24 DNA viruses from sterile sites (e.g., plasma, whole blood, and tissue), and mNGS identified only 14 of these DNA viruses. Likewise, conventional testing identified nine bacteria from sterile sites, and mNGS identified four of these bacteria. No fungi or parasites were identified from sterile sites by conventional testing.

Collection time, result time, and turnaround time.

In cases with positive agreement, the collection time, result time, and TAT were compared. For 73% of these patients, results from conventional testing were known before results from mNGS testing. Specimens for conventional testing were collected on average 1.6 (confidence interval [CI], 0.3, 2.9) days earlier than specimens for mNGS, and the mean result time was 3.5 (CI, 1.8, 5.2) days earlier for conventional testing. For 45% of these patients, the conventional test result was known prior to mNGS specimen collection. TAT (collection time to result time) for organism identification was also shorter for conventional testing than for mNGS testing (1.8 versus 4.0 days [P = 0.0001]) (Fig. 3). For 9 of 22 patients with positive agreement between mNGS and conventional testing, mNGS identified at least one organism earlier than conventional testing (Table S1). No change in antimicrobial management was made based on the mNGS result for any of these nine patients.

FIG 3.

Turnaround times of mNGS and conventional testing for tests with positive agreement. TATs for each organism identified in each patient by mNGS and conventional testing were compared. The collection time of the first test ordered for each patient was set at zero in order to calculate the timeline for subsequent tests ordered and the results. The width of each bar represents the TAT. Overall, TAT was 1.8 days for conventional testing versus 4.0 days for mNGS (P = 0.0001).

DISCUSSION

Our experience with a commercially available mNGS assay at a quaternary pediatric health system demonstrated that on average, organisms were identified earlier by conventional methods than by mNGS. The TAT for identification of the same organism was also shorter for conventional testing than for mNGS. The most common types of organisms identified by mNGS were bacteria and DNA viruses, which are easily identified by culture and PCR. When results matched (Table S1), the TAT was shorter for conventional testing in all patients except for two. In patient 18, Fusobacterium necrophorum was identified by broad-range 16S rRNA PCR/sequencing, and in patient 19, Mycoplasma pneumoniae was identified by PCR from a tracheal aspirate; in both cases, the conventional tests were send-out tests. Likewise, the result time was earlier for conventional testing in most cases where there was positive agreement. In the few cases in which mNGS identified an organism earlier than conventional testing, there was no change in antimicrobial management. Although mNGS has the potential to identify multiple pathogens with a single test and a rapid time to result, we found that conventional microbiological methods, such as culture and pathogen-specific PCR, identified the same pathogens sooner. This earlier result time for conventional testing was in part due to earlier order and collection times, as well as the lack of shipping and travel time for specimens worked up in-house. For all but two cases with positive agreement, conventional testing was done in the TCH hospital laboratories.

When mNGS identified additional organisms not identified by conventional testing, no change in therapy occurred in the majority (74%) of cases. This finding is consistent with a recent multicenter review of the clinical impact of mNGS, which found no clinical impact in most patients (85.4%) (14). In our study, when a change in therapy did occur, it was nearly always the result of identification of bacteria by mNGS that were not identified by conventional testing. We did not encounter any unusual pathogens identified by mNGS that resulted in a therapeutic change. When no change in therapy occurred, we postulated that it was the result of identifying an organism considered to be a commensal or an organism that did not fit the clinical picture. For example, in three patients (patients 5, 33, and 34), mNGS identified Torque teno virus, a DNA virus typically not associated with disease in humans but known to cause hepatitis in immunocompromised patients, and no change in therapy was made. We also found that a therapeutic change was less likely with polymicrobial mNGS results, defined as the identification of two or more organisms.

The positive agreement of mNGS with conventional testing in our pediatric study was 61%. This differs from the finding of a study in children where mNGS clinical sensitivity was determined to be 92% (10) but is similar to a report of recent single-center pediatric experience (15). Our results are also similar to the 64% positive agreement found in a multicenter review of mNGS for patients of all ages (14). In our study, mNGS missed three clinically significant organisms (Mycobacterium tuberculosis in patient 20, herpes simplex virus 1 in patient 24, and Staphylococcus aureus in patient 43), all of which were identified by conventional testing. One could argue that amounts of organism-specific DNA can vary in patients depending on when the specimen is collected. However, for our study patients, mNGS was ordered within 1 week of conventional testing, and conventional testing identified the causative pathogens by multiple methods (M. tuberculosis was identified by culture and PCR, and HSV-1 was identified by serology and PCR).

New diagnostic tests can add value by improving overall sensitivity, shortening turnaround time, or identifying pathogens not easily identified by conventional methods. In our study, mNGS did not add value by improving overall sensitivity or shortening turnaround time. In fact, in three instances, mNGS missed clinically significant pathogens that were detected by conventional tests. Furthermore, when mNGS did identify additional organisms, no change in therapy occurred in the majority (74%) of cases. The clinically relevant pathogens identified by mNGS were primarily bacteria and DNA viruses that were also identified by conventional testing, often prior to the reporting of mNGS results. This study suggests that an effective test utilization strategy may be to prioritize conventional testing and reserve ordering the more-costly mNGS test until the conventional tests fail to provide a clinically relevant answer.

There are limitations to this study. First, this is a single-center review of mNGS, and ordering practices and laboratory processes differ between institutions. Laboratory process differences can affect TAT based on the volume of tests, testing frequency, the number of tests performed in-house versus the number sent out to other institutions, and other such variables. Second, we reviewed only those charts in which mNGS identified organisms not identified by conventional testing in order to determine whether there was a therapeutic change based on the mNGS result. It is possible that organisms identified by conventional testing and mNGS were not plausible causes of infection in some cases. Third, we did not evaluate whether prior antimicrobial administration affected the ability of mNGS to detect the same organisms identified by conventional testing. It should be noted that conventional testing, particularly culture methods, is also affected by prior antimicrobial administration.

mNGS testing typically costs severalfold more than conventional testing. As clinicians, lab directors, and hospital administrators become more focused on diagnostic stewardship or utilization management, strategies for facilitating appropriate ordering of costly tests are becoming more relevant. We demonstrated that in the majority of cases, little or no additional value was gained when mNGS was ordered concurrently with conventional testing. This underscores the importance of implementing laboratory stewardship to optimize the diagnostic utility of mNGS for each patient. We propose that plasma metagenomic next-generation sequencing is not a substitute for routine diagnostic testing but may serve a complementary role when utilized in well-defined scenarios. Further investigation into the specific clinical scenarios for which mNGS may offer the greatest clinical value is warranted.