Clinical Metagenomics for Infectious Diseases: Progress toward Operational Value

ABSTRACT

The field of clinical metagenomics for infectious disease diagnostics has advanced to combining questions of technical methodologies with best-use practices due to lowering barriers of implementation. This commentary identifies challenges facing further development of the field and proposes methods for advancement by highlighting a recent prospective pilot study evaluating a targeted metagenomic approach for infectious endocarditis. This commentary introduces the concept of operational value as a method for standardizing results generated by differing clinical metagenomic approaches. Operational value includes assessments of result quality, utility, and cost through incorporating methodological aspects of metagenomics as applied to various infectious syndromes, patient populations, and specimen types. Focus is placed on standardizing outcome-based metrics using an operational value matrix. As ambitions of clinical metagenomics are increasingly realized, new models of study design and collaboration could promote progress toward routine use and positive benefits for patients with infectious diseases.

TEXT

The ambition of clinical metagenomics for infectious disease diagnostics is well summarized by, surprisingly enough, the English author, linguist, and academic J. R. R. Tolkien. In a lecture he gave on “The Story of Kullervo,” a fledgling tale adapted from Finnish poetry that is the literary genesis later developing into The Lord of the Rings and his larger legendarium, Tolkien describes the method of approaching malevolent beings in the original Finnish stories, “…you have only got to know the accurate detailed history of the origin birth and ancestry of anyone…to have the power to stop the evil and cure the damage he has done or otherwise deal with him” (1). This is the charge of clinical microbiology. The results generated by clinical microbiology tests must be actionable and lead to improved patient outcomes for patients with infectious diseases. We help our clinical colleagues “stop the evil and cure the damage…or otherwise deal with [it]” by identifying and characterizing pathogens. Clinical metagenomics represents a powerful method of knowing “the accurate detailed history of the origin birth and ancestry,” as it were, of those pathogens. This commentary uses a recent study from Flurin and colleagues published in the Journal of Clinical Microbiology (2) to highlight key principles of clinical metagenomics as applied to infectious disease diagnostics, comment on areas in need of development, and propose pathways forward for the field as a whole through use of standardized value assessments.

OPERATIONAL VALUE

A value-focused approach to medicine is highly important in the current health care structure of the United States (3). Laboratory medicine is well situated to focus on value due to quality initiatives foundational to the discipline (4). Accordingly, routine use of clinical metagenomics for infectious disease diagnostics requires high-value approaches generating actionable results. Ideally, these results would be reliable and trustworthy, clinically useful, and provided at the lowest possible cost to the patient. These principles are summarized within the concept of operational value. The operational component takes into consideration the methodologies of the clinical metagenomic approach used to analyze a particular specimen obtained at a given time point during an infectious course, as well as the specific infectious syndrome and patient population. Improving these operational considerations leads to the generation of valuable results, here defined as the highest-quality results that are the most useful for patient care at the lowest possible cost [or, as a representative equation, value = (quality + utility)/cost]. Although the concept of operational value is applicable to all tests offered by clinical microbiology laboratories, it takes on additional significance for clinical metagenomics. These approaches utilize complex technologies that are currently quite expensive and can yield tremendous amounts of data that require careful interpretation to generate actionable results. Presently, the field is primarily focused on investigating how to develop methods and apply them to clinical practice. Comparatively less effort has been directed toward investigating best-use practices to improve operational value. Beyond broad methodological considerations, the most beneficial clinical use of these approaches remains an open question. Unlike the writings of Tolkien, this field remains in relative infancy. It has not achieved its ambitions and has not yet developed into a full legendarium. It can, though, through collaborative groups focusing on methodological questions, best-use practices, and methods of standardization.

Multiple well-written reviews are available that detail the technical methods of clinical metagenomics for infectious disease diagnostics (5–9). It is important to note that although clinical metagenomics utilizes next-generation sequencing (NGS) technologies, the intended use of these technologies for clinical metagenomics differs fundamentally from other uses (such as whole-genome sequencing). Clinical metagenomics is a diagnostic approach that attempts to provide actionable results for patient care. Accordingly, a pressing technical question is the optimal clinical metagenomic approach to use in a given clinical situation. One approach is not overall superior to another, and the methodological differences between approaches have varying benefits and limitations that may be differentially adapted to increase operational value. Broadly, the steps of clinical metagenomics include sample acquisition, sample processing, nucleic acid extraction, library preparation (with potentially other steps preceding or following library preparation, such as host depletion, targeted or global amplification, probe-based target capture for enrichment, etc.), sequencing, bioinformatic analysis, interpretation, and result communication. Each step is a variable that can be modified, and the methods by which each step is performed can have meaningful impacts on the final results. As such, it is important to note that the term clinical metagenomics, or even NGS more broadly, encompasses various technical methods that can differ widely from one approach to another. It may be more accurate to think of clinical metagenomics as a philosophic approach to obtaining a diagnosis rather than a specific diagnostic test. It is certainly accurate to consider the term “clinical metagenomics” as pleural, not singular, as it comprises many varying techniques. Care must be taken to understand the various steps used in a specific clinical metagenomic approach, including the critically important step of data interpretation, as variations in these steps can positively or negatively impact the quality, utility, and cost of the final results.

Beyond the technical aspects of clinical metagenomics, the best application of these approaches is the primary consideration for improving operational value. It is unlikely that a single clinical metagenomic approach will be the best to use for all diagnostic dilemmas, and there are many infectious syndromes for which clinical metagenomics would questionably yield improvement over current clinical microbiology approaches. Accordingly, determining the best uses of clinical metagenomics is critical for this field to progress. This line of investigation falls into the realm of implementation research. When applied to clinical metagenomics for infectious diseases, implementation research presents particular challenges, given the methodological differences between varying approaches. However, this can be addressed through methods of standardization. One method is to standardize based on the approach used. As an example, multiple studies from differing institutions have recently been published investigating the use of a single clinical metagenomic approach offered by a reference laboratory. These studies demonstrate that unregulated ordering of this test is likely not the most valuable approach (10, 11). Additionally, the clinical performance of the test can be improved by focusing its use on specific patient populations with specific infectious syndromes suspected to be secondary to specific pathogen types (12). By extension, considering clinical metagenomics as a “hypothesis-free” approach is not a best-use practice to improve operational value. As with any other test in clinical microbiology, using it to support or refute a particular clinical hypothesis is a better use. Optimally, studies demonstrating the performance of a particular clinical metagenomic approach for use addressing a given clinical hypothesis would be available to help interpret results and thus better understand if the process actually helped answer the clinical question. Such a body of literature is building, and ongoing investigations are needed to further develop the technical methods and applications of clinical metagenomics.

CASE STUDY

The study from Flurin and colleagues (2) serves as a helpful example of the points detailed above. This study evaluated the performance of a targeted clinical metagenomic assay at a single institution. The authors restricted the study to an immunocompetent patient population with the clinical syndrome of infectious endocarditis, comparing detection of etiologic pathogens using this metagenomic approach to conventional approaches (blood culture, valve culture, valve sequencing, and serology). They compared two specimen sources, whole blood and plasma, and evaluated the timing of the metagenomic approach in relation to the last positive blood culture, as well as the time from the first antibiotic dose. Although no clinical outcomes were assessed in this study (the metagenomic results were not relayed to providers for patient care purposes), it is notable that this study was conducted in a prospective manner. A prospective study design allows assessment of variables that are important to considering operational value. The design of this study allows assessment of result quality through comparison of a metagenomic approach to standard methods. It suggests the utility of metagenomic results through extrapolation of how the results might have been utilized were they communicated to clinical teams. This study does not assess cost, but prospective designs can incorporate cost measures as an outcome.

Focusing on infective endocarditis allowed the authors to use a widely accepted diagnostic rubric (the modified Duke criteria) that is primarily clinical and does not entirely depend upon the identification of a causative pathogen (13, 14). However, identifying a pathogen confirms the diagnosis, and the design of this study allowed comparison of standard methods of pathogen identification with this metagenomic approach. Although the microbiology results factoring into a diagnosis of infective endocarditis are standardized in the modified Duke criteria, these approaches can be unrevealing and conclude with a diagnosis of culture-negative endocarditis. Variables like prior antibiotic therapy or the inability to detect fastidious pathogens contribute to negative results, and lacking the definitive identity of a causative pathogen can lead to suboptimal antimicrobial therapy with negative patient impacts. The use of clinical metagenomics in the syndrome of infective endocarditis could conceivably benefit diagnosis of culture-negative cases by providing evidence of a pathogen that is difficult to detect by current methods, as well as potentially inform diagnosis when culture is positive only from invasively obtained specimens (such as cardiac valves) later in a clinical course. It is less clear how clinical metagenomics could benefit settings where pathogens can be identified and/or phenotypically characterized by standard microbiology techniques. However, as best-use practices to improve operational value are not currently known, this remains an open question that is partially addressed in this study.

The specimens compared in this study, whole blood and plasma, deserve additional comment. The specimen source yielding the highest operational value for clinical metagenomic testing in infective endocarditis is unknown. The conventional and most readily accessible specimen types for diagnosis of infective endocarditis are whole blood (for culture) and plasma (for serology). These specimen sources have varying diagnostic benefits and challenges for clinical metagenomics. Whole blood is partially composed of plasma, and as such, whole blood conceivably contains pathogen nucleic acids present in all blood components. However, whole blood also contains a higher abundance of human DNA than does plasma. In clinical metagenomic assays, human DNA competes with pathogen nucleic acids in a stoichiometric relationship such that abundant human reads can dilute pathogen reads to an undetectable level (15). Accordingly, methods of host depletion can change the stoichiometric ratio to potentially improve detection of pathogen nucleic acids, and separation of plasma from whole blood is one such technique. Removal of the cell component from plasma, though, removes pathogen nucleic acids that could be present within human cells (due to phagocytosis of pathogens, intracellular pathogens, or other processes). Thus, the optimal specimen source for diagnosing infective endocarditis using clinical metagenomics is neither intuitive nor known, and studies such as this contribute helpful insights to the field at large. If plasma was not the superior specimen source in this study investigating a syndrome in which blood products are continually in direct contact with the focus of infection, perhaps plasma is not the best specimen source for clinical metagenomic testing of other infectious syndromes that are further removed from the blood.

As for the metagenomic strategy following nucleic acid extraction from whole blood and plasma specimens, the authors used a targeted approach by amplifying and sequencing portions of the 16S rRNA gene. This approach is useful given the unique ability of 16S rRNA to provide species-level identification. The primers used in this study restricted amplification of the 16S rRNA gene to the V1-V3 region, which can be adequate for genus and species-level identification (16) but does not allow further characterization, such as genotypic predictors of antimicrobial susceptibility. The bioinformatic approach utilized a fee-for-service platform referencing a curated and proprietary database. Importantly, the results generated by this approach were evaluated by experts in the field of metagenomics, including a clinical microbiology laboratory medical director, who were blinded to the results from conventional testing. Additionally, positive metagenomic results were considered in the clinical context to judge if they were clinically significant or clinically insignificant. This step of interpretation before metagenomic results were finalized contributes markedly to the strength of this study, given the methodological and interpretive complexities of this approach.

The most notable results relate to the comparative performance of the metagenomic and conventional approaches. Concordance and discordance between standard microbiology techniques and this metagenomic approach were evaluated based on the presence or absence of pathogen growth from conventional blood cultures. Higher concordance was found when specimens for targeted metagenomic testing were obtained from patients with positive blood cultures. A temporal association was noted, such that concordance was highest when obtaining specimens closer in time to a recent positive blood culture. No statistically significant impact was observed from obtaining specimens nearer or further in time from the first antibiotic dose, though the result trend was toward a longer time period between the first antibiotic dose and negative results. Although not statistically significant, whole blood tended to yield more concordant results than did plasma (whole blood, 61% concordant [17/28 specimens], versus plasma, 45% concordant [13/29 specimens]). Overall, no significant difference was found between metagenomic results obtained from whole blood or plasma specimens. Of note, discordance was found between whole blood and plasma in 8 specimens with positive cultures. The metagenomic approach did not detect Candida albicans from a blood culture, though this is expected, as fungi do not encode 16S rRNA. Additional bacterial results that were initially missed by the metagenomic approach were detected when repeating the procedure with differing extraction techniques and sequencing reagents. Importantly, the metagenomic approach detected pathogens otherwise missed by conventional approaches in 6 cases of culture-negative endocarditis. Clinical review suggested some that were detected could be etiologic pathogens driving the clinical presentation, and others were confirmed as such by orthogonal testing methods.

Although this approach added detection of some pathogens, the overall concordance between this metagenomic assay and conventional testing in patients with culture-positive and culture-negative endocarditis was 66%. This level of performance is somewhat below that needed for independent clinical use, though the acceptable level of performance is at the discretion of the clinical microbiology laboratory offering such testing. As this was a prospective pilot study, determining the clinical utility of this test is beyond the scope, as is establishing analytic or clinical performance criteria. It does, however, answer the question it set out to pursue, which is whether such a test could potentially be useful for diagnosing infective endocarditis. That answer is definitively, “yes.” Given the concordance, perhaps a near-future utility is as an adjunctive test included along with standard methods rather than an independent method replacing them. Alternatively, specimens obtained early in the course of infection could be stored for possible later testing if conventional microbiology approaches are unrevealing. Such questions of implementation would require additional studies. In addressing this question of potential use, the study provides helpful insight into the operational value of this specific metagenomic assay for infective endocarditis diagnosis and highlights many important considerations for further development as a broader field.

COMMENTARY

The development of clinical metagenomics for infectious diseases has been ongoing for over a decade. Though fundamental questions remain about best practices for methodological performance, the field is moving into questions of implementation and best use. Clinical metagenomics has not had a wide impact, due in part to the layered complexities of methods comprising this approach. Among other barriers to implementation, the historically high cost of NGS reagents and instruments (automated instruments for sample processing and library preparation, as well as sequencing instruments), the relatively steep learning curve needed to achieve technical expertise with nonautomated methods, the overall lack of readily available and easily used bioinformatic platforms for those without formal training in computer science, and the unreliability and incompleteness of publicly available reference databases has impeded the development of this field into a broadly applicable and widely available diagnostic approach.

However, the barriers to implementation are lowering. The overall cost of sequencing decreased precipitously over the past decade (17). NGS technologies with entirely different sequencing chemistries than those used when this field began to develop are now available, providing alternative and complementary approaches with various benefits. Expertise in these methods is more widely available, and process automation is developing to increase consistency and efficiency. End-user bioinformatics are also becoming more accessible via shared online resources, as well as commercial vendors. Improved public curated databases are available, and methods are being developed to further improve them (18, 19). Proprietary curated databases are proliferating as well. Although clinical metagenomic testing presently remains isolated in a small number of academic medical centers and commercial laboratories, lower barriers to implementation will presumably lead to more medical centers developing and/or utilizing these approaches, as well as more commercial laboratories offering clinical metagenomic testing in reference capacities. The field is moving from a place where the question of “How do we do this?” (focusing on technical aspects of the methods) is joined with the question of “How do we use this?” Questions of best-use practices improving operational value are of pressing importance to advance the field.

How can such questions be addressed? One method to evaluate operational value for clinical metagenomics is to utilize a value matrix. Taking the value components of quality, utility, and cost, a three-dimensional matrix can be constructed with quality (trustworthiness and reliability of results) on one axis, utility (actionability and impact of results) on another, and cost (financial and time requirements to generate the results) on the third (Fig. 1A). This matrix forms multiple value combinations. The combination with the highest value is an approach that generates high quality and highly useful results at low cost. The combination with the lowest value is an approach that generates low quality data with limited utility at a high cost. Other tests fall between these on a value spectrum. Alternatively, a scoring system using standardized numerical metrics with the equation value = (quality + utility)/cost could be developed, with different value rankings compared on a value continuum (Fig. 1B). Standardizing variables that place individual approaches to clinical metagenomic testing into the most desirable locations on an operational value matrix or continuum could guide best-use practices for infectious disease diagnostics.

FIG 1.

(A) Example of an operational value matrix to standardize assessment of clinical metagenomic approaches. Quality, utility, and cost are plotted on the axes and divided into high, moderate, or low values. Hypothetical approaches are plotted in circles to represent the most optimal value (approach 1), intermediate value (approaches 2 and 3), and the poorest value (approach 4). (B) Example of an operational value continuum using the value definition obtained with the equation value = (quality + utility)/cost. This approach requires standardized numerical values applied to each metric to generate a value assessment. As in the operational value matrix example, four studies with differing values are plotted along the continuum. Accepted definitions for these metrics (or methods of quantification) are needed to achieve standardization with either approach.

Such variables include all components of operational value (patient population, syndrome, specimen, timing, and testing approach, and quality, utility, and cost of results). However, there are multiple stakeholders with potentially competing opinions on the desired operational value for a given approach. These stakeholders include test developers, clinical laboratories, medical providers, hospital systems, insurance payers, and governmental agencies, but the stakeholders most impacted by these approaches are the patients afflicted with infectious diseases. Clinical metagenomics must be beneficial to patients for the field to advance, and the primary method of determining operational value is with studies investigating outcomes that follow clinical use of these approaches.

A major barrier to outcome studies is a lack of standardization. This can be thought of as disunified methods, though this may not be the best approach for the field to progress. Adhering to a unified approach would standardize testing through distribution of a single methodological approach to multiple laboratories or centralization of testing in a single laboratory. However, this approach to standardization does not allow for innovation at the multiple complex steps composing clinical metagenomics. Furthermore, it is unlikely that one approach will be adequate for all testing needs given the various patient populations, clinical syndromes, and specimen sources comprising diagnostic testing for infectious diseases.

Rather than methodological standardization, a way forward to increase operational value for clinical metagenomics is to standardize value-focused outcomes. Such outcomes would include clinical performance (clinical sensitivity, clinical specificity, positive predictive value, negative predictive value, etc.) and traditional measurements for infectious disease diagnostics (mortality, inpatient and intensive care unit [ICU] admission time, time to definitive antimicrobial therapy, duration of total antimicrobial therapy, etc.). Perhaps more importantly, outcomes would also include standardized metrics of operational value. Standardized assessments determining where a given approach resides on the operational value matrix allow comparison of approaches with differing methods and place more importance on the value of the results rather than the method used to generate them. Accordingly, differing methodological approaches could be compared in multi-institutional studies as long as clinical variables (patient populations, clinical syndromes, specimen sources, and timing of testing) are equivalent.

Achieving such a goal would likely require more formalized and ongoing collaborations, perhaps with members and a governing body forming a consortium. Establishing a consortium for clinical metagenomics in infectious diseases could involve multiple stakeholders with a decentralized governance structure. The consortium could standardize value metrics to compare metagenomic approaches, establish consistent outcome measurements focused on operational value, and pursue high-impact questions with rigorous study designs. Ideally, affiliated laboratories developing metagenomic approaches for clinical diagnostics could enroll in ongoing consortium-led studies to collaboratively contribute sequencing data and deidentified patient metadata into a central repository available to members. This approach could generate adequately powered studies spanning multiple institutions to address questions less amenable to publicly funded research, as well as provide avenues for replication and confirmation of impactful findings. Furthermore, the consortium could partner with agencies developing calibration standards to ensure the differing methods reach the same key results and use these calibration standards to assess the analytic performance of differing approaches. These calibration standards would need to assess the performance of all steps in a clinical metagenomic workflow, including contrived specimens to assess the laboratory steps and contrived sequencing data files to assess the bioinformatic processes. The ultimate goal of this consortium would be to develop clinical metagenomic approaches with the highest operational value for patients with infectious diseases.

CONCLUSION

Clinical outcomes as the primary determinate of operational value could serve as a standardization method to advance the field of clinical metagenomics for infectious diseases. Although developing and implementing complex metagenomic approaches using the model of an operational value matrix or continuum may seem daunting, it is possible to solve the implementation complexities of clinical metagenomics. Progress toward standardization is being made, as exemplified by guidelines for validating clinical metagenomic approaches (20), the development of strategic goals focused on genomic sequencing by the College of American Pathologists (21), and the development of the Centers for Disease Control and Prevention’s Next Generation Sequencing Quality Initiative (22). These efforts will advance the field of clinical metagenomics for infectious diseases, as will studies such as that conducted by Flurin and colleagues (2). A unified consortium for clinical metagenomics in infectious diseases would coordinate these efforts around patient-centered operational value. As described by J.R.R. Tolkien, such endeavors could go far in using clinical metagenomics to “…know the accurate detailed history of the origin birth and ancestry of anyone…to have the power to stop the evil and cure the damage he has done or otherwise deal with him.”

The views expressed in this article do not necessarily reflect the views of the journal or of ASM.