Generalizable Biomarkers in Critical Care: Toward Precision Medicine

REPRODUCIBILITY AND GENERALIZABILITY

The sequencing of the human genome and the subsequent availability of inexpensive, robust methods for “omics” profiling (e.g., genome-wide association studies, gene expression microarrays, and metabolomics) have led to optimism of a new era of biomarkers that would allow for a “precision medicine” approach to critical care. Unfortunately, this promise has yielded few tangible results, as the general biomedical reproducibility crisis (1–3) is particularly troublesome in critical care (4–8) and in omics biomarker studies (9–11). There are two broad problems that lead to seemingly similar studies of biomarkers in critical care producing different results. One problem is traditional nonreproducibility due to false positive biomarker selection or nonrobust statistical models. The other, more importantly, is a lack of generalizability in moving from a narrow study population into broader applications in critical care. We present here a contextual framework for addressing these problems and for assessing new biomarker studies.

SYNDROMIC ILLNESS AND GENERALIZABLE BIOMARKERS

Many critical illnesses are defined syndromically, such as sepsis, acute kidney injury (AKI), acute respiratory distress syndrome (ARDS), and delirium. These syndromes typically have clear, though changing, clinical criteria (12–14). Still, a syndrome may arise from multiple causes; as a result, it is unclear whether all cases of the syndrome really represent the same disease. Such uncertainty raises a major problem in the field. For example, if a positive clinical trial for adults with ARDS defined by Berlin criteria has failed to reproduce in an independent population of children with ARDS also defined by Berlin criteria, was the original finding a false positive or do adults and children have a “different” version of ARDS? Our reliance on syndromic definitions and the lack of clear gold-standard diagnostics linked to pathophysiology thus makes it difficult to assess clinical trial results. In theory, if the entire clinical spectrum of a disease has a common molecular pathophysiology, then a molecular biomarker should exist that is generalizable to the disease. Thus, finding a generalizable biomarker can help to define the disease, improving both patient care and clinical trial design, and potentially moving a whole field of study forward.

There are other practical reasons to search for biomarkers that are generalizable. First, requiring context-specific biomarkers for every variant on a clinical condition (e.g., a different biomarker for different sources of sepsis, or for each different cause of kidney injury) could end up requiring dozens of tests for each critical syndrome. Tests indicated for such increasingly fragmented populations will fail to overcome barriers to market entry. In addition, those that do make it may have overly specific indications for use, leaving many patients without help. Finally, since off-label uses of tests and therapies are common, if biomarkers fail to deliver similar performance in seemingly similar conditions, patients will be harmed.

We thus argue that research should focus first on finding generalizable, disease-defining molecular biomarkers for syndromes in critical illness, or alternatively on showing that such biomarkers do not exist (evidence-of-absence studies). If no generalizable biomarker exists, then more context-specific biomarkers can direct the effort to accurately characterize clinically actionable syndromic subtypes. In other words, we need to clearly define a disease before we begin to divide it into subtypes. Both are necessary components of a precision medicine approach, but due to the high heterogeneity of critical illness, research of both types can be challenging.

HETEROGENEITY IN CRITICAL ILLNESS AND THE CHALLENGE OF CLINICAL TRIALS

Clinical trials in the critical care setting are among the hardest to carry out, for reasons of practicality, patient protection, and patient heterogeneity; this leads to smaller, mostly homogeneous cohorts that do not represent the broad spectrum of critical illness. First, as described above, similar acute syndromes (such as sepsis, AKI, and ARDS) often have multiple possible definitions and span a range of severities. Second, critically ill patients span the entire range of ages, comorbid conditions, and demographics. Third, the medical, surgical, neurologic, and pediatric pathways of critical illness have widely varying primary problems. Fourth, the practicality of conducting a trial leads to differing sampling times and stages of disease at trial enrollment. Finally, the changing treatment patterns over time (such as the change in early sepsis resuscitation with early goal-directed therapy) can lead to different outcomes for the same intervention. The logistical and budgetary constraints of trying to represent all of these sources of heterogeneity means that most single-cohort studies cannot capture the broad spectrum of critical illness, and thus may have difficulty producing generalizable results.

Still, the bedrock of continued progress toward generalizable biomarkers is continued publication of clinical trials. One way to improve trials is to focus on not just size but also heterogeneity. Single-cohort studies are more likely to yield reproducible results when they are appropriately powered, and are more likely to yield generalizable results when they are designed with broad inclusion criteria that attempt to match the full spectrum of the condition under study. Thus, a biomarker that has been tested in 500 adults with pneumonia and ARDS at admission is more reliable than one that has been tested in only 50 such patients. However, until it is tested in children, or in ARDS arising from other causes, or at other clinical timepoints, its generalizability is unknown. We thus caution against the false security of solely relying on a high sample size in evaluating the robustness of a single study.

MULTICOHORT ANALYSIS AND DATA SHARING

An efficient, inexpensive way of tackling the problem of heterogeneity is to combine studies that represent the broad spectrum of disease. At a fixed total sample size, greater reproducibility is gained when the samples are integrated from a greater number of smaller sized studies, rather than vice versa (15). Our group has worked with many collaborators in repeatedly demonstrating that leveraging biological and technical heterogeneity across multiple cohorts can identify generalizable diagnostic and prognostic biomarkers in a diverse set of diseases including organ transplant, cancer, and autoimmune and infectious diseases (16–24). These early successes of multicohort analysis are firmly rooted in the hypothesis that although a broad representation of a disease could make the discovery of a biomarker challenging; such biomarkers are more likely to be reproducible and generalizable when tested in novel circumstances (Fig. 1). On the other hand, making full use of these studies often requires making imperfect comparisons (e.g., integrating datasets that use multiple different definitions for AKI). Although no hard rule can be set, we feel it both reasonable and pragmatic to use data to their fullest extent, even if the statistical methods are simple or some assumptions are slightly violated, as long as such caveats are fully explained and discussed.

Figure 1.

The benefit of incorporating heterogeneity. Biomarkers discovered in a homogeneous cohort are highly likely to work in external cohorts that are similar to the original cohort, but less likely to work in different settings. Biomarkers that are discovered in heterogeneous cohorts are more likely to be generalizable across a broad spectrum of patients.

However, multicohort studies are only possible when data are shared (such as is now required for most genome-wide expression studies). We thus argue for the increased appropriate sharing of molecular data from clinical trials, so that multiple cohorts can be combined in the discovery of new biomarkers (25). In many research areas, data are held privately, preventing such reuse. For instance, we searched the literature for metabolomics, clinical studies in critical care and identified 28 studies (total n = 2,322), out of which only two studies made their raw data publically available (Table 1) (26–53). Public sharing of these data would allow for meta-analysis and data-driven hypotheses generation, avoiding the need for each new cohort to “reinvent the wheel.” It is clear that studies that are performed on single cohorts can be successful at producing robust biomarkers if pitfalls are avoided; but we propose that these investigators make their data available for (and themselves take part in) efforts at later meta-analysis.

TABLE 1.

Metabolomics Studies in Critical Care

Year	Reference	Metabolomic Method	Cases	Controls	n	Tissue	Summary Data Available	Raw Data Available
2009	Mao et al (26)	NMR	Healthy/SIRS	Multiple organ dysfunction syndrome	78	Serum	No	No
2011	Stringer et al (27)	NMR	Healthy	Sepsis + acute lung injury	19	Plasma	Partial	No
2012	Schmerler et al (28)	LC/MS/MS	SIRS	Sepsis	143	Plasma	No	No
2013	Antti et al (29)	GC/MS	None	Sepsis	16	Serum	Partial	No
2013	Blaise et al (30)	NMR	None	Trauma	22	Plasma	No	No
2013	Mickiewicz et al (31)	NMR	Healthy/SIRS	Sepsis	140	Serum	No	No
2013	Langley et al (32)	GC/MS	SIRS	Sepsis	150	Plasma	Yes	Deposited but not available
2013	Kamisoglu et al (33)	LC and GC/MS	Healthy	Endotoxin	19	Plasma	Yes	No
2013	Seymour et al (34)	UHPLC/MS/MS	Sepsis survivor	Sepsis nonsurvivor	30	Plasma	Partial	No
2014	Bos et al (35)	GC/MS	Ventilated non-ARDS	ARDS	101	Breath	No	No
2014	Dessì et al (36)	GC/MS	Healthy	Sepsis (n = 1)	14	Urine	No	No
2014	Evans et al (37)	LC/MS	Healthy	ARDS	26	Bronchoalveolar lavage fluid	Partial	No
2014	Fanos et al (38)	GC/MS and NMR	Healthy	Sepsis	15	Urine	No	No
2014	Mickiewicz et al (39)	NMR	SIRS	Sepsis	59	Serum	Partial	No
2014	Rogers et al (40)	GC/MS	SIRS	Sepsis, ± ARDS	240	Plasma	Yes	No
2014	Su et al (41)	LC/MS/MS	Healthy/SIRS	Sepsis	65	Serum	No	No
2015	Puskarich et al (42)	NMR	Sepsis	Sepsis with L-carnitine	30	Serum	Partial	No
2015	Garcia-Simon et al (43)	NMR	Sepsis survivor	Sepsis nonsurvivor	64	Urine	Partial	No
2015	Mickiewicz et al (44)	NMR	ED nonsepsis	Sepsis	238	Serum	No	No
2015	Su et al (45)	LC/MS	Sepsis survivor	Sepsis nonsurvivor	35	Serum	Yes	No
2016	Richter et al (46)	LC/MS/MS	Febrile neutropenia	Sepsis	48	Plasma	Partial	No
2016	Kauppi et al (47)	GC/MS	ED nonsepsis	Sepsis	114	Whole blood	Partial	Yes
2016	Ferrario et al (48)	LC/MS/MS	Sepsis survivor	Sepsis nonsurvivor	20	Plasma	Yes	Yes
2016	Neugebauer et al (49)	LC/MS/MS	SIRS	Sepsis	406	Serum	Yes	No
2016	Liu et al (50)	UHPLC-MS	Sepsis survivor	Sepsis nonsurvivor	50	Serum	Partial	No
2016	Stewart et al (51)	LC/MS/MS	Healthy neonate	Necrotizing enterocolitis/sepsis	19	Serum	No	No
2016	Mogensen et al (52)	LC and GC/MS	SIRS	Sepsis	85	Plasma	Partial	No
2016	Lau et al (53)	UHPLC-MS	Noninfected controls	Bacteremia	76	Plasma	Yes	No

ARDS = acute respiratory distress syndrome, ED = emergency department, GC = gas chromatography, LC = liquid chromatography, MS = mass spectrometry, NMR = nuclear magnetic resonance, SIRS = systemic inflammatory response syndrome, UHPLC = ultra-high performance liquid chromatography.

All raw data and most summary data are unpublished, preventing their reuse.

To aid the broader community in this effort, we have made available on our website (http://khatrilab.stanford.edu/sepsis) a large number of existing studies of gene expression in sepsis along with source code for analysis (20, 21). This is one resource any researcher can use to further explore their biomarkers in broader clinical context and to test their generalizability in silico prior to embarking on a clinical trial.

BIG DATA AND BIOMARKERS

One of the biggest benefits of the data-driven omics approach to biomarker discovery is the possibility of discovering novel pathobiology in the heterogeneity of critical illness. Although hypothesis-driven studies of familiar cytokines (e.g., those resulting from activation of the nuclear factor-kB or interferon pathways) may be warranted by preclinical models, many common pathways are activated by multiple stimuli at a cellular level (54) and so are unlikely to be highly specific for a given syndrome. Similarly, clinical scores that use similar data available in an electronic health record (vitals, common laboratories, etc.) are unlikely to be highly specific for multiple conditions. An omics approach, by contrast, can sift through thousands or tens of thousands of candidate biomarkers to find the best fit for a given condition. Unfortunately, the promise of omics is also thus its major pitfall: false positives are likely when there are many more variables than samples in a study. This has contributed to some early failures in the field. It is thus worthwhile to have a general framework with which to approach biomarker development studies.

BIOMARKER STUDIES: A CONCEPTUAL FRAMEWORK

There are two excellent guidelines for how to determine the rigor of multivariable prediction models (the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis [TRIPOD] statement [www.tripod-statement.org] [55]) and of diagnostic accuracy studies generally (the Standards for Reporting Diagnostic accuracy studies [STARD] statement [www.stard-statement.org] [56]). In addition to these reporting guidelines, we suggest the following list of questions that we use to help place a study in context:

1. What is the context of the reported diagnostic comparison? Are there existing comparators/gold standards for this question, and what is their diagnostic accuracy in practice? Ideally, a study will compare a new biomarker to a gold standard, possibly with a net reclassification, but this is often not possible and only reported in later validations.

2. How “locked down” is the reported biomarker? Are the biomarkers themselves being selected? Is a statistical model being retrained in the new cohort (i.e., if using a regression model, were the coefficients determined prior to testing)? If a cutoff is used, was it determined prospectively, and is it standard?

3. How generalizable is the validation cohort being studied? Is this merely a random held-out set of the original discovery cohort? Is it from the same center as the discovery cohort? Does it represent a new area of application?

4. Is this biomarker useful? If applied clinically, would it change practice?

5. Is there a link to known biology? In our opinion, this may not be necessary at first, especially if the study is searching in spaces that are not well-studied (outside the “street lamp” of common studies). Procalcitonin, for instance, had not been well-characterized as part of the immune response (and the biology remains somewhat unclear today) at its first testing as a biomarker for bacterial infection (57).

6. Can the biomarker be measured in a reasonable amount of time to make it useful in critical care? Although not a reason to dismiss results, many of the diagnostic applications in critical care require a rapid turnaround time. A more complex process, or one that relies on new technologies, may take longer to be clinically translated, and will be harder to replicate in validation studies. For example, neutrophilic CD64 as measured by flow cytometry is highly diagnostic for sepsis but has a turnaround time of several hours (58).

In addition, it is helpful to put a study into context in terms of biomarker development (Fig. 2). Early validation may be simply the generation of receiver-operating characteristic curves in similar cohorts to the initial discovery cohort. As evidence accumulates, however, such studies should 1) investigate the application of the biomarker in a broader variety of cohorts that represent the full spectrum of disease and 2) compare the test to known standards for easy comparison. For instance, the later-stage validation of a biomarker for the prognosis of sepsis that is not compared to either lactate or clinical severity scores (e.g., Sequential Organ Failure Assessment) is unhelpful. Similarly, a study examining the diagnostic power for a locked, commercially available biomarker is important, but not as helpful as one examining outcome after intervention.

Figure 2.

Maturity of biomarkers: a conceptual framework. ROC = receiver-operating characteristic.

MOVING FORWARD IN THE BIG DATA ERA

The promise of precision medicine is to have the right treatment for the right patient at the right time. In critical care, our immediate need is to get the basics right. For instance, we should first try to answer urgent clinical questions (such as which patients need antibiotics), and then pose new ones that may not have been previously answerable (such as whether there are molecular subtypes of sepsis). As omics and big data technologies proliferate, so too will studies utilizing them as biomarkers in critical illness (studying the genome, epigenome, transcriptome, proteome, metabolome, lipidome, microbiome, and quantified self, to name a few). In all cases, we must remember the extreme heterogeneity of critical illness, and strive for generalizable disease-defining diagnostics and robust biomarkers that can help the entire spectrum of critical care research and delivery.