Discriminating Bacterial and Viral Infection Using a Rapid Host Gene Expression Test

Abstract

Objective:

Host gene expression signatures discriminate bacterial and viral infection but have not been translated to a clinical test platform. This study enrolled an independent cohort of patients to describe and validate a first-in-class host gene expression test for bacterial vs. viral discrimination (HR-B/V).

Design:

Subjects were recruited from 2006–2016. Enrollment blood samples were collected in an RNA preservative and banked for later testing. The reference standard was an expert panel clinical adjudication, which was blinded to gene expression and procalcitonin results.

Setting:

Four U.S. Emergency Departments.

Patients:

623 subjects with acute respiratory illness or suspected sepsis.

Interventions:

45-transcript signature measured on the BioFire® FilmArray® System in ~45 minutes.

Measurements and Main Results:

HR-B/V test performance characteristics were evaluated in 623 participants (mean age 46 years; 45% male) with bacterial infection, viral infection, co-infection, or non-infectious illness. Performance of the HR-B/V test was compared to procalcitonin. The test provided independent probabilities of bacterial and viral infection in ~45 minutes. In the 213-subject training cohort, the HR-B/V test had an area under the curve (AUC) for bacterial infection of 0.90 (95% CI 0.84-0.94) and 0.92 (95% CI 0.87-0.95) for viral infection. Independent validation in 209 subjects revealed similar performance with an AUC of 0.85 (95% CI 0.78-0.90) for bacterial infection and 0.91 (95% CI 0.85-0.94) for viral infection. The test had 80.1% (95% CI 73.7%-85.4%) average weighted accuracy for bacterial infection and 86.8% (95% CI 81.8%-90.8%) for viral infection in this validation cohort. This was significantly better than 68.7% (95% CI 62.4%-75.4%) observed for procalcitonin (P<0.001). An additional cohort of 201 subjects with indeterminate phenotypes (co-infection or microbiology-negative infections) revealed similar performance.

Conclusions:

The HR-B/V host gene expression test measured using the BioFire® System rapidly and accurately discriminated bacterial and viral infection better than procalcitonin, which can help support more appropriate antibiotic use.

Introduction

Acute respiratory illness (ARI) is the most common reason for acute healthcare visits (1, 2). Patients with ARI are inappropriately treated with antibacterials at high rates due to challenges in discriminating viral, bacterial, or non-infectious etiologies (3, 4). Diagnostics that reliably discriminate bacterial and viral etiologies in patients with ARI could improve clinical management.

Currently available diagnostic strategies for ARI largely focus on pathogen identification, which identifies a pathogen in only a minority of cases (5). Depending on the type of pathogen assay, additional limitations include long time to result, inability to discriminate infection from colonization, and the need for a priori suspicion for the specific pathogen. In contrast, measuring the host response to infection offers a complementary, unbiased strategy that overcomes many of these limitations. Procalcitonin, a commonly used host response marker, has exhibited mixed results in discriminating bacterial from viral ARI etiologies (6–8) and in guiding antibacterial use (9, 10). In contrast, recent studies have shown that peripheral blood host gene expression accurately discriminates bacterial, viral, and non-infectious etiologies of ARI (11–17).

Host gene expression tests are commercially available for non-infectious diseases such as oncology and transplant rejection (18, 19). Due to their complexity, these tests are typically performed in referral laboratories and require days to return results. Consequently, host gene expression tests are not currently available for infectious diseases, which require results at the point-of-need for real-time clinical decision-making. Our study objectives were to demonstrate the feasibility of developing a host gene expression test that could be utilized at the point-of-need and to demonstrate its ability to discriminate bacterial, viral, and non-infectious illness in a clinical cohort. To achieve these objectives, we developed a first-in-class research-use-only (RUO) test for the BioFire® System to quantify host gene expression for suspected infection. This host response bacterial/viral test (HR-B/V) was evaluated in a multisite study of patients presenting to the Emergency Department (ED) with suspected infection of up to 28-days duration using clinical adjudication as the reference standard. The findings presented here demonstrate the feasibility of a rapid, point-of-need, host response test that differentiates bacterial and viral illness.

Materials and Methods

Study Design

Studies were approved by relevant Institutional Review Boards and in accordance with the Declaration of Helsinki. All subjects or their legally authorized representatives provided written informed consent. Patients were enrolled by convenience sampling in four Emergency Departments (EDs) from 2006-2016: Duke University Medical Center (Durham, NC), Durham VA Health Care System (Durham, NC), UNC Health Care (Chapel Hill, NC), and Henry Ford Hospital (Detroit, MI). This was done as part of three consecutively executed observational studies: CAPSOD (ClinicalTrials.gov NCT00258869) (20–22), the Community Acquired Pneumonia and Sepsis Study (CAPSS), and the Rapid Diagnostics in Categorizing Acute Lung Infection (RADICAL). Patients were eligible for CAPSOD and CAPSS if they were ≥6-years with a known or suspected infection of <28-days duration and exhibited two or more systemic inflammatory response syndrome criteria (23). RADICAL enrolled patients age ≥2-years with ARI of <28-days duration. Prior antimicrobial exposure was not exclusionary. ARI was defined as having at least two qualifying symptoms or one qualifying symptom and at least one qualifying vital sign abnormality. Qualifying symptoms included headache, rhinorrhea, nasal congestion, sneezing, sore throat, itchy/watery eyes, conjunctivitis, cough, shortness of breath, sputum production, chest pain, and wheezing. Qualifying vital sign abnormalities included heart rate ≥90 (or ≥110 for children aged 2-6 years), respiratory rate ≥20, and temperature ≥38.0ºC or ≤36.0ºC. There were 1,274 subjects enrolled in CAPSOD, 1,320 in CAPSS, and 944 in RADICAL. Subjects were selected from this larger pool based on the availability of a PAXgene Blood RNA sample and confirmatory microbiology (with the exception of suspected bacterial and suspected viral cases). In suspected cases where no microbiological etiology was identified, consecutive subjects were selected for inclusion in this study.

Diagnostic Reference Standard

In the absence of a gold standard for bacterial/viral discrimination, we performed retrospective adjudications as previously described (20, 24). Clinician adjudicators had experience managing patients with ARI defined by subspecialty training in hospital medicine, emergency medicine, infectious diseases, or pulmonary/critical care medicine or by >2 years of post-graduate clinical experience in that field. Two independent adjudications were performed >28-days after enrollment using the full medical record, supplemental etiology testing, and case report forms but not host-response test nor procalcitonin results. This avoided incorporation bias and allowed procalcitonin to be used as an independent comparator. Supplemental testing included the BinaxNOW S. pneumoniae urinary antigen test (Alere) and a multiplex viral respiratory pathogen panel (ResPlex V2.0, Qiagen; Respiratory Viral Panel, Luminex; or Respiratory Pathogen Panel, Luminex). For discordant adjudications, a consensus panel of at least three adjudicators entered the final adjudication by consensus or majority vote.

BioFire Testing

A custom, RUO BioFire test was designed to measure 45 host mRNA transcripts (Supplementary Table 1) that were previously shown to be differentially expressed in viral, bacterial, or non-infectious causes of ARI (Supplementary Methods) (16, 25). A larger pool of targets was initially selected. The assays included in the final HR-B/V test were selected through iterative evaluations and selection for robust performance (strong linearity and low variability in quantitative reporting) with the BioFire nested, multiplex PCR chemistry. The test included internal and endogenous normalization controls, selected for their low coefficients of variation (<0.1). Upon loading 100µL of PAXgene-preserved blood (~27µl whole blood volume) into the disposable pouch, automated sample extraction, nucleic acid purification, reverse-transcription, and two stage (multiplexed-nested) real-time PCR was performed by the BioFire® FilmArray® Instrument (Supplementary Figure 1). All assays were tested in duplicate within each pouch in case of assay failure although failure rates were low in both the discovery (0.18%) and validation (0.13%) cohorts. The real-time PCR curve quantification results, as expressed in Cq values, were collected for each assay for each sample. The Cq value is a semi-quantitative measure of target abundance defined by the PCR cycle number at which a target is detected. Therefore, lower values (i.e., fewer PCR cycles) indicates a greater abundance of the target in a sample.

Statistical Analysis

Normalized target expression values were used to build two independent sparse logistic regression models: viral vs. non-viral infection and bacterial vs. non-bacterial infection (26). The two probabilities are independent, which allows for the identification of coinfection (i.e., both positive) or no infection (i.e., both negative). To generate these probabilities, the BioFire® Cq values were used to build a logistic regression model trained on subjects with known phenotype. The regularization parameter of the model and performance metrics were estimated using nested Leave-One-Out Cross-Validation (LOOCV), where the internal LOOCV was used for the regularization parameter and the outer LOOCV for performance estimates including Area Under the receiving operating Characteristic (AUC), Positive Percent Agreement (PPA) and Negative Percent Agreement (NPA) (27). PPA is calculated in the same manner as sensitivity whereas NPA is calculated in the same manner as specificity. The terms PPA and NPA were used instead of sensitivity and specificity given the use of a reference standard rather than a gold standard, as recommended for the clinical performance evaluation of molecular diagnostic tests (28). After training the model, parameters were fixed and applied to independent cohorts. Thresholds for the bacterial and viral tests (27.5% and 41.7%, respectively) were calculated to optimize the average weighted accuracy (AWA) (29). AWA is a pragmatic metric of diagnostic yield or global utility of a diagnostic test that integrates sensitivity and specificity, accounts for disease prevalence within the population, and accounts for the clinical implications of false-positive and false-negative results. The AWA was calculated assuming a 10-30% bacterial infection prevalence, 50-80% viral infection prevalence, r=0.25 for bacterial classification, and r=2 for viral classification. Details regarding the development of the AWA metric and how it specifically applies to this test are described elsewhere (29). Comparison of HR-B/V to procalcitonin was performed using the chi-square test. Procalcitonin concentrations ≥0.25ng/ml indicated bacterial infection (10).

Additional details regarding study design, case definitions, procalcitonin measurement, transcript selection process, and statistical analysis are included in the Supplementary Materials.

Results

Clinical Cohort

We enrolled 623 subjects at four EDs presenting with suspected sepsis or ARI (Figure 1). Of these, 422 had microbiologically confirmed phenotypes (Supplementary Table 2) and were randomly assigned to training (n=213) or validation (n=209) cohorts so the numbers of bacterial, viral, and non-infectious illness cases were balanced (Table 1). The remaining 201 subjects (82 suspected bacterial infections, 83 suspected viral infections, and 36 coinfections) were tested but not included in calculations of performance characteristics due to the absence of a reliable reference standard.

Figure 1: Experimental Flow.

The Indeterminate Phenotypes group was not used to calculate performance characteristics but used to demonstrate the distribution of HR-B/V test results in these groups. ARI = Acute Respiratory Illness.

Table 1:

Subjects Characteristics

Demographic and Clinical Variables	Total Cohort (N=623)	Training Cohort (N=213)	Validation Cohort (N=209)	Coinfection (N=36)	Suspected Bacterial Infection (N=82)	Suspected Viral Infection (N=83)
Age – mean years (SD)	46.1 (17.9)	48.6 (17.9)	47.4 (18.6)	46.1 (16.4)	47.2 (18.9)	35.3 (14.1)
Male sex – no. (%)	280 (44.9)	100 (46.9)	96 (45.9)	12 (33.3)	49 (59.8)	23 (27.7)
Race – no. (%)a
White	317 (50.9)	110 (51.6)	97 (46.4)	25 (69.4)	53 (64.6)	32 (38.6)
Black	284 (45.6)	96 (45.1)	104 (49.8)	11 (30.6)	25 (30.5)	48 (57.8)
Other	22 (3.5)	7 (3.3)	8 (3.8)	0	4 (4.9)	3 (3.6)
Etiology – no. (%)b
Bacterial	135 (32.0)	68 (31.9)	67 (32.1)
Viral	183 (43.4)	92 (43.2)	91 (43.5)
Non-infection	104 (24.6)	53 (24.9)	51 (24.4)
Abnormal temperature – no. (%)c	236 (37.9)	84 (39.4)	79 (37.8)	19 (52.8)	41 (50.0)	13 (15.7)
Comorbidities
Chronic Lung Disease	166 (26.6)	57 (26.8)	63 (30.1)	7 (19.4)	24 (29.3)	15 (18.1)
Chronic Liver Disease	14 (2.2)	5 (2.3)	6 (2.9)	2 (5.6)	0	1 (1.2)
Coronary Artery Disease	76 (12.2)	29 (13.6)	26 (12.4)	3 (8.3)	12 (14.6)	6 (7.2)
Diabetes	135 (21.7)	53 (24.9)	47 (22.5)	9 (25)	14 (17.1)	12 (14.5)
Dialysis	12 (1.9)	8 (3.8)	4 (1.9)	0	0	0
Heart Failure	42 (6.7)	23 (10.8)	11 (5.3)	1 (2.8)	6 (7.3)	1 (1.2)
HIV Infection	11 (1.8)	6 (2.8)	5 (2.4)	4 (11.1)	8 (9.8)	2 (2.4)
Hypertension	280 (44.9)	112 (52.6)	90 (43.1)	14 (38.9)	36 (43.9)	28 (33.7)
Immunosuppressive Therapy	62 (10.0)	24 (11.3)	25 (12.0)	2 (5.6)	7 (8.5)	4 (4.8)
Malignancy	59 (9.5)	23 (10.8)	25 (12.0)	2 (5.6)	7 (8.5)	2 (2.4)
Hospitalized – no. (%)	304 (48.8)	118 (55.4)	107 (51.2)	23 (63.9)	51 (62.2)	5 (6.0)

^aRace was reported by participants.

^bEtiology is only defined in subjects who had a microbiologically confirmed pathogen. Those with coinfection, suspected bacterial, or suspected viral etiologies were excluded from this calculation.

^cAbnormal temperature is defined as ≤35.5ºC or ≥38.0ºC.

Classification

Training Cohort:

Thresholds for positive and negative test results were selected to maximize the AWA, which incorporates the clinical significance of false positive and false negative errors (Supplementary Figure 2). Using nested LOOCV in the training cohort, the HR-B/V test had an AWA of 83.3% (95% CI 77.4%-88.2%) for the identification of bacterial infection and 85.9% (95% CI 80.4%-90.1%) for viral infection (Figure 2A and Table 2). The corresponding AUCs were 0.90 (95% CI 0.84%-0.94%) and 0.92 (95% CI 0.87-0.95), respectively. Precision-recall curves for this and subsequent comparisons are shown in Supplementary Figure 3. The model trained in these 213 subjects was then fixed and used for all subsequent tests. A heatmap highlighting the contribution of each transcript in the signature to discriminate bacterial, viral, and non-infectious illness is shown in Supplementary Figure 4.

Figure 2: Classification performance.

The test assigns each subject a probability of viral infection (y-axis) and bacterial infection (x-axis). The vertical line is the threshold for bacterial infection (0.275) while the horizontal line is the threshold for viral infection (0.417). The top left region is indicative of viral infection, bottom left indicates no infection, top right suggests bacterial/viral coinfection, and the bottom right indicates bacterial infection. For panels A and B, colors represent the adjudicated phenotype: blue=bacterial, yellow=non-infectious illness, red=viral. A) Classification of 213 training cohort subjects. B) Classification of 209 validation cohort subjects. (C) The probabilities of bacterial infection (y-axis) as measured by the HR-B/V test vs. procalcitonin (x-axis) are plotted for each subject. The vertical line represents a procalcitonin threshold of 0.25ng/mL. The horizontal line is the threshold for a positive bacterial HR-B/V test. Subjects in the training cohort (n=213) are represented by circles whereas validation cohort subjects (n=209) are represented by a plus. Blue represents cases adjudicated as bacterial. Red represents cases adjudicated as non-bacterial (viral or non-infectious illness). The top right region was identified as bacterial by both tests. The bottom left region represents a non-bacterial classification by both tests. (D) Receiver operating characteristic plot for bacterial vs. non-bacterial infection using the HR-B/V test vs. procalcitonin in the training (discovery) and validation cohorts. E) Classification of 36 subjects with bacterial/viral coinfection (superinfection). Red circles represent clinically suspected cases of superinfection without microbiological confirmation. Blue circles represent microbiologically confirmed superinfection. F) 83 suspected viral infections (red circles) and 82 suspected bacterial infections (blue circles) are shown.

Table 2:

Performance characteristics of the HR-B/V test in the training and validation cohorts.

Test Group	AWA	AUC	PPA	NPA	Overall Accuracy	LR+	LR−	F1	PTP
Training Cohort
Bacterial Infection	83.3 (77.4-88.2)	0.90 (0.84-0.94)	85.3 (75.8-92.9)	81.4 (75.0-86.8)	82.6 (77.5-86.9)	4.6 (3.2-6.8)	0.18 (0.09-0.3)	0.758 (0.676-0.829)	52.2 (44.0-61.7)
Viral Infection	85.9 (80.4-90.1)	0.92 (0.87-0.95)	85.9 (78.1-92.1)	86.0 (78.9-90.9)	85.9 (80.8-90.1)	6.1 (4.1-10.0)	0.16 (0.09-0.26)	0.840 (0.772-0.892)	91.6 (87.3-94.4)
Procalcitonin	77.1 (70.4-83.0)	0.84 (0.76-0.89)	67.6 (54.3-77.6)	86.2 (79.0-90.9)	80.3 (74.6-85.4)	4.9 (3.1-8.1)	0.38 (0.25-0.52)	0.687 (0.579-0.770)	53.9 (43.1-67.4)
Validation Cohort
Bacterial Infection	80.1 (73.7-85.4)	0.85 (0.78-0.90)	79.1 (68.9-87.9)	81.0 (73.6-86.9)	80.4 (75.0-85.6)	4.2 (2.9-5.9)	0.26 (0.15-0.41)	0.721 (0.626-0.794)	49.9 (41.5-58.9)
Viral Infection	86.8 (81.8-90.8)	0.91 (0.85-0.94)	89.0 (81.4-94.7)	84.7 (77.1-90.5)	86.6 (81.8-90.9)	5.8 (3.8-9.4)	0.13 (0.07-0.22)	0.853 (0.796-0.901)	91.2 (87.4-94.4)
Procalcitonin	68.7 (62.4-75.4)	0.72 (0.62-0.79)	53.7 (41.0-65.3)	83.1 (76.5-88.5)	73.7 (66.7-79.4)	3.2 (2.1-4.99)	0.56 (0.43-0.71)	0.567 (0.463-0.667)	43.5 (33.7-54.6)

All values are presented with a 95% confidence interval. AWA = Average Weighted Accuracy. PPA = Positive Percent Agreement. NPA = Negative Percent Agreement. LR+ = Likelihood Ratio Positive. LR- = Likelihood Ratio Negative. PTP = Bayesian post-test probability, representing the expected post-test probability for a pre-test probability with a uniform prevalence distribution in the range of 0.1-0.3 for bacterial infection and 0.5-0.8 for viral infection.

Validation Cohort:

In the 209-subject validation cohort, the test had an AWA of 80.1% (95% CI 73.7%-85.4%) for bacterial infection and 86.8% (95% CI 81.8%-90.8%) for viral infection (Figure 2B and Table 2). The corresponding AUC were 0.85 (95% CI 0.78-0.90) and 0.91 (95% CI 0.85-0.94), respectively.

Infection Site:

To evaluate the test in specific clinical subgroups, we combined the training and validation groups to increase the evaluable sample size. Whereas all viral infections were respiratory in nature, the bacterial infections included a variety of anatomic sites. The PPA for bacterial infection was 84% for respiratory tract (n=50), 71% for urinary tract (n=35), 86% for vascular device (n=14), 75% for skin/soft tissue (n=12), 100% for intra-abdominal (n=12), and 92% for other sites (n=12) (Supplementary Table 3).

Procalcitonin:

In the combined cohort of 422 subjects, the HR-B/V test was significantly better at discriminating bacterial from non-bacterial etiologies compared to procalcitonin (Figure 2C–D). This was driven by a higher PPA for the HR-B/V test (82.2% vs. 60.7% for procalcitonin; P<0.001). NPA for bacterial infection was similar (81.2% for HR-B/V vs. 84.7% for procalcitonin; P=0.27). Whereas the HR-B/V test distinguishes viral from non-infectious etiologies, procalcitonin does not, which precluded a comparison of the tests for this purpose.

Confounders:

We evaluated the impact of age, sex, ethnicity, and illness severity (as defined by the need for hospitalization). This was done in the combined 422-subject cohort to improve the ability to detect such differences though the study was not powered for these analyses. There were no statistically significant differences due to age, sex, or ethnicity, but there was a higher accuracy among non-hospitalized subjects compared to hospitalized subjects (83.8% vs. 74.2%; P=0.02). (Supplementary Table 4).

Alternative reporting schemes

There is no standardized approach to reporting results of composite biomarkers. Thus far, we applied a single threshold to determine the presence or absence of disease so every tested subject receives a potentially actionable result. However, values close to the thresholds may have greater uncertainty. To account for this uncertainty, we evaluated the impact of two alternative reporting schemes: probability quartiles and inclusion of an equivocal zone.

Infection Score Quartiles:

This scheme provides greater ability to rule-in or rule-out bacterial and viral infection for subjects in the highest and lowest quartiles, respectively. Probability thresholds defined by training cohort quartiles were applied to the validation group. Those thresholds were well-calibrated in both cohorts (Supplementary Figure 5). For bacterial infection diagnosis, the lowest quartile had a PPA of 100% and 94.0% (LR- 0.09) in the training and validation cohorts, respectively (Supplementary Table 5, Supplementary Figure 6). The highest quartile for bacterial infection in the training and validation cohorts had a NPA of 92.4% (LR+ 5.04) and 90.1% (LR+ 4.39), respectively. The HR-B/V test performed better with respect to viral infection with a PPA of 96.7% in both the training and validation cohorts as well as a NPA of 98.3% and 94.9%, respectively. Additional results are shown in Supplementary Table 5 and Supplementary Figure 6.

Equivocal zone:

An equivocal zone decreases the number of subjects with an actionable result, but the diagnostic confidence is higher for subjects above or below the zone’s thresholds. We defined probability thresholds for both bacterial (0.18-0.37) and viral (0.26-0.47) infection that maximized AWA in the training cohort while assigning <15% of subjects to the equivocal zone. Scatter plots of subjects in training and validation cohorts for both the bacterial and viral models are presented along with a graphical representation of the scheme in Supplementary Figure 7. Incorporating an equivocal zone, the HR-B/V test had an AUC of 0.92 and AWA of 87% in the training cohort, as compared to an AUC of 0.86 and AWA of 83% in the validation cohort. For viral infection diagnosis, the training cohort had an AUC of 0.94 and AWA of 91% as compared to an AUC of 0.91 and 87% AWA in the validation cohort. Confusion matrices are presented in Supplementary Table 6. A comparison of results for the three schemes (single threshold, quartiles, and equivocal zone) is shown in Table 6.

Coinfection

We identified 36 cases of respiratory superinfection defined as a bacterial infection arising during or after an antecedent viral infection. In 12 cases, both bacterial and viral pathogens were microbiologically confirmed. In the remaining 24 subjects, a bacterial superinfection was clinically suspected but not microbiologically confirmed. Using the first reporting scheme where a single threshold determined the presence or absence of bacterial and viral infection, the HR-B/V test identified a bacterial infection in all 12 microbiologically confirmed cases (100%) as compared to 75% for procalcitonin (P=0.07) (Figure 2E). Among the 24 cases of suspected superinfection, 11 were identified as having a viral infection (n=11, 45.8%), five (20.8%) had a bacterial host response, five (20.8%) had both bacterial and viral responses, and three (12.5%) were negative for infection. Procalcitonin was ≥0.25ng/ml in 10/24 (41.7%) suspected superinfection cases.

We then evaluated how many subjects in the training and validation cohorts adjudicated as having a monomicrobial infection would have been diagnosed with coinfection using the HR-B/V test. Among the 135 bacterial infections, eight (5.9%) also demonstrated a host response to viral infection. Among the 183 viral infections, twelve (6.6%) also demonstrated a host response to bacterial infection.

Suspected infection

A host-based approach might offer the greatest benefit to patients with ARI but no positive microbiology. In this study, these subjects had clinical syndromes compatible with bacterial or viral infection based on expert panel adjudication but no identified pathogen. Of the 82 suspected bacterial cases, 61 (74.4%) were classified as bacterial or bacterial/viral coinfection. An additional ten (12.2%) were classified as viral while eleven (13.4%) cases were classified as neither (Figure 2F). Procalcitonin was ≥0.25ng/mL in 38 cases (46.3%). In the group of 83 suspected viral cases, thirty (36.1%) had a viral host response, 31 (37.3%) had a bacterial host response, three (3.6%) had both bacterial and viral responses, and nineteen (22.9%) were negative for both. Procalcitonin was <0.25ng/mL in 77 cases (92.8%).

Discussion

The overlap in symptoms due to bacterial, viral, and non-infectious disease leads to diagnostic uncertainty and inappropriate antimicrobial use. Pathogen detection tests play an important clinical role but are insufficient to make a diagnosis in the majority of ARI cases. In the current study, we built upon our previous findings that host gene expression accurately discriminates bacterial, viral, and non-infectious disease (16, 25). Beyond simply validating the signature, this study provides proof of principle that a complex host gene expression signature based on machine learning algorithms can be translated to a clinical platform and validated in an independent test cohort. The HR-B/V test was superior to procalcitonin both with respect to the identification of bacterial infection and the ability to discriminate viral from non-infectious disease. We also showed that blood serves as an accurate biosensor for bacterial infection at multiple different anatomic sites of infection. Furthermore, the HR-B/V test provided clear results even in complex or ambiguous cases such as coinfection or suspected infection.

Host gene expression signatures have been identified for multiple conditions including coronary artery disease, oncology, transplant rejection, and sepsis (18, 19, 30, 31). However, the utility of these tests is limited by a turnaround time of many hours to days due to their high complexity. In this study, we utilized the widely available BioFire system to measure host gene expression of 45 host mRNA biomarkers with results available in about 45 minutes. The biological roles and associated pathways for these targets have previously been described (16, 25). HR-B/V overall accuracy was similar to that previously reported despite using a much smaller signature and translation to an integrated, sample-to-answer platform: 87% using microarray (25), 88% using Taqman Low Density Array RT-PCR (24), and 80-87% in this study (depending on the subgroup).

Multiple studies have described host response signatures to discriminate viral and bacterial infection (14, 16, 32–42). In most cases, these signatures focus only on subjects with bacterial or viral infection without adequately accounting for the possibility of non-infectious illness. To address this limitation, we utilized a composite of two signatures: bacterial vs. non-bacterial (i.e., viral or non-infectious) illness and viral vs. non-viral (i.e., bacterial or non-infectious) illness. The possible outputs of this composite signature are therefore bacterial infection, viral infection, coinfection, or no infection. Although this scheme increases generalizability, it comes at the expense of a lower overall test accuracy. First, the test must distinguish three categories rather than just two, increasing the opportunities for classification errors. Second, there is a high degree of overlap in the host’s response to bacterial infection and non-infectious illness. Third, the AWA statistical approach minimizes false negative bacterial errors, which carry the greatest risk of patient harm. In so doing, it maximizes the test’s sensitivity for bacterial infection at the expense of specificity. These factors may explain the lower overall accuracy among hospitalized subjects, which were more likely to have either bacterial or non-infectious etiologies.

Most biomarker tests measure a single analyte and typically report the value as a concentration (e.g., procalcitonin in ng/ml). However, multi-analyte host response assays convert raw measurements (e.g., Cq for mRNA) into a probability function or composite score. Presently, there are no standardized ways to report such results. Previously described schemes include the use of single thresholds to provide results for all tested patients, quartiles/bands, and equivocal zones (24, 25, 43–45). In this study, we compared results for all three schemes. Our findings do not specify which approach is best but highlight the challenges in reporting results of composite biomarker tests. Moreover, different clinical scenarios (e.g., screening vs. diagnosis) might warrant different approaches.

The HR-B/V test identified all cases of microbiologically confirmed bacterial superinfection. However, some patients with superinfection but no confirmed bacterial pathogen demonstrated a viral host response. This suggests that secondary or persistent viral infections may be responsible for some suspected superinfections. Along these lines, we observed a significant number of patients with suspected (microbiology-negative) bacterial infections who instead had a viral host response. Without microbiological confirmation, these could be adjudication errors, test errors, or perhaps infections due to atypical bacterial pathogens such as Mycoplasma.

The best currently available clinical laboratory test for bacterial vs. viral discrimination is procalcitonin. A meta-analysis demonstrated that procalcitonin-guided algorithms reduced antibiotic use and improved patient outcomes (46). However, this result was not reproduced in the US-based ProACT randomized controlled trial (9). Furthermore, the ability of procalcitonin to discriminate bacterial from viral infection has been limited: 55% sensitivity and 76% specificity in patients with community-acquired pneumonia (8). Procalcitonin performed better in this study (60.7% sensitivity and 84.7% specificity) although not as well as host gene expression, consistent with prior observations (25, 35, 38, 47, 48). Despite these low performance characteristics, procalcitonin is widely used to guide antibacterial use. This underscores that a diagnostic test need not have perfect or even exceptional accuracy to be clinically useful, desirable as that may be. If a biomarker is to be the sole determinant of treatment in the absence of additional clinical data, then performance characteristics should be sufficiently high after accounting for the clinical consequences of false positives and false negatives. However, when used as an adjunct to other clinical information, biomarkers can be clinically useful and actionable even with lower accuracies (e.g., procalcitonin, white blood cell counts, fever). It is noteworthy that our reference standard in this study was clinical adjudication, which is known to be inaccurate. As such, we would not expect (nor desire) performance metrics that are too good to be true. In such a situation, the test would have done little more than perfectly matched an imperfect comparator.

Among this study’s limitations are that it was not powered to detect differences due to demographics such as age, race, and ethnicity. A peripheral blood host gene expression test may not perform as expected in patients with profound abnormalities in their peripheral leukocyte counts or distributions such as neutropenia. A recent evaluation of host gene expression in subjects with immunocompromising conditions revealed slightly lower, but still clinically useful performance (49). We did not assess the kinetics of the host response and are therefore unable to assess response to treatment. Perhaps the greatest limitation is the absence of a gold standard to define the presence of bacterial or viral infection. We therefore relied on expert adjudication, which is imperfect despite being the best available standard. Lastly, a clinical utility study will be necessary to demonstrate that such a test actually mitigates antibiotic overuse without compromising (and perhaps improving) patient outcomes.

Conclusions

This study demonstrates the first translation of a host gene expression signature for the diagnosis of bacterial and viral infection. In doing so, we demonstrate the feasibility of quantifying the host transcriptional response for real-time clinical decision-making, opening a new pathway for test development in multiple clinical domains. The HR-B/V test was superior to procalcitonin.

Table 3. Comparison of result reporting schemes.

We evaluated three different reporting schemes. The single threshold scheme uses a single numerical threshold to determine whether a subject has a bacterial infection in the bacterial vs. non-bacterial model or a viral infection in the viral vs. non-viral model. This single threshold allows for all subjects to be classified. The Quartile scheme uses multiple thresholds to assign subjects into probability bands. The reported PPA and NPA values focus on the top and bottom quartiles, therefore representing 50% of the cohort. The equivocal zone model allows for a probability band in which no call can be made. The equivocal zone thresholds for bacterial vs. non-bacterial and viral vs. non-viral classification were selected to exclude no more than 15% of the cohort.

	Single Threshold			Quartiles			Equivocal Zone
	PPA	NPA	% cohort	PPA	NPA	% cohort	PPA	NPA	% cohort
Training Cohort-Bacterial vs. Non-Bacterial Model	85.3	81.4	100	100	92.4	50	85.4	86.2	85
Validation Cohort-Bacterial vs. Non-Bacterial Model	79.1	81.0	100	94.0	90.1	50	83.2	83.1	88
Training Cohort-Viral vs. Non-Viral Model	85.9	86.0	100	96.7	98.3	50	84.5	92.3	85
Validation Cohort-Viral vs. Non-Viral Model	89.0	84.7	100	96.7	94.9	50	81.5	91.8	85