Comprehensive Validation of the FAIM3:PLAC8 Ratio in Time-matched Public Gene Expression Data

To the Editor:

Scicluna and colleagues recently reported the gene expression ratio of Fas apoptotic inhibitory molecule 3 (FAIM3) to placenta-specific 8 (PLAC8) as a new sepsis diagnostic biomarker (1). Accurate sepsis diagnosis is critical, as the mortality rate of sepsis increases for each hour that antibiotics are not administered (2). The FAIM3:PLAC8 ratio was discovered in a cohort comparing critically ill patients within 24 hours of admission for community-acquired pneumonia with noninfected patients. Scicluna and colleagues found that the FAIM3:PLAC8 ratio had areas under the receiver operating characteristic curve (AUCs) of 0.845 and 0.784 for diagnosing community-acquired pneumonia in their discovery and validation cohorts, respectively (1).

Of paramount importance in the further study of any proposed biomarker is its validation in independent heterogeneous cohorts, as estimates in discovery cohorts are often overfit. As we recently reported, there are multiple public gene expression datasets that compare noninfected systemic inflammatory response syndrome (SIRS)/trauma patients with patients with sepsis from various sources; we comprehensively organized all publicly available cohorts into 13 time-matched subcohorts and performed gene expression meta-analysis to derive an 11-gene set diagnostic for sepsis compared with sterile SIRS/trauma (3). These public datasets can be reanalyzed to test the diagnostic power of new gene expression biomarkers as well.

We thus tested the FAIM3:PLAC8 ratio in the previously described time-matched gene expression cohorts (four intensive care unit admission cohorts, nine Glue Grant trauma subcohorts; total n = 881). In intensive care unit admission cohorts, the FAIM3:PLAC8 mean AUC was 0.72 (range, 0.60–0.88; Figure 1A). In the Glue Grant trauma cohorts (4), which were split into time-matched subcohorts comparing patients within ±24 hours of diagnosis of infection with time-matched trauma patients who were never infected, the mean AUC was 0.83 (range, 0.62–0.98; Figure 1B) and generally increased over time since injury. In comparison, our 11-gene set had mean AUCs of 0.82 and 0.87 in the intensive care unit admission and Glue Grant cohorts, respectively. Finally, we tested our 11-gene set for diagnostic power in the 134 discovery samples from the study of Scicluna and colleagues (GEO dataset GSE65682; AUC = 0.79).

Figure 1.

Receiver operating characteristic curves showing the diagnostic power of sterile SIRS/trauma patients versus patients with sepsis in multiple public gene expression datasets. (A) Datasets comparing patients at intensive care unit admission. (B) Subcohorts of time-matched trauma patients, split out by days since initial injury. Buffy coat and neutrophil data were from separate patient cohorts. AUC = area under the receiver operating characteristic curve; CAP = community-acquired pneumonia; FAIM3 = Fas apoptotic inhibitory molecule 3; ICU = intensive care unit; MICU = medical ICU; PICU = pediatric ICU; PLAC8 = placenta-specific 8; SIRS = systemic inflammatory response syndrome.

Perhaps the greatest surprise is poor performance of the FAIM3:PLAC8 ratio in GSE40012 (5), another dataset that specifically studied community-acquired pneumonia. However, both GSE40012 and GSE32707 (6) were run on Illumina platforms and performed worse than almost all other tested cohorts; technical differences may thus be contributing to the observed decrease in AUC. In addition, the performance of FAIM3:PLAC8 in the Glue Grant data (mean AUC, 0.83) was nearly as good as in the original discovery cohort.

As noted by Scicluna and colleagues, biomarkers often perform better in combination than alone. In the cohorts tested here, the FAIM3:PLAC8 ratio was generally highly correlated to our sepsis diagnostic 11-gene set (3) (mean Spearman correlation, 0.68; SD, 0.10), and combining the two separate scores into a single diagnostic did not yield a synergistic increase in AUC (not shown).

We wish to congratulate Scicluna and colleagues for a well-performed study and for their derivation of the FAIM3:PLAC8 sepsis diagnostic biomarker. The AUCs of the FAIM3:PLAC8 ratio may not be high enough to bring it into practice in sole use; however, the team demonstrated superiority to procalcitonin in their cohort. Thus, it may be worthy of further study in combination with other infection markers. In addition, the use of a simple gene expression ratio (similar to other popular methods such as differences of arithmetic or geometric means) allows for easy testing and comparison without constructing a complicated model. Overall, given the relative ease with which researchers can now test a gene expression biomarker in multiple public cohorts, we recommend that further gene expression biomarker studies should include these comparisons as a basic benchmark.