A Benchmark for Evaluation of Algorithms for Identification of Cellular Correlates of Clinical Outcomes

Abstract

The Flow Cytometry: Critical Assessment of Population Identification Methods (FlowCAP) challenges were established to compare the performance of computational methods for identifying cell populations in multidimensional flow cytometry data. Here we report the results of FlowCAP-IV where algorithms from seven different research groups predicted the time to progression to AIDS among a cohort of 384 HIV+ subjects, using antigen-stimulated peripheral blood mononuclear cell (PBMC) samples analyzed with a 14-color staining panel. Two approaches (FlowReMi.1 and flowDensity-flowType-RchyOptimyx) provided statistically significant predictive value in the blinded test set. Manual validation of submitted results indicated that unbiased analysis of single cell phenotypes could reveal unexpected cell types that correlated with outcomes of interest in high dimensional flow cytometry datasets.

Introduction

A wide range of computational tools is required for systematic analysis of high-dimensional and high-throughput single cell data in different clinical and experimental settings. Despite recent developments in flow cytometry bioinformatics [1], guidance for end users regarding the appropriate use of these methods has been scarce. The FlowCAP Consortium has developed a series of benchmarks for objective evaluation of computational tools in different settings. FlowCAP-I focused on the evaluation of cell population identification algorithms compared to manual analysis by human experts for a selection of different cell populations across several datasets [2]. While this approach was helpful to demonstrate that algorithms can identify known populations, it did not directly address the effectiveness of algorithms that identified cell types not previously delineated by the human analysts. To address this limitation the next FlowCAP challenges focused on the identification of cellular correlates of an independent biological variable (e.g. a clinical outcome). While these challenges demonstrated that computational analysis could outperform manual analysis for identification of cell types correlated with external factors, they did not establish a benchmark for evaluating the relative performance of data analysis pipelines because in one challenge many algorithms were able to achieve perfect classification accuracy, whereas in another challenge none of the algorithms (or human expert) could successfully identify a cell type that correlated with the clinical outcome. Therefore, a more challenging benchmark for evaluation of these algorithms was required to help evaluate their relative performance. In this article we report the results of the latest FlowCAP challenge that addresses this requirement through the use of a complex dataset to evaluate data analysis strategies for identification of correlates of clinical outcome – in this case the time to progression to AIDS in HIV positive participants.

Materials and Methods

We used data from a long-standing natural history study of HIV [3], from which 384 individuals were selected, with known HIV seroconversion dates (so that an estimated date of infection could be calculated) and long-term clinical follow-up [4]. For each individual, the time to AIDS diagnosis (“survival time”) was known, and patients were censored upon initiation of treatment or when lost to follow-up. We chose the earliest available cell specimen (<18 months post-infection (p.i.); mean = 7 months p.i.); our previous analysis showed that the timing of the cell specimen had no relationship to survival time. Cells were stained with two polychromatic flow cytometry panels: the first assessed the resting phenotypes of T-cells directly ex vivo [4] (but was not used for this study), while the second characterized cytokine and other functional responses to HIV, after stimulation with pools of overlapping peptides from HIV-Gag proteins (Chattopadhyay et al., under review). Specifically, the latter panel examined the cytokines IFNg, IL2, and TNF, along with a costimulatory molecule (CD154) upregulated by CD4+ T-cells upon activation and a marker of T-cell degranulation (CD107a). Cell surface markers that define the maturity of the HIV-specific T-cell were examined in this panel as well (CD45RO, CCR7, CD27, and CD57) so that the functional responses could be ascribed to cell types well-described in the literature (e.g., central memory T-cells, senescent T-cells, etc.). Manual analysis of the dataset was performed previously, and showed that no specific characteristics of the anti-HIV T-cell response in the selected samples could predict survival time (specifically, the following characteristics were tested: total magnitude of the HIV Gag or Env specific T-cell responses, magnitude of individual functional responses, polyfunctionality of the CD4 or CD8 response, and differentiation state of the response; these are described in greater detail in a manuscript under review (Chattopadhyay, et al)). It is important to note, however, that the manual data analysis examined only cell characteristics that were known to the biologists a priori. Manual data analysis of all possible combinations of markers in the dataset was not feasible. All selected samples from divided into two equal training- and test-sets randomly and uniformly. No correlation with the clinical outcome was observed across the two sets (pvalue > 0.22). The FlowCAP participants were blinded to the clinical outcomes in the testset, which was used for independently verifying the submitted results.

To ensure the feasibility of the study, members of the FlowCAP organizing committee not participating in the submission of algorithm results performed an independent analysis [5]. The raw Gag-stimulated samples were pre-processed using an optimized logicle transformation [6] and 1000 cells per patient were selected randomly. K-means clustering was used to identify 50 cell types simultaneously in all patients using the pooled cells. For each cluster, the frequency of the cells (i.e., the number of cells in the cluster divided by the total number of cells) was used for the correlative analysis (Cox-proportional hazards model) on the training set. The median expression of each marker for each cell type as well as log-rank p-values for each cluster were calculated (Supplemental Figure 1). The most significant clusters included CD3+CD4+ and CD3+CD8+ populations that also express a range of other markers (e.g., CD27, CCR7, and IL2) and lacked expression of others (e.g., CD57 and TNF). These clusters remained correlated with the clinical outcome in the test set, suggesting that successful identification of correlates of disease progression in this dataset was possible (Supplemental Figure 2).

Data from unstimulated and Gag-stimulated samples were distributed to all participants. The dataset was randomly partitioned into a training set and a test set of equal sizes. The clinical information (survival time and censorship status) of the patients in the test set was not provided to the participants. Each participant provided a vector of final predictions (extracted from one or a combination of several cell types) that was most correlated with the clinical outcome (as measured by a lon-rank test on a Cox proportional hazards regression). The complete source code as well as the required software packages for independent reproduction of the results was also required. The results were evaluated by an independent log-rank test on the blinded test set.

Results

Twenty-four groups requested to participate in FlowCAP-IV and received the data. We received 9 submissions from 7 groups (Table 1) using combinations of different algorithms including [7]-[17]. Log-rank test p-values for the submitted result indicated that while several algorithms identified cell types strongly correlated with the clinical outcome in the training dataset, only the flowDensity-flowType-RchyOptimyx and FloReMi.1 approaches maintained a significant correlation (p < 0.05) in the blinded test set. flowDensity is a supervised sequential bivariate clustering algorithm that mimics manual gating methods. flowType-RchyOptimyx is an approach that uses partitioning of cells into categories (e.g., positive or negative) using 1-D density estimate to calculate cut-offs for each marker and enumerate all cell types in a sample, followed by dynamic programing to efficiently construct k-shortest paths to important cell populations. FloReMi.1 combines flowType for cell population identification with a random forest approach using cell population-based features to build a survival regression model. One version of the three different versions of FloReMi submitted (FloReMi.1) remained significantly correlated with the clinical outcome after Bonferroni adjustment for multiple hypothesis testing to account for the multiple algorithm submissions submitted (i.e., as the number of submitted results increases it is possible that some will have significant correlations in the test dataset due to chance alone).

Table 1.

summary of all submitted results.

Name	Approach	Availability	Reference
BorFlowFP	Exhaustive all-relevant feature selection with the Boruta method over the flow cytometry fingerprints data followed by modeling with Survival Random Forest algorithm.	Boruta and FlowFP are free, open-source R packages available from CRAN and BioConductor, respectively.	[15]
FloReMi.1	Partitioning of cells (e.g., into positive or negative) to identify cell populations, followed by feature extraction of both cell population size and MFI values. Features with minimal redundancy were selected as input for a random survival forest for survival time prediction.	R Code to reproduce our results is available at www.github.com/SofieVG/FloReMi and in Supplementary data from FlowRepository.org	[12]
FloReMi.2	As FloReMi.1, but using a Cox-proportional hazards model for survival time prediction	R Code to reproduceresults is available at www.github.com/SofieVG/FloReMi and in supplementary data from FlowRepository.org	[12]
FloReMi.3	As FloReMi.1, but using an additive hazards model for survival time prediction	R Code to reproduce results is available at www.github.com/SofieVG/FloReMi and in supplementary data from FlowRepository.org	[12]
flowDensity/flowType/RchyOptimyx	Density-based getting and partitioning of cells (e.g., into positive or negative) followed by dynamic programing to ID k-shortest paths to important cell populations	Free, open-source R packages available from BioConductor	[10,7,8,9]
GANN	Identifying profiles of individual bin channels of fluorescence for the different cell markers which can be used for distinguishing different groups/categories	C++ written program, available by contacting the author and in supplementary data from FlowRepository.org	[11,13,14]
RTOMT	Partitioning of cells (e.g., into positive or negative). Combining clinical diagnosis and survival time into a single target vector. Regression tree.	Free, open-source code, Matlab available in supplementary data from FlowRepository.org	NA
SPADE.SNR	SPADE was used to derive cell clusters in high-dimensional space. Important clusters were selected by a signal-to-noise ratio based on cell frequency of subjects who showed disease progression.	Freely available implementation in Matlab	16
EMMIXflow	Clustering using expectation maximization fitting of skew-t mixture models was used for feature extraction. A random survival forest was used for building the predictive model.	R source code available in supplementary data from FlowRepository.org	17

The algorithms used different cell types for the final predictions. For example, FloReMi.1 identified a CD3+ CD4− CD45RO− CD27− CCR7− population (likely effector CD8+ cells [18], with mixed expression of CD57+ and CD57− cells. Higher frequencies of these cells early in HIV disease were correlated with worse prognosis, consistent with previous reports in chronic infection[19]. The flowType pipeline identified a CD27− TNF− CD154− population with predictive value, but this could not be interpreted further because there was either mixed or negative expression of all the other markers studied. This is not necessarily a limitation of the algorithm; it suggests that more markers were needed in the original experiment to positively identify a disease correlate. Still, the result is consistent with that obtained by FloReMi, in that CD27− cells were also identified.

Importantly, several algorithms identified populations within the CD14/VIVID+ channel, originally designed to exclude dead cells and monocytes. Cluster #3 of our independent analysis (Supplemental Figure 1) further describes this population, which includes CD3+ CD4− CD14/VIVID+ CD57− cells. This cluster was correlated with outcome both in the training and test sets. To validate this population, we constructed a gating strategy for this cell type (Supplemental Figure 3) and applied it to the unstimulated set of samples. The correlation with the clinical outcome was preserved (Supplemental Figure 4), arguing against the possibility of experimental artifact, since the identified population appeared significant in all samples (stimulated and unstimulated). Stated differently, since the population lacked cytokine-expressing cells, we expected to observe it (and find the same clinical correlations) in the unstimulated samples. Finally, we confirmed these results in a different dataset from the same cohort[10], using manual gates to identify the CD3+ CD4- CD14/VIVID+ CD57- cells. The boundaries of the gates were manually adjusted using FlowJo (Treestar Inc, Ashland, OR) to account for technical variations across the two datasets. The frequency of the manually identified cell type remained correlated with the clinical outcome (Supplemental Figure 5), confirming robustness of the results in this dataset.

We performed additional experiments to better define these cells. Based on their phenotype, we surmised that the correlation arose from dead cells or live monocytes. Notably, although the cells appeared CD3+, this was likely a consequence of well-known experimental artifacts: non-specific antibody binding by dead cells [20] and/or monocyte binding of our mouse anti-human IgG1 CD3 antibody via FCgRIIA (CD32A, [21]). To disentangle the staining patterns of the viability marker between dead cells and monocytes, we stained cells with VIVID and then measured CD14 expression using a different fluorochrome. Live, CD14+ monocytes bound intermediate levels of VIVID compared to live lymphocytes and dead monocytes (Supplemental Figure 6); a likely result of the increased size and protein content of live monocytes compared to lymphocytes[20]. This suggested that the correlation found in the dataset represented live monocytes, not dead cells (further confirmed by a separate experiment, where the correlate population expressed other markers of monocytes, CD16 and CD11c). This was confirmed by testing viability data from the original dataset, obtained using microscopic examination of ethidium bromide/acridine orange stained cells (EB/AO) at the time of thaw. There was no relationship between the frequency of dead cells, or the number of cells recovered from the vial as measured by a hemocytometer, and survival times (Supplemental Figure 7). This strongly suggests that monocytes accounted for the correlation between CD14+/VIVID+ cells in early infection and survival time, consistent with previous data [22]–[25]. This analysis demonstrates how careful analysis of biology, experimental artifacts, and computational results can provide an unbiased analysis of all available cells, revealing unexpected cell types that correlate with clinical outcomes.

Discussion

As high throughput and high dimensional flow and mass cytometry datasets become increasingly common, there is a critical need for automated data analysis approaches. As demonstrated by our HIV natural history studies, correlates of survival often lie within complex populations of cells not typically queried by directed, manual data analysis. Comprehensive analysis of a dataset, in which cells expressing all possible combinations of markers are examined, is unfeasible with classical, manual data analysis alone. Fortunately, there are a wide variety of approaches to automated data analysis. However, there have been few carefully controlled comparative studies that have assessed their performance. FlowCAP provides a forum for such comparison.

FlowCAP-IV was designed to address the limitations in some of the past challenges by using a dataset containing complex correlates of an externally defined variable (time to AIDS progression/diagnosis). Participants were provided a training set and their submitted results (survival status) on a blinded and independent test set. There were a number of reasons this dataset was selected for a FlowCAP challenge. It contained well-defined clinical data, providing a standardized and meaningful outcome variable. The data was also robustly collected using stringent quality control approaches, with previous studies performed in the same cohort of individuals. A variety of parameters were examined in this study, including multiple markers of T-cell maturity and function, providing a complex test. Our previous analysis also suggested that correlates of survival were contained within complex cell populations, and thus would be challenging to identify. Finally, the size of the dataset was consistent with the types of studies increasingly performed today, for which better data analysis solutions are needed.

A key product of our work with automated data analysis is the recognition that correlates of biologically important outcomes can lie in cell types that are not well described in the literature. In fact, the reliance on classically defined cell types for manual data analysis is problematic in high-dimensional experiments, as the definition of a particular cell type might differ by investigator or paper. For example, a central memory T-cell in one study may be CD45RA− CCR7+, while in another study the cell is defined as CD45RA− CD27+. When all of these markers are measured together in a sample, it becomes difficult to decide which combination of markers should define the cell type of interest and only a few cell populations are assessed due to the burden of manual analysis. In this regard, automated unbiased approaches that consider expression of all markers are vastly preferable since no potentially important cell populations are missed; this is a feature of all the automated approaches tested in this challenge. Notably, we show here that pre-gating data (to remove dead cells or monocytes, for example) can cause similar problems, as the excluded cell types may include correlates. However, this consideration is balanced by the problem of interpreting biology from the potentially artifactual staining patterns on the excluded cells.

The methods included in this study can be broken into several components: 1) Preprocessing (e.g., transformation [26] and normalization [27]); 2) feature extraction (e.g., clustering [28], probability binning [29], combinatorial gating [7], [10], and cluster matching [30], [31]); 3) supervised analysis (e.g., single-variate [32]–[34] and multi-variate models [35]); 4) characterization [8], [36] and visualization [16], [37], [38]. Not only do these components vary in their individual performance across different biological applications, their interactions with each other in a large data analysis pipeline also further complicates the choice of appropriate methods [39] The establishment of objective benchmarks like the one reported here enables further analysis in which different components are examined subject to a wide range of technical and biological variations to identify the combination with optimal performance. It is also possible that the optimal combination may differ by data type or study design. FlowCAP provides a platform for future evaluation of these possibilities.

Figure 1.

The log-rank test p-values of the submissions on (A) the training-set and (B) the test-set.