Supervised Machine Learning with CITRUS for Single Cell Biomarker Discovery

Abstract

CITRUS is a supervised machine learning algorithm designed to analyze single cell data, identify cell populations, and identify changes in the frequencies or functional marker expression patterns of those populations that are significantly associated with an outcome. The algorithm is a black box that includes steps to cluster cell populations, characterize these populations, and identify the significant characteristics. This chapter describes how to optimize the use of CITRUS by combining it with upstream and downstream data analysis and visualization tools.

1. Introduction

Analyzing single cell data with traditional, biaxial approaches can lead investigators to focus only on cell populations that they are already aware of and might lead them to miss important discoveries in other populations [1, 2]. This is particularly important in studies that are designed to identify novel biomarkers for a specific clinical endpoint like disease status or response to therapy.

CITRUS [3] is a supervised machine learning algorithm for single cell biomarker discovery. It has been used to identify single cell biomarkers in many contexts. For example, CITRUS has been used to identify: biomarkers for response and resistance to CD19 CAR-T cell therapy in chronic lymphocytic leukemia [4]; age-related epithelial subpopulations in breast tissue [5]; peripheral blood biomarkers for response to anti-CTLA-4 and anti-PD-1 checkpoint blockade [6]; and differences in immune populations between placebo-responders and controls in a mouse model [7].

The CITRUS algorithm includes steps to cluster cell populations, characterize these cell populations, and identify the characteristics of the cell populations that are statistically significant biomarkers that distinguish outcome groups. In this way, CITRUS is a black box that identifies statistically significant single cell biomarkers for any discrete outcomes of interest. Since the CITRUS algorithm seamlessly integrates these steps, it is able to search efficiently for the boundaries of a cell population that is significantly different between the outcome groups. The clustering step identifies nested clusters such that each “parent” cluster contains all of the cells within each of its “child” clusters. Then, the significance testing is performed for each of these nested levels of cell populations: clusters that are too narrow or too broad in their boundary of a population will not be identified as significant. Since CITRUS performs so many statistical tests, it uses permutations and the False Discovery Rate method [8] to report only single cell biomarkers that are significant after correction for multiple hypothesis testing. The single cell biomarkers it identifies can be either population abundances or population-specific median expression of functional markers. In addition to correlative analyses that identify all of the significant single cell biomarkers, CITRUS can build predictive models that identify a minimum set of biomarkers necessary to discriminate effectively between the groups, and estimate the accuracy with which these biomarkers will be able to do this.

This chapter describes our protocol for combining the CITRUS algorithm with upstream and downstream data analysis tools to take full advantage of CITRUS’s unbiased biomarker discovery. With any machine learning method, data QC and cleanup is essential to generating meaningful results. In addition, effective exploratory data analysis is essential to understanding the biology and heterogeneity that drives those results (or lack of results). Thus, our protocol includes considerations for experimental setup, data tidying and quality control, visualization and exploratory data analysis, clustering and biomarker discovery, and visualization for communication of results (Fig. 1). As a part of this pipeline, we utilize viSNE [9, 10], biaxial plots, and heatmaps for the visualization and exploration of significant clusters. Together, these tools allow unbiased identification of significant single cell biomarkers and effective communication of these results.

Fig. 1.

Single cell biomarker discovery pipeline with CITRUS. Key steps outlined for biomarker discovery analysis pipeline. (a) Experimental setup (b) data tidying and quality control and (c) exploratory data analysis, clustering and biomarker discovery, and visualizations to communicate results. Steps (a) and (b) are required before initiating any of the components in step (c). Each part of step (c) (i–iii) may be initiated following experimental setup and data tidying/quality control

2. Materials

CITRUS was developed in and may be run using the R programming language [3]. This chapter discusses CITRUS as implemented in Cytobank, a cloud-based informatics platform available at www.cytobank.org.

3. Methods

3.1. Considerations for Analysis Setup

Thinking through certain aspects of the analysis setup before beginning analysis will reduce time spent in later steps of the analysis.

1. During data acquisition, ensure that channel short names and long names are consistent across all of the files that will be analyzed together. As much as possible, include information about outcome groups, stratifying variables, and technical factors in file names.

2. Define the outcomes and stratifying dimensions you will compare. The outcomes should be discrete (i.e., have defined categories with groups of samples) and the samples in each group should be independent (i.e., not consist of multiple observations on the same individual, even under different conditions) (see Note 1). In addition to outcomes, there may be stratifying variables, or other dimensions in the data that you wish to compare across. These stratifying variables should also be discrete, but samples split across these stratifying variables do not have to be independent (see Note 2). An example of typical outcome, stratifying, and technical variables you might track is shown in Table 1.

3. Determine the number of samples you will have for each group of outcome and stratifying dimensions (see Note 3). CITRUS will be run for your outcome variable within each stratifying group. For each of these comparisons, you need a minimum of three samples per group to run CITRUS, and we recommend that you have at least eight samples per outcome group to obtain reliable results.

4. Determine whether the samples were run in multiple batches and whether control samples are available. If the samples were run in multiple batches, determine whether the outcome and stratifying variables were distributed equally across the batches and whether a technical control sample was run with each batch (see Note 4). Identify any factors based on experimental setup or lab observations while processing samples and acquiring data that were tracked; these can be used later for sensitivity analyses.

5. Determine whether the scientific question requires a correlative or predictive analysis, or both. Correlative analyses will identify all of the significant biomarkers that can be detected in the data, while predictive analyses will identify a minimum set of markers needed to best predict an outcome (see Note 5). The choice of analysis will dictate which models to select in CITRUS (Fig. 2).

6. Determine which markers in the panel will be used for phenotyping and which will be used as functional readouts (see Note 6).

7. Complete standard preprocessing steps for the mass cytometry data, which may include normalization for time and de- barcoding if barcoding was used.

Table 1.

Example of Stratifying, Outcome, and Technical Variables in an Experiment

Stratifying or outcome variable	# in each category
Baseline (pre) or post-therapy	Pre: 40 files Post: 40 files
Therapy response	Responders (R): 28 individuals (2 files each) Non-responders (NR): 12 individuals (2 files each)
Patient ID	40 unique patient IDs (2 files each)
Staining batch (1–4), labeled by date	1 = 20 files 2 = 20 files 3 = 20 files 4 = 20 files

In this example study, patient peripheral blood mononuclear cells (PBMCs) have been acquired in batches for samples pre- and post-therapy. The study outcome variable to be used for CITRUS (highlighted in italics) is therapy response. Timepoint (pre- or post- therapy) is a stratifying variable in this study; one CITRUS run will be performed on the pre-therapy samples, and a separate CITRUS run will be performed on the post-therapy samples. Staining batch will be used to look for batch effects. Timepoint, therapy response, patient ID, and staining batch are annotated by Cytobank sample tags. The number of files expected in each category is outlined in the second column and used to check the files present in Cytobank before starting the analysis

Fig. 2.

Flowchart to determine correlative vs. predictive CITRUS and CITRUS models to use. The CITRUS models to select are determined by the overall analysis goals. Use the above flowchart to determine which models will address the scientific question. Regardless of the model selection, significant populations identified by CITRUS can be visualized on viSNE maps

3.2. Data Tidying and Quality Control

The goal of the data tidying and quality control (QC) steps is to identify any problems in the data that may result in the exclusion of samples, markers, or batches of samples from analysis. For issues that are identified related to batches of samples, normalization may be performed in an attempt to control for batch effects.

1. Upload data to Cytobank and annotate the data according to outcome groups, stratifying variables, and technical factors using Cytobank’s Sample Tags (see Note 7).

2. Check that the number of files in each tagged group matches what you expect according to your experimental design (Table 1) (see Note 8).

3. Check that the panels are annotated as expected based on the assay(s) that were run. Each CITRUS analysis should be performed on a set of samples run on a single panel (see Note 9). Fix any discrepancies in channel names that may have happened accidentally during acquisition. Identify any channels that will need to be excluded because they differed across samples in the experiment. If there are a few samples that have extensive panel differences, you may choose to exclude these samples instead of excluding discrepant channels.

4. Identify files that are outliers for total event count and check whether there were issues acquiring data for that sample. These samples will likely need to be excluded from the analysis.

5. Check the scales for all markers in the panel across all samples. Perform or adjust transformations as required (see Note 10). If your data are off the plot or bunched on the axis you may need to adjust your scale range settings (see Note 11).

6. Examine technical controls for evidence of batch effects or problems with marker staining using histograms or other basic plot types (see example in Fig. 3a, Note 12). If you find markers for which staining does not appear to have worked (either within a particular batch or across all batches), these markers should be excluded from the CITRUS analysis.

7. If you think that observed differences across technical control replicates are due to batch effects, run viSNE on the technical control samples from each batch to identify whether observed differences between batches will impact downstream analysis results (see example in Fig. 3b, Note 13).

8. If you identify batch effects that will impact your downstream analysis results, one approach is to normalize the data [11, 12] (see Note 14). After normalization, reexamine the scales, rerun viSNE on the technical controls, and look again for evidence of batch effects to assess whether the normalization was successful (see example in Fig. 3b). Alternatively, batches for which large batch effects are identified may be excluded, so that only batches without any apparent differences across batches are included.

Fig. 3.

Technical control evaluation. Peripheral blood from one healthy donor was collected and cryopreserved to use as a technical control in this study. A different antibody cocktail was prepared for each of the five indicated staining batches and evaluated for differences across batch using the technical control. (a) Pre- normalized data for technical control samples shown. Histograms displaying a few measured markers are examined across batch. Intensity of indicated marker and the negative and positive spread are compared across batch. The red line indicates the middle of the plot. Data reveal staining differences. (b) Pre and post- normalization viSNE maps are compared by coloring across channel. Here, only CD19 expression is displayed. Technical control samples per batch were assessed using viSNE as an exploratory data analysis tool. Row 1 show data pre-normalization and row 2 show data post z-score normalization by batch. Data post- normalization highlight more uniform intensity of CD19 expression and reveal more uniform viSNE “island” shapes. The effect of normalization can be quantified using a clustering algorithm to define cell populations and then measuring the correlation of the cell populations across the batches pre- and post-normalization

3.3. Pre-gating

Pre-gating to intact singlets or similar might be required in order to perform some of the investigations described above. For the rest of the CITRUS analysis, further pre-gating might be desired. This decision will depend on the cell populations the panel is designed to measure, the number of samples being comparing, and the scientific questions being answered. If a large number of samples are being compared, where differences in rare cell populations are being investigated, further pre-gating may be required to include enough cells per sample to detect these rare cell populations reliably.

1. At a minimum, pre-gate to the events the panel and scientific question are focused on. For example, gate to select CD45+ cells if lymphocytes are the focus, or gate to remove cells stained with antibodies in a dump channel.

2. Determine the maximum number of events you can include in a single viSNE map. This is limited in Cytobank by your subscription options [13]. Care should be taken in interpreting viSNE plots of over 100,000 cells, as data crowding will start to skew population distributions.

3. Determine the number of samples to include in the viSNE map. At a minimum, this is the number of samples that will be included in a single CITRUS run for a set outcomes within a given stratifying variable group (see Note 15). Exclude samples based on the conclusions from data tidying and QC.

4. Determine the number of events per file in the initial pre-gated population. For the exercise below, use the file with the minimum number of events in the pre-gated population. If one or more files have a very low number of events in this population, plan to exclude them and use the next lowest number.

5. Determine the expected frequency of the rarest cell population of interest as a fraction of your initial pre-gated population.

6. Use the flowchart in Fig. 4 to determine whether you need to pre-gate further (see Note 16) and whether you will use multiple pre-gated populations for your analysis (see Note 17).

7. For the final pre-gated population(s), check whether any samples have pre-gated event counts lower than the number of events you determined was necessary per file. These samples should be excluded from the analysis.

Fig. 4.

Determine required pre-gating using event count. For example, we use the following logic to determine how many events are needed to be able to detect Tregs with a desired 5% CV starting with all CD3+ T cells for 50 samples. Let’s say the file with the lowest number of CD3+ T cells has 50,000 events in this population, and the maximum number of events we can include on viSNE is 1.3 M. (a) 1.3 M max events/ 50 samples = 26,000 events, which is <50,000. (b) We estimate that Tregs have a frequency of about 2% of CD3+ T cells. Thus, the # of events needed per file to detect Tregs is 20,000. 26,000 events is greater than the 20,000 events needed per file. (ii) No additional pre-gating is required in this example. If our samples did not meet these conditions, we could (i) reduce the number of samples being analyzed, or (c) perform further pre-gating to isolate a smaller and more specific group of cells containing the Treg cells

3.4. Visualization and Exploratory Data Analysis

1. Run one viSNE for each pre-gated population and set of samples that are necessary for the data and scientific questions above. Exclude any samples selected for exclusion during file QC, data QC, and pre-gating. Set the total number of events based on pre-gating event count calculations, and select equal sampling. Select the channels that were identified as phenotyping markers in Subheading 3.1 as clustering channels, but exclude any channels used during pre-gating (for a discussion on exceptions see Note 18). If including a large number of total events, increase the number of iterations and perplexity (see Note 19).

2. Check the quality of the resulting viSNE map to ensure that no further tuning of any of the algorithm settings is required. In Cytobank, set up a working illustration that shows the viSNE map for every file on the rows colored by every channel on the columns. If this map is not high enough quality (see Note 20), adjust the advanced settings and rerun viSNE.

3. Create a concatenated viSNE map across all samples, and use the working illustration dot plots to color it by the expression of each of your clustering marker channels (see Note 21, and an example of separate viSNE maps in Fig. 5 compared to a concatenated viSNE map in Fig. 6c).

4. Create views of the viSNE map to help with exploration of the data. viSNE analysis, as described here, can also be used after performing CITRUS clustering (described below) to help explain the CITRUS results (see Note 22). Samples may be kept individual or concatenated in groups according to the outcome, stratifying, and technical variables you annotated with Cytobank’s Sample Tags. To visualize differences in individual samples or groups of samples, use dot plots colored by marker expression to visualize heterogeneity. When using viSNE after CITRUS clustering (as described below) dot plots colored by marker expression can be used to visualize differences between and within cell clusters identified by CITRUS, and dot plots colored by density can be used to visualize differences in abundance of cell clusters identified by CITRUS.

5. If using some markers as functional readouts that weren’t included in the parameters used to generate the viSNE map (such as phosphorylated antigens or cytokines) these can still be visualized on the viSNE map by creating a colored dot plot (see example Fig. 5 and Note 23).

Fig. 5.

Using viSNE for Exploratory Data Analysis. Synthetic example viSNE grid, colored by marker expression, shown for patients with B cell malignancies. Two individuals from each group of interest are shown—two patients from Outcome A (therapy responders) and two patients from Outcome B (therapy non-responders). PBMCs were isolated from each patient prior to therapy (baseline) and stimulated with anti-BCR, to see if baseline stimulation responses will correlate with the outcome of therapy. Expression of each marker is indicated by scale bar (red for high, blue for low). In this example, each column represents the expression of the indicated marker (p-S6, p-PLCγ, p-ERK) for intact live cells from peripheral blood mononuclear cells (PBMCs). The B cell “island” of the viSNE map is circled in blue and reveals the inter- and intra-heterogeneity in expression of measured markers between groups and across patients. This exploratory data analysis highlights how patients who responded to therapy (from outcome group A) were able to respond to BCR stimulation prior to therapy; whereas, patients from outcome group B were not. The strongest response from patients in outcome group A was for p-S6 expression

Fig. 6.

Biomarker discovery results visualization. CITRUS using SAM with a 5% FDR was run on non-T cell subsets (i.e., CD3 negative pre-gated cells in PBMC samples) to look for any differences in abundances of non-T cells among three different disease types—Group A, Group B, Group C. (a) Display of the CITRUS defined clusters. Highlighted clusters show cell subsets that had a significantly different abundance among the Fig. 6 three cohorts. The parent clusters (which are “highest” up the tree), 1 and 2, are highlighted in orange and blue, respectively. (b) Boxplot displays show that the both clusters are most abundant in Group A and least abundant in Group C. (c) viSNE plots for each sample were concatenated and the expression of the displayed marker for the combined assessed patient population is displayed (e.g., CD11c, CD14). The significant clusters identified by CITRUS are shown on the viSNE coordinates in the bottom row to the right. Comparison of these plots helps identify the phenotype of these significant clusters. (d) Heatmap and (e) histograms show the expression of the indicated marker on the significant clusters (1 and 2) vs. all other non-T cells

3.5. Clustering and Correlative Biomarker Discovery Analysis

1. Run one CITRUS analysis to compare a set of outcome groups (such as therapy responders vs. non-responders) within one stratifying variable (such as pre-therapy, or post-therapy) that was defined above. For each CITRUS, start from the viSNE experiment that contains the appropriate samples (see Note 24) and select the pre-gated population. For each CITRUS run, choose to either look at differences in the abundances of cell populations between your groups (as a percentage of your pre-gated population) or the medians of functional marker expression in cell populations between your groups (see Note 25). Regardless of which choice you make, select the clustering channels that you chose to use for phenotyping markers in Subheading 3.1. Assign the “file groupings” based on the outcome groups you are planning to compare (see Note 26). For correlative analysis, choose the SAM model [14] and in the “Additional Settings” box, make sure that the number of events sampled per file is large enough that all of the events from your viSNE will be included in the CITRUS. Set the minimum cluster size (%) to the approximate percentage of the lowest abundance population you are interested in detecting. Set the cross-validation folds to 1 and leave the FDR set at the default (see Note 27). Select “Normalize Scales.”

2. Select a false discovery rate (FDR) level at which you will assess the SAM results (see Note 28).

3. Determine whether you have any significant results. If there are no significant biomarkers found, no model output will be returned (see Note 29).

4. If the model returns significant results, select the clusters to focus on. Use the “featurePlots” output from CITRUS for the chosen FDR and identify the significant nodes highlighted in red on the tree. When there are multiple significant clusters that are immediate neighbors of each other, CITRUS will draw a bubble around these neighbors. Choose the highest cluster on the tree within each bubble to focus on (see example in Fig. 6a and Note 30).

5. Review how these clusters differ between the outcome groups on the “features” boxplots output from CITRUS for the chosen FDR (see example in Fig. 6b and Note 31).

6. From the completed CITRUS page, export the clusters you selected to focus on to a new experiment along with the original files.

7. Map these clusters back to viSNE to understand their phenotypes (see example in Fig. 6c and Note 32). Concatenate the events separately for each selected cluster and all of the original files. Create a viSNE dot plot for all of the concatenated files with the significant cluster files overlaid on the original file (see Note 33). Compare this viSNE map showing the significant clusters to the concatenated viSNE expression maps that were created in Subheading 3.4 to help explain the phenotype of the significant CITRUS clusters.

8. Examine the selected clusters on the “markerPlots” (see Note 34) and the “clusters” histograms (see example in Fig. 6e and Note 35) output by CITRUS. Check that this expression matches what you see on the viSNE map in the region where the significant clusters are overlaid.

9. Create a heatmap in the working illustration (see example in Fig. 6d) of the exported clusters experiment with the clustering channels as the rows and the concatenated files for each significant cluster as the columns. Check that this expression matches what is seen on the viSNE map in the region where the significant clusters are overlaid (see Note 36).

10. Back-gate the significant CITRUS clusters onto biaxial plots using color overlay dot plots (see Note 37).

3.6. Clustering and Predictive Biomarker Model Development

1. Run one CITRUS for each predictive model you would like to build (see Note 38). Set the parameters for this CITRUS as for the correlative analysis (see Subheading 3.5), except: (1) choose the PAM model [15], and additionally choose the lasso model [16] if only two groups are being compared (see Note 39); and (2) in the “additional settings” box, set the cross-validation folds to at least 5 and no more than 10 (more will take longer) and set the FDR to the significance level at which you would like to evaluate your results. Take note of the number of samples in each CITRUS group (see Note 40).

2. Determine whether the biomarkers included in the predictive model are able to discriminate the groups using the model error rate plot (see example Fig. 7 and Note 41). Identify the green dot representing the CV min constrained model (see Note 42) and calculate the cross-validated estimation of the accuracy of your model to discriminate your outcome groups as one minus the model cross-validation error rate (y-axis) at this point. Determine whether the accuracy of your model is “good” given the context of your analysis (see Note 43).

3. Determine whether the predictive model results are also significant (when using PAM) by comparing the predictive model constrained by the minimum cross-validated error rate (the CV min model, green circle) to the model constrained by the false discovery rate (the FDR model, yellow triangle) on the model error rate plot (see Note 44). Check that the FDR constrained model contains more features than the CV min constrained model (i.e., the yellow triangle should be further to the right on the X-axis); this indicates that all of the features in the predictive model are statistically significant.

4. As described for the correlative analysis results in Subheading 3.5, select the clusters to focus on, review how these clusters differ between the outcome groups, and map the clusters back to the viSNE map to understand their phenotypes. Compare the results of the predictive and correlative analyses and of all of the predictive models that you build (see Note 45).

Fig. 7.

“Good” vs. “bad” example model error rate plots. Two examples of model error rate plots output from predictive CITRUS analyses using PAM and an FDR of 5% with equally balanced groups. In each plot, the cross-validation error rate (red line) and false discovery rate (blue line) are shown as the number of features in the model is increased (top x-axis) according to the regularization threshold used to build the model (bottom x-axis). (a) A “good” model error rate plot. An FDR constrained model (yellow triangle) was identified with more features than the CV constrained models (cv.min, cv.1se). The CV min model had a CV error rate of around 5% using three features (cluster abundances in this example), which was considered excellent for this context. (b) A “bad” model error rate plot. The CV error rate never drops below 50%, meaning the model fails to accurately predict the groups 50% of the time. No FDR constrained model was identified (no yellow triangle), meaning that the features included in the CV constrained models are not statistically significant

3.7. Further Analyses to Validate the CITRUS Results

There are several reasons why additional CITRUS analyses might be run, in order to understand how your results are affected by certain factors in your data. Generally, sensitivity analyses involve excluding certain samples or markers from the CITRUS analysis and checking that you identify the same trends that you see when you use all of the data to ensure the effects that you find are consistent across your entire dataset and are not being driven by outlying samples or markers (see Note 46). In addition to the sensitivity analyses, once you have identified biomarkers of interest, you should repeat your CITRUS analysis to guard against false positives due to stochasticity (see Note 47).

1. Rerun CITRUS excluding any samples or markers for which there are concerns about data quality. Such data quality concerns may have been identified during the data tidying and QC phase.

2. Rerun CITRUS excluding any batches of samples for which there are concerns about batch effects, or, if several samples per group within each batch are available, rerun CITRUS stratifying analyses by batch.

3. Rerun CITRUS excluding, combining or splitting samples from certain outcome groups if there are more than two (see Note 48).

4. If a predictive model is being built with unbalanced groups, rerun CITRUS many times, each time arbitrarily selecting subsets of the samples so that the groups used to build the model and assess accuracy are equal in size.

5. Rerun the entire viSNE + CITRUS analysis pipeline described above to reproduce any results of interest. Do not set the same seed for viSNE (as rerunning with the same seed will result in an identical analysis). For correlative CITRUS analyses, significant clusters with the same phenotype in each repeat should be identifiable (see Note 49). For predictive CITRUS analyses, the cross-validated model error rate seen on the model error rate plot should be similar for each repeat (see Note 50).

6. Experimental replication of findings in additional samples is also an important step in any project.

7. Use exploratory data analysis on the viSNE map as described in Subheading 3.4 to assist in understanding why a particular CITRUS cluster is significant or not in its clinical, biological, and experimental context.