Data analysis as a source of variability of the HLA-peptide multimer assay: from manual gating to automated recognition of cell clusters

Abstract

Multiparameter flow cytometry is an indispensable method for assessing antigen-specific T cells in basic research and cancer immunotherapy. Proficiency panels have shown that cell sample processing, test protocols and data analysis may all contribute to the variability of the results obtained by laboratories performing ex vivo T cell immune monitoring. In particular, analysis currently relies on a manual, step-by-step strategy employing serial gating decisions based on visual inspection of one- or two-dimensional plots. It is therefore operator dependent and subjective. In the context of continuing efforts to support inter-laboratory T cell assay harmonization, the CIMT Immunoguiding Program organized its third proficiency panel dedicated to the detection of antigen-specific CD8⁺ T cells by HLA-peptide multimer staining. We first assessed the contribution of manual data analysis to the variability of reported T cell frequencies within a group of laboratories staining and analyzing the same cell samples with their own reagents and protocols. The results show that data analysis is a source of variation in the multimer assay outcome. To evaluate whether an automated analysis approach can reduce variability of proficiency panel data, we used a hierarchical statistical mixture model to identify cell clusters. Challenges for automated analysis were the need to process non-standardized data sets from multiple centers, and the fact that the antigen-specific cell frequencies were very low in most samples. We show that this automated method can circumvent difficulties inherent to manual gating strategies and is broadly applicable for experiments performed with heterogeneous protocols and reagents.

Electronic supplementary material

The online version of this article (doi:10.1007/s00262-014-1649-1) contains supplementary material, which is available to authorized users.

Introduction

HLA-class I peptide multimers (HLA-multimers) have evolved as essential tools for T cell research [1, 2]. The relatively easy access to such reagents, together with recent technological developments in multiparameter flow cytometry (FCM), allows the enumeration and characterization of pathogen- or tumor-specific CD8⁺ cytotoxic T lymphocytes with improved precision and sensitivity [3–6]. HLA-multimers are also widely used to monitor T cell responses induced by experimental vaccination against cancer [7–9]. So far, however, only a few studies have demonstrated a positive correlation between the induction of anti-vaccine T cells as measured with HLA-multimers and clinical benefit [10–12].

A prerequisite for evaluating the usefulness of HLA-multimer staining as a biomarker of clinical efficacy is that the assay is sufficiently sensitive to detect antigen-specific cells and to do so robustly when performed in different laboratories. Several technical publications have provided optimized protocols and guidelines for cell staining, instrument set-up, test analysis and data reporting for FCM that can improve sensitivity and robustness [13–18]. Validation and standardization issues are also being addressed [19, 20].

Despite these efforts, methods still vary considerably between institutions and sometimes even between different investigators from the same institution. The results of immune monitoring studies performed at different laboratories are consequently difficult to compare, and this delays progress in the field. One approach to improve the reproducibility of FCM results is to establish core facilities that perform centralized immune monitoring for single- and multicenter studies [21]. Alternatively, one can coordinate inter-laboratory projects that aim to optimize and harmonize T cell assays between institutions, so that similar results are achieved using laboratory-specific protocols and reagents [22].

The Immunoguiding Program of the Association for Cancer Immunotherapy (CIP, www.CIMT/workgroups/CIP) is an international working group which is actively developing the concept of assay harmonization. An important instrument of harmonization is the coordination of iterative proficiency panels for assays commonly used in immune monitoring. Proficiency panels have so far focused on the monitoring of antigen-specific CD8⁺ T cells by HLA-multimers, IFNγ ELISPOT and intracellular cytokine staining (ICS). Such panels organized by CIP and others [23–29] have shown that there is considerable heterogeneity of assay protocols and performance among laboratories with respect to many steps of the assay, including cell handling (freezing, thawing, culture), test conditions and data analysis. CIP and the Cancer Immunotherapy Consortium (CIC) demonstrated in two independent studies that inter-center variability in ICS decreased significantly when common gating instructions were provided to operators analyzing the same data sets [26, 30]. Centralized analyses also reduced heterogeneity [29, 31]. In light of these results, algorithms for automated recognition of cell clusters constitute attractive tools that are increasingly accessible to flow users [32–34].

Impact of the gating step on the outcome of the multimer assay has never been systematically addressed, and it is not known whether different gating strategies would affect variability in the same way as demonstrated for the ICS. Hence, in continuation of our efforts to harmonize the HLA-multimer assay, we report here on the comparison of different analysis methods applied to data generated from the same cell samples by a heterogeneous group of seventeen laboratories. Both the detection rate and the frequencies of CD8⁺HLA-multimer⁺ T cells were assessed. The analysis methods evaluated consisted of two preselected gating procedures applied by the participants, a manual centralized analysis by a single investigator and the utilization of a statistical mixture model for identification of rare cell clusters [35].

Materials and methods

The study was divided into three successive steps (1) inter-laboratory testing, i.e., proficiency panel, (2) analysis of selected data sets by a single operator, i.e., central analysis, and (3) computer-guided analysis (Fig. 1). The methods section is structured according to the guidelines of the Minimal Information About T cell Assays framework [36].

Fig. 1.

Flow chart of the study. Experimental settings and conclusions of the three successive steps are indicated

Proficiency panel

Preparation of reagents at the central laboratory

Samples leukapheresis samples were obtained from healthy volunteers at the Department of Transfusion Medicine of the University Hospital in Tübingen after informed consent. Low-resolution DNA HLA-class I typing and human cytomegalovirus (HCMV) serological status were known. The products were transported to the laboratory at room temperature (RT) and processed within 8 h. After dilution ¼ with sterile PBS, peripheral mononuclear cells (PBMC) were isolated by standard density gradient centrifugation (PAA, Pasching, Austria). PBMC were washed twice in PBS and counted using trypan blue. For freezing, cells were resuspended gently in cold 90 % heat-inactivated bovine serum (Hyclone, Bonn, Germany; serum was pretested for in vitro cell proliferation) plus 10 % DMSO and distributed in cryovials at 15-20 x10⁶ cells/1 ml on ice. Samples were transferred in freezing containers at -80 °C then to a liquid nitrogen tank.

HLA-peptide multimer production synthetic peptides representing two immunodominant, HLA-A*0201 restricted, virus-derived epitopes were used for HLA-monomer refolding, i.e., HCMV (pp65 495-503 NLVPMVATV) and Influenza A (Flu Matrix 58-66 GILGFVFTL) [23]. Fluorescent HLA-multimers were generated by co-incubating monomers with streptavidin-PE or streptavidin-APC (Invitrogen, Darmstadt, Germany) at a 4:1 molar ratio. They were used for screening experiments either directly or after a freezing step at -80 °C (in Tris 20 mM, 16 % glycerol, 0.5 % human serum albumin and 1X Complete Protease Inhibitor, Roche Diagnostics, Mannheim, Germany).

Pretesting and selection of the PBMC samples PBMC were screened ex vivo with CMV or Flu HLA-multimers at the central laboratory, with an additional test being performed at the co-organizing laboratory. Stainings at the central laboratory were done in two steps following the CIP guidelines (www.cimt.eu/workgroups/CIP), with CD3-FITC or CD4-FITC (OKT3- or HP2/6-FITC, in-house labeling) and CD8-PE-Cy7 (clone SFCI21Thy2D3, Beckman Coulter, Krefeld, Germany) at pretested concentrations. Acquisition was performed on a FACS Canto II (BD Biosciences, Heidelberg, Germany) using Diva software. PMT channels and compensations were adjusted using unstained PBMC and fluorescent beads (BD Biosciences). Analysis was done with FlowJo version 7.2. PBMC from 5 donors (D1–D5) with a total of 7 CMV- and Flu-specific T cell responses showing different levels of reactivity (n = 4 low, i.e., <0.1 %, n = 1 intermediate, and n = 2 high, i.e., >1 % multimer⁺ in the CD8⁺ subset) were selected. One donor was HLA-A*02 negative and HCMV seropositive (D5), one was HLA-A*02 positive and HCMV seronegative (D1) and the remaining three were HLA-A*02 positive and HCMV seropositive (D2, D3, D4).

Inter-laboratory testing

Seventeen European laboratories (ID01 to ID17) participated. All laboratories self-reported as being experienced with at least 3 (up to 10)-color staining, and the majority (n = 16) had performed HLA-multimer stainings before. Every laboratory received two PBMC vials from each donor (D1 to D5) as well as two HLA-A*0201-multimer aliquots (Flu- and CMV-multimers labeled with PE) shipped on dry ice and delivered via courier (n = 14 labs) or collected at the central laboratory (n = 3). Upon receipt, reagents had to be stored in liquid nitrogen (cells) or at −80 °C (HLA-multimers) until use, and experiments had to be performed within 6 weeks. A panel guideline and a report form were distributed.

The main objective of the proficiency panel was to address the influence of data analysis on the variability of HLA-multimer frequency estimates. Two gating strategies were compared; gating strategy 1 was: lymphocyte gate (FSC/SSC), CD3⁺ gate in lymphocytes (CD3/CD8), CD8/multimer in CD3⁺; gating strategy 2 was: low granularity CD3⁺ gate (CD3/SSC), lymphocyte gate (FSC/SSC) in CD3⁺, CD8/multimer in lymphocytes (Suppl. Figure 1). PBMC had to be tested twice, either immediately after thawing (15/17 laboratories counted living cells by trypan blue exclusion, and two used a Guava counter), or after a cell-resting phase; results of immediate testing are shown in the manuscript. In addition, the utility of including a fluorescence minus one (FMO) staining (all mAb, but no multimer) for setting quadrants or gates was assessed. In total, each laboratory was expected to perform 15 ex vivo stainings (FMO, Flu-multimer and CMV-multimer, i.e., 3 tests × 5 donors), each analyzed with the two predefined gating strategies.

Test conditions reagents (except HLA-multimers), staining protocols and flow cytometer setup were not standardized, but some procedures were mandatory following the recommendations of previous CIP proficiency panels [23]. Participants had to (1) use at least 1 × 10⁶ (up to 2 × 10⁶) PBMC per stain and acquire all cells contained in the sampling tubes, (2) include CD3 and CD8 mAb, (3) include a FMO control sample and (4) stain cells with the multimers for 30 min at RT before adding mAb (recommended concentration of multimer was 5 μg/ml). Participants were free to (5) include or exclude a dump channel and/or a dead cell dye, (6) choose the mAb clones and fluorescent dyes and (7) choose the staining protocol and buffers (a suggested protocol was provided).

Data reporting parameters collected for central analysis were the thawing conditions, cell recovery and viability, the number of PBMC distributed per test, and the number of CD3⁺, CD8⁺ and multimer⁺ cells counted. The mAb (specificity, clone, company, fluorochrome), the use of a dump channel or of a dye for excluding dead cells and the cytometer type were also recorded (Suppl. Table 1). Frequencies of antigen-specific cells were expressed as the number and % of multimer⁺ among CD3⁺CD8⁺ cells using analysis strategies 1 or 2.

Central data analysis

Analysis of data generated by the individual participants all dot- or pseudocolor-plots of HLA-multimer stains were analyzed by a group of five experts, who scored the tests with “0” (no CD8⁺multimer⁺ cell population), “1” (plausible CD8⁺multimer⁺ cell population) or “2” (obvious CD8⁺multimer⁺ cell population) for a maximum aggregate score of 10. Stains with an aggregate score of 6 or more were considered positive (i.e., detection of CD8⁺multimer⁺ cells). In addition, the central analysis also evaluated whether the gating strategies conformed to the provided guideline and whether appropriate intermediate steps for the dump or dead cell exclusion were added when applicable. Data sets were excluded from the final analysis if the test results were not accurately reported by the laboratory, or if the frequencies reported were obvious outliers, suggesting technical problems (Suppl. Table 2). One laboratory recorded “false-positive” stains (CMV- and Flu-multimer-binding cells in the HLA-A*02^neg donor D5), presumably arising from artifacts due to technical problems during cell staining or sample acquisition (not shown).

Final results were generated based on the information provided by the participants. Detection rates were defined as: [number of detected responses as per central review/number of expected responses = 7] × 100; antigen-specific T cell frequencies are given in 1 CD8⁺multimer ⁺ T cell in x CD3⁺CD8⁺ lymphocytes. For statistical analysis, mean, median, standard deviations and % coefficient of variation (% CV = SD/mean × 100) were calculated. For group comparisons, a two-tailed paired Student t test was performed using GraphPad Prism 4.02. Relative differences are expressed as Δ = (x − y)/y × 100.

Central analysis of FCS files re-analysis of selected data sets was performed at the central laboratory by one investigator on the same day, using FCS files provided by 14 of the participants (ID04, ID13 and ID24 were excluded) and Flow Jo v7.6.3. Gating strategy 1 was applied as per panel guideline. The number of parameters included was identical to that used by the individual laboratories, i.e., gating steps for dump channel and exclusion of dead cells were included when applicable. Five representative specificities, i.e., D2 Flu and CMV, D3 Flu and CMV, and D4 Flu were examined. The fluorescence compensation was slightly modified for two laboratories. FSC/SSC lymphocyte gates were kept constant within each donor, and CD8⁺multimer⁺ lymphocytes were identified using quadrants or gates set on the CD8 versus multimer dot-plots.

Automated analysis of FCS files

We used a hierarchical statistical Gaussian mixture model that we recently described [35] to analyze FCS files (FMO and multimer tests) obtained from 14/17 participants. File preprocessing included compensation, the application of the same biexponential transformation to all data sets and extraction of data of only the subset of channels common to mAb panels from all laboratories for the statistical model fitting. The hierarchical mixture model was fitted to all data samples from a single laboratory simultaneously, and the same procedure was then repeated for each separate laboratory. A complete description of the preprocessing and processing pipeline is provided in Supplementary Materials (see download link below). More than 98 % of the FCS files (207/210) could be analyzed, demonstrating the feasibility of applying such program for non-standardized data. Three files from one laboratory were excluded because they were recorded on a different scale (max range = 1,024) than the other twelve files (max range = 262,144), violating the assumptions of the hierarchical modeling software.

The automated analysis differed from the manual or central analyses in the following ways: (1) data annotation needed to be preprocessed, (2) the original fluorescence compensation was used for all laboratories, (3) only information from markers common to all laboratories was used for the model fitting, although viability information was used to screen out dead cell events in preprocessing for laboratories that had included a marker for viability, and (4) positivity was determined using an arbitrary event count rather than by expert consensus. The positivity threshold was set at ≥10 events detected, so that the rate of false-positives was very low and close to that obtained by manual analysis (for this data set, 2 and 3 false-positives were recorded for individual and automated analyses, respectively), possibly at the risk of missing very rare events.

Identification of antigen-specific cell clusters was performed using a small set of conservative heuristic rules: (1) the probability density evaluated on the fitted mixture model at cluster center is at least fivefold higher for CMV- or Flu-multimer than for FMO for the same donor, (2) the FSC intensity at cluster center is less than the mean across all samples, (3) the SSC intensity at cluster center is less than the mean across all samples, (4) the CD3 fluorescence intensity at cluster center is greater than the mean across all samples, (5) the CD8 fluorescence intensity at cluster center is greater than the mean across all samples, and (6) the multimer fluorescence intensity at cluster center is greater than the mean + 2 SD across all samples. Rule (1) filters for events that are likely to be antigen specific, rules (2) and (3) exclude clusters that are more likely to be monocytes or macrophages, and rules (4), (5) and (6) filter for clusters that are likely to be comprised of CD3⁺CD8⁺multimer⁺ cells. However, because of a large degree of variability in the separation of multimer⁺ from multimer^neg events across samples from different laboratories, visual inspection was used to select only multimer-positive clusters that were consistently present in multiple samples. In particular, a CD8⁺multimer^low cluster was sometimes observed in CMV- and Flu-multimer stained samples but not in the FMO samples; these were considered by the experts to be non-antigen specific even though they met the criteria listed above. These nonspecific background multimer events were typically assigned into different clusters from truly multimer-positive events, allowing simple exclusion after visual inspection.

An executable IPython Notebook (CIP_Analysis.ipynb) and a read-only version (CIP_Analysis.html) showing the full analysis pipeline from the FCS files to the frequency estimates are provided as Suppl. Materials (https://duke.box.com/s/9kphs2300kumjyowjq06), along with instructions for installing the prerequisite software (README.md file). The FCS files from one participating laboratory are uploaded at the FlowRepository website (www.flowrepository.org, experiment: CIP ID03 panel, ID: FR-FCM-ZZER).

Laboratory environment

The central laboratory operated under exploratory research conditions using established protocols and trained personal. Participants followed their own procedures, and most of them (n = 14) had taken part in previous HLA-multimer proficiency panels organized by CIP or CIC. Three other laboratories had never been engaged in such activities.

Results

General panel characteristics

Centers used local protocols for thawing, counting, resting and staining the cells. The median number of living cells after thawing was 12.5 × 10⁶ per vial with 85 % cell viability, and most laboratories used at least 1 × 10⁶ cells per stain as per panel requirement (mean 1.6 × 10⁶ cells/test, range 0.9–2.7 × 10⁶). In addition to the mandatory markers (CD3, CD8, multimers), some operators included a dump channel mAb and/or a dye for excluding dead cells (Suppl. Tables 1a, b).

In previous panels, we had noticed that inaccurate placement of quadrants or gates on the CD8/multimer dot-plots generally results in erroneous overestimation of the percentage of multimer⁺ cells among the CD8⁺ cell subset [23, 37]. A fluorescence minus one staining (FMO for the PE, i.e., multimer channel) was therefore introduced to guide participants in placing markers or gates. Central review of the CD8/FMO-PE dot-plots showed that all but two laboratories analyzed the same number of cells in the FMO and in the multimer-containing tests; quadrants/gates were adjusted either “low” (close to the CD8⁺multimer^neg population) or “high” (close to the CD8⁺multimer⁺ population) on the PE channel (see examples in Suppl. Figure 2); no serious mistakes were observed, except for one laboratory. Of note, a CD8⁺multimer^low population was often seen in the multimer-containing tests that was absent from the FMO stains, probably due to weak binding of the multimers independently of the antigen specificity. Thus, a FMO test does not constitute an ideal negative control for the multimer assay.

HLA-multimer assay outcome may be influenced by the gating strategy

From the screening experiments, it was expected that a maximum of 7 antigen specificities (4 × Flu and 3 × CMV) could be detected in all five PBMC samples. The majority of the laboratories were able to detect multimer⁺CD8⁺ T cells for the 5 T cell populations present at the highest frequencies (D3 CMV, D4 CMV, D2 Flu, D1 Flu and D4 Flu) using either of the two gating strategies (Fig. 2a and Suppl. Table 2c). The group performance was, however, drastically decreased for the two lowest frequencies, i.e., D3 Flu and D2 CMV (<0.04 %, Fig. 2a). For both specificities, the best detection rates were obtained when applying gating strategy 1. A detailed analysis (Fig. 2b) revealed that many laboratories performed equally well with the two analysis conditions; however, five participants missed between 1 and 2 T cell responses when applying strategy 2 as compared to strategy 1 (e.g., ID08, ID14). This suggests that operator analysis decisions may impact on the test outcome and sensitivity of the multimer assay. Of note, gating strategy 1 led to slightly higher CD8⁺ T cell numbers than gating strategy 2 (mean and median for the group = 106,500 vs. 102,000 and 91,000 vs. 90,500 respectively, Suppl. Figure 3). However, we found no correlation between the mean numbers of CD8⁺ T cells and the individual laboratory performances, suggesting that sufficient numbers of cells were included in most tests to support detection of even low-frequency T cells.

Fig. 2.

Detection rates for Flu- and CMV-specific CD8⁺ T cells following the two gating strategies. a Group analysis for each of the seven antigen specificities to be detected separately. b Single-center and group analysis for all seven antigen specificities together. For details, see Suppl. Table 2

Next, we assessed the frequency of antigen-specific cells (for absolute numbers, see Suppl. Tables 2a–c). As shown in Fig. 3, the inter-laboratory variation (% CV) was limited for antigen-specific CD8⁺ T cells present at high (>1 %, D3 CMV and D4 CMV) or intermediate (0.1 % < D2 Flu < 1 %,) frequencies among PBMC and calculated mean frequencies appear to be similar with the two gating strategies (p > 0.05). Percentage CVs were higher for 3 out of 4 low-frequency T cell specificities (<0.1 %), especially in one case (D3 Flu: % CV = 104 with gating strategy 1), confirming our previous observations [23]. Interestingly, for the lowest T cell frequency that could be compared, i.e., D2 CMV, the variability within the group was smaller after analysis with gating strategy 1 as compared to gating strategy 2 (Fig. 3: % CV 53 to 87, Δ = 64 %).

Fig. 3.

Frequencies of Flu- and CMV-specific CD8⁺ T cells. Scatter plots display laboratory-individual frequencies of CD8⁺multimer⁺ lymphocytes obtained for each of the seven expected antigen/donor combinations with the two gating strategies (positive stainings only). Means (black lines) and % CV are indicated

Overall, superior results were obtained when applying a classical three-step analysis strategy including a standard FSC/SSC scatter plot for identifying the lymphocyte subset. More importantly, we concluded that the gating procedure used for data analysis is a source of inter-center variability in detecting and enumerating HLA-multimer-binding cells. This is especially apparent for rare antigen-specific cells (less than 0.04 % in this panel), which is of high relevance for tumor immunologists since the frequencies of tumor-directed T cells are expected to be low in many settings.

Central gating analysis decreases inter-laboratory variability

To further evaluate the contribution of gating decisions on variability, we performed a central re-analysis of selected FCS files. A single user used gating strategy 1 to analyze five representative tests with low (D2 CMV, D3 Flu, D4 Flu), intermediate (D2 Flu) and high (D3 CMV) frequencies of antigen-specific T cells provided by 14 panel participants. Frequencies of CD8⁺multimer⁺ T cells were determined only for stains defined as positive during the central review process. Figure 4 displays the results of this central analysis (C) as compared to those compiled from the individual participants (I) for the same tests. Strikingly, 3 out of the 5 calculated % CV were substantially decreased (Δ = 72–114 %) after central analysis, and all 3 related to the detection of low-frequency T cells (D2 CMV, D3 Flu and D4 Flu). For the two other antigen/donor combinations, the CV remained essentially unchanged. One obvious reason for discrepancy between the deduced T cell frequencies was the setting of the lower bound of the multimer channel gate to be distant from the CD8⁺multimer^neg subset in the central analysis; some participants chose a position closer to this multimer^neg cell population, thereby including diffuse CD8⁺PE^low events in the multimer⁺ population that most likely reflect nonspecific binding. Hence, the mean frequencies were slightly lower after central analysis.

Fig. 4.

Frequencies of antigen-specific cells determined by individual and central analyses. Scatter plots display the frequencies of CD8⁺multimer⁺ lymphocytes obtained by the individual laboratories (black circles, I) or by the central analysis (gray triangles, C) of FCS files. Means (black lines) and % CV are indicated. Five antigen specificities are included in the analysis (number of positive stainings: 5, 13, 14, 5 and 11 for D2 CMV, D2 Flu, D3 CMV, D3 Flu and D4 Flu, respectively)

These analyses demonstrate that not only protocol parameters, but also individual gating choices contribute to the inter-laboratory variability of the HLA-multimer assay and corroborate previous findings for the ICS assay [26, 30].

Automated analysis of heterogeneous data sets is feasible, allows detection of even rare antigen-specific T cells and leads to results comparable to that of manual gating

We next explored the possibility of identifying multimer-binding cells using a hierarchical statistical mixture model for automated, multidimensional recognition of cell clusters that allows robust, linear and sensitive (<0.02 % of all cells) detection of antigen-specific CD8⁺ T cells spiked at defined numbers in autologous PBMC [35]. Examples of complete automated analyses and deduced CD8⁺multimer⁺ T cell frequencies are given in Suppl. Figure 4a and Suppl. Table 3.

The results obtained after automated processing of the FCS files were consistent with those generated by individual or central manual analyses applying gating strategy 1 (Table 1; Fig. 5a). The detection rates for CD8⁺multimer⁺ T cells were equivalent for 4/7 low- to high-frequency specificities; for the two rare specificities, it was increased in one case and decreased in the other (D2 CMV: 38 vs. 54 % and D3 Flu: 38 vs. 23 % for the central vs. automated assessments, respectively). The mean and median frequencies obtained after automated analysis were in essence lower than for the manually generated data, suggesting a more stringent selection of multimer⁺ T cells by the statistical mixture model. As compared to the laboratory-individual results, the % CV was notably decreased for 1 stain (D2 CMV, Δ = 58 %), essentially unchanged for 4 tests (D2 Flu, D3 Flu, D4 CMV and D4 Flu, Δ < 40 %) and increased for 2 stains (D1 Flu and D3 CMV, Δ > 50 %). For D3 CMV, test of one laboratory (i.e., ID10) contributed greatly to the discrepancy, since HLA-multimer⁺ cells of low intensity were included in the individual (and central) analyses but ignored by the software-guided identification [% CV 29 vs. 158 and 29 vs. 60 with or without ID10 staining, respectively (Table 1 and Suppl. Figure 4b)]. Further dot-plot inspection revealed that what can be interpreted as low-intensity, nonspecific background is more easily “ignored” by the automated analysis than by visual gating. Also, within “diffuse” cell populations disregarded by visual analysis, distinct cell clusters may be captured by automated recognition, as illustrated in Fig. 5b and Suppl. Figure 4b, c. Importantly, this suggests that the automated analysis may be not only more objective, but also more selective than the manual analysis.

Table 1.

Comparison of the three analysis methods

	D1 Flu	D2 CMV	D2 Flu	D3 CMV	D3 Flu	D4 CMV	D4 Flu
Detection rate
I/C	71.4 (10/14)	38.5 (5/13)	100 (13/13)	100 (14/14)	38.5 (5/13)	100 (14/14)	84.6 (11/13)
A	57.1 (8/14)	53.8 (7/13)	92.3 (12/13)	100 (14/14)	23.1 (3/13)	100 (14/14)	92.3 (12/13)
Mean
I	1,627	3,299	676	34	3,208	47	1,699
C	n.d.	3,562	843	36	4,951	n.d.	2,071
A	3,795	7,064	1,201	73	4,170	47	2,852
Median
I	1,616	3,148	610	32	1,286	44	1,586
C	n.d.	3,490	754	34	6,463	n.d.	2,120
A	2,743	7,311	968	40	4,801	43	2,082
% CV
I	36	60	39	29	105	30	60
C	n.d.	28	39	36	61	n.d.	34
A	70	38	54	158 (60)a	77	41	58

Results are given for the seven specificities analyzed for n = 14 data sets by individual (I) or central (C) manual analyses (gating strategy 1), or by the automated approach (A)

Detection rates represent the percentage of participants having detected multimer⁺cells (number of positive stainings/total number of stainings)

Mean and median frequencies of antigen-specific cells are given as 1 multimer⁺cell/x CD8⁺ lymphocytes

n.d. not determined (D1 Flu and D4 CMV were not assessed by central analysis)

^aCV after excluding test from laboratory ID10

Fig. 5.

Automated analysis of heterogeneous FCS files. a Comparison of the frequencies of antigen-specific cells determined by individual and automated analyses. Scatter plots display the frequencies of CD8⁺multimer⁺ lymphocytes for all antigen specificities obtained by n = 14 individual laboratories (black circles, I) or by the automated analysis of the same FCS files (gray diamonds, A). Means (black lines) and % CV are indicated. b Examples of individual and automated analyses. (A) Automated analysis with back-gated multimer⁺ events shown in black. (I) CD8/HLA-multimer dot-plots provided by the participants. The upper staining was scored by the central inspection as being positive, whereas the middle and lower ones were considered to be negative

In summary, these results highlight the considerable potential of computer-based approaches in general and of the hierarchical modeling software in particular for objective, centralized analysis of rare cell events in FCM data generated by single laboratories over time or by multicenter studies.

Discussion

Fluorescent HLA-multimers are excellent tools to detect antigen-specific CD8⁺ T cells independently of their function. The method is considered to be robust and has been shown to be very sensitive in the hands of experts (described limit of detection is below 0.02 % = 1 multimer⁺ among 5,000 CD8⁺ cells) [6, 38]. Considering these favorable characteristics, it was rather unexpected that earlier proficiency panels organized by CIP, CIC and others reported considerable inter-laboratory variability in detecting rare antigen-specific T cells. Some parameters of cell handling that influence assay performance were identified, including the number of cells per test and whether a dump channel was used [23, 28, 37, 39]. Furthermore, gating strategies and marker settings varied, including cases of incorrect cell subset gating that may contribute to the variability of the results [23]. Based on these findings, our study focused on the impact of individual gating decisions on assay outcome. Furthermore, we evaluated the feasibility of an alternative, automated method for analyzing data generated by a group of non-standardized laboratories.

The inter-laboratory variability (% CV) in this panel was decreased as compared to our previous experience; we believe that this is because most of the centers had taken part in proficiency panels before, underlining the value of inter-laboratory harmonization. Despite an excellent overall performance, differences (detection rates and frequencies of CD8⁺ HLA-multimer⁺ T cells) were clearly discernible between participants, especially for very low (<0.04 %) frequencies of antigen-specific T cells.

Manual gating relies on the skill and experience of individual users, with few basic rules (such as gating sequences) that may vary between laboratories and often between operators. Analysis of FCS files by a single operator (central analysis) reduced the inter-center variability, indicating that gating indeed plays an essential role in the HLA-multimer staining outcome. These results reinforce previous observations made for ICS, where it was shown that single-operator analysis or harmonization of the gating strategy among laboratories analyzing the same set of FCS files reduced variability considerably [26, 29–31]. However, central analysis remains highly dependent on operator judgment, is time-consuming especially for multiparametric data sets and may simply not be feasible. Automated methods in which cell subsets are directly quantified by machine algorithms are therefore attractive tools to overcome the problem of user subjectivity. A number of algorithms for cell subset identification have been recently proposed [32, 34]. With such approaches, analysis of cell populations is in principle independent of any sequential gating strategy, since clusters are identified using the full set of data dimensions simultaneously.

We have developed a hierarchical statistical mixture that allows detection of low-frequency antigen-specific T cells in spiked samples with a sensitivity and reproducibility comparable with that of cytometry experts [35]. Here, we show that this program is also suited for analyzing FCS files generated at different centers using unmanipulated PBMC samples and centrally prepared HLA-peptide multimers, but different protocols, staining reagents and cytometer settings. Statistical mixture modeling may therefore be used not only for individual purposes, but also for analyzing data generated by proficiency panels or multicentric studies. A similar, non-hierarchical Gaussian mixture model was also recently shown to identify rare antigen-specific cell subsets with high sensitivity [40].

The detection rates obtained with the modeling were overall in the range of that of the manual analysis; however, the automated analysis can occasionally result in higher variability as inclusion/exclusion applies to entire clusters rather than individual events potentially amplifying differences between samples. Also, mean frequencies were mainly decreased after automated analysis. Unlike manual gating where events must be discriminated using only two markers at a time, the multivariate clustering method simultaneously uses information from multiple markers in order to determine cluster membership: This facilitates the separation of cell subsets that can only be distinguished on the basis of multiple markers simultaneously. At the same time, events with similar characteristics in all dimensions will be clustered together and not excluded by “hard” gating boundaries because they are relative outliers in a two-dimensional dot-plot. In line with this, we observed that low-intensity multimer⁺ events that are often interpreted as background by immunologists are classified into different clusters than the distinct high-intensity events, allowing a clean separation either using a rule-based method or by visual inspection (Suppl. Figure 4c). Conversely, in some cases, the automated analysis revealed CD3⁺CD8⁺multimer⁺ events that would be excluded by a manual scatter gating (FSC/SSC) as nonspecific binding of the reagents to debris or dying/dead cells (Fig. 5b, Suppl. Figure 4c). Notably, these events only appeared for samples that also contain multimer-binding CD8⁺ cells within the standard FSC/SSC lymphocyte gate. It is known that binding of HLA multimers to cognate TCR may induce activation-driven cell death, a process which is temperature and likely protocol dependent [41, 42]. If so, these events correspond to “true-positive” antigen-specific cells and should not be excluded from the analysis.

Our data support current efforts to improve the availability of robust and sensitive automated tools for flow cytometry analysis and to make these accessible to the broader community. The potential payoff is clear: Currently, data analysis takes up a large fraction of time for staff members in immune monitoring laboratories; automated tools have the potential to dramatically lower the need for manual analysis of standard Ab panels, freeing up time for other activities. However, automated analysis still faces significant challenges before it can be widely adopted for routine analysis of multicenter data.

First, in the absence of consistent and sufficiently complete annotation, there is a large preprocessing burden before the data can be fed to automated analysis pipelines. In this panel, laboratories were allowed to use their own annotation, leading to an extensive variation in the naming of FCS files and channels. In some cases, the FCS metadata did not provide information on the channel marker, and cross-referencing with data provided by the laboratories was necessary. Such problems can be overcome if laboratories develop consistent data annotation protocols that minimize human error (e.g., use of barcoding), and bioinformaticians develop tools to harmonize laboratory annotations, for example via the intermediary of appropriate ontologies.

Second, it is challenging to map event clusters to their biologically meaningful nomenclature for non-standardized data from multiple laboratories. We describe a heuristic set of filter criteria based on statistical comparison across data sets and controls to identify candidate antigen-specific clusters, but final visual inspection was still necessary to disambiguate clusters that were CD8⁺ and multimer^{+ (low)} into nonspecific or antigen-positive subsets. We suggest that a pragmatic solution might be to move away from a fully automated solution, and instead use the automated analysis pipeline to detect candidate clusters for targeted cell subsets using heuristic rules, and then use visual inspection to accept or reject the candidates based on subtle properties that are currently not captured by our algorithms (e.g., shape, compactness, consistency across different data sets, known biological context of the sample).

Third, the statistical mixture model software must be operated by a computer expert both for initial parameter settings and for the following analysis process; a browser-based version that is more accessible to average flow analysts, both in the context of multicenter studies and for center-specific projects, is currently being evaluated by CIP core laboratories [44].

Finally, some recommendations for data analysis can be added to our HLA-multimer guideline (www.CIMT/workgroup/CIP): (1) run systematic comparisons of gating strategies when establishing or optimizing new mAb panels [39, 43], (2) carefully set and audit quadrant/gate placement for selecting CD8⁺multimer⁺ cells. A FMO staining—performed on the same number of cells as the test stains—may help inexperienced users but cannot be utilized for setting thresholds accurately because it provides no information on nonspecific multimer binding. As an alternative, irrelevant HLA-multimers have been introduced as controls in subsequent CIP proficiency panels, in line with recently published recommendations [1, 28].

In summary, the results of this proficiency panel extend our knowledge on parameters that impact performance of the HLA-multimer assay and underline the importance of data analysis strategies. Automated methods are attractive tools that bear the promise to reduce inter-laboratory variability of flow cytometry and increase robustness, accuracy and comparability of results, but much needs to be done before fully automated methods can be used for routine analysis of non-standardized multicenter data.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Abbreviations

CIC

Cancer Immunotherapy Consortium

CIP

CIMT Immunoguiding Program

CIMT

Association for Cancer Immunotherapy

HCMV

Human cytomegalovirus

CV

Coefficient of variation

FCM

Flow cytometry

FCS file

Flow Cytometry Standard format for data files

FMO

Fluorescence minus one

HLA

Human leukocyte antigen

ICS

Intracellular cytokine staining

IFNγ

Interferon gamma

mAb

Monoclonal antibody

PBMC

Peripheral blood mononuclear cells

RT

Room temperature

TCR

T cell receptor