A comparison of clustering models for inference of T cell receptor antigen specificity

Abstract

The vast potential sequence diversity of TCRs and their ligands has presented an historic barrier to computational prediction of TCR epitope specificity, a holy grail of quantitative immunology. One common approach is to cluster sequences together, on the assumption that similar receptors bind similar epitopes. Here, we provide the first independent evaluation of widely used clustering algorithms for TCR specificity inference, observing some variability in predictive performance between models, and marked differences in scalability. Despite these differences, we find that different algorithms produce clusters with high degrees of similarity for receptors recognising the same epitope. Our analysis strengthens the case for use of clustering models to identify signals of common specificity from large repertoires, whilst highlighting scope for improvement of complex models over simple comparators.

Graphical abstract

1. Introduction

T lymphocytes recognise peptide epitopes presented at the cell surface by Major Histocompatibility Complexes (MHC) in jawed vertebrates [1]. Recognition is mediated by diverse heterodimeric $\alpha$ or $\beta$ TCR domains positioned on the $T$ cell surface. The chains of the more common $\alpha$ $\beta$ TCR contain variable (V), joining (J) gene segments, constant (C) regions, and an additional diversity (D) segment in the $\beta$ polypeptide. Each $T$ cell expresses many copies of a single TCR, which bind to peptide-MHC (pMHC) via the complementarity determining regions (CDR) 1-3 of the TCR [2]. Productive TCR engagement triggers a context-dependent signalling cascade, which in turn promotes activation and differentiation of diverse immune effector cells [3].

The central role of the TCR in immune surveillance and response to disease has encouraged efforts to decode the rules of TCR-pMHC binding. Determination of specificity, or “receptor de-orphanisation”, can be achieved experimentally using sequencing and repertoire analysis, or with functional, multimer binding, or TCR screening methods, reviewed in [4], [5]. Indeed, the advent of high-throughput single cell methods has vastly increased the volume and functional annotation of $T$ cell repertoires. However, our understanding of epitope specificity remains limited to a small fraction of the universe of TCR-epitope pairs, and the ability to accurately predict the cognate epitope of any TCR in silico could vastly accelerate our understanding of fundamental and translational $T$ cell biology [6].

The availability of large repositories of TCR sequences and their known ligands has enabled the development of two major families of computational model for prediction of TCR antigen specificity: Supervised Predictive Models (SPMs) and Unsupervised Clustering Models (UCMs) (Fig. 1) [6]. These families are representative of two distinct approaches to machine learning. In supervised learning, predictive models are trained on a set of input instances having a known label (in this case, the cognate epitope for a given TCR). In unsupervised learning, models learn the underlying statistical features or patterns of a dataset to differentiate between input TCRs, applying techniques such as clustering or dimensionality reduction.

Fig. 1.

Supervised and unsupervised learning in $T$ cell epitope specificity inference. (A) SPMs (left) fit a predictive function f(x) to training data having an independent variable X (TCR sequences and other features) and dependent variable y (epitopes or pMHC complexes). This function may then be applied to predict the cognate epitopes of orphan TCRs. UCMs (right) generate a mapping from TCR sequences to a cluster allocation, such that each TCR is assigned to one or more clusters having common epitope specificity. (B) Application of UCMs to de-orphanise TCRs by co-clustering.

The use of deep neural networks (DNNs) including large language models and convolutional neural networks has contributed significantly to recent improvements in UCM and SPM performance [7], [8], [9], [10], [11]. Despite these advances, no publicly available SPM is yet capable of accurately predicting the specificity of TCRs recognising “unseen” epitopes that were not encountered during model training [8], [12]. This is likely due at least in part to the limited volume of experimentally determined receptor-epitope pairs, which constitutes just a small fraction of the vast theoretical diversity of TCRs [6], [13].

Unlike SPMs, UCMs do not require receptor–ligand pairs as an input, but have been designed to group together TCRs on the basis of features thought to underly common antigen specificity [14], [15]. Such models are of particular use in an era when bulk and single-cell sequencing experiments can yield thousands of unique orphan TCRs per sample, applying them to group receptors sharing similar sequence, structure, or physico-chemical properties and so to shortlist TCRs of interest for later experimental de-orphanisation. Such approaches have been successfully applied to identify and characterise TCRs associated with mycobacterial and viral infection, cancer, and autoimmune disease [15], [16], [17], [18], [19], [20].

UCMs take as their input single or paired TCR CDR3 nucleotide or amino acid sequences, with or without V and J gene usage information, and return a mapping of sequences to unique clusters. This has historically been achieved using some form of distance measure, typically either direct sequence similarity and/ or the frequency enrichment of short sequence snippets (kmers) compared to a reference dataset. Recent approaches leverage DNNs to generate a compressed numeric representation of the input TCR as a precursor to clustering [11], [21], [22], [23].

Whilst this is not the purpose for which they were originally conceived, UCMs may be used to infer TCR epitope specificity from datasets having full or partial epitope labels, by assigning the most frequent epitope of a cluster as the predicted binder for all TCRs in that cluster (Fig. 1B). However, there has to date been no independent benchmarking study of UCMs as predictors of TCR specificity, despite their widespread use in the field. In the present work, we provide the first confirmation of the direct predictive capacity of UCMs by applying the models to sets of known TCR-epitope pairs, comparing performance using a balanced measure of precision and recall. We evaluate five commonly used UCMs, investigating the impact on performance of the number of TCR representatives provided per epitope, the receptor chain, the epitope studied, and the input dataset. We find that all five models outperform CDR3 length and random baselines, but that relative predictive performance is sensitive to each of these four factors. The five models studied are not consistently superior to of simple Hamming sequence distance and V-gene clustering models, highlighting significant room for improvement. Extending the analysis to qualitative comparison of the clusters formed, we observe that the five models also produce similar cluster motifs for common viral epitopes. Taken together, we conclude that whilst UCMs are of considerable value in de-orphanisation pipelines, further work is needed, and that in practice clustering algorithms, experimental methods, and predictive models should be combined to maximise the likelihood of identifying true epitope-specific clusters.

2. Methods

2.1. Datasets

A consolidated dataset of paired $\alpha \beta$ TCR amino acid sequences of human origin was developed using instances drawn from VDJdb [24] for initial model comparison, and TCR-epitope pairs from McPas-TCR [25] and from the MIRA dataset [26] were prepared as separate non-overlapping test sets (Table S1). Sequences derived from a 10X study of healthy donors, and CDR3 sequences containing non amino acid symbols, were removed from the input data. V and J gene codes were processed for consistency with IMGT reference sequences [27]. Duplicates were removed within and between datasets using CDR3-V-J bio-identities for both $\alpha$ and $\beta$ chains, such that a given TCR was encoded in the format CDR3$\alpha$-TRAV-TRAJ-CDR3$\beta$-TRBV-TRBJ. Only those TCRs having V or J genes included in the reference IMGT alleles of the tcrdist module of the CoNGA conda package (v.0.1.1) [28] were retained, to ensure that consistent numbers of sequences were provided to each model. Benchmarking experiments were performed on VDJdb data after chain selection and down-sampling as described in Results. Neither McPas-TCR nor MIRA datasets were downsampled, to give a representative view of model performance on imbalanced datasets more representative of real world datasets.

2.2. Models

A systematic review of the literature was conducted to identify studies presenting novel methods for prediction of antigen specificity from TCR sequences (Section S1). ClusTCR [29], GIANA [30], GLIPH2 [17], iSMART [20], and tcrdist3 [31] were shortlisted for analysis based on the availability of open-source python packages or executable files. ALICE [32] was excluded as more appropriately applied to the identification of expanded clones in individual patient repertoire data. To the five test models we added Hamming distance, V-gene, length and random baseline comparators. Implementation and background methodological detail is included for each of the selected algorithms in Section S2. All benchmarking experiments were run on a single remote Intel(R) Xeon(R) Gold 6126 CPU at 2.60 GHz to ensure fair comparison of algorithms with and without parallel processing capability.

2.3. Metrics

Performance of each model was initially analysed using a broad panel of UCM-specific, entropy, and predictive measures. Clustering metrics included purity, consistency and retention, as described previously in [29], [33], where purity is the proportion of TCRs in a given cluster having the modal epitope; consistency is the proportion of epitope-specific TCRs that are assigned to a single cluster; and retention is the proportion of TCRs successfully assigned to a cluster. Entropy was calculated using a scikit learn implementation of adjusted mutual information (AMI) that accounts for class imbalance in computation of the mutual information of true and predicted labels [34]. To these were added the predictive metrics accuracy, precision, recall, and f1-score, as described in Eqs. (1a)–(1d). Each of these measures was weighted to adjust for per-epitope class imbalance, and computed for all input instances to account for differences in retention between models.

$$\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}$$ (1a)

$$\text{Precision}=\frac{TP}{TP+FP}$$ (1b)

$$\text{Recall}=\frac{TP}{TP+FN}$$ (1c)

$$\text{F1-Score}=\frac{2\ast \text{Precision}\ast \text{Recall}}{\text{Precision}+\text{Recall}}$$ (1d)

Eqs. (1a)–(1d): Predictive performance metrics, where TP $=$ True Positive, TN $=$ True Negative, FP $=$ False Positive, FN $=$ False Negative.

2.4. Synthetic TCRs

Synthetic TCR sequences were produced using OLGA [35] (see Study Limitations), which uses dynamic programming to generate TCR sequences (CDR3, V and J gene codes) with probabilities inferred from human T cell repertoires.

2.5. Statistics

Statistical significance was determined using a two-tailed Mann–Whitney test (consistent with [36]). Statistical analyses were performed with GraphPad Prism [37]. Plots were produced with GraphPad, Matplotlib, and Seaborn Python libraries [37], [38], [39].

2.6. CDR3 amino acid motifs

Sequence logos were produced from $\beta$ chain selections of dataset V1000 by retaining TCRs having the modal length from the largest cluster for each of the four epitopes present. Sequences were aligned with MUSCLEv5.1 [40], and logos were produced from the resulting multiple sequence alignments with WebLogo v3.7.12 [41].

3. Results

Five commonly used open-source UCMs were identified from the literature for which a python implementation was readily available: ClusTCR, GIANA, GLIPH2, iSMART, and tcrdist3 [17], [20], [29], [30], [31]. Four baseline models were added: A Hamming distance model that grouped together sequences having identical length and differing by not more than one amino acid, CDR3 length, V-gene clustering, and random baselines. A summary of the respective model methodologies is provided in Section S2.

Benchmarking analyses were performed on paired $\alpha \beta$ TCRs data drawn from VDJdb, a large, public, curated source of TCRs of known epitope specificity and which has been used as a reference for all of the five models selected [24]. However, past studies have presented comparisons using a single pre-processing strategy, for example by retaining only a subset of TCRs binding selected epitopes [17] or meeting a specific quality threshold [29]. To explore whether selective sub-setting of data might mask inherent differences between models, performance was analysed for data subsets generated by retaining epitopes having 10, 50, 100, 500, or 1000 cognate TCRs (datasets V10, V50, V100, V500, and V1000 respectively, Table S1 and Table S2). Instances were randomly down-sampled after pre-processing, such that each experimental run was performed on the same number of TCR sequences per epitope. All models were applied to $\alpha$ or $\beta$ chain selections from the same set of paired TCRs. Sampling was repeated 25 times to account for variance between random TCR selections. We present performance on $\alpha$ or $\beta$ chain selections independently (see Discussion and Study Limitations).

UCMs have historically been evaluated using cluster quality metrics such as purity, consistency, diversity, and retention (see Methods). By applying each model to sets of TCRs of known specificity, we were able to supplement these metrics with direct measures of predictive capacity following the schema depicted in Fig. 1B. Observing a positive correlation between performance according to cluster purity, consistency, adjusted mutual information, precision, recall, and F1-score (Fig. S2), we present results for F1-score alone, weighting scores to account for epitope-specific class imbalance and computing performance across all input TCRs to allow for differences in retention between models.

Differential model performance was first inspected at a global level for datasets V10 through V1000 combined, grouping over $\alpha$ and $\beta$ chain selections and over all epitopes (Fig. 2A and Table S3). All study models outperformed length, V-gene, and random baselines (p$<$0.0001). GIANA and iSMART performed equally well (p=0.67), surpassing both a Hamming distance and the other models studied (p$<$0.0006). V-gene usage (vcluster) was better predictive of epitope specificity than CDR3 length, both of which outperformed a random baseline. However, significant variation in relative performance was observed for each model, sensitive to the choice of chain selection (Fig. 2B and Table S4), the number of representatives per epitope (Fig. S3A and Table S5), the epitope (Fig. S3B and Table S6), and the data source (Fig. S3C and Table S7), with no one model achieving superiority across all factors. The predictive capacity of all models was improved when provided with more TCR instances per epitope (Table S5), whereas performance was lower for $\alpha$ than $\beta$ chain selections for all except V-gene, length and random baselines (Table. S4). Notably, clustering with V-gene identity alone achieved superior performance to the study models for AVFDRKSDAK and KLGGALQAK epitopes, and for McPas-TCR (Fig. S3) where performance was similar to a Hamming distance model.

Fig. 2.

Comparison of model performance across datasets. Weighted F1-scores shown for V10-V1000 grouped, (A) $\alpha$ and $\beta$ chain selections combined; and (B) split by chain selection. Significance values: ****, p $\le$ 0.0001; ***, p $\le$ 0.001; **, p $\le$ 0.01; *, p $\le$ 0.05, n.s. $=$ not significant.

Whilst the size and epitope distribution of clusters varied between models (Fig. S4), CDR3$\beta$ motifs for the largest clusters associated with each of these epitopes were strikingly similar for all except length, V-gene and random baselines, and were near-identical when applied to GILGFVFTL or RAKFKQLL (Fig. 3). Motifs were more varied for the other two epitopes in the V1000 dataset (KLGGALQAK and AVFDRKSDAK, Fig. S5).

Fig. 3.

CDR3$\beta$ sequence logos for the largest clusters produced per epitope, dataset V1000. Logos were produced with WebLogo [41] for TCRs in the largest cluster produced for a given epitope per model following sequence alignment with MUSCLE [40].

Finally, we compared UCM computational speeds by adding synthetic TCRs, produced with OLGA [35], to V1000 (Fig. 4). ClusTCR and GIANA were the most scalable of the five models tested, completing their cluster assignments within 10s for 100,000 TCRs, outpaced only by the baseline models. Time limitations prevented extension of the present analysis to comparative performance over parallel CPUs, and to the GPU-enabled versions of ClusTCR and GIANA. A batch-clustering implementation of tcrdist3 was used in order to avoid CPU overloading during distance calculation, which ran slowly compared to the other test models. Model predictions were negatively impacted by the addition of 100,000 synthetic TCR sequences, with F1-scores dropping to near zero for all but tcrdist3 (Fig. S6).

Fig. 4.

Investigating model scalability, comparing model runtimes as a function of the number of synthetic TCR sequences introduced with OLGA [35]. All experiments conducted on dataset V1000 ($\beta$ chain selection).

4. Discussion

Despite the exponential growth of orphan TCR datasets, and the widespread use of UCMs in de-orphanisation pipelines, an independent comparison of the predictive capacity of clustering models has thus far been missing from the field. Here, we present an initial, modest attempt to address this need.

Our work suggests that clustering models can indeed be used to infer epitope specificity, by co-clustering TCR sequences having known and unknown cognate epitopes, and assigning the most common epitope as the predicted binder for a given TCR cluster. Whilst this analytical strategy has been deployed elsewhere [29], we report performance using metrics that can be applied equally to clustering and predictive models, over a wide variety of datasets and preprocessing strategies, which therefore give a more balanced view of true model performance. The comparative analytical framework and model datasets are made freely available at https://github.com/hudsondan/tcr-scapes.

We find that model ranking is sensitive to the number of TCRs provided per epitope, to the epitope studied, and to the dataset tested. This is most easily explained by the different model methodologies deployed (reviewed in Section S2). Whilst GIANA and iSMART perform well on VDJdb instances (Fig. 2), GLIPH2 provides superior predictions when applied to the large MIRA SARS-CoV-2 dataset, whilst Hamming distance and V-gene clustering baselines outperform the study models for McPas-TCR data (Fig. S3). The improved ability of V-gene identity clustering to group together TCRs recognising AVFDRKSDAK and KLGGALQAK compared with the study models (Fig. S3) may be reflective of the contribution of germline CDRs to recognition, although absolute performance was poor (under 10%). Only tcrdist3 was able to retain predictive performance in the presence of many synthetic TCRs, albeit not for all epitopes (Fig. S6). This is of particular relevance when applied to the task of extracting expanded antigen-specific clones from a mixed repertoire of effector and antigen-naive $T$ cells. Adding to the complexity, different models produced similar CDR3 motifs for TCRs recognising common viral epitopes, suggesting that one might reach a similar immunological conclusion irrespective of the model used.

With this in mind, should experimentalists deploy UCMs in the complex task of TCR de-orphanisation, and if so how? We observe that the five UCMs studied are able to group TCRs into antigen-specific clusters, even when provided with as few as ten representatives per epitope (Fig. S3). Whether these common specificity group are driven by recognition of unknown (unseen) or known (seen) antigens, will depend on the condition under investigation, the dataset in question, and the experimental methodology. We would nonetheless encourage their continued use, in combination with state of the art predictive models and experimental methods, as a means of improving confidence in putative assignment of epitope specificity. In the absence of a consistently superior model, one strategy could be to deploy two or more of the models studied to improve confidence in TCR cluster assignment in a manner robust to epitope-specific performance variations, for example combining a Hamming distance model with GIANA (a scalable follow-on to iSMART), ClusTCR (which performed equivalently to GLIPH2 for many datasets) or tcrdist3 (which appears to be more robust to noise). Choice of model will also depend on practical considerations including ease of use, and we note for example that at present on GLIPH2 is available as both a web tool and a command line executable file.

Finally, although recent structural [42], statistical [43], and predictive [9] analyses suggest that both polypeptide chains play an important role in epitope recognition, we observed consistently lower F1-scores for $\alpha$ compared to $\beta$ chain selections (Table S4). One possible explanation is that the default model hyperparameters have been optimised for $\beta$ chain data, which make up the majority of published TCR-epitope pairs [6]. Alternatively, the $\beta$ chain may contribute more to determination of overall epitope specificity, as a product of its increased diversity relative to $\alpha$ chains [13], as we will discuss further in Study Limitations. Modelling strategies that permit integration of $\alpha$ and $\beta$ chain pairing with transcriptomic and phenotypic information, including graph network approaches such as CoNGA [28], may help efforts to decode the relative contribution of chain pairing to epitope specificity, which remains an open question in the field.

5. Study limitations

The significant scientific and economic potential of a generalisable solution to prediction of TCR epitope specificity has encouraged the development of a multitude of new SPMs and UCMs, summarised in [6]. However, the scope of the present study is limited to a handful of commonly used open-source models, on the basis of their widespread use and relative freedom from training data bias as compared to DNN-UCMs and SPMs. Nonetheless, an independent comparison of UCMs, DNN-UCMs and SPMs would be of great use to the community. We also limit our analysis to performance in an inference task, overlooking a host of useful features such as motif extraction, computation of probability of generation, and visualisation tools, which will likely influence user choice for a given immunological question.

There is also growing evidence that inclusion of both $\alpha$ and $\beta$ chains improves predictive performance in SPMs and DNN-SPMs [9]. However, whilst ClusTCR, tcrdist3, and GLIPH2 may all theoretically be applied to paired chain data, cluster assignments are produced for a given CDR3 independent of the other chain for all but tcrdist3. An investigation of whether the integration of $\alpha$ and $\beta$ chain information improves performance equally across models might reveal the relative merits of each, when applied to large orphan repertoires.

Finally, comparison of the generation probabilities of TCRs drawn from true repertoires and those produced with OLGA have concluded that the latter produces TCRs that are too distant from native repertoires to represent a true background [18]. It may be that the unique inclusion of a CDR2.5 loop in tcrdist3 distance calculations, or of a more permissive edit distance than other models, allows the model to differentiate between synthetic and input TCRs and so to cluster TCRs effectively with or without noise. Use of a mixed background [18] or of true orphan repertoires such as [44], might confirm this and so test the robustness of the models to other sources of noise.

CRediT authorship contribution statement

Dan Hudson: Software, Methodology, Formal analysis, Data curation. Alex Lubbock: Software. Mark Basham: Supervision. Hashem Koohy: Writing – review & editing, Writing – original draft, Supervision, Investigation, Funding acquisition, Conceptualization.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: D.H. provides consultancy services to companies active in T cell antigen discovery and vaccine development. The other authors declare no competing interests.

Appendix A. Supplementary data

The following is the Supplementary material related to this article.

MMC S1

Supplementary Material includes a supportive literature review, model methodology, supplementary figures and tables.

Data availability

The codes and data are available in our public github that will be publicly available upon publication.