Leveraging Deep Learning to Improve Vaccine Design

Abstract

Deep learning has led to incredible breakthroughs in areas of research, from self-driving vehicles to solutions, to formal mathematical proofs. In the biomedical sciences, however, the revolutionary results seen in other fields are only now beginning to be realized. Given its public health significance, vaccine research and development efforts, including protein structure prediction, immune repertoire analysis, and phylogenetics are three principal areas in which deep learning is poised to provide key advances. Here, we opine on some of the current challenges with deep learning and how they are being addressed. Despite the nascent stage of deep learning applications in immunological studies, there is ample opportunity to utilize this new technology to address the most challenging and burdensome infectious diseases confronting global populations.

A New Era in Deep Learning

Deep learning has garnered widespread attention in the scientific community and beyond, with groundbreaking results in various domains. The ability of deep learning programs to defeat world champions at games, drive vehicles with human-level performance, and discover novel mathematical proofs[1–4] has spurred intense desire to translate similar results in the biomedical sciences, including immunology and vaccinology. Recent deep learning applications have shown encouraging results in biology for predictive and descriptive tasks[5, 6]. For example, models have been developed to detect cancer from histology images or from genetic information at a higher accuracy than the standard of care[7–9]. To date, the greatest breakthrough of a deep learning model in biology is arguably AlphaFold’s solution to the “protein folding problem”, considered one of the most fundamental and longstanding challenges in biology[10, 11]. As deep learning slowly starts to find its place in biology, it raises the question of what type of impact this area of research may have in contributing to the development of efficacious vaccines (Figure 1), particularly those against viral pathogens, which burden society in the form of continuously evolving circulating strains, and new zoonoses that jump from animal hosts into humans. While antiviral therapies can contribute to disease treatments (e.g. HIV-1), the most effective countermeasure for preventing infectious diseases is the development of highly effective vaccines. To this end, while smallpox and polio eradication campaigns serve as examples of what is possible, developing safe and effective viral vaccines is a difficult and complex process with many more failures than successes. The new era of deep learning and big data suggests that similar potential breakthroughs might be realized in vaccine development for the benefit of public health. To this end, we foresee protein structure prediction, immune repertoire analysis, and phylogenetics as three complementary areas in which deep learning methods will contribute to efforts to advance vaccine research and development.

Figure 1: Deep learning areas of focus in vaccine design.

Prediction of protein structures, analysis of antibody and T cell receptor repertoires, and viral phylogenetics are three areas in which deep learning is supporting rapid advances. Deep learning has made the greatest progress so far in structure prediction “solving” the protein folding problem and is now commonly being used to generate antibodies bypassing experimentation steps. Immune repertoire data growth has coincided with deep learning development allowing for prediction of the specificity or disease outcomes of immune responses from sequencing data alone. Phylogenetic analysis of global viral variants can leverage deep learning to better understand mutational patterns and the effect mutations may have on subsequent immune responses as well as pathogen fitness and population susceptibility. This figure was created using BioRender (https://biorender.com/)

Surface level introduction to deep learning

Deep learning is a subset of machine learning that utilizes more complex learning algorithms on large datasets. Although deep learning and classical machine learning (ML) techniques are similar, there are a few key distinctions (Figure 2). At a high level, classical ML and deep learning are computational modeling approaches to take a training data set, learn trends about how data input features relate to outcomes of interest, and create rules to make a prediction (Figure 3). These approaches are validated using a test data set, leading to the determination of the prediction accuracy on “unseen” examples. The overall task of learning how to make correct predictions is the same for both modeling approaches; however, the methods the models use to accomplish this task are different. Deep learning models are modeled after neural networks in which information flows between nodes connected in layers. While the relationships between individual features and prediction outcomes become more abstract at each layer, prediction accuracy can be dramatically better than observed from classical ML approaches. As a result, although deep learning may pose challenges to supporting mechanistic biological insights, these models may more accurately represent true biological complexity, and their improved performance has the potential to lead to new insights in vaccine and therapeutic antibody[12] development, ideally helping to address the most challenging infectious diseases.

Figure 2. Comparison of common machine learning and deep learning models.

A. Examples of common classical machine learning algorithms. Algorithms are a mix of supervised approaches, such as linear regression, logistic regression, random forest, and support vector machines, in which the models are trained and tested on labelled data, and unsupervised algorithms, such as principal component analysis and K-means, in which the algorithm uses unlabeled data. B. Examples of common deep learning model architectures and associated tasks. Deep learning architectures pass information among nodes within layers to create more abstract data representations that can result in more accurate model predictions. Deep learning models generally have greater performance than machine learning algorithms however are generally more complex to create and are computationally more expensive. This figure was created using BioRender (https://biorender.com/)

Figure 3. Deep Learning Model Workflow.

Deep learning models are made using the training data set. Model parameters are refined and tuned until the error is minimized when making predictions in the training set. The model is then tested by making predictions on the test data set, which the model has not seen previously. Standard metrics for classification model evaluation include generation of a confusion matrix which breaks down where the misclassifications happened and a receiver operating characteristic (ROC) curve providing information on how model performance compares to random. This figure was created using BioRender (https://biorender.com/)

Protein structure prediction and immunogen design

In theory, the total surface area of the entire proteome of a pathogen can elicit an adaptive immune response, but in practice, different parts and conformational states of these target antigen surfaces offer differing degrees of protection, as was recently reported for SARS-CoV-2 (responsible for the current COVID-19 pandemic)[13] and has long been appreciated for other viruses, such as HIV-1, and has been referred to as the “neutralizing antibody problem”[14]. Whereas reverse vaccinology sought to employ bioinformatics for antigen selection and has driven great inroads against bacterial pathogens[15], next generation vaccinology approaches for viruses clearly heavily rely on insights from structural biology (Table 1). Knowledge of the three-dimensional structure of relevant immunogens can aid in vaccine development by providing a physical representation above the raw amino acid sequence that can guide studies into directing responses towards certain epitopes or pre-fusion conformational states, and away from others. While for decades the only method of accurately obtaining the structure of a protein was experimentally, deep learning methods have recently predicted structures from amino acid sequences with accuracy equivalent to experimental methods[10, 11, 16–18]. These deep learning models have been evaluated using data from the Critical Assessment of Structure Prediction (CASP)[19], in which models are evaluated by predicting structure from sequence on solved structures that have not been released publicly providing an accuracy benchmark. These new models are able to achieve accuracies on the CASP test structures that are within the same level of variation from true structure as x-ray crystallography and cryo-electron microscopy. Given the importance of structure-based vaccine design against the metastable fusion proteins of viruses, the ability to accurately and rapidly predict structure from sequence alone has the potential to usher in a new era of vaccine discovery.

Table 1:

Structure-based vaccinology for human viruses

Virus	Structural Modification	Outcome	Ref
RSV	Identification of pre-fusion F protein structure	Allows vaccination with pre-fusion F-containing neutralizing epitopes	[105]
	Pre-fusion F stabilization	Highly immunogenic responses in vaccines	[106–108]
	Epitope focused vaccine design	Proof-of-concept study developing an RSV vaccine for neutralizing epitopes of interest	[109]
SARS-CoV-2	SARS-CoV-2 spike stabilization	Highly immunogenic vaccines with the S2P and HexaPro stabilizations	[26]
SARS-CoV	Identification of SARS-CoV prefusion spike structure	Revealed new epitopes for vaccine design	[34, 110, 111]
MERS-CoV	Identification of MERS-CoV prefusion spike structure	Highly immunogenic epitopes for vaccine development	[112, 113]
HIV-1	Stabilization of HIV-1 envelope protein	Generation of BG-SOSIP trimer immunogens capable of eliciting neutralizing antibodies	[27, 114–116]
	Structure of pre-fusion envelope	Atomic resolution of pre-fusion spike immunogens	[117, 118]
	Engineered HIV-1 immunogens	Structural design of germline targeting immunogens	[53, 54, 119, 120]
	Structural guided nanoparticle design	Structure based nanoparticle formulations are highly immunogenic for multiple viruses	[38, 121–123]

The clearest example of the value that structural information can have for vaccines arguably comes from Respiratory Syncytial Virus (RSV)[20, 21]. Specifically, whereas the native F protein of the virus is most frequently presented in its post-fusion conformation, antibody responses with superior neutralizing capacity are associated with recognition of the pre-fusion conformation[22]. These advances in structural knowledge for RSV led to the development of pre-fusion F-based vaccines which have recently shown highly encouraging results in advanced clinical trials (e.g. NCT04785612, NCT03982199, NCT04032093, NCT03334695). These and other studies have demonstrated that the RSV pre-fusion F protein conformation was highly immunogenic and vaccination reduced RSV infection risk compared to placebo [23–25]. Similar observations have been made regarding distinct conformations of HIV-1 and SARS-CoV-2 fusion proteins in structural studies[26, 27] that have informed clinical studies. Indeed, SARS-CoV-2 fusion proteins with structural modifications are the basis of the highly efficacious mRNA vaccines mRNA-1273 and BNT162b2 (NCT04470427, NCT04368728)[28, 29]. Prior efforts to capture metastable proteins in their most vulnerable conformations have sometimes required extensive and iterative exploration of rational modifications or directed evolution[30]. Reliable predictions of structural states from sequence lends naturally to computational alternatives to experimental protein design, which have recently gained traction in diverse design tasks such as designing immunoglobulin scaffolds, protein biosensors, and specific binding proteins using deep learning and sequence alone (Table 2)[31–33].

Table 2:

Recent advances in structure prediction and computational protein design

Category	Method	Result	Ref
Structure Prediction	AlphaFold	Highly accurate results predicting protein structure from amino acid sequence	[124]
	AlphaFold-2	Updated version of AlphaFold that has solved the protein folding problem	[125]
	RosettaFold	Similar protein structure prediction as AlphaFold	[126]
	ProteinMPNN	Protein backbone sequence design using deep learning	[127]
	trRosetta	De novo protein structure prediction using deep neural networks	[128]
	RaptorX	Web based server for protein structure prediction from amino acid sequence	[129]
	ProGen	Language models can predict protein function from sequence families	[130]
	AminoBERT	Structure prediction using a language model	[131]
	Pfam	Annotating protein function from amino acid sequence with a deep learning model	[17]
		Prediction of protein fitness from evolutionary data	[132]
Protein Design	Deep learning-based design of zinc finger nucleases for specific DNA binding regions		[133]
	Design of IL-2 mimetic protein with reduced toxicity		[134]
	Development of a capsid protein using deep learning		[135]
	De novo design of a chimeric antigen receptor, small molecule regulated, kill switch		[136]
	Computational design of membrane permeable proteins		[137]
	Protein design of axel-rotator-like components		[138]
	Design of proteins binding to specific targets from aa sequence alone		[31]
	Development of nanocage structural proteins		[139]
	Computational design of large multicomponent proteins		[140]
	Rational design of donut-shaped proteins		[141]
	Design of IgG antibodies using multi-state design simulations		[142]
	Design of helical membrane proteins		[143]
	De novo design of a β barrel protein		[144]

Beyond viral antigens themselves, new approaches to present immunogens in defined spatial densities and orientations[34–37] further leverage advances in computation protein design[38]; indeed, approaches to predict attributes of immune responses that may be elicited by these immunogens are advancing. With respect to this latter goal, deep learning approaches appear to be gaining traction. For instance, progress has been made predicting linear B cell epitopes[39–42], in some cases substantially improving sensistivity and specificity, facilitated by growing databases such as the Immune Epitope DataBase (https://www.iedb.org/)[43]. Conformational B cell epitope prediction has been more challenging; however, Graph-based neural networks have recently become more successful at modeling structural space of conformation epitopes by essentially representing interactions in which graph nodes represent atoms and edges represent connections between them[44, 45]. Although substantial room for improvement certainly remains, B cell epitope prediction is expected to continue improving with the combination of larger experimental data sets and novel neural network architectures that might best model the epitope-paratope interactions.

Collectively, each of these aspects of protein structure modeling and prediction advanced by deep learning are in the process of being productively deployed toward vaccine design. At the most advanced end of this spectrum, engineered versions of natural viral proteins are presented, sometimes on designed particles, with the intent of driving recognition of specific epitopes whose recognition might be predicted and result in pathogen neutralization.

Immune repertoire analysis to understand vaccine-induced responses

Whereas the human genome project’s sequencing efforts were widely thought to have provided more information than insight, at least initially, large scale sequencing efforts directed at B and T cell receptors appear to be primed to be effectively coupled to advances in ML in ways that could meaningfully inform vaccine research and development. Given their central positions in building long term immunity after vaccination, B and T cell receptor sequencing has been an intense area of study in the last decade, being revolutionized by next generation sequencing (NGS) technology, resulting, for example, in over 1.5 billion unique human B cell receptor (BCR) sequences available[46]. Coupled to deeper phenotypic characterization, we expect that this rapid expansion in input data can provide rich resources to ML models. To date, deep learning models trained on immune repertoire sequence data have successfully predicted treatment outcomes from immunotherapy[47], as well as infection status or history[48], and infectious disease severity[49].

For example, basic studies of how antibody sequences vary among individuals[50, 51], and with disease[52], have started to be paired with immunogen engineering[53] and in vitro assays[54, 55] to build vaccine strategies that aim to generalize the induction of specific antibody responses[56]. With early “natural history” studies[57, 58] maturing toward cohort-sized undertakings[59–61], progress has been made in making inferences of antigen specificity[62–64], positioning deep learning to support more explicit links between antigenic stimuli and resultant responses. Coupled to insights from elegant animal model experiments[65], iterative cycles of vaccination and repertoire sequencing may provide the raw data needed to gain fundamental and quantitative insights into phenomena such as “original antigenic sin”[66](or antigenic imprinting), and might provide a better understanding of how immune history impacts future immune responses at molecular-level resolution.

In the context of the T cell receptor (TCR), previous studies started to uncover features of TCR sequencing datasets that support prediction of epitope specificity[67, 68]. Relative to antibody-antigen complexes, the structural conservation in TCR-peptide-MHC has supported more facile learning of quantifiable descriptive features[47, 69] that can contribute to prediction of specificity or relationships to other biological attributes. Comparisons of TCR repertoires in individuals with progressive and controlled disease have been used in the context of experimental antigen screens to define TCR specificities that are associated with pathogen control, therefore representing promising vaccine targets. For example, in studies of Mycobacterium Tuberculosis, comparison of TCR sequences defined common T cell specificities for peptide-MHC, that represent novel targets for vaccine design[70, 71].

Deep learning has also been used to model antibody-antigen interactions based on data from directed evolution studies of libraries of antibody sequences with changes in antigen binding over rounds of diversification using methods such as error prone PCR to introduce sequence mutations and selection of mutants of interest that have mutations that impact binding affinity or kinetics. Initial applications of deep learning to enrich for antigen binding molecules focused on phage display experiments[72, 73]. Models were used to predict binders, thereby speeding up the experimental process of affinity maturation. Models have also been applied to yeast surface display libraries focused on identifying immunoglobulin CDR3 regions of antibody heavy and light chains with the goal of understanding the impact of sequence mutations[74]. Results from library studies have demonstrated that deep learning models can identify useful patterns in relationships between antibody sequence and structural space, showing that geometric similarity and structural commonalities in CDRs can reflect attributes of antigen recognition[75], particularly over accumulated mutations. Abstracting these efforts toward the analysis of sequence repertoires from the study of immunized and vaccinated individuals, we expect that rapid gains in insights into the specificity and affinity maturation of antibody responses may ensue.

Phylogenetic analysis to understand viral evolution and population susceptibility

Continuous evolution that leads to escape from an immune response within individuals and across populations poses a persistent challenge to the development of vaccines for viral infections. Historically, predicting which viral strains will become dominant relies on modeling that can approximate educated guess work. A leading example is the influenza virus vaccine, which is designed each year based on predictions of which strains will be dominant. Incorrect predictions result in compromised vaccine efficacy and higher burdens of seasonal flu. Modeling whether or when strains may shift host species is even more difficult, but such changes in tropism can be highly consequential, given the vulnerability of naïve populations. Yet, the rapid expansion and exceptional penetrance of SARS-CoV-2 variants of concern demonstrate that viruses can exhibit substantial evolution and point estimates of population susceptibility can vary dramatically, even within a seasonal time scale in populations with a high degree of prior exposure[76–78]. To this end, the insufficiency of the immune system to outpace antigenically variant viruses ranging from common colds to HIV-1 infection, may highlight the value of new approaches to combine advances in repertoire sequencing with insights into viral phylogenies.

Fortunately, advances in technology and global infrastructure have made it easier to sequence viral variants observed in many individuals, raising the prospect of moving well beyond technical enhancements that define sequence phylogenies[79] and toward more functionally informed inferences into the future directions of viral evolution. For example, mutation-resistant amino acid residues and CD8 T cell responses prevalent among individuals able to suppress HIV-1 replication were studied using analysis combining structural data and network theory in order to quantify the structural importance of amino acid mutations on viral evasion from T cell responses. These studies have opened the door to deploying network theory approaches to design novel T cell epitope-based vaccine concepts that are resistant to viral mutation[80].

Similarly, the rapid and thorough study of SARS-CoV-2 and its continued evolution exemplify the push toward data-driven approaches that rely on more comprehensive sequencing and novel functional data streams, combined with ML models. Viral sequence information is now captured globally through robust sampling and sequencing networks[81] and can then be viewed through the lens of deep mutational scanning data[82–84] allowing for investigation of the impact specific amino acid mutations have on antibody and vaccine responses from emerging sequence variants. In these approaches, libraries comprised of thousands to billions of virus or viral antigen sequence variants are screened for phenotypes such as loss of binding to monoclonal or polyclonal antibody pools[85], enhanced infectivity[86] or binding to entry receptors[87], revealing advantageous amino acid mutations in viral sequences.

Integrating data that link attributes of host and pathogen biology from both in vivo and in vitro sources enables estimation of the tendency of different viral sequence variants to escape from contemporaneous antibody responses in the population as well as monoclonal therapies. Thus, these approaches offer opportunities to predict population-level susceptibility and to anticipate the identity of future viral variants of concern[88]. While there is clearly significant utility afforded by classical ML approaches, deep learning models are beginning to be employed on these tasks[89–91]. Given the ambitious plans for serosurveillance[92] across viruses[93], as well as antibody[94] and T cell epitopes[95] by collecting and profiling samples across global populations, and the creation of scientific networks with the capacity for rapid mechanistic and efficacy experiments[96], we envision that the development of models informed by protein structure and predicted interactions, in particular, have the potential to be greatly enhanced by deep learning.

Challenges and considerations

While deep learning can successfully tackle complex biological problems, its application is likely to stretch current immunology datasets and leave gaps in interpretation. Biological datasets are often considerably smaller than those commonly used in deep learning tasks. Immunological datasets often suffer from being “wide”, with a greater number of features than outcomes, are comprised of experimental input data that are inherently noisy due to both technical and biological variability, and are likely to confront the challenges of representing diverse populations that have been popularized by settings such as facial recognition[97]. Understanding how to address experimental and human variability while creating models that are still robust in making predictions is a challenge that is independent of the specific learning task. Moreover, while “interpretable” deep learning models are under development[98–100], the numerous parameters and layered construction inherent to current approaches drive greater and greater abstraction at each level, ultimately resulting in a scenario in which accurate and reliable predictions are made, but insights into mechanisms are obscured. Moreover, computational constraints must be considered; unlike classical ML and other types of statistical analysis, usually, deep learning models cannot run on a personal computer in a reasonable amount of time, and instead require high performance computing clusters with graphics processing units. Indeed, specialized hardware for deep learning is an area of development for several companies and research laboratories, while other groups work on algorithms and data structures that may yield more efficient run times for processing data[101–104]. Fortunately, the excellent predictive performance of deep learning models in various tasks suggests that progress will continue toward these current and future challenges from which vaccine research and development can benefit.

Concluding Remarks

Cutting-edge technological advancements in immunology and data science provide an opportunity for applying new analytical approaches to the development of vaccines, but leave a number of outstanding questions. Specialized domains of structure prediction, immune repertoire analysis, and phylogenetics are current areas of vaccine research on which deep learning is poised to have impact. We anticipate that insights from the application of deep learning on these tasks will offer opportunities to refine the enormous space of molecular possibilities in basic, translational, and clinical trials testing promising vaccine candidates; this can reduce research time and contribute to the development of highly efficacious vaccines for some of the most challenging viruses confronting human populations.

Figure 4. Immune repertoire and deep learning model analysis.

A. Overview of experimental workflow for B and T cell sequencing experiments. After cells are sorted and analyzed on a sequencer, deep learning models can make predictions on various aspects of the immune repertoire. B. Simplified schematic of technology development for sequencing B cells and T cells. C. Over time, sequencing data has continued to accrue with a corresponding growth in deep learning models with improved performance. This figure was created using BioRender (https://biorender.com/)

Glossary

Affinity maturation

the process by which antibodies are selected over iterative rounds within germinal centers to select for clones with the greatest binding, which has served as a model for in vitro strategies of protein engineering.

AlphaFold

protein structure prediction model developed by DeepMind that has solved the protein folding problem by Critical Assessment of protein Structure Prediction (CASP) metrics.

BCR

B cell receptor, a transmembrane receptor on the surface of B cells that binds antigens

CASP

Critical Assessment of Structural Predictions, a biannual benchmarking study of novel approaches to predict protein structures that have been solved by experimental methods but not yet deposited in the Protein Data Bank

Deep learning

a subset of machine learning in which models are built from extracting simpler components into more complicated ones forming many layers that allowing the model to make accurate predictions

Deep mutational scanning

a set of experimental methods in which a protein or pathogen of interest is diversified in amino acid sequence and then screened in high-throughput fashion to define sequence-function relationships

Directed evolution

an experimental method in protein engineering that uses sequence variation and selective pressure to iteratively screen and identify variants of a desired phenotype

Epitope

surface area on an antigen that is recognized by another protein

Graph-based neural networks

a class of deep learning models that are used when data can be represented by nodes and edges within a graph structure

Immunogen

an antigen capable of generating an immunological response

Immune repertoire

the antibodies, BCRs, and TCRs that compose the adaptive immune response observed in an individual

Layer

a unit of a deep learning model composed of nodes that takes input data, transforms the data with model weights, and applies an activation function to the data

Neural network

a deep learning model representation that aims to mimic the biological learning process in the brain

Next generation sequencing

extremely high throughput methods of sequencing that rely on highly parallel processing to determine expression levels and genetic variation in RNA or DNA

Node

a unit in a deep learning model that is comprised of input connections, weights, and an activation function. Multiple nodes at the same depth comprise a layer

Machine learning

a set of methods that enable the detection of patterns in data.

Neutralizing antibody problem

the distinction between antibodies that bind to a given pathogen and those that provide broad anti-pathogen activity

Next-generation vaccinology

new approaches to vaccine design and research that move beyond empirical approaches

Original antigenic sin

described more often now as antigenic imprinting, the experimentally supported theory that initial adaptive immune responses to antigen influence characteristics of subsequent exposures to antigenic variants

Paratope

the part of an antibody that recognizes the respective antigen

Point estimates

a statistical method that infers information about a population by using a sample statistic

Predictive model

a model that uses previous data to make a forecast related to unseen data

Protein folding problem

as initially described, the lack of clarity as to fundamental forces at play in supporting rapid protein folding; as later described, considered the greatest challenge in bioinformatics, the task of computationally predicting the structure of a protein from amino acid sequence alone with the same accuracy as experimental methods

Reverse vaccinology

a vaccine development approach that uses bioinformatics to systematically identify, prioritize, and then experimentally evaluate the suitability of proteins in the genome of a pathogen as vaccine immunogens.

Structure-based vaccine design

a vaccine development approach that leverages structural biology in the selection or engineering of candidate immunogens

TCR

T cell receptor, a T cell surface receptor that recognizes peptides presented by MHC molecules

Test data

the portion of the data set not previously seen by the model that is used to measure performance after training

Training data

the portion of the data set a model uses to learn to make predictions for a given task