A 5’ UTR Language Model for Decoding Untranslated Regions of mRNA and Function Predictions

Abstract

The 5’ UTR, a regulatory region at the beginning of an mRNA molecule, plays a crucial role in regulating the translation process and impacts the protein expression level. Language models have showcased their effectiveness in decoding the functions of protein and genome sequences. Here, we introduced a language model for 5’ UTR, which we refer to as the UTR-LM. The UTR-LM is pre-trained on endogenous 5’ UTRs from multiple species and is further augmented with supervised information including secondary structure and minimum free energy. We fine-tuned the UTR-LM in a variety of downstream tasks. The model outperformed the best known benchmark by up to 5% for predicting the Mean Ribosome Loading, and by up to 8% for predicting the Translation Efficiency and the mRNA Expression Level. The model also applies to identifying unannotated Internal Ribosome Entry Sites within the untranslated region and improves the AUPR from 0.37 to 0.52 compared to the best baseline. Further, we designed a library of 211 novel 5’ UTRs with high predicted values of translation efficiency and evaluated them via a wet-lab assay. Experiment results confirmed that our top designs achieved a 32.5% increase in protein production level relative to well-established 5’ UTR optimized for therapeutics.

The 5’ untranslated region (5’ UTR) is a region at the beginning of an mRNA that precedes the coding sequence of the protein. It plays a critical role in regulating the translation from mRNA to proteins, as it can influence the stability, localization, and translation of the mRNA molecule¹. There has been a significant amount of research^2–9 exploring the biology of the 5’ UTR, including its secondary structure², RNA-binding proteins that may interact with it³, and the effect of mutations within the 5’ UTR on the gene expression⁴. The complex functions of mRNA and their potential implications for human health underscore the necessity for more universally applicable computational approaches.

Investigation into the role of 5’ UTRs encompasses various aspects of translational control. With the growing interest in studying and designing 5’ UTRs, various computational tools^5–11 have been developed to study its functions. For example, the ribosome load measures the number of ribosomes engaged in translating a given mRNA at a given time. Supervised machine learning models were shown to predict the mean ribosome loading (MRL)^5,6,9,10 based on the UTR sequence or its biological features. Additionally, RNABERT¹² and RNA-FM¹⁰ are language models specific to RNA sequences and were shown useful for predicting the MRL. Scientists also tried to predict the mRNA translation efficiency (TE)^7,9 that quantifies the rate of translation into proteins and the mRNA expression level (EL)^7,9 that reflects the relative abundance of the mRNA transcript in the cell. MTtrans⁹ uses a multi-task model that harnesses diverse data types for translation rate prediction. While there exist specialized models for individual prediction tasks, there lacks a unified methodology to decode functions of 5’ UTR.

In this study, we adopted the principled approach of a language model in order to extract meaningful semantic representations from untranslated regions of raw mRNA sequences and further map them to predict functions of interest. Specifically, we developed a semi-supervised language model, which we refer to as the UTR-LM, trained using sequences of 5’ UTR from multiple data sources (Fig. 1). The transformer-based model is pre-trained to extract representations from the raw sequences via nucleotides masking and reconstruction. It also incorporated supervised information such as the secondary structure (SS) and the minimum free energy (MFE). We applied the UTR-LM and fine-tuned it for a variety of downstream tasks, such as predicting the mean ribosome loading, the mRNA translation efficiency and the mRNA expression level. Experiment results showed that the UTR-LM accurately predicts these regulatory functions. When compared with existing baselines in each downstream task^{5–7,9,10,12,13}, the UTR-LM demonstrated state-of-the-art performances across modalities and test sets. In particular, the UTR-LM outperforms the MTtrans⁹ by 5% for MRL prediction, outperforms the Cao et al.⁷ (Cao-RF) by up to 8% for TE and EL predictions, in terms of the Spearman R score. The model generalized well to unseen data, especially human 5’ UTRs with varying lengths. Additionally, we adapted the UTR-LM to identify unannotated internal ribosome entry sites (IRESs)^13–16, which are sequences within mRNAs that enable ribosomes to initiate translation internally, bypassing the traditional cap-dependent mechanism. The UTR-LM outperforms IRESpy¹³ by 0.15 in terms of the AUPR score.

Fig. 1. Overview of the UTR-LM model for 5’ UTR function prediction and design.

(a) The input of the proposed pre-trained model is the 5’ UTR sequence, which is fed into the transformer layer through a randomly generated 128-dimensional embedding for each nucleotide and a special [CLS] token. The pre-training phase employs a combination of masked nucleotide (MN) prediction, 5’ UTR secondary structure (SS) prediction, and 5’ UTR minimum free energy (MFE) prediction. (b) Following pre-training, the [CLS] token is used for downstream task-specific training. (c) The UTR-LM is fine-tuned for downstream tasks such as predicting mean ribosome loading, translation efficiency, mRNA expression level, and internal ribosome entry site. (d) Designing a in-house library of 5’ UTRs with highly predicted translation efficiency and the wet-lab experimental validation employing mRNA transfection and luciferase assays. Drawings were created using BioRender.com.

Given the vital role of untranslated regions of mRNA on the translation process, artificially designing the 5’ UTR holds potential to improve the translation efficiency and optimize the protein production⁶. Leveraging this biological principle, we designed an in-house library of 211 novel 5’ UTRs with high predicted values of translation efficiency. We conducted experiments of mRNA transfection and luciferase assay to evaluate our designs. Wet-lab experiments revealed that top 5’ UTRs in our design library achieved up to 32.5% increase in protein production efficiency, when compared to the benchmark NCA-7d-5’UTR¹⁷, which was optimized for encoding SARS-CoV-2 antigens and elicited strong immunity when delivered using liquid nanoparticle (LNP) for in vivo vaccination. The observed increase in protein production efficiency suggests that the newly designed 5’ UTRs could have practical applications in various biotechnological processes. Further, we used the in-house data as an independent set to test UTR-LM for zero-shot fitness prediction, and showed that the model significantly outperformed other methods by up to 51% in terms of the Spearman R score.

Additionally, we analyzed the pre-trained embedding and the attention score of the UTR-LM. The embedding of UTR-LM is able to differentiate between species and captures features like the minimal fold energy. We also sought to identify motif patterns using an attention-based detection pipeline. In particular, the presence of the Kozak consensus sequence with higher GC content was found to be significant, a finding that aligns with previous research^18,19. Further, attention score analysis also identified that upstream ATG (uATG) were depleted in high-expression UTR and enriched in low-expression ones, consistent with the biological insight that uATG driving upstream open reading frame (uORF) could compete and thus lower downstream translation efficiency and protein expression²⁰. In summary, our study presents the UTR-LM, a novel self-supervised language model for studying untranslated regions of mRNA and decoding its functions. This research holds promising implications for advancing our understanding of gene regulation and innovating therapeutic interventions.

Results

UTR-LM predicts the Ribosome Loading

Ribosome loading refers to the number of ribosomes that are actively translating a specific mRNA molecule at any given time. It is a measure of how efficiently a particular mRNA is being translated into protein, and can be measured by various experimental techniques such as Ribo-seq or polysome profiling²¹. The ribosome loading can influence the rate of protein production and is influenced by factors such as the 5’ untranslated region (5’ UTR) sequence, secondary structures within the mRNA, and the availability of ribosomes. Designing the 5’ UTR can impact ribosome loading, which in turn can be utilized to optimize protein expression levels for various applications, including biotechnology and therapeutic protein production. Scientists had attempted to predict the effect of 5’ UTR sequence on the mean ribosome loading (MRL). Several machine learning models were developed for this specific task, including Optimus⁶, FramePool⁵, and MTtrans⁹. RNA language models namely RNABERT¹² and RNA-FM¹⁰ were also tested using this task.

In our study, we used the pre-trained foundation model for 5’ UTR and further fine-tuned it for the task of MRL prediction. The baseline model is pre-trained on unlabeled 5’ UTR sequences from five species within the Ensembl database, via the masked nucleotides (MN) task, and subsequently fine-tuned for MRL prediction. As shown in Fig. 2a, we tested variants of the model with additional training, including raw sequences from the downstream library, supervised information such as secondary structure (SS), minimum free energy (MFE) and other biology features. Fig. 2b reveals that the baseline model alone attains satisfactory performance, and minor differences are observed among UTR-LM variants. For the final model, we chose the version that incorporates the downstream library, SS and MFE, and we will refer to it as UTR-LM MRL in the rest of the subsection. To investigate the effect of pretraining, we also compared the pre-training+fine-tuning strategy with other strategies that do not use pretrained weights or do not finetune the full model. As shown in Fig. 2d, our UTR-LM model showed the highest performance, significantly outperforming other baselines. This aligns with our expectation and demonstrates the effectiveness of our pre-training and fine-tuning strategy (more details in Supplementary Materials B.5).

Fig. 2. Prediction of mean ribosome loading (MRL) based on 5’ UTR sequences.

(a) Illustration of the UTR-LM framework, which includes variants integrating downstream library (DSL), secondary structure (SS), and minimum free energy (MFE) tasks during pre-training, along with biological features (BF) during downstream prediction. (b) Ablation study of UTR-LM hyperparameters under the U₁ library (a sub-library of synthetic 50-nucleotide 5’ UTRs, see the Methods Section for details) with Rank Splitting. For subsequent experiments, we used the baseline UTR-LM enhanced by DSL, SS, and MFE, termed UTR-LM MRL. (c) Comparison of various pre-trained methods followed by either a simple multilayer perceptron (MLP) with 1 hidden layer or a complex 32-layer residual network (ResNet) under U₁ library with Rank Splitting. (d) Ablation study on the training strategy. We compared the pre-training+fine-tuning strategy with other training strategies that use random weights and/or freezed weights (U₁ library with Rank Splitting). (e-f) Evaluation of various methods across eight libraries with random 50-nucleotide 5’ UTRs. (g-h) Evaluation of various methods using independent tests. In independent tests, we fine-tuned UTR-LM MRL and re-trained baselines on 76,319 random 5’ UTRs (25–100 bp), and tested models on 7,600 random and 7,600 human 5’ UTRs.

We tested the UTR-LM MRL model with two downstream predictors: a simple multilayer perceptron (MLP) with 1 hidden layer and a much more complex 32-layer residual network (ResNet). In Fig. 2c, UTR-LM MRL followed by both predictors achieve similar performances, outperforming two other RNA foundation language models, i.e., RNA-FM and RNABERT, on this same task. Moreover, RNA-FM and RNABERT worked well with a deep ResNet predictor, but performed poorly with the simpler MLP predictor. This observation hints that RNA-FM and RNABERT rely on deep neural networks to learn the sequence-to-MRL relation. In contrast, our pre-trained UTR-LM MRL model does not rely on such complex predictors and achieves robustly high performances, indicating that it has extracted better semantic embeddings from 5’ UTR sequences.

We compared the UTR-LM MRL model with benchmark methods from the MRL prediction literature^5,6,9,10 across eight synthetic libraries. This analysis focused on the 50-bp segment within the 5’ UTR closest to the start codon. Variants within this region are believed to be subject to stronger negative selection, likely because they can have more immediate effects on the gene’s ability to produce proteins²². We tested two different splitting strategies (Supplementary Materials B.3): Rank Splitting⁶ selects the 5’ UTRs with the highest read counts as the test data and uses the rest for training; Random Splitting splits data randomly into training and test sets. As illustrated in Fig. 2e,f, UTR-LM MRL consistently shows higher performance compared to other methods across all these tests. In particular, the UTR-LM MRL outperforms Optimus by up to 0.08, outperforms FramePool by up to 0.07, and outperforms RNA-FM by up to 0.43, in terms of the Spearman R score (Additional metrics and results reported in Supplementary Materials D.1, D.2, D.3).

We aimed to assess whether our model, specifically trained on 50-bp synthetic sequences, could predict the regulatory functions of human 5’ UTR sequences with varying lengths. While human 5’ UTR sequences can span from tens to thousands of nucleotides, a mere 13% of them are less than 50-bp long. We used two datasets that were originally proposed by Sample et al.⁶ and later as independent tests in several studies^5,6,9,10. They allow evaluating the model’s adaptability from the training data (including only synthetic 5’ UTRs of 50-bp length) to human 5’ UTRs of varying lengths. These datasets encompass both synthetic 5’ UTRs and human 5’ UTRs, with their lengths ranging from 25 to 100 bp, and they do not overlap with our training data. By following the length-based held-out testing approach suggested by Optimus⁶, we fine-tuned the UTR-LM MRL model and retrained available baselines^5,6,9,10 on the 76,319 random 5’ UTRs with 25 to 100 bp. Then, we tested and compared UTR-LM MRL and baselines on 7,600 random and 7,600 human 5’ UTRs. Further details can be found in the Methods Section and Supplementary Materials B.4. Our results are illustrated in Fig. 2g,h. They show that the UTR-LM MRL, when paired with a ResNet downstream predictor, outperforms all other methods in both tests, while UTR-LM MRL with a simpler MLP exhibits similarly competitive performance. In addition, we analyzed the prediction results across various sequence lengths. The results demonstrate that UTR-LM MRL can be effectively extended to both longer and shorter 5’ UTRs. Notably, our model shows the state-of-the-art performance on human 5’ UTRs (Fig. 2h). In particular, the UTR-LM MRL outperforms Optimus, FramePool, and RNA-FM by about 1%−6%, in terms of Spearman R score. It highlights the generalizability of UTR-LM MRL for decoding functions of endogenous 5’ UTRs.

UTR-LM Predicts mRNA Translation Efficiency and Expression

Protein production involves two primary processes: transcription and translation. The level of protein expression is highly dependent on the mRNA expression level (EL) and also the translation efficiency (TE) of the transcripts^23,24. EL is measured based on the relative abundance of the mRNA transcript in the cell and is quantified using RNA-seq RPKM⁷, where RPKM denotes Reads Per Kilobase of transcript per Million mapped reads. On the other hand, the TE of a gene, reflecting the rate of mRNA translation into protein, is calculated by dividing the Ribo-seq RPKM (indicative of ribosomal footprints on the mRNA) by the RNA-seq RPKM⁷.

In this section, we applied the pre-trained UTR-LM and fine-tuned it for TE and EL prediction via 10-fold cross-validation. We used three endogenous datasets⁷ gathered from human muscle tissue (Muscle), human prostate cancer cell line PC3 (PC3), and human embryonic kidney 293T cell line (HEK) for training and testing. For benchmarking, we compared our method with a random forest model based on 3000+ handcrafted biological features by Cao et al.⁷ (Cao-RF), and several other sequence-based models including Optimus⁶, FramePool⁵, MTtrans⁹, RNABERT¹², and RNA-FM¹⁰. Here note that the Optimus, FramePool, RNABERT, and RNA-FM were not developed for the mRNA TE and EL tasks, thus for comparison we re-trained these models using TE and EL datasets.

For the prediction of TE, we conducted ablation studies using various UTR-LM variants. Unlike the MRL task (as shown in Fig. 2b), the TE task (illustrated in Fig. 3a,b,c) reveals notable performance discrepancy among different UTR-LM variants. When augmented with downstream library and MFE loss, the fine-tuned model demonstrates substantially higher performance across various datasets. We observed that the use of the downstream library alone or SS could lead to performance degradation, possibly due to the fact that we had to truncate long sequences in the model and it might have introduced noticeable bias. We also observed that the additional use of biological features only affects the performance very minorly, so we chose to not include it in the final model. In the rest of the section, we chose the UTR-LM enhanced by the downstream library and MFE for the tasks TE and EL prediction. We will refer to this variant as UTR-LM TE for brevity. We also adapted the UTR-LM TE architecture to predict EL, which we refer to as UTR-LM EL. Further, we conducted an ablation study to compare our training strategy with other strategies. Fig. 3d showed that the fully fine-tuned pretrained model consistently outperforms others. Models with random initialization, even when fully fine-tuned, do not reach the same effectiveness, highlighting the critical role of pre-training. See Supplementary Materials B.5 for details.

Fig. 3. Prediction of mRNA translation efficiency (TE) and expression level (EL) for endogenous datasets.

Sources for these datasets include human muscle tissue (Muscle), human prostate cancer cells (PC3), and human embryonic kidney 293T cells (HEK). (a-c) Ablation study of UTR-LM hyperparameters on TE tasks, including the downstream library’s 5’ UTRs (DSL), a downstream task-independent secondary structure (SS), and a downstream task-related minimum free energy (MFE) during the pre-training phase, and the inclusion of biological features (BF) during the downstream prediction phase. For the subsequent experiments, we used the baseline UTR-LM enhanced by DSL and MFE as the final model. (d) Ablation study of the training strategy. We compared the pre-training+fine-tuning strategy of our UTR-LM model with other training strategies that use random weights and/or freezed weights. (e) For the TE prediction, the UTR-LM model outperforms Cao-RF by up to 5% and outperforms Optimus by up to 27% in terms of Spearman R. (f) For the EL prediction, the UTR-LM model outperforms Cao-RF by up to 8% and outperforms Optimus by up to 47% in terms of Spearman R. In panel (e,f), data are presented as mean values +/− SD obtained via 10-fold cross-validation. Paired t-tests showed that UTR-LM outperforms other benchmarks significantly (p-value < 0.05).

For the prediction of TE and EL, we used UTR-LM TE and UTR-LM EL and compared their performances with benchmark methods respectively^{5–7,9,10,12}. Fig. 3e,f illustrates that UTR-LM TE and UTR-LM EL performs competitively with Cao-RF and outperforms other methods. Specifically, for Spearman R, UTR-LM model outperforms Cao-RF by up to 5% and 8% for TE and EL tasks, and outperforms Optimus by up to 27% and 47% for TE and EL tasks. While Cao-RF proves effective in predicting TE and EL, it relies on more than 3000 handcrafted features including k-mer frequency, RNA folding energy, 5’ UTR length, and number of open reading frameworks, and its random forest model that may encounter scalability issues with larger datasets. In contrast, the training of a language model offers a more principled solution to modeling 5’ UTR sequences and only uses information (for example MFE) that can be easily computed. Thus, we believe that the UTR-LM provides a more robust and generalizable model for understanding 5’ UTR sequences.

URR-LM Identifies Internal Ribosome Entry Sites

Internal ribosome entry sites (IRESs) are unique RNA sequences, most located within the 5’ UTR of mRNAs. Unlike the typical cap-dependent translation initiation that starts at the 5’ end of an mRNA, IRESs enables ribosomes to initiate translation directly at the internal sites. Approximately 10% of cellular and viral mRNA are believed to employ IRESs for translation initiation²⁵. However, there existed only a limited number of verified IRESs, and most of them remain unannotated.

In this study, we applied the pretrained UTR-LM model to identify unannotated IRESs. We assembled a library of 46,774 sequences including both viral and cellular mRNAs, sourced from multiple databases^14,26–29 (see details in the Methods Section), with 37,602 sequences labeled as non-IRESs and 9,172 as IRESs. Building on the pre-trained UTR-LM, we developed a contrastive learning approach to train a downstream IRES classifier; see the Methods Section.

Several baseline methods exist for predicting IRES, such as IRESfinder¹⁵, IRESpred¹⁴, IRESpy¹³, and DeepCIP¹⁶. See Supplementary Materials C. for more details about these methods. For our comparison, we selected IRESpy as the benchmark because it represents the most recent method for both viral and cellular IRES detection and has shown advantages over IRESpred. We also compared with CNN baselines that were reported as top performers in previous literature^5,6. As illustrated in Fig. 4, the UTR-LM IRES classifier substantially outperforms the best known benchmark, improving the AUPR from 0.37 to 0.52.

Fig. 4. UTR-LM accurately identifies unannotated internal ribosome entry sites (IRES).

Our method, the UTR-LM IRES classifier, achieves higher test accuracy than best known baselines for this task. All models were trained and tested on a dataset of 46,774 sequences via 10-fold cross-validation. Here we chose the metric AUPR because it is more suitable for imbalanced data.

Novel 5’ UTR designs validated in wet-lab experiments

Finally, we conducted a set of wet-lab experimental assay to validate our prediction model and generate novel 5’ UTR designs with high translation efficiency. We designed a library of 211 distinct 5’ UTR sequences with high predicted values of translation efficiency. In the experiment, we used expression of a luciferase reporter gene to measure mRNA translation in human cells (Fig. 5a). We first cloned all designed 5’ UTRs upstream of the same, standard luciferase reporter gene, and then transfected the synthesized mRNAs into C2C12 cells, followed by quantitative luciferase assay (Fig. 5b). Specifically, we measured the relative light units (RLU) that quantifies the luciferase activity. This allowed us to assess the protein production level from each mRNA, which provided a direct measurement of how the designed 5’ UTRs might affect the protein synthesis process. We also compared the predicted values of MRL and TE of the designed library with the background distribution (i.e., distribution of training data in each task). As shown in Fig. 5c,d, the UTR-LM model predicts that these in-house designed 5’ UTRs exhibit significantly higher MRL and TE values than the background.

Fig. 5. Experimental validation of UTR-LM model and top sequence designs generated by UTR-LM.

A library of 211 in-house 5’ UTRs with high translation efficiency was designed and validated through wet-lab experiments. These sequences were subsequently used as an independent test set to verify the efficiency of our model. (a) The mRNA structure with in-house designed 5’ UTR. (b) Experimental workflow for mRNA transfection and luciferase assay. (c-d) The in-house 5’ UTRs had higher predicted values of mean ribosome loading (MRL) and translation efficiency (TE) compared to the background. (e) Wet-Lab Results: Relative translation efficiency of the top 20 designed 5’ UTRs compared to the benchmarks 5’ UTRs NCA-7d-5’UTR and HBB-5’UTR. (f) Using the wet-lab results as an independent testset, the UTR-LM gave substantially more accurate zero-shot predictions of the Relative Light Units (RLU) compared to the benchmark. Drawings in panels (a-b) were created with BioRender.com.

We next compared the efficiencies of the top in-house designed 5’ UTRs with well-established UTRs used for mRNA therapeutics. For benchmarks, we measured our designs against two well-known 5’ UTRs, namely the HBB-5’UTR³⁰ and the NCA-7d-5’UTR¹⁷. The HBB-5’UTR, a 5’ UTR from the human hemoglobin subunit beta, is commonly used in studies of mRNA translation and stability³⁰. The NCA-7d-5’UTR, an optimized 5’ UTR for protein-coding mRNAs, has demonstrated effective delivery via lipid-derived TT3 nanoparticles, resulting in pronounced expression of potential SARS-CoV-2 antigens¹⁷. Our experiment validated that top candidates in our designed library achieved significantly improved protein production level (Fig. 5e). In particular, the top 5’ UTR sequence found in our assay had a 32.5% increase in protein production level compared to NCA-7d-5’UTR. Full details of the experimental design and methods are given in Supplementary Materials A.6.

Next, we evaluated our UTR-LM for zero-shot fitness prediction using our in-house design library and wet-lab results as an independent test set. In this test, we do not have measurements of TE or MRL, but instead we need to predict a new target RLU, measured by the fold change of log2-transformed RLU relative to that of NCA-7d-5’UTR. For zero-shot fitness prediction, we transferred the learnt models UTR-LM MRL and UTR-LM TE to the new test set for predicting RLU, without any additional training or fine-tuning. For comparison, we also tested the benchmark model, Optimus⁶, which was trained on MRL. As illustrated in Fig. 5(f), UTR-LM MRL and UTR-LM TE significantly outperformed the benchmark Optimus on this task of zero-shot prediction of RLU. In particular, UTR-LM TE’s accuracy exceeds that of Optimus by more than one fold, demonstrating the transferability of our model across tasks and modalities.

Further, we applied the pre-trained UTR-LM embedding, trained a new RLU predictor using our in-house data and tested it via 10-fold cross-validation. As illustrated in Fig. 5f, our UTR-LM RLU model substantially outperformed the benchmark Optimus and the zero-shot models. The outcome underscores the significant potential of the language model approach in modeling genome sequences, showcasing its robustness and ability to generalize in predicting wet-lab outcomes.

Discussion

Attention scores reveal proven genome patterns

We examine the 5’ UTR embeddings across the five species used for pre-training. In Fig. 6a, we illustrated the frequency of each nucleotide at each base, per species. In Fig. 6b, we reported the silhouette scores, a metric that measures the separation between clusters, for pairs of species using both the UTR-LM embedding and the 4-mer representation. They showed that the UTR-LM embeddings achieved significantly higher silhouette scores than the 4-mer representation. It suggests that the pre-trained transformer provides more meaningful representations for differentiating between species. Fig. 6c showed that the UTR-LM embedding of 5’ UTR captured most variation in the MFE value, while the traditional 4-mer representation struggled. See also Supplementary Materials E.1, E.2 for additional visualizations.

Fig. 6. The UTR language model recognizes underlying patterns and reveals known motif patterns.

(a) Sequence logo of the five species. (b) Silhouette scores for pairs of species suggest the language model gives better differentiation between species based on 5’UTR. (c) UMAP visualization of 4-mer representation and UTR-LM embeddings, colored by the minimal fold energy (MFE). (d) Attention score of an instance human 5’ UTR, where a higher score means the corresponding site could have higher influence on the target function measured by ribosome loading. (e) Average attention scores per position and nucleotide across 15,555 human 5’ UTRs revealed the Kozak consensus sequence and its conserved pattern CCACC, which is a known motif that largely affects the ribosome loading.

The attention mechanism in the transformer architecture helps to draw connections between any parts of the sequence. In Fig. 6d, we visualize the attention scores of an instance of human 5’ UTR associated with the MRL prediction. A high attention score for a specific site suggests it could be influential in determining the target function. See Supplementary Materials E.3 for more examples and visualizations. Next, we analyzed the average attention score per each position and per nucleotide across all human 5’ UTRs, with the hope to identify potential motif patterns. As shown in Fig. 6e, we saw that high attention scores in positions 1–6 align with the proven conserved nucleotide patterns³¹ of the Kozak consensus sequence (KCS)^18,19. KCS is a nucleotide motif that functions as the protein translation initiation site in most mRNA transcripts. More specifically, our analysis found a most common conserved pattern CCACC³² (Fig. 6e) among the KCS variations. Finally, we screened all 3–12 mer fragments, filtered and ranked them by their attention scores, to identify new potential motifs; see Supplementary Materials E.4 for the top found motifs. Notably, attention scores identified uATG being critical for UTR design, and a potential positional effect that inspires future investigation into the influence of uATG (and thus uORF) within the 5’ UTR. While these motif patterns call for biological validation and further research, they hold the potential to unlock new understanding into mRNA translation and de novo design.

Summary

We introduced a language model for 5’ UTRs that integrates sequence, secondary structure, and minimum fold energy (MFE). The UTR-LM model has learned meaningful semantic representations from 5’UTR sequences that pertain to the mRNA translation process. It applies to predicting the mean ribosome loading (MRL), translation efficiency (TE), mRNA expression levels (EL), and internal ribosome entry site (IRES) and outperforms the best known baseline in each task. Also, we generated novel 5’ UTR designs with highly predicted translation efficiency and measured their performances via wet-lab experiments. Our experiment revealed highly-efficient novel 5’UTR designs with potential therapeutic value. While our model’s current performance is not limited by sequence length, it will be an exciting future direction to consider sparse transformers for modeling longer RNA sequences and more complex biological functions.

Methods

Overview of UTR-LM

We developed a unified foundation language model to provide meaningful and rich representations for 5’ UTRs. The model adopts a transformer architecture and it is trained using multi-modal data, including raw sequences, secondary structure, and minimum free energy. The model is pre-trained in a self-supervised learning manner via mask reconstruction, secondary structure prediction and MFE prediction. It is later fine-tuned for a variety of downstream function prediction tasks, where it is shown to improve the state-of-the-art performances in each task.

We trained UTR-LM on two computing clusters. The first cluster, hosted on the Amazon Web Services cloud platform, was equipped with four Tesla V100-SXM2 GPUs, each boasting 16 GB of high-bandwidth memory. The second cluster, the Stanford University’s Sherlock high-performance computing system, employed four TESLA_P100_PCIE GPUs, each with 32 GB of memory. We configured the training to a maximum of 200 epochs and two days.

Datasets

For pre-training the language model, we collected unlabeled 5’ UTR sequences from three sources: the Ensembl database³³, synthetic libraries from Sample et al.⁶, and endogenous human 5’ UTR data analyzed by Cao et al.⁷ We pre-processed the raw data to keep only high-quality and well-defined 5’ UTRs, details are shown in Supplementary Materials A.1. After the pre-processing steps, the final large-scale dataset from several sources is obtained:

We obtained 214,349 unlabeled 5’ UTR sequences from the Ensembl database³³, spanning five species: human, rat, mouse, chicken, and zebrafish.

We obtained eight synthetic libraries of random 5’ UTRs truncated to 50 nucleotides long from Sample et al.⁶ These libraries are sorted into two groups based on their coding sequence (CDS): six libraries are linked to the enhanced green fluorescent protein (eGFP), and two are linked to mCherry. The eGFP-linked libraries consist of two unmodified uridine (U) libraries (U₁ and U₂), two pseudouridine (Ψ) libraries (Ψ₁ and Ψ₂), and two 1-methyl pseudouridine (m₁Ψ) libraries (m₁Ψ₁ and m₁Ψ₂). Each eGFP library contains approximately 280,000 distinct 5’ UTRs. The mCherry-linked libraries, named mC-U₁ and mC-U₂, each contain around 200,000 unique 5’ UTRs.

We obtained three endogenous human 5’ UTR datasets analyzed by Cao et al.⁷, each originating from a distinct cell line or tissue type: human embryonic kidney 293T (HEK), human prostate cancer cell (PC3), and human muscle tissue (Muscle). The HEK, PC3, and Muscle datasets comprised 14,410, 12,579, and 1,257 sequences, respectively.

In addition, we also included unlabeled raw sequences from datasets of downstream tasks in the pre-training. For more in-depth description of each dataset, please see Supplementary Materials A.

Architecture and pre-training of UTR-LM

We developed a specialized language model for studying 5’ UTRs, called UTR-LM (as illustrated in Fig. 1). The main architecture comprises an encoder block and a predictor block. The encoder block consists of a six-layer transformer³⁴ with 16 multi-heads self-attention. The layer normalization and residual connections are applied before and after each encoder block. The predictor block is a two-layer feed-forward neural network.

In our UTR-LM model, a 5’ UTR sequence of length L is input as a series of nucleotide tokens (such as ‘A’, ‘G’, ‘C’, ‘T’), along with a special [CLS] token. These tokens are first converted into 128-dimensional vectors via an embedding layer, forming an (L+1) × 128 matrix. This matrix then goes through the encoder block, generating a representation for each nucleotide token in the sequence.

In the pre-training stage, we used a mix of self-supervised learning and supervised learning. For the self-supervised training part, we followed the masked language modeling (MLM)³⁵, where we randomly masked 15% of the nucleotide tokens in the 5’ UTR sequence. The model is then trained to predict these masked tokens by minimizing the cross-entropy loss to encourage accurate reconstruction. We refer to this part as the masked nucleotide (MN) training task. This training objective function is

$$L_{MN}=E_{x~X}{E}_M\sum _{i\in M}-logp\left(x_i\right|x_{/M})$$ (1)

In Eq.1, we randomly select a set of indices, denoted as $M$, from each input sequence $x$ (making up 15% of the entire sequence). The token at each index $i$ is replaced with the mask token [MASK]. The objective is to minimize the negative log-likelihood of the correct reconstruction of each $x_i$, when the unmasked part $x_{/M}$ is given as context.

For the supervised training part, we used two labels to provide auxiliary supervision. First, we included the secondary structure (SS) of 5’ UTR for training, which is calculated using the software ViennaRNA³⁶. We represented the SS using the “dot-bracket” notation³⁶ , where paired nucleotides are given by “(“ and “)” characters and unpaired nucleotides are given by a “.”. For instance, the SS of the sequence AUGCAUGCGAUCAGC is given by “(((..)))..((.))”. To utilize the SS, we introduced an MLM-inspired task (which we call the SS task) to predict the SS symbols associated with masked nucleotides. The training objective function is

$$L_{SS}=E_{x~X}{E}_M\sum _{i\in M}-logp\left(s_i\right|x_{/M})$$ (2)

Eq.2 differs from Eq.1 in that it uses the secondary structure $s$ as input. It attempts to reconstruct the masked secondary structure, when the unmasked part $x_{/M}$ is given as context. The set of mask indices $M$ is the same as for the MN task. Second, we used the minimum free energy (MFE) of the 5’ UTR as an additional target of prediction, because there exists proven correlation between the MFE and the translation efficiency⁶. We refer to this step as the MFE training task, where the predictor block uses the [CLS] token’s representation to estimate the MFE value. We used the ViennaRNA software³⁶ to calculate the actual MFE. The training objective is to minimize the mean squared error:

$$L_{MFE}=E_{x~X}{({MFE}_x-\widehat{{MFE}_x})}^2$$ (3)

where ${MFE}_x$ and $\widehat{{MFE}_x}$ are the actual and predicted MFE of the input sequence $x$.

Downstream training

We used the learned representations from UTR-LM for several downstream tasks, as illustrated in Fig. 1b,c. We extended the predictor block of the pre-trained UTR-LM from predicting MFE to predicting the downstream label. Instead of using UTR-LM merely as a static encoder to generate embeddings, we fine-tuned the model using the pre-trained base model as the initial point. We provided a full chart of model sizes and runtime needed for fine-tuning the proposed and benchmark models on synthesis library in Supplementary Materials B.6.

Prediction of Mean Ribosome Loading (MRL)

Ribosome loading, a measure of mRNA translation efficiency influenced by factors like the 5’ UTR, is experimentally assessed through techniques like polysome profiling to derive the mean ribosome loading (MRL). Understanding and designing the 5’ UTR can optimize MRL, thereby enhancing protein production for applications such as biotechnology and therapeutics. Here, we attempted to predict MRL based on the 5’ UTR sequence.

Firstly, we used eight libraries of randomized 50-nucleotide oligomer 5’ UTRs⁶, which are U₁, U₂, Ψ₁, Ψ₂, m₁Ψ₁, m₁Ψ₂, mC-U₁ and mC-U₂. These libraries include the measured MRL for each 5’ UTR sequence. Details and statistics can be found in Supplementary Material A.3. Following Sample et al.⁶, we fine-tuned the UTR-LM for each downstream library, which we collectively refer to as UTR-LM MRL.

We performed independent tests on a dataset containing 83,919 random 5’ UTRs and another one containing 15,555 human 5’ UTRs, both with varying lengths from 25 to 100 bp. We employed two testing strategies for independent tests. The first strategy, proposed by Optimus⁶, employed a length-based held-out testing approach. Each test set contains 7,600 random 5’ UTRs or 7,600 human 5’ UTRs with the most reads. The remaining 76,319 random 5’ UTRs were used as the training set for both independent tests. The second strategy is to use a 10-fold cross-validation. Because the U₁ is commonly used in prior UTR research^5,6, we started with the UTR-LM MRL model that was trained on U₁ as the initial model and then fine-tuned it using the training set.

Prediction of mRNA Expression Level (EL) and Translation Efficiency (TE)

Protein production is influenced by mRNA expression level (EL), measured in RNA-seq RPKM⁷, and translation efficiency (TE), calculated as the ratio of Ribo-seq to RNA-seq RPKM⁷. Both factors are essential for understanding how the 5’ UTR influences the rate at which mRNA is translated into protein.

We used three different human datasets⁷ for the TE and the EL tasks. These datasets are named Muscle, PC3, and HEK, and together they contain 41,446 unique 5’ UTRs. Each sequence of these datasets provides measurements of TE and EL. In alignment with the limitation of commercially available single-stranded DNA template biosynthesis⁷, a fixed 5’ UTR length of 100-bp was chosen for training. More detailed statistics can be found in Supplementary Materials A.4.

For each dataset, we trained a TE prediction model and EL prediction model, respectively. The model is fine-tuned based on the parameter of the UTR-LM model pre-trained on both Ensembl database and 41,446 human 5’ UTRs as an initial point.

While our 5’ UTR model was able to achieve reasonable performance to predict mRNA EL, 3’ UTR is thought to be more critical in determining mRNA stability and other properties. Nonetheless, recent work showcased that 5’ UTR may also contain structured RNA elements that influence mRNA stability³⁷. Thus, we’d like to note that the EL is ultimately determined by a combination of factors that included 5’ and 3’ UTRs, and requires further investigation.

Identification of Internal Ribosome Entry Site (IRES) with Contrastive Learning

Internal ribosome entry sites (IRESs) are unique sequences that are often found within the 5’ UTRs of mRNAs. These specialized sequences enable cap-independent translation initiation and are thought to regulate translation for a subset of cellular and viral mRNAs²⁵.

We assembled an unbalanced dataset sourced from several studies, including Weingarten-Gabbay et al.²⁶, IRESbase²⁷, IRESite²⁸, Rfam²⁹, and IRESpred¹⁴. This assembled dataset contains 46,774 sequences, with 37,602 sequences labeled as non-IRESs and 9,172 identified as IRESs. More detailed statistics can be found in Supplementary Materials A.5.

We adopted a contrastive learning approach to distinguish between IRES and non-IRES. First, we constructed contrastive samples where each sample consists of a pair of IRES and non-IRES sequences. For each sequence in the training set, we made one contrastive sample by randomly selecting another sequence with the opposite label to pair with it.

Although many IRESs are often found in the 5’ UTR of mRNA, they can also appear elsewhere. To address this, we transferred the UTR-LM model to fine-tune using the IRES labeled data. We started by using the U₁-trained version of UTR-LM as the initial base model. Then, we fine-tuned this model by minimizing a combination of three loss functions. The first one is the cross-entropy loss of the MN task (Eq.1). The second one is binary cross-entropy loss for IRES vs. non-IRES classification, as follows:

$$L_{classification}=-E_{x~X}(y_{x}log{\widehat{y}}_x+({1-y}_x\left)log\right(1-{\widehat{y}}_x\left)\right)$$ (4)

where $y_x$ and ${\widehat{y}}_x$ stand for the ground-truth label and the predicted probability of the label being IRES for the sequence $x$, respectively. The third one is a specially designed contrastive loss that measures the difference between the predicted probabilities for sequences labeled as non-IRES and sequences labeled as IRES. This loss function is computed over contrastive samples, i.e., pairs of an IRES and a non-IRES. It is given by

$$L_{contrastive}=E_{{(x}_i,x_j)~C}|{\widehat{y}}_{x_i}-{\widehat{y}}_{x_j}|,if{y}_{x_i}\ne y_{x_i}$$ (5)

where the pair ${(x}_i,x_j)$ consists of two sequences selected from the set of contrastive samples, denoted by $C$. Since the pair comes from contrastive samples, their ground-truth labels are not equal $y_{x_i}\ne y_{x_j}$. The variables ${\widehat{y}}_{x_i}$ and ${\widehat{y}}_{x_j}$ represent the predicted probabilities of the label being IRES for sequences $x_i$ and $x_j$, respectively.

Prediction of Relative Light Units (RLU)

The Relative Light Units (RLU) quantifies the level of luciferase activity, providing a direct measurement of how the designed 5’ UTRs affects the protein synthesis process. In our experiment, we designed a library of 211 unique 50-bp 5’ UTR sequences, and obtained the RLU by measuring the fold change of log2-transformed RLU relative to the well-established benchmark NCA-7d-5’UTR¹⁷.

We fine-tuned the UTR-LM for the in-house designed library, termed UTR-LM RLU. Furthermore, we explored the model generalization ability in zero-shot fitness prediction. For this, we applied the learned UTR-LM MRL and UTR-LM TE models to predict RLU, without any additional training or fine-tuning.

Implementation of Benchmarks

For comparisons, We re-trained benchmark models including Optimus⁶, FramePool⁵, MTtrans⁹, RNABERT¹², RNA-FM¹⁰, and Cao-RF⁷ using the same settings as used for training UTR-LM. Each model’s parameters and hyperparameters adhered to their respective GitHub implementations, such as learning rate and batch size, etc. For RNA-FM and RNABERT, the generated embeddings are of dimensions (sequence_length, 640) and (sequence_length, 120), respectively. These 2-D embeddings were averaged into 1-D (size 640 and 120) and then fed into an MLP downstream predictor.

Attention-based motif detection

Attention scores provide insights into the contribution of specific positions in the input sequence to the model’s prediction. Analysis of the attention scores of UTR-LM allows us to pinpoint important regions and potential motifs in 5’ UTRs. At the level of individual sequences, we examined the attention scores to find regions that highly contribute to the predictive result. At the position-level, we averaged attention scores from the same nucleotide positions across human sequences, revealing potential motif patterns. The computation workflow is given in Extended Fig. 1..

To do this, we adopted a methodology inspired by ESM³⁸. We generate an initial attention matrix for each sequence, which has dimensions (# layers × # heads, L, L), where # layers and # heads are hyper-parameters of the self-attention layer, and L is the sequence length. We first make the attention matrix symmetrical by taking the sum of it and its transpose. We then normalize the symmetrized matrix across the (L, L) dimension using the formula:

$$x_{normalized}=x-\frac{\left({\sum }_{i=1}^n{x}_{ij}\right)\left({\sum }_{j=1}^n{x}_{ij}\right)}{{\sum }_{i=1}^n{\sum }_{j=1}^n{x}_{ij}}\odot \mathbf{1}$$ (6)

Eq.6 describes how to perform element-wise subtraction to compute a normalized matrix $x_{normalized}$ from an input matrix $x$. Here, $\odot$ denotes element-wise multiplication, and 1 is a matrix of ones with dimensions matching $x$. This allows the average value to be broadcasted across all elements of $x$ during subtraction. Specifically, the average value is calculated by taking the product of the sums of the rows $i$ and columns $j$ of $x$ and dividing it by the sum of all elements in $x$. Next, we sum the normalized matrix across the # layers × # heads dimension, thereby reducing the dimensions to (L, L). Finally, we sum along the rows of the resulting (L, L) matrix to generate the final attention vector.

To analyze attention at the positional level, we generated attention vectors for 15,555 human sequences. Across these different sequences, we computed the average attention score for each nucleotide type at each specific position. This provides insight into how specific nucleotides at particular positions contribute to the overall sequence behavior.

Extended Data

Extended Fig. 1.

The computation flow of identifying patterns in 5’ UTR sequences based on attention scores.

Data Availability

The datasets are available and can be downloaded at https://codeocean.com/capsule/6711822³⁹. This link includes training data for the pre-trained model as well as datasets for various downstream tasks. Detailed statistics for these datasets are provided in Supplementary Materials A.